IR-0014: Large Build Artifact Upload Corruption on Chunk Retry - Fimple (Aytemiz) & Boehringer Ingelheim¶

Status: Draft - fix pending
Date: 2026-07-22
Severity: SEV-2 - High
Duration: Latent defect, triggered by transient network conditions; no bounded outage window
Affected: Fimple (Aytemiz Android release build) and Boehringer Ingelheim, both Appcircle Cloud - Export Build Artifacts step / BuildServer chunked upload / artifact object storage
Owners: @emre, @harun

Summary¶

Build artifacts larger than 100 MB are uploaded to BuildServer in 100 MB chunks. On 2026-07-22, two customers hit a server-side integrity failure on that step: when a chunk upload stalled and the client component retried it, the retried request sent an empty or partial body. The server overwrote the already-complete chunk object with the near-empty one, so the merged artifact was truncated (a 143 MB APK appeared as ~93 MB in storage; exact size not yet measured) and its SHA256 no longer matched the client's hash.

The build was marked Success + Warning while the endpoint returned HTTP 500, so a corrupted artifact reached storage in a build the customer saw as successful. The server-side integrity check was the only control that caught it; without that check a truncated APK would have been distributed.

Root cause is client-side: the chunk POST is wrapped in a retry block, but the file handle and the HTTP request object are built once outside it, so the retry re-serializes a request whose IO is already at EOF. The fix is a component version bump (no agent binary update, no server deploy), plus a server-side guard rejecting zero-byte or shrinking writes to an existing chunk object. As of 2026-07-28 the root cause is agreed but the fix is not yet implemented (BE-9062, Investigating).

Timeline¶

All times UTC.

Time	Date	Event
14:00-23:00	Jul 21, 2026	Network slowness confirmed on GitHub CDN access. Related but separate condition; it set the context for the following day.
~09:00	Jul 22, 2026	Customers report failing builds. Support ticket #1236 (Fimple). Boehringer Ingelheim reports the same symptom the same day. No general network problem detected on Jul 22.
-	Jul 22, 2026	Call with Fimple; the build log showing the corrupted artifacts is shared.
~15:17-17:15	Jul 22, 2026	Two-hour diagnostic session. Upload throughput measured across the platform; a comparison test isolates the bottleneck; the duplicated chunk log line is spotted and the truncation hypothesis is formed.
20:18	Jul 22, 2026	BE-9060 opened with the collected evidence, code walkthrough, and open questions - explicitly marking hypotheses as unproven.
20:25	Jul 22, 2026	Diagnosis recorded on BE-9060: the customer network is the trigger, not the cause; the defect is that a retried chunk truncates the merged file.
07:30	Jul 23, 2026	BE-9062 opened with the root cause: the retry re-sends an empty body because the request/IO is built outside the retry block. One BE-9060 hypothesis is corrected in the process.
11:18	Jul 24, 2026	BE-9062 moved to Investigating. Fix not yet implemented.
08:18	Jul 28, 2026	BE-9077 opened to produce this report.

Root Cause¶

Empty-body chunk retry overwrites a complete chunk¶

Artifacts above 100 MB are uploaded by the appcircle-export-build-artifacts component (v1.0.12 in the failing builds) in 100 MB chunks. Each chunk is POSTed to the BuildServer callback endpoint with an offset field and stored as a separate object named {QueueId}_{Name}_{Offset}. On completeUpload, the server lists those objects by prefix, parses each offset from the object name, merges them into the final artifact, computes its SHA256, and compares it against the hash the client sends.

The chunk POST is wrapped in a retry block, but the file handle and the Net::HTTP::Post object are created once outside that block; only the send call is inside it. On the first attempt the IO is read to EOF. When the attempt fails and the retry fires, the same request object is re-serialized from an IO that is already at EOF, so the retry sends an empty or partial body.

The server treats that retry as a normal write to the same object name and overwrites the already-complete chunk with the near-empty one. Nothing rejects it: the write path has no duplicate detection and no size consistency check. The subsequent merge therefore produces an artifact missing an entire chunk, and the SHA256 comparison fails with BuildServer.FileHash.ValidationFailed.

Evidence. In both failing files, the chunk that was retried is exactly the chunk that went missing from the merged object - the signed APK lost chunk 2 at offset 100000000, the other APK lost chunk 1 at offset 0. In the failing builds one uploading... log line is printed exactly twice; in every clean file in the same builds, no line repeats. The merged object appeared to be ~93 MB against a 143 MB source - inspected at a glance, exact size not yet measured - i.e. the file shrank rather than grew, which rules out double-append and matches an overwrite with a short body.

Trigger, not cause. The customer network is the trigger: a transient stall causes the retry. The defect is that our upload path does not handle that retry safely. Runner egress was measured at 10 Gbit during the investigation, so a client bandwidth ceiling is not part of the mechanism.

Confidence. High, based on the correlation above. Two verifications remain open: (a) the size and SHA256 of the corrupted object in storage, measured against the local file, which would confirm the truncation and its magnitude; (b) the fileSize the server recorded on the second chunk response for the retried chunk of that QueueId - if it is zero or very small, the empty-body mechanism is confirmed end to end. Until then this section describes a strongly evidenced mechanism, not a directly observed empty request.

Contributing factors¶

Chunk writes are unvalidated. The chunk object name carries only the offset, not the chunk size, and the write path performs no duplicate, size, or shrink check. A retry was assumed to be idempotent because it targets the same offset - which holds only if the retry actually re-sends the same bytes.
No authoritative total size. The client sends the size of each chunk, and completeUpload carries only the file list and per-file hashes. The server therefore has no expected total size to validate the merged object against; the per-file SHA256 is the only integrity signal, and it can only report failure after the corrupted object has already been written.
Success + Warning combined with HTTP 500. When validation fails, the build is marked Success + Warning but the endpoint returns HTTP 500. Two consequences: the component retries completeUpload (~68 s wasted, no-op'd by the already-completed guard), and the customer sees a successful build whose artifact is corrupt.
The logs cannot show the failure. The upload log prints the size of the temp chunk file, not the number of bytes actually sent, so an empty retry body is invisible in the log. Root cause had to be inferred from a repeated log line rather than read directly - which is why diagnosis took a two-hour session and a further day.
Ingest throughput (separate finding, noted here as it widens the exposure window). During the investigation the same ~350 MB file was uploaded three ways from the same runner: ~4-6 s to Cloudflare, ~6-12 s to Google Cloud Storage via cache-push, and ~38-40 s through export-build-artifacts to BuildServer - 3-5x slower. Runner egress was ruled out, so the bottleneck is on our ingest path (nginx buffering, Kubernetes ingress, or the BuildServer request path). A longer upload window means more opportunity for a stall, and therefore for the retry that triggers this defect. Not a root cause and not part of the fix; tracked separately in PL-366.

Impact¶

Customers affected: Fimple (Aytemiz) and Boehringer Ingelheim, both on Appcircle Cloud, confirmed 2026-07-22. By mechanism, any build uploading a >100 MB artifact while the network stalls is affected, on Cloud and self-hosted alike - the faulty component is cloned fresh on every build regardless of deployment model.
Services degraded: Export Build Artifacts step returns HTTP 500 on integrity validation; the build itself is marked Success + Warning and completes.
Data integrity: Impacted. The merged artifact in object storage was truncated (143 MB appeared as ~93 MB; exact size not yet measured) and did not match the artifact produced by the build. The corrupted object was retained and made available as a build artifact.
Data loss: No. No source code, build input, or customer content was lost; the artifact is reproducible by re-running the build.
Data confidentiality: No impact.
Downstream risk: A truncated APK is not installable and would fail signature verification. Had it been picked up by distribution or store submission, the corruption would have surfaced at the customer or the store instead of in our pipeline. In the reported cases the integrity check stopped it before that point.
Duration of impact: Latent. The defect is triggered by a chunk retry, so exposure tracks network conditions rather than a fixed window. Both reported occurrences were on 2026-07-22; two builds 19 minutes apart on the same machine behaved differently (one failed, one succeeded), confirming the intermittent nature.

Resolution¶

Not resolved as of 2026-07-28. The root cause is identified and the fix is scoped in BE-9062 (Investigating), but no change has been implemented or shipped. The scoped work is:

Primary fix - export-build-artifacts component. Re-open the file and rebuild the request object inside the retry block on every attempt, so a retry sends the full chunk instead of an empty body. This closes the root cause directly. The component is cloned fresh on every build and versioned separately from the agent binary, so it ships as a component version bump - no agent update and no server deploy.
Defense-in-depth - BuildServer. Reject a zero-byte or shrinking write to an existing chunk object, so no client's faulty retry can overwrite a complete chunk. Backward-compatible.
Cleanup - BuildServer. Return HTTP 200 in the accepted-but-warning case instead of 500, removing the pointless completeUpload retry.

Verification plan. Force a chunk retry by injecting a transient failure and confirm the retried chunk arrives complete; upload multi-chunk artifacts repeatedly and confirm the server hash matches the client hash; confirm the server guard rejects a zero-byte or short write to an existing chunk. A reproduction attempt during the diagnostic session, based on interrupting the runner's network mid-upload, did not succeed - the mechanism was established from log correlation instead, so a working reproduction is still needed to validate the fix.

Customer-facing workaround (active). Do not use the artifact of a build that completed with a warning; re-run the build. The integrity check makes the corruption visible, so an affected build is identifiable.

Action Items¶

#	Action	Owner	Tracked in	Status
1	Ship the fix: component retry rebuild, BuildServer zero-byte/shrinking write guard, HTTP 200 on accepted-but-warning, and a working reproduction to validate it (see Resolution for the breakdown)	@harun	BE-9062	Investigating
2	Close the two open verifications: size and SHA256 of the corrupted object against the local file, and the server-recorded `fileSize` on the retried chunk's second response	@emre	BE-9060 / BE-9062	Pending
3	Investigate the 3-5x ingest throughput gap on the runner → build server path (Ingress / Nginx / Load Balancer limits and buffering)	@osmank	PL-366	Triage
4	Log the bytes actually sent per chunk (client) and received (server), so an empty retry is visible in the log instead of inferred	@emre	Not tracked	Pending - task needed
5	Decide how a build whose artifact failed integrity validation should be surfaced - today it is Success + Warning with the corrupted object retained and offered as a build artifact	@emre	Not tracked	Pending - decision needed

Links¶

Linear issue(s): BE-9060 (investigation), BE-9062 (fix), PL-366 (ingest throughput investigation), BE-9077 (this report)
Support ticket: Zendesk #1236 - Fimple
Related PR(s): none yet
Related IRs: IR-0001, IR-0002 - earlier cloud runner network degradation incidents; context for the network conditions on 2026-07-21
Post-mortem meeting notes: diagnostic session, 2026-07-22 (internal recording and transcript)