IR-0007: Appcircle CLI 503 Upload Failure - Marriott¶
- Status: Resolved
- Date: 2025-09-03
- Severity: SEV-1 - Critical
- Duration: ~2.5 days (2025-09-03 to 2025-09-08)
- Affected: Marriott (Cloud) - Testing Distribution upload via Appcircle CLI in their CI/CD
- Owners: @buraky, @burako
Summary¶
The customer received HTTP 503 Service Unavailable errors when uploading binaries to Testing Distribution via Appcircle CLI 2.3.2 from their CI environment. This blocked all CI/CD uploads for the Marriott Bonvoy app for an extended period. The error originated on the Google Cloud Storage side (Appcircle server errors return 502), reached during upload through a pre-signed URL.
Timeline¶
All times UTC+3 (Istanbul) unless noted.
| Time | Date | Event |
|---|---|---|
| ~20:00 EST | Sep 3, 2025 | Marriott reports all CLI uploads failing with 503, blocking CI/CD. |
| 23:38 | Sep 3, 2025 | Ticket #758 opened with a 503 screenshot. |
| 08:51 | Sep 4, 2025 | Burak Öztopuz alerts internal team; backend begins investigation. |
| 11:27 | Sep 4, 2025 | Issue not reproducible internally; Google connectivity suspected; CLI lacks logging. |
| 16:20 | Sep 4, 2025 | Customer confirms CLI 2.3.2; works locally, fails in CI/CD, all repos affected. |
| 21:49 | Sep 4, 2025 | Version gap identified: customer on 2.3.2, latest 2.7.3. |
| ~night | Sep 4, 2025 | Customer tests 2.7.3: "socket hang up" instead of 503. |
| 10:13 | Sep 5, 2025 | Confirmed issue is specific to the customer's CI runners. |
| 12:40 | Sep 5, 2025 | Burak Yıldırım identifies root cause; implements Google retry strategy in CLI 2.7.4. |
| 14:51 | Sep 5, 2025 | CLI 2.7.4 provided to customer. |
| 16:52 | Sep 5, 2025 | Customer confirms fix works on first server; rolls out to remaining servers. |
| 16:26 | Sep 8, 2025 | Customer confirms complete resolution; requests ticket closure. |
Root Cause¶
The pre-signed URL used for the upload to Google Cloud Storage became non-functional due to a transient communication issue between the customer's CI environment and Google services. Google classifies this class of 5xx errors as transient and recommends a retry strategy. Contributing factor: the Appcircle CLI had not implemented Google's recommended retry strategy, so a transient error surfaced directly to the customer as a hard failure.
Impact¶
- Customers affected: Marriott (Cloud)
- Services degraded: Testing Distribution upload via Appcircle CLI from customer CI/CD
- Data loss: No
Resolution¶
Implemented Google's recommended retry strategy for transient GCS errors in the Appcircle CLI, released as CLI 2.7.4. No direct fix was possible for the underlying network/access path because the failing request did not traverse Appcircle servers, but the retry strategy resolved the customer impact.
Action Items¶
| # | Action | Owner | Due Date | Status |
|---|---|---|---|---|
| 1 | Adopt Google retry policy in Appcircle CLI (released in 2.7.4) | @buraky | 2025-09-05 | Done |
| 2 | Add upload diagnostics/logging to CLI for faster triage of CI-side failures | Backend | Pending |
Links¶
- Linear issue(s): BE-7060
- Support ticket: Zendesk #758
- Release: appcircle-cli v2.7.4
- Reference: https://cloud.google.com/storage/docs/retry-strategy