PREP)¶

Status: Resolved
Date: 2025-10-01
Severity: SEV-1 - Critical
Duration: ~90 minutes
Affected: Production and PREP environments (all users); DEV intermittent - APK/IPA upload
Owners: @enver

Summary¶

Production and PREP environments experienced a critical failure preventing APK and IPA uploads for approximately 90 minutes. Uploads initiated but never completed (hanging 30+ minutes). Root cause was the storage service returning 401 Unauthorized on retrieval, while the resource server and Kafka remained operational.

Timeline¶

All times UTC+3 (Istanbul). Relative to first report (T+0).

Time	Event
T-2h	Last successful APK upload test in PREP (F. Zehra).
T+0	Upload process reported hanging 30+ minutes (Osman Kibar); no APK upload possible to PROD.
T+5	Platform team confirms no known issues.
T+10	Enver begins investigation.
T+15	Resource server and Kafka operational; storage returning errors; 401 Unauthorized on GET identified.
T+45	PREP confirmed also affected (regression from 2h prior).
T+75	DEV shows intermittent success with hanging uploads.
T+85	IPA uploads also confirmed failing in PROD.
T+90	Storage credentials refreshed/rotated; connections re-established; resolution verified.

Root Cause¶

Storage service authentication/authorization failure prevented read access to uploaded files. Upload initiation was accepted, but subsequent verification/retrieval operations returned 401 Unauthorized due to invalid or expired credentials between the resource server and the storage backend. A synchronized deployment across environments is the suspected trigger for the credential/config issue.

Impact¶

Customers affected: All users on PROD and PREP (internal + customers); DEV intermittent
Services degraded: APK/IPA upload pipeline (storage retrieval); mobile deployment pipeline blocked
Data loss: No

Resolution¶

Emergency response team assembled; service health verified (resource server and Kafka healthy, storage failing). Storage service authentication credentials were refreshed/rotated and connections re-established. Upload testing across environments and file types (APK, IPA) confirmed recovery in PROD and PREP.

Action Items¶

#	Action	Owner	Due Date	Status
1	Add automated alerting for storage 401/auth failures before user reports	DevOps		Pending
2	Review credential lifetime/rotation between resource server and storage backend	Backend		Pending

Links¶

Linear issue(s): BE-7209 (PROD-2025-001)
Related systems: Resource Server, Apache Kafka, Storage Service (primary failure point), Upload UI