IR-0012: Testing Distribution Portal Instability - FWD¶
- Status: Resolved
- Date: 2026-01-12
- Severity: SEV-1 - Critical
- Duration: ~6 hours
- Affected: FWD (Cloud) - Testing Distribution portal (application list)
- Owners: @burako
Summary¶
The Testing Distribution portal intermittently failed to load the application list: roughly 1 in 2-3 page loads returned an error/empty screen instead of the app list. This is the same class of instability the customer had reported around the prior Christmas period.
Timeline¶
All times UTC+3 (Istanbul).
| Time | Date | Event |
|---|---|---|
| ~14:28 | Jan 10, 2026 | Customer first observes the portal intermittently not returning the app list. |
| 08:44 | Jan 12, 2026 | Customer reports via Slack; Appcircle reproduces on one of the shared links. |
| 08:51 | Jan 12, 2026 | Customer opens support request #1011. |
| 09:09 | Jan 12, 2026 | Backend engaged; investigating in the same thread. |
| 09:59 | Jan 12, 2026 | Issue resolved (manual restart of TesterWeb/StoreWeb services); customer told they may continue working. |
| 10:06 | Jan 12, 2026 | Customer requests an incident report. |
Root Cause¶
Under some circumstances all services restarted unexpectedly. TesterWeb and StoreWeb depend on the tester APIs and store APIs; on a simultaneous restart (only in cloud, where multiple StoreWeb/TesterWeb instances are active) the web services could not reach their APIs and generated random .NET data-protection keys, so they could not decrypt/trust each other's requests. Previously a Redis-based auto-repair mechanism (subscription + service close on detection) handled this, but Redis was no longer present on the StoreWeb/TesterWeb side, and an unexpected whole-system restart was not accounted for.
Impact¶
- Customers affected: FWD (Cloud); 1+ profile
- Services degraded: Testing Distribution portal application listing (intermittent empty/error state)
- Data loss: No
Resolution¶
- Immediate: manual restart of TesterWeb and StoreWeb services once their API dependencies were available.
- Permanent: patch applied in the 3.29.8 cloud release so TesterWeb and StoreWeb wait for dependent services to become healthy, retry, and self-restart if the dependency is still unreachable, repeating until dependencies are available.
Action Items¶
| # | Action | Owner | Due Date | Status |
|---|---|---|---|---|
| 1 | Ship dependency-aware wait/retry/self-restart for TesterWeb & StoreWeb (3.29.8) | Backend | 2026-01 | Done |
| 2 | Add monitoring for data-protection key mismatch / decrypt-trust failures between web and API tiers | DevOps | Pending | |
| 3 | Review unexpected whole-system restart trigger | Backend / DevOps | Pending |
Links¶
- Linear issue(s): BE-8097 (related: BE-8095)
- Support ticket: Zendesk #1011
- Fix release: 3.29.8