IR-0002: Appcircle Cloud Runner Network Instability – GitHub CDN Download Speed Degradation¶
| Field | Detail |
|---|---|
| Document ID | IR-2026-06-09-PL197 |
| Classification | Confidential – Internal Use Only |
| Version | 1.1 |
| Date Issued | 2026-06-10 |
| Prepared By | arc@appcircle.io |
| Reviewed By | osmank@appcircle.io |
| Approved By | osman@appcircle.io |
1. Incident Overview¶
| Field | Detail |
|---|---|
| Incident Date | 2026-06-09 |
| Detection Date | 2026-06-09 |
| Resolution Date | 2026-06-10 |
| Duration | ~16:53 – ~02:28 UTC+3 (approximately 9.5 hours) |
| Affected Systems | Appcircle Cloud Runners (macOS hosts) |
| Severity | SEV2 |
| Status | Resolved |
| Incident Reference | PL-197 |
2. Executive Summary¶
On 2026-06-09, multiple Appcircle Cloud customers experienced intermittent build failures caused by severe download speed degradation from GitHub and its CDN infrastructure. Affected builds failed at various stages including git clone, CocoaPods dependency resolution, Gradle dependency downloads, and npm installs. The issue self-resolved overnight with no active fix applied; a definitive root cause has not been established.
Customers affected: 7 (Idefix, HangiKredi, Logiwa, KKB, AmericanExpress/GBT, Taxifoni, Terabank)
3. Impact Assessment¶
3.1 Service Impact¶
| Dimension | Assessment |
|---|---|
| Service Availability | Degraded – build pipelines dependent on external network fetches failed intermittently |
| Affected Feature | All build steps relying on GitHub (git clone, Releases), githubusercontent CDN, CocoaPods CDN, Gradle Services, npm registry |
| Data Integrity | No impact |
| Data Confidentiality | No impact |
| Affected Customers | 7 cloud customers |
| Duration of Impact | ~16:53 – ~02:28 UTC+3 (~9.5 hours) |
3.2 Business Impact¶
- 7 enterprise customers experienced intermittent build failures for approximately 9.5 hours during evening and nighttime hours.
- Builds failed non-deterministically across multiple pipeline types (Android, iOS, npm), increasing diagnosis difficulty.
- Customer notification was issued approximately 6 hours after the first report (16:53 → 22:59), exceeding acceptable response time for a SEV2 incident.
- No active fix was available; resolution depended entirely on the external issue self-resolving.
4. Timeline¶
All times UTC+3 (Istanbul).
| Time | Date | Event |
|---|---|---|
| 16:53 | Jun 9, 2026 | First customer-reported runner network issues received; incident thread opened in #eng-general |
| 16:54 | Jun 9, 2026 | İlhan Cengiz and Burak Öztopuz engaged |
| 17:14 | Jun 9, 2026 | Infrastructure investigation begins; DNS/HTTPS access checks from prod mac hosts show github.com, cdn.cocoapods.org, services.gradle.org reachable — no broad issue detected |
| 17:30 | Jun 9, 2026 | No specific blocked addresses identified; customer logs and affected IP addresses requested |
| 18:52 | Jun 9, 2026 | Suspicious external IP pairs shared from admin panel logs (77.92.x.x egress range) |
| 18:53 | Jun 9, 2026 | Build logs from 5 failing builds shared (HangiKredi Android/iOS, GBT iOS, Taxifoni Android, Terabank Android) |
| 19:51 | Jun 9, 2026 | 2 additional customers report access problems; total confirmed affected customers rises |
| 22:40 | Jun 9, 2026 | Osman Kibar engaged for deeper network investigation |
| 22:44 | Jun 9, 2026 | Pattern identified as potentially consistent with Turkey-egress IP blocking (similar to IR-0001) |
| 22:46 | Jun 9, 2026 | GitHub DNS checked from runner hosts; single IP returned (140.82.121.4) — no obvious multi-IP block scenario |
| 22:56 | Jun 9, 2026 | İlhan begins scanning full GitHub IP range list from https://api.github.com/meta for firewall-blocked entries |
| 22:57 | Jun 9, 2026 | Joint investigation session started (Osman Kibar, İlhan Cengiz, B. Berk Keskin) |
| 22:59 | Jun 9, 2026 | Customer notifications sent by Burak Öztopuz: community announcement channel, all enterprise customer channels, cloud support channel |
| 23:08 | Jun 9, 2026 | All enterprise channel notifications confirmed delivered |
| 23:59 | Jun 9, 2026 | IP block scan finding: some GitHub IP ranges unreachable, but same ranges also unreachable from local machines and non-Teknotel hosts — issue not isolated to Appcircle infrastructure |
| 01:21 | Jun 10, 2026 | Full GitHub IP block scan completes clean; results consistent with Netherlands server — no firewall blocking confirmed |
| 02:58 | Jun 10, 2026 | Root cause hypothesis updated: severe and inconsistent download speed degradation from githubusercontent CDN observed (speeds to IP 185.199.111.133 ranged from ~80 KB/s to ~4.6 MB/s across consecutive requests). Issue began self-resolving approximately 30 minutes prior |
| ~02:28 | Jun 10, 2026 | Issue begins resolving |
| 08:57 | Jun 10, 2026 | Morning check confirms situation stable; Çelik briefed |
5. Trigger & Root Cause¶
Trigger: Customer builds started timing out or failing on network-dependent steps (git clone, dependency resolution) due to extreme download speed degradation from external services.
Root cause: Not definitively determined. Observed evidence points to a performance degradation on GitHub's CDN infrastructure (githubusercontent CDN, IP 185.199.111.133) — download speeds fluctuated between ~80 KB/s and ~4.6 MB/s to the same IP address across consecutive requests. The same degradation was observable from non-Teknotel hosts, ruling out a Teknotel firewall or routing issue specific to Appcircle infrastructure. No IP blocks were found. The issue self-resolved; whether the cause was GitHub-side rate limiting, upstream transit congestion, or peering degradation is unknown.
Most likely hypotheses (ranked by evidence):
- GitHub CDN-side throughput degradation — speed fluctuation to the same IP across consecutive requests is the strongest signal; consistent with a CDN-side issue rather than a network-path issue.
- Upstream transit / peering degradation (Turkey → GitHub) — degradation was also observable from non-Teknotel hosts, suggesting a broader Turkey-egress routing issue cannot be fully excluded.
- Teknotel routing change — ruled out as primary cause; no IP blocks found and issue was reproducible outside Teknotel infrastructure.
6. Containment & Resolution¶
No active fix was applied — the issue self-resolved overnight. During the incident:
- Investigated and ruled out Teknotel firewall IP blocking.
- Investigated and ruled out DNS resolution failures.
- Scanned full GitHub IP range list for blocked entries on port 443 — all came back clean:
| CIDR | IP Count |
|---|---|
| 192.30.252.0/22 | 1,024 |
| 140.82.112.0/20 | 4,096 |
| 143.55.64.0/20 | 4,096 |
| 192.30.255.164/31 | 2 |
- Sent customer notifications via enterprise channels, community announcement, and cloud support channel at 22:59.
7. Corrective & Preventive Actions (CAPA)¶
| ID | Action | Detail | Owner | Due Date | Status |
|---|---|---|---|---|---|
| CAPA-01 | Implement download speed monitoring on runner hosts | Set up continuous synthetic monitoring for key external endpoints (github.com, githubusercontent.com, cdn.cocoapods.org, services.gradle.org, registry.npmjs.org) with alerting thresholds. Tracked under PL-112. | Platform Team | TBD | Backlog |
| CAPA-02 | Investigate dependency caching / mirror proxy | Evaluate options for caching or proxying high-traffic external resources (CocoaPods, Gradle, npm) to reduce blast radius of external CDN degradation. Tracked under BE-1705. | Platform | TBD | Backlog |
| CAPA-03 | Evaluate foreign egress + NAT pool for runner outbound traffic | Assess routing runner internet egress through an overseas node using an IP pool to bypass BTK-blocked IPs and distribute rate-limit risk. Note: does not resolve GitHub-side throughput limits; operational overhead must be evaluated. | Platform Team | TBD | Under Evaluation |
| CAPA-04 | Evaluate DNS-layer mitigation for BTK-blocked IPs | Implement custom DNS infrastructure that avoids resolving to known-blocked addresses or redirects to unblocked alternates. | Platform Team | TBD | Under Evaluation |
| CAPA-05 | Define runbook for external dependency slowness incidents + formal SEV2 notification SLA | Document detection criteria, escalation path, customer communication trigger, and retry/workaround guidance. Establish a formal SLA for customer-facing notification time during SEV2 incidents — this incident had a ~6 hour gap between first report (16:53) and customer notification (22:59). Tracked under PL-13. | Engineering Manager | TBD | Backlog |
8. Lessons Learned¶
What Worked Well¶
- Timeline reconstruction was thorough; investigation logs were well-preserved in Slack threads.
- Teknotel firewall and DNS failure hypotheses were systematically ruled out through comparative scanning, avoiding premature conclusions.
- Customer notification, once triggered, was executed efficiently across all channels simultaneously.
- Morning stability check and stakeholder briefing (Çelik) were handled promptly.
Areas for Improvement¶
- Detection relied entirely on customer reports. The first 6 hours of investigation produced no definitive finding. Per-IP and per-endpoint download speed monitoring (CAPA-01) would have surfaced anomalies before builds started failing.
- Customer notification took ~6 hours. A pre-defined SEV2 communication SLA and templated notifications (CAPA-05) would have reduced this significantly — even without a confirmed root cause, earlier acknowledgment is expected.
- No active mitigation was available. Unlike IR-0001 where a
/etc/hostsworkaround was deployable, this incident had no customer-side or platform-side workaround. Dependency caching/proxy (CAPA-02) would reduce exposure to this class of failure.
Would IR-0001 CAPA Actions Have Prevented This Incident?¶
IR-0001 (February 2026) identified the same infrastructure layer as vulnerable and produced 12 CAPA items — most of which remain in Draft/Backlog status as of this writing.
| IR-0001 CAPA | Relevant to this incident? | Would it have helped? |
|---|---|---|
| CAPA-08: Per-IP connectivity monitoring | Yes | Yes — would have detected download speed degradation within minutes rather than relying on customer reports |
| CAPA-05: Internal DNS resolver with override capability | Partially | Partially — root cause was not a hard IP block, so DNS override would not have prevented failures; but the infrastructure would have been in place for faster response |
| CAPA-06: Gradle bootstrap retry wrapper | Yes | Yes — retry logic on download timeout would have reduced build failure rate during degraded CDN periods |
| CAPA-07: Gradle distribution mirror | Yes | Yes — if Gradle distributions were served from Appcircle infrastructure, CDN degradation would have had zero impact on Android builds |
| CAPA-11: ISP escalation runbook | Partially | Partially — Teknotel was ruled out, but the investigation spent significant time on that hypothesis; a runbook would have accelerated that ruling-out process |
| CAPA-12: Customer communication templates | Yes | Yes — would have reduced the 6-hour notification gap |
Conclusion: Had the highest-priority IR-0001 CAPA items (CAPA-06, CAPA-07, CAPA-08) been implemented, the customer impact of this incident would have been significantly reduced or eliminated. The recurrence of a GitHub CDN-related incident within 4 months of IR-0001 makes the implementation of these items urgent.
9. Data & Privacy Impact Assessment¶
| Criterion | Assessment |
|---|---|
| Personal Data Breach | No — incident was limited to network connectivity degradation; no data was accessed, leaked, or modified |
| Data Loss | No — no build artifacts, source code, or customer data was lost |
| Security Breach | No — no unauthorized access; root cause was external CDN performance degradation |
| Service Availability | Degraded — cloud runner builds dependent on external network fetches failed intermittently for approximately 9.5 hours |
| Regulatory Notification Required (GDPR / KVKK) | No — no personal data breach or data loss occurred |
| Third-Party Impact | GitHub CDN / githubusercontent CDN exhibited the underlying performance degradation; Teknotel infrastructure was investigated and ruled out |
Conclusion: A service-availability degradation affecting cloud build pipelines; not a security incident or data breach under SOC 2 / ISO 27001.
10. Communication Log¶
| Date | Time (UTC+3) | Channel | Audience | Summary |
|---|---|---|---|---|
| 2026-06-09 | 16:53 | Slack – #eng-general | Engineering Team | Incident thread opened; first customer report received |
| 2026-06-09 | 22:59 | Slack – community announcement channel | All Appcircle community | Service notice: runner network connectivity issues affecting builds |
| 2026-06-09 | 22:59 | Slack – enterprise customer channels (x7) | Enterprise customers | Service notice: runner network connectivity issues |
| 2026-06-09 | 22:59 | Slack – cloud support channel | Cloud customers | Service notice: runner network connectivity issues |
| 2026-06-10 | 08:57 | Internal | Çelik | Morning stability briefing |
11. Approval & Sign-Off¶
| Role | Name | Date |
|---|---|---|
| Prepared By | arc@appcircle.io | 2026-06-10 |
| Reviewed By | osmank@appcircle.io | 2026-06-11 |
| Approved By | osman@appcircle.io |
Document Control¶
| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0 | 2026-06-10 | arc@appcircle.io | Initial release |
| 1.1 | 2026-06-11 | osmank@appcircle.io | Added classification, numbered sections, Lessons Learned with IR-0001 CAPA analysis, CAPA table with owners, root cause hypothesis ranking, business impact detail, Document Control |
Classification
This document is Confidential and intended for internal use and authorized distribution only.