IR-0001: Cloud Runner Network Connectivity – GitHub CDN IP Block (IR-2026-0224-BE8351)¶
| Field | Detail |
|---|---|
| Document ID | IR-2026-0224-BE8351 |
| Classification | Confidential – Internal Use Only |
| Version | 1.2 |
| Date Issued | 2026-02-24 |
| Prepared By | cagkan@appcircle.io |
| Reviewed By | osmank@appcircle.io |
| Approved By | osman@appcircle.io |
1. Incident Overview¶
| Field | Detail |
|---|---|
| Incident Date | 2026-02-24 |
| Detection Date | 2026-02-24 |
| Resolution Date | 2026-02-24 |
| Duration | ~5 hours (active investigation); ~7 hours including ISP coordination and customer follow-up |
| Affected Systems | Appcircle Cloud Runners, Self-Hosted Runners (Anadolu Sigorta) |
| Deployment Model | Cloud + Self-Hosted |
| Severity | SEV-1 – Critical |
| Status | Resolved |
| Incident Reference | PL-202 |
2. Executive Summary¶
On February 24, 2026, starting at 11:15 (UTC+3), Appcircle build runners began experiencing Android build failures. Affected builds either timed out or remained in a Waiting state for an extended period. The issue was first observed on macOS unit test components; Anadolu Sigorta (self-hosted) was identified as an affected customer within minutes.
Investigation revealed that one of four IP addresses associated with release-assets.githubusercontent.com — specifically 185.199.108.133 — had become unreachable from all Appcircle runners hosted in the Teknotel data center. Because Gradle resolves the download host via DNS round-robin across four IPs, requests that resolved to the blocked IP timed out, causing Gradle distribution downloads to fail non-deterministically across all Android builds.
A support ticket was opened with Teknotel at approximately 13:51. Teknotel responded at 15:27, confirming that 185.199.108.133 had been automatically blocked by their internet traffic security systems due to detected SSH and Telnet brute-force attack traffic originating from that IP. Teknotel's response referenced the IP owner as a "US-based cybersecurity firm." In practice, 185.199.108.133 is a GitHub CDN endpoint, part of the 185.199.108.0/22 prefix used by GitHub's content delivery infrastructure. The discrepancy between Teknotel's description and the IP's actual ownership suggests the block decision was made based on automated threat intelligence data — likely a shared threat feed — without verification of the IP's actual function in context. Notably, this IP range had previously been communicated to Teknotel as part of an allowlist for Appcircle's runner infrastructure, yet the automated block was applied regardless.
A /etc/hosts workaround was deployed to affected customers at 11:42, and connectivity was fully restored at approximately 16:20 following Teknotel's removal of the block. Incident was formally confirmed resolved at 17:37, with customer notification issued at 17:45.
3. Impact Assessment¶
3.1 Service Impact¶
| Dimension | Assessment |
|---|---|
| Service Availability | Degraded – Android builds using Gradle wrapper could not complete |
| Affected Feature | Android build pipeline (Gradle distribution download via release-assets.githubusercontent.com) |
| Data Integrity | No impact |
| Data Confidentiality | No impact |
| Affected Users | All Appcircle Cloud users; Self-hosted deployment at Anadolu Sigorta |
| Duration of Impact | 11:15–16:20 (UTC+3), approximately 5 hours |
3.2 Business Impact¶
- All Android builds on the platform were non-functional for the duration of the incident.
- Customers experienced prolonged build timeouts without clear error messages pointing to a network issue, increasing confusion.
- Anadolu Sigorta (self-hosted on Teknotel infrastructure) was directly impacted and contacted Appcircle support.
- A customer ("Hani Kredi") called Appcircle directly after receiving no clear status update.
- A workaround requiring manual workflow modification was necessary to unblock urgent customer pipelines.
- Resolution time from public customer notification to confirmed fix was approximately 4 hours and 49 minutes (12:48 → 17:37).
3.3 Platform Impact¶
- Identified a monitoring gap: partial IP-level connectivity failures for external dependencies were not detected by existing alerting.
- Identified a resilience gap: Gradle download retries do not account for DNS-level round-robin resulting in silent routing failures at the TCP connection layer.
4. Root Cause Analysis¶
4.1 Primary Cause¶
Teknotel's automated internet traffic security systems blocked the IP address 185.199.108.133 in both inbound and outbound directions. Per Teknotel's support response, the block was triggered by SSH and Telnet brute-force attack traffic detected from that IP targeting Teknotel's infrastructure and customers.
The IP 185.199.108.133 is part of GitHub's CDN infrastructure (185.199.108.0/22), used to serve content at release-assets.githubusercontent.com. Teknotel's own response described the IP owner as a "US-based cybersecurity firm," which indicates that the blocking decision was driven by a threat intelligence feed or WHOIS classification rather than direct traffic analysis in context. It is plausible that the underlying cause was not an actual attack from GitHub's IP, but rather that GitHub's CDN IP — which services very high request volumes globally — was flagged by an automated threat intelligence system, possibly due to traffic pattern heuristics, and that Teknotel applied the block without verifying whether the IP was serving legitimate HTTP/HTTPS traffic rather than SSH/Telnet. This represents a limitation of undiscriminating automated ISP-level blocking.
Critically, this IP range had previously been communicated to Teknotel as part of an allowlist for Appcircle's infrastructure. The fact that the automated block was applied regardless indicates that Teknotel's security automation does not consult customer-submitted allowlists before applying blocks, or that the allowlist was not configured at the appropriate layer in their systems.
This is a third-party ISP-level network control action, not a failure within Appcircle's systems.
4.2 Contributing Factor – DNS Round-Robin Without TCP Retry Fallback¶
release-assets.githubusercontent.com resolves to four IP addresses via DNS round-robin:
185.199.108.133 ← Blocked by Teknotel
185.199.109.133 ← Reachable
185.199.110.133 ← Reachable
185.199.111.133 ← Reachable
When a build agent resolved the hostname to the blocked IP, the TCP connection attempt to port 443 timed out (confirmed via nc -vz 185.199.108.133 443). The Gradle wrapper does not automatically retry with an alternative IP upon TCP connection timeout; it treats the failure as a download error and hangs until the timeout threshold is reached. This made the failure non-deterministic (roughly 1 in 4 build attempts would hit the blocked IP) and initially difficult to reproduce consistently.
4.3 Contributing Factor – No Monitoring for Partial Upstream Connectivity¶
Appcircle's monitoring infrastructure did not have checks in place to detect IP-level partial reachability for external download dependencies. The failure manifested as slow or hung builds rather than an explicit network error, which increased time-to-detection and complicated initial triage.
4.4 Traceroute Evidence¶
Traceroute from the Teknotel-hosted runner confirmed traffic was dropped at the Teknotel network edge:
traceroute to 185.199.108.133 (185.199.108.133), 64 hops max, 40 byte packets
1 host-77-92-124-1.reverse.teknotel.com (77.92.124.1) 0.895 ms
2 77.92.96.45 (77.92.96.45) 0.369 ms
3 mx304-06-ae2-qfx5k-06.teknotel.com (77.92.108.213) 0.768 ms
4 * * * ← Traffic dropped at Teknotel edge
Comparative traceroute to a reachable IP in the same subnet:
traceroute to 185.199.109.133 (185.199.109.133), 64 hops max, 40 byte packets
1 host-77-92-124-1.reverse.teknotel.com (77.92.124.1) 0.680 ms
2 77.92.96.45 (77.92.96.45) 0.376 ms
3 mx304-06-ae2-qfx5k-06.teknotel.com (77.92.108.213) 1.790 ms
4 10.155.100.14 (10.155.100.14) 0.818 ms
5 5.253.168.30 (5.253.168.30) 2.261 ms
6 et252-8.rt.tlp.sof.bg.retn.net (87.245.232.204) 10.148 ms
5. Timeline¶
All times are UTC+3 (Istanbul).
| Time | Date | Event |
|---|---|---|
| 11:15 | Feb 24, 2026 | First report by Doruk Batur: macOS unit test component throwing socket exception during build. Android build failing; Linux build unaffected. |
| 11:26 | Feb 24, 2026 | Burak Berk Keskin identifies Anadolu Sigorta (self-hosted) as a second affected party with the same symptoms. |
| 11:40 | Feb 24, 2026 | Root cause isolated: 185.199.108.133 unreachable from all Teknotel-hosted runners. Three other IPs confirmed reachable. |
| 11:42 | Feb 24, 2026 | Workaround identified and shared internally: pin release-assets.githubusercontent.com to a working IP via /etc/hosts. |
| 11:46 | Feb 24, 2026 | Berk Keskin flags that cloud customers are also at risk. Zehra and Burak Öztopuz notified. |
| 11:52 | Feb 24, 2026 | Scope confirmed: all customers with Android builds using Gradle may be affected, not only unit test component users. |
| 12:01 | Feb 24, 2026 | Osman Kibar initiates internal call with Berk Keskin and İlhan Cengiz to assess resolution path. |
| 12:16 | Feb 24, 2026 | Osman determines issue will take time to resolve; requests customer-facing notification. |
| 12:48 | Feb 24, 2026 | F. Zehra sends customer notification: network-related issue causing Android builds to remain in Waiting state; team is actively working on resolution. |
| 13:00 | Feb 24, 2026 | All prod, dev, prep, and build machines confirmed unable to reach 185.199.108.133. |
| 13:51 | Feb 24, 2026 | Osman Kibar opens support ticket with Teknotel, providing traceroute evidence and comparative analysis. |
| 15:27 | Feb 24, 2026 | Teknotel responds: confirms 185.199.108.133 was automatically blocked due to detected SSH/Telnet brute-force attack traffic. Teknotel describes IP owner as "US-based cybersecurity firm." |
| ~16:00 | Feb 24, 2026 | Appcircle responds to Teknotel: IP belongs to GitHub's CDN infrastructure, used exclusively for downloading build dependencies. No attack traffic originated from Appcircle systems. |
| 16:20 | Feb 24, 2026 | Connectivity to 185.199.108.133 restored. Berk Keskin confirms from a runner. |
| 16:22 | Feb 24, 2026 | İlhan Cengiz confirms all four IPs reachable from all prod machines. |
| 17:37 | Feb 24, 2026 | Confirmed: builds succeeding normally. Incident considered resolved. |
| 17:45 | Feb 24, 2026 | Customer resolution notification sent by F. Zehra. |
| 19:14 | Feb 24, 2026 | Final confirmation by İlhan Cengiz: all four IPs reachable across all machines. Incident formally closed. |
6. Containment & Resolution¶
6.1 Immediate Workaround¶
A host entry workaround was made available to affected customers at 11:42 to force DNS resolution to a known-good IP, bypassing the blocked address. Anadolu Sigorta applied this at approximately 12:00 and confirmed it resolved their builds.
Steps:
- Open the affected workflow in the Appcircle editor.
- Add a Custom Script step (type: Bash) before the Android build step.
- Insert the following script:
echo "185.199.109.133 release-assets.githubusercontent.com" | sudo tee -a /etc/hosts
This pins release-assets.githubusercontent.com to 185.199.109.133 on the runner, ensuring Gradle downloads are routed away from the blocked IP.
6.2 Resolution¶
- Appcircle opened a support ticket with Teknotel, providing traceroute evidence demonstrating the block at Teknotel's network edge.
- Appcircle clarified to Teknotel that
185.199.108.133is a GitHub CDN IP used for downloading build dependencies; no malicious activity originated from Appcircle's systems. - Teknotel removed the block at approximately 16:20.
- Normal connectivity to all four GitHub CDN IPs was restored.
- Android builds across all affected deployments resumed without requiring the workaround.
7. Corrective & Preventive Actions (CAPA)¶
This section outlines actions across four defense layers: ISP relationship, platform resilience, monitoring & detection, and operational response.
Layer 1 — ISP Relationship & Contract Controls¶
| ID | Action | Detail | Owner | Status |
|---|---|---|---|---|
| CAPA-01 | Formal written notice to Teknotel | Document: (1) SEV-1 outage caused; (2) blocked IP was on submitted allowlist; (3) demand automated blocks are not applied to allowlist entries without prior notification; (4) request SLA commitments around network change notifications. | Operations / Engineering Manager | Draft |
| CAPA-02 | Submit comprehensive IP allowlist with documented business justification | Compile full list of external IP ranges and hostnames Appcircle runner infrastructure depends on; resubmit to Teknotel with explicit business context. Request written confirmation that these ranges will not be subject to automated blocking. | Engineering / DevOps | Draft |
| CAPA-03 | Request proactive notification SLA from Teknotel | Request that Teknotel notify Appcircle before applying any block affecting Appcircle's IP ranges, with a minimum 2-hour response window — except in cases of active DDoS. | Operations | Draft |
| CAPA-04 | Evaluate datacenter/ISP diversification | Assess whether critical runner infrastructure should be distributed across multiple ISPs or datacenters to eliminate single-ISP dependency for outbound connectivity. | Engineering Manager | Draft |
Layer 2 — Platform Resilience¶
| ID | Action | Detail | Owner | Status |
|---|---|---|---|---|
| CAPA-05 | Deploy internal DNS resolver on runner infrastructure | Run a local recursive DNS resolver (e.g., Unbound or dnsmasq) on all runner hosts to enable DNS-level overrides for critical hostnames centrally. Combined with monitoring (CAPA-08), enables automated remediation. | Engineering / DevOps | Draft |
| CAPA-06 | Implement Gradle bootstrap retry wrapper | Wrap Gradle bootstrap invocation in a retry script that, on TCP connection timeout, resolves the hostname again and retries with a different IP. Transparent to customers. | Engineering | Draft |
| CAPA-07 | Evaluate hosting a Gradle distribution mirror | Consider proxying or caching Gradle distribution archives within Appcircle's own infrastructure to eliminate dependency on GitHub CDN for the Gradle bootstrap step. | Engineering | Draft |
Layer 3 — Monitoring & Detection¶
| ID | Action | Detail | Owner | Status |
|---|---|---|---|---|
| CAPA-08 | Implement per-IP connectivity monitoring for critical external dependencies | Continuous synthetic monitoring checks probing each individual resolved IP for critical external hostnames every 1–2 minutes. Alert on partial reachability within 2–3 minutes. Would have detected this incident before the first customer build failure. | Engineering / DevOps | Draft |
| CAPA-09 | Add network diagnostics to build logs on download timeout | When a Gradle distribution download times out, automatically emit a network diagnostic summary: attempted hostname, resolved IPs, selected IP, TCP connection result. | Engineering | Draft |
| CAPA-10 | Add pre-build connectivity gate on runner agents | Before each build, runner agents perform a lightweight connectivity check against known-critical external hostnames. Fail fast with an explicit network error if all IPs are unreachable. | Engineering | Draft |
Layer 4 — Operational Response¶
| ID | Action | Detail | Owner | Status |
|---|---|---|---|---|
| CAPA-11 | Create ISP escalation runbook | Step-by-step runbook: diagnosis commands (nc -vz, traceroute, comparative subnet analysis), Teknotel contact details and ticket portal, standard traceroute evidence template, escalation path. Target: reduce time from detection to ISP ticket from ~2 hours to under 30 minutes. |
Engineering / Operations | Draft |
| CAPA-12 | Pre-draft customer communication templates for network incidents | Prepare templated notifications for network-related build failures (initial acknowledgment, status update, resolution) so communication can be issued within 15 minutes of incident declaration. | CSM / Operations | Draft |
8. Lessons Learned¶
What Worked Well¶
- Root cause was isolated to a specific IP within ~25 minutes of investigation start, using
nc -vzconnectivity tests and traceroute comparative analysis. - Comparative traceroute (blocked IP vs. working IP in the same
/22subnet) provided clear, unambiguous evidence for the ISP ticket. - The workaround was simple, deployable without Appcircle platform changes, and verified effective by Anadolu Sigorta within ~20 minutes.
- Customer notification was issued before the root cause was fully confirmed, correctly prioritizing customer communication over internal completeness.
Areas for Improvement¶
- Detection time was high. The failure presented as slow/hung builds rather than a network error. Per-IP connectivity monitoring (CAPA-08) would have surfaced the issue within 2 minutes rather than relying on customer reports.
- Gradle's DNS behavior created a non-obvious, intermittent failure mode. Approximately 1 in 4 builds would fail depending on which IP the DNS resolver returned. A local DNS resolver (CAPA-05) and Gradle retry wrapper (CAPA-06) would have prevented builds from failing entirely.
- A previously submitted allowlist had no effect. Appcircle had taken the correct preventive action and it failed. Process-only controls at the ISP level are insufficient. Platform-level resilience (CAPA-05, CAPA-06, CAPA-07) is essential as a backstop regardless of ISP cooperation.
- The ideal end state is full automation. CAPA-05 (internal DNS resolver) + CAPA-08 (per-IP monitoring) → automated detection and remediation, zero customer impact.
- ISP ticket-to-resolution took ~2.5 hours. A pre-drafted escalation runbook (CAPA-11) and established contacts would compress this significantly.
9. Data & Privacy Impact Assessment¶
| Criterion | Assessment |
|---|---|
| Personal Data Breach | No – no personal data was accessed, exposed, or compromised |
| Data Loss | No – no customer or platform data was lost |
| Security Breach | No – no unauthorized access occurred; brute-force traffic was associated with the GitHub CDN IP, not Appcircle's systems |
| Service Availability | Degraded – Android build functionality unavailable for ~5 hours |
| Regulatory Notification Required (GDPR / KVKK) | No – no personal data breach or data loss occurred |
| Third-Party Impact | Teknotel's automated blocking of a GitHub CDN IP was the proximate cause; no Appcircle customer data was exposed to third parties |
Conclusion: Service availability degradation caused by a third-party ISP-level automated network control action. Not a security incident or data breach under SOC 2 / ISO 27001.
10. Communication Log¶
| Date | Time (UTC+3) | Channel | Audience | Summary |
|---|---|---|---|---|
| 2026-02-24 | 11:42 | Internal (Slack) | Engineering Team | Root cause identified; workaround shared internally |
| 2026-02-24 | 11:46 | Internal (Slack) | CSM / Customer-facing teams | Risk to cloud customers flagged; workaround communicated for customer distribution |
| 2026-02-24 | 12:48 | Customer notification | All affected customers | Network-related issue acknowledged; team actively working on resolution |
| 2026-02-24 | 13:51 | Support Ticket | Teknotel Support | Ticket opened with traceroute evidence and comparative analysis |
| 2026-02-24 | 15:27 | Support Ticket | Teknotel → Appcircle | Teknotel confirms automated block on 185.199.108.133 due to brute-force detection |
| 2026-02-24 | ~16:00 | Support Ticket | Appcircle → Teknotel | Appcircle clarifies GitHub CDN ownership; requests removal of block |
| 2026-02-24 | 16:20 | Internal (Slack) | Engineering Team | Connectivity restored; builds confirmed succeeding |
| 2026-02-24 | 17:45 | Customer notification | All affected customers | Resolution confirmed; service restored to normal operation |
11. Approval & Sign-Off¶
| Role | Name | Date |
|---|---|---|
| Prepared By | Çağkan Aksun Yüksel – Engineering Manager | 2026-02-24 |
| Reviewed By | osmank@appcircle.io | |
| Approved By | osman@appcircle.io |
Document Control¶
| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0 | 2026-02-24 | Çağkan Aksun Yüksel | Initial release |
| 1.1 | 2026-03-05 | Çağkan Aksun Yüksel | Enriched with Slack thread timeline, Teknotel block analysis, internal DNS CAPA item, corrected resolution timestamps |
| 1.2 | 2026-03-05 | Çağkan Aksun Yüksel | CAPA section fully restructured into four defense layers with 12 detailed action items; Lessons Learned expanded |
Classification
This document is Confidential and intended for internal use and authorized distribution only.