IR-0004: TesterWeb WAF Rate Limiting – Internal Traffic Not Excluded (IR-2026-0407-001)¶
| Field | Detail |
|---|---|
| Document ID | IR-2026-0407-001 |
| Classification | Confidential – Internal Use Only |
| Version | 1.0 |
| Date Issued | 2026-04-08 |
| Prepared By | cagkan@appcircle.io |
| Reviewed By | osmank@appcircle.io |
| Approved By | cagkan@appcircle.io |
1. Incident Overview¶
| Field | Detail |
|---|---|
| Incident Date | 2026-04-07 |
| Detection Date | 2026-04-07 |
| Resolution Date | 2026-04-07 |
| Duration | ~9 hours (04:35–17:42 UTC+3) |
| Detection Source | N/A – detected internally via GCP monitoring |
| Affected Systems | dist.appcircle.io (disttesterweb-deployment), API Gateway, Tester API |
| Deployment Model | Cloud-Hosted (GCP) |
| Severity | SEV-3 – Low |
| Status | Resolved |
| Incident Reference | GCP Alert: prod-tester-deep-YZC4kDQW1bM |
2. Executive Summary¶
On April 7, 2026, GCP Uptime Monitoring triggered multiple alerts for dist.appcircle.io between 04:35 and 08:15 UTC+3. The disttesterweb-deployment pods became intermittently unresponsive, returning HTTP 500 errors and failing Kubernetes liveness probes.
The root cause was a WAF rate limiting misconfiguration: internal Appcircle IP addresses were not excluded from rate limiting rules. The TesterWeb → API Gateway → Tester API chain — all originating from the same internal IP — exhausted the rate limit quota, triggering 429 Too Many Requests responses that degraded pod health.
The high request volume was driven by enterprise customer FWD Group / Cube PH performing a large distribution download. No service-wide outage occurred and no other customers were impacted. The issue was resolved the same day by adding a WAF exception for the internal Appcircle IP.
3. Impact Assessment¶
3.1 Service Impact¶
| Dimension | Assessment |
|---|---|
| Service Availability | Degraded – intermittent 500 errors on dist.appcircle.io; no full outage |
| Data Integrity | No impact |
| Data Confidentiality | No impact |
| Affected Users | FWD Group / Cube PH – intermittent failures (~2K errors out of ~12K requests) |
| Other Customers | No observed impact |
| Duration of Impact | ~3.5 hours (07:24–08:17 UTC+3 active errors; alerts from 04:35) |
No SLA breach. No data loss. No customer escalation received.
3.2 Business Impact¶
- A single enterprise customer's distribution downloads experienced intermittent failures during the incident window.
- No customer-facing communication was required; the customer did not escalate.
3.3 Platform Impact¶
- Identified a systemic gap: IP-based rate limiting applied uniformly to both external and internal service-to-service traffic, treating internal fan-out calls as if they originated from a single external client.
4. Root Cause Analysis¶
Internal Appcircle IP addresses were not excluded from WAF rate limiting rules. The TesterWeb frontend issues 4–5 server-side calls to the API Gateway per user request, all originating from the same internal IP via the load balancer — consuming rate limit quota as a single external client. When FWD Group / Cube PH generated ~11.5K user-facing requests (~88K including internal fan-out), the aggregate traffic tripped the rate limit.
The customer's request volume (~12K) is within expected enterprise usage and should not have constituted a risk under a correctly configured system. The systemic gap was that IP-based rate limiting applied uniformly to both external and internal service-to-service traffic.
5. Timeline¶
All times are UTC+3 (Istanbul).
| Time | Date | Event |
|---|---|---|
| 04:35 | Apr 7, 2026 | First GCP uptime alert fires for prod-tester-deep-YZC4kDQW1bM |
| 05:14 | Apr 7, 2026 | Alert confirmed: uptime check failure recorded |
| 07:24 | Apr 7, 2026 | HTTP 500 responses begin appearing in GCP uptime probe logs |
| 08:46 | Apr 7, 2026 | 429 Too Many Requests errors identified on testerweb |
| 08:58 | Apr 7, 2026 | Source org identified: Cube PH (D8A6973E-E174-4BA5-BCBC-E871A7EE214C) |
| 10:15 | Apr 7, 2026 | Liveness probe failures confirmed as root signal for GCP health check failures |
| 15:05 | Apr 7, 2026 | Root org confirmed as FWD Group (f59b58b7-0615-4cd0-9e54-d7588a1c391f) |
| 16:11 | Apr 7, 2026 | Real user-facing request count refined: ~11.5K; fan-out ratio confirmed at 4–5x |
| 17:42 | Apr 7, 2026 | Fix applied: WAF exception added for internal Appcircle IP. No new blocks observed |
6. Containment & Resolution¶
No manual containment was required. The system self-recovered as customer traffic subsided.
A WAF exception was added by enver@appcircle.io at 17:42 UTC+3 to exclude the internal Appcircle IP from rate limiting rules, exempting the TesterWeb → API Gateway → Tester API chain from client-facing rate limits. No new blocked requests were observed after the change.
7. Corrective & Preventive Actions (CAPA)¶
| ID | Action | Detail | Owner | Status |
|---|---|---|---|---|
| CAPA-01 | WAF exception added for internal Appcircle IP | Rate limiting rules updated to exclude the internal IP used by the TesterWeb → API Gateway → Tester API chain. |
Platform | Done |
| CAPA-02 | Rate limiting strategy reviewed | Reviewed for other affected internal service chains; none identified. | Platform | Done |
| CAPA-03 | Expand internal rate limiting documentation | Cover GCP-side configuration and WAF exception guidance so future diagnosis is faster. Tracked under PL-16. | Platform | Open |
8. Lessons Learned¶
What Worked Well¶
- Root cause identified and fix applied within the same day of detection.
Areas for Improvement¶
- Internal service-to-service traffic must always be exempt from IP-based rate limiting; this should be validated in future configuration changes.
- Internal rate limiting documentation did not cover the GCP/cloud-side setup, which slowed diagnosis.
9. Data & Privacy Impact Assessment¶
| Criterion | Assessment |
|---|---|
| Personal Data Breach | No – no personal data was accessed, exposed, or compromised |
| Data Loss | No – no customer or end-user data was lost |
| Security Breach | No – no unauthorized access to systems, data, or infrastructure occurred |
| Service Availability | Degraded – intermittent 500 errors; no full outage. Other customers unaffected |
| Regulatory Notification Required (GDPR / KVKK) | No – does not meet notification threshold |
| Third-Party Impact | No – no third-party systems or data were affected |
Conclusion: This incident did not constitute a security incident, data breach, or service outage as defined under SOC 2 Type II or ISO/IEC 27001 controls.
10. Communication Log¶
| Date | Channel | Audience | Summary |
|---|---|---|---|
| 2026-04-07 | Slack (internal) | Engineering Team | Alert raised, investigation initiated, root cause identified, fix applied. |
| 2026-04-07 | Direct message | CEO (osman@appcircle.io) | Status update: high-volume enterprise traffic; intermittent alerts; no outage. |
11. Approval & Sign-Off¶
| Role | Name | Date |
|---|---|---|
| Prepared By | cagkan@appcircle.io | 2026-04-08 |
| Reviewed By | osmank@appcircle.io | 2026-04-08 |
| Approved By | cagkan@appcircle.io | 2026-04-08 |
Document Control¶
| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0 | 2026-04-08 | cagkan@appcircle.io | Initial release |
Classification
This document is Confidential and intended for internal use only. Unauthorized distribution is prohibited.