IR-0006: Repository & Distribution Pipeline Queue Lag - Ziraat Teknoloji¶
- Status: Resolved
- Date: 2025-09-03
- Severity: SEV-2 - High
- Duration: ~3 days (intermittent, 2025-09-03 to 2025-09-08)
- Affected: Ziraat Teknoloji (Self-Hosted 3.29.0) - repository connection, build distribution, notifications
- Owners: @recep
Summary¶
Repository operations and the distribution pipeline experienced intermittent blocking: processes queued for 1-2 hours and then executed all at once. Build artifacts were delayed before reaching Testing Distribution profiles, repository refresh operations hung, and test notification emails were delayed, all during a critical customer release cycle.
Timeline¶
All times UTC+3 (Istanbul).
| Time | Date | Event |
|---|---|---|
| 15:45 | Sep 2, 2025 | Zeynep (customer) reports disconnect/reconnect error; branches not refreshing after 10 min wait. |
| 15:48 | Sep 2, 2025 | Repository can be selected but will not fetch; PAT authentication error observed. |
| 16:08 | Sep 2, 2025 | A ~1h20m blockage observed; queued distribution emails arrive simultaneously. |
| 17:47 | Sep 2, 2025 | Second occurrence: test package emails not delivered. |
| 09:34 | Sep 3, 2025 | Customer opens Ticket #755 with logs and screenshots. |
| 16:56 | Sep 3, 2025 | Third occurrence reported; investigation meeting held. |
| 09:17 | Sep 4, 2025 | Recep identifies a specific Testing Distribution profile involved; missing GitLab webhook events for the Android project confirmed. |
| 21:01 | Sep 5, 2025 | Priority escalated to URGENT; internal deep-dive meeting held. |
| 12:30 | Sep 6, 2025 | Hotfix package generated, verified in customer test env, deployed to production. |
| 17:30 | Sep 8, 2025 | Customer confirms the issue no longer occurs under real-world usage. |
Root Cause¶
On repository connection, the system queried commits across all branches. For a repository with 1600+ branches this took far longer than expected, and additional per-branch existence-check queries during connection tests compounded the delay. As a result, other events could not be processed on time, degrading overall pipeline throughput. This was a customer-specific scenario (very high branch count) not seen in standard repository structures.
Impact¶
- Customers affected: Ziraat Teknoloji (self-hosted)
- Services degraded: Repository connection/refresh, build distribution to Testing Distribution, notification emails
- Data loss: No
Resolution¶
- Commit retrieval limited to the default branch; commits from other branches are queried only when explicitly selected in the UI.
- For GitLab, increased the page size for commit and branch queries, reducing request volume.
- Removed the real-time branch existence check during selection; validation now happens only once for the branch used to start a build.
- Delivered via a build server patch release deployed to the customer, verified across refresh + Publish + Distribution scenarios.
Action Items¶
| # | Action | Owner | Due Date | Status |
|---|---|---|---|---|
| 1 | Apply the changes to the new self-hosted image | Infra Team | 2025-09-18 | Done |
| 2 | Add automated alerting for pipeline performance-degradation patterns | Backend / DevOps | Pending |
Links¶
- Linear issue(s): BE-7022, BE-7009
- Support ticket: Zendesk #755
- Related PR(s): build server patch release (3.29.x)