IR-0008: Workflow Steps Deleted on Workflow Rename - Boehringer Ingelheim¶
- Status: Resolved
- Date: 2025-09-09
- Severity: SEV-2 - High
- Duration: ~20 hours (2025-09-09 18:08 to 2025-09-10 14:06)
- Affected: Boehringer Ingelheim (Cloud) - Build module workflow configuration
- Owners: @harun, @tolga
Summary¶
Editing a workflow name in the Build module deleted all configured workflow steps, leaving the workflow blank. Affected users had to recreate entire workflow configurations, with no warning or recovery mechanism. Up to 60+ sub-organizations were potentially exposed.
Timeline¶
All times UTC+3 (Istanbul).
| Time | Date | Event |
|---|---|---|
| 18:08 | Sep 9, 2025 | Customer reports all workflow steps deleted when renaming workflows. |
| 18:15 | Sep 9, 2025 | Frontend team acknowledges; commits to same-day hotfix. |
| 18:51 | Sep 9, 2025 | FE-1901 deployed to production. |
| 18:58 | Sep 9, 2025 | Marked resolved. |
| 09:04 | Sep 10, 2025 | Issue confirmed still occurring in production. |
| 09:29 | Sep 10, 2025 | Root cause identified: steps deleted when a workflow is renamed without opening it first. |
| 10:54 | Sep 10, 2025 | API incompatibility after scope/role-management changes confirmed. |
| 11:01 | Sep 10, 2025 | Backend fix (BE-7049) deployed to dev; testing passed at 11:25. |
| 14:06 | Sep 10, 2025 | BE-7049 deployed to production. |
| 14:51 | Sep 10, 2025 | Backward-compatibility gap found: previously lost workflows not auto-recoverable. |
| 15:11 | Sep 10, 2025 | Manual recovery plan created (restore from prior builds by profile + workflow name). |
Root Cause¶
Changes to the role-management/scope system introduced an unintended side effect in workflow editing. When a workflow name was updated via a PATCH operation, the existing workflow document could not be retrieved due to the scope/permission change. The backend then performed a "replace" with an empty configuration: the workflow ID was preserved but its content was overwritten with an empty document. No safeguard or confirmation prevented the destructive overwrite.
Impact¶
- Customers affected: Boehringer Ingelheim (Cloud); up to 60+ sub-organizations potentially exposed
- Services degraded: Build module workflow editing (data loss on rename)
- Data loss: Yes - workflow step configurations overwritten; recovered manually from prior builds
Resolution¶
- Frontend fix (FE-1901) deployed 2025-09-09 18:51 to address the workflow editor interface behavior.
- Backend fix (BE-7049) deployed 2025-09-10 14:06 to correct the role-management integration so the existing workflow document is retrieved and not replaced with an empty one.
- Affected customer workflows restored manually from prior builds using profile and workflow names.
Action Items¶
| # | Action | Owner | Due Date | Status |
|---|---|---|---|---|
| 1 | Implement workflow backup before mutating operations | Backend | Pending | |
| 2 | Add confirmation/safeguard before destructive workflow operations | Frontend / Backend | Pending | |
| 3 | Enhanced integration testing for scope/role-management changes vs workflow functionality | QA | Pending | |
| 4 | Add automated test coverage for critical workflow-editing journeys (incl. content preservation after rename) | QA | Pending | |
| 5 | Restore affected customer workflow profiles | Backend | Done |
Links¶
- Linear issue(s): BE-7051, BE-7049, FE-1901
- Support ticket: Zendesk #767