IR-0003: PublishCount Usage Inflation – Anadolu Sigorta (IR-2026-06-29-BE8914)¶

Field	Detail
Document ID	IR-2026-06-29-BE8914
Classification	Confidential – Internal Use Only
Version	1.1
Date Issued	2026-07-01
Prepared By	emre@appcircle.io
Reviewed By	enver@appcircle.io
Approved By	enver@appcircle.io

1. Incident Overview¶

Field	Detail
Incident Date	Chronic since 2026-04-29 (introduced by BE-7761); reported 2026-06-26
Detection Date	2026-06-29 (systematic investigation)
Resolution Date	Fix merged to master; rolling out to self-hosted deployments
Duration	~2 months of silent metering inflation (2026-04-29 → detection)
Affected Systems	Appcircle License Server, Publish Service (Appcircle.Core shared event model)
Deployment Model	Self-Hosted (confirmed: Anadolu Sigorta, v3.31.0) + Cloud (by mechanism)
Severity	SEV-2
Status	Resolved – rolling out
Incident Reference	BE-8914

2. Executive Summary¶

Anadolu Sigorta's self-hosted Appcircle deployment (v3.31.0) reported that the publish usage counter (PublishCount) displayed values far higher than the number of publishes actually performed — for example, 51 recorded against an expectation of ~4–5 in a month with only 3 builds. Because the inflated counter made the customer appear to approach their license limit, the account was erroneously throttled and the license limit was subsequently raised (125 → 150) — a mitigation of the symptom, not the cause.

The root cause is that a server-side multi-step publish was counted once per step instead of once per publish. A change introduced by BE-7761 (commit cb3d093, 2026-04-29) began folding event fields into a single JSON envelope (AC_PUBLISH_ENV_BUNDLE_JSON). The step-index field (AC_STEP_INDEX) was not added to the pass-through allowlist, so it was folded into the envelope. The License Server continued to read the field at the top level, failed to find it, defaulted it to 0, and therefore treated every server-side step as the "first step" and counted it. A publish with N server-side steps was recorded as N publishes instead of one.

A fix was authored in the License Server, verified locally end-to-end (a 6-step publish now produces 1 count instead of 6), hardened following code review, and merged to master. It is being rolled out to self-hosted deployments.

Customers affected: Anadolu Sigorta (self-hosted, confirmed). By mechanism, any organization running server-side multi-step publishes since 2026-04-29 is potentially affected.

3. Impact Assessment¶

3.1 Service Impact¶

Dimension	Assessment
Service Availability	Not down. Feature-level: publish metering produced inflated values, erroneously throttling publish operations against the license limit.
Affected Feature	`PublishCount` license metering for server-side multi-step publishes, and downstream license-limit enforcement.
Data Integrity	Impacted — `PublishCount` usage records inflated in the datastore; no source code, artifact, or customer content affected.
Data Confidentiality	No impact
Affected Users	Anadolu Sigorta (self-hosted, confirmed). All deployments running server-side multi-step publishes since 2026-04-29 are potentially affected; inflation was also reproduced on an internal cloud test organization during investigation.
Duration of Impact	Chronic since 2026-04-29 until fix deployment.

3.2 Business Impact¶

The customer was erroneously throttled against a license limit they had not actually reached, blocking legitimate publish operations.
The license limit was raised (125 → 150) on the basis of inflated data; the increase should be revisited once usage figures are accurate (the inflated period value clears at the next billing-period reset, see 6.3).
The reliability of usage figures was undermined.

3.3 Platform Impact¶

Contract-test gap: a change to a shared event serialization (AC_PUBLISH_ENV_BUNDLE_JSON) silently invalidated a read contract in another service, undetected by any test.
Silent-default anti-pattern: EventData.TryGetValueAsInt returns 0 when a key is absent, conflating "missing data" with a meaningful business value (step 0 = first step).
Traceability gap: usage records carry no source identifier (publish/task id), so "where did 51 come from" could not be answered directly from the data.

4. Root Cause Analysis¶

4.1 Root Cause — `AC_STEP_INDEX` envelope inflation (confirmed)¶

Intended behavior. In the License Server, PublishEventController.PublishStart (handling the Publish|PublishQueueServer event) reads AC_STEP_INDEX and increments PublishCount only when stepIndex == 0, so a multi-step publish is counted once.

Event data model. MessageBrokerEvent.AdditionalData is an array of named boxes: [{ Id, Data, Version }, ...]. The License Server locates the step index via FirstOrDefault(i => i.Id == "AC_STEP_INDEX") — by name, in the array.

The break. BE-7761 (cb3d093) introduced BundlePlainEventData, which folds event fields into a single AC_PUBLISH_ENV_BUNDLE_JSON envelope. The pass-through allowlist of keys kept outside the envelope (_publishEventDirectAccessKeys) did not include AC_STEP_INDEX, so it was folded inside:

// BEFORE bundling (worked)
"AdditionalData": [ { "Id": "AC_STEP_INDEX", "Data": "3" }, ... ]

// AFTER bundling (broke — AC_STEP_INDEX now INSIDE the envelope string)
"AdditionalData": [
  { "Id": "AC_PUBLISH_ENV_BUNDLE_JSON",
    "Data": "{\"AC_ORGANIZATION_ID\":\"...\",\"AC_STEP_INDEX\":\"3\"}" }
]

Because the License Server still searched the top level, it did not find the box. A second design weakness then applied: EventData.TryGetValueAsInt returns 0 when the key is absent. Consequently stepIndex was 0 for every step, the stepIndex == 0 condition was true on every step, and every server-side step was counted as a new publish.

Result: N enabled server-side steps = N PublishCount (should be 1). The observed 51 is the sum, over the month's publishes, of each publish's enabled server-side step count.

Why the agent path is unaffected. Inflation appears only on the server-side path (Publish|PublishQueueServer). The agent-side path is also metered (Build|PublishQueue → Build Server → Build|PublishStart → the License Server's BuildEventController, which increments PublishCount when PublishStepIndex == 0), but it is not inflated: the Build Server unpacks the envelope via MergePublishEnvBundle and reads the step index as a proper top-level field, so only genuine first steps are counted. This is why single-step or agent-side publishes show no inflation.

Code locations:

File	Location	Note
`ac-service-publish/.../PublishService.Domain/ApplicationAdapter.cs`	`:10617`	`_publishEventDirectAccessKeys` allowlist - `AC_STEP_INDEX` missing (root defect, still open)
`ac-service-publish/.../PublishService.Domain/ApplicationAdapter.cs`	`:10636`	`BundlePlainEventData` folds non-allowlisted keys into the envelope
`ac-server-license/.../WebApi/Events/PublishEventController.cs`	`PublishStart` / `GetPublishStepIndex`	step-index read point (fix applied here)
`ac-lib-core/.../Business/MessageBroker/EventData.cs`	`:161`	`TryGetValueAsInt` returns `0` on missing key (silent default)

4.2 Contributing Factors¶

Silent default on missing key (EventData.TryGetValueAsInt → 0) turned "field not present" into a valid, meaningful business value.
No consumer-side contract test guarded the License Server's dependency on the top-level presence of AC_STEP_INDEX, so the bundling change passed CI undetected.
No cross-service impact analysis was performed when the shared event serialization was changed.

5. Timeline¶

All dates in UTC+3 (Istanbul).

Date	Event
2026-04-29	BE-7761 (commit `cb3d093`) merged. `BundlePlainEventData` goes live; `AC_STEP_INDEX` is not on the pass-through allowlist and is folded into `AC_PUBLISH_ENV_BUNDLE_JSON`. Inflation begins silently.
2026-04-29 → 2026-06-24	`PublishCount` inflates on server-side multi-step publishes; Anadolu Sigorta usage climbs far above actual.
2026-06-24	Anadolu Sigorta license limit raised 125 → 150 after the customer hit the limit (symptom mitigation; not the cause).
2026-06-26	Anomaly reported: 51 recorded against an expectation of ~4–5.
2026-06-29	Systematic investigation. Root cause confirmed via an isolated unit test and a local Kafka end-to-end reproduction. Fix authored, committed (`c3a3f46`), and pushed to `feature/BE-8914`.
2026-06-30	Fix hardened after CodeRabbit review (`int?` return, null-and-skip, parse-failure logging). Merged to develop and to master (hotfix PR #196); rollout to self-hosted deployments begins.

6. Containment & Resolution¶

6.1 Fix (merged, rolling out)¶

Approach (most isolated option): change only the License Server so it resolves AC_STEP_INDEX from inside the envelope when it is not present at the top level. The agent-side path was left untouched (already correct), keeping the change to a single service.

File: Appcircle.LicenseServer/src/Appcircle.LicenseServer.WebApi/Events/PublishEventController.cs

Final behavior after code review:

A GetPublishStepIndex helper returns int?: if the step index is found neither at the top level nor in the envelope, it returns null (previously a silent 0, which caused the miscount).
PublishStart handles null explicitly: it logs a warning (...skipped: step index could not be resolved...) and returns without counting (defensive layer).
Envelope parse failures are logged (LogWarning(ex, ...)) rather than silently swallowed.

6.2 Verification (local; debugger + datastore)¶

Stage	Scenario	Result
Isolated unit test	envelope contains `AC_STEP_INDEX` → legacy top-level read	returns `0` (bug mechanism confirmed)
Kafka repro — before fix	4 envelope events (steps 0–3)	4 records (inflation)
Kafka repro — after fix	4 envelope events	1 record
Kafka repro — after fix	6 envelope events (steps 0–5)	1 record
Control	top-level `AC_STEP_INDEX=2`	0 records (correct)
Combined final test	2 rounds × 3 events (step 0 / 3 / none)	2 records (only the step-0 events)

6.3 Outstanding¶

The fix has been merged to master and is being rolled out to self-hosted deployments; impact ends for each environment as it updates.
Historical inflated counters are not corrected retroactively; a manual backfill was assessed as impractical, and the inflated period value clears at the next billing-period reset.

7. Corrective & Preventive Actions (CAPA)¶

ID	Action	Detail	Owner	Status
CAPA-01	Complete rollout of the fix	Merged to master; complete deployment of the updated License Server to all self-hosted deployments so new publishes are counted once.	Backend / Ops	In progress
CAPA-02	Add usage-record traceability (data)	Add `publishId` / `taskId` / `source` to the usage store object so a counter value can be traced to its origin.	Backend	Open
CAPA-03	Usage history / audit trail (product)	Expose an event-level usage breakdown to the customer and support (which publish, when, which organization) instead of only an aggregate total, so a count can be reconciled. Requires a product decision on the surface (drill-down from the limit view, history tab) and a supporting endpoint.	Product	Open
CAPA-04	Consumer-side contract test for shared event keys	Add a regression test asserting the License Server can resolve `AC_STEP_INDEX` regardless of bundling; require cross-service impact analysis when changing shared event serialization.	Backend	Open
CAPA-05	Audit the silent-default read	Replace or wrap `EventData.TryGetValueAsInt`'s "missing key → 0" behavior with explicit not-found semantics, or audit all call sites relying on it.	Backend / Core	Open
CAPA-06	Revisit the license-limit increase	Review the 125 → 150 change made on inflated data once usage figures are accurate (after the next billing-period reset).	Product / Licensing	Open

8. Lessons Learned¶

What Worked Well¶

The root cause was isolated to a specific mechanism (envelope bundling + silent default) through static analysis of the event model, then confirmed with an isolated unit test and a controlled end-to-end reproduction.
The chosen fix was the most isolated option (single service), reducing deployment risk, and was verified against multiple scenarios including the "field entirely absent" edge case.
Code review materially improved the fix, converting a silent 0 default into an explicit null-and-skip path with logging.

Areas for Improvement¶

A serialization change silently broke a downstream read contract. Shared event schema changes need consumer-side contract tests and cross-service impact analysis before shipping.
A "Try"-named accessor returned a meaningful default on missing input. "Not found" must be represented explicitly, never collapsed into a business-significant constant.
Lack of traceability delayed diagnosis. Without a source identifier on usage records, "where did 51 come from" could not be answered from the data — and there is no customer-facing usage history either: only aggregate totals are exposed, not the underlying events, so neither the customer nor support can reconcile a count.
Detection depended on a customer report. The inflation ran for ~2 months undetected; a sanity check comparing publish counts against actual publish/build activity would have surfaced it earlier.
The change exceeded its ticket's scope. BE-7761 was scoped to secret transfer to agents on dequeue and should not have altered the behavior of non-secret environment variables. Folding AC_STEP_INDEX into the bundle was an unintended, out-of-scope change; an earlier iteration was reverted when side effects surfaced, but the refactor re-entered through a large pull request and was not caught. Changes should stay within the issue's stated intent — behavioral changes to fields the ticket does not call for should not ship.
Review effectiveness must improve — human and AI. The out-of-scope refactor passed both human and AI (CodeRabbit) review, partly because it arrived inside a very large diff. Keep pull requests small and scoped, share the plan and target up front and iterate on reviewer feedback, and treat AI review as a complement to — not a replacement for — careful human review of shared and event-path changes.

9. Data & Privacy Impact Assessment¶

Criterion	Assessment
Personal Data Breach	No — no personal data was accessed, exposed, or modified.
Data Loss	No — no source code, build artifacts, or customer content was lost.
Security Breach	No — no unauthorized access; this is a metering/accounting defect.
Data Integrity	Impacted — `PublishCount` usage/metering records were inflated; the inflated period value clears at the next billing-period reset.
Service Availability	Feature-level degradation — publish operations were erroneously throttled by incorrect license enforcement.
Regulatory Notification Required (GDPR / KVKK)	No — no personal data breach or data loss occurred.
Third-Party Impact	None.

Conclusion: A metering-accuracy defect causing feature-level throttling; not a security incident or personal-data breach under SOC 2 / ISO 27001.

10. Communication Log¶

Date	Time (UTC+3)	Channel	Audience	Summary
2026-06-24	—	Licensing action	Anadolu Sigorta	License limit raised 125 → 150 in response to erroneous throttling (symptom mitigation).
2026-06-26	—	Internal (Slack)	Engineering	Anomaly reported and discussed (inflated counter); investigation initiated.

No formal customer incident notification has been issued yet. A customer-facing communication should follow deployment of the fix.

11. Approval & Sign-Off¶

Role	Name	Date
Prepared By	emre@appcircle.io	2026-07-01
Reviewed By	enver@appcircle.io	2026-07-01
Approved By	enver@appcircle.io	2026-07-01

Document Control¶

Version	Date	Author	Changes
1.0	2026-07-01	emre@appcircle.io	Initial release
1.1	2026-07-01	emre@appcircle.io	Review fix-ups: clarified agent-side metering in 4.1; scoped inflation to enabled server-side steps; corrected code-location line numbers against master; aligned limit-revisit wording with the billing-period reset.

Classification

This document is Confidential and intended for internal use and authorized distribution only.

IR-0003: PublishCount Usage Inflation – Anadolu Sigorta (IR-2026-06-29-BE8914)¶

1. Incident Overview¶

2. Executive Summary¶

3. Impact Assessment¶

3.1 Service Impact¶

3.2 Business Impact¶

3.3 Platform Impact¶

4. Root Cause Analysis¶

4.1 Root Cause — AC_STEP_INDEX envelope inflation (confirmed)¶

4.2 Contributing Factors¶

5. Timeline¶

6. Containment & Resolution¶

6.1 Fix (merged, rolling out)¶

6.2 Verification (local; debugger + datastore)¶

6.3 Outstanding¶

7. Corrective & Preventive Actions (CAPA)¶

8. Lessons Learned¶

What Worked Well¶

Areas for Improvement¶

9. Data & Privacy Impact Assessment¶

10. Communication Log¶

11. Approval & Sign-Off¶

Document Control¶

4.1 Root Cause — `AC_STEP_INDEX` envelope inflation (confirmed)¶