0008-Android (Google Play) Staged Rollout – Manual & Automatic – Technical Design Document¶

TL;DR

Appcircle now manages Google Play staged rollout for Android app versions directly from the Publish module, in two modes: manual (set a single target percentage, applied to Play immediately) and automatic (a per-day percentage plan that a background scheduler advances one step per day, with halt and resume). Rollout state is held server-side per track in the publish service and reconciled against live Google Play data on every read, instead of being driven from a Fastlane build step. The feature shipped in 3.31.0 and every operation is gated by a paired publish# permission scope and license# feature scope.

Status: 📝 Draft (retrospective; feature shipped in 3.31.0) • Linear: BE-7205 (analysis: BE-7299) • Primary PR: appcircleio/ac-service-publish#462 (feature/BE-7205)

Introduction¶

Purpose: This document describes the design that gives Appcircle users control over Google Play staged rollout directly from the Publish module, without opening the Google Play Console. The feature originates from a customer request (Intertech, Zendesk #803 / CSM-244) for the same staged-release experience that already exists for iOS phased release. Two modes are supported:

Manual rollout – the user sets a single target rollout percentage, and Appcircle submits it to Play immediately.
Automatic rollout – the user defines a per-day percentage plan (for example 10 → 20 → 50 → 100), and a background scheduler advances the live rollout one step per day until the plan completes, with halt and resume control.

Scope: - New rollout API surface in ac-service-publish (list tracks, read state, update rollout, halt/resume). - A new per-track rollout state persisted on the app version (PlayStoreRolloutState). - A periodic background service that advances active automatic rollouts. - Kafka events for activity log + notifications, consumed by ac-server-reporting and ac-server-notification. - API Gateway routes, permission scopes and per-operation license gating. - Android / Google Play only. iOS is explicitly rejected at the controller (PublishPlatformTypeInvalid).

System Overview¶

High-Level Description: The feature manages the userFraction / status of a Google Play track release for a given app version. Appcircle keeps a local mirror of the rollout (PlayStoreRolloutStateStoreObject) per track and reconciles it with the live Play state on every read, so the UI always reflects what Google actually reports. For automatic rollouts, a scheduled job (RolloutPeriodicService) walks each active plan forward one day at a time, submitting the next percentage to Play and emitting events.

Rollout percentages are presented to the user as whole percentages (10–100) but stored internally as fractions (0.1–1.0), matching the Google Play Developer API contract.

Architecture¶

System Architecture (data flow):

                 ┌─────────────────────────── ac-service-publish ───────────────────────────┐
  Web App  ──►   RolloutController (api/v1/rollout)                                            │
  (via Gateway)        │  GET  {platform}/{profileId}/{appVersionId}/tracks                    │
                       │  GET  {platform}/{profileId}/{appVersionId}?trackValue=               │
                       │  POST {platform}/{profileId}/{appVersionId}        (set manual/auto)  │
                       │  PATCH{platform}/{profileId}/{appVersionId}/status (halt/resume)      │
                       ▼                                                                        │
              IApplicationAdapter (rollout business logic + helpers)                           │
                       │  ├─ playStoreService.SubmitReleaseToTrack ──────────►  Google Play    │
                       │  ├─ GetPlayStoreDistributionTracksFromStore ◄───────   Developer API  │
                       │  ├─ _applicationService.UpdateAppVersionRolloutStateAsync ─► MongoDB  │
                       │  └─ PublishPlayStoreRolloutEvent ──► Kafka topic "Publish"            │
                       ▼                                                                        │
   RolloutPeriodicService (every 120 min, RedLock-guarded) ─► ProcessAutomaticRollouts()       │
                 └────────────────────────────────────────────────────────────────────────────┘
                                              │ Kafka "Publish"
                            ┌─────────────────┴──────────────────┐
                            ▼                                     ▼
                  ac-server-reporting                   ac-server-notification
                  (publish activity log)                (in-app toast messages)

Technology Stack: - .NET (C#), ASP.NET Core Web API (ac-service-publish). - MongoDB (rollout state embedded on the AppVersions collection). - Kafka (topic Publish) via the shared message-broker builder. - Redis / RedLock (distributed lock for the periodic service). - Google Play Developer API (via the internal playStoreService). - Shared contracts in ac-lib-core (Appcircle.Core).

External Dependencies: - Google Play Developer API – read track releases, submit userFraction / status. Authenticated with the profile's configured Google service-account JSON key (resolved through GetCachedGooglePlayStoreCredentials).

Key Components¶

Component 1 – RolloutController¶

Purpose: HTTP surface for rollout management.
Responsibilities: Validate platform is Android, extract UserInfo/OrganizationId, delegate to the adapter.
Interactions: IApplicationAdapter.
Endpoints:
GET {platformType}/{profileId}/{appVersionId}/tracks – eligible Play tracks that contain the app version.
GET {platformType}/{profileId}/{appVersionId}?trackValue= – current rollout info (reconciles + persists state).
POST {platformType}/{profileId}/{appVersionId} – set manual percentage or automatic plan.
PATCH {platformType}/{profileId}/{appVersionId}/status – halt / resume an automatic rollout.

Component 2 – IApplicationAdapter (rollout logic)¶

Purpose: All rollout business rules and Play Store interaction.
Key methods: GetRolloutTracks, GetRolloutInfo, UpdateRolloutInfo, UpdateRolloutStatus, ProcessAutomaticRollouts.
Helpers: SubmitPlayStoreRolloutTrackUpdate, ResolveCurrentAutomaticDay, GetCurrentRolloutFraction, GetPreferredTrackRelease, PercentageToFraction / FractionToPercentage, GetFreshDefaultPercentages, BuildRolloutInfoFromState, ToRolloutTrackResponse, PublishPlayStoreRolloutEvent.
Data Flow: Reads live track from Play → reconciles with stored state → submits change to Play → persists state → emits Kafka event.

Component 3 – RolloutPeriodicService¶

Purpose: Advance active automatic rollouts on a schedule.
Responsibilities: Runs PeriodicHostedServiceBase with Period = 120 min, StartDelay = 30 s. Acquires a RedLock named RolloutPeriodicService (TTL 119 min) so only one instance processes; calls ProcessAutomaticRollouts().
Interactions: IApplicationAdapter, Redis.

Component 4 – Persistence (`PlayStoreRolloutStateStoreObject`)¶

Purpose: Per-track mirror of the rollout, embedded as an array on AppVersionStoreObject.
Repository: ProfileRepository.UpdateAppVersionRolloutState (upsert by TrackValue) and GetProfilesWithAutomaticPlayStoreRolloutsAsync (find due rollouts).

Component 5 – Event consumers¶

ac-server-reporting – writes a publish activity-log entry per rollout event.
ac-server-notification – raises an in-app toast per rollout event.

Detailed Design¶

Database Design¶

Rollout state is embedded on the app version document (AppVersions collection), one entry per track:

PlayStoreRolloutStateStoreObject | Field | Type | Meaning | |---|---|---| | TrackValue | string | Play track name (key, case-insensitive). | | Mode | enum | Manual = 0, Automatic = 1. | | CurrentRolloutPercentage | double? | Live fraction (0.1–1.0) from last successful sync. | | LastKnownRolloutPercentage | double? | Last synced fraction. | | CurrentStatus | string | Play release status (inProgress / completed / halted / draft). | | AutomaticPercentages | List | The per-day plan as fractions. | | CurrentDay | int? | 1-based index of the day reached. | | IsAutomaticActive | bool | Plan running (not halted, not completed). | | IsCompleted | bool | Reached 100% or final day. | | NextExecutionAt | DateTimeOffset? | When the scheduler should advance next (UtcNow + 1 day). | | LastSyncedAt | DateTimeOffset? | Last reconciliation timestamp. |

Due-rollout query (GetProfilesWithAutomaticPlayStoreRolloutsAsync) uses an ElemMatch over the embedded array: IsAutomaticActive == true && IsCompleted == false && NextExecutionAt != null && NextExecutionAt <= now.

Note. There is no dedicated index on the embedded rollout fields; the existing Migration4 indexes target other AppVersionStoreObject fields, so the periodic ElemMatch is currently a collection scan over AppVersions. See Risks and Mitigation Strategies.

Upsert (UpdateAppVersionRolloutState) is a read-modify-write: load the document, find the matching track entry, add or mutate it, then Set the whole array. It is not an atomic positional update. See Risks (concurrency).

Algorithms and Logic¶

Fraction/percentage conversion – percentages are whole numbers in the API; internally PercentageToFraction = round(pct/100, 2) and FractionToPercentage = round(fraction*100, 0). GetCurrentRolloutFraction returns 1.0 for a completed release, otherwise the rounded UserFraction.

Manual rollout validation (UpdateRolloutInfo, Mode == Manual): - value required, >= 10, > current live Play percentage, <= 100, whole number, and != current value. - Submits to Play with status inProgress (or completed at 100%); clears AutomaticPercentages and CurrentDay.

Automatic plan validation: - 1–7 days; no null entries; each day >= 10, >= current live Play percentage, monotonically non-decreasing, <= 100, whole numbers. - Already-elapsed days are skipped in validation when a plan is active (activePastDayCount = CurrentDay), so users can edit the remaining tail of a running plan (PR #491 no-null-gap rule, #492 skip past-days, #515 allow same percentage on different days). - ResolveCurrentAutomaticDay maps the live fraction back to a day index (highest day whose target ≤ current fraction); the first submitted day = that resolved day (default 1).

Automatic progression (ProcessAutomaticRollouts, per due rollout): 1. Re-fetch the live track from Play; resolve current fraction/day. 2. If Play reports halted or completed (or live fraction ≥ 1.0): persist terminal state (IsAutomaticActive = false, NextExecutionAt = null), emit PlayStoreRolloutHalted / PlayStoreAutoRolloutCompleted, stop. 3. Else compute nextDay = liveCurrentDay + 1. If beyond the plan length, mark completed. 4. Else submit AutomaticPercentages[nextDay-1] to Play, persist CurrentDay = nextDay, NextExecutionAt = UtcNow + 1 day (or null if the step is 100%), emit PlayStoreAutoRolloutProgressed / ...Completed. - The whole loop body is wrapped in try/catch per rollout so one failure does not stop the batch.

Halt / resume (UpdateRolloutStatus): only allowed for automatic, non-completed rollouts. Submits Play status halted (pause) or inProgress (resume); sets IsAutomaticActive and NextExecutionAt accordingly; emits PlayStoreRolloutHalted / PlayStoreRolloutResumed.

Read reconciliation (GetRolloutInfo): always fetches the live track, recomputes day/completed/active flags from the live fraction, persists the reconciled state, and returns it. ToRolloutTrackResponse also computes MinimumAllowedPercentage = clamp(current+1, 10, 100) and suggests default plans (GetFreshDefaultPercentages → 10,20,30,50,75,100 filtered by the minimum) for the UI (PR #487 UX defaults).

Kafka events (topic Publish, via PublishPlayStoreRolloutEvent) carry callback data: ProfileName, TrackName, PreviousRolloutPercentage, NewRolloutPercentage, CurrentRolloutPercentage, DayNumber, ProfileId, AppVersionId, StoreStatus (IN_PROGRESS / COMPLETED / HALTED).

Event names (defined in ac-lib-core EventConst.Publish): PlayStoreManualRolloutSucceeded, PlayStoreManualRolloutFailed, PlayStoreAutoRolloutProgressed, PlayStoreAutoRolloutCompleted, PlayStoreAutoRolloutFailed, PlayStoreRolloutHalted, PlayStoreRolloutResumed.

Security Considerations¶

AuthN: Gateway routes use userToken auth (user-scoped, not service token).
AuthZ: Each operation carries a dedicated permission scope and a paired license scope:
GET rollout / tracks → publish#get_rollout_appversion_android + license#get_rollout_appversion_android
POST rollout → publish#update_rollout_appversion_android + license#update_rollout_appversion_android
PATCH status → publish#update_rollout_status_appversion_android + license#update_rollout_status_appversion_android (License gating added in apigateway #347; dedicated scopes in #341.)
Multi-tenancy: OrganizationId is taken from the user context and threaded through every adapter call and Mongo query; the gateway routes carry no org segment (tenant isolation enforced downstream).
Secrets: Google Play access uses the profile's stored service-account JSON key, fetched via the cached store-credential lookup; the key is never exposed in API responses or events.

User Interface (if applicable)¶

Web App surfaces per-track rollout state, the manual percentage input, the automatic day-plan editor (with suggested defaults and a computed minimum), and halt/resume controls. Progress, completion, halt, resume and automatic-rollout failure are surfaced as in-app toast notifications (see Integration). Manual-rollout failure (PlayStoreManualRolloutFailed) is intentionally not toasted; it is recorded in the publish activity log only, since the failure is reported inline in the API response the user is already looking at.

Note. The web UI is delivered separately in ac-web-app. The separation of the App Release and Rollout Information sections from the metadata details view is tracked in BE-8517.

Integration and Interfaces¶

External Interfaces: - Google Play Developer API – playStoreService.SubmitReleaseToTrack(packageName, { Status, Name, VersionCodes, UserFraction }, trackValue) and track read. At 100% the request sends status=completed with UserFraction=null; below 100% it sends inProgress/halted with the target fraction.

Internal Interfaces:

API Gateway routes (ac-server-apigateway, module publish, downstream {{PUBLISH}}): | Method | Gateway path | Downstream | |---|---|---| | GET | /publish/{version}/rollout/android/{profileId}/{appVersionId}/tracks | /api/{version}/rollout/android/.../tracks | | GET | /publish/{version}/rollout/android/{profileId}/{appVersionId} | /api/{version}/rollout/android/... | | POST | /publish/{version}/rollout/android/{profileId}/{appVersionId} | /api/{version}/rollout/android/... | | PATCH | /publish/{version}/rollout/android/{profileId}/{appVersionId}/status | /api/{version}/rollout/android/.../status |

Kafka (topic Publish, producer → consumers): - ac-server-reporting consumes all 7 events → publish activity log. New PublishAction values 55–61 (e.g. PlayStoreAutoRolloutProgressed = 57). Stored fields: platform, action, PreviousRolloutPercentage, NewRolloutPercentage, CurrentRolloutPercentage, DayNumber, trackName (platform/action backfill in publish #501/#502). - ac-server-notification consumes 6 events (all but PlayStoreManualRolloutFailed) → in-app ToastMessage (IDs PUB-0108…PUB-0117), alert types Info/Success/Warning/Danger. No external channel. PlayStoreManualRolloutFailed is deliberately excluded: a manual failure surfaces inline in the synchronous API response, so it is logged by reporting (above) but not toasted.

Deployment and Infrastructure¶

Cloud Deployment¶

No new infrastructure. The periodic service runs in-process inside ac-service-publish; it relies on the existing Redis for the RedLock and the existing Kafka topic Publish. Ensure at least one publish-service replica runs the hosted service (lock guarantees single execution across replicas).

Self-Hosted Deployment¶

No new containers or compose changes. Requires the standard Redis + Kafka already present. New gateway scopes/license entries ship with the apigateway config; license-gated so the feature can be toggled per plan.

Note. This changes customer-facing behaviour, so the public documentation is tracked separately in BE-8725 and docusaurus#1214 ("App Rollout Information").

Testing and Quality Assurance¶

Test Strategy: - Unit: percentage/fraction rounding, manual & automatic validation rules, ResolveCurrentAutomaticDay, day-advance/terminal logic. - Integration: reconciliation against a mocked Play track (in-progress / halted / completed / version-code mismatch); upsert by track; due-rollout ElemMatch. - End-to-end: full plan 10→…→100 advancing across scheduler ticks; halt then resume; manual override; Kafka → reporting/notification assertions. - Regression: the follow-up fixes encode edge cases worth permanent tests, namely #491 (no null gap / relax 100%), #492 (skip past-day validation on active plan), #515 (same percentage on different days), #520 (execution period).

Quality Assurance: Standard PR review (CODEOWNERS), CI build, license/scope checks at the gateway as a release gate.

Performance Considerations¶

Requirements: - Each GET/POST performs a live Google Play track fetch, so latency and Play API quota are the dominant cost. Reads are user-initiated and infrequent. - The periodic job runs every 120 min, single-instance via RedLock, and processes only due rollouts.

Optimizations: - Google Play credentials are cached (GetCachedGooglePlayStoreCredentials). - Per-rollout try/catch isolates failures so one bad rollout doesn't block the batch. - Add a Mongo index to back the due-rollout ElemMatch (see Risks) before this scales to many active rollouts.

Maintenance and Support¶

Monitoring and Logging: Structured logs throughout ProcessAutomaticRollouts (start, per-step progression, stop/complete, and a per-rollout error log with org/profile/appVersion/track/day/plan). Activity-log entries (reporting) and toast notifications give product-level traceability per rollout transition.

Support and Maintenance: Watch Google Play API quota/errors; the rollout will simply retry on the next tick (state is reconciled, not assumed). The 120-min period means a step can land up to ~2 h after its nominal NextExecutionAt.

Design Decisions & Alternatives¶

Why server-side, not Fastlane (architecture rationale). The original analysis (BE-7299) proposed driving the rollout from the build pipeline: fastlane supply --track production --rollout "$percentage" ... (optionally with --skip_upload_*) re-run per day, scheduled by cron. That keeps rollout state implicit in the build step. The "Business Decisions" call (Burak Öztopuz, BE-7205) explicitly overrode this: App Release and Rollout Information are decoupled from Fastlane and moved to a server-side architecture, with data synchronised against Google Play when the user opens the rollout modal. The implemented design follows that decision: rollout state lives in the publish service (PlayStoreRolloutStateStoreObject), progression is driven by RolloutPeriodicService calling the Google Play Developer API directly, and every read reconciles against live Play data, rather than being a side effect of a Fastlane build. This is why there is no build-step / cron component in the final design.

Manual ⇄ Automatic switching semantics. The validation rules in Detailed Design enforce these, but the intended switch semantics (per the business decisions) are: - Manual → Automatic: the latest manual percentage becomes Day 1 of the automatic plan; remaining days may only be equal or higher (monotonic non-decreasing). E.g. manual at 60% → Day 1 = 60%, later days ≥ 60%; once a day hits 100% the remaining days are disabled. - Automatic → Manual: the latest automatic percentage pre-fills the manual slider. - In both directions the live Google Play percentage is the floor: the user can never set a value below the current GPC rollout, and for an active automatic plan the already-elapsed days are locked.

Phase scope: halt/resume pulled into Phase 1. Hold/Resume/Cancel were originally scoped as Phase 2 ("to be analysed and implemented later"). In the delivered feature, halt and resume are already implemented (UpdateRolloutStatus → Play status halted / inProgress, with IsAutomaticActive and NextExecutionAt updated and PlayStoreRolloutHalted / PlayStoreRolloutResumed events). So the Phase 1 surface is broader than the initial split. A full "Cancel" (tear down an in-flight rollout) is not part of this delivery; halt + manual override are the available stop paths.

Risks and Mitigation Strategies¶

Risk	Impact	Mitigation
No Mongo index on embedded rollout fields	`ElemMatch` collection scan grows with `AppVersions`	Add a partial/multikey index on `PlayStoreRolloutState.IsAutomaticActive` + `NextExecutionAt`.
Non-atomic read-modify-write upsert	Concurrent GET (reconcile) + scheduler write could clobber a track entry	Scheduler is RedLock-guarded; consider positional array updates or per-doc optimistic concurrency for the GET path.
Google Play API quota / transient errors	A step fails	Per-rollout try/catch + reconcile-on-read; retries next tick; failure events emitted.
120-min period drift	Step applied later than the nominal day boundary	Acceptable for staged rollout; documented. Reduce period if tighter timing needed.
Rounding to 2-decimal fractions	Sub-percent precision lost	Percentages constrained to whole numbers ≥10; acceptable for Play (10% granularity typical).
License/scope misconfiguration	Endpoint open or blocked	Paired `publish#` + `license#` scopes enforced at gateway (#341, #347).

Appendix¶

Repositories & PRs - ac-service-publish #462 (core), follow-ups: #487, #491, #492, #495, #501, #502, #515, #520. - ac-lib-core #328 – event constants + PublishPlatformTypeInvalid error code. - ac-server-apigateway #326 (routes), #341 (track-filtered endpoints + dedicated scopes), #347 (license checks). - ac-server-reporting #100 – activity-log consumer (PublishAction 55–61). - ac-server-notification #305 – toast notifications (PUB-0108…PUB-0117). - appcircle-docusaurus #1214 – public docs ("App Rollout Information").

State machine (automatic)

An automatic rollout reaches Completed by either terminal condition: the live fraction reaches 100%, or the scheduler advances past the final scheduled day (nextDay > AutomaticPercentages.Count) even if the last planned percentage is below 100%.

Draft ──set plan──► InProgress(day n) ──scheduler──► InProgress(day n+1) ──► … ──┐
   │                      │  ▲                                                    │
   │                      │  │                            reaches 100% ──────────►├──► Completed
   │                      │  │              final scheduled day reached ─────────►┘
   │                      │  └── resume (PATCH status, IsAutomaticActive=true) ──────┐
   │                      └──── halt (PATCH status, IsAutomaticActive=false) ──► Halted
   └── manual % ──► InProgress/Completed (clears the day plan)