Appcircle Disaster Recovery Plan¶
Effective date: 2025-04-10 Last reviewed: 2026-02-05 Policy owner: Engineering Management (Security Program Owner)
1. Purpose¶
This document describes Appcircle's Disaster Recovery (DR) strategy for restoring platform services in the event of a significant infrastructure failure. It covers the scope, recovery objectives, architecture overview, recovery procedures, and testing approach.
2. Scope¶
This DR plan covers Appcircle's cloud-hosted SaaS platform, including:
- Application services (API, Web, Background Workers)
- Data stores (Relational databases, Document databases)
- Secrets management (Vault)
- Object storage (Artifacts, build outputs, resources)
- Container registry (Application images)
- Kubernetes orchestration layer
- DNS and CDN infrastructure
Self-hosted (on-premise) deployments are excluded; those are managed by the customer's own DR processes.
3. Recovery Objectives¶
| Metric | Target | Notes |
|---|---|---|
| RTO (Recovery Time Objective) | < 16 hours | Full platform restoration |
| RPO (Recovery Point Objective) | < 24 hours | Based on daily automated backups |
These targets are based on documented recovery procedures and automated migration scripts. Partial failures (single component) have significantly shorter recovery times.
4. Infrastructure Overview¶
Appcircle runs on Google Cloud Platform (GCP) with the following high-availability characteristics:
| Capability | Production | Non-Production |
|---|---|---|
| Multi-region storage | Yes | No |
| Automated daily backups | Yes | No |
| Cross-region backup replication | Yes | No |
| High-availability secrets management | Yes (global KMS) | Yes (global KMS) |
5. DR Strategy¶
Appcircle's DR strategy is based on cross-project migration within GCP. In the event of a disaster affecting the primary GCP project, all services are restored to a pre-designated secondary project in a different region.
5.1 Backup Mechanisms¶
| Component | Backup Method | Frequency | Retention |
|---|---|---|---|
| Relational databases (PostgreSQL) | GCP Cloud SQL automated backups | Daily + on-demand | Per retention policy |
| Document databases (MongoDB) | GCP Backup & DR vault backups | Daily | Per retention policy |
| Object storage (Buckets) | Multi-region replication (prod); Transfer jobs (non-prod) | Continuous (prod) | Per retention policy |
| Container images | Artifact Registry | On every deployment | Retained per version policy |
| Secrets (Vault) | GCS backend with KMS auto-seal; global KMS keys | Continuous | Per retention policy |
5.2 Recovery Procedures Summary¶
Recovery follows a defined sequence to respect component dependencies:
| Step | Component | Procedure | Dependencies |
|---|---|---|---|
| 1 | PostgreSQL | Restore from latest automated backup to target instance | Target SQL instance pre-created |
| 2 | MongoDB | Restore VMs from Backup & DR vault backups with source configurations | Backup vault access; static IP allocation |
| 3 | Object Storage | Transfer jobs to copy buckets; reconfigure CDN, load balancers, SSL | Target project; DNS update preparation |
| 4 | Container Registry | Copy tagged images to target registry using multi-arch-aware tooling | Target registry pre-created |
| 5 | Vault | Migrate auto-seal configuration to target KMS; transfer storage backend | Recovery keys available; target KMS provisioned |
| 6 | Kubernetes Cluster | Deploy via Helm with updated configurations; assign static IPs | All data stores restored; service accounts configured |
| 7 | DNS | Update A records for API, CDN, Auth, Distribution, and Store endpoints | All services verified in target environment |
| 8 | CI/CD Pipelines | Reconfigure deployment pipelines for target registry and cluster | Target cluster and registry operational |
5.3 Automation¶
Migration procedures are supported by automated scripts covering:
- Cross-project database backup and restore
- Storage bucket transfer with CDN/LB reconfiguration
- Container image copying (multi-architecture aware)
- Vault seal migration between KMS keys
- Kubernetes cluster provisioning and Helm deployment
- Static IP allocation and DNS record generation
6. Roles and Responsibilities¶
| Role | Responsibility |
|---|---|
| Engineering Management | DR plan ownership; initiates DR activation; stakeholder communication |
| Platform / DevOps Team | Executes recovery procedures; monitors restoration progress |
| Development Team | Validates application functionality post-recovery |
| QA Team | Runs verification tests on restored environment |
7. DR Activation Criteria¶
The DR plan is activated when any of the following occur:
- GCP project-level outage exceeding 8 hours with no ETA from Google
- Data corruption affecting production databases with no viable in-place recovery
- Security incident requiring full environment isolation and rebuild
- Regional infrastructure failure affecting multiple availability zones
Activation is authorized by Engineering Management.
8. Communication Plan¶
| Phase | Audience | Channel | Timing |
|---|---|---|---|
| Detection | Internal engineering | Internal chat + incident channel | Immediate |
| Activation | Leadership + Engineering Management | Direct notification | Within 30 minutes |
| Progress updates | Affected customers | Status page + customer Slack workspace | Every 2-4 hours during recovery |
| Resolution | All stakeholders | Post-incident report (status page + customer Slack workspace) | Within 96 hours of resolution |
9. Testing Strategy¶
Appcircle validates its DR readiness through a layered testing approach:
| Test Type | Description | Frequency |
|---|---|---|
| Tabletop Exercise | Walk-through of DR scenarios with engineering team; identify gaps in procedures and documentation | Quarterly |
| Component Restore Drill | Restore individual components (e.g., database, storage) to an isolated target project to validate scripts and procedures | Semi-annually |
| Integrated DR Simulation | Full-stack migration of a non-production environment to a secondary project | Annually |
Test results are documented and used to update this plan.
10. Plan Maintenance¶
- This plan is reviewed and updated at least annually, or after any significant infrastructure change.
- Lessons learned from DR tests and real incidents are incorporated into plan updates.
- All changes are version-controlled and reviewed by Engineering Management.
Note
This document describes Appcircle's disaster recovery strategy at a high level for external stakeholders. Detailed internal procedures, automation scripts, and infrastructure-specific configurations are maintained separately and may be shared under appropriate agreements when required.
11. Related Policies¶
- Information Security Policy
- Incident Response Plan
- Business Continuity Plan
- Data Retention and Disposal Policy
- Logging and Monitoring Policy