Appcircle Environment Migration Plan (for Disaster Recovery)

Overview

Appcircle uses Google Cloud Platform for cloud services and uses below components for each environment.

This document covers steps to migrate a specific environment to be migrated to another project in GCP (e.g dev-appcircle) in case of a DR scenario happened.

Environment	GCP Project	Region	Multi-Region
Local	Appcircle	europe-west-1	❌
Dev	Appcircle	europe-west-1	❌
Prep	Appcircle	europe-west-3	❌
Prod	Appcircle	europe-west-4	✅ (eu)

GCP Component Usage	local	dev	prep	prod
1. SQL (Postgresql)	❌	✅	✅	✅
2. Compute Engine (MongoDB)	❌	❌ (hosted in OVH)	✅	✅
3. Storage Buckets	✅	✅	✅	✅
4. Artifact Registry (Docker Images)	❌	✅	✅	✅
5. Kubernetes Engine	❌	✅	✅	✅
6. Cloud CDN & Load Balancers	✅	✅	✅	✅
7. Service Accounts (IAM)	✅	✅	✅	✅
8. Key Management	✅	✅	✅	✅
8. SSL Certificates	✅	✅	✅	✅

Appcircle Server uses below gcp components within each component to be run properly.

Server Component	GCP Components	Usage
Postresql DB	SQL Postgres instance	Keycloak & CodePush uses Postgresql for DB
MongoDB	Compute Engine, Static IP Address	APIs uses mongodb as main storage and access via dns record. The dns records points to the static ip which forwards to VMs in compute engine.
Resource Server	Storage Buckets, Cloud CDN & Load Balancer & Static IP & SSL Certificates, Service Account	Resource Server uses buckets for file storage via cdn records with a service account in https. CDN dns records points to the static ip which forwards to load balancer which targets storage buckets as backend buckets.
Vault	Storage Bucket, Key Management, Service Account	Vault uses storage buckets as file system and KMS key for autosealing.

Since Appcircle Server is deployed to kubernetes cluster below components are also used; - GCP Docker Artifact Registry - Kubernetes Engine - Compute Engine (VM, Disk) - Static IP

In order to release app updates smoothly, CI/CD pipelines must be configured for Jenkins. - GCP Docker Artifact Registry - Service Account to use new artifact registry and update kubernetes cluster

Migration Steps:

1. Postresql

All environment's sql instances have daily backup enabled. So in case of a dr scenario, new sql instance will be restored with the latest successfull backup. If source sql is available, having a manual backup first should be considered for min RTO.

Prerequisites : - Target sql instance must be created before the migration. - Script runner must have required permissions for sql instance restoring.

Steps : - Create a manual Backup from source instance (Optional) - Find the latest successfull backup of the source instance - Restore the target sql instance with the source backup

Script:

sh ./postgres/migrate.sh -h
sh ./postgres/migrate.sh -s source-sql-instance -p source-gcp-project -t target-sql-instance -P target-gcp-project -c (optional : create manual backup for source first)

2. MongoDB

Since all environments has different setups for mongodb, migration steps differ per environment.

Environment	Hosting	Migration strategy
dev	hosted in OVH	No migration strategy
prep	hosted in GCP VMs with no dr backup configured	Creating disk snapshots manually then restoring vms.
prod	hosted in GCP VMs with dr backup configured	Restoring from vault backup and data from the source vms's dr backups.

Prerequisites : - Script runner must have required permissions for vm and disk restoring and vault backup access (for prod).

Steps for Restoring with Vault Backup (prod): - Find the vault backup and data-sources for each VM - Find the latest backup of each VM and their source configurations (scopes & tags) - Restore each VM in the target project and region with their latest backup and source configs

Script (params should be edited):

./mongo/migrate-dr-backup.sh

Steps for Restoring with creating snapshot (prep): - Create a snapshot of VM disk - Move snapshot to target project - Restore VM with snapshot in target project

Script (params should be edited):

./mongo/migrate-snapshot.sh

Post Restore Steps: - After restoring new instances with either option, static ip (internal - GCE_ENDPOINT) creations must be dne for each new instance. - DNS records should be updated with new ip addresses.

:warning: Warning: - Both scripts have not been tested and may have errors in the first run.

3. Storage Buckets

Bucket migration can be made via using GCP Transfer Jobs. Each environment has buckets with their names prefixed below. Since prod buckets are multi-region and HA enabled, migrating buckets may not be needed in case of a DR scenario.

Environment	Bucket Prefix	Region	Is Multi Region
local	appcircle-local-	europe-west1	❌
dev	appcircle-dev-	europe-west1	❌
prep	appcircle-prep-	europe-west3	❌
prod	appcircle-prod-	eu	✅

Since bucket names should be unique, it is not possible to copy a bucket to another project or region with the same name. So when copying a bucket to a target, new bucket names are designed to be versioned like appcircle-dev{version}. Some buckets are accessed via DNS records from the Appcircle Server services. So in order to migrate properly, bucket's CDN records, Load balancer and static ip configurations should also be migrated. Once all migration process completes, DNS records should be updated (in Cloudflare). Also a Service Account should be created with necessary roles (roles/storage.admin) to administrate new buckets for api usages.

Prerequisites : - Script runner must have required permissions for transfer job creation. - Target project's google managed storage transfer SA (project-{target-project-id}@storage-transfer-service.iam.gserviceaccount.com) must have access to source buckets with these roles. (roles/storage.objectViewer, roles/storage.legacyBucketReader) - mongo cli installed in order to update persisted bucketnames in mongo

Steps : - Start transfer job for each source bucket to copy data for the environment specified. - Check for Service Account existence, create if none is exist. - Sync roles of target SA with source SA for new buckets. When all jobs completed; - Create cdn records within GCP for the buckets accessed via services. Give google managed CDN SA (service-$PROJECT_NUMBER@cloud-cdn-fill.iam.gserviceaccount.com) to access to view buckets' objects. (roles/storage.objectViewer) - Create backend buckets of CDN with signing-keys. (use "Restrict access using signed URLs and signed cookies"). (check existing values in existing helm values for source environment) - Create a loadbalancer for cdn buckets (with ssl certificates). - Assign a static ip for the lb. - Update A records of the cdn DNS records (after all migration completes). - Add CORS policy to *-resource-temp to enable uploading files via signed urls. - :warning: Data update : Bucketnames stored in mongodb (resources collection) must also be updated with new bucket names. (make sure mongodb migration completed first.)

Script:

#Coping buckets;
./buckets/migrate.sh -h
./buckets/migrate.sh -p source-gcp-project -P target-gcp-project -e environment -v bucket-version

#Creating cdn, lb, etc.;
./buckets/cdn-lb.sh -h
./buckets/cdn-lb.sh -P target-gcp-project -e environment -v bucket-version -s ssl-certificate-name -skv cdn-bucket-sign-key-value

#add CORS policy to *-resource-temp bucket
./cors.sh -e environment -v bucket-version

#Mongodb resource bucketname update (after mongodb migration)
./update-bucketnames-mongo.sh -h
./update-bucketnames-mongo.sh -e environment -v bucket-version -mcs mongo-conn-string

4. Docker Images

Appcircle's private docker registry is located at europe-west1 region in Appcircle project at GCP. In case of a dr scenario, images can be copied to another registry in another region or project. Copying docker images can be made per environment also. Each environment has their own docker tag system, so when migrating images to another registry, only related tagged images will be copied to target registry. Custom tagged images also can be transferred with scripts -f option.

Environment	Image Prefix
dev	alpha-latest
prep	beta-latest
prod	latest
Self Hosted	vX.Y.Z

Prerequisites : - Script runner must have required permissions for artifact registry copying. - Target registry should be created before migration. - In order to copy multi arch images properly, crane should be installed on the host that runs migration script.

Steps : - Find images having desired tags in source repository - Copy from source to target repository via crane cp.

Script:

sh ./docker-images/migrate.sh -h
sh ./docker-images/migrate.sh -p source-gcp-project -P target-gcp-project -r source-repo-region -R target-repo-region -n registry-name -f tag-to-transfer

Note: - Docker registry can be configured as multi-region for delegating HA and multi replication processes to GCP.

5. Vault

Vault migration should be handled carefully since it has sealing logic and a failing migration can not be irreversable. In order to migrate vault properly, storage bucket and kms key migration should be considered in case of DR scenario. Bucket is used as filesystem for vault and kms keys are used for auto sealing vault.

Since prod buckets are multi-region and HA enabled, migrating bucket may not be needed in case of a DR scenario. Also all dev, prep and prod kms rings are located globally and highly available, migrating kms keys also may not be needed in case of a DR scenario.

Prerequisites : - Script runner must have required permissions for bucket copying (via transfer job), kms key (creation), service account creation (and cross role iam role binding). - docker installed to run temporary container for migrating auto seal stanza for target project. - 3 recovery key and the root token must be provided for vault to migrate auto seal configuration.

Steps : - Configure target kms ring and key (and bucket) - Create SA if not exists in target project - Set source vault's SA cross project policy binding to use target project's key(roles/cloudkms.cryptoKeyEncrypterDecrypter) - Set target vault's SA policy for target bucket (roles/storage.objectAdmin) and target kms key (roles/cloudkms.cryptoKeyEncrypterDecrypter). - Migrate vault auto sealing, providing 3 recovery keys (to unseal manually once to migrate autosealing) and root token (for manually sealing) - Using source vault's bucket, source vault's kms key as disabled and target's key as enabled run vaults auto migration. Once the migration is completed, seal the vault. - Copy source bucket to target (via bucket migration). In this stage, bucket has sealed with target kms keys and ready to be run in the target. - Migrate source vault's kms keys back with disabling target and enabling source key this time with same steps. - Create key for target SA. - Generate new vault config for migration.

Script:

sh ./vault/migrate.sh -h
sh ./vault/migrate.sh -p source-gcp-project -P target-gcp-project -e environment -v bucket-version --source-sa-cred 'source-sa-key.json' --root-token "token" --recovery-key-1 "vault-rec-key-1" --recovery-key-2 "vault-rec-key-2" --recovery-key-3 "vault-rec-key-3"

6. Kubernetes Cluster

Migrating cluster is a straightforward process with helm for cloud release. But there are some dependencies that must be solved before exporting helm files and applying them to cluster. Cluster needs below static ip addresses and service accounts to be set before migration:

* stands for environment.

Ip Address Name	Type
*-api-internal-static-ip	INTERNAL (SHARED_LOADBALANCER_VIP)
*-ingress-static-ip	EXTERNAL (global)
*-webevent-redis-static-ip	EXTERNAL (regional)
*-external-ingress-static-ip (may be obsolete)	EXTERNAL (regional)

Service Account Name	Role
*-appcircle-minio	should be available w/ #3.Storage Buckets
*-appcircle-vault	should be available w/ #5.Vault

Prerequisites : - Cluster must be created before migration. - Script runner must have required permissions for applying cluster changes, (and cluster creation). - docker should be installed in order to generate helm values file.

Steps : - Create cluster if not already exist - Prepare static ip configurations. - Generate helm values and export. - Update cluster

Script:

#Create cluster
sh ./gke/cluster.sh #edit params in the file
#Create static ips
sh ./gke/static-ip.sh #edit params in the file
#Create helm values
sh ./gke/helm-values.sh #edit params in the file

7. DNS A Records

Below DNS records' A values should be updated after all migration steps completed with the related GCP IP address values.

GCP IP Address Name	Migration Source	DNS records
{environment}-resource-cdn-static-ip	Buckets' Migration (CDN creation)	{environment}-cdn.appcircle.io {environment}-cdn1.appcircle.io {environment}-cdn2.appcircle.io {environment}-cdn3.appcircle.io {environment}-cdn4.appcircle.io {environment}-cdn5.appcircle.io
{environment}-appcircle-ingress-static-ip	GKE Migration (Static IP creation)	{environment}-my.appcircle.io {environment}-api.appcircle.io {environment}-auth.appcircle.io {environment}-dist.appcircle.io {environment}-hook.appcircle.io *.{environment}-store.appcircle.io
{environment}-appcircle-external-ingress-static-ip	GKE Migration (Static IP creation)	{environment}-store-domain.appcircle.io
{environment}-appcircle-webevent-redis-static-ip	GKE Migration (Static IP creation)	{environment}-webeventredis.appcircle.io

8. Jenkins Pipelines

In order to release app updates smoothly, CI/CD pipelines must be configured for Jenkins. - GCP Docker Artifact Registry (GCLOUD_DOCKER_REGISTRY, GCLOUD_DOCKER_REGISTRY_PATH) - K8S Cluster Vars (CLUSTER_REGION, CLUSTER_NAME, K8_NAMESPACE) - Service Account to use new artifact registry and update kubernetes cluster (credentialsId: 'PipelineGcloudCredentials')