Skip to content

Create DR documentation for Vault

In order to restore vault in another project inside gcloud we need to migrate below gcp components;

  • KMS keys (used for auto sealing) : not possible to copy to another project
  • Storage Bucket (used as backend) : possible to copy to another project
  • Vault itself (helm)

With these components in mind we have different options to restore the vault in another project;

~~Solution 1 : Migrate only bucket and keep using existing kms keys~~

~~Since we cannot copy kms keys to another project, in this solution we just copy the bucket (data) to another project with vault itself but keep the existing kms keys in the source project.~~

~~⚠️ The provided account service should have cross project permissions.~~

~~This solution has also two possible options itself:~~

  1. ~~Copy bucket with a transfer job as defined in BE-6773 (check if vault needs to be stopped before this). Run the vault in the target project with using target bucket using source kms keys.~~
  2. ~~Use vault operator migrate~~~~ : defining source and destination in a hcl file, vault operator migrate copies data from source to destination.~~

~~Since we will transfer data between gcp buckets, first options seems more optimal without including the vault itself in the solution.~~

~~Solution 2 : Using shamir during migration~~

~~In this solution, we first stop using kms and use shamir for migration.~~

  • ~~Remove kms source, restart and unseal & seal with shamir.~~
  • ~~Copy bucket to target and run target vault with same shamir and unseal.~~
  • ~~Update target vault to use kms created in target project.~~

~~Solution 3 : Export source kv and import in newly created target vault (like dump)~~

~~In this solution, we don't copy buckets either. Using a tool like ~~~~medusa~~~~ we export all keys in the source to an encrypted file. Deploy a vault with new kms and an empty bucket in the target project and then import all keys from exported file.~~

~~⚠️ In order to export and import, vault api must be accessible.~~

Solution 4 : Using vault operator seal -migrate

In this solution, we let vault to handle seal migration. Since version 1.5.1 vault operator can handle seal migrations automatically.

Documentations:

https://developer.hashicorp.com/vault/docs/v1.10.x/concepts/seal

https://developer.hashicorp.com/vault/docs/v1.10.x/concepts/seal#seal-migration

https://developer.hashicorp.com/vault/docs/v1.10.x/concepts/seal#migration-post-vault-1-5-1

Steps:

  • Create kms ring, keys and bucket in the target project/region.
  • Create a service account for target project that has;
  • object admin on target bucket
  • kms key encryptdecrytor on target key
  • Give source project's service account to access target kms key (temporarily).
  • Scale down the source vault to 0 in kubernetes cluster (if any running).
  • Seal the source vault using the source project's kms. (needs verification if can be made while running unsealed).
  • Run auto kms migration (source kms -> target kms) with vault image using the source bucket. Once migration is completed seal the vault.
  • Transfer source vault bucket contents to target project. (if transfer will be made via SA, sa needs to be authorised on source bucket (object viewer).
  • Revert auto kms migration (target kms -> source kms) for the source vault with source bucket (and scale up the source vault in the k8s cluster if needed).
  • Run vault on the target project with the target kms key and bucket.
## Source KMS
seal "gcpckms" {  
    project     = "appcircle"
    region      = "europe-west1"
    key_ring    = "local-appcircle-ring"
    crypto_key  = "local-appcircle-key"
    disabled    = "true" #-> should be disabled
}

## Target KMS
seal "gcpckms" {
    project     = "dev-appcircle"
    region      = "global"
    key_ring    = "local-dev-appcircle-ring"
    crypto_key  = "local-dev-appcircle-key"
}

## Source Bucket
storage "gcs" {
    bucket = "local-dev-appcircle-vault"
}
  • In order to migrate seal from kms to kms, vault needs to be unsealed with recovery keys at first time like;
vault operator unseal -migrate
  • Once the vault is unsealed, vault operator should handle seal migration itself. once the migration is completed, the old gcpkms config can be removed and vault can be restarted with auto-sealing again for target.
## Target KMS
seal "gcpckms" {
    project     = "dev-appcircle"
    region      = "global"
    key_ring    = "local-dev-appcircle-ring"
    crypto_key  = "local-dev-appcircle-key"
}

## Target Bucket
storage "gcs" {
    bucket = "local-dev-appcircle-vault"
}
  • In order to revert kms keys for the source vault, re-run auto kms migration with disabling the target kms and enabling the source key back again with source bucket. Once the migration completed, source vault should be able to auto seal with the original vault configuration.
## Source KMS
seal "gcpckms" {  
    project     = "appcircle"
    region      = "europe-west1"
    key_ring    = "local-appcircle-ring"
    crypto_key  = "local-appcircle-key"
}

## Target KMS
seal "gcpckms" {
    project     = "dev-appcircle"
    region      = "global"
    key_ring    = "local-dev-appcircle-ring"
    crypto_key  = "local-dev-appcircle-key"
    disabled    = "true" #-> should be disabled this time
}

## Source Bucket
storage "gcs" {
    bucket = "local-dev-appcircle-vault"
}
  • Revoke source kms decryptor (and source bucket object viewer if authorised) from the new SA.

Bucket sizes (as of 07.08.2025 16:30 +3))

local : 0,53 MB

dev : 2,22 GB

prep : 96.36 MB

prod : 2.17 GB

➜  ~ gcloud storage du gs://appcircle-local-vault --project=appcircle --summarize
557530       gs://appcircle-local-vault
➜  ~ gcloud storage du gs://appcircle-dev-vault --project=appcircle --summarize
2382744600   gs://appcircle-dev-vault
➜  ~ gcloud storage du gs://appcircle-dev-vault --project=appcircle --summarize
➜  ~ gcloud storage du gs://appcircle-prep-vault --project=appcircle --summarize
101030305    gs://appcircle-prep-vault
➜  ~ gcloud storage du gs://appcircle-prod-vault --project=appcircle --summarize
2332578251   gs://appcircle-prod-vault

Metadata