Vault to OpenBao Migration¶

Scope: HashiCorp Vault 1.4.2 (GCS backend) → OpenBao 2.5.4 (PostgreSQL backend)
Environment: GKE, GCP KMS auto-unseal, Helm-managed installation

Overview¶

	Start	Intermediate	Final
App	HashiCorp Vault 1.4.2	HashiCorp Vault 1.9.2	OpenBao 2.5.4
Helm Chart	vault-0.6.0	vault-0.19.0	openbao-0.28.3
Storage	GCS (`appcircle-dev2-vault`)	GCS	PostgreSQL
Seal	GCP KMS (gcpckms)	GCP KMS (gcpckms)	GCP KMS (gcpckms)

Why OpenBao?¶

HashiCorp switched Vault to the BSL (Business Source License) starting with version 1.14.x. OpenBao is an open-source fork maintained by the Linux Foundation. GCP KMS auto-unseal is fully supported in OpenBao 2.x.

Why Switch from GCS to PostgreSQL?¶

OpenBao v2.0.0 removed cloud-specific storage backends (GCS, S3, Azure). Only Raft and PostgreSQL (v2.1.0+) are supported.

Prerequisites¶

1. Backup¶

BACKUP_DATE=$(date +%Y%m%d-%H%M%S)

gsutil -m cp -r \
  gs://appcircle-dev2-vault/* \
  gs://appcircle-backup/dev-vault-pre-pg-migration-${BACKUP_DATE}/

# Verify
gsutil du -sh gs://appcircle-backup/dev-vault-pre-pg-migration-${BACKUP_DATE}/

2. Check Vault Status¶

VAULT_POD=$(kubectl get pod -l app.kubernetes.io/name=vault -o jsonpath='{.items[0].metadata.name}')

kubectl exec ${VAULT_POD} -- vault status
# Sealed: false is expected

GCP KMS Permissions¶

Vault 1.9+ requires the cloudkms.cryptoKeys.get permission. This must be granted before the 1.9.2 upgrade. roles/cloudkms.viewer was added to all environments.

# DEV (project: dev-appcircle)
gcloud kms keys add-iam-policy-binding dev-appcircle-key \
  --keyring=dev-appcircle-ring \
  --location=global \
  --project=dev-appcircle \
  --member="serviceAccount:dev-appcircle-vault-sa@dev-appcircle.iam.gserviceaccount.com" \
  --role="roles/cloudkms.viewer"

# PREP (project: appcircle)
gcloud kms keys add-iam-policy-binding prep-appcircle-key \
  --keyring=prep-appcircle-ring \
  --location=global \
  --project=appcircle \
  --member="serviceAccount:prep-appcircle-vault@appcircle.iam.gserviceaccount.com" \
  --role="roles/cloudkms.viewer"

# PROD (project: appcircle)
gcloud kms keys add-iam-policy-binding prod-appcircle-vault \
  --keyring=prod-appcircle-ring \
  --location=global \
  --project=appcircle \
  --member="serviceAccount:prod-appcircle-vault@appcircle.iam.gserviceaccount.com" \
  --role="roles/cloudkms.viewer"

Note

The permission is bound to the KMS key resource, not the project. This follows the principle of least privilege.

Stage 1: Vault 1.4.2 → 1.9.2¶

Remove Existing Resources¶

cd ./export/appcircle-helm/charts/vault/templates
kubectl --context dev2-appcircle delete -f .

Chart.yaml Changes¶

# Before
- name: vault
  version: 0.6.0
  repository: https://helm.releases.hashicorp.com

# After
- name: vault
  version: 0.19.0
  repository: https://helm.releases.hashicorp.com

Helm Upgrade¶

pwsh export.ps1 google dev

# Pull chart
helm repo add hashicorp https://helm.releases.hashicorp.com
helm pull hashicorp/vault --version 0.19.0 -d appcircle-helm/charts/

cd ./export/appcircle-helm/charts/vault/templates
kubectl --context dev2-appcircle apply -f .

# OnDelete strategy — pod must be restarted manually
kubectl delete pod appcircle-v1-vault-0 -n dev-appcircle

Stage 2: GCS to PostgreSQL Migration¶

Create Cloud SQL Database¶

Create a new database and user on the existing Cloud SQL instance:

# Create database
gcloud sql databases create vault \
  --instance=<instance-name> \
  --project=dev-appcircle

# Create user (use a strong, unique password)
gcloud sql users create vault-user \
  --instance=<instance-name> \
  --password=<strong-password> \
  --project=dev-appcircle

Create PostgreSQL Table¶

CREATE TABLE IF NOT EXISTS vault_kv_store (
  parent_path TEXT COLLATE "C" NOT NULL,
  path        TEXT COLLATE "C",
  key         TEXT COLLATE "C",
  value       BYTEA,
  CONSTRAINT pkey PRIMARY KEY (path, key)
);

CREATE INDEX IF NOT EXISTS parent_path_idx ON vault_kv_store (parent_path);

Kubernetes Secret — Migration¶

Warning

Do not use these special characters in the password: @ / : ? # + &

kubectl create secret generic vault-pg-secret \
  -n dev-appcircle \
  --from-literal=connection_url="postgres://USER:PASS@HOST:5432/DBNAME?sslmode=require"

Migration Job¶

Since OpenBao v2.x does not support the GCS backend, the migration is done using the Vault 1.14.0 binary.

apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-migrate-config
  namespace: dev-appcircle
data:
  migrate.hcl: |
    storage_source "gcs" {
      bucket = "appcircle-dev2-vault"
    }
    storage_destination "postgresql" {
      connection_url = "${PG_CONN_URL}"
      table          = "vault_kv_store"
    }
---
apiVersion: batch/v1
kind: Job
metadata:
  name: vault-migrate-gcs-to-pg
  namespace: dev-appcircle
spec:
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: appcircle-v1-vault
      volumes:
        - name: google-secret
          secret:
            secretName: vault-google-secret
        - name: migrate-config
          configMap:
            name: vault-migrate-config
      containers:
        - name: vault-migrate
          image: hashicorp/vault:1.14.0
          command:
            - "/bin/sh"
            - "-c"
            - |
              sed "s|\${PG_CONN_URL}|${PG_CONN_URL}|g" /migrate/migrate.hcl > /tmp/migrate.hcl
              vault operator migrate -config=/tmp/migrate.hcl
          env:
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: /vault/userconfig/vault-google-secret/credentials.json
            - name: GOOGLE_PROJECT
              value: dev-appcircle
            - name: GOOGLE_REGION
              value: global
            - name: PG_CONN_URL
              valueFrom:
                secretKeyRef:
                  name: vault-pg-secret
                  key: connection_url
          volumeMounts:
            - name: google-secret
              mountPath: /vault/userconfig/vault-google-secret
              readOnly: true
            - name: migrate-config
              mountPath: /migrate

kubectl apply -f migration-job.yaml
kubectl logs job/vault-migrate-gcs-to-pg -n dev-appcircle -f

Cleanup

The Job and Secret are one-time resources. Delete them after the migration completes.

kubectl delete job vault-migrate-gcs-to-pg -n dev-appcircle
kubectl delete secret vault-pg-secret -n dev-appcircle
kubectl delete configmap vault-migrate-config -n dev-appcircle

Stage 3: Vault 1.9.2 → OpenBao 2.x¶

Chart.yaml Changes¶

# Before
- name: vault
  version: 0.19.0
  repository: https://helm.releases.hashicorp.com

# After
- name: openbao
  version: 0.28.3
  repository: https://openbao.github.io/openbao-helm/
  alias: vault          # existing values files work without changes
  condition: vault.enabled

Kubernetes Secret — PostgreSQL Config¶

The storage block is stored in a Kubernetes Secret instead of a ConfigMap, so the connection URL is not stored in plaintext. Because the HCL parser does not expand shell environment variables, the connection details are written directly into the Secret.

kubectl create secret generic vault-pg-config \
  -n dev-appcircle \
  --from-literal=pg.hcl='storage "postgresql" {
  connection_url = "postgres://USER:PASS@HOST:5432/DBNAME?sslmode=require"
  table          = "vault_kv_store"
  ha_enabled     = "false"
}'

Values Structure¶

vault:
  server:
    extraEnvironmentVars:
      GOOGLE_REGION: global
      GOOGLE_PROJECT: <project>
      GOOGLE_APPLICATION_CREDENTIALS: /openbao/userconfig/vault-google-secret/credentials.json
    extraArgs: "-config=/openbao/userconfig/vault-pg-config/pg.hcl"
    extraVolumes:
    - type: 'secret'
      name: 'vault-google-secret'
    - type: 'secret'
      name: 'vault-pg-config'
    standalone:
      enabled: true
      config: |
        ui = true

        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }

        seal "gcpckms" {
          project    = "<project>"
          region     = "global"
          key_ring   = "<keyring>"
          crypto_key = "<key>"
        }
        # no storage block — loaded via extraArgs from the vault-pg-config secret

Warning

The OpenBao chart does not merge extraVolumes lists — it replaces them. Define the full list only in the most specific (lowest) values file instead of splitting it across multiple overrides.

Helm Upgrade¶

# Remove existing resources
cd ./export/appcircle-helm/charts/vault/templates
kubectl --context dev2-appcircle delete -f .

# Pull chart
helm repo add openbao https://openbao.github.io/openbao-helm/
helm pull openbao/openbao --version 0.28.3 -d appcircle-helm/charts/

# Upgrade
pwsh export.ps1 google dev
cd ./export/appcircle-helm/charts/vault/templates
kubectl --context dev2-appcircle apply -f .

# OnDelete strategy — pod must be restarted manually
kubectl delete pod appcircle-v1-vault-0 -n dev-appcircle

Verification¶

# Check pod status
kubectl get pod -n dev-appcircle -l app.kubernetes.io/name=vault

# Check volumes
kubectl get pod appcircle-v1-vault-0 -n dev-appcircle \
  -o jsonpath='{.spec.volumes[*].name}' | tr ' ' '\n'
# vault-google-secret and vault-pg-config should appear

# OpenBao status
kubectl exec appcircle-v1-vault-0 -n dev-appcircle -- bao status
# Sealed: false, Version: 2.5.4

Known Issues¶

Issue	Cause	Fix
`unknown storage type gcs`	GCS was removed in OpenBao v2.0.0	Use PostgreSQL or Raft
`cloudkms.cryptoKeys.get denied`	New permission required in Vault 1.9+	Add `roles/cloudkms.viewer` before upgrade
`cannot parse ${PG_CONN_URL}`	HCL parser does not expand env vars	Move storage block to a Secret, load with `extraArgs`
Credentials not found `/openbao/userconfig/...`	`extraVolumes` list lost after override	Redefine the full list in the overriding values file
Pod keeps running with old config	StatefulSet `OnDelete` update strategy	Delete the pod manually after `helm upgrade`