Skip to content

OpenBao Migration & Env Bootstrap – Technical Design Document

Introduction

Purpose: This document defines the technical design for migrating Appcircle’s secrets backend from HashiCorp Vault to OpenBao (Vault-compatible OSS direction) and incrementally removing the OpenSecrets dependency.

The work is split into two ordered stages:

  1. Stage 1 — Runtime secrets migration: Services that currently fetch secrets at runtime (direct Vault API usage) must be switched to OpenBao with minimal behavioral change.

  2. Stage 2 — Startup env/config bootstrap: Helm/Compose-provided base env values and startup secrets (e.g., DB connection strings) will be generated dynamically at service startup time by fetching required values from OpenBao and rendering them into .env/config artifacts.

This approach reduces vendor/licensing risk while keeping service changes minimal.

Scope:

  • Migrate Vault KV v1 data and access patterns to OpenBao KV v1.

  • Introduce a standardized env/config bootstrap mechanism that works in:

  • Kubernetes/Openshift (Helm)

  • Self-hosted (Docker Compose/Podman)
  • Roll out the new bootstrap pattern across ~35 microservices (≈30 .NET, 3 Node.js, 2 Java).

Note

Related ADR: ADR-0007 (Accepted) — OpenBao chosen as the Vault replacement due to licensing/vendor risks.

System Overview

High-Level Description: Current secret consumption patterns:

  • Runtime secrets: Some services fetch secrets during runtime by calling Vault APIs (KV v1).
  • Startup configuration: Some secrets/env values are injected at startup via Helm/Compose (e.g., DB connection). OpenSecrets currently bridges Vault secrets into runtime for this model.

Target state (incremental):

  1. Runtime secrets: Services that call Vault at runtime will be switched to OpenBao (KV v1) while preserving existing behavior.
  2. Startup configuration: Services will start using generated env/config artifacts produced at startup time from OpenBao, eliminating the need for OpenSecrets in the steady state.
  3. Application coupling: Prefer platform-managed bootstrap patterns so services do not need to embed Vault/OpenBao client logic.

Architecture

Vendor-agnostic principle (required):
In the target design, application services must not know which secrets backend is used (Vault/OpenBao/other). Services will integrate only with a vendor-agnostic secrets access interface (a stable contract).

  • This interface is responsible for authentication, authorization, and translating requests to the configured backend.
  • The backend can change (Vault → OpenBao, or customer-provided backend later) without requiring service-level code changes.

Note

The concrete deployment shape of this interface (central service, per-environment gateway, or library wrapper) will be finalized during Phase 1. This TDD treats it as a mandatory logical component.

System Architecture:

The design consists of two main parts:

Secrets backend migration: Vault → OpenBao

Env bootstrap: render templates into env/config artifacts consumed at service start

Data Flow A — Runtime secrets (Stage 1)

  1. Service requests a runtime secret via the vendor-agnostic secrets gateway (stable contract).
  2. The gateway forwards the request to OpenBao (backend). Authentication/authorization mechanisms for this hop will be finalized later.
  3. The response from OpenBao is returned to the service in the expected format, and the service continues operating without knowing the underlying secrets backend.

Data Flow B — Bootstrap & initial secret provisioning (Stage 2)

Goal: Establish initial/derived env values and ensure required secrets exist for both internal and external components before steady-state runtime secret reads.

  1. Services start with a minimal set of seed env values.
  2. A bootstrap service/job runs to determine/derive additional env values and identify missing required secrets.
  3. For internal components that do not require external dependencies, bootstrap generates missing secrets (e.g., passwords) and writes them to the system (preferably OpenBao, otherwise via an internal mechanism that will later be mapped to OpenBao).
  4. For components such as Redis and other third-party services that cannot directly fetch secrets at runtime, secrets/env are injected at startup (e.g., via init container) using the generated/derived values.
  5. For external service secrets, bootstrap may need to fetch values from an external system/engine depending on the customer environment. Bootstrap normalizes these values into env/config form and stores them so initial state is established.
  6. Bootstrap writes the resulting secrets to OpenBao (when available) and then exits. On re-run, it checks whether data already exists and either:
    • does nothing and exits, or
    • re-generates/re-provisions when explicitly forced.
  7. After bootstrap completes, the platform operates in steady state using Data Flow A (runtime secrets).

Technology Stack:

  • Services: .NET (~30), Node.js (3), Java (2)
  • Runtime: Kubernetes (Helm), Docker Compose
  • Secrets: OpenBao (KV v1)
  • Observability (baseline expectation): Prometheus metrics endpoints where applicable, centralized logs

External Dependencies:

  • OpenBao (deployed and operated by Appcircle)
  • Existing platform dependencies that consume secrets (e.g., Keycloak) may be configured via generated env/config.

Key Components

Component 1 – Vendor-Agnostic Secrets Access Interface (Logical)

  • Purpose: Provide a stable contract for secret reads so services are not coupled to Vault/OpenBao.

  • Responsibilities:

    • Authenticate and authorize callers.
    • Enforce least-privilege access mapping (service identity → allowed secret scope).
    • Read secrets from the configured backend (OpenBao in the target state).
    • Optionally apply standard behaviors (timeouts, retries, audit enrichment).

  • Interactions:

    • Services (runtime secret reads)
    • OpenBao (backend)

Note

The concrete implementation may be a central gateway/service, an environment-scoped endpoint, or a thin compatibility layer. The key requirement is that services integrate only with this interface, not directly with the backend.

Component 2 – OpenBao (Secrets Backend)

  • Purpose:: Central KV v1 secrets store replacing HashiCorp Vault.

  • Responsibilities:

    • Store KV v1 secrets required by microservices and platform components.
    • Provide audit logs for secret access.
    • Support backup/restore and safe upgrades.

  • Interactions:

    • Runtime secret reads (Stage 1) via the vendor-agnostic interface
    • Bootstrap mechanism authenticates and reads secrets (Stage 2)
    • Admin tooling/jobs perform migration and provisioning.

  • Data Flow:

    • Read-heavy during runtime; write operations restricted to provisioning/migration.

Component 3 – Secrets Inventory & Migration Tooling

  • Purpose: Convert existing Vault KV v1 data and access structure to OpenBao.

  • Responsibilities:

    • Generate an inventory: which services consume which keys/paths.
    • Export from Vault KV v1 and import to OpenBao KV v1.
    • Verify correctness (presence, structure, access/policy).

  • Interactions:

    • Reads Vault; writes OpenBao.
    • Produces reports for rollout readiness.

Component 4 – Runtime Secrets Compatibility (Stage 1)

  • Purpose: Ensure services that currently call Vault at runtime can continue doing so against OpenBao (KV v1).

  • Responsibilities:

    • Discover and document runtime secret access patterns used today.
    • Ensure OpenBao auth and policies support these patterns.
    • Define the minimal migration steps per service (config-only when possible).

  • Interactions:

    • Services (existing runtime secret calls)
    • OpenBao (read operations)

Note

The concrete implementation (direct OpenBao endpoint vs a controlled compatibility endpoint/proxy) will be finalized during Phase 1 based on how services currently call Vault.

Component 5 – Env/Config Bootstrap Mechanism (Stage 2)

  • Purpose: Generate env/config artifacts for each service before it starts.

  • Responsibilities:

    • Authenticate to OpenBao.
    • Fetch required KV v1 values.
    • Render templates to .env and/or structured config files.
    • Write artifacts to a shared filesystem location for the service container.

  • Interactions:

    • Runs as K8s init container or Compose bootstrap step.
    • Writes files into a shared volume mounted by the app container.

  • Data Flow:

    • Bootstrap → render → write artifacts to shared filesystem volume → service starts.

Component 6 – Template & Schema Standard

  • Purpose: Ensure consistent key naming and artifact generation across services.

  • Responsibilities:

    • Define env variable naming rules.
    • Define required/optional variables per service.
    • Provide reusable templates.

Detailed Design

Database Design: OpenBao KV v1 is used; no relational schema changes are required for this feature. However, we define path conventions and a logical schema for key organization.

Secrets Path Conventions (Logical)

A recommended baseline (subject to finalization in Phase 1 inventory):

  • secret/appcircle/<environment>/<service>/...

Examples:

  • secret/appcircle/prod/api-gateway/DB_PASSWORD
  • secret/appcircle/prep/notification-service/SMTP_PASSWORD

Note

This is a logical structure for consistency. Exact prefixes may differ based on existing patterns discovered during inventory.

Algorithms and Logic:

Bootstrap Rendering Flow (Stage 2)

  1. Determine service identity and environment.
  2. Load template(s) for that service.
  3. Fetch required keys from env.
  4. Render template into .env and/or config outputs.
  5. Validate output (required keys present, format correct).

  6. Write artifacts to openbao atomically:

  7. Write to a temp file

  8. Move/rename into final location

Missing Secret Strategy

Two possible approaches:

  • Option A — Bootstrap-only provisioning (recommended for initial rollout):

  • Bootstrap reads and renders only.

  • Missing secrets are provisioned via controlled processes (bootstrap job/admin).

  • Option B — Auto-generate missing secrets at runtime:

  • Bootstrap detects missing keys and generates them.

Warning

This TDD assumes Option A for the initial rollout to reduce concurrency/idempotency risks.

Security Considerations:

Authentication & Authorization

  • Enforce least privilege: each service can access only its own secrets.
  • Avoid long-lived broad-scoped tokens.

Encryption

  • TLS for OpenBao endpoints.
  • Sensitive artifacts (.env) must be stored on volumes with restrictive permissions.

Auditability

  • Enable OpenBao audit logging.
  • Track and alert on denied access attempts.

Docker Compose Note

Docker Compose does not have Kubernetes workload identity. Compose deployments require an explicit bootstrap and secret distribution model.

Note

Compose credential distribution will be implemented in Phase 3 as part of the bootstrap reference design.

User Interface (if applicable)

Not applicable.

Integration and Interfaces

External Interfaces:

  • Vendor-agnostic secrets access interface (internal platform API/contract)
  • OpenBao HTTP API (used by the interface, bootstrap, and tooling—services must not call OpenBao directly)

Internal Interfaces:

  • Bootstrap writes env/config artifacts consumed by services.

Artifact formats:

  • Primary: .env
  • Optional: config.json / config.yaml for structured configurations

Deployment and Infrastructure

Cloud Deployment (Kubernetes / Helm)

  • Add OpenBao deployment (or update existing secrets component).

  • Add bootstrap init container pattern to service Helm charts:

  • Shared volume mount for generated artifacts

  • Init container responsible for fetching and rendering

Info

If Helm chart changes are required across many services, open a dedicated Helm issue/epic to track the shared template update.

Self-Hosted Deployment (Docker Compose)

  • Ensure OpenBao is available in Compose.
  • Introduce a bootstrap step and shared volumes for generated artifacts.

Info

If installation scripts or compose templates need changes, open a self-hosted script issue.

Testing and Quality Assurance

Test Strategy:

  • Regression tests:
  • Pilot wave validation
  • Rollback verification

Quality Assurance:

  • Code review requirements for shared bootstrap components.
  • CI checks for template validation.

Performance Considerations

Requirements:

  • Startup-time secret resolution is acceptable (non-performance-critical path).
  • The bootstrap process must not cause unacceptable startup delays.

Optimizations:

  • Batch reads where possible.
  • Avoid repeated calls; render once per startup.

Maintenance and Support

Monitoring and Logging:

  • OpenBao health checks, audit logs ingestion.

  • Alerts on:

    • OpenBao unavailability
    • Authentication failures
    • Excessive denied reads

Support and Maintenance:

  • Runbooks:

    • Migration
    • Provisioning missing secrets (Option A)
    • Backup/restore
    • Upgrades

Risks and Mitigation Strategies

  • Risk: Env variable inconsistencies across services

    • Mitigation: Standard schema, pilot-first rollout, deprecation list

  • Risk: Compose secret distribution security gaps

    • Mitigation: Reference design with strict volume permissions, documented bootstrap workflow

  • Risk: Startup dependency issues (DB/Keycloak/other services)

    • Mitigation: Readiness/startup probe patterns, retries, wave rollout

  • Risk: Migration errors (missing keys/paths)

    • Mitigation: Inventory + automated verification + pilot validation

Cost & Timeline

Assumptions

  • Secrets usage is Vault KV v1 only.
  • Minimal application code changes (deployment/bootstrap pattern changes).
  • Rollout is performed in waves with pilot validation.

Phase Plan and Estimates

Phase Scope Estimate (work days)
Phase 1 OpenBao foundation + runtime secrets migration (deploy, auth/audit, PoC migration, runtime validation) 8–12
Phase 2 Discovery & design (inventory, schema, rollout plan; finalize runtime patterns; identify pilot services) 4–9
Phase 3 Startup bootstrap framework (K8s + Compose reference patterns, pilot) 8–12
Phase 4 Service rollout (~32 services) 15–25
Total 35–58

Ownership and parallelization:

  • Phase 4 is owned primarily by the development teams (service owners).

  • Phase 4 can start after Phase 2 for the subset of services that only require runtime-secret migration and do not depend on the Stage 2 bootstrap framework.

  • Architecture 20-33 work days

  • Development Teams 15-25 work days

Calendar estimate (with capacity constraints)

Capacity assumptions
  • Architecture team: ~3 days/week
  • Development teams (service owners): full-time (can run Phase 4 in parallel once prerequisites are done)
Sequential phases (Architecture-led)
  • Phase 1: 8–12 work days → ~2.7–4.0 weeks (3 days/week)
  • Phase 2: 4–9 work days → ~1.3–3.0 weeks (3 days/week)
  • Phase 3: 8–12 work days → ~2.7–4.0 weeks (3 days/week)

Architecture-led calendar subtotal: ~6.7–11.0 weeks

Parallel rollout (Development-led)
  • Phase 4 (service rollout): starts after Phase 2
  • Can run in parallel with Phase 3 for services that are ready.
  • Remaining services complete after Phase 3 delivers the bootstrap reference pattern.

Note

Because Phase 4 is owned by development teams and can be parallelized, the overall calendar time is primarily bounded by Phases 1–3 (architecture capacity).

Appendix

Phase Deliverables Checklist

  • Phase 1:

    • OpenBao deployed
    • Auth/audit enabled
    • PoC migration executed
    • Runtime secret reads validated for pilot services

  • Phase 2:

    • Vault KV inventory
    • Env schema proposal
    • Runtime secret access pattern discovery
    • Pilot + wave rollout plan

  • Phase 3:

    • Bootstrap reference implementation
    • Helm/Compose examples
    • Pilot services live

  • Phase 4:

    • All services migrated
    • OpenSecrets no longer required
    • Rollback validated