OpenBao¶

Overview¶

OpenBao is the secrets management backend for all Appcircle infrastructure. It stores VM passwords, service credentials, and database passwords. Secrets are delivered to applications at runtime through OpenBao Agent — no secret ever lives in a config file or environment file committed to source control.

Domain: secrets.appcircle.io

Infrastructure¶

Component	Details
Runtime	LXC container on Proxmox (cloned from `prod-ready` template)
Network	`mgmt` VNet (`172.16.100.0/24`)
Address	`http://172.16.100.107:8080`
Storage backend	PostgreSQL (Google Cloud SQL)
Database	`openbao`
HA	Enabled (PostgreSQL-based)
Service user	`openbao` (system account, no login shell)

TLS not yet configured

OpenBao currently runs over plain HTTP on the management network. TLS with secrets.appcircle.io is a pending task. Until TLS is in place, the agent address uses http://.

Installation¶

Version:  2.5.5
Binary:   /usr/local/bin/bao
Config:   /etc/openbao/openbao.hcl
Data:     /opt/openbao/data
Service:  systemd — openbao.service

PostgreSQL Storage¶

Two tables are required in the openbao database.

CREATE TABLE vault_kv_store (
  parent_path TEXT COLLATE "C" NOT NULL,
  path        TEXT COLLATE "C",
  key         TEXT COLLATE "C",
  value       BYTEA,
  CONSTRAINT pkey PRIMARY KEY (path, key)
);
CREATE INDEX parent_path_idx ON vault_kv_store (parent_path);

CREATE TABLE vault_ha_locks (
  ha_key      TEXT COLLATE "C" NOT NULL,
  ha_identity TEXT COLLATE "C" NOT NULL,
  ha_value    TEXT COLLATE "C",
  valid_until TIMESTAMP WITH TIME ZONE NOT NULL,
  CONSTRAINT ha_key PRIMARY KEY (ha_key)
);

The openbao database user must have only the necessary permissions — do not grant cloudsqlsuperuser.

Network access¶

The OpenBao LXC connects to Cloud SQL on port 5432. This requires a Proxmox cluster firewall FORWARD rule:

FORWARD ACCEPT — protocol: tcp — source: +sdn/isolated-all — dport: 5432

Initialization and Unseal¶

OpenBao uses Shamir's secret sharing with 3 key shares and a threshold of 2.

bao operator init -key-shares=3 -key-threshold=2 | tee /root/openbao-init.txt

Danger

The output contains 3 Unseal Keys and 1 Root Token. These are shown only once. Store them immediately in a secure location outside of OpenBao (e.g., a password manager). Remove /root/openbao-init.txt from the server once saved.

Unseal after every restart:

bao operator unseal <key-1>
bao operator unseal <key-2>

bao status
# Sealed: false

Secret Engine¶

KV v2 is enabled at the secret/ path.

bao secrets enable -path=secret kv-v2

Secret Path Conventions¶

Path	Content
`secret/vm/<vm-name>/sysadmin`	`sysadmin` password for the named VM
`secret/vm/<vm-name>/appcircle`	`appcircle` password for the named VM
`secret/services/guacamole/db`	Guacamole PostgreSQL password

Example:

bao kv put secret/vm/obs-signoz/sysadmin  password="<value>"
bao kv put secret/vm/obs-signoz/appcircle password="<value>"
bao kv put secret/services/guacamole/db   password="<value>"

Tip

Generate passwords without characters that break shell quoting:

openssl rand -base64 32 | tr -d "'/+"

AppRole Authentication¶

AppRole is the authentication method used by OpenBao Agents running on application hosts.

Policy¶

Each agent gets a dedicated policy scoped to the paths it needs.

Example — Guacamole agent policy (guac-agent-policy.hcl):

path "secret/data/vm/obs-signoz/sysadmin"  { capabilities = ["read"] }
path "secret/data/vm/obs-signoz/appcircle" { capabilities = ["read"] }
path "secret/data/services/guacamole/db"   { capabilities = ["read"] }

bao policy write guac-agent guac-agent-policy.hcl

Role¶

bao write auth/approle/role/guac-agent \
  token_policies="guac-agent" \
  token_ttl=1h \
  token_max_ttl=4h \
  secret_id_num_uses=0

Current role configuration for guac-agent:

Parameter	Value
`token_ttl`	1h
`token_max_ttl`	4h
`secret_id_num_uses`	0 (unlimited)
`token_policies`	`guac-agent`

Credentials¶

# Get role-id (stable, write once to the host)
bao read auth/approle/role/guac-agent/role-id

# Generate secret-id (write to host before starting agent)
bao write -f auth/approle/role/guac-agent/secret-id

Place on the application host:

echo "<role-id>"   > /opt/appcircle/config/role-id
echo "<secret-id>" > /opt/appcircle/config/secret-id
chmod 0600 /opt/appcircle/config/role-id /opt/appcircle/config/secret-id

OpenBao Agent¶

The agent runs as a systemd service on each application host. It authenticates with AppRole, fetches secrets, and renders them to files via templates. When a template is rendered, an optional command can be triggered (e.g., to reload the application).

Example agent config (/opt/appcircle/config/openbao-agent.hcl):

vault {
  address = "http://172.16.100.107:8080"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path                   = "/opt/appcircle/config/role-id"
      secret_id_file_path                 = "/opt/appcircle/config/secret-id"
      remove_secret_id_file_after_reading = false
    }
  }
}

template {
  source      = "/opt/appcircle/config/app.env.tpl"
  destination = "/opt/appcircle/config/app.env"
  perms       = "0600"
  command     = "/opt/appcircle/bin/on-secret-change.sh"
}

remove_secret_id_file_after_reading = false

Set to false because secret_id_num_uses = 0 (unlimited reuse) and the file is protected by root:root 0600. This allows the agent to re-authenticate after a restart without requiring a new secret-id to be provisioned.

Systemd service¶

# /etc/systemd/system/openbao-agent.service
[Unit]
Description=OpenBao Agent
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/bao agent -config=/opt/appcircle/config/openbao-agent.hcl
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Operational Commands¶

# Check status
bao status

# Unseal (run twice with different keys)
bao operator unseal <key>

# List secrets
bao kv list secret/vm/

# Read a secret (returns metadata + data)
bao kv get secret/vm/obs-signoz/sysadmin

# Rotate a password
bao kv put secret/vm/obs-signoz/sysadmin password="<new-value>"

# Check AppRole role
bao read auth/approle/role/guac-agent

Pending Tasks¶

Configure TLS for secrets.appcircle.io
Switch agent address from http:// to https:// after TLS is in place
Define policies and AppRole roles for additional application hosts