Guacamole¶

Overview¶

Apache Guacamole provides browser-based SSH access to all Appcircle VMs and LXC containers. Operators authenticate through SSO (OIDC) and connect to target machines without needing a local SSH client or VPN.

Accessible at: remote.appcircle.io

Guacamole uses two connection groups per VM — one for infrastructure operators (sysadmin) and one for developers (appcircle). Passwords for both accounts are stored in OpenBao and injected automatically. Operators never see or handle VM passwords directly.

Infrastructure¶

Component	Details
Runtime	LXC container on Proxmox
`eth0` (`isolated`)	`172.16.10.115/24`, gateway `172.16.10.1` — internet access via SNAT
`eth1` (`mgmt`)	`172.16.100.2/24`, gateway `172.16.100.1` — OpenBao connectivity
Stack	Docker Compose (guacamole + guacd + postgresql)

The Guacamole host sits on two networks: isolated for internet egress (reaching external services, updates), and mgmt for internal service communication (OpenBao at 172.16.100.107).

Directory Structure¶

/opt/appcircle/
  bin/
    Makefile                      operational commands for appcircle user
    update-guac-creds.sh          updates VM passwords in Guacamole database
  config/
    openbao-agent.hcl             OpenBao Agent configuration
    guac-creds.env.tpl            template — VM passwords
    guac-creds.env                rendered VM passwords (root:root 0600)
    guac-compose-secrets.env.tpl  template — Guacamole DB password
    role-id                       AppRole role-id (root:root 0600)
    secret-id                     AppRole secret-id (root:root 0600)
    guacamole.properties          Guacamole application config
    extensions/
      guacamole-auth-sso-1.6.0/  SSO/OIDC extension
  compose/
    compose.yaml                  Docker Compose stack definition
    .env                          DB connection parameters (non-secret)
    .env.secrets                  DB password rendered by OpenBao Agent (root:root 0600)

Services¶

Two systemd services manage the stack.

Service	Purpose	Managed by
`openbao-agent.service`	Authenticates to OpenBao, renders secret files, triggers credential update	root only
`guacamole-stack.service`	Runs `docker compose up/down` for the full Guacamole stack	`appcircle` via scoped sudo

Startup order¶

guacamole-stack.service waits for .env.secrets to be populated before starting containers:

ExecStartPre=/bin/bash -c 'for i in $(seq 1 30); do [ -s /opt/appcircle/compose/.env.secrets ] && exit 0; sleep 2; done; exit 1'

This prevents the Guacamole container from starting with an empty or stale database password.

`appcircle` user permissions¶

The appcircle user can manage only guacamole-stack.service via a scoped sudoers file (/etc/sudoers.d/appcircle-guacamole):

appcircle ALL=(root) NOPASSWD: /usr/bin/systemctl start guacamole-stack.service
appcircle ALL=(root) NOPASSWD: /usr/bin/systemctl stop guacamole-stack.service
appcircle ALL=(root) NOPASSWD: /usr/bin/systemctl restart guacamole-stack.service
appcircle ALL=(root) NOPASSWD: /usr/bin/systemctl status guacamole-stack.service

openbao-agent.service is intentionally excluded — appcircle must not be able to stop secret rendering.

OpenBao Integration¶

OpenBao Agent runs continuously on the Guacamole host. It renders two files from templates and triggers a credential update script when VM passwords change.

Flow¶

OpenBao KV v2
  secret/vm/<name>/sysadmin    → SYSADMIN_PASSWORD
  secret/vm/<name>/appcircle   → APPCIRCLE_PASSWORD
  secret/services/guacamole/db → POSTGRESQL_PASSWORD
         ↓
OpenBao Agent renders guac-creds.env.tpl
         ↓
/opt/appcircle/config/guac-creds.env  (root:root 0600)
         ↓
update-guac-creds.sh triggered automatically
         ↓
psql UPDATE guacamole_connection_parameter
  connection_id = 3  → sysadmin password
  connection_id = 4  → appcircle password

OpenBao Agent also renders guac-compose-secrets.env.tpl → compose/.env.secrets. When the Guacamole DB password rotates, the stack is restarted automatically via a command hook on that template.

Agent configuration (`openbao-agent.hcl`)¶

vault {
  address = "http://172.16.100.107:8080"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path                   = "/opt/appcircle/config/role-id"
      secret_id_file_path                 = "/opt/appcircle/config/secret-id"
      remove_secret_id_file_after_reading = false
    }
  }
}

template {
  source      = "/opt/appcircle/config/guac-creds.env.tpl"
  destination = "/opt/appcircle/config/guac-creds.env"
  perms       = "0600"
  command     = "/opt/appcircle/bin/update-guac-creds.sh"
}

template {
  source      = "/opt/appcircle/config/guac-compose-secrets.env.tpl"
  destination = "/opt/appcircle/compose/.env.secrets"
  perms       = "0600"
  command     = "systemctl restart guacamole-stack.service"
}

SSO / OIDC¶

Authentication is handled by the guacamole-auth-sso-1.6.0 extension. Users log in with their organizational SSO account — no local Guacamole credentials are required.

Access to each connection group is controlled by SSO group membership:

SSO group	Connection group	Target
Infrastructure / ops team	`vm-<name>-admin`	`sysadmin` on the VM
Developer / deployment team	`vm-<name>-app`	`appcircle` on the VM

Connection Groups¶

Each managed VM has two Guacamole connections.

Connection	Account used	Who has access
`vm-<name>-admin`	`sysadmin`	Infrastructure and ops team
`vm-<name>-app`	`appcircle`	Developers and deployment operators

Passwords are never entered manually. They are injected from OpenBao via update-guac-creds.sh into the guacamole_connection_parameter table.

Session Recording¶

Every session is recorded. Recordings are stored under /recordings/ on the Guacamole host, organized by username.

Recording path pattern:

/recordings/${GUAC_USERNAME}/${GUAC_DATE}T${GUAC_TIME}_<vm-name>
/recordings/${GUAC_USERNAME}/${GUAC_DATE}T${GUAC_TIME}_<vm-name>.txt

Future: Cloud Storage

Recordings will be moved to a Cloud Storage bucket. Until that migration is complete, recordings remain on the Guacamole host's local filesystem.

Minimum retention: 90 days.

Adding a New VM¶

To make a new VM accessible through Guacamole:

1. Store passwords in OpenBao:

bao kv put secret/vm/<vm-name>/sysadmin  password="$(openssl rand -base64 32 | tr -d "'/+")"
bao kv put secret/vm/<vm-name>/appcircle password="$(openssl rand -base64 32 | tr -d "'/+")"

2. Set those passwords on the VM:

passwd sysadmin   # enter the value stored in OpenBao
passwd appcircle  # enter the value stored in OpenBao

3. Add the VM to the agent template (guac-creds.env.tpl):

{{ with secret "secret/data/vm/<vm-name>/sysadmin" }}
<VM_NAME>_SYSADMIN_PASSWORD={{ .Data.data.password }}
{{ end }}
{{ with secret "secret/data/vm/<vm-name>/appcircle" }}
<VM_NAME>_APPCIRCLE_PASSWORD={{ .Data.data.password }}
{{ end }}

4. Update the AppRole policy to include the new secret paths.

5. Create two connections in Guacamole (vm-<name>-admin and vm-<name>-app) with SSH protocol, pointing to the VM IP and custom SSH port.

6. Update update-guac-creds.sh to include UPDATE statements for the new connection IDs.

7. Restart the agent to trigger a re-render and apply the new credentials.

Operational Commands¶

The appcircle user operates the stack via the Makefile:

cd /opt/appcircle/bin

make help        # show available commands
make start       # start the Guacamole stack
make stop        # stop the Guacamole stack
make restart     # restart the Guacamole stack
make status      # show stack service status
make logs        # show last 50 log lines
make logs-tail   # follow logs in real time
make agent-logs  # follow OpenBao Agent logs
make ps          # list running containers

Pending Tasks¶

Migrate session recordings to Cloud Storage bucket
Switch OpenBao Agent address to https:// after TLS is configured on OpenBao
Refine mgmt VNet firewall rules

Guacamole¶

Overview¶

Infrastructure¶

Directory Structure¶

Services¶

Startup order¶

appcircle user permissions¶

OpenBao Integration¶

Flow¶

Agent configuration (openbao-agent.hcl)¶

SSO / OIDC¶

Connection Groups¶

Session Recording¶

Adding a New VM¶

Operational Commands¶

Pending Tasks¶

`appcircle` user permissions¶

Agent configuration (`openbao-agent.hcl`)¶