Skip to content

ADR-0004: Virtualization Platform Standard for Appcircle (On-Prem + OVH)

  • Status: ✅ Accepted
  • Date: 2026-02-05
  • Owners: @osmank,@Berk,@Çagkan

Context

Appcircle runs and will continue to expand multiple environments across: - On-Prem / internal lab infrastructure (5–6 physical hosts) - OVH platform (Appcircle-operated infrastructure hosting VMs and supporting services)

Today, VM lifecycle management is not fully standardized across environments. We need a single, consistent virtualization approach for: - Kubernetes clusters (dev/prep/test) - CRC / OpenShift Local (nested virtualization required in some cases) - Git provider services (e.g., Bitbucket) and supporting tooling - Over time, we may also migrate additional services such as a blog website or other low/medium critical workloads.

Hosts may have Docker installed; however, running random workloads on hypervisor nodes increases attack surface and operational risk.

This ADR selects a virtualization platform standard that can be used both on-prem and in OVH with strong security controls, standardized backup/restore, and a path to commercial support when needed.

Requirements / Constraints

Operational - Central UI/API to manage VMs across multiple hosts (create/start/stop/console/templates) - Cluster support and lifecycle operations (maintenance, upgrades, migrations where feasible) - Practical support for nested virtualization (CRC/OpenShift Local) - Automation capability (API / scripting / IaC integration potential) - Must work both on-prem and OVH (separate clusters are acceptable; the standard and runbooks must be shared)

Security - RBAC and least-privilege administration - MFA/2FA support for human access - Management plane isolation (no direct internet exposure; access via VPN/Zero-Trust) - Network segmentation (MGMT / STORAGE / WORKLOAD) - Host and VM-level network controls (firewalling / policy) - Patch and upgrade process with change management discipline

Backup / Recovery - Standardized backup/restore procedures - Offsite copy strategy (at least one additional location, depending on data class) - Restore drills and documented RPO/RTO targets for critical workloads

Commercial posture - Free-to-use baseline acceptable - Optional paid subscription/support path preferred (to reduce operational risk in production-like usage)

Decision

We will adopt Proxmox VE as Appcircle’s standard virtualization platform for both on-prem and OVH VM hosting, operated as one Proxmox cluster per environment (e.g., one cluster on-prem, one cluster in OVH), with Proxmox Backup Server (PBS) used for standardized VM backups.

Key implementation principles: 1. Hypervisor nodes must remain minimal. - No application workloads on the host OS. - If Docker is required, it runs inside dedicated “Tools VM” (or Tools VM pool), not on hypervisor nodes. 2. Management plane must be isolated. - Proxmox UI/API reachable only through MGMT network via VPN/Zero-Trust access. - No public exposure of admin interfaces. 3. Network segmentation is mandatory. - Separate networks/VLANs for MGMT, STORAGE, and WORKLOAD. 4. Access control is mandatory. - RBAC roles defined (admin/operator/read-only). - MFA enforced for all human access. 5. Backup is mandatory. - PBS configured with encryption. - Backups include verification and periodic restore drills. 6. Phased adoption. - Phase 1: Build clusters (on-prem + OVH), validate nested virtualization, define templates and runbooks. - Phase 2: Migrate existing OVH VMs into the standard platform where feasible. - Phase 3: Consider migrating additional low/medium critical services (e.g., blog site) with explicit RPO/RTO and security classification.

Options considered

Option A — Proxmox VE + Proxmox Backup Server (Chosen)

Pros - Cohesive platform UX: web UI + cluster operations for small/medium environments - Supports both VMs (KVM) and containers (LXC) where useful - Strong backup story with PBS (standardized restore workflows) - Free baseline with an optional subscription/support path - Practical fit for nested virtualization workloads

Cons - Best HA/live-migration depends on storage/network architecture - Requires operational discipline to keep hypervisors minimal and consistent - Multi-site “single pane” is not automatic; separate clusters need standard runbooks and automation for consistency

Option B — KVM/libvirt + Cockpit + Ansible

Pros - Fully Linux-native and free - Highly flexible and minimal component footprint

Cons - Less cohesive platform experience across on-prem + OVH - More internal engineering effort for standardization (HA/backup/UI/ops)

Option C — XCP-ng + Xen Orchestra

Pros - Mature centralized management model - Commercial support options exist

Cons - Higher adoption/learning overhead vs Proxmox - Tooling/operational choices (XO deployment, backup) require additional standardization work

Option D — KubeVirt / OpenShift Virtualization / Harvester

Pros - Unified VM + container operations under Kubernetes-style management

Cons - Higher operational complexity than needed for this scope - Adds platform overhead and additional failure modes

Consequences

Positive

  • Standard platform across on-prem + OVH improves consistency, reduces drift
  • Security posture improves via mgmt isolation, RBAC, and MFA baseline
  • Standardized backups/restores and measurable recovery drills via PBS
  • Enables gradual migration of additional workloads (e.g., blog) under controlled policy

Negative / Trade-offs

  • Storage/HA design must be intentional for production-like expectations
  • PBS adds another component to operate (monitoring/patching needed)
  • Requires initial investment in runbooks/templates and access model definition

Follow-ups / migrations

  1. Platform bootstrap
  2. Create on-prem Proxmox cluster (5–6 nodes)
  3. Create OVH Proxmox cluster (node count based on OVH capacity plan)
  4. Network baseline
  5. Define MGMT/STORAGE/WORKLOAD networks and naming conventions
  6. Enforce VPN/Zero-Trust access path to MGMT
  7. Storage baseline
  8. Phase 1: Local ZFS + PBS backups
  9. Phase 2 (optional): Shared storage / Ceph only if HA/migration requirements justify it and infra supports it
  10. Security baseline
  11. RBAC matrix, MFA enforcement, admin access restrictions
  12. Host + template hardening checklist
  13. Backup policy
  14. Define RPO/RTO per workload class (test vs production-like)
  15. Daily incremental backups, retention rules, and monthly restore drills
  16. Offsite copy approach (e.g., replicate backups cross-site where needed)
  17. Migration plan (OVH)
  18. Inventory existing VMs
  19. Prioritize “low risk” VMs for first migrations
  20. Validate performance and restore before broader rollout
  21. Workload onboarding
  22. Standard VM templates (cloud-init, baseline monitoring agents, logging)
  23. “Blog site” migration requires explicit classification, RPO/RTO, and rollback plan
  • Related issues/PRs: (to be added)
  • Supersedes / Superseded by: N/A