Proxmox Node Operations Runbook¶

Overview¶

This document covers node addition, node removal, and workload migration operations on the OVHCluster (our OVH-based Proxmox VE cluster). For the cluster's network/firewall design, see Network & Firewall Design.

Cluster: OVHCluster Current nodes: pve1 (192.168.10.1), pve3 (192.168.10.3), pve4 (192.168.10.4), pve5 (192.168.10.5) NFS Backup Server: 192.168.10.3 (pve3) → /srv/nfs-backup Private Network: 192.168.10.0/24 (OVH vRack, over vmbr1)

1. Adding a New Node¶

1.1 Prerequisites¶

Proxmox VE must already be installed on the server via the OVH panel.
The new node's private IP (192.168.10.x) must be decided and must not conflict with existing nodes.
The server must be attached to the OVH vRack.

1.2 Clearing SSH Host Keys¶

Whenever the OS has been reinstalled, or you connect to a new server for the first time, clear the known_hosts entry:

ssh-keygen -R <new_node_public_ip>
ssh root@<new_node_public_ip>

1.3 Private Network (vRack) Configuration¶

Identify the private NIC on the new node:

ip link show

The second port in DOWN state is the one used for vRack (e.g. eno2, enp66s0f1). Add a bridge to /etc/network/interfaces:

auto vmbr1
iface vmbr1 inet static
    address 192.168.10.X/24
    bridge-ports <private_nic_name>
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

Apply and verify:

ifreload -a
ping -c3 192.168.10.1   # ping an existing node

1.4 Updating /etc/hosts¶

Add the hostnames of the existing nodes for SSL hostname validation during the cluster join:

echo "192.168.10.1 pve1" >> /etc/hosts
echo "192.168.10.3 pve3" >> /etc/hosts
# ... other nodes

1.5 Cluster Join¶

pvecm add 192.168.10.1 --link0 <new_node_private_ip>

You will be prompted for pve1's root password. Confirm the SSH fingerprint prompt with yes.

Verify (on any node):

pvecm status
pvecm nodes

Check that Expected votes increased by 1 and that the new node shows as online.

Warning

Do not add two nodes at the same time. Wait for the first one to finish joining and appear in the cluster before adding the second (corosync config sync conflicts otherwise).

1.6 Installing dnsmasq (Simple Zone DHCP)¶

apt update
apt install -y dnsmasq

# Disable the default instance (conflicts with Simple Zone)
systemctl stop dnsmasq
systemctl disable dnsmasq

1.7 SDN Apply¶

Apply the cluster's SDN config on the new node (Web UI: Datacenter → SDN → Apply, or via CLI):

pvesh set /cluster/sdn

Verify:

systemctl status dnsmasq                 # should be inactive
systemctl status dnsmasq@simple.service   # should be active

ip link show ingress
ip link show isolated
ip link show k8group

All three VNet bridges should be UP.

1.8 NAT Service Setup¶

Create /etc/network/nat.nft (fill in public DNAT rules if traffic will be routed to this node, otherwise it can stay empty):

table inet nat {
    chain prerouting {
        type nat hook prerouting priority -100;
        # DNAT rules go here
    }
}

Create /etc/systemd/system/nat.service:

[Unit]
Description=Load NAT nftables rules after Proxmox firewall
After=proxmox-firewall.service
Wants=proxmox-firewall.service
PartOf=proxmox-firewall.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/nft -f /etc/network/nat.nft
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable nat.service
systemctl start nat.service
systemctl status nat.service   # should be active (exited)

1.9 NFS Backup Access¶

On pve3, add the new node to /etc/exports:

# on pve3
nano /etc/exports
# add a new line: /srv/nfs-backup 192.168.10.X(rw,sync,no_subtree_check,no_root_squash)

exportfs -ra
exportfs -v    # confirm the new IP appears in the list

Test connectivity from the new node:

nc -zv 192.168.10.3 2049
mount -t nfs 192.168.10.3:/srv/nfs-backup /mnt/test

1.10 Firewall Check¶

systemctl is-active proxmox-firewall.service   # should be active
systemctl is-active pve-firewall.service       # should be inactive

Danger

If pve-firewall.service shows as active, stop it immediately:

systemctl stop pve-firewall
systemctl disable pve-firewall

1.11 Final Operational Checks¶

systemctl is-active proxmox-firewall.service   # active
systemctl is-active pve-firewall.service       # inactive
sysctl net.ipv4.ip_forward                     # 1
systemctl status nftables                      # active
systemctl status dnsmasq                       # inactive
systemctl status dnsmasq@simple.service        # active
systemctl status nat.service                   # active (exited)

2. Removing a Node¶

2.1 Prerequisites — Guest Check¶

No guests should be running on the node to be removed:

# on the target node
qm list
pct list
pvesm list local-lvm

If any running or stopped guests exist, migrate them first (see Section 3).

2.2 Quorum Impact Check¶

pvecm status

Check whether quorum can still be maintained after the node is removed:

Current Node Count	Quorum Threshold	After Removal	Safe?
5	3	4 nodes remain	✅
4	3	3 nodes remain	✅
3	2	2 nodes remain	⚠️ Proceed with caution

2.3 Remove from NFS Exports¶

On pve3, remove the node's IP from /etc/exports:

# on pve3
nano /etc/exports
exportfs -ra
exportfs -v

2.4 Remove from Cluster¶

After stopping the node, from any active node:

pvecm delnode <node_name>

Verify:

pvecm nodes
pvecm status   # confirm Expected votes decreased by 1

2.5 DNS / Public IP Rerouting¶

If DNAT rules were active on the removed node, traffic to that node's public IP will no longer be answered. Update DNS records and floating IP mappings accordingly.

3. Workload Migration (Node-to-Node)¶

3.1 Choosing a Migration Method¶

Method	When	Pros	Cons
`qm migrate` / `pct migrate`	Shared storage or online migration	Fast, IP is preserved	Since Simple Zone is node-local, additional network configuration may be required
Backup → Restore	Any case	Safe, controlled	Requires downtime

Note

Simple Zone operates node-local (subnets such as 172.16.5.0/24 run as an isolated instance on each node). The guest's private IP can stay the same after migration, but DNAT/public traffic will need to be rerouted to the target node's public IP.

3.2 Migration via Backup (Recommended)¶

Step 1 — Take a backup:

Via Web UI or CLI:

vzdump <CT_ID> --compress zstd --mode snapshot --storage nfs-backup --node <source_node>

Step 2 — Confirm the NFS storage is visible on the target node:

pvesm status
ls /mnt/pve/nfs-backup/dump/

Step 3 — Restore:

# for a CT
pct restore <new_CT_ID> /mnt/pve/nfs-backup/dump/vzdump-lxc-<ID>-<date>.tar.zst \
  --storage local-lvm

# for a VM
qmrestore /mnt/pve/nfs-backup/dump/vzdump-qemu-<ID>-<date>.vma.zst <new_VM_ID> \
  --storage local-lvm

Step 4 — Verify the network config:

pct config <CT_ID>
# check the net0 line: bridge, IP, gateway, hwaddr

If Simple Zone is in use, the gateway (172.16.x.1) can stay the same — each node provides its own local gateway. The MAC address remaining unchanged is not an issue (different bridges, L2 isolation is in place).

Step 5 — Confirm the SDN bridges exist on the target node:

ip link show ingress
ip link show isolated
ip link show k8group

If they don't exist, apply SDN:

pvesh set /cluster/sdn

Step 6 — Start and test the CT/VM:

pct start <CT_ID>
pct exec <CT_ID> -- ping -c3 <gateway_ip>
pct exec <CT_ID> -- ping -c3 8.8.8.8

Step 7 — Stop the original on the source node:

Make sure both copies are never running at the same time:

# on the source node
pct stop <original_CT_ID>

3.3 Updating Public Routing¶

If the guest serves a domain/IP:

Add the DNAT rules to the target node's nat.nft file.
Run systemctl restart nat.service.
Point the DNS record / floating IP to the target node's public IP.
Wait out the TTL and test access from outside.

3.4 External Service Access (Cloud SQL, GCP, etc.)¶

If the guest connects to any GCP or external service (e.g. Cloud SQL, the OpenBao PostgreSQL backend), the target node's public IP must be added to that service's allowlist/Authorized Networks:

# connectivity test from inside the CT
nc -zv <external_ip> <port>

If you get a timeout:

GCP Console → Cloud SQL → Connections → Authorized Networks → add the target node's public IP.
Check firewall/security group rules.

4. Quick Checklist¶

Adding a New Node¶

Removing a Node¶

Workload Migration¶

Backup taken, visible on NFS
Restored on target node
SDN bridges present
Network config (IP/GW/MAC) verified
CT/VM started, gateway and internet ping OK
External service allowlists updated (if needed)
Original copy on source node stopped
Public DNAT rules moved to target node
DNS/floating IP mapping updated