Crashes¶

Summary¶

On Proxmox nodes using a ZFS pool as local storage for VM disks (qcow2, virtio), VMs may unexpectedly crash or stop with guest-level I/O errors. Root cause is ZFS's Direct I/O write verification feature (direct=standard), which is incompatible with how QEMU issues O_DIRECT writes to qcow2-backed virtio disks.

This is not a hardware or data corruption issue — it is a known ZFS/QEMU compatibility gap.

Symptoms¶

A VM stops or crashes with no corresponding manual qmstop/qmshutdown task in Proxmox task history (i.e., not operator-initiated)
Guest OS or Proxmox reports an I/O error on the virtio disk
Sudden drop in VM CPU/memory graphs coinciding with the crash
No disk hardware indicators: SMART passes, zpool status shows 0 READ/WRITE/CKSUM errors, pool state ONLINE

Diagnosis¶

Check for I/O errors reported by the guest/QEMU around the crash window (journalctl on the host, or guest console/logs if accessible).
Rule out physical disk failure:
```
smartctl -a /dev/<nvme_or_disk>
zpool status -v
```
Healthy output (0 errors, PASSED) points away from hardware.
Check ZFS event log for Direct I/O verification failures:
```
zpool events -v | grep dio_verify_wr
```
Presence of ereport.fs.zfs.dio_verify_wr events (with zio_err = 0x5 / EIO) confirms this failure mode. Each event includes a dio_verify_errors counter.
Check the pool's Direct I/O setting:
```
zfs get direct <pool_name>
```
standard (the default) means ZFS performs verification reads whenever an application (e.g., QEMU) requests O_DIRECT — which qcow2 + virtio does.

Fix¶

Disable ZFS Direct I/O verification at the pool level:

zfs set direct=disabled <pool_name>

This applies to all datasets/VMs under that pool. I/O falls back to ZFS's standard buffered path, still protected by ZFS's own checksumming — no meaningful data-integrity or performance tradeoff for typical VM workloads on NVMe-backed pools.

Verify:

zfs get direct <pool_name>          # should show: disabled   local
zpool events -v | grep -c dio_verify_wr   # count should stop increasing

Restart the affected VM(s) after applying.

Notes / Prevention¶

Apply this setting on any Proxmox node using ZFS local storage for qcow2/virtio VM disks — the issue is not VM-specific, it's pool-wide.
Any prior "unexplained" VM crash on ZFS-backed local storage should be re-evaluated against zpool events -v for dio_verify_wr entries before assuming hardware failure.
Consider setting direct=disabled proactively as a standard config item when provisioning new ZFS pools intended for VM storage.
Re-check after major ZFS/kernel version upgrades — this is a known interaction that may change in future OpenZFS releases.

References¶

ZFS event class: ereport.fs.zfs.dio_verify_wr
Relevant ZFS property: direct (disabled / standard / always)