OFFSET偏移
← All articlesOFFSET / FIELD NOTESNO. 139

KVM firmware, VFIO, virtiofs, and image chains

Explain advanced parameters and document compatibility and recovery requirements for passthrough, image changes, and shared directories.

English translation · Code and commands follow the Chinese original.

IN THESE READING PATHSAdvanced KVM capabilities and platform choices ↗
About 20 min readStart reading

The commands and XML are generic examples and must be adjusted for the host version and actual resources. Network addresses use documentation ranges; choose a nonconflicting private network for real deployments. See the relevant sections for the conditions that apply to image modification and device passthrough.

From basic virtualization to production-grade virtual machines, this chapter covers modern UEFI/TPM firmware support, VFIO hardware passthrough, virtio paravirtualized driver optimization, virtiofs file sharing, and advanced disk-image operations.


1. UEFI firmware and TPM: modern OS support#

1.1 Why use UEFI?

Choose firmware based on guest and boot requirements. SeaBIOS remains usable for supported legacy guests, while UEFI provides a modern firmware interface:

Scenario BIOS limitation UEFI solution
Windows 11 installation Not supported OVMF + TPM 2.0 + Secure Boot
Disks > 2TB Do not confuse MBR addressing limits with BIOS; BIOS can boot GPT disks with a suitable bootloader OVMF supports GPT natively
Secure Boot Not supported OVMF supports signature verification
Boot speed Initialization time varies by implementation Shorter total boot time is not guaranteed
Multi-OS boot Traditional bootloaders can manage multiple operating systems, but lack the UEFI boot-entry interface UEFI boot manager with multiple entries

1.2 OVMF (Open Virtual Machine Firmware)

The filenames and directories below are distribution-specific examples. CODE/VARS sizes, architecture, and Secure Boot keys must match; check domcapabilities and firmware metadata first. New domains can use automatic firmware selection. An existing domain's NVRAM contains boot entries and keys, so do not casually delete or recreate it just to move paths.

OVMF is a UEFI firmware implementation built from Tianocore EDK2 for QEMU/KVM.

Installation:

# Arch Linux
sudo pacman -S edk2-ovmf

# Ubuntu / Debian
sudo apt install ovmf

# Fedora
sudo dnf install edk2-ovmf

Firmware file locations:

# Arch / Fedora
ls /usr/share/edk2-ovmf/
# OVMF_CODE.fd        -- 只读固件代码 (UEFI 运行时)
# OVMF_VARS.fd        -- 可变存储模板 (NVRAM, 启动项, 证书)
# OVMF_CODE.secboot.fd -- 带 Secure Boot 的固件代码

# Ubuntu / Debian
ls /usr/share/OVMF/
# OVMF_CODE.fd
# OVMF_CODE_4M.fd       -- 4MB 版本 (推荐用于现代 OS)
# OVMF_VARS.fd
flowchart LR
    A[Host: OVMF_CODE.fd] -->|readonly, shared| B[Guest NVRAM]
    C[Host: OVMF_VARS.fd] -->|copy-on-write| D[Per-VM VARS file]
    D --> E[Boot entries / Keys / Certificates]
    B --> F[UEFI Runtime in Guest]

libvirt XML configuration:

<os firmware='efi'>
  <type arch='x86_64' machine='q35'>hvm</type>
  <firmware>
    <feature enabled='yes' name='enrolled-keys'/>
    <feature enabled='yes' name='secure-boot'/>
  </firmware>
  <!-- 新建域让 libvirt 根据固件元数据选择兼容的 Secure Boot CODE/VARS。 -->
</os>

Each virtual machine needs its own NVRAM file OVMF_VARS.fd is a template; libvirt copies it to /var/lib/libvirt/qemu/nvram/ when creating a virtual machine. VMs must not share a VARS file, or they will overwrite one another's boot entries and certificates.

1.3 Secure Boot

<!-- 启用 Secure Boot -->
<os firmware='efi'>
  <loader readonly='yes' secure='yes' type='pflash'>
    /usr/share/edk2-ovmf/OVMF_CODE.secboot.fd
  </loader>
</os>

In Secure Boot mode, OVMF verifies bootloader signatures. Linux distributions usually include signed bootloaders (shim + GRUB), and Windows also supports this natively.

Secure Boot conflicts with some GPU ROMs With both Secure Boot and GPU passthrough enabled, a GPU's Video ROM may fail to load because signature verification fails. First check the firmware, ROM type, signatures, and guest logs, then decide whether to use a verified ROM or adjust policy. Disabling Secure Boot is not the default fix and may also trigger disk-encryption recovery.

1.4 TPM 2.0

Windows 11 officially requires TPM 2.0. SWTPM is a common emulated implementation, but a compliant hardware TPM backend can also be used. Check the full requirements as well, including CPU, memory, storage, UEFI, and Secure Boot capability.

Installation:

# Arch Linux
sudo pacman -S swtpm

# Ubuntu / Debian
sudo apt install swtpm swtpm-tools

# Fedora
sudo dnf install swtpm swtpm-tools

libvirt XML configuration:

<tpm model='tpm-crb'>
  <backend type='emulator' version='2.0'>
    <encryption secret='...'/>
  </backend>
</tpm>

In virt-manager:

  1. Add Hardware -> TPM
  2. Model: CRB (recommended; better performance than TIS)
  3. Backend: Emulated
  4. Version: 2.0

Persisting TPM state The TPM emulator stores its state under /var/lib/libvirt/swtpm/<vm-uuid>/. If TPM-dependent full-disk encryption such as BitLocker is enabled, backing up this directory is critical: losing the TPM state means the disk cannot be decrypted.

1.5 Complete Windows 11-compatible configuration

<!-- 演示 UEFI/TPM 相关片段;并非全部 Windows 11 硬件要求 -->
<os firmware='efi'>
  <type arch='x86_64' machine='q35'>hvm</type>
  <firmware>
    <feature enabled='yes' name='secure-boot'/>
  </firmware>
</os>
<tpm model='tpm-crb'>
  <backend type='emulator' version='2.0'/>
</tpm>

2. VFIO / PCI Passthrough: hardware passthrough#

The VFIO/IOMMU group is the unit of isolation. Before passthrough, check every device in the group, host dependencies, and a recovery control path. ID-based binding may match multiple devices of the same model, and blacklisting may disable all of the host's GPUs. ACS override is a dangerous software method of splitting groups available in some patched kernels; it does not create hardware DMA isolation and is not a default step.

VFIO (Virtual Function I/O) directly assigns physical PCIe devices to virtual machines for near-bare-metal I/O performance. Typical uses include GPU passthrough for gaming/rendering/AI training, NVMe passthrough for databases, and USB controller passthrough for latency-sensitive peripherals.

2.1 IOMMU basics

The IOMMU (Input-Output Memory Management Unit) is the hardware foundation of VFIO, responsible for mapping DMA addresses into the virtual machine's address space.

Enable IOMMU:

# /etc/default/grub (GRUB 用户)
GRUB_CMDLINE_LINUX_DEFAULT="... intel_iommu=on iommu=pt"
# Intel: 按内核/固件需要检查 intel_iommu=on;AMD 通常自动初始化,iommu.strict=1 不是启用开关
# iommu=pt: 直通模式下不对 host 设备做 DMA 重映射,减少性能开销

# systemd-boot 用户
# /boot/loader/entries/xxx.conf
options ... intel_iommu=on iommu=pt

# 更新引导配置
sudo grub-mkconfig -o /boot/grub/grub.cfg   # GRUB

Verify that IOMMU is enabled:

# 检查内核日志
dmesg | grep -i iommu
# Intel 输出示例: DMAR-IR: Enabled IRQ remapping in x2apic mode
# AMD 输出示例: AMD-Vi: Enabling IOMMU

# 检查 DMAR / AMD-Vi 表
dmesg | grep -i -e DMAR -e IOMMU

# 验证 IOMMU 分组
find /sys/kernel/iommu_groups/ -type l | sort -V

2.2 IOMMU groups and isolation

IOMMU groups determine which devices can be passed through independently. VFIO requires all relevant devices in a group to meet safe binding conditions, such as being bound to VFIO or unbound from their host driver. This does not mean every function must be exposed to the same guest. Devices in the same group that are still used by the host must not compromise isolation.

# 查看每个分组的设备
#!/bin/bash
for g in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $(find $g/devices/ -type l); do
        echo "  $(basename $d) ($(lspci -nns $(basename $d) | cut -d' ' -f2-))"
    done
done

Security risks of the ACS patch If a GPU shares an IOMMU group with other devices, such as a USB controller, NVMe drive, or NIC, the pcie_acs_override=downstream,multifunction kernel parameter can forcibly split the group. However, this breaks hardware isolation: devices in the same group can access one another through DMA. It must not be an automatic fallback after moving slots fails. If reliable isolation is unavailable, abandon the assignment or replace the hardware.

flowchart TD
    A[物理 PCIe 设备] --> B{IOMMU 组内仅有目标设备?}
    B -->|是| C[跳过 ACS]
    B -->|否| D{可以更换 PCIe 插槽?}
    D -->|是| E[换插槽后重试]
    D -->|否| F[停止分配并评估硬件隔离]
    F --> G["ACS override 不提供真实硬件隔离"]
    G --> H[⚠️ 安全隔离被破坏,注意风险]
    C --> I[绑定到 vfio-pci]

2.3 Complete GPU passthrough workflow

GPU passthrough is the most common VFIO use case. The following example uses an NVIDIA graphics card.

Step 1: Find the GPU PCI ID

lspci -nn | grep -i nvidia
# 输出示例:
# <PCI_BDF> VGA compatible controller [...] [<VENDOR_DEVICE_ID>]
# <PCI_BDF> Audio device [...] [<VENDOR_DEVICE_ID>]
#
# <VENDOR_DEVICE_ID> = GPU 本体
# <VENDOR_DEVICE_ID> = HDMI Audio 功能
# 检查同组与关联功能、宿主依赖和VFIO绑定条件;需要分配的功能明确列出

Step 2: Configure early binding to vfio-pci

Prevent host drivers from claiming the GPU at boot:

# /etc/modprobe.d/vfio.conf
# 用你的 GPU PCI ID 替换
options vfio-pci ids=<VENDOR_DEVICE_ID>,<VENDOR_DEVICE_ID>

# 同时黑名单宿主机显卡驱动
# /etc/modprobe.d/blacklist.conf
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
# /etc/mkinitcpio.conf (Arch Linux)
MODULES=(... vfio_pci vfio vfio_iommu_type1 ...)
# 旧内核可能有 vfio_virqfd;Linux 6.2 起功能并入 VFIO,勿盲目加载不存在的模块。
# 然后 sudo mkinitcpio -P

# /etc/initramfs-tools/modules (Debian/Ubuntu)
vfio
vfio_iommu_type1
vfio_pci

# 然后 sudo update-initramfs -u -k all

Add to the kernel command line:

vfio-pci.ids=<VENDOR_DEVICE_ID>,<VENDOR_DEVICE_ID>

Verify the binding:

lspci -k -s "<PCI_BDF>"
# Kernel driver in use: vfio-pci  <-- 成功

Step 3: Configure libvirt XML

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='<PCI_DOMAIN>' bus='<GPU_BUS>' slot='<GPU_SLOT>' function='<GPU_FUNCTION>'/>
  </source>
  <rom bar='on'/>
  <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='<PCI_DOMAIN>' bus='<GPU_BUS>' slot='<GPU_SLOT>' function='<GPU_AUDIO_FUNCTION>'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</hostdev>

NVIDIA Error 43 Some older NVIDIA drivers imposed virtualization-detection restrictions; this cannot be assumed of modern drivers in general. Error 43 may also result from ROM, reset, firmware, or driver mismatches. The KVM/vendor_id hiding settings below are historical compatibility fragments, not defaults required by every GPU. Diagnose the actual versions and logs first:

<features>
  <hyperv>
    <vendor_id state='on' value='KVM Hv'/>
  </hyperv>
  <kvm>
    <hidden state='on'/>
  </kvm>
</features>

Also add the following to the <cpu> block: <feature policy='disable' name='hypervisor'/>

2.4 Single-GPU passthrough (without a second graphics card)

If you have only one graphics card, the host loses graphical output after passing it through. The following additional configuration is needed:

  1. Make the host headless: manage it remotely over SSH
  2. Keep the boot process from claiming the GPU: use the video=efifb:off kernel parameter
  3. Extract the GPU ROM: some GPUs require a ROM file
    echo 1 > /sys/bus/pci/devices/<PCI_BDF>/rom
    cat /sys/bus/pci/devices/<PCI_BDF>/rom > gpu.rom
    echo 0 > /sys/bus/pci/devices/<PCI_BDF>/rom
  4. Return the GPU on shutdown: a libvirt hook script unbinds vfio-pci when the VM stops

2.5 USB controller passthrough

Advantages of passing through an entire USB controller rather than individual USB devices:

Passthrough method Hotplug Latency Compatibility
USB redirection (Spice) Manual selection required Higher Depends on guest tools
Individual USB device Not supported Low Only selected devices are passed through
Entire USB controller Native support Native latency All USB devices are available
# 找到独立的 USB 控制器
lspci | grep USB
# 示例: <USB_CONTROLLER_BDF> USB controller: Intel Corp. ...

# 在 virt-manager 中: Add Hardware → PCI Host Device → 选择 USB 控制器

Best practice for gaming VMs For a gaming VM, pass through a dedicated USB 3.0 PCIe expansion card. This lets you physically switch peripherals such as the keyboard, mouse, and headset.


3. virtio drivers and performance optimization#

3.1 How the virtio architecture works

virtio is a paravirtualized I/O framework. The guest knows it is running in a virtual machine and communicates directly with QEMU's virtio backend through virtio drivers, bypassing the intermediate layer of full device emulation.

flowchart LR
    subgraph Guest
        A[Guest Application]
        B[virtio Frontend Driver]
    end
    subgraph Hypervisor
        C[QEMU virtio Backend]
        D["vhost (kernel)"]
        E["vhost-user (userspace)"]
    end
    subgraph Host Hardware
        F[Physical I/O Device]
    end

    A --> B
    B -->|virtqueue / shared memory| C
    C --> D
    C --> E
    D --> F
    E --> F

3.2 Comparing virtio device types

Device type Storage/network device Performance characteristics Recommended use
virtio-blk Block device Multiqueue support; actual IOPS depend on the backend and configuration General storage
virtio-scsi SCSI controller Multiple LUNs and SCSI command support Workloads requiring SCSI features
virtio-net NIC Multiqueue and vhost acceleration All networking scenarios
virtio-gpu Graphics card virgl 3D acceleration (Linux guests) Linux VMs requiring 3D
virtio-serial Serial port VM-guest communication channel Guest-agent communication
virtio-rng Random numbers Obtain entropy from the host Cryptography-intensive applications
virtio-balloon Memory Dynamically adjust guest memory Memory overcommit
virtio-fs Filesystem Host-guest directory sharing Shared file directories
virtio-input Input devices Low-latency mouse/keyboard Alternative to USB redirection
virtio-mem Memory hotplug Add or remove memory at runtime Flexible memory management

3.3 Installing virtio drivers in a Windows guest

Windows does not include virtio drivers; download them from the Fedora project:

# 下载 virtio-win ISO
wget https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso

# 挂载到虚拟机,然后在 Windows 中浏览安装

Installation order:

  1. During installation: when selecting a disk in Windows Setup, choose "Load driver" -> browse the virtio ISO -> select the viostor directory for the corresponding Windows version
  2. After installation: run virtio-win-guest-tools.exe or install them individually:
    • NetKVM — network driver
    • viostor — disk driver
    • vioinput — mouse/keyboard input
    • qxldod — QXL display driver
    • vioserial — serial driver for the guest agent
  3. guest-agent: install qemu-ga-x64.msi

Linux Guest:

# virtio 驱动内置于内核主线,只需安装 guest-agent
sudo apt install qemu-guest-agent        # Debian/Ubuntu
sudo pacman -S qemu-guest-agent           # Arch
sudo dnf install qemu-guest-agent         # Fedora

# 启用并启动服务
sudo systemctl enable --now qemu-guest-agent

3.4 CPU and memory performance tuning

CPU Pinning:

Pin vCPUs to specific physical CPU cores to avoid scheduling jitter, which is crucial for latency-sensitive workloads.

<cputune>
  <vcpupin vcpu='0' cpuset='2'/>
  <vcpupin vcpu='1' cpuset='8'/>
  <vcpupin vcpu='2' cpuset='3'/>
  <vcpupin vcpu='3' cpuset='9'/>
  <emulatorpin cpuset='0-1'/>
  <iothreadpin iothread='1' cpuset='10-11'/>
</cputune>

SMT / Hyper-Threading considerations The cpuset numbers are illustrative logical CPU IDs; 2 and 8 are not guaranteed to be SMT siblings. First check the actual topology with lscpu -e and thread_siblings_list, reserve housekeeping CPUs, and measure the effects of pinning against shared-cache and latency goals. iothreadpin also requires the corresponding iothread to be defined in the domain.

CPU isolation (isolcpus):

# 内核命令行添加,防止宿主机使用指定核心
isolcpus=2-9,10-17
# isolcpus 的 domain 隔离标志被标为 deprecated,其他标志语义不同;优先评估可动态管理的 cpuset,并保留 housekeeping CPU

vCPU topology:

<cpu mode='host-passthrough' check='none'>
  <topology sockets='1' dies='1' cores='4' threads='2'/>
  <cache mode='passthrough'/>
  <feature policy='require' name='topoext'/>
</cpu>
  • host-passthrough: expose the host's full CPU feature set to the guest for maximum performance
  • check='none': skip QEMU's CPU model validation
  • sockets='1' cores='4' threads='2': emulate 1 socket x 4 cores x 2 threads = 8 vCPUs

HugePages:

# 宿主机配置 2MB 大页
# /etc/default/grub 或内核命令行
hugepagesz=2M hugepages=4096
# 4096 × 2MB = 8GB 大页池

# 运行时配置
sudo sysctl vm.nr_hugepages=4096

# 查看大页状态
cat /proc/meminfo | grep Huge
<!-- libvirt XML -->
<memoryBacking>
  <hugepages/>
  <locked/>
</memoryBacking>

When should you use HugePages? For virtual machines with memory > 8GB that need stable performance, such as gaming, databases, and AI inference, huge pages can significantly reduce TLB misses. For lightweight VMs, however, THP (Transparent Huge Pages) is sufficient.

Performance optimization quick reference:

Technique Effect Use case Complexity
CPU Pinning Eliminate vCPU scheduling jitter Latency-sensitive workloads (gaming/real-time) Medium
isolcpus / cgroup Prevent the host from preempting cores Use with CPU Pinning Medium
HugePages Reduce TLB misses Large-memory VMs Medium
host-passthrough Maximize exposed CPU features For cases that do not require heterogeneous migration; measurement is still needed Low
IOThreads Dedicated disk I/O threads High-IOPS workloads Low
Multi-queue NIC Parallel network processing Multiple vCPUs + high throughput Low

3.5 Network optimization

Multi-queue virtio-net:

<interface type='network'>
  <model type='virtio'/>
  <driver name='vhost' queues='8'/>
</interface>

The recommended number of queues equals the number of vCPUs.

# Guest 内设置队列数
sudo ethtool -L eth0 combined 8

# 验证
ethtool -l eth0

vhost-net vs vhost-user:

Backend Location Latency Features
QEMU userspace QEMU process High Best compatibility
vhost-net Kernel Low Standard choice
vhost-user DPDK/SPDK userspace process Very low Additional configuration required

3.6 Disk I/O optimization

<!-- iothreads 在 domain 顶层,disk driver 引用对应 ID;本片段须放入正确位置。 -->
<iothreads>1</iothreads>
<!-- devices 内: -->
<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap' iothread='1'/>
  <source file='/var/lib/libvirt/images/vm.qcow2'/>
  <target dev='vda' bus='virtio'/>
</disk>
Parameter Option Meaning
cache none Bypass the host page cache with direct O_DIRECT access
cache writeback Use the host page cache; correctness depends on flush handling and storage, and performance must be measured
cache writethrough Synchronous acknowledgment; does not guarantee safety under every device failure, and latency must be measured
cache directsync O_DIRECT + synchronous acknowledgment; still requires durability guarantees from the underlying storage
io native Use Linux AIO, usually with cache=none/directsync; measure performance, and check version-specific io_uring support
discard unmap Support TRIM/DISCARD to reclaim disk space

4. virtiofs: file sharing between host and virtual machine#

virtiofs is a paravirtualized filesystem introduced in QEMU 5.0+ that replaces the aging virtio-9p and provides high-performance file sharing between host and guest.

4.1 Comparison with alternatives

Approach Throughput POSIX semantics Configuration complexity Recommendation
virtiofs Very high Complete Medium First choice
virtio-9p Low Incomplete Low Legacy compatibility
NFS Medium-high Mostly complete Medium Sharing across multiple VMs
Samba/CIFS Low-medium Incomplete Low Windows guest
VirtFS (9p) Low Partial Low Obsolete

4.2 Requirements

  • Linux Guest kernel >= 5.4 (virtiofs driver; the host also needs the corresponding software support)
  • QEMU >= 5.0 (recommended: >= 8.0)
  • libvirt >= 6.2
  • The virtiofsd daemon, now commonly a standalone Rust package; check paths and options for the version in use
# 确认 virtiofsd 可用
which virtiofsd
# 通常位于 /usr/lib/qemu/virtiofsd 或 /usr/libexec/virtiofsd

4.3 Complete configuration

Step 1: Create a shared directory on the host

mkdir -p ~/vm-shared
chmod 755 ~/vm-shared

Step 2: Configure libvirt memory backing

virtiofs requires shared memory (memfd) as its communication channel:

<domain type='kvm'>
  <!-- ... -->
  <memoryBacking>
    <source type='memfd'/>
    <access mode='shared'/>
  </memoryBacking>
  <!-- ... -->
</domain>

Step 3: Add a filesystem device

<filesystem type='mount' accessmode='passthrough'>
  <driver type='virtiofs'/>
  <source dir='/home/user/vm-shared'/>
  <target dir='share_tag'/>
  <binary path='/usr/libexec/virtiofsd' xattr='on'>
    <cache mode='always'/>
    <sandbox mode='chroot'/>
  </binary>
</filesystem>
Parameter Description
accessmode='passthrough' Handle permissions/ownership using passthrough semantics; check actual UID/GID mappings and daemon privileges
source dir Directory path on the host
target dir Mount tag in the guest: an identifier, not an actual path
cache mode='always' More aggressive caching; check consistency semantics when the host modifies files concurrently
cache mode='auto' Default: automatically selected based on the access pattern
xattr='on' Enable extended-attribute support (SELinux)

Step 4: Mount inside the guest

# 手动挂载
sudo mount -t virtiofs share_tag /mnt/shared

# 验证挂载
mount | grep virtiofs
df -h /mnt/shared

# 自动挂载 (/etc/fstab)
share_tag  /mnt/shared  virtiofs  defaults  0  0

fstab mount ordering virtiofs is a local virtual device and need not be treated as a network filesystem. Choose nofail,x-systemd.automount as appropriate for the guest's systemd/boot dependencies, and verify device readiness and the mount result. _netdev changes only mount-ordering semantics:

share_tag  /mnt/shared  virtiofs  defaults,nofail,x-systemd.automount  0  0

Navigation in virt-manager:

  1. Add Hardware -> Filesystem
  2. Type: mount, Driver: virtiofs
  3. Source path: /home/user/vm-shared
  4. Target path: share_tag (a custom tag)

4.4 Configuring a DAX (Direct Access) window

This section illustrates the DAX design; it does not mean the memfd/cache fragments below enable a DAX window. DAX requires matching QEMU, virtiofsd, and guest-kernel support, as well as actual window configuration. Check the corresponding versions to see whether the current implementation supports it; adding mount -o dax alone does not guarantee it will work.

DAX is designed to let the guest map host file pages, reducing some caching paths. Its benefits and support requirements must be verified across the full software stack:

<memoryBacking>
  <source type='memfd'/>
  <access mode='shared'/>
</memoryBacking>
<!-- ... -->
<filesystem type='mount' accessmode='passthrough'>
  <driver type='virtiofs'/>
  <source dir='/home/user/vm-shared'/>
  <target dir='share_tag'/>
  <binary path='/usr/libexec/virtiofsd'>
    <cache mode='always'/>
  </binary>
</filesystem>

Mount with DAX:

sudo mount -t virtiofs -o dax share_tag /mnt/shared

5. Advanced disk-image operations#

qemu-img is intended for offline disks. Stop all users of a disk and take backups before any modification, conversion, repair, snapshot, resize, commit, or rebase; even online queries may see inconsistent metadata. check -r all can lose data, so copy the original first. qemu-img internal disk snapshots do not save guest RAM, UEFI NVRAM, or TPM state.

5.1 qemu-img command reference

# ========== 创建镜像 ==========

# 创建 qcow2 格式(精简配置,按需增长)
qemu-img create -f qcow2 disk.qcow2 40G

# 创建预分配 qcow2(避免碎片)
qemu-img create -f qcow2 -o preallocation=full disk.qcow2 40G

# 创建 raw 格式(通常是稀疏文件;完整预分配可显式选择 preallocation=full)
qemu-img create -f raw disk.raw 40G

# 创建 qcow2 带集群大小(默认 64K)
qemu-img create -f qcow2 -o cluster_size=2M disk.qcow2 100G

# ========== 转换镜像 ==========

# raw -> qcow2(最常用:回收空间)
qemu-img convert -f raw -O qcow2 old.img new.qcow2

# qcow2 -> raw(性能优化)
qemu-img convert -f qcow2 -O raw old.qcow2 new.raw

# 压缩转换(-c 压缩目标镜像)
qemu-img convert -c -f raw -O qcow2 old.img compressed.qcow2

# 离线转换:-W 允许目标乱序写,不提供在线一致性;仅在合适的预分配目标上考虑
qemu-img convert -f raw -O qcow2 source.img target.qcow2
# 进阶示意:预分配 raw 目标;仍要求源离线
qemu-img convert -W -f raw -O raw -o preallocation=full source.img target.raw

# ========== 查看信息 ==========

# 查看镜像元数据
qemu-img info disk.qcow2
# virtual size: 40 GiB
# disk size: 2.3 GiB
# cluster_size: 65536
# backing file: base.qcow2

# 查看详细 JSON 输出
qemu-img info --output=json disk.qcow2

# 一致性检查
qemu-img check disk.qcow2

# 修复发现的错误(慎用)
qemu-img check -r all disk.qcow2

# ========== 快照操作 ==========

# 创建快照
qemu-img snapshot -c snap1_before_update disk.qcow2

# 列出所有快照
qemu-img snapshot -l disk.qcow2

# 应用/回滚到某快照
qemu-img snapshot -a snap1_before_update disk.qcow2

# 删除快照
qemu-img snapshot -d snap1_before_update disk.qcow2

# ========== 调整大小 ==========

# 扩容(仅扩大虚拟磁盘,分区/文件系统需在 guest 内扩展)
qemu-img resize disk.qcow2 +20G

# 缩容(先在客体/离线工具中安全缩文件系统与分区;新边界必须容纳全部结构与数据,XFS 等不支持常规缩容)
qemu-img resize disk.qcow2 -10G --shrink

# 显式截短该镜像的虚拟大小(不是重新计算链;满足全部缩容前提才可操作)
qemu-img resize --shrink disk.qcow2 20G

5.2 Backing Chains (incremental image chains)

qcow2 supports incremental layers based on a backing file, similar to an overlay filesystem:

# 创建基础镜像
qemu-img create -f qcow2 base.qcow2 40G

# 基于基础镜像创建 overlay(记录差异)
qemu-img create -f qcow2 -b base.qcow2 -F qcow2 overlay.qcow2

# 查看 backing chain
qemu-img info --backing-chain overlay.qcow2
flowchart TD
    A["base.qcow2 / OS 安装 + 基础软件 / 40GB virtual, 15GB actual"]
    B["overlay1.qcow2 / backing: base.qcow2 / Application 1 改变 / diff: 2GB"]
    C["overlay2.qcow2 / backing: overlay1.qcow2 / Application 2 改变 / diff: 500MB"]
    A -->|backing file| B
    B -->|backing file| C

Backing Chain operations:

# 合并 overlay 到 base(commit)
qemu-img commit overlay.qcow2

# 重构 backing chain(rebase)
qemu-img rebase -b new_base.qcow2 -F qcow2 overlay.qcow2

# 拍平 backing chain(合并所有 overlay 生成独立镜像)
qemu-img convert -f qcow2 -O qcow2 overlay2.qcow2 standalone.qcow2

Dangerous backing-chain operations Sectors in the base that are not overridden by the overlay are still read, so rewriting a shared base changes the data view of every guest that depends on it. This should not be interpreted as "sector offsets necessarily become invalid." The base should remain immutable during normal operation.

  1. Before making changes, list all dependent chains, stop all writers, and back up the entire chain.
  2. commit writes to the backing file and may affect other overlays; it is not a generally safe preparatory step before modifying a shared base.
  3. Safe-mode rebase reads the old and new bases while preserving the guest's view. -u changes only metadata, and misuse can corrupt that view. Check the formats, paths, and underlying data.
  4. When you need an independent copy, prefer using convert to flatten the offline chain, then verify the copy. Do not run these tools on online images.

5.3 Storage pool management

Before defining an LVM/disk pool, check the device's stable ID, capacity, existing partitions, and users. Defining a pool is different from building/formatting it; the latter may destroy existing data. A physical block device is not a "blank image," so do not act on a /dev/sdX name alone. Backend support for a feature depends on the current libvirt build and version.

libvirt storage pool types:

Type Use case Features
Directory General purpose Simplest option; host filesystem
LVM High-performance block storage Snapshots and thin provisioning
ZFS Advanced storage Snapshots, clones, send/recv, compression, and deduplication
NFS Centralized storage/migration Remote access and sharing across multiple hosts
iSCSI SAN High availability, enterprise-grade
GlusterFS Distributed Horizontal scaling
RBD (Ceph) Cloud scale Very large scale, replication, and automatic repair

Create a storage pool:

# Directory 类型 (最常用)
virsh pool-define-as --name images --type dir --target /var/lib/libvirt/images
virsh pool-start images
virsh pool-autostart images

# LVM 类型
virsh pool-define-as --name lvm_pool --type logical \
  --source-device "/dev/disk/by-id/<LVM_SOURCE_DISK_ID>" --target /dev/vg_kvm

Storage pool management commands:

virsh pool-list --all              # 列出所有存储池
virsh pool-info images             # 存储池详情
virsh pool-refresh images          # 刷新(重新扫描卷)
virsh vol-list images              # 列出池中所有卷
virsh vol-info disk.qcow2 --pool images  # 卷信息
virsh vol-create-as images new_disk.qcow2 40G --format qcow2
virsh vol-delete disk.qcow2 --pool images

5.4 Disk performance comparison

Format Thin provisioning Snapshots Compression Performance Recommended use
raw Files can be sparse; backends can be thin-provisioned No qcow2 internal snapshots; backend snapshots may be available No native compression Depends on the backend and workload Simple file or block-device images
qcow2 Yes Yes Yes High (about 95% of raw) Recommended for general use
vmdk Yes Yes Partial Medium VMware compatibility
vdi Yes Yes Partial Medium VirtualBox compatibility

5.5 Expanding partitions inside the guest

qemu-img resize enlarges only the virtual disk; you must also expand the partition and filesystem inside the guest:

# === Guest 内操作 ===

# 1. 查看当前分区表
lsblk
sudo fdisk -l /dev/vda

# 2. 扩展分区(以 GPT + systemd-growfs 为例)
sudo growpart /dev/vda 1          # 扩展第 1 个分区

# 3. 扩展文件系统
# ext4
sudo resize2fs /dev/vda1
# XFS
sudo xfs_growfs /
# Btrfs
sudo btrfs filesystem resize max /

# 4. 验证
df -h /

Offline expansion with virt-resize If the guest has a complex partition layout, you can use libguestfs tools after shutting it down:

先离线备份并检查客体分区;预先创建足够大的目标文件(如 `qemu-img create -f qcow2 new.qcow2 <LARGER_SIZE>`),再执行:
`virt-resize --expand /dev/sda1 old.qcow2 new.qcow2`。这里 /dev/sda1 指镜像内的客体分区,不是宿主磁盘;支持的文件系统与扩展结果需核对。

6. Advanced network configuration#

The security and performance of NAT, isolated, bridge, and macvtap depend on rules, topology, and workload. The high/low ratings in the table are only an overview from the original notes, not measured results. NAT/isolated do not block guests from accessing the host bridge or other guests on the same subnet. bridge/macvtap require authorization from the external network, and macvtap is not PCI passthrough. Ordinary Wi-Fi clients often cannot transparently bridge multiple MAC addresses. Prepare local or out-of-band recovery before changing the host IP or routes.

6.1 Network mode comparison

Mode NAT Isolation Externally accessible Performance Use case
NAT (default) Yes High No (requires port forwarding) Medium Desktop virtualization
Bridged No Low Yes (IP on the same subnet) High Server virtualization
macvtap (VEPA) No Medium Yes Very high High-performance workloads
macvtap (Bridge) No Low Yes Very high High performance on a single host
PCI Passthrough (SR-IOV) No None Yes Near bare metal Maximum performance

6.2 Bridge networking configuration

# /etc/network/interfaces (Debian/Ubuntu) 或使用 NetworkManager / systemd-networkd

# 使用 nmcli (NetworkManager)
nmcli connection add type bridge ifname br0
nmcli connection add type ethernet slave-type bridge con-name bridge-br0 ifname eth0 master br0
nmcli connection up br0
<!-- libvirt XML -->
<interface type='bridge'>
  <!-- 未指定 MAC,由 libvirt 生成;迁移现有 VM 应保留其地址。 -->
  <source bridge='br0'/>
  <model type='virtio'/>
</interface>

6.3 High-performance macvtap passthrough

macvtap associates a guest virtual NIC with a host physical interface without a traditional bridge. It still traverses the host networking path and is not PCI/VFIO NIC passthrough:

<interface type='direct'>
  <source dev='eth0' mode='bridge'/>
  <model type='virtio'/>
</interface>

macvtap modes:

Mode Behavior Use case
VEPA Traffic between VMs is forwarded through an external switch Enterprise environments
Bridge VMs on the host can communicate directly Multiple VMs on a single host
Private VMs are completely isolated from one another Security isolation
Passthrough One VM has exclusive use of the NIC High performance for a single VM

macvtap limitations In macvtap bridge mode, the host itself cannot communicate with its VMs, even on the same physical NIC. This is a design limitation of Linux macvlan. If host-to-VM communication is required, configure an additional macvlan on the host's macvtap interface or use a traditional bridge instead.



References & further reading

  1. VFIO and IOMMU groups ↗
  2. libvirt virtiofs ↗
  3. QEMU qemu-img ↗
  4. Domain XML ↗
  5. Linux kernel parameters ↗

Adapted from technical notes. Examples use generic names.

Continue along a reading path

Advanced KVM capabilities and platform choices

  1. 01KVM firmware, VFIO, virtiofs, and image chains
  2. 02Declaring KVM host capabilities on NixOS
  3. 03Choosing KVM management and automation tools
CONTINUE READING / NEXT ARTICLEDeclaring KVM host capabilities on NixOS

Configure libvirt through version-pinned module interfaces while managing networking, firmware, and VM runtime state outside the configuration.