QEMU GPU Passthrough

Tutorials
1

QEMU GPU Passthrough

Configure GPU passthrough using QEMU/KVM.

Key Value
Summary How to get GPU passthrough working on Ubuntu.
Categories desktop
Difficulty 5
Author Phil Guinchard phil.guinchard@slackdaystudio.ca

Overview

Duration: 01:00

What you’ll learn

This tutorial will guide you through the steps required to configure GPU passthrough to virtual machines using QEMU.

What you’ll need

  • A laptop or PC with one or more virtual machines
  • 2 GPUs

1.0 Required Software

Duration: 01:00

I’m running an Ubuntu 24.10-based system. I needed the following;

sudo apt install libvirt-daemon-system libvirt-clients qemu-kvm qemu-utils virt-manager ovmf

2.0 The Hardware

Duration: 2:00

This system is an older model Comet Lake CPU from Intel. Still, it has all the bells and whistles I need to run VMs. This computer is also equipped with two GPUs: the primary GPU is an AMD Radeon RX 7900 RT, and a Radeon RX 6400 GPU, which I dedicate to my VMs.

I only run one VM at a time because I typically shift most of my physical resources to my VMs, as that is where I will likely need them.

2.1 CPU Requirements

Check to see if you have Intels VT-d or AMDs AMD-Vi virtualization CPU extensions enabled. Make sure it is enabled in the BIOS first; this depends on your mobo make and model, and you will need to consult the manufacturer’s docs on how to enable this in your BIOS.

Once enabled in your BIOS, you can verify the extensions are on by running:

sudo dmesg | grep -E "VT-d|AMD-Vi"

Assuming you can see some output from that last command, you can begin by adding the correct kernel param for your CPU model.

3.0 Kernel Params

Duration: 5:00

For Intel, add intel_iommu=on, and for AMD, add amd_iommu=on to your kernel params.

I needed to edit GRUB first to make everything persist between reboots on my Ubuntu system.

sudo vi /etc/default/grub

Find this line…

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"`

…and add your param.

For example, on my Intel machine, it looks like:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on"

Update GRUB:

sudo update-grub

3.1 VFIO Kernel Modules

Add the following to the end of /etc/initramfs-tools/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vhost-net

Once you have added the VFIO modules, you must update your initramfs image to load these on host boot.

sudo update-initramfs -u

3.2 Reboot

Reboot your system; we will check to see if things work.

3.3 Validating Things

Run the following command to check if things came up appropriately configured.

sudo dmesg | grep -e DMAR -e IOMMU

The output for me that confirms things are configured is this line:

[    0.354991] DMAR: Intel(R) Virtualization Technology for Directed I/O

4.0 Add The Hooks

Duration: 5:00

Now, we can add our QEMU hooks to bind and unbind the GPU automatically. QEMU has a powerful hook system in place that allows us to do all sorts of things, including automatically controlling hardware.

The easiest thing to do is check out the qemu_hook_skeleton directory from my GitHub repo and drop it into /etc/libvirt/hooks/.

Let’s look at a few files and go over what they do.

/etc/libvirt/hooks/
├── kvm.conf
├── qemu
└── qemu.d
   └── gpu-passthrough
       ├── prepare
       │   └── begin
       │       └── bind_vfio.sh
       └── release
           └── end
               └── unbind_vfio.sh
  1. qemu - This is a key script for enabling our hooks to work. It is responsible for invoking our hooks.
  2. kvm.conf - These are some functions and vars common to all hooks. This is where you define your device to pass through
  3. qemu.d This is where we place our actual hook files
  4. qemu.d/gpu-passthrough - This is a stub that we will be symlinking our VMs too

4.1 Finding Your IOMMU Group

To enable the binding and unbinding of your GPU, you must first determine its iommu group.

find /sys/kernel/iommu_groups/ -type l -exec basename {} \;  | sort | xargs -I % lspci -nns %

In my case, it was these lines that were important:

02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 24 [Radeon RX 6400/6500 XT/6500M] [1002:743f] (rev c7)
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]

The relevant part is at the start of the lines; those are the iommu groups for your hardware. This is what you need to assign to your kvm.conf file on lines 2 and 3.

4.1 Symlinking

Now that everything is ready, you can start creating symlinks to define the hooks for your domains. Simply create a symlink to gpu-passthrough like so:

ln -s /etc/libvirt/hooks/qemu.d/gpu-passthrough /etc/libvirt/hooks/qemu.d/<VM_NAME>

Note: if you are unsure of the name of your VMs, run sudo virsh list --all

5.0 Assigning Hardware

Duration: 2:00

At this point, I used Virtual Machine Manager to add the GPU device to my VMs via the GUI. It’s a bit out of scope, but it was easy. Add Hardware->PCI Host Device and then pick the GPU and the GPU audio devices individually.

6.0 Wrapping Up

Duration: 1:00

From then on, I could get all my VMs binding and unbinding to the secondary GPU on my machine. Things are nice and snappy, with GPU hardware in the mix now. Adding new VMs via symlinking is easy, which is a nice plus to this approach.

Enjoy!

References

https://passthroughpo.st/simple-per-vm-libvirt-hooks-with-the-vfio-tools-hook-helper/