Virtualization - libvirt


The libvirt library is used to interface with different virtualization technologies. Before getting started with libvirt it is best to make sure your hardware supports the necessary virtualization extensions for KVM. Enter the following from a terminal prompt:


A message will be printed informing you if your CPU does or does not support hardware virtualization.


On many computers with processors supporting hardware assisted virtualization, it is necessary to activate an option in the BIOS to enable it.

Virtual Networking

There are a few different ways to allow a virtual machine access to the external network. The default virtual network configuration includes bridging and iptables rules implementing usermode networking, which uses the SLIRP protocol. Traffic is NATed through the host interface to the outside network.

To enable external hosts to directly access services on virtual machines a different type of bridge than the default needs to be configured. This allows the virtual interfaces to connect to the outside network through the physical interface, making them appear as normal hosts to the rest of the network.

There is a great example how to configure an own bridge and combining it with libvirt so that guests will use it at the


To install the necessary packages, from a terminal prompt enter:

sudo apt update
sudo apt install qemu-kvm libvirt-daemon-system

After installing libvirt-daemon-system, the user used to manage virtual machines will need to be added to the libvirt group. This is done automatically for members of the sudo group, but needs to be done in additon for anyone else that should access system wide libvirt resources. Doing so will grant the user access to the advanced networking options.

In a terminal enter:

sudo adduser $USER libvirt


If the user chosen is the current user, you will need to log out and back in for the new group membership to take effect.

You are now ready to install a Guest operating system. Installing a virtual machine follows the same process as installing the operating system directly on the hardware.

You either need:

  • a way to automate the installation
  • a keyboard and monitor will need to be attached to the physical machine.
  • use cloud images which are meant to self-initialize (see Multipass and UVTool)

In the case of virtual machines a Graphical User Interface (GUI) is analogous to using a physical keyboard and mouse on a real computer. Instead of installing a GUI the virt-viewer or virt-manager application can be used to connect to a virtual machine’s console using VNC. See Virtual Machine Manager / Viewer for more information.

Virtual Machine Management

The following section covers the command-line tools around virsh that are part of libvirt itself. But there are various options at different levels of complexities and feature-sets, like:


There are several utilities available to manage virtual machines and libvirt. The virsh utility can be used from the command line. Some examples:

  • To list running virtual machines:

    virsh list
  • To start a virtual machine:

    virsh start <guestname>
  • Similarly, to start a virtual machine at boot:

    virsh autostart <guestname>
  • Reboot a virtual machine with:

    virsh reboot <guestname>
  • The state of virtual machines can be saved to a file in order to be restored later. The following will save the virtual machine state into a file named according to the date:

    virsh save <guestname> save-my.state

    Once saved the virtual machine will no longer be running.

  • A saved virtual machine can be restored using:

    virsh restore save-my.state
  • To shutdown a virtual machine do:

    virsh shutdown <guestname>
  • A CDROM device can be mounted in a virtual machine by entering:

    virsh attach-disk <guestname> /dev/cdrom /media/cdrom
  • To change the definition of a guest virsh exposes the domain via

    virsh edit <guestname>

That will allow one to edit the XML representation that defines the guest and when saving it will apply format and integrity checks on these definitions.

Editing the XML directly certainly is the most powerful way, but also the most complex one. Tools like Virtual Machine Manager / Viewer can help unexperienced users to do most of the common tasks.

If virsh (or other vir* tools) shall connect to something else than the default qemu-kvm/system hipervisor one can find alternatives for the connect option in man virsh or libvirt doc

system and session scope

Virsh - as well as most other tools to manage virtualization - can be passed connection strings.

$ virsh --connect qemu:///system

There are two options for the connection.

  • qemu:///system - connect locally as root to the daemon supervising QEMU and KVM domains
  • qemu:///session - connect locally as a normal user to his own set of QEMU and KVM domains

The default always was (and still is) qemu:///system as that is the behavior users are used to.
But there are a few benefits (and drawbacks) to qemu:///session to consider it.

qemu:///session is per user and can on a multi-user system be used to separate the people.
But most importantly things run under the permissions of the user which means no permission struggle on the just donwloaded image in your $HOME or the just attached USB-stick.
On the other hand it can’t access system resources that well, which includes network setup that is known to be hard with qemu:///session. It falls back to slirp networking which is functional but slow and makes it impossible to be reached from other systems.

qemu:///system is different in that it is run by the global system wide libvirt that can arbitrate resources as needed.
But you might need to mv and/or chown files to the right places permssions to have them usable.

Applications usually will decide on their primary use-case. Desktop-centric applications often choose qemu:///session while most solutions that involve an administrator anyway continue to default to qemu:///system.

Read more about that in the libvirt FAQ and this blog about the topic.


There are different types of migration available depending on the versions of libvirt and the hipervisor being used. In general those types are:

There are various options to those methods, but the entry point for all of them is virsh migrate. Read the integrated help for more detail.

 virsh migrate --help 

Some useful documentation on constraints and considerations about live migration can be found at the Ubuntu Wiki

Device Passthrough / Hotplug

If instead of the here described hotplugging you want to always pass through a device add the xml content of the device to your static guest xml representation via e.g. virsh edit <guestname>. In that case you don’t need to use attach/detach. There are different kinds of passthrough. Types available to you depend on your Hardware and software setup.

  • USB hotplug/passthrough

  • VF hotplug/Passthrough

But both kinds are handled in a very similar way and while there are various way to do it (e.g. also via qemu monitor) driving such a change via libvirt is recommended. That way libvirt can try to manage all sorts of special cases for you and also somewhat masks version differences.

In general when driving hotplug via libvirt you create a xml snippet that describes the device just as you would do in a static guest description. A usb device is usually identified by Vendor/Product id’s:

<hostdev mode='subsystem' type='usb' managed='yes'>
    <vendor id='0x0b6d'/>
    <product id='0x3880'/>

Virtual functions are usually assigned via their PCI-ID (domain, bus, slot, function).

<hostdev mode='subsystem' type='pci' managed='yes'>
    <address domain='0x0000' bus='0x04' slot='0x10' function='0x0'/>


To get the Virtual function in the first place is very device dependent and can therefore not be fully covered here. But in general it involves setting up an iommu, registering via VFIO and sometimes requesting a number of VFs. Here an example on ppc64el to get 4 VFs on a device:

$ sudo modprobe vfio-pci
# identify device
$ lspci -n -s 0005:01:01.3
0005:01:01.3 0200: 10df:e228 (rev 10)
# register and request VFs
$ echo 10df e228 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
$ echo 4 | sudo tee /sys/bus/pci/devices/0005\:01\:00.0/sriov_numvfs

You then attach or detach the device via libvirt by relating the guest with the xml snippet.

virsh attach-device <guestname> <device-xml>
# Use the Device int the Guest
virsh detach-device <guestname> <device-xml>

Access Qemu Monitor via libvirt

The Qemu Monitor is the way to interact with qemu/KVM while a guest is running. This interface has many and very powerful features for experienced users. When running under libvirt that monitor interface is bound by libvirt itself for management purposes, but a user can run qemu monitor commands via libvirt still. The general syntax is virsh qemu-monitor-command [options] [guest] 'command'

Libvirt covers most use cases needed, but if you every want/need to work around libvirt or want to tweak very special options you can e.g. add a device that way:

virsh qemu-monitor-command --hmp focal-test-log 'drive_add 0 if=none,file=/var/lib/libvirt/images/test.img,format=raw,id=disk1'

But since the monitor is so powerful, you can do a lot especially for debugging purposes like showing the guest registers:

virsh qemu-monitor-command --hmp y-ipns 'info registers'
RAX=00ffffc000000000 RBX=ffff8f0f5d5c7e48 RCX=0000000000000000 RDX=ffffea00007571c0
RSI=0000000000000000 RDI=ffff8f0fdd5c7e48 RBP=ffff8f0f5d5c7e18 RSP=ffff8f0f5d5c7df8

Huge Pages

Using huge pages can help to reduce TLB pressure, page table overhead and speed up some further memory relate actions. Furthermore by default Transparent huge pages are useful, but can be quite some overhead - so if it is clear that using huge pages is preferred making them explicit usually has some gains.

While huge page are admittedly harder to manage, especially later in the lifetime of a system if memory is fragmented they provide a useful boost especially for rather large guests.

Bonus: When using device pass through on very large guests there is an extra benefit of using huge pages as it is faster to do the initial memory clear on vfio dma pin.

Huge page allocation

Huge pages come in different sizes. A normal page usually is 4k and huge pages are eithe 2M or 1G, but depending on the architecture other options are possible.

The most simple, yet least reliable way to allocate some huge pages is to just echo a value to sysfs
Be sure to re-check if it worked.

$ echo 256 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
$ cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

There one of these sizes is “default huge page size” which will be used in the auto-mounted /dev/hugepages.
Changing the default size requires a reboot and is set via default_hugepagesz

You can check the current default size:

$ grep Hugepagesize /proc/meminfo
Hugepagesize:       2048 kB

But there can be more than one at the same time one better check:

$ tail /sys/kernel/mm/hugepages/hugepages-*/nr_hugepages` 
==> /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages <==
==> /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages <==

And even that could on bigger systems be further split per Numa node.

One can allocate huge pages at boot or runtime, but due to fragmentation there are no guarantees it works later. The kernel documentation lists details on both ways.

Huge pages need to be allocated by the kernel as mentioned above but to be consumable they also have to be mounted.
By default systemd will make /dev/hugepages available for the default huge page size.
Feel free to add more mount points if you need different sized.
An overview can be queried with

$ hugeadm --list-all-mounts
Mount Point          Options
/dev/hugepages       rw,relatime,pagesize=2M

A one-stop info for the overall huge page status of the system can be reported with

$ hugeadm --explain

Huge page usage in libvirt

With the above in place libvirt can map guest memory to huge pages.
In a guest definition add the most simple form of


That will allocate the huge pages using the default huge page size from a autodetected mountpoint.
For more control e.g. how memory is spread over Numa nodes or which page size to use check out the details at the libvirt doc.

Apparmor isolation

By default libvirt will spawn qemu guests using apparmor isolation for enhanced security. The apparmor rules for a guest will consist of multiple elements:

  • a static part that all guests share => /etc/apparmor.d/abstractions/libvirt-qemu
  • a dynamic part created at guest start time and modified on hotplug/unplug => /etc/apparmor.d/libvirt/libvirt-f9533e35-6b63-45f5-96be-7cccc9696d5e.files

Of the above the former is provided and updated by the libvirt-daemon package and the latter is generated on guest start. None of the two should be manually edited. They will by default cover the vast majority of use cases and work fine. But there are certain cases where users either want to:

  • further lock down the guest (e.g. by explicitly denying access that usually would be allowed)
  • open up the guest isolation (most of the time this is needed if the setup on the local machine does not follow the commonly used paths.

To do so there are two files to do so. Both are local overrides which allow you to modify them without getting them clobbered or command file prompts on package upgrades.

  • /etc/apparmor.d/local/abstractions/libvirt-qemu this will be applied to every guest. Therefore it is rather powerful, but also a rather blunt tool. It is a quite useful place thou to add additional deny rules.
  • /etc/apparmor.d/local/usr.lib.libvirt.virt-aa-helper the above mentioned dynamic part that is individual per guest is generated by a tool called libvirt.virt-aa-helper. That is under apparmor isolation as well. This is most commonly used if you want to use uncommon paths as it allows to ahve those uncommon paths in the guest XML (see virsh edit) and have those paths rendered to the per-guest dynamic rules.


  • See the KVM home page for more details.

  • For more information on libvirt see the libvirt home page

    • xml configuration of domains and storage being the most often used libvirt reference
  • Another good resource is the Ubuntu Wiki KVM page.

  • For basics how to assign VT-d devices to qemu/KVM, please see the linux-kvm page.

1 Like

@powersj Thank you very much for such a nice guideline !

It could be nice to add sudo apt-get update before running installation command for libvirt under Installation, header. I tried to edit and submit for confirmation but seems it is not possible.

@powersj arr, you get all the fame for my improved howto :slight_smile:

@mrturkmen it is generally recommended to be “up to date” in regard to “apt update” and it is running automatically in background. If we’d add it here we’d have to add it throughout all of the serverguide. Is there a particular reason why you think an “apt update” is needed in this case that makes it special to be worth adding it here?

@paelzer Thank you for your reply. I have followed through the article however, ubuntu was unable to find required packages which are installed in first step of the installation guide. Yeah, I fully agree that it is recommended to be “up to date” in regard to “apt update” . However, in my case, I was using a vagrant installation of ubuntu and it was not up to date, since it is not running all the time, I am running it whenever I need it. So, apart from it, there are many scenarios which may require to run update command, I think, it does not harm anything to have it from the beginning of the installation guide of libvirt.

Yeah @mrturkmen, while I think it would be overload to add the update to any ‘apt install’ call adding it to the first one of the page should be fine. Thanks for the discussion.

P.S. I also fixed a (now bad) link on the page.