Setting up Virgl VMs with QEMU and libvirt on Ubuntu 18.04

VirGL

VirGL is a driver that allows creating a virtual GPU within Virtual Machines that leverages host’s GPU without the need for passing it through (VFIO). Currently it’s mature enough to support window managers and maybe some really, really simple quake clones (I tried running Half Life 2… It did not work well). Currently it only supports Linux guests, although there was an experiment to get Windows guests working as well.

Hardware wise you need a GPU with DRM support. Intel iGPUs should work, so should most AMD cards. With nvidia cards you’re probably stuck on using the nouveau driver which is not ideal. Obviously, you also need a system capable of virtualisation.

Setting up QEMU

Sadly the version of QEMU that comes with Ubuntu does not have VirGL support. Because of this, we’ll need to compile it with the necessary support.

Setting up QEMU is the easy part. In short -

1. sudo apt install build-essential libepoxy-dev libdrm-dev libgbm-dev libx11-dev libvirglrenderer-dev libpulse-dev libsdl2-dev python python-dev libpixman-1-dev build-essential.
2. wget http://download.qemu-project.org/qemu-2.12.0.tar.xz -O qemu.tar.xz (or whatever the latest version is at the time of reading).
3. mkdir qemu && tar -xf qemu.tar.xz -C qemu --strip-components=1 && cd qemu.
4. ./configure --enable-sdl --with-sdlabi=2.0 --enable-opengl --enable-virglrenderer --enable-system --enable-modules --audio-drv-list=pa --target-list=x86_64-softmmu --enable-kvm.

After configure finishes it should output all of the features the new QEMU version will support. The ones that interest us are - SDL Support, virgl support, KVM support, OpenGL support, OpenGL dmabufs - all of them should be set to yes.

5. make -j$(nproc).
6. sudo make install.

This will install our newly compiled version to /usr/local. You CAN install it over the distribution’s packages by setting the prefix for configure, but personally I would refrain from doing so if only because you can accidentally mess up your existing virtual machines.

Setting up Libvirt

To install Libvirt you have two options - compile it manually or install it from distribution’s repositories. The first option has the added ‘benefit’ of being able to disable Apparmor which will block you from using Libvirt with a custom version of QEMU.

To simply install Libvirt without compiling it, just run sudo apt install libvirt-bin and move on to the next section. Otherwise, if you choose to compile it yourself (which I don’t really recommend) then these commands should work -

1. sudo apt-get install git build-essential xsltproc libxml-xpath-perl libyajl-dev libdevmapper-dev libpciaccess-dev libnl-3-dev libnl-route-3-dev systemtap-sdt-dev uuid-dev libtool autoconf pkg-config libxml2 libxml2-utils autopoint python-dev libnuma-dev gettext gnutls-dev libxml2-dev numad librbd-dev build-essential.
2. wget https://libvirt.org/sources/libvirt-4.5.0.tar.xz -O libvirt.tar.xz (or find the latest version here).
3. mkdir libvirt && tar xf libvirt.tar.xz -C libvirt --strip-components=1 && cd libvirt.
4. ./configure --with-qemu=yes --with-dtrace --with-numad --with-storage-rbd --disable-nls. If you want to compile without Apparmor support, append --disable-apparmor to the command.
5. make -j$(nproc) && sudo make install.

This will install Libvirt to /usr/local.

Setting up apparmor for libvirt

If you compiled Libvirt without Apparmor support you can skip this section. Otherwise, you’ll need to manually add the executables to Apparmor access lists.

At the bottom of the file /etc/apparmor.d/usr.sbin.libvirtd, just before the closing curly bracket (}) add the following -

1
2
3
4
5
6
7
8

/usr/local/bin/qemu-system-x86_64 rmix,
/usr/local/bin/qemu-io rmix,
/usr/local/bin/qemu-pr-helper rmix,
/usr/local/bin/qemu-img rmix,
/usr/local/bin/qemu-nbd rmix,
/usr/local/bin/qemu-ga rmix,
/usr/local/bin/qemu-keymap rmix,
/usr/local/share/qemu/* r,

At the bottom of the /etc/apparmor.d/abstractions/libvirt-qemu add the following (you might need to replace /dev/dri/renderD128 with your own render device) -

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

/usr/local/bin/qemu-system-x86_64 rmix,
/usr/local/bin/qemu-io rmix,
/usr/local/bin/qemu-pr-helper rmix,
/usr/local/bin/qemu-img rmix,
/usr/local/bin/qemu-nbd rmix,
/usr/local/bin/qemu-ga rmix,
/usr/local/bin/qemu-keymap rmix,
/usr/local/share/qemu/* r,

# gpu access
/dev/dri/ r,
/etc/drirc r,
/usr/share/glvnd/egl_vendor.d/ r,
/usr/share/glvnd/egl_vendor.d/* r,
/etc/glvnd/egl_vendor.d/ r,
/dev/dri/renderD128 rw,
/usr/lib/x86_64-linux-gnu/dri/* rm,
/sys/devices/** r,

You will also need to symlink the required libraries using sudo ln -s /usr/lib/x86_64-linux-gnu/dri /usr/lib/dri because Libvirt (even the one from Ubuntu repositories) will not look into x86_64-linux-gnu by default.

Creating a virtual machine

Now the fun part. First create a VM the way you usually would (i.e. using virt-install, using a web GUI like Kimchi etc.) and set it up. What you definitely want to set up is some sort of remote desktop software, like x2go, nomachine, one of the VNC server options (1, 2…), or even something like teamviewer. The reason for this is that enabling virgl will effectively kill QEMU’s SPICE socket - there’s no support for tcp sockets (it simply won’t let you start the VM), and I could not get file sockets to work with virgl enabled (the spice client connects, even shows you the name of the VM you’ve connected to, but there’s no video output).

After setting up your virtual machine, shut it down, and edit its Libvirt xml definition (virsh edit <vm-name>). In it, you have to make the following changes (look below to see how the file should look like) -

1. Under <devices>, change (or add?) the <emulator> used to your new qemu installation, i.e. - <emulator>/usr/local/bin/qemu-system-x86_64</emulator>.
2. Set the <graphics> type to spice and also enable the Opengl support, i.e. -

1
2
3
4

<graphics type='spice'>
    <listen type='none'/>
    <gl enable='yes' rendernode='/dev/dri/renderD128'/>
</graphics>

3. Set the video adapter to VirtIO, i.e. -

1
2
3
4
5
6

<video>
    <model type='virtio' heads='1' primary='yes'>
    <acceleration accel3d='yes'/>
    </model>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>

Once done, save, let Libvirt validate it, and if it works, you can just start it. To test whether it works, connect to the VM and check if VirGL was enabled by running dmesg | grep "virgl 3d acceleration enabled". If your dmesg contains this line, you’re good to go.

Full xml definition file

For reference -

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91

<domain type='kvm'>
  <name>test-vm</name>
  <uuid>97d73144-ab79-4b8e-b03f-6a335207edb7</uuid>
  <maxMemory slots='256' unit='KiB'>8388608</maxMemory>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memtune>
    <hard_limit unit='KiB'>9437184</hard_limit>
  </memtune>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.12'>hvm</type>
    <boot dev='hd'/>
    <boot dev='cdrom'/>
    <boot dev='network'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu>
    <topology sockets='1' cores='1' threads='2'/>
    <numa>
      <cell id='0' cpus='0' memory='4194304' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/97d73144-ab79-4b8e-b03f-6a335207edb7-0.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/www/files/isos/ubuntu-18.04-desktop-amd64.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:db:cd:ab'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice'>
      <listen type='none'/>
      <gl enable='yes' rendernode='/dev/dri/renderD128'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='virtio' heads='1' primary='yes'>
        <acceleration accel3d='yes'/>
      </model>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>