LXD, integrated GPU passthrough, now fully blank desktop, how to get desktop back?

[ifsystems]

Hi everyone

Ubuntu Version: Ubuntu 22.04
Desktop Environment (if applicable): GNOME
Problem Description:
Please excuse me for this post, it’s definitely a newbie problem and a schoolkid error. I have searched a lot to try and find a direct answer but have found nothing that I can easily follow. If this is the wrong place to ask this question then please point me to where I should ask.

I have made a mistake setting up a VM GPU passthrough on a work Ubuntu laptop, and now the laptop is unusable. I enabled GPU passthrough for the integrated Intel GPU for a Win11 VM (trying to get CAD working) and when I started the VM the screen went blank. I did this in the LXD UI, not the terminal. The GPU passthrough was physical. I tried lots of keyboard shortcuts to try and get the screen to come back, but it didn’t work, though I imagine there’s probably a quick way to recover from this issue which I haven’t found yet. I have not been able to get the terminal to show at all. I have been able to turn the laptop off with the power button, both soft shutdown and hard stop, though soft shutdown takes a loooonnng time, takes ages to kill XOrg. The laptop will boot to the login screen, but then the screen goes blank after a few seconds. I believe this is because LXD starts and isolates the integrated GPU for the use of the VM, and Ubuntu on the laptop cannot use it, so can no longer display the desktop. The laptop does have a second discrete GPU, an NVidia one, which I would have done GPU passthrough with but at the time it was actually the one running XOrg according to the error message I got when I tried to start the VM with this config. I had expected Ubuntu to use the discrete GPU for itself, but I was probably naive thinking this.

One thing I can do is get to the terminal through GRUB, but this is of course with nothing loaded. From here I installed openssh-server but when I let the laptop boot to login screen again it did not appear on my local network, so I don’t think I can even SSH in and fix things.

Relevant System Information:
Dell Precision 3490, Intel integrated GPU and discrete Nvidia Ada 500 GPU.

Screenshots or Error Messages:
There’s nothing I can show as I can’t see the screen or SSH in.

What I’ve Tried:
Loads of shortcuts to try and get the desktop or terminal to show, but nothing works.

I installed OpenSSH-server from the GRUB terminal, however when I let the laptop boot to the login screen, and I blindly enter the password (because the screen is blank) the laptop doesn’t appear on the local network anyway so I’ve not been able to attempt to SSH in.

I imagine I need to somehow change the lxd config to remove the GPU passthrough, but so far I’ve not wrapped my head around how I can do that without starting lxd and therefore activating the passthrough and losing the screen again. Could you please advise how I can fix the problem? I will continue to try and find answers and attempt to fix this.

All help much appreciated

[thingizkhan]

Get your display back first stop LXD from grabbing the GPU

Boot to the GRUB menu
(hold Shift or tap Esc right after the Dell logo).

Highlight the default entry, press e, and add at the end of the line that starts with linux:

systemd.unit=multi-user.target

Press Ctrl + X to boot.
Ubuntu comes up in pure text mode, no GUI, so even if LXD starts you still have a usable console.

Disable the LXD daemon so it can’t claim the iGPU on the next boot:

sudo systemctl disable --now snap.lxd.daemon.service

(If you prefer, sudo systemctl mask snap.lxd.daemon.service works too.)

Reboot normally (just sudo reboot).
The GUI should appear again because nothing is pulling the Intel GPU away.

---

Remove the passthrough device from the VM

Once the desktop is back:

sudo systemctl start snap.lxd.daemon   # start LXD by hand

# list your VMs
lxc list

# pick the VM name (say "win11") and delete the GPU device you added
lxc config device remove win11 gpu0    # or whatever you called it

(If you also created a /etc/modprobe.d/vfio.conf or similar to bind the Intel
GPU to VFIO, delete or comment out that file and run sudo update-initramfs -u.)

---

Re-enable LXD and reboot

sudo systemctl enable --now snap.lxd.daemon
sudo reboot

Ubuntu should boot with its normal display, LXD will start, and the VM will run
without stealing the integrated GPU.

---

Tip next time: test GPU passthrough with the discrete NVIDIA card first,
and set boot.autostart=false on the VM so a bad config can’t brick the host
at startup.

[ifsystems]

Hi @thingizkhan,

Thank you for the helpful advice. I could almost follow all of the steps, but unfortunately after forcing Ubuntu to boot into the console it seems LXD was still running and taking the GPU away, so I got the login prompt for a few seconds and then the screen went blank again. It seems that with a blank screen it is not possible to blindly enter commands either, at least, I could not get that to work.

I then realised I could disable virtualisation in the BIOS and decided to try that. This worked, to a certain extent. It seems the LXD tried to start and take the GPU away a few times because the screen kept going blank and then coming back. After a few times it stopped repeating this and I was able to log in and then follow the rest of your steps. I then rebooted and went back into the BIOS and re-enabled virtualisation, and then booted into the desktop and everything seems to be working again.

I’ll go back to trying to get the GPU passthrough working with the discrete GPU and disabling autostart tomorrow.

Cheers

[thingizkhan]

Keep the VM from auto-starting

lxc config set win11 boot.autostart false

Now if a future GPU setting misbehaves, the host will still come up clean and you can tweak the VM at leisure.

---

Hand the discrete GPU to the VM safely

Find the PCI address

lspci -nn | grep -i nvidia

Attach it with a vendorid filter (avoids grabbing the display card Ubuntu is using)

lxc config device add win11 nvidia gpu \
    pci=0000:01:00.0 \
    vendorid=10de \
    primary=true

Pass the matching NVIDIA vBIOS (often required on laptops):

lxc config device set win11 nvidia rom-file=/path/to/dgpu.rom

Start the VM and install the GeForce drivers inside Windows.

If the host display disappears again, just boot with virtualization disabled in BIOS, remove the device, and re-enable virtualization.

Happy CAD’ing, and shout if you get stuck!

[ifsystems]

Hi @thingizkhan,

Thanks for the additional recommendations. Setting the VM to not autostart worked well, thanks. For attaching the GPU to the VM your string didn’t work, but I checked through the docs and worked out this, which seems to have worked although I could see no way to set primary=true:

lxc config device add win11 nvidia gpu gputype=physical pci=01:00.0

I also couldn’t find reference to rom-file in the docs, so didn’t try to passthrough the vBIOS as well yet, although I have found reference to this elsewhere on the web.

EDIT ignore below, it’s amazing what a reboot can do…! The VM starts, now I need to install the drivers and see if I need to do the vBIOS passthrough as well.

Cheers

rest of the post before edit

However, I tried to run the VM and it failed with

Failed to start device "nvidia": Cannot use device "/dev/nvidia0", 1 processes are still attached to it: 3735

Process 3735 is XOrg. After looking into this some more it seems that the Nvidia software is preventing the laptop being run solely on the iGPU, the dGPU is being retained “just in case” it is needed. The Nvidia GUI tool PRIME has the Intel-only option greyed-out so I can’t switch to iGPU-only that way. I tried adding the kernel parameter nogpumanager and using prime-select intel but that still doesn’t seem to have released the dGPU because for some reason it’s still being used by XOrg.

My initial call for help has certainly been addressed, thank you. I’ll keep trying and if I need more help on the specific dGPU issue I’ll post again (though I may also ask for help on the Nvidia forums).

Cheers