This topic addresses common issues when launching or starting instances, such as timeouts or “unknown state” errors.
These problems can occur for a few different reasons. Since Multipass relies on instances having an IP address on the default interface to establish an SSH connection, they are often (but not always) linked to IP assignment or connectivity issues.
The possible reasons that can lead the launch
or start
commands to fail are:
- When you launch a new instance, it fails to load the required image due to stale network cache.
- The VM didn’t manage to boot properly and didn’t get to the point where it requests an IP address.
- The VM requested an IP address, but didn’t obtain one.
- The VM obtained an IP address, but:
- Multipass can’t find it.
- Multipass finds an IP that doesn’t match the one that was assigned to the instance.
- SSH doesn’t function properly in the VM, or Multipass is blocked from accessing it.
- When you launch a new instance, it times out waiting for initialization to complete.
Diagnose your issue
Follow these steps to diagnose your issue and identify the most likely scenario:
-
If the
multipass launch
command fails with the message “Downloaded image hash does not match”, see: Stale network cache. -
[Windows, Hyper-V driver] Inspect the file
C:\WINDOWS\System32\drivers\etc\hosts.ics
and see if there is more than one entry with your instance name in it. If that’s the case, see: Stale internet connection sharing lease. -
[Linux/macOS, QEMU driver] Inspect the Multipass logs and look for a message mentioning
NIC_RX_FILTER_CHANGED
. This message indicates that the network interface has been initialised.- If you don’t find it, it means that the VM didn’t manage to bring up the interface; see: VM boot failure.
- If the message is present, proceed to check DHCP traffic in the next step.
-
[Linux/macOS, QEMU driver] Check DHCP traffic from your host to the instance, to find out if there are requests and replies. Adapt and run the following command right after starting/launching the instance:
sudo tcpdump -i <bridge> udp port 67 and port 68
You will need to replace
<bridge>
withmpqemubr0
on Linux and withbridge100
on macOS.Note that, on macOS,
bridge100
is a virtual network interface that only appears when at least a VM is running.- If you see
NIC_RX_FILTER_CHANGED
, you should also see DHCP requests. If you don’t, see VM boot failure and please let us know. - If you see a DHCP request, but no reply, it means that the VM is still waiting for an IP address to be assigned; see: No IP assigned.
- If you see DHCP requests and replies, continue to the next step.
- If you see
-
Look for messages regarding SSH in Multipass’s logs. The instance may have obtained an IP and/or be properly connected, but still refuse Multipass when it tries to SSH into it.
-
Look for the message in the CLI or GUI spinner. Once it reads “Waiting for initialization to complete”, Multipass willl have succeeded SSH-ing into the instance but remain waiting for cloud-init to finish.
Troubleshooting steps
VM boot failure
To find out if something is failing during boot, you’d need to attach to the VM’s console/serial and observe the output and try to find out where the VM is getting stuck. Here is how you can do that, depending on the driver:
-
[Linux/macOS, QEMU driver] Relaunch QEMU manually:
- Look for the
qemu-system-*
command line corresponding to the failing VM in Multipass logs - copy it to an editor and modify it
- remove
-serial chardev:char0 -nographic
- escape any spaces in paths (e.g.
Application Support
should becomeApplication\ Support
)
- remove
- Run the edited line in a terminal, with
sudo
. Here is an example:
/Library/Application\ Support/com.canonical.multipass/bin/qemu-system-aarch64 -machine virt,gic-version=3 -accel hvf -drive file=/Library/Application\ Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -cpu host -nic vmnet-shared,model=virtio-net-pci,mac=52:54:00:e2:30:dd -device virtio-scsi-pci,id=scsi0 -drive file=/var/root/Library/Application\ Support/multipassd/qemu/vault/instances/superior-chihuahua/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 2 -m 4096M -qmp stdio -cdrom /var/root/Library/Application\ Support/multipassd/qemu/vault/instances/superior-chihuahua/cloud-init-config.iso
This will open a QEMU window where you can see the boot output. You may need to select the correct display output (Serial or VGA) from the QEMU menu.
- Look for the
-
[macOS/Windows, VirtualBox driver] Observe the output in the VirtualBox GUI:
- Run the VirtualBox GUI as admin/root:
- [macOS]
sudo VirtualBox
- [Windows] Run with
psexec.exe
:
- [macOS]
- Start or launch the instance with
multipass start|launch
- Select and attach to the VM in the VirtualBox GUI and observe the boot output. If it eventually arrives at a login screen, it means that the instance should’ve started correctly.
- Run the VirtualBox GUI as admin/root:
-
[Windows, Hyper-V driver]
- Open the Hyper-V Manager GUI (look for it in your Start menu)
- Start or launch the instance with
multipass start|launch
- Select the VM in Hyper-V manager and click “Connect” on the Actions pane, at the right-hand side. Observe the boot output.
VM image corruption
Boot failures are often caused by VM image corruption, which can happen when the VM is killed without a proper shutdown.
Here are some options to attempt recovery:
-
If you took a snapshot before incurring this issue, you could try to restore it. However, snapshots are typically stored layers against an original image file, so they may not be enough.
-
Run
fsck
in the Serial Console:The
fsck
tool (short for “file system consistency check”) is used to scan the file system for errors and attempt repairs.To use it, access the VM’s console as described above and follow these steps:
-
Access the VM’s Console
- Use the method appropriate for your driver to access the VM’s console, as described in the VM Boot Failure section.
-
Interrupt the Boot Process
- As the VM starts booting, interrupt the boot process to access the GRUB menu:
- Press the
Esc
key repeatedly during the VM’s startup until the GRUB menu appears. - On some systems, you might need to hold down the
Shift
key instead. - The key needs to be pressed at just the right time, after UEFI loading (to avoid getting into the UEFI screen), but before Ubuntu starts booting (to trigger GRUB)
- Press the
- As the VM starts booting, interrupt the boot process to access the GRUB menu:
-
Enter Recovery Mode
- In the GRUB menu:
- Use the arrow keys to select Advanced options for Ubuntu (or your distribution’s equivalent) and press
Enter
. - Select a kernel version with
(recovery mode)
appended and pressEnter
.
- Use the arrow keys to select Advanced options for Ubuntu (or your distribution’s equivalent) and press
- In the GRUB menu:
-
Run
fsck
from Recovery Menu- Once in the recovery menu:
- Use the arrow keys to highlight
fsck
and pressEnter
. - You will be prompted to remount the filesystem in read/write mode. Select Yes.
- The system will run
fsck
and attempt to repair any detected issues.
- Use the arrow keys to highlight
- Once in the recovery menu:
-
Alternatively, Drop to a Root Shell
- If you prefer to run
fsck
manually:-
From the recovery menu, select
root
to drop to a root shell prompt. -
At the prompt, run the following commands:
mount -o remount,ro / fsck -f /
-
After
fsck
completes, remount the filesystem in read/write mode:mount -o remount,rw /
-
Type
exit
to return to the recovery menu.
-
- If you prefer to run
-
Resume Normal Boot
- In the recovery menu, select
resume
to continue with the normal boot process. - The system should now boot normally if
fsck
was able to repair the filesystem.
- In the recovery menu, select
-
-
[Linux/macOS] Alternatively, run
fsck
over a mounted image on the host. -
Run
qemu-img check -r
on the image.qemu-img
, shipped with Multipass, can also be used to check and repair disk images.- See below how to locate it.
- See here on how to locate images.
- For example:
/Library/Application\ Support/com.canonical.multipass/bin/qemu-img check -r /var/root/Library/Application\ Support/multipassd/qemu/vault/instances/<instance>/<img>
-
If none of the above works, you can still try to mount the image manually to recover data.
No IP assigned
Sometimes VMs request an IP address, but don’t obtain one. That can happen because of interference from other software, VPNs, network misconfiguration and, firewall settings.
[macOS, QEMU driver] Firewall blocks bootp
The macOS firewall is known to cause vmnet
to malfunction, because it blocks Apple’s own bootp
from giving out IPs. The effect of this problem on Multipass is tracked in this issue, which we internally call the dreaded firewall issue.
You may be able to work around it by disabling the firewall entirely, or executing
/usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
/usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
We are aware that this requires administrative privileges, which managed Macs won’t have. We unfortunately don’t have a better fix for those cases. We continue hoping that Apple will eventually fix the problem which, to the best of our knowledge, affects all products using vmnet
. Chances of that happening will probably increase if enough people report it to them.
See also: How to troubleshoot networking.
[macOS, QEMU driver] bootp
not coming up
The DHCP server should be launched automatically when there is a request, but you can also launch it manually if needed. To do so, run:
sudo launchctl start com.apple.bootpd
If that doesn’t work for you, try :
sudo launchctl load -w /System/Library/LaunchDaemons/bootps.plist
[Windows, with Hyper-V] Stale internet connection sharing lease
Another possible reason for instance timeouts is a problem with the Internet Connection Sharing
hosts file. This file sometimes gets corrupted, with jumbled or incomplete text. Other times, it contains duplicate or stale IP addresses.
Using Administrator privileges, edit the file C:\WINDOWS\System32\drivers\etc\hosts.ics
and look for any corruption or entries that have your instance name in it. If there is more than one entry, remove any of them except for the first one listed. Save the file and try again. If that does not work, stop any running instances, delete the file, and reboot.
SSH issues (DRAFT)
If SSH doesn’t function properly in the VM, or Multipass is blocked from accessing it, your instance may need to be reconfigured or repaired.
-
If the default user is not
ubuntu
, Multipass cannot connect. If you used a custom cloud-init config file, make sure that the default user isubuntu
. -
if SSH keys are missing or incorrect, you will have to add your public SSH key from
~/.ssh/id_rsa.pub
on the host to~/.ssh/authorized_keys
in the instance. To do so you may need to gain access to the instance through a method besides SSH.
To gain access to an instance without SSH you can try the following methods.
-
Mount the instance’s image file on your host and make necessary changes through the filesystem.
-
Run the instance VM directly. This will require a username and password to log in. The username is the default user,
ubuntu
, and the password is what was set in cloud-init if you used a custom cloud-init config. If you do not have a password you can modify the instance’scloud-init-config.iso
file to change it. One way to do so is as follows.- Back up your existing
cloud-init-config.iso
. - Make a new instance by running
multipass launch --cloud-init config.yaml
, the contentsconfig.yaml
are shown below. - Replace your existing
cloud-init-config.iso
with the newly generatedcloud-init-config.iso
. - Restart the VM and use the password
ubuntu
. The instance’s password will remainubuntu
unless it is changed again - Make necessary changes.
- Back up your existing
#cloud-config
password: ubuntu
chpasswd: { expire: false }
Cloud-init tarries during an instance launch (DRAFT)
-
When launching a new instance, once Multipass obtains an SSH session to the instance, it will wait for cloud-init to complete. During this phase, the CLI/GUI spinner reads “Waiting for initialization to complete”.
-
At this point, the initialization continues in the background, even if you interrupt the launch command or if it times out.
-
So if you wait for a little while longer, your instance may eventually finish setting up. When it does, it will have this file:
/var/lib/cloud/instance/boot-finished
.- Consider passing a longer timeout to the
launch
command. For example,multipass launch --timeout 1000
sets the launch timeout to 1000 seconds.
- Consider passing a longer timeout to the
- You can use
multipass shell
to get a shell in the instance when Multipass is waiting for cloud-init to finish. To diagnose problems, inspect cloud-init’s logs in/var/log/cloud-init*log
.
[Windows, VirtualBox driver] Running VirtualBox with the system account
To run the VirtualBox GUI with the system account, you will need a Windows tool called PsExec:
- Install PsExec
- Add it to your
PATH
- Run PowerShell as Administrator
- Execute
psexec.exe -s -i "C:\Program Files\Oracle\VirtualBox\VirtualBox.exe"
(adapt the path to the VirtualBox executable as needed)
When successful, you should see Multipass’s instances in VirtualBox
Stale network cache
This can be caused by a known Qt bug (see issue #1714 on our GitHub).
A workaround to resolve this issue is to run the command multipass find --force-update
, which forces downloading the image information from the network. As a result, if the download is successful, the network-cache
will be overwritten.
Alternatively, try deleting the network-cache
folder and restart the Multipass service:
- (on Linux)
sudo snap stop multipass sudo rm -rf /var/snap/multipass/common/cache/multipassd/network-cache/ sudo snap start multipass
- (on macOS)
sudo launchctl unload /Library/LaunchDaemons/com.canonical.multipassd.plist sudo rm -rf /System/Volumes/Data/private/var/root/Library/Caches/multipassd/network-cache sudo launchctl load /Library/LaunchDaemons/com.canonical.multipassd.plist
- (on Windows)
RemoveC:\ProgramData\Multipass\cache\network-cache
and restart the Multipass service.
Reading data from a QCOW2 image
The images that Multipass uses with the QEMU driver follow a standard format — QCOW2 — which other tools can read.
One example is qemu-nbd
, which allows mounting the image. This tool is not shipped with Multipass, so you would need to install it manually.
Once you have it, you can search the web for recipes to “mount a QCOW2 image”. For example, here is a a recipe.
Locating Multipass Binaries (DRAFT)
You may need to locate where Multipass is installed. There are several ways to do so, depending on your platform:
-
(on Linux)
- Run the command
which multipass
orwhereis multipass
. - By default, Multipass is installed in the
/snap/bin
folder.
- Run the command
-
(on Windows)
- Run the command
where.exe multipass
. - Right-click a shortcut to Multipass in your files or Start menu and select “Open file location”.
- By default, Multipass is installed in the
C:\Program Files\Multipass\bin
folder.
- Run the command
-
(on MacOS)
- Run the command
readlink -f $(which multipass)
- By default, Multipass is installed in the
/Library/Application\ Support/com.canonical.multipass/bin/
folder.
- Run the command
Locating Multipass Images (DRAFT)
You may need to locate where Multipass is storing instances. The location changes depending on your platform:
-
(Linux)
/root/.local/share/multipassd/vault/instances/<instance/<img>
-
(Windows)
C:\ProgramData\Multipass\data\vault\instances\<instance>\<img>
-
(MacOS)
/var/root/Library/Application\ Support/multipassd/qemu/vault/instances/<instance>/<img>