As part of the 24.10 cycle, the Foundations team worked on a proof of concept
(PoC) demonstrating the installation of Ubuntu Server on a disk-less machine
using NVMe over TCP [1] (NVMe/TCP).
We are hosting the documentation for this PoC at
This work involved growing experimental support in the Subiquity installer (and related components) for:
- listing and formatting remote NVMe drives
- determining if the firmware supports NVMe/TCP booting
- installing Ubuntu to a remote NVMe drive (provided that the firmware supports booting from it)
The setup differs from a typical Ubuntu installation in some aspects, namely:
- dracut (currently a universe package) manages the generation of the initramfs instead of initramfs-tools
- when booting the target system, the firmware takes care of loading the kernel and the initramfs from over the network
- on the target system, a network interface called “nbft0” will be present.
Keeping this network interface up and refraining from adjusting its IP
settings (see the known issue section below) is essential while the system is running.
How to try it at home
Unless you have access to some specific hardware that supports booting using NVMe/TCP, virtual machines (VMs) are the way to go.
The documentation at GitHub - canonical/nvme-tcp-poc is a step by step guide to help you run the PoC using two VMs:
- a first VM with a NVMe drive that will be exposed over the network
- a second one, with no disk, that will boot from the NVMe drive over the network
Feedback is appreciated!
Constructive feedback, positive and negative is very appreciated!
If you run into an error caused by one of the scripts from the PoC itself, you can open a new issue on the GitHub repository for the PoC.
On the other hand, if you are encountering difficulties using the newly installed system, a bug report on Launchpad Subiquity is preferred.
Known issue
If you are not following exactly the steps from the PoC documentation, you will
probably run into the following issue when booting the initiator VM:
nvme0c0n1: I/O Cmd(0x2) @ LBA 5726464, 32 blocks, I/O Error (sct 0x3 / sc 0x71)
nvme0c0n1: I/O Cmd(0x2) @ LBA 4181768, 80 blocks, I/O Error (sct 0x3 / sc 0x71)
I/O error, dev nvme0c0n1, sector 4181768 op 0x0:(READ) flags 0x2080700 phys_seg 4 prio class 0
I/O error, dev nvme0c0n1, sector 5726464 op 0x0:(READ) flags 0x2080700 phys_seg 4 prio class 0
[...]
nvme nvme0: failed nvme_keep_alive_end_io error=10
nvme nvme0: failed to bind queue 0 socket -99
This indicates that the network went down during boot; when the network daemon took over the management of network interfaces. This is tracked in LP: #2084012 and a workaround is in place in the PoC to avoid it.
NOTE: An exhaustive list of known issues (including those that have already
been addressed) is available at GitHub - canonical/nvme-tcp-poc