Abstract
Supply the kernel modules and firmware files pre-compressed and do not re-compress them in the
initramfs to make booting faster and the initramfs smaller and faster to generate.
Rationale
An initramfs contains a temporary root file system to be used during boot. It contains everything that is needed to find and mount the root file system of the host (e. g. NVMe in a laptop). The initramfs needs to include all potentially required kernel modules and firmware files. This also includes USB and graphics drivers (in case the user has to enter a password to decrypt the root partition).
The initramfs should be small and fast to boot. Creating the initramfs should be not too slow and must not need more memory than the system has.
The initramfs can be compressed (e.g. with xz, lz4 or zstd). Picking the default compression type and level depends on the hardware it runs on. The slowest hardware with the lowest amount of memory dictates what can be used. Starting with Ubuntu 22.04, the default compression is zstd -1, because zstd -19 was too slow (it took more than an hour on a Nezha development board). That leads to bigger initramfs sizes that triggered other bugs. Big initramfs sizes can also easily fill small-sized /boot partitions (grooming the 850 initramfs-tools bugs revealed several such reports).
Specification
Ship the kernel modules and firmware files zstd-compressed (LP: #2028568 and #1942260). The Debian build machines are powerful enough to compress the files with zstd level 19. Change initramfs-tools to not re-compress compressed kernel modules and firmware files (LP: #2028567). This is done by putting the compressed files into a separate cpio archive that is not compressed. The remaining files are put into a separate cpio archive that is compressed (default zstd level 2) and concatenated.
When generating the initramfs, most time is spent in the Bash auto_add_modules function. Replacing the Bash code with dracut-install reduces the execution time significantly (LP: #2031185).
A further improvement is to avoid storing the intermediate main cpio archive on disk.
Distribution upgrade
To ensure reliable dist-upgrades in case of partial upgrades or upgrade failures, backport support to tolerate zstd-compressed firmware files to the Ubuntu 22.04 “jammy” kernels. initramfs-tools in jammy supports zstd-compressed kernel modules but not zstd-compressed firmware files. Therefore, the zstd-compressed firmwares declare breaking the old initramfs-tools.
Benchmark results
To evaluate the impact of the change, there are three different aspects that can be measured: the size of the generated initramfs, the time it takes to generate the initramfs, and how fast the system boots. To put those values into perspective, measure the system before the change (Ubuntu 22.04 LTS “jammy” and Ubuntu 23.04 “lunar”) and afterwards (Ubuntu 23.10 “mantic”).
initramfs size
| test | distro | kernel | size / bytes | size / MiB | uncompressed size / bytes | uncompressed size / MiB |
|---|---|---|---|---|---|---|
| amd64 minimal | jammy | 5.15.0-83.92 | 62762675 | 59.9 | 198801710 | 189.6 |
| amd64 minimal | lunar | 6.2.0-32.32 | 66729453 | 63.6 | 223378396 | 213.0 |
| amd64 minimal | mantic uncomp¹ | 6.5.0-5.5 | 62302654 | 59.4 | 205924538 | 196.4 |
| amd64 minimal | mantic | 6.5.0-5.5 | 55326431 | 52.8 | 64224628 | 61.2 |
| amd64 full | jammy | 5.15.0-83.92 | 112782161 | 107.6 | 345507840 | 329.5 |
| amd64 full | lunar | 6.2.0-32.32 | 124663880 | 118.9 | 406902468 | 388.1 |
| amd64 full | mantic uncomp¹ | 6.5.0-5.5 | 115110927 | 109.8 | 375100662 | 357.7 |
| amd64 full | mantic | 6.5.0-5.5 | 103623579 | 98.8 | 126072148 | 120.2 |
| amd64 nvidia | jammy | 5.15.0-83.92 | 202413553 | 193.0 | 470515992 | 448.7 |
| amd64 nvidia | lunar | 6.2.0-32.32 | 213388241 | 203.5 | 530733877 | 506.1 |
| amd64 nvidia | mantic uncomp¹ | 6.5.0-5.5 | 203887505 | 194.4 | 499346193 | 476.2 |
| amd64 nvidia | mantic | 6.5.0-5.5 | 189517267 | 180.7 | 230569794 | 219.9 |
| Pi Zero 2 | jammy | 5.15.0-1037.40 | 36769927 | 35.1 | 64928449 | 61.9 |
| Pi Zero 2 | lunar | 6.2.0-1013.15 | 39577844 | 37.7 | 69974064 | 66.7 |
| Pi Zero 2 | mantic | 6.5.0-1002.2 | 31046627 | 29.6 | 40592364 | 38.7 |
| VisionFive 2 | lunar | 6.2.0-33.33.1 | 112822390 | 107.6 | 376176126 | 358.7 |
| VisionFive 2 | mantic uncomp kernel only | 6.2.0-33.33.1 | 109754816 | 104.7 | 275491464 | 262.7 |
| VisionFive 2 | mantic | 6.2.0-33.33.1² | 101693476 | 97.0 | 125650851 | 119.8 |
Legend
- minimal: chroot with the following packages: linux-image-generic initramfs-tools zstd
- full: minimal plus the following packages: busybox-initramfs cryptsetup-initramfs isc-dhcp-client kbd lvm2 mdadm ntfs-3g plymouth plymouth-theme-spinner
- nvidia: full plus the following packages: linux-headers-generic nvidia-driver-535
- Pi Zero 2: Raspberry Pi Zero 2 ARM board
- VisionFive 2: VisionFive 2 RISC-V board
- uncomp¹: Revert the compression change by decompressing the kernel modules and use a linux-firmware package that was built without compressing the firmware with zstd
- 6.2.0-33.33.1²: The kernel modules were manually compressed with
zstd -19 - nvidia-driver-535 is not yet easily installable on mantic (see bug #2037305) and needs patching to be included in the initramfs (bug #2037400
How to reproduce
- chroot setup: Use a minimal chroot and install the packages with the
--no-install-recommendsparameter. - Compress kernel modules (for mantic uncomp¹ only):
find /lib/modules/$(uname -r) -type f -name '*.ko' -print0 | xargs -0 -P 4 sudo zstd -19 -z --rm && sudo depmod "$(uname -r)" - Check the size:
stat -L -c '%s' /boot/initrd.img - Extract the initrasmfs content and check its size:
rm -rf unpacked && unmkinitramfs /boot/initrd.img unpacked && du -sb unpacked
initramfs build time and memory usage
| test / machine | distro | kernel | build time / s | memory usage / MiB |
|---|---|---|---|---|
| amd64 minimal 5700G tmpfs | jammy | 5.15.0-83.92 | 4.37 ± 0.01 | 39.02 ± 0.43 |
| amd64 minimal 5700G tmpfs | lunar | 6.2.0-32.32 | 4.43 ± 0.01 | 39.98 ± 0.88 |
| amd64 minimal 5700G tmpfs | mantic | 6.5.0-5.5 | 1.84 ± 0.01 | 25.75 ± 0.13 |
| amd64 full 5700G tmpfs | jammy | 5.15.0-83.92 | 6.90 ± 0.03 | 39.38 ± 0.33 |
| amd64 full 5700G tmpfs | lunar | 6.2.0-32.32 | 7.31 ± 0.02 | 40.86 ± 0.88 |
| amd64 full 5700G tmpfs | mantic | 6.5.0-5.5 | 3.89 ± 0.02 | 55.99 ± 0.08 |
| amd64 nvidia 5700G tmpfs | jammy | 5.15.0-83.92 | 7.09 ± 0.03 | 45.92 ± 0.76 |
| amd64 nvidia 5700G tmpfs | lunar | 6.2.0-32.32 | 7.17 ± 0.02 | 47.18 ± 0.92 |
| amd64 nvidia 5700G tmpfs | mantic | 6.5.0-5.5 | 4.10 ± 0.01 | 160.57 ± 0.11 |
| Pi Zero 2 | jammy | 5.15.0-1037.40 | 102.77 ± 0.97 | TBD |
| Pi Zero 2 | lunar | 6.2.0-1013.15 | 127.91 ± 1.36 | 15.61 ± 0.09 |
| Pi Zero 2 | mantic | 6.5.0-1002.2 | 99.36 ± 0.92 | 20.04 ± 0.05 |
| VisionFive 2 | lunar | 6.2.0-33.33.1 | 134.18 ± 0.96 | 36.10 ± 0.07 |
| VisionFive 2 | mantic | 6.2.0-33.33.1¹ | 85.07 ± 0.21 | 40.53 ± 0.06 |
Legend
- 5700G tmpfs: AMD Ryzen 7 5700G with schroot and overlay on tmpfs
- Pi Zero 2: Raspberry Pi Zero 2 ARM board with 64 GB Samsung EVO Plus 2020 microSDXC (running armhf)
- VisionFive 2: VisionFive 2 RISC-V board with 32 GB Sandisk Ultra microSDHC
- 6.2.0-33.33.1¹: The kernel modules were manually compressed with
zstd -19
How to reproduce
- Run
update-initramfs -uten or twenty times (and build the average and standard deviation):for i in $(seq 20); do time update-initramfs -u > /dev/null 2>&1; done - Measure the memory consumption in Kbytes with time:
/usr/bin/time -f "%M" - Build the average and standard deviation:
for i in $(seq 20); do /usr/bin/time -f "%M" update-initramfs -u > /dev/null; done
Boot time
Systemd-analyze can be used to measure the boot time. The reported kernel time also contains the time spent in the initramfs because systemd is not used in the initramfs.
| test / machine | distro | kernel | kernel / s | userspace / s | total / s |
|---|---|---|---|---|---|
| amd64 5700G VM | lunar | 6.2.0.33.33 | 1.65 ± 0.05 | 22.20 ± 0.37² | 23.85 ± 0.42² |
| amd64 5700G VM | mantic uncomp¹ | 6.5.0-5.5 | 1.36 ± 0.03 | 23.28 ± 0.48² | 24.64 ± 0.50² |
| amd64 5700G VM | mantic | 6.5.0-5.5 | 1.17 ± 0.04 | 22.71 ± 0.55² | 23.88 ± 0.57² |
| Pi Zero 2 | lunar | 6.2.0-1013.15 | 6.84 ± 0.02 | 28.79 ± 0.47 | 35.64 ± 0.47 |
| Pi Zero 2 | mantic | 6.5.0-1002.2 | 7.34 ± 0.02 | 27.07 ± 0.16 | 34.40 ± 0.16 |
| VisionFive 2 | lunar | 6.2.0-33.33.1 | 25.42 ± 0.21 | 30.74 ± 0.56 | 56.16 ± 0.59 |
| VisionFive 2 | mantic | 6.2.0-33.33.1³ | 16.61 ± 0.07 | 28.98 ± 0.34 | 45.58 ± 0.38 |
Legend
- uncomp¹: Revert the compression change by decompressing the kernel modules and use a linux-firmware package that was built without compressing the firmware with zstd.
- ² Take those slow userspace times with a grain of salt (see bug #2036592)
- 5700G VM: Virtual machine with 8 vCPUs (host AMD Ryzen 7 5700G) and 4 GB memory
- 6.2.0-33.33.1³: The kernel modules were manually compressed with
zstd -19
How to reproduce
- Reboot the machine at least ten times and take the boot time by running
systemd-analyze. Remove the entries with too long boot times.
Further Information
The improvement was discussed on the ubuntu-devel mailing list. See
Reducing initramfs size and speed up the generation and
Update on reducing initramfs size and speed up.