I’ve been asked to try and summarize everything here so we can conclude this thread. The following applies specifically to the issue @irihapeti ran into. There are some other boot issues currently being dealt with, but I suggest we don’t confuse this thread any further (than I already have!). So, if you’re running into something other than the issue below, please feel free to start another thread (or ping me from another open one).
Root cause
The issue here was that 26.04 switched initramfs generation from the initramfs-tools package to dracut. The latter unfortunately has rather silly defaults for compression (zstd level 15 with multi-threading) which work perfectly fine if you’re on a fat multi-core machine with 8+GB of RAM, but is a terrible choice on anything with a minimal set of RAM like a Pi Zero 2W (512MB) or 3B (1GB), depending on what else is loaded (for reference, zstd at level 15 will happily eat 500MB or more in the course of its duties).
It’s not the first time we’ve seen this; I had to workaround the exact same settings in initramfs-tools a few years ago. However, it turns out there’s a compounding error in dracut which we weren’t aware of: when the compressor fails and runs out of RAM, dracut merrily exits with “0” (indicating “success”). This leads the flash-kernel hook to assume that the initramfs was generated successfully (it wasn’t) so it writes it to the boot partition in the “new” slot for piboot-try to test.
At the next boot, piboot-try does its thing, the initramfs duly fails and piboot-try falls back to the original assets. On the plus side: yay, piboot-try works! On the minus side: I didn’t really want it to get a work-out this soon after resolute’s release…
Are you affected?
You can tell if you’re affected by checking the size of the initramfs in your /boot folder:
$ ls -lh /boot/initrd.img-*
-rw------- 1 root root 67M Jul 3 13:46 /boot/initrd.img-7.0.0-1011-raspi
-rw------- 1 root root 67M Jul 3 06:30 /boot/initrd.img-7.0.0-1014-raspi
The size of all such files should be in the region of ~60MB. If they’re a much smaller (e.g. a few hundred KB) instead, then you’re hitting this issue.
Workarounds
There is an SRU to fix this working its way through the system. If you don’t want to tinker with anything, just wait for the SRU to land and this will be resolved. In the meantime piboot-try should keep you booting successfully. To re-iterate a note from further up: piboot-try is not testing the “bad” assets on every boot, but it will (annoyingly) warn you on every login that the “new” assets are “bad”.
If you want to workaround it right now, you can do the following:
$ cat << EOF | sudo tee /etc/dracut.conf.d/zz-compress.conf
compress_level_zstd="1"
compress_level_xz="1"
EOF
(I included xz as that is another compressor that eats a very large amount of RAM with the defaults that dracut uses; this guards against users configuring a different compressor and inadvertantly running into this same issue)
The SRU will (eventually) create those same settings (but in a different file, so the upgrade won’t collide). Once the settings are in place, regenerate any truncated initramfs images. For example, if you saw the 1014 revision was truncated:
$ sudo dracut --force /boot/initrd.img-7.0.0-1014-raspi 7.0.0-1014-raspi
Repeat for each truncated version (replacing both version strings in the command as necessary). Once finished, copy the boot assets to the boot partition with flash-kernel:
$ sudo flash-kernel
At the next boot (assuming all goes well) you should find yourself on the newest version.
