Lost 3 servers upgrading from 22.04 to 24.04

I upgraded 7 virtual servers from 22.04 to 24.04 today spread across 5 different hosts

I did extensive testing & verification post-upgrade but pre-reboot and they were all working perfectly.

I then rebooted 6 of the 7 (all except my main server)

of the 6 rebooted, the 3 non-UEFI ones rebooted successfully but the 3 UEFI ones failed to boot

the 3 unbootable systems are on Oracle Cloud, Google Cloud, and Scaleway

connecting via console or VNC, I’m not even able to get to GRUB, I’m just seeing messages like this

error: no suitable video mode found.
error: file `/boot/grub/x86_64-efi/bli.mod' not found.

according to apt-file, this file should come from package grub-efi-amd64-bin but I have never had this package installed on any of my systems, because there were no dependencies demanding it, not when I was on 20.04, not on 22.04, and not on 24.04

the package is “Priority: extra” and the only relevent package I can see that depends on it is grub-efi-amd64, which I don’t use, I use these GRUB packages instead:

grub-common/noble-updates,now 2.12-1ubuntu7.3 amd64 [installed,automatic]
grub-gfxpayload-lists/noble,now 0.7build2 amd64 [installed,automatic]
grub-pc-bin/noble-updates,now 2.12-1ubuntu7.3 amd64 [installed,automatic]
grub-pc/noble-updates,now 2.12-1ubuntu7.3 amd64 [installed,automatic]
grub2-common/noble-updates,now 2.12-1ubuntu7.3 amd64 [installed,automatic]

given that this was never an issue before, why is 24.04 demanding bli.mod in order to boot, but didn’t make any effort to ensure the relevant package was installed?

My 7th server, my main one, which I have not rebooted yet, is UEFI but unlike all the others, it’s ARM. And it does have a bli.mod, located at /boot/grub/arm64-efi/bli.mod and /usr/lib/grub/arm64-efi/bli.mod from the package grub-efi-arm64-bin, installed as a dependency of grub-efi-arm64. I’m not sure exactly why I have the EFI GRUB on my ARM server but have the non-EFI GRUB on my AMD servers. I think at one point the dependencies forced it to be that way but I can’t recall for sure.

because my main server has the bli.mod that the others didn’t, I think it’s probably safe to reboot but I’m not going to do it yet until I understand more about exactly what happened to the other servers (which I’m probably going to have to reinstall and then restore stuff from backup)

thanks for any insight anyone can provide

I was able to recover one of my servers and am working on the others

Installing grub-efi-amd64-bin and updating GRUB does indeed fix the boot failures

fixing this after-the-fact (after the system has become unbootable) requires mounting the unbootable filesystem onto a working system and setting up a chroot in order to be able to install the package and run update-grub

if this package is now required for booting, why is there nothing in place to ensure that it’s actually installed, though?