How do you deal with Linux kernels on 26.04 (TPM-backed FDE)?

Support and Help
,
1

Ubuntu Version: 26.04

Desktop Environment (if applicable): GNOME

Problem Description:

On Ubuntu 24.04 and prior:

  1. On booting, Grub would allow you to choose which kernel to boot from (default to the latest one).
  2. You could manually install or remove kernels, including non-standard ones (I used the latter with zabbly when helping a dev to debug a regression and as a temporary workaround for my problem).

I’ve installed Ubuntu 26.04 using TPM-backed full-disk encryption (FDE). On this:

  1. Grub doesn’t offer which kernel to boot from. It just boots from the latest version.

    New Grub menu
    New Grub menu698×181 4.41 KB

  2. I can’t even see which versions are installed. I used to use apt list --installed 'linux-image*', but 26.04 returns nothing.

  3. I have tried manually installing a different kernel using apt install, but on rebooting the machine, the installed kernel mysteriously disappears.

So, my questions are:

  1. How do you select a previous kernel when booting, eg if the latest update fails to boot properly?
  2. How do you see which kernels are installed?
  3. How do you install a different kernel, specifically:
    • An older kernel?
    • A non-standard kernel, eg zabbly?

Relevant System Information:

I’m still learning and testing Ubuntu 26.04, so this is in VirtualBox (host: Ubuntu 24.04), in case it makes a difference.

2 Likes
2

Found this but I only use WSL now (for work)
https://www.progressiverobot.com/2026/05/29/ubuntu-26-04-lts-older-kernels-missing-from-grub-menu-fix-prevention/
I recommend testing in a VM first, or dev system

3

Thanks. Unfortunately, the fixes listed there don’t work, because:

  • Changing Grub causes the system to fail completely.
  • As already explained, when manually installing another kernel, that kernel simply disappears when rebooting.

EDIT: WSL is a very different beast from a native installation of Ubuntu 26.04.

4

Oh I know. It’s just what I use for work. I don’t run any Linux on tin now.

5

That’s just the first GRUB menu. Select run mode, press Enter and then immediately press Esc again to get to the second menu and select “Fallback on failed update”:

image
image1920×1080 51 KB

I hope the comment is still readable enough.

I have just tested this on my own tinker toy VM, which pretty much mirrors your setup, I believe. It still booted the 7.0.0-22 kernel, though. I don’t quite know why, because:

$ snap list --all pc-kernel 
Name       Version      Rev   Tracking      Publisher   Notes
pc-kernel  7.0.0-14.14  3390  26.04/stable  canonical✓  kernel,disabled
pc-kernel  7.0.0-22.22  3492  26.04/stable  canonical✓  kernel

Perhaps because it’s disabled? But I’d have thought that’s the normal status of previous (fallback) versions of Snaps. Maybe GRUB/<boot magic> doesn’t believe me that the update failed, because there is no actual error to recover from?

2 Likes
6

It is clever and knows you want to cheat :wink:

If there is actually an upgrade to verify snapd sets some cmdline kernel variables that grub uses to do the automatic upgrade verification (and the fallback when this fails), if these variables are not found it will do nothing…

1 Like
7

Oh, so it really doesn’t believe me. But what if one wants to boot an older kernel for regression tests, for instance? I guess, one can run snap refresh pc-kernel --revision=<old-kernel>, but that requires a usable system. So, what if, for arguments sake, the new kernel had a regression which made it panic sometime after a successful boot, which, I think gives the “all clear” to said <boot magic>?

BTW, I wanted to see what’s behind Recover using enhanced-secureboot-desktop, got an Ubuntu Core splash screen with a turning spinner, but it seemed to go nowhere. The host showed 25% CPU load on a 4c/8t AMD APU the whole time; after 10 minutes I gave up and sent CtrlAltDel, which actually worked; usually it doesn’t at that stage. Was I just too impatient or should I open a separate Support and Help request?

1 Like
8

Thank you! Goodness, I’d never have guessed that.

So, that’s what we need to use instead of apt list --installed.

Where are all of these documented? How did you know about these?

I’ve already raised a bug report, discussed in this post.

9

I vaguely remembered reading about them in the Snap docs or from “browsing” snap help; can’t say for certain. Plus, Tab (menu) completion; the day my Tab dies, my shell days are over. :grin:

Oh, right, I was actually present but forgot. :innocent: Thanks for the reminder.

1 Like
10

Right, that would work (if you are in a state to call the command)

I sadly do not know a lot about the changes made to the original Ubuntu Core design, there you would be able to switch to a former state of the system in recovery mode in case something would prevent you from accessing the system (due to a late oops or whatnot)

https://documentation.ubuntu.com/core/how-to-guides/manage-ubuntu-core/use-a-recovery-mode/#use-a-recovery-mode

Not sure if that feature was carried over to desktop (it does heavily depend on the system to be all-snap based, not sure it can even work with a traditional rootfs)

2 Likes
11

With a bit of help from AI (it’s not totally useless :slightly_smiling_face:), I found that you can use revert instead of refresh in Oliver’s command:

sudo snap revert pc-kernel --revision=[old kernel]

What’s the difference between the two? For this, I’ll use three terms: current (the current kernel version); previous (the previous kernel version); and new (the next kernel version, yet to be released).

  • revert: Goes backwards in time to previous.
    • It attempts to revert the snap’s configuration and data back to the state it was when previous was active.
    • It signals to snap that current failed or is undesirable, and so it won’t automatically reinstate current on the next refresh cycle.
  • refresh: Tells snap to treat this as a new deployment.
    • Snap changes to previous as if it were a new installation, so it doesn’t roll back the configuration and data; it rolls them forward instead.
    • When new is released, snap will update to new, bypassing current.

Apparently, the difference is understandable as an intent:

  • revert: Roll back due to a problem.
  • refresh: Force installation of a specific version.

Apologies for the lengthy wording. This is mainly for me, trying to understand all of this!

1 Like
12

FYI, --revision shouldn’t be necessary for reverting to the previous revision, because that is the default action:

$ snap help revert
Usage:
  snap revert [revert-OPTIONS] <snap>

The revert command reverts the given snap to its state before
the latest refresh. This will reactivate the previous snap revision,
and will use the original data that was associated with that revision,
discarding any data changes that were done by the latest revision. As
an exception, data which the snap explicitly chooses to share across
revisions is not touched by the revert process.

[revert command options]
      --no-wait     Do not wait for the operation to finish but just print the change id.
      --devmode     Put snap in development mode and disable security confinement
      --jailmode    Put snap in enforced confinement mode
      --classic     Put snap in classic mode and disable security confinement
      --revision=   Revert to the given revision

All options are optional — hence the surrounding [] in the usage section — and the help text implies in so many words that --revision=<prev> is the default, so one doesn’t even need to know about revisions, unless an older one is desired. But, since snapd, by default, only keeps one extra revision for fallbacks, most users won’t have any choice anyway.

Good find, though; looks like what one actually wants in such a situation.

1 Like
13

Note that revert will only be able to revert to revisions on disk so the–revision arg will only make sense if you have more than two kernels (running and last one), else revert simply goes back to the last one on disk …

FWIW, I’d also recommend to take a look at snap help which will point you to things like snap help revert which has a bunch more details about how the revert command works …

EDIT: bah, peter beats me with the help stuff :stuck_out_tongue:

2 Likes
14

Thank you. I’ll test and document this for myself.

Ha ha! You’ve both been most helpful!

1 Like
15

The grub on the ESP (loaded by firmware) by default simply chainloads the grub from the ubuntu-boot partition. Both grub binaries are identical, so you can intercept the second grub as was already mentioned, edit the menu:

menuentry "Run Ubuntu Core" {
    # use $prefix because the symlink manipulation at runtime for kernel snap
    # upgrades, etc. should only need the /boot/grub/ directory, not the
    # /EFI/ubuntu/ directory
    chainloader $prefix/$kernel snapd_recovery_mode=run $cmdline_args
}

and change it to explicitly boot a previous kernel:

menuentry "Run Ubuntu Core" {
    # use $prefix because the symlink manipulation at runtime for kernel snap
    # upgrades, etc. should only need the /boot/grub/ directory, not the
    # /EFI/ubuntu/ directory
    chainloader $prefix/pc-kernel_3419.snap/kernel.efi snapd_recovery_mode=run $cmdline_args
}

and boot with Ctrl-X or F10 .

Doing this will invalidate kernel command line measurements so you will need to use the recovery key.

Or you can simply invoke chainloader and then boot (this is the standard grub, not Ubuntu specific. What is specific are various variables read from the environment file(s) on ESP and/or ubuntu-boot. In particiuar, you must supply the snapd_recovery_mode=run, otherwise the system will refuse to boot).

You can also check the available snaps in grub at boot time by entering the command line (c in the menu) and using

ls -l /EFI/ubuntu

(sorry, the site refuses to upload images).

If you stop in the “first” grub you can find out the ubuntu-boot partition using

ls -l

After boot the ubuntu-boot is mounted as /run/mnt/ubuntu-boot, so

bor@resolute:~$ ll /run/mnt/ubuntu-boot/EFI/ubuntu/
total 24
drwxr-xr-x 4 root root 4096 Jun 20 09:03 ./
drwxr-xr-x 4 root root 4096 Apr 27 21:21 ../
-rw-r--r-- 1 root root 1952 Apr 27 21:21 grub.cfg
-rw-r--r-- 1 root root 1024 Jun 27 14:55 grubenv
lrwxrwxrwx 1 root root   30 Jun 20 08:57 kernel.efi@ -> pc-kernel_3492.snap/kernel.efi
drwxr-xr-x 2 root root 4096 Apr 28 12:18 pc-kernel_3419.snap/
drwxr-xr-x 2 root root 4096 Jun  5 11:32 pc-kernel_3492.snap/
bor@resolute:~$

I do not see any way to install an older kernel. E.g. currently only the revision 3492 in the channel 26.04/stable is available (according to the snap info pc-kernel). I do have the previous snap 3419 and I most likely can revert to it with snap revert pc-kernel, but it is not installing an older revision. So, for all practical purposes I am limited to the current and previous versions.

I do not think you can install non-standard kernel without breaking TPM based FDE.

3 Likes
16

Thank you for the details! These are helpful. I’ve documented this for my future self.

Yes, this is an ongoing problem. You have to try multiple times before the site eventually accepts the image. Sometimes, you have to wait a half-minute, but that’s no guarantee.