Ubuntu Server Gazette - Issue 10 - The unbelievable number of supported kernels in Ubuntu

This issue of the Gazette started when a coworker told me he only realized Canonical does our own backports after he joined the company… From there, more questions came up and were answered, so I wanted to explain everything you might have wanted to know about the Linux kernel in Ubuntu. We’ll start with how many supported kernels there are and why, and then add a few further related topics - like how to know which kernel version contains a specific patch. Thanks for reading!

How many?

The current number of supported Ubuntu kernel sources (not even including the ones in the ongoing development series) is … drumroll …

So there are 2 then, the normal and hardware enablement kernel - right?
No!

Ah ok, so maybe like 25 with hardware specific kernels?
No!

How about 42, because it is the answer anyway?
No comment, but bonus points.

Wow really, so maybe 75 accounting for the different releases?
No!

There are a whopping 163 supported variants of them right now!
163 !!!

The kernel series information file lists all kernels and their repo locations, even if the repositories of a few might be private. Some kernels there have codenames, others are well recognizable. Mostly, there are a lot of them :slight_smile:

163 is an impressive number, but why are there so many?

Why so many?

A look at how Ubuntu supports Linux kernels for 15 years

a.k.a scaling by time due to long-term support

The Linux kernel is a central component of Ubuntu, so it needs bug and security fixes for a long time.
Recently, 15 years of coverage with Ubuntu Legacy was announced, implying that all still-active LTS releases of the last 15 years, as well as recent interim releases, will receive this coverage. At the time of writing, that would be 14.04, 16.04, 18.04, 20.04, 22.04, 24.04 - and a shorter time for active interim releases 25.04, 25.10 - and the current development release that will become 26.04. This keeps growing over time.

How does Ubuntu provide kernel updates for longer than a kernel.org-kernel gives you?
Simple: We do the kernel patch backporting ourselves.
That’s why stable Ubuntu kernel versions are named x.y.0-ABI instead of kernel.org’s stable names x.y.z.

Linux kernel development generally happens on kernel.org, and from there, a kernel base version is picked when an Ubuntu release is drafted:
We pick an upstream kernel before its release (and follow upstream updates until it is finalized) to ensure you get the latest upstream kernel version right when a new Ubuntu version is released.

From then on, the Ubuntu team backports relevant patches and stable fixes from kernel.org, or develops Ubuntu-specific ones.
We do this because we do not generally base our kernels on upstream’s stable releases, since even a release denoted “stable” with long term support from kernel.org won’t receive updates for 15 years.
This means Ubuntu maintains its own stable kernels with cherry-picked patches.

Each Ubuntu kernel is released as creating x.y.0-ABI.
In this context, ABI in a Ubuntu kernel version represents the tracking of exported functions via the kernel team’s build system. An ABI version over a thousand usually indicates a derivative kernel (more below). The kernel team starts them at 1000 to avoid ABI version clashes with the -generic/-hwe kernels, which start at 1 and usually grow to a couple of hundreds over their lifetime.

But not all patches originate from upstream.
Additionally, Ubuntu might include patches that are not included in the kernel.org trees, either because the patch hasn’t been pulled in, or because it is obtained from other non-upstream sources and is unlikely to be upstreamed – we call that the “Sauce”.
More features and drivers are sometimes provided by out of tree driver modules to ensure, for example, that ZFS works fine.

Okay, this explains only 10-ish of our active 163 kernel releases, why are there more?

A look at how Ubuntu supports many different platforms

a.k.a scaling by derivatives

There is a balance to strike. Ideally everyone would have their very own kernel optimized for their platform, their use case and their needs. But it is hard to support the many kernel variants we have – let us not make them millions of variants yet. On the other hand, saying “you all shall work with the very same kernel” is wasting too much potential.

“Works well everywhere”

The most common standard kernel is called “linux-generic” and that is a great balancing act for the needs of everyone. It comes with defaults that work well in almost all cases, but these defaults are not optimized for any scenario that would mostly come at the cost of other use cases.

“Optimized for platforms”

Starting from the generic kernels, there are optimized or opinionated kernels, again by modifications and configurations, to be better where the substrate they run on or the use-case they will be serving is known.

Some are optimized to provide feature support and optimization for clouds, while others are optimized for a particular hardware variant like different RISC-V models, or developer boards. Or the common case of the (usually) pre-installed OEM kernels to enable/optimize for the hardware of a particular device, if it is not yet supported by the generic kernel.

“Optimized for special requirements”

There are further use cases which need to modify or configure the kernel in a way that is detrimental to other users. Therefore, this isn’t just done on the kernel used by everyone. Instead those become derivative kernels, mostly based on the LTS kernels but with individual changes on top and their own set of verifications.

Rather well known examples of that would be the FIPS kernels for those under regulations that require this kind of operation. Or, real-time kernels for use cases that need more determinism.

“Optimized for recent hardware and features”

The following scenario is quite a common request. “I’d like to have everything as stable as possible, except this one detail I need”.
Combine enough people saying that and you are faced with the mutually exclusive demand to keep everything stable and to change everything.

One particularly frequent aspect of that is the request for the support of new hardware. But, backporting all the changes would be too disruptive to the existing users that do not need that. This is where the hardware enablement kernel comes into play to extend the Ubuntu kernel lifecycle. That again doubles the number of supported kernels. The so-called HWE-kernel is a backport of the kernel of newer Ubuntu releases to the Ubuntu LTS releases. This allows users to get the more recent hardware support and features, while keeping the rest of the OS stable. Thanks to the Linux kernel for being a great abstraction layer to make that possible.

163 or are there even more?

Strictly speaking, all that was mentioned so far is only about the different kernel source packages. A single source can be built for a combination of architectures+flavors. Flavors are built from the same source but with different kernel configs. For instance, the questing/linux generic kernel builds for amd64/arm64/armhf/ppc64el/s390x. Most of these arches have only one flavor, but for example arm64 builds 2 flavors (with 4KB and 64KB page size).
So the actual number of “linux-image-<foo>” packages, for all architectures and flavors, is much higher than that.
But fully counting that would be bragging right … right?
Although I’m not in the kernel team - so it would be praise I guess?
Hey Kernel Team, in case you didn’t know: You are awesome!

All of that allows you to get the best out of your Ubuntu system and then be able to rely on it for a long time!

How to navigate all these kernels

Which kernel contains a certain patch?

a.k.a to not see the fix-forest for all the kernel-trees

Out of all of the above you might have realized that it is quite a challenge to support all users and all variations they might be using, yet this is just what we do.
We often get questions like :

  • But I only know a particular CVE is fixed upstream in >=6.14.3 and >=6.17, how do I map this to x.y.0-thousands?
  • I’m looking for a particular fix that was added >=6.15, how could I track if it was or will be applied?
  • A commit was accepted upstream, in which Ubuntu kernel releases is it included ?

Let us start with the easy mode. If upstream says “6.15 has fixed XYZ”, then any >=6.15.0-ABI will have that fix.

For fixes and feature backports you can proactively file bugs or check those filed by others. To do so, you can either look at an individual bug, which will list the release it is targeted at and update the state (per target release) when it is fixed. Furthermore, on the bug you will see regular updates along the process to prepare, test and release these changes via stable release updates. These updates happen regularly and you can see the dates of the ongoing and upcoming stable and security updates at https://kernel.ubuntu.com/.

Or you can look the other way around, at the changelogs of the linux kernel you are interested in. Changelogs will list the applied changes and refer to bugs for more details.

Luckily CVEs have standardized identifiers, which we map and provide at https://ubuntu.com/security/cves, or on your system via the Pro Client’s ability to list and handle vulnerabilities.

And finally, you can go all in on a per-commit level: the option of tracing a change not only via bugs is made possible by the nature of open source - the Ubuntu kernel repositories are public. The SHA1 of backported upstream fixes is referenced in the commit message - to do so the commit usually will get an entry like “([cherry picked|backported] from commit <sha1>)”. With that in mind it is easy to find the change you are looking for even after potential adaptation for the backport.

How to find a particular change

a.k.a tracing the journey of an upstream commit to ubuntu kernels

Imagine you’d have a suspend issue and found it is addressed in bd820906ea9d “platform/x86/intel/pmc: Add Arrow Lake U/H support to intel_pmc_core driver” upstream.

You could find the topic via a web search and spot the bug report https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2112469.

Or you can find it mentioned in the changelog like here https://launchpad.net/ubuntu/+source/linux/6.8.0-88.89.

Or you go all in, fetch the kernel repositories that are important to you and see where it is referenced:

$ git --no-pager log --all --grep='from commit bd820906ea9d' --pretty='%H %s'
3b56343cd08a025f3eb632ff145f27d7c3c0ee48 platform/x86/intel/pmc: Add Arrow Lake U/H support to intel_pmc_core driver
44db804982800348c70c759cd8a5e9e4c976322d platform/x86/intel/pmc: Add Arrow Lake U/H support to intel_pmc_core driver

You now know that in those two commits the change you looked for was mentioned and you can look them up.
But usually, as the next step, you then want to check which exact kernel version(s) this change was added to.

You can use git tag –contains and similar on this as usual probing for these new hashes that you found.

$ git tag --contains 3b56343cd08a025f3eb632ff145f27d7c3c0ee48
Ubuntu-6.8.0-88.89
Ubuntu-6.8.0-89.90
Ubuntu-6.8.0-90.91
Ubuntu-6.8.0-91.92
Ubuntu-hwe-6.8-6.8.0-90.91_22.04.1
Ubuntu-lowlatency-6.8.0-88.89.1
Ubuntu-lowlatency-6.8.0-90.91.1
Ubuntu-lowlatency-hwe-6.8-6.8.0-90.91.1_22.04.1

Or, if you need this often, consider an alias or script doing something like:

$ git --no-pager log --all --grep="from commit bd820906ea9d" --pretty='%H' | while read -r c; do   echo "Commit: $c"; echo "Subject: $(git log -1 --pretty=%s "$c")";   echo "Message:";   git log -1 --pretty=format:'  %b' "$c" | sed 's/^/  /';   echo "Branches:";   git branch --contains "$c" --format='  %(refname:short)' 2>/dev/null || echo "  (none)";   echo "Tags:";   git tag --contains "$c" --format='  %(refname:short)' 2>/dev/null || echo "  (none)";   echo; done
Commit: 3b56343cd08a025f3eb632ff145f27d7c3c0ee48
Subject: platform/x86/intel/pmc: Add Arrow Lake U/H support to intel_pmc_core driver
Message:
    BugLink: https://bugs.launchpad.net/bugs/2112469
  
  Add Arrow Lake U and Arrow Lake H support in intel_pmc_core driver.
  
  Signed-off-by: Rajvi Jingar <rajvi.jingar@linux.intel.com>
  Signed-off-by: Xi Pardee <xi.pardee@linux.intel.com>
  Link: https://lore.kernel.org/r/20250207225615.401235-7-xi.pardee@linux.intel.com
  Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
  Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
  (backported from commit bd820906ea9dc3acfcac9de4f1be89b78609e2ac)
  [thibf: due to a large refactor missing:
   ac6bef0d5 platform/x86:intel/pmc: Create generic_core_init() for all platforms
   Duplicate initialization function and update values to reflect arl_h correct state.]
  Signed-off-by: Thibault Ferrante <thibault.ferrante@canonical.com>
  Acked-by: Stefan Bader <stefan.bader@canonical.com>
  Acked-by: Manuel Diewald <manuel.diewald@canonical.com>
  Signed-off-by: Mehmet Basaran <mehmet.basaran@canonical.com>
  Signed-off-by: Edoardo Canepa <edoardo.canepa@canonical.com>
Branches:
Tags:
  Ubuntu-6.8.0-88.89
  Ubuntu-6.8.0-89.90
  Ubuntu-6.8.0-90.91
  Ubuntu-6.8.0-91.92
  Ubuntu-hwe-6.8-6.8.0-90.91_22.04.1
  Ubuntu-lowlatency-6.8.0-88.89.1
  Ubuntu-lowlatency-6.8.0-90.91.1
  Ubuntu-lowlatency-hwe-6.8-6.8.0-90.91.1_22.04.1

Commit: 44db804982800348c70c759cd8a5e9e4c976322d
Subject: platform/x86/intel/pmc: Add Arrow Lake U/H support to intel_pmc_core driver
Message:
    BugLink: https://bugs.launchpad.net/bugs/2112469
  
  Add Arrow Lake U and Arrow Lake H support in intel_pmc_core driver.
  
  Signed-off-by: Rajvi Jingar <rajvi.jingar@linux.intel.com>
  Signed-off-by: Xi Pardee <xi.pardee@linux.intel.com>
  Link: https://lore.kernel.org/r/20250207225615.401235-7-xi.pardee@linux.intel.com
  Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
  Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
  (cherry picked from commit bd820906ea9dc3acfcac9de4f1be89b78609e2ac)
  Signed-off-by: Thibault Ferrante <thibault.ferrante@canonical.com>
  Acked-by: Manuel Diewald <manuel.diewald@canonical.com>
  Acked-by: Edoardo Canepa <edoardo.canepa@canonical.com>
  Signed-off-by: Mehmet Basaran <mehmet.basaran@canonical.com>
Branches:
Tags:
  Ubuntu-hwe-6.14-6.14.0-37.37_24.04.1

After you have tracked the change you are interested in, you might ask what is the most recent kernel you could get out of the automatic updates?
The easiest way is to check right on the system. You can see the current version and available updates, for example via apt policy:

$ apt policy linux-generic linux-generic-hwe-24.04
linux-generic:
  Installed: (none)
  Candidate: 6.8.0-88.89
  Version table:
     6.8.0-91.92 100
        100 http://de.archive.ubuntu.com/ubuntu noble-proposed/main amd64 Packages
     6.8.0-88.89 500
        500 http://de.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages
     6.8.0-31.31 500
        500 http://de.archive.ubuntu.com/ubuntu noble/main amd64 Packages
linux-generic-hwe-24.04:
  Installed: 6.14.0-36.36~24.04.1
  Candidate: 6.14.0-36.36~24.04.1
  Version table:
 *** 6.14.0-36.36~24.04.1 500
        500 http://de.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages
        100 http://de.archive.ubuntu.com/ubuntu noble-proposed/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages
        100 /var/lib/dpkg/status
     6.8.0-31.31 500
        500 http://de.archive.ubuntu.com/ubuntu noble/main amd64 Packages

For a check while not on the system, you can refer to the publishing records of the related source package. The base kernels share the same source name “linux / linux-meta” and hence have one publishing history that shows you all of them. If you want instead to check on a particular hardware enablement stream, those are versioned - like for 6.14, you can instead check the publishing history of linux-meta-hwe-6.14 for example.

Closing thoughts

I hope this post helps to explain why there are so many active Ubuntu kernel source trees for support and optimization, and makes you realize that Ubuntu does all the backports for them! And if you ever wondered how to find a particular change you are interested in, I hope this post was useful for you too by providing some examples.
Either way - I hope that each reader found something to take away that was just the right thing for them!

9 Likes