The Canonical Kernel Team (CKT) is responsible for all things relating to the Linux kernel for any Ubuntu distribution release, including version selection, preparation, qualification, the actual release, and then ongoing maintenance for 9 months or 12 years depending on the release type. One of the most common questions asked is “what will be the kernel version for upcoming release YY.MM?”, but before we talk about that process let’s familiarize ourselves with the differences between the upstream Linux kernel (henceforth referred to as ‘upstream’) and Ubuntu release philosophies.
As described here, the upstream developers “use a loosely time-based release process, with a new major kernel release happening every two or three months”, interpreted to mean that while the intention is to put out a release every 2-3 months, the actual release date is fluid to be able to adjust for any issues that may be found.
In contrast, Ubuntu follows a hard time-based release process, with release dates committed 6 months (or more) in advance with only the most extreme of circumstances able to delay a release.
Given the non-aligned nature of the two release schedules, it is inevitable that there will be instances where both release dates happen to fall on or near the same date. This can be further exacerbated by the nature of the upstream kernel releasing when it is deemed ready, not at some predetermined deadline, which therefore could mean a much later release date than originally anticipated. A general rule of thumb, that the CKT has determined, is that about a month is required between an upstream release and the associated Ubuntu kernel to be considered stable enough for release. That can pose a problem when the upstream release is expected to land either within the 4 weeks before the Ubuntu release or even a few weeks after the Ubuntu release is scheduled.
This is a situation we are heading towards with the upcoming Ubuntu 24.10 which is currently expected to have its release date on or very near the upstream 6.11 release. Here is an illustrative example:
This puts the CKT in a bit of a dilemma. Does Ubuntu release with a 2-3 month old upstream kernel that will likely be superseded on or near the release date? Or should the qualification period be shortened to make the release date, possibly with a lower confidence in stability? Or should the Ubuntu release date be adjusted accordingly, even with prior commitments made?
Please note that none of this is in any way intended to be a criticism of the upstream Linux kernel release process. It is merely describing a reality that the CKT must account for as part of their Ubuntu release planning.
Current Policy
The way the CKT has historically chosen an upstream Linux kernel version was with a conservative ‘wait and see’ approach. Given the month-long stabilization window required, an upstream kernel version all but certain to be released would be the tentative selection, with a possible last minute jump to a more recent version should it turn out to release in a workable time-frame. This approach would guarantee stability on the appointed release day, but was proving unpopular with consumers looking to adopt the latest features and hardware support as well as silicon vendors looking for a firmer version commitment to align their Ubuntu support.
New Policy
The intent behind this post is to describe a new policy the CKT is taking in regards to kernel version selection for an upcoming Ubuntu release. To provide users with the absolute latest in features and hardware support, Ubuntu will now ship the absolute latest available version of the upstream Linux kernel at the specified Ubuntu release freeze date, even if upstream is still in Release Candidate (RC) status.
But as previously mentioned, there are issues with this approach so let’s further describe how this will happen.
Glossary
Before going any further lets define a few recurring terms up front.
Variant Kernel
From the perspective of the CKT, the master or base kernel for a release is the code base that creates the linux-image-generic package binary. Variant Kernels are additional kernel binaries released that have additional support and/or configuration changes from the base kernel to provide an optimized experience tuned for a specific runtime environment. It is important to note that these are alterations to the base kernel which therefore follows that the base kernel must be stable before a derived variant can be stable. See here for more general information about Ubuntu kernels and here for more specific information about kernel variants.
Dependent Components
From the kernel perspective, device driver or other kernel source code that is not a part of the upstream kernel.org repository but that Canonical has decided to integrate and support as part of the Ubuntu Linux kernel. This can include user space as well as kernel software that can be owned by external entities, other teams within Canonical, or the CKT itself. The important distinction here is, that these are all components that are dependent on the kernel version but not fully controlled by the CKT and therefore may not be supported for an indeterminate period of time after upstream release. Some important examples would be NVIDIA GPU drivers and support for the ZFS file system.
However, as the Ubuntu release is not solely a Linux kernel, Dependent Components can also mean any user space libraries or applications that directly interact with the kernel and therefore require the kernel to be stable to verify functioning as intended.
Tight Release
When an upstream kernel is in -rc[4-6] at Feature Freeze, this will be known as a Tight Release. The assumption here is that the upstream kernel is far enough along that the Kernel Team has high confidence it will release before Beta Freeze. However, the upstream kernel release will be so close to the Ubuntu release that it will necessitate a constrained period for testing, addressing issues, and integrating dependent components.
Unstable Release
When an upstream kernel is still in the open merge window or -rc[1-3] at Feature Freeze, this will be known as an Unstable Release. In this situation the CKT is confident that the upstream kernel will still be in an RC state at Beta Freeze where the kernel version is frozen and therefore complete stability or even full dependent component support cannot be expected.
Late Release
Generic term meaning the release is considered to be Tight or Unstable and used as a shorthand term when describing situations common to both.
Stabilized Kernel
An Ubuntu Linux kernel and all Dependent Components that the CKT has declared working as intended and passing all known tests, including full release integration testing as well as kernel testing. In normal situations where Late releases are not in play, this is the kernel as of the GA release date.
Stabilization Period
The period of time between the Ubuntu release GA date with a declared Late kernel and when the CKT declares a Stabilized Kernel. During this time there will be different expectations in terms of features enabled or resolution of issues.
Bridge Kernel
The previous released version of the Ubuntu kernel shipped in some Late release situations that is expected to have full Dependent Component support. The Dependent Components are also expected to be older, or at best equal to, the target version in the pending Ubuntu release. The intent for this is for users who wish to upgrade to the pending release but require Dependent Components that are not yet stabilized, such as a file system, to be able to boot or be at all usable. This will ‘bridge’ them to the new release environment until such time as the stabilized kernel is available. Users on a bridge kernel would be automatically upgraded as soon as the release kernel has stabilized.
Guidelines
Open Merge Window
An upstream kernel that has a merge window opened after Feature Freeze is considered to be too unstable and its eventual release too far in the future to adopt in the pending Ubuntu release. Put another way, an upstream kernel not in active development by Feature Freeze will not be considered for an Ubuntu release.
LTS vs Interim
Ubuntu Long Term Support and Interim releases have differing support timelines and use cases and therefore may have slightly different policies in Late release situations. These will be explicitly specified when a distinction is necessary.
Late Release
Guidelines and caveats common to both Tight and Unstable release situations.
- The image is not expected to be rebuilt once the kernel is stabilized. It will be a snapshot of the stabilization period for the life of the release.
- No prior release upgrades to unstable or bridge kernels, upgrades will be deferred to kernel stabilization.
- Late interim kernels will not roll to the former LTS as HWE until they are stabilized.
- As there is a new image generated for the .1 release, the .1 release will provide the stabilized kernel. As the .1 release is several months after the LTS GA release this is not expected to be a problem.
- For GA, the CKT will commit to providing a late kernel for -generic and any associated build flavors, architectures, and snaps.
- Bridge Kernels (see below) will be provided for any late kernel at Ubuntu release.
Tight Release
In a Tight Release, the upstream kernel is assumed to be in its final form but due to the short window between its release and the Ubuntu release, any number of the Dependent Components may not be in a functional state. It may also be the case that even with testing through the upstream RC period that issues may be present at Ubuntu release.
Additional Guidelines for a Tight Release
- Kernel SRU process continues as normal for the tight kernels that are released.
- Livepatches continue as normal, for released kernels.
- As mentioned above, the CKT will commit to providing -generic and related build flavors for a tight GA, but variant kernels will be best-effort and may need to release later depending on timing or other factors. In some situations a Bridge Kernel (see below) may be prepared.
Unstable Release
In an unstable release the upstream Linux kernel is expected to be still in RC state and therefore not in its final form. Even though in RC there should not be any features added, only fixes to committed features, these fixes could be substantial or it is even possible for features to be removed. As the upstream kernel is still in a state of flux there is no point doing a full regression battery on it or deriving from it and so the CKT will deliver a minimally supported unstable kernel.
Additional Guidelines for an Unstable Release
- No standard SRU on an unstable kernel, the CKT will issue updates as deemed beneficial and at their discretion. Most likely per each RC release, but that is an example and not a requirement.
- No variant unstable kernels will be released, however some variant bridge kernels may be released, see below for more information.
- Dependent components on the unstable kernel will be best-effort and will likely be unavailable until stabilization, at the discretion of the CKT.
- An unstable kernel is considered unsupported so any reported issues or formal support requests will be addressed best-effort by the CKT, but it will be considered acceptable to defer any investigation until after kernel stabilization.
- No livepatches on an unstable kernel. Please note that livepatches cannot be used to upgrade a kernel version, even from X.Y-rcZ to X.Y.
Bridge Kernels
By declaring a late release situation, the Kernel Team is making a public announcement that the version of record for the upcoming Ubuntu release may not be fully qualified and/or in its final form. This can range anywhere from disabled/unavailable dependent components up through lack of confidence in the stability of the currently available version from upstream (which would be in an early RC state in this case). As the primary value provided to consumers of the Ubuntu distribution is assurance of stability and ease of adoption, a means must be provided to users for a seamless and transparent transition to the stabilized release kernel. Or, to put another way, provide a mechanism for users to ‘bridge the gap’ between release and stabilization, hence we introduce the concept of a Bridge Kernel.
The Bridge Kernel is a kernel from a prior upstream release that has been fully prepared and qualified to be a release kernel, but has a short lifetime expectation as it exists solely to provide compatibility until the release kernel of record stabilizes. The original intention for the Bridge Kernel version was to use the prior Ubuntu release kernel version, however with the introduction of the monthly release snapshots, it now seems more feasible, as well as a better user experience, to provide the kernel version immediately previous to the release late kernel version. In any case, the final decision on which version to release as a bridge will be at the discretion of the CKT based on the circumstances at the time.
General Guidelines for Bridge Kernels
- Lifetime is expected to be on the order of weeks, and for users to be updated immediately to the stabilized release kernel once available.
- Because of the short lifetime, no standard support or updates are expected. CVEs and/or issues may be addressed at the discretion of the CKT depending on severity, impact, and latency from the Ubuntu release to kernel stabilization, as part of the SRU process.
- No livepatching of a Bridge Kernel. Which therefore follows that it will not be possible to livepatch from a bridge to a stabilized release kernel.
- There will be no HWE bridge kernels. HWE will be deferred until kernel stabilization.
- No prior release upgrading to a bridge kernel, upgrades will be deferred to kernel stabilization.
Variant Bridge Kernel Guidelines For Unstable
The default policy for variants is to defer everything until kernel stabilization. However, there are certain variants that have their own associated release image, such as Raspberry Pi, which should be available at GA along with the general images. For these kernels a bridge kernel will be prepared, and its associated release image, but there will be no associated unstable kernel. Users will then be required to upgrade to the release kernel once it has stabilized. Which variants will receive bridge images at GA is at the discretion of the CKT based on the circumstances at hand, but the default policy will be any variant with a dedicated release image.
Schedule Timeline
The countdown of milestones from the CKT perspective to Ubuntu release.
| Release Week | Stage | Description (CKT perspective) |
|---|---|---|
| R-8 | Feature Freeze | Upstream kernel must be in an -rc or open merge window by this date to be considered for release. If it is in the merge window or rc[1-3], it is declared an Unstable Release. If it is in rc[4-6] it is declared a Tight Release. Any other rc state has enough confidence to release before Kernel Feature Freeze. In an unstable release situation a bridge kernel will be selected and prepared. |
| R-5 | Kernel Submission Freeze | No major feature submissions will be committed to git past this date. Dependent components in a late release situation are exempt. |
| R-4 | Kernel Feature Freeze | Any features not coming from the direct upstream kernel must be committed with a candidate in -proposed. Dependent components in a late release situation are exempt. |
| R-3 | Beta Freeze / HWE Freeze | At this date the release kernel version is frozen to whatever version it is as of the Monday (upstream historically releases on a Sunday), should be either released or at least rc4. The current state of any dependent components is also frozen barring critical fixes. |
| R-2 | Kernel Freeze | Shipping kernel is frozen in -release |
| R-1 | Final Freeze | |
| R | Final Release | The die is cast |
| R+N | Kernel Stabilization | In a late release situation kernels are updated as the kernel and/or dependent components stabilize. |
See the 24.10 release schedule for an example of the full Ubuntu release timeline.
Going Forward
This is to be the kernel selection policy for all future Ubuntu releases, hence the description of LTS situations as well as the Interim 24.10 release.
With this policy we will be able to be more aggressive about making kernel version commitment announcements for an upcoming release at a much earlier date than previously. However, due to the uncoupled nature of the upstream and Ubuntu releases as described above the CKT will only be able to announce the kernel version for the next upcoming release, not any successive ones.
It is actually expected that Late Releases will be the exception rather than the norm and in most releases these guidelines will not be necessary as the upstream kernel will release with enough time for the Ubuntu kernel to stabilize. However, adopting a more aggressive kernel version commitment policy does require us to be prepared for a possible Late Release situation and therefore informing the community on what they can expect.
)?