The Canonical Kernel Team (CKT) is responsible for all things relating to the Linux kernel for any Ubuntu distribution release, including version selection, preparation, qualification, the actual release, and then ongoing maintenance for 9 months or 12 years depending on the release type. One of the most common questions asked is “what will be the kernel version for upcoming release YY.MM?”, but before we talk about that process let’s familiarize ourselves with the differences between the upstream Linux kernel (henceforth referred to as ‘upstream’) and Ubuntu release philosophies.
As described here, the upstream developers “use a loosely time-based release process, with a new major kernel release happening every two or three months”, interpreted to mean that while the intention is to put out a release every 2-3 months, the actual release date is fluid to be able to adjust for any issues that may be found.
In contrast, Ubuntu follows a hard time-based release process, with release dates committed 6 months (or more) in advance with only the most extreme of circumstances able to delay a release.
Given the non-aligned nature of the two release schedules, it is inevitable that there will be instances where both release dates happen to fall on or near the same date. This can be further exacerbated by the nature of the upstream kernel releasing when it is deemed ready, not at some predetermined deadline, which therefore could mean a much later release date than originally anticipated. A general rule of thumb, that the CKT has determined, is that about a month is required between an upstream release and the associated Ubuntu kernel to be considered stable enough for release. That can pose a problem when the upstream release is expected to land either within the 4 weeks before the Ubuntu release or even a few weeks after the Ubuntu release is scheduled.
This is a situation we are heading towards with the upcoming Ubuntu 24.10 which is currently expected to have its release date on or very near the upstream 6.11 release. Here is an illustrative example:
This puts the CKT in a bit of a dilemma. Does Ubuntu release with a 2-3 month old upstream kernel that will likely be superseded on or near the release date? Or should the qualification period be shortened to make the release date, possibly with a lower confidence in stability? Or should the Ubuntu release date be adjusted accordingly, even with prior commitments made?
Please note that none of this is in any way intended to be a criticism of the upstream Linux kernel release process. It is merely describing a reality that the CKT must account for as part of their Ubuntu release planning.
Current Policy
The way the CKT has historically chosen an upstream Linux kernel version was with a conservative ‘wait and see’ approach. Given the month-long stabilization window required, an upstream kernel version all but certain to be released would be the tentative selection, with a possible last minute jump to a more recent version should it turn out to release in a workable time-frame. This approach would guarantee stability on the appointed release day, but was proving unpopular with consumers looking to adopt the latest features and hardware support as well as silicon vendors looking for a firmer version commitment to align their Ubuntu support.
New Policy
The intent behind this post is to describe a new policy the CKT is taking in regards to kernel version selection for an upcoming Ubuntu release. To provide users with the absolute latest in features and hardware support, Ubuntu will now ship the absolute latest available version of the upstream Linux kernel at the specified Ubuntu release freeze date, even if upstream is still in Release Candidate (RC) status.
But as previously mentioned, there are issues with this approach so let’s further describe how this will happen.
Glossary
Before going any further lets define a few recurring terms up front.
Variant Kernel
From the perspective of the CKT, the master or base kernel for a release is the code base that creates the linux-image-generic package binary. Variant Kernels are additional kernel binaries released that have additional support and/or configuration changes from the base kernel to provide an optimized experience tuned for a specific runtime environment. It is important to note that these are alterations to the base kernel which therefore follows that the base kernel must be stable before a derived variant can be stable. See here for more general information about Ubuntu kernels and here for more specific information about kernel variants.
Dependent Components
From the kernel perspective, device driver or other kernel source code that is not a part of the upstream kernel.org repository but that Canonical has decided to integrate and support as part of the Ubuntu Linux kernel. This can include user space as well as kernel software that can be owned by external entities, other teams within Canonical, or the CKT itself. The important distinction here is, that these are all components that are dependent on the kernel version but not fully controlled by the CKT and therefore may not be supported for an indeterminate period of time after upstream release. Some important examples would be NVIDIA GPU drivers and support for the ZFS file system.
However, as the Ubuntu release is not solely a Linux kernel, Dependent Components can also mean any user space libraries or applications that directly interact with the kernel and therefore require the kernel to be stable to verify functioning as intended.
Tight Release
When an upstream kernel is in -rc[4-6] at Feature Freeze, this will be known as a Tight Release. The assumption here is that the upstream kernel is far enough along that the Kernel Team has high confidence it will release before Beta Freeze. However, the upstream kernel release will be so close to the Ubuntu release that it will necessitate a constrained period for testing, addressing issues, and integrating dependent components.
Unstable Release
When an upstream kernel is still in the open merge window or -rc[1-3] at Feature Freeze, this will be known as an Unstable Release. In this situation the CKT is confident that the upstream kernel will still be in an RC state at Beta Freeze where the kernel version is frozen and therefore complete stability or even full dependent component support cannot be expected.
Late Release
Generic term meaning the release is considered to be Tight or Unstable and used as a shorthand term when describing situations common to both.
Stabilized Kernel
An Ubuntu Linux kernel and all Dependent Components that the CKT has declared working as intended and passing all known tests, including full release integration testing as well as kernel testing. In normal situations where Late releases are not in play, this is the kernel as of the GA release date.
Stabilization Period
The period of time between the Ubuntu release GA date with a declared Late kernel and when the CKT declares a Stabilized Kernel. During this time there will be different expectations in terms of features enabled or resolution of issues.
Bridge Kernel
The previous released version of the Ubuntu kernel shipped in some Late release situations that is expected to have full Dependent Component support. The Dependent Components are also expected to be older, or at best equal to, the target version in the pending Ubuntu release. The intent for this is for users who wish to upgrade to the pending release but require Dependent Components that are not yet stabilized, such as a file system, to be able to boot or be at all usable. This will ‘bridge’ them to the new release environment until such time as the stabilized kernel is available. Users on a bridge kernel would be automatically upgraded as soon as the release kernel has stabilized.
Guidelines
Open Merge Window
An upstream kernel that has a merge window opened after Feature Freeze is considered to be too unstable and its eventual release too far in the future to adopt in the pending Ubuntu release. Put another way, an upstream kernel not in active development by Feature Freeze will not be considered for an Ubuntu release.
LTS vs Interim
Ubuntu Long Term Support and Interim releases have differing support timelines and use cases and therefore may have slightly different policies in Late release situations. These will be explicitly specified when a distinction is necessary.
Tight Release
In a Tight Release, the upstream kernel is assumed to be in its final form but due to the short window between its release and the Ubuntu release, any number of the Dependent Components may not be in a functional state. It may also be the case that even with testing through the upstream RC period that issues may be present at Ubuntu release.
General Guidelines for a Tight Release
- Kernel SRU process continues as normal for the tight kernels that are released.
- The CKT will commit to providing -generic and flavors for a Tight GA, but variant kernels may need to release later depending on timing or other factors.
Interim Releases
- A bridge kernel of the previous stabilized kernel will be prepared for situations where dependent components required by the system are not yet in a stable state. If the prior release was an LTS release then this is just the stabilized LTS GA kernel. If the prior release was an interim, the prior stabilized kernel will be the bridge and its support lifetime will be extended to cover the new late kernel stabilization period. In either case, as this is expected to be a short term situation (in the order of weeks) while the release kernel stabilizes, the bridge kernel will receive no SRU updates as the intention is that users will be updated in that period. A possible exception would be a critical or high CVE that may warrant a Security Cycle release, but that is not expected to be the norm. It is intended that the installer will initially boot the bridge kernel and then make further decisions on which kernel is appropriate to install based on further runtime probes.
- The image is not expected to be rebuilt once the kernel is stabilized. It will be a snapshot of the stabilization period for the life of the release.
- Tight kernels will not roll to the former LTS as HWE until they are stabilized.
LTS Releases
- Livepatches continue as normal, for released kernels.
- There is no bridge kernel in this situation so all upgrades are disabled until stabilization.
- As there is a new image generated for the .1 release, the .1 release will provide the stabilized kernel. As the .1 release is several months after the LTS GA release this is not expected to be a problem.
Unstable Release
In an unstable release the upstream Linux kernel is expected to be still in RC state and therefore not in its final form. Even though in RC there should not be any features added, only fixes to committed features, these fixes could be substantial or it is even possible for features to be removed. As the upstream kernel is still in a state of flux there is no point doing a full regression battery on it or deriving from it and so the Kernel Team will deliver a minimally supported unstable kernel.
General Guidelines for an Unstable Release
All Tight Release guidelines apply to an unstable release with the following additions and clarifications.
- No standard SRU on an unstable kernel, the CKT will issue updates as deemed beneficial and at their discretion. Most likely per each RC release and then finally on transition to a stabilized kernel, but that is an example and not a requirement.
- No variant kernels will be released until base kernel stabilization. Any images based around a variant kernel will need to be delayed until base kernel stabilization. At GA there will be the current RC snapshot of the -generic unstable kernel and variants with a best-effort regression mitigation.
- Dependent Components on the unstable kernel will be best-effort and will likely be unavailable until stabilization, at the discretion of the CKT.
- An unstable kernel is considered unsupported so any reported issues or formal support requests will be addressed best-effort by the CKT, but it will be considered acceptable to defer any investigation until after kernel stabilization.
LTS Releases
- No livepatches on an unstable kernel.
Schedule Timeline
The countdown of milestones from the CKT perspective to Ubuntu release.
| Release Week | Stage | Description (CKT perspective) |
|---|---|---|
| R-8 | Feature Freeze | Upstream kernel must be in an -rc or open merge window by this date to be considered for release. If it is in the merge window or rc[1-3], it is declared an Unstable Release. If it is in rc[4-6] it is declared a Tight Release. Any other rc state has enough confidence to release before Kernel Feature Freeze. In an unstable release situation a bridge kernel will be selected and prepared. |
| R-5 | Kernel Submission Freeze | No major feature submissions will be committed to git past this date. Dependent components in a late release situation are exempt. |
| R-4 | Kernel Feature Freeze | Any features not coming from the direct upstream kernel must be committed with a candidate in -proposed. Dependent components in a late release situation are exempt. |
| R-3 | Beta Freeze / HWE Freeze | At this date the release kernel version is frozen to whatever version it is as of the Monday (upstream historically releases on a Sunday), should be either released or at least rc4. The current state of any dependent components is also frozen barring critical fixes. |
| R-2 | Kernel Freeze | Shipping kernel is frozen in -release |
| R-1 | Final Freeze | |
| R | Final Release | The die is cast |
| R+N | Kernel Stabilization | In a late release situation kernels are updated as the kernel and/or dependent components stabilize. |
See the 24.10 release schedule for an example of the full Ubuntu release timeline.
Going Forward
This is to be the kernel selection policy for all future Ubuntu releases, hence the description of LTS situations as well as the Interim 24.10 release.
With this policy we will be able to be more aggressive about making kernel version commitment announcements for an upcoming release at a much earlier date than previously. However, due to the uncoupled nature of the upstream and Ubuntu releases as described above the CKT will only be able to announce the kernel version for the next upcoming release, not any successive ones.
It is actually expected that Late Releases will be the exception rather than the norm and in most releases these guidelines will not be necessary as the upstream kernel will release with enough time for the Ubuntu kernel to stabilize. However, adopting a more aggressive kernel version commitment policy does require us to be prepared for a possible Late Release situation and therefore informing the community on what they can expect.
)?