SPEC: Closely Coupled Kernel Userspace Releases

Project Discussion Foundations
,
1

Abstract

Filesystem support is handled in the Linux kernel. Many filesystems come with userspace tools. As the Ubuntu hwe (hardware enablement) kernel moves forward, new capabilities are available for filesystems. However, the userspace packages are frozen. An example is btrfs-progs which maintained separately in github[1]. From the README in the repository: “The major version releases are time-based and follow the cycle of the linux kernel releases. The cycle usually takes 2 months [sic: from the upstream kernel release]. A minor version release may happen in the meantime if there are bug fixes or minor useful improvements queued.” The goal, in the case of btrfs-progs, is to have userspace and kernel capabilities in lockstep for users. This is not the current case in Ubuntu.

Userspace packages in Ubuntu also have the issue of being pulled from Debian. If Debian is seeking to keep userspace and kernel in-line, this will lead to misalignment with Ubuntu kernels. This is easily shown on an Ubuntu 24.04 Noble Numbat system, where the LTS kernel is 6.8, and btrfs-progs is at 6.6.3. The misalignment with kernel and difficulty in maintaining the package lead to bugs that can cause major repercussions.[2][3][4]

This specification begins with the general approach and workflow for packages that are closely coupled to the Ubuntu kernels. The second portion is a working example of btrfs-progs following the specification.

Rationale

  • When userspace and kernel are so tightly coupled, users of a Hardware Enablement Kernel should have the full capabilities of that kernel available in the associated user space packages.
  • Backporting specific fixes in tightly-coupled kernel userspace packages can be tricky, as bugfixes may be relying on kernel changes as well.
    • Example scenario: a bug is reported against btrfs. It turns out that this issue has a kernel and userspace component. The kernel portion is addressed in $NEW_KERNEL in an HWE roll. btrfs-progs stays on the same major version and a targeted fix is done. However, due to the mixing of new features and bug fixes related to new btrfs work in the kernel and userspace, extracting only the bugfix is difficult.

Specification

High Level Considerations for Inclusion

Not all packages can be considered a part of the userspace hwe-kernel tight coupling set. The following are base criteria to see if a package should be considered a part of this specification:

  1. Primary functionally of the package is directly tied to Kernel version
  2. Kernel capabilities added in an HWE roll are inaccessible to most users without the userspace package
  3. Upstream development does not maintain any “long term stable” branches
  4. Changes to upstream code mix bugfix and features
    1. As upstream code moves forward, a bug which affects “older” versions may be fixed on top of already changed code. Thus crafting specific bugfix patches is difficult, and can lead to incompatible changes with older versions.
    2. This may lead to situations where a new version is being backported that only contains enablement for the HWE kernel and no bugs being addressed. This should be safe for users, but can lead to an extra update.
  5. Upstream projects set acceptable minimum version compatibility against the kernel.
    1. Ex: an upstream project supports backwards compatibility with the LTS kernel available in a given Ubuntu release.
    2. Bonus points for backwards compatibility against all standard supported kernels, as it means keeping a consistent version through Ubuntu releases.
  6. Packages are end-user applications not strictly libraries
    1. Library changes create a huge blast radius. While applications may be called by other applications (package-a relies on ethtool) there are better guarantees on API compatibility compared to bumping a SONAME and ABI compatibility
    2. As is, it must be proven that a package is best served by rolling, and has minimal possible disruptions to the Ubuntu ecosystem at large.

Workflow

What follows is the general flow of the package, from development to SRU. This includes required steps and considerations. The content is generic in form at this point, with a specific example held in the btrfs-progs section later.

Ubuntu Devel

The Ubuntu kernel is taking a bleeding edge approach, taking as new a version as possible. This may mean releasing with an RC kernel from upstream. Many userspace packages do not update until after the release of the upstream kernel, with lag being different for each package. If a time differential is generally known, it must be documented by the package. Basic steps:

  1. Follow upstream releases as close as possible, ensuring the latest release is in Ubuntu devel
  2. If a kernel is released, and no userspace package is available at the time of release, the package must first be put into Ubuntu Devel and then SRU’d.
    1. It is acceptable to do an SRU before the development release is complete if it offers enablement for the kernel provided functionality in $PREVIOUS_RELEASE.
      1. Ex: Ubuntu Resolute Racoon 26.04 is shipping with kernel 7.0.0, taken from 7.0.0rc7 upstream. At 26.04 release, userspace tools are not released yet. It is acceptable to get the new tool into 26.10 Stonking Stingray, and SRU into 26.04 before 26.10 release.

Ubuntu LTS

The Ubuntu kernel follows two primary tracks during LTS releases – HWE and Stable. There are also specialized OEM kernels that are on fixed versions that may differ from HWE and Stable. This leads to a situation where there are, minimally, two versions of the kernel available. For a package to be able to be backported to a Stable Release, it must declare support for the Stable LTS kernel. An example upstream README from btrfs-progs states

Feature compatibility

The btrfs-progs of version X.Y declare support of kernel features of the same version. New progs on old kernel are expected to work, limited only by features provided by the kernel.[5]

If the above condition is met, a package may be backported from Devel into Stable Releases. This must follow standard procedures, not skipping any release unless there is a documented reason. A fictitious example:

  • tightly-coupled-package-a is at version 1.90 in devel

  • Ubuntu Development for 28.10 Winsome Weasel (WW) has begun

  • tightly-coupled-package-a 1.90 defines that it has support for kernel versions starting at 6.15.0

  • The Ubuntu Kernel moves to 7.6

  • tightly-coupled-package-a follows with version 1.91 enabling features in 7.6 during Ubuntu Development

    • It retains the same earliest starting version of the kernel

  • tightly-coupled-pakage-a is updated to 1.91 during development

  • Ubuntu is released

  • At 28.04 VV .2 release

    • The current LTS kernel is 7.4
    • HWE kernel rolls to 7.6

  • Based on the above criteria, rolling tightly-coupled-package-a version 1.91 into 28.04 is most desirable

  • A team member prepares an SRU, with full-template, for the release of 1.91 to enable the HWE kernel

  • They also check for any bugs that may be addressed…

  • Bundled in 1.91 are various bugfixes. Because of the enablement of new features, separating the fixes from the feature work is difficult.

  • A bug is reported in tightly-coupled-package-a on the original version of the package, 1.83, as found in Ubuntu 26.04 Resolute Racoon. This bug affected 28.04 as well with their version 1.89

  • Version 1.91 upstream specifies it fixes the bug.

  • After successfully testing 1.91 on 28.04, a team member prepares uploads for 26.04, as it is the next supported release

    • Note: because this happened after the .2 release, Ubuntu UU is an interim version with only 9 months of support and has already been deprecated. Therefore it does not need to receive an update

  • This follows a standard SRU procedure – do not skip releases, ensure no breaking changes, ensure compatibility of the tool, especially against both the LTS and HWE kernels.

Backport/SRU Release Timing

The Kernel team rolls the HWE kernel roughly 3 months after the release of an interim or the next LTS release. These coincide with the “dot” releases for an LTS.[6] This means that tools must also follow this timing. Due to the time between Ubuntu release and backport of the HWE kernel, a new userspace filesystem tool is likely. Use the “dot” releases as a milestone, with coordination with the release team and kernel team.

Things like filesystem tools are often seeded, which means needing to be prepared and ready for dot release candidate images. This is because filesystem tools are made available on the ISOs for users to freely choose root file systems.

Documentation

All packages meeting the above criteria must be documented in “niche package maintenance”[7] under “Tightly Coupled Kernel / Userspace Packages.” The documentation for each package must outline all the information for the package:

  • Release cadence (if publicized)
  • Statements of and links to compatibility statements
  • Testing documentation
    • Autopkgtests and their coverage
    • Kernel integration testing
    • Any manual testing steps required for packages.
  • Any exceptions that need to be made
    • Ex: inability of running autopkgtests due to environment

Testing

Autopkgtests

Tools should have autopkgtests. Minimally, these tests must exercise:

  • Installation of the package (in most cases, this will upgrade a system)
  • Basic operations

If a package does not have autopackage tests, it must be clearly documented why they are not appropriate and what testing will be done to ensure quality and lack of regressions. These tests must be run against the LTS and HWE kernels

Kernel Testing

In the case of the kernel containing tests exercising the capabilities, including using the userspace tool, those tests must be run. These tests must be run against the LTS and HWE kernels.

Manual Testing

If a manual test is required, either due to lack of autopkgtest capability or kernel integration tests, the manual test must be documented fully. These test plans must be written in such a manner that any individual can run the tests, given reasonable pre-requisites.

Brtfs-progs

What follows is an example of a niche package page for btrfs-progs.

Basic Information

  • Btrfs-progs is maintained in github and follows the kernel release.[8]
  • Btrfs-progs states “The btrfs-progs of version X.Y declare support of kernel features of the same version. New progs on old kernel are expected to work, limited only by features provided by the kernel.”[9]
  • Releases use tags. As can be seen in the tags, there are no long-term branches, instead continuous moving versions aligning with upstream kernels.[10]
  • Primary functionality of btrfs is contained in the kernel.
  • For minor releases, “A minor version release may happen in the meantime if there are bug fixes or minor useful improvements queued.”[11]

Autopgktest

As of 20260128, brtfs-progs does not have autopkgtests. Reverse run-time dependencies are:

Dependency Analysis

Reverse-Depends

* apt-btrfs-snapshot # create btrfs snapshots whenever apt is run : universe
* btrbk # backup tool for btrfs volumes : universe
* btrfsd [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x] # daemon for btrfs maintenance : universe
* btrfsmaintenance # scripts automating btrfs maintenance tasks : universe
* calamares-settings-mobian # calamares + Mobian for small touch screen devices : universe
* freedom-maker # FreedomBox image maker. Freedombox is a personal cloud server : universe
* golang-github-containerd-btrfs-dev : src:golang-github-containerd-btrfs # go bindings for btrfs : universe
* initramfs-tools-devices # Common initramfs scripts for Ubuntu Core and Classic : universe
* kiwi-dracut-lib [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x] : src: kiwi # modules for dracut for kiwi image builder
* kiwi-systemdeps-filesystems [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x] : src: kiwi # kiwi image builder host setup helper : universe
* libblockdev-btrfs3 : src: libblockdev # library plugin and standalone for btrfs and C++ projects : universe
* libguestfs0t64 [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x] : src: libguestfs # shared library for guest disk image management : universe
* mkosi
* ubuntu-server [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x] : meta
* ubuntu-server-minimal [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x] : meta
* ubuntu-server-raspi [arm64 armhf] : meta

Reverse-Recommends

* calamares [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x] # alternative graphical installer : universe
* calamares-extensions [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x] # mobile modules for calamares
* distrobuilder [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x]
* lubuntu-desktop [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x] : meta
* lubuntu-desktop-minimal [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x] : meta
* ntfs2btrfs [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x] # convert ntfs to brtfs : universe
* ubiquity [amd64 amd64v3 arm64 armhf ppc64el riscv64 s390x] # old installer

There are a further few reverse-build-depends

Reverse-Build-Depends

* btrfsd # yet another btffs daemon for maintenance tasks : universe
* containerd-app # daemon to control runc. Runc + containers = can use btrfs
* containerd-stable # as above
* docker.io-app # as above, except Docker
* golang-github-containerd-btrfs # stated above with golang-github-containerd-btrfs-dev
* libguestfs # stated above with libguestfs0t64

Of these dependencies, only libblockdev provides direct btrfs testing mounting btrfs filesystems, manipulating volumes and snapshots, and validating the plugin path from top level to the kernel module. Of the build dependencies, only docker.io-app contains any reference to btrfs, and it’s somewhat ancillary. Running the docker-in-lxd tests configure lxd with a btrfs backend.

The packages broadly show a need to have matching functionality with the kernel. For image building, container runtimes, and libraries it is important to ensure full-functionality with kernel features. For btrfs-specific maintenance scripts and daemons, stability of the interface is important.

Autopkgtest Requirements and Addition of Tests

Btrfs-progs needs to have autopkgtests added to be able to be SRU’d safely. Further, passing libblockdev tests are of high importance.

  1. Test installation and upgrade of btrfs-progs (it’s currently seeded on server, so will likely been in any VM, container, or chroot unless explicitly pruned)
  2. Test basic filesystem operations
    1. Create a bare partition
    2. Ensure partition is unmounted
    3. Mkfs.btrfs the new partition
    4. Btrfs check --read-only --force $device
    5. Ensure partition is mounted
    6. Write a bunch of files
    7. Run btrfs filesystem du , ensure nothing crashes
    8. Run btrfs device stats $device
    9. All commands need sudo
    10. Tests must be run with isolation-machine

The goal of the tests is not to run every command, nor to cover every case. It’s a selection of “low hanging fruit” that tests that the program is installed and common commands are available. We will not be able to cover all the cases in an autopkgtest, including functions such as subgroups, managing quota, replication, etc.

Tests must be run with the LTS and HWE kernel. Creating a proper dependency graph that ensures these tests are automatically run on upload of any filesystem tool may not be possible. In which case a single case will be covered for any interim bugfix updates, and manual testing against LTS and HWE done on`btrfs-progs updates.

Kernel Testing

The Ubuntu Kernel team maintains a test suite that exercises filesystem capabilities. During the coordinated dot release, the kernel team will run these tests against the LTS kernel and the HWE kernel with the new version of btrfs-progs that has been prepared. Aspirationally, this testing will be done with a package made in a PPA, as there is currently no way to block proposed-migration on an external test. The kernel team will also aid when the new version of any filesystem tool is made available to do similar testing. The Kernel team regularly runs filesystem tests using the version of userspace tools already released, so continuous testing will occur related to kernel updates. Test results from the kernel runs will be posted to the appropriate SRU bugs

Ubuntu Policy

Based on this spec, this process should meet the qualifications for a Micro Release Exception (MRE)[12]. This means that a specific exception for btrfs-progs and other filesystem tools should not be needed because this one is a generic template for all that apply. If in reviewing this spec with select members of the SRU team they decide that an SRU Exception needs to be filed, only the ways in which this spec deviates from an MRE must be specified.

Public Statements

This spec must be made public in a static location. The class of exceptions (Closely Coupled Kernel Userspace Packages) must be documented in the SRU docs. Each package will document the specific rationale for inclusion into this set in the SRU package-specific docs. This will be setup as a paged structure, with the definition at the top, and package specifics underneath.

Spec History and Changelog

This spec has been made public for comment on 2026-04-21. It has been reviewed internally at Canonical until 2026-04-20.

  1. ↩︎

  2. ↩︎

  3. ↩︎

  4. Note that in Debian it got orphaned, and then salvaged https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1088329 ↩︎

  5. ↩︎

  6. A “dot” release is, essentially, the kernel roll + a more fully tested ISO. Some packages may follow the “dot” releases due to tight coupling with the ISO (such as subiquity ) or internal Canonical teams may use dot releases as milestones to achieve partner commitments. Cloud images are continuously built, tested, and released. ↩︎

  7. https://documentation.ubuntu.com/project/maintainers/niche-package-maintenance/ ↩︎

  8. ↩︎

  9. ↩︎

  10. ↩︎

  11. ↩︎

  12. https://documentation.ubuntu.com/project/SRU/reference/special/ see “New upstream release that adds features without breaking existing behaviour” ↩︎

2 Likes
2

Awesome stuff, thanks for working on this.

One very vivid example of the consequences of mismatched kernel and filesystem utilities is https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2110891 – I don’t think the core of this issue has actually been addressed yet.

Would this situation have been better if the filesystem tools upgrade had happened “between” boots? so the new tools could use the new kernel’s interfaces. (Maybe we can’t do any better than asking upstreams to be more careful.)

Thanks

2 Likes
3

this is a tougher case, and highlights why it’s important to ensure backwards compatibility. This also crosses upgrade boundaries though, which is even more difficult.

If a tool in $NEW_RELEASE has breaking changes against kernel in $OLD_RELEASE, then during upgrade, when we’re still running $OLD_KERNEL until reboot, it can lead to issues. The way to solve that is usually in upgrader quirks. I’m not a fan of that mechanism either, but that’s a different problem for me to have opinions on :slight_smile:

In the upgrader case, it sounds like, yes, we should have a general quirk that is “upgrade the tools after reboot” which is … an annoying bit of code to write. Or, maybe more importantly, we need to be doing upgrade testing for supported filesystems to ensure the path is clear.

for this spec, i don’t think we need to worry about between boots. Backwards compatibility is a strong requirement. If there’s a breaking feature, that version cannot be backported to where it breaks.

1 Like