| Index | FO238 |
|---|---|
| Title | stubble: A secure-boot friendly device-tree loading EFI stub |
| Status | Approved |
| Authors | Tobias Heider |
| Type | Implementation |
| Created | 2025-07-21 |
| Reviewer | Status |
|---|---|
| @xypron | Approved Jul 30, 2025 |
| @waveform | Approved Jul 30, 2025 |
| @mkukri | Approved Jul 30, 2025 |
Abstract
Beginning with Ubuntu 24.10, we offer a universal arm64 desktop ISO designed for broad compatibility across arm64 desktop platforms that utilize UEFI for booting.
Certain platforms, particularly Qualcomm Snapdragon Laptops, lack functional ACPI support on Linux. Because they also don’t supply a suitable device tree in their firmware, we currently load device trees built from Linux manually via the Grub bootloader. While effective, this method does not support Secure Boot due to the unsigned nature of the device trees and requires a complex configuration to identify the correct device tree for each device.
This specification details stubble, a small Linux kernel EFI stub designed to address this limitation. It achieves this by bundling device trees directly with the kernel image and incorporating the loading logic into a compact EFI stub that executes prior to kernel boot. This results in a single file that can be signed using our existing Launchpad signing pipeline, encompassing both the kernel and its associated device trees.
Specification
Current Solution
Ubuntu currently relies on a bespoke grub configuration, ubuntu-drivers and flash kernel to manage device-specific DTBs on the generic arm64 installer ISO.
The bootloader grub configuration shipped as part of the debian-cd package uses the Grub smbios command to identify the running device based on the information provided in the SMBIOS table by the device firmware. A long if-else statement then configures device-specific command line options and a device tree if needed.
Here’s how this functions on the Lenovo Thinkpad T14s Gen 6, as illustrated in the code block below:
smbios --type 4 --get-string 5 --set proc_version
regexp "Snapdragon.*" "\$proc_version"
if [ $? = 0 ]; then
smbios --type 1 --get-string 6 --set system_product_version
if [ "$system_product_version" == "ThinkPad T14s Gen 6" ]; then
dtb="devicetree /casper/dtbs/x1e78100-lenovo-thinkpad-t14s.dtb"
fi
# ...
fi
menuentry "Try or Install Ubuntu" {
linux /casper/vmlinuz $KERNEL_PARAMS console=tty0
initrd /casper/initrd
$dtb
}
Grub script with its built-in commands offers a powerful identification logic, leveraging regular expressions and access to the arbitrary SMBIOS fields like manufacturer, product name, and SKU number but also processor information. Unfortunately recent changes to the Linux upstream device trees require acquiring additional device identifiers such as the display EDID which is not currently available in grub. We would need to implement the low-level code from scratch and expose it as a new grub script command to provide the required information.
Once the live system is booted, during the standard installation process, the Ubuntu desktop installer executes the ubuntu-drivers command. This command identifies device-specific packages by matching modalias signatures. We leverage the export of SMBIOS fields as modalias entries in /sys/devices/virtual/dmi/id/modalias to install the hwe-qcom-x1e-meta package. This package includes additional device-specific configuration files and pulls in flash-kernel as a dependency.
Flash-kernel installs a kernel update hook to copy a device-tree to the /boot partition. It selects a matching device-tree by comparing /proc/device-tree/model[^1] with an embedded database. The selected DTB is then copied to /boot/dtbs and linked to /boot, where update-grub finds it and generates a corresponding device-tree directive.
The current Ubuntu secure boot architecture lacks compatibility with this mechanism. Unsigned device trees are prohibited in secure-boot mode because they pose a risk of bypassing essential security checks or deactivating security components on the device. In GRUB, this is implemented in the so-called lockdown mode which is automatically activated if secure boot is enabled. In lockdown mode, some features we currently rely on such as the devicetree command are not available.
Signing and verifying individual device trees presents a significant challenge due to the absence of a reference implementation within our infrastructure. Our current secure boot verification logic relies on shim, which is limited to verifying EFI executables and would require substantial workarounds to extend its capabilities to device trees.
Another downside of the current approach is that resulting installations are not portable because the matching is only done on the boot media. The resulting installation is limited to the device-specific DTB installed by flash-kernel.
Related work
dtbloader
dtbloader is an improved rewrite of the earlier DtbLoader.efi app. It implements device identification and automated device-tree loading and patching based on so-called Computer Hardware IDs (CHIDs or HWIDs). CHIDs are derived from the SMBIOS table, but hash the contents in a set of pre-defined combinations. This makes them easier to compare but doesn’t allow matching substrings or regular expressions as we do in our current grub approach. Dtbloader embeds a CHID to device-tree lookup table. On boot it generates the device specific CHIDs, compares them to the lookup table and if a match is found loads the corresponding device-tree.
Once dtbloader has injected itself into the boot process and loaded a device-tree it is entirely opaque to the way the distribution works. From an operating system point of view there is no way to distinguish between a device-tree loaded by the boot firmware vs. one loaded by dtbloader. This is desirable because it means no additional changes in the system are required.
The downside of this approach is that it is loaded as an EFI driver which is currently not compatible with grub and doesn’t play well with our shim secure boot verification architecture in Ubuntu.
Systemd UKI
The systemd-stub, which is an essential building block of systemd Unified Kernel Images (UKIs), has merged support for dtb loading functionality inspired by dtbloader. The UKI, consisting of at least a Linux kernel image and initrd can embed a CHID database in the .hwids elf section and a list of device-tree binaries in the .dtbauto section. Device-tree loading and patching happens as part of the stub code before the actual Linux kernel is loaded.
The benefit of this approach is that it gives us a single image file that is compatible with grub and shim. There are also niche use-cases where Ubuntu already uses systemd UKIs, such as Ubuntu Core and TPM FDE.
The downside is that UKI and the systemd stub implement an opinionated boot architecture that is quite different from what Ubuntu assumes today. The initrd is always part of the UKI which means it can also benefit from secure-boot signing but in turn can’t be generated or modified on the local machine. Systemd also provides a solution in the form of system extensions and addons but those would require fundamental changes to the current Ubuntu security architecture and signing key enrollment.
Many Ubuntu packages, especially those inherited from Debian, rely on a mutable initrd. These packages ship additional initrd hooks or configurations. While the UKI approach does provide possible security benefits, those are orthogonal to the DTB loading problem and not the subject of this specification.
Migrating all of Ubuntu to the UKI model would require large-scale changes or break many non-compatible packages. Ubuntu is in the midst of transitioning from initramfs-tools to dracut. Another initrd paradigm switch is unlikely to happen in the near future.
Furthermore, defaulting to a UKI model solely for device tree architectures is undesirable due to the potential for fragmenting the community and increased development and QA efforts across all installations.
stubble
Rationale
None of the previously presented solutions are suitable for direct adoption by Ubuntu. dtbloader does not address the secure boot issue, and systemd UKI would necessitate substantial packaging changes. An alternative idea, briefly discussed, involved modifying systemd-stub to enable pre-loading an initrd in GRUB. This concept was ultimately rejected as it would have undermined systemd UKI’s architectural principles.
Our proposed solution is stubble, an UKI compatible stub that does not enforce any additional limitations and only implements one feature: loading device-specific DTBs before the kernel starts. This solution combines the strengths of both dtbloader and systemd UKIs. The resulting combined kernel+stub image behaves just like any regular kernel image in almost all regards, except it also loads device-trees when needed. No additional changes should be required throughout the system.
A regular kernel and the bundled kernel+stub should be interchangeable and behave identically unless one of the embedded CHIDs matches and there isn’t already a device tree loaded in memory. This makes it safe to default to the combined kernel+stub even on platforms without device-trees or working DTB management implemented in firmware.
Keeping the binary layout similar to systemd UKIs allows us to utilize existing tooling such as systemd-ukify and share a common CHID database.
Implementation
stubble is an EFI application compiled to the PE binary format. Bundled data such as the kernel, device trees and CHID database are stored in corresponding PE sections inherited from systemd UKI. We use systemd-ukify to populate those sections.
Building a kernel image
Given a stub binary stubble.efi, a simple ukify invocation could look as follows.
ukify build --stub=stubble.efi --linux=/boot/vmlinuz --dtbauto=machine1.dtb --dtbauto=machine2.dtb --hwids=hwids --output=vmlinuz.out
This creates a .linux section containing /boot/vmlinuz, two .dtbauto sections containing machine1.dtb and machine2.dtb respectively, and finally a .hwids section containing the CHIDs stored in the hwids/ directory in the PE executable. More device trees can be included by passing additional --dtbauto arguments.
CHIDs are generated with fwupdtool hwids, filtered to make sure we only include unambiguous device identifiers and then converted to a JSON format to be consumed by ukify. For example on the Lenovo Thinkpad T14s this might look like:
$ sudo fwupdtool hwids
Computer Information
--------------------
BiosVendor: LENOVO
BiosVersion: N42ET88W (2.18 )
BiosMajorRelease: 2
BiosMinorRelease: 18
FirmwareMajorRelease: 01
FirmwareMinorRelease: 1b
Manufacturer: LENOVO
Family: ThinkPad T14s Gen 6
ProductName: 21N2ZC5QUS
ProductSku: LENOVO_MT_21N2_BU_Think_FM_ThinkPad T14s Gen 6
EnclosureKind: a
BaseboardManufacturer: LENOVO
BaseboardProduct: 21N2ZC5QUS
Hardware IDs
------------
{1d9f3ebb-96de-5dd6-8c88-38308b0c1c44} <- Manufacturer + Family + ProductName + ProductSku + BiosVendor + BiosVersion + BiosMajorRelease + BiosMinorRelease
{578dd7d5-5871-5bd5-92a9-be07f1067b92} <- Manufacturer + Family + ProductName + BiosVendor + BiosVersion + BiosMajorRelease + BiosMinorRelease
{ed647f93-3075-598b-9d89-d0f30ec11707} <- Manufacturer + ProductName + BiosVendor + BiosVersion + BiosMajorRelease + BiosMinorRelease
...
When converted to JSON, this gives:
{
"type": "devicetree",
"name": "LENOVO ThinkPad T14s Gen 6 (LCD)",
"compatible": "lenovo,thinkpad-t14s-lcd",
"hwids": [
"1d9f3ebb-96de-5dd6-8c88-38308b0c1c44",
"578dd7d5-5871-5bd5-92a9-be07f1067b92",
"ed647f93-3075-598b-9d89-d0f30ec11707",
...
]
}
The compatible and name fields are filled in manually. Compatible is used to map the CHIDs to a specific device tree. A CHID database is shipped as part of the stubble Ubuntu package. The original kernel image and device trees come from the kernel package.
The kernel image build step can happen entirely in the kernel package build process in launchpad so that the resulting vmlinuz binary can be signed with the existing workflow. For a transitional period we might consider shipping both the original vmlinuz binary and the one bundled with stubble to provide a fallback for systems where the stub version causes unforeseen problems. This is discussed in more detail in the “Considerations” section below.
Inside the stub
Upon launch the stub will first parse the command line arguments it was booted with. Most of those will need to be forwarded to the kernel later, some might be relevant to the stub itself. We respect logging flags such as quiet and debug for the stub.
Next it finds sections embedded in its own binary image. The automatic CHID to DTB matching occurs in this step, if a .dtbauto section is present and there isn’t already a device-tree registered . The stub computes device-specific CHIDs from the UEFI SMBIOS table and then compares them to the CHIDs in the .hwids section. Upon finding a match, it stores the corresponding compatible string for subsequent steps.
The stub then iterates over all embedded sections and maps them for their corresponding use-case. If a .dtbauto section is encountered the device tree compatible field is compared to the stored one and if it matches the section is marked to be loaded for later.
After all sections have been found, the stub uses the EFI_DT_FIXUP_PROTOCOL to apply firmware specific fix-ups to the embedded DTB and then installs it using InstallConfigurationTable.
The stub may include an optional, embedded .cmdline section. If present, this section will be used to construct a new command line string by combining its options with those originally passed to the stub.
Subsequently, the kernel will be executed and passed this newly formed command line string.
Kernel build integration
Integrating stubble into our existing kernel packaging and signing workflow should be straightforward.
stubble needs to be packaged and uploaded to the Ubuntu archive, and then added as a kernel build dependency alongside systemd-ukify. Instead of directly installing the vmlinuz.efi binary produced by Linux, an additional step will be introduced to run ukify and build a combined kernel package that replaces vmlinuz.efi.
Initially, this new behavior will be conditionally enabled only for the arm64 architecture, by defining do_stubble as true in the architecture-specific rules configuration. Other architectures continue installing the unmodified vmlinuz.efi binary.
ifeq ($(do_stubble),true)
# Build kernel+stub image
dtb_files="$(find $(build_dir)/arch/arm64/boot/dts/qcom/ \
-name "x1*.dtb" -not -name "*-el2.dtb")" \
args=""; for f in ${dtb_files}; do args="${args} --devicetree-auto=${f}"; done; \
/usr/bin/ukify build --linux=$(build_dir)/$(kernfile) \
--stub=/usr/lib/stubble/stubble.efi \
--hwids=/usr/share/stubble/hwids \
$args \
--output=$(build_dir)/$(kernfile).stubble
# The main image
install -m600 -D $(build_dir)/$(kernfile).stubble \
$(pkgdir_bin)/boot/$(instfile)-$(abi_release)-$*
else
# The main image
install -m600 -D $(build_dir)/$(kernfile) \
$(pkgdir_bin)/boot/$(instfile)-$(abi_release)-$*
endif
No further changes are anticipated for the signing pipeline, as it should function correctly with the new kernel binary.
Considerations
Changing the default linux-generic kernel on arm64 to use stubble comes with a few potential challenges.
A major API break in comparison to our current arm64 generic kernel is the hard dependency on EFI. We believe the number of affected devices to be rather small. Most server and desktop systems come with EDK2, even smaller platforms will usually have some EFI support through u-boot. One big platform without EFI is Raspberry Pi but those use a dedicated kernel and aren’t affected by changes to our generic kernel. The UEFI secure boot signed kernel naturally already depends on EFI being available.
In any case we should provide a fallback option to install the raw kernel image without stub. There are legitimate use-cases where users might want to build their own systemd UKIs from our pre-built kernels. It would not make sense to build those from the combined kernel+stubble binaries since the stubs have overlapping functionality. Providing a fallback is also a good idea in case we find any unforeseen regressions or incompatibilities with the kernel+stub approach.
Finally, while our experimental evaluation has shown the bundled DTB approach is feasible for the current set of devices we care about (primarily Snapdragon X machines), a growing number of device trees might present challenges in the future. The current total size of device tree binaries shipped in Linux 6.14.0-24-generic is 68 MB. The size of those included initially adds up to 7.5 MB. There is an opportunity to optimize this further by using compression or deduplication using device tree overlays.
Our current kernel compression in arm64 relies on grub decompressing the binary before loading it. The stub has no such functionality so ukify will unpack the kernel before writing it to the .linux section. In order to make use of compression we should enable the EFI_ZBOOT kernel boot option which adds decompression to the Linux upstream stub and decompresses the kernel internally. This method can reduce an uncompressed 61 MB kernel to about 20 MB.
Timeline
A functional prototype has been developed, and we anticipate resolving this issue by the upcoming stable release. The proposed solution involves replacing the arm64 linux-generic kernel with a bundled stub version in Ubuntu 25.10.
Given the previously discussed challenges, implementing such a significant change in a stable release is likely inadvisable. By establishing this as the default for the preceding interim release, we can gather valuable testing feedback before it impacts the broader user base of the Ubuntu 26.04 LTS release.
Further Information
Linaro engineers have addressed this issue in their presentations at FOSDEM 2025 and DebConf 25. Their evaluation of the advantages and disadvantages of the existing solutions aligns closely with our own. One problem they identify with UKI is a lack of a central CHID database which we hope to be able to contribute to by adopting the same format.
The stubble source code is available on github. Pre-built binary packages of stubble and a kernel package utilizing stubble to ship a combined kernel + stub + device trees binary are available in the ~tobhe/hamoa PPA on Launchpad.