Kernel configuration in Ubuntu


Ubuntu provides a wide variety of kernels for many different architectures and flavours, such as generic, lowlatency, cloud kernels, kvm, etc.

Maintaining a separate .config for each of these kernels would be extremely challenging, Kconfig options need to always match the desired value in order to have a kernel functional for the scope that it was designed for.

Starting with Ubuntu 23.04 (Lunar Lobster) the kernel team introduced a new format of annotations: a JSON-like file to store all the kernel config options for the supported architectures and flavours.

Edit: This feature has been backported also to 22.04 (Jammy Jellyfish)

Quick start

  • Generate the Ubuntu kernel .config for a specific architecture & flavour (i.e., amd64, generic):
$ ./debian/scripts/misc/annotations \
  --arch amd64 --flavour generic --export > .config
  • Import your own kernel .config in the Ubuntu kernel for a specific architecture & flavour (i.e., amd64, generic):
$ ./debian/scripts/misc/annotations \
--arch amd64 --flavour generic --import .config
  • After a new config is imported run fakeroot debian/rules clean updateconfigs to update the dependencies, then you can review the actual changes doing a simple git diff.


Ubuntu kernel annotations

There is a main annotations file for each Ubuntu kernel in: debian.<kernel_name>/config/annotations

The main annotations file can include other annotations (recursively).

The annotations has 4 different sections:

  • a header (that defines the format version, the list of supported architectures, the list of supported flavours paired with each architecture and a list of flavour inheritance rules)

  • a list of includes (annotations can include other annotations files to create a hierarchy of configs with overrides)

  • a subset of mandatory config options that have a note associated (typically config options that must be enforced for certain reasons, the note can be simply a reference to a tracking bug, explaining why the specific config needs to be set to a certain value, or just a description, like security implications for that config value, etc.)

  • a subset of config options that are either generated as dependency of the previous subset of configs or configs that have been set using the default Ubuntu policy (enable everything as module when possible, except for debugging, testing or deprecated features), so they don’t need an explicit note.

The annotations format looks like the following:

# Menu: HEADER
# ARCH: <ARCH_1> <ARCH_2> <ARCH_3> ... <ARCH_N>

include <FILE_1>
include <FILE_2>
include <FILE_N>

CONFIG_<FOO>                                  policy<{'<ARCH_1>[-<FLAVOUR_1>]': '<VALUE_1_1>', ..., '<ARCH_N>[-<FLAVOUR_M>]': '<VALUE_N_M>'}>
CONFIG_<FOO>                                  note<'<DESCRIPTION>'>

# ---- Annotations without notes ----

CONFIG_<BAR>                                  policy<{'<ARCH_1>[-<FLAVOUR_1>]': '<VALUE_1_1>', ..., '<ARCH_N>[-<FLAVOUR_M>]': '<VALUE_N_M>'}>

  • ARCH: contains the list of comma-separated values that defines all the supported architectures for this kernel

  • FLAVOUR: contains the list of comma-separated values that defines the list of flavours associated to each supported architecture

  • FLAVOUR_DEP: contains a mapping of ARCH-FLAVOUR pairs mapped to a parent ARCH-FLAVOUR pair; this can be used to define flavours that derives from other flavours, for example amd64-lowlatency derives from amd64-generic (then the local annotations file will only contain an include rule of the generic kernel plus the small subset of config options to override the defaults from amd64-generic)

  • FILE_1 … FILE_N: are the other included annotations (e.g., used by derivative kernels that want to include the annotations from the kernel they derive from)

  • FOO, BAR: kernel config options with the associated values across architectures and flavours, represented by the policy<...> definition

A simple example of an annotations file is the following (from the lunar/linux-lowlatency kernel, that derives from the generic kernel lunar/linux):

# Menu: HEADER
# ARCH: amd64 arm64
# FLAVOUR: amd64-lowlatency arm64-lowlatency arm64-lowlatency-64k
# FLAVOUR_DEP: {'amd64-lowlatency': 'amd64-generic', 'arm64-lowlatency': 'arm64-generic', 'arm64-lowlatency-64k': 'arm64-generic-64k'}

include "../../debian.master/config/annotations"

CONFIG_HZ_1000                                  policy<{'amd64': 'y', 'arm64': 'y'}>
CONFIG_HZ_1000                                  note<'HZ for lowlatency must be set to 1000 to provide better system responsiveness'>

CONFIG_HZ_250                                   policy<{'amd64': 'n', 'arm64': 'n'}>
CONFIG_HZ_250                                   note<'Override default HZ used in generic'>

CONFIG_LATENCYTOP                               policy<{'amd64': 'y', 'arm64': 'y'}>
CONFIG_LATENCYTOP                               note<', LP#1655986'>

CONFIG_PREEMPT                                  policy<{'amd64': 'y', 'arm64': 'y'}>
CONFIG_PREEMPT                                  note<'Enable fully preemptible kernel'>

CONFIG_PREEMPT_VOLUNTARY                        policy<{'amd64': 'n', 'arm64': 'n'}>
CONFIG_PREEMPT_VOLUNTARY                        note<'Disable voluntary preemption model'>

# ---- Annotations without notes ----

CONFIG_HZ                                       policy<{'amd64': '1000', 'arm64': '1000'}>

Managing the annotations file

To help the management of annotations each Ubuntu kernel provides a helper script in debian/scripts/misc/annotations (use --help for an overview of the supported actions).

In the following examples we are going to define an alias for this script

$ alias annotations='./debian/scripts/misc/annotations'

Here some typical examples of what you can do with the annotations script:

  • Show settings for CONFIG_DEBUG_INFO_BTF for master kernel across all the supported architectures and flavours:
$ annotations --query --config CONFIG_DEBUG_INFO_BTF
    "policy": {
        "amd64": "y",
        "arm64": "y",
        "armhf": "n",
        "ppc64el": "y",
        "riscv64": "y",
        "s390x": "y"
    "note": "'Needs newer pahole for armhf'"
  • Dump kernel .config for arm64 and flavour generic-64k:
$ annotations --arch arm64 --flavour generic-64k --export
  • Update annotations file with a new kernel .config for amd64 flavour generic:
$ annotations --arch amd64 --flavour generic --import .config
  • Enable CONFIG_PROVE_LOCKING on amd64 for flavour generic:
$ annotations -c CONFIG_PROVE_LOCKING --arch amd64 --flavour generic --write y
        "policy": {
            "amd64": "y",
            "arm64": "n",
            "armhf": "n",
            "ppc64el": "n",
            "riscv64": "n",
            "s390x": "n"
$ git diff
diff --git a/debian.master/config/annotations b/debian.master/config/annotations
index bc466d731815..301be4f9aaec 100644
--- a/debian.master/config/annotations
+++ b/debian.master/config/annotations
@@ -9884,7 +9884,7 @@ CONFIG_PROFILE_ALL_BRANCHES                     policy<{'riscv64': 'n'}>
 CONFIG_PROFILE_ANNOTATED_BRANCHES               policy<{'amd64': 'n', 'arm64': 'n', 'armhf': 'n', 'ppc64el': 'n', 'riscv64': 'n', 's390x': 'n'}>
 CONFIG_PROFILING                                policy<{'amd64': 'y', 'arm64': 'y', 'armhf': 'y', 'ppc64el': 'y', 'riscv64': 'y', 's390x': 'y'}>
 CONFIG_PROTECTED_VIRTUALIZATION_GUEST           policy<{'s390x': 'y'}>
-CONFIG_PROVE_LOCKING                            policy<{'amd64': 'n', 'arm64': 'n', 'armhf': 'n', 'ppc64el': 'n', 'riscv64': 'n', 's390x': 'n'}>
+CONFIG_PROVE_LOCKING                            policy<{'amd64': 'y', 'arm64': 'n', 'armhf': 'n', 'ppc64el': 'n', 'riscv64': 'n', 's390x': 'n'}>
 CONFIG_PROVIDE_OHCI1394_DMA_INIT                policy<{'amd64': 'n'}>
 CONFIG_PRU_REMOTEPROC                           policy<{'arm64': 'm', 'armhf': 'm'}>
 CONFIG_PSAMPLE                                  policy<{'amd64': 'm', 'arm64': 'm', 'armhf': 'm', 'ppc64el': 'm', 'riscv64': 'm', 's390x': 'm'}>

Every time the annotations file is changed we need to make sure that all the dependent config options are refreshed.

To do so we need to run the following command:

$ fakeroot debian/rules clean updateconfigs

This command will take care of interfacing with the kernel Kconfig subsystem and it’ll interactively ask the value for all the potentially new enabled dependent options.


The annotations format introduced in Lunar provides an easier and more efficient way of managing kernel .config’s across the large variety of kernels, architectures and flavours that Ubuntu supports.

If you want to submit a patch to that needs to change the kernel .config, or simply if you want to recompile the Ubuntu kernel using a different .config you just need to modify the annotations file and then you can simply upload your custom modified kernel to a ppa to test it, or you can recompile it locally using the typical Ubuntu way of rebuilding packages.


I like this change. My problem now is that I don’t know how to build a kernel from a clone/pull of a recent 22.04/master-next. :frowning:

Could the "Kconfig mentioned be configured for the rest of the flavors?

There shouldn’t be any change in how to build the kernel because of the move to annotations. You need to build in a 22.04 chroot or on a 22.04 system and obviously have all the build dependencies installed.

Unfortunately, the whole Kconfig story is now architecture/compiler-dependent so you either need to have all the cross-compilers installed or disable unwanted architectures and flavors. If you’re only interested in certain architectures/flavors, modify the annotations header accordingly. I.e., if you only want amd64-generic:

$ head -5 debian.master/config/annotations 
# Menu: HEADER
# ARCH: amd64
# FLAVOUR: amd64-generic

Append a version suffix to the package version in debian.master/changelog. Make sure not to change the version numbers or the build might fail. Something like:

$ head -1 debian.master/changelog
linux (5.15.0-80.90+test1) UNRELEASED; urgency=medium

Then run updateconfigs to clean up the annotations:

$ fakeroot ./debian/rules clean
$ ./debian/rules updateconfigs

And finally build the (generic) kernel packages:

$ fakeroot ./debian/rules binary-generic

I gave it a try. Started with a recent clone of 22.04/master-next.

$ fakeroot debian/rules clean
dh_clean: Please specify the compatibility level in debian/compat

Put something in debian/compat. Clean worked.

$ echo 9 > debian/compat
$ fakeroot debian/rules clean
$ fakeroot debian/rules updateconfigs
* Run config-check for amd64-generic ...
check-config: loading annotations from debian.master/config/annotations
check-config: FAIL: (- != y): CONFIG_AS_TPAUSE policy<{'amd64': 'y'}>)
check-config: FAIL: (- != n): CONFIG_ZERO_CALL_USED_REGS policy<{'amd64': 'n', 'arm64': 'n', 'armhf': 'n', 'ppc64el': 'n', 's390x': 'n'}>)
check-config: 9591/9602 checks passed -- exit 1
ERROR: 1 config-check failures detected

Importing all configurations ...

* Import configs for amd64-generic ...
debian/rules.d/ recipe for target 'updateconfigs' failed
make: *** [updateconfigs] Error 1

Update configs failed. Maybe generate configs?

$ fakeroot debian/rules genconfigs
check-config: 9602/9602 checks passed -- exit 0

Yay! This is the farthest I’ve gotten.
And the build is running.

Thanks. Thanks.

This is where I was stuck. I didn’t understand enough to remove all the arch and flavours.

With this change and debian/compat, I’m on my way.
Next step: move my flavour into annotations. Looks like there are tools to help with that.
Thanks again.

You shouldn’t have to fiddle with debian/compat and updateconfigs should not report any failures. Are you running this on recent Jammy?

Also, genconfigs doesn’t help you here, it just creates kernel config files in CONFIGS/.

You can ignore the failure from updateconfigs. It’s just a sanity check for our build process to fail on unexpected changes. If you run updateconfigs a second time it will pass.

I’m building on a 18.04 machine. That might be the reason for debian/compat.

Thanks for the information about genconfigs and updateconfigs. Confirmed what you said.

I did get an error at the end of the kernel build.
BTFIDS vmlinux
FAILED: load BTF from vmlinux: No such file or directory

Not sure what is going on here.

Compiler too old? Kernel too new? Just create a Jammy chroot and use that:

$ sudo apt install ubuntu-dev-tools
$ mk-sbuild jammy

Log out and log back in as instructed and rerun the command:

$ mk-sbuild jammy

After the chroot is created, allow the use of /home in the chroot(s):

$ echo "/home   /home   none   rw,bind   0   0" | sudo tee -a /etc/schroot/sbuild/fstab

Now create a chroot session so that you don’t mess up your base chroot, enter it as root and install whatever build dependencies you need:

$ schroot -b -n jammy-amd64-session -c jammy-amd64
$ schroot -u root -r -c jammy-amd64-session
(jammy-amd64) $ apt update
(jammy-amd64) $ apt-get build-dep linux
(jammy-amd64) $ exit

Enter the chroot session as regular user and build your kernel:

$ schroot -r -c jammy-amd64-session
(jammy-amd64) $ fakeroot ./debian/rules clean
(jammy-amd64) $ ./debian/rules updateconfigs

If you need to install more packages later on, just enter the chroot session with -u root. Or install and configure sudo.

You can also install your build dependencies (and any other packages you want) directly into the base chroot rather than the session so that every new session has them available already.

If you want to start clean, you just throw away your session and create a new one.

$ schroot -e -c jammy-amd64-session
$ schroot -b -n new-jammy-amd64-session -c jammy-amd64
1 Like

I’m getting the exact same compilation error. The only way I can compile a Jammy 6.2 kernel is through the sbuild process, but that doesn’t pass my kernel patches (probably a knowledge deficit on my end). Has anyone figured this out?

You are supposed to build the Jammy 6.2 kernel in a Jammy environment with all the proper build dependencies installed. The BTF error can happen if you are missing some packages, or if you are using different versions of some tools (compiler, pahole, etc.) that are not the expected ones.

If you are on a 18.04 environment I would recommend to create a Jammy chroot and build the kernel inside that chroot, as described up above.

I’m trying to get my own flavour to work with annotations.
I’m building Jammy master-next in a u22 chroot.
I’ve made changes to the kernel source to add my flavour. New files. Updated files. Kconfig changes, etc.
In debian.myflav/config/annotations, I have:

# Menu: HEADER
# ARCH: amd64
# FLAVOUR: amd64-generic
# FLAVOUR_DEP: {'amd64-myflav': 'amd64-generic'}

include "../../debian.master/config/annotations"

And I should put all the CONFIG_ things I want, in it as well, correct?

CONFIG_MYNEW                                      policy<{'amd64': 'y'}>
CONFIG_OLD                                        policy<{'amd64': 'n'}>
CONFIG_ETC                                        policy<{'amd64': 'y'}>

Do I need to copy the rest of the debian.master directory into debian.myflav?

And, building like this will work?

fakeroot debian/rules -j5 binary-myflav

And, I don’t need anything else?

I think this line:

# FLAVOUR: amd64-generic

Should be:

# FLAVOUR: amd64-myflav

Then everything else looks good. If you have the include line you don’t need to copy the entire annotations from debian.master.

Oops. Good catch. I missed that from the example above.

Do I need to copy any other files from debian.master into debian.myflav?
Or is it enough to have just the new annotations file in debian.myflav/config?

It’s enough to have your new annotations file. If you include the one from master you can simply define the “delta” in your file and you should be fine.

It seems like the new CONFIG_ lines I’ve added to myflav annotations are not getting automatically used. CONFIG_ lines in myflav annotation which change existing values are used, I think.
The build prompts me for anything new:

My new thing (THING) [M/n/?] (NEW) :
My other thing (THING2) [N/m/?] (NEW) :

But doesn’t prompt for changed values.

Do I need an additional step before building? This?

fakeroot ./debian/rules updateconfigs

Do I need to add the new CONFIG_ lines to the master annotations?

Ideas are welcome.

A follow up.
I looked at the .config created by the build and it did not contain any of the CONFIG_ values in myflav annotations.
Something is missing. It’s likely my problem and I’ll keep looking.

Could it be that I’m building Jammy master-next? Do I need to move up to a newer version?

I think I need amd64-myflav in the # FLAVOURS line for debian.master/config/annotations.

You don’t need to touch debian.master/config/annotations, you should be able to just add the new configs to your annotations. If you can push your branch somewhere I can take a look.

Unfortunately, I can’t push it. Proprietary code, etc.

I really seems like

fakeroot debian/build binary-myflav

is completely ignoring debian.myflav/config/annotations
And, it prompts for the new configs.

The only change I’ve made is to add the new annotations file for myflav. I feel like I’m missing some step prior to doing the build.

Is there a way to confirm that the build is actually using the myflav annotations?
I changed FORMAT to ‘6’ in the annotations files and did a build. It seems to ignore it.
It seems to prompt for the new configs before it even looks at the annotations.

So, I added just new configs to the build.master annnotations and did a build. The build didn’t prompt for the new configs, so that’s good. It’s still running. I’m not convinced that it will actually build binary-myflav.

So, the build completed without errors but did not create the deb files.

I’ve made two changes: added debian.myflav/config/annotations, and edited debian.master/config/annotations to add FLAVOUR binary-myflav and my new CONFIGs.

I’m using Jammy master-next.
I’ve applied my kernel changes to the sources.

I see only two possibilities: annotations as we’ve discussed are not fully supported in Jammy master-next, or I’ve corrupted my source tree in some way. I’ll keep looking.

Suggestions are welcome.