Kernel 'one-and-done' current workflow requested

I suspect there are many experienced kernel and other developers who like me, aren’t expert in Canonical/Ubuntu’s ways of packaging, who on rare occasion need to patch a bug in the particular kernel on the particular Ubuntu machine/arch/flavor they’re using.

The ‘official’ online resources canonical publishes are so out of date they give wrong instructions (they reference the ‘disco’ version of some years ago, specify command options no longer supported in the current releases, or talk about the many possible variants on git, etc.)

What’s needed, all in one place and current, is easy to describe:

  1. How to get the source for the kernel they’re running,
  2. How to add a short string to the packagename/changelog.
  3. The cycle of how compile our .c/.h file fixes/changes into a suite .debs that are installable locally.
  4. Optional – a way to create a patch file against the currently running default source to enable easy sharing to upstream of any results.

There are lots of us out there who have to deal with the kernel among thousands of other lines of app and embedded code-- it would make using Ubuntu a lot more attractive if there was a maintained page that spells out everything needed for ‘one and done’ patches of the current LTS release.


I totally agree and understand the frustration. Our documentation is a mess :frowning: We’re in a difficult place at the moment with our tooling that is still work in progress and difficult to set up and use. I’m not sure how much value there is in trying to document that at this very moment. It will change, it will be outdated again, it might scare people off. On the other hand, the current situation is not acceptable either.

There are different ways of doing things which complicates documentation. Let me think about it and run through some scenarios.

1 Like

Documenting the tooling necessary to support multiple architectures across multiple named performance emphasis configurations across multiple ‘official’ releases – quite a challenge. Then managing ‘acceptance of patches’ and ‘testing’ – well, this isn’t anything near to that.

Maybe if there were just these three commands as ‘commands’ get maintained better than documentation.

$# Default: $(uname -r)

$cd ($uname -r)

#makes fresh key/cert pair for interim rev secure module signing.
#’.config’ set to official release version.
#“debian/rules <frobbaz> to edit config”

#Takes one optional argument ‘clean’


Then just ‘double pinky swear’ promise to make sure that works before releasing an LTS.

Even the slightest hint about how to build from a git clone/pull of a recent release would be welcome. We use 22.04/master. I’ve cobbled together a repeatable build with our local changes (a new flavour). But, 22.04/master-next is different and I’m not finding any good hints.

I like the change to move all the config into annotations, but I have no idea where to start for a build of amd64 generic, for example.


I suggest to start with the simplest scenario possible:

Rebuild of the running kernel

This would allow for simple patching, where the configuration does not need to be changed. And it would be a test for build readiness.

Of course, it already includes some challenges:

1. Fetching the kernel source

There are three ways: kernel_org, APT source, or git

I assume that kernel_org sources do not contain the specific support for building suitable Ubuntu packages, so this is not recommended for a start and an emergency measure if there is no other way to get the build done.

APT source is quite nice, because network load and disk space requirement are low. Of course, it would be nice to get some fixes uploaded, but there can be a more advanced git guide which perfectly adds the knowledge missing to do that.

After having slow progress with the APT source, I tried git, and it has occupied 3.6 GB of disk space! I had to create a branch to be able to track my changes, which is common, but maybe too much hassle for a start. Although my machine and internet connection are not so slow, it was a test of patience to clone the repository. An outdated guide tells several 100 MB as a requirement.

2. Install all prerequisites as preparation for the build

My actual list for Ubuntu 6.2.0-26.26~22.04.1-generic 6.2.13 on jammy/amd64 contains:

libncurses-dev gawk flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev debhelper binutils-dev python-is-python3 libiberty-dev pahole libreadline-dev llvm ubuntu-dev-tools libdw-dev systemtap-sdt-dev libunwind-dev libslang2-dev libzstd-dev libcap-dev libbabeltrace-ctf-dev openjdk-8-jdk libtraceevent-dev

and for Rust some more:

rustc cargo llvm-15 lld-15 clang-15 wasi-libc bindgen librust-core-foundation-dev rust-src

It would be nice to have a metapackage to install them all. Maybe it exists already. I admit that I have not built Ubuntu kernels for quite some time, and the outdated guides are contradictory and confusing. The Ubuntu kernel could provide the prerequisites, because it depends on the version. Even providing dependencies upstream would be possible, specified by the components which require it, and the distribution just would need to do the mapping from <name,version> to package. Of course, it could add distribution-specific dependencies.

3. Apply the actual configuration residing in /boot/$(uname -r)

The expectation is that the whole config is accepted in the build, and that there are no differences anymore. Let me show the progress after several hours of tweaking:

$ scripts/diffconfig /boot/config-6.2.0-26-generic  .config
 CC_VERSION_TEXT "x86_64-linux-gnu-gcc-11 (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0" -> "gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0"
 GCC_VERSION 110300 -> 110400
 PAHOLE_VERSION 122 -> 125

Having a static package configuration older than a fully updated system cannot be expected by the casual user, for security reasons alone. On build servers, it is another matter of course. If there would be an easy way to create a VM for that, it would help, but only for those who already use VMs (BIOS settings, guest OS installation and extra space required) and with preferably high automation.

The casual user usually does not need a specific kernel version for a start, the version of the running kernel would be the smallest version acceptable. One could be suggested which matches the system nicely. I understand that this might be wishful thinking.

A static baseline only used for the kernel build could be kept in the repository and installed by the metapackage for installing build prerequisites. I have seen the GCC_VERSION difference, but maybe this change is not affecting anything.

Usually, for testing the configuration can be stripped to the hardware available, which would make it easier by far to have an accepted configuration with no important differences. In my case, I could not do it, because the Presonus 1810, NOT 1810c, audio interface (194f:0106) makes my system freeze (Firefox not responding, …). Although the USB device is not supported by kernels upto the most recent one, 6.5rc6, it is not ignored. Quite the contrary.

4. Build steps

We all remember probably how it was in ancient times. If I remember correctly:

$ cp -a /boot/config-$(uname -r) .config
$ make oldconfig
$ make
$ make modules
$ make install

Most guides avoid the Ubuntu procedure and recommend to get sources from Debian packages can be created somehow anyway, surprisingly.

Many guides tell the following procedure:

$ debian/rules clean
$ debian/rules binary

When the configuration shall be applied, diversity starts.

I did two things:

4a. adapt header in debian.master/config/annotations:

ARCH: amd64
FLAVOUR: amd64-generic

And because I found a sister directory, debian.hwe-6.2/, with the same annotations header, I have changed it as well. I do not understand the difference between these two directories in relation to the build.

4b. set a suffix in debian.master/changelog (first line)

I appended +presonus1810 to mark the specific kernel.

A bit awkward to configure things like that, editing a changelog file at the beginning.
Usually a changelog has documentation purposes only, for development history sorted in chronological order.

4c. Build procedure

Although I did not want to change the configuration, somehow check-config always complains about differences.

First try:

$ cp /boot/config-6.2.0-26-generic .config
$ ./debian/scripts/misc/annotations --arch amd64 --flavour generic --import .config
$ LANG=C fakeroot debian/rules clean
No change in config, so I left out the following step.
#$ LANG=C fakeroot debian/rules updateconfigs
$ LANG=C fakeroot debian/rules binary

Second try:

$ ./debian/scripts/misc/annotations --arch amd64 --flavour generic --export
$ LANG=C fakeroot debian/rules clean
No change in config, so I left out the following step.
#$ LANG=C fakeroot debian/rules updateconfigs
$ LANG=C fakeroot debian/rules binary

Third try:

$ git clone git://
$ cd jammy
$ git checkout -b Ubuntu-hwe-6.2-6.2.0-26.26_22.04.1-presonus1810 Ubuntu-hwe-6.2-6.2.0-26.26_22.04.1
$ patch -p1 <../presonus_1810.patch
$ git commit -a -m "Replace device id for Presonus 1810c by id for older 1810 model"
$ patch -p1 <../build.patch
$ git commit -a -m "Preparation for build:
- add variant -presonus1810
- reduce archs to build to amd64"
$ chmod a+x debian/rules
$ chmod a+x debian/scripts/*
$ chmod a+x debian/scripts/misc/*
$ LANG=C fakeroot debian/rules clean
$ LANG=C fakeroot debian/rules binary

Is LANG=C necessary? Somehow strange that a locale setting is assumed to affect a kernel build.

Fourth try:

Like third try, but with steps from another guide:

LANG=C fakeroot debian/rules clean
LANG=C fakeroot debian/rules editconfigs do_enforce_all=false
LANG=C fakeroot debian/rules editconfigs do_enforce_all=false
LANG=C fakeroot debian/rules do_enforce_all=false binary

and answering ‘n’ for editconfigs both times

One guide mentions that still there is an enforcement done by annotations, and it recommends to do the following:

echo '' >debian.master/config/annotations

Fifth try:

Returning to APT source because there does not seem to be a difference concerning the build.

$ make olddefconfig
# using defaults found in /boot/config-6.2.0-26-generic
# configuration written to .config
Note: target olddefconfig is quite comfortable. I am glad to know about it now.
$ LANG=C fakeroot debian/rules clean
$ LANG=C fakeroot debian/rules binary

The question is, in which way the configuration from /boot/config-$(uname -r) is applied. I had problems with the annotations script probably because it read the default configuration. This could be detected. I would even go so far as to suggest that if the user does not configure the kernel, the default Ubuntu configuration is applied, and it is mentioned that it is done of course.

I am not a fan of the check-config steps so far because usually a maintainer does not change the configuration without a very good reason, and with git a personalized commit message for such a change is required. But checking for a non-configured kernel is a help for beginners because of the outdated and incomplete guides out there. Because of all the automation sometimes steps are left out which are still vital for a build.

Unfortunately, I am stuck in the process:

make[1]: Leaving directory '/[...]/debian/build/tools-perarch/tools/perf'
mv /[...]/debian/build/tools-perarch/tools/bpf/bpftool/vmlinux /[...]/debian/build/tools-perarch/vmlinux
mv: cannot stat '/[...]/debian/build/tools-perarch/tools/bpf/bpftool/vmlinux': No such file or directory
make: *** [debian/rules.d/ /[...]/debian/stamps/stamp-build-perarch] Error 1

Not the first problem with bpftool, unfortunately.

I have looked for the log file of the build, and could not find it, having grepped the whole directory with

$ find . -type f -exec fgrep -Il bpftool/vmlinux {} \;

What is shown is a top-level view only:

  CC      pmu-events/pmu-events.o
  LD      util/perf-in.o
  LD      perf-in.o
  LD      pmu-events/pmu-events-in.o
  LINK    perf

I wonder how to enable a full build log. In the early days I used the straightforward:

$ make 2>&1 | tee build.log

Maybe it is not helpful to have it and just a waste of space, just I have thought about it because I am stuck. mv cannot move a file, but I have not seen an error message before that and could not even see what happened.

I have written a script which uses rsync to remove all changes and applies the two patches to let me try the build again. But I know that git can help with it as well.

This is my personal experience so far, kernel rebuild not a quickie at all after a prolonged break.