What's up with Ubuntu mainline builds?

juergh · July 3, 2023, 6:24am

You might have been wondering lately what’s up with Ubuntu mainline builds [1] and why a lot of the recent 6.x builds failed. Well, in the 6.0 time frame we moved some checker scripts around [2] and changed how we handle kernel configs. Away from split config files towards a single annotations file [3]. This is still work in progress and we’ll keep making changes as we go along.

An unexpected side-effect of this is that it broke mainline builds Some kernels still built properly - by sheer luck - but the majority failed due to config errors.

To give some context: When building kernel packages, there are a couple of checks that make sure we don’t release potentially broken kernels. Amongst other things, we check that kernel configs are as expected and that we don’t accidentally break the kernel ABI. This is not relevant for mainline builds, so these checks are disabled - theoretically. The code that generates mainline builds as well as the packaging bits have obviously changed over time and so have these checks. The build scripts disable the checks in two ways: 1) replace the checker scripts with empty shell scripts and 2) set a do_mainline_build build flag that is honored by the build scripts/rules to ignore some checks. With the above mentioned changes, we moved the checker scripts so they didn’t get stubbed out by the mainline build scripts and we introduced new config handling tools that ignored the do_mainline_build flag…

Long story short, it’s all fixed and hopefully in a more robust way than before. Be aware though that we don’t actively monitor mainline builds so might not notice immediately when builds fail. In that case, don’t hesitate to notify us on IRC [4] or via email [5]. Or do so for any kernel related question/comment/suggestion/…

…Juerg

[1] https://kernel.ubuntu.com/~kernel-ppa/mainline/
[2] https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/lunar/commit/?id=f2c0b00ac49fff8b78142be661982fe790d6a309
[3] https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/lunar/commit/?id=6146e6e42d91559b0cec0fa86492c12403972d1e
[4] irc.libera.chat - channel #ubuntu-kernel
[5] kernel-team@lists.ubuntu.com

derrickchapman · April 2, 2023, 5:13pm

$ mainline --install-latest
mainline 1.1.9
Distribution: Linux Mint 21.1
Architecture: amd64
Running kernel: 6.2.8-060208-generic
Updating from: ‘https://kernel.ubuntu.com/~kernel-ppa/mainline/’
OK
Found installed : 6.2-6ubuntu1~jammy
Found installed : 6.2.5-060205.202303110831
Found installed : 6.2.8-060208.202303220943
Latest point update: 6.2.9

Install Kernel Version 6.2.9 ? (y/n):
y

** (process:50799): CRITICAL **: 12:31:22.174: download_item_construct: assertion ‘cksum != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_task_add_to_queue: assertion ‘item != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_item_construct: assertion ‘cksum != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_task_add_to_queue: assertion ‘item != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_item_construct: assertion ‘cksum != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_task_add_to_queue: assertion ‘item != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_item_construct: assertion ‘cksum != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_task_add_to_queue: assertion ‘item != NULL’ failed
Downloading 6.2.9

derrickchapman · April 2, 2023, 6:39pm

This solved the problem for me:

Known Issue in 1.1.9 : assertion ‘cksum != NULL’

Work-Around: Settings -> enable Verify Checksums
As soon as the ppa updates to 1.1.11 the problem will be gone.

keith-myers · June 26, 2023, 7:29pm

Why hasn’t Mainline put up the 6.3.8 or 6.3.9 kernels?

For that matter, where is the 6.4 kernel that was published yesterday?

juergh · June 27, 2023, 4:46am

Thanks for the polite inquiry. There’s is an issue with processing new tags that I need to look into.

smd88 · July 4, 2023, 9:11pm

Hello, is the processing new tags issue still what keeps new builds from showing? Or is it something else as I see 6.3.10, 6.3.11 and 6.4.1 are missing?

smd88 · July 7, 2023, 2:50pm

I see that the newer builds are now available, all up to 6.3.12 and 6.4.2, but they (6.3.10-6.3.12, 6.4.1 and 6.4.2) appear as invalid in the Ubuntu Mainline Kernel Installer.

juergh · July 8, 2023, 7:00am

They should be all there by now. I don’t know what Ubuntu Mainline Kernel Installer is nor how it works but note that the existence of the main directory (for example ~kernel-ppa/mainline/v6.4.1/) is only the first step of the build process which will then trigger the actual builds of the binary packages.

I should probably add an overall build status that indicates the build status/progress of the whole operation. And also some PASS/FAIL logic to better detect missing/bad builds.

smd88 · July 9, 2023, 5:34pm

Hey @juergh, thank you for the clarifications. When I first checked, the tests were not yet ran for the builds, but after the tests finished (successfully) the builds were also available in the kernel installer app I’m using. It’s nice to have a bit more context on what is going on. It would be great to have the build status easily available as well.