What's up with Ubuntu mainline builds?

You might have been wondering lately what’s up with Ubuntu mainline builds [1] and why a lot of the recent 6.x builds failed. Well, in the 6.0 time frame we moved some checker scripts around [2] and changed how we handle kernel configs. Away from split config files towards a single annotations file [3]. This is still work in progress and we’ll keep making changes as we go along.

An unexpected side-effect of this is that it broke mainline builds :frowning: Some kernels still built properly - by sheer luck - but the majority failed due to config errors.

To give some context: When building kernel packages, there are a couple of checks that make sure we don’t release potentially broken kernels. Amongst other things, we check that kernel configs are as expected and that we don’t accidentally break the kernel ABI. This is not relevant for mainline builds, so these checks are disabled - theoretically. The code that generates mainline builds as well as the packaging bits have obviously changed over time and so have these checks. The build scripts disable the checks in two ways: 1) replace the checker scripts with empty shell scripts and 2) set a do_mainline_build build flag that is honored by the build scripts/rules to ignore some checks. With the above mentioned changes, we moved the checker scripts so they didn’t get stubbed out by the mainline build scripts and we introduced new config handling tools that ignored the do_mainline_build flag… :flushed:

Long story short, it’s all fixed and hopefully in a more robust way than before. Be aware though that we don’t actively monitor mainline builds so might not notice immediately when builds fail. In that case, don’t hesitate to notify us on IRC [4] or via email [5]. Or do so for any kernel related question/comment/suggestion/…

…Juerg

[1] Index of /mainline
[2] ~ubuntu-kernel/ubuntu/+source/linux/+git/lunar - [no description]
[3] ~ubuntu-kernel/ubuntu/+source/linux/+git/lunar - [no description]
[4] irc.libera.chat - channel #ubuntu-kernel
[5] kernel-team@lists.ubuntu.com

9 Likes

$ mainline --install-latest
mainline 1.1.9
Distribution: Linux Mint 21.1
Architecture: amd64
Running kernel: 6.2.8-060208-generic
Updating from: ‘https://kernel.ubuntu.com/~kernel-ppa/mainline/
OK
Found installed : 6.2-6ubuntu1~jammy
Found installed : 6.2.5-060205.202303110831
Found installed : 6.2.8-060208.202303220943
Latest point update: 6.2.9

Install Kernel Version 6.2.9 ? (y/n):
y

** (process:50799): CRITICAL **: 12:31:22.174: download_item_construct: assertion ‘cksum != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_task_add_to_queue: assertion ‘item != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_item_construct: assertion ‘cksum != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_task_add_to_queue: assertion ‘item != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_item_construct: assertion ‘cksum != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_task_add_to_queue: assertion ‘item != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_item_construct: assertion ‘cksum != NULL’ failed

** (process:50799): CRITICAL **: 12:31:22.174: download_task_add_to_queue: assertion ‘item != NULL’ failed
Downloading 6.2.9

This solved the problem for me:

Known Issue in 1.1.9 : assertion ‘cksum != NULL’

Work-Around: Settings -> enable Verify Checksums
As soon as the ppa updates to 1.1.11 the problem will be gone.

Why hasn’t Mainline put up the 6.3.8 or 6.3.9 kernels?

For that matter, where is the 6.4 kernel that was published yesterday?

Thanks for the polite inquiry. There’s is an issue with processing new tags that I need to look into.

4 Likes

Hello, is the processing new tags issue still what keeps new builds from showing? Or is it something else as I see 6.3.10, 6.3.11 and 6.4.1 are missing?

1 Like

I see that the newer builds are now available, all up to 6.3.12 and 6.4.2, but they (6.3.10-6.3.12, 6.4.1 and 6.4.2) appear as invalid in the Ubuntu Mainline Kernel Installer.

They should be all there by now. I don’t know what Ubuntu Mainline Kernel Installer is nor how it works but note that the existence of the main directory (for example ~kernel-ppa/mainline/v6.4.1/) is only the first step of the build process which will then trigger the actual builds of the binary packages.

I should probably add an overall build status that indicates the build status/progress of the whole operation. And also some PASS/FAIL logic to better detect missing/bad builds.

1 Like

Hey @juergh, thank you for the clarifications. When I first checked, the tests were not yet ran for the builds, but after the tests finished (successfully) the builds were also available in the kernel installer app I’m using. It’s nice to have a bit more context on what is going on. It would be great to have the build status easily available as well.

Index of /mainline is broken again. Last kernels that built the amd64 kernels was Index of /mainline/v6.10-rc2

1 Like

I noticed the same thing. I compiled it myself but sadly the nvidia driver is not working yet :frowning:

Half the time, if the amd64 builds are actually there, they don’t have Nvidia driver support until at least a month later.

Sometimes, you can upgrade to a later Nvidia driver release to find Nvidia driver support.

I had to move from Nvidia 535 drivers to the 555 drivers to get the 6.10-rc2 kernel to build properly.

And sometimes, that is all for naught because the mainline ppa maintainers move to the next GLIBC version meaning that you are never going to get LTS release support for the later kernels.

Is there a way to fix 6.10 amd64 build? Its been a week already and its still not working correctly

It’s broken, we know. We’re under-staffed and it’s summer vacation time so please be patient.

4 Likes

Thanks, glad to hear you know about the issue.

Thanks for the fix, I’ve already managed to update my kernel to 6.10.2

Also kernel 6.11 RC1 is available

corrado@corrado-n02-oo-0718:~$ inxi -SCxc
System:
  Host: corrado-n02-oo-0718 Kernel: 6.11.0-061100rc1-generic arch: x86_64
    bits: 64 compiler: gcc v: 13.3.0
  Desktop: GNOME v: 46.3.1 Distro: Ubuntu 24.10 (Oracular Oriole)
CPU:
  Info: quad core model: 12th Gen Intel Core i3-12100 bits: 64 type: MT MCP
    arch: Alder Lake rev: 5 cache: L1: 320 KiB L2: 5 MiB L3: 12 MiB
  Speed (MHz): avg: 800 min/max: 800/4300 cores: 1: 800 2: 800 3: 800 4: 800
    5: 800 6: 800 7: 800 8: 800 bogomips: 52838
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
corrado@corrado-n02-oo-0718:~$

same issue on kubuntu

  1. Ubuntu mainline builds are unsupported. They don’t receive security updates. They’re unsupported. You should only use them for testing/debugging. They’re unsupported.
  2. 6.2 is very, very old. You should upgrade.
  3. You should report this problem against whatever tool you’re using to install/upgrade Ubuntu mainline builds. 6.2.9 does not exist, it failed to build: Index of /mainline/v6.2.9
1 Like

For a very, very long time, I have wished we could make this unsupported much more :rotating_light: obvious :rotating_light: to users.

Maybe we should create a separate ‘Mainline Kernel Support’ category here and block the entire Ubuntu Kernel team from accessing it so they don’t get distracted. I honestly think this idea has legs :smiley: Maybe add a Chatty Jeeps bot in there for “fun.”

Here are my not-at-all serious and increasingly deranged suggestions to inform and discourage the use of the mainline kernel; don’t @ me.

  • While running mainline, your machine shows an enormous [-U-N-S-U-P-P-O-R-T-E-D-] banner on every other line of the system journal

  • Also, the system only boots in French :fr: (gonna be honest, that will be fine for many of our lovely French amis (Bonsoir @jibel @seb128 @didrocks :heart: - Vive la révolution!)

  • Mainline kernel builds will only be made available to paying :moneybag:Ubuntu Pro :money_mouth_face: subscribers (not gonna lie; this one isn’t a completely terrible idea, and I am 42% certain someone at Canonical has probably already suggested it at a Product Sprint :innocent: gotta pay those kernel developer salaries somehow!)

  • Alternatively, the mainline is only available under a new free-tier “Ubuntu Amateur” account that expires after 30 days a year a random period. Once it expires, you get thrown back to Linux Kernel 2.6.8, which (fact fans) shipped with Ubuntu 4.10 (Warty Warthog)

  • Mainline kernel users can install any font they like, but if they take a screenshot, the font gets in-line replaced with MS Comic Sans. Beautiful :kiss:

  • The kernel introduces an exponentially larger boot delay every day. It’s an almost imperceptible one Millisecond on the first day. But like a grain of rice on a chessboard, after 64 days, it will take an extra 9,223,372,036,854,775.808 seconds to boot

  • If you ask support questions in any official location, your avatar gets changed to a :clown_face: for a week

  • If you’re using anything other than Ubuntu (like Zorin, Mint, or PopOS) with a mainline kernel, in the background, slowly, non-interactively, a few packages a day are migrated back to Ubuntu, starting with snapd, and ending with linux-image-generic, getting you right back on track :+1:

  • While running the mainline kernel, the display is rotated slightly, maybe 3 degrees. Not 90, not 180, but an irritating small amount that will cause some text to be slightly off-screen and annoy people. Like this:

  • If the above is implemented, while you install the mainline debs, the desktop should quietly play You’re the Fool by The Three Degrees, but like really, super quietly, so you wonder where it’s coming from. Subliminal messaging.

  • Your IP address gets blocked from accessing the mainline kernel server, every day except your birthday. Send us a photocopy of your government-issued ID to enable this feature.

Further suggestions are very welcome.
Support questions, less so.

Humour: the quality of being amusing or comic, especially as expressed in literature or speech.

(please don’t send copies of your government-issued ID)

3 Likes