What's up with Ubuntu mainline builds? (2023)

tenplus1 · February 3, 2025, 3:20pm

Today a slew of 6.12.x builds and the new 6.13 builds were added to the mainline kernel list but no downloadable files for any users to test them out, shame!

dsmythies · February 6, 2025, 3:17pm

I see lots of activity on the mainline PPA page, but most builds are still failing.
My, and I know for many others, bookmarked link for the mainline PPA is:

https://kernel.ubuntu.com/mainline/?C=N;O=D

So, typically, I would never see your new pseudo files:

1_ANY_PRODUCTION_USE_DISCOURAGED_AND_AT_OWN_RISK
0_For_debugging_purposes_only

Suggest repeating those two files with the names:

z_ANY_PRODUCTION_USE_DISCOURAGED_AND_AT_OWN_RISK
z_For_debugging_purposes_only

I compile the mainline kernel myself, but I like to keep synchronized with the Ubuntu kernel configuration, which I obtain via installing the compiled kernel from the Ubuntu PPA. Given the ongoing issues with the Ubuntu mainline PPA is there another way to obtain Ubuntu’s kernel configuration? I.E. for kernel 6.14-rc1? Right now I am just using default kernel changes since the last successful Ubuntu mainline compile for AMD64, 6.13-rc1.

juergh · February 6, 2025, 8:10pm

Bits are slowly trickling in as the builders are grinding through the backlog. Once the queue is empty, I’ll retrigger some of the newer failed builds.

FWIW, the kernel dashboard shows some mainline builds status. For example: RabbitMQ: ps5/pks/rabbitmq → Queue: mainline:mainline-todo.x86 - The number of messages is the number of queued build jobs.

juergh · February 10, 2025, 2:33pm

FWIW, this dashboard should now update hourly and give a better overview of where things stand.

keith-myers · March 31, 2025, 2:56am

Something wrong with the Mainline Kernel PPA downloads over the last couple of days.
Won’t download the latest 6.13.9 kernel. Looked at the build logs and there aren’t any errors noted with the CI builds.

As long as the CI builds were successful I’ve never had an issue downloading and installing the new kernels so far.

Never seen this message in the Mainline installer before. I was simply trying to upgrade from the 6.13.8 kernel I’m running to the new 6.13.9 kernel released and available in the installer.

mainline 1.4.10
install 6.13.9
6.13.9 download failed
Install: no installable kernels specified
mainline: done

Anyone have an explanation of what is failing and what the “no installable kernels specified” means?

mpellizzer · March 31, 2025, 7:35am

From mainline dasboard it is possible to see that 6.13.9 debian packages have been built correctly. It may be a problem on the mainline tool’s side. I suggest you to open an issue in the mainline tool github.

juergh · March 31, 2025, 9:17am

Someone reported seeing intermittent issues when trying to access builds. Could be a problem with our internal infrastructure

juergh · March 31, 2025, 9:18am

Sorry we can’t help with that, that’s not an Ubuntu tool.

peci1 · April 16, 2025, 9:15am

There are some differences in how 6.14.1 and 6.14.2 are built and packaged (at least for amd64):

$ file 6.14.1/usr/src/linux-headers-6.14.1-061401-generic/scripts/genksyms
6.14.1/usr/src/linux-headers-6.14.1-061401-generic/scripts/genksyms: directory
$ ll 6.14.1/usr/src/linux-headers-6.14.1-061401-generic/scripts/genksyms/
celkem 360
drwxr-xr-x 2 peci1 peci1  4096 dub  7 12:48 ./
drwxr-xr-x 8 peci1 peci1 12288 dub  7 12:48 ../
-rwxr-xr-x 1 peci1 peci1 72624 dub  7 12:48 genksyms*
lrwxrwxrwx 1 peci1 peci1    64 dub 16 11:06 genksyms.c -> ../../../linux-headers-6.14.1-061401/scripts/genksyms/genksyms.c
lrwxrwxrwx 1 peci1 peci1    64 dub 16 11:06 genksyms.h -> ../../../linux-headers-6.14.1-061401/scripts/genksyms/genksyms.h
-rw-r--r-- 1 peci1 peci1 27200 dub  7 12:48 genksyms.o
lrwxrwxrwx 1 peci1 peci1    64 dub 16 11:06 keywords.c -> ../../../linux-headers-6.14.1-061401/scripts/genksyms/keywords.c
lrwxrwxrwx 1 peci1 peci1    59 dub 16 11:06 lex.l -> ../../../linux-headers-6.14.1-061401/scripts/genksyms/lex.l
-rw-r--r-- 1 peci1 peci1 59635 dub  7 12:48 lex.lex.c
-rw-r--r-- 1 peci1 peci1 38384 dub  7 12:48 lex.lex.o
lrwxrwxrwx 1 peci1 peci1    62 dub 16 11:06 Makefile -> ../../../linux-headers-6.14.1-061401/scripts/genksyms/Makefile
-rw-r--r-- 1 peci1 peci1 81578 dub  7 12:48 parse.tab.c
-rw-r--r-- 1 peci1 peci1  4995 dub  7 12:48 parse.tab.h
-rw-r--r-- 1 peci1 peci1 34336 dub  7 12:48 parse.tab.o
lrwxrwxrwx 1 peci1 peci1    61 dub 16 11:06 parse.y -> ../../../linux-headers-6.14.1-061401/scripts/genksyms/parse.y

versus

$ file 6.14.2/usr/src/linux-headers-6.14.2-061402-generic/scripts/genksyms
6.14.2/usr/src/linux-headers-6.14.2-061402-generic/scripts/genksyms: symbolic link to ../../linux-headers-6.14.2-061402/scripts/genksyms
$ ll 6.14.2/usr/src/linux-headers-6.14.2-061402-generic/scripts/genksyms/
celkem 80
drwxr-xr-x  2 peci1 peci1  4096 dub 10 15:48 ./
drwxr-xr-x 20 peci1 peci1 12288 dub 10 15:48 ../
-rw-r--r--  1 peci1 peci1 20969 dub 10 15:48 genksyms.c
-rw-r--r--  1 peci1 peci1  2178 dub 10 15:48 genksyms.h
-rw-r--r--  1 peci1 peci1    50 dub 10 15:48 .gitignore
-rw-r--r--  1 peci1 peci1  2251 dub 10 15:48 keywords.c
-rw-r--r--  1 peci1 peci1  8931 dub 10 15:48 lex.l
-rw-r--r--  1 peci1 peci1   359 dub 10 15:48 Makefile
-rw-r--r--  1 peci1 peci1 12037 dub 10 15:48 parse.y

So e.g. the genksyms binary is missing from 6.14.2 .

The changelog for 6.14.2 doesn’t contain the string genksyms. The amd64 build log for 6.14.2 doesn’t contain string genksyms at all (6.14.1 has 16 matches).

juergh · April 16, 2025, 2:57pm

Different builds use potentially different packaging bits, so differences are expected.

dsmythies · December 15, 2025, 11:11pm

After many months of the mainline builds always working fine (thank you very much), yesterdays AMD64 build of 6.19-rc1 failed. The relevant portion of the build log is:

Mainline 6.19-rc1 log extract:

  CC      drivers/gpu/drm/drm_displayid.o
error[E0432]: unresolved import `kernel::firmware`
  --> /home/kernel/COD/linux/drivers/gpu/nova-core/firmware.rs:10:5
   |
10 |     firmware,
   |     ^^^^^^^^ no `firmware` in the root
   |
   = help: consider importing this module instead:
           crate::firmware
note: found an item that was configured out
  --> /home/kernel/COD/linux/rust/kernel/lib.rs:97:9
   |
97 | pub mod firmware;
   |         ^^^^^^^^
note: the item is gated here
  --> /home/kernel/COD/linux/rust/kernel/lib.rs:96:1
   |
96 | #[cfg(CONFIG_RUST_FW_LOADER_ABSTRACTIONS)]
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  CC [M]  drivers/acpi/video_detect.o
error[E0432]: unresolved import `kernel::firmware`
  --> /home/kernel/COD/linux/drivers/gpu/nova-core/firmware/riscv.rs:10:5
   |
10 |     firmware::Firmware,
   |     ^^^^^^^^ could not find `firmware` in `kernel`
   |
note: found an item that was configured out
  --> /home/kernel/COD/linux/rust/kernel/lib.rs:97:9
   |
97 | pub mod firmware;
   |         ^^^^^^^^
note: the item is gated here
  --> /home/kernel/COD/linux/rust/kernel/lib.rs:96:1
   |
96 | #[cfg(CONFIG_RUST_FW_LOADER_ABSTRACTIONS)]
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  CC [M]  net/netfilter/xt_addrtype.o
error[E0433]: failed to resolve: could not find `module_firmware` in `kernel`
  --> /home/kernel/COD/linux/drivers/gpu/nova-core/nova_core.rs:33:9
   |
33 | kernel::module_firmware!(firmware::ModInfoBuilder);
   |         ^^^^^^^^^^^^^^^ could not find `module_firmware` in `kernel`

  CC [M]  fs/nfs/sysctl.o
  CC [M]  sound/soc/codecs/ak4375.o

The compile worked fine for me, using the config file from kernel 6.18-rc4 (my last mainline kernel compile) and accepting default changes, of which there were many (123).

juergh · December 16, 2025, 7:55am

Oh rust fun Well, it’s an -rc1 so build failures are not totally unexpected.

redwoodsec · December 16, 2025, 3:50pm

6.19 where are you………

keith-myers · January 1, 2026, 11:41pm

Or the 6.18 point releases for that matter. Auto CI build system has broken again like earlier in the year. Hoping that the CI maintainers take a look soon in this new year.

redwoodsec · January 1, 2026, 11:55pm

Yeah depends for 6.18 are broken now for a couple weeks it seems

juergh · January 5, 2026, 10:03am

This looks to be a bug in upstream that is exposed by our configs. Will look into it.

juergh · January 5, 2026, 10:46am

Needs this which will land eventually: https://lore.kernel.org/lkml/20251106-b4-select-rust-fw-v3-0-771172257755@nvidia.com/

The relevant commit is in linux-next so should show up in v6.19-rc5.

dsmythies · January 5, 2026, 10:42pm

Isn’t the commit you are referring to included already and since v6.19-rc1?

doug@s19:~/kernel/linux$ git show 9906efa545d1
commit 9906efa545d1d2cf25a614eeb219d3f8d5a302cd
Author: Alexandre Courbot <acourbot@nvidia.com>
Date:   Thu Nov 6 11:40:54 2025 +0900

    firmware_loader: make RUST_FW_LOADER_ABSTRACTIONS select FW_LOADER

doug@s19:~/kernel/linux$ git tag --contains 9906efa545d1d2cf25a614eeb219d3f8d5a302cd
v6.19-rc1
v6.19-rc2
v6.19-rc3
v6.19-rc4

EDIT: Oh, never mind. I think it is patch 2 of 3 that didn’t make it into the main branch back then (like why not?) So this one:

[PATCH v3 2/3] gpu: nova-core: select RUST_FW_LOADER_ABSTRACTIONS

doug@s19:~/kernel/linux$ git log --oneline | grep "nova-core: select RUST_FW_LOADER_ABSTRACTIONS"
doug@s19:~/kernel/linux$

juergh · January 6, 2026, 7:33am

https://lore.kernel.org/lkml/DEZE2MJFASEH.91JN02AU26VA@nvidia.com/ has the explanation.

dsmythies · January 9, 2026, 4:51pm

Ok, as of a few minutes ago the patch has finally made it into the mainline kernel. It seems it was missed last week by the maintainer. See here. And (after a git pull):

doug@s19:~/kernel/linux$ git log --oneline | grep “nova-core: select RUST_FW_LOADER_ABSTRACTIONS”
3d3352e73a55 gpu: nova-core: select RUST_FW_LOADER_ABSTRACTIONS

So the next Ubuntu mainline build is expected to work.

EDIT (The next day): The daily Ubuntu mainline compile worked fine.