Thanks for your pointers, let me try to study a bit more since I am not familiar with the schroot thing. Anyway, when I tried loading the kernel module from the sbuild (replacing the option.ko in the /lib/modules…/driver/…) , I am getting the following error:
kernel: BPF: [127570] STRUCT a…percpu
kernel: BPF: size=44 vlen=11
kernel: BPF:
kernel: BPF: Invalid name
kernel: BPF:
kernel: failed to validate module [option] BTF: -22
What is this BTF thing? Anything that I missed this time? modprobe --dump-modversions are identical. Why it seems like getting harder to do it with newer ubuntu versions 
BTF things is additional metadata created by pahole tool which enables additional kernel features used by all sophisticated sysadmins. It enables one to compile BTF programmes and load them into the kernel to dynamically tune the kernel and add trace points to monitor performance and collect telemetry. WIthout need to stop the kernel, recompile the kernel, or restart workloads.
Ubuntu kernels used for all variety of contexts, thus we enable as many features like that as possible to ensure Ubuntu can just be used out of the box without need to recompile it.
If you missbuild the kernel module it will not load. Which typically means using the same config as generated by debian/rules clean and using the toolchains as installed by apt build-dep ./ including the same gcc compiler, the same pahole, and the same make invocation as done by ./debian/rules binary as otherwise one can trivially produce incompatible modules with the given existing binary build of Ubuntu.
Yes, building kernels is hard - but this is also why Ubuntu kernel is so good for so many people, we did all the hard work. Which is documented in code and reproducible by using the ubuntu kernel build process via debian/rules entry points.
@nabilmiri your questions have drifted far away from being specifically about Ubuntu kernel by now, and are now actually very generic (it is coincidental that you are using Ubuntu kernel, and substituting any other kernel would produce the same results). I recommend you to search answers on askubuntu.com and unix.stackexchange.com as they are not at all unique or specific to the Ubuntu Kernels anymore. Good luck on your journey of learning how to compile and reproduce ABI/API compatible builds of Linux based software.
Quick question 
in Fedora/RHEL if I want to see if the bpf LSM is active/available in the kernel I can go here:
[root@virtualrocky]# cat /sys/kernel/security/lsm
lockdown,capability,yama,selinux,bpf[root@virtualrocky]#
but if I do the same thing in Ubuntu 22.0.4 bpf is NOT there:
root@virtual-ubuntu2204:/# cat /sys/kernel/security/lsm
lockdown,capability,landlock,yama,apparmorroot@virtual-ubuntu2204:/#
bpf is enabled in Ubuntu kernels right? Why is it not listed in /sys/kernel/security/lsm? where could i look to get the list of all available LSMs on a ubuntu kernel, including bpf?
Thank you!
@why2jjj BPF LSM is enabled but it’s not activated by default (it seems):
$ grep CONFIG_LSM= /boot/config-6.2.0-32-generic
CONFIG_LSM="landlock,lockdown,yama,integrity,apparmor"
You would have to activate it explicitly via a kernel commandline option:
lsm=apparmor,bpf,...
Maybe our defaults are wrong?
I do believe that the defaults are wrong and the automatic activation of bpf LSM upon install should behave the same as what RHEL/Red-Hat does. bpf LSM is becoming such a goto tool for performance and security measuring, only good can come out of having bpf LSM default to “activate” like RHEL does. That is at least one less uncomfortable conversation a sw company has with a potential customer why their server needs to be rebooted for the company’s SW solution to use a Linux driver.
Plus the consistency where all the Ubuntu kernel LSM modules are set to activate out-of-box eases the mind a bit.
I can file a feature request that an initial ubuntu install should have bpf LSM set to activate if you like me too…if u could point me to the the place to. write it up I’ll be very happy to do it.
Please open a bug on LP against the kernel and we’ll take it from there: https://bugs.launchpad.net/ubuntu/+source/linux/+filebug
Sorry to point this here but Bug #2033735 “No HDMI in Nvidia on-demand mode” : Bugs : linux package : Ubuntu .
Quite unusual since we already are in a (nearly) stable state of Mantic’s kernel and this bug is quite annoying too. It’s odd that I’m the only one subscribing this bug, so maybe I missed something or this bug is hardware-related.
Well, help!
I recently found this guide https://wiki.ubuntu.com/Kernel/Dev/KernelModuleRebuild that does exactly what I wanted but it doesn’t give the exact module symvers of the original module. Do you happen to know anything that looks like this but will work on the Jammy kernel?
Well, I just followed the guide here https://docs.kernel.org/kbuild/modules.html , simply copied out the specific .c and .h files for the kernel module from the kernel source, build it like a out-of-tree module, and I am able to build a working kernel module that has correct symvers and loads fine, what a surprise.
1 Like
What’s the right repository to add to get the debug symbols of the kernels published under ESM.
http://ddebs.ubuntu.com/ only includes debug symbols for kernels released outside of ESM.
https://esm.ubuntu.com/infra/ubuntu/dists/ includes the packages of the newer kernels (e.g., 4.15.0-216-generic under bionic) but the package lists in that repo don’t contain the debug symbols packages.
By default pro attach ${TOKEN} doesn’t add the debug symbols repository so I am at a loss on where to find them.
So I recently decided to jump ship to Ubuntu 23.10 beta just because of the fact that it has a kernel 6.5 kernel that helps my issues with Starfield not launching (would’ve just gone the PPA route, but it’s close enough to 23.10’s launch date I figured I’ll just jump ship
https://github.com/ValveSoftware/Proton/issues/7064#issuecomment-1745438028
Are there plans on working with Valve and the community working on Proton/Linux gaming to improve the latest supported kernels in Ubuntu to make sure that it works amazing with the latest games without needing to install PPAs? Or adding newer kernels without waiting for a new release, like Pop!_OS? IIRC, I remember the reason why this was never done is because of all the patches you do on Ubuntu, but I think it’s time to revisit that.
Newer kernels tend to resolve a lot of the issues Linux games experience, unless you’re Valve who customized the kernel for gaming purposes.
I debated on posting it here instead, but I figured since the update to the latest kernel solved my problems, I figure I’ll do it here.
Which release version did you choose to do the jump from?
For example, Jammy already has one kernel flavour available on x86 based off 6.5 and soon other flavours will be available too, like the hwe-22.04-edge and cloud kernel edge. Kernel team is currently focusing on resolving last minute regressions in the v6.5 kernel to ensure successful launch of 23.10. Once that is out the door, we will start rolling out more backports.
Unfortunately these are not currently trivially available, please contact Canonical support and they can share those with you on request.
This is largely due to lack of reprepro support for .ddeb files; and I have a plan to re-engineer ESM publishing to auto-rename .ddeb to .deb and publish it correctly. Separately Server team is working on making ESM .ddeb to be available in the dbgsymbols server.
1 Like
Does this mean Canonical support would only be able to provide them for specific kernels so I can’t automate their download whenever a new kernel is published?
For context, I use the debug symbols to properly support eBPF programs that use BTF. For this I have been able to use the debug symbols of the kernel packages to create a BTF file for specific kernel versions.
I’ve sent a message through the Ubuntu website, I hope that’s what you meant by “Canonical support”.
you don’t need debug symbols to use eBPF, because eBPF support is compiled into all of our kernels and should be available to be usable as is, without debug symbols. We had toolchains issues in the past that prevented that but it is expected for eBPF to “just work” without debug symbols. Can you please check if that is the case on i.e. jammy 22.04, and if it is not the case on the ESM kernels, please use ubuntu-bug linux and contact support to escalate fixing that. Or paste the bug number here.
It is expected for eBPF to work on Ubuntu without installing debug symbols.
edit: basically debug symbols are a huge download and ebpf information is tiny, and should be available from normal debs, sans bugs. Also a working example of the things you do with eBPF would be useful, as we can also add this to our certification test suite to ensure it always works and doesn’t regress again.
I may not have been super clear. eBPF is enabled and supported in the kernel, I am trying to create a BTF file so that eBPF programs written with CO-RE in mind can be used instead of only programs that are built specifically only for this kernel. In newer kernels a BTF file exists under /sys/kernel/btf/vmlinux and it is used by BPF loaders to relocate fields that the BPF programs use so that the program can be written once for any kernels and loaded anywhere even if the fields move because the loader will know where the field is at load time for the currently running kernel by using the vmlinux file.
In older kernels (such as 4.15.x) this BTF file does not exist so it needs to be created manually and given to the loader. BTF files can be created using debug symbols of that kernel and pahole once and then used on any machine with that same kernel. That is my goal. This has been working for 4.15.x kernels in Ubuntu 18 until it hit the ESM period because now newer kernels are not published without a license. I signed up for a PRO license so I can still have access to these kernels (that works) but pahole needs the debug symbols when creating the BTF file.
yes, but ideally we don’t want anybody running pahole manually, and we can and should run it during build and ship the relevant btf file with linux-image package straight away. This is a way too common requirement that shouldn’t need everyone to go through dbg package gymnastics.
It’d definitely be ideal if the BTF file was already in the linux image package. It looks like getting a support ticket requires using the paid PRO license, I only have the free PRO license since I only need one machine to create the BTF file so there isn’t a way for me to get the debug symbols packages. I’ll find another way to support running CO-RE ebpf programs in these kernels.
Thanks for the replies, at least now I know my google-fu wasn’t failing me when trying to find a repo for these ESM debug kernel packages.
1 Like
A public ubuntu-bug linux should work too. if you open one asking for BTF for ESM kernels i can escalate it from there. Or use https://bugs.launchpad.net/ubuntu/+source/linux/+filebug/?no-redirect