Kernel crash Lenovo QCNFA765 / ath11k_pci when attching to certain networks

Support and Help
1

This is a brand new Lenovo ThinkPad L16 laptop, with default WiFi configuration.

https://termbin.com/4xe3

The log above shows a working connection to my Unifi AP as the script snips it just after the failure.

If I try to connect to a HostAP served network on a Pi2Zero the kernel driver crashes on a loop.
From the UI point of view this looks like an incorrect PSK password and repeated prompts for a new one.

journlctl -f :

Dec 02 10:49:52 Wills-ThinkPad kernel: wlp1s0: associate with 2c:cf:67:9d:4f:32 (try 1/3)
Dec 02 10:49:52 Wills-ThinkPad kernel: wlp1s0: associate with 2c:cf:67:9d:4f:32 (try 2/3)
Dec 02 10:49:52 Wills-ThinkPad kernel: wlp1s0: associate with 2c:cf:67:9d:4f:32 (try 3/3)
Dec 02 10:49:52 Wills-ThinkPad kernel: wlp1s0: association with 2c:cf:67:9d:4f:32 timed out
Dec 02 10:49:52 Wills-ThinkPad kernel: ath11k_pci 0000:01:00.0: failed to lookup peer 2c:cf:67:9d:4f:32 on vdev 0
Dec 02 10:49:55 Wills-ThinkPad wpa_supplicant[1495]: wlp1s0: CTRL-EVENT-SSID-TEMP-DISABLED id=0 ssid="RoboCon2025-Team22" auth_failures=1 duration=10 reason=CONN_FAILED
Dec 02 10:49:55 Wills-ThinkPad kernel: ------------[ cut here ]------------
Dec 02 10:49:55 Wills-ThinkPad kernel: WARNING: CPU: 3 PID: 218096 at drivers/net/wireless/ath/ath11k/mac.c:7698 ath11k_mac_op_unassign_vif_chanctx+0x28f/0x2f0 [ath11k]
Dec 02 10:49:55 Wills-ThinkPad kernel: Modules linked in: uas usb_storage cdc_mbim cdc_wdm cdc_ncm cdc_ether usbnet mii usbhid typec_displayport ipheth apple_mfi_fastcharge ccm michael_mic rfcomm snd_seq_dummy snd_hrtimer snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device cmac algif_hash algif_skcipher af_alg bnep btusb uvcvideo btrtl videobuf2_vmalloc btintel uvc btbcm videobuf2_memops btmtk videobuf2_v4l2 binfmt_misc bluetooth videodev videobuf2_common ecdh_generic qrtr_mhi nls_iso8859_1 ecc mc intel_rapl_msr snd_soc_dmic snd_soc_acp6x_mach snd_acp6x_pdm_dma intel_rapl_common joydev amdgpu snd_sof_amd_acp63 edac_mce_amd snd_sof_amd_vangogh snd_sof_amd_rembrandt snd_sof_amd_renoir snd_sof_amd_acp snd_sof_pci snd_sof_xtensa_dsp kvm_amd snd_sof input_leds mac_hid snd_sof_utils kvm snd_soc_core irqbypass crct10dif_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 snd_compress sha1_ssse3 aesni_intel ac97_bus snd_pcm_dmaengine crypto_simd snd_ctl_led cryptd snd_pci_ps qrtr snd_rpl_pci_acp6x amdxcp
Dec 02 10:49:55 Wills-ThinkPad kernel:  snd_hda_codec_realtek snd_acp_pci drm_exec snd_acp_legacy_common snd_hda_codec_generic ath11k_pci gpu_sched rapl snd_hda_codec_hdmi ath11k drm_buddy snd_hda_intel drm_suballoc_helper drm_ttm_helper snd_intel_dspcfg snd_pci_acp6x think_lmi qmi_helpers snd_intel_sdw_acpi ttm snd_pci_acp5x snd_hda_codec firmware_attributes_class wmi_bmof k10temp mac80211 drm_display_helper snd_hda_core snd_rn_pci_acp3x cec snd_acp_config snd_soc_acpi snd_hwdep rc_core snd_pcm snd_pci_acp3x i2c_algo_bit cfg80211 i2c_piix4 snd_timer thinkpad_acpi libarc4 ccp mhi nvram amd_pmc acpi_tad serio_raw sch_fq_codel msr parport_pc ppdev lp parport efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 snd hid_multitouch soundcore r8169 nvme video xhci_pci hid_generic ucsi_acpi crc32_pclmul psmouse typec_ucsi thunderbolt nvme_core xhci_pci_renesas typec i2c_hid_acpi realtek ledtrig_audio nvme_auth platform_profile i2c_hid wmi hid
Dec 02 10:49:55 Wills-ThinkPad kernel: CPU: 3 PID: 218096 Comm: kworker/u24:3 Tainted: G        W          6.8.0-49-generic #49-Ubuntu
Dec 02 10:49:55 Wills-ThinkPad kernel: Hardware name: LENOVO 21L7CTO1WW/21L7CTO1WW, BIOS R2IET29W (1.29 ) 11/01/2024
Dec 02 10:49:55 Wills-ThinkPad kernel: Workqueue: events_unbound cfg80211_wiphy_work [cfg80211]
Dec 02 10:49:55 Wills-ThinkPad kernel: RIP: 0010:ath11k_mac_op_unassign_vif_chanctx+0x28f/0x2f0 [ath11k]
Dec 02 10:49:55 Wills-ThinkPad kernel: Code: fe ff ff 4c 89 e7 e8 30 90 ff ff 85 c0 0f 84 8d fe ff ff 49 8b 3c 24 89 c2 48 c7 c6 88 8d c0 c0 e8 76 b7 01 00 e9 76 fe ff ff <0f> 0b e9 c2 fd ff ff 4c 8d b3 f4 04 00 00 8b b3 30 04 00 00 4c 89
Dec 02 10:49:55 Wills-ThinkPad kernel: RSP: 0018:ffffb1278ec73b90 EFLAGS: 00010246
Dec 02 10:49:55 Wills-ThinkPad kernel: RAX: 0000000000000000 RBX: ffff9837767edc58 RCX: ffff98376ee4a358
Dec 02 10:49:55 Wills-ThinkPad kernel: RDX: 0000000000000000 RSI: ffff9837767edc58 RDI: 0000000000000000
Dec 02 10:49:55 Wills-ThinkPad kernel: RBP: ffffb1278ec73bc8 R08: 0000000000000000 R09: 0000000000000000
Dec 02 10:49:55 Wills-ThinkPad kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff98376c419fc0
Dec 02 10:49:55 Wills-ThinkPad kernel: R13: ffff983748dc0000 R14: 0000000000000000 R15: ffff98376c41d098
Dec 02 10:49:55 Wills-ThinkPad kernel: FS:  0000000000000000(0000) GS:ffff983e5ed80000(0000) knlGS:0000000000000000
Dec 02 10:49:55 Wills-ThinkPad kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 02 10:49:55 Wills-ThinkPad kernel: CR2: 000074bf1308b000 CR3: 00000001b783c000 CR4: 0000000000f50ef0
Dec 02 10:49:55 Wills-ThinkPad kernel: PKRU: 55555554
Dec 02 10:49:55 Wills-ThinkPad kernel: Call Trace:
Dec 02 10:49:55 Wills-ThinkPad kernel:  <TASK>
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? show_regs+0x6d/0x80
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? __warn+0x89/0x160
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? ath11k_mac_op_unassign_vif_chanctx+0x28f/0x2f0 [ath11k]
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? report_bug+0x17e/0x1b0
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? handle_bug+0x51/0xa0
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? exc_invalid_op+0x18/0x80
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? asm_exc_invalid_op+0x1b/0x20
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? ath11k_mac_op_unassign_vif_chanctx+0x28f/0x2f0 [ath11k]
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Dec 02 10:49:55 Wills-ThinkPad kernel:  drv_unassign_vif_chanctx+0x126/0x1f0 [mac80211]
Dec 02 10:49:55 Wills-ThinkPad kernel:  ieee80211_assign_link_chanctx+0x59/0x200 [mac80211]
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Dec 02 10:49:55 Wills-ThinkPad kernel:  __ieee80211_link_release_channel+0x5e/0x140 [mac80211]
Dec 02 10:49:55 Wills-ThinkPad kernel:  ieee80211_link_release_channel+0x21/0x40 [mac80211]
Dec 02 10:49:55 Wills-ThinkPad kernel:  ieee80211_destroy_assoc_data+0x161/0x1e0 [mac80211]
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? _printk+0x60/0x90
Dec 02 10:49:55 Wills-ThinkPad kernel:  ieee80211_sta_work+0x3f9/0x710 [mac80211]
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Dec 02 10:49:55 Wills-ThinkPad kernel:  ieee80211_iface_work+0x16a/0x1a0 [mac80211]
Dec 02 10:49:55 Wills-ThinkPad kernel:  cfg80211_wiphy_work+0xbf/0xe0 [cfg80211]
Dec 02 10:49:55 Wills-ThinkPad kernel:  process_one_work+0x178/0x350
Dec 02 10:49:55 Wills-ThinkPad kernel:  worker_thread+0x306/0x440
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? __pfx_worker_thread+0x10/0x10
Dec 02 10:49:55 Wills-ThinkPad kernel:  kthread+0xf2/0x120
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? __pfx_kthread+0x10/0x10
Dec 02 10:49:55 Wills-ThinkPad kernel:  ret_from_fork+0x47/0x70
Dec 02 10:49:55 Wills-ThinkPad kernel:  ? __pfx_kthread+0x10/0x10
Dec 02 10:49:55 Wills-ThinkPad kernel:  ret_from_fork_asm+0x1b/0x30
Dec 02 10:49:55 Wills-ThinkPad kernel:  </TASK>
Dec 02 10:49:55 Wills-ThinkPad kernel: ---[ end trace 0000000000000000 ]---
Dec 02 10:49:55 Wills-ThinkPad kernel: ath11k_pci 0000:01:00.0: failed to find peer vdev_id 0 addr 2c:cf:67:9d:4f:32 in delete
Dec 02 10:49:55 Wills-ThinkPad kernel: ath11k_pci 0000:01:00.0: failed to delete peer 2c:cf:67:9d:4f:32 for vdev 0: -22
Dec 02 10:49:55 Wills-ThinkPad NetworkManager[1471]: <info>  [1733136595.9083] device (wlp1s0): supplicant interface state: associating -> disconnected
Dec 02 10:49:57 Wills-ThinkPad NetworkManager[1471]: <warn>  [1733136597.9859] device (wlp1s0): Activation: (wifi) association took too long
Dec 02 10:49:57 Wills-ThinkPad NetworkManager[1471]: <info>  [1733136597.9859] device (wlp1s0): state change: config -> need-auth (reason 'none', sys-iface-state: 'managed')
Dec 02 10:49:57 Wills-ThinkPad NetworkManager[1471]: <warn>  [1733136597.9863] device (wlp1s0): Activation: (wifi) asking for new secrets
Dec 02 10:49:59 Wills-ThinkPad NetworkManager[1471]: <warn>  [1733136599.8662] device (wlp1s0): no secrets: User canceled the secrets request.

On the PI the configuration is:

# jdh: https://frillip.com/using-your-raspberry-pi-3-as-a-wifi-access-point-with-hostapd/
interface=wlan0
#country_code=GB
# Use the nl80211 driver with the brcmfmac driver
driver=nl80211
# The SSID (name of the network)
ssid=$SSID$
# Use the 2.4 GHz band
hw_mode=g
# Use channel 6
channel=6
# Enable 802.11n
ieee80211n=1
# Enable WMM
#wmm_enabled=1
wmm_enabled=0
# 40 MHz channels with 20 ns guard interval
#ht_capab=[HT40][SHORT-GI-20][DSSS_CCK-40]
# Accept all MAC addresses
macaddr_acl=0
# Use WPA
auth_algs=1
# Do not require clients to know the SSID
ignore_broadcast_ssid=0
# Use WPA2
wpa=2
# Use a pre-shared key (WPA2-PSK)
wpa_key_mgmt=WPA-PSK
# The network passphrase (8 characters mimimum)
wpa_passphrase=$PSK$
# Use AES instead of TKIP
rsn_pairwise=CCMP
2

Not super helpful, but you’re not alone with connectivity issues on that wifi adapter: https://discussion.fedoraproject.org/t/laptop-intermittently-disconnects-from-wifi-about-every-5-mins/94662

I note you’re using the official 6.8.0-49-generic kernel from the archive. I don’t have a specific fix to mention, but on my ThinkPad Z13, which sports a QCNFA765 adapter, I’ve found the 6.8.0-*-oem kernel to work better.

I have not had the specific issue you report, but I have certainly had wifi issues in the past with stock kernels on this laptop. That’s all gone with the oem kernel.

Mine is currently on 6.8.0-1013-oem but I need to reboot to get to 6.8.0-1017-oem. My uptime is decent, and it’s permanently connected to wifi at work.

11:43:31 up 55 days, 3:09, 5 users, load average: 1.04, 1.02, 1.00

Right now if you sudo apt update ; sudo apt install linux-image-oem-24.04a that should pull in 6.8.0-1017-oem which should be worth a look.

3

Thanks, I gave that a try and its still crashing the same way.

I thought I’d just pop in the old wifi adaptor to my new laptop having brought a thinkpad because I expect the machines to be upgradable


Apparently not, the wifi is soldered in :frowning:

4

Ok, I’d be inclined to revert back to the stock kernel and run ubuntu-bug linux and report the issue in launchpad. There’s 2007486 which hasn’t had much attention.

Ok, so on my system, I see this on boot:

[Wed Oct 16 11:29:23 2024] ath11k_pci 0000:01:00.0: BAR 0 [mem 0xb0000000-0xb01fffff 64bit]: assigned
[Wed Oct 16 11:29:23 2024] ath11k_pci 0000:01:00.0: enabling device (0000 -> 0002)
[Wed Oct 16 11:29:23 2024] ath11k_pci 0000:01:00.0: MSI vectors: 32
[Wed Oct 16 11:29:23 2024] ath11k_pci 0000:01:00.0: wcn6855 hw2.1
[Wed Oct 16 11:29:25 2024] ath11k_pci 0000:01:00.0: chip_id 0x12 chip_family 0xb board_id 0xff soc_id 0x400c1211
[Wed Oct 16 11:29:25 2024] ath11k_pci 0000:01:00.0: fw_version 0x1106196e fw_build_timestamp 2024-01-12 11:30 fw_build_id WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.37
[Wed Oct 16 11:29:25 2024] ath11k_pci 0000:01:00.0 wlp1s0: renamed from wlan0
[Wed Oct 16 11:29:25 2024] ath11k_pci 0000:01:00.0: Failed to set the requested Country regulatory setting
[Wed Oct 16 11:29:25 2024] ath11k_pci 0000:01:00.0: failed to process regulatory info -22

So mine is wcn6855 hw2.1.
This seems old: fw_version 0x1106196e fw_build_timestamp 2024-01-12 11:30

Let’s go digging:

$ ls -l /lib/firmware/ath11k/WCN6855/hw2.1/
total 0
lrwxrwxrwx 1 root root 21 Apr  9  2024 amss.bin.zst -> ../hw2.0/amss.bin.zst
lrwxrwxrwx 1 root root 24 Apr  9  2024 board-2.bin.zst -> ../hw2.0/board-2.bin.zst
lrwxrwxrwx 1 root root 19 Apr  9  2024 m3.bin.zst -> ../hw2.0/m3.bin.zst
lrwxrwxrwx 1 root root 22 Apr  9  2024 regdb.bin.zst -> ../hw2.0/regdb.bin.zst
$ ls -l /lib/firmware/ath11k/WCN6855/hw2.0
total 2060
-rw-r--r-- 1 root root 1924867 Apr  9  2024 amss.bin.zst
-rw-r--r-- 1 root root   47257 Apr  9  2024 board-2.bin.zst
-rw-r--r-- 1 root root  116906 Apr  9  2024 m3.bin.zst
-rw-r--r-- 1 root root   12057 Apr  9  2024 Notice.txt.zst
-rw-r--r-- 1 root root    2417 Apr  9  2024 regdb.bin.zst

These files come in the linux-firmware package.

$ dpkg -S /lib/firmware/ath11k/WCN6855/hw2.0/amss.bin.zst
linux-firmware: /lib/firmware/ath11k/WCN6855/hw2.0/amss.bin.zst

If we look at https://packages.ubuntu.com/noble/linux-firmware we can see in the changelog that the latest revision was made back in April (which tallies with the date on my 24.04 system above)

linux-firmware (20240318.git3b128b60-0ubuntu2) noble; urgency=medium

  * No-change rebuild for CVE-2024-3094

 -- Steve Langasek <steve.langasek@ubuntu.com>  Tue, 09 Apr 2024 22:14:13 +0000

However, if we look at the oracular changelog we find some updates for WCN6855.

linux-firmware (20240826.git2cdc11a7-0ubuntu1) oracular; urgency=medium

  * Miscellaneous Ubuntu changes
    - [Packaging]: scripts/create-quilt-series: Correctly handle file renames and spaces
    - Rebase to upstream commit 2cdc11a7b3bf06042f1fb179fa1775fde381491f
      Rebase against git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
      - amdgpu: Update ISP FW for isp v4.1.1
      - rtw89: 8852bt: add firmware 0.29.91.0
      - rtw89: 8852c: add fw format-1 v0.27.97.0
      - QCA: Update Bluetooth WCN685x 2.1 firmware to 2.1.0-00642
      - qcom: update path for video firmware for vpu-1/2/3.0
      - amdgpu: DMCUB updates for various AMDGPU ASICs
      - qcom: vpu: add video firmware for sa8775p
      - ath11k: IPQ5018 hw1.0: update to WLAN.HK.2.6.0.1-01291-QCAHKSWPL_SILICONZ-1
      - ath11k: QCA2066 hw2.1: add board-2.bin
      - ath11k: QCA2066 hw2.1: add to WLAN.HSP.1.1-03926.13-QCAHSPSWPL_V2_SILICONZ_CE-2.52297.3
      - ath11k: WCN6855 hw2.0: update board-2.bin
      - ath11k: WCN6855 hw2.0: update to WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.41
      - ath12k: WCN7850 hw2.0: update board-2.bin
      - Revert "i915: Update MTL DMC v2.22"
      - linux-firmware: update firmware for MT7996
      - linux-firmware: Update AMD SEV firmware
      - rename rtl8723bs_config-OBDA8723.bin -> rtl_bt/rtl8723bs_config.bin
      - rtl_bt: de-dupe identical config.bin files
      - rtl_bt: Add firmware file for the the RTL8723CS Bluetooth part
      - rtl_bt: Add firmware and config files for RTL8922A
      - linux-firmware: update firmware for mediatek bluetooth chip (MT7925)
      - linux-firmware: update firmware for MT7925 WiFi device

 -- Juerg Haefliger <juerg.haefliger@canonical.com>  Mon, 26 Aug 2024 08:38:31 +0200

But those updates don’t appear to have made it to noble-updates, nor is there anything seemingly related in the plucky changelog

It might be okay to install the linux-firmware package from oracular or plucky.

Or we could ask the maintainer @juergh if it’s possible to get a backport of the linux-firmware package to noble, or if it’s best to test the oracular or plucky ones first. :pray:

5

We don’t backport firmware packages to earlier releases, only specific and isolated commits that can be tested by the bug reporter. The risk of introducing regressions is just too high with bulk backports.

1 Like
6

And with firmware, it’s typically a case of try the latest blob for your HW from kernel.org and if that helps we’ll add it.

1 Like
7

Thanks for the quick response :+1:
Understandable process.

Should we file a bug to keep a track of this, test the newer firmware blobs and report findings there?

8

Yes, Launchpad please. And you can file it against linux-firmware if you want my attention :wink:

1 Like
9

@wiil - Do you need help getting the relevant blobs and then testing and reporting an issue? Or are you good with what to do next?

1 Like
10

Help would be appreciated, I’m not sure how to proceed because my bug doesn’t seem to behave like the bugs they are expecting. It isn’t a program crash so apport doesn’t run, and it doesn’t crash and freeze the kernel. It is self recovering, I can connect back to my main wifi without doing something like a modprobe, but I can never connect to this PI.

Treat me like a engineer who used to do software in a large company environment 20 years ago, with an above average (but not expert) experience of using and configuring Linux and idea of what is going on in the system, but with no idea of the processes of bug reporting for this or who owns what here :slight_smile:

You know, the sort of person who is really great at following instructions but misses a step because they skim read over it and didn’t think it was important.

1 Like
11

Perfect, let’s do this! Bust out a terminal and away we go! :nerd_face::keyboard:

Instructions to replace ath11k firmware blobs

This looks more complicated than it is really. I just tried to make it clear what’s happening, and make sure you have a backup. My laptop didn’t burst into flames or kick anyones cats, after doing this, and the wifi still works [BONUS].

  1. Check firmware version in use before we do this.

This is just to see what version of the firmware is loaded, so we can be certain at the end, that we actually did it right.

sudo dmesg | grep ath11k | grep fw_version

Here’s what mine says, before we embark on this journey together. :two_men_holding_hands:

[    5.231096] ath11k_pci 0000:01:00.0: fw_version 0x1106196e fw_build_timestamp 2024-01-12 11:30 fw_build_id WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.37

2024-01-12 - we hope to see something newer, later. :crossed_fingers:

  1. Make a folder to work in, and jump into it
mkdir ~/new_firmware
cd ~/new_firmware
  1. Grab the latest (as of writing) firmware tarball from kernel.org

Note to future readers: Scroll to the bottom of the kernel.org firmware directory to find the latest tarball.

Note: I do the apt install bit to make sure you have wget & tree - as I don’t want to assume anything.

sudo apt update
sudo apt install wget tree
wget https://www.kernel.org/pub/linux/kernel/firmware/linux-firmware-20241210.tar.xz
  1. Unpack the kernel firmware blobs

Note: This should spit out a list of files starting with linux-firmware-20241210/ath11k/ and finishing with linux-firmware-20241210/ath11k/WCN6855/hw2.0/regdb.bin.

tar xvf ./linux-firmware-20241210.tar.xz "linux-firmware-20241210/ath11k/"
  1. Backup existing blobs

Just in case we mess up, let’s make a backup of your existing blobs. This will also spit out a list of files.

tar czvf ./backup_ath11k_firmware.tgz /lib/firmware/ath11k

Make sure the backup was good

Note: The size may differ, as will the date/time, but should be in this ballpark

ls -l ./backup_ath11k_firmware.tgz
-rw-rw-r-- 1 alan alan 15412653 Dec 11 12:39 ./backup_ath11k_firmware.tgz

You could also list the files in the tarball, which, again should spit out a list of around 100 files.

tar tf ./backup_ath11k_firmware.tgz
  1. Remove old blobs and replace with the new ones

First, a quick confirmation that the files already installed differ from the new ones we unpacked - they almost certainly will.

Note: This will spit out a load of lines starting “Only in /lib/firmware/ath11k
” and some starting “Only in ./linux-firmware-20241210/ath11k
” which is to be expected.

diff -r ./linux-firmware-20241210/ath11k/ /lib/firmware/ath11k/

:rotating_light: Next danger we will remove the existing blobs. Don’t reboot at this point. Because you’ll be missing the blobs to get online. So once this is done, immediately move to the next step which copies the newly unpacked blobs.

sudo rm -rf /lib/firmware/ath11k

Here we just copy the newly unpacked blobs to replace the old ones.

Note: You’ll need your sudo password for this, and it should output a bunch of lines starting "‘./linux-firmware-20241210/ath11k/IPQ5018/hw1.0/Notice.txt’ → ‘/lib/firmware/ath11k/IPQ5018/hw1.0/Notice.txt’and ending in’./linux-firmware-20241210/ath11k/WCN6855/hw2.0/regdb.bin’ → ‘/lib/firmware/ath11k/WCN6855/hw2.0/regdb.bin’` .

sudo cp -av ./linux-firmware-20241210/ath11k /lib/firmware

We want to add in the symlinks (which is what Ubuntu ships with) for people with hardware 2.1 cards (like us) to point to the 2.0 firmware (that the kernel tarball ships with).

sudo mkdir /lib/firmware/ath11k/WCN6855/hw2.1/
pushd  /lib/firmware/ath11k/WCN6855/hw2.1/
sudo ln -s ../hw2.0/amss.bin amss.bin
sudo ln -s ../hw2.0/board-2.bin board-2.bin
sudo ln -s ../hw2.0/m3.bin m3.bin
sudo ln -s ../hw2.0/regdb.bin regdb.bin
popd

We should set the permissions on these files correctly, too.

sudo chown -R root:root  /lib/firmware/ath11k

Let’s check they ended up in the right place by re-doing the diff again.

Note: This time you’ll see only one line of output “Only in /lib/firmware/ath11k/WCN6855: hw2.1” because we completely removed the ath11k folder and replaced the contents, but added the hw2.1 folder, above.

diff -r ./linux-firmware-20241210/ath11k/ /lib/firmware/ath11k/

Final check of the directory tree. The output on my machine looks like this.

tree /lib/firmware/ath11k
  1. Cross fingers :crossed_fingers: and reboot :electric_plug:

:pray: Pray to the computer gods that it comes back.

  1. Check loaded firmware version again
sudo dmesg | grep ath11k | grep fw_version

Mine now reports:

[    5.583817] ath11k_pci 0000:01:00.0: fw_version 0x11088c35 fw_build_timestamp 2024-04-17 08:34 fw_build_id WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.41

2024-04-17 is a whole three months newer than 2024-01-12. No idea if this fixes it for you. My machine boots, and is on the wifi, so there’s that at least, as far as testing goes! :smiley:

  1. Do your thing

Test the wireless. Let us know if it works!

12

Confirmation at this step.

I see ONLY the “Only In” lines, there are no “Files differ”, e.g.:

Only in ./linux-firmware-20241210/ath11k/IPQ5018/hw1.0: board-2.bin
Only in /lib/firmware/ath11k/IPQ5018/hw1.0: board-2.bin.zst

Should there have been a step to unpack these compressed files, or is this to be expected?

13

Step 5 has an unpack line, starting with tar which unpacks the tarball from the kernel website. No other uncompression is needed.

14

Sorry, yes, /lib/firmware has compressed versions, and only compressed versions, is that OK?

$ ls /lib/firmware/ath11k/IPQ5018/hw1.0/ -al
total 2072
drwxr-xr-x 2 root root    4096 Nov 21 16:50 .
drwxr-xr-x 3 root root    4096 Feb 20  2024 ..
-rw-r--r-- 1 root root    6472 Nov  8 07:30 board-2.bin.zst
-rw-r--r-- 1 root root      82 Nov  8 07:30 m3_fw.b00.zst
-rw-r--r-- 1 root root     110 Nov  8 07:30 m3_fw.b01.zst
-rw-r--r-- 1 root root  117179 Nov  8 07:30 m3_fw.b02.zst
-rw-r--r-- 1 root root      75 Nov  8 07:30 m3_fw.flist.zst
-rw-r--r-- 1 root root     186 Nov  8 07:30 m3_fw.mdt.zst
-rw-r--r-- 1 root root   12313 Nov  8 07:30 Notice.txt.zst
-rw-r--r-- 1 root root     248 Nov  8 07:30 q6_fw.b00.zst
-rw-r--r-- 1 root root     439 Nov  8 07:30 q6_fw.b01.zst
-rw-r--r-- 1 root root    2699 Nov  8 07:30 q6_fw.b02.zst
-rw-r--r-- 1 root root  302691 Nov  8 07:30 q6_fw.b03.zst
-rw-r--r-- 1 root root   24157 Nov  8 07:30 q6_fw.b04.zst
-rw-r--r-- 1 root root    4667 Nov  8 07:30 q6_fw.b05.zst
-rw-r--r-- 1 root root     300 Nov  8 07:30 q6_fw.b07.zst
-rw-r--r-- 1 root root     307 Nov  8 07:30 q6_fw.b08.zst
-rw-r--r-- 1 root root 1448294 Nov  8 07:30 q6_fw.b09.zst
-rw-r--r-- 1 root root   82933 Nov  8 07:30 q6_fw.b10.zst
-rw-r--r-- 1 root root   41246 Nov  8 07:30 q6_fw.b11.zst
-rw-r--r-- 1 root root    1908 Nov  8 07:30 q6_fw.b13.zst
-rw-r--r-- 1 root root     686 Nov  8 07:30 q6_fw.b14.zst
-rw-r--r-- 1 root root     100 Nov  8 07:30 q6_fw.flist.zst
-rw-r--r-- 1 root root     713 Nov  8 07:30 q6_fw.mdt.zst
15

Yes, but we’re deleting those, and replacing them. The kernel has had support for compressed firmware files, but they can also be uncompressed. We’re just removing those, and dropping the uncompressed versions in the firmware folder.

To clarify based on your previous question. Yes, this is expected, no, there are no additional steps to compress or uncompress anything.

1 Like
16

You’ll be please to know I’m still alive and so is my laptop and its now running the April build.

Also, and this one is a bit bittersweet, the crash is still there if I think a little more verbose than before.

So, although you have not solved my problem, it wasn’t a problem that would just go away if I sat on it, so its a worthwhile endeavor!

Lead on

1 Like
17

Ok, time for a bug report, and in parallel, more testing!
We can’t rule out the firmware, as it’s possible this is just an unknown bug upstream in the kernel, driver, or firmware.

First, let’s get that bug filed.

Filing a bug against the kernel

Prerequisites

  1. Ubuntu Single Sign-On (SSO) account
  2. Access to the affected system
  3. Internet connection (even if intermittent)

Setting Up Launchpad Account

  1. Go to launchpad.net
  2. Click “Log in / Register”
  3. Choose “Log in with Ubuntu”
  4. Use your Ubuntu SSO credentials
  5. Complete your Launchpad profile if prompted

Preparing Your System

Open a terminal (Ctrl+Alt+T) and run these commands:

# Update package lists
sudo apt update

# Install all available updates
sudo apt upgrade

# If a kernel update was installed, reboot your system
# Check if reboot is needed:
[ -f /var/run/reboot-required ] && echo "Reboot required"

If a reboot is required, restart your system before proceeding.

Filing the Bug Report

  1. Open a terminal and run:

    ubuntu-bug linux

  2. This will collect system information and open your web browser to complete the bug report.

Writing an Effective Bug Report

Include the following information in your description:

  1. What you were doing when the problem occurred
  2. What you expected to happen
  3. What actually happened
  4. Clear steps to reproduce the issue:
    • Starting conditions (fresh boot, after suspend, etc.)
    • Exact actions taken
    • How frequently the issue occurs
  5. Your wireless hardware details (if known)
  6. Any relevant error messages from system logs
  7. A link to this discussion

Example format:

Summary: Wireless connection drops after suspend on ThinkPad X1 Carbon

Steps to reproduce:
1. Suspend laptop by closing lid
2. Wait 5 minutes
3. Open lid to resume
4. Wireless connection fails to reconnect

Expected: Wireless should automatically reconnect
Actual: Network manager shows no wireless networks available

Frequency: Happens 100% of the time after suspend

Additional notes:
- Issue started after upgrading to Ubuntu 22.04
- Using Intel AX200 wireless card
- Only way to fix is to reboot

After Submission

  1. Note your bug report number
  2. Monitor your email for questions from developers
  3. Update the bug report if you discover new information
  4. Test any proposed solutions and report back on their effectiveness

Important Tips

  • Keep the bug report focused on one specific issue
  • Respond promptly to requests for additional information
  • If asked to test with a newer kernel or different driver, provide clear feedback
  • Update the bug status if you find a workaround or if the issue resolves

If you have time, inclination and nothing else going on, we could look at trying a newer kernel, from the “Kernel PPA” (note: Not actually a PPA), which contains unmodified versions of the upstream Linux Kernel, built automatically for people like us. It’s not meant to be run in production, and isn’t supported at all.

But they’re useful for testing if a really new kernel works, and if it does, can we “bisect” back and forth until we find the exact (or near enough) release it broke in.

At the start of this thread we saw you are currently on 6.8.0-49-generic. So we could jump somewhat higher to jump to like v6.12.3 for example.

Installing a Mainline Kernel in Ubuntu

Prerequisites

  • Ensure you have wget installed
  • A stable internet connection
  • Sufficient disk space
  • Administrator (sudo) privileges

Finding the Right Kernel Version

  1. Visit the Ubuntu Mainline Kernel PPA website:

    https://kernel.ubuntu.com/~kernel-ppa/mainline/

  2. Versions are listed by date and kernel version. Choose the version you need:

    • v6.x.x - Latest stable mainline kernels
    • v6.x.x-rc - Release candidates (testing)
    • Navigate to the specific version directory

Downloading the Kernel Packages

For AMD64/x86_64 systems, you need these files:

  • linux-headers-*-generic
  • linux-headers-*
  • linux-image-unsigned-*-generic
  • linux-modules-*-generic

Example commands (replace VERSION with your chosen version, e.g., 6.7.0):

cd ~/Downloads
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v${VERSION}/amd64/linux-headers-${VERSION}-generic_${VERSION}.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v${VERSION}/amd64/linux-headers-${VERSION}_${VERSION}.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v${VERSION}/amd64/linux-image-unsigned-${VERSION}-generic_${VERSION}.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v${VERSION}/amd64/linux-modules-${VERSION}-generic_${VERSION}.deb

Installing the Kernel

  1. Install the packages in the correct order:

    cd ~/Downloads
sudo dpkg -i linux-headers-*.deb linux-image-unsigned-*.deb linux-modules-*.deb

  2. Reboot your system:

    sudo reboot

Verifying the Installation

After reboot, check your kernel version:

uname -r

Reverting if Needed

If the new kernel causes issues:

  1. Reboot and hold SHIFT to access GRUB menu
  2. Select “Advanced options for Ubuntu”
  3. Choose your previous kernel version
  4. Remove the mainline kernel:
    sudo apt remove linux-headers-VERSION-generic linux-image-unsigned-VERSION-generic linux-modules-VERSION-generic

Important Notes

  • Replace VERSION with actual kernel version numbers in all commands
  • These kernels are unsupported and for testing only
  • Some hardware-specific features might not work
  • Keep your stable kernel installed as a backup
  • Not all kernel versions will work with all hardware
  • Proprietary drivers might need to be reinstalled

Enjoy!

18

Thanks for your help

Bug opened as

The package was autodetected as linux-signed-oem-6.8, is this correct?

1 Like
19

Much fingerpoking later.

The bug is fixed in mainline (or at least, it isn’t showing up!)

The fix happens between 6.8.12 and 6.9.0

Do the numbers mean this: a.b.c increments huge things in a and big things on b and small things on c, so 6.11.11 is the best 6.11, and its likely to be more stable than a 6.12.x until a 6.13 comes along

Or am I just better reverting to the standard kernel for now?

20

It works on mainline 6.9.0, but not on 6.8.12?

Good work! This is useful additional information for your bug 2091587 (which I see you’ve already updated). Nice one! Excellent contribution! :raised_hand:

Unfortunately version numbers don’t work quite like that, but kinda. It’s called semantic versioning.

It looks like v6.9.0 was released (or at least built in the mainline “ppa”) on 12th May, and v6.8.12 was built on 30th May. So 6.8.12 is a patch to the 6.8 series. If someone wants to ‘track’ the 6.8 series then they may want to stay on that until it’s no longer supported. However, some people just grab whatever the latest version is.

I don’t fully understand the process by which patches end up in which release of the kernel, so I’m not sure if the kernel team can “cherry pick” something from 6.9 and put it on 6.8. The Ubuntu kernels are not the same as the upstream mainline ones.

The word “stable” is a tricky one. Do you mean “unchanging” or “not crashing”?

If you’ve found a mainline kernel that works, and you’re happy running that (unsupported) kernel so your life is simpler and work/play can get done, then so be it, it’s your computer.

If someone on the kernel team, or you, if you have time, wants, you could go grubbing around in the source code for the changes that might be appropriate between those two releases of the kernel.

Now, the kernel codebase is enormous, but by bisecting, you’ve massively reduced the search space down. Plus, you’re really only looking for changes in the ath11k driver (probably), although there may be changes elsewhere in the network codebase which cause this.

How much time and curiosity do you have? You’ve come this far, why stop here!? :wink:

After a mild spelunking, I found this commit, no idea if it’s the one that we need to be concerned about. :person_shrugging:

Maybe @juergh can comment? :pray: