Foundations Team Updates - Thursday 14 Sep 2023

Previous week’s status can be found at Foundations Team Updates - Thursday 7 Sep 2023



Ubuntu Packaging Guide

  • attended weekly Ubuntu Packaging Guide meeting
  • finished article explanation/archive
  • writing definitions for glossary terms
  • cleaning up commit history



Thursday 14 September 2023


  • Debugged an issue with launching s390x instances in bos01. Submitted an MP which resolved the issue.
  • Submitted an RT regarding failure to provision s390x instances in bos02.
  • Subscribed Canonical Ubuntu QA to the autopkgtest development branch - this was missed somehow.
  • Reported bug 2035126 regarding updating the autopkgtest web servers to display more information about who requested a test.
  • Reviewed and merged Tim’s MP adding a link to the service status discourse post on the autopkgtest main page.

ISO Tracker

  • Investigated issues with the ISO tracker sending email as I now have root access on limequat.
  • Reviewed and merged an MP which adds FDE test cases to the ISO tracker.
  • Fixed the download links for the daily-legacy images in the ISO tracker which I’d reported as bug 2034957.


  • Kicked off an Ubuntu Budgie daily build as they asked.
  • Reported a bug regarding corrupt fonts during a distribution upgrade from lunar to mantic.
  • Reported a bug regarding Kubuntu distribution upgrades failing. This was discovered during auto-upgrade testing.
  • Reported a bug about a subiquity crash on a Lenovo G50-80 system.
  • Asked a LP question regarding not being able to subscribe Canonical Ubuntu QA to ubuntu bugs tagged iso-testing.
  • Reviewed and merged seb128’s MP regarding smoke testing of the Ubuntu Desktop image.
  • Discovered an issue with the Ubuntu Desktop (Legacy) image building which was ironically due to a change I’d requested. I made a change which should resolve the issue.
  • Reviewed an MP from @andersson123 which would fix the command-not-found index age metric.


  • 360 reviews.
  • Got added to the UTAH dev team as an admin.
  • Worked with IS regarding the migration of the Error Tracker environment to PS6.
  • Worked with IS regarding the amount of storage used on s0lp4 in bos02.
  • focused on bug fixing this week
  • worked on recovery implementation using zkey on s390x
  • worked on codecs vs network status bug in subiquity.
  • merged fix in subiquity for wrong URL to access the -security pocket.
  • internal meetings


  • ubiquity: Upload 23.10.3 to update legacy timezones in tzsetup (LP: #2031838)
  • initramfs-tools: Some bug triaging. The bug count is now down to 590 bugs (excluding incomplete without response).




  • 360 review
  • Adjusted default of ssh_pw_auth on Ubuntu Server for Pi images; SSH password login is now disabled by default
  • Landed one fix for livecd-rootfs; addressing fixes in the remaining ones before posting more
  • Posted SD card blog post
  • Still working on NBD write-up, but created bug to track the move of the nbd module (LP: #2035373)
  • Finished upstreaming GPIO changes
  • Bumped raspberry-userland to fix an issue with kernel 6.5 (LP: #2035371)
  • Started filling out release notes for mantic
  • Investigating linker issue on arm64 (LP: #2035094)
  • Pi meetings


  • Sponsoring of CPC livecd-rootfs changes, including follow-through on the
    lsb-release \EFI\debian issue
  • SecureBoot cross-vendor weekly meeting


  • FFe reviews



  • catch-up
  • 360 reviews


  • libevent FTBFS with glibc 2.38. Fixed in Debian experimental. Send MP to merge it from Debian experimental

+1 Maintenance

  • ffmpeg transition

    I’m looking at NBS page. It seems that many packages are stuck there because the unfinished ffmpeg transition.

    • moc

      Uploaded moc 1:2.6.0~svn-r3005-3.1 to Debian to fix building with ffmpeg 6.0.

    • gpac

      Uploaded gpac 2.2.1+dfsg1-2 to Debian with patch from upstream to fix building with ffmpeg 6.0.
      But I miss that gpac has already been removed from mantic, although it’s shown at NBS page.
      So I stop here and don’t go further with FFe for this new version.

    • xpra

      Uploaded xpra 3.1.5+dfsg1-0.1 to Debian to fix building with ffmpeg 6.0.

    • liquidsoap

      Still have no idea how to debug the OCaml failures.

  • poppler transition

    • inkscape

      Currently debugging the failures on i386.


This has been my first week at Canonical and on the Foundations team!

I’ve been spending my time getting to know the Subiquity code base, learning how installers and ISOs work, and working with @dbungert who has been helping me get started.


Fairly short week due to PTOs:

  • Depile a week’s worth of emails and pings
  • 360 reviews
  • Investigated pastescript FTBFS (dh-python new cleaning code is a bit aggressive)
  • Patch Pilot



Missed last week’s update due to a national holiday, so sending two week’s worth of updates:


  • Fixed the nvme-stas s390x autopkgtest hang (LP: #2026878).
    • Issue was caused by an erroneous byte-order on s390x (thanks @enr0n). MP has been submitted here.
    • The s390x autopkgtest is now passing successfully.
  • Investigating nvme-of test case on ARM64.
  • Investigated libnvme FTBFS on latest mantic test rebuild.
    • Forward fix upstream since it’s also failing to build on Debian.
  • Reported FTBFS bug from latest mantic test rebuild on Debian for libnvme (#1051722)
    • Fixed in 1.5-3
    • Requested sync from Debian Unstable (LP: #2035203)


  • Investigated september updates build error on ARM64
  • Started packaging .NET 8 RC1 released Tuesday, September 12th
    • Building bootstrap packages for AMD64 and ARM64
    • Investigating ARM64 build error happening since latest toolchain updates on Mantic





  • some (but sadly not much) work on SRU’ing patches for openssl
  • armhf time_t work: added a bunch of optimizations to the script which lead to a much more comfortable feedback loop and enabled me to fix 140 packages so far (almost all are missing dependencies)


  • 360
  • curtin
    • MP: #451298 (LP: #2035034) - fix reading existing partition info on disks that have non-512 byte logical sector size
    • MP: #451309 - add some logging for the layers that are being extracted.
  • subiquity
    • lunar branch build for LP: #2033977, stable release shortly for this fix
    • PR: #1791 (LP: #2034939) - guided zfs crash when creating swap part, forgot to align the partition size
    • PR: #1792 - Fix the snap build, and implement building the snap in CI.
    • PR: #1793 - fix deprecation warnings in workflows due to checkout@v2
    • FDE bug triage and some investigation. Two of interest:
      • LP: #2031896 - “cannot seal the encryption keys” failure
      • LP: #2034715 - timeouts communicating with snapd
  • probert
    • PR: #138 - while investigating Subiquity snap build failures, found that the probert unit tests may fail depending on filesystem tool availability. Add mocks.
  • livecd-rootfs
    • MP: #450980 (LP: #2026225), having to do with being able to install kernels other than the default when using the desktop ISO
  • 24.04 planning
  • patch pilot
  • help onboarding @cpete
  • 360

I am attending the Image-based Linux summit and All Systems Go in Berlin this week. Not really attending many talks but focusing on the hallway track, talking to many different people.

I gave a talk about sandboxing in APT, enticing suggestions from the community how to sandbox further. Had some great conversations. I also presented the numbers of our armhf time_t work and highlights from the script in a lightning talk.

Had a couple of discussions from the shim-review side, about openSUSE signing systemd-boot, to NixOS people that were looking for advise on how to submit a shim.

My kernel crashed in 3 places, my gnome-shell is getting SIGKILLED whenever I lock the screen and I spent a too much time trying to trace what is killing it, but even bpf tracing doesn’t tell me, I think it’s the kernel itself killing it for some reason, but I haven’t figured out how to use ftrace to trace that exit code path yet.

My USB hub fails to start after resume, my bluetooth stack attached to said USB hub also fails to respond all the time, and the PCIE chip that USB hub is on is sending spurious interrupts, so I did upgrade and downgrade kernels, upgrade firmware, reset firmware, but nothing helps. Maybe my firmware is toast.

My systemd-resolved was hanging one boot so I spent 15 minutes or so trying to figure out what was going on there.

Talked to Paul Menzel about fast booting and grub-common.service. Realized I forgot to move it to use the

Lots of talking to Bastian Blank from Debian.

Talks with Luca and Lennart about systemd-boot stuff.

Some people had questions about Ubuntu support levels, so I tried my best to answer those.

Continuing with helping Mate.


This is my first week in the Foundations team, and at Canonical.

I have been mainly discovering the tooling (packaging process, etc.) and the image build infrastructure, specifically doing a deep dive in ubuntu-image code. Also some work with @schopin to understand how to deal with a FTBFS issue.



  • MP for slight amendment to our documents
  • MP for adding a link at the top of the webpage to a post on discourse detailing the current service status
  • MP’s for adding new unit tests and adding unit tests to CI for autopkgtest-web
  • Investigating an issue with apt dependencies not being satisfied when doing a mojo run

autopkgtest specs

  • Merged a few mp’s moving packages to large/long

iso testing

  • Couple of MP’s amending test instructions w.r.t TPM install and changes to default/minimal/full and testing of the gnome suite of applications
  • testing issues with upgrade test failures from lunar->mantic on flavors

grafana dashboards

  • Setting up email alerts for service shortages - set up an alert and I’m waiting for IS to sort out the smtp settings on the grafana instance, then I’ll expand it and make proper alerts
  • amended an issue with a new collector for the age of the command not found index
  • created a new tile to show the data from the above collector


  • Multiple MP reviews
  • Reviewed and merged MP from @andersson123 running the unit tests in CI

ISO testing

  • Continued monitoring of the status of the Server ISO testing jobs
  • Test of beta subiquity fixing the security mirror
  • Venonat cleanup, upgrade and retrigger of failed jobs


  • More work on getting OpenQA up and running on beldam, with frontend on a PS5 environment

Ubuntu Server

  • General Jenkins maintenance
  • More debugging of the “unresponsive LXD” issue which is interfering with integration testing and metrics
  • Bug triage rotation


  • Greenhouse work
  • Patch pilot rotation


  • jexec patch was merged upstream, opened MR on salsa.


  • summarised failures for default Java 21, most issues are general compile errors, a big cluster of packages requiring Gradle update and quite a few hardcoded -target 7 arguments in the Makefile.
    Started discussion on debian-java



  • added MP to add upstream testsuite as autopkgtest


  • working on adding unit tests (for build time tests)