An Unexpected Attempted Migration in Devel Release
At 13:59 UTC on 2025-10-24, @jbicha reported on Matrix ubuntu:release that several packages were beginngin to migrate from resolute-proposed to the release pocket that should have been blocked by failing autopkgtests. The message was quickly noticed by various members of Ubuntu with proper powers to intercede, including @paride, @schopin, @jchittum, and @seb128. Many of us were already in a planned, unrelated call, and turned it into an impromptu white hat bridge which included @apw, @uralt, @simpoir, and @bdrung. By 14:05 UTC, the cause was found to be a bug introduced to Britney roughly two hours prior. In attempting to skip RISCV testing while RISCV nodes were offline, it was inadvertently set to skip all tests. The change was quickly fixed in Britney, reviewed in the call. At 14:15, after discussion to understand current state and ramifications of the automatic migration in the devel suite, @paride contacted the Launchpad team via Canonical’s internal chat to disable the Launchpad Publisher for a period of 2 hours. The Publisher was still in its last run, and the packages marked for migration by Britney were not picked up by the Publisher. This gave the team time to clear the Launchpad queue and copy the packages back into resolute-proposed. Britney was then run in a “non-save” mode, which means the results of proposed migration are run, but not saved to Launchpad as package state. Once it was determined that the changes to Britney were correct, and that the outcome of a run was successfully picking up past test results and seeing currently queued and running tests, the manual run of Britney was halted. This led to a time where the proposed-migration report for resolute was in an unfinished state. By 16:00, all packages had been fixed and the Launchpad Publisher was asked to be re-enabled. In the end, no packages migrated,
The incident is considered resolved and Resolute Racoon development has progressed smoothly since. Some bugs may have been closed early due to automation.
What Does This Really Mean?
There have been a lot of blog posts done in the past few months around how Ubuntu tests and publishes packages. @mclemenceau published a blog breaking down package lifecycle simply, and showing off a tool to visualize everything a bit better.[1] @jchittum published a blog talking about the testing lifecycle and a high level discussions about autopkgtest infrastructure.[2] @uralt has been publishing an ongoing series about revamping autopkgtest infrastructure.[3] These provide good background reading before diving further.
The tl;dr of package lifecycle is:
- Package is uploaded to proposed
- Britney (the service) reads what is in proposed from Launchpad
- Britney queues a message requesting autopkgtest runs
- There is one message, per test run, with reference to what triggered the test run
- There are tests run for the package and for any reverse dependencies of the package
- Autopkgtest-cloud pops this message from the queue, and runs the test
- The result is saved to a database and a message sent back to the queue
- Britney consumes information, and produces the Proposed Migration Report
- If Britney is run in “save” mode, then Britney edits the Launchpad publication metadata for the package, pointing those records to the release pocket and marking them as Pending. Crucially, it doesn’t actually interact with the archive itself.
- Only the Ubuntu release currently under development has Britney running in “save” mode.
- Launchpad uses a queue system to track these changes. Archive Admins have the power to move things between these queues by setting package publishing information. A bit more on this later.
This is a very high level set of statements that may not be exactly right as I’m not an archive admin, Launchpad administrator, nor expert in any field other than mid to late 20th century and early 21st century experimental classical music, which is of little help here.[4]
This means that only Resolute Racoon, the current Ubuntu Development version, was possibly affected by this change. For Stable Releases, all changes go through the Stable Release Update.[5] This process requires an SRU team member to run manual commands, essentially the reverse commands of what we ran to fix this situation.
Why Didn’t Packages Fully Migrate?
Excellent question astute reader! For those reading between the lines and who followed on Matrix, they’ll see that the packages were marked for migration, but had not yet fully migrated. They were in a bit of a limbo state – gone from proposed, but not in release yet. This is because of the Launchpad Publisher.
The Launchpad Publisher is a large service that handles all the moving of packages. This includes publishing built artifacts to the correct places, processing binary moves between pockets, and is inclusive of the primary archive and all PPAs. This includes areas such as security and esm. The Publisher is not a continuous process – it takes in the state of Packages and their requests at the start, and then goes through all the required pieces. This means that, for this incident, we were partially saved by time and quick response. A Publisher run had started prior to the change to Britney. This meant that all the affected packages were queued for the next run of the Publisher, which gave us time to alert Launchpad and ask them to pause the next run. The Publisher does not run concurrently, so we figured delaying a little while wouldn’t cause too much harm, as we estimated the current run to take at least another hour.
Outcomes
The bug introduced in Britney was a very human mistake that was not easily spotted in code review. Britney itself doesn’t have great test coverage, and this is an area improvement for the team. However, it showed just how quickly members of Ubuntu can mobilize around a potential issue. I took notes during the call, we shared any scripts via pastebins prior to running for quick review and short-term saving.[6] The scripts and chat have also been saved off so we can review and put together a small playbook in case of another similar incident.
I’d like to especially thank @apw and @lgp171188 who moved quickly to help. @apw gave us a lot of technical know-how on the Launchpad API side, publishing and package movement knowledge, and thorough understanding of the system. @lgp171188 responded quickly and was able to pause the next Launchpad Publisher run. And, of course, every one listed at the top who was planning to have a regular team meeting, and instead turned it into a white hat call where we moved quickly and decisively to address the possible incident before anything could happen. It also illustrates that the processes we put in place to guard stable releases are incredibly important, as the tight controls made it so this bug could not affect them.
[1] Visualizing and Exploring Ubuntu Excuses
[2] Debcrafters Digest -- Vol 2
[3] Topics tagged autopkgtests
[4] yes, this is a true statement. The Dr. in my signature is real.
[5] https://documentation.ubuntu.com/project/SRU/howto/
[6] pastebin is not a reliable source of keeping knowledge forever.