Hiya folks,
I wanted to take some time today and talk about the work being done to modernize the processes behind the Ubuntu archive.
1. Archive? What archive?
You can think of the “Ubuntu archive” here as a collection of software packages, which are what you install when you run, for example, apt install neovim. Each release of Ubuntu has multiple collections of these packages which we call “pockets”. Namely, each release will have a default pocket (also called a release pocket) and a proposed pocket to start. Once a release is out there, the default pocket is frozen and further versions of packages for that release will go into an updates pocket, a backports pocket, or a security pocket as needed. You can see the pockets your system draws its packages from every time you run apt update:
Get:1 http://security.ubuntu.com/ubuntu resolute-security InRelease [124 kB]
Get:2 http://archive.ubuntu.com/ubuntu resolute InRelease [124 kB]
Get:3 http://security.ubuntu.com/ubuntu resolute-security/main amd64 Components [208 B]
Get:4 http://archive.ubuntu.com/ubuntu resolute-updates InRelease [124 kB]
Get:5 http://security.ubuntu.com/ubuntu resolute-security/restricted amd64 Components [212 B]
Get:6 http://security.ubuntu.com/ubuntu resolute-security/universe amd64 Components [212 B]
Get:7 http://security.ubuntu.com/ubuntu resolute-security/multiverse amd64 Components [212 B]
Get:8 http://archive.ubuntu.com/ubuntu resolute-backports InRelease [124 kB]
Get:9 http://archive.ubuntu.com/ubuntu resolute-proposed InRelease [124 kB]
<snip>
2. Processes? What processes?
All uploads for new versions of packages will initially go into the proposed pocket, where they will stay until they are sufficiently tested. There are multiple layers of what we consider to be “sufficiently tested”, such as running the test suite provided by the package itself, the test suites of any other packages that depend directly on the package in question (performed by autopkgtest which is undergoing its own modernization process), as well as some other checks such as making sure nothing in the archive becomes impossible to install or remove as a result of this new version. This is the most important process in the Ubuntu archive called proposed-migration.
All of these checks are kicked off and all of its results are collected by a piece of software called britney (well, britney1 and britney2 in tandem but let’s not get lost in the details too much) and collected in handy reports called update-excuses and update-output. update-excuses (pictured below) details how the new package is behaving against the archive, that is to say the package builds, doesn’t regress any tests, and doesn’t have impossible to fill dependencies such as depending on a package that doesn’t exist or a package in main depending on a package in universe. update-output, conversely, focuses on how the archive is behaving against the new package by making sure the count of packages in the archive which are impossible to install does not increase with the new version.
If that report doesn’t seem all that digestible to you, I would advise you to go check out @mclemenceau’s excellent visual excuses project (and highly advise you AGAINST trying to read update-output, which is a lot harder to parse to the naked eye).
As an example, let’s say we have a package providing sounds in a binary called package-sounds in the archive when upstream decides to rework things and provide videos instead. Following this, we (or Debian) decide to rename the deb to package-video. Other packages in the archive which depend on package-sounds will need to update their Depends: fields (and possibly implement code changes if necessary). update-excuses might give package-video a pass if it builds fine, all its related tests pass, and it itself does not have impossible dependencies. However, update-output will likely halt this migration as any other package which still explicitly depends on package-sounds would become impossible to install if package-video is moved out of proposed. In this scenario, we would fix all affected packages in the proposed pocket until everything is ready to migrate as per update-excuses, then migrate everything out of proposed at once as a set.
Another important archive process is germinate. Ubuntu images are built from what are called “seeds”, which are lists of packages that are required to be on a complete installation of Ubuntu. Germinate turns the seed packages into lists including all the individual dependencies required for the seed packages to work and is thus a crucial part of Ubuntu.
The way these reports are generated is through a series of scripts triggered regularly via cronjob. These scripts live on a privileged machine and are maintained mostly by the Ubuntu Archive Admins team and have served Ubuntu well over the past decades, having been manually maintained this whole time. However, as you might have read on @seb128’s previous blog post about Merge-o-Matic, there is an ongoing effort on the part of Ubuntu Engineering to charm all of our processes while we move datacenters, and I have taken on the task of charming the archive scripts.
3. Current charming efforts
Our initial goal as Debcrafters is to very carefully put all existing Ubuntu archive processes into a charm and deploy it. At the end of the initial phase, the only difference as far as the reports are concerned is that they run on a different machine, and we’ll be using systemd units and timers instead of the previous model of a single crontab to rule them all. I have made some good progress on this already, and a repository for a machine charm capable of running all of these scripts is already up. The intent with the machine charm is to be able to run a single juju deploy command and end up with a machine that only needs the appropriate credentials to start running all the Ubuntu archive processes which keep the world going around.
4. Future efforts
Once the initial form of the charm is functional and we have transitioned all archive reports to be generated from the new machine charm instead of the currently manually maintained setup, our intent is to take a look at the actual scripts being run and how we can optimize them directly. Next cycle, here are some things I want to think about (note: none of these are firm commitments, just ideas that I have!):
- Splitting the single charm we have now into multiple units. While some of the archive reports are somewhat coupled to each other, there are some that can run separately. Personally, I would be interested in seeing if we can decouple germinate and britney.
- Integrating ubuntu-archive-scripts as well as any other scripts required for report generation directly into the charm(s), so we have a single source of truth for all archive processes.
- Tighter integration of autopkgtest with britney. Currently, britney does a full database download of autopkgtest every time test results need to be analyzed to see if a package can be migrated from the proposed pocket. Now that autopkgtest has made significant progress towards being a set of modernized machine charms itself, it should be possible for us to better leverage the capabilities of juju and optimize this process.
- The version of britney we are using is a fork of Debian’s britney. We can look at any improvements Debian has recently made to see if they can be ported to our fork of britney as well.
Thank you all for reading and I hope to show off more of this work soon!