Today I’m writing about one of the regular Ubuntu Archive activities: test rebuilds.
Introduction
Test rebuilds are builds of one or more components of a release series, which don’t land in the main archive, but in a separate test archive.
Test rebuilds are done for several reasons:
- To check the buildability of the archive. When build dependencies for a package get updated in an incompatible way, then the package may fail to build; a test rebuild detects these build failures. It is usually done once or twice during a release cycle.
- To check for major planned changes (affecting large parts of the archive) before they are done in the main archive.
Typically rebuilds will be used for changes to a core component like Glibc or for toolchains updates (newer gcc version).
If the set of the affected packages is small or if the changes start a transition that would require dealing with several dependency levels, it would be better to use a PPA.
Newer versions of poppler, which tend to bump sonames with every release, is an example of “smaller impact” case.
Python is an example of high impact but requires rebuilds in dependency-levels order.
Handling those rebuilds has historically been done by the Ubuntu Foundations team and now moved to Debcrafters under the supervision of @ginggs.
What does it look like
Those steps are done in cooperation with the Launchpad team.
- A test archive is created and populated from an existing Ubuntu suite ($series-$pocket). We define the components and architectures to include.
A ppa can be added to the configuration, it allows us to provide specific changes we want to test for (a new toolchain version of example).
- Builds for the defined set of packages are queued.
Test rebuilds are prioritized in the order main, restricted, universe then multiverse. All of these are prioritized lower than regular archive and PPA builds to not disturb normal Ubuntu development. However, higher priority builds may still need to wait for a single build from a test rebuild to finish.
The builds can be viewed or retried in the archive view on launchpad, the page is similar to what you get for a ppa:
https://launchpad.net/ubuntu/+archive/test-rebuild-20250530-questing-coreutils
Note that the Ubuntu rebuilds are usually owned by this launchpad team, Ubuntu Archive Test Rebuilds in Launchpad , the team members are allowed to retry failing builds.
A report is generated using https://launchpad.net/lp-ftbfs-report and published (usually on people.canonical.com).
https://people.canonical.com/~ginggs/ftbfs-report/test-rebuild-20250530-questing-coreutils-questing.html
Examples
Let’s take the Questing Quokka cycle as an example where those rebuilds happened
- Regular end of cycle rebuild.
We usually do a rebuild at the beginning and one at the end of the cycle. The earliest one didn’t happen this time since Debian was frozen and we didn’t have enough changes in the archive to make it worth having one.
The end of the cycle one report was published there:
https://people.canonical.com/~ginggs/ftbfs-report/test-rebuild-20250910-questing-questing.html
We usually do one for newer Glibc versions since that’s a high impact package but this cycle it was ready quite late and we decided to have it covered by the end of the cycle rebuild instead.
- Rust-coreutils
Coreutils is another central component, since Ubuntu was switching to rust-coreutils this cycle we did a rebuild with it as default at the start of the cycle, before the change actually landed in Ubuntu. It gave us some early feedback and the opportunity to work with upstream on resolving some of the problems ahead of the default change in the archive.
- RISC-V RVA23
This rebuild is an example of a more restricted set. Our RISC-V toolchain was updated this cycle to target RVA23 so we did a rebuild to have an idea of the impact of the change. It is worth noting that the rebuild also included amd64 since that’s where arch:all binaries are built, it also had the advantage of giving us a hint on whether the build issue was also happening on amd64 and such more likely to be due to other factors.
https://people.canonical.com/~ginggs/ftbfs-report/test-rebuild-20250731-questing-rva23-questing.html
Potential improvements
There is some manual work we might try automate going forward
-
Retrying build failures. At least in the cases where the problem is due to the builder (typically a launchpad build failing without log).
-
Refreshing the reports. It shouldn’t be too difficult do automatically but it needs to happen at a time where there are no build pending (otherwise a failing build being retried would temporarily drop from the report)