About a year ago, we embarked on a journey to modernize the Ubuntu release process. Our goal was to adopt modern release-engineering practices while retaining the resilience and stability of our six-month release cadence.
Today, I would like to cover our progress toward a more flexible, transparent, and resilient testing framework.
Ubuntu release management
The Ubuntu Release Management Team is responsible for the lifecycle of Ubuntu release artifacts, which includes building, testing, and publishing these artifacts.
We build and publish 38 artifact images for Stonking, like the Desktop and Server ISO, and several preinstalled images, across all supported architectures.
A few examples:
stonking-desktop-amd64.isostonking-desktop-arm64.isostonking-live-server-amd64.isostonking-preinstalled-server-amd64.img.xzstonking-live-server-arm64.isostonking-live-server-riscv64.iso
and many more, including Ubuntu flavors. You can find them all on ubuntu.com/download (when they are officially released).
Ubuntu artifacts are built daily from packages that have already passed their package-level testing, including unit tests and autopkgtests, and successfully migrated to the release pocket. Artifact testing then adds another layer of validation.
Artifact testing
Release management runs a variety of automated tests on freshly built artifacts every day, including basic ISO validation done by UTAH: Ubuntu Test Automation Harness. Some run more elaborate automated GUI testing with Ubuntu GUI Testing, such as testing the installer with various configurations. We also have more niche testing, including automated testing for the mini.iso artifacts that were introduced this cycle.
Finally, we have Ubuntu manual testing, which can happen at any time but usually gets more intense 
around major milestones (beta and release weeks).
The daily automated tests are handled by a Jenkins instance. Results are available and observable within Canonical infrastructure and the Release team reviews them daily and notify various teams of regressions and problems discovered.
It would be nice to make these results public, wouldn’t it?
Test Observer
After 15 years of great service, the ISO Tracker service had become a maintenance burden, and we decided to sunset it. Last year, we announced its retirement and the introduction of Test Observer as a replacement.
While the initial priority for Test Observer was to provide a stop-gap solution for Ubuntu manual testing during release weeks, it also gave us another really nice feature: a stable API built on a modern code base and ready for modern deployment.
So, why not push more test results there?
Aggregating test results
During the 26.10 cycle, we tasked the Release team with publishing all of our daily automated test results to Test Observer. I’m pleased to report that the following reports are now public:
- Static ISO validation
- Desktop GUI testing
mini.isosmoke testing- Upgrade testing; yes, we test all sorts of upgrades from LTS to LTS on various Ubuntu flavors on a daily basis!
Where can I see these results, you might ask.
At the same place you can look at the manual testing:
For example, here you can see that three tests environments are available:
- cdimage.ubuntu.com: this represents the artifact build itself; if that step fails, there would be no artifact in Test Observer to test.
- platform-qa-jenkins: this is the Jenkins orchestrating our automated tests.
- user manual tests: available to anyone who wants to run manual tests.
This screenshot shows all the various automated tests running against this artifact, including end-to-end installation testing using the GUI testing framework.
While Test Observer stores our artifacts and testing results, navigating the UI to aggregate data could be cumbersome. Introducing Ubuntu Mission Control! 
Ubuntu Mission Control
So, last year, before the Ubuntu 26.04 LTS release week, I put together a tool that took advantage of the testobserver API. I called it Ubuntu Mission Control. The intent was to offer the Release team a single pane of glass to be able to quickly identify release blockers, new artifacts, tests failures, and new bugs.
During release week, Ubuntu Mission Control proved very useful for the team!
It offers several features:
- organizing artifacts by daily swimlane to quickly identify recent and stale artifacts,
- KPI to monitor test progress (manual only then),
- bugs discovered by testers, and
- approved artifacts (which is really neat to monitor on release week).
Per-artifact detail view including the test results
Test transparency
As we pushed so many test results to Test Observer this cycle, I thought we could share all the details and metrics we have about them.
So I’ve implemented two additional views allowing to monitor current test runs (all the test runs on the latest artifacts:
And a testing history dedicated page that allows us to observe test results over the last 7, 14, 30, 60, or 90 days:
On this page, you find a section showing all the testers, human and automated, and below is the list of the top testers during the 24.04.5 release week:
For the 24.04.5 release, we received many test contributions. Shout out and thank you to all the contributors who make our release week possible and so exciting 
Special thanks to our top three manual-testing contributors for 24.04.5:
- @arraybolt3
with 15 manual tests - @fuseteam
with 13 manual tests - @leok
with 11 manual tests
Our testing modernization journey is just starting
Our goal is to run as many automated tests as possible, often on as many artifacts as possible. As you explore these dashboards, you will see some gaps in our testing - we will want to close them. Constructive feedback welcome! 
We are in the process of refining our set of test harnesses. We will likely deprecate some of them (sorry, UTAH) in favor of more modern tooling, potentially creating some new ones as well. That said, we won’t change where the results end up: Test Observer.
Contributor’s journey
We also want to make it easier to submit tests, so they can be included in the Release Management’s machinery. We are working on an improved test-language definition to enable anyone to contribute additional tests. Ideally, anyone should be able to design their own tests, validate them end to end against their artifact of choice, and submit them to the Release team for review and inclusion.
If you want to contribute tests, contact me or the Release team, and we’ll be happy to help!
There’s still much more work to do, but our team is growing, and we will be able to accelerate results in the next few cycles. Stay tuned!
That’s all folks.
Resources
Ubuntu Test Observer - https://tests.ubuntu.com/#/images
Ubuntu Mission Control - Dashboard - Github
UTAH - https://launchpad.net/utah
Ubuntu GUI Testing - https://github.com/canonical/ubuntu-gui-testing
Ubuntu Manual Tests - https://github.com/ubuntu/ubuntu-manual-tests
Retiring the ISO Tracker - discourse post