On April 26th, we released Ubuntu 26.04 LTS Beta, and this was our first opportunity to experiment with tests.ubuntu.com (see Retiring the ISO Tracker! Introducing tests.ubuntu.com for 26.04 LTS).
Since this was a fairly new experience for most of us, we took the time to look into the numbers, gather feedback, and put together a quick retrospective to make sure that the next iteration goes even better.
Beta week effort 
First, let’s look at a few raw numbers for what was done during the Beta week.
- Out of 34 builds, we released 31 images
- 3 images stayed on hold and didn’t get released
- 79 manual tests were performed and recorded in tests.ubuntu.com
- 6 bugs opened and documented in tests.ubuntu.com
Community support 
While our daily images are going through numerous automated tests, the release week is always a great opportunity to have a human touch our products before releasing them. We make sure that established mandatory manual tests are successful.
I wanted to take a moment to recognize some outstanding support we received from the community for Beta artifact manual testing. I especially want to take a moment to recognize:
- @arraybolt3 for contributing 32 manual tests results
- @leok for contributing 15 manual tests results
- and @fossfreedom with 4 manual tests.
On behalf of the release team: Thank you! 
The top three manual testers are community members with 51 manual tests out of 79, which is 65% of the submitted manual tests. 
Bugs 
Due to some issues with the ACL on tests.ubuntu.com, it was very difficult to open a bug against a specific manual test. So, while the initial perception was that no bugs were open, it turns out testers have been diligent in opening bugs during their testing - they were just not showing up in tests.ubuntu.com, or simply couldn’t be attached to the test case.
Looking deeper into the data, we found out that testers had been leaving bug details in comments while documenting their test results. Below is just a highlight of some of the bugs identified.
Feedback 
Adjusting to a new system is never easy, and there is some room to grow for tests.ubuntu.com. We greatly appreciate all of you who provided numerous pieces of feedback during and after the testing week. Below is the list of the most critical problems that were reported, as well as their current status.
ACL issues preventing individual actions (respin, open bug) - Fix In Progress 
- Fine grained team ACLs · Issue #609 · canonical/test_observer · GitHub
- There is still work required to enable appropriate ACL between Launchpad teams and test_observer. In particular, this bug made it impossible to link Launchpad bugs with manual tests unless you had elevated privileges.
User experience issue with URL access - 
- It wasn’t possible to copy the URL of each artifact. Instead, each click would trigger a download.
- [TO-340] Enable copy link by rpbritton · Pull Request #691 · canonical/test_observer · GitHub
Ctrl-click to open in new tabs - Fixed -
- When you want to open each artifact into a new tab and rapidly CTRL+click several artifacts.
- Improve mouse navigation and enable CRTL+Click by mclemenceau · Pull Request #685 · canonical/test_observer · GitHub
Test coverage overview - TBD 
- This one is a little more complicated as it involves more changes, but the idea would be to provide an aggregate view of test progress without having to go into the artifact for more details.
- Show inline aggregate test status in a new Tests Status column · Issue #549 · canonical/test_observer · GitHub
- Have a better overview of the artifacts' testing coverage · Issue #676 · canonical/test_observer · GitHub
–
We would like everyone to be prepared for release week in the best possible way, and there might be some time for a few easy fixes.
If you have any other questions, let us know.
Thanks again! 