Hello again folks,
Today I want to talk about some issues you may have noticed in the autopkgtest infrastructure, how we fixed it, and improvements that are shipping soon.
What happened?
As you may know, we deployed a new iteration of the autopkgtest infrastructure in early May with tests running on LXD-backed containers instead of OpenStack based VMs. This was followed by an extended period of outage where essentially no tests were being run. While we had tested the new testing logic in our staging environment, the increased load that came from running 10x the number of tests proved too much for our cluster-first approach.
After many hours of debugging by us (especially @paride!) and the LXD team, we believe the problem was us overloading the websocket connections going into the testing clusters by requesting too many containers in a short time. The LXD team has already landed a number of fixes that address this issue. In the meanwhile, we got around it by unclustering the remotes. Essentially, instead of having all tests for each arch be run on a single large cluster made up of roughly a dozen nodes, we now treat each node as separate remotes. This way, instead of flooding just a few nodes on the cluster with all the connection demands of autopkgtest, we can spread the workload evenly between all the nodes.
This has led to the dramatically increased reliability of the system we see today. While there are still some lingering issues during periods of high stress, the system performs much better than it was last month.
New improvements
Now that the immediate concern of firefighting is behind us, I want to talk about other improvements that we have shipped and will continue to ship in the near future. First off, the most noticeable one for the daily users of autopkgtest: the frontend has gotten a much needed facelift. I have updated the version of bootstrap that the frontend was based on to the most recent version, and took the opportunity to give every page a design pass while I was at it. Some highlights include:
- An improved home page design with more relevant information visible at a first glance
- A page where you can request tests, without having to know all the magic parameters you have to use
- A user page which is easier to navigate, featuring more accessible pagination and collapsible tables
- A search bar to jump directly to a package’s results
And last but not least:
- Dark mode!
Here’s a quick comparison of a before and after:
Looking forward
Of course, the improvements do not end with just frontend work. In the very near future, we will be adding some resiliency to the workers to make sure one bad worker does not repeatedly retry more tests from the queue, turning dozens of test requests into tmpfails while other workers are busy processing actual tests. Speaking of the worker, I also have an in-progress rewrite of the entire worker script, with the aim of making it much easier to test and maintain.
In the backend, we will be switching to a postgresql database, featuring the Canonical postgresql operator charm. This will allow us to move away from keeping all autopkgtest results as a sqlite database which lives locally on the frontend machine, and allow all charms access to the database. This greatly simplifies the logic of the charms, since all database writes no longer have to go through the frontend machine. Of course, there are other advantages such as being able to deploy more than one frontend for load balancing purposes without having to worry about a split-brain scenario, as well as taking advantage of more robust backups and potential access to a high-availability deployment of postgresql, should we need it.
The postgresql changes will be a great building block for the next set of changes that we are working on, which will be first-class treatment of PPA tests. Gone will be the days of having to read logs yourself (or, to be honest, having ppa-dev-tools read them on your behalf). You will be getting a fully featured results table for all of your PPA tests, including pass/fail information at a glance, version under test, triggers, and a retry button.
That’s all we have for now. Stay tuned for more updates!