Almost everyone loves cookies - famously used when trying to convince you of the dark side.
Almost everyone loves tests - famously used when trying to convince you of the code quality.
Ingredients:
- 120g of Cooperation
- 110g of Quality Assurance
- 60g of Network Test Complexity
The Preparation
1. Sift the 120g of Cooperation over all your peers, ignore country, company or philosophy borders
Cooperation is for the better of everyone. True in general, even more true in open source and sadly worth reminding everyone in today’s aggressive geopolitical landscape. There could be many books about cooperation and I’m sure there are, the aspect that I touch on today is that between Ubuntu and its upstreams. Those are the various upstream projects for the code as well as Debian for the packaging. Each of those might differ in their choices and preferences, thereby there are a few things that can’t be shared and will stay different - how boring would an “only one opinion” open source world be.
At the end of the day, we all improve the same solutions to make them more consumable for every user out there. Doing so is built on the reliable base of cooperation, upstreams that file bugs in distributions, distributions passing fixes to each other and to upstream, distributions backporting changes to releases used by many, generally helping each other to make the best of it, and much much more …
2. Weigh the 110g of Quality Assurance
Let’s add in some QA. In regard to cooperation this is the best of all ingredients, because here you can help yourself by helping others. Upstreams might spot an issue before it becomes an issue to end users. Distributions might have a wider reach of users and thereby use-cases and therefore can help upstreams to find and fix corner cases.
Testing is done in various ways: from per-commit actions on the platform of choice, via automated integration tests to massive scale out tests - and a lot in between. Today in this recipe we want to look at integration testing. In Ubuntu (and Debian) we follow the DEP-8 specification, as implemented by autopkgtest, for testing of packages and their reverse-dependencies. The result of that is an ever-growing set of tests executed whenever a package is changed as seen on https://ci.debian.net and https://autopkgtest.ubuntu.com.
3. Melt the 60g Network Test Complexity
The autopkgtest infrastructure has some peculiarities, in that it does not handle all the architectures equally, similarly to Debian’s DebCI implementation. Tests for some architectures are executed in virtual machines, while others are executed in containers. This is fine for most packages, but network testing sometimes requires access to low-level system components, which cannot always be provided inside a container. This led to a situation where many tests were just skipped or even never implemented because nobody wanted to go the extra mile in writing and running them on different platforms.
Now whisk the Quality Assurance into the molten Network Test Complexity – the step might be extra effort, but the cookies are extra tasty as a result.
4. Fold in the Cooperation
As part of his connectivity focus, @slyon worked on improving the tests of several network related packages, such as avahi, dnsmasq & WireGuard in Ubuntu.
As outlined above this is a perfect case where fixing it only in Ubuntu would only give us half the value. Not only is it the right thing to help each other (it is!), but even the Grinch would admit that implementing it in a way that helps both Distributions will help Ubuntu more. Issues can be spotted earlier, already when Debian applies new changes, thereby avoiding potentially slow feedback loops.
The approach that was taken can be split into 3 major categories (vanilla, salt, chocolate chips?): Containerization, implementation of new tests and fixes for existing tests. For all of them, we increased coverage (and reduced delta between Debian and Ubuntu) as much as possible, by making it run on major platforms, such as Ubuntu autopkgtests, DebCI and SalsaCI. This helps to make all maintainers aware of any issues and thus detecting failure scenarios early in the development cycle. We had good and friendly discussions with the upstream maintainers and were able to request a DebCI VM-testing exception in the case of NetworkManager. Overall, the testing improvements were well received by package maintainers, e.g.: “Thanks a lot, this is really nice. […] Keep those MRs coming!” – @biebl
- Containerized to enable test coverage on Ubuntu armhf and/or DebCI:
- Added integration tests for increased test coverage of reverse-depends:
- Avahi (upstreamed to Debian & SalsaCI !17)
- ModemManager (upstreamed to Debian & SalsaCI !26)
- Fixed & refactored deprecated tests:
Spread your testing one teaspoon each, evenly distributed for best test coverage and bake at 160 °C until you like the color (about 15 minutes).
Enjoy the tasty network test cookies!
- Cooperation is all-purpose flour
- Quality Assurance is granulated white sugar
- Network Test Complexity is unsalted butter
Not the best recipe, but the one that matched my structure.