Hi everyone,
Today I want to share an update on the work that has recently been happening on the Ubuntu Merge-o-Matic service, as well as some thoughts on where we could go from here.
1. What is Merge-o-Matic?
As you probably know, Ubuntu is based on Debian. It means most of our packages come from there. We try to contribute fixes and improvements back upstream and to keep our packages “in sync” as much as possible, but we have to carry changes (which we also call “delta”) in some of our packages. Sometimes this is temporary, when we want to land fixes or new versions without waiting, but sometimes it is longer-term because we want to provide a different level of support or user experience.
Merge-o-Matic was created in 2004 by Scott James Remnant as a tool to help Ubuntu developers rebase their packages on Debian.
The Ubuntu Merge-o-Matic produces automated merges of the differences between source packages in two distributions, as well as collections of useful patches to aid manual merging or review.
For the past 20 years, Merge-o-Matic has been identifying the “base” version of packages, calculating the diffs between Ubuntu and Debian, and generating the HTML reports that drive our merge workflow.
2. Technical details and maintenance history
Merge-o-Matic saw some improvements over the years but has not changed fundamentally since it was introduced.
The way it works is basically:
- Create a local pool with the content of the Debian and Ubuntu archives.
- Go through the Ubuntu packages and compare their versions with the Debian ones. If the package has an Ubuntu delta, look for the common base between the Debian and Ubuntu versions.
- Perform a “3-Way merge” (trying to apply combined changes from Debian and Ubuntu to the base).
- Export artifacts (Patches, updated changelog entry, …).
- Generate HTML reports.
In recent years, the code has been ported to Python 3 thanks to Colin Watson, Zixing Liu, and Skia.
3. Landing the Python 3 port and Charming
Until recently, the service in production was still using Python 2 and running as a cronjob on a Bionic VM, mostly manually set up.
There is currently an Ubuntu Engineering effort to charm our services as part of moving them to a new datacenter. I’ve been charming a few services in the past cycle, and Merge-o-Matic was next on the list.
My first step was to merge the WIP Python 3 branch Skia had working.
Then (with the help of Gemini) I moved the statistics code away from pychart (which is deprecated and was included as a bundled local copy hacked to work with Python 3) to matplotlib and refreshed the style of the page a bit .
Next was the charming part. I created a new repository, moved the codebase under app/, and started to write the machine charm around it.
The goal was to make running Merge-o-Matic as simple as doing a juju deploy ubuntu-merges.
Instead of manually configuring Apache, setting up DNS, and wrestling with firewalls, the logic is now encapsulated in the ubuntu-merges-operator. The charm installs the application, configures Apache, and manages the systemd timers that trigger the merge reports .
4. The Result
The service is currently running in our new datacenter (PS7), fully managed by the charm. You can see the result at:
https://merges.ubuntu.com/
You might notice that the default reports are shorter than before. That’s because the local deb pool wasn’t migrated to the new instance, and Merge-o-Matic isn’t able to download the “base” version from Debian if that is not in the current archive. Those entries are still there but listed under the manual variant of the reports.
It also means it will pick new bases as packages get updated, and the problem will resolve itself over time. (I could manually try to import those old bases, but my backlog is long, and there are other, more important tasks I would prefer to focus on).
5. Going Forward
Now let’s talk a bit about the next steps.
Merge-o-Matic currently serves three primary purposes:
- An Index: A dashboard showing what needs merging.
- An Artifact Generator: Producing the
.patchand.difffiles used by developers to perform the merge. - A Patches Index: The same service is also hosting patches.ubuntu.com, which provides the delta of each package as a patch.
Regarding 1: This is something we still use. We need an index of the merges that need to happen. Some Canonical teams have their own reports and use Launchpad bugs to track their merges, but we also need an overview at the distribution level.
Regarding 2: This can probably go away at this point. It is much easier nowadays to do merges using git-ubuntu.
Regarding 3: These are currently referenced by the Debian PTS, which gives an easy way for Debian maintainers to see the delta Ubuntu carries for their packages. git-ubuntu is probably a better alternative here as well, providing not only the diff but often a rich VCS history explaining the changes.
Next steps:
I’m proposing that we stop the 3-way-merge patch generation now, as well as patches.ubuntu.com once we have worked out how to provide what the Debian PTS needs via git-ubuntu. After working with Debian to replace the current patches.ubuntu.com/PTS integration, I would like to stop that part of the service as well.
These steps would simplify the codebase and make it lighter in resource usage.
Then…
Matthieu recently wrote about his experiments reworking the merge UI in AI for Ubuntu tooling modernization, which is indeed overdue and would be a nice thing to work on (if anyone is interested in helping, please let us know).
From a Debcrafters perspective, we also started thinking about how we could build a better merge workflow based on Launchpad and git-ubuntu. We would love to see the standard merge experience become as simple as “review a git-ubuntu pull request generated by the service and approve it on Launchpad”.
There is work to do before we get there, and everyone is currently busy with the upcoming LTS and the services migrations, but I hope we will be able to start pushing some of those ideas forward next cycle.