How we delivered the new APT solver in 25.10

As previously announced, Ubuntu 25.04 ships with the new APT solver as the default.

To achieve this, during the last cycle, we implemented a platform to evaluate the correctness of the solver as outlined in Evaluating the new APT solver in 25.04. In this cycle, we used the errors reported to the error tracker to improve the solver to a quality suitable for release.

Here’s how the system works: The error tracker (errors.ubuntu.com) receives textual error reports from a program called whoopsie and stores them inside a Cassandra database. One challenge we faced was the size of the solver dumps we sent this way: They exceeded database limits. We realized that we could overcome this issue by minimizing them to only include relevant packages , compressing them with zstd, and encoding them into the error report format as a base64 value.

@skia figured out how to query these reports in Cassandra, and implemented APIs to download reports by date. Afterward, I implemented some Python scripts to retrieve and extract the error reports before passing them to a custom test suite (built on top of https://salsa.debian.org/apt-team/apt-tests).

This differential test suite lets me compare the solver in a development git branch against both the classic and new solver in the installed apt version. Over the course of several months, I analyzed over 430 automatic reports in this way. For every solver change, I review the output of the test suite to determine if the change is an overall improvement or regression in solver quality.

This effort led to the identification and resolution of multiple bugs.

One of the most impactful outcomes is the handling of manual packages. The solver was allowed to remove them for the purpose of the evaluation, and it sometimes ended up removing them even when not necessary. Starting in 25.10, the solver will not suggest removing manually installed packages if another solution exists, but it will still do so if necessary to complete installations or removals.

Another issue we identified involves the handling of packages that are newer than the version available in your configured repositories. The solver considered them to be obsolete and was overly eager to replace them with a different alternative if one existed, such as in a dependency satisfied by multiple packages such as Depends: postfix | mail-transport-agent.

While the solver is the default now, the work is not done. There are still further opportunities to improve its reliability (producing safe results), consistency (producing the same results across systems), and performance (especially for complex problems). We will continue analyzing more reports from 25.04 systems, and look forward to receiving your bug reports for 25.10. Beginning with 26.04, we may use the solver evaluation mechanism to guide further improvements.

8 Likes

Moderator Note:

Folks, this is a technical explanation in the Foundations category.
Comments and questions are welcome, but should show a high level of understanding.

It’s not the place for general complaints or opinions.
It’s not the place to ask how to install Steam or other support questions.
Off-topic comments may be removed to keep the signal/noise ratio of this topic high.

5 Likes