An agentic workflow to triage Ubuntu FTBFS

A couple weeks ago my application for Ubuntu Contributor was approved by the DMB :tada::tada: and with this new recognition I felt empowered to go and do some good for Ubuntu and contribute a few packages.

Looking for contributions is sometimes a daunting task for new and even seasoned contributors, there are a lot of places to look at within the Ubuntu package life cycle where it is possible to contribute (this previous post cover this life cycle at a high level):

  • Merges: merges.ubuntu.com is the first gate for Debian packages to migrate to Ubuntu, and sometimes new versions just don’t merge with Ubuntu changes and require work.
  • FTBFS: When a package doesn’t build on LaunchPad, it is recorded as a Fail to Build From Sources and ends up on the ftbfs report: http://qa.ubuntuwire.com/ftbfs.
  • Excuses: Freshly built packages on LaunchPad go through Autopkgtest which may sometimes fail due to new dependencies or requirements in the other packages in the distro (see Visualizing and Exploring Ubuntu Excuses, which covers excuses in detail).
  • Bug Fixing: LaunchPad has plenty of them to choose from. If there’s a package you really care about, that is not a bad place to start.

So, I chose to look at FTBFS in universe.

Finding FTBFS

The main page I looked at was http://qa.ubuntuwire.com/ftbfs/universe (filtered on universe). At the time, it had about 1000 FTBFS, so to filter them even further, I chose to focus on amd64 failures as they would be easier to reproduce locally. Then for each FTBFS, I started a fairly predictable process:

  1. Searching for a candidate on http://qa.ubuntuwire.com/ftbfs/universe.
  2. Looking for existing FTBFS bug on Ubuntu or upstream in Debian where applicable.
  3. Reproducing the FTBFS locally with sbuild.
  4. Once confirmed, look for a solution.
  5. Created FTBFS bug in LaunchPad.
  6. Once a tentative fix is available, confirm with a new build.
  7. If build succeeds, start the submission process.
  8. Create PPA and upload to make sure it builds across architecture.
  9. Run autopkgtest on that PPA.
  10. Submit Merge Request.
  11. Repeat for another candidate.

The Ubuntu team has done some fantastic improvements to the Ubuntu Project documentation, and it is worth checking there first when blocked.

I was able to fix 2 packages [1] [2] following this approach, but while I’ve done several uploads in the past, something has changed since my last upload. My brain has this new function prompting me:

This is a repetitive process. Should I be able to accelerate this at scale with AI?

Living in the agentic era

So I embarked on an experiment to design an agentic workflow to help me process FTBFS faster and make the process more automated.

These were my requirements:

  • The workflow should be designed as a directed graph with various phases similar to the current manual process.
  • Pipeline management should be flexible and extensible.
  • Any phase that can be deterministic should be.
  • Workflow should identify patterns (over thousands of build logs, there has to be some that are related).
  • Agent-assisted phase should use the appropriate model (low/medium/high cost) based on the level of judgment required.
  • Human should remain in the loop to approve packages from one phase to another, especially to:
    • Review potential high cost operations.
    • Review operations that require heavy building capacity.
    • Review write operations.
    • Open bugs or merge proposals on LaunchPad.
  • The process needs to be heavily observable and measurable.
  • A web UI to observe and help assist with the decisions.
  • Integration with LaunchPad for authentication.

AI Setup

For this operation, I used Claude Code with the default model, Opus 5.5 at the time, and provided a fairly large prompt with the requirements above. For the deployment I’ve set up 2 different possible uses: Claude Code in headless mode or Open Router. For this experiment I favored using OpenRouter with a dedicated API key allowing more fine grained tuning and monitoring.

TIPS: for large pieces of software like this, ask for the plan to be split in phases, keep a documented copy of the plan, and run one phase at a time after clearing the context. This makes a huge difference and will save you millions! (of tokens).

Ubuntu agentic FTBFS triage

After a few hours, I had a working workflow that started looking at the current list of FTBFS in universe on amd64 without open bugs.

The pipeline

The FTBFS pipeline: 8 stages built

The pipeline is designed to be cost-effective and extensible. Everything that can be deterministic is, so only three steps require an agent. Each of these steps can be customized with a dedicated agent with a model (small, medium, or large) that can be used for the task at hand.

excerpt: cut the Launchpad log to keep only the failure deterministic

classify: help clarify the failure type and potential issue cluster deterministic

facts: grab upstream details (Debian versions, bugs, builds) deterministic

triage: per cluster: category, fixable, action small model

diagnose: root cause of the problem, propose fix strategy and risk medium model

reproduce: use configured builders to reproduce build with sbuild deterministic

dev: an agent edits the sources and proposes a fix medium/large model

verify: build the proposed fix; on failure, back to dev, twice deterministic

The pipeline is defined in a pipeline.toml file that allows for easy tune up and extensibility in the future.

A tour of the dashboard

Everything the agents do is visible and reversible. A small local web app reads the same SQLite database as the pipeline, so a maintainer can watch it run live and act on its output.

Overview. The latest snapshot of the FTBFS page, what changed since the last one, and the pipeline with live counts per stage. Each count links to its items.

Next steps: The maintainer’s inbox. Verified fixes wait for review with their debdiff and the agent’s notes; one click accepts or rejects. Sync and merge candidates from Debian are listed apart since they need no patch at all.

Packages page: That page allows a deep dive in each package going through the pipeline and each of them has a detailed investigation report, including: failing builds per arch, the log excerpt, classification, diagnosis, and every agent attempt with its prompt, transcript, cost, and debdiff.

Cluster page: When you start looking at hundreds of build failures, you quickly find clusters of problems potentially caused by systemic change in Ubuntu (new GCC, deprecated library, etc). Identifying those clusters can enable you to fix multiple packages at the same time**.** For example, the tool highlighted cmake-minimum-version as a potential cluster, which – after some brief search – was explained clearly in this Debian MBF.

Results and costs

The initial pipeline run has processed and organized 421 packages in clusters and led to 38 build verified debdiffs and the cost was $29 of total LLM spend, including every experiment and A/B run along the way. That is under $1 per verified fix before any tuning.

Of course, these debdiffs are not ready to be submitted without deeper reviews, but they could usually put the maintainer or contributor on the right path. Agent prompts will likely need some tuning to handle specific Debian and Ubuntu practices. For example, a few debdiffs recommended disabling lto and providing a patch for it, but that isn’t how it’s done in Ubuntu as we use the lto-disabled-list package instead.

Cost details for the run using OpenRouter with Haiku, Sonnet, and Opus (small, medium, large)

Stage Unit Results Cost per result
Triage cluster 421 $0.003 to $0.008
Diagnose cluster 296 $0.03 to $0.07
Reproduce item 146 0 (local sbuild)
Dev item 42 $0.08 to $0.23 per attempt
Verify item 38 pass, 4 fail 0 (local sbuild)

What’s next?

I can’t wait for Ubuntu 27.04 Tenacious T-Rex :t_rex: to be open to put this tool to the test and add some features to improve output quality and make the workflow more natural. Couple features I’d like to see next:

  • More integrated development Experience: Once the machinery is tuned and the quality of the results increase, we could consider automatically creating FTBFS bugs and merge proposals, so the development could continue with Launchpad.
  • Multi-arch verification: right now, this is limited to amd64, but it would be great to leverage Launchpad PPA to enable multi-architecture support or allow new LXD builders for different architectures. (I have an arm64 laptop and should be going on that ;))
  • Extended Launchpad support: right now, there’s minimal Launchpad integration for authentication. Deeper integration could allow different access and capabilities for various Launchpad members (Core Dev, MOTU).
  • New Adversarial review phase: Adding a second agent that tries to break each debdiff before a human sees it.
  • Prompt tuning.
  • And why not add Ubuntu Excuses as a new input stream…

Conclusion

This was a fun experiment, and this will remain a fantastic helper tool to identify potential places where I can contribute to Ubuntu and amplify my potential impact (a great assist for +1 Maintenance!)

I also hope it can be used as inspiration for more distro agentic workflows that take care of a good portion of the Archive maintenance and free up time to do more exciting things.

Demo is live at ftbfs.oclemenceau.com and source code available at github.com/mclemenceau/ubuntu-agentic-ftbfs.

Feedback is welcome!

6 Likes

Thanks man for sharing this helpful blog