Welcome to The Rockcrafter’s Diary, a series where we share the day-to-day adventures of building, maintaining, and improving Ubuntu Rocks. Across the series, different Rockcrafters will share what they’ve been working on, the problems they’ve encountered, the lessons learned, and occasionally the unexpected rabbit holes that come along the way. This is the first entry–and, as it turns out, supporting two new architectures is a pretty good way to kick things off.
Chapter I: Teaching Rocks New Tricks
One of the initial thoughts I had about this task was: “surely this one should just need a couple of lines in the rockcraft.yaml…” Well, yes and… no.
To be fair, a surprising number of them actually did. rabbitmq, curl, bind9… even squid mostly just needed the new platforms added and everything happily carried on. Those are the kind of PRs that make you think supporting new architectures is going to be a relaxing afternoon.
Then came Go and Rust, leading straight into the GCC rabbit hole.
Following the dependency chain
The dependency graph is roughly:
rust / cargo
│
├── gcc
│ └── cpp
│ └── binutils
│
go ─────┘
The interesting part is that this stack already existed for Ubuntu 24.04 and 26.04 on amd64 and arm64, but the IBM architectures (s390x and ppc64le) weren’t there yet.
22.04 was even more interesting: Nothing existed.
So instead of simply adding two architectures, the whole compiler stack had to be backported first, before Go and Rust could even be considered.
That meant binutils, CPP, GCC… and eventually Go and Rust themselves.
Binutils: where the adventure actually started
I started with binutils. The plan looked simple enough:
- create the new slices for s390x and ppc64le on 26.04;
- split the work into three PRs (binutils, CPP and GCC);
- once happy, backport everything to 24.04;
- finally create the 22.04 slices from scratch.
The first part went fairly smoothly. The second… less so.
During the review of the 24.04 changes, it became apparent that the cross-compilation tests behaved differently depending on the architecture. That’s not necessarily unusual, but something felt off.
After quite a bit of digging—and rerunning everything with set -eu enabled—the culprit started to appear.
Some of the tests were silently succeeding despite missing the libopcodes cross-compilation library. Enabling stricter shell options immediately exposed the issue.
Then another surprise appeared: The dynamic linker installed by the cross-compilation packages wasn’t named consistently across architectures, causing this innocent-looking line to fail on s390x and ppc64le:
linker_lib="$(ls "$rootfs_ld"/lib*/ld-*.so*)"
At that point the easiest way to verify everything was simply running emulated containers for both architectures and testing the cross-compilation setup directly.
Huge thanks to @lczyk for helping with the emulated environments—it made reproducing and validating the issues significantly easier.
Finding these problems meant another round of backports… and then forward ports… and then backports again… The glamorous life of keeping multiple Ubuntu bases in sync.
The nice side effect, though, is that during this work s390x and ppc64le integration tests were enabled in chisel-releases! So from now on, new slices are exercised on all four supported architectures instead of only amd64 and arm64, which should hopefully prevent this class of bugs from sneaking in again.
CPP: AI does pretty well… until cross-compilation appears
By the time I reached CPP, the 26.04 work had already landed. That meant backporting to 24.04 became an excellent candidate for Claude Sonnet 4.6.
Honestly, it did a pretty decent job. 22.04, however, was a different story: CPP didn’t exist there at all, so everything had to be sliced from scratch.
Fortunately CPP is considerably smaller than binutils or GCC. It’s essentially one slice per architecture plus the accompanying tests.
The amusing part is that the tests ended up being larger than the slice definitions themselves.
Only a handful of slice definition files…
…and around eighteen test files to make sure everything behaves correctly across four architectures.
As expected, AI was very good at mechanical work. It happily inspected packages across architectures, downloaded .deb files, explored their contents and generated a reasonable first draft using the previous PRs as context.
Where it still struggled was the multi-architecture cross-compilation logic. That’s still very much a human job.
During this process I also stumbled across an interesting packaging detail: cpp-11-x86-64-linux-gnu lives in Universe for every architecture except s390x, where the package doesn’t exist at all.
That raised an uncomfortable question. Would this turn rust and go into pro rocks for 22.04? Digging a bit deeper revealed something interesting: The standard Go and Rust installations from apt don’t actually install the cross-compilation cc1 binary.
This means including it inside the rock would be inconsistent with the normal package installation, and would unnecessarily pull universe content into something we’d rather keep publishable as a normal rock with the same support window as Ubuntu itself.
That investigation ended up changing the slicing strategy. The real cc1 compiler already lives inside cpp-11, while the Universe packages are only there for cross-compilation.
So GCC in 22.04 probably shouldn’t include the cross-compilation pieces at all. Instead, CPP should provide the native compiler, and GCC should be sliced accordingly.
Sometimes supporting two new architectures ends up teaching you something entirely unrelated.
GCC: mostly AI, mostly
By the time I reached GCC the workflow had become fairly familiar.
For 24.04, AI handled most of the scaffolding surprisingly well. Most of the manual review ended up fixing the usual places where LLMs still get confused: multi-arch layouts, cross-compilation edge cases and small slice definition details.
22.04 again required more manual intervention. The scaffolding was useful, but several slices needed adjusting, a few tests complained about missing headers already provided elsewhere, and some symlinks were being recreated despite already existing.
Nothing particularly dramatic. Just the normal “the last 10% takes 50% of the time” kind of work.
Finally, Go and Rust
Compared to the compiler stack, Go and Rust were almost relaxing. For Ubuntu 24.04 and 26.04 the required compiler infrastructure was already available, so it mostly came down to fixing tests and making everything pass on the new architectures.
22.04 is still future work. But the important part is already done. Once binutils, CPP and GCC are available, bringing Go and Rust along becomes a much smaller task.
The final boss: building the Rocks
After all the compiler work, package slicing, cross-compilation fixes, architecture-specific debugging and more backports than I’d like to count…
There was finally the satisfying step: Actually building the Rocks! As expected, this entire go/rust story was pretty much about the compiler slices, so the rocks didn’t require much work other than just adding the new platforms to the rockcraft.yaml.
There’s something particularly rewarding about watching an OCI image build successfully on an architecture that wasn’t supported a few weeks earlier. It’s a nice reminder that all those tiny slice definitions, tests and review comments eventually become something tangible.
So, what’s next?
The immediate goal is getting everything released to the Edge channel and validating the stack end-to-end. Then after some time goes by and everything is all right, the rocks will land in stable.
Looking back, this started with what sounded like a relatively innocent goal:
“Let’s support s390x and ppc64le.”
A few weeks later it turned into compiler stacks, cross-compilation quirks, linker names, AI-assisted slicing, package archaeology, integration testing and enough backports to keep three Ubuntu releases aligned.
Exactly the sort of thing that somehow always sounds much simpler before you begin.
Until next time,
@alesancor