Some recent `cdimage` changes for improved consistency

Project Discussion Release
, ,
1

Some recent cdimage changes for improved consistency

A bit of context

Here is a quick update on some recent changes happening on the release
infrastructure.

First, what do we call cdimage? That designate both the Launchpad
ubuntu-cdimage project, the server
actually running this code, and the exposed web service accessible here:
https://cdimage.ubuntu.com/. People used to work with all kinds of Ubuntu images
usually know this service well, because that’s where they’re used to grab their
daily images.

But the file structure exposed here is notoriously difficult to figure out, and
a lot of cruft accumulated over the years. Given that we’re working on
modernizing the whole machinery, and that the present cycle is the first interim
after an LTS, now is the perfect timing to rollout changes that could potentially
impact many consumers of the service.

The changes

So, what did we roll out recently? Well, @utkarsh worked on putting together two
very welcome changes in the code:

Feel free to go have a deeper look, but let me give you a high level view that
should hopefully be enough for most people.

Simplifying the tree a lot, here is what we had before:

/
├── daily-live
│   └── there you found the daily stonking `ubuntu` images
├── noble
│   └── daily-live
│       └── there you found the daily noble `ubuntu` images
├── releases
│   └── noble
│       └── there you found the released noble `ubuntu` images
├── ubuntu  ← only the "ubuntu" project behaved weirdly like that
│   ├── symlink to ../daily-live
│   ├── symlink to ../noble
│   └── symlink to ../releases
└── ubuntu-server  ← most other flavors behaved like this
    ├── daily-live
    │   └── there you found the daily stonking `ubuntu-server` images
    ├── noble
    │   └── daily-live
    │       └── there you found the daily noble `ubuntu-server` images
    └── releases
        └── noble
            └── there you found the released noble `ubuntu-server` images

And here is what the new tree looks like now:

/
├── ubuntu
│   ├── noble
│   │   └── daily-live
│   │       └── there you find the daily noble `ubuntu` images
│   ├── stonking
│   │   └── daily-live
│   │       └── there you find the daily stonking `ubuntu` images
│   └── releases
│       └── noble
│           └── there you find the released noble `ubuntu` images
└── ubuntu-server
    ├── noble
    │   └── daily-live
    │       └── there you find the daily noble `ubuntu-server` images
    ├── stonking
    │   └── daily-live
    │       └── there you find the daily stonking `ubuntu-server` images
    └── releases
        └── noble
            └── there you find the released noble `ubuntu-server` images

What changed?

  • The devel series daily builds now behave exactly like other stable series, by
    having their own per-project folders to store the daily-live directory.
  • The ubuntu project now behaves exactly like the other projects, instead of
    invading the root directory directly with both its daily-live and releases
    folders.

Of course, in the real tree, many daily-live also have a daily-preinstalled
sibling, and noble currently has its resolute sibling, because those are the
two stable series for which we still build images.

NOTES: at the time of publishing this, the content-cache on top of the
web-service still hasn’t fully pick-up the change, so you might experience
some inconsistencies in the coming few hours.

Breaking changes

  • The /releases folder at the root is now located in /ubuntu/releases.
    We have put a redirect in place to hopefully mitigate broken links, but please
    update your pointers as well to follow the change.
  • Daily builds for stonking now have an additional /stonking/ nesting level.
    Given that these are daily images for an unreleased series, we deliberately
    didn’t put redirects in place, to force consumers to adapt to the new path.
    That’s in preparation for further changes coming in the next few months, and we
    take the occasion to discover what breaks early-ish in the cycle.

Closing notes

We hope you’ll find the changes to be welcome, and we’ll be very happy to
hear from you if any of those changes broke your workflow. So please come to
us, nothing is set in stone, and we’ll do our best to find an arrangement if
something here is making you sad :wink:

12 Likes
2

Thank you for the update!

Would it be possible to add a symlink pointing to the latest series? It’s very helpful for things like CI to have a stable path to the equivalent of ubuntu:devel. With this new tree, for every new release, we now need to go and update paths to point to the latest one.

For instance:

/
├── ubuntu
│   ├── devel
│   │   └── symlink to series of the latest/devel series of `ubuntu` (e.g., `stonking`).
4 Likes
3

The release directory seems to be inconsistent to me compared to daily-live (e.g. ubuntu/releases/resolute/release/ vs ubuntu/stonking/daily-live/). Is there a reason for having ubuntu/releases/resolute/release/ instead of ubuntu/resolute/release/?

2 Likes
4

Well, in principle, we could. But the overall artifact storing story is undergoing more fundamental changes, like moving from filesystem to object storage, for better flexibility on the infrastructure side. Investigating how to do that change in cdimage’s code would likely be a waste of time, both to do it, and to support the devel URLs in the upcoming redirect farm that we’ll have to set up.

That means I’d recommend sticking to hardcoding stonking for now if that’s OK with you. The URLs in your CI will require another change before the next cycle anyway (hopefully). We’ll make sure to provide a devel symlink in the next infra.

Good question. My grand-parents unfortunately weren’t even born when that tree was designed, we’d have to ask to someone way older! :older_person:

Jokes aside, from what I’ve gathered reading IRC logs from 2004, cdimage was already there in some form, and my guess is it just evolved organically with time, so it never was really designed with the whole picture in mind.
Now that we’re building a storage infrastructure from scratch, we actually get a chance to fix that, and hopefully improve things.

1 Like
5

It actually was (being that grey old timer here that was aruond), but the picture has changed :wink:

1 Like
6

Right, true, I implicitly referred to the 2026 whole picture in that sentence, and I’m sure the tree made sense at the time, before the introduction of (in no particular order):

  1. many flavors
  2. many architectures
  3. preinstalled images
  4. netboot
  5. ubuntu-base tarballs
  6. ubuntu-core
  7. WSL
  8. “almost flavor” products (:eyes: ubuntu-server)
  9. product-specific subtrees (:eyes: nvidia-tegra)

I’d actually be interested to chat with you on that part of history whenever I bump into you at some sprint. Nothing specific, just curious :slight_smile:

2 Likes