Ubuntu Server Gazette - Issue 12 - Upgrading RabbitMQ across Ubuntu releases

Project Discussion Server
, ,
1

Upgrading RabbitMQ across Ubuntu releases

If you’re running RabbitMQ on Ubuntu, you might have encountered a challenging situation when upgrading your system from one Ubuntu release to another. The good news is, we’ve been working on a solution to make your life easier.

Jackie (a rabbit) looking at the camera.
Jackie (a rabbit) looking at the camera.1484×1113 174 KB

The challenge

RabbitMQ has strict upgrade requirements that don’t always align with the versions packaged in consecutive Ubuntu releases. It’s possible that a direct upgrade from the version in one release to the version on the next is not supported — RabbitMQ requires stepping through intermediate versions to properly migrate internal data structures.

Here are the RabbitMQ versions packaged for the last three Ubuntu LTS releases:

Ubuntu Release RabbitMQ Version
22.04 LTS (Jammy) 3.9
24.04 LTS (Noble) 3.12
26.04 LTS (Resolute) 4.0

The problem? To upgrade RabbitMQ from 3.9 to 3.12, you need to go through versions 3.10 and 3.11. Similarly, upgrading from 3.12 to 4.0 requires version 3.13 (the latest release in the 3.X series) as an intermediate step.

This left users in a difficult position: either stay on an older Ubuntu release or manually manage RabbitMQ upgrades outside the package manager. Neither option is ideal for production environments.

Stepping stone PPAs

We’ve created a series of PPAs that provide the missing intermediate RabbitMQ versions built for the Ubuntu release where your upgrade path starts, enabling a safe upgrade path across Ubuntu releases.

Data safety considerations

The packages in the PPAs are built from backported packages with minimal changes to increase compatibility. We’ve tested the complete upgrade path in controlled environments, including upgrades with running queues and live data. However, as with any major system upgrade, it’s better to be safe than sorry.

  • Back up your data before starting.
  • Test the upgrade in a non-production environment first.
  • Plan for downtime.
  • Check RabbitMQ’s status after each upgrade.

If you’re running a clustered setup, follow RabbitMQ’s cluster upgrade procedures, upgrading one node at a time.

22.04 LTS (Jammy) to 24.04 LTS (Noble)

Starting with Jammy running RabbitMQ 3.9, you’ll need to step through RabbitMQ 3.10 and 3.11 to upgrade to Noble.

  1. Add the RabbitMQ 3.10 PPA for Jammy:
    1. sudo add-apt-repository ppa:canonical-server/rabbitmq-3.10-jammy
    2. sudo apt update && sudo apt upgrade
  2. Enable all feature flags:
    1. sudo rabbitmqctl enable_feature_flag all
  3. Add the RabbitMQ 3.11 PPA for Jammy:
    1. sudo add-apt-repository ppa:canonical-server/rabbitmq-3.11-jammy
    2. sudo apt update && sudo apt upgrade
  4. Enable all feature flags again:
    1. sudo rabbitmqctl enable_feature_flag all
  5. Perform the Ubuntu release upgrade manually. Since RabbitMQ is installed, do-release-upgrade will block the upgrade because the versions before and after the upgrade are not compatible. Instead, upgrade manually:
    1. sudo sed -i s/jammy/noble/g /etc/apt/sources.list
    2. sudo apt update
    3. sudo apt dist-upgrade -o APT::Get::Always-Include-Phased-Updates=true

After the upgrade you’ll be on Noble with RabbitMQ 3.12, and the output of apt policy rabbitmq-server should be something similar to:

rabbitmq-server:
  Installed: 3.12.1-1ubuntu1.2
  Candidate: 3.12.1-1ubuntu1.2
  Version table:
 *** 3.12.1-1ubuntu1.2 500
        500 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages
        100 /var/lib/dpkg/status
     3.12.1-1ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages
     3.11.28-0ubuntu1~ppa1 500
        500 https://ppa.launchpadcontent.net/canonical-server/rabbitmq-3.11-jammy/ubuntu jammy/main amd64 Packages
     3.10.25-0ubuntu1~ppa1 500
        500 https://ppa.launchpadcontent.net/canonical-server/rabbitmq-3.10-jammy/ubuntu jammy/main amd64 Packages

You can now safely remove the PPAs by deleting the .list files in /etc/apt/sources.list.d/ and the trusted GPG keys in /etc/apt/trusted.gpg.d/.

sudo rm /etc/apt/sources.list.d/canonical-server-ubuntu-rabbitmq-3_10-jammy-jammy.list 
        /etc/apt/sources.list.d/canonical-server-ubuntu-rabbitmq-3_11-jammy-jammy.list 
        /etc/apt/trusted.gpg.d/canonical-server-ubuntu-rabbitmq-3_10-jammy.gpg 
        /etc/apt/trusted.gpg.d/canonical-server-ubuntu-rabbitmq-3_11-jammy.gpg

24.04 LTS (Noble) to 26.04 LTS (Resolute)

Starting with Noble running RabbitMQ 3.12, you’ll need to step through RabbitMQ 3.13.

  1. Add the RabbitMQ 3.13 PPA for Noble:
    1. sudo apt-add-repository ppa:canonical-server/rabbitmq-3.13-noble
    2. sudo apt update && sudo apt upgrade
  2. Enable all feature flags:
    1. sudo rabbitmqctl enable_feature_flag all
  3. Perform the Ubuntu release upgrade:
    1. sudo do-release-upgrade

After the upgrade you’ll be on Resolute with RabbitMQ 4.0.

Why can we use do-release-upgrade here? The check performed before upgrading from Noble to Resolute is smarter than the one in Jammy to Noble. Instead of blocking all upgrades when RabbitMQ is installed, it checks whether you’ve properly prepared for the upgrade by having RabbitMQ 3.13 installed and all required feature flags enabled. If you’ve followed the steps above, the tool will allow the upgrade to proceed. Only if you try to upgrade with an older RabbitMQ version or without enabling the required feature flags will it block you.

Complete upgrade path

If you’re going all the way from Jammy to Resolute, combine both procedures, stepping through all intermediate versions (3.10 → 3.11 → 3.12 → 3.13 → 4.0), always enabling all feature flags before moving to the next version.

Understanding feature flags

You might have noticed that we enable all feature flags after every RabbitMQ upgrade. This is a critical step and let’s understand why.

Feature flags are RabbitMQ’s mechanism for managing compatibility across different versions, especially in mixed-version clusters during upgrades. Each feature flag controls whether a specific feature or behavior is enabled across all nodes in the cluster. They are switches that need to be flipped to active new functionality or data formats.

When you upgrade RabbitMQ to a new version, the software gains knowledge of new feature flags, but they start in a disabled state. This allows the new version to remain compatible with older versions that don’t know about these flags, making rolling upgrades in clustered environments possible.

However, feature flags are designed to be enabled after upgrades, not to remain disabled indefinitely. Once you’ve completed an upgrade to a new version, enabling all feature flags signals RabbitMQ to migrate its internal data structures to the latest format. This is crucial because:

  1. Future versions require them. RabbitMQ versions eventually make older feature flags mandatory. For example, RabbitMQ 4.0 requires all feature flags from the 3.13 release to be enabled before you can upgrade it. If you skip enabling feature flags in 3.13, you won’t be able to safely upgrade to 4.0.
  2. Once enabled, they cannot be disabled. This is by design: once RabbitMQ converts its data to a new format, there’s no going back. This is why enabling feature flags is a one-way operation.
  3. They enable new behaviors, and leaving them disabled means you’re not getting the full benefit of the version you’re currently running.

It’s important to mention that rabbitmqctl enable_feature_flags (which is what we recommend running after each update) does not enable all feature flags, but only the ones that are considered stable. This provides a safe path forward, without necessarily opting into new experimental behavior.

Looking forward

These PPAs are available now and will remain available to support users throughout the lifetime of these Ubuntu releases. While we hope this situation won’t repeat with future RabbitMQ versions, this solution provides a template for handling similar challenges.

Resources

Happy upgrading, and may your queues keep flowing!

Thanks to @paelzer and @sally-makin for reviewing this post, and @alanbach for testing the upgrade process between Jammy and Noble.

3 Likes