How to upgrade Anbox Cloud

Note:

If you’re interested in getting notified for the latest Anbox Cloud releases, make sure you subscribe to notifications on the announcements category on the Anbox Cloud discourse.

Anbox Cloud allows upgrades from older versions to newer version. This describes the steps necessary to perform the upgrade.

The upgrade instructions detail the revisions each charm needs to be upgraded to, to bring it to the latest version. Next to the upgrade of the charms any used images or addons need to be updated as well.

Before you begin

As with all upgrades, there is a possibility that there may be unforeseen difficulties. It is highly recommended that you make a backup of any important data, including any running workloads.

You should also make sure that:

  • Your deployment is running normally.
  • Your Juju client and controller/models are running the latest version.
  • You have read the release notes for the version you are upgrading to, which will alert you to any important changes to the operation of your cluster.
Note:

The following assume you’re using Juju >= 3.1. If you’re using Juju 2.9, you have to map the following commands:

Juju 3.x Juju 2.9
juju refresh juju upgrade-charm
juju exec juju run

Upgrade OS

Before you run the upgrade of the charms, you should make sure all packages on the machines that are part of the deployment are up-to-date. To do so, run the following commands on each machine:

sudo apt update
sudo apt upgrade

You can either run the package update manually or use the Juju command to run it for all machines.

juju exec --all -- /bin/sh -c 'sudo apt update && sudo apt upgrade -y'

If the LXD charm is deployed on a machine that has an NVIDIA GPU installed, running the above command for the machine may upgrade the NVIDIA drivers, which accidentally suspends running containers with GPU support. Starting with the 1.17.1 release, the NVIDIA drivers are held from being upgraded until you upgrade the LXD charm using the Juju command. To check if the currently installed NVIDIA drivers are held from being upgraded:

apt-mark showhold "libnvidia-.*-$(nvidia-smi --query-gpu=driver_version --format=csv,noheader | cut -d'.' -f1)"

If they are not, run the following command to do so:

sudo apt-mark hold 'linux-modules-nvidia-*' 'nvidia-*' 'libnvidia-*'

Check Juju version

Before you upgrade, check the required Juju version.

If your deployment uses an earlier Juju version, you must upgrade your controller and all models first. See the Juju documentation for instructions on how to upgrade the Juju controller and all models to a newer Juju version.

Upgrade all charms

The deployed Juju charms need to be upgraded next.

Note:
  • You can find a list of all charm, snap and Debian package versions for each Anbox Cloud release in the component versions overview. This also includes the charm and bundle revisions and channels for each release.

  • Starting with the 1.14 release, all charms come from Charmhub and use the concept of channels to track particular versions. The instructions below address how to upgrade from a 1.13.x release, where charms were still from the old Juju charm store. The --channel=1.17/stable argument instructs Juju to switch to the Charmhub version of the charm and track the right channel.

  • With the 1.14 release, the name of the lxd charm changed to ams-lxd. If you run a deployment older than 1.14 and want to upgrade, add --switch=ams-lxd to the upgrade command to make Juju switch to the new charm. The charm itself remains identical with the same functionality and features.

  • Starting with the 1.15 release, AMS enforces TLS 1.3 on its HTTPS endpoint. Images older than 1.15.0 will fail to reach AMS in this case. To still allow older images to work with AMS, you can temporarily enable TLS 1.2 support again in AMS by setting the force_tls12 configuration option of the AMS charm.

  • If you want to deploy a particular revision of a charm, you can do so by adding --revision=<rev> to the juju upgrade-charm command.

For any of the charm upgrades, you can watch the upgrade status by running:

`juju status`

Continue with the next step only when the current step has completed successfully and all units in the output are marked as active.

Note:

If you don’t run Anbox Cloud in a high availability configuration, upgrading the charms will cause a short down time of individual service components during the process.

Upgrade infrastructure components

As a first step, we will update all infrastructure components. This includes deployed internal certificate authorities and etcd.

First we update easyrsa:

```
juju refresh internal-ca --revision=26
juju refresh etcd-ca --revision=26

Upgrade application registry

The Anbox Application Registry (AAR) can be updated independently of the other services. The upgrade process will cause a short down time of the service providing the registry API but connected AMS instances will retry connecting with it automatically.

To upgrade the registry, run

juju refresh --channel=1.17/stable aar

Upgrade control plane

If you have the streaming stack deployed, you need to update the charms responsible for the control plane next. If you do not use the streaming stack, you can skip this step.

Note:

If you don’t run any of the services in a high availability configuration, upgrading the charms will cause a short down time of the service.

To upgrade all charms, run the following commands:

juju refresh --channel=1.17/stable anbox-cloud-dashboard
juju refresh --channel=1.17/stable anbox-stream-gateway
juju refresh --channel=1.17/stable anbox-stream-agent
juju refresh --channel=1.17/stable coturn
juju refresh nats

Upgrade AMS

The AMS service needs to be updated independently of the other service components to ensure minimal down time. The charm can be upgraded by running the following command.

juju refresh --channel=1.17/stable ams

Upgrade LXD

As the last step, you have to upgrade the LXD cluster. Upgrading LXD will not restart running containers but it’s recommended to take a backup before continuing.

As the first step, you need to upgrade the AMS node controller by running:

juju refresh --channel=1.17/stable ams-node-controller

Once the upgrade is completed, you can continue upgrading LXD:

juju refresh --channel=1.17/stable lxd

In some cases, specifically when you maintain bigger LXD clusters or want to keep a specific set of LXD nodes active until users have dropped, it makes sense to run the upgrade process manually on a per node basis. To enable this, you can set the following configuration option for the LXD charm before running the refresh command above:

juju config lxd enable_manual_upgrade=true

This will allow you to run the actual upgrade process for each deployed LXD instance separately. After the charm has been refreshed, the upgrade for a single LXD deployment unit can be triggered by running:

juju run --wait=30m lxd/0 upgrade

Once the upgrade has completed, the unit will be marked as active.

For major and minor version upgrades, an update of the LXD charm may upgrade kernel modules or GPU drivers. This requires stopping any running containers before applying the upgrade and performing a reboot of the machine once the upgrade completed.

In case a reboot of the machine is required, a status message will be shown. When the machine has been rebooted, the status message can be cleared by running:

juju run --wait=1m lxd/0 clear-notification

If the LXD charm is deployed on a machine with an NVIDIA GPU installed, by default, the NVIDIA drivers are held from being upgraded in case of downtime for all running containers due to either a manual upgrade or an unattended-upgrade. The downside to this is that the machine may miss security updates for the NVIDIA drivers. To manually upgrade the NVIDIA drivers, you need to:

  1. Stop all running containers.

  2. Unhold the NVIDIA drivers to allow an upgrade:

     sudo apt-mark unhold 'linux-modules-nvidia-*' 'nvidia-*' 'libnvidia-*'
    
  3. Update OS:

     sudo apt update && sudo apt upgrade -y
    
  4. Hold the NVIDIA drivers from being updated again:

     sudo apt-mark hold 'linux-modules-nvidia-*' 'nvidia-*' 'libnvidia-*'
    

A Juju action that simplifies the manual upgrade for NVIDIA drivers will be added in a future release.

Upgrade Debian packages

Some parts of Anbox Cloud are distributed as Debian packages coming from the Anbox Cloud Archive. In order to apply all pending upgrades, run the following commands on your machines:

sudo apt update
sudo apt upgrade

or apply the updates via Landscape if available.

Upgrade LXD images

LXD images are automatically being fetched by AMS from the image server once they are published.

Existing applications will be automatically updated by AMS as soon as the new image is uploaded. Watch out for new versions being added for any of the existing applications based on the new image version.

You can check for the status of an existing application by running

amc application show <application id or name>