Key rotation in the Ceph charms

Recently, the Ceph charms have gained the capability to rotate the keys that an entity uses for authorization within the cluster. This is important from a security standpoint, as it allows operators to quickly replace the keys of any compromised entity.

This capability is presented as an action for the ceph-mon charm, aptly named rotate-key. This action takes a single parameter, called entity, that specifies which entities will be affected. Currently, we support the following entities:

  • mgr: Rotates the key for the local manager. In a Ceph cluster, each monitor is paired with a manager. Note that at any point in time, only a single manager is active in the cluster, so key rotation for managers should be done on every ceph-mon unit, unless the operator knows for certain where the active manager currently resides.

  • client.rgw.<RGW-ID>: Rotates the key for a specific Rados gateway service daemon.

  • mds.<MDS-ID>: Rotates the key for a Ceph metadata server. This is of particular use for ceph-fs charm units.

  • osd.<OSD-ID>: Rotates the key for a specific Ceph OSD.

  • osd: Rotates the key for all Ceph OSDs in the cluster.

Note that in every case but the manager one, this action involves coordination between the ceph-mon unit on which the action is executed, and the units on which the affected units reside. This means that even after the action finishes, operators should look at the other units to see if the run was successful. This action also involves restarting systemd services, so if it’s run on many units (as is the case when specifying osd for the entity), this can take a considerable time.

As an example, let’s consider a deployment in which we have a Ceph model:

Model  Controller     Cloud/Region     Version  SLA          Timestamp
ceph   my-controller  maas/default  2.8.1    unsupported  19:34:16Z

App          Version  Status  Scale  Charm     Store       Rev  OS      Notes
ceph-fs      17.2.6   active      3  ceph-fs   jujucharms   24  ubuntu  
ceph-mon     17.2.6   active      3  ceph-mon  jujucharms   49  ubuntu  
ceph-osd     17.2.6   active      3  ceph-osd  jujucharms  304  ubuntu  

Unit            Workload  Agent  Machine  Public address  Ports  Message
ceph-fs/0       active    idle   0/lxd/0             Unit is ready
ceph-fs/1       active    idle   1/lxd/0             Unit is ready
ceph-fs/2*      active    idle   2/lxd/0             Unit is ready
ceph-mon/0      active    idle   0/lxd/1             Unit is ready and clustered
ceph-mon/1      active    idle   1/lxd/1             Unit is ready and clustered
ceph-mon/2*     active    idle   2/lxd/1             Unit is ready and clustered
ceph-osd/0      active    idle   0             Unit is ready (2 OSD)
ceph-osd/1*     active    idle   1             Unit is ready (2 OSD)
ceph-osd/2      active    idle   2             Unit is ready (2 OSD)

If we wanted to rotate the keys for one of the ceph-fs charms, we need to get the actual entity name. For ceph-fs, this is composed of the prefix mds. and the hostname of the unit. So if we wanted to rotate the key for the unit ceph-fs/0, we could do:

juju ssh ceph-fs/0 -- hostname

And then proceed to rotate it via:

juju run-action ceph-mon/0 rotate-key entity=mds.$HOSTNAME

The unit on which we call the key rotation can be any of the ceph-mon ones.

The process is similar for the ceph-radosgw charms, with the exception that the prefix is client.rgw..

In general, the list of entities for which the key can be rotated is found by calling:

juju run ceph-mon/0 list-entities.