The HPC team is developing a new lustre-server charm to add the popular Lustre parallel filesystem to our set of core Charmed HPC projects. Lustre is the de facto standard for HPC systems so this charm interest anyone looking to add a large-scale, performant filesystem to their Juju deployments.
Lustre is the most common filesystem in the TOP500 list of the world’s most powerful supercomputers. It’s a scalable, parallel filesystem built for fast I/O from a large number of clients. It’s also complex to stand up: a deployment combines Management Servers (MGS) and Metadata Servers (MDS) with multiple Object Storage Servers (OSS) exposing the actual file data, all wired together over Lustre’s LNet network layer.
Lustre data is held on “targets” (typically RAID disk arrays) corresponding to each of the server types: Management Targets (MGT), Metadata Targets (MDT), Object Storage Targets (OST). Each target is formatted with a backing filesystem. ZFS is the modern option and Ubuntu’s first-class ZFS support gives it a real advantage here.
Source: https://wiki.lustre.org/Introduction_to_Lustre#Lustre_Architecture
The complexity of a Lustre setup makes it an ideal candidate for a new charm. The lustre-server charm wraps all these components in a single operator and exposes management through standard Juju commands, so no manual `zpool create`, `mkfs.lustre`, or `mount -t lustre` plumbing.
Charm approach
A single lustre-server unit can run any of the Lustre MGS, MDS, OSS server roles, an approach inspired by the microceph charm (a large thank you to the Ceph team for their invaluable input).
At present, lustre-server roles are fixed at deployment time:
- The initial application leader becomes a combined MGS+MDS. It formats the MGT and MDT on a ZFS-backed pool and publishes its LNet Network Identifiers (NIDs) to the peer relation so other units can register against it.
- Every non-leader unit becomes an OSS. Each OSS formats an OST on its own ZFS-backed pool and joins the filesystem by pointing at the leader’s MGS NIDs, retrieved from peer relation data.
One requirement is that the MGS must exist before any OSS can join the Lustre cluster. To ensure this, the charm uses a peer relation to coordinate start-up:
- The initial leader MGS+MDS writes its NIDs and unit name into the peer application databag once it has completed deployment.
- Non-leaders wait. When they receive a peer relation change event from the MGS+MDS writing into the application databag, they retrieve the published MGS NIDs and configure an OSS against them.
Lustre’s LNet network layer is automatically configured by the charm by default. The charm selects the Ethernet device associated with the default route and any InfiniBand RDMA devices detected on the node. For finer control, an “lnet-networks” configuration option is provided to allow users to override the automatic configuration with their own list of network devices.
Storage is provisioned as ZFS-backed pools and datasets: the MGT/MDT pool is built from mirror vdevs, while each OST pool uses RAIDZ2. Targets are formatted with calls to “mkfs.lustre --backfstype=zfs” made by the charm.
Integrating using filesystem-client
To mount Lustre on a client (for example, your Charmed HPC cluster’s Slurm compute nodes or login nodes), the filesystem-client subordinate charm is used. This is a charm we worked on in previous cycles to provide CephFS and NFS access to Charmed HPC clusters. We’ve now extended it to support Lustre as well, so a single, consistent “juju integrate” command is all that’s needed to mount a shared filesystem of any type.
This is particularly useful as a typical Charmed HPC deployment runs multiple filesystems side by side: Lustre for the active, short-lived scratch data driving HPC workloads, and a more general filesystem such as CephFS (deployed via the existing charmed Ceph solutions) for longer-term storage.
Demo
A video demo showing the lustre-server charm in action is available in our announcement LinkedIn post: https://www.linkedin.com/feed/update/urn:li:activity:7498029484554571776
For those without access to LinkedIn: https://drive.google.com/file/d/1NorAY6k_4aEzSrxy1kS9fdGmLRyImTNZ/view
The demo walks through adding a Lustre filesystem to a minimal Charmed Slurm deployment: a Slurm controller, a login node, and a single compute node, all backed by LXD virtual machines.
What’s next?
The charm is published at https://charmhub.io/lustre-server. To try the latest snapshot, deploy with “juju deploy lustre-server --channel edge” to your preferred cloud.
Block-device detection is on the roadmap before a general release. The charm currently backs the filesystem with image files and we will be replacing this with provisioning full disks.
Our long-term goal is to provide a Lustre solution with integrated tuning such as finding the best ratio of MDSes to OSSes, as well as OSTs to OSSes, given customer hardware and workload I/O profiles.
If you’d like to help, the charm lives at GitHub - canonical/filesystem-charms: Juju charms for automatically providing, requesting and mounting shared filesystems. · GitHub and issues are tracked there.
Come find us on the Ubuntu HPC Matrix channel or the weekly Ubuntu HPC community call: Wednesdays 16:30 UTC.