| Project | MicroCloud |
|---|---|
| Status | Pending Review |
| Author(s) | @jpelizaeus |
| Approver(s) | @tomp @maria-seralessandri |
| Release | 1.x / 2.x |
| Internal ID | LX073 |
Abstract
Allow all members of a MicroCloud to explicitly establish trust so they can securely join the cluster. Grant both the joining member and the cluster the possibility to verify its peer
and do not transfer critical information like API secrets across the network.
Rationale
In its current version MicroCloud offers a convenient approach to bootstrap clusters built using LXD, MicroCeph and MicroOVN. By controlling the entire process, MicroCloud can generate join tokens for each of the services and distribute them across the cluster members. This allows forming additional clusters for each of the services without additional manual intervention.
As MicroCloud itself offers many additional settings to configure its behavior, additional support for a preseed configuration file has been added. This allows and administrator to skip the setup dialog and apply the configuration for an entire MicroCloud in one step.
Having this one step configuration mechanism requires MicroCloud to make decisions on the administrators behalf. One of them is to accept additional cluster members that have been selected or configured (using preseed) by the administrator. As there is no additional step involved to ensure the integrity of either one of the joining peers, this might lead to security risks as currently the network is considered to be a trusted party.
Specification
This specification supersedes the already existing cluster join mechanism with proactive tasks that have to be executed on each cluster member individually to ensure integrity before starting the join procedure.
Existing mechanism
Existing mechanism
In the latest release of MicroCloud the cluster join mechanism is largely dependent on mDNS in order to discover its peers, share relevant connection details and to bootstrap the final cluster. Therefore on each node of the cluster a microcloudd daemon is running that both broadcasts its own set of details onto the network but also receives details sent by others in the same network.
This ultimately allows to construct a picture of the available resources so that MicroCloud can offer the administrator a straightforward question and answer input dialog:
The details broadcasted on the network by each of the daemons consist of the following:
- The current version of the mDNS broadcast/lookup format (currently
1.0) - The hostname of the node
- The address of the nodeâs MicroCloud API endpoint
- The nodes network interface over which the broadcast was sent
- A list of services (MicroCloud, LXD, MicroCeph and MicroOVN) present on this node
- An authentication secret that can be used to access this nodes API endpoint using the
X-MicroCloud-Authheader
Using those information the initial microcloudd requests further details on network interfaces and storage disks from the local and peer LXD servers for later selection by the administrator.
Discovery
At the beginning MicroCloud requires three independent microcloudd daemons to be running on a shared network. As a node running microcloudd could potentially have more than a single network interface, each microcloudd broadcasts its details on each of the network interfaces which are available on the underlying node that have an IP address configured.
An administrator will then pick any of the microcloudd daemons to be the initial one that will be used to bootstrap the MircoCloud. Itâs the responsibility of this microcloudd to listen only to broadcasts on the network(s) selected by the administrator:
Select an address for MicroCloud's internal traffic:
Space to select; enter to confirm; type to filter results.
Up/down to move; right to select all; left to select none.
+----------------------------------------+--------+
| ADDRESS | IFACE |
+----------------------------------------+--------+
> [x] | 10.237.170.93 | enp5s0 |
[ ] | fd42:e287:8e5c:b221:216:3eff:fe6f:45cb | enp5s0 |
+----------------------------------------+--------+
...
Limit search for other MicroCloud servers to 10.237.170.93/24? (yes/no) [default=yes]:
After selecting the network(s) on which MicroCloud should discover any potential peers, MicroCloud prompts the user with a list of peers that have been discovered on the respective network interface:
Scanning for eligible servers ...
Space to select; enter to confirm; type to filter results.
Up/down to move; right to select all; left to select none.
+------+--------+----------------+
| NAME | IFACE | ADDR |
+------+--------+----------------+
> [x] | m3 | enp5s0 | 10.237.170.140 |
[x] | m2 | enp5s0 | 10.237.170.61 |
+------+--------+----------------+
After this the administrator is guided through multiple dialogs allowing further configuration of the final MicroCloud in regards to storage and networking.
As a last step MicroCloud instructs its peers to form a cluster for each of the services (MicroCloud, LXD, MicroCeph and MicroOVN). In order to allow access from the initial MicroCloud node to every other node in the cluster, each node has broadcasted an API secret that is now used to invoke RPC requests on the clusterâs nodes by setting the X-MicroCloud-Auth header.
As part of those requests MicroCloud initiates the cluster forming process for the various services.
Cluster forming
MicroCloud is using MicroCluster under the hood to form a cluster of nodes. The mechanism on the MicroCluster side currently relies on the fact that the join token is considered to be a secret. In addition a node joining using this secret is automatically trusted to be the one for which the secret has been created.
The joining node however is checking if the fingerprint of the clusters certificate (embedded into the token) is matching the one returned by the cluster API when issuing the join request:
The response of the join request contains the clusterâs certificate and key and a list of certificates from all the nodes who have already joined the cluster. This list is used to extend the truststore of the newly joined node.
The nodes own certificate is added automatically to the truststore.
After adding the other peerâs certificates to the nodes truststore, it will start its API and join the already existing dqlite cluster
using mutual TLS with the certificates that have been obtained during the join process.
Updated mechanism
As the current mechanism fully trusts the local LAN, every microcloudd broadcasts an authentication secret and trusts the broadcasts received from other peers in the network.
This can lead to man in the middle (MITM) attacks as the broadcasts itself arenât protected and could potentially be read and modified by somebody sitting in the same network.
By removing the authentication secret from the broadcast message, the initial microcloudd cannot anymore talk to its peers as there isnât a trust relationship anymore. This breaks the disk and network discovery as well as the final cluster forming as they are currently making use of this secret.
Instead this communication could already make use of the mutual TLS that currently gets established during the final cluster forming. By moving the exchange of trust right after the discovery of peers and extending it with a proactive human verification option, it can be ensured
that the nodes in the cluster are the ones they pretend to be.
Instead of forming the MicroCloudâs MicroCluster at the end to establish the base for mTLS, a new temporary trust store is build up during the cluster forming. This store can be used for any follow up tasks to discover the required information and to finally form the clusters for MicroCloud, LXD, MicroCeph and MicroOVN.
To allow exchanging the public keys for the temporary trust store, the already existing flow of communication (discovery) is extended with a HMAC to prevent broadcasting a secret across the network but still allow the sender and receiver to validate and trust the received payloads. This requires having a shared secret on both ends that never gets transmitted across the network.
Discovery (KDF and HMAC)
The discovery is relying on HMAC to sign the messages exchanged between the initial microcloudd and potential peers so that each side can verify any received contents. For added security a key derivation function (KDF) is used together with a salt that allows having a stronger secret when computing the messageâs HMAC.
The overall idea is to allow establishing a verified mTLS connection as soon as possible so that both ends can talk via an encrypted channel to exchange further information. Therefore the discovery ensures that both sides exchange their own public key rather quick so that if a new HTTPS connection gets opened up from one end to the other, we can rely on TLS to perform a proper exchange and to setup a secure session.
Even before the peers can be verified using mTLS, both the joiner and initiator open up TLS connections to the other end (if allowed by the underlying protocol) so that the flow of communication is encrypted.
In case of preseed the steps that require human interaction are skipped and only the HMAC comparison is performed on both ends. Additionally there is no random password generated on the initial microcloudd. Instead the password has to be generated by the administrator and injected accordingly when running microcloud preseed.
The initial public key exchange is what is depicted in the next six steps which are also shown in the graphic above.
Startup
An administrator starts the cluster forming process by reading a randomly generated password displayed by the initial microcloudd and setting it on any of the potential joiners. The password itself is a concatenation of strings which have been selected randomly from a given word list (e.g. EFF wordlist for random passphrases). There are various approaches for the word lists. One of them might be picking a list that only contains words which have a unique three-character prefix (see example list) so that an administrator only has to type in the first three characters of each word and the remainder of the characters can be guessed using auto completion.
The length of the password is based on the number of words selected from the words list. It takes around n^k/2 guesses to crack the password where n is the length of the overall word list and k the number of words chosen from the list. Picking between 4-6 words from a list with a length of around 5000 should be sufficient. The password is displayed on the initial microcloudd and has to be typed in on any other microcloudd that should join the cluster.
Using this password the joining microcloudd can derive a key using a random salt with an appropriate length. We have chosen argon2, but other KDFs (HKDF, scrypt) might work too:
// Nonce S, which is a salt for password hashing applications.
// May have any length from 8 to 2^(32)-1 bytes.
// 16 bytes is recommended for password hashing.
// Salt must be unique for each password.
// See https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-argon2-03#section-3.1
salt := make([]byte, 16)
rand.Read(salt)
// The draft RFC recommends[2] time=1, and memory=64*1024 is a sensible number.
// If using that amount of memory (64 MB) is not possible in some contexts
// then the time parameter can be increased to compensate.
// The number of threads can be adjusted to the numbers of available CPUs.
// See https://pkg.go.dev/golang.org/x/crypto/argon2
//
// func IDKey(password, salt []byte, time, memory uint32, threads uint8, keyLen uint32) []byte
key := argon2.IDKey([]byte("some words from the password list"), salt, 1, 64*1024, 4, 32)
As each joining microcloudd will pick another random salt, the initial microcloudd can derive the key only after receiving the intent from the joiner which also includes the salt. This is covered in the next section.
For the purpose of human validation, the respective local microcloudd will also print itâs fingerprint on startup either when running microcloud init or microcloud join for visual comparison on the other end.
Discover joiner
As both ends need to be aware of each other, the joining microcloudd has to send its intent to join an existing MicroCloud cluster. This intent (ServerInfoSigned in the next section) is sent to the initiating side and the body contains at least the following information:
- The local public key
- The version of MicroCloud
- The name of the node
- The local address of the API
- The random salt
- The HMAC
The value of the HMAC field is created by taking the contents of ServerInfo (see the section below) and creating the MAC by using the key which was computed by the KDF.
Authenticate joiner
After receiving the intent to join from any potential joiner, the initial microcloudd first has to validate the contents of the received payload. This is required to filter out joiners that donât have the same version of MicroCloud as well as the ones that have sent a payload with an HMAC that cannot be reproduced on the receiving side. Those have to be marked with extra care as this could be the result of a MITM attack.
Using the salt and the random password that has been generated by the initial microcloudd during the startup, the exact same key can now be derived using the same KDF as on the joining side. Now the HMAC over ServerInfo can be computed using the key and compared to the one that got sent over the wire as part of ServerInfoSigned. If the HMACs match, the administrator has the possibility to approve the join request from the joiner. Joiners with invalid versions and non matching HMACâs are âgreyed outâ and cannot be selected. A reason for this is provided in the Note column:
Scanning for eligible servers ...
Space to select; enter to confirm; type to filter results.
Up/down to move; right to select all; left to select none.
+------+--------+----------------+-------------+--------------+
| NAME | IFACE | ADDR | Fingerprint | Note |
+------+--------+----------------+-------------+--------------+
> [x] | m3 | enp5s0 | 10.237.170.140 | aabbccddeef | |
| m2 | enp5s0 | 10.237.170.61 | ff001122334 | Invalid HMAC |
+------+--------+----------------+-------------+--------------+
After the joiner is accepted, its public key gets added to the local microclouddâs temporary trust store (bound to its address) which allows for certificate validation of new mTLS connections during the remainder of the cluster forming. Now when opening up a new mTLS connection from the initial microcloudd to the joiner, the certificate provided from the other end has to match the one which is tracked in the local temporary trust store.
Authenticate cluster
The last step of the discovery allows the joiner to also verify that it is joining the right cluster. As the initial microcloudd already knows the address of the joiner (from the received payload), a new HTTPS request is made to the API of the joiner. As the received payload from before, the requestâs body contains:
- The local public key
- The version of MicroCloud
- The name of the node
The HMAC of the request body is sent alongside the Authentication header.
After receiving the request, the joiner now computes the HMAC itself using the key from before and the contents of the requestâs body. If the HMAC doesnât match, this might be an indication for a MITM attack. As the protocol doesnât foresee multiple clusters contacting the same joiner, such a mismatch is ignored but an appropriate warning message is logged to the daemonâs log of the joiner. Also if the version doesnât match, the request should be ignored too. The initial microcloudd should have never contacted the joiner in the first place if the versions donât match.
If both the version and HMAC matches, the administrator is asked to approve the request from the cluster:
Scanning for response ...
Would you like to join m1 (fingerprint): (yes/no) [default=yes]:
After the cluster is accepted, its public key gets added to the local microclouddâs temporary trust store (bound to its address) which allows for certificate validation of new mTLS connections during the remainder of the cluster forming. Now if the joiner receives a new mTLS connection from the initial microcloudd, it can verify the provided public key based on the entry in its local trust store.
The response of the HTTPS request indicates a successful pairing and marks the end of the discovery/authentication protocol. This also marks the end of the session and discards the random password on each end.
Cluster forming
The initial microcloudd can now use mTLS to retrieve further information from the joiner and both ends can validate the other side based on their temporary trust store entries. Furthermore join tokens are created on the initial microcloudd for each of the services (LXD, MicroCeph and MicroOVN). Those tokens are now sent through the encrypted and trusted mTLS channel in order to form each of the services MicroClusterâs.
Cleanup
During the cleanup stage both ends discard their temporary trust store as the serviceâs MicroClusters are formed and the trust is established in each MicroClusterâs own truststore.
Daemon and API changes
MicroCloud
A new API extension is added that indicates the change in how MicroCloud performs the discovery/authentication.
Joiner request
The request payload is extended with the following information. Check the previous sections on some more explanations:
type ServerInfoSigned struct {
// MAC generated from HMAC(key, ServerInfo)
HMAC string
// The existing ServerInfo struct
ServerInfo
}
type ServerInfo struct {
// The current version of the payload format
Version string
// The hostname of the node
Name string
// The address of the node's MicroCloud API endpoint
Address string
// The interface used on the sending side
Interface string
// A list of services (e.g. LXD, MicroCeph, MicroOVN) present on this node
Services []types.ServiceType
// The node's public certificate for mTLS
Certificate string
// The random salt
Salt string
}
Temporary trust store
MicroCloud will maintain a temporary trust store on both ends that gets filled up with the public key of the respective peer. This temporary trust store has to be made available to MicroCluster so that the custom API endpoints of MicroCloud can use this temporary store instead.
There is an open proposal in the MicroCluster repo (#120) which makes the authentication handler public so that an importer of MicroCluster (like MicroCloud) can inject itâs own trust store information for every custom API endpoint. A more detailed description on the specifics can be found here.
In any case the X-MicroCloud-Auth header is being removed as there isnât anymore a secret being broadcasted to the local network.
Requests to any peers of the MicroCloud have to be made using a mTLS connection which can be trusted on both ends using the temporary trust store.
This is only possible if both ends have successfully finished the discovery/authentication protocol.
Joining existing services
As part of #259 MicroCloud grows support to reuse existing MicroCeph and MicroOVN clusters. The process behind relies on one of the microcloudd within the existing cluster being able to create a join token on the peer that allows joining into the already existing remote cluster(s).
This concept wouldnât be blocked as microcloudd would continue to use the same paths of communication to reuse the existing clusters.
MicroCluster
To have as little impact as possible on other active importers of MicroCluster (e.g. MicroCeph, MicroOVN), the modifications for the temporary trust store wonât affect any of the existing setups. Itâs the choice of the importer to make use of this added functionality using the temporary trust store.
CLI changes
MicroCloud
Join command
A new command microcloud join gets added to allow a peer joining into MicroCloud.
This command is also the starting point after which the joinerâs microcloudd can send its cluster forming intent to the initiating side.
The command prompts the administrator to enter the random password that got displayed by the initial microcloudd.
The command blocks until the request got approved on both sides.
Preseed command
A new command microcloud preseed gets added to allow for unattended deployments.
All participants of the deployment load the password from the preseed file that gets passed via stdin.
Session timeout
In addition a new --session-timeout flag is added to both the init and join subcommands. It allows exiting the session at time x so that both ends discard their temporary trust stores and forget about the random password. Afterwards a new discovery/authentication session has to be started by running the init and join subcommands again on both ends.
The default session timeout value is set to ten minutes.
UX
To approve the requests on both ends, the dialogs displaying the information have to be extended to allow âgreying outâ invalid requests as well as setting a notification in case a peer cannot be selected due to version or integrity mismatches.
Database changes
No database changes expected.
Packaging changes
No packaging changes expected.