| Project | microcluster |
|---|---|
| Status | In Review |
| Author | @masnax |
| Internal ID | LX071 |
Abstract
This adds the concept of cluster roles to a microcluster. A role can be one of cluster or non-cluster, and will determine whether a microcluster daemon will join the dqlite cluster, or simply serve the microcluster public REST API (/cluster/1.0 and /1.0).
Rationale
Both the database and API of microcluster are highly extensible, but both are necessarily required to be set up on each node of the cluster. To facilitate large-scale deployments of up to 500+ systems without the added overhead on dqlite, microcluster should be able to serve only the REST API on a node, without necessarily including it in the dqlite cluster. The dqlite cluster can continue to keep a record of these nodes for establishing and revoking trust.
The dqlite database allows for direct database actions by any node, and keeps the nodes’ internal states that are dependent on the database in-sync. In many cases database actions are not necessary on every node after it joins the cluster, as it may simply serve another service with a custom REST API on top. On these nodes, we can just serve the REST API, with the option to later promote the node to join dqlite.
Specification
Design
When a node joins an existing microcluster, it serves the REST API with the given API extensions, and joins the dqlite cluster. We can introduce the concept of role to determine whether it should be included in the dqlite cluster. This can be included in the local truststore yamls, as one of:
clusterfor a node that should join dqlite, and be considered a dqlite cluster membernon-clusterfor a node does not set up dqlite and is not considered a dqlite cluster member, but is trusted to communicate with the cluster over the REST API.
A microcluster establishes trust with cluster members by recording information about them locally in the state directory under /state/truststore/node01.yaml. We can add a role like so:
name: node01
address: 127.0.0.1:9000
role: cluster # or non-cluster
certificate: |
...
Interface
-
state.Remotes() --> state.Remotes(trust.Role):- Updates this function to take a
trust.Roleas an argument for returning a set of cluster members with the particular role oftrust.Clusterortrust.NonCluster.
- Updates this function to take a
-
microcluster.IssueToken(string) --> microcluster.IssueToken(string, bool)- Updates this function to take a boolean value for whether the node should be considered a
non-clustermember, in addition to the node name to issue the token for. When the token is used by the node to join the cluster, the cluster will detect that the token is authorized for a particular role, and set the node up accordingly.
- Updates this function to take a boolean value for whether the node should be considered a
-
client.UpgradeClusterMember(ctx, types.ClusterMemberUpgrade)- Upgrades the
non-clustermember with the given args toclustermember. Any node, whetherclusterornon-clustercan request anon-clusternode to upgrade, including itself.
- Upgrades the
For the CLI, the recommended implementation will be
# Adds a `--non-cluster` flag. This will also be reported in `microctl token list`
token=$(microctl token add node02 --non-cluster --state-dir ${node01_dir})
# The join process is unchanged, as the role information is stored in the token.
microctl init --join ${token} --state-dir ${node02_dir}
# Upgrading the node to `cluster` role will add it to the dqlite cluster and fire the appropriate hooks.
microctl cluster upgrade node02 --state-dir ${node02_dir} # or ${node01_dir}
API Upgrades
The default role will be cluster, and existing systems upgrading from prior to this change will have their non-existent roles considered as cluster, as these nodes will all have dqlite already set up. After receiving the corresponding schema update, they will update the truststore schema to include the cluster role.
When a non-cluster member that has already been initialized is restarted, this will be one of the few times it reaches out to a cluster member automatically in order to determine whether it needs to wait for all other cluster and non-cluster members to upgrade their API versions, or whether the node itself still needs upgrading.
REST API
-
PUT /cluster/1.0/cluster/{name}/upgradewill be added. This endpoint will elevate thenon-clustermember with the specified name toclusterstatus, and add it to dqlite. This can be run on any node, whetherclusterornon-clustermember, as long as it targets anon-clustermember. The request will then be forwarded to the node-to-be-upgraded, where we will run thePreUpgradehook, before forwarding the request to a randomclustermember, who will then forward the request to the dqlite leader. The leader will prepare the existing cluster for the incoming node, who will then attempt to join dqlite. At this point, allclustermembers will run theirOnUpgradedMemberhooks, and the newly upgraded node will run itsPostUpgradehook. -
GET /cluster/1.0/clusterwill have arole={roleName}query parameter indicating what role of cluster members we should fetch. The default will becluster. If the role isnon-cluster, their connection status will not be updated. -
PUT /cluster/1.0/clusterwill haverole={roleName}andupgrade={boolean}query parameters to indicate what role the registered cluster member should have, and whether it should be re-registered as aclusterrole (ifupgrade=true). -
POST /cluster/1.0/clusterwill have aupgrade={boolean}query parameter to indicate whether the node should be added or upgraded. -
POST /cluster/1.0/tokenswill now accept payloads with theRolefield set, to determine token role to generate.
non-cluster members will be trusted over the /cluster/1.0 (internal) and /1.0 (external) public APIs. If an endpoint in this set requires database access, it can choose one of two methods to handle this:
- Manually handle the endpoint, checking
state.Role()to change the behaviour based on the cluster role of the node. - Use
access.AllowClusterMembersoraccess.AllowNonClusterMembersas therest.EndpointAction.AccessHandlerto restrict access to the endpoint method to a particular role.
By default, only cluster members will be able to use DELETE /cluster/1.0/cluster/{name}. All other public endpoints will be accessible by non-cluster members. POST and PUT /cluster/1.0/cluster will be re-used for upgrade purposes by non-cluster member nodes.
Modified and new API types:
// ClusterMemberUpgrade represents information about upgrading a non-cluster member to a dqlite member.
type ClusterMemberUpgrade struct {
Name string
SchemaVersion int
InitConfig map[string]string
}
// Role is a string representing the cluster role for the token.
types.TokenRecord.Role
types.Token.Role
// NonClusterMembers is a []ClusterMemberLocal representing the list of non-cluster members that the joining node should record locally in its truststore.
types.TokenResponse.NonClusterMembers
Database
In addition to the truststore update above, this adds an internal_non_cluster_members table, similar to internal_cluster_members to keep track of API versions of non-cluster members, and facilitate joining. The internal_token_records table will have a new column for role.
ALTER TABLE internal_token_records ADD COLUMN role TEXT NOT NULL;
CREATE TABLE internal_non_cluster_members (
id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
name TEXT NOT NULL,
address TEXT NOT NULL,
certificate TEXT NOT NULL,
internal_api_extensions TEXT,
external_api_extensions TEXT,
UNIQUE(name),
UNIQUE(certificate)
);
Action Hooks
Microcluster supports various action hooks that are specified by the project using microcluster, and are run at specific times during cluster creation.
-
clustermember only hooks:PreBootstrap,PostBootstrapare only run on bootstrapping nodes which must necessarily becluster.OnHeartbeatrequires dqlite and will only run onclustermembers.OnNewMemberwill run only on pre-existingclustermembers, even whennon-clustermembers join the cluster.PostRemovewill run only on pre-existingclustermembers, even when anon-clustermember is removed from the cluster.OnUpgradedMemberis a new hook that will run only on pre-existingclustermembers when anon-clustermember is upgraded tocluster.PostUpgradeis a new hook that runs on a newly upgradecclustermember, which was previously anon-clustermember.
-
non-clustermember only hooks:PreUpgradeis a new hook that runs only on anon-clustermember before it is upgraded toclustermember.
-
role-agnostic hooks:
OnStartruns on a node when the daemon starts, regardless of cluster role.PreRemoveruns a node before removing it from the cluster, regardless of its cluster role.PreJoinruns on a node before it joins the cluster, regardless of its cluster role.PostJoinruns on a node after it first joins the cluster, regardless of its cluster role.