| Project | LXD |
| Status | Drafting |
| Author(s) | @gabrielmougard |
| Approver(s) | |
| Release | |
| Internal ID | |
Abstract:
This document outlines an approach to enable the development of self-sustaining software applications capable of adapting to dynamic operational conditions through the utilization of a Deployments API.
The crux of this proposal revolves around creating a specialized, bidirectional Deployments API, serving as a universal conduit for seamless interaction between platforms, such as LXD, and application workloads, like Microk8s.
This Deployments API is pivotal, concentrating on providing a standardized, platform-agnostic interaction mechanism that enables automatic scaling, instigated either by the platform or the application workloads.
The proposed Deployments API, with its dual components—Platform API (LXD in our case) and Governor facing API—enables customized control over instance allocation, ensuring optimized resource distribution and responsiveness to scaling requests and offers. Let us define a few important terms for a better understanding:
-
A Governor is a component that has special access to the platform’s deployments API, allowing it to control the quantity of instances in a deployment. It is responsible for managing the scaling of instances, responding to requests and demands from the platform for scaling, and ensuring optimal allocation of resources. In short, it helps in maintaining and adjusting the deployment size as per the needs, acting as a mediator between the application workloads and the platform they run on.
-
In the context of a deployment, managed by a Governor, let’s also define the Deployment Shape (or simply, Shape). A shape is a predefined template within a deployment on a platform. It defines the configuration for instances that are spawned within the deployment. Each Shape specifies the setup for the instances, tailoring them to fit specific needs within the application workload. A deployment can contain multiple shapes, allowing for a variety of instance configurations within the same deployment to meet different requirements of the application services. In essence, a Shape outlines how instances in a deployment should be configured and managed.
In this document, we’ll only focus on the Platform specific management API and the Governor facing API.
Rationale:
Modern software frameworks demands solutions that are adaptable and able to meet varied and dynamic needs. As an example, an e-commerce platform will require:
- Web Servers: To host the website, handle user requests, and serve content.
- Database Servers: To store and manage product information, user data, and transaction records.
- Caching Servers: To cache frequent queries and improve data retrieval performance.
- Background Workers: To process tasks like order processing, email notifications, and other asynchronous jobs.
- Search Servers: To provide fast and accurate search results to the users.
- Load Balancers: To distribute incoming network traffic across multiple servers to ensure no single server is overwhelmed with too much traffic.
- Security Services: To manage authentication, authorization, and protect against malicious attacks.
Each of these components could have distinct scaling needs, availability requirements, resource consumptions, and operating environments, making the overall workload multifaceted.
The Deployments API aims at managing such a scenario. We’d initially grant a cloud substrate (or platform) like LXD, the autonomy and adaptability to react to these environmental variables and demands.
Then, the Deployments API will be an input for governing softwares, to maximize the resource allocation efficiency, foster scalability, and facilitate the flawless integration of varied applications within the dynamic ecosystems of software deployment.
Specification
We can describe the deployment workflow as following:
- An admin would typically create the boundaries of a deployment (the deployment (let’s say it is called
dep1for our example) itself with its shapes, each having different scaling needs and instance templates) - An admin would then create a key / certificate pair that would be added to LXD’s config trust store (
lxc config trust add <my-cert.crt> --type=deployments), get the fingerprint of the newly inserted certificate and create the deployment key using this fingerprint (lxc deployment key create dep1 dep1-key1 <fingerprint> --role rw) - Then, a governor program living outisde of LXD can authenticate using TLS with the generated key pair and LXD will then detect its role on the deployment.
- A governor living inside a LXD instance communicates through the devlxd vsock with a JWT signature (using the same key pair for the signature) to authenticate itself for the asked deployment.
- A webhook endpoint (if specified when creating the deployment) could be used in option in order for the governor to listen for platform specific metrics. The goal is to enable the governor to scale based on resource usage (opportunistic scaling) in order to optimize the system utilization.
Database design
First, we’ll need to introduce new SQL schemas to store the deployment related records, here is a proposal:
-
The
deploymentstable holds the Deployment records. Each deployment has 0 or more config key/value pairs (stored in thedeployments_configtable), 0 or more access tokens (stored in thedeployments_keystable) and 0 or more deployment shapes (stored in thedeployments_shapestable)- An access token can be useful to restrict the use of a particular deployment through authz and RBAC rules for examples.
-
A Deployment Shape record hold the scale information (i.e, the scale boundaries. The current scale is simply derived from the number of associated records in
deployment_shape_instances) and an instance template of the to-be-spun instances which will be part of each deployment shape (this relation is stored in thedeployments_shapes_instancestable). Thisinstance_templatecould be a copy of an existing instance/snapshot configuration but it could also be built using an instance profile. Each shape record also has 0 or more config key/value pairs (stored in thedeployments_shapes_configtable). -
When an instance is created / deleted as part of a deployment shape, the associated LXD controller will ensure that this operation is allowed (i.e, the number of instances after this operation fits within the scale boundaries) before effectively creating / deleting the instance and affecting its database record in
deployments_shapes_instances. -
Some edge cases to take into consideration:
- An instance that is already part of a deployment shape can also be deleted through the normal
DELETE 1.0/instances/{instanceName}endpoint, but the aforementioned scale check will be enforced as well. In case of a scale error, the user must adapt the scale parameter of the shape (decreasing thescaling_minimumvalue of the shape) containing the instance before deleting it. - We might want to introduce a
forcequery API parameter (with its CLI equivalent) to forcefully delete a deployment shape : it will then delete all the underlying instances before deleting their parent shape. The same could apply for a deployment : we’ll then need to delete all the instances of all the shapes contained in a deployment before removing the deployment database record (the cascade delete relation will then automatically remove the associated records)
- An instance that is already part of a deployment shape can also be deleted through the normal
Here are some sequence diagrams of common scenarios that illustrate how the management and the governor facing API will interact with LXD, its database and a to-be-defined Governor application:
[Management] Creating a Deployment
[Management] Creating a Deployment Shape
[Governor] Reconciling the Governor with a desired Deployment
[Management] Creating a Deployment Key to manage access
For the governor facing API, it is important to stress that all the calls to the platform must be authenticated using the deployment key as a governor can be outside the actual lxd cluster/server. If no deployment key exists for a deployment, then a governor won’t be able to communicate with it.
API Struct changes
Deployment related
// DeploymentPost represents the fields required to rename a LXD deployment
type DeploymentPost struct {
// The name for the deployment
// Example: myapp
Name string `json:"name" yaml:"name"`
}
// DeploymentPut represents the modifiable fields of a LXD deployment
type DeploymentPut struct {
// Description of the deployment
// Example: My new app and its required services
Description string `json:"description" yaml:"description"`
// Deployment configuration map (refer to doc/deployments.md)
// Example: {"user.mykey": "foo"}
Config map[string]string `json:"config" yaml:"config"`
// Governor webhook URL for provider triggered scaling requests
// Example: https://n.n.n.n/scale
GovernorWebhookURL string `json:"governor_webhook_url" yaml:"governor_webhook_url"`
}
// Deployment used for displaying a LXD Deployment
DeploymentPost `json:",inline" yaml:",inline"`
DeploymentPut `json:",inline" yaml:",inline"`
// DeploymentShapes keyed by name
// Example: k8s-kubelet (map key)
DeploymentShapes map[string]DeploymentShape `json:"deployment_shapes" yaml:"deployment_shapes"`
// List of URLs of objects using this deployment
// Read only: true
// Example: ["/1.0/instances/c1", "/1.0/instances/c2"]
UsedBy []string `json:"used_by" yaml:"used_by"`
}
Deployment Key related
// DeploymentKeysPost represents the fields required to create a LXD Deployment key
type DeploymentKeysPost struct {
DeploymentKeyPost `json:",inline" yaml:",inline"`
DeploymentKeyPut `json:",inline" yaml:",inline"`
CertificateFingerprint string `json:"certificate_fingerprint" yaml:"certificate_fingerprint"`
}
// DeploymentKeyPost represents the fields required to rename a LXD Deployment key
type DeploymentKeyPost struct {
// The name for the deployment key
Name string `json:"name" yaml:"name"`
}
// DeploymentKeyPut represents the modifiable fields of a deployment key
type DeploymentKeyPut struct {
// Description of the deployment key
// Example: Deployment key for myapp
Description string `json:"description" yaml:"description"`
// The role for a deployment key
// this deployment. Could be either "admin" or "read-only"
Role string `json:"role" yaml:"role"`
}
Deployment Shape related
// DeploymentShapesPost represents the fields required to create a LXD shape
type DeploymentShapesPost struct {
DeploymentShapePost `json:",inline" yaml:",inline"`
DeploymentShapePut `json:",inline" yaml:",inline"`
}
// DeploymentShapePost represents the fields required to rename a LXD shape
type DeploymentShapePost struct {
// The name for the shape
// Example: myapp
Name string `json:"name" yaml:"name"`
}
// DeploymentShapePut represents the modifiable fields of a shape template
type ShapePut struct {
// Description of the shape
// Example: Web servers
Description string `json:"description" yaml:"description"`
// Shape configuration map
// Example: {"user.mykey": "foo"}
Config map[string]string `json:"config" yaml:"config"`
// Instance definition to use for instances in this set
InstanceTemplate InstancesPost `json:"instance_template" yaml:"instance_template"`
// Maximum allowed size of instance set
ScalingMaximum int `yaml:"scaling_maximum" json:"scaling_maximum"`
// Minimum allowed size of instance set
ScalingMinimum int `yaml:"scaling_minimum" json:"scaling_minimum"`
}
// DeploymentShape represents the fields of a shape template
type DeploymentShape struct {
DeploymentShapePost `yaml:",inline"`
DeploymentShapePut `yaml:",inline"`
// Current size of instance set
ScalingCurrent int `yaml:"scaling_current" json:"scaling_current"`
}
Deployment Instance related
// DeploymentInstancesPost represents the fields required to create an instance in an existing LXD deployment shape.
type DeploymentInstancesPost struct {
// The shape name in which to create the instance
// Example: k8s-kubelet
DeploymentShapeName string `json:"shape_name" yaml:"shape_name"`
// The instance name to use
// Example: k8s-kubelet01
InstanceName string `json:"instance_name" yaml:"instance_name"`
}
Management API endpoint changes:
We’ll list the Deployment API REST endpoints. Some of them will be annotated with the MANAGEMENT label (meaning that its use is targeted toward the platform admin) and some others with GOVERNOR (meaning that it will be called from the governor application)
deployments related:
-
GET /1.0/deployments
- List the deployments (filtering with projects and deployment names available). No
recursionURL parameter will return a list of the deployment URLs,recursion=1will return the list of the deployment API objects.
- List the deployments (filtering with projects and deployment names available). No
-
POST /1.0/deployments
- Add a deployment
-
DELETE /1.0/deployments/{deploymentName}
- Delete a deployment
-
GET /1.0/deployments/{deploymentName}
- Get the specific details of a deployment
-
PUT /1.0/deployments/{deploymentName}
- Update a deployment (the update replace and existing deployment with the new one)
-
PATCH /1.0/deployments/{deploymentName}
- Update a deployment (the update merge the existing deployment’s config with the new one)
-
POST /1.0/deployments/{deploymentName}
- Rename a deployment
deployment_keys related:
-
GET /1.0/deployments/{deploymentName}/keys
- List a deployment’s keys. No
recursionURL parameter will return a list of the deployment keys URLs,recursion=1will return the list of the deployment key API objects.
- List a deployment’s keys. No
-
POST /1.0/deployments/{deploymentName}/keys
- Add a deployment key
-
DELETE /1.0/deployments/{deploymentName}/keys/{keyName}
- Delete a deployment key
-
GET /1.0/deployments/{deploymentName}/keys/{keyName}
- Get the specific details of a deployment key
-
POST /1.0/deployments/{deploymentName}/keys/{keyName}
- Rename a deployment key
deployment_shapes related:
-
GET /1.0/deployments/{deploymentName}/shapes
- List the shapes for a given deployment. No
recursionURL parameter will return a list of the shape URLs for a given deployment,recursion=1will return the list of the shape API objects for a given deployment.
- List the shapes for a given deployment. No
-
POST /1.0/deployments/{deploymentName}/shapes
- Creates a new shape within a deployment.
-
DELETE /1.0/deployments/{deploymentName}/shapes/{shapeName}
- Delete a shape.
-
GET /1.0/deployments/{deploymentName}/shapes/{shapeName}
- Gets a specific shape.
-
PUT /1.0/deployments/{deploymentName}/shapes/{shapeName}
- Updates a specific shape.
-
POST /1.0/deployments/{deploymentName}/shapes/{shapeName}
- Rename a specific shape.
Governor-facing API endpoint changes:
deployment_shape_instances related:
-
GET /1.0/deployments/{deploymentName}/shapes/{shapeName}/instances
- List instances in a shape. No
recursionURL parameter will return a list of the instance URLs,recursion=1will return the list of the instance API objects.
- List instances in a shape. No
-
POST /1.0/deployments/{deploymentName}/shapes/{shapeName}/instances/{instanceName}
- Creates a new instance (with the name
instanceName) within this shape. If the shape current size allow a one unit add-up, then the to-be-created instance will be launched using the shapeinstance_template. Should the new request exceed the maximum number (scaling_maximum), the request will fail.
- Creates a new instance (with the name
-
DELETE /1.0/deployments/{deploymentName}/shapes/{shapeName}/instances/{instanceName}
- Deletes an existing instance (with the name
instanceName) within this shape. If the shape current size can be shrinked from one unit, then the instance will be effectively deleted. Should the new request subceed the minimum number (scaling_minimum), the request will fail.
- Deletes an existing instance (with the name
-
PUT /1.0/deployments/{deploymentName}/shapes/{shapeName}/instances/{instanceName}/state
- Update the state of an instance (start / stop / etc.) within this shape.
CLI Changes:
We will introduce the following CLI changes:
deployment
├─ list [--format]
├─ show [<remote>:]<deployment_name>
├─ get [<remote>:]<deployment_name> <key/property> [--property]
├─ set [<remote>:]<deployment_name> (<key1>=<value1> <key2>=<value2> ...) [--property]
├─ unset [<remote>:]<deployment_name> (<key1/property1> <key2/property2> ...) [--property]
├─ create [<remote>:]<new_deployment_name>
├─ edit [<remote>:]<deployment_name>
├─ rename [<remote>:]<deployment_name> <new_deployment_name>
├─ delete [<remote>:]<deployment_name>
├─ key
├─ list [<remote>:]<deployment_name>
├─ show [<remote>:]<deployment_name> <key_name>
├─ get [<remote>:]<deployment_name> <key_name> <key/property> [--property]
├─ create [<remote>:]<deployment_name> <new_key_name> <certificate_full_fingerprint> [--role]
├─ rename [<remote>:]<deployment_name> <key_name> <new_key_name>
├─ delete [<remote>:]<deployment_name> <key_name>
├─ shape
├─ list [--format]
├─ show [<remote>:]<deployment_name> <shape_name>
├─ create [<remote>:]<deployment_name> <new_shape_name> [--from-profile <instance_profile>, --from-image <remote:alias>, --scaling-min <min>, --scaling-max <max>, --vm]
├─ get [<remote>:]<deployment_name> <shape_name> <key/property> [--property]
├─ set [<remote>:]<deployment_name> <shape_name> (<key1>=<value1> <key2>=<value2> ...) [--property]
├─ unset [<remote>:]<deployment_name> <shape_name> (<key1/property1> <key2/property2> ...) [--property]
├─ edit [<remote>:]<deployment_name> <shape_name>
├─ rename [<remote>:]<deployment_name> <shape_name> <new_shape_name>
├─ delete [<remote>:]<deployment_name> <shape_name>
├─ instance
├─ launch [<remote>:]<deployment_name> <shape_name> <instance_name>
├─ delete [<remote>:]<deployment_name> <shape_name> <instance_name>
├─ list [<remote>:]<deployment_name> <shape_name> [--format]
link to WIP PR: https://github.com/canonical/lxd/pull/12284