Hi everyone,
With the UI team working on their 6 month planning I wanted to propose an architectural addition to the LXD API.
A proxy/query abstraction layer for historical metrics and audit events.
The problem: Observability gap in LXD UI / MicroCloud
Historically, LXD was CLI and API focussed. If you wanted metrics you could choose between cli tools such as htop, or set up Prometheus/Grafana and look at dashboards.
Over the past few years LXD has evolved significantly.
- The UI has matured into the primary daily management interface for many
- MicroCloud positions LXD as an out-of-the-box private cloud solution
- Fine-grained ACL is now a first class citizen
In basically every competing platform (Proxmox, Harvester, Nutanix, OpenShift, OpenStack, OpenNebula, Public clouds), the web console has the ability to show performance graphs (CPU, Memory, Storage, Network etc)
Currently in the UI, only the latest snapshot of metrics can be displayed, scraped on page load from /1.0/metrics.
To see historical data, users must rely on external tools such as Grafana dashboards, switching between the UI and a Grafana instance. Access and RBAC would have to be configured for both, if at all possible. This is impractical in multi-tenant environments and for restricted users.
Proposal 1: Time-Series metrics API
Building on the existing Prometheus integration: an external Prometheus instance (or multiple) scraping and storing metrics from /1.0/metrics.
LXD could introduce an RBAC aware time-series endpoint.
lxc config set core.metrics_provider_url="http://prometheus.internal.domain:9090"
A new REST API endpoint, for example:
GET /1.0/instances/{name}/metrics/history?start=...&end=...&step=...
GET /1.0/cluster/members/{name}/metrics/history
LXD authenticates the user, checks RBAC, queries the configured Prometheus backend ( {instance="<name>", project="<project>"}) and returns a clean JSON.
MicroCloud added value
MicroCloud could optionally deploy a Prometheus instance on a selected or multiple nodes, and configure core.metrics_provider_url automatically, providing the user a more complete out-of-the-box private cloud solution.
Proposal 2: Audit & Event history in Loki
As mentioned in Feature: Operation History + Requestor Address Tracking by larrasket · Pull Request #17776 · canonical/lxd · GitHub, an operations rework is underway.
The new operations subsystem will keep a short history (<24 hours). For longer history, users are directed to Grafana/Loki. This leaves us with the same access & RBAC issue as we have with metrics.
We can apply the same proxy pattern here, once the operation rework lands:
- LXD ships lifecycle and operation events to Loki.
- LXD exposes an RBAC-scoped audit endpoint:
GET /1.0/projects/{project}/events/history?start=...&limit=...&identity=...
GET /1.0/instances/{name}/events/history
Both the CLI and the UI can then display audit data. We will finally have long term insight into who, what and when. (User X restarted container Y 3 days ago)
Similar to proposal 1, MicroCloud could optionally deploy a Loki instance on a selected or multiple nodes. Providing an even more complete private cloud solution.
Benefits
-
LXD sticks to its core principles. No bloat in the database. Utilizing external systems built for this principle.
-
Strict security & RBAC. Users do not need access to Prometheus/Loki/Grafana. LXD enforces isolation and prevents cross-tenant access.
-
Unified Management Experience. No more switching between the UI and an external Grafana instance.
Questions for you
-
Is the proxy pattern architecturally acceptable, or is there hesitation around LXD querying an external TSDB at request time?
-
Is my suggestion regarding operations history/audit in line with the rework on the operation subsystem?
I’d love to hear your thoughts and feedback about my suggestion.
The UI is absolutely amazing (hence my active contribution). I can’t wait for the next release with the new dashboard. A feature like these will make the whole project feel so much more complete.
Related feature requests:
edit: Edlerd even requested a similar feature: Instance CPU and Memory usage via API · Issue #15639 · canonical/lxd · GitHub