Salvage a dead sqlite database

LXD
1

Hi all,

I have a 6/stable LXD that I’m trying to rescue from what I can gather to be a pretty bad state:

DEBUG  [2025-10-29T18:49:26-05:00] Database error                                err="Failed to ensure schema: Failed to update cluster member version info for \"\": updated 0 rows instead of 1"
ERROR  [2025-10-29T18:49:26-05:00] Failed starting daemon                        err="Failed to initialize global database: Failed to ensure schema: Failed to ensure schema: Failed to update cluster member version info for \"\": updated 0 rows instead of 1"
INFO   [2025-10-29T18:49:26-05:00] Starting shutdown sequence                    signal=interrupt

The database itself appears to be readable and I see the nodes table is populated with the one node:

root@supermicro:/var/snap/lxd/common/lxd/database/global# **sqlite3 -json $PWD/db.bin "select * from nodes;"  | jq**
[
  {
    "id": 1,
    "name": "supermicro",
    "description": "",
    "address": "172.16.66.10:8443",
    "schema": 73,
    "api_extensions": 433,
    "heartbeat": "2025-10-29 18:19:35.620437543-05:00",
    "state": 0,
    "arch": 2,
    "failure_domain_id": null

if I can’t get the current /var/snap/lxd/common/lxd/database/global/ database rehabilitated, is it possible to somehow dump the contents of the database via sqlite, delete the global directory or any other database, and then restore just the configuration data and hopefully the recognition of existing VMs and storage pools?

Trying sqlite3 db.bin < db.dump.sql unfortunately didn’t work.

2

Hi there, sorry to hear you’re having trouble with your LXD dqlite database.

Is this a LXD cluster?

Please can you provide me with the output of snap list lxd, and also please can you provide me with the output of the following command for each server:

apt install sqlite3
sqlite3 -header /var/snap/lxd/common/lxd/database/local.db 'select * from config'
sqlite3 -header /var/snap/lxd/common/lxd/database/local.db 'select * from patches'
3

Here are the requested outputs: Ubuntu Pastebin

root@supermicro:/var/snap/lxd/common/lxd/database# snap list lxd
Name  Version      Rev    Tracking  Publisher   Notes
lxd   6.5-ccdfb39  36020  6/stable  canonical✓  -

This was originally a cluster which I eventually downsized into a single-node instance after running recover-from-quorum-loss. This worked fine for some time.

Yesterday, I wanted to further remove the clustering altogether (as it was still enabled just as a single-node cluster which causes it to appear as a cluster in MAAS KVM pod functionality), which led me to this post to try to disable clustering:

https://discuss.linuxcontainers.org/t/disable-or-reconfig-lxd-cluster-to-standalone-lxd/8415/3

Against better judgement I ran the queries, which likely caused the breaking:

rm $LXD_DIR/cluster.crt $LXD_DIR/cluster.key
sqlite3 $LXD_DIR/database/local.db "DELETE FROM config WHERE key='cluster.https_address'"
sqlite3 $LXD_DIR/database/local.db "DELETE FROM raft_nodes"

after which the node was no longer starting:

Oct 30 10:18:38 supermicro lxd.daemon[13181]: time="2025-10-30T10:18:38-05:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.pr
iority instead"
Oct 30 10:18:39 supermicro lxd.daemon[13181]: time="2025-10-30T10:18:39-05:00" level=error msg="Failed starting daemon" err="Failed to initialize global database: Failed to ensure schema: Failed to ensure schema: Failed to update clu
ster member version info for \"\": updated 0 rows instead of 1"
Oct 30 10:18:39 supermicro lxd.daemon[13181]: time="2025-10-30T10:18:39-05:00" level=warning msg="Dqlite last entry" index=0 term=0
Oct 30 10:18:39 supermicro lxd.daemon[13181]: Error: Failed to initialize global database: Failed to ensure schema: Failed to ensure schema: Failed to update cluster member version info for "": updated 0 rows instead of 1
Oct 30 10:18:39 supermicro lxd.daemon[12990]: Killed
Oct 30 10:18:39 supermicro lxd.daemon[12990]: => LXD failed to start
Oct 30 10:18:39 supermicro systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Oct 30 10:18:39 supermicro systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Oct 30 10:18:40 supermicro systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 2.
Oct 30 10:18:40 supermicro systemd[1]: Started snap.lxd.daemon.service - Service for snap application lxd.daemon.
Oct 30 10:18:40 supermicro lxd.daemon[13247]: INFO: the gpu-2404 interface isn't connected. Skipping gpu-2404 wrapper.
Oct 30 10:18:40 supermicro lxd.daemon[13247]: => Preparing the system (36020)

I’ve encountered this state a few times before in other scenarios, and while this is a lab environment and I can always delete and start fresh, I am hoping to at least be able to recover the VMs and maybe come up with a disaster recovery strategy

4

A quick follow-up:

I eventually removed the global/ directory, restarted LXD which started with an empty config, and used lxd recover to re-discover my pools/vms.

2 Likes