Wow… That’s it! Thanks a lot @masnax , @tomp for your guidance.
1 Like
There’s now another problem. LXD has again failed to start up after a server reboot.
Running lxc ls now returns Error: The LXD daemon doesn't appear to be started (socket path: /var/snap/lxd/common/lxd/unix.socket).
And, lxd --debug --group lxd, returns:
DEBUG [2024-02-14T01:00:44+02:00] Connecting to a local LXD over a Unix socket
DEBUG [2024-02-14T01:00:44+02:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0"
INFO [2024-02-14T01:00:44+02:00] LXD is starting mode=normal path=/var/snap/lxd/common/lxd version=5.0.3
INFO [2024-02-14T01:00:44+02:00] Kernel uid/gid map:
INFO [2024-02-14T01:00:44+02:00] - u 0 0 4294967295
INFO [2024-02-14T01:00:44+02:00] - g 0 0 4294967295
INFO [2024-02-14T01:00:44+02:00] Configured LXD uid/gid map:
INFO [2024-02-14T01:00:44+02:00] - u 0 1000000 1000000000
INFO [2024-02-14T01:00:44+02:00] - g 0 1000000 1000000000
INFO [2024-02-14T01:00:44+02:00] Kernel features:
INFO [2024-02-14T01:00:44+02:00] - closing multiple file descriptors efficiently: yes
INFO [2024-02-14T01:00:44+02:00] - netnsid-based network retrieval: yes
INFO [2024-02-14T01:00:44+02:00] - pidfds: yes
INFO [2024-02-14T01:00:44+02:00] - core scheduling: yes
INFO [2024-02-14T01:00:44+02:00] - uevent injection: yes
INFO [2024-02-14T01:00:44+02:00] - seccomp listener: yes
INFO [2024-02-14T01:00:44+02:00] - seccomp listener continue syscalls: yes
INFO [2024-02-14T01:00:44+02:00] - seccomp listener add file descriptors: yes
INFO [2024-02-14T01:00:44+02:00] - attach to namespaces via pidfds: yes
INFO [2024-02-14T01:00:44+02:00] - safe native terminal allocation : yes
INFO [2024-02-14T01:00:44+02:00] - unprivileged file capabilities: yes
INFO [2024-02-14T01:00:44+02:00] - cgroup layout: cgroup2
WARNING[2024-02-14T01:00:44+02:00] - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored
WARNING[2024-02-14T01:00:44+02:00] - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead
INFO [2024-02-14T01:00:44+02:00] - shiftfs support: yes
INFO [2024-02-14T01:00:44+02:00] - idmapped mounts kernel support: yes
INFO [2024-02-14T01:00:44+02:00] Instance type operational driver=lxc features="map[]" type=container
ERROR [2024-02-14T01:00:44+02:00] Unable to run feature checks during QEMU initialization: QEMU premature exit: exit status 1 (qemu-system-x86_64: -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd: Could not open '/usr/share/OVMF/OVMF_CODE.fd': No such file or directory)
WARNING[2024-02-14T01:00:44+02:00] Instance type not operational driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
INFO [2024-02-14T01:00:44+02:00] Initializing local database
DEBUG [2024-02-14T01:00:44+02:00] Refreshing local trusted certificate cache
INFO [2024-02-14T01:00:44+02:00] Set client certificate to server certificate fingerprint=5eaba09b34ee628564424b60a85f6569c4b0ecdf3f7818b6845b4fa5a0751671
DEBUG [2024-02-14T01:00:44+02:00] Initializing database gateway
INFO [2024-02-14T01:00:44+02:00] Starting database node id=1 local=1 role=voter
ERROR [2024-02-14T01:00:44+02:00] Failed to start the daemon err="Failed to start dqlite server: raft_start(): io: closed segment 0000000000671011-0000000000671051 is past last snapshot snapshot-1-670720-39270450"
INFO [2024-02-14T01:00:44+02:00] Starting shutdown sequence signal=interrupt
INFO [2024-02-14T01:00:44+02:00] Not unmounting temporary filesystems (instances are still running)
INFO [2024-02-14T01:00:44+02:00] Daemon stopped
Error: Failed to start dqlite server: raft_start(): io: closed segment 0000000000671011-0000000000671051 is past last snapshot snapshot-1-670720-39270450
How may this get resolved?
Looks like your database got a bit corrupted somehow. First I would back up everything in /var/snap/lxd/common/lxd/database and then proceed from there.
You will likely have to remove some segment files and snapshots from the global database directory.
Could you please post the output of ls -lah /var/snap/lxd/common/lxd/database/global/
2 Likes
I’ve looped in our colleagues on the dqlite team to help answer this.
I suspect the server was shutdown abruptly and this has left inconsistent raft snapshots behind.
1 Like
Please can you show the output of:
sudo ls -lah /var/snap/lxd/common/lxd/database/global
After posting about the LXD failure to start again after Server reboot, I actually decided to go ahead and restore
/var/snap/lxd/common/lxd
from a backup that I took earlier, after the initial stall. Thereafter I went ahead to again
delete from storage_pools where name='lxddsk';
using
/var/snap/lxd/common/lxd/database/patch.global.sql
as previously advised; and actually got LXD up and running again. So, I decided to let it run that way to minimize user disruption; until over the weekend when I can have more time to conclusively resolve the issue.
As requested, @tomp @masnax the output of ls -lah /var/snap/lxd/common/lxd/database/global/ (hoping that this could nonetheless provide useful insight, though it is taken before the server reboot that gave rise to the latter
Error: The LXD daemon doesn't appear to be started (socket path: /var/snap/lxd/common/lxd/unix.socket)
issue that I reported above), is as below. The actual output is quite large (like 8810 lines), and so I have attempted shorten what I would have had to share here as follows (I hope that it can be easily understood ):
total 96M
drwxr-x--- 2 root root 768K Feb 14 01:25 .
drwx------ 4 root root 4.0K Feb 14 01:25 ..
-rw------- 1 root root 4.1K Feb 11 12:18 0000000000659457-0000000000659457
-rw------- 1 root root 4.1K Feb 11 12:18 0000000000659458-0000000000659458
-rw------- 1 root root 4.1K Feb 11 12:18 0000000000659459-0000000000659459
-rw------- 1 root root 4.1K Feb 11 12:19 0000000000659460-0000000000659460
-rw------- 1 root root 4.1K Feb 11 12:19 0000000000659461-0000000000659461
-rw------- 1 root root 4.1K Feb 11 12:19 0000000000659462-0000000000659462
-rw------- 1 root root 4.1K Feb 11 12:19 0000000000659463-0000000000659463
-rw------- 1 root root 4.1K Feb 11 12:19 0000000000659464-0000000000659464
-rw------- 1 root root 4.1K Feb 11 12:20 0000000000659465-0000000000659465
-rw------- 1 root root 4.1K Feb 11 12:20 0000000000659466-0000000000659466
-rw------- 1 root root 4.1K Feb 11 12:20 0000000000659467-0000000000659467
-rw------- 1 root root 4.1K Feb 11 12:20 0000000000659468-0000000000659468
-rw------- 1 root root 4.1K Feb 11 12:20 0000000000659469-0000000000659469
-rw------- 1 root root 4.1K Feb 11 12:21 0000000000659470-0000000000659470
-rw------- 1 root root 4.1K Feb 11 12:21 0000000000659471-0000000000659471
-rw------- 1 root root 4.1K Feb 11 12:21 0000000000659472-0000000000659472
-rw------- 1 root root 4.1K Feb 11 12:21 0000000000659473-0000000000659473
-rw------- 1 root root 4.1K Feb 11 12:21 0000000000659474-0000000000659474
-rw------- 1 root root 4.1K Feb 11 12:22 0000000000659475-0000000000659475
-rw------- 1 root root 4.1K Feb 11 12:22 0000000000659476-0000000000659476
-rw------- 1 root root 4.1K Feb 11 12:22 0000000000659477-0000000000659477
-rw------- 1 root root 4.1K Feb 11 12:22 0000000000659478-0000000000659478
-rw------- 1 root root 4.1K Feb 11 12:22 0000000000659479-0000000000659479
-rw------- 1 root root 4.1K Feb 11 12:22 0000000000659480-0000000000659480
-rw------- 1 root root 4.1K Feb 11 12:23 0000000000659481-0000000000659481
-rw------- 1 root root 4.1K Feb 11 12:23 0000000000659482-0000000000659482
-rw------- 1 root root 4.1K Feb 11 12:23 0000000000659483-0000000000659483
-rw------- 1 root root 4.1K Feb 11 12:23 0000000000659484-0000000000659484
-rw------- 1 root root 4.1K Feb 11 12:23 0000000000659485-0000000000659485
-rw------- 1 root root 4.1K Feb 11 12:24 0000000000659486-0000000000659486
-rw------- 1 root root 4.1K Feb 11 12:24 0000000000659487-0000000000659487
-rw------- 1 root root 4.1K Feb 11 12:24 0000000000659488-0000000000659488
-rw------- 1 root root 4.1K Feb 11 12:24 0000000000659489-0000000000659489
-rw------- 1 root root 4.1K Feb 11 12:24 0000000000659490-0000000000659490
-rw------- 1 root root 4.1K Feb 11 12:25 0000000000659491-0000000000659491
-rw------- 1 root root 4.1K Feb 11 12:25 0000000000659492-0000000000659492
-rw------- 1 root root 4.1K Feb 11 12:25 0000000000659493-0000000000659493
-rw------- 1 root root 4.1K Feb 11 12:25 0000000000659494-0000000000659494
-rw------- 1 root root 4.1K Feb 11 12:25 0000000000659495-0000000000659495
-rw------- 1 root root 4.1K Feb 11 12:26 0000000000659496-0000000000659496
-rw------- 1 root root 4.1K Feb 11 12:26 0000000000659497-0000000000659497
-rw------- 1 root root 4.1K Feb 11 12:26 0000000000659498-0000000000659498
-rw------- 1 root root 4.1K Feb 11 12:26 0000000000659499-0000000000659499
-rw------- 1 root root 4.1K Feb 11 12:26 0000000000659500-0000000000659500
-rw------- 1 root root 4.1K Feb 11 12:27 0000000000659501-0000000000659501
...
-rw------- 1 root root 4.1K Feb 11 12:46 0000000000659599-0000000000659599
-rw------- 1 root root 4.1K Feb 11 12:46 0000000000659600-0000000000659600
-rw------- 1 root root 4.1K Feb 11 12:46 0000000000659601-0000000000659601
...
-rw------- 1 root root 4.1K Feb 11 13:28 0000000000659799-0000000000659799
-rw------- 1 root root 4.1K Feb 11 13:28 0000000000659800-0000000000659800
-rw------- 1 root root 4.1K Feb 11 13:28 0000000000659801-0000000000659801
...
-rw------- 1 root root 4.1K Feb 11 13:49 0000000000659899-0000000000659899
-rw------- 1 root root 4.1K Feb 11 13:49 0000000000659900-0000000000659900
-rw------- 1 root root 4.1K Feb 11 13:49 0000000000659901-0000000000659901
...
-rw------- 1 root root 4.1K Feb 11 14:10 0000000000659999-0000000000659999
-rw------- 1 root root 4.1K Feb 11 14:10 0000000000660000-0000000000660000
-rw------- 1 root root 4.1K Feb 11 14:10 0000000000660001-0000000000660001
...
-rw------- 1 root root 4.1K Feb 11 17:40 0000000000660999-0000000000660999
-rw------- 1 root root 4.1K Feb 11 17:40 0000000000661000-0000000000661000
-rw------- 1 root root 4.1K Feb 11 17:40 0000000000661001-0000000000661001
...
-rw------- 1 root root 4.1K Feb 11 21:09 0000000000661999-0000000000661999
-rw------- 1 root root 4.1K Feb 11 21:09 0000000000662000-0000000000662000
-rw------- 1 root root 4.1K Feb 11 21:09 0000000000662001-0000000000662001
...
-rw------- 1 root root 4.1K Feb 12 00:39 0000000000662999-0000000000662999
-rw------- 1 root root 4.1K Feb 12 00:39 0000000000663000-0000000000663000
-rw------- 1 root root 4.1K Feb 12 00:39 0000000000663001-0000000000663001
...
-rw------- 1 root root 4.1K Feb 12 04:08 0000000000663999-0000000000663999
-rw------- 1 root root 4.1K Feb 12 04:08 0000000000664000-0000000000664000
-rw------- 1 root root 4.1K Feb 12 04:08 0000000000664001-0000000000664001
...
-rw------- 1 root root 4.1K Feb 12 07:38 0000000000664999-0000000000664999
-rw------- 1 root root 4.1K Feb 12 07:38 0000000000665000-0000000000665000
-rw------- 1 root root 4.1K Feb 12 07:38 0000000000665001-0000000000665001
...
-rw------- 1 root root 4.1K Feb 12 11:08 0000000000665999-0000000000665999
-rw------- 1 root root 4.1K Feb 12 11:08 0000000000666000-0000000000666000
-rw------- 1 root root 4.1K Feb 12 11:08 0000000000666001-0000000000666001
...
-rw------- 1 root root 4.1K Feb 12 14:43 0000000000666999-0000000000666999
-rw------- 1 root root 4.1K Feb 12 14:43 0000000000667000-0000000000667000
-rw------- 1 root root 4.1K Feb 12 14:43 0000000000667001-0000000000667001
...
-rw------- 1 root root 4.1K Feb 12 18:31 0000000000667999-0000000000667999
-rw------- 1 root root 4.1K Feb 12 18:31 0000000000668000-0000000000668000
-rw------- 1 root root 4.1K Feb 12 18:32 0000000000668001-0000000000668001
...
-rw------- 1 root root 4.1K Feb 12 18:48 0000000000668079-0000000000668079
-rw------- 1 root root 4.1K Feb 12 18:48 0000000000668080-0000000000668080
-rw------- 1 root root 4.1K Feb 12 18:49 0000000000668081-0000000000668081
... ...
-rw------- 1 root root 4.1K Feb 12 19:19 0000000000668226-0000000000668226
-rw------- 1 root root 4.1K Feb 12 19:20 0000000000668227-0000000000668227
-rw------- 1 root root 4.1K Feb 12 19:20 0000000000668228-0000000000668228
-rw------- 1 root root 4.1K Feb 12 19:20 0000000000668229-0000000000668229
-rw------- 1 root root 4.1K Feb 12 19:20 0000000000668230-0000000000668230
-rw------- 1 root root 4.1K Feb 12 19:21 0000000000668231-0000000000668231
-rw------- 1 root root 4.1K Feb 12 19:21 0000000000668232-0000000000668232
-rw------- 1 root root 4.1K Feb 12 19:21 0000000000668233-0000000000668233
-rw------- 1 root root 4.1K Feb 12 19:21 0000000000668234-0000000000668234
-rw------- 1 root root 4.1K Feb 12 19:21 0000000000668235-0000000000668235
-rw------- 1 root root 4.1K Feb 12 19:22 0000000000668236-0000000000668236
-rw------- 1 root root 4.1K Feb 12 19:22 0000000000668237-0000000000668237
-rw------- 1 root root 4.1K Feb 12 19:22 0000000000668238-0000000000668238
-rw------- 1 root root 4.1K Feb 12 19:22 0000000000668239-0000000000668239
-rw------- 1 root root 4.1K Feb 12 19:22 0000000000668240-0000000000668240
-rw------- 1 root root 4.1K Feb 12 19:23 0000000000668241-0000000000668241
-rw------- 1 root root 4.1K Feb 12 19:23 0000000000668242-0000000000668242
-rw------- 1 root root 4.1K Feb 12 19:23 0000000000668243-0000000000668243
-rw------- 1 root root 4.1K Feb 12 19:23 0000000000668244-0000000000668244
-rw------- 1 root root 4.1K Feb 12 19:23 0000000000668245-0000000000668245
-rw------- 1 root root 4.1K Feb 12 19:24 0000000000668246-0000000000668246
-rw------- 1 root root 4.1K Feb 12 19:24 0000000000668247-0000000000668247
-rw------- 1 root root 4.1K Feb 12 19:24 0000000000668248-0000000000668248
-rw------- 1 root root 4.1K Feb 12 19:24 0000000000668249-0000000000668249
-rw------- 1 root root 4.1K Feb 12 19:25 0000000000668250-0000000000668250
-rw------- 1 root root 4.1K Feb 12 19:25 0000000000668251-0000000000668251
-rw------- 1 root root 4.1K Feb 12 19:25 0000000000668252-0000000000668252
-rw------- 1 root root 4.1K Feb 12 19:25 0000000000668253-0000000000668253
-rw------- 1 root root 4.1K Feb 12 19:25 0000000000668254-0000000000668254
-rw------- 1 root root 4.1K Feb 12 19:26 0000000000668255-0000000000668255
-rw------- 1 root root 4.1K Feb 12 19:26 0000000000668256-0000000000668256
-rw------- 1 root root 4.1K Feb 12 19:26 0000000000668257-0000000000668257
-rw------- 1 root root 4.1K Feb 12 19:26 0000000000668258-0000000000668258
-rw------- 1 root root 4.1K Feb 12 19:26 0000000000668259-0000000000668259
-rw------- 1 root root 4.1K Feb 12 19:27 0000000000668260-0000000000668260
-rw------- 1 root root 4.1K Feb 12 19:27 0000000000668261-0000000000668261
-rw------- 1 root root 4.1K Feb 12 19:27 0000000000668262-0000000000668262
-rw------- 1 root root 4.1K Feb 12 19:27 0000000000668263-0000000000668263
-rw------- 1 root root 4.1K Feb 12 19:28 0000000000668264-0000000000668264
-rw------- 1 root root 4.1K Feb 12 19:28 0000000000668265-0000000000668265
-rw------- 1 root root 4.1K Feb 12 19:28 0000000000668266-0000000000668266
-rw------- 1 root root 4.1K Feb 12 19:28 0000000000668267-0000000000668267
-rw------- 1 root root 4.1K Feb 12 19:28 0000000000668268-0000000000668268
-rw------- 1 root root 4.1K Feb 12 19:29 0000000000668269-0000000000668269
-rw------- 1 root root 4.1K Feb 12 19:29 0000000000668270-0000000000668270
-rw------- 1 root root 4.1K Feb 12 19:29 0000000000668271-0000000000668271
-rw------- 1 root root 4.1K Feb 12 19:29 0000000000668272-0000000000668272
-rw------- 1 root root 4.1K Feb 12 19:29 0000000000668273-0000000000668273
-rw------- 1 root root 552K Feb 12 19:29 db.bin
-rw------- 1 root root 1.7M Feb 12 19:29 db.bin-wal
-rw------- 1 root root 32 Sep 30 2022 metadata1
-rw------- 1 root root 8.0M Feb 14 17:25 open-1
-rw------- 1 root root 8.0M Feb 14 01:25 open-2
-rw------- 1 root root 8.0M Feb 14 01:25 open-3
-rw------- 1 root root 86K Feb 12 13:26 snapshot-1-666624-6004746
-rw------- 1 root root 56 Feb 12 13:26 snapshot-1-666624-6004746.meta
-rw------- 1 root root 86K Feb 12 17:10 snapshot-1-667648-1265697
-rw------- 1 root root 56 Feb 12 17:10 snapshot-1-667648-1265697.meta
whereby, the
...
gap represents contiguous files from
-rw------- 1 root root 4.1K Feb dd HH:MM 00000000006NNN02-00000000006NNN02
to
-rw------- 1 root root 4.1K Feb dd HH:MM 00000000006NNN98-00000000006NNN98
as fitting into the names sequence; whereas the
... ...
gap represents contiguous files from
-rw------- 1 root root 4.1K Feb 12 18:49 0000000000668082-0000000000668082
to
-rw------- 1 root root 4.1K Feb 12 19:19 0000000000668225-0000000000668225
also as fitting into the sequence respectively.
Thanks.
2 Likes
dqlite team member here—thanks for the information!
The first thing I notice is that there are quite a lot of closed segment files that only contain a single entry (these are the files where the start and end of the range in the filename are the same). That is not incorrect per se but it’s definitely pathological. I will try to figure out what could cause dqlite to create so many segments of this type.
With the list of files that you posted, you indeed won’t get the error closed segment ... is past last snapshot that you reported (as expected since your server is now running smoothly). If you get that error again, please report it here with the corresponding list of files and I’ll advise on how you can fix it.
2 Likes
I did perform the promised Server reboot like 36 hours ago. And, the reboot was done after performing manual APT system updates. After the mentioned Server reboot, LXD fired up without any issues; and has been running smoothly ever since. Just wondering, did the issue get resolved through the updates?
1 Like
There’s been no updates to the LXD snap in this time.
Glad to hear it working now 
1 Like
I tried these steps but still keep getting the same issue
Error: Failed to connect to local LXD: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied
The patch.global.sql file has been created with the queries to be inputed, I re-ran the comand
lxd init --minimal , but i got the same error.
What is your error and what is the contents of the patch.global.sql file you’re using (as each one will likely need to be different).
Here is the error message
Error: Failed to connect to local LXD: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied
Here is the SQL content
delete from storage_pools where name='lxddsk';
And what is the error you’re seeing in /var/snap/lxd/common/lxd/logs/lxd.log?
It seems likely if you’re being affected by the same issue, then your storage pool is likely named something different.
These are the logs in the /var/snap/lxd/common/lxd/logs/logs.txt
time="2024-02-28T18:48:17+01:00" level=warning msg="AppArmor support has been disabled because of lack of kernel support" time="2024-02-28T18:48:17+01:00" level=warning msg=" - AppArmor support has been disabled, Disabled because of lack of kernel support" time="2024-02-28T18:48:17+01:00" level=warning msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored" time="2024-02-28T18:48:17+01:00" level=warning msg="Instance type not operational" driver=qemu err="vhost_vsock kernel module not loaded" type=virtual-machine time="2024-02-28T18:48:17+01:00" level=error msg="Failed to start the daemon" err="Failed to initialize global database: failed to ensure schema: failed to execute
Looks like your logs got cut off before the actual error - is it possible for you to get the whole logs, maybe look also in journalctl?
i do this, but its still not help me.
lxc list
Error: LXD unix socket "/var/snap/lxd/common/lxd-user/unix.socket" not accessible: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd-user/unix.socket: connect: connection refused
lxc version
Client version: 5.21.1 LTS
Server version: unreachable
1 Like
This thread is regarding LXD 5.0.3, have you looked at the LXD 5.21.1 release notes page?
Yes, I use different version (auto updates/ latest stable release)
but after update happened, same issue that described in current discussion.
The 5.21.1 require manual actions to migrate ?
If yes, please share it.
If no, help me solve my issue. It’s blocker for my servers.
Please can you post on the LXD 5.21.1 thread.
1 Like
i moved my question to 5.12.1 thread
1 Like