LXD 5.0.3 on Ubuntu 22.04 stopped working, with lxc ls returning: Error: Get "http://unix.socket/1.0": EOF

rmodiah · February 13, 2024, 12:02pm

Hello Community.

LXD 5.0.3 on Ubuntu 22.04 (kernel 5.15.0-94-generic and snap 2.61.1) stopped working and all instances suddenly went down. Server had hitherto been running since Nov 2022. I believe that Ubuntu auto-update (and the accompanying snap refresh) may have caused this; as no manual update of any kind has been done in the past month or so.

Running lxc ls returns:
Error: Get "http://unix.socket/1.0": EOF,

and,
lxd --debug --group lxd
returns:

DEBUG  [2024-02-13T12:19:51+02:00] Connecting to a local LXD over a Unix socket
DEBUG  [2024-02-13T12:19:51+02:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0"
INFO   [2024-02-13T12:19:53+02:00] LXD is starting                               mode=normal path=/var/snap/lxd/common/lxd
 version=5.0.3
INFO   [2024-02-13T12:19:53+02:00] Kernel uid/gid map:
INFO   [2024-02-13T12:19:53+02:00]  - u 0 0 4294967295
INFO   [2024-02-13T12:19:53+02:00]  - g 0 0 4294967295
INFO   [2024-02-13T12:19:53+02:00] Configured LXD uid/gid map:
INFO   [2024-02-13T12:19:53+02:00]  - u 0 1000000 1000000000
INFO   [2024-02-13T12:19:53+02:00]  - g 0 1000000 1000000000
INFO   [2024-02-13T12:19:53+02:00] Kernel features:
INFO   [2024-02-13T12:19:53+02:00]  - closing multiple file descriptors efficiently: yes
INFO   [2024-02-13T12:19:53+02:00]  - netnsid-based network retrieval: yes
INFO   [2024-02-13T12:19:53+02:00]  - pidfds: yes
INFO   [2024-02-13T12:19:53+02:00]  - core scheduling: yes
INFO   [2024-02-13T12:19:53+02:00]  - uevent injection: yes
INFO   [2024-02-13T12:19:53+02:00]  - seccomp listener: yes
INFO   [2024-02-13T12:19:53+02:00]  - seccomp listener continue syscalls: yes
INFO   [2024-02-13T12:19:53+02:00]  - seccomp listener add file descriptors: yes
INFO   [2024-02-13T12:19:53+02:00]  - attach to namespaces via pidfds: yes
INFO   [2024-02-13T12:19:53+02:00]  - safe native terminal allocation : yes
INFO   [2024-02-13T12:19:53+02:00]  - unprivileged file capabilities: yes
INFO   [2024-02-13T12:19:53+02:00]  - cgroup layout: cgroup2
WARNING[2024-02-13T12:19:53+02:00]  - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored
WARNING[2024-02-13T12:19:53+02:00]  - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead
INFO   [2024-02-13T12:19:53+02:00]  - shiftfs support: yes
INFO   [2024-02-13T12:19:53+02:00]  - idmapped mounts kernel support: yes
ERROR  [2024-02-13T12:19:53+02:00] Unable to run feature checks during QEMU initialization: QEMU premature exit: exit status 1 (qemu-system-x86_64: -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd: Could not open '/usr/share/OVMF/OVMF_CODE.fd': No such file or directory)
WARNING[2024-02-13T12:19:53+02:00] Instance type not operational                 driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
INFO   [2024-02-13T12:19:53+02:00] Instance type operational                     driver=lxc features="map[]" type=container
INFO   [2024-02-13T12:19:53+02:00] Initializing local database
DEBUG  [2024-02-13T12:19:53+02:00] Refreshing local trusted certificate cache
INFO   [2024-02-13T12:19:53+02:00] Set client certificate to server certificate  fingerprint=5eaba09b34ee628564424b60a85f6569c4b0ecdf3f7818b6845b4fa5a0751671
DEBUG  [2024-02-13T12:19:53+02:00] Initializing database gateway
INFO   [2024-02-13T12:19:53+02:00] Starting database node                        id=1 local=1 role=voter
INFO   [2024-02-13T12:19:54+02:00] Loading daemon configuration
INFO   [2024-02-13T12:19:54+02:00] Binding socket                                socket=/var/snap/lxd/common/lxd/unix.socket type="REST API Unix socket"
INFO   [2024-02-13T12:19:54+02:00] Binding socket                                socket=/var/snap/lxd/common/lxd/devlxd/sock type="devlxd socket"
INFO   [2024-02-13T12:19:54+02:00] Binding socket                                socket="[::]:8443" type="REST API TCP socket"
INFO   [2024-02-13T12:19:54+02:00] Initializing global database
INFO   [2024-02-13T12:19:54+02:00] Connecting to global database
DEBUG  [2024-02-13T12:19:54+02:00] Dqlite: attempt 1: server 1: connected
INFO   [2024-02-13T12:19:54+02:00] Connected to global database
INFO   [2024-02-13T12:19:54+02:00] Initialized global database
INFO   [2024-02-13T12:19:54+02:00] Firewall loaded driver                        driver=nftables
INFO   [2024-02-13T12:19:54+02:00] Initializing storage pools
DEBUG  [2024-02-13T12:19:54+02:00] Initializing storage pool                     pool=lxddsk
DEBUG  [2024-02-13T12:19:54+02:00] Mount started                                 driver=btrfs pool=lxddsk
DEBUG  [2024-02-13T12:20:04+02:00] Mount finished                                driver=btrfs pool=lxddsk
ERROR  [2024-02-13T12:20:04+02:00] Failed mounting storage pool                  err="Failed to mount \"/dev/disk/by-uuid/27051cec-9c46-4759-ada6-7f14e48b17b2\" on \"/var/snap/lxd/common/lxd/storage-pools/lxddsk\" using \"btrfs\": no such file or directory" pool=lxddsk
DEBUG  [2024-02-13T12:20:04+02:00] Initializing storage pool                     pool=lxdsp-01
DEBUG  [2024-02-13T12:20:04+02:00] Mount started                                 driver=btrfs pool=lxdsp-01
DEBUG  [2024-02-13T12:20:04+02:00] Mount finished                                driver=btrfs pool=lxdsp-01
INFO   [2024-02-13T12:20:04+02:00] Initialized storage pool                      pool=lxdsp-01
INFO   [2024-02-13T12:20:04+02:00] Applying patch                                name=storage_delete_old_snapshot_records
ERROR  [2024-02-13T12:20:04+02:00] Failed to start the daemon                    err="Failed applying patch \"storage_delete_old_snapshot_records\": Unvailable storage pools: [lxddsk]"
INFO   [2024-02-13T12:20:04+02:00] Starting shutdown sequence                    signal=interrupt
DEBUG  [2024-02-13T12:20:04+02:00] Cancel ongoing or future gRPC connection attempts
INFO   [2024-02-13T12:20:04+02:00] Closing the database
INFO   [2024-02-13T12:20:04+02:00] Stop database gateway
INFO   [2024-02-13T12:20:04+02:00] Closing socket                                socket="[::]:8443" type="REST API TCP socket"
INFO   [2024-02-13T12:20:04+02:00] Closing socket                                socket=/var/snap/lxd/common/lxd/unix.socket type="REST API Unix socket"
INFO   [2024-02-13T12:20:04+02:00] Closing socket                                socket=/var/snap/lxd/common/lxd/devlxd/sock type="devlxd socket"
INFO   [2024-02-13T12:20:04+02:00] Unmounting temporary filesystems
INFO   [2024-02-13T12:20:04+02:00] Done unmounting temporary filesystems
INFO   [2024-02-13T12:20:04+02:00] Daemon stopped
Error: Failed applying patch "storage_delete_old_snapshot_records": Unvailable storage pools: [lxddsk]

How can I restore the instances into full operation, without any loss of data, metadata or configs?

Thanks

tomp · February 13, 2024, 12:19pm

So the problem here is that you have a storage pool configured that is unavailable:

ERROR  [2024-02-13T12:20:04+02:00] Failed mounting storage pool                  err="Failed to mount \"/dev/disk/by-uuid/27051cec-9c46-4759-ada6-7f14e48b17b2\" on \"/var/snap/lxd/common/lxd/storage-pools/lxddsk\" using \"btrfs\": no such file or directory" pool=lxddsk

Normally that wouldn’t cause a start up failure, but as LXD 5.0.3 has recently been released it is trying to apply a storage patch which requires all storage pools to be online.

Are you aware of why /dev/disk/by-uuid/27051cec-9c46-4759-ada6-7f14e48b17b2 isn’t available for your BTRFS pool called lxddsk?

rmodiah · February 13, 2024, 1:12pm

I believe that /dev/disk/by-uuid/27051cec-9c46-4759-ada6-7f14e48b17b2 is supposed to reference lxddsk . I created the lxddsk storage pool at the time when I was trying to acquaint myself with LXD; and I went ahead to delete it - and thought it was gone forever - when I got to ready setup the real thing. Therefore, hither to this LXD outage, this lxddsk storage pool was no longer needed, and I was actually surprised to see it being referenced again.

I guess what is therefore needed now is to make LXD forget about the lxddsk storage pool, for the system would spring up again?

masnax · February 13, 2024, 5:24pm

You should be able to get rid of lxddsk by adding a new file /var/snap/lxd/common/lxd/database/patch.global.sql

With the content

delete from storage_pools where name='lxddsk';

Then the next time LXD starts, it will apply the patch and clean up the orphaned storage pool and things should start working again.

Here’s a link to the documentation for this feature:
https://documentation.ubuntu.com/lxd/en/latest/debugging/#running-custom-queries-at-lxd-daemon-startup

rmodiah · February 13, 2024, 7:59pm

Wow… That’s it! Thanks a lot @masnax , @tomp for your guidance.

rmodiah · February 13, 2024, 11:05pm

There’s now another problem. LXD has again failed to start up after a server reboot.

Running lxc ls now returns Error: The LXD daemon doesn't appear to be started (socket path: /var/snap/lxd/common/lxd/unix.socket).

And, lxd --debug --group lxd, returns:

DEBUG  [2024-02-14T01:00:44+02:00] Connecting to a local LXD over a Unix socket
DEBUG  [2024-02-14T01:00:44+02:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0"
INFO   [2024-02-14T01:00:44+02:00] LXD is starting                               mode=normal path=/var/snap/lxd/common/lxd version=5.0.3
INFO   [2024-02-14T01:00:44+02:00] Kernel uid/gid map:
INFO   [2024-02-14T01:00:44+02:00]  - u 0 0 4294967295
INFO   [2024-02-14T01:00:44+02:00]  - g 0 0 4294967295
INFO   [2024-02-14T01:00:44+02:00] Configured LXD uid/gid map:
INFO   [2024-02-14T01:00:44+02:00]  - u 0 1000000 1000000000
INFO   [2024-02-14T01:00:44+02:00]  - g 0 1000000 1000000000
INFO   [2024-02-14T01:00:44+02:00] Kernel features:
INFO   [2024-02-14T01:00:44+02:00]  - closing multiple file descriptors efficiently: yes
INFO   [2024-02-14T01:00:44+02:00]  - netnsid-based network retrieval: yes
INFO   [2024-02-14T01:00:44+02:00]  - pidfds: yes
INFO   [2024-02-14T01:00:44+02:00]  - core scheduling: yes
INFO   [2024-02-14T01:00:44+02:00]  - uevent injection: yes
INFO   [2024-02-14T01:00:44+02:00]  - seccomp listener: yes
INFO   [2024-02-14T01:00:44+02:00]  - seccomp listener continue syscalls: yes
INFO   [2024-02-14T01:00:44+02:00]  - seccomp listener add file descriptors: yes
INFO   [2024-02-14T01:00:44+02:00]  - attach to namespaces via pidfds: yes
INFO   [2024-02-14T01:00:44+02:00]  - safe native terminal allocation : yes
INFO   [2024-02-14T01:00:44+02:00]  - unprivileged file capabilities: yes
INFO   [2024-02-14T01:00:44+02:00]  - cgroup layout: cgroup2
WARNING[2024-02-14T01:00:44+02:00]  - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored
WARNING[2024-02-14T01:00:44+02:00]  - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead
INFO   [2024-02-14T01:00:44+02:00]  - shiftfs support: yes
INFO   [2024-02-14T01:00:44+02:00]  - idmapped mounts kernel support: yes
INFO   [2024-02-14T01:00:44+02:00] Instance type operational                     driver=lxc features="map[]" type=container
ERROR  [2024-02-14T01:00:44+02:00] Unable to run feature checks during QEMU initialization: QEMU premature exit: exit status 1 (qemu-system-x86_64: -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd: Could not open '/usr/share/OVMF/OVMF_CODE.fd': No such file or directory)
WARNING[2024-02-14T01:00:44+02:00] Instance type not operational                 driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
INFO   [2024-02-14T01:00:44+02:00] Initializing local database
DEBUG  [2024-02-14T01:00:44+02:00] Refreshing local trusted certificate cache
INFO   [2024-02-14T01:00:44+02:00] Set client certificate to server certificate  fingerprint=5eaba09b34ee628564424b60a85f6569c4b0ecdf3f7818b6845b4fa5a0751671
DEBUG  [2024-02-14T01:00:44+02:00] Initializing database gateway
INFO   [2024-02-14T01:00:44+02:00] Starting database node                        id=1 local=1 role=voter
ERROR  [2024-02-14T01:00:44+02:00] Failed to start the daemon                    err="Failed to start dqlite server: raft_start(): io: closed segment 0000000000671011-0000000000671051 is past last snapshot snapshot-1-670720-39270450"
INFO   [2024-02-14T01:00:44+02:00] Starting shutdown sequence                    signal=interrupt
INFO   [2024-02-14T01:00:44+02:00] Not unmounting temporary filesystems (instances are still running)
INFO   [2024-02-14T01:00:44+02:00] Daemon stopped
Error: Failed to start dqlite server: raft_start(): io: closed segment 0000000000671011-0000000000671051 is past last snapshot snapshot-1-670720-39270450

How may this get resolved?

masnax · February 14, 2024, 12:25am

Looks like your database got a bit corrupted somehow. First I would back up everything in /var/snap/lxd/common/lxd/database and then proceed from there.

You will likely have to remove some segment files and snapshots from the global database directory.

Could you please post the output of ls -lah /var/snap/lxd/common/lxd/database/global/

tomp · February 14, 2024, 8:18am

I’ve looped in our colleagues on the dqlite team to help answer this.
I suspect the server was shutdown abruptly and this has left inconsistent raft snapshots behind.

tomp · February 14, 2024, 8:22am

Please can you show the output of:

sudo ls -lah /var/snap/lxd/common/lxd/database/global

rmodiah · February 14, 2024, 8:46pm

After posting about the LXD failure to start again after Server reboot, I actually decided to go ahead and restore

/var/snap/lxd/common/lxd

from a backup that I took earlier, after the initial stall. Thereafter I went ahead to again

delete from storage_pools where name='lxddsk';

using

/var/snap/lxd/common/lxd/database/patch.global.sql

as previously advised; and actually got LXD up and running again. So, I decided to let it run that way to minimize user disruption; until over the weekend when I can have more time to conclusively resolve the issue.

As requested, @tomp @masnax the output of ls -lah /var/snap/lxd/common/lxd/database/global/ (hoping that this could nonetheless provide useful insight, though it is taken before the server reboot that gave rise to the latter

Error: The LXD daemon doesn't appear to be started (socket path: /var/snap/lxd/common/lxd/unix.socket)

issue that I reported above), is as below. The actual output is quite large (like 8810 lines), and so I have attempted shorten what I would have had to share here as follows (I hope that it can be easily understood ):

total 96M
drwxr-x--- 2 root root 768K Feb 14 01:25 .
drwx------ 4 root root 4.0K Feb 14 01:25 ..
-rw------- 1 root root 4.1K Feb 11 12:18 0000000000659457-0000000000659457
-rw------- 1 root root 4.1K Feb 11 12:18 0000000000659458-0000000000659458
-rw------- 1 root root 4.1K Feb 11 12:18 0000000000659459-0000000000659459
-rw------- 1 root root 4.1K Feb 11 12:19 0000000000659460-0000000000659460
-rw------- 1 root root 4.1K Feb 11 12:19 0000000000659461-0000000000659461
-rw------- 1 root root 4.1K Feb 11 12:19 0000000000659462-0000000000659462
-rw------- 1 root root 4.1K Feb 11 12:19 0000000000659463-0000000000659463
-rw------- 1 root root 4.1K Feb 11 12:19 0000000000659464-0000000000659464
-rw------- 1 root root 4.1K Feb 11 12:20 0000000000659465-0000000000659465
-rw------- 1 root root 4.1K Feb 11 12:20 0000000000659466-0000000000659466
-rw------- 1 root root 4.1K Feb 11 12:20 0000000000659467-0000000000659467
-rw------- 1 root root 4.1K Feb 11 12:20 0000000000659468-0000000000659468
-rw------- 1 root root 4.1K Feb 11 12:20 0000000000659469-0000000000659469
-rw------- 1 root root 4.1K Feb 11 12:21 0000000000659470-0000000000659470
-rw------- 1 root root 4.1K Feb 11 12:21 0000000000659471-0000000000659471
-rw------- 1 root root 4.1K Feb 11 12:21 0000000000659472-0000000000659472
-rw------- 1 root root 4.1K Feb 11 12:21 0000000000659473-0000000000659473
-rw------- 1 root root 4.1K Feb 11 12:21 0000000000659474-0000000000659474
-rw------- 1 root root 4.1K Feb 11 12:22 0000000000659475-0000000000659475
-rw------- 1 root root 4.1K Feb 11 12:22 0000000000659476-0000000000659476
-rw------- 1 root root 4.1K Feb 11 12:22 0000000000659477-0000000000659477
-rw------- 1 root root 4.1K Feb 11 12:22 0000000000659478-0000000000659478
-rw------- 1 root root 4.1K Feb 11 12:22 0000000000659479-0000000000659479
-rw------- 1 root root 4.1K Feb 11 12:22 0000000000659480-0000000000659480
-rw------- 1 root root 4.1K Feb 11 12:23 0000000000659481-0000000000659481
-rw------- 1 root root 4.1K Feb 11 12:23 0000000000659482-0000000000659482
-rw------- 1 root root 4.1K Feb 11 12:23 0000000000659483-0000000000659483
-rw------- 1 root root 4.1K Feb 11 12:23 0000000000659484-0000000000659484
-rw------- 1 root root 4.1K Feb 11 12:23 0000000000659485-0000000000659485
-rw------- 1 root root 4.1K Feb 11 12:24 0000000000659486-0000000000659486
-rw------- 1 root root 4.1K Feb 11 12:24 0000000000659487-0000000000659487
-rw------- 1 root root 4.1K Feb 11 12:24 0000000000659488-0000000000659488
-rw------- 1 root root 4.1K Feb 11 12:24 0000000000659489-0000000000659489
-rw------- 1 root root 4.1K Feb 11 12:24 0000000000659490-0000000000659490
-rw------- 1 root root 4.1K Feb 11 12:25 0000000000659491-0000000000659491
-rw------- 1 root root 4.1K Feb 11 12:25 0000000000659492-0000000000659492
-rw------- 1 root root 4.1K Feb 11 12:25 0000000000659493-0000000000659493
-rw------- 1 root root 4.1K Feb 11 12:25 0000000000659494-0000000000659494
-rw------- 1 root root 4.1K Feb 11 12:25 0000000000659495-0000000000659495
-rw------- 1 root root 4.1K Feb 11 12:26 0000000000659496-0000000000659496
-rw------- 1 root root 4.1K Feb 11 12:26 0000000000659497-0000000000659497
-rw------- 1 root root 4.1K Feb 11 12:26 0000000000659498-0000000000659498
-rw------- 1 root root 4.1K Feb 11 12:26 0000000000659499-0000000000659499
-rw------- 1 root root 4.1K Feb 11 12:26 0000000000659500-0000000000659500
-rw------- 1 root root 4.1K Feb 11 12:27 0000000000659501-0000000000659501
...
-rw------- 1 root root 4.1K Feb 11 12:46 0000000000659599-0000000000659599
-rw------- 1 root root 4.1K Feb 11 12:46 0000000000659600-0000000000659600
-rw------- 1 root root 4.1K Feb 11 12:46 0000000000659601-0000000000659601
...
-rw------- 1 root root 4.1K Feb 11 13:28 0000000000659799-0000000000659799
-rw------- 1 root root 4.1K Feb 11 13:28 0000000000659800-0000000000659800
-rw------- 1 root root 4.1K Feb 11 13:28 0000000000659801-0000000000659801
...
-rw------- 1 root root 4.1K Feb 11 13:49 0000000000659899-0000000000659899
-rw------- 1 root root 4.1K Feb 11 13:49 0000000000659900-0000000000659900
-rw------- 1 root root 4.1K Feb 11 13:49 0000000000659901-0000000000659901
...
-rw------- 1 root root 4.1K Feb 11 14:10 0000000000659999-0000000000659999
-rw------- 1 root root 4.1K Feb 11 14:10 0000000000660000-0000000000660000
-rw------- 1 root root 4.1K Feb 11 14:10 0000000000660001-0000000000660001
...
-rw------- 1 root root 4.1K Feb 11 17:40 0000000000660999-0000000000660999
-rw------- 1 root root 4.1K Feb 11 17:40 0000000000661000-0000000000661000
-rw------- 1 root root 4.1K Feb 11 17:40 0000000000661001-0000000000661001
...
-rw------- 1 root root 4.1K Feb 11 21:09 0000000000661999-0000000000661999
-rw------- 1 root root 4.1K Feb 11 21:09 0000000000662000-0000000000662000
-rw------- 1 root root 4.1K Feb 11 21:09 0000000000662001-0000000000662001
...
-rw------- 1 root root 4.1K Feb 12 00:39 0000000000662999-0000000000662999
-rw------- 1 root root 4.1K Feb 12 00:39 0000000000663000-0000000000663000
-rw------- 1 root root 4.1K Feb 12 00:39 0000000000663001-0000000000663001
...
-rw------- 1 root root 4.1K Feb 12 04:08 0000000000663999-0000000000663999
-rw------- 1 root root 4.1K Feb 12 04:08 0000000000664000-0000000000664000
-rw------- 1 root root 4.1K Feb 12 04:08 0000000000664001-0000000000664001
...
-rw------- 1 root root 4.1K Feb 12 07:38 0000000000664999-0000000000664999
-rw------- 1 root root 4.1K Feb 12 07:38 0000000000665000-0000000000665000
-rw------- 1 root root 4.1K Feb 12 07:38 0000000000665001-0000000000665001
...
-rw------- 1 root root 4.1K Feb 12 11:08 0000000000665999-0000000000665999
-rw------- 1 root root 4.1K Feb 12 11:08 0000000000666000-0000000000666000
-rw------- 1 root root 4.1K Feb 12 11:08 0000000000666001-0000000000666001
...
-rw------- 1 root root 4.1K Feb 12 14:43 0000000000666999-0000000000666999
-rw------- 1 root root 4.1K Feb 12 14:43 0000000000667000-0000000000667000
-rw------- 1 root root 4.1K Feb 12 14:43 0000000000667001-0000000000667001
...
-rw------- 1 root root 4.1K Feb 12 18:31 0000000000667999-0000000000667999
-rw------- 1 root root 4.1K Feb 12 18:31 0000000000668000-0000000000668000
-rw------- 1 root root 4.1K Feb 12 18:32 0000000000668001-0000000000668001
...
-rw------- 1 root root 4.1K Feb 12 18:48 0000000000668079-0000000000668079
-rw------- 1 root root 4.1K Feb 12 18:48 0000000000668080-0000000000668080
-rw------- 1 root root 4.1K Feb 12 18:49 0000000000668081-0000000000668081
... ...
-rw------- 1 root root 4.1K Feb 12 19:19 0000000000668226-0000000000668226
-rw------- 1 root root 4.1K Feb 12 19:20 0000000000668227-0000000000668227
-rw------- 1 root root 4.1K Feb 12 19:20 0000000000668228-0000000000668228
-rw------- 1 root root 4.1K Feb 12 19:20 0000000000668229-0000000000668229
-rw------- 1 root root 4.1K Feb 12 19:20 0000000000668230-0000000000668230
-rw------- 1 root root 4.1K Feb 12 19:21 0000000000668231-0000000000668231
-rw------- 1 root root 4.1K Feb 12 19:21 0000000000668232-0000000000668232
-rw------- 1 root root 4.1K Feb 12 19:21 0000000000668233-0000000000668233
-rw------- 1 root root 4.1K Feb 12 19:21 0000000000668234-0000000000668234
-rw------- 1 root root 4.1K Feb 12 19:21 0000000000668235-0000000000668235
-rw------- 1 root root 4.1K Feb 12 19:22 0000000000668236-0000000000668236
-rw------- 1 root root 4.1K Feb 12 19:22 0000000000668237-0000000000668237
-rw------- 1 root root 4.1K Feb 12 19:22 0000000000668238-0000000000668238
-rw------- 1 root root 4.1K Feb 12 19:22 0000000000668239-0000000000668239
-rw------- 1 root root 4.1K Feb 12 19:22 0000000000668240-0000000000668240
-rw------- 1 root root 4.1K Feb 12 19:23 0000000000668241-0000000000668241
-rw------- 1 root root 4.1K Feb 12 19:23 0000000000668242-0000000000668242
-rw------- 1 root root 4.1K Feb 12 19:23 0000000000668243-0000000000668243
-rw------- 1 root root 4.1K Feb 12 19:23 0000000000668244-0000000000668244
-rw------- 1 root root 4.1K Feb 12 19:23 0000000000668245-0000000000668245
-rw------- 1 root root 4.1K Feb 12 19:24 0000000000668246-0000000000668246
-rw------- 1 root root 4.1K Feb 12 19:24 0000000000668247-0000000000668247
-rw------- 1 root root 4.1K Feb 12 19:24 0000000000668248-0000000000668248
-rw------- 1 root root 4.1K Feb 12 19:24 0000000000668249-0000000000668249
-rw------- 1 root root 4.1K Feb 12 19:25 0000000000668250-0000000000668250
-rw------- 1 root root 4.1K Feb 12 19:25 0000000000668251-0000000000668251
-rw------- 1 root root 4.1K Feb 12 19:25 0000000000668252-0000000000668252
-rw------- 1 root root 4.1K Feb 12 19:25 0000000000668253-0000000000668253
-rw------- 1 root root 4.1K Feb 12 19:25 0000000000668254-0000000000668254
-rw------- 1 root root 4.1K Feb 12 19:26 0000000000668255-0000000000668255
-rw------- 1 root root 4.1K Feb 12 19:26 0000000000668256-0000000000668256
-rw------- 1 root root 4.1K Feb 12 19:26 0000000000668257-0000000000668257
-rw------- 1 root root 4.1K Feb 12 19:26 0000000000668258-0000000000668258
-rw------- 1 root root 4.1K Feb 12 19:26 0000000000668259-0000000000668259
-rw------- 1 root root 4.1K Feb 12 19:27 0000000000668260-0000000000668260
-rw------- 1 root root 4.1K Feb 12 19:27 0000000000668261-0000000000668261
-rw------- 1 root root 4.1K Feb 12 19:27 0000000000668262-0000000000668262
-rw------- 1 root root 4.1K Feb 12 19:27 0000000000668263-0000000000668263
-rw------- 1 root root 4.1K Feb 12 19:28 0000000000668264-0000000000668264
-rw------- 1 root root 4.1K Feb 12 19:28 0000000000668265-0000000000668265
-rw------- 1 root root 4.1K Feb 12 19:28 0000000000668266-0000000000668266
-rw------- 1 root root 4.1K Feb 12 19:28 0000000000668267-0000000000668267
-rw------- 1 root root 4.1K Feb 12 19:28 0000000000668268-0000000000668268
-rw------- 1 root root 4.1K Feb 12 19:29 0000000000668269-0000000000668269
-rw------- 1 root root 4.1K Feb 12 19:29 0000000000668270-0000000000668270
-rw------- 1 root root 4.1K Feb 12 19:29 0000000000668271-0000000000668271
-rw------- 1 root root 4.1K Feb 12 19:29 0000000000668272-0000000000668272
-rw------- 1 root root 4.1K Feb 12 19:29 0000000000668273-0000000000668273
-rw------- 1 root root 552K Feb 12 19:29 db.bin
-rw------- 1 root root 1.7M Feb 12 19:29 db.bin-wal
-rw------- 1 root root   32 Sep 30  2022 metadata1
-rw------- 1 root root 8.0M Feb 14 17:25 open-1
-rw------- 1 root root 8.0M Feb 14 01:25 open-2
-rw------- 1 root root 8.0M Feb 14 01:25 open-3
-rw------- 1 root root  86K Feb 12 13:26 snapshot-1-666624-6004746
-rw------- 1 root root   56 Feb 12 13:26 snapshot-1-666624-6004746.meta
-rw------- 1 root root  86K Feb 12 17:10 snapshot-1-667648-1265697
-rw------- 1 root root   56 Feb 12 17:10 snapshot-1-667648-1265697.meta

whereby, the
...
gap represents contiguous files from

-rw------- 1 root root 4.1K Feb dd HH:MM 00000000006NNN02-00000000006NNN02
to
-rw------- 1 root root 4.1K Feb dd HH:MM 00000000006NNN98-00000000006NNN98

as fitting into the names sequence; whereas the
... ...
gap represents contiguous files from

-rw------- 1 root root 4.1K Feb 12 18:49 0000000000668082-0000000000668082
to
-rw------- 1 root root 4.1K Feb 12 19:19 0000000000668225-0000000000668225
also as fitting into the sequence respectively.

Thanks.

colemiller · February 14, 2024, 9:50pm

dqlite team member here—thanks for the information!

The first thing I notice is that there are quite a lot of closed segment files that only contain a single entry (these are the files where the start and end of the range in the filename are the same). That is not incorrect per se but it’s definitely pathological. I will try to figure out what could cause dqlite to create so many segments of this type.

With the list of files that you posted, you indeed won’t get the error closed segment ... is past last snapshot that you reported (as expected since your server is now running smoothly). If you get that error again, please report it here with the corresponding list of files and I’ll advise on how you can fix it.

rmodiah · February 19, 2024, 8:39am

I did perform the promised Server reboot like 36 hours ago. And, the reboot was done after performing manual APT system updates. After the mentioned Server reboot, LXD fired up without any issues; and has been running smoothly ever since. Just wondering, did the issue get resolved through the updates?

tomp · February 19, 2024, 9:20am

There’s been no updates to the LXD snap in this time.
Glad to hear it working now

philip-21 · February 27, 2024, 8:39am

I tried these steps but still keep getting the same issue
Error: Failed to connect to local LXD: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied

The patch.global.sql file has been created with the queries to be inputed, I re-ran the comand
lxd init --minimal , but i got the same error.

tomp · February 27, 2024, 9:03am

What is your error and what is the contents of the patch.global.sql file you’re using (as each one will likely need to be different).

philip-21 · February 28, 2024, 7:30am

Here is the error message
Error: Failed to connect to local LXD: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied

Here is the SQL content

delete from storage_pools where name='lxddsk';

tomp · February 28, 2024, 9:14am

And what is the error you’re seeing in /var/snap/lxd/common/lxd/logs/lxd.log?

It seems likely if you’re being affected by the same issue, then your storage pool is likely named something different.

philip-21 · February 28, 2024, 5:55pm

These are the logs in the /var/snap/lxd/common/lxd/logs/logs.txt

time="2024-02-28T18:48:17+01:00" level=warning msg="AppArmor support has been disabled because of lack of kernel support" time="2024-02-28T18:48:17+01:00" level=warning msg=" - AppArmor support has been disabled, Disabled because of lack of kernel support" time="2024-02-28T18:48:17+01:00" level=warning msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored" time="2024-02-28T18:48:17+01:00" level=warning msg="Instance type not operational" driver=qemu err="vhost_vsock kernel module not loaded" type=virtual-machine time="2024-02-28T18:48:17+01:00" level=error msg="Failed to start the daemon" err="Failed to initialize global database: failed to ensure schema: failed to execute

tomp · February 29, 2024, 9:29am

Looks like your logs got cut off before the actual error - is it possible for you to get the whole logs, maybe look also in journalctl?

endrii · April 15, 2024, 8:42pm

i do this, but its still not help me.

lxc list 
Error: LXD unix socket "/var/snap/lxd/common/lxd-user/unix.socket" not accessible: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd-user/unix.socket: connect: connection refused

lxc version
Client version: 5.21.1 LTS
Server version: unreachable