Missing disk metrics in the Grafana LXD dashboard

LXD Support
1

I setup the LXD monitoring dashboard, according to the instructions here.

I have noticed that the metrics related to disk are not available (lxd_disk_*) and as a result some plots remain empty in the dashboard (Disk reads, Disk writes, Disk R/W Data, Disk I/O Data).
Directly requesting metrics from the endpoint

I’m using lxd 5.15 and ZFS as storage driver.

Here the list of the metrics I get:

lxd_cpu_seconds_total
lxd_filesystem_avail_bytes
lxd_filesystem_free_bytes
lxd_filesystem_size_bytes
lxd_memory_Active_anon_bytes
lxd_memory_Active_bytes
lxd_memory_Active_file_bytes
lxd_memory_Cached_bytes
lxd_memory_Dirty_bytes
lxd_memory_Inactive_anon_bytes
lxd_memory_Inactive_bytes
lxd_memory_Inactive_file_bytes
lxd_memory_Mapped_bytes
lxd_memory_MemAvailable_bytes
lxd_memory_MemFree_bytes
lxd_memory_MemTotal_bytes
lxd_memory_Shmem_bytes
lxd_memory_Swap_bytes
lxd_memory_Unevictable_bytes
lxd_memory_Writeback_bytes
lxd_network_receive_bytes_total
lxd_network_receive_drop_total
lxd_network_receive_errs_total
lxd_network_receive_packets_total
lxd_network_transmit_bytes_total
lxd_network_transmit_drop_total
lxd_network_transmit_errs_total
lxd_network_transmit_packets_total
lxd_procs_total

2

I have just tested this with an empty instance of LXD. If you don’t have any disk related workload, it doesn’t show the corresponding metrics.
Do you have a running instance for which you are missing the metrics?

3

Ok… In my case, I’m testing uploading files to a container that hosts a web application.

In the dashboard I can see the network traffic in the proxy (a container itself) and the target container but no disk activity:

image
image1254×738 34 KB

Maybe because the target disk is actually a custom volume?

4

I was testing with a VM and not a container but can confirm that the lxd_disk_ metrics do not show up if you have a running container. This seems odd.

5

@pmarini yeah, unfortunately ZFS IO stats have been broken for a while. This is due to a kernel/cgroup/zfs issue, not something LXD can do anything about.

I remember it used to work, then broken then started working again with the 5.19 kernel but eventually broke and remained broken ever since, even on 6.2 kernel :confused:

6

Thanks, do you know if there is a kernel issue tracking this?

7

I’m not aware of any existing tracking issue. Maybe @amikhalitsyn would know more about the source of the issue. So Aleksandr, here I have a container running off of a zpool and making tons of read IOs:

lxc exec c1 -- find / -type f -exec cat {} + > /dev/null

But there are no read bytes in the metrics output:

$ lxc query /1.0/metrics | grep -E '^lxd_disk_(read|written)_bytes_total' | grep -F 'name="c1"' 
lxd_disk_read_bytes_total{device="dm-0",name="c1",project="default",type="container"} 0
lxd_disk_read_bytes_total{device="nvme0n1",name="c1",project="default",type="container"} 0
lxd_disk_written_bytes_total{device="dm-0",name="c1",project="default",type="container"} 335872
lxd_disk_written_bytes_total{device="nvme0n1",name="c1",project="default",type="container"} 0

And the dm-0 part above seems totally unrelated to the container’s activity. The dm-0 device is the cryptsetup/luks backing my rootfs which is not where the zpool reside (it is on a dedicated NVME partition).

Here’s the container’s IO stats:

$ cat /sys/fs/cgroup/lxc.payload.c1/io.stat 
259:0 rbytes=0 wbytes=0 rios=0 wios=2 dbytes=0 dios=0
253:0 rbytes=0 wbytes=335872 rios=0 wios=6 dbytes=0 dios=0

$ lsblk | grep -E '25[39]:0'
nvme0n1                 259:0    0 931.5G  0 disk  
│ └─nvme0n1p3_crypt     253:0    0  38.1G  0 crypt /var/snap/firefox/common/host-hunspell

So clearly, the wrong bits end up in the cgroup file. Do you know if ZFS is a special case here or if something else can explain the absence of IO stats?

8

I tested with a container on a btrfs pool backed by a dedicated NVME partition (nvme0n1p7) and it’s not better:

$ lxc query /1.0/metrics | grep -E '^lxd_disk_(read|written)_bytes_total' | grep -F 'name="c1"' 
lxd_disk_read_bytes_total{device="dm-0",name="c1",project="default",type="container"} 0
lxd_disk_read_bytes_total{device="nvme0n1",name="c1",project="default",type="container"} 3.002368e+06
lxd_disk_written_bytes_total{device="dm-0",name="c1",project="default",type="container"} 434176
lxd_disk_written_bytes_total{device="nvme0n1",name="c1",project="default",type="container"} 1.656201216e+09

It seems the whole NVME disk stats (nvme0n1) are not including the IOs happening on that nvme0n1p7 partition.

9

I’m trying to reproduce the same test case in a 5.0.2 installation and I don’t get any _disk_ related metrics in the output of the query.

Moreover from what I understand above disk metrics are not reported not just for ZFS but also for btrfs? Is there a storage backend for which they work? I’d like to see how the metrics are reported when using custom volumes.

10

@sdeziel1 so, you have a reproducer with btrfs, right? Couldn’t you also show lsblk for your setup with btrfs, lxc exec c1 -- cat /proc/1/mountinfo and the whole io.stat file output.
I’ve tested on my local system with device mapper on top of normal (GPT) partition and it accounts IO properly.

I can say that btrfs and ZFS can be a problem for IO accounting because both of them are using a virtual devices to represent partitions (zvols) to mount them later.

But of course, if we have reproducer then we need to at least debug this issues (and probably fix).

11 
$ lsblk
NAME                    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0                     7:0    0     4K  1 loop  /snap/bare/5
...                   # multiple snap loopX
zd0                     230:0    0    10G  0 disk  
zd16                    230:16   0    10G  0 disk  
zd32                    230:32   0    10G  0 disk  
zd48                    230:48   0     5G  0 disk  
zd64                    230:64   0     5G  0 disk  
zd80                    230:80   0     5G  0 disk  
nvme0n1                 259:0    0 931.5G  0 disk  
├─nvme0n1p1             259:1    0   487M  0 part  /boot/efi
├─nvme0n1p2             259:2    0   977M  0 part  /boot
├─nvme0n1p3             259:3    0  38.1G  0 part  
│ └─nvme0n1p3_crypt     253:0    0  38.1G  0 crypt /var/snap/firefox/common/host-hunspell
│                                                  /home
│                                                  /
├─nvme0n1p4             259:4    0     2G  0 part  
│ └─nvme0n1p4_cryptswap 253:1    0     2G  0 crypt [SWAP]
├─nvme0n1p5             259:5    0   160G  0 part  
├─nvme0n1p6             259:6    0   180G  0 part    # the part used by the zfs pool
└─nvme0n1p7             259:7    0    32G  0 part     # the part used by the btrfs pool

The info from c1:

$ lxc exec c1 -- cat /proc/1/mountinfo
3993 2316 0:83 /containers/c1/rootfs / rw,relatime,idmapped shared:435 master:1065 - btrfs /dev/nvme0n1p7 rw,ssd,discard=async,space_cache=v2,user_subvol_rm_allowed,subvolid=263,subvol=/containers/c1
3994 3993 0:114 / /dev rw,relatime shared:469 - tmpfs none rw,size=492k,mode=755,uid=1000000,gid=1000000,inode64
3995 3993 0:115 / /proc rw,nosuid,nodev,noexec,relatime shared:643 - proc proc rw
3996 3993 0:116 / /sys rw,relatime shared:698 - sysfs sysfs rw
3997 3994 0:5 /fuse /dev/fuse rw,nosuid,relatime shared:481 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
3998 3994 0:5 /net/tun /dev/net/tun rw,nosuid,relatime shared:545 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
3999 3995 0:21 /sys/fs/binfmt_misc /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:649 master:13 - proc proc rw,gid=999,hidepid=invisible
4000 3996 0:32 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:699 master:11 - efivarfs efivarfs rw
4001 3996 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:700 master:25 - fusectl fusectl rw
4002 3996 0:31 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:701 master:10 - pstore pstore rw
4003 3996 0:19 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:702 master:39 - configfs configfs rw
4004 3996 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:703 master:8 - securityfs securityfs rw
4005 3996 0:20 /kernel/tracing /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:704 master:7 - sysfs sysfs rw
4006 3994 0:18 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:557 master:15 - mqueue mqueue rw
4007 3994 0:85 / /dev/lxd rw,relatime shared:600 - tmpfs tmpfs rw,size=100k,mode=755,inode64
4008 3994 0:82 /c1 /dev/.lxd-mounts rw,relatime master:922 - tmpfs tmpfs rw,size=100k,mode=711,inode64
4009 3996 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:705 - cgroup2 none rw
4010 3995 0:80 /proc/cpuinfo /proc/cpuinfo rw,nosuid,nodev,relatime shared:690 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
4011 3995 0:80 /proc/diskstats /proc/diskstats rw,nosuid,nodev,relatime shared:691 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
4012 3995 0:80 /proc/loadavg /proc/loadavg rw,nosuid,nodev,relatime shared:692 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
4013 3995 0:80 /proc/meminfo /proc/meminfo rw,nosuid,nodev,relatime shared:693 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
4014 3995 0:80 /proc/slabinfo /proc/slabinfo rw,nosuid,nodev,relatime shared:694 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
4015 3995 0:80 /proc/stat /proc/stat rw,nosuid,nodev,relatime shared:695 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
4016 3995 0:80 /proc/swaps /proc/swaps rw,nosuid,nodev,relatime shared:696 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
4017 3995 0:80 /proc/uptime /proc/uptime rw,nosuid,nodev,relatime shared:697 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
4018 3996 0:80 /sys/devices/system/cpu /sys/devices/system/cpu rw,nosuid,nodev,relatime shared:706 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
4019 3994 0:5 /full /dev/full rw,nosuid,relatime shared:634 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
4020 3994 0:5 /null /dev/null rw,nosuid,relatime shared:635 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
4021 3994 0:5 /random /dev/random rw,nosuid,relatime shared:636 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
4022 3994 0:5 /tty /dev/tty rw,nosuid,relatime shared:637 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
4023 3994 0:5 /urandom /dev/urandom rw,nosuid,relatime shared:638 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
4024 3994 0:5 /zero /dev/zero rw,nosuid,relatime shared:639 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
4025 3994 0:117 / /dev/pts rw,nosuid,noexec,relatime shared:640 - devpts devpts rw,gid=1000005,mode=620,ptmxmode=666,max=1024
4045 3994 0:117 /ptmx /dev/ptmx rw,nosuid,noexec,relatime shared:641 - devpts devpts rw,gid=1000005,mode=620,ptmxmode=666,max=1024
4048 3994 0:117 /0 /dev/console rw,nosuid,noexec,relatime shared:642 - devpts devpts rw,gid=1000005,mode=620,ptmxmode=666,max=1024
4049 3995 0:114 /.lxc-boot-id /proc/sys/kernel/random/boot_id ro,nosuid,nodev,noexec,relatime shared:469 - tmpfs none rw,size=492k,mode=755,uid=1000000,gid=1000000,inode64
930 3994 0:118 / /dev/shm rw,nosuid,nodev shared:612 - tmpfs tmpfs rw,uid=1000000,gid=1000000,inode64
1026 3993 0:119 / /run rw,nosuid,nodev shared:707 - tmpfs tmpfs rw,size=6328600k,nr_inodes=819200,mode=755,uid=1000000,gid=1000000,inode64
1130 1026 0:120 / /run/lock rw,nosuid,nodev,noexec,relatime shared:708 - tmpfs tmpfs rw,size=5120k,uid=1000000,gid=1000000,inode64
5787 1026 0:123 / /run/user/0 rw,nosuid,nodev,relatime shared:1016 - tmpfs tmpfs rw,size=3164296k,nr_inodes=791074,mode=700,uid=1000000,gid=1000000,inode64

c1's io.stat:

$ cat /sys/fs/cgroup/lxc.payload.c1/io.stat 
259:0 rbytes=66088960 wbytes=8970240 rios=543 wios=151 dbytes=0 dios=0
7:2 rbytes=3072 wbytes=0 rios=1 wios=0 dbytes=0 dios=0

LXD’s disk metrics for c1:

$ lxc query /1.0/metrics | grep -E '^lxd_disk_(read|written)_bytes_total' | grep -F 'name="c1"' 
lxd_disk_read_bytes_total{device="nvme0n1",name="c1",project="default",type="container"} 6.608896e+07
lxd_disk_read_bytes_total{device="loop2",name="c1",project="default",type="container"} 3072
lxd_disk_written_bytes_total{device="nvme0n1",name="c1",project="default",type="container"} 9.0112e+06
lxd_disk_written_bytes_total{device="loop2",name="c1",project="default",type="container"} 0
12

Hm, as I can see IO has been accounted for nvme0n1 (259:0). So it looks like everything is fine at least with the btrfs variant. Or not?

13

So yes, the btrfs variant seems to be working fine as IOs for the main NVME drive are accounted for.

Now here with c2 that’s backed by ZFS:

$ lxc exec c2 -- cat /proc/self/mountinfo
6846 6316 0:125 /rootfs / rw,noatime shared:976 master:975 - zfs default/containers/c2 rw,xattr,posixacl
6847 6846 0:127 / /dev rw,relatime shared:977 - tmpfs none rw,size=492k,mode=755,uid=1000000,gid=1000000,inode64
6848 6846 0:128 / /proc rw,nosuid,nodev,noexec,relatime shared:992 - proc proc rw
6849 6846 0:129 / /sys rw,relatime shared:1002 - sysfs sysfs rw
6850 6847 0:5 /fuse /dev/fuse rw,nosuid,relatime shared:978 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
6851 6847 0:5 /net/tun /dev/net/tun rw,nosuid,relatime shared:979 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
6852 6848 0:21 /sys/fs/binfmt_misc /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:993 master:13 - proc proc rw,gid=999,hidepid=invisible
6853 6849 0:32 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:1003 master:11 - efivarfs efivarfs rw
6854 6849 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:1004 master:25 - fusectl fusectl rw
6855 6849 0:31 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:1005 master:10 - pstore pstore rw
6856 6849 0:19 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:1006 master:39 - configfs configfs rw
6857 6849 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:1014 master:8 - securityfs securityfs rw
6858 6849 0:20 /kernel/tracing /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:1015 master:7 - sysfs sysfs rw
6859 6847 0:18 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:980 master:15 - mqueue mqueue rw
6860 6847 0:85 / /dev/lxd rw,relatime shared:981 - tmpfs tmpfs rw,size=100k,mode=755,inode64
6861 6847 0:82 /c2 /dev/.lxd-mounts rw,relatime master:922 - tmpfs tmpfs rw,size=100k,mode=711,inode64
6862 6849 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:1016 - cgroup2 none rw
6863 6848 0:80 /proc/cpuinfo /proc/cpuinfo rw,nosuid,nodev,relatime shared:994 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
6864 6848 0:80 /proc/diskstats /proc/diskstats rw,nosuid,nodev,relatime shared:995 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
6865 6848 0:80 /proc/loadavg /proc/loadavg rw,nosuid,nodev,relatime shared:996 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
6866 6848 0:80 /proc/meminfo /proc/meminfo rw,nosuid,nodev,relatime shared:997 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
6867 6848 0:80 /proc/slabinfo /proc/slabinfo rw,nosuid,nodev,relatime shared:998 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
6868 6848 0:80 /proc/stat /proc/stat rw,nosuid,nodev,relatime shared:999 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
6869 6848 0:80 /proc/swaps /proc/swaps rw,nosuid,nodev,relatime shared:1000 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
6870 6848 0:80 /proc/uptime /proc/uptime rw,nosuid,nodev,relatime shared:1001 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
6871 6849 0:80 /sys/devices/system/cpu /sys/devices/system/cpu rw,nosuid,nodev,relatime shared:1042 master:1063 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
6872 6847 0:5 /full /dev/full rw,nosuid,relatime shared:983 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
6873 6847 0:5 /null /dev/null rw,nosuid,relatime shared:984 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
6874 6847 0:5 /random /dev/random rw,nosuid,relatime shared:985 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
6875 6847 0:5 /tty /dev/tty rw,nosuid,relatime shared:986 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
6876 6847 0:5 /urandom /dev/urandom rw,nosuid,relatime shared:987 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
6877 6847 0:5 /zero /dev/zero rw,nosuid,relatime shared:988 master:2 - devtmpfs udev rw,size=15786868k,nr_inodes=3946717,mode=755,inode64
6878 6847 0:130 / /dev/pts rw,nosuid,noexec,relatime shared:989 - devpts devpts rw,gid=1000005,mode=620,ptmxmode=666,max=1024
6879 6847 0:130 /ptmx /dev/ptmx rw,nosuid,noexec,relatime shared:990 - devpts devpts rw,gid=1000005,mode=620,ptmxmode=666,max=1024
6880 6847 0:130 /0 /dev/console rw,nosuid,noexec,relatime shared:991 - devpts devpts rw,gid=1000005,mode=620,ptmxmode=666,max=1024
6881 6848 0:127 /.lxc-boot-id /proc/sys/kernel/random/boot_id ro,nosuid,nodev,noexec,relatime shared:977 - tmpfs none rw,size=492k,mode=755,uid=1000000,gid=1000000,inode64
930 6847 0:131 / /dev/shm rw,nosuid,nodev shared:982 - tmpfs tmpfs rw,uid=1000000,gid=1000000,inode64
1026 6846 0:132 / /run rw,nosuid,nodev shared:1054 - tmpfs tmpfs rw,size=6328600k,nr_inodes=819200,mode=755,uid=1000000,gid=1000000,inode64
1130 1026 0:133 / /run/lock rw,nosuid,nodev,noexec,relatime shared:1064 - tmpfs tmpfs rw,size=5120k,uid=1000000,gid=1000000,inode64

c2's (empty) io.stat:

$ cat /sys/fs/cgroup/lxc.payload.c2/io.stat
$ wc -l /sys/fs/cgroup/lxc.payload.c2/io.stat
0 /sys/fs/cgroup/lxc.payload.c2/io.stat

And the host’s lsblk output:

$ lsblk
NAME                    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0                     7:0    0     4K  1 loop  /snap/bare/5
...   # many loopX devices omitted for brevity
zd0                     230:0    0    10G  0 disk  
zd16                    230:16   0    10G  0 disk  
zd32                    230:32   0    10G  0 disk  
zd48                    230:48   0     5G  0 disk  
zd64                    230:64   0     5G  0 disk  
zd80                    230:80   0     5G  0 disk  
zd96                    230:96   0    10G  0 disk  
nvme0n1                 259:0    0 931.5G  0 disk  
├─nvme0n1p1             259:1    0   487M  0 part  /boot/efi
├─nvme0n1p2             259:2    0   977M  0 part  /boot
├─nvme0n1p3             259:3    0  38.1G  0 part  
│ └─nvme0n1p3_crypt     253:0    0  38.1G  0 crypt /var/snap/firefox/common/host-hunspell
│                                                  /home
│                                                  /
├─nvme0n1p4             259:4    0     2G  0 part  
│ └─nvme0n1p4_cryptswap 253:1    0     2G  0 crypt [SWAP]
├─nvme0n1p5             259:5    0   160G  0 part  
├─nvme0n1p6             259:6    0   180G  0 part  
└─nvme0n1p7             259:7    0    32G  0 part