Missing metrics for disk reads and writes

LXD Support
1

Have no metrics (or only 0) from disk reads and writes. LXD 5.21.1 LTS
use a profile

name: bridgel2k
description: bridge to br0 of host
config: {}
devices:
  eth0:
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: lxd-pool
    type: disk
used_by:
- /1.0/instances/lxd-dashboard
- /1.0/instances/ascv00067
- /1.0/instances/ascv00069

storage info

info:
  description: ""
  driver: zfs
  name: lxd-pool
  space used: 4.73GiB
  total space: 96.41GiB
used by:
  images:
  - 7ee17012a04afa3d8c9a30b9169c4a607f9e9ed420306ece578a5c6b0e657204
  - fbc54fa1f9cbdec452a99891feca3aec4c33200aa76e4eaec0e87263f7577e9f
  instances:
  - ascv00067
  - ascv00069
  - lxd-dashboard
  profiles:
  - bridgel2k
  - default

image
image1536Ă—313 63.3 KB

# HELP lxd_disk_read_bytes_total The total number of bytes read.
# TYPE lxd_disk_read_bytes_total counter
lxd_disk_read_bytes_total{device="loop7",name="ascv00069",project="default",type="container"} 0
lxd_disk_read_bytes_total{device="loop4",name="ascv00067",project="default",type="container"} 0
# HELP lxd_disk_reads_completed_total The total number of completed reads.
# TYPE lxd_disk_reads_completed_total counter
lxd_disk_reads_completed_total{device="loop7",name="ascv00069",project="default",type="container"} 0
lxd_disk_reads_completed_total{device="loop4",name="ascv00067",project="default",type="container"} 0
# HELP lxd_disk_written_bytes_total The total number of bytes written.
# TYPE lxd_disk_written_bytes_total counter
lxd_disk_written_bytes_total{device="loop7",name="ascv00069",project="default",type="container"} 0
lxd_disk_written_bytes_total{device="loop4",name="ascv00067",project="default",type="container"} 0
# HELP lxd_disk_writes_completed_total The total number of completed writes.
# TYPE lxd_disk_writes_completed_total counter
lxd_disk_writes_completed_total{device="loop7",name="ascv00069",project="default",type="container"} 0
lxd_disk_writes_completed_total{device="loop4",name="ascv00067",project="default",type="container"} 0
# HELP lxd_filesystem_avail_bytes The number of available space in bytes.
# TYPE lxd_filesystem_avail_bytes gauge
lxd_filesystem_avail_bytes{device="lxdpool/containers/ascv00069",fstype="zfs",mountpoint="/",name="ascv00069",project="default",type="container"} 9.8433236992e+10
lxd_filesystem_avail_bytes{device="lxdpool/containers/ascv00067",fstype="zfs",mountpoint="/",name="ascv00067",project="default",type="container"} 9.8433236992e+10
lxd_filesystem_avail_bytes{device="lxdpool/containers/lxd-dashboard",fstype="zfs",mountpoint="/",name="lxd-dashboard",project="default",type="container"} 9.8433236992e+10
# HELP lxd_filesystem_free_bytes The number of free space in bytes.
# TYPE lxd_filesystem_free_bytes gauge
lxd_filesystem_free_bytes{device="lxdpool/containers/ascv00069",fstype="zfs",mountpoint="/",name="ascv00069",project="default",type="container"} 9.8433236992e+10
lxd_filesystem_free_bytes{device="lxdpool/containers/ascv00067",fstype="zfs",mountpoint="/",name="ascv00067",project="default",type="container"} 9.8433236992e+10
lxd_filesystem_free_bytes{device="lxdpool/containers/lxd-dashboard",fstype="zfs",mountpoint="/",name="lxd-dashboard",project="default",type="container"} 9.8433236992e+10
# HELP lxd_filesystem_size_bytes The size of the filesystem in bytes.
# TYPE lxd_filesystem_size_bytes gauge
lxd_filesystem_size_bytes{device="lxdpool/containers/ascv00069",fstype="zfs",mountpoint="/",name="ascv00069",project="default",type="container"} 1.00210704384e+11
lxd_filesystem_size_bytes{device="lxdpool/containers/ascv00067",fstype="zfs",mountpoint="/",name="ascv00067",project="default",type="container"} 1.00203364352e+11
lxd_filesystem_size_bytes{device="lxdpool/containers/lxd-dashboard",fstype="zfs",mountpoint="/",name="lxd-dashboard",project="default",type="container"} 9.9265019904e+10

any idea?

2

@sdeziel1 is this expected currently?

3

Just started using LXD metrics on Grafana. I believe this is because I’m utilizing a ZFS pool for storage!? instead of “loop” device

4

I believe there is an issue with ZFS blkio accounting currently.

@sdeziel1 is there a Github issue for this btw?

Other storage drivers should work fine.

5

I agree. Just added a new container with the pool driver seted to “dir”, and that works, instead of using ZFS. ZFS pool still isn’t working

image
image1537Ă—299 64.8 KB

lxd_disk_read_bytes_total{device="loop4",name="ascv00067",project="default",type="container"} 0
lxd_disk_read_bytes_total{device="loop7",name="ascv00069",project="default",type="container"} 0
lxd_disk_read_bytes_total{device="vda",name="ascv00070",project="default",type="container"} 5.046272e+06
# HELP lxd_disk_reads_completed_total The total number of completed reads.
# TYPE lxd_disk_reads_completed_total counter
lxd_disk_reads_completed_total{device="loop4",name="ascv00067",project="default",type="container"} 0
lxd_disk_reads_completed_total{device="loop7",name="ascv00069",project="default",type="container"} 0
lxd_disk_reads_completed_total{device="vda",name="ascv00070",project="default",type="container"} 179
# HELP lxd_disk_written_bytes_total The total number of bytes written.
# TYPE lxd_disk_written_bytes_total counter
lxd_disk_written_bytes_total{device="loop4",name="ascv00067",project="default",type="container"} 0
lxd_disk_written_bytes_total{device="loop7",name="ascv00069",project="default",type="container"} 0
lxd_disk_written_bytes_total{device="vda",name="ascv00070",project="default",type="container"} 1.449984e+06
# HELP lxd_disk_writes_completed_total The total number of completed writes.
# TYPE lxd_disk_writes_completed_total counter
lxd_disk_writes_completed_total{device="loop4",name="ascv00067",project="default",type="container"} 0
lxd_disk_writes_completed_total{device="loop7",name="ascv00069",project="default",type="container"} 0
lxd_disk_writes_completed_total{device="vda",name="ascv00070",project="default",type="container"} 117
# HELP lxd_filesystem_avail_bytes The number of available space in bytes.
# TYPE lxd_filesystem_avail_bytes gauge
lxd_filesystem_avail_bytes{device="lxdpool/containers/ascv00067",fstype="zfs",mountpoint="/",name="ascv00067",project="default",type="container"} 9.8429698048e+10
lxd_filesystem_avail_bytes{device="lxdpool/containers/ascv00069",fstype="zfs",mountpoint="/",name="ascv00069",project="default",type="container"} 9.8429698048e+10
lxd_filesystem_avail_bytes{device="lxdpool/containers/lxd-dashboard",fstype="zfs",mountpoint="/",name="lxd-dashboard",project="default",type="container"} 9.8429698048e+10
lxd_filesystem_avail_bytes{device="",fstype="ext4",mountpoint="/",name="ascv00070",project="default",type="container"} 1.401720832e+10
# HELP lxd_filesystem_free_bytes The number of free space in bytes.
# TYPE lxd_filesystem_free_bytes gauge
lxd_filesystem_free_bytes{device="lxdpool/containers/ascv00067",fstype="zfs",mountpoint="/",name="ascv00067",project="default",type="container"} 9.8429698048e+10
lxd_filesystem_free_bytes{device="lxdpool/containers/ascv00069",fstype="zfs",mountpoint="/",name="ascv00069",project="default",type="container"} 9.8429698048e+10
lxd_filesystem_free_bytes{device="lxdpool/containers/lxd-dashboard",fstype="zfs",mountpoint="/",name="lxd-dashboard",project="default",type="container"} 9.8429698048e+10
lxd_filesystem_free_bytes{device="",fstype="ext4",mountpoint="/",name="ascv00070",project="default",type="container"} 1.510772736e+10
# HELP lxd_filesystem_size_bytes The size of the filesystem in bytes.
# TYPE lxd_filesystem_size_bytes gauge
lxd_filesystem_size_bytes{device="lxdpool/containers/ascv00067",fstype="zfs",mountpoint="/",name="ascv00067",project="default",type="container"} 1.00201791488e+11
lxd_filesystem_size_bytes{device="lxdpool/containers/ascv00069",fstype="zfs",mountpoint="/",name="ascv00069",project="default",type="container"} 1.00207296512e+11
lxd_filesystem_size_bytes{device="lxdpool/containers/lxd-dashboard",fstype="zfs",mountpoint="/",name="lxd-dashboard",project="default",type="container"} 9.926148096e+10
lxd_filesystem_size_bytes{device="",fstype="ext4",mountpoint="/",name="ascv00070",project="default",type="container"} 2.0957446144e+10
6

@brinkib52 ZFS I/O metrics are indeed not working, apparently due to how ZFS integrates with the kernel.

@tomp It seems it was only discussed in forum posts, some of them with @amikhalitsyn so I put a reference to those I could find in this new GH bug: https://github.com/canonical/lxd/issues/13314

1 Like