Been trying to figure out why my LXD Grafana dashboard is not displaying any data in the “Loki logs” section. Both panels display: “No data”
I verified that my LXD server logs are getting into Loki via some queries in the “Explore” menu item in Grafana. Also installed logcli and verified that way as well, tailing the logs via:
./logcli query -t ‘{app=“lxd”}’
The queries used in the two “Loki logs” panels are the following:
In my logs I’m seeing instance=“none” rather than the instance=“lxd” as used in the LXD dashboard panels. This appears to be the issue, “none” versus “lxd”
Where does instance=“lxd” come from? Why did mine end up as instance=“none”?
I kept fairly meticulous notes during my install process, so if someone can point me where to look regarding where instance=“lxd” comes from, where it is set, maybe I can figure out if I did something wrong to cause mine to be named “none” instead of “lxd”. Or maybe something in the instruction docs is missing. Not really sure.
Aside from that, also need to figure out what needs to be done to correct this so that my LXD logs start displaying in the LXD Grafana dashboard.
I thought I saw something showing they were working on this, but in case it was something else:
If you make the dashboard editable, and edit the panels, remove the instance=$jobs from the panels’ queries, it will work (though not be filterable), or alternatively create a separate variable with the server names.
Instance in this case is supposed to be the server name according to LXD (or cluster member name in my case, I don’t know how it looks without a cluster), but I think they are accidentally using the variable associated with the prometheus job name.
The other day I stumbled upon something about a commit to the dashboard in version 5 which covered that.
I’m using dashboard version 5 and still have the “No data” issue, though.
I’ll probably try the dashboard edit that you mentioned.
Knowing where instance=“lxd ” gets the “lxd” value from might help me track this down. When I initialized LXD (lxd init) I didn’t name anything “lxd.” The only part with “lxd” in it is my ZFS dataset name which is serverpool/lxd , other than that I’m not sure where that value in instance=“lxd” or on my server “none” is coming from.
Its from the prometheus job, assuming you followed the instructions for setting it up
Like I said, if you make the dashboard editable, then edit the panels, and remove the instance=$job it should work, assuming you have the same issue I have.
It’s still incorrect on the latest version of the dashboard they provided, at least for clusters
After selecting the dashboard, you can hit the gear icon and should have an option somewhere to mark it editable. Then go to the main dashboard, and mouse over the bottom panels and you will see 3 dots appear where you can click edit. You will see a query where you remove the instance=$job. You then need to save and repeat for the other panel.
RE: the instance name, its pulled from the job name field in the prometheus.yaml file, specifically:
job_name: lxd
(unless I’m totally wrong, its been a minute since I figured this out)
Not sure what the root cause is, but I suppose no worries as at least I can still dig into that info if I need to via queries and also with the changes you suggested. Thanks again for your help.