How to set labels for Loki?

LXD Support
1

Hi,

I’m trying to filter the Loki data by LXD server for the Grafana Dashboard.
It looks like the “instance” label is set to the hostname. (server1.domain.com)

I’d like to filter based on the Grafana “job” name, which i have given the short hostname.

“{app="lxd", type="lifecycle", instance="$job"}”

I’m trying to overrule the “instance” label and add a random label as test.

lxc config set loki.labels “instance=‘server1’, bla=‘bla2’”

lxc config show, shows

loki.labels: instance=‘server1’, bla=‘bla2’

But the “instance” label stays unchanged and I don’t see the new ‘bla’ label coming in to Loki.

What am I missing?

Kind regards, Justin

2

I am currently setting up an environment to replicate and troubleshoot but, looking at these lines, I think you can’t override labels that are already set.

3

Yeah, I think overriding existing label names won’t work. I think we should simply reject the loki.labels config when it includes one of the reserved label names.

So for the missing bla label, this feels like a bug.

4

Considering instance is a reserved/already used label, could you instead use another label name like job="$job"?

5

@sdeziel1 not sure if relevant but importing this might be useful https://github.com/lxc/incus/pull/403

1 Like
6

@tomp seems like yes, I’ll pull those bits.

1 Like
7

Figured out how this works. Setting loki.labels allows one to take key-values pairs from the log message part and put the pair in the key part. If I have this for instance:

2024-02-09T21:24:00Z {app="lxd", instance="node3", location="node3", name="c2", project="default", type="lifecycle"} requester-protocol="unix" requester-username="egelinas" action="instance-started" source="/1.0/instances/c2" requester-address="@" instance-started

and want the requester-username to become a key, I can set do this by setting the key as label.

lxc config set loki.labels="requester-username"

to get this:

2024-02-09T21:26:32Z {app="lxd", instance="node3", location="node3", name="c2", project="default", requester_username="egelinas", type="lifecycle"} action="instance-started" source="/1.0/instances/c2" requester-address="@" requester-protocol="unix" instance-started

then I can look for it:

logcli query -t '{requester_username="egelinas"}'

Note the replacement of - by _ in the key.

8

@egelinas @ru-fu could the docs do with clarification here do you think?

1 Like
9

Thank you very much for looking in to this @egelinas !

Clear, label “instance” is reserved.

Do I understand correctly that I cannot add custom labels and or assign a fixed value to a label?
I can only “upgrade” existing log fields to labels?

Wanted to go for the suggestion of @sdeziel1
“Considering instance is a reserved/already used label, could you instead use another label name like job=”$job"?"

Only option is to wait for “loki.instance” to become available? So I can make it match my job name?

10

Correct and “loki.instance” is the works.

1 Like