Send OVN logs to the events API

Project LXD
Status Implemented
Author(s) @monstermunchkin
Approver(s)
Release 5.18
Internal ID LX054

Abstract

This makes OVN logs accessible from the events API, which is used by both lxc monitor and Loki, by having OVN send the logs to a unix socket LXD is listening on.

Rationale

By default, OVN logs are written to files which LXD currently is not able to read. For a user to get any information regarding OVN, they needs to check the logs themselves. In order to make things easier, LXD should be able to capture these logs and make them available in the events API. This way, they can be viewed by querying the events API itself or using lxc monitor. They can also be sent to Loki.

Specification

Design

It’s possible to configure OVN in such a way that it sends it logs to a unix socket (see --syslog-method in ovn-controller(8), ovn-sbctl(8), ovn-nbctl(8)). In order for LXD to make use of this, it needs to gain a new config key which is core.syslog_socket. If this configuration key is set to true, LXD will listen on this socket and receive log messages from OVN.

By default, the log message pattern for syslog messages from OVN is %05N|%c|%p|%m. LXD will support this default pattern as well as %A|%05N|%c|%p|%m which includes the application name.

The variables are expanded as follows:

%A The name of the application logging the message, e.g. ovn-controller.
%N A serial number for this message within this run of the program, as a decimal number. The first message a program logs has serial number 1, the second one has serial number 2, and so on.
%c The name of the module (as shown by ovn-appctl -list) logging the message.
%p The level at which the message is logged, e.g. DBG.
%m The message being logged.

See ovn-appctl(8) for more on patterns.

LXD will receive the log messages from OVN, and send them to the events API as a new event type ovn. The event message will be the same as for event type logging, i.e. api.EventLogging. The event context will contain the module name, sequence number, and application name (if available).

event := api.EventLogging{
	Level:   logLevel,
	Message: message,
	Context: map[string]string{
		"application": applicationName, // will be omitted if not present
		"module":      moduleName,
		"sequence":    sequenceNumber,
	},
}

Loki integration

In order to have Loki receive OVN logs, the loki.types configuration key will accept the ovn value. The verbosity can be controlled via loki.loglevel as before. That means if OVN is configured to send debug logs, but loki.loglevel is set to info, only log messages with info (or lower log level) will be sent to Loki.

API changes

A new event type ovn will be added which will be supported by GET /1.0/events.

CLI changes

No CLI changes, but lxc monitor will be able to set --type=ovn to receive OVN logs.

Database changes

No database changes.

Upgrade handling

No upgrade handling.

Further information

No further information.

I feel like this is too vague. Will this be a generic syslog listener that will only (for now at least) handle ovn type messages?

Could it potentially be re-used in the future for other services?

If so would core.syslog_address be more appropriate?

How does this work in an OVN & LXD cluster?

Would each OVN chassis need to be configured to send its local logs into the local LXD syslog listener?

If so then would this new syslog listener address need to be a per-member configuration item?

Please can you also include how to configure OVN in the PR’s docs too.

I feel like this is too vague. Will this be a generic syslog listener that will only (for now at least) handle ovn type messages?

Could it potentially be re-used in the future for other services?

If so would core.syslog_address be more appropriate?

It could potentially be used for other services as well. For now, it’s only aware of OVN’s log pattern though.

How does this work in an OVN & LXD cluster?

Since this is a node config key, each cluster member would need to enable the listener separately.

Would each OVN chassis need to be configured to send its local logs into the local LXD syslog listener?

I haven’t tried it on a cluster, but I believe each OVN cluster member would have their own logs. So, I’d say yes.

If so then would this new syslog listener address need to be a per-member configuration item?

Yes, this is per-member.

Please can you also include how to configure OVN in the PR’s docs too.

I can do that.

OK thanks, lets confirm that.

Cool, lets use core.syslog_address then.

Did you look at the --syslog-method flag, it appears it can be used to send to a unix socket:

https://man7.org/linux/man-pages/man8/ovn-controller.8.html

unix: file, to use a UNIX domain socket directly.

If that works we could make the config option just core.syslog_socket=true and put it in a predictable location with the other socket(s) files.

This would avoid having additional network listeners and worrying about ports etc.