Set up your LMA stack

[bryce]

Note:
This documentation has moved to a new home! Please update your bookmarks to the new URL for the up-to-date version of this page.

[rick_h]

@bryce

20.04

Let’s go with 20.10 so we can use telegraf from universe and not document/use the PPA that’s used for testing at all please.

@bryce

aggregator:~# add-apt-repository -yus ppa:canonical-server/lma-dev

so this will go away

@bryce

sent

as noted earlier, it’s published on the web UI endpoint, but prometheus is responsible for coming and collecting the information. This might need a little tweaking to the wording to be accurate.

@bryce

aggregator

does this dns style work because of lxd magic?

@bryce

which you should find already enabled by default:

yay!

@bryce

aggregator:~# apt install lynx

since this is a lxd environment can we skip lynx and just document the URL that a user would leverage from the lxd host machine?

@bryce

$ lxc launch ubuntu:20.04 visualizer

again maybe use groovy since that’s where this “feature” was dev’d with?

@bryce

visualizer:~# apt install snapd

shouldn’t snapd be there already? is this because of the lxd images we’re using it doesn’t include snapd?

@bryce

like Node.js:

I’d just cut this and leave it at “dependencies.”

@bryce

Verify that Prometheus is listening on the port as we expect:

Do we need this section? I wonder if we can cut it for space/time?

[rick_h]

@bryce

On the Workload, install Telegraf from the PPA:

This isn’t the PPA any longer.

@bryce

workload:~# wget -O- http://127.0.0.1:9273/metrics

Nice!

@bryce

monitor:~# lynx workload:9273/metrics

can we use wget here again?

@bryce

cpu_usage_system

what do you think of making things that are variables or things that would line up with a config file being monospaced even when it’s inline here?

@bryce

/var/snap/prometheus/32/prometheus.yml

Like here, I like this a lot.

[bryce]

Thanks, I’ve updated with these changes.

[sed-i]

Under section “B. Alerts”,

@bryce

There is a sample configuration installed to /etc/prometheus/alertmanager.yml, however it’s full of example data. Instead, replace it entirely with this content:

## /var/snap/prometheus/32/alertmanager.yml
global:
  resolve_timeout: 5m

In my case the file was in /var/snap/prometheus-alertmanager/11/alertmanager.yml, and not in

• /var/snap/prometheus/32/alertmanager.yml or
• /etc/prometheus/alertmanager.yml

[ckchase]

In section C Grafana:

@bryce

So we see it’s getting its defaults from /snap/grafana/36/conf/, however it’s trying to load the actual configuration from /var/snap/grafana/common/conf/ so that’s where we should put our customizations. (/snap is read-only.)

@bryce

(preferably in a /var/snap/grafana/common/conf/custom.ini file)

These are incorrect references to the local configuration file. We should not assume that just because the provisioning path is overridden with an environment variable pointing to a certain location that the config should be in the same place. I suggest that the only way to know is to inspect the wrapper file. For example, with the grafana snap I recently installed you will see:
CONF="$SNAP_DATA/conf/grafana.ini"

You should confirm the value of the $SNAP_DATA variable also. I did this in a snap shell but there may be another way… I’m no snap expert.

So in my case the actual location for settings was:
/var/snap/grafana/36/conf/grafana.ini

This is a great page, and certainly a common search result for “Ubuntu logging”. That one misdirection had me beating my head against the wall though.

[andyvan92117]

The second edit session under B. Alerts uses the wrong name:

Now create /var/snap/prometheus/32/alerts.yml with the following contents:

## /var/snap/prometheus/32/alertmanager.yml

The ‘alertsmanager.yml’ should be ‘alerts.yml’.

[sergiodj]

@andyvan92117

Thank you for your feedback. Sorry this took so long, but we’ve been working through lots of feedback received on these docs. We think we’ve now addressed all feedback above this comment by editing the original text above. As the feedback is incorporated, we’ll delete the feedback comments to clean up in about a month from now.

[sergiodj]

@sed-i

Thank you for your feedback. Sorry this took so long, but we’ve been working through lots of feedback received on these docs. We think we’ve now addressed all feedback above this comment by editing the original text above. Are our edits OK, and have we correctly understood and addressed your feedback? If not, please let us know. Otherwise, as the feedback is incorporated, we’ll delete the feedback comments to clean up in about a month from now.

[sergiodj]

@ckchase

Thank you for your feedback. Sorry this took so long, but we’ve been working through lots of feedback received on these docs. We think we’ve now addressed all feedback above this comment by editing the original text above. Are our edits OK, and have we correctly understood and addressed your feedback? If not, please let us know. Otherwise, as the feedback is incorporated, we’ll delete the feedback comments to clean up in about a month from now.

On a side note, I would like to point out that I believe the right way to address this issue is to use $SNAP_COMMON (i.e., /var/snap/grafana/common/) as the right path for the customization file. I happen to be working on the grafana snap now, so I will take this opportunity to also improve its configuration scheme.