Add logging

[fbegyn]

Adds a logging collection stack and configures Grafana to be able to query them. promtail can also be added to the stations, so we can collect logs from them.

Potential improvements: configure promtail to accept rsyslog streams and also configure network devices to send logs.

[mcbloch]

When trying to start docker-compose I get the following error

promtail_1    | level=error ts=2023-02-23T14:19:07.884666368Z caller=main.go:167 msg="error creating promtail" error="failed to make journal target manager: creating journal reader: failed to open journal in directory \"/var/log/journal\": no such file or directory"

I do not have a journal log file on my system. I think there should be a better way to do this without making assumptions on the existence of the log files.

I also think the promtail client is something we want on more systems to collect logs, not only on the monitoring one. Only the monitoring is ran in docker so the promtail deploy should probably just be on the host, not containerized.
As default we could add a promtail in docker that only collects docker logs?

[mcbloch]

The extra labels added in the Loki pipeline also don't appear in Grafana for me. I don't have the container_name, image_name, ... labels.

[fbegyn]

The docker logging did not work on my system (like the journal at yours I guess, I'm blaming this on nixos configuring docker for journald logging). It does seem to contain a stark difference in assumption based on the system.
For the monitoring (and other docker hosts) we should determine which behavior we want and shape the host to match our assumptions for the logging.

I propose the following:

• on docker hosts accept the current assumption that the docker host is running in json-file logging mode. The other (maybe better option here that requires testing) is the configure docker sd on the docker host.
• bare metal/VM hosts, assume these systems are running journald and syslog in combination (as is the case on most debian based systems I believe). Implement a config that parses both these and sends the logs to the loki host.

[fbegyn]

I'm guessing the additional docker labels not being processed will have something to do with the docker log driver.