Don't bail out from systemd log reader when reaching the end

[Ma27]

The log search starts at the current time, i.e. the end of the journal
(SeekTail() / SeekRealtimeUsec) and thus runs into an EOF.

Not a big deal because the code takes care of that, but the log-reading
loop is exited then and further messages are ignored.

Potentially fixes #55.

[elsgaard]

The log search starts at the current time, i.e. the end of the journal (SeekTail() / SeekRealtimeUsec) and thus runs into an EOF.

Not a big deal because the code takes care of that, but the log-reading loop is exited then and further messages are ignored.

Potentially fixes #55.

I made a build with the PR, and deployed the exporter to a couple of our production servers (RHEL 8), but they went to 100% CPU until i stopped the exporters again 15 min later. Has anybody else seen this ?

[ktosiek]

I don't think this is the right fix: log sources should wait for new data, and handle basic problems inside Read (at least that's what FileLogSource is doing by using a tailer).

sd_journal_next doesn't wait for new data, the fix is to .Wait() if sd_journal_next returns 0. See the example in sd_journal_wait man page: https://manpages.debian.org/testing/libsystemd-dev/sd_journal_wait.3.en.html#EXAMPLES

[Rudd-O]

We need this fix too, and sadly the CPU spinning problem is a blocker for us.

[Ma27]

Yeah I only tested it on a mailserver which doesn't have too much traffic (and relatively few logs).

However I think that it's perhaps better to just use loki for that purpose (and define alerts in Grafana), at least that's what I'm planning to do these days. Since I'm not planning to find a better solution, I'll close this now, sorry.