Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

journald metrics not available #55

Open
breed808 opened this issue May 2, 2020 · 5 comments · May be fixed by #112
Open

journald metrics not available #55

breed808 opened this issue May 2, 2020 · 5 comments · May be fixed by #112

Comments

@breed808
Copy link

breed808 commented May 2, 2020

When running postfix_exporter built from master, journald metrics are not available, and there is only a single path for the postfix_up metric present when querying the exporter.
The exporter is printing "Reading log events from systemd" on startup.

Bisecting with git reveals that commit 26d0642 introduced the issue.

@breed808
Copy link
Author

breed808 commented May 3, 2020

Additionally, I've run the exporter with a debugger and the CollectFromLogLine function is not reached.

@ktosiek
Copy link

ktosiek commented Aug 31, 2021

I believe postfix_exporter currently only reads the log once, and stops on first 0-length read.

@dswarbrick
Copy link

dswarbrick commented Jan 4, 2022

I co-maintain the Debian package prometheus-postfix-exporter, and have also just recently discovered that systemd journal support inexplicably broke somewhere between v0.2.0 and v0.3.0. This is somewhat disappointing, since v0.2.0 was working quite reliably.

From the minimal debugging that I've done so far, it appears to bail out of the SystemdLogSource.Read() function with io.EOF when s.journal.Next() returns zero, and never actually calls s.journal.GetEntry()

func (s *SystemdLogSource) Read(ctx context.Context) (string, error) {
    c, err := s.journal.Next()
    if err != nil {
        return "", err
    }
    if c == 0 {
        return "", io.EOF
    }

    e, err := s.journal.GetEntry()
...

That subsequently causes the for-loop in PostfixExporter.StartMetricCollection() to bail out, and that's pretty much game over.

By commenting out the "Start at end of journal" seek in logsource_systemd.go, I can get the exporter to "replay" historical systemd journal entries, and it appears to produce the expected metrics. However, when it reaches the end of the events, the Read() function still bails out with io.EOF. This seems to be the main issue - it doesn't wait for further events, and if it is allowed to seek to the end of the journal (i.e., unmodified code from the 0.3.0 tag), it will immediately bail out as there are no events to read.

@fpletz
Copy link

fpletz commented Jan 24, 2022

This might also be an issue with recent versions of go-systemd: coreos/go-systemd#392

@Ma27
Copy link

Ma27 commented May 26, 2022

I did some digging today because this bothered me a bit:

  • We go to the bottom of the journal (https://github.com/kumina/postfix_exporter/blob/master/logsource_systemd.go#L55 via SeekRealtimeUsec and then it runs into an EOF if nothing else is written to the journal in the meantime. Removing the return fixes the problem and seems to work fine for me.
  • For some reason the locking of GetEntry from go-systemd hangs, but I'm not sure here what's wrong so far. <- this was just an error by me while debugging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants