Are logs with different message formats supported in one file?

[maxim-lbmv]

For example, if logs from the entire system are collected into one file, the programs in which have their own format for recording messages.

[tstack]

Sorry, no, lnav expects a single format per file. There can be multiple regex patterns in a format, though.

The reason is that lnav uses the regex patterns to find the start of a log message. If none of the patterns in a format match a given line, then it's considered a continuation of the current log message. Supporting multiple log formats in a single file would mean that we would have to try every single pattern whenever a mismatch occurred, which would seriously affect performance.

[maxim-lbmv]

It may not have much impact, you just need to improve the string matching algorithm

1. Remember the format of the last correct line and use it first for the next checks; this will allow you not to go through all the formats if the file contains only one format.
2. Remember the format of all detected correct strings in a list, use the formats from the list, if the last format does not fit, this will allow you not to go through all the formats if the file contains several formats.
3. Define a simple invalid string check, for example, check that the string length is less than the minimum string length of any of the formats, perform this check before any other check, this will allow you to immediately cut off strings that do not fit any format with a simple check.

Thus, if the file contains lines of only one format, then performance will not change. If the file contains lines of different formats, then the execution time of the program will slow down in the worst case by the number of formats times, provided that the lines are constantly interleaved. If the file contains wrap lines for which the simple invalid line condition is not met, then program execution will slow down by the number of such lines multiplied by the number of possible formats. Usually there are not many messages with line breaks in the logs.

In any case, you can make this behavior optional and define it in the configuration. This will not be a problem for users who care about speed, because the option will be disabled by default. And for users who analyze small logs, but with different message formats, it will make their work easier.

But if the problem is really solved by the ability to define several string formats in one file format, then there is no need to complicate the program logic. The only thing that would be appropriate is to supplement the log file format so that it can be composite and contain links to the file formats used in it.

[tstack]

1. Remember the format of the last correct line and use it first for the next checks; this will allow you not to go through all the formats if the file contains only one format.

It already works this way.

2. Remember the format of all detected correct strings in a list, use the formats from the list, if the last format does not fit, this will allow you not to go through all the formats if the file contains several formats.

I don't see what you mean here. If a line doesn't match the last format, we would have to try all the formats, no matter what. There's no getting around that.

3. Define a simple invalid string check, for example, check that the string length is less than the minimum string length of any of the formats, perform this check before any other check, this will allow you to immediately cut off strings that do not fit any format with a simple check.

The minimum length for most formats is not very long. So, it wouldn't take much for a multiline message to meet that minimum length.

If lnav was written in rust, I could use RegexSet, which seems to be optimized for finding a match in a bunch of regexes. But, it isn't and PCRE doesn't seem to have a similar mechanism.

[maxim-lbmv]

I don't see what you mean here. If a line doesn't match the last format, we would have to try all the formats, no matter what. There's no getting around that.

Here I meant the case when the file uses, for example, two formats that are usually checked last, if you remember them in a separate list, you don’t have to check all the previous ones every time.

Probably using several formats in one file is really not optimal.