Quick first feedback from trying to index a local log file

[nh2]

I have a 100 GB log file and wanted to explore whether quickwit can help me do quieries against it faster than grep.

I followed https://quickwit.io/docs/ingest-data/ingest-local-file after installing quickwit from my linux distribution (nix-shell -p quickwit).

I hit the following obstacles I want to share:

It says to run quickwit run. When I do that, I get:

quickwit run
✘ Command failed: Failed to load node config.

Caused by:
    0: StorageError(kind=NotFound, source=No such file or directory (os error 2))
    1: No such file or directory (os error 2)
    2: Failed to read file file:///home/niklas/tmp/quickwit/config/quickwit.yaml

The tutorial does not explain where to get that config/quickwit.yaml from.
I eventually figured that probably a default config is in the Github repo (it was), but it would be nice if it was somehow clearer where I can get the default config from, from either the tutorial or the CLI output.

Next, I used the proposed stackoverflow-schemaless-config.yaml.

I'm assuming "schemaless" will allow me to ingest any kind of text file.

When I run quickwit tool local-ingest on a 500 KB file, I get this output:

% quickwit tool local-ingest --index stackoverflow-schemaless --input-path /var/log/mylog.log
❯ Ingesting documents locally...

---------------------------------------------------
 Connectivity checklist 
 ✔ metastore storage
 ✔ metastore
 ✔ index storage
 ✔ _ingest-cli-source

 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  2.04MB/s Time 00:00:02
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  1.36MB/s Time 00:00:03
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  1.02MB/s Time 00:00:04
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.81MB/s Time 00:00:05
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:06
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:07
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:08
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:09
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:10
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:11
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:12
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:13
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:14
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:15
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:16
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:17
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:18
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:19
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:20
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:21
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:22
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:23
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:24
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:25
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:26
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:27
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:28
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:29
 Num docs   32068 Parse errs 32068 PublSplits   0 Input size     4MB Thrghput  0.00MB/s Time 00:00:30

Indexed 0 out of 32,068 documents in 30s. Failed to index 32,068 document(s). (100.0% error rate)

Clearing local cache directory...
✔ Local cache directory cleared.
✘ Command failed: Failed to ingest all the documents.

I'm immediately confused by the fact that this invocation runs for, and only fails after, 30 seconds even though my file is only 500 KB and it apparently parsed everything in the first line of output.

What is it doing in those 30 seconds? Unclear.

Also the fact that some lines show e.g. 1.36MB/s when it probably isn't doing anything, is confusing. Probably some rolling average causing it (it's weird that the rolling average counts on when the program is apparently already done with the file).

Why is it failing to parse the file? Unclear. I have plenty of Parse errs but the UX is not discoverable in the sense that there's no clue what I should do next to investigate those errors.

Is schema-less not what I expected? It would be nice if the tutorial could explain that if that's the case.

I hope this is useful feedback regarding "let's try this tool out quickly" experience, thanks!

[PSeitz]

I have a 100 GB log file and wanted to explore whether quickwit can help me do quieries against it faster than grep.

I followed https://quickwit.io/docs/ingest-data/ingest-local-file after installing quickwit from my linux distribution (nix-shell -p quickwit).

I hit the following obstacles I want to share:

It says to run quickwit run. When I do that, I get:

quickwit run
✘ Command failed: Failed to load node config.

Caused by:
    0: StorageError(kind=NotFound, source=No such file or directory (os error 2))
    1: No such file or directory (os error 2)
    2: Failed to read file file:///home/niklas/tmp/quickwit/config/quickwit.yaml

The tutorial does not explain where to get that config/quickwit.yaml from. I eventually figured that probably a default config is in the Github repo (it was), but it would be nice if it was somehow clearer where I can get the default config from, from either the tutorial or the CLI output.

nixOs does have a quickwit package? That's pretty cool - it's incomplete without the config though.
The config is part of the installation, which is linked in the tutorial.

The error should be better though (maybe even have some mechanism to create a default config?)

Next, I used the proposed stackoverflow-schemaless-config.yaml.

I'm assuming "schemaless" will allow me to ingest any kind of text file.

No, that's not how it works.
The local file docs are pretty bare bones, anything in get started (quickstart, tutorials) would you get better started...
Not sure if we should add links here or change it into something that's more like a reference and less like a tutorial.

Why is it failing to parse the file? Unclear. I have plenty of Parse errs but the UX is not discoverable in the sense that there's no clue what I should do next to investigate those errors.

Logging some errors would probably make sense.

[fmassot]

@nh2 thanks a lot for your feedback, you hit several Quickwit's limitations, and you had clearly a bad UX, very sorry for that.

I opened an issue on nixpkgs about the config file, it would be really nice to have it directly in your path.

We are working on document validation and it should be shipped in the next version, with that quickwit will return better errors and help you fix your config/setup.

We definitely need to improve the UX, especially for those quick starts. We will work on that, I opened this issue to track this.

[nh2]

Thanks for the responses so far!

@PSeitz

I'm assuming "schemaless" will allow me to ingest any kind of text file.

No, that's not how it works. The local file docs are pretty bare bones, anything in get started (quickstart, tutorials) would you get better started...

I did read the Quickstart and other tutorials. It says e.g.

There are no more explicit field definitions as we can use the default dynamic mode: the undeclared fields will still be indexed, by default fast fields are enabled to enable aggregation queries. and the raw tokenizer is used for text.

And the stackoverflow-schemaless-config.yaml uses the mode: dynamic.

If this isn't it, what is the way to just index a plain text file / where is it explained in the quickstart or the tutorials?