Is there a way to determine how far along the search indexing has progressed?

[StingyJack]

GH built in search is really buggy lately and burned me twice in the last week, so I'm giving hound a try. I added all my org's repos to the config, set the max indexers to 20, and its going through the repos but I dont know how far along it is or how long it has left to complete.

I thought that I could count the number of data folders but that more than the number of repos currently (~1300 repos, > 1400 folders). I did have to stop and restart houndd because running 20 indexers was making other programs slow.

Is there a way to tell roughly when I can start using the indexed results?

[salemhilal]

Hey! I don't know off-hand of a way to monitor indexer progress, but the logs should show you how many searchers have been started, which gives you a rough idea of how far along in the indexing process Hound is. Does that help at all?

[StingyJack]

Counting the number of "Searcher started" is not really feasible when they are scrolling by. I cant count that fast. =D

I found that counting the two subfolder types (vcs, idx) in the data directory was close enough for the initial seeding where its downloading and indexing repos. The closer they get to the count of the repos in the config file, the closer it is to being ready. (If you are curious, it took about 2-3 hours total for a pared down repo list of only about 800 and with 20 searchers/indexers. About 70 or so repos failed due to "filename too long" or similar issues.)

It may be useful to post a stat line to the console/log every 30 seconds with the count of done, in progress, and awaiting repos. Thats up to you, I dont know Go or this program's design well enough to PR it and I think I can get the info I need without it.

[salemhilal]

Totally — that makes sense. I'm gonna leave this issue open and flag it as a "help wanted" for the time being. I'm glad to hear this is (mostly) working as expected on your windows machine! I'm unclear about what the bottleneck is for that startup time (i.e. if it's network time, or if the indexers take a while due to the size of your repo, etc), but if you have any intuitions there, I'd love to hear them. If you're really struggling for a loading indicator in the interim, it could be worth teeing these logs through a script that counts "Searcher started for" log lines for you.

[StingyJack]

The 799 of our repos I'm targeting average about 462 files per repo. That doesnt seem out of the "medium" size range, so i suspect its just the number of repos that it has to index and possibly network speed (speed test reports ~80 Mbps down).

Is there a specific log message to look for when the Web UI is ready?

[salemhilal]

Yes! I believe the All indexes built! should do the trick there.