Is --browsers-file ignored?

[AndrewKvalheim]

I’m running an ActivityPub server and would like peers to be classified as ActivityPub, not as Crawlers. I’ve tried defining a Mastodon “browser” but it seems to have no effect:

$ cat /etc/goaccess/browsers.list
Mastodon	ActivityPub
$ goaccess /var/log/caddy/mastodon.log --browsers-file=/etc/goaccess/browsers.list --output=json | jq --compact-output '.browsers.data[] | { group: .data, match: .items[] | .data }'
{"group":"Crawlers","match":"(Mastodon/4.0.2"}
{"group":"Crawlers","match":"CloudFront"}
{"group":"Crawlers","match":"(Mastodon/3.5.3"}
{"group":"Crawlers","match":"(Mastodon/3.4.7"}
{"group":"Crawlers","match":"agent"}
{"group":"Crawlers","match":"3.5.5"}
{"group":"Crawlers","match":"glitch"}
{"group":"Crawlers","match":"2022.10-1484"}
{"group":"Crawlers","match":"(Mastodon/3.2.2"}
{"group":"Crawlers","match":"2020.12-dev-1382"}
{"group":"Crawlers","match":"(Mastodon/3.2.0"}
{"group":"Crawlers","match":"(Mastodon/3.5.1"}
{"group":"Crawlers","match":"SemrushBot/7~bl"}
{"group":"Crawlers","match":"MastodonInstances"}
{"group":"Crawlers","match":"(Mastodon/3.2.1"}
{"group":"Crawlers","match":"2022.12-dev-1497"}
{"group":"Crawlers","match":"(Mastodon/3.5.5"}
{"group":"Unknown","match":"Unknown"}
{"group":"Firefox","match":"Firefox/107.0"}
{"group":"Firefox","match":"Firefox/106.0"}
{"group":"Others","match":"python-requests/2.28.1"}
{"group":"Others","match":"python-requests/2.25.1"}
{"group":"Others","match":"python-requests/2.27.1"}
{"group":"Others","match":"python-requests/2.22.0"}
{"group":"Chrome","match":"Chrome/87.0.4280.88"}
$ goaccess /var/log/caddy/mastodon.log --browsers-file=/etc/goaccess/browsers.list --output=json | jq --raw-output '.hosts.data[].items[]' | grep 'Mastodon' | head -n 10
http.rb/5.1.0 (Mastodon/4.0.2; +https://mastodon.social/)
http.rb/5.1.0 (Mastodon/4.0.2; +https://fosstodon.org/)
http.rb/5.1.0 (Mastodon/4.0.2; +https://hachyderm.io/)
http.rb/5.1.0 (Mastodon/4.0.2; +https://social.coop/)
http.rb/5.1.0 (Mastodon/4.0.2; +https://artisan.chat/)
http.rb/5.1.0 (Mastodon/4.0.2; +https://mstdn.social/)
http.rb/5.1.0 (Mastodon/4.0.2+glitch; +https://infosec.exchange/)
http.rb/5.1.0 (Mastodon/4.0.2; +https://en.osm.town/)
http.rb/5.1.0 (Mastodon/4.0.2; +https://social.ridetrans.it/)
http.rb/5.1.0 (Mastodon/4.0.2; +https://mastodon.seattlematrix.org/)

What am I doing wrong?

---

GoAccess 1.6.5, Ubuntu 22.04.1 LTS

[allinurl]

Just tested this by adding an entry to the browsers.list as:

Mastodon\t\t\t\t\t\tCrawlers

Where \t is the tab character. It did work. Please make sure your file has actual tabs in between the key/name and value/category. Also, please note that the SIZE of the list is proportional to the run time. The longer the list, the more time GoAccess will take to parse it.

[AndrewKvalheim]

Thanks. It does use a tab:

$ od -c /etc/goaccess/browsers.list
0000000   M   a   s   t   o   d   o   n  \t   A   c   t   i   v   i   t
0000020   y   P   u   b  \n
0000025

In your example it still says Crawlers. Does it work with another value?

[allinurl]

Here's what I get:

[AndrewKvalheim]

How about with one of the sample user agent strings from above?

[allinurl]

You're right. It won't as it checks for crawlers here. We could move block starting at 577 above line 574, however, I believe I'm checking for possible crawlers before the user's list since it would be much faster to identify them than going through a potential big list of stuff given by the user. On the other hand, it breaks what you're trying to do...

[allinurl]

I don't see any other way of doing this other than checking the user's file first. Would you be able to submit a PR by swapping those lines? I'm happy to merge that after running some tests. Thanks.

[AndrewKvalheim]

I see, thanks! I’ve submitted #2426.

[allinurl]

Merged. Thanks again. The changes will be out in the upcoming release.