AI bots

[staabm]

I wonder whether this lib could/would add support to detect AI bots, so crawlers which are used to feed AI engines.
I found a repo which lists most of them: https://github.com/ai-robots-txt/ai.robots.txt

maybe thats something which could be used for/with crawler-detect?

[JayBizzle]

Hmmm, interesting suggestion.

Would you see our current isCrawler() method returning true for these AI bots, with a new additional method called something like isAiBot()?

Open to implementation suggestions here 👍

[staabm]

I think a separate method would be good, as I think some users might be interested in 'whatever' definition this lib had of a crawler before.

AI bots are a different class which might need different handling IMO

[beberlei]

A crawler is a crawler or not? Regardless of what the crawler actually does with the responses. If users want to differentiate then they should look at https://github.com/VolkswAIgen/VolkswAIgen - but for the purpose of this library I would think that a bot from OpenAI or Claude is detected as crawler.

[module17]

Agreed @beberlei -- I don't see a use case where it'd be necessary to treat an "AI" bot differently. Certainly if that were the case, I'd imagine that is beyond the scope of this library.