Bots list review

[ekemeyer]

Because

Since finding a partner IP address in the bad bots list, we need to take time to review the IP ranges.

From Drew: parse the bad bots log and group by the first 2 or 3 IP octets. The current apache log parser that creates the bad bot IP config already does this, so little to no work required there. But, then if we see any IPs in the bad bot log that are "alone" in their address space, then we can probably reasonably assume that they are not bots, and unblock their address space again.

To do

• Come up with policy to define what a bad bot is
• Flag the non-bots based on that policy

Done when

Bad bot list has been reviewed and we have documented reasoning for removal of certain IPs

Additional context

Notes here

related, past tickets and documentation #1908 and #2750