Support asynchronous mode

[zrquan]

Description

This PR adds the --async parameter to enable asynchronous mode (close #643). In asynchronous mode, coroutines are used instead of threads to send network requests (use httpx).

I ran a few quick tests and found that the asynchronous mode really speeds up the scanning process. (uvloop can makes it even faster)

python dirsearch.py -u https://security-crawl-maze.app -e html -t 50  150.96s user 5.89s system 98% cpu 2:38.45 total
python dirsearch.py -u https://security-crawl-maze.app -e html --async  41.34s user 0.94s system 40% cpu 1:43.80 total
python dirsearch.py -u https://security-crawl-maze.app -e html --async -t 50  33.49s user 0.65s system 53% cpu 1:03.24 total

This PR isn't quite finished yet!

• I just copied and modified the existing code, so it needs to be refactored.
• The rate-limiting logic is likely to require further optimisation.
• HttpNtlmAuth hasn't been implemented yet.
• There might be some bugs in concurrency and timeout control.
• Should we replace requests with httpx? Since httpx supports both synchronous and asynchronous modes. There should be a new issue for discussion.
• Handle all request exceptions properly.
• Change the minimum version requirement to Python3.9. (httpx requires 3.8+)
• Adjust log/output in async mode.
• Fix errors from pickle (requester, loop, future...)
• httpx doesn't let me choose different proxy for each request.

Requirements

• Add your name to CONTRIBUTORS.md
• If this is a new feature, then please add some additional information about it to CHANGELOG.md

[zrquan]

Should we think about raising the minimum version requirement to Python3.9?

[maurosoria]

I like this!

We can jump to even newer versions if it is worth it.

DM me on twitter if you want to chat with us.

[maurosoria]

Thank you so much and congratulations for such important contribution!!!!