doc: Add robots txt feature proposal

clemlesne · Aug 16, 2024 · 3cc6018 · 3cc6018
1 parent 445ff97
commit 3cc6018
Showing 1 changed file with 1 addition and 0 deletions.
diff --git a/README.md b/README.md
@@ -19,6 +19,7 @@ Scraper:
 - [x] Load dynamic JavaScript content with [Playwright](https://github.com/microsoft/playwright-python)
 - [x] Show progress with a status command
 - [x] Track progress of total network usage
+- [ ] Respect [`robots.txt`](https://en.wikipedia.org/wiki/Robots.txt)
 
 Indexer: