Skip to content

Commit

Permalink
Merge branch 'master' of https://github.com/madeindjs/spider
Browse files Browse the repository at this point in the history
  • Loading branch information
madeindjs committed Feb 6, 2018
2 parents a758e91 + 1cb409c commit ff81169
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,10 @@ fn main() {

## TODO

- [ ]: multi-threaded system
- [ ]: respect _robot.txt_ file
- [ ]: add configuratioon object for polite delay, etc..
- [ ]: parse command line arguments
- [ ] multi-threaded system
- [ ] respect _robot.txt_ file
- [ ] add configuratioon object for polite delay, etc..
- [ ] parse command line arguments

## Contribute

Expand Down

0 comments on commit ff81169

Please sign in to comment.