a novel spider that use muti threads and mongodb for 2.7 version
由于python GIL的存在，对于计算密集型的任务多线程处理没有任何优势，反而会因为同步、切换线程等造成消耗，所以解析response只能单的线程处理，网络请求和磁盘读写可以多线程处理
- 20个线程请求数据
- 1个线程解析response
- 3个线程写入MongoDB
-
Notifications
You must be signed in to change notification settings - Fork 0
silencesdg/novelSpider
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
a novel spider that use muti threads and mongodb for 2.7 version
Topics
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published