This crawler can crawl tens of thousands of apps information from the Xiaomi App Store, and save them into the MongoDB. It is based on Python + Scrapy.
- Crawing data from specified url
- Persisting data in MongoDB
- Crawl more content by following next page links
| Stage | Start | End | Goals |
|---|---|---|---|
| 1 | 07/25/16 | 07/31/16 | Environment setup, Resource Discovery, Plan Discussion, and Proposal Draft Writing |
| 2 | 08/01/16 | 08/07/16 | System Design, System architecure setup, Basic crawling function implementation and Document Writing |
| 3 | 08/08/16 | 08/14/16 | Persistent layer setup, Multi-thread & link analysis study and Document writing |
| 4 | 08/15/16 | 08/21/16 | Auto crawl function implementation and Document writing |
| 5 | 08/23/16 | 08/28/16 | User Manual Writing and Presentation Making |
See the LICENSE file for license rights and limitations (MIT).
- category: big data
- team: KLV Team
- description: Crawling information from Xiaomi APP store.
- stack: python, scrapy, mongodb