####About Main idea behind proxyHarvester is to get as reliable free proxy list as possible and maintaining it in real time. First thing harvester is doing is scraping all defined free proxy lists. Next, it check if each single proxy is healthy or sick. Sick proxies (most of them... free proxies are crap...) are removed from list. Healthy proxies are stored and constantly checked if still healthy. Healthy proxies are stored in python pickle file each 30s. To get as good proxy list as possible, harvester should run while whole scrapping processes.
-
Notifications
You must be signed in to change notification settings - Fork 2
tulski-s/robots
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
About
Proxy Harvester - get free and healthy proxy
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published