add ScrapyCloudCollectionCache #30
Conversation
Codecov Report
@@ Coverage Diff @@
## master #30 +/- ##
==========================================
- Coverage 85.24% 82.12% -3.12%
==========================================
Files 9 9
Lines 488 526 +38
==========================================
+ Hits 416 432 +16
- Misses 72 94 +22
|
scrapy_autoextract/cache.py
Outdated
| ) | ||
|
|
||
| def close(self): | ||
| pass |
There was a problem hiding this comment.
Shouldn't be closing the sc client here?
There was a problem hiding this comment.
@asadurski it looks great. I left a minor comment. The approach in general makes sense to me.
If I understand it well, it will create a collection using the jobid, so it will be empty, and then for every request there will be one request to the cache, even if nothing will be found there (because the request is new). This can be inefficient and can make crawling slower for no reason.
At the same time, it would be nice to have a reusable cache that is using collections. I imagine that all jobs for a particular spider could be using this cache, and this will speed up it by a lot!
So I propose here may be to be more flexible in the configuration:
- To have a write-only flag, so that the responses are saved but no requests are done against the cache.
- To have a way to customize the name of the collection to be used. So that it can be reused across jobs.
scrapy_autoextract/cache.py
Outdated
|
|
||
| @classmethod | ||
| def fingerprint(cls, request: Request) -> str: | ||
| return request.url |
There was a problem hiding this comment.
hey! What's the reason just an URL is used as a fingerprint?
There was a problem hiding this comment.
Agreed, that probably wasn't optimal. Introduced the same fingerprint mechanism as in AutoExtractCache.
It was like this for simplicity - those cached items are later retrieved in QA tool, MATT, and compared with job items. Now I just have to modify the retrieval in MATT to reach to item contents instead of the collection key.
|
Hi @asadurski . I'm moving the cache mechanism to |
Adds a custom cache that stores raw responses from AutoExtract to ScrapyCloud collections.
Adds settings:
Adds docs.
(Bonus: minor fixes to docs).