Directly harvest oai sources

[david-caro]

Currently in order to harvest oai services we rely on invenio-oaiharvester from an app like inspire to do it, save the result in xml files on disk, and then harvest those. It would be really nice if we can instead harvest directly from them.

Expected Behavior

For example, for the arxiv spider, instead of harvesting from a file, we should be able to pass a server (like http://export.arxiv.org/oai2), and the parameters for the harvest (like the date from, to and the sets to take into account).

As an example of similar functionality, you can check the inspirehep oaharvester harvest command for parameters, we can use https://sickle.readthedocs.io/en/latest/index.html too to do the oai protocol handling.

Current Behavior

There's no support to directly harvest from oai enabled services.

[michamos]

#198 looks like @kaplun's attempt at solving this issue.

[kaplun]

Yeah was writing it to @david-caro on gitter: for the OAI-PMH crawler see: #198
I never sent it for integration because I wasn't sure for testing.
Basically the OAI-PMH spider is directly embedded into our hepcrawl project (not assuming it will be integrated into scrapy. I noticed you were pointing Szymon to it)
The PR is working. It's just not complete WRT tests.
And it's showing how to use OAI-PMH (for CDS, but would be valid for arXiv too)

[szymonlopaciuk]

Closed by #203