pygetpapers

getpapers (https://github.com/petermr/openVirus/wiki/getpapers), the primary scraper that we've been using so far, is written in Java and requires Node.js to run. Driven by the problems of maintaining and extending the Node-based getpapers, we've decided to re-write the whole thing in Python and call it pygetpapers.

People

1. PMR
2. Ayush
3. Dheeraj
4. Shweata

Our Initial Plans

PMR: This project is well suited to a modular approach, both in content and functionality. For example, each target repo is a subproject and as long as the framework is well designed it should be possible to add repos independently. An important aspect (missing at the moment) is "how to add a new repo" for example.

Requirements and Bugs to Fix

These are much too general. Who contributed them? Please expand:

General API

What does this mean?

Sort the Date

Which date?

Specifically, download only the Review, Research etc.

This is an EPMC option (I think). What is the current query format? We will need to customise this for the user.

Add attributes for repository specific functions

This is too general

Add option to get raw files as well as files in format such as xml and pdfs

Which raw files? Does EPMC have an interface? Do we want these files? Why? What are they used for?

Convert XML papers in a user readable format.

Why? This is not part of getpapers. It is already done by ami

Specify a wordlist and then get the count of those words for each paper.

Out of scope. This is ami-search.

Requirements from PMR (not exhaustive)

default number of hits