Ferret

Installing Ferret on MacOS

GO (currently doesn't work on MacOS):

brew install golang

go get github.com/MontFerret/ferret

Binary: Download the ferret_darwin_x86_64.tar.gz binary from the ferret releases page, unzip your local directory and link an alias to it

alias ferret="/your/local/directory/ferret_darwin_x86_64/ferret"

To test type the ferret command

$ ferret
Welcome to Ferret REPL 0.10.1
Please use `exit` or `Ctrl-D` to exit this program.
>  

Further information and tutorials about ferret can be found here

Sample Ferret Code for Scraping a Biorxiv page:

LET doc = DOCUMENT(@url, { driver: "cdp" })
LET authors = (
     FOR auth in ELEMENTS(doc, '.highwire-citation-authors')
       RETURN {
            firstname : INNER_TEXT(auth,'.nlm-given-names'),
            surname : INNER_TEXT(auth,'.nlm-surname'),
            orcid_id : auth.a
       }
)

RETURN {
        abstract: INNER_TEXT(doc, '.abstract'),
        acknowledgements: INNER_TEXT(doc,'.ack'),
        title:  INNER_TEXT(doc,'.highwire-cite-title'),
        pub_time: ELEMENT(doc, 'meta[name="description"]'),
        authors: authors,
        sections: INNER_TEXT_ALL(doc, '[id^="sec-"]')
}

Ferret Command:

ferret --param=url:\"https://www.biorxiv.org/content/10.1101/2020.02.02.931162v2.full\" get_data.fql