Provide first party bulk download

[cthoyt]

I would like to download the entire database all at once. Having a paginated API isn't ideal for this. It would be great if you had a bulk download link.

Perhaps this could be periodically dumped to zenodo

[cthoyt]

In the mean time, I wrote a library that extracts everything from the API (https://github.com/cthoyt/biotools-client, takes about 2 hours) and uploaded the results to Zenodo under the CC BY 4.0 license (https://zenodo.org/records/13869530)

[redmitry]

I did similar task for tools monitoring...
https://gitlab.bsc.es/inb/elixir/openebench/tools-monitoring/-/blob/main/tools-monitoring-import/biotools-import.py
It converts biotools to bioschemas, so removing all the conversion part and saving to a disk instead of pushing to the server...
It may need some adoption :-(

Cheers,

D.

P.S. importing all biotools should take 10 minutes... 2 hours sounds a lot.

[veitveit]

This is a very good point and we are well aware of it.

The future synchronization with the ecosystem might partly solve this.

I assume a zipped json would be preferred?

We are backing up the content in as sql files which could be an alternative (after removal of user information).

[cthoyt]

Yes, a zipped or otherwise compressed JSON would get the job done! Make sure this archive is on an external system outside of ELIXIR infrastructure, so it will still exist after ELIXIR ends (future planning ;))

[redmitry]

Hi All,
I made a simplest bio.tools importer I could. On my notebook it takes 15 min to load all data.
I also noticed, that bio.tools doesn't compress the data sent via the API.
You may save some time and a lot of traffic enabling gzip on the server...

Best,

Dmitry
biotools-backup.zip

P.S. I put Apache 2.0 license coz without a license usage of any piece of code may be troublesome....