Return all results in single API call

[jayrdi]

Hi, sorry if this is not the right place for this as it's not an issue as such. I only have today to get this sorted though as I am moving on from this role.

Anyway, I am trying to list all tools associated with RD-Connect within the WordPress site for RD-Connect but it only pulls 10 items at a time.

It's difficult to implement pagination with the theme we are using and so I am hoping there is an endpoint parameter that will allow me to pull all results in a single page?

Thank you for your help!

[joncison]

I'm afraid there isn't currently. More flexibility has been requested a lot (e.g. #111, #325, #76, #310) - an improvement has to be made soon, but it can't be done today.

cc @piotrgithub1 @hansioan - the requests around this are building up. Let's discuss / act during the next dev call?

[jayrdi]

Okay, thanks for your help. For now then, I will set it up as if that functionality exists, and then when it does I can just add the endpoint parameter to list all items to the API request and the web page should display properly..

Thanks for getting back to me so quickly!

[joncison]

no probs ... please cc your replacement on this thread so they can nudge us / keep up-to-date with developments

[jayrdi]

cc @lmatalonga

[albangaignard]

 Dear all, Maybe I could help on this topic. I wrote a small script to crawl all bio.tools entries from the REST API. It should be easy to reuse/extend it to filter specific “RD” resources. The code is in Python but its crawling part could be easily translated to any other language. Best regards, Alban Le 11 oct. 2018 à 12:12 +0200, Jon Ison <[email protected]>, a écrit :

…

no probs ... please cc your replacement on this thread so they can nudge us / keep up-to-date with developments — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[joncison]

@albangaignard it'd be nice to put this in a repo, so others can see it. Lot's of people have to do this, so having a worked example would be very helpful.

We could create https://github.com/bio-tools/utilities or some such for it (and similar things) - what do you think?

[albangaignard]

Hi all, I just commited a notebook to show how to crawl page per page.

@joncison, if you think its usefull we could create a dedicated repo to share kind of "recipes".

[joncison]

nice !

I just created:
https://github.com/bio-tools/utilities

Please upload it there, and add a note in the table in the README.md to give the file name and
a short description of what it is / does.

[jayrdi]

Thanks everyone for working on this, and thanks @albangaignard for the script to crawl pages. The problem we have is that we are displaying the tools on a web page within a WordPress site so we can't use Python (or many scripting languages, unfortunately!).

Would it be possible to make a new endpoint parameter for this? So that at the end of the REST request I could just add something like &limit=1000 to the end of https://bio.tools/api/tool/?domain=RD-Connect&sort=name&ord=asc to increase the number of results pulled through?

[joncison]

Understood @jayrdi - we'll take your suggestion when we take the other issues (#111, #325, #76, #310) - so this stays open until we have a resolution.

I personally want what you're proposing, but we have to be sure the implementation isn't a potential server-killer.

Me and @piotrgithub1 @hansioan will discuss / get back early Nov I expect

[piotrgithub1]

@joncison It seems we are getting more and more requests for "batch download" functionality, please bump this one up on your development todo list and we can have a code review of it mid next week if that's enough time?

[joncison]

yup, sounds good. I'll speak to @hansioan today about it - better support for relationships between tool entries and use of publication data are also in the mix (from a different thread) so we'll draw up a plan / nail some of this stuff

[piotrgithub1]

I don't think this is a good idea, Hans should be focusing on other things right now so he actually makes some good progress on that front...

[joncison]

@hansioan @piotrgithub1 we also have the request from EBI re paging for XML downloads, also more flexibility on paging / downloads requested (albeit indirectly) by the reviewer of one of the papers recently accepted ... so I think we should try to improve things somehow, sooner rather than later.