Can't get over 10000 results from the API

[jaanisoe]

Or more precisely, the maximum working value of the page parameter is 1000:
https://bio.tools/api/tool?page=1000

When the next page is tried (as listed by "next": "?page=1001"), I get a "Server Error (500)":
https://bio.tools/api/tool?page=1001

[albangaignard]

I'm writing a crawler for data exchange and the same happens when page = 1001.

[hansioan]

In the case of bio.tools it would make sense to have page 1001 since there are 10,000+ tools and 10 tools per page. But I think this is a hard limit in the code/framework/server which gives the server error.

I've tried the same on dev.bio.tools:
https://dev.bio.tools/api/tool?page=1001
and get the same error, but since on dev.bio.tools there are only 9,582 resources we can enforce this hard limit assumption by checking that page 999 and 1000:
https://dev.bio.tools/api/tool?page=999
https://dev.bio.tools/api/tool?page=1000
actually give a 404 which is supposed to happen after the page number is too large to hold any results, which should be the case of 1001 as well.

[redmitry]

It looks DJango limit:
http://www.django-rest-framework.org/api-guide/pagination/

one should modify the limit:

class LargeResultsSetPagination(PageNumberPagination):
    page_size = 1000
    page_size_query_param = 'page_size'
    max_page_size = 10000

class StandardResultsSetPagination(PageNumberPagination):
    page_size = 100
    page_size_query_param = 'page_size'
    max_page_size = 1000

[joncison]

Reassigning to you @piotrgithub1 - this looks like it's easy to fix

[piotrgithub1]

Hi @jaanisoe
Apologies for the inconvenience, we honestly did not expect anyone to go so deep in the page number as supporting ontology is supposed to be helpful in quickly get what you are looking for.
The fix is not as easy as one would expect, we'll put it on top of the new todo list after we are done with current batch of issues.

Best regards,
Piotr

[joncison]

thanks @piotrgithub1 - we have already quite a few groups / dependencies (in USA, France, Spain ...) which are taking the whole bio.tools data, so there's a real use-case here.

[piotrgithub1]

Made a temporary workaround for the bug.
Long-term we need to take a closer look at how this is used by the dependant services and make a more use specialized functionality for getting larger amount of content out of the registry as tight coupling with ontologies is clearly insufficient.

NOTE: leaving this open until we get a confirmation from @jaanisoe

Best regards,
Piotr

[albangaignard]

Thanks a lot, Piotr ! I will try it right know. The use case is the following : - iterate over all entries and get its JSON serialization - transform it into RDF (JSON-LD) - populate a knowledge base for further semantic web querying. Alban

…

Le 16 févr. 2018 à 15:02, piotrgithub1 ***@***.***> a écrit : Made a temporary workaround for the bug. Long-term we need to take a closer look at how this is used by the dependant services and make a more use specialized functionality for getting larger amount of content out of the registry as tight coupling with ontologies is clearly insufficient. NOTE: leaving this open until we get a confirmation from @jaanisoe <https://github.com/jaanisoe> Best regards, Piotr — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#310 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKLGELJDYRjFcfeMFTsgxBfO8ENZrHOkks5tVYpggaJpZM4RWi3C>.

-- Alban Gaignard, PhD, CNRS

[jaanisoe]

Yes, it's working now. Thanks!

[piotrgithub1]

Awesome :)
As it is a temporary workaround while we are in the process of making a permanent fix, I will monitor it daily if it works.

[albangaignard]

Hi all, I confirm, I've been able to crawl all the 10059 entries ! Thanks again, Alban

…

Le 16 févr. 2018 à 15:09, Alban Gaignard ***@***.***> a écrit : Thanks a lot, Piotr ! I will try it right know. The use case is the following : - iterate over all entries and get its JSON serialization - transform it into RDF (JSON-LD) - populate a knowledge base for further semantic web querying. Alban > Le 16 févr. 2018 à 15:02, piotrgithub1 ***@***.*** ***@***.***>> a écrit : > > Made a temporary workaround for the bug. > Long-term we need to take a closer look at how this is used by the dependant services and make a more use specialized functionality for getting larger amount of content out of the registry as tight coupling with ontologies is clearly insufficient. > > NOTE: leaving this open until we get a confirmation from @jaanisoe <https://github.com/jaanisoe> > Best regards, > Piotr > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub <#310 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKLGELJDYRjFcfeMFTsgxBfO8ENZrHOkks5tVYpggaJpZM4RWi3C>. > -- Alban Gaignard, PhD, CNRS

-- Alban Gaignard, PhD, CNRS

[joncison]

We need to check this still works when we make the next big release

[joncison]

See also #355

[hmenager]

fix confirmed

[joncison]

cc @jaanisoe @albangaignard @redmitry