Be able to export a full list of media clusters.

[caiosba]

Description

The current export limit for media lists is 10.000 because this is the maximum size of a result window in ElasticSearch. The solution is to paginate the results.

Fixes: CV2-5205.

How has this been tested?

This is just a refactoring, the existing test should cover it.

Checklist

• I have performed a self-review of my own code
• I have added unit and feature tests, if the PR implements a new feature or otherwise would benefit from additional testing
• I have added regression tests, if the PR fixes a bug
• I have added logging, exception reporting, and custom tracing with any additional information required for debugging
• I considered secure coding practices when writing this code. Any security concerns are noted above.
• I have commented my code in hard-to-understand areas, if any
• I have made needed changes to the README
• My changes generate no new warnings
• If I added a third party module, I included a rationale for doing so and followed our current guidelines

[melsawy]

@caiosba to get all existing items not recommended to use pagination as maximum item to get through pagination is 10.000 items but you should iterated using search_after option this is a reference to get items using search_after

[caiosba]

Thanks Sawy, I'll make this change!

[DGaffney]

As long as this offset/limit syntax works it otherwise looks good to me:

      search.set_option('eslimit', page_size)
      search.set_option('esoffset', offset)

[caiosba]

@caiosba to get all existing items not recommended to use pagination as maximum item to get through pagination is 10.000 items but you should iterated using search_after option this is a reference to get items using search_after

Thanks @melsawy ! Indeed, I tested it locally with more than 20.000 items and my previous approach only exported 10.000, with no alerts from ElasticSearch. I re-implemented it using search_after and then I was able to export more than 20.000 items in less than 15 minutes locally. Thanks a lot for the reference you provided, it helped me a lot!

Kindly re-review, @melsawy :)

[melsawy]

LGTM and I suggest to use do..whileLoop instead of while true to fix CC issue