Plugin Restart Behavior and Search Scroll

[thanthos]

Would like to query the design behavior for this plugin.

1. What should be the behavior of the plugin when it encounter error and perform a restart?
2. How should it be handling the scroll?

I hit an OOM issue while trying to reindex my docs and I discovered that the search scroll context is not handled properly. While the plugin restarts, the search scroll context is left hanging when querying the status.
GET /_nodes/stats/indices/search?pretty
If the behavior is to continue after restart, I am not seeing it as the doc count exceed the original count. If the behavior is to restart the query, then the search scroll should be explicitly closed off before restarting.

Not too big an issue if interval settings is small but it can have significant impact if the interval is long or when resources are limited.

[ppf2]

This also happens if the scroll timeout is not set long enough. This will result in an apparent restart of the scroll, so if you are running a lengthy job where the scroll may timeout, you can end up with a whole bunch of superfluous documents in the resulting output. Easiest way to reproduce this is to explicitly issue a clear scroll to clear the search context while the job is running and you will see the resulting count to be completely off at the end.