Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Json parsing speed too slow #10

Open
johniez opened this issue Jul 11, 2019 · 0 comments
Open

Json parsing speed too slow #10

johniez opened this issue Jul 11, 2019 · 0 comments
Assignees

Comments

@johniez
Copy link
Collaborator

johniez commented Jul 11, 2019

JsonCpp library used to parse scroll results is extremely slow. Scrolling through larger indices show the delay much more. In numbers: what the current implementation iterates in 85 seconds, scroll implemented using rapidjson did in 25 seconds.

Unfortunately JsonCpp is used in Scroll header interface, so the change would break API/ABI. But the speed is worth it.

Possible implementation with rapidjson (as it is header only library):

  • hide it completely inside elasticlient, without publishing it in public interface
  • make the rapidjson parsing public (header only), so each client could compile it again

First option means that parsing has to be done twice. Internally and then for data parsing itself. Otherwise there is a risk of using different versions (binary incompatible) of rapidjson.
The second option allows shared parsing (internall and client's data) without possibility of using different versions of rapidjson (one compiled into elasticlient, another version in client code).

johniez added a commit that referenced this issue Jul 11, 2019
… this must be resolved:

  * reconsider using std::unique_ptr on the interface, maybe raw pointer would be more ABI secure
  * CMake && doc update
@johniez johniez self-assigned this Jul 12, 2019
johniez added a commit that referenced this issue Jul 12, 2019
…to be tested with -DUSE_SYSTEM_RAPIDJSON=NO.
johniez added a commit that referenced this issue Jul 17, 2019
…rary itself has to pass the reference to the header-only part and it cannot access it internally. Otherwise the binary compatibility could be harmed!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant