Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maybe we can eliminate group committer thread #130

Open
zz-jason opened this issue Sep 6, 2024 · 1 comment · May be fixed by #132
Open

Maybe we can eliminate group committer thread #130

zz-jason opened this issue Sep 6, 2024 · 1 comment · May be fixed by #132

Comments

@zz-jason
Copy link
Owner

zz-jason commented Sep 6, 2024

Enhancement

Currently, the group committer thread is used for:

  1. Calculate the minimal global user and system transaction IDs, determining whether a user transaction with remote dependency can be committed safely.
  2. Collect and flush WAL records from all worker threads to a single WAL file

The proposed solution is:

  1. remove group committer
  2. each worker writes its WAL records to the local WAL file, updates the flushed system and user transaction id on the worker context
  3. for user transactions with remote dependency, collect flushed system and user transaction id from other workers, calculate the global min flushed user and system transaction id, and commit the user transaction if safe
  4. for recovery, read all the wall files, shuffle by page id, sort by page sequence number in the wall record, and replay the wal records
  5. for checkpoint, the global user and system transaction id can be used to indicate lower watermarks and to truncate wal files
@zz-jason
Copy link
Owner Author

zz-jason commented Sep 8, 2024

A simple POC on ycsb workload A and B shows scalable performance improvements as worker thread increased, and great performance improvement over the group committer.

@zz-jason zz-jason linked a pull request Sep 9, 2024 that will close this issue
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant