Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CNDB-10907: Implement overlap diagnostics #1453

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

CNDB-10907: Implement overlap diagnostics #1453

wants to merge 1 commit into from

Conversation

blambov
Copy link

@blambov blambov commented Dec 5, 2024

What is the issue

Extra diagnostics methods to better understand UCS compaction hierarchy structure.

For Cassandra itself this provides a new --overlaps (-O) option to nodetool compactionstats that generates an overlap map showing the maximum overlaps for every shard of every level in the strategy, as well as a total for all sstables in the system (not necessarily the sum of the per-level overlaps).

Also implements methods to be used by CNDB-side diagnostics.

What does this PR fix and why was it fixed

...

Checklist before you submit for review

  • Make sure there is a PR in the CNDB project updating the Converged Cassandra version
  • Use NoSpamLogger for log lines that may appear frequently in the logs
  • Verify test results on Butler
  • Test coverage for new/modified code is > 80%
  • Proper code formatting
  • Proper title for each commit staring with the project-issue number, like CNDB-1234
  • Each commit has a meaningful description
  • Each commit is not very long and contains related changes
  • Renames, moves and reformatting are in distinct commits

Copy link

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be great to expose this in CNDB, it should be pretty simple, it is only a matter of exposing the API in the REST API of the CompactorLeader

@blambov
Copy link
Author

blambov commented Dec 5, 2024

Sample printout:

Max overlap map for density.blobs:
  all: 26, per shard: [11, 11, 17, 17, 14, 14, 20, 20, 17, 17, 23, 23, 20, 20, 26, 26]
  unrepaired:0: 18, per shard: [6, 10, 14, 18]
  unrepaired:1: 9, per shard: [6, 9, 5, 8, 4, 7, 3, 6]
  unrepaired:2: 3, per shard: [3, 3, 3, 3, 3, 3, 2, 2, 3, 3, 3, 3, 3, 3, 2, 2, 3, 3, 3, 3, 3, 3, 2, 2, 3, 3, 3, 3, 3, 3, 2, 2]

One thing this makes very visible is the effect of early open: L0 overlaps are small for the lower keys because ongoing compactions have already early-opened the result. L1 also has this (every second overlap is higher).

Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
70.2% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

@cassci-bot
Copy link

✔️ Build ds-cassandra-pr-gate/PR-1453 approved by Butler


Approved by Butler
See build details here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants