CNDB-11532: Adaptive compression #1432

pkolaczk · 2024-11-20T15:27:14Z

This commit introduces a new AdaptiveCompressor class.

AdaptiveCompressor uses ZStandard compression with a dynamic compression level based on the current write load. AdaptiveCompressor's goal is to provide similar write performance as LZ4Compressor for write heavy workloads, but a significantly better compression ratio for databases with a moderate amount of writes or on systems with a lot of spare CPU power.

If the memtable flush queue builds up, and it turns out the compression is a significant bottleneck, then the compression level used for flushing is decreased to gain speed. Similarly, when pending compaction tasks build up, then the compression level used for compaction is decreased.

In order to enable adaptive compression:

set -Dcassandra.default_sstable_compression=adaptive JVM option to automatically select AdaptiveCompressor as the main compressor for flushes and new tables, if not overriden by specific options in cassandra.yaml or table schema
set flush_compression: adaptive in cassandra.yaml to enable it for flushing
set AdaptiveCompressor in Table options to enable it for compaction

Caution: this feature is not turned on by default because it
may impact read speed negatively in some rare cases.

Checklist before you submit for review

Make sure there is a PR in the CNDB project updating the Converged Cassandra version
Use NoSpamLogger for log lines that may appear frequently in the logs
Verify test results on Butler
Test coverage for new/modified code is > 80%
Proper code formatting
Proper title for each commit staring with the project-issue number, like CNDB-1234
Each commit has a meaningful description
Each commit is not very long and contains related changes
Renames, moves and reformatting are in distinct commits

blambov · 2024-11-21T11:29:34Z

src/java/org/apache/cassandra/schema/CompressionParams.java

+                                                                          DEFAULT_MIN_COMPRESS_RATIO,
+                                                                          Collections.emptyMap());
+
+    public static final CompressionParams FAST_ADAPTIVE = new CompressionParams(AdaptiveCompressor.createForFlush(Collections.emptyMap()),


I'm not very happy with this name, and generally with the use of "FAST" for "FLUSH" and "GENERAL" for "COMPACTION", but we can't change the enum at this point, can we?

Im not happy either... However if I change the naming, then we should change it for all the things consistently and this is a part of the config (we use "fast" in cassandra.yaml). :(

src/java/org/apache/cassandra/io/compress/AdaptiveCompressor.java

src/java/org/apache/cassandra/io/sstable/format/SortedTableWriter.java

src/java/org/apache/cassandra/io/compress/AdaptiveCompressor.java

blambov · 2024-11-21T12:04:05Z

src/java/org/apache/cassandra/io/compress/AdaptiveCompressor.java

+    @Override
+    public Set<Uses> recommendedUses()
+    {
+        return Set.of(params.uses);


This can use EnumSet.of.

Further, IMHO it makes better sense to return both here; or rather, add a new method to adapt the compressor for a certain use, e.g.

default ICompressor forUse(Uses use) { return recommendedUses.contains(use) ? this : null; }

in ICompressor, overridden to change params and recreate in AdaptiveCompressor.

Actually, such a change is also necessary to preserve encryption for HCD.

+1 to EnumSet.of

However, I don't agree with returning both uses here.
This is because a particular AdaptiveCompressor can be either adapting to "flush pressure" or "compaction pressure" but not both. I don't want to accidentally end up using AdaptiveCompressor configured for compaction (aka GENERAL use) in flush, so it cannot advertise as FAST_COMPRESSION unless configured for flushing.
The use is locked at the moment of creating the compressor. It could be actually two separate compressor classes, each for different use, but because they share like 99% of logic and the only real difference is defaults + pressure source, there is likely no point in separating them.

But on the second thought, indeed, if someone configures flushCompression as table and then AdaptiveCompression is selected in the table schema... this would result in using a wrong source of pressure. So adding this forUse API looks like a good idea to me.

This is precisely why I described an adaptation method: we do want to use a compressor derived from the current one, suitable for the flush usage. This can be done without requiring any changes to existing ICompressor implementations via the default method above, and is important for this patch, but also critical for preserving encryption when DSE moves to this code base.

blambov · 2024-11-26T16:42:18Z

src/java/org/apache/cassandra/utils/ExpMovingAverage.java

+            current = average.get();
+
+            if (!Double.isNaN(current))
+                update = current + Math.pow(alpha, n) * (val - current);


This doesn't appear to agree with the javadoc. The effect of val should be stronger for higher n, not weaker.

E.g. I believe calling update(val) twice results in (1-alpha)^2 * current + (1-(1-alpha)^2) * val, which is not the same as (1 - alpha^2) * current + alpha^2 * val, which this translates to.

Right. I got it reversed.

blambov · 2024-11-26T16:53:33Z

src/java/org/apache/cassandra/utils/concurrent/RateLimiter.java

+ * also measures and exposes the actual rate.
+ */
+@SuppressWarnings("UnstableApiUsage")
+public class RateLimiter


It would be better to port over CASSANDRA-13890 which introduces a compaction metric for the current compaction rate (with a few aggregations, e.g. 1-minute rate).

blambov

I'm curious why not port over the few other changes in CASSANDRA-13890. Has the code diverged too much?

pkolaczk · 2024-11-27T12:53:42Z

I'm curious why not port over the few other changes in CASSANDRA-13890. Has the code diverged too much?

Will take a look. The merge wasn't clean and I didn't initially want to spend time on something that is not essential for this feature.

What was ported: - current compaction throughput measurement by CompactionManager - exposing current compaction throughput in StorageService and CompactionMetrics - nodetool getcompactionthroughput, including tests Not ported: - changes to `nodetool compactionstats`, because that would require porting also the tests which are currently missing in CC and porting those tests turned out to be a complex task without porting the other changes in the CompactionManager API - Code for getting / setting compaction throughput as double

This commit introduces a new AdaptiveCompressor class. AdaptiveCompressor uses ZStandard compression with a dynamic compression level based on the current write load. AdaptiveCompressor's goal is to provide similar write performance as LZ4Compressor for write heavy workloads, but a significantly better compression ratio for databases with a moderate amount of writes or on systems with a lot of spare CPU power. If the memtable flush queue builds up, and it turns out the compression is a significant bottleneck, then the compression level used for flushing is decreased to gain speed. Similarly, when pending compaction tasks build up, then the compression level used for compaction is decreased. In order to enable adaptive compression: - set `-Dcassandra.default_sstable_compression=adaptive` JVM option to automatically select `AdaptiveCompressor` as the main compressor for flushes and new tables, if not overriden by specific options in cassandra.yaml or table schema - set `flush_compression: adaptive` in cassandra.yaml to enable it for flushing - set `AdaptiveCompressor` in Table options to enable it for compaction Caution: this feature is not turned on by default because it may impact read speed negatively in some rare cases. Fixes riptano/cndb#11532

Reduces some overhead of setting up / tearing down those contexts that happened inside the calls to Zstd.compress / Zstd.decompress. Makes a difference with very small chunks. Additionally, added some compression/decompression rate metrics.

sonarqubecloud · 2024-12-18T20:51:14Z

Quality Gate passed

Issues
0 New issues
4 Accepted issues

Measures
0 Security Hotspots
82.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

cassci-bot · 2024-12-18T20:54:37Z

❌ Build ds-cassandra-pr-gate/PR-1432 rejected by Butler

1 new test failure(s) in 16 builds
See build details here

Found 1 new test failures

Test	Explanation	Branch history	Upstream history
...ToolEnableDisableBinaryTest.testMaybeChangeDocs	regression	🔴🔵🔵🔴🔴🔵🔵	🔵🔵🔵🔵🔵🔵🔵

Found 120 known test failures

pkolaczk requested a review from blambov November 20, 2024 15:27

blambov reviewed Nov 21, 2024

View reviewed changes

pkolaczk requested a review from blambov November 22, 2024 13:00

blambov approved these changes Nov 22, 2024

View reviewed changes

pkolaczk force-pushed the c11532-adaptive-compression branch 3 times, most recently from 1806608 to 9bdc785 Compare November 26, 2024 10:01

blambov reviewed Nov 26, 2024

View reviewed changes

pkolaczk force-pushed the c11532-adaptive-compression branch from 787932c to f2bfb36 Compare November 27, 2024 09:16

blambov approved these changes Nov 27, 2024

View reviewed changes

pkolaczk force-pushed the c11532-adaptive-compression branch from f2bfb36 to 7222496 Compare November 27, 2024 12:14

pkolaczk force-pushed the c11532-adaptive-compression branch 4 times, most recently from a38fa30 to 9efb5fa Compare December 3, 2024 08:46

maoling and others added 2 commits December 16, 2024 17:26

pkolaczk force-pushed the c11532-adaptive-compression branch from 9efb5fa to 90da921 Compare December 16, 2024 16:27

Reuse ZStd compression/decompression context

e6baaee

Reduces some overhead of setting up / tearing down those contexts that happened inside the calls to Zstd.compress / Zstd.decompress. Makes a difference with very small chunks. Additionally, added some compression/decompression rate metrics.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CNDB-11532: Adaptive compression #1432

CNDB-11532: Adaptive compression #1432

pkolaczk commented Nov 20, 2024 •

edited

Loading

blambov Nov 21, 2024

pkolaczk Nov 22, 2024

blambov Nov 21, 2024

blambov Nov 21, 2024

pkolaczk Nov 22, 2024

pkolaczk Nov 22, 2024

blambov Nov 22, 2024

blambov Nov 26, 2024

pkolaczk Nov 26, 2024

blambov Nov 26, 2024

blambov left a comment

pkolaczk commented Nov 27, 2024

sonarqubecloud bot commented Dec 18, 2024

cassci-bot commented Dec 18, 2024

CNDB-11532: Adaptive compression #1432

Are you sure you want to change the base?

CNDB-11532: Adaptive compression #1432

Conversation

pkolaczk commented Nov 20, 2024 • edited Loading

Checklist before you submit for review

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blambov left a comment

Choose a reason for hiding this comment

pkolaczk commented Nov 27, 2024

sonarqubecloud bot commented Dec 18, 2024

Quality Gate passed

cassci-bot commented Dec 18, 2024

❌ Build ds-cassandra-pr-gate/PR-1432 rejected by Butler

Found 1 new test failures

Found 120 known test failures

pkolaczk commented Nov 20, 2024 •

edited

Loading