Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[server] Fix NPE when key value size profiling is not enabled #1378

Closed
wants to merge 1 commit into from
Closed

[server] Fix NPE when key value size profiling is not enabled #1378

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions services/venice-server/src/main/java/com/linkedin/venice/stats/ServerHttpRequestStats.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -445,11 +445,15 @@ public void recordEarlyTerminatedEarlyRequest() {
}

public void recordKeySizeInByte(int keySize) {
requestKeySizeSensor.record(keySize);
if (requestKeySizeSensor != null) {
requestKeySizeSensor.record(keySize);
}
Comment on lines -448 to +450
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... but it's weird though... why is this sensor null?

The server config should control both whether this sensor is null, and also whether this function gets called or not 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean is:

This function is only called from MultiGetResponseStatsWithSizeProfiling (and the equivalent compute-related class). And MultiGetResponseStatsWithSizeProfiling can only get constructed when serverConfig.isKeyValueProfilingEnabled() is true...

And AggServerHttpRequestStats also gets passed the same config, which (after a few indirections) should determine whether that sensor gets created or not.

So how can these be mismatched?

This comment was marked as outdated.

Sign in to view

Copy link
Collaborator Author

@sushantmane sushantmane Dec 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's coming from the following stack trace.

java.lang.NullPointerException: Cannot invoke "io.tehuti.metrics.Sensor.record(double)" because "this.requestValueSizeSensor" is null
	at com.linkedin.venice.stats.ServerHttpRequestStats.recordValueSizeInByte(ServerHttpRequestStats.java:452) ~[venice-server.jar:?]
	at com.linkedin.venice.listener.response.stats.ResponseStatsUtil.consumeIntIfAbove(ResponseStatsUtil.java:22) ~[venice-server.jar:?]
	at com.linkedin.venice.listener.response.stats.SingleGetResponseStats.recordMetrics(SingleGetResponseStats.java:30) ~[venice-server.jar:?]
	at com.linkedin.venice.listener.ServerStatsContext.recordBasicMetrics(

In SingleGetResponseStats we've this method:

  @Override
  public void recordMetrics(ServerHttpRequestStats stats) {
    super.recordMetrics(stats);

    ResponseStatsUtil.consumeIntIfAbove(stats::recordValueSizeInByte, this.valueSize, 0);
    stats.recordKeySizeInByte(this.keySize);
  }

IMO in the above NPE case SingleGetResponseStats::valueSize is set from one of these places where we don't check size profiling config is enabled or not:

  1. venice/clients/da-vinci-client/src/main/java/com/linkedin/davinci/storage/chunking/ChunkingUtils.java

    Line 326 in 6111853

    responseStats.addValueSize(valueLength);

  2. venice/clients/da-vinci-client/src/main/java/com/linkedin/davinci/storage/chunking/ChunkingUtils.java

    Line 382 in 6111853

    responseStats.addValueSize(actualSize);

Copy link
Collaborator Author

@sushantmane sushantmane Dec 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is regression due to: #1377. It assumed that single get request stats has similar logic to multi-key requests

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about that. Any idea why this wasn't caught in any of the tests?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah, right. I see it now: #1377 (comment)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the motivation to disable KV size profiling for single gets?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Metric bloat maybe? IDK. @nisargthakkar ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any idea why this wasn't caught in any of the tests?

Probably we don't have E2E a test which verifies all metrics for single-get request type?

}

public void recordValueSizeInByte(int valueSize) {
requestValueSizeSensor.record(valueSize);
if (requestValueSizeSensor != null) {
requestValueSizeSensor.record(valueSize);
}
}

public void recordMisroutedStoreVersionRequest() {
Expand Down
Loading