grafana: Add gRPC Received commands rate panel
#8921
Conversation
Signed-off-by: okJiang <[email protected]>
ti-chi-bot
bot
added
release-note-none
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
okJiang
changed the title
gRPC Received commands rategRPC Received commands rate panel
ti-chi-bot
bot
added
the
size/L
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #8921 +/- ##
==========================================
+ Coverage 76.20% 76.31% +0.11%
==========================================
Files 461 462 +1
Lines 70448 70447 -1
==========================================
+ Hits 53686 53765 +79
+ Misses 13402 13336 -66
+ Partials 3360 3346 -14
Flags with carried forward coverage won't be shown. Click here to find out more. |
metrics/grafana/pd.json
Outdated
| "spaceLength": 10, | ||
| "targets": [ | ||
| { | ||
| "expr": "sum(rate(grpc_server_started_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\"}[1m])) by (instance, grpc_method)", |
There was a problem hiding this comment.
How about also adding the error rate?
There was a problem hiding this comment.
I added it. But btw, it can't catch the error hide in the header, like this
Line 2071 in 2d970a6
There was a problem hiding this comment.
Please add a comment or raise an issue for further improvement.
Signed-off-by: okJiang <[email protected]>
| "spaceLength": 10, | ||
| "targets": [ | ||
| { | ||
| "expr": "sum(rate(grpc_server_handled_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", grpc_type=\"unary\", grpc_code!=\"OK\"}[1m])) by (grpc_method)", |
There was a problem hiding this comment.
How about adding one expr as unary total about ignoring grpc_code?
There was a problem hiding this comment.
grpc_server_handled_total only records the count after handling. If we want to record the receive msg count, we can use grpc_server_msg_received_total, but it does not record grpc_code label. So we can't record the error rate of the stream, no grpc native metrics we can use. https://github.com/grpc-ecosystem/go-grpc-middleware/blob/66bb7d8818ff22b450b82111a1260ca6dfe3486c/providers/prometheus/server_metrics.go#L27-L53
Signed-off-by: okJiang <[email protected]>
|
/retest |
Signed-off-by: okJiang <[email protected]>
…g/pd into add-grpc-received-metrics
|
/retest |
|
ping @rleungx @bufferflies |
What problem does this PR solve?
Issue Number: Close #8920
What is changed and how does it work?
Check List
Tests
Release note