Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a throttled logger for exceeded memory warnings #15424

Merged
merged 3 commits into from
Mar 26, 2024
Merged

Use a throttled logger for exceeded memory warnings #15424

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
eecffd2
feat: create a throttled logger and use that for exeeced memory warnings
GuptaManan100 Mar 8, 2024
1b6c139
return a mysql warning
frouioui Mar 12, 2024
d27dec5
fix typo
frouioui Mar 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions go/vt/vtgate/executor.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,15 @@
"vitess.io/vitess/go/acl"
"vitess.io/vitess/go/cache/theine"
"vitess.io/vitess/go/mysql/collations"
"vitess.io/vitess/go/mysql/sqlerror"
"vitess.io/vitess/go/sqltypes"
"vitess.io/vitess/go/stats"
"vitess.io/vitess/go/streamlog"
"vitess.io/vitess/go/trace"
"vitess.io/vitess/go/vt/callerid"
"vitess.io/vitess/go/vt/key"
"vitess.io/vitess/go/vt/log"
"vitess.io/vitess/go/vt/logutil"
binlogdatapb "vitess.io/vitess/go/vt/proto/binlogdata"
querypb "vitess.io/vitess/go/vt/proto/query"
topodatapb "vitess.io/vitess/go/vt/proto/topodata"
Expand Down Expand Up @@ -73,6 +75,8 @@

queriesProcessedByTable = stats.NewCountersWithMultiLabels("QueriesProcessedByTable", "Queries processed at vtgate by plan type, keyspace and table", []string{"Plan", "Keyspace", "Table"})
queriesRoutedByTable = stats.NewCountersWithMultiLabels("QueriesRoutedByTable", "Queries routed from vtgate to vttablet by plan type, keyspace and table", []string{"Plan", "Keyspace", "Table"})

exceedMemoryRowsLogger = logutil.NewThrottledLogger("ExceedMemoryRows", 1*time.Minute)
)

const (
Expand Down Expand Up @@ -231,7 +235,12 @@
if err != nil {
piiSafeSQL = logStats.StmtType
}
log.Warningf("%q exceeds warning threshold of max memory rows: %v. Actual memory rows: %v", piiSafeSQL, warnMemoryRows, len(result.Rows))
warningMsg := fmt.Sprintf("%q exceeds warning threshold of max memory rows: %v. Actual memory rows: %v", piiSafeSQL, warnMemoryRows, len(result.Rows))
exceedMemoryRowsLogger.Warningf(warningMsg)
safeSession.RecordWarning(&querypb.QueryWarning{
Code: uint32(sqlerror.EROutOfMemory),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if EROutOfMemory is appropriate, we can also remove the Code altogether

Copy link
Member

@deepthi deepthi Mar 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be fine, because the warning we return to the client is descriptive.

Message: warningMsg,
})
}

logStats.SaveEndTime()
Expand Down Expand Up @@ -365,7 +374,12 @@
if err != nil {
piiSafeSQL = logStats.StmtType
}
log.Warningf("%q exceeds warning threshold of max memory rows: %v. Actual memory rows: %v", piiSafeSQL, warnMemoryRows, srr.rowsReturned)
warningMsg := fmt.Sprintf("%q exceeds warning threshold of max memory rows: %v. Actual memory rows: %v", piiSafeSQL, warnMemoryRows, srr.rowsReturned)
exceedMemoryRowsLogger.Warningf(warningMsg)
safeSession.RecordWarning(&querypb.QueryWarning{
Code: uint32(sqlerror.EROutOfMemory),
Message: warningMsg,
})

Check warning on line 382 in go/vt/vtgate/executor.go

View check run for this annotation

Codecov / codecov/patch

go/vt/vtgate/executor.go#L377-L382 

Added lines #L377 - L382 were not covered by tests
}

logStats.SaveEndTime()
Expand Down
Loading