control-plane: use btree index for log_lines #1197
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
Replaces the current BRIN index on the
internal.log_lines
table with a btree index. The BRIN index would require queries to use thelogged_at
timestamp in the where clause in order for it to be effective. Neither the UI or flowctl do that currently. The btree index works well when only filtering on token, so this changes to just using that.Notes for reviewers:
The main issue this is seeking to address is that clients are getting HTTP 500 responses when they try to fetch log lines during a publication or discover. Another option that might also work to address this would be to add
where logged_at > (now() - '1h'::interval)
to the query in theview_logs
RPC. I decided to use the btree index instead because it felt somewhat less hacky.BUT, we may still want a brin index on (only)
logged_at
, so that we can use it in apg_cron
job to clean out old logs. I've opted to ignore this for the time being, and address that in a future PR along with the cron job to delete the logs.This change is