Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(query): Add metrics for failover #1856

Merged
merged 1 commit into from
Sep 26, 2024

Conversation

kvpetrov
Copy link
Contributor

@kvpetrov kvpetrov commented Sep 26, 2024

Pull Request checklist

  • The commit(s) message(s) follows the contribution guidelines ?
  • Tests for the changes have been added (for bug fixes / features) ?
  • Docs have been added / updated (for bug fixes / features) ?

Current behavior : (link exiting issues here : https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests)

If query server fails over to the buddy cluster because local FiloDB clusters have shard(s) down no metric is emitted.

New behavior :
Now we emit 4 metrics:

  • legacy failover
  • full failover with local planning
  • partial failover (shard level failover) with local planning
  • failure to execute the query because no shards are available

BREAKING CHANGES
N/A

@kvpetrov kvpetrov changed the title adding metrics for failover feat(query): Add metrics for failover Sep 26, 2024
Copy link
Contributor

@alextheimer alextheimer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with one comment (ignore it if the suggested log already exists)

@@ -360,6 +384,7 @@ class HighAvailabilityPlanner(dsRef: DatasetRef,
buddyGrpcEndpoint = Some(remoteGrpcEndpoint.get)
)
val context = qContext.copy(plannerParams = haPlannerParams)
logger.info(context.getQueryLogLine("Using shard level failover"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be useful to have lists of shards such as:

Materializing locally: [0, 1, 2, 3]
Materializing on buddy: [4, 5, 6, 7]

I know technically that info exists in the plan structure, but this would be a nice quick reference to have.

@kvpetrov kvpetrov merged commit f9cbbcf into filodb:develop Sep 26, 2024
1 check passed
kvpetrov added a commit to kvpetrov/FiloDB that referenced this pull request Sep 27, 2024
kvpetrov added a commit to kvpetrov/FiloDB that referenced this pull request Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants