storage controller: tenant creation sometimes disagrees with background optimization #8969
Assignees
Labels
Comments
jcsp
added
a/tech_debt
5 tasks
VladLazar
added a commit
that referenced
this issue
Sep 24, 2024
## Problem Scheduling on tenant creation uses different heuristics compared to the scheduling done during background optimizations. This results in scenarios where shards are created and then immediately migrated by the optimizer. ## Summary of changes 1. Make scheduler aware of the type of the shard it is scheduling (attached vs secondary). We wish to have different heuristics. 2. For attached shards, include the attached shard count from the context in the node score calculation. This brings initial shard scheduling in line with what the optimization passes do. 3. Add a test for (2). This looks like a bigger change than required, but the refactoring serves as the basis for az-aware shard scheduling where we also need to make the distinction between attached and secondary shards. Closes #8969
|
I think our fix isn't quite complete, I'm analyzing a log where this happened again... |
|
There's a unit test that reproduces the unstable case in #9275 |
This was referenced Nov 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Tenant creation uses
schedule_shard, which chooses affinity scores based on the absolute number of shards that a tenant has on a node, treating attached and secondary locations the same.However, the optimize_all logic prefers to move attached locations away from nodes that have lots of existing attachments for the same tenant.
This can lead to a situation where we create a tenant, and then immediately start migrating one of its shards, because several attached locations for the same tenant were already on that node.
We can fix this by modifying schedule_shard to know whether it is scheduling an attached or secondary location, and if scheduling an attached location then include ScheduleContext::attached_nodes in the affinity score calculation.
This will probably get solved implicitly when implementing AZ-aware scheduling, as that will also need to distinguish attached vs. secondary locations.
The text was updated successfully, but these errors were encountered: