Epic: Scaling latency metrics

[sharnoff]

Motivation

Two reasons:

1. We're currently flying blind w.r.t. how long scaling takes
2. Scaling latency should be part of any autoscaling SLOs

DoD

We should have histogram metrics recording:

• end-to-end latency of scaling (down and up; cpu and memory)
• latency of all the components:
  • requests to scheduler plugin (including retries)
  • requests to vm-monitor (including retries)
  • delay between initial NeonVM patch request and when status was updated

Implementation ideas

AFAICT the basic idea is that we store some extra info in agent/core.State and add some extra callbacks in agent/core.Config to increment some metrics when we determine that various parts of scaling (and the entire thing) have occurred.

More design work is required, because the edge cases are quite subtle.

Tasks

Blockers

Preview Give feedback

1. Bug: VM memory scale-up leaves "Scaling" phase before updating status resources #453
   c/autoscaling/neonvm t/bug +2
   
2. Bug: Fractional CPU scale-down temporarily exceeds spec maximum #462
   c/autoscaling/neonvm t/bug +2
   
3. Bug: pkg/agent/state should use VM status, not spec to represent "current" resources #592
   1 of 2
   c/autoscaling/autoscaler-agent t/bug +2

Implementation

Preview Give feedback

1. Decide how we should measure latency (difficult due to eventual consistency)
2. agent/core: Measure scaling latency

Follow-ups

Preview Give feedback

1. agent: Plugin/NeonVM/Monitor metrics with retries #1012
   +0

Other related tasks, Epics, and links

• Proposed RFC

[sharnoff]

Latency metrics measured by pkg/agent/core.State would have allowed us to notice the effects of #614 (i.e. core.State believed there was a super long-running request).

[sharnoff]

Status: waiting on @sharnoff and @stradig to review the internal RFC.