Add VM informant

[sharnoff]

Currently, this mostly does nothing. The protocol is designed so that it'll be easy to add to it later, which I'll do shortly.

Remaining TODO items before merging:

• agent must check with the informant before downscaling and after upscaling (0bb61ba)
• ARCHITECTURE.md needs updating (3c40d14, 8389afe)

[sharnoff]

I think this is ready to merge. It's now clear to me that there's some heavy refactoring needed in pkg/agent/run.go (and .../serve.go, added by this PR), so that the existing long-running tasks can share state & signals — in addition to the work required by #5 (which I think is mostly separate and could be done in parallel).

While the refactoring in pkg/agent/{run,serve}.go is a prerequisite to further work, I think it's more important to merge this as-is & get CI-built vm-informant binaries (see #6) into the compute images, so that we can iterate on something ­— especially considering that this autoscaler-agent requires the informant to be present, so we already have to wait to deploy it until after the updated storage images are built.

[kelvich]

Hm, do we actually need a new binary inside? Not saying that we shouldn't, but trying to understand the list of reasons.

Vector now provides metrics that agent can read, and we also can embed some UDF's in postgres to up/down scale things and agent can call them. That leaves us with postgres -> scheduler requests on spiky load. Is there anything else?

[sharnoff]

@kelvich

Is there anything else?

Short answer: not yet. The key benefits from having a binary inside, as I see it:

1. We can handle "memory high" events along the lines of Christian's proposal with cgroups
2. Calling postgres UDF(s) is through localhost (easier to harden postgres auth rules later)
3. A binary inside has the flexibility to supply metrics that vector might not
4. [not sure] Better tracking of disk usage? We want to track this as closely as possible so we can handle "low disk space" in the same way as (1). It's not clear to me whether this is required. Related to (3).

These are all directions for future work — none of this is implemented yet, although (1) is planned and (3) is tenative. (4) may be required.

[kelvich]

👌