Epic: Standard interface for running containers AND/OR systemd services in VMs

[sharnoff]

Split off from #577.

Motivation

• vm-builder must make hacky modifications to files (see e.g. /neonvm/bin/...) to avoid conflicting with the root filesystem supplied by the user's image
• the "image spec" yaml file is hacky, dockerfile-in-yaml with a separate "merge" step to combine images into a single root filesystem
• it's currently difficult to limit CPU usage of postgres inside the VM
• logs routing is annoying, and maybe systemd can help here; see Epic: Separately tagged logs for VM processes, dmesg, and runner  #578
• if a user breaks out of postgres, they can freely disrupt neonvm internal programs running in the VM

DoD

1. Use systemd services and/or containers to run programs inside the VM
2. Maybe expose separate configuration for these in the VM spec, to allow easier parameterization by control plane
3. Maybe allow "delayed start" so e.g. compute_ctl can start a daemon-ish program only after a certain point.

Implementation ideas

There's kind of two approaches we've discussed for the internal details:

1. Via containers running inside the VM
   • i.e.: vm-builder loads docker images, exports to tarball, unpacks into VM image; on startup, we use e.g. runc to run the images stored there
2. Via systemd "services"
   • i.e.: vm-builder still puts things into the same root filesystem, but systemd is responsible for running the various daemons. We can also allow compute_ctl to "start" a service (e.g. pgbouncer) so that it's not running until postgres is.
   • See also Switch NeonVMs to systemd #728
   • Potentially helps a lot with Epic: Separately tagged logs for VM processes, dmesg, and runner  #578

We also could use some combination of the two, e.g. using systemd to runc run the containers as a service. We also discussed that systemd maybe has a way to run containers natively? also an option.

Tasks

Tasks

Preview Give feedback

1. Change vm-builder to run daemons as containers and/or systemd services
2. Maybe add support to VM spec to change env/args to containers at runtime
3. Maybe support "delayed start" of dameons

Other related tasks, Epics, and links

• Switch NeonVMs to systemd #728
• Epic: Separately tagged logs for VM processes, dmesg, and runner  #578