warnet-rpc as a k8s custom resource definition

[josibake]

Today, we run a single RPC server in the cluster that manages all scenario activity. My concerns are:

• The RPC server is a central point of failure and can only run with as much resources as the largest node in the cluster (e.g., 2vcpu, 4gb ram)
• Its a bit unclear how a single RPC server can be used by multiple "teams" , e.g., red team vs blue team, but both going through rpc-o
• Seems difficult to recover if the central RPC server falls over

An alternative would be to create the RPC server as a K8s Custom Resource Definition, were rpc-0 can spin up "Scenario" resources as needed, and then these Scenario resources can manage a subset (up to all) nodes in the cluster. They also manage cleaning themselves up , recovering from failure, etc.

Generally speaking, trying to manage things in a more Kubernetes native way seems the best path forward.