Service ports: expose customization for multi-tenancy?

[vsoch]

Hi @AkihiroSuda ! We are prototyping / preparing for a deployment of user space Kubernetes on our on-premises clusters this fiscal here. In addition to a Python wrapper that (so far) is working on AWS (headless) with flux, either as a system instance, or setup / teardown in a job:

$ flux run -N2 --setattr=attributes.user.usernetes=yes /bin/bash batch.sh 

flux-job: ƒ3SwLtzHu started                                                                                                                                                            00:02:03
Found kubeconfig at /tmp/usernetes-ƒ3swltzhu/kubeconfig
NAME                      STATUS   ROLES           AGE   VERSION
u7s-i-0348057d85534bdeb   Ready    <none>          31s   v1.31.2
u7s-i-056b730dd06cee178   Ready    control-plane   42s   v1.31.2

we are going to want to do the same on a system with multi-tenancy. This means that everyone using the same ports won't fly - and actually even the same user using the same ports won't! So I wanted to discuss / brainstorm with you about ways we can customize the various services to ensure they don't conflict. In HPC we often choose very high numbers too. Thank you!

[AkihiroSuda]

The ports of kube-apiserver, kubelet, etc. can be customized via https://github.com/rootless-containers/usernetes/blob/master/docker-compose.yaml and https://github.com/rootless-containers/usernetes/blob/master/kubeadm-config.yaml

I agree that these ports should be configurable, probably via env vars

[vsoch]

Yes exactly! Do you want me to take a shot at a PR this week / weekend?

[AkihiroSuda]

Yes, thanks

[vsoch]

Awesome! Ping @milroy - let's discuss what other envars we might want to tweak this week for our setup, and I'll follow up here with a pull request to continue discussion.

Thank you @AkihiroSuda 🙏