Service ports: expose customization for multi-tenancy

[vsoch]

Hi @AkihiroSuda ! We are prototyping / preparing for a deployment of user space Kubernetes on our on-premises clusters this fiscal here. In addition to a Python wrapper that (so far) is working on AWS (headless) with flux, either as a system instance, or setup / teardown in a job:

$ flux run -N2 --setattr=attributes.user.usernetes=yes /bin/bash batch.sh 

flux-job: ƒ3SwLtzHu started                                                                                                                                                            00:02:03
Found kubeconfig at /tmp/usernetes-ƒ3swltzhu/kubeconfig
NAME                      STATUS   ROLES           AGE   VERSION
u7s-i-0348057d85534bdeb   Ready    <none>          31s   v1.31.2
u7s-i-056b730dd06cee178   Ready    control-plane   42s   v1.31.2

we are going to want to do the same on a system with multi-tenancy. This means that everyone using the same ports won't fly - and actually even the same user using the same ports won't! So I wanted to discuss / brainstorm with you about ways we can customize the various services to ensure they don't conflict. In HPC we often choose very high numbers too. Thank you!

Proposed plan: tackle each in separate pull request.

Items to do

• kube-apiserver and refactor of testing to handle multiple cases of multi-node feat: customization of exposed service ports #346
• etcd 10ce86f
• flannel (Not going to do)
• kubelet ea48bba
• testing setup that affords customization of nodes for single and multi-node tests https://github.com/rootless-containers/usernetes/actions/runs/12219828578

@AkihiroSuda you can assign this issue to me - I can cover the remaining three bullets. I can also do #347.

[AkihiroSuda]

The ports of kube-apiserver, kubelet, etc. can be customized via https://github.com/rootless-containers/usernetes/blob/master/docker-compose.yaml and https://github.com/rootless-containers/usernetes/blob/master/kubeadm-config.yaml

I agree that these ports should be configurable, probably via env vars

[vsoch]

Yes exactly! Do you want me to take a shot at a PR this week / weekend?

[AkihiroSuda]

Yes, thanks

[vsoch]

Awesome! Ping @milroy - let's discuss what other envars we might want to tweak this week for our setup, and I'll follow up here with a pull request to continue discussion.

Thank you @AkihiroSuda 🙏

[vsoch]

All set!

#346

And I had a follow up question in #347 - if there is a means to do that I can open a second PR (and do additional testing for it). We have tested connecting a control plane from EKS with a node they provide, but haven't tested Usernetes on EC2 with the same node (or a different strategy).

[vsoch]

@AkihiroSuda I've finished work for this issue, and the PR #346 is ready for review, and checkboxes in the description above updated! Specifically:

• We expose (and test) service ports for the kubelet, kube-apiserver, and etcd.
• This required some redundancy of testing, so I refactored to add single and multi-node reusable templates
• I decided to not expose flannel, because it's based on vxlan, and that's a Linux default.

For the last point, I read about the setup and my naive instinct is that we should not be customizing such a default, at least until there is a compelling need (e.g., different default on a different platform). If you'd like to do this I can follow your guidance, either for this PR or a later one.

Thanks again for the review! I learned a lot working on this and appreciate it.

[AkihiroSuda]

VXLAN ports have to be changeable for supporting multi-tenancy

[vsoch]

VXLAN ports have to be changeable for supporting multi-tenancy

Gotcha. Please see my questions in #346 (review). I think we have two options:

1. Expose the ports for customization, assuming that someone doing so will know how to configure the vxlan.
2. Also vendor the flannel yaml here, with the added Port variable customizable from the environment.

If we need to test this, I need to ask for your guidance on how to do that. Note that the current failure in the test looks like a flaky download for the fedora lima image, we can re-run it when the rest finish.

[vsoch]

Flannel is added, and commits squashed #346.

The helm charts don't expose the variables we need (customizing args and the Port) so we clone the repository with the helm chart, write our custom values.yaml (with the changes we need) and then install from that. I thought we might need more system level changes, but it seems to work with that.