Operator assumes default cluster domain (cluster.local) is used

[flostadler]

What happened?

The operator does not work in kubernetes clusters that are not using the default cluster.local domain. The controller cannot connect to workspace pods when a custom domain is used.
The root cause of this is that cluster.local is hardcoded here:

pulumi-kubernetes-operator/operator/internal/controller/auto/workspace_controller.go

Line 398 in b7c71da

return fmt.Sprintf("%s.%s.svc.cluster.local", nameForService(w), w.Namespace)

Example

2024-11-25T15:55:21.126Z	INFO	Connecting to workspace pod	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"stack-of-stacks-70ab85b5","namespace":"pulumi-stacks"}, "namespace": "pulumi-stacks", "name": "stack-of-stacks-70ab85b5", "reconcileID": "3d0a3e3c-a604-4a5f-9c16-225ba6ed88df", "revision": "71012", "addr": "stack-of-stacks-70ab85b5-workspace.pulumi-stacks.svc.cluster.local:50051"}
2024-11-25T15:55:21.135Z	ERROR	unable to connect; retrying later	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"stack-of-stacks-70ab85b5","namespace":"pulumi-stacks"}, "namespace": "pulumi-stacks", "name": "stack-of-stacks-70ab85b5", "reconcileID": "3d0a3e3c-a604-4a5f-9c16-225ba6ed88df", "revision": "71012", "addr": "stack-of-stacks-70ab85b5-workspace.pulumi-stacks.svc.cluster.local:50051", "error": "unable to connect to workspace: TRANSIENT_FAILURE"}
github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile
	/go/operator/internal/controller/auto/workspace_controller.go:229
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222

It's trying to connect to stack-of-stacks-70ab85b5-workspace.pulumi-stacks.svc.cluster.local:50051, but the correct address would be stack-of-stacks-70ab85b5-workspace.pulumi-stacks.svc.local.lab.flostadler.de:50051

Output of pulumi about

n/a

Additional context

Operator version v2.0.0-beta.2

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[EronWright]

I believe that a reasonable solution is to use the shorter DNS name, e.g. guestbook-workspace.default rather than guestbook-workspace.default.svc.cluster.local, per DNS for Services and Pods.