Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set some default resource requests on the workspace pod #707

Merged
merged 14 commits into from
Oct 7, 2024
Merged

Set some default resource requests on the workspace pod #707

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
928cbab
bugfix for fileserver
EronWright Oct 2, 2024
c24b663
use tini for pulumi container
EronWright Oct 2, 2024
14764ef
use tini for all containers
EronWright Oct 2, 2024
fda6898
use workspace dir as working dir
EronWright Oct 2, 2024
72c9eb0
set agent memory limit
EronWright Oct 3, 2024
c48402c
default workspace resources
EronWright Oct 4, 2024
796f41c
remove webhook scaffolding
EronWright Oct 4, 2024
f4f2f9a
linting
EronWright Oct 4, 2024
ac134bd
fix copyright
EronWright Oct 7, 2024
ecb323c
remove unnecessary code
EronWright Oct 7, 2024
ea811b6
Update API docs for securityProfile
EronWright Oct 7, 2024
8008704
fix for make run
EronWright Oct 7, 2024
d5d708c
add port to advertised address
EronWright Oct 7, 2024
2fb59f1
bugfix for advertised address
EronWright Oct 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Build a base image with modules cached.
FROM --platform=${BUILDPLATFORM} golang:1.23 AS base
ARG TARGETARCH

# Install tini to reap zombie processes.
ENV TINI_VERSION=v0.19.0
ADD --chmod=755 https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static-${TARGETARCH} /tini

COPY /go.mod go.mod
COPY /go.sum go.sum
Expand Down Expand Up @@ -37,8 +42,11 @@ RUN --mount=type=cache,target=${GOCACHE} \
# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
FROM gcr.io/distroless/static-debian12:debug-nonroot

COPY --from=base /tini /tini
COPY --from=op-builder /manager /manager
COPY --from=agent-builder /agent /agent
USER 65532:65532

ENTRYPOINT ["/manager"]
ENTRYPOINT ["/tini", "--"]
CMD ["/manager"]
EronWright marked this conversation as resolved.
Show resolved Hide resolved
11 changes: 11 additions & 0 deletions agent/cmd/serve.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (
"os"
"os/signal"
"path/filepath"
"runtime/debug"
"syscall"

"github.com/pulumi/pulumi-kubernetes-operator/v2/agent/pkg/server"
Expand All @@ -30,6 +31,7 @@ import (
"github.com/spf13/cobra"
"go.uber.org/zap"
"go.uber.org/zap/zapio"
"k8s.io/apimachinery/pkg/api/resource"
)

var (
Expand All @@ -52,6 +54,15 @@ var serveCmd = &cobra.Command{
log.Infow("Pulumi Kubernetes Agent", "version", version.Version)
log.Debugw("executing serve command", "WorkDir", _workDir)

// limit the agent's memory usage to the configured quantity (e.g. 64Mi)
if limit, ok := os.LookupEnv("AGENT_MEMLIMIT"); ok {
val := resource.MustParse(limit)
if !val.IsZero() {
log.Debugf("setting memory limit to %s", limit)
debug.SetMemoryLimit(val.Value())
}
}

// open the workspace using auto api
workspaceOpts := []auto.LocalWorkspaceOption{}
workDir, err := filepath.EvalSymlinks(_workDir) // resolve the true location of the workspace
Expand Down
8 changes: 6 additions & 2 deletions deploy/crds/auto.pulumi.com_workspaces.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8413,8 +8413,12 @@ spec:
type: object
securityProfile:
default: restricted
description: SecurityProfile applies a security profile to the workspace,
'restricted' by default.
description: |-
SecurityProfile applies a security profile to the workspace.
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
the Restricted policy of the Pod Security Standards.
The baseline profile runs the pod as the root user and with a security context that conforms with
the Baseline policy of the Pod Security Standards.
type: string
serviceAccountName:
default: default
Expand Down
16 changes: 12 additions & 4 deletions deploy/crds/pulumi.com_stacks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9334,8 +9334,12 @@ spec:
type: object
securityProfile:
default: restricted
description: SecurityProfile applies a security profile to
the workspace, 'restricted' by default.
description: |-
SecurityProfile applies a security profile to the workspace.
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
the Restricted policy of the Pod Security Standards.
The baseline profile runs the pod as the root user and with a security context that conforms with
the Baseline policy of the Pod Security Standards.
type: string
serviceAccountName:
default: default
Expand Down Expand Up @@ -18871,8 +18875,12 @@ spec:
type: object
securityProfile:
default: restricted
description: SecurityProfile applies a security profile to
the workspace, 'restricted' by default.
description: |-
SecurityProfile applies a security profile to the workspace.
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
the Restricted policy of the Pod Security Standards.
The baseline profile runs the pod as the root user and with a security context that conforms with
the Baseline policy of the Pod Security Standards.
type: string
serviceAccountName:
default: default
Expand Down
8 changes: 6 additions & 2 deletions deploy/helm/pulumi-operator/crds/auto.pulumi.com_workspaces.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8413,8 +8413,12 @@ spec:
type: object
securityProfile:
default: restricted
description: SecurityProfile applies a security profile to the workspace,
'restricted' by default.
description: |-
SecurityProfile applies a security profile to the workspace.
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
the Restricted policy of the Pod Security Standards.
The baseline profile runs the pod as the root user and with a security context that conforms with
the Baseline policy of the Pod Security Standards.
type: string
serviceAccountName:
default: default
Expand Down
16 changes: 12 additions & 4 deletions deploy/helm/pulumi-operator/crds/pulumi.com_stacks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9334,8 +9334,12 @@ spec:
type: object
securityProfile:
default: restricted
description: SecurityProfile applies a security profile to
the workspace, 'restricted' by default.
description: |-
SecurityProfile applies a security profile to the workspace.
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
the Restricted policy of the Pod Security Standards.
The baseline profile runs the pod as the root user and with a security context that conforms with
the Baseline policy of the Pod Security Standards.
type: string
serviceAccountName:
default: default
Expand Down Expand Up @@ -18871,8 +18875,12 @@ spec:
type: object
securityProfile:
default: restricted
description: SecurityProfile applies a security profile to
the workspace, 'restricted' by default.
description: |-
SecurityProfile applies a security profile to the workspace.
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
the Restricted policy of the Pod Security Standards.
The baseline profile runs the pod as the root user and with a security context that conforms with
the Baseline policy of the Pod Security Standards.
type: string
serviceAccountName:
default: default
Expand Down
5 changes: 2 additions & 3 deletions deploy/helm/pulumi-operator/templates/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,12 @@ spec:
{{- toYaml .Values.extraSidecars | nindent 8 }}
{{- end}}
- name: manager
command:
- /manager
args:
- /manager
- --leader-elect
- --health-probe-bind-address=:8081
- --metrics-bind-address=:8383
- --program-fs-adv-addr=pulumi-kubernetes-operator.$(POD_NAMESPACE).svc.cluster.local
- --program-fs-adv-addr=pulumi-kubernetes-operator.$(POD_NAMESPACE).svc.cluster.local:80
- --zap-log-level={{ .Values.controller.logLevel }}
- --zap-time-encoding=iso8601
env:
Expand Down
29 changes: 20 additions & 9 deletions deploy/yaml/install.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9569,8 +9569,12 @@ spec:
type: object
securityProfile:
default: restricted
description: SecurityProfile applies a security profile to
the workspace, 'restricted' by default.
description: |-
SecurityProfile applies a security profile to the workspace.
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
the Restricted policy of the Pod Security Standards.
The baseline profile runs the pod as the root user and with a security context that conforms with
the Baseline policy of the Pod Security Standards.
type: string
serviceAccountName:
default: default
Expand Down Expand Up @@ -19106,8 +19110,12 @@ spec:
type: object
securityProfile:
default: restricted
description: SecurityProfile applies a security profile to
the workspace, 'restricted' by default.
description: |-
SecurityProfile applies a security profile to the workspace.
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
the Restricted policy of the Pod Security Standards.
The baseline profile runs the pod as the root user and with a security context that conforms with
the Baseline policy of the Pod Security Standards.
type: string
serviceAccountName:
default: default
Expand Down Expand Up @@ -27868,8 +27876,12 @@ spec:
type: object
securityProfile:
default: restricted
description: SecurityProfile applies a security profile to the workspace,
'restricted' by default.
description: |-
SecurityProfile applies a security profile to the workspace.
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
the Restricted policy of the Pod Security Standards.
The baseline profile runs the pod as the root user and with a security context that conforms with
the Baseline policy of the Pod Security Standards.
type: string
serviceAccountName:
default: default
Expand Down Expand Up @@ -28388,14 +28400,13 @@ spec:
spec:
containers:
- args:
- /manager
- --leader-elect
- --health-probe-bind-address=:8081
- --metrics-bind-address=:8383
- --program-fs-adv-addr=pulumi-kubernetes-operator.$(POD_NAMESPACE).svc.cluster.local
- --program-fs-adv-addr=pulumi-kubernetes-operator.$(POD_NAMESPACE).svc.cluster.local:80
- --zap-log-level=error
- --zap-time-encoding=iso8601
command:
- /manager
env:
- name: POD_NAMESPACE
valueFrom:
Expand Down
4 changes: 2 additions & 2 deletions operator/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ build: manifests generate fmt vet ## Build manager binary.

.PHONY: run
run: manifests generate fmt vet ## Run a controller from your host.
go run ./cmd/main.go --program-fs-adv-addr=localhost:9090
WORKSPACE_LOCALHOST=localhost:50051 SOURCE_CONTROLLER_LOCALHOST=localhost:9090 go run ./cmd/main.go

# If you wish to build the manager image targeting other platforms you can use the --platform flag.
# (i.e. docker build --platform linux/arm64). However, you must enable docker buildKit for it.
Expand Down Expand Up @@ -183,7 +183,7 @@ endif

.PHONY: install
install: manifests kustomize ## Install CRDs into the K8s cluster specified in ~/.kube/config.
$(KUSTOMIZE) build config/crd | $(KUBECTL) apply --server-side=true -f -
$(KUSTOMIZE) build config/crd | $(KUBECTL) apply --server-side=true --force-conflicts -f -

.PHONY: uninstall
uninstall: manifests kustomize ## Uninstall CRDs from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
Expand Down
11 changes: 5 additions & 6 deletions operator/api/auto/v1alpha1/workspace_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,6 @@ const (
SecurityProfileBaseline SecurityProfile = "baseline"
// SecurityProfileRestricted applies the restricted security profile.
SecurityProfileRestricted SecurityProfile = "restricted"

// SecurityProfileBaselineDefaultImage is the default image used when the security profile is 'baseline'.
SecurityProfileBaselineDefaultImage = "pulumi/pulumi:latest"
// SecurityProfileRestrictedDefaultImage is the default image used when the security profile is 'restricted'.
SecurityProfileRestrictedDefaultImage = "pulumi/pulumi:latest-nonroot"
Comment on lines -37 to -41
Copy link
Contributor Author

@EronWright EronWright Oct 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rationale: moving these constants and the associated 'defaulting' logic to webhook/auto/v1alpha1 where, in the future, a true webhook would apply the defaults eagerly. This PR does NOT implement a webhook, it simply applies the defaults during reconciliation for simplicity.

See the latest in webhook scaffolding: kubernetes-sigs/kubebuilder#4150

Copy link
Contributor Author

@EronWright EronWright Oct 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me say, the benefit of applying defaults eagerly (with a webhook) rather than lazily (during reconciliation) is stability; one may change the default later without affecting existing workloads. The implicit becomes explicit.

)

// WorkspaceSpec defines the desired state of Workspace
Expand All @@ -47,7 +42,11 @@ type WorkspaceSpec struct {
// +kubebuilder:default="default"
ServiceAccountName string `json:"serviceAccountName,omitempty"`

// SecurityProfile applies a security profile to the workspace, 'restricted' by default.
// SecurityProfile applies a security profile to the workspace.
// The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
// the Restricted policy of the Pod Security Standards.
// The baseline profile runs the pod as the root user and with a security context that conforms with
// the Baseline policy of the Pod Security Standards.
// +kubebuilder:default="restricted"
// +optional
SecurityProfile SecurityProfile `json:"securityProfile,omitempty"`
Expand Down
14 changes: 4 additions & 10 deletions operator/cmd/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ import (
// Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
// to ensure that exec-entrypoint and run can make use of them.
_ "k8s.io/client-go/plugin/pkg/client/auth"
"sigs.k8s.io/controller-runtime/pkg/client"

sourcev1 "github.com/fluxcd/source-controller/api/v1"
sourcev1b2 "github.com/fluxcd/source-controller/api/v1beta2"
Expand Down Expand Up @@ -150,7 +149,10 @@ func main() {

// Create a new ProgramHandler to handle Program objects. Both the ProgramReconciler and the file server need to
// access the ProgramHandler, so it is created here and passed to both.
pHandler := newProgramHandler(mgr.GetClient(), programFSAdvAddr)
if programFSAdvAddr == "" {
programFSAdvAddr = determineAdvAddr(programFSAddr)
}
pHandler := pulumicontroller.NewProgramHandler(mgr.GetClient(), programFSAdvAddr)

if err = (&autocontroller.WorkspaceReconciler{
Client: mgr.GetClient(),
Expand Down Expand Up @@ -246,14 +248,6 @@ func (fs pFileserver) Start(ctx context.Context) error {
}
}

func newProgramHandler(k8sClient client.Client, advAddr string) *pulumicontroller.ProgramHandler {
if advAddr == "" {
advAddr = determineAdvAddr(advAddr)
}

return pulumicontroller.NewProgramHandler(k8sClient, advAddr)
}

func determineAdvAddr(addr string) string {
host, port, err := net.SplitHostPort(addr)
if err != nil {
Expand Down
8 changes: 6 additions & 2 deletions operator/config/crd/bases/auto.pulumi.com_workspaces.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8413,8 +8413,12 @@ spec:
type: object
securityProfile:
default: restricted
description: SecurityProfile applies a security profile to the workspace,
'restricted' by default.
description: |-
SecurityProfile applies a security profile to the workspace.
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
the Restricted policy of the Pod Security Standards.
The baseline profile runs the pod as the root user and with a security context that conforms with
the Baseline policy of the Pod Security Standards.
type: string
serviceAccountName:
default: default
Expand Down
16 changes: 12 additions & 4 deletions operator/config/crd/bases/pulumi.com_stacks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9334,8 +9334,12 @@ spec:
type: object
securityProfile:
default: restricted
description: SecurityProfile applies a security profile to
the workspace, 'restricted' by default.
description: |-
SecurityProfile applies a security profile to the workspace.
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
the Restricted policy of the Pod Security Standards.
The baseline profile runs the pod as the root user and with a security context that conforms with
the Baseline policy of the Pod Security Standards.
type: string
serviceAccountName:
default: default
Expand Down Expand Up @@ -18871,8 +18875,12 @@ spec:
type: object
securityProfile:
default: restricted
description: SecurityProfile applies a security profile to
the workspace, 'restricted' by default.
description: |-
SecurityProfile applies a security profile to the workspace.
The restricted profile (default) runs the pod as a non-root user and with a security context that conforms with
the Restricted policy of the Pod Security Standards.
The baseline profile runs the pod as the root user and with a security context that conforms with
the Baseline policy of the Pod Security Standards.
type: string
serviceAccountName:
default: default
Expand Down
7 changes: 3 additions & 4 deletions operator/config/manager/manager.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,18 +33,18 @@ spec:
runAsUser: 65532
runAsGroup: 65532
containers:
- command:
- /manager
- name: manager
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
args:
- /manager
- --leader-elect
- --health-probe-bind-address=:8081
- --metrics-bind-address=:8383
- --program-fs-adv-addr=pulumi-kubernetes-operator.$(POD_NAMESPACE).svc.cluster.local
- --program-fs-adv-addr=pulumi-kubernetes-operator.$(POD_NAMESPACE).svc.cluster.local:80
- --zap-log-level=error
- --zap-time-encoding=iso8601
ports:
Expand All @@ -56,7 +56,6 @@ spec:
protocol: TCP
image: controller:latest
imagePullPolicy: IfNotPresent
name: manager
securityContext:
allowPrivilegeEscalation: false
capabilities:
Expand Down
Loading
Loading