Setting an existingSecret for externalPostgresql means sentry web and worker fail to start

[threesquared]

Issue submitter TODO list

• I've searched for an already existing issues here

Describe the bug (actual behavior)

Whenever I set a existingSecret for externalPostgresql in my values file the sentry web and worker pods do not start and fail the health checks. No logs whatsoever are written out from these pods.

Expected behavior

The web and worker pods should eventually come up and then the rest of the hooks should fire.

values.yaml

sentry.yaml

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sentry
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://sentry-kubernetes.github.io/charts
    chart: sentry
    targetRevision: 26.8.0
    helm:
      releaseName: sentry
      values: |
        asHook: true

        serviceAccount:
          enabled: true
          annotations:
            argocd.argoproj.io/sync-wave: "-100"
            eks.amazonaws.com/role-arn: "arn:aws:iam::999999999999:role/SentryRole"
            eks.amazonaws.com/sts-regional-endpoints: "true"

        system:
          url: "https://sentry.domain.io"
          adminEmail: "[email protected]"

        nginx:
          enabled: false

        rabbitmq:
          enabled: false

        sentry:
          web:
            resources:
              limits:
                memory: 3Gi
                ephemeral-storage: 2Gi
              requests:
                cpu: 400m
                memory: 2Gi
                ephemeral-storage: 1Gi
            service:
              annotations:
                alb.ingress.kubernetes.io/healthcheck-path: /_health/
                alb.ingress.kubernetes.io/healthcheck-port: traffic-port
          worker:
            resources:
              limits:
                memory: 3Gi
                ephemeral-storage: 2Gi
              requests:
                cpu: 400m
                memory: 2Gi
                ephemeral-storage: 1Gi

        relay:
          service:
            annotations:
              alb.ingress.kubernetes.io/healthcheck-path: /api/relay/healthcheck/ready/
              alb.ingress.kubernetes.io/healthcheck-port: traffic-port

        postgresql:
          enabled: false

        externalPostgresql:
          existingSecret: db-user-pass
          existingSecretKeys:
            password: password
            host: host
          port: 3306
          database: sentry
          username: sentry

        metrics:
          enabled: true

        ingress:
          enabled: true
          hostname: sentry.domain.io
          regexPathStyle: aws-alb
          annotations:
            kubernetes.io/ingress.class: alb
            alb.ingress.kubernetes.io/scheme: internet-facing
            alb.ingress.kubernetes.io/target-type: ip
            alb.ingress.kubernetes.io/tags: Owner=Me,Environment=env,Service=sentry
            alb.ingress.kubernetes.io/load-balancer-attributes: "access_logs.s3.enabled=true,access_logs.s3.bucket=my-logs"
            alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
            alb.ingress.kubernetes.io/ssl-redirect: "443"
            alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-1:999999999999:certificate/UUID
            alb.ingress.kubernetes.io/security-groups: "alb-sg-sentry"
            external-dns.alpha.kubernetes.io/hostname: sentry.domain.io

        filestore:
          backend: s3
          s3:
            bucketName: sentry-bucket
            region_name: eu-west-1

        extraManifests:
          - apiVersion: vpcresources.k8s.aws/v1beta1
            kind: SecurityGroupPolicy
            metadata:
              name: sentry
              namespace: sentry
              annotations:
                argocd.argoproj.io/sync-wave: "-100"
            spec:
              podSelector:
                matchLabels:
                  app: sentry
              securityGroups:
                groupIds:
                  - "sg-999999999999999"

          - apiVersion: external-secrets.io/v1beta1
            kind: ExternalSecret
            metadata:
              name: sentry
              namespace: sentry
              annotations:
                argocd.argoproj.io/sync-wave: "-80"
            spec:
              refreshInterval: 5m
              secretStoreRef:
                kind: SecretStore
                name: sentry
              target:
                name: db-user-pass
                creationPolicy: Owner
              dataFrom:
              - extract:
                  key: "sentry/secrets"

          - apiVersion: external-secrets.io/v1beta1
            kind: SecretStore
            metadata:
              name: sentry
              namespace: sentry
              annotations:
                argocd.argoproj.io/sync-wave: "-90"
            spec:
              provider:
                aws:
                  service: SecretsManager
                  region: eu-west-1
                  auth:
                    jwt:
                      serviceAccountRef:
                        name: sentry-web

  destination:
    server: https://kubernetes.default.svc
    namespace: sentry
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true

Helm chart version

26.8.0

Steps to reproduce

I appreciate it might be hard to re-create this setup exactly but running kubectl apply -f sentry.yaml in a cluster with ArgoCD should reproduce it? I am more looking for tips to try and get any kind of debug information out of the pods or anything else I can try.

Screenshots

No response

Logs

There are no logs from the web or worker pods. This is the description of the web pod:

Name:             sentry-web-c7d558d99-68ffw
Namespace:        sentry
Priority:         0
Service Account:  sentry-web
Node:             ip-10-224-52-214.eu-west-1.compute.internal/10.224.52.214
Start Time:       Tue, 10 Dec 2024 23:32:56 +0000
Labels:           app=sentry
                  pod-template-hash=c7d558d99
                  release=sentry
                  role=web
Annotations:      checksum/config.yaml: 9ea15b23df10c4f4ea41d56073417b5b04dd08f92adb876fdcaa2484052487e4
                  checksum/configYml: 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
                  checksum/sentryConfPy: d5f85a6a8afbc55eebe23801e1a51a0fb4c0428c9a73ef6708d8dc83e079cd49
Status:           Running
IP:               10.224.52.189
IPs:
  IP:           10.224.52.189
Controlled By:  ReplicaSet/sentry-web-c7d558d99
Containers:
  sentry-web:
    Container ID:  containerd://13c1c0b77539aced575dc64bbd5d65542e23c9efe6c6aeed99607137c80bc815
    Image:         getsentry/sentry:24.9.0
    Image ID:      docker.io/getsentry/sentry@sha256:1830c64c38383ff8e317bc7ba2274d27d176a113987fdc67e8a5202a67a70bad
    Port:          9000/TCP
    Host Port:     0/TCP
    Command:
      sentry
    Args:
      run
      web
    State:          Running
      Started:      Tue, 10 Dec 2024 23:40:57 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 10 Dec 2024 23:39:37 +0000
      Finished:     Tue, 10 Dec 2024 23:40:57 +0000
    Ready:          False
    Restart Count:  6
    Limits:
      ephemeral-storage:  2Gi
      memory:             3Gi
    Requests:
      cpu:                400m
      ephemeral-storage:  1Gi
      memory:             2Gi
    Liveness:             http-get http://:9000/_health/ delay=10s timeout=2s period=10s #success=1 #failure=5
    Readiness:            http-get http://:9000/_health/ delay=10s timeout=2s period=10s #success=1 #failure=5
    Environment:
      SNUBA:                        http://sentry-snuba:1218
      VROOM:                        http://sentry-vroom:8085
      SENTRY_SECRET_KEY:            <set to the key 'key' in secret 'sentry-sentry-secret'>  Optional: false
      POSTGRES_PASSWORD:            <set to the key 'password' in secret 'db-user-pass'>     Optional: false
      POSTGRES_USER:                sentry
      POSTGRES_NAME:                sentry
      POSTGRES_HOST:                <set to the key 'host' in secret 'db-user-pass'>  Optional: false
      POSTGRES_PORT:                3306
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           eu-west-1
      AWS_REGION:                   eu-west-1
      AWS_ROLE_ARN:                 arn:aws:iam::999999999999:role/SentryRole
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /etc/sentry from config (ro)
      /var/lib/sentry/files from sentry-data (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mjmxv (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sentry-sentry
    Optional:  false
  sentry-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-mjmxv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  9m1s                    default-scheduler  Successfully assigned sentry/sentry-web-c7d558d99-68ffw to ip-10-224-52-214.eu-west-1.compute.internal
  Normal   Killing    8m10s                   kubelet            Container sentry-web failed liveness probe, will be restarted
  Normal   Pulled     7m40s (x2 over 9m)      kubelet            Container image "getsentry/sentry:24.9.0" already present on machine
  Normal   Created    7m40s (x2 over 9m)      kubelet            Created container sentry-web
  Normal   Started    7m40s (x2 over 9m)      kubelet            Started container sentry-web
  Warning  Unhealthy  7m20s (x7 over 8m50s)   kubelet            Liveness probe failed: Get "http://10.224.52.189:9000/_health/": dial tcp 10.224.52.189:9000: connect: connection refused
  Warning  Unhealthy  3m50s (x35 over 8m50s)  kubelet            Readiness probe failed: Get "http://10.224.52.189:9000/_health/": dial tcp 10.224.52.189:9000: connect: connection refused

Additional context

I can jump into the web/worker pods before they are killed and the right values from the secret are present in the env. If I try to run sentry run worker in the pod then it just hangs. Turning on debug level logs does spit out some lines like def send_activity_notifications_to_slack_threads... but nothing helpful and then also just hangs.

I have tried both using the ExternalSecret provider and with just a pre-created Secret object the outcome is the same.

The only thing that helps is if I just hardcode the externalPostgresql values then the web and worker pods will eventually start and the db-init hook is fired as expected.