Server-Side Dry-Runs Fallback to Client-Side for New Resources in ArgoCD

[andyflatt]

Issue Description

I am running ArgoCD with Kyverno configured as an admission controller and am attempting to enable the Argo Syncs in dry-run server mode to report blocking issues to users. This setup is intended to allow Kyverno errors to be captured during CI, helping to validate resource policies before applying them. While this works as expected for resources that already exist on the server, it fails for new resources.

The problem seems to stem from this hardcoded behavior in ArgoCD

Configuration Details

I have configured the following options:

syncPolicy:
  syncOptions:
    - CreateNamespace=false
    - ServerSideApply=true

Additionally, I have set these annotations:

annotations:
  argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true

Steps to reproduce

1. Deploy a Resource: Deploy a Deployment resource in a namespace (e.g., bad-namespace):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: bad-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: badtest
          image: nginx:latest
          ports:
          - containerPort: 567
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "500m"
              memory: "256Mi"

2. Apply a Kyverno Policy: Deploy a Kyverno policy to block deployments to the namespace.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-deployments-to-bad-namespace
spec:
  validationFailureAction: Enforce
  rules:
  - name: block-deployments-to-bad-namespace
    match:
      resources:
        kinds:
        - Deployment
        - Pod
        namespaces:
        - bad-namespace
    validate:
      message: "Deployments to the namespace 'bad-namespace' are not allowed."
      deny: {}

3. Dry Run the Change:* Perform a dry run sync of a change to the deployment. Observe the error message returned:

ComparisonError: Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error running server side apply in dryrun mode for resource Deployment/nginx-deployment: admission webhook "validate.kyverno.svc-ignore" denied the request:

resource Deployment/bad-namespace/nginx-deployment was blocked due to the following policies:

block-deployments-to-bad-namespace:
  block-deployments-to-bad-namespace: Deployments to the namespace 'bad-namespace' are not allowed.

4. Reverse the order: Create the Kyverno policy first, then attempt to create the resource. The dry run incorrectly shows a successful result:

Succeeded  Synced  deployment.apps/nginx-deployment created (dry run)

Expected Behavior

The dry-run server mode should consistently return Kyverno errors for both existing and new resources in alignment with the policy rules, regardless of the resource’s presence on the server.

Actual Behavior

Dry-run server validation works only for existing resources. For new resources, it defaults to dry-run client and therefore the expected Kyverno policy error is not returned, likely due to the hardcoded handling in the ArgoCD sync context.

Additional Information

This behavior impacts CI validation workflows by allowing non-compliant resources to appear valid in dry-run scenarios.