istio-cni by default

[juliusvonkohout]

• Istio-CNI as default
• We should use the new "Values.gateways.seccompProfile" and securitycontext to get rid of the manual patch at

  manifests/.github/workflows/pss_test.yaml

  Lines 35 to 45 in 26d970d

  	- name: Configure istio init container with seccompProfile attribute
  	run: |
  	kubectl get cm istio-sidecar-injector -n istio-system -o yaml > temporary_patch.yaml
  	sed -i '0,/runAsNonRoot: true/{s//&\n seccompProfile:\n type: RuntimeDefault/}' temporary_patch.yaml
  	sed -i '/runAsNonRoot: true/{N; /runAsUser: {{ .ProxyUID | default "1337" }}/a\
  	seccompProfile:\n type: RuntimeDefault
  	}' temporary_patch.yaml
  	kubectl apply -f temporary_patch.yaml
  	rm temporary_patch.yaml

  both for istio and istio-cni, so it is related to Rootless Kubeflow #2528
• We also need to add a comment about kserve with istio-cni and migrate https://github.com/kubeflow/manifests/blob/35298e042850e22a929ebdbcae94c225d69e33f8/.github/workflows/kserve_m2m_test.yaml to use a proper user namespace

@tarekabouzeid @kimwnasptd @DnPlas @kromanow94

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from juliusvonkohout. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[juliusvonkohout]

@hansinikarunarathne can you take a look at the ray tests?

[kromanow94]

I asked @MaxKavun to check from our end. He should have some findings this week.