fix: Make generated manifests compatible with various Yaml parsers (a…

…rgoproj#20532)

Can help with argoproj#20532

The manifests often used the plain style of strings, which could move a part of value to a new line, which could cause some parsers fail to parse. Using this style is not recommended. It was switched to the literal style when relevant and manifests were updated. To make sure next updates are good, the helper script to fix manifests is a part of the script to update manifests. It's in Python, hope that's okay as it doesn't get added to images.

There's one weird value for haproxy.cfg which is rendered as a quotes-enclosed string, but it was correct when I tried to load yamls from upstream and updated ones. In any case, it's more readable now.

Let's cherry pick to 2.13, as the apply of install manifests is currently broken.

Signed-off-by: Andrii Korotkov <[email protected]>

hack/installers/install-codegen-tools.sh

@@ -1,4 +1,4 @@
 #!/bin/bash
 set -eux -o pipefail
 
-KUSTOMIZE_VERSION=4.5.7 "$(dirname $0)/../install.sh" kustomize protoc
+KUSTOMIZE_VERSION=5.4.3 "$(dirname $0)/../install.sh" kustomize protoc

hack/update-manifests-normalizer.py

@@ -0,0 +1,23 @@
+import math
+import sys
+import yaml
+
+def represent_str(dumper, data):
+    use_literal_style = '\n' in data or ' ' in data or '\\' in data
+    if use_literal_style:
+        data = data.replace("\\", "\\\\")
+        return dumper.represent_scalar("tag:yaml.org,2002:str", data, style="|")
+    return dumper.represent_str(data)
+
+yaml.add_representer(str, represent_str, Dumper=yaml.SafeDumper)
+
+# Technically we can write to the same file, but that can erase the file if dump fails.
+input_filename = sys.argv[1]
+output_filename = sys.argv[2]
+
+with open(input_filename, "r") as input_file:
+    data = yaml.safe_load_all(input_file.read())
+
+with open(output_filename, "w") as output_file:
+    # Without large width the multiline strings enclosed in "..." are rendered weirdly.
+    yaml.safe_dump_all(data, output_file, width=math.inf)

hack/update-manifests.sh

@@ -9,6 +9,10 @@ AUTOGENMSG="# This is an auto-generated file. DO NOT EDIT"
 
 KUSTOMIZE=kustomize
 [ -f "$SRCROOT/dist/kustomize" ] && KUSTOMIZE="$SRCROOT/dist/kustomize"
+# Really need advanced Yaml tools to fix https://github.com/argoproj/argo-cd/issues/20532.
+PYTHON=python3
+NORMALIZER="$SRCROOT/hack/update-manifests-normalizer.py"
+TEMP_FILE=/tmp/argocd-normalized-manifest.yaml
 
 cd ${SRCROOT}/manifests/ha/base/redis-ha && ./generate.sh
 
@@ -35,17 +39,38 @@ cd ${SRCROOT}/manifests/base && $KUSTOMIZE edit set image quay.io/argoproj/argoc
 cd ${SRCROOT}/manifests/ha/base && $KUSTOMIZE edit set image quay.io/argoproj/argocd=${IMAGE_NAMESPACE}/argocd:${IMAGE_TAG}
 cd ${SRCROOT}/manifests/core-install && $KUSTOMIZE edit set image quay.io/argoproj/argocd=${IMAGE_NAMESPACE}/argocd:${IMAGE_TAG}
 
-echo "${AUTOGENMSG}" > "${SRCROOT}/manifests/install.yaml"
-$KUSTOMIZE build "${SRCROOT}/manifests/cluster-install" >> "${SRCROOT}/manifests/install.yaml"
+OUTPUT_FILE="${SRCROOT}/manifests/install.yaml"
+$KUSTOMIZE build "${SRCROOT}/manifests/cluster-install" > $OUTPUT_FILE
+$PYTHON $NORMALIZER $OUTPUT_FILE $TEMP_FILE
+# Avoid printing the whole file contents
+set +x
+echo -e "${AUTOGENMSG}\n$(cat $TEMP_FILE)" > $OUTPUT_FILE
+set -x
 
-echo "${AUTOGENMSG}" > "${SRCROOT}/manifests/namespace-install.yaml"
-$KUSTOMIZE build "${SRCROOT}/manifests/namespace-install" >> "${SRCROOT}/manifests/namespace-install.yaml"
+OUTPUT_FILE="${SRCROOT}/manifests/namespace-install.yaml"
+$KUSTOMIZE build "${SRCROOT}/manifests/namespace-install" > $OUTPUT_FILE
+$PYTHON $NORMALIZER $OUTPUT_FILE $TEMP_FILE
+set +x
+echo -e "${AUTOGENMSG}\n$(cat $TEMP_FILE)" > $OUTPUT_FILE
+set -x
 
-echo "${AUTOGENMSG}" > "${SRCROOT}/manifests/ha/install.yaml"
-$KUSTOMIZE build "${SRCROOT}/manifests/ha/cluster-install" >> "${SRCROOT}/manifests/ha/install.yaml"
+OUTPUT_FILE="${SRCROOT}/manifests/ha/install.yaml"
+$KUSTOMIZE build "${SRCROOT}/manifests/ha/cluster-install" > $OUTPUT_FILE
+$PYTHON $NORMALIZER $OUTPUT_FILE $TEMP_FILE
+set +x
+echo -e "${AUTOGENMSG}\n$(cat $TEMP_FILE)" > $OUTPUT_FILE
+set -x
 
-echo "${AUTOGENMSG}" > "${SRCROOT}/manifests/ha/namespace-install.yaml"
-$KUSTOMIZE build "${SRCROOT}/manifests/ha/namespace-install" >> "${SRCROOT}/manifests/ha/namespace-install.yaml"
+OUTPUT_FILE="${SRCROOT}/manifests/ha/namespace-install.yaml"
+$KUSTOMIZE build "${SRCROOT}/manifests/ha/namespace-install" > $OUTPUT_FILE
+$PYTHON $NORMALIZER $OUTPUT_FILE $TEMP_FILE
+set +x
+echo -e "${AUTOGENMSG}\n$(cat $TEMP_FILE)" > $OUTPUT_FILE
+set -x
 
-echo "${AUTOGENMSG}" > "${SRCROOT}/manifests/core-install.yaml"
-$KUSTOMIZE build "${SRCROOT}/manifests/core-install" >> "${SRCROOT}/manifests/core-install.yaml"
+OUTPUT_FILE="${SRCROOT}/manifests/core-install.yaml"
+$KUSTOMIZE build "${SRCROOT}/manifests/core-install" > $OUTPUT_FILE
+$PYTHON $NORMALIZER $OUTPUT_FILE $TEMP_FILE
+set +x
+echo -e "${AUTOGENMSG}\n$(cat $TEMP_FILE)" > $OUTPUT_FILE
+set -x

hack/update-openapi.sh

@@ -12,6 +12,9 @@ PROJECT_ROOT=$(
 PATH="${PROJECT_ROOT}/dist:${PATH}"
 GOPATH=$(go env GOPATH)
 GOPATH_PROJECT_ROOT="${GOPATH}/src/github.com/argoproj/argo-cd"
+# Really need advanced Yaml tools to fix https://github.com/argoproj/argo-cd/issues/20532.
+PYTHON=python3
+NORMALIZER="$PROJECT_ROOT/hack/update-manifests-normalizer.py"
 
 VERSION="v1alpha1"
 
@@ -31,3 +34,10 @@ openapi-gen \
 export GO111MODULE=on
 go build -o ./dist/gen-crd-spec "${PROJECT_ROOT}/hack/gen-crd-spec"
 ./dist/gen-crd-spec
+
+CRD_FILE="$PROJECT_ROOT/manifests/crds/application-crd.yaml"
+$PYTHON $NORMALIZER $CRD_FILE $CRD_FILE
+CRD_FILE="$PROJECT_ROOT/manifests/crds/applicationset-crd.yaml"
+$PYTHON $NORMALIZER $CRD_FILE $CRD_FILE
+CRD_FILE="$PROJECT_ROOT/manifests/crds/appproject-crd.yaml"
+$PYTHON $NORMALIZER $CRD_FILE $CRD_FILE