Skip to content

Commit

Permalink
Merge branch '1632-generate-stackgres-crds-reference-docs-automatical…
Browse files Browse the repository at this point in the history
…ly' into 'main'

Resolve "Generate StackGres CRDs reference docs automatically"

Closes #1632

See merge request ongresinc/stackgres!1424
  • Loading branch information
teoincontatto committed Jul 28, 2023
2 parents c799c48 + 0ac8f5f commit 3e3fdad
Show file tree
Hide file tree
Showing 51 changed files with 1,482 additions and 2,843 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@ doc/public
doc/data/crds
doc/data/versions.yml
doc/generated
doc/content/en/06-crd-reference/*/_index.md

# Node exclusions
node_modules/
Expand Down
56 changes: 52 additions & 4 deletions .gitlab-ci/e2e-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -251,12 +251,24 @@ e2e ex tests jvm 6 amd64:
- $E2E_TEST
- $E2E_EXCLUSIVE_JOBS =~ /^[12345]$/

e2e ui tests jvm amd64:
e2e ui chrome tests jvm amd64:
extends: .e2e ex tests jvm amd64
variables:
E2E_SKIP_OPERATOR_INSTALL: "true"
E2E_RUN_ONLY: ui:1/1
IS_WEB: "true"
E2E_UI_TEST_BROWSER: chrome
except:
variables:
- $E2E_TEST != null && $E2E_TEST !~ /(^ui$|^ui | ui | ui$)/

e2e ui firefox tests jvm amd64:
extends: .e2e ex tests jvm amd64
variables:
E2E_SKIP_OPERATOR_INSTALL: "true"
E2E_RUN_ONLY: ui:1/1
IS_WEB: "true"
E2E_UI_TEST_BROWSER: firefox
except:
variables:
- $E2E_TEST != null && $E2E_TEST !~ /(^ui$|^ui | ui | ui$)/
Expand Down Expand Up @@ -370,12 +382,24 @@ e2e ex tests native 6 amd64:
- $E2E_TEST
- $E2E_EXCLUSIVE_JOBS =~ /^[12345]$/

e2e ui tests native amd64:
e2e ui chrome tests native amd64:
extends: .e2e ex tests native amd64
variables:
E2E_SKIP_OPERATOR_INSTALL: "true"
E2E_RUN_ONLY: ui:1/1
IS_WEB: "true"
E2E_UI_TEST_BROWSER: chrome
except:
variables:
- $E2E_TEST != null && $E2E_TEST !~ /(^ui$|^ui | ui | ui$)/

e2e ui firefox tests native amd64:
extends: .e2e ex tests native amd64
variables:
E2E_SKIP_OPERATOR_INSTALL: "true"
E2E_RUN_ONLY: ui:1/1
IS_WEB: "true"
E2E_UI_TEST_BROWSER: firefox
except:
variables:
- $E2E_TEST != null && $E2E_TEST !~ /(^ui$|^ui | ui | ui$)/
Expand Down Expand Up @@ -492,12 +516,24 @@ e2e ex tests jvm 6 arm64:
- $E2E_TEST
- $E2E_EXCLUSIVE_JOBS =~ /^[12345]$/

e2e ui tests jvm arm64:
e2e ui chrome tests jvm arm64:
extends: .e2e ex tests jvm arm64
variables:
E2E_SKIP_OPERATOR_INSTALL: "true"
E2E_RUN_ONLY: ui:1/1
IS_WEB: "true"
E2E_UI_TEST_BROWSER: chrome
except:
variables:
- $E2E_TEST != null && $E2E_TEST !~ /(^ui$|^ui | ui | ui$)/

e2e ui firefox tests jvm arm64:
extends: .e2e ex tests jvm arm64
variables:
E2E_SKIP_OPERATOR_INSTALL: "true"
E2E_RUN_ONLY: ui:1/1
IS_WEB: "true"
E2E_UI_TEST_BROWSER: firefox
except:
variables:
- $E2E_TEST != null && $E2E_TEST !~ /(^ui$|^ui | ui | ui$)/
Expand Down Expand Up @@ -611,12 +647,24 @@ e2e ex tests native 6 arm64:
- $E2E_TEST
- $E2E_EXCLUSIVE_JOBS =~ /^[12345]$/

e2e ui tests native arm64:
e2e ui chrome tests native arm64:
extends: .e2e ex tests native arm64
variables:
E2E_SKIP_OPERATOR_INSTALL: "true"
E2E_RUN_ONLY: ui:1/1
IS_WEB: "true"
E2E_UI_TEST_BROWSER: chrome
except:
variables:
- $E2E_TEST != null && $E2E_TEST !~ /(^ui$|^ui | ui | ui$)/

e2e ui firefox tests native arm64:
extends: .e2e ex tests native arm64
variables:
E2E_SKIP_OPERATOR_INSTALL: "true"
E2E_RUN_ONLY: ui:1/1
IS_WEB: "true"
E2E_UI_TEST_BROWSER: firefox
except:
variables:
- $E2E_TEST != null && $E2E_TEST !~ /(^ui$|^ui | ui | ui$)/
133 changes: 133 additions & 0 deletions doc/CRD.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
{{- range .Groups }}
{{- $group := . }}
{{- range .Kinds}}
{{- $kind := .}}
{{- $levelTypes := dict }}
{{- $hiddenTypes := dict }}
{{- $onlyOnCreationTypes := dict }}
{{- $requireRestartTypes := dict }}
{{- range .Types}}
{{- $key := regexReplaceAll "\\[index\\]" .Name "" }}
{{- range .Fields -}}
{{- $description := .Description }}
{{- $fieldKey := (print $key "." .Name) }}
{{- if (and $key (get $levelTypes $key)) }}
{{- $levelTypes = merge $levelTypes (dict $fieldKey (add (get $levelTypes $key) 1)) }}
{{- else }}
{{- $levelTypes = merge $levelTypes (dict $fieldKey 2) }}
{{- end }}
{{- if or ($description | regexMatch "See: http[^ ]+$") (and $key (get $hiddenTypes $key)) }}
{{- $hiddenTypes = merge $hiddenTypes (dict $fieldKey true) }}
{{- end }}
{{- if or ($description | regexMatch "\\*\\*This field can only be set on creation\\.\\*\\*") (and $key (get $onlyOnCreationTypes $key)) }}
{{- $onlyOnCreationTypes = merge $onlyOnCreationTypes (dict $fieldKey true) }}
{{- end }}
{{- if or ($description | regexMatch "\\*\\*Changing this field may require a restart\\.\\*\\*") (and $key (get $requireRestartTypes $key)) }}
{{- $requireRestartTypes = merge $requireRestartTypes (dict $fieldKey true) }}
{{- end }}
{{- end }}
{{- end }}
{{- range .Types}}
{{- $key := regexReplaceAll "\\[index\\]" .Name "" }}
{{- if not (get $hiddenTypes $key) }}
{{- if not .IsTopLevel}}

{{range $i, $e := until (min 5 (get $levelTypes $key) | int)}}#{{end}} {{.Name}}
{{ if .ParentKey}}<sup><sup>[↩ Parent](#{{.ParentKey}})</sup></sup>{{end}}
{{- end}}

{{.Description}}

<table>
<thead>
<tr>
<th><div style="width:12rem">Property</div></th>
<th><div style="width:5rem">Required</div></th>
<th><div style="width:5rem">Updatable</div></th>
<th><div style="width:5rem">May Require Restart</div></th>
<th><div style="width:5rem">Type</div></th>
<th>Description</th>
</tr>
</thead>
<tbody><tr style="display:none;">
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>

Workaround for hugo bug not rendering first table row
<br/>
</td>
</tr>
{{- if .IsTopLevel -}}
<tr>
<td><b>apiVersion</b></td>
<td>✓</td>
<td></td>
<td></td>
<td>string</td>
<td>{{$group.Group}}/{{$group.Version}}</td>
</tr>
<tr>
<td><b>kind</b></td>
<td>✓</td>
<td></td>
<td></td>
<td>string</td>
<td>{{$kind.Name}}</td>
</tr>
<tr>
<td><b><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta">metadata</a></b></td>
<td>✓</td>
<td>✓</td>
<td></td>
<td>object</td>
<td>Refer to the Kubernetes API documentation for the fields of the `metadata` field.</td>
</tr>
{{- end -}}
{{- range .Fields -}}
{{- $fieldKey := (print $key "." .Name) }}
<tr>
<td><b>{{if and .TypeKey (not (get $hiddenTypes $fieldKey))}}<a href="#{{.TypeKey}}">{{.Name}}</a>{{else}}{{.Name}}{{end}}</b></td>
<td>{{if .Required}}✓{{end}}</td>
<td>{{if not (get $onlyOnCreationTypes $fieldKey)}}✓{{end}}</td>
<td>{{if get $requireRestartTypes $fieldKey}}✓{{end}}</td>
<td>{{.Type}}</td>
<td>
{{.Description}}<br/>
{{- if or .Schema.XValidations .Schema.Format .Schema.Enum .Schema.Default .Schema.Minimum .Schema.Maximum }}
<br/>
{{- end}}
{{- if .Schema.XValidations }}
<i>Validations</i>:
{{- range .Schema.XValidations -}}
<li>{{ .Rule }}: {{ .Message }}</li>
{{- end -}}
{{- end }}
{{- if .Schema.Format }}
<i>Format</i>: {{ .Schema.Format }}<br/>
{{- end }}
{{- if .Schema.Enum }}
<i>Enum</i>: {{ .Schema.Enum | toStrings | join ", " }}<br/>
{{- end }}
{{- if .Schema.Minimum }}
<i>Minimum</i>: {{ .Schema.Minimum }}<br/>
{{- end }}
{{- if .Schema.Maximum }}
<i>Maximum</i>: {{ .Schema.Maximum }}<br/>
{{- end }}
{{- if .Schema.Default }}
<i>Default</i>: {{ .Schema.Default }}<br/>
{{- end }}
</td>
</tr>
{{- end -}}
</tbody>
</table>

{{- end}}{{/* hidden type check */}}
{{- end}}{{/* range .Types */}}
{{- end}}{{/* range .Kinds */}}
{{- end}}{{/* range .Groups */}}
15 changes: 15 additions & 0 deletions doc/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,5 +33,20 @@ helm-docs \
-f "values.yaml" \
-t "VALUES.md.gotmpl"
)
CRD_PATH="$(dirname "$0")/../stackgres-k8s/src/common/src/main/resources/crds"
cat "$CRD_PATH/index.txt" \
| {
while read -r CRD_FILE
do
crdoc --resources "$CRD_PATH/$CRD_FILE" \
-o "$(dirname "$0")/generated/${CRD_FILE%.yaml}.md" \
--template "$(dirname "$0")/CRD.tmpl"
TARGET_PATH="$(ls -d "$(dirname "$0")/content/en/06-crd-reference/"*"-$(echo "${CRD_FILE%.yaml}" | tr 'A-Z' 'a-z')")"
INCLUDE_LINE="$(cat "$TARGET_PATH/_index.template.md" | grep -nxF '{{% include "generated/'"${CRD_FILE%.yaml}"'.md" %}}' | cut -d : -f 1)"
head -n "$((INCLUDE_LINE - 1))" "$TARGET_PATH/_index.template.md" > "$TARGET_PATH/_index.md"
cat "$(dirname "$0")/generated/${CRD_FILE%.yaml}.md" >> "$TARGET_PATH/_index.md"
tail -n +"$((INCLUDE_LINE + 1))" "$TARGET_PATH/_index.template.md" >> "$TARGET_PATH/_index.md"
done
}
mv "$(dirname "$0")/../stackgres-k8s/install/helm/stackgres-operator/generated.md" \
"$(dirname "$0")/generated/stackgres-operator.md"
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ spec:
disableClusterResourceRequirements: true
```
After setting those options you will have to restart the Postgres cluster Pods by simply deleting them (or using a [restart SGDbOps]({{% relref "06-crd-reference/09-sgdbops#restart" %}})):
After setting those options you will have to restart the Postgres cluster Pods by simply deleting them (or using a [restart SGDbOps]({{% relref "06-crd-reference/08-sgdbops#restart" %}})):
```bash
kubectl delete pod -l app=StackGresCluster,stackgres.io/cluster-name=simple
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ In general, these steps are optional, but we do recommend to consider these feat

### Configuring an Instance Profile

You can create your cluster with different hardware specifications using an [SGInstanceProfile](https://stackgres.io/doc/latest/04-postgres-cluster-management/03-instance-profiles/) custom resource (CR) as follows:
You can create your cluster with different hardware specifications using an [SGInstanceProfile]({{% relref "06-crd-reference/02-sginstanceprofile" %}}) custom resource (CR) as follows:

```yaml
cat << EOF | kubectl apply -f -
Expand All @@ -33,7 +33,7 @@ EOF

### Configuring Postgres and PGBouncer

You can also change Postgres' configuration using an [SGPostgresConfig]({{% relref "06-crd-reference/03-sgpostgresconfig" %}}) CR, or the PGBouncer settings using [SGPoolingConfig]({{% relref "06-crd-reference/04-sgpoolingconfig" %}}), the backup storage specification using [SGObjectStorage]({{% relref "06-crd-reference/10-sgobjectstorage" %}}), and more.
You can also change Postgres' configuration using an [SGPostgresConfig]({{% relref "06-crd-reference/03-sgpostgresconfig" %}}) CR, or the PGBouncer settings using [SGPoolingConfig]({{% relref "06-crd-reference/04-sgpoolingconfig" %}}), the backup storage specification using [SGObjectStorage]({{% relref "06-crd-reference/09-sgobjectstorage" %}}), and more.

The next code snippets will show you how to use these CRs.

Expand Down Expand Up @@ -82,7 +82,7 @@ EOF

### Configuring Backups

The [SGObjectStorage]({{% relref "06-crd-reference/10-sgobjectstorage" %}}) CRs are used to configure how backups are being taken.
The [SGObjectStorage]({{% relref "06-crd-reference/09-sgobjectstorage" %}}) CRs are used to configure how backups are being taken.

The following shows and example configuration using [Google Cloud Storage](https://cloud.google.com/storage/):

Expand Down Expand Up @@ -156,7 +156,7 @@ kubectl -n my-cluster create secret generic pgbench-user-password-secret \
--from-literal=pgbench-create-user-sql="create user pgbench password 'admin123'"
```

Then we reference the secret in a [SGScript]({{% relref "06-crd-reference/11-sgscript" %}}):
Then we reference the secret in a [SGScript]({{% relref "06-crd-reference/10-sgscript" %}}):

```yaml
cat << EOF | kubectl apply -f -
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ kubectl create secret generic \
```

Having the credentials secret created, we now need to create the object storage configuration and to set the backup configuration.
The object storage configuration it is governed by the [SGObjectStorage]({{% relref "06-crd-reference/10-sgobjectstorage" %}}) CRD.
The object storage configuration it is governed by the [SGObjectStorage]({{% relref "06-crd-reference/09-sgobjectstorage" %}}) CRD.
This CRD allows to specify the object storage technology, required parameters, as well as a reference to the credentials secret.

Create a file `sgobjectstorage-backupconfig1.yaml` with the following contents:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ kubectl -n $CLUSTER_NAMESPACE create secret generic s3-backup-bucket-secret \
## StackGres Object Storage

Having the credentials secret created, we now need to create the object storage configuration and to set the backup configuration.
The object storage configuration it is governed by the [SGObjectStorage]({{% relref "06-crd-reference/10-sgobjectstorage" %}}) CRD.
The object storage configuration it is governed by the [SGObjectStorage]({{% relref "06-crd-reference/09-sgobjectstorage" %}}) CRD.
This CRD allows to specify the object storage technology, required parameters, as well as a reference to the credentials secret.

Create a file `sgobjectstorage-backupconfig1.yaml` with the following contents:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ rm -rfv my-creds.json
```

Having the resources created, we now need to create the object storage configuration and to set the backup configuration.
The object storage configuration it is governed by the [SGObjectStorage]({{% relref "06-crd-reference/10-sgobjectstorage" %}}) CRD.
The object storage configuration it is governed by the [SGObjectStorage]({{% relref "06-crd-reference/09-sgobjectstorage" %}}) CRD.
This CRD allows to specify the object storage technology, required parameters, as well as a reference to the credentials secret.

Create a file `sgobjectstorage-backupconfig1.yaml` with the following contents:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ kubectl create secret generic oke-backup-bucket-secret \
```

Having the credential secret created, we now need to create the object storage configuration and to set the backup configuration.
The object storage configuration it is governed by the [SGObjectStorage]({{% relref "06-crd-reference/10-sgobjectstorage" %}}) CRD.
The object storage configuration it is governed by the [SGObjectStorage]({{% relref "06-crd-reference/09-sgobjectstorage" %}}) CRD.
This CRD allows to specify the object storage technology, required parameters, as well as a reference to the credentials secret.

Create a file `sgobjectstorage-backupconfig1.yaml` with your endpoint and region:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ StackGres support backups with the following storage options:
> The examples are using the [MinIO](https://min.io/) service as a S3 compatible service for a quick setup on local Kubernetes clusters.
> Although StackGres definitely recommends to choose a Storage-as-a-Service for production setups.
All the storage-related configuration is defined in the [SGObjectStorage]({{% relref "06-crd-reference/10-sgobjectstorage" %}}) CRD.
All the storage-related configuration is defined in the [SGObjectStorage]({{% relref "06-crd-reference/09-sgobjectstorage" %}}) CRD.
```yaml
apiVersion: stackgres.io/v1beta1
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ showToc: true
---

In StackGres, you can define SQL scripts that are executed on the database instances.
The [SGScript]({{% relref "06-crd-reference/11-sgscript" %}}) type is used to define these scripts that are referenced in a StackGres cluster.
The [SGScript]({{% relref "06-crd-reference/10-sgscript" %}}) type is used to define these scripts that are referenced in a StackGres cluster.

There are multiple ways to define scripts, depending on whether you need to define them as custom resource, within a config map, or within a secret.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@ url: /administration/sharded-cluster-creation/nonproduction
description: Important notes for non-production options in the production environment.
---

There are certain [non-production options]({{% relref "06-crd-reference/12-sgshardedcluster" %}}#non-production-options) supported in StackGres, that we recommend to disable in a production environment.
There are certain [non-production options]({{% relref "06-crd-reference/11-sgshardedcluster" %}}#non-production-options) supported in StackGres, that we recommend to disable in a production environment.
To disable all of these options, create a YAML values file to include in the helm installation (`-f` or `--values` parameters) of the StackGres operator similar to the following:

```yaml
nonProductionOptions: {}
```
For reference, you can see a list of all of these [non-production options]({{% relref "06-crd-reference/12-sgshardedcluster" %}}#non-production-options).
For reference, you can see a list of all of these [non-production options]({{% relref "06-crd-reference/11-sgshardedcluster" %}}#non-production-options).
## Scaling with limited resources
Expand All @@ -35,7 +35,7 @@ spec:
disableClusterResourceRequirements: true
```
After setting those options you will have to restart the Postgres cluster Pods by simply deleting them (or using a [restart SGDbOps]({{% relref "06-crd-reference/09-sgdbops#restart" %}})):
After setting those options you will have to restart the Postgres cluster Pods by simply deleting them (or using a [restart SGDbOps]({{% relref "06-crd-reference/08-sgdbops#restart" %}})):
```bash
kubectl delete pod -l app=StackGresCluster,stackgres.io/cluster-name=simple
Expand Down
Loading

0 comments on commit 3e3fdad

Please sign in to comment.