diff --git a/CHANGELOG.md b/CHANGELOG.md index 7da712d8..7d338a2b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,18 @@ # SAS Viya Monitoring for Kubernetes +## Version 1.0.0 (18Nov20) + +This is the first public release. + +* **Overall** + * Minor edits and cleanup to README files and sample user response files + +* **Monitoring** + * Grafana version bumped to 7.3.1 + * Prometheus Operator version bumped to 0.43.2 + * Prometheus version bumped to 2.22.2 + * Prometheus Pushgateway version bumped to 1.3.0 + ## Version 0.1.3 (11NOV20) * **Overall** diff --git a/README.md b/README.md index d7f34ccc..8c71f5f5 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ The monitoring solution includes these components: - [Prometheus Operator](https://github.com/coreos/prometheus-operator) - [Prometheus](https://prometheus.io/docs/introduction/overview/) - - [Alert Manager](https://prometheus.io/docs/alerting/alertmanager/) + - [Alertmanager](https://prometheus.io/docs/alerting/alertmanager/) - [Grafana](https://grafana.com/) - Prometheus Exporters - [node-exporter](https://github.com/prometheus/node_exporter) @@ -72,7 +72,7 @@ for more information about using the logging components. ### Monitoring See the [monitoring README](monitoring/README.md) to deploy the monitoring -components, including Prometheus Operator, Prometheus, Alert Manager, Grafana, +components, including Prometheus Operator, Prometheus, Alertmanager, Grafana, metric exporters, service monitors, and custom dashboards. ### Logging diff --git a/logging/AZURE_LOG_ANALYTICS_WORKSPACES.md b/logging/AZURE_LOG_ANALYTICS_WORKSPACES.md index db8c4463..15828ec7 100644 --- a/logging/AZURE_LOG_ANALYTICS_WORKSPACES.md +++ b/logging/AZURE_LOG_ANALYTICS_WORKSPACES.md @@ -81,7 +81,7 @@ To remove this solution, run this command: ``` By default, this script does NOT delete the namespace. -## Using Connection Information From Kubernetes Secret +## Using Connection Information From a Kubernetes Secret The deployment script creates a Kubernetes secret named `connection-info-azmonitor`containing the connection information. This ensures that the connection information is available in case the Fluent Bit pods are diff --git a/logging/Log_Retention.md b/logging/Log_Retention.md index 6e947204..083954c1 100644 --- a/logging/Log_Retention.md +++ b/logging/Log_Retention.md @@ -6,7 +6,7 @@ These are the default log retention periods: If you want to change either of these retention periods, follow these steps: - 1. If you have not already set up your `USER_DIR` directory (as discussed in the 'Customize the Deployment' section of the [README.md](README.md) file in the /logging directory), set up an empty directory with a `logging` subdirectory to contain the customization files. Export a `USER_DIR` environment variable that points to this location. For example: + 1. If you have not already set up your `USER_DIR` directory (as discussed in the 'Customize the Deployment' section of the [README.md](README.md) file in the `/logging` directory), set up an empty directory with a `logging` subdirectory to contain the customization files. Export a `USER_DIR` environment variable that points to this location. For example: ```bash mkdir -p ~/my-cluster-files/ops/user-dir/logging diff --git a/logging/README.md b/logging/README.md index 73aab5ac..2389435a 100644 --- a/logging/README.md +++ b/logging/README.md @@ -3,9 +3,9 @@ ## Introduction This document outlines the steps needed to deploy a set of log collection and -monitoring components for SAS Viya 4.x. These components provide a +monitoring components for SAS Viya. These components provide a comprehensive solution for collecting, transforming and surfacing all of the -log messages generated throughout SAS Viya 4.x. These components collect logs +log messages generated throughout SAS Viya. These components collect logs from all pods in a Kubernetes cluster, not only the pods used for SAS Viya. You must have cluster-admin access to any cluster in which you deploy these @@ -23,15 +23,37 @@ These components are deployed: Provides detailed Elasticsearch performance information for Prometheus ## Perform Pre-Deployment Tasks +Before deploying, you must select the release that you want to deploy, then create a local copy of the repository. -### Clone the Repository +### Select the Release to Copy -* From a command line, create a directory to contain the cloned repository. -* Change to the directory you created. -* Clone the repository -* `cd` to the repository directory -* If you have already cloned the repository, use the `git pull` command to -ensure that you have the most recent updates. +1. Click on **tags** above the repository tree. +2. On the **Tags** page, click [Releases](https://github.com/sassoftware/viya4-monitoring-kubernetes/releases) to view the list of available releases. +3. Use the release notes to determine the release you want to deploy. + +### Create a Local Copy of the Repository + +There are two methods to create a local copy of the repository: +- download a compressed copy +- clone the repository + +#### Download a Compressed Copy of the Repository + +1. On the [Releases](https://github.com/sassoftware/viya4-monitoring-kubernetes/releases) page, locate the release that you want to deploy. +2. Expand **Assets** for the release, which is located below the release notes. +3. Select either **Source code (.zip)** or **Source code (.tar.gz)** to download the repository +as a compressed file. +4. Expand the downloaded file to create a local copy of the repository. The repository is created +in a directory named `viya4-monitoring-kubernetes-`. + +#### Clone the Repository + +1. From the main page for the repository, click **Code**. +2. Copy the HTTPS URL for the repository. +3. From a directory where you want to create the local copy, enter the +command `git clone `. +4. Change to the `viya4-monitoring-kubernetes` directory. +5. Enter the command `git checkout ` ### Customize the Deployment diff --git a/monitoring/README.md b/monitoring/README.md index d78f3d93..104a54dd 100644 --- a/monitoring/README.md +++ b/monitoring/README.md @@ -3,8 +3,8 @@ ## Introduction This document outlines the steps needed to deploy a set of monitoring -components that provide the ability to monitor resources in a SAS Viya 4.x -environment. These components support monitoring of SAS Viya 4.x resources +components that provide the ability to monitor resources in a SAS Viya +environment. These components support monitoring of SAS Viya resources for each SAS Viya namespace as well as monitoring for Kubernetes cluster resources. @@ -18,7 +18,7 @@ These components are deployed: * Prometheus Operator * Prometheus -* Alert Manager +* Alertmanager * Grafana * node-exporter * kube-state-metrics @@ -29,18 +29,44 @@ These components are deployed: ## Perform Pre-Deployment Tasks -### Clone the Repository +Before deploying, you must select the release that you want to deploy, then +create a local copy of the repository. -* From a command line, create a directory to contain the cloned repository. -* Change to the directory you created. -* Clone the repository -* `cd` to the repository directory +If you use TLS to encrypt network traffic, you must also perform manual steps prior +to deployment. See the **TLS Support** section below for more information. -If you have already cloned the repository, use the `git pull` command to ensure -that you have the most recent updates. +### Select the Release to Copy -If you use TLS to encrypt network traffic, you must perform manual steps prior -to deployment. See the **TLS Support** section below for more information. +1. Click on **tags** above the repository tree. +2. On the **Tags** page, click [Releases](https://github.com/sassoftware/viya4-monitoring-kubernetes/releases) +to view the list of available releases. +3. Use the release notes to determine the release you want to deploy. + +### Create a Local Copy of the Repository + +There are two methods to create a local copy of the repository: + +* download a compressed copy +* clone the repository + +#### Download a Compressed Copy of the Repository + +1. On the [Releases](https://github.com/sassoftware/viya4-monitoring-kubernetes/releases) +page, locate the release that you want to deploy. +2. Expand **Assets** for the release, which is located below the release notes. +3. Select either **Source code (.zip)** or **Source code (.tar.gz)** to download +the repository as a compressed file. +4. Expand the downloaded file to create a local copy of the repository. The +repository is created in a directory named `viya4-monitoring-kubernetes-`. + +#### Clone the Repository + +1. From the main page for the repository, click **Code**. +2. Copy the HTTPS URL for the repository. +3. From a directory where you want to create the local copy, enter the +command `git clone `. +4. Change to the `viya4-monitoring-kubernetes` directory. +5. Enter the command `git checkout ` ## Customize the Deployment @@ -81,43 +107,31 @@ The monitoring stack uses the following Helm charts: These charts are highly customizable. Although the default values might be suitable, you might need to customize some values (such as for ingress, -for example). The kube-prometheus-stack helm chart includes the Prometheus +for example). The `kube-prometheus-stack` Helm chart includes the Prometheus Operator and aggregates other helm charts such as Grafana and the Prometheus Node Exporter. Links to the charts and default values are included in the `user-values-prom-operator.yaml` file. **Note:** If you are using a cloud provider, you must use ingress, rather than -NodePorts. Use the samples in the -[samples/ingress](/samples/ingress) +NodePorts. Use the samples in the [samples/ingress](/samples/ingress) area of this repository to set up either host-based or path-based ingress. ## Workload Node Placement -SAS Viya is deployed using a workload node placement strategy, which uses the -`workload.sas.com/class` taint to optimize the placement of its components on -Kubernetes nodes. By default, the monitoring components do **not** participate in the -workload node placement strategy. This is the recommended approach, because it enables the -monitoring components to function even if `workload.sas.com/class`-tainted nodes -are scaled to zero (in other words, are shut down). Therefore, by default, -most of the monitoring components are deployed to cluster nodes that do not -have `workload.sas.com/class` taints. On Microsoft Azure, this results -in pods being deployed on nodes in the `system` nodepool. - -To deploy the monitoring components so that they participate in the SAS Viya workload node -placement strategy rather than use this recommended deployment, set `MON_NODE_PLACEMENT_ENABLE` -to `true` in `$USER_DIR/monitoring/user.env`. - -## Istio Integration - -This repository contains an optional set of dashboards for Istio. These -dashboards use these assumptions: - -* Istio is installed in the `istio-system` namespace -* The optional Prometheus instance that is included with Istio is deployed - -To enable Istio support, set `ISTIO_ENABLED=true` in `$USER_DIR/user.env` -or `$USER_DIR/monitoring/user.env` before deploying monitoring using the -`deploy_monitoring_cluster.sh` script. +SAS Viya is deployed using a workload node placement strategy, which uses the +`workload.sas.com/class` taint to optimize the placement of its components on +Kubernetes nodes. By default, the monitoring components do **not** participate +in the workload node placement strategy. This is the recommended approach, +because it enables the monitoring components to function even if +`workload.sas.com/class`-tainted nodes are scaled to zero (in other words, are +shut down). Therefore, by default, most of the monitoring components are +deployed to cluster nodes that do not have `workload.sas.com/class` taints. On +Microsoft Azure, this results in pods being deployed on nodes in the `system` +nodepool. + +To deploy the monitoring components so that they participate in the SAS Viya +workload node placement strategy rather than use this recommended deployment, +set `MON_NODE_PLACEMENT_ENABLE` to `true` in `$USER_DIR/monitoring/user.env`. ### Recommendations @@ -212,7 +226,7 @@ You must perform manual steps prior to deployment in order to enable TLS. In addition, configuring HTTPS ingress involves a separate set of steps, which are similar to those needed for SAS Viya. -See the [TLS Sample](samples/tls) for more information. +See the [TLS Sample](/samples/tls) for more information. ## Miscellaneous Notes and Troubleshooting diff --git a/monitoring/bin/deploy_monitoring_cluster.sh b/monitoring/bin/deploy_monitoring_cluster.sh index 24b3811f..df6c940d 100755 --- a/monitoring/bin/deploy_monitoring_cluster.sh +++ b/monitoring/bin/deploy_monitoring_cluster.sh @@ -68,7 +68,7 @@ fi # Check if Prometheus Operator CRDs are already installed PROM_OPERATOR_CRD_UPDATE=${PROM_OPERATOR_CRD_UPDATE:-true} -PROM_OPERATOR_CRD_VERSION=${PROM_OPERATOR_CRD_VERSION:-v0.43.0} +PROM_OPERATOR_CRD_VERSION=${PROM_OPERATOR_CRD_VERSION:-v0.43.2} if [ "$PROM_OPERATOR_CRD_UPDATE" == "true" ]; then log_info "Updating Prometheus Operator custom resource definitions" kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/$PROM_OPERATOR_CRD_VERSION/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml diff --git a/monitoring/user-values-pushgateway.yaml b/monitoring/user-values-pushgateway.yaml index 7fffb5f5..90dc1e8e 100644 --- a/monitoring/user-values-pushgateway.yaml +++ b/monitoring/user-values-pushgateway.yaml @@ -1,10 +1,6 @@ # Chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-pushgateway # Default values: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus-pushgateway/values.yaml -# image: -# repository: prom/pushgateway -# tag: v1.2.0 - # persistentVolume: # enabled: true # size: 2Gi diff --git a/monitoring/user.env b/monitoring/user.env index 96876416..7ddef7cf 100644 --- a/monitoring/user.env +++ b/monitoring/user.env @@ -31,10 +31,10 @@ # This setting overrides TLS_ENABLE only for monitoring # MON_TLS_ENABLE=false -# Enable TLS (HTTPS) for monitoring components -# WARNING: Not fully working yet - for development/testing only -# See samples/tls for related customizations needed for TLS support -# MON_TLS_ENABLE=false +# Enables tolerations and pod affinity to enable the monitoring +# components to participate in the SAS Viya workload node +# placement strategy +# MON_NODE_PLACEMENT_ENABLE=false # Set to true to force an update of the Prometheus Operator CRDs # PROM_OPERATOR_CRD_UPDATE=false @@ -44,16 +44,11 @@ # match the value of prometheusOperator.image.tag in the helm YAML # if changed from the default. # See https://github.com/prometheus-operator/prometheus-operator/releases -# PROM_OPERATOR_CRD_VERSION=v0.43.0 +# PROM_OPERATOR_CRD_VERSION=v0.43.2 # Version of the kube-prometheus-stack helm chart to use # KUBE_PROM_STACK_CHART_VERSION=11.0.0 -# Enables tolerations and pod affinity to enable the monitoring -# components to participate in the SAS Viya workload node -# placement strategy -# MON_NODE_PLACEMENT_ENABLE=false - # Namespace of NGINX ingress controller (if applicable) # NGINX_NS=ingress-nginx @@ -67,7 +62,6 @@ # PGMONITOR_DASH=true # RABBITMQ_DASH=true # LOGGING_DASH=true -# ISTIO_DASH=false ## deploy_monitoring_viya.sh options # ---------------------------------- diff --git a/monitoring/values-prom-operator.yaml b/monitoring/values-prom-operator.yaml index c6e22f00..57159aaf 100644 --- a/monitoring/values-prom-operator.yaml +++ b/monitoring/values-prom-operator.yaml @@ -72,7 +72,7 @@ prometheus: nodePort: 31090 prometheusSpec: image: - tag: v2.21.0 + tag: v2.22.2 logLevel: info logFormat: json podAntiAffinity: soft @@ -165,7 +165,7 @@ nodeExporter: # https://github.com/grafana/helm-charts/tree/main/charts/grafana grafana: image: - tag: "7.2.2" + tag: "7.3.1" "grafana.ini": analytics: check_for_updates: false diff --git a/monitoring/values-pushgateway.yaml b/monitoring/values-pushgateway.yaml index 1039cf0d..b8b85b93 100644 --- a/monitoring/values-pushgateway.yaml +++ b/monitoring/values-pushgateway.yaml @@ -1,6 +1,6 @@ image: repository: prom/pushgateway - tag: v1.2.0 + tag: v1.3.0 # Bug in the helm chart - podLabels are put on the deployment, not the pod podLabels: diff --git a/samples/azure-deployment/README.md b/samples/azure-deployment/README.md index a79f81fd..89b345e5 100644 --- a/samples/azure-deployment/README.md +++ b/samples/azure-deployment/README.md @@ -6,22 +6,34 @@ modify it for other solutions. ## Instructions -1. Copy this directory to a local directory -2. Edit `azuredisk-v4m.yaml` file if necessary to customize the -storage class that will be used for the deployment +1. Copy this directory to a local directory. + +2. If necessary, edit the `azuredisk-v4m.yaml` file to customize the +storage class that will be used for the deployment. + 3. Edit the hostnames in the sample yaml files in both the `monitoring` -and `logging` subdirectories to match your environment's actual hostname(s) -4. Add additional customization as desired -5. Run the command (adjust to local path created in step 1): -`export USER_DIR=path/to/my/copy/azure-deployment` -6. Run the command: +and `logging` subdirectories to match your environment's actual hostnames. + +4. Add additional customization as desired. + +5. Run this command to set the `USER_DIR` environment variable to the local path (change the command to use local path that you created in step 1): + +```bash +export USER_DIR=path/to/my/copy/azure-deployment +``` + +6. Run this command: + +```bash `kubectl apply -f $USER_DIR/azuredisk-v4m.yaml` -7. Run the scripts to deploy monitoring and logging components in Azure +``` + +7. Run the standard deployment scripts to deploy monitoring and logging components in Azure. ## Access the Applications The monitoring and logging applications in this sample are configured for -path-based ingress and will be available at (replace the hostnames): +path-based ingress and are available at (replace the hostnames for your deployment): * `http://host.mycluster.example.com/grafana` * `http://host.mycluster.example.com/prometheus` diff --git a/samples/azure-monitor/README.md b/samples/azure-monitor/README.md index f77192a2..88777bf0 100644 --- a/samples/azure-monitor/README.md +++ b/samples/azure-monitor/README.md @@ -3,33 +3,29 @@ ## Scrape Prometheus Metrics Endpoints SAS Viya components are natively instrumented to expose a Prometheus-compatible -HTTP(S) metrics endpoint. This sample shows how to configure Azure Monitor to automatically discover and scrape these endpoints. +HTTP or HTTPS metrics endpoint. This sample shows how to configure Azure Monitor to automatically discover and scrape these endpoints. -Follow these steps: +See the [Azure documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-prometheus-integration) +to understand how Azure Monitor discovers and scrapes endpoints. -1. See the [Azure documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-prometheus-integration) -to understand how Azure Monitor discovers and scrapes endpoints. +Follow these steps to enable Azure Monitor for your cluster. -2. Enabling Azure Monitor for your cluster. - -3. Download the +1. Download the [template](https://github.com/microsoft/Docker-Provider/blob/ci_dev/kubernetes/container-azm-ms-agentconfig.yaml) ConfigMap yaml. -4. Customize the `prometheus-data-collection-settings` section. Use the yaml file provided in this sample [here](container-azm-ms-agentconfig.yaml) as a guide. Recommended changes include: - -* `interval` - Update from `1m` to `30s` (recommended, but not required) -* `monitor_kubernetes_pods` - Set to `true`. +2. Customize the `prometheus-data-collection-settings` section. Use the yaml file provided in this sample [here](container-azm-ms-agentconfig.yaml) as a guide. Recommended changes include: -Setting `monitor_kubernetes_pods` to `true` enables Azure Monitor to auto-discover pods to monitor, based on the standard Prometheus annotations. SAS Viya +- `interval` - Update from `1m` to `30s` (recommended, but not required) +- `monitor_kubernetes_pods` - Set to `true`. +- Set `monitor_kubernetes_pods` to `true` to enable Azure Monitor to auto-discover pods to monitor, based on the standard Prometheus annotations. SAS Viya components that expose metrics endpoints should include these annotations: + - `promethues.io/scrape` - `true` or `false` + - `promethues.io/path` - path to metrics endpoint + - `promethues.io/port`- metrics port + - `promethues.io/scheme`- `http` or `https` -* `promethues.io/scrape` - `true` or `false` -* `promethues.io/path` - path to metrics endpoint -* `promethues.io/port`- metrics port -* `promethues.io/scheme`- `http` or `https` - -5. After customizing the template, apply it to the cluster using this command: +3. After customizing the template, apply it to the cluster using this command: ```bash kubectl apply -f /path/to/container-azm-ms-agentconfig.yaml diff --git a/samples/esmulti/README.md b/samples/esmulti/README.md index 1fe35b56..d6a75adc 100644 --- a/samples/esmulti/README.md +++ b/samples/esmulti/README.md @@ -1,10 +1,10 @@ # Multi-Role Elasticsearch Nodes This sample demonstrates how to deploy Elasticsearch so that each of its nodes perform all three roles: primary (also known as master), data and -client/ingest. The sample deploys the nodes as primary (master) nodes and updates them to include the data and client roles. While it is possible to deploy additional data and client nodes, these additional nodes cannot be made multi-role; this sample only supports enabling multi-role for primary nodes. In the standard configuration of this project, each node performs a single node. +client/ingest. The sample deploys the nodes as primary (master) nodes and updates them to include the data and client roles. While it is possible to deploy additional data and client nodes, these additional nodes cannot be made multi-role; this sample only supports enabling multi-role for primary nodes. In the standard configuration of this project, each node performs a single role. Although multi-role Elasticsearch nodes are widely used, they are not natively supported by the Open Distro for Elasticsearch -Helm chart that is used to deploy the logging components of this project. In order to enable multi-role Elasticsearch nodes, our deployment process must patch various Kubernetes objects. Because performance and other criteria are still being evaluated, this use of multi-role nodes is experimental and is not yet part of our standard configuration. +Helm chart that is used to deploy the logging components of this project. In order to enable multi-role Elasticsearch nodes, the deployment process must patch various Kubernetes objects. Because performance and other criteria are still being evaluated, this use of multi-role nodes is experimental and is not yet part of our standard configuration. ## Preparation and Deployment @@ -21,7 +21,6 @@ export USER_DIR=/path/to/my/copy/esmulti 3. Note that the `ES_MULTIROLE_NODES` environment variable has been set to `true` in the `$USER_DIR/logging/user.env` file in order to enable multi-role Elasticsearch nodes. - 4. Review the contents of the `$USER_DIR/logging/user-values-elasticsearch-open.yaml` file and make any changes needed for your environment. These are two areas that could be customized: **Storage:** When primary nodes also serve as data nodes, they need more disk space than when they serve only as primary nodes. The `master.persistence.size` property specifies the amount of storage that is assigned to the primary nodes. Review this setting and adjust it as necessary. diff --git a/samples/external-alertmanager/README.md b/samples/external-alertmanager/README.md index e30b1f22..5c558871 100644 --- a/samples/external-alertmanager/README.md +++ b/samples/external-alertmanager/README.md @@ -1,6 +1,6 @@ -# Using an External AlertManager +# Using an External Alertmanager -In a typical deployment, each cluster uses a separate instance of AlertManager. If you have multiple clusters, you might want alerts from all clusters to come from a single instance of AlertManager. You might also have an existing instance of AlertManager and want alerts to come from the existing instance. Use this sample to configure monitoring to use an external instance of AlertManager. +In a typical deployment, each cluster uses a separate instance of Alertmanager. If you have multiple clusters, you might want alerts from all clusters to come from a single instance of Alertmanager. You might also have an existing instance of Alertmanager and want alerts to come from the existing instance. Use this sample to configure monitoring to use an external instance of Alertmanager. ## Installation @@ -8,25 +8,27 @@ Follow these steps: 1. Copy the directory in the sample to a separate local path. -2. Set the `USER_DIR` environment variable to the local path: +2. Run this command to set the `USER_DIR` environment variable to your local path: ```bash export USER_DIR=/your/path/to/external-alertmanager ``` -3. Define a service that points to the AlertManager instance that you want to use. +3. Define a service that points to the Alertmanager instance that you want to use. -4. Edit `alertmanager-endpoint.yaml` to point to the existing AlertManager -instance, then deploy the yaml file to the monitoring namespace: +4. Edit `alertmanager-endpoint.yaml` to point to the existing Alertmanager +instance. + +5. Deploy the yaml file to the monitoring namespace: ```bash kubectl apply -n monitoring -f $USER_DIR/alertmanager-endpoint.yaml ``` -5. If you changed the service name `my-alertmanager`, make the same change +6. If you changed the service name `my-alertmanager`, make the same change to your copy of `user-values-prom-operator.yaml`. -6. Deploy monitoring using the standard deployment script: +7. Deploy monitoring using the standard deployment script: ```bash monitoring/bin/deploy_monitoring_cluster.sh diff --git a/samples/generic-base/README.md b/samples/generic-base/README.md index 629b7973..2549a71b 100644 --- a/samples/generic-base/README.md +++ b/samples/generic-base/README.md @@ -1,6 +1,8 @@ # Generic Base This sample contains all of the files that can be customized as part of the -monitoring and logging deployments. See the comments in each file for reference links and variable listings. +monitoring and logging deployments. See the comments in each file for +reference links and variable listings. -See the [main README](../../README.md#customization) for information about the customization process. +See the [main README](../../README.md#customization) for information about +the customization process. diff --git a/samples/generic-base/monitoring/user-values-pushgateway.yaml b/samples/generic-base/monitoring/user-values-pushgateway.yaml index 7fffb5f5..90dc1e8e 100644 --- a/samples/generic-base/monitoring/user-values-pushgateway.yaml +++ b/samples/generic-base/monitoring/user-values-pushgateway.yaml @@ -1,10 +1,6 @@ # Chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-pushgateway # Default values: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus-pushgateway/values.yaml -# image: -# repository: prom/pushgateway -# tag: v1.2.0 - # persistentVolume: # enabled: true # size: 2Gi diff --git a/samples/generic-base/monitoring/user.env b/samples/generic-base/monitoring/user.env index 96876416..bb9e0e1e 100644 --- a/samples/generic-base/monitoring/user.env +++ b/samples/generic-base/monitoring/user.env @@ -12,15 +12,6 @@ # Deploy components with TLS enabled # TLS_ENABLE=false -# USER_DIR - enables the ability to locate user-customized files -# outside of this repository - for example, to check in environment-specific -# customimzations into a separate Git repository. -# -# USER_DIR must refer to a directory. The directory can include any or all of -# the following files in the structure below. Additional content is allowed, but -# will be ignored. If unset, the root of this repository is used. -# USER_DIR= - # deploy_monitoring_cluster.sh options # ------------------------------------ # Cluster monitoring namespace for the Prometheus Operator, @@ -31,10 +22,10 @@ # This setting overrides TLS_ENABLE only for monitoring # MON_TLS_ENABLE=false -# Enable TLS (HTTPS) for monitoring components -# WARNING: Not fully working yet - for development/testing only -# See samples/tls for related customizations needed for TLS support -# MON_TLS_ENABLE=false +# Enables tolerations and pod affinity to enable the monitoring +# components to participate in the SAS Viya workload node +# placement strategy +# MON_NODE_PLACEMENT_ENABLE=false # Set to true to force an update of the Prometheus Operator CRDs # PROM_OPERATOR_CRD_UPDATE=false @@ -44,15 +35,10 @@ # match the value of prometheusOperator.image.tag in the helm YAML # if changed from the default. # See https://github.com/prometheus-operator/prometheus-operator/releases -# PROM_OPERATOR_CRD_VERSION=v0.43.0 +# PROM_OPERATOR_CRD_VERSION=v0.43.2 # Version of the kube-prometheus-stack helm chart to use -# KUBE_PROM_STACK_CHART_VERSION=11.0.0 - -# Enables tolerations and pod affinity to enable the monitoring -# components to participate in the SAS Viya workload node -# placement strategy -# MON_NODE_PLACEMENT_ENABLE=false +# KUBE_PROM_STACK_CHART_VERSION=11.1.3 # Namespace of NGINX ingress controller (if applicable) # NGINX_NS=ingress-nginx @@ -67,7 +53,6 @@ # PGMONITOR_DASH=true # RABBITMQ_DASH=true # LOGGING_DASH=true -# ISTIO_DASH=false ## deploy_monitoring_viya.sh options # ---------------------------------- diff --git a/samples/generic-base/user.env b/samples/generic-base/user.env index 41818aca..02883f56 100644 --- a/samples/generic-base/user.env +++ b/samples/generic-base/user.env @@ -7,6 +7,8 @@ # LOG_LEVEL_ENABLE=true # LOG_COLOR_ENABLE=true +# Enables in-cluster service-to-service TLS for monitoring and +# logging components. See the respective README for details. # TLS_ENABLE=false # Enables tolerations and pod affinity to enable the monitoring diff --git a/samples/ingress/README.md b/samples/ingress/README.md index b1ff1d97..530b6b11 100644 --- a/samples/ingress/README.md +++ b/samples/ingress/README.md @@ -1,7 +1,7 @@ # Ingress This sample demonstrates how to deploy monitoring and logging components -configured with ingress instead of node ports. You can choose to use host-based routing or path-based routing. Use host-based routing if the monitoring applications (Prometheus, AlertManager, and Grafana) will be on different hosts. Use path-based routing if the applications use different paths on the same host. +configured with ingress instead of node ports. You can choose to use host-based routing or path-based routing. Use host-based routing if the monitoring applications (Prometheus, Alertmanager, and Grafana) will be on different hosts. Use path-based routing if the applications use different paths on the same host. If you are using a cloud provider, you must deploy using ingress. @@ -24,10 +24,10 @@ Follow these steps to deploy monitoring using ingress: 1. The monitoring deployment process requires that the user response file be named `$USER_DIR/monitoring/user-values-prom-operator.yaml`. -If you are using host-based routing, rename the + - If you are using host-based routing, rename the `user-values-prom-host.yaml` to `user-values-prom-operator.yaml`. -If you are using path-based routing, rename the `user-values-prom-path.yaml` to `user-values-prom-operator.yaml`. + - If you are using path-based routing, rename the `user-values-prom-path.yaml` to `user-values-prom-operator.yaml`. 2. Edit `$USER_DIR/monitoring/user-values-prom-operator.yaml` and replace all instances of `host.cluster.example.com` with hostnames that match your cluster. @@ -45,10 +45,10 @@ Follow these steps to deploy logging using ingress: 1. The logging deployment process requires that the user response file be named `$USER_DIR/logging/user-values-elasticsearch-open.yaml`. -If you are using host-based routing, rename the + - If you are using host-based routing, rename the `user-values-elasticsearch-host.yaml` to `user-values-elasticsearch-open.yaml`. -If you are using path-based routing, rename the `user-values-elasticsearch-path.yaml` to `user-values-elasticsearch-open.yaml`. + - If you are using path-based routing, rename the `user-values-elasticsearch-path.yaml` to `user-values-elasticsearch-open.yaml`. 2. Edit `$USER_DIR/logging/user-values-elasticsearch-open.yaml` and replace all instances of `host.cluster.example.com` with hostnames that match your cluster @@ -66,7 +66,7 @@ locations (with hostnames replaced with those in the actual environment that you * Grafana - `https://grafana.host.mycluster.example.com` * Prometheus - `https://prometheus.host.mycluster.example.com` -* AlertManager - `https://alertmanager.host.mycluster.example.com` +* Alertmanager - `https://alertmanager.host.mycluster.example.com` * Kibana - `https://kibana.host.mycluster.example.com` If you deploy using path-based ingress, the applications are available at these @@ -74,7 +74,7 @@ locations (with hostnames replaced with those in the actual environment that you * Grafana - `http://host.mycluster.example.com/grafana` * Prometheus - `http://host.mycluster.example.com/prometheus` -* AlertManager - `http://host.mycluster.example.com/alertManager` +* Alertmanager - `http://host.mycluster.example.com/alertManager` * Kibana - `http://host.mycluster.example.com/kibana` The default credentials for Grafana and Kibana are `admin`:`admin`. diff --git a/samples/namespace-monitoring/README.md b/samples/namespace-monitoring/README.md index 962bc96a..d0f7b75a 100644 --- a/samples/namespace-monitoring/README.md +++ b/samples/namespace-monitoring/README.md @@ -17,8 +17,8 @@ their respective SAS Viya namespaces. All resources in this sample are configured for host-based ingress. In this example, all three Prometheus instances share the same -instance of AlertManager, to demonstrate how you can centralize alerts. You -can use AlertManager CRs to deploy a separate AlertManager for each instance +instance of Alertmanager, in order to demonstrate how you can centralize alerts. You +can use Alertmanager CRs to deploy a separate Alertmanager for each instance of Prometheus. This sample assumes that you are deploying two SAS Viya namespaces, but you can @@ -70,7 +70,7 @@ kubectl label ns $VIYA_TWO_NS sas.com/viya-namespace=$VIYA_TWO_NS VIYA_DASH=false monitoring/bin/deploy_monitoring_cluster.sh ``` -8. Deploy standard SAS Viya monitoring components each Viya namespace. +8. Deploy standard SAS Viya monitoring components each SAS Viya namespace. ```bash VIYA_NS=$VIYA_ONE_NS monitoring/bin/deploy_monitoring_viya.sh VIYA_NS=$VIYA_ONE_NS monitoring/bin/deploy_monitoring_viya.sh diff --git a/samples/tls/logging/README.md b/samples/tls/logging/README.md index aa1b53a0..5b198efb 100644 --- a/samples/tls/logging/README.md +++ b/samples/tls/logging/README.md @@ -17,15 +17,13 @@ you run any of the scripts in this repository: * kubernetes.io/tls secret - `elasticsearch-ingress-tls-secret` Generating these certificates is outside the scope of this example. However, you -can use the process documented in ["Configure NGINX Ingress TLS for SAS -Applications"](https://go.documentation.sas.com/?cdcId=sasadmincdc&cdcVersion=default&docsetId=calencryptmotion&docsetTarget=n1xdqv1sezyrahn17erzcunxwix9.htm&locale=en#n0oo2yu8440vmzn19g6xhx4kfbrq) in SAS Viya Administration documentation and specify the `logging` namespace. +can use the process documented in ["Configure NGINX Ingress TLS for SAS Applications"](https://go.documentation.sas.com/?cdcId=sasadmincdc&cdcVersion=default&docsetId=calencryptmotion&docsetTarget=n1xdqv1sezyrahn17erzcunxwix9.htm&locale=en#n0oo2yu8440vmzn19g6xhx4kfbrq) in SAS Viya Administration documentation and specify the `logging` namespace. ## Instructions -1. Set up an empty directory with a `logging` subdirectory to contain the customization files. +1. Set up an empty directory with a `/logging` subdirectory to contain the customization files. -2. Export a `USER_DIR` environment variable that points to this -location. For example: +2. Set the `USER_DIR` environment variable to your local path: ```bash mkdir -p ~/my-cluster-files/ops/user-dir/logging @@ -44,7 +42,7 @@ cp path/to/this/repo/loggingsamples/tls/user-values-elasticsearch-open.yaml $USE 5. Edit `$USER_DIR/monitoring/user-values-elasticsearch-open.yaml` and replace any sample hostnames with hostnames for your deployment. Specifically, you must replace -`host.cluster.example.com` with the name of the ingress node. Often, the ingress node is the cluster master node, but environments vary. +`host.cluster.example.com` with the name of the ingress node. Often, the ingress node is the cluster master node, but your environment might be different. 6. Specify any other customizations in `user-values-elasticsearch-open.yaml`. diff --git a/samples/tls/monitoring/README.md b/samples/tls/monitoring/README.md index 133db56e..68e3e3eb 100644 --- a/samples/tls/monitoring/README.md +++ b/samples/tls/monitoring/README.md @@ -26,10 +26,9 @@ For in-cluster (east-west traffic) TLS for monitoring components, ## Instructions -1. Set up an empty directory with a `monitoring` subdirectory to contain the customization files. +1. Set up an empty directory with a `/monitoring` subdirectory to contain the customization files. -2. Export a `USER_DIR` environment variable that points to this -location. For example: +2. Set the `USER_DIR` environment variable to your local path: ```bash mkdir -p ~/my-cluster-files/ops/user-dir/monitoring @@ -39,7 +38,7 @@ export USER_DIR=~/my-cluster-files/ops/user-dir 3. Create `$USER_DIR/monitoring/user.env`. Specify this value in the file: * `MON_TLS_ENABLE=true` - This flag modifies the deployment of Prometheus, -Grafana, and AlertManager to be TLS-enabled. +Grafana, and Alertmanager to be TLS-enabled. 4. Copy the sample TLS Helm user response file to your `USER_DIR`: @@ -49,7 +48,7 @@ cp path/to/this/repo/monitoring/samples/tls/user-values-prom-operator.yaml $USER 5. Edit `$USER_DIR/monitoring/user-values-prom-operator.yaml` and replace any sample hostnames with hostnames for your deployment. Specifically, you must replace -`host.cluster.example.com` with the name of the ingress node. Often, the ingress node is the cluster master node, but environments vary. +`host.cluster.example.com` with the name of the ingress node. Often, the ingress node is the cluster master node, but your environment might be different. 6. Specify any other customizations in `user-values-prom-operator.yaml`. @@ -61,16 +60,16 @@ path/to/this/repo/monitoring/bin/deploy_monitoring_cluster.sh ## Limitations and Known Issues -* There is a [bug in the AlertManager Helm template](https://github.com/helm/charts/issues/22939) -that prevents mounting the TLS certificates for the reverse proxy sidecar. -It is expected that this issue will be addressed before general availability of SAS Viya 4. HTTPS is still -supported for AlertManager at the ingress level, but it is not supported for the pod (in-cluster). +* There is a [bug in the Prometheus template](https://github.com/prometheus-community/helm-charts/issues/152) +that prevents mounting the TLS certificates for the reverse proxy sidecar for Alertmanager. +HTTPS is still +supported for Alertmanager at the ingress level, but it is not supported for the pod (in-cluster). * The Prometheus node exporter and kube-state-metrics exporters do not currently support TLS. These components are not exposed over ingress, so in-cluster -access will be over HTTP and not HTTPS. +access is over HTTP and not HTTPS. -* If needed, a self-signed cert-manager Issuer is created that generates +* If needed, a self-signed cert-manager issuer is created that generates self-signed certificates when TLS is enabled and the secrets do not already exist. By default, in-cluster traffic between monitoring components (not ingress) is configured to skip TLS CA verification.