From 7e200d847657929617f35cf665896d7df9c2fd83 Mon Sep 17 00:00:00 2001 From: danmacpherson Date: Mon, 14 Oct 2024 06:04:59 +0000 Subject: [PATCH] deploy: c322796be573f58f337f27d14956249cbb4f7c31 --- blog/2021-12-31-medical-diagnosis/index.html | 3 ++- blog/2022-03-23-acm-mustonlyhave/index.html | 3 ++- blog/2022-03-30-multicloud-gitops/index.html | 3 ++- blog/2022-06-30-ansible-edge-gitops/index.html | 3 ++- blog/2022-07-15-push-vs-pull/index.html | 3 ++- blog/2022-08-24-clustergroups/index.html | 3 ++- blog/2022-09-02-route/index.html | 3 ++- blog/2022-10-12-acm-provisioning/index.html | 3 ++- blog/2022-11-20-argo-rollouts/index.html | 3 ++- blog/2022-12-01-multicluster-devsecops/index.html | 3 ++- blog/2023-11-17-argo-configmanagement-plugins/index.html | 3 ++- blog/2023-12-01-new-pattern-tiers/index.html | 3 ++- blog/2023-12-05-nutanix-testing/index.html | 3 ++- blog/2023-12-15-understanding-namespaces/index.html | 3 ++- blog/2023-12-20-private-repos/index.html | 3 ++- .../index.html | 3 ++- blog/2024-01-26-more-secrets-options/index.html | 3 ++- blog/2024-02-07-hcp-htpasswd-config/index.html | 3 ++- blog/2024-03-05-intel-accelerated-patterns/index.html | 3 ++- blog/2024-07-12-in-cluster-git/index.html | 3 ++- blog/2024-07-19-write-token-kubeconfig/index.html | 3 ++- blog/2024-08-30-push-secrets/index.html | 3 ++- blog/2024-09-13-using-hypershift/index.html | 3 ++- blog/2024-09-26-slimming-of-common/index.html | 3 ++- blog/2024-10-12-disconnected/index.html | 3 ++- contribute/background-on-pattern-development/index.html | 3 ++- contribute/contribute-to-docs/index.html | 3 ++- contribute/creating-a-pattern/index.html | 3 ++- contribute/extending-a-pattern/index.html | 3 ++- contribute/support-policies/index.html | 3 ++- learn/about-pattern-tiers-types/index.html | 3 ++- learn/about-validated-patterns/index.html | 3 ++- learn/clustergroup-in-values-files/index.html | 3 ++- learn/faq/index.html | 3 ++- learn/implementation/index.html | 3 ++- learn/importing-a-cluster/index.html | 3 ++- learn/infrastructure/index.html | 3 ++- learn/keyconcepts/index.html | 3 ++- learn/maintained/index.html | 3 ++- learn/ocp-cluster-general-sizing/index.html | 3 ++- learn/quickstart/index.html | 3 ++- learn/sandbox/index.html | 3 ++- .../index.html | 3 ++- learn/secrets/index.html | 3 ++- learn/test-artifacts/index.html | 3 ++- learn/tested/index.html | 3 ++- learn/using-validated-pattern-operator/index.html | 3 ++- learn/validated_patterns_frameworks/index.html | 3 ++- learn/values-files/index.html | 3 ++- learn/vault/index.html | 3 ++- learn/vp_openshift_framework/index.html | 3 ++- learn/vp_structure_vp_pattern/index.html | 3 ++- learn/workflow/index.html | 3 ++- .../ansible-edge-gitops/ansible-automation-platform/index.html | 3 ++- patterns/ansible-edge-gitops/cluster-sizing/index.html | 3 ++- patterns/ansible-edge-gitops/getting-started/index.html | 3 ++- .../ansible-edge-gitops/ideas-for-customization/index.html | 3 ++- patterns/ansible-edge-gitops/index.html | 3 ++- patterns/ansible-edge-gitops/installation-details/index.html | 3 ++- .../ansible-edge-gitops/openshift-virtualization/index.html | 3 ++- patterns/ansible-edge-gitops/troubleshooting/index.html | 3 ++- patterns/ansible-gitops-framework/index.html | 3 ++- patterns/cockroachdb/index.html | 3 ++- patterns/connected-vehicle-architecture/index.html | 3 ++- patterns/devsecops/cluster-sizing/index.html | 3 ++- patterns/devsecops/devel-cluster/index.html | 3 ++- patterns/devsecops/getting-started/index.html | 3 ++- patterns/devsecops/ideas-for-customization/index.html | 3 ++- patterns/devsecops/index.html | 3 ++- patterns/devsecops/production-cluster/index.html | 3 ++- patterns/devsecops/secure-supply-chain-demo/index.html | 3 ++- .../emerging-disease-detection/edd-getting-started/index.html | 3 ++- patterns/emerging-disease-detection/index.html | 3 ++- .../gaudi-rag-chat-qna-getting-started/index.html | 3 ++- .../gaudi-rag-chat-qna-required-hardware/index.html | 3 ++- .../gaudi-rag-chat-qna-troubleshooting/index.html | 3 ++- patterns/gaudi-rag-chat-qna/index.html | 3 ++- patterns/hypershift/index.html | 3 ++- patterns/industrial-edge/application/index.html | 3 ++- patterns/industrial-edge/cluster-sizing/index.html | 3 ++- patterns/industrial-edge/demo-script/index.html | 3 ++- patterns/industrial-edge/factory/index.html | 3 ++- patterns/industrial-edge/getting-started/index.html | 3 ++- patterns/industrial-edge/ideas-for-customization/index.html | 3 ++- patterns/industrial-edge/index.html | 3 ++- patterns/industrial-edge/troubleshooting/index.html | 3 ++- patterns/kong-gateway/index.html | 3 ++- patterns/medical-diagnosis-amx/cluster-sizing/index.html | 3 ++- patterns/medical-diagnosis-amx/getting-started/index.html | 3 ++- .../medical-diagnosis-amx/ideas-for-customization/index.html | 3 ++- patterns/medical-diagnosis-amx/index.html | 3 ++- patterns/medical-diagnosis-amx/troubleshooting/index.html | 3 ++- patterns/medical-diagnosis/cluster-sizing/index.html | 3 ++- patterns/medical-diagnosis/demo-script/index.html | 3 ++- patterns/medical-diagnosis/getting-started/index.html | 3 ++- patterns/medical-diagnosis/ideas-for-customization/index.html | 3 ++- patterns/medical-diagnosis/index.html | 3 ++- patterns/medical-diagnosis/troubleshooting/index.html | 3 ++- patterns/mlops-fraud-detection/index.html | 3 ++- patterns/mlops-fraud-detection/mfd-getting-started/index.html | 3 ++- patterns/mlops-fraud-detection/mfd-running-the-demo/index.html | 3 ++- patterns/multicloud-gitops-amx-rhoai/index.html | 3 ++- .../mcg-amx-rhoai-bert-script/index.html | 3 ++- .../mcg-amx-rhoai-cluster-sizing/index.html | 3 ++- .../mcg-amx-rhoai-demo-script/index.html | 3 ++- .../mcg-amx-rhoai-getting-started/index.html | 3 ++- .../mcg-amx-rhoai-ideas-for-customization/index.html | 3 ++- .../mcg-amx-rhoai-imperative-actions/index.html | 3 ++- .../mcg-amx-rhoai-managed-cluster/index.html | 3 ++- patterns/multicloud-gitops-amx/index.html | 3 ++- .../multicloud-gitops-amx/mcg-amx-cluster-sizing/index.html | 3 ++- .../multicloud-gitops-amx/mcg-amx-getting-started/index.html | 3 ++- .../mcg-amx-ideas-for-customization/index.html | 3 ++- .../mcg-amx-imperative-actions/index.html | 3 ++- .../multicloud-gitops-amx/mcg-amx-managed-cluster/index.html | 3 ++- patterns/multicloud-gitops-portworx/cluster-sizing/index.html | 3 ++- patterns/multicloud-gitops-portworx/getting-started/index.html | 3 ++- .../ideas-for-customization/index.html | 3 ++- patterns/multicloud-gitops-portworx/index.html | 3 ++- patterns/multicloud-gitops-portworx/managed-cluster/index.html | 3 ++- patterns/multicloud-gitops-qat/index.html | 3 ++- .../multicloud-gitops-qat/mcg-qat-cluster-sizing/index.html | 3 ++- .../multicloud-gitops-qat/mcg-qat-getting-started/index.html | 3 ++- .../mcg-qat-ideas-for-customization/index.html | 3 ++- .../mcg-qat-imperative-actions/index.html | 3 ++- .../multicloud-gitops-qat/mcg-qat-managed-cluster/index.html | 3 ++- patterns/multicloud-gitops-sgx-hello-world/index.html | 3 ++- .../mcg-sgx-hello-world-cluster-sizing/index.html | 3 ++- .../mcg-sgx-hello-world-demo-script/index.html | 3 ++- .../mcg-sgx-hello-world-getting-started/index.html | 3 ++- .../mcg-sgx-hello-world-ideas-for-customization/index.html | 3 ++- .../mcg-sgx-hello-world-imperative-actions/index.html | 3 ++- .../mcg-sgx-hello-world-managed-cluster/index.html | 3 ++- patterns/multicloud-gitops-sgx/index.html | 3 ++- .../multicloud-gitops-sgx/mcg-sgx-cluster-sizing/index.html | 3 ++- patterns/multicloud-gitops-sgx/mcg-sgx-demo-script/index.html | 3 ++- .../multicloud-gitops-sgx/mcg-sgx-getting-started/index.html | 3 ++- .../mcg-sgx-ideas-for-customization/index.html | 3 ++- .../mcg-sgx-imperative-actions/index.html | 3 ++- .../multicloud-gitops-sgx/mcg-sgx-managed-cluster/index.html | 3 ++- patterns/multicloud-gitops/index.html | 3 ++- patterns/multicloud-gitops/mcg-cluster-sizing/index.html | 3 ++- patterns/multicloud-gitops/mcg-demo-script/index.html | 3 ++- patterns/multicloud-gitops/mcg-getting-started/index.html | 3 ++- .../multicloud-gitops/mcg-ideas-for-customization/index.html | 3 ++- patterns/multicloud-gitops/mcg-imperative-actions/index.html | 3 ++- patterns/multicloud-gitops/mcg-managed-cluster/index.html | 3 ++- patterns/openshift-ai/getting-started/index.html | 3 ++- patterns/openshift-ai/index.html | 3 ++- patterns/rag-llm-gitops/getting-started/index.html | 3 ++- patterns/rag-llm-gitops/gpu_provisioning/index.html | 3 ++- patterns/rag-llm-gitops/index.html | 3 ++- patterns/retail/application/index.html | 3 ++- patterns/retail/cluster-sizing/index.html | 3 ++- patterns/retail/components/index.html | 3 ++- patterns/retail/getting-started/index.html | 3 ++- patterns/retail/ideas-for-customization/index.html | 3 ++- patterns/retail/index.html | 3 ++- patterns/retail/store/index.html | 3 ++- patterns/retail/troubleshooting/index.html | 3 ++- patterns/travelops/demo-script/index.html | 3 ++- patterns/travelops/getting-started/index.html | 3 ++- patterns/travelops/ideas-for-customization/index.html | 3 ++- patterns/travelops/index.html | 3 ++- 164 files changed, 328 insertions(+), 164 deletions(-) diff --git a/blog/2021-12-31-medical-diagnosis/index.html b/blog/2021-12-31-medical-diagnosis/index.html index 3a353b991..ce82b6541 100644 --- a/blog/2021-12-31-medical-diagnosis/index.html +++ b/blog/2021-12-31-medical-diagnosis/index.html @@ -9,7 +9,8 @@ deploying and the xraylab dashboard being available via its route.

The charts for the pattern deployment are located: $GIT_REPO_DIR/charts/datacenter/

Pattern Deployed Technology

OperatorUpstream Project
openshift data foundation (odf)ceph, rook, noobaa
openshift-gitopsargoCD
openshift serverlessknative
amq streamskafka
opendatahubopendatahub
grafanagrafana

Challenges

With the imperative dependence on the originating content, there were some resources that didn’t align 1:1 and needed to be overcome. For example, there are a number of tasks that are interrogating the cluster for information to transform into a variable and finally apply that variable to some resource. As you can imagine, this can be very challenging when you’re declaring the state of your cluster. In order to -maneuver around these imperative actions we took what we could and created openshift jobs to execute the task.

Conclusion

Speed, accuracy, efficiency all come to mind when considering what this pattern provides. Patients get the treatment they need, when they need it because we’re able to use technology to quickly and accurately diagnosis anomalies detected in X-rays. The validated patterns framework enables administrators to quickly meet their user demands by providing solutions that only require them to bring their own data to complete the last 20-25% of the architecture.

QUICKLINKS

QUICKLINKS

CONTRIBUTE

Changing the complianceType to mustonlyhave fixed the issue as it enforced the template fully on the remote cluster and solved this issue.

Thanks

Special thanks to Ilkka Tengvall and Christian Stark for their help and patience.

QUICKLINKS

Changing the complianceType to mustonlyhave fixed the issue as it enforced the template fully on the remote cluster and solved this issue.

Thanks

Special thanks to Ilkka Tengvall and Christian Stark for their help and patience.

QUICKLINKS

CONTRIBUTE

Multi-Cloud GitOps

by Martin Jackson
March 30, 2022
multi-cloud gitops xray -validated-pattern

Validated Pattern: Multi-Cloud GitOps

Validated Patterns: The Story so far

Our first foray into the realm of Validated Patterns was the adaptation of the MANUela application and its associated tooling to ArgoCD and Tekton, to demonstrate the deployment of a fairly involved IoT application designed to monitor industrial equipment and use AI/ML techniques to predict failure. This resulted in the Industrial Edge validated pattern, which you can see here.

This was our first use of a framework to deploy a significant application, and we learned a lot by doing it. It was good to be faced with a number of problems in the “real world” before taking a look at what is really essential for the framework and why.

All patterns have at least two parts: A “common” element (which we expect to be the basic framework that nearly all of our patterns will share) and a pattern-specific element, which uses the common pattern and expands on it with pattern-specific content. In the case of Industrial Edge, the common component included secret handling, installation of the GitOps operator, and installation of Red Hat Advanced Cluster Management. The pattern-specific components included the OpenShift Pipelines Operator, the AMQ Broker and Streams operators, Camel-K, the Seldon Operator, OpenDataHub, and Jupyter Notebooks and S3 storage buckets.

Multi-Cloud GitOps: The Why and What

After finishing with Industrial Edge, we recognized that there were some specific areas that we needed to tell a better story in. There were several areas where we thought we could improve the user experience in working with our tools and repos. And we recognized that the pattern might be a lot clearer in form and design to those of us who worked on it than an interested user from the open Internet.

So we had several categories of work to do as we scoped this pattern:

  • Make a clear starting point: Make a clear “entry point” into pattern development, and define the features that we think should be common to all patterns. This pattern should be usable as a template for both us and other users to be able to clone as a starting point for future pattern development.
  • Make “common” common: Since this pattern is going to be foundational to future patterns, remove elements from the common framework that are not expected to be truly common to all future patterns (or at least a large subset of them). Many elements specific to Industrial Edge found their way into common; in some cases we thought those elements truly were common and later re-thought them.
  • Improve secrets handling: Provide a secure credential store such that we can manage secrets in that store rather than primarily as YAML files on a developer workstation. Broker access to that secret store via the External Secrets Operator to ensure a level of modularity and allow users to choose different secret stores if they wish. We also want to integrate the usage of that secret store into the cluster and demonstrate how to use it.
  • Improve support for cluster scalability: For the Industrial Edge pattern, the edge cluster was technically optional (we have a supported model where both the datacenter and factory applications can run on the same cluster). We want these patterns to be more clearly scalable, and we identified two categories of that kind of scalability:
  1. Clusters vary only in name, but run the same applications and the same workloads
  2. Clusters vary in workloads, sizing, and configuration, and allow for considerable variability.

Many of the elements needed to support these were present in the initial framework, but it may not have been completely clear how to use these features, or they were couched in terms that only made sense to the people who worked on the pattern. This will now be clearer for future patterns, and we will continue to evolve the model with user and customer feedback.

Key Learning: Submodules are hard, and probably not worth it

  1. Copy and Paste
  2. Git Submodules
  3. Git Subtrees

We rejected the notion of copy and paste because we reasoned that once patterns diverged in their “common” layer it would be too difficult and painful to bring them back together later. More importantly, there would be no motivation to do so.

In Industrial Edge, we decided to make common a git submodule. Git submodules have been a feature of git for a long time, originally intended to make compiling a large project with multiple libraries more straightforward, by having a parent repo and an arbitrary set of submodule repos. Git submodule requires a number of exceptions to the “typical” git workflow - the initial clone works differently, and keeping the submodule updated to date can trip users up. Most importantly, it requires the practical management of multiple repositories, which can make life difficult in disconnected environments, which are important to us to support. It was confusing for our engineers to understand how to contribute code to the submodule repository. Finally, user response to the exceptions they had to make because of submodules was universally negative.

So going forward, because it is still important to have a common basis for patterns, and a clear mechanism and technical path to get updates to the common layer, we have moved to the subtree model as a mechanism for including common. This allows consumers of the pattern to treat the repo as a single entity instead of two, and does not require special syntax or commands to be run when switching branches, updating or, in many cases, contributing to common itself.

Key Learning: Secrets Management

One of our biggest challenges in following GitOps principles for the deployment of workloads is the handling of secrets. GitOps tells us that the git repo should be the source of truth - but we know that we should not store secrets directly in publicly accessible repositories. Previously, our patterns standardized the use of YAML files on the developer workstation as the de-facto authoritative secret store. This can be problematic for at least two reasons: for one, if two people are working on the same repo, which secret store is “right”? Secondly, it might be easier to retrieve credentials from a developer workstation due to information breach or theft. Our systems will not work without secrets, and we need to have a better way of working with them.

Highlight: Multi-cloud GitOps is the “Minimum Viable Pattern”

One of the goals of this pattern is to provide the minimal pattern that both demonstrates the goals, aims and purpose of the framework and does something that we think users will find interesting and valuable. We plan to use it as a starting point for our own future pattern development; and as such we can point to it as the pattern to clone and start with if a user wants to start their own pattern development from scratch.

New Feature: Hub Cluster Vault instance

In this pattern, we introduce the ability to reference upstream helm charts, and pass overrides to them in a native ArgoCD way. The first application we are treating this way is Hashicorp Vault. The use of Vault also allows us to make Vault the authoritative source of truth for secrets in the framework. This also improves our security posture by making it significantly easier to rotate secrets and have OpenShift “do the right thing” by re-deploying and re-starting workloads as necessary.

For the purposes of shipping this pattern as a runnable demonstration, we take certain shortcuts with security that we understand are not best practices - storing the vault keys unencrypted on a developer drive, for example. If you intend to run code derived from this pattern in production, we strongly recommend you consider and follow the practices documented here.

New Feature: External Secrets Operator

While it is important to improve the story for secret handling in the pattern space overall, it is also important to provide space for multiple solutions inside patterns. Because of this, we include the external secrets operator, which in the pattern uses vault, but can be used to support a number of other secret providers, and can be extended to support secrets providers that it does not already support. Furthermore, the external secrets approach is less disruptive to existing applications, since it works by managing the secret objects applications are already used to consuming. Vault provides different options for integration, including the agent injector, but this approach is very specific to Vault and not clearly portable.

In a similar note to the feature about Vault and secrets: the approach we take in this release of the pattern has some known security deficiencies. In RHACM prior to 2.5, policies containing secrets will not properly cloak the secrets in the policy objects, and will not properly encrypt the secrets at rest. RHACM 2.5+ includes a fromSecret function that will secure these secrets in transit and at rest in both of these ways. (Of course, any entity with cluster admin access can recover the contents of a secret object in the cluster.) One additional deficiency of this approach is that the lookup function we use in the policies to copy secrets only runs when the policy object is created or refreshed - which means there is not a mechanism within RHACM presently to detect when a secret has changed and the policy needs to be refreshed. We are hoping this functionality will be included in RHACM 2.6.

New Feature: clusterGroups can have multiple cluster members

Using Advanced Cluster Management, we can inject per-cluster configuration into the ArgoCD application definition. We do this, for example, with the global.hubClusterDomain and global.localClusterDomain variables, which are available to use in helm templates in applications that use the framework.

This enables one of our key new features, the ability to deploy multiple clusters that differ only in local, cluster-defined ways (such as the FQDNs that they would publish for their routes). This is a need we determined when we were working on Industrial Edge, where we had to add the FQDN of the local cluster to a config map, for use in a browser application that was defined in kubernetes, but runs in a user’s browser.

The config-demo namespace uses a deployment of the Red Hat Universal Base Image of httpd to demonstrate how to use the framework to pass variables from application definition to actual use in config maps. The config-demo app shows the management of a secret defined and securely transferred from the hub cluster to remote clusters, as well as allowing for the use of the hub cluster base domain and the local cluster base domain in configuration of applications running on either the hub or managed clusters.

Where do we go from here?

One of the next things we are committed to delivering in the new year is a pattern to extend the concept of GitOps to include elements that are outside of OpenShift and Kubernetes - specifically Red Hat Enterprise Linux nodes, including Red Hat Enterprise Linux For Edge nodes, as well as Red Hat Ansible Automation Platform.

We plan on developing a number of new patterns throughout the new year, which will showcase various technologies. Keep watching this space for updates, and if you would like to get involved, visit our site at https://validatedpatterns.io!

QUICKLINKS

Validated Pattern: Multi-Cloud GitOps

Validated Patterns: The Story so far

Our first foray into the realm of Validated Patterns was the adaptation of the MANUela application and its associated tooling to ArgoCD and Tekton, to demonstrate the deployment of a fairly involved IoT application designed to monitor industrial equipment and use AI/ML techniques to predict failure. This resulted in the Industrial Edge validated pattern, which you can see here.

This was our first use of a framework to deploy a significant application, and we learned a lot by doing it. It was good to be faced with a number of problems in the “real world” before taking a look at what is really essential for the framework and why.

All patterns have at least two parts: A “common” element (which we expect to be the basic framework that nearly all of our patterns will share) and a pattern-specific element, which uses the common pattern and expands on it with pattern-specific content. In the case of Industrial Edge, the common component included secret handling, installation of the GitOps operator, and installation of Red Hat Advanced Cluster Management. The pattern-specific components included the OpenShift Pipelines Operator, the AMQ Broker and Streams operators, Camel-K, the Seldon Operator, OpenDataHub, and Jupyter Notebooks and S3 storage buckets.

Multi-Cloud GitOps: The Why and What

After finishing with Industrial Edge, we recognized that there were some specific areas that we needed to tell a better story in. There were several areas where we thought we could improve the user experience in working with our tools and repos. And we recognized that the pattern might be a lot clearer in form and design to those of us who worked on it than an interested user from the open Internet.

So we had several categories of work to do as we scoped this pattern:

  • Make a clear starting point: Make a clear “entry point” into pattern development, and define the features that we think should be common to all patterns. This pattern should be usable as a template for both us and other users to be able to clone as a starting point for future pattern development.
  • Make “common” common: Since this pattern is going to be foundational to future patterns, remove elements from the common framework that are not expected to be truly common to all future patterns (or at least a large subset of them). Many elements specific to Industrial Edge found their way into common; in some cases we thought those elements truly were common and later re-thought them.
  • Improve secrets handling: Provide a secure credential store such that we can manage secrets in that store rather than primarily as YAML files on a developer workstation. Broker access to that secret store via the External Secrets Operator to ensure a level of modularity and allow users to choose different secret stores if they wish. We also want to integrate the usage of that secret store into the cluster and demonstrate how to use it.
  • Improve support for cluster scalability: For the Industrial Edge pattern, the edge cluster was technically optional (we have a supported model where both the datacenter and factory applications can run on the same cluster). We want these patterns to be more clearly scalable, and we identified two categories of that kind of scalability:
  1. Clusters vary only in name, but run the same applications and the same workloads
  2. Clusters vary in workloads, sizing, and configuration, and allow for considerable variability.

Many of the elements needed to support these were present in the initial framework, but it may not have been completely clear how to use these features, or they were couched in terms that only made sense to the people who worked on the pattern. This will now be clearer for future patterns, and we will continue to evolve the model with user and customer feedback.

Key Learning: Submodules are hard, and probably not worth it

  1. Copy and Paste
  2. Git Submodules
  3. Git Subtrees

We rejected the notion of copy and paste because we reasoned that once patterns diverged in their “common” layer it would be too difficult and painful to bring them back together later. More importantly, there would be no motivation to do so.

In Industrial Edge, we decided to make common a git submodule. Git submodules have been a feature of git for a long time, originally intended to make compiling a large project with multiple libraries more straightforward, by having a parent repo and an arbitrary set of submodule repos. Git submodule requires a number of exceptions to the “typical” git workflow - the initial clone works differently, and keeping the submodule updated to date can trip users up. Most importantly, it requires the practical management of multiple repositories, which can make life difficult in disconnected environments, which are important to us to support. It was confusing for our engineers to understand how to contribute code to the submodule repository. Finally, user response to the exceptions they had to make because of submodules was universally negative.

So going forward, because it is still important to have a common basis for patterns, and a clear mechanism and technical path to get updates to the common layer, we have moved to the subtree model as a mechanism for including common. This allows consumers of the pattern to treat the repo as a single entity instead of two, and does not require special syntax or commands to be run when switching branches, updating or, in many cases, contributing to common itself.

Key Learning: Secrets Management

One of our biggest challenges in following GitOps principles for the deployment of workloads is the handling of secrets. GitOps tells us that the git repo should be the source of truth - but we know that we should not store secrets directly in publicly accessible repositories. Previously, our patterns standardized the use of YAML files on the developer workstation as the de-facto authoritative secret store. This can be problematic for at least two reasons: for one, if two people are working on the same repo, which secret store is “right”? Secondly, it might be easier to retrieve credentials from a developer workstation due to information breach or theft. Our systems will not work without secrets, and we need to have a better way of working with them.

Highlight: Multi-cloud GitOps is the “Minimum Viable Pattern”

One of the goals of this pattern is to provide the minimal pattern that both demonstrates the goals, aims and purpose of the framework and does something that we think users will find interesting and valuable. We plan to use it as a starting point for our own future pattern development; and as such we can point to it as the pattern to clone and start with if a user wants to start their own pattern development from scratch.

New Feature: Hub Cluster Vault instance

In this pattern, we introduce the ability to reference upstream helm charts, and pass overrides to them in a native ArgoCD way. The first application we are treating this way is Hashicorp Vault. The use of Vault also allows us to make Vault the authoritative source of truth for secrets in the framework. This also improves our security posture by making it significantly easier to rotate secrets and have OpenShift “do the right thing” by re-deploying and re-starting workloads as necessary.

For the purposes of shipping this pattern as a runnable demonstration, we take certain shortcuts with security that we understand are not best practices - storing the vault keys unencrypted on a developer drive, for example. If you intend to run code derived from this pattern in production, we strongly recommend you consider and follow the practices documented here.

New Feature: External Secrets Operator

While it is important to improve the story for secret handling in the pattern space overall, it is also important to provide space for multiple solutions inside patterns. Because of this, we include the external secrets operator, which in the pattern uses vault, but can be used to support a number of other secret providers, and can be extended to support secrets providers that it does not already support. Furthermore, the external secrets approach is less disruptive to existing applications, since it works by managing the secret objects applications are already used to consuming. Vault provides different options for integration, including the agent injector, but this approach is very specific to Vault and not clearly portable.

In a similar note to the feature about Vault and secrets: the approach we take in this release of the pattern has some known security deficiencies. In RHACM prior to 2.5, policies containing secrets will not properly cloak the secrets in the policy objects, and will not properly encrypt the secrets at rest. RHACM 2.5+ includes a fromSecret function that will secure these secrets in transit and at rest in both of these ways. (Of course, any entity with cluster admin access can recover the contents of a secret object in the cluster.) One additional deficiency of this approach is that the lookup function we use in the policies to copy secrets only runs when the policy object is created or refreshed - which means there is not a mechanism within RHACM presently to detect when a secret has changed and the policy needs to be refreshed. We are hoping this functionality will be included in RHACM 2.6.

New Feature: clusterGroups can have multiple cluster members

Using Advanced Cluster Management, we can inject per-cluster configuration into the ArgoCD application definition. We do this, for example, with the global.hubClusterDomain and global.localClusterDomain variables, which are available to use in helm templates in applications that use the framework.

This enables one of our key new features, the ability to deploy multiple clusters that differ only in local, cluster-defined ways (such as the FQDNs that they would publish for their routes). This is a need we determined when we were working on Industrial Edge, where we had to add the FQDN of the local cluster to a config map, for use in a browser application that was defined in kubernetes, but runs in a user’s browser.

The config-demo namespace uses a deployment of the Red Hat Universal Base Image of httpd to demonstrate how to use the framework to pass variables from application definition to actual use in config maps. The config-demo app shows the management of a secret defined and securely transferred from the hub cluster to remote clusters, as well as allowing for the use of the hub cluster base domain and the local cluster base domain in configuration of applications running on either the hub or managed clusters.

Where do we go from here?

One of the next things we are committed to delivering in the new year is a pattern to extend the concept of GitOps to include elements that are outside of OpenShift and Kubernetes - specifically Red Hat Enterprise Linux nodes, including Red Hat Enterprise Linux For Edge nodes, as well as Red Hat Ansible Automation Platform.

We plan on developing a number of new patterns throughout the new year, which will showcase various technologies. Keep watching this space for updates, and if you would like to get involved, visit our site at https://validatedpatterns.io!

QUICKLINKS

CONTRIBUTE

Ansible Edge GitOps

by Martin Jackson
June 30, 2022
ansible-edge-gitops -patterns

Validated Pattern: Ansible Edge GitOps

Ansible Edge GitOps: The Why and What

As we have been working on new validated patterns and the pattern framework, we have seen a need and interest from the community in expanding the use cases covered by the framework to include other parts of the portfolio besides OpenShift. We understand the Edge computing environments are very complex, and while OpenShift may be the right choice for some Edge environments, it will not be feasible or practical for all of them. Can other environments besides Kubernetes-native ones benefit from GitOps? If so, what would those look like? This pattern works to answer those questions.

GitOps is currently a hot topic in technology. It is a natural outgrowth of the Kubernetes approach in particular, and is informed by now decades of practice in managing large fleets of systems. But is GitOps a concept that is only for Kubernetes? Or can we use the techniques and patterns of GitOps in other systems as well? We believe that by applying specific practices and techniques to Ansible code, and using a Git repo as the authoritative source for configuration results, that we can do exactly that.

One of the first problems we knew we would have to solve in developing this pattern was to work out how to model an Edge environment that was running Virtual Machines. We started with the assumption that we were going to use the Ansible Automation Platform Operator for OpenShift to manage these VMs. But how should we run the VMs themselves?

It is certainly possible to use the different public cloud offerings to spin up instances within the clouds, but that would require a lot of maintenance to the pattern over the long haul to pay attention to different image types and to address any changes to the provisioning schemes the different clouds might make. Additionally, since the purpose of including VMs in this pattern is to model an Edge environment, modeling them as ordinary public cloud instances might seem odd. As a practical matter, the pattern user would have to keep track of the instances and spin them down when spinning down the pattern.

To begin solving these problems, this pattern introduces OpenShift Virtualization to the pattern framework. While OpenShift Virtualization today supports AWS and on-prem baremetal clusters, we hope that it will also bring support to GCP and Azure in the not too distant future. The use of OpenShift Virtualization enables the simulated Edge environment to be modeled entirely in a single cluster, and any instances will be destroyed along with the cluster.

The pattern itself focuses on the installation of a containerized application (Inductive Automation Ignition) on simulated kiosks running RHEL 8 in kiosk mode. This installation pattern is based on work Red Hat did with a customer in the Petrochemical industry.

Highlight: Imperative and Declarative Automation, and GitOps

The validated patterns framework has been committed to GitOps as a philosophy and operational practice since the beginning. The framework’s use of ArgoCD as a mechanism for deploying applications and components is proof of our commitment to GitOps core principles of having a declared desired end state, and a designated agent to bring about that end state.

Many decades of automation practice that focus on individual OS instances (whether they be virtual machines or baremetal) may lead us to believe that the only way to manage such instances is imperatively - that is, focusing on the steps required to configure a machine to the state you want it to be in as opposed to the actual end state you want.

By way of example, consider a situation where you want an individual OS instance to synchronize its time to a source you specify. The imperative way to do this would be to write a script that does some or all of the following:

  1. Install the software that manages system time synchronization
  2. Write a configuration file for the service that specifies the time source in question
  3. If the configuration file or other configuration mechanism that influences the service has changed, restart the time synchronization service.

Along the way, there are subtle differences between different operating systems, such as the name of the time synchronization package (ntp or chrony, for example); differences in which package manager to use; differences in configuration file formats; differences in service names. It is all rather a lot to consider, and the kinds of scripts that managed these sorts of things at scale, when written in Shell or Perl, could get quite convoluted.

Meanwhile, would it not be great if we could put the focus on end state, instead of on the steps required to get to that end state? So we could specify what we want, and we could trust the framework to “make it so” for us? Languages that have this capability rose to the forefront of IT consciousness this century and became wildly popular - languages like Puppet, Chef, Salt and, of course, Ansible. (And yes, they all owe quite a lot to CFEngine, which has been around, and is still around.) The development and practices that grew up around these languages significantly influenced Kubernetes and its development in turn.

Because these languages all provide a kind of hybrid model, they all have mechanisms that allow you to violate one or more of the core tenets of GitOps. For example, while many people run their configuration management code from a Git repository, none of these languages specifically require that, and all provide mechanisms to run in an ad-hoc mode. And yet, all of these languages have a fairly strong declarative flavor that can be used to specify configurations with them; again, this is not mandatory, but it is still quite common. So maybe there is a way to apply the stricter definitions of GitOps to these languages and include their use in a larger GitOps system.

Even within Kubernetes, where have clearly have first-class support for declarative systems, there are aspects of configuration that we may want to make deterministic, but not explicitly code into a git repository. For example, best practice for cluster availability is to spread a worker pool over three different availability zones in a public cloud region. Which three zones should they be? Those decisions are bound to the region the cluster is installed in. Do the AZs themselves really matter, or is the only important constraint that there be three? These kinds of things are state that matters to operators, and an imperative framework for dealing with questions like this can vastly simplify the task of cluster administrators, who can then use this automation to create clusters in multiple regions and clouds and trust that resources will be optimized for maximum availability.

Another crucial point of Declarative and Imperative systems is that it is impossible to conceive of a declarative system that does not have or require reconciliation loops. These reconciliation loops are by definition imperative processes. They often have additional conventions that apply - for example the convention that in Kubernetes Operators the reconciliation loop will change one thing, and then retry - but those processes are still inherently imperative.

A final crucial point on Declarative and Imperative systems is that, especially when we are talking about Edge installations, many of the systems that are important parts of those ecosystems do not have the same level of support for declarative-style configuration management that server operating systems and layers like Kubernetes have. Here we consider crucial elements of Edge environments like routers, switches, access points, and other network gear; as we consider IoT sensors like IP Cameras, it seems unlikely that we will have a Kubernetes-native way to manage devices like these in the foreseeable future.

With these points in mind, it seems that if we cannot bring devices to GitOps, perhaps we should bring GitOps to devices. Ansible has long been recognized for its ability to orchestrate and manage devices in an agentless model. Is there a way to run Ansible that we can recognize as a GitOps mechanism? We believe that there is, by using it with the Ansible Automation Platform components (formerly known as Tower), and recording the desired state in the git repository or repositories that the system uses. In doing so, we believe that we can and should bring GitOps to Edge environments.

Highlight: Including Ansible in Validated Patterns

A new element of this pattern is the use of the Ansible Automation Platform Operator, which we install in the hub cluster of the pattern.

The Ansible Automation Platform Operator is the Kubernetes-native way of running AAP. It provides the Controller function, which supports Execution Environments. The pattern provides its own Execution Environment (with the definition files, so that you can see what is in it or customize it if you like), and loads its own Ansible content into the AAP instance. It uses a dynamic inventory technique to deal with certain aspects of running the VMs it manages under Kubernetes.

The key function of AAP in this pattern is to configure and manage the kiosks. The included content takes the fresh templates, registers them to the Red Hat CDN, installs Firefox, configures kiosk mode, and then downloads and manages the Ignition application container so that both firefox and the application container start at boot time.

The playbook that configures the kiosks is configured to run every 10 minutes on all kiosks, so that if there is some temporary error on the kiosk the configuration will simply attempt configuration again when the schedule tells it to.

Highlight: Including Virtualization in Validated Patterns

As discussed above, another key element in this pattern is the introduction of of OpenShift Virtualization to model the Edge environment with kiosks. The pattern installs the OpenShift Virtualization operator, configures it, and provisions a metal node in order to run the virtual machines. It is possible to emulate hardware acceleration, but the resulting VMs have terrible performance.

The virtual machines we build as part of this pattern are x86_64 RHEL machines, but it should be straightforward to extend this pattern to model other architectures, or other operating systems or versions.

The chart used to define the virtual machines is designed to be open and flexible - replacing the values.yaml file in the chart’s directory will allow you to define different kinds of virtual machine sets; the chart may give you some ideas on how to manage virtual machines under OpenShift in a GitOps way.

Highlight: Including RHEL in Validated Patterns

One of the highlights of this pattern is the use of RHEL in it. There are a number of interesting developments in RHEL that we have been working on, and we expect to highlight more of these in future patterns. We expect that this pattern will be the basis for future patterns that include RHEL, Ansible, and/or Virtualization.

Where do we go from here?

We believe this pattern breaks some new and interesting ground in bringing Ansible, Virtualization, and RHEL to the validated pattern framework. Like all of our patterns, this pattern is Open Source, and we encourage you to use it, tinker with it, and submit your ideas, changes and fixes.

Documentation for how to install the pattern is here, where there are detailed installation instructions, more technical details on the different components in the pattern (especially the use of AAP and OpenShift Virtualization), and some ideas for customization.

QUICKLINKS

Validated Pattern: Ansible Edge GitOps

Ansible Edge GitOps: The Why and What

As we have been working on new validated patterns and the pattern framework, we have seen a need and interest from the community in expanding the use cases covered by the framework to include other parts of the portfolio besides OpenShift. We understand the Edge computing environments are very complex, and while OpenShift may be the right choice for some Edge environments, it will not be feasible or practical for all of them. Can other environments besides Kubernetes-native ones benefit from GitOps? If so, what would those look like? This pattern works to answer those questions.

GitOps is currently a hot topic in technology. It is a natural outgrowth of the Kubernetes approach in particular, and is informed by now decades of practice in managing large fleets of systems. But is GitOps a concept that is only for Kubernetes? Or can we use the techniques and patterns of GitOps in other systems as well? We believe that by applying specific practices and techniques to Ansible code, and using a Git repo as the authoritative source for configuration results, that we can do exactly that.

One of the first problems we knew we would have to solve in developing this pattern was to work out how to model an Edge environment that was running Virtual Machines. We started with the assumption that we were going to use the Ansible Automation Platform Operator for OpenShift to manage these VMs. But how should we run the VMs themselves?

It is certainly possible to use the different public cloud offerings to spin up instances within the clouds, but that would require a lot of maintenance to the pattern over the long haul to pay attention to different image types and to address any changes to the provisioning schemes the different clouds might make. Additionally, since the purpose of including VMs in this pattern is to model an Edge environment, modeling them as ordinary public cloud instances might seem odd. As a practical matter, the pattern user would have to keep track of the instances and spin them down when spinning down the pattern.

To begin solving these problems, this pattern introduces OpenShift Virtualization to the pattern framework. While OpenShift Virtualization today supports AWS and on-prem baremetal clusters, we hope that it will also bring support to GCP and Azure in the not too distant future. The use of OpenShift Virtualization enables the simulated Edge environment to be modeled entirely in a single cluster, and any instances will be destroyed along with the cluster.

The pattern itself focuses on the installation of a containerized application (Inductive Automation Ignition) on simulated kiosks running RHEL 8 in kiosk mode. This installation pattern is based on work Red Hat did with a customer in the Petrochemical industry.

Highlight: Imperative and Declarative Automation, and GitOps

The validated patterns framework has been committed to GitOps as a philosophy and operational practice since the beginning. The framework’s use of ArgoCD as a mechanism for deploying applications and components is proof of our commitment to GitOps core principles of having a declared desired end state, and a designated agent to bring about that end state.

Many decades of automation practice that focus on individual OS instances (whether they be virtual machines or baremetal) may lead us to believe that the only way to manage such instances is imperatively - that is, focusing on the steps required to configure a machine to the state you want it to be in as opposed to the actual end state you want.

By way of example, consider a situation where you want an individual OS instance to synchronize its time to a source you specify. The imperative way to do this would be to write a script that does some or all of the following:

  1. Install the software that manages system time synchronization
  2. Write a configuration file for the service that specifies the time source in question
  3. If the configuration file or other configuration mechanism that influences the service has changed, restart the time synchronization service.

Along the way, there are subtle differences between different operating systems, such as the name of the time synchronization package (ntp or chrony, for example); differences in which package manager to use; differences in configuration file formats; differences in service names. It is all rather a lot to consider, and the kinds of scripts that managed these sorts of things at scale, when written in Shell or Perl, could get quite convoluted.

Meanwhile, would it not be great if we could put the focus on end state, instead of on the steps required to get to that end state? So we could specify what we want, and we could trust the framework to “make it so” for us? Languages that have this capability rose to the forefront of IT consciousness this century and became wildly popular - languages like Puppet, Chef, Salt and, of course, Ansible. (And yes, they all owe quite a lot to CFEngine, which has been around, and is still around.) The development and practices that grew up around these languages significantly influenced Kubernetes and its development in turn.

Because these languages all provide a kind of hybrid model, they all have mechanisms that allow you to violate one or more of the core tenets of GitOps. For example, while many people run their configuration management code from a Git repository, none of these languages specifically require that, and all provide mechanisms to run in an ad-hoc mode. And yet, all of these languages have a fairly strong declarative flavor that can be used to specify configurations with them; again, this is not mandatory, but it is still quite common. So maybe there is a way to apply the stricter definitions of GitOps to these languages and include their use in a larger GitOps system.

Even within Kubernetes, where have clearly have first-class support for declarative systems, there are aspects of configuration that we may want to make deterministic, but not explicitly code into a git repository. For example, best practice for cluster availability is to spread a worker pool over three different availability zones in a public cloud region. Which three zones should they be? Those decisions are bound to the region the cluster is installed in. Do the AZs themselves really matter, or is the only important constraint that there be three? These kinds of things are state that matters to operators, and an imperative framework for dealing with questions like this can vastly simplify the task of cluster administrators, who can then use this automation to create clusters in multiple regions and clouds and trust that resources will be optimized for maximum availability.

Another crucial point of Declarative and Imperative systems is that it is impossible to conceive of a declarative system that does not have or require reconciliation loops. These reconciliation loops are by definition imperative processes. They often have additional conventions that apply - for example the convention that in Kubernetes Operators the reconciliation loop will change one thing, and then retry - but those processes are still inherently imperative.

A final crucial point on Declarative and Imperative systems is that, especially when we are talking about Edge installations, many of the systems that are important parts of those ecosystems do not have the same level of support for declarative-style configuration management that server operating systems and layers like Kubernetes have. Here we consider crucial elements of Edge environments like routers, switches, access points, and other network gear; as we consider IoT sensors like IP Cameras, it seems unlikely that we will have a Kubernetes-native way to manage devices like these in the foreseeable future.

With these points in mind, it seems that if we cannot bring devices to GitOps, perhaps we should bring GitOps to devices. Ansible has long been recognized for its ability to orchestrate and manage devices in an agentless model. Is there a way to run Ansible that we can recognize as a GitOps mechanism? We believe that there is, by using it with the Ansible Automation Platform components (formerly known as Tower), and recording the desired state in the git repository or repositories that the system uses. In doing so, we believe that we can and should bring GitOps to Edge environments.

Highlight: Including Ansible in Validated Patterns

A new element of this pattern is the use of the Ansible Automation Platform Operator, which we install in the hub cluster of the pattern.

The Ansible Automation Platform Operator is the Kubernetes-native way of running AAP. It provides the Controller function, which supports Execution Environments. The pattern provides its own Execution Environment (with the definition files, so that you can see what is in it or customize it if you like), and loads its own Ansible content into the AAP instance. It uses a dynamic inventory technique to deal with certain aspects of running the VMs it manages under Kubernetes.

The key function of AAP in this pattern is to configure and manage the kiosks. The included content takes the fresh templates, registers them to the Red Hat CDN, installs Firefox, configures kiosk mode, and then downloads and manages the Ignition application container so that both firefox and the application container start at boot time.

The playbook that configures the kiosks is configured to run every 10 minutes on all kiosks, so that if there is some temporary error on the kiosk the configuration will simply attempt configuration again when the schedule tells it to.

Highlight: Including Virtualization in Validated Patterns

As discussed above, another key element in this pattern is the introduction of of OpenShift Virtualization to model the Edge environment with kiosks. The pattern installs the OpenShift Virtualization operator, configures it, and provisions a metal node in order to run the virtual machines. It is possible to emulate hardware acceleration, but the resulting VMs have terrible performance.

The virtual machines we build as part of this pattern are x86_64 RHEL machines, but it should be straightforward to extend this pattern to model other architectures, or other operating systems or versions.

The chart used to define the virtual machines is designed to be open and flexible - replacing the values.yaml file in the chart’s directory will allow you to define different kinds of virtual machine sets; the chart may give you some ideas on how to manage virtual machines under OpenShift in a GitOps way.

Highlight: Including RHEL in Validated Patterns

One of the highlights of this pattern is the use of RHEL in it. There are a number of interesting developments in RHEL that we have been working on, and we expect to highlight more of these in future patterns. We expect that this pattern will be the basis for future patterns that include RHEL, Ansible, and/or Virtualization.

Where do we go from here?

We believe this pattern breaks some new and interesting ground in bringing Ansible, Virtualization, and RHEL to the validated pattern framework. Like all of our patterns, this pattern is Open Source, and we encourage you to use it, tinker with it, and submit your ideas, changes and fixes.

Documentation for how to install the pattern is here, where there are detailed installation instructions, more technical details on the different components in the pattern (especially the use of AAP and OpenShift Virtualization), and some ideas for customization.

QUICKLINKS

CONTRIBUTE

Push or Pull?

by Martin Jackson
July 15, 2022
ansible-edge-gitops patterns -GitOps

Push or Pull? Strategies for Large Scale Technology Change Management on the Edge

What is Technology Change Management?

There is a segment of the technology industry dedicated to keeping track of what is changing in an IT environment, when and how. These include systems like Service Now, Remedy, JIRA, and others. This is definitely a kind of change management, and these systems are important - but the focus of this blog post is not how the work of change management is tracked, but the actual means and strategy of doing those changes.

Edge technology solutions involve hardware and software, and they all require some kind of technology maintenance. Our focus here is on software maintenance - these task can involve updating applications, patching underlying operating systems, performing remote administration - restarting applications and services. Coordinating change for a complex application can be daunting for a centralized datacenter application - but on the Edge, where we have hundreds, thousands, maybe millions of devices and application instances to keep track of, it is harder.

What Do You Mean, Push or Pull?

In this article, we are going to discuss two primary strategies for systems that are responsible for making and recording changes on other systems. We are making the assumption here that the system in question is making changes and also recording the results of those changes for later review or troubleshooting. Highly regulated organizations often have audit requirements to show that their financial statements are accurate, and this means demonstrating that there are business processes in place to authorize and schedule changes.

In this context, when we say “Push”, we mean that a hub or centralized system originates and makes changes on other systems. The key differentiator is that the “Push” system stays in contact with the managed system throughout the process of the change. The “Push” system may also keep a record of changes made for its own purposes.

In a “Pull” system, on the other hand, the centralized system waits for managed systems to connect to it to get their configuration instructions. “Pull” systems often have agents that use a dedicated protocol to define changes. There may be several steps in a “Pull” conversation, as defined by the system. A “Pull” system might also be able to cache and apply a previous configuration. The key differentiator of a “Pull” system is that it does not need to maintain constant contact with the central system to do its work.

Push and Pull, in this context represent “strategies” for managing change. A given system can have both “push” and “pull” aspects; for example, ansible has an ansible-pull command which clearly works in a “pull” mode, even though most people recognize ansible as being primarily a push based system. While specific systems or products may be mentioned, the goal is not to evaluate systems themselves, but to talk about the differences and the relative merits and pitfalls of push and pull as strategies for managing change.

What do you mean by “Large”?

The term “Large” is quoted because it can mean different things in different contexts. For change management systems, it can include, but is not necessarily limited to:

  • The count of individual systems managed

As an arbitrary number, a system that manages 10,000 systems could probably be considered “large” regardless of other considerations. But sheer numbers of managed systems are only one aspect in play here. But you may still have a “large” problem if you do not have that many instances. Sheer volume of managed systems impose many constraints on systems that have to change and track other systems - records of changes have to be stored and indexed; there has to be a way to represent the different desired configurations.

  • The complexity of different configurations across those systems

The number of configurations represented across your fleet might be a better predictor for how “large” the problem is. It is easier to manage 10 configurations with 1,000 instances each than to manage 50 configurations with 100 instances each, for example.

  • The organizational involvement in managing configurations

Fleets have a certain team overhead in managing them as they grow. Who decides what hardware gets deployed, and when? Who decides when new Operating System versions are rolled out? Who does testing? If there are several teams involved in these activities, the problem is almost certainly a “large” one.

  • The geographic distribution of systems managed

Another aspect of complexity is how widely dispersed the fleet is. It is easier to manage 10,000 instances in one or even two locations than it is to manage 5 instances in each of 2,000 locations. Geographic distribution also includes operating in multiple legal jurisdictions, and possible in multiple nations. These impose requirements of various kinds on systems and thus also on the systems responsible for maintaining and changing them.

  • It feels like a “large” problem to you

If none of the other criteria listed so far apply to you, but it still feels like a “large” problem to you, it probably is one.

Managing Change - An Ongoing Challenge

In a perfect world, we could deploy technology solutions that maintain themselves. We would not need to update them; they would know how to update themselves. They could coordinate outage times, and could ensure that they can run successfully on a proposed platform. They would know about their own security vulnerabilities, and know how to fix them. Best of all, they could take requests from their users, turn those into systematic improvements, and deploy them without any other interaction.

What is the Edge?

Are you laughing yet? Most of the work of IT administrators is done in one of the areas listed above. Even very competent IT organizations sometimes struggle balancing some of these priorities. One aspect of the current computing environment is the prominence of Edge computing, which places its focus on the devices and applications that use computing resources far from central clouds and data centers - these compute resources run in retail stores, in pharmacies, hospitals, and warehouses; they run in cars and sometimes even spacecraft. In Edge environments, compute resources have to deal with intermittent network connectivity, if they have network connectivity at all. Sometimes, groups of devices have access to local server resources - as might the case in a large retail store, or in a factory or warehouse - but sometimes the closest “servers” are in the cloud and centralized. One interesting aspect of Edge deployments is that there are often many copies of effectively the same deployment. A large chain retail store might deploy the same set of applications to each of its stores. In such an installation, there may be many devices of a single type installed in that location. Think of the number of personal computers or cash registers you can see at a large retail store, for example. It would not be unusual to have ten PCs and twenty cash registers (per store) in this kind of deployment. And a large retail chain could have hundreds or even thousands of locations. Newer technologies, like Internet of Things deployments, require an even higher degree of connectivity - the single retail store example we are considering could have three hundred cameras to manage, which would need to be integrated into its IoT stack. And there could be hundreds, or thousands, of sites just like this one. The scope and scale of systems to manage can get daunting very quickly.

So, some of the defining qualities of Edge environments are: scale (anyone operating some edge installations probably has a lot of edge installations, and the success of their business depends on them operating more of them) and network limitations (whether there is a connection at all, and if so, how reliable it is; bandwidth - how much data it can transfer at a time; and latency - how long it takes to get where it is going). This makes making changes in these environments challenging, because it means keeping track of large numbers of entities in an environment where our ability to contact those entities and verify their status may be limited if it is present at all. But we still must make changes in those environments, because those solutions need maintenance - their platforms may need security updates; their operators may want to update application features and functionality. This requires us to make changes to these devices, and requires technology change management.

Consideration: Workload Capacity Management

Workload Capacity Management focuses on what is needed to manage the scale of deployments. With large Edge deployments, the work needs to get done, and that work needs to be replicated on every node in scope for the deployment - so the same job or configuration content may need to apply to hundreds, thousands, or more individual nodes. Since the control point (central or distributed) is different between push and pull based systems, how they distribute the work needed to distribute changes. Push based systems must send the work out directly; but pull-based systems can potentially overwhelm a centralized system with a “thundering herd.”

Common Consideration: Inventory/Data Aggregation

Inventory and Data Aggregation are crucial considerations for both kinds of systems. Inventory is the starting point for determining what systems are in scope for a given unit of work; data aggregation is important to them because as units of work get done, we often need proof or validation that the work was done. With large numbers of edge nodes, there are certain to be exceptions, and the ability to keep track of where the work is crucial to completing the task.

Common Consideration: Authentication/Authorization

Since these systems are responsible for making changes on devices, how they interact with authentication and authorization systems is an important aspect of how they work. Is the user who the user claims to be? Which users are allowed to make which changes? Authentication and authorization are things we must consider for systems that make changes. Additionally many large organizations have additional technology requirements for systems that can make changes to other systems.

Common Consideration: Dealing with Network Interruptions

In Edge deployments, network connectivity is by definition limited, unreliable, or non-existent. There are differences in how respective types of systems can detect and behave in the presence of network interruptions.

Common Consideration: Eventual Consistency / Idempotence

Regardless of whether a system is push-based or pull-based, it is valuable and useful for the configuration units managed by that system to be safe to apply and re-apply at will. This is the common meaning of the term idempotence. One strategy for minimizing the effect of many kinds of problems in large-scale configuration management is writing content that is idempotent, that is, the effect of running the same content multiple times is the same as the effect of running it once. Practically speaking, this means that such systems make changes only when they need to, and do not have “side effects”. This makes it safe to run the same configuration content on the same device many times, so if it cannot be determined whether a device has received a particular configuration or not, the solution would be to apply the configuration to it, and the devices should then be in the desired, known state when the configuration is done.

Approach 1: Push Strategy

The first approach we will consider is the “push” strategy. In a “push” strategy, the centralized change management system itself reaches out to managed devices and triggers updates in some way. This could involve making an API call to a device, logging in to a device through SSH, or using a dedicated client/server protocol. Red Hat’s Ansible operates as a push-based system, where a central console reaches out to devices and manages them.

Push Consideration: Workload Capacity Management

A push based system has much more control over how it parcels out configuration workload, since it is in control of how configuration workloads are driven. It can more easily perceive its own load state, and potentially “back off” or “throttle” if it is processing too much work too quickly - or increase it if the scope of the desired change is smaller than the total designed capacity of the system. It is easier to influence the “rate of change” on a push-based system for this reason.

Push Consideration: Dealing with Network Interruptions

Push-based systems are at a disadvantage when dealing with network interruptions and limitations. The most common network failure scenarios are ambiguous: if an attempt to reach an individual device fails, is that because there was a problem with the device, or a problem with the network path to reach the device? The push-based system can only know things about the devices it manages when it is told. An additional potential with network interruption is that a device can successfully apply a unit of configuration change but can fail to report that because of a network problem - the report is dropped, for example, because of a network path outage or problem, or the central collection infrastructure was overwhelmed. In such a situation, it is best to have the option to re-apply the configuration, for which it is best if you can have the confidence that such configuration will not have any undesired side-effects, and will only make the changes it needs to make.

Approach 2: Pull Strategy

The second approach we will consider is the “pull” strategy. The key difference in the “pull” strategy is that devices themselves initiate communication with the central management system. They can do this by making a request to the management system (which can be a notification, API call, or some other mechanism). That is to say - the central management system “waits” for check-ins from the managed devices. Client-server Puppet is a pull-based system, in which managed devices reach out to server endpoints, which give the devices instructions on what configurations to apply to themselves. Puppet also has options for operating in a push-based model; historically this could be done through puppet kick, mcollective orchestration, application orchestration, or bolt.

Pull Consideration: Workload Capacity Management

Pull-based systems have some challenges in regard to workload capacity for the pieces that need to be centralized (particularly reporting and inventory functions). The reason for this is that the devices managed will not have a direct source of information about the load level of centralized infrastructure, unless this is provided by an API; some load balancing schemes can do this in a rudimentary way by directing new requests to an instance via a “least connection” balancing scheme. Large deployments typically have to design a system to stagger check-ins to ensure the system does not get overwhelmed by incoming requests.

Pull Consideration: Authentication/Authorization

Pull-based systems typically have agents that run on the systems that are managed, and as such are simpler to operate from an authentication and authorization standpoint. Agents on devices can often be given administrative privilege, and the practical authentication/authorization problems have to do with access to the central management console, and the ability to change the configurations distributed or see the inventory and reports of attempts to configure devices.

Pull Consideration: Dealing with Network Interruptions

Pull-based systems have a distinct advantage when they encounter network interruptions. While it is in no way safe to assume that a managed device is still present or relevant from the standpoint of central infrastructure, it is almost always safe for a device, when it finds it cannot connect central infrastructure, to assume that it is experiencing a temporary network outage, and to simply retry the operation later. Care must be taken, especially in large deployments, not to overwhelm central infrastructure with requests. Additionally, we must remember that since network interruption can occur at any time on the edge, that the operation we are interested in may indeed have completed successfully, but the device was simply unable to report this to us for some reason. As was the case for push-based systems, the best cure for this is to ensure that content can be safely re-applied as needed or desired.

Conclusions

Push and Pull based systems have different scaling challenges as they grow

Push and Pull-based systems have different tradeoffs. It can be easier to manage a push-based system for smaller numbers of managed devices; some of the challenges of both styles clearly increase as systems grow to multiple thousands of nodes.

Meanwhile, both push and pull based systems, as a practical matter, have to make sense and be usable for small installations as well as large, and grow and scale as smoothly as possible. Many installations will never face some or maybe even any of these challenges - and systems of both types must be easy to understand and learn, or else they will not be used.

Pull-based systems are better for Edge uses despite scaling challenges

Pull based systems can deal better with the network problems that are inherent with edge devices. When connectivity to central infrastructure is unreliable, pull-based systems can still operate. Pull-based systems can safely assume that a network partition is temporary, and thus do not suffer from the inherent ambiguity of “could not reach target system” kinds of errors.

Idempotence matters more than whether a system is push or pull based

The fix for nearly all the operational problems in large scale configurations management problems is to be able to apply the same configuration to the same device multiple times and expect the same result. This takes discipline and effort, but that effort pays off well in the end.

To help scale, introduce a messaging or queuing layer to hold on to data in flight if possible

Many of the operational considerations are related to limited network connectivity or overtaxing centralized infrastructure. Both of these problems can be mitigated significantly by introducing a messaging or queuing layer in the configuration management system to hold on to reports, results, and inventory updates until the system can confirm receipt and processing of those elements.

QUICKLINKS

Push or Pull? Strategies for Large Scale Technology Change Management on the Edge

What is Technology Change Management?

There is a segment of the technology industry dedicated to keeping track of what is changing in an IT environment, when and how. These include systems like Service Now, Remedy, JIRA, and others. This is definitely a kind of change management, and these systems are important - but the focus of this blog post is not how the work of change management is tracked, but the actual means and strategy of doing those changes.

Edge technology solutions involve hardware and software, and they all require some kind of technology maintenance. Our focus here is on software maintenance - these task can involve updating applications, patching underlying operating systems, performing remote administration - restarting applications and services. Coordinating change for a complex application can be daunting for a centralized datacenter application - but on the Edge, where we have hundreds, thousands, maybe millions of devices and application instances to keep track of, it is harder.

What Do You Mean, Push or Pull?

In this article, we are going to discuss two primary strategies for systems that are responsible for making and recording changes on other systems. We are making the assumption here that the system in question is making changes and also recording the results of those changes for later review or troubleshooting. Highly regulated organizations often have audit requirements to show that their financial statements are accurate, and this means demonstrating that there are business processes in place to authorize and schedule changes.

In this context, when we say “Push”, we mean that a hub or centralized system originates and makes changes on other systems. The key differentiator is that the “Push” system stays in contact with the managed system throughout the process of the change. The “Push” system may also keep a record of changes made for its own purposes.

In a “Pull” system, on the other hand, the centralized system waits for managed systems to connect to it to get their configuration instructions. “Pull” systems often have agents that use a dedicated protocol to define changes. There may be several steps in a “Pull” conversation, as defined by the system. A “Pull” system might also be able to cache and apply a previous configuration. The key differentiator of a “Pull” system is that it does not need to maintain constant contact with the central system to do its work.

Push and Pull, in this context represent “strategies” for managing change. A given system can have both “push” and “pull” aspects; for example, ansible has an ansible-pull command which clearly works in a “pull” mode, even though most people recognize ansible as being primarily a push based system. While specific systems or products may be mentioned, the goal is not to evaluate systems themselves, but to talk about the differences and the relative merits and pitfalls of push and pull as strategies for managing change.

What do you mean by “Large”?

The term “Large” is quoted because it can mean different things in different contexts. For change management systems, it can include, but is not necessarily limited to:

  • The count of individual systems managed

As an arbitrary number, a system that manages 10,000 systems could probably be considered “large” regardless of other considerations. But sheer numbers of managed systems are only one aspect in play here. But you may still have a “large” problem if you do not have that many instances. Sheer volume of managed systems impose many constraints on systems that have to change and track other systems - records of changes have to be stored and indexed; there has to be a way to represent the different desired configurations.

  • The complexity of different configurations across those systems

The number of configurations represented across your fleet might be a better predictor for how “large” the problem is. It is easier to manage 10 configurations with 1,000 instances each than to manage 50 configurations with 100 instances each, for example.

  • The organizational involvement in managing configurations

Fleets have a certain team overhead in managing them as they grow. Who decides what hardware gets deployed, and when? Who decides when new Operating System versions are rolled out? Who does testing? If there are several teams involved in these activities, the problem is almost certainly a “large” one.

  • The geographic distribution of systems managed

Another aspect of complexity is how widely dispersed the fleet is. It is easier to manage 10,000 instances in one or even two locations than it is to manage 5 instances in each of 2,000 locations. Geographic distribution also includes operating in multiple legal jurisdictions, and possible in multiple nations. These impose requirements of various kinds on systems and thus also on the systems responsible for maintaining and changing them.

  • It feels like a “large” problem to you

If none of the other criteria listed so far apply to you, but it still feels like a “large” problem to you, it probably is one.

Managing Change - An Ongoing Challenge

In a perfect world, we could deploy technology solutions that maintain themselves. We would not need to update them; they would know how to update themselves. They could coordinate outage times, and could ensure that they can run successfully on a proposed platform. They would know about their own security vulnerabilities, and know how to fix them. Best of all, they could take requests from their users, turn those into systematic improvements, and deploy them without any other interaction.

What is the Edge?

Are you laughing yet? Most of the work of IT administrators is done in one of the areas listed above. Even very competent IT organizations sometimes struggle balancing some of these priorities. One aspect of the current computing environment is the prominence of Edge computing, which places its focus on the devices and applications that use computing resources far from central clouds and data centers - these compute resources run in retail stores, in pharmacies, hospitals, and warehouses; they run in cars and sometimes even spacecraft. In Edge environments, compute resources have to deal with intermittent network connectivity, if they have network connectivity at all. Sometimes, groups of devices have access to local server resources - as might the case in a large retail store, or in a factory or warehouse - but sometimes the closest “servers” are in the cloud and centralized. One interesting aspect of Edge deployments is that there are often many copies of effectively the same deployment. A large chain retail store might deploy the same set of applications to each of its stores. In such an installation, there may be many devices of a single type installed in that location. Think of the number of personal computers or cash registers you can see at a large retail store, for example. It would not be unusual to have ten PCs and twenty cash registers (per store) in this kind of deployment. And a large retail chain could have hundreds or even thousands of locations. Newer technologies, like Internet of Things deployments, require an even higher degree of connectivity - the single retail store example we are considering could have three hundred cameras to manage, which would need to be integrated into its IoT stack. And there could be hundreds, or thousands, of sites just like this one. The scope and scale of systems to manage can get daunting very quickly.

So, some of the defining qualities of Edge environments are: scale (anyone operating some edge installations probably has a lot of edge installations, and the success of their business depends on them operating more of them) and network limitations (whether there is a connection at all, and if so, how reliable it is; bandwidth - how much data it can transfer at a time; and latency - how long it takes to get where it is going). This makes making changes in these environments challenging, because it means keeping track of large numbers of entities in an environment where our ability to contact those entities and verify their status may be limited if it is present at all. But we still must make changes in those environments, because those solutions need maintenance - their platforms may need security updates; their operators may want to update application features and functionality. This requires us to make changes to these devices, and requires technology change management.

Consideration: Workload Capacity Management

Workload Capacity Management focuses on what is needed to manage the scale of deployments. With large Edge deployments, the work needs to get done, and that work needs to be replicated on every node in scope for the deployment - so the same job or configuration content may need to apply to hundreds, thousands, or more individual nodes. Since the control point (central or distributed) is different between push and pull based systems, how they distribute the work needed to distribute changes. Push based systems must send the work out directly; but pull-based systems can potentially overwhelm a centralized system with a “thundering herd.”

Common Consideration: Inventory/Data Aggregation

Inventory and Data Aggregation are crucial considerations for both kinds of systems. Inventory is the starting point for determining what systems are in scope for a given unit of work; data aggregation is important to them because as units of work get done, we often need proof or validation that the work was done. With large numbers of edge nodes, there are certain to be exceptions, and the ability to keep track of where the work is crucial to completing the task.

Common Consideration: Authentication/Authorization

Since these systems are responsible for making changes on devices, how they interact with authentication and authorization systems is an important aspect of how they work. Is the user who the user claims to be? Which users are allowed to make which changes? Authentication and authorization are things we must consider for systems that make changes. Additionally many large organizations have additional technology requirements for systems that can make changes to other systems.

Common Consideration: Dealing with Network Interruptions

In Edge deployments, network connectivity is by definition limited, unreliable, or non-existent. There are differences in how respective types of systems can detect and behave in the presence of network interruptions.

Common Consideration: Eventual Consistency / Idempotence

Regardless of whether a system is push-based or pull-based, it is valuable and useful for the configuration units managed by that system to be safe to apply and re-apply at will. This is the common meaning of the term idempotence. One strategy for minimizing the effect of many kinds of problems in large-scale configuration management is writing content that is idempotent, that is, the effect of running the same content multiple times is the same as the effect of running it once. Practically speaking, this means that such systems make changes only when they need to, and do not have “side effects”. This makes it safe to run the same configuration content on the same device many times, so if it cannot be determined whether a device has received a particular configuration or not, the solution would be to apply the configuration to it, and the devices should then be in the desired, known state when the configuration is done.

Approach 1: Push Strategy

The first approach we will consider is the “push” strategy. In a “push” strategy, the centralized change management system itself reaches out to managed devices and triggers updates in some way. This could involve making an API call to a device, logging in to a device through SSH, or using a dedicated client/server protocol. Red Hat’s Ansible operates as a push-based system, where a central console reaches out to devices and manages them.

Push Consideration: Workload Capacity Management

A push based system has much more control over how it parcels out configuration workload, since it is in control of how configuration workloads are driven. It can more easily perceive its own load state, and potentially “back off” or “throttle” if it is processing too much work too quickly - or increase it if the scope of the desired change is smaller than the total designed capacity of the system. It is easier to influence the “rate of change” on a push-based system for this reason.

Push Consideration: Dealing with Network Interruptions

Push-based systems are at a disadvantage when dealing with network interruptions and limitations. The most common network failure scenarios are ambiguous: if an attempt to reach an individual device fails, is that because there was a problem with the device, or a problem with the network path to reach the device? The push-based system can only know things about the devices it manages when it is told. An additional potential with network interruption is that a device can successfully apply a unit of configuration change but can fail to report that because of a network problem - the report is dropped, for example, because of a network path outage or problem, or the central collection infrastructure was overwhelmed. In such a situation, it is best to have the option to re-apply the configuration, for which it is best if you can have the confidence that such configuration will not have any undesired side-effects, and will only make the changes it needs to make.

Approach 2: Pull Strategy

The second approach we will consider is the “pull” strategy. The key difference in the “pull” strategy is that devices themselves initiate communication with the central management system. They can do this by making a request to the management system (which can be a notification, API call, or some other mechanism). That is to say - the central management system “waits” for check-ins from the managed devices. Client-server Puppet is a pull-based system, in which managed devices reach out to server endpoints, which give the devices instructions on what configurations to apply to themselves. Puppet also has options for operating in a push-based model; historically this could be done through puppet kick, mcollective orchestration, application orchestration, or bolt.

Pull Consideration: Workload Capacity Management

Pull-based systems have some challenges in regard to workload capacity for the pieces that need to be centralized (particularly reporting and inventory functions). The reason for this is that the devices managed will not have a direct source of information about the load level of centralized infrastructure, unless this is provided by an API; some load balancing schemes can do this in a rudimentary way by directing new requests to an instance via a “least connection” balancing scheme. Large deployments typically have to design a system to stagger check-ins to ensure the system does not get overwhelmed by incoming requests.

Pull Consideration: Authentication/Authorization

Pull-based systems typically have agents that run on the systems that are managed, and as such are simpler to operate from an authentication and authorization standpoint. Agents on devices can often be given administrative privilege, and the practical authentication/authorization problems have to do with access to the central management console, and the ability to change the configurations distributed or see the inventory and reports of attempts to configure devices.

Pull Consideration: Dealing with Network Interruptions

Pull-based systems have a distinct advantage when they encounter network interruptions. While it is in no way safe to assume that a managed device is still present or relevant from the standpoint of central infrastructure, it is almost always safe for a device, when it finds it cannot connect central infrastructure, to assume that it is experiencing a temporary network outage, and to simply retry the operation later. Care must be taken, especially in large deployments, not to overwhelm central infrastructure with requests. Additionally, we must remember that since network interruption can occur at any time on the edge, that the operation we are interested in may indeed have completed successfully, but the device was simply unable to report this to us for some reason. As was the case for push-based systems, the best cure for this is to ensure that content can be safely re-applied as needed or desired.

Conclusions

Push and Pull based systems have different scaling challenges as they grow

Push and Pull-based systems have different tradeoffs. It can be easier to manage a push-based system for smaller numbers of managed devices; some of the challenges of both styles clearly increase as systems grow to multiple thousands of nodes.

Meanwhile, both push and pull based systems, as a practical matter, have to make sense and be usable for small installations as well as large, and grow and scale as smoothly as possible. Many installations will never face some or maybe even any of these challenges - and systems of both types must be easy to understand and learn, or else they will not be used.

Pull-based systems are better for Edge uses despite scaling challenges

Pull based systems can deal better with the network problems that are inherent with edge devices. When connectivity to central infrastructure is unreliable, pull-based systems can still operate. Pull-based systems can safely assume that a network partition is temporary, and thus do not suffer from the inherent ambiguity of “could not reach target system” kinds of errors.

Idempotence matters more than whether a system is push or pull based

The fix for nearly all the operational problems in large scale configurations management problems is to be able to apply the same configuration to the same device multiple times and expect the same result. This takes discipline and effort, but that effort pays off well in the end.

To help scale, introduce a messaging or queuing layer to hold on to data in flight if possible

Many of the operational considerations are related to limited network connectivity or overtaxing centralized infrastructure. Both of these problems can be mitigated significantly by introducing a messaging or queuing layer in the configuration management system to hold on to reports, results, and inventory updates until the system can confirm receipt and processing of those elements.

QUICKLINKS

CONTRIBUTE

Using clusterGroups in the Validated Patterns Framework

by Martin Jackson
August 24, 2022
validated-pattern

What Is a clusterGroup?

The Validated Patterns framework defines itself in terms of clusterGroups. A clusterGroup is a set of one or more Kubernetes clusters that are managed to have the same deployments (that is, subscriptions and applications, namespaces and projects) applied to each member of the same clustergroup. Two different members of the same clusterGroup will differ in cluster-name and in other cluster-specific details (such as PKI and tokens) but will have the same subscriptions and applications as other members of the same clusterGroup.

Essentially, a clusterGroup is an abstract definition of what the pattern will install on each respective cluster in the pattern. It is possible to install multiple clusterGroups on the same cluster, as we do (for example) in Industrial Edge where we have both datacenter and factory clusterGroups, and the factory clusterGroup is installed on the datacenter cluster by default.

Single-Cluster and Multi-Cluster Patterns

Because Validated Patterns started as an Edge initiative, we designed the notion of multi-cluster patterns into the framework from the beginning. The first Validated Pattern, Industrial Edge models a central data-lake and an optional remote factory cluster. The factory cluster does not need the CI or test system, nor the central data lake. Since we have different configuration needs on the two types of cluster, we define them as different cluster groups.

Other patterns only (so far) require a single cluster, since they model their Edge requirements in different ways. Medical Diagnosis brings a pre-provided set of data (which would ordinarily come in from Edge clusters). Ansible Edge GitOps has RHEL instances as its Edge environment, not OpenShift clusters - and it runs its VMs through OpenShift Virtualization so it only uses the one cluster. In these cases, it is simplest to -designate the single cluster as a Hub cluster.

Uses for a Hub Cluster in a Multi-Cluster Deployment

But in multi-cluster deployments, it is often helpful to define a hierarchy. Even if there are exactly two clusters as part of a multi-cluster pattern, it may be helpful to define one as the “hub”, in case the pattern grows to more clusters later. Modern architectures can scale to many instances, in some cases thousands, and there are certain responsibilities that are convenient to define as “central” or “hub” functions. Some examples that exist in the Framework and its applications so far:

  1. Configuration Management (via Advanced Cluster Management, or by Ansible Automation Platform)
  2. Data Aggregation (via AMQ Streams)
  3. Continuous Integration/Continuous Delivery Pipelines (via OpenShift Pipelines)
  4. Data Visualization (via Grafana)
  5. Secrets storage and maintenance (via Vault and the External Secrets Operator)

Some other potential uses for the “hub” role or function include:

  1. General “control plane” functions
  2. Metrics aggregation

The hub role defines a place in the architecture to run these vital functions, while reserving capacity on the Edge for data gathering and pre-process functions.

But this naturally brings up a question - what are the different roles we have considered in the Validated Patterns framework, and how should they be used?

Types of clusterGroup

While we describe the Validated Patterns framework as “opinionated”, we do not want to make it overly constricting. The framework currently considered two types of clusterGroups: Hub and non-hub. We consider “Hub” as a role, primarily, but it is also used as the default name for the Hub clusterGroup. There are two important aspects of a Hub clusterGroup:

  1. It is expected to be a singleton (there is expected to be a single hub cluster in the hub clusterGroup)
  2. There is expected to be a single Hub clusterGroup in a given pattern

Generally, we expect that non-Hub clusterGroups will be Edge clusters, but this is not essential. A clusterGroup should have at least one cluster that it will apply to (otherwise, why define one?) and can have multiple clusters. The framework sets ArgoCD’s ignoreMissingValueFiles setting to true unconditionally, and the framework also provides an extraValueFiles variable, which can define multiple optional additional values files on a per-clusterGroup basis.

How the Hierarchy Works - the clustergroup Chart

The clustergroup chart begins by looking in the pattern’s values-global.yaml for the main.clusterGroupName value. This value is then used to compute the next values file to process - if the value of that variable is hub then the application(s) that are created will use both the values-global.yaml and values-hub.yaml files in the root of the pattern repository. The clusterGroup structure will then be parsed for name and isHubCluster values; namespaces, subscriptions, projects and applications will be applied. Any managedClusterGroups (on the hubCluster) will be defined in terms for Advanced Cluster Management (as is done in Industrial Edge). Because the factory match expression is very general (vendor In OpenShift) this will match the hub cluster and any cluster that is joined to the hub cluster’s ACM instance.

Frequently Asked Questions (FAQs)

I have a single-cluster Pattern. Do I need to call my hub cluster “hub”?

You can, but you do not have to. If you want to call it something other than hub:

  1. Define a different main.clusterGroupName value in values-global.yaml
  2. Create the values-yourhubgroupname.yaml in the root of your pattern repository.
  3. Make sure clusterGroup.name variable in values-yourhubgroupname.yaml matches your intended hub group.

I have a multi-cluster Pattern. All of my Edge clusters will have the same configuration. Do I need multiple clusterGroups to model it?

No! This is the exact scenario we had in mind when we designed clusterGroups. Just add multiple clusters with the same criteria you defined to the ACM instance on your hub cluster, and each cluster you add will get the same subscriptions and applications.

I have a multi-cluster Pattern. How do I decide if I need multiple clusterGroups to model it?

Do you have different subscriptions or applications that you plan to use on your different clusterGroups? If so, you should define different clusterGroups and define them as managedClusterGroups for your hub cluster.

QUICKLINKS

QUICKLINKS

CONTRIBUTE

As you can see the subdomain property was replaced with the host property but it includes the ingress domain for the cluster. Why is this important? I can now apply this to any OpenShift cluster without worrying it’s ingress domain since it will get appended automatically.

There is currently a known issue when using the subdomain property in which the name given is not published with the correct template, instead of that they are published with the default ‘${name}-${namespace}.subdomain.local’. This has been fixed and it’s available in OpenShift 4.11.

Conclusion

Using the subdomain property when defining route is super useful if you are deploying your application to different clusters and it will allow you to not have to hard code the ingress domain for every cluster.

If you have any questions or want to see what we are working on please feel free to visit our Validated Patterns site. If you are excited or intrigued by what you see here we’d love to hear your thoughts and ideas! Try the patterns contained in our Validated Patterns Repo. We will review your pull requests to our pattern repositories.

QUICKLINKS

As you can see the subdomain property was replaced with the host property but it includes the ingress domain for the cluster. Why is this important? I can now apply this to any OpenShift cluster without worrying it’s ingress domain since it will get appended automatically.

There is currently a known issue when using the subdomain property in which the name given is not published with the correct template, instead of that they are published with the default ‘${name}-${namespace}.subdomain.local’. This has been fixed and it’s available in OpenShift 4.11.

Conclusion

Using the subdomain property when defining route is super useful if you are deploying your application to different clusters and it will allow you to not have to hard code the ingress domain for every cluster.

If you have any questions or want to see what we are working on please feel free to visit our Validated Patterns site. If you are excited or intrigued by what you see here we’d love to hear your thoughts and ideas! Try the patterns contained in our Validated Patterns Repo. We will review your pull requests to our pattern repositories.

QUICKLINKS

CONTRIBUTE

QUICKLINKS

QUICKLINKS

CONTRIBUTE

In the argoCD interface we see that the rollout is paused just like with blue-green

canary

But what about our application - we said we only want 20% of the traffic to go to the new app:

demo

That is awesome! So now let’s promote the application and see what happens - the expectation is that it will incrementally update the percentage of connections to the new application until completely promoted.

fullpromote

Let’s take a look at what it looks like using the argo rollouts plugin

argo rollout plugin

Conclusion

Argo Rollouts makes progressive delivery of our applications super easy. Whether you want to deploy using blue-green or the more advanced canary rollout is up to you. The canary rollout is very powerful and as we saw gives us the ultimate control, with insights and flexibility to deploy applications. There is so much more that argo rollouts can do - this demo barely scratches the surface! Keep an eye out for argo rollouts as part of openshift-gitops in ‘23.

QUICKLINKS

In the argoCD interface we see that the rollout is paused just like with blue-green

canary

But what about our application - we said we only want 20% of the traffic to go to the new app:

demo

That is awesome! So now let’s promote the application and see what happens - the expectation is that it will incrementally update the percentage of connections to the new application until completely promoted.

fullpromote

Let’s take a look at what it looks like using the argo rollouts plugin

argo rollout plugin

Conclusion

Argo Rollouts makes progressive delivery of our applications super easy. Whether you want to deploy using blue-green or the more advanced canary rollout is up to you. The canary rollout is very powerful and as we saw gives us the ultimate control, with insights and flexibility to deploy applications. There is so much more that argo rollouts can do - this demo barely scratches the surface! Keep an eye out for argo rollouts as part of openshift-gitops in ‘23.

QUICKLINKS

CONTRIBUTE

Multicluster DevSecOps

Software supply chain security: The why and what

Today more and more organizations are turning to agile development models and DevOps. With this approach, development organizations can deliver more enhancements and bug fixes in a timely manner, providing more value to their customers. While DevOps can include security earlier in the software lifecycle, in practice this has not always been the case. DevSecOps explicitly calls on organizations to pay attention to security best practices and to automate them or “Shift Left” as much as possible.

DevSecOps means baking in application and infrastructure security from the start. In order to be successful, organizations must look both upstream where their dependencies come from, and also how their components integrate together in the production environment. It also means automating security gates to keep the DevOps workflow from slowing down. As we learn from experience, we codify that into the automation process.

A successful DevSecOps based supply chain must consider four areas of concern:

  1. Secure developer dependencies
  2. Secure code development
  3. Secure deployment of resources into a secure environment
  4. Software Bill of Materials (SBOM)

Within each of these areas there are also many best practices to be applied particularly in Cloud Native development using container technology.

  • Scanning new development code for potential vulnerabilities
  • Scanning dependent images that new code will be layered upon
  • Attesting to the veracity of images using image signing
  • Scanning images for know CVEs
  • Scanning the environment for potential networking vulnerabilities
  • Scanning for misconfiguration of images and other assets
  • Ensuring consistent automated deployment of secure configuration using GitOps
  • Continuous upgrading of security policies from both trusted third parties and experience

This pattern deploys several Red Hat Products:

Highlight: Multicluster

While all of the components can be deployed on a single cluster, which makes for a simple demo, this pattern deploys a real world architecture where the central management, development environments, and production are all deployed on different clusters. This ensures that the pattern is structured for real-world deployments, with all the functionality needed to make such an architecture work already built-in, so that pattern consumers can concentrate on what is being delivered, rather than how.

Multicluster DevSecOps Architecture

The heavy lifting in the pattern includes a great deal of integration between components, especially those spanning across clusters:

  • Deployment of Quay Enterprise with OpenShift Data Foundations as a storage backend
  • Deployment of Quay Bridge operator configured to connect with Quay Enterprise on hub cluster
  • Deployment of ACS on managed nodes with integration back to ACS central on the hub
  • Deployment of a secure pipeline with scanning and signing tools, including ACS

Highlight: DevSecOps with Pipelines

“OpenShift Pipelines makes CI/CD concepts such as a ‘pipeline’, a ’task’, a ‘step’ natively instantiatable [sic] so it can use the scalability, security, ease of deployment capabilities of Kubernetes.” (Introducing OpenShift Pipelines). The pattern consumes many of the OpenShift Pipelines out of the box tasks but also defines new tasks for scanning and signing and includes them in enhanced DevSecOps pipelines.

Multicluster DevSecOps Architecture - development schema

While these pipelines are included in the pattern, the pattern also implements the use of Pipelines-as-Code feature where the pipeline can be part of the application code repository. “This allows developers to ship their CI/CD pipelines within the same git repository as their application, making it easier to keep both of them in sync in terms of release updates.”

Highlight: Using the CI Pipeline to provide supply chain security

This pattern includes some other technologies in the development CI pipeline, including cosign, a SIGSTORE project, implemented with Tekton Chains. Cosign supports container signing, verification, and storage in an OCI registry. It enables consumers to sign their pipeline resources and images and share the attestation files providing downstream consumers assurances that they are consuming a trusted artifact.

We also implement open source tools like Sonarqube for static code analysis, nexus for securely storing build artifacts in-cluster, and an open source reports application that is used to upload and present the reports from the security pipeline.

Not using these tools in your environment? That’s not a problem. The pattern framework is flexible. Organizations using different services can swap out what’s in the pattern with their software of choice to fit their environment.

Where do we go from here?

This pattern provides a complete deployment solution for Multicluster DevSecOps that can be used as part of a supply chain deployment pattern across different industries.

Documentation for how to install the pattern is here, where there are detailed installation instructions and more technical details on the different components in the pattern.

QUICKLINKS

Multicluster DevSecOps

Software supply chain security: The why and what

Today more and more organizations are turning to agile development models and DevOps. With this approach, development organizations can deliver more enhancements and bug fixes in a timely manner, providing more value to their customers. While DevOps can include security earlier in the software lifecycle, in practice this has not always been the case. DevSecOps explicitly calls on organizations to pay attention to security best practices and to automate them or “Shift Left” as much as possible.

DevSecOps means baking in application and infrastructure security from the start. In order to be successful, organizations must look both upstream where their dependencies come from, and also how their components integrate together in the production environment. It also means automating security gates to keep the DevOps workflow from slowing down. As we learn from experience, we codify that into the automation process.

A successful DevSecOps based supply chain must consider four areas of concern:

  1. Secure developer dependencies
  2. Secure code development
  3. Secure deployment of resources into a secure environment
  4. Software Bill of Materials (SBOM)

Within each of these areas there are also many best practices to be applied particularly in Cloud Native development using container technology.

  • Scanning new development code for potential vulnerabilities
  • Scanning dependent images that new code will be layered upon
  • Attesting to the veracity of images using image signing
  • Scanning images for know CVEs
  • Scanning the environment for potential networking vulnerabilities
  • Scanning for misconfiguration of images and other assets
  • Ensuring consistent automated deployment of secure configuration using GitOps
  • Continuous upgrading of security policies from both trusted third parties and experience

This pattern deploys several Red Hat Products:

Highlight: Multicluster

While all of the components can be deployed on a single cluster, which makes for a simple demo, this pattern deploys a real world architecture where the central management, development environments, and production are all deployed on different clusters. This ensures that the pattern is structured for real-world deployments, with all the functionality needed to make such an architecture work already built-in, so that pattern consumers can concentrate on what is being delivered, rather than how.

Multicluster DevSecOps Architecture

The heavy lifting in the pattern includes a great deal of integration between components, especially those spanning across clusters:

  • Deployment of Quay Enterprise with OpenShift Data Foundations as a storage backend
  • Deployment of Quay Bridge operator configured to connect with Quay Enterprise on hub cluster
  • Deployment of ACS on managed nodes with integration back to ACS central on the hub
  • Deployment of a secure pipeline with scanning and signing tools, including ACS

Highlight: DevSecOps with Pipelines

“OpenShift Pipelines makes CI/CD concepts such as a ‘pipeline’, a ’task’, a ‘step’ natively instantiatable [sic] so it can use the scalability, security, ease of deployment capabilities of Kubernetes.” (Introducing OpenShift Pipelines). The pattern consumes many of the OpenShift Pipelines out of the box tasks but also defines new tasks for scanning and signing and includes them in enhanced DevSecOps pipelines.

Multicluster DevSecOps Architecture - development schema

While these pipelines are included in the pattern, the pattern also implements the use of Pipelines-as-Code feature where the pipeline can be part of the application code repository. “This allows developers to ship their CI/CD pipelines within the same git repository as their application, making it easier to keep both of them in sync in terms of release updates.”

Highlight: Using the CI Pipeline to provide supply chain security

This pattern includes some other technologies in the development CI pipeline, including cosign, a SIGSTORE project, implemented with Tekton Chains. Cosign supports container signing, verification, and storage in an OCI registry. It enables consumers to sign their pipeline resources and images and share the attestation files providing downstream consumers assurances that they are consuming a trusted artifact.

We also implement open source tools like Sonarqube for static code analysis, nexus for securely storing build artifacts in-cluster, and an open source reports application that is used to upload and present the reports from the security pipeline.

Not using these tools in your environment? That’s not a problem. The pattern framework is flexible. Organizations using different services can swap out what’s in the pattern with their software of choice to fit their environment.

Where do we go from here?

This pattern provides a complete deployment solution for Multicluster DevSecOps that can be used as part of a supply chain deployment pattern across different industries.

Documentation for how to install the pattern is here, where there are detailed installation instructions and more technical details on the different components in the pattern.

QUICKLINKS

CONTRIBUTE

QUICKLINKS

QUICKLINKS

CONTRIBUTE

Introducing a Simplified Tier Naming Scheme

by Andrew Beekhof
December 1, 2023
patterns -announce

The efforts here started off with 2 different classes of patterns: “Community” and “Validated”, however this terminology dates back to before the effort had arrived at “Validated Patterns” as the official project name.

Having standardized on the use of “Validated Patterns” to refer to the overall initiative, it became confusing to refer to “Community” Validated Patterns and “Validated” Validated Patterns.

In addressing that confusion, we took the opportunity to design a new set of tiers:

Generally speaking, Sandbox aligns with the old Community tier, and Maintained aligns with the old Validated tier. However some of the requirements associated with those previous tiers were structured around our bandwidth and our priorities. In revisiting the tiers, we’ve removed many of those value judgements and shifted the emphasis to be on where the bar is, rather than who the work is being done by.

With a new set of tier names, and a new set of requirements, we are going to start off all patterns in the Sandbox tier rather than grandfather them into the new names. Don’t panic, the code behind your favorite patterns have not suddenly regressed, we’re using the opportunity to work out any kinks in the promotion process and ensure patterns are classified consistently.

We expect to have finished re-reviewing the previous validated tier patterns by the end of 2023 and the previous community tier patterns by the end of Q1 2024.

QUICKLINKS

The efforts here started off with 2 different classes of patterns: “Community” and “Validated”, however this terminology dates back to before the effort had arrived at “Validated Patterns” as the official project name.

Having standardized on the use of “Validated Patterns” to refer to the overall initiative, it became confusing to refer to “Community” Validated Patterns and “Validated” Validated Patterns.

In addressing that confusion, we took the opportunity to design a new set of tiers:

Generally speaking, Sandbox aligns with the old Community tier, and Maintained aligns with the old Validated tier. However some of the requirements associated with those previous tiers were structured around our bandwidth and our priorities. In revisiting the tiers, we’ve removed many of those value judgements and shifted the emphasis to be on where the bar is, rather than who the work is being done by.

With a new set of tier names, and a new set of requirements, we are going to start off all patterns in the Sandbox tier rather than grandfather them into the new names. Don’t panic, the code behind your favorite patterns have not suddenly regressed, we’re using the opportunity to work out any kinks in the promotion process and ensure patterns are classified consistently.

We expect to have finished re-reviewing the previous validated tier patterns by the end of 2023 and the previous community tier patterns by the end of Q1 2024.

QUICKLINKS

CONTRIBUTE

Pattern Testing on Nutanix

by Andrew Beekhof
December 4, 2023
patterns announce -nutanix

I am pleased to announce the addition of the Nutanix platform to our CI dashboard for the Multi-cloud GitOps pattern.

Pattern consumers can now rest assured that the core pattern functionality will remain functional for deployments of OpenShift on the Nutanix platform.

This would not be possible without the wonderful co-operation of Nutanix, who are doing all the work of deploying OpenShift and our pattern on their platform, executing the tests, and reporting the results.

To facilitate this, the patterns team have begun the process of open sourcing the downstream tests for all our patterns. Soon all tests will live alongside the the patterns they target, allowing them to be easily executed and/or improved by pattern consumers and platform owners.

Our thanks once again to Nutanix.

QUICKLINKS

I am pleased to announce the addition of the Nutanix platform to our CI dashboard for the Multi-cloud GitOps pattern.

Pattern consumers can now rest assured that the core pattern functionality will remain functional for deployments of OpenShift on the Nutanix platform.

This would not be possible without the wonderful co-operation of Nutanix, who are doing all the work of deploying OpenShift and our pattern on their platform, executing the tests, and reporting the results.

To facilitate this, the patterns team have begun the process of open sourcing the downstream tests for all our patterns. Soon all tests will live alongside the the patterns they target, allowing them to be easily executed and/or improved by pattern consumers and platform owners.

Our thanks once again to Nutanix.

QUICKLINKS

CONTRIBUTE

QUICKLINKS

QUICKLINKS

CONTRIBUTE

QUICKLINKS

QUICKLINKS

CONTRIBUTE

QUICKLINKS

You will also need to delete the managedcluster resource
oc delete managedcluster <cluster_name>

Conclusion

Use this blog as a practical guide for creating, deleting and managing your hostedCluster resources using the Hosted Control Planes feature!

QUICKLINKS

CONTRIBUTE

QUICKLINKS

QUICKLINKS

CONTRIBUTE

# Points to the mirrored VP install chart
 export PATTERN_DISCONNECTED_HOME=registry.internal.disconnected.net/hybridcloudpatterns
 ./pattern.sh make install

After a while the cluster will converge to its desired final state and the -MultiCloud Gitops pattern will be installed successfully.

QUICKLINKS

QUICKLINKS

CONTRIBUTE

QUICKLINKS

To enable syntax highlighting for a programming language, use [source] tags used in the code block. For example:

  • [source,yaml]

  • [source,go]

  • [source,javascript]

QUICKLINKS

CONTRIBUTE

Size matters

If things are taking a long time to deploy, use the OpenShift console to check on memory and other potential capacity issues with the cluster. If running in a cloud you may wish to up the machine size. Check the sizing charts.

QUICKLINKS

Size matters

If things are taking a long time to deploy, use the OpenShift console to check on memory and other potential capacity issues with the cluster. If running in a cloud you may wish to up the machine size. Check the sizing charts.

QUICKLINKS

CONTRIBUTE

Add, Commit & Push

Steps:

  1. Use git status to see what’s changed that you need to add to your commit and add them using git add

  2. Commit the changes to the branch

  3. Push the branch to your fork.

~/git/multicloud-gitops> git status
 ~/git/multicloud-gitops> git add <the assets created/changed>
 ~/git/multicloud-gitops> git commit -m “Added Kafka using AMQ Stream operator and Helm charts”
-~/git/multicloud-gitops> git push origin multicloud-gitops

Watch OpenShift GitOps hub cluster UI and see Kafka get deployed

Let’s check the OpenShift console. This can take a bit of time for ArgoCD to pick it up and deploy the assets.

  1. Select installed operators. Is AMQ Streams Operator deployed?

  2. Select the Red Hat Integration - AMQ Streams operator.

  3. Select Kafka tab. Is there a new lab-cluster created?

  4. Select the Kafka Topic tab. Is there a lab-streams topic created?

This is a very simple and minimal Kafka set up. It is likely you will need to add more manifests to the Chart but it is a good start.

QUICKLINKS

Watch OpenShift GitOps hub cluster UI and see Kafka get deployed

Let’s check the OpenShift console. This can take a bit of time for ArgoCD to pick it up and deploy the assets.

  1. Select installed operators. Is AMQ Streams Operator deployed?

  2. Select the Red Hat Integration - AMQ Streams operator.

  3. Select Kafka tab. Is there a new lab-cluster created?

  4. Select the Kafka Topic tab. Is there a lab-streams topic created?

This is a very simple and minimal Kafka set up. It is likely you will need to add more manifests to the Chart but it is a good start.

QUICKLINKS

CONTRIBUTE

QUICKLINKS

CONTRIBUTE

FAQ

What is a Validated Pattern?

Validated Patterns are collections of applications (in the ArgoCD sense) that demonstrate aspects of hub/edge computing that seem interesting and useful. Validated Patterns will generally have a hub or centralized component, and an edge component. These will interact in different ways.

Many things have changed in the IT landscape in the last few years - containers and kubernetes have taken the industry by storm, but they introduce many technologies and concepts. It is not always clear how these technologies and concepts play together - and Validated Patterns is our effort to show these technologies working together on non-trivial applications in ways that make sense for real customers and partners to use.

The first Validated Pattern is based on MANUela, an application developed by Red Hat field associates. This application highlights some interesting aspects of the industrial edge in a cloud-native world - the hub component features pipelines to build the application, a "twin" for testing purposes, a central data lake, an s3 component to gather data from the edge installations (which are factories in this case). The edge component has machine sensors, which are responsible for only gathering data from instrumented line devices and shares them via MQTT messaging. The edge also features Seldon, an AI/ML framework for making predictions, a custom Node.js application to show data in real time, and messaging components supporting both MQTT and Kafka protocols. The local applications use MQTT to retrieve data for display, and the Kafka components move the data to the central hub for storage and analysis.

We are actively developing new Validated Patterns. Watch this space for updates!

How are they different from XYZ?

Many technology demos can be very minimal - such demos have an important place in the ecosystem to demonstrate the intent of an individual technology. Validated Patterns are meant to demonstrate groups of technologies working together in a cloud native way. And yet, we hope to make these patterns general enough to allow for swapping application components out — for example, if you want to swap out ActiveMQ for RabbitMQ to support MQTT - or use a different messaging technology altogether, that should be possible. The other components will require reconfiguration.

What technologies are used?

Key technologies in the stack for Industrial Edge include:

  • Red Hat OpenShift Container Platform

  • Red Hat Advanced Cluster Management

  • Red Hat OpenShift GitOps (based on ArgoCD)

  • Red Hat OpenShift Pipelines (based on tekton)

  • Red Hat Integration - AMQ Broker (ActiveMQ Artemis MQTT)

  • Red Hat Integration - AMQ Streams (Kafka)

  • Red Hat Integration - Camel K

  • Seldon Operator

In the future, we expect to further use Red Hat OpenShift, and expand the integrations with other elements of the ecosystem. How can the concept of GitOps integrate with a fleet of devices that are not running Kubernetes? What about integrations with baremetal or VM servers? Sounds like a job for Ansible! We expect to tackle some of these problems in future patterns.

How are they structured?

Validated Patterns come in parts - we have a common repository with logic that will apply to multiple patterns. Layered on top of that is our first pattern - industrial edge. This layout allows for individual applications within a pattern to be swapped out by pointing to different repositories or branches for those individual components by customizing the values files in the root of the repository to point to different branches or forks or even different repositories entirely. (At present, the repositories all have to be on github.com and accessible with the same token.)

The common repository is primarily concerned with how to deploy the GitOps operator, and to create the namespaces that will be necessary to manage the pattern applications.

The pattern repository has the application-specific layout, and determines which components are installed in which places - hub or edge. The pattern repository also defines the hub and edge locations. Both the hub and edge are expected to have multiple components each - the hub will have pipelines and the CI/CD framework, as well as any centralization components or data analysis components. Edge components are designed to be smaller as we do not need to deploy Pipelines or the test and staging areas to the Edge.

Each application is described as a series of resources that are rendered into GitOps (ArgoCD) via Helm and Kustomize. The values for these charts are set by values files that need to be "personalized" (with your local cluster values) as the first step of installation. Subsequent pushes to the gitops repository will be reflected in the clusters running the applications.

Who is behind this?

Today, a team of Red Hat engineers including Andrew Beekhof (@beekhof), Lester Claudio (@claudiol), Martin Jackson (@mhjacks), William Henry (@ipbabble), Michele Baldessari (@mbaldessari), Jonny Rickard (@day0hero) and others.

Excited or intrigued by what you see here? We’d love to hear your thoughts and ideas! Try the patterns contained here and see below for links to our repositories and issue trackers.

How can I get involved?

Try out what we’ve done and submit issues to our issue trackers.

We will review pull requests to our pattern repositories.

QUICKLINKS

FAQ

What is a Validated Pattern?

Validated Patterns are collections of applications (in the ArgoCD sense) that demonstrate aspects of hub/edge computing that seem interesting and useful. Validated Patterns will generally have a hub or centralized component, and an edge component. These will interact in different ways.

Many things have changed in the IT landscape in the last few years - containers and kubernetes have taken the industry by storm, but they introduce many technologies and concepts. It is not always clear how these technologies and concepts play together - and Validated Patterns is our effort to show these technologies working together on non-trivial applications in ways that make sense for real customers and partners to use.

The first Validated Pattern is based on MANUela, an application developed by Red Hat field associates. This application highlights some interesting aspects of the industrial edge in a cloud-native world - the hub component features pipelines to build the application, a "twin" for testing purposes, a central data lake, an s3 component to gather data from the edge installations (which are factories in this case). The edge component has machine sensors, which are responsible for only gathering data from instrumented line devices and shares them via MQTT messaging. The edge also features Seldon, an AI/ML framework for making predictions, a custom Node.js application to show data in real time, and messaging components supporting both MQTT and Kafka protocols. The local applications use MQTT to retrieve data for display, and the Kafka components move the data to the central hub for storage and analysis.

We are actively developing new Validated Patterns. Watch this space for updates!

How are they different from XYZ?

Many technology demos can be very minimal - such demos have an important place in the ecosystem to demonstrate the intent of an individual technology. Validated Patterns are meant to demonstrate groups of technologies working together in a cloud native way. And yet, we hope to make these patterns general enough to allow for swapping application components out — for example, if you want to swap out ActiveMQ for RabbitMQ to support MQTT - or use a different messaging technology altogether, that should be possible. The other components will require reconfiguration.

What technologies are used?

Key technologies in the stack for Industrial Edge include:

  • Red Hat OpenShift Container Platform

  • Red Hat Advanced Cluster Management

  • Red Hat OpenShift GitOps (based on ArgoCD)

  • Red Hat OpenShift Pipelines (based on tekton)

  • Red Hat Integration - AMQ Broker (ActiveMQ Artemis MQTT)

  • Red Hat Integration - AMQ Streams (Kafka)

  • Red Hat Integration - Camel K

  • Seldon Operator

In the future, we expect to further use Red Hat OpenShift, and expand the integrations with other elements of the ecosystem. How can the concept of GitOps integrate with a fleet of devices that are not running Kubernetes? What about integrations with baremetal or VM servers? Sounds like a job for Ansible! We expect to tackle some of these problems in future patterns.

How are they structured?

Validated Patterns come in parts - we have a common repository with logic that will apply to multiple patterns. Layered on top of that is our first pattern - industrial edge. This layout allows for individual applications within a pattern to be swapped out by pointing to different repositories or branches for those individual components by customizing the values files in the root of the repository to point to different branches or forks or even different repositories entirely. (At present, the repositories all have to be on github.com and accessible with the same token.)

The common repository is primarily concerned with how to deploy the GitOps operator, and to create the namespaces that will be necessary to manage the pattern applications.

The pattern repository has the application-specific layout, and determines which components are installed in which places - hub or edge. The pattern repository also defines the hub and edge locations. Both the hub and edge are expected to have multiple components each - the hub will have pipelines and the CI/CD framework, as well as any centralization components or data analysis components. Edge components are designed to be smaller as we do not need to deploy Pipelines or the test and staging areas to the Edge.

Each application is described as a series of resources that are rendered into GitOps (ArgoCD) via Helm and Kustomize. The values for these charts are set by values files that need to be "personalized" (with your local cluster values) as the first step of installation. Subsequent pushes to the gitops repository will be reflected in the clusters running the applications.

Who is behind this?

Today, a team of Red Hat engineers including Andrew Beekhof (@beekhof), Lester Claudio (@claudiol), Martin Jackson (@mhjacks), William Henry (@ipbabble), Michele Baldessari (@mbaldessari), Jonny Rickard (@day0hero) and others.

Excited or intrigued by what you see here? We’d love to hear your thoughts and ideas! Try the patterns contained here and see below for links to our repositories and issue trackers.

How can I get involved?

Try out what we’ve done and submit issues to our issue trackers.

We will review pull requests to our pattern repositories.

QUICKLINKS

CONTRIBUTE

Technical requirements

Consider these requirements specific to the implementation of all Validated Patterns and their tiers.

The requirements are categorized as follows:

Must

These are nonnegotiable, core requirements that must be implemented.

Should

These are important but not critical; their implementation enhances the pattern.

Can

These are optional or desirable features, but their absence does not hinder the implementation of a pattern.

Must

  1. Patterns must include one or more Git repositories in a publicly accessible location, containing configuration elements that can be consumed by the Red Hat OpenShift GitOps Operator without supplying custom Argo CD images.

  2. Patterns must be useful without all content stored in private Git repositories.

  3. Patterns must include a list of names and versions of all the products and projects that the pattern consumes.

  4. Patterns must be useful without any sample applications that are private or that lack public sources.

  5. Patterns must not degrade due to lack of updates or opaque incompatibilities in closed source applications.

  6. Patterns must not store sensitive data elements including, but not limited to, passwords in Git repositories.

  7. Patterns must be possible to deploy on any installer-provisioned infrastructure OpenShift cluster (BYO).

    Validated Patterns distinguish between the provisioning and configuration requirements of the initial cluster (Patterns) and of clusters or machines that are managed by the initial cluster (Managed clusters).

  8. Patterns must use a standardized clustergroup Helm chart as the initial Red Hat OpenShift GitOps application that describes all namespaces, subscriptions, and any other GitOps applications which contain the configuration elements that make up the solution.

  9. Managed clusters must operate on the premise of eventual consistency (automatic retries, and an expectation of idempotence), which is one of the essential benefits of the GitOps model.

  10. Imperative elements must be implemented as idempotent code stored in Git repository.

Should

  1. Patterns should include sample applications to demonstrate the business problems addressed by the pattern.

  2. Patterns should try to indicate which parts are foundational as opposed to being for demonstration purposes.

  3. Patterns should use the Validated Patterns Operator to deploy patterns. However, anything that creates the OpenShift GitOps subscription and initial clustergroup application could be acceptable.

  4. Patterns should embody the Open Hybrid Cloud model unless there is a compelling reason to limit the availability of functionality to a specific platform or topology.

  5. Patterns should use industry standards and Red Hat products for all required tooling.

    Patterns require current best practices at the time of pattern development. Solutions that do not conform to best practices should expect to justify non-conformance or expend engineering effort to conform.

  6. Patterns should not make use of upstream or community Operators and images except, depending on the market segment, where it is critical to the overall solution.

    Such Operators are forbidden to be deployed into an increasing number of customer environments, which limits the pattern reuse. Alternatively, consider to productize the Operator, and build it in-cluster from trusted sources as part of the pattern.

  7. Patterns should be decomposed into modules that perform a specific function, so that they can be reused in other patterns.

    For example, Bucket Notification is a capability in the Medical Diagnosis pattern that could be used for other solutions.

  8. Patterns should use Red Hat Ansible Automation Platform to drive the declarative provisioning and management of managed hosts, for example, Red Hat Enterprise Linux (RHEL). See also Imperative elements.

  9. Patterns should use Red Hat Advanced Cluster Management (RHACM) to manage policy and compliance on any managed clusters.

  10. Patterns should use RHACM and a standardized RHACM chart to deploy and configure OpenShift GitOps to managed clusters.

  11. Managed clusters should be loosely coupled to their hub, and use OpenShift GitOps to consume applications and configuration directly from Git as opposed to having hard dependencies on a centralized cluster.

  12. Managed clusters should use the pull deployment model for obtaining their configuration.

  13. Imperative elements should be implemented as Ansible playbooks.

  14. Imperative elements should be driven declaratively implying that the playbooks should be triggered by Jobs or CronJobs stored in Git and delivered by OpenShift GitOps.

Can

  1. Patterns can include additional configuration and/or demo elements located in one or more additional private Git repositories.

  2. Patterns can include automation that deploys a known set of clusters and/or machines in a specific topology.

  3. Patterns can limit functionality/testing claims to specific platforms, topologies, and cluster/node sizes.

  4. Patterns can consume Operators from established partners (for example, Hashicorp Vault, and Seldon)

  5. Patterns can include managed clusters.

  6. Patterns can include details or automation for provisioning managed clusters, or rely on the admin to pre-provision them out-of-band.

  7. Patterns can also choose to model multi-cluster solutions as an uncoordinated collection of initial hub clusters.

  8. Imperative elements can interact with cluster state or external influences.

QUICKLINKS

Technical requirements

Consider these requirements specific to the implementation of all Validated Patterns and their tiers.

The requirements are categorized as follows:

Must

These are nonnegotiable, core requirements that must be implemented.

Should

These are important but not critical; their implementation enhances the pattern.

Can

These are optional or desirable features, but their absence does not hinder the implementation of a pattern.

Must

  1. Patterns must include one or more Git repositories in a publicly accessible location, containing configuration elements that can be consumed by the Red Hat OpenShift GitOps Operator without supplying custom Argo CD images.

  2. Patterns must be useful without all content stored in private Git repositories.

  3. Patterns must include a list of names and versions of all the products and projects that the pattern consumes.

  4. Patterns must be useful without any sample applications that are private or that lack public sources.

  5. Patterns must not degrade due to lack of updates or opaque incompatibilities in closed source applications.

  6. Patterns must not store sensitive data elements including, but not limited to, passwords in Git repositories.

  7. Patterns must be possible to deploy on any installer-provisioned infrastructure OpenShift cluster (BYO).

    Validated Patterns distinguish between the provisioning and configuration requirements of the initial cluster (Patterns) and of clusters or machines that are managed by the initial cluster (Managed clusters).

  8. Patterns must use a standardized clustergroup Helm chart as the initial Red Hat OpenShift GitOps application that describes all namespaces, subscriptions, and any other GitOps applications which contain the configuration elements that make up the solution.

  9. Managed clusters must operate on the premise of eventual consistency (automatic retries, and an expectation of idempotence), which is one of the essential benefits of the GitOps model.

  10. Imperative elements must be implemented as idempotent code stored in Git repository.

Should

  1. Patterns should include sample applications to demonstrate the business problems addressed by the pattern.

  2. Patterns should try to indicate which parts are foundational as opposed to being for demonstration purposes.

  3. Patterns should use the Validated Patterns Operator to deploy patterns. However, anything that creates the OpenShift GitOps subscription and initial clustergroup application could be acceptable.

  4. Patterns should embody the Open Hybrid Cloud model unless there is a compelling reason to limit the availability of functionality to a specific platform or topology.

  5. Patterns should use industry standards and Red Hat products for all required tooling.

    Patterns require current best practices at the time of pattern development. Solutions that do not conform to best practices should expect to justify non-conformance or expend engineering effort to conform.

  6. Patterns should not make use of upstream or community Operators and images except, depending on the market segment, where it is critical to the overall solution.

    Such Operators are forbidden to be deployed into an increasing number of customer environments, which limits the pattern reuse. Alternatively, consider to productize the Operator, and build it in-cluster from trusted sources as part of the pattern.

  7. Patterns should be decomposed into modules that perform a specific function, so that they can be reused in other patterns.

    For example, Bucket Notification is a capability in the Medical Diagnosis pattern that could be used for other solutions.

  8. Patterns should use Red Hat Ansible Automation Platform to drive the declarative provisioning and management of managed hosts, for example, Red Hat Enterprise Linux (RHEL). See also Imperative elements.

  9. Patterns should use Red Hat Advanced Cluster Management (RHACM) to manage policy and compliance on any managed clusters.

  10. Patterns should use RHACM and a standardized RHACM chart to deploy and configure OpenShift GitOps to managed clusters.

  11. Managed clusters should be loosely coupled to their hub, and use OpenShift GitOps to consume applications and configuration directly from Git as opposed to having hard dependencies on a centralized cluster.

  12. Managed clusters should use the pull deployment model for obtaining their configuration.

  13. Imperative elements should be implemented as Ansible playbooks.

  14. Imperative elements should be driven declaratively implying that the playbooks should be triggered by Jobs or CronJobs stored in Git and delivered by OpenShift GitOps.

Can

  1. Patterns can include additional configuration and/or demo elements located in one or more additional private Git repositories.

  2. Patterns can include automation that deploys a known set of clusters and/or machines in a specific topology.

  3. Patterns can limit functionality/testing claims to specific platforms, topologies, and cluster/node sizes.

  4. Patterns can consume Operators from established partners (for example, Hashicorp Vault, and Seldon)

  5. Patterns can include managed clusters.

  6. Patterns can include details or automation for provisioning managed clusters, or rely on the admin to pre-provision them out-of-band.

  7. Patterns can also choose to model multi-cluster solutions as an uncoordinated collection of initial hub clusters.

  8. Imperative elements can interact with cluster state or external influences.

QUICKLINKS

CONTRIBUTE

Importing a managed cluster

Many validated patterns require importing a cluster into a managed group. These groups have specific application sets that will be deployed and managed. Some examples are factory clusters in the Industrial Edge pattern, or development clusters in Multi-cluster DevSecOps pattern.

Red Hat Advanced Cluster Management (RHACM) can be used to create a cluster of a specific cluster group type. You can deploy a specific cluster that way if you have RHACM set up with credentials for deploying clusters. However in many cases an OpenShift cluster has already been created and will be imported into the set of clusters that RHACM is managing.

While you can create and deploy in this manner this section concentrates on importing an existing cluster and designating a specific managed cluster group type.

To deploy a cluster that can be imported into RHACM, use the openshift-install program provided at console.redhat.com. You will need login credentials.

Importing a cluster using the RHACM User Interface

Getting to the RHACM user interface

After ACM is installed a message regarding a "Web console update is available" will be displayed. Click on the "Refresh web console" link.

On the upper-left side you’ll see a pull down labeled "local-cluster". Select "All Clusters" from this pull down. This will navigate to the RHACM console and to its "Clusters" section

Select the "Import cluster" option.

Importing the cluster

On the "Import an existing cluster" page, enter the cluster name (arbitrary) and choose Kubeconfig as the "import mode". Add the tag clusterGroup= using the appropriate cluster group specified in the pattern. Press Import.

Using this method, you are done. Skip to the section in your pattern documentation that describes how you can confirm the pattern deployed correctly on the managed cluster.

Other potential import tools

There are a two other known ways to join a cluster to the RHACM hub. These methods are not supported but have been tested once. The patterns team no longer tests these methods. If these methods become supported we will maintain the documentation here.

Importing a cluster using the cm-cli tool

  1. Install the cm-cli (cm) (cluster management) command-line tool. See installation instructions here: cm-cli installation

  2. Obtain the KUBECONFIG file from the managed cluster.

  3. On the command-line login into the hub/datacenter cluster (use oc login or export the KUBECONFIG).

  4. Run the following command:

    cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-KUBECONFIG>

Importing a cluster using clusteradm tool

You can also use clusteradm to join a cluster. The following instructions explain what needs to be done. clusteradm is still in testing.

  1. To deploy a edge cluster you will need to get the hub/datacenter cluster’s token. You will need to install clusteradm. When it is installed run the following on existing hub/datacenter cluster:

    clusteradm get token

  2. When you run the clusteradm command above it replies with the token and also shows you the command to use on the managed. Login to the managed cluster with either of the following methods:

    oc login

    or

    export KUBECONFIG=~/<path-to-kubeconfig>

  3. Then request to that the managed join the datacenter hub.

    clusteradm join --hub-token <token from clusteradm get token command > <managed-cluster-name>

  4. Back on the hub cluster accept the join request.

    clusteradm accept --clusters <managed-cluster-name>

Designate the new cluster as a devel site

If you use the command line tools above you need to explicitly indicate that the imported cluster is part of a specific clusterGroup. If you haven’t tagged the cluster as clusterGroup=<managed-cluster-group> then do that now. Some examples of clusterGroup are factory, devel, or prod.

We do this by adding the label referenced in the managedSite’s clusterSelector.

  1. Find the new cluster.

    oc get managedclusters.cluster.open-cluster-management.io

  2. Apply the label.

    oc label managedclusters.cluster.open-cluster-management.io/<your-cluster> clusterGroup=<managed-cluster-group>

QUICKLINKS

Importing a managed cluster

Many validated patterns require importing a cluster into a managed group. These groups have specific application sets that will be deployed and managed. Some examples are factory clusters in the Industrial Edge pattern, or development clusters in Multi-cluster DevSecOps pattern.

Red Hat Advanced Cluster Management (RHACM) can be used to create a cluster of a specific cluster group type. You can deploy a specific cluster that way if you have RHACM set up with credentials for deploying clusters. However in many cases an OpenShift cluster has already been created and will be imported into the set of clusters that RHACM is managing.

While you can create and deploy in this manner this section concentrates on importing an existing cluster and designating a specific managed cluster group type.

To deploy a cluster that can be imported into RHACM, use the openshift-install program provided at console.redhat.com. You will need login credentials.

Importing a cluster using the RHACM User Interface

Getting to the RHACM user interface

After ACM is installed a message regarding a "Web console update is available" will be displayed. Click on the "Refresh web console" link.

On the upper-left side you’ll see a pull down labeled "local-cluster". Select "All Clusters" from this pull down. This will navigate to the RHACM console and to its "Clusters" section

Select the "Import cluster" option.

Importing the cluster

On the "Import an existing cluster" page, enter the cluster name (arbitrary) and choose Kubeconfig as the "import mode". Add the tag clusterGroup= using the appropriate cluster group specified in the pattern. Press Import.

Using this method, you are done. Skip to the section in your pattern documentation that describes how you can confirm the pattern deployed correctly on the managed cluster.

Other potential import tools

There are a two other known ways to join a cluster to the RHACM hub. These methods are not supported but have been tested once. The patterns team no longer tests these methods. If these methods become supported we will maintain the documentation here.

Importing a cluster using the cm-cli tool

  1. Install the cm-cli (cm) (cluster management) command-line tool. See installation instructions here: cm-cli installation

  2. Obtain the KUBECONFIG file from the managed cluster.

  3. On the command-line login into the hub/datacenter cluster (use oc login or export the KUBECONFIG).

  4. Run the following command:

    cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-KUBECONFIG>

Importing a cluster using clusteradm tool

You can also use clusteradm to join a cluster. The following instructions explain what needs to be done. clusteradm is still in testing.

  1. To deploy a edge cluster you will need to get the hub/datacenter cluster’s token. You will need to install clusteradm. When it is installed run the following on existing hub/datacenter cluster:

    clusteradm get token

  2. When you run the clusteradm command above it replies with the token and also shows you the command to use on the managed. Login to the managed cluster with either of the following methods:

    oc login

    or

    export KUBECONFIG=~/<path-to-kubeconfig>

  3. Then request to that the managed join the datacenter hub.

    clusteradm join --hub-token <token from clusteradm get token command > <managed-cluster-name>

  4. Back on the hub cluster accept the join request.

    clusteradm accept --clusters <managed-cluster-name>

Designate the new cluster as a devel site

If you use the command line tools above you need to explicitly indicate that the imported cluster is part of a specific clusterGroup. If you haven’t tagged the cluster as clusterGroup=<managed-cluster-group> then do that now. Some examples of clusterGroup are factory, devel, or prod.

We do this by adding the label referenced in the managedSite’s clusterSelector.

  1. Find the new cluster.

    oc get managedclusters.cluster.open-cluster-management.io

  2. Apply the label.

    oc label managedclusters.cluster.open-cluster-management.io/<your-cluster> clusterGroup=<managed-cluster-group>

QUICKLINKS

CONTRIBUTE

Background

Each validated pattern has infrastructure requirements. The majority of the validated patterns will run Red Hat OpenShift while some parts will run directly on Red Hat Enterprise Linux or (RHEL), more likely, a version of RHEL called RHEL for Edge. It is expected that consumers of validated patterns already have the infrastructure in place using existing reliable and supported deployment tools. For more information and tools head over to console.redhat.com

Sizing

In this section we provide general minimum sizing requirements for such infrastructure but it is important to review specific requirements for a specific validated pattern. For example, Industrial Edge 2.0 employs AI/Ml technology that requires large machine instances to support the applications deployed on OpenShift at the datacenter.

QUICKLINKS

Background

Each validated pattern has infrastructure requirements. The majority of the validated patterns will run Red Hat OpenShift while some parts will run directly on Red Hat Enterprise Linux or (RHEL), more likely, a version of RHEL called RHEL for Edge. It is expected that consumers of validated patterns already have the infrastructure in place using existing reliable and supported deployment tools. For more information and tools head over to console.redhat.com

Sizing

In this section we provide general minimum sizing requirements for such infrastructure but it is important to review specific requirements for a specific validated pattern. For example, Industrial Edge 2.0 employs AI/Ml technology that requires large machine instances to support the applications deployed on OpenShift at the datacenter.

QUICKLINKS

CONTRIBUTE

1The Chart.yaml file contains chart metadata, such as the name and version of the chart.
2The templates directory contains files that define application resources such as deployments.
3The values.yaml file contains default values for the chart.

ArgoCD and Helm Integration

ArgoCD integrates with Helm to provide a powerful GitOps-based deployment mechanism. The validated patterns framework uses ArgoCD and Helm to streamline application deployment by defining applications as Helm charts stored in Git repositories. ArgoCD is the tool of choice to apply the desired state of desired application to the target cluster environment.

ArgoCD automates the deployment and synchronization of these applications to OpenShift Container Platform clusters, ensuring consistency, reliability, and efficiency in managing Kubernetes applications. This integration supports automated, declarative, and version-controlled deployments, enhancing operational efficiency and maintaining application state across environments. ArgoCD helps implement continuous deployment for cloud-native applications.

Values

Values files are essential for customizing settings in applications, services, or validated patterns, particularly in Kubernetes deployments using Helm charts. These files, written in plain YAML format, provide a structured and flexible approach to set parameters and configurations for deploying validated patterns. The values files contain the variables that drive the configurations of your namespaces, subscriptions, applications, and other resources. The variables defined in your values files are referenced within your Helm chart templates. This ensures consistency and enables dynamic configurations. Combined with the power of the Helm’s templating language you can implement conditionals and loops for adaptable and scalable configurations.

Key characteristics of values files include:

  • Plain YAML Format: The human-readable and easy-to-edit syntax of YAML makes configuration settings accessible and straightforward to manage.

  • Hierarchical Nature: Values files support a hierarchy of values, allowing logical organization and structuring of configurations, which is especially useful in handling complex deployments. -In Helm charts, values files define configuration settings for deploying applications and managing resources within an OpenShift Container Platform cluster. They enable flexible, per-cluster customization while ensuring consistency with the overall validated pattern. This ensures that organizations can achieve efficient, secure, and consistent deployments across multiple OpenShift Container Platform clusters.

A common practice is to use a base values file, such as values-global.yaml, for global settings, and then have cluster-specific values files for example values-cluster1.yaml, values-cluster2.yaml that override or add to the global settings. This approach allows for comprehensive customization while maintaining a centralized and organized configuration structure, promoting best practices for deployment and resource management.

For more information, see Exploring values.

Applications

The applications section in the Helm values file plays a crucial role in defining and managing the deployment of various applications within an OpenShift Container Platform cluster. By leveraging Helm charts and adhering to validated patterns, it ensures consistency, best practices, and simplified management, leading to reliable and scalable application deployments.

The path field in each application entry points in the values file points to the location of the Helm chart and associated configuration files. These charts contain the Kubernetes manifests and configuration necessary to deploy the application. Helm charts are used to package Kubernetes applications and manage their deployment in a consistent and reproducible manner.

When these applications are deployed, the following Kubernetes resources are typically created:

  • Deployments: Define the desired state and replicas for the application’s pods.

  • Services: Expose the application’s pods to other services or external traffic.

  • ConfigMaps and Secrets: Store configuration data and sensitive information.

  • PersistentVolumeClaims (PVCs): Request storage resources for the application.

  • Ingress or Routes: Provide external access to the application.

  • RBAC (Role-Based Access Control): Define access permissions and roles.

Red Hat Advanced Cluster Management (RHACM)

One of the applications deployed by the Validated Patterns Operator is Red Hat Advanced Cluster Management (RHACM). RHACM is a comprehensive solution designed to manage multiple OpenShift Container Platform clusters, whether that is ten clusters or a thousand clusters and enforce policies across those clusters from a single pane of glass.

RHACM plays a pivotal role in the validated pattern framework by providing robust capabilities for managing Kubernetes clusters and enforcing policies across heterogeneous environments. RHACM is only installed when a pattern spans multiple clusters. It supports operational efficiency, scalability, compliance, and security, making it an essential tool for organizations looking to manage their Kubernetes infrastructure effectively.

The Validated Patterns framework uses ACM policies to ensure that applications, targeted for specific clusters, are deployed to the appropriate cluster environments. The single pane of glass allows you to see information about your clusters. RHACM supports multiple cloud providers out of the box and it gives you a clear insight into the resources for that cluster using the observability feature.

ClusterGroups

In a validated pattern, a ClusterGroup organizes and manages clusters sharing common configurations, policies, or deployment needs, with the default group initially encompassing all clusters unless assigned elsewhere. Multiple cluster groups within a pattern allow for tailored management, enabling specific configurations and policies based on roles, environments, or locations. This segmentation enhances efficiency, consistency, and simplifies complex environments. In the validated patterns framework, a ClusterGroup is a key entity representing either a single cluster or a collection of clusters with unique configurations, determined by Helm charts and Kubernetes features. Typically, a ClusterGroup serves as the foundation for each pattern, with the primary one named in values-global.yaml, often referred to as hub. Managed ClusterGroups can also be defined, specifying characteristics and policies for additional clusters. Managed cluster groups are sets of clusters, grouped by function, that share a common configuration set. There is no limitation on the number of groups, or the number of clusters within each group.

When joining a managed cluster to Red Hat Advanced Cluster Management (RHACM) or deploying a new cluster with RHACM, it must be assigned to at least one ClusterGroup. RHACM identifies the managed cluster’s membership in a ClusterGroup and proceeds to set up the cluster, including installing the RHACM agent. Once the setup is complete, RHACM deploys GitOps and supplies it with information about the ClusterGroup. GitOps then retrieves the associated values file and proceeds to deploy the Operators, configurations, and charts accordingly.

For more information, see ClusterGroup configuration in values files.

GitOps

GitOps is a way to manage cloud-native systems that are powered by Kubernetes. It leverages a policy-as-code approach to define and manage every layer of the modern application stack from infrastructure, networking application code, and the GitOps pipeline itself.

The key principle of GitOps are:

  • Declarative: The methodology requires describing the desired state, achieved through raw manifests, helm charts, kustomize, or other forms of automation.

  • Versioned and immutability: Git ensures versioning and immutability, serving as the definitive source of truth. Version control and historical tracking offer insights into changes that impact the clusters.

  • Pulled automatically: The GitOps controller pulls the state automatically to prevent any errors introduced by humans, and it also allows the application an opportunity to heal itself.

  • Continuously reconciled: The GitOps controller has a reconciliation loop that by default runs every 3 minutes. When the reconciler identifies a diff between git and the cluster, it will reconcile the change onto the cluster during the next synchronization.

GitOps within the validated pattern framework ensures that infrastructure and application configurations are managed declaratively, consistently, and securely. GitOps ensures consistency across our environments, platforms and applications.

For more information, see GitOps.

Namespaces

Namespaces in a validated pattern are essential for organizing and managing resources within an OpenShift Container Platform cluster, ensuring security, consistency, and efficient resource allocation. Recommendations for defining namespaces include using consistent naming conventions, ensuring isolation and security through policies and RBAC, setting resource quotas, tailoring configurations to specific environments, and designing namespaces for modularity and reusability across patterns or applications.

Operators generally create their own namespaces, but you might need to create additional ones. Check if there are expected namespaces with the product before creating new ones.

For more information, see Understanding Namespace Creation using the Validated Patterns Framework.

Subscriptions

Subscriptions in a validated pattern typically refer to a methodical approach to managing and deploying applications and services within an OpenShift cluster. Subscriptions within a ClusterGroup in validated patterns streamline access management and resource allocation, allowing administrators to efficiently control user and application permissions, allocate resources precisely, and enforce governance policies.

Subscriptions are defined in the values files and they are OpenShift Operator subscriptions from the Operator Hub. Subscriptions contribute to the creation of a software bill of materials (SBOM), enhancing transparency and security by detailing all intended installations within the ClusterGroup. Managed through the Operator Lifecycle Manager (OLM), these subscriptions ensure continuous operation and upgrades of operators, similar to RPM packages on RHEL, thus maintaining cluster health and security.

To maximize the benefits of subscriptions, it is crucial to align them with organizational needs, integrate automated monitoring and alerting, and regularly review and update subscription plans.

Secrets

Enterprise applications, especially in multi-cluster and multi-site environments, require robust security measures, including the use of certificates and other secrets to establish trust. Managing these secrets effectively is crucial.

Ignoring security during the development of distributed enterprise applications can lead to significant technical debt. The DevSecOps model addresses this by emphasizing the need to integrate security early in the development lifecycle, known as "shifting security to the left."

In the OpenShift Container Platform, secrets are used to securely store sensitive information like passwords, API keys, and certificates. These secrets are managed using Kubernetes secret objects within validated patterns, ensuring consistent, secure, and compliant deployments. This approach promotes best practices for security and simplifies the management of sensitive data across OpenShift Container Platform Container Platform deployments.

For more information, see Overview of secrets management.

Shared values

Shared values files are YAML files used in a validated pattern to centralize and manage configuration settings. They define common parameters and settings that can be reused across multiple clusters, applications, and environments. This approach promotes consistency and simplifies configuration management.

Shared values files are a powerful mechanism in a validated pattern, enabling centralized, consistent, and reusable configuration management across multiple clusters and environments. By defining global settings and leveraging cluster-specific overrides, they ensure that configurations are both standardized and flexible enough to accommodate specific needs of individual clusters.

Tests

Tests within the Validated Pattern Framework are essential components to validate and ensure that the deployed patterns and configurations adhere to operational standards, security protocols, performance expectations, and compliance requirements in Kubernetes and OpenShift Container Platform environments.

QUICKLINKS

A common practice is to use a base values file, such as values-global.yaml, for global settings, and then have cluster-specific values files for example values-cluster1.yaml, values-cluster2.yaml that override or add to the global settings. This approach allows for comprehensive customization while maintaining a centralized and organized configuration structure, promoting best practices for deployment and resource management.

For more information, see Exploring values.

Applications

The applications section in the Helm values file plays a crucial role in defining and managing the deployment of various applications within an OpenShift Container Platform cluster. By leveraging Helm charts and adhering to validated patterns, it ensures consistency, best practices, and simplified management, leading to reliable and scalable application deployments.

The path field in each application entry points in the values file points to the location of the Helm chart and associated configuration files. These charts contain the Kubernetes manifests and configuration necessary to deploy the application. Helm charts are used to package Kubernetes applications and manage their deployment in a consistent and reproducible manner.

When these applications are deployed, the following Kubernetes resources are typically created:

  • Deployments: Define the desired state and replicas for the application’s pods.

  • Services: Expose the application’s pods to other services or external traffic.

  • ConfigMaps and Secrets: Store configuration data and sensitive information.

  • PersistentVolumeClaims (PVCs): Request storage resources for the application.

  • Ingress or Routes: Provide external access to the application.

  • RBAC (Role-Based Access Control): Define access permissions and roles.

Red Hat Advanced Cluster Management (RHACM)

One of the applications deployed by the Validated Patterns Operator is Red Hat Advanced Cluster Management (RHACM). RHACM is a comprehensive solution designed to manage multiple OpenShift Container Platform clusters, whether that is ten clusters or a thousand clusters and enforce policies across those clusters from a single pane of glass.

RHACM plays a pivotal role in the validated pattern framework by providing robust capabilities for managing Kubernetes clusters and enforcing policies across heterogeneous environments. RHACM is only installed when a pattern spans multiple clusters. It supports operational efficiency, scalability, compliance, and security, making it an essential tool for organizations looking to manage their Kubernetes infrastructure effectively.

The Validated Patterns framework uses ACM policies to ensure that applications, targeted for specific clusters, are deployed to the appropriate cluster environments. The single pane of glass allows you to see information about your clusters. RHACM supports multiple cloud providers out of the box and it gives you a clear insight into the resources for that cluster using the observability feature.

ClusterGroups

In a validated pattern, a ClusterGroup organizes and manages clusters sharing common configurations, policies, or deployment needs, with the default group initially encompassing all clusters unless assigned elsewhere. Multiple cluster groups within a pattern allow for tailored management, enabling specific configurations and policies based on roles, environments, or locations. This segmentation enhances efficiency, consistency, and simplifies complex environments. In the validated patterns framework, a ClusterGroup is a key entity representing either a single cluster or a collection of clusters with unique configurations, determined by Helm charts and Kubernetes features. Typically, a ClusterGroup serves as the foundation for each pattern, with the primary one named in values-global.yaml, often referred to as hub. Managed ClusterGroups can also be defined, specifying characteristics and policies for additional clusters. Managed cluster groups are sets of clusters, grouped by function, that share a common configuration set. There is no limitation on the number of groups, or the number of clusters within each group.

When joining a managed cluster to Red Hat Advanced Cluster Management (RHACM) or deploying a new cluster with RHACM, it must be assigned to at least one ClusterGroup. RHACM identifies the managed cluster’s membership in a ClusterGroup and proceeds to set up the cluster, including installing the RHACM agent. Once the setup is complete, RHACM deploys GitOps and supplies it with information about the ClusterGroup. GitOps then retrieves the associated values file and proceeds to deploy the Operators, configurations, and charts accordingly.

For more information, see ClusterGroup configuration in values files.

GitOps

GitOps is a way to manage cloud-native systems that are powered by Kubernetes. It leverages a policy-as-code approach to define and manage every layer of the modern application stack from infrastructure, networking application code, and the GitOps pipeline itself.

The key principle of GitOps are:

  • Declarative: The methodology requires describing the desired state, achieved through raw manifests, helm charts, kustomize, or other forms of automation.

  • Versioned and immutability: Git ensures versioning and immutability, serving as the definitive source of truth. Version control and historical tracking offer insights into changes that impact the clusters.

  • Pulled automatically: The GitOps controller pulls the state automatically to prevent any errors introduced by humans, and it also allows the application an opportunity to heal itself.

  • Continuously reconciled: The GitOps controller has a reconciliation loop that by default runs every 3 minutes. When the reconciler identifies a diff between git and the cluster, it will reconcile the change onto the cluster during the next synchronization.

GitOps within the validated pattern framework ensures that infrastructure and application configurations are managed declaratively, consistently, and securely. GitOps ensures consistency across our environments, platforms and applications.

For more information, see GitOps.

Namespaces

Namespaces in a validated pattern are essential for organizing and managing resources within an OpenShift Container Platform cluster, ensuring security, consistency, and efficient resource allocation. Recommendations for defining namespaces include using consistent naming conventions, ensuring isolation and security through policies and RBAC, setting resource quotas, tailoring configurations to specific environments, and designing namespaces for modularity and reusability across patterns or applications.

Operators generally create their own namespaces, but you might need to create additional ones. Check if there are expected namespaces with the product before creating new ones.

For more information, see Understanding Namespace Creation using the Validated Patterns Framework.

Subscriptions

Subscriptions in a validated pattern typically refer to a methodical approach to managing and deploying applications and services within an OpenShift cluster. Subscriptions within a ClusterGroup in validated patterns streamline access management and resource allocation, allowing administrators to efficiently control user and application permissions, allocate resources precisely, and enforce governance policies.

Subscriptions are defined in the values files and they are OpenShift Operator subscriptions from the Operator Hub. Subscriptions contribute to the creation of a software bill of materials (SBOM), enhancing transparency and security by detailing all intended installations within the ClusterGroup. Managed through the Operator Lifecycle Manager (OLM), these subscriptions ensure continuous operation and upgrades of operators, similar to RPM packages on RHEL, thus maintaining cluster health and security.

To maximize the benefits of subscriptions, it is crucial to align them with organizational needs, integrate automated monitoring and alerting, and regularly review and update subscription plans.

Secrets

Enterprise applications, especially in multi-cluster and multi-site environments, require robust security measures, including the use of certificates and other secrets to establish trust. Managing these secrets effectively is crucial.

Ignoring security during the development of distributed enterprise applications can lead to significant technical debt. The DevSecOps model addresses this by emphasizing the need to integrate security early in the development lifecycle, known as "shifting security to the left."

In the OpenShift Container Platform, secrets are used to securely store sensitive information like passwords, API keys, and certificates. These secrets are managed using Kubernetes secret objects within validated patterns, ensuring consistent, secure, and compliant deployments. This approach promotes best practices for security and simplifies the management of sensitive data across OpenShift Container Platform Container Platform deployments.

For more information, see Overview of secrets management.

Shared values

Shared values files are YAML files used in a validated pattern to centralize and manage configuration settings. They define common parameters and settings that can be reused across multiple clusters, applications, and environments. This approach promotes consistency and simplifies configuration management.

Shared values files are a powerful mechanism in a validated pattern, enabling centralized, consistent, and reusable configuration management across multiple clusters and environments. By defining global settings and leveraging cluster-specific overrides, they ensure that configurations are both standardized and flexible enough to accommodate specific needs of individual clusters.

Tests

Tests within the Validated Pattern Framework are essential components to validate and ensure that the deployed patterns and configurations adhere to operational standards, security protocols, performance expectations, and compliance requirements in Kubernetes and OpenShift Container Platform environments.

QUICKLINKS

CONTRIBUTE

About the Validated Patterns Maintained tier

A pattern categorized under the maintained tier implies that the pattern was known to be functional on all currently supported extended update support (EUS) versions of Red Hat OpenShift Container Platform. Qualifying for this tier might require additional work for the pattern’s owner who might be a partner or a sufficiently motivated subject matter expert (SME).

Nominating a pattern for the maintained tier

If your pattern qualifies or meets the criteria for maintained tier, submit your nomination to validatedpatterns@googlegroups.com.

Each maintained pattern represents an ongoing maintenance, support, and testing effort. Finite team capacity means that it is not possible for the team to take on this responsibility for all Validated Patterns.

For this reason we have designed the tiers and our processes to facilitate this to occur outside of the team by any sufficiently motivated party, including other parts of Red Hat, partners, and even customers.

In limited cases, the Validated Patterns team may consider taking on that work, however, it is recommended that you contact the team at least 4 weeks prior to the end of a given quarter for the necessary work to be considered as part of the following quarter’s planning process.

Requirements for the maintained tier

The maintained patterns have deliverable and requirements in addition to those -specified for the Tested tier.

The requirements are categorized as follows:

Must

These are nonnegotiable, core requirements that must be implemented.

Should

These are important but not critical; their implementation enhances the pattern.

Can

These are optional or desirable features, but their absence does not hinder the implementation of a pattern.

Must

A maintained pattern must continue to meet the following criteria to remain in maintained tier:

  • A maintained pattern must conform to the common technical implementation requirements.

  • A maintained pattern must only make use of components that are either supported, or easily substituted for supportable equivalents, for example, HashiCorp vault which has community and enterprise variants.

  • A maintained pattern must not rely on functionality in tech-preview, or hidden behind feature gates.

  • A maintained pattern must have their architectures reviewed by a representative of each Red Hat product they consume to ensure consistency with the product teams` intentions and roadmaps. Your patterns SME (eg. services rep) can help coordinate this.

  • A maintained pattern must include a link to a hosted presentation (Google Slides or similar) intended to promote the solution. The focus should be on the architecture and business problem being solved. No customer, or otherwise sensitive, information should be included.

  • A maintained pattern must include test plan automation that runs on every change to the pattern, or a schedule no less frequently than once per week.

  • A maintained pattern must be tested on all currently supported Red Hat OpenShift Container Platform extended update support (EUS) releases.

  • A maintained pattern must fix breakage in timely manner.

  • A maintained pattern must document their support policy.

    The individual products used in a Validated Patterns are backed by the full Red Hat support experience conditional on the customer’s subscription to those products, and the individual products`s support policy.

    Additional components in a Validated Patterns that are not supported by Red Hat; for example, Hashicorp Vault, and Seldon Core, require a customer to obtain support from that vendor directly.

The Validated Patterns team is will try to address any problems in the Validated Patterns Operator, and in the common Helm charts, but cannot not offer any SLAs at this time.

The maintained patterns do not imply an obligation of support for partner or community Operators by Red Hat.

Can

  • If you are creating Validated Patterns, you can provide your own SLA.

QUICKLINKS

The requirements are categorized as follows:

Must

These are nonnegotiable, core requirements that must be implemented.

Should

These are important but not critical; their implementation enhances the pattern.

Can

These are optional or desirable features, but their absence does not hinder the implementation of a pattern.

Must

A maintained pattern must continue to meet the following criteria to remain in maintained tier:

  • A maintained pattern must conform to the common technical implementation requirements.

  • A maintained pattern must only make use of components that are either supported, or easily substituted for supportable equivalents, for example, HashiCorp vault which has community and enterprise variants.

  • A maintained pattern must not rely on functionality in tech-preview, or hidden behind feature gates.

  • A maintained pattern must have their architectures reviewed by a representative of each Red Hat product they consume to ensure consistency with the product teams` intentions and roadmaps. Your patterns SME (eg. services rep) can help coordinate this.

  • A maintained pattern must include a link to a hosted presentation (Google Slides or similar) intended to promote the solution. The focus should be on the architecture and business problem being solved. No customer, or otherwise sensitive, information should be included.

  • A maintained pattern must include test plan automation that runs on every change to the pattern, or a schedule no less frequently than once per week.

  • A maintained pattern must be tested on all currently supported Red Hat OpenShift Container Platform extended update support (EUS) releases.

  • A maintained pattern must fix breakage in timely manner.

  • A maintained pattern must document their support policy.

    The individual products used in a Validated Patterns are backed by the full Red Hat support experience conditional on the customer’s subscription to those products, and the individual products`s support policy.

    Additional components in a Validated Patterns that are not supported by Red Hat; for example, Hashicorp Vault, and Seldon Core, require a customer to obtain support from that vendor directly.

The Validated Patterns team is will try to address any problems in the Validated Patterns Operator, and in the common Helm charts, but cannot not offer any SLAs at this time.

The maintained patterns do not imply an obligation of support for partner or community Operators by Red Hat.

Can

  • If you are creating Validated Patterns, you can provide your own SLA.

QUICKLINKS

CONTRIBUTE

OpenShift General Sizing

The OpenShift Container Platform node configuration file contains important options. For example, two parameters control the maximum number of pods that can be scheduled to a node: podsPerCore and maxPods.

When both options are in use, the lower of the two values limits the number of pods on a node. Exceeding these values can result in:

  • Increased CPU utilization.

  • Slow pod scheduling.

  • Potential out-of-memory scenarios, depending on the amount of memory in the node.

  • Exhausting the pool of IP addresses.

  • Resource overcommitting, leading to poor user application performance.

In Kubernetes, a pod that is holding a single container actually uses two containers. The second container is used to set up networking prior to the actual container starting. Therefore, a system running 10 pods will actually have 20 containers running.

podsPerCore sets the number of pods the node can run based on the number of processor cores on the node. For example, if podsPerCore is set to 10 on a node with 4 processor cores, the maximum number of pods allowed on the node will be 40.

kubeletConfig:
   podsPerCore: 10

Setting podsPerCore to 0 disables this limit. The default is 0. podsPerCore cannot exceed maxPods.

maxPods sets the number of pods the node can run to a fixed value, regardless of the properties of the node.

 kubeletConfig:
-    maxPods: 250

For more information about sizing and Red Hat standard host practices see the Official OpenShift Documentation Page for recommended host practices.

Control plane node sizing

The control plane node resource requirements depend on the number of nodes in the cluster. The following control plane node size recommendations are based on the results of control plane density focused testing. The control plane tests create the following objects across the cluster in each of the namespaces depending on the node counts:

  • 12 image streams

  • 3 build configurations

  • 6 builds

  • 1 deployment with 2 pod replicas mounting two secrets each

  • 2 deployments with 1 pod replica mounting two secrets

  • 3 services pointing to the previous deployments

  • 3 routes pointing to the previous deployments

  • 10 secrets, 2 of which are mounted by the previous deployments

  • 10 config maps, 2 of which are mounted by the previous deployments

Number of worker nodesCluster load (namespaces)CPU coresMemory (GB)

25

500

4

16

100

1000

8

32

250

4000

16

96

On a cluster with three masters or control plane nodes, the CPU and memory usage will spike up when one of the nodes is stopped, rebooted or fails because the remaining two nodes must handle the load in order to be highly available. This is also expected during upgrades because the masters are cordoned, drained, and rebooted serially to apply the operating system updates, as well as the control plane Operators update. To avoid cascading failures on large and dense clusters, keep the overall resource usage on the master nodes to at most half of all available capacity to handle the resource usage spikes. Increase the CPU and memory on the master nodes accordingly.

The node sizing varies depending on the number of nodes and object counts in the cluster. It also depends on whether the objects are actively being created on the cluster. During object creation, the control plane is more active in terms of resource usage compared to when the objects are in the running phase.

If you used an installer-provisioned infrastructure installation method, you cannot modify the control plane node size in a running OpenShift Container Platform 4.5 cluster. Instead, you must estimate your total node count and use the suggested control plane node size during installation.

The recommendations are based on the data points captured on OpenShift Container Platform clusters with OpenShiftSDN as the network plug-in.

In OpenShift Container Platform 4.5, half of a CPU core (500 millicore) is now reserved by the system by default compared to OpenShift Container Platform 3.11 and previous versions. The sizes are determined taking that into consideration.

For more information about sizing and Red Hat standard host practices see the Official OpenShift Documentation Page for recommended host practices.

For large and dense clusters, etcd can suffer from poor performance if the keyspace grows excessively large and exceeds the space quota. Periodic maintenance of etcd, including defragmentation, must be performed to free up space in the data store. It is highly recommended that you monitor Prometheus for etcd metrics and defragment it when required before etcd raises a cluster-wide alarm that puts the cluster into a maintenance mode, which only accepts key reads and deletes. Some of the key metrics to monitor are etcd_server_quota_backend_bytes which is the current quota limit, etcd_mvcc_db_total_size_in_use_in_bytes which indicates the actual database usage after a history compaction, and etcd_debugging_mvcc_db_total_size_in_bytes which shows the database size including free space waiting for defragmentation. Instructions on defragging etcd can be found in the Defragmenting etcd data section.

Etcd writes data to disk, so its performance strongly depends on disk performance. Etcd persists proposals on disk. Slow disks and disk activity from other processes might cause long fsync latencies, causing etcd to miss heartbeats, inability to commit new proposals to the disk on time, which can cause request timeouts and temporary leader loss. It is highly recommended to run etcd on machines backed by SSD/NVMe disks with low latency and high throughput.

Some of the key metrics to monitor on a deployed OpenShift Container Platform cluster are p99 of etcd disk write ahead log duration and the number of etcd leader changes. Use Prometheus to track these metrics. etcd_disk_wal_fsync_duration_seconds_bucket reports the etcd disk fsync duration, etcd_server_leader_changes_seen_total reports the leader changes. To rule out a slow disk and confirm that the disk is reasonably fast, 99th percentile of the etcd_disk_wal_fsync_duration_seconds_bucket should be less than 10ms.

For more information about sizing and Red Hat standard host practices see the Official OpenShift Documentation Page for recommended host practices.

QUICKLINKS

For more information about sizing and Red Hat standard host practices see the Official OpenShift Documentation Page for recommended host practices.

Control plane node sizing

The control plane node resource requirements depend on the number of nodes in the cluster. The following control plane node size recommendations are based on the results of control plane density focused testing. The control plane tests create the following objects across the cluster in each of the namespaces depending on the node counts:

  • 12 image streams

  • 3 build configurations

  • 6 builds

  • 1 deployment with 2 pod replicas mounting two secrets each

  • 2 deployments with 1 pod replica mounting two secrets

  • 3 services pointing to the previous deployments

  • 3 routes pointing to the previous deployments

  • 10 secrets, 2 of which are mounted by the previous deployments

  • 10 config maps, 2 of which are mounted by the previous deployments

Number of worker nodesCluster load (namespaces)CPU coresMemory (GB)

25

500

4

16

100

1000

8

32

250

4000

16

96

On a cluster with three masters or control plane nodes, the CPU and memory usage will spike up when one of the nodes is stopped, rebooted or fails because the remaining two nodes must handle the load in order to be highly available. This is also expected during upgrades because the masters are cordoned, drained, and rebooted serially to apply the operating system updates, as well as the control plane Operators update. To avoid cascading failures on large and dense clusters, keep the overall resource usage on the master nodes to at most half of all available capacity to handle the resource usage spikes. Increase the CPU and memory on the master nodes accordingly.

The node sizing varies depending on the number of nodes and object counts in the cluster. It also depends on whether the objects are actively being created on the cluster. During object creation, the control plane is more active in terms of resource usage compared to when the objects are in the running phase.

If you used an installer-provisioned infrastructure installation method, you cannot modify the control plane node size in a running OpenShift Container Platform 4.5 cluster. Instead, you must estimate your total node count and use the suggested control plane node size during installation.

The recommendations are based on the data points captured on OpenShift Container Platform clusters with OpenShiftSDN as the network plug-in.

In OpenShift Container Platform 4.5, half of a CPU core (500 millicore) is now reserved by the system by default compared to OpenShift Container Platform 3.11 and previous versions. The sizes are determined taking that into consideration.

For more information about sizing and Red Hat standard host practices see the Official OpenShift Documentation Page for recommended host practices.

For large and dense clusters, etcd can suffer from poor performance if the keyspace grows excessively large and exceeds the space quota. Periodic maintenance of etcd, including defragmentation, must be performed to free up space in the data store. It is highly recommended that you monitor Prometheus for etcd metrics and defragment it when required before etcd raises a cluster-wide alarm that puts the cluster into a maintenance mode, which only accepts key reads and deletes. Some of the key metrics to monitor are etcd_server_quota_backend_bytes which is the current quota limit, etcd_mvcc_db_total_size_in_use_in_bytes which indicates the actual database usage after a history compaction, and etcd_debugging_mvcc_db_total_size_in_bytes which shows the database size including free space waiting for defragmentation. Instructions on defragging etcd can be found in the Defragmenting etcd data section.

Etcd writes data to disk, so its performance strongly depends on disk performance. Etcd persists proposals on disk. Slow disks and disk activity from other processes might cause long fsync latencies, causing etcd to miss heartbeats, inability to commit new proposals to the disk on time, which can cause request timeouts and temporary leader loss. It is highly recommended to run etcd on machines backed by SSD/NVMe disks with low latency and high throughput.

Some of the key metrics to monitor on a deployed OpenShift Container Platform cluster are p99 of etcd disk write ahead log duration and the number of etcd leader changes. Use Prometheus to track these metrics. etcd_disk_wal_fsync_duration_seconds_bucket reports the etcd disk fsync duration, etcd_server_leader_changes_seen_total reports the leader changes. To rule out a slow disk and confirm that the disk is reasonably fast, 99th percentile of the etcd_disk_wal_fsync_duration_seconds_bucket should be less than 10ms.

For more information about sizing and Red Hat standard host practices see the Official OpenShift Documentation Page for recommended host practices.

QUICKLINKS

CONTRIBUTE

QUICKLINKS

QUICKLINKS

CONTRIBUTE

About the Validated Patterns Sandbox tier

A pattern categorized under the sandbox tier provides you with an entry point to onboard to the Validated Patterns. The minimum requirement to qualify for the sandbox tier is that you must start with the patterns framework and include minimal documentation.

Nominating a pattern for the sandbox tier

The Validated Patterns team has a preference for empowering others, and not taking credit for their work.

Where there is an existing application or a demonstration, there is also a strong preference for the originating team to own any changes that are needed for the implementation to become a validated pattern. Alternatively, if the Validated Patterns team drives the conversion, then to prevent confusion and duplicated efforts, we are likely to ask for a commitment to phase out use of the previous implementation for future engagements such as demos, presentations, and workshops.

The goal is to avoid bringing a parallel implementation into existence which divides engineering resources, and creates confusion internally and with customers as the implementations drift apart.

In both scenarios the originating team can choose where to host the primary repository, will be given admin permissions to any fork in https://github.com/validatedpatterns, and will receive on-going assistance from the Validated Patterns team.

Requirements for the sandbox tier

Consider these requirements for all sandbox tier.

The requirements are categorized as follows:

Must

These are nonnegotiable, core requirements that must be implemented.

Should

These are important but not critical; their implementation enhances the pattern.

Can

These are optional or desirable features, but their absence does not hinder the implementation of a pattern.

Must

A sandbox pattern must continue to meet the following criteria to remain in the sandbox tier:

  • A sandbox pattern must conform to the common technical implementation requirements.

  • A sandbox pattern must be able to be deployed onto a freshly deployed OpenShift cluster without prior modification or tuning.

  • A sandbox pattern must include a top-level README file that highlights the business problem and how the pattern solves it.

  • A sandbox pattern must include an architecture drawing. The specific tool or format is flexible as long as the meaning is clear.

  • A sandbox pattern must undergo an informal technical review by a community leader to ensure that it meets basic reuse standards.

  • A sandbox pattern must undergo an informal architecture review by a community leader to ensure that the solution has the right components, and they are generally being used as intended. For example, not using a database as a message bus.

    As community leaders, contributions from within Red Hat might be subject to a higher level of scrutiny. While we strive to be inclusive, the community will have quality standards and generally using the framework does not automatically imply a solution is suitable for the community to endorse/publish.

  • A sandbox pattern must document their support policy.

It is anticipated that most sandbox pattern will be supported by the community on a best-effort basis, but this should be stated explicitly. -The Validated Patterns team commits to maintaining the framework, but will also accept help.

Can

QUICKLINKS

Can

QUICKLINKS

CONTRIBUTE

Sub-parameters

  • region: The AWS region where the cluster is running.

  • server: The letsencrypt server endpoint for authentication.

  • email: The e-mail address to use as a DNS contact.

QUICKLINKS

Sub-parameters

  • region: The AWS region where the cluster is running.

  • server: The letsencrypt server endpoint for authentication.

  • email: The e-mail address to use as a DNS contact.

QUICKLINKS

CONTRIBUTE

Secrets

Secret Management

One area that has been impacted by a more automated approach to security is in the secret management. DevOps (and DevSecOps) environments require the use of many different services:

  1. Code repositories

  2. GitOps tools

  3. Image repositories

  4. Build pipelines

All of these services require credentials. (Or should do!) And keeping those credentials secret is very important. E.g. pushing your credentials to your personal GitHub/GitLab repository is not a secure solution.

While using a file based secret management can work if done correctly, most organizations opt for a more enterprise solution using a secret management product or project. The Cloud Native Computing Foundation (CNCF) has many such projects. The Validated Patterns project has started with Hashicorp Vault secret management product but we look forward to other project contributions.

QUICKLINKS

Secrets

Secret Management

One area that has been impacted by a more automated approach to security is in the secret management. DevOps (and DevSecOps) environments require the use of many different services:

  1. Code repositories

  2. GitOps tools

  3. Image repositories

  4. Build pipelines

All of these services require credentials. (Or should do!) And keeping those credentials secret is very important. E.g. pushing your credentials to your personal GitHub/GitLab repository is not a secure solution.

While using a file based secret management can work if done correctly, most organizations opt for a more enterprise solution using a secret management product or project. The Cloud Native Computing Foundation (CNCF) has many such projects. The Validated Patterns project has started with Hashicorp Vault secret management product but we look forward to other project contributions.

QUICKLINKS

CONTRIBUTE

QUICKLINKS

QUICKLINKS

CONTRIBUTE

About the Validated Patterns Tested tier

The tested tier provides you with additional collateral and reassurance that the pattern was known to be recently working on at least one recent version of Red Hat OpenShift Container Platform. Inclusion in this tier requires some additional work for the pattern’s owner, which might be a partner or a sufficiently motivated subject matter expert (SME).

Nominating a a pattern for the tested tier

If your pattern qualifies or meets the criteria for tested tier, submit your nomination to validatedpatterns@googlegroups.com.

Each tested pattern represents an ongoing maintenance, support, and testing effort. Finite team capacity means that it is not possible for the team to take on this responsibility for all Validated Patterns.

For this reason we have designed the tiers and our processes to facilitate this to occur outside of the team by any sufficiently motivated party, including other parts of Red Hat, partners, and even customers.

In limited cases, the Validated Patterns team may consider taking on that work, however please get in contact at least 4 weeks prior to the end of a given quarter in order for the necessary work to be considered as part of the following quarter’s planning process

Requirements for the tested tier

A tested patterns have deliverable and requirements in addition to those specified for the Sandbox tier.

The requirements are categorized as follows:

Must

These are nonnegotiable, core requirements that must be implemented.

Should

These are important but not critical; their implementation enhances the pattern.

Can

These are optional or desirable features, but their absence does not hinder the implementation of a pattern.

Must

A tested pattern must continue to meet the following criteria to remain in the tested tier:

  • A tested pattern must conform to the common technical implementation requirements.

  • A tested pattern must be meaningful without specialized hardware, including flavors of architectures not explicitly supported.

    Qualification is a Validated Patterns Technical Oversight Committee (TOC) decision with input from the pattern owner.

  • A tested pattern must have their implementation reviewed by the patterns team to ensure that it is sufficiently flexible to function across a variety of platforms, customer environments, and any relevant verticals.

  • A tested pattern must include a standardized architecture drawing, created with (or at least conforming to) the standard Validated Patterns tooling.

  • A tested pattern must include a written guide for others to follow when demonstrating the pattern.

  • A tested pattern must include a test plan covering all features or attributes being highlighted by the demonstration guide. Negative flow tests (such as resiliency or data retention in the presence of network outages) are also limited to scenarios covered by the demonstration guide.

    The test plan must define how to validate if the pattern has been successfully deployed and is functionally operational. Example: Validating an Industrial Edge Deployment.

  • A tested pattern must nominate at least one currently supported Red Hat OpenShift Container Platform release to test against.

  • A tested pattern must ensure the test plan passes at least once per quarter.

  • A tested pattern must create a publicly available JSON file (for example, in an AWS bucket) that records the result of the latest test for each combination of pattern, platform, and Red Hat OpenShift Container Platform version. See testing artifacts.

A tested pattern does not imply an obligation of support for partner or community operators by Red Hat or the pattern owner.

Should

  • A tested pattern should be broadly applicable.

  • A tested pattern should focus on functionality not performance.

Can

Teams creating tested pattern can provide their own service level agreement (SLA).

A technical document for Quality Engineering (QE) team that defines how to validate if the pattern has been successfully deployed and is functionally operational. -For example, see Validating an Industrial Edge Deployment.

A tested pattern meeting additional criteria can be nominated for promotion to the Maintained tier.

QUICKLINKS

A tested pattern meeting additional criteria can be nominated for promotion to the Maintained tier.

QUICKLINKS

CONTRIBUTE

About the Validated Patterns Operator

You can use the Validated Patterns Operator to install and manage Validated Patterns. Use the Red Hat Hybrid Cloud Console to install the Validated Patterns Operator. After installing the Operator, you can create an instance where you can specify the details for your pattern. The Validated Patterns Operator then installs and manages the required assets and artifacts that the pattern requires.

Installing the Validated Patterns Operator

Prerequisites

  • Access to an OpenShift Container Platform cluster using an account with cluster-admin permissions.

Procedure

  1. Navigate in the Red Hat Hybrid Cloud Console to the Operators → OperatorHub page.

  2. Scroll or type a keyword into the Filter by keyword box to find the Operator you want. For example, type validated patterns to find the Validated Patterns Operator.

  3. Select the Operator to display additional information.

    Choosing a Community Operator warns that Red Hat does not certify Community Operators; you must acknowledge the warning before continuing.

  4. Read the information about the Operator and click Install.

  5. On the Install Operator page:

    1. Select an Update channel (if more than one is available).

    2. Select a Version (if more than one is available).

    3. Select an Installation mode:

      • All namespaces on the cluster (default) installs the Operator in the default openshift-operators namespace to watch and be made available to all namespaces in the cluster. This option is not always available.

      • A specific namespace on the cluster allows you to choose a specific, single namespace in which to install the Operator. The Operator will only watch and be made available for use in this single namespace.

    4. Select Automatic or Manual approval strategy.

  6. Click Install to make the Operator available to the selected namespaces on this OpenShift Container Platform cluster.

Verification

To confirm that the installation is successful:

  1. Navigate to the Operators → Installed Operators page.

  2. Check that the Operator is installed in the selected namespace and its status is Succeeded.

Creating a pattern instance

Prerequisites

The Validated Patterns Operator is successfully installed in the relevant namespace.

Procedure

  1. Navigate to the Operators → Installed Operators page.

  2. Click the installed Validated Patterns Operator.

  3. Under the Details tab, in the Provided APIs section, in the -Pattern box, click Create Instance that displays the Create Pattern page.

  4. On the the Create Pattern page, select Form view and enter information in the following fields:

    • Name - A name for the pattern deployment that is used in the projects that you created.

    • Labels - Apply any other labels you might need for deploying this pattern.

    • Cluster Group Name - Select a cluster group name to identify the type of cluster where this pattern is being deployed. For example, if you are deploying the Industrial Edge pattern, the cluster group name is datacenter. If you are deploying the Multicloud GitOps pattern, the cluster group name is hub.

      To know the cluster group name for the patterns that you want to deploy, check the relevant pattern-specific requirements.

  5. Expand the Git Config section to reveal the options and enter the required information.

    1. Change the Target Repo URL to your forked repository URL. For example, change https://github.com/validatedpatterns/<pattern_name> to https://github.com/<your-git-username>/<pattern-name>

    2. Optional: You might need to change the Target Revision field. The default value is HEAD. However, you can also provide a value for a branch, tag, or commit that you want to deploy. For example, v2.1, main, or a branch that you created, my-branch.

  6. Ensure that you have made any required changes to your values-*.yaml files locally and pushed them to your forked repository on the correct branch or target that you chose in the previous step.

  7. Click Create.

Verification

The Red Hat OpenShift GitOps Operator displays in list of Installed Operators. The Red Hat OpenShift GitOps Operator installs the remaining assets and artifacts for this pattern. To view the installation of these assets and artifacts, such as Red Hat Advanced Cluster Management (RHACM), ensure that you switch to Project:All Projects.

For more information about post-installation instructions for a pattern, see its Getting started page.

QUICKLINKS

Verification

The Red Hat OpenShift GitOps Operator displays in list of Installed Operators. The Red Hat OpenShift GitOps Operator installs the remaining assets and artifacts for this pattern. To view the installation of these assets and artifacts, such as Red Hat Advanced Cluster Management (RHACM), ensure that you switch to Project:All Projects.

For more information about post-installation instructions for a pattern, see its Getting started page.

QUICKLINKS

CONTRIBUTE

Introduction

Validated patterns provides two frameworks to deploy applications: the OpenShift-based Validated Patterns framework and the Ansible GitOps Framework (AGOF).

The OpenShift-based validated patterns framework is the most common method for deploying applications and infrastructure on the OpenShift Container Platform. It offers a set of predefined configurations and patterns that follow best practices and are validated by Red Hat.

Ansible GitOps Framework (AGOF) is an alternative framework, designed to provide a framework for GitOps without Kubernetes. AGOF is not a pattern itself; it is a framework for installing Ansible Automation Platform (AAP), and then using that as the GitOps engine to drive other pattern work. AGOF comes with code to install VMs in AWS, if desired, or else it can work with previously provisioned VMs, or a functional AAP Controller endpoint.

The goal with either framework, is that developers, operators, security, and architects build a secure and repeatable day one deployment mechanism and maintenance automation for day two operations.

QUICKLINKS

Introduction

Validated patterns provides two frameworks to deploy applications: the OpenShift-based Validated Patterns framework and the Ansible GitOps Framework (AGOF).

The OpenShift-based validated patterns framework is the most common method for deploying applications and infrastructure on the OpenShift Container Platform. It offers a set of predefined configurations and patterns that follow best practices and are validated by Red Hat.

Ansible GitOps Framework (AGOF) is an alternative framework, designed to provide a framework for GitOps without Kubernetes. AGOF is not a pattern itself; it is a framework for installing Ansible Automation Platform (AAP), and then using that as the GitOps engine to drive other pattern work. AGOF comes with code to install VMs in AWS, if desired, or else it can work with previously provisioned VMs, or a functional AAP Controller endpoint.

The goal with either framework, is that developers, operators, security, and architects build a secure and repeatable day one deployment mechanism and maintenance automation for day two operations.

QUICKLINKS

CONTRIBUTE

In this example, the coffeeshop-test application has specific overrides, such as setting the projectnamespace to quarkuscoffeeshop-demo, the Release.release-namespace to quarkuscoffeeshop-demo, and the storeid to TEST. These values can be easily adjusted based on the desired configuration for the application, showcasing the flexibility provided by values files in tailoring deployments to specific requirements.

Outline of a basic values file

In a values file, each variable declares an element with a default value, influencing Helm templates and guiding validated patterns. As files or overrides enter the rendering pipeline, they replace default values in the values files, following the principle that the last value takes precedence for adaptability.

The three-fold distinction in validated patterns, "Global," "Main," and "ClusterGroup," practically applies the Helm override scheme. While "main" and "global" terms are used interchangeably, Helm differentiates by allowing "global" variables to extend to sub-charts, ensuring comprehensive configuration access. Values files, structured in free-form YAML, provide flexibility for any YAML variable type and are organized into Global, Main, and ClusterGroup configurations.

In Validated patterns, values files follow Helm rules, but the "global" and "ClusterGroup" sections have distinct meanings within this framework.

Increasingly, more value files are designed solely for overriding base chart values. These files may not always have "ClusterGroup," "Main," or "Global" values. The decision to include these sections depends on the specific overrides required for each case.

Key sections of values files:

Global

Global values refer to configuration settings shared across multiple charts within a deployment. These values are distributed among various charts to avoid redundancy and configuration setting repetition, promoting consistency. -Each pattern includes a values-global.yaml file, which contains configurations that are automatically incorporated into the application specification processed by the GitOps operator.

ClusterGroup

The ClusterGroup holds a specific significance within validated patterns. It defines clusters that are anticipated to share nearly identical configurations, serving a common architectural purpose. A ClusterGroup may consist of one or many members. Conventionally, there is a main ClusterGroup defined in main.clusterGroupName in values-global.yaml, acting as the hub or starting point for each pattern. Additional ClusterGroups can be managed by the main ClusterGroup through tools like Red Hat Advanced Cluster Management (RHACM).

Not all value files contain a ClusterGroup section; instead values-global sets the starting point, and these value files can subsequently define managed clusters.

QUICKLINKS

Not all value files contain a ClusterGroup section; instead values-global sets the starting point, and these value files can subsequently define managed clusters.

QUICKLINKS

CONTRIBUTE

The vault’s root token is needed to log into the vault’s UI and the unseal keys are needed whenever the vault pods are restarted. -In the OpenShift console click on the nine box at the top and click on the vault line:

Vault Nine Box

Copy the root_token field which in the example above has the value hvs.VNFq7yPuZljq2VDJTkgAMs2Z and paste it in the sign-in page:

Vault Sign In</a>

After signing in you will see the secrets that have been created.

Vault Secrets Engine</a>

Unseal

If you don’t see the sign in page but instead see an unseal page, something may have happened the cluster and you need to unseal it again. Make sure that the imperative/vaultkeys secret exists and wait for the CronJob called unseal-vault inside the imperative namespace to run. Alternatively you could unseal the vault manually by running vault operator commands inside the vault-0 pod. See these instructions for additional information.

What’s next?

Check with the validated pattern instructions to see if there are further steps you need to perform. Sometimes this might be deploying a pattern on an edge cluster and checking to see if the correct Vault handshaking and updating occurs.

QUICKLINKS

Vault Nine Box

Copy the root_token field which in the example above has the value hvs.VNFq7yPuZljq2VDJTkgAMs2Z and paste it in the sign-in page:

Vault Sign In</a>

After signing in you will see the secrets that have been created.

Vault Secrets Engine</a>

Unseal

If you don’t see the sign in page but instead see an unseal page, something may have happened the cluster and you need to unseal it again. Make sure that the imperative/vaultkeys secret exists and wait for the CronJob called unseal-vault inside the imperative namespace to run. Alternatively you could unseal the vault manually by running vault operator commands inside the vault-0 pod. See these instructions for additional information.

What’s next?

Check with the validated pattern instructions to see if there are further steps you need to perform. Sometimes this might be deploying a pattern on an edge cluster and checking to see if the correct Vault handshaking and updating occurs.

QUICKLINKS

CONTRIBUTE

QUICKLINKS

QUICKLINKS

CONTRIBUTE

Creating a validated pattern

The high level steps to create a validated pattern are as follows:

  1. Identify the business case you want to address.

  2. Use the Multicloud GitOps as a starting point by using the template to create a new Git repository in your local GitHub account.

  3. Identify the technology components that you need.

  4. Ensure that the configuration of those components, and the integration points are handled in the pattern.

Best Practices for creating Validated Patterns

Following these best practices, to create robust, scalable, and maintainable validated patterns that are easy to deploy and manage across various environments.

  1. Start with OpenShift Multicloud GitOps

    • When creating a validated pattern, begin with OpenShift Multicloud GitOps as your foundation. This is the most fundamental pattern and should serve as your base.

  2. Centralize all manifests in a Git Repository

    • Ensure that all pattern manifests, configuration files, and desired state definitions are stored in a Git repository. This provides a single source of truth and facilitates the use of GitOps for deployment.

  3. Include all necessary deployment resources

    • The Git repository should include everything required to deploy the pattern, including manifests, configuration, and any additional resources. Avoid relying on manual commands post-deployment; instead, incorporate these into the manifests.

  4. Adopt a clear separation of concerns

    • Maintain a clear separation between configuration files and development source code. This helps manage complexity and ensures that each aspect of the pattern is independently maintainable.

  5. Leverage GitOps for deployment consistency

    • Use GitOps to push the desired state from your Git repository to the Kubernetes cluster. This approach ensures consistency across multiple deployments and clusters.

  6. Use Helm charts for grouping manifests

    • If your deployment process involves multiple oc apply commands, group these manifests into a Helm chart. This simplifies deployment and integrates well with the pattern framework, which can apply the chart automatically.

  7. Integrate existing automation scripts

    • Review existing Ansible scripts or shell scripts that deploy resources. Where possible, convert these scripts into manifests or use the pattern’s framework to apply them imperatively. Avoid adding new scripts, as they often introduce technical debt.

  8. Ensure comprehensive deployment capabilities

    • The pattern should be capable of deploying all necessary resources to support the applications it includes. This means that once the pattern is applied, no additional manual steps should be required.

  9. Plan for cluster sizing and storage needs

    • Before deploying a pattern, review its cluster sizing and storage requirements. For instance, patterns using Red Hat OpenShift Data Foundation (ODF) may require substantial storage resources. Consult pattern documentation for accurate sizing.

  10. Customize and fork patterns appropriately

    • When extending a pattern or deploying it at a customer site, create a personal or organizational fork. This allows for experimentation and customization while facilitating potential contributions back to the upstream pattern repository.

  11. Consider managed cloud services for deployment

    • For ease of deployment, especially in public cloud environments, consider using managed OpenShift services like Red Hat OpenShift Service on AWS (ROSA) or Microsoft Azure Red Hat® OpenShift (ARO). These services simplify the setup and management of OpenShift clusters.

  12. Manage Multi-Cluster and Multi-Cloud deployments

    • Use the patterns framework to centrally manage multiple clusters and workloads across public and private clouds. Incorporate tools like Red Hat Advanced Cluster Management (RHACM) and Red Hat OpenShift GitOps (ArgoCD) to streamline operations.

  13. Secure secrets across deployments

    • Utilize tools such as Hashicorp Vault to manage secrets securely across multi-cloud deployments, ensuring sensitive data is protected.

QUICKLINKS

Creating a validated pattern

The high level steps to create a validated pattern are as follows:

  1. Identify the business case you want to address.

  2. Use the Multicloud GitOps as a starting point by using the template to create a new Git repository in your local GitHub account.

  3. Identify the technology components that you need.

  4. Ensure that the configuration of those components, and the integration points are handled in the pattern.

Best Practices for creating Validated Patterns

Following these best practices, to create robust, scalable, and maintainable validated patterns that are easy to deploy and manage across various environments.

  1. Start with OpenShift Multicloud GitOps

    • When creating a validated pattern, begin with OpenShift Multicloud GitOps as your foundation. This is the most fundamental pattern and should serve as your base.

  2. Centralize all manifests in a Git Repository

    • Ensure that all pattern manifests, configuration files, and desired state definitions are stored in a Git repository. This provides a single source of truth and facilitates the use of GitOps for deployment.

  3. Include all necessary deployment resources

    • The Git repository should include everything required to deploy the pattern, including manifests, configuration, and any additional resources. Avoid relying on manual commands post-deployment; instead, incorporate these into the manifests.

  4. Adopt a clear separation of concerns

    • Maintain a clear separation between configuration files and development source code. This helps manage complexity and ensures that each aspect of the pattern is independently maintainable.

  5. Leverage GitOps for deployment consistency

    • Use GitOps to push the desired state from your Git repository to the Kubernetes cluster. This approach ensures consistency across multiple deployments and clusters.

  6. Use Helm charts for grouping manifests

    • If your deployment process involves multiple oc apply commands, group these manifests into a Helm chart. This simplifies deployment and integrates well with the pattern framework, which can apply the chart automatically.

  7. Integrate existing automation scripts

    • Review existing Ansible scripts or shell scripts that deploy resources. Where possible, convert these scripts into manifests or use the pattern’s framework to apply them imperatively. Avoid adding new scripts, as they often introduce technical debt.

  8. Ensure comprehensive deployment capabilities

    • The pattern should be capable of deploying all necessary resources to support the applications it includes. This means that once the pattern is applied, no additional manual steps should be required.

  9. Plan for cluster sizing and storage needs

    • Before deploying a pattern, review its cluster sizing and storage requirements. For instance, patterns using Red Hat OpenShift Data Foundation (ODF) may require substantial storage resources. Consult pattern documentation for accurate sizing.

  10. Customize and fork patterns appropriately

    • When extending a pattern or deploying it at a customer site, create a personal or organizational fork. This allows for experimentation and customization while facilitating potential contributions back to the upstream pattern repository.

  11. Consider managed cloud services for deployment

    • For ease of deployment, especially in public cloud environments, consider using managed OpenShift services like Red Hat OpenShift Service on AWS (ROSA) or Microsoft Azure Red Hat® OpenShift (ARO). These services simplify the setup and management of OpenShift clusters.

  12. Manage Multi-Cluster and Multi-Cloud deployments

    • Use the patterns framework to centrally manage multiple clusters and workloads across public and private clouds. Incorporate tools like Red Hat Advanced Cluster Management (RHACM) and Red Hat OpenShift GitOps (ArgoCD) to streamline operations.

  13. Secure secrets across deployments

    • Utilize tools such as Hashicorp Vault to manage secrets securely across multi-cloud deployments, ensuring sensitive data is protected.

QUICKLINKS

CONTRIBUTE

You only need to change subtrees if you want to test changes in the common/ area of the pattern repositories, or if you wish to contribute to the common/ repository itself in conjunction with one of the patterns. Using the pattern by itself does not require changing subtrees.

For the common cases (use and consumption of the pattern), users do not need to be aware that the pattern uses a subtree at all.

$ git clone https://github.com/<your-workspace>/industrial-edge

If you want to change and track your own version of common, you should fork and clone our common repository separately:

$ git clone https://github.com/<your-workspace>/common

Now, you can make changes in your fork’s main branch, or else make a new branch and make changes there.

If you want to track these changes in your fork of the pattern repository (industrial-edge in this case), you will need to swap out the subtree in industrial-edge for the version of common you forked. We have provided a script to make this a bit easier:

$ common/scripts/make_common_subtree.sh <subtree_repo> <subtree_branch> <subtree_remote_name>

This script will set up a new remote in your local working directory with the repository you specify. It will replace the common directory with a new common from the fork and branch you specify, and commit it. The script will not push the result.

For example:

$ common/scripts/make_common_subtree.sh https://github.com/mhjacks/common.git wip-main common-subtree

This will replace common in the current repository with the wip-main branch from the common in mhjacks’s common repository, and call the remote common-subtree.

From that point, changes from mhjacks’s wip-main branch on mhjacks’s fork of common can be pulled in this way:

$ git subtree pull --prefix common common-subtree wip-main

When run without arguments, the script will run as if it had been given the following arguments:

$ common/scripts/make_common_subtree.sh https://github.com/validatedpatterns/common.git main common-subtree

Which are the defaults the repository is normally configured with.

Subtree vs. Submodule

It has always been important to us to be have a substrate for patterns that is as easy as possible to share amongst multiple patterns. While it is possible to share changes between multiple unrelated git repositories, it is an almost -entirely manual process, prone to error. We feel it is important to be able to provide a "pull" experience (i.e. one git "pull" type action) to update the shared components of a pattern. Two strategies exist for repository sharing in this way: submodule and subtree. We started with submodules but have since moved to subtree.

Atlassian has some good documentation on what subtree is here and here. In short, a subtree integrates another repository’s history into a parent repository, which allows for most of the benefits of a submodule workflow, without most of the caveats.

Earlier versions of this document described the usage of patterns with submodules instead of subtrees. In the earliest stages of pattern development, we used submodules because the developers of the project were familiar with submodules and had used them previously, but we had not used subtrees. User feedback, as well as some of the unavoidable complexities of submodules, convinced us to try subtrees and we believe we will stick with that strategy. Some of the unavoidable complexities of submodules include:

  • Having to remember to checkout repositories with --recurse-submdules, or else doing git submodule init && git submodule sync. Experienced developers asked in several of our support channels early on why common was empty.

  • Hoping that other tools that are interacting with the repository are compatible with the submodule approach. (To be fair, tools like ArgoCD and Tekton Pipelines did this very well; their support of submodules was one of the key reasons we started with submodules)

  • When changing branches on a submoduled repository, if the branch you were changing to was pointed to a different revision of the submoduled repository, the repository would show out of sync. While this behavior is correct, it can be surprising and difficult to navigate.

  • In disconnected environments, submodules require mirroring more repositories.

  • Developing with a fork of the submoduled repository means maintaining two forked repositories and multiple branches in both.

Subtrees have some pitfalls as well. In the subtree strategy, it is easier to diverge from the upstream version of the subtree repository, and in fact with a typical git clone, the user may not be aware that a subtree is in use at all. This can be considered a feature, but could become problematic if the user/consumer later wants to update to a newer version of the subtree but local changes might conflict. Additionally, since subtrees are not as well understood generally, there can be some surprising effects. In practice, we have run into the following:

  • Cherry picking from a subtree commit into the parent puts the change in the parent location, not the subtree

Contributing to Patterns using Common Subtrees

Once you have forked common and changed your subtree for testing, changes from your fork can then be proposed to [https://github.com/validatedpatterns/common.git] and can then be integrated into other patterns. A change to upstream common for a particular upstream pattern would have to be done in two stages:

  1. PR the change into upstream’s common

  2. PR the updated common into the pattern repository

QUICKLINKS

Atlassian has some good documentation on what subtree is here and here. In short, a subtree integrates another repository’s history into a parent repository, which allows for most of the benefits of a submodule workflow, without most of the caveats.

Earlier versions of this document described the usage of patterns with submodules instead of subtrees. In the earliest stages of pattern development, we used submodules because the developers of the project were familiar with submodules and had used them previously, but we had not used subtrees. User feedback, as well as some of the unavoidable complexities of submodules, convinced us to try subtrees and we believe we will stick with that strategy. Some of the unavoidable complexities of submodules include:

  • Having to remember to checkout repositories with --recurse-submdules, or else doing git submodule init && git submodule sync. Experienced developers asked in several of our support channels early on why common was empty.

  • Hoping that other tools that are interacting with the repository are compatible with the submodule approach. (To be fair, tools like ArgoCD and Tekton Pipelines did this very well; their support of submodules was one of the key reasons we started with submodules)

  • When changing branches on a submoduled repository, if the branch you were changing to was pointed to a different revision of the submoduled repository, the repository would show out of sync. While this behavior is correct, it can be surprising and difficult to navigate.

  • In disconnected environments, submodules require mirroring more repositories.

  • Developing with a fork of the submoduled repository means maintaining two forked repositories and multiple branches in both.

Subtrees have some pitfalls as well. In the subtree strategy, it is easier to diverge from the upstream version of the subtree repository, and in fact with a typical git clone, the user may not be aware that a subtree is in use at all. This can be considered a feature, but could become problematic if the user/consumer later wants to update to a newer version of the subtree but local changes might conflict. Additionally, since subtrees are not as well understood generally, there can be some surprising effects. In practice, we have run into the following:

  • Cherry picking from a subtree commit into the parent puts the change in the parent location, not the subtree

Contributing to Patterns using Common Subtrees

Once you have forked common and changed your subtree for testing, changes from your fork can then be proposed to [https://github.com/validatedpatterns/common.git] and can then be integrated into other patterns. A change to upstream common for a particular upstream pattern would have to be done in two stages:

  1. PR the change into upstream’s common

  2. PR the updated common into the pattern repository

QUICKLINKS

CONTRIBUTE

Pattern AAP Configuration Details

In this section, we describe the details of the AAP configuration we apply as part of installing the pattern. All of the configuration discussed in this section is applied by the ansible_load_controller.sh script.

Loading a Manifest

After validating that AAP is ready to be configured, the first thing the script does is to install the manifest you specify in the values-secret.yaml file in the files.manifest setting. The value of this setting is expected to be a fully-pathed file that represents a Red Hat Satellite manifest file with a valid entitlement for AAP. The only thing this manifest is used for is entitling AAP.

Instructions for creating a suitable manifest file can be found here.

While it is absolutely possible to entitle AAP via a username/password on first login, the automated mechanisms for entitling only support manifests, that is the technique the pattern uses.

Organizations

The pattern installs an Organization called HMI Demo is installed. This makes it a bit easier to separate what the pattern is doing versus the default configuration of AAP. The other resources created in AAP as part of the load process are associated with this Organization.

Credential Types (and their Credentials)

Kubeconfig (Kubeconfig)

The Kubeconfig credential is for holding the OpenShift cluster admin kubeconfig file. This is used to query the edge-gitops-vms namespace for running VM instances. Since the kubeconfig is necessary for installing the pattern and must be available when the load script is running, the load script pulls it into an AAP secret and stores it for later use (and calls it Kubeconfig).

The template for creating the Credential Type was taken from here.

RHSMcredential (rhsm_credential)

This credential is required to register the RHEL VMs and configure them for Kiosk mode. The registration process allows them to install packages from the Red Hat Content Delivery Network.

Machine (kiosk-private-key)

This is a standard AAP Machine type credential. kiosk-private-key is created with the username and private key from your values-secret.yaml file in the kiosk-ssh.username and kiosk-ssh.privatekey fields.

KioskExtraParams (kiosk_container_extra_params)

This CredentialType is considered “secret” because it includes the admin login password for the Ignition application. This passed to the provisioning playbook(s) as extra_vars.

Inventory

The pattern installs an Inventory (HMI Demo), but no inventory sources. This is due to the way that OpenShift Virtualization provides access to virtual machines. The IP address associated with the SSH service that a given VM is running is associated with the Service object on the VM. This is not the way the Kubernetes inventory plugin expects to work. So to make inventory dynamic, we are instead using a play to discover VMs and add them to inventory “on the fly”. What is unusual about DNS inside a Kubernetes cluster is that resources outside the namespace must use the cluster FQDN - which is resource-name.resource-namespace.svc.

It is also possible to define a static inventory - an example of how this would like is preserved in the pattern repository as hosts.

A standard dynamic inventory script is available here. This will retrieve the object names, but it will not (currently) map the FQDN properly. Because of this limitation, we moved to using the inventory pre-play method.

Templates (key playbooks in the pattern)

Dynamic Provision Kiosk Playbook

This combines all three key workflows in this pattern:

  • Dynamic inventory (inventory preplay)
  • Kiosk Mode
  • Podman Playbook

It is safe to run multiple times on the same system. It is run on a schedule, every 10 minutes, to demonstrate this.

Kiosk Mode Playbook

This playbook runs the kiosk_mode role.

Podman Playbook

This playbook runs the container_lifecycle role with overrides suitable for the Ignition application container.

Ping Playbook

This playbook is for testing basic connectivity - making sure that you can reach the nodes you wish to manage, and that the credentials you have given will work on them. It will not change anything on the VMs - just gather facts from them (which requires elevating to root).

Schedules

Update Project AEG GitOps

This job runs every 5 minutes to update the GitOps repository associated with the project. This is necessary when any of the Ansible code (for example, the playbooks or roles associated with the pattern) changes, so that the new code is available to the AAP instance.

Dynamic Provision Kiosk Playbook

This job runs every 10 minutes to provision and configure any kiosks it finds to run the Ignition application in a podman container, and configure firefox in kiosk mode to display that application. The playbook is designed to be idempotent, so it is safe to run multiple times on the same targets; it will not make user-visible changes to those targets unless it must.

This playbook combines the inventory_preplay and the Provision Kiosk Playbook.

Execution Environment

The pattern includes an execution environment definition that can be found here.

The execution environment includes some additional collections beyond what is provided in the Default execution environment, including:

The execution environment definition is provided if you want to customize or change it; if so, you should also change the Execution Environment attributes of the Templates (in the load script, those attributes are set by the variables aap_execution_environment and aap_execution_environment_image).

Roles included in the pattern

kiosk_mode

This role is responsible does the following:

  • RHEL node registration
  • Installation of GUI packages
  • Installation of Firefox
  • Configuration of Firefox kiosk mode

container_lifecycle

This role is responsible for:

  • Downloading and running a podman image on the system (and configure it to auto-update)
  • Setting the container up to run at boot time
  • Passing any other runtime arguments to the container. In this container’s case, that includes specifying an admin password override.

Extra Playbooks in the Pattern

inventory_preplay.yml

This playbook is designed to be included in other plays; its purpose is to discover the desired inventory and add those hosts to inventory at runtime. It uses a kubernetes query via the cluster-admin kube config file.

Provision Kiosk Playbook

This does the work of provisioning the kiosk, which configures kiosk mode, and also installs Ignition and configures it to start at boot. It runs the kiosk_mode and container_lifecycle roles.

Next Steps

Help & Feedback

Report Bugs

QUICKLINKS

Pattern AAP Configuration Details

In this section, we describe the details of the AAP configuration we apply as part of installing the pattern. All of the configuration discussed in this section is applied by the ansible_load_controller.sh script.

Loading a Manifest

After validating that AAP is ready to be configured, the first thing the script does is to install the manifest you specify in the values-secret.yaml file in the files.manifest setting. The value of this setting is expected to be a fully-pathed file that represents a Red Hat Satellite manifest file with a valid entitlement for AAP. The only thing this manifest is used for is entitling AAP.

Instructions for creating a suitable manifest file can be found here.

While it is absolutely possible to entitle AAP via a username/password on first login, the automated mechanisms for entitling only support manifests, that is the technique the pattern uses.

Organizations

The pattern installs an Organization called HMI Demo is installed. This makes it a bit easier to separate what the pattern is doing versus the default configuration of AAP. The other resources created in AAP as part of the load process are associated with this Organization.

Credential Types (and their Credentials)

Kubeconfig (Kubeconfig)

The Kubeconfig credential is for holding the OpenShift cluster admin kubeconfig file. This is used to query the edge-gitops-vms namespace for running VM instances. Since the kubeconfig is necessary for installing the pattern and must be available when the load script is running, the load script pulls it into an AAP secret and stores it for later use (and calls it Kubeconfig).

The template for creating the Credential Type was taken from here.

RHSMcredential (rhsm_credential)

This credential is required to register the RHEL VMs and configure them for Kiosk mode. The registration process allows them to install packages from the Red Hat Content Delivery Network.

Machine (kiosk-private-key)

This is a standard AAP Machine type credential. kiosk-private-key is created with the username and private key from your values-secret.yaml file in the kiosk-ssh.username and kiosk-ssh.privatekey fields.

KioskExtraParams (kiosk_container_extra_params)

This CredentialType is considered “secret” because it includes the admin login password for the Ignition application. This passed to the provisioning playbook(s) as extra_vars.

Inventory

The pattern installs an Inventory (HMI Demo), but no inventory sources. This is due to the way that OpenShift Virtualization provides access to virtual machines. The IP address associated with the SSH service that a given VM is running is associated with the Service object on the VM. This is not the way the Kubernetes inventory plugin expects to work. So to make inventory dynamic, we are instead using a play to discover VMs and add them to inventory “on the fly”. What is unusual about DNS inside a Kubernetes cluster is that resources outside the namespace must use the cluster FQDN - which is resource-name.resource-namespace.svc.

It is also possible to define a static inventory - an example of how this would like is preserved in the pattern repository as hosts.

A standard dynamic inventory script is available here. This will retrieve the object names, but it will not (currently) map the FQDN properly. Because of this limitation, we moved to using the inventory pre-play method.

Templates (key playbooks in the pattern)

Dynamic Provision Kiosk Playbook

This combines all three key workflows in this pattern:

  • Dynamic inventory (inventory preplay)
  • Kiosk Mode
  • Podman Playbook

It is safe to run multiple times on the same system. It is run on a schedule, every 10 minutes, to demonstrate this.

Kiosk Mode Playbook

This playbook runs the kiosk_mode role.

Podman Playbook

This playbook runs the container_lifecycle role with overrides suitable for the Ignition application container.

Ping Playbook

This playbook is for testing basic connectivity - making sure that you can reach the nodes you wish to manage, and that the credentials you have given will work on them. It will not change anything on the VMs - just gather facts from them (which requires elevating to root).

Schedules

Update Project AEG GitOps

This job runs every 5 minutes to update the GitOps repository associated with the project. This is necessary when any of the Ansible code (for example, the playbooks or roles associated with the pattern) changes, so that the new code is available to the AAP instance.

Dynamic Provision Kiosk Playbook

This job runs every 10 minutes to provision and configure any kiosks it finds to run the Ignition application in a podman container, and configure firefox in kiosk mode to display that application. The playbook is designed to be idempotent, so it is safe to run multiple times on the same targets; it will not make user-visible changes to those targets unless it must.

This playbook combines the inventory_preplay and the Provision Kiosk Playbook.

Execution Environment

The pattern includes an execution environment definition that can be found here.

The execution environment includes some additional collections beyond what is provided in the Default execution environment, including:

The execution environment definition is provided if you want to customize or change it; if so, you should also change the Execution Environment attributes of the Templates (in the load script, those attributes are set by the variables aap_execution_environment and aap_execution_environment_image).

Roles included in the pattern

kiosk_mode

This role is responsible does the following:

  • RHEL node registration
  • Installation of GUI packages
  • Installation of Firefox
  • Configuration of Firefox kiosk mode

container_lifecycle

This role is responsible for:

  • Downloading and running a podman image on the system (and configure it to auto-update)
  • Setting the container up to run at boot time
  • Passing any other runtime arguments to the container. In this container’s case, that includes specifying an admin password override.

Extra Playbooks in the Pattern

inventory_preplay.yml

This playbook is designed to be included in other plays; its purpose is to discover the desired inventory and add those hosts to inventory at runtime. It uses a kubernetes query via the cluster-admin kube config file.

Provision Kiosk Playbook

This does the work of provisioning the kiosk, which configures kiosk mode, and also installs Ignition and configures it to start at boot. It runs the kiosk_mode and container_lifecycle roles.

Next Steps

Help & Feedback

Report Bugs

QUICKLINKS

CONTRIBUTE

OpenShift Cluster Sizing for the Ansible Edge GitOps Pattern

Tested Platforms

The Ansible Edge GitOps pattern has been tested on AWS:

Certified Cloud Providers4.94.10
Amazon Web ServicesTested

The pattern is adaptable to running on bare metal/on-prem clusters but has not yet been tested there.

General OpenShift Minimum Requirements

OpenShift 4 has the following minimum requirements for sizing of nodes:

  • Minimum 4 vCPU (additional are strongly recommended).
  • Minimum 16 GB RAM (additional memory is strongly recommended, especially if etcd is colocated on Control Planes).
  • Minimum 40 GB hard disk space for the file system containing /var/.
  • Minimum 1 GB hard disk space for the file system containing /usr/local/bin/.

There is one application that comprises the Ansible Edge GitOps pattern. In addition, the Ansible Edge GitOps pattern also includes the Advanced Cluster Management (ACM) supporting operator that is installed by OpenShift GitOps using ArgoCD.

Ansible Edge GitOps Pattern Components

Here’s an inventory of what gets deployed by the Ansible Edge GitOps pattern on the Datacenter/Hub OpenShift cluster:

NameKindNamespaceDescription
Ansible Edge GitOps-hubApplicationAnsible Edge GitOps-hubHub GitOps management
Red Hat OpenShift GitOpsOperatoropenshift-operatorsOpenShift GitOps
Red Hat Ansible Automation PlatformOperatoransible-automation-platformAnsible Automation
Red Hat OpenShift Data FoundationsOperatoropenshift-storageOpenShift Storage solution
Red Hat OpenShift VirtualizationOperatoropenshift-cnvVirtualization software to run VMs
Edge GitOps VMsVMsedge-gitops-vmsSimulated Edge environment with VMs to manage
Hashicorp VaultOperatorvaultSecrets Storage
External Secrets Operator (ESO)Operatorgolang-external-secretsAbstraction for secrets storage

Ansible Edge GitOps Pattern OpenShift Datacenter HUB Cluster Size

The Ansible Edge GitOps pattern has been tested with a defined set of specifically tested configurations that represent the most common combinations that Red Hat OpenShift Container Platform (OCP) customers are using or deploying for the x86_64 architecture.

The Hub OpenShift Cluster is made up of the the following on the AWS deployment tested:

Node TypeNumber of nodesCloud ProviderInstance Type
Control Plane3Amazon Web Servicesm5.xlarge
Worker3Amazon Web Servicesm5.4xlarge
Worker1Amazon Web Servicesc5n.metal

The metal node is added to the cluster by the installation process after initial provisioning. The pattern on the hub requires OpenShift Data Fabric to support Virtual Machine storage and is a minimum size for a Hub cluster. In the next few sections we take some snapshots of the cluster utilization while the Ansible Edge GitOps pattern is running. Keep in mind that resources will have to be added as more developers are working building their applications.

Datacenter Cluster utilization

Below is a snapshot of the OpenShift cluster utilization while running the Ansible Edge GitOps pattern:

CPUCPU%MemoryMemory%
321m0%12511Mi6%
736m21%7533Mi51%
673m4%9298Mi14%
920m26%8635Mi59%
673m4%9258Mi14%
921m26%9407Mi65%
395m2%5149Mi8%

AWS Instance Types

The Ansible Edge GitOps pattern was tested with the highlighted AWS instances in bold. The OpenShift installer will let you know if the instance type meets the minimum requirements for a cluster.

The message that the openshift installer will give you will be similar to this message

INFO Credentials loaded from default AWS environment variables
 FATAL failed to fetch Metadata: failed to load asset "Install Config": [controlPlane.platform.aws.type: Invalid value: "m4.large": instance type does not meet minimum resource requirements of 4 vCPUs, controlPlane.platform.aws.type: Invalid value: "m4.large": instance type does not meet minimum resource requirements of 16384 MiB Memory]
-

Below you can find a list of the AWS instance types that can be used to deploy the Ansible Edge GitOps pattern.

Instance typeDefault vCPUsMemory (GiB)DatacenterFactory/Edge
3x3 OCP Cluster3 Node OCP Cluster
m4.xlarge416NN
m4.2xlarge832YY
m4.4xlarge1664YY
m4.10xlarge40160YY
m4.16xlarge64256YY
m5.xlarge416YN
m5.2xlarge832YY
m5.4xlarge1664YY
m5.8xlarge32128YY
m5.12xlarge48192YY
m5.16xlarge64256YY
m5.24xlarge96384YY

The OpenShift cluster is made of 3 Control Plane nodes and 4 Workers for the Hub cluster; 3 workers are standard compute nodes and one is c5n.metal. For the node sizes we used the m5.4xlarge on AWS and this instance type met the minimum requirements to deploy the Ansible Edge GitOps pattern successfully on the Hub cluster.

This pattern is currently only usable on AWS because of the integration of OpenShift Virtualization; it would be straightforward to adapt this pattern also to run on bare metal/on-prem clusters. If and when other public cloud providers support metal node provisioning in OpenShift Virtualization, we will document that here.

QUICKLINKS

Below you can find a list of the AWS instance types that can be used to deploy the Ansible Edge GitOps pattern.

Instance typeDefault vCPUsMemory (GiB)DatacenterFactory/Edge
3x3 OCP Cluster3 Node OCP Cluster
m4.xlarge416NN
m4.2xlarge832YY
m4.4xlarge1664YY
m4.10xlarge40160YY
m4.16xlarge64256YY
m5.xlarge416YN
m5.2xlarge832YY
m5.4xlarge1664YY
m5.8xlarge32128YY
m5.12xlarge48192YY
m5.16xlarge64256YY
m5.24xlarge96384YY

The OpenShift cluster is made of 3 Control Plane nodes and 4 Workers for the Hub cluster; 3 workers are standard compute nodes and one is c5n.metal. For the node sizes we used the m5.4xlarge on AWS and this instance type met the minimum requirements to deploy the Ansible Edge GitOps pattern successfully on the Hub cluster.

This pattern is currently only usable on AWS because of the integration of OpenShift Virtualization; it would be straightforward to adapt this pattern also to run on bare metal/on-prem clusters. If and when other public cloud providers support metal node provisioning in OpenShift Virtualization, we will document that here.

QUICKLINKS

CONTRIBUTE

  • Apply the changes to your cluster. This will install the pattern via the Validated Patterns Operator, and then run any necessary follow-up steps.

    ./pattern.sh make install

    • The installation process will take between 45-60 minutes to complete. If you want to know the details of what is happening during that time, the entire process is documented here.

    Installation Validation

    • Check the operators have been installed using the OpenShift console

      OpenShift Console Web UI -> Installed Operators

    The screen should like this when installed via make install:

    ansible-edge-gitops-operators

    • Check all applications are synchronised

    Under the project ansible-edge-gitops-hub click on the URL for the hubgitopsserver. All applications will sync, but this takes time as ODF has to completely install, and OpenShift Virtualization cannot provision VMs until the metal node has been fully provisioned and ready. Additionally, the Dynamic Provision Kiosk Template in AAP must complete; it can only start once the VMs have provisioned and are running:

    ansible-edge-gitops-applications

    • While the metal node is building, the VMs in OpenShift console will show as “Unschedulable.” This is normal and expected, as the VMs themselves cannot run until the metal node completes provisioning and is ready.

    ansible-edge-vms-unschedulable

    • Under Virtualization > Virtual Machines, the virtual machines will eventually show as “Running.” Once they are in “Running” state the Provisioning workflow will run on them, and install Firefox, Kiosk mode, and the Ignition application on them:

    ansible-edge-gitops-vmlist

    • Finally, the VM Consoles will show the Ignition introduction screen. You can choose any of these options; this tutorial assumes you chose “Ignition”:

    ansible-edge-gitops-ignition-options

    • You should be able to login to the application with the userid “admin” and the password you specified as the GATEWAY_ADMIN_PASSWORD in container_extra_params in your values-secret.yaml file.

    ansible-edge-gitops-vmconsole

    Please see Installation Details for more information on the steps of installation.

    Please see Ansible Automation Platform for more information on how this pattern uses the Ansible Automation Platform Operator for OpenShift.

    Please see OpenShift Virtualization for more information on how this pattern uses OpenShift Virtualization.

    Infrastructure Elements of this Pattern

    Ansible Automation Platform

    A fully functional installation of the Ansible Automation Platform operator is installed on your OpenShift cluster to configure and maintain the VMs for this demo. AAP maintains a dynamic inventory of kiosk machines and can configure a VM from template to fully functional kiosk in about 10 minutes.

    OpenShift Virtualization

    OpenShift Virtualization is a Kubernetes-native way to run virtual machine workloads. It is used in this pattern to host VMs simulating an Edge environment; the chart that configures the VMs is designed to be flexible to allow easy customization to model different VM sizes, mixes, versions and profiles for future pattern development.

    Inductive Automation Ignition

    The goal of this pattern is to configure 2 VMs running Firefox in Kiosk mode displaying the demo version of the Ignition application running in a podman container. Ignition is a popular tool in use with Oil and Gas companies; it is included as a real-world example and as an item to spark imagination about what other applications could be installed and managed this way.

    The container used for this pattern is the container image published by Inductive Automation.

    HashiCorp Vault

    Vault is used as the authoritative source for the Kiosk ssh pubkey via the External Secrets Operator. -As part of this pattern HashiCorp Vault has been installed. Refer to the section on Vault.

    Next Steps

    Help & Feedback

    Report Bugs

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    And to use the template to create a VM:

    oc process -n openshift rhel8-desktop-medium | oc apply -f -
 virtualmachine.kubevirt.io/rhel8-q63yuvxpjdvy18l7 created

    In just a few minutes, you will have a blank rhel8 VM running, which you can then login to (via console) and customize.

    1. Get the details of this template as a local YAML file:
    oc get template -n openshift rhel8-desktop-medium -o yaml > my-template.yaml
-

    Once you have this local template, you can view the elements you want to customize, possibly using this as an example.

    HOWTO Define your own Ansible Controller Configuration

    The ansible_load_controller.sh is designed to be relatively easy to customize with a new controller configuration. Structurally, it is principally based on configure_controller.yml from the Red Hat Community of Practice controller_configuration collection. The order and specific list of roles invoked is taken from there.

    To customize it, the main thing would be to replace the different variables in the role tasks with the your own. The script includes the roles for variable types that this pattern does not manage in order to make that part straightforward. Feel free to add your own roles and playbooks (and add them to the controller configuration script).

    The reason this pattern ships with a script as it does instead of invoking the referenced playbook directly is that several of the configuration elements depend on each other, and there was not a super-convenient place to put things like the controller credentials as the playbook suggests.

    HOWTO substitute your own container application (instead of ignition)

    1. Adjust the query in the inventory_preplay.yml either by overriding the vars for the play, or forking the repo and replacing the vars with your own query terms. (That is, use your own label(s) and namespace to discover the services you want to connect to.

    2. Adjust or override the vars in the provision_kiosk.yml playbook to suitable values for your own container application. The roles it calls are fairly generic, so changing the vars is all you should need to do.

    Next Steps

    Help & Feedback

    Report Bugs

    QUICKLINKS

    Once you have this local template, you can view the elements you want to customize, possibly using this as an example.

    HOWTO Define your own Ansible Controller Configuration

    The ansible_load_controller.sh is designed to be relatively easy to customize with a new controller configuration. Structurally, it is principally based on configure_controller.yml from the Red Hat Community of Practice controller_configuration collection. The order and specific list of roles invoked is taken from there.

    To customize it, the main thing would be to replace the different variables in the role tasks with the your own. The script includes the roles for variable types that this pattern does not manage in order to make that part straightforward. Feel free to add your own roles and playbooks (and add them to the controller configuration script).

    The reason this pattern ships with a script as it does instead of invoking the referenced playbook directly is that several of the configuration elements depend on each other, and there was not a super-convenient place to put things like the controller credentials as the playbook suggests.

    HOWTO substitute your own container application (instead of ignition)

    1. Adjust the query in the inventory_preplay.yml either by overriding the vars for the play, or forking the repo and replacing the vars with your own query terms. (That is, use your own label(s) and namespace to discover the services you want to connect to.

    2. Adjust or override the vars in the provision_kiosk.yml playbook to suitable values for your own container application. The roles it calls are fairly generic, so changing the vars is all you should need to do.

    Next Steps

    Help & Feedback

    Report Bugs

    QUICKLINKS

    CONTRIBUTE

    CI status:
    Links:

    Ansible Edge GitOps

    Background

    Organizations are interested in accelerating their deployment speeds and improving delivery quality in their Edge environments, where many devices may not fully or even partially embrace the GitOps philosophy. Further, there are VMs and other devices that can and should be managed with Ansible. This pattern explores some of the possibilities of using an OpenShift-based Ansible Automated Platform deployment and managing Edge devices, based on work done with a partner in the Chemical space.

    This pattern uses OpenShift Virtualization (the productization of Kubevirt) to simulate the Edge environment for VMs.

    Solution elements

    Red Hat Technologies

    Other Technologies this Pattern Uses

    Architecture

    Similar to other patterns, this pattern starts with a central management hub, which hosts the AAP and Vault components.

    Logical architecture

    Ansible-Edge-Gitops-Architecture

    Physical Architecture

    Ansible-Edge-GitOps-Physical-Architecture

    Recorded Demo

    TBD

    Other Presentations Featuring this Pattern

    Registration Required

    Ansible-Automates-June-2022-Deck

    Ansible-Automates-June-2022-Video

    What Next

    QUICKLINKS

    Ansible Edge GitOps

    Background

    Organizations are interested in accelerating their deployment speeds and improving delivery quality in their Edge environments, where many devices may not fully or even partially embrace the GitOps philosophy. Further, there are VMs and other devices that can and should be managed with Ansible. This pattern explores some of the possibilities of using an OpenShift-based Ansible Automated Platform deployment and managing Edge devices, based on work done with a partner in the Chemical space.

    This pattern uses OpenShift Virtualization (the productization of Kubevirt) to simulate the Edge environment for VMs.

    Solution elements

    Red Hat Technologies

    Other Technologies this Pattern Uses

    Architecture

    Similar to other patterns, this pattern starts with a central management hub, which hosts the AAP and Vault components.

    Logical architecture

    Ansible-Edge-Gitops-Architecture

    Physical Architecture

    Ansible-Edge-GitOps-Physical-Architecture

    Recorded Demo

    TBD

    Other Presentations Featuring this Pattern

    Registration Required

    Ansible-Automates-June-2022-Deck

    Ansible-Automates-June-2022-Video

    What Next

    QUICKLINKS

    CONTRIBUTE

    When the metal-worker is showing “READY” and “AVAILABLE”, the virtual machines will begin provisioning on it.

    The metal node will be destroyed when the cluster is destroyed. The script is idempotent and will create at most one metal node per cluster.

    post-install

    Note that all the steps of post-install are idempotent. If you want or need to reconfigure vault or AAP, the recommended way to do so is to call make post-install. This may change as we move elements of this pattern into the new imperative framework in common.

    Specific processes that are called by post-install include:

    vault-init

    Vault requires extra setup in the form of unseal keys and configuration of secrets. The vault-init task does this. Note that it is safe to run vault-init as it will exit successfully if it can connect to a cluster with a running, unsealed vault.

    load-secrets

    This process (which calls push_secrets) calls an Ansible playbook that reads the values-secret.yaml file and stores the data it finds there in vault as keypairs. These values are then usable in the kubernetes cluster. This pattern uses the ssh pubkey for the kiosk VMs via the external secrets operator.

    This script will update secrets in vault if re-run; it is safe to re-run if the secret values have not changed as well.

    configure-controller

    There are two parts to this script - the first part, with the code here, retrieves the admin credentials from OpenShift to enable login to the AAP Controller.

    The second part, which is the bulk of the ansible-load-controller process is here and uses the controller configuration framework to configure the Ansible Automation Platform instance that is installed by the helm chart.

    This division is so that users can adapt this pattern more easily if they’re running AAP, but not on OpenShift.

    The script waits until AAP is ready, and then proceeds to:

    1. Install the manifest to entitle AAP
    2. Configure the custom Credential Types the demo needs
    3. Define an Organization for the Demo
    4. Add a Project for the Demo
    5. Add the Credentials for jobs to use
    6. Configure Host inventory and inventory sources, and smart inventories to define target hosts
    7. Configure an Execution environment for the Demo
    8. Configure Job Templates for the Demo
    9. Configure Schedules for the jobs that need to repeat

    Note: This script has defaults that it overrides when run as part of make install that it derives from the environment (the repo that it is attached to and the branch that it is on). So if you need to re-run it, the most straightforward way to do this is to run make upgrade when using the make-based installation process.

    OpenShift GitOps (ArgoCD)

    OpenShift GitOps is central to this pattern as it is responsible for installing all of the other components. The installation process is driven through the installation of the clustergroup chart. This in turn reads the repo’s global values file, which instructs it to read the hub values file. This is how the pattern knows to apply the Subscriptions and Applications listed further in the pattern.

    ODF (OpenShift Data Foundations)

    ODF is the storage framework that is needed to provide resilient storage for OpenShift Virtualization. It is managed via the helm chart here. This is basically the same chart that our Medical Diagnosis pattern uses (see here for details on the Medical Edge pattern’s use of storage).

    Please note that this chart will create a Noobaa S3 bucket named nb.epoch_timestamp.cluster-domain which will not be destroyed when the cluster is destroyed.

    OpenShift Virtualization (KubeVirt)

    OpenShift Virtualization is a framework for running virtual machines as native Kubernetes resources. While it can run without hardware acceleration, the performance of virtual machines will suffer terribly; some testing on a similar workload indicated a 4-6x delay running without hardware acceleration, so at present this pattern requires hardware acceleration. The pattern provides a script deploy-kubevirt-worker.sh which will provision a metal worker to run virtual machines for the pattern.

    OpenShift Virtualization currently supports only AWS and on-prem clusters; this is because of the way that baremetal resources are provisioned in GCP and Azure. We hope that OpenShift Virtualization can support GCP and Azure soon.

    The installation of the OpenShift Virtualization HyperConverged deployment is controlled by the chart here.

    OpenShift Virtualization was chosen in this pattern to avoid dealing with the differences in galleries and templates of images between the different public cloud providers. The important thing from this pattern’s standpoint is the availability of machine instances to manage (since we are simulating an Edge deployment scenario, which could either be bare metal instances or virtual machines); OpenShift Virtualization was the easiest and most portable way to spin up machine instances. It also provides mechanisms for defining the desired machine set declaratively.

    The creation of virtual machines is controlled by the chart here.

    More details about the way we use OpenShift Virtualization are available here.

    Ansible Automation Platform (AAP, formerly known as Ansible Tower)

    The use of Ansible Automation Platform is really the centerpiece of this pattern. We have recognized for some time that the notion and design principles of GitOps should apply to things outside of Kubernetes, and we believe this pattern gives us a way to do that.

    All of the Ansible interactions are defined in a Git Repository; the Ansible jobs that configure the VMs are designed -to be idempotent (and are scheduled to run every 10 minutes on those VMs).

    The installation of AAP itself is governed by the chart here. The post-installation configuration of AAP is done via the ansible-load-controller.sh script.

    It is very much the intention of this pattern to make it easy to replace the specific Edge management use case with another one. Some ideas on how to do that can be found here.

    Specifics of the Ansible content for this pattern can be seen here.

    More details of the specifics of how AAP is configured are available here.

    Next Steps

    Help & Feedback

    Report Bugs

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    You could also use the “Create VM Wizard” in the OpenShift console.

    Another option - capturing template output and converting it into a Helm Chart

    See details here.

    Components of the virtual-machines template

    Setup - the mechanism for creating identifiers declaratively

    The first part of the template file sets up some variables that we will use later as the template is expanded. We use a sequential numbering scheme for VM name creation because that is an easy way to make each item in the set declarative - it ensures that if you ask for 5 VMs of a particular type, they will have predictable names, and if one is deleted, it will be replaced by a VM with the same name.

    We use explicit “range” variables for the Go templating. This is because the implicit range variable is easily “trampled”, and we have at least two different dimensions to iterate on - vm “role” and “index” within that role.

    The External Secret - SSH pubkey

    The first item we define as part of this structure is an external secret to hold an SSH pubkey. This pubkey will be mounted in the VM under an unprivileged user’s home directory - and generally that unprivileged user is expected to be able to sudo root without password. By default, RHEL images are configured to only allow SSH access via pubkey. In this pattern, the private key and public key for the SSH connections are loaded into both Vault (which we inherited from previous patterns) and Ansible Automation Platform.

    Since the keys are defined per VM “group”, it is possible and expected that you could have different keypairs for different groups of VMs. Nothing would prevent you from using the same keypair for all machines if you have different groups, though.

    While the pubkey is not truly a “secret”, the availability of the External Secrets Operator made for a nice opportunity to allow for variance in configuration without necessarily requiring local customization of the pattern. The OpenShift Virtualization model has no way of knowing that multiple servers may have the same SSH credentials, and in fact cannot depend on this. So it creates a pubkey object by default for each VM, and we imitate this behavior in the pattern.

    The VirtualMachine definition

    The VirtualMachine definition is the biggest part of the template. All of it is derived from customization of the default templates that OpenShift Virtualization installs in the openshift namespace - especially most of the labels and annotations, with the following exceptions:

    labels

    • app

    This is set to $identifier to match a general pattern with other applications.

    • edge-gitops-role

    This is set explicitly and used elsewhere in this pattern to help identify resources by role. The intention is to be able to use the edge-gitops-role as a selector for targeting various kind of queries, including (especially) Ansible inventories. Though please note - because of the way Kubernetes (and OpenShift) work, when you connect to a VM with Ansible you are connecting to the Service object directly, not to the VM. (Another way to look at it is that the Service object is providing network abstraction over the VM object.)

    Other resources in the rest of the VirtualMachine definition are copied from the default template, with appropriate Helm variables included.

    Initial user access

    Note that the initial user (default: cloud-user) and initial password are customizable via values overrides. The kiosk type shows an example of how to either use a user/password specific to the type or a default for the chart using the coalesce function.

    The Service definition

    The Service definition is potentially complex. The purpose of this Service object is to expose all of the needed TCP and UDP network ports within the cluster. (Providing access to them from outside the cluster would require Route or Ingress objects, and would have some significant security implications; access to these entities from outside the cluster is not the focus of this pattern, so we do not provide it at this time.)

    A given VM may expose one port (for Ansible access, you need at least TCP/22), or it may expose many ports. You are free to define a service per port if you like, but it seems more convenient to define them all as a single service.

    One aspect of the templating you may find interesting is the use of the toPrettyJson filter in Go. Since YAML is a proper superset of JSON, this is a neat trick that allows to include a nested data structure without having to worry about how to indent it. (As toPrettyJson uses the square bracket ([]) and curly bracket ({}) notation for arrays and hashes, YAML can interpret it without worrying about its indentation.

    Accessing the VMs

    There are three mechanisms for access to these VMs:

    Ansible - keypair authentication

    The ssh keypairs from your values-secret.yaml are loaded into both Vault and AAP for use later. The pattern currently -defines one such keypair, kiosk-ssh, but could support more, such as iot-ssh, gateway-ssh, etc. more details on how to expand on this pattern are described below.

    AAP only needs the private key and the username as a machine credential. The public key is not truly a secret, but it seemed interesting and useful to use the external secret operator to associate the public key with VM instances this way and prevent having to diverge from the upstream pattern to include local ssh pubkey specifications.

    Note that the default SSH setting for RHEL does not allow password-based logins via SSH, and it’s at the very least inconvenient to copy the SSH private key into a VM inside the cluster, so the typical way the keypair will be used is through Ansible.

    Virtual Machine Console Access via OpenShift Console

    Navigate to Virtualization -> VirtualMachines and make sure Project: All Projects or edge-gitops-vms is selected:

    show-vms

    Click on the “three dots” menu on the right, which will open a dialog like the following:

    show-vm-open-console

    Note: In OpenShift Virtualization 4.11, the “Open Console” option appears when you click on the virtual machine name in openshift console. The dialog looks like this:

    kubevirt411-vm-open-console

    The virtual machine console view will either show a standard RHEL console login screen, or if the demo is working as designed, it will show the Ignition application running in kiosk mode. If the console shows a standard RHEL login, it can be accessed using the the initial user name (cloud-user by default) and password (which is what is specified in the Helm chart Values as either the password specific to that machine group, the default cloudInit, or a hardcoded default which can be seen in the template here. On a VM created through the wizard or via oc process from a template, the password will be set on the VirtualMachine object in the volumes section.

    Initial User login (cloud-user)

    In general, and before the VMs have been configured by the Ansible Jobs, you can log in to the VMs on the console using the user and password you specified in the Helm chart, or else you can look at the VirtualMachine object and see what the username and password setting are. The pattern, by design, replaces the typical console view with Firefox running in kiosk mode. But this mechanism can still be used if you change the console from “VNC Console” to “Serial Console”.

    The “extra” VM Template

    Also included in the edge-gitops-vms chart is a separate template that will allow the creation of VMs with similar (though not identical characteristics) to the ones defined in the chart.

    The rhel8-kiosk-with-svc template is preserved as an intermediate step to creating your own VM types, to see how the pipeline from default VM template -> customized template -> Helm-variable chart can work.

    Next Steps

    Help & Feedback

    Report Bugs

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    Ansible GitOps Framework

    Background

    Ansible GitOps Framework (AGOF) is an alternative to the OpenShift-based Validated Patterns framework, designed to provide a framework for GitOps without Kubernetes. AGOF is not a pattern itself; it is a framework for installing Ansible Automation Platform (AAP), and then using that as the GitOps engine to drive other pattern work. AGOF comes with code to install VMs in AWS, if desired, or else it can work with previously provisioned VMs, or a functional AAP Controller endpoint.

    The Pattern is then expressed as an Infrastructure as Code repository, which will be loaded into AAP.

    Solution elements

    Red Hat Technologies

    QUICKLINKS

    CONTRIBUTE

    Cockroach

    Validation status:
    Sandbox -Sandbox

    Cockroach

    A multicloud pattern using cockroachdb and submariner, deployed via RHACM.

    Repo

    QUICKLINKS

    Cockroach

    A multicloud pattern using cockroachdb and submariner, deployed via RHACM.

    Repo

    QUICKLINKS

    CONTRIBUTE

    Connected Vehicle Architecture

    Validation status:
    Sandbox -Sandbox

    Connected Vehicle Architecture

    A distributed cloud-native application that implements key aspects of a modern IoT architecture.

    Repo

    QUICKLINKS

    Connected Vehicle Architecture

    A distributed cloud-native application that implements key aspects of a modern IoT architecture.

    Repo

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    Below you can find a list of the AWS instance types that can be used to deploy the Multicluster DevSecOps pattern.

    Instance typeDefault vCPUsMemory (GiB)DatacenterFactory/Edge
    3x3 OCP Cluster3 Node OCP Cluster
    m4.xlarge416NN
    m4.2xlarge832YY
    m4.4xlarge1664YY
    m4.10xlarge40160YY
    m4.16xlarge64256YY
    m5.xlarge416YN
    m5.2xlarge832YY
    m5.4xlarge1664YY
    m5.8xlarge32128YY
    m5.12xlarge48192YY
    m5.16xlarge64256YY
    m5.24xlarge96384YY

    The OpenShift cluster is made of 4 Control Plane nodes and 3 Workers for the Datacenter and the Edge/managed data center cluster are made of 3 Control Plane and 3 Worker nodes. For the node sizes we used the m5.xlarge on AWS and this instance type met the minimum requirements to deploy the Multicluster DevSecOps pattern successfully on the Datacenter hub. On the managed data center cluster we used the m5.xlarge since the minimum cluster was comprised of 3 nodes. .

    To understand better what types of nodes you can use on other Cloud Providers we provide some of the details below.

    Azure Instance Types

    The Multicluster DevSecOps pattern was also deployed on Azure using the Standard_D8s_v3 VM size. Below is a table of different VM sizes available for Azure. Keep in mind that due to limited access to Azure we only used the Standard_D8s_v3 VM size.

    The OpenShift cluster is made of 3 Control Plane nodes and 3 Workers for the Datacenter cluster.

    The OpenShift cluster is made of 3 Control Plane nodes and 3 or more workers for each of the managed data center clusters.

    TypeSizesDescription
    General purposeB, Dsv3, Dv3, Dasv4, Dav4, DSv2, Dv2, Av2, DC, DCv2, Dv4, Dsv4, Ddv4, Ddsv4Balanced CPU-to-memory ratio. Ideal for testing and development, small to medium databases, and low to medium traffic web servers.
    Compute optimizedF, Fs, Fsv2, FXHigh CPU-to-memory ratio. Good for medium traffic web servers, network appliances, batch processes, and application servers.
    Memory optimizedEsv3, Ev3, Easv4, Eav4, Ev4, Esv4, Edv4, Edsv4, Mv2, M, DSv2, Dv2High memory-to-CPU ratio. Great for relational database servers, medium to large caches, and in-memory analytics.
    Storage optimizedLsv2High disk throughput and IO ideal for Big Data, SQL, NoSQL databases, data warehousing and large transactional databases.
    GPUNC, NCv2, NCv3, NCasT4_v3, ND, NDv2, NV, NVv3, NVv4Specialized virtual machines targeted for heavy graphic rendering and video editing, as well as model training and inferencing (ND) with deep learning. Available with single or multiple GPUs.
    High performance computeHB, HBv2, HBv3, HC, HOur fastest and most powerful CPU virtual machines with optional high-throughput network interfaces (RDMA).

    For more information please refer to the Azure VM Size Page.

    Google Cloud (GCP) Instance Types

    The Multicluster DevSecOps pattern was also deployed on GCP using the n1-standard-8 VM size. Below is a table of different VM sizes available for GCP. Keep in mind that due to limited access to GCP we only used the n1-standard-8 VM size.

    The OpenShift cluster is made of 3 Control Plane and 3 Workers for the Datacenter cluster.

    The OpenShift cluster is made of 3 Nodes combining Control Plane/Workers for the Edge/managed data center cluster.

    The following table provides VM recommendations for different workloads.

    | General purpose | Workload optimized

    Cost-optimizedBalancedScale-out optimizedMemory-optimizedCompute-optimizedAccelerator-optimized
    E2N2, N2D, N1T2DM2, M1C2A2
    Day-to-day computing at a lower costBalanced price/performance across a wide range of VM shapesBest performance/cost for scale-out workloadsUltra high-memory workloadsUltra high performance for compute-intensive workloadsOptimized for high performance computing workloads

    For more information please refer to the GCP VM Size Page.

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    Multicluster DevSecOps

    Background

    With this Pattern, we demonstrate a horizontal solution for multicluster DevSecOps use cases.

    It is derived from the multi-cloud GitOps pattern with added products to provide a complete DevSecOps workflow. This includes CI/CD pipelines with security gates; image scanning, signing and storage in a secure registry; deployment to secured clusters that provide advanced security monitoring and alerting.

    Solution elements

    • How to use a GitOps approach to keep in control of configuration and operations
    • How to centrally manage multiple clusters, including workloads
    • How to build and deploy workloads across clusters using modern CI/CD
    • How to deploy different security products into the pattern

    Red Hat Products

    • Red Hat OpenShift Container Platform (Kubernetes)
    • Red Hat Advanced Cluster Management (Open Cluster Management)
    • Red Hat OpenShift GitOps (ArgoCD)
    • Red Hat OpenShift Pipelines (Tekton)
    • Red Hat Quay (container image registry with security features enabled)
    • Red Hat Open Data Foundation (highly available storage)
    • Red Hat Advanced Cluster Security (scanning and monitoring)

    Other technologies and products

    • Hashicorp Vault community edition (secrets management)

    Context on Multicluster DevSecOps

    Effective cloud native DevSecOps is about securing both the platform and the applications deployed to the platform. Securing the applications deployed is also about securing the supply chain. Not all applications are built in-house. Confidence in external applications and technologies is critical. OpenShift Platform Plus enables DevSecOps for both platform and supply chain.

    OpenShift Platform Plus includes OpenShift Container Platform, Advanced Cluster Management, Advanced Cluster Security, OpenShift Data Foundation and Red Hat Quay. The capabilities delivered across these components combine to provide policy-based cluster lifecycle management and policy based risk and security management across your fleet of clusters. You can see the flow at the bottom of this graphic.

    Multi-cluster DevSecOps

    The Hub cluster is where lifecycle, deployment, security, compliance and risk management policies are defined and is the central management point across clusters. DevSecOps for the platform includes pulling images from Red Hat’s registry, pulling day two configuration code from Git via our integration with ArgoCD, and ensuring that all optional operators are deployed and configured.

    Policy based deployment also specifies which admission controllers should be deployed to which clusters, including the ACS admission controller. The Hub also provides a unified view of health, security, risk and compliance across your fleet. We have many of these capabilities in place today, however, they each have their own UI. Over the next few releases, we will be working to provide an integrated multi-cluster user experience for admin, security and developer persona in the OpenShift Console.

    Demo Scenario

    The Multicluster DevSecOps Pattern / Demo Scenario reflects this by having 3 layers:

    • Managed/Secured edge - the edge or production a more controlled environment.
    • Devel - where AppDev, Testing etc. is happening
    • Central Data Center / Hub - the cloud/core, (and ERP systems of course, not part of the demo).

    There are ways of combing these three clusters into a two cluster (hub/devel and secured/edge) and single cluster (all in one). The documentation provides instructions (TBD Link).

    Pattern Logical Architecture

    The following diagram explains how different roles have different concerns and focus when working with this distributed AL/ML architecture.

    Multicluster DevSecOps Architecture

    In the Multi-Cluster DevSecOps architecture there are three logical types of sites.

    • The Hub. This is where the cloud native infrastructure is monitored and managed. It performs cluster management, advanced cluster security and a secure image registry.
    • Devel. This is where the development pipeline is hosted. Developers submit code builds to the pipeline and various security tools are used in the pipeline to mitigate the risk of harmful applications or code being deployed in production.
    • Secured Production. This is where applications are securely deployed and monitored.

    Pattern Architecture Schema

    The following diagram shows various management functions, including products and components, that are deployed on central hub and managed clusters (Devel. and Prod.) in order to maintain secure clusters. Consider this the GitOps schema.

    Multicluster DevSecOps Architecture - management schema

    The following diagram shows various development pipeline functions, including products and components, that are deployed on the central hub and development (Devel) clusters in order to provide security features and services for development pipelines. Consider this the DevSecOps schema.

    Multicluster DevSecOps Architecture - development schema

    QUICKLINKS

    CONTRIBUTE

    Remember to commit the changes and push to GitHub so that GitOps can see -your changes and apply them.

    Deploy a Production (prod) cluster

    For instructions on how to prepare and import a production (prod) cluster please read the section importing a cluster. Use clusterGroup=prod.

    You are done importing the production cluster

    That’s it! Go to your production OpenShift console and check for the open-cluster-management-agent pod being launched. Be patient, it will take a while for the ACM agent and agent-addons to launch. After that, the operator OpenShift GitOps will run. When it’s finished coming up launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    GitOps Dashboard prod

    Next up

    Work your way through the Multicluster DevSecOps GitOps/DevOps demos (TBD)

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Viewing the Sepsis Detection dashboard

    TO-DO: Describe how to examine the various parts of the Sepsis application

    Next Steps

    Help & Feedback -Report Bugs

    Using the Emerging Disease Detection pattern

    The following steps describes how you can use this pattern in a demo.

    Viewing the Sepsis Detection dashboard

    TO-DO: Describe how to examine the various parts of the Sepsis application

    QUICKLINKS

    Using the Emerging Disease Detection pattern

    The following steps describes how you can use this pattern in a demo.

    Viewing the Sepsis Detection dashboard

    TO-DO: Describe how to examine the various parts of the Sepsis application

    QUICKLINKS

    CONTRIBUTE

    About the Emerging Disease Detection pattern

    Use case

    • Use a GitOps approach to manage a hybrid cloud deployment for an AI emerging disease detection system.

    • Use a AI automation at the edge to detect emerging diseases.

    • Use an event driven architecture

    • Enable application lifecycle management.

    • Securely manage secrets across the deployment.

    Background

    No technology is better poised to transform Healthcare as AI and Business Process Automation. Coupled with an Edge Architecture, these continuous monitoring and detection systems can scale to provide early warning intervention and measurable process improvements, anywhere.

    Detection of disease states like sepsis, stroke, pulmonary embolism, and heart attack requires low-latency, broadband, asynchronous streaming capabilities. We have prototyped an early warning platform built with a distributed edge architecture, fed by at-home post-operative monitoring (fitbit, smart phone, wifi devices) and automated Clinical care escalation and coordination processes. This platform has the potential to significantly lower network traffic and cost while providing early warning interventions for our nation’s Veterans.

    About the solution

    To demonstrate the effectiveness of the solution this pattern focuses on the specific problem of Sepsis. Sepsis is the body’s extreme response to an infection. It is a life-threatening medical emergency. Sepsis is a costly and life threatening condition that can result in multi-organ failure. Beating conditions like sepsis requires rapid detection and mitigation of risks. With the immune system compromised, recovery at home is often preferred to minimize the risk for cross-infections, yet medical teams often lack the capability to perform constant surveillance for emerging risks across their patient cohorts, especially in rural settings. In this session, we will demonstrate an early warning system driven by Clinical AI at the Edge, fed by at-home post-operative monitoring and automated Clinical care escalation and coordination processes.

    Technology Highlights:

    • Event-Driven Architecture

    • Red Hat OpenShift Data Science

    • Process Automation

    Solution Discussion and Demonstration

    In this demonstration, we will follow a vulnerable, post-operative patient, Alani, as she recovers from surgery in her home setting. The pattern demonstrates an early warning system driven by Clinical AI at the Edge, fed by at-home post-operative monitoring and automated Clinical care escalation and coordination processes.

    This architecture pattern demonstrates three strengths:

    • Enabling healthcare at the edge

      • Taking healthcare past the edge of the cloud to support care processes in Low Bandwidth environments.

    • AI integrated into healthcare team workflows

      • BPM+ Health workflows support plug & play AI modules embedded into clinical workflows as Clinical Decision Support.

    • Event driven, Intelligent automation

      • Clinical best practices & processes automated using BPM+ Health authored workflows using FHIR API endpoints.

    For a thorough explanation of this solution in the context of Sepsis detection please consider reviewing the following 25 minute video.

    Overview of the solution in Sepsis Detection

    Overview of the Architecture

    As data arrives from Alani’s various monitoring devices her latest metrics are collected in the FHIR database. Debezium is an open source distributed platform for change data capture. It will create an observation bundle for streaming to the AI model. This, in turn, will create a risk assessment and provide that to the process automation for review with Alani’s doctor or the on-call doctor that is available. Their assessment may trigger further process workflows.

    edd rh hl architecture
    Figure 1. Overview of the solution reference architecture

    In the following figure, logically, this solution can be viewed as being composed of an automation component, unified management including secrets management, and the clusters under management, all running on top of a user-chosen mixture of on-premise data centers and public clouds.

    edd data architecture
    Figure 2. Sepsis Data Architecture
    edd logical architecture legend
    Figure 3. Logical Architecture
    edd schema dataflow
    Figure 4. Data Flow Architecture

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Red Hat AMQ Streams

    Red Hat AMQ streams is a massively scalable, distributed, and high-performance data streaming platform based on the Apache Kafka project. It offers a distributed backbone that allows microservices and other applications to share data with high throughput and low latency. Red Hat AMQ Streams is available in the Red Hat AMQ product.

    Red Hat Single Sign-On

    Based on the Keycloak project, Red Hat Single Sign-On enhances security by enabling you to secure your web applications with Web single sign-on (SSO) capabilities based on popular standards such as SAML 2.0, OpenID Connect and OAuth 2.0.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Next steps

    QUICKLINKS

    About the Emerging Disease Detection pattern

    Use case

    • Use a GitOps approach to manage a hybrid cloud deployment for an AI emerging disease detection system.

    • Use a AI automation at the edge to detect emerging diseases.

    • Use an event driven architecture

    • Enable application lifecycle management.

    • Securely manage secrets across the deployment.

    Background

    No technology is better poised to transform Healthcare as AI and Business Process Automation. Coupled with an Edge Architecture, these continuous monitoring and detection systems can scale to provide early warning intervention and measurable process improvements, anywhere.

    Detection of disease states like sepsis, stroke, pulmonary embolism, and heart attack requires low-latency, broadband, asynchronous streaming capabilities. We have prototyped an early warning platform built with a distributed edge architecture, fed by at-home post-operative monitoring (fitbit, smart phone, wifi devices) and automated Clinical care escalation and coordination processes. This platform has the potential to significantly lower network traffic and cost while providing early warning interventions for our nation’s Veterans.

    About the solution

    To demonstrate the effectiveness of the solution this pattern focuses on the specific problem of Sepsis. Sepsis is the body’s extreme response to an infection. It is a life-threatening medical emergency. Sepsis is a costly and life threatening condition that can result in multi-organ failure. Beating conditions like sepsis requires rapid detection and mitigation of risks. With the immune system compromised, recovery at home is often preferred to minimize the risk for cross-infections, yet medical teams often lack the capability to perform constant surveillance for emerging risks across their patient cohorts, especially in rural settings. In this session, we will demonstrate an early warning system driven by Clinical AI at the Edge, fed by at-home post-operative monitoring and automated Clinical care escalation and coordination processes.

    Technology Highlights:

    • Event-Driven Architecture

    • Red Hat OpenShift Data Science

    • Process Automation

    Solution Discussion and Demonstration

    In this demonstration, we will follow a vulnerable, post-operative patient, Alani, as she recovers from surgery in her home setting. The pattern demonstrates an early warning system driven by Clinical AI at the Edge, fed by at-home post-operative monitoring and automated Clinical care escalation and coordination processes.

    This architecture pattern demonstrates three strengths:

    • Enabling healthcare at the edge

      • Taking healthcare past the edge of the cloud to support care processes in Low Bandwidth environments.

    • AI integrated into healthcare team workflows

      • BPM+ Health workflows support plug & play AI modules embedded into clinical workflows as Clinical Decision Support.

    • Event driven, Intelligent automation

      • Clinical best practices & processes automated using BPM+ Health authored workflows using FHIR API endpoints.

    For a thorough explanation of this solution in the context of Sepsis detection please consider reviewing the following 25 minute video.

    Overview of the solution in Sepsis Detection

    Overview of the Architecture

    As data arrives from Alani’s various monitoring devices her latest metrics are collected in the FHIR database. Debezium is an open source distributed platform for change data capture. It will create an observation bundle for streaming to the AI model. This, in turn, will create a risk assessment and provide that to the process automation for review with Alani’s doctor or the on-call doctor that is available. Their assessment may trigger further process workflows.

    edd rh hl architecture
    Figure 1. Overview of the solution reference architecture

    In the following figure, logically, this solution can be viewed as being composed of an automation component, unified management including secrets management, and the clusters under management, all running on top of a user-chosen mixture of on-premise data centers and public clouds.

    edd data architecture
    Figure 2. Sepsis Data Architecture
    edd logical architecture legend
    Figure 3. Logical Architecture
    edd schema dataflow
    Figure 4. Data Flow Architecture

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Red Hat AMQ Streams

    Red Hat AMQ streams is a massively scalable, distributed, and high-performance data streaming platform based on the Apache Kafka project. It offers a distributed backbone that allows microservices and other applications to share data with high throughput and low latency. Red Hat AMQ Streams is available in the Red Hat AMQ product.

    Red Hat Single Sign-On

    Based on the Keycloak project, Red Hat Single Sign-On enhances security by enabling you to secure your web applications with Web single sign-on (SSO) capabilities based on popular standards such as SAML 2.0, OpenID Connect and OAuth 2.0.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Next steps

    QUICKLINKS

    CONTRIBUTE

  • Verify that MachineConfigPool is ready. To check the status run following command:

    oc get mcp

  • Verify that all applications are healthy and synchronized. Under the project gaudi-rag-chat-qna click the URL for the hub gitops server.

    Gaudi RAG Chat QnA GitOps Hub

    • Setup Red Hat OpenShift AI

    After all components are properly deployed user can proceed to setup Red Hat OpenShift AI. The procedures consists of two manual steps:

    1. Uploading AI model to S3 bucket (RGW storage) on the cluster

    2. Deploying TGI

    Upload AI model

    1. First step is to go to RHOAI dashboard/Data Science Projects tab and select gaudi-llm project (If the gaudi-llm project is missing user should check if app is ready in ArgoCD dashboard):

      Select RHOAI project

    2. Go to Workbenches tab and click Create workbench:

      Create RHOAI workbench

    3. Fill the input form following images below:

      Deploy Jupyter notebook 1
      Deploy Jupyter notebook 2

    4. After workbench is created go to this Jupyter notebook dashboard and upload Jupyter notebook file /download-model.ipynb to the file explorer so it looks like this:

      Jupyter notebook view

    5. When download-model is uploaded, run notebook’s all commands. After notebook is executed, model should be uploaded to S3 bucket. To check if model is present please run following commands:

      export AWS_ACCESS_KEY_ID=$(oc -n openshift-storage get secret s3-secret-bck -ojsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d)
 export AWS_SECRET_ACCESS_KEY=$(oc -n openshift-storage get secret s3-secret-bck -ojsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d)
 export CEPH_RGW_ENDPOINT=http://$(oc -n openshift-storage get route s3-rgw -ojsonpath='{.spec.host}')
-aws --endpoint ${CEPH_RGW_ENDPOINT} s3 ls model-bucket/models/

      Response should show that model directory is present.

    Deploy TGI

    1. First step is to go to RHOAI dashboard/Data Science Projects tab and select gaudi-llm project:

      Select RHOAI project

    2. Now select Data connections tab and click Add data connection:

      Add Data connection

    3. There should appear form Add data connection with couple of inputs. Form should be looking similar to this:

      Complete Data connection details

      1. To get value for Access key run command:

        oc -n openshift-storage get secret s3-secret-bck -ojsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d

      2. To get value for Secret key run command:

        oc -n openshift-storage get secret s3-secret-bck -ojsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d

      3. To get value for Endpoint run command:

        echo "http://$(oc -n openshift-storage get route s3-rgw -ojsonpath='{.spec.host}')"

    4. Next step is to go to tab Models and click Deploy model in Single-model serving platform section:

      Create Single Model Serving

    5. There should appear form Deploy model. Fill all the inputs like in following images and then click Deploy button:

      Deploy model 1
      Deploy model 2

    6. If everything is setup correctly go to tab Models again to check status of TGI. It should be looking like this:

      Status of TGI

    RAG Chat demo

    After whole setup is complete demo application is ready to use. Chat QnA UI address can be obtained by running following command:

    echo "http://$(oc -n gaudi-llm get route chatqna-gaudi-ui-server -ojsonpath='{.spec.host}')"

    QUICKLINKS

    Response should show that model directory is present.

    Deploy TGI

    1. First step is to go to RHOAI dashboard/Data Science Projects tab and select gaudi-llm project:

      Select RHOAI project

    2. Now select Data connections tab and click Add data connection:

      Add Data connection

    3. There should appear form Add data connection with couple of inputs. Form should be looking similar to this:

      Complete Data connection details

      1. To get value for Access key run command:

        oc -n openshift-storage get secret s3-secret-bck -ojsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d

      2. To get value for Secret key run command:

        oc -n openshift-storage get secret s3-secret-bck -ojsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d

      3. To get value for Endpoint run command:

        echo "http://$(oc -n openshift-storage get route s3-rgw -ojsonpath='{.spec.host}')"

    4. Next step is to go to tab Models and click Deploy model in Single-model serving platform section:

      Create Single Model Serving

    5. There should appear form Deploy model. Fill all the inputs like in following images and then click Deploy button:

      Deploy model 1
      Deploy model 2

    6. If everything is setup correctly go to tab Models again to check status of TGI. It should be looking like this:

      Status of TGI

    RAG Chat demo

    After whole setup is complete demo application is ready to use. Chat QnA UI address can be obtained by running following command:

    echo "http://$(oc -n gaudi-llm get route chatqna-gaudi-ui-server -ojsonpath='{.spec.host}')"

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    CONTRIBUTE

    HyperShift

    Validation status:
    Sandbox Sandbox
    CI status:
    Links:

    About the HyperShift pattern (hosted control plane)

    Background

    This pattern simplifies the deployment of an hosted control plane or hosted control plane cluster. Use this pattern to create hosted control plane clusters.

    Workflow

    • Install multicluster engine for Kubernetes Operator

    • Create an instance of the MultiClusterEngine to enable hypershift, which is a technology preview feature.

    • Install the AWS Controllers for Kubernetes - Amazon S3 Operator

    • Create an S3 bucket that hosted control plane will use for OpenID Connect (OIDC)

    • Create a buildconfig and imagestream that provide the HyperShift cli (hypershift) as an imagestream to be used in further automation if desired.

    hypershift high level architecture
    Figure 1. source: https://hypershift-docs.netlify.app/

    If you have any questions or concerns contact Jonny Rickard.

    About the solution elements

    The solution enables the rapid provisioning of hosted control plane.

    The HyperShift pattern uses the following products and technologies:

    • Red Hat OpenShift Container Platform for container orchestration

    • Red Hat OpenShift GitOps, a GitOps continuous delivery (CD) solution

    • The multicluster engine for Kubernetes Operator, the multicluster-engine provider

    • AWS Controllers for Kubernetes - Amazon S3 Operator, an S3 storage controller

    QUICKLINKS

    About the HyperShift pattern (hosted control plane)

    Background

    This pattern simplifies the deployment of an hosted control plane or hosted control plane cluster. Use this pattern to create hosted control plane clusters.

    Workflow

    • Install multicluster engine for Kubernetes Operator

    • Create an instance of the MultiClusterEngine to enable hypershift, which is a technology preview feature.

    • Install the AWS Controllers for Kubernetes - Amazon S3 Operator

    • Create an S3 bucket that hosted control plane will use for OpenID Connect (OIDC)

    • Create a buildconfig and imagestream that provide the HyperShift cli (hypershift) as an imagestream to be used in further automation if desired.

    hypershift high level architecture
    Figure 1. source: https://hypershift-docs.netlify.app/

    If you have any questions or concerns contact Jonny Rickard.

    About the solution elements

    The solution enables the rapid provisioning of hosted control plane.

    The HyperShift pattern uses the following products and technologies:

    • Red Hat OpenShift Container Platform for container orchestration

    • Red Hat OpenShift GitOps, a GitOps continuous delivery (CD) solution

    • The multicluster engine for Kubernetes Operator, the multicluster-engine provider

    • AWS Controllers for Kubernetes - Amazon S3 Operator, an S3 storage controller

    QUICKLINKS

    CONTRIBUTE

    Now its time to kick off the CI pipeline. Due to the need for GitHub secrets and Quay secrets as part of this process, we currently can’t use the OpenShift console’s Pipelines to kick off the pipeline in the demo environment. Instead, use the command-line. While in the industrial-edge repository directory, run the following:

    make build-and-test
-

    This build takes some time because the pipeline is rebuilding all the images. You can monitor the pipeline’s progress in the Openshift console’s pipelines section.

    Alternatively you can can try and run the shorter build-iot-consumer pipeline run in the OpenShift console. This should just run and test the specific application.

    build-and-test-pipeline

    You can also see some updates happening in the manuela-tst application in OpenShift GitOps (ArgoCD).

    When the pipeline is complete check the lines-dashboard application again in the browser. More reasonable, Celsius, temperatures are displayed. (Compare with above.)

    celsius-temp

    The steps above have successfully applied the change to the Manuela test environment at the data center. In order for these changes to be pushed out to the factories it must be accepted and pushed to the Git repository. Examine the project in GitHub. There is a new Pull Request (PR) called Pull request created by Tekton task github-add-pull-request. Select that PR and merge the pull request.

    tekton-pull-request

    OpenShift GitOps will see the new change and apply it out to the factories.

    Application AI model changes with DevOps

    After a successful deployment of Industrial Edge 2.0, check to see that Jupyter Hub is running. To do this go to project manuela-ml-workspace check that jupyterhub pods are up and running.

    jupyerhub-pods

    Then, in the same project manuela-ml-namespace, select Networking/Routes and click on the URL associated with jupyterhub in the Location column.

    jupyterhub-url

    This will bring you to a web page at an address in the following format:

    • jupyterhub-manuela-ml-workspace.apps.*clustername*.*your-domain*

    Options for different types of Jupyter servers are shown. There are two options that are useful for this demo.

    • Standard Data Science. Select this notebook image for simpler notebooks like Data Analyses.ipynb
    • Tensorflow Notebook Image. Select this notebook image for more a complex notebook that require Tensorflow. E.g. Anomaly Detection-using-TF-and-Deep-Learning.ipynb

    At the bottom of the screen there is a Start server button. Select the type of Notebook server image and press Start server.

    jupyterhub-init-console

    Selecting Tensorflow notebook image:

    jupyter-tf-server

    On the next screen upload the following files from manuela-dev/ml-models/anomaly-detection:

    • One of the Jupyter notebooks
      • Data-Analyses.ipynb for a somewhat simpler demo
      • Anomaly Detection-using-TF-and-Deep-Learning.ipynb for a Tensorflow demo.
    • raw-data.cvs

    upload-ml-files

    Open the notebook by double clicking on the notebook file (ending in .ipynb)

    anomaly-detection-notebook

    After opening the notebook successfully, walk through the demonstration by pressing play and iterating through the commands in the playbook. Jupyter playbooks are interactive and you may make changes and also save those changes. Also, some steps in the notebook take milliseconds, however, other steps can take a long time (up to an hour), so check on the completion of steps.

    Remember that changes to the notebook will require downloading, committing, and pushing that notebook to the git repository so that it gets redeployed to the factories.

    QUICKLINKS

    This build takes some time because the pipeline is rebuilding all the images. You can monitor the pipeline’s progress in the Openshift console’s pipelines section.

    Alternatively you can can try and run the shorter build-iot-consumer pipeline run in the OpenShift console. This should just run and test the specific application.

    build-and-test-pipeline

    You can also see some updates happening in the manuela-tst application in OpenShift GitOps (ArgoCD).

    When the pipeline is complete check the lines-dashboard application again in the browser. More reasonable, Celsius, temperatures are displayed. (Compare with above.)

    celsius-temp

    The steps above have successfully applied the change to the Manuela test environment at the data center. In order for these changes to be pushed out to the factories it must be accepted and pushed to the Git repository. Examine the project in GitHub. There is a new Pull Request (PR) called Pull request created by Tekton task github-add-pull-request. Select that PR and merge the pull request.

    tekton-pull-request

    OpenShift GitOps will see the new change and apply it out to the factories.

    Application AI model changes with DevOps

    After a successful deployment of Industrial Edge 2.0, check to see that Jupyter Hub is running. To do this go to project manuela-ml-workspace check that jupyterhub pods are up and running.

    jupyerhub-pods

    Then, in the same project manuela-ml-namespace, select Networking/Routes and click on the URL associated with jupyterhub in the Location column.

    jupyterhub-url

    This will bring you to a web page at an address in the following format:

    • jupyterhub-manuela-ml-workspace.apps.*clustername*.*your-domain*

    Options for different types of Jupyter servers are shown. There are two options that are useful for this demo.

    • Standard Data Science. Select this notebook image for simpler notebooks like Data Analyses.ipynb
    • Tensorflow Notebook Image. Select this notebook image for more a complex notebook that require Tensorflow. E.g. Anomaly Detection-using-TF-and-Deep-Learning.ipynb

    At the bottom of the screen there is a Start server button. Select the type of Notebook server image and press Start server.

    jupyterhub-init-console

    Selecting Tensorflow notebook image:

    jupyter-tf-server

    On the next screen upload the following files from manuela-dev/ml-models/anomaly-detection:

    • One of the Jupyter notebooks
      • Data-Analyses.ipynb for a somewhat simpler demo
      • Anomaly Detection-using-TF-and-Deep-Learning.ipynb for a Tensorflow demo.
    • raw-data.cvs

    upload-ml-files

    Open the notebook by double clicking on the notebook file (ending in .ipynb)

    anomaly-detection-notebook

    After opening the notebook successfully, walk through the demonstration by pressing play and iterating through the commands in the playbook. Jupyter playbooks are interactive and you may make changes and also save those changes. Also, some steps in the notebook take milliseconds, however, other steps can take a long time (up to an hour), so check on the completion of steps.

    Remember that changes to the notebook will require downloading, committing, and pushing that notebook to the git repository so that it gets redeployed to the factories.

    QUICKLINKS

    CONTRIBUTE

    OpenShift Cluster Sizing for the Industrial Edge Pattern

    Tested Platforms

    The Industrial-Edge pattern has been tested in the following Certified Cloud Providers.

    Certified Cloud Providers4.84.94.10
    Amazon Web Services:heavy_check_mark:
    Microsoft Azure:heavy_check_mark:
    Google Cloud Platform:heavy_check_mark:

    General OpenShift Minimum Requirements

    OpenShift 4 has the following minimum requirements for sizing of nodes:

    • Minimum 4 vCPU (additional are strongly recommended).
    • Minimum 16 GB RAM (additional memory is strongly recommended, especially if etcd is colocated on masters).
    • Minimum 40 GB hard disk space for the file system containing /var/.
    • Minimum 1 GB hard disk space for the file system containing /usr/local/bin/.

    There are several applications that comprise the industrial-edge pattern. In addition, the industrial-edge pattern also includes a number of supporting operators that are installed by OpenShift GitOps using ArgoCD.

    Industrial-Edge Pattern Components

    Here’s an inventory of what gets deployed by the Industrial-Edge pattern on the Datacenter/Hub OpenShift cluster:

    NameKindNamespaceDescription
    line-dashboardApplicationmanuela-tst-allFrontend application
    machine-sensor-1Applicationmanuela-tst-allData publisher
    machine-sensor-2Applicationmanuela-tst-allData publisher
    messagingApplicationmanuela-tst-allData subscriber
    mqtt2kafka-integrationApplicationmanuela-tst-allKafka Integration
    anomaly-detection-predictor-0-anomaly-detectionApplicationmanuela-tst-allAnomaly detection application
    manuela-kafka-cluster-entity-operatorOperatormanuela-tst-allKafka
    Red Hat Advanced Cluster ManagementOperatoropen-cluster-managementAdvance Cluster Management
    Red Hat Integration - AMQ BrokerOperatormanuela-tst-allAMQ Broker
    Red Hat Integration - AMQ StreamsOperatormanuela-tst-allAMQ Streams
    Open Data HubOperatoropenshift-operatorsOpen Data Hub
    Red Hat OpenShift GitOpsOperatoropenshift-operatorsOpenShift GitOps
    Red Hat Integration - Camel KOperatormanuela-tst-allIntegration Platform, Kamelet Binding, Kamelet
    Red Hat OpenShift PipelinesOperatorAll NamespacesTekton Config, Pipelines, Triggers, Addons
    Seldon OperatorOperatormanuela-tst-allSeldon Deployment

    Industrial-Edge Pattern OpenShift Datacenter HUB Cluster Size

    The Industrial-Edge pattern has been tested with a defined set of specifically tested configurations that represent the most common combinations that Red Hat OpenShift Container Platform (OCP) customers are using or deploying for the x86_64 architecture.

    The Datacenter HUB OpenShift Cluster is made up of the the following on the AWS deployment tested:

    Node TypeNumber of nodesCloud ProviderInstance Type
    Master3Amazon Web Servicesm5.xlarge
    Worker4Amazon Web Servicesm5.xlarge

    The Datacenter HUB OpenShift cluster needs to be a bit bigger than the Factory/Edge clusters because this is where the developers will be running pipelines to build and deploy the Industrial Edge pattern on the cluster. The above cluster sizing is close to a minimum size for a Datacenter HUB cluster. In the next few sections we take some snapshots of the cluster utilization while the Industrial Edge pattern is running. Keep in mind that resources will have to be added as more developers are working building their applications.

    Datacenter Cluster utilization

    Below is a snapshot of the OpenShift cluster utilization while running the Industrial-Edge pattern:

    CPUMemoryFile SystemNetworkPod Count
    13.84 Used 42.16 available of 5673.5 GiB 146.3 GiB available of 219.8 GiB106 GiB 732.9 GiB available of 838.9 GiB20.65 MBps in 22.84 MBps out354 pods

    Industrial-Edge Pattern OpenShift Factory Edge Cluster Size

    The OpenShift cluster is made of 3 Nodes combining Master/Workers for the Edge/Factory cluster.

    Node TypeNumber of nodesCloud ProviderInstance Type
    Master/Worker3Google Cloudn1-standard-8
    Master/Worker3Amazon Cloud Servicesm5.2xlarge
    Master/Worker3Microsoft AzureStandard_D8s_v3

    Factory/Edge Cluster Utilization

    GCP

    This is a snapshot of a Google Cloud Factory Edge cluster running the production Industrial-Edge pattern.

    CPUMemoryFile SystemNetworkPod Count
    6.55 17.45 available of 2443.19 GiB usage 45.09 GiB available of 88.28 GiB48.45 GiB usage 334 GiB available of 382.5 GiB9.64 MBps in15.79 MBps out187 pods

    AWS

    This is a snapshot of a Amazon Web Services Factory Edge cluster running the production Industrial-Edge pattern.

    CPUMemoryFile SystemNetworkPod Count
    5.1 18.9 available of 2442.91 GiB 49.27 GiB available of 92.18 GiB51.54 GiB 308 GiB available of 359.5 GiB9.41 MBps in 10.38 MBps out194 pods

    Azure

    This is a snapshot of an Azure Factory Edge cluster running the production Industrial-Edge pattern.

    CPUMemoryFile SystemNetworkPod Count
    7.86 15.65 available of 2442.76 Gib used 51.15 GiB available of 94.2 GiB71.29 GiB used 2.93 TiB available of 3 TiB8.98 MBps in 9.64 MBps out192 *pods

    AWS Instance Types

    The industrial-edge pattern was tested with the highlighted AWS instances in bold. The OpenShift installer will let you know if the instance type meets the minimum requirements for a cluster.

    The message that the openshift installer will give you will be similar to this message

    INFO Credentials loaded from default AWS environment variables
 FATAL failed to fetch Metadata: failed to load asset "Install Config": [controlPlane.platform.aws.type: Invalid value: "m4.large": instance type does not meet minimum resource requirements of 4 vCPUs, controlPlane.platform.aws.type: Invalid value: "m4.large": instance type does not meet minimum resource requirements of 16384 MiB Memory]
-

    Below you can find a list of the AWS instance types that can be used to deploy the industrial-edge pattern.

    Instance typeDefault vCPUsMemory (GiB)DatacenterFactory/Edge
    3x3 OCP Cluster3 Node OCP Cluster
    m4.xlarge416NN
    m4.2xlarge832YY
    m4.4xlarge1664YY
    m4.10xlarge40160YY
    m4.16xlarge64256YY
    m5.xlarge416YN
    m5.2xlarge832YY
    m5.4xlarge1664YY
    m5.8xlarge32128YY
    m5.12xlarge48192YY
    m5.16xlarge64256YY
    m5.24xlarge96384YY

    The OpenShift cluster is made of 4 Masters and 3 Workers for the Datacenter and the Edge/Factory cluster are made of 3 Master/Worker nodes. For the node sizes we used the m5.xlarge on AWS and this instance type met the minimum requirements to deploy the industrial-edge pattern successfully on the Datacenter hub. On the Factory/Edge cluster we used the m5.2xlarge since the minimum cluster was comprised of 3 nodes. .

    To understand better what types of nodes you can use on other Cloud Providers we provide some of the details below.

    Azure Instance Types

    The industrial-edge pattern was also deployed on Azure using the Standard_D8s_v3 VM size. Below is a table of different VM sizes available for Azure. Keep in mind that due to limited access to Azure we only used the Standard_D8s_v3 VM size.

    The OpenShift cluster is made of 3 Master and 3 Workers for the Datacenter cluster.

    The OpenShift cluster is made of 3 Nodes combining Master/Workers for the Edge/Factory cluster.

    TypeSizesDescription
    General purposeB, Dsv3, Dv3, Dasv4, Dav4, DSv2, Dv2, Av2, DC, DCv2, Dv4, Dsv4, Ddv4, Ddsv4Balanced CPU-to-memory ratio. Ideal for testing and development, small to medium databases, and low to medium traffic web servers.
    Compute optimizedF, Fs, Fsv2, FXHigh CPU-to-memory ratio. Good for medium traffic web servers, network appliances, batch processes, and application servers.
    Memory optimizedEsv3, Ev3, Easv4, Eav4, Ev4, Esv4, Edv4, Edsv4, Mv2, M, DSv2, Dv2High memory-to-CPU ratio. Great for relational database servers, medium to large caches, and in-memory analytics.
    Storage optimizedLsv2High disk throughput and IO ideal for Big Data, SQL, NoSQL databases, data warehousing and large transactional databases.
    GPUNC, NCv2, NCv3, NCasT4_v3, ND, NDv2, NV, NVv3, NVv4Specialized virtual machines targeted for heavy graphic rendering and video editing, as well as model training and inferencing (ND) with deep learning. Available with single or multiple GPUs.
    High performance computeHB, HBv2, HBv3, HC, HOur fastest and most powerful CPU virtual machines with optional high-throughput network interfaces (RDMA).

    For more information please refer to the Azure VM Size Page.

    Google Cloud (GCP) Instance Types

    The industrial-edge pattern was also deployed on GCP using the n1-standard-8 VM size. Below is a table of different VM sizes available for GCP. Keep in mind that due to limited access to GCP we only used the n1-standard-8 VM size.

    The OpenShift cluster is made of 3 Master and 3 Workers for the Datacenter cluster.

    The OpenShift cluster is made of 3 Nodes combining Master/Workers for the Edge/Factory cluster.

    The following table provides VM recommendations for different workloads.

    | General purpose | Workload optimized

    Cost-optimizedBalancedScale-out optimizedMemory-optimizedCompute-optimizedAccelerator-optimized
    E2N2, N2D, N1T2DM2, M1C2A2
    Day-to-day computing at a lower costBalanced price/performance across a wide range of VM shapesBest performance/cost for scale-out workloadsUltra high-memory workloadsUltra high performance for compute-intensive workloadsOptimized for high performance computing workloads

    For more information please refer to the GCP VM Size Page.

    QUICKLINKS

    Below you can find a list of the AWS instance types that can be used to deploy the industrial-edge pattern.

    Instance typeDefault vCPUsMemory (GiB)DatacenterFactory/Edge
    3x3 OCP Cluster3 Node OCP Cluster
    m4.xlarge416NN
    m4.2xlarge832YY
    m4.4xlarge1664YY
    m4.10xlarge40160YY
    m4.16xlarge64256YY
    m5.xlarge416YN
    m5.2xlarge832YY
    m5.4xlarge1664YY
    m5.8xlarge32128YY
    m5.12xlarge48192YY
    m5.16xlarge64256YY
    m5.24xlarge96384YY

    The OpenShift cluster is made of 4 Masters and 3 Workers for the Datacenter and the Edge/Factory cluster are made of 3 Master/Worker nodes. For the node sizes we used the m5.xlarge on AWS and this instance type met the minimum requirements to deploy the industrial-edge pattern successfully on the Datacenter hub. On the Factory/Edge cluster we used the m5.2xlarge since the minimum cluster was comprised of 3 nodes. .

    To understand better what types of nodes you can use on other Cloud Providers we provide some of the details below.

    Azure Instance Types

    The industrial-edge pattern was also deployed on Azure using the Standard_D8s_v3 VM size. Below is a table of different VM sizes available for Azure. Keep in mind that due to limited access to Azure we only used the Standard_D8s_v3 VM size.

    The OpenShift cluster is made of 3 Master and 3 Workers for the Datacenter cluster.

    The OpenShift cluster is made of 3 Nodes combining Master/Workers for the Edge/Factory cluster.

    TypeSizesDescription
    General purposeB, Dsv3, Dv3, Dasv4, Dav4, DSv2, Dv2, Av2, DC, DCv2, Dv4, Dsv4, Ddv4, Ddsv4Balanced CPU-to-memory ratio. Ideal for testing and development, small to medium databases, and low to medium traffic web servers.
    Compute optimizedF, Fs, Fsv2, FXHigh CPU-to-memory ratio. Good for medium traffic web servers, network appliances, batch processes, and application servers.
    Memory optimizedEsv3, Ev3, Easv4, Eav4, Ev4, Esv4, Edv4, Edsv4, Mv2, M, DSv2, Dv2High memory-to-CPU ratio. Great for relational database servers, medium to large caches, and in-memory analytics.
    Storage optimizedLsv2High disk throughput and IO ideal for Big Data, SQL, NoSQL databases, data warehousing and large transactional databases.
    GPUNC, NCv2, NCv3, NCasT4_v3, ND, NDv2, NV, NVv3, NVv4Specialized virtual machines targeted for heavy graphic rendering and video editing, as well as model training and inferencing (ND) with deep learning. Available with single or multiple GPUs.
    High performance computeHB, HBv2, HBv3, HC, HOur fastest and most powerful CPU virtual machines with optional high-throughput network interfaces (RDMA).

    For more information please refer to the Azure VM Size Page.

    Google Cloud (GCP) Instance Types

    The industrial-edge pattern was also deployed on GCP using the n1-standard-8 VM size. Below is a table of different VM sizes available for GCP. Keep in mind that due to limited access to GCP we only used the n1-standard-8 VM size.

    The OpenShift cluster is made of 3 Master and 3 Workers for the Datacenter cluster.

    The OpenShift cluster is made of 3 Nodes combining Master/Workers for the Edge/Factory cluster.

    The following table provides VM recommendations for different workloads.

    | General purpose | Workload optimized

    Cost-optimizedBalancedScale-out optimizedMemory-optimizedCompute-optimizedAccelerator-optimized
    E2N2, N2D, N1T2DM2, M1C2A2
    Day-to-day computing at a lower costBalanced price/performance across a wide range of VM shapesBest performance/cost for scale-out workloadsUltra high-memory workloadsUltra high performance for compute-intensive workloadsOptimized for high performance computing workloads

    For more information please refer to the GCP VM Size Page.

    QUICKLINKS

    CONTRIBUTE

    Industrial Edge Deployment Script

    Objectives

    There’s no experience like hands-on experience and being able to see industrial edge scenarios. This is a demo for the Industrial Edge Validated Pattern using the latest product and technology improvements.

    • Show Red Hat Operators being deployed
    • Show available Red Hat Pipelines for the Industrial Edge pattern
    • Show the seed pipeline running and explain what is is doing
    • Demonstration of the Red Hat ArgoCD views
    • Show the openshift-gitops-server view
    • Show the datacenter-gitops-server view
    • Show the factory-gitops-server view

    For Information on the Red Hat Validated Patterns, visit our website

    See the pattern in action

    Watch the following video for a demonstration of OpenShift Pipelines in the Industrial Edge Pattern

    In this article, we give an overview of the demo and step by step instructions on how to get started.

    Getting Started

    NOTE: This demo takes a “bring your own cluster” approach, which means this pattern/demo will not deploy any OpenShift clusters.

    This demo script begins after the completion of you running ./pattern.sh make install from our Getting Started Guide

    Demo: Quick Health Check

    NOTE: This is a complex setup, and sometimes things can go wrong. Do a quick check of the essentials:

    There is an initial Seed Pipeline run in namespace manuela-ci that builds all required container images into the local registry. Check that the run was successful like this:

    Pipeline Success

    If it did fail, try “Rerun” on the Pipeline run page:

    Re-run Pipeline

    Check that the “Line Dashboard” in the development namespace is showing Data. The Link is in the bottom of the email under “Deployed Applications”. The Application should open - click on the “Realtime Data” Navigation on the left and wait a bit. Data should be visualized as received. Note that there is only vibration data! We will soon change that and activate temperature data also.

    If you wait a bit more (usually every 2-3 minutes), you will see an anomaly and alert (which is created by an ML Model)

    line dashboard

    ArgoCD - all healthy and synced? -Login to the datacenter Argo. Link and password are in the email under ArgoCD Deployments. Make sure you get the right “datacenter one - there is another one for OpenShift gitops which could be confusing. It should look like this:

    Healthy Argo

    Demo: Configuration Changes with GitOps

    Follow the procedures here

    Demo: Application Changes with DevOps

    Follow the procedures here

    Demo: Application AI Model Changes with DevOps

    Follow the procedures here

    Troubleshooting

    If you run into any problems, checkout the potential/Known issues list: http://validatedpatterns.io/industrial-edge/troubleshooting/

    Summary

    In this demo we: , we show you how to get started with the Industrial Edge Validated Pattern. In

    • Show you how to get started with the Industrial Edge Pattern
    • Make configuration changes with GitOps
    • Make application changes with DevOps
    • Use DevOps to make changes to an Application AI model
    • Stream events from the edge to the datacenter

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Remember to commit the changes and push to GitHub so that GitOps can see -your changes and apply them.

    Deploy a factory cluster

    For instructions on how to prepare and import a factory cluster please read the section importing a cluster. Use clusterGroup=factory.

    You’re done

    That’s it! Go to your factory (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. Be patient, it will take a while for the ACM agent and agent-addons to launch. After that, the operator OpenShift GitOps will run. When it’s finished coming up launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    Next up

    Work your way through the Industrial Edge 2.0 GitOps/DevOps demos

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    The most important ArgoCD instance to examine at this point is data-center-gitops-server. This is where all the applications for the datacenter, including the test environment, can be tracked.

  • Apply the secrets from the values-secret-industrial-edge.yaml to the secrets management Vault. This can be done through Vault’s UI - manually without the file. The required secrets and scopes are:

    • secret/hub/git git username & password (GitHub token)
    • secret/hub/imageregistry Quay or DockerHub username & password
    • secret/hub/aws - AWS values read from your ~/.aws/credentials

    Using the Vault UI check that the secrets have been setup.

    For more information on secrets management see here. For information on Hashicorp’s Vault see here

  • Check all applications are synchronised

    • Next Steps

    Help & Feedback{: .btn .fs-5 .mb-4 .mb-md-0 .mr-2 } Report Bugs{: .btn .btn-red .fs-5 .mb-4 .mb-md-0 .mr-2 }

    Once the data center has been setup correctly and confirmed to be working, you can:

    1. Add a dedicated cluster to deploy the factory pieces using ACM

    2. Once the data center and the factory have been deployed you will want to check out and test the Industrial Edge 2.0 demo code. You can find that here

      a. Making configuration changes with GitOps a. Making application changes using DevOps -a. Making AI/ML model changes with DevOps

    Uninstalling

    We currently do not support uninstalling this pattern.

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Ideas for Customization

    Why change it?

    One of the major goals of the Red Hat patterns development process is to create modular, customizable demos. The Industrial Edge demonstration includes multiple, simulated, IoT devices publishing their temperature and vibration telemetry to our data center and ultimately persisting the data into an AWS S3 storage service bucket which we call the Data Lake. All of this is done using our Red Hat certified products running on OpenShift.

    This demo in particular can be customized in a number of ways that might be very interesting - and here are some starter ideas with some instructions on exactly what and where changes would need to be made in the pattern to accommodate those changes.

    HOWTO Forking the Industrial Edge repository to your github account

    Hopefully we are all familiar with GitHub. If you are not GitHub is a code hosting platform for version control and collaboration. It lets you and others work together on projects from anywhere. Our Industrial Edge GitOps repository is available in our Validated Patterns GitHub organization.

    To fork this repository, and deploy the Industrial Edge pattern, follow the steps found in our Getting Started section. This will allow you to follow the next few HOWTO guides in this section.

    Our sensors have been configured to send data relating to the vibration of the devices. To show the power of GitOps, and keeping state in a git repository, we can make a change to the config map of one of the sensors to detect and report data on temperature. This is done via a variable called SENSOR_TEMPERATURE_ENABLED that is initially set to false. Setting this variable to true will trigger the GitOps engine to synchronize the application, restart the machine sensor and apply the change.

    There are two environments in the Industrial Edge demonstration:

    • The staging environment that lives in the manuela-tst-all namespace
    • The production environment which lives in the stormshift namespaces

    As an operator you would first make changes to the staging first. Here are the steps to see how the GitOps engine does it’s magic. These changes will be reflected in the staging environment Line Dashboard UI in the manuela-tst-all namespace.

    • The config maps in question live in the charts/datacenter/manuela-tst/templates/machine-sensor directory
    • There are two config maps that we can change:
      • machine-sensor-1-configmap.yaml
      • machine-sensor-2-configmap.yaml
    • Change the following variable in machine-sensor-1-configmap.yaml
      • SENSOR_TEMPERATURE_ENABLED: “true”
    • Make sure you commit the changes to git
      • git add machine-sensor-1-configmap.yaml
      • git commit -m “Changed SENSOR_TEMPERATURE_ENABLED to true”
      • git push
    • Now you can go to the Line Dashboard application and see how the UI shows the temperature for that device. You can find the route link by:
      • Change the Project context to manuela-tst-all
      • Navigate to Networking->Routes
      • Press on the Location link to see navigate to the UI.

    HOWTO Applying the pattern to a new use case

    There are a lot of IoT devices that we could add to this pattern. In today’s world we have IoT devices that perform different functions and these devices are connected to a network where they have the ability of sending telemetry data to other devices or a central data center. In this particular use case we address an Industrial sector but what about applying this use case to other sectors such as Automotive or Delivery service companies?

    If we take the Deliver Service use case, and apply it to this pattern, we would have to take into account the following aspects:

    • The main components in the pattern architecture can be used as is.
      • The broker and kafka components are the vehicles for the streaming data coming from the devices.
    • The IoT sensor software would have to be developed. The IoT devices will now be mobile so that presents a few challenges tracking the devices in part due to spotty connectivity to send the data stream.
    • The number of IoT devices to be tracked will increase depending on the fleet of delivery trucks out in the field.
      • Scalability will be an important aspect for the pattern to be able to handle.
    • A new AI/ML model would have to be developed to “learn” through the analysis of the data stream from the IoT devices.

    The idea is that this pattern can be used for other use cases keeping the main components in place. The components that would be new to the pattern are: IoT device code, AI/ML models, and specific kafka/broker topics to keep track of.

    Next Steps

    What ideas for customization do you have? Can you use this pattern for other use cases? Let us know through our feedback link below.

    Help & Feedback{: .btn .fs-5 .mb-4 .mb-md-0 .mr-2 } -Report Bugs{: .btn .btn-red .fs-5 .mb-4 .mb-md-0 .mr-2 }

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Industrial Edge Pattern

    Red Hat Validated Patterns are detailed deployments created for different use cases. These pre-defined computing configurations bring together the Red Hat portfolio and technology ecosystem to help you stand up your architectures faster. Example application code is provided as a demonstration, along with the various open source projects and Red Hat products required for the deployment to work. Users can then modify the pattern for their own specific application.

    Use Case: Boosting manufacturing efficiency and product quality with artificial intelligence/machine learning (AI/ML) out to the edge of the network.

    Background: Microcontrollers and other types of simple computers have long been widely used on factory floors and processing plants to monitor and control the many machines required to implement the many machines required to implement many modern manufacturing workflows. The manufacturing industry has consistently used technology to fuel innovation, production optimization, and operations. However, historically, control systems were mostly “dumb” in that they mostly took actions in response to pre-programmed triggers and heuristics. For example, predictive maintenance commonly took place on either a set length of time or the number of hours was in service. Supervisory control and data acquisition (SCADA) has often been used to collectively describe these hardware and software systems, which mostly functioned independently of the company’s information technology (IT) systems. Companies increasingly see the benefit of bridging these operational technology (OT) systems with their IT. Factory systems can be much more flexible as a result. They can also benefit from newer technologies such as AI/ML, thereby allowing for tasks like maintenance to be scheduled based on multiple real-time measurements rather than simple programmed triggers while bringing processing power closer to data.

    Solution Overview

    High level view of Industrial Edge

    Figure 1. Industrial edge solution overview.

    Figure 1 provides an overview of the industrial edge solution. It is applicable across a number of verticals including manufacturing.

    This solution:

    • Provides real-time insights from the edge to the core datacenter
    • Secures GitOps and DevOps management across core and factory sites
    • Provides AI/ML tools that can reduce maintenance costs

    Different roles within an organization have different concerns and areas of focus when working with this distributed AL/ML architecture across two logical types of sites: the core datacenter and the factories. (As shown in Figure 2.)

    • The core datacenter. This is where data scientists, developers, and operations personnel apply the changes to their models, application code, and configurations.
    • The factories. This is where new applications, updates and operational changes are deployed to improve quality and efficiency in the factory..

    Industrial Edge Architecture

    Figure 2. Mapping of organizational roles to architectural areas.

    Industrial Edge Computing

    Figure 3. Overall data flows of solution.

    Figure 3 provides a different high-level view of the solution with a focus on the two major dataflow streams.

    1. Moving sensor data and events from the operational/shop floor edge towards the core. The idea is to centralize as much as possible, but decentralize as needed. For example, sensitive production data might not be allowed to leave the premises. Think of a temperature curve of an industrial oven; it might be considered crucial intellectual property of the customer. Or the sheer amount of raw data (maybe 10,000 events per second) might be too expensive to transfer to a cloud datacenter. In the above diagram, this is from left to right. In other diagrams the edge / operational level is usually at the bottom and the enterprise/cloud level at the top. Thus, this is also referred to as northbound traffic.

    2. Push code, configuration, master data, and machine learning models from the core (where development, testing, and training is happening) towards the edge / shop floors. As there might be 100 plants with 1000s of lines, automation and consistency is key. In the above diagram, this is from right to left, in a top/down view, it is called southbound traffic.

    Logical Diagrams

    Conceptual view of Industrial Edge deployed at various locations

    Figure 4: Industrial Edge solution as logically and physically distributed across multiple sites.

    The following technology was chosen for this solution as depicted logically in Figure 4.

    The Technology

    Red Hat OpenShift is an enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere.

    Red Hat Application Foundations (also sold as Red Hat Integration) includes frameworks and capabilities for designing, building, deploying, connecting, securing, and scaling cloud-native applications, including foundational patterns like microservices, API-first, and data streaming. When combined with Red Hat OpenShift, Application Foundations creates a hybrid cloud platform for development and operations teams to build and modernize applications efficiently and with attention to security, while balancing developer choice and flexibility with operational control. -It includes, among other components::

    Red Hat Runtimes is a set of products, tools, and components for developing and maintaining cloud-native applications. It offers lightweight runtimes and frameworks for highly distributed cloud architectures, such as microservices. Built on proven open source technologies, it provides development teams with multiple modernization options to enable a smooth transition to the cloud for existing applications.

    Red Hat AMQ is a massively scalable, distributed, and high-performance data streaming platform based on the Apache Kafka project. It offers a distributed backbone that allows microservices and other applications to share data with high throughput and low latency.

    Red Hat Data Foundation is software-defined storage for containers. Engineered as the data and storage services platform for Red Hat OpenShift, Red Hat Data Foundation helps teams develop and deploy applications quickly and efficiently across clouds. It is based on the open source Ceph, Rook, and Noobaa projects.

    Red Hat Advanced Cluster Management for Kubernetes (RHACM) controls clusters and applications from a single console, with built-in security policies. It extends the value of Red Hat OpenShift by deploying applications, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat Enterprise Linux is the world’s leading enterprise Linux platform. It’s an open source operating system (OS). It’s the foundation from which you can scale existing apps—and roll out emerging technologies—across bare-metal, virtual, container, and all types of cloud environments.

    Architectures

    Edge manufacturing with messaging and ML

    Data interaction of various Industrial Edge components

    Figure 5: Industrial Edge solution showing messaging and ML components schematically.

    As shown in Figure 5, data coming from sensors is transmitted over MQTT (Message Queuing Telemetry Transport) to Red Hat AMQ, which routes sensor data for two purposes: model development in the core data center and live inference in the factory data centers. The data is then relayed on to Red Hat AMQ for further distribution within the factory datacenter and out to the core datacenter. MQTT is the most commonly used messaging protocol for Internet of Things (IoT) applications.

    The lightweight Apache Camel K, a lightweight integration framework built on Apache Camel that runs natively on Kubernetes, provides MQTT (Message Queuing Telemetry Transport) integration that normalizes and routes sensor data to the other components.

    That sensor data is mirrored into a data lake that is provided by Red Hat OpenShift Data Foundation. Data scientists then use various tools from the open source Open Data Hub project to perform model development and training, pulling and analyzing content from the data lake into notebooks where they can apply ML frameworks.

    Once the models have been tuned and are deemed ready for production, the artifacts are committed to git which kicks off an image build of the model using OpenShift Pipelines (based on the upstream Tekton), a serverless CI/CD system that runs pipelines with all the required dependencies in isolated containers.

    The model image is pushed into OpenShift’s integrated registry running in the core datacenter which is then pushed back down to the factory datacenter for use in inference.

    Using network segragation to protect factories and operations infrastructure from cyber attacks

    Figure 6: Industrial Edge solution showing network flows schematically.

    As shown in Figure 6, in order to protect the factories and operations infrastructure from cyber attacks, the operations network needs to be segregated from the enterprise IT network and the public internet. The factory machinery, controllers, and devices need to be further segregated from the factory data center and need to be protected behind a firewall.

    Edge manufacturing with GitOps

    Using GitOps for managing any changes to clusters and applications

    Figure 7: Industrial Edge solution showing a schematic view of the GitOps workflows.

    GitOps is an operational framework that takes DevOps best practices used for application development such as version control, collaboration, compliance, and CI/CD, and applies them to infrastructure automation. Figure 6 shows how, for these industrial edge manufacturing environments, GitOps provides a consistent, declarative approach to managing individual cluster changes and upgrades across the centralized and edge sites. Any changes to configuration and applications can be automatically pushed into operational systems at the factory.

    Secrets exchange and management

    Secret exchange and management

    Figure 8: Schematic view of secrets exchange and management in an Industrial Edge solution.

    Authentication is used to securely deploy and update components across multiple locations. The credentials are stored using a secrets management solution like Hashicorp Vault. The external secrets component is used to integrate various secrets management tools (AWS Secrets Manager, Google Secrets Manager, Azure Key Vault). As shown in Figure 7, these secrets are then passed to Red Hat Advanced Cluster Management for Kubernetes (RHACM) which pushes the secrets to the RHACM agent at the edge clusters based on policy. RHACM is also responsible for providing secrets to OpenShift for GitOps workflows( using Tekton and Argo CD).

    For logical, physical and dataflow diagrams, please see excellent work done by the Red Hat Portfolio Architecture team

    Demo Scenario

    This scenario is derived from the MANUela work done by Red Hat Middleware Solution Architects in Germany in 2019/20. The name MANUela stands for MANUfacturing Edge Lightweight Accelerator, you will see this acronym in a lot of artifacts. It was developed on a platform called stormshift.

    The demo has been updated 2021 with an advanced GitOps framework.

    Demo Scenario

    Figure 9. High-level demo summary. The specific example is machine condition monitoring based on sensor data in an industrial setting, using AI/ML. It could be easily extended to other use cases such as predictive maintenance, or other verticals.

    The demo scenario reflects the data flows described earlier and shown in Figure 3 by having three layers.

    Line Data Server: the far edge, at the shop floor level.

    Factory Data Center: the near edge, at the plant, but in a more controlled environment.

    Central Data Center: the cloud/core, where ML model training, application development, testing, and related work happens. (Along with ERP systems and other centralized functions that are not part of this demo.)

    The northbound traffic of sensor data is visible in Figure 9. It flows from the sensor at the bottom via MQTT to the factory, where it is split into two streams: one to be fed into an ML model for anomaly detection and another one to be streamed up to the central data center via event streaming (using Kafka) to be stored for model training.

    The southbound traffic is abstracted in the App-Dev / Pipeline box at the top. This is where GitOps kicks in to push config or version changes down into the factories.

    Demo Script

    To deploy the Industrial Edge Pattern demo yourself, follow the demo script

    Download diagrams

    View and download all of the diagrams above in our open source tooling site.

    [Open Diagrams]

    Pattern Structure

    Presentation

    View a presentation slide deck about Industrial Edge here

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    When this happens, the pipeline may not entirely stop running. It is safe to stop/cancel the pipeline run, and desirable to do so, since multiple pipelines attempting to change the repository at the same time could cause more failures.

    Resolution: Run make seed in the root of the repository OR re-run the failed pipeline segment (e.g. seed-iot-frontend or seed-iot-consumer).

    We’re looking into better long-term fixes for a number of the situations that can cause these situations as #40.

    Symptom: Error in “push-*” pipeline tasks

    Cause: Multiple processes or people were trying to make changes to the repository at the same time. The state of the repository changed in the middle of the process in such a way that the update was not a “fast-forward” in git terms.

    Resolution: Re-run the failed pipeline segment OR run make seed from the root of your fork of the industrial-edge repository.

    It is also possible that multiple pipelines were running at the same time and were making conflicting changes. We recommend running one pipeline at a time.

    Symptom: Pipelines application perpetually “progressing” and not showing green/healthy. May show “degraded”

    Cause: Most likely the application is missing the images that are built by the seed pipeline.

    Resolution: Run make seed from the root of your forked repository directory, which will build the images and deploy them to both test and production.

    Symptom: There is a “spinny” next to one of the resources in the app that never resolves

    Cause: Check for a PersistentVolumeClaim that is not in use.

    Resolution: Delete the unused PVC

    ArgoCD not syncing

    Symptom: ArgoCD shows an error and “Unknown” sync status

    Cause: A change has been made in the repository that renders invalid YAML

    Resolution: Fix the issue as identified by the error message, and commit and push the fix OR revert the last one.

    Certain changes might invalidate objects in ArgoCD, and this will prevent ArgoCD from deploying the change related to that commit. The error message for that situation might look like this (this particular change removed the Image details from the kustomization.yaml file, and we resolved it by re-adding the image entries:

    rpc error: code = Unknown desc = Manifest generation error (cached): `/bin/bash -c helm template . --name-template ${ARGOCD_APP_NAME:0:52} -f https://github.com/claudiol/industrial-edge/raw/deployment/values-global.yaml -f https://github.com/claudiol/industrial-edge/raw/deployment/values-datacenter.yaml --set global.repoURL=$ARGOCD_APP_SOURCE_REPO_URL --set global.targetRevision=$ARGOCD_APP_SOURCE_TARGET_REVISION --set global.namespace=$ARGOCD_APP_NAMESPACE --set global.pattern=industrial-edge --set global.valuesDirectoryURL=https://github.com/claudiol/industrial-edge/raw/deployment --post-renderer ./kustomize` failed exit status 1: Error: error while running post render on files: error while running command /tmp/https:__github.com_claudiol_industrial-edge/charts/datacenter/manuela-tst/kustomize. error output: ++ dirname /tmp/https:__github.com_claudiol_industrial-edge/charts/datacenter/manuela-tst/kustomize + BASE=/tmp/https:__github.com_claudiol_industrial-edge/charts/datacenter/manuela-tst + '[' /tmp/https:__github.com_claudiol_industrial-edge/charts/datacenter/manuela-tst = /tmp/https:__github.com_claudiol_industrial-edge/charts/datacenter/manuela-tst ']' + BASE=./ + cat + echo / /tmp/https:__github.com_claudiol_industrial-edge/charts/datacenter/manuela-tst / /tmp/https:__github.com_claudiol_industrial-edge/charts/datacenter/manuela-tst + ls -al total 44 drwxr-xr-x. 3 default root 166 Oct 6 20:59 . drwxr-xr-x. 7 default root 98 Oct 6 20:28 .. -rw-r--r--. 1 default root 1105 Oct 6 20:28 Chart.yaml -rw-r--r--. 1 default root 22393 Oct 6 20:59 helm.yaml -rw-r--r--. 1 default root 98 Oct 6 20:59 kustomization.yaml -rwxr-xr-x. 1 default root 316 Oct 6 20:28 kustomize -rw-r--r--. 1 default root 348 Oct 6 20:28 system-image-builder-role-binding.yaml drwxr-xr-x. 7 default root 115 Oct 6 20:28 templates -rw-r--r--. 1 default root 585 Oct 6 20:28 values.yaml + kubectl kustomize ./ Error: json: cannot unmarshal object into Go struct field Kustomization.images of type []image.Image : exit status 1 Use --debug flag to render out invalid YAML
-

    Symptom: Applications show “not in sync” status in ArgoCD

    Cause: There is a discrepancy between what the git repository says the application should have, and how that state is realized in ArgoCD.

    The installation mechanism currently installs operators as parts of multiple applications when running on the same cluster, so it is a race condition in ArgoCD to see which one “wins.” This is a problem with the way we are installing the patterns. We are tracking this as #38.

    QUICKLINKS

    Symptom: Applications show “not in sync” status in ArgoCD

    Cause: There is a discrepancy between what the git repository says the application should have, and how that state is realized in ArgoCD.

    The installation mechanism currently installs operators as parts of multiple applications when running on the same cluster, so it is a race condition in ArgoCD to see which one “wins.” This is a problem with the way we are installing the patterns. We are tracking this as #38.

    QUICKLINKS

    CONTRIBUTE

    Kong

    Validation status:
    Sandbox -Sandbox

    Kong

    A pattern for Kong Gateway Control Plane and Data Plane demo

    Repo

    QUICKLINKS

    Kong

    A pattern for Kong Gateway Control Plane and Data Plane demo

    Repo

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    Making some changes on the dashboard

    You can change some of the parameters and watch how the changes effect the dashboard.

    1. You can increase or decrease the number of image generators.

      $ oc scale deploymentconfig/image-generator --replicas=2

      Check the dashboard.

      $ oc scale deploymentconfig/image-generator --replicas=0

      Watch the dashboard stop processing images.

    2. You can also simulate the change of the AI model version - as it’s only an environment variable in the Serverless Service configuration.

      $ oc patch service.serving.knative.dev/risk-assessment --type=json -p '[{"op":"replace","path":"/spec/template/metadata/annotations/revisionTimestamp","value":"'"$(date +%F_%T)"'"},{"op":"replace","path":"/spec/template/spec/containers/0/env/0/value","value":"v2"}]'

      This changes the model version value, and the revisionTimestamp in the annotations, which triggers a redeployment of the service.

    QUICKLINKS

    CONTRIBUTE

    About customizing the pattern Medical Diagnosis pattern

    One of the major goals of the Validated Patterns development process is to create modular and customizable demos. The Medical Diagnosis pattern is just an example of how AI/ML workloads built for object detection and classification can be run on OpenShift clusters. Consider your workloads for a moment - how would your workload best consume the pattern framework? Do your consumers require on-demand or near real-time responses when using your application? Is your application processing images or data that is protected by either Government Privacy Laws or HIPAA? -The Medical Diagnosis pattern can answer the call to either of these requirements by using OpenShift Serverless and OpenShift Data Foundation.

    Understanding different ways to use the Medical Diagnosis pattern

    1. The Medical Diagnosis pattern is scanning X-Ray images to determine the probability that a patient might or might not have Pneumonia. Continuing with the medical path, the pattern could be used for other early detection scenarios that use object detection and classification. For example, the pattern could be used to scan C/T images for anomalies in the body such as Sepsis, Cancer, or even benign tumors. Additionally, the pattern could be used for detecting blood clots, some heart disease, and bowel disorders like Crohn’s disease.

    2. The Transportation Security Agency (TSA) could use the Medical Diagnosis pattern in a way that enhances their existing scanning capabilities to detect with a higher probability restricted items carried on a person or hidden away in a piece of luggage. With Machine Learning Operations (MLOps), the model is constantly training and learning to better detect those items that are dangerous but which are not necessarily metallic, such as a firearm or a knife. The model is also training to dismiss those items that are authorized; ultimately saving passengers from being stopped and searched at security checkpoints.

    3. Militaries could use images collected from drones, satellites, or other platforms to identify objects and determine with probability what that object is. For example, the model could be trained to determine a type of ship, potentially its country of origin, and other such identifying characteristics.

    4. Manufacturing companies could use the pattern to inspect finished products as they roll off a production line. An image of the item, including using different types of light, could be analyzed to help expose defects before packaging and distributing. The item could be routed to a defect area.

    These are just a few ideas to help you understand how you could use the Medical Diagnosis pattern as a framework for your application.

    QUICKLINKS

    Understanding different ways to use the Medical Diagnosis pattern

    1. The Medical Diagnosis pattern is scanning X-Ray images to determine the probability that a patient might or might not have Pneumonia. Continuing with the medical path, the pattern could be used for other early detection scenarios that use object detection and classification. For example, the pattern could be used to scan C/T images for anomalies in the body such as Sepsis, Cancer, or even benign tumors. Additionally, the pattern could be used for detecting blood clots, some heart disease, and bowel disorders like Crohn’s disease.

    2. The Transportation Security Agency (TSA) could use the Medical Diagnosis pattern in a way that enhances their existing scanning capabilities to detect with a higher probability restricted items carried on a person or hidden away in a piece of luggage. With Machine Learning Operations (MLOps), the model is constantly training and learning to better detect those items that are dangerous but which are not necessarily metallic, such as a firearm or a knife. The model is also training to dismiss those items that are authorized; ultimately saving passengers from being stopped and searched at security checkpoints.

    3. Militaries could use images collected from drones, satellites, or other platforms to identify objects and determine with probability what that object is. For example, the model could be trained to determine a type of ship, potentially its country of origin, and other such identifying characteristics.

    4. Manufacturing companies could use the pattern to inspect finished products as they roll off a production line. An image of the item, including using different types of light, could be analyzed to help expose defects before packaging and distributing. The item could be routed to a defect area.

    These are just a few ideas to help you understand how you could use the Medical Diagnosis pattern as a framework for your application.

    QUICKLINKS

    CONTRIBUTE

    About the Medical Diagnosis pattern

    Background

    This validated pattern is a modified version of the Medical Diagnosis pattern. It was extended to showcase 5th Generation Intel Xeon Scalable Processors capabilities, especially Intel AMX that speeds up AI workloads. The pattern is based on a demo implementation of an automated data pipeline for chest X-ray analysis that was previously developed by Red Hat. It includes the same functionality as the original demonstration, but uses the GitOps framework to deploy the pattern including Operators, creation of namespaces, and cluster configuration.

    Compared to the original Medical Diagnosis pattern, this one was extended by the Node Feature Discovery Operator, whose task is to detect hardware features and expose them as labels for this node. -Red Hat OpenShift Serverless component is modified to assign AI workload to nodes with available Intel AMX feature.

    Moreover, the Machine Learning model was quantized to int8 precision to improve efficiency thanks to Intel AMX, with only marginal accuracy loss.

    This pattern was also adapted to run in on-premise environments.

    Workflow

    • Node Feature Discovery operator labels nodes with Intel AMX capabilities.

    • Ingest chest X-rays from a simulated X-ray machine and puts them into an objectStore based on Ceph.

    • The objectStore sends a notification to a Kafka topic.

    • A KNative Eventing listener to the topic triggers a KNative Serving function.

    • KNative Serving was modified to schedule pods with AI workload only on nodes with enabled Intel AMX feature

    • An ML-trained model running in a container makes a risk assessment of Pneumonia for incoming images.

    • A Grafana dashboard displays the pipeline in real time, along with images incoming, processed, anonymized, and full metrics collected from Prometheus.

    The simplified pipeline without Intel AMX is showcased in this video.

    dashboard

    About the solution elements

    The solution aids the understanding of the following:

    • How to use a GitOps approach to keep in control of configuration and operations.

    • How to deploy AI/ML technologies for medical diagnosis using GitOps.

    The Medical Diagnosis pattern uses the following products and technologies:

    • Red Hat OpenShift Container Platform for container orchestration

    • Red Hat OpenShift GitOps, a GitOps continuous delivery (CD) solution

    • Red Hat AMQ, an event streaming platform based on the Apache Kafka

    • Red Hat OpenShift Serverless for event-driven applications

    • Red Hat OpenShift Data Foundation for cloud native storage capabilities

    • Grafana Operator to manage and share Grafana dashboards, data sources, and so on

    • Node Feature Discovery Operator to label nodes with Intel AMX capabilities

    About the architecture

    Presently, the Intel AMX accelerated Medical Diagnosis pattern does not have an edge component. Edge deployment capabilities are planned as part of the pattern architecture for a future release.

    edge medical diagnosis marketing slide

    Components are running on OpenShift either at the data center, at the medical facility, or public cloud running OpenShift.

    The diagram below shows the components that are deployed with the the data flows and API calls between them.

    physical dataflow

    Next steps

    QUICKLINKS

    Moreover, the Machine Learning model was quantized to int8 precision to improve efficiency thanks to Intel AMX, with only marginal accuracy loss.

    This pattern was also adapted to run in on-premise environments.

    Workflow

    • Node Feature Discovery operator labels nodes with Intel AMX capabilities.

    • Ingest chest X-rays from a simulated X-ray machine and puts them into an objectStore based on Ceph.

    • The objectStore sends a notification to a Kafka topic.

    • A KNative Eventing listener to the topic triggers a KNative Serving function.

    • KNative Serving was modified to schedule pods with AI workload only on nodes with enabled Intel AMX feature

    • An ML-trained model running in a container makes a risk assessment of Pneumonia for incoming images.

    • A Grafana dashboard displays the pipeline in real time, along with images incoming, processed, anonymized, and full metrics collected from Prometheus.

    The simplified pipeline without Intel AMX is showcased in this video.

    dashboard

    About the solution elements

    The solution aids the understanding of the following:

    • How to use a GitOps approach to keep in control of configuration and operations.

    • How to deploy AI/ML technologies for medical diagnosis using GitOps.

    The Medical Diagnosis pattern uses the following products and technologies:

    • Red Hat OpenShift Container Platform for container orchestration

    • Red Hat OpenShift GitOps, a GitOps continuous delivery (CD) solution

    • Red Hat AMQ, an event streaming platform based on the Apache Kafka

    • Red Hat OpenShift Serverless for event-driven applications

    • Red Hat OpenShift Data Foundation for cloud native storage capabilities

    • Grafana Operator to manage and share Grafana dashboards, data sources, and so on

    • Node Feature Discovery Operator to label nodes with Intel AMX capabilities

    About the architecture

    Presently, the Intel AMX accelerated Medical Diagnosis pattern does not have an edge component. Edge deployment capabilities are planned as part of the pattern architecture for a future release.

    edge medical diagnosis marketing slide

    Components are running on OpenShift either at the data center, at the medical facility, or public cloud running OpenShift.

    The diagram below shows the components that are deployed with the the data flows and API calls between them.

    physical dataflow

    Next steps

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    You must accept the security risks / self signed certificates before scaling the image-generator application

    Scale up the deployment

    As we mentioned earlier, we don’t have an x-ray machine hanging around that we can use for this demo, so we emulate one by creating an s3 bucket and hosting the x-ray images within it. In the "real world" an x-ray would be taken at an edge medical facility and then uploaded to an OpenShift Data Foundations (ODF) S3 compatible bucket in the Core Hospital, triggering the AI/ML workflow.

    To emulate the edge medical facility we use an application called image-generator which (when scaled up) will download the x-rays from s3 and put them in an ODF s3 bucket in the cluster, triggering the AI/ML workflow.

    Let’s scale the image-generator deploymentConfig up to start the pipeline

    Make sure that you are in the xraylab-1 project under the Developer context in the OpenShift Console

    In the Topology menu under the Developer context in the OpenShift Console:

    • Search for the image-generator application in the Topology console

    image generator

    • Click on the image-generator application ( you may have to zoom in on the highlighted application)

    • Switch to the Details menu in the application menu context

    • Click the ^ next to the pod donut

    image generator scale

    Demo Dashboard

    Now let’s jump over to the dashboard

    • Return to the topology screen

    • Select “Grafana” in the drop down for Filter by resource

    • Click the grafana icon

    • Open url to go open a browser for the grafana dashboard.

    Within the grafana dashboard:

    • click the dashboards icon

    • click Manage

    • select xraylab-1

    • finally select the XRay Lab folder

    dashboard

    In the dashboard on the right we see the images that have been uploaded, processed and anonymized. Images in the processed view have been through the AI/ML pipeline and images in the lower third are images that have been stripped of Personally Identifiable Information or PII.

    In the lower middle section of the dashboard we can see the distribution of images that are normal, unsure or pneumonia has been detected. We can also see the number of risk assessment containers running as well as the cpu and memory metrics for the pods.

    Summary

    You did it! You have completed the deployment of the medical diagnosis pattern! Hopefully you are getting ideas of how you can take advantage of our GitOps framework to deploy and manage your applications.

    The medical diagnosis pattern is more than just the identification and detection of pneumonia in x-ray images. It is an object detection and classification model built on top of Red Hat OpenShift and can be transformed to fit multiple use-cases within the object classification paradigm. Similar use-cases would be detecting contraband items in the Postal Service or even in luggage in an airport baggage scanner.

    For more information on Validated Patterns visit our website

    QUICKLINKS

    CONTRIBUTE

    Examine the medical-diagnosis-hub ArgoCD instance. You can track all the applications for the pattern in this instance.

  • Check that all applications are synchronized. There are thirteen different ArgoCD applications that are deployed as part of this pattern.

    • Viewing the Grafana based dashboard

    1. Accept the SSL certificates on the browser for the dashboard. In the OpenShift Container Platform web console, go to the Routes for project openshift-storage`. Click the URL for the s3-rgw.

      storage route

      Ensure that you see some XML and not the access denied error message.

      storage rgw route

    2. While still looking at Routes, change the project to xraylab-1. Click the URL for the image-server. Ensure that you do not see an access denied error message. You must to see a Hello World message.

      grafana routes

    3. Turn on the image file flow. There are three ways to go about this.

      You can go to the command-line (make sure you have KUBECONFIG set, or are logged into the cluster).

      $ oc scale deploymentconfig/image-generator --replicas=1 -n xraylab-1

      Or you can go to the OpenShift UI and change the view from Administrator to Developer and select Topology. From there select the xraylab-1 project.

      dev topology

      Right-click on the image-generator pod icon and select Edit Pod count.

      dev topology menu

      Up the pod count from 0 to 1 and save.

      dev topology pod count

      Alternatively, you can have the same outcome on the Administrator console.

      Go to the OpenShift UI under Workloads, select Deploymentconfigs for Project xraylab-1. -Click image-generator and increase the pod count to 1.

      start image flow

    Making some changes on the dashboard

    You can change some of the parameters and watch how the changes effect the dashboard.

    1. You can increase or decrease the number of image generators.

      $ oc scale deploymentconfig/image-generator --replicas=2

      Check the dashboard.

      $ oc scale deploymentconfig/image-generator --replicas=0

      Watch the dashboard stop processing images.

    2. You can also simulate the change of the AI model version - as it’s only an environment variable in the Serverless Service configuration.

      $ oc patch service.serving.knative.dev/risk-assessment --type=json -p '[{"op":"replace","path":"/spec/template/metadata/annotations/revisionTimestamp","value":"'"$(date +%F_%T)"'"},{"op":"replace","path":"/spec/template/spec/containers/0/env/0/value","value":"v2"}]'

      This changes the model version value, and the revisionTimestamp in the annotations, which triggers a redeployment of the service.

    QUICKLINKS

    start image flow

    Making some changes on the dashboard

    You can change some of the parameters and watch how the changes effect the dashboard.

    1. You can increase or decrease the number of image generators.

      $ oc scale deploymentconfig/image-generator --replicas=2

      Check the dashboard.

      $ oc scale deploymentconfig/image-generator --replicas=0

      Watch the dashboard stop processing images.

    2. You can also simulate the change of the AI model version - as it’s only an environment variable in the Serverless Service configuration.

      $ oc patch service.serving.knative.dev/risk-assessment --type=json -p '[{"op":"replace","path":"/spec/template/metadata/annotations/revisionTimestamp","value":"'"$(date +%F_%T)"'"},{"op":"replace","path":"/spec/template/spec/containers/0/env/0/value","value":"v2"}]'

      This changes the model version value, and the revisionTimestamp in the annotations, which triggers a redeployment of the service.

    QUICKLINKS

    CONTRIBUTE

    About customizing the pattern Medical Diagnosis pattern

    One of the major goals of the Validated Patterns development process is to create modular and customizable demos. The Medical Diagnosis pattern is just an example of how AI/ML workloads built for object detection and classification can be run on OpenShift clusters. Consider your workloads for a moment - how would your workload best consume the pattern framework? Do your consumers require on-demand or near real-time responses when using your application? Is your application processing images or data that is protected by either Government Privacy Laws or HIPAA? -The Medical Diagnosis pattern can answer the call to either of these requirements by using OpenShift Serverless and OpenShift Data Foundation.

    Understanding different ways to use the Medical Diagnosis pattern

    1. The Medical Diagnosis pattern is scanning X-Ray images to determine the probability that a patient might or might not have Pneumonia. Continuing with the medical path, the pattern could be used for other early detection scenarios that use object detection and classification. For example, the pattern could be used to scan C/T images for anomalies in the body such as Sepsis, Cancer, or even benign tumors. Additionally, the pattern could be used for detecting blood clots, some heart disease, and bowel disorders like Crohn’s disease.

    2. The Transportation Security Agency (TSA) could use the Medical Diagnosis pattern in a way that enhances their existing scanning capabilities to detect with a higher probability restricted items carried on a person or hidden away in a piece of luggage. With Machine Learning Operations (MLOps), the model is constantly training and learning to better detect those items that are dangerous but which are not necessarily metallic, such as a firearm or a knife. The model is also training to dismiss those items that are authorized; ultimately saving passengers from being stopped and searched at security checkpoints.

    3. Militaries could use images collected from drones, satellites, or other platforms to identify objects and determine with probability what that object is. For example, the model could be trained to determine a type of ship, potentially its country of origin, and other such identifying characteristics.

    4. Manufacturing companies could use the pattern to inspect finished products as they roll off a production line. An image of the item, including using different types of light, could be analyzed to help expose defects before packaging and distributing. The item could be routed to a defect area.

    These are just a few ideas to help you understand how you could use the Medical Diagnosis pattern as a framework for your application.

    QUICKLINKS

    Understanding different ways to use the Medical Diagnosis pattern

    1. The Medical Diagnosis pattern is scanning X-Ray images to determine the probability that a patient might or might not have Pneumonia. Continuing with the medical path, the pattern could be used for other early detection scenarios that use object detection and classification. For example, the pattern could be used to scan C/T images for anomalies in the body such as Sepsis, Cancer, or even benign tumors. Additionally, the pattern could be used for detecting blood clots, some heart disease, and bowel disorders like Crohn’s disease.

    2. The Transportation Security Agency (TSA) could use the Medical Diagnosis pattern in a way that enhances their existing scanning capabilities to detect with a higher probability restricted items carried on a person or hidden away in a piece of luggage. With Machine Learning Operations (MLOps), the model is constantly training and learning to better detect those items that are dangerous but which are not necessarily metallic, such as a firearm or a knife. The model is also training to dismiss those items that are authorized; ultimately saving passengers from being stopped and searched at security checkpoints.

    3. Militaries could use images collected from drones, satellites, or other platforms to identify objects and determine with probability what that object is. For example, the model could be trained to determine a type of ship, potentially its country of origin, and other such identifying characteristics.

    4. Manufacturing companies could use the pattern to inspect finished products as they roll off a production line. An image of the item, including using different types of light, could be analyzed to help expose defects before packaging and distributing. The item could be routed to a defect area.

    These are just a few ideas to help you understand how you could use the Medical Diagnosis pattern as a framework for your application.

    QUICKLINKS

    CONTRIBUTE

    About the Medical Diagnosis pattern

    Background

    This validated pattern is based on a demo implementation of an automated data pipeline for chest X-ray analysis that was previously developed by Red Hat. You can find the original demonstration here. It was developed for the US Department of Veteran Affairs.

    This validated pattern includes the same functionality as the original demonstration. The difference is that this solution uses the GitOps framework to deploy the pattern including Operators, creation of namespaces, and cluster configuration. Using GitOps provides an efficient means of implementing continuous deployment.

    Workflow

    • Ingest chest X-rays from a simulated X-ray machine and puts them into an objectStore based on Ceph.

    • The objectStore sends a notification to a Kafka topic.

    • A KNative Eventing listener to the topic triggers a KNative Serving function.

    • An ML-trained model running in a container makes a risk assessment of Pneumonia for incoming images.

    • A Grafana dashboard displays the pipeline in real time, along with images incoming, processed, anonymized, and full metrics collected from Prometheus.

    This pipeline is showcased in this video.

    dashboard

    About the solution elements

    The solution aids the understanding of the following:

    • How to use a GitOps approach to keep in control of configuration and operations.

    • How to deploy AI/ML technologies for medical diagnosis using GitOps.

    The Medical Diagnosis pattern uses the following products and technologies:

    • Red Hat OpenShift Container Platform for container orchestration

    • Red Hat OpenShift GitOps, a GitOps continuous delivery (CD) solution

    • Red Hat AMQ, an event streaming platform based on the Apache Kafka

    • Red Hat OpenShift Serverless for event-driven applications

    • Red Hat OpenShift Data Foundation for cloud native storage capabilities

    • Grafana Operator to manage and share Grafana dashboards, data sources, and so on

    • S3 storage

    About the architecture

    Presently, the Medical Diagnosis pattern does not have an edge component. Edge deployment capabilities are planned as part of the pattern architecture for a future release.

    edge medical diagnosis marketing slide

    Components are running on OpenShift either at the data center, at the medical facility, or public cloud running OpenShift.

    About the physical schema

    The following diagram shows the components that are deployed with the various networks that connect them.

    physical network

    The following diagram shows the components that are deployed with the the data flows and API calls between them.

    physical dataflow

    Recorded demo

    Demo</a>

    Presentation

    View presentation for the Medical Diagnosis Validated Pattern here

    Demo Script

    Use this demo script to successfully complete the Medical Diagnosis pattern demo here

    Next steps

    QUICKLINKS

    About the Medical Diagnosis pattern

    Background

    This validated pattern is based on a demo implementation of an automated data pipeline for chest X-ray analysis that was previously developed by Red Hat. You can find the original demonstration here. It was developed for the US Department of Veteran Affairs.

    This validated pattern includes the same functionality as the original demonstration. The difference is that this solution uses the GitOps framework to deploy the pattern including Operators, creation of namespaces, and cluster configuration. Using GitOps provides an efficient means of implementing continuous deployment.

    Workflow

    • Ingest chest X-rays from a simulated X-ray machine and puts them into an objectStore based on Ceph.

    • The objectStore sends a notification to a Kafka topic.

    • A KNative Eventing listener to the topic triggers a KNative Serving function.

    • An ML-trained model running in a container makes a risk assessment of Pneumonia for incoming images.

    • A Grafana dashboard displays the pipeline in real time, along with images incoming, processed, anonymized, and full metrics collected from Prometheus.

    This pipeline is showcased in this video.

    dashboard

    About the solution elements

    The solution aids the understanding of the following:

    • How to use a GitOps approach to keep in control of configuration and operations.

    • How to deploy AI/ML technologies for medical diagnosis using GitOps.

    The Medical Diagnosis pattern uses the following products and technologies:

    • Red Hat OpenShift Container Platform for container orchestration

    • Red Hat OpenShift GitOps, a GitOps continuous delivery (CD) solution

    • Red Hat AMQ, an event streaming platform based on the Apache Kafka

    • Red Hat OpenShift Serverless for event-driven applications

    • Red Hat OpenShift Data Foundation for cloud native storage capabilities

    • Grafana Operator to manage and share Grafana dashboards, data sources, and so on

    • S3 storage

    About the architecture

    Presently, the Medical Diagnosis pattern does not have an edge component. Edge deployment capabilities are planned as part of the pattern architecture for a future release.

    edge medical diagnosis marketing slide

    Components are running on OpenShift either at the data center, at the medical facility, or public cloud running OpenShift.

    About the physical schema

    The following diagram shows the components that are deployed with the various networks that connect them.

    physical network

    The following diagram shows the components that are deployed with the the data flows and API calls between them.

    physical dataflow

    Recorded demo

    Demo</a>

    Presentation

    View presentation for the Medical Diagnosis Validated Pattern here

    Demo Script

    Use this demo script to successfully complete the Medical Diagnosis pattern demo here

    Next steps

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    About the MLOps Fraud Detection

    MLOps Credit Card Fraud Detection use case

    • Build and train models in RHODS to detect credit card fraud

    • Track and store those models with MLFlow

    • Serve a model stored in MLFlow using RHODS Model Serving (or MLFlow serving)

    • Deploy a model application in OpenShift that runs sends data to the served model and displays the prediction

    Background

    AI technology is already transforming the financial services industry. AI models can be used to make rapid inferences that benefit the FS institute and its customers. This pattern deploys a AI model to detect fraud on crdit card transactions

    About the solution

    The model is built on a Credit Card Fraud Detection model, which predicts if a credit card usage is fraudulent or not depending on a few parameters such as: distance from home and last transaction, purchase price compared to median, if it’s from a retailer that already has been purchased from before, if the PIN number is used and if it’s an online order or not.

    Technology Highlights:

    • Event-Driven Architecture

    • Data Science on OpenShift

    • Model registry using MLFlow

    Solution Discussion

    This architecture pattern demonstrates four strengths:

    • Real-Time Processing: Analyze transactions in real-time, quickly identifying and flagging potentially fraudulent activities. This speed is crucial in preventing unauthorized transactions before they are completed.

    • Pattern Recognition: Detect patterns and anomalies in data and learn from historical transaction data to identify typical spending patterns of a cardholder and flag transactions that deviate from these patterns.

    • Cost Efficiency: By automating the detection process, AI reduces the need for extensive manual review of transactions, which can be time-consuming and costly.

    • Flexibility and Agility: An cloud native architecture that supports the use of microservices, containers, and serverless computing, allowing for more flexible and agile development and deployment of AI models. This means faster iteration and deployment of new fraud detection algorithms.

    Demo Video

    Overview of the solution for credit card fraud detection

    Overview of the Architecture

    Description of each component:

    • Data Set: The data set contains the data used for training and evaluating the model we will build in this demo.

    • RHODS Notebook: We will build and train the model using a Jupyter Notebook running in RHODS.

    • MLFlow Experiment tracking: We use MLFlow to track the parameters and metrics (such as accuracy, loss, etc) of a model training run. These runs can be grouped under different "experiments", making it easy to keep track of the runs.

    • MLFlow Model registry: As we track the experiment we also store the trained model through MLFlow so we can easily version it and assign a stage to it (for example Staging, Production, Archive).

    • S3 (ODF): This is where the models are stored and what the MLFlow model registry interfaces with. We use ODF (OpenShift Data Foundation) according to the MLFlow guide, but it can be replaced with another solution.

    • RHODS Model Serving: We recommend using RHODS Model Serving for serving the model. It’s based on ModelMesh and allows us to easily send requests to an endpoint for getting predictions.

    • Application interface: This is the interface used to run predictions with the model. In our case, we will build a visual interface (interactive app) using Gradio and let it load the model from the MLFlow model registry.

    mfd reference architecture
    Figure 1. Overview of the solution reference architecture

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Container Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift AI

    Red Hat® OpenShift® AI is an AI-focused portfolio that provides tools to train, tune, serve, monitor, and manage AI/ML experiments and models on Red Hat OpenShift. Bring data scientists, developers, and IT together on a unified platform to deliver AI-enabled applications faster.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat AMQ Streams

    Red Hat AMQ streams is a massively scalable, distributed, and high-performance data streaming platform based on the Apache Kafka project. It offers a distributed backbone that allows microservices and other applications to share data with high throughput and low latency. Red Hat AMQ Streams is available in the Red Hat AMQ product.

    Hashicorp Vault (community)

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    MLFlow Model Registry (community)

    A centralized model store, set of APIs, and UI, to collaboratively manage the full lifecycle of an MLflow Model. It provides model lineage (which MLflow experiment and run produced the model), model versioning, model aliasing, model tagging, and annotations.

    Other

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Next steps

    QUICKLINKS

    About the MLOps Fraud Detection

    MLOps Credit Card Fraud Detection use case

    • Build and train models in RHODS to detect credit card fraud

    • Track and store those models with MLFlow

    • Serve a model stored in MLFlow using RHODS Model Serving (or MLFlow serving)

    • Deploy a model application in OpenShift that runs sends data to the served model and displays the prediction

    Background

    AI technology is already transforming the financial services industry. AI models can be used to make rapid inferences that benefit the FS institute and its customers. This pattern deploys a AI model to detect fraud on crdit card transactions

    About the solution

    The model is built on a Credit Card Fraud Detection model, which predicts if a credit card usage is fraudulent or not depending on a few parameters such as: distance from home and last transaction, purchase price compared to median, if it’s from a retailer that already has been purchased from before, if the PIN number is used and if it’s an online order or not.

    Technology Highlights:

    • Event-Driven Architecture

    • Data Science on OpenShift

    • Model registry using MLFlow

    Solution Discussion

    This architecture pattern demonstrates four strengths:

    • Real-Time Processing: Analyze transactions in real-time, quickly identifying and flagging potentially fraudulent activities. This speed is crucial in preventing unauthorized transactions before they are completed.

    • Pattern Recognition: Detect patterns and anomalies in data and learn from historical transaction data to identify typical spending patterns of a cardholder and flag transactions that deviate from these patterns.

    • Cost Efficiency: By automating the detection process, AI reduces the need for extensive manual review of transactions, which can be time-consuming and costly.

    • Flexibility and Agility: An cloud native architecture that supports the use of microservices, containers, and serverless computing, allowing for more flexible and agile development and deployment of AI models. This means faster iteration and deployment of new fraud detection algorithms.

    Demo Video

    Overview of the solution for credit card fraud detection

    Overview of the Architecture

    Description of each component:

    • Data Set: The data set contains the data used for training and evaluating the model we will build in this demo.

    • RHODS Notebook: We will build and train the model using a Jupyter Notebook running in RHODS.

    • MLFlow Experiment tracking: We use MLFlow to track the parameters and metrics (such as accuracy, loss, etc) of a model training run. These runs can be grouped under different "experiments", making it easy to keep track of the runs.

    • MLFlow Model registry: As we track the experiment we also store the trained model through MLFlow so we can easily version it and assign a stage to it (for example Staging, Production, Archive).

    • S3 (ODF): This is where the models are stored and what the MLFlow model registry interfaces with. We use ODF (OpenShift Data Foundation) according to the MLFlow guide, but it can be replaced with another solution.

    • RHODS Model Serving: We recommend using RHODS Model Serving for serving the model. It’s based on ModelMesh and allows us to easily send requests to an endpoint for getting predictions.

    • Application interface: This is the interface used to run predictions with the model. In our case, we will build a visual interface (interactive app) using Gradio and let it load the model from the MLFlow model registry.

    mfd reference architecture
    Figure 1. Overview of the solution reference architecture

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Container Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift AI

    Red Hat® OpenShift® AI is an AI-focused portfolio that provides tools to train, tune, serve, monitor, and manage AI/ML experiments and models on Red Hat OpenShift. Bring data scientists, developers, and IT together on a unified platform to deliver AI-enabled applications faster.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat AMQ Streams

    Red Hat AMQ streams is a massively scalable, distributed, and high-performance data streaming platform based on the Apache Kafka project. It offers a distributed backbone that allows microservices and other applications to share data with high throughput and low latency. Red Hat AMQ Streams is available in the Red Hat AMQ product.

    Hashicorp Vault (community)

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    MLFlow Model Registry (community)

    A centralized model store, set of APIs, and UI, to collaboratively manage the full lifecycle of an MLflow Model. It provides model lineage (which MLflow experiment and run produced the model), model versioning, model aliasing, model tagging, and annotations.

    Other

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Next steps

    QUICKLINKS

    CONTRIBUTE

    The most important ArgoCD instance to examine at this point is mlops-fraud-detection-hub. This is where all the applications for the pattern can be tracked.

  • Check all applications are synchronised. There are thirteen different ArgoCD "applications" deployed as part of this pattern.

    • Next Steps

    QUICKLINKS

    The most important ArgoCD instance to examine at this point is mlops-fraud-detection-hub. This is where all the applications for the pattern can be tracked.

  • Check all applications are synchronised. There are thirteen different ArgoCD "applications" deployed as part of this pattern.

    • Next Steps

    QUICKLINKS

    CONTRIBUTE

    After pressing the "Add data connection" button. Here is an example of how to fill the form out:

    mfd data connection details
    Figure 6. Add data connection details

    Next, configure a model server, which will serve our models.

    mfd configure model server
    Figure 7. Configure model server

    Add Model Server

    • Model server name = credit card fraud

    • Serving runtime = OpenVINO Model Server

    • Model server replicas = 1

    • Model server size = Small

    • Check the Make deployed models available through an external route box external access model is required. This is not needed in this dmeo.

    Deploy Model

    Finally, we will deply the model, to do that, press the "Deploy model" button which is in the same place that "Configure Model" was before. We need to fill out a few settings here:

    • Name: credit card fraud

    • Model framework: onnx-1 - Since we saved the model as ONNX in the model training section

    • Model location:

      • Name: mlflow-connection

      • *Folder path: This is the full path we can see in the MLFlow interface from the end of the previous section. In my case it’s 1/b86481027f9b4b568c9efa3adc01929f/artifacts/models.Beware that we only need the last part, which looks something like: 1/…​./artifacts/models

    Note: use models not model. There are 2 folder in MLflow that might cause confusion.

    mfd mlflow model path
    Figure 8. Review MLFlow Model Path

    Note the models in highlighted folder.

    mfd deployment model options
    Figure 9. Deploy the Model

    Press Deploy and wait for it to complete. A green checkmark will be displayed when done. The status is displayed on the line for model "credit fraud" to the right.

    Access the model application

    The model application is a visual interface for interacting with the model. You can use it to send data to the model and get a prediction of whether a transaction is fraudulent or not. It is deployed in inferencing-app project. You can access the model application from the images::grid.png short-cut on top right in openshift console: "Inferencing App"

    Check the INFERENCE_ENDPOINT env variable value.

    Go to https://<your-uri>/ns/inferencing-app/deployments/credit-fraud-detection-demo/environment. Make sure correct INFERENCE_ENDPOINT value is set. In this case it is http://modelmesh-serving.credit-fraud-model:8008/v2/models/credit-card-fraud/infer

    You can get this value from

    Congratulations, you now have an application running your AI model!

    Try entering a few values and see if it predicts it as a credit fraud or not. You can select one of the examples at the bottom of the application page.

    Next Steps

    Help & Feedback -Report Bugs

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    About the Intel AMX accelerated Multicloud GitOps pattern with Openshift AI

    Use case

    • Use a GitOps approach to manage hybrid and multi-cloud deployments across both public and private clouds.

    • Enable cross-cluster governance and application lifecycle management.

    • Accelerate AI operations and improve computational performance by using Intel Advanced Matrix Extensions together with Openshift AI operator.

    • Securely manage secrets across the deployment.

      Based on the requirements of a specific implementation, certain details might differ. However, all validated patterns that are based on a portfolio architecture, generalize one or more successful deployments of a use case.

    Background

    Organizations are aiming to develop, deploy, and operate applications on an open hybrid cloud in a stable, simple, and secure way. This hybrid strategy includes multi-cloud deployments where workloads might be running on multiple clusters and on multiple clouds, private or public. This strategy requires an infrastructure-as-code approach: GitOps. GitOps uses Git repositories as a single source of truth to deliver infrastructure-as-code. Submitted code checks the continuous integration (CI) process, while the continuous delivery (CD) process checks and applies requirements for things like security, infrastructure-as-code, or any other boundaries set for the application framework. All changes to code are tracked, making updates easy while also providing version control should a rollback be needed. Moreover, organizations are looking for solutions that increase efficiency and at the same time reduce costs, what is possible using 5th Generation Intel Xeon Scalable Processors with a new build-in accelerator - Intel Advanced Matrix Extensions.

    About the solution

    This architecture covers hybrid and multi-cloud management with GitOps as shown in following figure. At a high level this requires a management hub, for DevOps and GitOps, and infrastructure that extends to one or more managed clusters running on private or public clouds. The automated infrastructure-as-code approach can manage the versioning of components and deploy according to the infrastructure-as-code configuration.

    Benefits of Hybrid Multicloud management with GitOps:

    • Unify management across cloud environments.

    • Dynamic infrastructure security.

    • Infrastructural continuous delivery best practices.

    Multicloud Architecture
    Figure 1. Overview of the solution including the business drivers, management hub, and the clusters under management

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Advanced Cluster Management for Kubernetes

    Controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Node Feature Discovery Operator

    Manages the detection of hardware features and configuration in an OpenShift Container Platform cluster by labeling the nodes with hardware-specific information. NFD labels the host with node-specific attributes, such as PCI cards, kernel, operating system version, and so on.

    Red Hat Openshift AI

    A flexible, scalable MLOps platform with tools to build, deploy, and manage AI-enabled applications. OpenShift AI (previously called Red Hat OpenShift Data Science) supports the full lifecycle of AI/ML experiments and models, on-premise and in the public cloud.

    OpenVINO Toolkit Operator

    The Operator includes OpenVINO™ Notebooks for development of AI optimized workloads and OpenVINO™ Model Server for deployment that enables AI inference execution at scale, and exposes AI models via gRPC and REST API interfaces.

    Intel® Advanced Matrix Extensions

    A new built-in accelerator that improves the performance of deep-learning training and inference on the CPU and is ideal for workloads like natural-language processing, recommendation systems and image recognition.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Intel AMX accelerated Multicloud GitOps pattern with Openshift AI

    The basic Multicloud GitOps pattern has been extended to highlight the 5th Generation Intel Xeon Scalable Processors capabilities, offering developers a streamlined pathway to accelerate their workloads through the integration of cutting-edge Intel AMX, providing efficiency and performance optimization in AI workloads.

    The basic pattern has been extended with two components: Openshift AI and OpenVINO Toolkit Operator.

    • Openshift AI, serves as a robust AI/ML platform for the creation of AI-driven applications and provides a collaborative environment for data scientists and developers that helps to move easily from experiment to production. It offers Jupyter application with selection of notebook servers, equipped with pre-configured environments and necessary support and optimizations (such as CUDA, PyTorch, Tensorflow, HabanaAI, etc.).

    • OpenVINO Toolkit Operator manages OpenVINO components within Openshift environment. First one, OpenVINO™ Model Server (OVMS) is a scalable, high-performance solution for serving machine learning models optimized for Intel® architectures. The other component, that was used in the proposed pattern is Notebook resource. This element integrates Jupyter from OpenShift AI with a container image that includes developer tools from the OpenVINO toolkit. It also enables selecting a defined image OpenVINO™ Toolkit from the Jupyter Spawner choice list.

    BERT-Large model is used as an example of AI workload using Intel AMX in the pattern. The BERT-Large inference is running in the Jupyter Notebook that uses OpenVINO optimizations.

    As a side note, BERT_Large is a wide known model used by various enterprise Natural Language Processing workloads. Intel has demonstrated, that 5th Generation Intel Xeon Scalable Processors perform up to 1.49 times better in NLP flows on Red Hat OpenShift vs. prior generation of processors- read more: -Level Up Your NLP applications on Red Hat OpenShift and 5th Gen

    Next steps

    QUICKLINKS

    Next steps

    QUICKLINKS

    CONTRIBUTE

    Introduction

    The Intel AMX accelerated Multicloud GitOps pattern with Openshift AI provides developers and data scientists with Red Hat OpenShift AI product that is fully configured and ready to go. It also helps to boost their workloads by integrating AMX, which ensures efficiency and performance optimization for AI workloads.

    AMX demo

    Using 5th Generation Intel Xeon Scalable Processors the kernel detects Intel® AMX at run-time, there is no need to enable and configure it additionally to improve performance. However, we need Intel optimized tools and frameworks to take advantage of AMX acceleration, such as OpenVINO Toolkit.

    Before proceeding with this demo steps, please make sure you have your pattern deployed with Getting started.

    1. Verify if the openvino-notebooks-v2022.3-1 build is completed under Builds > Builds. Build might take some time and before it is finished it won’t be accessible from Openshift AI console.

    2. Open OpenShift AI dashboard and go to Applications > Enabled window.

    3. Open Jupyter by clicking Launch application.

    4. Choose OpenVINO™ Toolkit v2022.3 notebook image with X Large container size and start the notebook server. Server launching will take several minutes. Once it is ready, go to access notebook server.

    5. On the Jupyter Launcher window choose Notebook with Python 3 (ipykernel).

    6. Download BERT-large example, that uses AMX accelerator, by typing in the opened notebook:

      !wget https://raw.githubusercontent.com/validatedpatterns-sandbox/amx-accelerated-rhoai-multicloud-gitops/main/scripts/BERT.ipynb -O BERT.ipynb

    7. On the left-hand side menu the BERT.ipynb script should show up. Open it and run instructions one by one with play button or with Ctr+Enter from keyboard.

    All necessary tools like Model Downloader and Benchmark Python Tool are built in and ready to use.

    Description of BERT.ipynb

    In case of issues with downloading the script, you can copy the following steps into your notebook and run.

    %env ONEDNN_VERBOSE=1

    Download the BERT-Large model compatible with FP32&BF16 precision bert-large-uncased-whole-word-masking-squad-0001:

    !omz_downloader --name bert-large-uncased-whole-word-masking-squad-0001

    Go to the directory with downloaded model and run the benchmark tool with parameter infer_precision bf16 to use BF16 precision:

    %cd /opt/app-root/src/intel/bert-large-uncased-whole-word-masking-squad-0001/FP32/
 
-!benchmark_app -m bert-large-uncased-whole-word-masking-squad-0001.xml -infer_precision bf16

    In ONEDNN verbose you should see avx_512_core_amx entry, what confirms that AMX instructions are being used.

    Logs from amx-app pod
    Figure 1. BERT inference log

    QUICKLINKS

    In ONEDNN verbose you should see avx_512_core_amx entry, what confirms that AMX instructions are being used.

    Logs from amx-app pod
    Figure 1. BERT inference log

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    CONTRIBUTE

    Deploying the Intel AMX accelerated Multicloud GitOps pattern with Openshift AI

    Prerequisites

    • An OpenShift cluster

      • To create an OpenShift cluster, go to the Red Hat Hybrid Cloud console and select Services -> Containers -> Create cluster.

      • The cluster must have a dynamic StorageClass to provision PersistentVolumes.

    • The cluster must have worker nodes with Intel AMX feature enabled, so the 5th Generation of Intel Xeon Processors is highly recommended

    • Optional: A second OpenShift cluster for multicloud demonstration.

    • Install the tooling dependencies.

    The use of this pattern depends on having at least one running Red Hat OpenShift cluster. However, consider creating a cluster for deploying the GitOps management hub assets and a separate cluster for the managed cluster.

    If you do not have a running Red Hat OpenShift cluster, you can start one on a -public or private cloud by using Red Hat Hybrid Cloud Console.

    Procedure

    1. Fork the amx-accelerated-rhoai-multicloud-gitops repository on GitHub.

    2. Clone the forked copy of this repository.

      git clone git@github.com:your-username/amx-accelerated-rhoai-multicloud-gitops.git

    3. Create a local copy of the secret values file that can safely include credentials for the config-demo application and edit it if you want to customize the secret. If not, the framework generates a random password.

      cp values-secret.yaml.template ~/values-secret-multicloud-gitops.yaml

      Do not commit this file. You do not want to push personal credentials to GitHub.

    4. (Optional) You may customize the deployment for your cluster depending on your needs by editing values-global.yaml and values-hub.yaml. To do this run the following commands:

      git checkout -b my-branch
      vi values-global.yaml
      git add values-global.yaml
      git commit values-global.yaml
      git push origin my-branch

    5. Deploy the pattern by running ./pattern.sh make install or by using the Validated Patterns Operator - both methods are described below.

    Deploying the cluster by using the pattern.sh file

    To deploy the cluster by using the pattern.sh file, complete the following steps:

    1. Login to your cluster by running the following command:

      oc login

      Optional: Set the KUBECONFIG variable for the kubeconfig file path:

      export KUBECONFIG=~/<path_to_kubeconfig>

    2. Deploy the pattern to your cluster. Run the following command:

      ./pattern.sh make install

    Verify that the Operators have been installed.

    1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

    2. Check that the following Operators are installed with Succeeded status (Figure 1):

      • Advanced Cluster Management for Kubernetes

      • multicluster engine for Kubernetes

      • Node Feature Discovery Operator

      • Red Hat Openshift GitOps

      • Validated Patterns Operator

      • OpenVINO Toolkit Operator

      • Red Hat Openshift AI

        AMX accelerated Multicloud GitOps Hub with Openshift AI installed operators
        Figure 1. List of Installed Operators for AMX accelerated Multicloud GitOps with Openshift AI pattern

    Deploying the cluster by using the Validated Patterns Operator

    To install the Validated Patterns Operator:

    1. Log in to the Openshift Container Platform web console and select Operators > OperatorHub.

    2. Search for Validated Patterns Operator, open it and click Install.

      Install Validated Patterns Operator
      Figure 2. Install Validated Patterns Operator step 1

    3. Choose default settings for the installation mode, namespaces and update strategy and confirm it by clicking Install.

      Install Validated Patterns Operator
      Figure 3. Install Validated Patterns Operator step 2

    4. Select Operators > Installed Operators.

    5. Ensure that Validated Patterns Operator is listed in the openshift-operators project with a status Succeeded.

    Create Intel AMX accelerated Multicloud GitOps pattern with Openshift AI

    After a successful installation, open Validated Patterns Operator page. Next, go to Pattern tab and click Create Pattern.

    1. Set the Name field to multicloud-gitops-amx-rhoai and Cluster Group Name to hub, Values must be the same as in the values-global.yaml file (Figure 3).

    2. As a Git Config > Target Repo value, paste the link to your fork. Under Git Config > Target Revision write the name of your branch (Figure 3).

    3. Click Create button to create the pattern.

      Create pattern AMX accelerated Multicloud GitOps with Openshift AI
      Figure 4. Create Pattern Form

    Verify that the rest of Operators have been installed:

    1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

    2. Check that the following Operators are installed with Succeeded status (Figure 1):

      • Advanced Cluster Management for Kubernetes

      • multicluster engine for Kubernetes

      • Node Feature Discovery Operator

      • Red Hat Openshift GitOps

      • OpenVINO Toolkit Operator

      • Red Hat Openshift AI

    Verification

    Go to the Hub ArgoCD and verify that all applications are synchronized. The URL can be found in Openshift Container Platform web console under Networking > Routes for the project multicloud-gitops-amx-rhoai-hub or use command:

    oc -n multicloud-gitops-amx-rhoai-hub get route hub-gitops-server -ojsonpath='{.spec.host}'

    All applications should be Healthy and Synced:

    Multicloud GitOps Hub with rhods and openvino toolkit
    Figure 5. ArgoCD panel with all applications

    As part of this pattern, HashiCorp Vault has been installed. Refer to the section on Vault.

    Start your workload

    Now, you are ready to use installed pattern.

    Next steps

    After the management hub is set up and works correctly, attach one or more managed clusters to the architecture.

    For instructions on deploying the edge, refer to Attach a managed cluster (edge) to the management hub.

    QUICKLINKS

    Procedure

    1. Fork the amx-accelerated-rhoai-multicloud-gitops repository on GitHub.

    2. Clone the forked copy of this repository.

      git clone git@github.com:your-username/amx-accelerated-rhoai-multicloud-gitops.git

    3. Create a local copy of the secret values file that can safely include credentials for the config-demo application and edit it if you want to customize the secret. If not, the framework generates a random password.

      cp values-secret.yaml.template ~/values-secret-multicloud-gitops.yaml

      Do not commit this file. You do not want to push personal credentials to GitHub.

    4. (Optional) You may customize the deployment for your cluster depending on your needs by editing values-global.yaml and values-hub.yaml. To do this run the following commands:

      git checkout -b my-branch
      vi values-global.yaml
      git add values-global.yaml
      git commit values-global.yaml
      git push origin my-branch

    5. Deploy the pattern by running ./pattern.sh make install or by using the Validated Patterns Operator - both methods are described below.

    Deploying the cluster by using the pattern.sh file

    To deploy the cluster by using the pattern.sh file, complete the following steps:

    1. Login to your cluster by running the following command:

      oc login

      Optional: Set the KUBECONFIG variable for the kubeconfig file path:

      export KUBECONFIG=~/<path_to_kubeconfig>

    2. Deploy the pattern to your cluster. Run the following command:

      ./pattern.sh make install

    Verify that the Operators have been installed.

    1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

    2. Check that the following Operators are installed with Succeeded status (Figure 1):

      • Advanced Cluster Management for Kubernetes

      • multicluster engine for Kubernetes

      • Node Feature Discovery Operator

      • Red Hat Openshift GitOps

      • Validated Patterns Operator

      • OpenVINO Toolkit Operator

      • Red Hat Openshift AI

        AMX accelerated Multicloud GitOps Hub with Openshift AI installed operators
        Figure 1. List of Installed Operators for AMX accelerated Multicloud GitOps with Openshift AI pattern

    Deploying the cluster by using the Validated Patterns Operator

    To install the Validated Patterns Operator:

    1. Log in to the Openshift Container Platform web console and select Operators > OperatorHub.

    2. Search for Validated Patterns Operator, open it and click Install.

      Install Validated Patterns Operator
      Figure 2. Install Validated Patterns Operator step 1

    3. Choose default settings for the installation mode, namespaces and update strategy and confirm it by clicking Install.

      Install Validated Patterns Operator
      Figure 3. Install Validated Patterns Operator step 2

    4. Select Operators > Installed Operators.

    5. Ensure that Validated Patterns Operator is listed in the openshift-operators project with a status Succeeded.

    Create Intel AMX accelerated Multicloud GitOps pattern with Openshift AI

    After a successful installation, open Validated Patterns Operator page. Next, go to Pattern tab and click Create Pattern.

    1. Set the Name field to multicloud-gitops-amx-rhoai and Cluster Group Name to hub, Values must be the same as in the values-global.yaml file (Figure 3).

    2. As a Git Config > Target Repo value, paste the link to your fork. Under Git Config > Target Revision write the name of your branch (Figure 3).

    3. Click Create button to create the pattern.

      Create pattern AMX accelerated Multicloud GitOps with Openshift AI
      Figure 4. Create Pattern Form

    Verify that the rest of Operators have been installed:

    1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

    2. Check that the following Operators are installed with Succeeded status (Figure 1):

      • Advanced Cluster Management for Kubernetes

      • multicluster engine for Kubernetes

      • Node Feature Discovery Operator

      • Red Hat Openshift GitOps

      • OpenVINO Toolkit Operator

      • Red Hat Openshift AI

    Verification

    Go to the Hub ArgoCD and verify that all applications are synchronized. The URL can be found in Openshift Container Platform web console under Networking > Routes for the project multicloud-gitops-amx-rhoai-hub or use command:

    oc -n multicloud-gitops-amx-rhoai-hub get route hub-gitops-server -ojsonpath='{.spec.host}'

    All applications should be Healthy and Synced:

    Multicloud GitOps Hub with rhods and openvino toolkit
    Figure 5. ArgoCD panel with all applications

    As part of this pattern, HashiCorp Vault has been installed. Refer to the section on Vault.

    Start your workload

    Now, you are ready to use installed pattern.

    Next steps

    After the management hub is set up and works correctly, attach one or more managed clusters to the architecture.

    For instructions on deploying the edge, refer to Attach a managed cluster (edge) to the management hub.

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Ensure that you commit the changes and push them to GitHub so that GitOps can fetch your changes and apply them.

    Deploying a managed cluster by using Red Hat Advanced Cluster Management

    Prerequisites

    • An OpenShift cluster

    • Red Hat Advanced Cluster Management (RHACM) web console to join the managed cluster to the management hub

      After RHACM is installed, a message regarding a Web console update is available" might be displayed. Follow the instructions and click the Refresh web console link.

    Procedure

    1. In the left navigation panel of web console, click local-cluster. Select All Clusters. The RHACM web console is displayed with Cluster* on the left navigation panel.

    2. On the Managed clusters tab, click Import cluster.

    3. On the Import an existing cluster page, enter the cluster name and choose KUBECONFIG as the "import mode". Add the tag clusterGroup=region-one. Click Import.

    Now that RHACM is no longer deploying the managed cluster applications everywhere, you must indicate that the new cluster has the managed cluster role.

    Optional: Deploying a managed cluster by using cm-cli tool

    Prerequisites
    Procedure

    1. Obtain the KUBECONFIG file from the managed cluster.

    2. Open a shell prompt and login into the management hub cluster by using either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Run the following command:

      cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-path_to_kubeconfig>
    Next steps

    Optional: Deploying a managed cluster by using the clusteradm tool

    Prerequisites
    Procedure

    1. To deploy an edge cluster, you must to get the token from the management hub cluster. Run the following command on the existing management hub or datacenter cluster:

      clusteradm get token

      The command generates a token and shows you the command to use on the managed cluster.

    2. Login to the managed cluster with either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. To request that the managed join the hub cluster, run the following command:

      clusteradm join --hub-token <token_from_clusteradm_get_token_command> <managed_cluster_name>

    4. Accept the join request on the hub cluster:

      clusteradm accept --clusters <managed_cluster_name>
    Next steps

    Designate the new cluster as a managed cluster site

    If you use the command line tools such as clusteradm or cm-cli, you must explicitly indicate that the imported cluster is part of a specific clusterGroup. Some examples of clusterGroup are factory, devel, or prod.

    To tag the cluster as clusterGroup=<managed-cluster-group>, complete the following steps.

    Procedure

    1. To find the new cluster, run the following command:

      oc get managedcluster.cluster.open-cluster-management.io

    2. To apply the label, run the following command:

      oc label managedcluster.cluster.open-cluster-management.io/YOURCLUSTER site=managed-cluster

    Verification

    Go to your managed cluster (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. It might take a while for the RHACM agent and agent-addons to launch. After that, the OpenShift GitOps Operator is installed. On successful installation, launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    QUICKLINKS

    Ensure that you commit the changes and push them to GitHub so that GitOps can fetch your changes and apply them.

    Deploying a managed cluster by using Red Hat Advanced Cluster Management

    Prerequisites

    • An OpenShift cluster

    • Red Hat Advanced Cluster Management (RHACM) web console to join the managed cluster to the management hub

      After RHACM is installed, a message regarding a Web console update is available" might be displayed. Follow the instructions and click the Refresh web console link.

    Procedure

    1. In the left navigation panel of web console, click local-cluster. Select All Clusters. The RHACM web console is displayed with Cluster* on the left navigation panel.

    2. On the Managed clusters tab, click Import cluster.

    3. On the Import an existing cluster page, enter the cluster name and choose KUBECONFIG as the "import mode". Add the tag clusterGroup=region-one. Click Import.

    Now that RHACM is no longer deploying the managed cluster applications everywhere, you must indicate that the new cluster has the managed cluster role.

    Optional: Deploying a managed cluster by using cm-cli tool

    Prerequisites
    Procedure

    1. Obtain the KUBECONFIG file from the managed cluster.

    2. Open a shell prompt and login into the management hub cluster by using either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Run the following command:

      cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-path_to_kubeconfig>
    Next steps

    Optional: Deploying a managed cluster by using the clusteradm tool

    Prerequisites
    Procedure

    1. To deploy an edge cluster, you must to get the token from the management hub cluster. Run the following command on the existing management hub or datacenter cluster:

      clusteradm get token

      The command generates a token and shows you the command to use on the managed cluster.

    2. Login to the managed cluster with either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. To request that the managed join the hub cluster, run the following command:

      clusteradm join --hub-token <token_from_clusteradm_get_token_command> <managed_cluster_name>

    4. Accept the join request on the hub cluster:

      clusteradm accept --clusters <managed_cluster_name>
    Next steps

    Designate the new cluster as a managed cluster site

    If you use the command line tools such as clusteradm or cm-cli, you must explicitly indicate that the imported cluster is part of a specific clusterGroup. Some examples of clusterGroup are factory, devel, or prod.

    To tag the cluster as clusterGroup=<managed-cluster-group>, complete the following steps.

    Procedure

    1. To find the new cluster, run the following command:

      oc get managedcluster.cluster.open-cluster-management.io

    2. To apply the label, run the following command:

      oc label managedcluster.cluster.open-cluster-management.io/YOURCLUSTER site=managed-cluster

    Verification

    Go to your managed cluster (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. It might take a while for the RHACM agent and agent-addons to launch. After that, the OpenShift GitOps Operator is installed. On successful installation, launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    QUICKLINKS

    CONTRIBUTE

    About the Intel AMX accelerated Multicloud GitOps pattern

    Use case

    • Use a GitOps approach to manage hybrid and multi-cloud deployments across both public and private clouds.

    • Enable cross-cluster governance and application lifecycle management.

    • Accelerate AI operations and improve computational performance by using Intel Advanced Matrix Extensions.

    • Securely manage secrets across the deployment.

      Based on the requirements of a specific implementation, certain details might differ. However, all validated patterns that are based on a portfolio architecture, generalize one or more successful deployments of a use case.

    Background

    Organizations are aiming to develop, deploy, and operate applications on an open hybrid cloud in a stable, simple, and secure way. This hybrid strategy includes multi-cloud deployments where workloads might be running on multiple clusters and on multiple clouds, private or public. This strategy requires an infrastructure-as-code approach: GitOps. GitOps uses Git repositories as a single source of truth to deliver infrastructure-as-code. Submitted code checks the continuous integration (CI) process, while the continuous delivery (CD) process checks and applies requirements for things like security, infrastructure-as-code, or any other boundaries set for the application framework. All changes to code are tracked, making updates easy while also providing version control should a rollback be needed. Moreover, organizations are looking for solutions that increase efficiency and at the same time reduce costs, what is possible using 5th Generation Intel Xeon Scalable Processors with a new build-in accelerator - Intel Advanced Matrix Extensions.

    About the solution

    This architecture covers hybrid and multi-cloud management with GitOps as shown in following figure. At a high level this requires a management hub, for DevOps and GitOps, and infrastructure that extends to one or more managed clusters running on private or public clouds. The automated infrastructure-as-code approach can manage the versioning of components and deploy according to the infrastructure-as-code configuration.

    Benefits of Hybrid Multicloud management with GitOps:

    • Unify management across cloud environments.

    • Dynamic infrastructure security.

    • Infrastructural continuous delivery best practices.

    Multicloud Architecture
    Figure 1. Overview of the solution including the business drivers, management hub, and the clusters under management

    In the following figure, logically, this solution can be viewed as being composed of an automation component, unified management including secrets management, and the clusters under management, all running on top of a user-chosen mixture of on-premise data centers and public clouds.

    Logical Architecture
    Figure 2. Logical diagram of Intel AMX accelerated hybrid multi-cloud management with GitOps

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Advanced Cluster Management for Kubernetes

    Controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Node Feature Discovery Operator

    Manages the detection of hardware features and configuration in an OpenShift Container Platform cluster by labeling the nodes with hardware-specific information. NFD labels the host with node-specific attributes, such as PCI cards, kernel, operating system version, and so on.

    Intel® Advanced Matrix Extensions

    A new built-in accelerator that improves the performance of deep-learning training and inference on the CPU and is ideal for workloads like natural-language processing, recommendation systems and image recognition.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Overview of the architectures

    The following figure provides a schematic diagram overview of the complete solution including both components and data flows.

    Physical Architecture
    Figure 3. Overview schematic diagram of the complete solution

    Subsequent schematic diagrams provide details on:

    • Bootstrapping the management hub (Figure 4)

    • Hybrid multi-cloud GitOps (Figure 5)

    • Dynamic security management (Figure 6)

    Bootstrapping the management hub

    The following figure provides a schematic diagram showing the setup of the management hub using Ansible playbooks.

    Schematic diagram of bootstrapping the management hub
    Figure 4. Schematic diagram of bootstrapping the management hub

    • Set up the OpenShift Container Platform that hosts the Management Hub. The OpenShift installation program provides flexible ways to install OpenShift. An Ansible playbook starts the installation with necessary configurations.

    • Ansible playbooks deploy and configure Red Hat Advanced Cluster Management for Kubernetes and, later, other supporting components such as external secrets management, on top of the provisioned OpenShift cluster.

    • Another Ansible playbook installs HashiCorp Vault, a Red Hat partner product chosen for this solution that can be used to manage secrets for OpenShift clusters.

    • An Ansible playbook configures and installs the Openshift GitOps operator on the hub cluster. This deploys the Openshift GitOps instance to enable continuous delivery.

    Hybrid Multicloud GitOps

    The following figure provides a schematic diagram showing remaining activities associated with setting up the management hub and clusters using Red Hat Advanced Cluster Management.

    Schematic diagram of hybrid multi-cloud management with GitOps
    Figure 5. Schematic diagram of hybrid multi-cloud management with GitOps

    • Manifest and configuration are set as code template in the form of a Kustomization YAML file. The file describes the desired end state of the managed cluster. When complete, the Kustomization YAML file is pushed into the source control management repository with a version assigned to each update.

    • OpenShift GitOps monitors the repository and detects changes in the repository.

    • OpenShift GitOps creates and updates the manifest by creating Kubernetes objects on top of Red Hat Advanced Cluster Management.

    • Red Hat Advanced Cluster Management provisions, updates, or deletes managed clusters and configuration according to the manifest. In the manifest, you can configure what cloud provider the cluster will be on, the name of the cluster, infrastructure node details and worker node. Governance policy can also be applied as well as provision an agent in the cluster as the bridge between the control center and the managed cluster.

    • OpenShift GitOps continuously monitors the code repository and the status of the clusters reported back to Red Hat Advanced Cluster Management. Any configuration drift or in case of any failure, OpenShift GitOps will automatically try to remediate by applying the manifest or by displaying alerts for manual intervention.

    Dynamic security management

    The following figure provides a schematic diagram showing how secrets are handled in this solution.

    Schematic showing the setup and use of external secrets management
    Figure 6. Schematic showing the setup and use of external secrets management

    • During setup, the token to securely access HashiCorp Vault is stored in Ansible Vault. It is encrypted to protect sensitive content.

    • Red Hat Advanced Cluster Management for Kubernetes acquires the token from Ansible Vault during install and distributes it among the clusters. As a result, you have centralized control over the managed clusters through RHACM.

    • To allow the cluster access to the external vault, you must set up the external secret management with Helm in this study. OpenShift Gitops is used to deploy the external secret object to a managed cluster.

    • External secret management fetches secrets from HashiCorp Vault by using the token that was generated in step 2 and constantly monitors for updates.

    • Secrets are created in each namespace, where applications can use them.

    Additional resources

    View and download all of the diagrams above from the Red Hat Portfolio Architecture open source tooling site.

    Intel AMX accelerated Multicloud GitOps pattern

    The basic Multicloud GitOps pattern has been extended to highlight the 5th Generation Intel Xeon Scalable Processors capabilities, offering developers a streamlined pathway to accelerate their workloads through the integration of cutting-edge Intel AMX, fostering efficiency and performance optimization in AI workloads.

    The basic pattern has been extended by the AI application named amx-app. It runs Deep Interest Evolution Network (DIEN) inference using the Intel-optimized TensorFlow and measures its accuracy. DIEN is a machine learning model used in the field of recommender systems, particularly in the domain of personalized content recommendation.

    Since the amx-app application must be running on the node with CPU supporting Intel AMX, Node Feature Discovery Operator (NFD) is deployed as a part of this pattern. -NFD manages the detection of hardware features and configuration in an OpenShift Container Platform cluster. It labels the nodes with hardware-specific information. The kernel detects Intel AMX at run-time, so there is no need to enable and configure it separately.

    A deployment of amx-app was created based on instructions from Model Zoo for Intel® Architecture repository - TF DIEN inference and uses intel/recommendation:tf-spr-dien-inference image.

    An amx-app use persistent volume claim to download and prepare dataset. When the dataset is ready, an application runs and measures the inference accuracy. By enabling ONEDNN verbose all the compiled instructions are shown in the logs. The appearance of the avx512_core_amx_bf16 flag confirms that Intel AMX is used.

    Logs from amx-app pod
    Figure 7. Logs from amx-app pod

    Next steps

    QUICKLINKS

    A deployment of amx-app was created based on instructions from Model Zoo for Intel® Architecture repository - TF DIEN inference and uses intel/recommendation:tf-spr-dien-inference image.

    An amx-app use persistent volume claim to download and prepare dataset. When the dataset is ready, an application runs and measures the inference accuracy. By enabling ONEDNN verbose all the compiled instructions are shown in the logs. The appearance of the avx512_core_amx_bf16 flag confirms that Intel AMX is used.

    Logs from amx-app pod
    Figure 7. Logs from amx-app pod

    Next steps

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    CONTRIBUTE

    Ensure that you commit the changes and push them to GitHub so that GitOps can fetch your changes and apply them.

    Deploying a managed cluster by using Red Hat Advanced Cluster Management

    Prerequisites

    • An OpenShift cluster

    • Red Hat Advanced Cluster Management (RHACM) web console to join the managed cluster to the management hub

      After RHACM is installed, a message regarding a Web console update is available" might be displayed. Follow the instructions and click the Refresh web console link.

    Procedure

    1. In the left navigation panel of web console, click local-cluster. Select All Clusters. The RHACM web console is displayed with Cluster* on the left navigation panel.

    2. On the Managed clusters tab, click Import cluster.

    3. On the Import an existing cluster page, enter the cluster name and choose KUBECONFIG as the "import mode". Add the tag clusterGroup=region-one. Click Import.

    Now that RHACM is no longer deploying the managed cluster applications everywhere, you must indicate that the new cluster has the managed cluster role.

    Optional: Deploying a managed cluster by using cm-cli tool

    Prerequisites
    Procedure

    1. Obtain the KUBECONFIG file from the managed cluster.

    2. Open a shell prompt and login into the management hub cluster by using either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Run the following command:

      cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-path_to_kubeconfig>
    Next steps

    Optional: Deploying a managed cluster by using the clusteradm tool

    Prerequisites
    Procedure

    1. To deploy an edge cluster, you must to get the token from the management hub cluster. Run the following command on the existing management hub or datacenter cluster:

      clusteradm get token

      The command generates a token and shows you the command to use on the managed cluster.

    2. Login to the managed cluster with either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. To request that the managed join the hub cluster, run the following command:

      clusteradm join --hub-token <token_from_clusteradm_get_token_command> <managed_cluster_name>

    4. Accept the join request on the hub cluster:

      clusteradm accept --clusters <managed_cluster_name>
    Next steps

    Designate the new cluster as a managed cluster site

    If you use the command line tools such as clusteradm or cm-cli, you must explicitly indicate that the imported cluster is part of a specific clusterGroup. Some examples of clusterGroup are factory, devel, or prod.

    To tag the cluster as clusterGroup=<managed-cluster-group>, complete the following steps.

    Procedure

    1. To find the new cluster, run the following command:

      oc get managedcluster.cluster.open-cluster-management.io

    2. To apply the label, run the following command:

      oc label managedcluster.cluster.open-cluster-management.io/YOURCLUSTER site=managed-cluster

    Verification

    Go to your managed cluster (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. It might take a while for the RHACM agent and agent-addons to launch. After that, the OpenShift GitOps Operator is installed. On successful installation, launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    QUICKLINKS

    Ensure that you commit the changes and push them to GitHub so that GitOps can fetch your changes and apply them.

    Deploying a managed cluster by using Red Hat Advanced Cluster Management

    Prerequisites

    • An OpenShift cluster

    • Red Hat Advanced Cluster Management (RHACM) web console to join the managed cluster to the management hub

      After RHACM is installed, a message regarding a Web console update is available" might be displayed. Follow the instructions and click the Refresh web console link.

    Procedure

    1. In the left navigation panel of web console, click local-cluster. Select All Clusters. The RHACM web console is displayed with Cluster* on the left navigation panel.

    2. On the Managed clusters tab, click Import cluster.

    3. On the Import an existing cluster page, enter the cluster name and choose KUBECONFIG as the "import mode". Add the tag clusterGroup=region-one. Click Import.

    Now that RHACM is no longer deploying the managed cluster applications everywhere, you must indicate that the new cluster has the managed cluster role.

    Optional: Deploying a managed cluster by using cm-cli tool

    Prerequisites
    Procedure

    1. Obtain the KUBECONFIG file from the managed cluster.

    2. Open a shell prompt and login into the management hub cluster by using either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Run the following command:

      cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-path_to_kubeconfig>
    Next steps

    Optional: Deploying a managed cluster by using the clusteradm tool

    Prerequisites
    Procedure

    1. To deploy an edge cluster, you must to get the token from the management hub cluster. Run the following command on the existing management hub or datacenter cluster:

      clusteradm get token

      The command generates a token and shows you the command to use on the managed cluster.

    2. Login to the managed cluster with either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. To request that the managed join the hub cluster, run the following command:

      clusteradm join --hub-token <token_from_clusteradm_get_token_command> <managed_cluster_name>

    4. Accept the join request on the hub cluster:

      clusteradm accept --clusters <managed_cluster_name>
    Next steps

    Designate the new cluster as a managed cluster site

    If you use the command line tools such as clusteradm or cm-cli, you must explicitly indicate that the imported cluster is part of a specific clusterGroup. Some examples of clusterGroup are factory, devel, or prod.

    To tag the cluster as clusterGroup=<managed-cluster-group>, complete the following steps.

    Procedure

    1. To find the new cluster, run the following command:

      oc get managedcluster.cluster.open-cluster-management.io

    2. To apply the label, run the following command:

      oc label managedcluster.cluster.open-cluster-management.io/YOURCLUSTER site=managed-cluster

    Verification

    Go to your managed cluster (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. It might take a while for the RHACM agent and agent-addons to launch. After that, the OpenShift GitOps Operator is installed. On successful installation, launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    QUICKLINKS

    CONTRIBUTE

    About OpenShift cluster sizing for the Multicloud GitOps Pattern

    Support matrix for Multicloud GitOps pattern

    The Multicloud GitOps pattern has been tested in the following Certified Cloud Providers.

    Certified Cloud Providers4.84.94.10
    Amazon Web Services:heavy_check_mark:
    Microsoft Azure:heavy_check_mark:
    Google Cloud Platform:heavy_check_mark:

    Minimum requirements for OpenShift Container Platform

    OpenShift 4 has the following minimum requirements for sizing of nodes:

    • Minimum 4 vCPU** (additional are strongly recommended)
    • Minimum 16 GB RAM** (additional memory is strongly recommended, especially if etcd is colocated on Control Planes)
    • Minimum 40 GB hard disk space for the file system containing /var/
    • Minimum 1 GB hard disk space for the file system containing /usr/local/bin/

    There is one application that comprises the Medical Diagnosis pattern. In addition, the Multicloud GitOps pattern also includes the Red Hat Advanced Cluster Management (RHACM) supporting operator that is installed by OpenShift GitOps using ArgoCD.

    Understanding Multicloud GitOps pattern components

    Here’s an inventory of what gets deployed by the Multicloud GitOps pattern on the Datacenter/Hub OpenShift cluster:

    NameKindNamespaceDescription
    multicloud-gitops-hubApplicationmulticloud-gitops-hubHub GitOps management
    Red Hat Advanced Cluster ManagementOperatoropen-cluster-managementAdvance Cluster Management
    Red Hat OpenShift GitOpsOperatoropenshift-operatorsOpenShift GitOps

    Multicloud GitOps pattern with OpenShift datacenter hub cluster size

    The Multicloud GitOps pattern has been tested with a defined set of specifically tested configurations that represent the most common combinations that Red Hat OpenShift Container Platform (OCP) customers are using or deploying for the x86_64 architecture.

    The datacenter hub OpenShift cluster is made up of the the following on the AWS deployment tested:

    Node TypeNumber of nodesCloud ProviderInstance Type
    Control Plane4Amazon Web Servicesm5.xlarge
    Worker3Amazon Web Servicesm5.xlarge

    The datacenter hub OpenShift cluster needs to be a bit bigger than the Factory/Edge clusters because this is where the developers will be running pipelines to build and deploy the Multicloud GitOps pattern on the cluster. The above cluster sizing is close to a minimum size for a Datacenter HUB cluster. In the next few sections we take some snapshots of the cluster utilization while the Multicloud GitOps pattern is running. Keep in mind that resources will have to be added as more developers are working building their applications.

    Datacenter cluster utilization

    Below is a snapshot of the OpenShift cluster utilization while running the Multicloud GitOps pattern:

    CPUMemoryFile SystemNetworkPod Count

    Multicloud GitOps pattern with OpenShift managed datacenter cluster size

    The OpenShift cluster is a standard datacenter deployment of 3 control plane nodes and 3 or more worker nodes.

    Node TypeNumber of nodesCloud ProviderInstance Type
    Control Plane/Worker3Google Cloudn1-standard-8
    Control Plane/Worker3Amazon Cloud Servicesm5.2xlarge
    Control Plane/Worker3Microsoft AzureStandard_D8s_v3

    Managed Datacenter Cluster Utilization

    GCP

    This is a snapshot of a Google Cloud managed data center cluster running the production Multicloud GitOps pattern.

    CPUMemoryFile SystemNetworkPod Count

    AWS

    This is a snapshot of a Amazon Web Services managed data center cluster running the production Multicloud GitOps pattern.

    CPUMemoryFile SystemNetworkPod Count

    Azure

    This is a snapshot of an Azure managed data center cluster running the production Multicloud GitOps pattern.

    CPUMemoryFile SystemNetworkPod Count

    AWS Instance Types

    The Multicloud GitOps pattern was tested with the highlighted AWS instances in bold. The OpenShift installer will let you know if the instance type meets the minimum requirements for a cluster.

    The message that the openshift installer will give you will be similar to this message

    INFO Credentials loaded from default AWS environment variables
 FATAL failed to fetch Metadata: failed to load asset "Install Config": [controlPlane.platform.aws.type: Invalid value: "m4.large": instance type does not meet minimum resource requirements of 4 vCPUs, controlPlane.platform.aws.type: Invalid value: "m4.large": instance type does not meet minimum resource requirements of 16384 MiB Memory]
-

    Below you can find a list of the AWS instance types that can be used to deploy the Multicloud GitOps pattern.

    Instance typeDefault vCPUsMemory (GiB)DatacenterFactory/Edge
    3x3 OCP Cluster3 Node OCP Cluster
    m4.xlarge416NN
    m4.2xlarge832YY
    m4.4xlarge1664YY
    m4.10xlarge40160YY
    m4.16xlarge64256YY
    m5.xlarge416YN
    m5.2xlarge832YY
    m5.4xlarge1664YY
    m5.8xlarge32128YY
    m5.12xlarge48192YY
    m5.16xlarge64256YY
    m5.24xlarge96384YY

    The OpenShift cluster is made of 4 Control Plane nodes and 3 Workers for the Datacenter and the Edge/managed data center cluster are made of 3 Control Plane and 3 Worker nodes. For the node sizes we used the m5.xlarge on AWS and this instance type met the minimum requirements to deploy the Multicloud GitOps pattern successfully on the Datacenter hub. On the managed data center cluster we used the m5.xlarge since the minimum cluster was comprised of 3 nodes. .

    To understand better what types of nodes you can use on other Cloud Providers we provide some of the details below.

    Azure Instance Types

    The Multicloud GitOps pattern was also deployed on Azure using the Standard_D8s_v3 VM size. Below is a table of different VM sizes available for Azure. Keep in mind that due to limited access to Azure we only used the Standard_D8s_v3 VM size.

    The OpenShift cluster is made of 3 Control Plane nodes and 3 Workers for the Datacenter cluster.

    The OpenShift cluster is made of 3 Control Plane nodes and 3 or more workers for each of the managed data center clusters.

    TypeSizesDescription
    General purposeB, Dsv3, Dv3, Dasv4, Dav4, DSv2, Dv2, Av2, DC, DCv2, Dv4, Dsv4, Ddv4, Ddsv4Balanced CPU-to-memory ratio. Ideal for testing and development, small to medium databases, and low to medium traffic web servers.
    Compute optimizedF, Fs, Fsv2, FXHigh CPU-to-memory ratio. Good for medium traffic web servers, network appliances, batch processes, and application servers.
    Memory optimizedEsv3, Ev3, Easv4, Eav4, Ev4, Esv4, Edv4, Edsv4, Mv2, M, DSv2, Dv2High memory-to-CPU ratio. Great for relational database servers, medium to large caches, and in-memory analytics.
    Storage optimizedLsv2High disk throughput and IO ideal for Big Data, SQL, NoSQL databases, data warehousing and large transactional databases.
    GPUNC, NCv2, NCv3, NCasT4_v3, ND, NDv2, NV, NVv3, NVv4Specialized virtual machines targeted for heavy graphic rendering and video editing, as well as model training and inferencing (ND) with deep learning. Available with single or multiple GPUs.
    High performance computeHB, HBv2, HBv3, HC, HOur fastest and most powerful CPU virtual machines with optional high-throughput network interfaces (RDMA).

    For more information please refer to the Azure VM Size Page.

    Google Cloud (GCP) Instance Types

    The Multicloud GitOps pattern was also deployed on GCP using the n1-standard-8 VM size. Below is a table of different VM sizes available for GCP. Keep in mind that due to limited access to GCP we only used the n1-standard-8 VM size.

    The OpenShift cluster is made of 3 Control Plane and 3 Workers for the Datacenter cluster.

    The OpenShift cluster is made of 3 Nodes combining Control Plane/Workers for the Edge/managed data center cluster.

    The following table provides VM recommendations for different workloads.

    | General purpose | Workload optimized

    Cost-optimizedBalancedScale-out optimizedMemory-optimizedCompute-optimizedAccelerator-optimized
    E2N2, N2D, N1T2DM2, M1C2A2
    Day-to-day computing at a lower costBalanced price/performance across a wide range of VM shapesBest performance/cost for scale-out workloadsUltra high-memory workloadsUltra high performance for compute-intensive workloadsOptimized for high performance computing workloads

    For more information please refer to the GCP VM Size Page.

    QUICKLINKS

    Below you can find a list of the AWS instance types that can be used to deploy the Multicloud GitOps pattern.

    Instance typeDefault vCPUsMemory (GiB)DatacenterFactory/Edge
    3x3 OCP Cluster3 Node OCP Cluster
    m4.xlarge416NN
    m4.2xlarge832YY
    m4.4xlarge1664YY
    m4.10xlarge40160YY
    m4.16xlarge64256YY
    m5.xlarge416YN
    m5.2xlarge832YY
    m5.4xlarge1664YY
    m5.8xlarge32128YY
    m5.12xlarge48192YY
    m5.16xlarge64256YY
    m5.24xlarge96384YY

    The OpenShift cluster is made of 4 Control Plane nodes and 3 Workers for the Datacenter and the Edge/managed data center cluster are made of 3 Control Plane and 3 Worker nodes. For the node sizes we used the m5.xlarge on AWS and this instance type met the minimum requirements to deploy the Multicloud GitOps pattern successfully on the Datacenter hub. On the managed data center cluster we used the m5.xlarge since the minimum cluster was comprised of 3 nodes. .

    To understand better what types of nodes you can use on other Cloud Providers we provide some of the details below.

    Azure Instance Types

    The Multicloud GitOps pattern was also deployed on Azure using the Standard_D8s_v3 VM size. Below is a table of different VM sizes available for Azure. Keep in mind that due to limited access to Azure we only used the Standard_D8s_v3 VM size.

    The OpenShift cluster is made of 3 Control Plane nodes and 3 Workers for the Datacenter cluster.

    The OpenShift cluster is made of 3 Control Plane nodes and 3 or more workers for each of the managed data center clusters.

    TypeSizesDescription
    General purposeB, Dsv3, Dv3, Dasv4, Dav4, DSv2, Dv2, Av2, DC, DCv2, Dv4, Dsv4, Ddv4, Ddsv4Balanced CPU-to-memory ratio. Ideal for testing and development, small to medium databases, and low to medium traffic web servers.
    Compute optimizedF, Fs, Fsv2, FXHigh CPU-to-memory ratio. Good for medium traffic web servers, network appliances, batch processes, and application servers.
    Memory optimizedEsv3, Ev3, Easv4, Eav4, Ev4, Esv4, Edv4, Edsv4, Mv2, M, DSv2, Dv2High memory-to-CPU ratio. Great for relational database servers, medium to large caches, and in-memory analytics.
    Storage optimizedLsv2High disk throughput and IO ideal for Big Data, SQL, NoSQL databases, data warehousing and large transactional databases.
    GPUNC, NCv2, NCv3, NCasT4_v3, ND, NDv2, NV, NVv3, NVv4Specialized virtual machines targeted for heavy graphic rendering and video editing, as well as model training and inferencing (ND) with deep learning. Available with single or multiple GPUs.
    High performance computeHB, HBv2, HBv3, HC, HOur fastest and most powerful CPU virtual machines with optional high-throughput network interfaces (RDMA).

    For more information please refer to the Azure VM Size Page.

    Google Cloud (GCP) Instance Types

    The Multicloud GitOps pattern was also deployed on GCP using the n1-standard-8 VM size. Below is a table of different VM sizes available for GCP. Keep in mind that due to limited access to GCP we only used the n1-standard-8 VM size.

    The OpenShift cluster is made of 3 Control Plane and 3 Workers for the Datacenter cluster.

    The OpenShift cluster is made of 3 Nodes combining Control Plane/Workers for the Edge/managed data center cluster.

    The following table provides VM recommendations for different workloads.

    | General purpose | Workload optimized

    Cost-optimizedBalancedScale-out optimizedMemory-optimizedCompute-optimizedAccelerator-optimized
    E2N2, N2D, N1T2DM2, M1C2A2
    Day-to-day computing at a lower costBalanced price/performance across a wide range of VM shapesBest performance/cost for scale-out workloadsUltra high-memory workloadsUltra high performance for compute-intensive workloadsOptimized for high performance computing workloads

    For more information please refer to the GCP VM Size Page.

    QUICKLINKS

    CONTRIBUTE

  • Verify that the Operators have been installed.

    1. To verify, in the *OpenShift Container Platform web console, navigate to Operators → Installed Operators page. 2.Check that the Operator is installed in the openshift-operators namespace and its status is Succeeded.
    1. Verify that all applications are synchronized. Under the project multicloud-gitops-hub click the URL for the hub gitops server. The Vault application is not synched.

    Multicloud GitOps Hub

    Multicloud GitOps application demos

    As part of this pattern, HashiCorp Vault has been installed. Refer to the section on Vault.

    Next steps

    Deploying the managed cluster applications

    After the management hub is set up and works correctly, attach one or more managed clusters to the architecture (see diagrams below).

    For instructions on deploying the edge, refer to Managed Cluster Sites.

    Contribute to this pattern: Help & Feedback -Report Bugs

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Ideas for customization

    Why change it?

    One of the major goals of the Red Hat patterns development process is to create modular, customizable demos. The Multicloud Gitops is just an example of a pattern managing multiple clusters in a gitops fashion. It contains a very simple ‘config-demo’ application which prints out a secret that was injected into the vault via an out-of-band mechanism.

    It could be an interesting exercise to customize this demo in different ways.

    Split the config-demo across hub and regional clusters

    Currently hub and regional clusters are reusing the exact same helm chart found at charts/all/config-demo. The first customization step could be to split the demo app in two separate charts: one in charts/hub/config-demo and one in charts/region/config-demo. Once charts/all/config-demo has been copied to charts/hub/config-demo and charts/region/config-demo, we need to include them in the respective values-hub.yaml and values-region-one.yaml, respectively.

    Once this is done we can start customizing the two apps and make them output a different web page entirely depending if the pod is running on the hub or on the cluster.

    Rest API addition

    Another idea, after splitting the charts, could be to implement a small Rest API server on the hub. This API could serve read-only values out to anyone and could provide some update write-APIs only if the client provides a secret (for example via the X-API-KEY mechanism). The config-demo application could be tweaked to talk to the hub and use a vault-injected secret as the X-API-KEY. So the hub would possess the key via the External-Secrets generated K8s secret and the regional app would possess that same secret via the ACM policy pushing out secrets via the {{hub fromSecret}} mechanism.

    In the end the possibilities to tweak this pattern are endless. Do let us know if you have an awesome idea that you’d like to add

    Contribute to this pattern: Help & Feedback{: .btn .fs-5 .mb-4 .mb-md-0 .mr-2 } -Report Bugs{: .btn .btn-red .fs-5 .mb-4 .mb-md-0 .mr-2 }

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Multicloud GitOps with Portworx Enterprise

    Some details will differ based on the requirements of a specific implementation but all validated patterns, based on a portfolio architecture, generalize one or more successful deployments of a use case.

    Use case:

    • Use a GitOps approach to manage hybrid and multi-cloud deployments across both public and private clouds.
    • Enable cross-cluster governance and application lifecycle management.
    • Securely manage secrets across the deployment.
    • Deploy and configure Portworx Enterprise persistent storage for stateful applications.

    Background: Organizations are aiming to develop, deploy, and operate applications on an open hybrid cloud in a stable, simple, and secure way. This hybrid strategy includes multi-cloud deployments where workloads might be running on multiple clusters and on multiple clouds—private or public. It include Portworx Enterprise for persistent storage and Kubernetes data services required for stateful applications.

    This strategy requires an infrastructure-as-code approach: GitOps. GitOps uses Git repositories as a single source of truth to deliver infrastructure-as-code. Submitted code will be checked by the continuous integration (CI) process, while the continuous delivery (CD) process checks and applies requirements for things like security, infrastructure-as-code, or any other boundaries set for the application framework. All changes to code are tracked, making updates easy while also providing version control should a rollback be needed.

    Solution overview

    This architecture covers hybrid and multi-cloud management with GitOps as shown in Figure 1. At a high level this requires a management hub, for DevOps and GitOps, and infrastructure that extends to one or more managed clusters running on private or public clouds. The automated infrastructure-as-code approach can manage the versioning of components and deploy according to the infrastructure-as-code configuration.

    Why Hybrid Multicloud management with GitOps ?

    • Unify management across cloud environments.
    • Dynamic infrastructure security.
    • Infrastructural continuous delivery best practices.

    Multi-Cloud Architecture

    Figure 1 shows a high-level overview of the solution including the business drivers, management hub, and the clusters under management.

    Logical diagram

    Logical Architecture

    Figure 2. Logical diagram of hybrid multi-cloud management with GitOps.

    As you can see in Figure 2, logically this solution can be viewed as being composed of an automation component, unified management (including secrets management), and the cluster(s) under management—all running on top of a user-chosen mixture of on-prem data center(s) and public cloud(s).

    The technology

    The following technology was chosen for this solution.

    Red Hat OpenShift Platform is an enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems (like HashiCorp Vault in this case) to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps is a declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Advanced Cluster Management for Kubernetes controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat Ansible Automation Platform provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation—from development and operations to security and network teams.

    Portworx Enterprise provides persistent storage and Kubernetes data services to Red Hat OpenShift. Persistence is necessary for stateful applications in Kubernetes environments. Portworx also provides business continuity with Portworx Backup and Portworx DR products that will be incorporated in a future GitOps pattern.

    Hashicorp Vault provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Architectures

    Figure 3 provides a schematic diagram overview of the complete solution including both components and data flows.

    Subsequent schematic diagrams go into more detail on:

    • Bootstrapping the management hub (Figure 4)
    • Hybrid multi-cloud GitOps (Figure 5)
    • Dynamic security management (Figure 6)
    • Observability in hybrid multi-cloud environments (Figure 7)

    Physical Architecture

    Figure 3. Overview schematic diagram of the complete solution.

    Bootstrapping the management hub

    Schematic diagram of bootstrapping the management hub

    Figure 4. Schematic diagram of bootstrapping the management hub.

    As detailed below, Figure 4 provides a schematic diagram showing the setup of the management hub using Ansible playbooks.

    • Set up the Red Hat OpenShift Platform that hosts the Management Hub. The OpenShift installation program provides flexible ways to install OpenShift. An Ansible playbook kicks off the installation with necessary configurations.
    • Ansible playbooks are again used to deploy and configure Red Hat Advanced Cluster Management for Kubernetes and, later, other supporting components (such as external secrets management) on top of the provisioned OpenShift cluster.
    • Another Ansible playbook installs HashiCorp Vault, a Red Hat partner product chosen for this solution that can be used to manage secrets for OpenShift clusters.
    • An Ansible playbook is used again to configure and trigger the Openshift GitOps operator on the hub cluster. This deploys the Openshift GitOps instance to enable continuous delivery.

    Hybrid multicloud GitOps

    Schematic diagram of hybrid multi-cloud management with GitOps

    Figure 5. Schematic diagram of hybrid multi-cloud management with GitOps.

    As detailed below, Figure 5 provides a schematic diagram showing remaining activities associated with setting up the management hub and clusters using Red Hat Advanced Cluster Management.

    • Manifest and configuration are set as code template in the form of “Kustomization” yaml. It describes the end desire state of how the managed cluster is going to be like. When done, it is pushed into the source control management repository with a version assigned to each update.
    • OpenShift GitOps watches the repository and detects changes in the repository.
    • OpenShift GitOps creates/updates the manifest by creating Kubernetes objects on top of Red Hat Advanced Cluster Management.
    • Red Hat Advanced Cluster Management provision/update/delete managed clusters and configuration according to the manifest. In the manifest, you can configure what cloud provider the cluster will be on, the name of the cluster, infra node details and worker node. Governance policy can also be applied as well as provision an agent in the cluster as the bridge between the control center and the managed cluster.
    • OpenShift GitOps will continuously watch between the code repository and status of the clusters reported back to Red Hat Advanced Cluster Management. Any configuration drift or in case of any failure, it will automatically try to remediate by applying the manifest (Or showing alerts for manual intervention).

    Dynamic security management

    Schematic showing the setup and use of external secrets management

    Figure 6. Schematic showing the setup and use of external secrets management.

    As detailed below, Figure 6 provides a schematic diagram showing how secrets are handled in this solution.

    During setup, the token to securely access HashiCorp Vault is stored in Ansible Vault. It is encrypted to protect sensitive content.

    Red Hat Advanced Cluster Management for Kubernetes allows us to have centralized control over the managed clusters. It acquires the token from Ansible Vault during install and distributes it among the clusters.

    To allow the cluster access to the external vault, we need to set up the external secret management (with Helm in this study). OpenShift Gitops is used to deploy the external secret object to a managed cluster.

    External secret management fetches secrets from HashiCorp Vault using the token we created in step 2 and constantly watches for updates. -Secrets are created in each namespace, where applications can use them.

    Demo Scenario

    Download diagrams

    View and download all of the diagrams above in our open source tooling site.

    [Open Diagrams]

    What Next

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    About the Intel QAT accelerated Multicloud GitOps pattern

    Use case

    • Use a GitOps approach to manage hybrid and multi-cloud deployments across both public and private clouds.

    • Enable cross-cluster governance and application lifecycle management.

    • Accelerate TLS handshaking operations and improve computational performance by offloading cryptographic tasks to QAT.

    • Securely manage secrets across the deployment.

    • Simplify managing different microservices that make up a cloud-native application with Istio service mesh.

    Based on the requirements of a specific implementation, certain details might differ. However, all validated patterns that are based on a portfolio architecture, generalize one or more successful deployments of a use case.

    Background

    Organizations are aiming to develop, deploy, and operate applications on an open hybrid cloud in a stable, simple, and secure way. This hybrid strategy includes multi-cloud deployments where workloads might be running on multiple clusters and on multiple clouds, private or public. This strategy requires an infrastructure-as-code approach: GitOps. GitOps uses Git repositories as a single source of truth to deliver infrastructure-as-code. Submitted code checks the continuous integration (CI) process, while the continuous delivery (CD) process checks and applies requirements for things like security, infrastructure-as-code, or any other boundaries set for the application framework. All changes to code are tracked, making updates easy while also providing version control should a rollback be needed. -Moreover, organizations which are looking to deliver distributed applications at scale can make these applications faster by offloading cryptographic tasks to Intel QuickAssist Technology.

    About the solution

    This architecture covers hybrid and multi-cloud management with GitOps as shown in following figure. At a high level this requires a management hub, for DevOps and GitOps, and infrastructure that extends to one or more managed clusters running on private or public clouds. The automated infrastructure-as-code approach can manage the versioning of components and deploy according to the infrastructure-as-code configuration.

    Benefits of Hybrid Multicloud management with GitOps:

    • Unify management across cloud environments.

    • Dynamic infrastructure security.

    • Infrastructural continuous delivery best practices.

    Multicloud Architecture
    Figure 1. Overview of the solution including the business drivers, management hub, and the clusters under management

    In the following figure, logically, this solution can be viewed as being composed of an automation component, unified management including secrets management, and the clusters under management, all running on top of a user-chosen mixture of on-premise data centers and public clouds.

    Logical Architecture
    Figure 2. Logical diagram of Intel QAT accelerated hybrid multi-cloud management with GitOps

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Advanced Cluster Management for Kubernetes

    Controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Node Feature Discovery Operator

    Manages the detection of hardware features and configuration in an OpenShift Container Platform cluster by labeling the nodes with hardware-specific information. NFD labels the host with node-specific attributes, such as PCI cards, kernel, operating system version, and so on.

    Intel® QuickAssist Technology

    Accelerate data encryption and compression for applications from networking to enterprise, cloud to storage, and content delivery to database with Intel® QuickAssist Technology.

    Intel® Device Plugin Operator

    The Intel Technology Enabling for OpenShift project provides Intel Data Center hardware feature-provisioning technologies with the Red Hat OpenShift Container Platform (RHOCP).

    Sail Operator, Istio Operator for Red Hat OpenShift

    A service mesh is a dedicated infrastructure layer that you can add to your applications. It allows you to transparently add capabilities like observability, traffic management, and security, without adding them to your code. Sail Operator serves as the foundation for Red Hat OpenShift Service Mesh 3.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Overview of the architectures

    The following figure provides a schematic diagram overview of the complete solution including both components and data flows.

    Physical Architecture
    Figure 3. Overview schematic diagram of the complete solution

    Subsequent schematic diagrams provide details on:

    • Bootstrapping the management hub (Figure 4)

    • Hybrid multi-cloud GitOps (Figure 5)

    • Dynamic security management (Figure 6)

    Bootstrapping the management hub

    The following figure provides a schematic diagram showing the setup of the management hub using Ansible playbooks.

    Schematic diagram of bootstrapping the management hub
    Figure 4. Schematic diagram of bootstrapping the management hub

    • Set up the OpenShift Container Platform that hosts the Management Hub. The OpenShift installation program provides flexible ways to install OpenShift. An Ansible playbook starts the installation with necessary configurations.

    • Ansible playbooks deploy and configure Red Hat Advanced Cluster Management for Kubernetes and, later, other supporting components such as external secrets management, on top of the provisioned OpenShift cluster.

    • Another Ansible playbook installs HashiCorp Vault, a Red Hat partner product chosen for this solution that can be used to manage secrets for OpenShift clusters.

    • An Ansible playbook configures and installs the Openshift GitOps operator on the hub cluster. This deploys the Openshift GitOps instance to enable continuous delivery.

    Hybrid Multicloud GitOps

    The following figure provides a schematic diagram showing remaining activities associated with setting up the management hub and clusters using Red Hat Advanced Cluster Management.

    Schematic diagram of hybrid multi-cloud management with GitOps
    Figure 5. Schematic diagram of hybrid multi-cloud management with GitOps

    • Manifest and configuration are set as code template in the form of a Kustomization YAML file. The file describes the desired end state of the managed cluster. When complete, the Kustomization YAML file is pushed into the source control management repository with a version assigned to each update.

    • OpenShift GitOps monitors the repository and detects changes in the repository.

    • OpenShift GitOps creates and updates the manifest by creating Kubernetes objects on top of Red Hat Advanced Cluster Management.

    • Red Hat Advanced Cluster Management provisions, updates, or deletes managed clusters and configuration according to the manifest. In the manifest, you can configure what cloud provider the cluster will be on, the name of the cluster, infrastructure node details and worker node. Governance policy can also be applied as well as provision an agent in the cluster as the bridge between the control center and the managed cluster.

    • OpenShift GitOps continuously monitors the code repository and the status of the clusters reported back to Red Hat Advanced Cluster Management. Any configuration drift or in case of any failure, OpenShift GitOps will automatically try to remediate by applying the manifest or by displaying alerts for manual intervention.

    Dynamic security management

    The following figure provides a schematic diagram showing how secrets are handled in this solution.

    Schematic showing the setup and use of external secrets management
    Figure 6. Schematic showing the setup and use of external secrets management

    • During setup, the token to securely access HashiCorp Vault is stored in Ansible Vault. It is encrypted to protect sensitive content.

    • Red Hat Advanced Cluster Management for Kubernetes acquires the token from Ansible Vault during install and distributes it among the clusters. As a result, you have centralized control over the managed clusters through RHACM.

    • To allow the cluster access to the external vault, you must set up the external secret management with Helm in this study. OpenShift Gitops is used to deploy the external secret object to a managed cluster.

    • External secret management fetches secrets from HashiCorp Vault by using the token that was generated in step 2 and constantly monitors for updates.

    • Secrets are created in each namespace, where applications can use them.

    Additional resources

    View and download all of the diagrams above from the Red Hat Portfolio Architecture open source tooling site.

    Intel QAT accelerated Multicloud GitOps pattern

    The basic Multicloud GitOps pattern has been extended to highlight the 5th Generation Intel Xeon Scalable Processors capabilities, offering developers a streamlined pathway to accelerate their workloads through the integration of Intel QAT.

    The basic pattern has been extended by 4 components: Intel Device Plugin Operator, Node Feature Discovery Operator (NFD), Sail Operator, and simple HTTP request and response application. The main purpose of this extension is to enable TLS handshaking and showcase how Intel QAT accelerates cryptographic operations.

    Intel Device Plugin Operator’s task is to expose QAT resources to the Openshift layer, so it makes it easy to use when creating deployments. After QAT resources are properly exposed Node Feature Discovery Operator is deployed. NFD manages the detection of hardware features and configuration in an OpenShift Container Platform cluster. It labels the nodes with hardware-specific information.

    Sail Operator deploys Istio Service Mesh. It is basically an extra layer added on top of user applications, and it adds capabilities like traffic management between services and security, without manually adding them to code. The Istio version deployed by Sail Operator allows for additional configuration, that is enabling Intel QAT in cryptographic operations. The last added component is the httpbin application, which is HTTP request and response service used for testing and showcase purposes. The workload idea was based on Istio’s official instructions on how to setup secure TLS ingress gateway.

    The appearance of qat as a private key provider in Istio ingress gateway pod logs confirms that Intel QAT is used:

    Logs from Istio ingress gateway pod
    Figure 7. Logs from Istio ingress gateway pod

    Next steps

    QUICKLINKS

    About the solution

    This architecture covers hybrid and multi-cloud management with GitOps as shown in following figure. At a high level this requires a management hub, for DevOps and GitOps, and infrastructure that extends to one or more managed clusters running on private or public clouds. The automated infrastructure-as-code approach can manage the versioning of components and deploy according to the infrastructure-as-code configuration.

    Benefits of Hybrid Multicloud management with GitOps:

    • Unify management across cloud environments.

    • Dynamic infrastructure security.

    • Infrastructural continuous delivery best practices.

    Multicloud Architecture
    Figure 1. Overview of the solution including the business drivers, management hub, and the clusters under management

    In the following figure, logically, this solution can be viewed as being composed of an automation component, unified management including secrets management, and the clusters under management, all running on top of a user-chosen mixture of on-premise data centers and public clouds.

    Logical Architecture
    Figure 2. Logical diagram of Intel QAT accelerated hybrid multi-cloud management with GitOps

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Advanced Cluster Management for Kubernetes

    Controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Node Feature Discovery Operator

    Manages the detection of hardware features and configuration in an OpenShift Container Platform cluster by labeling the nodes with hardware-specific information. NFD labels the host with node-specific attributes, such as PCI cards, kernel, operating system version, and so on.

    Intel® QuickAssist Technology

    Accelerate data encryption and compression for applications from networking to enterprise, cloud to storage, and content delivery to database with Intel® QuickAssist Technology.

    Intel® Device Plugin Operator

    The Intel Technology Enabling for OpenShift project provides Intel Data Center hardware feature-provisioning technologies with the Red Hat OpenShift Container Platform (RHOCP).

    Sail Operator, Istio Operator for Red Hat OpenShift

    A service mesh is a dedicated infrastructure layer that you can add to your applications. It allows you to transparently add capabilities like observability, traffic management, and security, without adding them to your code. Sail Operator serves as the foundation for Red Hat OpenShift Service Mesh 3.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Overview of the architectures

    The following figure provides a schematic diagram overview of the complete solution including both components and data flows.

    Physical Architecture
    Figure 3. Overview schematic diagram of the complete solution

    Subsequent schematic diagrams provide details on:

    • Bootstrapping the management hub (Figure 4)

    • Hybrid multi-cloud GitOps (Figure 5)

    • Dynamic security management (Figure 6)

    Bootstrapping the management hub

    The following figure provides a schematic diagram showing the setup of the management hub using Ansible playbooks.

    Schematic diagram of bootstrapping the management hub
    Figure 4. Schematic diagram of bootstrapping the management hub

    • Set up the OpenShift Container Platform that hosts the Management Hub. The OpenShift installation program provides flexible ways to install OpenShift. An Ansible playbook starts the installation with necessary configurations.

    • Ansible playbooks deploy and configure Red Hat Advanced Cluster Management for Kubernetes and, later, other supporting components such as external secrets management, on top of the provisioned OpenShift cluster.

    • Another Ansible playbook installs HashiCorp Vault, a Red Hat partner product chosen for this solution that can be used to manage secrets for OpenShift clusters.

    • An Ansible playbook configures and installs the Openshift GitOps operator on the hub cluster. This deploys the Openshift GitOps instance to enable continuous delivery.

    Hybrid Multicloud GitOps

    The following figure provides a schematic diagram showing remaining activities associated with setting up the management hub and clusters using Red Hat Advanced Cluster Management.

    Schematic diagram of hybrid multi-cloud management with GitOps
    Figure 5. Schematic diagram of hybrid multi-cloud management with GitOps

    • Manifest and configuration are set as code template in the form of a Kustomization YAML file. The file describes the desired end state of the managed cluster. When complete, the Kustomization YAML file is pushed into the source control management repository with a version assigned to each update.

    • OpenShift GitOps monitors the repository and detects changes in the repository.

    • OpenShift GitOps creates and updates the manifest by creating Kubernetes objects on top of Red Hat Advanced Cluster Management.

    • Red Hat Advanced Cluster Management provisions, updates, or deletes managed clusters and configuration according to the manifest. In the manifest, you can configure what cloud provider the cluster will be on, the name of the cluster, infrastructure node details and worker node. Governance policy can also be applied as well as provision an agent in the cluster as the bridge between the control center and the managed cluster.

    • OpenShift GitOps continuously monitors the code repository and the status of the clusters reported back to Red Hat Advanced Cluster Management. Any configuration drift or in case of any failure, OpenShift GitOps will automatically try to remediate by applying the manifest or by displaying alerts for manual intervention.

    Dynamic security management

    The following figure provides a schematic diagram showing how secrets are handled in this solution.

    Schematic showing the setup and use of external secrets management
    Figure 6. Schematic showing the setup and use of external secrets management

    • During setup, the token to securely access HashiCorp Vault is stored in Ansible Vault. It is encrypted to protect sensitive content.

    • Red Hat Advanced Cluster Management for Kubernetes acquires the token from Ansible Vault during install and distributes it among the clusters. As a result, you have centralized control over the managed clusters through RHACM.

    • To allow the cluster access to the external vault, you must set up the external secret management with Helm in this study. OpenShift Gitops is used to deploy the external secret object to a managed cluster.

    • External secret management fetches secrets from HashiCorp Vault by using the token that was generated in step 2 and constantly monitors for updates.

    • Secrets are created in each namespace, where applications can use them.

    Additional resources

    View and download all of the diagrams above from the Red Hat Portfolio Architecture open source tooling site.

    Intel QAT accelerated Multicloud GitOps pattern

    The basic Multicloud GitOps pattern has been extended to highlight the 5th Generation Intel Xeon Scalable Processors capabilities, offering developers a streamlined pathway to accelerate their workloads through the integration of Intel QAT.

    The basic pattern has been extended by 4 components: Intel Device Plugin Operator, Node Feature Discovery Operator (NFD), Sail Operator, and simple HTTP request and response application. The main purpose of this extension is to enable TLS handshaking and showcase how Intel QAT accelerates cryptographic operations.

    Intel Device Plugin Operator’s task is to expose QAT resources to the Openshift layer, so it makes it easy to use when creating deployments. After QAT resources are properly exposed Node Feature Discovery Operator is deployed. NFD manages the detection of hardware features and configuration in an OpenShift Container Platform cluster. It labels the nodes with hardware-specific information.

    Sail Operator deploys Istio Service Mesh. It is basically an extra layer added on top of user applications, and it adds capabilities like traffic management between services and security, without manually adding them to code. The Istio version deployed by Sail Operator allows for additional configuration, that is enabling Intel QAT in cryptographic operations. The last added component is the httpbin application, which is HTTP request and response service used for testing and showcase purposes. The workload idea was based on Istio’s official instructions on how to setup secure TLS ingress gateway.

    The appearance of qat as a private key provider in Istio ingress gateway pod logs confirms that Intel QAT is used:

    Logs from Istio ingress gateway pod
    Figure 7. Logs from Istio ingress gateway pod

    Next steps

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    As a part of this pattern, HashiCorp Vault has been installed. Refer to the section on Vault.

    Next steps

    After the management hub is set up and works correctly, attach one or more managed clusters to the architecture.

    For instructions on deploying the edge, refer to Attach a managed cluster (edge) to the management hub.

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Ensure that you commit the changes and push them to GitHub so that GitOps can fetch your changes and apply them.

    Deploying a managed cluster by using Red Hat Advanced Cluster Management

    Prerequisites

    • An OpenShift cluster

    • Red Hat Advanced Cluster Management (RHACM) web console to join the managed cluster to the management hub

      After RHACM is installed, a message regarding a Web console update is available" might be displayed. Follow the instructions and click the Refresh web console link.

    Procedure

    1. In the left navigation panel of web console, click local-cluster. Select All Clusters. The RHACM web console is displayed with Cluster* on the left navigation panel.

    2. On the Managed clusters tab, click Import cluster.

    3. On the Import an existing cluster page, enter the cluster name and choose KUBECONFIG as the "import mode". Add the tag clusterGroup=region-one. Click Import.

    Now that RHACM is no longer deploying the managed cluster applications everywhere, you must indicate that the new cluster has the managed cluster role.

    Optional: Deploying a managed cluster by using cm-cli tool

    Prerequisites
    Procedure

    1. Obtain the KUBECONFIG file from the managed cluster.

    2. Open a shell prompt and login into the management hub cluster by using either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Run the following command:

      cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-path_to_kubeconfig>
    Next steps

    Optional: Deploying a managed cluster by using the clusteradm tool

    Prerequisites
    Procedure

    1. To deploy an edge cluster, you must to get the token from the management hub cluster. Run the following command on the existing management hub or datacenter cluster:

      clusteradm get token

      The command generates a token and shows you the command to use on the managed cluster.

    2. Login to the managed cluster with either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. To request that the managed join the hub cluster, run the following command:

      clusteradm join --hub-token <token_from_clusteradm_get_token_command> <managed_cluster_name>

    4. Accept the join request on the hub cluster:

      clusteradm accept --clusters <managed_cluster_name>
    Next steps

    Designate the new cluster as a managed cluster site

    If you use the command line tools such as clusteradm or cm-cli, you must explicitly indicate that the imported cluster is part of a specific clusterGroup. Some examples of clusterGroup are factory, devel, or prod.

    To tag the cluster as clusterGroup=<managed-cluster-group>, complete the following steps.

    Procedure

    1. To find the new cluster, run the following command:

      oc get managedcluster.cluster.open-cluster-management.io

    2. To apply the label, run the following command:

      oc label managedcluster.cluster.open-cluster-management.io/YOURCLUSTER site=managed-cluster

    Verification

    Go to your managed cluster (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. It might take a while for the RHACM agent and agent-addons to launch. After that, the OpenShift GitOps Operator is installed. On successful installation, launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    QUICKLINKS

    Ensure that you commit the changes and push them to GitHub so that GitOps can fetch your changes and apply them.

    Deploying a managed cluster by using Red Hat Advanced Cluster Management

    Prerequisites

    • An OpenShift cluster

    • Red Hat Advanced Cluster Management (RHACM) web console to join the managed cluster to the management hub

      After RHACM is installed, a message regarding a Web console update is available" might be displayed. Follow the instructions and click the Refresh web console link.

    Procedure

    1. In the left navigation panel of web console, click local-cluster. Select All Clusters. The RHACM web console is displayed with Cluster* on the left navigation panel.

    2. On the Managed clusters tab, click Import cluster.

    3. On the Import an existing cluster page, enter the cluster name and choose KUBECONFIG as the "import mode". Add the tag clusterGroup=region-one. Click Import.

    Now that RHACM is no longer deploying the managed cluster applications everywhere, you must indicate that the new cluster has the managed cluster role.

    Optional: Deploying a managed cluster by using cm-cli tool

    Prerequisites
    Procedure

    1. Obtain the KUBECONFIG file from the managed cluster.

    2. Open a shell prompt and login into the management hub cluster by using either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Run the following command:

      cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-path_to_kubeconfig>
    Next steps

    Optional: Deploying a managed cluster by using the clusteradm tool

    Prerequisites
    Procedure

    1. To deploy an edge cluster, you must to get the token from the management hub cluster. Run the following command on the existing management hub or datacenter cluster:

      clusteradm get token

      The command generates a token and shows you the command to use on the managed cluster.

    2. Login to the managed cluster with either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. To request that the managed join the hub cluster, run the following command:

      clusteradm join --hub-token <token_from_clusteradm_get_token_command> <managed_cluster_name>

    4. Accept the join request on the hub cluster:

      clusteradm accept --clusters <managed_cluster_name>
    Next steps

    Designate the new cluster as a managed cluster site

    If you use the command line tools such as clusteradm or cm-cli, you must explicitly indicate that the imported cluster is part of a specific clusterGroup. Some examples of clusterGroup are factory, devel, or prod.

    To tag the cluster as clusterGroup=<managed-cluster-group>, complete the following steps.

    Procedure

    1. To find the new cluster, run the following command:

      oc get managedcluster.cluster.open-cluster-management.io

    2. To apply the label, run the following command:

      oc label managedcluster.cluster.open-cluster-management.io/YOURCLUSTER site=managed-cluster

    Verification

    Go to your managed cluster (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. It might take a while for the RHACM agent and agent-addons to launch. After that, the OpenShift GitOps Operator is installed. On successful installation, launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    QUICKLINKS

    CONTRIBUTE

    About the Intel SGX protected application in Multicloud GitOps pattern

    Use case

    • Use a GitOps approach to manage hybrid and multi-cloud deployments across both public and private clouds.

    • Enable cross-cluster governance and application lifecycle management.

    • Securely manage secrets across the deployment.

    • Effectively protect data in use with Intel Software Guard Extensions.

      Based on the requirements of a specific implementation, certain details might differ. However, all validated patterns that are based on a portfolio architecture, generalize one or more successful deployments of a use case.

    Background

    Organizations are aiming to develop, deploy, and operate applications on an open hybrid cloud in a stable, simple, and secure way. This hybrid strategy includes multi-cloud deployments where workloads might be running on multiple clusters and on multiple clouds, private or public. This strategy requires an infrastructure-as-code approach: GitOps. GitOps uses Git repositories as a single source of truth to deliver infrastructure-as-code. Submitted code checks the continuous integration (CI) process, while the continuous delivery (CD) process checks and applies requirements for things like security, infrastructure-as-code, or any other boundaries set for the application framework. All changes to code are tracked, making updates easy while also providing version control should a rollback be needed. Moreover, organizations are looking for solutions that are secure for AI, ML, data processing etc. It is the case especially for cloud computing, which uses heavily multi-tenancy and multiple processes runs on single bare-metal machine and they do not know who might be their neighbors and what are their intentions. -Memory encryption technologies can protect well the data and separate application from other ones run on the same machine - it is possible using 5th Generation Intel Xeon Scalable Processors with Intel Software Guard Extensions.

    About the solution

    This architecture covers hybrid and multi-cloud management with GitOps as shown in following figure. At a high level this requires a management hub, for DevOps and GitOps, and infrastructure that extends to one or more managed clusters running on private or public clouds. The automated infrastructure-as-code approach can manage the versioning of components and deploy according to the infrastructure-as-code configuration.

    Benefits of Hybrid Multicloud management with GitOps:

    • Unify management across cloud environments.

    • Dynamic infrastructure security.

    • Infrastructural continuous delivery best practices.

    Multicloud Architecture
    Figure 1. Overview of the solution including the business drivers, management hub, and the clusters under management

    In the following figure, logically, this solution can be viewed as being composed of an automation component, unified management including secrets management, and the clusters under management, all running on top of a user-chosen mixture of on-premise data centers and public clouds.

    Logical Architecture
    Figure 2. Logical diagram of Intel SGX accelerated hybrid multi-cloud management with GitOps

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Advanced Cluster Management for Kubernetes

    Controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Node Feature Discovery Operator

    Manages the detection of hardware features and configuration in an OpenShift Container Platform cluster by labeling the nodes with hardware-specific information. NFD labels the host with node-specific attributes, such as PCI cards, kernel, operating system version, and so on.

    Intel® Device Plugins Operator

    Is a collection of device plugins advertising Intel specific hardware resources to the kubelet. Currently the operator has basic support for the QAT, GPU, FPGA, SGX, DSA, IAA device plugins: it validates container image references and extends reported statuses.

    Intel® Software Guard Extensions

    Helps protect data in use via unique application isolation technology. Protect selected code and data from modification using hardened enclaves with Intel SGX.

    Gramine Shielded Containers

    Transforms a Docker image into a new image which includes the Gramine Library OS, manifest files, Intel SGX related information, and executes the application inside an Intel SGX enclave using the Gramine Library OS.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Overview of the architectures

    The following figure provides a schematic diagram overview of the complete solution including both components and data flows.

    Physical Architecture
    Figure 3. Overview schematic diagram of the complete solution

    Subsequent schematic diagrams provide details on:

    • Bootstrapping the management hub (Figure 4)

    • Hybrid multi-cloud GitOps (Figure 5)

    • Dynamic security management (Figure 6)

    Bootstrapping the management hub

    The following figure provides a schematic diagram showing the setup of the management hub using Ansible playbooks.

    Schematic diagram of bootstrapping the management hub
    Figure 4. Schematic diagram of bootstrapping the management hub

    • Set up the OpenShift Container Platform that hosts the Management Hub. The OpenShift installation program provides flexible ways to install OpenShift. An Ansible playbook starts the installation with necessary configurations.

    • Ansible playbooks deploy and configure Red Hat Advanced Cluster Management for Kubernetes and, later, other supporting components such as external secrets management, on top of the provisioned OpenShift cluster.

    • Another Ansible playbook installs HashiCorp Vault, a Red Hat partner product chosen for this solution that can be used to manage secrets for OpenShift clusters.

    • An Ansible playbook configures and installs the Openshift GitOps operator on the hub cluster. This deploys the Openshift GitOps instance to enable continuous delivery.

    Hybrid Multicloud GitOps

    The following figure provides a schematic diagram showing remaining activities associated with setting up the management hub and clusters using Red Hat Advanced Cluster Management.

    Schematic diagram of hybrid multi-cloud management with GitOps
    Figure 5. Schematic diagram of hybrid multi-cloud management with GitOps

    • Manifest and configuration are set as code template in the form of a Kustomization YAML file. The file describes the desired end state of the managed cluster. When complete, the Kustomization YAML file is pushed into the source control management repository with a version assigned to each update.

    • OpenShift GitOps monitors the repository and detects changes in the repository.

    • OpenShift GitOps creates and updates the manifest by creating Kubernetes objects on top of Red Hat Advanced Cluster Management.

    • Red Hat Advanced Cluster Management provisions, updates, or deletes managed clusters and configuration according to the manifest. In the manifest, you can configure what cloud provider the cluster will be on, the name of the cluster, infrastructure node details and worker node. Governance policy can also be applied as well as provision an agent in the cluster as the bridge between the control center and the managed cluster.

    • OpenShift GitOps continuously monitors the code repository and the status of the clusters reported back to Red Hat Advanced Cluster Management. Any configuration drift or in case of any failure, OpenShift GitOps will automatically try to remediate by applying the manifest or by displaying alerts for manual intervention.

    Dynamic security management

    The following figure provides a schematic diagram showing how secrets are handled in this solution.

    Schematic showing the setup and use of external secrets management
    Figure 6. Schematic showing the setup and use of external secrets management

    • During setup, the token to securely access HashiCorp Vault is stored in Ansible Vault. It is encrypted to protect sensitive content.

    • Red Hat Advanced Cluster Management for Kubernetes acquires the token from Ansible Vault during install and distributes it among the clusters. As a result, you have centralized control over the managed clusters through RHACM.

    • To allow the cluster access to the external vault, you must set up the external secret management with Helm in this study. OpenShift Gitops is used to deploy the external secret object to a managed cluster.

    • External secret management fetches secrets from HashiCorp Vault by using the token that was generated in step 2 and constantly monitors for updates.

    • Secrets are created in each namespace, where applications can use them.

    Additional resources

    View and download all of the diagrams above from the Red Hat Portfolio Architecture open source tooling site.

    Intel SGX protected application in Multicloud GitOps pattern

    The basic Multicloud GitOps pattern has been extended to highlight the 5th Generation Intel Xeon Scalable Processors capabilities, offering developers Intel SGX as a streamlined pathway to protect data in use against threats with isolation, encryption, and attestation while also allowing users to maintain control and use their data.

    The application which has been added to the basic pattern is named hello-world-sgx. It is a simple Python application that runs securely inside the SGX enclave. SGX enclave is an encrypted memory, which prevents unauthorized access from other containers running on the same host or even some host’s processes.

    The sample application uses image converted with Gramine Shielded Containers (GSC) tools to include Intel SGX related information and execute the application inside an Intel SGX enclave using the Gramine Library OS. It’s an easy way to convert the application into its secure version.

    Since the hello-world-sgx application must be running on the node with CPU supporting Intel SGX, Node Feature Discovery Operator (NFD) and Intel Device Plugins Operator (IDP) are deployed as a part of this pattern.

    NFD manages the detection of hardware features and configuration in an OpenShift Container Platform cluster. It labels the nodes with hardware-specific information. It is a requirement for IDP installation. IDP provides resources which are the user interface to claim and consume the hardware feature by user pods and also advertises Intel-specific hardware resources to the kubelet.

    In the logs of hello-world-sgx pod, there is an information that "Gramine is starting" and it executes the application by printing "HelloWorld!".

    This pattern demonstrates basic capabilities of running docker applications inside SGX enclaves. Based on this pattern other applications can be secured with Intel SGX and Gramine Shielded Containers in a similar way as presented here.

    Logs from hello-world-sgx pod
    Figure 7. Logs from hello-world-sgx pod

    Next steps

    QUICKLINKS

    About the solution

    This architecture covers hybrid and multi-cloud management with GitOps as shown in following figure. At a high level this requires a management hub, for DevOps and GitOps, and infrastructure that extends to one or more managed clusters running on private or public clouds. The automated infrastructure-as-code approach can manage the versioning of components and deploy according to the infrastructure-as-code configuration.

    Benefits of Hybrid Multicloud management with GitOps:

    • Unify management across cloud environments.

    • Dynamic infrastructure security.

    • Infrastructural continuous delivery best practices.

    Multicloud Architecture
    Figure 1. Overview of the solution including the business drivers, management hub, and the clusters under management

    In the following figure, logically, this solution can be viewed as being composed of an automation component, unified management including secrets management, and the clusters under management, all running on top of a user-chosen mixture of on-premise data centers and public clouds.

    Logical Architecture
    Figure 2. Logical diagram of Intel SGX accelerated hybrid multi-cloud management with GitOps

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Advanced Cluster Management for Kubernetes

    Controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Node Feature Discovery Operator

    Manages the detection of hardware features and configuration in an OpenShift Container Platform cluster by labeling the nodes with hardware-specific information. NFD labels the host with node-specific attributes, such as PCI cards, kernel, operating system version, and so on.

    Intel® Device Plugins Operator

    Is a collection of device plugins advertising Intel specific hardware resources to the kubelet. Currently the operator has basic support for the QAT, GPU, FPGA, SGX, DSA, IAA device plugins: it validates container image references and extends reported statuses.

    Intel® Software Guard Extensions

    Helps protect data in use via unique application isolation technology. Protect selected code and data from modification using hardened enclaves with Intel SGX.

    Gramine Shielded Containers

    Transforms a Docker image into a new image which includes the Gramine Library OS, manifest files, Intel SGX related information, and executes the application inside an Intel SGX enclave using the Gramine Library OS.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Overview of the architectures

    The following figure provides a schematic diagram overview of the complete solution including both components and data flows.

    Physical Architecture
    Figure 3. Overview schematic diagram of the complete solution

    Subsequent schematic diagrams provide details on:

    • Bootstrapping the management hub (Figure 4)

    • Hybrid multi-cloud GitOps (Figure 5)

    • Dynamic security management (Figure 6)

    Bootstrapping the management hub

    The following figure provides a schematic diagram showing the setup of the management hub using Ansible playbooks.

    Schematic diagram of bootstrapping the management hub
    Figure 4. Schematic diagram of bootstrapping the management hub

    • Set up the OpenShift Container Platform that hosts the Management Hub. The OpenShift installation program provides flexible ways to install OpenShift. An Ansible playbook starts the installation with necessary configurations.

    • Ansible playbooks deploy and configure Red Hat Advanced Cluster Management for Kubernetes and, later, other supporting components such as external secrets management, on top of the provisioned OpenShift cluster.

    • Another Ansible playbook installs HashiCorp Vault, a Red Hat partner product chosen for this solution that can be used to manage secrets for OpenShift clusters.

    • An Ansible playbook configures and installs the Openshift GitOps operator on the hub cluster. This deploys the Openshift GitOps instance to enable continuous delivery.

    Hybrid Multicloud GitOps

    The following figure provides a schematic diagram showing remaining activities associated with setting up the management hub and clusters using Red Hat Advanced Cluster Management.

    Schematic diagram of hybrid multi-cloud management with GitOps
    Figure 5. Schematic diagram of hybrid multi-cloud management with GitOps

    • Manifest and configuration are set as code template in the form of a Kustomization YAML file. The file describes the desired end state of the managed cluster. When complete, the Kustomization YAML file is pushed into the source control management repository with a version assigned to each update.

    • OpenShift GitOps monitors the repository and detects changes in the repository.

    • OpenShift GitOps creates and updates the manifest by creating Kubernetes objects on top of Red Hat Advanced Cluster Management.

    • Red Hat Advanced Cluster Management provisions, updates, or deletes managed clusters and configuration according to the manifest. In the manifest, you can configure what cloud provider the cluster will be on, the name of the cluster, infrastructure node details and worker node. Governance policy can also be applied as well as provision an agent in the cluster as the bridge between the control center and the managed cluster.

    • OpenShift GitOps continuously monitors the code repository and the status of the clusters reported back to Red Hat Advanced Cluster Management. Any configuration drift or in case of any failure, OpenShift GitOps will automatically try to remediate by applying the manifest or by displaying alerts for manual intervention.

    Dynamic security management

    The following figure provides a schematic diagram showing how secrets are handled in this solution.

    Schematic showing the setup and use of external secrets management
    Figure 6. Schematic showing the setup and use of external secrets management

    • During setup, the token to securely access HashiCorp Vault is stored in Ansible Vault. It is encrypted to protect sensitive content.

    • Red Hat Advanced Cluster Management for Kubernetes acquires the token from Ansible Vault during install and distributes it among the clusters. As a result, you have centralized control over the managed clusters through RHACM.

    • To allow the cluster access to the external vault, you must set up the external secret management with Helm in this study. OpenShift Gitops is used to deploy the external secret object to a managed cluster.

    • External secret management fetches secrets from HashiCorp Vault by using the token that was generated in step 2 and constantly monitors for updates.

    • Secrets are created in each namespace, where applications can use them.

    Additional resources

    View and download all of the diagrams above from the Red Hat Portfolio Architecture open source tooling site.

    Intel SGX protected application in Multicloud GitOps pattern

    The basic Multicloud GitOps pattern has been extended to highlight the 5th Generation Intel Xeon Scalable Processors capabilities, offering developers Intel SGX as a streamlined pathway to protect data in use against threats with isolation, encryption, and attestation while also allowing users to maintain control and use their data.

    The application which has been added to the basic pattern is named hello-world-sgx. It is a simple Python application that runs securely inside the SGX enclave. SGX enclave is an encrypted memory, which prevents unauthorized access from other containers running on the same host or even some host’s processes.

    The sample application uses image converted with Gramine Shielded Containers (GSC) tools to include Intel SGX related information and execute the application inside an Intel SGX enclave using the Gramine Library OS. It’s an easy way to convert the application into its secure version.

    Since the hello-world-sgx application must be running on the node with CPU supporting Intel SGX, Node Feature Discovery Operator (NFD) and Intel Device Plugins Operator (IDP) are deployed as a part of this pattern.

    NFD manages the detection of hardware features and configuration in an OpenShift Container Platform cluster. It labels the nodes with hardware-specific information. It is a requirement for IDP installation. IDP provides resources which are the user interface to claim and consume the hardware feature by user pods and also advertises Intel-specific hardware resources to the kubelet.

    In the logs of hello-world-sgx pod, there is an information that "Gramine is starting" and it executes the application by printing "HelloWorld!".

    This pattern demonstrates basic capabilities of running docker applications inside SGX enclaves. Based on this pattern other applications can be secured with Intel SGX and Gramine Shielded Containers in a similar way as presented here.

    Logs from hello-world-sgx pod
    Figure 7. Logs from hello-world-sgx pod

    Next steps

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    CONTRIBUTE

  • Build the docker image using the following command:

    $ docker build -t hello-world .

  • Clone GSC repo:

    $ git clone https://github.com/gramineproject/gsc.git

  • Copy configuration file from sample template:

    $ cd gsc
 $ cp config.yaml.template config.yaml

    (Optional) This file can be adjusted if needed.

  • Create a new hello-world.manifest file containing parameters of conversion:

    loader.pal_internal_mem_size = "1G"
 sgx.enclave_size = "1G"
-sgx.thread_num = 1024

  • Generate the signing key:

    $ openssl genrsa -3 -out enclave-key.pem 3072

  • Build secure image using gsc command:

    $ ./gsc build hello-world hello-world.manifest

  • Sign the graminized docker image:

    $ ./gsc sign-image hello-world enclave-key.pem

  • Push secured container image to the registry which is available on the cluster:

    1. Tag secured container image with appropriate tag and push it to the registry:

      $ docker tag gsc-hello-world <DOCKER_REPOSITORY>:<TAG>

      <DOCKER_REPOSITORY> is name a docker repository, <TAG> is a version of an image

    2. Login to Docker Hub:

      $ docker login -u=<DOCKER_ID> docker.io

      <DOCKER_ID> is your login to Docker Hub

    3. Push the image to registry:

      $ docker push <DOCKER_REPOSITORY>:<TAG>

  • Update the file charts/all/hello-world-sgx/values.yaml with the information about your image in your forked repo with pattern.

    • Deploying the cluster by using the pattern.sh file

    To deploy the cluster by using the pattern.sh file, complete the following steps:

    1. Login to your cluster by running the following command:

      oc login

      Optional: Set the KUBECONFIG variable for the kubeconfig file path:

      export KUBECONFIG=~/<path_to_kubeconfig>

    2. Deploy the pattern to your cluster. Run the following command:

      ./pattern.sh make install

    Verify that the Operators have been installed.

    1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

    2. Check that the following Operators are installed with Succeeded status (Figure 1):

      • Advanced Cluster Management for Kubernetes

      • Intel Device Plugins Operator

      • multicluster engine for Kubernetes

      • Node Feature Discovery Operator

      • Red Hat Openshift GitOps

      • Validated Patterns Operator

        Multicloud GitOps Hub with SGX installed operators
        Figure 1. List of Installed Operators for Multicloud GitOps Validated Pattern with SGX

    Deploying the cluster by using the Validated Patterns Operator

    To install the Validated Patterns Operator:

    1. Log in to the Openshift Container Platform web console and select Operators > OperatorHub.

    2. Search for Validated Patterns Operator, open it and click Install.

      Install Validated Patterns Operator
      Figure 2. Install Validated Patterns Operator

    3. Choose default settings for the installation mode, namespaces and update strategy and confirm it by clicking Install.

    4. Select Operators > Installed Operators.

    5. Ensure that Validated Patterns Operator is listed in the openshift-operators project with a status Succeeded.

    After a successful installation open the Validated Patterns Operator page. Next, go to Pattern tab and click Create Pattern.

    1. Set the Name field to multicloud-gitops-sgx-hello-world, and the Cluster Group Name to hub. Values must be the same as in the values-global.yaml file (Figure 3).

    2. As a Git Config > Target Repo value, paste the link to your fork. Under Git Config > Target Revision write the name of your branch (Figure 3).

    3. Click the Create button to create the pattern.

      Create Pattern Form
      Figure 3. Create Pattern Form

    Verify that the rest of Operators have been installed:

    1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

    2. Check that the following Operators are installed with Succeeded status (Figure 1):

      • Advanced Cluster Management for Kubernetes

      • Intel Device Plugins Operator

      • multicluster engine for Kubernetes

      • Node Feature Discovery Operator

      • Red Hat Openshift GitOps

    Add a secret for config-demo application (from values-secret-multicloud-gitops.yaml) to Vault manually:

    1. Go to Vault service route. URL can be found:

      1. by running command:

        oc -n vault get route vault -ojsonpath='{.spec.host}'

      2. in Openshift Container Platform web console under Networking > Routes for vault project.

    2. Log into the Vault using root token. Root token can be found by executing command:

      oc -n imperative get secrets vaultkeys -ojsonpath='{.data.vault_data_json}' | base64 -d

    3. After login go to secret catalog and clik Create secret and fill all the fields manually (Figure 4):

      1. Put global/config-demo value in the Path for this secret field (the value comes from values-secret-multicloud-gitops.yaml file).

      2. Add one Secret data key-value pair. Put secret as a key (left field). Click Add button to confirm.

      3. Click Save to save changes.

      Create secret
      Figure 4. Create secret

    Verification

    Go to the Hub ArgoCD and verify that all applications are synchronized. The URL can be found in Openshift Container Platform web console under Networking > Routes for the project multicloud-gitops-sgx-hello-world-hub or use command:

    oc -n multicloud-gitops-sgx-hello-world-hub get route hub-gitops-server -ojsonpath='{.spec.host}'

    All applications should be Healthy and Synced:

    ArgoCD panel with hello-world-sgx app
    Figure 5. ArgoCD panel with hello-world-sgx app

    Check the logs of a pod hello-world-sgx to verify if it uses Intel SGX. In the OpenShift Container Platform web console, navigate to Workloads > Pods. Change project to hello-world-sgx and open the Logs tab in the pod details. The Gramine output confirms that Intel SGX is used.

    In case of pod hello-world-sgx crashing, delete it manually to be recreated - not restarted.

    As part of this pattern, HashiCorp Vault has been installed. Refer to the section on Vault.

    Next steps

    After the management hub is set up and works correctly, attach one or more managed clusters to the architecture.

    For instructions on deploying the edge, refer to Attach a managed cluster (edge) to the management hub.

    QUICKLINKS

  • Generate the signing key:

    $ openssl genrsa -3 -out enclave-key.pem 3072

  • Build secure image using gsc command:

    $ ./gsc build hello-world hello-world.manifest

  • Sign the graminized docker image:

    $ ./gsc sign-image hello-world enclave-key.pem

  • Push secured container image to the registry which is available on the cluster:

    1. Tag secured container image with appropriate tag and push it to the registry:

      $ docker tag gsc-hello-world <DOCKER_REPOSITORY>:<TAG>

      <DOCKER_REPOSITORY> is name a docker repository, <TAG> is a version of an image

    2. Login to Docker Hub:

      $ docker login -u=<DOCKER_ID> docker.io

      <DOCKER_ID> is your login to Docker Hub

    3. Push the image to registry:

      $ docker push <DOCKER_REPOSITORY>:<TAG>

  • Update the file charts/all/hello-world-sgx/values.yaml with the information about your image in your forked repo with pattern.

    • Deploying the cluster by using the pattern.sh file

    To deploy the cluster by using the pattern.sh file, complete the following steps:

    1. Login to your cluster by running the following command:

      oc login

      Optional: Set the KUBECONFIG variable for the kubeconfig file path:

      export KUBECONFIG=~/<path_to_kubeconfig>

    2. Deploy the pattern to your cluster. Run the following command:

      ./pattern.sh make install

    Verify that the Operators have been installed.

    1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

    2. Check that the following Operators are installed with Succeeded status (Figure 1):

      • Advanced Cluster Management for Kubernetes

      • Intel Device Plugins Operator

      • multicluster engine for Kubernetes

      • Node Feature Discovery Operator

      • Red Hat Openshift GitOps

      • Validated Patterns Operator

        Multicloud GitOps Hub with SGX installed operators
        Figure 1. List of Installed Operators for Multicloud GitOps Validated Pattern with SGX

    Deploying the cluster by using the Validated Patterns Operator

    To install the Validated Patterns Operator:

    1. Log in to the Openshift Container Platform web console and select Operators > OperatorHub.

    2. Search for Validated Patterns Operator, open it and click Install.

      Install Validated Patterns Operator
      Figure 2. Install Validated Patterns Operator

    3. Choose default settings for the installation mode, namespaces and update strategy and confirm it by clicking Install.

    4. Select Operators > Installed Operators.

    5. Ensure that Validated Patterns Operator is listed in the openshift-operators project with a status Succeeded.

    After a successful installation open the Validated Patterns Operator page. Next, go to Pattern tab and click Create Pattern.

    1. Set the Name field to multicloud-gitops-sgx-hello-world, and the Cluster Group Name to hub. Values must be the same as in the values-global.yaml file (Figure 3).

    2. As a Git Config > Target Repo value, paste the link to your fork. Under Git Config > Target Revision write the name of your branch (Figure 3).

    3. Click the Create button to create the pattern.

      Create Pattern Form
      Figure 3. Create Pattern Form

    Verify that the rest of Operators have been installed:

    1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

    2. Check that the following Operators are installed with Succeeded status (Figure 1):

      • Advanced Cluster Management for Kubernetes

      • Intel Device Plugins Operator

      • multicluster engine for Kubernetes

      • Node Feature Discovery Operator

      • Red Hat Openshift GitOps

    Add a secret for config-demo application (from values-secret-multicloud-gitops.yaml) to Vault manually:

    1. Go to Vault service route. URL can be found:

      1. by running command:

        oc -n vault get route vault -ojsonpath='{.spec.host}'

      2. in Openshift Container Platform web console under Networking > Routes for vault project.

    2. Log into the Vault using root token. Root token can be found by executing command:

      oc -n imperative get secrets vaultkeys -ojsonpath='{.data.vault_data_json}' | base64 -d

    3. After login go to secret catalog and clik Create secret and fill all the fields manually (Figure 4):

      1. Put global/config-demo value in the Path for this secret field (the value comes from values-secret-multicloud-gitops.yaml file).

      2. Add one Secret data key-value pair. Put secret as a key (left field). Click Add button to confirm.

      3. Click Save to save changes.

      Create secret
      Figure 4. Create secret

    Verification

    Go to the Hub ArgoCD and verify that all applications are synchronized. The URL can be found in Openshift Container Platform web console under Networking > Routes for the project multicloud-gitops-sgx-hello-world-hub or use command:

    oc -n multicloud-gitops-sgx-hello-world-hub get route hub-gitops-server -ojsonpath='{.spec.host}'

    All applications should be Healthy and Synced:

    ArgoCD panel with hello-world-sgx app
    Figure 5. ArgoCD panel with hello-world-sgx app

    Check the logs of a pod hello-world-sgx to verify if it uses Intel SGX. In the OpenShift Container Platform web console, navigate to Workloads > Pods. Change project to hello-world-sgx and open the Logs tab in the pod details. The Gramine output confirms that Intel SGX is used.

    In case of pod hello-world-sgx crashing, delete it manually to be recreated - not restarted.

    As part of this pattern, HashiCorp Vault has been installed. Refer to the section on Vault.

    Next steps

    After the management hub is set up and works correctly, attach one or more managed clusters to the architecture.

    For instructions on deploying the edge, refer to Attach a managed cluster (edge) to the management hub.

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Ensure that you commit the changes and push them to GitHub so that GitOps can fetch your changes and apply them.

    Deploying a managed cluster by using Red Hat Advanced Cluster Management

    Prerequisites

    • An OpenShift cluster

    • Red Hat Advanced Cluster Management (RHACM) web console to join the managed cluster to the management hub

      After RHACM is installed, a message regarding a Web console update is available" might be displayed. Follow the instructions and click the Refresh web console link.

    Procedure

    1. In the left navigation panel of web console, click local-cluster. Select All Clusters. The RHACM web console is displayed with Cluster* on the left navigation panel.

    2. On the Managed clusters tab, click Import cluster.

    3. On the Import an existing cluster page, enter the cluster name and choose KUBECONFIG as the "import mode". Add the tag clusterGroup=region-one. Click Import.

    Now that RHACM is no longer deploying the managed cluster applications everywhere, you must indicate that the new cluster has the managed cluster role.

    Optional: Deploying a managed cluster by using cm-cli tool

    Prerequisites
    Procedure

    1. Obtain the KUBECONFIG file from the managed cluster.

    2. Open a shell prompt and login into the management hub cluster by using either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Run the following command:

      cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-path_to_kubeconfig>
    Next steps

    Optional: Deploying a managed cluster by using the clusteradm tool

    Prerequisites
    Procedure

    1. To deploy an edge cluster, you must to get the token from the management hub cluster. Run the following command on the existing management hub or datacenter cluster:

      clusteradm get token

      The command generates a token and shows you the command to use on the managed cluster.

    2. Login to the managed cluster with either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. To request that the managed join the hub cluster, run the following command:

      clusteradm join --hub-token <token_from_clusteradm_get_token_command> <managed_cluster_name>

    4. Accept the join request on the hub cluster:

      clusteradm accept --clusters <managed_cluster_name>
    Next steps

    Designate the new cluster as a managed cluster site

    If you use the command line tools such as clusteradm or cm-cli, you must explicitly indicate that the imported cluster is part of a specific clusterGroup. Some examples of clusterGroup are factory, devel, or prod.

    To tag the cluster as clusterGroup=<managed-cluster-group>, complete the following steps.

    Procedure

    1. To find the new cluster, run the following command:

      oc get managedcluster.cluster.open-cluster-management.io

    2. To apply the label, run the following command:

      oc label managedcluster.cluster.open-cluster-management.io/YOURCLUSTER site=managed-cluster

    Verification

    Go to your managed cluster (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. It might take a while for the RHACM agent and agent-addons to launch. After that, the OpenShift GitOps Operator is installed. On successful installation, launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    QUICKLINKS

    Ensure that you commit the changes and push them to GitHub so that GitOps can fetch your changes and apply them.

    Deploying a managed cluster by using Red Hat Advanced Cluster Management

    Prerequisites

    • An OpenShift cluster

    • Red Hat Advanced Cluster Management (RHACM) web console to join the managed cluster to the management hub

      After RHACM is installed, a message regarding a Web console update is available" might be displayed. Follow the instructions and click the Refresh web console link.

    Procedure

    1. In the left navigation panel of web console, click local-cluster. Select All Clusters. The RHACM web console is displayed with Cluster* on the left navigation panel.

    2. On the Managed clusters tab, click Import cluster.

    3. On the Import an existing cluster page, enter the cluster name and choose KUBECONFIG as the "import mode". Add the tag clusterGroup=region-one. Click Import.

    Now that RHACM is no longer deploying the managed cluster applications everywhere, you must indicate that the new cluster has the managed cluster role.

    Optional: Deploying a managed cluster by using cm-cli tool

    Prerequisites
    Procedure

    1. Obtain the KUBECONFIG file from the managed cluster.

    2. Open a shell prompt and login into the management hub cluster by using either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Run the following command:

      cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-path_to_kubeconfig>
    Next steps

    Optional: Deploying a managed cluster by using the clusteradm tool

    Prerequisites
    Procedure

    1. To deploy an edge cluster, you must to get the token from the management hub cluster. Run the following command on the existing management hub or datacenter cluster:

      clusteradm get token

      The command generates a token and shows you the command to use on the managed cluster.

    2. Login to the managed cluster with either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. To request that the managed join the hub cluster, run the following command:

      clusteradm join --hub-token <token_from_clusteradm_get_token_command> <managed_cluster_name>

    4. Accept the join request on the hub cluster:

      clusteradm accept --clusters <managed_cluster_name>
    Next steps

    Designate the new cluster as a managed cluster site

    If you use the command line tools such as clusteradm or cm-cli, you must explicitly indicate that the imported cluster is part of a specific clusterGroup. Some examples of clusterGroup are factory, devel, or prod.

    To tag the cluster as clusterGroup=<managed-cluster-group>, complete the following steps.

    Procedure

    1. To find the new cluster, run the following command:

      oc get managedcluster.cluster.open-cluster-management.io

    2. To apply the label, run the following command:

      oc label managedcluster.cluster.open-cluster-management.io/YOURCLUSTER site=managed-cluster

    Verification

    Go to your managed cluster (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. It might take a while for the RHACM agent and agent-addons to launch. After that, the OpenShift GitOps Operator is installed. On successful installation, launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    QUICKLINKS

    CONTRIBUTE

    This is a sample usage of the SGX application, as it is, and should not be used in production; unless the above warnings related to sgx.allowed_files related to Gramine are resolved.

    Next steps

    QUICKLINKS

    This is a sample usage of the SGX application, as it is, and should not be used in production; unless the above warnings related to sgx.allowed_files related to Gramine are resolved.

    Next steps

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    CONTRIBUTE

  • Generate the signing key:

    openssl genrsa -3 -out enclave-key.pem 3072

  • Build a protected image using the gsc command:

    ./gsc build vault-without-sgx vault.manifest

  • Sign the graminized docker image:

    ./gsc sign-image vault-without-sgx enclave-key.pem

  • (Optional) Log in to an image registry server (if the registry requires it):

    docker login -u=<USER_NAME> docker.io

    <USER_NAME> is the name of a user, which will be used to log in to the image registry server

    Tag protected container image with appropriate tag:

    docker tag gsc-vault-without-sgx <DESTINATION_IMAGE_NAME>

    <DESTINATION_IMAGE_NAME> is the name of the image in a registry

    Push protected container image to a registry:

    docker push <DESTINATION_IMAGE_NAME>

    replace <DESTINATION_IMAGE_NAME> with proper name

    • Deploying the cluster by using the pattern.sh file

    To deploy the cluster by using the pattern.sh file, complete the following steps:

    1. Log in to your cluster by running the following command:

      oc login

    2. Optional: Set the KUBECONFIG variable for the kubeconfig file path:

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Deploy the pattern to your cluster. Run the following command:

      ./pattern.sh make install

    Verify that the Operators have been installed.

    1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

    2. Check that the following Operators are installed with Succeeded status (Figure 1):

      • Advanced Cluster Management for Kubernetes

      • Intel Device Plugins Operator

      • multicluster engine for Kubernetes

      • Node Feature Discovery Operator

      • Red Hat Openshift GitOps

      • Validated Patterns Operator

        Multicloud GitOps Hub with SGX installed operators
        Figure 1. List of Installed Operators for Multicloud GitOps Validated Pattern with SGX

    Deploying the cluster by using the Validated Patterns Operator

    To install the Validated Patterns Operator:

    1. Log in to the Openshift Container Platform web console and select Operators > OperatorHub.

    2. Search for Validated Patterns Operator, open it and click Install.

      Install Validated Patterns Operator
      Figure 2. Install Validated Patterns Operator

    3. Choose default settings for the installation mode, namespaces and update strategy and confirm it by clicking Install.

    4. Select Operators > Installed Operators.

    5. Ensure that Validated Patterns Operator is listed in the openshift-operators project with a status Succeeded.

    After a successful installation, open the Validated Patterns Operator page. Next, go to the Pattern tab and click Create Pattern.

    1. Set the Name field to multicloud-gitops-sgx, and the Cluster Group Name to hub. Values must be the same as in the values-global.yaml file (Figure 3).

    2. As a Git Config > Target Repo value, paste the link to your fork. Under Git Config > Target Revision write the name of your branch (Figure 3).

    3. Click the Create button to create the pattern.

      Create pattern Multicloud GitOps with SGX
      Figure 3. Create Pattern Form

    Verify that the rest of Operators have been installed:

    1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

    2. Check that the following Operators are installed with Succeeded status (Figure 1):

      • Advanced Cluster Management for Kubernetes

      • Intel Device Plugins Operator

      • multicluster engine for Kubernetes

      • Node Feature Discovery Operator

      • Red Hat Openshift GitOps

    Add a secret for config-demo application (from values-secret-multicloud-gitops.yaml) to Vault manually:

    1. Go to the Vault service route. URL can be found:

      1. by running the command:

        oc -n vault get route vault -ojsonpath='{.spec.host}'

      2. in Openshift Container Platform web console under Networking > Routes for vault project.

    2. Log into the Vault using the root token. Root token can be found by executing the command:

      oc -n imperative get secrets vaultkeys -ojsonpath='{.data.vault_data_json}' | base64 -d

    3. After login go to secret catalog click Create secret and fill in all the fields manually (Figure 2):

      1. Path for this secret is global/config-demo (from values.yaml file for config-demo charts)

      2. Under Secret data key is secret (from values-secret-multicloud-gitops.yaml file) and in the next field put its value.

      3. Click Add and then Save.

        Create secret in the vault
        Figure 4. Create secret

    Verification

    Go to the Hub ArgoCD and verify that all applications are synchronized. The URL can be found in Openshift Container Platform web console under Networking > Routes for the project vault or use the command:

    oc -n vault get route vault -ojsonpath='{.spec.host}'

    All applications should be Healthy and Synced:

    Multicloud GitOps Hub with `vault` app
    Figure 5. ArgoCD panel with sgx-app

    Check the logs of a pod vault-0 to verify if it contains Gramine header. In the OpenShift Container Platform web console, navigate to Workloads > Pods. Change the project to vault and open the Logs tab in the pod details. The appearance of Gramine is starting header confirms that Intel SGX is used.

    Next steps

    After the management hub is set up and works correctly, attach one or more managed clusters to the architecture.

    For instructions on deploying the edge, refer to Attach a managed cluster (edge) to the management hub.

    QUICKLINKS

  • Generate the signing key:

    openssl genrsa -3 -out enclave-key.pem 3072

  • Build a protected image using the gsc command:

    ./gsc build vault-without-sgx vault.manifest

  • Sign the graminized docker image:

    ./gsc sign-image vault-without-sgx enclave-key.pem

  • (Optional) Log in to an image registry server (if the registry requires it):

    docker login -u=<USER_NAME> docker.io

    <USER_NAME> is the name of a user, which will be used to log in to the image registry server

    Tag protected container image with appropriate tag:

    docker tag gsc-vault-without-sgx <DESTINATION_IMAGE_NAME>

    <DESTINATION_IMAGE_NAME> is the name of the image in a registry

    Push protected container image to a registry:

    docker push <DESTINATION_IMAGE_NAME>

    replace <DESTINATION_IMAGE_NAME> with proper name

    • Deploying the cluster by using the pattern.sh file

    To deploy the cluster by using the pattern.sh file, complete the following steps:

    1. Log in to your cluster by running the following command:

      oc login

    2. Optional: Set the KUBECONFIG variable for the kubeconfig file path:

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Deploy the pattern to your cluster. Run the following command:

      ./pattern.sh make install

    Verify that the Operators have been installed.

    1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

    2. Check that the following Operators are installed with Succeeded status (Figure 1):

      • Advanced Cluster Management for Kubernetes

      • Intel Device Plugins Operator

      • multicluster engine for Kubernetes

      • Node Feature Discovery Operator

      • Red Hat Openshift GitOps

      • Validated Patterns Operator

        Multicloud GitOps Hub with SGX installed operators
        Figure 1. List of Installed Operators for Multicloud GitOps Validated Pattern with SGX

    Deploying the cluster by using the Validated Patterns Operator

    To install the Validated Patterns Operator:

    1. Log in to the Openshift Container Platform web console and select Operators > OperatorHub.

    2. Search for Validated Patterns Operator, open it and click Install.

      Install Validated Patterns Operator
      Figure 2. Install Validated Patterns Operator

    3. Choose default settings for the installation mode, namespaces and update strategy and confirm it by clicking Install.

    4. Select Operators > Installed Operators.

    5. Ensure that Validated Patterns Operator is listed in the openshift-operators project with a status Succeeded.

    After a successful installation, open the Validated Patterns Operator page. Next, go to the Pattern tab and click Create Pattern.

    1. Set the Name field to multicloud-gitops-sgx, and the Cluster Group Name to hub. Values must be the same as in the values-global.yaml file (Figure 3).

    2. As a Git Config > Target Repo value, paste the link to your fork. Under Git Config > Target Revision write the name of your branch (Figure 3).

    3. Click the Create button to create the pattern.

      Create pattern Multicloud GitOps with SGX
      Figure 3. Create Pattern Form

    Verify that the rest of Operators have been installed:

    1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

    2. Check that the following Operators are installed with Succeeded status (Figure 1):

      • Advanced Cluster Management for Kubernetes

      • Intel Device Plugins Operator

      • multicluster engine for Kubernetes

      • Node Feature Discovery Operator

      • Red Hat Openshift GitOps

    Add a secret for config-demo application (from values-secret-multicloud-gitops.yaml) to Vault manually:

    1. Go to the Vault service route. URL can be found:

      1. by running the command:

        oc -n vault get route vault -ojsonpath='{.spec.host}'

      2. in Openshift Container Platform web console under Networking > Routes for vault project.

    2. Log into the Vault using the root token. Root token can be found by executing the command:

      oc -n imperative get secrets vaultkeys -ojsonpath='{.data.vault_data_json}' | base64 -d

    3. After login go to secret catalog click Create secret and fill in all the fields manually (Figure 2):

      1. Path for this secret is global/config-demo (from values.yaml file for config-demo charts)

      2. Under Secret data key is secret (from values-secret-multicloud-gitops.yaml file) and in the next field put its value.

      3. Click Add and then Save.

        Create secret in the vault
        Figure 4. Create secret

    Verification

    Go to the Hub ArgoCD and verify that all applications are synchronized. The URL can be found in Openshift Container Platform web console under Networking > Routes for the project vault or use the command:

    oc -n vault get route vault -ojsonpath='{.spec.host}'

    All applications should be Healthy and Synced:

    Multicloud GitOps Hub with `vault` app
    Figure 5. ArgoCD panel with sgx-app

    Check the logs of a pod vault-0 to verify if it contains Gramine header. In the OpenShift Container Platform web console, navigate to Workloads > Pods. Change the project to vault and open the Logs tab in the pod details. The appearance of Gramine is starting header confirms that Intel SGX is used.

    Next steps

    After the management hub is set up and works correctly, attach one or more managed clusters to the architecture.

    For instructions on deploying the edge, refer to Attach a managed cluster (edge) to the management hub.

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Ensure that you commit the changes and push them to GitHub so that GitOps can fetch your changes and apply them.

    Deploying a managed cluster by using Red Hat Advanced Cluster Management

    Prerequisites

    • An OpenShift cluster

    • Red Hat Advanced Cluster Management (RHACM) web console to join the managed cluster to the management hub

      After RHACM is installed, a message regarding a Web console update is available" might be displayed. Follow the instructions and click the Refresh web console link.

    Procedure

    1. In the left navigation panel of web console, click local-cluster. Select All Clusters. The RHACM web console is displayed with Cluster* on the left navigation panel.

    2. On the Managed clusters tab, click Import cluster.

    3. On the Import an existing cluster page, enter the cluster name and choose KUBECONFIG as the "import mode". Add the tag clusterGroup=region-one. Click Import.

    Now that RHACM is no longer deploying the managed cluster applications everywhere, you must indicate that the new cluster has the managed cluster role.

    Optional: Deploying a managed cluster by using cm-cli tool

    Prerequisites
    Procedure

    1. Obtain the KUBECONFIG file from the managed cluster.

    2. Open a shell prompt and login into the management hub cluster by using either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Run the following command:

      cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-path_to_kubeconfig>
    Next steps

    Optional: Deploying a managed cluster by using the clusteradm tool

    Prerequisites
    Procedure

    1. To deploy an edge cluster, you must to get the token from the management hub cluster. Run the following command on the existing management hub or datacenter cluster:

      clusteradm get token

      The command generates a token and shows you the command to use on the managed cluster.

    2. Login to the managed cluster with either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. To request that the managed join the hub cluster, run the following command:

      clusteradm join --hub-token <token_from_clusteradm_get_token_command> <managed_cluster_name>

    4. Accept the join request on the hub cluster:

      clusteradm accept --clusters <managed_cluster_name>
    Next steps

    Designate the new cluster as a managed cluster site

    If you use the command line tools such as clusteradm or cm-cli, you must explicitly indicate that the imported cluster is part of a specific clusterGroup. Some examples of clusterGroup are factory, devel, or prod.

    To tag the cluster as clusterGroup=<managed-cluster-group>, complete the following steps.

    Procedure

    1. To find the new cluster, run the following command:

      oc get managedcluster.cluster.open-cluster-management.io

    2. To apply the label, run the following command:

      oc label managedcluster.cluster.open-cluster-management.io/YOURCLUSTER site=managed-cluster

    Verification

    Go to your managed cluster (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. It might take a while for the RHACM agent and agent-addons to launch. After that, the OpenShift GitOps Operator is installed. On successful installation, launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    QUICKLINKS

    Ensure that you commit the changes and push them to GitHub so that GitOps can fetch your changes and apply them.

    Deploying a managed cluster by using Red Hat Advanced Cluster Management

    Prerequisites

    • An OpenShift cluster

    • Red Hat Advanced Cluster Management (RHACM) web console to join the managed cluster to the management hub

      After RHACM is installed, a message regarding a Web console update is available" might be displayed. Follow the instructions and click the Refresh web console link.

    Procedure

    1. In the left navigation panel of web console, click local-cluster. Select All Clusters. The RHACM web console is displayed with Cluster* on the left navigation panel.

    2. On the Managed clusters tab, click Import cluster.

    3. On the Import an existing cluster page, enter the cluster name and choose KUBECONFIG as the "import mode". Add the tag clusterGroup=region-one. Click Import.

    Now that RHACM is no longer deploying the managed cluster applications everywhere, you must indicate that the new cluster has the managed cluster role.

    Optional: Deploying a managed cluster by using cm-cli tool

    Prerequisites
    Procedure

    1. Obtain the KUBECONFIG file from the managed cluster.

    2. Open a shell prompt and login into the management hub cluster by using either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Run the following command:

      cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-path_to_kubeconfig>
    Next steps

    Optional: Deploying a managed cluster by using the clusteradm tool

    Prerequisites
    Procedure

    1. To deploy an edge cluster, you must to get the token from the management hub cluster. Run the following command on the existing management hub or datacenter cluster:

      clusteradm get token

      The command generates a token and shows you the command to use on the managed cluster.

    2. Login to the managed cluster with either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. To request that the managed join the hub cluster, run the following command:

      clusteradm join --hub-token <token_from_clusteradm_get_token_command> <managed_cluster_name>

    4. Accept the join request on the hub cluster:

      clusteradm accept --clusters <managed_cluster_name>
    Next steps

    Designate the new cluster as a managed cluster site

    If you use the command line tools such as clusteradm or cm-cli, you must explicitly indicate that the imported cluster is part of a specific clusterGroup. Some examples of clusterGroup are factory, devel, or prod.

    To tag the cluster as clusterGroup=<managed-cluster-group>, complete the following steps.

    Procedure

    1. To find the new cluster, run the following command:

      oc get managedcluster.cluster.open-cluster-management.io

    2. To apply the label, run the following command:

      oc label managedcluster.cluster.open-cluster-management.io/YOURCLUSTER site=managed-cluster

    Verification

    Go to your managed cluster (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. It might take a while for the RHACM agent and agent-addons to launch. After that, the OpenShift GitOps Operator is installed. On successful installation, launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    QUICKLINKS

    CONTRIBUTE

    About the Multicloud GitOps pattern

    Use case

    • Use a GitOps approach to manage hybrid and multi-cloud deployments across both public and private clouds.

    • Enable cross-cluster governance and application lifecycle management.

    • Securely manage secrets across the deployment.

      Based on the requirements of a specific implementation, certain details might differ. However, all validated patterns that are based on a portfolio architecture, generalize one or more successful deployments of a use case.

    Background

    Organizations are aiming to develop, deploy, and operate applications on an open hybrid cloud in a stable, simple, and secure way. This hybrid strategy includes multi-cloud deployments where workloads might be running on multiple clusters and on multiple clouds, private or public. -This strategy requires an infrastructure-as-code approach: GitOps. GitOps uses Git repositories as a single source of truth to deliver infrastructure-as-code. Submitted code will be checked by the continuous integration (CI) process, while the continuous delivery (CD) process checks and applies requirements for things like security, infrastructure-as-code, or any other boundaries set for the application framework. All changes to code are tracked, making updates easy while also providing version control should a rollback be needed.

    About the solution

    This architecture covers hybrid and multi-cloud management with GitOps as shown in following figure. At a high level this requires a management hub, for DevOps and GitOps, and infrastructure that extends to one or more managed clusters running on private or public clouds. The automated infrastructure-as-code approach can manage the versioning of components and deploy according to the infrastructure-as-code configuration.

    Benefits of Hybrid Multicloud management with GitOps:

    • Unify management across cloud environments.

    • Dynamic infrastructure security.

    • Infrastructural continuous delivery best practices.

    Multicloud Architecture
    Figure 1. Overview of the solution including the business drivers, management hub, and the clusters under management

    In the following figure, logically, this solution can be viewed as being composed of an automation component, unified management including secrets management, and the clusters under management, all running on top of a user-chosen mixture of on-premise data centers and public clouds.

    Logical Architecture
    Figure 2. Logical diagram of hybrid multi-cloud management with GitOps

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Advanced Cluster Management for Kubernetes

    Controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Overview of the architectures

    The following figure provides a schematic diagram overview of the complete solution including both components and data flows.

    Physical Architecture
    Figure 3. Overview schematic diagram of the complete solution

    Subsequent schematic diagrams provide details on:

    • Bootstrapping the management hub (Figure 4)

    • Hybrid multi-cloud GitOps (Figure 5)

    • Dynamic security management (Figure 6)

    Bootstrapping the management hub

    The following figure provides a schematic diagram showing the setup of the management hub using Ansible playbooks.

    Schematic diagram of bootstrapping the management hub
    Figure 4. Schematic diagram of bootstrapping the management hub

    • Set up the OpenShift Container Platform that hosts the Management Hub. The OpenShift installation program provides flexible ways to install OpenShift. An Ansible playbook starts the installation with necessary configurations.

    • Ansible playbooks deploy and configure Red Hat Advanced Cluster Management for Kubernetes and, later, other supporting components such as external secrets management, on top of the provisioned OpenShift cluster.

    • Another Ansible playbook installs HashiCorp Vault, a Red Hat partner product chosen for this solution that can be used to manage secrets for OpenShift clusters.

    • An Ansible playbook configures and installs the Openshift GitOps operator on the hub cluster. This deploys the Openshift GitOps instance to enable continuous delivery.

    Hybrid Multicloud GitOps

    The following figure provides a schematic diagram showing remaining activities associated with setting up the management hub and clusters using Red Hat Advanced Cluster Management.

    Schematic diagram of hybrid multi-cloud management with GitOps
    Figure 5. Schematic diagram of hybrid multi-cloud management with GitOps

    • Manifest and configuration are set as code template in the form of a Kustomization YAML file. The file describes the desired end state of the managed cluster. When complete, the Kustomization YAML file is pushed into the source control management repository with a version assigned to each update.

    • OpenShift GitOps monitors the repository and detects changes in the repository.

    • OpenShift GitOps creates and updates the manifest by creating Kubernetes objects on top of Red Hat Advanced Cluster Management.

    • Red Hat Advanced Cluster Management provisions, updates, or deletes managed clusters and configuration according to the manifest. In the manifest, you can configure what cloud provider the cluster will be on, the name of the cluster, infrastructure node details and worker node. Governance policy can also be applied as well as provision an agent in the cluster as the bridge between the control center and the managed cluster.

    • OpenShift GitOps continuously monitors the code repository and the status of the clusters reported back to Red Hat Advanced Cluster Management. Any configuration drift or in case of any failure, OpenShift GitOps will automatically try to remediate by applying the manifest or by displaying alerts for manual intervention.

    Dynamic security management

    The following figure provides a schematic diagram showing how secrets are handled in this solution.

    Schematic showing the setup and use of external secrets management
    Figure 6. Schematic showing the setup and use of external secrets management

    • During setup, the token to securely access HashiCorp Vault is stored in Ansible Vault. It is encrypted to protect sensitive content.

    • Red Hat Advanced Cluster Management for Kubernetes acquires the token from Ansible Vault during install and distributes it among the clusters. As a result, you have centralized control over the managed clusters through RHACM.

    • To allow the cluster access to the external vault, you must set up the external secret management with Helm in this study. OpenShift Gitops is used to deploy the external secret object to a managed cluster.

    • External secret management fetches secrets from HashiCorp Vault by using the token that was generated in step 2 and constantly monitors for updates.

    • Secrets are created in each namespace, where applications can use them.

    Additional resources

    View and download all of the diagrams above from the Red Hat Portfolio Architecture open source tooling site.

    Presentation

    View a short presentation slide deck about Multicloud GitOps here

    Next steps

    QUICKLINKS

    About the solution

    This architecture covers hybrid and multi-cloud management with GitOps as shown in following figure. At a high level this requires a management hub, for DevOps and GitOps, and infrastructure that extends to one or more managed clusters running on private or public clouds. The automated infrastructure-as-code approach can manage the versioning of components and deploy according to the infrastructure-as-code configuration.

    Benefits of Hybrid Multicloud management with GitOps:

    • Unify management across cloud environments.

    • Dynamic infrastructure security.

    • Infrastructural continuous delivery best practices.

    Multicloud Architecture
    Figure 1. Overview of the solution including the business drivers, management hub, and the clusters under management

    In the following figure, logically, this solution can be viewed as being composed of an automation component, unified management including secrets management, and the clusters under management, all running on top of a user-chosen mixture of on-premise data centers and public clouds.

    Logical Architecture
    Figure 2. Logical diagram of hybrid multi-cloud management with GitOps

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Advanced Cluster Management for Kubernetes

    Controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Overview of the architectures

    The following figure provides a schematic diagram overview of the complete solution including both components and data flows.

    Physical Architecture
    Figure 3. Overview schematic diagram of the complete solution

    Subsequent schematic diagrams provide details on:

    • Bootstrapping the management hub (Figure 4)

    • Hybrid multi-cloud GitOps (Figure 5)

    • Dynamic security management (Figure 6)

    Bootstrapping the management hub

    The following figure provides a schematic diagram showing the setup of the management hub using Ansible playbooks.

    Schematic diagram of bootstrapping the management hub
    Figure 4. Schematic diagram of bootstrapping the management hub

    • Set up the OpenShift Container Platform that hosts the Management Hub. The OpenShift installation program provides flexible ways to install OpenShift. An Ansible playbook starts the installation with necessary configurations.

    • Ansible playbooks deploy and configure Red Hat Advanced Cluster Management for Kubernetes and, later, other supporting components such as external secrets management, on top of the provisioned OpenShift cluster.

    • Another Ansible playbook installs HashiCorp Vault, a Red Hat partner product chosen for this solution that can be used to manage secrets for OpenShift clusters.

    • An Ansible playbook configures and installs the Openshift GitOps operator on the hub cluster. This deploys the Openshift GitOps instance to enable continuous delivery.

    Hybrid Multicloud GitOps

    The following figure provides a schematic diagram showing remaining activities associated with setting up the management hub and clusters using Red Hat Advanced Cluster Management.

    Schematic diagram of hybrid multi-cloud management with GitOps
    Figure 5. Schematic diagram of hybrid multi-cloud management with GitOps

    • Manifest and configuration are set as code template in the form of a Kustomization YAML file. The file describes the desired end state of the managed cluster. When complete, the Kustomization YAML file is pushed into the source control management repository with a version assigned to each update.

    • OpenShift GitOps monitors the repository and detects changes in the repository.

    • OpenShift GitOps creates and updates the manifest by creating Kubernetes objects on top of Red Hat Advanced Cluster Management.

    • Red Hat Advanced Cluster Management provisions, updates, or deletes managed clusters and configuration according to the manifest. In the manifest, you can configure what cloud provider the cluster will be on, the name of the cluster, infrastructure node details and worker node. Governance policy can also be applied as well as provision an agent in the cluster as the bridge between the control center and the managed cluster.

    • OpenShift GitOps continuously monitors the code repository and the status of the clusters reported back to Red Hat Advanced Cluster Management. Any configuration drift or in case of any failure, OpenShift GitOps will automatically try to remediate by applying the manifest or by displaying alerts for manual intervention.

    Dynamic security management

    The following figure provides a schematic diagram showing how secrets are handled in this solution.

    Schematic showing the setup and use of external secrets management
    Figure 6. Schematic showing the setup and use of external secrets management

    • During setup, the token to securely access HashiCorp Vault is stored in Ansible Vault. It is encrypted to protect sensitive content.

    • Red Hat Advanced Cluster Management for Kubernetes acquires the token from Ansible Vault during install and distributes it among the clusters. As a result, you have centralized control over the managed clusters through RHACM.

    • To allow the cluster access to the external vault, you must set up the external secret management with Helm in this study. OpenShift Gitops is used to deploy the external secret object to a managed cluster.

    • External secret management fetches secrets from HashiCorp Vault by using the token that was generated in step 2 and constantly monitors for updates.

    • Secrets are created in each namespace, where applications can use them.

    Additional resources

    View and download all of the diagrams above from the Red Hat Portfolio Architecture open source tooling site.

    Presentation

    View a short presentation slide deck about Multicloud GitOps here

    Next steps

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    CONTRIBUTE

    Deploying the Multicloud GitOps pattern

    Prerequisites

    The use of this pattern depends on having at least one running Red Hat OpenShift cluster. However, consider creating a cluster for deploying the GitOps management hub assets and a separate cluster for the managed cluster.

    If you do not have a running Red Hat OpenShift cluster, you can start one on a -public or private cloud by using Red Hat Hybrid Cloud Console.

    Procedure

    1. Fork the multicloud-gitops repository on GitHub.

    2. Clone the forked copy of this repository.

      git clone git@github.com:your-username/multicloud-gitops.git

    3. Create a local copy of the secret values file that can safely include credentials. Run the following commands:

      cp values-secret.yaml.template ~/values-secret-multicloud-gitops.yaml
      vi ~/values-secret-multicloud-gitops.yaml

      Do not commit this file. You do not want to push personal credentials to GitHub. If you do not want to customize the secrets, these steps are not needed. The framework generates a random password for the config-demo application.

    4. Customize the deployment for your cluster. Run the following command:

      git checkout -b my-branch
      vi values-global.yaml
      git add values-global.yaml
      git commit values-global.yaml
      git push origin my-branch

    5. Deploy the pattern by running ./pattern.sh make install or by using the Validated Patterns Operator.

    Deploying the cluster by using the pattern.sh file

    To deploy the cluster by using the pattern.sh file, complete the following steps:

    1. Login to your cluster by running the following command:

       oc login

      Optional: Set the KUBECONFIG variable for the kubeconfig file path:

       export KUBECONFIG=~/<path_to_kubeconfig>

    2. Deploy the pattern to your cluster. Run the following command:

       ./pattern.sh make install

    3. Verify that the Operators have been installed.

      1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

      2. Check that the Operator is installed in the openshift-operators namespace and its status is Succeeded.

    4. Verify that all applications are synchronized. Under the project multicloud-gitops-hub click the URL for the hub gitops server. The Vault application is not synched.

      Multicloud GitOps Hub

    As part of this pattern, HashiCorp Vault has been installed. Refer to the section on Vault.

    Next steps

    After the management hub is set up and works correctly, attach one or more managed clusters to the architecture.

    For instructions on deploying the edge, refer to Attach a managed cluster (edge) to the management hub.

    QUICKLINKS

    Procedure

    1. Fork the multicloud-gitops repository on GitHub.

    2. Clone the forked copy of this repository.

      git clone git@github.com:your-username/multicloud-gitops.git

    3. Create a local copy of the secret values file that can safely include credentials. Run the following commands:

      cp values-secret.yaml.template ~/values-secret-multicloud-gitops.yaml
      vi ~/values-secret-multicloud-gitops.yaml

      Do not commit this file. You do not want to push personal credentials to GitHub. If you do not want to customize the secrets, these steps are not needed. The framework generates a random password for the config-demo application.

    4. Customize the deployment for your cluster. Run the following command:

      git checkout -b my-branch
      vi values-global.yaml
      git add values-global.yaml
      git commit values-global.yaml
      git push origin my-branch

    5. Deploy the pattern by running ./pattern.sh make install or by using the Validated Patterns Operator.

    Deploying the cluster by using the pattern.sh file

    To deploy the cluster by using the pattern.sh file, complete the following steps:

    1. Login to your cluster by running the following command:

       oc login

      Optional: Set the KUBECONFIG variable for the kubeconfig file path:

       export KUBECONFIG=~/<path_to_kubeconfig>

    2. Deploy the pattern to your cluster. Run the following command:

       ./pattern.sh make install

    3. Verify that the Operators have been installed.

      1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

      2. Check that the Operator is installed in the openshift-operators namespace and its status is Succeeded.

    4. Verify that all applications are synchronized. Under the project multicloud-gitops-hub click the URL for the hub gitops server. The Vault application is not synched.

      Multicloud GitOps Hub

    As part of this pattern, HashiCorp Vault has been installed. Refer to the section on Vault.

    Next steps

    After the management hub is set up and works correctly, attach one or more managed clusters to the architecture.

    For instructions on deploying the edge, refer to Attach a managed cluster (edge) to the management hub.

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Ensure that you commit the changes and push them to GitHub so that GitOps can fetch your changes and apply them.

    Deploying a managed cluster by using Red Hat Advanced Cluster Management

    Prerequistes

    • An OpenShift cluster

    • Red Hat Advanced Cluster Management (RHACM) web console to join the managed cluster to the management hub

      After RHACM is installed, a message regarding a Web console update is available" might be displayed. Follow the instructions and click the Refresh web console link.

    Procedure

    1. In the left navigation panel of web console, click local-cluster. Select All Clusters. The RHACM web console is displayed with Cluster* on the left navigation panel.

    2. On the Managed clusters tab, click Import cluster.

    3. On the Import an existing cluster page, enter the cluster name and choose KUBECONFIG as the "import mode". Add the tag clusterGroup=region-one. Click Import.

    Now that RHACM is no longer deploying the managed cluster applications everywhere, you must indicate that the new cluster has the managed cluster role.

    Optional: Deploying a managed cluster by using cm-cli tool

    Prerequistes
    Procedure

    1. Obtain the KUBECONFIG file from the managed cluster.

    2. Open a shell prompt and login into the management hub cluster by using either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Run the following command:

      cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-path_to_kubeconfig>
    Next steps

    Optional: Deploying a managed cluster by using the clusteradm tool

    Prerequistes
    Procedure

    1. To deploy an edge cluster, you must to get the token from the management hub cluster. Run the following command on the existing management hub or datacenter cluster:

      clusteradm get token

      The command generates a token and shows you the command to use on the managed cluster.

    2. Login to the managed cluster with either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. To request that the managed join the hub cluster, run the following command:

      clusteradm join --hub-token <token_from_clusteradm_get_token_command> <managed_cluster_name>

    4. Accept the join request on the hub cluster:

      clusteradm accept --clusters <managed_cluster_name>
    Next steps

    Designate the new cluster as a managed cluster site

    If you use the command line tools such as clusteradm or cm-cli, you must explicitly indicate that the imported cluster is part of a specific clusterGroup. Some examples of clusterGroup are factory, devel, or prod.

    To tag the cluster as clusterGroup=<managed-cluster-group>, complete the following steps.

    Procedure

    1. To find the new cluster, run the following command:

      oc get managedcluster.cluster.open-cluster-management.io

    2. To apply the label, run the following command:

      oc label managedcluster.cluster.open-cluster-management.io/YOURCLUSTER site=managed-cluster

    Verification

    Go to your managed cluster (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. It might take a while for the RHACM agent and agent-addons to launch. After that, the OpenShift GitOps Operator is installed. On successful installation, launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    QUICKLINKS

    Ensure that you commit the changes and push them to GitHub so that GitOps can fetch your changes and apply them.

    Deploying a managed cluster by using Red Hat Advanced Cluster Management

    Prerequistes

    • An OpenShift cluster

    • Red Hat Advanced Cluster Management (RHACM) web console to join the managed cluster to the management hub

      After RHACM is installed, a message regarding a Web console update is available" might be displayed. Follow the instructions and click the Refresh web console link.

    Procedure

    1. In the left navigation panel of web console, click local-cluster. Select All Clusters. The RHACM web console is displayed with Cluster* on the left navigation panel.

    2. On the Managed clusters tab, click Import cluster.

    3. On the Import an existing cluster page, enter the cluster name and choose KUBECONFIG as the "import mode". Add the tag clusterGroup=region-one. Click Import.

    Now that RHACM is no longer deploying the managed cluster applications everywhere, you must indicate that the new cluster has the managed cluster role.

    Optional: Deploying a managed cluster by using cm-cli tool

    Prerequistes
    Procedure

    1. Obtain the KUBECONFIG file from the managed cluster.

    2. Open a shell prompt and login into the management hub cluster by using either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. Run the following command:

      cm attach cluster --cluster <cluster-name> --cluster-kubeconfig <path-to-path_to_kubeconfig>
    Next steps

    Optional: Deploying a managed cluster by using the clusteradm tool

    Prerequistes
    Procedure

    1. To deploy an edge cluster, you must to get the token from the management hub cluster. Run the following command on the existing management hub or datacenter cluster:

      clusteradm get token

      The command generates a token and shows you the command to use on the managed cluster.

    2. Login to the managed cluster with either of the following methods:

      oc login

      or

      export KUBECONFIG=~/<path_to_kubeconfig>

    3. To request that the managed join the hub cluster, run the following command:

      clusteradm join --hub-token <token_from_clusteradm_get_token_command> <managed_cluster_name>

    4. Accept the join request on the hub cluster:

      clusteradm accept --clusters <managed_cluster_name>
    Next steps

    Designate the new cluster as a managed cluster site

    If you use the command line tools such as clusteradm or cm-cli, you must explicitly indicate that the imported cluster is part of a specific clusterGroup. Some examples of clusterGroup are factory, devel, or prod.

    To tag the cluster as clusterGroup=<managed-cluster-group>, complete the following steps.

    Procedure

    1. To find the new cluster, run the following command:

      oc get managedcluster.cluster.open-cluster-management.io

    2. To apply the label, run the following command:

      oc label managedcluster.cluster.open-cluster-management.io/YOURCLUSTER site=managed-cluster

    Verification

    Go to your managed cluster (edge) OpenShift console and check for the open-cluster-management-agent pod being launched. It might take a while for the RHACM agent and agent-addons to launch. After that, the OpenShift GitOps Operator is installed. On successful installation, launch the OpenShift GitOps (ArgoCD) console from the top right of the OpenShift console.

    QUICKLINKS

    CONTRIBUTE

    Deploying the Red Hat OpenShift AI Pattern

    Prerequisites

    The use of this pattern depends on having at least one running Red Hat OpenShift cluster. However, consider creating a cluster for deploying the GitOps management hub assets and a separate cluster for the managed cluster.

    If you do not have a running Red Hat OpenShift cluster, you can start one on a public or private cloud by using Red Hat Hybrid Cloud Console.

    Procedure

    1. Fork the openshift-ai repository on GitHub.

    2. Clone the forked copy of this repository.

      git clone git@github.com:your-username/openshift-ai.git

    1. If you want a peak under the covers to see what the pattern contains, you can do so with the following command:

      cat values-hub.yaml

    But don’t worry if it looks intimidating.

    1. Deploy the pattern by running ./pattern.sh make install or by using the Validated Patterns Operator.

    Deploying the cluster by using the pattern.sh file

    To deploy the cluster by using the pattern.sh file, complete the following steps:

    1. Login to your cluster by running the following command:

       oc login

      Optional: Set the KUBECONFIG variable for the kubeconfig file path:

       export KUBECONFIG=~/<path_to_kubeconfig>

    2. Deploy the pattern to your cluster. Run the following command:

       ./pattern.sh make install

    Verify Red Hat OpenShift AI Pattern installation

    1. Verify that the Operators have been installed.

      1. To verify, in the OpenShift Container Platform web console, navigate to Operators → Installed Operators page.

      2. Set your project to All Projects and verify the operators are isntalled and have a status of Succeeded.

    2. Verify that all applications are synchronized. Under the project openshift-ai-hub click the URL for the hub gitops server.

      ArgoCD Applications

      As part of this pattern, HashiCorp Vault has been installed. Refer to the section on Vault.

    Verify installation by checking the OpenShift AI Dashboard

    1. Access the OpenShift AI dashboard

      RHODS=https://$(oc get route -n redhat-ods-applications -o jsonpath='{.spec.host}')
-echo ${RHODS}

    You can also get to the dashboard from the OpenShift Console by selecting the application shortcut icon and then selecting the link for Red Hat OpenShift Ai

    Application ShortCut

    Log in to the Dashboard using your OpenShift credentials. You will find an environment that is ready for further configuration. This pattern provides the fundamental platform pieces to support MLOps workflows. The installation of OpenShift Pipelines enables the immediate use of pipelines if that is the desired approach for deployment.

    OpenShift AI Dashboard

    QUICKLINKS

    You can also get to the dashboard from the OpenShift Console by selecting the application shortcut icon and then selecting the link for Red Hat OpenShift Ai

    Application ShortCut

    Log in to the Dashboard using your OpenShift credentials. You will find an environment that is ready for further configuration. This pattern provides the fundamental platform pieces to support MLOps workflows. The installation of OpenShift Pipelines enables the immediate use of pipelines if that is the desired approach for deployment.

    OpenShift AI Dashboard

    QUICKLINKS

    CONTRIBUTE

    About the Red Hat OpenShift AI pattern

    Use case

    • Use a GitOps approach to manage hybrid and multi-cloud deployments across both public and private clouds.

    • Enable cross-cluster governance and application lifecycle management.

    • Securely manage secrets across the deployment.

      Based on the requirements of a specific implementation, certain details might differ. However, all validated patterns that are based on a portfolio architecture, generalize one or more successful deployments of a use case.

    Background

    The {rhoai-pattern} deployed using OpenShift GitOps and is comprised of Red Hat OpenShift AI and OpenShift Pipelines. This pattern deploys the minimum capabilities necessary for consumers to quickly start implementing their MLOps or AI workloads in an automated way.

    Organizations are aiming to develop, deploy, and operate applications on an open hybrid cloud in a stable, simple, and secure way. This hybrid strategy includes multi-cloud deployments where workloads might be running on multiple clusters and on multiple clouds, private or public. -This strategy requires an infrastructure-as-code approach: GitOps. GitOps uses Git repositories as a single source of truth to deliver infrastructure-as-code. Submitted code will be checked by the continuous integration (CI) process, while the continuous delivery (CD) process checks and applies requirements for things like security, infrastructure-as-code, or any other boundaries set for the application framework. All changes to code are tracked, making updates easy while also providing version control should a rollback be needed.

    About the solution

    This architecture covers a single cluster for all DevOps and GitOps functionality. However, one could extend this architecture to meet hybrid or multicloud demand using a GitOps approach

    Benefits of Hybrid Multicloud management with GitOps:

    • Unify management across cloud environments.

    • Dynamic infrastructure security.

    • Infrastructural continuous delivery best practices.

    In the following figure, logically, this solution can be viewed as being composed of an automation component, unified management including secrets management, and the clusters under management, all running on top of a user-chosen mixture of on-premise data centers and public clouds.

    Logical Architecture
    Figure 1. Logical diagram of hybrid multi-cloud management with GitOps

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Advanced Cluster Management for Kubernetes

    Controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat OpenShift AI

    A flexible, scalable MLOps platform with tools to build, deploy, and manage AI-enabled applications. Built using open source technologies, it provides trusted, operationally consistent capabilities for teams to experiment serve models, and deliver innovative apps.

    Red Hat OpenShift Pipelines

    Red Hat OpenShift Pipelines is a cloud-native, continuous integratino and continuous delivery (CI/CD) solution based on Kubernetes resources. It uses Tekton building blocks to automate deployments across multiple platforms by abstracting away the underlying implementation details. Tekton introduces a number of standard custom resource definitions (CRDs) for defining CI/CD pipelines that are portable across Kubernetes distributions.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Overview of the architectures

    The following figure provides a high level architectural overview of the OpenShift AI pattern.

    Logical Architecture
    Figure 2. Overview schematic diagram of the complete solution

    Subsequent schematic diagrams provide details on:

    • Hybrid multi-cloud GitOps

    • Dynamic security management

    Hybrid Multicloud GitOps

    The following figure provides a schematic diagram showing remaining activities associated with setting up the management hub and clusters using Red Hat Advanced Cluster Management.

    Schematic diagram of hybrid multi-cloud management with GitOps
    Figure 3. Schematic diagram of hybrid multi-cloud management with GitOps

    • Manifest and configuration are set as code template in the form of a Kustomization YAML file. The file describes the desired end state of the managed cluster. When complete, the Kustomization YAML file is pushed into the source control management repository with a version assigned to each update.

    • OpenShift GitOps monitors the repository and detects changes in the repository.

    • OpenShift GitOps creates and updates the manifest by creating Kubernetes objects on top of Red Hat Advanced Cluster Management.

    • Red Hat Advanced Cluster Management provisions, updates, or deletes managed clusters and configuration according to the manifest. In the manifest, you can configure what cloud provider the cluster will be on, the name of the cluster, infrastructure node details and worker node. Governance policy can also be applied as well as provision an agent in the cluster as the bridge between the control center and the managed cluster.

    • OpenShift GitOps continuously monitors the code repository and the status of the clusters reported back to Red Hat Advanced Cluster Management. Any configuration drift or in case of any failure, OpenShift GitOps will automatically try to remediate by applying the manifest or by displaying alerts for manual intervention.

    Dynamic security management

    The following figure provides a schematic diagram showing how secrets are handled in this solution.

    Schematic showing the setup and use of external secrets management
    Figure 4. Schematic showing the setup and use of external secrets management

    • During setup, the token to securely access HashiCorp Vault is stored in Ansible Vault. It is encrypted to protect sensitive content.

    • Red Hat Advanced Cluster Management for Kubernetes acquires the token from Ansible Vault during install and distributes it among the clusters. As a result, you have centralized control over the managed clusters through RHACM.

    • To allow the cluster access to the external vault, you must set up the external secret management with Helm in this study. OpenShift Gitops is used to deploy the external secret object to a managed cluster.

    • External secret management fetches secrets from HashiCorp Vault by using the token that was generated in step 2 and constantly monitors for updates.

    • Secrets are created in each namespace, where applications can use them.

    Next steps

    QUICKLINKS

    About the solution

    This architecture covers a single cluster for all DevOps and GitOps functionality. However, one could extend this architecture to meet hybrid or multicloud demand using a GitOps approach

    Benefits of Hybrid Multicloud management with GitOps:

    • Unify management across cloud environments.

    • Dynamic infrastructure security.

    • Infrastructural continuous delivery best practices.

    In the following figure, logically, this solution can be viewed as being composed of an automation component, unified management including secrets management, and the clusters under management, all running on top of a user-chosen mixture of on-premise data centers and public clouds.

    Logical Architecture
    Figure 1. Logical diagram of hybrid multi-cloud management with GitOps

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Advanced Cluster Management for Kubernetes

    Controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat OpenShift AI

    A flexible, scalable MLOps platform with tools to build, deploy, and manage AI-enabled applications. Built using open source technologies, it provides trusted, operationally consistent capabilities for teams to experiment serve models, and deliver innovative apps.

    Red Hat OpenShift Pipelines

    Red Hat OpenShift Pipelines is a cloud-native, continuous integratino and continuous delivery (CI/CD) solution based on Kubernetes resources. It uses Tekton building blocks to automate deployments across multiple platforms by abstracting away the underlying implementation details. Tekton introduces a number of standard custom resource definitions (CRDs) for defining CI/CD pipelines that are portable across Kubernetes distributions.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Overview of the architectures

    The following figure provides a high level architectural overview of the OpenShift AI pattern.

    Logical Architecture
    Figure 2. Overview schematic diagram of the complete solution

    Subsequent schematic diagrams provide details on:

    • Hybrid multi-cloud GitOps

    • Dynamic security management

    Hybrid Multicloud GitOps

    The following figure provides a schematic diagram showing remaining activities associated with setting up the management hub and clusters using Red Hat Advanced Cluster Management.

    Schematic diagram of hybrid multi-cloud management with GitOps
    Figure 3. Schematic diagram of hybrid multi-cloud management with GitOps

    • Manifest and configuration are set as code template in the form of a Kustomization YAML file. The file describes the desired end state of the managed cluster. When complete, the Kustomization YAML file is pushed into the source control management repository with a version assigned to each update.

    • OpenShift GitOps monitors the repository and detects changes in the repository.

    • OpenShift GitOps creates and updates the manifest by creating Kubernetes objects on top of Red Hat Advanced Cluster Management.

    • Red Hat Advanced Cluster Management provisions, updates, or deletes managed clusters and configuration according to the manifest. In the manifest, you can configure what cloud provider the cluster will be on, the name of the cluster, infrastructure node details and worker node. Governance policy can also be applied as well as provision an agent in the cluster as the bridge between the control center and the managed cluster.

    • OpenShift GitOps continuously monitors the code repository and the status of the clusters reported back to Red Hat Advanced Cluster Management. Any configuration drift or in case of any failure, OpenShift GitOps will automatically try to remediate by applying the manifest or by displaying alerts for manual intervention.

    Dynamic security management

    The following figure provides a schematic diagram showing how secrets are handled in this solution.

    Schematic showing the setup and use of external secrets management
    Figure 4. Schematic showing the setup and use of external secrets management

    • During setup, the token to securely access HashiCorp Vault is stored in Ansible Vault. It is encrypted to protect sensitive content.

    • Red Hat Advanced Cluster Management for Kubernetes acquires the token from Ansible Vault during install and distributes it among the clusters. As a result, you have centralized control over the managed clusters through RHACM.

    • To allow the cluster access to the external vault, you must set up the external secret management with Helm in this study. OpenShift Gitops is used to deploy the external secret object to a managed cluster.

    • External secret management fetches secrets from HashiCorp Vault by using the token that was generated in step 2 and constantly monitors for updates.

    • Secrets are created in each namespace, where applications can use them.

    Next steps

    QUICKLINKS

    CONTRIBUTE

    Following commands will take about 15-20 minutes

    Validated pattern will be deployed

    ./pattern.sh make install

    1: Verify the installation

    • Login to the OpenShift web console.
    • Navigate to the Workloads –> Pods.
    • Select the rag-llm project from the drop down.
    • Following pods should be up and running.

    Pods

    Note: If the hf-text-generation-server is not running, make sure you have followed the steps to configure a node with GPU from the instructions provided above.

    2: Launch the application

    • Click the Application box icon in the header, and select Retrieval-Augmented-Generation (RAG) LLM Demonstration UI

    Launch Application

    • It should launch the application

      Application

    3: Generate the proposal document

    • It will use the default provider and model configured as part of the application deployment. The default provider is a Hugging Face model server running in the OpenShift. The model server is deployed with this validated pattern and requires a node with GPU.

    • Enter any company name

    • Enter the product as RedHat OpenShift

    • Click the Generate button, a project proposal should be generated. The project proposal also contains the reference of the RAG content. The project proposal document can be Downloaded in the form of a PDF document.

      Routes

    4: Add an OpenAI provider

    You can optionally add additional providers. The application supports the following providers

    • Hugging Face Text Generation Inference Server
    • OpenAI
    • NVIDIA

    Click on the Add Provider tab to add a new provider. Fill in the details and click Add Provider button. The provider should be added in the Providers dropdown under Chatbot tab.

    Routes

    5: Generate the proposal document using OpenAI provider

    Follow the instructions in step 3 to generate the proposal document using the OpenAI provider.

    Routes

    6: Rating the provider

    You can provide rating to the model by clicking on the Rate the model radio button. The rating will be captured as part of the metrics and can help the company which model to deploy in production.

    7: Grafana Dashboard

    By default, Grafana application is deployed in llm-monitoring namespace.To launch the Grafana Dashboard, follow the instructions below:

    • Grab the credentials of Grafana Application
      • Navigate to Workloads –> Secrets
      • Click on the grafana-admin-credentials and copy the GF_SECURITY_ADMIN_USER, GF_SECURITY_ADMIN_PASSWORD
    • Launch Grafana Dashboard
      • Click the Application box icon in the header, and select Grafana UI for LLM ratings -Launch Application
      • Enter the Grafana admin credentials.
      • Ratings are displayed for each model.

    Routes

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Provisioning NVIDIA daemonsets and compiling drivers may take some time (5-10 minutes)

    QUICKLINKS

    Provisioning NVIDIA daemonsets and compiling drivers may take some time (5-10 minutes)

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    QUICKLINKS

    CONTRIBUTE

    Review Travel Agency Application Graph

    In the Kiali dashboard we can see how all of the various components interact with each other within the service mesh. Just to get a glimpse of what we are able to see let’s take a look at the applications and services in the travel-agency namespace.

    In the left hand menu:

    • click Graph

    • in the Namespace dropdown, select travel-agency

    • exit the menu

    You should see all of the deployments and services that make up the travel-agency application.

    Travel Agency

    Next Steps

    To run through the demo, refer to Monitor the Mesh

    Like what you see, but can’t quite put your finger on how you could use a Service Mesh? Check out Ways to customize the Mesh for some ideas!

    QUICKLINKS

    CONTRIBUTE

    QUICKLINKS

    About the solution

    This architecture covers a single cluster for all DevOps and GitOps functionality. However, one could extend this architecture to meet hybrid or multicloud demand using a GitOps approach

    Benefits of Hybrid Multicloud management with GitOps:

    • Unify management across cloud environments.

    • Dynamic infrastructure security.

    • Infrastructural continuous delivery best practices.

    In the following figure, logically, this solution can be viewed as being composed of an automation component, unified management including secrets management, and the clusters under management, all running on top of a user-chosen mixture of on-premise data centers and public clouds.

    Logical Architecture
    Figure 1. Logical diagram of hybrid multi-cloud management with GitOps

    About the technology

    The following technologies are used in this solution:

    Red Hat OpenShift Platform

    An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. It delivers a complete application platform for both traditional and cloud-native applications, allowing them to run anywhere. OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. It also enables the use of external secret management systems, for example, HashiCorp Vault in this case, to securely add secrets into the OpenShift platform.

    Red Hat OpenShift GitOps

    A declarative application continuous delivery tool for Kubernetes based on the ArgoCD project. Application definitions, configurations, and environments are declarative and version controlled in Git. It can automatically push the desired application state into a cluster, quickly find out if the application state is in sync with the desired state, and manage applications in multi-cluster environments.

    Red Hat Advanced Cluster Management for Kubernetes

    Controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale.

    Red Hat Service Mesh

    Red Hat® OpenShift Service Mesh provides a uniform way to connect, manage, and observe microservices-based applications.

    Red Hat Ansible Automation Platform

    Provides an enterprise framework for building and operating IT automation at scale across hybrid clouds including edge deployments. It enables users across an organization to create, share, and manage automation, from development and operations to security and network teams.

    Hashicorp Vault

    Provides a secure centralized store for dynamic infrastructure and applications across clusters, including over low-trust networks between clouds and data centers.

    This solution also uses a variety of observability tools including the Prometheus monitoring and Grafana dashboard that are integrated with OpenShift as well as components of the Observatorium meta-project which includes Thanos and the Loki API.

    Overview of the architectures

    The following figure provides a high level architectural overview of the travelops pattern.

    Physical Architecture
    Figure 2. Overview schematic diagram of the complete solution

    Subsequent schematic diagrams provide details on:

    • Hybrid multi-cloud GitOps

    • Dynamic security management

    Hybrid Multicloud GitOps

    The following figure provides a schematic diagram showing remaining activities associated with setting up the management hub and clusters using Red Hat Advanced Cluster Management.

    Schematic diagram of hybrid multi-cloud management with GitOps
    Figure 3. Schematic diagram of hybrid multi-cloud management with GitOps

    • Manifest and configuration are set as code template in the form of a Kustomization YAML file. The file describes the desired end state of the managed cluster. When complete, the Kustomization YAML file is pushed into the source control management repository with a version assigned to each update.

    • OpenShift GitOps monitors the repository and detects changes in the repository.

    • OpenShift GitOps creates and updates the manifest by creating Kubernetes objects on top of Red Hat Advanced Cluster Management.

    • Red Hat Advanced Cluster Management provisions, updates, or deletes managed clusters and configuration according to the manifest. In the manifest, you can configure what cloud provider the cluster will be on, the name of the cluster, infrastructure node details and worker node. Governance policy can also be applied as well as provision an agent in the cluster as the bridge between the control center and the managed cluster.

    • OpenShift GitOps continuously monitors the code repository and the status of the clusters reported back to Red Hat Advanced Cluster Management. Any configuration drift or in case of any failure, OpenShift GitOps will automatically try to remediate by applying the manifest or by displaying alerts for manual intervention.

    Dynamic security management

    The following figure provides a schematic diagram showing how secrets are handled in this solution.

    Schematic showing the setup and use of external secrets management
    Figure 4. Schematic showing the setup and use of external secrets management

    • During setup, the token to securely access HashiCorp Vault is stored in Ansible Vault. It is encrypted to protect sensitive content.

    • Red Hat Advanced Cluster Management for Kubernetes acquires the token from Ansible Vault during install and distributes it among the clusters. As a result, you have centralized control over the managed clusters through RHACM.

    • To allow the cluster access to the external vault, you must set up the external secret management with Helm in this study. OpenShift Gitops is used to deploy the external secret object to a managed cluster.

    • External secret management fetches secrets from HashiCorp Vault by using the token that was generated in step 2 and constantly monitors for updates.

    • Secrets are created in each namespace, where applications can use them.

    Next steps

    QUICKLINKS

    CONTRIBUTE