Skip to content

Latest commit

 

History

History
399 lines (284 loc) · 10.5 KB

README.md

File metadata and controls

399 lines (284 loc) · 10.5 KB
Raw

How to monitor Openshift using Datadog Operator

 

In this article, I will demonstrate how to integrate Openshift with Datadog using datadog operator to collect metrics,logs and events.

In this article we use the following versions:

  • Openshift v4.13.11
  • Datadog Operator v1.3.0

 

About

  • This article is aimed at users who would like to integrate or monitor their Openshift Cluster using the Datadog monitoring solution.

  • We will use the datadog operator to instantiate our agent and collect all metrics, cluster and container/pod logs, network, cpu and memory consumption.

Prerequisites

  • User with the cluster-admin cluster role
  • Openshift 4.10 or +
  • Datadog Account

Procedure

Datadog

Add api keys

  • To add a new datadog api key, navigate to Organization Settings > Api Keys
  • If you have the permission to create api keys, click New Key in the top right corner.
  • Define the desired name, something that can help you identify in the future.
  • Once created, copy the Key so we can use it later.

Add application keys

  • To add a new datadog application key, navigate to Organization Settings > Application Keys
  • If you have the permission to create api keys, click New Key in the top right corner.
  • Define the desired name, something that can help you identify in the future.
  • Once created, copy the Key so we can use it later.

Openshift

Datadog Operator Install

  • In the Openshift console, in the left side menu, click Operator > OperatorHub > in the search field, type datadog

Tip

Whenever available, use a certified option.

 

  • As we can see, we are using version 1.3.0 of operator, click Install.

 

  • On this screen, we will keep all the default options:
    • Update channel: stable
    • Installation mode: All namespaces the cluster(default)
    • Installed Namespace: openshift-operators
    • Update approval: Automatic
      • Obs.: If you prefer, you can use the Manual option.
    • Click Install.

 

  • Wait until the installation is complete.

Create secret with datadog keys

  • In the terminal, access the openshift-operators namespace context
$ oc project openshift-operators

 

  • Now let's create a secret to store in this API Key and Application Key, replace the values below with the keys we generated previously in the Datadog console.
$ oc create secret generic datadog-secret \
--from-literal api-key=`REPLACE_ME` \
--from-literal app-key=`REPLACE_ME`

 

  • Let's now create our datadog agent using the yaml below
$ cat <<EOF > datadog_agent.yaml
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
  namespace: openshift-operators
spec:
  features:
    apm:
      enabled: true
      unixDomainSocketConfig:
        enabled: true
    clusterChecks:
      enabled: true
      useClusterChecksRunners: true
    dogstatsd:
      originDetectionEnabled: true
      unixDomainSocketConfig:
        enabled: true
    eventCollection:
      collectKubernetesEvents: true
    liveContainerCollection:
      enabled: true
    liveProcessCollection:
      enabled: true
    logCollection:
      containerCollectAll: true
      enabled: true
    npm:
      collectDNSStats: true
      enableConntrack: true
      enabled: true
  global:
    clusterName: DemoLab
    credentials:
      apiSecret:
        keyName: api-key
        secretName: datadog-secret
      appSecret:
        keyName: app-key
        secretName: datadog-secret
    criSocketPath: /var/run/crio/crio.sock
    kubelet:
      tlsVerify: false
    site: datadoghq.eu
  override:
    clusterAgent:
      containers:
        cluster-agent:
          securityContext:
            readOnlyRootFilesystem: false
      replicas: 2
      serviceAccountName: datadog-agent-scc
    nodeAgent:
      hostNetwork: true
      securityContext:
        runAsUser: 0
        seLinuxOptions:
          level: s0
          role: system_r
          type: spc_t
          user: system_u
      serviceAccountName: datadog-agent-scc
      tolerations:
      - operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
EOF
  • Some explanations about what we are enabling in this agent

Enabling the APM (Application Performance Management) feature

apm:
  enabled: true
  unixDomainSocketConfig:
    enabled: true

 

Cluster Check extends the autodiscover function to non-containerized resources

clusterChecks:
  enabled: true
  useClusterChecksRunners: true

 

Dogstatsd is responsible for collecting custom metrics and events and sending them from time to time to a metrics aggregation service on the Datadog server.

dogstatsd:
  originDetectionEnabled: true
  unixDomainSocketConfig:
    enabled: true

 

Here we are enabling the collection of all logs (including container logs) and events generated in our cluster and sending them to Datadog.

eventCollection:
  collectKubernetesEvents: true
liveContainerCollection:
  enabled: true
liveProcessCollection:
  enabled: true
logCollection:
  containerCollectAll: true
  enabled: true

 

With NPM (Network Performance Monitoring), we can have visibility of all traffic in our cluster, nodes, containers, availability zones, etc.

npm:
  collectDNSStats: true
  enableConntrack: true
  enabled: true

 

In the credentials block in Global, we have the definition of the secret previously created with the api and app key.

credentials:
  apiSecret:
    keyName: api-key
    secretName: datadog-secret
  appSecret:
    keyName: app-key
    secretName: datadog-secret

 

In this block, we define the path to the crio service socket, we define the non-checking of tls for communication with the kubelet and in website, we define which datadog server will receive the data sent.

criSocketPath: /var/run/crio/crio.sock
kubelet:
  tlsVerify: false
site: datadoghq.eu

 

In the clusterAgent block in override, we add SecurityContext(scc) settings and which serviceaccount should be used in the datadog-cluster-agent pods.

clusterAgent:
  containers:
    cluster-agent:
      securityContext:
        readOnlyRootFilesystem: false
  replicas: 2
  serviceAccountName: datadog-agent-scc

Note

The datadog-agent-scc serviceaccount is created automatically by the operator and already has all the necessary permissions for the agent to run correctly.

 

In the nodeAgent block in override, we define settings for SecurityContext for the datadog-agent pods, we will use the same datadog-agent-scc serviceaccount and we also define the tolerations for the nodes that have taints created, in our case for the master nodes.

nodeAgent:
  hostNetwork: true
  securityContext:
    runAsUser: 0
    seLinuxOptions:
      level: s0
      role: system_r
      type: spc_t
      user: system_u
  serviceAccountName: datadog-agent-scc
  tolerations:
  - operator: Exists
  - effect: NoSchedule
    key: node-role.kubernetes.io/master

 

  • After some explanations, let's create our datadog agent, execute this command to create this object:
$ oc -n openshift-operators create -f datadog_agent.yaml
  • Once created, we will validate that our agent was created correctly
$ oc -n openshift-operators get datadogagent
$ oc -n openshift-operators get pods

Obs.: Here we should have a datadog-agent running on each available openshift node.

Note

  • datadog-agent-xxxxx pods, is responsible for collecting all metrics, events, traces and logs from each node in the cluster.
  • datadog-cluster-agent-xxxxx pods, will act as a proxy between the API server and node-based agents, Cluster Agent helps to ease the server load.

 

  • Now let's validate the logs of the datadog-agent-xxxxx pods, to identify if there is any communication error.
$ oc logs -f -l app.kubernetes.io/managed-by=datadog-operator --max-log-requests 10

 

Datadog Dashboard

  • Now on the Datadog dashboard, in the left side menu, click on Infrastructure > and then on Infrastructure List

Note

Server data, such as status, cpu information, memory and other details, may take a few minutes to be displayed.

 

  • To view more details about a specific node, click on the node name and navigate through the available tabs.

 

  • To view more details about network traffic, in the left side menu, go to Infrastructure > Network Map

 

  • To view the logs received from the cluster, in the left side menu, go to Logs > Analytics, on this screen, we can view all the details, filter application logs and even view the processes.

 

  • To view all collected metrics, in the left side menu, go to Metrics > Explorer, here we can view all metrics, run and save queries or create dashboards based on queries.

 

  • Datadog provides ready-made dashboards that can be used and customized. To use one available, in the left side menu, go to Dashboards > Dashboard List > choose the dashboard and click on the name.

Obs.: To customize a dashboard provided by Datadog, use the Clone feature to make the desired changes and save.

   

Conclusion

Using the Datadog Operator solution, we can have a complete monitoring solution for our Openshift cluster with main features such as APM, Network Analysis, Logs, Events and Metrics.

To request an Openshift trial and learn more about our solution, click here.

To request a Datadog trial and be able to replicate this knowledge, click here.

 

References

For more details and other configurations, start with the reference documents below.