From c09a96d7a9e2cd1147122169a10dba4c6416be92 Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Thu, 26 Sep 2024 15:15:49 +0200 Subject: [PATCH 1/7] control plane metrics --- .../kubernetes-config.rst | 26 +++++++++++++++++++ .../kubernetes-helm-architecture.rst | 2 ++ 2 files changed, 28 insertions(+) diff --git a/gdi/opentelemetry/collector-kubernetes/kubernetes-config.rst b/gdi/opentelemetry/collector-kubernetes/kubernetes-config.rst index b5542a92e..10aed0e19 100644 --- a/gdi/opentelemetry/collector-kubernetes/kubernetes-config.rst +++ b/gdi/opentelemetry/collector-kubernetes/kubernetes-config.rst @@ -210,6 +210,32 @@ Set ``agent.hostNetwork`` to ``false`` to turn off host network access. This mig This value is disregarded for Windows. +.. _otel-kubernetes-config-control-plane-metrics: + +Collect control plane metrics from specific OTel components +====================================================================================== + +Control plane metrics are available for the following components: ``coredns``, ``etcd``, ``kube-controller-manager``, ``kubernetes-apiserver``, ``kubernetes-proxy``, and ``kubernetes-scheduler`` + +You can use the :ref:`Collector agent ` to obtain control plane metrics from a specific component by setting ``agent.controlPlaneMetrics.{otel_component}`` to ``true``. + +Supported distributions +------------------------------------------- + +Control plane metrics are supported for Kubernetes version 1.22 and Openshift version 4.9. + +Unsupported distributions include AKS, EKS, EKS/Fargate, GKE, and GKE/Autopilot. + +Activate control plane metrics histograms +------------------------------------------- + +To activate control plane metrics histograms, use the feature flag ``useControlPlaneMetricsHistogramData``, for example: + +.. code-block:: yaml + + featureGates: + useControlPlaneMetricsHistogramData: true + Activate AlwaysOn Profiling ================================= diff --git a/gdi/opentelemetry/collector-kubernetes/kubernetes-helm-architecture.rst b/gdi/opentelemetry/collector-kubernetes/kubernetes-helm-architecture.rst index 91204bdae..873b83dd2 100644 --- a/gdi/opentelemetry/collector-kubernetes/kubernetes-helm-architecture.rst +++ b/gdi/opentelemetry/collector-kubernetes/kubernetes-helm-architecture.rst @@ -13,6 +13,8 @@ The Helm chart for the Collector has three components: agent, cluster receiver, .. note:: For use cases about the different components, see the GitHub documentation :new-page:`Splunk OpenTelemetry Collector Helm Chart Components: Use Cases `. +.. _helm-chart-components-agent: + Agent component ============================================== From 3b48f21eb666fb82ffc5368bd5535db4db6731f9 Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Thu, 26 Sep 2024 15:31:05 +0200 Subject: [PATCH 2/7] WIP --- .../kubernetes-config-advanced.rst | 39 ++++++++++++------- .../kubernetes-config.rst | 26 ------------- 2 files changed, 26 insertions(+), 39 deletions(-) diff --git a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst index 3559d0dfa..093dbb86a 100644 --- a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst +++ b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst @@ -57,16 +57,16 @@ This custom configuration is merged into the default agent configuration. .. _otel-kubernetes-config-advanced-control-plane: -Override a control plane configuration +Configure control plane metrics ============================================================== -If any of the control plane metric receivers are activated under the ``agent.controlPlaneMetrics`` configuration section, then the Helm chart will configure the Collector to use the activated receivers to collect metrics from the control plane. +Control plane metrics are available for the following components: ``coredns``, ``etcd``, ``kube-controller-manager``, ``kubernetes-apiserver``, ``kubernetes-proxy``, and ``kubernetes-scheduler``. You can use the :ref:`Collector Helm agent ` to obtain control plane metrics from a specific component by setting ``agent.controlPlaneMetrics.{otel_component}`` to ``true``. -To collect control plane metrics, the Helm chart uses the Collector on each node to use the receiver creator to represent control plane receivers at runtime. The receiver creator has a set of discovery rules that know which control plane receivers to create. The default discovery rules can vary depending on the Kubernetes distribution and version. See :ref:`receiver-creator-receiver` for more information. +The Helm chart uses the Collector on each node to use the receiver creator to represent control plane receivers at runtime. The receiver creator has a set of discovery rules that know which control plane receivers to create. The default discovery rules can vary depending on the Kubernetes distribution and version. See :ref:`receiver-creator-receiver` for more information. If your control plane is using non-standard specifications, then you can provide a custom configuration to allow the Collector to successfully connect to it. -Availability and configuration instructions +Supported versions ----------------------------------------------------------------------------- The Collector relies on pod-level network access to collect metrics from the control plane pods. Since most cloud Kubernetes as a service distributions don't expose the control plane pods to the end user, collecting metrics from these distributions is not supported. @@ -90,7 +90,10 @@ The following table shows which Kubernetes distributions support control plane m See the :new-page:`agent template ` for the default configurations for the control plane receivers. -See the following documentation for information on the configuration options and supported metrics for each control plane receiver: +Availability +----------------------------------------------------------------------------- + +The following components provide control plane metrics: * :ref:`CoreDNS `. * :ref:`etcd`. To retrieve etcd metrics, see :new-page:`Setting up etcd metrics `. @@ -99,14 +102,6 @@ See the following documentation for information on the configuration options and * :ref:`Kubernetes proxy `. * :ref:`Kubernetes scheduler `. -Known issue ------------------------------------------------------------------------------ - -There is a known limitation for the Kubernetes proxy control plane receiver. When using a Kubernetes cluster created using kops, a network connectivity issue prevents proxy metrics from being collected. The limitation can be addressed by updating the kubeProxy metric bind address in the kops cluster specification: - -#. Set ``kubeProxy.metricsBindAddress: 0.0.0.0`` in the kops cluster specification. -#. Run ``kops update cluster {cluster_name}`` and ``kops rolling-update cluster {cluster_name}`` to deploy the change. - Use custom configurations for non-standard control plane components ----------------------------------------------------------------------------- @@ -135,6 +130,24 @@ The following example shows how to connect to a nonstandard API server that uses useHTTPS: true useServiceAccount: false +Activate control plane metrics histograms +------------------------------------------- + +To activate control plane metrics histograms, use the feature flag ``useControlPlaneMetricsHistogramData``, for example: + +.. code-block:: yaml + + featureGates: + useControlPlaneMetricsHistogramData: true + +Known issues +----------------------------------------------------------------------------- + +There is a known limitation for the Kubernetes proxy control plane receiver. When using a Kubernetes cluster created using kops, a network connectivity issue prevents proxy metrics from being collected. The limitation can be addressed by updating the kubeProxy metric bind address in the kops cluster specification: + +#. Set ``kubeProxy.metricsBindAddress: 0.0.0.0`` in the kops cluster specification. +#. Run ``kops update cluster {cluster_name}`` and ``kops rolling-update cluster {cluster_name}`` to deploy the change. + .. _kubernetes-config-advanced-non-root: Run the container in non-root user mode diff --git a/gdi/opentelemetry/collector-kubernetes/kubernetes-config.rst b/gdi/opentelemetry/collector-kubernetes/kubernetes-config.rst index 10aed0e19..b5542a92e 100644 --- a/gdi/opentelemetry/collector-kubernetes/kubernetes-config.rst +++ b/gdi/opentelemetry/collector-kubernetes/kubernetes-config.rst @@ -210,32 +210,6 @@ Set ``agent.hostNetwork`` to ``false`` to turn off host network access. This mig This value is disregarded for Windows. -.. _otel-kubernetes-config-control-plane-metrics: - -Collect control plane metrics from specific OTel components -====================================================================================== - -Control plane metrics are available for the following components: ``coredns``, ``etcd``, ``kube-controller-manager``, ``kubernetes-apiserver``, ``kubernetes-proxy``, and ``kubernetes-scheduler`` - -You can use the :ref:`Collector agent ` to obtain control plane metrics from a specific component by setting ``agent.controlPlaneMetrics.{otel_component}`` to ``true``. - -Supported distributions -------------------------------------------- - -Control plane metrics are supported for Kubernetes version 1.22 and Openshift version 4.9. - -Unsupported distributions include AKS, EKS, EKS/Fargate, GKE, and GKE/Autopilot. - -Activate control plane metrics histograms -------------------------------------------- - -To activate control plane metrics histograms, use the feature flag ``useControlPlaneMetricsHistogramData``, for example: - -.. code-block:: yaml - - featureGates: - useControlPlaneMetricsHistogramData: true - Activate AlwaysOn Profiling ================================= From b20c08ad180c64bee2fc7d898649311a0ce8b3e8 Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Thu, 26 Sep 2024 17:26:30 +0200 Subject: [PATCH 3/7] Link --- .../collector-kubernetes/kubernetes-config-advanced.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst index 093dbb86a..c5b5b7ba6 100644 --- a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst +++ b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst @@ -133,13 +133,15 @@ The following example shows how to connect to a nonstandard API server that uses Activate control plane metrics histograms ------------------------------------------- -To activate control plane metrics histograms, use the feature flag ``useControlPlaneMetricsHistogramData``, for example: +To activate control plane metrics histograms, use the feature flag ``useControlPlaneMetricsHistogramData``: .. code-block:: yaml featureGates: useControlPlaneMetricsHistogramData: true +To learn more about histogram metrics see :ref:`histograms`. + Known issues ----------------------------------------------------------------------------- From 7df1e0ae1fa42a926795c1beeeb092024cbedb65 Mon Sep 17 00:00:00 2001 From: Anna U <104845867+aurbiztondo-splunk@users.noreply.github.com> Date: Fri, 27 Sep 2024 07:20:10 +0200 Subject: [PATCH 4/7] Update gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst Co-authored-by: Aunsh Chaudhari --- .../collector-kubernetes/kubernetes-config-advanced.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst index c5b5b7ba6..9fdd6a969 100644 --- a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst +++ b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst @@ -133,7 +133,7 @@ The following example shows how to connect to a nonstandard API server that uses Activate control plane metrics histograms ------------------------------------------- -To activate control plane metrics histograms, use the feature flag ``useControlPlaneMetricsHistogramData``: +To activate control plane metrics with the OpenTelemetry prometheus receiver instead, use the feature flag ``useControlPlaneMetricsHistogramData``: .. code-block:: yaml From 79ef759f9c400081c59e76d4668184c165589923 Mon Sep 17 00:00:00 2001 From: Anna U <104845867+aurbiztondo-splunk@users.noreply.github.com> Date: Fri, 27 Sep 2024 07:20:15 +0200 Subject: [PATCH 5/7] Update gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst Co-authored-by: Aunsh Chaudhari --- .../collector-kubernetes/kubernetes-config-advanced.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst index 9fdd6a969..02adc9144 100644 --- a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst +++ b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst @@ -130,7 +130,7 @@ The following example shows how to connect to a nonstandard API server that uses useHTTPS: true useServiceAccount: false -Activate control plane metrics histograms +Activate Kubernetes control plane metrics with Prometheus receiver ------------------------------------------- To activate control plane metrics with the OpenTelemetry prometheus receiver instead, use the feature flag ``useControlPlaneMetricsHistogramData``: From 3e41c079d31474adea1ebb7b4451e2fcb0b98fd8 Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Fri, 27 Sep 2024 08:26:31 +0200 Subject: [PATCH 6/7] Edits --- .../kubernetes-config-advanced.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst index 02adc9144..ffaf3691d 100644 --- a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst +++ b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst @@ -130,17 +130,19 @@ The following example shows how to connect to a nonstandard API server that uses useHTTPS: true useServiceAccount: false -Activate Kubernetes control plane metrics with Prometheus receiver -------------------------------------------- +.. _kubernetes-control-plane-prometheus: -To activate control plane metrics with the OpenTelemetry prometheus receiver instead, use the feature flag ``useControlPlaneMetricsHistogramData``: +Activate Kubernetes control plane metrics with the Prometheus receiver +------------------------------------------------------------------------- + +To activate control plane metrics with the OpenTelemetry Prometheus receiver instead, use the feature flag ``useControlPlaneMetricsHistogramData``: .. code-block:: yaml featureGates: useControlPlaneMetricsHistogramData: true -To learn more about histogram metrics see :ref:`histograms`. +To learn more see :ref:`prometheus-receiver`. Known issues ----------------------------------------------------------------------------- From 18f69f393cac0731e37140d00355973332b78c3d Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Tue, 1 Oct 2024 09:40:53 +0200 Subject: [PATCH 7/7] Note --- .../collector-kubernetes/kubernetes-config-advanced.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst index ffaf3691d..311195e10 100644 --- a/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst +++ b/gdi/opentelemetry/collector-kubernetes/kubernetes-config-advanced.rst @@ -142,6 +142,8 @@ To activate control plane metrics with the OpenTelemetry Prometheus receiver ins featureGates: useControlPlaneMetricsHistogramData: true +.. Note:: Out-of-the-box dashboards and navigators for control plane metrics with the Prometheus receiver aren't supported yet, but are planned for a future release. + To learn more see :ref:`prometheus-receiver`. Known issues