Target allocator #261

okankoAMZ · 2024-10-25T17:37:21Z

Enhanced CloudWatch Agent Operator for Kubernetes

Summary

This PR introduces significant improvements to the CloudWatch Agent Operator for Kubernetes, addressing limitations in the previous Daemonset implementation and enhancing scalability and efficiency.

Key Changes

Deploying as Statefulset
Integration of Target Allocator component
Dynamic sharding of Prometheus targets

Detailed Description

Background

The Amazon CloudWatch Agent allows customers to collect and publish metrics in Prometheus format across various compute environments, including Kubernetes clusters. The CloudWatch Agent Operator simplifies the onboarding process for Prometheus scraping.

Previous Limitations

Metric duplication due to multiple agent instances scraping the same endpoints
Lack of horizontal scaling capability

New Features

Statefulset Deployment:
- Provides stable, persistent storage
- Allows ordered, graceful scaling of agent instances
- Needs : Target allocator aws-observability/helm-charts#126
Target Allocator Integration:
- Watches for Prometheus targets in the cluster
- Dynamically shards targets across multiple CloudWatch Agent replicas

Benefits:

Configurable number of agent replicas
Automatic distribution of Prometheus scrape targets
Improved efficiency and scalability in metric collection
Customizable Prometheus scrape configuration via custom resource

Automatic Updates

The operator automatically applies changes to the scrape configuration, updating both the Target Allocator and CloudWatch Agent instances.

Testing

TBA

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

NodeJS merging-in from main

Co-authored-by: Mitali Salvi <[email protected]>

* Ta https server (#2921) * Added https server, tests, secret marshalling --------- Co-authored-by: ItielOlenick <[email protected]>

) * Changes error to warning

…247) * edited workflow

* Reconciler now removes un-used managed resources for CWA collector

* added one-service per TA

* TLS tests

* Injecting Prom path if it doesn't exist

* Adding support for NodeJS auto instrumentation and integ tests (#220) * Support configurable resources for NodeJS. (#225) * Supporting JMX annotations (#240) * Add support for a supplemental YAML configuration for the CloudWatchAgent (#241) * Changed naming for OTLP container ports from agent JSON (#252) * Updated Release Notes for 1.8.0 (#251) * Adjust EKS add-on integration test service count expectations (#256) * Add integration tests for JMX. (#250) * Implemented Target Allocator Container (#214) * Implemented TargetAllocator resource deployments. (#208) * Update cmd/amazon-cloudwatch-agent-target-allocator/config/config.go Co-authored-by: Musa <[email protected]> * Update internal/config/main.go Co-authored-by: Musa <[email protected]> --------- Co-authored-by: Parampreet Singh <[email protected]> Co-authored-by: Musa <[email protected]> Co-authored-by: Mitali Salvi <[email protected]> Co-authored-by: Jeffrey Chien <[email protected]>

.github/workflows/build-and-upload-integration.yml

.github/workflows/build-and-upload-release.yml

internal/manifests/collector/config_replace.go

apis/v1alpha1/collector_webhook.go

internal/manifests/collector/container.go

cmd/amazon-cloudwatch-agent-target-allocator/Dockerfile

cmd/amazon-cloudwatch-agent-target-allocator/allocation/allocator.go

versions.txt

main.go

sky333999 · 2024-11-15T07:25:11Z

internal/naming/main.go

@@ -9,6 +9,16 @@ func ConfigMap(otelcol string) string {
 	return DNSName(Truncate("%s", 63, otelcol))
 }

+// TAConfigMap returns the name for the config map used in the TargetAllocator.
+func TAConfigMap(otelcol string) string {
+	return DNSName(Truncate("%s-target-allocator", 63, otelcol))


nit: %s-targetallocator to be consistent with upstream

we are following the cloudwatch-agent format here but I am open to changing

sky333999 · 2024-11-15T07:25:57Z

internal/naming/main.go

+
+// PrometheusConfigMap returns the name for the prometheus config map.
+func PrometheusConfigMap(otelcol string) string {
+	return DNSName(Truncate("%s-prometheus-config", 63, otelcol))


nit: %s-prometheus is probably good enough to be consistent with the other names and its better to keep these short to avoid hitting the max length of 64.

internal/manifests/manifestutils/labels.go

sky333999 · 2024-11-15T08:09:31Z

internal/manifests/targetallocator/labels.go

+)
+
+// Labels return the common labels to all TargetAllocator objects that are part of a managed AmazonCloudWatchAgent.
+func Labels(instance v1alpha1.AmazonCloudWatchAgent, name string) map[string]string {


nit: Do we need this?
Cant we just call manifestutils.Labels with a new component name like ComponentAmazonCloudWatchAgentTargetAllocator?

We don't need it, we can remove it and use manifestutils.Labels instead if that's preferable.

internal/manifests/targetallocator/volume.go

sky333999 · 2024-11-15T08:20:28Z

controllers/amazoncloudwatchagent_controller.go

+		Namespace:     owner.Namespace,
+		LabelSelector: labels.SelectorFromSet(selector),
+	}
+	// Define lists for different Kubernetes resources


why just these? are we only looking at the type of resources created for TA currently?
since the controller can create other resources too such as hpsa, pdbs, routes, ingresses etc.

Same question

mitali-salvi

Do we have any integration tests which validate the new component being added ?

mitali-salvi · 2024-11-20T21:17:46Z

apis/v1alpha1/collector_webhook.go

@@ -87,6 +90,9 @@ func (c CollectorWebhook) defaulter(r *AmazonCloudWatchAgent) error {
 	if r.Spec.Replicas == nil {
 		r.Spec.Replicas = &one
 	}
+	if r.Spec.TargetAllocator.Enabled && r.Spec.TargetAllocator.Replicas == nil {
+		r.Spec.TargetAllocator.Replicas = &one


Wondering if this needs to be documented or made configurable later ? This means TA is not scalable at the moment

Yeah I think this can be an enhancement.

mitali-salvi · 2024-11-20T21:19:56Z

apis/v1alpha1/collector_webhook.go

@@ -163,6 +169,32 @@ func (c CollectorWebhook) validate(r *AmazonCloudWatchAgent) (admission.Warnings
 		return warnings, fmt.Errorf("the OpenTelemetry Collector mode is set to %s, which does not support the attribute 'AdditionalContainers'", r.Spec.Mode)
 	}

+	// validate target allocation
+	if r.Spec.TargetAllocator.Enabled && r.Spec.Mode != ModeStatefulSet {
+		warnings = append(warnings, fmt.Sprintf("The Amazon CloudWatch Agent mode is set to %s, we do not recommend enabling Target Allocator when not running as a StatefulSet", r.Spec.Mode))


nit: could be updated with The Amazon CloudWatch Agent mode is set to %s, which does not support enabling Target Allocator

The mode could be Deployment and the error message would say StatefulSet which can be confusing

mitali-salvi · 2024-11-22T17:44:10Z

controllers/amazoncloudwatchagent_controller.go

+		Namespace:     owner.Namespace,
+		LabelSelector: labels.SelectorFromSet(selector),
+	}
+	// Define lists for different Kubernetes resources


Same question

okankoAMZ and others added 17 commits September 11, 2024 10:45

Implemented Target Allocator Container (#214)

59fb829

Merge main into target-allocator (#232)

bea6ed2

NodeJS merging-in from main

Supporting K8s 1.31 (#222) (#236)

0dce7f2

Co-authored-by: Mitali Salvi <[email protected]>

Implemented TargetAllocator resource deployments. (#208)

b3cf3cf

Adding target-allocator label to service selectors (#242)

2dc2f2b

Target allocator TLS Implementation (#239)

bfb6b3a

* Ta https server (#2921) * Added https server, tests, secret marshalling --------- Co-authored-by: ItielOlenick <[email protected]>

[Target Allocator] Enable Deployment and Daemonset modes for Agent (#253

b74b110

) * Changes error to warning

[CI/CD] Add Target Allocator(TA) Build to Build and Upload Workflow (#…

313843f

…247) * edited workflow

Clean up managed resources when disabled (#255)

95f77b0

* Reconciler now removes un-used managed resources for CWA collector

remove pprof endpoint (#260)

a877192

[TA] One service per Target Allocator (#259)

64e55f3

* added one-service per TA

Setup cert-watcher for TA server cert (#264)

31db083

[TA] Target Allocator TLS Unit-tests (#265)

7ae202e

* TLS tests

Injecting Prometheus path if not specified in agent config (#258)

f82db5a

* Injecting Prom path if it doesn't exist

merging main to target-allocator

b683196

go fmt

b5428a3

okankoAMZ marked this pull request as ready for review November 12, 2024 20:40

okankoAMZ commented Nov 12, 2024

View reviewed changes

.github/workflows/build-and-upload-integration.yml Outdated Show resolved Hide resolved

fixed unit-tests

b551012

okankoAMZ force-pushed the target-allocator branch 3 times, most recently from d59fed3 to a1a0f2c Compare November 13, 2024 16:26

fixed unit-test

50e29f0

okankoAMZ force-pushed the target-allocator branch from a1a0f2c to 50e29f0 Compare November 13, 2024 16:34

lisguo reviewed Nov 14, 2024

View reviewed changes

lisguo previously approved these changes Nov 14, 2024

View reviewed changes

sky333999 reviewed Nov 15, 2024

View reviewed changes

addressed comments

57c0f4e

okankoAMZ dismissed lisguo’s stale review via 57c0f4e November 15, 2024 20:27

mitali-salvi reviewed Nov 22, 2024

View reviewed changes

sky333999 approved these changes Nov 26, 2024

View reviewed changes

mitali-salvi approved these changes Nov 26, 2024

View reviewed changes

okankoAMZ merged commit cf01423 into main Nov 26, 2024
8 checks passed

okankoAMZ deleted the target-allocator branch November 26, 2024 21:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Target allocator #261

Target allocator #261

okankoAMZ commented Oct 25, 2024 •

edited

Loading

sky333999 Nov 15, 2024

okankoAMZ Nov 15, 2024

sky333999 Nov 15, 2024

sky333999 Nov 15, 2024

musa-asad Nov 17, 2024 •

edited

Loading

sky333999 Nov 15, 2024

mitali-salvi Nov 22, 2024

mitali-salvi left a comment

mitali-salvi Nov 20, 2024

sky333999 Nov 26, 2024

mitali-salvi Nov 20, 2024

mitali-salvi Nov 22, 2024

Target allocator #261

Target allocator #261

Conversation

okankoAMZ commented Oct 25, 2024 • edited Loading

Enhanced CloudWatch Agent Operator for Kubernetes

Summary

Key Changes

Detailed Description

Background

Previous Limitations

New Features

Benefits:

Automatic Updates

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

musa-asad Nov 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mitali-salvi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

okankoAMZ commented Oct 25, 2024 •

edited

Loading

musa-asad Nov 17, 2024 •

edited

Loading