feat(DMVP-2816): Added new variable for drop namespace logs

README.md

@@ -235,14 +235,14 @@ worker_groups = {
 | <a name="input_adot_version"></a> [adot\_version](#input\_adot\_version) | The version of the AWS Distro for OpenTelemetry addon to use. | `string` | `"v0.78.0-eksbuild.1"` | no |
 | <a name="input_alb_log_bucket_name"></a> [alb\_log\_bucket\_name](#input\_alb\_log\_bucket\_name) | n/a | `string` | `""` | no |
 | <a name="input_alb_log_bucket_path"></a> [alb\_log\_bucket\_path](#input\_alb\_log\_bucket\_path) | ALB-INGRESS-CONTROLLER | `string` | `""` | no |
-| <a name="input_api_gateway_resources"></a> [api\_gateway\_resources](#input\_api\_gateway\_resources) | Nested map containing API, Stage, and VPC Link resources | <pre>list(object({<br>    namespace = string<br>    api = object({<br>      name         = string<br>      protocolType = string<br>    })<br>    stages = optional(list(object({<br>      name        = string<br>      namespace   = string<br>      apiRef_name = string<br>      stageName   = string<br>      autoDeploy  = bool<br>      description = string<br>    })))<br>    vpc_links = optional(list(object({<br>      name      = string<br>      namespace = string<br>    })))<br>  }))</pre> | n/a | yes |
+| <a name="input_api_gateway_resources"></a> [api\_gateway\_resources](#input\_api\_gateway\_resources) | Nested map containing API, Stage, and VPC Link resources | <pre>list(object({<br>    namespace = string<br>    api = object({<br>      name         = string<br>      protocolType = string<br>    })<br>    stages = optional(list(object({<br>      name        = string<br>      namespace   = string<br>      apiRef_name = string<br>      stageName   = string<br>      autoDeploy  = bool<br>      description = string<br>    })))<br>    vpc_links = optional(list(object({<br>      name      = string<br>      namespace = string<br>    })))<br>  }))</pre> | `[]` | no |
 | <a name="input_api_gw_deploy_region"></a> [api\_gw\_deploy\_region](#input\_api\_gw\_deploy\_region) | Region in which API gatewat will be configured | `string` | `""` | no |
 | <a name="input_autoscaler_image_patch"></a> [autoscaler\_image\_patch](#input\_autoscaler\_image\_patch) | The patch number of autoscaler image | `number` | `0` | no |
 | <a name="input_autoscaler_limits"></a> [autoscaler\_limits](#input\_autoscaler\_limits) | n/a | <pre>object({<br>    cpu    = string<br>    memory = string<br>  })</pre> | <pre>{<br>  "cpu": "100m",<br>  "memory": "600Mi"<br>}</pre> | no |
 | <a name="input_autoscaler_requests"></a> [autoscaler\_requests](#input\_autoscaler\_requests) | n/a | <pre>object({<br>    cpu    = string<br>    memory = string<br>  })</pre> | <pre>{<br>  "cpu": "100m",<br>  "memory": "600Mi"<br>}</pre> | no |
 | <a name="input_autoscaling"></a> [autoscaling](#input\_autoscaling) | Weather enable autoscaling or not in EKS | `bool` | `false` | no |
 | <a name="input_bindings"></a> [bindings](#input\_bindings) | Variable which describes group and role binding | <pre>list(object({<br>    group     = string<br>    namespace = string<br>    roles     = list(string)<br><br>  }))</pre> | `[]` | no |
-| <a name="input_cluster_enabled_log_types"></a> [cluster\_enabled\_log\_types](#input\_cluster\_enabled\_log\_types) | A list of the desired control plane logs to enable. For more information, see Amazon EKS Control Plane Logging documentation (https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html) | `list(string)` | <pre>[<br>  "audit"<br>]</pre> | no |
+| <a name="input_cluster_enabled_log_types"></a> [cluster\_enabled\_log\_types](#input\_cluster\_enabled\_log\_types) | A list of the desired control plane logs to enable. For more information, see Amazon EKS Control Plane Logging documentation (https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html) | `list(string)` | `[]` | no |
 | <a name="input_cluster_endpoint_public_access"></a> [cluster\_endpoint\_public\_access](#input\_cluster\_endpoint\_public\_access) | n/a | `bool` | `true` | no |
 | <a name="input_cluster_name"></a> [cluster\_name](#input\_cluster\_name) | Creating eks cluster name. | `string` | n/a | yes |
 | <a name="input_cluster_version"></a> [cluster\_version](#input\_cluster\_version) | Allows to set/change kubernetes cluster version, kubernetes version needs to be updated at leas once a year. Please check here for available versions https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html | `string` | `"1.27"` | no |
@@ -259,9 +259,7 @@ worker_groups = {
 | <a name="input_enable_olm"></a> [enable\_olm](#input\_enable\_olm) | To install OLM controller (experimental). | `bool` | `false` | no |
 | <a name="input_enable_sso_rbac"></a> [enable\_sso\_rbac](#input\_enable\_sso\_rbac) | Enable SSO RBAC integration or not | `bool` | `false` | no |
 | <a name="input_external_secrets_namespace"></a> [external\_secrets\_namespace](#input\_external\_secrets\_namespace) | The namespace of external-secret operator | `string` | `"kube-system"` | no |
-| <a name="input_fluent_bit_name"></a> [fluent\_bit\_name](#input\_fluent\_bit\_name) | FLUENT-BIT | `string` | `""` | no |
-| <a name="input_log_group_name"></a> [log\_group\_name](#input\_log\_group\_name) | n/a | `string` | `""` | no |
-| <a name="input_log_retention_days"></a> [log\_retention\_days](#input\_log\_retention\_days) | n/a | `number` | `90` | no |
+| <a name="input_fluent_bit_configs"></a> [fluent\_bit\_configs](#input\_fluent\_bit\_configs) | Fluent Bit configs | <pre>object({<br>    fluent_bit_name       = optional(string, "")<br>    log_group_name        = optional(string, "")<br>    system_log_group_name = optional(string, "")<br>    log_retention_days    = optional(number, 90)<br>    values_yaml           = optional(string, "")<br>    configs = optional(object({<br>      inputs  = optional(string, "")<br>      filters = optional(string, "")<br>      outputs = optional(string, "")<br>    }), {})<br>    drop_namespaces        = optional(list(string), [])<br>    log_filters            = optional(list(string), [])<br>    additional_log_filters = optional(list(string), [])<br>  })</pre> | <pre>{<br>  "additional_log_filters": [<br>    "ELB-HealthChecker",<br>    "Amazon-Route53-Health-Check-Service"<br>  ],<br>  "configs": {<br>    "filters": "",<br>    "inputs": "",<br>    "outputs": ""<br>  },<br>  "drop_namespaces": [<br>    "kube-system",<br>    "opentelemetry-operator-system",<br>    "adot",<br>    "cert-manager"<br>  ],<br>  "fluent_bit_name": "",<br>  "log_filters": [<br>    "kube-probe",<br>    "health",<br>    "prometheus",<br>    "liveness"<br>  ],<br>  "log_group_name": "",<br>  "log_retention_days": 90,<br>  "system_log_group_name": "",<br>  "values_yaml": ""<br>}</pre> | no |
 | <a name="input_manage_aws_auth"></a> [manage\_aws\_auth](#input\_manage\_aws\_auth) | n/a | `bool` | `true` | no |
 | <a name="input_map_roles"></a> [map\_roles](#input\_map\_roles) | Additional IAM roles to add to the aws-auth configmap. | <pre>list(object({<br>    rolearn  = string<br>    username = string<br>    groups   = list(string)<br>  }))</pre> | `[]` | no |
 | <a name="input_metrics_exporter"></a> [metrics\_exporter](#input\_metrics\_exporter) | Metrics Exporter, can use cloudwatch or adot | `string` | `"cloudwatch"` | no |

examples/spot-instance/README.md

@@ -12,7 +12,7 @@
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.41 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | 4.67.0 |
 
 ## Modules
 

fluent-bit.tf

@@ -0,0 +1,43 @@
+module "fluent-bit" {
+  source = "./modules/fluent-bit"
+
+  count = var.create ? 1 : 0
+
+  account_id = local.account_id
+  region     = local.region
+
+  cluster_name                = module.eks-cluster[0].cluster_id
+  eks_oidc_root_ca_thumbprint = module.eks-cluster[0].eks_oidc_root_ca_thumbprint
+  oidc_provider_arn           = module.eks-cluster[0].oidc_provider_arn
+
+  fluent_bit_name       = try(var.fluent_bit_configs.fluent_bit_name, "") != "" ? var.fluent_bit_configs.fluent_bit_name : "${module.eks-cluster[0].cluster_id}-fluent-bit"
+  log_group_name        = try(var.fluent_bit_configs.log_group_name, "") != "" ? var.fluent_bit_configs.log_group_name : "fluent-bit-cloudwatch-${module.eks-cluster[0].cluster_id}"
+  system_log_group_name = try(var.fluent_bit_configs.system_log_group_name, "")
+  log_retention_days    = try(var.fluent_bit_configs.log_retention_days, 90)
+
+  values_yaml = try(var.fluent_bit_configs.values_yaml, "")
+
+  drop_namespaces = try(var.fluent_bit_configs.drop_namespaces, [
+    "kube-system",
+    "opentelemetry-operator-system",
+    "adot",
+    "cert-manager"
+  ])
+  log_filters = try(var.fluent_bit_configs.log_filters, [
+    "kube-probe",
+    "health",
+    "prometheus",
+    "liveness"
+  ])
+
+  additional_log_filters = try(var.fluent_bit_configs.additional_log_filters, [
+    "ELB-HealthChecker",
+    "Amazon-Route53-Health-Check-Service",
+  ])
+
+  fluent_bit_config = try(var.fluent_bit_configs.configs, {
+    inputs  = ""
+    outputs = ""
+    filters = ""
+  })
+}

main.tf

@@ -251,22 +251,6 @@ module "alb-ingress-controller" {
   # alb_log_bucket_path = var.alb_log_bucket_path != "" ? var.alb_log_bucket_path : module.eks-cluster[0].cluster_id
 }
 
-module "fluent-bit" {
-  source = "./modules/fluent-bit"
-
-  count = var.create ? 1 : 0
-
-  account_id = local.account_id
-  region     = local.region
-
-  fluent_bit_name             = var.fluent_bit_name != "" ? var.fluent_bit_name : "${module.eks-cluster[0].cluster_id}-fluent-bit"
-  log_group_name              = var.log_group_name != "" ? var.log_group_name : "fluent-bit-cloudwatch-${module.eks-cluster[0].cluster_id}"
-  log_retention_days          = var.log_retention_days
-  cluster_name                = module.eks-cluster[0].cluster_id
-  eks_oidc_root_ca_thumbprint = module.eks-cluster[0].eks_oidc_root_ca_thumbprint
-  oidc_provider_arn           = module.eks-cluster[0].oidc_provider_arn
-}
-
 module "metrics-server" {
   source = "./modules/metrics-server"
 

modules/adot/tests/template_file/README.md

@@ -12,7 +12,7 @@
 
 | Name | Version |
 |------|---------|
-| <a name="provider_helm"></a> [helm](#provider\_helm) | n/a |
+| <a name="provider_helm"></a> [helm](#provider\_helm) | 2.9.0 |
 | <a name="provider_test"></a> [test](#provider\_test) | n/a |
 
 ## Modules

modules/fluent-bit/README.md

@@ -47,18 +47,23 @@ No modules.
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_account_id"></a> [account\_id](#input\_account\_id) | AWS Account Id to apply changes into | `string` | n/a | yes |
+| <a name="input_additional_log_filters"></a> [additional\_log\_filters](#input\_additional\_log\_filters) | Fluent bit doesn't send logs if message consists of this values | `list(string)` | <pre>[<br>  "ELB-HealthChecker",<br>  "Amazon-Route53-Health-Check-Service"<br>]</pre> | no |
 | <a name="input_cluster_name"></a> [cluster\_name](#input\_cluster\_name) | AWS EKS Cluster name. | `string` | n/a | yes |
 | <a name="input_create_log_group"></a> [create\_log\_group](#input\_create\_log\_group) | Wether or no to create log group. | `bool` | `true` | no |
 | <a name="input_create_namespace"></a> [create\_namespace](#input\_create\_namespace) | Wether or no to create namespace. | `bool` | `false` | no |
+| <a name="input_drop_namespaces"></a> [drop\_namespaces](#input\_drop\_namespaces) | Flunt bit doesn't send logs for this namespaces | `list(string)` | <pre>[<br>  "kube-system",<br>  "opentelemetry-operator-system",<br>  "adot",<br>  "cert-manager"<br>]</pre> | no |
 | <a name="input_eks_oidc_root_ca_thumbprint"></a> [eks\_oidc\_root\_ca\_thumbprint](#input\_eks\_oidc\_root\_ca\_thumbprint) | n/a | `string` | n/a | yes |
+| <a name="input_fluent_bit_config"></a> [fluent\_bit\_config](#input\_fluent\_bit\_config) | You can add other inputs,outputs and filters which module doesn't have by default | `any` | <pre>{<br>  "filters": "",<br>  "inputs": "",<br>  "outputs": ""<br>}</pre> | no |
 | <a name="input_fluent_bit_name"></a> [fluent\_bit\_name](#input\_fluent\_bit\_name) | Container resource name. | `string` | `"fluent-bit"` | no |
+| <a name="input_log_filters"></a> [log\_filters](#input\_log\_filters) | Fluent bit doesn't send logs if message consists of this values | `list(string)` | <pre>[<br>  "kube-probe",<br>  "health",<br>  "prometheus",<br>  "liveness"<br>]</pre> | no |
 | <a name="input_log_group_name"></a> [log\_group\_name](#input\_log\_group\_name) | Log group name fluent-bit will be streaming logs into. | `string` | `"fluentbit-default-log-group"` | no |
 | <a name="input_log_retention_days"></a> [log\_retention\_days](#input\_log\_retention\_days) | If set to a number greater than zero, and newly create log group's retention policy is set to this many days. Valid values are: [0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653] | `number` | `90` | no |
 | <a name="input_namespace"></a> [namespace](#input\_namespace) | k8s namespace fluent-bit should be deployed into. | `string` | `"kube-system"` | no |
 | <a name="input_oidc_provider_arn"></a> [oidc\_provider\_arn](#input\_oidc\_provider\_arn) | n/a | `string` | n/a | yes |
 | <a name="input_region"></a> [region](#input\_region) | AWS Region name. | `string` | n/a | yes |
 | <a name="input_s3_permission"></a> [s3\_permission](#input\_s3\_permission) | If you want send logs to s3 you should enable s3 permission | `bool` | `false` | no |
-| <a name="input_values_yaml"></a> [values\_yaml](#input\_values\_yaml) | Content of the values.yaml given to the helm chart. This disables the rendered values.yaml file from this module. | `string` | `null` | no |
+| <a name="input_system_log_group_name"></a> [system\_log\_group\_name](#input\_system\_log\_group\_name) | Log group name fluent-bit will be streaming kube-system logs. | `string` | `""` | no |
+| <a name="input_values_yaml"></a> [values\_yaml](#input\_values\_yaml) | Content of the values.yaml if you want override all default configs. | `string` | `""` | no |
 
 ## Outputs
 

modules/fluent-bit/locals.tf

@@ -3,10 +3,18 @@ locals {
   log_group_name = var.log_group_name != "" ? var.log_group_name : "fluent-bit-cloudwatch"
   region         = var.region
   config_settings = {
-    log_group_name     = local.log_group_name,
-    region             = local.region,
-    log_retention_days = var.log_retention_days
-    auto_create_group  = var.create_log_group ? "On" : "Off"
+    log_group_name         = local.log_group_name
+    system_log_group_name  = var.system_log_group_name == "" ? "${local.log_group_name}-kube" : "${var.system_log_group_name}"
+    region                 = local.region
+    log_retention_days     = var.log_retention_days
+    auto_create_group      = var.create_log_group ? "On" : "Off"
+    drop_namespaces        = "(${join("|", var.drop_namespaces)})"
+    log_filters            = "(${join("|", var.log_filters)})"
+    additional_log_filters = "(${join("|", var.additional_log_filters)})"
+    inputs                 = try(var.fluent_bit_config.inputs, "")
+    outputs                = try(var.fluent_bit_config.outputs, "")
+    filters                = try(var.fluent_bit_config.filters, "")
   }
-  values = var.values_yaml == null ? templatefile("${path.module}/values.yaml", local.config_settings) : var.values_yaml
+
+  values = var.values_yaml == "" ? templatefile("${path.module}/values.yaml.tpl", local.config_settings) : var.values_yaml
 }

modules/fluent-bit/tests/advanced/0-setup.tf

@@ -0,0 +1,21 @@
+terraform {
+  required_providers {
+    test = {
+      source = "terraform.io/builtin/test"
+    }
+
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 4.37"
+    }
+    kubernetes = {
+      source  = "hashicorp/kubernetes"
+      version = "~>2.23"
+    }
+    helm = ">= 2.0"
+  }
+}
+
+provider "aws" {}
+provider "helm" {}
+provider "kubernetes" {}

modules/fluent-bit/tests/advanced/1-example.tf

@@ -0,0 +1,52 @@
+locals {
+  oidc_provider_arn = "arn:aws:iam::000000000000:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/6F40EA94327Dh8956DDB9S0AE7907CFD"
+}
+
+module "fluent-bit" {
+  source = "../../"
+
+  account_id = 000000000000
+  region     = "eu-central-1"
+
+  cluster_name                = "Test"
+  oidc_provider_arn           = local.oidc_provider_arn
+  eks_oidc_root_ca_thumbprint = replace(local.oidc_provider_arn, "/.*id//", "")
+
+
+  log_group_name        = "fluent-bit"
+  system_log_group_name = "fluent-bit-kube"
+  create_log_group      = true
+  log_retention_days    = 7
+
+  drop_namespaces = [
+    "kube-system",
+    "opentelemetry-operator-system",
+    "adot",
+    "cert-manager"
+  ]
+
+  additional_log_filters = [
+    "ELB-HealthChecker",
+    "Amazon-Route53-Health-Check-Service",
+  ]
+
+  log_filters = [
+    "ELB-HealthChecker",
+    "Amazon-Route53-Health-Check-Service",
+    "kube-probe",
+    "health",
+    "prometheus",
+    "liveness"
+  ]
+
+  fluent_bit_config = {
+    inputs  = templatefile("${path.module}/templates/inputs.yaml.tpl", {})
+    outputs = templatefile("${path.module}/templates/outputs.yaml.tpl", {})
+    filters = templatefile("${path.module}/templates/filters.yaml.tpl", {})
+  }
+
+}
+
+output "merged_inputs" {
+  value = module.fluent-bit
+}

modules/fluent-bit/tests/advanced/2-assert.tf

@@ -0,0 +1,9 @@
+resource "test_assertions" "api_url" {
+  component = "Basic-Setup"
+
+  equal "scheme" {
+    description = "As module does not have any output and data just make sure the case runs. Probably can be thrown away."
+    got         = "all good"
+    want        = "all good"
+  }
+}

modules/fluent-bit/tests/advanced/README.md

@@ -0,0 +1,39 @@
+# basic
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | ~> 4.37 |
+| <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.0 |
+| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | ~>2.23 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_test"></a> [test](#provider\_test) | n/a |
+
+## Modules
+
+| Name | Source | Version |
+|------|--------|---------|
+| <a name="module_fluent-bit"></a> [fluent-bit](#module\_fluent-bit) | ../../ | n/a |
+
+## Resources
+
+| Name | Type |
+|------|------|
+| test_assertions.api_url | resource |
+
+## Inputs
+
+No inputs.
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| <a name="output_merged_inputs"></a> [merged\_inputs](#output\_merged\_inputs) | n/a |
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

modules/fluent-bit/tests/advanced/templates/filters.yaml.tpl

@@ -0,0 +1,9 @@
+[FILTER]
+    Name          grep
+    Match         kube.*
+    Exclude       $log (test)
+
+[FILTER]
+    Name          grep
+    Match         audit.*
+    regex         $log (test)

modules/fluent-bit/tests/advanced/templates/inputs.yaml.tpl

@@ -0,0 +1,9 @@
+[INPUT]
+    Name               tail
+    Tag                test.*
+    Path               /var/log/containers/*.log
+    Read_from_head     true
+    multiline.parser   docker, cri
+    Docker_Mode        On
+    Parser             docker
+    Mem_Buf_Limit      50MB

modules/fluent-bit/tests/advanced/templates/outputs.yaml.tpl

@@ -0,0 +1,7 @@
+[OUTPUT]
+    Name s3
+    Match test.*
+    bucket s3-bucket
+    region eu-central-1
+    total_file_size 250M
+    s3_key_format /%Y/%m/%d/%H_%M_%S.gz