I use deploy Beyla as daemon set to instrument everything in k8s cluster but it return nothing

[bin1st090104]

Log of a beyla instance here:

beyla-logs.log

It can get trace from my k8s cluster, and only get beyla_network_flow_bytes of this own daemonset and target_info metrics.

My beyla config here:

kind: ConfigMap
metadata:
  name: beyla-config
  namespace: tuyennt33
data:
  config.yaml: |
    log_level: DEBUG
    trace_printer: json_indent
    discovery:
      services:
        - k8s_namespace: ".*"
          k8s_pod_name: ".*"
          k8s_deployment_name: ".*"
          k8s_replicaset_name: ".*"
          k8s_statefulset_name: ".*"
          k8s_daemonset_name: ".*"
          k8s_owner_name: ".*"
          k8s_pod_labels:
            instrument: beyla
    attributes:
      kubernetes:
        enable: true
    routes:
      unmatched: heuristic
    otel_traces_export:
      endpoint: http://grafana.tuyennt33.svc.cluster.local:4317
    otel_metrics_export:
      endpoint: http://grafana.tuyennt33.svc.cluster.local:4317
      features: ["application","application_span","application_service_graph","application_process","network"]
---

My Daemon-set config:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: tuyennt33
  name: beyla
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: beyla
rules:
  - apiGroups: [ "apps" ]
    resources: [ "replicasets" ]
    verbs: [ "list", "watch" ]
  - apiGroups: [ "" ]
    resources: [ "pods", "services", "nodes" ]
    verbs: [ "list", "watch" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: beyla
subjects:
  - kind: ServiceAccount
    name: beyla
    namespace: tuyennt33
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: beyla
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: tuyennt33
  name: beyla
spec:
  selector:
    matchLabels:
      instrumentation: beyla
  template:
    metadata:
      labels:
        instrumentation: beyla
    spec:
      serviceAccountName: beyla
      hostPID: true # mandatory!
      containers:
      - name: beyla
        image: grafana/beyla:latest
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true # mandatory!
          # readOnlyRootFilesystem: true
        args:
        - -config=/etc/beyla/config.yaml
        volumeMounts:
        - mountPath: /etc/beyla
          name: beyla-config
        env:
        - name: BEYLA_KUBE_CLUSTER_NAME
          value: demo-beyla
      volumes:
      - name: beyla-config
        configMap:
          name: beyla-config
---

While beyla cannot get traces and metrics, I use alloy with beyla.ebpf component still get many metrics and traces with this config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: alloy-config
  namespace: tuyennt33
data:
  config.alloy: |
    logging {
      level = "debug"
    }

    beyla.ebpf "default" {
      debug = true
      attributes {
        kubernetes {
          enable = "true"
        }
      }
      discovery {
        services {
          kubernetes {
            namespace = "."
            pod_name = "."
          }
        }
      }
      metrics {
        features = [
          "application",
          "application_span",
          "application_service_graph",
          "application_process",
          "network",
        ]
        network {
          enable = true
        }
      }
      output {
        traces = [otelcol.processor.batch.default.input]
      }
    }

    prometheus.scrape "beyla" {
      targets = beyla.ebpf.default.targets
      honor_labels = true // required to keep job and instance labels
      forward_to = [prometheus.remote_write.default.receiver]
    }

    prometheus.remote_write "default" {
      endpoint {
        url = "http://grafana.tuyennt33.svc.cluster.local:9008/api/v1/push"
      }
    }

    otelcol.receiver.otlp "beyla" {
      http {}
      grpc {}
      output {
        metrics = [otelcol.processor.batch.default.input]
        logs    = [otelcol.processor.batch.default.input]
        traces  = [otelcol.processor.batch.default.input]
      }
    }

    otelcol.processor.batch "default" {
      output {
        metrics = [otelcol.exporter.otlphttp.mimir.input]
    //    logs    = [otelcol.exporter.otlp.beyla.input]
        traces  = [otelcol.exporter.otlp.tempo.input]
      }
    }

    otelcol.exporter.otlp "tempo" {
      client {
        endpoint = "grafana.tuyennt33.svc.cluster.local:44317"
        tls {
          insecure             = true
          insecure_skip_verify = true
        }
      }
    }

    otelcol.exporter.otlphttp "mimir" {
      metrics_endpoint = "http://grafana.tuyennt33.svc.cluster.local:9008/otlp/v1/metrics"
      client {
        endpoint = "grafana.tuyennt33.svc.cluster.local:9008"
        // tls {
        //   insecure             = true
        //   insecure_skip_verify = true
        // }
      }
    }
---

[grcevski]

Hi, based on your configuration, we'll instrument only the pods which have label "beyla", can you confirm that your pods actually have this label?

What happens if your discovery section of the Beyla config has only this:

    discovery:
      services:
        - k8s_namespace: ".*"

[bin1st090104]

thank you, sorry for my mistake, it run well now