[Prometheus Receiver] Prometheus Receiver configuration failing with new 0.112.0 (latest)

[developer1622]

Component(s)

receiver/prometheus

What happened?

Description

Thank you.
Previously following scrape config used to work fine, but with the new latest changes , I am getting error, attached

    receivers:
      prometheus/coredns:
        config:
          scrape_configs:
            - job_name: "dns"
              scrape_interval: 2m
              scrape_timeout: 59s
              kubernetes_sd_configs:
                - role: pod
                  namespaces:
                    names:
                      - kube-system
              relabel_configs:
                - source_labels:
                    - "__meta_kubernetes_namespace"
                    - "__meta_kubernetes_pod_label_k8s_app"
                  action: keep
                  regex: "kube-system;kube-dns"
                - source_labels:
                    - "__meta_kubernetes_pod_container_port_name"
                  action: keep
                  regex: "metrics"
                - source_labels: [ __meta_kubernetes_pod_ip, __meta_kubernetes_pod_container_port_number ]
                  action: replace
                  target_label: __address__
                  regex: (.*);(.*)
                  replacement: $$1:$$2

Steps to Reproduce

Run the above scrape config, results in the error

Expected Result

No error

Actual Result

failed to get config: cannot resolve the configuration: cannot convert the confmap.Conf: environment variable "2" has invalid name: must match regex ^[a-zA-Z_][a-zA-Z0-9_]*$

Collector version

v0.112.0

Environment information

Environment

OS: (e.g., "Ubuntu 22.04")
go version go1.22.8 linux/amd64

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

Seems like two dollars are not taking correctly. is that the issue ?

failed to get config: cannot resolve the configuration: cannot convert the confmap.Conf: environment variable "2" has invalid name: must match regex ^[a-zA-Z_][a-zA-Z0-9_]*$

Thank you

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

That does seem odd. Based on https://opentelemetry.io/docs/collector/configuration/#environment-variables, that should work.

Might be related to open-telemetry/opentelemetry-collector#7111?

cc @mx-psi @TylerHelmuth
Any ideas what might have caused $$ to not work anymore? I wonder if it could be related to open-telemetry/opentelemetry-collector#10510?

[mx-psi]

If using the operator this could be it open-telemetry/opentelemetry-operator#3262 (comment)

[TylerHelmuth]

@developer1622 I wasn't able to reproduce your issue locally. Are you running in kubernetes? If so you need to deal with the fact that k8s wants to escape $$ also, so you need $$$.