How to debug host service / kernel logging not working?

[AverageMarcus]

I have tried enabling sendings logs on one of my nodes by following the documentation but I'm having some trouble getting it working with Promtail and unsure how to go about debugging why it isn't working.

What I've changed:

• I have updated my promtail (running in my cluster) config to have a loki_push_api job configured:

   - job_name: talos-machine
     loki_push_api:
       server:
         http_listen_port: 3500
         grpc_listen_port: 3600
       labels:
         pushserver: talos-machine
  

• I've created a new NodePort service that points to this http_listen_port port for promtail and exposes it as 30001 on each node.
• On one of my worker nodes I have edited the machineconfig to add a logging destination:

   talosctl -n 192.168.1.168 get machineconfig -o yaml | yq '.spec.machine.logging'                                      
   destinations:
     - endpoint: "tcp://127.0.0.1:30001/promtail/api/v1/raw/"
       format: "json_lines"
  

• I have also updated the extraKernelArgs to also send kernel logs to the same endpoint:

   talosctl -n 192.168.1.168 get machineconfig -o yaml | yq '.spec.machine.install.extraKernelArgs'                      
   - talos.logging.kernel=tcp://127.0.0.1:30001/promtail/api/v1/raw/
  

What I've confirmed so far:

• The endpoint http://127.0.0.1:30001/promtail/api/v1/raw/ is available from a debug pod (kubectl debug) on the node in question and I can send a payload to it and see the entry show up in Loki
• No other logs are ending up in Loki besides my manual tests.
• I have rebooted the node. I have also performed an "upgrade" to the same version (v1.7.7)
• I've checked talosctl dmesg and cannot see any errors related to logging.
• From the dmesg output I can see that the kernal flag as been set:

   192.168.1.168: kern:  notice: [2024-12-12T09:35:30.05624826Z]: Kernel command line: talos.platform=metal talos.config=none console=ttyS0 console=tty0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on ima_template=ima-ng ima_appraise=fix ima_hash=sha512 talos.logging.kernel=tcp://127.0.0.1:30001/promtail/api/v1/raw/
  

• From the dmesg, some relevant logs lines that might indicate that logging is enabled but I'm not 100% sure if that is what they are for or not:

192.168.1.168: kern:    info: [2024-12-12T09:35:31.17615726Z]: netconsole: network logging started
...
192.168.1.168: user: warning: [2024-12-12T09:35:46.61671126Z]: [talos] enabling JSON logging {"component": "controller-runtime"}

Is there anything more I can do to debug this and figure out why the logs might not be sending? Can anyone see anything that might be wrong in my setup?

[smira]

The only supported protocol (in the docs you referenced) is straight TCP/UDP, so not sure what you mean by /promtail/api/v1/raw/.

You should be able to echo '{"message": "foo"}' | nc 127.0.0.1 3001

[AverageMarcus]

🤦 Doh!

Totally overlooked that!

I guess it's not compatible with Promtail then! 😔 Unless there's something I'm or aware of.

[AverageMarcus]

For those interested, I managed to come up with a hacky workaround 😅

I deployed https://github.com/bakins/tcp2stdout as a daemonset on all nodes with:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: tcp2stdout
  namespace: monitoring
  labels:
    app.kubernetes.io/name: tcp2stdout
  annotations:
    reloader.stakater.com/search: "true"
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: tcp2stdout
  template:
    metadata:
      labels:
        app.kubernetes.io/name: tcp2stdout
      annotations:
        prometheus.io/scrape: "false"
    spec:
      hostNetwork: true
      containers:
        - name: tcp2stdout
          image: "quay.io/bakins/tcp2stdout:v0.1.0"
          imagePullPolicy: IfNotPresent
          args:
            - "-addr"
            - "0.0.0.0:1313"
          ports:
            - containerPort: 1313
              name: tcplistener
          securityContext:
            readOnlyRootFilesystem: true
            runAsGroup: 1000
            runAsUser: 1000
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/not-ready
          operator: Exists
        - effect: NoExecute
          key: node.kubernetes.io/unreachable
          operator: Exists
        - effect: NoSchedule
          key: node.kubernetes.io/disk-pressure
          operator: Exists
        - effect: NoSchedule
          key: node.kubernetes.io/memory-pressure
          operator: Exists
        - effect: NoSchedule
          key: node.kubernetes.io/pid-pressure
          operator: Exists
        - effect: NoSchedule
          key: node.kubernetes.io/unschedulable
          operator: Exists

I then updated my promtail config with the following additional pipeline_stage:

        - match:
            selector: '{app="tcp2stdout"}'
            stages:
            - json:
                expressions:
                  level: "\"talos-level\""
                  service: "\"talos-service\""
                  timestamp: "\"talos-time\""
                  output: msg
            - labels:
                level:
                service:
                timestamp:
            - timestamp:
                source: timestamp
                format: RFC3339
            - output:
                source: output

I then updated my talos machineconfig to point to that tcp2stdout proxy:

    logging:
        destinations:
        - endpoint: "tcp://127.0.0.1:1313/"
          format: "json_lines"

It's not perfect but it at least gives me a way to persist some logs for later debugging.