unable to see Kubernetes pod logs in vmui

[slopesaka]

Is your question request related to a specific component?

Victoria-logs

Describe the question in detail

the documentation says to set vector.enabled to true to enable vector logs shipper but I see no logs in vmui when using this values.yaml file to install Victoria-logs-single using helm

vector:
  # -- Enable deployment of vector
  enabled: true
  role: Agent
  dataDir: /vector-data-dir
  resources: {}
  podMonitor:
    enabled: true
  containerPorts:
    - name: prom-exporter
      containerPort: 9090
      protocol: TCP
  service:
    enabled: false
  existingConfigMaps:
    - vl-config
  customConfig:
    data_dir: /vector-data-dir
    api:
      enabled: false
      address: 127.0.0.1:8686
      playground: true
    sources:
      k8s:
        type: kubernetes_logs
      internal_metrics:
        type: internal_metrics
    transforms:
      parser:
        type: remap
        inputs: [k8s]
        source: |
          .log = parse_json(.message) ?? .message
          del(.message)
    sinks:
      console:
        type: console
        inputs: [ parser ]
        encoding:
          codec: json

Troubleshooting docs

• General - https://docs.victoriametrics.com/troubleshooting/
• vmagent - https://docs.victoriametrics.com/vmagent/#troubleshooting
• vmalert - https://docs.victoriametrics.com/vmalert/#troubleshooting

[AndrewChubatiuk]

hey @slopesaka
could you please leave only custom chart values in a description?
what is your kubernetes version?
how do you setup it?
also migrating this issue to helm-charts

[AndrewChubatiuk]

hey @slopesaka
do you still experience issues?

[trudzkimedo]

Hey.
Possibly related.

When used victoria-logs-single-0.8.1 / v0.41.0 with default config with vector set to enable on azure k8s, every json message that was generated by pod was not logged in but in it's place got:

missing _msg field; see https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field

It's same for today (victoria-logs-single-0.8.2 / v1.0.0) release.

this is easily to see when using istio ingress as it's logs are missing.

[trudzkimedo]

Quick update.
In my situation I misunderstood some concept here (especially that in fluent-bit it behaved bit differently)
My json message is parsed into log dict but since there was no message/msg/_msg field inside json payload default message (as above) is injected.
In fluent-bit the whole json payload was still left as _msg field.
Not sure if OP issue is same.

[AndrewChubatiuk]

hey @trudzkimedo
you can try to update processing pipeline to write a whole unparsed json into message field if neither msg nor message are found

vector:
  customConfig:
    transforms:
      parser:
        source: |
          message = parse_json(.message) ?? {}
          if is_nullish(message) || (is_null(message.msg) && is_null(message.message)) {
             message = .message
          }
          .log = message
          del(.message)

or add expected field names to VL-Msg-Field

vector:
  customConfig:
    sinks:
      vlogs:
        request:
          headers:
            VL-Msg-Field: message,msg,_msg,log.msg,log.message,log,<expected-value>