Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bad encoding in Logstash in Filebeat + Logstash + Elasticsearch stack #16835

Open
java-juggled-jazz opened this issue Dec 25, 2024 · 0 comments

Comments

@java-juggled-jazz
Copy link

Hi! I've just deployed Filebeat, Logstash and Elasticsearch
I use Filebeat on Linux machine that uses POSIX locale by default (tried to specify UTF-8 encoding, but nothing has changed)
Logstash within Docker container (locale is en_US.UTF-8) on Linux machine ( locale is C.UTF-8) receives beats but messages have bad decoding
Elasticsearch only receives Logstash entries and stores them as is

Here is Logstash config

input {
  tcp {
    port => 5044
    type => beats
    ssl_enable => false
    codec => plain { charset => "ASCII" }
  }
}

filter {
  if [type] == "beats" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601} %{POSINT:pid} %{WORD:log_level} %{DATA:module} \[(?<request>req-([a-fA-F0-9]{8}(?:-[a-fA-F0-9]{4}){3}-[a-fA-F0-9]{12})\S*)?%{DATA}\] %{GREEDYDATA:message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://XXXXXXXX:9200"]
    index => "XXXXXXXXX-[host][hostname]-%{+YYYY.MM.dd}"
  }
  file {
    path => "logstash-output"
  }
}

Here is Logstash output example

{"message":"\u000FY�*���./����<Udh��3?����3+��\u000F��Rqi�1�\u0005o�\u000F��9,�T�ah��\u001D���\u001E��W\u0003<��\u0000\u0015������\\���fI=g=���6y���w\u0000����2W\u0000\u0000\u0000\u00012C\u0000\u0000\u0003\u0001x^�Tak�6\u0010M�?��O-H>I�e[\u0010��^r��z\u001F�O\ra�����m��v�t����S�n�\u0014\u0002�`���\u001B�{��~�����������\u0018�����\u0011\u0016J(���*�Uj+��&�T�\u001B\u0018>�\u0014]���=bK.��i;Z\u001E\u0019��H������@���\u0001\u0016e*�T����G�#B��\u0014a�R��5[@`�\u0018]�����M\u001F���a\b\u0007���\u000F����v�\u0011\f5\u001DZO������)��\u000E�&X(��","@version":"1","tags":["_grokparsefailure"],"@timestamp":"2024-12-25T14:22:22.436611203Z","event":{"original":"\u000FY�*���./����<Udh��3?����3+��\u000F��Rqi�1�\u0005o�\u000F��9,�T�ah��\u001D���\u001E��W\u0003<��\u0000\u0015������\\���fI=g=���6y���w\u0000����2W\u0000\u0000\u0000\u00012C\u0000\u0000\u0003\u0001x^�Tak�6\u0010M�?��O-H>I�e[\u0010��^r��z\u001F�O\ra�����m��v�t����S�n�\u0014\u0002�`���\u001B�{��~�����������\u0018�����\u0011\u0016J(���*�Uj+��&�T�\u001B\u0018>�\u0014]���=bK.��i;Z\u001E\u0019��H������@���\u0001\u0016e*�T����G�#B��\u0014a�R��5[@`�\u0018]�����M\u001F���a\b\u0007���\u000F����v�\u0011\f5\u001DZO������)��\u000E�&X(��"},"type":"beats"}

I've tried to run Filebeat but write beats into file and charset was decoded properly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant