LogStash::Event should guard against incorrect use of reserved tags field.

[yaauie]

Logstash information:

Everything, up to and including 8.5.0

Description of the problem including expected versus actual behavior:

LogStash::Event does not guard against incorrect use of the top-level tags reserved field, so it is easy to create an event or modify an event such that later processing fails in surprising and hard-to-debug ways when something downstream attempts to use Event#tag that requires the underlying tags to be a single string or a list-of-strings. Since tagging is frequently done in failure-handling scenarios, this exception is generally not caught by the plugin that is already handling an error and results in a pipeline crash.

I would expect an attempt to set the top-level tags to an unsupported value to result in an error, so that the code doing so can handle it. If an event is being created with Event::new, the caller typically already has failure handling (such as JSON and CEF codec's or Elasticsearch input's behaviour of emitting a tagged event with the original payload). If an event is being modified by a plugin with Event#set, that plugin should similarly have error handling.

We have prior art for handling @timestamp reserved field:

• in Event::new, the resulting event is tagged with _timestampparsefailure and the original @timestamp value is retained in _@timestamp
• when using Event#set, an unparsable timestamp results in an exception, which is handled by the caller.

Steps to reproduce:

If we create an event whose top-level tags is a key/value map:

input {
  generator {
    message => '{"tags":{"poison":true}}'
    codec => json
  }
}

Subsequent use of that field can easily cause pipeline crash:

filter {
  # ruby filter catches exceptions and tags the event
  ruby {  code => 'fail "intentional"' }
}

OR

filter{
  # json filter catches parse errors and tags the event
  mutate { "add_field" => {"message" => "not_valid_json" } }
  json { source => "message" }
}

OR

filter{
  # date filter tags when it fails to parse
  mutate { "add_field" => {"when" => "yesterday" } }
  date { match => ["when", "yyyymm"] }
}

Notably, The filter add_tags decorator is immune because it manipulates the event directly instead of routing through Event#tag:

filter {
  mutate { "add_tag" => "kuzco" }
}

Errata

• @timestamp is well-guarded, both on Event::new and Event#set
• @metadata is not explicitly guarded, but reliably rejects anything other than a key/value map (with an ugly Java class cast exception); It may benefit from Event::new re-routing invalid metadata to _@metadata and a more descriptive error from Event#set.
• @version is unguarded, but also has no APIs that rely on it keeping shape
• tags is unguarded and needs to be

[roaksoax]

For future reference:

This change was release-noted in https://www.elastic.co/guide/en/logstash/current/logstash-8-7-0.html .
The setting available is event_api.tags.illegal as documented in https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html
Users can event_api.tags.illegal to warn to rollback the behavior