Skip to content

Commit

Permalink
LIR: merge hash attributes of same name to support legacy configurations
Browse files Browse the repository at this point in the history
A [build failure][] of the grok plugin when run through LIR/lscl indicates that
there is an expectation for multiple Attributes of the same name to be merged
together.

This commit ports the failing spec in the plugin to an abstraction that can be
tested within logstash-core, and adds a caveat to the LSCL LIR-builder to
ensure that we merge the hashes in a way that is compatible with legacy
behaviour.

NOTE: when multiple Attributes of the same name are used in a single config,
it's possible to create configurations that circumvent `AST::Hash`'s ability to
report duplicate keys.

[build failure]: https://travis-ci.org/logstash-plugins/logstash-filter-grok/builds/293778268

Fixes #8597
  • Loading branch information
yaauie committed Nov 7, 2017
1 parent 69845ec commit db50857
Show file tree
Hide file tree
Showing 2 changed files with 39 additions and 2 deletions.
12 changes: 10 additions & 2 deletions logstash-core/lib/logstash/compiler/lscl.rb
Original file line number Diff line number Diff line change
Expand Up @@ -103,10 +103,18 @@ def expr_attributes
end
}.reduce({}) do |hash, kv|
k, v = kv
if hash[k].nil?
existing = hash[k]
if existing.nil?
hash[k] = v
elsif existing.kind_of?(::Hash)
# For legacy reasons, a config can contain multiple `AST::Attribute`s with the same name
# and a hash-type value (e.g., "match" in the grok filter), which are merged into a single
# hash value; e.g., `{"match" => {"baz" => "bar"}, "match" => {"foo" => "bulb"}}` is
# interpreted as `{"match" => {"baz" => "bar", "foo" => "blub"}}`.
# (NOTE: this bypasses `AST::Hash`'s ability to detect duplicate keys)
hash[k] = existing.merge(v)
else
hash[k] += v
hash[k] = existing + v
end
hash
end
Expand Down
29 changes: 29 additions & 0 deletions logstash-core/spec/logstash/compiler/compiler_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,35 @@ def j
expect(c_plugin).to ir_eql(j.iPlugin(INPUT, "generator", expected_plugin_args))
end
end

describe "a filter plugin that repeats a Hash directive" do
let(:source) { "input { } filter { #{plugin_source} } output { } " }
subject(:c_plugin) { compiled[:filter] }

let(:plugin_source) do
%q[
grok {
match => { "message" => "%{WORD:word}" }
match => { "examplefield" => "%{NUMBER:num}" }
break_on_match => false
}
]
end

let(:expected_plugin_args) do
{
"match" => {
"message" => "%{WORD:word}",
"examplefield" => "%{NUMBER:num}"
},
"break_on_match" => "false"
}
end

it "should merge the contents of the individual directives" do
expect(c_plugin).to ir_eql(j.iPlugin(FILTER, "grok", expected_plugin_args))
end
end
end

context "inputs" do
Expand Down

0 comments on commit db50857

Please sign in to comment.