Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PipelineBusV2 block shutdown #16657

Closed
kaisecheng opened this issue Nov 7, 2024 · 1 comment · Fixed by #16682
Closed

PipelineBusV2 block shutdown #16657

kaisecheng opened this issue Nov 7, 2024 · 1 comment · Fixed by #16682
Assignees
@yaauie
Labels
bug int-shortlist status:work-in-progress

Comments

@kaisecheng
Copy link
Contributor

kaisecheng commented Nov 7, 2024 •
edited by andsel
Loading

Logstash version: 8.15.3

Pipeline to pipeline has upgraded to use PipelineBusV2. Logstash is unable to shutdown in 8.15.3 while downgrade to 8.14.3 has no such issue.

jstack found deadlock

Found one Java-level deadlock:
=============================
"pipeline_1":
  waiting to lock monitor 0x00007f5fd00062d0 (object 0x000000069bcb4fa8, a java.util.concurrent.ConcurrentHashMap$Node),
  which is held by "pipeline_2"

"pipeline_2":
  waiting to lock monitor 0x00007f5fe8001730 (object 0x00000006b3233ea0, a java.util.concurrent.ConcurrentHashMap$Node),
  which is held by "pipeline_3"

"pipeline_3":
  waiting to lock monitor 0x00007f6000002cd0 (object 0x0000000695d3fcf0, a org.jruby.gen.LogStash$$Plugins$$Builtin$$Pipeline$$Input_934266047),
  which is held by "Converge PipelineAction::StopAndDelete<pipeline_4>"

"Converge PipelineAction::StopAndDelete<pipeline_4>":
  waiting to lock monitor 0x00007f5fe8001730 (object 0x00000006b3233ea0, a java.util.concurrent.ConcurrentHashMap$Node),
  which is held by "pipeline_3"
Java stack information for the threads listed above 
"pipeline_1":
	at java.util.concurrent.ConcurrentHashMap.compute([email protected]/ConcurrentHashMap.java:1931)
	- waiting to lock <0x000000069bcb4fa8> (a java.util.concurrent.ConcurrentHashMap$Node)
	at org.logstash.plugins.pipeline.PipelineBusV2.unregisterSender(PipelineBusV2.java:63)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a8fb2c00.invokeInterface([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a90ac800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.DelegatingMethodHandle$Holder.delegate([email protected]/DelegatingMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.DelegatingMethodHandle$Holder.delegate([email protected]/DelegatingMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.output.RUBY$method$close$0(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/output.rb:43)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.output.RUBY$method$close$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/output.rb:42)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugin.RUBY$method$do_close$0(/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:98)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a955f000.invokeStatic([email protected]/LambdaForm$DMH)
	at java.lang.invoke.Invokers$Holder.invokeExact_MT([email protected]/Invokers$Holder)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:152)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:148)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:212)
	at org.jruby.RubyClass.finvoke(RubyClass.java:598)
	at org.jruby.runtime.Helpers.invoke(Helpers.java:708)
	at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:349)
	at org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.close(OutputStrategyExt.java:270)
	at org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.doClose(OutputStrategyExt.java:137)
	at org.logstash.config.ir.compiler.OutputDelegatorExt.close(OutputDelegatorExt.java:121)
	at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.doClose(AbstractOutputDelegatorExt.java:75)
	at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt$INVOKER$i$0$0$doClose.call(AbstractOutputDelegatorExt$INVOKER$i$0$0$doClose.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:841)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)
	at org.jruby.RubySymbol$SymbolProcBody.yieldInner(RubySymbol.java:1513)
	at org.jruby.RubySymbol$SymbolProcBody.doYield(RubySymbol.java:1528)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:108)
	at org.jruby.runtime.Block.yield(Block.java:189)
	at org.jruby.RubyArray.each(RubyArray.java:1981)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86f4c00.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8708800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fbc00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fc000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fbc00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fc000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$shutdown_workers$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:484)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.Invokers$Holder.invokeExact_MT([email protected]/Invokers$Holder)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:152)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:148)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:212)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:456)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:346)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:76)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:164)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:151)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:212)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:456)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:346)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)
	at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)
	at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
	at org.jruby.runtime.Block.call(Block.java:144)
	at org.jruby.RubyProc.call(RubyProc.java:354)
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)
"pipeline_2":
	at java.util.concurrent.ConcurrentHashMap.compute([email protected]/ConcurrentHashMap.java:1931)
	- waiting to lock <0x00000006b3233ea0> (a java.util.concurrent.ConcurrentHashMap$Node)
	at org.logstash.plugins.pipeline.PipelineBusV2$AddressStateMapping.mutate(PipelineBusV2.java:137)
	at org.logstash.plugins.pipeline.PipelineBusV2.lambda$unregisterSender$5(PipelineBusV2.java:64)
	at org.logstash.plugins.pipeline.PipelineBusV2$$Lambda/0x00007f60a9554000.accept(Unknown Source)
	at java.lang.Iterable.forEach([email protected]/Iterable.java:75)
	at org.logstash.plugins.pipeline.PipelineBusV2.lambda$unregisterSender$6(PipelineBusV2.java:64)
	at org.logstash.plugins.pipeline.PipelineBusV2$$Lambda/0x00007f60a946bc18.apply(Unknown Source)
	at java.util.concurrent.ConcurrentHashMap.compute([email protected]/ConcurrentHashMap.java:1940)
	- locked <0x000000069bcb4fa8> (a java.util.concurrent.ConcurrentHashMap$Node)
	at org.logstash.plugins.pipeline.PipelineBusV2.unregisterSender(PipelineBusV2.java:63)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a8fb2c00.invokeInterface([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a90ac800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.DelegatingMethodHandle$Holder.delegate([email protected]/DelegatingMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.DelegatingMethodHandle$Holder.delegate([email protected]/DelegatingMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.output.RUBY$method$close$0(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/output.rb:43)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.output.RUBY$method$close$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/output.rb:42)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugin.RUBY$method$do_close$0(/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:98)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a955f000.invokeStatic([email protected]/LambdaForm$DMH)
	at java.lang.invoke.Invokers$Holder.invokeExact_MT([email protected]/Invokers$Holder)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:152)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:148)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:212)
	at org.jruby.RubyClass.finvoke(RubyClass.java:598)
	at org.jruby.runtime.Helpers.invoke(Helpers.java:708)
	at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:349)
	at org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.close(OutputStrategyExt.java:270)
	at org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.doClose(OutputStrategyExt.java:137)
	at org.logstash.config.ir.compiler.OutputDelegatorExt.close(OutputDelegatorExt.java:121)
	at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.doClose(AbstractOutputDelegatorExt.java:75)
	at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt$INVOKER$i$0$0$doClose.call(AbstractOutputDelegatorExt$INVOKER$i$0$0$doClose.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:841)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)
	at org.jruby.RubySymbol$SymbolProcBody.yieldInner(RubySymbol.java:1513)
	at org.jruby.RubySymbol$SymbolProcBody.doYield(RubySymbol.java:1528)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:108)
	at org.jruby.runtime.Block.yield(Block.java:189)
	at org.jruby.RubyArray.each(RubyArray.java:1981)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86f4c00.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8708800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fbc00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fc000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fbc00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fc000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$shutdown_workers$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:484)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$shutdown_workers$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:475)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:209)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8718800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a80b9c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
	at org.jruby.runtime.Block.call(Block.java:144)
	at org.jruby.RubyProc.call(RubyProc.java:354)
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)
"pipeline_3":
	at org.logstash.plugins.pipeline.PipelineBusV2$AddressStateMapping.lambda$mutate$0(PipelineBusV2.java:148)
	- waiting to lock <0x0000000695d3fcf0> (a org.jruby.gen.LogStash$$Plugins$$Builtin$$Pipeline$$Input_934266047)
	at org.logstash.plugins.pipeline.PipelineBusV2$AddressStateMapping$$Lambda/0x00007f60a8f6df88.apply(Unknown Source)
	at java.util.concurrent.ConcurrentHashMap.compute([email protected]/ConcurrentHashMap.java:1940)
	- locked <0x00000006b3233ea0> (a java.util.concurrent.ConcurrentHashMap$Node)
	at org.logstash.plugins.pipeline.PipelineBusV2$AddressStateMapping.mutate(PipelineBusV2.java:137)
	at org.logstash.plugins.pipeline.PipelineBusV2.lambda$unregisterSender$5(PipelineBusV2.java:64)
	at org.logstash.plugins.pipeline.PipelineBusV2$$Lambda/0x00007f60a9554000.accept(Unknown Source)
	at java.lang.Iterable.forEach([email protected]/Iterable.java:75)
	at org.logstash.plugins.pipeline.PipelineBusV2.lambda$unregisterSender$6(PipelineBusV2.java:64)
	at org.logstash.plugins.pipeline.PipelineBusV2$$Lambda/0x00007f60a946bc18.apply(Unknown Source)
	at java.util.concurrent.ConcurrentHashMap.compute([email protected]/ConcurrentHashMap.java:1940)
	- locked <0x000000069bd0d9a8> (a java.util.concurrent.ConcurrentHashMap$Node)
	at org.logstash.plugins.pipeline.PipelineBusV2.unregisterSender(PipelineBusV2.java:63)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a8fb2c00.invokeInterface([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a90ac800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.output.RUBY$method$close$0(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/output.rb:43)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.output.RUBY$method$close$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/output.rb:42)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugin.RUBY$method$do_close$0(/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:98)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a955f000.invokeStatic([email protected]/LambdaForm$DMH)
	at java.lang.invoke.Invokers$Holder.invokeExact_MT([email protected]/Invokers$Holder)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:152)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:148)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:212)
	at org.jruby.RubyClass.finvoke(RubyClass.java:598)
	at org.jruby.runtime.Helpers.invoke(Helpers.java:708)
	at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:349)
	at org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.close(OutputStrategyExt.java:270)
	at org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.doClose(OutputStrategyExt.java:137)
	at org.logstash.config.ir.compiler.OutputDelegatorExt.close(OutputDelegatorExt.java:121)
	at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.doClose(AbstractOutputDelegatorExt.java:75)
	at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt$INVOKER$i$0$0$doClose.call(AbstractOutputDelegatorExt$INVOKER$i$0$0$doClose.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:841)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)
	at org.jruby.RubySymbol$SymbolProcBody.yieldInner(RubySymbol.java:1513)
	at org.jruby.RubySymbol$SymbolProcBody.doYield(RubySymbol.java:1528)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:108)
	at org.jruby.runtime.Block.yield(Block.java:189)
	at org.jruby.RubyArray.each(RubyArray.java:1981)
	at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:561)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)
	at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:209)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8718800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a80b9c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
	at org.jruby.runtime.Block.call(Block.java:144)
	at org.jruby.RubyProc.call(RubyProc.java:354)
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)
"Converge PipelineAction::StopAndDelete<pipeline_4>":
	at java.util.concurrent.ConcurrentHashMap.compute([email protected]/ConcurrentHashMap.java:1931)
	- waiting to lock <0x00000006b3233ea0> (a java.util.concurrent.ConcurrentHashMap$Node)
	at org.logstash.plugins.pipeline.PipelineBusV2$AddressStateMapping.mutate(PipelineBusV2.java:137)
	at org.logstash.plugins.pipeline.PipelineBusV2.tryUnlistenOrphan(PipelineBusV2.java:115)
	at org.logstash.plugins.pipeline.PipelineBusV2.unlistenBlocking(PipelineBusV2.java:100)
	- locked <0x0000000695d3fcf0> (a org.jruby.gen.LogStash$$Plugins$$Builtin$$Pipeline$$Input_934266047)
	at org.logstash.plugins.pipeline.PipelineBusV2.unlisten(PipelineBusV2.java:86)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a8fb2c00.invokeInterface([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a90ac800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.input.RUBY$method$stop$0(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/input.rb:77)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.input.RUBY$method$stop$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/input.rb:76)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.inputs.base.RUBY$method$do_stop$0(/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:102)
	at usr.share.logstash.logstash_minus_core.lib.logstash.inputs.base.RUBY$method$do_stop$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:99)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)
	at org.jruby.RubySymbol$SymbolProcBody.yieldInner(RubySymbol.java:1513)
	at org.jruby.RubySymbol$SymbolProcBody.doYield(RubySymbol.java:1528)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:108)
	at org.jruby.runtime.Block.yield(Block.java:189)
	at org.jruby.RubyArray.each(RubyArray.java:1981)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86f4c00.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8708800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fbc00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fc000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fbc00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fc000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$stop_inputs$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:468)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$stop_inputs$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:466)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.performIndirectCall(InvokeSite.java:735)
	at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:680)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86de800.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a870b000.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$shutdown$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:456)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$shutdown$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:448)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.performIndirectCall(InvokeSite.java:735)
	at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:657)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86e3800.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8721800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e2c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e3000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e2c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e3000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e2c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e3000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.stop_and_delete.RUBY$block$execute$1(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/stop_and_delete.rb:30)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8786800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e2c00.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e3000.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.pipelines_registry.RUBY$method$terminate_pipeline$0(/usr/share/logstash/logstash-core/lib/logstash/pipelines_registry.rb:192)
	at usr.share.logstash.logstash_minus_core.lib.logstash.pipelines_registry.RUBY$method$terminate_pipeline$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipelines_registry.rb:184)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.performIndirectCall(InvokeSite.java:725)
	at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:657)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86e3800.invokeVirtual([email protected]/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a875d800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.stop_and_delete.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/stop_and_delete.rb:29)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a884ac00.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard([email protected]/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite([email protected]/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$1(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8718800.invoke([email protected]/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a80b9c00.invokeExact_MT([email protected]/LambdaForm$MH)
	at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
	at org.jruby.runtime.Block.call(Block.java:144)
	at org.jruby.RubyProc.call(RubyProc.java:354)
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

Found 1 deadlock.

Workaround

PipelineBusV1 is not subject to this issue, and can be selected by adding the following to config/jvm.options:

# Use PipelineBusV1 to avoid possibility of deadlock during shutdown
# See https://github.com/elastic/logstash/issues/16657
-Dlogstash.pipelinebus.implementation=v1
@kaisecheng
Copy link
Contributor Author

kaisecheng commented Nov 8, 2024 •
edited by andsel
Loading

StopAndDelete try to lock A and then lock B
pipeline_3 try to lock B and then lock A

A is PipelineInput
B is AddressStateMapping.mapping

yaauie added a commit that referenced this issue Nov 8, 2024
@yaauie
Add Known Issue for 8.15.x releases and PipelineBusV2
921e603 
#16657
@yaauie yaauie mentioned this issue Nov 8, 2024
Merged
yaauie added a commit that referenced this issue Nov 11, 2024
@yaauie @karenzone
Add known issue for PipelineBusv2 to 8.15.x release notes (#16661)
e17ad0c 
* Add Known Issue for 8.15.x releases and PipelineBusV2

#16657

* Sync 8.15.4 with others

* Apply suggestions from code review

Co-authored-by: Karen Metts <[email protected]>

* Propagate suggestions to other entries

* Apply suggestions from code review

formatting:
- plus-to-continue only in lists,
- consistency of linebreaks before next section

Co-authored-by: Karen Metts <[email protected]>

---------

Co-authored-by: Karen Metts <[email protected]>
@yaauie yaauie mentioned this issue Nov 12, 2024
Merged
3 tasks
This was referenced Nov 18, 2024
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yaauie yaauie

Labels
bug int-shortlist status:work-in-progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Forwardport PR #16671 to main: PipelineBusV2 deadlock proofing elastic/logstash
3 participants
@yaauie @robbavey @kaisecheng