logstash[]: java.util.concurrent.RejectedExecutionException: event executor terminated

[pezthebasic]

Hi There

I try to using Logstash version 7.4.1 
am use Logstash config type input HTTP 

Example config
input { tcp { host => "127.0.0.1" port => "50001" } } filter { } output { elasticsearch { hosts => ["http://192.168.1.150:9200"] index => "TESTLOG" } }

my Logstash can work.
but I try to get Log from another machine with HTTP input type
change the host input => 192.168.1.151
Logstash can't create Pipeline and can't use it.

this error message below

[2020-05-20T19:45:24,729][ERROR][logstash.javapipeline ][main] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::Tcp host=>"192.168.1.151", port=>50001, id=>"26f2c5ac68c09557db4873c1707cb4e78a365618958e617c3cc76ba607537c32", enable_metric=>true, codec=><LogStash::Codecs::Line id=>"line_c8b61485-6d78-4f2a-9562-aa43c60fc9ba", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>, tcp_keep_alive=>false, dns_reverse_lookup_enabled=>true> Error: event executor terminated Exception: Java::JavaUtilConcurrent::RejectedExecutionException Stack: io.netty.util.concurrent.SingleThreadEventExecutor.reject(io/netty/util/concurrent/SingleThreadEventExecutor.java:821) io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(io/netty/util/concurrent/SingleThreadEventExecutor.java:327) io.netty.util.concurrent.SingleThreadEventExecutor.addTask(io/netty/util/concurrent/SingleThreadEventExecutor.java:320) io.netty.util.concurrent.SingleThreadEventExecutor.execute(io/netty/util/concurrent/SingleThreadEventExecutor.java:746) io.netty.channel.AbstractChannel$AbstractUnsafe.register(io/netty/channel/AbstractChannel.java:479) io.netty.channel.SingleThreadEventLoop.register(io/netty/channel/SingleThreadEventLoop.java:80) io.netty.channel.SingleThreadEventLoop.register(io/netty/channel/SingleThreadEventLoop.java:74) io.netty.channel.MultithreadEventLoopGroup.register(io/netty/channel/MultithreadEventLoopGroup.java:86) io.netty.bootstrap.AbstractBootstrap.initAndRegister(io/netty/bootstrap/AbstractBootstrap.java:331) io.netty.bootstrap.AbstractBootstrap.doBind(io/netty/bootstrap/AbstractBootstrap.java:282) io.netty.bootstrap.AbstractBootstrap.bind(io/netty/bootstrap/AbstractBootstrap.java:278) io.netty.bootstrap.AbstractBootstrap.bind(io/netty/bootstrap/AbstractBootstrap.java:260) org.logstash.tcp.InputLoop.run(org/logstash/tcp/InputLoop.java:87) java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440) org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_3_minus_java.lib.logstash.inputs.tcp.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.3-java/lib/logstash/inputs/tcp.rb:152) RUBY.inputworker(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:314) usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_input(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:306) org.jruby.RubyProc.call(org/jruby/RubyProc.java:295) org.jruby.RubyProc.call(org/jruby/RubyProc.java:274) org.jruby.RubyProc.call(org/jruby/RubyProc.java:270) java.lang.Thread.run(java/lang/Thread.java:748) [2020-05-20T19:45:26,006][INFO ][logstash.inputs.tcp ][main] Starting tcp input listener {:address=>"192.168.1.151:50001", :ssl_enable=>"false"} [2020-05-20T19:45:26,006][WARN ][io.netty.channel.AbstractChannel][main] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x676b3e5f] java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) ~[logstash-input-tcp-6.0.3.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) ~[logstash-input-tcp-6.0.3.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) ~[logstash-input-tcp-6.0.3.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) ~[logstash-input-tcp-6.0.3.jar:?] at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:479) [logstash-input-tcp-6.0.3.jar:?] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [logstash-input-tcp-6.0.3.jar:?] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [logstash-input-tcp-6.0.3.jar:?] at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [logstash-input-tcp-6.0.3.jar:?] at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:331) [logstash-input-tcp-6.0.3.jar:?] at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:282) [logstash-input-tcp-6.0.3.jar:?] at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:278) [logstash-input-tcp-6.0.3.jar:?] at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:260) [logstash-input-tcp-6.0.3.jar:?] at org.logstash.tcp.InputLoop.run(InputLoop.java:87) [logstash-input-tcp-6.0.3.jar:?] at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222-ea] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222-ea] at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:440) [jruby-complete-9.2.8.0.jar:?] at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:304) [jruby-complete-9.2.8.0.jar:?] at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36) [jruby-complete-9.2.8.0.jar:?] at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_3_minus_java.lib.logstash.inputs.tcp.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.3-java/lib/logstash/inputs/tcp.rb:152) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) [jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:172) [jruby-complete-9.2.8.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316) [jruby-complete-9.2.8.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72) [jruby-complete-9.2.8.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:171) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:158) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) [jruby-complete-9.2.8.0.jar:?] at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:306) [jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) [jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) [jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.Block.call(Block.java:129) [jruby-complete-9.2.8.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:295) [jruby-complete-9.2.8.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:274) [jruby-complete-9.2.8.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:270) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.8.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222-ea] [2020-05-20T19:45:26,007][ERROR][io.netty.util.concurrent.DefaultPromise.rejectedExecution][main] Failed to submit a listener notification task. Event loop shut down? java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) ~[logstash-input-tcp-6.0.3.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) ~[logstash-input-tcp-6.0.3.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) ~[logstash-input-tcp-6.0.3.jar:?] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) ~[logstash-input-tcp-6.0.3.jar:?] at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760) [logstash-input-tcp-6.0.3.jar:?] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428) [logstash-input-tcp-6.0.3.jar:?] at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) [logstash-input-tcp-6.0.3.jar:?] at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) [logstash-input-tcp-6.0.3.jar:?] at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1148) [logstash-input-tcp-6.0.3.jar:?] at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:490) [logstash-input-tcp-6.0.3.jar:?] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [logstash-input-tcp-6.0.3.jar:?] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [logstash-input-tcp-6.0.3.jar:?] at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [logstash-input-tcp-6.0.3.jar:?] at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:331) [logstash-input-tcp-6.0.3.jar:?] at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:282) [logstash-input-tcp-6.0.3.jar:?] at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:278) [logstash-input-tcp-6.0.3.jar:?] at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:260) [logstash-input-tcp-6.0.3.jar:?] at org.logstash.tcp.InputLoop.run(InputLoop.java:87) [logstash-input-tcp-6.0.3.jar:?] at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222-ea] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222-ea] at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:440) [jruby-complete-9.2.8.0.jar:?] at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:304) [jruby-complete-9.2.8.0.jar:?] at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36) [jruby-complete-9.2.8.0.jar:?] at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_3_minus_java.lib.logstash.inputs.tcp.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.3-java/lib/logstash/inputs/tcp.rb:152) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) [jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:172) [jruby-complete-9.2.8.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316) [jruby-complete-9.2.8.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72) [jruby-complete-9.2.8.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:171) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:158) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) [jruby-complete-9.2.8.0.jar:?] at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:306) [jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) [jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) [jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.Block.call(Block.java:129) [jruby-complete-9.2.8.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:295) [jruby-complete-9.2.8.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:274) [jruby-complete-9.2.8.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:270) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.8.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222-ea]

Thank you

[kares]

sorry for the long response - seems like an issue with the Netty library in LS 7.4.1, latest LS 7.x should do the trick.

[hackery]

This event executor terminated exception can also be thrown when the configuration selects a privileged port to listen on (< 1024).
This is an unfortunate cause to report for this case, as it gives no indication of the actual error.
It's also a strange result, as other cases (e.g. specifying the same port in multiple http inputs) are correctly reported as bind errors.