Logstash hangs when connection to remote syslog server hangs

[micahlmartin]

I'm sending logs both to ES and a remote syslog server after parsing them. I ran into an issue today where after a few logs were sent to ES it would just stop processing completely. It turns out that I was unable to connect to my remote syslog server. The IP address is reachable but the security groups were preventing me from connecting which would just make my connection attempt hang. Turns out that this plugin also hangs trying to connect and then eventually just stops all logs from being processed at all with no indication as to why. I think there should be a timeout setting on the socket connection so that if it can't connect it just fails gracefully and stops trying to send messages.

[jordansissel]

This behavior (preventing data loss on output) is intended. If you want to
allow data loss, I am open to having a new setting that would choose to
lose data instead of waiting.

On Thursday, May 26, 2016, Micah Martin [email protected] wrote:

I'm sending logs both to ES and a remote syslog server after parsing them.
I ran into an issue today where after a few logs were sent to ES it would
just stop processing completely. It turns out that I was unable to connect
to my remote syslog server. The IP address is reachable but the security
groups were preventing me from connecting which would just make my
connection attempt hang. Turns out that this plugin also hangs trying to
connect and then eventually just stops all logs from being processed at all
with no indication as to why. I think there should be a timeout setting on
the socket connection so that if it can't connect it just fails gracefully
and stops trying to send messages.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#32,
or mute the thread
https://github.com/notifications/unsubscribe/AAIC6vYP_VrLY-l-Xqwyj23mYkGYIEnWks5qFjnygaJpZM4IoElP
.

[breml]

We implemented a codec, which decouples the respective output plugin from the rest of the outputs. Unfortunately, currently I am not allowed to open source this codec.
The basic idea is the following:

• The codec creates a copy of the event and puts the copy into a buffer (the buffer is based on the buffer used in the Elasticsearch output). After this step, the codec returns without calling the callback of the output plugin, which would effectively send the event.
• An additional thread reads on the buffer and sends the event via the callback of the output plugin.
• The buffer has implemented a high water mark. If this limit is reached, new events will not be placed into the buffer and are therefore dropped.

This decoupling allows us to achieve exactly what @micahlmartin requested. Our Logstash event processing becomes independent of the availability of some external target (in our case, the target is maintained by an other party).

[mvernimmen-CG]

Yes I would like to have such a setting where I can choose to loose data and not block message processing for other outputs, or not lose data but block all further processing.

[joaopauloksn]

Hey guys have you decided to implement this change? I believe this is important when dealing with third part products as output. A new setting would be awesome.

[breml]

@joaopauloksn No, there has no decision taken on this. As I mentioned above, this this problem was solved with a codec (unfortunately closed source). In my opinion, it does not make sense to implement this specific feature only in logstash-output-syslog. So the following two options come to my mind:

1. Implement an open source codec like the described above, so it could be used with every output plugin
2. Add this functionality to logstash core

The second one would be a long way to go so I suggest to stick to option 1.

[antoinetran]

@micahlmartin do you confirm this behavior is only visible in case we configure syslog output with TCP? Indeed, in UDP this behavior should not be visible from logstash point of view.