Elasticsearch - Send Buffer is full

[Pacerino]

Hello everyone,

go-dnscollector is running in my Kubernetes cluster. Everything is running fine so far. Unfortunately, I regularly get the message that the buffer is full and packets are therefore being dropped.

I have already played with the config, but I don't quite understand it. Maybe someone can help me.

ERROR: 2024/09/20 22:08:27.798948 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:29.167903 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:29.895738 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:31.696430 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:33.759606 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:35.533266 worker - [elastic] elasticsearch - automatic flush, send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:38.231494 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:40.265980 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:42.023559 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:44.002391 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:45.535341 worker - [elastic] elasticsearch - automatic flush, send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:46.698195 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:49.575917 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:51.698154 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:53.699330 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:55.432225 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:56.406654 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped
ERROR: 2024/09/20 22:08:58.226573 worker - [elastic] elasticsearch - Send buffer is full, bulk dropped

Here is my config:

global:
  trace:
    verbose: false
  worker:
    interval-monitor: 10
    buffer-size: 4096
pipelines:
  - name: pdns-01
    powerdns:
      listen-ip: 0.0.0.0
      listen-port: 6000
    transforms:
      latency:
        measure-latency: true
        unanswered-queries: true
        queries-timeout: 2
    routing-policy:
      forward: [ prom, elastic ]
  - name: pdns-02
    powerdns:
      listen-ip: 0.0.0.0
      listen-port: 6001
    routing-policy:
      forward: [ prom, elastic ]
  - name: prom
    prometheus:
      listen-ip: 0.0.0.0
      listen-port: 8080
      histogram-metrics-enabled: true
      basic-auth-login: prometheus
      basic-auth-pwd XXXXXXXXX
  - name: elastic
    elasticsearch:
      server: "http://XXXXXXX"
      index:  "dnscollector"
      chan-buffer-size: 16384
      bulk-size: 5242880 # chuck of 5MB
      flush-interval: 10 # flush every 5s
      compression: gzip
      bulk-channel-size: 50 

According to Prometheus, the server has a throughput of 4000 operations per second. So that you know how much the PDNS has to do.

Thank you very much!

[dmachard]

Hi,

The message Send buffer is full, bulk dropped appears because Elasticsearch is too slow to ingest all the data. To prevent the DNS collector from being blocked, DNS message bulks are dropped when this happens, minimizing further impact.

Have you tried increasing the bulk-size and bulk-channel-size parameters?

Additionally, some optimizations might be possible in the DNS collector, such as establishing multiple connections to the Elasticsearch instance.

You also have other options, such as applying sampling to your DNS logs using the filtering transformer.

Denis

[Pacerino]

Hi Denis!

First of all, thank you for your answere!
Could you explain the difference between bulk-size and bulk-channel-size? From the names i guess they are in some relation.

Also with multiple connections, my ES got 6 nodes, can i put all the logs to the nodes at the sametime and if so, how could i do that? Simply define 5 more pipelines and define them in the PDNS Pipeline?

Thanks in advance!

[Pacerino]

Small update: Setting following config killed my Kubernetes Nodes.

chan-buffer-size: 16384
bulk-size: 33554432 # chuck of 32MB
flush-interval: 5 # flush every 5s
compression: gzip
bulk-channel-size: 50 

[dmachard]

Could you explain the difference between bulk-size and bulk-channel-size? From the names i guess they are in some relation.

bulk-size refers to the size of the batch of DNS messages sent to your Elasticsearch instance.
Since sending these batches can take time, bulk-channel-size defines the number of batches the DNS collector can hold in memory before dropping them.

I think it's better to have smaller batches (e.g., 1MB) and increase the bulk-channel-size to 100.

Also with multiple connections, my ES got 6 nodes, can i put all the logs to the nodes at the sametime and if so, how could i do that?

the elasticsearch logger does not support multiple connections but you can try something like that

pipelines:
  - name: pdns-01
    powerdns:
      ....
    routing-policy:
      forward: [ prom, elastic1 ]
  - name: pdns-02
    powerdns:
      ....
    routing-policy:
      forward: [ prom, elastic2 ]
  - name: prom
    prometheus:
      listen-ip: 0.0.0.0
      listen-port: 8080
      histogram-metrics-enabled: true
      basic-auth-login: prometheus
      basic-auth-pwd XXXXXXXXX
  - name: elastic1
    elasticsearch:
      server: "http://XXXXXXX"
      index:  "dnscollector"
      chan-buffer-size: 16384
      bulk-size: 5242880 # chuck of 5MB
      flush-interval: 10 # flush every 5s
      compression: gzip
      bulk-channel-size: 50 
  - name: elastic2
    elasticsearch:
      server: "http://XXXXXXX"
      index:  "dnscollector"
      chan-buffer-size: 16384
      bulk-size: 1048576 # chuck of 5MB
      flush-interval: 10 # flush every 5s
      compression: gzip
      bulk-channel-size: 100