The problem of data latency when network traffic is particularly high

[alleniverson33]

When the network traffic is particularly high, the Suricata alarm shows a delay of one hour in the dashboard. Is there any good solution to this？
I feel like Logstash can't forward it anymore

[mmguero]

I don't understand what you're saying: what do you mean the suricata alarm? Just that the data is taking an hour to show up? Or does the data have incorrect time stamps? What are your system specs? Is it with or without a network sensor? Is the other data (zeek?) showing up correctly?

I'm going to be on vacation until December 2nd, but I will follow up here when I return.

[alleniverson33]

I don't understand what you're saying: what do you mean the suricata alarm? Just that the data is taking an hour to show up? Or does the data have incorrect time stamps? What are your system specs? Is it with or without a network sensor? Is the other data (zeek?) showing up correctly?

I'm going to be on vacation until December 2nd, but I will follow up here when I return.

malcolm k8s
When I attack a target, theoretically Suricata can detect the attack record and display it on the dashboard
Testing in a testing environment is normal, as long as an attack occurs, the attack record can be displayed
But in production environments, network traffic may be particularly high, and attack records may not be real-time. It may take an hour to see the logs of the attack now
I'm not sure if it's a problem with Filebeat collection or if opensearch insertion is experiencing a bottleneck

[mmguero]

Converting to a troubleshooting discussion, we can continue the conversation there.

[mmguero]

Is Malcolm doing the traffic capture in an all-in-one/standalone mode, or are you using a network sensor running hedgehog linux?

What are the resources (CPU and memory, and are the disks HDD or SSD?) for the malcolm server and sensor, if applicable?

There are some things you can try to tweak resources for performance:

• Suricata tuning
• environment variables in logstash.env, particularly the pipeline variables and the memory allowances in LS_JAVA_OPTS.

This is not an issue I've ever seen before.