Not Populate Malcolm_beats_*

[devilman85]

I set up elasticsearch as the remote source where to send the data. i set up elasticsearch username and password. i am having problems populating the malcolm_beats_* index and in the logstash logs this message appears [WARN ][logstash.outputs.elasticsearch] Badly formatted index, after interpolation still contains placeholder: [%{[@metadata][malcolm_elasticsearch_index]}]

I cannot understand the error

[mmguero]

Hmmm... could we double-check some of your settings? Could you post the results from this command:

grep -v '^#' ./config/opensearch.env

If there are any hostnames/IP addresses you want to redact, that's fine, but there won't be anything sensitive in that file.

[devilman85]

During the configuration phase, does not this question appear: Expose Logstash port to external hosts? (y/N):

How can I get it so that the malcom beats index populates?

[mmguero]

You didn't answer my question. And we need to establish: what data are you expecting to be in the malcolm_beats index? The data that goes in that index includes:

• Metrics/statistics from a Hedgehog Linux sensor. Are you using an external hedgehog Linux sensor? Have you set it up to send the "miscbeat" data?
• NGINX access and error logs. You'd need to manually enable this variable in ./config/nginx.env

What data are you expecting to see in the malcolmbeats index that you're not seeing?

[devilman85]

no i'm using malcolm to send data to a remote elasticsearch cluster i already have. i don't have Hedgehog Linux sensor

[mmguero]

That's what I'm saying, there won't be any data in the malcolm_beats index because mostly what that is used for is resource utilization for tracking the sensor. All of the network log data is in the arkime_sessions3* index, the malcolm_beats_* indexes won't have anything in them unless you enable the nginx access and error logs, which is optional only if you need/want them.

[mmguero]

If you would post the contents of your opensearch.env like I asked for, I can get a better idea for what your settings are.

[devilman85]

grep -v '^#' ./config/opensearch.env
OPENSEARCH_PRIMARY=elasticsearch-remote
OPENSEARCH_URL=https://192.168.1.10:9200
OPENSEARCH_CREDS_CONFIG_FILE=/var/local/curlrc/.opensearch.primary.curlrc
OPENSEARCH_SSL_CERTIFICATE_VERIFICATION=false
OPENSEARCH_SECONDARY=
OPENSEARCH_SECONDARY_URL=
OPENSEARCH_SECONDARY_CREDS_CONFIG_FILE=/var/local/curlrc/.opensearch.secondary.curlrc
OPENSEARCH_SECONDARY_SSL_CERTIFICATE_VERIFICATION=false
OPENSEARCH_JAVA_OPTS=-server -Xmx10g -Xms10g -Xss256k -XX:-HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dlog4j.formatMsgNoLookups=true

MALCOLM_NETWORK_INDEX_PATTERN=arkime_sessions3-*
MALCOLM_NETWORK_INDEX_TIME_FIELD=firstPacket
MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}
MALCOLM_OTHER_INDEX_PATTERN=malcolm_beats_*
MALCOLM_OTHER_INDEX_TIME_FIELD=@timestamp
MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%d}
ARKIME_NETWORK_INDEX_PATTERN=arkime_sessions3-*
ARKIME_NETWORK_INDEX_TIME_FIELD=firstPacket

logger.level=WARN
bootstrap.memory_lock=true
MAX_LOCKED_MEMORY=unlimited
discovery.type=single-node
cluster.routing.allocation.disk.threshold_enabled=false
cluster.routing.allocation.node_initial_primaries_recoveries=8
indices.query.bool.max_clause_count=4096
path.repo=/opt/opensearch/backup

but not malcolm_beats_* populate

[mmguero]

Thanks. I think maybe without a sensor there's just nothing even enabled to be written into that index.

If you want to try something, edit ./config/nginx.env and change NGINX_LOG_ACCESS_AND_ERRORS to true, then restart Malcolm. My guess is then you'll start seeing at least some data in that index.

[devilman85]

thanks for suggestion... i have the logstash conteiner unhelathy.... i see in the log the host is 0.0.0.0:5044... But shouldn't it have as its address the docker address which in my case is 172.18.0.17?

in fact the conteiner tries to connect to 172.18017:5044 connection refused

[mmguero]

I don't think the logs internally will show the IP address of the docker network. Once you restart Malcolm, it takes a few minutes for all the pipelines to come up. After a few minutes the connection refused errors should go away.

[devilman85]

thank for suggestio