How to Read PCAPs and Generate Logs with Zeek and Suricata in Hedgehog

[MLIndeed]

Hi everyone,

First of all, thank you for creating this amazing project!

I’ve recently set up Hedgehog and launched it using the Malcolm/scripts/start command. According to Docker, everything seems to be running smoothly—Zeek, Zeek-Live, Suricata, and Suricata-Live are all healthy.

However, when I place PCAP files in the Malcolm/pcap directory, it looks like they're not being processed, as neither Zeek nor Suricata are generating log files.

I’ve already confirmed that the directories are correctly mounted. I also accessed the containers and verified that the PCAP files are present in the correct location with the proper permissions.

Is there something I might have overlooked? Are there any additional containers or processes that need to be running to get this working?

I’m running Hedgehog in standalone mode, so I don’t have access to a web interface.

I appreciate any help or suggestions!

Thank you!

[mmguero]

A couple of things just terminology-wise just to avoid confusion: Malcolm, whether installed via the ISO installer or running in Docker on another platform, is the "aggregator" or server portion of the project. Hedgehog Linux is a network sensor OS installed with an installation ISO for capturing live traffic and forwarding information about to a Malcolm server/aggregator.

To me what it looks like is you're running Malcolm, not Hedgehog Linux, which is good if what you want to do is process existing PCAP like you're describing.

Now as far as your issue, let's first double check and make sure that the files are being uploaded the right way. There is an upload user interface you can use to upload PCAP files, or you can drop them into the ./pcap/upload/ directory under the Malcolm installation directory. In your original comment above you say you're putting them in the ./pcap/ directory, perhaps that's the problem?

Another thing you could double check is your answers to some of the configuration questions, particularly answering "yes" to both of these:

• Automatically analyze all PCAP files with Suricata?
• Automatically analyze all PCAP files with Zeek?

Also, verify that the packets are in PCAP format rather than PCAPNG format, which is not fully supported by the project yet.

Those are a few things you can double check, then report back here and we'll continue to debug if needed.

[MLIndeed]

Thank you for your answer!

I will try out everything you wrote to me in the list and get back to you!

Yes it is malcolm, I just disabled some containers and ran it using docker and in the "hedgehog mode" (profile "hedgehog" of the composefile) with the minimal opensearch - so i think there is no web-acess. Is that correct?

[mmguero]

Well, the hedgehog run profile doesn't provide any OpenSearch, it's only use is to forward metadata to another instance of Malcolm. If you want Malcolm to maintain its own OpenSearch instance, you need to use the malcolm run profile which will provide both OpenSearch and the UI elements. You may wish to review the configuration questions carefully to decide what is is you're really wanting to do.

[MLIndeed]

Thx for the answer. Putting the PCAP files into Malcolm/pcap/upload did not help unfortunately. I made sure they are PCAP (without NG) files.

auto-zeek and auto-suricata are on.

Furthermore, i modified the composefile and turned off filebeat, arkime and logstash by removing "hedgehog" from the profiles-list.

This is my installation setup:
./scripts/install.py
--verbose true
--defaults true
--configure false
--runtime docker
--malcolm-profile false
--dark-mode true
--image-arch amd64
--https true
--ldap false
--ldap-mode openldap
--ldap-start-tls false
--restart-malcolm false
--reverse-proxied false
--traefik-host ""
--traefik-host-opensearch ""
--traefik-entrypoint ""
--traefik-resolver ""
--network-name "hh-network"
--opensearch opensearch-local
--opensearch-memory "4g"
--opensearch-url ""
--opensearch-ssl-verify true
--opensearch-compress-snapshots true
--opensearch-secondary ""
--opensearch-secondary-url ""
--opensearch-secondary-ssl-verify true
--dashboards-url ""
--logstash-host ""
--logstash-memory "2g"
--logstash-workers 1
--logstash-expose false
--opensearch-expose false
--filebeat-tcp-expose false
--sftp-expose false
--pcap-path "pcap"
--zeek-path "zeek"
--suricata-path "suricata"
--opensearch-path "opensearch"
--opensearch-snapshot-path ""
--delete-old-pcap false
--delete-pcap-threshold "250GB"
--extracted-file-max-size-threshold "250GB"
--extracted-file-total-disk-usage-percent-threshold "90%"
--delete-index-threshold "1TB"
--index-management-enable true
--index-management-hot-warm-enable false
--index-management-optimization-time-period "30d"
--index-management-spi-data-retention "90d"
--index-management-replicas 0
--index-management-weeks-of-history 13
--index-management-segments 1
--auto-arkime false
--auto-suricata true
--suricata-rule-update true
--auto-zeek true
--zeek-ics false
--zeek-ics-best-guess false
--reverse-dns false
--auto-oui false
--auto-freq false
--file-extraction none
--file-preservation none
--extracted-file-server false
--extracted-file-server-zip false
--extracted-file-server-password ""
--extracted-file-clamav false
--extracted-file-yara false
--extracted-file-capa false
--virustotal-api-key ""
--file-scan-rule-update true
--netbox true
--netbox-enrich false
--netbox-autopopulate false
--netbox-auto-prefixes false
--netbox-site-name ""
--live-capture-iface ""
--live-capture-filter ""
--live-capture-iface-tweak false
--live-capture-arkime false
--live-capture-arkime-node-host ""
--live-capture-netsniff false
--live-capture-tcpdump false
--live-capture-zeek false
--live-capture-suricata false
--node-name ""

And the configuration:

./scripts/auth_setup
--auth-noninteractive true
--verbose
--file "${loc}/docker-compose.yml"
--environment-dir ./env
--profile hedgehog
--runtime docker
--no-tmpdir-override
--start
--namespace default
--reclaim-persistent-volume
--auth
--auth-admin-username admin
--auth-admin-password-openssl abc
--auth-admin-password-htpasswd abc
--auth-arkime-password abc
--auth-generate-webcerts
--auth-generate-fwcerts
--auth-generate-netbox-passwords
--logs
--lines 100
--status
--urls
--service default
--netbox-backup netbox_backup.sql

[mmguero]

I'm still not coming through clearly: the Malcolm profile is the only profile that processes uploaded PCAP. The way you're configuring it (with the hedgehog run profile) is only for live traffic capture.

If you want to process PCAP files by uploading them, you need to configure Malcolm in its full Malcolm run profile.

Also, you should never have to modify the docker-compose file directly to turn on and off services. You don't understand what the services are doing: for example, by disabling the filebeat service, Zeek logs generated from the PCAP files will never make their way into the parsing pipeline.

Blow away your Malcolm installation, start over, and walk through the configuration wizard step by step. If you want to process PCAP files that you're uploading or copying into the pcap upload directory, you need to set up the Malcolm run profile, not the hedgehog run profile.

[MLIndeed]

Ok thx for your answer.
I got it now.

One more question: When Zeek/Suricata analyze live traffic, that would also be fine for me - therefore i think that I'm still good with Hedgehog. Do they capture directly into the RAM then or do they store the files somewhere?

Thank you!

[mmguero]

Both the Malcolm aggregator and Hedgehog Linux network sensor store artifacts locally on the disk. Here's the documentation for how you can configure the parameters for retention and storage management:

• for Malcolm
• for Hedgehog

Generally speaking, during the Malcolm setup during the configuration questions it will ask you where you want to store artifacts like PCAP, database indices, etc. Similarly, during the capture configuration for of Hedgehog Linux it will ask you where you want to store PCAP files and logs. As the disk becomes full, the older artifacts are deleted to make room for the new ones.

[MLIndeed]

Alright, thank you!