Malcolm v3.3.0

List of changes in Malcolm v3.3.0:

v3.2.1...v3.3.0

Version 3.3.0 is a feature release of Malcolm.

• New features

  • Automatically create some broadly useful anomaly detectors when initializing Kibana
    • connection size
    • file transfer MIME type
    • action and result (by application protocol)
  • Configurable event severity scoring (idaholab#19) and new Severity dashboard

• Other changes

  • vagrant-based ISO build can now work with either VirtualBox or libvirt providers
  • change wording of terms such as "master"/"slave" to "client"/"server" as instructed by DHS directive

• Version updates

  • Update base image for Debian-based Docker images from 10 (buster) to 11 (bullseye)
  • Update Yara to 4.1.2
  • Update Capa to 2.0.0
  • Update Spicy to 1.2.1
  • Update remainder of python 2 code to python 3

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v3.2.1

List of changes in Malcolm v3.2.1:

v3.2.0...v3.2.1

This is a minor match release to fix a single regression bug in the Zeek LDAP parser that slipped into Malcolm v3.2.0. It is otherwise identical to that release.

• Bugs:
  • LDAP parser broken (ldap.spicy:393 unset optional value) if built from source since May 31 (idaholab#52)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v3.2.0

List of changes in Malcolm v3.2.0:

v3.1.0...v3.2.0

• New features

  • "Best Guess" Fingerprinting for ICS Protocols - In an effort to help identify more ICS traffic, Malcolm can use "buest guess" method based on transport protocol (e.g., TCP or UDP) and port(s) to categorize potential traffic communicating over some ICS protocols without full parser support. This feature involves a mapping table and a Zeek script to look up the transport protocol and destination and/or source port to make a best guess at whether a connection belongs to one of those protocols. These potential ICS communications are categorized by vendor where possible. The list of ICS protocols' ports was adapted from various public sources, including, but not limited to, Grassmarlin's fingerprints and ITI/ICS-Security-Tools' list of Control Systems Ports.

• Improvements and bug fixes

  • Allow configuring the number of concurrent requests for ClamAV scanning, Yara and Capa via environment variables (CLAMD_MAX_REQUESTS, YARA_MAX_REQUESTS and CAPA_MAX_REQUESTS)
  • Zeek plugins to detect CVE-2021-31166 and pingback vulnerabilities
  • Move creation of custom fields and views to Arkime's config.ini
  • LDAP bind credentials world readable in docker (idaholab#47 and #171)
  • zeek_template index template not created if index management not enabled (idaholab#50)
  • kibana offline maps server not started (idaholab#51)

• Version bumps

  • Yara to 4.1.1
  • Zeek to 4.0.3
  • Spicy to 1.1.0
  • Alpine to 3.14
  • NGINX to 1.20.1
  • Linux kernel to 5.10 (for ISO installs)
  • urllib3 to 1.26.5 (#169)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v3.1.0

• Network analyzers

  • Added support for EtherCAT (ICS protocol)
  • Fixed and improved Spicy-based LDAP analyzer
  • Detect VPN protocols IPsec, OpenVPN and WireGuard

• New or improved

  • Updated many Kibana dashboards and added dashbaords for newly-supported network protocols
  • Improved output of debug logs from docker images
  • Many minor improvements to underlying system for ISO installations
  • Massively cut build time for Hedgehog ISO and Zeek Docker container by using .deb packages from released versions rather than building from source
  • During build, install all Zeek plugins via zkg

• Version updates

  • Zeek v4.0.1
  • Spicy v1.0.0
  • Open Distro For Elasticsearch v1.13.2
  • Yara v4.1.0
  • Capa v1.6.3
  • switch from centos:7 to amazonlinux:2 for base Docker image to build Kibana plugins
  • stunnel v5.59
  • NGINX v1.20.0
  • LLVM/clang toolchain v11
  • Flask-Cors v3.0.9 for Hedgehog kiosk interface (dependabot-flagged security alert)
  • latest updates of various Zeek plugins, system and python packages, etc.
  • all Python scripts updated to Python 3

• Bugs fixed

  • When LDAP authentication is used instead of BASIC authentication, show a landing page rather than a server error when attempting to browse to the local authentication management interface
  • Fixed a regression bug where Malcolm fails to start correctly if not using UID/GID 1000:1000
  • Don't automatically expose elasticsearch (and logstash) ports unless explicitly configured to do so
  • freshclam should update the clamav database during docker image build

Malcolm v3.0.1

Malcolm v3.0.1 contains some important version updates for several of its components and fixes a few bugs. Please continue reading for more details.

List of changes in Malcolm v3.0.1:

v3.0.0...v3.0.1

• Version bumps
  • Open Distro for Elastic (v1.13.0), which adds the following functionality over the previous release
    • Reporting
    • Historical data anomaly detection
  • ODFE v1.13.0 is based on the Elastic components 7.10.2 (elasticsearch, kibana, logstash, beats)
  • Zeek 3.0.13
  • NGINX 1.19.7
  • Alpine Linux 3.13 Docker base layer
  • docker-compose 1.28.5 in Malcolm installable ISO version
• Restored the sankey visualization which was temporarily removed in Malcolm v3.0.0 (although there are still a few minor cosmetic issues with it)
• Removed port 8443 for upload (now just use /upload over the regular HTTPS port)
• Fixed issue with ODFE email alerts not being able to use self-signed SMTP certificates by importing CA certs in nginx/ca-trust into the JDK trust store for Elasticsearch and Logstash (see idaholab#37)
• Don't expose the Elasticsearch 9200 by default, it must now be explicitly be enabled during install.py -c (see idaholab#38)
• For ISO-installed versions of Malcolm and Hedgehog Linux, populate /etc/os-release with information about the build/release version
• Populate user-agent for a few clients (Arkime's moloch-capture, some hedgehog test connection processes) so they're not just sent as blank when communicating with Malcolm
• Added Arkime link to Kibana dashboards' navigation pane
• Fix some issues in control script with older python3 versions (3.6.x) with contextlib.nullcontext not being available
• Fix suggestion for yum-based distributions to install python 3 requests via pip

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v3.0.0

Malcolm v3.0.0 is a major release with some big replacements in the project's underpinnings, including a few backwards compatibility-breaking changes. Please continue reading for more details.

List of changes in Malcolm v3.0.0:

v2.6.1...v3.0.0

• Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see idaholab#15. This is a major change which breaks backwards compatibility for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run install.py --configure (see System configuration and tuning) prior to running Malcolm v3.0.0.
  • Kibana comments replaced with Notebooks
  • Kibana elastalert plugin replaced with Alerting plugin
  • Elasticsearch curator replaced with Index Management plugin
  • The third-party Sankey visualization plugin has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15)
  • The third-party Kibana drill-down plugin providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5)
  • In addition to those replacements, the Real Time Anomaly Detection feature is now available:
    • Real Time Anomaly Detection in Open Distro for Elasticsearch blog announcement
    • Anomaly Detection documentation and source code for Elasticsearch and Kibana components
    • Random Cut Forests writeup
  • If you are not up-to-date on the recent developments in Elasticsearch's licensing, here are a few of the official statements from the various parties involved:
    • Elastic.co's original announcement, clarification, Elastic License v2 announcement, "Why we had to change" post and FAQ on 2021 License Change
    • Open Distro for Elasticsearch initial response post, Amazon AWS Open Source Blog post and fork updates post
• Malcolm startup time (especially the Logstash container) has been reduced drastically
• Improvements to Malcolm's prebuilt Kibana dashboards
• Improvements to build scripts
• Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux
• Minor other bug fixes and performance improvements
• Version bump
  • Yara v4.0.5

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v2.6.1

Malcolm v2.6.1 contains the following changes:

v2.6.0...v2.6.1

• Added TFTP Zeek parser and corresponding Logstash parsing, Arkime WISE support and Kibana dashboards
• Provide browser-based access to zeek/extracted-files directory (idaholab#34)
• Fix LDAP analyzer not parsing all events (idaholab#35)
• Provide more fine-tuned controls for Zeek's node.cfg in Hedgehog sensor (idaholab#36, /pull/158)
• set zeek.uid to conn_uids for files.log entries (idaholab#33)
• Modify Zeek build chain to use default GCC compilers instead of LLVM/clang,which reduces build dependencies
• Use Firefox instead of Chromium for browser in ISO-installed versions of Malcolm and in Hedgehog Linux
• Updated copyright notices in text from "2020" to "2021" (which is the bulk of the changed files in this commit)
• Version bumps
  • Yara to 4.0.4

Malcolm v2.6.0

Malcolm v2.6.0 contains the following changes:

v2.5.0...v2.6.0

• Replace some of the Amazon ICS parsers for Zeek with parsers developed at the Idaho National Lab supporting DHS CISA

  • BACnet
  • Ethernet/IP

• Incorporated updates to some default Zeek ICS protocols

  • DNP3
  • Modbus

• Added new parsers for BSAP ICS protocol

  • BSAP IP
  • BSAP Serial

Component version bumps:

• Supercronic 0.1.12 (used in some Malcolm Docker images)
• alpine:3.12 (base layer of some Malcolm Docker images)
• nginx 1.19.6 (the web server handling encryption, authentication and proxying for Malcolm's Docker containers)
• CMake 3.19.3 (for building some Malcolm source code)
• netsniff-ng 0.6.8 (for packet capture)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v2.5.0

Malcolm v2.5.0 contains the following changes:

v2.4.2...v2.5.0

• Updated packaged Yara rules (from github.com/Neo23x0/signature-base, originally github.com/fireeye/sunburst_countermeasures) for Yara scanning of carved files to detect artifacts from the SolarWinds SUNBURST attack
• Version bumps:
  • Zeek 3.0.12
  • Bison, CMake and LLVM/Clang tools for building Zeek for Docker image and Hedgehog OS ISO

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v2.4.2

Malcolm v2.4.2 contains the following changes:

v2.4.1...v2.4.2

• Added code to allow periodic updates of Yara and Capa rules in addition to ClamAV rules for file scanners
• Bump to Arkime (Moloch up until recently) 2.7.1 and all possible related user-facing code/documentation changed
• Bump kernel to 5.9.0 for ISO installer
• minor bug fixes and documentation tweaks

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.