Merge pull request #295 from cisagov/v24.01.0_merge_cisagov

Malcolm v24.01.0 contains new features, improvements, bug fixes and component version updates.

v23.12.1...v24.0.1

* Features and enhancements
    + new Malcolm instance landing page (idaholab#252)
    + file carve download with password-protected .zip file (idaholab#288)
    + new "all files exept common plain text files" option for Malcolm's file carving to match Hedgehog capability (idaholab#290)
    + allow customizing indexes for logs written to OpenSearch/Elasticsearch (idaholab#313)
    + more consistently differentiate between uploaded and live-captured traffic (idaholab#321)
    + make download extracted file context item from Arkime smarter (idaholab#330)
    + improve netbox device type library import by using "official" import script (idaholab#384)
* Component version updates
    + Alpine Linux to [v3.19](https://alpinelinux.org/posts/Alpine-3.19.0-released.html) as the base for some Docker images
    + Fluent Bit to [v2.2.2](https://github.com/fluent/fluent-bit/releases/tag/v2.2.2)
    + Beats to [v8.11.4](https://www.elastic.co/guide/en/beats/libbeat/8.11/release-notes-8.11.4.html)
    + LogStash to [v8.11.4](https://www.elastic.co/guide/en/logstash/current/logstash-8-11-4.html)
* Bug fixes
    + Suricata Alerts dashboard "Alerts - Tags" visualization is useless (idaholab#314)
    + third party logs are not parsed correctly from fluentbit -> fluentd aggregator -> Malcolm (idaholab#318)
    + update document lookup APIs to search either network or host data (idaholab#322)
    + suricata rule update is broken (idaholab#323)
    + time sync from hedgehog to Malcolm opensearch instance not working (idaholab#324)
    + fix issue specifying database mode via command-line
    + have pruning of OpenSearch indices (based on size) include "other" Malcolm indices as well (e.g., nginx logs, system resources, third-party logs, etc.)
* Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/cisagov/Malcolm/tree/v24.0.1/config))
    + added the following variables with relation to idaholab#313
        - added `ARKIME_ROTATE_INDEX` to [`arkime.env`](https://github.com/cisagov/Malcolm/tree/v24.0.1/arkime.env.example) with default value of `daily` (see [Arkime docs on rotateIndex](https://arkime.com/settings#rotateIndex))
        - added the following variables and defaults to [`opensearch.env`](https://github.com/cisagov/Malcolm/tree/v24.0.1/opensearch.env.example):
        ```
        # OpenSearch index patterns and timestamp fields
        # Index pattern for network traffic logs written via Logstash (e.g., Zeek logs, Suricata alerts)
        MALCOLM_NETWORK_INDEX_PATTERN=arkime_sessions3-*
        # Default time field to use for network traffic logs in Logstash and Dashboards
        MALCOLM_NETWORK_INDEX_TIME_FIELD=firstPacket
        # Suffix used to create index to which network traffic logs are written (supports Ruby strftime strings in %{})
        MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}
        # Index pattern for other logs written via Logstash (e.g., nginx, beats, fluent-bit, etc.)
        MALCOLM_OTHER_INDEX_PATTERN=malcolm_beats_*
        # Default time field to use for other logs in Logstash and Dashboards
        MALCOLM_OTHER_INDEX_TIME_FIELD=@timestamp
        # Suffix used to create index to which other logs are written (supports Ruby strftime strings in %{})
        MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%d}
        # Index pattern used specifically by Arkime (will probably match MALCOLM_NETWORK_INDEX_PATTERN, should probably be arkime_sessions3-*)
        ARKIME_NETWORK_INDEX_PATTERN=arkime_sessions3-*
        # Default time field used by for sessions in Arkime viewer
        ARKIME_NETWORK_INDEX_TIME_FIELD=firstPacket
        ```
    + changed default for `EXTRACTED_FILE_HTTP_SERVER_KEY` to `infected` in [`zeek-secret.env`](https://github.com/cisagov/Malcolm/tree/v24.0.1/zeek-secret.env.example)
    + added `EXTRACTED_FILE_HTTP_SERVER_ZIP` with default value of `false` in [`zeek.env`](https://github.com/cisagov/Malcolm/tree/v24.0.1/zeek.env.example), see (idaholab#288)

.github/workflows/api-build-and-push-ghcr.yml

@@ -39,11 +39,26 @@ jobs:
       -
         name: Checkout
         uses: actions/checkout@v4
+      -
+        name: Generate build timestamp
+        shell: bash
+        run: echo "btimestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT
+        id: generate_build_timestamp
       -
         name: Extract branch name
         shell: bash
         run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
         id: extract_branch
+      -
+        name: Extract commit SHA
+        shell: bash
+        run: echo "sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+        id: extract_commit_sha
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        id: extract_malcolm_version
       -
         name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -66,6 +81,10 @@ jobs:
         with:
           context: .
           file: ./Dockerfiles/api.Dockerfile
+          build-args: |
+            MALCOLM_VERSION=${{ steps.extract_malcolm_version.outputs.mversion }}
+            BUILD_DATE=${{ steps.generate_build_timestamp.outputs.btimestamp }}
+            VCS_REVISION=${{ steps.extract_commit_sha.outputs.sha }}
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/api:${{ steps.extract_branch.outputs.branch }}
       -

.github/workflows/arkime-build-and-push-ghcr.yml

@@ -39,11 +39,26 @@ jobs:
       -
         name: Checkout
         uses: actions/checkout@v4
+      -
+        name: Generate build timestamp
+        shell: bash
+        run: echo "btimestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT
+        id: generate_build_timestamp
       -
         name: Extract branch name
         shell: bash
         run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
         id: extract_branch
+      -
+        name: Extract commit SHA
+        shell: bash
+        run: echo "sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+        id: extract_commit_sha
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        id: extract_malcolm_version
       -
         name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -67,6 +82,9 @@ jobs:
           context: .
           file: ./Dockerfiles/arkime.Dockerfile
           build-args: |
+            MALCOLM_VERSION=${{ steps.extract_malcolm_version.outputs.mversion }}
+            BUILD_DATE=${{ steps.generate_build_timestamp.outputs.btimestamp }}
+            VCS_REVISION=${{ steps.extract_commit_sha.outputs.sha }}
             MAXMIND_GEOIP_DB_LICENSE_KEY=${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/arkime:${{ steps.extract_branch.outputs.branch }}

.github/workflows/dashboards-build-and-push-ghcr.yml

@@ -39,11 +39,26 @@ jobs:
       -
         name: Checkout
         uses: actions/checkout@v4
+      -
+        name: Generate build timestamp
+        shell: bash
+        run: echo "btimestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT
+        id: generate_build_timestamp
       -
         name: Extract branch name
         shell: bash
         run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
         id: extract_branch
+      -
+        name: Extract commit SHA
+        shell: bash
+        run: echo "sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+        id: extract_commit_sha
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        id: extract_malcolm_version
       -
         name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -66,6 +81,10 @@ jobs:
         with:
           context: .
           file: ./Dockerfiles/dashboards.Dockerfile
+          build-args: |
+            MALCOLM_VERSION=${{ steps.extract_malcolm_version.outputs.mversion }}
+            BUILD_DATE=${{ steps.generate_build_timestamp.outputs.btimestamp }}
+            VCS_REVISION=${{ steps.extract_commit_sha.outputs.sha }}
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/dashboards:${{ steps.extract_branch.outputs.branch }}
       -

.github/workflows/dashboards-helper-build-and-push-ghcr.yml

@@ -39,11 +39,26 @@ jobs:
       -
         name: Checkout
         uses: actions/checkout@v4
+      -
+        name: Generate build timestamp
+        shell: bash
+        run: echo "btimestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT
+        id: generate_build_timestamp
       -
         name: Extract branch name
         shell: bash
         run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
         id: extract_branch
+      -
+        name: Extract commit SHA
+        shell: bash
+        run: echo "sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+        id: extract_commit_sha
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        id: extract_malcolm_version
       -
         name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -66,6 +81,10 @@ jobs:
         with:
           context: .
           file: ./Dockerfiles/dashboards-helper.Dockerfile
+          build-args: |
+            MALCOLM_VERSION=${{ steps.extract_malcolm_version.outputs.mversion }}
+            BUILD_DATE=${{ steps.generate_build_timestamp.outputs.btimestamp }}
+            VCS_REVISION=${{ steps.extract_commit_sha.outputs.sha }}
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/dashboards-helper:${{ steps.extract_branch.outputs.branch }}
       -

.github/workflows/dirinit-build-and-push-ghcr.yml

@@ -31,11 +31,26 @@ jobs:
       -
         name: Checkout
         uses: actions/checkout@v4
+      -
+        name: Generate build timestamp
+        shell: bash
+        run: echo "btimestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT
+        id: generate_build_timestamp
       -
         name: Extract branch name
         shell: bash
         run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
         id: extract_branch
+      -
+        name: Extract commit SHA
+        shell: bash
+        run: echo "sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+        id: extract_commit_sha
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        id: extract_malcolm_version
       -
         name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -58,6 +73,10 @@ jobs:
         with:
           context: .
           file: ./Dockerfiles/dirinit.Dockerfile
+          build-args: |
+            MALCOLM_VERSION=${{ steps.extract_malcolm_version.outputs.mversion }}
+            BUILD_DATE=${{ steps.generate_build_timestamp.outputs.btimestamp }}
+            VCS_REVISION=${{ steps.extract_commit_sha.outputs.sha }}
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/dirinit:${{ steps.extract_branch.outputs.branch }}
       -

.github/workflows/file-monitor-build-and-push-ghcr.yml

@@ -39,11 +39,26 @@ jobs:
       -
         name: Checkout
         uses: actions/checkout@v4
+      -
+        name: Generate build timestamp
+        shell: bash
+        run: echo "btimestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT
+        id: generate_build_timestamp
       -
         name: Extract branch name
         shell: bash
         run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
         id: extract_branch
+      -
+        name: Extract commit SHA
+        shell: bash
+        run: echo "sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+        id: extract_commit_sha
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        id: extract_malcolm_version
       -
         name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -66,6 +81,10 @@ jobs:
         with:
           context: .
           file: ./Dockerfiles/file-monitor.Dockerfile
+          build-args: |
+            MALCOLM_VERSION=${{ steps.extract_malcolm_version.outputs.mversion }}
+            BUILD_DATE=${{ steps.generate_build_timestamp.outputs.btimestamp }}
+            VCS_REVISION=${{ steps.extract_commit_sha.outputs.sha }}
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/file-monitor:${{ steps.extract_branch.outputs.branch }}
       -

.github/workflows/file-upload-build-and-push-ghcr.yml

@@ -39,11 +39,26 @@ jobs:
       -
         name: Checkout
         uses: actions/checkout@v4
+      -
+        name: Generate build timestamp
+        shell: bash
+        run: echo "btimestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT
+        id: generate_build_timestamp
       -
         name: Extract branch name
         shell: bash
         run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
         id: extract_branch
+      -
+        name: Extract commit SHA
+        shell: bash
+        run: echo "sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+        id: extract_commit_sha
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        id: extract_malcolm_version
       -
         name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -66,6 +81,10 @@ jobs:
         with:
           context: .
           file: ./Dockerfiles/file-upload.Dockerfile
+          build-args: |
+            MALCOLM_VERSION=${{ steps.extract_malcolm_version.outputs.mversion }}
+            BUILD_DATE=${{ steps.generate_build_timestamp.outputs.btimestamp }}
+            VCS_REVISION=${{ steps.extract_commit_sha.outputs.sha }}
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/file-upload:${{ steps.extract_branch.outputs.branch }}
       -

.github/workflows/filebeat-build-and-push-ghcr.yml

@@ -39,11 +39,26 @@ jobs:
       -
         name: Checkout
         uses: actions/checkout@v4
+      -
+        name: Generate build timestamp
+        shell: bash
+        run: echo "btimestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT
+        id: generate_build_timestamp
       -
         name: Extract branch name
         shell: bash
         run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
         id: extract_branch
+      -
+        name: Extract commit SHA
+        shell: bash
+        run: echo "sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+        id: extract_commit_sha
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        id: extract_malcolm_version
       -
         name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -66,6 +81,10 @@ jobs:
         with:
           context: .
           file: ./Dockerfiles/filebeat.Dockerfile
+          build-args: |
+            MALCOLM_VERSION=${{ steps.extract_malcolm_version.outputs.mversion }}
+            BUILD_DATE=${{ steps.generate_build_timestamp.outputs.btimestamp }}
+            VCS_REVISION=${{ steps.extract_commit_sha.outputs.sha }}
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/filebeat-oss:${{ steps.extract_branch.outputs.branch }}
       -

.github/workflows/freq-build-and-push-ghcr.yml

@@ -39,11 +39,26 @@ jobs:
       -
         name: Checkout
         uses: actions/checkout@v4
+      -
+        name: Generate build timestamp
+        shell: bash
+        run: echo "btimestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT
+        id: generate_build_timestamp
       -
         name: Extract branch name
         shell: bash
         run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
         id: extract_branch
+      -
+        name: Extract commit SHA
+        shell: bash
+        run: echo "sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+        id: extract_commit_sha
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        id: extract_malcolm_version
       -
         name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -66,6 +81,10 @@ jobs:
         with:
           context: .
           file: ./Dockerfiles/freq.Dockerfile
+          build-args: |
+            MALCOLM_VERSION=${{ steps.extract_malcolm_version.outputs.mversion }}
+            BUILD_DATE=${{ steps.generate_build_timestamp.outputs.btimestamp }}
+            VCS_REVISION=${{ steps.extract_commit_sha.outputs.sha }}
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/freq:${{ steps.extract_branch.outputs.branch }}
       -

.github/workflows/htadmin-build-and-push-ghcr.yml

@@ -39,11 +39,26 @@ jobs:
       -
         name: Checkout
         uses: actions/checkout@v4
+      -
+        name: Generate build timestamp
+        shell: bash
+        run: echo "btimestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT
+        id: generate_build_timestamp
       -
         name: Extract branch name
         shell: bash
         run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
         id: extract_branch
+      -
+        name: Extract commit SHA
+        shell: bash
+        run: echo "sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+        id: extract_commit_sha
+      -
+        name: Extract Malcolm version
+        shell: bash
+        run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
+        id: extract_malcolm_version
       -
         name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -66,6 +81,10 @@ jobs:
         with:
           context: .
           file: ./Dockerfiles/htadmin.Dockerfile
+          build-args: |
+            MALCOLM_VERSION=${{ steps.extract_malcolm_version.outputs.mversion }}
+            BUILD_DATE=${{ steps.generate_build_timestamp.outputs.btimestamp }}
+            VCS_REVISION=${{ steps.extract_commit_sha.outputs.sha }}
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/htadmin:${{ steps.extract_branch.outputs.branch }}
       -