Skip to content

Commit

Permalink
List of changes in Malcolm v3.0.0:
Browse files Browse the repository at this point in the history
- Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see idaholab#15. This is a major change which **breaks backwards compatibility** for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run `install.py --configure` (see [System configuration and tuning](https://github.com/cisagov/Malcolm#ConfigAndTuning)) prior to running Malcolm v3.0.0.
  - Kibana [comments](https://github.com/gwintzer/kibana-comments-app-plugin) replaced with [Notebooks](https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/)
  - Kibana [elastalert](https://github.com/nsano-rururu/elastalert-kibana-plugin) plugin replaced with [Alerting](https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/) plugin
  - Elasticsearch [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) replaced with [Index Management](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) plugin
  - The third-party [Sankey visualization plugin](https://github.com/mmguero-dev/kbn_sankey_vis] has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15)
  - The third-party [Kibana drill-down plugin](https://github.com/mmguero-dev/kibana-plugin-drilldownmenu/) providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5)
  - In addition to those replacements, the Real Time Anomaly Detection feature is now available:
    - [Real Time Anomaly Detection in Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/real-time-anomaly-detection-in-open-distro-for-elasticsearch/) blog announcement
     - [Anomaly Detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/) documentation and source code for [Elasticsearch](https://github.com/opendistro-for-elasticsearch/anomaly-detection) and [Kibana](https://github.com/opendistro-for-elasticsearch/anomaly-detection-kibana-plugin) components
     - [Random Cut Forests writeup](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/random-cut-forests/)

- Malcolm startup time (especially the Logstash container) has been reduced drastically

- Improvements to Malcolm's prebuilt Kibana dashboards

- Improvements to build scripts

- Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux

- Minor other bug fixes and performance improvements

- Version bump
  - Yara [v4.0.5](https://github.com/VirusTotal/yara/releases/tag/v4.0.5)
  • Loading branch information
mmguero committed Feb 16, 2021
1 parent 83f0627 commit eddc803
Show file tree
Hide file tree
Showing 134 changed files with 6,269 additions and 6,305 deletions.
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
/nginx/nginx_ldap.conf
/htadmin/config.ini
/htadmin/metadata
/elastalert/config/smtp-auth.yaml

# development
.vagrant
Expand Down
4 changes: 1 addition & 3 deletions Dockerfiles/arkime.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -214,10 +214,8 @@ RUN groupadd --gid $DEFAULT_GID $PGROUP && \
ln -sfr /data/pcap_moloch_and_zeek_processor.py /data/pcap_moloch_processor.py && \
cp -f /data/moloch_update_geo.sh $ARKIMEDIR/bin/moloch_update_geo.sh && \
chmod u+s $ARKIMEDIR/bin/moloch-capture && \
chown -R $PUSER:$PGROUP $ARKIMEDIR/logs && \
mkdir -p /var/run/moloch && \
chown -R $PUSER:$PGROUP /var/run/moloch

chown -R $PUSER:$PGROUP $ARKIMEDIR/etc $ARKIMEDIR/logs /var/run/moloch
#Update Path
ENV PATH="/data:$ARKIMEDIR/bin:${PATH}"

Expand Down
115 changes: 0 additions & 115 deletions Dockerfiles/curator.Dockerfile

This file was deleted.

50 changes: 0 additions & 50 deletions Dockerfiles/elastalert.Dockerfile

This file was deleted.

24 changes: 18 additions & 6 deletions Dockerfiles/elasticsearch.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2
FROM amazon/opendistro-for-elasticsearch:1.12.0

# Copyright (c) 2021 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
Expand All @@ -7,8 +7,8 @@ LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'
LABEL org.opencontainers.image.vendor='Cybersecurity and Infrastructure Security Agency'
LABEL org.opencontainers.image.title='malcolmnetsec/elasticsearch-oss'
LABEL org.opencontainers.image.description='Malcolm container providing Elasticsearch (the Apache-licensed variant)'
LABEL org.opencontainers.image.title='malcolmnetsec/elasticsearch-od'
LABEL org.opencontainers.image.description='Malcolm container providing Elasticsearch (the Apache-licensed Open Distro variant)'

ARG DEFAULT_UID=1000
ARG DEFAULT_GID=1000
Expand All @@ -20,11 +20,23 @@ ENV PUSER_PRIV_DROP true

ENV TERM xterm

ARG GITHUB_OAUTH_TOKEN=""
ARG DISABLE_INSTALL_DEMO_CONFIG=true
ENV DISABLE_INSTALL_DEMO_CONFIG $DISABLE_INSTALL_DEMO_CONFIG

# Malcolm manages authentication and encryption via NGINX reverse proxy
# https://opendistro.github.io/for-elasticsearch-docs/docs/security/configuration/disable/
# https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/#customize-the-docker-image
# https://github.com/opendistro-for-elasticsearch/opendistro-build/issues/613
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro_security && \
/usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro_performance_analyzer && \
echo -e 'cluster.name: "docker-cluster"\nnetwork.host: 0.0.0.0' > /usr/share/elasticsearch/config/elasticsearch.yml && \
chown -R $PUSER:$PGROUP /usr/share/elasticsearch/config/elasticsearch.yml && \
sed -i "s/\b1000\b/\${PUID:-${DEFAULT_UID}}/g" /usr/local/bin/docker-entrypoint.sh

# just used for initial keystore creation
ADD shared/bin/docker-uid-gid-setup.sh /usr/local/bin/

ENTRYPOINT ["/usr/local/bin/docker-uid-gid-setup.sh", "/usr/local/bin/docker-entrypoint.sh"]


# to be populated at build-time:
ARG BUILD_DATE
ARG MALCOLM_VERSION
Expand Down
3 changes: 2 additions & 1 deletion Dockerfiles/file-monitor.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ ENV EXTRACTED_FILE_ENABLE_CAPA $EXTRACTED_FILE_ENABLE_CAPA
ENV EXTRACTED_FILE_CAPA_VERBOSE $EXTRACTED_FILE_CAPA_VERBOSE
ENV SRC_BASE_DIR "/usr/local/src"
ENV CLAMAV_RULES_DIR "/var/lib/clamav"
ENV YARA_VERSION "4.0.4"
ENV YARA_VERSION "4.0.5"
ENV YARA_URL "https://github.com/VirusTotal/yara/archive/v${YARA_VERSION}.tar.gz"
ENV YARA_RULES_URL "https://github.com/Neo23x0/signature-base"
ENV YARA_RULES_DIR "/yara-rules"
Expand Down Expand Up @@ -228,6 +228,7 @@ VOLUME ["$YARA_RULES_DIR"]
VOLUME ["$YARA_RULES_SRC_DIR"]

EXPOSE 3310
EXPOSE $EXTRACTED_FILE_HTTP_SERVER_PORT

ENTRYPOINT ["/usr/local/bin/docker-uid-gid-setup.sh", "/docker-entrypoint.sh"]

Expand Down
2 changes: 1 addition & 1 deletion Dockerfiles/filebeat.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM docker.elastic.co/beats/filebeat-oss:7.6.2
FROM docker.elastic.co/beats/filebeat-oss:7.10.0

# Copyright (c) 2021 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
Expand Down
92 changes: 92 additions & 0 deletions Dockerfiles/kibana-helper.Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
FROM alpine:3.12

# Copyright (c) 2020 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
LABEL org.opencontainers.image.authors='[email protected]'
LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'
LABEL org.opencontainers.image.vendor='Cybersecurity and Infrastructure Security Agency'
LABEL org.opencontainers.image.title='malcolmnetsec/kibana-helper'
LABEL org.opencontainers.image.description='Malcolm container providing Kibana support functions'

ARG DEFAULT_UID=1000
ARG DEFAULT_GID=1000
ENV DEFAULT_UID $DEFAULT_UID
ENV DEFAULT_GID $DEFAULT_GID
ENV PUSER "helper"
ENV PGROUP "helper"
ENV PUSER_PRIV_DROP true

ENV TERM xterm

ARG ARKIME_INDEX_PATTERN="sessions2-*"
ARG ARKIME_INDEX_PATTERN_ID="sessions2-*"
ARG ARKIME_INDEX_TIME_FIELD="firstPacket"
ARG CREATE_ES_ARKIME_SESSION_INDEX="true"
ARG ELASTICSEARCH_URL="http://elasticsearch:9200"
ARG ISM_SNAPSHOT_COMPRESSED=false
ARG ISM_SNAPSHOT_REPO=logs
ARG KIBANA_OFFLINE_REGION_MAPS_PORT="28991"
ARG KIBANA_URL="http://kibana:5601/kibana"

ENV ARKIME_INDEX_PATTERN $ARKIME_INDEX_PATTERN
ENV ARKIME_INDEX_PATTERN_ID $ARKIME_INDEX_PATTERN_ID
ENV ARKIME_INDEX_TIME_FIELD $ARKIME_INDEX_TIME_FIELD
ENV CREATE_ES_ARKIME_SESSION_INDEX $CREATE_ES_ARKIME_SESSION_INDEX
ENV ELASTICSEARCH_URL $ELASTICSEARCH_URL
ENV ISM_SNAPSHOT_COMPRESSED $ISM_SNAPSHOT_COMPRESSED
ENV ISM_SNAPSHOT_REPO $ISM_SNAPSHOT_REPO
ENV KIBANA_OFFLINE_REGION_MAPS_PORT $KIBANA_OFFLINE_REGION_MAPS_PORT
ENV KIBANA_URL $KIBANA_URL
ENV PATH="/data:${PATH}"

ENV SUPERCRONIC_VERSION "0.1.12"
ENV SUPERCRONIC_URL "https://github.com/aptible/supercronic/releases/download/v$SUPERCRONIC_VERSION/supercronic-linux-amd64"
ENV SUPERCRONIC "supercronic-linux-amd64"
ENV SUPERCRONIC_SHA1SUM "048b95b48b708983effb2e5c935a1ef8483d9e3e"
ENV SUPERCRONIC_CRONTAB "/etc/crontab"

ADD kibana/dashboards /opt/kibana/dashboards
ADD kibana/maps /opt/maps
ADD kibana/scripts /data/
ADD kibana/supervisord.conf /etc/supervisord.conf
ADD kibana/zeek_template.json /data/zeek_template.json
ADD shared/bin/docker-uid-gid-setup.sh /usr/local/bin/
ADD shared/bin/elastic_search_status.sh /data/
ADD shared/bin/elastic_index_size_prune.py /data/

RUN apk --no-cache add bash python3 py3-pip curl procps psmisc npm shadow jq && \
npm install -g http-server && \
pip3 install supervisor humanfriendly && \
curl -fsSLO "$SUPERCRONIC_URL" && \
echo "${SUPERCRONIC_SHA1SUM} ${SUPERCRONIC}" | sha1sum -c - && \
chmod +x "$SUPERCRONIC" && \
mv "$SUPERCRONIC" "/usr/local/bin/${SUPERCRONIC}" && \
ln -s "/usr/local/bin/${SUPERCRONIC}" /usr/local/bin/supercronic && \
addgroup -g ${DEFAULT_GID} ${PGROUP} ; \
adduser -D -H -u ${DEFAULT_UID} -h /nonexistant -s /sbin/nologin -G ${PGROUP} -g ${PUSER} ${PUSER} ; \
addgroup ${PUSER} tty ; \
addgroup ${PUSER} shadow ; \
mkdir -p /data/init && \
chown -R ${PUSER}:${PGROUP} /opt/kibana/dashboards /opt/maps /data/init && \
chmod 755 /data/*.sh /data/*.py /data/init && \
chmod 400 /opt/maps/* && \
(echo -e "*/2 * * * * /data/kibana-create-moloch-sessions-index.sh\n0 10 * * * /data/kibana_index_refresh.py --template zeek_template\n*/20 * * * * /data/elastic_index_size_prune.py" > ${SUPERCRONIC_CRONTAB})

EXPOSE $KIBANA_OFFLINE_REGION_MAPS_PORT

ENTRYPOINT ["/usr/local/bin/docker-uid-gid-setup.sh"]

CMD ["/usr/bin/supervisord", "-c", "/etc/supervisord.conf", "-n"]

VOLUME ["/data/init"]

# to be populated at build-time:
ARG BUILD_DATE
ARG MALCOLM_VERSION
ARG VCS_REVISION

LABEL org.opencontainers.image.created=$BUILD_DATE
LABEL org.opencontainers.image.version=$MALCOLM_VERSION
LABEL org.opencontainers.image.revision=$VCS_REVISION
Loading

0 comments on commit eddc803

Please sign in to comment.