-
Notifications
You must be signed in to change notification settings - Fork 332
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see idaholab#15. This is a major change which **breaks backwards compatibility** for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run `install.py --configure` (see [System configuration and tuning](https://github.com/cisagov/Malcolm#ConfigAndTuning)) prior to running Malcolm v3.0.0. - Kibana [comments](https://github.com/gwintzer/kibana-comments-app-plugin) replaced with [Notebooks](https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/) - Kibana [elastalert](https://github.com/nsano-rururu/elastalert-kibana-plugin) plugin replaced with [Alerting](https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/) plugin - Elasticsearch [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) replaced with [Index Management](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) plugin - The third-party [Sankey visualization plugin](https://github.com/mmguero-dev/kbn_sankey_vis] has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15) - The third-party [Kibana drill-down plugin](https://github.com/mmguero-dev/kibana-plugin-drilldownmenu/) providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5) - In addition to those replacements, the Real Time Anomaly Detection feature is now available: - [Real Time Anomaly Detection in Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/real-time-anomaly-detection-in-open-distro-for-elasticsearch/) blog announcement - [Anomaly Detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/) documentation and source code for [Elasticsearch](https://github.com/opendistro-for-elasticsearch/anomaly-detection) and [Kibana](https://github.com/opendistro-for-elasticsearch/anomaly-detection-kibana-plugin) components - [Random Cut Forests writeup](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/random-cut-forests/) - Malcolm startup time (especially the Logstash container) has been reduced drastically - Improvements to Malcolm's prebuilt Kibana dashboards - Improvements to build scripts - Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux - Minor other bug fixes and performance improvements - Version bump - Yara [v4.0.5](https://github.com/VirusTotal/yara/releases/tag/v4.0.5)
- Loading branch information
Showing
134 changed files
with
6,269 additions
and
6,305 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
FROM docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2 | ||
FROM amazon/opendistro-for-elasticsearch:1.12.0 | ||
|
||
# Copyright (c) 2021 Battelle Energy Alliance, LLC. All rights reserved. | ||
LABEL maintainer="[email protected]" | ||
|
@@ -7,8 +7,8 @@ LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm' | |
LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md' | ||
LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm' | ||
LABEL org.opencontainers.image.vendor='Cybersecurity and Infrastructure Security Agency' | ||
LABEL org.opencontainers.image.title='malcolmnetsec/elasticsearch-oss' | ||
LABEL org.opencontainers.image.description='Malcolm container providing Elasticsearch (the Apache-licensed variant)' | ||
LABEL org.opencontainers.image.title='malcolmnetsec/elasticsearch-od' | ||
LABEL org.opencontainers.image.description='Malcolm container providing Elasticsearch (the Apache-licensed Open Distro variant)' | ||
|
||
ARG DEFAULT_UID=1000 | ||
ARG DEFAULT_GID=1000 | ||
|
@@ -20,11 +20,23 @@ ENV PUSER_PRIV_DROP true | |
|
||
ENV TERM xterm | ||
|
||
ARG GITHUB_OAUTH_TOKEN="" | ||
ARG DISABLE_INSTALL_DEMO_CONFIG=true | ||
ENV DISABLE_INSTALL_DEMO_CONFIG $DISABLE_INSTALL_DEMO_CONFIG | ||
|
||
# Malcolm manages authentication and encryption via NGINX reverse proxy | ||
# https://opendistro.github.io/for-elasticsearch-docs/docs/security/configuration/disable/ | ||
# https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/#customize-the-docker-image | ||
# https://github.com/opendistro-for-elasticsearch/opendistro-build/issues/613 | ||
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro_security && \ | ||
/usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro_performance_analyzer && \ | ||
echo -e 'cluster.name: "docker-cluster"\nnetwork.host: 0.0.0.0' > /usr/share/elasticsearch/config/elasticsearch.yml && \ | ||
chown -R $PUSER:$PGROUP /usr/share/elasticsearch/config/elasticsearch.yml && \ | ||
sed -i "s/\b1000\b/\${PUID:-${DEFAULT_UID}}/g" /usr/local/bin/docker-entrypoint.sh | ||
|
||
# just used for initial keystore creation | ||
ADD shared/bin/docker-uid-gid-setup.sh /usr/local/bin/ | ||
|
||
ENTRYPOINT ["/usr/local/bin/docker-uid-gid-setup.sh", "/usr/local/bin/docker-entrypoint.sh"] | ||
|
||
|
||
# to be populated at build-time: | ||
ARG BUILD_DATE | ||
ARG MALCOLM_VERSION | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
FROM docker.elastic.co/beats/filebeat-oss:7.6.2 | ||
FROM docker.elastic.co/beats/filebeat-oss:7.10.0 | ||
|
||
# Copyright (c) 2021 Battelle Energy Alliance, LLC. All rights reserved. | ||
LABEL maintainer="[email protected]" | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,92 @@ | ||
FROM alpine:3.12 | ||
|
||
# Copyright (c) 2020 Battelle Energy Alliance, LLC. All rights reserved. | ||
LABEL maintainer="[email protected]" | ||
LABEL org.opencontainers.image.authors='[email protected]' | ||
LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm' | ||
LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md' | ||
LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm' | ||
LABEL org.opencontainers.image.vendor='Cybersecurity and Infrastructure Security Agency' | ||
LABEL org.opencontainers.image.title='malcolmnetsec/kibana-helper' | ||
LABEL org.opencontainers.image.description='Malcolm container providing Kibana support functions' | ||
|
||
ARG DEFAULT_UID=1000 | ||
ARG DEFAULT_GID=1000 | ||
ENV DEFAULT_UID $DEFAULT_UID | ||
ENV DEFAULT_GID $DEFAULT_GID | ||
ENV PUSER "helper" | ||
ENV PGROUP "helper" | ||
ENV PUSER_PRIV_DROP true | ||
|
||
ENV TERM xterm | ||
|
||
ARG ARKIME_INDEX_PATTERN="sessions2-*" | ||
ARG ARKIME_INDEX_PATTERN_ID="sessions2-*" | ||
ARG ARKIME_INDEX_TIME_FIELD="firstPacket" | ||
ARG CREATE_ES_ARKIME_SESSION_INDEX="true" | ||
ARG ELASTICSEARCH_URL="http://elasticsearch:9200" | ||
ARG ISM_SNAPSHOT_COMPRESSED=false | ||
ARG ISM_SNAPSHOT_REPO=logs | ||
ARG KIBANA_OFFLINE_REGION_MAPS_PORT="28991" | ||
ARG KIBANA_URL="http://kibana:5601/kibana" | ||
|
||
ENV ARKIME_INDEX_PATTERN $ARKIME_INDEX_PATTERN | ||
ENV ARKIME_INDEX_PATTERN_ID $ARKIME_INDEX_PATTERN_ID | ||
ENV ARKIME_INDEX_TIME_FIELD $ARKIME_INDEX_TIME_FIELD | ||
ENV CREATE_ES_ARKIME_SESSION_INDEX $CREATE_ES_ARKIME_SESSION_INDEX | ||
ENV ELASTICSEARCH_URL $ELASTICSEARCH_URL | ||
ENV ISM_SNAPSHOT_COMPRESSED $ISM_SNAPSHOT_COMPRESSED | ||
ENV ISM_SNAPSHOT_REPO $ISM_SNAPSHOT_REPO | ||
ENV KIBANA_OFFLINE_REGION_MAPS_PORT $KIBANA_OFFLINE_REGION_MAPS_PORT | ||
ENV KIBANA_URL $KIBANA_URL | ||
ENV PATH="/data:${PATH}" | ||
|
||
ENV SUPERCRONIC_VERSION "0.1.12" | ||
ENV SUPERCRONIC_URL "https://github.com/aptible/supercronic/releases/download/v$SUPERCRONIC_VERSION/supercronic-linux-amd64" | ||
ENV SUPERCRONIC "supercronic-linux-amd64" | ||
ENV SUPERCRONIC_SHA1SUM "048b95b48b708983effb2e5c935a1ef8483d9e3e" | ||
ENV SUPERCRONIC_CRONTAB "/etc/crontab" | ||
|
||
ADD kibana/dashboards /opt/kibana/dashboards | ||
ADD kibana/maps /opt/maps | ||
ADD kibana/scripts /data/ | ||
ADD kibana/supervisord.conf /etc/supervisord.conf | ||
ADD kibana/zeek_template.json /data/zeek_template.json | ||
ADD shared/bin/docker-uid-gid-setup.sh /usr/local/bin/ | ||
ADD shared/bin/elastic_search_status.sh /data/ | ||
ADD shared/bin/elastic_index_size_prune.py /data/ | ||
|
||
RUN apk --no-cache add bash python3 py3-pip curl procps psmisc npm shadow jq && \ | ||
npm install -g http-server && \ | ||
pip3 install supervisor humanfriendly && \ | ||
curl -fsSLO "$SUPERCRONIC_URL" && \ | ||
echo "${SUPERCRONIC_SHA1SUM} ${SUPERCRONIC}" | sha1sum -c - && \ | ||
chmod +x "$SUPERCRONIC" && \ | ||
mv "$SUPERCRONIC" "/usr/local/bin/${SUPERCRONIC}" && \ | ||
ln -s "/usr/local/bin/${SUPERCRONIC}" /usr/local/bin/supercronic && \ | ||
addgroup -g ${DEFAULT_GID} ${PGROUP} ; \ | ||
adduser -D -H -u ${DEFAULT_UID} -h /nonexistant -s /sbin/nologin -G ${PGROUP} -g ${PUSER} ${PUSER} ; \ | ||
addgroup ${PUSER} tty ; \ | ||
addgroup ${PUSER} shadow ; \ | ||
mkdir -p /data/init && \ | ||
chown -R ${PUSER}:${PGROUP} /opt/kibana/dashboards /opt/maps /data/init && \ | ||
chmod 755 /data/*.sh /data/*.py /data/init && \ | ||
chmod 400 /opt/maps/* && \ | ||
(echo -e "*/2 * * * * /data/kibana-create-moloch-sessions-index.sh\n0 10 * * * /data/kibana_index_refresh.py --template zeek_template\n*/20 * * * * /data/elastic_index_size_prune.py" > ${SUPERCRONIC_CRONTAB}) | ||
|
||
EXPOSE $KIBANA_OFFLINE_REGION_MAPS_PORT | ||
|
||
ENTRYPOINT ["/usr/local/bin/docker-uid-gid-setup.sh"] | ||
|
||
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisord.conf", "-n"] | ||
|
||
VOLUME ["/data/init"] | ||
|
||
# to be populated at build-time: | ||
ARG BUILD_DATE | ||
ARG MALCOLM_VERSION | ||
ARG VCS_REVISION | ||
|
||
LABEL org.opencontainers.image.created=$BUILD_DATE | ||
LABEL org.opencontainers.image.version=$MALCOLM_VERSION | ||
LABEL org.opencontainers.image.revision=$VCS_REVISION |
Oops, something went wrong.