-
Notifications
You must be signed in to change notification settings - Fork 0
/
Dockerfile-atune-engine
208 lines (194 loc) · 12.5 KB
/
Dockerfile-atune-engine
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
# Copyright (c) 2021 Huawei Technologies Co., Ltd.
# A-Tune is licensed under the Mulan PSL v2.
# You can use this software according to the terms and conditions of the Mulan PSL v2.
# You may obtain a copy of Mulan PSL v2 at:
# http://license.coscl.org.cn/MulanPSL2
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
# PURPOSE.
# See the Mulan PSL v2 for more details.
# Create: 2021-01-14
#
# Dockerfile for building atune-engine docker image.
#
# Usage:
# docker build -f Dockerfile-atune-engine -t atune-engine .
# docker run -p [<HOST_IP>:]<HOST_PORT>:3838 -e ENGINE_TLS=false atune-engine
#
# Supported running arguments:
# -e PORT=<CONTAINER_PORT> The atune-engine running port inside container, should be consistent with the second port of -p, default 3838
#
# -e ENGINE_TLS=<true/false> Whether using TLS to secure atuned client connection, default true
# -e ENGINE_TLS_CA_CERT_FILE=<path> TLS ca.crt file path, default /etc/atuned/engine_certs/ca.crt
# -e ENGINE_TLS_SERVER_CERT_FILE=<path> TLS server.crt file path, default /etc/atuned/engine_certs/server.crt
# -e ENGINE_TLS_SERVER_KEY_FILE=<path> TLS server.key file path, default /etc/atuned/engine_certs/server.key
# -e ENGINE_IP_ADDR=<ip addr> Binded IP/domain of newly generated TLS certificates, default localhost
#
# -e DB_ENABLE=<true/false> Whether using database to store atune-engine data, default false
# -e DB_HOST=<ip address> Database host ip address, default localhost
# -e DB_PORT=<port number> Database running port, default 5432
# -e DB_NAME=<database name> Database name used to store atune-engine tables, default atune_db
# -e DB_USER_NAME=<username> Database login username, default admin
# -e DB_USER_PASSWD=<password> Database login password, default no password, add if necessary
#
# Running example:
# 1. In atune-engine server host, build atune-engine docker image:
# docker build -f Dockerfile-atune-engine -t atune-engine:latest .
#
# 2. Run atune-engine service container at container port 3838, which is mapped to host port 3737:
# docker run -p 3737:3838 -e ENGINE_TLS=false atune-engine
#
# The output will be like this if the atune-engine container is successfully started:
# Starting enhanced syslogd: rsyslogd.
# * Serving Flask app "analysis.app" (lazy loading)
# * Environment: production
# WARNING: This is a development server. Do not use it in a production deployment.
# Use a production WSGI server instead.
# * Debug mode: off
#
# 3. In atuned client host, install A-tune service, and update atuned configuration(/etc/atuned/atuned.cnf):
# engine_host = <atune-engine host ip>
# engine_port = 3737
#
# 4. In atuned client host, start atuned service and run default atune analysis:
# systemctl daemon-reload
# systemctl start atuned
# atune-adm analysis
#
# 5. In atune-engine host, check analysis log in atune-engine container:
# docker ps
# docker exec -it <atune-engine container ID> /bin/bash
# cat /var/log/messages
#
# You will get log record like following if atune-engine successfully finishes the analysis:
# ...
# Jan 15 07:04:33 f1e85f6c7a4d kernel: [347948.341795] docker0: port 2(veth8472648) entered disabled state
# Jan 15 07:04:33 f1e85f6c7a4d kernel: [347948.351181] device veth8472648 left promiscuous mode
# Jan 15 07:04:33 f1e85f6c7a4d kernel: [347948.354295] docker0: port 2(veth8472648) entered disabled state
# Jan 15 09:33:49 f1e85f6c7a4d atuned: 2021-01-15 09:33:49,965 [INFO] transfer [/home/A-Tune/analysis/../analysis/engine/transfer.py:37] : ImmutableMultiDict([('file', <FileStorage: 'test-1610703229683.csv' ('application/octet-stream')>)])
# Jan 15 09:33:49 f1e85f6c7a4d atuned: 2021-01-15 09:33:49,967 [INFO] transfer [/home/A-Tune/analysis/../analysis/engine/transfer.py:47] : /var/atune_data/analysis//test-1610703229683.csv
# Jan 15 09:33:49 f1e85f6c7a4d atuned: 2021-01-15 09:33:49,967 [INFO] transfer [/home/A-Tune/analysis/../analysis/engine/transfer.py:48] : 9.85.179.193
# Jan 15 09:33:49 f1e85f6c7a4d atuned: 2021-01-15 09:33:49,973 [INFO] classification [/home/A-Tune/analysis/../analysis/engine/classification.py:42] : {'modelpath': '/usr/libexec/atuned/analysis/models', 'data': '/var/atune_data/analysis//test-1610703229683.csv', 'model': None}
# Jan 15 09:33:50 f1e85f6c7a4d atuned: 2021-01-15 09:33:50,779 [INFO] transfer [/home/A-Tune/analysis/../analysis/engine/transfer.py:37] : ImmutableMultiDict([('file', <FileStorage: 'test-1610442581644.log' ('application/octet-stream')>)])
# Jan 15 09:33:50 f1e85f6c7a4d atuned: 2021-01-15 09:33:50,780 [INFO] transfer [/home/A-Tune/analysis/../analysis/engine/transfer.py:47] : /var/atune_data/analysis//test-1610442581644.log
# Jan 15 09:33:50 f1e85f6c7a4d atuned: 2021-01-15 09:33:50,780 [INFO] transfer [/home/A-Tune/analysis/../analysis/engine/transfer.py:48] : 9.85.179.193
#
# Running example for using database:
# docker run -p 3737:3838 -e ENGINE_TLS=false -e DB_ENABLE=true -e DB_HOST=<database ip> -e DB_PORT=5432 -e DB_NAME=atune_db \
# -e DB_USER_NAME=admin -e DB_USER_PASSWD=Ha123456# atune-engine
#
# Running example for using TLS:
# docker run -p <host ip>:3838:3838 -v /path/to/existing/tls/certs:/etc/atuned/engine_certs atune-engine
# here /path/to/existing/tls/certs is the atune-engine host directory where ca.crt, server.crt, server.key files are provided.
# these certificate files should be consisted with those in atuned client machine.
#
# docker run -p <host ip>:3838:3838 -e ENGINE_IP_ADDR=<host ip> -v /path/to/generate/new/tls/certs:/etc/atuned/engine_certs atune-engine
# here /path/to/generate/new/tls/certs is an empty atune-engine host directory where ca.crt, server.crt, server.key files will be generated.
# later we need to copy ca.crt, client.crt, client.key in the directory to atuned client machine and update atuned configuration.
#
# base image
FROM python:3.8-slim
# update Debian source
RUN echo 'deb https://mirrors.tuna.tsinghua.edu.cn/debian/ buster main' > /etc/apt/sources.list && \
echo 'deb https://mirrors.tuna.tsinghua.edu.cn/debian/ buster-updates main' >> /etc/apt/sources.list && \
echo 'deb https://mirrors.tuna.tsinghua.edu.cn/debian-security buster/updates main' >> /etc/apt/sources.list
# update pip source
RUN pip3 config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
# install python3 library dependencies and rsyslog service
RUN apt update && \
apt install -y --no-install-recommends rsyslog gcc g++ && \
pip3 install --no-cache-dir Flask-RESTful scipy==1.5.4 pandas scikit-learn==0.23.2 scikit-optimize xgboost pyyaml && \
apt purge -y gcc g++ && \
apt clean
# in this RUN, we pull latest code from https://gitee.com/openeuler/A-Tune, and generate models
# then keep necessary files, remove useless files and cache to reduce image size
RUN cd ~ && \
apt install -y --no-install-recommends git && \
git -c http.sslVerify=false clone https://gitee.com/openeuler/A-Tune.git --depth=1 && \
cd ~/A-Tune/tools && \
python3 generate_models.py && \
mkdir -p /usr/libexec/atuned/analysis && \
mv ~/A-Tune/analysis/models /usr/libexec/atuned/analysis/ && \
chmod -R 750 /usr/libexec/atuned/analysis/* && \
cd ~/A-Tune && \
rm -rf analysis/__pycache__ analysis/engine/__pycache__ analysis/engine/utils/__pycache__ analysis/optimizer/__pycache__/&& \
rm -rf analysis/app_rest.py analysis/atuned/ analysis/dataset/ && \
mkdir -p /etc/atuned && \
cp ~/A-Tune/misc/engine.cnf /etc/atuned/ && \
mkdir -p ~/atune-engine && \
cp -r ~/A-Tune/analysis ~/atune-engine && \
cp -r ~/A-Tune/License ~/atune-engine && \
mkdir -p ~/atune-engine/tools && \
cp ~/A-Tune/tools/encrypt.py ~/atune-engine/tools/ && \
cd ~ && \
rm -rf ~/A-Tune && \
apt purge -y git && \
apt clean
# default ENV variables in /etc/atuned/engine.cnf
ENV PORT=3838
ENV ENGINE_TLS=true
ENV ENGINE_TLS_CA_CERT_FILE=/etc/atuned/engine_certs/ca.crt
ENV ENGINE_TLS_SERVER_CERT_FILE=/etc/atuned/engine_certs/server.crt
ENV ENGINE_TLS_SERVER_KEY_FILE=/etc/atuned/engine_certs/server.key
ENV ENGINE_IP_ADDR=localhost
ENV DB_ENABLE=false
ENV DB_HOST=localhost
ENV DB_PORT=5432
ENV DB_NAME=atune_db
ENV DB_USER_NAME=admin
# atuned client should communicate with atune-engine through <host ip>:3838
EXPOSE ${PORT}/tcp
# remove imklog module in rsyslogd service to avoid root permission need
RUN sed -i '/imklog/s/^/#/' /etc/rsyslog.conf
# update engine.cnf, add tls certificates and database support if needed, start rsyslog service and atune-engine
CMD sed -i "/^engine_host = /c\engine_host = 0.0.0.0" /etc/atuned/engine.cnf && \
sed -i "/^engine_port = /c\engine_port = $PORT" /etc/atuned/engine.cnf && \
sed -i "/^engine_tls = /c\engine_tls = $ENGINE_TLS" /etc/atuned/engine.cnf && \
sed -i "/^tlsenginecacertfile = /c\tlsenginecacertfile = $ENGINE_TLS_CA_CERT_FILE" /etc/atuned/engine.cnf && \
sed -i "/^tlsengineservercertfile = /c\tlsengineservercertfile = $ENGINE_TLS_SERVER_CERT_FILE" /etc/atuned/engine.cnf && \
sed -i "/^tlsengineserverkeyfile = /c\tlsengineserverkeyfile = $ENGINE_TLS_SERVER_KEY_FILE" /etc/atuned/engine.cnf && \
(if [ $ENGINE_TLS = true ]; then \
ENGINE_CERT_PATH=`dirname $ENGINE_TLS_CA_CERT_FILE`; \
(if [ ! -f $ENGINE_TLS_CA_CERT_FILE ]; then \
mkdir -p $ENGINE_CERT_PATH; \
openssl genrsa -out $ENGINE_CERT_PATH/ca.key 2048; \
openssl req -new -x509 -days 3650 -subj "/CN=ca" -key $ENGINE_CERT_PATH/ca.key -out $ENGINE_CERT_PATH/ca.crt; \
fi); \
(if [ ! -f $ENGINE_TLS_SERVER_CERT_FILE ] || [ ! -f $ENGINE_TLS_SERVER_KEY_FILE ]; then \
(for name in server client; do \
openssl genrsa -out $ENGINE_CERT_PATH/$name.key 2048; \
(if [ $ENGINE_IP_ADDR = localhost ]; then \
openssl req -new -subj "/CN=localhost" -key $ENGINE_CERT_PATH/$name.key -out $ENGINE_CERT_PATH/$name.csr; \
openssl x509 -req -sha256 -CA $ENGINE_CERT_PATH/ca.crt -CAkey $ENGINE_CERT_PATH/ca.key -CAcreateserial -days 3650 \
-in $ENGINE_CERT_PATH/$name.csr -out $ENGINE_CERT_PATH/$name.crt; \
else \
openssl req -new -subj "/CN=${ENGINE_IP_ADDR}" -key $ENGINE_CERT_PATH/$name.key -out $ENGINE_CERT_PATH/$name.csr; \
echo "subjectAltName=IP:${ENGINE_IP_ADDR}" > $ENGINE_CERT_PATH/extfile.cnf; \
openssl x509 -req -sha256 -CA $ENGINE_CERT_PATH/ca.crt -CAkey $ENGINE_CERT_PATH/ca.key -CAcreateserial -days 3650 \
-extfile $ENGINE_CERT_PATH/extfile.cnf -in $ENGINE_CERT_PATH/$name.csr -out $ENGINE_CERT_PATH/$name.crt; \
fi); \
done); \
rm -rf $ENGINE_CERT_PATH/*.srl $ENGINE_CERT_PATH/*.csr $ENGINE_CERT_PATH/extfile.cnf; \
fi); \
fi) && \
sed -i "/^db_enable = /c\db_enable = $DB_ENABLE" /etc/atuned/engine.cnf && \
sed -i "/^db_host = /c\db_host = $DB_HOST" /etc/atuned/engine.cnf && \
sed -i "/^db_port = /c\db_port = $DB_PORT" /etc/atuned/engine.cnf && \
sed -i "/^db_name = /c\db_name = $DB_NAME" /etc/atuned/engine.cnf && \
(if [ $DB_ENABLE = true ]; then \
apt update && \
apt install -y gcc libpq-dev && \
pip3 install sqlalchemy cryptography psycopg2-binary; \
fi) && \
sed -i "/user_name = /c\user_name = $DB_USER_NAME" /etc/atuned/engine.cnf && \
(if [ $DB_USER_PASSWD ]; then \
res=`python3 ~/atune-engine/tools/encrypt.py -e $DB_USER_PASSWD | awk '{print $2}'`; \
pwd=`echo $res | cut -d' ' -f 1`; \
key=`echo $res | cut -d' ' -f 2`; \
iv=`echo $res | cut -d' ' -f 3`; \
sed -i "/user_passwd = /c\user_passwd = $pwd" /etc/atuned/engine.cnf; \
sed -i "/passwd_key = /c\passwd_key = $key" /etc/atuned/engine.cnf; \
sed -i "/passwd_iv = /c\passwd_iv = $iv" /etc/atuned/engine.cnf; \
fi) && \
service rsyslog start && \
python3 ~/atune-engine/analysis/app_engine.py /etc/atuned/engine.cnf