docker-compose repository for MaRDI
git clone [email protected]:MaRDI4NFDI/portal-compose.git
cd portal-compose
cp ./mediawiki/template.env ./.env
Change parameters for your local installation in .env as required, this file will not be committed. Change at least the passwords and secret to any password for local usage:
MW_SECRET_KEY=some-secret-key
MW_ADMIN_PASS=change-this-password
DB_PASS=change-this-sqlpassword
The local install has 2 additional containers:
- Selenium for running tests
- Openrefine for data manipulation
The local install also has open ports, so that the services can be accessed without using the reverse proxy
- Wikibase, http://localhost:8080
- WDQS Frontend, http://localhost:8834
- Quickstatements, http://localhost:8840
- OpenRefine, http://localhost:3333
- Grafana, http://localhost:3000
- Reverse proxy (traefik) dashboard: https://localhost/
Note that the containers for local development are set to not restart so that they do not start automatically when you start your computer.
Some containers are pulled from special MaRDI images:
- wikibase and wikibase_jobrunner are pulled from https://github.com/MaRDI4NFDI/docker-wikibase
- backup is pulled from https://github.com/MaRDI4NFDI/docker-wikibase/tree/main/backup
- quickstatements is pulled from https://github.com/MaRDI4NFDI/docker-quickstatements
traefik is an edge router (or reverse proxy), with the purpose of routing incoming requests to the corresponding services. E.g., requests to https://portal.mardi4nfdi.de are forwarded to the wikibase service. Services are discovered and monitored automatically, but rules to make services accessible on a domain, authentication, redirections, etc, must be defined in the docker-compose files via docker labels for the services' containers. See the docs.
Be careful, the containers need about 13GB of free memory.
The services are defined in the following three files:
docker-compose.yml: basic stack of containers (wikibase, mysql, elasticsearch, redis, wdqs, quickstatements, traefik, portainer).docker-compose-extra.yml: additional set of APIs and analytic tools (importer, matomo, goaccess, prometheus, grafana, etc.).docker-compose-dev.yml: additional parameters for the development environment.
Start up all containers for development:
docker-compose -f docker-compose.yml -f docker-compose-extra.yml -f docker-compose-dev.yml up -d
Stop the containers:
docker-compose -f docker-compose.yml -f docker-compose-extra.yml -f docker-compose-dev.yml down
Define the COMPOSE_FILE variable in your .env file with a value equal to
docker-compose.yml:docker-compose-extra.yml:docker-compose-dev.yml
With this configuration all three files will be automatically read when calling
docker-compose up -d | down | logs
Alternatively, if you only want to start the basic containers defined in docker-compose.yml you can also
cp docker-compose-dev.yml docker-compose.override.yml and modify the new docker-compose.override.yml according
to your needs. The docker-compose.override.yml will also be automatically picked up when calling
docker-compose up -d | down | logs without the need to define COMPOSE_FILE in .env.
See here and here for further info.
- Start up the containers locally as explained above
- Run the tests:
bash ./run_tests.sh
The images to be pulled are defined at the beginning of the docker-compose.yml and docker-compose-extra.yml. For instance
x-wikibase: &wikibase-image
ghcr.io/mardi4nfdi/docker-wikibase:main
To update a given image to a newer version only the url for the corresponding image needs to be updated. Thus, we make sure that all services based on a given image are using the same image version. This also simplifies reproducibility if we need to rollback to a previous state.
After merging a PR the main branch can be deployed to mardi03 (staging server) by manually calling the deployment workflow in srv-mardi03.
When the main branch is ready for release into mardi02 (production server) a tagged commit must be created.
First run git tag to list the latest tags.
Create and push a new signed tag with the newer version.
git tag -s <tag_version>
git push origin <tag_version>
The specified <tag_version> can be referenced in the deploy script in mardi02. The new tagged version will be deployed automatically after pushing to srv-mardi02.
Grafana is a tool to visualize metrics collected by Prometheus. Here, Prometheus
is set up to scrape metrics provided by the edge router traefik, the backup script, and system metrics of the host
system via node-exporter. The Grafana UI can be accessed via https://grafana.portal.mardi4nfdi.de or https://localhost:3000, locally. The dashboards need to be added manually after initializing the Grafana container in the UI via Create->Import:
- backup monitor: import the file grafana/backup_monitor.json
- node-exporter: import e.g. the dashboard id 1860
- traefik: import e.g. the dashboard id 4475
Currently, Grafana does not offer import/export of alerting rules. These have to be created manually, e.g., for the disk usage of the backup drive and failure of the backups.
GoAccess is an opensource log analyzer, set up in docker-compose.yml to
parse the access.log of the reverse proxy traefik.
The resulting report is served with a nginx webserver at
https://stats.portal.mardi4nfdi.de.
We use a custom docker image
docker-goaccess-cron, running
goaccess via cron. The configuration contained in docker-compose.yml and
goaccess/goaccess.conf of
portal-compose is complete with
exception of the following requirements:
The size of traefik logs can easily take gigabytes and should be rotated with the unix
tool logrotate (already running on mardi01). A logrotate config for traefik is given by
traefik/logrotate.conf
(should be placed in /etc/logrotate.d, requires root; see also the project's server
setup notes and
man logrotate).
Our goaccess image expects the two log files access.log and access.log.1, but can
handle the case where the rotated file log.1 does not exist.
In order to resolve IP geo locations, download the free database GeoLite2 City from
here.
An account was already registered with our MaRDI4NFDI groupware email account.
Extract the file GeoLite2-City.mmdb to the directory ./goaccess/.
Create a docker-compose.override.yml like this
version: '3.4'
services:
wikibase:
image: "ghcr.io/mardi4nfdi/docker-wikibase:dev"
environment:
XDEBUG_CONFIG: "remote_host=host.docker.internal"
expose:
- 9000
volumes:
- ./extensions-dev/<extension_to_debug>:/var/www/html/extensions/<extension_to_debug>
- ./debugging/php.ini:/usr/local/etc/php/php.ini
Here ./extensions-dev/<extension_to_debug> is the path of your local development checkout of the extension, you modify.
For extended documentation on debugging with xdebug, see.
Eventually, add the docker-compose.override.yml file to your startup command:
Adjust host.docker.internal on linux as described.
docker-compose -f docker-compose.yml -f docker-compose-dev.yml -f docker-compose.override.yml up -dThe containers will be built and tested automatically by GitHub after each commit on the main branch. The CI steps are defined in .github/workflows/main.yml.
Preparations this has already been done on GitHub:
- create a GitHub environment
- call it "staging" (specified in .github/workflows/main.yml)
- set (required) these to test passwords, change the default values:
MW_SECRET_KEY=some-secret-key
MW_ADMIN_PASS=change-this-password
DB_PASS=change-this-sqlpassword
- also set these environment variables:
WIKIBASE_HOST=localhost
WIKIBASE_PORT=8080
WDQS_FRONTEND_HOST=localhost
WDQS_FRONTEND_PORT=8834
QUICKSTATEMENTS_HOST=localhost
QUICKSTATEMENTS_PORT=8840
WB_PUBLIC_HOST_AND_PORT=localhost:8080
QS_PUBLIC_HOST_AND_PORT=localhost:8840
MW_ELASTIC_HOST=localhost
MW_ELASTIC_PORT=9200