Skip to content

Latest commit

 

History

History
269 lines (215 loc) · 11.2 KB

README.md

File metadata and controls

269 lines (215 loc) · 11.2 KB
Raw

portal-compose

docker-compose repository for MaRDI

Local installation

git clone [email protected]:MaRDI4NFDI/portal-compose.git
cd portal-compose
cp ./mediawiki/template.env ./.env

Change parameters for your local installation in .env as required, this file will not be committed. Change at least the passwords and secret to any password for local usage:

MW_SECRET_KEY=some-secret-key
MW_ADMIN_PASS=change-this-password
DB_PASS=change-this-sqlpassword

The local install has 2 additional containers:

  • Selenium for running tests
  • Openrefine for data manipulation

The local install also has open ports, so that the services can be accessed without using the reverse proxy

Note that the containers for local development are set to not restart so that they do not start automatically when you start your computer.

Some containers are pulled from special MaRDI images:

Notes on the traefik reverse proxy

traefik is an edge router (or reverse proxy), with the purpose of routing incoming requests to the corresponding services. E.g., requests to https://portal.mardi4nfdi.de are forwarded to the wikibase service. Services are discovered and monitored automatically, but rules to make services accessible on a domain, authentication, redirections, etc, must be defined in the docker-compose files via docker labels for the services' containers. See the docs.

Start up the containers

Be careful, the containers need about 13GB of free memory.

The services are defined in the following three files:

  • docker-compose.yml: basic stack of containers (wikibase, mysql, elasticsearch, redis, wdqs, quickstatements, traefik, portainer).
  • docker-compose-extra.yml: additional set of APIs and analytic tools (importer, matomo, goaccess, prometheus, grafana, etc.).
  • docker-compose-dev.yml: additional parameters for the development environment.

Start up all containers for development:

docker-compose -f docker-compose.yml -f docker-compose-extra.yml -f docker-compose-dev.yml up -d

Stop the containers:

docker-compose -f docker-compose.yml -f docker-compose-extra.yml -f docker-compose-dev.yml down

Tip 1:

Define the COMPOSE_FILE variable in your .env file with a value equal to docker-compose.yml:docker-compose-extra.yml:docker-compose-dev.yml With this configuration all three files will be automatically read when calling docker-compose up -d | down | logs

Tip 2:

Alternatively, if you only want to start the basic containers defined in docker-compose.yml you can also cp docker-compose-dev.yml docker-compose.override.yml and modify the new docker-compose.override.yml according to your needs. The docker-compose.override.yml will also be automatically picked up when calling docker-compose up -d | down | logs without the need to define COMPOSE_FILE in .env.

See here and here for further info.

Test locally

  1. Start up the containers locally as explained above
  2. Run the tests: bash ./run_tests.sh

Define images to pull

The images to be pulled are defined at the beginning of the docker-compose.yml and docker-compose-extra.yml. For instance

x-wikibase: &wikibase-image
  ghcr.io/mardi4nfdi/docker-wikibase:main

To update a given image to a newer version only the url for the corresponding image needs to be updated. Thus, we make sure that all services based on a given image are using the same image version. This also simplifies reproducibility if we need to rollback to a previous state.

Deploy on the MaRDI server

After merging a PR the main branch can be deployed to mardi03 (staging server) by manually calling the deployment workflow in srv-mardi03.

When the main branch is ready for release into mardi02 (production server) a tagged commit must be created. First run git tag to list the latest tags. Create and push a new signed tag with the newer version.

git tag -s <tag_version>
git push origin <tag_version>

The specified <tag_version> can be referenced in the deploy script in mardi02. The new tagged version will be deployed automatically after pushing to srv-mardi02.

Configure Grafana

Grafana is a tool to visualize metrics collected by Prometheus. Here, Prometheus is set up to scrape metrics provided by the edge router traefik, the backup script, and system metrics of the host system via node-exporter. The Grafana UI can be accessed via https://grafana.portal.mardi4nfdi.de or https://localhost:3000, locally. The dashboards need to be added manually after initializing the Grafana container in the UI via Create->Import:

Currently, Grafana does not offer import/export of alerting rules. These have to be created manually, e.g., for the disk usage of the backup drive and failure of the backups.

GoAccess log analyzer

GoAccess is an opensource log analyzer, set up in docker-compose.yml to parse the access.log of the reverse proxy traefik. The resulting report is served with a nginx webserver at https://stats.portal.mardi4nfdi.de.

We use a custom docker image docker-goaccess-cron, running goaccess via cron. The configuration contained in docker-compose.yml and goaccess/goaccess.conf of portal-compose is complete with exception of the following requirements:

Logrotation

The size of traefik logs can easily take gigabytes and should be rotated with the unix tool logrotate (already running on mardi01). A logrotate config for traefik is given by traefik/logrotate.conf (should be placed in /etc/logrotate.d, requires root; see also the project's server setup notes and man logrotate). Our goaccess image expects the two log files access.log and access.log.1, but can handle the case where the rotated file log.1 does not exist.

GeoIP Database

In order to resolve IP geo locations, download the free database GeoLite2 City from here. An account was already registered with our MaRDI4NFDI groupware email account. Extract the file GeoLite2-City.mmdb to the directory ./goaccess/.

Develop/debug locally

Create a docker-compose.override.yml like this

version: '3.4'

services:
  wikibase:
    image: "ghcr.io/mardi4nfdi/docker-wikibase:dev"
    environment:
      XDEBUG_CONFIG: "remote_host=host.docker.internal"
    expose:
      - 9000
    volumes:
      - ./extensions-dev/<extension_to_debug>:/var/www/html/extensions/<extension_to_debug>
      - ./debugging/php.ini:/usr/local/etc/php/php.ini

Here ./extensions-dev/<extension_to_debug> is the path of your local development checkout of the extension, you modify.

For extended documentation on debugging with xdebug, see.

Eventually, add the docker-compose.override.yml file to your startup command:

Adjust host.docker.internal on linux as described.

docker-compose -f docker-compose.yml -f docker-compose-dev.yml -f docker-compose.override.yml up -d

Build on CI

The containers will be built and tested automatically by GitHub after each commit on the main branch. The CI steps are defined in .github/workflows/main.yml.

Preparations this has already been done on GitHub:

  • create a GitHub environment
  • call it "staging" (specified in .github/workflows/main.yml)
  • set (required) these to test passwords, change the default values:
MW_SECRET_KEY=some-secret-key
MW_ADMIN_PASS=change-this-password
DB_PASS=change-this-sqlpassword
  • also set these environment variables:
WIKIBASE_HOST=localhost
WIKIBASE_PORT=8080

WDQS_FRONTEND_HOST=localhost
WDQS_FRONTEND_PORT=8834

QUICKSTATEMENTS_HOST=localhost
QUICKSTATEMENTS_PORT=8840

WB_PUBLIC_HOST_AND_PORT=localhost:8080
QS_PUBLIC_HOST_AND_PORT=localhost:8840

MW_ELASTIC_HOST=localhost
MW_ELASTIC_PORT=9200