MySQL for DataJoint

Docker image of MySQL (version 5.7 or 8.0) that comes configured for use with DataJoint!

Administering the MySQL server

WARNING Whether hosted via Docker or directly on a machine, the MySQL server needs to be administered properly. Please refer to an appropriate reference to administer the server safely and efficiently.

How to use

Using docker-compose

The simplest and the recommended way to configure and run a MySQL server with Docker is to use the Docker compose. Once you have docker and docker-compose installed on your system (the one on which you'd want to run the MySQL server), copy this docker-compose.yaml to a folder, and run docker-compose up -d to start the MySQL server. Here is a series of commands to run in the terminal to achieve this:

mkdir mysql-docker
cd mysql-docker
wget https://raw.githubusercontent.com/datajoint/mysql-docker/master/docker-compose.yaml
docker-compose up -d

This will start the MySQL server mapped to localhost's port 3306, and any MySQL data will be stored in the directory ./data, or if you followed the above instructions, mysql-docker/data directory.

By default the database sets up user root with password simple (refer to below on how to change this at the startup). You can access your locally running MySQL server using mysql client. On Ubuntu system, you can obtain this by installing apt package mysql-client as follows:

sudo apt-get install mysql-client

Once mysql client is installed, you can access the running server:

mysql -h 127.0.0.1 -u root -p
Enter password: [type in your password here: default is "simple"]

mysql >

Read on to find out the details about the content of the docker-compose.yaml and how you can customize it to fit your needs.

What's in docker-compose.yaml?

The content of the docker-compose.yaml is rather simple:

version: '2.4'
services:
  db:
    image: datajoint/mysql:5.7 # or datajoint/mysql:8.0
    ports:
      - "3306:3306"
    environment:
      - MYSQL_ROOT_PASSWORD=simple
      # - DATAJOINT_DATABASE_CONFIG_MONITOR=FALSE
    volumes:
      - ./data:/var/lib/mysql
      # - ./my.cnf:/etc/mysql/my.cnf

Let's step through the parts you might want to customize. The line

- MYSQL_ROOT_PASSWORD=simple

configures the default password. If you would like to use something else, you can modify this prior to starting your server with docker-compose up.

The line:

# - DATAJOINT_DATABASE_CONFIG_MONITOR=FALSE

has been commented to indicate that it is optional. It is a DataJoint-unique feature that can be enabled to allow live reloading of the database server if it detects MySQL config changes. The default is FALSE as it is depicted.

The lines:

volumes:
  - ./data:/var/lib/mysql
  # - ./my.cnf:/etc/mysql/my.cnf

maps the local directory ./data to the /var/lib/mysql inside the container where MySQL stores all of its data by default. The next line is commented to indicate that it is optional. It overriding the built-in MySQL configuration that is optimized for DataJoint use to allow it to be easily modified. Uncommenting it will allow you to directly modify the configuration from your machine which is useful when coupling with the above DATAJOINT_DATABASE_CONFIG_MONITOR feature.

WARNING: If you decide to map volume /var/lib/mysql (like in the example), then settings for your MySQL server will persist across separate Docker mysql instances. In particular, this means that the MYSQL_ROOT_PASSWORD setting will be used only when the very first mysql Docker container is created. To change the root password on an alredy created mysql Docker instance, access the database via mysql client as root and run:

$ mysql -h 127.0.0.1 -u root -p
Enter password: [type in your old password]

mysql > SET PASSWORD FOR root = PASSWORD('your_new_password');

replacing the 'your_new_password' with your new desired password surrounded by quotes (').

Backup

If you wish to take a disk-based backup of the database, simply back up the local directory matching the :/var/lib/mysql volume in your compose file when the database is stopped. To restore, simply copy the backed up data into place before launching the container. Please note that this type of backup is version-specific to the mysql version used with this container.

To take a backup without stopping the container, you can run:

mysql> flush tables with read lock;

prior to taking the backup, and

mysql> unlock tables;

after taking the backup. Please note that this will prevent all write activity in the database while the backup is running, and may not provide a consistent state in some environments (e.g. docker-machine or virtualization environments such as docker for windows). Be sure to check related volume mount documentation for your platform and perform regular recovery checks for your environment before relying on any backup strategy.

More advanced methods for backup such as mysqldump and xtrabackup can also be used to backup the database; please see relevent documentation for further details.

A strategy such as mysqldump has the benefit of being able to perform a backup without requiring database downtime. The following is an example:

mysqldump -h$DJ_HOST -u${DJ_USER} -p$DJ_PASS --system=user --quick -v --databases schema_a schema_b schema_c > backup.sql

In the above, we connect to MySQL server $DJ_HOST using user $DJ_USER with password $DJ_PASS. We perform a backup of the specific schemas schema_a, schema_b, schema_c written to a file backup.sql. Optional flags were added to include: user management info (usernames + grants), skip caching to commit more often to file, and write more verbose logs.