Doc db migration guide (#696)

* doc: inital DB migration guide Summary of changes: - Initial guide. - Testing is still in progress. * fix: removed shell * fix: change type of cells * fix: updated username and password * fix: added password retrieval for mysql-k8s charm * Update docs/db-migration-guide.ipynb Co-authored-by: Daniela Plascencia <[email protected]> * doc: updated according to comments * fix: updated according to comments * fix: added markdown version * Wording updates * doc: updated markdown version * fix: updated according to comments --------- Co-authored-by: Daniela Plascencia <[email protected]> Co-authored-by: ColmBhandal <[email protected]>
canonical · Sep 22, 2023 · 0b66611 · 0b66611
1 parent 0054bd2
commit 0b66611
Show file tree

Hide file tree

Showing 2 changed files with 480 additions and 0 deletions.
diff --git a/docs/db-migration-guide.ipynb b/docs/db-migration-guide.ipynb
@@ -0,0 +1,302 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Charmed Kubeflow Databases Migration Guide\n",
+    "\n",
+    " When upgrading to Charmed Kubeflow 1.8 the databases must be migrated to MySQL. There are two databases in Charmed Kubeflow - `katib-db` and `kfp-db`. In Charmed Kubeflow 1.7 these databases are backed by the `charmed-osm-mariadb-k8s` charm. In the same release (1.7) we introduced support for MySQL databases, which can be deployed using the `mysql-k8s` charm. In Charmed Kubeflow 1.8 release MariaDB is not supported. Databases must be migrated prior to upgrading to Charmed Kubeflow 1.8. Migration process describe is only applicable for migration from CKF 1.7 to 1.8."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Do you need to migrate?\n",
+    "\n",
+    "A database migration is only required if the output of the following command is `mariadb-k8s`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "shellscript"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "    # replace DB_CHARM with desired unit name: `katib-db` or `kfp-db`\n",
+    "    DB_CHARM= <kfp-db | katib-db>\n",
+    "    juju show-application $DB_CHARM | yq '.[] | .charm'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Prerequisites\n",
+    "\n",
+    "- Client machine with access to deployed Charmed Kubeflow 1.7.\n",
+    "- Juju version 2.9.\n",
+    "- Enough storage in the cluster to support backup/restore of the databases.\n",
+    "- `mysql-client` on client machine (install by running `sudo apt install mysql-client`)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Obtain existing database credentials\n",
+    "\n",
+    "To obtain credentials for existing databases execute the following commands for each database to be migrated. Use those credentials in migration steps."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "shellscript"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# replace DB_UNIT with desired unit name: `katib-db/0` or `kfp-db/0`\n",
+    "DB_UNIT= <katib-db/0 | kfp-db/0>\n",
+    "\n",
+    "# obtain username and password of existing MariadDB database from DB relation\n",
+    "# in Charmed Kubeflow 1.7 username used is `root` and password is specified in `mysql` relation by\n",
+    "# 'root_password'\n",
+    "DB_MYSQL_RELATION_ID=$(juju show-unit $DB_UNIT | yq '.[] | .relation-info | select(.[].endpoint == \"mysql\") | .[0] | .relation-id')\n",
+    "DB_USER=root\n",
+    "DB_PASSWORD=$(bash -c \"juju run --unit $DB_UNIT 'relation-get -r $DB_MYSQL_RELATION_ID - $DB_UNIT' | grep root_password\" | awk '{print $2}')\n",
+    "DB_IP=$(juju show-unit $DB_UNIT | yq '.[] | .address')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Deploy new MySQL databases and obtain credentials\n",
+    "\n",
+    "Deploy new MySQL databases and obtain credentials for each new database by executing the following commands, once per database:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "shellscript"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# replace MYSQL_DB with desired application name: `katib-db-mysql` or `kfp-db-mysql`\n",
+    "MYSQL_DB= <katib-db-mysql | kfp-db-mysql>\n",
+    "\n",
+    "# deploy new MySQL database charm\n",
+    "juju deploy mysql-k8s $MYSQL_DB --channel 8.0/stable --trust\n",
+    "\n",
+    "# obtain username and password of new MySQL database from MySQL charm\n",
+    "MYSQL_DB_USER=$(juju run-action $MYSQL_DB/0 get-password --wait | yq '.[] | .results.username')\n",
+    "MYSQL_DB_PASSWORD=$(juju run-action $MYSQL_DB/0 get-password --wait | yq '.[] | .results.password')\n",
+    "MYSQL_DB_IP=$(juju show-unit $MYSQL_DB/0 | yq '.[] | .address')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Katib DB migration\n",
+    "\n",
+    "Use the credentials and information obtained in previous steps for `katib-db` to perform the database migration by executing the following commands:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "shellscript"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# ensure that there are no new connections are made and that dababase is not altered\n",
+    "# remove relation between katib-db-manager charm and katib-db charm\n",
+    "juju remove-relation katib-db-manager katib-db\n",
+    "\n",
+    "# connect to unit's IP address using mysql client and verify that `katib` database exists\n",
+    "# this command will log you into mysql monitor prompt\n",
+    "mysql --host=$DB_IP --user=$DB_USER --password=$DB_PASSWORD\n",
+    "\n",
+    "# exit mysql client session\n",
+    "\n",
+    "# create backup of all databases file using `mysqldump` utility, username, password, and unit's IP address\n",
+    "# obtained earlier\n",
+    "# file `katib-db.sql` will be created that can be used to restore database\n",
+    "mysqldump --host=$DB_IP --user=$DB_USER --password=$DB_PASSWORD --column-statistics=0 --databases katib  > katib-db.sql\n",
+    "\n",
+    "# connect to new DB using username, password, and unit's IP address and restore database from backup\n",
+    "mysql --host=$MYSQL_DB_IP --user=$MYSQL_DB_USER --password=$MYSQL_DB_PASSWORD < katib-db.sql\n",
+    "\n",
+    "# relate katib-db-manager and new MySQL database charm\n",
+    "juju relate katib-db-manager:relational-db $MYSQL_DB:database"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## KFP DB migration\n",
+    "\n",
+    "Use the credentials and information obtained in the previous steps for `kfp-db` to perform the database migration by executing the following commands:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "shellscript"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# ensure that there are no new connections are made and that dababase is not altered\n",
+    "# remove relation between kfp-api charm and kfp-db charm\n",
+    "juju remove-relation kfp-api kfp-db\n",
+    "\n",
+    "# connect to unit's IP address using mysql client and verify that `mlpipeline` database exists\n",
+    "# this command will log you into mysql monitor prompt\n",
+    "mysql --host=$DB_IP --user=$DB_USER --password=$DB_PASSWORD\n",
+    "\n",
+    "# exit mysql client session\n",
+    "\n",
+    "# create backup of all databases file using `mysqldump` utility, username, password, and unit's IP address\n",
+    "# obtained earlier\n",
+    "# file `kfp-db.sql` will be created that can be used to restore database\n",
+    "mysqldump --host=$DB_IP --user=$DB_USER --password=$DB_PASSWORD --column-statistics=0 --databases mlpipeline  > kfp-db.sql\n",
+    "\n",
+    "# connect to new DB using username, password, and unit's IP address and restore database from backup\n",
+    "mysql --host=$MYSQL_DB_IP --user=$MYSQL_DB_USER --password=$MYSQL_DB_PASSWORD < kfp-db.sql\n",
+    "\n",
+    "# relate kfp-api and new MySQL database charm\n",
+    "juju relate kfp-api:relational-db $MYSQL_DB:database"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Verify DB migration\n",
+    "\n",
+    "Note: some variables will vary per Katib vs. KFP, namely: `$MYSQL_DB_PASSWORD` and `$MYSQL_DB_IP`. These must be adjusted for the correct database, accordingly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "shellscript"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# compare new MySQL database and compare it to the backup created earlier\n",
+    "mysqldump --host=$MYSQL_DB_IP --user=$MYSQL_DB_USER --password=$MYSQL_DB_PASSWORD --column-statistics=0 --databases katib  > katib-db-new.sql\n",
+    "diff katib-db.sql katib-db-new.sql\n",
+    "mysqldump --host=$MYSQL_DB_IP --user=$MYSQL_DB_USER --password=$MYSQL_DB_PASSWORD --column-statistics=0 --databases mlpipeline  > kfp-db-new.sql\n",
+    "diff kfp-db.sql kfp-db-new.sql"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The difference between two SQL backup files should be limited to server versions, IP addresses, timestamps and other non data related information. Example of difference:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "shellscript"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "< -- Host: 10.1.45.226    Database: katib\n",
+    "---\n",
+    "> -- Host: 10.1.46.40    Database: katib\n",
+    "5c5\n",
+    "< -- Server version\t5.5.5-10.3.17-MariaDB-1:10.3.17+maria~bionic\n",
+    "---\n",
+    "> -- Server version\t8.0.34-0ubuntu0.22.04.1\n",
+    "16a17,26\n",
+    "> SET @MYSQLDUMP_TEMP_LOG_BIN = @@SESSION.SQL_LOG_BIN;\n",
+    "> SET @@SESSION.SQL_LOG_BIN= 0;\n",
+    "> \n",
+    "> --\n",
+    "> -- GTID state at the beginning of the backup \n",
+    "> --\n",
+    "> \n",
+    "> SET @@GLOBAL.GTID_PURGED=/*!80000 '+'*/ '0d3210b9-587f-11ee-acf3-b26305f815ec:1-4,\n",
+    "> 34442d83-587f-11ee-84f5-b26305f815ec:1-85,\n",
+    "> 34444583-587f-11ee-84f5-b26305f815ec:1';\n",
+    "22c32\n",
+    "< CREATE DATABASE /*!32312 IF NOT EXISTS*/ `katib` /*!40100 DEFAULT CHARACTER SET latin1 */;\n",
+    "---\n",
+    "> CREATE DATABASE /*!32312 IF NOT EXISTS*/ `katib` /*!40100 DEFAULT CHARACTER SET latin1 */ /*!80016 DEFAULT ENCRYPTION='N' */;\n",
+    "34c44\n",
+    "<   `id` int(11) NOT NULL,\n",
+    "---\n",
+    ">   `id` int NOT NULL,\n",
+    "60c70\n",
+    "<   `id` int(11) NOT NULL AUTO_INCREMENT,\n",
+    "---\n",
+    ">   `id` int NOT NULL AUTO_INCREMENT,\n",
+    "75a86\n",
+    "> SET @@SESSION.SQL_LOG_BIN = @MYSQLDUMP_TEMP_LOG_BIN;\n",
+    "86c97\n",
+    "< -- Dump completed on 2023-09-21 17:05:54\n",
+    "---\n",
+    "> -- Dump completed on 2023-09-21 17:09:40\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Remove old databases"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "shellscript"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "juju remove-application --destroy-storage katib-db kfp-db "
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}