examples/04_model_select_and_optimize/azureml_hyperdrive_surprise_svd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.<br>\n",
    "Licensed under the MIT License.</i>\n",
    "<br><br>\n",
    "# SVD Hyperparameter Tuning with Azure Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we show how to tune the hyperparameters of a matrix factorization algorithm by utilizing **Azure Machine Learning service** ([AzureML](https://azure.microsoft.com/en-us/services/machine-learning-service/)) in the context of movie recommendations. To use AzureML you will need an Azure subscription. We use the SVD algorithm from the Surprise library.\n",
    "\n",
    "We present the overall process of utilizing AzureML by demonstrating some key steps while avoiding too much detail. \n",
    "\n",
    "For more details about the **SVD** algorithm:\n",
    "* [Surprise SVD deep-dive notebook](../02_model/surprise_svd_deep_dive.ipynb)\n",
    "* [Original paper](http://papers.nips.cc/paper/3208-probabilistic-matrix-factorization.pdf)\n",
    "* [Surprise homepage](https://surprise.readthedocs.io/en/stable/)\n",
    "  \n",
    "Regarding **AzureML**, please refer to:\n",
    "* [Quickstart notebook](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-create-workspace-with-python)\n",
    "* [Hyperdrive](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-tune-hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Global Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.8 |Anaconda, Inc.| (default, Feb 21 2019, 18:30:04) [MSC v.1916 64 bit (AMD64)]\n",
      "Surprise version: 1.0.6\n",
      "Azure ML SDK Version: 1.0.10\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append(\"../../\")\n",
    "import os\n",
    "import surprise\n",
    "import papermill as pm\n",
    "import pandas as pd\n",
    "import shutil\n",
    "from tempfile import TemporaryDirectory\n",
    "\n",
    "import azureml as aml\n",
    "import azureml.widgets\n",
    "import azureml.train.hyperdrive as hd\n",
    "\n",
    "from reco_utils.dataset import movielens\n",
    "from reco_utils.dataset.python_splitters import python_random_split\n",
    "from reco_utils.evaluation.python_evaluation import rmse, precision_at_k, ndcg_at_k\n",
    "from reco_utils.recommender.surprise.surprise_utils import predict, compute_ranking_predictions\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Surprise version: {}\".format(surprise.__version__))\n",
    "print(\"Azure ML SDK Version:\", aml.core.VERSION)\n",
    "\n",
    "# Temp dir to cache temporal files while running this notebook\n",
    "tmp_dir = TemporaryDirectory()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We assume that an AzureML workspace has already been created. For instructions how to do this, see [here](README.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# AzureML workspace info. Note, will look up \"aml_config\\config.json\" first, then fall back to using this\n",
    "SUBSCRIPTION_ID = '<subscription-id>'\n",
    "RESOURCE_GROUP  = '<resource-group>'\n",
    "WORKSPACE_NAME  = '<workspace-name>'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found the config file in: C:\\Users\\anargyri\\aml_config\\config.json\n",
      "AzureML workspace name:  anargyri\n"
     ]
    }
   ],
   "source": [
    "# Connect to a workspace\n",
    "try:\n",
    "    ws = aml.core.Workspace.from_config()\n",
    "except aml.exceptions.UserErrorException:\n",
    "    try:\n",
    "        ws = aml.core.Workspace(\n",
    "            subscription_id=SUBSCRIPTION_ID,\n",
    "            resource_group=RESOURCE_GROUP,\n",
    "            workspace_name=WORKSPACE_NAME\n",
    "        )\n",
    "        ws.write_config()\n",
    "    except aml.exceptions.AuthenticationException:\n",
    "        ws = None\n",
    "\n",
    "if ws is None:\n",
    "    raise ValueError(\n",
    "        \"\"\"Cannot access the AzureML workspace w/ the config info provided.\n",
    "        Please check if you entered the correct id, group name and workspace name\"\"\"\n",
    "    )\n",
    "else:\n",
    "    print(\"AzureML workspace name: \", ws.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the following cells, we\n",
    "1. Create a *remote compute target* (cpu_cluster) if it does not exist already,\n",
    "2. Mount a *data store* and upload the data set, and\n",
    "3. Run a hyperparameter tuning experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Create a Remote Compute Target\n",
    "\n",
    "We create an AI Compute for our remote compute target. The script will load the cluster if it already exists. You can look at [this document](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets) to learn more about setting up a *compute target*.\n",
    "\n",
    "> Note: we create a low priority cluster to save costs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found existing cluster, use it.\n",
      "Succeeded\n",
      "AmlCompute wait for completion finished\n",
      "Minimum number of nodes requested have been provisioned\n",
      "{'allocationState': 'Steady', 'allocationStateTransitionTime': '2019-03-05T14:25:20.509000+00:00', 'creationTime': '2019-02-06T15:20:44.924560+00:00', 'currentNodeCount': 0, 'errors': None, 'modifiedTime': '2019-02-06T15:21:35.653142+00:00', 'nodeStateCounts': {'idleNodeCount': 0, 'leavingNodeCount': 0, 'preemptedNodeCount': 0, 'preparingNodeCount': 0, 'runningNodeCount': 0, 'unusableNodeCount': 0}, 'provisioningState': 'Succeeded', 'provisioningStateTransitionTime': None, 'scaleSettings': {'minNodeCount': 0, 'maxNodeCount': 4, 'nodeIdleTimeBeforeScaleDown': 'PT120S'}, 'targetNodeCount': 0, 'vmPriority': 'Dedicated', 'vmSize': 'STANDARD_D2_V2'}\n"
     ]
    }
   ],
   "source": [
    "# Remote compute (cluster) configuration. If you want to save costs decrease these.\n",
    "# Each standard_D2_V2 VM has 2 vCPUs, 7GB memory, 100GB SSD storage\n",
    "\n",
    "VM_SIZE = 'STANDARD_D2_V2'\n",
    "VM_PRIORITY = 'lowpriority'\n",
    "# Cluster nodes\n",
    "MIN_NODES = 0\n",
    "MAX_NODES = 8\n",
    "\n",
    "# Choose a name for your CPU cluster\n",
    "cpu_cluster_name = \"cpuclustersvd\"\n",
    "\n",
    "# Verify that cluster does not exist already\n",
    "try:\n",
    "    cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
    "    print('Found existing cluster, use it.')\n",
    "except ComputeTargetException:\n",
    "    compute_config = AmlCompute.provisioning_configuration(vm_size=VM_SIZE, \n",
    "                                                           min_nodes=MIN_NODES, \n",
    "                                                           vm_priority=VM_PRIORITY,\n",
    "                                                           max_nodes=MAX_NODES)\n",
    "    cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
    "\n",
    "cpu_cluster.wait_for_completion(show_output=True)\n",
    "\n",
    "# Use the 'status' property to get a detailed status for the current cluster. \n",
    "print(cpu_cluster.status.serialize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Prepare Dataset\n",
    "1. Download data and split into training, validation and testing sets. The metric used for tuning the hyperparameters is evaluated on the valdation set and the final reported results are evaluated on the test set.\n",
    "2. Upload the data set to the default **blob storage** of the workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select MovieLens data size: 100k, 1m, 10m, or 20m\n",
    "MOVIELENS_DATA_SIZE = '100k'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>itemID</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID  itemID  rating\n",
       "0     196     242     3.0\n",
       "1     186     302     3.0\n",
       "2      22     377     1.0\n",
       "3     244      51     2.0\n",
       "4     166     346     1.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = movielens.load_pandas_df(\n",
    "    size=MOVIELENS_DATA_SIZE,\n",
    "    header=[\"userID\", \"itemID\", \"rating\"]\n",
    ")\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, validation, test = python_random_split(data, [0.7, 0.15, 0.15], seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading aml_data\\movielens_100k_test.pkl\n",
      "Uploading aml_data\\movielens_100k_train.pkl\n",
      "Uploading aml_data\\movielens_100k_val.pkl\n",
      "Uploaded aml_data\\movielens_100k_test.pkl, 1 files out of an estimated total of 3\n",
      "Uploaded aml_data\\movielens_100k_val.pkl, 2 files out of an estimated total of 3\n",
      "Uploaded aml_data\\movielens_100k_train.pkl, 3 files out of an estimated total of 3\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "$AZUREML_DATAREFERENCE_a19c6926d4734c98bf63a762190f454f"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DATA_DIR = os.path.join(tmp_dir.name, 'aml_data')\n",
    "os.makedirs(DATA_DIR, exist_ok=True)\n",
    "\n",
    "TRAIN_FILE_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_train.pkl\"\n",
    "train.to_pickle(os.path.join(DATA_DIR, TRAIN_FILE_NAME))\n",
    "\n",
    "VAL_FILE_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_val.pkl\"\n",
    "validation.to_pickle(os.path.join(DATA_DIR, VAL_FILE_NAME))\n",
    "\n",
    "TEST_FILE_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_test.pkl\"\n",
    "test.to_pickle(os.path.join(DATA_DIR, TEST_FILE_NAME))\n",
    "\n",
    "# Note, all the files under DATA_DIR will be uploaded to the data store\n",
    "ds = ws.get_default_datastore()\n",
    "ds.upload(\n",
    "    src_dir=DATA_DIR,\n",
    "    target_path='data',\n",
    "    overwrite=True,\n",
    "    show_progress=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Prepare Hyperparameter Tuning "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also prepare a training script [svd_training.py](../../reco_utils/azureml/svd_training.py) for the hyperparameter tuning, which will log our target metrics such as [RMSE](https://en.wikipedia.org/wiki/Root-mean-square_deviation) and/or [NDCG](https://en.wikipedia.org/wiki/Discounted_cumulative_gain) to AzureML experiment so that we can track the metrics and optimize the primary metric via **hyperdrive**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "SCRIPT_DIR = os.path.join(tmp_dir.name, 'aml_script')\n",
    "\n",
    "# Clean-up scripts if already exists\n",
    "shutil.rmtree(SCRIPT_DIR, ignore_errors=True)\n",
    "\n",
    "# Copy scripts to SCRIPT_DIR temporarly\n",
    "shutil.copytree(os.path.join('..', '..', 'reco_utils'), os.path.join(SCRIPT_DIR, 'reco_utils'))\n",
    "\n",
    "# add training script examples\n",
    "shutil.copytree(\"train_scripts\", os.path.join(SCRIPT_DIR, 'train_scripts'))\n",
    "\n",
    "ENTRY_SCRIPT_NAME = 'train_scripts/svd_training.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we define a search space for the hyperparameters. All the parameter values will be passed to our training script.\n",
    "\n",
    "We specify the output directory as ./outputs. The outputs directory is specially treated by Azure ML in that all the content in this directory gets uploaded to the workspace as part of the run history. The files written to this directory are therefore accessible even once the remote run is over. In the training script (svd_training.py), we use the output directory for saving the trained models. \n",
    "\n",
    "AzureML hyperdrive provides `RandomParameterSampling`, `GridParameterSampling`, and `BayesianParameterSampling`. Details about each approach are beyond the scope of this notebook and can be found in [Azure doc](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-tune-hyperparameters). Here, we use the Bayesian sampling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "EXP_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_svd_model\"\n",
    "PRIMARY_METRIC = 'precision_at_k'\n",
    "RATING_METRICS = ['rmse']\n",
    "RANKING_METRICS = ['precision_at_k', 'ndcg_at_k']  \n",
    "USERCOL = 'userID'\n",
    "ITEMCOL = 'itemID'\n",
    "REMOVE_SEEN = True\n",
    "K = 10\n",
    "RANDOM_STATE = 0\n",
    "VERBOSE = True\n",
    "NUM_EPOCHS = 30\n",
    "BIASED = True\n",
    "\n",
    "script_params = {\n",
    "    '--datastore': ds.as_mount(),\n",
    "    '--train-datapath': \"data/\" + TRAIN_FILE_NAME,\n",
    "    '--validation-datapath': \"data/\" + VAL_FILE_NAME,\n",
    "    '--output_dir': './outputs',\n",
    "    '--surprise-reader': 'ml-100k',\n",
    "    '--rating-metrics': RATING_METRICS,\n",
    "    '--ranking-metrics': RANKING_METRICS,\n",
    "    '--usercol': USERCOL,\n",
    "    '--itemcol': ITEMCOL,\n",
    "    '--k': str(K),\n",
    "    '--random-state': str(RANDOM_STATE),\n",
    "    '--epochs': str(NUM_EPOCHS),\n",
    "}\n",
    "\n",
    "if BIASED:\n",
    "    script_params['--biased'] = ''\n",
    "if VERBOSE:\n",
    "    script_params['--verbose'] = ''\n",
    "if REMOVE_SEEN:\n",
    "    script_params['--remove-seen'] = ''\n",
    "    \n",
    "# hyperparameters search space\n",
    "# We do not set 'lr_all' and 'reg_all' because they will be overwritten by the other lr_ and reg_ parameters\n",
    "\n",
    "hyper_params = {\n",
    "    'n_factors': hd.choice(10, 50, 100, 150, 200),\n",
    "    'init_mean': hd.uniform(-0.5, 0.5),\n",
    "    'init_std_dev': hd.uniform(0.01, 0.2),\n",
    "    'lr_bu': hd.uniform(1e-6, 0.1), \n",
    "    'lr_bi': hd.uniform(1e-6, 0.1), \n",
    "    'lr_pu': hd.uniform(1e-6, 0.1), \n",
    "    'lr_qi': hd.uniform(1e-6, 0.1), \n",
    "    'reg_bu': hd.uniform(1e-6, 1),\n",
    "    'reg_bi': hd.uniform(1e-6, 1), \n",
    "    'reg_pu': hd.uniform(1e-6, 1), \n",
    "    'reg_qi': hd.uniform(1e-6, 1)\n",
    "}\n",
    "\n",
    "# Note, BayesianParameterSampling only support choice, uniform, and quniform\n",
    "ps = hd.BayesianParameterSampling(hyper_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you submit the experiment, you can see the progress from the notebook by using `azureml.widgets.RunDetails`. You can directly check the details from the Azure portal as well. To get the link, run `run.get_portal_url()`.\n",
    "\n",
    "For RandomSampling, you can use early termnination policy\n",
    "```\n",
    "policy = hd.BanditPolicy(evaluation_interval=1, slack_factor=0.1, delay_evaluation=3)\n",
    "```\n",
    "\n",
    "> Since we will do hyperparameter tuning, we create a `HyperDriveRunConfig` and pass it to the experiment object. If you already know what hyperparameters to use and still want to utilize AzureML for other purposes (e.g. model management), you can set the hyperparameter values directly to `script_params` and run the experiment, `run = exp.submit(est)`, instead.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hyperdrive experimentation configuration\n",
    "MAX_TOTAL_RUNS = 100  # Number of runs (training-and-evaluation) to search for the best hyperparameters. \n",
    "MAX_CONCURRENT_RUNS = 8\n",
    "\n",
    "est = azureml.train.estimator.Estimator(\n",
    "    source_directory=SCRIPT_DIR,\n",
    "    entry_script=ENTRY_SCRIPT_NAME,\n",
    "    script_params=script_params,\n",
    "    compute_target=cpu_cluster,\n",
    "    conda_packages=['pandas', 'scikit-learn'],\n",
    "    pip_packages=['scikit-surprise', 'psutil']\n",
    ")\n",
    "\n",
    "hd_config = hd.HyperDriveRunConfig(\n",
    "    estimator=est, \n",
    "    hyperparameter_sampling=ps,\n",
    "    primary_metric_name=PRIMARY_METRIC,\n",
    "    primary_metric_goal=hd.PrimaryMetricGoal.MAXIMIZE, \n",
    "    max_total_runs=MAX_TOTAL_RUNS,\n",
    "    max_concurrent_runs=MAX_CONCURRENT_RUNS\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Execute Runs in AzureML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "61cd3c66d91f433888ffffc6e45b46fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_HyperDriveWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO'…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RunId: movielens_100k_svd_model_1551957798853\n",
      "\n",
      "Execution Summary\n",
      "=================\n",
      "RunId: movielens_100k_svd_model_1551957798853\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'runId': 'movielens_100k_svd_model_1551957798853',\n",
       " 'target': 'cpuclustersvd',\n",
       " 'status': 'Completed',\n",
       " 'endTimeUtc': '2019-03-07T11:57:23.000Z',\n",
       " 'properties': {'primary_metric_config': '{\"name\": \"precision_at_k\", \"goal\": \"maximize\"}',\n",
       "  'runTemplate': 'HyperDrive',\n",
       "  'azureml.runsource': 'hyperdrive'},\n",
       " 'logFiles': {'azureml-logs/hyperdrive.txt': 'https://anargyri6699545766.blob.core.windows.net/azureml/ExperimentRun/dcid.movielens_100k_svd_model_1551957798853/azureml-logs/hyperdrive.txt?sv=2018-03-28&sr=b&sig=5vpEl9OQZvTgU4v%2BiIAKATIHBJg8UyBw8e3jD0xtAz4%3D&st=2019-03-07T11%3A47%3A26Z&se=2019-03-07T19%3A57%3A26Z&sp=r'}}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "313732b33930494d8245cf305ffd99f0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_UserRunWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', '…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create an experiment to track the runs in the workspace\n",
    "exp = aml.core.Experiment(workspace=ws, name=EXP_NAME)\n",
    "run = exp.submit(config=hd_config)\n",
    "\n",
    "azureml.widgets.RunDetails(run).show()\n",
    "run.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see the experiment progress from this notebook by using `azureml.widgets.RunDetails(hd_run).show()` or check from the Azure portal with the url link you can get by running `hd_run.get_portal_url()`.\n",
    "To load an existing Hyperdrive run, use `hd_run = hd.HyperDriveRun(exp, <user-run-id>, hyperdrive_run_config=hd_run_config)`. You also can cancel a run with `hd_run.cancel()`.\n",
    "![](https://recodatasets.z20.web.core.windows.net/images/svd_hyperdrive1.PNG)\n",
    "![](https://recodatasets.z20.web.core.windows.net/images/svd_hyperdrive2.PNG)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Show Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get best run and printout metrics\n",
    "best_run = run.get_best_run_by_primary_metric()\n",
    "\n",
    "best_run_metrics = best_run.get_metrics()\n",
    "parameter_values = best_run.get_details()['runDefinition']['arguments']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Number of epochs': 30,\n",
       " 'rmse': 1.0343498081373697,\n",
       " 'precision_at_k': 0.10000000000000002,\n",
       " 'ndcg_at_k': 0.11498322243961594}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_run_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--datastore $AZUREML_DATAREFERENCE_workspaceblobstore --train-datapath data/movielens_100k_train.pkl --validation-datapath data/movielens_100k_val.pkl --output_dir ./outputs --surprise-reader ml-100k --rating-metrics rmse --ranking-metrics precision_at_k ndcg_at_k --usercol userID --itemcol itemID --k 10 --random-state 0 --epochs 30 --biased --verbose --n_factors 150 --init_mean -0.4163305768968 --init_std_dev 0.159711436379793 --lr_bu 0.0386753983834255 --lr_bi 4.48660045721016E-05 --lr_pu 0.0119378772073106 --lr_qi 0.0936873305814469 --reg_bu 0.385400397115581 --reg_bi 0.975251474623207 --reg_pu 0.906537637834819 --reg_qi 0.801240951271603\n"
     ]
    }
   ],
   "source": [
    "print(\" \".join(parameter_values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now evaluate the metrics on the test data. To do this, get the SVD model that was saved as model.dump in the training script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_DIR = os.path.join(tmp_dir.name, 'aml_model') \n",
    "os.makedirs(MODEL_DIR, exist_ok=True)\n",
    "best_run.download_file('outputs/model.dump', output_file_path=MODEL_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "svd = surprise.dump.load(os.path.join(MODEL_DIR, 'model.dump')[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'rmse': 1.0331492610799313, 'precision_at_k': 0.09968017057569298, 'ndcg_at_k': 0.1160964958978592}\n"
     ]
    }
   ],
   "source": [
    "test_results = {}\n",
    "predictions = predict(svd, test, usercol=USERCOL, itemcol=ITEMCOL)\n",
    "for metric in RATING_METRICS:\n",
    "    test_results[metric] = eval(metric)(test, predictions, col_user=USERCOL, col_item=ITEMCOL)\n",
    "\n",
    "all_predictions = compute_ranking_predictions(svd, train, usercol=USERCOL, itemcol=ITEMCOL, remove_seen=REMOVE_SEEN)\n",
    "for metric in RANKING_METRICS:\n",
    "    test_results[metric] = eval(metric)(test, all_predictions, col_prediction='prediction', k=K, col_user=USERCOL, col_item=ITEMCOL)\n",
    "\n",
    "print(test_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cleanup files\n",
    "tmp_dir.cleanup()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Concluding Remarks\n",
    "\n",
    "We showed how to tune **all** the hyperparameters accepted by Surprise SVD simultaneously, by utilizing the Azure Machine Learning service. \n",
    "For example, training and evaluation of a single SVD model takes about 50 seconds on the 100k MovieLens data on a Standard D2_V2 VM. Searching through 100 different combinations of hyperparameters sequentially would take about 80 minutes whereas this notebook took less than half that. With AzureML, one can easily specify the size of the cluster according to the problem at hand and use Bayesian sampling to navigate efficiently through a large space of hyperparameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "* [Matrix factorization algorithms in Surprise](https://surprise.readthedocs.io/en/stable/matrix_factorization.html) \n",
    "* [Surprise SVD deep-dive notebook](../02_model/surprise_svd_deep_dive.ipynb)\n",
    "* [Fine-tune natural language processing models using Azure Machine Learning service](https://azure.microsoft.com/en-us/blog/fine-tune-natural-language-processing-models-using-azure-machine-learning-service/)\n",
    "* [Training, hyperparameter tune, and deploy with TensorFlow](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-tensorflow/train-hyperparameter-tune-deploy-with-tensorflow.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (reco_base)",
   "language": "python",
   "name": "reco_base"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}