Model-based-Collaborative-Filtering.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<module 'test' from 'C:\\\\Users\\\\dbsrh\\\\OneDrive\\\\Desktop\\\\model-based-CF\\\\test.py'>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import time\n",
    "import json\n",
    "import math\n",
    "from numba import njit, prange, types\n",
    "from numba.typed import Dict\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "import preprocessing\n",
    "import training\n",
    "import test\n",
    "import tools\n",
    "import importlib\n",
    "\n",
    "# reload moudules\n",
    "importlib.reload(tools)\n",
    "importlib.reload(training)\n",
    "importlib.reload(preprocessing)\n",
    "importlib.reload(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing GroupLens Dataset\n",
    "For testing a model, divide dataset into two parts, training and test. The original dataset is sequential, hence we should shuffle the dataset randomly. We will build an adjacency matrix for the rating matrix because of the memory limits. The whole rating matrix, about **`280000 X 58000`**, cannot be loaded on memory practically. \"userMapper\" and \"movieMapper\" are a hashmap to convert an user and a movie ids into continuous natural number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Load shuffled data : (27753444, 4)\n",
      "Train / Test ratio : 0.8 / 0.19999999999999996\n",
      "Split data : [0:22202755] , [22202755:27753444]\n",
      "##### Get mapper and adjacency matrix\n",
      "userMapper processing Time : 0.3211400508880615\n",
      "movieMapper processing Time : 0.18952274322509766\n",
      "adjacency matrix processing Time : 8106.937525510788\n"
     ]
    }
   ],
   "source": [
    "# set training - test ratio\n",
    "train_ratio = 0.8\n",
    "test_ratio = 1 - train_ratio\n",
    "\n",
    "# for testing, shuffle dataset\n",
    "if not os.path.isfile('shuffled_ratings.csv'):\n",
    "    preprocessing.shuffle_data()\n",
    "    \n",
    "ratings = pd.read_csv('shuffled_ratings.csv')\n",
    "split_bound = int(ratings.shape[0] * train_ratio)\n",
    "\n",
    "print(f\"Load shuffled data : {ratings.shape}\")\n",
    "print(f\"Train / Test ratio : {train_ratio} / {test_ratio}\")\n",
    "print(f\"Split data : [0:{split_bound}] , [{split_bound}:{ratings.shape[0]}]\")\n",
    "\n",
    "train_data = ratings[:split_bound]\n",
    "test_data = ratings[split_bound:]\n",
    "train_data.reset_index(inplace=True, drop=True)\n",
    "test_data.reset_index(inplace=True, drop=True)\n",
    "\n",
    "# training data\n",
    "print(\"##### Get mapper and adjacency matrix\")\n",
    "start_time = time.time()\n",
    "userMapper = preprocessing.userIdMapper(train_data)\n",
    "print(f\"userMapper processing Time : {time.time() - start_time}\")\n",
    "start_time = time.time()\n",
    "movieMapper = preprocessing.movieIdMapper(train_data)\n",
    "print(f\"movieMapper processing Time : {time.time() - start_time}\")\n",
    "start_time = time.time()\n",
    "user_item = preprocessing.build_adjacency_matrix(train_data, userMapper, movieMapper)\n",
    "print(f\"adjacency matrix processing Time : {time.time() - start_time}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training model\n",
    "\n",
    "LFM(Latent Factor Model) is a modeling assuming that there are latent factors to determine an entity. We set 20 dimension features to represent characteristics for each user and movie. These features are the unknown characteristics to calculate a movie rating of a user. This is driven by the assumption that \"Some factors absolutely affect to determine rating but, I do not know what the factors are\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time : 307.0417127609253\n",
      "###### epoch 0 1.300770809690487\n",
      "Time : 302.2743446826935\n",
      "###### epoch 1 1.2679080557498\n",
      "Time : 302.48478055000305\n",
      "###### epoch 2 1.2543538343538112\n",
      "Time : 303.6577317714691\n",
      "###### epoch 3 1.2458869421464702\n",
      "Time : 302.0409677028656\n",
      "###### epoch 4 1.239882007357872\n",
      "Time : 302.5303153991699\n",
      "###### epoch 5 1.2352502303697226\n",
      "Time : 299.1184973716736\n",
      "###### epoch 6 1.231351467493274\n",
      "Time : 308.9363920688629\n",
      "###### epoch 7 1.2277740141279259\n",
      "Time : 307.31588649749756\n",
      "###### epoch 8 1.2243977204112657\n",
      "Time : 307.5912232398987\n",
      "###### epoch 9 1.2213255480747798\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAD4CAYAAAD2FnFTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxV5b3v8c8vMySBzBCGhHlwAIEAEVAcWocWa62trQK1iuXq8ba2PT313Pa2nlPb01fvaXs62hYRsVWx1qGtOFWtiMoYZhRBIIwBEgjzHPK7f+wdDHSTbJLsrAzf9+u1X2attVfWr7u6v3nW86znMXdHRETkbHFBFyAiIi2TAkJERCJSQIiISEQKCBERiUgBISIiESUEXUBTysnJ8V69egVdhohIq7FkyZLd7p4b6VibCohevXpRUlISdBkiIq2GmW0+1zHdYhIRkYgUECIiEpECQkREIlJAiIhIRAoIERGJSAEhIiIRKSBERCSimAaEmc0ws3IzW32O4zea2UozW25mJWY2rtax283sw/Dr9ljVeOzkKR6eu5H5G/bE6hIiIq1SrFsQM4Hr6jj+BjDU3S8B7gSmA5hZFvAAMBoYBTxgZpmxKDA+znj47Y1Mm7shFr9eRKTVimlAuPtcoLKO44f8oxWLUoGan68FXnP3SnffC7xG3UHTYInxcdw6qoA56yrYsudILC4hItIqBd4HYWY3mdkHwIuEWhEA3YGttd62Lbwv0vlTw7enSioqKhpUw62jCogz44lF53ziXESk3Qk8INz9eXcfBHwaeDC82yK99RznT3P3Incvys2NON9Uvbp2TuHjg7vw9OKtHDt5qkG/Q0SkrQk8IGqEb0f1NbMcQi2GnrUO9wDKYnn9yZcWsvfISV5atSOWlxERaTUCDQgz62dmFv55OJAE7AFeBa4xs8xw5/Q14X0xM6ZvNn1yU/njAt1mEhGB2A9znQXMBwaa2TYzm2Jmd5vZ3eG33AysNrPlwG+Az3tIJaHbTYvDr++H98WyViaNLmTZln2s3r4/lpcSEWkV7KNBRK1fUVGRN2Y9iP1HTzL6v17npmHd+dFnhjRhZSIiLZOZLXH3okjHWkwfREvQuUMin76kO39ZVsb+oyeDLkdEJFAKiLNMKi7k6MlTPLd0W9CliIgESgFxlou6d2ZYQQZ/XLCZtnT7TUTkfCkgIpg0upCNFYc1P5OItGsKiAg+OSSfjI6JGvIqIu2aAiKClMR4Pl/Uk7+/v4ud+48FXY6ISCAUEOdw2+gCqt2ZtWhL0KWIiARCAXEOhdmpjB+Qy1OLt3DyVHXQ5YiINDsFRB0mFxey68BxXn9/V9CliIg0OwVEHa4YmEf3jA7qrBaRdkkBUYf4OGNicQHzNuxhffnBoMsREWlWCoh63FLUk8R44/EF6qwWkfZFAVGPnLRkPnFxPs8u2caRE1VBlyMi0mwUEFGYXFzIweNV/HV5TNcsEhFpURQQURhRmMmgrun8cb7mZxKR9kMBEQUzY/Klhby/4wDLtu4LuhwRkWYRs4AwsxlmVm5mq89xfKKZrQy/5pnZ0FrH7jOz1Wb2npl9LVY1no9PX9KdtOQEHp+vIa8i0j7EsgUxE7iujuOlwHh3H0JoedFpAGZ2EfBlYBQwFJhgZv1jWGdUUpMTuHl4d2av3EHl4RNBlyMiEnMxCwh3nwuccx1pd5/n7nvDmwuAHuGfBwML3P2Iu1cBbwE3xarO8zGpuJATp6p5umRr0KWIiMRcS+mDmAK8HP55NXC5mWWbWUfgE0DPc51oZlPNrMTMSioqKmJaZP8u6RT3yeKJhZs5Va3OahFp2wIPCDO7klBA3A/g7muAHwOvAa8AK4BzPoDg7tPcvcjdi3Jzc2Ne76TiQrZWHmXuutiGkYhI0AINCDMbAkwHbnT308u3ufsj7j7c3S8ndJvqw6BqPNs1F3QlNz1Z8zOJSJsXWECYWQHwHDDZ3deddSyv1ns+A8xq/gojS0qI49aRPXlzbTlbK48EXY6ISMzEcpjrLGA+MNDMtpnZFDO728zuDr/le0A28JCZLTezklqnP2tm7wMvAPfW6sxuEW4dXUCcGU8s1PxMItJ2JcTqF7v7rfUcvwu46xzHLotJUU0kv3MHPjY4j6dLtvL1j/cnOSE+6JJERJpc4J3UrdXk4l5UHj7By6t2Bl2KiEhMKCAaaEzfbPrkpKqzWkTaLAVEA8XFGROLC1myeS/vle0PuhwRkSangGiEzw7vQUpinBYTEpE2SQHRCJ07JvKpod34y7LtHDh2MuhyRESalAKikSYX9+LoyVM8t2Rb0KWIiDQpBUQjXdyjM0N7ZvD4wi1aTEhE2hQFRBOYXFzI+vJDLNh4zslrRURaHQVEE5gwJJ+Mjok8riGvItKGKCCaQEpiPLcU9eTV93ay68CxoMsREWkSCogmMnF0AVXVzlOLtJiQiLQNCogmUpidyuUDcnly0WZOnqoOuhwRkUZTQDShycWF7DpwnDfW7Aq6FBGRRlNANKGrBuXRPaOD5mcSkTYhqum+zWwM0Kv2+939DzGqqdWKjzNuG13Af7+6lg0Vh+ibmxZ0SSIiDVZvC8LM/gj8BBgHjAy/imJcV6t1S1FPEuONJzQ/k4i0ctG0IIqAC/w8HxM2sxnABKDc3S+KcHwicH948xBwj7uvCB/7OqHFhBxYBdzh7q1i/GhuejLXX5TPn5ds5ZvXDqBjUszWZBIRialo+iBWA10b8LtnAtfVcbwUGO/uQ4AHgWkAZtYd+CpQFA6WeOALDbh+YCZfWsjBY1W8sKIs6FJERBrsnH/emtkLhP6CTwfeN7NFwPGa4+7+qbp+sbvPNbNedRyfV2tzAdDjrLo6mNlJoCPQqr5piwozGdQ1nT/M38wtRT0xs6BLEhE5b3Xd//hJs1UBU4CXAdx9u5n9BNgCHAX+7u5/P9eJZjYVmApQUFDQDKXWz8yYVFzI//3LapZv3cewgsygSxIROW/nvMXk7m+5+1uEvqgX1tpeBDTZOE4zu5JQQNwf3s4EbgR6A92AVDObVEed09y9yN2LcnNzm6qsRvv0sO6kJsVryKuItFrR9EH8Gaj9aPCp8L5GM7MhwHTgRnffE979MaDU3Svc/STwHDCmKa7XnNKSE/jM8B7MXrmDysMngi5HROS8RRMQCe5++hsu/HNSYy9sZgWEvvwnu/u6Woe2AMVm1tFCN++vBtY09npBmFRcyImqav5covmZRKT1iSYgKszsdIe0md0I7K7vJDObBcwHBprZNjObYmZ3m9nd4bd8D8gGHjKz5WZWAuDuC4FngKWEhrjGER7h1NoM7JrOqN5ZPLFwC9XVWkxIRFoXq+/xBjPrCzwBdA/v2kror/4NMa7tvBUVFXlJSUnQZZzhhRVlfGXWMmbeMZIrBuYFXY6IyBnMbIm7R3z4ud6nuMJBUGxmaYQC5WBTF9iWXXthV3LSknl8wWYFhIi0KtFMtdHZzH4GzAHeNLOfmlnnmFfWRiQlxHHrqJ688UE5WyuPBF2OiEjUoumDmAEcBG4Jvw4Aj8ayqLbm1lEFGDBrkeZnEpHWI5qA6OvuD7j7xvDrP4E+sS6sLemW0YGrB3fhT4u3crzqVNDliIhEJZqAOGpm42o2zGwsoSec5TxMLi5kz+ETvLJ6Z9CliIhEJZqpRu8BHgv3OxhQCdwe06raoHH9cuiV3ZE/zt/MjZd0r/8EEZGA1duCcPfl7j4UGAJc7O7D3H1l7EtrW+LiQvMzlWzey5odB4IuR0SkXtGMYso2s1/y0SimX5hZdswra4M+O6IHyQlxPK75mUSkFYimD+IpoAK4Gfhs+Oc/xbKotiqjYxKfGtqN55dt5+Cxk0GXIyJSp2gCIsvdH3T30vDrB0BGrAtrqyZfWsiRE6d4ftn2oEsREalTNAHxppl9wcziwq9bgBdjXVhbNaRHBkN7dOaP8zdznqu4iog0q2gC4n8BTwInCK0o9xTwDTM7aGbqbW2AScWFfFh+iIWllUGXIiJyTtGMYkp39zh3T3D3xPDP6eFXp+Yosq25YWg3OndI1GJCItKiRTOKycxskpl9N7zd08xGxb60tislMZ7PjejBq6t3Un7gWNDliIhEFM0tpoeAS4HbwtuHgN/ErKJ2YmJxIVXVzlOLtZiQiLRM0QTEaHe/FzgG4O57aYIV5dq73jmpXNY/h1mLtlB1qrr+E0REmlk0AXHSzOIBBzCzXM5cozoiM5thZuVmtvocxyea2crwa56ZDQ3vHxheYa7mdcDMvnYe/5tajcnFhezYf4w3PigPuhQRkX8STUD8EngeyDOzHwLvAP8VxXkzgevqOF4KjHf3IcCDhJcVdfe17n6Ju18CjACOhK/f5lw1KI9unVP0ZLWItEjRrCj3hJktAa4mNFnfp919TRTnzTWzXnUcn1drcwHQI8LbrgY2uHub/AZNiI/jttEF/OTv69hYcYg+uWlBlyQiclo0LQjc/QN3/427/zqacGiAKcDLEfZ/AZhV14lmNtXMSsyspKKiIgalxdYtI3uSGG88sVCLCYlIyxJVQMSSmV1JKCDuP2t/EvAp4M91ne/u09y9yN2LcnNzY1dojOSlp3DthV35c8lWjp7QYkIi0nIEGhBmNgSYDtzo7nvOOnw9sNTddzV/Zc1rcnEhB45V8cKKsqBLERE5rc6AMLN4M3s9Fhc2swLgOWCyu6+L8JZbqef2UlsxqncWA7qk6clqEWlR6gwIdz8FHAmvJndezGwWMB8YaGbbzGyKmd1tZneH3/I9IBt4KDyctaTWuR2BjxMKkDbPzJhcXMiq7ft5d/3uoMsREQHA6ptR1MyeBoqB14DDNfvd/auxLe38FRUVeUlJSf1vbIEOHa9iwi/fZvehEzx25yhGFGYGXZKItANmtsTdiyIdi6YP4kXgu8BcYEmtlzShtOQEZk0tJictidtnLGLJ5r1BlyQi7Vy9LQgAM+sAFLj72tiX1HCtuQVRY+f+Y3xh2vxwS2IkIwqzgi5JRNqwRrUgzOwGYDnwSnj7EjP7W9OWKDW6dk7hqamXkpuezBcfWcSSzVozQkSCEc0tpv8ARgH7ANx9OdA7hjW1e107pzDry8XkdUpRSIhIYKIJiCp333/WPq2VGWOhlsRHIVGySSEhIs0rmoBYbWa3AfFm1t/MfgXMq+8kabwunUIh0aVTCrfPWMRihYSINKNoAuIrwIWE1qOeBRwA2uT02y1Rl04pzFJIiEgAolmT+oi7f4fQzKpXuvt33F3rZDajmpZEV4WEiDSjaEYxjTSzVcBKYJWZrTCzEbEvTWrLqwmJzqGQWFSqkBCR2IrmFtMjwL+4ey937wXcCzwa06okorxOKTz15WLyO6fwpUcXsXDj2fMbiog0nWgC4qC7v12z4e7vAAdjV5LUJa9TaAhsfucU7pi5WCEhIjETTUAsMrPfm9kVZjbezB4C5pjZcDMbHusC5Z/lhTuuFRIiEkvRTNb3Zh2H3d2vatqSGq4tTLVxPsoPHuO2hxeyfe9RHr1jJMV9soMuSURambqm2ohqLqbWor0FBJwZEjO+NJJL+yokRCR6jZ3NVVqwvPRQn0T3zA7cOXMx8zfodpOINA0FRBuQm57MrC8X00MhISJNKGYBYWYzzKzczFaf4/hEM1sZfs0zs6G1jmWY2TNm9oGZrTGzS2NVZ1uRm57Mk+GQuGPmIuZt0Mp0ItI40Two9zkzSw///H/N7LkoRy/NBK6r43gpMN7dhwAPAtNqHfsF8Iq7DwKGAmuiuF67VxMSPTM7cufMxQoJEWmUaFoQ33X3g2Y2DrgWeAz4bX0nuftc4JyP+7r7PHevWTZtAdADwMw6AZcTekAPdz/h7vuiqFMI326aWkxBVjgktMa1iDRQNAFxKvzPTwK/dfe/AklNXMcU4OXwz32ACuBRM1tmZtPNLPVcJ5rZVDMrMbOSioqKJi6rdcpJC7UkCrI6cudji3lXISEiDRBNQGw3s98DtwAvmVlylOdFxcyuJBQQ94d3JQDDCYXRMOAw8O/nOt/dp7l7kbsX5ebmNlVZrV5NSBRmpXLnTIWEiJy/aL7obwFeBa4L3+rJAv6tKS5uZkOA6cCN7l4z9GYbsM3dF4a3nyEUGHKeQiExml7ZCgkROX91BoSZxQGL3P05d/8QwN13uPvfG3thMysAngMmu/u6mv3uvhPYamYDw7uuBt5v7PXaq+yzQuKdDxUSIhKdOgPC3auBFeEv8/NiZrOA+cBAM9tmZlPM7G4zuzv8lu8B2cBDZrbczGo/Av0V4AkzWwlcAvzX+V5fPlITEr1zUpny2GLe/lB9NSJSv2jmYvoHMBJYRKg/AAB3/1RsSzt/7XGqjfOx59BxJk5fSOnuw0y/vYjL+qvPRqS9a9RcTGY2PtJ+d3+rCWprUgqI+lUePsFtDy+gdPdhHv5iEZcPUEiItGeNmospHAQfAOnh15qWGA4SnazUJJ78cjG9c1K56w8lzF2n200iElk0T1LfQuj20ucIjWhaaGafjXVhEjs1IdE3N427/lDCWwoJEYkgmmGu3wFGuvvt7v5FYBTw3diWJbGWlZrEk3eNpm9uGl9WSIhIBNEERJy7l9fa3hPledLCZZ4VEnPWltd/koi0G9F80b9iZq+a2ZfM7EvAi8BLsS1LmktNSPTLTWPqH5coJETktGg6qf8N+D0whNDMqtPc/f66z5LWJDM1iSfuGk3/PIWEiHykviep483s9fCT1N9w96+7+/PNVZw0nzNC4g9L+NUbH3KiqjroskQkQPU9SX0KOGJmnZupHglQRsdQSHzsgjx++to6PvHLt1m4UavTibRX0fRBHANWmdkjZvbLmlesC5NgZHRM4qGJI5jxpSKOnjjF56ct4FvPrGDv4RNBlyYizSwhive8GH5JO3LVoC4UfyObX7zxIdPfLuX1NeV85xOD+czw7phZ0OWJSDOoc6oNM4sHHnP3Sc1XUsNpqo3YWLPjAN9+fhXLtuzj0j7Z/OCmi+ibmxZ0WSLSBBo81Ua4DyLXzJp6BTlpRQbnd+LZu8fwg09fxOqy/Vz/87f5+evrOF51qv6TRaTViuYW0ybgXTP7G2fO5vqzWBUlLU9cnDGpuJBrLuzCg7PX8PPXP+Rvy8v4wU0XMaZvTtDliUgMRNNJXQbMDr83vdZL2qG89BR+deswHrtzFFXVzm0PL+QbTy9nz6HjQZcmIk2s3um+T7/RLNXdD9f/ztPvnwFMAMrd/aIIxyfy0TrUh4B73H1F+Ngm4CBwCqg61/2xs6kPonkdO3mKX/3jQ37/1kbSUhL49vWD+VxRD3Vii7QijZru28wuNbP3gTXh7aFm9lAU150JXFfH8VJgvLsPAR4Epp11/Ep3vyTacJDml5IYz79dO4iX7ruM/nlpfOvZlXx+2gLWlx8MujQRaQLR3GL6OXAtoUn6CP+Vf3l9J7n7XKCyjuPz3H1veHMB0COKWqQFGtAlnT9NvZQf33wxa3ce5PpfvM1PXl3LsZPqxBZpzaKaldXdt561q6n/y58CvFz7ksDfzWyJmU1t4mtJDMTFGZ8fWcAb/zqeG4Z049dvrufan8/V+tcirVg0AbHVzMYAbmZJZvZNwrebmoKZXUkoIGpPADjW3YcD1wP3mtk5WyxmNtXMSsyspKJCX0ZBy0lL5mefv4Qn7hpNnBmTH1nEfU8to+KgOrFFWptoAuJu4F6gO7ANuCS83WhmNgSYDtzo7qcn/XH3svA/y4HnCS1SFJG7T3P3Incvys3V+sotxdh+Obx832V89er+vLxqJ1f/dA5PLtxCdXV0gyJEJHjRTPe9290nunsXd89z90m1v8wbyswKgOeAye6+rtb+VDNLr/kZuAZY3djrSfNLSYznGx8fwEv3Xcbg/E58+/lVfO7381m7U53YIq1B1MNcz/sXm80CrgBygF3AA0AigLv/zsymAzcDm8OnVLl7kZn1IdRqgNCDfE+6+w+juaaGubZc7s6zS7fzwxff5+CxKr58eR++elV/OiTFB12aSLtW1zDXmAVEEBQQLV/l4RP86KU1/HnJNnpmdeD7N17ElQPzgi5LpN1q1HMQIk0pKzWJ//7cUJ6aWkxSfBx3PLqYe59cSvmBY0GXJiJnqbcFYWbJhG4F9aLW3E3u/v2YVtYAakG0LserTjHtrY386s31JMfH8a3rBnLb6ELi4/QktkhzaWwL4q/AjUAVocn6al4ijZKcEM9Xru7Pq1+7nKE9M/juX9/jM7+dx3tl+4MuTUSIrgWxOtJcSi2RWhCtl7vz1+Vl/ODF99l75CR3ju3F1z42gNTkaCYcFpGGamwLYp6ZXdzENYmcwcz49LDuvP6N8dxS1IOH3y7lqp/O4bdzNrD/yMmgyxNpl6JpQbwP9CM0ud5xwAAPT7LXoqgF0XaUbKrkf15fx7vr99AhMZ7PFfXgjrG96Z2TGnRpIm1Ko4a5mllhpP3uvjnS/iApINqeNTsOMOOdUv66vIyT1dVcPSiPO8f15tI+2ZpWXKQJNCggzKyTux8ws6xIx939nDO1BkUB0XZVHDzO4ws28/iCzew5fIIL8jtx57je3DA0n+QEPWwn0lANDYjZ7j7BzEoJza5a+881d/c+TV9q4ygg2r5jJ0/x1+XbeeSdUtbtOkRuejKTiwuZOLqA7LTkoMsTaXX0JLW0Oe7OO+t388g7pcxZW0FyQhw3DevOneN6M6CLVsQViVZdARHVGEIzywT6Ayk1+8ILAokEwsy4rH8ul/XPZX35QWa8u4nnlm7jqcVbuXxALlPG9eby/jnqpxBphGg6qe8C7iO04ttyoBiY7+5Xxb6886MWRPu29/AJnly0hcfmbaL84HH656Vx57je3DSsOymJ6qcQiaSxo5hWASOBBe5+iZkNAv7T3T/f9KU2jgJCAE5UVTN7ZRmPvFPKe2UHyEpNYuLoAiZfWkheekr9v0CkHWnsLaZj7n7MzDCzZHf/wMwGNnGNIk0mKSGOzwzvwU3DurOwtJJH3inl12+u53dvbeCGod2YMq43F3brHHSZIi1eNAGxzcwygL8Ar5nZXqAstmWJNJ6ZUdwnm+I+2WzafZiZ8zbxdMlWnlu6neI+Wdw1rg9XDcojTpMDikR0XqOYzGw80Bl4xd1PxKyqBtItJqnP/qMneSrcT1G2/xi9c1K5Y2wvbh7eQ/M+SbvU4D4IM4sDVjZksj4zmwFMAMojnW9mE4H7w5uHgHvcfUWt4/FACbDd3SdEc00FhETr5KlqXlm9k0feKWX51n10Skng1tEF3H5pL7pldAi6PJFm0+DJ+ty9GlgRXj/6fM0ErqvjeCkwPjyn04PAtLOO3wesacB1ReqVGB/HDUO78Zd7x/LsPWO4rH8uD8/dyGX/702+MmsZy7fuC7pEkcBF06bOB94zs0XUWgfC3T9V10nuPtfMetVxfF6tzQWEhtECYGY9gE8CPwS+EUWNIg02ojCTEYWZbK08wmPzNvGnxVt5YUUZIwozmTKuN9dc0IWEeC2+KO1PNMNcx0fa7+5v1fvLQwExu75bVGb2TWCQu98V3n4G+BGQDnyzrltMZjYVmApQUFAwYvPmFjeHoLQyh45X8fTirTw6r5StlUfp0imZT1yczw1DuzGsZ4YevpM2pbHDXD/h7vfX3mFmPwbqDYgoi7sSmAKMC2/X9FssMbMr6jvf3acRvj1VVFTUduYNkcCkJSdw57je3D6mF6+9v4tnl27jiQVbePTdTXTP6MCEIflMGNKNi7p3UlhImxZNC2Kpuw8/a9/KaNaDqK8FYWZDgOeB6919XXjfj4DJhJY4TQE6Ac+5+6T6rqdOaomVA8dO8tp7u5i9soy3P9xNVbXTK7sjE4Z0Y8LQfAZ2SVdYSKvU0Nlc7wH+BegDbKh1KB14N5ov7LoCItzx/Q/gi2f1R9R+zxXUc4upNgWENIe9h0/w6ns7mb1yB/M27KbaoV9eGjeEw6JvblrQJYpEraEB0RnIJNQX8O+1Dh2MZi0IM5sFXAHkALuAB4BEAHf/nZlNB24GajoNqs4uUgEhLV3FweO8snoHL6zcweJNlbjD4PxOTBiSzw1DulGQ3THoEkXqpOm+RZrBzv3HeGnVDmavLGPpltAw2aE9OjNhSDc+OSRfz1dIi6SAEGlm2/Ye4cWVO5i9cgertu8HQsNpbxiSzycuzievkyYNlJZBASESoE27DzN7ZRmzV+7gg50HMYPRvbO4YWg3rr8on6zUpKBLlHZMASHSQqwvP8gLK0K3oTZUHCY+zhjTN5sbhnTj2gu70rljYtAlSjujgBBpYdydNTsOnm5ZbKk8QmK8cXn/XCYMzedjg7uQnqKwkNhTQIi0YO7Oqu37eWFFGS+u3EHZ/mMkJcRx5cBcbhjajasG5dExSTPNSmwoIERaiepqZ9nWvbywYgcvrdpB+cHjdEiM5+rBeVx3UVcu65er21DSpBQQIq3QqWpnUWkls1eW8fLqnVQePkF8nDG8IIMrBuZxxcBcLsjXdB/SOAoIkVau6lQ1K7btY87aCuasrTg9dDYvPZkrBuZyxcA8xvXPoZP6LeQ8KSBE2pjyg8eYu243b64t5+11FRw4VkV8nDGiMJMrBuZy5cA8BnXV/FBSPwWESBtWdaqaZVv3MWdtOW9+UMH7Ow4A0LVTCuMH5HLloFzG9svRqCiJSAEh0o7sOnCMt9ZWMGddOW+v283B41UkxBlFvTK5YmAeVw7MY0CXNLUuBFBAiLRbJ09Vs3TzXt5cW8GcteV8sPMgAPmdU073XYztl0NasobRtlcKCBEBYMf+o6HWxdoK3lm/m0PHq0iMN0b2yjodGP3z1LpoTxQQIvJPTlRVs2TzXuasLWfO2grW7gq1LrpndGB8uKN7TN9sUtW6aNMUECJSr7J9R8PDaMt5d/1uDp84RVJ8HKN617Qucumbq9ZFW6OAEJHzcqKqmpJNlbwZbl18WH4IgB6ZHbisfw5j++Uwpm+OZqJtAwIJCDObAUwAys+x5OhE4P7w5iHgHndfYWYpwFwgGUgAnnH3B6K5pgJCJDa27T1y+iG9BRv3cOh4FQAX5HdibL9sxvbLYVTvLM0Z1QoFFRCXE/ri/8M5AmIMsMbd95rZ9cB/uPtoC7VfU939kJklAvnJawgAAAkwSURBVO8A97n7gvquqYAQib2qU9Ws3L6fdz/czbsbdrN08z5OnKomMd4Y1jOTsf1yGNsvm6E9M0iMjwu6XKlHYLeYzKwXMDtSQJz1vkxgtbt3P2t/R0IBcY+7L6zvegoIkeZ39MQpFm+q5N0Nu3l3/W7eKzuAO6QmxTO6TzZj+mYzrn8OA7voye6WqK6AaCntwSnAyzUbZhYPLAH6Ab+pKxzMbCowFaCgoCDGZYrI2TokxXP5gFwuH5ALwN7DJ1iwcQ/vrN/NvA17+McH5QDkpCVxad8cxvXLZkzfHHpmdQyybIlC4C0IM7sSeAgY5+57zjqWATwPfMXdV9d3PbUgRFqe7fuO8u763cxbv5t31u9h96HjABRmd2RM39DtKHV4B6fFtiDMbAgwHbj+7HAAcPd9ZjYHuA6oNyBEpOXpntGBW4p6cktRT9ydD8sP8e760O2oF1aUMWvRFiDU4T2ufw5j+marw7uFCOz/ATMrAJ4DJrv7ulr7c4GT4XDoAHwM+HFAZYpIEzIzBnRJZ0CXdO4Y2/ufOrxnvruJaXM3hjq8CzIZ2zeHcf2zGdJDHd5BiOUoplnAFUAOsAt4AEgEcPffmdl04GZgc/iUKncvCrcqHgPigTjgaXf/fjTX1C0mkdatvg7vmhFS6vBuOnpQTkRapbM7vEt3HwZCHd6j+2RT3DuL4j7Z9NP8UQ3WYvsgRETqkpmaxPUX53P9xfnAmR3eC0sreXHlDgCyU5MY3ScUFsV9sjXhYBNRC0JEWiV3Z2vlURZs3HP6Vbb/GABZqUmM7p3F6N5ZFPfNZkBeOnFxCoxI1IIQkTbHzCjI7khBdkduGRkaIbVt71Hmb9zDwo2VLNi4h5dX7wQgs2Mio8K3o0b3zmZQVwVGNBQQItImmBk9szrSM6sjtxT1BGBr5REWlobCYmHpHl59bxcAGR0TGdmr5pZUFoO7dlJgRKCAEJE2qyYwPjuiBxDqw1h4+pZUJa+9HwqMTikJjOodCoviPtkMzu9EvAJDfRAi0n6V7TvKwtI9LNhQycLSPWzacwSA9JQERoVbGKP7ZHFBficS2uhzGBrmKiIShZ37j4UCI9zCqBlWm56cwMiaTu8+2VzYre0EhjqpRUSi0LVzCjde0p0bLwlNLL3rwLHTYbGw9KOJB9OSEyjqlcno3tkU9crk4u6dSUmMD7L0mFBAiIicQ5dOZwZG+YFjpzu9F2zcw5y1FQAkxhsXdOvMiIJMRhSGXl07pwRZepPQLSYRkQbac+g4S7fsY8nmvSzdvJcV2/ZxvKoaCE1SOLwwkxEFGYwozGJQfnqLnE9Kt5hERGIgOy2Zj1/QhY9f0AUIreW9ZscBlmzey5Ite1lcWskLK8oASEmMY2iPjNMtjOEFmWS28CnO1YIQEYmhsn1HQy2MLaFWxntlB6iqDn3v9slNPX1banhhJv1y05r9eQy1IEREAtItowPdMjpww9BuQGjG2pXb9rEkHBhvfFDOn5dsA0LPYwyr1Y8xtGcGacnBfU0rIEREmlGH8NTlo/tkA6E5pTbtORK6LRXuy/if19fhDnEGg7p2CrcwMhhRkEXPrA7NNhGhbjGJiLQw+4+eZPnWfSwN35patmUfh45XAZCTlsyIwo/6Mi7s1rghtrrFJCLSinTukMj4AbmMH5ALwKlqZ92ug6dbGEu37D09r1RSfByX9MzgqanFTd5/EbOAMLMZwASg3N0vinB8InB/ePMQcI+7rzCznsAfgK5ANTDN3X8RqzpFRFq6+DhjcH4nBud3YlJxIQC7Dx1naXi01P4jJ2PSuR3LFsRM4NeEvuwjKQXGu/teM7semAaMBqqAf3X3pWaWDiwxs9fc/f0Y1ioi0qrkpCVzzYVduebCrjG7Rsye2nD3uUBlHcfnufve8OYCoEd4/w53Xxr++SCwBugeqzpFRCSylvJY3xTg5bN3mlkvYBiw8FwnmtlUMysxs5KKioqYFSgi0t4EHhBmdiWhgLj/rP1pwLPA19z9wLnOd/dp7l7k7kW5ubmxLVZEpB0JdBSTmQ0BpgPXu/ueWvsTCYXDE+7+XFD1iYi0Z4G1IMysAHgOmOzu62rtN+ARYI27/yyo+kRE2rtYDnOdBVwB5JjZNuABIBHA3X8HfA/IBh4KPxVYFX5YYywwGVhlZsvDv+7b7v5SrGoVEZF/FrOAcPdb6zl+F3BXhP3vAFoMVkQkYIF3UouISMvUpuZiMrMKYHMDT88BdjdhOa2ZPosz6fM4kz6Pj7SFz6LQ3SMOAW1TAdEYZlZyrgmr2ht9FmfS53EmfR4faeufhW4xiYhIRAoIERGJSAHxkWlBF9CC6LM4kz6PM+nz+Eib/izUByEiIhGpBSEiIhEpIEREJKJ2HxBmdp2ZrTWz9Wb270HXEyQz62lmb5rZGjN7z8zuC7qmoJlZvJktM7PZQdcSNDPLMLNnzOyD8L8jlwZdU5DM7Ovh/05Wm9ksM0sJuqam1q4Dwszigd8A1wMXALea2QXBVhWomtX8BgPFwL3t/PMAuI/QolUCvwBecfdBwFDa8ediZt2BrwJF4SWV44EvBFtV02vXAQGMAta7+0Z3PwE8BdwYcE2B0Wp+ZzKzHsAnCU1J366ZWSfgckIzLePuJ9x9X7BVBS4B6GBmCUBHoCzgeppcew+I7sDWWtvbaMdfiLVFs5pfO/Bz4FtAddCFtAB9gArg0fAtt+lmlhp0UUFx9+3AT4AtwA5gv7v/Pdiqml57D4hIs8a2+3G/0a7m15aZ2QSg3N2XBF1LC5EADAd+6+7DgMNAu+2zM7NMQncbegPdgFQzmxRsVU2vvQfENqBnre0etMFm4vnQan6njQU+ZWabCN16vMrMHg+2pEBtA7a5e02L8hlCgdFefQwodfcKdz9JaPGzMQHX1OTae0AsBvqbWW8zSyLUyfS3gGsKjFbz+4i7/x937+HuvQj9e/EPd29zfyFGy913AlvNbGB419XA+wGWFLQtQLGZdQz/d3M1bbDTPtA1qYPm7lVm9r+BVwmNQpjh7u8FXFaQtJqf1OUrwBPhP6Y2AncEXE9g3H2hmT0DLCU0+m8ZbXDaDU21ISIiEbX3W0wiInIOCggREYlIASEiIhEpIEREJCIFhIiIRKSAEBGRiBQQIiIS0f8HO8xQMYSyPEwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Collaborative Filtering (Model-based), Trainig\n",
    "epoch = 10\n",
    "learning_rate = 0.005\n",
    "regular_term = 0.1\n",
    "latent_dim = 20\n",
    "\n",
    "user_count = len(userMapper.keys())\n",
    "movie_count = len(movieMapper.keys())\n",
    "\n",
    "[fm_users, fm_movies, train_error] = tools.load_files(user_count, movie_count, latent_dim)\n",
    "\n",
    "for i in range(epoch):\n",
    "    start_time = time.time()\n",
    "    lossValue = training.training(user_item, fm_users, user_count, fm_movies, learning_rate, regular_term)\n",
    "    print(f\"Time : {time.time() - start_time}\")\n",
    "    train_error.append(lossValue)\n",
    "    print(f\"###### epoch {i} {lossValue}\")\n",
    "    \n",
    "plt.plot(train_error)\n",
    "plt.ylabel('train errors per epoch')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing\n",
    "\n",
    "After training model, we should check whether model is trained well. We already splited the dataset to test the model, therefore we just use it! We measure performance of the model using RMSE(Root Mean Squared Error)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time : 3141.4417424201965\n",
      "Test dataset error : 0.8761482181999753\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "test_err = test.test_error(test_data, fm_users, userMapper, fm_movies, movieMapper)\n",
    "print(f\"Time : {time.time() - start_time}\")\n",
    "print(f\"Test dataset error : {test_err}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This result shows that the model is well trained successfully, because validation set is used to test the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "tools.save_files(fm_users, fm_movies, train_error)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}