training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Estimation of Direction of Arrival (DOA) for First Order Ambisonic Audio Files using Artificial Neural Networks\n",
    "\n",
    "**Pedro Pablo Lucas Bravo**\n",
    "\n",
    "**pedropl@uio.no**\n",
    "\n",
    "Return to [file.ipybn](file.ipynb)\n",
    "\n",
    "Go back to [feature_extraction.ipybn](feature_extraction.ipynb)\n",
    "\n",
    "Continue to [inference.ipybn](inference.ipynb)\n",
    "\n",
    "# Training\n",
    "\n",
    "**Before running**: If you DONT want to save the models set the next vatiable to FALSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "save = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Packages and Utilitiy Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import sklearn\n",
    "import joblib\n",
    "\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.neural_network import MLPRegressor\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import RepeatedKFold\n",
    "from sklearn.model_selection import cross_validate\n",
    "\n",
    "import time\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "#Taken from: https://stackoverflow.com/questions/2827393/angles-between-two-n-dimensional-vectors-in-python\n",
    "def unit_vector(vector):\n",
    "    \"\"\" Returns the unit vector of the vector.  \"\"\"\n",
    "    return vector / np.linalg.norm(vector)\n",
    "\n",
    "def angle_between(v1, v2):\n",
    "    \"\"\" Returns the angle in radians between vectors 'v1' and 'v2'::\n",
    "\n",
    "            >>> angle_between((1, 0, 0), (0, 1, 0))\n",
    "            1.5707963267948966\n",
    "            >>> angle_between((1, 0, 0), (1, 0, 0))\n",
    "            0.0\n",
    "            >>> angle_between((1, 0, 0), (-1, 0, 0))\n",
    "            3.141592653589793\n",
    "    \"\"\"\n",
    "    v1_u = unit_vector(v1)\n",
    "    v2_u = unit_vector(v2)\n",
    "    return np.arccos(np.clip(np.dot(v1_u, v2_u), -1.0, 1.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Training Features from CSV File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading features...\n",
      "(15798, 768)\n",
      "(15798, 3)\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "#Load from file\n",
    "print('Loading features...')\n",
    "dataset = pd.read_csv('features_train.csv')\n",
    "num_features = 768\n",
    "features = dataset.iloc[:,0:num_features].to_numpy()\n",
    "target = np.zeros((features.shape[0],3))\n",
    "target[:,0] = dataset['x'] \n",
    "target[:,1] = dataset['y']\n",
    "target[:,2] = dataset['z']\n",
    "print(features.shape)\n",
    "print(target.shape)\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "The training process includes a pipeline with 3 steps: \n",
    "\n",
    "1. A 'whitening' transformation using a StandardScaler.\n",
    "2. Dimensionality reduction using Principal Component Analysis(PCA) to take 40 component.\n",
    "3. A Multi-layer Perceptron regressor with 2 hidden layers (50 and 25 respectively) with iteration limit of 500, a logistic activation function, learning rate of 0.0001 (learning rate is constant), tolerance for converging of 0.00001, a L2 regularization parameter alpha of 0.1 to avoid overfitting, and the Adam optimization algorithm as solver.\n",
    "\n",
    "Before the training, this pipeline is tested with a repeated k-fold validation (k = 5 with 5 repetitions) strategy to check relevant scores for the application. To arrive to this pipeline and network architecture, I tested several approaches regarding feature extraction and pipeline by using a Grid Search, specially in terms of hyperparameters for the ANN regarding hidden layers, activation function, and alpha; also training several PCA components. Part of these experiments are found in the [experiments](./experiments) folder (Remember that the notebooks here will not run unless they are in the root folder)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explained Variance for PCA\n",
    "\n",
    "For chosing the number of components for PCA. The explained variance was taken into accounto and the 'Elbow method' was applied to keep a certain amount of variance for the training process. The next section shows the plot that helps to find the value, which was 40 components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEWCAYAAACT7WsrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAqBUlEQVR4nO3deZxddX3/8df7LnMnmZnsAwaSEAgRRRGlkcUNEK2AVWx/toIbtCJVUete/dUfrn086lJrF5VSUdQiihZbSkGkggu4QAKySiCsCQSSkG2WzP75/XHOndwMc2fOJHPv3Mm8n4/Hfdxzvmf73Mzkfua7nO9RRGBmZjaa3FQHYGZmjctJwszMqnKSMDOzqpwkzMysKicJMzOryknCzMyqcpKwaU/SJZI+m3HfaySdXYMYlksKSYXJPneV63VKOqwe17KZzUnC6kbSw5J2pV9w5de/1DOGiDgtIr5Vz2tK+rGkT49SfoakJ/YmsUREa0Q8ODkRmlXnJGH19pr0C678evdUB1QH3wLeLEkjyt8CXBoRA1lPVK+ailmZk4Q1BElfk/QfFeufk/RTJU6StEHS/5W0Ja2RvKnKeeZLukrSZknb0uUlFdt/JuncdPkcSTdK+mK670OSTqvYd66kiyVtlPSYpM9Kyqfb8ulxWyQ9CLx6jI/3n8BC4KWVcQJ/BHxb0rGSfi1pe3qtf5HUVLFvSDpf0v3A/RVlh6fLr5Z0m6SdktZL+mTFseVmsLMlPZrG+zcV2/Ppv+sDkjokrZG0NN32LEnXSdoqaa2kPxvjM9p+yknCGsUHgaPSL+6XAm8Dzo7d88Y8A1gEHAycDVwk6YhRzpMDvgkcAiwDdgFjNWkdB6xNz/154OKKv/gvAQaAw4EXAH8InJtuezvJl/wLgFXA66tdICJ2AZcDb60o/jPg3oi4HRgE3p/GcAJwCvCuEad5XRrrkaNcois99zySZPVOSa8bsc9LgCPSc18g6dlp+QeAs4DTgTnAXwDdklqA64DvAgcAZwJflTTa9W1/FhF++VWXF/Aw0Alsr3i9vWL7ccBW4BHgrIryk0i+rFsqyi4H/l+6fAnw2SrXfD6wrWL9Z8C56fI5wLqKbbOBIElIBwK9wKyK7WcBN6TL1wPvqNj2h+mxhSpxvCT9vM3p+k3A+6vs+z7gRxXrAbx8xD4BHF7l+C8D/5AuL0/3XVKx/WbgzHR5LXDGKOd4A/DLEWX/Cnxiqn+P/Krvy+2bVm+vi4j/HW1DRPw2bbo5gCQJVNoWEV0V648AB408h6TZwD8ApwLz0+I2SfmIGBzlsk9UXL87rUS0AguAIrCxoishB6xPlw+qWC7HU1VE3ChpC/A6SbcAxwJ/ksb8TOBLJDWS2UABWDPiFOupQtJxwN8BzwWagBLwg2qfE+hOPyPAUuCBUU57CHCcpO0VZQXgO9XisP2Tm5usYUg6n+QL7nHgIyM2z0+bQMqWpfuN9EGSZpXjImIO8LLy6ScYznqSmsSiiJiXvuZExHPS7RtJvmAr4xnPt0mahd4MXBsRT6blXwPuBVamMf/fUeIda7rm7wJXAksjYi5w4SjHV7MeWFGl/OcVn31eJAMN3pnxvLafcJKwhpD+Nf1Zki/QtwAfkfT8Ebt9SlJT2mfxRzz9r2WANpJ+iO2SFgCf2Jt4ImIj8BPg7yXNkZSTtELSiekulwPvlbQk7YT+aIbTfht4BUl/RuUw3DZgJ9Ap6VnARL+I24CtEdEj6VjgjRM49uvAZyStTAcJPE/SQuAq4JmS3iKpmL5eWNGXYTOEk4TV239rz/skfpQO6/x34HMRcXtE3E/y1/R3JJXS454AtpHUHi4l6Q+4d5TzfxmYBWwBfgP8eB9ifStJ88096bV/CCxOt/0bcC1wO3ArcMV4J4uIh4FfAS0kf/mXfYjki70jPe/3Jxjnu4BPS+oALuDpTXVj+VK6/09IEtXFJP0wHST9LGeS/Js/AXyOpKZnM4gi/NAha2ySTgL+PSKWjLOrmU0y1yTMzKwqJwkzM6vKzU1mZlaVaxJmZlbVtLuZbtGiRbF8+fKpDsPMbFpZs2bNlohon+hx0y5JLF++nNWrV091GGZm04qkMWcFqMbNTWZmVpWThJmZVeUkYWZmVTlJmJlZVU4SZmZWlZOEmZlV5SRhZmZVzZgksfaJDv7+J2t5qrN3qkMxM5s2ZkySWLepk3++fh1bOvumOhQzs2ljxiSJYj55mmP/4NAUR2JmNn3MoCSRfFQnCTOz7GZMkiikNYmBIU+NbmaW1YxJEsM1iQHXJMzMsppBSSLtk3BNwswssxmTJAq55KMOuE/CzCyzGZMkdndcuyZhZpZVzZKEpG9I2iTprirb3yTpDkl3SvqVpKNrFQt4CKyZ2d6oZU3iEuDUMbY/BJwYEUcBnwEuqmEsFNKaxMCQk4SZWVY1e3xpRPxC0vIxtv+qYvU3wJJaxQJQyJVrEm5uMjPLqlH6JN4GXFNto6TzJK2WtHrz5s17dYGmgm+mMzObqClPEpJOJkkSf11tn4i4KCJWRcSq9vb2vbpOuSYx4JqEmVlmNWtuykLS84CvA6dFxFO1vFbB03KYmU3YlNUkJC0DrgDeEhH31fp6TR4Ca2Y2YTWrSUi6DDgJWCRpA/AJoAgQERcCFwALga9KAhiIiFW1imd47ibXJMzMMqvl6Kazxtl+LnBura4/0vDoJk/LYWaW2ZR3XNeLJIp5uSZhZjYBMyZJQDJ/kzuuzcyym1lJIi93XJuZTcCMShJN+Zyn5TAzm4AZlSQKedE/4JqEmVlWMytJ5HL0uyZhZpbZjEoSyegm1yTMzLKaYUnCo5vMzCZiRiWJQj7n0U1mZhMwo5JEMS+PbjIzm4AZliTc3GRmNhEzKkkUcr6ZzsxsImZUkijmc567ycxsAmZYkhADngXWzCyzGZUkCvkcfQOuSZiZZTWjkoRrEmZmEzPDkoT7JMzMJmJGJYnkeRKuSZiZZTWjkkQxL98nYWY2ATMqSRTcJ2FmNiEzKkkU8zn6PbrJzCyzmZckPHeTmVlmMypJFHJ+noSZ2UTMqCRRzOcYGAoinCjMzLKoWZKQ9A1JmyTdVWW7JP2TpHWS7pB0TK1iKSvmBeBhsGZmGdWyJnEJcOoY208DVqav84Cv1TAWIJmWA/AzJczMMqpZkoiIXwBbx9jlDODbkfgNME/S4lrFA0lzE7gmYWaW1bhJQtKBki6WdE26fqSkt03CtQ8G1lesb0jLRovhPEmrJa3evHnzXl9wd3OTaxJmZllkqUlcAlwLHJSu3we8r0bxjCoiLoqIVRGxqr29fa/PU8ilzU2uSZiZZZIlSSyKiMuBIYCIGAAGJ+HajwFLK9aXpGU1U3BNwsxsQrIkiS5JC4EAkHQ8sGMSrn0l8NZ0lNPxwI6I2DgJ562qabhPwknCzCyLQoZ9PkDyhb5C0k1AO/D68Q6SdBlwErBI0gbgE0ARICIuBK4GTgfWAd3An+9F/BNSrkl4/iYzs2zGTRIRcaukE4EjAAFrI6I/w3FnjbM9gPOzBjoZyn0SrkmYmWWTZXTT+UBrRNwdEXcBrZLeVfvQJl9TwTfTmZlNRJY+ibdHxPbySkRsA95es4hqaPfoJtckzMyyyJIk8pJUXpGUB5pqF1LtFDwth5nZhGTpuP4x8H1J/5qu/2VaNu14dJOZ2cRkSRJ/TZIY3pmuXwd8vWYR1ZDnbjIzm5gso5uGSCbfq/kEfLVWyLm5ycxsIsZNEpJeDHwSOCTdXyQjWA+rbWiTr6ngaTnMzCYiS3PTxcD7gTVMznQcU2Z3TcLNTWZmWWRJEjsi4pqaR1IHRXdcm5lNSJYkcYOkLwBXAL3lwoi4tWZR1Yin5TAzm5gsSeK49H1VRVkAL5/8cGrLNQkzs4nJMrrp5HoEUg/FnJ9MZ2Y2EVlqEkh6NfAcoLlcFhGfrlVQtTLc3OSahJlZJlkm+LsQeAPwHpLhr39KMhx22nFzk5nZxGSZu+lFEfFWYFtEfAo4AXhmbcOqjaLnbjIzm5AsSWJX+t4t6SCgH1hcu5BqRxL5nDwth5lZRln6JK6SNA/4AnArycimaTl3EyS1Cd9xbWaWTZbRTZ9JF/9D0lVAc0RMxjOup0Qxl6PPfRJmZplUTRKSXh4R10v6k1G2ERFX1Da02ii4JmFmltlYNYkTgeuB14yyLUjuwJ52ivmc+yTMzDKqmiQi4hOScsA1EXF5HWOqqWI+R9+AaxJmZlmMObopfZbER+oUS10U8h7dZGaWVZYhsP8r6UOSlkpaUH7VPLIaKeTcJ2FmllWWIbBvSN/PrygLYNo9dAjS5iaPbjIzy2TcmkREHDrKK1OCkHSqpLWS1kn66Cjbl0m6QdJtku6QdPrefIiJKOZznrvJzCyjrBP8PRc4kj0n+Pv2OMfkga8ArwQ2ALdIujIi7qnY7ePA5RHxNUlHAlcDyyf0CSYo6ZNwc5OZWRZZnnH9CeAkkiRxNXAacCMwZpIAjgXWRcSD6Xm+B5wBVCaJAOaky3OBxycQ+15JRje5JmFmlkWWjuvXA6cAT0TEnwNHk3yhj+dgYH3F+oa0rNIngTdL2kCSgN4z2okknSdptaTVmzdvznDp6koF90mYmWWVaYK/dCjsgKQ5wCZg6SRd/yzgkohYApwOfCe9N2MPEXFRRKyKiFXt7e37dMFSIU9Pv5OEmVkWWfokVqcT/P0bsAboBH6d4bjH2DOZLEnLKr0NOBUgIn4tqRlYRJKIaqK5mKO3f7BWpzcz269UrUlI+oqkF0fEuyJie0RcSNIJfXba7DSeW4CVkg6V1AScCVw5Yp9HSZqykPRsko7xfWtPGkepkKfXfRJmZpmMVZO4D/iipMXA5cBlEXFb1hNHxICkdwPXAnngGxFxt6RPA6sj4krgg8C/SXo/SSf2ORFR06FHzcUcPa5JmJllMtbcTf8I/KOkQ0hqAd+QNAu4jCRh3DfeySPiapIO6cqyCyqW7wFevJex7xXXJMzMsstyM90jEfG5iHgBSUfz64Df1zqwWnFNwswsu3GThKSCpNdIuhS4BlgLPO0ZE9NFczHPwFD4rmszswzGeujQK0lqDqcDNwPfA86LiK46xVYTpUKSF3sHhijks4wANjObucbquP4Y8F3ggxGxrU7x1FxzMQ9AT/8gLaVMs5KYmc1YY3Vcv7yegdRLczGpPfS489rMbFwzrr2lVEhqEr6hzsxsfGPdTFeqZyD1Ukz7Ifr94CEzs3GNVZP4NYCk79Qplroo5AVAv0c3mZmNa6ye2yZJbwReJOlpQ14j4orahVU7xTRJ+JkSZmbjGytJvAN4EzAPeM2IbQFMyyRRyCWVJ98nYWY2vrFGN90I3ChpdURcXMeYasp9EmZm2WW5UeA7kt4LvCxd/zlwYUT01y6s2im6T8LMLLMsSeKrQDF9B3gL8DXg3FoFVUvlu6wHhpwkzMzGkyVJvDAijq5Yv17S7bUKqNYKuXJNws1NZmbjyXIz3aCkFeUVSYcB0/ZOtHKfxICThJnZuLLUJD4M3CDpQUDAIUCWJ9M1pN1DYN3cZGY2nnGTRET8VNJK4Ii0aG1E9NY2rNrx6CYzs+wyTYOaJoU7ahxLXfiOazOz7GbcBH++mc7MLLsZlySa3NxkZpZZlseXStKbJV2Qri+TdGztQ6uNgjuuzcwyy1KT+CpwAsmjTAE6gK/ULKIa290n4ZqEmdl4snRcHxcRx0i6DSAitklqqnFcNVPMlZubXJMwMxtPlppEv6Q8ycyvSGoHMn3DSjpV0lpJ6yR9tMo+fybpHkl3S/pu5sj3Ui4ncvLNdGZmWWSpSfwT8CPgAEl/C7we+Ph4B6WJ5SvAK4ENwC2SroyIeyr2WQl8DHhxWkM5YC8+w4QV8zn63SdhZjauLDfTXSppDXAKyR3Xr4uI32c497HAuoh4EEDS94AzgHsq9nk78JWI2JZea9ME498rLaUCHT0D9biUmdm0Nm6SkLQA2ARcVlFWzDBV+MHA+or1DcBxI/Z5Znq+m4A88MmI+HGGuPfJAW0lNu2ctjeNm5nVTZbmpluBpcA2kprEPOAJSU8Cb4+INft4/ZXAScAS4BeSjoqI7ZU7SToPOA9g2bJl+3C5xDPmNvPkzp59Po+Z2f4uS8f1dcDpEbEoIhYCpwFXAe9i9zMmRvMYSXIpW5KWVdoAXBkR/RHxEHAfSdLYQ0RcFBGrImJVe3t7hpDHdmBbM084SZiZjStLkjg+Iq4tr0TET4ATIuI3QGmM424BVko6NB0yeyZw5Yh9/pOkFoGkRSTNTw9mjn4vzZtdZOeuaflgPTOzusqSJDZK+mtJh6SvjwBPpqOXqg4RiogB4N3AtcDvgcsj4m5Jn5b02nS3a4GnJN0D3AB8OCKe2qdPlEGpmKd3YIgID4M1MxtLlj6JNwKfIPmrH+CmtCwP/NlYB0bE1cDVI8ouqFgO4APpq26ai0lu7B0YormYr+elzcymlSxDYLcA76myed3khlMfzYUkMfT0DzpJmJmNIcsQ2HbgI8BzgOZyeUS8vIZx1VQ5MfT0+4Y6M7OxZOmTuBS4FzgU+BTwMEmn9LRVbm7q6Z+2j+o2M6uLLEliYURcDPRHxM8j4i+AaVuLACiVm5sGnCTMzMaSpeO6PFZ0o6RXA48DC2oXUu3trkm4ucnMbCxZksRnJc0FPgj8MzAHeF8tg6q13X0SrkmYmY0lS5LYFhE7gB3AyQCSXlzTqGrMfRJmZtlk6ZP454xl00a5T6J3wM1NZmZjqVqTkHQC8CKgXVLlzW5zSG6km7ZmNbm5ycwsi7Gam5qA1nSftorynSQPHpq2WkvJx+7s9TMlzMzGUjVJRMTPgZ9LuiQiHqljTDXXkiaJLicJM7MxZem4Lkm6CFheuf90vuN6djq6qdNPpzMzG1OWJPED4ELg68B+0Yify4nWUoHO3v3i45iZ1UyWJDEQEV+reSR11lLKu7nJzGwcWYbA/rekd0laLGlB+VXzyGqspVSgs89JwsxsLFlqEmen7x+uKAvgsMkPp35aSwX3SZiZjSPL8yQOrUcg9TZ3VpEdfoSpmdmYxm1ukjRb0sfTEU5IWinpj2ofWm3Nn93Etu6+qQ7DzKyhZemT+CbQR3L3NcBjwGdrFlGdLGhpYmunk4SZ2ViyJIkVEfF50inDI6IbUE2jqoOFLU109A7Q62dKmJlVlSVJ9EmaRdJZjaQVQG9No6qD+S1NAGzrcr+EmVk1WZLEJ4AfA0slXQr8lOSZ19PavNlFAHb2OEmYmVWTZXTTdZJuBY4naWb6q4jYUvPIaqw8f1OHh8GamVWVZXTTH5Pcdf0/EXEVMCDpdTWPrMbaPBOsmdm4MjU3pU+mAyAitpM0QY1L0qmS1kpaJ+mjY+z3fySFpFVZzjsZWpvTJOGahJlZVVmSxGj7jNtMJSkPfAU4DTgSOEvSkaPs1wb8FfDbDLFMmlZPF25mNq4sSWK1pC9JWpG+vgSsyXDcscC6iHgwIvqA7wFnjLLfZ4DPAT2Zo54EbaWk47rDScLMrKosSeI9JDfTfZ/ki74HOD/DcQcD6yvWN6RlwyQdAyyNiP8Z60SSzpO0WtLqzZs3Z7j0+FpKfqaEmdl4xmw2SpuMroqIkyf7wpJywJeAc8bbNyIuAi4CWLVqVUzG9Qv5HG2lgqfmMDMbw5g1iYgYBIYkzd2Lcz8GLK1YX5KWlbUBzwV+JulhkiG2V9az8/oZc5vZuGNXvS5nZjbtZJkqvBO4U9J1QFe5MCLeO85xtwArJR1KkhzOBN5YcfwOYFF5XdLPgA9FxOrM0e+jxfNm8fj2unaFmJlNK1mSxBXpa0IiYkDSu4FrgTzwjYi4W9KngdURceVEzznZDp43izUPb6Wjp5+25uJUh2Nm1nCy3HH9rXTupmURsXYiJ4+Iq4GrR5RdUGXfkyZy7snwmuct5rKbH+WGtZt57dEH1fvyZmYNL8sd168BfkcyfxOSni9pymsBk+G5S5Kulid3uMnJzGw0WYbAfpLknoftABHxO6b5o0vL2koFZhXzPLnTScLMbDRZkkR/5bQcqaFaBFNvkjhgToknO6b9zOdmZjWRpeP6bklvBPKSVgLvBX5V27DqZ9mC2azb1DnVYZiZNaSsd1w/h+RBQ98FdgDvq2FMdfXC5Qu494md7Oj2cyXMzEaqWpOQ1Ay8AzgcuBM4ISL2uzksjjt0ARFw88NbeeWRB051OGZmDWWsmsS3gFUkCeI04It1iajOjl46j1Ihx28efGqqQzEzazhj9UkcGRFHAUi6GLi5PiHVV3MxzwuWzeO3DzlJmJmNNFZNYriRfn9sZqp0zLL53Luxg76B/WLQlpnZpBmrJnG0pJ3psoBZ6bqAiIg5NY+uTo54RhsDQ8FDW7o44hltUx2OmVnDqJokIiJfz0Cm0jMPTBLD2ic7nCTMzCpkGQK73zusvYV8Ttz/ZMdUh2Jm1lCcJIBSIc+hi1q4Y8PIG8vNzGY2J4nUKc8+gBvXbWFbl59UZ2ZW5iSROnFlO4NDwZ2PuTZhZlbmJJF6zkHJtOG3PrptiiMxM2scThKpubOLHHfoAn6wegODQzHV4ZiZNQQniQpnv2g5j23fxXd+/fBUh2Jm1hCcJCq86jnP4NjlC/jqzx5gV9/gVIdjZjblnCQq5HPiHScdxqaOXr76s3VTHY6Z2ZRzkhjh5c86kJeuXMTlq9ezpdNPrDOzmc1JYhQfedWz2N7dz6lf/iW/37hz/APMzPZTThKjOGrJXC4773iGIvjL76zhKdcozGyGcpKo4phl8/nb1z2XR7d2c8F/3U1PvzuyzWzmqWmSkHSqpLWS1kn66CjbPyDpHkl3SPqppENqGc9EnXbUYj78qiP4nzs3csrf/5wfrtkw1SGZmdVVzZKEpDzwFZJHnx4JnCXpyBG73QasiojnAT8EPl+rePbW+ScfzqXnHkdbc4EP/eB2/vybN/PL+zcT4RvuzGz/V8uaxLHAuoh4MCL6gO8BZ1TuEBE3RER3uvobYEkN49lrLz58EVe++yW8++TDuWfjTt5y8c2c881buObOjfQOuBnKzPZfYz2Zbl8dDKyvWN8AHDfG/m8Drhltg6TzgPMAli1bNlnxTUhTIceHXnUE7znlcL7+y4f45k0P8c5LN9OUz3HcYQt47dEHcfxhC1m6YPaUxGdmVgu1TBKZSXozsAo4cbTtEXERcBHAqlWrprSdp1TIc/7Jh/OOE1fws7Wb+PUDT3HtPU/w4R/eAcARB7Zx4hHtnLBiIccsm8/cWcWpDNfMbJ/UMkk8BiytWF+Slu1B0iuAvwFOjIhpM9Y0nxOnPPtATnn2gXzs9Gdz35Md3Hj/Fm5Yu4lv3vQQF/3iQXJKHo167KELeP7SeTxvyVwOW9RKLqepDt/MLBPVqgNWUgG4DziFJDncArwxIu6u2OcFJB3Wp0bE/VnOu2rVqli9enUNIp48Xb0D3ProNm55eBu3PbqNNY9sozudC2pRaxMrD2jjqCVzOby9lSOe0caKA1ppLTVEpc7M9lOS1kTEqokeV7NvpogYkPRu4FogD3wjIu6W9GlgdURcCXwBaAV+IAng0Yh4ba1iqpeWUoGXrmznpSvbARgcCh7Y3Mnv1m/nNw88xQNburj4xof2mJL8oLnNrDiglRXtrSxbMJtDFiavJfNn01zMT9VHMbMZrmY1iVqZDjWJLAYGh1i/bRf3PdnBuk2drNvUyf2bOnhwc9dwraNs8dxmVrS3snTBbFa0t3DIwhYOmtfMknmzmTOrQJpgzcyqariahI2tkM9x6KIWDl3Uwques7s8Itja1ccjW7tZv7WbR57q5uEtXTywuZMf37WRbd39e5ynpSnPQfNmsXjeLA6e18xBc2cl63ObOXBuMwfOaXZTlpntNX97NBhJLGwtsbC1xDHL5j9t+9auPtZv7ebx7bt4LH1t3N7D4zt2cc/jO9jS2fe0Y1qa8hwwp5mFLU0cMKfEQXNncfD8JJkcNHcWB8wpsaCliWLes7SY2Z6cJKaZBS1NLGhp4uil80bd3tM/yMYdPWzcvotNHb08ubOHJ3f2sqmjh6c6+7h3YwfX37uJnv6hPY6TYP7sJha2NLGwtYn2tmbaW0ssamuivbVEe1uJRa0l5s0uMn92E7Ob8m7mMpsBnCT2M83F/HAzVjXlJq3H0xrI5o5etnT2srmjl6c6+3iqq5e7HtvB5o5eOnsHRj1HUyHH/DRhzBt+b2JBy+7l+bOLw+8LWpqY01z08F+zacZJYgaqbNI6asncMfft7htgS0cfmzt7eaqzl+3d/Wzt7mNbdx/bu/qT9+5+7t/UyfbuPrZ19+8xaqtSTjB31p6JZX5LZTLZvbygoryp4GYws6niJGFjmt1UYNnCAssWZptuJCLo6B0YTiBbu/uS5NHVP5xEymUbd/Tw+4072dbdz64xpmJvaconSSStpZSTydzZTcydVeTAOSXmzioyd1aROc3Je1tzgYL7WMz2mZOETSpJzGlOvqyzJhZI+lK2jUgmyXqyvD2tvWzr7mf91m62dvWxs2f0prCy1lKBOc0F5swqMidNIgtmNzGvZXcymTOryJzmwvDyvPTdnfhmCScJawjNxTyL585i8dxZmY8ZHAp27upnU0cvO3v62dHdn7zv6mfnroHkPV3fsStJLnds2M62rn76BofGPHc5wbQ2F2gtFWhtLtJWKi8XaCkVkvXm3WXl9ZamAm3pPk42Nt05Sdi0lc8p6dNoaZrwsT39g+zc9fSkUn5t707eu3oH6OobYOeufh7fvovOngE6eweqduiP1FzM0VpKmr9aSwVaSvk91stJZvf20ZJPkeZizqPJbEo4SdiM1FzM01xM7h/ZG0NDQVdfkiy6egfoKCePngE60vfhbRXrnT0DPLZ9F529/XT1DtLR00//4PizHuRzoqUpT1tzcY/kUq7BtJT2TDaVyWd4W6lISynvvhqbECcJs72Qy4m25iJtzfs+FXzvwOBwEumoSC6dI5JPeb28bfuufjZs6x7e3tWX7QFYs4r53UmmVFGDad5d02kt5Yeb2VpLeVpKBWY3Jdtbmgrpep5SwTWc/Z2ThNkUKxXylFrzLGwt7dN5Kms3I2s0lc1k5WTT0ZM2p/UO7lXtBqCQ03BNZXZTfni5pVS5XKAl3TZqWVOB2WnycbNa43GSMNtP7FG7Gfv2l3H1DgzS1TuYJpt+uvsG6eodGH5P+mqevlyuBZVvxOzqG6C7d3DcgQJlEmlNpSLJpOuzm0YmnKSsnKCGk09FkmppKpD3DZz7xEnCzJ6mVMhTKuRZsBeDAkbTNzC0O4n0DQzXYLr7kveuvt3r5USzu3yAx7f3p9uSfUbOlDyWWcVywhmjNpNur2xSK+/TWjGoYHYxP+NmDXCSMLOaayrkaCrs3Ui00QwNBd39g3SPUovpLCeYioTU2bu71tPZO8Cmjh66tgwOH5M16VTWdMr9Oc3FPLObdieYOc2777+ZM6s4PNigrbmQbivQ1lycNjUcJwkzm3ZyOQ1/SU+GwaFgV5p0OtOkUR4k0NW3e8BAebRauabT2TvArr5BNnf20t3XTVfvADt3DYw5g0BZW3oTZ+X0NOUZBRa07J6eZl4691l7a2lKRqY5SZjZjJevSDoHTML5+gaG6OjpZ2fPwHC/TjJYYPf9ODvT9/JMAg9v6WJbdx8dVWYSaGsu8N6Xr+TtLztsEiLMzknCzGySNRVyw5NoTlT/4BDb95iWpo+tXf38bv02Fs/bu/t69oWThJlZAynmc7S3Jc9wqfTG45ZNSTy+9dLMzKpykjAzs6qcJMzMrConCTMzq8pJwszMqnKSMDOzqpwkzMysKicJMzOrShHZ5o1vFJI2A4/s5eGLgC2TGM5kc3x7r5FjA8e3rxo5vkaODXbHd0hEtE/04GmXJPaFpNURsWqq46jG8e29Ro4NHN++auT4Gjk22Pf43NxkZmZVOUmYmVlVMy1JXDTVAYzD8e29Ro4NHN++auT4Gjk22Mf4ZlSfhJmZTcxMq0mYmdkEOEmYmVlVMyZJSDpV0lpJ6yR9dIpi+IakTZLuqihbIOk6Sfen7/PTckn6pzTeOyQdU+PYlkq6QdI9ku6W9FcNFl+zpJsl3Z7G96m0/FBJv03j+L6kprS8lK6vS7cvr2V86TXzkm6TdFUDxvawpDsl/U7S6rSsIX626TXnSfqhpHsl/V7SCY0Sn6Qj0n+38munpPc1UHzvT/9P3CXpsvT/yuT97kXEfv8C8sADwGFAE3A7cOQUxPEy4BjgroqyzwMfTZc/CnwuXT4duAYQcDzw2xrHthg4Jl1uA+4Djmyg+AS0pstF4LfpdS8HzkzLLwTemS6/C7gwXT4T+H4dfr4fAL4LXJWuN1JsDwOLRpQ1xM82vea3gHPT5SZgXiPFVxFnHngCOKQR4gMOBh4CZlX8zp0zmb97dfmHneoXcAJwbcX6x4CPTVEsy9kzSawFFqfLi4G16fK/AmeNtl+d4vwv4JWNGB8wG7gVOI7kTtLCyJ8zcC1wQrpcSPdTDWNaAvwUeDlwVfoF0RCxpdd5mKcniYb42QJz0y86NWJ8I2L6Q+CmRomPJEmsBxakv0tXAa+azN+9mdLcVP6HLNuQljWCAyNiY7r8BHBgujxlMadV0BeQ/LXeMPGlzTm/AzYB15HUDrdHxMAoMQzHl27fASysYXhfBj4CDKXrCxsoNoAAfiJpjaTz0rJG+dkeCmwGvpk2131dUksDxVfpTOCydHnK44uIx4AvAo8CG0l+l9Ywib97MyVJTAuRpPcpHZMsqRX4D+B9EbGzcttUxxcRgxHxfJK/2o8FnjVVsVSS9EfApohYM9WxjOElEXEMcBpwvqSXVW6c4p9tgaQZ9msR8QKgi6T5ZthU/+4BpO36rwV+MHLbVMWX9oOcQZJoDwJagFMn8xozJUk8BiytWF+SljWCJyUtBkjfN6XldY9ZUpEkQVwaEVc0WnxlEbEduIGkGj1PUmGUGIbjS7fPBZ6qUUgvBl4r6WHgeyRNTv/YILEBw39xEhGbgB+RJNlG+dluADZExG/T9R+SJI1Gia/sNODWiHgyXW+E+F4BPBQRmyOiH7iC5Pdx0n73ZkqSuAVYmfb4N5FUGa+c4pjKrgTOTpfPJukLKJe/NR0pcTywo6JqO+kkCbgY+H1EfKkB42uXNC9dnkXSX/J7kmTx+irxleN+PXB9+tfepIuIj0XEkohYTvK7dX1EvKkRYgOQ1CKprbxM0q5+Fw3ys42IJ4D1ko5Ii04B7mmU+Cqcxe6mpnIcUx3fo8Dxkman/4fL/3aT97tXj86eRniRjDi4j6Qd+2+mKIbLSNoN+0n+enobSXvgT4H7gf8FFqT7CvhKGu+dwKoax/YSkuryHcDv0tfpDRTf84Db0vjuAi5Iyw8DbgbWkTQDlNLy5nR9Xbr9sDr9jE9i9+imhogtjeP29HV3+fe/UX626TWfD6xOf77/CcxvsPhaSP7inltR1hDxAZ8C7k3/X3wHKE3m756n5TAzs6pmSnOTmZntBScJMzOryknCzMyqcpIwM7OqnCTMzKwqJwnbL0l6hqTvSXognYriaknPnOq49pakkyS9aKrjsJnHScL2O+lNRT8CfhYRKyLiD0gmdTxw7CMb2kmAk4TVnZOE7Y9OBvoj4sJyQUTcDtwo6QvpvPt3SnoDDP+V/nNJ/yXpQUl/J+lNSp5fcaekFel+l0i6UNJqSfelczaVn3XxzXTf2ySdnJafI+kKST9W8syBz5fjkfSHkn4t6VZJP0jnzCo/9+FTafmdkp6VTrj4DuD9Sp5n8FJJf5p+jtsl/aJO/642AxXG38Vs2nkuyUyYI/0JyZ29RwOLgFsqvmCPBp4NbAUeBL4eEccqefjSe4D3pfstJ5n3aAVwg6TDgfNJ5ng7StKzSGZbLTdtPZ9kRt1eYK2kfwZ2AR8HXhERXZL+muRZFJ9Oj9kSEcdIehfwoYg4V9KFQGdEfBFA0p3AqyLisfJ0JWa14JqEzSQvAS6LZDbZJ4GfAy9Mt90SERsjopdkOoWfpOV3kiSGsssjYigi7idJJs9Kz/vvABFxL/AIUE4SP42IHRHRQzKnziEkD6I5ErhJydTnZ6flZeXJFdeMuHalm4BLJL2d5EE4ZjXhmoTtj+5m9+RmWfVWLA9VrA+x5/+TkfPYjDevTeV5B9NzCbguIs4a55jy/k8TEe+QdBzwamCNpD+IiJrOJGszk2sStj+6Hihp98N1kPQ8YDvwBiUPL2oneZzszRM8959KyqX9FIeRPHXsl8Cb0us8E1iWllfzG+DFaVNVeZbW8UZedZA8Vrb8eVZExG8j4gKSB/YsrXqk2T5wTcL2OxERkv4Y+HLa3t9D8vjO9wGtJLOhBvCRiHgi7UfI6lGSxDIHeEdE9Ej6KvC1tJ9gADgnInqTQVajxrdZ0jnAZZJKafHHSWYprua/gR9KOoOkj+T9klaS1Ep+mn4ms0nnWWDNMpJ0Cck04D+c6ljM6sXNTWZmVpVrEmZmVpVrEmZmVpWThJmZVeUkYWZmVTlJmJlZVU4SZmZW1f8HcA8GrOOXiZsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pipe_PCA = Pipeline([\n",
    "        ('scaler', StandardScaler()),\n",
    "        ('dim_red', PCA(n_components = 768))])\n",
    "pipe_PCA.fit(features)\n",
    "%matplotlib inline\n",
    "plt.plot(pipe_PCA.get_params()['steps'][1][1].explained_variance_ratio_ * 100)\n",
    "plt.title('Explained Variance')\n",
    "plt.xlabel('Components')\n",
    "plt.ylabel('Percentage of Variance')\n",
    "#plt.savefig('./output/pca_variance.pdf')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start Validation...\n",
      "\n",
      "Mean Squared Error mean and variance 0.12726583751465803 7.314085693343502e-06\n",
      "Mean Absolute Error mean and variance 0.2647714426842764 6.010623667684166e-06\n",
      "Median Absolute Error mean and variance 0.20662328671238353 6.147080624115529e-06\n",
      "R2 mean and variance 0.5799225189834 6.434772412392846e-05 \n",
      "\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "\n",
    "print('Start Validation...\\n')\n",
    "#creating pipeline\n",
    "pipe = Pipeline([\n",
    "        ('scaler', StandardScaler()),\n",
    "        ('dim_red', PCA(n_components = 40)),\n",
    "        ('regressor', MLPRegressor(hidden_layer_sizes=(50, 25), max_iter=500, activation='logistic', verbose=False, random_state = 1, learning_rate_init = 0.0001, tol = 0.00001, alpha = 0.1))\n",
    "        ])\n",
    "\n",
    "#Perform cross-validation\n",
    "rkf = RepeatedKFold(n_splits=5, n_repeats=5)\n",
    "\n",
    "#initializing and running the cross validator with pipe, features, labels, scores, and kfold object\n",
    "scores = cross_validate(pipe, features, target, cv=rkf, scoring=('neg_mean_squared_error', 'neg_mean_absolute_error', 'neg_median_absolute_error', 'r2'),return_train_score=True)\n",
    "\n",
    "#print(scores,'\\n')\n",
    "print('Mean Squared Error mean and variance', -np.mean(scores['test_neg_mean_squared_error']),np.var(scores['test_neg_mean_squared_error']))\n",
    "print('Mean Absolute Error mean and variance', -np.mean(scores['test_neg_mean_absolute_error']),np.var(scores['test_neg_mean_absolute_error']))\n",
    "print('Median Absolute Error mean and variance', -np.mean(scores['test_neg_median_absolute_error']),np.var(scores['test_neg_median_absolute_error']))\n",
    "print('R2 mean and variance', np.mean(scores['test_r2']),np.var(scores['test_r2']),'\\n')\n",
    "\n",
    "print('Done!')\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('scaler', StandardScaler()), ('dim_red', PCA(n_components=40)),\n",
       "                ('regressor',\n",
       "                 MLPRegressor(activation='logistic', alpha=0.1,\n",
       "                              hidden_layer_sizes=(50, 25),\n",
       "                              learning_rate_init=0.0001, max_iter=500,\n",
       "                              random_state=1, tol=1e-05))])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#training the pipeline\n",
    "pipe.fit(features, target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAknklEQVR4nO3de5hddX3v8fdn77llJvcw5B4SIAihBaQhiEXsQeSiPaBWCtgL9HgO1h4srVVLa4sWj+fx0nrUc6glbfGOHMVq0UaB4g2taEKESAiBEEJIuCSQ+21m9uxv/1hrZnZ2ZnZ2MnvNnsz+vJ5nP7PWb132d2Un88lvrb1+SxGBmZnZUHL1LsDMzEY3B4WZmVXkoDAzs4ocFGZmVpGDwszMKnJQmJlZRQ4KMzOryEFhDUHSBkkX1em9l0haJmmHpG2Sfi7pD+pRi9nRcFCYZUjSecD3gB8CJwPTgHcClx3l/vK1q86sOg4Ka2iSWiV9UtJz6euTklrTZcdJ+nZJT+ABSbl02Z9L2ixpt6S1kl43xFt8HPh8RHw0Il6KxEMR8dvpfq6T9OOymkLSyen05yR9Ju2R7AXeI+mF0sCQ9GZJq9LpnKSbJD0l6WVJX5U0teZ/cNZQHBTW6N4PvAo4CzgTWAL8Vbrsz4BNQCcwHfhLICS9ArgBOCciJgCXABvKdyypHTgPuGuYNb4N+DAwAfgUsBe4sGz5Hen0u4A3Aa8FZgHbgVuH+f7W4BwU1uh+B7glIrZExFbgb4DfS5f1ADOBEyKiJyIeiGRwtF6gFVgkqTkiNkTEU4PsewrJv7Hnh1njv0bETyKiGBEHgK8A1wBImgC8IW0D+EPg/RGxKSK6gA8Cb5XUNMwarIE5KKzRzQKeKZl/Jm2D5LTROuBeSesl3QQQEeuAPyH5JbxF0p2SZnGo7UCRJGyG49my+TuAt6SnyN4CrIyIvmM4AfhGerpsB7CGJNimD7MGa2AOCmt0z5H8cu0zL20jInZHxJ9FxInA5cC7+65FRMQdEXF+um0AHy3fcUTsA34K/FaF998LtPfNSJoxyDoHDfEcEY+RBNplHHzaCZJQuSwiJpe82iJic4UazCpyUFgjaZbUVvJqIjll81eSOiUdB9wMfAlA0m9KOlmSgJ0k/zMvSnqFpAvT/9EfAPaT9BwG8z7gOknvlTQt3e+Zku5Mlz8CnC7pLEltJL2UatwB3AhcAHytpP0fgA9LOiF9r05JV1S5T7NBOSiskSwj+aXe9/og8L+AFcAq4JfAyrQNYCHw78Aekp7B30fE90muT3wEeAl4ATge+IvB3jAi/oPkwvOFwHpJ24ClaS1ExBPALen7PAn8eLD9DOIrJBesvxcRL5W0fwq4m+R02W7gQeDcKvdpNij5wUVmZlaJexRmZlaRg8LMzCpyUJiZWUUOCjMzq2jM3K153HHHxfz58+tdhpnZMeWhhx56KSI6K60zZoJi/vz5rFixot5lmJkdUyQ9c7h1fOrJzMwqclCYmVlFDgozM6vIQWFmZhU5KMzMrCIHhZmZVeSgMDOziho+KPZ0FfjEfU/wi43b612Kmdmo1PBB0VMo8un7n+SRZ3fUuxQzs1Ep06CQdKmktZLW9T1vuGz5uyU9JmmVpPv7nsqVLrtW0pPp69qsamxtTv4IDhSGekCZmVljyywoJOWBW0me67sIuEbSorLVfgEsjogzgLuAj6XbTgU+QPJkriXAByRNyaLO1qY8AF09Dgozs8Fk2aNYAqyLiPUR0Q3cCRz07N6I+H76AHpIHtk4J52+BLgvIrZFxHbgPuDSLIrM50RzXnQVerPYvZnZMS/LoJgNPFsyvyltG8rbge8cybaSrpe0QtKKrVu3HnWhrU15DrhHYWY2qFFxMVvS7wKLgY8fyXYRsTQiFkfE4s7OiqPkVtTalHOPwsxsCFkGxWZgbsn8nLTtIJIuAt4PXB4RXUeyba20Nefp8sVsM7NBZRkUy4GFkhZIagGuBu4uXUHSK4HbSEJiS8mie4CLJU1JL2JfnLZlorUpx4Ee9yjMzAaT2YOLIqIg6QaSX/B54PaIWC3pFmBFRNxNcqppPPA1SQAbI+LyiNgm6UMkYQNwS0Rsy6rWlqacexRmZkPI9Al3EbEMWFbWdnPJ9EUVtr0duD276gb41JOZ2dBGxcXsevOpJzOzoTkogFb3KMzMhuSgANqacnS5R2FmNigHBe5RmJlV4qDAPQozs0ocFCQjyLpHYWY2OAcFfWM9uUdhZjYYBwXQ5h6FmdmQHBQkPYpCMSj0OizMzMo5KEhuuAPcqzAzG4SDgmQID3BQmJkNxkHBQI/CF7TNzA7loCD5eiy4R2FmNhgHBdDW1HfqyT0KM7NyDgoGehR+braZ2aEcFJT0KHyNwszsEA4KfI3CzKwSBwXJDXfgbz2ZmQ3GQUEyhAe4R2FmNhgHBQM9CgeFmdmhHBT4hjszs0ocFCRPuAP3KMzMBpNpUEi6VNJaSesk3TTI8gskrZRUkPTWsmUfk7Ra0hpJn5akrOocGBTQPQozs3KZBYWkPHArcBmwCLhG0qKy1TYC1wF3lG37auDXgTOAXwHOAV6bVa0Dp57cozAzK9eU4b6XAOsiYj2ApDuBK4DH+laIiA3psvLf0AG0AS2AgGbgxawKlURrU849CjOzQWR56mk28GzJ/Ka07bAi4qfA94Hn09c9EbGmfD1J10taIWnF1q1bh1Vsa1OOLvcozMwOMSovZks6GTgNmEMSLhdKek35ehGxNCIWR8Tizs7OYb1nW3PePQozs0FkGRSbgbkl83PStmq8GXgwIvZExB7gO8B5Na7vIK3N7lGYmQ0my6BYDiyUtEBSC3A1cHeV224EXiupSVIzyYXsQ0491dK45jx7uwtZvoWZ2TEps6CIiAJwA3APyS/5r0bEakm3SLocQNI5kjYBVwK3SVqdbn4X8BTwS+AR4JGI+FZWtQJM62jl5T3dWb6FmdkxKctvPRERy4BlZW03l0wvJzklVb5dL/COLGsr1zmhlUc27RjJtzQzOyaMyovZ9dA5oZWtu7vqXYaZ2ajjoEh1TmhlX3cve7t8ncLMrJSDItU5vhXAvQozszIOilTnhDQo9jgozMxKOShS/UHhHoWZ2UEcFCkHhZnZ4BwUqSntLeRzclCYmZVxUKTyOTGto8VBYWZWxkFRonNCqy9mm5mVcVCU8E13ZmaHclCU6Bzfyou7DtS7DDOzUcVBUWJBZwdbdnexc39PvUsxMxs1HBQlTp0xAYAnXtxd50rMzEYPB0WJU2dMBODxFxwUZmZ9HBQlZk5qY0JbE48/v6vepZiZjRoOihKSOHXGBNa6R2Fm1s9BUeYVaVBERL1LMTMbFRwUZU6dMZHdXQU279hf71LMzEYFB0WZ02YmF7TXPO/TT2Zm4KA4xKkzJiDBY8/5graZGTgoDtHR2sSCaR089vzOepdiZjYqZBoUki6VtFbSOkk3DbL8AkkrJRUkvbVs2TxJ90paI+kxSfOzrLXUabMmsto9CjMzIMOgkJQHbgUuAxYB10haVLbaRuA64I5BdvEF4OMRcRqwBNiSVa3lTp81kU3b93soDzMzsu1RLAHWRcT6iOgG7gSuKF0hIjZExCqgWNqeBkpTRNyXrrcnIvZlWOtBFvVf0Havwswsy6CYDTxbMr8pbavGKcAOSf8i6ReSPp72UA4i6XpJKySt2Lp1aw1KTiyalQSFL2ibmY3ei9lNwGuA9wDnACeSnKI6SEQsjYjFEbG4s7OzZm9+/IQ2jhvfymPuUZiZZRoUm4G5JfNz0rZqbAIeTk9bFYBvAmfXtrzKFs2a6B6FmRlVBIWkj0maKKlZ0v2Stkr63Sr2vRxYKGmBpBbgauDuKutaDkyW1NdNuBB4rMpta+L0WRN5cstuugvFw69sZjaGVdOjuDgidgG/CWwATgbee7iN0p7ADcA9wBrgqxGxWtItki4HkHSOpE3AlcBtklan2/aSnHa6X9IvAQH/eKQHNxyLZk6kpzd4covv0DazxtZ0BOu8EfhaROyUVNXOI2IZsKys7eaS6eUkp6QG2/Y+4Iyq3igDpRe0T581qV5lmJnVXTU9im9Lehz4NZL/4XcCY/7B0vOndTCuOe8b78ys4R02KCLiJuDVwOKI6AH2UnY/xFiUz4lTZkzwY1HNrOFVczH7SqAnInol/RXwJWBW5pWNAqccP54nXtxT7zLMzOqqmlNPfx0RuyWdD1wE/DPwmWzLGh1OmT6Bl/Z0sX1vd71LMTOrm2qCojf9+UZgaUT8G9CSXUmjx8nTxwPw5Bb3KsyscVUTFJsl3QZcBSyT1Frldse8U6ZPAPB1CjNraNX8wv9tknshLomIHcBUqriPYiyYNamNjpY8TzoozKyBVfOtp33AU8Alkm4Ajo+IezOvbBSQxMnTJ/jUk5k1tGq+9XQj8GXg+PT1JUnvyrqw0WLh8eMdFGbW0Ko59fR24NyIuDm9q/pVwP/ItqzR48TODrbu7mL3AT/EyMwaUzVBIQa++UQ6Xd0YHmPAgmkdADzz8og9N8nMbFSpZqynzwI/k/SNdP5NJPdSNIQFnUlQrH9pL78y22M+mVnjOWxQRMQnJP0AOD9t+gPgxSyLGk1OmJoExYaX9ta5EjOz+qimR0FErARW9s1L2gjMy6qo0WRcS55Zk9p42kFhZg3qaG+ca5hrFADzj+twUJhZwzraoIiaVjHKLXBQmFkDG/LUk6T/y+CBIGByVgWNRguO62Dn/h627+1mSkdDDHNlZtav0jWKFUe5bMxZcFxyQfvpl/c6KMys4QwZFBHx+ZEsZDSb3xcUW/dy9rwpda7GzGxkNcQosMM1d0o7+ZzY8LKvU5hZ43FQVKGlKcecKeNY7wvaZtaAHBRVWnBch2+6M7OGdDTfegIgIv74cDuXdCnwKSAP/FNEfKRs+QXAJ4EzgKsj4q6y5ROBx4BvRsQNh3u/LM2f1sHPn95GRCA11G0kZtbgKvUoVgAPAW3A2cCT6essqngUqqQ8cCtwGbAIuEbSorLVNgLXAXcMsZsPAT863HuNhBM7O9jX3cvW3V31LsXMbEQd9ltPkt4JnB8RhXT+H4AHqtj3EmBdRKxPt7sTuIKkh9D3HhvSZcXyjSX9GjAd+C6wuLrDyc78aQODAx4/sa3O1ZiZjZxqrlFMASaWzI9P2w5nNvBsyfymtO2wJOWAvwPec5j1rpe0QtKKrVu3VrPro9Z3L4WvU5hZo6lmUMCPAL+Q9H2Su7IvAD6YZVHAHwHLImJTpesBEbEUWAqwePHiTIcVmTV5HE05sXGbn0thZo2lmmHGPyvpO8C5adOfR8QLVex7MzC3ZH5O2laN84DXSPojkh5Mi6Q9EXFTldvXXD4n5kwZ56Aws4ZTzTOzBVwEnBkR/0ryS3tJFfteDiyUtEBSC3A1cHc1RUXE70TEvIiYT3L66Qv1DIk+c6e286yDwswaTDXXKP6e5H/416Tzu0m+zVRRevH7BuAeYA3w1YhYLekWSZcDSDpH0ibgSuA2SauP4hhGzLyp7e5RmFnDqeYaxbkRcbakXwBExPa0h3BYEbEMWFbWdnPJ9HKSU1KV9vE54HPVvF/W5k1tZ/u+HnYd6GFiW3O9yzEzGxHV9Ch60nsiAkBSJ3DI11kbwbyp7QBsfNm9CjNrHNUExaeBbwDHS/ow8GPgf2da1Sg1Nw0KX6cws0ZS8dRTej/D08D7gNeRfD32TRGxZgRqG3XmTUt7FA4KM2sgFYMiIoqSbo2IVwKPj1BNo9bEtmamtDc7KMysoVRz6ul+Sb8lj4QH+JtPZtZ4qgmKdwBfA7ok7ZK0W9KujOsatXwvhZk1msMGRURMiIhcRLRExMR0fuLhthur5k1tZ9P2/fQWMx0xxMxs1KjmPgokTQEWkgw5DkBEjIrhv0favKntFIrB8zv3M2dKe73LMTPL3GGDQtJ/B24kuTHuYeBVwE+BCzOtbJTqv5di2z4HhZk1hGquUdwInAM8ExH/BXglsCPLokazub7pzswaTDVBcSAiDgBIao2Ix4FXZFvW6DVzUpuHGzezhlLNNYpNkiYD3wTuk7QdeCbLokazpnzOw42bWUOp5nkUb04nP5g+vGgSyeNJG5a/ImtmjaSa51HM63uRDOfxMDAj68JGM990Z2aNpJpTT/9GMnKsSL4euwBYC5yeYV2jmocbN7NGUs2pp18tnZd0NskzrRvWvJJRZE+fNanO1ZiZZauabz0dJCJWMvD87Ibk4cbNrJFUc8Pdu0tmc8DZwHOZVXQM8HDjZtZIqrlGMaFkukByzeLr2ZRzbJjY1szk9mae8U13ZtYAqrlG8TcjUcixxt98MrNGUc2pp2+RPi97MBFxeU0rOkbMm9rOo5t31rsMM7PMVXPqaT3JfRNfSuevAV4kuVO7Yc2b2s53H32B3mKQz/mZTmY2dlXzradfj4irIuJb6ettwGsi4ocR8cNKG0q6VNJaSesk3TTI8gskrZRUkPTWkvazJP1U0mpJqyRddeSHlq3S4cbNzMayaoKiQ9KJfTOSFgAdh9tIUh64FbgMWARcI2lR2WobgeuAO8ra9wG/HxGnA5cCn0zHmxo1SocbNzMby6o59fSnwA8krSe5O/sE4PoqtlsCrIuI9QCS7gSuAB7rWyEiNqTLiqUbRsQTJdPPSdoCdDKKhjc/6F6Kk+pcjJlZhqr51tN3JS0ETk2bHo+Irir2PRt4tmR+E0dxo56kJUAL8NQgy64nDa158+Yd6a6HxcONm1mjGPLUk6RzJM0ASIPhTOAW4OOSpo5EcZJmAl8E/iAiiuXLI2JpRCyOiMWdnZ0jUVK/pnyO2VPG+V4KMxvzKl2juA3ohuSiM/AR4AvATmBpFfveDMwtmZ+TtlVF0kSSm/veHxEPVrvdSJrn4cbNrAFUCop8RGxLp68ClkbE1yPir4GTq9j3cmChpAWSWoCrgburKSpd/xvAFyLirmq2qYe5vunOzBpAxaCQ1HcN43XA90qWVXNtowDcANwDrAG+GhGrJd0i6XLoP721CbgSuE3S6nTz3wYuAK6T9HD6OutIDmwknFAy3LiZ2VhV6Rf+V4AfSnoJ2A88ACDpZJLTT4cVEcuAZWVtN5dMLyc5JVW+3ZcYuMFv1PJw42bWCIYMioj4sKT7gZnAvRHRN4xHDnjXSBQ32s11UJhZA6h4Cmmwi8il9zg0Og83bmaN4IgfXGQD+oYbd1CY2VjmoBimZLhxj/dkZmOXg2KY5k5tZ+PLe+tdhplZZhwUwzRvajubtu+ntzjkIzvMzI5pDoph8nDjZjbWOSiG6QQPN25mY5yDYpgOGm7czGwMclAMk4cbN7OxzkExTH3DjW94yUFhZmOTg6IGTuocz1Nb99S7DDOzTDgoauDk48ez/qW9/oqsmY1JDooaOLlzPN2FIpu2+/STmY09DooaOOn4DgDWbfHpJzMbexwUNXBy5wTAQWFmY5ODogYmtTdz3PhWX9A2szHJQVEjJ3V2uEdhZmOSg6JGFk4fz5Nb9jDwIEAzs7HBQVEjp86YyO4DBZ7beaDepZiZ1ZSDokZOm5lc0H78+V11rsTMrLYcFDVyyvQ0KF7YXedKzMxqK9OgkHSppLWS1km6aZDlF0haKakg6a1ly66V9GT6ujbLOmthQlszc6aMY417FGY2xmQWFJLywK3AZcAi4BpJi8pW2whcB9xRtu1U4APAucAS4AOSpmRVa62cOmOiexRmNuZk2aNYAqyLiPUR0Q3cCVxRukJEbIiIVUCxbNtLgPsiYltEbAfuAy7NsNaaOG3mBNZv3cOBnt56l2JmVjNZBsVs4NmS+U1pW822lXS9pBWSVmzduvWoC62VRTMnUgxfpzCzseWYvpgdEUsjYnFELO7s7Kx3OfzqnEkArNq0o76FmJnVUJZBsRmYWzI/J23Letu6mT15HNM6Wnjk2Z31LsXMrGayDIrlwEJJCyS1AFcDd1e57T3AxZKmpBexL07bRjVJnDFnEr/cvKPepZiZ1UxmQRERBeAGkl/wa4CvRsRqSbdIuhxA0jmSNgFXArdJWp1uuw34EEnYLAduSdtGvTPmTGbdlj3s7SrUuxQzs5poynLnEbEMWFbWdnPJ9HKS00qDbXs7cHuW9WXhzLmTKAY8unkn5544rd7lmJkN2zF9MXs0OnPOZABWbtxR1zrMzGrFQVFj08a3clJnB8s3HBNnyszMDstBkYElC6ayfMM2eosectzMjn0OigwsWTCV3QcKrPWNd2Y2BjgoMrBkQXIR++dPv1znSszMhs9BkYHZk8cxe/I4frreQWFmxz4HRUYuOKWTn6x7mZ7e8vEOzcyOLQ6KjPzGKzrZ01XgoWe217sUM7NhcVBk5NUnTaMpJ36wtv6j2pqZDYeDIiMT2ppZPH8KP1i7pd6lmJkNi4MiQxcvmsHjL+xm3ZY99S7FzOyoOSgy9MYzZiLBt1c9V+9SzMyOmoMiQ9MntnHugql865HniPBd2mZ2bHJQZOy/njmLp7buZdUmP8zIzI5NDoqMXX7mLNpb8nzxwWfqXYqZ2VFxUGRsQlszb37lbL71yHNs39td73LMzI6Yg2IE/N55J9BVKLpXYWbHJAfFCDh1xkRev2g6//jAenbu66l3OWZmR8RBMULe/fpT2H2gwGd++FS9SzEzOyIOihFy2syJvOXs2fzTA+tZ8/yuepdjZlY1B8UI+us3LmLSuGbe87VHONDTW+9yzMyqkmlQSLpU0lpJ6yTdNMjyVkn/P13+M0nz0/ZmSZ+X9EtJayT9RZZ1jpQpHS189LfO4LHnd/G+u1ZR9KNSzewYkFlQSMoDtwKXAYuAayQtKlvt7cD2iDgZ+D/AR9P2K4HWiPhV4NeAd/SFyLHuokXTee8lr+DuR57jvXetouDnVZjZKJdlj2IJsC4i1kdEN3AncEXZOlcAn0+n7wJeJ0lAAB2SmoBxQDcwZk7sv/O1J/Hu15/C11du4qqlD/LMy3vrXZKZ2ZCyDIrZwLMl85vStkHXiYgCsBOYRhIae4HngY3A30bEtgxrHVGS+OPXLeRTV5/F2hd28/pP/IgPffsxNu/YX+/SzMwO0VTvAoawBOgFZgFTgAck/XtErC9dSdL1wPUA8+bNG/Eih+uKs2Zz7oJp/O29a/ncf2zg9p88zSvnTuaiRdM5d8E0Tps5gfaW0foRmVmjyPK30GZgbsn8nLRtsHU2paeZJgEvA28DvhsRPcAWST8BFgMHBUVELAWWAixevPiYvDI8Y1Ibf3vlmfzJRQv5+kObuf/xF/nYd9cCIMH8aR3MmTKO2ZPHMWty8nPmpDamjW9lakcLU9qbacr7y2tmlp0sg2I5sFDSApJAuJokAErdDVwL/BR4K/C9iAhJG4ELgS9K6gBeBXwyw1rrbs6Udm68aCE3XrSQLbsO8PCzO1j93C6e3LKbzdv3s+b5Xby0Z/CxoiaNa2ZaRwtTB3lNGtc88GpvZvK4pK2tOUdyOcjMrLLMgiIiCpJuAO4B8sDtEbFa0i3Aioi4G/hnkjBYB2wjCRNIvi31WUmrAQGfjYhVWdU62hw/sY2LT5/BxafPOKj9QE8vz+3Yzws7D/Dy3m627+vm5T3dbNs78Hrm5X2s3LiD7fu66a3w9duWfI6J45qZNK6Jye1lgVLy6mhtoqM1T3tLE+0teTpammhvTX46bMwag8bKA3UWL14cK1asqHcZo0ZEsOtAgV37e9ixr4ed+wdeO/Z3s3N/D7v65kuX7+thd1ehqveQSIKjJU9H66FBclB7axPjmvP9odP/s3T91jztzXmfSjMbQZIeiojFldbxldIxSlJ/r2Du1CPbttBbZPeBAjv397C3u8C+7l72dhXY393L3u5e9nUX2NtV9rO7l31dBfZ2F9i2t5tnt+3r325fdy+FI7i5sLUp1x8wbc15WvI5WpqSV2tTjpZ8jtbm3EHtLfn8Qeu09reXrpOjtWR/g61T2ubeklnCQWGHaMrnmNLRwpSOlprts7tQLAuUkp+DBk+BfV29HCj00l0o0lUo0l1IAqy7UKS7N5kfbLpWDg2ZgVBpyudoyYumXI6mvGjJJz+b8jmac6I5n6zTnK7TnO9rS3/mNLCPdL45nytZZ2DfzfkcOYmmnMinr77l+dzAfvv22ZQbWNdhZ7XgoLARkfzCbWFye7bvUyxGEhyl4ZEGSFdPke7e3v7QKQ+ZrtJ1D1qnN912oK2nGBR6i/T0FtnfE/T0Fin0Bj3F9GdvkZ7eoHDQfJGRHrWlOZ8GSy5HLg2PnEgCR+pvy2sghHLpdK4kcPISTem+mtJ1kvkceUE+lwRU6TY5Je+VywmJ5P363rNkWV9N/e872DrqC76B2tVfc9KDPmiddB85Jcv6a+mvgf628uUqXa9/+UCbSrbtW1+UzY+xgHZQ2JiSy4m2XHLKajQqFgfCpNCbhFppmBSKQXch+VlIw6a3mAROMZJteovRH1TJegOBVCiWbNObBFpvMdl3sRj0RlAM6O1Np9O23mL0778Yfe9ZOl3kQCFZv5Dus+9VKJsuRvJ+Ecm1sr73jP73qfenkL2BsBkkaCD5ik76Q/1hk06n2/et0bdsIETTUE23g2R06v/3trMzOx4HhdkIyuVEay5PawP/y4sIIugPqIOn09CJoFikP6iKfQHXP522962Thl7pOhH0r1ss2V9faBVLlg+sSzof9BYhKFm/WLr80PWT9Q6eP/j9BvaZ/DmU/Hmk80EMtPevM/B+vcWDw7dvYNETpmXbVW/gv65mVg99/4POIUZpx8/K+HuIZmZWkYPCzMwqclCYmVlFDgozM6vIQWFmZhU5KMzMrCIHhZmZVeSgMDOzisbMMOOStgLPDGMXxwEv1aicY4mPu/E06rE36nFD5WM/ISI6K208ZoJiuCStONyY7GORj7vxNOqxN+pxw/CP3aeezMysIgeFmZlV5KAYsLTeBdSJj7vxNOqxN+pxwzCP3dcozMysIvcozMysIgeFmZlV1PBBIelSSWslrZN0U73ryZqkDZJ+KelhSSvStqmS7pP0ZPpzSr3rHC5Jt0vaIunRkrZBj1OJT6d/B1ZJyu6Zkhkb4rg/KGlz+pk/LOkNJcv+Ij3utZIuqU/VwydprqTvS3pM0mpJN6btjfCZD3XstfvcI31sXyO+gDzwFHAi0AI8Aiyqd10ZH/MG4Liyto8BN6XTNwEfrXedNTjOC4CzgUcPd5zAG4DvkDya+FXAz+pdf42P+4PAewZZd1H6d74VWJD+W8jX+xiO8rhnAmen0xOAJ9Lja4TPfKhjr9nn3ug9iiXAuohYHxHdwJ3AFXWuqR6uAD6fTn8eeFP9SqmNiPgRsK2seajjvAL4QiQeBCZLmjkihdbYEMc9lCuAOyOiKyKeBtaR/Js45kTE8xGxMp3eDawBZtMYn/lQxz6UI/7cGz0oZgPPlsxvovIf8FgQwL2SHpJ0fdo2PSKeT6dfAKbXp7TMDXWcjfD34Ib0FMvtJacWx+RxS5oPvBL4GQ32mZcdO9Toc2/0oGhE50fE2cBlwP+UdEHpwkj6pmP+O9ONcpypzwAnAWcBzwN/V9dqMiRpPPB14E8iYlfpsrH+mQ9y7DX73Bs9KDYDc0vm56RtY1ZEbE5/bgG+QdLlfLGv253+3FK/CjM11HGO6b8HEfFiRPRGRBH4RwZOM4yp45bUTPKL8ssR8S9pc0N85oMdey0/90YPiuXAQkkLJLUAVwN317mmzEjqkDShbxq4GHiU5JivTVe7FvjX+lSYuaGO827g99NvwrwK2FlyuuKYV3bu/c0knzkkx321pFZJC4CFwM9Hur5akCTgn4E1EfGJkkVj/jMf6thr+rnX+4p9vV8k3354guTK//vrXU/Gx3oiybcdHgFW9x0vMA24H3gS+Hdgar1rrcGxfoWku91Dcg727UMdJ8k3X25N/w78Elhc7/prfNxfTI9rVfpLYmbJ+u9Pj3stcFm96x/GcZ9PclppFfBw+npDg3zmQx17zT53D+FhZmYVNfqpJzMzOwwHhZmZVeSgMDOzihwUZmZWkYPCzMwqclCYDULSnvTnfElvq/G+/7Js/j9quX+zWnNQmFU2HziioJDUdJhVDgqKiHj1EdZkNqIcFGaVfQR4TTqe/59Kykv6uKTl6WBr7wCQ9BuSHpB0N/BY2vbNdPDF1X0DMEr6CDAu3d+X07a+3ovSfT+q5JkhV5Xs+weS7pL0uKQvp3fjmo2Iw/3Px6zR3UQypv9vAqS/8HdGxDmSWoGfSLo3Xfds4FciGboZ4L9FxDZJ44Dlkr4eETdJuiEizhrkvd5CMoDbmcBx6TY/Spe9EjgdeA74CfDrwI9rfbBmg3GPwuzIXEwyRtDDJEM5TyMZKwfg5yUhAfDHkh4BHiQZhG0hlZ0PfCWSgdxeBH4InFOy702RDPD2MMkpMbMR4R6F2ZER8K6IuOegRuk3gL1l8xcB50XEPkk/ANqG8b5dJdO9+N+ujSD3KMwq203yeMk+9wDvTId1RtIp6Ui85SYB29OQOJXkcZt9evq2L/MAcFV6HaST5LGmx+Rorja2+H8lZpWtAnrTU0ifAz5FctpnZXpBeSuDPzr2u8AfSlpDMkLngyXLlgKrJK2MiN8paf8GcB7J6L4BvC8iXkiDxqxuPHqsmZlV5FNPZmZWkYPCzMwqclCYmVlFDgozM6vIQWFmZhU5KMzMrCIHhZmZVfSfrylC7Vf9kmwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plotting the loss curve over training iteration \n",
    "plt.plot(pipe.get_params()['steps'][2][1].loss_curve_)\n",
    "plt.title('Loss Curve')\n",
    "plt.xlabel('Iteration')\n",
    "plt.ylabel('Squared Loss')\n",
    "#plt.savefig('./output/loss_curve.pdf')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "if save:\n",
    "    #saving the model to file\n",
    "    joblib_file = \"doa_model.pkl\"\n",
    "    joblib.dump(pipe, joblib_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing the Model\n",
    "\n",
    "For this testing a different database is used. The database is separated in training and testing. In the [feature_extraction.ipybn](feature_extraction.ipynb) notebook the testing data is also transformed and saved so that it can be easyly and rapidly retrived for testing purposes. Here I load that data, get a prediction from it, and compare in terms of performance metrics and the delta angle between the actual DOA and the estimated DOA."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading Test Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading features...\n",
      "(3974, 768)\n",
      "(3974, 3)\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "#Load from file\n",
    "print('Loading features...')\n",
    "dataset_test = pd.read_csv('features_test.csv')\n",
    "num_features = 768\n",
    "features_test = dataset_test.iloc[:,0:num_features].to_numpy()\n",
    "target_test = np.zeros((features_test.shape[0],3))\n",
    "target_test[:,0] = dataset_test['x'] \n",
    "target_test[:,1] = dataset_test['y']\n",
    "target_test[:,2] = dataset_test['z']\n",
    "print(features_test.shape)\n",
    "print(target_test.shape)\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Restoring the Model from File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#restoring the classifier model from file\n",
    "restored_model = joblib.load(joblib_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predicting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Predicting from the testing data\n",
    "target_pred = restored_model.predict(features_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance Metrics on Testing Data\n",
    "\n",
    "It shows a plot with the performance of the ANN through the iterations, which converges fast to the best model. Also, this metrics are based on the error per each component (x, y, z) that represents the DOA vector. The last metric considers a different parameter which is the angle between the actual and the estimated vector in degrees, it is shown the average error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PERFORMANCE METRICS ON TESTING DATA\n",
      "\n",
      "Mean squared error: 0.1155\n",
      "Mean absolute error: 0.2518\n",
      "Median absolute error: 0.1966\n",
      "Coefficient of determination (R2 score): 0.6136\n",
      "Explained variance score: 0.6138\n",
      "R2 score on individual targets [x, y, z] [0.66936782 0.68238116 0.48891671] \n",
      "\n",
      "MEAN ANGLE ERROR: 30.2354 Degrees\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#computing a set of performance metrics\n",
    "def show_metrics(target_test, target_pred):\n",
    "    #mean squared error (lower the better)\n",
    "    print('Mean squared error: %.4f'% sklearn.metrics.mean_squared_error(target_test, target_pred))\n",
    "\n",
    "    #mean absolute error (lower the better)\n",
    "    print('Mean absolute error: %.4f'% sklearn.metrics.mean_absolute_error(target_test, target_pred))\n",
    "\n",
    "    #median absolute error (lower the better)\n",
    "    print('Median absolute error: %.4f'% sklearn.metrics.median_absolute_error(target_test, target_pred))\n",
    "\n",
    "    #coefficient of determination (r2 score): 1 is perfect prediction (it can get arbitrary negative)\n",
    "    print('Coefficient of determination (R2 score): %.4f'% sklearn.metrics.r2_score(target_test, target_pred))\n",
    "\n",
    "    #explained variance score: 1 is perfect prediction (it can get arbitrary worse)\n",
    "    print('Explained variance score: %.4f'% sklearn.metrics.explained_variance_score(target_test, target_pred))\n",
    "\n",
    "    #this is is a better represenation of the r2 score when working with multiple outputs\n",
    "    #as it provides a value for each target value\n",
    "    print('R2 score on individual targets [x, y, z]',sklearn.metrics.r2_score(target_test, target_pred, multioutput='raw_values'), '\\n')\n",
    "\n",
    "    #####----- Delta angle between actual DOA and estimated DOA\n",
    "    deg = []\n",
    "    for d in range(target_test.shape[0]):\n",
    "        deg = np.append(deg, np.rad2deg(angle_between(target_test[d], target_pred[d])))\n",
    "    print('MEAN ANGLE ERROR: %.4f'% np.mean(deg), 'Degrees\\n')\n",
    "\n",
    "print('PERFORMANCE METRICS ON TESTING DATA\\n')\n",
    "show_metrics(target_test, target_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance Metrics on Training Data\n",
    "\n",
    "This metrics will be obtained with the training data in order to stablish comparisions between cross-validation, testing, and training sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PERFORMANCE METRICS ON TRAINING DATA\n",
      "\n",
      "Mean squared error: 0.1249\n",
      "Mean absolute error: 0.2616\n",
      "Median absolute error: 0.2032\n",
      "Coefficient of determination (R2 score): 0.5883\n",
      "Explained variance score: 0.5884\n",
      "R2 score on individual targets [x, y, z] [0.645721   0.6469887  0.47206668] \n",
      "\n",
      "MEAN ANGLE ERROR: 31.6975 Degrees\n",
      "\n",
      "TOTAL EXECUTION TIME:  381.2887952327728  sec\n"
     ]
    }
   ],
   "source": [
    "#Predicting from the testing data\n",
    "target_pred_train = restored_model.predict(features)\n",
    "\n",
    "print('PERFORMANCE METRICS ON TRAINING DATA\\n')\n",
    "show_metrics(target, target_pred_train)\n",
    "\n",
    "print('TOTAL EXECUTION TIME: ', str(time.time() - start_time), ' sec')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Return to [file.ipybn](file.ipynb)\n",
    "\n",
    "Go back to [feature_extraction.ipybn](feature_extraction.ipynb)\n",
    "\n",
    "Continue to [inference.ipybn](inference.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}