part_6.modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the processed dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install -U -q datalearn19intro\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from datalearn19intro import get_processed_intro_dataset\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.display.float_format = '{:,.2f}'.format\n",
    "pd.set_option('display.max_columns', 150)\n",
    "pd.set_option('display.max_rows', 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = get_processed_intro_dataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1001, 22)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.set_index('account_id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_gmail_fromaccounts</th>\n",
       "      <th>collection_21_days</th>\n",
       "      <th>billed_users_count</th>\n",
       "      <th>log_max_tsize</th>\n",
       "      <th>os_android_avg</th>\n",
       "      <th>os_chrome_os_avg</th>\n",
       "      <th>os_ios_avg</th>\n",
       "      <th>os_linux_avg</th>\n",
       "      <th>os_mac_avg</th>\n",
       "      <th>total_events_sum</th>\n",
       "      <th>notification_events_sum</th>\n",
       "      <th>new_entry_events_sum</th>\n",
       "      <th>payment_events_sum</th>\n",
       "      <th>inbox_events_sum</th>\n",
       "      <th>communicating_events_sum</th>\n",
       "      <th>non_communicating_events_sum</th>\n",
       "      <th>web_events_sum</th>\n",
       "      <th>ios_events_sum</th>\n",
       "      <th>desktop_app_events_sum</th>\n",
       "      <th>empty_events_sum</th>\n",
       "      <th>lead_score</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>account_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2793496</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.57</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2793497</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.57</td>\n",
       "      <td>0.45</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.55</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2793498</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.43</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2793499</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.57</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2793500</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.43</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2793501</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.57</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2793502</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.43</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2793503</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.57</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            is_gmail_fromaccounts  collection_21_days  billed_users_count  \\\n",
       "account_id                                                                  \n",
       "2793496                      1.00                0.00                0.00   \n",
       "2793497                      1.00                0.00                0.00   \n",
       "2793498                      1.00                0.00                0.00   \n",
       "2793499                      1.00                0.00                0.00   \n",
       "2793500                      1.00                0.00                0.00   \n",
       "2793501                      0.00                0.00                0.00   \n",
       "2793502                      1.00                0.00                0.00   \n",
       "2793503                      1.00                0.00                0.00   \n",
       "\n",
       "            log_max_tsize  os_android_avg  os_chrome_os_avg  os_ios_avg  \\\n",
       "account_id                                                                \n",
       "2793496              0.57            0.00              0.00        0.00   \n",
       "2793497              0.57            0.45              0.00        0.55   \n",
       "2793498              0.43            0.00              0.00        0.00   \n",
       "2793499              0.57            1.00              0.00        0.00   \n",
       "2793500              0.43            0.00              1.00        0.00   \n",
       "2793501              0.57            0.00              0.00        0.00   \n",
       "2793502              0.43            0.00              0.00        0.00   \n",
       "2793503              0.57            1.00              0.00        0.00   \n",
       "\n",
       "            os_linux_avg  os_mac_avg  total_events_sum  \\\n",
       "account_id                                               \n",
       "2793496             0.00        0.00              0.00   \n",
       "2793497             0.00        0.00              0.01   \n",
       "2793498             0.00        0.00              0.00   \n",
       "2793499             0.00        0.00              0.00   \n",
       "2793500             0.00        0.00              0.00   \n",
       "2793501             0.00        1.00              0.00   \n",
       "2793502             0.00        0.00              0.00   \n",
       "2793503             0.00        0.00              0.00   \n",
       "\n",
       "            notification_events_sum  new_entry_events_sum  payment_events_sum  \\\n",
       "account_id                                                                      \n",
       "2793496                        0.00                  0.00                0.00   \n",
       "2793497                        0.04                  0.02                0.00   \n",
       "2793498                        0.00                  0.00                0.00   \n",
       "2793499                        0.00                  0.00                0.00   \n",
       "2793500                        0.00                  0.00                0.00   \n",
       "2793501                        0.00                  0.00                0.00   \n",
       "2793502                        0.00                  0.00                0.00   \n",
       "2793503                        0.00                  0.00                0.00   \n",
       "\n",
       "            inbox_events_sum  communicating_events_sum  \\\n",
       "account_id                                               \n",
       "2793496                 0.00                      0.00   \n",
       "2793497                 0.00                      0.00   \n",
       "2793498                 0.00                      0.00   \n",
       "2793499                 0.00                      0.00   \n",
       "2793500                 0.00                      0.00   \n",
       "2793501                 0.00                      0.00   \n",
       "2793502                 0.00                      0.00   \n",
       "2793503                 0.00                      0.00   \n",
       "\n",
       "            non_communicating_events_sum  web_events_sum  ios_events_sum  \\\n",
       "account_id                                                                 \n",
       "2793496                             0.00            0.00            0.00   \n",
       "2793497                             0.00            0.00            0.17   \n",
       "2793498                             0.00            0.00            0.00   \n",
       "2793499                             0.00            0.00            0.00   \n",
       "2793500                             0.00            0.00            0.00   \n",
       "2793501                             0.00            0.00            0.00   \n",
       "2793502                             0.00            0.00            0.00   \n",
       "2793503                             0.00            0.00            0.00   \n",
       "\n",
       "            desktop_app_events_sum  empty_events_sum  lead_score  \n",
       "account_id                                                        \n",
       "2793496                       0.00              0.00        0.00  \n",
       "2793497                       0.00              0.00        0.00  \n",
       "2793498                       0.00              0.00        0.00  \n",
       "2793499                       0.00              0.00        0.00  \n",
       "2793500                       0.00              0.00        0.00  \n",
       "2793501                       0.00              0.00        0.00  \n",
       "2793502                       0.00              0.00        0.00  \n",
       "2793503                       0.00              0.00        0.00  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop('lead_score', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df['lead_score']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's split the data into train and test sets, at a 80/20 ratio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/shaypalachy/.pyenv/versions/3.6.5/envs/py3/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='warn',\n",
       "          n_jobs=None, penalty='l2', random_state=None, solver='warn',\n",
       "          tol=0.0001, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, precision_score, recall_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9701492537313433"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OMG! That's amazing!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/shaypalachy/.pyenv/versions/3.6.5/envs/py3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh no! Maybe our model wasn't as good as we thought!\n",
    "What happened?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([0.]), array([201]))"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(y_pred, return_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([0., 1.]), array([980,  21]))"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(y, return_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 97.01%\n",
      "Precision: 0.00%\n",
      "Recall: 0.00%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/shaypalachy/.pyenv/versions/3.6.5/envs/py3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "print(\"Accuracy: {:,.2f}%\".format(100*accuracy_score(y_test, y_pred)))\n",
    "print(\"Precision: {:,.2f}%\".format(100*precision_score(y_test, y_pred)))\n",
    "print(\"Recall: {:,.2f}%\".format(100*recall_score(y_test, y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice, we have 980 negative examples and only 21 positive ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "97.9020979020979"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "980/1001 * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model was optimizing accuracy when fitting its parameters, and the easiest to do that is to simply predict 0 all the time for roughly 98% accuracy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `class_weight='balanced'` to each **class** equally important instead of each row/entry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/shaypalachy/.pyenv/versions/3.6.5/envs/py3/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "clf = LogisticRegression(class_weight='balanced')\n",
    "clf.fit(X_train, y_train)\n",
    "y_pred = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 68.66%\n",
      "Precision: 7.46%\n",
      "Recall: 83.33%\n"
     ]
    }
   ],
   "source": [
    "print(\"Accuracy: {:,.2f}%\".format(100*accuracy_score(y_test, y_pred)))\n",
    "print(\"Precision: {:,.2f}%\".format(100*precision_score(y_test, y_pred)))\n",
    "print(\"Recall: {:,.2f}%\".format(100*recall_score(y_test, y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can define, for each hyperparameter, a range of possible values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For logistic regression here we will play with just two hyperparameters:\n",
    "\n",
    "* `penalty` - The loss function use. Both L1 and L2 are common loss functions.\n",
    "* `C` - Inverse of regularization strength; smaller values specify stronger regularization. Regularization can prevent overfitting, a concept which you'll discuss in the advanced workshops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyparam_grid = [{\n",
    "    'penalty': ['l1', 'l2'],\n",
    "    'C': [1, 10, 100, 1000],\n",
    "    'class_weight': ['balanced'],\n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can optimize are hyperparameters for various metrics..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = ['precision', 'recall']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# silence annoying future warnings\n",
    "def warn(*args, **kwargs):\n",
    "    pass\n",
    "import warnings\n",
    "warnings.warn = warn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Tuning hyper-parameters for precision\n",
      "\n",
      "Best parameters set found on development set:\n",
      "\n",
      "{'C': 10, 'class_weight': 'balanced', 'penalty': 'l2'}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.525 (+/-0.014) for {'C': 1, 'class_weight': 'balanced', 'penalty': 'l1'}\n",
      "0.525 (+/-0.014) for {'C': 1, 'class_weight': 'balanced', 'penalty': 'l2'}\n",
      "0.515 (+/-0.036) for {'C': 10, 'class_weight': 'balanced', 'penalty': 'l1'}\n",
      "0.525 (+/-0.014) for {'C': 10, 'class_weight': 'balanced', 'penalty': 'l2'}\n",
      "0.508 (+/-0.022) for {'C': 100, 'class_weight': 'balanced', 'penalty': 'l1'}\n",
      "0.521 (+/-0.024) for {'C': 100, 'class_weight': 'balanced', 'penalty': 'l2'}\n",
      "0.516 (+/-0.032) for {'C': 1000, 'class_weight': 'balanced', 'penalty': 'l1'}\n",
      "0.513 (+/-0.031) for {'C': 1000, 'class_weight': 'balanced', 'penalty': 'l2'}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.99      0.68      0.81       195\n",
      "         1.0       0.07      0.83      0.14         6\n",
      "\n",
      "   micro avg       0.69      0.69      0.69       201\n",
      "   macro avg       0.53      0.76      0.47       201\n",
      "weighted avg       0.97      0.69      0.79       201\n",
      "\n",
      "\n",
      "# Tuning hyper-parameters for recall\n",
      "\n",
      "Best parameters set found on development set:\n",
      "\n",
      "{'C': 10, 'class_weight': 'balanced', 'penalty': 'l2'}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.786 (+/-0.160) for {'C': 1, 'class_weight': 'balanced', 'penalty': 'l1'}\n",
      "0.785 (+/-0.162) for {'C': 1, 'class_weight': 'balanced', 'penalty': 'l2'}\n",
      "0.669 (+/-0.380) for {'C': 10, 'class_weight': 'balanced', 'penalty': 'l1'}\n",
      "0.788 (+/-0.162) for {'C': 10, 'class_weight': 'balanced', 'penalty': 'l2'}\n",
      "0.586 (+/-0.219) for {'C': 100, 'class_weight': 'balanced', 'penalty': 'l1'}\n",
      "0.728 (+/-0.256) for {'C': 100, 'class_weight': 'balanced', 'penalty': 'l2'}\n",
      "0.621 (+/-0.242) for {'C': 1000, 'class_weight': 'balanced', 'penalty': 'l1'}\n",
      "0.641 (+/-0.326) for {'C': 1000, 'class_weight': 'balanced', 'penalty': 'l2'}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.99      0.68      0.81       195\n",
      "         1.0       0.07      0.83      0.14         6\n",
      "\n",
      "   micro avg       0.69      0.69      0.69       201\n",
      "   macro avg       0.53      0.76      0.47       201\n",
      "weighted avg       0.97      0.69      0.79       201\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "for score in scores:\n",
    "    print(\"# Tuning hyper-parameters for %s\" % score)\n",
    "    print()\n",
    "\n",
    "    clf = GridSearchCV(LogisticRegression(), hyparam_grid, cv=5,\n",
    "                       scoring='%s_macro' % score)\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    print(\"Best parameters set found on development set:\")\n",
    "    print()\n",
    "    print(clf.best_params_)\n",
    "    print()\n",
    "    print(\"Grid scores on development set:\")\n",
    "    print()\n",
    "    means = clf.cv_results_['mean_test_score']\n",
    "    stds = clf.cv_results_['std_test_score']\n",
    "    for mean, std, params in zip(means, stds, clf.cv_results_['params']):\n",
    "        print(\"%0.3f (+/-%0.03f) for %r\"\n",
    "              % (mean, std * 2, params))\n",
    "    print()\n",
    "\n",
    "    print(\"Detailed classification report:\")\n",
    "    print()\n",
    "    print(\"The model is trained on the full development set.\")\n",
    "    print(\"The scores are computed on the full evaluation set.\")\n",
    "    print()\n",
    "    y_true, y_pred = y_test, clf.predict(X_test)\n",
    "    print(classification_report(y_true, y_pred))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now fit a model with the tune hyperparameters over the entire training set and test its performace on the test set. This will provide an estimate of performance which is likely to be (possibly substantially) optimistically biased.\n",
    "\n",
    "A more advanced way to get estimates for hyperoptimization results - and a less biased one - is nested cross-validation; this is a more computationaly intensive method, and out of scope here, of course. :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 68.66%\n",
      "Precision: 7.46%\n",
      "Recall: 83.33%\n"
     ]
    }
   ],
   "source": [
    "clf = LogisticRegression(class_weight='balanced', C=10, penalty='l2')\n",
    "clf.fit(X_train, y_train)\n",
    "y_pred = clf.predict(X_test)\n",
    "print(\"Accuracy: {:,.2f}%\".format(100*accuracy_score(y_test, y_pred)))\n",
    "print(\"Precision: {:,.2f}%\".format(100*precision_score(y_test, y_pred)))\n",
    "print(\"Recall: {:,.2f}%\".format(100*recall_score(y_test, y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(note: this produced no difference in relation to the default hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Trying other models is easy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 97.01%\n",
      "Precision: 0.00%\n",
      "Recall: 0.00%\n"
     ]
    }
   ],
   "source": [
    "clf = RandomForestClassifier(class_weight='balanced')\n",
    "clf.fit(X_train, y_train)\n",
    "y_pred = clf.predict(X_test)\n",
    "print(\"Accuracy: {:,.2f}%\".format(100*accuracy_score(y_test, y_pred)))\n",
    "print(\"Precision: {:,.2f}%\".format(100*precision_score(y_test, y_pred)))\n",
    "print(\"Recall: {:,.2f}%\".format(100*recall_score(y_test, y_pred)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 68.66%\n",
      "Precision: 7.46%\n",
      "Recall: 83.33%\n"
     ]
    }
   ],
   "source": [
    "clf = SVC(class_weight='balanced')\n",
    "clf.fit(X_train, y_train)\n",
    "y_pred = clf.predict(X_test)\n",
    "print(\"Accuracy: {:,.2f}%\".format(100*accuracy_score(y_test, y_pred)))\n",
    "print(\"Precision: {:,.2f}%\".format(100*precision_score(y_test, y_pred)))\n",
    "print(\"Recall: {:,.2f}%\".format(100*recall_score(y_test, y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Don't be surprised that different linear models tend to achieve the same performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 2.99%\n",
      "Precision: 2.99%\n",
      "Recall: 100.00%\n"
     ]
    }
   ],
   "source": [
    "clf = SVC(class_weight='balanced', kernel='poly')\n",
    "clf.fit(X_train, y_train)\n",
    "y_pred = clf.predict(X_test)\n",
    "print(\"Accuracy: {:,.2f}%\".format(100*accuracy_score(y_test, y_pred)))\n",
    "print(\"Precision: {:,.2f}%\".format(100*precision_score(y_test, y_pred)))\n",
    "print(\"Recall: {:,.2f}%\".format(100*recall_score(y_test, y_pred)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyparam_grid = [{\n",
    "    'kernel': ['rbf', 'poly', 'sigmoid'],\n",
    "    'C': [0.1, 1, 10, 100],\n",
    "    'degree': [2, 3, 4, 5],\n",
    "    'class_weight': ['balanced'],\n",
    "}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Tuning hyper-parameters for precision\n",
      "\n",
      "Best parameters set found on development set:\n",
      "\n",
      "{'C': 0.1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'rbf'}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.530 (+/-0.004) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'rbf'}\n",
      "0.009 (+/-0.000) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'poly'}\n",
      "0.530 (+/-0.004) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'sigmoid'}\n",
      "0.530 (+/-0.004) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'rbf'}\n",
      "0.009 (+/-0.000) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'poly'}\n",
      "0.530 (+/-0.004) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'sigmoid'}\n",
      "0.530 (+/-0.004) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'rbf'}\n",
      "0.009 (+/-0.000) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'poly'}\n",
      "0.530 (+/-0.004) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'sigmoid'}\n",
      "0.530 (+/-0.004) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'rbf'}\n",
      "0.009 (+/-0.000) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'poly'}\n",
      "0.530 (+/-0.004) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'sigmoid'}\n",
      "0.530 (+/-0.004) for {'C': 1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'rbf'}\n",
      "0.527 (+/-0.010) for {'C': 1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'poly'}\n",
      "0.530 (+/-0.004) for {'C': 1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'sigmoid'}\n",
      "0.530 (+/-0.004) for {'C': 1, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'rbf'}\n",
      "0.009 (+/-0.000) for {'C': 1, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'poly'}\n",
      "0.530 (+/-0.004) for {'C': 1, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'sigmoid'}\n",
      "0.530 (+/-0.004) for {'C': 1, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'rbf'}\n",
      "0.009 (+/-0.000) for {'C': 1, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'poly'}\n",
      "0.530 (+/-0.004) for {'C': 1, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'sigmoid'}\n",
      "0.530 (+/-0.004) for {'C': 1, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'rbf'}\n",
      "0.009 (+/-0.000) for {'C': 1, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'poly'}\n",
      "0.530 (+/-0.004) for {'C': 1, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'sigmoid'}\n",
      "0.528 (+/-0.012) for {'C': 10, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'rbf'}\n",
      "0.530 (+/-0.004) for {'C': 10, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'poly'}\n",
      "0.528 (+/-0.012) for {'C': 10, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'sigmoid'}\n",
      "0.528 (+/-0.012) for {'C': 10, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'rbf'}\n",
      "0.527 (+/-0.011) for {'C': 10, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'poly'}\n",
      "0.528 (+/-0.012) for {'C': 10, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'sigmoid'}\n",
      "0.528 (+/-0.012) for {'C': 10, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'rbf'}\n",
      "0.491 (+/-0.000) for {'C': 10, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'poly'}\n",
      "0.528 (+/-0.012) for {'C': 10, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'sigmoid'}\n",
      "0.528 (+/-0.012) for {'C': 10, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'rbf'}\n",
      "0.491 (+/-0.000) for {'C': 10, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'poly'}\n",
      "0.528 (+/-0.012) for {'C': 10, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'sigmoid'}\n",
      "0.527 (+/-0.022) for {'C': 100, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'rbf'}\n",
      "0.527 (+/-0.012) for {'C': 100, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'poly'}\n",
      "0.528 (+/-0.012) for {'C': 100, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'sigmoid'}\n",
      "0.527 (+/-0.022) for {'C': 100, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'rbf'}\n",
      "0.530 (+/-0.004) for {'C': 100, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'poly'}\n",
      "0.528 (+/-0.012) for {'C': 100, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'sigmoid'}\n",
      "0.527 (+/-0.022) for {'C': 100, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'rbf'}\n",
      "0.521 (+/-0.014) for {'C': 100, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'poly'}\n",
      "0.528 (+/-0.012) for {'C': 100, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'sigmoid'}\n",
      "0.527 (+/-0.022) for {'C': 100, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'rbf'}\n",
      "0.009 (+/-0.000) for {'C': 100, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'poly'}\n",
      "0.528 (+/-0.012) for {'C': 100, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'sigmoid'}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.99      0.68      0.81       195\n",
      "         1.0       0.07      0.83      0.14         6\n",
      "\n",
      "   micro avg       0.69      0.69      0.69       201\n",
      "   macro avg       0.53      0.76      0.47       201\n",
      "weighted avg       0.97      0.69      0.79       201\n",
      "\n",
      "\n",
      "# Tuning hyper-parameters for recall\n",
      "\n",
      "Best parameters set found on development set:\n",
      "\n",
      "{'C': 0.1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'rbf'}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.848 (+/-0.025) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'rbf'}\n",
      "0.500 (+/-0.000) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'poly'}\n",
      "0.848 (+/-0.025) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'sigmoid'}\n",
      "0.848 (+/-0.025) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'rbf'}\n",
      "0.500 (+/-0.000) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'poly'}\n",
      "0.848 (+/-0.025) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'sigmoid'}\n",
      "0.848 (+/-0.025) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'rbf'}\n",
      "0.500 (+/-0.000) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'poly'}\n",
      "0.848 (+/-0.025) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'sigmoid'}\n",
      "0.848 (+/-0.025) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'rbf'}\n",
      "0.500 (+/-0.000) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'poly'}\n",
      "0.848 (+/-0.025) for {'C': 0.1, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'sigmoid'}\n",
      "0.848 (+/-0.025) for {'C': 1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'rbf'}\n",
      "0.825 (+/-0.083) for {'C': 1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'poly'}\n",
      "0.848 (+/-0.025) for {'C': 1, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'sigmoid'}\n",
      "0.848 (+/-0.025) for {'C': 1, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'rbf'}\n",
      "0.500 (+/-0.000) for {'C': 1, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'poly'}\n",
      "0.848 (+/-0.025) for {'C': 1, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'sigmoid'}\n",
      "0.848 (+/-0.025) for {'C': 1, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'rbf'}\n",
      "0.500 (+/-0.000) for {'C': 1, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'poly'}\n",
      "0.848 (+/-0.025) for {'C': 1, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'sigmoid'}\n",
      "0.848 (+/-0.025) for {'C': 1, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'rbf'}\n",
      "0.500 (+/-0.000) for {'C': 1, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'poly'}\n",
      "0.848 (+/-0.025) for {'C': 1, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'sigmoid'}\n",
      "0.818 (+/-0.133) for {'C': 10, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'rbf'}\n",
      "0.848 (+/-0.025) for {'C': 10, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'poly'}\n",
      "0.818 (+/-0.133) for {'C': 10, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'sigmoid'}\n",
      "0.818 (+/-0.133) for {'C': 10, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'rbf'}\n",
      "0.821 (+/-0.097) for {'C': 10, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'poly'}\n",
      "0.818 (+/-0.133) for {'C': 10, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'sigmoid'}\n",
      "0.818 (+/-0.133) for {'C': 10, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'rbf'}\n",
      "0.500 (+/-0.000) for {'C': 10, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'poly'}\n",
      "0.818 (+/-0.133) for {'C': 10, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'sigmoid'}\n",
      "0.818 (+/-0.133) for {'C': 10, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'rbf'}\n",
      "0.500 (+/-0.000) for {'C': 10, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'poly'}\n",
      "0.818 (+/-0.133) for {'C': 10, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'sigmoid'}\n",
      "0.801 (+/-0.247) for {'C': 100, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'rbf'}\n",
      "0.818 (+/-0.136) for {'C': 100, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'poly'}\n",
      "0.820 (+/-0.135) for {'C': 100, 'class_weight': 'balanced', 'degree': 2, 'kernel': 'sigmoid'}\n",
      "0.801 (+/-0.247) for {'C': 100, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'rbf'}\n",
      "0.847 (+/-0.025) for {'C': 100, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'poly'}\n",
      "0.820 (+/-0.135) for {'C': 100, 'class_weight': 'balanced', 'degree': 3, 'kernel': 'sigmoid'}\n",
      "0.801 (+/-0.247) for {'C': 100, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'rbf'}\n",
      "0.735 (+/-0.191) for {'C': 100, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'poly'}\n",
      "0.820 (+/-0.135) for {'C': 100, 'class_weight': 'balanced', 'degree': 4, 'kernel': 'sigmoid'}\n",
      "0.801 (+/-0.247) for {'C': 100, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'rbf'}\n",
      "0.500 (+/-0.000) for {'C': 100, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'poly'}\n",
      "0.820 (+/-0.135) for {'C': 100, 'class_weight': 'balanced', 'degree': 5, 'kernel': 'sigmoid'}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.99      0.68      0.81       195\n",
      "         1.0       0.07      0.83      0.14         6\n",
      "\n",
      "   micro avg       0.69      0.69      0.69       201\n",
      "   macro avg       0.53      0.76      0.47       201\n",
      "weighted avg       0.97      0.69      0.79       201\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "for score in scores:\n",
    "    print(\"# Tuning hyper-parameters for %s\" % score)\n",
    "    print()\n",
    "\n",
    "    clf = GridSearchCV(SVC(), hyparam_grid, cv=5,\n",
    "                       scoring='%s_macro' % score)\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    print(\"Best parameters set found on development set:\")\n",
    "    print()\n",
    "    print(clf.best_params_)\n",
    "    print()\n",
    "    print(\"Grid scores on development set:\")\n",
    "    print()\n",
    "    means = clf.cv_results_['mean_test_score']\n",
    "    stds = clf.cv_results_['std_test_score']\n",
    "    for mean, std, params in zip(means, stds, clf.cv_results_['params']):\n",
    "        print(\"%0.3f (+/-%0.03f) for %r\"\n",
    "              % (mean, std * 2, params))\n",
    "    print()\n",
    "\n",
    "    print(\"Detailed classification report:\")\n",
    "    print()\n",
    "    print(\"The model is trained on the full development set.\")\n",
    "    print(\"The scores are computed on the full evaluation set.\")\n",
    "    print()\n",
    "    y_true, y_pred = y_test, clf.predict(X_test)\n",
    "    print(classification_report(y_true, y_pred))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyparam_grid = [{\n",
    "    'n_estimators': [5, 20, 100],\n",
    "    'criterion': ['gini', 'entropy'],\n",
    "    'max_depth': [None, 3, 5],\n",
    "    'class_weight': ['balanced'],\n",
    "}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Tuning hyper-parameters for precision\n",
      "\n",
      "Best parameters set found on development set:\n",
      "\n",
      "{'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'n_estimators': 5}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.491 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': None, 'n_estimators': 5}\n",
      "0.491 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': None, 'n_estimators': 20}\n",
      "0.491 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': None, 'n_estimators': 100}\n",
      "0.524 (+/-0.058) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'n_estimators': 5}\n",
      "0.524 (+/-0.136) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'n_estimators': 20}\n",
      "0.511 (+/-0.083) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'n_estimators': 100}\n",
      "0.533 (+/-0.132) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'n_estimators': 5}\n",
      "0.491 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'n_estimators': 20}\n",
      "0.491 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'n_estimators': 100}\n",
      "0.491 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': None, 'n_estimators': 5}\n",
      "0.491 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': None, 'n_estimators': 20}\n",
      "0.491 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': None, 'n_estimators': 100}\n",
      "0.503 (+/-0.053) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'n_estimators': 5}\n",
      "0.533 (+/-0.132) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'n_estimators': 20}\n",
      "0.505 (+/-0.060) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'n_estimators': 100}\n",
      "0.518 (+/-0.071) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'n_estimators': 5}\n",
      "0.491 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'n_estimators': 20}\n",
      "0.491 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'n_estimators': 100}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.98      0.90      0.94       195\n",
      "         1.0       0.10      0.33      0.15         6\n",
      "\n",
      "   micro avg       0.89      0.89      0.89       201\n",
      "   macro avg       0.54      0.62      0.54       201\n",
      "weighted avg       0.95      0.89      0.92       201\n",
      "\n",
      "\n",
      "# Tuning hyper-parameters for recall\n",
      "\n",
      "Best parameters set found on development set:\n",
      "\n",
      "{'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'n_estimators': 5}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.499 (+/-0.003) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': None, 'n_estimators': 5}\n",
      "0.500 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': None, 'n_estimators': 20}\n",
      "0.500 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': None, 'n_estimators': 100}\n",
      "0.513 (+/-0.223) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'n_estimators': 5}\n",
      "0.503 (+/-0.133) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'n_estimators': 20}\n",
      "0.486 (+/-0.010) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 3, 'n_estimators': 100}\n",
      "0.508 (+/-0.145) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'n_estimators': 5}\n",
      "0.492 (+/-0.015) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'n_estimators': 20}\n",
      "0.496 (+/-0.006) for {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 5, 'n_estimators': 100}\n",
      "0.530 (+/-0.133) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': None, 'n_estimators': 5}\n",
      "0.499 (+/-0.003) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': None, 'n_estimators': 20}\n",
      "0.500 (+/-0.000) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': None, 'n_estimators': 100}\n",
      "0.557 (+/-0.257) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'n_estimators': 5}\n",
      "0.539 (+/-0.139) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'n_estimators': 20}\n",
      "0.512 (+/-0.140) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 3, 'n_estimators': 100}\n",
      "0.472 (+/-0.064) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'n_estimators': 5}\n",
      "0.492 (+/-0.019) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'n_estimators': 20}\n",
      "0.497 (+/-0.006) for {'class_weight': 'balanced', 'criterion': 'entropy', 'max_depth': 5, 'n_estimators': 100}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.98      0.81      0.88       195\n",
      "         1.0       0.05      0.33      0.09         6\n",
      "\n",
      "   micro avg       0.79      0.79      0.79       201\n",
      "   macro avg       0.51      0.57      0.48       201\n",
      "weighted avg       0.95      0.79      0.86       201\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "for score in scores:\n",
    "    print(\"# Tuning hyper-parameters for %s\" % score)\n",
    "    print()\n",
    "\n",
    "    clf = GridSearchCV(RandomForestClassifier(), hyparam_grid, cv=5,\n",
    "                       scoring='%s_macro' % score)\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    print(\"Best parameters set found on development set:\")\n",
    "    print()\n",
    "    print(clf.best_params_)\n",
    "    print()\n",
    "    print(\"Grid scores on development set:\")\n",
    "    print()\n",
    "    means = clf.cv_results_['mean_test_score']\n",
    "    stds = clf.cv_results_['std_test_score']\n",
    "    for mean, std, params in zip(means, stds, clf.cv_results_['params']):\n",
    "        print(\"%0.3f (+/-%0.03f) for %r\"\n",
    "              % (mean, std * 2, params))\n",
    "    print()\n",
    "\n",
    "    print(\"Detailed classification report:\")\n",
    "    print()\n",
    "    print(\"The model is trained on the full development set.\")\n",
    "    print(\"The scores are computed on the full evaluation set.\")\n",
    "    print()\n",
    "    y_true, y_pred = y_test, clf.predict(X_test)\n",
    "    print(classification_report(y_true, y_pred))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Without putting too much effort into hyperparameter tunning, a random forest classifier got an F1 score of 54% (averaged over classes) compared to 47% of logistic regression and SVM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even if that is not what we want to optimize for here (and that depends on a lot of factors), this still demonstrates why giving other models at least a superficial examination can be worthwhile."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final notes: The classification problem here is significantly unbalanced; we did not treat this at all except for balancing class weights. An imbalance of 98/2 probably justifies more sophisticated tools.\n",
    "\n",
    "## One such option is to treat positive examples as anomalies, and to then draw on knowledge and methods from the field of anomaly detection.\n",
    "\n",
    "## Other options can include more advnaced processing of the data.\n",
    "\n",
    "## The advanced workshops might go into more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# We're done! Thank you!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py3",
   "language": "python",
   "name": "py3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}