03-model-logistic-regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression \n",
    "\n",
    "\n",
    "In this notebook we will train a [Logistic Regression model](https://en.wikipedia.org/wiki/Logistic_regression) to distinguish between legitimate and fraudulent transactions. \n",
    "\n",
    "Logistic Regression is a classic statistical technique used for binary classification. Here the binary variable we are predicting is 'legitimate' or 'not legitimate' (i.e. fraudulent).\n",
    "\n",
    "We begin by loading in our generated data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "df = pd.read_parquet(\"fraud-cleaned-sample.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to split our data set into training and testing sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import model_selection\n",
    "train, test = model_selection.train_test_split(df, random_state=43)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also load in the feature engineering pipeline stage which we developed in [notebook 2](02-feature-engineering.ipynb). The model takes the feature vectors as input, rather than the raw data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cloudpickle as cp\n",
    "feature_pipeline = cp.load(open('feat_pipeline.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('feature_extraction',\n",
       "                 ColumnTransformer(transformers=[('interarrival_scaler',\n",
       "                                                  Pipeline(steps=[('median_imputer',\n",
       "                                                                   SimpleImputer(strategy='median')),\n",
       "                                                                  ('interarrival_scaler',\n",
       "                                                                   RobustScaler())]),\n",
       "                                                  ['interarrival']),\n",
       "                                                 ('amount_scaler',\n",
       "                                                  RobustScaler(), ['amount']),\n",
       "                                                 ('onehot',\n",
       "                                                  OneHotEncoder(categories=[['online',\n",
       "                                                                             'contactless',\n",
       "                                                                             'chip_and_pin',\n",
       "                                                                             'manual',\n",
       "                                                                             'swipe']],\n",
       "                                                                handle_unknown='ignore'),\n",
       "                                                  ['trans_type']),\n",
       "                                                 ('m_hashing',\n",
       "                                                  Pipeline(steps=[('dictify',\n",
       "                                                                   FunctionTransformer(accept_sparse=True,\n",
       "                                                                                       func=<function amap at 0x7fad54464b80>)),\n",
       "                                                                  ('hasher',\n",
       "                                                                   FeatureHasher(n_features=128))]),\n",
       "                                                  'merchant_id')]))])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dealing with Imbalanced Classes\n",
    "\n",
    "When the training data set contains unequal representation from each of your classes we say we are dealing with 'imbalanced classes'. In our data set fewer than 2% of the samples are fraudulent, and the remaining 98% are legitimate. Thus we have imbalanced classes. \n",
    "\n",
    "This causes problems for a few reasons:\n",
    "1. A model which classifies all transactions as 'legitimate' would be correct 98% of the time. This high accuracy can trick you into thinking that your model is working well, despite it just returning 'legitimate' for every sample it sees. \n",
    "2. Even if your model tries to learn patterns in the data, it may struggle to learn from the 'fraudulent' data since there simply isn't enough of it.\n",
    "\n",
    "\n",
    "There are a few approaches we could take to tackle the problem, and today we will use two of them: \n",
    "1. We will use metrics which are more informative than simply counting how often the model makes a correct prediction. \n",
    "2. We will weight the samples by the inverse of the frequency of their label within the data set. These weights will be passed into the logistic regression model, and used to ensure that the model is penalised proportionally to this weight for making a misclassification for each class when it is training. \n",
    "\n",
    "\n",
    "In this next cell we compute these weights for each of the data labels. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "fraud_frequency = train[train[\"label\"] == \"fraud\"][\"timestamp\"].count() / train[\"timestamp\"].count()\n",
    "train.loc[train[\"label\"] == \"legitimate\", \"weights\"] = fraud_frequency\n",
    "train.loc[train[\"label\"] == \"fraud\", \"weights\"] = (1 - fraud_frequency)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're now ready to train our Logistic Regression model. The model is trained on the feature vectors (generated using our `feature_pipeline` from the previous notebook) and we pass the class weights we computed above as a model parameter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(max_iter=500)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr = LogisticRegression(max_iter=500)\n",
    "\n",
    "svecs = feature_pipeline.fit_transform(train)\n",
    "lr.fit(svecs, train[\"label\"], sample_weight=train[\"weights\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to validate our model to check how well it performs on data it wasn't trained on. We use the model we just trained to make predictions for the data in our test set, and compare those predictions to the truth. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "       fraud       0.15      0.92      0.26     11424\n",
      "  legitimate       1.00      0.90      0.95    613576\n",
      "\n",
      "    accuracy                           0.90    625000\n",
      "   macro avg       0.58      0.91      0.60    625000\n",
      "weighted avg       0.98      0.90      0.94    625000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "predictions = lr.predict(feature_pipeline.fit_transform(test))\n",
    "print(classification_report(test.label.values, predictions))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The report shows that the model is performing okay, but is much better at identifying legitimate transactions than fraudulent ones. \n",
    "\n",
    "We can visualise the accuracy of classifications in a binary confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-f2c0af683c264306b546d388d6c332a9\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-f2c0af683c264306b546d388d6c332a9\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-f2c0af683c264306b546d388d6c332a9\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-25d663b7415ade2ca5e623f296f30d6d\"}, \"mark\": \"rect\", \"encoding\": {\"color\": {\"type\": \"quantitative\", \"field\": \"value\"}, \"tooltip\": [{\"type\": \"quantitative\", \"field\": \"raw_count\"}], \"x\": {\"type\": \"ordinal\", \"field\": \"predicted\"}, \"y\": {\"type\": \"ordinal\", \"field\": \"actual\"}}, \"height\": 215, \"width\": 215, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-25d663b7415ade2ca5e623f296f30d6d\": [{\"predicted\": \"fraud\", \"actual\": \"fraud\", \"raw_count\": 10539, \"value\": 0.9225315126050421}, {\"predicted\": \"fraud\", \"actual\": \"legitimate\", \"raw_count\": 885, \"value\": 0.07746848739495799}, {\"predicted\": \"legitimate\", \"actual\": \"fraud\", \"raw_count\": 58896, \"value\": 0.09598810905250531}, {\"predicted\": \"legitimate\", \"actual\": \"legitimate\", \"raw_count\": 554680, \"value\": 0.9040118909474947}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from mlworkflows import plot\n",
    "df, chart = plot.binary_confusion_matrix(test[\"label\"], predictions)\n",
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Viewing the raw counts, as well as the proportions of correctly and incorrectly classified items, emphasises that the model often misclassifies 'fraudulent' transactions as 'legitimate'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predicted</th>\n",
       "      <th>actual</th>\n",
       "      <th>raw_count</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>fraud</td>\n",
       "      <td>fraud</td>\n",
       "      <td>10539</td>\n",
       "      <td>0.922532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>fraud</td>\n",
       "      <td>legitimate</td>\n",
       "      <td>885</td>\n",
       "      <td>0.077468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>legitimate</td>\n",
       "      <td>fraud</td>\n",
       "      <td>58896</td>\n",
       "      <td>0.095988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>legitimate</td>\n",
       "      <td>legitimate</td>\n",
       "      <td>554680</td>\n",
       "      <td>0.904012</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    predicted      actual  raw_count     value\n",
       "0       fraud       fraud      10539  0.922532\n",
       "1       fraud  legitimate        885  0.077468\n",
       "2  legitimate       fraud      58896  0.095988\n",
       "3  legitimate  legitimate     554680  0.904012"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to save the model so that we can use it outside of this notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "cp.dump(lr, open(\"lr.pkl\", \"wb\"))\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.14",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}