stop_early.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "stop_early.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMSwuZ7fnLtZPg0S8/O2NH1"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C6nNsB7piGpF"
      },
      "source": [
        "# stopping training early to mitigate overfitting\n",
        "\n",
        "During stochastic gradient descent and other similar training processes, the model will typically tend to 'fit' the training data better and better as training proceeds.\n",
        "\n",
        "At the beginning of the training process, the model's parameters are initialized with 'random' values, so the model usually doesn't 'fit' the training data at all.\n",
        "\n",
        "The gradient information provided by backpropagation allows the model's parameter values to quickly improve the model's fit to the training data over the first few epochs of updates. Depending on the number of parameters in the model, this can happen very quickly (for small models), or it can take a larger number of epochs (for larger, more complex models).\n",
        "\n",
        "In many cases, during the first part of the training process, the model 'learns' to adjust its parameter values so they accurately represent the *patterns* in the training data. Only *after* the patterns in the data have been fit does the model begin to 'overfit' the error in the data.\n",
        "\n",
        "If we could somehow 'diagnose' when the model stops fitting the *patterns* in the data and *starts* to overfit the *error* in the data, we could potentially *stop* the training process early, *before* overfitting occurs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dX92QrVt0RRl"
      },
      "source": [
        "Run the code cell below, which simulates 100 data samples, splits them into 60% training and 40% validation data, and fits a fairly-complex neural network model to the data for 500 training epochs.\n",
        "\n",
        "Observe the model's \"loss\" on the training data and its \"val_loss\" on the validation data over the course of the training run."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WnTTz2D8iCAW"
      },
      "source": [
        "import sklearn.datasets\n",
        "import sklearn.model_selection\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "\n",
        "# simulate data\n",
        "x,y = sklearn.datasets.make_regression(n_samples=100,\n",
        "                                       n_features=2,\n",
        "                                       bias=100.0,\n",
        "                                       noise=10.0,\n",
        "                                       random_state=201188)\n",
        "y /= 100.0\n",
        "\n",
        "# partition into train and validation subsets\n",
        "train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(x,\n",
        "                                                                              y,\n",
        "                                                                              test_size=40,\n",
        "                                                                              random_state=221882)\n",
        "\n",
        "# package training and validation data into tensorflow Dataset objects\n",
        "train_data = tf.data.Dataset.from_tensor_slices((train_x, train_y)).batch(10)\n",
        "valid_data = tf.data.Dataset.from_tensor_slices((valid_x, valid_y)).batch(10)\n",
        "\n",
        "# build model\n",
        "model = tf.keras.models.Sequential()\n",
        "model.add(tf.keras.layers.Dense(units=4, activation=tf.keras.activations.relu, input_shape=[2]))\n",
        "model.add(tf.keras.layers.Dense(units=4, activation=tf.keras.activations.relu))\n",
        "model.add(tf.keras.layers.Dense(units=1))\n",
        "\n",
        "model.compile(optimizer=tf.keras.optimizers.SGD(),\n",
        "              loss=tf.keras.losses.MeanSquaredError())\n",
        "model.summary()\n",
        "\n",
        "# fit model\n",
        "model.fit(train_data, epochs=500, validation_data=valid_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wcp2dFQiiFO5"
      },
      "source": [
        "You may need to run the training process a few times, to get a feeling for when and how overfitting will occur in this example. Given that the data set is very small and the batch size is also small, each training run is likely to be highly variable.\n",
        "\n",
        "However, by the end of each run, you'll likely notice that the model's \"loss\" on the training data is quite a bit lower than its \"val_loss\" on the validation data, suggesting the model is overfitting."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ktt6B0tBouo6"
      },
      "source": [
        "## early stopping in tensorflow\n",
        "\n",
        "In tensorflow, we can implement early-stopping during training using a Keras \"Callback\" object.\n",
        "\n",
        "A Callback is an object that 'monitors' the neural network training process as it runs and performs specific actions whenever specific conditions are met. Callback objects can do all sorts of things during training, including:\n",
        "\n",
        "*   logging training information to a file\n",
        "*   saving the training state to disk, so training can be re-started from a checkpoint\n",
        "*   altering how the training process runs\n",
        "*   stopping the training process early\n",
        "\n",
        "In this notebook, we'll use a tf.keras.callbacks.EarlyStopping object to implement early-stopping and evaluate the extent to which early-stopping can mitigate model overfitting in our example.\n",
        "\n",
        "By default, the tf.keras.callbacks.EarlyStopping object monitors the \"val_loss\" (ie, the model's loss on the validation data) as training proceeds and stops the training process whenever the val_loss fails to improve. This is a fairly standard way to implement early-stopping during neural network training. It relies on the hypothesis that the model's validation loss will 'plateau' *before* significant model overfitting occurs.\n",
        "\n",
        "To implement early-stopping in tensorflow, we need to instantiate a tf.keras.callbacks.EarlyStopping object and provide it as the \"callbacks\" option to the model.fit(...) method call. The code will look like this:\n",
        "\n",
        "    callbacks=[tf.keras.callbacks.EarlyStopping()]\n",
        "\n",
        "the:\n",
        "\n",
        "    tf.keras.callbacks.EarlyStopping()\n",
        "\n",
        "part creates a new EarlyStopping object.\n",
        "\n",
        "The square brackets:\n",
        "\n",
        "    [ ... ]\n",
        "\n",
        "packages the EarlyStopping object into a python list, and the:\n",
        "\n",
        "    callbacks=...\n",
        "\n",
        "part sets the \"callbacks\" option during the model.fit(...) method call.\n",
        "\n",
        "The complete code example is shown in the following code cell. The relevant line is #35 at the very end.\n",
        "\n",
        "Notice that the *only* thing we needed to change to implement early-stopping during training is to supply the \"callbacks\" option when we fit the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FUkbpeR24isH"
      },
      "source": [
        "import sklearn.datasets\n",
        "import sklearn.model_selection\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "\n",
        "# simulate data\n",
        "x,y = sklearn.datasets.make_regression(n_samples=100,\n",
        "                                       n_features=2,\n",
        "                                       bias=100.0,\n",
        "                                       noise=10.0,\n",
        "                                       random_state=201188)\n",
        "y /= 100.0\n",
        "\n",
        "# partition into train and validation subsets\n",
        "train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(x,\n",
        "                                                                              y,\n",
        "                                                                              test_size=40,\n",
        "                                                                              random_state=221882)\n",
        "\n",
        "# package training and validation data into tensorflow Dataset objects\n",
        "train_data = tf.data.Dataset.from_tensor_slices((train_x, train_y)).batch(10)\n",
        "valid_data = tf.data.Dataset.from_tensor_slices((valid_x, valid_y)).batch(10)\n",
        "\n",
        "# build model\n",
        "model = tf.keras.models.Sequential()\n",
        "model.add(tf.keras.layers.Dense(units=4, activation=tf.keras.activations.relu, input_shape=[2]))\n",
        "model.add(tf.keras.layers.Dense(units=4, activation=tf.keras.activations.relu))\n",
        "model.add(tf.keras.layers.Dense(units=1))\n",
        "\n",
        "model.compile(optimizer=tf.keras.optimizers.SGD(),\n",
        "              loss=tf.keras.losses.MeanSquaredError())\n",
        "model.summary()\n",
        "\n",
        "# fit model\n",
        "model.fit(train_data, epochs=500, validation_data=valid_data, callbacks=[tf.keras.callbacks.EarlyStopping()])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kc32T7py5ALy"
      },
      "source": [
        "You may have noticed that the training run is halting after only a single epoch!\n",
        "\n",
        "Try re-running the code cell a few times... \n",
        "\n",
        "Yep, it stops after a single epoch of training (just about) every time. Not exactly what we wanted.\n",
        "\n",
        "During the very early stages of training, the model's loss and val_loss can fluctuate pretty wildly, as the randomly-initialized model parameter values start to 'hone in' to identify the patterns in the data.\n",
        "\n",
        "To avoid *premature* early-stopping during this initial phase of training, it's common practice to *turn off* early stopping and only enable it after a few training epochs have passed.\n",
        "\n",
        "This can be done in tensorflow by setting the \"patience\" option when you create the EarlyStopping callback object.\n",
        "\n",
        "The \"patience\" option sets the number of training epochs that must elapse with no improvement in the monitored value (val_loss, in this case) *before* early-stopping is triggered. The default value is 0, meaning that training will be stopped *whenever* the val_loss doesn't improve.\n",
        "\n",
        "In the following code cell, we set the patience option to 4 (line #38). The rest of the code is the same, although we reformatted the last line to make it a bit easier to read."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fbGQlW2Z9oyX"
      },
      "source": [
        "import sklearn.datasets\n",
        "import sklearn.model_selection\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "\n",
        "# simulate data\n",
        "x,y = sklearn.datasets.make_regression(n_samples=100,\n",
        "                                       n_features=2,\n",
        "                                       bias=100.0,\n",
        "                                       noise=10.0,\n",
        "                                       random_state=201188)\n",
        "y /= 100.0\n",
        "\n",
        "# partition into train and validation subsets\n",
        "train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(x,\n",
        "                                                                              y,\n",
        "                                                                              test_size=40,\n",
        "                                                                              random_state=221882)\n",
        "\n",
        "# package training and validation data into tensorflow Dataset objects\n",
        "train_data = tf.data.Dataset.from_tensor_slices((train_x, train_y)).batch(10)\n",
        "valid_data = tf.data.Dataset.from_tensor_slices((valid_x, valid_y)).batch(10)\n",
        "\n",
        "# build model\n",
        "model = tf.keras.models.Sequential()\n",
        "model.add(tf.keras.layers.Dense(units=4, activation=tf.keras.activations.relu, input_shape=[2]))\n",
        "model.add(tf.keras.layers.Dense(units=4, activation=tf.keras.activations.relu))\n",
        "model.add(tf.keras.layers.Dense(units=1))\n",
        "\n",
        "model.compile(optimizer=tf.keras.optimizers.SGD(),\n",
        "              loss=tf.keras.losses.MeanSquaredError())\n",
        "model.summary()\n",
        "\n",
        "# fit model\n",
        "model.fit(train_data, \n",
        "          epochs=500, \n",
        "          validation_data=valid_data, \n",
        "          callbacks=[tf.keras.callbacks.EarlyStopping(patience=4)])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "83KgaaaG94YG"
      },
      "source": [
        "Well, we've avoided *premature* early-stopping, but now it seems like the training run is proceeding through *all* 500 epochs, without triggering the early-stopping criterion.\n",
        "\n",
        "You can run the previous code cell a few times to convince yourself that it will (almost) always run through all 500 training epochs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P5LJjzAc-3gy"
      },
      "source": [
        "By default, tensorflow's EarlyStopping object will *only* trigger the early-stopping criterion when val_loss fails to improve *at all*.\n",
        "\n",
        "By setting the:\n",
        "\n",
        "    patience=4\n",
        "\n",
        "option, we've *also* required that val_loss must fail to improve *at all* for at least *4* epochs in a row, before early-stopping is triggered.\n",
        "\n",
        "Because stochastic gradient descent is stochastic, val_loss can fluctuate a bit during training, particularly when the data set and/or batch size is very small (like in this example!).\n",
        "\n",
        "To overcome this potential problem, we can set the \"min_delta\" option to some relatively *small* value, when we create our EarlyStopping callback.\n",
        "\n",
        "The \"min_delta\" option sets the *smallest* improvement in val_loss that will be counted as 'no improvement' by the early-stopping criterion. For example, if we set:\n",
        "\n",
        "    min_delta=0.1\n",
        "\n",
        "then an 'improvement' in val_loss of &leq;0.1 will be counted as *no improvement* in the evaluation of the early-stopping criterion. So, a sequence of 4 training epochs in which val_loss 'improves' by only 0.1 *would* trigger early-stopping.\n",
        "\n",
        "Let's try this in the following code cell."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IVLuSKQ0AOIo"
      },
      "source": [
        "import sklearn.datasets\n",
        "import sklearn.model_selection\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "\n",
        "# simulate data\n",
        "x,y = sklearn.datasets.make_regression(n_samples=100,\n",
        "                                       n_features=2,\n",
        "                                       bias=100.0,\n",
        "                                       noise=10.0,\n",
        "                                       random_state=201188)\n",
        "y /= 100.0\n",
        "\n",
        "# partition into train and validation subsets\n",
        "train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(x,\n",
        "                                                                              y,\n",
        "                                                                              test_size=40,\n",
        "                                                                              random_state=221882)\n",
        "\n",
        "# package training and validation data into tensorflow Dataset objects\n",
        "train_data = tf.data.Dataset.from_tensor_slices((train_x, train_y)).batch(10)\n",
        "valid_data = tf.data.Dataset.from_tensor_slices((valid_x, valid_y)).batch(10)\n",
        "\n",
        "# build model\n",
        "model = tf.keras.models.Sequential()\n",
        "model.add(tf.keras.layers.Dense(units=4, activation=tf.keras.activations.relu, input_shape=[2]))\n",
        "model.add(tf.keras.layers.Dense(units=4, activation=tf.keras.activations.relu))\n",
        "model.add(tf.keras.layers.Dense(units=1))\n",
        "\n",
        "model.compile(optimizer=tf.keras.optimizers.SGD(),\n",
        "              loss=tf.keras.losses.MeanSquaredError())\n",
        "model.summary()\n",
        "\n",
        "# fit model\n",
        "model.fit(train_data, \n",
        "          epochs=500, \n",
        "          validation_data=valid_data, \n",
        "          callbacks=[tf.keras.callbacks.EarlyStopping(patience=4, min_delta=0.1)])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EIULBoFqAVJr"
      },
      "source": [
        "In my case, setting:\n",
        "\n",
        "    patience=4\n",
        "\n",
        "and:\n",
        "\n",
        "    min_delta=0.1\n",
        "\n",
        "resulted in a training run of 22/500 epochs, so early-stopping was definitely triggered!.\n",
        "\n",
        "In my case, the final val_loss was 0.1754, and the loss on the *training* data was 0.0834. Your results are likely to vary, perhaps even quite a bit. But try re-running the previous code cell a few times, to see how the training run is likely to go, given the current early-stopping criteria.\n",
        "\n",
        "After a few runs, I was able to convince myself that, subjectively, it seems like the early-stopping criteria are not quite 'optimal'. Both the loss and val_loss values are relatively high, and the loss value is always a bit less than the val_loss. Additionally, <25/500 training epochs isn't a lot of training; it would be nice to have a slightly longer training run, so the model's parameter values have a chance to fit the data a little better.\n",
        "\n",
        "In practice, it can be pretty tricky to 'dial in' the options controlling the early-stopping criteria. The 'ideal' values will be highly dependent on the nature of the data, the model and the training procedure you are running.\n",
        "\n",
        "Take some time to try various combinations of early-stopping option values in the previous code cell, to see how they impact the training process and the resulting \"loss\" and \"val_loss\" values of the final model.\n",
        "\n",
        "For example, if you want to run the training process a bit longer, try either *increasing* the \"patience\" option or *decreasing* the \"min_delta\" option value. Increasing the \"patience\" value will require *more* epochs of 'no improvement' to trigger early-stopping. Decreasing the \"min_delta\" option will count only *very small* improvements in val_loss as 'no improvement'.\n",
        "\n",
        "As an example, I set:\n",
        "\n",
        "    min_delta=0.001\n",
        "\n",
        "and left:\n",
        "\n",
        "    patience=4\n",
        "\n",
        "This resulted in a training run of 176/500 epochs, final loss=0.0273, and val_loss=0.0850. As always, the results were pretty variable across replicate training runs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XLGMHBcUEXJ0"
      },
      "source": [
        "## evaluating the early-stopping approach to mitigating overfitting\n",
        "\n",
        "After playing around with various values for \"patience\" and \"min_delta\" in the previous code cell, were you able to reliably mitigate model overfitting using early-stopping criteria in this example?\n",
        "\n",
        "In my 'experiments', I was not able to find an early-stopping criterion that reliably produced final loss values (on the training data) that were not significantly lower than val_loss (on the validation data). So, in my case, I wasn't able to reliably mitigate model overfitting using early-stopping in this example.\n",
        "\n",
        "A quick 'scroll' back through the training output might suggest why early-stopping failed to mitigate overfitting in this example.\n",
        "\n",
        "Take a look at the values of \"loss\" and \"val_loss\" as the training run progressed.\n",
        "\n",
        "Do you notice that *both* \"loss\" and \"val_loss\" seem to nearly always *continuously decrease* throughout the *entire* training run? And \"loss\" is almost *always less than* \"val_loss\" throughout training.\n",
        "\n",
        "This pattern of training losses is *not* a good candidate for early-stopping.\n",
        "\n",
        "In some cases, you might observe that both \"loss\" and \"val_loss\" are *similar* to one another during early training, and only *later on* does loss become significantly lower than val_loss. In other cases, you might observe that \"val_loss\" actually starts to *increase* at some point during training. These patterns are much better-suited to using early-stopping to mitigate model overfitting.\n",
        "\n",
        "It just goes to show that there is no single approach that works well in *all* cases.\n",
        "\n",
        "Good neural-network 'engineers' will develop skills over time that allow them to select the right tool for a particular situation."
      ]
    }
  ]
}