keras_tuner_ann_sample.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mariajmolina/UMDAOSC650/blob/main/keras_tuner_ann_sample.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Choosing hyperparameters for your ANN\n",
        "\n",
        "How do you decide on how many hidden layers to use, how many nodes each layer should have, or what loss function to use?\n",
        "\n",
        "KerasTuner is an easy-to-use, scalable hyperparameter optimization framework that helps conduct a hyperparameter search for your specific problem.\n",
        "\n",
        "Docs: https://keras.io/keras_tuner/"
      ],
      "metadata": {
        "id": "qOksAfsbQssH"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "WSGzNdvgfyXP"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "import xarray as xr\n",
        "import sklearn\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "import keras\n",
        "from keras import layers"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# keras-tuner doesn't come installed in colab so lets install\n",
        "!pip install keras-tuner -q"
      ],
      "metadata": {
        "id": "klt6sxIYgDTx"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import keras_tuner"
      ],
      "metadata": {
        "id": "HhowxjsCf2Fc"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Similar to our first ann notebook, we will create some toy data to play with."
      ],
      "metadata": {
        "id": "ygw5psJCRTWK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "np.random.seed(123)\n",
        "\n",
        "# set the time range from 1999-2019\n",
        "times = pd.date_range(\"1999-01-01\", \"2019-12-31\", name=\"time\")\n",
        "\n",
        "# create an \"annual cycle\"\n",
        "annual_cycle = np.sin(2 * np.pi * (times.dayofyear.values / 365.25 - 0.28))\n",
        "\n",
        "# create some base \"temperatures\"\n",
        "base = 10 + 15 * annual_cycle.reshape(-1, 1)\n",
        "tmin_values = base + 3 * np.random.randn(annual_cycle.size, 1)\n",
        "tmax_values = base + 10 + 3 * np.random.randn(annual_cycle.size, 1)\n",
        "\n",
        "# let's put these dummy temperatures into an xarray dataset\n",
        "ds = xr.Dataset(\n",
        "    {\n",
        "        \"tmin\": ((\"time\", \"location\"), tmin_values),\n",
        "        \"tmax\": ((\"time\", \"location\"), tmax_values),\n",
        "    },\n",
        "    {\"time\": times, \"location\": [\"IA\"]},\n",
        ")"
      ],
      "metadata": {
        "id": "JuGfd3z8gXWi"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# lets split our data into train/test sets using dates\n",
        "train = ds.sel(time=slice('1999-01-01T00:00:00.000000000','2014-12-31T00:00:00.000000000'))\n",
        "test = ds.sel(time=slice('2015-01-01T00:00:00.000000000','2019-12-31T00:00:00.000000000'))\n",
        "\n",
        "# and grab the input feature and labels\n",
        "input_train = train.tmin\n",
        "label_train = train.tmax"
      ],
      "metadata": {
        "id": "e5aeZRTHgL5W"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create our scaler object\n",
        "scaler_input = MinMaxScaler()\n",
        "scaler_label = MinMaxScaler()\n",
        "\n",
        "# fit it to our training set\n",
        "scaler_input.fit(input_train)\n",
        "scaler_label.fit(label_train)\n",
        "\n",
        "# now transform our input data and labels\n",
        "input_train_scaled = scaler_input.transform(input_train)\n",
        "label_train_scaled = scaler_label.transform(label_train)"
      ],
      "metadata": {
        "id": "ISaBVKjDgRh8"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Searching hyperparameters\n",
        "\n",
        "To start our hyperparameter search, we will build our ANN sequentially, as we did in the first ann notebook. However, we will place this model into a function call build_model(hp). hp will be our hyperparameter object used during the search. Additionally, we will use hyperparameter objects with predefined ranges and sampling/steps, depending on what we want to explore."
      ],
      "metadata": {
        "id": "AR1PTx9LRpVy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def build_model(hp):\n",
        "    # start building model\n",
        "    model = keras.Sequential()\n",
        "    # input layer stays the same as before\n",
        "    model.add(keras.Input(shape=(1,), name=\"input_layer\"))\n",
        "    # instead of choosing the number of nodes, we add an hp object\n",
        "    hp_units_1 = hp.Int('units_1', min_value=2, max_value=12, step=1)\n",
        "    hp_units_2 = hp.Int('units_2', min_value=2, max_value=12, step=1)\n",
        "    model.add(\n",
        "        keras.layers.Dense(\n",
        "            units=hp_units_1,\n",
        "            activation=keras.activations.relu\n",
        "            )\n",
        "        )\n",
        "    model.add(\n",
        "        keras.layers.Dense(\n",
        "            units=hp_units_2,\n",
        "            activation=keras.activations.relu\n",
        "            )\n",
        "        )\n",
        "    # likewise, we will search for the learning rate\n",
        "    learning_rate = hp.Float(\n",
        "        \"lr\", min_value=1e-4, max_value=1e-2, sampling=\"log\"\n",
        "        )\n",
        "    model.compile(\n",
        "        keras.optimizers.SGD(learning_rate=learning_rate),\n",
        "        loss='mse'\n",
        "        )\n",
        "    return model"
      ],
      "metadata": {
        "id": "rq4eNl5dgRqi"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "hyperparameter method options include: https://keras.io/api/keras_tuner/hyperparameters/"
      ],
      "metadata": {
        "id": "BBM9njSUUd89"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "build_model(keras_tuner.HyperParameters())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_u7TZdv4WC05",
        "outputId": "8bf7b971-d42d-4d8f-bed5-a9f00e4f2802"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.src.engine.sequential.Sequential at 0x7e4db01ce8c0>"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next up, we initialize a tuner.\n",
        "\n",
        "Here we use RandomSearch (https://keras.io/api/keras_tuner/tuners/random/), but you can also use a Bayesian option (https://keras.io/api/keras_tuner/tuners/bayesian/).\n",
        "\n",
        "We use objective to specify the objective to select the best models, which in our case we base it on the validation set loss. We then use max_trials to specify the number of different models to try."
      ],
      "metadata": {
        "id": "_VrTBpIwUppJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#tuner = keras_tuner.RandomSearch(\n",
        "#    hypermodel=build_model,\n",
        "#    objective=\"val_loss\",\n",
        "#    max_trials=100,\n",
        "#    seed=123,\n",
        "#    overwrite=True\n",
        "#)\n",
        "\n",
        "tuner = keras_tuner.BayesianOptimization(\n",
        "    hypermodel=build_model,\n",
        "    objective=\"val_loss\",\n",
        "    max_trials=100,\n",
        "    num_initial_points=50,\n",
        "    alpha=0.0001,\n",
        "    beta=2.6,\n",
        "    seed=123,\n",
        "    # hyperparameters=None,\n",
        "    tune_new_entries=True,\n",
        "    allow_new_entries=True,\n",
        "    max_retries_per_trial=0,\n",
        "    max_consecutive_failed_trials=3,\n",
        "    overwrite=True\n",
        ")"
      ],
      "metadata": {
        "id": "3V1lpOmKh93A"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can print a summary of the search space:"
      ],
      "metadata": {
        "id": "nnbxe0T6WMlp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tuner.search_space_summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Edv-B-PNiIiZ",
        "outputId": "fef68f12-867f-4f34-dfb8-f4819de6c48d"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Search space summary\n",
            "Default search space size: 3\n",
            "units_1 (Int)\n",
            "{'default': None, 'conditions': [], 'min_value': 2, 'max_value': 12, 'step': 1, 'sampling': 'linear'}\n",
            "units_2 (Int)\n",
            "{'default': None, 'conditions': [], 'min_value': 2, 'max_value': 12, 'step': 1, 'sampling': 'linear'}\n",
            "lr (Float)\n",
            "{'default': 0.0001, 'conditions': [], 'min_value': 0.0001, 'max_value': 0.01, 'step': None, 'sampling': 'log'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Make sure input and label data are ready for training..."
      ],
      "metadata": {
        "id": "C5GLkq8dWWz1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "input_train_scaled.reshape(1,-1).squeeze().shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ckb61CVIjyN1",
        "outputId": "b1fe21ef-ed49-4303-89f3-8a77e8365d37"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(5844,)"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "label_train_scaled.reshape(1,-1).squeeze().shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "apeV4A7TkBuZ",
        "outputId": "b86ad1cb-082e-481d-96ad-f6ebeb59e192"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(5844,)"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now start the hyperparameter search! Here we set the first 5000 samples for training and the remaining samples as the validation set."
      ],
      "metadata": {
        "id": "ZlJXoToRWa_0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tuner.search(\n",
        "    input_train_scaled.reshape(1,-1).squeeze()[:5000],\n",
        "    label_train_scaled.reshape(1,-1).squeeze()[:5000],\n",
        "    epochs=20,\n",
        "    validation_data=(\n",
        "        input_train_scaled.reshape(1,-1).squeeze()[5000:],\n",
        "        label_train_scaled.reshape(1,-1).squeeze()[5000:])\n",
        "    )"
      ],
      "metadata": {
        "id": "Pr5J50MoiKzx",
        "outputId": "49899ca7-8340-4c8a-d7dd-47e86b6fb137",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Trial 100 Complete [00h 00m 09s]\n",
            "val_loss: 0.22613386809825897\n",
            "\n",
            "Best val_loss So Far: 0.008198599331080914\n",
            "Total elapsed time: 00h 16m 47s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Querying the results\n",
        "\n",
        "When hyperparameter search is over, you can retrieve the best model(s). The model is saved at its best performing epoch evaluated on the validation_loss because we set that as the objective."
      ],
      "metadata": {
        "id": "8eTQevp_WzBH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# we can now query the tuner object and grab the best models\n",
        "# here we grab the best two models\n",
        "models = tuner.get_best_models(num_models=2)"
      ],
      "metadata": {
        "id": "bJHkP2OxiWpw"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "models"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9LqK37RDXJlt",
        "outputId": "ed15922a-0454-4bc2-db78-cf828d92cd4c"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<keras.src.engine.sequential.Sequential at 0x7e4dafa5b370>,\n",
              " <keras.src.engine.sequential.Sequential at 0x7e4da4700640>]"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# here is the best model\n",
        "best_model = models[0]\n",
        "best_model.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nJqC7Y-7XLaW",
        "outputId": "479cc505-4cec-4fac-e670-8fdf662915e9"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense (Dense)               (None, 9)                 18        \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 2)                 20        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 38 (152.00 Byte)\n",
            "Trainable params: 38 (152.00 Byte)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# here is the second best model\n",
        "best_model = models[1]\n",
        "best_model.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mME219pKXPyE",
        "outputId": "94a314be-4f90-4220-ee87-6e3a5bec0d09"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense (Dense)               (None, 8)                 16        \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 2)                 18        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 34 (136.00 Byte)\n",
            "Trainable params: 34 (136.00 Byte)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can also print a summary of the search results."
      ],
      "metadata": {
        "id": "NOfyFZSvXa4d"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tuner.results_summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Bx2qodBmlP8c",
        "outputId": "635bb9a5-4783-443d-a16f-c78752607071"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Results summary\n",
            "Results in ./untitled_project\n",
            "Showing 10 best trials\n",
            "Objective(name=\"val_loss\", direction=\"min\")\n",
            "\n",
            "Trial 057 summary\n",
            "Hyperparameters:\n",
            "units_1: 9\n",
            "units_2: 2\n",
            "lr: 0.004401768822986867\n",
            "Score: 0.008198599331080914\n",
            "\n",
            "Trial 064 summary\n",
            "Hyperparameters:\n",
            "units_1: 8\n",
            "units_2: 2\n",
            "lr: 0.0012655560244433175\n",
            "Score: 0.009423202835023403\n",
            "\n",
            "Trial 022 summary\n",
            "Hyperparameters:\n",
            "units_1: 7\n",
            "units_2: 2\n",
            "lr: 0.003298757461307946\n",
            "Score: 0.010363856330513954\n",
            "\n",
            "Trial 065 summary\n",
            "Hyperparameters:\n",
            "units_1: 5\n",
            "units_2: 5\n",
            "lr: 0.00703649382753033\n",
            "Score: 0.010851570405066013\n",
            "\n",
            "Trial 077 summary\n",
            "Hyperparameters:\n",
            "units_1: 7\n",
            "units_2: 2\n",
            "lr: 0.003425133066484979\n",
            "Score: 0.012281861156225204\n",
            "\n",
            "Trial 005 summary\n",
            "Hyperparameters:\n",
            "units_1: 4\n",
            "units_2: 2\n",
            "lr: 0.0034928656354620575\n",
            "Score: 0.012883968651294708\n",
            "\n",
            "Trial 012 summary\n",
            "Hyperparameters:\n",
            "units_1: 6\n",
            "units_2: 3\n",
            "lr: 0.0010874771484415556\n",
            "Score: 0.014009159989655018\n",
            "\n",
            "Trial 009 summary\n",
            "Hyperparameters:\n",
            "units_1: 11\n",
            "units_2: 9\n",
            "lr: 0.009022079415210089\n",
            "Score: 0.049138378351926804\n",
            "\n",
            "Trial 041 summary\n",
            "Hyperparameters:\n",
            "units_1: 12\n",
            "units_2: 7\n",
            "lr: 0.0047156844970391324\n",
            "Score: 0.0522053986787796\n",
            "\n",
            "Trial 059 summary\n",
            "Hyperparameters:\n",
            "units_1: 12\n",
            "units_2: 9\n",
            "lr: 0.003802989312314547\n",
            "Score: 0.05479014292359352\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Retrain the model\n",
        "\n",
        "Keras-tuner also makes it easy to retrain a model using the set of hyperparameters that performed well.\n",
        "If you want to train the model with the entire dataset, you may retrieve the best hyperparameters and retrain the model by yourself."
      ],
      "metadata": {
        "id": "nY3xnVAEXoOh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the top hyperparameters.\n",
        "best_hps = tuner.get_best_hyperparameters(num_trials=1)"
      ],
      "metadata": {
        "id": "vHwtfeqZimEv"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "best_hps"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6-1njMTzXxZe",
        "outputId": "4cfc0f3d-ac66-48d9-d8b1-5243420ee18e"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<keras_tuner.src.engine.hyperparameters.hyperparameters.HyperParameters at 0x7e4daddc4c10>]"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Build the model with the best hp.\n",
        "model = build_model(best_hps[0])"
      ],
      "metadata": {
        "id": "NENhnlbjkxdY"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# now train!\n",
        "history = model.fit(\n",
        "    x=input_train_scaled.reshape(1,-1).squeeze(),\n",
        "    y=label_train_scaled.reshape(1,-1).squeeze(),\n",
        "    epochs=20,\n",
        "    validation_split=0.1,\n",
        "    batch_size=32\n",
        "    )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WY-BGK8pX1mj",
        "outputId": "7c1be700-c4b6-4592-ea91-4b564a8abb57"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "165/165 [==============================] - 1s 3ms/step - loss: 0.1673 - val_loss: 0.1775\n",
            "Epoch 2/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1589 - val_loss: 0.1760\n",
            "Epoch 3/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1584 - val_loss: 0.1757\n",
            "Epoch 4/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1583 - val_loss: 0.1755\n",
            "Epoch 5/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1582 - val_loss: 0.1754\n",
            "Epoch 6/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1581 - val_loss: 0.1753\n",
            "Epoch 7/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1580 - val_loss: 0.1753\n",
            "Epoch 8/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1580 - val_loss: 0.1752\n",
            "Epoch 9/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1579 - val_loss: 0.1751\n",
            "Epoch 10/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1579 - val_loss: 0.1751\n",
            "Epoch 11/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1578 - val_loss: 0.1750\n",
            "Epoch 12/20\n",
            "165/165 [==============================] - 0s 3ms/step - loss: 0.1578 - val_loss: 0.1750\n",
            "Epoch 13/20\n",
            "165/165 [==============================] - 1s 3ms/step - loss: 0.1577 - val_loss: 0.1749\n",
            "Epoch 14/20\n",
            "165/165 [==============================] - 0s 3ms/step - loss: 0.1577 - val_loss: 0.1749\n",
            "Epoch 15/20\n",
            "165/165 [==============================] - 1s 3ms/step - loss: 0.1577 - val_loss: 0.1749\n",
            "Epoch 16/20\n",
            "165/165 [==============================] - 1s 3ms/step - loss: 0.1576 - val_loss: 0.1748\n",
            "Epoch 17/20\n",
            "165/165 [==============================] - 1s 4ms/step - loss: 0.1576 - val_loss: 0.1748\n",
            "Epoch 18/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1576 - val_loss: 0.1748\n",
            "Epoch 19/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1576 - val_loss: 0.1747\n",
            "Epoch 20/20\n",
            "165/165 [==============================] - 0s 2ms/step - loss: 0.1576 - val_loss: 0.1747\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(range(0,20), history.history['loss'], c='k')\n",
        "plt.plot(range(0,20), history.history['val_loss'], c='r')\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 430
        },
        "id": "d2aE2Es0X60M",
        "outputId": "e0a304f7-ab39-4592-b818-ea3e7cfb2dee"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAj4AAAGdCAYAAAASUnlxAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABGGklEQVR4nO3dfVxUZcI38N8MMDMobwryZryIL6llRCITuvuwFSuaW7rrrdVdiWbaGunqbHsr95O6tnuDpRWfNW7JbiX38a6sfUpTs0LK6i4Ug/hsQpEWovLmKzO8CAMz1/PHPHNkZAaYEZgZ5vf9fK4Pc65zzXWuw2nk1znXmSMTQggQEREReQC5swdARERENFgYfIiIiMhjMPgQERGRx2DwISIiIo/B4ENEREQeg8GHiIiIPAaDDxEREXkMBh8iIiLyGN7OHoArMRqNqK2thb+/P2QymbOHQ0RERH0ghEBTUxMiIyMhl/d8TofBp4va2lpERUU5exhERETkgHPnzuGWW27psQ2DTxf+/v4ATL+4gIAAJ4+GiIiI+kKn0yEqKkr6O94TBp8uzJe3AgICGHyIiIjcTF+mqXByMxEREXkMBh8iIiLyGAw+RERE5DEYfIiIiMhjMPgQERGRx2DwISIiIo/B4ENEREQeg8GHiIiIPAaDDxEREXkMh4JPbm4uYmNjoVKpoFarUVxcbLNteXk55s+fj9jYWMhkMuTk5HRrY153Y8nIyAAAnDlzxup6mUyGd999V+rH2vq3337bkV0kIiKiIcju4LN3715oNBps3LgRpaWliI+PR1paGi5cuGC1fWtrK+Li4rB582aEh4dbbXPixAnU1dVJpaCgAACwYMECAEBUVJTF+rq6OmzatAl+fn6YPXu2RV/5+fkW7ebNm2fvLhIREdEQZfezul5++WUsW7YMS5YsAQDk5eXh0KFD2LVrF9atW9et/bRp0zBt2jQAsLoeAEaNGmWxvHnzZowdOxYpKSkAAC8vr26h6f3338fChQvh5+dnUR8UFGQzYBEREZFns+uMj16vR0lJCVJTU693IJcjNTUVRUVF/TIgvV6PPXv24IknnrD5sLGSkhKUlZVh6dKl3dZlZGQgJCQESUlJ2LVrF4QQNrfV3t4OnU5nUQbE1atAWhrQwyVBIiIiGnh2BZ9Lly7BYDAgLCzMoj4sLAz19fX9MqB9+/ahsbERixcvttlm586dmDRpEqZPn25R//zzz+Odd95BQUEB5s+fj6effhrbtm2z2U92djYCAwOlEhUV1S/70M1zzwGffALMng2Ulw/MNoiIiKhXdl/qGmg7d+7E7NmzERkZaXX9tWvX8Oabb2L9+vXd1nWtS0hIQEtLC7Zs2YJVq1ZZ7SszMxMajUZa1ul0AxN+Nm8GSkqA48eBX/8a+J//AeLi+n87RERE1CO7zviEhITAy8sLDQ0NFvUNDQ39Mq+muroaR44cwZNPPmmzzT/+8Q+0trZi0aJFvfanVqtx/vx5tLe3W12vVCoREBBgUQaEvz/w4YfA7bcDdXVAaipQWzsw2yIiIiKb7Ao+CoUCU6dORWFhoVRnNBpRWFiI5OTkmx5Mfn4+QkNDMWfOHJttdu7ciQcffLDbhGhrysrKMGLECCiVypse200bOdJ0uWvsWKCqynTm5/JlZ4+KiIjIo9h9qUuj0SA9PR2JiYlISkpCTk4OWlpapLu8Fi1ahNGjRyM7OxuAabJyRUWF9LqmpgZlZWXw8/PDuHHjpH6NRiPy8/ORnp4Ob2/rwzp9+jS++OILfPjhh93WHThwAA0NDbj77ruhUqlQUFCArKwsPPvss/bu4sCJiACOHAF+8QugogKYNQsoLAQG6kwTERERWRIO2LZtm4iOjhYKhUIkJSWJY8eOSetSUlJEenq6tFxVVSUAdCspKSkWfX788ccCgKisrLS53czMTBEVFSUMBkO3dYcPHxZ33nmn8PPzE8OHDxfx8fEiLy/PaltbtFqtACC0Wm2f3+OQigohgoOFAIT41a+EaG0d2O0RERENYfb8/ZYJ0cP93h5Gp9MhMDAQWq124Ob7mH3zDXDvvUBTE/DAA8D//b+Aj8/AbpOIiGgIsufvN5/V5SyJicCBA4BKZfq5eDFgNDp7VEREREMag48zpaQA//gH4O0NvPkm8MwzAE/AERERDRgGH2ebMwf4P/8HkMmA7duB//2/nT0iIiKiIYvBxxU8/DCQl2d6nZ0NvPCCc8dDREQ0RDH4uIrly68HnnXrgNdec+54iIiIhiAGH1fyb/8GZGaaXq9YAbz9tnPHQ0RENMQw+Lia//gPU+gRAnj8ceDQIWePiIiIaMhg8HE1Mhnw6qvAv/4r0NkJ/Mu/AJ9/7uxRERERDQkMPq5ILgfeeMP0xYZtbaaf33zj7FERERG5PQYfV+XjA7zzDvCrX5m+3XnWLNPzvYiIiMhhDD6uTKUCPvgAmDbN9CT3X//a9GR3IiIicgiDj6vz9wcOHwYmTwZqa03hp67O2aMiIiJySww+7iA4GCgoAMaMAX76CZg5E7hyxdmjIiIicjsMPu4iMhI4cgSIiABOngRmzzbN/SEiIqI+Y/BxJ3FxpjM/I0cCxcXAvHmmu76IiIioTxh83M1ttwEffQT4+QGffmp6zldHh7NHRURE5BYYfNzRtGmmu72USmD/fuCJJwCj0dmjIiIicnkMPu7qnnuAd98FvLyAPXuAlSuB1lZnj4qIiMilMfi4swceAP7+d9NjLv7zP4GAACA+Hli2DNixAygr42UwIiKiLmRCCOHsQbgKnU6HwMBAaLVaBAQEOHs4fbd7N/Dv/276np8bqVTAXXeZLo9NmwYkJQHjxpnCEhER0RBgz99vBp8u3Db4AKanudfUACdOmEpxsen5Xlpt97ZBQZZBaNo00+3yREREbojBx0FuHXysMRqBU6csw9C33wLt7d3bjh5tGYQSE00BiYiIyMUx+DhoyAUfazo6gO++ux6ETpwAysut3xU2YcL1MDRlChAWZiojRpieIE9EROQCGHwc5BHBx5qWFqC01DIM/fyz7fZeXsCoUaYQFBpq+6e5KJWDty9ERORxGHwc5LHBx5pLl0xzhMxh6NQp4MIF4OpV+/sKDLQeirrWjRplKkFBPJtERER2YfBxEINPH+j1wMWLphDU0NDzzwsXgM5O+/r38jI9lNUchEJCrr+2thwcDCgUA7OvRETkFuz5++09SGOioUKhME2EHj2697ZGI9DYaBmErIWkhgbTGSadDjAYrrftq8DA3oPSyJGmuUnm4uPj8K+AiIjcF4MPDRy53BQ4Ro4EJk3qvX17uykAXbpkOqt08aLl6xuXL182hSut1lROn+772Pz8TAHoxkDUddna68BA01kpIiJySww+5DqUyr6fTQJMoefq1e6ByNrrq1eBK1euf69Rc7OpnDtn/zgDA7sHJn9/U5jy9+9erNX7+fGsExGREzD4kPuSy01zfIKD+/4eg8EUfq5cMYUhc+m6bOt1c7OpD/MZpqqqmxu/Utl7QDKXwMDrJSjIctnPj9/ETUTURww+5Fm8vK5ffrOXXm+as3RjKGpsBJqaLEtzc/c6c9HrTf21t1+/vHcz5HLTc9q6hiFbIcla8fMDhg/nGSgi8ggOBZ/c3Fxs2bIF9fX1iI+Px7Zt25CUlGS1bXl5OTZs2ICSkhJUV1fjlVdewerVqy3axMbGorq6utt7n376aeTm5gIAfvWrX+Hzzz+3WP/UU08hLy9PWj579ixWrFiBzz77DH5+fkhPT0d2dja8vZnvqB8oFNdvxb8Zer3tYGStXqe7fpbpxtLZeX0SeWPjzY3Lx8cUgPq7qFSmwrlRROQC7E4Ee/fuhUajQV5eHtRqNXJycpCWlobKykqEWvmD0Nrairi4OCxYsABr1qyx2ueJEydgMBik5ZMnT+LXv/41FixYYNFu2bJleP7556XlYcOGSa8NBgPmzJmD8PBwfP3116irq8OiRYvg4+ODrKwse3eTaOAoFI6fdepKCODaNVPgsRWMbJWu7zF/9jo6+idA2eLtfT0EKZXXX1srfV3v62t5mbDr5UKVipcAiagbu7/HR61WY9q0aXj11VcBAEajEVFRUVi5ciXWrVvX43tjY2OxevXqbmd8brR69WocPHgQp06dguz//8P1q1/9CnfeeSdycnKsvufw4cP4zW9+g9raWoSFhQEA8vLysHbtWly8eBGKPnzXC7/HhzyOEKbLbS0tQGur6Wd/ldZW68+FGyxeXqYg1DUMWQtIPdUNG2YKV+aiUpnOjDFQEbmUAfseH71ej5KSEmRmZkp1crkcqampKCoqcmy0VraxZ88eaDQaKfSY/fd//zf27NmD8PBwPPDAA1i/fr101qeoqAhTpkyRQg8ApKWlYcWKFSgvL0dCQkK3bbW3t6O9yz/MOp2uX/aByG3IZNfPotgzSbyvOjtN4aetzVS6vrZW7F3f2mp5ebC52RS6gOsT2c138vUXudwyCFl73dvyja/Nyze+7lrHS/ZE/cKuT9KlS5dgMBgswgUAhIWF4YcffuiXAe3btw+NjY1YvHixRf2//uu/IiYmBpGRkfjnP/+JtWvXorKyEu+99x4AoL6+3uq4zOusyc7OxqZNm/pl3ERkhbe3qQwfPnjbNBhMgejGeVM3BqS+1LW2mi4ntrVd799ovH5WazB1vVTYW2DqeklQoTCdpVIoei+OtPPy4hkwcisu978QO3fuxOzZsxEZGWlRv3z5cun1lClTEBERgfvuuw8//fQTxo4d69C2MjMzodFopGWdToeoqCjHBk5ErsHL6/olq/5iviR47dr1IGTttT3rzMvm0nXZ/Np8ByBgOntm/v4pV3NjIOoajByt8/ExFW/v66+tLfelja3lrj+9vRngPIRdwSckJAReXl5oaGiwqG9oaEB4ePhND6a6uhpHjhyRzuL0RK1WAwBOnz6NsWPHIjw8HMXFxd3GBcDm2JRKJZR8cjgR9abrJcERIwZvu0Zj7+HI1utr10wT1vX63ktf2+n11p+/Z17n7ry8roegroHI2uu+rO+pdA13/dHOVuFDn7uxK/goFApMnToVhYWFmDdvHgDT5ObCwkI888wzNz2Y/Px8hIaGYs6cOb22LSsrAwBEREQAAJKTk/Ef//EfuHDhgnR3WUFBAQICAjB58uSbHhsR0aCTy00TrLvcwep0RuP1oHTjT2t19rS5sa6jwxS0zK/7stzX91hjMJiKMyfl9ze53P6wdGPY6u/lqChgwgSn/UrsvtSl0WiQnp6OxMREJCUlIScnBy0tLViyZAkAYNGiRRg9ejSys7MBmCYrV1RUSK9rampQVlYGPz8/jBs3TurXaDQiPz8f6enp3b5356effsKbb76J+++/H8HBwfjnP/+JNWvW4H/9r/+FO+64AwAwc+ZMTJ48GY8//jhefPFF1NfX47nnnkNGRgbP6hAR9Re53DR3yJ3/XRXCFHA6O6+Hoq4/bb3u6/obg1hfStew52ixFuiMxutfluoqli8HXnvNaZu3O/g89NBDuHjxIjZs2ID6+nrceeed+Oijj6SJxGfPnoW8y6m12tpaizuqtm7diq1btyIlJQVHjx6V6o8cOYKzZ8/iiSee6LZNhUKBI0eOSCErKioK8+fPx3PPPSe18fLywsGDB7FixQokJydj+PDhSE9Pt/jeHyIiIshk1y9NDSVCOB68egpgXetuXO/Iultuceqvye7v8RnK+D0+RERE7seev9+c9UREREQeg8GHiIiIPAaDDxEREXkMBh8iIiLyGAw+RERE5DEYfIiIiMhjMPgQERGRx2DwISIiIo/B4ENEREQeg8GHiIiIPAaDDxEREXkMBh8iIiLyGAw+RERE5DEYfIiIiMhjMPgQERGRx2DwISIiIo/B4ENEREQeg8GHiIiIPAaDDxEREXkMBh8iIiLyGAw+RERE5DEYfIiIiMhjMPgQERGRx2DwISIiIo/B4ENEREQeg8GHiIiIPAaDDxEREXkMBh8iIiLyGAw+RERE5DEYfIiIiMhjMPgQERGRx3Ao+OTm5iI2NhYqlQpqtRrFxcU225aXl2P+/PmIjY2FTCZDTk5OtzbmdTeWjIwMAMCVK1ewcuVK3HrrrfD19UV0dDRWrVoFrVZr0Y+1Pt5++21HdpGIiIiGILuDz969e6HRaLBx40aUlpYiPj4eaWlpuHDhgtX2ra2tiIuLw+bNmxEeHm61zYkTJ1BXVyeVgoICAMCCBQsAALW1taitrcXWrVtx8uRJvPHGG/joo4+wdOnSbn3l5+db9DVv3jx7d5GIiIiGKJkQQtjzBrVajWnTpuHVV18FABiNRkRFRWHlypVYt25dj++NjY3F6tWrsXr16h7brV69GgcPHsSpU6cgk8mstnn33Xfx2GOPoaWlBd7e3qadkcnw/vvvOxx2dDodAgMDodVqERAQ4FAfRERENLjs+ftt1xkfvV6PkpISpKamXu9ALkdqaiqKioocG62VbezZswdPPPGEzdADQNo5c+gxy8jIQEhICJKSkrBr1y70lOva29uh0+ksChEREQ1d3r03ue7SpUswGAwICwuzqA8LC8MPP/zQLwPat28fGhsbsXjx4h7H8Ze//AXLly+3qH/++edx7733YtiwYfjkk0/w9NNPo7m5GatWrbLaT3Z2NjZt2tQv4yYiIiLXZ1fwGQw7d+7E7NmzERkZaXW9TqfDnDlzMHnyZPz5z3+2WLd+/XrpdUJCAlpaWrBlyxabwSczMxMajcai76ioqJvfCSIiInJJdl3qCgkJgZeXFxoaGizqGxoabE5ctkd1dTWOHDmCJ5980ur6pqYmzJo1C/7+/nj//ffh4+PTY39qtRrnz59He3u71fVKpRIBAQEWhYiIiIYuu4KPQqHA1KlTUVhYKNUZjUYUFhYiOTn5pgeTn5+P0NBQzJkzp9s6nU6HmTNnQqFQ4IMPPoBKpeq1v7KyMowYMQJKpfKmx0ZERETuz+5LXRqNBunp6UhMTERSUhJycnLQ0tKCJUuWAAAWLVqE0aNHIzs7G4BpsnJFRYX0uqamBmVlZfDz88O4ceOkfo1GI/Lz85Gent5twrI59LS2tmLPnj0WE5FHjRoFLy8vHDhwAA0NDbj77ruhUqlQUFCArKwsPPvss479ZoiIiGjIsTv4PPTQQ7h48SI2bNiA+vp63Hnnnfjoo4+kCc9nz56FXH79RFJtbS0SEhKk5a1bt2Lr1q1ISUnB0aNHpfojR47g7NmzeOKJJ7pts7S0FMePHwcAi7AEAFVVVYiNjYWPjw9yc3OxZs0aCCEwbtw4vPzyy1i2bJm9u0hERERDlN3f4zOU8Xt8iIiI3M+AfY8PERERkTtj8CEiIiKPweBDREREHoPBh4iIiDwGgw8RERF5DAYfIiIi8hgMPkREROQxGHyIiIjIYzD4EBERkcdg8CEiIiKPweBDREREHoPBh4iIiDwGgw8RERF5DAYfIiIi8hgMPkREROQxGHyIiIjIYzD4EBERkcdg8CEiIiKPweBDREREHoPBh4iIiDwGgw8RERF5DAYfIiIi8hgMPkREROQxGHyIiIjIYzD4EBERkcdg8CEiIiKPweBDREREHoPBh4iIiDwGgw8RERF5DAYfIiIi8hgMPkREROQxHAo+ubm5iI2NhUqlglqtRnFxsc225eXlmD9/PmJjYyGTyZCTk9OtjXndjSUjI0Nq09bWhoyMDAQHB8PPzw/z589HQ0ODRT9nz57FnDlzMGzYMISGhuJPf/oTOjs7HdlFIiIiGoLsDj579+6FRqPBxo0bUVpaivj4eKSlpeHChQtW27e2tiIuLg6bN29GeHi41TYnTpxAXV2dVAoKCgAACxYskNqsWbMGBw4cwLvvvovPP/8ctbW1+N3vfietNxgMmDNnDvR6Pb7++mvs3r0bb7zxBjZs2GDvLhIREdFQJeyUlJQkMjIypGWDwSAiIyNFdnZ2r++NiYkRr7zySq/t/vCHP4ixY8cKo9EohBCisbFR+Pj4iHfffVdq8/333wsAoqioSAghxIcffijkcrmor6+X2mzfvl0EBASI9vb2Pu2bVqsVAIRWq+1TeyIiInI+e/5+23XGR6/Xo6SkBKmpqVKdXC5HamoqioqK+iWI6fV67NmzB0888QRkMhkAoKSkBB0dHRbbnThxIqKjo6XtFhUVYcqUKQgLC5PapKWlQafToby83Oq22tvbodPpLAoRERENXXYFn0uXLsFgMFiECwAICwtDfX19vwxo3759aGxsxOLFi6W6+vp6KBQKBAUF2dxufX291XGZ11mTnZ2NwMBAqURFRfXLPhAREZFrcrm7unbu3InZs2cjMjJywLeVmZkJrVYrlXPnzg34NomIiMh5vO1pHBISAi8vr253UzU0NNicuGyP6upqHDlyBO+9955FfXh4OPR6PRobGy3O+nTdbnh4eLe7y8zjtDU2pVIJpVJ50+MmIiIi92DXGR+FQoGpU6eisLBQqjMajSgsLERycvJNDyY/Px+hoaGYM2eORf3UqVPh4+Njsd3KykqcPXtW2m5ycjK+++47i7vLCgoKEBAQgMmTJ9/02IiIiMj92XXGBwA0Gg3S09ORmJiIpKQk5OTkoKWlBUuWLAEALFq0CKNHj0Z2djYA02TliooK6XVNTQ3Kysrg5+eHcePGSf0ajUbk5+cjPT0d3t6WwwoMDMTSpUuh0WgwcuRIBAQEYOXKlUhOTsbdd98NAJg5cyYmT56Mxx9/HC+++CLq6+vx3HPPISMjg2d1iIiIyMSR28a2bdsmoqOjhUKhEElJSeLYsWPSupSUFJGeni4tV1VVCQDdSkpKikWfH3/8sQAgKisrrW7z2rVr4umnnxYjRowQw4YNE7/97W9FXV2dRZszZ86I2bNnC19fXxESEiL++Mc/io6Ojj7vF29nJyIicj/2/P2WCSGE82KXa9HpdAgMDIRWq0VAQICzh0NERER9YM/fb5e7q4uIiIhooDD4EBERkcdg8CEiIiKPweBDREREHoPBh4iIiDwGgw8RERF5DAYfIiIi8hgMPkREROQxGHyIiIjIYzD4EBERkcdg8CEiIiKPweBDREREHoPBh4iIiDwGgw8RERF5DAYfIiIi8hgMPkREROQxGHyIiIjIYzD4EBERkcdg8CEiIiKPweBDREREHoPBh4iIiDwGgw8RERF5DAYfIiIi8hgMPkREROQxGHyIiIjIYzD4EBERkcdg8CEiIiKPweBDREREHoPBh4iIiDwGgw8RERF5DAYfIiIi8hgOBZ/c3FzExsZCpVJBrVajuLjYZtvy8nLMnz8fsbGxkMlkyMnJsdqupqYGjz32GIKDg+Hr64spU6bgm2++kdbLZDKrZcuWLVIb8za6ls2bNzuyi0RERDQE2R189u7dC41Gg40bN6K0tBTx8fFIS0vDhQsXrLZvbW1FXFwcNm/ejPDwcKttrl69ihkzZsDHxweHDx9GRUUFXnrpJYwYMUJqU1dXZ1F27doFmUyG+fPnW/T1/PPPW7RbuXKlvbtIREREQ5S3vW94+eWXsWzZMixZsgQAkJeXh0OHDmHXrl1Yt25dt/bTpk3DtGnTAMDqegB44YUXEBUVhfz8fKluzJgxFm1uDE379+/HPffcg7i4OIt6f39/mwGLiIiIPJtdZ3z0ej1KSkqQmpp6vQO5HKmpqSgqKnJ4EB988AESExOxYMEChIaGIiEhAa+//rrN9g0NDTh06BCWLl3abd3mzZsRHByMhIQEbNmyBZ2dnQ6Pi4iIiIYWu874XLp0CQaDAWFhYRb1YWFh+OGHHxwexM8//4zt27dDo9Hg3//933HixAmsWrUKCoUC6enp3drv3r0b/v7++N3vfmdRv2rVKtx1110YOXIkvv76a2RmZqKurg4vv/yy1e22t7ejvb1dWtbpdA7vAxEREbk+uy91DQSj0YjExERkZWUBABISEnDy5Enk5eVZDT67du3Co48+CpVKZVGv0Wik13fccQcUCgWeeuopZGdnQ6lUdusnOzsbmzZt6ue9ISIiIldl16WukJAQeHl5oaGhwaK+oaHhpubVREREYPLkyRZ1kyZNwtmzZ7u1/fLLL1FZWYknn3yy137VajU6Oztx5swZq+szMzOh1Wqlcu7cOYfG31cdHR0D2j8RERH1zK7go1AoMHXqVBQWFkp1RqMRhYWFSE5OdngQM2bMQGVlpUXdjz/+iJiYmG5td+7cialTpyI+Pr7XfsvKyiCXyxEaGmp1vVKpREBAgEUZCP/1X/+F0NBQPPPMMwPSPxEREfWN3Ze6NBoN0tPTkZiYiKSkJOTk5KClpUW6y2vRokUYPXo0srOzAZgmRFdUVEiva2pqUFZWBj8/P4wbNw4AsGbNGkyfPh1ZWVlYuHAhiouLsWPHDuzYscNi2zqdDu+++y5eeumlbuMqKirC8ePHcc8998Df3x9FRUVYs2YNHnvsMYvb4p1h2LBhuHjxovR7ICIiIicRDti2bZuIjo4WCoVCJCUliWPHjknrUlJSRHp6urRcVVUlAHQrKSkpFn0eOHBA3H777UKpVIqJEyeKHTt2dNvua6+9Jnx9fUVjY2O3dSUlJUKtVovAwEChUqnEpEmTRFZWlmhra+vzfmm1WgFAaLXaPr+nL8rKygQAMWLECGE0Gvu1byIiIk9nz99vmRBCOC92uRadTofAwEBotdp+vezV1taG4cOHw2g0ora2FhEREf3WNxERkaez5+83n9U1CFQqFcaOHQvA9AgPIiIicg4Gn0Fy2223AWDwISIiciYGn0HC4ENEROR8DD6DxBx8eGcXERGR8zD4DJKuZ3w4n5yIiMg5GHwGyYQJEyCXy9HY2Ii6ujpnD4eIiMgjMfgMEpVKJX1hI+f5EBEROQeDzyDiBGciIiLnYvAZRJzgTERE5FwMPoOIZ3yIiIici8FnEPHOLiIiIudi8BlEEyZMgJeXF7RaLWpra509HCIiIo/D4DOIlEol7+wiIiJyIgafQcYJzkRERM7D4DPIOMGZiIjIeRh8BhmDDxERkfMw+AyyyZMnA+CdXURERM7A4DPIzHd26XQ61NTUOHs4REREHoXBZ5AplUqMHz8eAC93ERERDTYGHyfgnV1ERETOweDjBJzgTERE5BwMPk7QdYIzERERDR4GHyfoeqmLd3YRERENHgYfJ5gwYQK8vb2h0+lw/vx5Zw+HiIjIYzD4OIFCoZDu7OIEZyIiosHD4OMknOBMREQ0+Bh8nITBh4iIaPAx+DgJ7+wiIiIafAw+TsI7u4iIiAYfg4+TjB8/Ht7e3mhqauKdXURERIOEwcdJFAoFJkyYAICXu4iIiAYLg48TcYIzERHR4HIo+OTm5iI2NhYqlQpqtRrFxcU225aXl2P+/PmIjY2FTCZDTk6O1XY1NTV47LHHEBwcDF9fX0yZMgXffPONtH7x4sWQyWQWZdasWRZ9XLlyBY8++igCAgIQFBSEpUuXorm52ZFdHBSc4ExERDS47A4+e/fuhUajwcaNG1FaWor4+HikpaXhwoULVtu3trYiLi4OmzdvRnh4uNU2V69exYwZM+Dj44PDhw+joqICL730EkaMGGHRbtasWairq5PKW2+9ZbH+0UcfRXl5OQoKCnDw4EF88cUXWL58ub27OGh4xoeIiGhwyYSdtxSp1WpMmzYNr776KgDAaDQiKioKK1euxLp163p8b2xsLFavXo3Vq1db1K9btw5fffUVvvzyS5vvXbx4MRobG7Fv3z6r67///ntMnjwZJ06cQGJiIgDgo48+wv3334/z588jMjKy133T6XQIDAyEVqtFQEBAr+1vVkVFBW677Tb4+flBp9NBJpMN+DaJiIiGGnv+ftt1xkev16OkpASpqanXO5DLkZqaiqKiIsdGC+CDDz5AYmIiFixYgNDQUCQkJOD111/v1u7o0aMIDQ3FrbfeihUrVuDy5cvSuqKiIgQFBUmhBwBSU1Mhl8tx/Phxq9ttb2+HTqezKINp/Pjx8PHxQXNzM86dOzeo2yYiIvJEdgWfS5cuwWAwICwszKI+LCwM9fX1Dg/i559/xvbt2zF+/Hh8/PHHWLFiBVatWoXdu3dLbWbNmoW///3vKCwsxAsvvIDPP/8cs2fPhsFgAADU19cjNDTUol9vb2+MHDnS5tiys7MRGBgolaioKIf3wRE+Pj68s4uIiGgQeTt7AIDpclliYiKysrIAAAkJCTh58iTy8vKQnp4OAHj44Yel9lOmTMEdd9yBsWPH4ujRo7jvvvsc2m5mZiY0Go20rNPpBj38TJ48GeXl5SgvL8fs2bMHddtERESexq4zPiEhIfDy8kJDQ4NFfUNDg82Jy30REREh3eFkNmnSJJw9e9bme+Li4hASEoLTp08DAMLDw7tNsO7s7MSVK1dsjk2pVCIgIMCiDDZOcCYiIho8dgUfhUKBqVOnorCwUKozGo0oLCxEcnKyw4OYMWMGKisrLep+/PFHxMTE2HzP+fPncfnyZURERAAAkpOT0djYiJKSEqnNp59+CqPRCLVa7fDYBhqDDxER0eCx+3Z2jUaD119/Hbt378b333+PFStWoKWlBUuWLAEALFq0CJmZmVJ7vV6PsrIylJWVQa/Xo6amBmVlZdKZGgBYs2YNjh07hqysLJw+fRpvvvkmduzYgYyMDABAc3Mz/vSnP+HYsWM4c+YMCgsLMXfuXIwbNw5paWkATGeIZs2ahWXLlqG4uBhfffUVnnnmGTz88MN9uqPLWfjMLiIiokEkHLBt2zYRHR0tFAqFSEpKEseOHZPWpaSkiPT0dGm5qqpKAOhWUlJSLPo8cOCAuP3224VSqRQTJ04UO3bskNa1traKmTNnilGjRgkfHx8RExMjli1bJurr6y36uHz5snjkkUeEn5+fCAgIEEuWLBFNTU193i+tVisACK1Wa98v5Cbo9Xrh4+MjAIgzZ84M2naJiIiGCnv+ftv9PT5D2WB/j4/Z7bffjvLychw6dAj333//oG2XiIhoKBiw7/GhgcF5PkRERIODwccFMPgQERENDgYfF9B1gjMRERENHAYfF9A1+BiNRiePhoiIaOhi8HEB48aNg4+PD1paWnr80kYiIiK6OQw+LsDb2xu33norAM7zISIiGkgMPi6CE5yJiIgGHoOPi2DwISIiGngMPi6Cd3YRERENPAYfF8E7u4iIiAYeg4+LGDt2LBQKBVpbW1FdXe3s4RAREQ1JDD4ugnd2ERERDTwGHxfCCc5EREQDi8HHhXCCMxER0cBi8HEhPONDREQ0sBh8XMjkyZMBAN9//z3v7CIiIhoADD4upOudXWfOnHH2cIiIiIYcBh8X4u3tjYkTJwLg5S4iIqKBwODjYjjPh4iIaOAw+LgY3tlFREQ0cBh8XAzP+BAREQ0cBh8Xwzu7iIiIBg6Dj4sZO3YslEolrl27hqqqKmcPh4iIaEhh8HExXl5evLOLiIhogDD4uCBOcCYiIhoYDD4uiBOciYiIBgaDjwsyT3Bm8CEiIupfDD4uyHzG5/vvv4fBYHDyaIiIiIYOBh8XFBcXB5VKhba2Nt7ZRURE1I8YfFxQ1zu7OMGZiIio/zD4uChOcCYiIup/DgWf3NxcxMbGQqVSQa1Wo7i42Gbb8vJyzJ8/H7GxsZDJZMjJybHarqamBo899hiCg4Ph6+uLKVOm4JtvvgEAdHR0YO3atZgyZQqGDx+OyMhILFq0CLW1tRZ9mLfRtWzevNmRXXQ6TnAmIiLqf3YHn71790Kj0WDjxo0oLS1FfHw80tLScOHCBavtW1tbERcXh82bNyM8PNxqm6tXr2LGjBnw8fHB4cOHUVFRgZdeegkjRoyQ+igtLcX69etRWlqK9957D5WVlXjwwQe79fX888+jrq5OKitXrrR3F10Cz/gQERH1P5kQQtjzBrVajWnTpuHVV18FABiNRkRFRWHlypVYt25dj++NjY3F6tWrsXr1aov6devW4auvvsKXX37Z53GcOHECSUlJqK6uRnR0dI/995VOp0NgYCC0Wi0CAgIc6qO/nD59GuPHj4dKpUJzczO8vLycOh4iIiJXZc/fb7vO+Oj1epSUlCA1NfV6B3I5UlNTUVRU5NhoAXzwwQdITEzEggULEBoaioSEBLz++us9vker1UImkyEoKMiifvPmzQgODkZCQgK2bNmCzs5Om320t7dDp9NZFFcxZswY6c6un3/+2dnDISIiGhLsCj6XLl2CwWBAWFiYRX1YWBjq6+sdHsTPP/+M7du3Y/z48fj444+xYsUKrFq1Crt377bavq2tDWvXrsUjjzxikexWrVqFt99+G5999hmeeuopZGVl4d/+7d9sbjc7OxuBgYFSiYqKcngf+puXlxcmTZoEgHd2ERER9RdvZw8AMF0uS0xMRFZWFgAgISEBJ0+eRF5eHtLT0y3adnR0YOHChRBCYPv27RbrNBqN9PqOO+6AQqHAU089hezsbCiVym7bzczMtHiPTqdzqfBz22234dtvv0V5eTnmzp3r7OEQERG5PbvO+ISEhMDLywsNDQ0W9Q0NDTYnLvdFRESEdBeT2aRJk3D27FmLOnPoqa6uRkFBQa/X8dRqNTo7O3HmzBmr65VKJQICAiyKK+GdXURERP3LruCjUCgwdepUFBYWSnVGoxGFhYVITk52eBAzZsxAZWWlRd2PP/6ImJgYadkcek6dOoUjR44gODi4137Lysogl8sRGhrq8NiciXd2ERER9S+7L3VpNBqkp6cjMTERSUlJyMnJQUtLC5YsWQIAWLRoEUaPHo3s7GwApgnR5jkqer0eNTU1KCsrg5+fH8aNGwcAWLNmDaZPn46srCwsXLgQxcXF2LFjB3bs2AHAFHr+5V/+BaWlpTh48CAMBoM0p2jkyJFQKBQoKirC8ePHcc8998Df3x9FRUVYs2YNHnvsMem2eHdjDj4//PADDAYD7+wiIiK6WcIB27ZtE9HR0UKhUIikpCRx7NgxaV1KSopIT0+XlquqqgSAbiUlJcWizwMHDojbb79dKJVKMXHiRLFjx45e+wAgPvvsMyGEECUlJUKtVovAwEChUqnEpEmTRFZWlmhra+vzfmm1WgFAaLVaR34t/c5gMAhfX18BQPz444/OHg4REZFLsufvt93f4zOUudL3+JhNnToVpaWleP/99zFv3jxnD4eIiMjlDNj3+NDg4wRnIiKi/sPg4+I4wZmIiKj/MPi4OAYfIiKi/sPg4+LMwaeysrLHx28QERFR7xh8XFxsbCx8fX3R3t7OZ3YRERHdJAYfFyeXy6VndvFyFxER0c1h8HEDnOdDRETUPxh83ACDDxERUf9g8HEDDD5ERET9g8HHDfDOLiIiov7B4OMGYmJiMGzYMOj1evz000/OHg4REZHbYvBxA7yzi4iIqH8w+LgJzvMhIiK6eQw+boLBh4iI6OYx+LgJc/CpqKhw8kiIiIjcF4OPm+CdXURERDePwcdNREdHS3d2nT592tnDISIicksMPm5CLpdj8uTJADjPh4iIyFEMPm6EE5yJiIhuDoOPG+EEZyIiopvD4ONGeMaHiIjo5jD4uBHzHJ/Kykp0dHQ4eTRERETuh8HHjURHR2P48OHo6OjgnV1EREQOYPBxI7yzi4iI6OYw+LgZzvMhIiJyHIOPm+GdXURERI5j8HEzPONDRETkOAYfN2Oe4/Pjjz/yzi4iIiI7Mfi4mejoaPj5+aGjowOnTp1y9nCIiIjcCoOPm5HJZLyzi4iIyEEMPm6IE5yJiIgc41Dwyc3NRWxsLFQqFdRqNYqLi222LS8vx/z58xEbGwuZTIacnByr7WpqavDYY48hODgYvr6+mDJlCr755htpvRACGzZsQEREBHx9fZGamtrtUs+VK1fw6KOPIiAgAEFBQVi6dCmam5sd2UWXxgnOREREjrE7+OzduxcajQYbN25EaWkp4uPjkZaWhgsXLlht39rairi4OGzevBnh4eFW21y9ehUzZsyAj48PDh8+jIqKCrz00ksYMWKE1ObFF1/E3/72N+Tl5eH48eMYPnw40tLS0NbWJrV59NFHUV5ejoKCAhw8eBBffPEFli9fbu8uujxe6iIiInKQsFNSUpLIyMiQlg0Gg4iMjBTZ2dm9vjcmJka88sor3erXrl0rfvGLX9h8n9FoFOHh4WLLli1SXWNjo1AqleKtt94SQghRUVEhAIgTJ05IbQ4fPixkMpmoqanpy64JrVYrAAitVtun9s5SXV0tAAhvb2/R3t7u7OEQERE5lT1/v+0646PX61FSUoLU1FSpTi6XIzU1FUVFRQ6Hrw8++ACJiYlYsGABQkNDkZCQgNdff11aX1VVhfr6eovtBgYGQq1WS9stKipCUFAQEhMTpTapqamQy+U4fvy41e22t7dDp9NZFHcQFRUFf39/dHZ28s4uIiIiO9gVfC5dugSDwYCwsDCL+rCwMNTX1zs8iJ9//hnbt2/H+PHj8fHHH2PFihVYtWoVdu/eDQBS3z1tt76+HqGhoRbrvb29MXLkSJtjy87ORmBgoFSioqIc3ofB1PXOLk5wJiIi6juXuKvLaDTirrvuQlZWFhISErB8+XIsW7YMeXl5A7rdzMxMaLVaqZw7d25At9efOMGZiIjIfnYFn5CQEHh5eaGhocGivqGhwebE5b6IiIiQzmCYTZo0CWfPngUAqe+ethseHt5tgnVnZyeuXLlic2xKpRIBAQEWxV1wgjMREZH97Ao+CoUCU6dORWFhoVRnNBpRWFiI5ORkhwcxY8YMVFZWWtT9+OOPiImJAQCMGTMG4eHhFtvV6XQ4fvy4tN3k5GQ0NjaipKREavPpp5/CaDRCrVY7PDZXxTM+RERE9vO29w0ajQbp6elITExEUlIScnJy0NLSgiVLlgAAFi1ahNGjRyM7OxuAaUK0eR6KXq9HTU0NysrK4Ofnh3HjxgEA1qxZg+nTpyMrKwsLFy5EcXExduzYgR07dgAwzWlZvXo1/vrXv2L8+PEYM2YM1q9fj8jISMybNw+A6QzRrFmzpEtkHR0deOaZZ/Dwww8jMjLypn9RrsYcfE6dOgW9Xg+FQuHkEREREbkBR24b27Ztm4iOjhYKhUIkJSWJY8eOSetSUlJEenq6tFxVVSUAdCspKSkWfR44cEDcfvvtQqlUiokTJ4odO3ZYrDcajWL9+vUiLCxMKJVKcd9994nKykqLNpcvXxaPPPKI8PPzEwEBAWLJkiWiqampz/vlLrezC2H6fQQEBAgA4rvvvnP2cIiIiJzGnr/fMiGEcF7sci06nQ6BgYHQarVuMd8nOTkZx44dw969e7Fw4UJnD4eIiMgp7Pn77RJ3dZFjOM+HiIjIPgw+box3dhEREdmHwceN8YwPERGRfRh83FjXO7va29udPBoiIiLXx+DjxkaPHo2AgAAYDAY+s4uIiKgPGHzcmEwm4+UuIiIiOzD4uDlOcCYiIuo7Bh83xzM+REREfcfg4+YYfIiIiPqOwcfNmYPP6dOneWcXERFRLxh83FxkZCQCAwNhMBjw448/Ons4RERELo3Bx83JZDJpgvPvf/97/Od//ifOnz/v5FERERG5JgafIeC3v/0tAODrr79GRkYGoqKikJiYiL/+9a/47rvvwOfQEhERmfDp7F2429PZu6qsrMT+/fuxf/9+FBUVWYSdMWPGYN68eZg7dy5mzJgBb29vJ46UiIiof9nz95vBpwt3Dj5dNTQ04MCBA9i/fz8KCgosJj0HBwfjN7/5DebOnYuZM2di+PDhThwpERHRzWPwcdBQCT5dNTc345NPPsH+/ftx8OBBXLlyRVqnUqnw61//GnPnzsUDDzyA0NBQJ46UiIjIMQw+DhqKwaerzs5OfPXVV9i3bx/279+PqqoqaZ1MJsP06dMxd+5czJ07FxMmTHDiSImIiPqOwcdBQz34dCWEwMmTJ6UQVFJSYrF+0qRJUghKSkqCXM558ERE5JoYfBzkScHnRufOncMHH3yA/fv347PPPkNnZ6e0Ljw8HPfeey+ioqIwevRoixIeHs7J0kRE5FQMPg7y5ODTlVarxeHDh7Fv3z58+OGHaGpqstlWLpcjLCysWyC6sXjy75OIiAYWg4+DGHy60+v1OHr0KL799lvU1NSgtrYWNTU1qKmpQV1dncWZoZ74+fl1C0ORkZHS67CwMAQHB8PPzw8ymWyA94qIiIYSBh8HMfjYx2Aw4OLFi1IQslW0Wm2f+/Tx8cHIkSMRHBxsUXqrUygUA7inRETkyhh8HMTgMzBaWlqsBqKuZ48uXryItrY2h7fh5+fXY0AaMWIEgoKCEBQUhMDAQOl1QEAAJ24TEbk5Bh8HMfg4V2trKy5fvozLly/jypUr0uuu5cb6q1evwmg0OrxNmUyGgIAAizDUtVir71oXGBjIyd1ERE5mz99v/otNLmPYsGEYNmwYoqKi+vweo9EIrVbba0DSarVobGy0KG1tbRBCQKvVQqvV4uzZsw6Ne/jw4QgICLjpwst1REQDj2d8uuAZH8/S3t7eLRBZC0i26pqbm/t1PEqlEv7+/t0Ckb+/P/z9/eHn52f1p7W64cOH80wUEXkMnvEh6gOlUonQ0FCHH9XR2dkphaKmpibodDqHSktLCwBTEGtvb8elS5f6Zf98fX1thqWuP4cPH95jGTZsmMWyl5dXv4yPiMgZGHyIHOTt7S1NoL4ZnZ2daG5uthqKzGeWmpqa+vzT/BUD165dw7Vr13Dx4sX+2F2JUqnsMRhZC072FKVSya80IKIBw+BD5GTe3t7SZOmbJYSAXq/vU0Ayl9bWVrS0tNgs5vXmq+LmM1NdH3jbn2QyWZ9Dkq+vL3x9faFSqWz+7Gmdr68vlEol7+wj8iAMPkRDiEwmg1KphFKpREhISL/1K4RAW1ubzVDUU7l27RpaW1t7LR0dHdK2zO8dLEql0mZoMhdzG2vLtl731s687OPjw7NcRIOEwYeIeiWTyaSzK/0ZqLrq6Ojoc0jqWtra2tDW1oZr1671+PPGOoPBIG3bfBbLni/b7G/mINTfRaFQ3FTx9vZmKKMhxaHgk5ubiy1btqC+vh7x8fHYtm0bkpKSrLYtLy/Hhg0bUFJSgurqarzyyitYvXq1RZs///nP2LRpk0Xdrbfeih9++AEAcObMGYwZM8Zq/++88w4WLFgAAFY/nG+99RYefvhhe3eRiAaZj48PfHx8Bu2Oys7Ozh7D0rVr19De3i6Fpq6v+2P5xse9mMOXK/Lx8bEaisz11n46uq5rm67F29vban1f2vBSJnVld/DZu3cvNBoN8vLyoFarkZOTg7S0NFRWVlq9O6a1tRVxcXFYsGAB1qxZY7Pf2267DUeOHLk+sC634kZFRaGurs6i/Y4dO7BlyxbMnj3boj4/Px+zZs2Slvtj3gQRDT3e3t7S1wE4g8FgkMLOQJWOjg7o9Xq7irXw1dHRgY6OjkG9/Nif5HK51XDk7e0tLZtfdy03W+/l5TUgy11/9vW1l5cXz9z9f3YHn5dffhnLli3DkiVLAAB5eXk4dOgQdu3ahXXr1nVrP23aNEybNg0ArK6XBuLtjfDwcKvrvLy8uq17//33sXDhQvj5+VnUBwUF2eyHiMhVeHl5SZO0XYkQAgaDoU8hyRyszMHoxjp72tiq61o6Ozut1nctXS9hmhmNRpc+ozZY5HK5w6HJWp0jbby9vTFjxgwsXLjQab8Hu4KPXq9HSUkJMjMzpTq5XI7U1FQUFRXd1EBOnTqFyMhIqFQqJCcnIzs7G9HR0VbblpSUoKysDLm5ud3WZWRk4Mknn0RcXBx+//vfY8mSJTZT7o0fBJ1Od1P7QETk7mQymXSGwdVCWV8IIfockMxhqmu5sa4vbW6sM/ff2dkp/TSX/lg299+1mNf1xGg0wmg0SjcSOEt7e7v7BJ9Lly7BYDAgLCzMoj4sLEyaj+MItVqNN954A7feeivq6uqwadMm/PKXv8TJkyetnobeuXMnJk2ahOnTp1vUP//887j33nsxbNgwfPLJJ3j66afR3NyMVatWWd1udnZ2t7lFRETkvmQymXQpyxMZjUYpBN0Yim58bc86a8uOtjFfBXIWl7irq+s8nTvuuANqtRoxMTF45513sHTpUou2165dw5tvvon169d366drXUJCAlpaWrBlyxabwSczMxMajUZa1ul0dj0nioiIyJXI5XI+968Xdk11DwkJgZeXFxoaGizqGxoa+nVeTVBQECZMmIDTp093W/ePf/wDra2tWLRoUa/9qNVqnD9/3uZ1XaVS2e25SERERDR02RV8FAoFpk6disLCQqnOaDSisLAQycnJ/Tao5uZm/PTTT4iIiOi2bufOnXjwwQcxatSoXvspKyvDiBEjoFQq+21sRERE5L7svtSl0WiQnp6OxMREJCUlIScnBy0tLdJdXosWLcLo0aORnZ0NwDQhuqKiQnpdU1ODsrIy+Pn5Ydy4cQCAZ599Fg888ABiYmJQW1uLjRs3wsvLC4888ojFtk+fPo0vvvgCH374YbdxHThwAA0NDbj77ruhUqlQUFCArKwsPPvss/buIhEREQ1Rdgefhx56CBcvXsSGDRtQX1+PO++8Ex999JE04fns2bMWXxZVW1uLhIQEaXnr1q3YunUrUlJScPToUQDA+fPn8cgjj+Dy5csYNWoUfvGLX+DYsWPdzurs2rULt9xyC2bOnNltXD4+PsjNzcWaNWsghMC4ceOkW++JiIiIAEAmzE8eJOh0OgQGBkKr1XK+DxERkZuw5+83v8ebiIiIPAaDDxEREXkMBh8iIiLyGAw+RERE5DEYfIiIiMhjMPgQERGRx2DwISIiIo/B4ENEREQewyWezu4qzN/lqNPpnDwSIiIi6ivz3+2+fCczg08XTU1NAICoqCgnj4SIiIjs1dTUhMDAwB7b8JEVXRiNRtTW1sLf3x8ymaxf+9bpdIiKisK5c+eG/OMwuK9DlyftL/d16PKk/fWUfRVCoKmpCZGRkRbPC7WGZ3y6kMvluOWWWwZ0GwEBAUP6P76uuK9DlyftL/d16PKk/fWEfe3tTI8ZJzcTERGRx2DwISIiIo/B4DNIlEolNm7cCKVS6eyhDDju69DlSfvLfR26PGl/PWlf+4qTm4mIiMhj8IwPEREReQwGHyIiIvIYDD5ERETkMRh8iIiIyGMw+PSj3NxcxMbGQqVSQa1Wo7i4uMf27777LiZOnAiVSoUpU6bgww8/HKSROi47OxvTpk2Dv78/QkNDMW/ePFRWVvb4njfeeAMymcyiqFSqQRqx4/785z93G/fEiRN7fI87HlOz2NjYbvsrk8mQkZFhtb07HdcvvvgCDzzwACIjIyGTybBv3z6L9UIIbNiwAREREfD19UVqaipOnTrVa7/2fuYHQ0/72tHRgbVr12LKlCkYPnw4IiMjsWjRItTW1vbYpyOfhcHS27FdvHhxt7HPmjWr137d7dgCsPr5lclk2LJli80+XfnYDhQGn36yd+9eaDQabNy4EaWlpYiPj0daWhouXLhgtf3XX3+NRx55BEuXLsW3336LefPmYd68eTh58uQgj9w+n3/+OTIyMnDs2DEUFBSgo6MDM2fOREtLS4/vCwgIQF1dnVSqq6sHacQ357bbbrMY9//8z//YbOuux9TsxIkTFvtaUFAAAFiwYIHN97jLcW1paUF8fDxyc3Otrn/xxRfxt7/9DXl5eTh+/DiGDx+OtLQ0tLW12ezT3s/8YOlpX1tbW1FaWor169ejtLQU7733HiorK/Hggw/22q89n4XB1NuxBYBZs2ZZjP2tt97qsU93PLYALPaxrq4Ou3btgkwmw/z583vs11WP7YAR1C+SkpJERkaGtGwwGERkZKTIzs622n7hwoVizpw5FnVqtVo89dRTAzrO/nbhwgUBQHz++ec22+Tn54vAwMDBG1Q/2bhxo4iPj+9z+6FyTM3+8Ic/iLFjxwqj0Wh1vbseVwDi/fffl5aNRqMIDw8XW7ZskeoaGxuFUqkUb731ls1+7P3MO8ON+2pNcXGxACCqq6tttrH3s+As1vY3PT1dzJ07165+hsqxnTt3rrj33nt7bOMux7Y/8YxPP9Dr9SgpKUFqaqpUJ5fLkZqaiqKiIqvvKSoqsmgPAGlpaTbbuyqtVgsAGDlyZI/tmpubERMTg6ioKMydOxfl5eWDMbybdurUKURGRiIuLg6PPvoozp49a7PtUDmmgOm/6T179uCJJ57o8YG97npcu6qqqkJ9fb3FsQsMDIRarbZ57Bz5zLsqrVYLmUyGoKCgHtvZ81lwNUePHkVoaChuvfVWrFixApcvX7bZdqgc24aGBhw6dAhLly7tta07H1tHMPj0g0uXLsFgMCAsLMyiPiwsDPX19VbfU19fb1d7V2Q0GrF69WrMmDEDt99+u812t956K3bt2oX9+/djz549MBqNmD59Os6fPz+Io7WfWq3GG2+8gY8++gjbt29HVVUVfvnLX6Kpqclq+6FwTM327duHxsZGLF682GYbdz2uNzIfH3uOnSOfeVfU1taGtWvX4pFHHunxAZb2fhZcyaxZs/D3v/8dhYWFeOGFF/D5559j9uzZMBgMVtsPlWO7e/du+Pv743e/+12P7dz52DqKT2cnh2VkZODkyZO9Xg9OTk5GcnKytDx9+nRMmjQJr732Gv7yl78M9DAdNnv2bOn1HXfcAbVajZiYGLzzzjt9+r8od7Zz507Mnj0bkZGRNtu463Elk46ODixcuBBCCGzfvr3Htu78WXj44Yel11OmTMEdd9yBsWPH4ujRo7jvvvucOLKBtWvXLjz66KO93nDgzsfWUTzj0w9CQkLg5eWFhoYGi/qGhgaEh4dbfU94eLhd7V3NM888g4MHD+Kzzz7DLbfcYtd7fXx8kJCQgNOnTw/Q6AZGUFAQJkyYYHPc7n5Mzaqrq3HkyBE8+eSTdr3PXY+r+fjYc+wc+cy7EnPoqa6uRkFBQY9ne6zp7bPgyuLi4hASEmJz7O5+bAHgyy+/RGVlpd2fYcC9j21fMfj0A4VCgalTp6KwsFCqMxqNKCwstPg/4q6Sk5Mt2gNAQUGBzfauQgiBZ555Bu+//z4+/fRTjBkzxu4+DAYDvvvuO0RERAzACAdOc3MzfvrpJ5vjdtdjeqP8/HyEhoZizpw5dr3PXY/rmDFjEB4ebnHsdDodjh8/bvPYOfKZdxXm0HPq1CkcOXIEwcHBdvfR22fBlZ0/fx6XL1+2OXZ3PrZmO3fuxNSpUxEfH2/3e9352PaZs2dXDxVvv/22UCqV4o033hAVFRVi+fLlIigoSNTX1wshhHj88cfFunXrpPZfffWV8Pb2Flu3bhXff/+92Lhxo/Dx8RHfffeds3ahT1asWCECAwPF0aNHRV1dnVRaW1ulNjfu66ZNm8THH38sfvrpJ1FSUiIefvhhoVKpRHl5uTN2oc/++Mc/iqNHj4qqqirx1VdfidTUVBESEiIuXLgghBg6x7Qrg8EgoqOjxdq1a7utc+fj2tTUJL799lvx7bffCgDi5ZdfFt9++610J9PmzZtFUFCQ2L9/v/jnP/8p5s6dK8aMGSOuXbsm9XHvvfeKbdu2Scu9feadpad91ev14sEHHxS33HKLKCsrs/gMt7e3S33cuK+9fRacqaf9bWpqEs8++6woKioSVVVV4siRI+Kuu+4S48ePF21tbVIfQ+HYmmm1WjFs2DCxfft2q32407EdKAw+/Wjbtm0iOjpaKBQKkZSUJI4dOyatS0lJEenp6Rbt33nnHTFhwgShUCjEbbfdJg4dOjTII7YfAKslPz9fanPjvq5evVr6vYSFhYn7779flJaWDv7g7fTQQw+JiIgIoVAoxOjRo8VDDz0kTp8+La0fKse0q48//lgAEJWVld3WufNx/eyzz6z+d2veH6PRKNavXy/CwsKEUqkU9913X7ffQUxMjNi4caNFXU+feWfpaV+rqqpsfoY/++wzqY8b97W3z4Iz9bS/ra2tYubMmWLUqFHCx8dHxMTEiGXLlnULMEPh2Jq99tprwtfXVzQ2Nlrtw52O7UCRCSHEgJ5SIiIiInIRnONDREREHoPBh4iIiDwGgw8RERF5DAYfIiIi8hgMPkREROQxGHyIiIjIYzD4EBERkcdg8CEiIiKPweBDREREHoPBh4iIiDwGgw8RERF5DAYfIiIi8hj/D9wQRRBt7yZrAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Seems like our results can vary depending on the validation set we \"hold out.\" Ensuring that our ANN is robust to subsets of our data distribution necessitates k-fold cross-validation. Read here for how to use sklearn cross validation utilities with keras models: https://github.com/christianversloot/machine-learning-articles/blob/main/how-to-use-k-fold-cross-validation-with-keras.md"
      ],
      "metadata": {
        "id": "nqURctigavi2"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "4BJ3kK4Ka_WK"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}