Copy_of_EECS_504_PS4_Backpropagation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of EECS 504 PS4: Backpropagation",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ix5dQS2rUMlu"
      },
      "source": [
        "#EECS 504 PS4: Backpropagation\n",
        "\n",
        "Please provide the following information \n",
        "(e.g. Andrew Owens, ahowens):\n",
        "\n",
        "[Mohammadreza] [Tavasoli Naeini], [tavasoli]\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W_Cst4k4tuBc"
      },
      "source": [
        "# Starting\n",
        "\n",
        "Run the following code to import the modules you'll need. After your finish the assignment, remember to run all cells and save the note book to your local machine as a .ipynb file for Canvas submission."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SHumIO-xt57H"
      },
      "source": [
        "import pickle\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import os\n",
        "from torchvision.datasets import CIFAR10\n",
        "download = not os.path.isdir('cifar-10-batches-py')\n",
        "dset_train = CIFAR10(root='.', download=download)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "apEPzDNtK0MC"
      },
      "source": [
        "# Problem 4.2 Multi-layer perceptron\n",
        "In this problem you will develop a two Layer neural network with fully-connected layers to perform classification, and test it out on the CIFAR-10 dataset.\n",
        "\n",
        "We train the network with a softmax loss function on the weight matrices. The network uses a ReLU nonlinearity after the first fully connected layer. In other words, the network has the following architecture:\n",
        "\n",
        "input - fully connected layer - ReLU - fully connected layer - softmax\n",
        "\n",
        "The outputs of the second fully-connected layer are the scores for each class.\n",
        "\n",
        "You cannot use any deep learning libraries such as PyTorch in this part."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SXfumCQ21JoK"
      },
      "source": [
        "# 4.2 (a) Layers\n",
        "In this problem, implement fully connected layer, relu and softmax. Filling in all TODOs in skeleton codes will be sufficient."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q-ljfgMv9PHx"
      },
      "source": [
        "def fc_forward(x, w, b):\n",
        "    \"\"\"\n",
        "    Computes the forward pass for a fully-connected layer.\n",
        "    \n",
        "    The input x has shape (N, Din) and contains a minibatch of N\n",
        "    examples, where each example x[i] has shape (Din,).\n",
        "    \n",
        "    Inputs:\n",
        "    - x: A numpy array containing input data, of shape (N, Din)\n",
        "    - w: A numpy array of weights, of shape (Din, Dout)\n",
        "    - b: A numpy array of biases, of shape (Dout,)\n",
        "    \n",
        "    Returns a tuple of:\n",
        "    - out: output, of shape (N, Dout)\n",
        "    - cache: (x, w, b)\n",
        "    \"\"\"\n",
        "    ###########################################################################\n",
        "    # TODO: Implement the forward pass. Store the result in out.              #\n",
        "    ###########################################################################\n",
        "    out = None\n",
        "    # Reshape x into rows\n",
        "    N = x.shape[0]\n",
        "    x_row = x.reshape(N, -1)         # (N,D)\n",
        "    out = np.dot(x_row, w) + b       # (N,M)\n",
        "  \n",
        "    ###########################################################################\n",
        "    #                             END OF YOUR CODE                            #\n",
        "    ###########################################################################\n",
        "    cache = (x, w, b)\n",
        "    return out, cache\n",
        "\n",
        "\n",
        "def fc_backward(dout, cache):\n",
        "    \"\"\"\n",
        "    Computes the backward pass for a fully_connected layer.\n",
        "    \n",
        "    Inputs:\n",
        "    - dout: Upstream derivative, of shape (N, Dout)\n",
        "    - cache: returned by your forward function. Tuple of:\n",
        "      - x: Input data, of shape (N, Din)\n",
        "      - w: Weights, of shape (Din, Dout)\n",
        "      - b: Biases, of shape (Dout,)\n",
        "      \n",
        "    Returns a tuple of:\n",
        "    - dx: Gradient with respect to x, of shape (N, Din)\n",
        "    - dw: Gradient with respect to w, of shape (Din, Dout)\n",
        "    - db: Gradient with respect to b, of shape (Dout,)\n",
        "    \"\"\"\n",
        "    x, w, b = cache\n",
        "    dx, dw, db = None, None, None\n",
        "    ###########################################################################\n",
        "    # TODO: Implement the affine backward pass.                               #\n",
        "    ########################################################################### \n",
        "    dx = np.dot(dout, w.T)                       # (N,D)    \n",
        "    dx = np.reshape(dx, x.shape)                 # (N,d1,...,d_k)   \n",
        "    x_row = x.reshape(x.shape[0], -1)            # (N,D)    \n",
        "    dw = np.dot(x_row.T, dout)                   # (D,M)    \n",
        "    db = np.sum(dout, axis=0, keepdims=True)     # (1,M)    \n",
        "    ###########################################################################\n",
        "    #                             END OF YOUR CODE                            #\n",
        "    ###########################################################################\n",
        "    return dx, dw, db\n",
        "\n",
        "def relu_forward(x):\n",
        "    \"\"\"\n",
        "    Computes the forward pass for a layer of rectified linear units (ReLUs).\n",
        "\n",
        "    Input:\n",
        "    - x: Inputs, of any shape\n",
        "\n",
        "    Returns a tuple of:\n",
        "    - out: Output, of the same shape as x\n",
        "    - cache: x\n",
        "    \"\"\"\n",
        "    out = x\n",
        "    ###########################################################################\n",
        "    # TODO: Implement the ReLU forward pass.                                  #\n",
        "    ###########################################################################\n",
        "    out = None    \n",
        "    out = np.maximum(x, 0)\n",
        "    ###########################################################################\n",
        "    #                             END OF YOUR CODE                            #\n",
        "    ###########################################################################\n",
        "    cache = x\n",
        "    return out, cache\n",
        "\n",
        "\n",
        "def relu_backward(dout, cache):\n",
        "    \"\"\"\n",
        "    Computes the backward pass for a layer of rectified linear units (ReLUs).\n",
        "\n",
        "    Input:\n",
        "    - dout: Upstream derivatives, of any shape\n",
        "    - cache: returned by your forward function. Input x, of same shape as dout\n",
        "\n",
        "    Returns:\n",
        "    - dx: Gradient with respect to x\n",
        "    \"\"\"\n",
        "    dx, x = dout, cache\n",
        "    ###########################################################################\n",
        "    # TODO: Implement the ReLU backward pass.                                 #\n",
        "    ###########################################################################\n",
        "    dx = dout    \n",
        "    dx[x <= 0] = 0  \n",
        "    \n",
        "    ###########################################################################\n",
        "    #                             END OF YOUR CODE                            #\n",
        "    ###########################################################################\n",
        "    return dx\n",
        "\n",
        "\n",
        "def softmax_loss(x, y):\n",
        "    \"\"\"\n",
        "    Computes the loss and gradient for softmax classification.\n",
        "\n",
        "    Inputs:\n",
        "    - x: Input data, of shape (N, C) where x[i, j] is the score for the jth\n",
        "      class for the ith input.\n",
        "    - y: Vector of labels, of shape (N,) where y[i] is the label for x[i] and\n",
        "      0 <= y[i] < C\n",
        "\n",
        "    Returns a tuple of:\n",
        "    - loss: Scalar giving the loss\n",
        "    - dx: Gradient of the loss with respect to x\n",
        "    \"\"\"\n",
        "    loss, dx = None, None\n",
        "    ###########################################################################\n",
        "    # TODO: Implement softmax loss                                            #\n",
        "    ###########################################################################\n",
        "    probs = np.exp(x - np.max(x, axis=1, keepdims=True))    \n",
        "    probs /= np.sum(probs, axis=1, keepdims=True)    \n",
        "    N = x.shape[0]   \n",
        "    loss = -np.sum(np.log(probs[np.arange(N), y])) / N    \n",
        "    dx = probs.copy()    \n",
        "    dx[np.arange(N), y] -= 1    \n",
        "    dx /= N  \n",
        "\n",
        "    \n",
        "    ###########################################################################\n",
        "    #                             END OF YOUR CODE                            #\n",
        "    ###########################################################################\n",
        "    return loss, dx"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LbFxtS3zK8oz"
      },
      "source": [
        "# 4.2 (b) Softmax Classifier\n",
        "\n",
        "In this problem, implement softmax classifier."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ytvxbx9UpxVL"
      },
      "source": [
        "class SoftmaxClassifier(object):\n",
        "    \"\"\"\n",
        "    A fully-connected neural network with\n",
        "    softmax loss that uses a modular layer design. We assume an input dimension\n",
        "    of D, a hidden dimension of H, and perform classification over C classes.\n",
        "\n",
        "    The architecture should be fc - relu - fc - softmax with one hidden layer\n",
        "\n",
        "    The learnable parameters of the model are stored in the dictionary\n",
        "    self.params that maps parameter names to numpy arrays.\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, input_dim=3072, hidden_dim=300, num_classes=10,\n",
        "                 weight_scale=1e-3):\n",
        "        \"\"\"\n",
        "        Initialize a new network.\n",
        "\n",
        "        Inputs:\n",
        "        - input_dim: An integer giving the size of the input\n",
        "        - hidden_dim: An integer giving the size of the hidden layer, None\n",
        "          if there's no hidden layer.\n",
        "        - num_classes: An integer giving the number of classes to classify\n",
        "        - weight_scale: Scalar giving the standard deviation for random\n",
        "          initialization of the weights.\n",
        "        \"\"\"\n",
        "        self.params = {}\n",
        "        ############################################################################\n",
        "        # TODO: Initialize the weights and biases of the two-layer net. Weights    #\n",
        "        # should be initialized from a Gaussian centered at 0.0 with               #\n",
        "        # standard deviation equal to weight_scale, and biases should be           #\n",
        "        # initialized to zero. All weights and biases should be stored in the      #\n",
        "        # dictionary self.params, with fc weights and biases using the keys        #\n",
        "        # 'W' and 'b', i.e., W1, b1 for the weights and bias in the first linear   #\n",
        "        # layer, W2, b2 for the weights and bias in the second linear layer.       #\n",
        "        ############################################################################\n",
        "        self.params['W1'] = np.random.normal(scale=weight_scale, size=(input_dim, hidden_dim))\n",
        "        self.params['W2'] = np.random.normal(scale=weight_scale, size=(hidden_dim, num_classes))\n",
        "        self.params['b1'] = np.zeros(hidden_dim)\n",
        "        self.params['b2'] = np.zeros(num_classes)\n",
        "        ############################################################################\n",
        "        #                             END OF YOUR CODE                             #\n",
        "        ############################################################################\n",
        "\n",
        "\n",
        "    def forwards_backwards(self, X, y=None):\n",
        "        \"\"\"\n",
        "        Compute loss and gradient for a minibatch of data.\n",
        "\n",
        "        Inputs:\n",
        "        - X: Array of input data of shape (N, Din)\n",
        "        - y: Array of labels, of shape (N,). y[i] gives the label for X[i].\n",
        "\n",
        "        Returns:\n",
        "        If y is None, then run a test-time forward pass of the model and return:\n",
        "        - scores: Array of shape (N, C) giving classification scores, where\n",
        "          scores[i, c] is the classification score for X[i] and class c.\n",
        "\n",
        "        If y is not None, then run a training-time forward and backward pass. And\n",
        "        return a tuple of:\n",
        "        - loss: Scalar value giving the loss\n",
        "        - grads: Dictionary with the same keys as self.params, mapping parameter\n",
        "          names to gradients of the loss with respect to those parameters.\n",
        "        \"\"\"\n",
        "        scores = None\n",
        "        ############################################################################\n",
        "        # TODO: Implement the forward pass for the two-layer net, computing the    #\n",
        "        # class scores for X and storing them in the scores variable.              #\n",
        "        ############################################################################\n",
        "        W1 = self.params[\"W1\"]\n",
        "        W2 = self.params[\"W2\"]\n",
        "        b1 = self.params[\"b1\"]\n",
        "        b2 = self.params[\"b2\"]\n",
        "        out1, cache1 =fc_forward(X, W1, b1)\n",
        "        out2, cache2 = relu_forward(out1)\n",
        "        scores, cache3 =fc_forward(out2, W2, b2)\n",
        "        ############################################################################\n",
        "        #                             END OF YOUR CODE                             #\n",
        "        ############################################################################\n",
        "\n",
        "        # If y is None then we are in test mode so just return scores\n",
        "        if y is None:\n",
        "            return scores\n",
        "\n",
        "        loss, grads = 0, {}\n",
        "        ############################################################################\n",
        "        # TODO: Implement the backward pass for the two-layer net. Store the loss  #\n",
        "        # in the loss variable and gradients in the grads dictionary. Compute data #\n",
        "        # loss using softmax, and make sure that grads[k] holds the gradients for  #\n",
        "        # self.params[k].                                                          # \n",
        "        ############################################################################\n",
        "        loss, dsoftmax = softmax_loss(scores, y)\n",
        "        dout2, dW2, db2 = fc_backward(dsoftmax, cache3)\n",
        "        dout1 = relu_backward(dout2, cache2)\n",
        "        dx, dW1, db1 = fc_backward(dout1, cache1)\n",
        "        grads = {\"W1\": dW1, \"W2\": dW2, \"b1\": db1, \"b2\":db2}\n",
        "        ############################################################################\n",
        "        #                             END OF YOUR CODE                             #\n",
        "        ############################################################################\n",
        "        return loss, grads\n",
        "\n",
        "  \n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lwp0waIL1h_e"
      },
      "source": [
        "# 4.2(c) Training\n",
        "\n",
        "In this problem, you need to preprocess the images and set up model hyperparameters. Notice that adjust the training and val split is optional."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kZPtQzXGMoCg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "280b5934-5aba-459c-b504-5e899da0beba"
      },
      "source": [
        "def unpickle(file):\n",
        "    with open(file, 'rb') as fo:\n",
        "        dict = pickle.load(fo, encoding=\"latin1\")\n",
        "    return dict\n",
        "\n",
        "def load_cifar10():\n",
        "    data = {}\n",
        "    meta = unpickle(\"cifar-10-batches-py/batches.meta\")\n",
        "    batch1 = unpickle(\"cifar-10-batches-py/data_batch_1\")\n",
        "    batch2 = unpickle(\"cifar-10-batches-py/data_batch_2\")\n",
        "    batch3 = unpickle(\"cifar-10-batches-py/data_batch_3\")\n",
        "    batch4 = unpickle(\"cifar-10-batches-py/data_batch_4\")\n",
        "    batch5 = unpickle(\"cifar-10-batches-py/data_batch_5\")\n",
        "    test_batch = unpickle(\"cifar-10-batches-py/test_batch\")\n",
        "    X_train = np.vstack((batch1['data'], batch2['data'], batch3['data'],\\\n",
        "                         batch4['data'], batch5['data']))\n",
        "    Y_train = np.array(batch1['labels'] + batch2['labels'] + batch3['labels'] + \n",
        "                       batch4['labels'] + batch5['labels'])\n",
        "    X_test = test_batch['data']\n",
        "    Y_test = test_batch['labels']\n",
        "    \n",
        "    #Preprocess images here                                     \n",
        "    X_train = (X_train-np.mean(X_train,axis=1,keepdims=True))/np.std(X_train,axis=1,keepdims=True)\n",
        "    X_test = (X_test-np.mean(X_test,axis=1,keepdims=True))/np.std(X_test,axis=1,keepdims=True)\n",
        "\n",
        "    data['X_train'] = X_train[:40000]\n",
        "    data['y_train'] = Y_train[:40000]\n",
        "    data['X_val'] = X_train[40000:]\n",
        "    data['y_val'] = Y_train[40000:]\n",
        "    data['X_test'] = X_test\n",
        "    data['y_test'] = Y_test\n",
        "    return data\n",
        "\n",
        "def test_network(model, X, y, num_samples=None, batch_size=100):\n",
        "    \"\"\"\n",
        "    Check accuracy of the model on the provided data.\n",
        "\n",
        "    Inputs:\n",
        "    - model: Image classifier\n",
        "    - X: Array of data, of shape (N, d_1, ..., d_k)\n",
        "    - y: Array of labels, of shape (N,)\n",
        "    - num_samples: If not None, subsample the data and only test the model\n",
        "      on num_samples datapoints.\n",
        "    - batch_size: Split X and y into batches of this size to avoid using\n",
        "      too much memory.\n",
        "\n",
        "    Returns:\n",
        "    - acc: Scalar giving the fraction of instances that were correctly\n",
        "      classified by the model.\n",
        "    \"\"\"\n",
        "\n",
        "    # Subsample the data\n",
        "    N = X.shape[0]\n",
        "    if num_samples is not None and N > num_samples:\n",
        "        mask = np.random.choice(N, num_samples)\n",
        "        N = num_samples\n",
        "        X = X[mask]\n",
        "        y = y[mask]\n",
        "\n",
        "    # Compute predictions in batches\n",
        "    num_batches = N // batch_size\n",
        "    if N % batch_size != 0:\n",
        "        num_batches += 1\n",
        "    y_pred = []\n",
        "    for i in range(num_batches):\n",
        "        start = i * batch_size\n",
        "        end = (i + 1) * batch_size\n",
        "        scores = model.forwards_backwards(X[start:end])\n",
        "        y_pred.append(np.argmax(scores, axis=1))\n",
        "    y_pred = np.hstack(y_pred)\n",
        "    acc = np.mean(y_pred == y)\n",
        "\n",
        "    return acc\n",
        "\n",
        "\n",
        "def train_network(model, data, **kwargs):\n",
        "    \"\"\"\n",
        "     Required arguments:\n",
        "    - model: Image classifier\n",
        "    - data: A dictionary of training and validation data containing:\n",
        "      'X_train': Array, shape (N_train, d_1, ..., d_k) of training images\n",
        "      'X_val': Array, shape (N_val, d_1, ..., d_k) of validation images\n",
        "      'y_train': Array, shape (N_train,) of labels for training images\n",
        "      'y_val': Array, shape (N_val,) of labels for validation images\n",
        "\n",
        "    Optional arguments:\n",
        "    - learning_rate: A scalar for initial learning rate.\n",
        "    - lr_decay: A scalar for learning rate decay; after each epoch the\n",
        "      learning rate is multiplied by this value.\n",
        "    - batch_size: Size of minibatches used to compute loss and gradient\n",
        "      during training.\n",
        "    - num_epochs: The number of epochs to run for during training.\n",
        "    - print_every: Integer; training losses will be printed every\n",
        "      print_every iterations.\n",
        "    - verbose: Boolean; if set to false then no output will be printed\n",
        "      during training.\n",
        "    - num_train_samples: Number of training samples used to check training\n",
        "      accuracy; default is 1000; set to None to use entire training set.\n",
        "    - num_val_samples: Number of validation samples to use to check val\n",
        "      accuracy; default is None, which uses the entire validation set.\n",
        "    \"\"\"\n",
        "    \n",
        "    \n",
        "    learning_rate =  kwargs.pop('learning_rate', 1e-3)\n",
        "    lr_decay = kwargs.pop('lr_decay', 1.0)\n",
        "    batch_size = kwargs.pop('batch_size', 100)\n",
        "    num_epochs = kwargs.pop('num_epochs', 10)\n",
        "    num_train_samples = kwargs.pop('num_train_samples', 1000)\n",
        "    num_val_samples = kwargs.pop('num_val_samples', None)\n",
        "    print_every = kwargs.pop('print_every', 10)   \n",
        "    verbose = kwargs.pop('verbose', True)\n",
        "    \n",
        "    epoch = 0\n",
        "    best_val_acc = 0\n",
        "    best_params = {}\n",
        "    loss_history = []\n",
        "    train_acc_history = []\n",
        "    val_acc_history = []\n",
        "    \n",
        "    \n",
        "    num_train = data['X_train'].shape[0]\n",
        "    iterations_per_epoch = max(num_train // batch_size, 1)\n",
        "    num_iterations = num_epochs * iterations_per_epoch\n",
        "    \n",
        "\n",
        "    \n",
        "    for t in range(num_iterations):\n",
        "        # Make a minibatch of training data\n",
        "        batch_mask = np.random.choice(num_train, batch_size)\n",
        "        X_batch = data['X_train'][batch_mask]\n",
        "        y_batch = data['y_train'][batch_mask]\n",
        "        \n",
        "        # Compute loss and gradient\n",
        "        loss, grads = model.forwards_backwards(X_batch, y_batch)\n",
        "        loss_history.append(loss)\n",
        "\n",
        "        # Perform a parameter update\n",
        "        for p, w in model.params.items():\n",
        "            model.params[p] = w - grads[p]*learning_rate\n",
        "          \n",
        "        # Print training loss\n",
        "        if verbose and t % print_every == 0:\n",
        "            print('(Iteration %d / %d) loss: %f' % (\n",
        "                   t + 1, num_iterations, loss_history[-1]))\n",
        "         \n",
        "        # At the end of every epoch, increment the epoch counter and decay\n",
        "        # the learning rate.\n",
        "        epoch_end = (t + 1) % iterations_per_epoch == 0\n",
        "        if epoch_end:\n",
        "            epoch += 1\n",
        "            learning_rate *= lr_decay\n",
        "        \n",
        "        # Check train and val accuracy on the first iteration, the last\n",
        "        # iteration, and at the end of each epoch.\n",
        "        first_it = (t == 0)\n",
        "        last_it = (t == num_iterations - 1)\n",
        "        if first_it or last_it or epoch_end:\n",
        "            train_acc = test_network(model, data['X_train'], data['y_train'],\n",
        "                num_samples= num_train_samples)\n",
        "            val_acc = test_network(model, data['X_val'], data['y_val'],\n",
        "                num_samples=num_val_samples)\n",
        "            train_acc_history.append(train_acc)\n",
        "            val_acc_history.append(val_acc)\n",
        "\n",
        "            if verbose:\n",
        "                print('(Epoch %d / %d) train acc: %f; val_acc: %f' % (\n",
        "                       epoch, num_epochs, train_acc, val_acc))\n",
        "\n",
        "            # Keep track of the best model\n",
        "            if val_acc > best_val_acc:\n",
        "                best_val_acc = val_acc\n",
        "                best_params = {}\n",
        "                for k, v in model.params.items():\n",
        "                    best_params[k] = v.copy()\n",
        "        \n",
        "    model.params = best_params\n",
        "        \n",
        "    return model, train_acc_history, val_acc_history\n",
        "        \n",
        "\n",
        "# load data\n",
        "data = load_cifar10() \n",
        "train_data = { k: data[k] for k in ['X_train', 'y_train', \n",
        "                                    'X_val', 'y_val']}\n",
        "#######################################################################\n",
        "# TODO: Set up model hyperparameters                                  #\n",
        "#######################################################################\n",
        "\n",
        "# initialize model\n",
        "model = SoftmaxClassifier(hidden_dim =2048, weight_scale=1e-2)\n",
        "\n",
        "# start training    \n",
        "model, train_acc_history, val_acc_history = train_network(\n",
        "    model, train_data, learning_rate =0.01,\n",
        "    lr_decay=0.95, num_epochs=10, \n",
        "    batch_size=256, print_every=1000)\n",
        "#######################################################################\n",
        "#                         END OF YOUR CODE                            #\n",
        "#######################################################################\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(Iteration 1 / 1560) loss: 2.318424\n",
            "(Epoch 0 / 10) train acc: 0.130000; val_acc: 0.111800\n",
            "(Epoch 1 / 10) train acc: 0.400000; val_acc: 0.367400\n",
            "(Epoch 2 / 10) train acc: 0.432000; val_acc: 0.402200\n",
            "(Epoch 3 / 10) train acc: 0.466000; val_acc: 0.418800\n",
            "(Epoch 4 / 10) train acc: 0.483000; val_acc: 0.432700\n",
            "(Epoch 5 / 10) train acc: 0.497000; val_acc: 0.443100\n",
            "(Epoch 6 / 10) train acc: 0.480000; val_acc: 0.448700\n",
            "(Iteration 1001 / 1560) loss: 1.492344\n",
            "(Epoch 7 / 10) train acc: 0.487000; val_acc: 0.451800\n",
            "(Epoch 8 / 10) train acc: 0.486000; val_acc: 0.455300\n",
            "(Epoch 9 / 10) train acc: 0.496000; val_acc: 0.464600\n",
            "(Epoch 10 / 10) train acc: 0.473000; val_acc: 0.468400\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fcovGmpXvXXa"
      },
      "source": [
        "# 4.2(c) Report Accuracy\n",
        "\n",
        "Run the given code and report the accuracy on test set.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FwCq8pBhu6dz",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "da9900db-f2f2-4f0a-f0a0-214c0ee486a9"
      },
      "source": [
        "# report test accuracy\n",
        "acc = test_network(model, data['X_test'], data['y_test'])\n",
        "print(\"Test accuracy: {}\".format(acc))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test accuracy: 0.4773\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oTrmbULS7i2N"
      },
      "source": [
        "# 4.2(d) Plot\n",
        "\n",
        "Using the train_acc_history and val_acc_history, plot the train & val accuracy versus epochs on one plot. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SPjtnbya9S7g",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "15e2a297-76fd-4405-c636-9dd6a9eb6eb6"
      },
      "source": [
        "plt.plot(range(len(train_acc_history)),train_acc_history,'b', label='Train_acc')\n",
        "plt.plot(range(len(val_acc_history)),val_acc_history,'r',label='val_acc')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f15e8a96128>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3TU9Z3/8eeboCiCKIJyC4KKKIjl\nEiBc6nbrDW0X2u3N2nbt1pa1LVu7vfxqL7/a1bWrbddTt2svrrVXLWttu+XXgpe6tVswA4SLF26C\neOFukJtUhCS8f3+8Z8gkBDJJZjLJd16Pc+bMzHe+35nPaHjlk8/V3B0REUmubsUugIiIFJaCXkQk\n4RT0IiIJp6AXEUk4Bb2ISMJ1L3YBmurXr58PGzas2MUQEelSli1bttPd+zf3WqcL+mHDhlFdXV3s\nYoiIdClm9tKxXlPTjYhIwinoRUQSTkEvIpJwCnoRkYTLKejNbIaZrTOzDWZ2UzOvf9jMasxsZfr2\n0azXrjOz9enbdfksvIiItKzFUTdmVgbcDVwObAaWmtk8d1/d5NT/cvc5Ta7tC9wMVAAOLEtfuzsv\npRcRkRblUqOfBGxw943ufgiYC8zK8f2vBB5z913pcH8MmNG2ooqISFvkEvSDgU1ZzzenjzX1LjN7\n2sweMrPy1lxrZrPNrNrMqmtqanIsukjHqK2FJUvgrrvggQdg585il0ikdfI1Yer/Ab9w94Nm9g/A\nT4C35nqxu98D3ANQUVGhBfKlqPbvh1QK/vxnWLgwHr/+esPrZjBpElx9NVx1FUyYAN00rKHLqa+H\n55+HVavg2WfjftUqePFFeMtb4Prr4W1vgxNOKHZJ2y+XoN8ClGc9H5I+doS7v5r19F7gG1nXvqXJ\ntU+0tpAihbRjByxa1BDsK1ZECHTrBm96U/yDf/ObYepU2LIFFiyA+fPha1+Dm2+G/v1hxowI/Suu\ngDPOKPY3kmyHD8MLLxwd6GvXwsGDDecNGwajR8OUKfC738XtrLPguuviZ+D884v2FdrNWtphysy6\nA88BlxLBvRS41t1XZZ0z0N23pR+/E/iCu1emO2OXAePTpy4HJrj7rmN9XkVFhWsJBCkU96jFZUL9\nz3+G9evjtZNOgsmTI9SnT49/8Keeeuz3qqmBRx+N0H/kEXj11fjlMHlyhP7VV8O4cartd5TDh+Hl\nl48O9DVr4MCBhvPKyyPQR4+Giy6K+wsvhF69Gs6pq4tf6D/8YQR+fX38XFx/Pbz73XDKKR3//Vpi\nZsvcvaLZ13LZStDMrga+DZQB97n7bWZ2C1Dt7vPM7F+BmUAdsAv4uLuvTV/7EeBL6be6zd1/dLzP\nUtBLPtXVwVNPNYT6woVRgwfo2zcCffr0+Ec8fjyceGLbPqe+HqqrI/QXLIClS+P4mWdGbf/qq6O2\nf/rp+flepcwdNm9uHOarVsHq1fCXvzScN2jQ0YE+atTxf3k3Z9s2+OlPI/TXr4feveH974ePfhQq\nKqIprzNod9B3JAW9tMfrr8PixQ3BXlUVbe4Qf5pnQn36dLjggsLVtl95JWr5CxbE/a5d8VmVlQ1t\n+2PHds7afm0tbN0ateNNm+KXWPfucTvhhOYfH++1Yz0uKzt+OdwjZJsL9H37Gs4766zmAz3fv1Td\n42fqhz+EX/4y/koYMyZq+R/8YPGb7BT0klg7dzZuX1+2LGrxZvGPMBPq06fDkCHFKWN9fYzaydT2\nly2L4wMGNLTtX355x9X29+6NEH/ppbjP3DLPt26NZpBCMzv+L429e2HPnobz+/VrHOaZWzECdu9e\nmDs3Qn/p0vhL8J3vjNC/9NLi/AJX0EtibNsGf/hDQ7CvWRPHe/SIkTCZUJ86FU47rbhlPZYdO+Dh\nhyP0H30Udu+O2u2UKQ1t+296U9uaBOrqGmrjzYX4yy83rg1DhGp5OZx9Ngwd2vhWXh4hVlcXt9ra\ntj9u7TUnn9w40M88Mz///fPt6acj8H/+8/jL7eyz4e//Pm5Dh3ZcORT00mXV1cXwxsxIl5Ur4/hp\np8G0aQ1NMRMmRGdqV1NXF01NCxbEbfnyOD5wYEPb/uWXQ58+cXzfvuOH+JYt8RdEtr59mw/xzLGz\nzuqcTUhdzRtvwH//d4T+H/4Qv6ivuCJq+TNnRmWkkBT00qVs3964xrtnT9R4p02LGu+MGXDxxckM\np8x3nz8/vvvevfHdzzsv/hLIbsqAaOYoL28+wDM18uzRJNIxXnwRfvQjuO++6Dju1w8+9KEI/dGj\nC/OZCnrp1Orro1abacPOrtVedVXcLrus8zbFFErmr5n586OJavDgo2vmAwa03KkpxVNfD489FrX8\n3/42mqUqKyPw3/e+GMGTLwp66XR27IjRKJmaa3Y7dWZUSlvbqUU6o5oa+NnPIvRXr46x+O99b4T+\n1Knt/1lX0EvRZUaeZNraiz3yRKRY3OMv2Hvvhf/6rxj+e8EFEfh/93dt73RW0EtRHGsseWZ0SWce\nSy7SEfbvhwcfjFr+k0/GDN1Vq9pWuz9e0OdrUTORo2aHVldH7eXMM+Fv/qah1t63b7FLKtI59OoF\nH/lI3NasieHDhWiuVNBLu+zc2dDW3nS9l3/+Z633IiXsjTdimNTu3XGf/biZYxfu3s2FI0fCWx/I\ne1EU9NJqhw7BD34A998f7e7usYJjphNVKzhKItTXN0zPPU5AH/NY9tKYzTn55BhKdvrpcT9wYKzT\nUQAKesmZO/z+9/DZz8Jzz8WCTl/7mtZkly7KPYZ/rV3b+Pbcc/GnadMpxE2VlUVAZ4f1kCFHH8vc\nN31c6BlUWRT0kpNVq+Azn4mhkCNHRlPNVVcVu1QiOaithY0bG4f5mjVxv3dvw3k9e8bwl8mTY7rw\nsUI6c9+rV5cZ/6ugl+N69dXYXOP734/JHd/+NnziE8nYdUcSZu9eWLfu6DDfsCFmn2UMHBjDWz7w\ngQj2zG3w4MT+Waqgl2bV1sL3vhdNM3v3wg03ROdqv37FLpmUtMxi9NlBnrlt29ZwXvfuMGJEBPo7\n39kQ5iNHNiwcVEIU9HKUBQuimWbt2hgOeeedsTSsSIeoq4vaxdatR7efr1vXeHeRPn0izK+8snHt\n/Jxz9GdnlpyC3sxmAHcRO0zd6+63H+O8dwEPARPdvdrMhgFrgHXpU1LufkN7Cy2FsXZtBPyCBVEZ\nmjcP3v72LtMMKZ1BXV10Yu7d2/i2Z8/Rx471WvZO7Blnnx0B/uY3N4T5hRfGJA39gLaoxaA3szLg\nbuByYDOw1MzmufvqJuf1Bm4EFjd5i+fdfWyeyisFsGtXNMt897ux/sa//RvMmdP2bfWkC3KPgH3t\ntbjt29fwuDVhnV3bPpaTTorOzD59Gm7l5Y2f9+kTHaIXXBC1js64SWsXkkuNfhKwwd03ApjZXGAW\nsLrJebcCdwCfz2sJpWDq6mI8/Fe/Gv9OP/YxuPXWGBMvXcDhwxGsTcM5O6Rbepx9LJdtpXr0ODqk\nBw1q/Lzp601vqkF0uFyCfjCwKev5ZmBy9glmNh4od/ffm1nToB9uZiuAfcBX3P3PTT/AzGYDswGG\nduSWLCXs0Ufhn/4pVtH767+O0TQXX1zsUpWww4djmvH27Q23bdsaHtfUHB3O+/dHTbwlZWUxZOrU\nUxvu+/SJMd+ZY01fzxzr3btxSHfg2G/Jn3Z3xppZN+BO4MPNvLwNGOrur5rZBOC/zWy0uzeaieDu\n9wD3QCxq1t4yybE991xMePrd7+Dcc+E3v4FZs9TMWTD79x87vLOPvfLK0VtDQYzVHjAg/sw644yY\nOXmsUD5WaJ90kv4Hl7hcgn4LUJ71fEj6WEZv4CLgCYsfpgHAPDOb6e7VwEEAd19mZs8D5wNanrKD\n7dkTzTLf+U78u7/jDrjxRlXQ2qSuLoK5pfDevr35Nuuysmh/HjAgxnSPGxePs28DB8Y52h5K8iCX\noF8KjDCz4UTAXwNcm3nR3fcCR0ZXm9kTwOfSo276A7vcvd7MzgFGABvzWH5pQX09/Od/wv/9vzH5\n6frr4V/+JTJEjsM9gvqZZxpuzz4bG7Pu3Nl8k8lppzUE9aRJzYf3gAFRM0/oxBzpnFoMenevM7M5\nwCPE8Mr73H2Vmd0CVLv7vONcfglwi5nVAoeBG9x9Vz4KLi37n/+BT386MuqSS6Idfty4YpeqE3rt\ntQjxpqH+6qsN5wwYAGPGxAI/x6p9d8XdyaUkaOORBNqwAT7/+diRftgw+Na34G//Vs201NbGhJvs\nMH/mmdjJOaNXr5gdNmZMw/2YMZoSLJ2eNh4pEfv2RbPMXXfFpMCvfz1G1pRcRdM9mlia1tLXro2w\nh2gnHzkyFrD66EcbAv3ss9WsIomjoE+A+nr40Y/gy1+OUXgf/jDcdlu0KCTe7t2NwzxTU89eYra8\nPEL86qsbAn3kSPVES8lQ0Hdxf/pTtMOvXAnTpsXywRMmFLtUBbJjByxaBKkUPP10hPrWrQ2vn3Za\nhPgHPtAQ6BddFMdFSpiCvotatQq+9KVYj2boUJg7F9773gS1w7vD+vWwcGHDbf36eO3EE2Odk7e+\ntSHQx4yJZWYT8x9AJH8U9F3Mpk2xdPCPfxz9hl//etToTz652CVrp9paWLGicbDX1MRrffvC9Omx\nRsP06TB+vJpdRFpBQd9F7N4Nt98O//7vMVv+05+OGn2X3Zt1375ogsmE+uLFDasWnnNObF81fXrc\nRo5UB6lIOyjoO7kDB+A//iNq7nv3woc+BLfcEoNDupStWxvX1p96Kn5jdesGY8fGyJfp06OjYdCg\nYpdWJFEU9J1UfT389KexsuTmzTFg5F//tYssPHb4cAxlzA72F16I13r2hMpK+MpXItgrK2NNFhEp\nGAV9J+MeC4598YvR4TppEvzsZ/CWtxS7ZMdx8CAsX94Q6osWNcwqPfPMCPR//Me4HztWO/+IdDAF\nfSdSVQVf+AL8+c+x18IvfwnvelcnHEjyxhvwv/8bYzsXLoQlS+IYwPnnx3KYmfb1887rhF9ApLQo\n6DuBtWujY/U3v4klU773vVh8rNNUfN1j6YBHHoGHH46AP3AgNmAePx4+8YkI9alTtVqaSCekoC+i\nrVtjqOQPfxg7pd16a4ym6RQr0+7bB48/3hDuL70Ux88/P4Y5zpgRK6VpizeRTk9BXwR79sA3vhGr\nSdbVRfP1l79c5C38Dh+O6bUPPxzh/uSTUbheveCyy+Cmm+DKK2H48CIWUkTaQkHfgQ4ehLvvjnVo\ndu2Ca6+NWvw55xSpQK+8Ao89FuH+6KPxHGIt489/PoJ9yhTt8SnSxSnoO0B9PTzwQGz+8dJLcMUV\nMfmpw9eGr62NSUqZ5phly+J4v35RqBkz4l7t7CKJoqAvIPfI05tuijW4xo+He++NlpAO89JLDcH+\n+OPR9l5WFuPXb701wn38eM08FUkwBX2BLFkSQyWfeCKaZubOhfe8pwPy9MCBGPr48MNxW7s2jpeX\nw/veF80xl16qFR1FSkhOQW9mM4C7iK0E73X3249x3ruAh4CJ6Y3BMbMvAtcD9cCn3P2RfBS8s1q/\nPoZKPvRQdK5+5zswe3YBm7ndI8wznah/+lOMae/RA/7qr+LDr7wyVnvUeHaRktRi0JtZGXA3cDmw\nGVhqZvPcfXWT83oDNwKLs46NIjYTHw0MAv5gZue7e33+vkLnsH17rEFzzz2xo9PNN8NnP1vA2f2H\nD8PPfx7jMzPLC4wcCf/wDw1DH3v2LNCHi0hXkkuNfhKwwd03ApjZXGAWsLrJebcCdwCfzzo2C5jr\n7geBF8xsQ/r9qtpb8M7koYfguuvg0CG44YbodC1of+bSpfCpT0XH6sSJ0UZ05ZWxQayISBO5BP1g\nYFPW883A5OwTzGw8UO7uvzezzze5NtXk2sFNP8DMZgOzAYYOHZpbyTuRO++EIUNijZoRIwr4Qa+8\nEu1C990Xa8j8+MexnKU6UkXkONqdEGbWDbgT+Gxb38Pd73H3Cnev6F/UWUOtl1nPa+bMAoZ8bW3M\nrjr/fPjJT6JN6Lnn4s8IhbyItCCXGv0WoDzr+ZD0sYzewEXAExadfQOAeWY2M4dru7ynnoqwr6ws\n0Af84Q9w442wenWMcb/rLrjgggJ9mIgkUS7VwaXACDMbbmYnEp2r8zIvuvted+/n7sPcfRjRVDMz\nPepmHnCNmfUws+HACGBJ3r9FEaXSDVN5D/oXX4ylKy+/PEbR/Pa3MbJGIS8irdRijd7d68xsDvAI\nMbzyPndfZWa3ANXuPu84164ysweJjts64JNJG3FTVRXt84OP6nloo9dfhzvuiMVwunWL9RI+85kY\nyiMi0gbm7sUuQyMVFRVeXV1d7GLkbPhwqKiItePbxR1+9atof3/5ZXj/+yPshwzJSzlFJNnMbJm7\nVzT3mnry2mH79mhhmTKlnW/07LMxW/U974kZq3/6UyyOo5AXkTxQ0LfD4vTUsDa3z+/eHePhx46N\nXt3vfjcWGrvkkryVUUREa920Q1VV7ALV6lUo6+tjLPyXvhTrFd9wQ0yrPeOMgpRTREqbavTtkEpF\nZfzkk1tx0ZNPxo7fs2fH+jPLlsUi9Qp5ESkQBX0b1dXFSgQ5t89v3RqzWKdNgx074Be/iLb4sWML\nWk4REQV9Gz37bIyEbLF9/uDBGD0zciQ8+GDsGbh2LVxzjVaTFJEOoTb6NsppotT8+bHb9/r1sUbC\nnXfCued2SPlERDJUo2+jqqpYV6zZBSPXr4e3vx3e9raotS9YEDNbFfIiUgQK+jZKpaJ9vlHry/79\nsW/g6NGxy9M3vwnPPBPrw4uIFImCvg1efTUWjzzSbOMO998f7fB33AHXXgvr1sHnPlfAraVERHKj\noG+DJell2Y4E/Ve+Ah/8IAwaFG06P/4xDBxYrOKJiDSiztg2qKqK9cYqMqtK/PrXsYTBo49qfXgR\n6XSUSm2QSsHFF0OvXsDOnTFc8rLLFPIi0ikpmVrp8OFY4+ZIs82TT8b99OlFK5OIyPEo6Ftp7VrY\nty8r6BcujA7XimZXBxURKToFfStVVcX9kaBftChCXhuDiEgnpaBvpVQKTj89vRH4G29AdXWsXyMi\n0knlFPRmNsPM1pnZBjO7qZnXbzCzZ8xspZktNLNR6ePDzOxA+vhKM/t+vr9AR0ulojbfrRsR8ocO\nKehFpFNrcXilmZUBdwOXA5uBpWY2z91XZ532gLt/P33+TOBOIDMd9Hl3T8QSjfv2wapVsREUEM02\nAFOnFq1MIiItyaVGPwnY4O4b3f0QMBeYlX2Cu+/LenoK0Lk2os2TJUtiEmyjjtiRI6F//6KWS0Tk\neHIJ+sHApqznm9PHGjGzT5rZ88A3gE9lvTTczFaY2Z/M7M3NfYCZzTazajOrrqmpaUXxO1ZmxcpJ\nk4hxlk8+qWGVItLp5a0z1t3vdvdzgS8AX0kf3gYMdfdxwGeAB8zs1GauvcfdK9y9on8nrh2nUjBq\nVOzfzbp1sQ2g2udFpJPLJei3AOVZz4ekjx3LXOAdAO5+0N1fTT9eBjwPnN+2ohaXe0NHLBDNNqCg\nF5FOL5egXwqMMLPhZnYicA0wL/sEMxuR9fRtwPr08f7pzlzM7BxgBLAxHwXvaM8/H6tWNho/379/\nepyliEjn1eKoG3evM7M5wCNAGXCfu68ys1uAanefB8wxs8uAWmA3cF368kuAW8ysFjgM3ODuuwrx\nRQrtqIlSCxdGbV7bAYpIJ5fT6pXuPh+Y3+TYV7Me33iM634F/Ko9BewsUino3Tva6NmxI6r4H/94\nsYslItIizYzNUSoVo23KymgYP6/2eRHpAhT0OXj9dXjqqSbNNiedBOPHF7VcIiK5UNDnoLoa6uub\ndMROmqRtAkWkS1DQ5yAzUaqykqjeL1+uZhsR6TIU9DlIpeC886BfP2IdhLo6zYgVkS5DQd8C9xha\n2ajZBmDKlKKVSUSkNRT0LXj5Zdi+vUlH7OjRsSi9iEgXoKBvQaZ9fsoUoke2qkrNNiLSpSjoW5BK\nwcknw5gxxGL0e/eqI1ZEuhQFfQtSqdgS9oQTaGifV41eRLoQBf1xHDwYIykbdcQOHAjDhhWzWCIi\nraKgP44VK2JLWC1kJiJdmYL+OBpNlNq8GV56Sc02ItLlKOiPI5WCoUNh0CC0kJmIdFkK+uNotKPU\nokVwyikwdmxRyyQi0loK+mPYti1aahoF/eTJ0D2nJfxFRDqNnILezGaY2Toz22BmNzXz+g1m9oyZ\nrTSzhWY2Kuu1L6avW2dmV+az8IXUaKLUa6/BypVqthGRLqnFoE/v+Xo3cBUwCnh/dpCnPeDuY9x9\nLPAN4M70taOIPWZHAzOA72b2kO3sUqlYhXjcOGDxYjh8WB2xItIl5VKjnwRscPeN7n4ImAvMyj7B\n3fdlPT0F8PTjWcBcdz/o7i8AG9Lv1+mlUhHyPXoQwyq7dctqxxER6TpyCfrBwKas55vTxxoxs0+a\n2fNEjf5Trbx2tplVm1l1TU1NrmUvmNpaWLq0Sfv8xRfDqacWtVwiIm2Rt85Yd7/b3c8FvgB8pZXX\n3uPuFe5e0b9//3wVqc2eeQYOHEi3z9fVRfVe7fMi0kXlEvRbgPKs50PSx45lLvCONl7bKTSaKPX0\n07B/v4JeRLqsXIJ+KTDCzIab2YlE5+q87BPMbETW07cB69OP5wHXmFkPMxsOjACWtL/YhZVKwYAB\nMVlKC5mJSFfX4qBwd68zsznAI0AZcJ+7rzKzW4Bqd58HzDGzy4BaYDdwXfraVWb2ILAaqAM+6e71\nBfoueZPZUcqM6IgtL4+biEgXlNPsH3efD8xvcuyrWY9vPM61twG3tbWAHW3nTtiwAT72MWIfwUWL\n4JJLil0sEZE208zYJhYvjvvKSmIfwS1b1D4vIl2agr6JVArKymDCBKLZBhT0ItKlKeibqKqKIfOn\nnEI02/Tund5HUESka1LQZ6mvhyVLmmw0MmVKVPFFRLooBX2WNWti/bIpU4A9e+DZZzWsUkS6PAV9\nlkYTpVKpGHWj9nkR6eIU9FlSKejbF847j2i2KSuLNehFRLowBX2WRhOlFi2K5StPOaXYxRIRaRcF\nfdqePbB6dbp9vrY2BtSr2UZEEkBBn7Z0adxXVgIrVsTyleqIFZEEUNCnpVLRZDNxIpooJSKJoqBP\nq6qCUaOgTx+ifX74cBg4sNjFEhFpNwU9MYoylUq3z2cWMlOzjYgkhIIeWL8edu9Ot88//zzs2KFm\nGxFJDAU9TSZKaaMREUkYBT3RPn/qqXDhhURH7GmnpZ+IiHR9CnqiRj95MnTrRtTop05NPxER6fpy\nSjMzm2Fm68xsg5nd1MzrnzGz1Wb2tJk9bmZnZ71Wb2Yr07d5Ta8ttr/8Jfb/rqwEXn01VjZTs42I\nJEiLWwmaWRlwN3A5sBlYambz3H111mkrgAp3f93MPg58A3hf+rUD7j42z+XOm+pqOHw4HfRPPhkH\n1RErIgmSS41+ErDB3Te6+yFgLjAr+wR3/6O7v55+mgKG5LeYhVNVFfeTJxPNNieckJ41JSKSDLkE\n/WBgU9bzzeljx3I9sCDr+UlmVm1mKTN7R3MXmNns9DnVNTU1ORQpf1IpGDECzjiD6IidMAFOPrlD\nyyAiUkh57XE0sw8CFcA3sw6f7e4VwLXAt83s3KbXufs97l7h7hX9+/fPZ5GOq9FEqYMHox1HzTYi\nkjC5BP0WoDzr+ZD0sUbM7DLgy8BMdz+YOe7uW9L3G4EngHHtKG9evfRSzI2qrASWLYuwV0esiCRM\nLkG/FBhhZsPN7ETgGqDR6BkzGwf8gAj5V7KOn25mPdKP+wHTgOxO3KJqNFEqs5DZ1KlFK4+ISCG0\nGPTuXgfMAR4B1gAPuvsqM7vFzGamT/sm0Av4ZZNhlBcC1Wb2FPBH4PYmo3WKqqoKevaEMWOIjtjz\nz4czzyx2sURE8qrF4ZUA7j4fmN/k2FezHl92jOueBMa0p4CFlErFAJvuZemFzGbObPkiEZEupmSn\nf77xRuwvUlkJrFsXk6XUESsiCVSyQb9iRewYqIXMRCTpSjboMxOljnTE9usXbfQiIglTskGfSsGw\nYTBgAFGjnzYt9hIUEUmYkg76ykpiIP369WqfF5HEKsmg37IFNm3SQmYiUhpKMuiP2lGqR49Y40ZE\nJIFKNuh79IBx44iO2IkT44CISAKVbNCPHw8n1r0Oy5drWKWIJFrJBX1tbSxSWVkJLF0aB9Q+LyIJ\nVnJB/9RTMSu20UQpLWQmIglWckGf6YidMoUI+lGjoG/fopZJRKSQSjLoBw2CIYMOx9BKNduISMKV\nZNBXVoKtWQ179qgjVkQSr6SCvqYGnn++yUYjqtGLSMKVVNAfNVHqrLPgnHOKWiYRkULLKejNbIaZ\nrTOzDWZ2UzOvf8bMVpvZ02b2uJmdnfXadWa2Pn27Lp+Fb61UCrp3T0+CXbQomm20kJmIJFyLQW9m\nZcDdwFXAKOD9ZjaqyWkrgAp3vxh4CPhG+tq+wM3AZGAScLOZnZ6/4rdOKgVvehP03LMVXnhBzTYi\nUhJyqdFPAja4+0Z3PwTMBWZln+Duf3T319NPU8CQ9OMrgcfcfZe77wYeA2bkp+itU18PS5ZooxER\nKT25BP1gYFPW883pY8dyPbCgjdcWzKpVsH9/Vkdsz54wdmwxiiIi0qHy2hlrZh8EKoBvtvK62WZW\nbWbVNTU1+SzSEUdNlJo0CU44oSCfJSLSmeQS9FuA8qznQ9LHGjGzy4AvAzPd/WBrrnX3e9y9wt0r\n+vfvn2vZWyWVit0CzzlzP6xcqWYbESkZuQT9UmCEmQ03sxOBa4B52SeY2TjgB0TIv5L10iPAFWZ2\neroT9or0sQ53ZKLUksXRYK+OWBEpES0GvbvXAXOIgF4DPOjuq8zsFjObmT7tm0Av4JdmttLM5qWv\n3QXcSvyyWArckj7WoXbvhjVrsjpizdJtOCIiydc9l5PcfT4wv8mxr2Y9vuw4194H3NfWAubDkiVx\nP2UKcPtCGDMG+vQpZpFERDpMScyMTaWiEj9xfH08UbONiJSQkgn6iy6C3i8+A6+9po5YESkpiQ/6\nw4cbOmK1kJmIlKLEB/1zz8VqxEfGzw8ZAkOHFrtYIiIdJvFB32jFyoULozavhcxEpISURND36QMj\nT34ZNm9Ws42IlJySCPrJkzyOl+AAAAffSURBVKFblRYyE5HSlOigf+01eOaZrGabXr1iDL2ISAlJ\ndNBXV8eomyMdsVOmxM4jIiIlJNFBn+mInXzBXnj6abXPi0hJSnzQjxwJp69LgbuCXkRKUmKD3h2q\nqrIWMisri15ZEZESk9igf+EFqKlJt88vXBibxfbuXexiiYh0uMQG/ZGJUhNqYfFiDasUkZKV6KA/\n5RQYXbsSXn9d7fMiUrISG/RVVTBxInRfnJ4opaAXkRKVyKA/cCC2hT3SPj9sGAweXOxiiYgURU5B\nb2YzzGydmW0ws5uaef0SM1tuZnVm9u4mr9Wntxc8ssVgoS1fDnV1UDnZY8SNavMiUsJanCZqZmXA\n3cDlwGZgqZnNc/fVWae9DHwY+Fwzb3HA3cfmoaw5y3TETh34Amzfro5YESlpuawHMAnY4O4bAcxs\nLjALOBL07v5i+rXDBShjq1VVwfDh0G+tNhoREcml6WYwsCnr+eb0sVydZGbVZpYys3c0d4KZzU6f\nU11TU9OKt25eKpW1vk2fPjB6dLvfU0Skq+qIztiz3b0CuBb4tpmd2/QEd7/H3SvcvaJ///7t+rDN\nm2HLlqwVK6dOhW6J7HMWEclJLgm4BSjPej4kfSwn7r4lfb8ReAIY14rytVqmfX7ahbtg9Wo124hI\nycsl6JcCI8xsuJmdCFwD5DR6xsxON7Me6cf9gGlkte0XQlUV9OgBY/ZXxQF1xIpIiWsx6N29DpgD\nPAKsAR5091VmdouZzQQws4lmthl4D/ADM1uVvvxCoNrMngL+CNzeZLRO3qVSMGECnLB4Yaw9P3Fi\nIT9ORKTTM3cvdhkaqaio8Orq6jZde+gQnHoqzJkD31pyCRw8GOvciIgknJktS/eHHiVRvZRPPRXZ\nPnXCQViyRM02IiIkLOgzHbHTey6PxFdHrIhIsoK+qiqWtDlzvRYyExHJSFTQH5kotXAhnHcenHVW\nsYskIlJ0iQn6HTtiVyktZCYi0lgua910Cb17w29/C+N6Pgc7d6ojVkQkLTE1+p49YeZMKH9Z7fMi\nItkSE/RHLFoEZ5wBF1xQ7JKIiHQKyQv6zEJmZsUuiYhIp5CsoK+pgeeeU7ONiEiWZAX9onT7vDpi\nRUSOSF7Qn3hirGomIiJAEoN+4kQ46aRil0REpNNITtAfOADV1WqfFxFpIjlBv28fvOc9cMUVxS6J\niEinkpiZsZx1Ftx/f7FLISLS6SSnRi8iIs3KKejNbIaZrTOzDWZ2UzOvX2Jmy82szsze3eS168xs\nffp2Xb4KLiIiuWkx6M2sDLgbuAoYBbzfzEY1Oe1l4MPAA02u7QvcDEwGJgE3m9np7S+2iIjkKpca\n/SRgg7tvdPdDwFxgVvYJ7v6iuz8NHG5y7ZXAY+6+y913A48BM/JQbhERyVEuQT8Y2JT1fHP6WC5y\nutbMZptZtZlV19TU5PjWIiKSi07RGevu97h7hbtX9O/fv9jFERFJlFyCfgtQnvV8SPpYLtpzrYiI\n5EEuQb8UGGFmw83sROAaYF6O7/8IcIWZnZ7uhL0ifUxERDqIuXvLJ5ldDXwbKAPuc/fbzOwWoNrd\n55nZROA3wOnAG8B2dx+dvvYjwJfSb3Wbu/+ohc+qAV5q6xcC+gE723F9V1Rq37nUvi/oO5eK9nzn\ns9292bbvnIK+KzGzanevKHY5OlKpfedS+76g71wqCvWdO0VnrIiIFI6CXkQk4ZIY9PcUuwBFUGrf\nudS+L+g7l4qCfOfEtdGLiEhjSazRi4hIFgW9iEjCJSboW1pKOWnMrNzM/mhmq81slZndWOwydRQz\nKzOzFWb2u2KXpSOY2Wlm9pCZrTWzNWY2pdhlKjQz+6f0z/WzZvYLM0vcRtBmdp+ZvWJmz2Yd62tm\nj6WXdX8sX6v9JiLoc1xKOWnqgM+6+yigEvhkCXznjBuBNcUuRAe6C3jY3S8A3kTCv7uZDQY+BVS4\n+0XERM1riluqgvgxR6/mexPwuLuPAB5PP2+3RAQ9OSylnDTuvs3dl6cfv0b84891VdEuy8yGAG8D\n7i12WTqCmfUBLgF+CODuh9x9T3FL1SG6AyebWXegJ7C1yOXJO3f/X2BXk8OzgJ+kH/8EeEc+Pisp\nQd+epZS7PDMbBowDFhe3JB3i28D/4ei9D5JqOFAD/CjdXHWvmZ1S7EIVkrtvAb5FbGi0Ddjr7o8W\nt1Qd5ix335Z+vB04Kx9vmpSgL1lm1gv4FfBpd99X7PIUkpm9HXjF3ZcVuywdqDswHvieu48D/kKe\n/pzvrNLt0rOIX3KDgFPM7IPFLVXH8xj7npfx70kJ+pJcDtnMTiBC/n53/3Wxy9MBpgEzzexFonnu\nrWb28+IWqeA2A5vdPfPX2kNE8CfZZcAL7l7j7rXAr4GpRS5TR9lhZgMB0vev5ONNkxL07VlKuUsy\nMyPabde4+53FLk9HcPcvuvsQdx9G/D/+H3dPdE3P3bcDm8xsZPrQpcDqIhapI7wMVJpZz/TP+aUk\nvAM6yzzguvTj64Df5uNNu+fjTYrN3evMbA6x1n1mKeVVRS5WoU0DPgQ8Y2Yr08e+5O7zi1gmKYx/\nBO5PV2I2An9f5PIUlLsvNrOHgOXE6LIVJHA5BDP7BfAWoJ+ZbQZuBm4HHjSz64nl2t+bl8/SEggi\nIsmWlKYbERE5BgW9iEjCKehFRBJOQS8iknAKehGRhFPQi4gknIJeRCTh/j+nEN6xOy7rNgAAAABJ\nRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    }
  ]
}