MNITS_TPU.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "MNITS_TPU.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyMZKsUqU4RCqA7Y24OcSs7c",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/wenxuan0923/My-notes/blob/master/MNITS_TPU.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rG22pheGNlL-",
        "colab_type": "text"
      },
      "source": [
        "## Train Keras model with TPU on Google Colab\n",
        "### - An Example using MNIST dataset with TF 2.0\n",
        "\n",
        "Most tutorials available online are using the function `tf.contrib.tpu.keras_to_tpu_model` to set up the distribution strategy, which has been deprecated in TF 2.0.  In this note I will use train a simple handwritten digits classifier using Keras and Colab TPU. Be sure to change **runtime type** to be **TPU** when running this notebook in Colab.\n",
        "\n",
        "Reference: \n",
        "\n",
        "[tf.distribute.Strategy Documentation](https://www.tensorflow.org/api_docs/python/tf/distribute/Strategy)<br>\n",
        "[A helpful Github issue](https://github.com/huan/tensorflow-handbook-tpu/issues/1)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DAA87jnZ3k7J",
        "colab_type": "code",
        "outputId": "f968b8cd-ae59-4880-bd9f-316747d86045",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import keras\n",
        "from keras.datasets import mnist\n",
        "from sklearn.model_selection import train_test_split\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YdaX4JqI49D3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Authenticates the Colab machine and also the TPU using your\n",
        "# credentials so that they can access your private GCS buckets.\n",
        "from google.colab import auth\n",
        "auth.authenticate_user()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NGbBLXnY5Kog",
        "colab_type": "code",
        "outputId": "d3f43358-9f59-4a5c-cb00-290380a926f6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "# Detect hardware\n",
        "try:\n",
        "  # TPU detection\n",
        "  tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n",
        "  print('Running on TPU ', tpu.cluster_spec().as_dict()['worker'])\n",
        "except ValueError:\n",
        "  tpu = None\n",
        "  gpus = tf.config.experimental.list_logical_devices(\"GPU\")"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Running on TPU  ['10.21.193.202:8470']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sEwEeZQA5UY7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 763
        },
        "outputId": "ffc44a85-bf5d-4fbe-b076-dfdfceb07dfc"
      },
      "source": [
        "# Select appropriate distribution strategy\n",
        "tf.config.experimental_connect_to_cluster(tpu)\n",
        "tf.tpu.experimental.initialize_tpu_system(tpu)\n",
        "strategy = tf.distribute.experimental.TPUStrategy(tpu)\n",
        "print(\"REPLICAS: \", strategy.num_replicas_in_sync)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Initializing the TPU system: grpc://10.21.193.202:8470\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Initializing the TPU system: grpc://10.21.193.202:8470\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Clearing out eager caches\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Clearing out eager caches\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished initializing TPU system.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished initializing TPU system.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Found TPU system:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Found TPU system:\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores: 8\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores: 8\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Workers: 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Workers: 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "REPLICAS:  8\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GfbRaAkhQaJG",
        "colab_type": "text"
      },
      "source": [
        "### Data preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "53Pcfmi-65cm",
        "colab_type": "code",
        "outputId": "c1da03fe-cd48-4890-b029-9c50d351b5dd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "# the original data is 3D tensor\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "print(x_train.shape)\n",
        "print(x_test.shape)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(60000, 28, 28)\n",
            "(10000, 28, 28)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9lCNM8hSCPWi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def preprocess_input(x):\n",
        "    desired_shape = (-1, 28, 28, 1)\n",
        "    return x.reshape(desired_shape) / 255.0\n",
        "\n",
        "def preprocess_output(y):\n",
        "    return keras.utils.to_categorical(y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IPVcw573CTCl",
        "colab_type": "code",
        "outputId": "d4df8762-a1ec-457d-8d66-790444353c45",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "x_train, x_test = map(preprocess_input, [x_train, x_test])\n",
        "y_train, y_test = map(preprocess_output, [y_train, y_test])\n",
        "\n",
        "print(x_train.shape)\n",
        "print(y_train.shape)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(60000, 28, 28, 1)\n",
            "(60000, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0GxeKoFd_Gsg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        },
        "outputId": "b7d83014-db3f-4334-fcdc-744d6b335b88"
      },
      "source": [
        "X_train, X_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.2, random_state=42)\n",
        "print(X_train.shape)\n",
        "print(X_val.shape)\n",
        "print(y_train.shape)\n",
        "print(y_val.shape)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(48000, 28, 28, 1)\n",
            "(12000, 28, 28, 1)\n",
            "(48000, 10)\n",
            "(12000, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TXd3v8fdQvUf",
        "colab_type": "text"
      },
      "source": [
        "### Keras model: 3 convolutional layers, 2 dense layers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KrgeofDF7GrQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras import models, layers\n",
        "def create_model():\n",
        "    model = models.Sequential()\n",
        "    model.add(layers.Conv2D(32, (3, 3), activation='relu'))\n",
        "    model.add(layers.MaxPool2D(2, 2))\n",
        "    model.add(layers.Dropout(0.25))\n",
        "\n",
        "    model.add(layers.Conv2D(32, (3, 3), activation='relu'))\n",
        "    model.add(layers.MaxPool2D(2, 2))\n",
        "    model.add(layers.Dropout(0.25))\n",
        "\n",
        "    model.add(layers.Conv2D(32, (3, 3), activation='relu'))\n",
        "    model.add(layers.MaxPool2D(2, 2))\n",
        "    model.add(layers.Dropout(0.25))\n",
        "\n",
        "    model.add(layers.Flatten())\n",
        "    model.add(layers.Dense(251, activation='relu'))\n",
        "    model.add(layers.Dropout(0.5))\n",
        "    model.add(layers.Dense(10, activation='softmax'))\n",
        "    return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oE2eIRrXQLYz",
        "colab_type": "text"
      },
      "source": [
        "### Model Parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ha_oLpUI5Ubk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "total_train = len(X_train)\n",
        "total_val = len(X_val)\n",
        "BATCH_SIZE = 64 * strategy.num_replicas_in_sync # Gobal batch size.\n",
        "# The global batch size will be automatically sharded across all\n",
        "# replicas by the tf.data.Dataset API. A single TPU has 8 cores.\n",
        "# The best practice is to scale the batch size by the number of\n",
        "# replicas (cores). The learning rate should be increased as well.\n",
        "LEARNING_RATE = 0.01\n",
        "LEARNING_RATE_EXP_DECAY = 0.6 if strategy.num_replicas_in_sync == 1 else 0.7\n",
        "# Learning rate computed later in call back function\n",
        "# 0.7 decay instead of 0.6 means a slower decay, i.e. a faster learnign rate."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ftcEfPT5QtvK",
        "colab_type": "text"
      },
      "source": [
        "### Callbacks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bcIpb_0h-0ru",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# set up learning rate decay\n",
        "lr_decay = keras.callbacks.LearningRateScheduler(\n",
        "    lambda epoch: LEARNING_RATE * LEARNING_RATE_EXP_DECAY**epoch,\n",
        "    verbose=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bfJFmMz2QUrS",
        "colab_type": "text"
      },
      "source": [
        "### Train and validate the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aB0gmtXM7Gt4",
        "colab_type": "code",
        "outputId": "01190007-7a8b-49dd-ff62-cfda51ed82c8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "with strategy.scope():\n",
        "  model = create_model()\n",
        "  model.compile(\n",
        "      optimizer='rmsprop',\n",
        "      loss='categorical_crossentropy',\n",
        "      metrics=['accuracy']\n",
        "  )\n",
        "\n",
        "model.fit(\n",
        "    X_train, \n",
        "    y_train,\n",
        "    epochs = 14,\n",
        "    steps_per_epoch = total_train//BATCH_SIZE,\n",
        "    callbacks = [lr_decay],\n",
        "    validation_data = (X_val, y_val),\n",
        "    validation_steps = total_val//BATCH_SIZE,\n",
        ")"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 48000 samples, validate on 12000 samples\n",
            "Epoch 1/14\n",
            "\n",
            "Epoch 00001: LearningRateScheduler setting learning rate to 0.01.\n",
            "93/93 [==============================] - 231s 2s/step - loss: 1.0727 - accuracy: 0.6451 - val_loss: 0.0070 - val_accuracy: 21.9133\n",
            "Epoch 2/14\n",
            "\n",
            "Epoch 00002: LearningRateScheduler setting learning rate to 0.006999999999999999.\n",
            "93/93 [==============================] - 227s 2s/step - loss: 0.2324 - accuracy: 0.9309 - val_loss: 0.0035 - val_accuracy: 22.4078\n",
            "Epoch 3/14\n",
            "\n",
            "Epoch 00003: LearningRateScheduler setting learning rate to 0.0049.\n",
            "93/93 [==============================] - 231s 2s/step - loss: 0.1439 - accuracy: 0.9571 - val_loss: 0.0030 - val_accuracy: 22.5266\n",
            "Epoch 4/14\n",
            "\n",
            "Epoch 00004: LearningRateScheduler setting learning rate to 0.003429999999999999.\n",
            "93/93 [==============================] - 228s 2s/step - loss: 0.1131 - accuracy: 0.9662 - val_loss: 0.0024 - val_accuracy: 22.6435\n",
            "Epoch 5/14\n",
            "\n",
            "Epoch 00005: LearningRateScheduler setting learning rate to 0.0024009999999999995.\n",
            "93/93 [==============================] - 227s 2s/step - loss: 0.0973 - accuracy: 0.9706 - val_loss: 0.0021 - val_accuracy: 22.6703\n",
            "Epoch 6/14\n",
            "\n",
            "Epoch 00006: LearningRateScheduler setting learning rate to 0.0016806999999999994.\n",
            "93/93 [==============================] - 227s 2s/step - loss: 0.0887 - accuracy: 0.9730 - val_loss: 0.0021 - val_accuracy: 22.6953\n",
            "Epoch 7/14\n",
            "\n",
            "Epoch 00007: LearningRateScheduler setting learning rate to 0.0011764899999999997.\n",
            "93/93 [==============================] - 230s 2s/step - loss: 0.0836 - accuracy: 0.9746 - val_loss: 0.0020 - val_accuracy: 22.7106\n",
            "Epoch 8/14\n",
            "\n",
            "Epoch 00008: LearningRateScheduler setting learning rate to 0.0008235429999999996.\n",
            "93/93 [==============================] - 230s 2s/step - loss: 0.0807 - accuracy: 0.9753 - val_loss: 0.0019 - val_accuracy: 22.7163\n",
            "Epoch 9/14\n",
            "\n",
            "Epoch 00009: LearningRateScheduler setting learning rate to 0.0005764800999999997.\n",
            "93/93 [==============================] - 228s 2s/step - loss: 0.0786 - accuracy: 0.9760 - val_loss: 0.0019 - val_accuracy: 22.7240\n",
            "Epoch 10/14\n",
            "\n",
            "Epoch 00010: LearningRateScheduler setting learning rate to 0.0004035360699999998.\n",
            "93/93 [==============================] - 232s 2s/step - loss: 0.0772 - accuracy: 0.9763 - val_loss: 0.0019 - val_accuracy: 22.7144\n",
            "Epoch 11/14\n",
            "\n",
            "Epoch 00011: LearningRateScheduler setting learning rate to 0.00028247524899999984.\n",
            "93/93 [==============================] - 229s 2s/step - loss: 0.0763 - accuracy: 0.9766 - val_loss: 0.0019 - val_accuracy: 22.7259\n",
            "Epoch 12/14\n",
            "\n",
            "Epoch 00012: LearningRateScheduler setting learning rate to 0.0001977326742999999.\n",
            "93/93 [==============================] - 231s 2s/step - loss: 0.0755 - accuracy: 0.9768 - val_loss: 0.0019 - val_accuracy: 22.7202\n",
            "Epoch 13/14\n",
            "\n",
            "Epoch 00013: LearningRateScheduler setting learning rate to 0.0001384128720099999.\n",
            "93/93 [==============================] - 231s 2s/step - loss: 0.0749 - accuracy: 0.9771 - val_loss: 0.0018 - val_accuracy: 22.7240\n",
            "Epoch 14/14\n",
            "\n",
            "Epoch 00014: LearningRateScheduler setting learning rate to 9.688901040699991e-05.\n",
            "93/93 [==============================] - 231s 2s/step - loss: 0.0747 - accuracy: 0.9771 - val_loss: 0.0018 - val_accuracy: 22.7240\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.callbacks.History at 0x7f7ab1280240>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ah2pV4ke7Gpf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.save_weights('./mnist_TPU.h5', overwrite=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lzZd9orWPbOL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}