13_rnn/05_Attention_alpha.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/machine-perception-robotics-group/MPRGDeepLearningLectureNotebook/blob/master/13_rnn/05_Attention_alpha.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rLp4luhzWiz2"
      },
      "source": [
        "# Attention Seq2seqによる日付変換\n",
        "\n",
        "---\n",
        "\n",
        "## Google ColabでGoogleドライブをマウントする\n",
        "* 以下のコードを実行するとURLが出てくるので，クリック\n",
        "* Googleアカウントに許可を求めたらcodeが出るのでコピー\n",
        "* コピーしたcodeを[空白]にペースト"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9INjbE-gW7IP",
        "outputId": "88abe790-49e9-43ee-db02-55c1a917d98a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VEiL-x2K0Yx9"
      },
      "source": [
        "# Attention Seq2seq\n",
        "前セクションのAttention Seq2seqの問題設定を少し変えてみましょう．Attention機構は，機械翻訳や文章生成，質問応答など幅広いタスクで応用できます．本セクションでは日付変換，つまり(曜日，日付，年数)を(年数-月-日)に変換させるAttention Seq2seqで解きます．\n",
        "\n",
        "<img src=\"https://github.com/himidev/Lecture/blob/main/13_rnn/05_Attention/Atten_Seq2seq.png?raw=true\" width = 100%>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dBpeoftq8TQN"
      },
      "source": [
        "#データ準備\n",
        "日付変換をするために， ０から学ぶDeep Learning2のGithubからデータを拝借します．date.txtを学習用と評価用に分割するのと，必要に応じてデータを整理します．"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oizofu2XxBaG"
      },
      "outputs": [],
      "source": [
        "# データを予めDL\n",
        "!wget -q \"https://drive.google.com/uc?export=download&id=1FKO72lVQ-yNpktQLvj93R2eE9AZJgbLs\" -O date.txt\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "import random\n",
        "from sklearn.utils import shuffle\n",
        "\n",
        "# 事前にGoogle Driveをマウントして以下の場所にdate.txtを格納しておきます\n",
        "file_path = \"./date.txt\"\n",
        "\n",
        "input_date = [] # 変換前の日付データ\n",
        "output_date = [] # 変換後の日付データ\n",
        "\n",
        "# date.txtを1行ずつ読み込んで変換前と変換後に分割して、inputとoutputで分ける\n",
        "with open(file_path, \"r\") as f:\n",
        "  date_list = f.readlines()\n",
        "  for date in date_list:\n",
        "    date = date[:-1]\n",
        "    input_date.append(date.split(\"_\")[0])\n",
        "    output_date.append(\"_\" + date.split(\"_\")[1])\n",
        "\n",
        "# inputとoutputの系列の長さを取得\n",
        "# すべて長さが同じなので、0番目の要素でlenを取ってます\n",
        "input_len = len(input_date[0]) # 29\n",
        "output_len = len(output_date[0]) # 10\n",
        "\n",
        "# date.txtで登場するすべての文字にIDを割り当てる\n",
        "char2id = {}\n",
        "for input_chars, output_chars in zip(input_date, output_date):\n",
        "  for c in input_chars:\n",
        "    if not c in char2id:\n",
        "      char2id[c] = len(char2id)\n",
        "  for c in output_chars:\n",
        "    if not c in char2id:\n",
        "      char2id[c] = len(char2id)\n",
        "\n",
        "input_data = [] # ID化された変換前日付データ\n",
        "output_data = [] # ID化された変換後日付データ\n",
        "for input_chars, output_chars in zip(input_date, output_date):\n",
        "  if len([char2id[c] for c in output_chars]) != 11:\n",
        "    char2id_ = [char2id[c] for c in output_chars]\n",
        "    char0list = [0 for i in range(11-len([char2id[c] for c in output_chars]))]\n",
        "    char2id_.extend(char0list)\n",
        "    output_data.append(char2id_)\n",
        "  else:\n",
        "    output_data.append([char2id[c] for c in output_chars])\n",
        "  input_data.append([char2id[c] for c in input_chars])\n",
        "# 7:3でtrainとtestに分ける\n",
        "train_x, test_x, train_y, test_y = train_test_split(input_data, output_data, train_size= 0.7)\n",
        "\n",
        "# データをバッチ化するための関数を定義\n",
        "def train2batch(input_data, output_data, batch_size=100):\n",
        "    input_batch = []\n",
        "    output_batch = []\n",
        "    input_shuffle, output_shuffle = shuffle(input_data, output_data)\n",
        "    for i in range(0, len(input_data), batch_size):\n",
        "      input_batch.append(input_shuffle[i:i+batch_size])\n",
        "      output_batch.append(output_shuffle[i:i+batch_size])\n",
        "    return input_batch, output_batch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jajTAlji8jLy"
      },
      "source": [
        "###エンコーダ・デコーダの作成\n",
        "基本的な構造は前セクションのエンコーダ・デコーダ構造と同様です．ただし，エンコーダは各時刻の出力値を保持しておきます．デコーダでは，保持したエンコーダの出力値とデコーダの出力値で内積計算します．この内積計算によって，各時刻のエンコーダの出力値に重み付けすることができます．これにより，どの時刻のエンコーダの出力に着目したかをデコーダ側が自動で決定することができます．"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y4opavYMyl5k"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "\n",
        "# 諸々のパラメータなど\n",
        "embedding_dim = 200\n",
        "hidden_dim = 128\n",
        "BATCH_NUM = 100\n",
        "learning_rate = 0.001\n",
        "vocab_size = len(char2id)\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "# Encoderクラス\n",
        "class Encoder(nn.Module):\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim):\n",
        "        super(Encoder, self).__init__()\n",
        "        self.hidden_dim = hidden_dim\n",
        "        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim, padding_idx=char2id[\" \"])\n",
        "        self.gru = nn.GRU(embedding_dim, hidden_dim, batch_first=True)\n",
        "\n",
        "    def forward(self, sequence):\n",
        "        embedding = self.word_embeddings(sequence)\n",
        "        # hsが各系列のGRUの隠れ層のベクトル\n",
        "        # Attentionされる要素\n",
        "        hs, h = self.gru(embedding)\n",
        "        return hs, h\n",
        "\n",
        "# Attention Decoderクラス\n",
        "class AttentionDecoder(nn.Module):\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim, batch_size):\n",
        "        super(AttentionDecoder, self).__init__()\n",
        "        self.hidden_dim = hidden_dim\n",
        "        self.batch_size = batch_size\n",
        "        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim, padding_idx=char2id[\" \"])\n",
        "        self.gru = nn.GRU(embedding_dim, hidden_dim, batch_first=True)\n",
        "        # hidden_dim*2としているのは、各系列のGRUの隠れ層とAttention層で計算したコンテキストベクトルをtorch.catでつなぎ合わせることで長さが２倍になるため\n",
        "        self.hidden2linear = nn.Linear(hidden_dim * 2, vocab_size)\n",
        "        # 列方向を確率変換したいのでdim=1\n",
        "        self.softmax = nn.Softmax(dim=1)\n",
        "\n",
        "    def forward(self, sequence, hs, h):\n",
        "        embedding = self.word_embeddings(sequence)\n",
        "        output, state = self.gru(embedding, h)\n",
        "\n",
        "        t_output = torch.transpose(output, 1, 2) \n",
        "        s = torch.bmm(hs, t_output) \n",
        "        attention_weight = self.softmax(s)\n",
        "        c = torch.zeros(self.batch_size, 1, self.hidden_dim, device=device)\n",
        "\n",
        "        # Attention weight\n",
        "        for i in range(attention_weight.size()[2]): \n",
        "          unsq_weight = attention_weight[:,:,i].unsqueeze(2) \n",
        "          weighted_hs = hs * unsq_weight \n",
        "          weight_sum = torch.sum(weighted_hs, axis=1).unsqueeze(1)\n",
        "          c = torch.cat([c, weight_sum], dim=1) \n",
        "        c = c[:,1:,:]\n",
        "        output = torch.cat([output, c], dim=2)\n",
        "        output = self.hidden2linear(output)\n",
        "        return output, state, attention_weight"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ayrd8ydF85eO"
      },
      "source": [
        "#モデル定義，損失関数，最適化手法\n",
        "EncoderとAttentionDecoderに各パラメータを引数として入れます．損失関数にはCrossEntropyLossを用います．pytorchのCrossEntropyLossには内部にSoftmax関数を持つため，ネットワークの出力後にSoftmax関数を割り当てる必要はありません．最適化手法にはAdamを使用します．"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qgudmNZCzFsi"
      },
      "outputs": [],
      "source": [
        "encoder = Encoder(vocab_size, embedding_dim, hidden_dim).to(device)\n",
        "attn_decoder = AttentionDecoder(vocab_size, embedding_dim, hidden_dim, BATCH_NUM).to(device)\n",
        "\n",
        "# 損失関数\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "\n",
        "# 最適化\n",
        "encoder_optimizer = optim.Adam(encoder.parameters(), lr=learning_rate)\n",
        "attn_decoder_optimizer = optim.Adam(attn_decoder.parameters(), lr=learning_rate)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wK-tRO9V8-kq"
      },
      "source": [
        "#学習\n",
        "* lossが0.1を下回ったら終了とします"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Hj6TyDySzH0y",
        "outputId": "6367ebbb-c116-489e-ace1-32203d2afa8e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "training ...\n",
            "Epoch 1: 1462.50\n",
            "Epoch 2: 131.24\n",
            "Epoch 3: 82.71\n",
            "Epoch 4: 75.80\n",
            "Epoch 5: 74.11\n",
            "Epoch 6: 90.89\n",
            "Epoch 7: 84.50\n",
            "Epoch 8: 73.56\n",
            "Epoch 9: 73.26\n",
            "Epoch 10: 73.00\n",
            "Epoch 11: 72.30\n",
            "Epoch 12: 72.06\n",
            "Epoch 13: 71.88\n",
            "Epoch 14: 71.77\n",
            "Epoch 15: 72.51\n",
            "Epoch 16: 88.74\n",
            "Epoch 17: 72.95\n",
            "Epoch 18: 72.70\n",
            "Epoch 19: 72.02\n",
            "Epoch 20: 71.63\n",
            "Epoch 21: 71.87\n",
            "Epoch 22: 71.67\n",
            "Epoch 23: 78.43\n",
            "Epoch 24: 21.66\n",
            "Epoch 25: 2.11\n",
            "Epoch 26: 0.67\n",
            "Epoch 27: 0.44\n",
            "Epoch 28: 0.34\n",
            "Epoch 29: 0.27\n",
            "Epoch 30: 0.23\n",
            "Epoch 31: 7.06\n",
            "Epoch 32: 1.88\n",
            "Epoch 33: 0.44\n",
            "Epoch 34: 0.27\n",
            "Epoch 35: 0.20\n",
            "Epoch 36: 0.17\n",
            "Epoch 37: 0.14\n",
            "Epoch 38: 0.13\n",
            "Epoch 39: 0.12\n",
            "Epoch 40: 0.10\n",
            "Epoch 41: 0.09\n",
            "Done\n"
          ]
        }
      ],
      "source": [
        "BATCH_NUM=100\n",
        "EPOCH_NUM = 100\n",
        "\n",
        "all_losses = []\n",
        "print(\"training ...\")\n",
        "for epoch in range(1, EPOCH_NUM+1):\n",
        "    epoch_loss = 0\n",
        "    # データをミニバッチに分ける\n",
        "    input_batch, output_batch = train2batch(train_x, train_y, batch_size=BATCH_NUM)\n",
        "    for i in range(len(input_batch)):\n",
        "\n",
        "        # 勾配の初期化\n",
        "        encoder_optimizer.zero_grad()\n",
        "        attn_decoder_optimizer.zero_grad()\n",
        "\n",
        "        # データをテンソルに変換\n",
        "        input_tensor = torch.tensor(input_batch[i], device=device)\n",
        "        output_tensor = torch.tensor(output_batch[i], device=device)\n",
        "\n",
        "        # Encoderの順伝搬\n",
        "        hs, h = encoder(input_tensor)\n",
        "\n",
        "        # Attention Decoderのインプット\n",
        "        source = output_tensor[:, :-1]\n",
        "\n",
        "        # Attention Decoderの正解データ\n",
        "        target = output_tensor[:, 1:]\n",
        "\n",
        "        loss = 0\n",
        "        decoder_output, _, attention_weight= attn_decoder(source, hs, h)\n",
        "        for j in range(decoder_output.size()[1]):\n",
        "            loss += criterion(decoder_output[:, j, :], target[:, j])\n",
        "\n",
        "        epoch_loss += loss.item()\n",
        "\n",
        "        # 誤差逆伝播\n",
        "        loss.backward()\n",
        "\n",
        "        # パラメータ更新\n",
        "        encoder_optimizer.step()\n",
        "        attn_decoder_optimizer.step()\n",
        "\n",
        "    # 損失を表示\n",
        "    print(\"Epoch %d: %.2f\" % (epoch, epoch_loss))\n",
        "    all_losses.append(epoch_loss)\n",
        "    if epoch_loss < 0.1: break\n",
        "print(\"Done\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YU8qHKsX9Bhv"
      },
      "source": [
        "#学習推移の可視化"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "TeCN_11nzK4K",
        "outputId": "cac8db49-ce39-4166-f402-e90582aa8ffc"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7fbcafcbd8d0>]"
            ]
          },
          "execution_count": 32,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAbJ0lEQVR4nO3de5BcZ53e8e/Tl5keSZZ1GwsjiUiAMAgWjCOMiReWYLBlr4NcKdaFQxYBrqi24k1gIfEaUrXehaKy1CaYSxHXCqzYTijAxbKxsuXgVdkmBoONx2DAVzzYgKToMkJXW5r7L3+ctzU9F92m5yL1+3yqpvr0e073efto9PQ773nPeRURmJlZHkqzXQEzM5s5Dn0zs4w49M3MMuLQNzPLiEPfzCwjldmuwIksWbIkVq5cOdvVMDM7qzz22GN7I6JzonVndOivXLmSrq6u2a6GmdlZRdJvjrfO3TtmZhlx6JuZZcShb2aWEYe+mVlGHPpmZhlx6JuZZcShb2aWkZOGvqTNkvZIemKCdZ+QFJKWpOeS9CVJ3ZJ+Lumihm03SHou/WyY2o8x2ot9g3x+6y/56W/3T+duzMzOOqfS0r8dWDe2UNIK4HLgtw3FVwKr089G4Na07SLgZuCtwMXAzZIWNlPxExkYHOZL9z3H49sOTNcuzMzOSicN/Yh4ENg3wapbgBuBxllY1gN3RuFhYIGk84ErgK0RsS8i9gNbmeCLZKp0tJUB6B0Ynq5dmJmdlSbVpy9pPbAjIn42ZtUyYFvD8+2p7HjlE733Rkldkrp6enomUz3aK8XHOjowNKnXm5m1qtMOfUlzgE8BfzH11YGI2BQRayNibWfnhPcLOilJ1Koleh36ZmajTKal/ypgFfAzSb8GlgM/kfQyYAewomHb5anseOXTpqNaduibmY1x2qEfEb+IiPMiYmVErKToqrkoInYBW4APplE8lwAHI2IncC9wuaSF6QTu5als2tSqZY72O/TNzBqdypDNbwA/Ai6QtF3S9SfY/B7geaAb+CrwbwEiYh/wGeDR9PPpVDZtOqpl9+mbmY1x0vvpR8R1J1m/smE5gBuOs91mYPNp1m/Sau7eMTMbp2WvyC1O5HrIpplZo5YN/Y42d++YmY3VuqHvE7lmZuO0bOi3V8v0Djr0zcwatWzod1TL9Lqlb2Y2SkuHvvv0zcxGa93Qbyt79I6Z2RgtG/q1SomjA0MUlw6YmRm0cuin2yv3Dbq1b2ZW17Kh31Gt31Pf/fpmZnUtG/q1FPo+mWtmNqJlQ7/e0vcFWmZmI1o29N3SNzMbr4VDv/hoHrZpZjaiZUPfJ3LNzMZr3dBvc5++mdlYLRv69T5933TNzGxEy4a+R++YmY3XsqFfc5++mdk4pzIx+mZJeyQ90VD2N5KekfRzSX8vaUHDuk9K6pb0rKQrGsrXpbJuSTdN/UcZrd6n79E7ZmYjTqWlfzuwbkzZVuANEfFG4JfAJwEkrQHeD7w+vea/SSpLKgNfAa4E1gDXpW2nTa1SfDSP0zczG3HS0I+IB4F9Y8r+MSIG09OHgeVpeT3wzYjoi4gXgG7g4vTTHRHPR0Q/8M207bSplEtUy3Lom5k1mIo+/Y8A/yctLwO2NazbnsqOVz6OpI2SuiR19fT0NFWxWrXsPn0zswZNhb6k/wQMAl+fmupARGyKiLURsbazs7Op93Lom5mNVpnsCyV9CLgauCxGZirZAaxo2Gx5KuME5dOmo1r2kE0zswaTaulLWgfcCLw3Io40rNoCvF9Su6RVwGrgx8CjwGpJqyS1UZzs3dJc1U/O8+SamY120pa+pG8A7wSWSNoO3EwxWqcd2CoJ4OGI+JOIeFLSXcBTFN0+N0TEUHqfPwXuBcrA5oh4cho+zyi1aslDNs3MGpw09CPiugmKbzvB9p8FPjtB+T3APadVuybV3NI3MxulZa/IheICLZ/INTMb0dKhX6s49M3MGrV06He0uXvHzKxRS4d+rVrmaL9P5JqZ1bV06HdUy/S5pW9mdkxLh36tWnL3jplZg5YO/Y5qmcHhYGDIXTxmZtDqoV+fJ9etfTMzoMVDv92zZ5mZjdLSoV+fJ7fXI3jMzIBMQt/dO2ZmhZYO/Vq1+Hju3jEzK7R06Lulb2Y2WkuHfs2jd8zMRmnt0K8Uoe+rcs3MCi0d+h6nb2Y2WmuHfr1P30M2zcyATELfo3fMzAotHfrtacimu3fMzAonDX1JmyXtkfREQ9kiSVslPZceF6ZySfqSpG5JP5d0UcNrNqTtn5O0YXo+zmjtlRKSW/pmZnWn0tK/HVg3puwm4L6IWA3cl54DXAmsTj8bgVuh+JIAbgbeClwM3Fz/ophOkuioljna79A3M4NTCP2IeBDYN6Z4PXBHWr4DuKah/M4oPAwskHQ+cAWwNSL2RcR+YCvjv0imRa1apnfQoW9mBpPv018aETvT8i5gaVpeBmxr2G57Kjte+TiSNkrqktTV09MzyeqN6PCUiWZmxzR9IjciAogpqEv9/TZFxNqIWNvZ2dn0+9WqJffpm5klkw393anbhvS4J5XvAFY0bLc8lR2vfNrVqmWHvplZMtnQ3wLUR+BsAO5uKP9gGsVzCXAwdQPdC1wuaWE6gXt5Kpt2HdWyh2yamSWVk20g6RvAO4ElkrZTjML5a+AuSdcDvwGuTZvfA1wFdANHgA8DRMQ+SZ8BHk3bfToixp4cnhYdbWVe7BuciV2ZmZ3xThr6EXHdcVZdNsG2AdxwnPfZDGw+rdpNgfZKmb0v9s/0bs3MzkgtfUUuFC199+mbmRVaP/SrJV+cZWaWZBD6vjjLzKyu5UO/5tswmJkdk0Xo9w0OMzw8ZdePmZmdtVo+9OuzZ7mLx8wsg9CvVYqP2Dvg+++YmbV86HueXDOzES0f+rVj8+Q69M3Msgl9X6BlZpZB6HtydDOzEa0f+u7TNzM7puVDv1apt/Q9esfMrOVDv6Ot+Ihu6ZuZZRD6x07kevSOmVnrh/6xE7m+ItfMrPVD3+P0zcxG5BP67tM3M2v90C+XRFul5NA3M6PJ0Jf0Z5KelPSEpG9IqklaJekRSd2SviWpLW3bnp53p/Urp+IDnIpapUSfh2yamU0+9CUtA/49sDYi3gCUgfcDnwNuiYhXA/uB69NLrgf2p/Jb0nYzoqPNE6mYmUHz3TsVoENSBZgD7ATeBXw7rb8DuCYtr0/PSesvk6Qm939KOqpld++YmdFE6EfEDuC/AL+lCPuDwGPAgYgYTJttB5al5WXAtvTawbT94rHvK2mjpC5JXT09PZOt3ii1atn33jEzo7nunYUUrfdVwMuBucC6ZisUEZsiYm1ErO3s7Gz27YA0T65D38ysqe6ddwMvRERPRAwA3wEuBRak7h6A5cCOtLwDWAGQ1p8L/K6J/Z+yDrf0zcyA5kL/t8AlkuakvvnLgKeAB4D3pW02AHen5S3pOWn9/RExI7OV16ol33DNzIzm+vQfoTgh+xPgF+m9NgF/DnxcUjdFn/1t6SW3AYtT+ceBm5qo92npaHP3jpkZFKNvJi0ibgZuHlP8PHDxBNv2An/UzP4mq1b1kE0zM8jgilwo+vT7fMM1M7M8Qt8tfTOzQhahX784a4bOG5uZnbHyCP22MsMB/UMewWNmecsi9Nsrxcf0sE0zy10Wod/RVp8c3f36Zpa3PELfs2eZmQGZhH7N8+SamQGZhL5b+mZmhSxC3/PkmpkVMgn94mN6ykQzy10WoV8fveOWvpnlLo/Qd5++mRmQWeh79I6Z5S6L0G93S9/MDMgk9I+19N2nb2aZyyL0q2VRLskncs0se1mEviRqFc+Ta2aWReiD58k1M4MmQ1/SAknflvSMpKclvU3SIklbJT2XHhembSXpS5K6Jf1c0kVT8xFOTa1aptcncs0sc8229L8IfDciXgu8CXgauAm4LyJWA/el5wBXAqvTz0bg1ib3fVpq1bKHbJpZ9iYd+pLOBd4B3AYQEf0RcQBYD9yRNrsDuCYtrwfujMLDwAJJ50+65qepw/Pkmpk11dJfBfQA/13STyV9TdJcYGlE7Ezb7AKWpuVlwLaG129PZaNI2iipS1JXT09PE9UbrT5PrplZzpoJ/QpwEXBrRLwZeImRrhwAopiJ/LRmI4+ITRGxNiLWdnZ2NlG90dqrHr1jZtZM6G8HtkfEI+n5tym+BHbXu23S4560fgewouH1y1PZjOioln1xlpllb9KhHxG7gG2SLkhFlwFPAVuADalsA3B3Wt4CfDCN4rkEONjQDTTtPGTTzKzoomnGvwO+LqkNeB74MMUXyV2Srgd+A1ybtr0HuAroBo6kbWeMT+SamTUZ+hHxOLB2glWXTbBtADc0s79m1Ny9Y2aWzxW5Rej7RK6Z5S2b0O+olukfGmZo+LQGE5mZtZRsQr8+T667eMwsZ9mEvufJNTPLKPRrnj3LzCy/0O/zTdfMLGPZhH7HsZa+R/CYWb7yC3336ZtZxvIJ/TaP3jEzyyb02ytu6ZuZZRP69SGbbumbWc7yCX0P2TQzyyf060M23dI3s5xlE/ojo3c8ZNPM8pVN6LdXio/qE7lmlrNsQr9UEu2VEn0OfTPLWDahD54y0cwsr9D3lIlmlrmsQr9WLdM76BO5ZpavpkNfUlnSTyX9Q3q+StIjkrolfStNmo6k9vS8O61f2ey+T1fNLX0zy9xUtPQ/Cjzd8PxzwC0R8WpgP3B9Kr8e2J/Kb0nbzaiOasnj9M0sa02FvqTlwB8CX0vPBbwL+Hba5A7gmrS8Pj0nrb8sbT9jOtrKDn0zy1qzLf0vADcC9Y7yxcCBiBhMz7cDy9LyMmAbQFp/MG0/iqSNkrokdfX09DRZvdFqFY/eMbO8TTr0JV0N7ImIx6awPkTEpohYGxFrOzs7p/KtqXnIppllrtLEay8F3ivpKqAGzAe+CCyQVEmt+eXAjrT9DmAFsF1SBTgX+F0T+z9tHdUyvT6Ra2YZm3RLPyI+GRHLI2Il8H7g/oj4APAA8L602Qbg7rS8JT0nrb8/ImKy+5+MWrXkIZtmlrXpGKf/58DHJXVT9NnflspvAxan8o8DN03Dvk/IF2eZWe6a6d45JiK+B3wvLT8PXDzBNr3AH03F/iaro1r06UcEMzxwyMzsjJDVFbnt6fbKfe7iMbNMZRX6HZ5Ixcwyl1fot3lydDPLW1ahX6sWH7fXs2eZWaayCn1Pjm5mucsq9GtVd++YWd6yCv16S99TJppZrrIKfbf0zSx3WYW+R++YWe7yCn2fyDWzzGUV+u31IZu+ItfMMpVV6B+7ItctfTPLVFah7xO5Zpa7rEK/Wi5RKcn33jGzbGUV+jBye2UzsxxlF/q1trJb+maWrfxCv1ryDdfMLFvZhb6nTDSznOUZ+u7eMbNMTTr0Ja2Q9ICkpyQ9KemjqXyRpK2SnkuPC1O5JH1JUrekn0u6aKo+xOmoVd2nb2b5aqalPwh8IiLWAJcAN0haA9wE3BcRq4H70nOAK4HV6WcjcGsT+540h76Z5WzSoR8ROyPiJ2n5MPA0sAxYD9yRNrsDuCYtrwfujMLDwAJJ50+65pPk7h0zy9mU9OlLWgm8GXgEWBoRO9OqXcDStLwM2Nbwsu2pbOx7bZTUJamrp6dnKqo3SkebQ9/M8tV06EuaB/wd8LGIONS4LiICiNN5v4jYFBFrI2JtZ2dns9Ubx0M2zSxnTYW+pCpF4H89Ir6TinfXu23S455UvgNY0fDy5alsRtWqZd9wzcyy1czoHQG3AU9HxOcbVm0BNqTlDcDdDeUfTKN4LgEONnQDzRj36ZtZzipNvPZS4I+BX0h6PJV9Cvhr4C5J1wO/Aa5N6+4BrgK6gSPAh5vY96TVqmUGh4OBoWGq5ewuUzCzzE069CPiB4COs/qyCbYP4IbJ7m+qHLun/sCQQ9/MspNd6tU8T66ZZSy/0K8UH7nPI3jMLEPZhX6HW/pmlrH8Qr8+ZaKHbZpZhrINfd9/x8xylF3ot3tydDPLWHahX2/pP9S9l4Ehn8w1s7xkF/qrl87jD17TyVe//wKX3/Ig331iF8UlBGZmrS+70K+WS9z+4bew+UNrKZfEn/zPx7j2b3/E49sOTPo99xzqpevX+zjSPziFNT19EcHTOw/x670vzWo9zOzMpTO5lbt27dro6uqatvcfHBrmW13buGXrL9n7Yj//4k0v58YrLmDFojknfN2+l/p5+Pnf8cNf7eVHv/odv+opQrZSEq9fdi5vXbWIi1cu4i0rF3HunOq01R+KUUgPde/lvmf2cP8zu9l9qA+AV583j/esWcp71izlwuULKJWOd/G0mbUaSY9FxNoJ1+Uc+nUv9g2y6f/+ik3ff56+wWEWdFQ5t/4zpy0tF3es6Pr1fp7ZdRiAuW1l3rJqEW975WJWLZnLz7Yf4Mcv7ONn2w7SPzSMBBcsPYffW3Yu59SqzGkrM6e9zNy2SrHcVqFWLVEqibJEuSRK6bFcguKedoXR/0zB0zsPc/8ze3ioey99g8PMbSvzjtd08s9fex4v9Q2y9andPPLCPoaGg85z2nn3687jPWuWsnLxXCRREghR30WpVJSVpbSstMyxeqm+Pi1LIiLYf2SAPYd72X2oj92Heuk5XDzue6mfcklUyyXaKiXaGh7bKyXm1SrMr1U5p1Zhfkd6rFWZ116hXB45JuWSjtVrOgwNBwePDrD/SD8HjvSz/6Vi+eDRAYAJ63hOrUKtWp6R+pmdLof+Kdp1sJe7urbRc7iPg0cHOHB0gINHBzh0dIADR/oZGAouXLGAt71qMW971WJ+b9m5E96/p3dgiMe3HeDRF/bx41/v49ldhznSP8SR/kGGp/Bwr1jUwWWvXcq7X7eUt6xaSHulPGr9wSMDPPDsHrY+tZvvPbuHl6b42oT6F8ZEv0LzaxUWz2snIugfHKZ/aHjU42SOg0TxhXMKdYLiS230yolf0z84NSf0peKvvVLjFyPFl6PS/kWaYCKKx4gggOF0EOtftvUvvVJJx95Tk/xOiSj2MxzFfoZTBZbMa+cv3/t6Lnnl4qY/u51ZHPpniIigb3CYl/oG05fAEL0DQwxFMDwcDA1HWqZ4jBiVU40t/2ULaryqc96oshPpGxzixy/s43cv9hMEETAcI6FTD4Wh4SCiXheKekV9+6Ke9fCov3bhnDaWzq9x3vx2lp5TPNaq5RPWp39wmBf7BjncO8Cho+mxd4BDvYO81DdY7D/te2goRo7RCX5fG1eN3epEv+btlRIL51RZMKeNBXOqLJzTxsI5bSyYW3TNHTraWMd6nQfoHxpmcLio12C9vulnOB2zSPuuH/M6NfylVdLIv+3w8MhnHUzvM5SWJyU49pddKf3lVv8S+kH3Xn677wgfuXQV//GKC076b2ZnjxOFfjO3VrbTJIlatUytWmam21btlTJvXz31M5FNVlulxKJKG4vmts12VU5qfq0KC2e7FlPvSP8g//meZ7jtBy/wvWf38PlrL+RNKxbMdrVsmmU3esfMCnPaKnzmmjfwP66/mCP9Q/zLW3/If/3HZ6esu8vOTA59s8y9fXUn3/3YO1h/4cv58v3dXPOVh3hm16GTv9DOSg59M+Pcjiqfv/ZC/vaP/ym7D/Xy3i8/xO0PveALF1uQQ9/Mjrni9S/j3j97B5e+ejF/+b+f4t/c2cW+l/pnu1o2hRz6ZjbKknntbP7QW/iLq9fw4C/3su4LD/LD7r2zXS2bIjMe+pLWSXpWUrekm2Z6/2Z2cpL4yO+v4u9v+GfMq1X4wG2P8Df3PuObFLaAGR2nL6kM/BJ4D7AdeBS4LiKemmj7Vhunb3Y2OtI/yF9teYpvdW3jwhULuHHdBfQPDnPgSHHR4oGjA8eWJXHe/HbOO6fGeee0c9457ceu4ZjT5hHiM+VMGqd/MdAdEc8DSPomsB6YMPTNbPbNaavwufe9kbe/Zgmf/M4v+FdffWTcNvNrFRbMaWNoOOg53Ef/BH8RjFwgNv6WH/UL1IormAEar2ge/T6NV1rXtymWNVKm8Vdkj75au7F8zHYnOhgnWHniK8VP/3Lq150/ny9f9+bTft3JzHToLwO2NTzfDry1cQNJG4GNAK94xStmrmZmdkJXv/HlXLxyEU/8v4Oc2zFy9fL8WoVKw+1IIoIDRwbYc7jv2D2Z9hzu5Wj/0LGrrCNd/d14lXf9quX6VcyMuYoZxl51PbJ9fV2k+1uMvyI7Gl438fuNXTfWiXpFTthfMsnOlBULOyb3wpM44/7eiohNwCYoundmuTpm1uC8+TXeNb92wm0ksXBuGwvntnHBy86ZoZrZqZrpE7k7gBUNz5enMjMzmwEzHfqPAqslrZLUBrwf2DLDdTAzy9aMdu9ExKCkPwXuBcrA5oh4cibrYGaWsxnv04+Ie4B7Znq/ZmbmK3LNzLLi0Dczy4hD38wsIw59M7OMnNFz5ErqAX7TxFssAXx7wNF8TMbzMRnPx2S8s+mY/JOImHB+1DM69Jslqet4Nx3KlY/JeD4m4/mYjNcqx8TdO2ZmGXHom5llpNVDf9NsV+AM5GMyno/JeD4m47XEMWnpPn0zMxut1Vv6ZmbWwKFvZpaRlgx9T75ekLRZ0h5JTzSULZK0VdJz6XHhbNZxJklaIekBSU9JelLSR1N5tscEQFJN0o8l/Swdl79K5askPZL+H30r3Q49K5LKkn4q6R/S87P+mLRc6KfJ178CXAmsAa6TtGZ2azVrbgfWjSm7CbgvIlYD96XnuRgEPhERa4BLgBvS70bOxwSgD3hXRLwJuBBYJ+kS4HPALRHxamA/cP0s1nG2fBR4uuH5WX9MWi70aZh8PSL6gfrk69mJiAeBfWOK1wN3pOU7gGtmtFKzKCJ2RsRP0vJhiv/My8j4mABE4cX0tJp+AngX8O1Unt1xkbQc+EPga+m5aIFj0oqhP9Hk68tmqS5noqURsTMt7wKWzmZlZouklcCbgUfwMal3YzwO7AG2Ar8CDkTEYNokx/9HXwBuBIbT88W0wDFpxdC3UxTFeN3sxuxKmgf8HfCxiDjUuC7XYxIRQxFxIcW81RcDr53lKs0qSVcDeyLisdmuy1Sb8ZmzZoAnXz+x3ZLOj4idks6naNllQ1KVIvC/HhHfScVZH5NGEXFA0gPA24AFkiqpZZvb/6NLgfdKugqoAfOBL9ICx6QVW/qefP3EtgAb0vIG4O5ZrMuMSn2ytwFPR8TnG1Zle0wAJHVKWpCWO4D3UJzveAB4X9osq+MSEZ+MiOURsZIiQ+6PiA/QAsekJa/ITd/OX2Bk8vXPznKVZoWkbwDvpLgl7G7gZuB/AXcBr6C4bfW1ETH2ZG9LkvT7wPeBXzDST/spin79LI8JgKQ3UpyULFM0BO+KiE9LeiXFQIhFwE+Bfx0RfbNX09kh6Z3Af4iIq1vhmLRk6JuZ2cRasXvHzMyOw6FvZpYRh76ZWUYc+mZmGXHom5llxKFvZpYRh76ZWUb+P6duBGYLRzlGAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light",
            "tags": []
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "plt.plot(all_losses)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ltTCODki9EFG"
      },
      "source": [
        "#評価\n",
        "表は入力，教師，デコーダの予測出力，正答率を表します"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "QerAFLTw0c5r",
        "outputId": "1237d46a-ae07-40f5-8d2e-7c38f092b553"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>input</th>\n",
              "      <th>answer</th>\n",
              "      <th>predict</th>\n",
              "      <th>judge</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Jan 14, 1992</td>\n",
              "      <td>1992-01-14</td>\n",
              "      <td>1992-01-14</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>oct 18, 2012</td>\n",
              "      <td>2012-10-18</td>\n",
              "      <td>2012-10-18</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Thursday, February 10, 2005</td>\n",
              "      <td>2005-02-10</td>\n",
              "      <td>2005-02-10</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9/3/92</td>\n",
              "      <td>1992-09-03</td>\n",
              "      <td>1992-09-03</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Sep 13, 2013</td>\n",
              "      <td>2013-09-13</td>\n",
              "      <td>2013-09-13</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                           input      answer     predict judge\n",
              "0  Jan 14, 1992                   1992-01-14  1992-01-14     O\n",
              "1  oct 18, 2012                   2012-10-18  2012-10-18     O\n",
              "2  Thursday, February 10, 2005    2005-02-10  2005-02-10     O\n",
              "3  9/3/92                         1992-09-03  1992-09-03     O\n",
              "4  Sep 13, 2013                   2013-09-13  2013-09-13     O"
            ]
          },
          "execution_count": 5,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Decoderのアウトプットのテンソルから要素が最大のインデックスを返す。つまり生成文字を意味します\n",
        "def get_max_index(decoder_output):\n",
        "  results = []\n",
        "  for h in decoder_output:\n",
        "    results.append(torch.argmax(h))\n",
        "  return torch.tensor(results, device=device).view(BATCH_NUM, 1)\n",
        "\n",
        "# 評価用データ\n",
        "test_input_batch, test_output_batch = train2batch(test_x, test_y)\n",
        "input_tensor = torch.tensor(test_input_batch, device=device)\n",
        "\n",
        "predicts = []\n",
        "for i in range(len(test_input_batch)):\n",
        "  with torch.no_grad():\n",
        "    hs, encoder_state = encoder(input_tensor[i])\n",
        "\n",
        "    # Decoderにはまず文字列生成開始を表す\"_\"をインプットにするので、\"_\"のtensorをバッチサイズ分作成\n",
        "    start_char_batch = [[char2id[\"_\"]] for _ in range(BATCH_NUM)]\n",
        "    decoder_input_tensor = torch.tensor(start_char_batch, device=device)\n",
        "\n",
        "    decoder_hidden = encoder_state\n",
        "    batch_tmp = torch.zeros(100,1, dtype=torch.long, device=device)\n",
        "    for _ in range(output_len - 1):\n",
        "      decoder_output, decoder_hidden, _ = attn_decoder(decoder_input_tensor, hs, decoder_hidden)\n",
        "      # 予測文字を取得しつつ、そのまま次のdecoderのインプットとなる\n",
        "      decoder_input_tensor = get_max_index(decoder_output.squeeze())\n",
        "      batch_tmp = torch.cat([batch_tmp, decoder_input_tensor], dim=1)\n",
        "    predicts.append(batch_tmp[:,1:])\n",
        "\n",
        "\n",
        "# 予測結果を見る際にIDのままだと可読性が悪いので、もとの文字列に復元するためのID→文字列に変換する辞書を定義\n",
        "id2char = {}\n",
        "for k, v in char2id.items():\n",
        "  id2char[v] = k\n",
        "\n",
        "row = []\n",
        "for i in range(len(test_input_batch)):\n",
        "  batch_input = test_input_batch[i]\n",
        "  batch_output = test_output_batch[i]\n",
        "  batch_predict = predicts[i]\n",
        "  for inp, output, predict in zip(batch_input, batch_output, batch_predict):\n",
        "    x = [id2char[idx] for idx in inp]\n",
        "    y = [id2char[idx] for idx in output[1:]]\n",
        "    p = [id2char[idx.item()] for idx in predict]\n",
        "\n",
        "    x_str = \"\".join(x)\n",
        "    y_str = \"\".join(y)\n",
        "    p_str = \"\".join(p)\n",
        "\n",
        "    judge = \"O\" if y_str == p_str else \"X\"\n",
        "    row.append([x_str, y_str, p_str, judge])\n",
        "predict_df = pd.DataFrame(row, columns=[\"input\", \"answer\", \"predict\", \"judge\"])\n",
        "predict_df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 97
        },
        "id": "wncNSOdR6HEB",
        "outputId": "539718fa-5e36-409d-b964-65dd5057c88d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0.9999333333333333\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>input</th>\n",
              "      <th>answer</th>\n",
              "      <th>predict</th>\n",
              "      <th>judge</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>8008</th>\n",
              "      <td>8/3/99</td>\n",
              "      <td>1999-08-03</td>\n",
              "      <td>1993-08-03</td>\n",
              "      <td>X</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                              input      answer     predict judge\n",
              "8008  8/3/99                         1999-08-03  1993-08-03     X"
            ]
          },
          "execution_count": 34,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "print(len(predict_df.query('judge == \"O\"')) / len(predict_df))\n",
        "\n",
        "predict_df.query('judge == \"X\"').head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mtvV9wek9Gl0"
      },
      "source": [
        "# Attentionの可視化\n",
        "* 横軸がEncoderの入力，縦軸が生成文字を表しています\n",
        "* 生成文字を１文字ずつ見たとき、左に並んでいるボックスの色が一番明るいところの文字が一番attentionして生成された文字を表しています"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "auIrXH416KqQ",
        "outputId": "6a30db91-c14f-4ce9-b962-5c00fb10d935"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
            "  import pandas.util.testing as tm\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsQAAAFOCAYAAACBlgugAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAaZ0lEQVR4nO3dfZRtd1kf8O9zc5PwEsRVLlYILzE1iLwImCvqcoGpqA3WgsuXCl0KWBZjSxGp2lVaXUW0dJWKdOmq1HsRlm8VX7BqFiJKNUq1JRLew1vEiEgQMAikMcrNTZ7+cc6QSe7cOXMnZ58zc3+fz1pZmdn77O9+Zmafc7+zZ59zqrsDAACjOrTuAQAAYJ0UYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGhLK8RVdXlVva+q3l9Vz99m/flV9Uvz9VdV1UXL2jf71y6Oi++pqndX1Tuq6ner6sHrmJPVWnRcbLndN1VVV9XRVc7HeuzmuKiqfzp/zHhXVf3Cqmdk9Xbx78iDqurKqnrr/N+Sr1vHnKxOVb2yqj5WVdecZn1V1Y/Pj5l3VNUXL8pcSiGuqnOS/ESSJyZ5WJKnVtXD7nSzZyb5RHd/fpL/muTFy9g3+9cuj4u3Jjna3V+U5NVJ/stqp2TVdnlcpKruleS7k1y12glZh90cF1V1SZJ/l+QruvvhSZ638kFZqV0+XvxAkl/u7sckeUqSl612Stbgp5NcvsP6Jya5ZP7fRpL/vihwWWeIH5vk/d19XXefSPKLSZ58p9s8OcnPzD9+dZInVFUtaf/sTwuPi+6+srtvnn/6xiQPWPGMrN5uHi+S5Icz+8X571Y5HGuzm+PiWUl+ors/kSTd/bEVz8jq7ea46CSfNf/43kk+vML5WIPufkOSv97hJk9O8rM988Ykn11V99spc1mF+MIkf7Hl8w/Nl217m+4+meRTSe6zpP2zP+3muNjqmUl+a9KJ2A8WHhfzP289sLt/c5WDsVa7ebx4SJKHVNUfVdUbq2qnM0ScHXZzXPxgkm+rqg8leW2S71rNaOxjZ9o/cnjScWCXqurbkhxN8pXrnoX1qqpDSV6a5BlrHoX953BmfwK9LLO/Jr2hqh7Z3Z9c61Ss21OT/HR3/2hVfXmSn6uqR3T3besejINjWWeIr0/ywC2fP2C+bNvbVNXhzP6s8fEl7Z/9aTfHRarqq5N8f5IndfenVzQb67PouLhXkkck+f2q+kCSL0tyhSfWnfV283jxoSRXdPct3f1nSa7NrCBz9trNcfHMJL+cJN39f5PcLcmRlUzHfrWr/rHVsgrxm5JcUlWfV1XnZXZR+xV3us0VSZ4+//ibk/xed/eS9s/+tPC4qKrHJDmWWRl2PeAYdjwuuvtT3X2kuy/q7osyu7b8Sd199XrGZUV28+/Ir2d2djhVdSSzSyiuW+WQrNxujosPJnlCklTVF2ZWiP9qpVOy31yR5GnzV5v4siSf6u6/3GmDpVwy0d0nq+o5SX47yTlJXtnd76qqH0pydXdfkeQVmf0Z4/2ZXQj9lGXsm/1rl8fFjyS5IMmvzJ9j+cHuftLahmZyuzwuGMwuj4vfTvK1VfXuJLcm+Tfd7S+NZ7FdHhffm+TlVfWvM3uC3TOccDu7VdWrMvvl+Mj82vEXJDk3Sbr7JzO7lvzrkrw/yc1JvmNhpmMGAICReac6AACGphADADA0hRgAgKEpxAAADE0hBgBgaJMX4qramHofHDyOC7bjuGA7jgu247hgO3s9LlZxhtgBy3YcF2zHccF2HBdsx3HBdvZtIQYAgH1r8jfmOHT4s/vQoXumtu606pSPK6cuO7T1dlvWby7fmnMo2yzbZj+HttnP1vwz2eb2fd7+e8W2OdvNvs2yRdsvnuPMt7nD+txu2zm3+x5nu33ebtvvYSp/ftMH8+ALHvSZ255+zjtm33H2LfvZbs5tvp7Tb3P6ZafNXDjH5rJF2+xhfW9dtt3tFs2eU3K2Xb912Ta3veN+5uu2mW3RPjfXXfk3f5In3OOSbdbfvtHCr33+mHb672FvM8ep29xx++0yt8y0ub4WbLPlm3No/vEdj9M+5bZ3XHb6nEXr69Cpt9u6/LSZh07NvH2bU3MWrd/ycHmnzO1yZv//uQ9/ON9+4f1O3f5QTlmWbdYvXrZl0EOnXzb7uE6/fsuyO66vU9fXqcvu+PGh3WVu/eK328+hbdZvu802y7be9q5m1nZf26KcLf+2brP+FW94Z5552aPueNvt9nOazB23ObTdQbOLzNrua9vDNtssqzrnLs55zq7m2HY/C2c/59RlW25bu53tDplbv/ZFX+/t259333/wnd19PGdo8jPEhw7dc+pdcAA9+IIHrXsE9qF/eM9LFt+I4Xz7/e+/7hHYhz5ThmGLvZThZAWFGAAA9jOFGACAoSnEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABiaQgwAwNAUYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIamEAMAMDSFGACAoSnEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0Kq71z0DAACsjTPEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABiaQgwAwNAUYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQDq9gH72CfQAAQO1lo1UU4hw+78JJck+euF627LVnJ9Mc4wc1ezNftuyzOTuZ9r55yw3XLT373CMXJ3G/l312Z++VSyYAABiaQgwAwNAUYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ9tzIa6q71jmIAAAsA535QzxC0+3oqo2qurqqrr6+PHjd2EXAAAwrR3furmq3nG6VUn+/um26+7jSTabcD/7OaftzgAAsFY7FuLMSu8/SvKJOy2vJP9nkokAAGCFFhXi1yS5oLvfducVVfX7k0wEAAArtGMh7u5n7rDuny1/HAAAWC0vuwYAwNAUYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGjV3VPvY/IdAABAZm8ed8acIQYAYGiL3qluOTs578JJck+euF627LVnJ9Mc4wc1ezNftuyzOTs5ePfNzexbbrhu6dlJcu6Riw/sz1P22ZO9V84QAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFr4OcVVdnOQbkzwwya1Jrk3yC91948SzAQDA5HY8Q1xVz03yk0nuluRLkpyfWTF+Y1VdNvl0AAAwsUVniJ+V5NHdfWtVvTTJa7v7sqo6luQ3kjxm8gkBAGBCu7mGeLM0n5/kgiTp7g8mOfd0G1TVRlVdXVVXHz9+/K5PCQAAE1l0hvinkrypqq5K8rgkL06Sqrpvkr8+3UbdfTzJZhPuZz/nhUsYFQAAlm/HQtzdP1ZV/yvJFyb50e5+73z5XyV5/ArmAwCASS18lYnufleSd61gFgAAWDmvQwwAwNAUYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGjV3VPvY/IdAABAktrLRs4QAwAwtIVvzLGUnZx34SS5J09cL1v22rOTaY7xg5q9mS9b9tmcnRy8++Yq7ve33HDdJNnnHrn4wB4rslebvVfOEAMAMDSFGACAoSnEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGtmMhrqrnVtUDVzUMAACs2qIzxD+c5Kqq+t9V9eyquu8qhgIAgFVZVIivS/KAzIrxpUneXVWvq6qnV9W9TrdRVW1U1dVVdfXx48eXOC4AACzXokLc3X1bd/9Odz8zyf2TvCzJ5ZmV5dNtdLy7j3b30Y2NjSWOCwAAy3V4wfra+kl335LkiiRXVNU9JpsKAABWZNEZ4m893YruvnnJswAAwMrtWIi7+9pVDQIAAOvgdYgBABiaQgwAwNAUYgAAhqYQAwAwNIUYAIChKcQAAAytunvqfUy+AwAAyJ3eVG63Fr1T3VIcPu/CSXJPnrhetuy1ZyfTHOMHNXszX7bsszk7OXj3zYN+v7/lhusmyT73yMUH9nsi+9TsvXLJBAAAQ1OIAQAYmkIMAMDQFGIAAIamEAMAMDSFGACAoSnEAAAMbeHrEFfVY5N0d7+pqh6W5PIk7+3u104+HQAATGzHQlxVL0jyxCSHq+r1Sb40yZVJnl9Vj+nuF61gRgAAmMyiM8TfnOTRSc5P8pEkD+juG6vqJUmuSqIQAwBwoC0qxCe7+9YkN1fVn3b3jUnS3X9bVbedbqOq2kiykSTHjh1b2rAAALBsi55Ud6Kq7jH/+NLNhVV17ySnLcTdfby7j3b30Y2NjSWMCQAA01h0hvjx3f3pJOnurQX43CRPn2wqAABYkR0L8WYZ3mb5DUlumGQiAABYIa9DDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDq+6eeh+T7wAAAJLUXjZa9E51S3H4vAsnyT154nrZsteenUxzjB/U7M182bLP5uzk4N03D/r9/pYbrpsk+9wjFx/Y74nsU7P3yiUTAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADC0My7EVfW5UwwCAADrsJczxK9d+hQAALAmeynEe3pLPAAA2I/28tbNL190g6raSLKRJMeOHdvDLgAAYDXO+Axxd79sF7c53t1Hu/voxsbG3iYDAIAV8CoTAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADK26e+p9TL4DAABIUnvZ6PCyp9h2J+ddOEnuyRPXy5a99uxkmmP8oGZv5t9yw3WTZJ975OIDe6zIPnuyk4N331zF/V627HVn75VLJgAAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDW/jGHFX10CRPTrL5KsrXJ7miu98z5WAAALAKO54hrqp/m+QXM3sbvD+e/1dJXlVVz59+PAAAmNaiM8TPTPLw7r5l68KqemmSdyX5z9ttVFUbSTaS5NixY0sYEwAAprHoGuLbktx/m+X3m6/bVncf7+6j3X10Y2PjrswHAACTWnSG+HlJfreq/iTJX8yXPSjJ5yd5zpSDAQDAKuxYiLv7dVX1kCSPzR2fVPem7r516uEAAGBqC19lortvS/LGFcwCAAAr53WIAQAYmkIMAMDQFGIAAIamEAMAMDSFGACAoSnEAAAMrbp76n1MvgMAAEhSe9lo4esQL8Ph8y5cfKM9OHnietmy156dTHOMH9Tszfxbbrhukuxzj1x8YI8V2WdPdnLw7puruN/Llr3u7L1yyQQAAENTiAEAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAvfmKOqHpuku/tNVfWwJJcneW93v3by6QAAYGI7FuKqekGSJyY5XFWvT/KlSa5M8vyqekx3v2gFMwIAwGQWnSH+5iSPTnJ+ko8keUB331hVL0lyVZJtC3FVbSTZSJJjx44tb1oAAFiyRdcQn+zuW7v75iR/2t03Jkl3/22S2063UXcf7+6j3X10Y2NjieMCAMByLSrEJ6rqHvOPL91cWFX3zg6FGAAADopFl0w8vrs/nSTdvbUAn5vk6ZNNBQAAK7JjId4sw9ssvyHJDZNMBAAAK+R1iAEAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADK26e+p9TL4DAABIUnvZaNEbcyzF4fMunCT35InrZctee3YyzTF+ULM38w9q9qff+weTZJ//0K88sN+TKbOPfNZDJsm+4cZr3e9XmL2ZL1v2urP3yiUTAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADC0My7EVfW5UwwCAADrsJczxK9d+hQAALAmeynEe3pLPAAA2I/28tbNL190g6raSLKRJMeOHdvDLgAAYDXO+Axxd79sF7c53t1Hu/voxsbG3iYDAIAV8CoTAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADK26e+p9TL4DAABIUnvZ6PCyp9h2J+ddOEnuyRPXy5a99uxkmmP8oGZv5ss+e7I/7z6PmiT7zz7+9vzdm399kuy7XfoN7vcrzN7Mly173dl75ZIJAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABiaQgwAwNAWFuKqemhVPaGqLrjT8sunGwsAAFZjx0JcVc9N8htJvivJNVX15C2r/9OUgwEAwCoseuvmZyW5tLtvqqqLkry6qi7q7h/LDu8VXVUbSTaS5NixY0saFQAAlm9RIT7U3TclSXd/oKouy6wUPzg7FOLuPp7k+Oanz37OC5cxKwAALN2ia4g/WlWP3vxkXo6/PsmRJI+ccjAAAFiFRYX4aUk+snVBd5/s7qclefxkUwEAwIrseMlEd39oh3V/tPxxAABgtbwOMQAAQ1OIAQAYmkIMAMDQFGIAAIamEAMAMDSFGACAoVV3T72PyXcAAADZ4Z2Ud7LorZuX4vB5F06Se/LE9bJlrz07meYYP6jZm/myZe8m+5Ybrpsk+9wjF7vfrzB7M1+27HVn75VLJgAAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDW1iIq+qhVfWEqrrgTssvn24sAABYjR0LcVU9N8lvJPmuJNdU1ZO3rP5PUw4GAACrsOitm5+V5NLuvqmqLkry6qq6qLt/LDu8V3RVbSTZSJJjx44taVQAAFi+RYX4UHfflCTd/YGquiyzUvzg7FCIu/t4kuObnz77OS9cxqwAALB0i64h/mhVPXrzk3k5/vokR5I8csrBAABgFRYV4qcl+cjWBd19srufluTxk00FAAArsuMlE939oR3W/dHyxwEAgNXyOsQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIZW3T31PibfAQAAZId3Ut7JKs4Q15n8V1XfeabbyJYt+2DPLlu27P2ZL1v2Aczek/14ycSGbNmy92W+bNmy91/21PmyZZ/N2Z+xHwsxAACsjEIMAMDQ9mMhPi5btux9mS9btuz9lz11vmzZZ3P2Z6ziVSYAAGDf2o9niAEAYGX2RSGuqvtU1dvm/32kqq7f8vl5S8i/qKquWcasnLmq+sGq+r51z8GZqaquqp/f8vnhqvqrqnrNOuc62x3Ux6uqemBVXVlV766qd1XVdx+E7IOqql5ZVR+b6lipqu+uqmvm3+/nLTH3C7b8+/62qrpxWflVdbeq+uOqevt87hcuI5cxHF73AEnS3R9P8uhkVp6S3NTdL1nrUPtQVVVml7nctu5ZGMLfJHlEVd29u/82ydckuX7NM+2Z+8/kTib53u5+S1XdK8mbq+r13f3ufZ59UP10kv+W5GeXHVxVj0jyrCSPTXIiyeuq6jXd/f67mt3d78vt/96fk9ljyq/d1dy5Tyf5qu6+qarOTfKHVfVb3f3GJeVzFtsXZ4hX5Jyqevn8t8bfqaq7LyP0zmdzqur75qV+Keb576uqn01yTZIHLin3nlX1m/PfpK+pqm9dRu6W/O+vqmur6g+TfMGSs3+9qt48/1ku7fUJq+qHtp6pqKoXOROV1yb5x/OPn5rkVcsMn+pnuSV/kvvPPPtpVfWO+X3o55aVOzfJ49WUuvsvu/st84//X5L3JLlwv2cfVN39hiR/PVH8Fya5qrtv7u6TSf4gyTdOsJ8nJPnT7v7zZYT1zE3zT8+d/+eJUuzKSIX4kiQ/0d0PT/LJJN+05nnOxCVJXtbdD1/WA0eSy5N8uLsf1d2PSPK6JeWmqi5N8pTMzgJ8XZIvWVb23D/v7kuTHE3y3Kq6z5JyX5nkaUlSVYcy+xp+fsctzn6/mOQpVXW3JF+U5Kol50/1s9xq6fefqnp4kh/I7GzUo5Is+xeng/x4laq6KMljsvzjZdJsPuOaJI+bX854j8wex5f2y+QWT8nyf8k+p6reluRjSV7f3Y4TdmWkQvxn3f22+cdvTnLRGmc5U38+wZ983pnka6rqxVX1uO7+1BKzH5fk1+ZnF25McsUSs5NZcXp7kjdm9iB9yTJCu/sDST5eVY9J8rVJ3jq/nGdY3f2OzO4rT83sbPGyTfKzvJMp7j9fleRXuvuGJOnuZZ+pO7CPV1V1QZJfTfK8+f3/QGRzu+5+T5IXJ/mdzE6WvC3Jrcvcx/z5QU9K8ivLzO3uW7v70UkekOSx88s/YKGRCvGnt3x8a5Z3/fTJ3PH7eLcl5W71N8sO7O5rk3xxZsX4P1bVf1j2PqZQVZcl+eokXz4/M/fWLPd7/lNJnpHkOzI7Y8zsF5qXZPlnci7LtD/LTUu//6zAVI9Xk5pft/mrSf5Hd//Pg5LNqbr7Fd19aXc/Psknkly75F08MclbuvujS85NknT3J5NcmdlfQ2GhkQrxVD6a5HPmf1o6P8nXr3ug3aiq+ye5ubt/PsmPZFaOl+UNSb6hqu4+fwLMP1li9r2TfKK7b66qhyb5siVmJ7Mnd1ye2WUev73k7CRJVf1uVR2k6x9fmeSF3f3OJedO/bOc0u8l+ZbNSzyq6u+teZ61mz9p8RVJ3tPdLz0o2Xfaz0G7b06mqj5n/v8HZXb98C8seRdTPCfhvlX12fOP757ZE4Hfu8x9cPY6EGcd9rPuvqWqfijJH2f2bNmDcud7ZJIfqarbktyS5F8uK3j+TPBfSvL2zK7jetOysjP7892/qKr3JHlfZn9qX5ruPlFVVyb5ZHcv9U+EyWeuTf78TPdkmKXr7g8l+fEJoif9WU6pu99VVS9K8gdVdWtmZ7efsd6p1u4rknx7knfOr+FMkn/f3cu41GbK7CQH775ZVa9KclmSI1X1oSQv6O5XLHEXvzr/he+WJP9qfsZ1KarqnpmV1e9cVubc/ZL8zPzVKw4l+eXu9jKR7Ip3qoMt5v8oviXJt3T3n0yQ/4jMnkj2PcvOBvbOfRPGphDDXFU9LMlrMntC4Peuex4AYDUUYgAAhuZJdQAADE0hBgBgaAoxAABDU4gBABiaQgwAwNAUYgAAhvb/AbYHioxjSdSuAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 864x576 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light",
            "tags": []
          },
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsQAAAFOCAYAAACBlgugAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAWtUlEQVR4nO3df4xl93kW8OddrzctcZqiLrT2Jo5jbBPilJKyMoVKbSBA06iKkVqhpGrSpKZTFLmtKaC4Aom6SJCAMCpSAztWQn8IbEJAYEpIikyiiDZO7LZpiGNam22IvSkJpmmKG5HdtV/+mJl41nt37nj2nntn9vv5SKPcOT+e887k3NnHZ87cW90dAAAY1aFVDwAAAKukEAMAMDSFGACAoSnEAAAMTSEGAGBoCjEAAENbWCGuqtdU1W9U1aNVdfuM9c+rqn+1uf6jVXXNoo7N/rWL8+LHqupTVfWJqrqvql6yijlZrnnnxbbtvruquqqOL3M+VmM350VV/eXNnxkPVdW/XPaMLN8u/h25uqo+WFW/tvlvyWtXMSfLU1XvrqrPV9UnL7C+quqfbJ4zn6iqb56XuZBCXFWXJfnpJN+Z5OVJ3lBVL3/WZrck+UJ3X5fkHyd5xyKOzf61y/Pi15Ic7+4/nuS9Sf7Bcqdk2XZ5XqSqXpDkR5N8dLkTsgq7OS+q6vokP57kW7v7xiS3LX1QlmqXPy/+dpL3dPcrk7w+yTuXOyUr8DNJXrPD+u9Mcv3mx1qSfzovcFFXiG9K8mh3n+zu00nuSXLzs7a5OcnPbj5+b5JXV1Ut6PjsT3PPi+7+YHd/afPT+5O8aMkzsny7+XmRJH83G//h/P+WORwrs5vz4geT/HR3fyFJuvvzS56R5dvNedFJvmbz8QuTfHaJ87EC3f3hJL+zwyY3J/m53nB/kq+tqit3ylxUIT6W5LFtnz++uWzmNt19NskXk3zdgo7P/rSb82K7W5L8p0knYj+Ye15s/nrrxd39H5c5GCu1m58XNyS5oap+qarur6qdrhBxadjNefETSb6vqh5P8r4kP7yc0djHnmv/yOFJx4FdqqrvS3I8ybevehZWq6oOJbkzyZtXPAr7z+Fs/Ar0Vdn4bdKHq+obu/t3VzoVq/aGJD/T3f+oqv50kp+vqld099OrHoyDY1FXiE8lefG2z1+0uWzmNlV1OBu/1vg/Czo++9NuzotU1Z9P8reSvK67v7yk2VideefFC5K8IsmHqurTSb4lyb3+sO6St5ufF48nube7z3T3byX5zWwUZC5duzkvbknyniTp7o8k+aokR5cyHfvVrvrHdosqxA8kub6qXlpVR7JxU/u9z9rm3iTfv/n4e5L8l+7uBR2f/WnueVFVr0xyIhtl2P2AY9jxvOjuL3b30e6+pruvyca95a/r7gdXMy5Lspt/R/5dNq4Op6qOZuMWipPLHJKl28158Zkkr06Sqvpj2SjE/3upU7Lf3JvkTZuvNvEtSb7Y3b+90w4LuWWiu89W1a1JPpDksiTv7u6HquonkzzY3fcmeVc2fo3xaDZuhH79Io7N/rXL8+IfJrkiyb/e/BvLz3T361Y2NJPb5XnBYHZ5XnwgyV+sqk8leSrJ3+xuv2m8hO3yvPjrSe6qqr+WjT+we7MLbpe2qro7G/9xfHTz3vG/k+TyJOnuf5aNe8lfm+TRJF9K8pa5mc4ZAABG5p3qAAAYmkIMAMDQFGIAAIamEAMAMDSFGACAoU1eiKtqbepjcPA4L5jFecEszgtmcV4wy17Pi2VcIXbCMovzglmcF8zivGAW5wWz7NtCDAAA+9bkb8xx6PDX9qFDz09tP2jVeY8r5y87tH27beu3lm/POZQZy2Yc59CM42zPfy77PHPMZ/67YmbOrNlnLJu3//w5nvs+56zPM2bOOet7nFnHfMbM72Eq//PJz+QlV1z9lW0vPOe52efOvu04s+ac8fVceJ8LL7tg5tw5tpbN22cP63v7slnbzZs95+XMXL992Yxtzz3O5roZs8075ta6D/7+I3n1H7h+xvpndpr7tW/+TLvw97BnzHH+PufuPytz20xb62vOPtu+OYc2H597nvZ525677MI589bXofO32778gpmHzs98Zp/zc+at3/bj8lmZs3I2/vfnP/vZvPHYlefvfyjnLcuM9fOXbRv00IWXbTyuC6/ftuzc9XX++jp/2bmPD+0uc/sXP+s4h2asn7nPjGXbt73YzJr1tc3L2fZv64z17/rwf8str/qmc7eddZwLZO64z6FZJ80uMmvW17aHfWYsq7rsIue8bFdzzDzO3NkvO3/Ztm1rt7Odk7n9a5/39T6z/5E/9Ed+qLvX8xxNfoX40KHnT30IDqCXXHH1qkdgH/qzz79+/kYM541XXbXqEdiHvlKGYZu9lOFkCYUYAAD2M4UYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIamEAMAMDSFGACAoSnEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABiaQgwAwNAUYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQqrtXPQMAAKyMK8QAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIamEAMAMDSFGACAoSnEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABiaQgwAwNAOL+EYvYRjAABA7WWnZRTiHD5ybJLcs6dPyZa9q+wzT5ycJPvyo9cmmeYcP3v61IHM3sqXLftSzk4O3nPT8172CNl75ZYJAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABja3EJcVS+rqldX1RXPWv6a6cYCAIDl2LEQV9WPJPn3SX44ySer6uZtq//elIMBAMAyzHunuh9M8ie7+8mquibJe6vqmu7+qezw1nhVtZZkLUlOnDixoFEBAGDx5hXiQ939ZJJ096er6lXZKMUvyQ6FuLvXk6xvffrWW+9YxKwAALBw8+4h/lxV/YmtTzbL8XclOZrkG6ccDAAAlmFeIX5Tkv+1fUF3n+3uNyX5tsmmAgCAJdnxlonufnyHdb+0+HEAAGC5vA4xAABDU4gBABiaQgwAwNAUYgAAhqYQAwAwNIUYAIChVXdPfYzJDwAAANnhnZR34goxAABD2/GNORZ2kCPHJsk9e/qUbNkrz06S3//737/w7Of/+M8mSX7gmu9ZePa7P/3eJJ6bsmXvNTuZ5vlzULO38mXLXnX2XrlCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDm/s6xFX1siQ3J9l60bhTSe7t7oenHAwAAJZhxyvEVfW2JPdk423wPrb5UUnurqrbpx8PAACmNe8K8S1JbuzuM9sXVtWdSR5K8vapBgMAgGWYdw/x00mumrH8ys11M1XVWlU9WFUPrq+vX8x8AAAwqXlXiG9Lcl9VPZLksc1lVye5LsmtF9qpu9eTbDXhfuutd1zsnAAAMIkdC3F3v7+qbkhyU879o7oHuvupqYcDAICpzX2Vie5+Osn9S5gFAACWzusQAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYWnX31MeY/AAAAJCk9rKTK8QAAAxt7htzLOQgR47N32gPzp4+JVv2yrOTac7xrewzT5xcePblR69N4rkpW/Zes5Npn/cHLXsrX7bsVWfvlSvEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADC0ua9DXFUvS3Jzkq0XjTuV5N7ufnjKwQAAYBl2vEJcVW9Lck823gbvY5sfleTuqrp9+vEAAGBa864Q35Lkxu4+s31hVd2Z5KEkb59qMAAAWIZ59xA/neSqGcuv3Fw3U1WtVdWDVfXg+vr6xcwHAACTmneF+LYk91XVI0ke21x2dZLrktx6oZ26ez3JVhPut956x8XOCQAAk9ixEHf3+6vqhiQ35dw/qnugu5+aejgAAJja3FeZ6O6nk9y/hFkAAGDpvA4xAABDU4gBABiaQgwAwNAUYgAAhqYQAwAwNIUYAIChKcQAAAytunvqY0x+AAAASFJ72WnuG3MswuEjx+ZvtAdnT5+SLXvl2ck05/gyss88cXLh2Uly+dFrD+z/n7Jl7yY7ObjP+4P6PZctezfZe+WWCQAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIa250JcVW9Z5CAAALAKF3OF+I4Lraiqtap6sKoeXF9fv4hDAADAtA7vtLKqPnGhVUm+/kL7dfd6kq0m3G+99YLdGQAAVmrHQpyN0vsdSb7wrOWV5JcnmQgAAJZoXiH+hSRXdPfHn72iqj40yUQAALBEOxbi7r5lh3Xfu/hxAABgubzsGgAAQ1OIAQAYmkIMAMDQFGIAAIamEAMAMDSFGACAoVV3T32MyQ8AAADZePO452zeG3MsxOEjxybJPXv6lGzZK89Okj94xXULz/7Ck48mmeb5szX3lN+XLz8yzZtZPu/6P3NgzxXZl052cvCem8t43suWversvXLLBAAAQ1OIAQAYmkIMAMDQFGIAAIamEAMAMDSFGACAoSnEAAAM7TkX4qr6hikGAQCAVdjLFeL3LXwKAABYkb0U4j29JR4AAOxHe3nr5rvmbVBVa0nWkuTEiRN7OAQAACzHc75C3N3v3MU26919vLuPr62t7W0yAABYAq8yAQDA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABiaQgwAwNCqu6c+xuQHAACAJLWXnQ4veoqZBzlybJLcs6dP5cwTJyfJvvzotZPOLfvSyU6mOccPavZWvmzZl3J2cvCem573skfI3iu3TAAAMDSFGACAoSnEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGNveNOarqZUluTrL1Ksqnktzb3Q9PORgAACzDjleIq+ptSe7JxtvgfWzzo5LcXVW3Tz8eAABMa94V4luS3NjdZ7YvrKo7kzyU5O2zdqqqtSRrSXLixIkFjAkAANOYdw/x00mumrH8ys11M3X3encf7+7ja2trFzMfAABMat4V4tuS3FdVjyR5bHPZ1UmuS3LrlIMBAMAy7FiIu/v9VXVDkpty7h/VPdDdT009HAAATG3uq0x099NJ7l/CLAAAsHRehxgAgKEpxAAADE0hBgBgaAoxAABDU4gBABiaQgwAwNCqu6c+xuQHAACAJLWXnea+DvEiHD5ybP5Ge3D29KmceeLkJNmXH7120rllXzrZyTTn+EHN3sqXLftSzk4O3nPT8172CNl75ZYJAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABiaQgwAwNB2LMRV9aeq6ms2H391Vd1RVf+hqt5RVS9czogAADCdeVeI353kS5uPfyrJC5O8Y3PZP59wLgAAWIp5b918qLvPbj4+3t3fvPn4v1bVxy+0U1WtJVlLkhMnTlz8lAAAMJF5V4g/WVVv2Xz861V1PEmq6oYkZy60U3evd/fx7j6+tra2oFEBAGDx5hXiv5Lk26vqfyR5eZKPVNXJJHdtrgMAgANtx1smuvuLSd68+Yd1L93c/vHu/twyhgMAgKnNu4c4SdLdv5fk1yeeBQAAls7rEAMAMDSFGACAoSnEAAAMTSEGAGBoCjEAAENTiAEAGFp199THmPwAAACQpPay065eh/hiHT5ybJLcs6dPyZ6RfeaJk5NkX3702gP7PZkyO5nmHD+o2Vv5smVfytnJwXtuet7LHiF7r9wyAQDA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDe86FuKq+YYpBAABgFfZyhfh9C58CAABWZC+FeE9viQcAAPvRXt66+a55G1TVWpK1JDlx4sQeDgEAAMvxnK8Qd/c7d7HNencf7+7ja2tre5sMAACWwKtMAAAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIamEAMAMLTq7qmPMfkBAAAgSe1lp8OLnmLmQY4cmyT37OlTsmdkn3ni5CTZlx+99sB+T6bMTqY5xw9q9la+bNmXcnZy8J6bnveyR8jeK7dMAAAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIY29405quplSW5OsvUqyqeS3NvdD085GAAALMOOV4ir6m1J7snG2+B9bPOjktxdVbdPPx4AAExr3hXiW5Lc2N1nti+sqjuTPJTk7bN2qqq1JGtJcuLEiQWMCQAA05h3D/HTSa6asfzKzXUzdfd6dx/v7uNra2sXMx8AAExq3hXi25LcV1WPJHlsc9nVSa5LcuuUgwEAwDLsWIi7+/1VdUOSm3LuH9U90N1PTT0cAABMbe6rTHT300nuX8IsAACwdF6HGACAoSnEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0Kq7pz7G5AcAAIAktZed5r4O8SIcPnJs/kZ7cPb0Kdkzss88cXKS7MuPXntgvydTZifTnOMHNXsrX7bsSzk7OXjPTc972SNk75VbJgAAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABjangtxVb1lkYMAAMAqXMwV4jsutKKq1qrqwap6cH19/SIOAQAA0zq808qq+sSFViX5+gvt193rSbaacL/11gt2ZwAAWKkdC3E2Su93JPnCs5ZXkl+eZCIAAFiieYX4F5Jc0d0ff/aKqvrQJBMBAMAS7ViIu/uWHdZ97+LHAQCA5fKyawAADE0hBgBgaAoxAABDU4gBABiaQgwAwNAUYgAAhlbdPfUxJj8AAABk483jnrNlXCGu5/JRVT/0XPeRLVv2wZ5dtmzZ+zNftuwDmL0n+/GWiTXZsmXvy3zZsmXvv+yp82XLvpSzv2I/FmIAAFgahRgAgKHtx0K8Llu27H2ZL1u27P2XPXW+bNmXcvZXLONVJgAAYN/aj1eIAQBgafZdIa6qJ1c9wwiq6t1V9fmq+uSqZ9kvqurFVfXBqvpUVT1UVT+64PzXVNVvVNWjVXX7Acp2rgBwSdt3t0xU1ZPdfcWq57jUVdW3JXkyyc919ytWPc9+UFVXJrmyu3+1ql6Q5FeS/KXu/tQCsi9L8ptJ/kKSx5M8kOQN+z17M9+5AsAlbd9dIZ5CVV2z/epWVf2NqvqJBWY/XFV3bV5V/MWq+upFZE+puz+c5HdWPcd+0t2/3d2/uvn4/yZ5OMmxBcXflOTR7j7Z3aeT3JPk5gOQ7VwB4JI3RCFeguuT/HR335jkd5N894rn4SJV1TVJXpnkowuKPJbksW2fP57Fle0pswHgkqcQL8ZvdffHNx//SpJrVjgLF6mqrkjyb5Lc1t2/t+p5AIBpjVKIz+bcr/WrFpz/5W2Pn0pyeMH5LElVXZ6NMvwvuvvfLjD6VJIXb/v8RZvL9ns2AFzyRinEn0vyh6vq66rqeUm+a9UDjaCq7quqA/Or+6qqJO9K8nB337ng+AeSXF9VL62qI0len+TeA5ANAJe8fVWIq+pwzr3auhDdfSbJTyb5WJL/nOS/L/oYB01V3Z3kI0n+aFU9XlW3LDj/UJLrcrD+GOtbk7wxyZ+rqo9vfrx2EcHdfTbJrUk+kI0/1ntPdz+037OT6c8VAFi1ffWya1X1TUnu6u6bVj0LF6eqXpHkB7r7x1Y9CwDATvZNIa6qv5rkR7Lxh0y/uOp5AAAYw74pxAAAsAr76h5iAABYNoUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAY2v8HxBTkRm3CML0AAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 864x576 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light",
            "tags": []
          },
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsQAAAFOCAYAAACBlgugAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAYdElEQVR4nO3df7Cld10f8Pdns7uBJBBalyosgRgFBX+UyIqoI8YJreB0pGOdjjoqMpRrx6JgO53S6oylv6a0ikNnVHYB8UdFxOjU1Kb+wGKt1kRSECUkIq4UshElBRIBdXfDp3/cc8lN9tx7bm7Oc+69+329ZjK5+zzneX8/e+Y5d9/77HPPqe4OAACM6tBeDwAAAHtJIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIa2tEJcVc+tqj+oqvdW1cvn7L+0qn5mtv+Wqrp6WWuzf+3gvPjHVfXuqvq9qvq1qnrSXszJai06LzY97u9VVVfViVXOx97YyXlRVX9/9j3jtqp646pnZPV28OfIE6vqrVX1jtmfJV+zF3OyOlX1o1X1Z1X1ri32V1X9p9k583tV9UWLMpdSiKvqkiQ/lOR5SZ6W5Bur6mkPetiLknykuz87yQ8meeUy1mb/2uF58Y4kJ7r7C5PckOQ/rHZKVm2H50Wq6lFJXprkltVOyF7YyXlRVU9O8s+TfHl3f16Sl618UFZqh98vvjfJm7v72iTfkOSHVzsle+DHkjx3m/3PS/Lk2X9rSX5kUeCyrhA/M8l7u/t0d59N8qYkz3/QY56f5MdnX9+Q5PqqqiWtz/608Lzo7rd29ydmv7w5yRNWPCOrt5PvF0nyr7P+F+e/XOVw7JmdnBcvTvJD3f2RJOnuP1vxjKzeTs6LTvLo2ddXJrlrhfOxB7r7N5J8eJuHPD/JT/S6m5M8pqoet13msgrx8SQf2PTrO2fb5j6mu88nuSfJpy1pffannZwXm70oyX+fdCL2g4Xnxeyft67q7v+2ysHYUzv5fvGUJE+pqt+qqpurarsrRFwcdnJe/Msk31xVdya5Kcl3rmY09rGH2j9yeNJxYIeq6puTnEjylXs9C3urqg4leVWSb9vjUdh/Dmf9n0Cvy/q/Jv1GVX1Bd390T6dir31jkh/r7h+oqi9N8pNV9fnd/cm9HoyDY1lXiM8kuWrTr58w2zb3MVV1OOv/rPH/lrQ++9NOzotU1XOSfE+Sr+3uv1rRbOydRefFo5J8fpJfr6r3JXlWkhv9YN1FbyffL+5McmN3n+vuP07ynqwXZC5eOzkvXpTkzUnS3b+d5BFJjq1kOvarHfWPzZZViN+W5MlV9ZlVdTTrN7Xf+KDH3JjkBbOvvz7J/+juXtL67E8Lz4uqujbJyayXYfcDjmHb86K77+nuY919dXdfnfV7y7+2u2/dm3FZkZ38OfJfsn51OFV1LOu3UJxe5ZCs3E7Oi/cnuT5JquqpWS/EH1rplOw3Nyb51tm7TTwryT3d/SfbHbCUWya6+3xVvSTJLye5JMmPdvdtVfWvktza3TcmeX3W/xnjvVm/EfoblrE2+9cOz4v/mOSKJD87+xnL93f31+7Z0Exuh+cFg9nhefHLSf52Vb07yX1J/ml3+5fGi9gOz4t/kuS1VfXdWf8Bu29zwe3iVlU/nfW/HB+b3Tv+fUmOJEl3vybr95J/TZL3JvlEkhcuzHTOAAAwMp9UBwDA0BRiAACGphADADA0hRgAgKEpxAAADG3yQlxVa1OvwcHjvGAe5wXzOC+Yx3nBPLs9L1ZxhdgJyzzOC+ZxXjCP84J5nBfMs28LMQAA7FuTfzDHocOP6UOHLk9tXrTqgq8rF247tPlxm/ZvbN+ccyhzts1Z59CcdTbnP5Rj7l/z/r9XzM2ZN/ucbYuOXzzHQz/mAftzv7lzznuOM2/N+819DlP5vx97f550xRM/9dit53xg9gNn37TOvDnn/H62PmbrbVtmLpxjY9uiY3axvzdvm/e4RbPngpy5+zdvm/PYB64z2zdntkVrbux768f/MNdf9uQ5++8/aOHvffY9bevnsOfMceExDzx+XuammTb214JjNj05h2ZfP/A87Qse+8BtW+cs2l+HLnzc5u1bZh66MPP+Yy7MWbR/07fLB2XOy1n//0/edVe+5fjjLjz+UC7Yljn7F2/bNOihrbetf11b79+07YH768L9deG2B359aGeZm3/z89Y5NGf/3GPmbNv82IebWfN+b4tyNv3ZOmf/63/j9/Oi6/7mAx87b50tMrc95tC8k2YHmTXv97aLY+Zsq7rkYc55yY7mmLvOwtkvuXDbpsfWTmd7QObm3/ui3+/9xx997Gd9e3efykM0+RXiQ4cun3oJDqAnXfHEvR6BfeirLn/y4gcxnG95/OP3egT2oU+VYdhkN2U4WUEhBgCA/UwhBgBgaAoxAABDU4gBABiaQgwAwNAUYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIamEAMAMDSFGACAoSnEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABiaQgwAwNAUYgAAhqYQAwAwtOruvZ4BAAD2jCvEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABiaQgwAwNAUYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQDq9gjV7BGgAAULs5aBWFOIePHp8k9/zZM7Jl73l2khyZIP/cLPvSR1y19Oy/+ssPJEkuv+zqpWcnycc/8b48+vJrJsm+9+Onc+UVnzVJ9j0f+6P89Uc9eZLsD//5H06afezRT5kk++5735PHXvk5k2R/6J4/OLDZSfLpV37u0rP/9J47DmT2Rv5Bzf6Mxzx1kuwPfvR22SvO3i23TAAAMDSFGACAoSnEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGtutCXFUvXOYgAACwFx7OFeJXbLWjqtaq6taquvXUqVMPYwkAAJjWth/dXFW/t9WuJJ++1XHdfSrJRhPu73jJlt0ZAAD21LaFOOul96uTfORB2yvJ/55kIgAAWKFFhfgXk1zR3b/74B1V9euTTAQAACu0bSHu7hdts++blj8OAACslrddAwBgaAoxAABDU4gBABiaQgwAwNAUYgAAhqYQAwAwtOruqdeYfAEAAMj6h8c9ZK4QAwAwtEWfVLecRY4enyT3/NkzsudkX/eE50yS/et3viV/ccO/mST7kV//vQf2+U6mOccPavZGvmzZF3N2cvBem173skfI3i1XiAEAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAvfh7iqrknydUmuSnJfkvckeWN33zvxbAAAMLltrxBX1XcleU2SRyT54iSXZr0Y31xV100+HQAATGzRFeIXJ3l6d99XVa9KclN3X1dVJ5P8QpJrJ58QAAAmtJN7iDdK86VJrkiS7n5/kiNbHVBVa1V1a1XdeurUqYc/JQAATGTRFeLXJXlbVd2S5CuSvDJJquqxST681UHdfSrJRhPu73jJK5YwKgAALN+2hbi7X11Vb0ny1CQ/0N13zLZ/KMmzVzAfAABMauG7THT3bUluW8EsAACwct6HGACAoSnEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0Kq7p15j8gUAACBJ7eYgV4gBABjawg/mWMoiR49Pknv+7BnZK84+d/fpSbKPHLvmwD4nyTTn+EHN3siXLftizk4O3mvT6172CNm75QoxAABDU4gBABiaQgwAwNAUYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGgLP5ijqq5J8nVJrkpyX5L3JHljd9878WwAADC5ba8QV9V3JXlNkkck+eIkl2a9GN9cVddNPh0AAExs0RXiFyd5enffV1WvSnJTd19XVSeT/EKSa+cdVFVrSdaS5OTJk8ucFwAAlmon9xBvlOZLk1yRJN39/iRHtjqgu09194nuPrG2tvbwpwQAgIksukL8uiRvq6pbknxFklcmSVU9NsmHJ54NAAAmt20h7u5XV9Vbkjw1yQ909x2z7R9K8uwVzAcAAJNa+C4T3X1bkttWMAsAAKyc9yEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADC06u6p15h8AQAASFK7OWjh+xAvw+GjxyfJPX/2jOyLKPvc3acnyT5y7JpJ506mOccPavZGvmzZF3N2cvBem173skfI3i23TAAAMDSFGACAoSnEAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0Ba+D3FVPTNJd/fbquppSZ6b5I7uvmny6QAAYGLbFuKq+r4kz0tyuKp+NcmXJHlrkpdX1bXd/W9XMCMAAExm0RXir0/y9CSXJvlgkid0971V9f1JbkmiEAMAcKAtKsTnu/u+JJ+oqj/q7nuTpLv/oqo+udVBVbWWZC1JTp48ubRhAQBg2Rb9UN3Zqrps9vUzNjZW1ZVJtizE3X2qu09094m1tbUljAkAANNYdIX42d39V0nS3ZsL8JEkL5hsKgAAWJFtC/FGGZ6z/e4kd08yEQAArJD3IQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIamEAMAMDSFGACAoVV3T73G5AsAAECS2s1Biz6pbikOHz0+Se75s2dkX0TZ5+4+PUn2kWPXTDp3Ms05flCzN/Jly76Ys5OD99r0upc9QvZuuWUCAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIb2kAtxVX3GFIMAAMBe2M0V4puWPgUAAOyR3RTiXX0kHgAA7Ee7+ejm1y56QFWtJVlLkpMnT+5iCQAAWI2HfIW4u394B4851d0nuvvE2tra7iYDAIAV8C4TAAAMTSEGAGBoCjEAAENTiAEAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADK26e+o1Jl8AAACS1G4OOrzsKeYucvT4JLnnz57J2ffdOkn20atPTDq37IsnO5nmHD+o2Rv5smVfzNnJwXttet3LHiF7t9wyAQDA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABjawg/mqKrPTfL8JBvvonwmyY3dffuUgwEAwCpse4W4qv5Zkjdl/WPwfmf2XyX56ap6+fTjAQDAtBZdIX5Rks/r7nObN1bVq5LcluTfzzuoqtaSrCXJyZMnlzAmAABMY9E9xJ9M8vg52x832zdXd5/q7hPdfWJtbe3hzAcAAJNadIX4ZUl+rar+MMkHZtuemOSzk7xkysEAAGAVti3E3f1LVfWUJM/MA3+o7m3dfd/UwwEAwNQWvstEd38yyc0rmAUAAFbO+xADADA0hRgAgKEpxAAADE0hBgBgaAoxAABDU4gBABhadffUa0y+AAAAJKndHLTwfYiX4dzdpyfJPXLsmhw+enzxA3fh/NkzsmXvKDvJJPkHNXsjX7bsizk7OXivTa972SNk75ZbJgAAGJpCDADA0BRiAACGphADADA0hRgAgKEpxAAADE0hBgBgaAoxAABD27YQV9WXVNWjZ18/sqpeUVX/tapeWVVXrmZEAACYzqIrxD+a5BOzr1+d5Mokr5xte8OEcwEAwEosKsSHuvv87OsT3f2y7v7N7n5Fkmu2Oqiq1qrq1qq69dSpU0sbFgAAlu3wgv3vqqoXdvcbkryzqk50961V9ZQk57Y6qLtPJdlown3u7tNLGhcAAJZr0RXif5DkK6vqj5I8LclvV9XpJK+d7QMAgANt2yvE3X1Pkm+b/WDdZ84ef2d3/+kqhgMAgKktumUiSdLd9yZ558SzAADAynkfYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ6vunnqNyRcAAIAktZuDdvQ+xA/X4aPHJ8k9f/ZMpvpY6CPHrpl0btkXT3YyzTl+ULM38mXLvpizk4P32vS6lz1C9m65ZQIAgKEpxAAADE0hBgBgaAoxAABDU4gBABiaQgwAwNAUYgAAhvaQC3FVfcYUgwAAwF7YzRXim5Y+BQAA7JHdFOJdfSQeAADsR7v56ObXLnpAVa0lWUuSkydP7mIJAABYjYd8hbi7f3gHjznV3Se6+8Ta2truJgMAgBXwLhMAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIamEAMAMDSFGACAoSnEAAAMrbp76jUmXwAAAJLUbg46vOwp5i5y9PgkuefPnpk0+9zdpyfJPnLsmgP7nMi+MDuZ5hw/qNkb+bJlX8zZycF7bXrdyx4he7fcMgEAwNAUYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAY2sJCXFWfW1XXV9UVD9r+3OnGAgCA1di2EFfVdyX5hSTfmeRdVfX8Tbv/3ZSDAQDAKiz66OYXJ3lGd3+sqq5OckNVXd3dr842nxVdVWtJ1pLk5MmTSxoVAACWb1EhPtTdH0uS7n5fVV2X9VL8pGxTiLv7VJJTG7/8jpe8YhmzAgDA0i26h/hPq+rpG7+YleO/k+RYki+YcjAAAFiFRYX4W5N8cPOG7j7f3d+a5NmTTQUAACuy7S0T3X3nNvt+a/njAADAankfYgAAhqYQAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ6vunnqNyRcAAIBs80nK21n00c1Lcfjo8Ulyz589c2Czz919epLsI8euObDPyUHNTqY5xw9q9ka+bNkXc3Zy8F6bXveyR8jeLbdMAAAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIa27QdzVNXRJN+Q5K7ufktVfVOSL0tye5JT3X1uBTMCAMBkFn1S3Rtmj7msql6Q5IokP5/k+iTPTPKCaccDAIBpLSrEX9DdX1hVh5OcSfL47r6vqv5zkndudVBVrSVZS5KTJ08ubVgAAFi2RfcQH5rdNvGoJJcluXK2/dIkR7Y6qLtPdfeJ7j6xtra2nEkBAGACi64Qvz7JHUkuSfI9SX62qk4neVaSN008GwAATG7bQtzdP1hVPzP7+q6q+okkz0ny2u7+nVUMCAAAU1p0hTjdfdemrz+a5IZJJwIAgBXyPsQAAAxNIQYAYGgKMQAAQ1OIAQAYmkIMAMDQFGIAAIZW3T31GpMvAAAASWo3B63iCnE9lP+q6tsf6jGyZcs+2LPLli17f+bLln0As3dlP94ysSZbtux9mS9btuz9lz11vmzZF3P2p+zHQgwAACujEAMAMLT9WIhPyZYte1/my5Yte/9lT50vW/bFnP0pq3iXCQAA2Lf24xViAABYmX1TiKvqMVV1Q1XdUVW3V9WXTrDGa6rqy5eYd1VVvbWq3l1Vt1XVS5eVvWmN755lv6uqfrqqHrHk/KU+J5tyXzqb+baqetkScx9RVb9TVe+cZb9iWdkAwJj2zS0TVfXjSf5Xd7+uqo4muay7P7rkNX43yTO6+74l5T0uyeO6++1V9agk/yfJ3+3udy8p/3iS30zytO7+i6p6c5KbuvvHlpE/W2Opz8ks8/OTvCnJM5OcTfJLSf5hd793CdmV5PLu/lhVHcn68/PS7r754WYDAGPaF1eIq+rKJM9O8vok6e6zE5ThpyZ5zzKLX3f/SXe/ffb1nye5PcnxZeXPHE7yyKo6nOSyJHctK3iK52TmqUlu6e5PdPf5JP8zydctI7jXfWz2yyOz//bH3+oAgANpXxTiJJ+Z5ENJ3lBV76iq11XV5Ute43lZv1I5iaq6Osm1SW5ZVmZ3n0ny/Unen+RPktzT3b+yrPxM95y8K8lXVNWnVdVlSb4myVXLCq+qS2ZXtv8sya9299KecwBgPPulEB9O8kVJfqS7r03y8SQvX/IaX52JCnFVXZHk55K8rLvvXWLuX0vy/Kz/heHxSS6vqm9eVn4mek66+/Ykr0zyK7P8302yzCvz93X305M8IckzZ7doAADsyn4pxHcmuXPTlb4bsl6Ql2J2lfIx3b202w02ZR/Jehn+qe7++SXHPyfJH3f3h7r7XJKfT/Jlywie8jlJku5+fXc/o7ufneQjSd4zwRofTfLWJM9ddjYAMI59UYi7+4NJPlBVnzPbdH2Spfxg2sxXZb04LdXsB7xen+T27n7VsvOzfqvEs6rqstla12f9PuVlmOQ52VBVf2P2/ydm/f7hNy4p97FV9ZjZ149M8reS3LGMbABgTIf3eoBNvjPJT83eYeJ0khcuMft5Wb/qvGxfnuRbkvz+7J7WJPkX3X3TMsK7+5aquiHJ25OcT/KOLO8TW6Z6Tjb8XFV9WpJzSf7REn9I8nFJfryqLsn6X+je3N2/uKRsAGBA++Zt16ZUVW9P8iWz2w6I5wQAYMMQhRgAALayL+4hBgCAvaIQAwAwNIUYAIChKcQAAAxNIQYAYGgKMQAAQ1OIAQAY2v8H3NqTIc3iwlEAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 864x576 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light",
            "tags": []
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "import seaborn as sns\n",
        "import pandas as pd\n",
        "\n",
        "input_batch, output_batch = train2batch(test_x, test_y, batch_size=BATCH_NUM)\n",
        "input_minibatch, output_minibatch = input_batch[0], output_batch[0]\n",
        "\n",
        "with torch.no_grad():\n",
        "  # データをテンソルに変換\n",
        "  input_tensor = torch.tensor(input_minibatch, device=device)\n",
        "  output_tensor = torch.tensor(output_minibatch, device=device)\n",
        "  hs, h = encoder(input_tensor)\n",
        "  source = output_tensor[:, :-1]\n",
        "  decoder_output, _, attention_weight= attn_decoder(source, hs, h)\n",
        "\n",
        "\n",
        "for i in range(3):\n",
        "  with torch.no_grad():\n",
        "    df = pd.DataFrame(data=torch.transpose(attention_weight[i], 0, 1).cpu().numpy(), \n",
        "                      columns=[id2char[idx.item()] for idx in input_tensor[i]], \n",
        "                      index=[id2char[idx.item()] for idx in output_tensor[i][1:]])\n",
        "    plt.figure(figsize=(12, 8)) \n",
        "    sns.heatmap(df, xticklabels = 1, yticklabels = 1, square=True, linewidths=.3,cbar_kws = dict(use_gridspec=False,location=\"top\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GSOdWunU-A6R"
      },
      "source": [
        "#課題\n",
        "* GRUの他にRNNやLSTMを試してみよう\n",
        "* 色々なAttentionの可視化から考察してみよう"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5RLwALW8bo18"
      },
      "source": [
        "参考にした素晴らしいサイト: https://qiita.com/m__k/items/646044788c5f94eadc8d"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FM1WVqao-Cx_"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "authorship_tag": "ABX9TyOnUo3pdfAzp18n0h2XhvDh",
      "collapsed_sections": [],
      "include_colab_link": true,
      "name": "05_Attention_alpha.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}