Tasks/task3/lab_03.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Третье практическое задание. Реализация дропаута в рекуррентных нейронных сетях\n",
    "Практикум на ЭВМ для 317 группы, весна 2021\n",
    "\n",
    "#### Фамилия, имя:\n",
    "\n",
    "Дата выдачи: 3 апреля 00:01\n",
    "\n",
    "Мягкий дедлайн: 17 апреля 00:01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Данное задание будет состоять из двух частей:\n",
    "1. Применение рекуррентной сети для решения задачи классификации текста. Более конкретно -- предсказания рейтинга отзыва фильма.\n",
    "2. Простейшая лингвистическая модель для генерации текста на основе LSTM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "При выполнении задания вы обучите LSTM с разным уровнем \"коробочности\", а также познакомитесь с различными способами применения DropOut к рекуррентным архитектурам. В рекуррентных архитектурах вариантов, куда можно наложить бинарную маску шума, гораздо больше, чем в нейросетях прямого прохода.\n",
    "\n",
    "Во второй части вы попробуете реализовать простейший рекуррентный декодер для генерации текстов.\n",
    "\n",
    "Задание сделано так, чтобы его можно было выполнять на CPU, однако RNN - это ресурсоёмкая вещь, поэтому на GPU с ними работать приятнее. Можете попробовать использовать [https://colab.research.google.com](https://colab.research.google.com) - бесплатное облако с GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Для корректного отображения картинок, вам может понадобится сделать ноутбук доверенным (Trusted) в правом верхнем углу**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Часть 0. Загрузка и предобработка данных. (1 балл)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Рекомендуемые гиперпараметры"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:04:28.454122Z",
     "start_time": "2021-04-02T00:04:28.438278Z"
    }
   },
   "outputs": [],
   "source": [
    "max_length = 200\n",
    "top_n_words = 5000\n",
    "\n",
    "hidden_dim = 128\n",
    "embedding_dim = 32\n",
    "\n",
    "num_epochs = 15\n",
    "batch_size = 64\n",
    "learning_rate = 1e-3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Первое, что нужно сделать -- скачать, предобработать данные и организовать их таким образом, чтобы их можно было подавать в нейронную сеть.\n",
    "\n",
    "Для обеих частей задания мы будем использовать [**Large Movie Review Dataset**](https://ai.stanford.edu/~amaas/data/sentiment/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Загрузка и предобработка данных"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Загрузите данные по ссылке выше. (**tip**: используйте `wget`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T18:34:47.261797Z",
     "start_time": "2021-03-30T18:34:33.768597Z"
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Распакуйте скачанные данные в папку `aclImdb` (**tip:** используйте `tar`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T17:48:59.763990Z",
     "start_time": "2021-03-30T17:48:56.998383Z"
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Посмотрите в файле `./aclImdb/README` как организованы данные"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T23:55:43.946032Z",
     "start_time": "2021-04-01T23:55:43.814779Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is not the typical Mel Brooks film. It was much less slapstick than most of his movies and actually had a plot that was followable. Leslie Ann Warren made the movie, she is such a fantastic, under-rated actress. There were some moments that could have been fleshed out a bit more, and some scenes that could probably have been cut to make the room to do so, but all in all, this is worth the price to rent and see it. The acting was good overall, Brooks himself did a good job without his characteristic speaking to directly to the audience. Again, Warren was the best actor in the movie, but \"Fume\" and \"Sailor\" both played their parts well."
     ]
    }
   ],
   "source": [
    "! cat ./aclImdb/train/pos/10003_8.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:04:33.729663Z",
     "start_time": "2021-04-02T00:04:33.710871Z"
    }
   },
   "outputs": [],
   "source": [
    "test_data_path = './aclImdb/test/'\n",
    "train_data_path = './aclImdb/train/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:04:33.709378Z",
     "start_time": "2021-04-02T00:04:32.220580Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /home/mnakhodnov/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from functools import partial\n",
    "from collections import defaultdict\n",
    "\n",
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "\n",
    "import regex\n",
    "import numpy as np\n",
    "\n",
    "import torch\n",
    "import torchtext\n",
    "from torch.utils.data import Dataset, DataLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Стандартной предобработкой данных является токенизация текстов. Полученные токены можно будет закодировать и затем подавать на вход нейронной сети. Ключевым моментом, который влияет на скорость работы нейросети и её размер в памяти -- размер словаря, используемого при токенизации. Для задачи классификации мы можем убрать часть слов (стоп слова, редкие слова), ускорив обучение без потери в качестве."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:04:35.270825Z",
     "start_time": "2021-04-02T00:04:35.250283Z"
    }
   },
   "outputs": [],
   "source": [
    "STOPWORDS = nltk.corpus.stopwords.words('english')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реализуйте функцию для токенизации текста. Выполнять токенизацию можно по-разному, но в данном задании предлагается это делать следующим образом:\n",
    "1. Привести текст к нижнему регистру\n",
    "2. Убрать html разметку из текстов (`<br />`)\n",
    "3. Убрать все символы кроме латинских букв\n",
    "4. Разбить строку по пробелам\n",
    "5. Убрать стоп слова"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:04:36.003194Z",
     "start_time": "2021-04-02T00:04:35.980408Z"
    }
   },
   "outputs": [],
   "source": [
    "def tokenize(text):\n",
    "    \"\"\"\n",
    "    :param str text: Input text \n",
    "    :return List[str]: List of words\n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:27:12.428149Z",
     "start_time": "2021-04-01T21:27:12.402448Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['hello', 'words']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenize('1. Hello <br /> words!! <br />')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь мы можем создать словарь, с помощью которого мы будем численно кодировать токены из текста и наоборот.\n",
    "\n",
    "Удобной обёрткой для создания словарей является класс `torchtext.vocab.Vocab`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:27:21.466887Z",
     "start_time": "2021-04-01T21:27:21.352085Z"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "torchtext.vocab.Vocab??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T19:51:55.300753Z",
     "start_time": "2021-04-01T19:51:55.275188Z"
    }
   },
   "source": [
    "Чтобы создать такой словарь, сначала нужно создать словарь со всеми токенами в тексте и их частотами встречаемости:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:13.547038Z",
     "start_time": "2021-04-02T00:04:38.190688Z"
    }
   },
   "outputs": [],
   "source": [
    "counter = defaultdict(int)\n",
    "\n",
    "for path in ['./aclImdb/test/neg', './aclImdb/test/pos', './aclImdb/train/neg', './aclImdb/train/pos']:\n",
    "    for file_path in os.listdir(path):\n",
    "        text = open(os.path.join(path, file_path), 'r', encoding='utf-8', errors='ignore').read().strip()\n",
    "        for token in tokenize(text):\n",
    "            counter[token] += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для работы с текстами нам необходимо зарезервировать два специальных токена:\n",
    "1. `<pad>` для токена означающего паддинг\n",
    "2. `<unk>` для токенов, которые отсутствуют в словаре"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:28:18.239274Z",
     "start_time": "2021-04-01T21:28:18.214979Z"
    }
   },
   "outputs": [],
   "source": [
    "specials = ['<pad>', '<unk>']\n",
    "for special in specials:\n",
    "    counter[special] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создайте словарь из словаря частот `counter`. Наименьшие id отдайте под специальные токены. Отбросьте низкочастотные слова, оставив только `top_n_words` слов"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:28:23.041153Z",
     "start_time": "2021-04-01T21:28:22.899444Z"
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "vocab = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:28:24.332126Z",
     "start_time": "2021-04-01T21:28:24.306890Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.lookup_indices(['<pad>', '<unk>'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:28:26.254910Z",
     "start_time": "2021-04-01T21:28:26.231012Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 3, 1, 254]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab.lookup_indices(['this', 'film', 'was', 'awful'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь мы готовы создать обёртку-датасет для наших данных. \n",
    "\n",
    "Необходимо добавить несколько опции, которые понадобятся во второй части задания:\n",
    "1. Ограничение на максимальную длину текста в токенах. Если текст оказывается длиннее, то последние токены отбрасываются\n",
    "2. Возможность добавить в специальные токены `<sos>`, `<eos>` в начало и конец токенизированного текста\n",
    "    \n",
    "**tips:**\n",
    "1. Обратите особое внимание, что у длинных текстов не должен обрезаться паддинг\n",
    "2. В исходных данных рейтинг закодирован в названии файла в виде числа от 1 до 10. Для удобства, вычтите 1, чтобы рейтинг был от 0 до 9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:13.573249Z",
     "start_time": "2021-04-02T00:05:13.548593Z"
    }
   },
   "outputs": [],
   "source": [
    "class LargeMovieReviewDataset(Dataset):\n",
    "    def __init__(self, data_path, vocab, max_len, pad_sos=False, pad_eos=False):\n",
    "        \"\"\"\n",
    "        :param str data_path: Path to folder with one of the data splits (train or test)\n",
    "        :param torchtext.vocab.Vocab vocab: dictionary with lookup_indices method\n",
    "        :param int max_len: Maximum length of tokenized text\n",
    "        :param bool pad_sos: If True pad sequence at the beginning with <sos> \n",
    "        :param bool pad_eos: If True pad sequence at the end with <eos>         \n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        \n",
    "        self.pad_sos = pad_sos\n",
    "        if self.pad_sos:\n",
    "            self.sos_id = vocab.lookup_indices(['<sos>'])[0]\n",
    "        self.pad_eos = pad_eos\n",
    "        if self.pad_eos:\n",
    "            self.eos_id = vocab.lookup_indices(['<eos>'])[0]\n",
    "        \n",
    "        self.vocab = vocab\n",
    "        self.max_len = max_len\n",
    "        self.data_path = data_path\n",
    "        self.negative_path = os.path.join(data_path, 'neg')\n",
    "        self.positive_path = os.path.join(data_path, 'pos')\n",
    "        \n",
    "        self.negative_paths = []\n",
    "        self.positive_paths = []\n",
    "\n",
    "        for file_path in os.listdir(self.negative_path):\n",
    "            self.negative_paths.append(os.path.join(self.negative_path, file_path))\n",
    "\n",
    "        for file_path in os.listdir(self.positive_path):\n",
    "            self.positive_paths.append(os.path.join(self.positive_path, file_path))\n",
    "        \n",
    "        self.texts = []\n",
    "        self.tokens = []\n",
    "        self.ratings = []\n",
    "        self.labels = [0] * len(self.negative_paths) + [1] * len(self.positive_paths)\n",
    "        \n",
    "        # Read each file in data_path, tokenize it, get tokens ids, its rating and store\n",
    "        for path in self.negative_paths + self.positive_paths:\n",
    "            # YOUR CODE HERE\n",
    "            pass\n",
    "        \n",
    "    def __getitem__(self, idx):\n",
    "        \"\"\"\n",
    "        :param int idx: index of object in dataset\n",
    "        :return dict: Dictionary with all useful object data \n",
    "            {\n",
    "                'text' str: unprocessed text,\n",
    "                'label' torch.tensor(dtype=torch.long): sentiment of the text (0 for negative, 1 for positive)\n",
    "                'rating' torch.tensor(dtype=torch.long): rating of the text\n",
    "                'tokens' torch.tensor(dtype=torch.long): tensor of tokens ids for the text\n",
    "                'tokens_len' torch.tensor(dtype=torch.long): number of tokens\n",
    "            }\n",
    "        \"\"\"\n",
    "        # YOUR CODE HERE\n",
    "        # Do not forget to add padding if needed!\n",
    "        pass\n",
    "    \n",
    "    def __len__(self):\n",
    "        \"\"\"\n",
    "        :return int: number of objects in dataset \n",
    "        \"\"\"\n",
    "        # YOUR CODE HERE\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создайте датасеты для тестовой и обучающей выборки. \n",
    "\n",
    "Обратите внимание, что для задачи классификации нам не потребуется паддинг с помощью `<sos>`, `<eos>`. \n",
    "\n",
    "Не забудьте обрезать длинные тексты, передав параметр `max_length`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:31:32.123618Z",
     "start_time": "2021-04-01T21:30:54.950259Z"
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "test_dataset = None\n",
    "train_dataset = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Посмотрим, как выглядит объект в датасете:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:31:36.017106Z",
     "start_time": "2021-04-01T21:31:35.988797Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': \"I like Christophe Lambert as an actor. He has played in several good movies (Highlander, Subway, Greystoke, ...). But I can't even think he has played in that movie. The story is nearly nothing, the special effects are very bad and the actors also. To resume the movie, I have only one thing to say : it's the first time that I go to watch a movie and want to leave after 10 minutes.\",\n",
       " 'label': tensor(0),\n",
       " 'tokens': tensor([  5,   1,   1, 167, 143, 320,   6,  25,   1,   1,   1,   8,  27, 143,\n",
       "           2,  10, 664,  66, 192, 174,  15,  57,  19,   1,   2,   4,  55,  49,\n",
       "          20,   7,  52,  28,   2,  79, 430, 121]),\n",
       " 'tokens_len': tensor(36),\n",
       " 'rating': tensor(1)}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dataset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь нам нужно создать `DataLoader` для наших данных. `DataLoader` умеет из коробки объединять список объектов из датасета в один батч, даже когда датасет возвращает словарь тензоров. Однако, это работает только в случае когда все эти тензоры имеют один и тот же размер во всех батчах. В нашем случае, это не так, так как разные тексты могут иметь разную длину.\n",
    "\n",
    "Чтобы обойти эту проблему у `DataLoader` есть параметр `collate_fn`, который позволяет задать функцию для объединения списка объектов в один батч."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Чтобы объединить несколько тензоров разной длины в один можно использовать функцию `torch.nn.utils.rnn.pad_sequence`\n",
    "\n",
    "Обратите внимание на её аргументы:\n",
    "1. `batch_first` определяет по какой оси \"складывать\" тензоры. Предпочтительнее использовать `batch_first=False` так как это может упростить выполнение задания в дальнейшем \n",
    "2. `padding_value` -- число, которое будет использоваться в качестве паддинга, чтобы сделать все тензоры одинаковой длины"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:33:06.372403Z",
     "start_time": "2021-04-01T21:33:06.344559Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1,  4,  6],\n",
       "        [ 2,  5,  7],\n",
       "        [ 3, -1,  8],\n",
       "        [-1, -1,  9]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.nn.utils.rnn.pad_sequence([\n",
    "    torch.tensor([1, 2, 3]),\n",
    "    torch.tensor([4, 5]),\n",
    "    torch.tensor([6, 7, 8, 9])\n",
    "], batch_first=False, padding_value=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:13.709466Z",
     "start_time": "2021-04-02T00:05:13.575053Z"
    }
   },
   "outputs": [],
   "source": [
    "def collate_fn(batch, padding_value, batch_first=False):\n",
    "    \"\"\"\n",
    "    :param List[Dict] batch: List of objects from dataset\n",
    "    :param int padding_value: Value that will be used to pad tokens\n",
    "    :param bool batch_first: If True resulting tensor with tokens must have shape [B, T] otherwise [T, B]\n",
    "    :return dict: Dictionary with all data collated\n",
    "        {\n",
    "            'ratings' torch.tensor(dtype=torch.long): rating of the text for each object in batch\n",
    "            'labels' torch.tensor(dtype=torch.long): sentiment of the text for each object in batch\n",
    "            \n",
    "            'texts' List[str]: All texts in one list\n",
    "            'tokens' torch.tensor(dtype=torch.long): tensor of tokens ids padded with @padding_value\n",
    "            'tokens_lens' torch.tensor(dtype=torch.long): number of tokens for each object in batch\n",
    "        }\n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создайте даталоадеры с использованием `collate_fn`.\n",
    "\n",
    "**tips**:\n",
    "1. Передать в `collate_fn` правильное значение паддинга можно, например, с помощью `functools.partial`\n",
    "2. Если вы работаете в Google Colab, то, возможно, вам будет необходимо установить `num_workers=0` во избежание падения ноутбука."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:33:37.872760Z",
     "start_time": "2021-04-01T21:33:37.847071Z"
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "test_dataloader = None\n",
    "train_dataloader = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Посмотрим на какой-нибудь батч:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:33:46.922744Z",
     "start_time": "2021-04-01T21:33:46.755275Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(dict_keys(['labels', 'ratings', 'texts', 'tokens', 'tokens_lens']),\n",
       " tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),\n",
       " tensor([1, 1, 0, 2, 1, 2, 0, 0, 1, 0, 1, 1, 3, 0, 2, 2, 0, 2, 1, 1, 0, 0, 2, 1,\n",
       "         0, 1, 1, 3, 1, 1, 0, 2, 1, 3, 0, 0, 2, 2, 1, 2, 0, 3, 2, 3, 0, 3, 3, 1,\n",
       "         1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 3, 1, 0, 3, 0, 1]),\n",
       " tensor([[   5,  739,  575,  ...,    3,    2,   35],\n",
       "         [   1, 3840,    1,  ...,   11,  723,    2],\n",
       "         [   1,  312,   30,  ...,   80,  154, 2832],\n",
       "         ...,\n",
       "         [   0,    0,  722,  ...,    0,    0,    0],\n",
       "         [   0,    0,   47,  ...,    0,    0,    0],\n",
       "         [   0,    0, 1043,  ...,    0,    0,    0]]),\n",
       " tensor([ 36,  83, 200, 169, 200, 158,  75, 128,  24,  64, 115,  49,  27,  83,\n",
       "         136,  55,  42,  51, 200,  57, 155,  82,  32,  61,  62, 160,  74, 200,\n",
       "          85,  62,  48,  49,  69, 127,  51,  80,  81,  45, 133,  67,  83, 171,\n",
       "         117,  68, 104, 171, 200,  44,  75,  82,  30, 200, 177,  98, 123,  68,\n",
       "         200,  31,  82,  96, 177,  62,  83, 149]))"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch = next(iter(test_dataloader))\n",
    "batch.keys(), batch['labels'], batch['ratings'], batch['tokens'], batch['tokens_lens']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Часть 1. Классификация текстов. (6 баллов)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Сборка и обучение RNN в pytorch (2 балла)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создадим переменные для device-agnostic кода:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:13.809416Z",
     "start_time": "2021-04-02T00:05:13.711383Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using device: cuda:0, dtype: torch.float32\n"
     ]
    }
   ],
   "source": [
    "dtype, device, cuda_device_id = torch.float32, None, 0\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = '{0}'.format(str(cuda_device_id) if cuda_device_id is not None else '')\n",
    "if cuda_device_id is not None and torch.cuda.is_available():\n",
    "    device = 'cuda:{0:d}'.format(0)\n",
    "else:\n",
    "    device = torch.device('cpu')\n",
    "print(f'Using device: {device}, dtype: {dtype}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Наша нейросеть будет обрабатывать входную последовательность по словам (word level). Мы будем использовать простую и стандартную рекуррентную архитектуру для классификации:\n",
    "1. Слой представлений, превращающий id токена в вектор-ембеддинг этого слова\n",
    "2. Слой LSTM\n",
    "3. Полносвязный слой, предсказывающий выход по последнему скрытому состоянию\n",
    "\n",
    "Ниже дан код для сборки и обучения нашей нейросети."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Допишите класс-обёртку над LSTM для задачи классификации. \n",
    "**Не используйте циклы.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T20:59:16.467178Z",
     "start_time": "2021-04-01T20:59:16.441112Z"
    }
   },
   "source": [
    "**Для каждого тензора в функции `forward` подпишите в комментарии его размеры**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:13.873713Z",
     "start_time": "2021-04-02T00:05:13.810690Z"
    }
   },
   "outputs": [],
   "source": [
    "class RNNClassifier(torch.nn.Module):\n",
    "    def __init__(\n",
    "        self, embedding_dim, hidden_dim, output_size, vocab,\n",
    "        rec_layer=torch.nn.LSTM, dropout=None, **kwargs\n",
    "    ):\n",
    "        super().__init__()\n",
    "\n",
    "        self.dropout = dropout\n",
    "        \n",
    "        self.vocab = vocab\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.output_size = output_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        \n",
    "        # Create a simple lookup table that stores embeddings of a fixed dictionary and size.\n",
    "        #    Use torch.nn.Embedding. Do not forget specify padding_idx!\n",
    "        # YOUR CODE HERE\n",
    "        self.word_embeddings = None\n",
    "        \n",
    "        if dropout is not None:\n",
    "            self.rnn = rec_layer(self.embedding_dim, self.hidden_dim, dropout=self.dropout, **kwargs)\n",
    "        else:\n",
    "            self.rnn = rec_layer(self.embedding_dim, self.hidden_dim, **kwargs)\n",
    "        \n",
    "        # Create linear layer for classification\n",
    "        # YOUR CODE HERE\n",
    "        self.output = None\n",
    "    \n",
    "    def forward(self, tokens, tokens_lens):\n",
    "        \"\"\"\n",
    "        :param torch.tensor(dtype=torch.long) tokens: Batch of texts represented with tokens.\n",
    "        :param torch.tensor(dtype=torch.long) tokens_lens: Number of non-padding tokens for each object in batch.\n",
    "        :return torch.tensor(dtype=torch.long): Vector representation for each sequence in batch\n",
    "        \"\"\"\n",
    "        # Evaluate embeddings\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "        # Make forward pass through recurrent network\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "        # Pass output from rnn to linear layer \n",
    "        # Note: each object in batch has its own length \n",
    "        #     so we must take rnn hidden state after the last token for each text in batch\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Исходный код LSTM](http://pytorch.org/docs/master/_modules/torch/nn/modules/rnn.html#LSTM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Допишите функции для обучения и оценки модели:\n",
    "\n",
    "**tip:**\n",
    "1. В функции `evaluate` при подсчёте метрик учитывайте, что батчи могут иметь разный размер. (в частности последний батч)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:13.969665Z",
     "start_time": "2021-04-02T00:05:13.875379Z"
    }
   },
   "outputs": [],
   "source": [
    "def train_epoch(dataloader, model, loss_fn, optimizer, device):\n",
    "    model.train()\n",
    "    for idx, data in enumerate(dataloader):\n",
    "        # 1. Take data from batch\n",
    "        # 2. Perform forward pass\n",
    "        # 3. Evaluate loss\n",
    "        # 4. Make optimizer step\n",
    "        # YOUR CODE HERE\n",
    "        pass\n",
    "    \n",
    "def evaluate(dataloader, model, loss_fn, device):\n",
    "    model.eval()\n",
    "    \n",
    "    total_loss = 0.0\n",
    "    total_accuracy = 0.0\n",
    "    with torch.no_grad():\n",
    "        for idx, data in enumerate(dataloader):\n",
    "            # 1. Take data from batch\n",
    "            # 2. Perform forward pass\n",
    "            # 3. Evaluate loss\n",
    "            # 4. Evaluate accuracy\n",
    "            # YOUR CODE HERE\n",
    "            pass\n",
    "        \n",
    "    return total_loss / len(dataloader.dataset), total_accuracy / len(dataloader.dataset)\n",
    "    \n",
    "\n",
    "def train(\n",
    "    train_loader, test_loader, model, loss_fn, optimizer, device, num_epochs\n",
    "):\n",
    "    test_losses = []\n",
    "    train_losses = []\n",
    "    test_accuracies = []\n",
    "    train_accuracies = []\n",
    "    for epoch in range(num_epochs):\n",
    "        train_epoch(train_loader, model, loss_fn, optimizer, device)\n",
    "        \n",
    "        train_loss, train_acc = evaluate(train_loader, model, loss_fn, device)\n",
    "        train_accuracies.append(train_acc)\n",
    "        train_losses.append(train_loss)\n",
    "        \n",
    "        test_loss, test_acc = evaluate(test_loader, model, loss_fn, device)\n",
    "        test_accuracies.append(test_acc)\n",
    "        test_losses.append(test_loss)\n",
    "        \n",
    "        print(\n",
    "            'Epoch: {0:d}/{1:d}. Loss (Train/Test): {2:.3f}/{3:.3f}. Accuracy (Train/Test): {4:.3f}/{5:.3f}'.format(\n",
    "                epoch + 1, num_epochs, train_losses[-1], test_losses[-1], train_accuracies[-1], test_accuracies[-1]\n",
    "            )\n",
    "        )\n",
    "    return train_losses, train_accuracies, test_losses, test_accuracies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создадим модель:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:39:36.315652Z",
     "start_time": "2021-04-01T21:39:33.667776Z"
    }
   },
   "outputs": [],
   "source": [
    "model = RNNClassifier(\n",
    "    embedding_dim=embedding_dim, hidden_dim=hidden_dim, output_size=10, vocab=vocab,\n",
    "    rec_layer=torch.nn.LSTM, dropout=None\n",
    ").to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создадим класс для подсчёта функции потерь и оптимизатор:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:39:38.737281Z",
     "start_time": "2021-04-01T21:39:38.711948Z"
    }
   },
   "outputs": [],
   "source": [
    "loss_fn = torch.nn.CrossEntropyLoss(reduction='mean')\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Попробуем обучить модель:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Сохраните все метрики и время работы модели. Это потребуется в конце первой части для построения графиков обучения и сравнения времени работы для всех моделей в этой секции**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:44:27.733745Z",
     "start_time": "2021-04-01T21:41:52.821042Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/15. Loss (Train/Test): 1.951/1.961. Accuracy (Train/Test): 0.278/0.284\n",
      "Epoch: 2/15. Loss (Train/Test): 1.819/1.858. Accuracy (Train/Test): 0.330/0.330\n",
      "Epoch: 3/15. Loss (Train/Test): 1.632/1.683. Accuracy (Train/Test): 0.366/0.354\n",
      "Epoch: 4/15. Loss (Train/Test): 1.541/1.624. Accuracy (Train/Test): 0.392/0.371\n",
      "Epoch: 5/15. Loss (Train/Test): 1.478/1.596. Accuracy (Train/Test): 0.412/0.386\n",
      "Epoch: 6/15. Loss (Train/Test): 1.437/1.594. Accuracy (Train/Test): 0.435/0.380\n",
      "Epoch: 7/15. Loss (Train/Test): 1.352/1.566. Accuracy (Train/Test): 0.470/0.396\n",
      "Epoch: 8/15. Loss (Train/Test): 1.299/1.571. Accuracy (Train/Test): 0.488/0.396\n",
      "Epoch: 9/15. Loss (Train/Test): 1.239/1.596. Accuracy (Train/Test): 0.515/0.388\n",
      "Epoch: 10/15. Loss (Train/Test): 1.186/1.602. Accuracy (Train/Test): 0.541/0.384\n",
      "Epoch: 11/15. Loss (Train/Test): 1.107/1.671. Accuracy (Train/Test): 0.572/0.380\n",
      "Epoch: 12/15. Loss (Train/Test): 1.046/1.721. Accuracy (Train/Test): 0.596/0.387\n",
      "Epoch: 13/15. Loss (Train/Test): 0.960/1.808. Accuracy (Train/Test): 0.634/0.377\n",
      "Epoch: 14/15. Loss (Train/Test): 0.866/1.830. Accuracy (Train/Test): 0.687/0.376\n",
      "Epoch: 15/15. Loss (Train/Test): 0.777/1.939. Accuracy (Train/Test): 0.726/0.363\n"
     ]
    }
   ],
   "source": [
    "train_losses_pure, train_accuracies_pure, test_losses_pure, test_accuracies_pure = train(\n",
    "    train_dataloader, test_dataloader, model, loss_fn, optimizer, device, num_epochs\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Нерегуляризованные LSTM часто быстро переобучаются (и мы это видим по точности на контроле). Чтобы с этим бороться, часто используют L2-регуляризацию и дропаут.\n",
    "Однако способов накладывать дропаут на рекуррентный слой достаточно много, и далеко не все хорошо работают. По [ссылке](https://medium.com/@bingobee01/a-review-of-dropout-as-applied-to-rnns-72e79ecd5b7b) доступен хороший обзор дропаутов для RNN.\n",
    "\n",
    "Мы реализуем два варианта DropOut для RNN (и третий дополнительно). Заодно увидим, что для реализации различных усовершенствований рекуррентной архитектуры приходится \"вскрывать\" слой до различной \"глубины\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Реализация дропаута по статье Гала и Гарамани. Variational Dropout. (1 балл)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Начнем с дропаута, описанного в [статье Гала и Гарамани](https://arxiv.org/abs/1512.05287).\n",
    "Для этого нам потребуется перейти от использования слоя `torch.nn.LSTM`, полностью скрывающего от нас рекуррентную логику, к использованию слоя `torch.nn.LSTMCell`, обрабатывающего лишь один временной шаг нашей последовательности (а всю логику вокруг придется реализовать самостоятельно). \n",
    "\n",
    "Допишите класс `RNNLayer`. При `dropout=0` ваш класс должен работать как обычный слой LSTM, а при `dropout > 0` накладывать бинарную маску на входной и скрытый вектор на каждом временном шаге, причем эта маска должна быть одинаковой во все моменты времени.\n",
    "\n",
    "Дропаут Гала и Гарамани в виде формул (m обозначает маску дропаута):\n",
    "\n",
    "$$\n",
    "h_{t-1} = h_{t-1}*m_h, \\, x_t = x_t * m_x\n",
    "$$\n",
    "\n",
    "Далее обычный шаг рекуррентной архитектуры, например, LSTM:\n",
    "\n",
    "$$\n",
    "i = \\sigma(h_{t-1}W^i + x_t U^i+b_i) \\quad\n",
    "o = \\sigma(h_{t-1}W^o + x_t U^o+b_o) \n",
    "$$\n",
    "$$\n",
    "f = \\sigma(h_{t-1}W^f + x_t U^f+b_f) \\quad \n",
    "g = tanh(h_{t-1} W^g + x_t U^g+b_g) \n",
    "$$\n",
    "$$\n",
    "c_t = f \\odot c_{t-1} +  i \\odot  g \\quad\n",
    "h_t =  o \\odot tanh(c_t) \\nonumber\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:14.068954Z",
     "start_time": "2021-04-02T00:05:13.971286Z"
    }
   },
   "outputs": [],
   "source": [
    "def init_h0_c0(num_objects, hidden_size, some_existing_tensor):\n",
    "    \"\"\"\n",
    "    return h0 and c0, use some_existing_tensor.new_zeros() to gen them\n",
    "    h0 shape: num_objects x hidden_size\n",
    "    c0 shape: num_objects x hidden_size\n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:14.131347Z",
     "start_time": "2021-04-02T00:05:14.071577Z"
    }
   },
   "outputs": [],
   "source": [
    "def gen_dropout_mask(input_size, hidden_size, is_training, p, some_existing_tensor):\n",
    "    \"\"\"\n",
    "    is_training: if True, gen masks from Bernoulli\n",
    "                 if False, gen masks consisting of (1-p)\n",
    "    \n",
    "    return dropout masks of size input_size, hidden_size if p is not None\n",
    "    return one masks if p is None\n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T21:09:12.282613Z",
     "start_time": "2021-04-01T21:09:12.256019Z"
    }
   },
   "source": [
    "Допишите класс-обёртку над LSTMCell для реализации Variational Dropout. **Используйте только цикл по времени**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Для каждого тензора в функции `forward` подпишите в комментарии его размеры**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:14.190066Z",
     "start_time": "2021-04-02T00:05:14.132804Z"
    }
   },
   "outputs": [],
   "source": [
    "class RNNLayer(torch.nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, dropout=None):\n",
    "        super().__init__()\n",
    "\n",
    "        self.dropout = dropout\n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "        \n",
    "        self.rnn_cell = torch.nn.LSTMCell(self.input_size, self.hidden_size)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # Initialize h_0, c_0\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "        # Gen masks for input and hidden state\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "        # Implement recurrent logic and return what nn.LSTM returns\n",
    "        # Do not forget to apply generated dropout masks!\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Протестируйте реализованную модель с выключенным дропаутом (слой `RNNLayer` надо передать в `RNNClassifier` в качестве `rec_layer`). Замерьте время обучения. Сильно ли оно увеличилось по сравнению с `torch.nn.LSTM` (LSTM \"из коробки\")?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Сохраните все метрики и время работы модели. Это потребуется в конце первой части для построения графиков обучения и сравнения времени работы для всех моделей в этой секции**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T22:04:19.104557Z",
     "start_time": "2021-04-01T21:50:02.992858Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/15. Loss (Train/Test): 2.009/2.015. Accuracy (Train/Test): 0.231/0.234\n",
      "Epoch: 2/15. Loss (Train/Test): 1.798/1.820. Accuracy (Train/Test): 0.332/0.328\n",
      "Epoch: 3/15. Loss (Train/Test): 1.629/1.682. Accuracy (Train/Test): 0.375/0.356\n",
      "Epoch: 4/15. Loss (Train/Test): 1.532/1.614. Accuracy (Train/Test): 0.403/0.377\n",
      "Epoch: 5/15. Loss (Train/Test): 1.459/1.574. Accuracy (Train/Test): 0.431/0.388\n",
      "Epoch: 6/15. Loss (Train/Test): 1.407/1.551. Accuracy (Train/Test): 0.446/0.395\n",
      "Epoch: 7/15. Loss (Train/Test): 1.358/1.545. Accuracy (Train/Test): 0.466/0.398\n",
      "Epoch: 8/15. Loss (Train/Test): 1.305/1.559. Accuracy (Train/Test): 0.487/0.401\n",
      "Epoch: 9/15. Loss (Train/Test): 1.231/1.590. Accuracy (Train/Test): 0.520/0.395\n",
      "Epoch: 10/15. Loss (Train/Test): 1.170/1.592. Accuracy (Train/Test): 0.551/0.393\n",
      "Epoch: 11/15. Loss (Train/Test): 1.108/1.641. Accuracy (Train/Test): 0.573/0.393\n",
      "Epoch: 12/15. Loss (Train/Test): 1.030/1.709. Accuracy (Train/Test): 0.614/0.376\n",
      "Epoch: 13/15. Loss (Train/Test): 0.942/1.757. Accuracy (Train/Test): 0.656/0.376\n",
      "Epoch: 14/15. Loss (Train/Test): 0.851/1.866. Accuracy (Train/Test): 0.693/0.369\n",
      "Epoch: 15/15. Loss (Train/Test): 0.803/2.034. Accuracy (Train/Test): 0.698/0.372\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Протестируйте полученную модель с `dropout=0.25`, вновь замерив время обучения. Получилось ли побороть переобучение? Сильно ли дольше обучается данная модель по сравнению с предыдущей? (доп. время тратится на генерацию масок дропаута)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T22:18:28.301613Z",
     "start_time": "2021-04-01T22:04:19.107850Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/15. Loss (Train/Test): 1.998/2.000. Accuracy (Train/Test): 0.242/0.245\n",
      "Epoch: 2/15. Loss (Train/Test): 1.825/1.840. Accuracy (Train/Test): 0.320/0.321\n",
      "Epoch: 3/15. Loss (Train/Test): 1.737/1.754. Accuracy (Train/Test): 0.343/0.347\n",
      "Epoch: 4/15. Loss (Train/Test): 1.623/1.655. Accuracy (Train/Test): 0.381/0.372\n",
      "Epoch: 5/15. Loss (Train/Test): 1.545/1.597. Accuracy (Train/Test): 0.400/0.384\n",
      "Epoch: 6/15. Loss (Train/Test): 1.491/1.556. Accuracy (Train/Test): 0.419/0.397\n",
      "Epoch: 7/15. Loss (Train/Test): 1.461/1.535. Accuracy (Train/Test): 0.424/0.403\n",
      "Epoch: 8/15. Loss (Train/Test): 1.425/1.530. Accuracy (Train/Test): 0.440/0.402\n",
      "Epoch: 9/15. Loss (Train/Test): 1.404/1.516. Accuracy (Train/Test): 0.441/0.409\n",
      "Epoch: 10/15. Loss (Train/Test): 1.373/1.505. Accuracy (Train/Test): 0.461/0.410\n",
      "Epoch: 11/15. Loss (Train/Test): 1.348/1.504. Accuracy (Train/Test): 0.470/0.410\n",
      "Epoch: 12/15. Loss (Train/Test): 1.332/1.497. Accuracy (Train/Test): 0.475/0.412\n",
      "Epoch: 13/15. Loss (Train/Test): 1.327/1.513. Accuracy (Train/Test): 0.472/0.412\n",
      "Epoch: 14/15. Loss (Train/Test): 1.284/1.495. Accuracy (Train/Test): 0.494/0.417\n",
      "Epoch: 15/15. Loss (Train/Test): 1.270/1.494. Accuracy (Train/Test): 0.500/0.414\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Реализация дропаута по статье Гала и Гарамани. Дубль 2 (1 балл)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<начало взлома pytorch>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "При разворачивании цикла по времени средствами python обучение рекуррентной нейросети сильно замедляется. Однако для реализации дропаута Гала и Гарамани необязательно явно задавать в коде домножение нейронов на маски. Можно схитрить и обойтись использованием слоя `torch.nn.LSTM`: перед вызовом `forward` слоя `torch.nn.LSTM` подменять его веса на веса, домноженные по строкам на маски. А обучаемые веса хранить отдельно. Именно так этот дропаут реализован в библиотеке `fastai`, код из которой использован в ячейке ниже.\n",
    "\n",
    "Такой слой реализуется в виде обертки над `torch.nn.LSTM`. Допишите класс:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:14.286206Z",
     "start_time": "2021-04-02T00:05:14.191730Z"
    }
   },
   "outputs": [],
   "source": [
    "import warnings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T02:06:54.953123Z",
     "start_time": "2021-04-02T02:06:54.917017Z"
    }
   },
   "outputs": [],
   "source": [
    "class FastRNNLayer(torch.nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, dropout=0.0, layers_dropout=0.0, num_layers=1):\n",
    "        super().__init__()\n",
    "\n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "\n",
    "        self.num_layers = num_layers\n",
    "\n",
    "        self.dropout = dropout\n",
    "        self.layers_dropout = layers_dropout\n",
    "        self.module = torch.nn.LSTM(input_size, hidden_size, dropout=layers_dropout, num_layers=num_layers)\n",
    "\n",
    "        self.layer_names = []\n",
    "        for layer_n in range(self.num_layers):\n",
    "            self.layer_names += [f'weight_hh_l{layer_n}', f'weight_ih_l{layer_n}']\n",
    "\n",
    "        for layer in self.layer_names:\n",
    "            # Get torch.nn.Parameter with weights from torch.nn.LSTM instance\n",
    "            w = getattr(self.module, layer)\n",
    "\n",
    "            # Remove it from model\n",
    "            delattr(self.module, layer)\n",
    "\n",
    "            # And create new torch.nn.Parameter with the same data but different name\n",
    "            self.register_parameter(f'{layer}_raw', torch.nn.Parameter(w.data))\n",
    "\n",
    "            # Note. In torch.nn.LSTM.forward parameter with name `layer` will be used\n",
    "            #     so we must initialize it using `layer_raw` before forward pass\n",
    "\n",
    "    def _setweights(self, x):\n",
    "        \"\"\"\n",
    "            Apply dropout to the raw weights.\n",
    "        \"\"\"\n",
    "        for layer in self.layer_names:\n",
    "            # Get torch.nn.Parameter with weights\n",
    "            raw_w = getattr(self, f'{layer}_raw')\n",
    "\n",
    "            # Generate mask (use function gen_dropout_mask)\n",
    "            # YOUR CODE HERE\n",
    "\n",
    "            # Apply dropout mask\n",
    "            # YOUR CODE HERE\n",
    "\n",
    "            # Set modified weights in its place\n",
    "            setattr(self.module, layer, masked_raw_w)\n",
    "\n",
    "    def forward(self, x, h_c=None):\n",
    "        \"\"\"\n",
    "        :param x: tensor containing the features of the input sequence.\n",
    "        :param Optional[Tuple[torch.tensor, torch.tensor]] h_c: initial hidden state and initial cell state\n",
    "        \"\"\"\n",
    "        with warnings.catch_warnings():\n",
    "            # To avoid the warning that comes because the weights aren't flattened.\n",
    "            warnings.simplefilter(\"ignore\")\n",
    "\n",
    "            # Set new weights of self.module and call its forward\n",
    "            # Pass h_c with x if it is not None. Otherwise pass only x\n",
    "            # YOUR CODE HERE\n",
    "            \n",
    "            pass\n",
    "            \n",
    "    def reset(self):\n",
    "        if hasattr(self.module, 'reset'):\n",
    "            self.module.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Протестируйте реализованную модель с выключенным дропаутом (слой `FastRNNLayer` надо передать в `RNNClassifier` в качестве `rec_layer`). Замерьте время обучения. Убедитесь, что модель выдаёт такое же качество, как и оригинальная реализация LSTM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Сохраните все метрики и время работы модели. Это потребуется в конце первой части для построения графиков обучения и сравнения времени работы для всех моделей в этой секции**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T22:25:23.843846Z",
     "start_time": "2021-04-01T22:22:43.059254Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/15. Loss (Train/Test): 2.028/2.031. Accuracy (Train/Test): 0.217/0.214\n",
      "Epoch: 2/15. Loss (Train/Test): 1.850/1.872. Accuracy (Train/Test): 0.312/0.314\n",
      "Epoch: 3/15. Loss (Train/Test): 1.621/1.680. Accuracy (Train/Test): 0.370/0.351\n",
      "Epoch: 4/15. Loss (Train/Test): 1.534/1.614. Accuracy (Train/Test): 0.395/0.376\n",
      "Epoch: 5/15. Loss (Train/Test): 1.459/1.581. Accuracy (Train/Test): 0.423/0.386\n",
      "Epoch: 6/15. Loss (Train/Test): 1.399/1.566. Accuracy (Train/Test): 0.443/0.393\n",
      "Epoch: 7/15. Loss (Train/Test): 1.345/1.558. Accuracy (Train/Test): 0.469/0.393\n",
      "Epoch: 8/15. Loss (Train/Test): 1.301/1.576. Accuracy (Train/Test): 0.483/0.397\n",
      "Epoch: 9/15. Loss (Train/Test): 1.251/1.589. Accuracy (Train/Test): 0.508/0.388\n",
      "Epoch: 10/15. Loss (Train/Test): 1.187/1.627. Accuracy (Train/Test): 0.532/0.395\n",
      "Epoch: 11/15. Loss (Train/Test): 1.094/1.643. Accuracy (Train/Test): 0.584/0.382\n",
      "Epoch: 12/15. Loss (Train/Test): 1.036/1.736. Accuracy (Train/Test): 0.612/0.368\n",
      "Epoch: 13/15. Loss (Train/Test): 0.952/1.784. Accuracy (Train/Test): 0.642/0.379\n",
      "Epoch: 14/15. Loss (Train/Test): 0.867/1.849. Accuracy (Train/Test): 0.689/0.367\n",
      "Epoch: 15/15. Loss (Train/Test): 0.765/1.968. Accuracy (Train/Test): 0.731/0.367\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Протестируйте полученный слой (вновь подставив его в `RNNClassifier` в качестве `rec_layer`) с `dropout=0.25`. Сравните время обучения с предыдущими моделями. Проследите, чтобы качество получилось такое же, как при первой реализации этого дропаута."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T22:28:38.168777Z",
     "start_time": "2021-04-01T22:25:56.717326Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/15. Loss (Train/Test): 1.993/1.998. Accuracy (Train/Test): 0.248/0.246\n",
      "Epoch: 2/15. Loss (Train/Test): 1.794/1.812. Accuracy (Train/Test): 0.333/0.331\n",
      "Epoch: 3/15. Loss (Train/Test): 1.705/1.731. Accuracy (Train/Test): 0.351/0.348\n",
      "Epoch: 4/15. Loss (Train/Test): 1.655/1.697. Accuracy (Train/Test): 0.364/0.353\n",
      "Epoch: 5/15. Loss (Train/Test): 1.587/1.635. Accuracy (Train/Test): 0.384/0.369\n",
      "Epoch: 6/15. Loss (Train/Test): 1.548/1.606. Accuracy (Train/Test): 0.394/0.381\n",
      "Epoch: 7/15. Loss (Train/Test): 1.507/1.578. Accuracy (Train/Test): 0.411/0.387\n",
      "Epoch: 8/15. Loss (Train/Test): 1.481/1.565. Accuracy (Train/Test): 0.420/0.394\n",
      "Epoch: 9/15. Loss (Train/Test): 1.454/1.560. Accuracy (Train/Test): 0.425/0.385\n",
      "Epoch: 10/15. Loss (Train/Test): 1.417/1.533. Accuracy (Train/Test): 0.440/0.399\n",
      "Epoch: 11/15. Loss (Train/Test): 1.384/1.519. Accuracy (Train/Test): 0.455/0.408\n",
      "Epoch: 12/15. Loss (Train/Test): 1.366/1.521. Accuracy (Train/Test): 0.465/0.406\n",
      "Epoch: 13/15. Loss (Train/Test): 1.350/1.517. Accuracy (Train/Test): 0.466/0.412\n",
      "Epoch: 14/15. Loss (Train/Test): 1.316/1.504. Accuracy (Train/Test): 0.484/0.411\n",
      "Epoch: 15/15. Loss (Train/Test): 1.292/1.504. Accuracy (Train/Test): 0.494/0.412\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "</конец взлома pytorch>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Реализация дропаута по статье Семениуты и др. (1 балл)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Перейдем к реализации дропаута для LSTM по статье [Semeniuta et al](http://www.aclweb.org/anthology/C16-1165). \n",
    "\n",
    "Этот метод применения дропаута не менее популярен, чем предыдущий. Его особенность состоит в том, что он придуман специально для гейтовых архитектур. В контексте LSTM этот дропаут накладывается только на информационный поток (m_h - маска дропаута):\n",
    "$$\n",
    "i = \\sigma(h_{t-1}W^i + x_t U^i+b_i) \\quad\n",
    "o = \\sigma(h_{t-1}W^o + x_t U^o+b_o) \n",
    "$$\n",
    "$$\n",
    "f = \\sigma(h_{t-1}W^f + x_t U^f+b_f) \\quad \n",
    "g = tanh(h_{t-1} W^g + x_t U^g+b_g) \n",
    "$$\n",
    "$$\n",
    "c_t = f \\odot c_{t-1} +  i \\odot g \\odot {\\bf m_h} \\quad\n",
    "h_t =  o \\odot tanh(c_t) \\nonumber\n",
    "$$\n",
    "На входы $x_t$ маска накладывается как в предыдущем дропауте. Впрочем, на входы маску можно наложить вообще до вызова рекуррентного слоя.\n",
    "\n",
    "Согласно статье, маска дропаута может быть как одинаковая, так и разная для всех моментов времени. Мы сделаем одинаковую для всех моментов времени.\n",
    "\n",
    "Для реализации этого дропаута можно: \n",
    "1. самостоятельно реализовать LSTM (интерфейса LSTMCell не хватит) \n",
    "2. снова воспользоваться трюком с установкой весов (но тут мы опираемся на свойство $tanh(0)=0$, к тому же, трюк в данном случае выглядит менее тривиально, чем с дропаутом Гала). \n",
    "\n",
    "Предлагается реализовать дропаут по сценарию 1. Допишите класс:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Для каждого тензора в функции `forward` подпишите в комментарии его размеры**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:14.447201Z",
     "start_time": "2021-04-02T00:05:14.350457Z"
    }
   },
   "outputs": [],
   "source": [
    "class HandmadeLSTM(torch.nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, dropout=0.0):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.dropout = dropout\n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "        \n",
    "        self.input_weights = torch.nn.Linear(input_size, 4 * hidden_size)\n",
    "        self.hidden_weights = torch.nn.Linear(hidden_size, 4 * hidden_size)\n",
    "        \n",
    "        self.reset_params()\n",
    "\n",
    "    def reset_params(self):\n",
    "        \"\"\"\n",
    "        Initialization as in Pytorch. \n",
    "        Do not forget to call this method!\n",
    "        https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html#LSTM\n",
    "        \"\"\"\n",
    "        stdv = 1.0 / np.sqrt(self.hidden_size)\n",
    "        for weight in self.parameters():\n",
    "            torch.nn.init.uniform_(weight, -stdv, stdv)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # Use functions init_h0_c0 and gen_dropout_masks defined above\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "        # Implement recurrent logic to mimic torch.nn.LSTM\n",
    "        # Do not forget to apply dropout mask\n",
    "        # YOUR CODE HERE\n",
    "    \n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Протестируйте вашу реализацию без дропаута (проконтролируйте качество и сравните время обучения с временем обучения `torch.nn.LSTM` и `RNNLayer`), а также с `dropout=0.25`. Сравните качество модели с таким дропаутом с качеством модели с дропаутом Гала и Гарамани."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Сохраните все метрики и время работы модели. Это потребуется в конце первой части для построения графиков обучения и сравнения времени работы для всех моделей в этой секции**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T23:01:53.046622Z",
     "start_time": "2021-04-01T22:30:08.733475Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/15. Loss (Train/Test): 1.999/2.003. Accuracy (Train/Test): 0.245/0.244\n",
      "Epoch: 2/15. Loss (Train/Test): 1.781/1.798. Accuracy (Train/Test): 0.335/0.337\n",
      "Epoch: 3/15. Loss (Train/Test): 1.648/1.692. Accuracy (Train/Test): 0.363/0.351\n",
      "Epoch: 4/15. Loss (Train/Test): 1.536/1.606. Accuracy (Train/Test): 0.400/0.375\n",
      "Epoch: 5/15. Loss (Train/Test): 1.482/1.585. Accuracy (Train/Test): 0.423/0.382\n",
      "Epoch: 6/15. Loss (Train/Test): 1.422/1.571. Accuracy (Train/Test): 0.442/0.390\n",
      "Epoch: 7/15. Loss (Train/Test): 1.362/1.557. Accuracy (Train/Test): 0.463/0.395\n",
      "Epoch: 8/15. Loss (Train/Test): 1.314/1.572. Accuracy (Train/Test): 0.482/0.384\n",
      "Epoch: 9/15. Loss (Train/Test): 1.257/1.587. Accuracy (Train/Test): 0.510/0.387\n",
      "Epoch: 10/15. Loss (Train/Test): 1.180/1.617. Accuracy (Train/Test): 0.536/0.394\n",
      "Epoch: 11/15. Loss (Train/Test): 1.126/1.636. Accuracy (Train/Test): 0.569/0.387\n",
      "Epoch: 12/15. Loss (Train/Test): 1.053/1.683. Accuracy (Train/Test): 0.596/0.386\n",
      "Epoch: 13/15. Loss (Train/Test): 0.967/1.725. Accuracy (Train/Test): 0.637/0.382\n",
      "Epoch: 14/15. Loss (Train/Test): 0.878/1.844. Accuracy (Train/Test): 0.677/0.374\n",
      "Epoch: 15/15. Loss (Train/Test): 0.798/1.926. Accuracy (Train/Test): 0.715/0.363\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T23:33:28.808063Z",
     "start_time": "2021-04-01T23:01:53.049547Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/15. Loss (Train/Test): 2.031/2.032. Accuracy (Train/Test): 0.216/0.213\n",
      "Epoch: 2/15. Loss (Train/Test): 1.857/1.873. Accuracy (Train/Test): 0.317/0.320\n",
      "Epoch: 3/15. Loss (Train/Test): 1.688/1.724. Accuracy (Train/Test): 0.351/0.352\n",
      "Epoch: 4/15. Loss (Train/Test): 1.632/1.677. Accuracy (Train/Test): 0.363/0.344\n",
      "Epoch: 5/15. Loss (Train/Test): 1.551/1.610. Accuracy (Train/Test): 0.394/0.374\n",
      "Epoch: 6/15. Loss (Train/Test): 1.524/1.586. Accuracy (Train/Test): 0.404/0.379\n",
      "Epoch: 7/15. Loss (Train/Test): 1.486/1.562. Accuracy (Train/Test): 0.416/0.393\n",
      "Epoch: 8/15. Loss (Train/Test): 1.455/1.552. Accuracy (Train/Test): 0.427/0.398\n",
      "Epoch: 9/15. Loss (Train/Test): 1.434/1.552. Accuracy (Train/Test): 0.435/0.397\n",
      "Epoch: 10/15. Loss (Train/Test): 1.402/1.522. Accuracy (Train/Test): 0.444/0.405\n",
      "Epoch: 11/15. Loss (Train/Test): 1.391/1.527. Accuracy (Train/Test): 0.450/0.408\n",
      "Epoch: 12/15. Loss (Train/Test): 1.369/1.526. Accuracy (Train/Test): 0.459/0.405\n",
      "Epoch: 13/15. Loss (Train/Test): 1.360/1.522. Accuracy (Train/Test): 0.463/0.414\n",
      "Epoch: 14/15. Loss (Train/Test): 1.351/1.528. Accuracy (Train/Test): 0.466/0.410\n",
      "Epoch: 15/15. Loss (Train/Test): 1.327/1.520. Accuracy (Train/Test): 0.472/0.417\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T23:33:28.831346Z",
     "start_time": "2021-04-01T23:33:28.810453Z"
    }
   },
   "source": [
    "## Сравнение всех предложенных моделей. (1 балл)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T23:48:05.361634Z",
     "start_time": "2021-04-01T23:48:05.333901Z"
    }
   },
   "source": [
    "Используя замеры времени заполните табличку с временем работы четырёх реализованных моделей в следующей ячейке:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| torch.nn.LSTM | RNNLayer | FastRNNLayer | HandmadeLSTM |\n",
    "|---------------|----------|--------------|--------------|\n",
    "| 2m 35s        | 14m 16s  | 2m 41s       | 31m 44s      |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:14.722913Z",
     "start_time": "2021-04-02T00:05:14.448857Z"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Крайне желательно рисовать графики в векторном формате. \n",
    "\n",
    "Если по каким-то причинам, отрисовка не будет работать, закомментируйте следующую ячейку."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:05:14.745238Z",
     "start_time": "2021-04-02T00:05:14.724188Z"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from IPython.display import set_matplotlib_formats\n",
    "\n",
    "set_matplotlib_formats('pdf', 'svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Нарисуйте два графика -- функция потерь и качество на обучающей и тестовой выборке для всех 7 моделей обученных выше."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T23:40:37.809057Z",
     "start_time": "2021-04-01T23:40:36.694896Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjEwIDAgb2JqCjw8IC9Bbm5vdHMgWyBdIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDcxMi43MDYyNSA3MTEuMjY4NzUgXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMSAwIFIgPj4Kc3RyZWFtCnicvVxNsxvZbd3zV3CZbFoXuN/LkT2eVCqbOKp44cpC9STPeOyZsSUnU/73Obgk++Ki2Ww2RT1X2fUEN0E0zj346AZIxx8Pb76h4/efj+74I/7765GO3x3f/Pbj//355ePvv3t7fPl8cJD/dMjEU3aJI/71V/WvTDRxKjlC6oZ//XA4/HyAbnzgO6j9/nDwbqL2IV95ouBxGRS7OEUj/auSplKmepbOCrQQ3/Onw9+PV5SnHKdw5Jqmcvz08fiH48/HN9+w3C0f/x0m4Y7x//yK/03VUU0ul3Cwt59wzcLoWagNOfzX4T+Pf798hcOdq6+ADyDZ0OahLV20vYXzfj28fXd88zs6kju++9Mh5ilxiu3SnKcYj+8+HP7F/evx3Y/Hb9+1b3/cDQfthsUpoFCmyC76NJiuxHtdsa1xyx0U4uSDK6EwPq1dwq/hEvZpYi9fPtyAEu91ybbGLZew95OLkTw+X5J2SXgNl3j2uOFKLg83oMR7XbKtccslnt2UUw2wOXDVLkmv4ZKAkJgS8PDDDSjxXpdsa9xyCTgzheIRZH3MpF1SXsUltU74zpTreANdvNslmxo3XVJBHCreiUeCdgm9SnyNJU+uyhkd7kCJ9/pkW+NmuilgjudYEEtiHnzyKgE25XAtW87S3cl3Q9+WP1ICbUIrb7Qv5sg6hp3IEw6VXJv8VHIhPl3/7d9+efnhVYLxRW8gnO2cQvRDvaXEd8biqwo5TemKviv+yxMfg6sT5xpzroQz1eqX6VWiTrc+RljsaqmjO7p4rzu0wu6OQd+aOyLh8CF0Bxczn87T9CoBp1tfy4SzGXmsxpV4rzu0wu6OQd+aO2qYPAqXCLNLObvjVWLNrDcGXBedSzS4Q4l3umNQOLtj1LfiDhTBU4meSkKZG87ueJ1Cbra++IlyiDWP7ujive7QCrs7Bn1r7sgtnWfmUKie3fEqRVxvPBmkrpV8GNyhxDvdMSic3THqW3FHYpqqY+TnxP6ci145lEpGZLQn4+Ho0r3OUOq6L7S2NVekMAVGnRdyOneAT4iiNMWTlYfx9lX7j468uEixSOfqwlzSc0XlVGqSYKc6WpRSU80lsqRIginn6jRm+JtriZIrVLsXoR9/JrmcUY2dC7coGimTk+ChWqGI2wwhUJbr4bGzlZEqyCUfgFi1CUBrwsWliZGtzkaGGuGiUlwReS+hQy6Tj5QoQR6DvxgZEtrwQsmzyHt5GYL8mVwReYJpZysDZ4BRo/imF1/SkyDQkVcB9gtQo6M8t/r9DejgcvKZc7DQRRQoKAXLEjpE4dTM1tDheGbCp6KFDhGthmKRw434EtgCR1BSq6sjcGhdQ3TVZQtckO8sXCxyUIMCNcQlchHdr68WOUSQXFOqFjkvj/FKDhY5WMyOaUQOdR96Js/PQU5wo5vIwT+5wocWOTjRUciNFfrBFAEs56uzyKHcdwl3ZkmXkWQLGkEDXQR3a8mtgNHYAejEzjWnK+xCnKo8qysGu4DbdA6F0Ihd9AHRvHKoI3YRoSRWnxqLFHYRAZIqOG2wQ4GFOqsgdo7YIUhNCDDFD9hFqdnxSfLPwQ55I17Shm8EPK4TMCAqIYhFC2OavHc4f5aAeUq+pnYoBwIiugWKDfY7CAhK+RwaFTSKjF4yxkJhETpRxIcW8wYG4vKEvpgtAxELkzyNNATEFegbG0UGAuLPCJuSJSDU1EjLwOmZT+FdR04UC0gSsT4FQ5LcDxaBHlJW3MYwJeQ5RC+yQVTyMf40GCaYCrcg5hgMUfcXBJxggyiACzghzibAjLsiXFEMivBnQFHdgrdCUdJ0BrUsF6NEYwTSZLgYAuJfgBcMFz3SMUJD06O56PG17IqlIlBwVCpnQ0XmKZeaYzJcxAGHF4ifgaNl4vDPm5AiKhKwzBZSuA2BqEGhaVk9sr8LzYeGlhGhOiRLS+gvkRoVBmLiyDM+azMjT6iOfLWZEV+U0D3bkoYcEMKdLjKjPIPxLrMhZoGeSqEhPRAT7kUFXskSEzdepMay1EScSIRq3KRGhtOIWyn1jNRIG6lRvplCzd7GVDgO1RUny0d0DOgwq61HdZ06pEac3IiSLhnwAIyjyC02D7kRYdLH1I605iO+CUVGK3eG3Cg8rextbkTOdBl0WvARtZccg2z5GCYn8bZYQorHkQlNXSNEZU+mIh0y6SPgdeD+eIzyuJBx4f8gSX5orc96UwHne1SJi8QI5ng6VQgDiPgkcys0THkDPsVWaw4xFZdwoOZMjWHBWailJNtU4P5qgJo0YpjhVhA5GAi1dg0hGvyI8slmxljFSndqNhSESRIsAlYwTUVCDkSEJ4ojhOg+cETyKX+rR8SwgPH95StgeGkxblQ4KCTJu0VyBL7SM+YljgkNUzvNA5Co9FCokU2OQECSlLMljibpAGSvazWOSNWIsMnmRhwHJy+JyAIJLSDdqSLSQMrxYeJsgUQ5F90pwCocKcGbyTeKDjhK54x+f+wxGkCo0cKjPcZNLp4ajpsp8fJYYuAjqgQXXFj0GyAMlXB3jYMMB+VcLYxIWfi8rXDgccRDNi2+mFhPDwQ0iNKaE5KTDaiJEd9R5lg2Qp4TL3oN9ImI7SGZbIiecko5SSc2xlNxXcBhMAUOxKjuY3202bgN4q7O42qWlDTj0A9fISbwDC2L3UVMdCQ+5GQj7BBJNTFxkFBT2A4SXbtDm9/epwzMxI14fKhYZiJkIreRJSYMRs/QugZNTNQmMOwMto6wjKOUT0+LBmbibkne74zMTBLkKk7HV4mwO1qRlacCaOw4ulMWH1gKsLlQMjRNzoue2mp8DWqFLYTwZOvWIvHTeW9LH6AXaiq2cIUfEN9QMBtQs1hcmzUDUYEpl5KLJSqOUmT7KA4Ft1yx6Cfl8sip2qyp+a552qXPpunjTQkKE9R79z2sS46lL0WwtYxVYBnGVpS7ccHYXuQYwqISK2zKWlyN9pwSLwk7d7EDYZP0gVQWNZEUdKh+gk2lvdDThHVImZGD95awEVVwiH5EN8mL6VJj/SqplDZSKbQ6zrGmZehleKUsalt0kb62R9RDLoU4pBgsjgX5DoC1hDSQFDeIhsY+LhBr8Le3pS2yYyp50Z4Mck1S1DLVOV6QFAValPxrWYoaNri4eHSn9QwshWUoC1MwNEVARngsy/f30jb2dywrL2+Geaprk5XQeW0686eV6UxcvmPCU1/dtdzQ7eSezu+PqN3Z9/OLIG4TFyRFTZuzSKi1Th+Vd0G/+fTL58/f/vyPT7/87Z/H/8DflzdDb77xpwP9/Wn+FR67jKrm+UZyH03EMfZt+mMW4lQi/2U0+N4rMUJAv7LgnFPWsnq+7OUwCyXtnF4tdAmAhlooVt8hT9CifIG7XNX+/dItPktwoJFpUFXiq88iCC6XzKqVzJ8VxfnW63yXSjjfUNffb/yK315kQvftYe3dmj2d9fKeGApCG3X96RCXQnMkxuNwmtbJ84dA+PYhOQxv33/+eMG/GTO8Llq1psgju8sI1RXpFXu+u2UTEg6X+YT+9/tPv8Xp/M0PH1/+sjRupUHpxiE7EhlPXWR7DZOS/jKqdDbrqkUb1XY3LnoJdhbILt1rYECqPn1STPzd+8//uO297SKy2yrPk9p8lzZ1Fu61FAQ5DcMbQ6/4c09Z1M3lIC9Yrb1dutdgxhfX2eJ/e//zh5/ef/i4eiqv5vpuHVJ3MKifRXvtQviqPBp1sedaIXL8sGaTzBdlZOTRLCU9W3ay6o9nZeuWyQtXPn3SxpYVw7YCTZCipVgDZ+GKfbe8hz50isVQesB0xdStsCPv/YOrJiYq6SPGyhvp06evBKB1Q++NRij+0RgYiy+yR+wNMqCxpPg9/t0RmAJKXJdsNlTSR0yXNyspXgmkW+5+KFgFpskvDsssfOQGCKXIfFSuRqvVA3M7dCG9oN4iE1iV9BFrUV/x6dPX4th5DWsuRc9B0exNZXQ9i6L6ItRbU+hwJ2+3prSwbU2Nrd9Fz5PXprrNXaYM2bc1dU0Z0t19O1OM/uDmytRTPaAXmbrZWrrTC9sKb3vCrEt1b6wM2D7VG3qHqRuvpTu9sa3wtjfMplT3xsp87VO9odeXuvFautMb2wpve8MsSXVvrIzXPtUbenOpG6+lO72xrfC2N8x+VPfGynTtc72hlpaU8Uq61xubCje8Ma5GdW+sbUY91R16X6lbr6U73bGtcCOhjFtRyh2vEUfVtpLKhV24N7NuqLvtCrUQpdywtQ8lC0D3bEM9N+LO2yky2uUpjcsuF+HeVZeurC+6KF2rS1BTSIlltaOG8w7UaxyeWW1Np0luHrzQpTvdoNXNfhi0rTiiogWJAc1XcLmeHeFf0xFSPQWZGBz3OZR4pysGhbMvRn0rziCcnBQ8FaTs5M/eeJVq5KKWHZo5HEsa9zmUeKc3BoWzN0Z9a96oMlpKKVBExXr2RnxVbyBiJWrvZAdvdPFeb2iF3RuDvhVvMA5Qgd+qT8g9Z2+8Rm3W1crOgYvo2Mfus4t3emNQOHtj1LfiDY8DVOW1bMyJLkuk+Uu9sb3uU0IbcYpmboSI2ounYN5CE6zIfB4iVW8vheiM/6OYEQOKQaBI2UwCUYJ3MrnTzE+v6amENiPX3o7215fIKTL4V515Dc2oar3s+4Tx9SWzDIGGXMxsLLd5Q2ZnXl9yBBFRaHgzDcTZyQSvC2bfh2ueqrRZ43toBDoZpg50dTZ2L2zb+z6nd/DFQie1ba2ULXRRVqNOI/gaOty7c5zJQpeCDPiHYpZGKLOMK+bIFjqWgdzTAJaGzk855vP4soKOxJuJ7VQls3jfJ85L6GL0rmQLXW2PV6OZ+eFCU5a1WwOdd4QmGcjVETrnxbIUr06I7Gfc5sLPRMF5M0ZJ8jguLQfSCecqI4MGM0ZJ4pOAwt2s+wy5WUMXQRefmgs1ckm8Q1wNcknK2nia4VLIiTwjnZvRD8L3MG7KUG5gtMKNABXibWhTQwo3KnF+Ra5gI3lNCuq6EbZrefwLYNux6yOrOFSoZgshUp9j5gX7hB04pWY2S55gFS+bbJZ9TvYvFpOwlGVSrrhaLfuCbCy6mgz7AFv2wZJPDK7xNLaoyUd5wtcUNgN3LOG9yGiZJZ8MMeXTCtYQNwM6I3xhtXGzyJBmquMorDyqKaujsLvj5p5tHxzxFBNZFBFDI3zfDrNGUXpCStkuu5LsShQZZLJElLuItXUMGsXQhlhztExEugS9ch1XtoS3aMxzXDCR0HUBNDM2OTJXcVFmv0rJ5C0X23xktMsFQ0gfyMiyh+bN9M7A0S8no6LiYcdcXUaxxTUFi6hkM46ZLKIVHs/e29A6hND7eOmRGilYXsrr1JhLXfCSIgJ3tMSUWcrsE1tiynCF89GsUoplAUwO1RJzraCRwgiFebLEZNR10eVsChp0s1RSeQ6iW7s+aBtiRYdki1E4hV1KdkvkFhtnTAc2KtYNaVGSiE/OFqMR6Tijbs8mL8JM52vIlo2qptVshM99LNXORJIEyRK42swos9aZOZJlozpKmo0DSzsbVSJ9ALq1kcjjh8OtTR9ZQCbZkLuvnZClylx9XhQ2HoVKCOQtgqBNyKmYiWUphJB0vV3WoiBPbFDUmayoxRpBWWUv9RQ2FYA6imsA9bdqAINoLDGZkWUSsuKAsxlDl8q8+lL8uFswHOPnIri95yN4wZGLKAqjffJ5wURxFmJssjiqOx5w7I42aXEuhgYY8+Rr4RoNjLKxLNOuFkalZsARjTgcnS0RtVzjqHEZcNR4aRwljXIKJi3iMJeIyu/BrvAmE7f2fFayocuTTGPbfS2SQsyjjY1LPs63cFd9g9tA6CSzeDf0mhrIikhIxZsNApIiG2rZElJ+nyCUaqpULdYw6mcQGkYt1zDKL5SgTXZjkUqxrTmE8FVgvL/vgA78JywQlXv3tSaL6EqO1Gf4ngirOTJQU8uHCDsfgHsirGasjrBVfs/i9OMDAzPR3GfKC2JKT0nBAuolcbrTlIvmpRI/Pb7e34K0TTx5IGxCrbxcqs7bZwEyXyQ/r2AbScEkyoqaAVQHJZMy52cHA6AqeWlAVe00ACo/8wMDskU049xVKpai0h5nCguOkkxu+cz3clQ1Spqj6vInc/TBNR801IXlVZBBN8JtoQW8O9k6P7sbwFX8GNgqnsVRWYCrE6wCVxc4Glx1Fgxd+2HQiVQXPmNBNAeVga/K/IGw8lQ6uzR2I1+xINrc8sH5dKcfJLnviR18DmuSs4Wt9HEyhLEAUt3xwNIe0e6JuqpOGnDU0VjnUVmIRkazON5g6Xz9wFKVGgeW9oSsSary6yNLPmoa6Noc4Ljjc3W+cLz67mHE4dpZxw3N2ws+7bd7qM0luPMn5d3PNy8v//vp/cs/d6z1SO1WzVbPLBuWembpvJgjSSXJTwJlLePTVWqlx8vP9cxbQrOwoO/NUK23ejzX86V9x2gWqd2eLtTrPV3a13T61yhZ7hrj7Ai+3LSSzTfYv2T2w9KHD274eJaSQk/YniU7dns8oWb/ksUej5gUsx24V9KduwryasTlm+P29y/2wPFTbX8p27pwp2lcytQXZ56x28O5/TyhGepW0r0WZo/QOPP6qbs9LA+irCsvsr12RknF/NUWe1h+B2cJfJfutRcZ0NfZ4i9c7JHJ2cVwfBfutU1+KnN1LH7feg8MkLVga1qX7lzvYRSeLswLUU9Y72Env0M1mncWPbBTIC/0bq0QPrjYg55zYme3pJT0AVPlAWt2fmWz8MsXe+T58GKxqwsfsTjT1A3+ers9BHNLtJsxSvqI7VGe2dYrgfSr7PZIi2MDwix7xHxpjW6uIT642CMTOZFsGlDSR4zl9ttnNrjO00iH/wdwYO2dCmVuZHN0cmVhbQplbmRvYmoKMTEgMCBvYmoKNTE0MQplbmRvYmoKMTYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4OCA+PgpzdHJlYW0KeJw1jLsNwDAIRHumuBH4OID3iVKR/dsQWy64e9IT5znAyD4PR+jELWRD4aZ4STmhPlE0Wm86tkhzv7xkMxcV/TPcfYXt5oBkNEnkIdHlcuVZKXo+l9obfgplbmRzdHJlYW0KZW5kb2JqCjE3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjU5ID4+CnN0cmVhbQp4nD1SSXLDMAy7+xV8AndJ70mnJ+f/1wL01JcQY4oACKZ3i8rZ+GkrWX7kxy47SyK3fAdtk/uKqgekboLMJcclV4hZyudKkFi0lJlYn6kOys81qFX4wk/MTEyHJBFLyEk9irB+Xgf3oH69pPNbZPyjY4PSW5JMBZ0a7pWjlNiwVJ8a5B403ZRWn5l2ZQcsjTdkpSZ1+vXT4wdLMKUvEtFBiMSf5HKHLAspB/F2rt5AeFe7INVTKw87RJqPkd5SBas5MwFLRtMOI2Bt1pggVQK8RWsItBbi8oVOKyI+OTd0ME1VnoXIsFLhWHvPaA0Xuoa80nB6knH/+R+8K97X7x/a82ZkCmVuZHN0cmVhbQplbmRvYmoKMTggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzIgPj4Kc3RyZWFtCnicNVE7cgUxCOt9Cl0gM+Zvn2czr0ru30awk2ZhAQkJ5z3YiMSXGNId5YpvWZ1mGX4ni7z4WSmcvBdRgVRFWCHt4FnOaobBcyNT4HImPsvMJ9NixwKqiTjOjpxmMAgxjetoOR1mmgc9IdcHI27sNMtVDGm9W6rX91r+U0X5yLqb5dYpm1qpW/SMPYnLzuupLe0Lo47ipiDS4WOH9yBfxJzFRSfSzX4z5bCSNASnBfAjMZTq2eE1wsTPjARP2dPpfZSG1z5our53L+jIzYRM5RbKSMWTlcaYMVS/Ec0k9f0/0LM+f5owVEcKZW5kc3RyZWFtCmVuZG9iagoxOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJw9kEsSwyAMQ/ecQkfA+H+edLpK7r+tDZ1ssBiE9MB9YiKjFieCr8SHBqXDJPBsFYR7MNkRcoTkBE2GsoMkcQ0NBqXCpmOZ78mmddJKrLzRftl3NGaddIotRYd2If/n9SLco+Aa6xk8D2AxyNpKpeyZMFplpq7yqOi1H9PhPQ9Eq8Xl9Qau8NpHN6koKkvq/kR3NNj+kbf7Ht8fmWU4JAplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzkgPj4Kc3RyZWFtCnicTc27DcAgDATQnik8AuD/PlGqsH8bGyJCYz/pTjrBDhXc4rAYaHe4WvGlUZh96pkSklBzPURYMyU6hKRf+ssww5jYyLbvt1buF94bHBkKZW5kc3RyZWFtCmVuZG9iagoyMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDc0ID4+CnN0cmVhbQp4nDM1N1UwULC0ABKmhuYK5kaWCimGXEA+iJXLBRPLAbPMTMyALENLZJaJsSGQZWJhhsQyNrGAyiJYBkAabE0OzPQcrjQAA3EYkwplbmRzdHJlYW0KZW5kb2JqCjIyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzcgPj4Kc3RyZWFtCnicMzc1UjBQsLQAEmamJgrmRpYKKYZcQD6IlctlaGkOZuWAWSbGBkCWqakpEgsiC9MLYcHkYLSxiTnUBAQLJAe2NgdmWw5XGgCe4BuaCmVuZHN0cmVhbQplbmRvYmoKMjMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA1OSA+PgpzdHJlYW0KeJwzNTVXMFCwtAASpqZGCuZGlgophlxAPoiVy2VoaQ5m5YBZFsZABkgZnGEApMGac2B6crjSAKnhEFoKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDcwID4+CnN0cmVhbQp4nDOzMFEwULAAYjNzMwVzI0uFFEMuIwszoEAulwVYIIfL0NAQyjI2MVIwNDQFskzNjaFiMI1AWUuQQTlQ/TlcaQBPVBIvCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDQgPj4Kc3RyZWFtCnicPZI7ksMwDEN7nYIXyIz4k+TzZCeV9/7tPjLJVoBJiQAoL3WZsqY8IGkmCf/R4eFiO+V32J7NzMC1RC8TyynPoSvE3EX5spmNurI6xarDMJ1b9Kici4ZNk5rnKksZtwuew7WJ55Z9xA83NKgHdY1Lwg3d1WhZCs1wdf87vUfZdzU8F5tU6tQXjxdRFeb5IU+ih+lK4nw8KCFcezBGFhLkU9FAjrNcrfJeQvYOtxqywkFqSeezJzzYdXpPLm4XzRAPZLlU+E5R7O3QM77sSgk9ErbhWO59O5qx6RqbOOx+70bWyoyuaCF+yFcn6yVg3FMmRRJkTrZYbovVnu6hKKZzhnMZIOrZioZS5mJXq38MO28sL9ksyJTMCzJGp02eOHjIfo2a9HmV53j9AWzzczsKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIzMCA+PgpzdHJlYW0KeJw1UUluwzAMvOsV84EA4i6/x0FP7f+vHdIJYGBoS5zNERsbEXiJwc9B5MZb1oya+JvJXfG7PBUeCbeCJ1EEXoZ72QkubxiX/TjMfPBeWjmTGk8yIBfZ9PBEyGCXQOjA7BrUYZtpJ/qGhM+OSDUbWU5fS9BLqxAoT9l+pwtKtK3qz+2zLrTta0842e2pJ5VPIJ5bsgKXjVdMFmMZ9ETlLsX0QaqzhZ6E8qJ8DrL5qCESXaKcgScGB6NAO7Dntp+JV4WgdXWfto2hGikdT/82NDVJIuQTJZzZ0rhb+P6ee/38A6ZUU58KZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIyNyA+PgpzdHJlYW0KeJw1TzuyAyEM6zmFLpAZjG1gz7OZVC/3b59ksg0S/kjy9ERHJl7myAis2fG2FhmIGfgWU/GvPe3DhOo9uIcI5eJCmGEknDXruJun48W/XeUz1sG7Db5ilhcEtjCT9ZXFmct2wVgaJ3FOshtj10RsY13r6RTWEUwoAyGd7TAlyBwVKX2yo4w5Ok7kiediqsUuv+9hfcGmMaLCHFcFT9BkUJY97yagHRf039WN30k0i14CMpFgYZ0k5s5ZTvjVa0fHUYsiMSekGeQyEdKcrmIKoQnFOjsKKhUFl+pzyt0+/2hdW00KZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0NSA+PgpzdHJlYW0KeJxFULuNQzEM6z0FFwhg/Sx7nndIldu/PUpGcIUhWj+SWhKYiMBLDLGUb+JHRkE9C78XheIzxM8XhUHOhKRAnPUZEJl4htpGbuh2cM68wzOMOQIXxVpwptOZ9lzY5JwHJxDObZTxjEK6SVQVcVSfcUzxqrLPjdeBpbVss9OR7CGNhEtJJSaXflMq/7QpWyro2kUTsEjkgZNNNOEsP0OSYsyglFH3MLWO9HGykUd10MnZnDktmdnup+1MfA9YJplR5Smd5zI+J6nzXE597rMd0eSipVX7nP3ekZbyIrXbodXpVyVRmY3Vp5C4PP+Mn/H+A46gWT4KZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM5MiA+PgpzdHJlYW0KeJw9UktuBTEI288puECl8E1ynqne7t1/W5vMVKoKLwO2MZSXDKklP+qSiDNMfvVyXeJR8r1samfmIe4uNqb4WHJfuobYctGaYrFPHMkvyLRUWKFW3aND8YUoEw8ALeCBBeG+HP/xF6jB17CFcsN7ZAJgStRuQMZD0RlIWUERYfuRFeikUK9s4e8oIFfUrIWhdGKIDZYAKb6rDYmYqNmgh4SVkqod0vGMpPBbwV2JYVBbW9sEeGbQENnekY0RM+3RGXFZEWs/PemjUTK1URkPTWd88d0yUvPRFeik0sjdykNnz0InYCTmSZjncCPhnttBCzH0ca+WT2z3mClWkfAFO8oBA7393pKNz3vgLIxc2+xMJ/DRaaccE62+HmL9gz9sS5tcxyuHRRSovCgIftdBE3F8WMX3ZKNEd7QB1iMT1WglEAwSws7tMPJ4xnnZ3hW05vREaKNEHtSOET0ossXlnBWwp/yszbEcng8me2+0j5TMzKiEFdR2eqi2z2Md1Hee+/r8AS4AoRkKZW5kc3RyZWFtCmVuZG9iagozMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0NyA+PgpzdHJlYW0KeJxNUbttRDEM698UXOAA62t5ngtSXfZvQ8kIkMIgoS8ppyUW9sZLDOEHWw++5JFVQ38ePzHsMyw9yeTUP+a5yVQUvhWqm5hQF2Lh/WgEvBZ0LyIrygffj2UMc8734KMQl2AmNGCsb0kmF9W8M2TCiaGOw0GbVBh3TRQsrhXNM8jtVjeyOrMgbHglE+LGAEQE2ReQzWCjjLGVkMVyHqgKkgVaYNfpG1GLgiuU1gl0otbEuszgq+f2djdDL/LgqLp4fQzrS7DC6KV7LHyuQh/M9Ew7d0kjvfCmExFmDwVSmZ2RlTo9Yn23QP+fZSv4+8nP8/0LFShcKgplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggOTAgPj4Kc3RyZWFtCnicTY1BEsAgCAPvvCJPUETQ/3R60v9fq9QOvcBOAokWRYL0NWpLMO64MhVrUCmYlJfAVTBcC9ruosr+MklMnYbTe7cDg7LxcYPSSfv2cXoAq/16Bt0P0hwiWAplbmRzdHJlYW0KZW5kb2JqCjMyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTYzID4+CnN0cmVhbQp4nEWQuXUEMQxDc1WBEniAOuoZP0ez/acLabzeQPp4hHiIPQnDcl3FhdENP962zDS8jjLcjfVlxviosUBO0AcYIhNXo0n17YozVOnh1WKuo6JcLzoiEsyS46tAI3w6ssdDW9uZfjqvf+wh7xP/KirnbmEBLqruQPlSH/HUj9lR6pqhjyorax5q2r8IuyKUtn1cTmWcunsHtMJnK1f7fQOo5zqACmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MSA+PgpzdHJlYW0KeJw9zLsVgDAIBdA+U7wRQnyA7OOx0v1bwUQbuHzVAx0hGdQNbh2HtKxLd5N96nq1iaTIgNJTalwaToyoaX2pfWrguxvmS9WJP83P5wOHxxlrCmVuZHN0cmVhbQplbmRvYmoKMzQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNTUgPj4Kc3RyZWFtCnicRZFLkgMgCET3noIjgPzkPJmaVXL/7TSYTDZ2l6j9hEojphIs5xR5MP3I8s1ktum1HKudjQKKIhTM5Cr0WIHVnSnizLVEtfWxMnLc6R2D4g3nrpxUsrhRxjqqOhU4pufK+qru/Lgsyr4jhzIFbNY5DjZw5bZhjBOjzVZ3h/tEkKeTqaPidpBs+IOTxr7K1RW4Tjb76iUYB4J+oQlM8k2gdYZA4+YpenIJ9vFxu/NAsLe8CaRsCOTIEIwOQbtOrn9x6/ze/zrDnefaDFeOd/E7TGu74y8xyYq5gEXuFNTzPRet6wwd78mZY3LTfUPnXLDL3UGmz/wf6/cPUIpmiAplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTYxID4+CnN0cmVhbQp4nEWQSxLDIAxD95xCR/BHBnyedLpK77+tIU2zgKexQAZ3JwSptQUT0QUvbUu6Cz5bCc7GeOg2bjUS5AR1gFak42iUUn25xWmVdPFoNnMrC60THWYOepSjGaAQOhXe7aLkcqbuzvlHcPVf9Uex7pzNxMBk5Q6EZvUp7nybHVFd3WR/0mNu1mt/FfaqsLSspeWE285dM6AE7qkc7f0FqXM6hAplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjE0ID4+CnN0cmVhbQp4nD1QuxFDMQjrPQUL5M587TfPy6XL/m0knKRCNkISlJpMyZSHOsqSrClPHT5LYoe8h+VuZDYlKkUvk7Al99AK8X2J5hT33dWWs0M0l2g5fgszKqobHdNLNppwKhO6oNzDM/oNbXQDVocesVsg0KRg17YgcscPGAzBmROLIgxKTQb/rXL3UtzvPRxvooiUdPCu+eX0y88tvE49jkS6vfmKa3GmOgpEcEZq8op0YcWyyEOk1QQ1PQNrtQCu3nr5N2hHdBmA7BOJ4zSlHEP/1rjH6wOHilL0CmVuZHN0cmVhbQplbmRvYmoKMzcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MCA+PgpzdHJlYW0KeJxFjLsNwDAIRHumYAR+JmafKJWzfxsgStxwT7p7uDoSMlPeYYaHBJ4MLIZT8QaZo2A1uEZSjZ3so7BuX3WB5npTq/X3BypPdnZxPc3LGfQKZW5kc3RyZWFtCmVuZG9iagozOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIzNiA+PgpzdHJlYW0KeJxNUEtuRCEM23OKXOBJJCEBzkPVVef+27HDVO0qhhh/SA/pslUe61NidYns8qVNl8oyeRWo5U/b/1EMAm7/0MhBtLeMnWLmEtbFwiQ85TQjGyfXLB+PO08bZoXGxI3jnS4ZYJ8WATVblc2BOW06N0C6kBq3qrPeZFAMIupCzQeTLpyn0ZeIOZ6oYEp3JrWQG1w+1aEDcVq9Crlji5NvxBxZocBh0Exx1l8B1qjJslnIIEmGIc59o3uUCo2oynkrFcIPk6ER9YbVoAaVuYWiqeWS/B3aAjAFtox16QxKgaoAwd8qp32/ASSNXVMKZW5kc3RyZWFtCmVuZG9iagozOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDQ5ID4+CnN0cmVhbQp4nDM2tFAwUDA0MAeSRoZAlpGJQoohF0gAxMzlggnmgFkGQBqiOAeuJocrDQDG6A0mCmVuZHN0cmVhbQplbmRvYmoKNDAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNTcgPj4Kc3RyZWFtCnicRZC5EUMxCERzVUEJErAI6rHH0Xf/qRf5SrRvAC2HryVTqh8nIqbc12j0MHkOn00lVizYJraTGnIbFkFKMZh4TjGro7ehmYfU67ioqrh1ZpXTacvKxX/zaFczkz3CNeon8E3o+J88tKnoW6CvC5R9QLU4nUlQMX2vYoGjnHZ/IpwY4D4ZR5kpI3Fibgrs9xkAZr5XuMbjBd0BN3kKZW5kc3RyZWFtCmVuZG9iago0MSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMzMiA+PgpzdHJlYW0KeJwtUjmOJDEMy/0KfmAA6/Lxnh5M1Pv/dElVBQWqbMs85HLDRCV+LJDbUWvi10ZmoMLwr6vMhe9I28g6iGvIRVzJlsJnRCzkMcQ8xILv2/gZHvmszMmzB8Yv2fcZVuypCctCxosztMMqjsMqyLFg6yKqe3hTpMOpJNjji/8+xXMXgha+I2jAL/nnqyN4vqRF2j1m27RbD5ZpR5UUloPtac7L5EvrLFfH4/kg2d4VO0JqV4CiMHfGeS6OMm1lRGthZ4OkxsX25tiPpQRd6MZlpDgC+ZkqwgNKmsxsoiD+yOkhpzIQpq7pSie3URV36slcs7m8nUkyW/dFis0UzuvCmfV3mDKrzTt5lhOlTkX4GXu2BA2d4+rZa5mFRrc5wSslfDZ2enLyvZpZD8mpSEgV07oKTqPIFEvYlviaiprS1Mvw35f3GX//ATPifAEKZW5kc3RyZWFtCmVuZG9iago0MiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDY4ID4+CnN0cmVhbQp4nDMzNlMwULAwAhKmpoYK5kaWCimGXEA+iJXLBRPLAbPMLMyBLCMLkJYcLkMLYzBtYmykYGZiBmRZIDEgutIAcvgSkQplbmRzdHJlYW0KZW5kb2JqCjQzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzE3ID4+CnN0cmVhbQp4nDVSS3JDMQjbv1Nwgc6Yv32edLJq7r+thCcrsC1AQi4vWdJLftQl26XD5Fcf9yWxQj6P7ZrMUsX3FrMUzy2vR88Rty0KBFETPfgyJxUi1M/U6Dp4YZc+A68QTikWeAeTAAav4V94lE6DwDsbMt4Rk5EaECTBmkuLTUiUPUn8K+X1pJU0dH4mK3P5e3KpFGqjyQgVIFi52AekKykeJBM9iUiycr03VojekFeSx2clJhkQ3SaxTbTA49yVtISZmEIF5liA1XSzuvocTFjjsITxKmEW1YNNnjWphGa0jmNkw3j3wkyJhYbDElCbfZUJqpeP09wJI6ZHTXbtwrJbNu8hRKP5MyyUwccoJAGHTmMkCtKwgBGBOb2wir3mCzkWwIhlnZosDG1oJbt6joXA0JyzpWHG157X8/4HRVt7owplbmRzdHJlYW0KZW5kb2JqCjQ0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTcgPj4Kc3RyZWFtCnicMza0UDCAwxRDLgAalALsCmVuZHN0cmVhbQplbmRvYmoKNDUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMzEgPj4Kc3RyZWFtCnicRY/LDQQhDEPvVOES8hk+qYfVntj+r+swmkFC+EEiO/EwCKzz8jbQxfDRosM3/jbVq2OVLB+6elJWD+mQh7zyFVBpMFHEhVlMHUNhzpjKyJYytxvhtk2DrGyVVK2DdjwGD7anZasIfqltYeos8QzCVV64xw0/kEutd71Vvn9CUzCXCmVuZHN0cmVhbQplbmRvYmoKNDYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMzggPj4Kc3RyZWFtCnicNVI5rt1ADOt9Cl0ggHbNnOcFqX7u34aUXwpDtFaKmo4WlWn5ZSFVLZMuv+1JbYkb8vfJCokTklcl2qUMkVD5PIVUv2fLvL7WnBEgS5UKk5OSxyUL/gyX3i4c52NrP48jdz16YFWMhBIByxQTo2tZOrvDmo38PKYBP+IRcq5YtxxjFUgNunHaFe9D83nIGiBmmJaKCl1WiRZ+QfGgR61991hUWCDR7RxJcIyNUJGAdoHaSAw5sxa7qC/6WZSYCXTtiyLuosASScycYl06+g8+dCyovzbjy6+OSvpIK2tM2nejSWnMIpOul0VvN299PbhA8y7Kf17NIEFT1ihpfNCqnWMomhllhXccmgw0xxyHzBM8hzMSlPR9KH5fSya6KJE/Dg2hf18eo4ycBm8Bc9GftooDF/HZYa8cYIXSxZrkfUAqE3pg+v/X+Hn+/AMctoBUCmVuZHN0cmVhbQplbmRvYmoKNDcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDggPj4Kc3RyZWFtCnicLVE5kgNBCMvnFXpCc9PvscuR9//pCsoBg4ZDIDotcVDGTxCWK97yyFW04e+ZGMF3waHfynUbFjkQFUjSGFRNqF28Hr0HdhxmAvOkNSyDGesDP2MKN3pxeEzG2e11GTUEe9drT2ZQMisXccnEBVN12MiZw0+mjAvtXM8NyLkR1mUYpJuVxoyEI00hUkih6iapM0GQBKOrUaONHMV+6csjnWFVI2oM+1xL29dzE84aNDsWqzw5pUdXnMvJxQsrB/28zcBFVBqrPBAScL/bQ/2c7OQ33tK5s8X0+F5zsrwwFVjx5rUbkE21+Dcv4vg94+v5/AOopVsWCmVuZHN0cmVhbQplbmRvYmoKNDggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNzEgPj4Kc3RyZWFtCnicTZBNDkIhEIP3nKIXMKHzA4/zaFzp/bd28PnigvRLIUOnwwMdR+JGR4bO6HiwyTEOvAsyJl6N85+M6ySOCeoVbcG6tDvuzSwxJywTI2BrlNybRxT44ZgLQYLs8sMXGESka5hvNZ91k35+u9Nd1KV199MjCpzIjlAMG3AF2NM9DtwSzu+aJr9UKRmbOJQPVBeRstkJhailYpdTVWiM4lY974te7fkBwfY7+wplbmRzdHJlYW0KZW5kb2JqCjQ5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTM4ID4+CnN0cmVhbQp4nD2PQQ4DMQgD73mFPxApdkJY3rNVT9v/X0ua3V7QCIwxFkJDb6hqDpuCDceLpUuo1vApiolKDsiZYA6lpNIdZ5F6YjgY3B60G87isen6EbuSVn3Q5ka6JWiCR+xTadyWcRPEAzUF6inqXKO8ELmfqVfYNJLdtLKSazim373nqev/01XeX1/fLowKZW5kc3RyZWFtCmVuZG9iago1MCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxMCA+PgpzdHJlYW0KeJw1UMsNQzEIu2cKFqgUAoFknla9df9rbdA7YRH/QljIlAh5qcnOKelLPjpMD7Yuv7EiC611JezKmiCeK++hmbKx0djiYHAaJl6AFjdg6GmNGjV04YKmLpVCgcUl8Jl8dXvovk8ZeGoZcnYEEUPJYAlquhZNWLQ8n5BOAeL/fsPuLeShkvPKnhv5G5zt8DuzbuEnanYi0XIVMtSzNMcYCBNFHjx5RaZw4rPWd9U0EtRmC06WAa5OP4wOAGAiXlmA7K5EOUvSjqWfb7zH9w9AAFO0CmVuZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKPDwgL0Jhc2VGb250IC9EZWphVnVTYW5zIC9DaGFyUHJvY3MgMTUgMCBSCi9FbmNvZGluZyA8PAovRGlmZmVyZW5jZXMgWyAzMiAvc3BhY2UgNDYgL3BlcmlvZCA0OCAvemVybyAvb25lIC90d28gL3RocmVlIC9mb3VyIC9maXZlIC9zaXggL3NldmVuCi9laWdodCA2NSAvQSAvQiAvQyAvRCAvRSAvRiA3MiAvSCA3NiAvTCA4NiAvViA5NyAvYSA5OSAvYyAvZCAvZSAxMDQgL2ggMTA3Ci9rIDEwOSAvbSAvbiAvbyAvcCAxMTQgL3IgL3MgL3QgL3UgMTIxIC95IF0KL1R5cGUgL0VuY29kaW5nID4+Ci9GaXJzdENoYXIgMCAvRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAxMyAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvRGVqYVZ1U2FucwovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAxMiAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TmFtZSAvRGVqYVZ1U2FucyAvSXRhbGljQW5nbGUgMAovTWF4V2lkdGggMTM0MiAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMTIgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM0MiA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDIzIDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxMiA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjEyIDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDUKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk4MiA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMTUgMCBvYmoKPDwgL0EgMTYgMCBSIC9CIDE3IDAgUiAvQyAxOCAwIFIgL0QgMTkgMCBSIC9FIDIwIDAgUiAvRiAyMSAwIFIgL0ggMjIgMCBSCi9MIDIzIDAgUiAvViAyNCAwIFIgL2EgMjUgMCBSIC9jIDI2IDAgUiAvZCAyNyAwIFIgL2UgMjggMCBSIC9laWdodCAyOSAwIFIKL2ZpdmUgMzAgMCBSIC9mb3VyIDMxIDAgUiAvaCAzMiAwIFIgL2sgMzMgMCBSIC9tIDM0IDAgUiAvbiAzNSAwIFIgL28gMzYgMCBSCi9vbmUgMzcgMCBSIC9wIDM4IDAgUiAvcGVyaW9kIDM5IDAgUiAvciA0MCAwIFIgL3MgNDEgMCBSIC9zZXZlbiA0MiAwIFIKL3NpeCA0MyAwIFIgL3NwYWNlIDQ0IDAgUiAvdCA0NSAwIFIgL3RocmVlIDQ2IDAgUiAvdHdvIDQ3IDAgUiAvdSA0OCAwIFIKL3kgNDkgMCBSIC96ZXJvIDUwIDAgUiA+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTQgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMyA8PCAvQ0EgMC44IC9UeXBlIC9FeHRHU3RhdGUgL2NhIDAuOCA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCA+PgplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDEwIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKNTEgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIxMDQwMjAyNDAzNyswMycwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjMuMikgPj4KZW5kb2JqCnhyZWYKMCA1MgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAxNjc1NSAwMDAwMCBuIAowMDAwMDE2NTE4IDAwMDAwIG4gCjAwMDAwMTY1NTAgMDAwMDAgbiAKMDAwMDAxNjY5MiAwMDAwMCBuIAowMDAwMDE2NzEzIDAwMDAwIG4gCjAwMDAwMTY3MzQgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzk3IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwNTYxMyAwMDAwMCBuIAowMDAwMDE1MDU3IDAwMDAwIG4gCjAwMDAwMTQ4NTcgMDAwMDAgbiAKMDAwMDAxNDM3NiAwMDAwMCBuIAowMDAwMDE2MTEwIDAwMDAwIG4gCjAwMDAwMDU2MzQgMDAwMDAgbiAKMDAwMDAwNTc5NCAwMDAwMCBuIAowMDAwMDA2MTI2IDAwMDAwIG4gCjAwMDAwMDY0MzEgMDAwMDAgbiAKMDAwMDAwNjY2NCAwMDAwMCBuIAowMDAwMDA2ODE1IDAwMDAwIG4gCjAwMDAwMDY5NjEgMDAwMDAgbiAKMDAwMDAwNzExMCAwMDAwMCBuIAowMDAwMDA3MjQxIDAwMDAwIG4gCjAwMDAwMDczODMgMDAwMDAgbiAKMDAwMDAwNzc2MCAwMDAwMCBuIAowMDAwMDA4MDYzIDAwMDAwIG4gCjAwMDAwMDgzNjMgMDAwMDAgbiAKMDAwMDAwODY4MSAwMDAwMCBuIAowMDAwMDA5MTQ2IDAwMDAwIG4gCjAwMDAwMDk0NjYgMDAwMDAgbiAKMDAwMDAwOTYyOCAwMDAwMCBuIAowMDAwMDA5ODY0IDAwMDAwIG4gCjAwMDAwMTAwMTcgMDAwMDAgbiAKMDAwMDAxMDM0NSAwMDAwMCBuIAowMDAwMDEwNTc5IDAwMDAwIG4gCjAwMDAwMTA4NjYgMDAwMDAgbiAKMDAwMDAxMTAxOCAwMDAwMCBuIAowMDAwMDExMzI3IDAwMDAwIG4gCjAwMDAwMTE0NDggMDAwMDAgbiAKMDAwMDAxMTY3OCAwMDAwMCBuIAowMDAwMDEyMDgzIDAwMDAwIG4gCjAwMDAwMTIyMjMgMDAwMDAgbiAKMDAwMDAxMjYxMyAwMDAwMCBuIAowMDAwMDEyNzAyIDAwMDAwIG4gCjAwMDAwMTI5MDYgMDAwMDAgbiAKMDAwMDAxMzMxNyAwMDAwMCBuIAowMDAwMDEzNjM4IDAwMDAwIG4gCjAwMDAwMTM4ODIgMDAwMDAgbiAKMDAwMDAxNDA5MyAwMDAwMCBuIAowMDAwMDE2ODE1IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gNTEgMCBSIC9Sb290IDEgMCBSIC9TaXplIDUyID4+CnN0YXJ0eHJlZgoxNjk3MgolJUVPRgo=\n",
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"711.274375pt\" version=\"1.1\" viewBox=\"0 0 712.703125 711.274375\" width=\"712.703125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2021-04-02T02:40:37.172985</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.3.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 711.274375 \n",
       "L 712.703125 711.274375 \n",
       "L 712.703125 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 30.103125 319.118125 \n",
       "L 705.503125 319.118125 \n",
       "L 705.503125 22.318125 \n",
       "L 30.103125 22.318125 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 319.118125 \n",
       "L 60.803125 22.318125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"m5670c8524a\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"60.803125\" xlink:href=\"#m5670c8524a\" y=\"319.118125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(57.621875 333.716562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-48\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 148.517411 319.118125 \n",
       "L 148.517411 22.318125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"148.517411\" xlink:href=\"#m5670c8524a\" y=\"319.118125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(145.336161 333.716562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-50\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 236.231696 319.118125 \n",
       "L 236.231696 22.318125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"236.231696\" xlink:href=\"#m5670c8524a\" y=\"319.118125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(233.050446 333.716562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-52\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 323.945982 319.118125 \n",
       "L 323.945982 22.318125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"323.945982\" xlink:href=\"#m5670c8524a\" y=\"319.118125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(320.764732 333.716562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 33.015625 40.375 \n",
       "Q 26.375 40.375 22.484375 35.828125 \n",
       "Q 18.609375 31.296875 18.609375 23.390625 \n",
       "Q 18.609375 15.53125 22.484375 10.953125 \n",
       "Q 26.375 6.390625 33.015625 6.390625 \n",
       "Q 39.65625 6.390625 43.53125 10.953125 \n",
       "Q 47.40625 15.53125 47.40625 23.390625 \n",
       "Q 47.40625 31.296875 43.53125 35.828125 \n",
       "Q 39.65625 40.375 33.015625 40.375 \n",
       "z\n",
       "M 52.59375 71.296875 \n",
       "L 52.59375 62.3125 \n",
       "Q 48.875 64.0625 45.09375 64.984375 \n",
       "Q 41.3125 65.921875 37.59375 65.921875 \n",
       "Q 27.828125 65.921875 22.671875 59.328125 \n",
       "Q 17.53125 52.734375 16.796875 39.40625 \n",
       "Q 19.671875 43.65625 24.015625 45.921875 \n",
       "Q 28.375 48.1875 33.59375 48.1875 \n",
       "Q 44.578125 48.1875 50.953125 41.515625 \n",
       "Q 57.328125 34.859375 57.328125 23.390625 \n",
       "Q 57.328125 12.15625 50.6875 5.359375 \n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \n",
       "Q 6.984375 17.96875 6.984375 36.375 \n",
       "Q 6.984375 53.65625 15.1875 63.9375 \n",
       "Q 23.390625 74.21875 37.203125 74.21875 \n",
       "Q 40.921875 74.21875 44.703125 73.484375 \n",
       "Q 48.484375 72.75 52.59375 71.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-54\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 411.660268 319.118125 \n",
       "L 411.660268 22.318125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"411.660268\" xlink:href=\"#m5670c8524a\" y=\"319.118125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 8 -->\n",
       "      <g transform=\"translate(408.479018 333.716562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 31.78125 34.625 \n",
       "Q 24.75 34.625 20.71875 30.859375 \n",
       "Q 16.703125 27.09375 16.703125 20.515625 \n",
       "Q 16.703125 13.921875 20.71875 10.15625 \n",
       "Q 24.75 6.390625 31.78125 6.390625 \n",
       "Q 38.8125 6.390625 42.859375 10.171875 \n",
       "Q 46.921875 13.96875 46.921875 20.515625 \n",
       "Q 46.921875 27.09375 42.890625 30.859375 \n",
       "Q 38.875 34.625 31.78125 34.625 \n",
       "z\n",
       "M 21.921875 38.8125 \n",
       "Q 15.578125 40.375 12.03125 44.71875 \n",
       "Q 8.5 49.078125 8.5 55.328125 \n",
       "Q 8.5 64.0625 14.71875 69.140625 \n",
       "Q 20.953125 74.21875 31.78125 74.21875 \n",
       "Q 42.671875 74.21875 48.875 69.140625 \n",
       "Q 55.078125 64.0625 55.078125 55.328125 \n",
       "Q 55.078125 49.078125 51.53125 44.71875 \n",
       "Q 48 40.375 41.703125 38.8125 \n",
       "Q 48.828125 37.15625 52.796875 32.3125 \n",
       "Q 56.78125 27.484375 56.78125 20.515625 \n",
       "Q 56.78125 9.90625 50.3125 4.234375 \n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \n",
       "Q 19.734375 -1.421875 13.25 4.234375 \n",
       "Q 6.78125 9.90625 6.78125 20.515625 \n",
       "Q 6.78125 27.484375 10.78125 32.3125 \n",
       "Q 14.796875 37.15625 21.921875 38.8125 \n",
       "z\n",
       "M 18.3125 54.390625 \n",
       "Q 18.3125 48.734375 21.84375 45.5625 \n",
       "Q 25.390625 42.390625 31.78125 42.390625 \n",
       "Q 38.140625 42.390625 41.71875 45.5625 \n",
       "Q 45.3125 48.734375 45.3125 54.390625 \n",
       "Q 45.3125 60.0625 41.71875 63.234375 \n",
       "Q 38.140625 66.40625 31.78125 66.40625 \n",
       "Q 25.390625 66.40625 21.84375 63.234375 \n",
       "Q 18.3125 60.0625 18.3125 54.390625 \n",
       "z\n",
       "\" id=\"DejaVuSans-56\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-56\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 499.374554 319.118125 \n",
       "L 499.374554 22.318125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"499.374554\" xlink:href=\"#m5670c8524a\" y=\"319.118125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(493.012054 333.716562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-49\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 587.088839 319.118125 \n",
       "L 587.088839 22.318125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"587.088839\" xlink:href=\"#m5670c8524a\" y=\"319.118125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 12 -->\n",
       "      <g transform=\"translate(580.726339 333.716562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 674.803125 319.118125 \n",
       "L 674.803125 22.318125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"674.803125\" xlink:href=\"#m5670c8524a\" y=\"319.118125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 14 -->\n",
       "      <g transform=\"translate(668.440625 333.716562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_9\">\n",
       "     <!-- Epoch -->\n",
       "     <g transform=\"translate(352.492187 347.394687)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 9.8125 72.90625 \n",
       "L 55.90625 72.90625 \n",
       "L 55.90625 64.59375 \n",
       "L 19.671875 64.59375 \n",
       "L 19.671875 43.015625 \n",
       "L 54.390625 43.015625 \n",
       "L 54.390625 34.71875 \n",
       "L 19.671875 34.71875 \n",
       "L 19.671875 8.296875 \n",
       "L 56.78125 8.296875 \n",
       "L 56.78125 0 \n",
       "L 9.8125 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-69\"/>\n",
       "       <path d=\"M 18.109375 8.203125 \n",
       "L 18.109375 -20.796875 \n",
       "L 9.078125 -20.796875 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "z\n",
       "M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-112\"/>\n",
       "       <path d=\"M 30.609375 48.390625 \n",
       "Q 23.390625 48.390625 19.1875 42.75 \n",
       "Q 14.984375 37.109375 14.984375 27.296875 \n",
       "Q 14.984375 17.484375 19.15625 11.84375 \n",
       "Q 23.34375 6.203125 30.609375 6.203125 \n",
       "Q 37.796875 6.203125 41.984375 11.859375 \n",
       "Q 46.1875 17.53125 46.1875 27.296875 \n",
       "Q 46.1875 37.015625 41.984375 42.703125 \n",
       "Q 37.796875 48.390625 30.609375 48.390625 \n",
       "z\n",
       "M 30.609375 56 \n",
       "Q 42.328125 56 49.015625 48.375 \n",
       "Q 55.71875 40.765625 55.71875 27.296875 \n",
       "Q 55.71875 13.875 49.015625 6.21875 \n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \n",
       "Q 5.515625 13.875 5.515625 27.296875 \n",
       "Q 5.515625 40.765625 12.171875 48.375 \n",
       "Q 18.84375 56 30.609375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-111\"/>\n",
       "       <path d=\"M 48.78125 52.59375 \n",
       "L 48.78125 44.1875 \n",
       "Q 44.96875 46.296875 41.140625 47.34375 \n",
       "Q 37.3125 48.390625 33.40625 48.390625 \n",
       "Q 24.65625 48.390625 19.8125 42.84375 \n",
       "Q 14.984375 37.3125 14.984375 27.296875 \n",
       "Q 14.984375 17.28125 19.8125 11.734375 \n",
       "Q 24.65625 6.203125 33.40625 6.203125 \n",
       "Q 37.3125 6.203125 41.140625 7.25 \n",
       "Q 44.96875 8.296875 48.78125 10.40625 \n",
       "L 48.78125 2.09375 \n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \n",
       "Q 5.515625 14.109375 5.515625 27.296875 \n",
       "Q 5.515625 40.671875 12.859375 48.328125 \n",
       "Q 20.21875 56 33.015625 56 \n",
       "Q 37.15625 56 41.109375 55.140625 \n",
       "Q 45.0625 54.296875 48.78125 52.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-99\"/>\n",
       "       <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-104\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"63.183594\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"126.660156\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"187.841797\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"242.822266\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 30.103125 298.185422 \n",
       "L 705.503125 298.185422 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"md23709666f\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"298.185422\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.8 -->\n",
       "      <g transform=\"translate(7.2 301.984641)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 10.6875 12.40625 \n",
       "L 21 12.40625 \n",
       "L 21 0 \n",
       "L 10.6875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-46\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-56\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 30.103125 255.660886 \n",
       "L 705.503125 255.660886 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"255.660886\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 1.0 -->\n",
       "      <g transform=\"translate(7.2 259.460105)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 30.103125 213.13635 \n",
       "L 705.503125 213.13635 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"213.13635\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 1.2 -->\n",
       "      <g transform=\"translate(7.2 216.935569)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_23\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 30.103125 170.611814 \n",
       "L 705.503125 170.611814 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_24\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"170.611814\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 1.4 -->\n",
       "      <g transform=\"translate(7.2 174.411032)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_25\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 30.103125 128.087278 \n",
       "L 705.503125 128.087278 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_26\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"128.087278\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 1.6 -->\n",
       "      <g transform=\"translate(7.2 131.886496)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-54\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_27\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 30.103125 85.562741 \n",
       "L 705.503125 85.562741 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_28\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"85.562741\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 1.8 -->\n",
       "      <g transform=\"translate(7.2 89.36196)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-56\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_29\">\n",
       "      <path clip-path=\"url(#p73fe432886)\" d=\"M 30.103125 43.038205 \n",
       "L 705.503125 43.038205 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_30\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"43.038205\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_16\">\n",
       "      <!-- 2.0 -->\n",
       "      <g transform=\"translate(7.2 46.837424)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_31\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 53.456717 \n",
       "L 104.660268 81.52291 \n",
       "L 148.517411 121.283352 \n",
       "L 192.374554 140.632016 \n",
       "L 236.231696 154.027245 \n",
       "L 280.088839 162.744774 \n",
       "L 323.945982 180.817702 \n",
       "L 367.803125 192.086704 \n",
       "L 411.660268 204.844065 \n",
       "L 455.517411 216.113067 \n",
       "L 499.374554 232.910259 \n",
       "L 543.231696 245.880243 \n",
       "L 587.088839 264.165793 \n",
       "L 630.945982 284.152325 \n",
       "L 674.803125 303.075744 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_32\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 41.124601 \n",
       "L 104.660268 85.987987 \n",
       "L 148.517411 121.92122 \n",
       "L 192.374554 142.54562 \n",
       "L 236.231696 158.067075 \n",
       "L 280.088839 169.123455 \n",
       "L 323.945982 179.541966 \n",
       "L 367.803125 190.810968 \n",
       "L 411.660268 206.545047 \n",
       "L 455.517411 219.51503 \n",
       "L 499.374554 232.697636 \n",
       "L 543.231696 249.282206 \n",
       "L 587.088839 267.993001 \n",
       "L 630.945982 287.341665 \n",
       "L 674.803125 297.547554 \n",
       "\" style=\"fill:none;stroke:#ff0000;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_33\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 43.463451 \n",
       "L 104.660268 80.247174 \n",
       "L 148.517411 98.95797 \n",
       "L 192.374554 123.196956 \n",
       "L 236.231696 139.781525 \n",
       "L 280.088839 151.26315 \n",
       "L 323.945982 157.64183 \n",
       "L 367.803125 165.296247 \n",
       "L 411.660268 169.761323 \n",
       "L 455.517411 176.352626 \n",
       "L 499.374554 181.668193 \n",
       "L 543.231696 185.070156 \n",
       "L 587.088839 186.133269 \n",
       "L 630.945982 195.276045 \n",
       "L 674.803125 198.252762 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_34\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 37.08477 \n",
       "L 104.660268 74.931607 \n",
       "L 148.517411 123.622201 \n",
       "L 192.374554 142.120374 \n",
       "L 236.231696 158.067075 \n",
       "L 280.088839 170.824436 \n",
       "L 323.945982 182.306061 \n",
       "L 367.803125 191.661459 \n",
       "L 411.660268 202.292593 \n",
       "L 455.517411 215.900445 \n",
       "L 499.374554 235.674354 \n",
       "L 543.231696 248.006469 \n",
       "L 587.088839 265.866775 \n",
       "L 630.945982 283.939702 \n",
       "L 674.803125 305.627216 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_35\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 44.526564 \n",
       "L 104.660268 86.838477 \n",
       "L 148.517411 105.761896 \n",
       "L 192.374554 116.39303 \n",
       "L 236.231696 130.851372 \n",
       "L 280.088839 139.143657 \n",
       "L 323.945982 147.861187 \n",
       "L 367.803125 153.389377 \n",
       "L 411.660268 159.130189 \n",
       "L 455.517411 166.997228 \n",
       "L 499.374554 174.013777 \n",
       "L 543.231696 177.840985 \n",
       "L 587.088839 181.242948 \n",
       "L 630.945982 188.472119 \n",
       "L 674.803125 193.575063 \n",
       "\" style=\"fill:none;stroke:#ffa500;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_36\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 43.250828 \n",
       "L 104.660268 89.602572 \n",
       "L 148.517411 117.881389 \n",
       "L 192.374554 141.695129 \n",
       "L 236.231696 153.176754 \n",
       "L 280.088839 165.934115 \n",
       "L 323.945982 178.691476 \n",
       "L 367.803125 188.897364 \n",
       "L 411.660268 201.016857 \n",
       "L 455.517411 217.388803 \n",
       "L 499.374554 228.870428 \n",
       "L 543.231696 244.391884 \n",
       "L 587.088839 262.677434 \n",
       "L 630.945982 281.600853 \n",
       "L 674.803125 298.610667 \n",
       "\" style=\"fill:none;stroke:#800080;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_37\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 36.446902 \n",
       "L 104.660268 73.443249 \n",
       "L 148.517411 109.376482 \n",
       "L 192.374554 121.283352 \n",
       "L 236.231696 138.505789 \n",
       "L 280.088839 144.246601 \n",
       "L 323.945982 152.326263 \n",
       "L 367.803125 158.917566 \n",
       "L 411.660268 163.382643 \n",
       "L 455.517411 170.186568 \n",
       "L 499.374554 172.525418 \n",
       "L 543.231696 177.203117 \n",
       "L 587.088839 179.116721 \n",
       "L 630.945982 181.030325 \n",
       "L 674.803125 186.133269 \n",
       "\" style=\"fill:none;stroke:#00ffff;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_38\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 51.33049 \n",
       "L 104.660268 73.230626 \n",
       "L 148.517411 110.439595 \n",
       "L 192.374554 122.984333 \n",
       "L 236.231696 128.937768 \n",
       "L 280.088839 129.363014 \n",
       "L 323.945982 135.316449 \n",
       "L 367.803125 134.253335 \n",
       "L 411.660268 128.937768 \n",
       "L 455.517411 127.662032 \n",
       "L 499.374554 112.991067 \n",
       "L 543.231696 102.359933 \n",
       "L 587.088839 83.86176 \n",
       "L 630.945982 79.184061 \n",
       "L 674.803125 56.008189 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_39\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 39.848865 \n",
       "L 104.660268 81.310288 \n",
       "L 148.517411 110.652218 \n",
       "L 192.374554 125.11056 \n",
       "L 236.231696 133.615467 \n",
       "L 280.088839 138.505789 \n",
       "L 323.945982 139.781525 \n",
       "L 367.803125 136.804807 \n",
       "L 411.660268 130.213504 \n",
       "L 455.517411 129.788259 \n",
       "L 499.374554 119.369748 \n",
       "L 543.231696 104.911405 \n",
       "L 587.088839 94.705517 \n",
       "L 630.945982 71.529644 \n",
       "L 674.803125 35.809034 \n",
       "\" style=\"fill:none;stroke:#ff0000;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_40\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 43.038205 \n",
       "L 104.660268 77.057834 \n",
       "L 148.517411 95.343385 \n",
       "L 192.374554 116.39303 \n",
       "L 236.231696 128.725146 \n",
       "L 280.088839 137.442675 \n",
       "L 323.945982 141.907752 \n",
       "L 367.803125 142.970865 \n",
       "L 411.660268 145.947583 \n",
       "L 455.517411 148.286432 \n",
       "L 499.374554 148.499055 \n",
       "L 543.231696 149.987414 \n",
       "L 587.088839 146.585451 \n",
       "L 630.945982 150.412659 \n",
       "L 674.803125 150.625282 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_41\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 36.446902 \n",
       "L 104.660268 70.253908 \n",
       "L 148.517411 111.077463 \n",
       "L 192.374554 125.11056 \n",
       "L 236.231696 132.127108 \n",
       "L 280.088839 135.316449 \n",
       "L 323.945982 137.01743 \n",
       "L 367.803125 133.190222 \n",
       "L 411.660268 130.426127 \n",
       "L 455.517411 122.346465 \n",
       "L 499.374554 118.944502 \n",
       "L 543.231696 99.170593 \n",
       "L 587.088839 88.964704 \n",
       "L 630.945982 75.14423 \n",
       "L 674.803125 49.842131 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_42\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 43.463451 \n",
       "L 104.660268 83.011269 \n",
       "L 148.517411 100.233706 \n",
       "L 192.374554 107.462877 \n",
       "L 236.231696 120.645484 \n",
       "L 280.088839 126.811541 \n",
       "L 323.945982 132.764976 \n",
       "L 367.803125 135.529071 \n",
       "L 411.660268 136.592185 \n",
       "L 455.517411 142.332997 \n",
       "L 499.374554 145.309715 \n",
       "L 543.231696 144.884469 \n",
       "L 587.088839 145.73496 \n",
       "L 630.945982 148.499055 \n",
       "L 674.803125 148.499055 \n",
       "\" style=\"fill:none;stroke:#ffa500;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_43\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 42.400337 \n",
       "L 104.660268 85.987987 \n",
       "L 148.517411 108.525991 \n",
       "L 192.374554 126.811541 \n",
       "L 236.231696 131.276618 \n",
       "L 280.088839 134.253335 \n",
       "L 323.945982 137.230053 \n",
       "L 367.803125 134.040713 \n",
       "L 411.660268 130.851372 \n",
       "L 455.517411 124.472692 \n",
       "L 499.374554 120.432861 \n",
       "L 543.231696 110.439595 \n",
       "L 587.088839 101.509442 \n",
       "L 630.945982 76.207343 \n",
       "L 674.803125 58.772284 \n",
       "\" style=\"fill:none;stroke:#800080;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_44\">\n",
       "    <path clip-path=\"url(#p73fe432886)\" d=\"M 60.803125 36.234279 \n",
       "L 104.660268 70.041286 \n",
       "L 148.517411 101.722065 \n",
       "L 192.374554 111.715331 \n",
       "L 236.231696 125.961051 \n",
       "L 280.088839 131.063995 \n",
       "L 323.945982 136.166939 \n",
       "L 367.803125 138.293166 \n",
       "L 411.660268 138.293166 \n",
       "L 455.517411 144.671847 \n",
       "L 499.374554 143.608733 \n",
       "L 543.231696 143.821356 \n",
       "L 587.088839 144.671847 \n",
       "L 630.945982 143.396111 \n",
       "L 674.803125 145.097092 \n",
       "\" style=\"fill:none;stroke:#00ffff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 30.103125 319.118125 \n",
       "L 30.103125 22.318125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 705.503125 319.118125 \n",
       "L 705.503125 22.318125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 30.103125 319.118125 \n",
       "L 705.503125 319.118125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 30.103125 22.318125 \n",
       "L 705.503125 22.318125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_17\">\n",
       "    <!-- CrossEntropy Loss -->\n",
       "    <g transform=\"translate(312.982812 16.318125)scale(0.12 -0.12)\">\n",
       "     <defs>\n",
       "      <path d=\"M 64.40625 67.28125 \n",
       "L 64.40625 56.890625 \n",
       "Q 59.421875 61.53125 53.78125 63.8125 \n",
       "Q 48.140625 66.109375 41.796875 66.109375 \n",
       "Q 29.296875 66.109375 22.65625 58.46875 \n",
       "Q 16.015625 50.828125 16.015625 36.375 \n",
       "Q 16.015625 21.96875 22.65625 14.328125 \n",
       "Q 29.296875 6.6875 41.796875 6.6875 \n",
       "Q 48.140625 6.6875 53.78125 8.984375 \n",
       "Q 59.421875 11.28125 64.40625 15.921875 \n",
       "L 64.40625 5.609375 \n",
       "Q 59.234375 2.09375 53.4375 0.328125 \n",
       "Q 47.65625 -1.421875 41.21875 -1.421875 \n",
       "Q 24.65625 -1.421875 15.125 8.703125 \n",
       "Q 5.609375 18.84375 5.609375 36.375 \n",
       "Q 5.609375 53.953125 15.125 64.078125 \n",
       "Q 24.65625 74.21875 41.21875 74.21875 \n",
       "Q 47.75 74.21875 53.53125 72.484375 \n",
       "Q 59.328125 70.75 64.40625 67.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-67\"/>\n",
       "      <path d=\"M 41.109375 46.296875 \n",
       "Q 39.59375 47.171875 37.8125 47.578125 \n",
       "Q 36.03125 48 33.890625 48 \n",
       "Q 26.265625 48 22.1875 43.046875 \n",
       "Q 18.109375 38.09375 18.109375 28.8125 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 20.953125 51.171875 25.484375 53.578125 \n",
       "Q 30.03125 56 36.53125 56 \n",
       "Q 37.453125 56 38.578125 55.875 \n",
       "Q 39.703125 55.765625 41.0625 55.515625 \n",
       "z\n",
       "\" id=\"DejaVuSans-114\"/>\n",
       "      <path d=\"M 44.28125 53.078125 \n",
       "L 44.28125 44.578125 \n",
       "Q 40.484375 46.53125 36.375 47.5 \n",
       "Q 32.28125 48.484375 27.875 48.484375 \n",
       "Q 21.1875 48.484375 17.84375 46.4375 \n",
       "Q 14.5 44.390625 14.5 40.28125 \n",
       "Q 14.5 37.15625 16.890625 35.375 \n",
       "Q 19.28125 33.59375 26.515625 31.984375 \n",
       "L 29.59375 31.296875 \n",
       "Q 39.15625 29.25 43.1875 25.515625 \n",
       "Q 47.21875 21.78125 47.21875 15.09375 \n",
       "Q 47.21875 7.46875 41.1875 3.015625 \n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \n",
       "Q 10.6875 0.296875 5.421875 2 \n",
       "L 5.421875 11.28125 \n",
       "Q 10.40625 8.6875 15.234375 7.390625 \n",
       "Q 20.0625 6.109375 24.8125 6.109375 \n",
       "Q 31.15625 6.109375 34.5625 8.28125 \n",
       "Q 37.984375 10.453125 37.984375 14.40625 \n",
       "Q 37.984375 18.0625 35.515625 20.015625 \n",
       "Q 33.0625 21.96875 24.703125 23.78125 \n",
       "L 21.578125 24.515625 \n",
       "Q 13.234375 26.265625 9.515625 29.90625 \n",
       "Q 5.8125 33.546875 5.8125 39.890625 \n",
       "Q 5.8125 47.609375 11.28125 51.796875 \n",
       "Q 16.75 56 26.8125 56 \n",
       "Q 31.78125 56 36.171875 55.265625 \n",
       "Q 40.578125 54.546875 44.28125 53.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-115\"/>\n",
       "      <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-110\"/>\n",
       "      <path d=\"M 18.3125 70.21875 \n",
       "L 18.3125 54.6875 \n",
       "L 36.8125 54.6875 \n",
       "L 36.8125 47.703125 \n",
       "L 18.3125 47.703125 \n",
       "L 18.3125 18.015625 \n",
       "Q 18.3125 11.328125 20.140625 9.421875 \n",
       "Q 21.96875 7.515625 27.59375 7.515625 \n",
       "L 36.8125 7.515625 \n",
       "L 36.8125 0 \n",
       "L 27.59375 0 \n",
       "Q 17.1875 0 13.234375 3.875 \n",
       "Q 9.28125 7.765625 9.28125 18.015625 \n",
       "L 9.28125 47.703125 \n",
       "L 2.6875 47.703125 \n",
       "L 2.6875 54.6875 \n",
       "L 9.28125 54.6875 \n",
       "L 9.28125 70.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-116\"/>\n",
       "      <path d=\"M 32.171875 -5.078125 \n",
       "Q 28.375 -14.84375 24.75 -17.8125 \n",
       "Q 21.140625 -20.796875 15.09375 -20.796875 \n",
       "L 7.90625 -20.796875 \n",
       "L 7.90625 -13.28125 \n",
       "L 13.1875 -13.28125 \n",
       "Q 16.890625 -13.28125 18.9375 -11.515625 \n",
       "Q 21 -9.765625 23.484375 -3.21875 \n",
       "L 25.09375 0.875 \n",
       "L 2.984375 54.6875 \n",
       "L 12.5 54.6875 \n",
       "L 29.59375 11.921875 \n",
       "L 46.6875 54.6875 \n",
       "L 56.203125 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-121\"/>\n",
       "      <path id=\"DejaVuSans-32\"/>\n",
       "      <path d=\"M 9.8125 72.90625 \n",
       "L 19.671875 72.90625 \n",
       "L 19.671875 8.296875 \n",
       "L 55.171875 8.296875 \n",
       "L 55.171875 0 \n",
       "L 9.8125 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-76\"/>\n",
       "     </defs>\n",
       "     <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "     <use x=\"69.824219\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"108.6875\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "     <use x=\"169.869141\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "     <use x=\"221.96875\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "     <use x=\"274.068359\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"337.251953\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "     <use x=\"400.630859\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "     <use x=\"439.839844\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"478.703125\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "     <use x=\"539.884766\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     <use x=\"603.361328\" xlink:href=\"#DejaVuSans-121\"/>\n",
       "     <use x=\"662.541016\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"694.328125\" xlink:href=\"#DejaVuSans-76\"/>\n",
       "     <use x=\"748.291016\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "     <use x=\"809.472656\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "     <use x=\"861.572266\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 37.103125 314.118125 \n",
       "L 160.320313 314.118125 \n",
       "Q 162.320313 314.118125 162.320313 312.118125 \n",
       "L 162.320313 107.624375 \n",
       "Q 162.320313 105.624375 160.320313 105.624375 \n",
       "L 37.103125 105.624375 \n",
       "Q 35.103125 105.624375 35.103125 107.624375 \n",
       "L 35.103125 312.118125 \n",
       "Q 35.103125 314.118125 37.103125 314.118125 \n",
       "z\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_45\">\n",
       "     <path d=\"M 39.103125 113.722812 \n",
       "L 59.103125 113.722812 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_46\"/>\n",
       "    <g id=\"text_18\">\n",
       "     <!-- Base -->\n",
       "     <g transform=\"translate(67.103125 117.222812)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 19.671875 34.8125 \n",
       "L 19.671875 8.109375 \n",
       "L 35.5 8.109375 \n",
       "Q 43.453125 8.109375 47.28125 11.40625 \n",
       "Q 51.125 14.703125 51.125 21.484375 \n",
       "Q 51.125 28.328125 47.28125 31.5625 \n",
       "Q 43.453125 34.8125 35.5 34.8125 \n",
       "z\n",
       "M 19.671875 64.796875 \n",
       "L 19.671875 42.828125 \n",
       "L 34.28125 42.828125 \n",
       "Q 41.5 42.828125 45.03125 45.53125 \n",
       "Q 48.578125 48.25 48.578125 53.8125 \n",
       "Q 48.578125 59.328125 45.03125 62.0625 \n",
       "Q 41.5 64.796875 34.28125 64.796875 \n",
       "z\n",
       "M 9.8125 72.90625 \n",
       "L 35.015625 72.90625 \n",
       "Q 46.296875 72.90625 52.390625 68.21875 \n",
       "Q 58.5 63.53125 58.5 54.890625 \n",
       "Q 58.5 48.1875 55.375 44.234375 \n",
       "Q 52.25 40.28125 46.1875 39.3125 \n",
       "Q 53.46875 37.75 57.5 32.78125 \n",
       "Q 61.53125 27.828125 61.53125 20.40625 \n",
       "Q 61.53125 10.640625 54.890625 5.3125 \n",
       "Q 48.25 0 35.984375 0 \n",
       "L 9.8125 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-66\"/>\n",
       "       <path d=\"M 34.28125 27.484375 \n",
       "Q 23.390625 27.484375 19.1875 25 \n",
       "Q 14.984375 22.515625 14.984375 16.5 \n",
       "Q 14.984375 11.71875 18.140625 8.90625 \n",
       "Q 21.296875 6.109375 26.703125 6.109375 \n",
       "Q 34.1875 6.109375 38.703125 11.40625 \n",
       "Q 43.21875 16.703125 43.21875 25.484375 \n",
       "L 43.21875 27.484375 \n",
       "z\n",
       "M 52.203125 31.203125 \n",
       "L 52.203125 0 \n",
       "L 43.21875 0 \n",
       "L 43.21875 8.296875 \n",
       "Q 40.140625 3.328125 35.546875 0.953125 \n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \n",
       "Q 6 8.015625 6 15.921875 \n",
       "Q 6 25.140625 12.171875 29.828125 \n",
       "Q 18.359375 34.515625 30.609375 34.515625 \n",
       "L 43.21875 34.515625 \n",
       "L 43.21875 35.40625 \n",
       "Q 43.21875 41.609375 39.140625 45 \n",
       "Q 35.0625 48.390625 27.6875 48.390625 \n",
       "Q 23 48.390625 18.546875 47.265625 \n",
       "Q 14.109375 46.140625 10.015625 43.890625 \n",
       "L 10.015625 52.203125 \n",
       "Q 14.9375 54.109375 19.578125 55.046875 \n",
       "Q 24.21875 56 28.609375 56 \n",
       "Q 40.484375 56 46.34375 49.84375 \n",
       "Q 52.203125 43.703125 52.203125 31.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-97\"/>\n",
       "       <path d=\"M 56.203125 29.59375 \n",
       "L 56.203125 25.203125 \n",
       "L 14.890625 25.203125 \n",
       "Q 15.484375 15.921875 20.484375 11.0625 \n",
       "Q 25.484375 6.203125 34.421875 6.203125 \n",
       "Q 39.59375 6.203125 44.453125 7.46875 \n",
       "Q 49.3125 8.734375 54.109375 11.28125 \n",
       "L 54.109375 2.78125 \n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \n",
       "Q 5.515625 13.8125 5.515625 26.8125 \n",
       "Q 5.515625 40.234375 12.765625 48.109375 \n",
       "Q 20.015625 56 32.328125 56 \n",
       "Q 43.359375 56 49.78125 48.890625 \n",
       "Q 56.203125 41.796875 56.203125 29.59375 \n",
       "z\n",
       "M 47.21875 32.234375 \n",
       "Q 47.125 39.59375 43.09375 43.984375 \n",
       "Q 39.0625 48.390625 32.421875 48.390625 \n",
       "Q 24.90625 48.390625 20.390625 44.140625 \n",
       "Q 15.875 39.890625 15.1875 32.171875 \n",
       "z\n",
       "\" id=\"DejaVuSans-101\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use x=\"68.603516\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"129.882812\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"181.982422\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_47\">\n",
       "     <path d=\"M 39.103125 128.400937 \n",
       "L 59.103125 128.400937 \n",
       "\" style=\"fill:none;stroke:#ff0000;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_48\"/>\n",
       "    <g id=\"text_19\">\n",
       "     <!-- VarDropCheck -->\n",
       "     <g transform=\"translate(67.103125 131.900937)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 28.609375 0 \n",
       "L 0.78125 72.90625 \n",
       "L 11.078125 72.90625 \n",
       "L 34.1875 11.53125 \n",
       "L 57.328125 72.90625 \n",
       "L 67.578125 72.90625 \n",
       "L 39.796875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-86\"/>\n",
       "       <path d=\"M 19.671875 64.796875 \n",
       "L 19.671875 8.109375 \n",
       "L 31.59375 8.109375 \n",
       "Q 46.6875 8.109375 53.6875 14.9375 \n",
       "Q 60.6875 21.78125 60.6875 36.53125 \n",
       "Q 60.6875 51.171875 53.6875 57.984375 \n",
       "Q 46.6875 64.796875 31.59375 64.796875 \n",
       "z\n",
       "M 9.8125 72.90625 \n",
       "L 30.078125 72.90625 \n",
       "Q 51.265625 72.90625 61.171875 64.09375 \n",
       "Q 71.09375 55.28125 71.09375 36.53125 \n",
       "Q 71.09375 17.671875 61.125 8.828125 \n",
       "Q 51.171875 0 30.078125 0 \n",
       "L 9.8125 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-68\"/>\n",
       "       <path d=\"M 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "L 18.109375 31.109375 \n",
       "L 44.921875 54.6875 \n",
       "L 56.390625 54.6875 \n",
       "L 27.390625 29.109375 \n",
       "L 57.625 0 \n",
       "L 45.90625 0 \n",
       "L 18.109375 26.703125 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-107\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"60.658203\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"121.9375\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"163.050781\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"240.052734\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"278.916016\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"340.097656\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"403.574219\" xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"473.398438\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"536.777344\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"598.300781\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"653.28125\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_49\">\n",
       "     <path d=\"M 39.103125 143.079062 \n",
       "L 59.103125 143.079062 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_50\"/>\n",
       "    <g id=\"text_20\">\n",
       "     <!-- VarDrop -->\n",
       "     <g transform=\"translate(67.103125 146.579062)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"60.658203\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"121.9375\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"163.050781\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"240.052734\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"278.916016\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"340.097656\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_51\">\n",
       "     <path d=\"M 39.103125 157.757187 \n",
       "L 59.103125 157.757187 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_52\"/>\n",
       "    <g id=\"text_21\">\n",
       "     <!-- FastVarDropCheck -->\n",
       "     <g transform=\"translate(67.103125 161.257187)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 9.8125 72.90625 \n",
       "L 51.703125 72.90625 \n",
       "L 51.703125 64.59375 \n",
       "L 19.671875 64.59375 \n",
       "L 19.671875 43.109375 \n",
       "L 48.578125 43.109375 \n",
       "L 48.578125 34.8125 \n",
       "L 19.671875 34.8125 \n",
       "L 19.671875 0 \n",
       "L 9.8125 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-70\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"48.394531\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"109.673828\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"161.773438\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"200.982422\" xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"261.640625\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"322.919922\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"364.033203\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"441.035156\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"479.898438\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"541.080078\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"604.556641\" xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"674.380859\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"737.759766\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"799.283203\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"854.263672\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_53\">\n",
       "     <path d=\"M 39.103125 172.435312 \n",
       "L 59.103125 172.435312 \n",
       "\" style=\"fill:none;stroke:#ffa500;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_54\"/>\n",
       "    <g id=\"text_22\">\n",
       "     <!-- FastVarDrop -->\n",
       "     <g transform=\"translate(67.103125 175.935312)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"48.394531\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"109.673828\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"161.773438\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"200.982422\" xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"261.640625\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"322.919922\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"364.033203\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"441.035156\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"479.898438\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"541.080078\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_55\">\n",
       "     <path d=\"M 39.103125 187.113437 \n",
       "L 59.103125 187.113437 \n",
       "\" style=\"fill:none;stroke:#800080;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_56\"/>\n",
       "    <g id=\"text_23\">\n",
       "     <!-- HandmadeCheck -->\n",
       "     <g transform=\"translate(67.103125 190.613437)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 9.8125 72.90625 \n",
       "L 19.671875 72.90625 \n",
       "L 19.671875 43.015625 \n",
       "L 55.515625 43.015625 \n",
       "L 55.515625 72.90625 \n",
       "L 65.375 72.90625 \n",
       "L 65.375 0 \n",
       "L 55.515625 0 \n",
       "L 55.515625 34.71875 \n",
       "L 19.671875 34.71875 \n",
       "L 19.671875 0 \n",
       "L 9.8125 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-72\"/>\n",
       "       <path d=\"M 45.40625 46.390625 \n",
       "L 45.40625 75.984375 \n",
       "L 54.390625 75.984375 \n",
       "L 54.390625 0 \n",
       "L 45.40625 0 \n",
       "L 45.40625 8.203125 \n",
       "Q 42.578125 3.328125 38.25 0.953125 \n",
       "Q 33.9375 -1.421875 27.875 -1.421875 \n",
       "Q 17.96875 -1.421875 11.734375 6.484375 \n",
       "Q 5.515625 14.40625 5.515625 27.296875 \n",
       "Q 5.515625 40.1875 11.734375 48.09375 \n",
       "Q 17.96875 56 27.875 56 \n",
       "Q 33.9375 56 38.25 53.625 \n",
       "Q 42.578125 51.265625 45.40625 46.390625 \n",
       "z\n",
       "M 14.796875 27.296875 \n",
       "Q 14.796875 17.390625 18.875 11.75 \n",
       "Q 22.953125 6.109375 30.078125 6.109375 \n",
       "Q 37.203125 6.109375 41.296875 11.75 \n",
       "Q 45.40625 17.390625 45.40625 27.296875 \n",
       "Q 45.40625 37.203125 41.296875 42.84375 \n",
       "Q 37.203125 48.484375 30.078125 48.484375 \n",
       "Q 22.953125 48.484375 18.875 42.84375 \n",
       "Q 14.796875 37.203125 14.796875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-100\"/>\n",
       "       <path d=\"M 52 44.1875 \n",
       "Q 55.375 50.25 60.0625 53.125 \n",
       "Q 64.75 56 71.09375 56 \n",
       "Q 79.640625 56 84.28125 50.015625 \n",
       "Q 88.921875 44.046875 88.921875 33.015625 \n",
       "L 88.921875 0 \n",
       "L 79.890625 0 \n",
       "L 79.890625 32.71875 \n",
       "Q 79.890625 40.578125 77.09375 44.375 \n",
       "Q 74.3125 48.1875 68.609375 48.1875 \n",
       "Q 61.625 48.1875 57.5625 43.546875 \n",
       "Q 53.515625 38.921875 53.515625 30.90625 \n",
       "L 53.515625 0 \n",
       "L 44.484375 0 \n",
       "L 44.484375 32.71875 \n",
       "Q 44.484375 40.625 41.703125 44.40625 \n",
       "Q 38.921875 48.1875 33.109375 48.1875 \n",
       "Q 26.21875 48.1875 22.15625 43.53125 \n",
       "Q 18.109375 38.875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.1875 51.21875 25.484375 53.609375 \n",
       "Q 29.78125 56 35.6875 56 \n",
       "Q 41.65625 56 45.828125 52.96875 \n",
       "Q 50 49.953125 52 44.1875 \n",
       "z\n",
       "\" id=\"DejaVuSans-109\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"75.195312\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"136.474609\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "      <use x=\"199.853516\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"263.330078\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "      <use x=\"360.742188\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"422.021484\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"485.498047\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"547.021484\" xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"616.845703\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"680.224609\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"741.748047\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"796.728516\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_57\">\n",
       "     <path d=\"M 39.103125 201.791562 \n",
       "L 59.103125 201.791562 \n",
       "\" style=\"fill:none;stroke:#00ffff;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_58\"/>\n",
       "    <g id=\"text_24\">\n",
       "     <!-- Handmade -->\n",
       "     <g transform=\"translate(67.103125 205.291562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"75.195312\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"136.474609\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "      <use x=\"199.853516\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"263.330078\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "      <use x=\"360.742188\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"422.021484\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"485.498047\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_59\">\n",
       "     <path d=\"M 39.103125 216.469687 \n",
       "L 59.103125 216.469687 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_60\"/>\n",
       "    <g id=\"text_25\">\n",
       "     <!-- Base -->\n",
       "     <g transform=\"translate(67.103125 219.969687)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use x=\"68.603516\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"129.882812\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"181.982422\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_61\">\n",
       "     <path d=\"M 39.103125 231.147812 \n",
       "L 59.103125 231.147812 \n",
       "\" style=\"fill:none;stroke:#ff0000;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_62\"/>\n",
       "    <g id=\"text_26\">\n",
       "     <!-- VarDropCheck -->\n",
       "     <g transform=\"translate(67.103125 234.647812)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"60.658203\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"121.9375\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"163.050781\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"240.052734\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"278.916016\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"340.097656\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"403.574219\" xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"473.398438\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"536.777344\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"598.300781\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"653.28125\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_63\">\n",
       "     <path d=\"M 39.103125 245.825937 \n",
       "L 59.103125 245.825937 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_64\"/>\n",
       "    <g id=\"text_27\">\n",
       "     <!-- VarDrop -->\n",
       "     <g transform=\"translate(67.103125 249.325937)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"60.658203\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"121.9375\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"163.050781\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"240.052734\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"278.916016\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"340.097656\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_65\">\n",
       "     <path d=\"M 39.103125 260.504062 \n",
       "L 59.103125 260.504062 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_66\"/>\n",
       "    <g id=\"text_28\">\n",
       "     <!-- FastVarDropCheck -->\n",
       "     <g transform=\"translate(67.103125 264.004062)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"48.394531\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"109.673828\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"161.773438\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"200.982422\" xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"261.640625\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"322.919922\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"364.033203\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"441.035156\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"479.898438\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"541.080078\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"604.556641\" xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"674.380859\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"737.759766\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"799.283203\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"854.263672\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_67\">\n",
       "     <path d=\"M 39.103125 275.182187 \n",
       "L 59.103125 275.182187 \n",
       "\" style=\"fill:none;stroke:#ffa500;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_68\"/>\n",
       "    <g id=\"text_29\">\n",
       "     <!-- FastVarDrop -->\n",
       "     <g transform=\"translate(67.103125 278.682187)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"48.394531\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"109.673828\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"161.773438\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"200.982422\" xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"261.640625\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"322.919922\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"364.033203\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"441.035156\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"479.898438\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"541.080078\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_69\">\n",
       "     <path d=\"M 39.103125 289.860312 \n",
       "L 59.103125 289.860312 \n",
       "\" style=\"fill:none;stroke:#800080;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_70\"/>\n",
       "    <g id=\"text_30\">\n",
       "     <!-- HandmadeCheck -->\n",
       "     <g transform=\"translate(67.103125 293.360312)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"75.195312\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"136.474609\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "      <use x=\"199.853516\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"263.330078\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "      <use x=\"360.742188\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"422.021484\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"485.498047\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"547.021484\" xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"616.845703\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"680.224609\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"741.748047\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"796.728516\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_71\">\n",
       "     <path d=\"M 39.103125 304.538437 \n",
       "L 59.103125 304.538437 \n",
       "\" style=\"fill:none;stroke:#00ffff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_72\"/>\n",
       "    <g id=\"text_31\">\n",
       "     <!-- Handmade -->\n",
       "     <g transform=\"translate(67.103125 308.038437)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"75.195312\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"136.474609\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "      <use x=\"199.853516\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"263.330078\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "      <use x=\"360.742188\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"422.021484\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"485.498047\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 30.103125 673.718125 \n",
       "L 705.503125 673.718125 \n",
       "L 705.503125 376.918125 \n",
       "L 30.103125 376.918125 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_73\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 673.718125 \n",
       "L 60.803125 376.918125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_74\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"60.803125\" xlink:href=\"#m5670c8524a\" y=\"673.718125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_32\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(57.621875 688.316562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_10\">\n",
       "     <g id=\"line2d_75\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 148.517411 673.718125 \n",
       "L 148.517411 376.918125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_76\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"148.517411\" xlink:href=\"#m5670c8524a\" y=\"673.718125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_33\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(145.336161 688.316562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_11\">\n",
       "     <g id=\"line2d_77\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 236.231696 673.718125 \n",
       "L 236.231696 376.918125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_78\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"236.231696\" xlink:href=\"#m5670c8524a\" y=\"673.718125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_34\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(233.050446 688.316562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_12\">\n",
       "     <g id=\"line2d_79\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 323.945982 673.718125 \n",
       "L 323.945982 376.918125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_80\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"323.945982\" xlink:href=\"#m5670c8524a\" y=\"673.718125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_35\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(320.764732 688.316562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_13\">\n",
       "     <g id=\"line2d_81\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 411.660268 673.718125 \n",
       "L 411.660268 376.918125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_82\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"411.660268\" xlink:href=\"#m5670c8524a\" y=\"673.718125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_36\">\n",
       "      <!-- 8 -->\n",
       "      <g transform=\"translate(408.479018 688.316562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-56\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_14\">\n",
       "     <g id=\"line2d_83\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 499.374554 673.718125 \n",
       "L 499.374554 376.918125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_84\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"499.374554\" xlink:href=\"#m5670c8524a\" y=\"673.718125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_37\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(493.012054 688.316562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_15\">\n",
       "     <g id=\"line2d_85\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 587.088839 673.718125 \n",
       "L 587.088839 376.918125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_86\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"587.088839\" xlink:href=\"#m5670c8524a\" y=\"673.718125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_38\">\n",
       "      <!-- 12 -->\n",
       "      <g transform=\"translate(580.726339 688.316562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_16\">\n",
       "     <g id=\"line2d_87\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 674.803125 673.718125 \n",
       "L 674.803125 376.918125 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_88\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"674.803125\" xlink:href=\"#m5670c8524a\" y=\"673.718125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_39\">\n",
       "      <!-- 14 -->\n",
       "      <g transform=\"translate(668.440625 688.316562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_40\">\n",
       "     <!-- Epoch -->\n",
       "     <g transform=\"translate(352.492187 701.994687)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"63.183594\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"126.660156\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"187.841797\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"242.822266\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_89\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 30.103125 666.998715 \n",
       "L 705.503125 666.998715 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_90\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"666.998715\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_41\">\n",
       "      <!-- 0.2 -->\n",
       "      <g transform=\"translate(7.2 670.797933)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_91\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 30.103125 614.910263 \n",
       "L 705.503125 614.910263 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_92\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"614.910263\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_42\">\n",
       "      <!-- 0.3 -->\n",
       "      <g transform=\"translate(7.2 618.709481)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "z\n",
       "\" id=\"DejaVuSans-51\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-51\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_10\">\n",
       "     <g id=\"line2d_93\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 30.103125 562.821811 \n",
       "L 705.503125 562.821811 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_94\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"562.821811\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_43\">\n",
       "      <!-- 0.4 -->\n",
       "      <g transform=\"translate(7.2 566.621029)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_11\">\n",
       "     <g id=\"line2d_95\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 30.103125 510.733358 \n",
       "L 705.503125 510.733358 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_96\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"510.733358\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_44\">\n",
       "      <!-- 0.5 -->\n",
       "      <g transform=\"translate(7.2 514.532577)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-53\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_12\">\n",
       "     <g id=\"line2d_97\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 30.103125 458.644906 \n",
       "L 705.503125 458.644906 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_98\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"458.644906\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_45\">\n",
       "      <!-- 0.6 -->\n",
       "      <g transform=\"translate(7.2 462.444125)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-54\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_13\">\n",
       "     <g id=\"line2d_99\">\n",
       "      <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 30.103125 406.556454 \n",
       "L 705.503125 406.556454 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_100\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#md23709666f\" y=\"406.556454\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_46\">\n",
       "      <!-- 0.7 -->\n",
       "      <g transform=\"translate(7.2 410.355673)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 8.203125 72.90625 \n",
       "L 55.078125 72.90625 \n",
       "L 55.078125 68.703125 \n",
       "L 28.609375 0 \n",
       "L 18.3125 0 \n",
       "L 43.21875 64.59375 \n",
       "L 8.203125 64.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-55\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-55\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_101\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 626.369722 \n",
       "L 104.660268 599.283727 \n",
       "L 148.517411 580.531884 \n",
       "L 192.374554 566.988887 \n",
       "L 236.231696 556.571196 \n",
       "L 280.088839 544.590852 \n",
       "L 323.945982 526.359894 \n",
       "L 367.803125 516.983973 \n",
       "L 411.660268 502.920091 \n",
       "L 455.517411 489.377093 \n",
       "L 499.374554 473.229673 \n",
       "L 543.231696 460.728444 \n",
       "L 587.088839 440.934833 \n",
       "L 630.945982 413.327953 \n",
       "L 674.803125 393.013457 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_102\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 650.851295 \n",
       "L 104.660268 598.241958 \n",
       "L 148.517411 575.843924 \n",
       "L 192.374554 561.259157 \n",
       "L 236.231696 546.67439 \n",
       "L 280.088839 538.861123 \n",
       "L 323.945982 528.443432 \n",
       "L 367.803125 517.504857 \n",
       "L 411.660268 500.315668 \n",
       "L 455.517411 484.168248 \n",
       "L 499.374554 472.708788 \n",
       "L 543.231696 451.352523 \n",
       "L 587.088839 429.475373 \n",
       "L 630.945982 410.202646 \n",
       "L 674.803125 407.598223 \n",
       "\" style=\"fill:none;stroke:#ff0000;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_103\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 645.121565 \n",
       "L 104.660268 604.492572 \n",
       "L 148.517411 592.512228 \n",
       "L 192.374554 572.718616 \n",
       "L 236.231696 562.821811 \n",
       "L 280.088839 552.925005 \n",
       "L 323.945982 550.320582 \n",
       "L 367.803125 541.98643 \n",
       "L 411.660268 541.465545 \n",
       "L 455.517411 531.047855 \n",
       "L 499.374554 526.359894 \n",
       "L 543.231696 523.755471 \n",
       "L 587.088839 525.318125 \n",
       "L 630.945982 513.858666 \n",
       "L 674.803125 510.733358 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_104\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 658.143678 \n",
       "L 104.660268 608.659648 \n",
       "L 148.517411 578.448346 \n",
       "L 192.374554 565.426233 \n",
       "L 236.231696 550.841467 \n",
       "L 280.088839 540.423776 \n",
       "L 323.945982 526.880779 \n",
       "L 367.803125 519.588395 \n",
       "L 411.660268 506.566282 \n",
       "L 455.517411 494.065054 \n",
       "L 499.374554 466.979059 \n",
       "L 543.231696 452.394292 \n",
       "L 587.088839 436.767756 \n",
       "L 630.945982 412.286184 \n",
       "L 674.803125 390.409034 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_105\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 641.996258 \n",
       "L 104.660268 597.721073 \n",
       "L 148.517411 588.345152 \n",
       "L 192.374554 581.573653 \n",
       "L 236.231696 571.155963 \n",
       "L 280.088839 565.947118 \n",
       "L 323.945982 557.092081 \n",
       "L 367.803125 552.40412 \n",
       "L 411.660268 549.799697 \n",
       "L 455.517411 541.98643 \n",
       "L 499.374554 534.173162 \n",
       "L 543.231696 528.964317 \n",
       "L 587.088839 528.443432 \n",
       "L 630.945982 519.067511 \n",
       "L 674.803125 513.858666 \n",
       "\" style=\"fill:none;stroke:#ffa500;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_106\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 643.558911 \n",
       "L 104.660268 596.679304 \n",
       "L 148.517411 582.094538 \n",
       "L 192.374554 562.821811 \n",
       "L 236.231696 550.841467 \n",
       "L 280.088839 540.944661 \n",
       "L 323.945982 530.006086 \n",
       "L 367.803125 520.10928 \n",
       "L 411.660268 505.524513 \n",
       "L 455.517411 491.981516 \n",
       "L 499.374554 474.792326 \n",
       "L 543.231696 460.728444 \n",
       "L 587.088839 439.372179 \n",
       "L 630.945982 418.536798 \n",
       "L 674.803125 398.743186 \n",
       "\" style=\"fill:none;stroke:#800080;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_107\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 658.664562 \n",
       "L 104.660268 606.055226 \n",
       "L 148.517411 588.345152 \n",
       "L 192.374554 582.094538 \n",
       "L 236.231696 565.947118 \n",
       "L 280.088839 560.738272 \n",
       "L 323.945982 554.487658 \n",
       "L 367.803125 548.757928 \n",
       "L 411.660268 544.590852 \n",
       "L 455.517411 539.902892 \n",
       "L 499.374554 536.777584 \n",
       "L 543.231696 532.089624 \n",
       "L 587.088839 530.006086 \n",
       "L 630.945982 528.443432 \n",
       "L 674.803125 525.318125 \n",
       "\" style=\"fill:none;stroke:#00ffff;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_108\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 623.244415 \n",
       "L 104.660268 599.283727 \n",
       "L 148.517411 586.782498 \n",
       "L 192.374554 577.927462 \n",
       "L 236.231696 570.114194 \n",
       "L 280.088839 573.239501 \n",
       "L 323.945982 564.905349 \n",
       "L 367.803125 564.905349 \n",
       "L 411.660268 569.072425 \n",
       "L 455.517411 571.155963 \n",
       "L 499.374554 573.239501 \n",
       "L 543.231696 569.593309 \n",
       "L 587.088839 574.802154 \n",
       "L 630.945982 575.323039 \n",
       "L 674.803125 582.094538 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_109\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 649.288641 \n",
       "L 104.660268 600.325496 \n",
       "L 148.517411 585.740729 \n",
       "L 192.374554 574.802154 \n",
       "L 236.231696 569.072425 \n",
       "L 280.088839 565.426233 \n",
       "L 323.945982 563.86358 \n",
       "L 367.803125 562.300926 \n",
       "L 411.660268 565.426233 \n",
       "L 455.517411 566.468002 \n",
       "L 499.374554 566.468002 \n",
       "L 543.231696 575.323039 \n",
       "L 587.088839 575.323039 \n",
       "L 630.945982 578.969231 \n",
       "L 674.803125 577.406577 \n",
       "\" style=\"fill:none;stroke:#ff0000;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_110\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 643.558911 \n",
       "L 104.660268 603.971688 \n",
       "L 148.517411 590.42869 \n",
       "L 192.374554 577.406577 \n",
       "L 236.231696 571.155963 \n",
       "L 280.088839 564.384464 \n",
       "L 323.945982 561.259157 \n",
       "L 367.803125 561.780041 \n",
       "L 411.660268 558.13385 \n",
       "L 455.517411 557.612965 \n",
       "L 499.374554 557.612965 \n",
       "L 543.231696 556.571196 \n",
       "L 587.088839 556.571196 \n",
       "L 630.945982 553.966774 \n",
       "L 674.803125 555.529427 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_111\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 659.706331 \n",
       "L 104.660268 607.617879 \n",
       "L 148.517411 588.345152 \n",
       "L 192.374554 575.323039 \n",
       "L 236.231696 570.114194 \n",
       "L 280.088839 566.468002 \n",
       "L 323.945982 566.468002 \n",
       "L 367.803125 564.384464 \n",
       "L 411.660268 569.072425 \n",
       "L 455.517411 565.426233 \n",
       "L 499.374554 572.197732 \n",
       "L 543.231696 579.490115 \n",
       "L 587.088839 573.760385 \n",
       "L 630.945982 580.011 \n",
       "L 674.803125 580.011 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_112\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 643.038027 \n",
       "L 104.660268 598.762842 \n",
       "L 148.517411 589.907806 \n",
       "L 192.374554 587.303383 \n",
       "L 236.231696 578.969231 \n",
       "L 280.088839 572.718616 \n",
       "L 323.945982 569.593309 \n",
       "L 367.803125 565.947118 \n",
       "L 411.660268 570.635078 \n",
       "L 455.517411 563.342695 \n",
       "L 499.374554 558.654734 \n",
       "L 543.231696 559.696503 \n",
       "L 587.088839 556.571196 \n",
       "L 630.945982 557.092081 \n",
       "L 674.803125 556.571196 \n",
       "\" style=\"fill:none;stroke:#ffa500;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_113\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 644.079796 \n",
       "L 104.660268 595.637535 \n",
       "L 148.517411 588.345152 \n",
       "L 192.374554 575.843924 \n",
       "L 236.231696 572.197732 \n",
       "L 280.088839 568.030656 \n",
       "L 323.945982 565.426233 \n",
       "L 367.803125 571.155963 \n",
       "L 411.660268 569.593309 \n",
       "L 455.517411 565.947118 \n",
       "L 499.374554 569.593309 \n",
       "L 543.231696 570.114194 \n",
       "L 587.088839 572.197732 \n",
       "L 630.945982 576.364808 \n",
       "L 674.803125 582.094538 \n",
       "\" style=\"fill:none;stroke:#800080;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_114\">\n",
       "    <path clip-path=\"url(#pa3bc0c58dd)\" d=\"M 60.803125 660.227216 \n",
       "L 104.660268 604.492572 \n",
       "L 148.517411 587.824268 \n",
       "L 192.374554 591.991344 \n",
       "L 236.231696 576.364808 \n",
       "L 280.088839 573.760385 \n",
       "L 323.945982 566.468002 \n",
       "L 367.803125 563.86358 \n",
       "L 411.660268 564.384464 \n",
       "L 455.517411 560.217388 \n",
       "L 499.374554 558.654734 \n",
       "L 543.231696 560.217388 \n",
       "L 587.088839 555.529427 \n",
       "L 630.945982 557.612965 \n",
       "L 674.803125 553.966774 \n",
       "\" style=\"fill:none;stroke:#00ffff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 30.103125 673.718125 \n",
       "L 30.103125 376.918125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 705.503125 673.718125 \n",
       "L 705.503125 376.918125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 30.103125 673.718125 \n",
       "L 705.503125 673.718125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 30.103125 376.918125 \n",
       "L 705.503125 376.918125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_47\">\n",
       "    <!-- Accuracy -->\n",
       "    <g transform=\"translate(340.409375 370.918125)scale(0.12 -0.12)\">\n",
       "     <defs>\n",
       "      <path d=\"M 34.1875 63.1875 \n",
       "L 20.796875 26.90625 \n",
       "L 47.609375 26.90625 \n",
       "z\n",
       "M 28.609375 72.90625 \n",
       "L 39.796875 72.90625 \n",
       "L 67.578125 0 \n",
       "L 57.328125 0 \n",
       "L 50.6875 18.703125 \n",
       "L 17.828125 18.703125 \n",
       "L 11.1875 0 \n",
       "L 0.78125 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-65\"/>\n",
       "      <path d=\"M 8.5 21.578125 \n",
       "L 8.5 54.6875 \n",
       "L 17.484375 54.6875 \n",
       "L 17.484375 21.921875 \n",
       "Q 17.484375 14.15625 20.5 10.265625 \n",
       "Q 23.53125 6.390625 29.59375 6.390625 \n",
       "Q 36.859375 6.390625 41.078125 11.03125 \n",
       "Q 45.3125 15.671875 45.3125 23.6875 \n",
       "L 45.3125 54.6875 \n",
       "L 54.296875 54.6875 \n",
       "L 54.296875 0 \n",
       "L 45.3125 0 \n",
       "L 45.3125 8.40625 \n",
       "Q 42.046875 3.421875 37.71875 1 \n",
       "Q 33.40625 -1.421875 27.6875 -1.421875 \n",
       "Q 18.265625 -1.421875 13.375 4.4375 \n",
       "Q 8.5 10.296875 8.5 21.578125 \n",
       "z\n",
       "M 31.109375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-117\"/>\n",
       "     </defs>\n",
       "     <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "     <use x=\"66.658203\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "     <use x=\"121.638672\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "     <use x=\"176.619141\" xlink:href=\"#DejaVuSans-117\"/>\n",
       "     <use x=\"239.998047\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"281.111328\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"342.390625\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "     <use x=\"397.371094\" xlink:href=\"#DejaVuSans-121\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"legend_2\">\n",
       "    <g id=\"patch_13\">\n",
       "     <path d=\"M 37.103125 590.411875 \n",
       "L 160.320313 590.411875 \n",
       "Q 162.320313 590.411875 162.320313 588.411875 \n",
       "L 162.320313 383.918125 \n",
       "Q 162.320313 381.918125 160.320313 381.918125 \n",
       "L 37.103125 381.918125 \n",
       "Q 35.103125 381.918125 35.103125 383.918125 \n",
       "L 35.103125 588.411875 \n",
       "Q 35.103125 590.411875 37.103125 590.411875 \n",
       "z\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_115\">\n",
       "     <path d=\"M 39.103125 390.016562 \n",
       "L 59.103125 390.016562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_116\"/>\n",
       "    <g id=\"text_48\">\n",
       "     <!-- Base -->\n",
       "     <g transform=\"translate(67.103125 393.516562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use x=\"68.603516\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"129.882812\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"181.982422\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_117\">\n",
       "     <path d=\"M 39.103125 404.694687 \n",
       "L 59.103125 404.694687 \n",
       "\" style=\"fill:none;stroke:#ff0000;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_118\"/>\n",
       "    <g id=\"text_49\">\n",
       "     <!-- VarDropCheck -->\n",
       "     <g transform=\"translate(67.103125 408.194687)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"60.658203\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"121.9375\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"163.050781\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"240.052734\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"278.916016\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"340.097656\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"403.574219\" xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"473.398438\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"536.777344\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"598.300781\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"653.28125\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_119\">\n",
       "     <path d=\"M 39.103125 419.372812 \n",
       "L 59.103125 419.372812 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_120\"/>\n",
       "    <g id=\"text_50\">\n",
       "     <!-- VarDrop -->\n",
       "     <g transform=\"translate(67.103125 422.872812)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"60.658203\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"121.9375\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"163.050781\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"240.052734\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"278.916016\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"340.097656\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_121\">\n",
       "     <path d=\"M 39.103125 434.050937 \n",
       "L 59.103125 434.050937 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_122\"/>\n",
       "    <g id=\"text_51\">\n",
       "     <!-- FastVarDropCheck -->\n",
       "     <g transform=\"translate(67.103125 437.550937)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"48.394531\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"109.673828\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"161.773438\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"200.982422\" xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"261.640625\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"322.919922\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"364.033203\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"441.035156\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"479.898438\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"541.080078\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"604.556641\" xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"674.380859\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"737.759766\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"799.283203\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"854.263672\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_123\">\n",
       "     <path d=\"M 39.103125 448.729062 \n",
       "L 59.103125 448.729062 \n",
       "\" style=\"fill:none;stroke:#ffa500;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_124\"/>\n",
       "    <g id=\"text_52\">\n",
       "     <!-- FastVarDrop -->\n",
       "     <g transform=\"translate(67.103125 452.229062)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"48.394531\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"109.673828\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"161.773438\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"200.982422\" xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"261.640625\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"322.919922\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"364.033203\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"441.035156\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"479.898438\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"541.080078\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_125\">\n",
       "     <path d=\"M 39.103125 463.407187 \n",
       "L 59.103125 463.407187 \n",
       "\" style=\"fill:none;stroke:#800080;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_126\"/>\n",
       "    <g id=\"text_53\">\n",
       "     <!-- HandmadeCheck -->\n",
       "     <g transform=\"translate(67.103125 466.907187)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"75.195312\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"136.474609\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "      <use x=\"199.853516\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"263.330078\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "      <use x=\"360.742188\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"422.021484\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"485.498047\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"547.021484\" xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"616.845703\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"680.224609\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"741.748047\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"796.728516\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_127\">\n",
       "     <path d=\"M 39.103125 478.085312 \n",
       "L 59.103125 478.085312 \n",
       "\" style=\"fill:none;stroke:#00ffff;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_128\"/>\n",
       "    <g id=\"text_54\">\n",
       "     <!-- Handmade -->\n",
       "     <g transform=\"translate(67.103125 481.585312)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"75.195312\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"136.474609\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "      <use x=\"199.853516\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"263.330078\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "      <use x=\"360.742188\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"422.021484\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"485.498047\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_129\">\n",
       "     <path d=\"M 39.103125 492.763437 \n",
       "L 59.103125 492.763437 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_130\"/>\n",
       "    <g id=\"text_55\">\n",
       "     <!-- Base -->\n",
       "     <g transform=\"translate(67.103125 496.263437)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-66\"/>\n",
       "      <use x=\"68.603516\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"129.882812\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"181.982422\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_131\">\n",
       "     <path d=\"M 39.103125 507.441562 \n",
       "L 59.103125 507.441562 \n",
       "\" style=\"fill:none;stroke:#ff0000;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_132\"/>\n",
       "    <g id=\"text_56\">\n",
       "     <!-- VarDropCheck -->\n",
       "     <g transform=\"translate(67.103125 510.941562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"60.658203\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"121.9375\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"163.050781\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"240.052734\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"278.916016\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"340.097656\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"403.574219\" xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"473.398438\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"536.777344\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"598.300781\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"653.28125\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_133\">\n",
       "     <path d=\"M 39.103125 522.119687 \n",
       "L 59.103125 522.119687 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_134\"/>\n",
       "    <g id=\"text_57\">\n",
       "     <!-- VarDrop -->\n",
       "     <g transform=\"translate(67.103125 525.619687)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"60.658203\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"121.9375\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"163.050781\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"240.052734\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"278.916016\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"340.097656\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_135\">\n",
       "     <path d=\"M 39.103125 536.797812 \n",
       "L 59.103125 536.797812 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_136\"/>\n",
       "    <g id=\"text_58\">\n",
       "     <!-- FastVarDropCheck -->\n",
       "     <g transform=\"translate(67.103125 540.297812)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"48.394531\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"109.673828\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"161.773438\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"200.982422\" xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"261.640625\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"322.919922\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"364.033203\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"441.035156\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"479.898438\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"541.080078\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"604.556641\" xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"674.380859\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"737.759766\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"799.283203\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"854.263672\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_137\">\n",
       "     <path d=\"M 39.103125 551.475937 \n",
       "L 59.103125 551.475937 \n",
       "\" style=\"fill:none;stroke:#ffa500;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_138\"/>\n",
       "    <g id=\"text_59\">\n",
       "     <!-- FastVarDrop -->\n",
       "     <g transform=\"translate(67.103125 554.975937)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"48.394531\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"109.673828\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"161.773438\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"200.982422\" xlink:href=\"#DejaVuSans-86\"/>\n",
       "      <use x=\"261.640625\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"322.919922\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"364.033203\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"441.035156\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"479.898438\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"541.080078\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_139\">\n",
       "     <path d=\"M 39.103125 566.154062 \n",
       "L 59.103125 566.154062 \n",
       "\" style=\"fill:none;stroke:#800080;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_140\"/>\n",
       "    <g id=\"text_60\">\n",
       "     <!-- HandmadeCheck -->\n",
       "     <g transform=\"translate(67.103125 569.654062)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"75.195312\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"136.474609\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "      <use x=\"199.853516\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"263.330078\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "      <use x=\"360.742188\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"422.021484\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"485.498047\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"547.021484\" xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"616.845703\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"680.224609\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"741.748047\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"796.728516\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_141\">\n",
       "     <path d=\"M 39.103125 580.832187 \n",
       "L 59.103125 580.832187 \n",
       "\" style=\"fill:none;stroke:#00ffff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_142\"/>\n",
       "    <g id=\"text_61\">\n",
       "     <!-- Handmade -->\n",
       "     <g transform=\"translate(67.103125 584.332187)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"75.195312\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"136.474609\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "      <use x=\"199.853516\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"263.330078\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "      <use x=\"360.742188\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"422.021484\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "      <use x=\"485.498047\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p73fe432886\">\n",
       "   <rect height=\"296.8\" width=\"675.4\" x=\"30.103125\" y=\"22.318125\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"pa3bc0c58dd\">\n",
       "   <rect height=\"296.8\" width=\"675.4\" x=\"30.103125\" y=\"376.918125\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axes = plt.subplots(2, 1, figsize=(10, 10))\n",
    "\n",
    "# YOUR CODE HERE\n",
    "\n",
    "axes[0].legend()\n",
    "axes[0].grid(True)\n",
    "axes[0].set_xlabel('Epoch')\n",
    "axes[0].set_title('CrossEntropy Loss')\n",
    "\n",
    "axes[1].legend()\n",
    "axes[1].grid(True)\n",
    "axes[1].set_xlabel('Epoch')\n",
    "axes[1].set_title('Accuracy')\n",
    "\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Сделайте итоговые выводы о качестве работы моделей с разными реализациями DropOut:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Ответ:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Бонус. Zoneout (2 балла)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Это еще одна модификация идеи дропаута применительно к рекуррентным нейросетям. В Zoneout на каждом временном шаге с вероятностью p компонента скрытого состояния обновляется, а с вероятностью 1-p берется с предыдущего шага. \n",
    "В Виде формул (m^t_h - бинарная маска):\n",
    " \n",
    "(сначала обычный рекуррентный переход, например LSTM)\n",
    "$$\n",
    "i = \\sigma(h_{t-1}W^i + x_t U^i+b_i) \\quad\n",
    "o = \\sigma(h_{t-1}W^o + x_t U^o+b_o) \n",
    "$$\n",
    "$$\n",
    "f = \\sigma(h_{t-1}W^f + x_t U^f+b_f) \\quad \n",
    "g = tanh(h_{t-1} W^g + x_t U^g+b_g) \n",
    "$$\n",
    "$$\n",
    "c_t = f \\odot c_{t-1} +  i \\odot  g \\quad\n",
    "h_t =  o \\odot tanh(c_t) \\nonumber\n",
    "$$\n",
    "Затем Zoneout:\n",
    "$$\n",
    "h_t = h_t * m_h^t + h_{t-1}*(1-m_h^t)\n",
    "$$\n",
    "В этом методе маска уже должна быть разная во все моменты времени (иначе метод упрощается до дропаута Гала и Гарамани). На входы $x_t$ вновь можно накладывать маску до начала работы рекуррентного слоя.  \n",
    "\n",
    "Если у вас осталось время, вы можете реализовать этот метод. Выберите основу из трех рассмотренных случаев самостоятельно."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-31T16:05:00.702763Z",
     "start_time": "2021-03-31T16:05:00.674835Z"
    }
   },
   "source": [
    "# Часть 2. Language Modeling с помощью LSTM. (3 балла)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Во второй части мы попробуем обучить модель для генерации отзывов по их началу."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Концептуально модель будет выглядеть следующим образом:\n",
    "    \n",
    "![image info](https://blog.feedly.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-06-at-12.08.35-PM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В процессе обучения будем тренировать сеть предсказывать вероятность следующего символа при условии всех предыдущих. Эту вероятность можно моделировать с помощью скрытого состояния $h^{(t)}$ пропуская его через линейный слой с выходной размерностью равной размерности словаря:\n",
    "$$\n",
    "p(x^{t}|x^{t-1}, ..., x^{1}) = SoftMax(Linear(h^{(t)}))\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обратите внимание, что для вычисления $p(x^{t}|x^{t-1}, ..., x^{1})$ для всех моментов времени достаточно сделать один проход по RNN, а затем применить линейное преобразование ко всем скрытым состояниям."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:37:56.100520Z",
     "start_time": "2021-04-02T00:37:56.072747Z"
    }
   },
   "source": [
    "В качестве функции потерь необходимо использовать `CrossEntropy`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Рассмотрим другой важный момент. Для того, чтобы решить данную задачу, модель должна уметь определять момент начала генерации предложения и оповещать о завершении генерации -- конце предложения. Для этого добавим в словарь вспомогательные токены `<sos>`, `<eos>`. Добавив `<sos>` в начало каждого предложения и `<eos>` в конец.\n",
    "\n",
    "Модель сможет начинать генерацию как только ей будет передан токен `<sos>` и заканчивать генерацию, как только на очередном месте самым вероятным токеном оказывается `<eos>`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для решения этой задачи мы воспользуемся уже реализованной LSTM с дропаутом `FastRNNLayer` и классом `RNNClassifier`, то есть архитектура сети принципиально не поменяется. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Реализация модели и цикла обучения (1 балл)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Не используйте циклы в `RNNLM`, `LMCrossEntropyLoss`, `LMAccuracy`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T02:07:02.815198Z",
     "start_time": "2021-04-02T02:07:02.787445Z"
    }
   },
   "outputs": [],
   "source": [
    "class RNNLM(RNNClassifier):\n",
    "    def __init__(\n",
    "        self, embedding_dim, hidden_dim, vocab, dropout=0.5, layers_dropout=0.5, num_layers=1\n",
    "    ):\n",
    "        super().__init__(\n",
    "            embedding_dim=embedding_dim, hidden_dim=hidden_dim, output_size=len(vocab), vocab=vocab,\n",
    "            rec_layer=FastRNNLayer, dropout=dropout, layers_dropout=layers_dropout, num_layers=num_layers\n",
    "        )\n",
    "    \n",
    "    def forward(self, tokens, tokens_lens):\n",
    "        \"\"\"\n",
    "        :param torch.tensor(dtype=torch.long) tokens: Batch of texts represented with tokens. Shape: [T, B]\n",
    "        :param torch.tensor(dtype=torch.long) tokens_lens: Number of non-padding tokens for each object in batch. Shape: [B]\n",
    "        :return torch.tensor: Distribution of next token for each time step. Shape: [T, B, V], V -- size of vocabulary\n",
    "        \"\"\"\n",
    "        # Make embeddings for all tokens\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "        # Forward pass embeddings through network\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "        # Take all hidden states from the last layer of LSTM for each step and perform linear transformation\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реализуем функцию потерь для данной задачи. \n",
    "\n",
    "Моменты на которые нужно обратить внимание:\n",
    "1. Распределение вероятности следующего токена для последнего токена в последовательности не участвует в подсчёте функции потерь.\n",
    "2. Необходимо учитывать, что в одном батче могут быть тексты разной длины."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для решения второй проблемы можно воспользоваться функцией `torch.nn.utils.rnn.pack_padded_sequence`. \n",
    "\n",
    "Принимая на вход батч тензоров и длину каждого тензора без учёта паддинга эта функция позволяет получить все элементы в тензорах, которые не относятся к паддингу в виде плоского массива:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:54:40.004897Z",
     "start_time": "2021-04-02T00:54:39.977287Z"
    }
   },
   "outputs": [],
   "source": [
    "padded_tensors = torch.tensor([\n",
    "    [[1, 11, 111], [2, 22, 222], [3, 33, 333]],\n",
    "    [[4, 44, 444], [5, 55, 555], [6, 66, 666]],\n",
    "    [[7, 77, 777], [0, 0, 0], [8, 88, 888]],\n",
    "    [[9, 99, 999], [0, 0, 0], [0, 0, 0]]\n",
    "])\n",
    "tensors_lens = torch.tensor([4, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обратите внимание, что `torch.nn.utils.rnn.pack_padded_sequence` автоматически переупорядочивает тензоры в батче по убыванию их длины."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:54:24.517023Z",
     "start_time": "2021-04-02T00:54:24.490588Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PackedSequence(data=tensor([[  1,  11, 111],\n",
       "        [  3,  33, 333],\n",
       "        [  2,  22, 222],\n",
       "        [  4,  44, 444],\n",
       "        [  6,  66, 666],\n",
       "        [  5,  55, 555],\n",
       "        [  7,  77, 777],\n",
       "        [  8,  88, 888],\n",
       "        [  9,  99, 999]]), batch_sizes=tensor([3, 3, 2, 1]), sorted_indices=tensor([0, 2, 1]), unsorted_indices=tensor([0, 2, 1]))"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.nn.utils.rnn.pack_padded_sequence(padded_tensors, tensors_lens, batch_first=False, enforce_sorted=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T02:07:06.289671Z",
     "start_time": "2021-04-02T02:07:06.262883Z"
    }
   },
   "outputs": [],
   "source": [
    "class LMCrossEntropyLoss(torch.nn.CrossEntropyLoss):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        \n",
    "    def forward(self, outputs, tokens, tokens_lens):\n",
    "        \"\"\"\n",
    "        :param torch.tensor outputs: Output from RNNLM.forward. Shape: [T, B, V]\n",
    "        :param torch.tensor tokens: Batch of tokens. Shape: [T, B]\n",
    "        :param torch.tensor tokens_lens: Length of each sequence in batch\n",
    "        :return torch.tensor: CrossEntropyLoss between corresponding logits and tokens\n",
    "        \"\"\"\n",
    "        # Use torch.nn.utils.rnn.pack_padded_sequence().data to remove padding and flatten logits and tokens\n",
    "        # Do not forget specify enforce_sorted=False and correct value of batch_first \n",
    "        # YOUR CODE HERE\n",
    "        packed_outputs = None\n",
    "        packed_tokens = None\n",
    "        \n",
    "        # Use super().forward(..., ...) to compute CrossEntropyLoss\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для оценки качества нам также необходимо вычислять долю правильно предсказанных токенов. Реализуйте класс для вычисления точности."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T02:07:07.335981Z",
     "start_time": "2021-04-02T02:07:07.309586Z"
    }
   },
   "outputs": [],
   "source": [
    "class LMAccuracy(torch.nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        \n",
    "    def forward(self, outputs, tokens, tokens_lens):\n",
    "        \"\"\"\n",
    "        :param torch.tensor outputs: Output from RNNLM.forward. Shape: [T, B, V]\n",
    "        :param torch.tensor tokens: Batch of tokens. Shape: [T, B]\n",
    "        :param torch.tensor tokens_lens: Length of each sequence in batch\n",
    "        :return torch.tensor: Accuracy for given logits and tokens\n",
    "        \"\"\"\n",
    "        # Use torch.nn.utils.rnn.pack_padded_sequence().data to remove padding and flatten logits and tokens\n",
    "        # Do not forget specify enforce_sorted=False and correct value of batch_first \n",
    "        # YOUR CODE HERE\n",
    "        packed_outputs = None\n",
    "        packed_tokens = None\n",
    "        \n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Модифицируйте функции `train_epoch`, `evaluate`, `train` для обучения LM.\n",
    "\n",
    "**При вычислении точности, обратите внимание на то, что мы не предсказываем первый токен в каждой последовательности и токены, относящиеся к паддингу.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T02:07:31.492984Z",
     "start_time": "2021-04-02T02:07:31.459655Z"
    }
   },
   "outputs": [],
   "source": [
    "def train_epoch_lm(dataloader, model, loss_fn, optimizer, device):\n",
    "    model.train()\n",
    "    for idx, data in enumerate(dataloader):\n",
    "        # 1. Take data from batch\n",
    "        # 2. Perform forward pass\n",
    "        # 3. Evaluate loss\n",
    "        # 4. Make optimizer step\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "        pass\n",
    "    \n",
    "def evaluate_lm(dataloader, model, loss_fn, device):\n",
    "    model.eval()\n",
    "    \n",
    "    total_tokens = 0\n",
    "    total_loss = 0.0\n",
    "    total_accuracy = 0.0\n",
    "    \n",
    "    accuracy_fn = LMAccuracy()\n",
    "    with torch.no_grad():\n",
    "        for idx, data in enumerate(dataloader):\n",
    "            # 1. Take data from batch\n",
    "            # 2. Perform forward pass\n",
    "            # 3. Evaluate loss\n",
    "            # 4. Evaluate accuracy\n",
    "            # YOUR CODE HERE\n",
    "            \n",
    "            pass\n",
    "            \n",
    "    return total_loss / total_tokens, total_accuracy / total_tokens\n",
    "\n",
    "def train_lm(\n",
    "    train_loader, test_loader, model, loss_fn, optimizer, device, num_epochs\n",
    "):\n",
    "    test_losses = []\n",
    "    train_losses = []\n",
    "    test_accuracies = []\n",
    "    train_accuracies = []\n",
    "    for epoch in range(num_epochs):\n",
    "        train_epoch_lm(train_loader, model, loss_fn, optimizer, device)\n",
    "        \n",
    "        train_loss, train_acc = evaluate_lm(train_loader, model, loss_fn, device)\n",
    "        train_accuracies.append(train_acc)\n",
    "        train_losses.append(train_loss)\n",
    "        \n",
    "        test_loss, test_acc = evaluate_lm(test_loader, model, loss_fn, device)\n",
    "        test_accuracies.append(test_acc)\n",
    "        test_losses.append(test_loss)\n",
    "        \n",
    "        print(\n",
    "            'Epoch: {0:d}/{1:d}. Loss (Train/Test): {2:.3f}/{3:.3f}. Accuracy (Train/Test): {4:.3f}/{5:.3f}'.format(\n",
    "                epoch + 1, num_epochs, train_losses[-1], test_losses[-1], train_accuracies[-1], test_accuracies[-1]\n",
    "            )\n",
    "        )\n",
    "    return train_losses, train_accuracies, test_losses, test_accuracies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь у нас всё готово для обучения модели."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T01:06:12.736180Z",
     "start_time": "2021-04-02T01:06:12.708814Z"
    }
   },
   "source": [
    "Создадим словарь с `<sos>`, `<eos>` токенами.\n",
    "\n",
    "Обратите внимание, что в отличие от классификации текстов нам необходимо значительно увеличить размер словаря, чтобы доля `<unk>` токенов была не велика.\n",
    "\n",
    "Так же, так как задача генерации значительно сложнее задачи классификации текстов будем обучать модель только на префиксах рецензий длины $20$. Это позволяет значительно ускорить обучение."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:06:20.093645Z",
     "start_time": "2021-04-02T00:06:19.926668Z"
    }
   },
   "outputs": [],
   "source": [
    "specials = ['<pad>', '<unk>', '<sos>', '<eos>']\n",
    "for special in specials:\n",
    "    counter[special] = 0\n",
    "lm_vocab = torchtext.vocab.Vocab(counter, specials=specials, specials_first=True, max_size=30000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T00:06:58.566893Z",
     "start_time": "2021-04-02T00:06:21.430692Z"
    }
   },
   "outputs": [],
   "source": [
    "lm_test_dataset = LargeMovieReviewDataset(test_data_path, lm_vocab, max_len=20, pad_sos=True, pad_eos=True)\n",
    "lm_train_dataset = LargeMovieReviewDataset(train_data_path, lm_vocab, max_len=20, pad_sos=True, pad_eos=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создадим даталоадеры для тестовой и обучающей выборок:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T12:29:16.213723Z",
     "start_time": "2021-04-02T12:29:16.186954Z"
    }
   },
   "outputs": [],
   "source": [
    "lm_test_dataloader = DataLoader(\n",
    "    lm_test_dataset, batch_size=96, shuffle=False, num_workers=3, \n",
    "    collate_fn=partial(collate_fn, padding_value=lm_vocab.lookup_indices(['<pad>'])[0])\n",
    ")\n",
    "lm_train_dataloader = DataLoader(\n",
    "    lm_train_dataset, batch_size=96, shuffle=True, num_workers=3, \n",
    "    collate_fn=partial(collate_fn, padding_value=lm_vocab.lookup_indices(['<pad>'])[0])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Убедитесь, что все предложения имеют в начале `<sos>` токен, а в конце -- `<eos>` токен."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T12:29:17.218115Z",
     "start_time": "2021-04-02T12:29:16.922801Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[   2,    2,    2,  ...,    2,    2,    2],\n",
       "         [   1, 1495,   15,  ..., 1498,   37,    4],\n",
       "         [   1,  383,  296,  ..., 1658,    1,    8],\n",
       "         ...,\n",
       "         [ 355,   40,  258,  ...,   39,  347,   53],\n",
       "         [1294, 4617, 1661,  ...,    6, 9830,  423],\n",
       "         [   3,    3,    3,  ...,    3,    3,    3]]),\n",
       " tensor([20, 20, 20, 20, 20, 20, 20, 20, 18, 20, 20, 20, 20, 20, 20, 20, 20, 20,\n",
       "         20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20,\n",
       "         20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20,\n",
       "         20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20,\n",
       "         20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20,\n",
       "         20, 20, 20, 20, 20, 20]))"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch = next(iter(lm_train_dataloader))\n",
    "batch['tokens'], batch['tokens_lens']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создадим модель, функцию потерь и оптимизатор: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T14:15:32.916424Z",
     "start_time": "2021-04-02T14:15:32.525452Z"
    }
   },
   "outputs": [],
   "source": [
    "lm_model = RNNLM(\n",
    "    embedding_dim=512, hidden_dim=512, vocab=lm_vocab, dropout=0.6, layers_dropout=0.6, num_layers=2\n",
    ").to(device=device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T14:15:33.332806Z",
     "start_time": "2021-04-02T14:15:33.307749Z"
    }
   },
   "outputs": [],
   "source": [
    "lm_loss_fn = LMCrossEntropyLoss(reduction='mean')\n",
    "lm_optimizer = torch.optim.Adam(lm_model.parameters(), lr=0.005, weight_decay=1.2e-6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обучим модель:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T14:20:33.447251Z",
     "start_time": "2021-04-02T14:15:33.797444Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/10. Loss (Train/Test): 7.546/7.682. Accuracy (Train/Test): 0.079/0.079\n",
      "Epoch: 2/10. Loss (Train/Test): 7.315/7.519. Accuracy (Train/Test): 0.089/0.089\n",
      "Epoch: 3/10. Loss (Train/Test): 7.108/7.414. Accuracy (Train/Test): 0.097/0.095\n",
      "Epoch: 4/10. Loss (Train/Test): 6.911/7.334. Accuracy (Train/Test): 0.100/0.097\n",
      "Epoch: 5/10. Loss (Train/Test): 6.757/7.324. Accuracy (Train/Test): 0.105/0.100\n",
      "Epoch: 6/10. Loss (Train/Test): 6.555/7.283. Accuracy (Train/Test): 0.109/0.104\n",
      "Epoch: 7/10. Loss (Train/Test): 6.409/7.294. Accuracy (Train/Test): 0.114/0.106\n",
      "Epoch: 8/10. Loss (Train/Test): 6.242/7.287. Accuracy (Train/Test): 0.120/0.108\n",
      "Epoch: 9/10. Loss (Train/Test): 6.085/7.304. Accuracy (Train/Test): 0.123/0.109\n",
      "Epoch: 10/10. Loss (Train/Test): 5.940/7.341. Accuracy (Train/Test): 0.128/0.110\n"
     ]
    }
   ],
   "source": [
    "lm_train_losses, lm_train_accuracies, lm_test_losses, lm_test_accuracies = train_lm(\n",
    "    lm_train_dataloader, lm_test_dataloader, lm_model, lm_loss_fn, lm_optimizer, device, 10\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Реализация декодера. (2 балла)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь, реализуем последнюю деталь -- декодирование с использованием обученной модели.\n",
    "Есть несколько вариантов. Рассмотрим два самых простых:\n",
    "1. **Жадное декодирование.** На каждом шаге мы выбираем токен с максимальной вероятностью и используем его для обновления скрытого состояния RNN.\n",
    "2. **Top-k sampling.** На очередном шаге рассматриваются $k$ токенов с самыми большими вероятностями. Остальные токены игнорируются. Из выбранных токенов семплируется следующий токен пропорционально их вероятностям.\n",
    "\n",
    "Прочитать подробнее про разные варианты декодирования можно по ссылкам:\n",
    "1. [От huggingface](https://huggingface.co/blog/how-to-generate)\n",
    "2. [На towardsdatascience](https://towardsdatascience.com/decoding-strategies-that-you-need-to-know-for-response-generation-ba95ee0faadc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Существенным в процессе декодирования является критерий останова. Как только очередной самый вероятный символ оказался `<eos>`, то данная последовательность считается сгенерированной. Однако, может так оказаться, что `<eos>` никогда не будет выбран, тогда необходимо прекратить генерацию, как только длина последовательности перейдёт порог `max_generated_len`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T14:28:21.578336Z",
     "start_time": "2021-04-02T14:28:21.547183Z"
    }
   },
   "outputs": [],
   "source": [
    "def decode(model, start_tokens, start_tokens_lens, max_generated_len=20, top_k=None):\n",
    "    \"\"\"\n",
    "    :param RNNLM model: Model\n",
    "    :param torch.tensor start_tokens: Batch of seed tokens. Shape: [T, B]\n",
    "    :param torch.tensor start_tokens_lens: Length of each sequence in batch. Shape: [B]\n",
    "    :return Tuple[torch.tensor, torch.tensor]. Newly predicted tokens and length of generated part. Shape [T*, B], [B]\n",
    "    \"\"\"\n",
    "    # Get embedding for start_tokens\n",
    "    # YOUR CODE HERE\n",
    "    embedding = None\n",
    "    \n",
    "    # Pass embedding through rnn and collect hidden states and cell states for each time moment\n",
    "    all_h, all_c = [], []\n",
    "    h = embedding.new_zeros([model.rnn.num_layers, start_tokens.shape[1], model.hidden_dim])\n",
    "    c = embedding.new_zeros([model.rnn.num_layers, start_tokens.shape[1], model.hidden_dim])\n",
    "    for time_step in range(start_tokens.shape[0]):\n",
    "        # YOUR CODE HERE\n",
    "        pass\n",
    "    \n",
    "    all_h = torch.stack(all_h, dim=1)\n",
    "    all_c = torch.stack(all_c, dim=1)\n",
    "    # Take final hidden state and cell state for each start sequence in batch\n",
    "    # We will use them as h_0, c_0 for generation new tokens\n",
    "    h = all_h[:, start_tokens_lens - 1, torch.arange(start_tokens_lens.shape[0])]\n",
    "    c = all_c[:, start_tokens_lens - 1, torch.arange(start_tokens_lens.shape[0])]\n",
    "    \n",
    "    # List of predicted tokens for each time step\n",
    "    predicted_tokens = []\n",
    "    # Length of generated part for each object in the batch\n",
    "    decoded_lens = torch.zeros_like(start_tokens_lens, dtype=torch.long)\n",
    "    # Boolean mask where we store if the sequence has already generated\n",
    "    # i.e. `<eos>` was selected on any step\n",
    "    is_finished_decoding = torch.zeros_like(start_tokens_lens, dtype=torch.bool)\n",
    "    \n",
    "    # Stop when all sequences in the batch are finished\n",
    "    while not torch.all(is_finished_decoding) and torch.max(decoded_lens) < max_generated_len:\n",
    "        # Evaluate next token distribution using hidden state h.\n",
    "        # Note. Over first dimension h has hidden states for each layer of LSTM.\n",
    "        #     We must use hidden state from the last layer\n",
    "        # YOUR CODE HERE\n",
    "        logits = None\n",
    "        \n",
    "        if top_k is not None:\n",
    "            # Top-k sampling. Use only top-k most probable logits to sample next token\n",
    "            indices_to_remove = logits < torch.topk(logits, top_k)[0][..., -1, None]\n",
    "            # Mask non top-k logits\n",
    "            logits[indices_to_remove] = -1e10\n",
    "            # Sample next_token. \n",
    "            # YOUR CODE HERE\n",
    "            next_token = None\n",
    "        else:\n",
    "            # Select most probable token\n",
    "            # YOUR CODE HERE\n",
    "            next_token = None\n",
    "            \n",
    "        predicted_tokens.append(next_token)\n",
    "        \n",
    "        decoded_lens += (~is_finished_decoding)\n",
    "        is_finished_decoding |= (next_token == torch.tensor(model.vocab.lookup_indices(['<eos>'])[0]))\n",
    "\n",
    "        # Evaluate embedding for next token\n",
    "        # YOUR CODE HERE\n",
    "\n",
    "        # Update hidden and cell states\n",
    "        # YOUR CODE HERE\n",
    "\n",
    "    return torch.stack(predicted_tokens), decoded_lens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T01:38:06.232189Z",
     "start_time": "2021-04-02T01:38:06.205413Z"
    }
   },
   "source": [
    "Попробуем сгенерировать продолжения для нескольких префиксов:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T14:28:22.568613Z",
     "start_time": "2021-04-02T14:28:22.545810Z"
    }
   },
   "outputs": [],
   "source": [
    "start_tokens = torch.tensor([\n",
    "    lm_model.vocab.lookup_indices(['<sos>', '<pad>', '<pad>', '<pad>']),\n",
    "    lm_model.vocab.lookup_indices(['<sos>', 'my', 'favorite', 'movie']),\n",
    "    lm_model.vocab.lookup_indices(['<sos>', 'the', 'best', 'movie']),\n",
    "    lm_model.vocab.lookup_indices(['<sos>', 'the', 'worst', 'movie']),\n",
    "]).T\n",
    "\n",
    "start_tokens_lens = torch.tensor([1, 4, 4, 4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T14:28:28.222137Z",
     "start_time": "2021-04-02T14:28:27.930196Z"
    }
   },
   "outputs": [],
   "source": [
    "lm_model = lm_model.cpu()\n",
    "lm_model.eval()\n",
    "decoded_tokens, decoded_lens = decode(lm_model, start_tokens, start_tokens_lens, max_generated_len=20, top_k=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T14:28:28.751380Z",
     "start_time": "2021-04-02T14:28:28.708461Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<sos> first saw movie first time really disappointed really disappointed movie even one funniest films ever seen movie seen life <eos>\n",
      "<sos> <unk> favorite movie ever seen movie really enjoyed first saw movie years ago thought would good movie <eos>\n",
      "<sos> <unk> best movie ever seen movie ever made one thing bad acting bad acting bad effects bad bad script bad script bad acting\n",
      "<sos> <unk> worst movie ever seen life life <unk> one best movies ever seen bad movies bad acting bad acting bad <eos>\n"
     ]
    }
   ],
   "source": [
    "for text_idx in range(start_tokens.shape[1]):\n",
    "    decoded_text_tokens = decoded_tokens[:decoded_lens[text_idx], text_idx]\n",
    "    tokens = start_tokens[:start_tokens_lens[text_idx], text_idx].tolist() + decoded_text_tokens.tolist()\n",
    "    words = np.array(lm_model.vocab.itos)[np.array(tokens)]\n",
    "    print(' '.join(words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Попробуйте выполнить семплирование для разных $k$. Сравните результаты top-k семплирования с жадным декодированием. Опишите ваши наблюдения."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T17:06:36.404126Z",
     "start_time": "2021-04-02T17:06:36.399654Z"
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Ответ:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-02T15:04:51.678260Z",
     "start_time": "2021-04-02T15:04:51.673587Z"
    }
   },
   "source": [
    "## Бонус. Cущественное улучшение качества. (до 3 баллов)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Та модель, которая использовалась в предыдущей части во многом заимствует улучшения LSTM из статьи [Regularizing and Optimizing LSTM Language Models](https://arxiv.org/pdf/1708.02182.pdf). Вы можете попробовать применить другие варианты регуляризации из данной статьи для существенного улучшения качества LM.\n",
    "\n",
    "Например:\n",
    "1. Dropout для эмбеддингов\n",
    "2. Dropout входов и выходов RNN\n",
    "3. Регуляризация активаций (AR/TAR)\n",
    "4. NT-ASGD\n",
    "5. Tied веса эмбеддингов и софтмакса"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "313px",
    "left": "926px",
    "right": "27px",
    "top": "120px",
    "width": "343px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}