NLP-track-project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLP track project\n",
    "\n",
    "\n",
    "The goal of this project is to guide you through the latest techniques in NLP, we have chosen to frame that journey around the problem of *Neural Machine Translation (NMT)*, More specifically english-spanish translation. \n",
    "\n",
    "The motivation for centering around this particular problem is that it's a complex one that has been revolutionized lately. So It will guide you from the basics to the latest SoA techniques.\n",
    "\n",
    "The reason we choose English and Spanish is purely because we happen to know both of them, if you would like to explore other language feel free to do so but make sure you have a proper dataset that you can use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Baseline\n",
    "\n",
    "As a baseline we will first train a simple encoder-decoder architecture. The code is based on the [seq2seq notebook](https://github.com/fastai/course-nlp/blob/master/7-seq2seq-translation.ipynb) from the [lesson 12](https://www.youtube.com/watch?v=IfsjMg4fLWQ&list=PLtmWHNX-gukKocXQOkQjuVxglSDYWsSh9&index=13&t=0s) of the Fast.ai's NLP course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Dataset\n",
    "\n",
    "The dataset we will use was build by [Open Subtitles](https://www.opensubtitles.org/) and consists of movie and series subtitles, we chose this dataset over the alternatives because it contains casual and more simple language than the alternatives (UN resolutions, EU parlament sessions, etc). You can look at the alternatives on this [amazing site](http://opus.nlpl.eu/).\n",
    "\n",
    "Unfurtunately the spanish used is from Spain, not sure why the Latin American version of the subtitles is not available.\n",
    "\n",
    "The dataset is extremely large and not in a great format, you can see how we processed it on [this notebook](./subtitles-download.ipynb). There is no need to run it, ask for the csv file with the final result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'GeForce GTX 980 Ti'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from fastai.text import *\n",
    "\n",
    "base_path = Path('/data/NLP_track/')\n",
    "\n",
    "torch.cuda.set_device(0)\n",
    "torch.cuda.get_device_name(torch.cuda.current_device())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/javiber/.local/share/virtualenvs/javiber-jarMRqt3/lib/python3.6/site-packages/IPython/core/interactiveshell.py:3058: DtypeWarning: Columns (1) have mixed types. Specify dtype option on import or set low_memory=False.\n",
      "  interactivity=interactivity, compiler=compiler, result=result)\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv(base_path/'subtitles.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7990657\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>files</th>\n",
       "      <th>sub_id</th>\n",
       "      <th>en</th>\n",
       "      <th>es</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2386520</th>\n",
       "      <td>en/2010/1570179/3663037.xml#es/2010/1570179/36...</td>\n",
       "      <td>112</td>\n",
       "      <td>But I 'm hoping they 'll be more cooperative w...</td>\n",
       "      <td>Pero espero que sean más cooperativos cuando s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998663</th>\n",
       "      <td>en/2002/512294/5895863.xml#es/2002/512294/5179...</td>\n",
       "      <td>535</td>\n",
       "      <td>You came back for me .</td>\n",
       "      <td>Volviste por mí .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>989442</th>\n",
       "      <td>en/2002/512254/5235403.xml#es/2002/512254/5183...</td>\n",
       "      <td>782</td>\n",
       "      <td>He 's not going to be all right .</td>\n",
       "      <td>Él no va a estar bien .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1303299</th>\n",
       "      <td>en/2005/749711/178777.xml#es/2005/749711/40301...</td>\n",
       "      <td>570</td>\n",
       "      <td>I 'm fine .</td>\n",
       "      <td>Estaré bien .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3217283</th>\n",
       "      <td>en/2011/1832848/4659172.xml#es/2011/1832848/51...</td>\n",
       "      <td>544</td>\n",
       "      <td>At this point in time , I 'm pretty worried .</td>\n",
       "      <td>A esta altura , estoy preocupado .</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     files sub_id  \\\n",
       "2386520  en/2010/1570179/3663037.xml#es/2010/1570179/36...    112   \n",
       "998663   en/2002/512294/5895863.xml#es/2002/512294/5179...    535   \n",
       "989442   en/2002/512254/5235403.xml#es/2002/512254/5183...    782   \n",
       "1303299  en/2005/749711/178777.xml#es/2005/749711/40301...    570   \n",
       "3217283  en/2011/1832848/4659172.xml#es/2011/1832848/51...    544   \n",
       "\n",
       "                                                        en  \\\n",
       "2386520  But I 'm hoping they 'll be more cooperative w...   \n",
       "998663                              You came back for me .   \n",
       "989442                   He 's not going to be all right .   \n",
       "1303299                                        I 'm fine .   \n",
       "3217283      At this point in time , I 'm pretty worried .   \n",
       "\n",
       "                                                        es  \n",
       "2386520  Pero espero que sean más cooperativos cuando s...  \n",
       "998663                                   Volviste por mí .  \n",
       "989442                             Él no va a estar bien .  \n",
       "1303299                                      Estaré bien .  \n",
       "3217283                 A esta altura , estoy preocupado .  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(len(df))\n",
    "df.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "files     0\n",
       "sub_id    0\n",
       "en        8\n",
       "es        6\n",
       "dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "files     0\n",
       "sub_id    0\n",
       "en        0\n",
       "es        0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna(inplace=True)\n",
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.sample(50000, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Tokenization and Numericalization\n",
    "\n",
    "Unfurtunately the dataset was already tokenized, that's why you see spaces around punctuation marks. Not a big deal but we'd have rather the raw messages.\n",
    "\n",
    "Anyway for this baseline we will use the default settings from fast.ai which is to use spaCy's tokenizer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2.1 Data collate\n",
    "\n",
    "In order to run the RNN in parallel for a whole batch, all examples in a batch must have the same size. \n",
    "To accomplish this we'll add padding where necessary. \n",
    "\n",
    "Fast.ai has a [collate](https://docs.fast.ai/text.data.html#pad_collate) that does something similar but just in the Xs because is meant to be used on sentence classification, on seq2seq we need to padd both Xs and Ys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [],
   "source": [
    "def seq2seq_collate(samples, pad_idx=1, pad_first=True, backwards=False):\n",
    "    \"Function that collect samples and adds padding. Flips token order if needed\"\n",
    "    samples = to_data(samples)\n",
    "    max_len_x = max([len(s[0]) for s in samples])\n",
    "    max_len_y = max([len(s[1]) for s in samples])\n",
    "    \n",
    "    # create matrices of the target size we want (batch_lenght x max_sentence_lenght_in_batch) full of padding\n",
    "    res_x = torch.zeros(len(samples), max_len_x).long() + pad_idx\n",
    "    res_y = torch.zeros(len(samples), max_len_y).long() + pad_idx\n",
    "    \n",
    "    if backwards: \n",
    "        pad_first = not pad_first\n",
    "    \n",
    "    for i, s in enumerate(samples):\n",
    "        # replace the part of the correct part of the padded matrices with the actual values\n",
    "        if pad_first: \n",
    "            res_x[i, -len(s[0]):] = LongTensor(s[0])\n",
    "            res_y[i, -len(s[1]):] = LongTensor(s[1])\n",
    "        else:         \n",
    "            res_x[i,:len(s[0]):] = LongTensor(s[0])\n",
    "            res_y[i,:len(s[1]):] = LongTensor(s[1])\n",
    "            \n",
    "    if backwards: \n",
    "        res_x,res_y = res_x.flip(1),res_y.flip(1)\n",
    "    \n",
    "    return res_x,res_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2.2 Custom databunch\n",
    "\n",
    "On the training set, we don't want to mix really long senteces with really short because that'd create lots of padding, instead we'll use fast.ai's `SortishSampler` to keep the sentences of a batch relatively the same size while also having some randomization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "class Seq2SeqDataBunch(TextDataBunch):\n",
    "    \"Create a `TextDataBunch` suitable for training an RNN classifier.\"\n",
    "    \n",
    "    @classmethod\n",
    "    def create(\n",
    "            cls, train_ds, valid_ds, test_ds=None, path:PathOrStr='.', bs:int=32, val_bs:int=None, pad_idx=1,\n",
    "           dl_tfms=None, pad_first=False, device:torch.device=None, no_check:bool=False, backwards:bool=False, \n",
    "            **dl_kwargs) -> DataBunch:\n",
    "        \"\"\"\n",
    "        Function that transform the `datasets` in a `DataBunch` for classification. \n",
    "        Passes `**dl_kwargs` on to `DataLoader()`\n",
    "        \"\"\"\n",
    "        \n",
    "        datasets = cls._init_ds(train_ds, valid_ds, test_ds)\n",
    "        val_bs = bs if val_bs is None else val_bs \n",
    "        \n",
    "        collate_fn = partial(seq2seq_collate, pad_idx=pad_idx, pad_first=pad_first, backwards=backwards)\n",
    "        \n",
    "        # Sample data randomly but trying to keep the lenghts similar\n",
    "        train_sampler = SortishSampler(datasets[0].x, key=lambda t: len(datasets[0][t][0].data), bs=bs//2)\n",
    "        train_dl = DataLoader(datasets[0], batch_size=bs, sampler=train_sampler, drop_last=True, **dl_kwargs)\n",
    "        dataloaders = [train_dl]\n",
    "        \n",
    "        for ds in datasets[1:]:\n",
    "            lengths = [len(t) for t in ds.x.items]\n",
    "            sampler = SortSampler(ds.x, key=lengths.__getitem__)\n",
    "            \n",
    "            dataloaders.append(DataLoader(ds, batch_size=val_bs, sampler=sampler, **dl_kwargs))\n",
    "            \n",
    "        return cls(*dataloaders, path=path, device=device, collate_fn=collate_fn, no_check=no_check)\n",
    "    \n",
    "    \n",
    "class Seq2SeqTextList(TextList):\n",
    "    _bunch = Seq2SeqDataBunch\n",
    "    _label_cls = TextList"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "src = (Seq2SeqTextList\n",
    "       .from_df(df, path=base_path, cols='en')\n",
    "       .split_by_rand_pct(seed=42)\n",
    "       .label_from_df(cols='es', label_cls=TextList))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### 1.2.3 Analyze sentence length\n",
    "Some sentences are extremely large so we will remove those outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(18.0, 81)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_lengths = [len(o) for o in src.train.x.items] + [len(o) for o in src.valid.x.items]\n",
    "np.percentile(x_lengths, 90), np.max(x_lengths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17.0, 79)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_lengths = [len(o) for o in src.train.y.items] + [len(o) for o in src.valid.y.items]\n",
    "np.percentile(y_lengths, 90), np.max(y_lengths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "func = lambda x,y: len(x) > 20 or len(y) > 20\n",
    "filt = array([func(x, y) for x,y in zip(src.x.items, src.y.items)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3197"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(filt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   2,    5, 1673, 1050, 1050,   10,   20,   17, 1814,   23,  571,    9])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "src.x[0].data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fastai.data_block.LabelLists"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(src)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "__main__.Seq2SeqTextList"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(src.x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "fastai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "src = src.filter_by_func()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "45998"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(src.train) + len(src.valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### 1.2.4 Visualize and save dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "bs = 128"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data = src.databunch(bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos i don ' t like the banks 'cause of the xxunk and the screaming and all the xxunk .</td>\n",
       "      <td>xxbos xxmaj no me gustan los bancos por la xxunk y los gritos y toda la xxunk .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos - xxmaj okay , all right , that xxunk lost you another week of tv and video xxunk .</td>\n",
       "      <td>xxbos - xxmaj bien , muy bien , esa mentira acaba de xxunk otra semana sin tele o xxunk .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj now , now , xxmaj xxunk - xxmaj xxunk - xxmaj san , please don ' t .</td>\n",
       "      <td>xxbos xxmaj vamos , vamos , xxmaj xxunk - xxmaj xxunk xxmaj san , por favor , no ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj xxunk xxmaj xxunk has won by around xxunk xxunk votes in his xxunk seat , xxmaj xxunk .</td>\n",
       "      <td>xxbos xxmaj xxunk xxmaj xxunk xxunk aproximadamente xxunk xxunk xxmaj votos su asiento xxunk , xxmaj xxunk .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj almost all the xxunk have been let out of the room , only the actual xxunk remain .</td>\n",
       "      <td>xxbos xxmaj casi todos los espectadores han dejado la sala , solo quedan las partes .</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    ">**Note:** we didn't tell fast.ai what tokenizer it should use for spanish so it's using the english one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2.4 Load saved databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data = load_data(base_path, bs=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### 1.3 Embeddings\n",
    "\n",
    "We will use the embedings from [fastText](https://fasttext.cc/docs/en/support.html) because they have pretrained models that are good enough for our baseline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### 1.3.1 Download pretrained word vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# ! wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz -P {path}\n",
    "# ! wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.es.300.bin.gz -P {path}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# ! gunzip {path}/cc.en.300.bin.gz\n",
    "# ! gunzip {path}/cc.es.300.bin.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import fasttext as ft"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "es_vecs = ft.load_model(str((base_path/'cc.es.300.bin')))\n",
    "en_vecs = ft.load_model(str((base_path/'cc.en.300.bin')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30 missing words\n",
      "36 missing words\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(torch.Size([5808, 300]), torch.Size([7104, 300]))"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def create_emb(vecs, itos, em_sz=300, mult=1.):\n",
    "    emb = nn.Embedding(len(itos), em_sz, padding_idx=1)\n",
    "    wgts = emb.weight.data\n",
    "    vec_dic = {w: vecs.get_word_vector(w) for w in vecs.get_words()}\n",
    "    miss = 0\n",
    "    for i, w in enumerate(itos):\n",
    "        try: \n",
    "            wgts[i] = tensor(vec_dic[w])\n",
    "        except: \n",
    "            miss += 1\n",
    "    print(f'{miss} missing words')\n",
    "    return emb\n",
    "\n",
    "emb_enc = create_emb(en_vecs, data.x.vocab.itos)\n",
    "emb_dec = create_emb(es_vecs, data.y.vocab.itos)\n",
    "\n",
    "emb_enc.weight.size(), emb_dec.weight.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    ">**NOTE**: the words in our vocabulary that are missing on the fastText's one will be set to random vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "torch.save(emb_enc, base_path/'en_emb.pth')\n",
    "torch.save(emb_dec, base_path/'es_emb.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### 1.3.2 Load saved embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "emb_enc = torch.load(base_path/'en_emb.pth')\n",
    "emb_dec = torch.load(base_path/'es_emb.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### 1.4 Model\n",
    "\n",
    "The Model we will use is the classic Encoder-Decoder, using a GRU to encode the meaning of the sentence in the original language into a fixed-size vector that the decode will convert into the target language, hopefully conservint the meaning.\n",
    "\n",
    "\n",
    "<center style=\"margin-top: 20px\">\n",
    "    <video src=\"images/enc_dec.mp4\" width=\"80%\" controls autoplay muted loop></video>\n",
    "</center>\n",
    "\n",
    "\n",
    "<center>\n",
    "    <i>Diagram from Jay Alammar's <a href=\"http://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/\">Visualizing A Neural Machine Translation Model</a></i></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "code_folding": [],
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class EncDecRNN(nn.Module):\n",
    "    def __init__(self, emb_enc, emb_dec, nh, out_sl, nl=2, bos_idx=0, pad_idx=1, tf_weight=0):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.nl, self.nh, self.out_sl = nl, nh, out_sl\n",
    "        self.bos_idx, self.pad_idx = bos_idx, pad_idx\n",
    "        self.em_sz_enc = emb_enc.embedding_dim\n",
    "        self.em_sz_dec = emb_dec.embedding_dim\n",
    "        self.voc_sz_dec = emb_dec.num_embeddings\n",
    "                 \n",
    "        self.emb_enc = emb_enc\n",
    "        self.emb_enc_drop = nn.Dropout(0.15)\n",
    "        self.gru_enc = nn.GRU(\n",
    "            self.em_sz_enc, nh, num_layers=nl, dropout=0.25, batch_first=True)\n",
    "        self.out_enc = nn.Linear(nh, self.em_sz_dec, bias=False)\n",
    "        \n",
    "        self.emb_dec = emb_dec\n",
    "        self.gru_dec = nn.GRU(\n",
    "            self.em_sz_dec, self.em_sz_dec, num_layers=nl, dropout=0.1, batch_first=True)\n",
    "        \n",
    "        self.out_drop = nn.Dropout(0.35)\n",
    "        self.out = nn.Linear(self.em_sz_dec, self.voc_sz_dec)\n",
    "        self.out.weight.data = self.emb_dec.weight.data\n",
    "        \n",
    "        self.tf_weight = tf_weight  # teacher forcing weight\n",
    "        \n",
    "    def encoder(self, bs, inp):\n",
    "        h = self.initHidden(bs)\n",
    "        emb = self.emb_enc_drop(self.emb_enc(inp))\n",
    "        _, h = self.gru_enc(emb, h)\n",
    "        h = self.out_enc(h)\n",
    "        return h\n",
    "    \n",
    "    def decoder(self, dec_inp, h):\n",
    "        emb = self.emb_dec(dec_inp).unsqueeze(1)\n",
    "        outp, h = self.gru_dec(emb, h)\n",
    "        outp = self.out(self.out_drop(outp[:,0]))\n",
    "        return outp, h\n",
    "        \n",
    "    def forward(self, inp, targ=None):\n",
    "        bs, sl = inp.size()\n",
    "        h = self.encoder(bs, inp)\n",
    "        dec_inp = inp.new_zeros(bs).long() + self.bos_idx\n",
    "        \n",
    "        res = []\n",
    "        for i in range(self.out_sl):\n",
    "            outp, h = self.decoder(dec_inp, h)\n",
    "            dec_inp = outp.max(1)[1]\n",
    "            res.append(outp)\n",
    "            if (dec_inp==self.pad_idx).all(): \n",
    "                break\n",
    "                \n",
    "            # Teacher forcing\n",
    "            if (targ is not None) and (random.random() < self.tf_weight):\n",
    "                if i>=targ.shape[1]: continue\n",
    "                dec_inp = targ[:,i]\n",
    "        return torch.stack(res, dim=1)\n",
    "    \n",
    "    def initHidden(self, bs): \n",
    "        return one_param(self).new_zeros(self.nl, bs, self.nh)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([128, 20])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xb, yb = next(iter(data.valid_dl))\n",
    "xb.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "EncDecRNN(\n",
       "  (emb_enc): Embedding(5808, 300, padding_idx=1)\n",
       "  (emb_enc_drop): Dropout(p=0.15, inplace=False)\n",
       "  (gru_enc): GRU(300, 256, num_layers=2, batch_first=True, dropout=0.25)\n",
       "  (out_enc): Linear(in_features=256, out_features=300, bias=False)\n",
       "  (emb_dec): Embedding(7104, 300, padding_idx=1)\n",
       "  (gru_dec): GRU(300, 300, num_layers=2, batch_first=True, dropout=0.1)\n",
       "  (out_drop): Dropout(p=0.35, inplace=False)\n",
       "  (out): Linear(in_features=300, out_features=7104, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = EncDecRNN(emb_enc, emb_dec, 256, 20, tf_weight=1)\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "> **Note:** we are using teacher forcing with prob of 1 so that we can compare this result to the ones with transformer (which also use teacher forcing with prob of 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.6 Metrics and loss functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### 1.6.1 Cross-Entropy loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def cross_entropy(out, targ, pad_idx=1):\n",
    "    bs, targ_len = targ.size()\n",
    "    _, out_len, vs = out.size()\n",
    "    if targ_len > out_len: \n",
    "        out  = F.pad(out, (0, 0, 0, targ_len - out_len, 0, 0), value=pad_idx)\n",
    "    if out_len > targ_len: \n",
    "        targ = F.pad(targ, (0, out_len - targ_len, 0, 0), value=pad_idx)\n",
    "    return CrossEntropyFlat()(out, targ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### 1.6.2 Token-level accuracy "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def seq2seq_acc(out, targ, pad_idx=1):\n",
    "    bs, targ_len = targ.size()\n",
    "    _, out_len, vs = out.size()\n",
    "    if targ_len > out_len: \n",
    "        out  = F.pad(out,  (0,0,0,targ_len-out_len,0,0), value=pad_idx)\n",
    "    if out_len > targ_len: \n",
    "        targ = F.pad(targ, (0,out_len-targ_len,0,0), value=pad_idx)\n",
    "    out = out.argmax(2)\n",
    "    return (out==targ).float().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.6.3 BLEU\n",
    "\n",
    "BLEU is a very used meassurement on NMT and in seq2seq in general, is important to know how it works and its limitations, I strongly recommend reading Rachael Tatman's [post](https://towardsdatascience.com/evaluating-text-output-in-nlp-bleu-at-your-own-risk-e8609665a213) about it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "code_folding": [
     0,
     12,
     15,
     20
    ]
   },
   "outputs": [],
   "source": [
    "class NGram():\n",
    "    def __init__(self, ngram, max_n=5000): \n",
    "        self.ngram,self.max_n = ngram,max_n\n",
    "        \n",
    "    def __eq__(self, other):\n",
    "        if len(self.ngram) != len(other.ngram): \n",
    "            return False\n",
    "        return np.all(np.array(self.ngram) == np.array(other.ngram))\n",
    "    \n",
    "    def __hash__(self): \n",
    "        return int(sum([o * self.max_n**i for i,o in enumerate(self.ngram)]))\n",
    "\n",
    "def get_grams(x, n, max_n=5000):\n",
    "    return x if n==1 else [NGram(x[i:i+n], max_n=max_n) for i in range(len(x)-n+1)]\n",
    "\n",
    "def get_correct_ngrams(pred, targ, n, max_n=5000):\n",
    "    pred_grams, targ_grams = get_grams(pred, n, max_n=max_n), get_grams(targ, n, max_n=max_n)\n",
    "    pred_cnt, targ_cnt = Counter(pred_grams), Counter(targ_grams)\n",
    "    return sum([min(c, targ_cnt[g]) for g, c in pred_cnt.items()]), len(pred_grams)\n",
    "\n",
    "class CorpusBLEU(Callback):\n",
    "    def __init__(self, vocab_sz):\n",
    "        self.vocab_sz = vocab_sz\n",
    "        self.name = 'bleu'\n",
    "    \n",
    "    def on_epoch_begin(self, **kwargs):\n",
    "        self.pred_len, self.targ_len, self.corrects, self.counts = 0,0,[0]*4,[0]*4\n",
    "    \n",
    "    def on_batch_end(self, last_output, last_target, **kwargs):\n",
    "        last_output = last_output.argmax(dim=-1)\n",
    "        for pred, targ in zip(last_output.cpu().numpy(), last_target.cpu().numpy()):\n",
    "            self.pred_len += len(pred)\n",
    "            self.targ_len += len(targ)\n",
    "            for i in range(4):\n",
    "                c, t = get_correct_ngrams(pred, targ, i + 1, max_n=self.vocab_sz)\n",
    "                self.corrects[i] += c\n",
    "                self.counts[i]   += t\n",
    "    \n",
    "    def on_epoch_end(self, last_metrics, **kwargs):\n",
    "        precs = [c/t for c,t in zip(self.corrects,self.counts)]\n",
    "        len_penalty = exp(1 - self.targ_len/self.pred_len) if self.pred_len < self.targ_len else 1\n",
    "        bleu = len_penalty * ((precs[0]*precs[1]*precs[2]*precs[3]) ** 0.25)\n",
    "        return add_metrics(last_metrics, bleu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### 1.5 Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from fastai.callbacks import SaveModelCallback\n",
    "\n",
    "learn = Learner(\n",
    "    data, model, loss_func=cross_entropy, \n",
    "    metrics=[seq2seq_acc, CorpusBLEU(len(data.y.vocab.itos))])\n",
    "\n",
    "learn.callbacks.append(\n",
    "    SaveModelCallback(learn, every='improvement', monitor='valid_loss', name='baseline'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Min numerical gradient: 6.31E-03\n",
      "Min loss divided by 10: 2.29E-03\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAfzElEQVR4nO3deZxcZZ3v8c+vlt63JN1ZSICGAAEU2QLCuIEgN6JXRGGU1zDKlbmMOupFEcZlxtFxV5TXjL6uXpyLIioOm4p4FRBFRgGhISREJEAwezrdWaqru6uqu5bn/lGnkibpTjrddarq1Pm+X6/z6qqzPk86/TvPec5zfsecc4iISHhEql0AERGpLAV+EZGQUeAXEQkZBX4RkZBR4BcRCZlYtQswHd3d3a63t7faxRARCZQnnnhih3OuZ9/5gQj8vb299PX1VbsYIiKBYmYbJpuvrh4RkZBR4BcRCRkFfhGRkFHgFxEJGQV+EZGQUeAXEQkZBX4RkZBR4BcRqUHbkxm+dt9aXhwcKfu+FfhFRGrQhp0pvvGbF9iayJR93wr8IiI1KJEaB6CzOV72fSvwi4jUoKF0FoCuFgV+EZFQKAX+TgV+EZFwGEpniRi0NZQ/l6YCv4hIDUqksnQ2x4lErOz7VuAXEalBQ+msLzd2QYFfRKQmJdJZOlsafNm3Ar+ISA1Si19EJGSGUuN0KfCLiISHWvwiIiFSKDiG0llfHt4CBX4RkZozMp6j4PxJ1wAK/CIiNWco5T21q8AvIhIOCQV+EZFw2ZugTeP4RURCIZH2LyUzKPCLiNQcP1MygwK/iEjNUR+/iEjIJNNZGmMRmuJRX/avwC8iUmNKKZn9osAvIlJj/HxqFxT4RURqTiI9rha/iEiYDKVzdDb7M4YffAz8ZnaTmQ2Y2Zp95n/QzJ41sz+Z2Vf8Or6ISFANpYLb4v8esGLiDDM7F7gIONk59zLgeh+PLyISSIHt43fOPQTs2mf2+4AvOefGvHUG/Dq+iEgQZfMFRsfzgW3xT+Y44DVm9kcz+52ZnTHVimZ2lZn1mVnf4OBgBYsoIlI9fj+1C5UP/DFgLnAWcC1wm5nZZCs65250zi13zi3v6empZBlFRKrG76d2ofKBfzNwlyt6DCgA3RUug4hIzSq1+Osp8P8UOBfAzI4DGoAdFS6DiEjNGvIyc/qVkhmKXS++MLNbgXOAbjPbDPwLcBNwkzfEcxx4t3PO+VUGEZGgqUSL37fA75y7bIpFl/t1TBGRoCv18XfVUVePiIgcQKnF36HALyISDolUlvamGNHIpAMey0KBX0SkhiTT/qZkBgV+EZGakvA5XQMo8IuI1JQhtfhFRMIlkRqny8eUzKDALyJSU4bSWV9H9IACv4hIzXDO+Z6SGRT4RURqRmo8Tzbv1McvIhIWe1IyK/CLiIRDJVIygwK/iEjN2JOgTX38IiLhUErJrBa/iEhI7H3tosbxi4iEgvr4RURCZiidJRYxWhuivh5HgV9EpEYkvDw9Zv6lZAYFfhGRmjGUzvo+ogcU+EVEasZQyv/MnKDALyJSM4bSWd+f2gUFfhGRmpFIj/s+lBMU+EVEaoa6ekREQiRfcCQzOQV+EZGwGM5U5uEtUOAXEakJpad2/X4JCyjwi4jUhD2ZOdXiFxEJh0RaLX4RkVBRi19EJGSGUqVc/BrHLyISCmrxi4iETCKVpaUhSkPM/7CswC8iUgNKKZkrwbfAb2Y3mdmAma2ZZNk1ZubMrNuv44uIBIVzjhcHR5hTgTw94G+L/3vAin1nmtnhwAXARh+PLSISGL/+8wBPbkxw6fIlFTmeb4HfOfcQsGuSRTcA1wHOr2OLiATFWC7P537xDMfOb+Pys46syDEr2sdvZhcBW5xzq6ax7lVm1mdmfYODgxUonYhI5d30+/Vs2JniU//9ROLRyoTkigV+M2sBPgF8ajrrO+dudM4td84t7+np8bdwIiJVMJDM8M3fPM/5JyzgNcdWLs5VssW/FDgKWGVm64ElwJNmtrCCZRARqRlfuXct2bzjn950QkWPG6vUgZxzTwPzS9+94L/cObejUmUQEakVqzYluOOJzbz3dUvp7W6t6LH9HM55K/AIsMzMNpvZlX4dS0QkSAoFx6d//id62hv5wOuPqfjxfWvxO+cuO8jyXr+OLSJSqwoFx8fvepqVGxNcf+nJtDVWrONlj8ofUUQkpPIFxz/euZo7ntjMh15/DG8/bXFVyqHALyJSAfmC46O3r+InK7dw9fnHcvX5x1WtLAr8IiI+y+ULfOS2Vdy9aivXvOE4PnjesVUtjwK/iIiPcvkCV//nU9yzehvXrVjG+8+p/M3cfSnwi4j4JJcv8OHbVnHP6m187I3H897XLa12kQClZRYR8UWpe+fnq7bWVNAHBX4RkbLL5Qtcc3uxT/8fV9RW0AcFfhGRsnLOcd0dq/nZU1u5bsUy3ndObQV9UOAXESmrzbvT3LVyC1e99uiauJE7GQV+EZEy2p0aB+DM3rlVLsnUFPhFRMoomc4B0FGh9+fOhAK/iEgZJTNZANqbane0vAK/iEgZDXuBXy1+EZGQ2NPVoxa/iEg4JDNZIgatDQr8IiKhkExnaW+KE4lYtYsyJQV+EZEySmZydDTXbmsfphn4zWypmTV6n88xsw+ZWZe/RRMRCZ5kOktHU+3e2IXpt/jvBPJmdgxwI3A48CPfSiUiElDJTP0E/oJzLgdcDHzDOXctsMi/YomIBFMyXSddPUDWzC4D3g3c482r7VOaiEgV1FOL/38AZwOfd879xcyOAm7xr1giIsGUTGdr+uEtmOYbuJxzzwAfAjCzOUC7c+7LfhZMRCRocvkCo+P5+mjxm9mDZtZhZnOBJ4HvmNnX/S2aiEiwDGdKCdrqo4+/0zmXBN4GfN8590rgfP+KJSISPKUEbXXR4gdiZrYI+Gv23twVEZEJgpCSGaYf+P8VuBdY55x73MyOBp73r1giIsGzt8Vf21090725eztw+4TvLwJv96tQIiJBlEzXfkpmmP7N3SVm9hMzG/CmO81sid+FExEJkmQAcvHD9Lt6vgvcDRzmTT/35omIiKfUx1/Lb9+C6Qf+Hufcd51zOW/6HtDjY7lERAJnOJPFDNpqOBc/TD/w7zSzy80s6k2XAzv9LJiISNAkMznaG2M1nYsfph/430NxKGc/sA24BLjiQBuY2U3e/YA1E+Z91cyeNbPV3j0DpXYWkboRhHQNMM3A75zb4Jx7i3Ouxzk33zn3Vg4+qud7wIp95t0PvNw59wrgOeDjh1pgEZFaFYQEbTC7N3B95EALnXMPAbv2mXefl94Z4FFAI4NEpG4EISUzzC7wz7YT6z3AL6fcudlVZtZnZn2Dg4OzPJSIiP/C0OJ3M93QzD4J5IAfTrlz5250zi13zi3v6dEAIhGpfUHp4z/gNYmZDTN5gDegeSYHNLMrgDcD5znnZnzyEBGpNclMLhAt/gMGfudcezkPZmYrgOuA1znnUuXct4hINeXyBUbG6r+P/4DM7FbgEWCZmW02syuBbwLtwP1m9pSZfduv44uIVNLImJeZM+gt/tlwzl02yez/69fxRESqKSgpmcHHFr+ISJgEJSUzKPCLiJRFUFIygwK/iEhZBOW1i6DALyJSFnv7+NXVIyISCkF5CQso8IuIlEUyHYxc/KDALyJSFslMjrYA5OIHBX4RkbIISoI2UOAXESmLYkpmBX4RkdAotvhrv38fFPhFRMoiKCmZQYFfRKQshgOSkhkU+EVEyqLY4ldXj4hIKOQLjuExtfhFREJjJBOclMygwC8iMmtBSskMCvwiIrM2FKCUzKDALyIya0FKyQwK/CIisxaklMygwC8iMmtq8YuIhEyQXrsICvwiIrOWzOQwg/ZGdfWIiIRCMp2lrSEYufhBgV9EZNaSmeAkaAMFfhGRWUumc7QH5OEtUOAXEZm1YbX4RUTCJRmglMygwC8iMmtBSskMCvwiIrMWpBetgwK/iMisFAqOkbHgvGgdFPhFRGZleCyHc8FJyQwK/CIisxK0dA3gY+A3s5vMbMDM1kyYN9fM7jez572fc/w6vohIJQQtQRv42+L/HrBin3kfAx5wzh0LPOB9FxEJrKClZAYfA79z7iFg1z6zLwJu9j7fDLzVr+OLiFSCWvwHt8A5t8373A8smGpFM7vKzPrMrG9wcLAypRMROUSlPv5O9fEfnHPOAe4Ay290zi13zi3v6empYMlERKYvmfG6etTin9J2M1sE4P0cqPDxRUTKqtTib9NwzindDbzb+/xu4GcVPr6ISFklM1naG2NEA5KLH/wdznkr8AiwzMw2m9mVwJeAN5jZ88D53ncRkcDalsjQ095Y7WIcEt+uTZxzl02x6Dy/jikiUmlrtw+zbEF7tYtxSPTkrojIDGWyedbvHGXZQgV+EZFQeH77CM7B8Qr8IiLhsHb7MADHKfCLiITD2v4kjbEIvfNaq12UQ6LALyIyQ8/2D3PsgrZADeUEBX4RkRl7bvswxwVsRA8o8IuIzEgiNc725FjgbuyCAr+IyIw821+8sbtsYUeVS3LoFPhFRGbgOW9ET9Ae3gIFfhGRGXm2f5jO5jgLOoKVrgF8TNlQC+5ZvZXH/7ILBzgHBVfMAh2PRmiMR2iMRWmMRYhHDcOwCTfmC86RzTvyBUeu4CgUHAXn9uwLIBqBqBnRSIRoBMzbgXNuzzqRiBGPFteJRYxIxIhYcbtIxIhFjFg0QkPUiEcjxKMRopFiWSJm3gSxaHH7eDRCLGrEIxGiUSMeMaJ79hHxjmV7yiIi/ljbX0zVEMS/tboO/Ks3D/HTp7buCaKlX894vsBYrsB4rjDtfUVK+zAo7angiieFWmPmndyiERpiERpjpZ9RmuIRGuNRmuPFz62NMdobY7Q1xWhtjNHWGKPJW94cj9LcUJxaG2K0NERpbYzR2hilMRatdjVFqsY5x3P9w7z11MXVLsqM1HXg/8SFJ/CJC0+Ycnmh4BjPF8gVHM5ropda9DGvJR2PRogYBzyrF0pXBc7tOTGUVs8X9l415PIF8t7VQGl+cVmBbN6RzRfI5gvkC8WTSmHCuqV1cvm9n/N7fha3zRUc2VxxH2P5AtmcYyyXZzxXPNGN5fJksgXS2TyJ1DjpbJ7RsTwjYzlGxnLkD+Ek1hCL0NEUp6M5RkdTnPamGB3N8eK80ufmOJ0TprktDcxta6C1IRrIVpJIydahDMNjucDl6Cmp68B/MJGI0RSZfcs1EjEapniAIx6QhrFzjky2wMhYjkw2TyabJ53Nkx7Pk8rmSY3lSY3nSI0XTxTJTJZkOsdwJstQOstwJseWRJpkOkcynWU8P/XVVEMswrzWBuZ6U/FzI/PaGvbMn9fWSHdb8adOFFJrntszokeBXwLMzPZ065RDJpsnmS6eFErTrtHx/aado+Ns2Jli1+g4I2O5SffVEIvQ7Z0M5rU10NPWSE97I/PbG+lpb2J+RyMLvJ9NQTnTSqCVhnIG8eEtUOAXnzTFozTFo8zvaJr2Nplsfs8JYcfIGDtHxtk5OsbO0fHi55ExdoyM8+y2YXaMjE16f6WrJb7nJDC/vYkFHcUTxIKOJhZ0NrGgo4n57Y3EoxrQJjO3tj/JYZ1NgXrB+kQK/FIzmuJRDutq5rCu5oOuWyg4EuksA8MZBpJj9CczDCQzbC99Hh7jhYEdDA7vf4Iwg3mtjSzsbGRhRzOLOptY6J0U9l5JNDK3pYFIwHKwSGWs3T4SuIycEynwSyBFIrbnHsHxC6der1Bw7EqNsz2Z8aYx+oeKn/uTGTbvTtG3YReJVHa/bWMRo6d0tdBR/Lmws6l4ouho5rCu4slC3Uvhks0XWDcwwmuP6652UWZMgV/qWiRidLc10t3WyMsO65xyvfR4noHhDIPDYwwMjzHgXTVsT44xMJzhxcFRHl63k+HM/vchetobWTKnmSVzWlgyp5nFXd40p3j10taoP7N6sn7HKOP5QiBz9JTof6QI0NwQ5ch5rRx5kLzqo2M5+pMZtiUybBtKszWRYUsixebdaVZvTvCrNdvI5l/atdTZHGdRZxOLu5pZ1NXEYV3F7qUFHU0s6mxmYUdT2W6qi//2vHwloDd2QYFf5JC0NsZY2tPG0p62SZfnC47B4TG2JFJsSWTYsjvN1kSabUNptiQyPLFx96TdSqWTw6LOJhZ1NXNYZ/GkULznUexi0kNztWFt/zDRiHHM/Mn/DwSBAr9IGUUjxkLvZvHpR06+Tmo8R/9QpjglM2wbKl499A8VP6/aPMSu0fH9tpvf3sgRc1u8K5MWb2qld14LXS0NPtdMSp7tH+ao7tZAn4gV+EUqrKUhxtE9bRw9xVUDFIe2bhvKsDWR9qbijegNu1L84YUd3Plk5iXrdzTF6O1u5Yi5LfTOa+WIeS0c6Z0k5rc3anTSDP1p6xC/WL2NJXNa6J3XwhHzWljbP8xJi6e+XxQECvwiNagpHuWo7laO6p78nkMmm2fjrhQbdqbYsHOU9TtH2bAzxarNCX65pv8l6TfiUfO6jfbeX5jfXnyeofS8w/yOxkC3YP2Qyxe4+sdP8fzAyH7LLjl9SRVKVD4K/CIB1BSPctyC9klvMGbzBbbsTrNhV4qNu1ITrhrS/PHFXfQnM5PmZZrb2uDdcG6id14rpx7RxalHdLG4qzmUKTNufWwjzw+M8K2/OY2TlnSycWeK9TtT9CczvOOMw6tdvFlR4BepM/FohN7uVnqnuFooPdsw4A1VLT0A15/M7Ln38PC6Hdz0h78AxeGqpxzetWd6xZJO2puC+cTqdA2ls3z9/uc46+i5rHj5QsyMJXNa+Ktjql2y8lDgFwmZic82nMjkrw3M5gus7R9m5cbdrNyUYOXGBPc/sx0oPvm8tKeNVx/TzTvPPJzj93314Lp18LWvwQ9+ACMj0NYGl18O11wDS5f6Xb2y+MYDz5NIZ/nnN59Yl1c7VkpHXMuWL1/u+vr6ql0MkVBLpMZZvXmIVZsSrNyU4Pcv7GA8V+DUI7q47MwjePMrFtHywP1wySWQzRankni8ON1xB7zxjdWrxDT8ZccoF9zwO9526hK+fMkrql2cWTGzJ5xzy/ebr8AvIjOxe3Scu1Zu4Ud/3MC6wVFOSA1w93f+gXgmPfVGLS2wenVNt/yv+n4ff3hhB7+99hzmt08/yWAtmirwK0WhiMzInNYGrnz1Ufz6I6/j9veezbVrfoEb3//5g5fIZuGGGypTwBl4eN0O7ntmO+8/95jAB/0DUeAXkVkxM87oncvr++6joZA/8MrZLNxyS2UKdojyBcdn7/kzi7uaufLVR1W7OL5S4BeR8hjZf7z7rNarsB89tpE/b0vy8QuPr/uMq1UJ/Gb2YTP7k5mtMbNbzax+r6lEwqJtmrlrprteBe0aHef6e9dy9tHzeNNJi6pdHN9VPPCb2WLgQ8By59zLgSjwzkqXQ0TK7PLLiyN3DiAbifJfr1zBr5/ZzljuIN1CFfTVe59ldCzHZy56WV0O39xXtcbxx4BmM8sCLcDWKpVDRMrlmmvg5ptfOoxzHy4e58snvpE13++joynGf3vZQt5xxuEs751bwYK+1KpNCX78+CaufNVRgU61fCgq3uJ3zm0Brgc2AtuAIefcffuuZ2ZXmVmfmfUNDg5WupgicqiWLi2O029p2b/lH49DSwsNP7mLu65/F9+94gzOP3EBv1zTzyXffoRLv/0wv312gMmGlxcmSS9RLoWC41M/W0N3WyP/6/xjfTtOran4OH4zmwPcCbwDSAC3A3c4534w1TYaxy8SIOvWFYds3nLL3id3//Zv4cMf3m/8fmo8x38+vonvPPQiW4cyHL+wnXeecTi7Rsd5YXCEFwZGWL8jxeI5zVx86mIuPnUxh89tKVtRf/zYRj5219Pc8I6TufjUYCdem0zNPMBlZpcCK5xzV3rf3wWc5Zx7/1TbKPCL1LdsvsDPntrKt3+3jhcGRogYHDG3hWPmt3FUdytPbxni0Rd3AXBm71zecsphnL10Hkd3t864Tz6RGufc6x/kmPlt3Pb3Z9dl3/5Ugb8affwbgbPMrAVIA+cBiuoiIRaPRrjk9CW87dTFbNqdmvQl9pt3p/jZU1u588nN/NNP1wAwr7WB046cwxm9c3jTKw5jcVfztI63ZssQ196xmqF0lk+/JRw3dCeqSsoGM/sMxa6eHLAS+Dvn3NhU66vFLyIlzjnWDY7Qt343fRt207d+F+t3pmiIRfi7Vx/F+889ZsoX3I/l8nzjgRf41u/WMbe1gS9efBLnn7igwjWonJrp6pkJBX4ROZBNu1J8/f7n+MnKLXS3NXDNBcv46+WHE40YmWyeHSNjvDg4ymfveYbnB0Z4+2lL+Oc3n1D3r6xU4BeRurdqU4LP/eIZHl+/m3mtDYzlCoyM5fYsX9TZxBfedhLnLptfxVJWTi318YuI+OLkw7u47e/P5pdr+vn1M9vpbIl77x5ooLutkVcePW/KbqAw0b+AiNQVM+PCkxZxYQhSL8yUkrSJiISMAr+ISMgo8IuIhIwCv4hIyCjwi4iEjAK/iEjIKPCLiISMAr+ISMgEImWDmQ0CGybM6gSGJll13/kH+j7Z525gxyyLO1XZDmW9yZZNZ57qp/odTLXqN526qn4Hd6j1O9I517PfUudc4CbgxunMP9D3yT4DfX6V7VDWm2zZdOapfqpfrdZvOnVV/SpTP+dcYLt6fj7N+Qf6PtXn2Zruvg603mTLpjNP9Zs91e/g682kftOt62ypftNYLxBdPZViZn1ukkx29UL1CzbVL9hqqX5BbfH75cZqF8Bnql+wqX7BVjP1U4tfRCRk1OIXEQkZBX4RkZCp28BvZjeZ2YCZrZnBtqeb2dNm9oKZ/buZ2YRlHzSzZ83sT2b2lfKW+pDKWPb6mdmnzWyLmT3lTReWv+TTLqMvvz9v+TVm5sysu3wlPuQy+vH7+6yZrfZ+d/eZ2WHlL/m0y+hH/b7q/e2tNrOfmFlX+Us+7TL6Ub9LvbhSMDN/bwLPdlxprU7Aa4HTgDUz2PYx4CzAgF8Cb/Tmnwv8Gmj0vs+vs/p9GvhotX93ftXPW3Y4cC/FBwK766l+QMeEdT4EfLvO6ncBEPM+fxn4cp3V7wRgGfAgsNzP8tdti9859xCwa+I8M1tqZr8ysyfM7L/M7Ph9tzOzRRT/gB51xd/G94G3eovfB3zJOTfmHWPA31pMzaf61Qwf63cDcB1Q1VENftTPOZecsGorVayjT/W7zzlXenP6o8ASf2sxNZ/q92fn3NpKlL9uA/8UbgQ+6Jw7Hfgo8L8nWWcxsHnC983ePIDjgNeY2R/N7HdmdoavpT10s60fwAe8S+mbzGyOf0WdkVnVz8wuArY451b5XdAZmvXvz8w+b2abgL8BPuVjWWeiHP8/S95DsbVcS8pZP1+F5mXrZtYG/BVw+4Qu38ZD3E0MmEvxMu0M4DYzO9o7c1dVmer3LeCzFFuKnwW+RvEPrOpmWz8zawE+QbG7oOaU6feHc+6TwCfN7OPAB4B/KVshZ6Fc9fP29UkgB/ywPKWbvXLWrxJCE/gpXt0knHOnTJxpZlHgCe/r3RSD38RLyCXAFu/zZuAuL9A/ZmYFiomXBv0s+DTNun7Oue0TtvsOcI+fBT5Es63fUuAoYJX3h7kEeNLMznTO9ftc9ukox//PiX4I/D9qJPBTpvqZ2RXAm4HzaqHBNUG5f3/+qtbNkUpMQC8Tbr4ADwOXep8NOHmK7fa9+XKhN/+9wL96n48DNuE9BFcn9Vs0YZ0PAz+up9/fPuusp4o3d336/R07YZ0PAnfUWf1WAM8APdWsl9//P6nAzd2q/+P5+Eu5FdgGZCm21K+k2OL7FbDK+w/0qSm2XQ6sAdYB3ywFd6AB+IG37Eng9XVWv1uAp4HVFFsniypVn0rUb591qhr4ffr93enNX00xSdfiOqvfCxQbW095UzVHLflRv4u9fY0B24F7/Sq/UjaIiIRM2Eb1iIiEngK/iEjIKPCLiISMAr+ISMgo8IuIhIwCvwSSmY1U+Hj/YWYnlmlfeS+D5hoz+/nBskyaWZeZvb8cxxYBvYFLAsrMRpxzbWXcX8ztTQDmq4llN7Obgeecc58/wPq9wD3OuZdXonxS/9Til7phZj1mdqeZPe5Nr/Lmn2lmj5jZSjN72MyWefOvMLO7zew3wANmdo6ZPWhmd3h53384IVf6g6Uc6WY24iVDW2Vmj5rZAm/+Uu/702b2uWlelTzC3iRybWb2gJk96e3jIm+dLwFLvauEr3rrXuvVcbWZfaaM/4wSAgr8Uk/+DbjBOXcG8HbgP7z5zwKvcc6dSjFj5RcmbHMacIlz7nXe91OBq4ETgaOBV01ynFbgUefcycBDwP+ccPx/c86dxEszME7Ky+NyHsWnpAEywMXOudMovvvha96J52PAOufcKc65a83sAuBY4EzgFOB0M3vtwY4nUhKmJG1S/84HTpyQHbHDy5rYCdxsZsdSzDwan7DN/c65iXnVH3PObQYws6co5mP5/T7HGWdvArsngDd4n89mb+7/HwHXT1HOZm/fi4E/A/d78w34ghfEC97yBZNsf4E3rfS+t1E8ETw0xfFEXkKBX+pJBDjLOZeZONPMvgn81jl3sddf/uCExaP77GNswuc8k/+NZN3em2NTrXMgaefcKV6q6HuBfwD+nWIO/R7gdOdc1szWA02TbG/AF51z/+cQjysCqKtH6st9FLNSAmBmpRS5nexNfXuFj8d/lGIXE8A7D7aycy5F8RWJ15hZjGI5B7ygfy5wpLfqMNA+YdN7gfd4VzOY2WIzm1+mOkgIKPBLULWY2eYJ00coBtHl3g3PZyim0Qb4CvBFM1uJv1e5VwMfMbPVwDHA0ME2cM6tpJhN8zKKOfSXm9nTwLso3pvAObcT+IM3/POrzrn7KHYlPeKtewcvPTGIHJCGc4qUidd1k3bOOTN7J3CZc+6ig20nUmnq4xcpn9OBb3ojcRLUyGsrRfalFr+ISMioj19EJGQU+EVEQkaBX0QkZBT4RURCRoFfRCRk/j8YNRj0bGac/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(suggestion=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>seq2seq_acc</th>\n",
       "      <th>bleu</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.580141</td>\n",
       "      <td>2.943090</td>\n",
       "      <td>0.569072</td>\n",
       "      <td>0.273858</td>\n",
       "      <td>00:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.840177</td>\n",
       "      <td>2.816740</td>\n",
       "      <td>0.549650</td>\n",
       "      <td>0.251734</td>\n",
       "      <td>00:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.464854</td>\n",
       "      <td>2.669522</td>\n",
       "      <td>0.576558</td>\n",
       "      <td>0.296858</td>\n",
       "      <td>00:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.365971</td>\n",
       "      <td>3.272397</td>\n",
       "      <td>0.511141</td>\n",
       "      <td>0.332937</td>\n",
       "      <td>00:17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with valid_loss value: 2.9430902004241943.\n",
      "Better model found at epoch 1 with valid_loss value: 2.8167402744293213.\n",
      "Better model found at epoch 2 with valid_loss value: 2.6695215702056885.\n"
     ]
    }
   ],
   "source": [
    "learn.fit_one_cycle(4, 3e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.6 Cualitative evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('baseline');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.eval()\n",
    "model.cpu();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def greedy_decode(inp):\n",
    "    out =  model(inp)\n",
    "    return out.argmax(-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.6.1 Beam search\n",
    "\n",
    "Not a good implementation of beam search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def select_nucleus(probs, p=0.1, max_length=5):\n",
    "    idxs = torch.argsort(probs, descending=True)\n",
    "    res, cumsum = [], 0.\n",
    "    for idx in idxs:\n",
    "        res.append(idx)\n",
    "        cumsum += probs[idx]\n",
    "        if cumsum > p or len(res) >= max_length: \n",
    "            return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def beam_decode(inp, bos_idx=2, unk_idx=0, pad_idx=1, beam_size=5, nucleus=0.3, no_unk=False):\n",
    "    h = model.encoder(1, inp)\n",
    "    \n",
    "    finised = []  # finished phrases\n",
    "    next_word = None\n",
    "    current_ys = [(inp.new_zeros(1).long() + bos_idx, 0)]\n",
    "    while current_ys:\n",
    "        next_ys = []\n",
    "        for ys, path_score in current_ys:\n",
    "            out, h = model.decoder(ys[None, -1], h)\n",
    "            probs = F.softmax(out[0], dim=-1)\n",
    "            \n",
    "            if no_unk:\n",
    "                probs[unk_idx] = -float('Inf')\n",
    "            \n",
    "            likely = select_nucleus(probs, p=nucleus)\n",
    "            chosen_words = random.sample(likely, k=min(beam_size, len(likely)))\n",
    "            \n",
    "            for y in chosen_words:\n",
    "                phrase = torch.cat([ys, \n",
    "                        torch.Tensor([y]).long()], dim=0)\n",
    "                \n",
    "                score = path_score + torch.log(probs[y])\n",
    "                \n",
    "                if y == pad_idx or len(phrase) > 20:\n",
    "                    finised.append((phrase, score))\n",
    "                else:\n",
    "                    next_ys.append((phrase, score))\n",
    "                \n",
    "        current_ys = sorted(next_ys, key=lambda k: k[1], reverse=True)[:beam_size]\n",
    "    return sorted(finised, key=lambda k: k[1]/len(k[0]), reverse=True)[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pretty(text_):\n",
    "    if isinstance(text_, text.data.Text):\n",
    "        text_ = text_.text\n",
    "    maj = False\n",
    "    up = False\n",
    "    tokens = []\n",
    "    for t in text_.split(' '):\n",
    "        if t in ['xxbos']:\n",
    "            continue\n",
    "        elif t == 'xxmaj':\n",
    "            maj = True\n",
    "            continue\n",
    "        elif t == 'xxup':\n",
    "            up = True\n",
    "            continue\n",
    "        if maj:\n",
    "            t = t[0].upper() + t[1:]\n",
    "            maj = False\n",
    "        if up:\n",
    "            t = t.upper()\n",
    "            up = False\n",
    "        tokens.append(t)\n",
    "    return ' '.join(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_phrases = ['Let it go.', 'I\\'m your father.', 'You talking to me?', 'A census taker once tried to test me. I ate his liver with some fava beans and a nice Chianti.', 'I\\'ll be back.', 'I see dead people.', 'Houston, we have a problem.', 'Say hello to my little friend!', 'I did not hit her! I did not! Oh hi, Mark.', 'I\\'m king of the world!', 'This is Sparta!', 'I want to play a game.', 'I have had it with these motherfucking snakes on this motherfucking plane!', 'You\\'re a wizard, Harry.', 'With great power comes great responsibility.', 'I don\\'t like sand. It\\'s all coarse, and rough, and irritating. And it gets everywhere.', 'Not the bees!', 'Why so serious?', 'I\\'m gonna steal the Declaration of Independence.', 'You shall not pass!', 'Are you not entertained?', 'King Kong ain\\'t got shit on me!', 'I live my life a quarter mile at a time.', 'Water? Like out of the toilet?', 'They may take our lives, but they\\'ll never take our freedom!', 'I will look for you, I will find you, and I will kill you.', 'My precious.', 'The first rule of Fight Club is: You do not talk about Fight Club.', 'You\\'re gonna need a bigger boat.']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Let it go\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Xxunk xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "De\n",
      "==================================================\n",
      "\n",
      "i am your father\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¿ No que que que que que\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¿ Que que\n",
      "==================================================\n",
      "\n",
      "You talking to me ?\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Cómo\n",
      "==================================================\n",
      "\n",
      "a xxunk xxunk once tried to test me . i ate his xxunk with some xxunk xxunk and a nice Xxunk .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No que que y\n",
      "==================================================\n",
      "\n",
      "i 'll be back .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No que que que . . .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No que .\n",
      "==================================================\n",
      "\n",
      "i see dead people .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No que que que que que xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No a .\n",
      "==================================================\n",
      "\n",
      "Xxunk , we have a problem .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      ",\n",
      "==================================================\n",
      "\n",
      "Say hello to my little friend !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No xxunk xxunk xxunk xxunk xxunk xxunk ! ! ! ! ! ! ! !\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No no ! !\n",
      "==================================================\n",
      "\n",
      "i did not hit her ! i did not ! Oh hi , Mark .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No , , , , , , , , , , , , , , , , ,\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No , , , !\n",
      "==================================================\n",
      "\n",
      "i xxunk king of the world !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No xxunk xxunk xxunk xxunk xxunk ! !\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No he !\n",
      "==================================================\n",
      "\n",
      "This is Xxunk !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "\n",
      "==================================================\n",
      "\n",
      "i want to play a game .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No que que que que que que xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No no que\n",
      "==================================================\n",
      "\n",
      "i have had it with these xxunk xxunk on this xxunk plane !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¡ no xxunk xxunk xxunk xxunk ! !\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No no !\n",
      "==================================================\n",
      "\n",
      "You 're a xxunk , Harry .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      ", ,\n",
      "==================================================\n",
      "\n",
      "With great power comes great responsibility .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Que el\n",
      "==================================================\n",
      "\n",
      "i do xxunk like sand . It 's all xxunk , and rough , and xxunk . And it gets everywhere .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No xxunk y y y y y y y xxunk xxunk y y xxunk xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No y y .\n",
      "==================================================\n",
      "\n",
      "Not the xxunk !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "!\n",
      "==================================================\n",
      "\n",
      "Why so serious ?\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Qué qué qué qué qué qué qué qué qué qué qué qué qué qué qué qué qué\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Qué qué qué\n",
      "==================================================\n",
      "\n",
      "i xxunk gon na steal the Xxunk of Xxunk .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No que que\n",
      "==================================================\n",
      "\n",
      "You shall not pass !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No Xxunk xxunk xxunk xxunk ! ! !\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No\n",
      "==================================================\n",
      "\n",
      "Are you not xxunk ?\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "\n",
      "==================================================\n",
      "\n",
      "King Kong xxunk xxunk got shit on me !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No !\n",
      "==================================================\n",
      "\n",
      "i live my life a quarter mile at a time .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No que que que que que que que que que que xxunk de\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No que a\n",
      "==================================================\n",
      "\n",
      "Water ? Like out of the toilet ?\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "De\n",
      "==================================================\n",
      "\n",
      "They may take our lives , but they 'll never take our freedom !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      ", No\n",
      "==================================================\n",
      "\n",
      "i will look for you , i will find you , and i will kill you .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No , , , , , , , , , , , , , , , , ,\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No , , . .\n",
      "==================================================\n",
      "\n",
      "My precious .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Xxunk xxunk xxunk xxunk xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Que\n",
      "==================================================\n",
      "\n",
      "The first rule of Fight Club is : You do not talk about Fight Club .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "No que que que que Que xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "No que es\n",
      "==================================================\n",
      "\n",
      "You 're gon na need a bigger boat .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk xxunk\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Que que\n",
      "==================================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer = Tokenizer()\n",
    "tok = SpacyTokenizer('en')\n",
    "\n",
    "def show_predictions():\n",
    "    for x in test_phrases:\n",
    "        x = torch.tensor([[data.x.vocab.stoi[t] for t in tokenizer.process_text(x, tok)]])\n",
    "        print(pretty(data.x.reconstruct(x[0])))\n",
    "        print('~' * 10 + ' greedy ' + '~' * 10)    \n",
    "        print(pretty(data.y.reconstruct(greedy_decode(x)[0])))\n",
    "        print('~' * 10 + '  beam  ' + '~' * 10)    \n",
    "        print(pretty(data.y.reconstruct(beam_decode(x, no_unk=True))))\n",
    "        print('=' * 50 + '\\n')\n",
    "show_predictions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Extras\n",
    "\n",
    "This baseline was meant to provide a basic understanding of the problem but there are things that can be improved and potentially re-used for the more complex models. If you decide so, here are some things you can add to the baseline to get your feet wet, if you have other things to add to this list don't hessitate to propose it. \n",
    "\n",
    "- Align batches in a way that allows us to keep h between batches, similar to what is done in [notebook 6](https://github.com/fastai/course-nlp/blob/master/6-rnn-english-numbers.ipynb) of fast.ai's course.\n",
    "- Handle UNK better using a sub-word tokenizer like [BPT](https://github.com/rsennrich/subword-nmt) or [SentencePiece](https://github.com/google/sentencepiece). This would render fastext pre-trained embeddings obsolete so an alternative is necessary.\n",
    "- Train with more data and for longer.\n",
    "- [RAdam](https://medium.com/@lessw/new-state-of-the-art-ai-optimizer-rectified-adam-radam-5d854730807b) a new optimizer that can easily be integrated and promises better results.\n",
    "- Beam search, we are using a variant here, but didn't test it in depth. Take a look to [this paper](https://arxiv.org/abs/1904.09751) for an analysis on open-ended and non-open-ended text generation.\n",
    "- Try [QRNN](https://arxiv.org/abs/1611.01576) instead of GRU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Advanced solutions\n",
    "\n",
    "Now that we have a basic understanding of the problem we will dive into more novel solutions. The idea here is that you choose an approach that you want to try and implement it. It's beeter if you try a different solution than mine so that we can compare results later.\n",
    "\n",
    "Some ideas are:\n",
    "\n",
    "- ELMO embeddings. Pre-trained weights just in english.\n",
    "- BERT embeddings. English and multilingual pre-trained models could come handy.\n",
    "- TransformXL.\n",
    "- XLM embeddings. They have a [multilingual model](https://github.com/facebookresearch/XLM#pretrained-cross-lingual-language-models) pre-trained.\n",
    "- Distill some of the complex embeddings as described on [this paper](https://arxiv.org/pdf/1503.02531.pdf).\n",
    "- [LightConv and DynamicConv](https://arxiv.org/abs/1901.10430), an alternative to attention, [pytorch implementation](https://github.com/pytorch/fairseq) is available.\n",
    "\n",
    "Training one of these more advanced models from scratch is quite lengthy and it might not be possible on the available hardware we have. As a reference, I trained ULMFit from scratch on the whole wikipedia in spanish on a 2080 Ti with 11GB and took ~25hrs ([notebook](./nn-spanish.ipynb)).\n",
    "\n",
    "For this reason I highly recommend using pre-trained models and transfer learning where possible (for english there are pre-trained weights available for all models but for spanish there might not be available).\n",
    "\n",
    "Keep an eye on the latest news, NLP is in the middle of a revolution so new things come up each week."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Transformer\n",
    "\n",
    "Vanilla transformer. Implementation taken from fast.ai's [8th notebook](https://github.com/fastai/course-nlp/blob/master/8-translation-transformer.ipynb). I just changed it a bit to use some of the classes/functions that come in the library and used FastText pre-trained embeddings instead of random ones.\n",
    "\n",
    "Also highly recommend checking the [Annotated paper](http://nlp.seas.harvard.edu/2018/04/03/attention.html) with PyTorch code.\n",
    "\n",
    "<img src=\"images/transformer.png\"> </img>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import copy\n",
    "\n",
    "def clones(module, N):\n",
    "    \"Produce N identical layers.\"\n",
    "    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SublayerConnection(nn.Module):\n",
    "    \"\"\"\n",
    "    A residual connection followed by a layer norm.\n",
    "    Note for code simplicity the norm is first as opposed to last.\n",
    "    \"\"\"\n",
    "    def __init__(self, size, dropout):\n",
    "        super(SublayerConnection, self).__init__()\n",
    "        self.norm = LayerNorm(size)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x, sublayer):\n",
    "        \"Apply residual connection to any sublayer with the same size.\"\n",
    "        return x + self.dropout(sublayer(self.norm(x)))\n",
    "    \n",
    "class LayerNorm(nn.Module):\n",
    "    \"Construct a layernorm module (See citation for details).\"\n",
    "    def __init__(self, features, eps=1e-6):\n",
    "        super(LayerNorm, self).__init__()\n",
    "        self.a_2 = nn.Parameter(torch.ones(features))\n",
    "        self.b_2 = nn.Parameter(torch.zeros(features))\n",
    "        self.eps = eps\n",
    "\n",
    "    def forward(self, x):\n",
    "        mean = x.mean(-1, keepdim=True)\n",
    "        std = x.std(-1, keepdim=True)\n",
    "        return self.a_2 * (x - mean) / (std + self.eps) + self.b_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformerEmbedding(nn.Module):\n",
    "    \"Embedding + positional encoding + dropout\"\n",
    "    def __init__(self, emb, inp_p=0.):\n",
    "        super().__init__()\n",
    "        self.emb_sz = emb.embedding_dim\n",
    "        self.embed = emb\n",
    "        self.pos_enc = PositionalEncoding(self.emb_sz)  # https://github.com/fastai/fastai/blob/master/fastai/text/models/transformer.py#L11\n",
    "        self.drop = nn.Dropout(inp_p)\n",
    "    \n",
    "    def forward(self, inp): \n",
    "        pos = torch.arange(0, inp.size(1), device=inp.device).float()\n",
    "        return self.drop(self.embed(inp) * math.sqrt(self.emb_sz) + self.pos_enc(pos))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadAttention(nn.Module):\n",
    "    def __init__(self, n_heads, d_model, d_head=None, p=0., bias=True, scale=True):\n",
    "        super().__init__()\n",
    "        d_head = ifnone(d_head, d_model//n_heads)\n",
    "        self.n_heads,self.d_head,self.scale = n_heads,d_head,scale\n",
    "        self.q_wgt,self.k_wgt,self.v_wgt = [nn.Linear(\n",
    "            d_model, n_heads * d_head, bias=bias) for o in range(3)]\n",
    "        self.out = nn.Linear(n_heads * d_head, d_model, bias=bias)\n",
    "        self.drop_att,self.drop_res = nn.Dropout(p),nn.Dropout(p)\n",
    "        self.ln = nn.LayerNorm(d_model)\n",
    "        \n",
    "    def forward(self, q, kv, mask=None):\n",
    "        return self.ln(q + self.drop_res(self.out(self._apply_attention(q, kv, mask=mask))))\n",
    "    \n",
    "    def create_attn_mat(self, x, layer, bs):\n",
    "        return layer(x).view(bs, x.size(1), self.n_heads, self.d_head\n",
    "                            ).permute(0, 2, 1, 3)\n",
    "    \n",
    "    def _apply_attention(self, q, kv, mask=None):\n",
    "        bs, seq_len = q.size(0),q.size(1)\n",
    "        wq, wk, wv = map(lambda o: self.create_attn_mat(*o,bs),\n",
    "                       zip((q, kv, kv),(self.q_wgt, self.k_wgt, self.v_wgt)))\n",
    "        attn_score = wq @ wk.transpose(2,3)\n",
    "        if self.scale: \n",
    "            attn_score /= math.sqrt(self.d_head)\n",
    "        if mask is not None: \n",
    "            attn_score = attn_score.float().masked_fill(mask, -float('inf')).type_as(attn_score)\n",
    "        attn_prob = self.drop_att(F.softmax(attn_score, dim=-1))\n",
    "        attn_vec = attn_prob @ wv\n",
    "        return attn_vec.permute(0, 2, 1, 3).contiguous().view(bs, seq_len, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderBlock(Module):\n",
    "    \"Basic block of a Transformer model.\"\n",
    "    # Can't use Sequential directly cause more than one input...\n",
    "    def __init__(self, n_heads, d_model, d_head, d_inner, p=0., bias=True, scale=True, double_drop=True):\n",
    "        \n",
    "        self.mhra = MultiHeadAttention(n_heads, d_model, d_head, p=p, bias=bias, scale=scale)\n",
    "        self.ff   = feed_forward(d_model, d_inner, ff_p=p, double_drop=double_drop)\n",
    "        self.sublayer = clones(SublayerConnection(d_model, p), 2)\n",
    "\n",
    "    def forward(self, x, **kwargs):\n",
    "        x = self.sublayer[0](x, lambda x: self.mhra(x, x, **kwargs))\n",
    "        return self.sublayer[1](x, self.ff)\n",
    "    \n",
    "        return self.ff()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DecoderBlock(nn.Module):\n",
    "    \"Decoder block of a Transformer model.\"\n",
    "    #Can't use Sequential directly cause more than one input...\n",
    "    def __init__(self, n_heads, d_model, d_head, d_inner, p=0., bias=True, scale=True, double_drop=True):\n",
    "        super().__init__()\n",
    "        self.mha1 = MultiHeadAttention(n_heads, d_model, d_head, p=p, bias=bias, scale=scale)\n",
    "        self.mha2 = MultiHeadAttention(n_heads, d_model, d_head, p=p, bias=bias, scale=scale)\n",
    "        self.ff   = feed_forward(d_model, d_inner, ff_p=p, double_drop=double_drop)\n",
    "        self.sublayer = clones(SublayerConnection(d_model, p), 3)\n",
    "    \n",
    "    def forward(self, x, enc, mask_out=None): \n",
    "        x = self.sublayer[0](x, lambda x: self.mha1(x, x, mask_out))\n",
    "        x = self.sublayer[1](x, lambda x: self.mha2(x, enc))\n",
    "        return self.sublayer[2](x, self.ff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_output_mask(inp, pad_idx=1):\n",
    "    return torch.triu(inp.new_ones(inp.size(1),inp.size(1)), diagonal=1)[None,None].bool()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Transformer(Module):\n",
    "    def __init__(\n",
    "            self, emb_enc, emb_dec, n_layers=6, n_heads=8, d_model=300, d_head=32, \n",
    "            d_inner=1024, p=0.1, bias=True, scale=True, double_drop=True, pad_idx=1):\n",
    "        \n",
    "        self.enc_emb = TransformerEmbedding(emb_enc, p)\n",
    "        self.dec_emb = TransformerEmbedding(emb_dec, 0.)\n",
    "        \n",
    "        args = (n_heads, d_model, d_head, d_inner, p, bias, scale, double_drop)\n",
    "        \n",
    "        self.encoder = nn.ModuleList([EncoderBlock(*args) for _ in range(n_layers)])\n",
    "        self.decoder = nn.ModuleList([DecoderBlock(*args) for _ in range(n_layers)])\n",
    "        \n",
    "        self.out = nn.Linear(d_model, emb_dec.num_embeddings)\n",
    "        self.out.weight = self.dec_emb.embed.weight\n",
    "        self.pad_idx = pad_idx\n",
    "        \n",
    "    def forward(self, inp, out):\n",
    "        mask_out = get_output_mask(out, self.pad_idx)\n",
    "        \n",
    "        enc, out = self.enc_emb(inp), self.dec_emb(out)\n",
    "        \n",
    "        enc = compose(self.encoder)(enc)\n",
    "        out = compose(self.decoder)(out, enc, mask_out)\n",
    "        return self.out(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note:** Transformer is allways using TeacherForcing \n",
    "\n",
    "> **Note** Fast.ai's implementation of transformer is missing the residual connection and layer normalization, I add the code necessary from the [annotated transformer notebbok](/The%20Annotated%20Transformer.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def shift_tfm(b):\n",
    "    \"Adds the shifted y to the dataset\"\n",
    "    x,y = b\n",
    "    y = F.pad(y, (1, 0), value=1)\n",
    "    return [x, y[:,:-1]], y[:,1:]\n",
    "\n",
    "data.add_tfm(shift_tfm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model = Transformer(\n",
    "    emb_enc=emb_enc, \n",
    "    emb_dec=emb_dec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.2 Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(\n",
    "    data, model, \n",
    "    loss_func=FlattenedLoss(\n",
    "        LabelSmoothingCrossEntropy, axis=-1),  # the paper reports better results with label smoothing\n",
    "    metrics=[accuracy, CorpusBLEU(len(data.y.vocab.itos))], \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Min numerical gradient: 5.25E-03\n",
      "Min loss divided by 10: 7.59E-02\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xV9f3H8dcng4QkjEDCDBCWICAzAmpdtUW0/sSJWEWcOLCOWq2tv1+po1r3+DkqdaKI259YrUqpFAcrbJAVlgSBBAIkEBIyvr8/7kEjJCSEe+5Nbt7Px+M+cu73nvHOgeSTc77nfI855xARETmUqHAHEBGRuk/FQkREqqViISIi1VKxEBGRaqlYiIhItWLCHcAPKSkpLj09PdwxRETqlXnz5m1zzqVW9llEFov09HQyMzPDHUNEpF4xsw1VfabTUCIiUi0VCxERqZaKhYiIVEvFQkREqqViISIi1VKxEBGRaqlYiIhItXwrFmbWwcy+MLNvzWyZmd3stT9sZivMbLGZfWBmzSss8wczyzKzlWZ2eoX24V5blpnd6VfmI/Xl6lxWbikIdwwRkaDz88iiFLjNOdcLGAqMM7NewFSgj3OuL7AK+AOA99kooDcwHHjWzKLNLBp4BjgD6AVc7M0bdPtKy/nzlGVszCs87GW/217Ila/M5ZIXZrNtd7EP6UREwse3YuGc2+ycm+9NFwDLgfbOuc+dc6XebLOANG96BPCmc67YObcOyAIGe68s59xa59w+4E1v3qDbsquIDxZsYszLc9hZuO+wln106kqio4z8vSXc+d5i9FApEYkkIRnuw8zSgQHA7AM+uhJ4y5tuT6B47JfttQFsPKB9SCXbGAuMBejYsWOtcnZsmcDfL8vg0hdnc83ETF67agjxsdHVLrd00y4+XPg9407tSovEOO79x7e8Mec7LhnSqVY5jsSuwhI27ihkw/ZC1m/fw9rcPazfvof42Ch6tmlKr7ZN6dWuKT3bNMHMQp5PROon34uFmSUB7wG3OOfyK7TfReBU1aRgbMc5NwGYAJCRkVHrP+sHd27BYyP7ceMbC7jtnUX876gBREUd+pfqg5+uoHlCLNee3JWkRjFMX5nDvf/4liGdW9KtVVJto9RIflEJr8/awMeLN/NdXiEFRaU/+bx10zjSWyZSUFTK67M2UFxaDsAJ3Vry+EX9adUk3td8IhIZfC0WZhZLoFBMcs69X6H9cuAs4DT34/maTUCHCouneW0cot0XZ/Vtx/c793L/JytIa96YP5x5dJXzfrk6ly9Xb+O/f3U0TeNjAXjkwn4Mf2IGt7y1gHeuPZ7Gjao/OjlcOflFvPj1OibN+o7dxaUMTm/BuQPak5bcmA7JCXRokUDnlEQS4378Jy4tK2f99kJmrMrloc9WcOaTX/HERf35WfeUoOcTkchifp1bt8A5jleBPOfcLRXahwOPASc753IrtPcG3iDQR9EOmAZ0B4xAR/hpBIrEXODXzrllVW07IyPDHemos845xk9ZxsSZGxiZkcbdZ/c56Jd+ebnj7Ge+YseeEv79u5OJi/nx88+WbeHa1+YRZdCpZSLdWiVxVOsk+ndIZlCnZFokNvph3l17S1i+OZ912/aweVcRW3cVsTm/iBYJsRzbuQWD01vQrVUS2Tv2Mm35VqatyGHW2u2UlTt+1bcd157UhT7tmx3W97dqawHjJs0nK3c3407pxo0/73bQKbdVWwu4+6NlLN9cwHFdWnJi9xROOiqVds0b12KPikhdZ2bznHMZlX7mY7H4GfAlsAQo95r/CDwFxAHbvbZZzrnrvGXuItCPUUrgtNU/vfYzgSeAaOAl59xfDrXtYBQLgLJyx2NTV/Ls9DV0b5XEM78eSPfWTQAoKCph0uzv+Os/V/DYyH6cNzDtoOW/XJ3L3PU7yMopYPXW3azbtofS8sD+7pqaSMcWCazO2U32jr0/LGMGqUlxtG4az+ZdRT9cWZXYKJo9+8oA6JKayC+Obs2vB3ckPSWx1t/f3n1ljJ+ylLczs2nWOJYLB6VxydBOpDaJ48l/reLlr9eTGBfDyUelMmddHlvyiwLbT0lkaNeWHNelJUO7tCS1SVytM4hI3RGWYhFOwSoW+81Ylcutby2kcF8Zlw7tyNJN+cxdn0dpuWNgx+a8c93xRFfTrwFQVFLG4uxdZG7IY976HWTv2Ev31kn0ahfoeO7WKonWTeOJjQ5cpOacY8P2Quasz2PRxp10TknktKNb0/kICkRlZq3dzmuzNvDZ0i2UljuaxMdQUFTKqGM7cMfwnrRIbIRzjtU5u5mxKpdv1mxnzro8dhcH+keO69KSy09I5xdHt67RfhCRuknFIghy8ou4+c2FzFy7nZ5tmnBKj1ac2iOVgZ2Sf/jlXt/l5Bfx5tyNrNiSzzUndmFAx+Qq5y0tK2fJpl18uXobb83dyKade0lLbszooZ247Lh0X/ppRMRfKhZB4pyjoLj0h45sCSgtK+dfy7fy8tfrmb0uj44tEnjgvGM4oZs6zkXqk0MVi8j4kzhEzEyFohIx0VEM79OWt649jsnXDCXK4JIXZnPHu4vYVVgS7ngiEgQqFhJUx3Vtyae3nMR1J3flvfmbOO2x/zBp9gZKysqrX1hE6iwVCwm6+Nho7jyjJx+OO4H0lgnc9cFShj0+g0+WbNYwKCL1lPosxFfOOaYtz+HBT1ewOmc3XVMTGdy5Bf3SmtOvQ3OOat1EV1CJ1BHq4JawKyt3vD8/mymLvmfRxp3ke8OStG/emGtP7sLIjA41GodLRPyjYiF1inOO9dsLmb9hB2/M+Y55G3aQktSIK3/WmV8c3Zq05MYkNArJGJciUoGKhdRZzjnmrMvjmelrmLHqh9FfaJnYiA4tEujfoTnHprfg2M7JGvRQxGcqFlIvrN5awPItBWzMKyR7RyHrtu1h0cZd7C0JDHPSoUVjOrVIpF3zeNo1b0z/Ds05pUerMKcWiRyHKhY61pc6o3vrJj+MvbVfSVk5SzftYs66PBZn72LTzr1MX5lLTkFgzKzLj0/nrl8dHTF30YvUVSoWUqfFRkcxoGPyQUOPFJWU8chnK3nhq3Us35zPs5cMpGWSBjQU8Yv+HJN6KT42mv8+qxePjezHgo07Ofvpr1n2/a5wxxKJWCoWUq+dNzCNd687jnLnGP3iHLJ3FIY7kkhEUrGQeq9vWnNev3oIJWXljJ04j8J9pdUvJCKHRcVCIkLX1CSeungAy7fkc/s7izWsiEiQqVhIxDi1Ryt+P7wnHy/ZzLPT14Q7jkhE0dVQElGuPakLyzfn88jnKykpK+e8AWl0bJkQ7lgi9Z6KhUQUM+PB8/uys7CEJ/61mif+tZq+ac04o09burdKIqVJHKlN4khNiqNRjA6sRWpKxUIiTnxsNK9eOZiNeYX8c+lmPl68mQc/XfGTeRIbRfO30YM4sXtqmFKK1C8a7kMahJyCIjbvLCK3oJhtu4t55Zv1rN++h1evGMyQLi3DHU+kTgjLY1XNrIOZfWFm35rZMjO72WtvYWZTzWy19zXZazcze8rMssxssZkNrLCuMd78q81sjF+ZJXK1ahJPvw7N+UWv1owa3JHXrx5C++aNufKVuSz4bke444nUeX6etC0FbnPO9QKGAuPMrBdwJzDNOdcdmOa9BzgD6O69xgLPQaC4AOOBIcBgYPz+AiNSWylJcbxxzVBSmsQx5qU5LN2ku79FDsW3Pgvn3GZgszddYGbLgfbACOAUb7ZXgenA7732iS5wXmyWmTU3s7bevFOdc3kAZjYVGA5M9iu7NAytm8Yz6eohjPzbTM76369o1jiWts0CI9pmpCdz2XHpJMWpW08EQtTBbWbpwABgNtDaKyQAW4DW3nR7YGOFxbK9tqraD9zGWAJHJHTs2DF44SWipSUn8O71x/Phwu/5fudeNu/aS/aOvTz0aQ4vfLmOG07pyqVDO+kpftLg+V4szCwJeA+4xTmXb/bj85adc87MgtLD7pybAEyAQAd3MNYpDUO75o25/pSuP2lb8N0OHv18Ffd9vJwXvlzHuJ93Y2RGGnExKhrSMPl6obmZxRIoFJOcc+97zVu900t4X3O89k1AhwqLp3ltVbWL+GZAx2Rev3oIk68ZSvvkxvzP/y3l1IenM2n2BvaVloc7nkjI+Xk1lAEvAsudc49V+GgKsP+KpjHAhxXaL/OuihoK7PJOV30GDDOzZK9je5jXJuK747q25N3rjuPVKwfTqmk8d32wlFMfmc4D/1zOV6u3UeQ9xU8k0vl2n4WZ/Qz4ElgC7P9T7I8E+i3eBjoCG4CRzrk8r7g8TaDzuhC4wjmX6a3rSm9ZgL84514+1LZ1n4X4wTnH9FW5TPjPWjI35FFS5mgUE8VxXVoy/r960SU1KdwRRY6InsEtEmR7ikuZsy6Pr7K28f78bErLHI9f1J9f9Gpd/cIidVRYbsoTiWSJcTGc2rMV/3NWLz76zc/olJLA1RMzeXzqKsrLI+8PMBEVC5EjlJacwLvXHc/5A9N4ctpqrpmYSUFRSbhjiQSVioVIEMTHRvPIhX25Z0Rvpq/K5bxnv2HD9j3hjiUSNCoWIkFiZlx2XDqvXTWY3N3FjHjma75Zsy3csUSCQsVCJMiO75rCh+NOICUpjstenMPbczdWv5BIHadiIeKDTi0Tef+G4zmua0vufH8xX63WEYbUbyoWIj5pGh/L3y4dRLdWSdw4eT4b8wrDHUmk1lQsRHyUGBfDhNEZlJc7xr42j737dMe31E8qFiI+S09J5MmLB7BiSz6/f28xkXgjrEQ+FQuREDi1Ryt+N6wHUxZ9z9/+szbccUQOm57sIhIiN5zSlRVbCnjw0xWkNonjgkFp4Y4kUmMqFiIhYmY8cmFfduzZx+/fW0xyQiynHa2xpKR+0GkokRCKi4nmb6MH0btdU8a9MZ/M9XnhjiRSIyoWIiGWFBfDy5cfS7tmjbnylbms2loQ7kgi1VKxEAmDlklxTLxqMPGx0Vz16lzy9uwLdySRQ1KxEAmTtOQEJlyWwdb8Yq5/fZ4e1yp1moqFSBj179Cchy/oy+x1eYyfskz3YEidpauhRMJsRP/2rNxSwLPT19CzTRPGHJ8e7kgiB9GRhUgd8LthPfhlr9bc849vmbEqN9xxRA6iYiFSB0RFGY9f1J/urZIY98Z8snJ2hzuSyE+oWIjUEUlxMbwwJoO4mCiuenUuO3SFlNQhvhULM3vJzHLMbGmFtv5mNsvMFppZppkN9trNzJ4ysywzW2xmAyssM8bMVnuvMX7lFakL0pITeH50Bpt3FXGdrpCSOsTPI4tXgOEHtD0E3O2c6w/8yXsPcAbQ3XuNBZ4DMLMWwHhgCDAYGG9myT5mFgm7QZ2Seej8wBVSf/pwafULiISAb8XCOTcDOHAsAwc09aabAd970yOAiS5gFtDczNoCpwNTnXN5zrkdwFQOLkAiEeecAe0Zd2pX3py7kffnZ4c7jkjIL529BfjMzB4hUKiO99rbAxUfVJzttVXVfhAzG0vgqISOHTsGN7VIGPz2lz2Yu34H//N/SxnQMZnOKYnhjiQNWKg7uK8HbnXOdQBuBV4M1oqdcxOccxnOuYzU1NRgrVYkbKKjjCdH9Sc2JoqbJi9Q/4WEVaiLxRjgfW/6HQL9EACbgA4V5kvz2qpqF2kQ2jZrzIPn92XJpl088vnKcMeRBizUxeJ74GRv+ufAam96CnCZd1XUUGCXc24z8BkwzMySvY7tYV6bSINxeu82jB7aiQkz1vIf3bAnYeLnpbOTgZlADzPLNrOrgGuAR81sEXA/Xh8D8AmwFsgC/g7cAOCcywPuBeZ6r3u8NpEG5a5fHU2P1k249a2FbNq5N9xxpAGySBy4LCMjw2VmZoY7hkhQrcndzTlPf016SiLvXHcc8bHR4Y4kEcbM5jnnMir7THdwi9QTXVOTeOyi/izZtIu7PliqEWolpFQsROqRX/Zqzc2ndee9+dm8NmtDuONIA6JiIVLP3Hxad07r2Yp7PvqWuXqGt4SIioVIPRMVZTx2UX86tEjghknzySkoCnckaQBULETqoWaNY3nu0oEUFJVw8+SFlJbphj3xl4qFSD3Vs01T7jvnGGau3c7j/1oV7jgS4VQsROqxCwalcfHgDjzzxRqmLd8a7jgSwVQsROq58f/Vm97tmnLrWwvZmFcY7jgSoVQsROq5+NhonrtkEABjX5tH4b7SMCeSSKRiIRIBOrZM4KmLB7BySz63v7NYN+xJ0KlYiESIU3q04o7hPfl4yWaenb4m3HEkwqhYiESQa0/qwtn92vHI5yv59wp1eEvwqFiIRBAz48Hz+9KrbVNunryQddv2hDuSRAgVC5EI07hRNM+PHkRUlPGbyfMpLi0LdySJACoWIhEoLTmBhy7oy9JN+Tz0qZ6wJ0dOxUIkQp3euw2XHdeJF79ap/4LOWIqFiIR7I9nHk3PNk343TuL2ZqvAQel9lQsRCJYfGw0T/96IHv3lXHLmwspK9f9F1I7KhYiEa5bqyTuHtGbmWu3M2HG2nDHkXpKxUKkAbhwUBpn9GnDu29PZ/uYq6FpU4iKCny94QZYo5v45NBiwh1ARPxnZjyUuImYF8YRU1YK5d7ltAUF8MIL8Oqr8O67cMYZ4Q0qdZZvRxZm9pKZ5ZjZ0gPaf2NmK8xsmZk9VKH9D2aWZWYrzez0Cu3DvbYsM7vTr7wiEW3NGppcejGNS4qJLT/gvouSEigshAsu0BGGVMnP01CvAMMrNpjZqcAIoJ9zrjfwiNfeCxgF9PaWedbMos0sGngGOAPoBVzszSsih+PRRwNF4VBKSuDxx0OTR+qdGhULM+tqZnHe9ClmdpOZNT/UMs65GcCBT5O/Hvirc67YmyfHax8BvOmcK3bOrQOygMHeK8s5t9Y5tw9405tXRA7H66/XrFi89lpo8ki9U9Mji/eAMjPrBkwAOgBv1GJ7RwEnmtlsM/uPmR3rtbcHNlaYL9trq6r9IGY21swyzSwzNze3FtFEItju3cGdTxqcmhaLcudcKXAu8L/OuduBtrXYXgzQAhgK3A68bWZWi/UcxDk3wTmX4ZzLSE1NDcYqRSJHUlJw55MGp6bFosTMLgbGAP/w2mJrsb1s4H0XMAcoB1KATQSOVvZL89qqaheRw3HppRBbzY9sbCyMHh2aPFLv1LRYXAEcB/zFObfOzDoDtTm5+X/AqQBmdhTQCNgGTAFGmVmct+7uwBxgLtDdzDqbWSMCneBTarFdkYbtttuqLRYuNhZuvTVEgaS+qVGxcM5965y7yTk32cySgSbOuQcPtYyZTQZmAj3MLNvMrgJeArp4l9O+CYzxjjKWAW8D3wKfAuOcc2Xeqa8bgc+A5cDb3rwicji6dg3cR5GQcFDRKI+JoTA2jpdveTgwn0glrCbP6jWz6cDZBPoc5gE5wNfOud/6mq6WMjIyXGZmZrhjiNQ9a9YELo997bVAZ3ZSEowezYSBZ3P/qhKeu2QgZxxTm+5IiQRmNs85l1HpZzUsFguccwPM7Gqgg3NuvJktds71DXbYYFCxEDk8JWXlXPDcN6zfXsg/bz6Rds0bhzuShMGhikVN+yxizKwtMJIfO7hFJELERkfxxKgBlJSV89u3NTqtHKymxeIeAv0Ga5xzc82sC7Dav1giEmqdUxL583/1ZtbaPI1OKwepaQf3O865vs656733a51z5/sbTURC7cKMNM48pg2Pfr6Sxdk7wx1H6pCaDveRZmYfeAMD5pjZe2aW5nc4EQktM+OBc/uS2iSOmyYvYE9xabgjSR1R09NQLxO4v6Gd9/rIaxORCNMsIZYnLurPhrxC/jxFV6pLQE2LRapz7mXnXKn3egXQmBoiEWpIl5bceGo33pmXzUeLvg93HKkDalostpvZpfuHDTezS4HtfgYTkfC66bTuDOjYnD9+sITsHYXhjiNhVtNicSWBy2a3AJuBC4DLfcokInVAbHQUT140AOfgljcXUlpWHu5IEkY1vRpqg3PubOdcqnOulXPuHEBXQ4lEuI4tE7jvnD5kbtjBk9N0tXxDdiRPyquTQ32ISHCdM6A9IzPSePqLLGas0rNiGqojKRZBeQ6FiNR9d5/dh+6tkrj1rYVszS8KdxwJgyMpFhoPQKSBaNwommcvGUjhvjJumrxA/RcN0CGLhZkVmFl+Ja8CAvdbiEgD0a1VE+47pw+z1+Wp/6IBijnUh865JqEKIiJ13/mD0pi9bjtPf5HF4M4tOLG7brdqKI7kNJSINEB3n92HbqmB/oucAvVfNBQqFiJyWBo3iuaZSwayu7iUW97UcOYNhYqFiBy2o1o34Z4RffhmzXae+SIr3HEkBFQsRKRWLhyUxrkD2vPEv1Yxa61G/4l0KhYiUitmxn3n9CG9ZSI3TV5AbkFxuCOJj1QsRKTWEuNiePbSgeQXlXDT5AXqv4hgvhULM3vJe1DS0ko+u83MnJmleO/NzJ4ysywzW2xmAyvMO8bMVnuvMX7lFZHa6dmmKfedcwwz127n8amrwh1HfOLnkcUrwPADG82sAzAM+K5C8xlAd+81FnjOm7cFMB4YAgwGxptZso+ZRaQWLhiUxkUZHXj6iyz+vWJruOOID3wrFs65GUBeJR89DtzBT4cLGQFMdAGzgOZm1hY4HZjqnMtzzu0AplJJARKR8Lt7RG+ObtuUW99apOdfRKCQ9lmY2Qhgk3Nu0QEftQc2Vnif7bVV1V7ZuseaWaaZZebmamRMkVCLj43muUsGUl7uGDdpPsWlZeGOJEEUsmJhZgnAH4E/+bF+59wE51yGcy4jNVVDEIiEQ3pKIg9f2I9F2bv4y8fLwx1HgiiURxZdgc7AIjNbD6QB882sDbAJ6FBh3jSvrap2EamjhvdpwzUndmbizA1M0fO7I0bIioVzbon3lL1051w6gVNKA51zW4ApwGXeVVFDgV3Ouc3AZ8AwM0v2OraHeW0iUofdMbwnGZ2SufO9xWTlFIQ7jgSBn5fOTgZmAj3MLNvMrjrE7J8Aa4Es4O/ADQDOuTzgXmCu97rHaxOROiw2Ooqnfz2QxrHRXP/6fAr3lYY7khwhcy7ybqLJyMhwmZmZ4Y4h0uB9nbWN0S/O5ld92/HUqP6Y6QGbdZmZzXPOZVT2me7gFhHfnNAthd+d3oOPFn3Pi1+tC3ccOQIqFiLiq+tP7sqZx7Th/k+W803WtnDHkVpSsRARX5kZD1/Qj26tkhj3xnzdsFdPqViIiO8S42J4fnQGpeWOa1+bx959umGvvlGxEJGQ6JySyJOj+vPt5nzu+mAJkXhxTSRTsRCRkPl5z9bcctpRvL9gE6/N2hDuOHIYVCxEJKR+8/NunNazFfd89C2Z63XbVH2hYiEiIRUVZTx2UX/Skhtz/aT55OQXhTuS1ICKhYiEXLPGsTw/OoPdRaVcP2k++0rLwx1JqqFiISJh0aNNEx6+sC/zNuxg/JRl6vCu42LCHUBEGq6z+rZj2ff5PDd9Db3aNWX00E7hjiRV0JGFiITV74b14Oc9W3H3lGXMWrs93HGkCioWIhJW0VHGE6P607FlAjdM0h3edZWKhYiEXdP4WP5+WQYlZeVcM3GehjSvg1QsRKRO6JqaxFMXD2DFlnxuf2exOrzrGBULEakzTu3RijuH9+TjJZt5+t9Z4Y4jFehqKBGpU8ae1IUVWwp4dOoqerRpwrDebcIdSdCRhYjUMWbGA+cdQ7+0Ztz61kJWbtEzvOsCFQsRqXPiY6N5fnQGiXExXD1xLnl79oU7UoOnYiEidVKbZvE8P3oQW/OLuWHSPErKNCRIOKlYiEidNaBjMg+efwyz1ubx5ynLwh2nQfOtWJjZS2aWY2ZLK7Q9bGYrzGyxmX1gZs0rfPYHM8sys5VmdnqF9uFeW5aZ3elXXhGpm84dkMa1J3dh0uzveG3m+nDHabD8PLJ4BRh+QNtUoI9zri+wCvgDgJn1AkYBvb1lnjWzaDOLBp4BzgB6ARd784pIA3LH6T35ec9W/Pmjb/k6a1u44zRIvhUL59wMIO+Ats+dc/tvzZwFpHnTI4A3nXPFzrl1QBYw2HtlOefWOuf2AW9684pIAxIdZTw5qj9dUxO57vV5rN6qK6RCLZx9FlcC//Sm2wMbK3yW7bVV1X4QMxtrZplmlpmbm+tDXBEJpybxsbx0+bHExURzxStzyS0oDnekBiUsxcLM7gJKgUnBWqdzboJzLsM5l5Gamhqs1YpIHZKWnMCLYzLYtruYqydmsndfWbgjNRghLxZmdjlwFnCJ+3Hwl01AhwqzpXltVbWLSAPVr0Nznhw1gMXZO7n1rYWUl2sMqVAIabEws+HAHcDZzrmK4xBPAUaZWZyZdQa6A3OAuUB3M+tsZo0IdIJPCWVmEal7Tu/dhrvOPJpPl23h3o+/1aCDIeDb2FBmNhk4BUgxs2xgPIGrn+KAqWYGMMs5d51zbpmZvQ18S+D01DjnXJm3nhuBz4Bo4CXnnC62FhGu+llnvt9ZxEtfr6NN03iuPblruCNFNIvEipyRkeEyMzPDHUNEfFZe7rjpzQX8Y/FmHhvZj/MGplW/kFTJzOY55zIq+0yjzopIvRUVZTw6sh/bd+/jjncXk5IUx0lH6QIXP2i4DxGp1+Jionn+skF0b92E616fx+LsneGOFJFULESk3msaH8urVxxLi8RGXPHyXNZt2xPuSBFHxUJEIkKrpvFMvHIwDrjspdnkFBSFO1JEUbEQkYjRJTWJly4/lu279zHmpbnkF5WEO1LEULEQkYjSv0Nznrt0EKu3FjB2YiZFJbrLOxhULEQk4px8VCqPjuzH7HV53DBpvh6cFAQqFiISkUb0b8995/Th3ytyuPWthZQ1gGFB3pzj3zM/dJ+FiESsS4Z0Yk9xKfd/soKERtH89by+REVZuGP5ZuLMDSQnxjL6uPSgr1vFQkQi2tiTurK7uIynpq0moVEM4/+rF95wQxElv6iE5Vvyufm07r6sX8VCRCLerb/oTmFxKS98tY642CjuHN4z4grG/A07cA6OTW/hy/pVLEQk4pkZd/3qaIpKy3j+P2uJj4nm1l8eFe5YQZW5fgfRUUb/Ds19Wb+KhYg0CGbGPWf3YV9pOU9OW01cbBQ3nNIt3LGCZo7p1ZcAAA53SURBVM76PPq0a0pinD+/1lUsRKTBiIoyHjivL8Wl5Tz06UoaRUdx9Yldwh3riBWXlrFo404uHdrJt22oWIhIgxIdZTx6YT9Kyxz3fbwc5+Cak+p3wVi6KZ/i0nKOTU/2bRsqFiLS4MRER/HEqP4A/OWT5TgcY0+qvw9Pmrs+D4AMnzq3QcVCRBqo2OgonhzVHwzu/2QFzlFvn7aXuT6PLimJpCTF+bYNFQsRabBioqN48qL+GPDAP1dQWu4Yd2r96vQuL3dkbtjBsF6tfd2OioWINGgx0VE8cVF/YqKMhz9byZ7iUm4/vUe9uQ8jK3c3OwtLfLu/Yj8VCxFp8GKio3hsZH8aN4rh2elrKNxXxp/O6lUvhgbZ31+hYiEiEgJRUcb95/YhsVE0L3y1jj3FpTxw3jHERNft8VYz1+8gJSmOTi0TfN2Ob3vBzF4ysxwzW1qhrYWZTTWz1d7XZK/dzOwpM8sys8VmNrDCMmO8+Veb2Ri/8oqI7L/T+6bTuvPOvGyunzS/zj8PY+76PAZ3Tvb9tJmfJfMVYPgBbXcC05xz3YFp3nuAM4Du3mss8BwEigswHhgCDAbG7y8wIiJ+MDN++8ujuPvs3vxr+VYueWE2Owv3hTtWpTbv2kv2jr1kdPL3FBT4WCycczOAvAOaRwCvetOvAudUaJ/oAmYBzc2sLXA6MNU5l+ec2wFM5eACJCISdGOOT+eZXw9kSfYuLvjbTDbt3BvuSAeZu34HAIM71+NiUYXWzrnN3vQWYP+1Xu2BjRXmy/baqmoXEfHdmce0ZeJVg9maX8S5z3zN4uyd4Y70EzNW5dIkPoaebZr4vq2w9dw45xwQtEdXmdlYM8s0s8zc3NxgrVZEGrihXVry7nXHExsdxcjnZ/LJks3VLxQCRSVlfLp0C8N7twlJJ3yoi8VW7/QS3tccr30T0KHCfGleW1XtB3HOTXDOZTjnMlJTU4MeXEQarh5tmvDhjSfQu10zbpg0n/+dtprA37vh88WKHHYXl3LOgNCcbAl1sZgC7L+iaQzwYYX2y7yrooYCu7zTVZ8Bw8ws2evYHua1iYiEVEpSHJOuHsK5A9rz6NRV3PTmQgr3lYYtz4cLvye1SRxDu7QMyfZ8u8/CzCYDpwApZpZN4KqmvwJvm9lVwAZgpDf7J8CZQBZQCFwB4JzLM7N7gbnefPc45w7sNBcRCYn42GgeG9mPbq2SeOTzlazeWsDzowfRqWViSHPs2lvCv1fmcOmQTkSH6MZBC/ehlB8yMjJcZmZmuGOISAT7z6pcbpq8AOccT108gFN6tArZtt/O3Mgd7y7mw3En0C+IT8Yzs3nOuYzKPqvbtyaKiNRRJx+Vykc3/oz2yQlc8cpcHvt8JSVl5SHZ9ocLN5HeMoG+ac1Csj1QsRARqbWOLRN4//rjOW9AGk/9O4sL/zaT9dv2+LrNnPwivlmznbP7tw/pYIcqFiIiR6Bxo2geHdmPp389gLW5uznzqS95a+53vl0t9dHizTgHZ/dr58v6q6JiISISBGf1bcdnt55E/w7N+f17S7jspTms8+EoY8rCTfRp35RurZKCvu5DUbEQEQmSts0a8/pVQ7j77N4s/G4npz8xg8enrgraYIRZObtZlL2LEf1CP5CFioWISBBFRRljjk9n2m0nM7x3G56ctprTn5jBO5kbj6gDfNfeEsZNmk+TuBhG9A/tKShQsRAR8UWrpvE8dfEAXr9qCAmNYrj93cWc8vB0Xv1m/WEfaRSVlHHNxEzWbtvN86MH0appvE+pq6b7LEREfOacY/rKXJ7+Iot5G3aQnBDLBYPSGDW4I11TD933UFbu+M3k+XyyZAtPXTzA147tQ91noSfliYj4zMw4tWcrTumRyux1ebz6zXpe/no9f/9yHYM7t+CMPm3o2aYpPds0ITmxEQC7CkvYuKOQSbO/45MlW/jvXx0d8iugKlKxEBEJETNjaJeWDO3SktyCYt6dl81bc7/j7o++/WGelKQ49pWWkV/047hT15zYmatP7BKOyD/QaSgRkTByzpFTUMyKLQWs2lLAqq0FxMdG07FFAh1aNCY9JZEerZuE5AY8nYYSEamjzIzWTeNp3TSek4+qu49X0NVQIiJSLRULERGploqFiIhUS8VCRESqpWIhIiLVUrEQEZFqqViIiEi1VCxERKRaEXkHt5nlAhsq+agZsKuatorvK5ve/zUF2FaLeJVlqOk8h8paVd6qpv3KX9PslbX5ve8Pla+6z6vLXxf2fU1yVtUWKfu+4vu6sO8Pla+y94fa9+D/z20n51zldwY65xrMC5hQXVvF95VNV/iaGawMNZ3nUFlrkj0U+WuaPRz73s/8dWHf13Q/R/K+ryxzOPd9dfv6cPa9n/lr8v01tNNQH9Wg7aNqpitbx5FmqOk8h8p64PuaTNdGdcvXNHtlbX7v+5qso7b568K+r2qehrTvK76vC/u+svb6tO9/EJGnoULBzDJdFQNu1Qf1OX99zg71O399zg7KfyQa2pFFME0Id4AjVJ/z1+fsUL/z1+fsoPy1piMLERGplo4sRESkWioWIiJSLRULwMxeMrMcM1tai2UHmdkSM8sys6eswuOszOw3ZrbCzJaZ2UPBTf3DNoKe3cz+bGabzGyh9zoz+Ml/yODLvvc+v83MnJmlBC/xQRn82P/3mtlib99/bma+PHjZp+wPe//nF5vZB2bWPPjJf8jgR/4LvZ/XcjMLekfykWSuYn1jzGy19xpTof2QPxu1UttrpiPpBZwEDASW1mLZOcBQwIB/Amd47acC/wLivPet6lH2PwO/q6/73vusA/AZgZszU+pTfqBphXluAv5Wj7IPA2K86QeBB+vZvj8a6AFMBzLqSmYvT/oBbS2Atd7XZG86+VDf35G8dGQBOOdmAHkV28ysq5l9ambzzOxLM+t54HJm1pbAD/YsF/gXmgic4318PfBX51yxt42cepQ9ZHzM/zhwB+DrFRx+5HfO5VeYNRGfvgefsn/unCv1Zp0FpPmR3cf8y51zK+ta5iqcDkx1zuU553YAU4Hhfv1sq1hUbQLwG+fcIOB3wLOVzNMeyK7wPttrAzgKONHMZpvZf8zsWF/T/tSRZge40TuV8JKZJfsXtVJHlN/MRgCbnHOL/A5ahSPe/2b2FzPbCFwC/MnHrAcKxv+d/a4k8FdtKAUzf6jUJHNl2gMbK7zf/3348v3FHOkKIpGZJQHHA+9UONUXd5iriSFweDgUOBZ428y6eJXeN0HK/hxwL4G/aO8FHiXwg++7I81vZgnAHwmcDgm5IO1/nHN3AXeZ2R+AG4HxQQtZhWBl99Z1F1AKTApOuhptM2j5Q+VQmc3sCuBmr60b8ImZ7QPWOefODXVWFYvKRQE7nXP9KzaaWTQwz3s7hcAv1YqH2WnAJm86G3jfKw5zzKycwCBguX4GJwjZnXNbKyz3d+AffgY+wJHm7wp0BhZ5P3xpwHwzG+yc2+JzdgjO/52KJgGfEIJiQZCym9nlwFnAaX7/cXSAYO/7UKg0M4Bz7mXgZQAzmw5c7pxbX2GWTcApFd6nEejb2IQf31+wO3Dq6wtIp0KnE/ANcKE3bUC/KpY7sCPpTK/9OuAeb/ooAoeLVk+yt60wz63Am/Vp3x8wz3p87OD2af93rzDPb4B361H24cC3QKqf+9zv/zv41MFd28xU3cG9jkDndrI33aIm31+tcofiH7Suv4DJwGaghMARwVUE/jr9FFjk/ef/UxXLZgBLgTXA0/x4V3wj4HXvs/nAz+tR9teAJcBiAn+JtfUju1/5D5hnPf5eDeXH/n/Pa19MYIC39vUoexaBP4wWei9fruTyMf+53rqKga3AZ3UhM5UUC6/9Sm+fZwFXHM7PxuG+NNyHiIhUS1dDiYhItVQsRESkWioWIiJSLRULERGploqFiIhUS8VCGgwz2x3i7b1gZr2CtK4yC4xCu9TMPqpuNFcza25mNwRj2yKgJ+VJA2Jmu51zSUFcX4z7cdA8X1XMbmavAqucc385xPzpwD+cc31CkU8in44spEEzs1Qze8/M5nqvE7z2wWY208wWmNk3ZtbDa7/czKaY2b+BaWZ2iplNN7N3LfAch0n7nx3gtWd407u9wQEXmdksM2vttXf13i8xs/tqePQzkx8HTUwys2lmNt9bxwhvnr8CXb2jkYe9eW/3vsfFZnZ3EHejNAAqFtLQPQk87pw7FjgfeMFrXwGc6JwbQGDU1/srLDMQuMA5d7L3fgBwC9AL6AKcUMl2EoFZzrl+wAzgmgrbf9I5dww/HSm0Ut44R6cRuLMeoAg41zk3kMAzVB71itWdwBrnXH/n3O1mNgzoDgwG+gODzOyk6rYnsp8GEpSG7hdArwojfjb1RgJtBrxqZt0JjL4bW2GZqc65is8kmOOcywYws4UExv756oDt7OPHARnnAb/0po/jx2cNvAE8UkXOxt662wPLCTy7AAJj/9zv/eIv9z5vXcnyw7zXAu99EoHiMaOK7Yn8hIqFNHRRwFDnXFHFRjN7GvjCOXeud/5/eoWP9xywjuIK02VU/nNV4n7sIKxqnkPZ65zr7w3B/hkwDniKwPMuUoFBzrkSM1sPxFeyvAEPOOeeP8ztigA6DSXyOYGRXQEws/1DRTfjx2GdL/dx+7MInP4CGFXdzM65QgKPWr3NzGII5MzxCsWpQCdv1gKgSYVFPwOu9I6aMLP2ZtYqSN+DNAAqFtKQJJhZdoXXbwn84s3wOn2/JTC0PMBDwANmtgB/j8BvAX5rZosJPOBmV3ULOOcWEBiR9mICz7vIMLMlwGUE+lpwzm0HvvYutX3YOfc5gdNcM7153+WnxUTkkHTprEgYeaeV9jrnnJmNAi52zo2objmRUFOfhUh4DQKe9q5g2kmIHl8rcrh0ZCEiItVSn4WIiFRLxUJERKqlYiEiItVSsRARkWqpWIiISLX+H3o4tpqO2CYmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(suggestion=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>bleu</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>35.909977</td>\n",
       "      <td>33.639030</td>\n",
       "      <td>0.540811</td>\n",
       "      <td>0.395665</td>\n",
       "      <td>00:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>9.407427</td>\n",
       "      <td>6.137264</td>\n",
       "      <td>0.564513</td>\n",
       "      <td>0.392775</td>\n",
       "      <td>00:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>4.319383</td>\n",
       "      <td>3.770583</td>\n",
       "      <td>0.583217</td>\n",
       "      <td>0.403667</td>\n",
       "      <td>00:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.366669</td>\n",
       "      <td>3.444857</td>\n",
       "      <td>0.601924</td>\n",
       "      <td>0.409635</td>\n",
       "      <td>00:49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(4, 3e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('transformer-fastext')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 ULMFit with transformers\n",
    "\n",
    "I will use the embeddings from [ULMFit](https://arxiv.org/pdf/1801.06146.pdf) in english as the encoder. The decoder layer will be the same but it will also receive spanish embeddings from ULMFit. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### 2.2.1 English embeddings\n",
    "\n",
    "We will start with ULMFit's pre-trained model for english and fine-tuned it on our subtitles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "path = base_path/'lm_en'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_lm = (\n",
    "    TextList\n",
    "        .from_df(df, path=path, cols='en')\n",
    "        .split_by_rand_pct(0.1, seed=42)  # We randomly split for validation\n",
    "        .label_for_lm()  # We want to do a language model so we label accordingly\n",
    "        .databunch(bs=48, num_workers=12))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_lm.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_lm = load_data(path, bs=48)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SequentialRNN(\n",
       "  (0): AWD_LSTM(\n",
       "    (encoder): Embedding(25784, 400, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(25784, 400, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): WeightDropout(\n",
       "        (module): LSTM(400, 1152, batch_first=True)\n",
       "      )\n",
       "      (1): WeightDropout(\n",
       "        (module): LSTM(1152, 1152, batch_first=True)\n",
       "      )\n",
       "      (2): WeightDropout(\n",
       "        (module): LSTM(1152, 400, batch_first=True)\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "      (2): RNNDropout()\n",
       "    )\n",
       "  )\n",
       "  (1): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=25784, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, drop_mult=1.0)\n",
    "learn_lm.model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_lm.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Min numerical gradient: 3.98E-02\n",
      "Min loss divided by 10: 3.63E-02\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxc1X338c9PGu2bZUveDd7ABgIGW+wEcCFsKaEUhzY7ZHEdErL3gTZ90j7Jk7ZpHkhInGBcBxJC0iYx0IQUDKQJGINZZPCGbYzxKtuytdjal5H0e/6YKywL2Ra27iya7/v1mpfu3Hvm3p/Ho/npnHPPOebuiIhI+spIdAAiIpJYSgQiImlOiUBEJM0pEYiIpDklAhGRNBdJdADvVllZmU+ePDnRYYiIpJRVq1bVunv5QMdSLhFMnjyZysrKRIchIpJSzGzHkY6paUhEJM0pEYiIpDklAhGRNKdEICKS5pQIRETSnBKBiEiaUyIQEUlzSgQiIingnj+8yXNv1oRy7lATgZmNMLOlZrbJzDaa2YUDlLnczFab2etm9myY8YiIpCJ35wd/fJOXttaHcv6wRxbfAyxz93lmlg3k9z1oZiOAHwPXuPtOMxsdcjwiIimnpbOb7h6nOC+cr+zQEoGZlQCXArcAuHsn0Nmv2IeBR9x9Z1Bmf1jxiIikqsa2KADFuVmhnD/MpqEpQA3wgJm9ZmZLzKygX5lTgVIze8bMVpnZxwc6kZnNN7NKM6usqQmnjUxEJFk1tscSQUle6iWCCDAbuNfdzwFagDsHKDMHeD9wNfC/zezU/idy98XuXuHuFeXlA06eJyIybDW0BjWCFEwEVUCVu78UPF9KLDH0L/Oku7e4ey2wHJgVYkwiIimnsb0LSMGmIXevBnaZ2Yxg1xXAhn7FfgtcYmYRM8sHzgc2hhWTiEgqeruPINU6iwO3A78I7hjaCtxqZgsA3H2Ru280s2XAWqAHWOLu60OOSUQkpfT2EYRVIwg1Ebj7aqCi3+5F/cp8F/humHGIiKSyxrZY01BRbjhf2RpZLCKS5BraohTmRIhkhvOVrUQgIpLkGtujFIdUGwAlAhGRpNfYFg3t1lFQIhARSXqN7UoEIiJpraGtK7Q7hkCJQEQk6cWahtRHICKStmKdxaoRiIikpZ4ep7mjS30EIiLpqqmjC/fwZh4FJQIRkaR2aC0C9RGIiKSlhrZwp6AGJQIRkaQW9oRzoEQgIpLUeiec0+2jIiJpqrePQJ3FIiJp6u2mISUCEZH01NgWxQwKs9U0JCKSlhrbuyjKiZCRYaFdI9REYGYjzGypmW0ys41mduERyp1rZl1mNi/MeEREUk1jW5SS/PCahSD8NYvvAZa5+7xg3eL8/gXMLBP4DvBUyLGIiKSchrZw5xmCEGsEZlYCXAr8BMDdO9394ABFbwceBvaHFYuISKoKe8I5CLdpaApQAzxgZq+Z2RIzK+hbwMwmADcC9x7tRGY238wqzayypqYmvIhFRJJMY1tXqGMIINxEEAFmA/e6+zlAC3BnvzLfB+5w956jncjdF7t7hbtXlJeXhxOtiEgSikeNIMw0UwVUuftLwfOlvDMRVAD/aWYAZcB1Ztbl7v8VYlwiIimjoS0a6mAyCDERuHu1me0ysxnu/gZwBbChX5kpvdtm9lPg90oCIiIx0e4eWju7Qx1MBuHfNXQ78IvgjqGtwK1mtgDA3ReFfG0RkZTW1B7MMxTiFNQQciJw99XEmn/6GjABuPstYcYiIpJqGuMwBTVoZLGISNLqnWco7D4CJQIRkSQVj0VpQIlARCRpvb0WQQoPKBMRkRNwaArq1B1QJiIiJ+DQwvWqEYiIpKWGtiiRDCM/OzPU6ygRiIgkqcb2KMV5WQSzL4RGiUBEJEk1tnWFPpgMlAhERJJWb40gbEoEIiJJqjEOi9KAEoGISNKKx8yjoEQgIpK0GtvDX5QGlAhERJKWmoZERNJYe7Sbjq4edRaLiKSrQ9NLKBGIiKSlQxPOpXgfgZmNMLOlZrbJzDaa2YX9jn/EzNaa2Toze8HMZoUZj4hIqohnjSDsVHMPsMzd5wXLVeb3O74NuMzdD5jZtcBi4PyQYxIRSXrxmnAOQkwEZlYCXArcAuDunUBn3zLu/kKfpy8CE8OKR0QklTQG6xWXpPjto1OAGuABM3vNzJaYWcFRyn8KeCLEeEREUka8VieDcBNBBJgN3Ovu5wAtwJ0DFTSzucQSwR1HOD7fzCrNrLKmpiaseEVEkkY8m4bCTARVQJW7vxQ8X0osMRzGzM4ClgA3uHvdQCdy98XuXuHuFeXl5aEFLCKSLBrbo2RHMsjNCnctAggxEbh7NbDLzGYEu64ANvQtY2YnAY8AH3P3zWHFIiKSahpa4zOqGMK/a+h24BfBHUNbgVvNbAGAuy8CvgGMAn4cLLzQ5e4VIcckIpL06lo6KSvMjsu1Qk0E7r4a6P/FvqjP8U8Dnw4zBhGRVFTX3MGoOCUCjSwWEUlC9S2djCrIicu1lAhERJJQXXOnagQiIumqPdpNU0cXZYWqEYiIpKX6ltgkDCMLVCMQEUlLdc2xRDBKiUBEJD3VtnQAMEpNQyIi6am3RhCvcQRKBCIiSaZeNQIRkfRW19xJTiSDguzw5xkCJQIRkaRT29zJqIJsgql3QqdEICKSZOpaOuLWLARKBCIiSSeeo4pBiUBEJOnUNXfEbZ4hUCIQEUkq7h7XKahBiUBEJKm0dHbT0dWjpiERkXRV1xyMIVDTkIhIeqoNRhWPHC41AjMbYWZLzWyTmW00swv7HTcz+4GZbTGztWb2jsXtRUTSSW+NoCyONYKw1yy+B1jm7vOCdYvz+x2/FjgleJwP3Bv8FBFJS3XBFNTDoo/AzEqAS4GfALh7p7sf7FfsBuBBj3kRGGFm48KKSUQk2fXWCOK1FgGE2zQ0BagBHjCz18xsiZkV9CszAdjV53lVsE9EJC3VtXRSlBMhNys+8wxBuIkgAswG7nX3c4AW4M7jOZGZzTezSjOrrKmpGcoYRUSSSrxHFUO4iaAKqHL3l4LnS4klhr52A5P6PJ8Y7DuMuy929wp3rygvLw8lWBGRZFDX0hHXZiEIMRG4ezWwy8xmBLuuADb0K/Y74OPB3UMXAA3uvjesmEREkl2sRhC/O4Yg/LuGbgd+EdwxtBW41cwWALj7IuBx4DpgC9AK3BpyPCIiSa22uZNzThoR12uGmgjcfTVQ0W/3oj7HHfhcmDGIiKSKnh7nQGtnXEcVg0YWi4gkjYa2KN09Pqw6i0VE5F2oi/Naxb2UCEREkkTvPENlw+WuIREReXfqEjDhHCgRiIgkjbebhpKxs9jMpplZTrB9uZl9wczie3+TiMgwV9vciRmU5mfF9bqDrRE8DHSb2XRgMbHRwL8MLSoRkTRU19xBaX42kcz4NtYM9mo97t4F3Aj80N3/FtAsoSIiQ6i+pZNRce4ohsEngqiZfQj4BPD7YF986y4iIsNcIiacg8EngluBC4Fvu/s2M5sC/Dy8sERE0k9tS0fcO4phkFNMuPsG4AsAZlYKFLn7d8IMTEQk3SR1jcDMnjGzYjMbCbwK/LuZ3R1uaCIi6aOzq4eGtmhCagSDbRoqcfdG4C+JLS15PnBleGGJiKSXA63xX6u412ATQSRYS/hmDnUWi4jIEKlt7h1MlryJ4JvAk8Bb7v6KmU0F3gwvLBGR9FLd0A7A2JLcuF97sJ3FvwF+0+f5VuCmsIISEUk3u+pbAZhYmh/3aw+2s3iimT1qZvuDx8NmNjHs4ERE0kXVgTZyszIoS+I+ggeIrS88Png8Fuw7KjPbbmbrzGy1mVUOcLzEzB4zszVm9rqZaalKEUlLVQfamFiaj5nF/dqDTQTl7v6Au3cFj58C5YN87Vx3P9vd+y9ZCbFlKje4+yzgcuCuYH1jEZG0UnWwlYmleQm59mATQZ2ZfdTMMoPHR4G6Ibi+A0UWS4GFQD3QNQTnFRFJKbEaQXIngk8Su3W0GtgLzANuGcTrHHjKzFaZ2fwBji8ETgP2AOuAL7p7zyBjEhEZFpraoxxsjSakoxgGmQjcfYe7f8Ddy919tLv/BYO7a+gSd58NXAt8zswu7Xf8amA1sX6Hs4GFZlbc/yRmNt/MKs2ssqamZjAhi4ikjKoDbQBMSuZEcARfOVYBd98d/NwPPAqc16/IrcAjHrMF2AbMHOA8i929wt0ryssH2zUhIpIaehNBsjcNDeSoXdtmVmBmRb3bwFXA+n7FdgJXBGXGADOArScQk4hIyqk60DuGIDGJYFADyo7Aj3F8DPBocCtUBPiluy8zswUA7r4I+BbwUzNbRyyx3OHutScQk4hIyqk60EZeViYjEzC9BBwjEZhZEwN/4Rtw1NQVjD6eNcD+RX229xCrKYiIpK1d9a1MGpmXkDEEcIxE4O5F8QpERCRd9Q4mS5T4rpAsIiLvUHUgcYPJQIlARCShGtqiNLZ3KRGIiKSr3W/fOqqmIRGRtLQruHU0UYPJQIlARCShEj2YDJQIREQSqupAKwXZmYzIz0pYDEoEIiIJlMh1CHopEYiIJFDVgTYmjUxcsxAoEYiIJIy7U1XfmtA7hkCJQEQkYRrbumjqSOwYAjixSedSytMb9nHnw2vJz8mkIDtCfnYm00cXMm/OJM6dXHpY+5y70x7tIS87M4ERi8hwtyvBs472SptEMLY4l2veM5bWzm5aOrpo6ezi8XXV/LqyiillBdxw9njaOrtZW9XA+j0NNHd0Mb28kHNOGsHsk0o5c2IJ00cXkhNRchCRoVGVBIPJII0SwZkTSzhz4pmH7Wvt7OKJddX8unIX3//Dm2RnZjBzXBHXzxpPWWEO63c38PSGffy6sgqAzAxjalkBM8YWMWFEHqOLcxldlMP00YXMHFuU0F5/EUk9VUkwmAzSKBEMJD87wk1zJnLTnInUNXdQlJtFduTwbhN3Z3tdKxv2NLJxbyObqhtZW9XAUxv20dl1aHnlaeUF3HD2BD4wazyTywqGNM7Orh6ee7OGg61R2qLdtEe76epxIhlGhhmRTCM3K5OinAiFuRFyIpm0dHTR2B6bw6S7O9bMlZuVSX52hMKcCEW5EYpzs8jJyqC5o4vm9i6aO7oYV5LLlLICJTWROKg60EZRToTivMR+Fad1IuhrVGHOgPvNjCllBUwpK+D9Z417e7+709AWpbqxnVd3HOS3q3dz99ObufvpzRTnRhhVmMOogmzKCnOYUJrHpNI8Jo3MZ2xJLqMKchhZkP2OpNOfu/P4umr+7clN7KhrHdJ/79GMLc7lommjOG/KSEYX51Ccm0VxXhb52ZlEMjLIzDAiGUZRboRI5jsTZ1NHFx3RHgpyMsmNZJKRoaQiMpCqA61MKE3cOgS9lAiOk5kxIj+bEfnZzBxbzIfPP4k9B9tYtr6anfWt1DZ3UN/SyZv7m3hm837aoz3vOEdRboQzJ5Rw/pRRnD91JDPHFnGwNUpdSwd7DrbzkxXbWL3rIDPGFLH4Y3OYMbaIvKxMcrIyyco0unuc7h6nq8dp6+yO/WXf0UV7tJvCnAjFeVkU5UbINKO9q4e2zm5aO2N//Te2d9HUHqWjq+ftGkJ+doSttc288FYdz26u4ZHXdh/jPeDtZFeUG6GmqYN9jR20RbsPK1eQncnU8kJOH1fM6eOLOWV0IWVFsWRYmp9NphKFpKld9W1MGpnYZiEAcz/WipMncHKz7UAT0A10uXvFAGUuB74PZAG17n7Z0c5ZUVHhlZWVQx9siNyd2uZOdh1oZV9DO/WtndQ3d7KvKVab2FjdyED/DWOKc/jqVTO4afbEuH9Zujs761s50BqlsS1KY3uU1o5uut3p6u4h2u0cbItS09RBbXMHTe1RyotyGVOUw9iSXLIjGbR2dtPa2U1jW5Qt+5vZsLeR+pbOw65jBuWFOYwfkcf4EbmMKc4lOzMDMyPDYjW18yaP5LRxRe+ofYiksmh3D6d/Yxmffu9U7rhmZujXM7NVA30HQ3xqBHOPtA6xmY0Afgxc4+47zWx0HOKJOzOjvCiH8qKBm58aWqO8vL2e7bUtlBZkU1YY+yt7+uhCcrMSc5eSmXHyqAJOHjV053R39jd18Nb+ZupaOqlv6aSuuYPqxnb2NrSzqbqJ5Ztr6erpocdj5aPdsQxZmBOhYnIp08oLGVcSSxjlRTkUZEfIy84gJ5JJeVFOwt4vkXdra00L0W5n5tjELwSZ6KahDwOPuPtOAHffn+B4EqIkP4v3nT4m0WGEzswYUxz7Eh+sfY3tvLStnpe21lG5/QAvba1/R9NTrwyDyaNid3VNH11Ie7Sb2uZOaps76Ij2MLo4h3EluYwtyWPCiDxOHpXPpJH5FOYk+tdA0tGm6kYAZo4tTnAk4ScCB54yMwfuc/fF/Y6fCmSZ2TNAEXCPuz8YckySQsYU5/KBWeP5wKzxQKyW0NjeRXVDO7XNHbR1dtMW7aats5vdB9t4o7qJjXsbWfZ6NdmZGZQV5lBWlENOJOPt24E7ug7vrynJyyKSYfS40xM00UUyjMzg4U5wzMmJZHLelJFcPL2Mi6ePYlxJYgcCSeraVN1EVqYxtXxo7zI8HmEngkvcfXfQ5PO0mW1y9+X9rj8HuALIA1aa2YvuvrnvScxsPjAf4KSTTgo5ZElmZkZJXhYleVnM4MhV6mh3D5EMe8fdGO7OgdYouw+0sbO+lZ31rew52IbjZFjsdlx3p9uDjvhuxyw2hsTMaGiLsnxzDY8GHem9t9v2PsaV5DG6OIcxRbmMKcnRAEQ5ok17G5lWXkhWEvR9hZoI3H138HO/mT0KnAf0TQRVQJ27twAtZrYcmAVs7neexcBiiHUWhxmzDA9H+uUyM0YWZDOyIJszJ5Yc17l7epw39jWx4s1aNuxtZFttC4+t2UNje9dh5SIZxhnji5l9cilzTi7ljPElTCrNU6e3APBGdRPnTRmZ6DCAEBOBmRUAGe7eFGxfBXyzX7HfAgvNLAJkA+cD3wsrJpGhkJFhnDaumNPGHWrbdXcOtkbZ19TO/sYO9jW2s7W2hVd3HOA/Xt7JA89vByA7M4MpZQWcMqaQP5s5mitOG0NJXuIWJJHEaGiNsqehnZnjEt8/AOHWCMYAjwZV8wjwS3dfZmYLANx9kbtvNLNlwFqgB1ji7utDjEkkFGZGaUE2pQXZzBx7+LFod08wKr2Jt/Y3s2V/M69sr+f3a/eSlWlcPL2M684cx9VnjFVSSBNv7GsCYEYS3DEEIY8jCEMqjiMQ6a+nx1lddZBl66t5Yv1edtW3kZ2ZwWUzyrnh7PFcedoY3Qo7jD24cjvf+O3rvPh3VzC2ZPB30Z2IRI8jEJF+MjKM2SeVMvukUv7u2pmsqWrgd6v38Pu1e3h6wz5GFmTzofMm8dELTtadScPQpuomSvKyGFM88NiieFMiEEkwM+PsSSM4e9IIvv7+01j5Vh0PrtzOvc+8xaJnt3LNGWO5be40zhh/fJ3bknw27W1MqhmLlQhEkkhmhnHJKWVcckoZu+pbeejFHfzypZ3897q9XHnaGL5wxXTOmjgi0WHKCejpcd6obmLenImJDuVtuo9NJElNGpnP3113Givu/DO+fOWpvLytjg8sfJ6P3/8yyzfXkGr9exKz+2AbLZ3dSXPHECgRiCS9krwsvnjlKTx/55/xt1fPYOPeRj5+/8tc/f3l/OqVnYetiyHJb+Pe3qklkuOOIVAiEEkZRblZfG7udFbcMZe7PjiLzIwM7nh4Hdfes5xnN9ckOjwZpDeqY7eOnjpGiUBEjlNOJJOb5kzk8S9cwk8+UUF3j/OJ+1/m0z+rZEddS6LDk2PYVN3EyaPyKUiiyQ6VCERSlJlxxWljePLLl3LntTNZ+VYt77t7OXc99QZtnQPP0CqJt6m6kRlJVBsAJQKRlJcTyWTBZdP449cu59ozx/LDP27hyrufZdn6anUoJ5n2aDfbaluSqqMYlAhEho0xxbnc89fn8Kv5F1CUG2HBQ6v45E9fYc/BtkSHJoEt+5vp8eTqKAYlApFh5/ypo/j97ZfwD+8/jRe31vO+u5/lwZXb6elR7SDRkvGOIVAiEBmWIpkZfPq9U3nqy5cy++RSvvHb17n5vpVs2d+c6NDS2qs7D1CUG+HkUYlfjKYvJQKRYWzSyHwe/OR5/L8PzuLN/c1cd89z/PB/3tTYgwRZsaWWC6eOIjMjOaaW6KVEIDLMmRnz5kzkD1+5jKvOGMNdT2/m+h+uYPWug4kOLa3sqGthV30bl5xSluhQ3kGJQCRNlBflsPDDs1ny8Qoa26PMu/cF7nv2LfUdxMmKLbUAXDxdiUBEEuzK02NjD646Ywz/8sQmPv1gJfUtnYkOa9h7fkst40tymVqWXP0DoEQgkpaKc7P40Ydn880bzmDFm7W8/wfPsWpHfaLDGra6e5wX3qrj4ullSTP1dF+hJgIz225m68xstZkdcVkxMzvXzLrMbF6Y8YjIIWbGxy+czCO3XUR2JIO/uu9FHnh+mwahhWDDnkYOtkaTsn8A4lMjmOvuZx9piTQzywS+AzwVh1hEpJ/3TCjhd5+/hMtnjOb/PLaB2//jNVo6uhId1rDS2z9w0bT0TQTHcjvwMLA/0YGIpKuSvCwWf2wOd1wzk8fX7eWGHz3PrvrWRIc1bDy/pZaZY4soL0qOpSn7CzsROPCUma0ys/n9D5rZBOBG4N6Q4xCRY8jIMD57+TQe+tT57G9sZ96iF9hU3ZjosFJee7Sbl7fXJ+XdQr3CTgSXuPts4Frgc2Z2ab/j3wfucPejjm4xs/lmVmlmlTU1mnddJEwXTS/jNwsuAuDmRSt5Zbs6kU/Eqh0H6Ozq4ZJ0TQTuvjv4uR94FDivX5EK4D/NbDswD/ixmf3FAOdZ7O4V7l5RXl4eZsgiAswYW8TSBRdRVpjDR5e8xB827Et0SClrxZZaIhnGeVNGJjqUIwotEZhZgZkV9W4DVwHr+5Zx9ynuPtndJwNLgdvc/b/CiklEBm/SyHx+s+BCZowt4m8eWsXSVVWJDiklPb+lltknlSbVQjT9hVkjGAOsMLM1wMvAf7v7MjNbYGYLQryuiAyRUYU5/PIzF3DB1JF87TdrWLz8rUSHlFIOtHSybncDF00flehQjiq0FOXuW4FZA+xfdITyt4QVi4gcv8KcCPffci5f+dUa/vnxTdQ1d3LntTOTcmBUsln+Zg3ucPmM0YkO5aiSt64iIkkjJ5LJDz50DqUFWdy3fCsNbVG+feOZSTeLZrJ55o0aRhVkc9aEkkSHclRKBCIyKJkZxrdueA+l+dn88I9baO7o4u6bzyY7kgzDkZJPd4/z7OYaLj+1nIwkT5hKBCIyaGbGV6+aQWFOhH95YhOtnd38+COzyc3KTHRoSWdt1UHqWzq5fGZyNwtBcowsFpEU8zeXTePbN76HP72xn1sfeIX2jZvhttuguBgyMmI/b7sN3krfzuU/vVFDhsGlSTq/UF9KBCJyXD5y/sncffMscv7wJHb2LHzJEmhqAvfYzyVL4Kyz4IknEh1qQjzzxn5mn1TKiPzsRIdyTEoEInLcbizuYMlj3yGnsx2LRg8/GI1CayvMm5d2NYOapg7WVjUwNwWahUCJQEROxF13Eek+xkyl0Sh873vxiSdJPLs5NhXO5TNSYyYEJQIROX4PPRT7oj+aaBR+/vP4xJMk/vTGfkYX5XD6uOJEhzIoSgQicvyam4e23DDQ1d3D8s01zJ0xOmUG3SkRiMjxKywc2nLDwKs7D9LU3sXcmanRLARKBCJyIj76UcjKOmoRz8qCj30sTgEl3h837SeSYUm9/kB/SgQicvy++tVjJoIOy6T207fFKaDEinb38NvVu7lw2iiKco/+viQTJQIROX7TpsHSpZCf/86EkJVFd24eX7zp7/nzx/eyfndDYmKMoyfWV7O3oZ1bLpqc6FDeFSUCETkx114La9fC/PmHjyyeP5/M9ev40l1fJDPDmLfoBZ5YtzfR0YbG3fnJc1uZWlbA3CSfbbQ/JQIROXHTpsHChdDQAN3dsZ8LF8K0aZw2rpj/+tzFnD6umM/+4lX+fflW3D3REQ+5VTsOsKaqgVsvnpz0k8z1p0QgIqErL4otcHPdmWP59uMb+affvU53z/BKBj9ZsY2SvCxumjMx0aG8a0oEIhIXuVmZLPzQbD7z3in8bOUOFjy0itbOY4xKThG76lt58vVqPnz+SeRnp96kzqEmAjPbbmbrzGy1mVUOcPwjZrY2KPOCmb1jRTMRGT4yMoyvv/90/un60/nDxn3cdO9KdtW3JjqsE/bA89vJMOMTF05OdCjHJR41grnufra7VwxwbBtwmbufCXwLWByHeEQkwW65eAr333Iuuw+0cv3CFax4szbRIR23xvYov3plJ39+1jjGluQmOpzjktCmIXd/wd0PBE9fBFKvcU1EjsvcGaP53ecvYXRRDh+//yUWPftWSvYb/OhPW2jp7OZTl0xNdCjHLexE4MBTZrbKzOYfo+yngPScuFwkTU0uK+DR2y7mmveM5V+f2MQHFq6gcnt9osMatPW7G1jy3Db+qmISZ05M7nWJjybsRHCJu88GrgU+Z2aXDlTIzOYSSwR3HOH4fDOrNLPKmpqa8KIVkbgryInwow/P5ocfOof6lk7mLVrJl3+1mv2N7YkO7ai6unu44+G1lOZn8/fXnZbocE5IqInA3XcHP/cDjwLn9S9jZmcBS4Ab3L3uCOdZ7O4V7l5RXp46EzmJyOCYGdfPGs//fPUyPj93Ov+9di9Xf385T71enejQjugnK7bx+p5GvnnDGZTkp850EgMJLRGYWYGZFfVuA1cB6/uVOQl4BPiYu28OKxYRSQ352RG+dvUMnvjSe5lQmsf8n6/iH/5rHe3R7kSHdpjttS3c/fRmrjp9DNe+Z2yiwzlhYdYIxgArzGwN8DLw3+6+zMwWmNmCoMw3gFHAj490i6mIpJ9p5YU88tmLmX/pVB56cSfX/3AF66qSY66i9mg3/+vhtWRnZvDNG96TMmsOHI2l2lDviooKr6xUvhBJF8+9WcNXf72G2uYOPvPeqXzpylPJy85MSCzNHV3Mf7CSF96q4+6bZ/GXs1PnRkczW3WE2/g1slhEktt7T3rveyMAAAoYSURBVCnn6a9cxs0Vk7hv+VauvWc5K98asDsxVAdaOvnIkpd4aVt9yiWBY1EiEJGkV5KXxb/edBa//Mz5OPChf3+Rf/zt+rhNUVHd0M5fLV7Jxr2NLPronGGVBECJQERSyEXTylj2xUu59eLJ/GzlDq695zleCXncwepdB7nhRyvYfaCNn956Lu87fUyo10sEJQIRSSl52Zn84/Vn8J/zL6DHnZvvW8kdS9eGMmfRw6uquPm+lWRlZrD0sxdx0bTUWX7y3VBnsYikrJaOLu56ajMPvbSDnh7ngxUTue3y6UwamX9C561v6WThH7dw//PbuGjaKBZ+eDYjC7KHKOrEOFpnsRKBiKS86oZ27n1mC//x8i663blo2iiunzWeq88YS0ne4AZ7NbZHeer1fTy2Zg8rttTS3ePcctFk/uH9pxHJTP3GEyUCEUkLexvaeOjFHTy2Zi8761vJzszg7JNGMKk0n4mleUwamc/Jo2KP8sIcOrt7eOaNGn67ejd/2Lifzq4eJpbmcf2s8Vx/1nhOH1+c6H/SkFEiEJG04u6srWrgsTV7WFN1kKoDbVQ3ttP36y4/O5NMM5o6uhhVkB378p81ntknjRgWg8T6O1oiSL2ldEREjsHMmDVpBLMmjXh7X0dXN3sOtrOjroUdda1sr2uhPdrDVWeM4ZLpZWQNg+af46VEICJpISeSyZSyAqaUFSQ6lKSTvilQREQAJQIRkbSnRCAikuaUCERE0pwSgYhImlMiEBFJc0oEIiJpTolARCTNpdwUE2ZWAxwE+i9gWnKMfcfa7v1ZBtQeR2gDXX8wx/vvP9rz/rH23Xc8cccz5r7biXiv9fnQ5+Nox1Px8/FuYgY4xd1LBjy7u6fcA1j8bvcda7vPz8qhimkwx/vvP9rz/rGeaNzxjDnR77U+H/p8DLfPx7uJ+VjXSNWmoceOY9+xtgd6/YnGNJjj/fcf7flAsZ5I3PGMue92It5rfT7ePX0+Br+d7DEf9Rop1zQUNjOr9CPM0JfMUjFuxRw/qRi3Yo6fVK0RhGlxogM4TqkYt2KOn1SMWzHHiWoEIiJpTjUCEZE0p0QgIpLmhnUiMLP7zWy/ma0/jtfOMbN1ZrbFzH5gfdauM7PbzWyTmb1uZv82tFGHE7eZ/ZOZ7Taz1cHjumSPuc/xr5qZm1nZ0EUc2vv8LTNbG7zHT5nZ+BSI+bvB53mtmT1qZiOOda4kifuDwe9gj5kNWQfticR6hPN9wszeDB6f6LP/qJ/7uDqee15T5QFcCswG1h/Ha18GLgAMeAK4Ntg/F/gDkBM8H50icf8T8LVUeq+DY5OAJ4EdQFmyxwwU9ynzBWBRCsR8FRAJtr8DfCcVPh/AacAM4BmgItGxBnFM7rdvJLA1+FkabJce7d+ViMewrhG4+3Kgvu8+M5tmZsvMbJWZPWdmM/u/zszGEfuFftFj/2MPAn8RHP4s8K/u3hFcY3+KxB2qEGP+HvC/gCG/qyGMmN29sU/RgqGOO6SYn3L3rqDoi8DEoYw5xLg3uvsbyRLrEVwNPO3u9e5+AHgauCaRv6sDGdaJ4AgWA7e7+xzga8CPBygzAajq87wq2AdwKvBeM3vJzJ41s3NDjfaQE40b4PNB9f9+MysNL9S3nVDMZnYDsNvd14QdaB8n/D6b2bfNbBfwEeAbIcbaayg+G70+Seyv03gYyrjDNphYBzIB2NXneW/8yfLvAtJs8XozKwQuAn7Tpzku512eJkKsmncBcC7wazObGmT1UAxR3PcC3yL2F+q3gLuI/dKH4kRjNrN84O+JNVvExRC9z7j714Gvm9nfAZ8H/nHIguxnqGIOzvV1oAv4xdBEd9RrDVncYTtarGZ2K/DFYN904HEz6wS2ufuN8Y71eKVVIiBWAzro7mf33WlmmcCq4OnviH1p9q0eTwR2B9tVwCPBF//LZtZDbKKpmmSO29339XndvwO/DzFeOPGYpwFTgDXBL99E4FUzO8/dq5M05v5+ATxOiImAIYrZzG4B/hy4Isw/avoY6vc6TAPGCuDuDwAPAJjZM8At7r69T5HdwOV9nk8k1pewm8T/uw5JVOdEvB7AZPp0+gAvAB8Mtg2YdYTX9e/IuS7YvwD4ZrB9KrFqn6VA3OP6lPky8J/JHnO/MtsZ4s7ikN7nU/qUuR1YmgIxXwNsAMqHOtZ4fD4Y4s7i442VI3cWbyPWUVwabI8c7Oc+Xo+EXDRu/zj4D2AvECX2l/yniP2VuQxYE3z4v3GE11YA64G3gIUcGoWdDTwUHHsV+LMUifvnwDpgLbG/tMYle8z9ymxn6O8aCuN9fjjYv5bYJF8TUiDmLcT+oFkdPIb0TqcQ474xOFcHsA94MpGxMkAiCPZ/MniPtwC3vpvPfbwemmJCRCTNpeNdQyIi0ocSgYhImlMiEBFJc0oEIiJpTolARCTNKRHIsGBmzXG+3hIzO32IztVtsdlK15vZY8ea/dPMRpjZbUNxbRHQCmUyTJhZs7sXDuH5In5oIrZQ9Y3dzH4GbHb3bx+l/GTg9+7+nnjEJ8OfagQybJlZuZk9bGavBI+Lg/3nmdlKM3vNzF4wsxnB/lvM7Hdm9kfgf8zscjN7xsyWWmy+/l/0zhkf7K8ItpuDiebWmNmLZjYm2D8teL7OzP7vIGstKzk06V6hmf2Pmb0anOOGoMy/AtOCWsR3g7J/G/wb15rZ/xnCt1HSgBKBDGf3AN9z93OBm4Alwf5NwHvd/Rxis4P+c5/XzAbmuftlwfNzgC8BpwNTgYsHuE4B8KK7zwKWA5/pc/173P1MDp9pckDBPDtXEBv5DdAO3Ojus4mtg3FXkIjuBN5y97Pd/W/N7CrgFOA84GxgjpldeqzrifRKt0nnJL1cCZzeZ8bI4mAmyRLgZ2Z2CrHZWLP6vOZpd+87F/3L7l4FYGaric1Bs6LfdTo5NInfKuB9wfaFHJpj/pfA/ztCnHnBuScAG4nNWQ+xOWj+OfhS7wmOjxng9VcFj9eC54XEEsPyI1xP5DBKBDKcZQAXuHt7351mthD4k7vfGLS3P9PncEu/c3T02e5m4N+ZqB/qbDtSmaNpc/ezg6m3nwQ+B/yA2HoG5cAcd4+a2XYgd4DXG/Av7n7fu7yuCKCmIRneniI2AygAZtY7jXAJh6b8vSXE679IrEkK4K+PVdjdW4ktb/lVM4sQi3N/kATmAicHRZuAoj4vfRL4ZFDbwcwmmNnoIfo3SBpQIpDhIt/Mqvo8vkLsS7Ui6EDdQGwKcYB/A/7FzF4j3Frxl4CvmNlaYouWNBzrBe7+GrGZSz9EbD2DCjNbB3ycWN8G7l4HPB/cbvpdd3+KWNPTyqDsUg5PFCJHpdtHRUISNPW0ubub2V8DH3L3G471OpF4Ux+BSHjmAAuDO30OEuLSoCInQjUCEZE0pz4CEZE0p0QgIpLmlAhERNKcEoGISJpTIhARSXP/H8tG9za3Bxl3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.recorder.plot(suggestion=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>5.728145</td>\n",
       "      <td>4.841912</td>\n",
       "      <td>0.301525</td>\n",
       "      <td>04:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.292948</td>\n",
       "      <td>3.955799</td>\n",
       "      <td>0.335768</td>\n",
       "      <td>04:06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(2, 3e-1, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm.save(f'en_fine_tuned_1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.777594</td>\n",
       "      <td>3.648559</td>\n",
       "      <td>0.367891</td>\n",
       "      <td>04:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.603667</td>\n",
       "      <td>3.507471</td>\n",
       "      <td>0.382600</td>\n",
       "      <td>05:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.531927</td>\n",
       "      <td>3.433340</td>\n",
       "      <td>0.390342</td>\n",
       "      <td>05:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.480464</td>\n",
       "      <td>3.387439</td>\n",
       "      <td>0.394144</td>\n",
       "      <td>05:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.395352</td>\n",
       "      <td>3.348521</td>\n",
       "      <td>0.397989</td>\n",
       "      <td>05:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.317074</td>\n",
       "      <td>3.316249</td>\n",
       "      <td>0.401825</td>\n",
       "      <td>05:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.219466</td>\n",
       "      <td>3.301881</td>\n",
       "      <td>0.404282</td>\n",
       "      <td>05:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.172760</td>\n",
       "      <td>3.305299</td>\n",
       "      <td>0.404410</td>\n",
       "      <td>05:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.unfreeze()\n",
    "learn_lm.fit_one_cycle(8, slice(3e-2/(2.6**4), 3e-2), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm.save(f'en_fine_tuned')\n",
    "learn_lm.save_encoder(f'en_fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm.load(f'en_fine_tuned')\n",
    "learn_lm.load_encoder(f'en_fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "encoder = get_model(learn_lm.model)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "with open(base_path/'ulm_en_enc.pkl', 'wb') as f:\n",
    "    torch.save(encoder,f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### 2.2.2 Spanish embeddings\n",
    "\n",
    "We'll do the same for spanish, in this case the pre-trained spanish LM was trained by me on [this notebook](./nn-spanish.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "del data_lm\n",
    "del learn_lm\n",
    "import gc\n",
    "gc.collect()\n",
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "path = base_path/'lm_es'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_lm_es = (\n",
    "    TextList\n",
    "        .from_df(df, path=path, cols='es')\n",
    "        .split_by_rand_pct(0.1, seed=42)  # We randomly split and keep 10% for validation\n",
    "        .label_for_lm()  # We want to do a language model so we label accordingly\n",
    "        .databunch(bs=48, num_workers=12))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_lm_es.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_lm_es = load_data(path, bs=48)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm_es = language_model_learner(data_lm_es, AWD_LSTM, pretrained_fnames=[f'es_wt', f'es_wt_vocab'], drop_mult=1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_lm_es.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Min numerical gradient: 5.75E-02\n",
      "Min loss divided by 10: 3.63E-02\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3xdZZ3v8c9vX3Jt2iRteqEt9EKp4KVQQqEqCINUwEFEcV4w6gDqdBDF8TJejp6jc2TGGeU4HrwMTAcBRcTRYhU5yE1lUKTYlhaopUCvNIE2aZI2zXVnZ//OH3ul3U132rTN2pfk+3691it7P+tZe/327m5+eZ5nrecxd0dERGSwSL4DEBGRwqQEISIiWSlBiIhIVkoQIiKSlRKEiIhkFct3ACNp0qRJPmvWrHyHISJSNNasWbPb3euy7RtVCWLWrFmsXr0632GIiBQNM9s+1D51MYmISFZKECIikpUShIiIZKUEISIiWSlBiIhIVkoQIiKSlRKEiIhkpQQhIlLEHt2wi9v+e3Mor60EISJSxB7dsJO7ntwWymsrQYiIFLH27iTjy8OZFEMJQkSkiLX39DG+LB7KaytBiIgUsfaePsaXK0GIiMgg7d1Jxpepi0lERAZRC0JERA7h7rR3awxCREQG6Uz0k3KK7yomM5tvZusytnYz++SgOmZm3zazTWb2nJktzNh3jZm9HGzXhBWniEixau/uAwitBRHainLu/iJwOoCZRYFGYMWgapcA84LtbOBW4GwzqwW+AtQDDqwxs/vdvS2seEVEik17T5AginwM4kJgs7sPXtrucuCHnrYSqDazacA7gEfdvTVICo8CF+coVhGRotDenQTCa0HkKkFcBdybpXw6sCPjeUNQNlT5IcxsqZmtNrPVzc3NIxSuiEjh29/FVGxjEAPMrAR4F/CzMF7f3Ze5e72719fV1YVxChGRgrS/i6mIWxCXAM+4+64s+xqBmRnPZwRlQ5WLiEjgQAuieBPE1WTvXgK4H/ib4Gqmc4C97v4a8DCwxMxqzKwGWBKUiYhIoL0nPQZRFdKd1KFdxQRgZpXARcDfZZRdD+DutwEPApcCm4Au4LpgX6uZ3QSsCg77qru3hhmriEixae/uo6IkSjwazt/6oSYId+8EJg4quy3jsQMfG+LYO4A7woxPRKSYhTmTK+hOahGRohXmWhCgBCEiUrTUghARkazCnMkVlCBERIpWmGtBgBKEiEjRUgtCREQOEfZaEKAEISJSlMJeCwKUIEREitLANBtVakGIiEimsCfqAyUIEZGitH8tCHUxiYhIprCXGwUlCBGRohT2cqOgBCEiUpQOtCDUxSQiIhkOrAWhFoSIiGTY19NHeTxKSSy8X+NKECIiRSjsqb5BCUJEpCiFPdU3KEGIiBSlsCfqg5AThJlVm9lyM9toZi+Y2eJB+z9rZuuCbb2Z9ZtZbbBvm5k9H+xbHWacIiLFJuypviHkNamBW4CH3P1KMysBKjJ3uvvNwM0AZnYZ8Cl3b82ocoG77w45RhGRotPe08ecuspQzxFagjCzCcB5wLUA7p4AEoc55Grg3rDiEREZTcKe6hvC7WKaDTQDd5rZWjO73cyypjszqwAuBu7LKHbgETNbY2ZLhzqJmS01s9Vmtrq5uXkk4xcRKUjuTntPcV/FFAMWAre6+xlAJ/CFIepeBjw5qHvpre6+ELgE+JiZnZftQHdf5u717l5fV1c3guGLiBSmrkQ//Skv6hZEA9Dg7k8Hz5eTThjZXMWg7iV3bwx+NgErgEUhxSkiUlRyMQ8ThJgg3H0nsMPM5gdFFwIbBtcLxireBvwyo6zSzKoGHgNLgPVhxSoiUkz2T/Udcgsi7KuYbgTuCa5g2gJcZ2bXA7j7bUGdK4BH3L0z47gpwAozG4jxx+7+UMixiogUhQMtiCK+zNXd1wH1g4pvG1TnLuCuQWVbgAVhxiYiUqxysRYE6E5qEZGiU/RjECIiEo4DYxDFe5mriIiEYKCLKcy1IEAJQkSk6LTnYC0IUIIQESk6uVgLApQgRESKTi7WggAlCBGRopOLtSBACUJEpOjkYi0IUIIQESk6akGIiEhWuVgLApQgRESKSq7WggAlCBGRopKrtSBACUJEpKjkah4mUIIQESkquVoLApQgRESKyt7u3KwFAUoQIiJFpbUzAUBtZUno51KCEBEpIm1dShAiIpLFQAuipqLIE4SZVZvZcjPbaGYvmNniQfvPN7O9ZrYu2L6cse9iM3vRzDaZ2RfCjFNEpFi0diaoLIlSFo+Gfq6wRzluAR5y9yvNrASoyFLn9+7+l5kFZhYFvgdcBDQAq8zsfnffEHK8IiIFra0zQU0OupcgxBaEmU0AzgO+D+DuCXffM8zDFwGb3H2LuyeAnwCXhxOpiEjxaO1K5GT8AcLtYpoNNAN3mtlaM7vdzCqz1FtsZs+a2a/N7PVB2XRgR0adhqBMRGRMa+1M5GT8AcJNEDFgIXCru58BdAKDxxKeAU5y9wXAd4BfHO1JzGypma02s9XNzc3HG7OISEFr7RwdLYgGoMHdnw6eLyedMPZz93Z37wgePwjEzWwS0AjMzKg6Iyg7hLsvc/d6d6+vq6sb6fcgIlJQ2kZDgnD3ncAOM5sfFF0IHDTIbGZTzcyCx4uCeFqAVcA8M5sdDG5fBdwfVqwiIsWgp6+fzkR/zhJE2Fcx3QjcE/yS3wJcZ2bXA7j7bcCVwEfNLAl0A1e5uwNJM/s48DAQBe5w9z+HHKuISEEbuEkuV2MQoSYId18H1A8qvi1j/3eB7w5x7IPAg+FFJyJSXA5MsxH+RH2gO6lFRIpGW2d6or7aytKcnE8JQkSkSLR2qQUhIiJZtOVwHiZQghARKRotnQnMoFoJQkREMrV1JqgujxONWE7OpwQhIlIkWrtyN1EfKEGIiBSNts4EtTnqXgIlCBGRotGaw6m+QQlCRKRotKoFISIig7k7bV0JascpQYiISIaO3iR9/a4WhIiIHGxgHiaNQYiIyEFyPVEfKEGIiBSFtv3zMOVmoj5QghARKQqtAzO5agxCREQy7Z+oT11MIiKSqaUzQTxqjCsNeyHQA5QgRESKQFtngtrKEsxyM1EfKEGIiBSF1q5EztaBGBBqgjCzajNbbmYbzewFM1s8aP/7zew5M3vezP5oZgsy9m0LyteZ2eow4xQRKXQDLYhcGlZnlpnNBRrcvdfMzgfeBPzQ3fcc4dBbgIfc/UozKwEqBu3fCrzN3dvM7BJgGXB2xv4L3H33cGIUERnNWjsTnHrC+Jyec7gtiPuAfjM7mfQv8ZnAjw93gJlNAM4Dvg/g7onBCcXd/+jubcHTlcCMo4hdRGTMaO3K7UR9MPwEkXL3JHAF8B13/yww7QjHzAaagTvNbK2Z3W5mlYep/2Hg1xnPHXjEzNaY2dKhDjKzpWa22sxWNzc3D+/diIgUkWR/ir3dfTnvYhpugugzs6uBa4AHgrIjXYwbAxYCt7r7GUAn8IVsFc3sAtIJ4vMZxW9194XAJcDHzOy8bMe6+zJ3r3f3+rq6umG+HRGR4rG3uw93CjZBXAcsBv7Z3bea2Wzg7iMc00B63OLp4Ply0gnjIGb2JuB24HJ3bxkod/fG4GcTsAJYNMxYRURGlXxM1AfDTBDuvsHdP+Hu95pZDVDl7l8/wjE7gR1mNj8ouhDYkFnHzE4Efg580N1fyiivNLOqgcfAEmD9cN+UiMhosn+ivhyPQQz3KqbHgXcF9dcATWb2pLt/+giH3gjcE1zBtAW4zsyuB3D324AvAxOBfw9u/ki6ez0wBVgRlMWAH7v7Q0f53kRERoUDE/UVYIIAJrh7u5l9hPTlrV8xs+eOdJC7rwPqBxXflrH/I8BHshy3BVgwuFxEZCzaP1FfIXYxATEzmwb8FQcGqUVEJAcGWhDVFbmbqA+GnyC+CjwMbHb3VWY2B3g5vLBERGRAS0eCypIoZfFoTs87rC4md/8Z8LOM51uA94YVlIiIHNDWlaB2XG67l2CYLQgzm2FmK8ysKdjuMzPd9SwikgOtnbm/ixqG38V0J3A/cEKw/SooExGRkLV1JXJ+DwQMP0HUufud7p4MtrsA3bYsIpIDu9p7mDQud2tRDxhugmgxsw+YWTTYPgC0HPEoERE5Lj19/exq72VmzeDJsMM33ATxIdKXuO4EXgOuBK4NKSYREQm8uqcbgJm15Tk/93Cn2tju7u9y9zp3n+zu70ZXMYmIhG5HWzpBzCjgFkQ2R5pmQ0REjlNDWxcAM2oKtAUxhNytnC0iMkbtaO0mHjWmjC/L+bmPJ0H4iEUhIiJZNbR1cUJ1OdFI7v8mP+yd1Ga2j+yJwIDct3dERMaYHW3debmCCY6QINy9KleBiIjIoRrbunj7qVPycu7j6WISEZEQdSWS7O5IMLM2Py0IJQgRkQLVuP8S1/z06CtBiIgUqB37L3FVC0JERDI0BC2ImaOxBWFm1Wa23Mw2mtkLZrZ40H4zs2+b2SYze87MFmbsu8bMXg62a8KMU0SkEO1o7aI0FqGuKvcT9cHw16Q+VrcAD7n7lWZWAgxuJ10CzAu2s4FbgbPNrBb4Cun1rB1YY2b3u3tbyPGKiBSMhrZupteUY5af+5JDa0GY2QTgPOD7AO6ecPc9g6pdDvzQ01YC1cHa1+8AHnX31iApPApcHFasIiKFaEdbV97ugYBwu5hmA83AnWa21sxuN7PKQXWmAzsynjcEZUOVH8LMlprZajNb3dzcPHLRi4jkWUNbd96uYIJwE0QMWAjc6u5nAJ3AF0b6JO6+zN3r3b2+rk5rGInI6LCvp489XX15u4IJwk0QDUCDuz8dPF9OOmFkagRmZjyfEZQNVS4iMibsv4IpD+tADAgtQbj7TmCHmc0Pii4ENgyqdj/wN8HVTOcAe939NeBhYImZ1ZhZDbAkKBMRGRMa8rgOxICwr2K6EbgnuIJpC3CdmV0P4O63AQ8ClwKbgC7gumBfq5ndBKwKXuer7t4acqwiIgVjR2v6Jrl83QMBIScId19H+lLVTLdl7HfgY0McewdwR3jRiYgUroa2bsrjUWorS/IWg+6kFhEpQDvauphZm797IEAJQkSkIKUvcc3f+AMoQYiIFBx3p6G1K6/jD6AEISJScNq7k+zrTaoFISIiBxuY5juf90CAEoSISMFpyPM6EAOUIERECsyO1vyuJDdACUJEpMBs2d1BbWUJ1RX5uwcClCBERArO5qZO5tYNnvw695QgREQKzObmDubWjct3GEoQIiKFpK0zQUtnQglCREQOtrm5A4C5k9XFJCIiGQYSxMl1VXmORAlCRKSgbG7upCQWYXqeL3EFJQgRkYKyuamDOZMqiUbyN4vrACUIEZECUihXMIEShIhIwejp6+eV1q6CuAcClCBERArG9pYuUg5zJ6sFISIiGfZf4logXUyhrkltZtuAfUA/kHT3+kH7Pwu8PyOWU4E6d2890rEiIqPN5qZ0gphTIF1MoSaIwAXuvjvbDne/GbgZwMwuAz7l7q3DOVZEZLTZ3NzB9OpyKkpy8av5yAqpi+lq4N58ByEiki+bmzsLpvUA4ScIBx4xszVmtnSoSmZWAVwM3HcMxy41s9Vmtrq5uXnEAhcRyaVUytnc3MHJBTJADeF3Mb3V3RvNbDLwqJltdPcnstS7DHhyUPfSsI5192XAMoD6+noP402IiIRtZ3sPXYn+ghmghpBbEO7eGPxsAlYAi4aoehWDupeO4lgRkaJXaFcwQYgJwswqzaxq4DGwBFifpd4E4G3AL4/2WBGR0WLgCqZCmMV1QJhdTFOAFWY2cJ4fu/tDZnY9gLvfFtS7AnjE3TuPdGyIsYqI5NXm5k6qymLUjSvNdyj7hZYg3H0LsCBL+W2Dnt8F3DWcY0VERquBAergD+OCUEiXuYqIjFmbmgpnkr4BShAiInm2t7uPpn29BXUPBOTmTuqCd+ktv2dSVSlvOGE8b5w+gZMmVtK4p5stzR1sbu4gmXJmTazkpIkVzKipYF9PH6/t7eG1Pd3s600yaVwpdVXpbUZ1OTNrKyiLR/P9tkSkSKzZnr7C//QZ1XmO5GBjPkEkkineMH08zze2s+yJLSRTB99KMWlcKbGI8fNnGg851gwq4lE6E/2HlE8dX8aJtRVMHl/GxMoSJo0robaylJqKONUVJdRUximNRYkYRMwoiUWorSwhHlWjTmSsWbmllZJohIUn1eQ7lIOM+QRREovwjSvT4+E9ff28uHMfr7R2MaOmnDl145hQHt+/75XWLhrauqgqizNtQhlTxpcRj0bo6euneV8vTft6aGjrZntLF9taOnmlpYv1jXvZ3dHLvp7kEWMxg4mVJdRVlVFVFiNiYBiRCJTFopSXRCmPR6ksjTGhPM6E8jjVFXFi0QiplJNMOamUE40YsagRjRi9fSl27euhqb2XXe09dPf1059y+vpT9Kc8vTm4+4HnKSflTk1Fyf6WUV3QSpo8vpTJVWWMK40RixqxSIR41PbHNtQAWyrlNO3rpa0rwfSacsaXxUfuH1GkyK3c0sLpJ1YXXM/DmE8QmcriURbMrGbBzEObeWXxKKdMqeKUKYcuJF4WjzKztoKZtRWceVL21+5N9tPW2UdbVyK9dfaR6O8nlYJ+dxLJVJBkemlq76GjN0nKwT1Fqh/2dPXRneinu6+fjt7ksBJOpqqyGJOrSqksjRGLpH+xxyIRSmNGJGJEg5ZMNJLezNLn3NTUwVNbWtjT1Tes81SURKkoiVFZmv5ZURKltTNBY1s3if7U/nq1lSWcNLGCunGllMWjlMYilMYjdCX62deTpKMnSV9/iuqKODUVJdRUllAejxKLGNGoYRg9fenPoyuRpKu3n85Ekq5EP12JfqIRoyJIWuXxKKXxKGXxCGXxKN2Jfl7b283OvT007+ulqizO5PHpJDihIp24PGhImpE+ZyRCaSzCnLpK5k2uYk5dZcH9Z5bi1N7Tx/rGvXz8L+blO5RDKEHkSGksytQJUaZOKBuR1+tPOe3dfezt7iM50GoIfrGnUpBMpVsI8WiEyeNLj3t2yN5kf0YC66UrkSTZn2619PWngl/U/XQnknQm+unqTdLRm/7lfdq08Sx5/RRm1FRQXR7n1T3dbGvpYtvuTl5p7aI3maKnr5/eZIryeJSqshhVZTHi0QiNe3pY39hOW1eC3mTqkLjK4hEqSmKUx6OMK41RUZpOCP0pp7UzQXeQMHqT/fT0pc9TFo8ybUIZUyeUMXfyODp6kjTt62VLcyd7u/sw0onBzEi5k+xPt6oyE1zEoK6qlNLYgeRWEU8nxsrSGFVlcSZWljBxXAm1lSVUlcXSSTlqlEQjVJWlW38TyuNKNGPcqq2tpBzOmVOb71AOoQRRpKIRo6Yy/Zd1LpTGosyoSQ/S58tAN1gy6AIri0WJ5HBh90Qyxdbdnby0ax8v79rHzvYeEskUvcHW2ZukuaOXbS1dtHenW4upYcwOVlGSTlgnVJczvbqcqRPKggRWvj+RqUtu9Fq5pYWSWISFJxbW+AMoQUgRMUuPrcTy9Ad3SSzC/KlVzJ96aDdjNqmUs6e7j5aOXjoT/ST7U/T1p1si+3rSrb89XX20dCR4bW83r+7tYePGJpr39R7yWuNKY0ydUMb06nJmT6pk1sQKZk2qZG7dOE6oLieaw0QpI2vlllbOmFl44w+gBCESmkjEqK1MdzEdjUQyxa72Hna29/Dqnm52tfcEl1X3sKOti9XbWg+6cq4kFmH2xEpmT6pkWnXZ/gsoTp02nnkFdmeuHGxvdx9/fnUvNxbg+AMoQYgUnJJYZP9FD9m4e7ora3cXW5o72LK7k81NHbzctI/fv9x8UPKYXl3O+fPr+IvXTWbx3IkFs1KZpA2MPyyeOzHfoWSlb4tIkTEzJleVMbmqjEWzDx3YHLiRc832Nn63sYkVaxu55+lXKIlFOHt2LefPn8zbT53MSRML667dsWhg/OH0LFdOFgIlCJFRpqosTlVZnFOmVHH1ohPpTfazamsbj7/YxO9ebOKmBzZw0wMbOHt2LVctmsklb5hWkP3fY8HKrS0sLMD7HwaY++hZhK2+vt5Xr16d7zBECtqO1i5+9dyr/NeqHWxv6WJ8WYwlr5/KeafUce7Jk3J2ZdxYt7erj9NveoS/v3Aen3z7KXmLw8zWuHt9tn1qQYiMMTNrK7jh/JO5/ry5rNzawk9X7eDRDbtYvqYBM3jTjGouX3ACl59+AhMLaG2C0eZP21pxh8VzCnP8AZQgRMasSMR489xJvHnuJPpTznMNe3jipd08+sJOvvrABr724Atc8LrJvHfhdM6fP7lgu0GK1cotLZTGIllnbigUShAiQjRinHFiDWecWMPfv30eL+7cx33PNLBibSOPbthFVWm6G+pdp5/AW0+epPsujpO789gLu1g0u7agE68ShIgcYv7UKr546al87h3zeWpLC/eve5WH/ryT+55pYN7kcXz6olN4x+un5vRO9tHkmVf2sL2li49fcHK+QzmsUOeWNrNtZva8ma0zs0NGj83sfDPbG+xfZ2Zfzth3sZm9aGabzOwLYcYpItnFohHOnVfHze9bwKovvZ1vX30GDnz0nme47Lt/4Dcv7GI0XeiSK79Y20hpLMLFb5ia71AOKxctiAvcffdh9v/e3f8ys8DMosD3gIuABmCVmd3v7htCjFNEDqMsHuVdC07gnW+cxi/XNfJ/H3uZD/9gNa+bWsXfnjuHyxacQElM65kcSSKZ4oHnXuWi06ZQVeBzbBXqv+YiYJO7b3H3BPAT4PI8xyQipMcr3rNwBr/5zNv45vsWkHLnMz97lrfd/DvuenIriSyz7soBT7zUTFtXH+9ZOD3foRxR2AnCgUfMbI2ZLR2izmIze9bMfm1mrw/KpgM7Muo0BGWHMLOlZrbazFY3NzePXOQicljxaIT3njmDhz95HndeexYzayv4x19t4O3/9t/c/+yrpIYzle0YtGJdI7WVJZw7ry7foRxR2Anire6+ELgE+JiZnTdo/zPASe6+APgO8IujPYG7L3P3enevr6sr/A9cZLQxMy543WT+a+k53HXdWVSURPnEvWu5/HtP8sdNh+tdHnvae/p4bMMuLnvTtKJYXjjUCN29MfjZBKwg3XWUub/d3TuCxw8CcTObBDQCMzOqzgjKRKRAmRnnz5/Mg584l3/7qwW0dib469uf5to7/8TGne35Dq8gPPT8TnqTKa5YOCPfoQxLaAnCzCrNrGrgMbAEWD+ozlQL5iI2s0VBPC3AKmCemc02sxLgKuD+sGIVkZETyRij+OKlr+OZ7W1ccsvv+fzy52jtTOQ7vLxasbaR2ZMqWTBjQr5DGZYwWxBTgD+Y2bPAn4D/5+4Pmdn1ZnZ9UOdKYH1Q59vAVZ6WBD4OPAy8APzU3f8cYqwiMsLK4lGWnjeXJz53AR9+y2zue6aBC7/5OD9bvSN9aezmzXDDDTB+PEQi6Z833JAuH4Ve3dPNyq0tvPv06UWzRocm6xORnHhx5z6+tOJ5Vm9v4+86X+Tz3/9fRJJ90Nd3oFI8nt6WL4dLLslfsCH4pwc2cMeTW3n8Hy7gxIn5W7p3sMNN1lf4oyQiMirMn1rFT/9uMd9ZNJ5P/scXiXR3HZwcIP28qwuuvHJUtSQa93Tzw5Xbee/CGQWVHI5ECUJEciYSMS577F7KOMK9En198K1v5SaoHLjlsZfA4ZMX5W9a72OhBCEiufWjH2GDWw6D9fXB3XfnJp6QbWrqYPmaBj64+CSmV5fnO5yjogQhIrnV0TGy9QrcNx95kfJ4lBvOn5vvUI6aEoSI5Na4cSNbr4A9u2MPv16/k789b05RLr6kBCEiufWBD6SvVDqMvkiUVee+k1f3dOcoqJGXSjn/+uuN1FaW8JFz5+Q7nGOiBCEiufWZzxwxQaTicT4380LO+8bv+MxPn+WF14rvTuxvPfYST21p4TNLTmFcaXEuvaMEISK5NXdu+j6HiopDE0U8DhUVlK74OXd/7Wo+uPgkHnz+NS655fdctewpHv7zTvqLYBLAX6xt5Du/3cRVZ83krxedmO9wjplulBOR/Ni8OX0p6913pwekx42DD34QPvWpdBIJ7O3q4yerXuGHT22ncU83M2vLuWbxLN5XP5MJ5YW3nsKa7W1c/Z8rOWNmNXd/+OyCXyPjcDfKKUGISFFI9qd4ZMMu7nxyK6u2tVEej/KehdO59s2zmDelKt/hAdDQ1sW7v/cklaUxfnHDW6ipLMl3SEekBCEio8r6xr384I/b+OWzr5JIpjhnTi3XLJ7FRadNIZanabSf2tzCjfeupTfZz4ob3szJkwsjaR2JEoSIjEotHb38dHUDP1qZ7n6aOr6MKxZO5z1nTM9ZqyKVcv7jiS3c/PBGZk2q5Nb3n8n8qcWRHEAJQkRGuf6U87uNTdzz9HaeeHk3/SnnDdPH896FM3jPGTOYUBHOWEVTew9fXPE8j73QxDvfNI2vv/dNRXfFkhKEiIwZzft6uf/ZV/n5Mw38+dV2SmMRLltwAu8/+0ROn1k9IlNt9yb7+f4ftvK9326ir9/54qWv45o3zyqaabwzKUGIyJi0vnEvP/7TK/xybSOdiX5mT6pkyWlTWPL6qZwxs5pI5Oh+oe/c28MfNu3mO799me0tXVx02hT+5ztP5aSJlSG9g/ApQYjImNbRm+RXz77Kg8+/xlObW0imnEnjSjhrVi1nnlTDmSfVcMqUKipKovtbAT19/by0ax/rG9t5dscent7awraWLgDmTR7Hly87jXPn1eXzbY0IJQgRkcDe7j4ef7GJ321sYvX2NhraDkznEY8a48viVJRGeW1PD8ngprwJ5XHOmlXLOXNqOWfORE6dNp7oUbY+CtXhEkRxjaaIiBynCeVxLj99OpefPh2AXe09rNnexiutXezt7qO9u4+O3iQzFpTz+hMm8IYTJjCztrwoxxeOV6gJwsy2AfuAfiA5OEuZ2fuBzwMW1Puouz87nGNFREbClPFlXPrGafkOoyDlogVxgbvvHmLfVuBt7t5mZpcAy4Czh3msiIiEKK9dTO7+x4ynK4EZ+YpFREQOFvY96Q48YmZrzGzpEep+GPj1MR4rIiIjLOwWxFvdvdHMJgOPmugAKh4AAAhxSURBVNlGd39icCUzu4B0gnjrMRy7FFgKcOKJxTutrohIoQm1BeHujcHPJmAFsGhwHTN7E3A7cLm7txzNscH+Ze5e7+71dXXFf02yiEihCC1BmFmlmVUNPAaWAOsH1TkR+DnwQXd/6WiOFRGRcIXZxTQFWBFcOxwDfuzuD5nZ9QDufhvwZWAi8O9BvYHLWbMeG2KsIiIyiO6kFhEZw8bMVBtmthd4OcuuCcDeYT4feJytbBJwtPdlDD7XcPdnK88W01CPjyfmw8U13PiKJeZs5cX4/RhOzJmP9f0Y/v7R/v2Y5+4Tsr66u4+aDVg2nPLDPR94PETZ6pGK6WhjHiqmI8V/LDEfa9zFGPNo+X4MJ+Z8f9b6fhT+92PwVtiraR+9Xw2z/HDPf3WYspGM6Uj7s5UPFdOR4j8WxxJ3McacrbwYvx/DiTnzsb4fw98/lr4fBxlVXUxhM7PVXmRzQinm3CnGuBVz7hRj3KOtBRG2ZfkO4Bgo5twpxrgVc+4UXdxqQYiISFZqQYiISFZKECIiktWYTRBmdoeZNZnZUU/hYWZnmtnzZrbJzL5tGUtNmdmNZrbRzP5sZt8o9JjN7B/NrNHM1gXbpYUec8b+z5iZm9mkkYt4/2uH8VnfZGbPBZ/zI2Z2QhHEfHPwfX7OzFaYWXURxPy+4P9fysxGbFD4eGId4vWuMbOXg+2ajPLDfu9z6liuJx4NG3AesBBYfwzH/gk4h/RKeL8GLgnKLwAeA0qD55OLIOZ/BP6hmD7nYN9M4GFgOzCpGOIGxmfU+QRwWxHEvASIBY+/Dny9CGI+FZgPPA7U5zvWII5Zg8pqgS3Bz5rgcc3h3lc+tjHbgvD01OGtmWVmNtfMHgrWoPi9mb1u8HFmNo30f/SVnv7X/CHw7mD3R4F/dffe4BxNRRBzqEKM+VvA50ivG1IUcbt7e0bVypGOPaSYH3H3ZFB1xBf1CinmF9z9xZGM83hiHcI7gEfdvdXd24BHgYvz+X81mzGbIIawDLjR3c8E/gH49yx1pgMNGc8bgjKAU4BzzexpM/tvMzsr1GjTjjdmgI8HXQh3mFlNeKHud1wxm9nlQKMH65fn0HF/1mb2z2a2A3g/6ckqwzYS348BH+LgRb3CMpIxh204sWYzHdiR8Xwg/kJ5X0CelxwtJGY2Dngz8LOMLr/So3yZGOkm4znAWcBPzWxO8JfAiBuhmG8FbiL91+xNwDdJ/yIIxfHGbGYVwBdJd33kzAh91rj7l4Avmdn/AD4OfGXEghxkpGIOXutLQBK4Z2SiG/I8IxZz2A4Xq5ldB/x9UHYy8KCZJYCt7n5FrmM9VkoQB0SAPe5+emahmUWBNcHT+0n/Qs1sZs8AGoPHDcDPg4TwJzNLkZ6gq7lQY3b3XRnH/SfwQEixDjjemOcCs4Fng/+UM4BnzGyRu+8s4LgHuwd4kBATBCMUs5ldC/wlcGFYf+xkGOnPOUxZYwVw9zuBOwHM7HHgWnffllGlETg/4/kM0mMVjeT/fR2Qr8GPQtiAWWQMOAF/BN4XPDZgwRDHDR5EujQovx74avD4FNJNSCvwmKdl1PkU8JNC/5wH1dlGCIPUIX3W8zLq3AgsL4KYLwY2AHVhfMZhfj8Y4UHqY42VoQept5IeoK4JHtcO93ufqy0vJy2EDbgXeA3oI/2X/4dJ/2X6EPBs8J/iy0McW096hbvNwHc5cEd6CfCjYN8zwF8UQcx3A88Dz5H+y2xaocc8qM42wrmKKYzP+r6g/DnSE6RNL4KYN5H+Q2ddsI30lVdhxHxF8Fq9wC7g4XzGSpYEEZR/KPh8NwHXHc33PlebptoQEZGsdBWTiIhkpQQhIiJZKUGIiEhWShAiIpKVEoSIiGSlBCGjmpl15Ph8t5vZaSP0Wv2Wnvl1vZn96kgzqZpZtZndMBLnFgGtKCejnJl1uPu4EXy9mB+YvC5UmbGb2Q+Al9z9nw9TfxbwgLu/IRfxyeinFoSMOWZWZ2b3mdmqYHtLUL7IzJ4ys7Vm9kczmx+UX2tm95vZb4HfmNn5Zva4mS239FoJ9wzM2R+U1wePO4LJ+Z41s5VmNiUonxs8f97M/mmYrZynODBZ4Tgz+42ZPRO8xuVBnX8F5gatjpuDup8N3uNzZva/R/BjlDFACULGoluAb7n7WcB7gduD8o3Aue5+BumZVr+WccxC4Ep3f1vw/Azgk8BpwBzgLVnOUwmsdPcFwBPA32ac/xZ3fyMHz9yZVTAP0YWk73QH6AGucPeFpNcg+WaQoL4AbHb30939s2a2BJgHLAJOB840s/OOdD6RAZqsT8aitwOnZczAOT6YmXMC8AMzm0d6dtt4xjGPunvmWgB/cvcGADNbR3qOnj8MOk+CA5MfrgEuCh4v5sAc/z8G/s8QcZYHrz0deIH0mgGQnqPna8Ev+1Swf0qW45cE29rg+TjSCeOJIc4nchAlCBmLIsA57t6TWWhm3wV+5+5XBP35j2fs7hz0Gr0Zj/vJ/n+pzw8M8g1V53C63f30YIrzh4GPAd8mvZZEHXCmu/eZ2TagLMvxBvyLu//HUZ5XBFAXk4xNj5CeTRUAMxuYrnkCB6ZWvjbE868k3bUFcNWRKrt7F+klSj9jZjHScTYFyeEC4KSg6j6gKuPQh4EPBa0jzGy6mU0eofcgY4AShIx2FWbWkLF9mvQv2/pg4HYD6WnaAb4B/IuZrSXc1vUngU+b2XOkF5PZe6QD3H0t6Vlgrya9lkS9mT0P/A3psRPcvQV4Mrgs9mZ3f4R0F9ZTQd3lHJxARA5Ll7mK5FjQZdTt7m5mVwFXu/vlRzpOJNc0BiGSe2cC3w2uPNpDiEu8ihwPtSBERCQrjUGIiEhWShAiIpKVEoSIiGSlBCEiIlkpQYiISFb/H8rLsEQDVVpmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm_es.recorder.plot(suggestion=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>6.161613</td>\n",
       "      <td>5.348360</td>\n",
       "      <td>0.287702</td>\n",
       "      <td>04:24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.578285</td>\n",
       "      <td>4.179093</td>\n",
       "      <td>0.330367</td>\n",
       "      <td>04:29</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm_es.fit_one_cycle(2, 3e-1, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm_es.save(f'es_fine_tuned_1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='4' class='' max='8', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      50.00% [4/8 21:26<21:26]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.046064</td>\n",
       "      <td>3.888487</td>\n",
       "      <td>0.355104</td>\n",
       "      <td>05:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.848106</td>\n",
       "      <td>3.719486</td>\n",
       "      <td>0.372208</td>\n",
       "      <td>05:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.753581</td>\n",
       "      <td>3.634134</td>\n",
       "      <td>0.379355</td>\n",
       "      <td>05:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.684632</td>\n",
       "      <td>3.581652</td>\n",
       "      <td>0.383838</td>\n",
       "      <td>05:21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1356' class='' max='1359', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      99.78% [1356/1359 05:09<00:00 3.6147]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm_es.unfreeze()\n",
    "learn_lm_es.fit_one_cycle(8, slice(3e-2/(2.6**4), 3e-2), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm_es.save(f'es_fine_tuned')\n",
    "learn_lm_es.save_encoder(f'es_fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm_es.load(f'es_fine_tuned')\n",
    "learn_lm_es.load_encoder(f'es_fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "encoder = get_model(learn_lm_es.model)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "with open(base_path/'ulm_es_enc.pkl', 'wb') as f:\n",
    "    torch.save(encoder,f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "del learn_lm_es\n",
    "del data_lm_es\n",
    "import gc\n",
    "gc.collect()\n",
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2.3 Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "class ULMEmbedding(nn.Module):\n",
    "    \"Embedding + positional encoding + dropout\"\n",
    "    def __init__(self, ulm_enc, embedding_dim=400, p=0., pos=True):\n",
    "        super().__init__()\n",
    "        self.emb_sz = embedding_dim\n",
    "        self.ulm_enc = ulm_enc\n",
    "        self.pos_enc = None\n",
    "        if pos:\n",
    "            self.pos_enc = PositionalEncoding(self.emb_sz)  # https://github.com/fastai/fastai/blob/1.0.57/fastai/text/models/transformer.py#L11\n",
    "        self.drop = nn.Dropout(p)\n",
    "    \n",
    "    def forward(self, inp): \n",
    "        self.ulm_enc.reset()  # reset hidden state of LSTM\n",
    "        \n",
    "        # take all hidden states of the last layer\n",
    "        hs, _  = self.ulm_enc(inp)\n",
    "        hs = hs[-1]\n",
    "        \n",
    "        if self.pos_enc:\n",
    "            pos = torch.arange(0, inp.size(1), device=inp.device).float()\n",
    "            return self.drop(hs * math.sqrt(self.emb_sz) + self.pos_enc(pos))\n",
    "        else:\n",
    "            return self.drop(hs * math.sqrt(self.emb_sz))\n",
    "    \n",
    "    \n",
    "class Transformer(Module):\n",
    "    def __init__(\n",
    "            self, out_dim, n_layers=6, n_heads=8, d_model=400, d_head=32, \n",
    "            d_inner=1024, p=0.1, bias=True, scale=True, double_drop=True, pad_idx=1, \n",
    "            position_encoder=True):\n",
    "        \n",
    "        with open(base_path/'ulm_en_enc.pkl', 'rb') as f:\n",
    "            self.enc_emb = ULMEmbedding(torch.load(f), p=p)\n",
    "                    \n",
    "        with open(base_path/'ulm_es_enc.pkl', 'rb') as f:\n",
    "            self.dec_emb = ULMEmbedding(torch.load(f), p=0)\n",
    "        \n",
    "        args = (n_heads, d_model, d_head, d_inner, p, bias, scale, double_drop)\n",
    "        self.encoder = nn.ModuleList([EncoderBlock(*args) for _ in range(n_layers)])\n",
    "        self.decoder = nn.ModuleList([DecoderBlock(*args) for _ in range(n_layers)])\n",
    "        \n",
    "        self.out = nn.Linear(d_model, out_dim)\n",
    "        self.pad_idx = pad_idx\n",
    "        \n",
    "    def forward(self, inp, out):\n",
    "        enc = self.encode(inp)\n",
    "        return self.decode(enc, out)\n",
    "    \n",
    "    def encode(self, inp):\n",
    "        enc = self.enc_emb(inp)\n",
    "        return compose(self.encoder)(enc)\n",
    "    \n",
    "    def decode(self, enc, out):\n",
    "        out = self.dec_emb(out)\n",
    "        mask_out = get_output_mask(out, self.pad_idx)\n",
    "        \n",
    "        out = compose(self.decoder)(out, enc, mask_out)\n",
    "        \n",
    "        return self.out(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note:** I can use the spanish embeddings without any masking because they only depend on previous words, if they were bi-directional we would be cheating as the embedding of a word could hint the next words in the target sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Transformer(len(data.y.vocab.itos))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### 2.2.4 Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from fastai.callbacks import EarlyStoppingCallback, SaveModelCallback\n",
    "\n",
    "learn = Learner(\n",
    "    data, model, \n",
    "    loss_func=FlattenedLoss(LabelSmoothingCrossEntropy, axis=-1),\n",
    "    metrics=[accuracy, CorpusBLEU(len(data.y.vocab.itos))],\n",
    "    callback_fns=[partial(EarlyStoppingCallback, monitor='valid_loss', min_delta=0.01, patience=3)],\n",
    ")\n",
    "learn.callbacks.append(SaveModelCallback(\n",
    "    learn, every='improvement', monitor='valid_loss', name='uml-transformer-enc'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n",
      "Min numerical gradient: 5.25E-05\n",
      "Min loss divided by 10: 2.75E-03\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd5hU5d3/8fd3O1tYYFnaUpbeEWRBimiQaIIaETsRBCVYYmKJSX5JfJ4kJo+JiU80MVYU7BoLGkMeG1HARlt6kw5St1B2l7LLlvv3x4zJipQF5uyZ8nld11zMnDlnzvdmdj975p773Mecc4iISOyI87sAERGpXwp+EZEYo+AXEYkxCn4RkRij4BcRiTEJfhdQF02bNnW5ubl+lyEiElEWLlxY7JzLPnJ5RAR/bm4u+fn5fpchIhJRzGzL0Zarq0dEJMYo+EVEYoyCX0Qkxij4RURijIJfRCTGKPhFRGKMgl9EJMZExDj+U7Xv4GFSEuNJSYw/5jqrd5by2YbdVFbXUF3jqK5xVNU4OjVLZ1inpjROS6rHikVEvBfVwf/QB+t5LX8r3+rVglF9WzG4QxYJ8XFU1zg+/LyQqZ9sYs7G3cfc3gz65GQyrHM23+iazZltGxMXZ/XYAhGR0Ivq4P92rxaUlVfy7opdvL5wG03TkxjRrTlzN+1my+6DtMpM4ecjuzH6zBzSkxOIjzMS4uJwzrFsewkfry3mo3VFPDZ7Aw/PXE/zhslc2Lsl3zmjFf3aNMJMfwREJPJYJFyBKy8vz53OlA3lldXMWlPIW0t28MHnhfRs1ZCJZ7fn2z1bkBB/4q85Sg5VMmtNIf+3bCez1hRxuLqGnEYNGNopi64tGtK9RQZdW2SQlZ58yjWKiISamS10zuV9bXksBH9tzrnTOlIvLa9kxsoC3l6+kyVb97H7wOF/P9e8YTIje7Xk0n45nNE68yv7cc7xxZ6DrNxRSnyckZoUH7wlkJWWRHZGsj5BiEhIKfg9UlRWwZpdZXy+q5SFW/byweeFHK6qITcrlVF9c2iUmkj+5r0s2LyHwrKKY75Og8R42mWl0i4rldymaQzukMXgjlkkJxz7i2kRkeNR8NeT0vJK3l2+i78v2c6cjbtxDnIaNWBAbmPycptwRutGmMGhymoOHq7mYEUVxfsr2Lz7IFt2H2Dz7oN8sfsgh6trSEuKZ1jnbEZ0b8Z53ZqpK0lEToqC3weFpeVU1ThaNWpwUtuVV1YzZ8NuZqwu4MPVhewqLSfOYFCHLEb2asG3eragWcMUj6oWkWih4I9QzjlW7ijlvZW7eHv5TjYUHcAM8to1ZkT35nyjazZdm2fo+wER+RoFf5RYV1DG28t38e7KXazeWQpAy8wUzu2SzTe7N2d4t2bE61wDEUHBH5V2lZQze20hs9YU8cm6YsoqqmjduAEThuRy1YA2NExJ9LtEEfGRgj/KVVbX8MHqAqZ+spn5m/eQlhTPFf1bc+2gdnRpnuF3eSLiAwV/DFmxvYSpn25i+tIdVFY7euU05LJ+rbmkbyuaamSQSMxQ8Meg4v0V/GPJDt5cvJ3l20uIjzNGdGvGry/pedIjjUQk8ij4Y9zagjKmLdrGC3O2EB9n3Du6N985o5XfZYmIh44V/JqPP0Z0aZ7Bz0d25+3bh9EhO50fvryYH72yhLLySr9LE5F6puCPMe2y0njt5sHcNqIzf1+ynZF/+ZgFm/f4XZaI1CMFfwxKjI/jR+d34bWbB2MGVz0xh99MX8Whw9V+lyYi9UDBH8P6t2vCu7efw9iz2jH1002M/MtHzN+ko3+RaOdZ8JtZipnNN7OlZrbSzO4JLm9vZvPMbL2ZvWJmurahj9KSE/jtpb14adJZVDvH1ZPn8Ot/rGR/RZXfpYmIR7w84q8AznPOnQH0Bb5tZoOAPwAPOuc6AXuBiR7WIHU0pGNT3r39HMYNasczn23mm3+azTvLdxIJo75E5OR4FvwuYH/wYWLw5oDzgNeDy58FLvWqBjk5ackJ/GZUL6bdMoTGaUnc8uIiJjy9gM3FB/wuTURCyNM+fjOLN7MlQCEwA9gA7HPOfdmPsA3I8bIGOXn92zVm+g+G8suLe7Bwy14u+PNHPD93i99liUiIeBr8zrlq51xfoDUwEOhW123N7EYzyzez/KKiIs9qlKNLiI/jhrPb88Fd5zKkYxb//fcV/N+ynX6XJSIhUC+jepxz+4CZwGCgkZklBJ9qDWw/xjaTnXN5zrm87Ozs+ihTjqJ5wxQeH9ufvHaNufPVJSzcolE/IpHOy1E92WbWKHi/AXA+sJrAH4ArgquNB97yqgYJjZTEeCZfl0erzBS+92y++vxFIpyXR/wtgZlmtgxYAMxwzv0T+H/Aj8xsPZAFTPGwBgmRJmlJPH39QACuf2YBew8c9rkiETlVmqRNTkr+5j1896l59MnJ5KEx/TTLp0gY0yRtEhJ5uU148Kq+LN66j2F/nMktLyxk7sbdGu8vEkESTryKyFdd1KclZ7TJ5Pm5W3hlwVbeWbGLbi0yuOObXfh2rxZ+lyciJ6AjfjklrRun8vOR3ZnzsxH84fLe1DjHzS8s5MmPNvpdmoicgIJfTkuDpHiuHtCW6T88m4t6t+Tet1fz+7dXq+tHJIypq0dCIjkhnofG9CMrPYknPtpI8f7D3Hd5bxLjdWwhEm4U/BIy8XHGPZf0pGl6Mg/MWMueAxU8cu2ZpCbpx0wknOhwTELKzLhtRGd+N7o3s9cWce1T8zTmXyTMKPjFE989qy2PXtuflTtKueLxz9i+75DfJYlIkIJfPPPtXi14/oaBFJZVcPmjn7G2oMzvkkQEBb947KwOWbx602BqnOOKxz4jXxd2F/Gdgl88171lQ6bdMoSm6cmMmzKfBQp/EV8p+KVetGmSyis3DaZloxSuf3oBi77Y63dJIjFLwS/1JjsjmZe+N4is9CTGT53P8m0lfpckEpMU/FKvWmSm8NKkQWQ2SGTslHms2lHqd0kiMUfBL/Uup1EDXp40iNSkeMZOmcfGov1+lyQSUxT84os2TVJ5adIgqmscv3hzueb2EalHCn7xTfumafz4W12Zu3EP03Uhd5F6o+AXX313YFt65TTk3v9bxYGKKr/LEYkJCn7xVWBit14kb9nM55ePg4YNIS4u8O/3vw8bNvhdokjU0TV3xX/vvMPh0ZdBZSVJNdX/WZ6YGLi9/jqMHOlffSIRStfclfC0YQNccQVJFeVfDX2Ayko4eBCuuEJH/iIhpOAXf/3pT4GAP57KSnjwwfqpRyQGKPjFXy+8ULfgf/75+qlHJAZ4Fvxm1sbMZprZKjNbaWa3B5f3NbO5ZrbEzPLNbKBXNUgE2F/Hk7fqup6InJCXR/xVwF3OuR7AIOBWM+sB/BG4xznXF/hl8LHEqvT00K4nIifkWfA753Y65xYF75cBq4EcwAENg6tlAju8qkEiwNixgZE7x5OYCOPG1U89IjGgXoZzmlku8BHQi0D4vwcYgT88Q5xzW46yzY3AjQBt27btv2XL11aRaLBhA/TpExi9cyypqbBsGXTsWH91iUQB34Zzmlk6MA24wzlXCtwC3OmcawPcCUw52nbOucnOuTznXF52drbXZYpfOnYMjNNPTf3akX9VfAIHE5P5YvJzCn2REPI0+M0skUDov+iceyO4eDzw5f3XAH25G+tGjgwc0d9441fO3K2a+D3G3PoEP9jbnKrqGr+rFIkaXo7qMQJH86udcw/UemoHcG7w/nnAOq9qkAjSsSM8/DCUlEB1NZSUkPLEY0y6/nyWbSvhF28up7yy+sSvIyInlODhaw8FxgHLzWxJcNkvgEnAX8wsASgn2I8vcjQX9W7J6uGlPDJzA8u2lfDItWfSMVsjfEROh+bqkYgwc00hd726lPLKau4d3YvR/Vr7XZJI2NNcPRLRhndtxtu3DaNXq0zufGUpP3ltKQcPaxpnkVOh4JeIEbhe71n88LxOvL5oG9/56ye6Zq/IKVDwS0RJiI/jrgu68uLEsygrr+LSRz/l2c8269KNIidBwS8RaUinprxz+zCGdsziV/9YyY3PL2TfwcN+lyUSERT8ErGy0pOZMn4A/3VRd2atKeS6qfM5dFhDPkVORMEvES0uzvjesA48dm1/lm8v4Y5XFlNdo24fkeNR8EtU+GaP5vzy4h68t7KA+95Z7Xc5ImHNyxO4ROrV9UPbs2X3QZ78eBNts9IYN6id3yWJhCUFv0SV/764B1v3HORXb62gdaMGDO/WzO+SRMKOunokqsTHGQ+N6Uf3lg35wUuLWLZtn98liYQdBb9EnbTkBKZOGEDjtCTGT53PuoIyv0sSCSsKfolKzRum8OL3ziIhPo6xU+axdc9xLvQiEmMU/BK12mWl8cLEsyivrGHslHkUlpb7XZJIWFDwS1Tr2iKDZ64fQFFZBeOmzNfZvSIo+CUG9GvbmCevy2NT8QGFvwgKfokRQzs15fFxZ7KmoIwxT85j9/4Kv0sS8Y2CX2LGed2a89R1eWws2s+YJ+dSWKY+f4lNCn6JKed0yebp6wewdc8hrnliLrtKFP4SexT8EnOGdGzKcxMHUlhWwdWT56jbR2KOgl9i0oDcJjw3cSA79h3iN/9c5Xc5IvVKwS8x68y2jbl1eCfeWrKDDz8v8LsckXqj4JeY9v1vdKJr8wzufnMFZeWVfpcjUi8U/BLTkhLiuO/y3uwqLecP737udzki9cKz4DezNmY208xWmdlKM7u91nM/NLPPg8v/6FUNInXRr21jbhjanhfmfsG8jbv9LkfEc14e8VcBdznnegCDgFvNrIeZDQdGAWc453oC/+thDSJ1ctcFXWjTpAE/e2M55ZW6bq9EN8+C3zm30zm3KHi/DFgN5AC3APc55yqCzxV6VYNIXaUmJfD70X3YVHyAn76+jJ0lh/wuScQz9dLHb2a5QD9gHtAFGGZm88xstpkNOMY2N5pZvpnlFxUV1UeZEuPO7tyUW4d35J/LdnDOH2dy16tLWbNLc/lL9DHnnLc7MEsHZgP3OufeMLMVwEzgNmAA8ArQwR2nkLy8PJefn+9pnSJf2rrnIFM+2cQrC7ZyqLKa4V2z+f1lfWiRmeJ3aSInxcwWOufyjlzu6RG/mSUC04AXnXNvBBdvA95wAfOBGqCpl3WInIw2TVL59SU9+exn5/Gj87swf9Mexjw5lwLN5y9RwstRPQZMAVY75x6o9dTfgeHBdboASUCxV3WInKrGaUncNqJzYHqH0nLGTJ6ri7lIVPDyiH8oMA44z8yWBG8XAlOBDsEun78B44/XzSPit/7tmvDMDQPZVVrONZrVU6KA5338oaA+fgkH8zftYcLT82nVqAEvTxpEdkay3yWJHNdp9fGbWUczSw7e/4aZ3WZmjUJdpEg4G9i+CVMnDGD73kOMmzKPAxVVfpckckrq2tUzDag2s07AZKAN8JJnVYmEqUEdsnh8XH/WFpRx5ytLqKkJ/0/MIkeqa/DXOOeqgNHAX51zPwFaeleWSPg6t0s2d1/Ug/dXFfDnf631uxyRk5ZQx/UqzWwMMB74TnBZojcliYS/G4bmsmZXKQ99uJ4uLTK4uE8rv0sSqbO6HvFfDwwmcBLWJjNrDzzvXVki4c3M+O2lvejfrjE/fm0pK7aX+F2SSJ3VKfidc6ucc7c55142s8ZAhnPuDx7XJhLWkhPieXxsf5qkJjHpuXwN85SIUddRPbPMrKGZNQEWAU+a2QMn2k4k2mVnJPPk+Dz2HazklhcWUVGlmT0l/NW1qyfTOVcKXAY855w7C/imd2WJRI6erTL53yvPYOGWvfzXmyuIhHNjJLbVNfgTzKwlcBXwTw/rEYlIF/VpyW0jOvPawm08/elmv8sROa66Bv9vgPeADc65BWbWAVjnXVkikeeOEZ35Vs/m/M//reLjdZpKXMJXXb/cfc0518c5d0vw8Ubn3OXeliYSWeLijAeu6kuX5hn84KXFbCo+4HdJIkdV1y93W5vZm2ZWGLxNM7PWXhcnEmnSkhN48ro84gwmPruA4v0Vfpck8jV17ep5GvgH0Cp4mx5cJiJHaNMklcnX5bFzXzljn5rH3gOH/S5J5CvqGvzZzrmnnXNVwdszQLaHdYlEtAG5TXhqfB4biw8wdso8Sg5V+l2SyL/VNfh3m9lYM4sP3sYCu70sTCTSDe3UlCfG9WddwX6umzqfsnKFv4SHugb/DQSGcu4CdgJXABM8qkkkagzv2oyHv9uPldtLuP7pBZrKWcJCXUf1bHHOXeKcy3bONXPOXQpoVI9IHVzQswV/uaYfi77Yq6mcJSyczqUXfxSyKkSi3EV9WvJfwamcH/pQp8CIv04n+C1kVYjEgOuH5nL5ma3587/W8f7KXX6XI2Fu38HD/PT1pazZVRby1z6d4NfnVZGTYGbcO7oXZ7TO5M5XlrCuIPS/0BI9Pl5XzKv52zhwOPTfCx03+M2szMxKj3IrIzCeX0ROQkpiPI+P60+DpHgmPZevYZ5yTLPXFtEoNZEzWof+8ubHDX7nXIZzruFRbhnOubpevUtEammZ2YDHxvZn295D/OClRRrmKV9TU+OYvbaIYZ2ziY8Lfa/66XT1HJeZtTGzmWa2ysxWmtntRzx/l5k5M2vqVQ0i4WpAbhN+N7o3n64v5sKHPmbhlj1+lyRhZPWuUorKKji3izfnyXoW/EAVcJdzrgcwCLjVzHpA4I8CcAHwhYf7FwlrVw1ow2s3Dwbgysfn8MCMtVRV1/hclYSDWWsCs7ue08Wb42LPgt85t9M5tyh4vwxYDeQEn34Q+Cn6glhiXP92TXj7tmFc2i+Hhz5Yx5VPzGH7vkN+lyU+m722iJ6tGtIsI8WT1/fyiP/fzCwX6AfMM7NRwHbn3NITbHOjmeWbWX5RkeY2l+iVkZLIA1f15a9j+rG+YD83P7+QSh35x6zS8koWbdnrWTcP1EPwm1k6MA24g0D3zy+AX55oO+fcZOdcnnMuLztb88FJ9PvOGa344xV9WL69hEdmrve7HPHJZ+uLqapxkRv8ZpZIIPRfdM69AXQE2gNLzWwz0BpYZGYtvKxDJFKM7N2S0f1y+OuH61m2bZ/f5YgPZq8tIiM5gTPbNfZsH16O6jFgCrDaOfcAgHNueXCun1znXC6wDTjTOafTGEWCfn1JT7LTk7nzlSWUV1b7XY7UI+ccs9cUMbRTUxLjvTsu9/KIfygwDjjPzJYEbxd6uD+RqJDZIJH7r+zDhqID3P/eGr/LkXq0rnA/O0rKObert93bnp2E5Zz7hBPM5xM86heRIwzrnM11g9sx5ZNNjOjejCEddbpLLJgdHMbpZf8+1NOoHhE5eT8b2Y32TdP4yWvLdPnGGDF7bRFdmqfTqlEDT/ej4BcJU6lJCTx4dV+K9ldw0/MLqahSf380O1BRxfxNezw/2gcFv0hY69umEfdf0Yf5m/fw8zeW45zOeYxWczfu5nB1Dd/o2szzfWmiNZEwN6pvDpuLD/Lgv9bSMTudW4d38rsk8cCsNUU0SIwnL9e7YZxfUvCLRIDbRnRi8+7AKJ92Walc3EezokeT6hrHjFUFDO3UlOSEeM/3p64ekQhgZtx3eW/y2jXmrleXsuiLvX6XJCE0d+NudpWWc2m/+vmDruAXiRDJCfE8Ma4/zRumMPGZBbqCVxR5Y9F2MpIT+Gb35vWyPwW/SATJSk/m+YkDSYiPY9yU+Wzdc9DvkuQ0HTpczbsrdjKydwtSEr3v5gEFv0jEaZeVxvMTB3LwcBXjpsyjqKzC75LkNLy/ahcHDlczul/retungl8kAnVr0ZCnrx9IQWkF102dr2v3RrC/L95Oq8wUzmrfpN72qeAXiVD92zXmiXH9WV9YxsRnFnDwcJXfJclJKiqr4KN1xYzql0OcB9fWPRYFv0gEO6dLNn++uh+LvtjLhKcXcKBC4R9Jpi/dQXWNY3S/nBOvHEIKfpEId1Gflvz5mn7kb97D9U8vYL/CP2L8fcl2erZqSJfmGfW6XwW/SBS45IxWPDSmHwu/2MuEqfMV/hFgfeF+lm0rqfejfVDwi0SNi/u04q9j+rF46z7GT51PWbm+8A1nf1+8nTgL/NGubwp+kShyYe+WPDymH0u37mPclPnsO6jpnMNRTY3jzcXbObtzNs0aptT7/hX8IlFmZO+WPHrtmazaUco1k+dqnH8Y+nh9Mdv3HWJ0PU3RcCQFv0gUuqBnC6ZMyGPL7oNc9cQctu875HdJElReWc2v3lpB2yapjOzV0pcaFPwiUWpY52yenziQ4rIKrnp8DpuKD/hdkgB//tc6Nu8+yH2X9a63KRqOpOAXiWJ5uU14+cZBHKqs5srH5/DFbs3t46cV20t48uONXJXXmiGd/LuOsoJfJMr1ysnk1ZsGUVFVzW1/W0xldY3fJcWkquoafvbGMhqnJnH3hT18rUXBLxIDOjXL4PeX9WbJ1n38+V9r/S4nJk35ZBMrtpfym1E9yUxN9LUWBb9IjLi4TyuuymvNo7M2MGfDbr/LiSmbiw/wwIy1XNCjOSN7tfC7HO+C38zamNlMM1tlZivN7Pbg8vvN7HMzW2Zmb5pZI69qEJGv+tV3etI+K407X1nC3gMa419f7pm+kqT4OH4zqhdm9TcZ27F4ecRfBdzlnOsBDAJuNbMewAygl3OuD7AW+LmHNYhILWnJCTw0ph+7D1Tw/6Ytwznnd0lRr+RgJbPXFjF+SC4tMuv/ZK2j8Sz4nXM7nXOLgvfLgNVAjnPufefclxOJzAXq7+oDIkKvnEx++q1uvL+qgBfmfeF3OVHvk/XF1DgY3i3b71L+rV76+M0sF+gHzDviqRuAd46xzY1mlm9m+UVFRd4WKBJjJp7dnnO7ZPOb6SuZt1H9/V76aG0RGSkJnNE6fHq1PQ9+M0sHpgF3OOdKay2/m0B30ItH2845N9k5l+ecy8vODp+/lCLRIC7OeOiafrRtkspNLyzUyV0ecc4xe20Rwzo3JSE+fMbSeFqJmSUSCP0XnXNv1Fo+AbgYuNapk1HEF5mpiUydMIA4M254ZoG+7PXA2oL97Cot59wu4XXw6uWoHgOmAKudcw/UWv5t4KfAJc45nUYo4qN2WWlMHtef7XsPcdMLC6moqva7pKjy0dpAN/U5sRL8wFBgHHCemS0J3i4EHgYygBnBZY97WIOInEBebhPuv7IP8zft4edvLNdInxCavbaILs3TaZnZwO9SviLBqxd2zn0CHG3A6tte7VNETs2ovjlsLj7Ig/9aS3JCYLx5Yhj1SUeig4ermL9pD+OHtPO7lK/xLPhFJLLcNqITh6ureWTmBrbtPcQj155JwxR/pxaIZPM27uFwdU3YdfOApmwQkSAz4yff6sYfr+jDnA27ueKxz9i6R1/DnarZa4tISYxjQG4Tv0v5GgW/iHzFVXlteO6GgewqKWf0o5+yZOs+v0uKSLPXFjG4Q5Zvc+4fj4JfRL5mSKemvPH9ITRIiue7T85l/qY9fpcUUb7YfZBNxQfCbhjnlxT8InJUnZplMO2WIbTMTGHC0/MV/idh9rrwHMb5JQW/iBxTs4wUXp40SOF/kmavKaJNkwa0b5rmdylHpeAXkeNq1jAQ/i2C4b9gs8L/eA5X1TBnQzHndskOiymYj0bBLyIn1KxhCn8Lhv/4qTryP55P1xdz4HA153QOz24eUPCLSB19Gf4tg+H/ybpiv0sKO/M37eG2lxeT06gBQ328mPqJKPhFpM6aNUzhbzcOpl1WKjc8u4APPy/wu6Sw8eHnBYybMo/shsm8evNg0pLD9/xYBb+InJTsjGRenjSIrs0zuOn5hby7YqffJfnurSXbufG5hXRuns5rNw0mp1F4zc1zJAW/iJy0xmlJvPC9s+idk8mtLy3mrSXb/S7JFxVV1TwxewN3vLKE/u0a8/KkQWSlJ/td1gmF72cREQlrmQ0SeW7iWUx8ZgF3vrKEtKQEvtmjud9l1YtdJeW8MHcLL8//gt0HDnN+j+b8dUy/sDxL92gU/CJyytKTE5g6YQBjnpzLD15exEuTBnFm28Z+l+WZwtJy7pm+indX7qLGOUZ0a8b4Ibmc3alp2A7dPBp19YjIaUkLhn+LhilMfGYBG4r2+12SZ/7ywTpmrCrghqG5zP7xcJ4aP4BhncN3vP6xKPhF5LQ1TU/m2RsGEh9nXDdlPoWl5X6XFHI1NY4ZqwoY0b0Zd1/Ug7ZZqX6XdMoU/CISEu2y0pg6YQB7Dx5m/NMLKCuv9LukkFq6bR+FZRVc0DPyv8dQ8ItIyPRp3YjHxvZnXUEZ339xEVXVNX6XFDIzVhUQH2ec11XBLyLyFed2yeZ3l/Xm43XF/OofK6PmGr7vrypgUIcmZKZG/lXJNKpHRELuqrw2bCo+wGOzNtC+aRrfG9bB75JOy4ai/awv3M/Ys9r6XUpIKPhFxBM/uaArm4sPcO/bq2mXlcb5ETzGf8aqwNQU5/ds4XMloaGuHhHxRFyc8cBVfemTk8ltLy9mxfYSv0s6ZTNWFdArp2HYT8VQVwp+EfFMg6R4nhyfR5O0JG54ZgHrCyNvjH9hWTmLvtjL+d2j42gfPAx+M2tjZjPNbJWZrTSz24PLm5jZDDNbF/w3ek/zExGaZaTw9PUDqHGOq5+YE3FH/h+sLsQ5omIY55e8POKvAu5yzvUABgG3mlkP4GfAB865zsAHwcciEsW6NM/g1ZsGk5IYz5jJkXXx9vdX7qJNkwZ0a5Hhdykh41nwO+d2OucWBe+XAauBHGAU8GxwtWeBS72qQUTCR4fsdF67eTDZDZO5buo8Zq4p9LukE9pfUcWnG3ZzfvcWETctw/HUSx+/meUC/YB5QHPn3JcTeO8Cjvr5ycxuNLN8M8svKiqqjzJFxGOtGjXgtZsG0zE7nUnP5jN96Q6/Szquj9YWcbiqJqq6eaAegt/M0oFpwB3OudLaz7nAmR1HPbvDOTfZOZfnnMvLzg7fa1eKyMnJSk/m5RsDs3je9rfFvDhvi98lHdP7K3fRODWRvHbR9VWkp8FvZokEQv9F59wbwcUFZtYy+HxLIPw/74lISDVMSeTZGwYyvGsz7n5zBY/MXB92Z/gePFzFh58XMqJ7cxLio2sApJejegyYAqx2zj1Q66l/AOOD95Ge5/MAAAsBSURBVMcDb3lVg4iErwZJ8Twxrj+j+rbi/vfW8Lu3V4dV+D/72RZKy6u4ekAbv0sJOS/P3B0KjAOWm9mS4LJfAPcBr5rZRGALcJWHNYhIGEuMj+PBq/qS2SCRJz/eRMmhSn43urfvR9il5ZU8PnsD3+iazYDcJr7W4gXPgt859wlwrK/BR3i1XxGJLHFxxj2X9KRRg0Qe+nA9xfsP8/B3+5Ga5N+MMk99tJGSQ5X8+IKuvtXgpejquBKRiGRm/OiCrvz20l7MWlPINZPnUry/wpdadu+vYMonm7iod0t65WT6UoPXFPwiEjbGDWrHE+PyWFtQxmWPfsam4gP1XsNjszZwqLKaO8/vUu/7ri8KfhEJK+f3aM7LkwZxoKKKyx79lEVf7K23fe8sOcRzc7dw2Zmt6dQsvd72W98U/CISdvq1bcy0W4aQ2SCRMZPn8v7KXfWy34c+CAwrvX1E53rZn18U/CISlnKbpjHtliF0b9mQm19YyPNzNnu6v83FB3g1fyvfHdiWNk0i90LqdaHgF5GwlZWezMuTBnFet2b891srue+dz6mpCf1Yf+ccv56+ksR449bzOoX89cONgl9EwlqDpHgeH9ufa89qy+OzN3Dnq0soLC0P6T6em7OFWWuK+MWF3WmWkRLS1w5HuvSiiIS9hPg4/ufSXrRq1ID731vD9KU7OLdLNlf0b8OI7s1ISYw/5ddeW1DGvW+vZnjXbMYNahfCqsOXhdMp0seSl5fn8vPz/S5DRMLAxqL9TFu0jTcWbWdnSTmZDRIZ1bcVVw9oQ89WJzfuvryymksf+ZTi/RW8c/s5ZGcke1S1P8xsoXMu72vLFfwiEomqaxyfbSjmtfxtvLtyF4eraujTOpOrB7ThO2e0Yue+cuZv3kP+5j3kb95LQrzxvbPbc2Vem39/QvjtP1cx5ZNNPD1hAMO7NfO5RaGn4BeRqFVysJI3F2/jbwu28vmusq881ywjmQHtm7CrpJyFW/bSND2ZiWe3JzcrlVteXMT4we24Z1Qvnyr3loJfRKKec46l20r416oCcpumMSC3MW2bpGJmOOeYt2kPj8xcz8frigHo3Cyd6T88+7S+Iwhnxwp+fbkrIlHDzOjbphF92zQ66nODOmQxqEMWy7bt47X8bYwf0i5qQ/94FPwiEnP6tG5En9Zf/+MQKzSOX0Qkxij4RURijIJfRCTGKPhFRGKMgl9EJMYo+EVEYoyCX0Qkxij4RURiTERM2WBmRcCWWosygZIjVqvLstqPj3W/KVB8miUfrZaTXc+rNoZz+462vK6PI+U9PNryaPs5PdryaGtjuPwuHquWL7VzzmV/balzLuJuwORTWVb78XHu53tR38mu51Ubw7l9J2rP8R5Hynt4sm2MxJ/TWGhjuPwunkwba98itatn+ikum16H+6FQ19c73nrh3Eav2ne05XV9HCnv4dGWR9vP6dGWR1sbw+V38ZReLyK6euqTmeW7o8xmFy2ivX2gNkaLaG+jn+2L1CN+L032uwCPRXv7QG2MFtHeRt/apyN+EZEYoyN+EZEYo+AXEYkxURv8ZjbVzArNbMUpbNvfzJab2Xoze8jMrNZzPzSzz81spZn9MbRVn3SdIW+jmf3azLab2ZLg7cLQV35SdXryPgafv8vMnJk1DV3FJ8+j9/G3ZrYs+B6+b2atQl95nWv0on33B38Pl5nZm2bm61VVPGrjlcGcqTGz0H4JHIpxpOF4A84BzgRWnMK284FBgAHvACODy4cD/wKSg4+bRWEbfw382O/3z8s2Bp9rA7xH4MTAptHWRqBhrXVuAx6PsvZdACQE7/8B+EMUvofdga7ALCAvlPVG7RG/c+4jYE/tZWbW0czeNbOFZvaxmXU7cjsza0ngl2auC/zvPwdcGnz6FuA+51xFcB+F3rbi+DxqY1jxsI0PAj8FfB/d4EUbnXOltVZNw8d2etS+951zVcFV5wKtvW3F8XnUxtXOuTVe1Bu1wX8Mk4EfOuf6Az8GHj3KOjnAtlqPtwWXAXQBhpnZPDObbWYDPK321JxuGwF+EPwIPdXMGntX6ik7rTaa2Shgu3NuqdeFnobTfh/N7F4z2wpcC/zSw1pPRSh+Tr90A4Ej5XATyjaGVMxcbN3M0oEhwGu1unqTT/JlEoAmBD6WDQBeNbMOwb/UvgtRGx8DfkvgCPG3wJ8I/GKFhdNto5mlAr8g0FUQlkL0PuKcuxu428x+DvwA+FXIijwNoWpf8LXuBqqAF0NTXWiEso1eiJngJ/DpZp9zrm/thWYWDywMPvwHgeCr/bGxNbA9eH8b8EYw6OebWQ2BiZaKvCz8JJx2G51zBbW2exL4p5cFn4LTbWNHoD2wNPgL2RpYZGYDnXO7PK69rkLxs1rbi8DbhEnwE6L2mdkE4GJgRLgcfNUS6vcwtPz8QsTrG5BLrS9bgM+AK4P3DTjjGNsd+WXLhcHlNwO/Cd7vAmwleBJcFLWxZa117gT+Fm3v4xHrbMbnL3c9eh8711rnh8DrUda+bwOrgGy/3zuv2ljr+VmE+Mtd3/+zPHwTXgZ2ApUEjtQnEjjSexdYGvyh+eUxts0DVgAbgIe/DHcgCXgh+Nwi4LwobOPzwHJgGYEjkpb11Z76auMR6/ge/B69j9OCy5cRmMQrJ8rat57AgdeS4M23UUsetnF08LUqgALgvVDVqykbRERiTKyN6hERiXkKfhGRGKPgFxGJMQp+EZEYo+AXEYkxCn6JSGa2v57395SZ9QjRa1UHZ81cYWbTTzSzpJk1MrPvh2LfIqArcEmEMrP9zrn0EL5egvvPpF+eql27mT0LrHXO3Xuc9XOBfzrnetVHfRL9dMQvUcPMss1smpktCN6GBpcPNLM5ZrbYzD4zs67B5RPM7B9m9iHwgZl9w8xmmdnrwbneX6w1N/qsL+dEN7P9wQnQlprZXDNrHlzeMfh4uZn9Tx0/lczhP5PHpZvZB2a2KPgao4Lr3Ad0DH5KuD+47k+CbVxmZveE8L9RYoCCX6LJX4AHnXMDgMuBp4LLPweGOef6EZil8ne1tjkTuMI5d27wcT/gDqAH0AEYepT9pAFznXNnAB8Bk2rt/y/Oud58dcbFowrO2zKCwBnSAOXAaOfcmQSu/fCn4B+enwEbnHN9nXM/MbMLgM7AQKAv0N/MzjnR/kS+FEuTtEn0+ybQo9ZsiA2DsyRmAs+aWWcCs44m1tpmhnOu9jzq851z2wDMbAmB+Vc+OWI/h/nP5HULgfOD9wfznzn/XwL+9xh1Ngi+dg6wGpgRXG7A74IhXhN8vvlRtr8geFscfJxO4A/BR8fYn8hXKPglmsQBg5xz5bUXmtnDwEzn3Ohgf/msWk8fOOI1Kmrdr+bovyOV7j9fjh1rneM55JzrG5wi+j3gVuAhAvPmZwP9nXOVZrYZSDnK9gb83jn3xEnuVwRQV49El/cJzEQJgJl9OSVuJv+Z6naCh/ufS6CLCeCaE63snDtI4LKId5lZAoE6C4OhPxxoF1y1DMiotel7wA3BTzOYWY6ZNQtRGyQGKPglUqWa2bZatx8RCNG84BeeqwhMow3wR+D3ZrYYbz/l3gH8yMyWAZ2AkhNt4JxbTGAGzTEE5s3PM7PlwHUEvpvAObcb+DQ4/PN+59z7BLqS5gTXfZ2v/mEQOS4N5xQJkWDXzSHnnDOza4AxzrlRJ9pOpL6pj18kdPoDDwdH4uwjjC5ZKVKbjvhFRGKM+vhFRGKMgl9EJMYo+EVEYoyCX0Qkxij4RURizP8HAbq5hN/0KXEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()\n",
    "learn.recorder.plot(suggestion=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>bleu</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>6.558002</td>\n",
       "      <td>4.477090</td>\n",
       "      <td>0.584353</td>\n",
       "      <td>0.408594</td>\n",
       "      <td>01:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.787544</td>\n",
       "      <td>3.464381</td>\n",
       "      <td>0.639028</td>\n",
       "      <td>0.446606</td>\n",
       "      <td>01:41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.492886</td>\n",
       "      <td>3.223813</td>\n",
       "      <td>0.671886</td>\n",
       "      <td>0.472002</td>\n",
       "      <td>01:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.353385</td>\n",
       "      <td>3.184466</td>\n",
       "      <td>0.678727</td>\n",
       "      <td>0.479429</td>\n",
       "      <td>01:41</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with valid_loss value: 4.477090358734131.\n",
      "Better model found at epoch 1 with valid_loss value: 3.4643805027008057.\n",
      "Better model found at epoch 2 with valid_loss value: 3.223813056945801.\n",
      "Better model found at epoch 3 with valid_loss value: 3.1844663619995117.\n"
     ]
    }
   ],
   "source": [
    "learn.fit_one_cycle(4, 1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### 2.3 Bigger model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model = Transformer(len(data.y.vocab.itos), n_layers=12, n_heads=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### 2.3.1 training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from fastai.callbacks import EarlyStoppingCallback, SaveModelCallback\n",
    "\n",
    "learn = Learner(\n",
    "    data, model, \n",
    "    loss_func=FlattenedLoss(LabelSmoothingCrossEntropy, axis=-1),\n",
    "    metrics=[accuracy, CorpusBLEU(len(data.y.vocab.itos))],\n",
    "    callback_fns=[partial(EarlyStoppingCallback, monitor='valid_loss', min_delta=0.01, patience=3)],\n",
    ")\n",
    "learn.callbacks.append(\n",
    "    SaveModelCallback(learn, every='improvement', monitor='valid_loss', name='ulm-transformer-big-enc'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Min numerical gradient: 4.37E-05\n",
      "Min loss divided by 10: 1.58E-03\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd5hV5bXH8e+azgBDHXovoqCAMKCgYMGusSQaNbGbEGPsXhOT3Hi9xsTYrhoxiYiFqLFhDfZgQRDQoYgIogxFQMqAtJmBqev+cfbgiAMMMPuUOb/P85xnzt5n73PW6+Ca97z73es1d0dERJJHSqwDEBGR6FLiFxFJMkr8IiJJRolfRCTJKPGLiCSZtFgHUBetW7f2bt26xToMEZGEMnPmzHXunrvj/oRI/N26dSM/Pz/WYYiIJBQzW1bbfg31iIgkGSV+EZEkE9pQj5n1AZ6psasHcBPQHPg5UBjs/527vxZWHCIi8l2hJX53XwgMBDCzVGAl8CJwMXCPu98V1meLiMjORWuoZxRQ4O61XmgQEZHoiVbiPwd4qsb2FWY218weMbMWtZ1gZqPNLN/M8gsLC2s7RERE9kLoid/MMoBTgeeCXX8HehIZBloF3F3bee4+1t3z3D0vN/d701BFRGQvRaPHfyIwy93XALj7GnevdPcq4CFgaBRiEBFJKKs3bePutxayuLCo3t87Gon/XGoM85hZ+xqvnQHMi0IMIiIJZfmGEu5/ZxErN26t9/cO9c5dM2sMHAv8osbuO8xsIODA0h1eExERoKi0AoDGmfWfpkNN/O5eDLTaYd/5YX6miEhDUBwk/iYhJH7duSsiEodKSiuBcHr8SvwiInFo+1BPRmq9v7cSv4hIHCoOcYxfiV9EJA4Vl1WSkZZCemr9p2klfhGROFRcWhHKMA8o8YuIxKXi0opQhnlAiV9EJC4Vl1WEMpUTlPhFROJScWkl2RrqERFJHkUa6hERSS7FpRrqERFJKiVllerxi4gkkyJN5xQRSR7urumcIiLJpKyyiooqV+IXEUkWxdWVOTXUIyKSHMIs0AZK/CIicacoxEVYIMTEb2Z9zGxOjcdmM7vGzFqa2dtm9mXws0VYMYiIJKKSskjiz060xO/uC919oLsPBAYDJcCLwI3AJHfvDUwKtkVEJFAUjPE3yUzsMf5RQIG7LwNOA8YH+8cDp0cpBhGRhNBQxvjPAZ4Knrd191XB89VA29pOMLPRZpZvZvmFhYXRiFFEJC5sT/wZCZr4zSwDOBV4bsfX3N0Br+08dx/r7nnunpebmxtylCIi8aMh9PhPBGa5+5pge42ZtQcIfq6NQgwiIgmjuCyYx5/AY/zn8u0wD8ArwIXB8wuBl6MQg4hIwigurSA91chMS8DEb2aNgWOBF2rs/gtwrJl9CRwTbIuISKC4tILskMb3AcJ7Z8Ddi4FWO+xbT2SWj4iI1KKotDK0m7dAd+6KiMSdSGXOcIZ5QIlfRCTuFJeFO9SjxC8iEmfCXHYRlPhFROJOcWmlhnpERJJJcVl4q2+BEr+ISNwpLq0IrVwDKPGLiMSdyFCPEr+ISFIoq6iirLIqtJLMoMQvIhJXti/CoqEeEZHkEPayi6DELyISV4pLqytzKvGLiCSF4u3r7WqMX0QkKRRrqEdEJLmEvewiKPGLiMSV6jF+9fhFRJKExvhFRJJMwk/nNLPmZjbBzD43swVmNszMbjazlWY2J3icFGYMIiKJpLi0gtQUIzMtvPQc6tKLwH3AG+5+ppllANnA8cA97n5XyJ8tIpJwiksryc5IxcxC+4zQEr+ZNQNGAhcBuHsZUBZmY0REEl3Yi7BAuEM93YFC4FEzm21m48yscfDaFWY218weMbMWtZ1sZqPNLN/M8gsLC0MMU0QkfoRdix/CTfxpwCDg7+5+MFAM3Aj8HegJDARWAXfXdrK7j3X3PHfPy83NDTFMEZH4EXZJZgg38a8AVrj7jGB7AjDI3de4e6W7VwEPAUNDjEFEJKFEFmEJbyonhJj43X01sNzM+gS7RgHzzax9jcPOAOaFFYOISKIpKg1/qCfsWT1XAk8GM3oWAxcDfzWzgYADS4FfhByDiEjCKC4L/+JuqO/u7nOAvB12nx/mZ4qIJLKSYDpnmHTnrohIHClK8OmcIiKyByoqqyitqEroWT0iIrIHissilTk11CMikiSisQgLKPGLiMSN7YuwKPGLiCSH6qEe9fhFRJJEdY9fY/wiIkmiSEM9IiLJRRd3RUSSzPbpnCGutwtK/CIicUM9fhGRJFNcWoEZNEpXj19EJCkUl1bSOCMt1PV2QYlfRCRuFJdW0Djk8X1Q4hcRiRtFUVhvF5T4RUTiRnEUSjKDEr+ISNyIxiIsEHLiN7PmZjbBzD43swVmNszMWprZ22b2ZfCzRZgxiIgkimgswgLh9/jvA95w9/2BAcAC4EZgkrv3BiYF26EorajE3cN6exGRelUcpTH+0D7BzJoBI4GLANy9DCgzs9OAI4PDxgPvAb8JI4Y731jIMx8vp1fbJvTKbULvtk3o1aYJuU2yaJ6dTrPsdJpmhj91SkSkLopLK8nOSODED3QHCoFHzWwAMBO4Gmjr7quCY1YDbcMKYFjPVpRVVvHlmiLeXVjIczNXfO+Y1BSjWaN0crLSyGmUTk5WOk2z0shMSyE9NYW01BQyUo0mWWm0y8miXbNGtG+WRducLFpkp5OWqsskIlI/Ihd3wx/jDzPxpwGDgCvdfYaZ3ccOwzru7mZW61iMmY0GRgN06dJlrwIYdUBbRh3w7d+VDcVlLF5XxPqiMjZuLWdTSTkbt5axsaScLdsq2LytnM1by1mzeRtllVWUV1RRXuWUV1axZVsFlVXfD7VpZhrNG6fTvFEGbZpm0rllNl2CR+eW2eQ2zaR5o3RSUvStQkR2rrLK2VpemdhDPcAKYIW7zwi2JxBJ/GvMrL27rzKz9sDa2k5297HAWIC8vLx6Gahv0TiDwY1b7tW5lVXO+qJSVm3axurN21izeRsbisvZUFLGpq2Rn19v2sb0xeu3F1qqlpZitGycQW7TTFo2zqBFdgYtstNpnp1By8aRPxhtg28RuU0yyUjTtwiRZFNcFpRkTuShHndfbWbLzayPuy8ERgHzg8eFwF+Cny+HFUN9Sk0x2uRk0SYniwG7OM7d2VBSzlfflLD8mxIKt5Syrqj6UcY3xWUs/6aEb4rL2Lytotb36Ni8EX075HBA+xz6tm/KAe1z6NwiW98aRBqwktJIhzHRe/wAVwJPmlkGsBi4mMhMomfN7FJgGfDjkGOIKrNI775l4wwGdm6+y2MrKqvYUFLO2i3bWLu5lDWbI98mCgqLWbBqM5MWrKF6dCkrPYVebZqwX5um9GrbhL7tcxjYuTnNszOi0CoRCdu3i7Ak9hg/7j4HyKvlpVFhfm6iSEtNIbdpJrlNM+nX4fuvby2r5Is1W1iwajNfri3iizVb+LBgPS/MXrn9mG6tshnQuTl5XVtwcv8OtGysPwQiiShaJZkh/B6/7INGGakM6NycATt8c9i0tZzPVm5izoqNzPlqI9MK1vPynK+5ZeJ8juvbjh8P6czhvVqTqqEhkYRRPcaf6NM5JSTNGqUzvFdrhvdqvX3fwtVbeObj5bw4ewWvfrqKDs2yOGVAB47q04a8bi1I17RTkbhWHIzxq8cvddanXVNu+kFffnNiH96ev4bn8lfw6NQljJ28mKaZaYzYrzXHHNCWEw5sF5UehYjsmeKGMsYv0ZeZlsop/TtwSv8OFJVWMHXROt79fC3vLlzLa5+u5n9e+YwfHtyRnxzSlT7tmsY6XBEJbJ/OqR6/7IsmmWkc368dx/drh7vz8dIN/GvGMp76eDnjpy1jcNcWXDi8Gycd2E53IIvE2Lc9fiV+qSdmxtDuLRnavSU3FZfx/MwVPDljGVc9NZvbmzfi0sO78+MhnaMyvigi31cUjPFnh7zeLqgef1Jq2TiDn4/swTvXH8nY8wfToXkWt0ycz/DbJnHb6wtYsq441iGKJJ3i0gqyM1KjcqOmundJLCXFOK5fO47r145ZX23gocmLeWjyYh58fzFDu7XkrLxOnHRQ+6h89RRJdiVRKskM6vFLYFCXFvz9vMFM++0ofn1CHwqLSrlhwlyG/uk/3PXmQrbuUH9IROpXUWll1IZalfjlO9rmZHH5kb145/ojeO6yYRy1fxvGvLuIY/7vfd6Yt0oL24iEZGNJGU2z4ijxm1lPM8sMnh9pZleZ2a4L0UhCMzOGdGvJmJ8M4pnRh9I0K43LnpjFBY98xOLColiHJ9LgLFlXTLdWjaPyWXXt8T8PVJpZLyKlkjsD/wotKokrh/RoxcQrD+emU/oy56uNnPTXD3h5zsrdnygidbKtvJKVG7fSIze+En+Vu1cAZwD3u/sNQPvwwpJ4k5aawiWHd2fS9UfQv1Nzrn56Dn+cOJ+KyqpYhyaS8JasK8YdeuY2icrn1TXxl5vZuUTq508M9qWHE5LEszY5WTz5s0O4aHg3Hp6yhPMensH6otJYhyWS0BYXRqZQx1uP/2JgGPAnd19iZt2Bx8MLS+JZemoKN5/aj7vPGsDsrzbyg/un8MWaLbEOSyRhVV836946jhK/u89396vc/SkzawE0dffbQ45N4tyPBnfi+V8Op6LKueiRj1izeVusQxJJSAWFRXRs3ihqBRTrOqvnPTPLMbOWwCzgITP7v3BDk0RwYMdmPHLREDZuLefS8R9TUlb7cpIisnOL1xVHbZgH6j7U08zdNwM/BP7p7ocAx4QXliSSAzs2Y8xPDmb+15u56qnZVFZprr9IXbk7iwuL6RGlYR6oe+JPM7P2RNbHnbi7g6uZ2VIz+9TM5phZfrDvZjNbGeybY2Yn7UXcEmeO3r8tN5/aj/8sWMutr86PdTgiCWPtllKKSivo2SY6M3qg7rV6bgHeBKa6+8dm1gP4so7nHuXu63bYd4+731XXICUxXDCsG0vXlfDI1CV0bN6In43oEeuQROJeQXBht0frOEv87v4c8FyN7cXAj8IKShLX708+gBUbSrj11QXM/3ozN5/Wj5wszfwV2ZloT+WEul/c7WRmL5rZ2uDxvJl1qsOpDrxlZjPNbHSN/VeY2VwzeySYJVTbZ442s3wzyy8sLKxLmBIHUlOMB346iKtH9ealOSs58d4P+GjJN7EOSyRuFRQWkZ2RSrucrKh9Zl3H+B8FXgE6BI9/B/t253B3HwScCPzKzEYCfwd6AgOBVcDdtZ3o7mPdPc/d83Jzc+sYpsSD9NQUrj12P567bDipKcbZY6dxxxufU15ZBQUFcPnlkJMDKSmRn5dfHtkvkoQWFxbTvXXjqNThr1bXxJ/r7o+6e0XweAzYbTZ295XBz7XAi8BQd1/j7pXuXgU8BAzdy9glzg3u2oLXrh7Bjwd35m/vFXDv9ffh/fvDuHGwZQu4R36OGwf9+8Prr8c6ZJGoW7yuiB5RKtVQra6Jf72ZnWdmqcHjPGD9rk4ws8Zm1rT6OXAcMC+YHVTtDGDe3gQuiaFJZhq3n9mfBw7N4Vd/+y1WUgLl5d89qLwcSkrgzDPV85eksq28khUbttIziuP7UPfEfwmRqZyriQzPnAlctJtz2gJTzOwT4CPgVXd/A7gjmOI5FzgKuHZvApfEcvJbT5Hlu1nMpbwc7rknOgGJxIGl6yPF2aLd46/rrJ5lwKk195nZNcC9uzhnMTCglv3n72GM0hA88QQpFbu5q7e8HB5/HMaMiU5MIjG2fUZPFG/egn1bgeu6eotCGr6iOi7eUtfjRBqAgrXBHP44HeqpTfQuQUvia1LHr7J1PU6kAVi8rpgOzbKiVpyt2r4kfhVkkbo77zxI382NXOnpcL5GAiV5LC4simqphmq7TPxmtsXMNtfy2EJkPr9I3Vx//W4Tv6enw7W61i/Jwd0piHJxtmq7TPzu3tTdc2p5NHX36H43kcTWsydMmADZ2d/7A1CVlkZJeiZ3/fxWSrt2i018IlFWGBRni/aMHti3oR6RPXPiiTB3Lowe/Z07d1N+8Qvee/ZtHsjaj9Mf+JCFq7WalzR8BcGMnmits1uTEr9EV8+ekemamzZBZWXk55gxnHT6CMZdkEfhlm384P4pjJ1coLr+0qBtr8oZ5Rk9oMQvceSYvm1585qRHNknlz+/9jnnPjSd5d+UxDoskVAsLiymUXp0i7NVU+KXuNKqSSYPnj+Yu84awPyvN3PCvZN5fPoyqtT7lwYmUqMnusXZqinxS9wxM84c3Ik3rhnBoK4t+MNL8/jJuOksW18c69BE6k1BYfSLs1VT4pe41alFNv+8ZCi3/+ggPlu5mRPu/YCHpyyhrKIq1qGJ7JPq4myxmMoJSvwS58yMs4d04a3rRnJoj5b8ceJ8Dr1tErdOnM+XazT7RxLTsvUlQXG22CR+zcWXhNC+WSMeuWgI739RyDMfL+exD5cybsoSBndtweiRPTi+X7tYhyhSZ9UzenrF4K5dUOKXBGJmHNmnDUf2acO6olJemLWCpz9azi8en8n5h3blv085gMy01FiHKbJb24uzRXGB9Zo01CMJqXWTTEaP7Mmb145k9MgePD59GT/6+4d8tV7TPyX+FRQW0bF5IxplxKajosQvCS09NYXfnXQAY88fzFfrSzj5/g94Y97qWIclskuLYlScrZoSvzQIx/Vrx6tXjaB768Zc9sRM/vzaAioqNftH4k9VlVOwtjjqyy3WFGriN7OlwTKLc8wsP9jX0szeNrMvg58twoxBkkfnltk8d9kwzju0C2MnL+a8h2dQuKU01mGJfMfqzdvYWl4Zkxo91aLR4z/K3Qe6e16wfSMwyd17A5OCbZF6kZmWyq2nH8TdZw1g9lcbOeX+D5i5bEOswxLZrnpGT0NP/Ds6DRgfPB8PnB6DGKSB+9HgTrxw+XAy01I5Z+w0HpmyBHeVfZDYq57RE6upnBB+4nfgLTObaWajg31t3X1V8Hw10La2E81stJnlm1l+YWFhyGFKQ9SvQzP+fcXhjOydyy0T53PBIx+xZvO2WIclSa6gsJicrDRaN8mIWQxhJ/7D3X0QcCLwKzMbWfNFj3TBau2GuftYd89z97zc3NyQw5SGqll2OuMuzOOPpx/Ix0u/4fh7J/Pap6t2f6JISAqCGT1msVu2PNTE7+4rg59rgReBocAaM2sPEPxcG2YMImbG+Yd25dWrRtClZTaXPzmL656dQ3FpRaxDkyRUUFgU0/F9CDHxm1ljM2ta/Rw4DpgHvAJcGBx2IfByWDGI1NQztwnP/3I4Vx3di5dmr+TUMVP4QvV+JIq2bCtnzebShpv4iYzdTzGzT4CPgFfd/Q3gL8CxZvYlcEywLRIV6akpXHdcH5742SFs2lrBaWOm8sKsFbEOS5LE4u3LLcZuDj+EWKvH3RcDA2rZvx4YFdbnitTF8J6tee2qw7niqdlc9+wnfLx0A//zg75kpavWj4Rn+1TOGM7oAd25K0msTU4W//rZIVx2RE+e+ugrTn9gKp99vSnWYUkDVlBYRFqK0aVldkzjUOKXpJaWmsKNJ+7PoxcNYV1RGac/MJUH3l2kcg8SikVri+jaKpv01NimXiV+EeCo/dvw1rUjOa5vO+58cyFnPTiNJeu01KPUr4LC4phf2AUlfpHtWjbOYMxPDua+cwayuLCYE++LLPSuO36lPpRXVrFsfXHMx/dBiV/kO8yM0wZ25M1rRjK0eyv+8NI8Ln7sY9bqjl/ZR8u/KaG80tXjF4lX7ZplMf7iIfzvqf2YVrCe4++dzBvzdMev7L2COJnKCUr8IjtlZlw4vBuvXjWCTi2yueyJWfz3S5/qwq/slXiZyglK/CK71atN5I7f0SN78MT0r7hkfD5btpXHOixJMAVri2jTNJOcrPRYh6LEL1IXGWmRJR5v++FBTF20jrP+MY2vN26NdViSQOKhRk81JX6RPXDu0C48dvEQVm7YyukPTGXeSt3wJbvn7pGpnG1iP74PSvwie2xE71wm/HI4aSnGWf+Yxr9mfKUpn7JL64vL2LS1XD1+kUTWp11TXvrVYQzu2oLfvfgpP/9nPuuKtL6v1K561S0lfpEE1yYni39eMpQ/nNKXyV+u44R7JzNpwZpYhyVxaPtUzjiY0QNK/CL7JCXFuPTw7vz7isPJbZrFpePzufH5uWzWrB+pYdHaIhqlp9I+JyvWoQBK/CL1IjL0M5zLjujJs/nLOf6eyby7UIvLCRSXVvDKJ18zuGsLUlJit9xiTUr8IvUkMy2VG0/cnxcuP4wmmWlc/OjHXP/sJ2wqUe8/mY37YAnrikq59tj9Yh3Kdkr8IvVsYOfmTLzqcK44qhcvzVnJ8fdO1hKPSWpdUSljJxdwQr92DO7aItbhbBd64jezVDObbWYTg+3HzGyJmc0JHgPDjkEk2jLTUvmv4/vw0uWHUeXO2Q9O05z/JPTXSV+yraKKG07oE+tQviMaPf6rgQU77LvB3QcGjzlRiEEkJg7q1IxnfzGM7Iw0zn1oOjOXbYh1SBIlS9YV868ZX3HOkM5xM42zWqiJ38w6AScD48L8HJF41q11Y569bBitGmdw/sMz+LBgXaxDkii4682FZKSlcPUxvWMdyveE3eO/F/g1sGM5wz+Z2Vwzu8fMMms70cxGm1m+meUXFhaGHKZIuDo2b8SzvxhGpxaNuPjRj3l7vub7N2Rzlm/k1U9X8bMRPWjTND6mcNYUWuI3s1OAte4+c4eXfgvsDwwBWgK/qe18dx/r7nnunpebmxtWmCJR0yYni6dHD2P/dk0Z/Xg+D75foFIPDVBVlXPbawto1TiD0SN7xDqcWoXZ4z8MONXMlgJPA0eb2RPuvsojSoFHgaEhxiASV1o2zuDp0cM46aD23Pb65/zXc3MpraiMdVhSTzZtLWf04zOZseQbrjl2P5pkpsU6pFqFlvjd/bfu3snduwHnAO+4+3lm1h7AzAw4HZgXVgwi8ahRRipjzj2Ya47pzfOzVvDTh2aozk8DMP/rzZw6ZgrvLVzL//ygL+cd0iXWIe1ULObxP2lmnwKfAq2BW2MQg0hMmRnXHLMfD/xkEPO+3sTpD0xl+TclsQ5L9tKEmSs4429T2VZeydOjD+Xiw7oT6dvGJ0uEMca8vDzPz8+PdRgiofhk+UbOf3gGzbLTeXr0MDo2bxTrkGQ3yiur+GT5RqYuWs+URYV8vHQDw3q04q/nHkxu01rnq8SEmc1097zv7VfiF4m9uSs28tNxM2jZOINnRg+jXbP4mwkiUFpRyW8mzOXt+WsoLqvEDPp1yOGkg9ozekQP0lLjqxjCzhJ/fF55EEky/Ts155+XDOX8hz/i3Iem88zoQ2kTJ5Uc5VvjP1zKS3O+5sd5nTiqTxsO7dGKFo0zYh3WHouvP08iSezgLi0Yf8kQ1m7exrkPTWfVJq3pG0/WF5Vy/6RFHNUnlzvOHMCJB7VPyKQPSvwicWVw15Y8evFQVm3axrH/N5lxHyymonLH+x8lFu79z5eUlFfy+5MPiHUo+0yJXyTODO3ektevHkFetxbc+uoCTrl/Ch8v/SbWYSW1L9ds4V8ffcVPD+lCrzZNYx3OPlPiF4lDXVs15tGLhvCP8wazeWs5Z/1jGjc894lW9oqRP722gOyMVK45Jn5q6u8LJX6ROGVmnHBgO/5z/RFcdkRPXpi9khPv/YAZi9fHOrSk8v4Xhby3sJCrju5NywQd09+REr9InMvOSOPGE/dnwmXDSE81znloOn95/XPKKjT2H7aKyir+9Op8urbK5oLhXWMdTr1R4hdJEAd3acGrV43gnCGd+cf7BZz+wFQWFxbFOqwG7dn8FXyxpojfnrg/mWmpsQ6n3ijxiySQxplp3PbD/jx0QR6rN2/j9AemMuVL1fcPQ1lFFWPe+ZKDuzTn+H7tYh1OvVLiF0lAx/Zty8u/Ooz2zRpx4aMf8fi0pbEOqcF5cfYKvt60jatG9Y7rujt7Q4lfJEF1bpnN85cP58j9cvnDy59x08vzNOe/nlRUVvHAuwX079SMI/dreOuBKPGLJLAmmWmMvSCPX4zswT+nLeOCRz6icItKPO+rVz75mq++KeGKo3o1uN4+KPGLJLzUFOO3Jx3AnWf2Z+ayDZz01w/4cJHG/fdWZZUz5t1F7N+uKcf2bRvrcEKhxC/SQJyV15mXrziMnKw0fvrwDP7v7S+orIr/6rvx5rVPV7G4sJgrj254Y/vVlPhFGpD92+Xw7ysP50eDOvHXSV/y03HTmbdyU6zDShhVVc6YdxbRM7cxJxzYsGby1KTEL9LAZGekcddZA7j7rAF8umITp9w/hZPu+4DHpi5hY0lZrMOLa2/NX8PCNVu44uhepKY0zN4+RCHxm1mqmc02s4nBdnczm2Fmi8zsGTNrGPdAi8SZHw3uxIe/HcUfT+tHSgrc/O/5DP3zJK57do6WeaxFcWkF9036kq6tsvlB/w6xDidU0ejxXw0sqLF9O3CPu/cCNgCXRiEGkaTUrFE65w/rxsQrR/DqVYdzzpDOvDp3FaPufp8/TpzPhmJ9AwBYsGozPxgzhc9Xb+bXx+8fdytp1bdQW2dmnYCTgXHBtgFHAxOCQ8YDp4cZg4hE9OvQjFtOO5D3bjiS0w/uwKNTlzDyznf523uLKCmriHV4MeHuPDljGac9MJUt2yp48meHcHL/9rEOK3Rh/1m7F/g1UH1XSStgo7tX/ytbAXQMOQYRqaF9s0bcceYAXr96JId0b8kdbyzksL+8w/2TvmTT1uQp+1xUWsGVT83m9y/O45BgDYThPVvHOqyoCC3xm9kpwFp3n7mX5482s3wzyy8sLKzn6ESkT7umjLtwCM//cjgHd2nB3W9/wWF/eYfb3/icdUUN+yawsooqfjb+Y16ft5pfn9CH8RcPpXWTzFiHFTXmHs48XzO7DTgfqACygBzgReB4oJ27V5jZMOBmdz9+V++Vl5fn+fn5ocQpIhGffb2Jv71XwGufriI7PZXLj+rFpYd3Jyu94VSlhMjwzg0T5jJh5gruOXsAZxzcKdYhhcbMZrp73o77Q+vxu/tv3b2Tu3cDzgHecfefAu8CZwaHXQi8HFYMIlJ3/To044GfDOLta49geK/W3PnmQkbd/T4vzwLNfh8AAAvTSURBVFlJWB3EWPjbewVMmLmCq0f1btBJf1dicen6N8B1ZraIyJj/wzGIQUR2olebJjx0QR7/+vkhNGuUztVPz+GHf/+QBas2xzq0fTZx7tfc+eZCThvYgWuO6R3rcGImtKGe+qShHpHYqKxyXpi1gtvf+JxNW8u59tj9GD2iR0JOd5z11QbOGTud/h2b8cTPDmlwQ1i1ifpQj4gkvtQU46y8zrx17REc27ctd7yxkLMenJZQK39t2lrOvf/5ggsf/oh2OVk8eP7gpEj6u6Iev4jUibvzyidfc9PLn1FaUcnZeZ3p3DKb9s0a0b55Fh2bN6JN08y4KWy2aWs5j0xZwiNTl7BlWwXH9W3Lf5/cly6tsmMdWtTsrMefFotgRCTxmBmnDezIoT1a8YeX5vFM/nK2lX934Ze2OZkM6tIi8ujanIM6NicjLfoDC//+5Gt+9+KnbNlWwfH92nLVqN7069As6nHEKyV+EdkjbXOyGHtBHu7Opq3lrNq0jVWbtrJsfQlzlm9k1lcbeH3e6uDYTH55RE/OGdolasMrz+Yv5zfPz2VQlxbcclo/JfxaaKhHROrd2i3byF+6gcc+XMpHS76hbU4mlx3Rk3ND/gPw+LSl/OHlzxjRuzVjz8+jUUaSj+XvZKhHiV9EQjWtYD33/ucLZiz5htymmZyd15kf53Wu97H2cR8s5tZXF3DMAW0Y85NBSX8BF5T4RSTGpi9ez9jJi3lv4VqqHIb3bMXZQzpzfL92+5ykH3h3EXe+uZCTD2rPPWcPjMl1hXiki7siElOH9mjFoT1asWrTVibkr+CZ/OVc/fQcWjfJ4IJh3Tjv0K60bLzny3M8+H4Bd765kDMO7sidZ/ZPyHsMok09fhGJiaoqZ2rBOh6ZsoR3FxaSlZ7CWYM7c+nh3enWunGd3uOf05Zy08ufcUr/9tx3zsENetWsvaEev4jElZQUY0TvXEb0zuWLNVsY98Finvl4OY9PX8aBHXM49oB2HNO3DX3b59R6b8CzHy/nppc/49i+bbnn7IFK+ntAPX4RiRtrN2/j+VkreXv+amYv34g7dGzeiEN7tKJ/p2Yc1KkZfdvn8OZnq7nmmTmM6J3LQxcMJjNNF3Jro4u7IpJQCreU8u7na/nPgjXM+moD64oiy0SmphjuzpBuLXns4qFJP2VzVzTUIyIJJbdpJj8e0pkfD+mMu7N68zbmrtjEvJWbKK2o4qpRvZX095ISv4jEPTOL1ARq1ojj+7WLdTgJT/OeRESSjBK/iEiSUeIXEUkySvwiIkkmtMRvZllm9pGZfWJmn5nZ/wb7HzOzJWY2J3gMDCsGERH5vjBn9ZQCR7t7kZmlA1PM7PXgtRvcfUKIny0iIjsRWuL3yJ1h1QtzpgeP+L9bTESkgQt1jN/MUs1sDrAWeNvdZwQv/cnM5prZPWaWuZNzR5tZvpnlFxYWhhmmiEhSiUrJBjNrDrwIXAmsB1YDGcBYoMDdb9nN+YXAshq7mgGbajl0x/272q7teWtg3e5btEs7i21PjqvttbrsU/vUvt2JVfvq0la1b/f2tH1d3T33e6+6e1QewE3Af+2w70hg4l6819i67N/Vdm3Pgfx6aGetse3JcbW9Vpd9ap/aF6/tq0tb1b7otM/dQ53Vkxv09DGzRsCxwOdm1j7YZ8DpwLy9ePt/13H/rrZ39nxf1fW9dnVcba/VZZ/at+/Uvt0ftzftq2tb95XaV4fjQhvqMbP+wHgglci1hGfd/RYzewfIBQyYA1zm7kU7f6foMbN8r6WSXUOh9iU2tS+xxVP7wpzVMxc4uJb9R4f1mfVgbKwDCJnal9jUvsQWN+1LiHr8IiJSf1SyQUQkySjxi4gkmQab+M3sETNba2Z7PGvIzAab2admtsjM/mo1Vno2syvN7POg/tAd9Rv1HsVY7+0zs5vNbGWNOkon1X/kdY4xlN9f8Pr1ZuZm1rr+It7jGMP4/f0xuDFyjpm9ZWYd6j/yOscYRvvuDP7fm2tmL1bPGoyFkNp3VpBXqsws3IvA+zqvNF4fwEhgEDBvL879CDiUyMyj14ETg/1HAf8BMoPtNg2sfTezw70WDal9wWudgTeJ3BDYuiG1D8ipccxVwD8aWPuOA9KC57cDtzew9h0A9AHeA/LCjL/B9vjdfTLwTc19ZtbTzN4ws5lm9oGZ7b/jecF9BjnuPt0jv41/ErnfAOCXwF/cvTT4jLXhtmLnQmpf3AixffcAvybGdaPCaJ+7b65xaGNi2MaQ2veWu1cEh04HOoXbip0LqX0L3H1hNOJvsIl/J8YCV7r7YOC/gL/VckxHYEWN7RXBPoD9gBFmNsPM3jezIaFGu+f2tX0AVwRfpR8xsxbhhbpX9ql9ZnYasNLdPwk70L20z78/M/uTmS0Hfkrkbvl4Uh//PqtdQqS3HE/qs32hSprF1s2sCTAceK7GkG+tBeJ2IQ1oSeRr2hDgWTPrEfzljql6at/fgT8S6Sn+EbibyP9gMbev7TOzbOB3RIYL4k49/f5w998Dvzez3wJXAP9Tb0Hug/pqX/BevwcqgCfrJ7p9V5/ti4akSfxEvt1sdPfvLPxiZqnAzGDzFSLJr+ZXyE7AyuD5CuCFINF/ZGZVRAovxUP50H1un7uvqXHeQ8DEMAPeQ/vavp5Ad+CT4H/MTsAsMxvq7qtDjr0u6uPfZ01PAq8RJ4mfemqfmV0EnAKMiocOVw31/fsLV6wujkTjAXSjxsUX4EPgrOC5AQN2ct6OF19OCvZfBtwSPN8PWE5wE1wDaV/7GsdcCzzdkH5/OxyzlBhe3A3p99e7xjFXAhMaWPtOAOYDubFsV9j/PonCxd2Y/8cL8ZfyFLAKKCfSU7+USI/vDeCT4B/QTTs5N49I8bgCYEx1cidSSvqJ4LVZRFYYa0jtexz4FJhLpHfSPlrtiUb7djgmpok/pN/f88H+uUSKdHVsYO1bRKSzNSd4xHLWUhjtOyN4r1JgDfBmWPGrZIOISJJJtlk9IiJJT4lfRCTJKPGLiCQZJX4RkSSjxC8ikmSU+CUhmVlUl+s0s3Fm1ree3qsyqKA5z8z+vbsqk2bW3Mwur4/PFgGtwCUJysyK3L1JPb5fmn9bACxUNWM3s/HAF+7+p10c3w2Y6O4HRiM+afjU45cGw8xyzex5M/s4eBwW7B9qZtPMbLaZfWhmfYL9F5nZK2b2DjDJzI40s/fMbEJQ9/3JGrXS36uukW5mRUExtE/MbLqZtQ329wy2PzWzW+v4rWQa3xaRa2Jmk8xsVvAepwXH/AXoGXxLuDM49oagjXPN7H/r8T+jJAElfmlI7gPucfchwI+AccH+z4ER7n4wkYqVf65xziDgTHc/Itg+GLgG6Av0AA6r5XMaA9PdfQAwGfh5jc+/z90P4rsVGGsV1HEZReQuaYBtwBnuPojI2g93B394bgQK3H2gu99gZscBvYGhwEBgsJmN3N3niVRLpiJt0vAdA/StUR0xJ6ia2AwYb2a9iVQeTa9xztvuXrOu+kfuvgLAzOYQqccyZYfPKePbAnYzgWOD58P4tvb/v4C7dhJno+C9OwILgLeD/Qb8OUjiVcHrbWs5/7jgMTvYbkLkD8HknXyeyHco8UtDkgIc6u7bau40szHAu+5+RjBe/l6Nl4t3eI/SGs8rqf3/kXL/9uLYzo7Zla3uPjAoFf0m8Cvgr0Rq6OcCg9293MyWAlm1nG/Abe7+4B5+rgigoR5pWN4iUpUSADOrLpHbjG9L314U4udPJzLEBHDO7g529xIiSyReb2ZpROJcGyT9o4CuwaFbgKY1Tn0TuCT4NoOZdTSzNvXUBkkCSvySqLLNbEWNx3VEkmhecMFzPpEy2gB3ALeZ2WzC/ZZ7DXCdmc0FegGbdneCu88mUk3zXCI19PPM7FPgAiLXJnD39cDUYPrnne7+FpGhpGnBsRP47h8GkV3SdE6RehIM3Wx1dzezc4Bz3f203Z0nEm0a4xepP4OBMcFMnI3EybKVIjtSj19EJMlojF9EJMko8YuIJBklfhGRJKPELyKSZJT4RUSSzP8DO8rmcAD1UVkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(suggestion=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>bleu</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>5.880165</td>\n",
       "      <td>4.530049</td>\n",
       "      <td>0.580322</td>\n",
       "      <td>0.386800</td>\n",
       "      <td>03:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.053812</td>\n",
       "      <td>3.560559</td>\n",
       "      <td>0.651603</td>\n",
       "      <td>0.436420</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.732540</td>\n",
       "      <td>3.336047</td>\n",
       "      <td>0.676306</td>\n",
       "      <td>0.461640</td>\n",
       "      <td>03:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.589565</td>\n",
       "      <td>3.249422</td>\n",
       "      <td>0.682652</td>\n",
       "      <td>0.465569</td>\n",
       "      <td>03:07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with valid_loss value: 4.530048847198486.\n",
      "Better model found at epoch 1 with valid_loss value: 3.560558557510376.\n",
      "Better model found at epoch 2 with valid_loss value: 3.3360471725463867.\n",
      "Better model found at epoch 3 with valid_loss value: 3.249422311782837.\n"
     ]
    }
   ],
   "source": [
    "learn.fit_one_cycle(4, 1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 more data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = load_data(base_path, base_path/'500k_data.pk', bs=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Transformer(len(data.y.vocab.itos), n_layers=12, n_heads=12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.callbacks import EarlyStoppingCallback, SaveModelCallback\n",
    "\n",
    "learn = Learner(\n",
    "    data, model, \n",
    "    loss_func=FlattenedLoss(LabelSmoothingCrossEntropy, axis=-1),\n",
    "    metrics=[accuracy, CorpusBLEU(len(data.y.vocab.itos))],\n",
    "    callback_fns=[partial(EarlyStoppingCallback, monitor='valid_loss', min_delta=0.01, patience=3)],\n",
    ")\n",
    "learn.callbacks.append(\n",
    "    SaveModelCallback(learn, every='improvement', monitor='valid_loss', name='best'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n",
      "Min numerical gradient: 4.79E-04\n",
      "Min loss divided by 10: 1.58E-03\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xUZbrA8d+TTgrpQOgQQFAglFAEsevVtWB3VRQUxa7XddV1d+/eveuqu5blWu7uiogF7NhREVdlFZESSgDpvUNIAqSQOs/9YyYaMUAgOXOmPN/P53wyc+acOc/LhCdn3vOe5xVVxRhjTPiIcDsAY4wx/mWJ3xhjwowlfmOMCTOW+I0xJsxY4jfGmDAT5XYAjZGRkaGdO3d2OwxjjAkqCxYs2KOqmQevD4rE37lzZ/Ly8twOwxhjgoqIbGpovXX1GGNMmLHEb4wxYcYSvzHGhBlL/MYYE2Ys8RtjTJixxG+MMWHGEr8xxoQZS/zGGBOAdu2v4InPVrG+oLTZ39sSvzHGBKANe8p49qu17NhX0ezvbYnfGGMCUFFZFQDpiTHN/t6W+I0xJgAV+hJ/WoIlfmOMCQuFpZUApMZb4jfGmLBQVFZFSnw00ZHNn6Yt8RtjTAAqLKtypJsHLPEbY0xAKiytJN0SvzHGhI+isirSE2IdeW9L/MYYE4CKyqpIc2AoJ1jiN8aYgOPxqO+M3xK/McaEhb0HqvEolviNMSZc1I3hT0u0Pn5jjAkLdXft2hm/McaECSfr9IAlfmOMCTg/dPXYGb8xxoSHHwq0OVCnByDKkXcNEHvLqyipqEEVPKp4VIkQoWNaPBER4nZ4xhjToLo6PVEO1OmBEE/8T8xYxZQ5m3+2vlurRMaN6MrI/m2JjYp0ITJjjDm0wlLn6vRAiCf+i/u3J6d9ChEiRERAhAj7K2p4be5m7n9nCU/MWMX1w7swsl9b4mMiiYmKICYywrG/ssYY0xiFZZVkOFSuAUI88Q/slMrATqk/Wz9qSEdmrd3Dc/9ez1+nr+Sv01f+5PXICCEtIYbMxFgyk7xLfEwkVTUeqmo8VNZ6qK1VkltEk5EUQ0ZiLOmJsbRLiSM7M5EUh/rljDHhoaisiq4ZiY69v6OJX0RSgIlAb0CBG4D/AG4CCnyb/VZVP3EyjgbiYkT3TEZ0z+T77ftYuHnvD0m9qsZDRU0txWVVFJRUUlBayaqdJVTU1BITGeH9VhAVQaQIew9UU1RWRa1Hf/L+GYkxdM1MpFurRAZ2TGVI1zTap8b7s4nGmCBWWFpFbufg7ep5CpiuqpeJSAwQjzfxj1fVJxw+dqOc0DaZE9omH/P+Ho+y90A1e0or2VZ8gLW7S1lX4F2m5W/ntbneawztUlowpEsa3VonkhQbRWJcFImx0bSMi6JLZgKZibGI2AVnY8JdrUcpLq8iIxj7+EUkGTgZGAOgqlVAVagltwhft1BaQgw9WidxWs9WP7zm8SirdpUwb0MRczcU8vWaAt5dtK3B92kZF0W3Vol0b5VEn/bJDMtOp0tGgv0xMCbM7C2vwqPOjeEHZ8/4u+DtznlRRHKABcDdvtfuEJHrgDzgXlUtPnhnERkHjAPo2LGjg2E6JyJC6JXVkl5ZLRk9rDOqSmWNh5KKGkorayitqKGovIr1BaWs3e1dPl+xizfztgCQlRzHidnpDO2aTo/WSXTNTKBlXLTLrTLGOKnurl2n6vSAs4k/ChgA3Kmqc0XkKeA3wLPAQ3j7/B8CnsTb9/8TqjoBmACQm5urB78ejESEuOhI4qIjyUz68UM9pUfmD49VlY2F5cxet4fZawuZuaqAdxf++C0hIzGWrpkJ5LRP5qTumQzunEaLGBuSakyoqLt5Kyi7eoCtwFZVnet7PhX4jaruqttARJ4HpjkYQ9AREbpkJNAlI4FrhnTC41E2FJaxbncp6wrKWO+7fvDy7E08/80GYiIjGNAphdN7tmL0sM52X4IxQa6wtO6MPwgTv6ruFJEtInKcqq4CzgCWi0iWqu7wbXYxsMypGEJBRISQnZlIduZPh3YdqKpl/sYivl27h1lr9/DIJyv5MH87z141gM4ZCS5Fa4xpqqIyZ+v0gPOjeu4EXvWN6FkPXA88LSL98Hb1bARudjiGkNQiJpKTe2Rysq+b6PPlu/j12/mc/8wsHrmkDxfmtHU5QmPMsXC6Tg84nPhVdTGQe9Dqa508Zrg66/jWfHL3CO56fRF3vb6I79bt4Z6zetgwUWOCTGGps3V6IMTv3A037VJa8Ma4ofzt89X8Y+Y6Xp+3hYSYSDqkxdMpPZ7js5IZO6ILibH2sRsTqIrKnK3TA5b4Q050ZAQPnNOTc05ow8LNxWwqLGdLUTnrCsqYsXwXb+Vt4dFL+vzQRWSMCSxO1+kBS/whK6dDCjkdUn6ybsGmIu6buoTrJs3jytwO/Pa8XiS3sPsCjAkkhaVVPxvM0dysDGUYGdgpjU/uGsEtp2Tz9oIt/Mf4r3lr/hYqqmvdDs0Y41NUVuXoUE6wxB924qIj+c25PXnvtuGkJ8Zw/ztLGProFzzyyQo2F5a7HZ4xYa3WoxQ5XKcHLPGHrZwOKUy78yTeGDeUYdnpvDBrA6c88RU3vZLHuoJSt8MzJiztLa9CHa7TA9bHH9ZEhKFdvbWAdu6r4LW5m3jx242c879fc8PwLtx5RncbAWSMH/mjTg/YGb/xaZMcx6/OPo4vf30qF/Vrx3Nfr+f0J2by3qKtqIZEqSRjAt6eUufr9IAlfnOQzKRYHr88h/duG0ZWchz3vJnPn6YtdzssY8LCj2f8lviNC/p3TOW924YzZlhnXvx2I+8s2Op2SMaEvLo6PekOj+O3xG8OKSJC+P15vRiWnc6D7y1l6dZ9bodkTEir6+pJjXf2/hpL/OawoiIjeOaq/mQmxnLz5DwKSyvdDsmYkFVU5nydHrDEbxohPTGW564dSGFZFbe/tpDqWo/bIRkTkorKqkh3+MIuWOI3jdS7XTKPXtKHOeuLeGjacjweG+ljTHPbU1rpeP8+2Dh+cxQuGdCe5dv3M3HWBjYXlfPk5TmkOzze2JhwUlTmfJ0esDN+c5R+d14v/nxRb2avK+QXT3/DnPWFbodkTMgoLKsi3eGhnGCJ3xwlEWHU0E68f9twEmKiuPr5OTz1rzXUWtePMU1S61GKy62P3wSw49u25MM7T+LCnLaM/9dqfvfeUrvD15gm8FedHrA+ftMEibFRjL+yH+1T43n2q7V0SIvn9tO6uR2WMUGpbq5df1w3s8RvmkREuPfsHmwtLufxz1bRPrUFI/u1czssY4JOoe/mLX909VjiN00mIvz1sr7s2FfBfW8voU3LOIZ0TXc7LGOCir/q9ID18ZtmEhsVyYRrc+mQ1oJxkxdYTX9jjlKhn+r0gCV+04yS46N56frBREcK1zw/l+/W2VBPYxqr0E91esASv2lmHdLimTx2CC1iIrl64hwe/XQFVTVW4sGYI/FXnR6wxG8c0CurJR/fdRK/HNSR5/69nov+71vW7CpxOyxjAlphWaVfhnKCJX7jkPiYKB69pA/PX5fLzv0VnP/MLL5audvtsIwJWNv3VtA2uYVfjmWJ3zjqrONbM/0/R5Cdmchdbyxiw54yt0MyJiBtKSqnQ1oIJH4RSRGRqSKyUkRWiMiJIpImIp+LyBrfz1QnYzDua5UUx3PXDiQqQhj3Sh5llTVuh2RMQCmrrKGwrIoOafF+OZ7TZ/xPAdNVtSeQA6wAfgN8oardgS98z02I65AWzzNXDWBdQSn3Tc238g7G1LOluByAjsGe+EUkGTgZeAFAVatUdS8wEnjZt9nLwEVOxWACy0ndM7j/nJ58snQnz3293u1wjAkYmwu9ib9DapAnfqALUAC8KCKLRGSiiCQArVV1h2+bnUDrhnYWkXEikicieQUFBQ6Gafzp5pO7cl6fLB6bvpJv1tjnagzA5qIQOePHWw5iAPAPVe0PlHFQt456v+83+J1fVSeoaq6q5mZmZjoYpvEnEeGxy/rSrVUit05ZaPX8jQG2Fh8gKTaKFD/cvAXOJv6twFZVnet7PhXvH4JdIpIF4PtpY/zCTEJsFC/fMJg2yXFcN2keny/f5XZIxrhqc1E57dPiERG/HM+xxK+qO4EtInKcb9UZwHLgQ2C0b91o4AOnYjCBKyu5BW/ffCK9slpyy5QFvJ23xe2QjHHN5qJyOvppKCc4P6rnTuBVEVkC9AMeAf4CnCUia4Azfc9NGEpNiOG1G4cwLDud+6Yu4Xm74GvCkKqypajcb/374HBZZlVdDOQ28NIZTh7XBI+E2Cgmjs7lV2/m8/AnK2gRE8mooZ3cDssYvykoqaSyxhM6id+YxoiNiuTpq/pTVlXDn6Ytp1+HFHq3S3Y7LGP8om5ET3s/Jn4r2WACQmSE8OTlOaTFx3DHawspqah2OyRj/MLfQznBEr8JIOmJsTx9VX+2FB/gt+8ts7t7TVjYUnQAEWiXEjoXd405KoO7pPGrs3rwUf52Xp9nI31M6NtcVE7rpDjioiP9dkxL/Cbg3HpKNiO6Z/DHj75n+fb9bodjjKP8PaIHLPGbABQRIYy/sh8pLaIZ9cJcXp+3mVqPdfuY0LSluNxvVTnrWOI3ASkjMZZXbxxCt8xEHnx3KRc8M4u5Vt7BhJiK6lp27q/wWx3+Opb4TcDq3jqJN28eyrNX92dveRVXTpjD7a8tZItvFIQxwW7b3gOo+ndED9g4fhPgRITz+7bljJ6tmfD1ev7x77XM+H4no4Z24o7TupGeGOt2iMYcMzeGcoKd8Zsg0SImkrvP7M7MX5/GpQPa8/LsjZzy+Eye/XIN5VU2o5cJTlst8RtzZG2S4/jLpX2Zcc/JDMtO54kZqzn3qW/4fvs+t0Mz5qhtLionNiqCzCT/fnO1xG+CUrdWSUy4Lpc3xg2lstrDJX+fzdQFW90Oy5ijsrnIO6LHX+WY61jiN0FtaNd0pt11EgM6pvLrt/N58N2lVFTXuh2WMY2ypeiA37t5wBK/CQEZibFMHjuYW0/N5vV5m7n8n9+xe3+F22EZc1h15Zg7pPp3KCdY4jchIioyggfO6cmEaweyrqCUKyfMYce+A26HZcwh7S2vpqSyxu83b4ElfhNizj6hDa/cMJiCkkqufG4OW4ttzL8JTFuK3RnRA5b4TQjK7ZzG5LGDKS6v4srn5tgNXyYg1Y3htzN+Y5pJ/46pvHbjUEora7jiue/YsKfM7ZCM+QlL/MY4oE/7ZF6/aSiVNR5GPjuL6ct2uh2SMT/YUnSA9IQYEmP9X0DBEr8Jace3bcl7tw2jc0YCt0xZwB8+WGbDPU1A2FLk/6qcdSzxm5DXKT2BqbcM48aTuvDKd5u4+O+zWVdQ6nZYJsxttsRvjLNioiL4/fnHM2lMLjv3HeCCZ2Yxx8o8G5fU1HrYtvcAHf1cjrmOJX4TVk7v2ZpP7z6ZdiktGPvSfBZuLnY7JBOGtu09QK1H6ZBqZ/zG+EWb5DhevXEIGUmxjJk0zwq8Gb/714rdAOR2TnXl+Jb4TVhq1dKb/BNjo7j2hXms3V3idkgmjLy/aBt92iXTrVWSK8e3xG/CVvvUeF69aSiREcLVz89lU6GN9TfOW7u7hKXb9nFR/3auxdCoxC8i2SIS63t8qojcJSIpzoZmjPO6ZCTw6o1DqK71cN2keRSVVbkdkglx7y3aRoTABTlZrsXQ2DP+d4BaEekGTAA6AK8daScR2SgiS0VksYjk+db9UUS2+dYtFpFfHHP0xjSDHq2TmDh6EDv2VXDz5Dwqa2ycv3GGx6O8v2g7I7pn0iopzrU4Gpv4PapaA1wMPKOq9wGN/XN1mqr2U9XceuvG+9b1U9VPjiZgY5wwsFMqT16ew/yNxfzmnaWoqtshmRCUt6mYbXsPcLGL3TzQ+MnWq0XkKmA0cIFvXbQzIRnjjgty2rKpsIwnZqymc3oCd5/Z3e2QTIh5b9FW4mMiOfuE1q7G0dgz/uuBE4GHVXWDiHQBJjdiPwVmiMgCERlXb/0dIrJERCaJSIPjmURknIjkiUheQUFBI8M0pmluP60blw5oz/h/reaDxdvcDseEkIrqWqYt2cE5J7QhPsb/9Xnqa1TiV9XlqnqXqr7uS9RJqvrXRux6kqoOAM4FbheRk4F/ANlAP2AH8OQhjjlBVXNVNTczM7NRjTGmqUSERy/pw5Auadz39hK+XbvH7ZBMiJi5ajclFTWujuap09hRPTNFpKWIpAELgedF5G9H2k9Vt/l+7gbeAwar6i5VrVVVD/A8MPjYwzem+cVERfDctQPpkpHATa/kscju7jXN4N2F28hMimV4twy3Q2l0V0+yqu4HLgFeUdUhwJmH20FEEkQkqe4xcDawTETqXxS+GFh29GEb46yU+Bgmjx1MZlIsY16cz4od+90OyQSxveVVfLVqNyNz2hIZIW6H0+jEH+VL2FcA0xq5T2tglojkA/OAj1V1OvCYb4jnEuA04J6jDdoYf2jVMo4pY4fQIjqSa1+YZ5O5mGP28dIdVNdqQHTzQOMT/5+Az4B1qjpfRLoCaw63g6quV9Uc33KCqj7sW3+tqvZR1b6qeqGq7mhaE4xxToe0eKbcOASPKqMmzmX7XpvA3Rwdj0d5ZfYmerZJ4oS2Ld0OB2j8xd23fYn6Vt/z9ap6qbOhGRMYurVK5JUbBrP/QDXXTZrHvvJqt0MyQWT69ztZtauEW0/NRsT9bh5o/MXd9iLynojs9i3viEh7p4MzJlD0bpfMhOty2VxYzk2v5NksXqZRPB7l6S/W0DUzgfP7tnU7nB80tqvnReBDoK1v+ci3zpiwcWJ2Ok9ckcO8jUXc+1Y+Ho/d3WsOb8bynazcWcKdp3cLiIu6dRqb+DNV9UVVrfEtLwE2uN6EnQtz2vLbX/Tk46U7ePiTFW6HYwKYx6M89cVaumQkcEEAne1D4xN/oYiMEpFI3zIKsHnrTFi6aURXxgzrzAuzNjDxm/Vuh2MC1OcrdrFix37uOK0bUZGBVQG/sdHcgHco5068d9teBoxxKCZjApqI8F/nH8+5vdvw549X8P4iK+1gfkrV27ffOT2ekf0C62wfGj+qZ5Nv6GWmqrZS1YsAG9VjwlZkhDD+yn4M7ZrGvW/n8+XKXW6HZALIv1bs5vvt+7nj9O4Bd7YPTZuB61fNFoUxQSguOpLnr8vl+KyW3DplIfM2FLkdkgkAqspTX6ymU3o8FwXg2T40LfEHziVqY1ySFBfNS9cPol1qC8a+NN8mbjd8s2YPy7bt5/ZTA69vv05TorKxbMYA6YmxTB47hMS4KEZPmsdGK+0Q1l6YtYGMxFhG9g/Ms304QuIXkRIR2d/AUoJ3PL8xBmiX0oLJY4dQ61FumbLAbvAKU6t3lfDv1QWMPrETsVGRbodzSIdN/KqapKotG1iSVNXdmQSMCTDdWiXytyv7sXJnCY/aGP+wNGnWBmKjIrhmaCe3QzmswOyAMiZInXZcK24Y3oWvPpnDlquuh5YtISLC+/O222DdOrdDNA4pLK3k3UXbuGRAe9ISYtwO57As8RvTzB6U9cx46U7avDUZSkpA1ftz4kTo2xc+/dTtEI0DpszZTFWNh7EndXY7lCOyxG9Mc1q3jugrrySuqoJoz0H9/NXVUF4Ol11mZ/4hpqK6lslzNnLacZl0a5XkdjhHZInfmOb05JPeBH841dUwfrx/4jF+8WH+dvaUVjH2pK5uh9IolviNaU5TpjQu8U+e7J94jONUlUmzNtCzTRLDu6W7HU6jWOI3pjmVljbvdibgfbu2kJU7S7jhpC4BM9HKkVjiN6Y5JSY273Ym4E2Zs4n0hBguzAmeW5ss8RvTnEaNgujow28THQ3XXuufeIyjSiqq+XLVbi7IaUtcdODesHUwS/zGNKd7721c4r/nHv/EYxz1+fJdVNV4uCAny+1QjoolfmOaU3Y2TJ0K8fE/+wNQGxlFeVQsq//vRe92Juh9lL+ddiktGNAx1e1QjoolfmOa27nnwpIlMG7cT+7c9dx4E6Pvfp47i9tQXetxO0rTRMVlVXyzZg/n52QFzUXdOpb4jXFCdjY8+yzs2we1tbBvH9H//Ds3XX8Wq3aV8OyXa92O0DTRp8t2UuPRgJtPtzEs8RvjR2ef0IZL+rfjqS/W8FH+drfDMU3wUf52umYkcELblm6HctQs8RvjZ49e2odBnVO59+18Fmwqdjsccwx2769gzoZCzs9pG3TdPGCJ3xi/i42K5Llrc8lKjmPcK3lsKSp3OyRzlD5eugNVuDDIRvPUcTTxi8hGEVkqIotFJM+3Lk1EPheRNb6fwXU53JhmkJYQw6Qxg6iu9XDDS/PZd+AIZR5MQPkwfzu9sloGRUG2hvjjjP80Ve2nqrm+578BvlDV7sAXvufGhJ3szET+ee1ANuwp4/ZXF9pInyCxpaicRZv3Bt3Y/frc6OoZCbzse/wycJELMRgTEIZlZ/DIJX2YtXYPD7yzBFWbyjrQTVuyAyAoR/PUcTrxKzBDRBaIyDjfutaqusP3eCfQuqEdRWSciOSJSF5BQYHDYRrjnityO3DPmT14d+E2npyx2u1wzBF8lL+d/h1T6JAW73Yox8zpeXNPUtVtItIK+FxEVtZ/UVVVRBo8xVHVCcAEgNzcXDsNMiHtrjO6sXP/AZ79ai1ZKXFcMySw52wNV8u372f5jv384fzj3Q6lSRxN/Kq6zfdzt4i8BwwGdolIlqruEJEsYLeTMRgTDESEh0b2Ztf+Sv7r/WW0SorjrOMb/DJsXDR5zkbioiO4dEB7t0NpEse6ekQkQUSS6h4DZwPLgA+B0b7NRgMfOBWDMcEkKjKCZ6/uT592ydz5+kIWbrYx/oFkX3k17y/azsicdiTHH6EQX4Bzso+/NTBLRPKBecDHqjod+AtwloisAc70PTfGAPExUbwwZhCtW8Zx48t5bCosczsk4/P2gi0cqK7l2hODvxvOscSvqutVNce3nKCqD/vWF6rqGaraXVXPVNUip2IwJhhlJMby4phBeFQZ8+J8isqq3A4p7Hk8ypQ5mxjYKZXe7ZLdDqfJ7M5dYwJQ18xEJl6Xy7a9Bxj3Sh4V1bVuhxTWvlm7h42F5VwXAmf7YInfmICV2zmN8Vf0I29TMfe+lY/HY4Pb3PLK7I1kJMZwTu82bofSLCzxGxPAzuubxW9/0ZOPl+7gL9NXHnkH0+y2FJXz5ardXDW4I7FRwTO94uE4PY7fGNNEN43oytbiA0z4ej1ZyXFcP7yL2yGFlSlzNxEhwtVDOrodSrOxxG9MgBMR/vuCE9i5r4I/TVtOq6Q4zusbvHVigklFdS1vzd/CWb1ak5Xcwu1wmo119RgTBCIjhKev6s/Ajqnc8+Zi5qwvdDuksPDB4m0Ul1dz3bDQuKhbxxK/MUEiLjqSiaNz6ZDWgpteyWPVzhK3Qwppu/dX8JdPV9K3fTIndk13O5xmZYnfmCCSEh/DyzcMJj4mktGT5rFt7wG3QwpJqsr97yyhvKqWv12RE5SzbB2OJX5jgkz71Hheun4wZZU1XPP8HHbuq3A7pJAzZc4mZq4q4Le/6BW0k60cjiV+Y4JQr6yWvDx2MHtKq7j6+Tns3m/Jv7ms3V3Kw5+s4OQemSFzw9bBLPEbE6QGdEzlpesHsXN/BVdPnEtBSaXbIQW9qhoP97y5mLjoSB6/rG/IdfHUscRvTBDL7ZzGpDGD2FpczjUT51BYasm/KZ7+Yg1Lt+3j0Yv70LplnNvhOMYSvzFBbmjXdCaNHsSmwnKumTiXPZb8j8nCzcX8feZaLh3QnnP7hPZ9Epb4jQkBw7plMHF0Lhv2lHHVBOvzP1oHqmr59Vv5ZCW34I8XBvfsWo1hid+YEDGieyYvXT+YbXsP8MsJNtrnaDz+2SrW7ynjscv6khQX3JOsNIYlfmNCyInZ6bxyw2B2l1RyxXPfsbW43O2QAt6c9YVM+nYDo0/sxPBuGW6H4xeW+I0JMbmd05g8djDF5VVc+dwcm8XrMEora7hvaj6d0+N54NyebofjN5b4jQlB/Tum8vpNQymvquGyf35n5R0O4ZFPVrC1+ABPXJ5DfEz41Ky0xG9MiOrdLpk3bz6RCIErnvuORTZ5+098tXI3r83dzLgRXcntnOZ2OH5lid+YENajdRJTbxlGcotorpk4l2/X7nE7pICwZlcJd72+iJ5tkrjnrB5uh+N3lviNCXEd0uKZesuJdEiN5/oX5zPj+51uh+SqPaWVXP/SfGKjI3lhzCDiokNjVq2jYYnfmDDQqmUcb948lF5tW3LXG4vYsCc8L/hWVNcy7pU89pRW8sLoXNqlhM7kKkfDEr8xYSIlPoYJ1w4kJjKCX7+dT22YTd6uqtw3dQkLN+9l/BX9yOmQ4nZIrrHEb0wYad0yjv8ZeQILNhUz8Zv1bofjV+P/tYaP8rfzwDk9Q74kw5FY4jcmzFzUrx1nH9+aJz9fzZpd4THMc/a6PTz9xRouG9ieW07p6nY4rrPEb0yYEREevrgPCTGR3Pt2PjW1HrdDclRpZQ33T11Cl4wEHhrZO2RLLR8NS/zGhKHMpFj+fFEflmzdxz9mrnM7HEc9/PEKtu89wBOX96VFTPiN4GmI44lfRCJFZJGITPM9f0lENojIYt/Sz+kYjDE/d17fLM7vm8XTX65hzvpCt8NxxL9XF/D6vM3cNKIrAzuF101ah+OPM/67gRUHrbtPVfv5lsV+iMEY04CHRvamQ1o8170wjw/zt7sdTrPad6CaB6YuoXurxLC8SetwHE38ItIeOA+Y6ORxjDHHJjUhhndvHUa/Dinc9foi/jFzHaqhMczzTx8tp6C0kievyAnLm7QOx+kz/v8F7gcOvnr0sIgsEZHxIhLrcAzGmMNIiY9h8o2DuTCnLX+dvpLfvb8s6C/4fr26gHcWbuW2U7Pp2z58x+sfimOJX0TOB3ar6oKDXnoQ6AkMAtKABw6x/zgRyRORvIKCAqfCNMYAsVGR/O+V/bjt1Gxem7uZMS/OD+pZvD5YvJ3U+GjuPL2726EEJCfP+IcDF4rIRuAN4HQRmaKqO9SrEngRGNzQzqo6QVVzVTU3MzPTwTCNMQAREcL95/TksUv7krepiHOe+iYo6/qoKrPX7ZMMKnMAAA1QSURBVOHE7HRiomzgYkMc+1dR1QdVtb2qdgZ+CXypqqNEJAtAvINpLwKWORWDMeboXTGoA9PuHEHblDjGTV7Ag+8uobyqxu2wGm1jYTk79lUwLDs8ZtM6Fm78OXxVRJYCS4EM4M8uxGCMOYxurRJ599bh3HpqNm/M38J5T89ifUGp22E1Sl3p6XCZRvFY+CXxq+pMVT3f9/h0Ve2jqr1VdZSqBsdvkzFhJiYqggfO6cnrNw1l34FqRk2cGxRz+M5et4es5Dg6p8e7HUrAsg4wY8xhDe3qncC9pLKGURPnBvRFX49H+W5dIcOyM6w0w2FY4jfGHFHvdsm8dP1gdpdUMuqFuRSXVbkdUoOW79hPcXk1w7ulux1KQLPEb4xplIGdUpl4XS4bC8sZ/eI8Siqq3Q7pZ2avs/79xrDEb4xptGHdMvjHNQNYvn0/Vzw3h6Vb97kd0k98u7aQ7MwEWreMczuUgGaJ3xhzVM7o1Zrnr8tlT2klI/9vFn+etpyySveHe1bVeJi/scjO9hvBEr8x5qid1rMV//rVKfxycEcmztrA2eO/5quVu12NKX/rXsqrahmWbf37R2KJ3xhzTJJbRPPIxX14+5YTaRETyfUvzXe1tv+3a/cg4h2FZA7PEr8xpkkGdU7j47tO+qHI29NfrHEljtlrC+ndNpmU+BhXjh9MotwOwBgT/GKjIhl/ZT+iIoS/fb6amloP95zVw29j6curali0pZgbTuril+MFO0v8xphmERkhPH55DtGRETz95VqqPcr9/3GcX5L/vA1FVNcqw60+T6NY4jfGNJvICOHRS/oQFSm+SV3gN+f2dPy4360rJDpSyO2c6vixQoElfmNMs4qIEP58UW8A/vnvdbRPbcGooZ0cPea36/bQv2Mq8TGW0hrDLu4aY5qdiPA/F57Aacdl8t8ffs+/Vzs3mdLW4nKWbdvPKT1s3o7GssRvjHFEVGQEz1w9gB6tk7j91YWs3LnfkeN8lL8DgAtz2jry/qHIEr8xxjGJsVFMGpNLfEwkY1/KY3dJ81f2/GDxNvp3TKFDmpVhbixL/MYYR2Ult+CF0YMoKqti7Et5zF1fSK1Hm+W9V+8qYeXOEkba2f5RscRvjHFcn/bJPHNVf9buLuXKCXMY+ugX/Nf7y/huXdP+CHy4eDsRAuf1tcR/NOwSuDHGL848vjXzf38mX63czSdLd/D2gi1MnrOJVkmxXJjTlov6t+OEti0bPe5fVfkwfzvDu2WQmRTrcPShxRK/McZvEmOjuCCnLRfktKW8qoavVhbwweJtvPzdRibO2kCP1olcOqA9Y4Z3JjYq8rDvlb91H5uLyrnj9G7+CT6EWOI3xrgiPiaK8/pmcV7fLPaWVzFtyQ7eX7SNRz9dyfuLt/PMVf3o1irpkPt/sHgbMVERnNO7jR+jDg3Wx2+McV1KfAyjhnZi6q3DeGF0Lrv2V3D+M7N4de4mVH9+DaDWo0xbsoPTjsukZVy0CxEHN0v8xpiAckav1ky/ewSDOqfxu/eWcfPkBRQdNMfvnPWFFJRUcmFOO5eiDG6W+I0xAadVyzhevn4wvz+vF1+t2s0pj3/F32eu5UBVLeAdzZMQE8kZvVq5HGlwsj5+Y0xAiogQbhzRlZN7ZPLY9JU8Nn0VL8/eyF1ndOfTZTv4jxPaEBd9+AvApmF2xm+MCWg9WicxcfQg3rr5RNqltOB37y1jf0UNF/azsfvHys74jTFBYXCXNN65dRiffb+LxVv2cpJNqn7MLPEbY4KGiHBO7zY2hLOJHO/qEZFIEVkkItN8z7uIyFwRWSsib4qITZBpjDF+5I8+/ruBFfWe/xUYr6rdgGJgrB9iMMYY4+No4heR9sB5wETfcwFOB6b6NnkZuMjJGIwxxvyU02f8/wvcD3h8z9OBvapa43u+FbA7MIwxxo8cS/wicj6wW1UXHOP+40QkT0TyCgqcm7bNGGPCjZNn/MOBC0VkI/AG3i6ep4AUEakbTdQe2NbQzqo6QVVzVTU3M9Pm0jTGmObiWOJX1QdVtb2qdgZ+CXypqtcAXwGX+TYbDXzgVAzGGGN+zo07dx8AfiUia/H2+b/gQgzGGBO2pKGSp4FGRAqATfVWJQP7Gtj04PWHe97Q4wxgTxPDPVRsR7NdQ681Zp21z9p3JG61rzFttfYd2dG2r5Oq/ryvXFWDbgEmNGb94Z439BjIcyq2o9muodcas87aZ+0L1PY1pq3WPv+0T1WDtkjbR41cf7jnh3rcVI19r8Nt19BrjVln7Ws6a9+RtzuW9jW2rU1l7WvEdkHR1eMvIpKnqrlux+EUa19ws/YFt0BqX7Ce8TtlgtsBOMzaF9ysfcEtYNpnZ/zGGBNm7IzfGGPCjCV+Y4wJMyGb+EVkkojsFpFlx7DvQBFZ6psz4GlfVdG61+4UkZUi8r2IPNa8UR9VjM3ePhH5o4hsE5HFvuUXzR95o2N05PPzvX6viKiIuDaFk0Of30MissT32c0QEdfmJnSofY/7/u8tEZH3RCSl+SNvdIxOtO9yX17xiIizF4GbOq40UBfgZGAAsOwY9p0HDAUE+BQ417f+NOBfQKzveasQa98fgV+7/dk51T7fax2Az/DeEJgRSu0DWtbb5i7gnyHWvrOBKN/jvwJ/DbH29QKOA2YCuU7GH7Jn/Kr6NVBUf52IZIvIdBFZICLfiEjPg/cTkSy8/4HmqPfTeIUf5wy4FfiLqlb6jrHb2VYcmkPtCxgOtm883lLhro5qcKJ9qrq/3qYJuNhGh9o3Q38s6T4Hb5FHVzjUvhWqusof8Yds4j+ECcCdqjoQ+DXw9wa2aYd3noA69ecM6AGMEO/Ukf8WkUGORnv0mto+gDt8X6UniUiqc6Eekya1T0RGAttUNd/pQI9Rkz8/EXlYRLYA1wB/cDDWY9Ecv591bsB7thxImrN9jgqbydZFJBEYBrxdr8s39ijfJgpIw/s1bRDwloh09f3ldlUzte8fwEN4zxQfAp7E+x/MdU1tn4jEA7/F210QcJrp80NVfwf8TkQeBO4A/rvZgmyC5mqf771+B9QArzZPdE3XnO3zh7BJ/Hi/3exV1X71V4pIJFA3WcyHeJNf/a+Q9ecM2Aq860v080TEg7fwUiDMFNPk9qnqrnr7PQ9MczLgo9TU9mUDXYB833/M9sBCERmsqjsdjr0xmuP3s75XgU8IkMRPM7VPRMYA5wNnBMIJVz3N/fk5y62LI/5YgM7Uu/gCzAYu9z0WIOcQ+x188eUXvvW3AH/yPe4BbMF3E1yItC+r3jb3AG+E0ud30DYbcfHirkOfX/d629wJTA2x9p0DLAcy3WyX07+f+OHiruv/eA5+KK8DO4BqvGfqY/Ge8U0H8n2/QH84xL65wDJgHfBsXXIHYoApvtcWAqeHWPsmA0uBJXjPTrL81R5/tO+gbVxN/A59fu/41i/BW6SrXYi1by3ek63FvsXNUUtOtO9i33tVAruAz5yK30o2GGNMmAm3UT3GGBP2LPEbY0yYscRvjDFhxhK/McaEGUv8xhgTZizxm6AkIqV+Pt5EETm+md6r1ldBc5mIfHSkKpMikiIitzXHsY0Bm4HLBCkRKVXVxGZ8vyj9sQCYo+rHLiIvA6tV9eHDbN8ZmKaqvf0Rnwl9dsZvQoaIZIrIOyIy37cM960fLCLficgiEZktIsf51o8RkQ9F5EvgCxE5VURmishUX933V+vVSp9ZVyNdREp9xdDyRWSOiLT2rc/2PV8qIn9u5LeS7/ixiFyiiHwhIgt97zHSt81fgGzft4THfdve52vjEhH5n2b8ZzRhwBK/CSVPAeNVdRBwKTDRt34lMEJV++OtWPlIvX0GAJep6im+5/2B/wSOB7oCwxs4TgIwR1VzgK+Bm+od/ylV7cNPKzA2yFfH5Qy8d0kDVAAXq+oAvHM/POn7w/MbYJ2q9lPV+0TkbKA7MBjoBwwUkZOPdDxj6oRTkTYT+s4Ejq9XHbGlr2piMvCyiHTHW3k0ut4+n6tq/brq81R1K4CILMZbj2XWQcep4scCdguAs3yPT+TH2v+vAU8cIs4WvvduB6wAPvetF+ARXxL3+F5v3cD+Z/uWRb7niXj/EHx9iOMZ8xOW+E0oiQCGqmpF/ZUi8izwlape7Osvn1nv5bKD3qOy3uNaGv4/Uq0/Xhw71DaHc0BV+/lKRX8G3A48jbeGfiYwUFWrRWQjENfA/gI8qqrPHeVxjQGsq8eElhl4q1ICICJ1JXKT+bH07RgHjz8HbxcTwC+PtLGqluOdIvFeEYnCG+duX9I/Dejk27QESKq362fADb5vM4hIOxFp1UxtMGHAEr8JVvEisrXe8iu8STTXd8FzOd4y2gCPAY+KyCKc/Zb7n8CvRGQJ0A3Yd6QdVHUR3mqaV+GtoZ8rIkuB6/Bem0BVC4FvfcM/H1fVGXi7kr7zbTuVn/5hMOawbDinMc3E13VzQFVVRH4JXKWqI4+0nzH+Zn38xjSfgcCzvpE4ewmQaSuNOZid8RtjTJixPn5jjAkzlviNMSbMWOI3xpgwY4nfGGPCjCV+Y4wJM/8PTfj+zV+u0x8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()\n",
    "learn.recorder.plot(suggestion=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='25' class='' max='30', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      83.33% [25/30 14:12:56<2:50:35]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>bleu</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.368910</td>\n",
       "      <td>4.038377</td>\n",
       "      <td>0.632494</td>\n",
       "      <td>0.398985</td>\n",
       "      <td>33:13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.722131</td>\n",
       "      <td>3.532021</td>\n",
       "      <td>0.683106</td>\n",
       "      <td>0.438262</td>\n",
       "      <td>34:36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.272898</td>\n",
       "      <td>3.134663</td>\n",
       "      <td>0.717071</td>\n",
       "      <td>0.472063</td>\n",
       "      <td>34:34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.853325</td>\n",
       "      <td>2.827026</td>\n",
       "      <td>0.746694</td>\n",
       "      <td>0.507011</td>\n",
       "      <td>34:32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.640528</td>\n",
       "      <td>2.643712</td>\n",
       "      <td>0.769186</td>\n",
       "      <td>0.537538</td>\n",
       "      <td>35:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.502939</td>\n",
       "      <td>2.532196</td>\n",
       "      <td>0.784315</td>\n",
       "      <td>0.558068</td>\n",
       "      <td>34:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.445807</td>\n",
       "      <td>2.461506</td>\n",
       "      <td>0.794475</td>\n",
       "      <td>0.574374</td>\n",
       "      <td>34:51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.404784</td>\n",
       "      <td>2.404141</td>\n",
       "      <td>0.804026</td>\n",
       "      <td>0.587864</td>\n",
       "      <td>34:48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.327967</td>\n",
       "      <td>2.360392</td>\n",
       "      <td>0.809635</td>\n",
       "      <td>0.596664</td>\n",
       "      <td>34:42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.291917</td>\n",
       "      <td>2.336402</td>\n",
       "      <td>0.814616</td>\n",
       "      <td>0.604423</td>\n",
       "      <td>35:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.304498</td>\n",
       "      <td>2.309420</td>\n",
       "      <td>0.818315</td>\n",
       "      <td>0.610129</td>\n",
       "      <td>34:35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.207526</td>\n",
       "      <td>2.301975</td>\n",
       "      <td>0.820380</td>\n",
       "      <td>0.613838</td>\n",
       "      <td>35:05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.251492</td>\n",
       "      <td>2.283147</td>\n",
       "      <td>0.823041</td>\n",
       "      <td>0.618127</td>\n",
       "      <td>34:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.221489</td>\n",
       "      <td>2.265422</td>\n",
       "      <td>0.825980</td>\n",
       "      <td>0.622625</td>\n",
       "      <td>35:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.182941</td>\n",
       "      <td>2.256617</td>\n",
       "      <td>0.826849</td>\n",
       "      <td>0.623805</td>\n",
       "      <td>35:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>2.213764</td>\n",
       "      <td>2.247851</td>\n",
       "      <td>0.828860</td>\n",
       "      <td>0.627221</td>\n",
       "      <td>34:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>2.181971</td>\n",
       "      <td>2.239224</td>\n",
       "      <td>0.830254</td>\n",
       "      <td>0.629447</td>\n",
       "      <td>34:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>2.160002</td>\n",
       "      <td>2.227829</td>\n",
       "      <td>0.831259</td>\n",
       "      <td>0.630587</td>\n",
       "      <td>35:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>2.103742</td>\n",
       "      <td>2.225420</td>\n",
       "      <td>0.832231</td>\n",
       "      <td>0.632486</td>\n",
       "      <td>33:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>2.131206</td>\n",
       "      <td>2.216890</td>\n",
       "      <td>0.833462</td>\n",
       "      <td>0.634222</td>\n",
       "      <td>32:05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>2.139117</td>\n",
       "      <td>2.213911</td>\n",
       "      <td>0.833905</td>\n",
       "      <td>0.634894</td>\n",
       "      <td>32:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>2.084992</td>\n",
       "      <td>2.206584</td>\n",
       "      <td>0.834638</td>\n",
       "      <td>0.636454</td>\n",
       "      <td>31:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>2.095920</td>\n",
       "      <td>2.207472</td>\n",
       "      <td>0.835471</td>\n",
       "      <td>0.637932</td>\n",
       "      <td>32:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>2.125198</td>\n",
       "      <td>2.205842</td>\n",
       "      <td>0.836184</td>\n",
       "      <td>0.638846</td>\n",
       "      <td>31:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>2.107288</td>\n",
       "      <td>2.204391</td>\n",
       "      <td>0.836439</td>\n",
       "      <td>0.639427</td>\n",
       "      <td>32:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1439' class='' max='1439', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [1439/1439 02:21<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with valid_loss value: 4.038376808166504.\n",
      "Better model found at epoch 1 with valid_loss value: 3.5320210456848145.\n",
      "Better model found at epoch 2 with valid_loss value: 3.1346631050109863.\n",
      "Better model found at epoch 3 with valid_loss value: 2.8270263671875.\n",
      "Better model found at epoch 4 with valid_loss value: 2.643712282180786.\n",
      "Better model found at epoch 5 with valid_loss value: 2.532196283340454.\n",
      "Better model found at epoch 6 with valid_loss value: 2.461505651473999.\n",
      "Better model found at epoch 7 with valid_loss value: 2.4041407108306885.\n",
      "Better model found at epoch 8 with valid_loss value: 2.3603923320770264.\n",
      "Better model found at epoch 9 with valid_loss value: 2.3364017009735107.\n",
      "Better model found at epoch 10 with valid_loss value: 2.309419870376587.\n",
      "Better model found at epoch 11 with valid_loss value: 2.3019750118255615.\n",
      "Better model found at epoch 15 with valid_loss value: 2.2478508949279785.\n",
      "Better model found at epoch 17 with valid_loss value: 2.2278285026550293.\n",
      "Better model found at epoch 19 with valid_loss value: 2.2168900966644287.\n",
      "Better model found at epoch 21 with valid_loss value: 2.2065839767456055.\n",
      "Better model found at epoch 25 with valid_loss value: 2.2030985355377197.\n",
      "Epoch 25: early stopping\n"
     ]
    }
   ],
   "source": [
    "learn.fit_one_cycle(30, 1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note:** Even though this model has 12 layers and 12 heads is not significally better than the model with 6 layers and 8 heads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cualitative analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('best');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.cpu();\n",
    "model.eval();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def greedy_decode(\n",
    "        inp, \n",
    "        unk_idx=0, \n",
    "        pad_idx=1,\n",
    "        bos_idx=2, \n",
    "        no_unk=True):\n",
    "    \n",
    "    enc = model.encode(inp)\n",
    "    \n",
    "    ys = torch.ones(1, 1).fill_(bos_idx).type_as(inp.data)\n",
    "    next_word = None\n",
    "    while next_word != pad_idx and len(ys[0]) < 20:\n",
    "        out = model.decode(enc, ys)\n",
    "        if no_unk:\n",
    "            out[0][-1][unk_idx] = -float('Inf')\n",
    "            \n",
    "        next_word = out[0][-1].argmax(0)\n",
    "        next_word = next_word.data\n",
    "        ys = torch.cat([ys, \n",
    "                        torch.tensor([[next_word]]).long()\n",
    "                       ], dim=1)\n",
    "    return ys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def beam_decode(inp, bos_idx=2, unk_idx=0, pad_idx=1, beam_size=5, nucleus=0.3, no_unk=False):\n",
    "    enc = model.encode(inp)\n",
    "    \n",
    "    finised = []  # finished phrases\n",
    "    next_word = None\n",
    "    current_ys = [(inp.new_zeros([1]).long() + bos_idx, 0)]\n",
    "    while current_ys:\n",
    "        next_ys = []\n",
    "        for ys, path_score in current_ys:\n",
    "            out = model.decode(enc, ys.unsqueeze(0))\n",
    "            probs = F.softmax(out[0, -1], dim=-1)\n",
    "            \n",
    "            if no_unk:\n",
    "                probs[unk_idx] = -float('Inf')\n",
    "            \n",
    "            likely = select_nucleus(probs, p=nucleus)\n",
    "            chosen_words = random.sample(likely, k=min(beam_size, len(likely)))\n",
    "            \n",
    "            for y in chosen_words:\n",
    "                phrase = torch.cat([ys, y.unsqueeze(0).long()], dim=0)\n",
    "                \n",
    "                score = path_score + torch.log(probs[y])\n",
    "                \n",
    "                if y == pad_idx or len(phrase) > 20:\n",
    "                    finised.append((phrase, score))\n",
    "                else:\n",
    "                    next_ys.append((phrase, score))\n",
    "                \n",
    "        current_ys = sorted(next_ys, key=lambda k: k[1], reverse=True)[:beam_size]\n",
    "    return sorted(finised, key=lambda k: k[1]/len(k[0]), reverse=True)[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Let it go .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Déjalo ir .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Déjalo ir .\n",
      "==================================================\n",
      "\n",
      "i xxunk your father .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Yo le he dicho a tu padre .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Le he dicho a tu padre .\n",
      "==================================================\n",
      "\n",
      "You talking to me ?\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¿ Me hablas ?\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¿ Me hablas ?\n",
      "==================================================\n",
      "\n",
      "a xxunk taker once tried to test me . i ate his liver with some xxunk beans and a nice Xxunk .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Una vez más intentó probar probar su abrigo con algunas pastillas y un bonito\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Una vez me intentó probar una prueba de prueba , me comió su abrigo con un bonito y agradable\n",
      "==================================================\n",
      "\n",
      "i 'll be back .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Volveré . Volveré .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Volveré . Volveré .\n",
      "==================================================\n",
      "\n",
      "i see dead people .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Veo gente muerta .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Veo gente muerta .\n",
      "==================================================\n",
      "\n",
      "Houston , we have a problem .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Houston , tenemos un problema .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Houston , tenemos un problema .\n",
      "==================================================\n",
      "\n",
      "Say hello to my little friend !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¡ Dile hola a mi amigo !\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¡ Dile hola a mi pequeño amigo !\n",
      "==================================================\n",
      "\n",
      "i did not hit her ! i did not ! Oh hi , Mark .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¡ No la he hecho ! Hola , Mark .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¡ No la golpeó ! Hola , Mark .\n",
      "==================================================\n",
      "\n",
      "i xxunk king of the world !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¡ Yo soy el rey del mundo !\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¡ Soy el rey del mundo !\n",
      "==================================================\n",
      "\n",
      "This is Xxunk !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¡ Esto es\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¡ Este es Dios !\n",
      "==================================================\n",
      "\n",
      "i want to play a game .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Quiero jugar un juego .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Quiero jugar un juego .\n",
      "==================================================\n",
      "\n",
      "i have had it with these motherfucking snakes on this motherfucking plane !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¡ Lo he tenido con estas cartas de la Comunidad de Este avión !\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¡ Lo he tenido con estas ! ¡ En este avión ! ¡ Lo he tenido !\n",
      "==================================================\n",
      "\n",
      "You 're a wizard , Harry .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Eres un mentiroso , Harry .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Eres un mentiroso , Harry .\n",
      "==================================================\n",
      "\n",
      "With great power comes great responsibility .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Con gran poder viene gran responsabilidad .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Con gran poder llega gran responsabilidad .\n",
      "==================================================\n",
      "\n",
      "i do xxunk like sand . It 's all coarse , and rough , and irritating . And it gets everywhere .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Hago como arena , todo está lleno , y es duro , y se pone en todas partes\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Hago como arena , todo está lleno , y todo en todas partes , y se pone .\n",
      "==================================================\n",
      "\n",
      "Not the bees !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¡ No el título !\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¡ No el título !\n",
      "==================================================\n",
      "\n",
      "Why so serious ?\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¿ Por qué tan seria ?\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¿ Por qué tan seria ?\n",
      "==================================================\n",
      "\n",
      "i xxunk gon na steal the Declaration of Independence .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Voy a robar el País de África .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Voy a robar el Presidente de África .\n",
      "==================================================\n",
      "\n",
      "You shall not pass !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¡ No te vas !\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¡ No vas a pasar !\n",
      "==================================================\n",
      "\n",
      "Are you not entertained ?\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¿ No estás de acuerdo ?\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¿ No estás de mal humor ?\n",
      "==================================================\n",
      "\n",
      "King Kong ai xxunk got shit on me !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¡ El rey Kong Kong me ha jodido mierda !\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¡ El rey Kong Kong me tiene mierda !\n",
      "==================================================\n",
      "\n",
      "i live my life a quarter mile at a time .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Vivo mi vida de cuarto en un momento .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Vivo mi vida de cuarto en un momento .\n",
      "==================================================\n",
      "\n",
      "Water ? Like out of the toilet ?\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "¿ Como salir del agua ?\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "¿ Como salir del agua ?\n",
      "==================================================\n",
      "\n",
      "They may take our lives , but they 'll never take our freedom !\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Podrían tomar nuestras vidas , pero nunca se harán la libertad !\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Podrían tomar nuestras vidas , pero nunca harán nuestra libertad !\n",
      "==================================================\n",
      "\n",
      "i will look for you , i will find you , and i will kill you .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Te veré , te encontraré y te mataré .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Te voy a buscar , te encontraré y te mataré .\n",
      "==================================================\n",
      "\n",
      "My precious .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Mi preciosa .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Mi preciosa .\n",
      "==================================================\n",
      "\n",
      "The first rule of Fight Club is : You do not talk about Fight Club .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "La primera regla del club de Club Club no habla del club .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "La primera regla del club de Club Club no habla del club .\n",
      "==================================================\n",
      "\n",
      "You 're gon na need a bigger boat .\n",
      "~~~~~~~~~~ greedy ~~~~~~~~~~\n",
      "Vas a necesitar un barco más grande .\n",
      "~~~~~~~~~~  beam  ~~~~~~~~~~\n",
      "Vas a necesitar un barco más grande .\n",
      "==================================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer = Tokenizer()\n",
    "tok = SpacyTokenizer('en')\n",
    "\n",
    "def show_predictions():\n",
    "    for x in test_phrases:\n",
    "        x = torch.tensor([[data.x.vocab.stoi[t] for t in tokenizer.process_text(x, tok)]])\n",
    "        print(pretty(data.x.reconstruct(x[0])))\n",
    "        print('~' * 10 + ' greedy ' + '~' * 10)\n",
    "        print(pretty(data.y.reconstruct(greedy_decode(x)[0])))\n",
    "        print('~' * 10 + '  beam  ' + '~' * 10)\n",
    "        print(pretty(data.y.reconstruct(beam_decode(x, no_unk=True))))\n",
    "        print('=' * 50 + '\\n')\n",
    "show_predictions()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}