5 - Convolutional Sequence to Sequence Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 - Convolutional Sequence to Sequence Learning\n",
    "\n",
    "In this notebook we'll be implementing the [Convolutional Sequence to Sequence Learning](https://arxiv.org/abs/1705.03122) model. \n",
    "\n",
    "![](assets/convseq2seq0.png)\n",
    "\n",
    "## Introduction\n",
    "\n",
    "This model is drastically different to the previous models used in these tutorials. There are no recurrent components used at all. Instead it makes use of convolutional layers, typically used for image processing. For an introduction to convolutional layers on text for sentiment analysis, see [this](https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/4%20-%20Convolutional%20Sentiment%20Analysis.ipynb) tutorial. \n",
    "\n",
    "In short, a convolutional layer uses *filters*. These filters have a *width* (and also a *height* in images, but usually not text). If a filter has a width of 3, then it can see 3 consecutive tokens. Each convolutional layer has many of these filters (1024 in this tutorial). Each filter will slide across the sequence, from beginning to the end, looking at all 3 consectuive tokens at a time. The idea is that each of these 1024 filters will learn to extract a different feature from the text. The result of this feature extraction will then be used by the model - potentially as input to another convolutional layer. This can then all be used to extract features from the source sentence to translate it into the target language.\n",
    "\n",
    "\n",
    "## Preparing the Data\n",
    "\n",
    "First, let's import all the required modules and set the random seeds for reproducability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "\n",
    "from torchtext.datasets import Multi30k\n",
    "from torchtext.data import Field, BucketIterator\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "\n",
    "import spacy\n",
    "import numpy as np\n",
    "\n",
    "import random\n",
    "import math\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "SEED = 1234\n",
    "\n",
    "random.seed(SEED)\n",
    "np.random.seed(SEED)\n",
    "torch.manual_seed(SEED)\n",
    "torch.cuda.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll load the spaCy models and define the tokenizers for the source and target languages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "spacy_de = spacy.load('de')\n",
    "spacy_en = spacy.load('en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_de(text):\n",
    "    \"\"\"\n",
    "    Tokenizes German text from a string into a list of strings\n",
    "    \"\"\"\n",
    "    return [tok.text for tok in spacy_de.tokenizer(text)]\n",
    "\n",
    "def tokenize_en(text):\n",
    "    \"\"\"\n",
    "    Tokenizes English text from a string into a list of strings\n",
    "    \"\"\"\n",
    "    return [tok.text for tok in spacy_en.tokenizer(text)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll set up the `Field`s which decide how the data will be processed. By default RNN models in PyTorch require the sequence to be a tensor of shape **[sequence length, batch size]** so TorchText will, by default, return batches of tensors in the same shape. However in this notebook we are using CNNs which expect the batch dimension to be first. We tell TorchText to have batches be **[batch size, sequence length]** by setting `batch_first = True`.  \n",
    "\n",
    "We also append the start and end of sequence tokens as well as lowercasing all text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "SRC = Field(tokenize = tokenize_de, \n",
    "            init_token = '<sos>', \n",
    "            eos_token = '<eos>', \n",
    "            lower = True, \n",
    "            batch_first = True)\n",
    "\n",
    "TRG = Field(tokenize = tokenize_en, \n",
    "            init_token = '<sos>', \n",
    "            eos_token = '<eos>', \n",
    "            lower = True, \n",
    "            batch_first = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we load our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data, valid_data, test_data = Multi30k.splits(exts=('.de', '.en'), \n",
    "                                                    fields=(SRC, TRG))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We build our vocabulary as before, by converting any tokens that appear less than 2 times into `<unk>` tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "SRC.build_vocab(train_data, min_freq = 2)\n",
    "TRG.build_vocab(train_data, min_freq = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final bit of data preparation is defining the device and then building the iterator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 128\n",
    "\n",
    "train_iterator, valid_iterator, test_iterator = BucketIterator.splits(\n",
    "    (train_data, valid_data, test_data), \n",
    "     batch_size = BATCH_SIZE,\n",
    "     device = device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Model\n",
    "\n",
    "Next up is building the model. As before, the model is made of an encoder and decoder. The encoder *encodes* the input sentence, in the source language, into a *context vector*. The decoder *decodes* the context vector to produce the output sentence in the target language.\n",
    "\n",
    "### Encoder\n",
    "\n",
    "Previous models in these tutorials had an encoder that compresses an entire input sentence into a single context vector, $z$. The convolutional sequence-to-sequence model is a little different - it gets two context vectors for each token in the input sentence. So, if our input sentence had 6 tokens, we would get 12 context vectors, two for each token. \n",
    "\n",
    "The two context vectors per token are a *conved* vector and a *combined* vector. The conved vector is the result of each token being passed through a few layers - which we will explain shortly. The combined vector comes from the sum of the convolved vector and the embedding of that token. Both of these are returned by the encoder to be used by the decoder.\n",
    "\n",
    "The image below shows the result of an input sentence - *zwei menschen fechten.* - being passed through the encoder.\n",
    "\n",
    "![](assets/convseq2seq1.png)\n",
    "\n",
    "First, the token is passed through a *token embedding layer* - which is standard for neural networks in natural language processing. However, as there are no recurrent connections in this model it has no idea about the order of the tokens within a sequence. To rectify this we have a second embedding layer, the *positional embedding layer*. This is a standard embedding layer where the input is not the token itself but the position of the token within the sequence - starting with the first token, the `<sos>` (start of sequence) token, in position 0.\n",
    "\n",
    "Next, the token and positional embeddings are elementwise summed together to get a vector which contains information about the token and also its position with in the sequence - which we simply call the *embedding vector*. This is followed by a linear layer which transforms the embedding vector into a vector with the required hidden dimension size. \n",
    "\n",
    "The next step is to pass this hidden vector into $N$ *convolutional blocks*. This is where the \"magic\" happens in this model and we will detail the contents of the convolutional blocks shortly. After passing through the convolutional blocks, the vector is then fed through another linear layer to transform it back from the hidden dimension size into the embedding dimension size. This is our *conved* vector - and we have one of these per token in the input sequence. \n",
    "\n",
    "Finally, the conved vector is elementwise summed with the embedding vector via a residual connection to get a *combined* vector for each token. Again, there is a combined vector for each token in the input sequence.\n",
    "\n",
    "### Convolutional Blocks\n",
    "\n",
    "So, how do these convolutional blocks work? The below image shows 2 convolutional blocks with a single filter (blue) that is sliding across the tokens within the sequence. In the actual implementation we will have 10 convolutional blocks with 1024 filters in each block.\n",
    "\n",
    "![](assets/convseq2seq2.png)\n",
    "\n",
    "First, the input sentence is padded. This is because the convolutional layers will reduce the length of the input sentence and we want the length of the sentence coming into the convolutional blocks to equal the length of it coming out of the convolutional blocks. Without padding, the length of the sequence coming out of a convolutional layer will be `filter_size - 1` shorter than the sequence entering the convolutional layer. For example, if we had a filter size of 3, the sequence will be 2 elements shorter. Thus, we pad the sentence with one padding element on each side. We can calculate the amount of padding on each side by simply doing `(filter_size - 1)/2` for odd sized filters - we will not cover even sized filters in this tutorial.\n",
    "\n",
    "These filters are designed so the output hidden dimension of them is twice the input hidden dimension. In computer vision terminology these hidden dimensions are called *channels* - but we will stick to calling them hidden dimensions. Why do we double the size of the hidden dimension leaving the convolutional filter? This is because we are using a special activation function called *gated linear units* (GLU). GLUs have gating mechanisms (similar to LSTMs and GRUs) contained within the activation function and actually half the size of the hidden dimension - whereas usually activation functions keep the hidden dimensions the same size.\n",
    "\n",
    "After passing through the GLU activation the hidden dimension size for each token is the same as it was when it entered the convolutional blocks. It is now elementwise summed with its own vector before it was passed through the convolutional layer. \n",
    "\n",
    "This concludes a single convolutional block. Subsequent blocks take the output of the previous block and perform the same steps. Each block has their own parameters, they are not shared between blocks. The output of the last block goes back to the main encoder - where it is fed through a linear layer to get the conved output and then elementwise summed with the embedding of the token to get the combined output.\n",
    "\n",
    "### Encoder Implementation\n",
    "\n",
    "To keep the implementation simple, we only allow for odd sized kernels. This allows padding to be added equally to both sides of the source sequence.\n",
    "\n",
    "The `scale` variable is used by the authors to \"ensure that the variance throughout the network does not change dramatically\". The performance of the model seems to vary wildly using different seeds if this is not used.\n",
    "\n",
    "The positional embedding is initialized to have a \"vocabulary\" of 100. This means it can handle sequences up to 100 elements long, indexed from 0 to 99. This can be increased if used on a dataset with longer sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    def __init__(self, \n",
    "                 input_dim, \n",
    "                 emb_dim, \n",
    "                 hid_dim, \n",
    "                 n_layers, \n",
    "                 kernel_size, \n",
    "                 dropout, \n",
    "                 device,\n",
    "                 max_length = 100):\n",
    "        super().__init__()\n",
    "        \n",
    "        assert kernel_size % 2 == 1, \"Kernel size must be odd!\"\n",
    "        \n",
    "        self.device = device\n",
    "        \n",
    "        self.scale = torch.sqrt(torch.FloatTensor([0.5])).to(device)\n",
    "        \n",
    "        self.tok_embedding = nn.Embedding(input_dim, emb_dim)\n",
    "        self.pos_embedding = nn.Embedding(max_length, emb_dim)\n",
    "        \n",
    "        self.emb2hid = nn.Linear(emb_dim, hid_dim)\n",
    "        self.hid2emb = nn.Linear(hid_dim, emb_dim)\n",
    "        \n",
    "        self.convs = nn.ModuleList([nn.Conv1d(in_channels = hid_dim, \n",
    "                                              out_channels = 2 * hid_dim, \n",
    "                                              kernel_size = kernel_size, \n",
    "                                              padding = (kernel_size - 1) // 2)\n",
    "                                    for _ in range(n_layers)])\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, src):\n",
    "        \n",
    "        #src = [batch size, src len]\n",
    "        \n",
    "        batch_size = src.shape[0]\n",
    "        src_len = src.shape[1]\n",
    "        \n",
    "        #create position tensor\n",
    "        pos = torch.arange(0, src_len).unsqueeze(0).repeat(batch_size, 1).to(self.device)\n",
    "        \n",
    "        #pos = [0, 1, 2, 3, ..., src len - 1]\n",
    "        \n",
    "        #pos = [batch size, src len]\n",
    "        \n",
    "        #embed tokens and positions\n",
    "        tok_embedded = self.tok_embedding(src)\n",
    "        pos_embedded = self.pos_embedding(pos)\n",
    "        \n",
    "        #tok_embedded = pos_embedded = [batch size, src len, emb dim]\n",
    "        \n",
    "        #combine embeddings by elementwise summing\n",
    "        embedded = self.dropout(tok_embedded + pos_embedded)\n",
    "        \n",
    "        #embedded = [batch size, src len, emb dim]\n",
    "        \n",
    "        #pass embedded through linear layer to convert from emb dim to hid dim\n",
    "        conv_input = self.emb2hid(embedded)\n",
    "        \n",
    "        #conv_input = [batch size, src len, hid dim]\n",
    "        \n",
    "        #permute for convolutional layer\n",
    "        conv_input = conv_input.permute(0, 2, 1) \n",
    "        \n",
    "        #conv_input = [batch size, hid dim, src len]\n",
    "        \n",
    "        #begin convolutional blocks...\n",
    "        \n",
    "        for i, conv in enumerate(self.convs):\n",
    "        \n",
    "            #pass through convolutional layer\n",
    "            conved = conv(self.dropout(conv_input))\n",
    "\n",
    "            #conved = [batch size, 2 * hid dim, src len]\n",
    "\n",
    "            #pass through GLU activation function\n",
    "            conved = F.glu(conved, dim = 1)\n",
    "\n",
    "            #conved = [batch size, hid dim, src len]\n",
    "            \n",
    "            #apply residual connection\n",
    "            conved = (conved + conv_input) * self.scale\n",
    "\n",
    "            #conved = [batch size, hid dim, src len]\n",
    "            \n",
    "            #set conv_input to conved for next loop iteration\n",
    "            conv_input = conved\n",
    "        \n",
    "        #...end convolutional blocks\n",
    "        \n",
    "        #permute and convert back to emb dim\n",
    "        conved = self.hid2emb(conved.permute(0, 2, 1))\n",
    "        \n",
    "        #conved = [batch size, src len, emb dim]\n",
    "        \n",
    "        #elementwise sum output (conved) and input (embedded) to be used for attention\n",
    "        combined = (conved + embedded) * self.scale\n",
    "        \n",
    "        #combined = [batch size, src len, emb dim]\n",
    "        \n",
    "        return conved, combined"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoder\n",
    "\n",
    "The decoder takes in the actual target sentence and tries to predict it. This model differs from the recurrent neural network models previously detailed in these tutorials as it predicts all tokens within the target sentence in parallel. There is no sequential processing, i.e. no decoding loop. This will be detailed further later on in the tutorials.\n",
    "\n",
    "The decoder is similar to the encoder, with a few changes to both the main model and the convolutional blocks inside the model.\n",
    "\n",
    "![](assets/convseq2seq3.png)\n",
    "\n",
    "First, the embeddings do not have a residual connection that connects after the convolutional blocks and the transformation. Instead the embeddings are fed into the convolutional blocks to be used as residual connections there.\n",
    "\n",
    "Second, to feed the decoder information from the encoder, the encoder conved and combined outputs are used - again, within the convolutional blocks. \n",
    "\n",
    "Finally, the output of the decoder is a linear layer from embedding dimension to output dimension. This is used make a prediction about what the next word in the translation should be.\n",
    "\n",
    "### Decoder Convolutional Blocks\n",
    "\n",
    "Again, these are similar to the convolutional blocks within the encoder, with a few changes.\n",
    "\n",
    "![](assets/convseq2seq4.png)\n",
    "\n",
    "First, the padding. Instead of padding equally on each side to ensure the length of the sentence stays the same throughout, we only pad at the beginning of the sentence. As we are processing all of the targets simultaneously in parallel, and not sequentially, we need a method of only allowing the filters translating token $i$ to only look at tokens before word $i$. If they were allowed to look at token $i+1$ (the token they should be outputting), the model will simply learn to output the next word in the sequence by directly copying it, without actually learning how to translate.\n",
    "\n",
    "Let's see what happens if we **incorrectly** padded equally on each side, like we do in the encoder.\n",
    "\n",
    "![](assets/convseq2seq5.png)\n",
    "\n",
    "The filter at the first position, which is trying use the first word in the sequence, `<sos>` to predict the second word, `two`, can now directly see the word `two`. This is the same for every position, the word the model trying to predict is the second element covered by the filter. Thus, the filters can learn to simply copy the second word at each position allowing for perfect translation without actually learning how to translate.\n",
    "\n",
    "Second, after the GLU activation and before the residual connection, the block calculates and applies attention - using the encoded representations and the embedding of the current word. **Note**: we only show the connections to the rightmost token, but they are actually connected to all tokens - this was done for clarity. Each token input uses their own, and only their own, embedding for their own attention calculation.\n",
    "\n",
    "The attention is calculated by first using a linear layer to change the hidden dimension to the same size as the embedding dimension. Then the embedding summed via a residual connection. This combination then has the standard attention calculation applied by finding how much it \"matches\" with the *encoded conved* and then this is applied by getting a weighted sum over the *encoded combined*. This is then projected back up to the hidden dimenson size and a residual connection to the initial input to the attention layer is applied.\n",
    "\n",
    "Why do they calculate attention first with the encoded conved and then use it to calculate the weighted sum over the encoded combined? The paper argues that the encoded conved is good for getting a larger context over the encoded sequence, whereas the encoded combined has more information about the specific token and is thus therefore more useful for makng a prediction.\n",
    "\n",
    "### Decoder Impementation\n",
    "\n",
    "As we only pad on one side the decoder is allowed to use both odd and even sized padding. Again, the `scale` is used to reduce variance throughout the model and the position embedding is initialized to have a \"vocabulary\" of 100.\n",
    "\n",
    "This model takes in the encoder representations in its `forward` method and both are passed to the `calculate_attention` method which calculates and applies attention. It also returns the actual attention values, but we are not currently using them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "    def __init__(self, \n",
    "                 output_dim, \n",
    "                 emb_dim, \n",
    "                 hid_dim, \n",
    "                 n_layers, \n",
    "                 kernel_size, \n",
    "                 dropout, \n",
    "                 trg_pad_idx, \n",
    "                 device,\n",
    "                 max_length = 100):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.kernel_size = kernel_size\n",
    "        self.trg_pad_idx = trg_pad_idx\n",
    "        self.device = device\n",
    "        \n",
    "        self.scale = torch.sqrt(torch.FloatTensor([0.5])).to(device)\n",
    "        \n",
    "        self.tok_embedding = nn.Embedding(output_dim, emb_dim)\n",
    "        self.pos_embedding = nn.Embedding(max_length, emb_dim)\n",
    "        \n",
    "        self.emb2hid = nn.Linear(emb_dim, hid_dim)\n",
    "        self.hid2emb = nn.Linear(hid_dim, emb_dim)\n",
    "        \n",
    "        self.attn_hid2emb = nn.Linear(hid_dim, emb_dim)\n",
    "        self.attn_emb2hid = nn.Linear(emb_dim, hid_dim)\n",
    "        \n",
    "        self.fc_out = nn.Linear(emb_dim, output_dim)\n",
    "        \n",
    "        self.convs = nn.ModuleList([nn.Conv1d(in_channels = hid_dim, \n",
    "                                              out_channels = 2 * hid_dim, \n",
    "                                              kernel_size = kernel_size)\n",
    "                                    for _ in range(n_layers)])\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "      \n",
    "    def calculate_attention(self, embedded, conved, encoder_conved, encoder_combined):\n",
    "        \n",
    "        #embedded = [batch size, trg len, emb dim]\n",
    "        #conved = [batch size, hid dim, trg len]\n",
    "        #encoder_conved = encoder_combined = [batch size, src len, emb dim]\n",
    "        \n",
    "        #permute and convert back to emb dim\n",
    "        conved_emb = self.attn_hid2emb(conved.permute(0, 2, 1))\n",
    "        \n",
    "        #conved_emb = [batch size, trg len, emb dim]\n",
    "        \n",
    "        combined = (conved_emb + embedded) * self.scale\n",
    "        \n",
    "        #combined = [batch size, trg len, emb dim]\n",
    "                \n",
    "        energy = torch.matmul(combined, encoder_conved.permute(0, 2, 1))\n",
    "        \n",
    "        #energy = [batch size, trg len, src len]\n",
    "        \n",
    "        attention = F.softmax(energy, dim=2)\n",
    "        \n",
    "        #attention = [batch size, trg len, src len]\n",
    "            \n",
    "        attended_encoding = torch.matmul(attention, encoder_combined)\n",
    "        \n",
    "        #attended_encoding = [batch size, trg len, emd dim]\n",
    "        \n",
    "        #convert from emb dim -> hid dim\n",
    "        attended_encoding = self.attn_emb2hid(attended_encoding)\n",
    "        \n",
    "        #attended_encoding = [batch size, trg len, hid dim]\n",
    "        \n",
    "        #apply residual connection\n",
    "        attended_combined = (conved + attended_encoding.permute(0, 2, 1)) * self.scale\n",
    "        \n",
    "        #attended_combined = [batch size, hid dim, trg len]\n",
    "        \n",
    "        return attention, attended_combined\n",
    "        \n",
    "    def forward(self, trg, encoder_conved, encoder_combined):\n",
    "        \n",
    "        #trg = [batch size, trg len]\n",
    "        #encoder_conved = encoder_combined = [batch size, src len, emb dim]\n",
    "                \n",
    "        batch_size = trg.shape[0]\n",
    "        trg_len = trg.shape[1]\n",
    "            \n",
    "        #create position tensor\n",
    "        pos = torch.arange(0, trg_len).unsqueeze(0).repeat(batch_size, 1).to(self.device)\n",
    "        \n",
    "        #pos = [batch size, trg len]\n",
    "        \n",
    "        #embed tokens and positions\n",
    "        tok_embedded = self.tok_embedding(trg)\n",
    "        pos_embedded = self.pos_embedding(pos)\n",
    "        \n",
    "        #tok_embedded = [batch size, trg len, emb dim]\n",
    "        #pos_embedded = [batch size, trg len, emb dim]\n",
    "        \n",
    "        #combine embeddings by elementwise summing\n",
    "        embedded = self.dropout(tok_embedded + pos_embedded)\n",
    "        \n",
    "        #embedded = [batch size, trg len, emb dim]\n",
    "        \n",
    "        #pass embedded through linear layer to go through emb dim -> hid dim\n",
    "        conv_input = self.emb2hid(embedded)\n",
    "        \n",
    "        #conv_input = [batch size, trg len, hid dim]\n",
    "        \n",
    "        #permute for convolutional layer\n",
    "        conv_input = conv_input.permute(0, 2, 1) \n",
    "        \n",
    "        #conv_input = [batch size, hid dim, trg len]\n",
    "        \n",
    "        batch_size = conv_input.shape[0]\n",
    "        hid_dim = conv_input.shape[1]\n",
    "        \n",
    "        for i, conv in enumerate(self.convs):\n",
    "        \n",
    "            #apply dropout\n",
    "            conv_input = self.dropout(conv_input)\n",
    "        \n",
    "            #need to pad so decoder can't \"cheat\"\n",
    "            padding = torch.zeros(batch_size, \n",
    "                                  hid_dim, \n",
    "                                  self.kernel_size - 1).fill_(self.trg_pad_idx).to(self.device)\n",
    "                \n",
    "            padded_conv_input = torch.cat((padding, conv_input), dim = 2)\n",
    "        \n",
    "            #padded_conv_input = [batch size, hid dim, trg len + kernel size - 1]\n",
    "        \n",
    "            #pass through convolutional layer\n",
    "            conved = conv(padded_conv_input)\n",
    "\n",
    "            #conved = [batch size, 2 * hid dim, trg len]\n",
    "            \n",
    "            #pass through GLU activation function\n",
    "            conved = F.glu(conved, dim = 1)\n",
    "\n",
    "            #conved = [batch size, hid dim, trg len]\n",
    "            \n",
    "            #calculate attention\n",
    "            attention, conved = self.calculate_attention(embedded, \n",
    "                                                         conved, \n",
    "                                                         encoder_conved, \n",
    "                                                         encoder_combined)\n",
    "            \n",
    "            #attention = [batch size, trg len, src len]\n",
    "            \n",
    "            #apply residual connection\n",
    "            conved = (conved + conv_input) * self.scale\n",
    "            \n",
    "            #conved = [batch size, hid dim, trg len]\n",
    "            \n",
    "            #set conv_input to conved for next loop iteration\n",
    "            conv_input = conved\n",
    "            \n",
    "        conved = self.hid2emb(conved.permute(0, 2, 1))\n",
    "         \n",
    "        #conved = [batch size, trg len, emb dim]\n",
    "            \n",
    "        output = self.fc_out(self.dropout(conved))\n",
    "        \n",
    "        #output = [batch size, trg len, output dim]\n",
    "            \n",
    "        return output, attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seq2Seq\n",
    "\n",
    "The encapsulating `Seq2Seq` module is a lot different from recurrent neural network methods used in previous notebooks, especially in the decoding. \n",
    "\n",
    "Our `trg` has the `<eos>` element sliced off of the end of the sequence. This is because we do not input the `<eos>` token into the decoder.\n",
    "\n",
    "The encoding is similar, insert the source sequence and receive a \"context vector\". However, here we have two context vectors per word in the source sequence, `encoder_conved` and `encoder_combined`. \n",
    "\n",
    "As the decoding is done in parallel we do not need a decoding loop. All of the target sequence is input into the decoder at once and the padding is used to ensure each convolutional filter in the decoder can only see the current and previous tokens in the sequence as it slides across the sentence.\n",
    "\n",
    "This also, however, means we cannot do teacher forcing using this model. We do not have a loop in which we can choose whether to input the predicted token or the actual token in the sequence as everything is predicted in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2Seq(nn.Module):\n",
    "    def __init__(self, encoder, decoder):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        \n",
    "    def forward(self, src, trg):\n",
    "        \n",
    "        #src = [batch size, src len]\n",
    "        #trg = [batch size, trg len - 1] (<eos> token sliced off the end)\n",
    "           \n",
    "        #calculate z^u (encoder_conved) and (z^u + e) (encoder_combined)\n",
    "        #encoder_conved is output from final encoder conv. block\n",
    "        #encoder_combined is encoder_conved plus (elementwise) src embedding plus \n",
    "        #  positional embeddings \n",
    "        encoder_conved, encoder_combined = self.encoder(src)\n",
    "            \n",
    "        #encoder_conved = [batch size, src len, emb dim]\n",
    "        #encoder_combined = [batch size, src len, emb dim]\n",
    "        \n",
    "        #calculate predictions of next words\n",
    "        #output is a batch of predictions for each word in the trg sentence\n",
    "        #attention a batch of attention scores across the src sentence for \n",
    "        #  each word in the trg sentence\n",
    "        output, attention = self.decoder(trg, encoder_conved, encoder_combined)\n",
    "        \n",
    "        #output = [batch size, trg len - 1, output dim]\n",
    "        #attention = [batch size, trg len - 1, src len]\n",
    "        \n",
    "        return output, attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the Seq2Seq Model\n",
    "\n",
    "The rest of the tutorial is similar to all of the previous ones. We define all of the hyperparameters, initialize the encoder and decoder, and initialize the overall model - placing it on the GPU if we have one.\n",
    "\n",
    "In the paper they find that it is more beneficial to use a small filter (kernel size of 3) and a high number of layers (5+)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_DIM = len(SRC.vocab)\n",
    "OUTPUT_DIM = len(TRG.vocab)\n",
    "EMB_DIM = 256\n",
    "HID_DIM = 512 # each conv. layer has 2 * hid_dim filters\n",
    "ENC_LAYERS = 10 # number of conv. blocks in encoder\n",
    "DEC_LAYERS = 10 # number of conv. blocks in decoder\n",
    "ENC_KERNEL_SIZE = 3 # must be odd!\n",
    "DEC_KERNEL_SIZE = 3 # can be even or odd\n",
    "ENC_DROPOUT = 0.25\n",
    "DEC_DROPOUT = 0.25\n",
    "TRG_PAD_IDX = TRG.vocab.stoi[TRG.pad_token]\n",
    "    \n",
    "enc = Encoder(INPUT_DIM, EMB_DIM, HID_DIM, ENC_LAYERS, ENC_KERNEL_SIZE, ENC_DROPOUT, device)\n",
    "dec = Decoder(OUTPUT_DIM, EMB_DIM, HID_DIM, DEC_LAYERS, DEC_KERNEL_SIZE, DEC_DROPOUT, TRG_PAD_IDX, device)\n",
    "\n",
    "model = Seq2Seq(enc, dec).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also see that the model has almost twice as many parameters as the attention based model (20m to 37m)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 37,351,685 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "\n",
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define the optimizer and the loss function (criterion). As before we ignore the loss where the target sequence is a padding token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss(ignore_index = TRG_PAD_IDX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we define the training loop for the model.\n",
    "\n",
    "We handle the sequences a little differently than previous tutorials. For all models we never put the `<eos>` into the decoder. This is handled in the RNN models by the having the decoder loop not reach having the `<eos>` as an input to the decoder. In this model, we simply slice the `<eos>` token off the end of the sequence. Thus:\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\text{trg} &= [sos, x_1, x_2, x_3, eos]\\\\\n",
    "\\text{trg[:-1]} &= [sos, x_1, x_2, x_3]\n",
    "\\end{align*}$$\n",
    "\n",
    "$x_i$ denotes actual target sequence element. We then feed this into the model to get a predicted sequence that should hopefully predict the `<eos>` token:\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\text{output} &= [y_1, y_2, y_3, eos]\n",
    "\\end{align*}$$\n",
    "\n",
    "$y_i$ denotes predicted target sequence element. We then calculate our loss using the original `trg` tensor with the `<sos>` token sliced off the front, leaving the `<eos>` token:\n",
    "\n",
    "$$\\begin{align*}\n",
    "\\text{output} &= [y_1, y_2, y_3, eos]\\\\\n",
    "\\text{trg[1:]} &= [x_1, x_2, x_3, eos]\n",
    "\\end{align*}$$\n",
    "\n",
    "We then calculate our losses and update our parameters as is standard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion, clip):\n",
    "    \n",
    "    model.train()\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    \n",
    "    for i, batch in enumerate(iterator):\n",
    "        \n",
    "        src = batch.src\n",
    "        trg = batch.trg\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        output, _ = model(src, trg[:,:-1])\n",
    "        \n",
    "        #output = [batch size, trg len - 1, output dim]\n",
    "        #trg = [batch size, trg len]\n",
    "        \n",
    "        output_dim = output.shape[-1]\n",
    "        \n",
    "        output = output.contiguous().view(-1, output_dim)\n",
    "        trg = trg[:,1:].contiguous().view(-1)\n",
    "        \n",
    "        #output = [batch size * trg len - 1, output dim]\n",
    "        #trg = [batch size * trg len - 1]\n",
    "        \n",
    "        loss = criterion(output, trg)\n",
    "        \n",
    "        loss.backward()\n",
    "        \n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), clip)\n",
    "        \n",
    "        optimizer.step()\n",
    "        \n",
    "        epoch_loss += loss.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The evaluation loop is the same as the training loop, just without the gradient calculations and parameter updates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, iterator, criterion):\n",
    "    \n",
    "    model.eval()\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    \n",
    "    with torch.no_grad():\n",
    "    \n",
    "        for i, batch in enumerate(iterator):\n",
    "\n",
    "            src = batch.src\n",
    "            trg = batch.trg\n",
    "\n",
    "            output, _ = model(src, trg[:,:-1])\n",
    "        \n",
    "            #output = [batch size, trg len - 1, output dim]\n",
    "            #trg = [batch size, trg len]\n",
    "\n",
    "            output_dim = output.shape[-1]\n",
    "            \n",
    "            output = output.contiguous().view(-1, output_dim)\n",
    "            trg = trg[:,1:].contiguous().view(-1)\n",
    "\n",
    "            #output = [batch size * trg len - 1, output dim]\n",
    "            #trg = [batch size * trg len - 1]\n",
    "            \n",
    "            loss = criterion(output, trg)\n",
    "\n",
    "            epoch_loss += loss.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we have a function that tells us how long each epoch takes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we train our model. Notice we have reduced the `CLIP` value from 1 to 0.1 in order to train this model more reliably. With higher `CLIP` values, the gradient occasionally explodes.\n",
    "\n",
    "Although we have almost twice as many parameters as the attention based RNN model, it actually takes around half the time as the standard version and about the same time as the packed padded sequences version. This is due to all calculations being done in parallel using the convolutional filters instead of sequentially using RNNs. \n",
    "\n",
    "**Note**: this model always has a teacher forcing ratio of 1, i.e. it will always use the ground truth next token from the target sequence. This means we cannot compare perplexity values against the previous models when they are using a teacher forcing ratio that is not 1. See [here](https://github.com/bentrevett/pytorch-seq2seq/issues/39#issuecomment-529408483) for the results of the attention based RNN using a teacher forcing ratio of 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Time: 0m 30s\n",
      "\tTrain Loss: 4.240 | Train PPL:  69.408\n",
      "\t Val. Loss: 2.994 |  Val. PPL:  19.961\n",
      "Epoch: 02 | Time: 0m 30s\n",
      "\tTrain Loss: 3.043 | Train PPL:  20.971\n",
      "\t Val. Loss: 2.379 |  Val. PPL:  10.798\n",
      "Epoch: 03 | Time: 0m 30s\n",
      "\tTrain Loss: 2.604 | Train PPL:  13.521\n",
      "\t Val. Loss: 2.124 |  Val. PPL:   8.361\n",
      "Epoch: 04 | Time: 0m 30s\n",
      "\tTrain Loss: 2.369 | Train PPL:  10.685\n",
      "\t Val. Loss: 1.991 |  Val. PPL:   7.323\n",
      "Epoch: 05 | Time: 0m 30s\n",
      "\tTrain Loss: 2.209 | Train PPL:   9.107\n",
      "\t Val. Loss: 1.908 |  Val. PPL:   6.737\n",
      "Epoch: 06 | Time: 0m 30s\n",
      "\tTrain Loss: 2.097 | Train PPL:   8.139\n",
      "\t Val. Loss: 1.864 |  Val. PPL:   6.448\n",
      "Epoch: 07 | Time: 0m 30s\n",
      "\tTrain Loss: 2.009 | Train PPL:   7.456\n",
      "\t Val. Loss: 1.810 |  Val. PPL:   6.110\n",
      "Epoch: 08 | Time: 0m 31s\n",
      "\tTrain Loss: 1.932 | Train PPL:   6.904\n",
      "\t Val. Loss: 1.779 |  Val. PPL:   5.922\n",
      "Epoch: 09 | Time: 0m 30s\n",
      "\tTrain Loss: 1.868 | Train PPL:   6.474\n",
      "\t Val. Loss: 1.762 |  Val. PPL:   5.825\n",
      "Epoch: 10 | Time: 0m 30s\n",
      "\tTrain Loss: 1.817 | Train PPL:   6.156\n",
      "\t Val. Loss: 1.736 |  Val. PPL:   5.674\n"
     ]
    }
   ],
   "source": [
    "N_EPOCHS = 10\n",
    "CLIP = 0.1\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "    \n",
    "    start_time = time.time()\n",
    "    \n",
    "    train_loss = train(model, train_iterator, optimizer, criterion, CLIP)\n",
    "    valid_loss = evaluate(model, valid_iterator, criterion)\n",
    "    \n",
    "    end_time = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'tut5-model.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {math.exp(valid_loss):7.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then load the parameters which obtained the lowest validation loss and calculate the loss over the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| Test Loss: 1.795 | Test PPL:   6.022 |\n"
     ]
    }
   ],
   "source": [
    "model.load_state_dict(torch.load('tut5-model.pt'))\n",
    "\n",
    "test_loss = evaluate(model, test_iterator, criterion)\n",
    "\n",
    "print(f'| Test Loss: {test_loss:.3f} | Test PPL: {math.exp(test_loss):7.3f} |')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can can translations from our model with the `translate_sentence` function below.\n",
    "\n",
    "The steps taken are:\n",
    "- tokenize the source sentence if it has not been tokenized (is a string)\n",
    "- append the `<sos>` and `<eos>` tokens\n",
    "- numericalize the source sentence\n",
    "- convert it to a tensor and add a batch dimension\n",
    "- feed the source sentence into the encoder\n",
    "- create a list to hold the output sentence, initialized with an `<sos>` token\n",
    "- while we have not hit a maximum length\n",
    "  - convert the current output sentence prediction into a tensor with a batch dimension\n",
    "  - place the current output and the two encoder outputs into the decoder\n",
    "  - get next output token prediction from decoder\n",
    "  - add prediction to current output sentence prediction\n",
    "  - break if the prediction was an `<eos>` token\n",
    "- convert the output sentence from indexes to tokens\n",
    "- return the output sentence (with the `<sos>` token removed) and the attention from the last layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def translate_sentence(sentence, src_field, trg_field, model, device, max_len = 50):\n",
    "\n",
    "    model.eval()\n",
    "        \n",
    "    if isinstance(sentence, str):\n",
    "        nlp = spacy.load('de')\n",
    "        tokens = [token.text.lower() for token in nlp(sentence)]\n",
    "    else:\n",
    "        tokens = [token.lower() for token in sentence]\n",
    "\n",
    "    tokens = [src_field.init_token] + tokens + [src_field.eos_token]\n",
    "        \n",
    "    src_indexes = [src_field.vocab.stoi[token] for token in tokens]\n",
    "\n",
    "    src_tensor = torch.LongTensor(src_indexes).unsqueeze(0).to(device)\n",
    "\n",
    "    with torch.no_grad():\n",
    "        encoder_conved, encoder_combined = model.encoder(src_tensor)\n",
    "\n",
    "    trg_indexes = [trg_field.vocab.stoi[trg_field.init_token]]\n",
    "\n",
    "    for i in range(max_len):\n",
    "\n",
    "        trg_tensor = torch.LongTensor(trg_indexes).unsqueeze(0).to(device)\n",
    "\n",
    "        with torch.no_grad():\n",
    "            output, attention = model.decoder(trg_tensor, encoder_conved, encoder_combined)\n",
    "        \n",
    "        pred_token = output.argmax(2)[:,-1].item()\n",
    "        \n",
    "        trg_indexes.append(pred_token)\n",
    "\n",
    "        if pred_token == trg_field.vocab.stoi[trg_field.eos_token]:\n",
    "            break\n",
    "    \n",
    "    trg_tokens = [trg_field.vocab.itos[i] for i in trg_indexes]\n",
    "    \n",
    "    return trg_tokens[1:], attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we have a function what will display how much the model pays attention to each input token during each step of the decoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_attention(sentence, translation, attention):\n",
    "    \n",
    "    fig = plt.figure(figsize=(10,10))\n",
    "    ax = fig.add_subplot(111)\n",
    "        \n",
    "    attention = attention.squeeze(0).cpu().detach().numpy()\n",
    "    \n",
    "    cax = ax.matshow(attention, cmap='bone')\n",
    "   \n",
    "    ax.tick_params(labelsize=15)\n",
    "    ax.set_xticklabels(['']+['<sos>']+[t.lower() for t in sentence]+['<eos>'], \n",
    "                       rotation=45)\n",
    "    ax.set_yticklabels(['']+translation)\n",
    "\n",
    "    ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "    ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "\n",
    "    plt.show()\n",
    "    plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we'll finally start translating some sentences. **Note**: these sentences have been cherry picked.\n",
    "\n",
    "First, we'll get an example from the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "src = ['ein', 'kleines', 'mädchen', 'klettert', 'in', 'ein', 'spielhaus', 'aus', 'holz', '.']\n",
      "trg = ['a', 'little', 'girl', 'climbing', 'into', 'a', 'wooden', 'playhouse', '.']\n"
     ]
    }
   ],
   "source": [
    "example_idx = 2\n",
    "\n",
    "src = vars(train_data.examples[example_idx])['src']\n",
    "trg = vars(train_data.examples[example_idx])['trg']\n",
    "\n",
    "print(f'src = {src}')\n",
    "print(f'trg = {trg}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we pass it into our `translate_sentence` function which gives us the predicted translation tokens as well as the attention.\n",
    "\n",
    "We can see that it doesn't give the exact same translation, however it does capture the same meaning as the original. It is actual a more literal translation as *aus holz* literally translates to *of wood*, so a *wooden playhouse* is the same thing as a *playhouse made of wood*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted trg = ['a', 'little', 'girl', 'is', 'climbing', 'into', 'a', 'playhouse', 'made', 'of', 'wood', '.', '<eos>']\n"
     ]
    }
   ],
   "source": [
    "translation, attention = translate_sentence(src, SRC, TRG, model, device)\n",
    "\n",
    "print(f'predicted trg = {translation}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can view the attention of the model, making sure it gives sensibile looking results.\n",
    "\n",
    "We can see it correctly pays attention to *aus* when translation both *made* and *of*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlsAAAJ1CAYAAADuYzU3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdd5gkVdn38e/NLjkjUaISxAgKiogBxQSiiIqYXkUfXQM8iI9gQMUABhAVM6ICKgoiimAmCCooyqKCgqCgCCJJQMICC7t7v3+c02xtM8vO7E5Ndc98P9dV10xXV3efrq6u+vU5p05FZiJJkqR2LNV1ASRJkiYzw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRp4ETEtPp3qd7/0rAybEmSBkpETM/MuRGxIvAZ4CURsVrX5ZIW1/SuCyCpGxGxVGbO67ocUlPdLudExMrAOcA9wMXArG5LJi2+8ELU0tQTEdMyc279fxvgVuD2zLw+IiLdMahDEbEMcBYwG5gBXJWZ93RbKmnxWbMlTTE1TPWC1teBZwArAjMj4sOZebaBSx17JLAqsFdmXg4QEU8DdgHmAOdm5o87LJ80JoYtaQrpq9HaH9gBeDewIfAs4PiIeF1m/sTApYnS3C6re4CNgUdExK2U2q3/A/4IrAU8PSKuycwLJ7600tgZtqQppBG0Hk85mH06M79R5/0CeAdwXES8ysCliVI7w68AbAXMBK4AjqR0jr+5LvbKzDwhIp4JnAqs3UlhpcXg2YjSFBMR+wK/BZ4P/KM3PzPPBT5C6ZR8XEQ8JzMzIqKbkmqK+TRl29sxM+8GPga8CNgbeFYNWgHcQtluPblDQ8MO8tIUFBHHAq8GvgK8MzNvady3HfBO4IXAszPzjE4KqSklIlYHTqT019oLOLPZtBgRywNbAF+kNDM+w7NpNSys2ZImsYUNBpmZewHfBl4DvLSeZt+777fAJ4DjgasnoJiaYvq3yzqu1i3Ai4FLga8Bz4yI6fX+1Sh9C4+hHLeelZnzIsJjmIaCNVvSJNXXGX43YF1K88u/MvOSOv9k4NmUzsffyszbG49frjbnSOOu9tH6n8z8bL09vY6vtQrwfeBhwOuAMyk1Wq8G5gLvr8tNz8w5HRVfGhPDljQJNTu2R8R3gKcCdwNrAucDJ2Xm5xr37wLsB3w7M2/rptRF/5lpdtKfnCLiQOBA4IjMfG+d1wtcawM/A5YH9s3M0yJi+cy8qy7Xf/aiNNCsgpUmoUbQ+hCwDbAn8HBgS+A24DP1rC4ycw/gh8CXgBd10SE+IqZFxFYRsUqjNm735nvRpPNN4Fjg5RHxEYAatJbOzBuAEyg1Wt+PiCf0glZdzqCloeLQD9ISGIJL3mwL/BT4bWbeVZtodqT0ffl1r/yZuWdE3AWc11G42ZwS9k4DDoqIHwE7RsRDM/P6DsqjcTRSTVRm/jMiDgOmAS+LCDLzwMy8ty4yD/hovf+CiS2xNL4MW9Ji6usT9eDM/HfXZeqpHYdXpNRm/aYGrYcD5wI/Af43M++MiFdFxF8z83e103xX/gp8CzgiIvYAVgCeatAafo2mweWBpwGbAb8HLsvMqyLiY3XRPeuFpw+k9Nd6HvC9zPxMfR6bDjW07LMlLYa+oPVZSlPdvpk5s6PyNPtoNf8/jnLgeitwCuV6c/+TmbfXIR4OpAwe+dOumuv61uXfKYOtHktZn158eIj1Ptt6tuvZlB/46wDXUAL2/pl5TURsBLwFeAPlR8ItwHXA4+0Er8nAPlvSGPVdW/Ak4DnAd4D/dlSepRvhahoL1lj/AOgd6H6TmS+tQWsNyiVQHgxc1GHQaq7LlwMXUga33At4V3NICg2fGrSWpzQP3wK8KDPXBZYBdgW+GhEbZuZVwKHA04H31mnbWiM24vAl0jCxGVEao0aw2R/YGngV8PvMvKceWJYC7mqzL1dELAOsk5lX9/q4RMShlD5at0TESZl5QmZ+OyIeDbwSWC4itqV0kn8u5WD3lMy8pq1yLuI93NffLSJOpDQvzaA0MV0NHF7uisN6Z0jWYHnvwp5TA+n/AbcDb87MK+pnvQZlcNI9gCMj4vWZeS0lkF3Ue6BNh+rSeJ4JbdiaYvo3nt4Bz9PrF8vmwJ8z8zyAGmo+BKwP/DEivpCZfxzvF42IpSn9rq6PiPdm5t8j4hjKhaR/V1//GxGxSWZ+LDPfGxHXAS8AfkEJMv8EnpyZfx7v8o1WI2itSBkR/AOU0DoP+FQ9KfJwYG5EHAHcRTkw/zEzj+im1FoM1wMn1KB1BLA9sHNmXlRrWF8HHBURe9carvsYtNSVvh+Da1AG3L0zM7+5OM9n2JpC+vryrE7pgLpaRHw5M2d3W7rhUUe1nku5QO4GEbEP8CDKJW5+R7mQ7muBWcC4h63MvDcizquvcUBEHA+sBLw6M38eERtSrif3kVoTdHBmfi4ivkDpv3VNeZr5A5h2JSI+Q6l1uwH4SLM2MDObgespwGxKM9PnOyiqRmEhZ+f+AFg6ItalNLl/DPhLve/LwM6UGtm31UnqTOM4OS0iVqX8gF4HeAkwOyJ+AVwz1soJw9YU0Nh4lunbeHYDrqDUklzRYREHWn9TRq/DbkR8D3gUsD+lM+97MvNT9T6AbSNi2fEMso2hGt4TEbcB+1KaZDamNL+RmVdHxKeAOcAHI2JeZn641mD+dVBqC2po/S0lSG0GrFLn37e+a+C6jtJ5+mZgm8y8uKMiayHq2a9L9cbJAtYDbsnM22v4mh0RGwAbANc1moK3omwDJwDf7aLsUlNmZkTsSBmb8MXAVZQfzXcAn87Mfy3O8xq2poC68exESeYvAf5O2XhmUav3uyzfIOs7U25/YENKLdYxmXlm7dS9CrBMZl5Zl1uTEn4updSAtSIzD60HtndTgtWG1E76mXl9PUsS4P1RLr3zvi6DVn+tRz0wn0JpHjyC0nT4lF6n6EbgOj4iTgXmWAM7WCLiEZTLP90GzKsnNJxK2RZXjIiPAidn5tXAxZQfJa+NiH8Cy1H6O/4uM0+sz2cfLXUmIt4MbEfZLk8DPgccQgldWwKn1+XG3O3GoR8muYh4C6XmYE9KDdZvMvOQiHg+5SB9UGaeYZ+tB1ZrsZ5ACVArUfprHQ8clZnNDr2PozThvYAyTtRfRni6xS1DM/g9qtffKiLeSmmaORE4JDP/1njM2pTPeS9gs8y8abzKMxZ9ZX94nf3fzLy2BsZdKYOaXgI8o9bCDfqAsVNalAFyf0052/XRmXlbRJxLGYz0FEqt1W6Ui0p/PjMvjYgdKNvpOpQrGVwBPKk2jbsPUiciYiXK5creTrmc2ReAszPzv/X+n1L2+09Z7G00M50m4QQsDXwWuJIygvguwKqN+38M/Lrrcg7DBLyZ0qn8CcDKdd7hlFqr11KaT6B08P4FpT/KY8a5DNMa/38B+CXwpsa8d1I6In+REqqaj10LWKvD9bdU4/9jKQfYG2t530o5qxJgd0rNx9mNdbrURJfXadSfa2/4hr9QhuxYm9IH6zGNZd7T2C43rfPWpFxUerfedg1M7/r9OE3tCXgEZRDo1evtXmXU7pR+rk+rtxdrn2Qz4iSV5ZfiZyjVoDdk5i21XwUR8QLgkZSzgKy6X7SHApcBf8oyEvtDKAeLEyjNsL3alx8CNwGnZuY/x7MAOb9W6ATgicC7KH1devcfGqWj2P/V5Q7P2jycmTeOZ1nGKuef0fNlSgf3A4BbKde9+zSwTUS8HfgRpW/WZ4ALImKbtGZr4ETElpRm84si4jTgXkqY+gvloPS+3rKZ+eGImEvZLjMiPp2ZlwFfbzzftHTgUnUkItbPzGsy85LGvKAM4TMXeCplu74c5u/PxsqwNQlFxHrATblgc1IAvQsMP5XS2fiv4OnVo7AOsGYNWhtTrtN2OjCjznszcEVmnhYRv28rIES5jM0TKWNRnZF1yA7KL625mfmx2jH/f4GVIuL9mfn3NsoyVrVz9OMpJ2d8P8uYZL+ihK2bgDvqvB9Sakw+SOn3M66hVUumngL/ZWCNiHhFZl4YEWdTms7fS6kdWBu4LiKWycx7GtvlvsDqEXFANjoZu/9RVyLi48AmEXFEZp7bm5+lCmtuRDyK0rKxTy7heISOID/JRMSnKR36dmjOz2JulLGg9ga+mKXTqqpezV/jdi+cngusUPu//YHScXJGZs6KiIdSOk8+arz6GEXECrX2oN/GlB9IF/dep+4U7utDkJkfozTVbQfcuaRlWVy1H1bTusBjgMtrqHo48C/gJMpZnHdFxNaZeQ9wMmX0cIPWgMnMmykXMb+VMubZ1llOWjiLst+5HvhmRKxSP+dl6uM+Vh+3MjAw1xDV1BUR36Hsu3/OCNtk3XZfTumn+8MlfT3D1iQSZWTmXYE/U5q9+u+fRukncSn1rAoVtSmj19y1aUSsRjlbCsrZVXMpTbIXAP8vM2+tnc8PpNTAfG+cglZQmtEuiYjH1nm97+nalM7H/+6VGRZopntevf0eYPvMvG5JyzPGsk/rdX7P+aPaP6fe/S/gH8BD6hls51K2wddluSD2HsAhUS7dMjsHYAwwLai3HWbm0ZR+g9OAL9bAdTfloPVWYHngnIhYtS9wvQd4fq2R9dijzkTEQZSrf+xJObP8HxGxdEQs21hsHrA6cP547Evd4CeJiDiQ0sT0ckqt1b97G05vZ0cJD48GzsnMf3RT0sFTz4Lq9Yn6EuVXzO+Bj0bEI7NcRuS5lLCwDnBoRLyfUoO0O/DSrMM+LKlaU/VZSifxH0XE4xoh7kRK5+IP12Xva36JiEcCb4uIZ9VZN49HecZoG+CI2v+KKMM1vKM2PV1P6fPwIeAc4AzKtjorylAZO1Oaue/ooNwahRqSegH/OMqPgmbgmk3ZbvcGVgB+0ajhml4fl/X7Zl+8JdCoddfieSjwq8w8PzPvrj8SjwZ+HBGHRcRKtR/hFynNiEu8zg1bk0DdAW4O/CAzf1c3ni2B4+opq8dExDqZOYtSE/Ou+rgp/4WtNVq9UfUPpZy1eQzwK0rA+nwNPP+kdO7+BWW0650pnSZ3yMwLx7NM9fn2ptSi/SAitqp3XQF8FXh1RNzXCbn20duPMpDkJfU5ujiF/gbK2a/vjIgLgcdSTqW+vZbnZXWZ1SgX7l6+LnM4ZaiMAzLzlg7KrUVohKzm4L7HUUbzn87IgWt54LKIWLHZAb6jbXPSiIjpjX3WWl2XZ5jU2vdlKANBrxYRz4uId1P2tVsA/6H0eX0XQGb+qbftLul26zhbk0REfIUyntabgKdRNpbzKBd23ZxSm7Bv7RMz5fX3r4qITShB9KzMPL7Oew3loDEX2Dszf193dHMaTSrj9gs9FhyL6uXAppSaoMuBPTPzDxHxMMqI9a+i9B+7nVKL8HBgp/EOfqMs932d9CNiI0oT4frAZzJzv7rMMrWGYz1Ks+yqlGbRK4FlgZd1UXYtWmObX45yck0At+b8a4K+khL251AuNv3HekDbmXIpppfbCX589O0jvgDcDbw7Hex3TCJia0oLRlIGgv5GZh5W92XfpOyfnj+u+3fD1uRQa7K+SBkL6mLgO5n58brxnEwJ5rt3WcZBEGWE6w2yMdhoRBxMGfjzcmCv3kGk3vcqyllUc4E3ZjndvdXBF6MMoPowSlheDdiRcl3A52fmBTWwbAv8D6VW4a/AkZn517bKtJByTgOWz8w7GvNeCuxECVJPAj6emYfX+3qBa1q9b3PKcAFX1qZaDZiYf6H6lSlju61EqUG9kxKa96k16a+k9Ne6F3hLlrMUl2703XN4mSXUF7S+S+nCcCdlXLOBOOt4UEXES4CNKBe8PyszL46IdSgnbETWM/cj4kGUIX3+Arx1XPfzOQCDiTkt1gBse1KaaN4CbNeY/xhgw8btVSlh60jKgTm6LnuH62yp+kX6cXM9UGphzqR0iNybMoZQ83GvooyU/RfgkS2X8WWUquydmD+w51MpNZPXAY8b4TGdfKaUZtbTgQfX298Fvle3uc3rNncDpXmw95hluiir0xJ9zstQmtXPpgzfsRXw7Pp9OQ5Yri73SuA3wN/oG1h3WCZgja7LMEKZgsagr5SQexXwesrJMuM6gPJkmyh9Xa+u+8/rKD9c9wFW6FvuYZR+W9cDDxvvcjjO1hCqp6w+jfKr5sHA9RHx3czcLxe8dMxjKZ37ngI8Oaf4wIFZfqEfRL1iez3z7erMvKb+Mj+Z0kR3YUT8OucPr3BcPdngFZTrSbbpQZRQ/JecX4X9K0ozzbeB70TE7llq2Lq+nM0NlL4Pf4qI31PGWNo9M28Fbo2IT1IOFPtHuRj2Jyitju8Brs1yVpsGSK/2sW/2wyknZrwNmFm/O3vW+/6Y5UxEMvObtQZsW8rJJEMlIg4HpkXEt7NRu91heaZn5pwsSWBOnfcTyufxfMo6/ijlh03rNe7DKCI+QBkG6dWU6wGvB7yBci3W6ZQTeoJyubNnU2pun51l4N3x1XXqdBpzSn8f5VfNUykdUDcCPkU58H2hsdybKc1QlwFbdV3uEd5HpzVswDsov8y3asxbl9JR8m/Ak+m7LAOwyji99irAO0ZaH8BLKb9WnzzC4w6vZb6NAfk1SzkD8U5K9fxudV7z0kJbUK55eAvlWpLfoPQzeXjXZR/2aby/Q5SD0vHUy5U05j+ZcrHw7evtl9Xt8J319hqUPoX9zzdtPMvX8ro8Efg75eoG67W5nkdZnmUpY9B9sDFv1/rdf3y9vSLlslev73r9DeJEqZH9IfD1/s+TEq7upYzn19uPfQB4SFvl8WzE4fNYyrAEv8nMuzLzKsqvmy8DL4iIPWtS/yeliv85OQAdj3tnM/Vk3cI79EtKs8iZtbMkWcZS2YUyYOOxwJOiMR5QZt42Tq+9B7BzRKzQeO7e+ug1Z749IlbtO2P0X5QmhJ9QqsI701gvq1H6CF4OfDUiNs3SUX5pgCz9yA6ljBT/KMqYZE/IcbxA91QxAd+hFYHr8v5nhN5BqVlZIyJ2B75FGYi2d4mo5wGvj3IZq2b5hqKPVu2zuR0lRH4+518cHbhvuIqJPlauQqnlfmFEHFDL8UNg88w8v9ZizaI0j23Ye1BErBQRu0W50sVUdy/lGsGrjHDfZykVEf9T+xZeAHwo2xwSqev06TTqlL4UJan/Bji+zpvO/H4961MC1qebj+m63P3loFxq5mBK4Nhwgl5/xF/YwOMogesWYOvG/HUotYI3U3/Nj1M5VqF0al+rMW9zyvhoKwPL1nnPoDRXHg88qs5bm3KWzCfosN9T/7qkjN32IEqtyG8p/c02622ffcuuSONi6E5jWu8T9h2inN16GPXC0XXe0ZQayXmUjsO9+VtSfrh8mSHsD0qpQToROKQxbzPK8C/fpjY3TXCZerXc61H6QV5MOeNwgfvr/6cDJ9b/V6b8SLyZvtq5qTY11uHnKUP0PGqEZX5DOZFsYsrU9UpxGsWHVNqRexvP/pRajR3q7emN+35E6fw9kNX3dad2Q934/0OpoXlsy6/ZbNLaEXgm8NTGvEdTxs7qD1zrUS5BMm4dfSknNJzduP0lSpPlHZRaq/9jfmfzXes6ugK4CDifMsxDZ81vfevy6cCzgKc05m1Xy/kfanU8JYwdQKnN6nwbHPZpIr5DlDNF51FOJul9jlvU2/cAL6F0kn8B8DtgZi+QMJyB68eUE1AeTTkr+a56IP55/X4e0kGZej+iH8z8wPWOxv3L1L/HUALXsnV/chuwTdfrtMPPcj3Kj+VV6u0VKbXuvwIe2lhu3brf/zilIqP17bbzleO0yI3nsDo9ot5+GPBT4EIWPAtxLUptzBFdl7lRpubBeadavh0pfc1eX2//tq2dAwv+AjyB0tftP3VnegL1zD7KGZxnUQLXViOVf5zKs1RjJ/kNSpDao94+nxK4DgXWrfM2oPQjOIZS7b1lh5/lotblY+t9T6yf602UMd++TgmJW3S9PQ7jNNHfIeb/cHs25UfASZShUnr7nqPr9+S/lHHefggs3V/WQZ+A1zX2qU+k/KCZVf8eWOevSKn5PnYCyxX9/1NaLb5HX+Cq9+1HGcj4hPpdbPXH6yBPlNrVmZSavdOpfdkowyH9nTJEzgfrOvtO3Y7H/azDhZav6xXk9IAbz4mUVH4QjWphyvgqZ9cd3oHAeylNTv/t8oBcy9b7RdbcaXyQMjjnMSx4CvOe432w6H/tevuIGg52olwP6+GUX4C/odYUUfrCnU75Rf/oltfRq+pO4Sn19tso/Qt+Sulsfii1eaixPjs5kI1xXW5Zl9mO0rfsWkr/wq0nutzDPE3kd4gHaCKjDEo6i1KzskFj/lb1ALYZ8wPBhDa1LeH6fWL9nh8DbFznrUwZ1mKzxnKrUGq9PsEE1H6wYLCeRulv1Ksx3JARAhewV30vs6by94zyo+5q4I2U8d56JxO9p96/FnAKJZheRTl+TuhJRp2vJKeFbjwfofTB2g5Ysc5btnH/FvXAdz2lhuSsid54RijzCpSLYG/ZmLcSpSPiPOAH3P8Mv97B4lwaNXWL+frNfi29g0Dv1+m7mT8e0Ep153QUjbFW6s72VFr+tQP8P+CT9f8ZtSy9Gq4TKb9QP9I7EDTfzwR+lku6Lpem9OdZs8ttctimifwOUQ/u9fk/Qzmh5qj6PVip3rdL/XxPYiFnavWXZxgmyphg8yh9nDYf4f5H1nVxAxNQK8uCQetDlIB7HiXoPabObwau3pmg0+syrY7/N8gT5UzCSyjN2r1a1ifUz/crLHjcXJvS1LjyhJez6xXlNOLGs0rdqR7UmPcQyiCR3wLew/wAtnY9CK40AOV+OPBJ+oZIqGX8CaVZYhfu38l6D+BSysWJl13M116h7hxf0Df/wfV196m3N6dUH5/YCweUX/C9X5CL9foPUK6Fdc5/MOXX9IXAIb0vP6Uv1J2UqvAPLOzxLX+OS7oul+t6WxzWaaK+Q8wP0CtQas8vpXR0/yvlB9z7qbXp9TO9g9JUtemSvseO1+8yjffeC1xHs2CN1j6UQYwvZ4KHzaE0b/0b+BqlifbvlLOjn17v34gSuC4E3l/nDU3zbUvrbMe6ffaGJtms7j+/2dgvdd682vmKcmp8GI3AxPzO7o+jdJy+k9JUMLPuEN9GqdoeqC8aC54FskNj/lp1B/aP+uXoP1jsDmyyBK+7Vf3CnU4Z7qI3f2VK58hPUH4B3Uw5y6jXgXJHSufYce/AzYK1Q2+lhOQ3NuY9rJb5+Y15e9ad6cfoqJ/TOKzLbbveDkfxHgfqe9NXtla/QzSapinB7ceU/oG9/oRHUq5Z+T7m13A9hxJMDu56/SzmOj2U+V0Glub+geurlNaCpSgnABxIo0P1BJVxT0pT2FMb5Xsy5VhwJ/PHhFqfEqrPYwBHvJ+gddU8Vj6z7q82rt+R3n6pt+2+iPJDYZ1Oy9z1SnNqfBjll+Un6v8vpPSDuZPSrPDeOn9pSj+Yo7oub1/Zm/1LNqKM1nsrdQC+On+t+p6uHOlgsaSvTemLcTnlLKLnNu4/hDJO0J3U06Tr/AdRqpl/Cazd4rr5FqU/3V8p11j8HrAJJbz8FTipsd6+Cnyl689xUNflOL3HZpPNUyiBYyXmN6t1dfmjCfsOUc4SPZYSjr9PGeix+ePgS5RLmzy2MW87hqhvVqPcW1MC6n2XEWLBwLUfJXB9hhqwxmvfNMZy7k2pVdyob/62lJMRTmF+gFiXRj+6qTbROFbW2+fW/dUtdbvutRSsQ6nhOp4Omg4XKHPXK83pvo3l8ZS2+NfV29PrDvdJzP9FFsDqlI7UH663Oz/VeqQdE6Uq92eUM9Ge0JjfO1j8jTJ0wBL392DBDsNPY34ftl0a84+sO9T3UvpjPJ3SR+VmRhiDZQnL0zxobUwJLNtRag+eSTmL70zK5W3eQOkXchulKedmOux7N2jrsuX3egLlrMl5dUc9o7GTnug+cq1/h1gwZG5MCXM3Aac25i/T+P8K4Iv964PhDFzPoZzxezm1jxbzx7XbhBIs51FqE5eegPIs1f8/pbXiRuqPlb7P4qOUs5Uf1PW67Hpi/rHytcwPzM+ntPrcxoLDlRxDOVGn0xPHMg1bAzPVA9e/GaGzZmOZRzC/0+ZCl5vgcjd34K8F3km5FM6GlDP8Th3hYLFmDRYX0ncx0MV4/f4hCZ5NCahXUEJOMyR8th6gZlE6VP6OcQ42fetjmbpj+BmNS6BQOm/+h9Ivb1tge0qz4fvpcIiEQVuXLby/pRv/v4HS6fzFlCbRH1FOSHkndeBVJihwTcR3iPl9ElegDKy7PiVknEEZt2/f5nZAqfk6Dzim689tCdbr1pSLpfeGeHgKpVXg8ub3rH7+nwFe3lu25XI1Q9QazB/qZR1KMPhO4/5emHgjpf/Wul2v164nRjhW1n3tyyg/IG6un/MFlBrNgThLs/MCOCWUs7auBd5eb9/vNGPKr56f141nEK91+F3KKbWXUJo4rqWMs7QzpaPnfdf0qss/iMXso0UZY+hJjdvTKGeYXMf8sbN6IeEsYOfGsltQws1D6bsG3Divj89R+tf8kNLPqVf932uq2o4SuH7U2/H3f+YT9LkN/Lpcwve3IvC0vnmvoJwy3z9m0fF1G34XExy46mu18h1qbHMr14PQWcDedd4jKbWsf6aODF+3gYdSDu6Hd/0ZLua6/DrwJ0pz9/nUbheUPlAXUGqJdqzb71GUAS5bazrs/57VeUcBf6mf80cpweu1lNrG46h9cuvnfFJzPzJVJ0Y+Vt7X/E/5MbEP5QfLy5igq5SMquxdF2AqT42N5FV1B9s7m6JXrbwq9ZTeenA4iAE8G4gyFMDVlH4+69R5P6D8wtiZ8uv8tLoTedISvlZQ+pNcT7k6e2/e2nVe82A0Yq1Mm59l/f9zlBG+v1Ffe4GOxY3P9/H1vlOYgKaLYVmX4/z+vkU5s6v3XduxrvN5wP/Vec1Tw3uB6wBgtQksa6vfoXoQ+n19jsfTOGOUErjOqOvkx5QfAGdQAtgwNhkeQwmrz6UElVPre/tevf+xjfd7PeWHRWs/YBfyPfs05YfzR+r+4u66v9iW0nerd/WIcygh6zSTRl0AACAASURBVBYGvOa45c90UcfK1Rnw4S86L8BUnyjJ/DLgm415K1M67f6k7hD+t35hJ/yAPMr38PW6g+udzbRh3bF8i/njnmxFOZvyKpZwaADKWXynUX65PqfOW6UemJ5M6e/W+3LuUHdaZ9A3lEFL6+JxlDOfdqm316f0r5tHPcmh97k3lu+y6XBg1+U4vb/1mT9MSm/Q1Rn1APuzxnLNpp1vUM5u2o+Ja0ps9TtE+bF2CY0x5ChXTnhW3QbXrp/rfylh61mN5YYmcFHOyPw9sFO9/VbKJYa+Qul+cVJj2V0pgaz12o++79nzKN0Gdmvc/5y6zR1Pqb15GKVp83jKeIoTNtL5oE4s+lg5h4VcQ3IQps4LMFUn5rfFv47SN6J3uZMDKb8u59Yv2hsZ0JBVy7sU5cyQk+vth1J+hX2b+WOcvB7YlDKG0Ebj9LqbUZo//lS/bNPqQeh+o79TfsneQjnrasUW18UBlA6u/6bRUZzSF+NDLCRwdT0N4rps8bPpjVf0RkptwtcayzQD11eYoH6RE/EdolxT9U/MbyZ+GyVYXc/8H3Rr1+3gXOqJOoO0nY7iPQblygb719v/Qxkg+KX1oPzF+l5P7OJAXD+/Mymdu++rTWP+j5mdKH3zTqYxTEEXZR2kibEdKwd3SJeuCzDVJ8rZL5dSqpXPrweEI4Fn9C03sF+4WvbzKL/YemOc9Pq8bEnpu/TyFl63FxIuplS9X0Fpq38JsBulQ+z2lA7pezCOF5VeSHm2pHSGnwO8rO++tSmB6x7go11/ZoO+Llv6bH5OufZdM3DNZsHANa6D2o6hfK1+hyi1YrPra8yk1KL8b/08P0gZYmJFSpg+s+6L9u76c1uM97kS5WzNlep7PYT5gXVTSvP+PGqw7aB8m1NquOY1P0/m13TvRGkqPpvaH2+Q9/0TvO6G+ljZeQGm8kTpK9HrP3IK8AVKLUiv2SCafwd1Ah5Vd97zKL/Kemc+rUmpIbiIlqrqa0g4nfmXMzmtHjhuo3SCvbVO60/QungIpWbgHzTGp6r3rU25ZtctDOBlbAZtXbbw/jalBK4/c//AdXTHZWv9O0Tpd3c0pZbvcY35+1Ka3tZolGVmPeCv2vXntpjv9cF1m92/Me/F9bv5eiZ4wNK+sj10pH1EI3A9l1KzPGXH0RphnQ39sbJXQHUgIlagfPH/C/wgM2+p8yOH7IOJiGdSmpZ+Sxk2AErH3qdTzga7qMXX3pzSyXRzyhlmJ0XEOpTmmWWBWZl5Y1uvP0J5NqWcabRWLc9PG/etSdkhTFh5xmLQ1uV4i4jNKJ/N2sD/ZuZZEfEGSgfmIzPzLR2WbUK/QxGxNCWAfplS4/OKzJxX73sE5bP+53i+5kSJiN5YZH+knFh0F6WmdkXgTZk5q8PiLXQfERFLZea8iFghM+/ssoyDZDIcKw1bHYuIaZk5t3F7aDaefhHxeMopzBtQ2tEvA96XmRdPwGtvRumTsR6wX2ae0fZrjqI8X6Ic1N+emad1WZ6xGLR1Od76AtfemfmLiHgtcF5m/qXjsk3IdygiHkTp0/R8SpPb4zNzTkRMA+YN6z6oKSKeQmnWn0UJW8tTOs639sNvLIZ5H/FAIuLJmXlOC8871MdKw5bGVUQsTxlgbi5wb2bOnsDX3oxSvfwI4DWZeeZEvfYDlOfzlGaZV3ddnrEYtHU53hrv75HAnm0cHBbXRHyHIuJZlEF0/wG8tgat6Zk5Z7xfq0sR8WhKn8O7gFMy8/KOi7SAYd5HjCQidqJ0RTggMz/RdXkGiWFLk0pEPAw4DHhbZv7d8iy+YS77aNT393FK7d2ke38PJCKCMrzE1ZmZ/bUGmjiT6XsWEasB/wd8KzMv7bo8g8SwpUknIpbJzHu6LkfPoJVnLIa57KMx2d/faPT6CXVdjqlsMm2Hbk8jM2xJkiS1aKmuCyBJkjSZGbYkSZJaZNiSJElqkWFLkiSpRYatSSAiZnRdhkUZhjKC5RxvlnN8DUM5h6GMYDnHm+V8YIatyWEYNvJhKCNYzvFmOcfXMJRzGMoIlnO8Wc4HYNiSJElqkeNsTbCIcIWPo2222WZcn+/GG29krbXWGtfnvOCCC8b1+SRJgyMzY1HLGLYmmGFrfA3D9luujCJJmoxGE7ZsRpQkSWqRYUuSJKlFhi1JkqQWGbYkSZJaZNiSJElqkWFLkiSpRYYtSZKkFhm2JEmSWmTYkiRJapFhS5IkqUWGLUmSpBYZtiRJklpk2JIkSWqRYUuSJKlFhi1JkqQWGbYkSZJaZNhaDBGxfUScGhHXRsSsiPhjRLyy63JJkqTBM73rAgypjYFzgSOBu4EdgGMiYl5mHt9pySRJ0kCJzOy6DEMtIgKYBnwe2Dwzn7GI5V3h42gYtt+yiUiSJqPMXORO3pqtxRARqwMfBHYD1qeELYBrFrL8DGDGxJROkiQNEmu2FkNEnAI8ETgYuAS4DXgzsFtmrrmIx7rCx9EwbL/WbEnS5GXNVgsiYjlgV2DvzDyyMd+TDSRJ0v0YEMZuWcp6m92bERErAy/orESSJGlgWbM1Rpl5a0ScDxwUEbcB84B3AbcCq3RaOEmSNHDss7UYImIz4EuUfls3AZ8DVgD2sc/WxBqG7dc+W5I0eY2mz5Zha4IZtsbXMGy/hi1JmrxGE7bssyVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUosMW5IkSS0ybEmSJLXIsCVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUosMW5IkSS0ybEmSJLXIsCVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUokkXtiLi2IiYWf/fKyIyIlaqt9eOiA9ExCZ9j9mizl+tb/4Cj5ckSRqrSRe2+vwI2B64s95eG3g/sEnfclvU+ashSZI0jqZ3XYA2ZeaNwI1dl0OSJE1dk7pmq9kMWJsO/1TvOqvOz4jYEfhBnf+POu/KB3jO5SLisIi4OiJmR8SFEbFLm+9DkiQNr0kdtvpcC7yy/r83pXlxe+D3wP51/ovqvN0f4HlOAvYCPgI8HzgfODUith7/IkuSpGE3qZsRmzJzdkRcVG9ekpnn9e6LiMvqv3/IzCsX9hwRsRPwPGDHzPxFnX1aRGwBvAfYYyGPmwHMWMK3IEmShtBUqtkaD88ErgPOjYjpvQk4E9h2YQ/KzKMyc9vMXOgykiRpcpoyNVvjZE1gXeDeEe6bO8FlkSRJQ8CwNTY3A9cAL+y6IJIkaThMtbB1T/273Cjn9zsTeDtwR2ZeOp4FkyRJk9NUC1tXAXcBr4mIW4F7M3Mm0Osg/8aIOAG4MzP/NMLjTwd+BpweEYcCFwOrAFsDy2Xmu1t/B5IkaahMqQ7ymXk38AZgG+AXlGEbyMx/UoZ/eBFwLvPH3ep/fNZljgb2owSvL1GGizin5eJLkqQhFCU/aKJEhCt8HA3D9hsRXRdBktSSzFzkTn5K1WxJkiRNNMOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUosMW5IkSS0ybEmSJLXIsCVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUouldF0BaEhGLvNh65zKz6yKMyjCsS0kaRtZsSZIktciwJUmS1CLDliRJUosMW5IkSS0ybEmSJLXIsCVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUosMWw8gIjIi9hnFch+IiP9MRJkkSdJwmd51AQbc9sA/ui6EJEkaXoatB5CZ5z3Q/RGxNDBvgoojSZKG0JRuRoyIfSLi6oiYFRHfj4idatPhjvX+BZoRI+LsiDgpImZExBXA3cCDOyq+JEkaAlO2Zisidgc+C3wBOAV4MvDVUTx0B2BT4J3AncCtbZVRkiQNvykbtoADgR9n5t719mkRsSbw5kU8bjVg68y8vjcjIh7wARExA5ixBGWVJElDako2I0bEdOCxwKl9d/XfHskFzaA1Gpl5VGZum5nbjuVxkiRp+E3JsAWsCUwDbuyb3397JGMKWpIkaWqbqmHrP8BcYK2++f23R5LjXxxJkjRZTcmwlZlzgD8Au/Xd9YIOiiNJkiaxqdxB/qPAdyPic5S+WjsAz6v3OXaWJEkaF1OyZgsgM78H7Au8EPg+8Hhg/3r3bV2VS5IkTS6RaReknoh4L/AeYI3MvKul13CFTzHD8h1b1BAmkqT7y8xF7jynbDNiRKwFvBs4izI46VMoA5V+ta2gJUmSpp4pG7aAe4AtgVcDqwLXAp8G3tdloSRJ0uRiM+IEsxlx6hmW75jNiJI0dqNpRpyyHeQlSZImgmFLkiSpRYYtSZKkFhm2JEmSWmTYkiRJapFhS5IkqUWGLUmSpBYZtiRJklpk2JIkSWrRVL5cjzQhhmVkdke6l6R2WLMlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUosMW5IkSS0ybEmSJLXIsCVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwNUoRcWxEzOy6HJIkabhEZnZdhqEQEZsCy2fmn5fweVzhGkjDsi+IiK6LIEn3ycxF7pQMWxPMsKVBNSz7AsOWpEEymrBlM+IoNZsRI2K1iPhKRPw7Iu6OiKsi4stdl1GSJA2e6V0XYEh9EngS8DbgOmBD4KmdlkiSJA0kw9bieQLw+cz8dmPecQtbOCJmADNaL5UkSRo4hq3F80fggIiYC5yRmX99oIUz8yjgKLDPliRJU419thbPPsD3gYOAyyLibxHxso7LJEmSBpBhazFk5n8zc9/MXBfYCvgt8M2IeETHRZMkSQPGsLWEMvMi4ADKutyy4+JIkqQBY5+txRAR5wAnA38GEngDMAv4XZflkiRJg8ewtXh+A+wFbALMBf4A7JyZ/+qwTJIkaQA5gvwE82xEDaph2Rc4grykQeII8pIkSR0zbEmSJLXIsCVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUosMW5IkSS0ybEmSJLXIsCVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUosMW5IkSS0ybEmSJLXIsCVJktQiw5YkSVKLDFuSJEkt6iRsRcReEZERsVK9vUm9ves4PPeO9bketYjljo2ImUv6epIkSQ9ketcFqK4FtgcuncDXPBhYfgJfT5IkTUEDEbYyczZw3gS/5hUT+XqSJGlqarUZMSKeGhFnRcQdEXFrRJwdEY8dYbn7NSNGxJURcXhEvCsirq2P/0QUu0TExRFxe0R8PyJWH+HlHxwRP4yIWRFxVUS8qe81F2hGbDRtPjoiTq+PuzQiXtT3uIiIgyPihoi4LSKOjoiX1cdussQrTZIkTSqtha2I2BE4E7gXeA2wJ/ArYP0xPM3LgCcArwUOA/4P+CSlCfB9wJuApwEfHeGxXwUuAl4E/Bj44ij7hH0LOBXYHfgbcEJEbNC4fz/gQOBI4CXAXbVskiRJ95eZrUzAb4CZQIxw315AAivV25vU27s2lrkSuByY1pj3O2AO8JDGvMOA6xu3d6zPdVTfa54OnNe4fSwwc4Qyva4x70H19d5Ub0+j9C/7fN9z/7g+dpOFrIsZdV3MrMs5OQ3cNCy6Xk9OTk5OzSlHkYlaqdmKiBWB7YCv1Z3j4jo7M+c2bl8OXJmZ/+ibt1ZELNP32JP7bn8P2CYipi3iNU/r/ZOZNwE3AL2arQ2BdSk1X039txeQmUdl5raZue0iXluSJE0ybTUjrg4EpRZoSfy37/Y9C5kXQH/YumGE29OBNRfjNZer/69b/97Yt0z/bUmSJKC9sHULMA9Yr6XnH421R7g9B/jPEjzndfXvWn3z+29LkiQBLYWtzJwF/BZ4dUREG68xCruPcPuCvmbJsbqaErh265v/giV4TkmSNIm1Oc7Wu4AzgJ9ExFHALMrApRM1avvOEfFh4BeUMxKfxf1D0phk5tyI+Djw8Yi4ETiXErQeXReZtyTPL0mSJp/Whn7IzF9SAs4KwHHAtynDNPyrrdfs83rgccD3gV2BvTPzATuyj9KnKENNvAX4LqV/2kfqfbeNw/NLkqRJJJbsZEEBRMRXgGdl5sajWNYVroE0LPuC7nomSNL9ZeYid0oDcbmeYVIvcL0n8GtKs+HOlEFX39lluSRJ0mAybI3dLODJwD7AisA/KUHrE10WSpIkDSabESeYzYgaVMOyL7AZUdIgGU0zYqsXopYkSZrqDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUou8XI8kYHhGZr9z9uyuizAqKy63fNdFGJXMeV0XYZSGYfscjqswDMe6hOFZn4tmzZYkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUosMW5IkSS0ybEmSJLXIsCVJktQiw5YkSVKLDFuSJEktMmxJkiS1aEqErYg4NiJmjvExMyLihW2VSZIkTQ3Tuy7ABDkYWH6Mj5kB/Bn4/vgXR5IkTRVTImxl5hVdl0GSJE1NU64ZMSL2ioiMiEdHxOkRMSsiLo2IFzWWPxvYBnhNXTYjYq9637SI+EBEXBURsyPi4oh4RRfvS5IkDb4pEbYW4lvAqcDuwN+AEyJig3rfW4BLgR8D29fpR/W+DwHvAY4CXgCcC3wzIl4+cUWXJEnDYko0Iy7EpzLzaICIuAC4HtgVODIzL4mIWcCNmXle7wERsQawH3BIZh5SZ/+shrQPAMdP5BuQJEmDbyrXbJ3W+yczbwJuADZY+OIAPApYAfhO3/xvA1tExFojPaie2ThzrGdESpKk4TeVw9Z/+27fAyy3iMesV/9e3ze/d3uNkR6UmUdl5raZue3YiihJkobdVA5bi+Pa+nftvvnr1L83T2BZJEnSEDBsLdxINV1/Bu4E9uib/1Lgr5l540QUTJIkDY+p3EF+US4FnhMRzwFuAv6RmTdFxBHAeyNiDjATeBGwC+DZiJIk6X4MWwt3CLARcCKwCvBa4FjgIGAO8GZK8+HlwKsy84RuiilJkgZZZGbXZZhSIsIVLi2BO2fP7roIo7LicmO9Qlg3Mud1XYRRiq4LMArDsnsfhnUJw7I+M3ORK9Q+W5IkSS0ybEmSJLXIsCVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUosMW5IkSS2KzOG4qvZkERGucGkKmDN3btdFGJXp06Z1XYRJJLouwCh5GBpPmbnID96aLUmSpBYZtiRJklpk2JIkSWqRYUuSJKlFhi1JkqQWGbYkSZJaZNiSJElqkWFLkiSpRYYtSZKkFhm2JEmSWmTYkiRJapFhS5IkqUWGLUmSpBYZtiRJklpk2JIkSWqRYUuSJKlFhi1JkqQWGbYkSZJaZNhaDBGxfUScGhHXRsSsiPhjRLyy63JJkqTBM73rAgypjYFzgSOBu4EdgGMiYl5mHt9pySRJ0kCJzOy6DEMtIgKYBnwe2DwznzHCMjOAGfXmNhNYPEkdmTN3btdFGJXp06Z1XYRJJLouwCh53B9PmbnID96wtRgiYnXgg8BuwPqUsAVwTWZusIjHusKlKcCwNRUZtqai0YQtmxEXz7HAE4GDgUuA24A3U8KXJEnSfQxbYxQRywG7Antn5pGN+Z5sIEmS7seAMHbLUtbb7N6MiFgZeEFnJZIkSQPLmq0xysxbI+J84KCIuA2YB7wLuBVYpdPCSZKkgWMH+cUQEZsBX6L027oJ+BywArBPZq65iMe6wqUpwA7yU5Ed5Kciz0YcQIYtaWowbE1Fhq2paDRhyz5bkiRJLTJsSZIktciwJUmS1CLDliRJUosMW5IkSS0ybEmSJLXIsCVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLYpMr/49kSLCFS5pYAzLMSAiui6CNKLMXOTGac2WJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUosMW5IkSS0ybEmSJLXIsCVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktWjcwlZEZETsM17P13jeYyNi5ng/ryRJ0kSwZkuSJKlFhi1JkqQWjSps9ZryIuKFEXFpRNwdEedExCMe4DHPi4jTI+KGiLgtIs6LiGc37n9EbXrcse9xK0XEHRHx1r75z4qIiyJiVn3tR/bdv0JEfCYirqvlO7/5enWZKyPi8L55e9VyrFRvLx0Rh0fEVRExOyL+HREnR8QyjcdsFBEnRMTNEXFnRPwsIh42mnUpSZKmlrHUbG0MfBI4GHgFsCrws4hYbiHLPwT4AfD/gBcDvwZ+EhE7AGTmJcB5wF59j9sDWBo4rjFvI+DjwIeBlwNrA9+OiGgs82XgtXWZ3YGrgR9FxJPH8B4B3g28Engf8CxgP+BWYBpARKwBnAM8DHgT8FJgReCMiFh+jK8lSZImu8xc5AQcCyTwpMa8jYE5wJvq7QT2WcjjlwKmAz8Djm7Mfz1wB7BSY94vgZP6XnsOsHlj3gvr621Zbz8cmAe8pu81/wz8rDHvSuDwvrLtVZ9rpXr7h8AnHmBdHAzcBKzRmLc6JZDtvZDHzABm1imdnJycBmUaFl2vJyenhU05ihw1lpqtGzLz170bmflP4ALgCSMtHBEbRMTXIuIaSli6F3g2sEVjsW/Xv3vUx2wKPBk4pu/prszMvzVuX1L/blD/Ph4I4DuN8s2rt8das/VHYK+IeEdEPKav9gzgmcDpwG0RMT0ipgO3U9bFtiM9YWYelZnbZuaI90uSpMlrTGFrIfPW658ZEUsBpwJPAg4Cnk4JRD8B7mt2zMzbgRMpzX9QapmuA37a95T/7bt9T/3be671gDsy886+5a4HVoiIZRf2pkZwCPB54C3AhcDVff3H1gT2pITH5vR0YMMxvI4kSZoCpo9h2bUXMu/iEeZvBjwW2Dkz7wtOC+nT9BXgnIjYHHg18PXMnDuGcgFcC6wUESv0Ba51gDszc3a9fTewTN9jV2/eyMy7KQHxoFqmNwFHRMRl9b3cTAmSB49QjtvHWG5JkjTJjaVma+2IeFLvRkRsBDwO+N0Iy/ZC1ezG8hsDO/QvWJsmLwOOpnSEP3YMZeo5n9J2+pLG60W9fU5juX9R+nc1PZuFqE2X+1PeR+/MyzOBRwIXZ+bMvumyxSi7JEmaxMZSs/Uf4LiIeC9wF/BBSjPisSMseykl2HwiIt4HrFyXv2Yhz/1VytmGv8nMS8dQJgAy8y8RcTzwuYhYGbgCeAOwJfDmxqInA5+NiAMpAe3FlOB0n4g4mdL/6g/1fb6Esp5+WRf5JPAq4OcR8dn6ntYBngack5nHj7X8kiRp8hpLzdY/KbU8HwBOoDSZPac2uy2gNtu9iNIx/iRKk9tHgV8s5Lm/X/8ePYby9HsD8DVKE+AplLMld83MZs3WUcARwL6UvmKzKX20mn5NOdvxW/V5tgFenJkz63v7D/BESqD8FHAacBhlKIyLlqD8kiRpEopyRu0iFoo4FnhUW2fTRcRbKIHlwZl5WxuvMSgiYtErXJImyGiOAYPg/ieGS4MhMxe5cY6lGXHcRcQmlKEgDgSOnexBS5IkTT1dXxvxA5RBRP9CGbFdkiRpUhlVM6LGj82IkgbJsBwDbEbUoBpNM2LXNVuSJEmTmmFLkiSpRYYtSZKkFhm2JEmSWmTYkiRJapFhS5IkqUWGLUmSpBYZtiRJklpk2JIkSWqRYUuSJKlFnV6IWpLUrWG5DM4wXFZoWNalJp41W5IkSS0ybEmSJLXIsCVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUosMW5IkSS0ybEmSJLXIsCVJktQiw9YDiIhdIyIjYpOuyyJJkoaTYUuSJKlFhi1JkqQWDVXYiohjI2JmRDwvIi6JiDsj4kcRsUZEbBYRZ0XErLrMYxqPe3tEnB8Rt0bE9RHxg4jYrO+5IyI+EBE3RMTtEfF1YJURyrBcRBwWEVdHxOyIuDAidpmAty9JkobQUIWtaiPgQ8B7gRnAk4CjgBPq9BJgOnBCRER9zAbA54DdgDcA04BfR8SqjefdFzioPtdLgLuAw0Z4/ZOAvYCPAM8HzgdOjYitx+0dSpKkySMzh2YCjgXmAJs25h0GJPDqxrxd6ryHj/Ac04Dlgdt7j6nz/g18sW/Z0+vzbFJv71RvP61vuV8C33mAcs8AZtYpnZycnJzGNg2DrteRU2fb5iLzyzDWbF2ZmVc0bl9e//58hHnrA0TEEyPi9Ii4iRLW7gRWAraoy20IrAec0vda3+u7/UzgOuDciJjem4AzgW0XVuDMPCozt83MhS4jSZImp+ldF2Ax/Lfv9j0jzO/NWy4iNgJOA34HvJFSg3UP8CNgubrcuvXvDX3P3X97zbrsvSOUa+5oCi9JkqaWYQxbY/VcYAVgt8ycBVBro9ZoLHNd/bt232P7b98MXAO8sIVySpKkSWgqhK3lgXmU5sOel7Lge7+aErh2A37amP+ivuc6E3g7cEdmXjr+RZUkSZPNVAhbP6d0gD8mIr4KPBLYn0azY2bOjYjDgMMj4j/Ar4AXAw/ve67TgZ8Bp0fEocDFlOEhtgaWy8x3t/1mJEnScBnGj5ny/QAADUlJREFUDvJjkpl/ogzVsB3wQ+AVwB7ArX2LHkEZzuFNwHcpHejf0fdcSantOhrYjxK8vgRsD5zT1nuQJEnDK0p+0ESJCFe4JI3RMByr5g/tqKkkMxf5wU/6mi1JkqQuGbYkSZJaZNiSJElqkWFLkiSpRYYtSZKkFhm2JEmSWmTYkiRJapFhS5IkqUWGLUmSpBYZtiRJklpk2JIkSWqRYUuSJKlFhi1JkqQWTe+6AJI0NtF1AUYlYjjK+eIXv63rIozK3gd8vOsiLNKqq67VdRFG5a67bu+6CKOSmV0XYZHmzLlnVMtZsyVJktQiw5YkSVKLDFuSJEktMmxJkiS1yLAlSZLUIsOWJElSiwxbkiRJLTJsSZIktciwJUmS1CLDliRJUosMW5IkSf+/vfuPtbuu7zj+fEGF0g1wWKg4EaNmmXEkgoXM+BNR6eISnIu4ybJChE43jDMws5BsgsxotiBk6KKVKSVDqWRzomyWHxXFH5AgsHTTKGyUlIkFWpQo0tL2vT++3wvHm8u9F3Y//Z5z+3wkJ7fnez7f73mf80+f+X6/vW3I2JIkSWrI2JIkSWrI2JIkSWrI2JIkSWrI2FoASU5J8v0kO5NsHnoeSZI0PpYMPcCkS7I/cAXw78BZwM+HnUiSJI0TY+v/70jgEOBzVfXNoYeRJEnjxcuI85Dk1CSbkuxIsiXJh5MsSXI6sKVf9qUkleT84SaVJEnjxtiaQ5I3A+uB24FTgEuBc4GPA9cCb+uXngu8ErhsgDElSdKY8jLi3D4E3FRVq/vnX00C8BHgb4A7+u0/qKpbZjpAkjXAmtaDSpKk8eOZrVn0N78fB1w97aX1dN/dK+dznKpaW1Urq2rlAo8oSZLGnLE1u+XAs4Ct07ZPPT9s744jSZImjbE1u4eAx4Ejpm1f0f/cvnfHkSRJk8bYmkVV7Qa+C7x92kunAnuA7+z1oSRJ0kTxBvm5fRDYkOSzwFXAMcCFwKer6r4kLxxwNkmSNOY8szWHqroO+ANgJfBl4M+Bi4Czh5xLkiRNBs9szUNVraf7F4gzvbYZyF4dSJIkTQzPbEmSJDVkbEmSJDVkbEmSJDVkbEmSJDVkbEmSJDVkbEmSJDVkbEmSJDVkbEmSJDVkbEmSJDVkbEmSJDVkbEmSJDVkbEmSJDWUqhp6hn1KEr9wSVqEJuXv0yRDj7CoVNWcX6hntiRJkhoytiRJkhoytiRJkhoytiRJkhoytiRJkhoytiRJkhoytiRJkhoytiRJkhoytiRJkhoytiRJkhoytiRJkhoytiRJkhoytiRJkhoytiRJkhoytiRJkhoythZAkrOT1NBzSJKk8WNsSZIkNWRsSZIkNTSRsZXkxCSV5Hkj276TZHeSZ49s25Tkw/2fX57kxiSPJnk4yZVJVkw77vIk65Js69fdlGTltDUHJvl4kp8k2Z7kYuBZjT+yJEmaUBMZW8CtwOPAawCSLANeAewEXtVvOwx4GXBzksOBm4BlwDuB9wKvA65PcsDIcf8VOBk4F3gH3ffztSQvGVnzUeBM4ELgNOBo4JwWH1KSJE2+JUMP8ExU1aNJvksXW+uB3wZ+CtzYb7sWeDVQwLeB8/pdT66qRwCS3AXcAvw+8Pkkq+hC7fVV9fV+zUZgM/AXwJ8keQ7wbuCDVXVRv2YD8L3Wn1mSJE2mST2zBfAN+jNbwGuBbwJfn7btP/q4OgG4biq0AKrqVrqQenW/6QTgganQ6tf8HPjKyJpjgKXAl0bW7Bl9PpMka5LcluS2p/8xJUnSJJvk2LoZ+K3+Hq3X9M9vBlYmWTqyDeBIYOsMx9gKHDay5oE51jy3/zl93Uz7PaGq1lbVyqpaOds6SZK0+ExybH2r//l6usuI3wD+C/gZcBJwHE/G1v3AETMcYwWw/Wms+XH/c/q6mfaTJEma3NiqqoeB/wTeD+wG7qiqoruc+AG6+9GmYutW4OQkB0/tn+R44IX9+qk1RyR57ciaZcBbRtZsAh4DThlZs9/oc0mSpFETG1u9m+nuzfp2Ve2etu2uqpq6dPix/ueGJKckOQ34F7p4+meAqtpAdzP9+iSrk/wu8G/AQcDf9Wu2AWuBC5Kc099UfzXwq40/pyRJmlCLIbagu4Q4fdvU2Siq6kHgRLqzUp8HPtGve1NV7RzZ963A9cAldBEV4A1VdffImg8AnwH+uj/Wj3gy5iRJkn5Juitv2lv8PxQlaXGalL9Pkww9wqJSVXN+oZN+ZkuSJGmsGVuSJEkNGVuSJEkNGVuSJEkNGVuSJEkNGVuSJEkNGVuSJEkNGVuSJEkNGVuSJEkNGVuSJEkNGVuSJEkNGVuSJEkNGVuSJEkNLRl6AEl6Og5fftTQI8zLgw9tGXqEeVmy5IChR5iXAw5YOvQIczrqqJcOPcK8XPbVG4YeYV7OXPXGoUdYMJ7ZkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJasjYkiRJamjJ0APsC5KsAdYMPYckSdr7jK29oKrWAmsBktTA40iSpL3Iy4iSJEkNGVuSJEkNGVsLJMkfJ9mV5OihZ5EkSePD2Fo4+wH7Axl6EEmSND6MrQVSVZdXVapq89CzSJKk8WFsSZIkNWRsSZIkNWRsSZIkNWRsSZIkNWRsSZIkNWRsSZIkNWRsSZIkNWRsSZIkNWRsSZIkNWRsSZIkNWRsSZIkNWRsSZIkNWRsSZIkNbRk6AEk6el48KEtQ4+wqOzatXPoEeZl2UEHDz3CnM6+4PyhR5iXL1x05dAjzEsy/ueDqvbMa934fxJJkqQJZmxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1ZGxJkiQ1NHGxleTFA7znc5Ms29vvK0mSJt9ExFaSpUlOS7IRuGtk+35J/jLJ3Ul2JPlhktUz7H92krv6NXcnef+015+f5AtJHkjyiyT/neTCkSWrgPuTfCrJ8c0+qCRJWnSWDD3AbJIcC7wLOA1YBlwDvGVkyaXAauBDwO3Am4DPJNlWVV/pj3FWv+5jwAbgROCiJAdW1Uf741wBHASsAX4CvAj4zZH3+SJwCHAGsCbJJuAy4J+qavtCf25JkrR4pKqGnuGXJDmULq7eBRwH3Al8lmlhk+QlwA+BM6pq3cj2K4CXVtXxSfYDtgDXVdUZI2v+oX+PFVX1WJKfAX9YVV+ex3zH0UXXO4FfoQuxfwRurKf4MpOsoQs5gFfM75uQJE055ODnDD3CnM675BNDjzAvG6/aMPQI83L9DevmXjSwqj1UVeZaN1aXEZOsAu4HLgS+BRxbVcdW1d/PcAbpJGAP8MUkS6YewI3Ay5PsDzwfeB5w9bR919OdqTqmf34n8JEkpyd5wWwzVtXtVfXe/rirgV+jO2P2P7Pss7aqVlbVyrm+A0mStLiMVWwBO4BHgaXAocCzkzxVMS4H9gd+Cjw+8ric7vLokf0DYOu0faeeH9b/fAdwG3AxcG+SO5OcNMesT8xI9z0+PMd6SZK0Dxqre7aq6mtJfh34PeBMYCOwOcnlwLqqundk+XZgF/AqujNc0z3AkzF5xLTXVowcg6r6X+D0/rLjCcD5wDVJXlBV26Z26sPvDXSXEd8G7AQ+B7ynqu54Jp9ZkiQtbuN2Zouq2lFVV1XVG4EXA1cCZwH3JLkhyR/1SzfSndk6tKpum+GxE7gP+BHw9mlvcyrwCLBp2nvvqapbgAvobsg/GiDJiiTnA/cANwBHAe8GjqyqPzW0JEnSUxmrM1vTVdU9wF/1obOK7mzX1M3yP0jySeCqJH9LdxlwKfAy4Deq6syq2tPv+6kk24DrgdcB7wHO62+OP5Tunqsr6G64PxA4B/gx8P1+lN+hi6t1wGVV9cSvn5AkSZrNWMfWlKraDVwLXJtkxchLf0YXSGfR/fqHR4Dv0f3rwKl9P51kKfC+/nEfcE5VXdwveYzuDNf76M5YPQrcAry5qn7Rr7mGLvB2tfmEkiRpsZqI2BpVVVtH/lzAJf1jtn0upftdWzO9toMu1mbb39+lJUmSnpGxu2dLkiRpMTG2JEmSGjK2JEmSGjK2JEmSGjK2JEmSGjK2JEmSGjK2JEmSGjK2JEmSGjK2JEmSGjK2JEmSGjK2JEmSGjK2JEmSGjK2JEmSGkpVDT3DPiXJg8C9C3zY5cBDC3zMhTYJM4JzLjTnXFiTMOckzAjOudD21TmPrqrD51pkbC0CSW6rqpVDzzGbSZgRnHOhOefCmoQ5J2FGcM6F5pyz8zKiJElSQ8aWJElSQ8bW4rB26AHmYRJmBOdcaM65sCZhzkmYEZxzoTnnLLxnS5IkqSHPbEmSJDVkbEmSJDVkbEmSJDVkbEmSJDVkbEmSJDX0fzqt3pi0ldOtAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_attention(src, translation, attention)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how well it translates an example that is not in the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "src = ['ein', 'junge', 'mit', 'kopfhörern', 'sitzt', 'auf', 'den', 'schultern', 'einer', 'frau', '.']\n",
      "trg = ['a', 'boy', 'wearing', 'headphones', 'sits', 'on', 'a', 'woman', \"'s\", 'shoulders', '.']\n"
     ]
    }
   ],
   "source": [
    "example_idx = 2\n",
    "\n",
    "src = vars(valid_data.examples[example_idx])['src']\n",
    "trg = vars(valid_data.examples[example_idx])['trg']\n",
    "\n",
    "print(f'src = {src}')\n",
    "print(f'trg = {trg}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model manages to do a decent job at this one, except for changing *wearing* to *in*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted trg = ['a', 'boy', 'in', 'headphones', 'sits', 'on', 'the', 'shoulders', 'of', 'a', 'woman', '.', '<eos>']\n"
     ]
    }
   ],
   "source": [
    "translation, attention = translate_sentence(src, SRC, TRG, model, device)\n",
    "\n",
    "print(f'predicted trg = {translation}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we can see the attention is applied to sensible words - *junge* for *boy*, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApQAAAJ9CAYAAACLhOJ2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdebxv9bz48df7dJon0SCaECEZj3BNuSiEVGS8hOSSe7nCj6tr6rpdMl9DNyRzJVMZKiJDV1RUSBQiaZSkeTjv3x/vz7ezzrd9hn3W3nvtfc7r+Xh8H3t/11rf9X1/13d913qvz7QiM5EkSZJW1LyhA5AkSdLcZkIpSZKkXkwoJUmS1IsJpSRJknoxoZQkSVIvJpSSJEnqxYRSkiRJvZhQSpIkqRcTSkmSJPViQilJkqReTCglSZLUiwmlpFVKRKy2hOkx07FI0soiMnPoGCRpRkTEapl5S0SsA+wD3AD8NjO/3eZHelCUpEkzoZS0SomIdYHTgHWATYCLgS9m5mvbfJNKSZokq7wlrfQiYn77G8ArgPOBXYCHAV8CXhARHwHIzLT6W5ImZ/7QAUjSdMvMm1vJ5MuBewLfysxzACLiEuBvwL9FBJn5slFSaUmlJC0fE0pJK61Om8nVgPsD72iz9hstk5kXR8T/tqevjIiFmbmfyaQkLT+rvCWtlCJiXksmNwTeT7WbfFKb/U8Rce/Rspl5KXAI8F7gZRHxmhkPWJLmMBNKSSudVjK5MCJWB74B3B24S2YeBzwDeCjw+oi4x+g1mXkZ8HHgRcD7BghbkuYse3lLWilFxJrAVsBBwMHA6Zl5c5u3F3AE8BngwMw8d4LXzx8tL02HVoq+cGyabXc1J9mGUtJKp/XS/grVk/tc4ILWMWc1YGFmHtU6ch8BLIyId2bm2d11mExqOo3a97b/70wNMPBnk0nNVSaUklY6rZf2O4DbUZ1x7gP8ubWpnNd6cx8VEQkcCfwWOHspq5SmVCeZ/CjweGBeRHwf2NuLGc1FVnlLmvO6pT2daasDDwY+CVwJ7J6Zf2rz5lF5Z0bEPwLf9ySumdCt5o6ItwPPAQ4F7gDsDfwKeFZmXjhYkNIKMKGUNKeN2jpGxFrAI4CNgJ8B57fpD6Oqti8E9uomld32a7aZ1EyKiEcAewI/yczPt2mPoi6ALgaeblKpucSEUtKcNUoKI2J94AdUKc+6wNrUUEFHZOYZLan8HPBnKqn0RK0ZNVYy+XpqkP2FwG6ZeWabHtTdmz4LXAQ8w31Vc4XDBkmas1oyuQZwLHA58HRgW6rq8OXAWyJik8z8EVW1eEfgpIjYZKCQtQoaSybvRd3u83xqFIJdRstl+T/gucDGwLcjYvOZj1iaPBNKSXPdlsCdgA8BP83MK4C/AutRt1i8DKAllS8GzgSuGChWrWLaMECjZPKLwFeBP1D3lD8ZeH5EPK/7mpZU7gPcAKw5sxFLK8aEUtKcEhHz29/R8WsrqlTyl5l5Uzs5Hwe8MTM/FBEbRsQTI2LNzDwpM5/euR2jNG1ayWS2/3eiSshfSjU3Owt4FfAX6paf40nl94GHZub5Mxq0tIJMKCXNGa205+aIuD1waLut4jnA74HHRcTzgU9RyeRB7WXPpUp7Nu6ua7xXuDTVOiWTBwPPAq4CfpCZ17dk83Rgf+BaKql8ztjrr5/pmKUV5TiUkuaETgec1ag73GxF9ei+kKrGPhBYH3hzZh7USjC3BZ4JnEd1yJGGsBPwIOCX1Hl3dMemyMzTImJ/4B3A2yLi5sw8arBIpRVkCaWkOaElk2sCuwHXU1WHf8rMm4DnU4OTXwdcHRF3pe7Z/UmqLeVL25iTMUz0WlV097FRs4zMfDDwRWB7YN+IWLeVXsYoqQTeSJW2nzpA2FJvDhskaVaKiNVbsjh6Po+qzn44kMB9MvPaiFirVSGuS5Vc3o06cZ8FXADs2dpW3mbwc2kqjd1OcXVg9cy8tjP/BGqw/X8Fvtj23+4g+2tm5g2DBC/1ZAmlpFmnDfD8pja+JHBre7QvUZ0YtgH2aNOvj4g1MvMaYC/qNnaPAnalxvi7qQ1abjKpaTOWTL4P+Cbw64h4V0TsCpCZOwOnAx8A9oyItUcllW01Nw4QujQlbEMpaTZ6EHD7zPx7d2JmfikirgbeBbwqIi7PzOMy88aWNN4EXNIewK1tL70DjqZVJ5n8PHXHpk9Td2xaAOwaEVtn5ocz83GtpPLdwFoR8ZnMvK6twypDzVlWeUuaNSLiocDOwIGtCnANYAdgE+APmfmrttxTgTcB1wAHZeZxbXp4UtZQImJn4H+BFwI/yswbIuLpwFHA+4B/H/XcjoifUPv1/TLzqqFilqaKJZSSZpPnA1e2ZHJ94NvU2H1bAn+OiFOBPTLzmNb54T+A17c88niTSc2kCdrlbkINRP7blkzeHTgE+DxwQGuecc/MPCczd4yIrUwmNaSpvAi3hFLLbXzH6wzjYqmQplQrmfwe1Wv7HVTnmmdTd7q5CFjQks6nAAdQPblflJk/HihkrSJaJ5q1gBvHm1JExAuBj2TmWhGxKdVr+wRgn8y8OiKeC9wd+EC7o5M0mLFbgt4e2BO4NjM/uyLrs4RSy6WbNEbERlSHh9tFxEftlaipMHZhsgDYGtg7M09o8w8CfkrdYvFoqvf2sa037a7AaQOErVXPntRIA/8JXB4RPwTOycx9qIugcyPiGOCRwNeB/Voyeaf22ouogcylQXSOtau1m0O8DdgMeDpwQ0R8D7hwsgVFJpRaqs6Ot8bYjrcbNe7fN9tfLcNYUn7rlaHK2MFrLWAD4Gq4dQiha1tnhsOB50bE3TPz3Mz8EtX7e6IqSGmq/QH4BHDXNi7qltTdbgDOp/bFlwOXA6/LzL9FxFbAW6ghgx7rHXA0pFa7sxN104c9gT8CZ1DH2/dn5p9WZL0OG6SlajveY6kG5b8EHgBcSnWGOCIzTSaXQ+uBnBExv42XuMnQMc1yf6TGmvxHgM7QP9cAR1In8a3GX2QyqekQEWu33y2Z+RPgMVSp+MOAl4+aWrSLxAOphPMG4MSIOJbaZ3cBnpKZvxngI0gARMTLIuJwqn361sAHgR2B46mxe7/Vlpv0TSBMKLVEEfHyNgTGt6iT9/sz8yHA14CfU9U7K7TjrUpaqdnNrZPJF4EfAydHxGERcY/R3TRWNRExLyLmt/8X24cy8zzgYOAtEfHsNm3UXm0rqpTo8hkMV6uo9vs8CjgvIjZok7eh2vcCvCwitmjLRttP3wD8C/Bl6gL8KOCRmXnGTMa+LFG3MdUqICLWi4gDgP8C7kRVbz8nM9/WLoRe3Bb9AazYEFZ2ytFttDZp7wGeQjUq/wBwcmb+rc3/BnC7zPyH4aKcWyJiHeAn1MnlBOBK4HVUW6oDgaNWlY5NEbFZZnbHiVyPKgHfELiYuhf3FRGxNfB24FlUU4vvAetSPbtvAnay2YCmW0u6nkRd4NxAJYZXRcRmwD2pKu5TgJdl5h+Hi3RyxprgvAo41hqnlVtE3Juq+bk4M/862gciYneqpPI5mfm9FW2SZRtK3UarXvwAtYNd2na8eXDr+H/bAy9qz22ztnyeTd0N45XAL9qP+GbgUODmVSiZvAtwbEQcmZkHtpP1acDNVInjTsAjI2KfzDwtIv6duqh5LZWAXwmcC+zcRhiwLaqmVWbe0i6ibwA+AvwwIh7RLoouaeNMHg18JCL2zcwL20X5Ppn5kQFDX6Kx3r2HAQ8FfhQRv1tVjkWrkoi4c2ZemJlnd6YFVUt9C3VnsQuB8+DWphuTtkpWtWnJImLzqNvYnZuZvx5dxbDo1mCPAq4AfgO2WZuEewNk5s9bMvlP1Ph0b8zML0bEuq3T00qrdUzYikoInxcRrwa2A86mBjPfleo9Ow/4bEQ8NDP/mJn/SbXdfQJ1u8XHdtpUmkxq2oyqhNtx7iSqs806VFK5QZv3Xapjw0OAwyLiOdTF+IfaBdSs00kmNwU2Av4dOM1kcuUTEQcD74uIh3enZ7klIu4DvAw4NDMv7PNeJpS6VUS8nxoKY0k73g7AftQ4axcMEeNcsIR2SRdQPeXnR8RewCepgY4PaqUZ/w08dybjnEkR8UiqB2wCbwROpU7OHwQWApdk5jWtw8ITgeuBwyPiYS1x/F1m/jAzf9IpmfR2ippWueh2infPzBuppHI/bptUnkSNfHF/4J1Up50HZubvBwh7uUTE/1DDcN2DGvboFtvDr1wi4gvUxc53gD9PMH8NqvbsHKpvRC8mlAIgIo4Cngz8Avj1BPNXow6Y59B6gWli7cC8dqsKGzmPagj9WeAz1C3Y/rvN2446EW3ASigiNqdKYw/NzO+3apd/o9qd3RuY3zlxr9aukp9EJZUfBR4+QaedWV0y2S4StBKIiNcCv46Ih2eNuXsSEyeVJ1PJ2R7AI2ZbB5wJHA9cRQ20vg14L/GVSUS8iTqvPBP4RGb+PiJWb0NdjSykSqhPzcyL+76nCaVo7dQeSl2pfCQz/zza6doVDNS4gDsAP5zNV91DGiuZfC/wjoh4CUBmfo3qePIMariGw9trHky1owyq0f/KaDVgbWqcMyLiZ8CmwFupbfGEiHg93JqMj5LKXYE7AC+dCye6iFgtIhZExKaZeVOb9pSoO1Bo7jqTKuE5cjmSyr+1UvRLB4t2AksYSeLbVE/0S4A3RIRDma1c7gr8IDNPzbrl572Aw4BvRMQ7I2K9VsvzEarKu/eILfbyXsW1JOhj1O2W9mvT7kn1PF4f+Avw6sy8JCLuRvUOu6bbQ1CLOie1HstPp0pzHw38DXh7Zn6sLfefwKup5GpdKpG8BnhUaxe40nVyiogtqYPWxtT4kb8BnpmZl0bEPYA3UeP5fTgz391eM9qemwBXzIVtEhEPpHqg/xF4PdX7947ALrMtwdDElvT7i4hHU9/t9sDTM/PkdtG9E/B+6rd878z8+0zGuzy6nykitqMu7n6X7R7iEfF44HPA6cDzMnPQ4bjGO9rZ8W5y2jl9Naqj2M3Ax4H7Uvvvz6mmR08F3p2ZB0zpe5sTKCI+Rt0m7J+pJOj1VHXkX6nqkB8C/9raEGkJogY+Po1KKE6hei2/gapWem9m/m9b7ilU55TNqdKPL7Xkaf7K2i6wfeYjqJ6y+2fmJzrz7km1q/wH4EOZ+Z42vdsTdU4k2hHxHupiYiFVqr9Lt2el5oaIeChwVmZe25m2E3Xf+O2BPTLzRy2p3JkqbX96Zv5uiHiXZOw39CmqU+VGwI1UW88vZeZvW1L5GeBnwHMz8y9DxTwSEf9LDad2oknl5EXE/al2kUmNjvHpzHxnK4X8LDVM21OmcruaUGp0Qv8INVr+L4EvZObBbcf7MtW0ZvchY5wLIuJfqBPOY0ZJRERsS3XA2Rg4eFRSOcFr50TCNFmjE0FEHEK1i7ycGhD6nZn51c5yo6TyIcBnM/OtgwS8groXAxFxNjU+4ZHAK1tJ7Kw5IUYNsP986k5XgycOSzLUb6K1mXwH8E9UwnVdZ95jqY5kGwC7ZQ1ttQawetZdnGallpztAvw/6qYAD6eG4foRVd15MfA4qinOH4FdM/OKQYKlhrmhStjuS12U/XA2/YZmo9ZmfyvqYuG7mfnLqLFS16dyvXPbcnegLu5/RR2fpiwJNKFcBUXEM4EtqBP76dluGxYR9wX+OurBHTWMzeFUG5tXALdYzb1kEfFWanzOrTIzo4ZfurEllT+gOpkclJmHtuVX2mYDSzr4txPy26kSvHeMJZXbUQPqX0NVic+JbTOWTO5AnbShqvG/Rn3nF8+GE2IrRT+H6ny3Z9b90QePa9xYNe2Tqf3losz82Qy9/4lUm/FXA18cSyoPoAbavwl4dGaeMhMxrajWKe544ENU54wbI2IjqjnTB4DXZ7u3eETsSrX/flzO4CDtEx0LI2J74CDgsZhULlVUp9qHAaPOgBtR95c/bKyUfTvq+LQr1czqNh1we8lMH6vQA/gCdbeW86krmQuA902w3AOoziKXA/ccOu658KDuLLSQKqGEah+5Rvv/NVS1w8lUddng8Y7FvtoUrmt++7sGcBfgfmPzd6aaBJxMlfJ0520NzBttv6G3y2S2G9UO7WOd+N8D/JY6ad+xs0+sBqw3ULxvpZLJ0X65ydDbcBnxHtl+N1dTF2Qvm67vb4J536GSrucB63amvxI4Dvg0cI+ht9FyfMb7tePSw9vze7fPdRSwTpu2I7Bm2z/XmeH45nX+X3Ns3n2oi7JrqJ7ziy3vIwHeQg1K/liqE+N9qHa9NwOvassEVer+M2oc4PtNSyxDbwwfM/dgUYeBR1ENs7eirkYvpTpEjJZ7WTvh/3q6drwp+CyDJRtLOgkB21K3VfwONQZdd95B7QD+c+BrA8Y+j7o63asz7Z3UYOFT9r1Q1Sw/pkq3F1K3TdxntO2Ax7d97IfAUyeKc+h9bDL7AdUBZyF114mHdKaPksr3siipfAc1BueUJfGTiPlV1I0J7g18k+qIscbQ23IJ2/TJVBOcx1IXIe9q2/eAqfj9A2t1/n8q1Qxgu+72aL/ly4CXtJP17ahODq+jk2TOxkfnt3g74HdUj+57tu//SNpFDTVO4VHA1gPHezCVHK03Nn174BvURcVD27RZf3yYoW22BpVwf2r8u6fGNr4JWNCmPaht37tMWzxDbxAfM/doJ72vUO19RtM2paog/0SNVxVUW7dXANsMHXMnzhk/+S4tDhbdU/oNVCP20fznUj24T2onwjWoEoKTqd7fT2iJx4KB4l+XagP2K+DFwFepzlf3nsJtM4+qYvsu8AKq7dYPqGT63SwqwfzHtl3OpapfBv9+V2R/bNvw99SwW1dQ1YXdZd9FjUN6CtUmeSGwwwAxB1US9X3qIvJCZmkJG3Xx8T/UUFujEt/1qQ6Dt7Tf3qSTSupC+ghgs860I4G/A9dSncYOoJNcAccCF1GjE/yYGrlhu6G30dL2yQk+85fb7+wKqo386Hd6B+AT1MXF7QeO/8T2O9ofWH9s3k7tO7qMamYw+PaeDY/2mz4e+MoE0+9M1UZ8hHbOZ5oT8cE3iI/pf7QDxxpUA+zPt2nzOwfqO1MNtd/ffc3QcU8UC7AvNaTRM4AtZziO0RX/qB3a76kSuL9SPehGyz2HqhJbSDV2v4K6I8V8YPd2Mr/7DMc+n0XVW5u0E8iV7QD9wO7n6/k+a1O9nD9Lq6Jq0zdqB7afAy/qTN+FaloxKy4YVuDznkQ1H3lQ+42dQ93DGTrVd9S9yI+mShO2Hzjms6jk6Red7342/d6f0PbLPwKva9NGv711qIu4W6jbBU5qn6VKOi+lRmPYmBpZ4GyqQ8r21BBWC6mS5bt2XvdK4MNtX511TYCWcozcpk27E5UQX0+Vto4uLg6jmjXN6D7Z/d5Y/OLsqPZ7ei23TSq/TSXzF7X9YNY3iZmJbUi1jb0QuM8Ey/yIdgExIzENvVF8TPMXDOt1drzXUFfgo7Y08zvzvk5VK8zaE3s72IxKVi6nkqIHzNB7d0vf9m6JwV2oau5/oapjvthZfpt2YjygLT96/ZHUUEEbz+B2W5tK5J7IotLBw6mk4lzg5Z1lVzixaNvmQ1QCfQVwtzZ9zfb39lQV93eXto1n84PFS/fvQZVGd6u4z6Y6G91me7aT+FrTHeMEMc/vvP+GwDFUCd8pVInbg0bzh96+nZhfRV2s/WY82WnJxOuoxO+1k1zvPCrR+gV1+8/9qREHusu8mkVJ5V3G5s3qfXQJx8gHt3l3omoEzqGGMvsF1aRgRps1jW/D0f7Zef5FKql8HYsugu9KlbI+iU7p8qr4oIab2wzYoD1fl6oB+QGLXwTdkWpqdHDb76f99z34xvExjV9utY17J606k2ofdByV0HRPgpu0k8ttOucMHH/3yvWxLcadqARpn/ET4gzEsw7VPOBoarDy0fT1qaTx78DRS3jtP1BVbX8F7jvAtjx4dABqzx9NlcqcQJVY7deZ1yepfBKVbC+k7nAzmj5KKkcdl+7V531meNvNo0qWd+pM+29q7NbRCW90YfZd4KMT7BtbzYLP8RI6nXCAZ7Xf0E8YKKkcTy7G5r2KuuA5irEq5nYS/TfgXpN9r/Z9PpOqNbgJ+K82vdumcpRUHkwnqZzp7TOZ7beMY+So7eHtgAe2ffLBtHa9A8X7Jqom46dU05htO/OOotp9fooaOeMz1EXxpkNv84G/749SpetXULdAHtWG7Ni212+ojnevojrg/nX8tzOt8Q29gXxM0xdbP8jz2o9288703alquiupKqMDgM+354NX5TBBD9/2A3kb1dZnfmf6M5nBpBJ4TDsBXQ38x9i8ddtB+ipqfL/uvNtTif3xzHDbufETNlVt988sSoS2ZOKkci2qt+DqS1n3hAlh204nUSVMe4zNewlVZbX10PvaJLbhZlSnmsvaSftLVBORe3aWGSWUH6c6coxKAz9GJSaDJpTUrVUXtt/Qdp3pz2agpJLFk4v9qJ6pB1BDRo2mv4YqUbtNUrmi79Wez6MS6rOo9uObtOndDjmvbNvs7YyVog38Xa7oMXLHgePuxnsEdX76IDUKwlVUk5j7dZY5mEoi/0IVgszKDqIzuP0+RY3K8tK2b76r7Z9vbPM3odpzn001FzmJGS68GHwj+ZiGLxX+q53wHkLricji7bnuQTV2v4Tqgfrdmd7xlhD3OlQ1TPdEvR7V23wh1Th+3thrRgfMk+mUuk5RPN0SjdFBfBcqIfo1bXigzvLrUlfaC4EDx+ZtBGw0C7bxadTFwwtYVGWyNZVUnkGV+mxOtRX7Lp1SzbH1jKpR16JGDXgai7eZfAx1BX0pNWrAvdu0H7cD3awvnRw7MT+EqnK7hmoXu80SXvOOdkAftRm9irEe/wN+nue2ffMwJk4qp/w3tJRYusnF0cCf237x07Z9P9aZ/2oqqfwcK9B5jMUT19dRJXcbsqj6+9y274+Syu6x8uUr8p7TuN1m1TFyBT/DgS3eUVX8i1rsl1NNce7TWXZLqm3rrB7eaga22YPaceWpLOpgs2Pbbh8b22c3bcfw9Wc8zqE3lI8p/kLrDg7HAm/qTLsLcEg7IL+RRUnmplQSNMiYeBPEfi+q3dIGY9M3pdoCXU1VqY6XNjyjnXC+zdg4ZlMQ0zpU1cGTWZRUPokq1TiWThLV5q3X5g9eojG+nTrTv04lOnuzKKnciqqqvoTqbHQxS+iJzuK9bk+nqloWUiUJxwG3a/MfQ7XhWUg1pv8M1X5vze56ZuOjfe9fB57cmfaZ9lkuHU2nSiKjs01eTJW8fIlqozrjyeT4dqU6C41KUJeUVD6z+xtiGkopJ/q+qZLJ31EXJaMLuANajM/pLPfqtm8exlJKzSdYfzdxHVWjvovWDo9KKveikrSf0do2T/VxZAq34ZQcI6fj+12e75260DoKeEF7/lpqvMQnUe1ZF1JJ5eAFHLPpQTVjuBp4WHu+LVXt/VkW1TbNSH+CpcY5dAA+puiL7CSFLOpg88B2IL6WKhk6jWpj8W/tQDrrGpizeM+1h3embwL8H5Xs7DTBAXN3pmGYI6q6+nqqBOPxnZPeU6mk8muMJZWd1w6WVI4dxF9JXUi8rDPta9w2qbwjdbu5V9I61Cxl/WtQHWxOotpjbk8NNfUHqpRpw7bcY9qJ7hzgn7qvH3pfW8bnuydVit8d0Ho3apDrY9p3/9TRPtvZbx/DovEoB62io0pL79X+X53bJpUfp9MGkRqPcJtpimUtqhR8FxZP8g6hSrJHJ8VtqJKqTzM2wHbbv5a6Xy7l/d9OlYI+jEUX1KOLgFGbyl9QVf+zup3ebDtGLi3OsePQ3lQy+wTqTm2Poi7O9uks8zWqQ9ERzIImWAN/z91z+uOohHLr9j2PjyW6B2PDYQ0S89AbzccUfZE1tty72/9Po4YLGA0NckCbvjp1sj906HgniL97ktmKSuD+RqsWadM3aZ/r/IkOmFP03t3/R1ULm1KJ0s+5bVJ5AdVu5R+H3oZL+Gyfo6q4f0MlOceyqBTmWBYllcusHhnbNgvaZ++W4K1B3SP4j8A3O9MfR5Vcng08Zehtsryfk0UJx9uAl3TmP6ptu1uTys7n34kqdRt0fEfg/lRicS6tswOLJ5WvopLK9zED7Xqpkqkz2r7xGBY1mTgKOLb9f6/OiXKU9L0GeEbP916Har/8QW5bettt1rIn1Zzle8xQr9gV2S/b/zN6jJxknGswNug7Vbp/Fou3kfzXdky9U2faV9s+e053+qr4oHNOb89Ppmo//kqV4q7fpm9GlVR+fnmO49Ma89AbzccUfInVW++XtPH9qOGAtqJ6Fo9KKKId1I+jrtZvLVUZ+jHRQY8q0j+e6jm9Y2f66IB5LpXc9a42ZVHpyPzu3/b/qHp203YyHE8qn0JV2byzbxxTtC27JQJbUx1EHkKVCDyOKv05aXSwphKjv9DpqLO074hFt+xbQKte674vVQLxSqoDS3ckgcdQJecXjV4zWx8s3uZuM6pDwBnAizvTH82iksrdqKTlg+0z3mHI770zbRdqaJzzaOOedvbnbahmDQtb3Mtdjdwjxk2owasvZtHtSV9BdXR7ClVa9QUWnSi3o5oO7DfRMWJ5twXVXvKPwHuW8pp1WNSbf4VKQWdqn+xMm7Fj5CTi3KC9d/e3cqf2vT+BxY+tbwau7zzfmLqYeBBLaLu9qjxYdE5/IYsuAp9C1TJeRRt5gOoP8Yl2XB28RHfwDedjCr7EanP0Z5YyWDbVKeJQBhhUexmxd0/eL6RuXP86qjH2A9pJe/yAuTF1BXsmPe87297jB7Se8FRCdDLw6s4y3aTyAqqt1S4sSrIeMZkT3gxtyzXaQel4Op2BqIbcl1GlMKPP/P12wt1wCesdHdDWokq4H0u1y72cCcZcZNG9g580tp5dqDHmZt0Jewnb8EntO78LVS37Uxavnns0deephW2fuIoB2jGNxXz/duIeDRX2yBb3eXRKTdtJ+wNUh5xp7XTC4knEltRF2S+oi4zNqWRjIfCNznKbUFXyv2IS1bRj2+KJtLZ4VPOMbzHB3WDaNpjSe4RP4/c748fIScS5BnUs/QmLLgo+QiVBp7DoInZ0nLhX2y/Pogar/0o7Nk3brQHnyrmM+IAAACAASURBVIMJzult+z6Lurgd3SzjdKom4v5Dx5xpQjnnH1Rbr4uA/dvz21TVUG0mv9N2vFk59EJLNP5IVYue3z7TP7eTwqi9X7dq5w6TOdEs5X2fR7sbR3u+IYtuRzjROIoPpUrmTqBKprolgoMnlS2OD1Jtqb7WDvCjdjajBHiUVJ4IbNGmbTm2ju2Bx49NuzeVRI4S0VdSpbPPH1vucVTnh38Y7ZOdeTNyclvB7dY9cX+6HbDf3J7fr22v8aTyPlSv4f+iM47eDMbc3bafopK1a6mSyUPb9Ee0E8+fqGrQh1EXl9+b7n2WxatpD6V6yv+KSiDPa/E8mbrwubr95v+zHQ8mNWbrBN/f6bThvajq7NF9wNfpLDfqJPJlZv+9uQc5Rk4ivh3a7/4J7fm+1GgSl1MJ0IKx5edTx9CT2uc5ZTLf98r6YOJz+ujYHVRp+iuoC4tnMcN3jFtq7EMH4GMFv7hFO9jz2gFm1PtrdPW3Ie0OE1QbuTcxS0uGqKvTC6hkbdT78th2EHoidRV+AlU1+w/TFMNaVNXBPdqB+HDqqv+lY8ttw6IhOj4z9Lbr7gvt/w9Sjdo/TV1ELDaEUWf/eHCb9zVu23h/TaojzQXAzm1aUEnVpZ3v6A7UuJY3U2PGPY2qMvwxVcq72F1iht5Ok9ien6GG03oqi9/zeQcqqTyTTpvK2fBo++75VOnkHahSq4XAl9r8B1A9fBdSvaUvZgYvLtt+eVHbR3agelb/hEUdSHZo+9CZVDK8WIehFfz+nkJn4G4qmbyFam/2bKpz0tHtuDJrhgZawmca/Bi5HDFuQF3QnMyiUQ7uT51/rqYueNZuy3YvNOZRt/8dtP3f0A+WfU7fiIFv27rMzzB0AD56fHn1Q/w18NnOtPWpqrpvtpPHv7RkYNrbSPX4HJ9qJ8RR+7wt20nvcyzqGHO/lqj8kWm4fR2wa1v396jbfHWTym7v6AVUR4ZNmCUlkp3YHkj17B21a7wz1V52Ia1j1mi/6Sw/YecRKrE+vp0gdmnT7kl1Ati2c/DbmCqpvKSdNP5AJS6j721WbaPl2IaPpxKzf+xMu33b/7ahesKfSFXj7TdUnGMx705ritCevxK4kRqf7lI6d2+iSgOfwAyWarTtdwrwrs60aL/zk6nkb9SmcjQ6wAqNArCE7+92VIn7I9qx8QJqPNELmCOlYrPhGLmM+EbNYh7d9r2raW2oqYv1F1G3/f0wi4+ZOGcuNGdoOy7rnH4z8IbZuv0GD8DHCnxpi368L2oHxAe05/9ODRd0C9Xj66XM4kSyxTyPasP35fb8rlRV15Es6iyzD3A3qs3NtN1xhCqxOK2d5EZJ5ceoTgMfpsZMOwU4sfOaWZEwtdguo9rddAcG3ozqpTxhUrmMdW5LJU8/p6qxN2sn4c0nWPaO7YR2XxYlrIOPxbkC23FPqtpuQ2qM1sdQPeQvZNEF2uZtP/keS2h3OoPxBtWm9TXt+YuB66gSwPWpNmwLqWrdQU4+VIn3mcDhE8zbqSUfPwN2pnPf8Sn8/s6lqvsXUoPsz6Muju429Pe3nJ9p1hwjlyPWg6jS78updqujY8HaLJ5UzuphwwbYbpM5p8+Kc86En2PoAHz0+PJqHLJzqFuWndoSikMYG8JmqBPJJD7H+9uPaFcWDRsyKqm4J9Ue8NnT+P7d6pd/otpejZLKdYDXs2jonRNYVCIwa7Zr207HU1ewzxqbtymVVN4IHDTJ9W5LVZ3/kmoneFE7yL2SGvbjuVQ15r/ShiNqr5u1g5Yv4/Nu1056x7QD+TVUydBuVJXpTVTyvD2z5PaR1GD6m7S/p1BtEEeJxt1YlAx/eaD45lNV2GcwNkQRlfSd0eI7h/6d7Jb0/T2tfX83MgdKJCf4XIMeIycR59ZUu+LdqIvbH7LoImGUVI6qv00qb7v95vQ5ffAAfKzgF1cntIXt8VXqqm+z8WRntu54Y5/lPu0gs5BqHD86AG1MlRCexTRX0bHkpHK7Nm0Tauid0XaddaVvVG/kk6l2aU8Ym7cpdYeQv9JJ/JZzvdtSifTojjhHsajU7tdUNfepzOIr50l+3p2oJPoDwHM70/egOpLcdegYlxD3naiSuNd0pu3Z9ol9hoyb6tB1NTX4cre3+V2p3r0LgDvPwPd3LjPYUWUKt9/gx8hJxrs6lcRPlFS+nCrFvOOQMc62x8pwTh8FqDkmItahThJXUgMD/7VNj5yDX2pEPI46sfyYOulANTZ/DPDozDxrBmK4ddtFxOiOMTcAL8/MM0fzI2JeZi6c7nhWRETcjepNuwnVe/24zryNqYPRZSuw3rtTHSu2poZU+kbbB6GqNK+c7dtmMiJiPnBLZ3/YlGqPugOwa2b+Zcj4JhIRo/EHz6A64V1H9QRdF/jnzLxmwPCIiF2oZOh06rd+AdX+88HU3aYunsL3mnPf37LMhmPkZETEmlR8H6YuRnfKzJsjYi2qHeXfBg1wllkZzukmlHNYRKyWmbd0ns+ZHW8iEfFgqg3OFlSbkV9Tw378cgZjGE8q/5U6Me+Tmb+ZqTj6iIhtgf+lSiX3z8wTpnC9h1BVvvtn5vFj22ulSCbHRcRzqM4eT2MWnri7IuKRVNOHa6j9dm2qs86siDki7kdV592DKsW6AtgrM8+cxvecM9/fssyGY+RkRMQaVIeSD1DDGt2ve86aqyLiEZn5w2lY75w+p5tQalaJiLWpAVxvAW7KzBsGiKGbJD2Xuh/6T6iSuetmOp4V0ZK/D1FVZc/PzBOnaL13o5LK+1K3HfzxVKx3toqIh1IJ0I1Ub/9fDBzSMkXEDlQbtuuAr2bmeQOHtJiIWJ/qNLMR8OfpLC2ci9/fssyGY+RktKTyacBbgSdm5vnDRtRPRDyWGiT/tZn57qHjmU1MKKUJjCWVn6B6Tz56th+8uyJiO+CdwL9l5u+meL37tfXO+dKGpYmIeVRp2l9WpKmAhuX3Nzu0pHKNzLx66Fj6iojbUYUMn8vMc4aOZzYxoZSWoNNm8lDg4dRAs1cNHddkRMQamXnjNK5/tZU9qZSkrpW1eU9f84YOQJqtWjK5BnXXgmfNtWQSYDqTybZ+k0lJqxSTyYlZQiktg1ejkiQtnQmlJEmSerHKW5IkSb2YUK7iImLfoWNYHsY5teZCnHMhRjDOqWacU2cuxAjGOdWGitOEUnPiB4JxTrW5EOdciBGMc6oZ59SZCzGCcU41E0pJkiTNPXbKmUMiwi9rCj3oQQ+a8nVedtllbLLJJlO6ztNPP31K1ydJ0iRdnplLPbmZUM4hJpRTa67s+xExdAiSpFXb6Zm5YGkLWOUtSZKkXkwoJUmS1IsJpSRJknoxoZQkSVIvJpSSJEnqxYRSkiRJvZhQSpIkqRcTSkmSJPViQilJkqReTCglSZLUiwmlJEmSejGhlCRJUi8mlJIkSerFhFKSJEm9mFBKkiSpFxNKSZIk9WJCKUmSpF5MKAcQEQ+LiGMi4qKIuCYizoiI5w4dlyRJ0oqYP3QAq6itgZOBQ4DrgYcDn4iIhZn5+UEjkyRJmiQTygFk5hGj/yMigO8DWwAvARZLKCNiX2DfGQ1QkiRpEiIzh45hlRMRGwFvBXYD7gys1mZdmJlbLOV1fllTaK7s+3XNIUnSYE7PzAVLW8ASymEcDjwUOBA4G7gKeBmVYEqSJM0pJpQzLCLWAp4M7JeZh3Sm20FKkiTNSSYxM29NarvfMJoQEesDTx0sIkmSpB4soZxhmfm3iDgVeFNEXAUsBF4P/A3YYNDgJEmSVoAllMN4DvA74FPA+4Evtv8lSZLmHHt5zyH28p5ac2Xft5e3JGlgy+zlbQmlJEmSejGhlCRJUi8mlJIkSerFhFKSJEm9mFBKkiSpFxNKSZIk9WJCKUmSpF5MKCVJktSLCaUkSZJ6MaGUJElSLyaUkiRJ6sWEUpIkSb2YUEqSJKkXE0pJkiT1YkIpSZKkXkwoJUmS1IsJpSRJknoxoZQkSVIvJpSSJEnqxYRSkiRJvZhQSpIkqRcTSkmSJPViQilJkqReTCglSZLUiwmlJEmSejGhlCRJUi8mlJIkSerFhFKSJEm9mFAup4g4PCJOGzoOSZKk2caEUpIkSb2YUEqSJKkXE8pJioinRcQ5EXF9RPwwIu7dmbdORHwgIi5u80+NiJ07818eEVdHxHpj69wpIjIi7jeTn0WSJGkqmFBOztbAe4ADgecAGwLHR8Rabf5HgRcCbwd2By4Avh4Rj2jzPwesBjx9bL0vBH6amWdOb/iSJElTb/7QAcwxGwO7Zeb/AUTE6cBvgb0j4nvAs4EXZuYn2/zjgbOA/wB2ycwrI+KLVAJ5eFtmPWBP4PUTvWFE7AvsO50fSpIkqQ9LKCfn0lEyCZCZfwBOB3YEHgwE8IXO/IXt+SM66/g48MiIuGt7vheV2H9uojfMzEMzc0FmLpjKDyJJkjRVTCgn59IlTNu8Pa7OzGvH5l8CrBMRa7bnJwG/A/Zuz18IfDUzr5jyaCVJkmaACeXkbLqEaRe1x3oRsc7Y/M2AazPzBoDMTOAw4PkRcQ+q9PIT0xeyJEnS9DKhnJxNI+IfRk8iYivggcBPgFOBpNPhJiKiPf/h2HoOB7agqr8vBL41rVFLkiRNIzvlTM7lwGci4gDgOuCtVJX34Zl5fUR8HvhgRKxPddZ5CXBP4GXdlWTmnyPiOGBX4KDMvGUmP4QkSdJUsoRycv4AvAZ4C3AE8Heq9/b1bf5LgE8CbwK+Sg0z9OTMHC+hBPhK+2t1tyRJmtOimvRppkXEUcDmmfnISbzGL2sKzZV9v1pOSJI0mNOXNdqMVd4zLCJ2ABYAewDPGjgcSZKk3kwoZ96x1ADpH87Mo4cORpIkqS8TyhmWmdsMHYMkSdJUslOOJEmSejGhlCRJUi8mlJIkSerFhFKSJEm9mFBKkiSpFxNKSZIk9WJCKUmSpF5MKCVJktSLCaUkSZJ6MaGUJElSL956UausiBg6hOWSmUOHsExzZVtKkqaHJZSSJEnqxYRSkiRJvZhQSpIkqRcTSkmSJPViQilJkqReTCglSZLUiwmlJEmSejGhlCRJUi8mlJIkSerFhFKSJEm9mFBKkiSpFxNKSZIk9WJCKUmSpF5MKCVJktSLCaUkSZJ6MaGcZhFxeEScNnQckiRJ02X+0AGsAg4E1h46CEmSpOliQjnNMvO3Q8cgSZI0nazynmbdKu+I2DsiMiJ2iIhvRcQ1EXFOROwxdJySJEkryoRyGJ8DjgF2B84FjoiILYYNSZIkacVY5T2M92bmYQARcTpwCfBk4JDxBSNiX2DfmQ1PkiRp+ZlQDuOE0T+Z+ZeIuBSYsIQyMw8FDgWIiJyZ8CRJkpafVd7DuHLs+Y3AWkMEIkmS1JcJpSRJknoxoZQkSVIvJpSSJEnqxYRSkiRJvUSmHYfnCnt5r5rmwm80IoYOQZI0fU7PzAVLW8ASSkmSJPViQilJkqReTCglSZLUiwmlJEmSejGhlCRJUi8mlJIkSerFhFKSJEm9mFBKkiSpFxNKSZIk9WJCKUmSpF5MKCVJktSLCaUkSZJ6mT90AJKWLiKGDmGZMnPoEJbLXNiWkjQXWUIpSZKkXkwoJUmS1IsJpSRJknoxoZQkSVIvJpSSJEnqxYRSkiRJvZhQSpIkqRcTSkmSJPViQilJkqReTCglSZLUiwmlJEmSejGhlCRJUi8mlJIkSerFhFKSJEm9mFBKkiSpFxNKSZIk9bJcCWVEHB4Rp013MEt5//UiIiNi70m+bqf2uvtMU2iSJEmrPEsoJUmS1IsJpSRJknqZVEIZEY+PiLMi4pqI+GFEbN+ZNy8iXh8R50XEDRHxm4h4wdjrd42Ib0XEpRFxVUScEhE7T/A+e7bXXxcR3wfuOcEy50fEuyLiPyLi4oi4OiI+GxEbThD6xhHxhbbM7yLi5ROsb6+I+HmL/YKIeHtEzO/M37tVn+/QPsM1EXFOROwxwbp2i4jTIuL6Fts7I2L1zvwtIuKoth2ui4jfRsSBS9v2kiRJs9VkEsqtgIOBtwPPBjYFjoyIaPP/BzgAOBTYFfgycFhEPLmzjrsAxwL/BOwJ/B/wzYh4+GiBiHggcCRwJrBHW/6oJcT0bOBxwEuAV7f3/dgEy320rW934CTgQxGxY+c9d27v+VNgt/ZZXgN8cIJ1fQ44pq3rXOCIiNiis669gC8BPwGeCrwV2Bc4qLOOTwFbtulPpLbpmkv4jJIkSbNbZi7zARwO3AzcvTPtaUBSpYfbAguBF4y97lPAqUtY5zxgPnA8cFhn+lHA2UB0pr2xvdfenWnnA1cA63WmPbfFca/2fKf2urd1llkduAz47860U4DvjsX3OuAWYIv2fO+2rhd1lrlD2y7/3J4H8AfgE2PrehFwHXCH9vxq4CnLue33BU5rj/ThYzY+5oqht5MPHz58zNHHabmMfGUyJZTnZ+a5nednt79bAI+lErkvR8T80QM4Ebh/RKwGt1b1fjIiLqQSsZuAnYF7dNa7I3BMO/iPfGkJMX0rM6/uPP8yldQ9eGy5E0b/ZOZNVMniFi2m1YAHAl8Ye82RVNL7sKWs6y/ApaN1tc+xFXDU2Hb4DrAWMOptfgZwUKtG32oJn230Hodm5oLMXLC05SRJkoYyf9mL3OrKsec3tr9rARsDqwF/W8JrN4+IP1NVxesDbwLOA64B3kZVn4/ckUrSusafTzg9M6+NiKuBzZcj9rXa/xtTpZaXjC0zen77Sa4L4BtLiHfL9veZVDX3e4HbRcSZwP6ZeeISXidJkjRrTSahXJorqBLHh1MlleMuparFHwA8MTOPG82IiLXHlr2YxRNMJng+4fSIWAdYD7houSOHy6mS0vH32Kz9vWIS6xotuy/wswnm/x4gMy8E9o6IeVSJ7FuAYyJiq1bqKUmSNGdMVUL5HaqEcsPM/NZEC3QSxxs607amktCzOoueCjw1It7Qqfa+TU/q5vERsV6n2nt3Wl3/8gaembdExOnAM4CPdGbtRSXHP1redQG/Bi4EtsnMjy7Hey8ETomIt1IdlLYGTCglSdKcMiUJZWb+OiIOoXo8v5NK6NYCtgfukZn7AOcAfwLeHRH/QVV9v5VKwLreAfyYaof4card4YuX8NbXAV+PiIOpau6DgS9n5tlLWH5J3gwcHxGfAI4AdgAOBD6amX9a3pVk5sKI2B/4dERsAHyTqhK/K9WJ6elU9frxVIel31C9u/enSmZ/Ncm4JUmSBjdVJZQA+1EJ0kuodpFXUR13Pg6QmTe0MRs/BBxNJZdvp3pi33prxMw8LSKeRQ2z8xUqOX0mNQzPuCOAv7f3WI9qo/myyQaemSe09zyA6il+KfBuKtGc7LqOjIirgH+nenffAvwO+BqVXN4C/Bx4JdWm8lqql/nOmXndZN9PkiRpaLF4Z+q5IyLOB47OzNcMHctMiYi5+WVppTdXjiOLhs2VJE3C6csabcZbL0qSJKkXE0pJkiT1MpVtKGdUZm4zdAySJEmyhFKSJEk9mVBKkiSpFxNKSZIk9WJCKUmSpF5MKCVJktSLCaUkSZJ6MaGUJElSLyaUkiRJ6sWEUpIkSb3M2TvlSJo9ImLoEJZLZg4dwnKZK9tTkkYsoZQkSVIvJpSSJEnqxYRSkiRJvZhQSpIkqRcTSkmSJPViQilJkqReTCglSZLUiwmlJEmSejGhlCRJUi8mlJIkSerFhFKSJEm9mFBKkiSpFxNKSZIk9WJCKUmSpF5MKCVJktSLCaUkSZJ6MaGcQhFxUkQc3Xm+c0S8asiYJEmSptv8oQNYybwcuKnzfGfg6cD7hglHkiRp+plQTqHMPHvoGCRJkmaaVd6TFBHbR8RxEXFFRFwTEb+KiP3avFurvCPiLcD+wNYRke1x+LLWIUmSNNdYQjl5xwK/Ap4H3ABsB2wwwXIfA+4O/COwe5t22STXIUmSNOuZUE5CRGwM3AXYLTN/3iafONGymfmniLgIuCEzT1mRdbTl9wX2nYr4JUmSpoNV3pNzBXABcEhEPDMiNp3udWTmoZm5IDMXrMB7SZIkTTsTyknIzIVUz+2LgcOAiyPiBxHxgJlchyRJ0mxiQjlJmXlOZu4J3A54HLAW8PWIWO5tORXrkCRJmi1MYFZQZt6Umd8B3gNsTiWH426kksU+65AkSZrV7JQzCRFxX+BdwJHA74CNgP8HnJmZV0TE+EvOATaLiL2BXwCXU725l7iOGfgYkiRJU8qEcnIuBi4B3gjcCbgS+C6VEE7kKOAxwDuBTYBPAq+b5DokSZJmtcjMoWPQcooIvyyph7lyvJugtkOShnT6skabsQ2lJEmSejGhlCRJUi8mlJIkSerFhFKSJEm9mFBKkiSpFxNKSZIk9WJCKUmSpF5MKCVJktSLCaUkSZJ6MaGUJElSLyaUkiRJ6sWEUpIkSb3MHzoASZop8+evPnQIy+WMP/xh6BCWy4K73X3oEJbL6quvOXQIy2XevNlfxnPNNX8bOgTNUrN/75UkSdKsZkIpSZKkXkwoJUmS1IsJpSRJknoxoZQkSVIvJpSSJEnqxYRSkiRJvZhQSpIkqRcTSkmSJPViQilJkqReTCglSZLUiwmlJEmSejGhlCRJUi8mlJIkSerFhFKSJEm9mFBKkiSpFxNKSZIk9WJCKUmSpF5MKCVJktSLCeUUi4i9IuLnEXFDRFwQEW+PiPlt3t4RkRGxQ0R8KyKuiYhzImKPoeOWJElaUSaUUygidgaOBH4K7Ab8D/Aa4INji34OOAbYHTgXOCIitpjBUCVJkqbM/KEDWMm8DTgpM1/Qnh8XEQAHRcR/dpZ7b2YeBhARpwOXAE8GDhlfYUTsC+w7rVFLkiT1YAnlFImI1YAHAl8Ym3UktZ0f1pl2wuifzPwLcCkwYQllZh6amQsyc8HURixJkjQ1TCinzsbA6lRpY9fo+e07064cW+ZGYK1pikuSJGlamVBOncuBm4BNx6Zv1v5eMbPhSJIkzQwTyimSmbcApwPPGJu1F7AQ+NGMByVJkjQD7JQztd4MHB8RnwCOAHYADgQ+mpl/ah10JEmSViqWUE6hzDwBeBawADgWeBXwbuAVQ8YlSZI0nSyhnGKZeSTVs3uieYcDh08wfZtpDUqSJGkaWUIpSZKkXkwoJUmS1IsJpSRJknoxoZQkSVIvJpSSJEnqxYRSkiRJvZhQSpIkqRcTSkmSJPViQilJkqReTCglSZLUiwmlJEmSejGhlCRJUi/zhw5AkrS4+2+99dAhLJfNNttm6BCWy7dP+/7QISyXHbbcaugQpBVmCaUkSZJ6MaGUJElSLyaUkiRJ6sWEUpIkSb2YUEqSJKkXE0pJkiT1YkIpSZKkXkwoJUmS1IsJpSRJknoxoZQkSVIvJpSSJEnqxYRSkiRJvZhQSpIkqRcTSkmSJPViQilJkqReTChXQETsFRF7j007KSKOHigkSZKkwZhQrpi9gL2HDkKSJGk2MKGUJElSLyaUkxQRhwN7Ao+OiGyPt3TmPycizouIqyLimxGxxdjr14qId0bEBRFxQ0ScGRFPmtlPIUmSNHXmDx3AHHQgsBVwO+DlbdqfgJ2AhwB3AvYH1gbeDxwKdBPGo4EdgTcDv6Wqz4+JiAWZecYMxC9JkjSlTCgnKTN/GxFXAPMy85TR9IgA2ADYNTP/2qbdEXhvRKydmddFxGOBXYGdMvN77aUnRMQ9gDcCzxh/v4jYF9h3Wj+UJElSD1Z5T61TR8lkc3b7e+f293HAxcDJETF/9ABOBBZMtMLMPDQzF2TmhPMlSZKGZgnl1Lpy7PmN7e9a7e/GwB2BmyZ47S3TFZQkSdJ0MqGcWVcAFwJPGzoQSZKkqWJCuWJuZFGp42ScSHXYuTozz5nakCRJkoZhQrlizgF2i4inUT28/7ycr/sWcDzwrYh4B/BLqiPP/YG1MvMN0xGsJEnSdDKhXDEfBh4AHAZsBLx1eV6UmRkRewD/DryKGn7oCuAM4H+mJ1RJkqTpZUK5AjLzcmD35VjuJCDGpt1AjUH55mkJTpIkaYY5bJAkSZJ6MaGUJElSLyaUkiRJ6sWEUpIkSb2YUEqSJKkXE0pJkiT1YkIpSZKkXkwoJUmS1IsJpSRJknoxoZQkSVIvJpSSJEnqxYRSkiRJvZhQSpIkqZfIzKFj0HKKCL8sSZqkuXKei4ihQ5CW5PTMXLC0BSyhlCRJUi8mlJIkSerFhFKSJEm9mFBKkiSpFxNKSZIk9WJCKUmSpF5MKCVJktSLCaUkSZJ6MaGUJElSLyaUkiRJ6sWEUpIkSb2YUEqSJKkXE0pJkiT1YkIpSZKkXkwoJUmS1MusSygj4qSIOHoG32/viMiIWG8Zyx0dESfNUFiSJElzxqxLKCVJkjS3mFDOsIhYe+gYJEmSptIgCWVEbB8Rx0XEFRFxTUT8KiL2G1vmORFxXkRcFRHfjIgtxuZvHBGfjIi/RMS1rap8wdgyGRGvGJv2loi4fBnxbRkR34iI6yLi/IjYZwnL3Scivh4Rf2+PL0TEHTvzd2ox7BIRx0TE1cAH27wXR8TZ7T0uj4jvRcT2y7UBJUmSZpH5A73vscCvgOcBNwDbARt05j8EuBOwP7A28H7gUOBJnWW+AmwLvAa4HHgt8N2IeEBmnreigUVEAF8FNgZeDFwPvBW4PXBuZ7ltgZOB09rnmA8cCBwbETtmZnZW+3HgE8D7gOsj4lHAIcCbgB+1z/4wYMMVjVuSJGkoM55QRsTGwF2A3TLz523yiWOLbQDsmpl/ba+5I/DeiFg7M6+LiCcADwd2yszvtWW+A5xPJZYv7RHiE4EHAA/NzB+3dZ8O/JZOQgm8GbgYeGJm3tiWOws4h0p8v95Z9guZ+R+dbfAa4KzMPKizzDETBRMR+wL79vg8kiRJ02qIKu8rgAuAQyLimRGx6QTLnDpKJpuz2987t787ApeOkkmAzLwG+BrwiJ7x7QhcMkom27r/AJw+ttzjgIjI0gAAEOZJREFUgC8DCyNifkTMB35PJbULxpb9+tjzM4AHRMR7I+JREbHGkoLJzEMzc0Fmjq9TkiRpVpjxhDIzFwI7U6V7hwEXR8QPIuIBncWuHHvZje3vWu3v5sClE6z+Eqpquo87LmHd49M2Bv4fcNPY467AlhPEdavM/DbwQuBRwEnA5RHxoYhYt2fskiRJM26QNpSZeQ6wZ0SsDjwSeAfw9fGON0txETBRyeZmVAnoyA3AeOnfRstY98VLWPemwHWd51dQJZQfm2DZ8U4/Ob5AZn4S+GREbALsAbwX+Dvw+mXEJ0mSNKsMOmxQZt6Umd8B3kOVOt5uOV/6Y2DT1rkFgIhYB9gV+GFnuT8B9+osMw947DLWfSqwWUQ8pPO6rYAHji13IrA9cHpmnjb2OH85PweZeVlm/i/wA+Dey/s6SZKk2WKITjn3Bd4FHAn8jiox/H/AmZl5RXWyXrrMPD4i/o//3969x8pWl3cc/r5wLEcaUQlKFQpqtdVYE8GDlVgVNVZTjVQbLYoteOEkiqSaGm9NWhGtaIPWAkaPyi1WocSaam9AURS8czNUjUAJFC1FFJF4A4G3f8wc3WzPld/eZ/bs8zzJ5OxZs9aad8/8sT+stWZIzqqqNyb5fiaf9r53kr9dsOonkhxdVZdNn+sVufunyTfl35J8LcnZVfWGTI5yHptfPeX9liRfyeTI6imZHJXcJ8kzkpzW3Rds7gmqauOnxi+YbndAkqfE0UkAYA7N4pT3/2VyTeFfZvLVQLck+UwmUbk9/ijJCZl8Fc/aTOLuaYu+MujYTE5Vvy2T6zBPSvL1JEdnM7q7q+q5mXxN0SmZhOTfZBKKey1Y78qqesJ03xsyidnvZHLkcmtfW/TVJK9NcliS+yS5LpNAfe/Wf20AgJWl7v51iaxkVeXNAthO8/J3blvO0MGMXLK1b5vxv14EAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGDImlkPAADLqapmPcI26e5Zj7BV8/JasuM5QgkAwBBBCQDAEEEJAMAQQQkAwBBBCQDAEEEJAMAQQQkAwBBBCQDAEEEJAMAQQQkAwBBBCQDAEEEJAMAQQQkAwBBBCQDAEEEJAMAQQTljVXVoVX2zqm6vqmtnPQ8AwPZaM+sBdmZVtWuSM5L8e5Kjkvx4thMBAGw/QTlbD0qyR5KPdvdFsx4GAOCecMp7mVXVC6vqiqq6raqur6q3V9WaqjoyyfXT1f65qrqq3jK7SQEA7hlBuYyq6g+SnJXk0iSHJjkxyeuSnJTkX5M8f7rq65IcnORDMxgTAGCIU97L661JLujuI6b3/6OqkuQdSd6W5LLp8m9195c2tYOqWp9k/XIPCgBwTzlCuUymH7g5MMnZix46K5PX/eBt2U93b+judd29bolHBABYEoJy+eyV5F5Jbly0fOP9PXfsOAAAy0NQLp/vJfl5kgcuWr739N+bd+w4AADLQ1Auk+6+M8klSV6w6KEXJrkryRd3+FAAAMvAh3KW118nOaeqTk1yZpLHJDkuyQe7+9tV9ZAZzgYAsCQcoVxG3X1uksOSrEvyqSSvSXJCklfPci4AgKVU3T3rGdhGVeXNAlil5uHv8fSr79j5XLK1b5txhBIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYJyBqrq4Kr6ZFXdUFU/rqrLq+rwWc8FAHBPrJn1ADup/ZN8Psn7k/wsyROTnFpVd3X3x2Y6GQDAdhKUM9DdZ278uaoqyeeS7JvkqCR3C8qqWp9k/Q4dEABgO1R3z3qGnU5V3T/JsUkOTbJPkl2nD32nu/fdwnbeLIBVah7+Hk+OgbATuqS7121pBUcoZ+O0JE9IclySbyS5NckrMwlMAIC5Iih3sKpam+Q5SY7u7vcvWO4DUgDAXBIxO95umbzut21cUFX3SfLcmU0EADDAEcodrLt/WFVfTfJXVXVrkruSvDHJD5PsMdPhAADuAUcoZ+PFSa5JckaS9yb5+PRnAIC541Pec8SnvAFWr3n4e+xT3jutrX7K2xFKAACGCEoAAIYISgAAhghKAACGCEoAAIYISgAAhghKAACGCEoAAIYISgAAhghKAACGCEoAAIYISgAAhqyZ9QAAQFJVsx5hq7p71iNsk3l4LVcbRygBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgyt0FZVU+tqq6qBy9Y9sWqurOq7rdg2RVV9fbpz4+tqvOr6idV9YOq+oeq2nvBug+Z7vOwqjq1qm6tqm9X1Uumj7++qv63qm6qqndW1S4Ltn1kVZ1ZVddP9//1qnrNonUOme7/kKo6u6p+VFXXVNWrlvv1AgBYLnMblEm+nOTnSZ6UJFW1e5LHJbk9yROny/ZM8ugkF1bVA5JckGT3JC9OckySpyQ5r6p+bdG+35nkhiR/nOTCJKdX1QlJHp/kZUn+Lsnrk7xwwTb7JPlWklcl+cMkH0xybJI3bGL2Dyb5WpLnTWc6uaoef49eBQCAWevuub0l+WKSk6Y/Py3JTUnOTHL8dNlzk9yZZI8kxye5JckeC7b/vSSd5EXT+w+Z3j91wTp7ZBKuVyXZdcHyryQ5azNzVZI1Sd6c5JoFyw+Z7v+tC5bdazr38ZvZ1/okF09v7ebm5ubmNqvbvJj167QKbxf3Vppsno9QJsnnMj1CmeTJSS5K8tlFy77W3bdmcnTx3OnPSZLu/nKSa5P8/qL9nr9gnVszCb7PdvedC9a5OpOjkkmSqlpbVcdW1dVJbsskQt+e5KFVtWbR/s9dsP+Nsbrvpn7B7t7Q3eu6e93mXgQAgFma96C8MMnvTq+ZfNL0/oVJ1lXV2gXLkuRBSW7cxD5uTLLnomW3LLp/+2aWrV1w/51JXpdkQyanvA9K8rbpY2vvvulW9wUAMDfmPSg/P/33kCRPyOSI5deT/CjJ05McmF8G5Q1JHriJfeyd5OYlmOUFSU7s7nd1939298VJ7liC/QIArGhzHZTd/YMk/5XktZlcK3nZ9NqJizL50Mya/DIov5zkmVV1n43bV9VBmVw3edESjHPvTE51b9z3rkkOW4L9AgCsaHMdlFMXZnKt5BcWXOO4cdlV3b3xNPe7p/+eU1WHVtXhSf4pyRVJPr4Ec5yX5Oiq+tOqenaSTyXZbQn2CwCwoq2WoEwmp7sXL/vFkcfuvinJU5P8LMnHkpw8Xe8Z3X37EsxxzHR/Jyc5JZMjp+9Ygv0CAKxoNTlDzDyoKm8WADMzL81QVbMeYbW5ZGvfNrMajlACADBDghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIWtmPQCwZVUr/7/7uu+a9QjbqGY9wCrTsx5gG83H+37KeefPeoStesQj1s16hG3ygAfsN+sRtslNN/3PrEdYMiv/LxUAACuaoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgyJpZD8CWVdX6JOtnPQcAwOYIyhWuuzck2ZAkVdUzHgcA4Fc45Q0AwBBBCQDAEEEJAMAQQbkCVNWfVdUdVbX/rGcBANhegnJl2CXJrklq1oMAAGwvQbkCdPdp3V3dfe2sZwEA2F6CEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCGCEgCAIYISAIAhghIAgCHV3bOegW1UVd4sAFgl5qXBquqS7l63pXUcoQQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgyE4VlFX1WzN4zt+oqt139PMCAOwoqz4oq2ptVR1eVZ9OctWC5btU1Rur6uqquq2qrqyqIzax/aur6qrpOldX1WsXPb5vVf1jVX23qn5aVf9dVcctWOVZSW6oqg9U1UHL9osCAMzImlkPsFyq6oAkL09yeJLdk3wyybMXrHJikiOSvDXJpUmekeSUqvp+d//LdB9HTdd7d5Jzkjw1yQlVtVt3Hz/dzxlJ7p1kfZJbkjwsySMXPM8nkuyR5KVJ1lfVFUk+lOQj3X3zUv/eAAA7WnX3rGdYMlV130wC8uVJDkxyeZJTsyjequrhSa5M8tLuPn3B8jOSPKq7D6qqXZJcn+Tc7n7pgnXeN32Ovbv7Z1X1oyQv6u5PbcN8B2YSli9O8uuZxOaHk5zfm3kjqmp9JrGaJI/btlcCAFjp5qXBquqS7l63pXVWzSnvqnpWkhuSHJfk80kO6O4DuvvvN3Ek8OlJ7kryiapas/GW5Pwkj62qXZPsm+TBSc5etO1ZmRxxfMz0/uVJ3lFVR1bVfluasbsv7e5jpvs9Isn9Mznyec0WttnQ3eu29kYCAMzKqgnKJLcl+UmStUnum+R+VVWbWXevJLsm+WGSny+4nZbJZQAPmt6S5MZF2268v+f03z9JcnGS9yS5rqour6qnb2XWX8yYyXvwg62sDwCwYq2aayi7+zNVtU+S5yV5RZJPJ7m2qk5Lcnp3X7dg9ZuT3JHkiZkcqVzsu/llbD9w0WN7L9hHuvs7SY6cniJ/fJK3JPlkVe3X3d/fuNE0bp+WySnv5ye5PclHk7yyuy+7J78zAMBKsKquoVyoqh6a5GVJjkyyTyaBeVp3f6SqfifJN5M8s7vP28z2G6+hPKe7X7Zg+clJXpLpNZSb2O7gJF9I8rjuvrSq9k7yyukc+yf5XCbXTZ7d3T/dzt9pdb5ZALATmpcG25ZrKFdtUG40vR7yWZkctXxOd99ruvx9mZyuflcmp6zXJnl0kt/u7ldM1zkqyQeSnJDkvCRPSfKmJG/u7uOnHwI6J5NPel+ZZLckf5HkUUke1t0/raojkxyf5PQkH+ruX3x10T34XVb3mwUAO5F5aTBBuUhV7d3dN05/riR/nuSoJA9PcmuSbyT5cHefsWCbY6br7Zfk20lO7O73TB/bLclJSZ6c5DczuYbzS0ne1N1XTNfZM8mt3X3HEsy/87xZALDKzUuDCcpVRlACwOoxLw22U31tEAAAsyEoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGCIoAQAYIigBABgiKAEAGLJm1gOwXb6X5Lol3ude0/2udOZcWvMw5zzMmJhzqZlz6czDjMlOPGdVLeXuNlqO13P/ra1Q3b3Ez8k8qaqLu3vdrOfYGnMurXmYcx5mTMy51My5dOZhxsScS21WczrlDQDAEEEJAMAQQcmGWQ+wjcy5tOZhznmYMTHnUjPn0pmHGRNzLrWZzOkaSgAAhjhCCQDAEEEJAMAQQQkAwBBBCQDAEEEJAMCQ/wew8tEnR7aTfwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_attention(src, translation, attention)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's check an example from the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "src = ['ein', 'mann', 'in', 'einer', 'weste', 'sitzt', 'auf', 'einem', 'stuhl', 'und', 'hält', 'magazine', '.']\n",
      "trg = ['a', 'man', 'in', 'a', 'vest', 'is', 'sitting', 'in', 'a', 'chair', 'and', 'holding', 'magazines', '.']\n"
     ]
    }
   ],
   "source": [
    "example_idx = 9\n",
    "\n",
    "src = vars(test_data.examples[example_idx])['src']\n",
    "trg = vars(test_data.examples[example_idx])['trg']\n",
    "\n",
    "print(f'src = {src}')\n",
    "print(f'trg = {trg}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get a generally correct translation here, although the model changed _sitting **in** a chair_ to _sitting **on** a chair_ and removes the *and*.\n",
    "\n",
    "The word *magazines* is not in our vocabulary, hence it was output as an unknown token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted trg = ['a', 'man', 'in', 'a', 'vest', 'is', 'sitting', 'on', 'a', 'chair', 'holding', '<unk>', '.', '<eos>']\n"
     ]
    }
   ],
   "source": [
    "translation, attention = translate_sentence(src, SRC, TRG, model, device)\n",
    "\n",
    "print(f'predicted trg = {translation}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The attention seems to be correct. No attention is payed to *und* as it never outputs *and* and the word *magazine* is correctly attended to even though it is not in the output vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAJgCAYAAAAaidKhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdd5gkZbmw8fuBhSULSBQERJIBBUURs2LABBIkGM4Bw/qpqHjMCVE8ejAhHBQEEUQkiUgQBVSMKAqrAkoQzOQkcMjIvt8fz9tMbTOzbJju6pm6f9fV10xXV1c/XVVd9dSbKkopSJIkqTsWazsASZIkDZcJoCRJUseYAEqSJHWMCaAkSVLHmABKkiR1jAmgJElSx5gASpIkdYwJoCRJUseYAEqSJHWMCaAkSVLHmABKkqSRFRHRdgzTkQmgJEkaSRExo5RSIs3se80cZhFEKaXtGCRJkuYSEYuXUu6PiOWAg4D1gNuBXwIHlFLu6M3TZpxTlQmgJEkaSRGxLDAb+D/gZ8D6wFrADODZpZTbWwxvSrP4VJIkjZRGu7/3AbcCu5ZS3l1K2R74FbAZ8LJx5td8MgGUJEkjpYxVT24A3Az8EyAidgL2BN5bSjk+IpaNiCWK1ZkLzARQkiSNlEYHj4cB95dS7o2IXYETgI+UUj5fO4W8D9i1rTinMhNASZLUqohYvPm8lDKn/ns08PyIOLT+/2Hgf+prTwKeh7nMQrETiCRJak0d6uXftUTvCQCllPPqa+sBnwFeAZxeStkpIpYANgS+CtwJvNiewAtuRtsBSJKk7qrJ3/LAD4GNgBkRMRvYqZTyt4j4PFCAl0TEMcDywJpAAM+pQ8U4HMwCsgRQkiQNXWOcvwAOJ4d3+QqwGvBu4D5g+1LKZRGxDrAVsBtwHXAZcGBNHmeUUv7dzreYukwAJUnSUEVE1Dt8LAW8CtgGOLqU8v36+mbAkcDSwLallMsmWI4lfwvJBFCSJA1cRDyNLOU7uVHydxCwI3AXsGUp5fpGcvgE4ChgJvDKWhK4WCllTm+e1r7MNGDPGUmSNDCNQZo/Dzy5V2JXE7iDgfOAdYG3N6ZTSrkQeB3Z0eOXEbFOr3ewyd+iswRQkiQNTUQsAzwD+GW9n++jgf8lk8AvlVK+3Df/k4B3AG+wunfymABKkqSBiogZ5IDOJSKOBl4N7EwO7XJXRGwMHED27v1KfxLYWI5t/iaJCaAkSRqYxjh/D/TWjYhfAusBewGn1SRwE+CLwBrAIaWUQ1oLugNsAyhJkgaikfwtB1wcEe8BKKU8Hfg7Wer3iohYupRyKfBO4CrgYxGxfWuBd4AlgJIkadI1kr8VgHOBTYDzgWeXUu6u8/yKLAl8J2MlgY8H9gDeZ3Xv3Caz97MJ4Ijp37h2eZc0Cmx7pQXRSP6WBy4Efg/8DngzsDFwV29/qkngumRHj++XUu5oLMf9rurlA/X/lcnhc+4spXxzYZbnreBGSDPJi4iVgJcBK0bEYaWUe9qNTlo041zceFEzRdQTT+9k/WpgZeDnwF9LKbe1GpxGUiP5uxz4I5n4bUne0m2pUsrtEbFEKeW+UspWEXEOcALwAuDsxnI6n/w1jpWLR8TDgE8AqwM7AfdExE+Bqxb0eGoCOAIaG3fJvo27HfBn4Pv1rzQljXcVb/I3dTRKHY4DXgQsAdwDfDMiPldK+Web8Wlk/T+y5O8NdYDnPwMrAhsAN5ZS7uuVFJZSnhERXwV+2mbAo6j2nH4usAtZ6vcPcr3eDhxQSrlyYZZrJ5ARUDfu1mTvpz8CmwPXA3cAx5VSTP40ZTWTv4j4WEQcFRE/jIjX1vG/NKIaA/gSEU8H1idPQOuQ927dBvjviFi3nQg14r4JvKqUclV9fi9Z8LRSY561I2IngFLKG+sdQhYfcpwjKyLeEhFHAj8kq8kPAp4KnElWrf+gzhcTLWMiJoAti4i3RsSx5EZch8zmtwS+C1xEvRpamI07XUXEIyNimYhYoj53P24YtYNnI/k7EXgTWXp0H7AP8IWIeGZ70WkiNXFvltLeTrbhOqeU8q9SyvuBb5DVep80Cey28Y47pZSrSyn/V19fDPgnWbixZp22HvBtYP9Ii9X3We0bsVxEfAT4FPAIsrr31aWUT9QS+TfUWX8OC1ejYhVwS2ry8gXgFcClwMvJA+utdZa3AJRSflj/Wl0GRMSXgGeT7UhOiYiDSylXNxvHdlltTtBLuHrja7VeghwRrwOeQh7ELqg9/T5AHtx+HhG/dPuNjr79aD+y5OFxwF+AB7ZTKeWT9dr0dcA+EbFvKeUvA45tafKWYYeVUv41yM/S/Gl0+FgWeB95v1+A/YC/1XZ+c8j2arcDG0fEI4BjyTxk/VoT1kr8o6i2kTyJTJCvLaX8q1cQVIfH2ZRMCMvCnv8sOWlJKeU+4EDgxcBupZTvAb0rpW3Jg+1H6/ORKtFpS0R8gWwX+W3gr8CuwGERsW7tKd3p/bkeBHqdiL4GzAJWaaP0eJzPXAe4CbikJn+PAt4NHAN8uW6/lfqXo+Hr248OJ0salidLbl8BvC4ilurNX0r5JHAkeRH7vsg7PgzS24EPAntHxIoD/iw9hLq/9Dp8nEN2XgyyZPgEYIfm/gJcRzYlOBZ4GLBFoy2gBR1ARKwFUEq5uJRySSP5653jnk2OlXhFnW+hLp4tAWxBRKwJ3FRKubwxLcgfDeTGvRn4E1gcDhAR6wAPB95ZSvl2nbYn8Ebg8Ih4Qynl710uCWw01F+NbGPzIeD8YR9UozHaf8PKwMNLKbdGxNrAbLLZw5tLKXdGxBuAeyPiGPf3djX2o1XIxO9V5Bhuhdxm+wF3RcTJvbHcSimfjoh7gZPH2faT7fNkFeJLyV6Re5dSbhnwZ2oC9eJtJnASeZG3W+3wcSKwA1nTFRFxainlTrJp05717xObHUHa+g6jJCI+C6wXEV8spZzTm16P4/dHjpH4FmDPRtvKhdLpEpM2RMQBwCfJG2E/oKT7I2JT4G3AwfasSxHxOeACsnPMA0lzKeUg4CvAqmQSuE7XSwIj4n+B3wIbAZfWfWooJYAR8dKIeHMZu9XTyRHxzvryD8mD16fIhstnksnfHTW53xF4FGMXQWpRRBxE7kfrA/8opdxVSrm7lPIsssnKgcAr64kfgFLK5wfd3CDGOhS9GziDrEH5hCWBrdsaWIZMSq6PiG+TTT42J9v9fQbYrtZmnQwcDDzZ5G9uEfEt8lh4NnD1OK8vCexG/ga/u6if19kTZRsi4gSymuQPwGXjvL44WcV5KbVnjwD4Ktn26PHAZs0qplLKweTBZEXg2xHxyK6WAFZnArcBG5Kj6w+l/Wjdd9cADo6ID0XEd4AtgF/UWc4FLgE+QJZs715KuSUiVgf2JgeG/eYwTgS9zkOapzPITh8bk9uV3u+ulPJs8hj1eWDXZhI4aL0eovU3/i5yfzcJHLJxmiWdQTbn+FNE7EMmf7uVUi4gB3dekWwb+LpSytmllLeVvnsDd11E7A1sRg71ckQp5a8RsUTf72sOWbtzXinl2kX+TKvchyMiPkSOibQTcGEp5e6ImFlKuSciliyl3Fsb0H4NuL6U8vZWAx4xEbEBcBrZe3R34HfNxCYi3kVWN7ymlPKPVoIcsvGqu2tbm2eQbbL+AuxUSrlhSPEsRe7jXwD+BTy/lHJB7VBQImIN4GiyR9sfySvcx5IHvRfUk8WgYlucLI34Rynl+jrtFWTHq5sH9blTwUPsR0eRF6s7l1JubJ6wI+JCYGmyJGegg0HHBHeDqNv1i+TYhGcCVgcPSUQsA6xT8v69zelnkvvMf9Uk79HA94BHAj8rpWwz/GhHX+RQL3NKKa+vzx9DNuN5BNlk5hO1Y8imZFvqf/eOrQv7mZYADkE9SG1I9sj8TU3+NgGOjogzgCMiYvWSt7/5EFlK0umhXyJis4h4fD3IUEq5AtgWWBY4DNi8uX5KKfsDr+hQ8tcrBSEiNq7ra4VaTfcj4PVkcvWN2pZrkLH0hm64G+g19l4J2L5OLzVxuJasvjgMmFnjuxB4xiCTv+qJwIeBD0fE0hHxfXLA9U63g56P/Wh3srfh0RGxSq/UBqCU8gQycR908jezjPVI3iFyXLRNI2LFOn0v4CyGXBIYETN6x6dRN1nnkl7JX13efwMXRMQTGq8vS/YYX71RsvcI4DfkMeGlkxHHdBIRi9eq3ZXJO3+9LCI+SCZ9GwE3kh2fPgBQSrmot24XuXanlOJjCA+yGvMy4Hnk+Gd3Az8BvkNWCR8CLNl2nKPwIEsdbiRLkS4CNm68tiF5V5TZZMlRtB1vC+tnsb519TfgVuAG4L3Ao+trLyR73J1BdsAYRCyLN/6fSV7lP64erO4H9hkv7pbW2xfqvnM52S7psW1vy6m6HzW3+4BiW5rsVLBqY9rxZPOGO8iagM8Cm/TiAf63HmP3B1YacHwzyWYN7wKWb3tbzmvbTuIyF69/lwX+i2x/Paf+njarr0X97V9Hlsz+R11PZ/eO1YPed6bqo57PrqzHpouA9zXW6THA6ZO9XVv/0l15AJsAP64Hr98A721s3JOB77Qd4yg8yCudK8hxxfasB4/rgGc15tmQbIP0F+AJbcfc4rr6Sj1p7wI8jWwYf13dn9as+9YLyeECfgWsPMmf30z+9gM+B6xZny9PDtVxP/DxxnwzgZc1ng88gQdmNP6/uJ60jgVWq9NaSUzrOnobA0rOp8p+NEFMTyVHQriILBl5Dnmh/ELyNpkfIS8SDwUe19sfa9JxA/A/g963gFPIobtmMQJJIFmjtyPZ7KM37YvADpP4GcuSbXjPIkvQv0Teluxq4El1ng3I9qE31td+BCxRX+vcBfs81uVOZCK9Z2MfXr2uvw0b8z2c7BNw4GSvv9ZXwnR91IPpu4G3Als2pj8BeGTj+cPIUsBDyOqozv5AyDHGPkcO9dKb9mjyXsg3AM9uTN+E7KX4qLbjbmldrUlWn76ZWnJMVrHMqQf9pRrzvqwetNeZxM+Pxv/fqsvfD1i7MX0lMgn8N1ld9GSyw87tZBXRMNZTM/nblGyDeDRZEngAsEZ9bahJYD2R/rPu28u0EcMo7EfziGtx4Plkx6Hfk21LP9O33/US1f4kcD9q6eWAYmvG8HXynsizgBWHvf364lqZTMpuAf6THC/1yt66mYzvTDajuALYqPHalmRnr2sYSwKXA9Ymm14sVqfNWNQ4psuDHB/xn8C19XEPmQgu0zffxmS/gOto1IRNWhxtr4jp+KgnxOvJq+p764b+4jjzbV4PXjdSqzK6+qgnoOvIK8Zt6rTeQWddxpLAZzbes0Tbcbe4vp5YT9LPqM8fS47BdUIjoXgqWeIW/QeWSYzj0+Sg3E9rfO6SjdeXIKsT5wB/J0uRNh/SOmqWUB5DNsPonYx61cEHMpYERk0glhtCbB8nS7R6Sdeqg/7MUd6P+mLqbaPFgReQQ0DNAQ4cZ//qJYEHk2PKDWOdNS8qliJvxfV38jaHK7SwDZvxPI68R/Pd9Rw0qRfIZBL+d2DpvulbkcnmFYxTK0PLzT9G6UE2AbuKHDrn4eToFgeQF8p71XmCvJD5HVnFPpB9u/WVMd0e5N07/kEO5rw0eQeE/euP8cuN+d5CVm9eNqwD10PE3WrJI7AKeV/R+6m3B+p7fV2yF/Ac4Oltr6+2txM5rMJfyCrzTcjqsuN7yQtZFXQCsO4AY1mWrArauzFtPfJm5d+osa1Ypz+JvHPLIwcVT19szeTvpLrf3M/cpfG9JHB/xpLA/chS+0G3cdurbrPHkhc3sxliG+BR2o/64lqs7/lMsnPHr+tx9ZF1ejMJfFc9eX4RWHKQxzLmLv07prHtbiHbLA+1OriunxPJHqK9aYeQBQ83k2NtjrtuF3J/eQdZsPGg9tdkYcacvu1ke7+519GS5Ph9R/WvX7LZwn3knVEga0z2YYC1XK2vkOn2qCebk2mUTgGrkVVgV5JVw0H2htoTWK+lOEfuh1lPRl+vJ6SP9cdIDhT8LRrVD9P9MdF2Ii8uvkNeHd4MfKtOX4y8qjyinpwG1l6rftYpZML+9HpyuAM4jyy1+QfZlnOoFxfMnfydQpZQ7lbX0wv65v0cWWpxbl2fc4BNBxxfkKVqPyMvDK8a9D49yvvRBHEdwFhNwJJk27+LySrhXkeUZhL41mEeF8gLhetrXOuRCdGJwJ1kbcZQSgLJQfB/XH9vH6jTnkWOmPB1snT0Lc19r39fXMD9ZY36vU8AVmDuhPhTZJvAn5MdHO3U+OD1F+RwRSePM30tslbgYMbaTA605LT1FTJdHvWAuSTZSPrYOm0GY9UZa5FF5wc039NWrI3/ZwH7krd7GkrpTOOzNySrDpcHZtZpDyevrC9h/CSwM+1I5rGd1qvTHkG2ybqbrH7qJRZfI5sVLHLbn8bnT3RC+E/y/p+31RP0RxkrLfgDcFiL6+8nZGnFk+tv81LgjfW1mY353ltP3t+dzHU2H/FdWBOGPzDWdmoQvTdHZj+az3g3JC8irgOeW6ctSVYHX8oESeAQ45tZ95VvjvPaMfW3MPA2gY1zy5pke79LqFWIdfoTyd7d/UngTBol4eMsd0b9uxRZtbsLmfj1SoVfTF7ofav+tpYjq57PIS8C92IIFzVT7dE4Ln6prp/HjzPPr6gXYUOJqe2VMh0e9QfQ27jvIRt09trUzGi8djo5IOZIlL6RV3G9EogbySv9YbXPOrSedG4nS0b/i7Fqg4cD36wHtI+Oyvoase30lPraI+qB99J64vkDOcjypDUrYO4Std3I6sLXA4+o0x5NljpsWp8HWaX/QxoJ4RDWU7PUfSOy80CzyvdiYL/G82ZiFDQ6PAwovhmNz3oYcGpdP+eSVZxP7r0+HfejBYz1mfVYeT3wvDqtmQReSEvtJmssZwI/HWfbrl2Pa1cA72QA1cHMXfLZSwIfQSaBf6SWBNbpmzFWErgnWctyMJloPGioHMaGelmeLMm7giwZvpoc3qX3m39RXeY/6/70D/IGB71jxLWYAPbW6Zpk794V6vNl63r9ObB+Y741gJ+SwxstNozjZusrZ6o/yEaxn6GOKUb22jmDLJJvnnxWrQf6B3UGGWKszRP51jWe55LVQG/sPxENMI6j6g/gVfX5eWQSuB9zJ4G9q9f3t72dR3Q7Pa3OsyLZxm538hZMa0xiLM0qnm/XA/slwF31JPLOcd7zWHJoketoDGcwoHW1GDng9HMb0/6HTEiXaX4HsqrssMZ8y9d1NvBerX0xv4m5x7fbtW7T3zCJSeAo7Ufz2n6N/5fse+2Z9VjanwRuXfetXzPoKrIHt0ns7UvvI5uqbNv3+oy6n91O9oqd1FLA+v3/Cry2fzuTSeBJjJ8EHk42b7iMTP6fNI/PWIZs1/gD6gUAmej1jtG9oZ5WJ6veP0WW/PXiOJUJEsyuPchB788nk+gfMFYD8dS6//yJ7BC2F1mi+i8G0Nt3wvjaXkFT+UFeUV9B3st0zcb07cnqp1vIO3t8hGwndQtD7u3L2BVi80T+cXK4gCOYuwfZLgw4CQReS57onlWfv4ts+HoGWR3WTAJXIUsKBzakw6g8FmE7PXVI8e1dTzxbUdv+1G12G/DyxnzvrdP/xhBKj+pJaH+yh/jW9QT49+bvjLGT9uHUAWnJEriv1pPi0BJAssnDnLpNmwOc78YkJIGjvh/NI+6DyWSifxiMZ5Idja5nrFZlSXJcwIEeF5g7gV6v7vdL1efrkInWOcALG/OtSZ4XNqaOMzmJ8SxBFiTswQRtMskSyPGSwEcBryQT1/Xn8RlBDt30fWCtOq03ZMkxZOeS/Xq/mb797Llkzc2/6PD4rI31cVRdb28mS4M/V3/7H66vr0q2Ub6YLEH9ybDXW+sraao+yKuev5NjIC1bpzXbFW1E9kq7juxp+OOhb9y8kvsDc58MlyOvAueQvWr7r3B7J4VzmEc7kUWI6bXA5+v/s8i2JL2SwBPIkqVP9Q5S/fFNx8eIbqf+zzuJTCB6+/qqZPJ3OI0hIchqpn0Z/Mm5mchsSXZkuIMsoVxvgvfsVw+2K5EJx23MoyRkgLG/pm7XrzF+ErhQ23QU96MFiP0c8gL5P3hwErgd2cv3Svo68QwwnmZic3Ddby4jq+c2qtMfS14UXUT2vN2T7AB4HY3xMCcpnplkR53tGtPeSV78HEeWeK9cp4+bBC7AZ70JeHfju/+d2l6NTFhuJYd/WrfxntXJHqu/YcCdqKbCg2wbeTHZGafXoeOp9Xf4VebOFVYjLxyGPph46ytqKj7IK8HTmHv4i0fVg8Ax5GCZvRPlamSd/8DHFhsnzseQQ12s0Dd9NfIK73ayN3J/R4tXke1sftjcUScxrjXJ6rcLgE/2dnzyNnl3ksXl+9CRgbEnazsNYl0BG9S/f2JsHLaNGBsupFfN+hrgMfX/gTbMJxOd05m75PHoenC9vjedLM0IxkrF3kCW2J9U97OBJn88ONl6YIgSJk4Cd1nYbTrK+1FvfZAdCLZpTPt47/uTHStuIaug+5PAn5LJ/Z/q9h/kUC/Nkr/PkqU4HyJrcS4j72zRS4jWJ2spLiET1N8ygJLveix8B/Cw+vxYMvk8vq6bf5A1Tb3hjHpJ4O9p3I5xvOVOMH1m/W5/IUsce7/z3clSwDuBt/a9ZwWG3Ft8VB9kaejtwFb1+QbkMfObjXU5lPb284yz7QCm0oNGEsdYh44nkR0Y7iSrUs6vB6l31QNeqx0YmLvn0TMa01cFflkPIs8d56SwPZM0RA15pfph5h6TauP6A3lFY9ou9aD1P7TYgLiNbTYK22mcmPYFTqj//2/dt19BtiE6nrFGzU8mq+m2H9K62oQsXV+2MW07snT5VPJEvG1vvTbW7fMYGw9waJ0byJLHXnK8BA9OAg/vvV6n77iw23QU96PGZ6wEHFn3o+3I0qSbaPQyrsfUW5k76diobtddaTS1GcJ2W7dum10a095Y4/85Y0ngTLJd5doMYfgXsirxz9Rqe3JM2Tl1v9+HsVscrkUm9b9inNsNMnZhtFz9fR9GNuHodWjp/V4e33jPO8jbvA18rMyp9mDu/OAF5Plt3frb6x9fcwey5HYod0SaMOa2V9pUepDjdvWqL19Zf1i9YRw+UqcvQV4FHtpyrM0qjHXIK8Fbqb3+6vRV63f423gnhUmK4xjyqv5P5In3JLI9zfJ12omNGA8Hvtryemte/T+LLDFZjrEGzpN7L8YR2U7jfTeyiv7fdT1sRZYy3AN8rzHPSvXE8VsmudprXjE2Tl6fAN7UeP3ZZOn8A0lgnb5kXXdvY7jjxW1GJl2XM1aa2kwC92LstmsLXXU2SvvRQ8T5+BrDzWRVaa/neLNn63fJi4zPk1XCR5CdEgZautS3Dj9FDotzMXVg3sZre9R4fk4jcR/S+nsMWfiwQ33+AbIN9Q5k6dJ9ZBLYKwl8xLx+l2TiegFZsvnr+p1PIY/Rq9Z95TjyeP1k8pZvzXt7mwSOrYsH8oP6/ByyxuFf5IVPr6Zr9bqtjqXle0i3vtKmyoPsFfdH4PX1+Yx6oH06Y1f3QZ4QzyAHfn6g9GHIsT7oR0kWQZ9J3rz8qY3pvZPC5eSgppPW5o68+jmbbKO1NnlVdCNZhfJYsq3J9WRbrEvrSWEkGg/Xg95N5Mn5Choj/E/WNh2V7TRRLHX60fUzlyOrf24iS2m2q897nZuGst2YO0FfvZ68fg+8oTH9OYyVBG5HVhkeRJ44H1QSMsnxPWi7kNWe59X9aMM6rTfu5Xpk1eacGuMC395wlPaj+dxux5GJxh+Zu3St2S7qYLKU62ayenWzYcTZ+Px1yQRvTv3tL9H3+h5kwnQhA+zpPsH+9I6677+0Hk9756TF6jq9lCwlHLcTCnP3vN6MbLf4SLIX+HPIDlU/Iu+d/da6f95HJuyz6dBYrAuwnXr5wR6MXdy9giwtvo16Nw+yNPsIsod467d/bX3FTZUH2b7i6nn92Mmk5lAyqRno8BfziKF5oN0DeD/Z8+uR5L2HTx3npLBKPWhcwCTd65MscXlKPQmt1Jj+1HrQOg3YgixZ+h9y0Oc2q32bY8i9ibwi3pG86j2dbAj9fsba4CxSEjgq22mcuI6u26N3UfMEsnTvk/X5f5AdLnpjxZ3OOAOaDmHffinZtu1R5PAKv6UOsVBff049sc0h76d5GwNuc9MX32bANowND/WsGuMVzf287l8Hkh1AHjtd9qN5xPh8ctisl5IJ1LnAro3XmyWBG9T9b2jVZGRp9rHkxfvaZMJzOdl7u7+E/P+RnfvWHVAszXWxMnUMvsa0T5NJ6qr1+Yy6Pq+rv81Vxllmr3p3CbLN3mvJWplmp6reMfpUsonTpuTYn3swVhNiEjj3en1QfkCeA3clL1Bvrr//2WSNwFAvaCaMu+0ApsKDbHN0DWM9ox40SCPZ5u/sunFH4d6+3yar7C4mi/GvqQesl5BVLLcxd/XQw5m8Nn8Hke2NvlsPUL12D72Dx5b1AHM6Yz3q2igpXRZ4Tt+0V5MlW+/rm35sXZ8fYJKSwLa30zixbEomTHPIDk3vr9M/WPfrx9XnS5Hti5ZmCAlEc9+p/3+jHkw/Vp8/kSyx6E8CH0+22foUtfp1gPE1S1WOInuG3kmW/B1apz+zngCuJKtgtyIvGH/KIlaljdJ+NI/t9k2y9LHXXOZJZBL4a2C3xnzLkon9UEcAABYne/LeA3ypTlu7bstLGD8JfNgkx7A0ffc6Z6yTyTV1X16pTj8a+HNjvjXJdmabME7pH2MlU8uTieuldd2fwVhi2GtasSVZEvh9+salW9R9dbo9GD8/eKDJEFkDsSd5cTa0+6HPV+xtBzDKj8ZGfG09sPZ69PR+JA9j7KS4OzlWWutj1pEn7H/WA9bqddpp5FXIS8iSgbPI6rynT/JnH0SODP8NMiGeA+zbeL237p5SXzuFhaj2moQ4g2yf+PXGdn4uYwnQf9VpzWqpXhL4XiZhgNc2t1Nz/+6b9h6yumf/um1+Qt7p48807l9JS8Pz9E565PAKqzemb0omgRfQaBPYQnxHkAnYNmSSdWrdn06qr29ONsyfQ5bUXMsiXjC2vR8txHZbozG9mQS+jiyVOpQslVp2wDGNV8HmJ/4AACAASURBVL26BFnSdQ9wcJ3WTAK3HNS+X49JvQHUX1SnHUBefH2KPLbeRd628BFkk5pb63HpNfV4djXjdJRpHOMWI0vMf0b2cD6p7ov79K8Xxo7RB7axz4z6g4fOD1ZiyLdRXODv0HYAo/6oP5jLaNz3kbyCeil5dTSHLB4PWkhkJoj5qHoiWrI+f2Q9qBzD2JhET6wH3X8wSbfAqgfz/YCX1udrkW0h51Cv+nvrtDF/m9W+azE2XM8m9e8s8qR8ZmO+ZlXMN8jeXXux6NXArWynceJ4cWObLEOW1BxMNjg/mSy16iUtew0ihvmM84VkcvX8xrSV6zpaj7yV0o/IdjdvayG+7clSyK3r83eSQ2Z8lWwWcmJj3peTSeIilwaMyn60kNttnRrvz8kSp0vIRHWLIca3ft/zJclbHfYngb+rx4aBxUaOjnAWmXC+jGyO0Rz778Vkae6J9fj5djL5v76uuwkvJsjSxZeQ1dyb12lrkr39JzpGPwZL/Oa1vR4qP/g38MHGayM1rFnrAYzqg7Hi8teT7Sp6P5gPkY3g7yevvN7MiCR+Nb7FyKu779Tn65O9kJpjtr2RLNV5DJN0FwSyVOwG8gq0OWzA6mRPzXEPMKPwaMTeu93Um8lG6l9vzNNMAr/KIrbxbGs7jRPHy+q2OZGx8fN2Jdtublaf70nepmgO2QtwoCUz84h1R3JcsoeR1YTPI3uSX8XYhdiaZAL4Uya5eu4hYgvyTiTvqc/fQJbW7FxPCAfXGE+YzJPAqOxHC7ndrq7rZE+yXeKeZA/WobWfJpOfX9M3CDZZEjiLPM5/oa7ndcmenYMe5PzR5IXMH2mUEDNW4rQ1eRF6LNlO8uF1207Ywanun4eQpcJ/bf42yLa04x2jm72iTQL71mf9Oz/5wciuu9YDGPUHOZ7WpWRR/HlkonAIjavZ5g4xCo8a67nkyb03/lCv3domZPu83Sb5Mzchk4Z/02jUXV/rHWDuBT7d9vqZIPazyR59zSTwHuZOAid1UOw2ttMEcWxVT4K/I8cDW6oexE5rzPPouk6GOuxFX5wb121yao3vDrLkazuyEfZ9ZCng4xhQw/yHiG85spftcnW7fpKxJOzRjCWq35mO+9EibrdWagLIjmgXke0k+5PAFRmrwj+sThtK5wdgQ7IkcA5zt4/slcxtXbf1j6m3bJuPZfaO0XOAHfteW40clPvfwH5t7StT7cEUzA/miqvtAEb5UU8kvTZhpwBfJku0etUq0fw7Kg+y8fvtvZMNYw18VyFLry5kAA1RyYbb55BXmNv0vbYaOTTBvxind1rbj3qCPpvsPdefBH5tOm2nCWJZixzQ/O9kQ/1eY/g92942fXE+t26nA4HXNKbvQPawnfA+p0OM8RFkR4/3NKbtWH8bb5zsGEdpP1rI7XY5Q+iQMo/Ynkgmgd/jwUnggeTF0bht6wYc1/rjHU8ZSwK3Iav053vb1mP0L+o6H+8YvX/9zJE6p43igymaHzQfvQA1johYhjxg30KWhvyrTo8y4isuIl5Att/6NTnuFmT7j+eRPV8vHNDnPppsxL0q2ZP2jMZrq5A/hhsG8dmLKiI2IGNfDXh7KeXHEfEmsmH2IaWUtw7gM1vZThPEMoPcbl8hSwtWIU98u5VSLhpWHA+lxnl/7zcYEauRbU03BV5WSrmp5fh6Y+39nuwYdhfZA3BZ4P+VUu4YwGeOzH40kVHebhHxRLLt6z+AT5RSzo2I1cnq3+8Ap5dS7mohrnGPpxGxWCllTkQsU0q5cyGXuRrw3r5j9ErALaWUMhXOc22ayvlBjwngQ4iIxUsp9zeeT52NG/EUcqyotck2CZcBHy2l/HHAn7sBmUSsRnaNP2uQnzeZ+pLAt5VSfhoRewDnllIuGdBntrKdHiKm15O9Mp9Atum8pq1Y5iUiXk12MnglI5LoAETEs8jqtjvIBHBpsnPIwOIbxf1oIqO43SLiCeS9mR9OVpuvQpYOblVK+WuLcU368bSxzFXJJPDMvtenzHluYUXEM0spv1jEZUzZ/ABMAKe9iFia7NV2P3BfKeWeIX3uBmT7iMcD/1FK+dEwPncy1Ni/TBbx77KoB4n5/MxWttM4cSxWSplT/18HuKeUcl0bsTyUiHga2fbmXuAtpZQ/tBzSXCJiU7Kd213AKaWUK4bwmSOxH83LKG+3iFifHNP1WWQ1/gdHofR7EMfTWhJ4MJnkbltK+fWiLnOqiIityeFw3ltK+Xzb8bTFBFADExEbk6P+v6uU8pe241kQNfbPksOeTKnYF9VUuYqNiMXIWyvdNKrNCvRgo77dIiLIEtsYRHX9whrE8bQu8211mfc/1PzTRUSsSLZ5PqaUcmnb8bTFBFADFRFLllLubTuOhTGVY5c0/QzymNRfnTndNWs7usoEUJIkqWMWazsASZIkDZcJoCRJUseYAEqSJHWMCeAQRMSstmPoZ0zzbxTjMqb5Y0zzbxTjMqb5Y0zzbxTjaismE8DhGLkdDmNaEKMYlzHNH2Oaf6MYlzHNH2Oaf6MYlwmgJEmSBs9hYCYQEa6Y+fDkJz950pZ1ww03sOqqq07KsmbPnj0py5EkaYo4s5SyzfzObAI4ARPA+TOq+08O5i9JUmcsUAJoFbAkSdLUt8qCzGwCKEmS1DEmgJIkSR1jAihJktQxJoCSJEkdYwIoSZLUMSaAkiRJHWMCKEmS1DEmgJIkSR1jAihJktQxJoCSJEkdYwIoSZLUMSaAkiRJHWMCKEmS1DEmgJIkSR0z7RPAiNgqIk6NiGsi4o6I+H1EvKbtuCRJktoyo+0AhmBd4BzgEOBu4BnAERExp5RybKuRSZIktWDaJ4CllON6/0dEAD8D1gbeBMyVAEbELGDWUAOUJEkasiiltB3DQEXESsDHge2AtYDF60tXlVLWnsf7pveKmSSjuv9kri9JUmfMLqVsMb8zT/sSQOBI4GnAvsDFwG3AW8iEUJIkqXOmdQIYEUsBLwfeVko5pDF92nd+kSRJmsh0T4Rmkt/xnt6EiFge2La1iCRJklo2rUsASym3RsR5wN4RcRswB/gAcCuwQqvBSZIktWS6lwACvBr4C3AUcADw7fq/JElSJ037XsALy17A82dU9x97AUuSOmaBegF3oQRQkiRJDSaAkiRJHWMCKEmS1DEmgJIkSR1jAihJktQxJoCSJEkdYwIoSZLUMSaAkiRJHWMCKEmS1DEmgJIkSR1jAihJktQxJoCSJEkdYwIoSZLUMSaAkiRJHWMCKEmS1DEmgJIkSR1jAihJktQxJoCSJEkdYwIoSZLUMSaAkiRJHWMCKEmS1DEmgJIkSR1jAihJktQxJoCSJEkdYwIoSZLUMSaAkiRJHWMCKEmS1DGtJoARcWREnB8RL4uIiyPizog4PSJWjogNIuLHEXFHnecJjfe9OyLOi4hbI+K6iDgtIjboW/ZPIuLEiHh1RFwREbdFxPcjYu3hf1NJkqTRMQolgOsAnwA+AswCng4cChxXHzsBM4DjIiLqe9YGDgK2A94ELA78MiIe1rfsLYE9gXfXZT+pLluSJKmzZrQdALAysFUp5c8AtaTvvcB/llKOqtMCOB3YBLiklPKu3psjYnHgB8D1ZEJ4VGPZKwAvK6X8q867BrB/RCxdSrmrP5CImEUmipIkSdPWKJQA/q2X/FVX1L9njzNtLYCIeFpE/CAibgL+DdwJLAds1Lfs83rJX3Vxczn9SimHllK2KKVssRDfQ5IkaUoYhQTwlr7n944zvTdtqYhYBzgLCODNwDOAp5AlgEvN57L755MkSeqMUagCXlDbAMsA25VS7gCIiBlkVbIkSZIewiiUAC6opYE5ZNVvz85MzWRWkiRp6KZi0nQ22ev3iIg4HHgc8B4eXN0rSZKkcUy5EsBSykXA7uQQL98FXg28Cri1xbAkSZKmjCiltB3DSIoIV8x8GNX9Z2zISEmSOmH2goxiMuVKACVJkrRoTAAlSZI6xgRQkiSpY0wAJUmSOsYEUJIkqWNMACVJkjrGBFCSJKljTAAlSZI6xgRQkiSpY0wAJUmSOmZG2wFoahvVW66N4i3qRnVdSZK6xxJASZKkjjEBlCRJ6hgTQEmSpI4xAZQkSeoYE0BJkqSOMQGUJEnqGBNASZKkjjEBlCRJ6hgTQEmSpI4xAZQkSeoYE0BJkqSOMQGUJEnqGBNASZKkjplWCWBEHBkR57cdhyRJ0iib0XYAk2xfYOm2g5AkSRpl0yoBLKX8ue0YJEmSRt20rQKOiN0jokTEphHxg4i4IyIujYgd2o5TkiSpTdMqAZzAMcCpwPbA5cBxEbF2uyFJkiS1Z1pVAU9g/1LK1wAiYjZwHfBy4JBWo5IkSWpJFxLAs3r/lFJuiojrgXFLACNiFjBrWIFJkiS1oQsJ4C19z+8FlhpvxlLKocChABFRBhyXJElSK7rQBlCSJEkNJoCSJEkdYwIoSZLUMSaAkiRJHROl2NdhPHYCmdpGcb+OiLZDkCRNX7NLKVvM78yWAEqSJHWMCaAkSVLHmABKkiR1jAmgJElSx5gASpIkdYwJoCRJUseYAEqSJHWMCaAkSVLHmABKkiR1jAmgJElSx5gASpIkdcyMtgOQBmEU77vr/YklSaPCEkBJkqSOMQGUJEnqGBNASZKkjjEBlCRJ6hgTQEmSpI4xAZQkSeoYE0BJkqSOMQGUJEnqGBNASZKkjjEBlCRJ6hgTQEmSpI4xAZQkSeoYE0BJkqSOMQGUJEnqGBNASZKkjpn2CWBEbBURp0bENRFxR0T8PiJe03ZckiRJbZnRdgBDsC5wDnAIcDfwDOCIiJhTSjm21cgkSZJaEKWUtmMYmogIYHHgS8CGpZTnz2Pe7qwYDcUo/tbyJyFJmgZml1K2mN+Zp30JYESsBHwc2A5Yi0wAAa4aZ95ZwKzhRSdJkjR8074EMCJOAZ4G7AtcDNwGvAXYrpSyyjzeN71XjIZuFH9rlgBK0rRhCWBPRCwFvBx4WynlkMb0ad/5RZIkaSLTPRGaSX7He3oTImJ5YNvWIpIkSWrZtC4BLKXcGhHnAXtHxG3AHOADwK3ACq0GJ0mS1JLpXgII8GrgL8BRwAHAt+v/kiRJnTTtO4EsLDuBaLKN4m/NTiCSNG0sUCeQLpQASpIkqcEEUJIkqWNMACVJkjrGBFCSJKljTAAlSZI6xgRQkiSpY0wAJUmSOsYEUJIkqWNMACVJkjrGBFCSJKljTAAlSZI6ZkbbAUhdMYr33fX+xJLUTZYASpIkdYwJoCRJUseYAEqSJHWMCaAkSVLHmABKkiR1jAmgJElSx5gASpIkdYwJoCRJUseYAEqSJHWMCaAkSVLHmABKkiR1jAmgJElSx3QmAYyInSNi97bjkCRJaltnEkBgZ2D3toOQJElqW5cSQEmSJDECCWBE7B4R90bEin3THxcRJSJeUJ9vFxHnR8TdEXFtRHwmIpZozL92RJwQEddHxF0R8eeI2Le+diSwI/CcuswSEfsM71tKkiSNjhltBwCcDHwF2B44ojF9F+A64McRsTNwbJ3vQ8CjgU+TCex76vxHAUsDs4BbgPWBTepr+wLrACsCb63TrhzM15EkSRptUUppOwYi4hRgZillm8a0y4AfAG8H/gacXUrZo/H664EvAWuXUm6KiNuB3Uopp03wGScCq5RSnjufMbW/YqQBG4Xff7+IaDsESZqKZpdStpjfmVuvAq6OB7aOiIcDRMRmwEZ1+kZk6d0JETGj9wDOBpYCHl+X8Xvg07VKeZ2FCSIiZtVq5vMX8ftIkiSNrFFJAE8F7iPb6UFW/14J/AJYpU77Xp2n9/hrnf7IxnvOB/YH/h4Rv4+IrRckiFLKoaWULRYkg5YkSZpqRqENIKWU2yPidDKJO5QcsuVbpZQSETfX2WYBvxvn7X+ty7gK2D0iFgOeCuwDnBoR65RSbhr0d5AkSZoqRiIBrI4Djo+IV5AdOI6r0y8DrgLWK6Uc9lALKaXMAc6NiI8DvwTWBW4C7iWrjCVJkjptlBLA7wF3kj19/1pK+Q1kQhcR7wa+ERErAN8nk7n1gVcCOwFLAGeSPYH/BMwE3g1cC1xSl38psF1EvJKsXr66lHL1kL6bJEnSyBiVNoCUUu4i2wKuSXb+aL52PLAdsBnwLeAkcjiX35LJ4N3ARcA76zK+TiaTL6rLBfgycBbwNeA8skpZkiSpc0ZiGJhR5DAw6oJR/P07DIwkLZQpOQyMJEmShsQEUJIkqWNMACVJkjrGBFCSJKljTAAlSZI6xgRQkiSpY0wAJUmSOsYEUJIkqWNMACVJkjrGBFCSJKljTAAlSZI6ZkbbAUhqzyjed/f+OXPaDuFBFl/Ma2VJ04tHNUmSpI4xAZQkSeoYE0BJkqSOMQGUJEnqGBNASZKkjjEBlCRJ6hgTQEmSpI4xAZQkSeoYE0BJkqSOMQGUJEnqGBNASZKkjjEBlCRJ6hgTQEmSpI6ZlglgRBwZEee3HYckSdIomtF2AAOyL7B020FIkiSNommZAJZS/tx2DJIkSaNq2lcBR8SKEfHViLg6Iu6OiH9ExGFtxyhJktSWaVkC2OcLwNOBdwHXAo8Ent1qRJIkSS3qQgL4VOBLpZTjG9OOHm/GiJgFzBpKVJIkSS3pQgL4e+C9EXE/8MNSyp8mmrGUcihwKEBElCHFJ0mSNFTTsg1gnz2Bk4G9gcsi4vKI2LXlmCRJkloz7RPAUsotpZR3lFLWAJ4I/Br4ZkQ8tuXQJEmSWjHtE8CmUsqFwHvJ771Jy+FIkiS1Ytq3AYyIXwDfAf4AFOBNwB3Ab9qMS5IkqS3TPgEEfgXsDqwH3A/8DnhJKeXKFmOSJElqTZRiZ9fx2AtYasf9c+a0HcKDLL5Yp1rLSJqaZpdStpjfmT2qSZIkdYwJoCRJUseYAEqSJHWMCaAkSVLHmABKkiR1jAmgJElSx5gASpIkdYwJoCRJUseYAEqSJHWMCaAkSVLHdOFewJKmkMUi2g5BkqY9SwAlSZI6xgRQkiSpY0wAJUmSOsYEUJIkqWNMACVJkjrGBFCSJKljTAAlSZI6xgRQkiSpY0wAJUmSOsYEUJIkqWNMACVJkjrGBFCSJKljTAAlSZI6ZigJYET8JCJObDx/UUTsNc58O0fE7g/1fkmSJC28GUP6nLcC9zWevwjYCfhi33w7A6sARz7E+yVJkrSQhpIAllIubvP9kiRJGjNpVcAR8biIOCMibo6IOyLikoh4W33tgSrciNgHeDewbkSU+jgyIo4EdgSe05i+T//7e8uIiBsjYvOIODci7oyI30XEs/pimhkRB0fELRFxU0R8NiL2iogyWd9bkiRpqpnMEsDTgEuA1wL3ABsDK4wz31eBDYHnA9vXaTfUv+sAK5JVvgBXzuPzlgG+DuwPXAt8DDgpItYtpdxZ5/kMsDvwoRrbHsCuC/i9JEmSppVJSQAjYhXgUcB2pZSL6uQfjTdvKeXKiLgGuKeUcm7fcm4GFuufPoGlgb1KKWfX914D/A54NnBGRDwcmAXsXUrZv85zJvCHeXyPWfU9kiRJ09ZkVQHfDPwTOCQidomI1SZpufNyL/CTxvNeO8G1699NgaWAU3szlFIKWVI5rlLKoaWULUopW0xuqJIkSaNjUhLAUsocsmfvtcDXgGsj4ucRsflkLH8C/1c/txfDvfXfperfNerfG5hb/3NJkqROmbROIKWUS0spO5Jt+F5AJmKnR0Rbg01fW/+u2je9/7kkSVKnTHpyVkq5r7bL+wKwJpkQ9ruXsZK6+Zm+MC4C7ga2602IiABeMUnLlyRJmpImqxPIE4DPAccDfwFWAt4PXFBKuTnzrrlcCqxe7/rxB+DGUsrf6vTtIuKVZA/gq0spVy9MTKWUmyLiMODjEXEfY72AVwAcBkaSJHXWZJUAXgtcB3wY+D7wZTLh2naC+U8g7/bxGeA8YJ86/cvAWWQ7wvNY9B6576ufsw9wbI3xcOC2RVyuJEnSlBXZMbY7IuKHwBKllOc8xHzdWjHSiBjFY9I4tRiSNGpmL8goJsO6F3ArIuJ5wJbAb4ElgF2ArYFXtRmXJElSm6Z1AgjcDrwS+CDZueRyYPdSyonzfJckSdI0Nq0TwFLKecDT2o5DkiRplLQ1Rp8kSZJaYgIoSZLUMSaAkiRJHWMCKEmS1DEmgJIkSR1jAihJktQxJoCSJEkdM63HAZQ09YzibddG8fZ0MJrrStLUYAmgJElSx5gASpIkdYwJoCRJUseYAEqSJHWMCaAkSVLHmABKkiR1jAmgJElSx5gASpIkdYwJoCRJUseYAEqSJHWMCaAkSVLHmABKkiR1jAmgJElSx5gASpIkdYwJoCRJUseYAEqSJHXMlE4AI2LniLgoIu6JiH9GxH9HxIz62u4RUSJi04j4QUTcERGXRsQObcctSZLUpimbAEbEi4Djgd8C2wH/C7wHOKhv1mOAU4HtgcuB4yJi7SGGKkmSNFJmtB3AIvgE8JNSyn/W52dEBMCnI+KTjfn2L6V8DSAiZgPXAS8HDhlmsJIkSaNiSpYARsTiwJOAb/W9dDz5nbZqTDur908p5SbgemDcEsCImBUR50fE+ZMbsSRJ0uiYkgkgsAqwBFma19R7vnJj2i1989wLLDXeQksph5ZStiilbDEpUUqSJI2gqZoA3gjcB6zWN331+vfm4YYjSZI0dUzJBLCUcj8wG3hV30s7A3OAXw09KEmSpCliKncC+RhwZkQcARwHbArsCxxWSrmydgiRJElSnylZAghQSjkL2BXYAjgN2Av4PLBnm3FJkiSNuiiltB3DSIoIV4wkAEb1OGlNh6SG2QvSiXXKlgBKkiRp4ZgASpIkdYwJoCRJUseYAEqSJHWMCaAkSVLHmABKkiR1jAmgJElSx5gASpIkdYwJoCRJUseYAEqSJHWMCaAkSVLHzGg7AEkadaN6z91RvEfxqK4rSXOzBFCSJKljTAAlSZI6xgRQkiSpY0wAJUmSOsYEUJIkqWNMACVJkjrGBFCSJKljTAAlSZI6xgRQkiSpY0wAJUmSOsYEUJIkqWNMACVJkjrGBFCSJKljTAAlSZI6xgRQkiSpY6Z9AhgRW0XEqRFxTUTcERG/j4jXtB2XJElSW2a0HcAQrAucAxwC3A08AzgiIuaUUo5tNTJJkqQWRCml7RiGJiICWBz4ErBhKeX585i3OytG0pQ0isfvPMxKasHsUsoW8zvztC8BjIiVgI8D2wFrkQkgwFXjzDsLmDW86CRJkoZv2pcARsQpwNOAfYGLgduAtwDblVJWmcf7pveKkTTljeLx2xJAqTWWAPZExFLAy4G3lVIOaUyf9p1fJEmSJjLdE6GZ5He8pzchIpYHtm0tIkmSpJZN6xLAUsqtEXEesHdE3AbMAT4A3Aqs0GpwkiRJLZnuJYAArwb+AhwFHAB8u/4vSZLUSdO+E8jCshOIpFE3isdvO4FIrVmgTiBdKAGUJElSgwmgJElSx5gASpIkdYwJoCRJUseYAEqSJHWMCaAkSVLHmABKkiR1jAmgJElSx5gASpIkdYwJoCRJUseYAEqSJHXMjLYDkCQtnFG87673J5amBksAJUmSOsYEUJIkqWNMACVJkjrGBFCSJKljTAAlSZI6xgRQkiSpY0wAJUmSOsYEUJIkqWNMACVJkjrGBFCSJKljTAAlSZI6xgRQkiSpY0YqAYyI3SOiRMRyk7CsfSLixsmIS5IkaToZqQRwkn0VeHHbQUiSJI2aGW0HMCillCuBK+c1T0QsXUq5a0ghSZIkjYRWSgAj4tkR8eOIuD0ibo2In0TE5o1ZHhURP4iIOyLi0ojYoe/9L6uvXx8Rt0XEuRHxor555qoCjojn1urlF0fEqRFxO3DQYL+pJEnS6Bl6AhgRzwV+BNwH/CewC/BzYK3GbMcApwLbA5cDx0XE2o3XHwWcBrwO2BH4JfD9iHjGfIRwOHABsG39X5IkqVPaqAL+NJmAvbiUUuq0MyA7gdTn+5dSvlanzQauA14OHAJQSnmg5C4iFgN+DDwOeANwzkN8/rdKKR+dlG8iSZI0BQ21BDAilgW2BL7eSP7Gc1bvn1LKTcD1wAMlgBGxdkR8PSKuAv5Nlia+CNhoPsI4fR7xzYqI8yPi/PlYjiRJ0pQ07BLAlYAArnmI+W7pe34vsBQ8UOJ3KrA8sDdwBXAH8AlgtfmI4bqJXiilHAocWj9nXgmqJEnSlDXsBPBfwBxgzUVYxgbA5sBLSiln9CZGxNLz+X4TO0mS1GlDrQIupdwB/Br4j4iIhVxML9G7pzchItYF5qcDiCRJUue10QnkA8APyV67h5LVt1sB89vu7lJyfL/PR8RHyargjwNXDSBWSZKkaWfow8CUUn4GvBBYBjgaOB54Dg8xaHPj/fcAO5CdP04E9iV7Fv90EPFKkiRNNzHvzrjdZScQSVpwo3hOWfgWR9KUMruUssX8zjyd7wUsSZKkcZgASpIkdYwJoCRJUseYAEqSJHWMCaAkSVLHmABKkiR1jAmgJElSx5gASpIkdYwJoCRJUseYAEqSJHWMCaAkSVLHzGg7AEnS9DGK9931/sTSg1kCKEmS1DEmgJIkSR1jAihJktQxJoCSJEkdYwIoSZLUMSaAkiRJHWMCKEmS1DEmgJIkSR1jAihJktQxJoCSJEkdYwIoSZLUMSaAkiRJHWMCKEmS1DHzlQBGxJERcf6iflhE/CQiTnyIeZaLiBIRuzem/S0iPreony9JkiSY0XYA82l74Ka2g5AkSZoOpkQCWEr5XdsxSJIkTRcL1AYwIl4YERdGxB0R8YuIeFzjtWUi4sCIuDYi7o6I8yLiRfOxzB0j4k8RcVdE/AzYZJx55qoC7lVJzyueOt9KEXFcff3qiHh/RHwuIv62IN9bkiRpOlmQBHAd4LPAfwO7AasBx0dE1NcPA/aor28P/BM4PSKeOdECI+JJwPHABcAOwGnACZMUD8CRwAuBdwKzc3VMBgAADBhJREFUgBcBu8zn8iVJkqalBakCXhl4RinlcoCIWAz4DrBxTbp2A/YopXy9vn4mcCHwUeDFEyzzA8CfgJ1LKQX4fkQsCXxyUeIBLo2IxwPb1mV/q87zIzIxvX28BUbELDJRlCRJmrYWpATwb71kq7q4/l0beAoQwLd6L5ZS5tTnE5YAAk8FTq3JX89JkxAPwBb172mNmO4CfjjRAksph5ZStiilbDHRPJIkSVPdgiSAt/Q9v7f+XQpYE7i9lHJn3zzXActExMwJlrkGcH3ftP7nCxNPb9n/V0q5u2++G+Zz+ZIkSdPSZA0EfQ2wXEQs0zd9deDOUso9E7zvWrLtXlP/84V1LbB8RCzVN33VSVq+JEnSlDRZCeB5QAF26k2o7QJ3An7xEO/btq/jxg6TFFNv4OptGzEtTXYKkSRJ6qxJGQewlHJJRBwLHBQRywN/Bt5EDunylnm8dT/g18AJEXE48HjgDZMU0x8i4jTg4BrTtcB/AXcCcybjMyRJkqaiybwX8JuArwN7A6cA6wIvL6VMWAJYSjkf2BXYHDgZeCWTO0zL7mSnjwOBrwE/Bc4AbpvEz5AkSZpSYu4OuNNbRMzg/7d3r6GWlXUcx78/nXIcwhvSeEst7SIlNDVJZhe6C+oLC/OF4mjppKKlGJFBJUkkkQVZgpPpKGYXQSuL8B5SoTWjgpcuZioqonjpYuro5L8Xa43tTnPZ55w97n3O8/3A4sxe+1nP+q81B86P51kXuAO4uaqWbaJtOydGkuaxSfw7979XPkkjsXo6TzGZE6+Cm6kkhwG7ALcD29CNUr4WOGqcdUmSJI3TvA6AwL/o3k6yN7AlXRA8pKp+N9aqJEmSxqipKeDpcApYkuaHSfw75xSwNoNpTQGP8iYQSZIkzQEGQEmSpMYYACVJkhpjAJQkSWqMAVCSJKkxBkBJkqTGGAAlSZIaM98fBC1JapzP3BvOJD4vEfz/21wcAZQkSWqMAVCSJKkxBkBJkqTGGAAlSZIaYwCUJElqjAFQkiSpMQZASZKkxhgAJUmSGmMAlCRJaowBUJIkqTEGQEmSpMYYACVJkhpjAJQkSWrMnAqASe5L8vVx1yFJkjSXzakAKEmSpNnbLAEwyV6bo99J37ckSdJcMLIAmGRhkiOSXA/c3a/bM0klOXhK25VJVg18PiPJY0mWJLkpydNJbk3yrk3sc9ckf0xybZJF/eprk/wuySeTbDOq45MkSZovZh0A+9D2beBh4ALgceCgGXS1CLgIOA/4KLAGuHwg2E3d757AjcA9wMFV9XT/1RHAncDZwMN92NxokJQkSWrJjAJgkm2TnJhkNXALcADwJWDnqjqsqn45g263Bk6pqgv77Y8HdgTevZ79700X/m4DDq2qZ9d9V1W/rapjgJ2Ak4G9gRuT/CnJZ5Ms3shxLU+yanB0UpIkab6ZdgBMciDdaN+ZwG+AJVW1pKq+VVVPzKKW54BfDXy+q/+525R2r6cLf78GDq+q59bXWVU9VVUXVNU7+20uB04BHkxy7Aa2WVFVS6tq6cwPQ5IkabLNZARwDfA0sBDYFtguSUZQyz+r6oV1HwaC3cIp7d4B7AycX1Vrh+x7u35ZBDxLV78kSVKTph0Aq+oGYFfgE/3P64F7knwxyR5Tmq+bmn35lPXbT3e/Ay4Evgv8JMl+G2qUZHGS05LcAdwMLAE+QzdNfeks9i9JkjSnzegawKpaU1U/rKoPAHsB3weOA+7t78g9sm/6KPA8sM+6bZO8gm4UbzaOB34O/DLJvoNfJDkoyU+BB4HTgWuAN1XV26vq/Kp6apb7liRJmtNmfRdwVd1bVV8A9gQOAf5JN0pHP6X7U+DUJEf2j4O5Enhmlvt8ATiK7jrAq/ubQtY5h26q90hgl6o6tarunM3+JEmS5pMFo+qoqv4N/AL4xZQ7bU8CVgDnAk8CX6EbAXzTLPe3NsnH6ALldUneWVUPAPtX1SOz6VuSJGk+S1WNu4aJlMQTI0lqxqTmgdHcZ9qE1dN5ionvApYkSWqMAVCSJKkxBkBJkqTGGAAlSZIaYwCUJElqjAFQkiSpMQZASZKkxhgAJUmSGmMAlCRJaowBUJIkqTEjexewJEmau8678qpxl7BeH/rgMeMu4f9cfc2F4y5h1hwBlCRJaowBUJIkqTEGQEmSpMYYACVJkhpjAJQkSWqMAVCSJKkxBkBJkqTGGAAlSZIaYwCUJElqjAFQkiSpMQZASZKkxhgAJUmSGmMAlCRJaowBUJIkqTEGQEmSpMYYACVJkhpjAJQkSWqMAVCSJKkxBkBJkqTGLBh3AZMkyXJg+bjrkCRJ2pwMgAOqagWwAiBJjbkcSZKkzcIpYEmSpMYYACVJkhrTXABMclSStUn2GHctkiRJ49BcAKQ75i2BjLsQSZKkcWguAFbVyqpKVd037lokSZLGobkAKEmS1DoDoCRJUmMMgJIkSY0xAEqSJDXGAChJktQYA6AkSVJjDICSJEmNMQBKkiQ1xgAoSZLUGAOgJElSYwyAkiRJjUlVjbuGiZTEEyNJ0phNYk5JMu4S1md1VS0dtrEjgJIkSY0xAEqSJDXGAChJktQYA6AkSVJjDICSJEmNMQBKkiQ1xgAoSZLUGAOgJElSYwyAkiRJjTEASpIkNcYAKEmS1BgDoCRJUmMMgJIkSY0xAEqSJDVmswXAJHttrr43ss+dkix6qfcrSZI0l4w0ACZZmOSIJNcDdw+s3yLJ55L8JcmaJH9Osmw925+U5O6+zV+SnDrl+92S/DjJo0meSXJPkjMHmhwIPJzkvCRvG+WxSZIkzRcLRtFJkiXAJ4AjgEXAz4CDBpqcAywDvgzcAnwQuCDJ41X1876P4/p23wCuAt4LnJ1kq6o6q+/nYmBrYDnwN+A1wBsG9nMFsA1wDLA8ye3A+cAlVfXEKI5VkiRpzquqGS3AtsCJwGqggFuBTwE7TGm3N/ACsGzK+ouB3/f/3gJ4CLhwSptzgb8DC/vPTwGHDFnfW+gC5ePAs8APgA8AGXL7cnFxcXFxcRnvMonGfU42sKyqaeS4GU0BJzkQeBg4E/gNsKSqllTVt+r/R9reTxcAr0iyYN0CXAe8OcmWwG7ALsBlU7b9Ed2I3r7959uAryY5OsnuG6uxqm6pqpP7fpcB29ONLP51I8e1PMmqJKs2dQ4kSZLmqpleA7gGeBpYSDcSuF2SbKDtjsCWdCN5zw8sK+mmoHfuF4BHpmy77vMO/c/DgVXAN4H7k9yW5P2bqPXFGumO98kNNayqFVW1tKqWbqJPSZKkOWtG1wBW1Q1JdgUOBY4FrgfuS7ISuKiq7h9o/gSwFjiAbiRwqkf5bxB95ZTvFg/0QVU9BBydZAtgP+AM4GdJdq+qx9dt1IfR99FdC/gR4DngUuCEqrp1JscsSZI0X6Sbyp5lJ8mrgY8DRwO70gXClVV1SZLXA38APlxV12xg+y2AB4CrqurjA+u/AxwJLK6qZ9ez3f7Ab4G3VtUtSRYDJ/R17AHcCHwPuKyqnpnmMc3+xEiSpFkZRU4ZtQ1Peo7V6unMYI4kAL7YWXc934F0o4IHV9XL+vXn0k3ffo1uCnch8EbgdVV1bN/mOOA84GzgGuA9wOnA56vqrCTb0l3DdzHwZ2Ar4DRgH+A1VfVMkqOBs4CLgPOr6sVH0czgWCbvN06SpMYYAIc2vgD4Px0ni6vqkf7fAT4NHEd3V/A/gLuA71XVxQPbnNy32x14EDinqr7Zf7cV8G3g3cCr6K5BvAk4vapu79vsAPyjqtaOoP7J+42TJKkxBsChTUYAnOsMgJIkjd8k5pT5EAB9F7AkSVJjDICSJEmNMQBKkiQ1xgAoSZLUGAOgJElSYwyAkiRJjTEASpIkNcYAKEmS1BgDoCRJUmMMgJIkSY0xAEqSJDVmwbgLmGCPAfePqK8d+/4miTUNbxLrsqbhWNPwJrEuaxrOvK5pxO/dnc/nao/pNM4kvmR5vkmyajovaH4pWNPwJrEuaxqONQ1vEuuypuFY0/Amsa5x1eQUsCRJUmMMgJIkSY0xAL40Voy7gPWwpuFNYl3WNBxrGt4k1mVNw7Gm4U1iXWOpyWsAJUmSGuMIoCRJUmMMgJIkSY0xAEqSJDXGAChJktQYA6AkSVJj/gO9YAUcfQDu/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_attention(src, translation, attention)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BLEU\n",
    "\n",
    "Finally, we calculate the BLEU score for the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.data.metrics import bleu_score\n",
    "\n",
    "def calculate_bleu(data, src_field, trg_field, model, device, max_len = 50):\n",
    "    \n",
    "    trgs = []\n",
    "    pred_trgs = []\n",
    "    \n",
    "    for datum in data:\n",
    "        \n",
    "        src = vars(datum)['src']\n",
    "        trg = vars(datum)['trg']\n",
    "        \n",
    "        pred_trg, _ = translate_sentence(src, src_field, trg_field, model, device, max_len)\n",
    "        \n",
    "        #cut off <eos> token\n",
    "        pred_trg = pred_trg[:-1]\n",
    "        \n",
    "        pred_trgs.append(pred_trg)\n",
    "        trgs.append([trg])\n",
    "        \n",
    "    return bleu_score(pred_trgs, trgs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get a BLEU score of ~33, compared to the attention based RNN model which gave us a ~28. This is a ~17% improvement in BLEU score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BLEU score = 33.29\n"
     ]
    }
   ],
   "source": [
    "bleu_score = calculate_bleu(test_data, SRC, TRG, model, device)\n",
    "\n",
    "print(f'BLEU score = {bleu_score*100:.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've now covered the first of our non-RNN using models! Next up is the `Transformer` model which doesn't even use convolutional layers - only linear layers and a lot of attention mechanisms."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}