Tasks/task1/lab_01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 практическое задание. Полносвязная нейронная сеть на `numpy`.\n",
    "\n",
    "## Практикум на ЭВМ для 317 группы, весна 2021\n",
    "\n",
    "#### Фамилия, имя: \n",
    "\n",
    "Дата выдачи: <span style=\"color:red\">__4 марта 02:00__</span>.\n",
    "\n",
    "Мягкий дедлайн: <span style=\"color:red\">__17 марта 23:59__</span>.\n",
    "\n",
    "Стоимость: __10 баллов__ (основная часть заданий) + __3 балла__ (дополнительные задания).\n",
    "\n",
    "<span style=\"color:red\">__В ноутбуке все клетки должны выполняться без ошибок при последовательном их выполнении.__</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:48.909044Z",
     "start_time": "2021-03-03T14:44:48.683650Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Теоретическая часть (3 балла)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В этом блоке вам нужно решить 3 задания."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Задание 1. Градиенты для слоя Batch normalization (1.5 балла)\n",
    "\n",
    "Рассмотрим слой Batch normalization. Пусть на вход этого слоя был подан батч из $n$ объектов, при этом у всех объектов по 1 признаку. Представим вход BN слоя в виде $X \\in \\mathbb{R}^{n \\times 1}$.\n",
    "\n",
    "Тогда в этом слое производятся следующие вычисления:\n",
    "\n",
    "$$ \\mu = \\frac1n \\sum_{i=1}^{n} X_i $$\n",
    "\n",
    "$$ \\sigma^2 = \\frac1n \\sum_{i=1}^{n} \\left( X_i - \\mu \\right) ^2 $$\n",
    "\n",
    "$$ \\tilde{y_i} = \\frac{X_i - \\mu}{\\sqrt{\\sigma^2 + \\varepsilon}} $$\n",
    "\n",
    "$$ y_i = \\gamma \\tilde{y_i} + \\delta $$\n",
    "\n",
    "Выходом BN слоя является $y_i$, а $\\gamma$ и $\\delta$ в нем — параметры, которые подбираются во время обучения вместе с другими параметрами нейронной сети (наряду, например, с весами линейного слоя).\n",
    "\n",
    "Рассмотрим нейронную сеть, в которой есть BN слой. Предположим, что вычисления в нейронной завершаются подсчетом функции потерь $\\mathcal{L}$. Пусть мы выполнили прямой проход по нейронной сети и сейчас делаем обратный проход с помощью метода обратного распространения ошибки. Пусть BN слою пришел градиент функции потерь по выходу BN слоя ($\\nabla_{y} \\mathcal{L}$).\n",
    "\n",
    "В этом задании вам нужно записать вычисление градиента функции потерь по параметрам слоя BN $\\gamma$ и $\\delta$ ($\\nabla_{\\gamma} \\mathcal{L}$, $\\nabla_{\\delta} \\mathcal{L}$) через $\\nabla_{y} \\mathcal{L}$, а также вам нужно записать вычисление градиента функции потерь по входу слоя BN $X$ ($\\nabla_{X} \\mathcal{L}$) через $\\nabla_{y} \\mathcal{L}$.\n",
    "\n",
    "Хочу заметить, в данном задании мы рассматриваем объекты всего с 1 признаком, чтобы упростить выкладки."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Ваше решение:__\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Задание 2. Вывод инициализации весов линейного слоя при использовании ReLU в качестве функции активации (1 балл)\n",
    "\n",
    "Рассмотрим полносвязную нейронную сеть с функцией активации $g(y)$. Пусть сеть состоит из $L$ слоев и размер входа слоя $l$ равен $n_l\\ \\ (l = \\overline{1, L})$.\n",
    "\n",
    "Обозначим за $x^l \\in \\mathbb{R}^{n_{l}}$ вход слоя $l$, за $y^l \\in \\mathbb{R}^{n_{l+1}}$ — выход слоя $l$, за $W^l \\in \\mathbb{R}^{n_{l+1} \\times n_{l}}$ — веса слоя $l$, за $b^l \\in \\mathbb{R}^{n_{l+1}}$ — вектор сдвига слоя $l$.\n",
    "\n",
    "Тогда\n",
    "$$y^l = W^l x^l + b^l,$$\n",
    "$$x^{l+1} = g(y^l).$$\n",
    "\n",
    "На паре вы выводили хорошую инициализацию для линейного слоя в случае, когда в качестве функции активации $g(y)$ в нейронной сети используется гиперболический тангенс $g(y) = \\tanh(y)$. Сейчас вам нужно сделать подобный вывод для случая, когда в сети в качестве функций активации используется $g(y) = ReLU(y) = \\max(0, y)$.\n",
    "\n",
    "Сделаем следующие предположения насчет того, как распределены веса $W^l$, вектор сдвига $b^l$, входной вектор $x^l$, выходной вектор $y^l$ линейного слоя и градиенты функции потерь $\\frac{\\partial L}{\\partial y^{l}}$ $(l = \\overline{1, L}$):\n",
    "\n",
    "1. Все компоненты в $W^l$ распределены одинаково и независимо друг от друга;\n",
    "2. Все компоненты в $y^l$ распределены одинаково и независимо друг от друга;\n",
    "3. Все компоненты в $x^l$ распределены одинаково и независимо друг от друга;\n",
    "4. Все компоненты в $\\frac{\\partial L}{\\partial y^{l}}$ распределены одинаково и независимо друг от друга;\n",
    "5. Все компоненты в $W^l$ и все компоненты в $x^l$ независимы друг от друга;\n",
    "6. Все компоненты в $W^l$ имеют четную плотность распределения (то есть симметричную относительно нуля: $p_{W^l}(-x) = p_{W^l}(x)$);\n",
    "7. Все компоненты в $W^l$ имеют конечное матожидание;\n",
    "8. Вектор $b^l$ инициализирован нулями.\n",
    "\n",
    "\n",
    "Подсказки:\n",
    "1. Из пунктов 6 и 7 следует, что все компоненты в $W^l$ имеют нулевое среднее ($\\mathbb{E} W^l_{ij} = 0\\ \\ \\forall i = \\overline{1, n_{l+1}}, j = \\overline{1, n_{l}}$) (докажите);\n",
    "2. Из пунктов 6 и 7 следует, что все компоненты в $y^l$ имеют нулевое среднее и четную плотность распределения (докажите)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Ваше решение:__\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Задание 3. Почему функция активации $ReLU(y) = \\max(0, y)$ предпочтительней сигмоиды $\\sigma(y) = \\frac{1}{1 + \\exp(-y)}$ в нейронных сетях? (0.5 балла)\n",
    "\n",
    "Дайте развернутый ответ на вопрос \"Почему функция активации $ReLU(y) = \\max(0, y)$ предпочтительней сигмоиды $\\sigma(y) = \\frac{1}{1 + \\exp(-y)}$ в нейронных сетях?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "__Ваш ответ:__\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Практическая часть (7 баллов)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Реализация нейронной сети (3 балла)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "В этом задании вы обучите полносвязную нейронную сеть распознавать рукописные цифры (а что же еще, если не их :), [почти] самостоятельно реализовав все составляющие алгоритма обучения и предсказания.\n",
    "\n",
    "Для начала нам понадобится реализовать прямой и обратный проход через слои. Наши слои будут соответствовать следующему интерфейсу (на примере \"тождественного\" слоя):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:48.924967Z",
     "start_time": "2021-03-03T14:44:48.912996Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class IdentityLayer:\n",
    "    \"\"\"\n",
    "    A building block. Each layer is capable of performing two things:\n",
    "\n",
    "    - Process input to get output:           \n",
    "    output = layer.forward(input)\n",
    "\n",
    "    - Propagate gradients through itself:    \n",
    "    grad_input = layer.backward(input, grad_output)\n",
    "\n",
    "    Some layers also have learnable parameters.\n",
    "\n",
    "    Modified code from cs.hse DL course *\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        \"\"\"\n",
    "        Here you can initialize layer parameters (if any) \n",
    "        and auxiliary stuff. You should enumerate all parameters\n",
    "        in self.params\n",
    "        \"\"\"\n",
    "        # An identity layer does nothing\n",
    "        self.params = []\n",
    "        pass\n",
    "\n",
    "    def forward(self, input):\n",
    "        \"\"\"\n",
    "        Takes input data of shape [batch, input_units], \n",
    "        returns output data [batch, output_units]\n",
    "        \"\"\"\n",
    "        # An identity layer just returns whatever it gets as input.\n",
    "        self.input = input\n",
    "        return input\n",
    "\n",
    "    def backward(self, grad_output): \n",
    "        \"\"\"\n",
    "        Performs a backpropagation step through the layer, \n",
    "        with respect to the given input.\n",
    "\n",
    "        To compute loss gradients w.r.t input, \n",
    "        you need to apply chain rule (backprop):\n",
    "\n",
    "        d `loss` / d `input` = (d `loss` / d `layer`) * (d `layer` / d `input`)\n",
    "\n",
    "        Luckily, you already receive d `loss` / d `layer` in argument, \n",
    "        so you only need to multiply it by d `layer` / d `input`.\n",
    "\n",
    "        NB: Sometimes d `layer` / d `input` can be a 3D or even 4D tensor.\n",
    "        So it's better to write down the `loss` differential and extract\n",
    "        d `layer` / d `input` from it so that only 2D tensors were present.\n",
    "\n",
    "        The method returns:\n",
    "        * gradient w.r.t input (will be passed to \n",
    "          previous layer's backward method)\n",
    "        * flattened gradient w.r.t. parameters (with .ravel() \n",
    "          applied to each gradient). \n",
    "          If there are no params, return []\n",
    "        \"\"\"\n",
    "        # The gradient of an identity layer is precisely grad_output\n",
    "        input_dim = self.input.shape[1]\n",
    "\n",
    "        d_layer_d_input = np.eye(input_dim)\n",
    "\n",
    "        return np.dot(grad_output, d_layer_d_input), [] # chain rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Слой нелинейности ReLU\n",
    "\n",
    "Для начала реализуем слой нелинейности $ReLU(y) = \\max(0, y)$. Параметров у слоя нет. Метод `forward` должен вернуть результат поэлементного применения $ReLU$ к входному массиву, метод `backward` — градиент функции потерь по входу слоя. В нуле будем считать производную равной 0. Обратите внимание, что при обратном проходе могут понадобиться величины, посчитанные во время прямого прохода, поэтому их стоит сохранить как атрибут класса."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:48.939926Z",
     "start_time": "2021-03-03T14:44:48.928957Z"
    }
   },
   "outputs": [],
   "source": [
    "class ReLU:\n",
    "    \"\"\"\n",
    "    Modified code from cs.hse DL course *\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        \"\"\"ReLU layer simply applies elementwise rectified linear unit to all inputs\"\"\"\n",
    "        self.params = [] # ReLU has no parameters\n",
    "\n",
    "    def forward(self, input):\n",
    "        \"\"\"Apply elementwise ReLU to [batch, num_units] matrix\"\"\"\n",
    "        ### your code here\n",
    "        \n",
    "    def backward(self, grad_output):\n",
    "        \"\"\"Compute gradient of loss w.r.t. ReLU input\n",
    "        grad_output shape: [batch, num_units]\n",
    "        output 1 shape: [batch, num_units]\n",
    "        output 2: []\n",
    "        \"\"\"\n",
    "        ### your code here\n",
    "    \n",
    "    def __repr__(self):\n",
    "        return 'Relu()'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Полносвязный слой\n",
    "Далее реализуем полносвязный слой без нелинейности. У слоя два параметра: матрица весов и вектор сдвига.\n",
    "\n",
    "Обратите внимание на второй аргумент: в нем надо возвращать градиент по всем параметрам в одномерном виде. Для этого надо сначала применить `.ravel()` ко всем градиентам, а затем воспользоваться `np.r_`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:48.971838Z",
     "start_time": "2021-03-03T14:44:48.943913Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 0., 0., 0., 1., 0., 0., 0., 1., 0., 1., 2., 3.])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# example\n",
    "np.r_[np.eye(3).ravel(), np.arange(4)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:48.987794Z",
     "start_time": "2021-03-03T14:44:48.975827Z"
    }
   },
   "outputs": [],
   "source": [
    "class Dense:\n",
    "    \"\"\"\n",
    "    Modified code from cs.hse DL course *\n",
    "    \"\"\"\n",
    "    def __init__(self, input_units, output_units):\n",
    "        \"\"\"\n",
    "        A dense layer is a layer which performs a learned affine transformation:\n",
    "        f(x) = x W + b\n",
    "        \"\"\"\n",
    "        # initialize weights with small random numbers from normal distribution\n",
    "        self.weights = np.random.randn(input_units, output_units) * 0.01\n",
    "        self.biases = np.zeros(output_units)\n",
    "        self.params = [self.weights, self.biases]\n",
    "\n",
    "    def forward(self,input):\n",
    "        \"\"\"\n",
    "        Perform an affine transformation:\n",
    "        f(x) = x W + b\n",
    "        \n",
    "        input shape: [batch, input_units]\n",
    "        output shape: [batch, output units]\n",
    "        \"\"\"\n",
    "        ### your code here\n",
    "\n",
    "    def backward(self, grad_output):\n",
    "        \"\"\"\n",
    "        compute gradients\n",
    "        grad_output shape: [batch, output_units]\n",
    "        output shapes: [batch, input_units], [num_params]\n",
    "        \n",
    "        hint: use function np.r_\n",
    "        np.r_[np.arange(3), np.arange(3)] = [0, 1, 2, 0, 1, 2]\n",
    "        \"\"\"\n",
    "        ### your code here\n",
    "    \n",
    "    def __repr__(self):\n",
    "        return f'Dense({self.weights.shape[0]}, {self.weights.shape[1]})'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Проверка градиента\n",
    "\n",
    "Проверим правильность реализации с помощью функции численной проверки градиента. Функция `eval_numerical_gradient` принимает на вход callable объект `f` (функцию от одного аргумента-матрицы) и аргумент `x` и вычисляет приближенный градиент функции `f` в точке `x`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:49.003751Z",
     "start_time": "2021-03-03T14:44:48.990787Z"
    }
   },
   "outputs": [],
   "source": [
    "def eval_numerical_gradient(f, x, verbose=False, h=0.00001):\n",
    "    \"\"\"\n",
    "    Evaluates gradient df/dx via finite differences:\n",
    "    df/dx ~ (f(x+h) - f(x-h)) / 2h\n",
    "    Adopted from https://github.com/ddtm/dl-course/\n",
    "    \"\"\"\n",
    "    fx = f(x) # evaluate function value at original point\n",
    "    grad = np.zeros_like(x)\n",
    "    # iterate over all indexes in x\n",
    "    it = np.nditer(x, flags=['multi_index'], op_flags=['readwrite'])\n",
    "    while not it.finished:\n",
    "\n",
    "        # evaluate function at x+h\n",
    "        ix = it.multi_index\n",
    "        oldval = x[ix]\n",
    "        x[ix] = oldval + h # increment by h\n",
    "        fxph = f(x) # evalute f(x + h)\n",
    "        x[ix] = oldval - h\n",
    "        fxmh = f(x) # evaluate f(x - h)\n",
    "        x[ix] = oldval # restore\n",
    "\n",
    "        # compute the partial derivative with centered formula\n",
    "        grad[ix] = (fxph - fxmh) / (2 * h) # the slope\n",
    "        if verbose:\n",
    "            print (ix, grad[ix])\n",
    "        it.iternext() # step to next dimension\n",
    "\n",
    "    return grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Вычислите аналитический и численный градиенты по входу слоя ReLU от функции\n",
    "$$ f(y) = \\sum_i y_i, \\quad y = ReLU(x) $$\n",
    "\n",
    "Выпишите аналитический градиент в этой ячейке:\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Следующая ячейка после заполнения должна не выдавать ошибку :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:49.034673Z",
     "start_time": "2021-03-03T14:44:49.006744Z"
    }
   },
   "outputs": [],
   "source": [
    "relu = ReLU()\n",
    "points = np.linspace(-1, 1, 10*12).reshape([10, 12])\n",
    "\n",
    "### your code here\n",
    "### grads = \n",
    "### numeric_grads = \n",
    "\n",
    "assert np.allclose(grads, numeric_grads, rtol=1e-3, atol=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Вычислите аналитический и численный градиенты по входу полносвязного слоя от функции\n",
    "$$ f(y) = \\sum_i y_i, \\quad y = W x + b $$\n",
    "\n",
    "Выпишите аналитический градиент в этой ячейке (советуем выписать градиент через дифференциал функции $f$):\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Следующая ячейка после заполнения должна не выдавать ошибку :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:49.066624Z",
     "start_time": "2021-03-03T14:44:49.041651Z"
    }
   },
   "outputs": [],
   "source": [
    "linear = Dense(12, 32)\n",
    "points = np.linspace(-1, 1, 10*12).reshape([10, 12])\n",
    "\n",
    "### your code here\n",
    "### grads = \n",
    "### numeric_grads = \n",
    "\n",
    "assert np.allclose(grads, numeric_grads, rtol=1e-3, atol=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Реализация softmax-слоя и функции потерь"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для решения задачи многоклассовой классификации обычно используют $softmax$ в качестве нелинейности на последнем слое, чтобы получить вероятности классов для каждого объекта:\n",
    "$$\\hat y = softmax(x)  = \\left \\{\\frac {\\exp(x^i)}{\\sum_{j=1}^K \\exp(x^j)} \\right \\}_{i=1}^K, \\quad K - \\text{число классов.}$$\n",
    "\n",
    "Здесь за $x^i$ мы обозначаем $i$-ый признак объекта $x$.\n",
    "\n",
    "В качестве функции потерь выберем отрицательный логарифм правдоподобия (по английски: negative log likelihood или NLL)\n",
    "$$L(y, \\hat y) = -\\sum_{i=1}^K y^i \\log \\hat y^i,$$\n",
    "где $y^i = 1$, если объект принадлежит $i$-му классу, и $y^i = 0$ иначе.\n",
    "\n",
    "NLL совпадает с выражением для [кросс-энтропии](https://ru.wikipedia.org/wiki/Перекрёстная_энтропия) (в качестве первого распределения берем вырожденное распределение $y$, в качестве второго — предсказанное распределение $\\hat y$). Очевидно, что эту функцию потерь также можно переписать через индексацию, если через $y$ обозначить класс данного объекта:\n",
    "$$L(y, \\hat y) = - \\log \\hat y_{y}$$\n",
    "\n",
    "В таком виде ее удобно реализовывать."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T13:13:02.970075Z",
     "start_time": "2021-03-03T13:13:02.961134Z"
    }
   },
   "source": [
    "Для обучения нейронной сети будем оптимизировать эту функцию потерь по параметрам нейронной сети:\n",
    "\n",
    "$$ \\frac1N \\sum_{i=1}^N L(y_i, \\hat y_i) = \\frac1N \\sum_{i=1}^N L(y_i, \\text{NN}(x_i)) \\rightarrow \\min_{w}\\,,$$\n",
    "где за $x_i$ и $y_i$ мы обозначили признаки и таргет $i$-ого объекта обучающей выборки, за $\\text{NN}$ мы обозначили нейронную сеть, которая по признакам объекта $x_i$ выдает распределение вероятностей $\\hat y_i$, за $w$ мы обозначили все веса нейронной сети, а $N$ — это число объектов в обучающей выборке."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реализуйте слой `LogSoftmax` (у этого слоя нет параметров). Метод `forward` должен вычислять логарифм от $softmax$, а метод `backward` — пропускать градиенты. В общем случае в промежуточных вычислениях `backward` получится трехмерный тензор, однако для нашей конкретной функции потерь все вычисления можно реализовать в матричном виде.  Поэтому мы будем предполагать, что аргумент `grad_output` — это матрица, у которой в каждой строке только одно ненулевое значение (не обязательно единица)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:49.272035Z",
     "start_time": "2021-03-03T14:44:49.070612Z"
    }
   },
   "outputs": [],
   "source": [
    "from scipy.special import logsumexp\n",
    "# use this function instead of np.log(np.sum(np.exp(...))) because it is more stable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:49.288035Z",
     "start_time": "2021-03-03T14:44:49.275031Z"
    }
   },
   "outputs": [],
   "source": [
    "class LogSoftmax:\n",
    "    def __init__(self):\n",
    "        self.params = []\n",
    "\n",
    "    def forward(self, input):\n",
    "        \"\"\"\n",
    "        Applies softmax to each row and then applies component-wise log\n",
    "        Input shape: [batch, num_units]\n",
    "        Output shape: [batch, num_units]\n",
    "        \"\"\"\n",
    "        ### your code here\n",
    "        \n",
    "    def backward(self, grad_output):\n",
    "        \"\"\"\n",
    "        Propagartes gradients.\n",
    "        Assumes that each row of grad_output contains only 1 \n",
    "        non-zero element\n",
    "        Input shape: [batch, num_units]\n",
    "        Output shape: [batch, num_units]\n",
    "        Do not forget to return [] as second value (grad w.r.t. params)\n",
    "        \"\"\"\n",
    "        ### your code here\n",
    "    \n",
    "    def __repr__(self):\n",
    "        return 'LogSoftmax()'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реализуйте функцию потерь и градиенты функции потерь. Во время вычисления NLL усредняйте (а не суммируйте) значения функции потерь по батчу. Обычно так делают для того, чтобы при двух запусках обучения нейронной сети с разными размерами батча получаемые значения функции потерь у этих сетей были сравнимы."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:49.302990Z",
     "start_time": "2021-03-03T14:44:49.290985Z"
    }
   },
   "outputs": [],
   "source": [
    "def NLL(activations, target):\n",
    "    \"\"\"\n",
    "    Returns negative log-likelihood of target under model represented by\n",
    "    activations (log probabilities of classes, it's just output of LogSoftmax layer).\n",
    "    `activations` has shape [batch, num_classes], `target` has shape [batch]\n",
    "    Output shape: 1 (scalar).\n",
    "    \"\"\"\n",
    "    ### your code here\n",
    "\n",
    "\n",
    "def grad_NLL(activations, target):\n",
    "    \"\"\"\n",
    "    Returns gradient of negative log-likelihood w.r.t. activations.\n",
    "    each arg has shape [batch, num_classes]\n",
    "    output shape: [batch, num-classes]\n",
    "    \"\"\"\n",
    "    ### your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Наконец, выполните проверку `LogSoftmax`-слоя, используя функцию потерь и ее градиент:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:49.350863Z",
     "start_time": "2021-03-03T14:44:49.305945Z"
    }
   },
   "outputs": [],
   "source": [
    "lsm = LogSoftmax()\n",
    "target = np.arange(10)\n",
    "points = np.linspace(-1, 1, 10 * 12).reshape([10, 12])\n",
    "\n",
    "### your code here\n",
    "### grads = \n",
    "### numeric_grads = \n",
    "\n",
    "assert np.allclose(grads, numeric_grads, rtol=1e-3, atol=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Загрузка данных\n",
    "\n",
    "Мы реализаовали все архитектурные составляющие нашей нейронной сети. Осталось загрузить данные и обучить модель. Мы будем работать с датасетом `digits`, каждый объект в котором — это 8x8 изображение рукописной цифры."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:49.872521Z",
     "start_time": "2021-03-03T14:44:49.354818Z"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:51.700670Z",
     "start_time": "2021-03-03T14:44:49.876510Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:51.873771Z",
     "start_time": "2021-03-03T14:44:51.703667Z"
    }
   },
   "outputs": [],
   "source": [
    "X, y = load_digits(return_X_y=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:51.889252Z",
     "start_time": "2021-03-03T14:44:51.877326Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1797, 64), (1797,))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Разделим данные на обучение и контроль:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:51.952606Z",
     "start_time": "2021-03-03T14:44:51.892280Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:51.968609Z",
     "start_time": "2021-03-03T14:44:51.955642Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:52.000485Z",
     "start_time": "2021-03-03T14:44:51.971599Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1347, 64), (450, 64))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Сборка и обучение нейронной сети (1 балл)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В нашей реализации мы представляем нейронную сеть в виде списка ее слоев. Например, следующая функция конструирует нейронную сеть заданной ширины (то есть с заданным размером скрытых слоев) и глубины (то есть с заданным количеством слоев) с заданным размером входа и выхода, а также с заданной функцией активации между линейными слоями."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:52.016442Z",
     "start_time": "2021-03-03T14:44:52.003513Z"
    }
   },
   "outputs": [],
   "source": [
    "def make_network(input_size, hidden_layers_size, output_size, n_layers=3, activation_class=ReLU):\n",
    "    network = []\n",
    "\n",
    "    for layer_idx in range(n_layers):\n",
    "        # Compute sizes of current linear layer\n",
    "        layer_in = input_size if layer_idx == 0 else hidden_layers_size\n",
    "        layer_out = output_size if layer_idx == n_layers - 1 else hidden_layers_size\n",
    "        \n",
    "        # Add linear layer to the network\n",
    "        network.append(Dense(layer_in, layer_out))\n",
    "\n",
    "        # Add activation after each layer except the last one\n",
    "        if layer_idx != n_layers - 1:\n",
    "            network.append(activation_class())\n",
    "\n",
    "    # Add LogSoftmax layer to the network\n",
    "    network.append(LogSoftmax())\n",
    "\n",
    "    return network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:52.048395Z",
     "start_time": "2021-03-03T14:44:52.020431Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Dense(64, 32), Relu(), Dense(32, 32), Relu(), Dense(32, 10), LogSoftmax()]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_size = X_train.shape[1]\n",
    "hidden_layers_size = 32\n",
    "output_size = 10\n",
    "\n",
    "network = make_network(input_size, hidden_layers_size, output_size, 3, ReLU)\n",
    "network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реализуйте функцию, которая выполнет прямой проход по нейронной сети:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:52.063319Z",
     "start_time": "2021-03-03T14:44:52.051386Z"
    }
   },
   "outputs": [],
   "source": [
    "def forward(network, X):\n",
    "    \"\"\"\n",
    "    Perform forward pass through the network.\n",
    "    \n",
    "    network: list of layers\n",
    "    X: raw data\n",
    "    X shape: [batch, features_num]\n",
    "\n",
    "    output: \n",
    "    output shape: [batch, out_features_num]\n",
    "    \"\"\"\n",
    "    ### your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для проверки, хорошо ли сеть обучилась, нам понадобится вычислять долю правильных ответов (accuracy) на данной выборке. Для этого реализуйте функцию, которая делает предсказания на каждом объекте (логично в качестве предсказания на очередном объекте выдавать тот класс, для которого предсказанный логарифм вероятности максимален):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:52.063319Z",
     "start_time": "2021-03-03T14:44:52.051386Z"
    }
   },
   "outputs": [],
   "source": [
    "def predict(network, X):\n",
    "    \"\"\"\n",
    "    Returns predictions for each object in X.\n",
    "    \n",
    "    network: list of layers\n",
    "    X: raw data\n",
    "    X shape: [batch, features_num]\n",
    "\n",
    "    output: array of classes, each from 0 to 9\n",
    "    output shape: [batch]\n",
    "    \"\"\"\n",
    "    ### your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Мы будем обучать параметры нейросети с помощью готовой функции оптимизации из модуля `scipy`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:52.079311Z",
     "start_time": "2021-03-03T14:44:52.070308Z"
    }
   },
   "outputs": [],
   "source": [
    "from scipy.optimize import minimize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:52.111226Z",
     "start_time": "2021-03-03T14:44:52.085292Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function minimize in module scipy.optimize._minimize:\n",
      "\n",
      "minimize(fun, x0, args=(), method=None, jac=None, hess=None, hessp=None, bounds=None, constraints=(), tol=None, callback=None, options=None)\n",
      "    Minimization of scalar function of one or more variables.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    fun : callable\n",
      "        The objective function to be minimized.\n",
      "    \n",
      "            ``fun(x, *args) -> float``\n",
      "    \n",
      "        where ``x`` is an 1-D array with shape (n,) and ``args``\n",
      "        is a tuple of the fixed parameters needed to completely\n",
      "        specify the function.\n",
      "    x0 : ndarray, shape (n,)\n",
      "        Initial guess. Array of real elements of size (n,),\n",
      "        where 'n' is the number of independent variables.\n",
      "    args : tuple, optional\n",
      "        Extra arguments passed to the objective function and its\n",
      "        derivatives (`fun`, `jac` and `hess` functions).\n",
      "    method : str or callable, optional\n",
      "        Type of solver.  Should be one of\n",
      "    \n",
      "            - 'Nelder-Mead' :ref:`(see here) <optimize.minimize-neldermead>`\n",
      "            - 'Powell'      :ref:`(see here) <optimize.minimize-powell>`\n",
      "            - 'CG'          :ref:`(see here) <optimize.minimize-cg>`\n",
      "            - 'BFGS'        :ref:`(see here) <optimize.minimize-bfgs>`\n",
      "            - 'Newton-CG'   :ref:`(see here) <optimize.minimize-newtoncg>`\n",
      "            - 'L-BFGS-B'    :ref:`(see here) <optimize.minimize-lbfgsb>`\n",
      "            - 'TNC'         :ref:`(see here) <optimize.minimize-tnc>`\n",
      "            - 'COBYLA'      :ref:`(see here) <optimize.minimize-cobyla>`\n",
      "            - 'SLSQP'       :ref:`(see here) <optimize.minimize-slsqp>`\n",
      "            - 'trust-constr':ref:`(see here) <optimize.minimize-trustconstr>`\n",
      "            - 'dogleg'      :ref:`(see here) <optimize.minimize-dogleg>`\n",
      "            - 'trust-ncg'   :ref:`(see here) <optimize.minimize-trustncg>`\n",
      "            - 'trust-exact' :ref:`(see here) <optimize.minimize-trustexact>`\n",
      "            - 'trust-krylov' :ref:`(see here) <optimize.minimize-trustkrylov>`\n",
      "            - custom - a callable object (added in version 0.14.0),\n",
      "              see below for description.\n",
      "    \n",
      "        If not given, chosen to be one of ``BFGS``, ``L-BFGS-B``, ``SLSQP``,\n",
      "        depending if the problem has constraints or bounds.\n",
      "    jac : {callable,  '2-point', '3-point', 'cs', bool}, optional\n",
      "        Method for computing the gradient vector. Only for CG, BFGS,\n",
      "        Newton-CG, L-BFGS-B, TNC, SLSQP, dogleg, trust-ncg, trust-krylov,\n",
      "        trust-exact and trust-constr.\n",
      "        If it is a callable, it should be a function that returns the gradient\n",
      "        vector:\n",
      "    \n",
      "            ``jac(x, *args) -> array_like, shape (n,)``\n",
      "    \n",
      "        where ``x`` is an array with shape (n,) and ``args`` is a tuple with\n",
      "        the fixed parameters. If `jac` is a Boolean and is True, `fun` is\n",
      "        assumed to return and objective and gradient as an ``(f, g)`` tuple.\n",
      "        Methods 'Newton-CG', 'trust-ncg', 'dogleg', 'trust-exact', and\n",
      "        'trust-krylov' require that either a callable be supplied, or that\n",
      "        `fun` return the objective and gradient.\n",
      "        If None or False, the gradient will be estimated using 2-point finite\n",
      "        difference estimation with an absolute step size.\n",
      "        Alternatively, the keywords  {'2-point', '3-point', 'cs'} can be used\n",
      "        to select a finite difference scheme for numerical estimation of the\n",
      "        gradient with a relative step size. These finite difference schemes\n",
      "        obey any specified `bounds`.\n",
      "    hess : {callable, '2-point', '3-point', 'cs', HessianUpdateStrategy}, optional\n",
      "        Method for computing the Hessian matrix. Only for Newton-CG, dogleg,\n",
      "        trust-ncg,  trust-krylov, trust-exact and trust-constr. If it is\n",
      "        callable, it should return the  Hessian matrix:\n",
      "    \n",
      "            ``hess(x, *args) -> {LinearOperator, spmatrix, array}, (n, n)``\n",
      "    \n",
      "        where x is a (n,) ndarray and `args` is a tuple with the fixed\n",
      "        parameters. LinearOperator and sparse matrix returns are\n",
      "        allowed only for 'trust-constr' method. Alternatively, the keywords\n",
      "        {'2-point', '3-point', 'cs'} select a finite difference scheme\n",
      "        for numerical estimation. Or, objects implementing\n",
      "        `HessianUpdateStrategy` interface can be used to approximate\n",
      "        the Hessian. Available quasi-Newton methods implementing\n",
      "        this interface are:\n",
      "    \n",
      "            - `BFGS`;\n",
      "            - `SR1`.\n",
      "    \n",
      "        Whenever the gradient is estimated via finite-differences,\n",
      "        the Hessian cannot be estimated with options\n",
      "        {'2-point', '3-point', 'cs'} and needs to be\n",
      "        estimated using one of the quasi-Newton strategies.\n",
      "        Finite-difference options {'2-point', '3-point', 'cs'} and\n",
      "        `HessianUpdateStrategy` are available only for 'trust-constr' method.\n",
      "    hessp : callable, optional\n",
      "        Hessian of objective function times an arbitrary vector p. Only for\n",
      "        Newton-CG, trust-ncg, trust-krylov, trust-constr.\n",
      "        Only one of `hessp` or `hess` needs to be given.  If `hess` is\n",
      "        provided, then `hessp` will be ignored.  `hessp` must compute the\n",
      "        Hessian times an arbitrary vector:\n",
      "    \n",
      "            ``hessp(x, p, *args) ->  ndarray shape (n,)``\n",
      "    \n",
      "        where x is a (n,) ndarray, p is an arbitrary vector with\n",
      "        dimension (n,) and `args` is a tuple with the fixed\n",
      "        parameters.\n",
      "    bounds : sequence or `Bounds`, optional\n",
      "        Bounds on variables for L-BFGS-B, TNC, SLSQP, Powell, and\n",
      "        trust-constr methods. There are two ways to specify the bounds:\n",
      "    \n",
      "            1. Instance of `Bounds` class.\n",
      "            2. Sequence of ``(min, max)`` pairs for each element in `x`. None\n",
      "               is used to specify no bound.\n",
      "    \n",
      "    constraints : {Constraint, dict} or List of {Constraint, dict}, optional\n",
      "        Constraints definition (only for COBYLA, SLSQP and trust-constr).\n",
      "    \n",
      "        Constraints for 'trust-constr' are defined as a single object or a\n",
      "        list of objects specifying constraints to the optimization problem.\n",
      "        Available constraints are:\n",
      "    \n",
      "            - `LinearConstraint`\n",
      "            - `NonlinearConstraint`\n",
      "    \n",
      "        Constraints for COBYLA, SLSQP are defined as a list of dictionaries.\n",
      "        Each dictionary with fields:\n",
      "    \n",
      "            type : str\n",
      "                Constraint type: 'eq' for equality, 'ineq' for inequality.\n",
      "            fun : callable\n",
      "                The function defining the constraint.\n",
      "            jac : callable, optional\n",
      "                The Jacobian of `fun` (only for SLSQP).\n",
      "            args : sequence, optional\n",
      "                Extra arguments to be passed to the function and Jacobian.\n",
      "    \n",
      "        Equality constraint means that the constraint function result is to\n",
      "        be zero whereas inequality means that it is to be non-negative.\n",
      "        Note that COBYLA only supports inequality constraints.\n",
      "    tol : float, optional\n",
      "        Tolerance for termination. For detailed control, use solver-specific\n",
      "        options.\n",
      "    options : dict, optional\n",
      "        A dictionary of solver options. All methods accept the following\n",
      "        generic options:\n",
      "    \n",
      "            maxiter : int\n",
      "                Maximum number of iterations to perform. Depending on the\n",
      "                method each iteration may use several function evaluations.\n",
      "            disp : bool\n",
      "                Set to True to print convergence messages.\n",
      "    \n",
      "        For method-specific options, see :func:`show_options()`.\n",
      "    callback : callable, optional\n",
      "        Called after each iteration. For 'trust-constr' it is a callable with\n",
      "        the signature:\n",
      "    \n",
      "            ``callback(xk, OptimizeResult state) -> bool``\n",
      "    \n",
      "        where ``xk`` is the current parameter vector. and ``state``\n",
      "        is an `OptimizeResult` object, with the same fields\n",
      "        as the ones from the return. If callback returns True\n",
      "        the algorithm execution is terminated.\n",
      "        For all the other methods, the signature is:\n",
      "    \n",
      "            ``callback(xk)``\n",
      "    \n",
      "        where ``xk`` is the current parameter vector.\n",
      "    \n",
      "    Returns\n",
      "    -------\n",
      "    res : OptimizeResult\n",
      "        The optimization result represented as a ``OptimizeResult`` object.\n",
      "        Important attributes are: ``x`` the solution array, ``success`` a\n",
      "        Boolean flag indicating if the optimizer exited successfully and\n",
      "        ``message`` which describes the cause of the termination. See\n",
      "        `OptimizeResult` for a description of other attributes.\n",
      "    \n",
      "    See also\n",
      "    --------\n",
      "    minimize_scalar : Interface to minimization algorithms for scalar\n",
      "        univariate functions\n",
      "    show_options : Additional options accepted by the solvers\n",
      "    \n",
      "    Notes\n",
      "    -----\n",
      "    This section describes the available solvers that can be selected by the\n",
      "    'method' parameter. The default method is *BFGS*.\n",
      "    \n",
      "    **Unconstrained minimization**\n",
      "    \n",
      "    Method :ref:`Nelder-Mead <optimize.minimize-neldermead>` uses the\n",
      "    Simplex algorithm [1]_, [2]_. This algorithm is robust in many\n",
      "    applications. However, if numerical computation of derivative can be\n",
      "    trusted, other algorithms using the first and/or second derivatives\n",
      "    information might be preferred for their better performance in\n",
      "    general.\n",
      "    \n",
      "    Method :ref:`CG <optimize.minimize-cg>` uses a nonlinear conjugate\n",
      "    gradient algorithm by Polak and Ribiere, a variant of the\n",
      "    Fletcher-Reeves method described in [5]_ pp.120-122. Only the\n",
      "    first derivatives are used.\n",
      "    \n",
      "    Method :ref:`BFGS <optimize.minimize-bfgs>` uses the quasi-Newton\n",
      "    method of Broyden, Fletcher, Goldfarb, and Shanno (BFGS) [5]_\n",
      "    pp. 136. It uses the first derivatives only. BFGS has proven good\n",
      "    performance even for non-smooth optimizations. This method also\n",
      "    returns an approximation of the Hessian inverse, stored as\n",
      "    `hess_inv` in the OptimizeResult object.\n",
      "    \n",
      "    Method :ref:`Newton-CG <optimize.minimize-newtoncg>` uses a\n",
      "    Newton-CG algorithm [5]_ pp. 168 (also known as the truncated\n",
      "    Newton method). It uses a CG method to the compute the search\n",
      "    direction. See also *TNC* method for a box-constrained\n",
      "    minimization with a similar algorithm. Suitable for large-scale\n",
      "    problems.\n",
      "    \n",
      "    Method :ref:`dogleg <optimize.minimize-dogleg>` uses the dog-leg\n",
      "    trust-region algorithm [5]_ for unconstrained minimization. This\n",
      "    algorithm requires the gradient and Hessian; furthermore the\n",
      "    Hessian is required to be positive definite.\n",
      "    \n",
      "    Method :ref:`trust-ncg <optimize.minimize-trustncg>` uses the\n",
      "    Newton conjugate gradient trust-region algorithm [5]_ for\n",
      "    unconstrained minimization. This algorithm requires the gradient\n",
      "    and either the Hessian or a function that computes the product of\n",
      "    the Hessian with a given vector. Suitable for large-scale problems.\n",
      "    \n",
      "    Method :ref:`trust-krylov <optimize.minimize-trustkrylov>` uses\n",
      "    the Newton GLTR trust-region algorithm [14]_, [15]_ for unconstrained\n",
      "    minimization. This algorithm requires the gradient\n",
      "    and either the Hessian or a function that computes the product of\n",
      "    the Hessian with a given vector. Suitable for large-scale problems.\n",
      "    On indefinite problems it requires usually less iterations than the\n",
      "    `trust-ncg` method and is recommended for medium and large-scale problems.\n",
      "    \n",
      "    Method :ref:`trust-exact <optimize.minimize-trustexact>`\n",
      "    is a trust-region method for unconstrained minimization in which\n",
      "    quadratic subproblems are solved almost exactly [13]_. This\n",
      "    algorithm requires the gradient and the Hessian (which is\n",
      "    *not* required to be positive definite). It is, in many\n",
      "    situations, the Newton method to converge in fewer iteraction\n",
      "    and the most recommended for small and medium-size problems.\n",
      "    \n",
      "    **Bound-Constrained minimization**\n",
      "    \n",
      "    Method :ref:`L-BFGS-B <optimize.minimize-lbfgsb>` uses the L-BFGS-B\n",
      "    algorithm [6]_, [7]_ for bound constrained minimization.\n",
      "    \n",
      "    Method :ref:`Powell <optimize.minimize-powell>` is a modification\n",
      "    of Powell's method [3]_, [4]_ which is a conjugate direction\n",
      "    method. It performs sequential one-dimensional minimizations along\n",
      "    each vector of the directions set (`direc` field in `options` and\n",
      "    `info`), which is updated at each iteration of the main\n",
      "    minimization loop. The function need not be differentiable, and no\n",
      "    derivatives are taken. If bounds are not provided, then an\n",
      "    unbounded line search will be used. If bounds are provided and\n",
      "    the initial guess is within the bounds, then every function\n",
      "    evaluation throughout the minimization procedure will be within\n",
      "    the bounds. If bounds are provided, the initial guess is outside\n",
      "    the bounds, and `direc` is full rank (default has full rank), then\n",
      "    some function evaluations during the first iteration may be\n",
      "    outside the bounds, but every function evaluation after the first\n",
      "    iteration will be within the bounds. If `direc` is not full rank,\n",
      "    then some parameters may not be optimized and the solution is not\n",
      "    guaranteed to be within the bounds.\n",
      "    \n",
      "    Method :ref:`TNC <optimize.minimize-tnc>` uses a truncated Newton\n",
      "    algorithm [5]_, [8]_ to minimize a function with variables subject\n",
      "    to bounds. This algorithm uses gradient information; it is also\n",
      "    called Newton Conjugate-Gradient. It differs from the *Newton-CG*\n",
      "    method described above as it wraps a C implementation and allows\n",
      "    each variable to be given upper and lower bounds.\n",
      "    \n",
      "    **Constrained Minimization**\n",
      "    \n",
      "    Method :ref:`COBYLA <optimize.minimize-cobyla>` uses the\n",
      "    Constrained Optimization BY Linear Approximation (COBYLA) method\n",
      "    [9]_, [10]_, [11]_. The algorithm is based on linear\n",
      "    approximations to the objective function and each constraint. The\n",
      "    method wraps a FORTRAN implementation of the algorithm. The\n",
      "    constraints functions 'fun' may return either a single number\n",
      "    or an array or list of numbers.\n",
      "    \n",
      "    Method :ref:`SLSQP <optimize.minimize-slsqp>` uses Sequential\n",
      "    Least SQuares Programming to minimize a function of several\n",
      "    variables with any combination of bounds, equality and inequality\n",
      "    constraints. The method wraps the SLSQP Optimization subroutine\n",
      "    originally implemented by Dieter Kraft [12]_. Note that the\n",
      "    wrapper handles infinite values in bounds by converting them into\n",
      "    large floating values.\n",
      "    \n",
      "    Method :ref:`trust-constr <optimize.minimize-trustconstr>` is a\n",
      "    trust-region algorithm for constrained optimization. It swiches\n",
      "    between two implementations depending on the problem definition.\n",
      "    It is the most versatile constrained minimization algorithm\n",
      "    implemented in SciPy and the most appropriate for large-scale problems.\n",
      "    For equality constrained problems it is an implementation of Byrd-Omojokun\n",
      "    Trust-Region SQP method described in [17]_ and in [5]_, p. 549. When\n",
      "    inequality constraints  are imposed as well, it swiches to the trust-region\n",
      "    interior point  method described in [16]_. This interior point algorithm,\n",
      "    in turn, solves inequality constraints by introducing slack variables\n",
      "    and solving a sequence of equality-constrained barrier problems\n",
      "    for progressively smaller values of the barrier parameter.\n",
      "    The previously described equality constrained SQP method is\n",
      "    used to solve the subproblems with increasing levels of accuracy\n",
      "    as the iterate gets closer to a solution.\n",
      "    \n",
      "    **Finite-Difference Options**\n",
      "    \n",
      "    For Method :ref:`trust-constr <optimize.minimize-trustconstr>`\n",
      "    the gradient and the Hessian may be approximated using\n",
      "    three finite-difference schemes: {'2-point', '3-point', 'cs'}.\n",
      "    The scheme 'cs' is, potentially, the most accurate but it\n",
      "    requires the function to correctly handles complex inputs and to\n",
      "    be differentiable in the complex plane. The scheme '3-point' is more\n",
      "    accurate than '2-point' but requires twice as many operations.\n",
      "    \n",
      "    **Custom minimizers**\n",
      "    \n",
      "    It may be useful to pass a custom minimization method, for example\n",
      "    when using a frontend to this method such as `scipy.optimize.basinhopping`\n",
      "    or a different library.  You can simply pass a callable as the ``method``\n",
      "    parameter.\n",
      "    \n",
      "    The callable is called as ``method(fun, x0, args, **kwargs, **options)``\n",
      "    where ``kwargs`` corresponds to any other parameters passed to `minimize`\n",
      "    (such as `callback`, `hess`, etc.), except the `options` dict, which has\n",
      "    its contents also passed as `method` parameters pair by pair.  Also, if\n",
      "    `jac` has been passed as a bool type, `jac` and `fun` are mangled so that\n",
      "    `fun` returns just the function values and `jac` is converted to a function\n",
      "    returning the Jacobian.  The method shall return an `OptimizeResult`\n",
      "    object.\n",
      "    \n",
      "    The provided `method` callable must be able to accept (and possibly ignore)\n",
      "    arbitrary parameters; the set of parameters accepted by `minimize` may\n",
      "    expand in future versions and then these parameters will be passed to\n",
      "    the method.  You can find an example in the scipy.optimize tutorial.\n",
      "    \n",
      "    .. versionadded:: 0.11.0\n",
      "    \n",
      "    References\n",
      "    ----------\n",
      "    .. [1] Nelder, J A, and R Mead. 1965. A Simplex Method for Function\n",
      "        Minimization. The Computer Journal 7: 308-13.\n",
      "    .. [2] Wright M H. 1996. Direct search methods: Once scorned, now\n",
      "        respectable, in Numerical Analysis 1995: Proceedings of the 1995\n",
      "        Dundee Biennial Conference in Numerical Analysis (Eds. D F\n",
      "        Griffiths and G A Watson). Addison Wesley Longman, Harlow, UK.\n",
      "        191-208.\n",
      "    .. [3] Powell, M J D. 1964. An efficient method for finding the minimum of\n",
      "       a function of several variables without calculating derivatives. The\n",
      "       Computer Journal 7: 155-162.\n",
      "    .. [4] Press W, S A Teukolsky, W T Vetterling and B P Flannery.\n",
      "       Numerical Recipes (any edition), Cambridge University Press.\n",
      "    .. [5] Nocedal, J, and S J Wright. 2006. Numerical Optimization.\n",
      "       Springer New York.\n",
      "    .. [6] Byrd, R H and P Lu and J. Nocedal. 1995. A Limited Memory\n",
      "       Algorithm for Bound Constrained Optimization. SIAM Journal on\n",
      "       Scientific and Statistical Computing 16 (5): 1190-1208.\n",
      "    .. [7] Zhu, C and R H Byrd and J Nocedal. 1997. L-BFGS-B: Algorithm\n",
      "       778: L-BFGS-B, FORTRAN routines for large scale bound constrained\n",
      "       optimization. ACM Transactions on Mathematical Software 23 (4):\n",
      "       550-560.\n",
      "    .. [8] Nash, S G. Newton-Type Minimization Via the Lanczos Method.\n",
      "       1984. SIAM Journal of Numerical Analysis 21: 770-778.\n",
      "    .. [9] Powell, M J D. A direct search optimization method that models\n",
      "       the objective and constraint functions by linear interpolation.\n",
      "       1994. Advances in Optimization and Numerical Analysis, eds. S. Gomez\n",
      "       and J-P Hennart, Kluwer Academic (Dordrecht), 51-67.\n",
      "    .. [10] Powell M J D. Direct search algorithms for optimization\n",
      "       calculations. 1998. Acta Numerica 7: 287-336.\n",
      "    .. [11] Powell M J D. A view of algorithms for optimization without\n",
      "       derivatives. 2007.Cambridge University Technical Report DAMTP\n",
      "       2007/NA03\n",
      "    .. [12] Kraft, D. A software package for sequential quadratic\n",
      "       programming. 1988. Tech. Rep. DFVLR-FB 88-28, DLR German Aerospace\n",
      "       Center -- Institute for Flight Mechanics, Koln, Germany.\n",
      "    .. [13] Conn, A. R., Gould, N. I., and Toint, P. L.\n",
      "       Trust region methods. 2000. Siam. pp. 169-200.\n",
      "    .. [14] F. Lenders, C. Kirches, A. Potschka: \"trlib: A vector-free\n",
      "       implementation of the GLTR method for iterative solution of\n",
      "       the trust region problem\", :arxiv:`1611.04718`\n",
      "    .. [15] N. Gould, S. Lucidi, M. Roma, P. Toint: \"Solving the\n",
      "       Trust-Region Subproblem using the Lanczos Method\",\n",
      "       SIAM J. Optim., 9(2), 504--525, (1999).\n",
      "    .. [16] Byrd, Richard H., Mary E. Hribar, and Jorge Nocedal. 1999.\n",
      "        An interior point algorithm for large-scale nonlinear  programming.\n",
      "        SIAM Journal on Optimization 9.4: 877-900.\n",
      "    .. [17] Lalee, Marucha, Jorge Nocedal, and Todd Plantega. 1998. On the\n",
      "        implementation of an algorithm for large-scale equality constrained\n",
      "        optimization. SIAM Journal on Optimization 8.3: 682-706.\n",
      "    \n",
      "    Examples\n",
      "    --------\n",
      "    Let us consider the problem of minimizing the Rosenbrock function. This\n",
      "    function (and its respective derivatives) is implemented in `rosen`\n",
      "    (resp. `rosen_der`, `rosen_hess`) in the `scipy.optimize`.\n",
      "    \n",
      "    >>> from scipy.optimize import minimize, rosen, rosen_der\n",
      "    \n",
      "    A simple application of the *Nelder-Mead* method is:\n",
      "    \n",
      "    >>> x0 = [1.3, 0.7, 0.8, 1.9, 1.2]\n",
      "    >>> res = minimize(rosen, x0, method='Nelder-Mead', tol=1e-6)\n",
      "    >>> res.x\n",
      "    array([ 1.,  1.,  1.,  1.,  1.])\n",
      "    \n",
      "    Now using the *BFGS* algorithm, using the first derivative and a few\n",
      "    options:\n",
      "    \n",
      "    >>> res = minimize(rosen, x0, method='BFGS', jac=rosen_der,\n",
      "    ...                options={'gtol': 1e-6, 'disp': True})\n",
      "    Optimization terminated successfully.\n",
      "             Current function value: 0.000000\n",
      "             Iterations: 26\n",
      "             Function evaluations: 31\n",
      "             Gradient evaluations: 31\n",
      "    >>> res.x\n",
      "    array([ 1.,  1.,  1.,  1.,  1.])\n",
      "    >>> print(res.message)\n",
      "    Optimization terminated successfully.\n",
      "    >>> res.hess_inv\n",
      "    array([[ 0.00749589,  0.01255155,  0.02396251,  0.04750988,  0.09495377],  # may vary\n",
      "           [ 0.01255155,  0.02510441,  0.04794055,  0.09502834,  0.18996269],\n",
      "           [ 0.02396251,  0.04794055,  0.09631614,  0.19092151,  0.38165151],\n",
      "           [ 0.04750988,  0.09502834,  0.19092151,  0.38341252,  0.7664427 ],\n",
      "           [ 0.09495377,  0.18996269,  0.38165151,  0.7664427,   1.53713523]])\n",
      "    \n",
      "    \n",
      "    Next, consider a minimization problem with several constraints (namely\n",
      "    Example 16.4 from [5]_). The objective function is:\n",
      "    \n",
      "    >>> fun = lambda x: (x[0] - 1)**2 + (x[1] - 2.5)**2\n",
      "    \n",
      "    There are three constraints defined as:\n",
      "    \n",
      "    >>> cons = ({'type': 'ineq', 'fun': lambda x:  x[0] - 2 * x[1] + 2},\n",
      "    ...         {'type': 'ineq', 'fun': lambda x: -x[0] - 2 * x[1] + 6},\n",
      "    ...         {'type': 'ineq', 'fun': lambda x: -x[0] + 2 * x[1] + 2})\n",
      "    \n",
      "    And variables must be positive, hence the following bounds:\n",
      "    \n",
      "    >>> bnds = ((0, None), (0, None))\n",
      "    \n",
      "    The optimization problem is solved using the SLSQP method as:\n",
      "    \n",
      "    >>> res = minimize(fun, (2, 0), method='SLSQP', bounds=bnds,\n",
      "    ...                constraints=cons)\n",
      "    \n",
      "    It should converge to the theoretical solution (1.4 ,1.7).\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(minimize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Эта функция имеет стандартный интерфейс: нужно передать callable объект, который вычисляет значение и градиент целевой функции, а также точку старта оптимизации — начальное приближение (одномерный `numpy`-массив). Поэтому нам понадобятся функции для сбора и задания всех весов нашей нейросети (именно для них мы всегда записывали параметры слоя в список `layer.params`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:52.127145Z",
     "start_time": "2021-03-03T14:44:52.114180Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_weights(network):\n",
    "    weights = []\n",
    "    for layer in network:\n",
    "        for param in layer.params:\n",
    "            weights += param.ravel().tolist()\n",
    "    return np.array(weights)\n",
    "\n",
    "\n",
    "def set_weights(weights, network):\n",
    "    i = 0\n",
    "    for layer in network:\n",
    "        for param in layer.params:\n",
    "            l = param.size\n",
    "            param[:] = weights[i:i+l].reshape(param.shape)\n",
    "            i += l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Вам нужно реализовать ту самую функцию, которую мы будем передавать в `minimize`. Эта функция должна брать на вход текущую точку (вектор всех параметров), а также список дополнительных параметров (мы будем передавать через них нашу сеть и обучающие данные) и возвращать значение критерия качества (NLL) и его градиент по параметрам модели."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:52.143103Z",
     "start_time": "2021-03-03T14:44:52.131134Z"
    }
   },
   "outputs": [],
   "source": [
    "def compute_loss_grad(weights, args):\n",
    "    \"\"\"\n",
    "    takes current weights and computes cross-entropy and gradients\n",
    "    weights shape: [num_parameters]\n",
    "    output 1: loss (scalar)\n",
    "    output 2: gradint w.r.t. weights, shape: [num_parameters]\n",
    "    \n",
    "    hint: firstly perform forward pass through the whole network\n",
    "    then compute loss and its gradients\n",
    "    then perform backward pass, transmitting first baskward output\n",
    "    to the previos layer and saving second baskward output in a list\n",
    "    finally flatten all the gradients in this list\n",
    "    (in the order from the first to the last layer)\n",
    "    \n",
    "    Do not forget to set weights of the network!\n",
    "    \"\"\"\n",
    "    network, X, y = args\n",
    "    ### your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь мы готовы обучать нашу нейросеть. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:52.175017Z",
     "start_time": "2021-03-03T14:44:52.148090Z"
    }
   },
   "outputs": [],
   "source": [
    "weights = get_weights(network)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:53.078366Z",
     "start_time": "2021-03-03T14:44:52.179006Z"
    }
   },
   "outputs": [],
   "source": [
    "res = minimize(\n",
    "    compute_loss_grad, weights,       # fun and start point\n",
    "    args=[network, X_train, y_train], # args passed to fun\n",
    "    method=\"L-BFGS-B\",                # optimization method\n",
    "    jac=True                          # says that gradient is computed in fun\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:53.093332Z",
     "start_time": "2021-03-03T14:44:53.081402Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['fun', 'jac', 'nfev', 'njev', 'nit', 'status', 'message', 'x', 'success', 'hess_inv'])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:53.109290Z",
     "start_time": "2021-03-03T14:44:53.096325Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "res[\"nit\"] # number of iterations (should be >> 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:53.125246Z",
     "start_time": "2021-03-03T14:44:53.112286Z"
    }
   },
   "outputs": [],
   "source": [
    "res[\"success\"] # should be True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:53.140244Z",
     "start_time": "2021-03-03T14:44:53.128239Z"
    }
   },
   "outputs": [],
   "source": [
    "res[\"x\"] # leraned weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Выведите качество на обучении (`X_train`, `y_train`) и на контроле (`X_test`, `y_test`). Не забудьте установить веса!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:53.172124Z",
     "start_time": "2021-03-03T14:44:53.144212Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NLL (Train / Test)     : 0.000 / 1.004\n",
      "Accuracy (Train / Test): 1.000 / 0.947\n"
     ]
    }
   ],
   "source": [
    "### your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "У `minimize` есть также аргумент `callback` — в нее можно передать функцию, которая будет вызываться после каждой итерации оптимизации. Такую функцию удобно оформить в виде метода класса, который будет сохранять качество на обучении контроле после каждой итерации. Реализуйте этот метод в классе `Callback`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:53.188082Z",
     "start_time": "2021-03-03T14:44:53.175117Z"
    }
   },
   "outputs": [],
   "source": [
    "class Callback:\n",
    "    def __init__(self, network, X_train, y_train, X_test, y_test, print=False):\n",
    "        self.network = network\n",
    "        self.X_train = X_train\n",
    "        self.X_test = X_test\n",
    "        self.y_train = y_train\n",
    "        self.y_test = y_test\n",
    "        self.print = print\n",
    "        self.train_acc = []\n",
    "        self.test_acc = []\n",
    "        \n",
    "    def call(self, weights):\n",
    "        \"\"\"\n",
    "        Computes quality on train and test set with given weights\n",
    "        and saves to self.train_acc and self.test_acc.\n",
    "        If self.print is True, also prints these 2 values\n",
    "        \"\"\"\n",
    "        ### your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:54.497924Z",
     "start_time": "2021-03-03T14:44:53.191074Z"
    }
   },
   "outputs": [],
   "source": [
    "cb = Callback(network, X_train, y_train, X_test, y_test, print=True)\n",
    "\n",
    "res = minimize(\n",
    "    compute_loss_grad, weights,  \n",
    "    args=[network, X_train, y_train], \n",
    "    method=\"L-BFGS-B\",\n",
    "    jac=True,\n",
    "    callback=cb.call\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Изобразите на графике кривую качества на обучени и контроле по итерациям:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:54.816117Z",
     "start_time": "2021-03-03T14:44:54.500922Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAFgCAYAAACmDI9oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABPbElEQVR4nO3dd3hc1Z3/8ffRqIzKqFiS5SL33sDGxgVTRMeEHkJCCQnJAlnCb5NNNgnpZZNdEjY9pJCEdCC0gGkJJQgI4N6wJfcq2ZZkWW1URpqZ8/vjjqSRrWZJoxlLn9fz6JHm3jt3vjq69nx0dO45xlqLiIiIiIg44qJdgIiIiIhILFFAFhEREREJo4AsIiIiIhJGAVlEREREJIwCsoiIiIhImPhoF3CqcnJy7MSJEwf9devr60lNTR301x0q1H59p7brH7Vf/6j9+k5t1z9qv/5R+/XO+vXrj1lrc0/cftoF5IkTJ7Ju3bpBf93CwkIKCgoG/XWHCrVf36nt+kft1z9qv75T2/WP2q9/1H69Y4w50Nl2DbEQEREREQmjgCwiIiIiEkYBWUREREQkzGk3BrkzLS0tlJSU0NTUFLHXyMjIoLi4OGLnjxa3201+fj4JCQnRLkVEREQkJgyJgFxSUoLH42HixIkYYyLyGnV1dXg8noicO1qstVRWVlJSUsKkSZOiXY6IiIhITBgSQyyamprIzs6OWDgeqowxZGdnR7TnXUREROR0MyQCMqBw3EdqNxEREZGOhkxAFhEREREZCArIA6C6upqf//znfXrulVdeSXV19cAWJCIiIiJ9FrGAbIx52BhTbozZ2sV+Y4z5iTFmtzFmizHmrEjVEmndBWS/39/tc1988UUyMzMjUJWIiIiI9EUke5B/D1zRzf4VwLTQx13ALyJYS0Tdd9997Nmzh/nz5/O5z32OwsJCzjvvPK655hpmz54NwHXXXcfChQuZM2cODz30UNtzJ06cyLFjx9i/fz+zZs3izjvvZM6cOVx22WU0Njae9FrPPfccS5YsYcGCBVxyySWUlZUB4PV6ueOOO5g3bx5nnHEGTz31FAB///vfOeusszjzzDO5+OKLB6E1RERERE5vEZvmzVr7pjFmYjeHXAv80VprgVXGmExjzGhr7ZH+vO43n9tG0eHa/pziJLPHpPOZgvFd7r///vvZunUrmzZtApz1zzds2MDWrVvbpk97+OGHGTFiBI2NjZx99tm8//3vJzs7u8N5du3axaOPPsqvf/1rbrrpJp566iluu+22Dsece+65rFq1CmMMv/nNb/je977H97//ff77v/+bjIwM3nvvPQCqqqqoqKjgzjvv5M0332TSpEkcP358AFtFREQGWzBoqW/24/X5qWtq/WihrslPY0sg2uUNqB0lLZSvOxTtMk5bp1v7nTcth9EZydEuo00050EeC4T/5EpC204KyMaYu3B6mcnLy6OwsLDD/oyMDOrq6gBoaW4hEBjY/yRaz9n6Gifyer0Eg8G2/Q0NDSxcuJCcnJy2bQ888ADPP/88AIcOHWLTpk0sXrwYay1erxev18uECROYMmUKdXV1zJ07lx07dpz0mjt27OBLX/oSZWVlNDc3M2HCBOrq6nj55Zd5+OGH246Pj4/nlVdeYdmyZW11JCQkdPo9NDU1ndSmA83r9Ub8NYYqtV3/qP36J5barzlgOVofpMRrKa0LUtYQJD4O3PGGlHiDOx5S4g3J8ZAcb3DHG3wBS6MfGv2W5gCkJ8IIdxwj3IYstyHRdfJMPo1+y57qIAdqA9S3OI8b/RZfANyu9nO74zv+GdYYp5bkUA1+XxNv/fkVjjdZjjdZapstQdv19xe0zms1+S0NfvAF7En7m/zQzSmGnq1bol3B6e00ar/PLEzijNzYWZ4jdirphrX2IeAhgEWLFtmCgoIO+4uLi9sW8fj2++dHpIbuFgpJS0sjLi6ubX9KSgrp6eltjwsLC3nrrbdYvXo1KSkpFBQU4HK58Hg8GGNIS0sDIDk5ucM5vF7vSa9533338ZnPfIZrrrmGwsJCvvGNb+DxeIiLiyMtLa3D8cnJySQkJPS4wInb7WbBggV9a5heKiws5MSfm/SO2q5/1H79E4n28weCHDjewN6KetKS4hmT6WZUhpukeBfgLGJU3xygqr6ZLSU1rDtwnA0Hqth2uBZ/KGHGxxnGZ6cQ9Fu8dX5qm/w0+4OnXEt2aiKjM92MzkgmIzmBraU17Cirw4ZSaKIrDo87njR3PMkJLip9Aby1Tq9tS6CnqGqAZgBy0pLI9STh6mZgY5wxjEyKx+OOx+NOICXRRXh8j4szeJKcfWlu57i00GNPqL6hNHPnqlWrWLp0abTLOG2dbu2Xk5aEO8EV7TLaRDMglwLjwh7nh7addjweT5e9ywA1NTVkZWWRkpLC9u3bWbVqVZ9fq6amhrFjxwLwhz/8oW37pZdeyoMPPsiPfvQjwBlisXTpUu655x727dvXNsRixIgRfX5tEYk91lr2Hatn/YEqNh6qxtcSDAWs1gCV0BbwPEnx1Da1cLi6iSM1jRyt8ZGcGMfojGTGZiYzOsNNcmLHN6hqXxBr7Ulzpvv8AfZW1LOzrI4dR+vYWeZlT4UXY3ACWyjoueLanxcIWvZXNrCn3Etz4OQwOyI1kZZAEK/P3xZQAZLi4zhzXCZ3nj+Z2aPTmTHKw8TsVBLjO6ZNnz+ANzTswOvzU+/zk5zoaguQSfFxHPM2c6S6kcM1Te2faxo5WNnA8YZmZo7ycPmcUSyamMWZ4zJJdyd02fbN/iA2rD83GASvr3X4Qwur1q7n8vOXdgj/0ns5yXHkZ6VEu4zTltqvf6IZkFcC9xpjHgOWADX9HX8cLdnZ2Sxfvpy5c+eyYsUK3ve+93XYf8UVV/DLX/6SWbNmMWPGjH79RveNb3yDD3zgA2RlZXHRRRexb98+AL7yla/wyU9+krlz5+Jyufj617/ODTfcwEMPPcQNN9xAMBhk5MiRvPLKK/36XkVkcFTVN7OzrA6vr30mHGuhqqGZIzVNHK5upLS6kW2Hazle7/RSetzxpLsTqG1qOSlknsgVZxjpSaKhOUBNY0u3tXxz9StMz/MwdWQaVQ3N7Dhax/7KBgJhvbmTc1OZNdr5q1jruNiy2iYCYUUYYNyIFM6flsP0PA+Tc1NpaA5wuLqRIzVNlNU2keCKIz0U6NPdCcwanc6s0eknheHOJMW7SEpzkZ2W1OUxHncCk3JSezxXb3RWU3Kii1yP8/rHd7uYkD0wryUigytiAdkY8yhQAOQYY0qArwMJANbaXwIvAlcCu4EG4I5I1TIYHnnkkQ6Pw/8kmZSUxEsvvdTp8/bv3w9ATk4OW7e2z4j3X//1X50ef+2113LttdeetD0tLa1Dj3KrFStWsGLFip7KF5EIq6jzsfFgVVuYBWcsqa8l0NbjWdvk59DxBnaU1VFR5+v2fDlpSYzJdHPhjJEsmpjFwglZTM1NIy7UYxsMWhpaWntUW6gL3dTlccczJiM59Od+59iGZj9Hapo4Ut2Ez99+D0fQwutrtmDT89hxtI7nNh9mRGoi0/M8XDlvNNPyPMzI8zAp5+TeXBGR01kkZ7G4uYf9FvhkpF5fRCQSgkHLngov6w5Usf5AFce8HYNsQmjMqifJ6QU9UtPE+gNVHKhs6Pa87oQ40pISGJPp5oLpuczI8zAtL40RqYkdjstMTiQvI6nHP9nHxRnSkpwxqqMy3N0em5IYz5TcNKbkpp20L6E8gYKCM7p9vojIUHNa3KQnIjJYrLVUN7RwuKaRI6GxuuHjVbcfqaW2yRn2MCI1kfys5A43Uvn8wbYe4bqmFkakJnLW+CxuWTyehROyGJOZ3OFGqqR4F2lJ8eqBFRGJIQrIIjIkFR+p5Sev7WLTvgbOPLSe6aOc4QBp7vgON2lV1je3zSNb1+Snst5HU0vHG8gSXIa8dDdjMpK5ct5oFk5whjRMykk96ea1cDY0/ra7Y0REJPYoIIvIacMfCHK4usnp3a1p5HB1E8GgZVpeGtPzPEzITmXfMS8/fHUXL2w5gicpnsnpcewsq+PloqMd5qA1BkZ6kshJS8LjjmfciBQ8SfFkpSYyOsPtzOqQmcyYDDc5aUltY3tPhYKxiMjpSQFZRGJOMGgpqWpkR1kdO0MfO47WsbeivtPpwVolxsfhDwRJTnBx74VTufO8yWxc8zYFBQU0tQTYXe6loTnA6Aw3eeluDWsQEZFOKSCLSNRZa9l2uJbnNh/m3b2V7Crzdlg2d2xmMtPz0rhgRi5TctIYk5kcWtzBuflsd7k3NBdvHSmJ8XzknIkn3dzmTnAxd2zGoH5fIiJyelJAHgDV1dU88sgj3HPPPX16/o9+9CPuuusuUlI0obcMLwcrG3hi/SGe23yY/ZUNxMcZFk3M4ubF45mel8b0UR6mjUzD081iDQBn5GdyRn7m4BQtIiJDngLyAKiurubnP/95vwLybbfdpoAsw8bGg1X8+q29/H3rUQCWTcnm7gumcMWcUWSd0PMrIjIkWQu1pVC+HSq2gzsdRs6G3BmQ5Bn41wsGofoAlBfDsZ0QaO75OW0MZE2A3JmQMx0Sup86cihQQB4A9913H3v27GH+/PlceumlPPDAAzzwwAM8/vjj+Hw+rr/+er75zW9SX1/PTTfdRElJCYFAgK9+9auUlZVx+PBhLrzwQnJycnj99dc7nPtb3/oWzz33HI2NjZxzzjn86le/whjD7t27+cQnPkFFRQUul4snnniCKVOm8N3vfpc///nPxMXFsWLFCu6///4otYqcDmqbWnh9ezml1e1TmtU2+hmZnuQMY8hwMzojmTGZzuectMRe33gWCFp2ltWxq9xLbWNL27Rna/dVsWb/cTzueO46fwofPWdij/P0Si80VsELn4WKnR23j5oHc2+AyQXg6r4nfliyFsqLYOvTULIG5twAC24b+LYKtEBDJaSOhLgIjH33+6DuCN0un5iQAmkjIRZvHm1pcuoPk16zA9aHAl1FMdRXdnxOciaMnOWEtpGzYeRMSM4anHobjkNTTdgGCzUlTtgtL4KKHeCr6/r5Ngg1h8BX2/n+jHHgzuxXiYu8XigOzW1uA1C1H1q6n4+9V0wcZE+FqZc6/7eMXRib11Q/Db2A/NJ9cPS9gT3nqHlw7pe73H3//fezdetWNm3aBMDLL7/Mrl27WLNmDdZarrnmGt58800qKioYM2YML7zwAgA1NTVkZGTwgx/8gNdff52cnJyTzn3vvffyta99DYAPf/jDPP/881x99dXceuut3HfffVx//fU0NTURDAZ56aWXePbZZ1m9ejUpKSkcP358YNtBhgyvz8/v397HQ2/ubZvTNyM5gdEZbtKTE9haWsPLRWU0+zveEJfoimNybiofXjaB95+VjzuhfbEKay1bSmp4fUc56w9UsfFgdYdlkgHijLPU8FfeN4sPLR5PWtLQ+y8oKo7vhb/c5LwBTr3EeQMDCPph+wuw+REnOMy6GhZ9DMYsiGq5URUMBZPyYji8AbY9A8d2OG2WOR6e/zT864dwwRfgjA8626v3O8Gnah8E28fGExfvBIWRM51AY4yzv2o/lBczYf+L8OQfQz12uyDYAgmpzvG5syB7MsR1E8Tjk0Lnnw2eUc75Ay3Oz7u8KCyMbYfKPU4I6ok7w3ntkWEfubMgLdfZ7/c5tZYXg68GcmY4r5+a7exvaYLK1v11oXA6C1JGnPrPwu+DPf90fjnZ8SI0ezvsPgtgI06wz50JmeOgbdZxC/UVsOlRaA4Lop7Roe9rtvPLAKcQ3FKyQ+0xAxJDS4Q3VjvtW17cHtTLi53X7oo70zlP5vjuX2/CsvZwnzvTae/y4vaA3dy/MNvkP0ZaZihXGAOTLnCuvdZe6sSTFwbqUtDf8bo7sgnW/hpWPeh8n3Oud365HH3mkAnLeneKgJdffpmXX36ZBQucNyGv18uuXbs477zz+OxnP8sXvvAFrrrqKs4777wez/X666/zve99j4aGBo4fP86cOXMoKCigtLSU66+/HgC32+l9e/XVV7njjjvahmqMGNGH/7BkSKtpaOGv6w7yyzf2cry+mYtnjuSeC6cwc1Q6qSeEVWstx+ubOVLTxOHqRg5XN3Kkpol391by5b9t5Qcv7+T2ZRM5d1oOrxWX8fyWIxw83oAxMHNUOtctGMPCCVnMHp1BZkoCHnc8yQmu4Tn1WXMD7HoZip51Qs+c62HyhRAfGk7SXA87/+6EtRN60Ugd2THMZE9rfx7AwVXw2C1Oj9Ttz8LE5R2fHx5Ctj4NG/4IM94HF34JRs3t2/dTf8z5Xnb+3XmTbQtaM0/407CB1ByI637VvzaN1ZCU3v8e1qr9sO1vsPs18De1b/f7nDf5tiBmYMJyWHIXzLrWqXXXy/D6d+DZe+C1bzohsDe9bokeSB/j/Ak79JqTwAkPI2fDtMuc/cf3OiFo18uwqbz335M7A9Ly4Pg+J2i31j9istP2s6+FrIlgumlrX20o7G132mf979r3peQ4PbLH93UetFNznZ9N1T7nWjtRWl57MG0NfRn5HcNSQ2V7ACzfDvv/5YRCd6bzb2L8svZf7oD3dh9i3kUfgMwJXV8TtrXntrj9l4XyIlj7m44/+1NinJ9boAXqDrdvTkxzguX0y51fKlKyT2iDkZA3x2mLvvw/l5rt/Dxnvq+PdXe0tbCQgoKCATkXca72f+etGqudX2y2PgXvPghv/xhGTHF+ljOvdHrYW3+Jq9wTdt3iXKcjJnW8Zk68XqJs6AXkFREaUlDXzZ9KTmCt5Ytf/CJ33333Sfs2bNjAiy++yFe+8hUuvvjitt7hzjQ1NXHPPfewbt06xo0bxze+8Q2amvr6D16GK6/PzytFR3l+8xHe3FVBS8By3rQcPnPpdBaM7/rPkcYYstOSyE5L6jD7g7WW1fuO89Cbe/nhqzv54as7ccUZlk/N4d6LpnL57FFkpAzzP+WH97Lt/AfseAla6p2QEWiGzY86oWDWVU543vl3J4SljXLeMFrfJKyFyt3O/tbQYlztvZaeMbDut07v5a1PQPaUk2uJT4IZK5yPplpY/Ut452fwy+Uw+zpY9knIP/vkN6bGatj1CmNL1sDq0LCNgM8JnfvedOoZMdnpMd32dNdtEe92xix21mvV1isV6plrqAz1Fs5wAkjudKfHtbd8tc4bdul65/HoMzuGGHcmTDgnrNduhhMKw02/3Amz21+A9x6H9LHtx2dPAVfYLyf+plBvayiY1ZTCtEvbemXf2l7OeZes6Lre5obOw2bb/nqnd7t8u9M+3nKYcWV7UMmZDgnJvW+fcNaCt6xjT3RTtXNNtJ7fneH0ZLb+fJpqYe7723shE9Oc/a29quVFsP73Pf9CYVyQM825/mdf5wz/Cf+lL6SyutAJUd2eyzg9y5njYPpl7duDAWhpPJUGgbqjoe811GPsSjghwI2LzPCY01VyJsy/xfloOA7FK51fwv/1A3jr/9qPS8lxrtXwX54DLbC30Pm/sNUH/+JcEzFi6AXkKPB4PNSFBejLL7+cr371q9x6662kpaVRWlpKQkICfr+fESNGcNttt5GZmclvfvObDs8/cYhFaxjOycnB6/Xy5JNPcuONN+LxeMjPz+eZZ57huuuuw+fzEQgEuPTSS/nWt77Frbfe2jbEQr3Iw4vX52f9gSp2Hq1rm0N4+9E6mv1BRme4+ciyiVw7fyzz8vs+3ZkxhqWTs1k6OZtdZXUUHanl3Kk5ZKclDeB3MkCs7frPztbCOz91evIGUt1RJ/S1vm5yFpzxAefPjxPPdd64977uvJFsC/Uon3mzM5Zv/LLOe1v9Picot/6Zt7wYjmyBopUw6Tz4wB969ydudzpc8HlYfCe8+3NY9QsoesZ5459zndOLWrXPqW3PaxBoZhrA7rBzZE2E5Z9y6s2b6wQUn9cJchU7wR8WSsKGG7DvTdjy2Mk1JXqcwDXzfU7vU2to2/NPZ2jIqRp1BlzyTacXK2vCqT8fnO9p1lU9v1knpTm9zhOWdbo7sLuw++cn9nBjdlIaePJg0vndH9cXxjjDNjyjYMpFXR+XkQ9TL+56f+Y4mHZJ++PWG8Eqtp/815Ck9NBfQaY6130kxbmc9jsVSR4nuHNNREoa0lJGwMKPOh/ectj7hnPthg/f6UxjVfsvaPmLBqvaXlFAHgDZ2dksX76cuXPnsmLFCh544AGKi4tZtsz5TzMtLY0///nP7N69m8997nPExcWRkJDAL37xCwDuuusurrjiCsaMGdPhJr3MzEzuvPNO5s6dy6hRozj77LPb9v3pT3/i7rvv5mtf+xoJCQk88cQTXHHFFWzatIlFixaRmJjIlVdeyf/8z/8MbmPIKauqbyYuzpCWFI8rtFpbsz9IWa0ztGHNET+1m9v/zGeAtKR4PO54PO4EWgJB3t59jMIdFaw7cJyWgHOTTq4niRl5Hj6ybAKXzxnFWeOz+rQaXHem5XmYlheBu60HQvl2eObfWVp5AMY95IzNbeX3wbP3Oj2Eo+Y5IW2g5M5wwlnrONOcaR1v+IpzOb2U0y+HgN8JKj0NQYhPcv50mzen43a/r29BIzkLLvoynPP/Qn8ifRpW/dL5hQGcXtPFd8GcG3i7qJTly0PDNoxxnntib3NSmnOjztiF3b+urw78YXfOd3W+Vk21Tk9Tb8W5Tu4RlsEVF+f0+vbU8ytDV9pIp1OgN5KznF8wu/glM5qM7e6O1xi0aNEiu27dug7biouLmTVrVhfPGBh1dXV4PDEaBPppMNqvcCDHQg0RXp+fbz9fxGNrD7VtS010kZTgoqqhudub0Tszc5SHC2bkct7UXOaMSR++06UFA7Dq5/Daf0NSGg3WTUpjKSy8Ay77thMq/3orHHwXLvoqnPfZmBr3FjWNVc7wiYx8yF/c9qdk/dvtO7Vd/6j9+kft1zvGmPXW2pO6r9WDLBIF7+6p5HNPbuZwdSMfPWci+VnJ1DX58fr8NLYEGOlJYkyGs1rcge3vsXTJ4rbnBq3F6/PjbfJT1+THHwyyeNIIRmf0cSziUGEtHN3izGRz8B3nRrSrf8S61Zs4P/CmM+52zz+dMFx3FG78nTNMQBzJWTDvxmhXISISExSQRQZJdUMzO8u8vPjeEX7/zn4mZqfwxCeWsXBC92NHA6VxTB15imPphpPy7c5NYlufdm6MS0qH634JZ34IjCHoSoSLv+3c3PTMvzs3Pn3keRh3ds/nFhGRYWnIBGRr7fCcPqqfTrchNqeTxuYAL7x3hOc2H6b4SC3ldb62fbcvm8B9K2aSkjhk/gmerLneucO9trTj9uQsGLcUXP343iv3tIfi8iLAODfALbvHuSu+sxvWJpwDn1zrzCJxqjfviIjIsDIk3p3dbjeVlZVkZ2crJJ8Cay2VlZVt8yjLqSuvbeIP7+7HHe9idGYyYzLcJCXEsXLTYZ7eWEpdk5+J2SmcNy2XGaPSmJbnYdao9KG1clz4wgLh0z1VHQC6+AUsJQdmX+NMGdXVzA2tmhtCU12Fzr3vDTiy2dk3bims+J4zB6xnVM+1xid2Op2UiIhIuCERkPPz8ykpKaGiopuVbfqpqalpSAZJt9tNfn5+tMs4La3cfJivPrOVuqYWgifkwERXHCvmjeLmxeNZMmnE0PjFLRC2klLrZPzlxR1X8GpdWWzMAjjzlvbVpMID8PG9ziIFmx6FdQ87c//Ouc4Jy63z8dYecaYf2/o0lKylLWi7Ep1pvC77tjNTRIauXRERGXhDIiAnJCQwaVJkp5QpLCxsWxlPhreq+ma+8uxWXthyhPnjMvn+TWcyNjOZozVNHK5ppLqhhWWTs4fGLBKH1sKah5wwfGynMzwBOGkFr7aFFKb23EM7ap7zHJ83tHrc32Dd75wFLDLGOVOMHVoNWMibB+d/zlnxbeRsyJrUv6EZIiIivaB3GpEuWGv5x7aj/OS13Xh9/rbtVfXNNPkDfO7yGdx9/mTiXc50WBNzUpmYcwqrfvVXMADv/gy2POHMxrDkbkg8xdf3eZ2FHEbO7rhClN8Hhf/rLB2anAVjFzmLBbSuKJUzvedFDnqSlObMmjDvRmdJ0u0vOuOK645CwX3Owhq50/v3GiIiIn2ggCzSidLqRr7+7FZeLS5nRp6HhRPal2ROdMXxkXMmMntMevQKrNwDz9wDh1ZB9jR47Zvw7oNw7n/C2R93FlhoHQ9cd8RZoax1aV1XAux6BbY+5SyD7G90lixuHebgSoC//TuUb4MFH4bL/8dZgS2S3Bkw/2bnQ0REJMoUkEXCtASC/OGd/fzglZ1YC1++chZ3LJ/Y1ks84IIBKNvWcbWwOBdkT+m4bn2rxip470l45WtOkL3h1zDvA8443de/Ay9/GV79BgTDzxcPwfYecFyJzlCJlByYfwuMPtMJymt/4yywAZCWB7c87qz2JiIiMswoIIvgDKco3FHBt18oYk9FPRfOyOVb185l3Ih+DiPoTDDo9Pxufdq5Ea2+i5tLM8bByFlMq4+DQz9u7w0GmHIxXPszSB/jPB63GG5/Fvb/C4qfh6wJzvjgkbOdIFx9oP3GuvpKmHYpTDyvfTzvwo9AY7Wz7HD1QWeZ4c6mShMRERkGFJBl2NtZVse3XyjmzZ0VTMpJ5de3L+KSWSMHfuaJlkbnZrR3f+bMDRyfDNMvg5lXgTuz/Th/k7PgRWhas5GV+yB3GkwucELvqDOcrzurb+K5zseJRkxyPmas6Lq+5EynR1lERGSYU0CWYaklEOS14jIeWXOIt3ZV4EmK56tXzebDSyeQGD/Awyn8PtjwR3jr+04P8MTz4NJvwfQrerVgxduFhRQUFAxsTSIiItIlBWQZVo7WNPGnVfv569oSjnl9jM5w8x8XTeMj50xkRFfTstVXQlN1++M4lzMVmSuh43FlRc4sDLtfBX9z2PPLnWEU45c5Y4YnnTfg35eIiIgMHAVkGRZ2HK3joTf3snJzKYGg5aKZedyyZBwXTB+JK66boRT734Y/XRc2/29IXALkTHOmPEsf44Tiiu1g4pwgnD62/diRM2H+rTDlos6HRYiIiEhMUUCWIaWpJcAT60sorWqkrqmFuiY/R2uaWLP/OMkJLm5dMoGPLZ/E+Oxe3HzX3AAr7wXPaLjwy+3bA81wfI8zRrh0vXOj3bilcOX/OQtgpI2M2PcnIiIikaeALEPG27uP8eW/vcf+ygYSXAaPOwGPO550dwKfvXQ6ty2dcGqr2xX+j7Ms8u0rYfIFXR8XDHZcZENEREROawrIctqr9Pr4zovFPL2hlAnZKfz540s4d1pO/05aut5ZeGPhR7sPx6BwLCIiMsQoIMtpxVrLmn3H2XSomp1lXnaW1bGzrI6gtdx74VTuvWgq7gRX/17E3wzP3gtpo5zZJkRERGRYUUCW00bxkVq+/UIRb++uBCAvPYnpeR4+vHQCN509jul5naw8B10PgWishu3POzfYpeQ4N9ONnA27XnYW1LjlcWcJZBERERlWFJAl5h3z+vj+yzv569qDeNwJfP3q2Vy/YCyZKT2MJ66vhLd/BGt/C4mp7QE4czzse8sJxsEW5ya85nrw1bY/d95NWmZZRERkmFJAlph2tKaJK3/yFrWNLXzknIl86uJpPQfjhuPO+OHVv3SC75zrISHF6RXe8CdoqXemYVtyN8y9Acac5TyvttSZmaJqP5xxU8S/NxEREYlNCsgSs6y1fOGpLTQ0+1l577nMHpPe/ROaamDVL5xw7KuFOTdAwX2QO6P9mGAQvGWQlnfysIuMfOdDREREhjUFZIlZf117iDd2VvCNq2d3H459XljzK3j7J86KdzOvggu/BHlzTj42Lg7SR0esZhERETn9KSBLTCqpauDbLxSzbHI2ty+bePIBAT/sf8tZ2rlopROMp18BBV+EMfMHuVoREREZShSQJeYEg5bPP7kFay3fu/EM4sKXgm5phH9+GzY/Bg3HIDENZlzpjCfOXxS9okVERGTIUECWmPPn1Qd4Z08l/3vDPMaNCFsS2lsOj97sLOIx+xqYeyNMuxQSkqNXrIiIiAw5CsgSM2oaWvjJP3fxh3f2c8H0XD509rj2nWVF8MgHob4CPvgnmHV19AoVERGRIU0BWaLOHwjyyJqD/PCVndQ0tvDBs8fxhStmYkxoaMXuV+HxjzpzGX/sJRizIKr1ioiIyNCmgCxRtf1oLZ96dBM7yupYNjmbr151wowV3nJ49BbImQ63/BUyxkavWBERERkWFJAlKqy1PLrmEN98bhsedwK/+vBCLpud195r3GrzoxDwwY2/VTgWERGRQaGALBFhraWpJUhdUwt1Pj8VDUGq6pvxuONpbAnwxaff4/ktRzhvWg4/uGk+uZ6kzk4CG/4I45d1XOxDREREJIIUkGVAeX1+PvrwGjYdqsYftB13vvkKAK7QtG2fv2IGnzh/Ssdp3MIdeAcqd8N5n41kySIiIiIdKCDLgGldGnrDwSo+tnwS2WlJpLnj8STF8962IvInTaWuyU99s5/L54zirPFZ3Z9wwx8hKR1mXzs434CIiIgICsgygP747gFe2HKEz18xg3sKpnbYl1mzi4Llk3p/ssZqKHoG5t/izF4hIiIiMkjiol2ADA0bD1bx7ReKuHjmSD5x/pT+n/C9J8DfBGfd3v9ziYiIiJwCBWTpt6r6Zu59ZCN56W6+f9OZXY8p7i1rYcMfYNQZmvNYREREBp0CsvTLngovd/95PRV1Pn5+61lkpiT2/6RHNsHR99R7LCIiIlGhMcjSJwcq6/nxa7t4ZmMp7gQX/3vDPM7Izzz1E1kLL30BWhpg5CznY8vjEO+GeR8Y8LpFREREehLRgGyMuQL4MeACfmOtvf+E/eOBPwCZoWPus9a+GMmapP8e+Md2fvnGXuLjDB8/dxJ3XzCFnLRO5jHujR0vwZpfQVIGbPxT+/YzPgTJmQNSr4iIiMipiFhANsa4gAeBS4ESYK0xZqW1tijssK8Aj1trf2GMmQ28CEyMVE3Sf3srvDz4+h7eN280X796NiPT3X0/mbXwxnchayLcux4aq6Ci2Jn7ePqKAatZRERE5FREsgd5MbDbWrsXwBjzGHAtEB6QLZAe+joDOBzBemQAPLL6IPFxhq9fM5uRnn6EY4Ddrzrjja/5KbjiIS3X+Zh0/oDUKiIiItIXxlrb81F9ObExNwJXWGv/LfT4w8ASa+29YceMBl4GsoBU4BJr7fpOznUXcBdAXl7ewsceeywiNXfH6/WSlpY26K8bS5oDlv8sbGBOtot75p9aOD6p/axlwcYvkOQ7zuolv8DGJQxwtUOHrr3+Ufv1j9qv79R2/aP26x+1X+9ceOGF6621i07cHu2b9G4Gfm+t/b4xZhnwJ2PMXGttMPwga+1DwEMAixYtsgUFBYNeaGFhIdF43Vjy1PoS6ls28+mrF3HOlJxTeu5J7be3EN7YAe/7PhecfemA1jnU6NrrH7Vf/6j9+k5t1z9qv/5R+/VPJKd5KwXGhT3OD20L93HgcQBr7buAGzi15CWD5s+rDzA5N5Vlk7P7f7I3vgeeMbDgw/0/l4iIiMgAimRAXgtMM8ZMMsYkAh8CVp5wzEHgYgBjzCycgFwRwZqkj7YdrmHjwWpuXTIBY/q5EMj+t+HA27D8UxDfx9kvRERERCIkYgHZWusH7gX+ARTjzFaxzRjzLWPMNaHDPgvcaYzZDDwKfNRGalC09MtfVh/EnRDHjWfl9+9EwSAU/i+kjoSFHxmY4kREREQGUETHIIfmNH7xhG1fC/u6CFgeyRqk/+qaWnhmYylXnzGGjJR+3ExnLbzwn7D/Lbjy/yAheeCKFBERERkgWmpaevTMxlIamgPcunRC309iLbz0eVj/ezj3M3D2vw1YfSIiIiIDKdqzWMhp4NE1h5g7Np0z8zP6dgJrmbLnYShZCcvuhYu/Bv0dxywiIiISIQrI0q2DlQ0UHanlq1fNPrWb84JBqN4P5dthxwuMK1kJSz4Bl31b4VhERERimgKydOvloqMAXDY7r3dPsBb+djcUPwctDW2bS8ZeRf4V9ysci4iISMxTQJZuvVJUxsxRHsaNSOndE/b8E7b8Fea+HyYXQO4syJ3B7lUbyFc4FhERkdOAArJ0qaq+mbX7j/PJC6f2/knv/BTSRsF1v9AcxyIiInJa0iwW0qV/bi8naOHS3g6vOPoe7H0dltytcCwiIiKnLQVk6dIrRWWMSnczb2wvZ69490FISIVFd0S2MBEREZEIUkCWTjW1BHhzVwWXzB7Zu9krakrhvSfgrA9DclbkCxQRERGJEAVk6dQ7e47R0Bzgklm9HF6x5ldgg7D03yNbmIiIiEiEKSBLp14pKiMtKZ5lU7J7PthXB+t+D7OugayJkS5NREREJKI0i4WcJBi0vFpczgXTc0mKd518gLUQ9Lc/Xv8H8NXAOf9v8IoUERERiRAFZDnJppJqKup8nc9e0dwAv1sBRzZ13D7+HMhfNCj1iYiIiESSArKc5JWiMlxxhgtnjDx55+vfccLx8k9DUlpoo4HZ1w5ihSIiIiKRo4AsHRysbODpDSUsmTSCjJSEjjtL1sGqn8PCO+DSb0anQBEREZEIU0CWNtsO1/CRh9fiDwa5b8XMjjv9Pnj2k+AZDZd+KzoFioiIiAwCBWQB4N09ldz1x3WkueN57K5lTB3p6XjAm/8HFdvhlifAnR6dIkVEREQGgQKy8EpRGZ98ZAPjR6Twx48tZkxmcscDjr4H//oBnPFBmH5ZdIoUERERGSQKyMNcMGi576ktTM9L408fW0JWamLHA6yF5z7trI53xf1RqVFERERkMGmhkGHuvdIaKuub+bdzJ58cjgH2vQml6+DCL0PKiMEvUERERGSQKSAPc4U7KjAGzpuW0/kB7/wUUnPhzJsHtzARERGRKFFAHube2FnOGWMzyE5LOnlneTHsfgUW3wUJ7sEvTkRERCQKFJCHseqGZjYdquaCzhYEAXj3ZxCfDIs+PriFiYiIiESRAvIw9tauYwQtXDA99+SddWWw5XFYcCukZg9+cSIiIiJRooA8jBXuqCAzJYH54zJP3rnmIQi0wNJ7Br0uERERkWhSQB6mgkHLGzsrOG9aLq4403Fncz2s/Q3MfB9kT4lOgSIiIiJRooA8TBUdqeWY19f58IpNj0BTNZzzH4Nel4iIiEi0KSAPU2/srADg/OknTO/W0gTv/ATyz4bxS6JQmYiIiEh0aSW9YeqNHRXMGZPOSM8J07e981OoPgjX/DQ6hYmIiIhEmXqQh6HaphbWH6yiYMYJwytqSuCt78Osa2ByQVRqExEREYk2BeQhaP2B42wtrely/9u7jhEIWgpOnP/45a8AFi7/TmQLFBEREYlhCshD0H89sYVPPbYRa22n+9/YWYHHHc+C8Ond9r0F2/4G5/4nZI4fnEJFREREYpAC8hDj9fnZd6yePRX1vNdJL3KzP8grRWWcPz2XeFfoxx/ww0tfgIzxsPxTg1yxiIiISGxRQB5ith+pbfv66Q2lJ+3/+7ajVNY3c9Oice0b1z0M5ducoRUJyYNRpoiIiEjMUkAeYopCAfms8Zms3HyYlkCww/4/rzrA+BEpnDc1NL2btfD2j2HCuTDr6sEuV0RERCTmKCAPMUWHa8lKSeCegqkcr2/mjR0Vbft2ldWxZt9xblkynrjW1fOq9kFtCcy5Dozp/KQiIiIiw4gC8hBTdKSW2WPSuWBGLtmpiTy9saRt319WHyTRFccHFua3P+HAO87niecOcqUiIiIisUkBeQjxB4JsP1rH7NHpJLjiuPrMMbxaVE5NQwsNzX6eWl/ClfNGkZ2W1P6k/W9D8gjImRG9wkVERERiiALyELLvWD3N/iCzx6QD8P6z8mkOBHn+vcM8t/kwdT4/ty6d0PFJB96GCedAnC4FEREREdBS00NK6w16s0Y7AXnu2HSmjUzj6Q2lNPuDzMjzsGhCVvsTakqg+gAs+UQ0yhURERGJSeo2HEKKDteS6IpjSm4aAMYYrj9rLOsPVPFeaQ23LR2PCb8Rr2388fIoVCsiIiISmxSQh5CiI7VMH5VGgqv9x3rd/LEYAymJLq5bMLbjE/b/C5IyIG/uIFcqIiIiErs0xGKIsNZSdLiWi2eN7LB9TGYyty2ZwKgMNx53QscnHXgbxi+FONcgVioiIiIS2xSQh4iKOh+V9c3MDo0/Dvff13XSQ1xXBpW74azbB6E6ERERkdOHhlgMEdvCb9ArWQ9Ht3b/hANvO58naP5jERERkXAKyENE0eFQQB6TDn+7Gx65CVoau37CgXcgIRVGnzFIFYqIiIicHhSQh4iiI7WMG5FMelyzM3SithTW/qbrJxx4G8YvAVdC18eIiIiIDEMKyENE8eFaZ/xxWRFgITkL3vo+NNWcfHB9JZQXwQRN7yYiIiJyIgXkIaCh2c++ynpmj86AsvecjVf/BBqr4O2fnPyEg+86nxWQRURERE6igDwEbD9ah7U4S0wf3QpJ6TDrapj7flj1c6g72vEJB96GeDeMPSs6BYuIiIjEMAXkIaDtBr3RHijbCnlzwBi48MsQaIY3H3AO9DfD2t/C5scg/2yIT4pi1SIiIiKxSfMgDwFFR2pJd8czNiMJyrbBmTc7O7KnwFkfgfW/h8zxzk171QchfzGs+G5UaxYRERGJVRHtQTbGXGGM2WGM2W2Mua+LY24yxhQZY7YZYx6JZD1DVdHhWmaNTsdUH4BmL4wKWxjkgs+DKxFe+RqkZMOtT8HHX3Z6mUVERETkJD32IBtjrgZesNYGT+XExhgX8CBwKVACrDXGrLTWFoUdMw34IrDcWltljBnZ+dmkK8GgZVdZHR9YNK59cZC8ee0HeEbBzY86wyumXeoMvRARERGRLvWmB/mDwC5jzPeMMTNP4dyLgd3W2r3W2mbgMeDaE465E3jQWlsFYK0tP4XzC1Ba3Uh9c4AZo0Ljj00cjJzV8aDJBTD9MoVjERERkV4w1tqeDzImHbgZuAOwwO+AR621dd0850bgCmvtv4UefxhYYq29N+yYZ4CdwHLABXzDWvv3Ts51F3AXQF5e3sLHHnust9/fgPF6vaSlpQ366/ZkY7mfH2/w8ZUlbq4r+S4pDSWsWfLzaJd1klhtv9OB2q5/1H79o/brO7Vd/6j9+kft1zsXXnjhemvtohO39+omPWttrTHmSSAZ+DRwPfA5Y8xPrLU/7Udd8cA0oADIB940xsyz1laf8PoPAQ8BLFq0yBYUFPTjJfumsLCQaLxuT7a9vhvYwQdXnI/nl5+GyYtjss5Ybb/Tgdquf9R+/aP26zu1Xf+o/fpH7dc/PQ6xMMZcY4z5G1AIJACLrbUrgDOBz3bz1FJgXNjj/NC2cCXASmtti7V2H05v8rTely/bj9YxNjMZDw3ODBV5c3t+koiIiIh0qTdjkN8P/NBaO89a+0DrOGFrbQPw8W6etxaYZoyZZIxJBD4ErDzhmGdweo8xxuQA04G9p/QdDHM7j9Yxc5THmd4NYNS87p8gIiIiIt3qTUD+BrCm9YExJtkYMxHAWvtaV0+y1vqBe4F/AMXA49babcaYbxljrgkd9g+g0hhTBLwOfM5aW9mXb2Q4avYH2VPhdW7Qa5vBQj3IIiIiIv3RmzHITwDnhD0OhLad3dMTrbUvAi+esO1rYV9b4DOhDzlF+47V4w9aJyAffA+SsyB9TLTLEhERETmt9aYHOT40TRsAoa8TI1eS9Nb2o84S0209yHlzNZWbiIiISD/1JiBXhA2JwBhzLXAsciVJb+04Wkd8nGHyiGQoL9b4YxEREZEB0JshFp8A/mKM+RlggEPA7RGtSnplZ1kdk3NTSazZB/5GjT8WERERGQA9BmRr7R5gqTEmLfTYG/GqpFe2H61j/rhMKHvP2TBKAVlERESkv3q1UIgx5n3AHMBtQmNcrbXfimBd0gOvz09JVSMfOnucM/44Lh5yT2UlcBERERHpTG8WCvkl8EHg/+EMsfgAMCHCdUkPdpY5q3zPGJUOZVshZzrEJ0W5KhEREZHTX29u0jvHWns7UGWt/SawDGdBD4minUdDAXlkGpSuh9FnRrkiERERkaGhNwG5KfS5wRgzBmgBRkeuJOmN7UfrSEl0kc9RaKiEcUuiXZKIiIjIkNCbMcjPGWMygQeADYAFfh3JoqRnO8vqmJbnIa4ktMihArKIiIjIgOg2IBtj4oDXrLXVwFPGmOcBt7W2ZjCKk67tOFrHJbPy4NBqSErXDXoiIiIiA6TbIRbW2iDwYNhjn8Jx9FXU+aisb2b6KA8cWgP5iyCuN6NlRERERKQnvUlVrxlj3m+M1jCOFa0zWMwZAZQXaXiFiIiIyADqTUC+G3gC8Bljao0xdcaY2gjXJd3YHprBYlZwJ2Bh3OLoFiQiIiIyhPRmJT3PYBQivbfzaB3ZqYlkHFsPGBi7KNoliYiIiAwZPQZkY8z5nW231r458OVId6y1PLGuhGc3l3LOlBznBr28OeBOj3ZpIiIiIkNGb6Z5+1zY125gMbAeuCgiFUmn6ppa+MozW3l202GWT83m/hvmwIPrYN6N0S5NREREZEjpzRCLq8MfG2PGAT+KVEFysu1Ha/nEn9Zz8HgD/3XZdP69YCquimLw1UK+xh+LiIiIDKTe9CCfqASYNdCFSNe+80IxNY0tPHbXMhZPGuFsPLTa+awb9EREREQGVG/GIP8UZ/U8cGa9mI+zop4MkuP1zSwYn9UejsGZ/zglB0ZMjl5hIiIiIkNQb3qQ14V97Qcetda+HaF6pBMNzQFSk074UR1a7cx/rOmpRURERAZUbwLyk0CTtTYAYIxxGWNSrLUNkS1NWnl9ftKSXO0b6ivh+B446/boFSUiIiIyRPVqJT0gOexxMvBqZMqRztT7/KQmhv0uU7LG+awV9EREREQGXG8Cstta6219EPo6JXIlSbhg0J48xOLQaoiLhzHzo1aXiIiIyFDVm4Bcb4w5q/WBMWYh0Bi5kiRcfbMfgNTWIRbWwu7XYPSZkJDczTNFREREpC96Mwb508ATxpjDgAFGAR+MZFHSrt4XAGjvQd7zTzi6Ba76UfSKEhERERnCerNQyFpjzExgRmjTDmttS2TLklZen9ODnJYU7/Qev/E9SB8L82+JcmUiIiIiQ1OPQyyMMZ8EUq21W621W4E0Y8w9kS9NwLlBD3Bu0tv/FhxaBcs/DfFJ0S1MREREZIjqzRjkO6211a0PrLVVwJ0Rq0g6aB+DHO/0HqeN0vRuIiIiIhHUm4DsMqZ9NQpjjAtIjFxJEq51DPLIqg1OD/Ly/4AEd5SrEhERERm6enOT3t+BvxpjfhV6fDfwUuRKknCtQyxGb/6ps7T0wjuiXJGIiIjI0NabgPwF4C7gE6HHW3BmspBB4PX5OdPsJuXQG3DJNyFRU1CLiIiIRFKPQyystUFgNbAfWAxcBBRHtixpVe/z82/xL2LdWXD2x6NdjoiIiMiQ12UPsjFmOnBz6OMY8FcAa+2Fg1OagBOQx5tyyF8ESZ5olyMiIiIy5HU3xGI78BZwlbV2N4Ax5j8HpSpp4/UFyDQNGHdGtEsRERERGRa6G2JxA3AEeN0Y82tjzMU4K+nJIGpo9pNuGkABWURERGRQdBmQrbXPWGs/BMwEXsdZcnqkMeYXxpjLBqm+Yc/b1IKHegVkERERkUHSm5v06q21j1hrrwbygY04M1vIIGhp8hJPQAFZREREZJD0ZqGQNtbaKmvtQ9baiyNVkJygqdb5rIAsIiIiMihOKSDL4DO+GucLBWQRERGRQaGAHONczepBFhERERlMCsgxLr4tIGdGtQ4RERGR4UIBOcYl+uucL9SDLCIiIjIoFJBjWCBocQe8zgMFZBEREZFBoYAcw+qb/aTT4Dxwp0e3GBEREZFhQgE5htX7/KSbevxxbohPinY5IiIiIsOCAnIMq/c5Pcj+RE+0SxEREREZNhSQY5jXFyDd1BNI1PAKERERkcGigBzDWnuQg0m6QU9ERERksCggxzBnDHKDZrAQERERGUQKyDHMmcWiHqOALCIiIjJoFJBjmDMGuYG4lMxolyIiIiIybMRHuwDpWn1TC+k0YBWQRURERAZNRHuQjTFXGGN2GGN2G2Pu6+a49xtjrDFmUSTrOd34Gr0kmAAJqVnRLkVERERk2IhYQDbGuIAHgRXAbOBmY8zsTo7zAJ8CVkeqltNVoKEaQGOQRURERAZRJHuQFwO7rbV7rbXNwGPAtZ0c99/Ad4GmCNZyWrKN1c4XCsgiIiIigyaSAXkscCjscUloWxtjzFnAOGvtCxGs4/TVVON8VkAWERERGTRRu0nPGBMH/AD4aC+OvQu4CyAvL4/CwsKI1tYZr9c76K9bf6wUgPVFe6grcQ3qaw+0aLTfUKG26x+1X/+o/fpObdc/ar/+Ufv1TyQDcikwLuxxfmhbKw8wFyg0xgCMAlYaY66x1q4LP5G19iHgIYBFixbZgoKCCJbducLCQgb7dTdv/Bc0wcJzLoKcqYP62gMtGu03VKjt+kft1z9qv75T2/WP2q9/1H79E8khFmuBacaYScaYROBDwMrWndbaGmttjrV2orV2IrAKOCkcD2fxzXXOFxpiISIiIjJoIhaQrbV+4F7gH0Ax8Li1dpsx5lvGmGsi9bpDSaK/NSCnR7cQERERkWEkomOQrbUvAi+esO1rXRxbEMlaTkdJ/jqaTRKJ8UnRLkVERERk2NBS0zHMHfDii/dEuwwRERGRYUUBOUb5A0FSrZdmBWQRERGRQaWAHKPqmwOk04A/UQFZREREZDApIMeoep+fdNNAIFE36ImIiIgMJgXkGFXv85NOPcEkTfEmIiIiMpgUkGOUN9SDrDmQRURERAaXAnKMqm/yk04DRgFZREREZFApIMeoxoY6EkwAV0pmtEsRERERGVYUkGNUs7cKgPiUrChXIiIiIjK8KCDHqECjE5AT0xSQRURERAaTAnKMCjTUAJCkgCwiIiIyqBSQY5RtrAbUgywiIiIy2BSQY5RtcnqQjTszuoWIiIiIDDMKyDEqLhSQNQ+yiIiIyOBSQI5RrpZa5wu3lpoWERERGUwKyDEqvrkOH0kQnxTtUkRERESGFQXkGJXor6MhLjXaZYiIiIgMOwrIMSrJX0ejKy3aZYiIiIgMOwrIMcod8NKsgCwiIiIy6BSQY1RK0EtzgifaZYiIiIgMOwrIMSrV1uNP0AwWIiIiIoNNATkGtQSCpNGAP1EBWURERGSwKSDHoPqmFtKpx2qREBEREZFBp4Acg+rr60g0AS0SIiIiIhIFCsgxqMlbBYBxZ0a3EBEREZFhSAE5BvnqnIDsSsmMbiEiIiIiw5ACcgxqrncCcrwCsoiIiMigU0COQf5QQE5Iy4pyJSIiIiLDjwJyDAo0VAOQpIAsIiIiMugUkGOQbawBwO0ZEeVKRERERIYfBeRY1OQE5NR0BWQRERGRwaaAHIPimmtosgkkuVOiXYqIiIjIsKOAHINcvlrqTCrGmGiXIiIiIjLsKCDHoPiWOupNarTLEBERERmWFJBjULrvKHVxGdEuQ0RERGRYUkCONTUlTPIVsznp7GhXIiIiIjIsKSDHmJYtTwJQMfHKKFciIiIiMjzFR7sA6ahp4xNsC07mzDMWRLsUERERkWFJPcixpHIPnuNbedEuZ+nk7GhXIyIiIjIsKSDHkq1PEcRwNH8FKYnq3BcRERGJBqWwWGEtLZsfZ31wJmfMnhXtakRERESGLfUgx4qybSQc38VzgWUUzMiNdjUiIiIiw5YCcqzY+hQB4tiUdj5TctOiXY2IiIjIsKWAHAusxW59infsGZw5c6qWmBYRERGJIgXkWFC6HlN9gGdallIwXcMrRERERKJJATkWbH0Kv0nkn+ZszpmaE+1qRERERIY1BeRYULqeba4ZzJyQT1qSJhYRERERiSYF5BgQOH6A7U3Zmr1CREREJAYoIEeb34er/iglNocLFJBFREREok4BOdpqSgCoc49hRp4nysWIiIiIiAJytFUfACAxZ6KmdxMRERGJAQrI0VZ9EICE7AlRLkREREREIMIB2RhzhTFmhzFmtzHmvk72f8YYU2SM2WKMec0YM+xSYkvlflqsi7TccdEuRURERESIYEA2xriAB4EVwGzgZmPM7BMO2wgsstaeATwJfC9S9cSqpop9HLbZjMnS8tIiIiIisSCSPciLgd3W2r3W2mbgMeDa8AOsta9baxtCD1cB+RGsJyYFqw5SYnMZm5kc7VJEREREhMgG5LHAobDHJaFtXfk48FIE64lJCXUlTkDOUkAWERERiQUxsWybMeY2YBFwQRf77wLuAsjLy6OwsHDwigvxer0D/rom2MIFvnJKOZ/iDavYMYRnsYhE+w0Xarv+Ufv1j9qv79R2/aP26x+1X/9EMiCXAuF3nuWHtnVgjLkE+DJwgbXW19mJrLUPAQ8BLFq0yBYUFAx4sT0pLCxkwF+3cg+8Cd7ksVx04YUDe+4YE5H2GybUdv2j9usftV/fqe36R+3XP2q//onkEIu1wDRjzCRjTCLwIWBl+AHGmAXAr4BrrLXlEawlNoXmQPanawYLERERkVgRsYBsrfUD9wL/AIqBx62124wx3zLGXBM67AEgDXjCGLPJGLOyi9MNTaE5kF1Zw252OxEREZGYFdExyNbaF4EXT9j2tbCvL4nk68e6YNVBAtZFWs6wm7xDREREJGbFxE16w5Xv2D7KbTajNQeyiIiISMzQUtNRFDh+gBKby5hMd7RLEREREZEQBeQoctUeosTmkq85kEVERERihgJyH1lrqW5oZtvhGl4tKmP70dpTO4HfR3JTOSU2hzFaRU9EREQkZmgMch/87u19fO/vO2hsCbRtG5uZzNv3XdT7k9SUAFCVOIqURP0YRERERGKFklkfPLPpMKMy3Ny6ZDxjMpNZs+84v39nP8frmxmRmti7k4TmQG5O0wwWIiIiIrFEQyxOUVNLgKLDNVw2J49/O28yV84bzSWz8gAoPnIKwyxCcyCjOZBFREREYooC8ikqOlJLS8CyYFxW27ZZoz3OvsOnFpBbrIvUbPUgi4iIiMQSDbE4RRsPVpOOl2UN/4TXdkP5drLLi/hDcg7PHvm/Xp+nuXI/R2w2YzQHsoiIiEhMUUA+RZsOVfO/KY+S8eLrYFyQPRXcGVxg1/LHkmJgfq/O46905kAeqyneRERERGKKhlicoo0Hq1jo2gNTL4EvH4F718DNj2IxzK16DZ8/0PNJgLjag6FFQhSQRURERGKJAvIpqKjzcbyqiryWQ5C/GOKTnB3pYzievZCr4t5hd1ldzyfy+3A3ts6BrFX0RERERGKJAvIp2HSomlnmAAYLo8/ssC8w5wamxZVyeOf6nk8UmgP5aNxIclKTIlGqiIiIiPSRAvIp2HSoijNc+50HJwTk7LNvwm/jSN7xTM8nCs2B3JQ6lrg4M7BFioiIiEi/KCCfgo0Hq1meWgqpI8EzqsM+lyeXLYkLmFbxMljb/Yla50DOHB+hSkVERESkrxSQeyloLVtKaphr9ju9x+bknt89eZeTFziKLVnX/cmqD+LHRfIIzYEsIiIiEmsUkHvpsNfS4mtgpG8fjJnf6TGB6Vfis/HUb3i86xMF/AQPraPUZjNmhOZAFhEREYk1Csi9tKcmwAxziDgbOGn8caup48dSGJxPQvEzEOxkujdfHTx2M3H73+Cv/gJN8SYiIiISgxSQe2lPdZDFSaGxw10E5Jmj01kZOIekpnI48E7HndWH4LeXw+7X2LPkO/w8cB1jFZBFREREYo5W0uulvdUBPpZaCsEsyBjX6TFpSfHsylxOU+NDuP/1A6jY7uwItMDbP4KWRrjtSTZVTwc2KyCLiIiIxCAF5F7w+vyUei2z3Pu6vEGv1eTRubx64Dyu2vMK7Pln+46siXD7Shg5k9LXdgEwKkOLhIiIiIjEGgXkXthSUk08fnIbdsPoy7o9dtbodP6j6CNc+PmfkZoY1rzuTHA5j0urGsn1JOFOcEWwahERERHpC41B7oWNB6uZZkqIC7Z0Of641ewx6QRtHNtrkyA1p/0jFI5X761k5ebDzBmTPhili4iIiMgpUkDuhfLaJpa79zsPRs/v9thZoz0AFB2pPWnf+gPHueP3axmT6eaBG7sP2iIiIiISHRpi0QvfvHYuh8pK4JgHsiZ1e+zYzGTS3fEUnxCQNx+q5qMPryUv3c2jdy4l15MUyZJFREREpI8UkHsp3bvHGV4R132nuzGGWaPTeWHLESq9PkZnJJPrSeJXb+whKzWRR+5cwsh03ZwnIiIiEqsUkHsjGCDNuw9mX9Krw+86fzJ/ePcAeyvq+deuY9Q3Bxg3IplH7lzC6AxN7SYiIiISyxSQe+PYLlzB5h5v0Gt18aw8Lp6VB4C1ltomPymJLhJcGvItIiIiEusUkHvjyGbncy8DcjhjDBnJCQNckIiIiIhEiro0e+PIJgJxiZAzLdqViIiIiEiEqQe5N8YupCS/gglxWthDREREZKhTQO6NeTeyrzKHCdGuQ0REREQiTkMsRERERETCKCCLiIiIiIRRQBYRERERCaOALCIiIiISRgFZRERERCSMArKIiIiISBgFZBERERGRMArIIiIiIiJhFJBFRERERMIoIIuIiIiIhFFAFhEREREJo4AsIiIiIhLGWGujXcMpMcZUAAei8NI5wLEovO5QofbrO7Vd/6j9+kft13dqu/5R+/WP2q93Jlhrc0/ceNoF5Ggxxqyz1i6Kdh2nK7Vf36nt+kft1z9qv75T2/WP2q9/1H79oyEWIiIiIiJhFJBFRERERMIoIPfeQ9Eu4DSn9us7tV3/qP36R+3Xd2q7/lH79Y/arx80BllEREREJIx6kEVEREREwiggi4iIiIiEUUDuBWPMFcaYHcaY3caY+6JdTywzxowzxrxujCkyxmwzxnwqtH2EMeYVY8yu0OesaNcay4wxLmPMRmPM86HHk4wxq0PX4F+NMYnRrjFWGWMyjTFPGmO2G2OKjTHLdP31jjHmP0P/brcaYx41xrh17XXNGPOwMabcGLM1bFun15px/CTUjluMMWdFr/LY0EX7PRD6t7vFGPM3Y0xm2L4vhtpvhzHm8qgUHSM6a7uwfZ81xlhjTE7osa69PlBA7oExxgU8CKwAZgM3G2NmR7eqmOYHPmutnQ0sBT4Zaq/7gNestdOA10KPpWufAorDHn8X+KG1dipQBXw8KlWdHn4M/N1aOxM4E6cddf31wBgzFvgPYJG1di7gAj6Err3u/B644oRtXV1rK4BpoY+7gF8MUo2x7Pec3H6vAHOttWcAO4EvAoTeRz4EzAk95+eh9+fh6vec3HYYY8YBlwEHwzbr2usDBeSeLQZ2W2v3WmubgceAa6NcU8yy1h6x1m4IfV2HE07G4rTZH0KH/QG4LioFngaMMfnA+4DfhB4b4CLgydAhar8uGGMygPOB3wJYa5uttdXo+uuteCDZGBMPpABH0LXXJWvtm8DxEzZ3da1dC/zROlYBmcaY0YNSaIzqrP2stS9ba/2hh6uA/NDX1wKPWWt91tp9wG6c9+dhqYtrD+CHwOeB8BkYdO31gQJyz8YCh8Iel4S2SQ+MMROBBcBqIM9aeyS06yiQF626TgM/wvkPLhh6nA1Uh71p6Brs2iSgAvhdaIjKb4wxqej665G1thT4P5yepyNADbAeXXunqqtrTe8lp+5jwEuhr9V+PTDGXAuUWms3n7BLbdcHCsgSEcaYNOAp4NPW2trwfdaZW1DzC3bCGHMVUG6tXR/tWk5T8cBZwC+stQuAek4YTqHrr3OhsbLX4vySMQZIpZM/4Urv6VrrO2PMl3GG7P0l2rWcDowxKcCXgK9Fu5ahQgG5Z6XAuLDH+aFt0gVjTAJOOP6Ltfbp0Oay1j/phD6XR6u+GLccuMYYsx9nOM9FOGNqM0N/9gZdg90pAUqstatDj5/ECcy6/np2CbDPWlthrW0Bnsa5HnXtnZqurjW9l/SSMeajwFXArbZ9sQa1X/em4Pxyuzn0/pEPbDDGjEJt1ycKyD1bC0wL3cmdiHOTwMoo1xSzQuNlfwsUW2t/ELZrJfCR0NcfAZ4d7NpOB9baL1pr8621E3GutX9aa28FXgduDB2m9uuCtfYocMgYMyO06WKgCF1/vXEQWGqMSQn9O25tO117p6ara20lcHtoRoGlQE3YUAwJMcZcgTPE7BprbUPYrpXAh4wxScaYSTg3nK2JRo2xyFr7nrV2pLV2Yuj9owQ4K/R/oq69PtBKer1gjLkSZ1yoC3jYWvud6FYUu4wx5wJvAe/RPob2SzjjkB8HxgMHgJustZ3dYCAhxpgC4L+stVcZYybj9CiPADYCt1lrfVEsL2YZY+bj3OCYCOwF7sDpDND11wNjzDeBD+L8aXsj8G84YxV17XXCGPMoUADkAGXA14Fn6ORaC/3S8TOcYSsNwB3W2nVRKDtmdNF+XwSSgMrQYaustZ8IHf9lnHHJfpzhey+deM7horO2s9b+Nmz/fpwZaY7p2usbBWQRERERkTAaYiEiIiIiEkYBWUREREQkjAKyiIiIiEgYBWQRERERkTAKyCIiIiIiYRSQRURigDHGG/o80RhzywCf+0snPH5nIM8vIjLUKCCLiMSWicApBeSwle660iEgW2vPOcWaRESGFQVkEZHYcj9wnjFmkzHmP40xLmPMA8aYtcaYLcaYu8FZSMYY85YxZiXOincYY54xxqw3xmwzxtwV2nY/kBw6319C21p7q03o3FuNMe8ZYz4Ydu5CY8yTxpjtxpi/hBYbEBEZFnrqdRARkcF1H6EVFAFCQbfGWnu2MSYJeNsY83Lo2LOAudbafaHHHwut2pYMrDXGPGWtvc8Yc6+1dn4nr3UDMB84E2dFrrXGmDdD+xYAc4DDwNvAcuBfA/3NiojEIvUgi4jEtsuA240xm3CWbM8GpoX2rQkLxwD/YYzZDKwCxoUd15VzgUettQFrbRnwBnB22LlLrLVBYBPO0A8RkWFBPcgiIrHNAP/PWvuPDhuNKQDqT3h8CbDMWttgjCkE3P14XV/Y1wH0fiEiw4h6kEVEYksd4Al7/A/g340xCQDGmOnGmNROnpcBVIXC8Uxgadi+ltbnn+At4IOhcc65wPnAmgH5LkRETmPqERARiS1bgEBoqMTvgR/jDG/YELpRrgK4rpPn/R34hDGmGNiBM8yi1UPAFmPMBmvtrWHb/wYsAzYDFvi8tfZoKGCLiAxbxlob7RpERERERGKGhliIiIiIiIRRQBYRERERCaOALCIiIiISRgFZRERERCSMArKIiIiISBgFZBERERGRMArIIiIiIiJh/j8Pf34G51jjUgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(10, 5))\n",
    "ax.plot(cb.train_acc, label=\"train acc\")\n",
    "ax.plot(cb.test_acc, label=\"test acc\")\n",
    "ax.set_xlabel(\"Iteration\")\n",
    "ax.set_ylabel(\"Accuracy\")\n",
    "ax.grid(True)\n",
    "ax.legend()\n",
    "\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Эксперименты с числом слоев (1 балл)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ясно, что из-за случайного начального приближения с каждым запуском обучения мы будем получать различное качество. Попробуем обучить нашу нейросеть с разным числом слоев несколько раз.\n",
    "\n",
    "Заполните матрицы `accs_train` и `accs_test`. В позиции `[i, j]` должна стоять величина доли правильных ответов сети с $i+1$ полносвязными слоями при $j$-м запуске (все запуски идентичны)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:44:54.832074Z",
     "start_time": "2021-03-03T14:44:54.819071Z"
    }
   },
   "outputs": [],
   "source": [
    "accs_train = np.zeros((5, 5))\n",
    "accs_test = np.zeros((5, 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:45:56.444624Z",
     "start_time": "2021-03-03T14:44:54.836024Z"
    }
   },
   "outputs": [],
   "source": [
    "### your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Построим боксплоты полученного качества (горизонтальная линия в каждом столбце — среднее, прямоугольник показывает разброс)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:45:57.060827Z",
     "start_time": "2021-03-03T14:45:56.447467Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAI4CAYAAAB3OR9vAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA78ElEQVR4nO3de7geZX3v//eHAHKGQNgRCRJbaWuqVDGC5y6PBbWgoBUqIP7UeMJqLe3Gy99WpPVnFe32AFuLihihIuKJKgoWskD9qZzKQaAgUpUEFDUBXKAo8N1/PPPAsExWnpBMZmXl/bquda2Ze07fZ3IbP9y5n5lUFZIkSZIGNum7AEmSJGk6MSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWpBEk+VqSl/ddR1uS+UkqyabN+jqrMclHk/yvdXEuSdrQxOcgS5qpkky0VrcC7gLuadZfU1Wnrv+q1p0k84H/BjarqrsnbTsCeFVVPbWHuk4G/hr4bat5+6q6Z+VHSNL04giypBmrqrYZ/gA/Af6y1XZfOB6OwGqdem/7/o8ajv2zkDQdGJAlbXSSjCVZmuR/Jvkp8Mkks5N8JcnPk6xolue1jhlP8qpm+Ygk30ryvmbf/06y3xTXe1ySS5P8Kslnk5yW5J/a55q0fyV5ZLP8/CT/meT2JDcmOWaK64wneVWSRwEfBZ6UZCLJrUmekORnSWa19j8wyeWrONfJrRqH9+vvktyS5OYkr1j9nV695vN/O8n/TvJL4JgkxyQ5pbXP5Kkk40n+sTnuV0nOSTKn2bZFklOS/LL53BclmbsuapW08TAgS9pYPRTYEdgdWMTg78NPNusPB34NHD/F8fsA1wJzgPcCn0iSyTsl2Rz4EvDp5nqfAw5agzrvAA4HdgCeD7wuyQunOqCqrgFeC3ynGb3doaouAn4JPLe162HA4hHreCiwPbAr8ErghCSzp9j/9UmWJ7kkyeo+7z7ADcBc4F0j1vPXwCuA/wFsDhzVtL+8qXM3YCcG9+HXI55TkgADsqSN173AO6rqrqr6dVX9sqo+X1V3VtWvGAS1P5/i+B9X1ceaqQOfAnZhEPAmeyKwGfCBqvpdVZ0BXDRqkVU1XlVXVtW9VXUF8JnV1DWVTwGHAiTZEfgL4N9GPPZ3wLHNZzgLmAD+eBX7fgjYg0F4/V/AyUmeMsW5b6qqD1fV3VU1apj9ZFVd1+x/OvDYVp07AY+sqnuq6pKqun3Ec0oSYECWtPH6eVX9ZriSZKsk/5rkx0luBy4AdmhPSZjkp8OFqrqzWdxmJfs9DFhWD/xG9I9HLTLJPkmWNFM/bmMwIjpn1OMnOQX4yyRbA38FfLOqbh7x2F9O+iLgnaz881JVlzb/wXF3E6ZPBQ6c4tw3jlhD209by+1aPg2cDZyW5KYk702y2YM4v6SNmAFZ0sZq8iN8/o7BiOg+VbUd8PSm/femTayhm4FdJ02/eHhr+Q4GT9gYXCx56KTj/w04E9itqrZnMLd4lJp+7xFFVbUM+A6DsHoYgzC5PhRT1zy51gfcEwbTO0a70GCE+51VtQB4MvACBlNUJGlkBmRJGtiWwVzVW5vpB+9YR+f9DnA38DdJNktyILB3a/vlwJ8meWySLYBjVlLX8qr6TZK9Gcy9HcXPgHnNHOi2xcA/AI8BvrBmH2U0SV6cZJskmyR5LoNpHWeuwSkuA56e5OFJtgfeugbXfkaSxzQj/7czmHJx7xpcW5IMyJLU+ACwJfAL4LvA19fFSavqtwxGbI8AlgMvpRVMq+o64FjgP4AfAN+adIrXA8cm+RXwdgbzbUdxHnAV8NMkv2i1f5HBFxG/2Joasq69CVgG3AocB7y6qsZHPbiqvgF8FrgCuAT4yhpc+6HAGQzC8TXA+ay/kXJJM4QvCpGk9ax5kcbSqvp/e7r+Dxm8KOU/+ri+JE13jiBL0kakeeRaMRhhliSthG8skqSNRJJxYAFwWFU5L1eSVsEpFpIkSVKLUywkSZKklhkzxWLOnDk1f/78vsvo3R133MHWW2/ddxmaBuwLGrIvaMi+oCH7wsAll1zyi6raeXL7jAnI8+fP5+KLL+67jN6Nj48zNjbWdxmaBuwLGrIvaMi+oCH7wkCSlb7Z1CkWkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWrpLCAnOSnJLUm+v4rtSfKhJNcnuSLJXpO2b5dkaZLju6pRkiRJmqzLEeSTgX2n2L4fsEfzswj4yKTt/whc0EllkiRJ0ip0FpCr6gJg+RS7HAAsroHvAjsk2QUgyeOBucA5XdUnSZIkrcymPV57V+DG1vpSYNckPwPeDxwKPHuqEyRZxGD0mblz5zI+Pt5NpRuQiYkJ74MA+4LuZ1/QkH1BQ/aFqfUZkFfl9cBZVbU0yZQ7VtWJwIkACxcurLGxse6rm+bGx8fxPgjsC7qffUFD9gUN2Rem1mdAXgbs1lqf17Q9CXhaktcD2wCbJ5moqqN7qFGSJEkbmT4D8pnAkUlOA/YBbquqm4GXDXdIcgSw0HAsSZKk9aWzgJzkM8AYMCfJUuAdwGYAVfVR4CzgecD1wJ3AK7qqRZIkSRpVZwG5qg5ZzfYC3rCafU5m8Lg4SZIkab3wTXqSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJklo6C8hJTkpyS5Lvr2J7knwoyfVJrkiyV9P+2CTfSXJV0/7SrmqUJEmSJutyBPlkYN8ptu8H7NH8LAI+0rTfCRxeVX/aHP+BJDt0V6YkSZJ0v027OnFVXZBk/hS7HAAsrqoCvptkhyS7VNV1rXPclOQWYGfg1q5qlSRJkoY6C8gj2BW4sbW+tGm7ediQZG9gc+CHKztBkkUMRp+ZO3cu4+PjXdW6wZiYmPA+CLAv6H72BQ3ZFzRkX5hanwF5Skl2AT4NvLyq7l3ZPlV1InAiwMKFC2tsbGz9FThNjY+P430Q2Bd0P/uChuwLGrIvTK3Pp1gsA3Zrrc9r2kiyHfBV4G1V9d0eapMkSdJGqs+AfCZwePM0iycCt1XVzUk2B77IYH7yGT3WJ0mSpI1QZ1MsknwGGAPmJFkKvAPYDKCqPgqcBTwPuJ7Bkyte0Rz6V8DTgZ2SHNG0HVFVl3VVqyRJkjTU5VMsDlnN9gLesJL2U4BTuqpLkiRJmopv0pMkSZJaVhuQk1yS5A1JZq+PgiRJkqQ+jTKC/FLgYcBFSU5L8hdJ0nFdkiRJUi9WG5Cr6vqqehvwR8C/AScBP07yziQ7dl2gJEmStD6NNAc5yZ7A+4HjgM8DLwFuB87rrjRJkiRp/VvtUyySXALcCnwCOLqq7mo2fS/JUzqsTZIkSVrvRnnM20uq6oaVbaiqA9dxPZIkSVKvRpli8aokOwxXksxO8k/dlSRJkiT1Z5SAvF9V3TpcqaoVDN6AJ0mSJM04owTkWUkeMlxJsiXwkCn2lyRJkjZYo8xBPhU4N8knm/VXAJ/qriRJkiSpP6sNyFX1niRXAM9qmv6xqs7utixJkiSpH6OMIFNVXwO+1nEtkiRJUu9WOwc5yROTXJRkIslvk9yT5Pb1UZwkSZK0vo3yJb3jgUOAHwBbAq8CTuiyKEmSJKkvI71quqquB2ZV1T1V9Ulg327LkiRJkvoxyhzkO5NsDlyW5L3AzYwYrCVJkqQNzShB97BmvyOBO4DdgIO6LEqSJEnqy5QjyElmAf9fVb0M+A3wzvVSlSRJktSTKUeQq+oeYPdmioUkSZI0440yB/kG4NtJzmQwxQKAqvqXzqqSJEmSejJKQP5h87MJsG235UiSJEn9GuVV0847liRJ0kZjtQE5yRKgJrdX1TM7qUiSJEnq0ShTLI5qLW/B4BFvd3dTjiRJktSvUaZYXDKp6dtJLuyoHkmSJKlXo0yx2LG1ugnweGD7ziqSJEmSejTKFItLGMxBDoOpFf8NvLLLoiRJkqS+jDLF4hHroxBJkiRpOpjyTXoASd6QZIfW+uwkr++0KkmSJKknqw3IwKur6tbhSlWtAF7dWUWSJElSj0YJyLOSZLiSZBaweXclSZIkSf0ZJSB/HfhskmcleRbwmaZtSklOSnJLku+vYnuSfCjJ9UmuSLJXa9vLk/yg+Xn5qB9GkiRJWlujBOT/CZwHvK75ORf4hxGOOxnYd4rt+wF7ND+LgI/AfY+VewewD7A38I4ks0e4niRJkrTWRnnM25bAx6rqo3DfFIuHAHdOdVBVXZBk/hS7HAAsrqoCvptkhyS7AGPAN6pqeXO9bzAI2p8ZoVZJkiRprYwSkM8Fng1MNOtbAucAT17La+8K3NhaX9q0rar99yRZxGD0mblz5zI+Pr6WJa2dsfEDer0+DP7rgvF+awAYH/ty3yX0yr5wP/uCfWHIvmBfGLIv2BeGpmtfGCUgb1FVw3BMVU0k2arDmkZWVScCJwIsXLiwxsbG+i1o7LZ+rw+Mj4/T+32g+R/exsy+cJ/+K+iZfeE+/VfQM/vCffqvoGf2hfv0X8HKjTIH+Y5JX6B7PPDrdXDtZcBurfV5Tduq2iVJkqTOjRKQ3wx8Lsk3k3wL+Cxw5Dq49pnA4c3TLJ4I3FZVNwNnA89tXkgyG3hu0yZJkiR1bpRXTV+U5E+AP26arq2q363uuCSfYTByPifJUgZPptisOedHgbOA5wHXM/jC3yuabcuT/CNwUXOqY4df2JMkSZK6NsocZBiE4wXAFsBeSaiqxVMdUFWHrGZ7AW9YxbaTgJNGrE2SJElaZ1YbkJO8g8FI8AIGo777Ad8CpgzIkiRJ0oZolDnILwaeBfy0ql4B/BmwfadVSZIkST0ZJSD/uqruBe5Osh1wCw98yoQkSZI0Y4wyB/niJDsAHwMuYfDCkO90WZQkSZLUl1GeYvH6ZvGjSb4ObFdVV3RbliRJktSPUZ9iAUBV/aijOiRJkqRpYZQ5yJIkSdJGw4AsSZIktYw0xSLJLGBue/+q+klXRUmSJEl9GeVFIW9k8JronwH3Ns0F7NlhXZIkSVIvRhlBfhPwx1X1y66LkSRJkvo2yhzkG4Hbui5EkiRJmg5GGUG+ARhP8lXgrmFjVf1LZ1VJkiRJPRklIP+k+dm8+ZEkSZJmrFHepPfO9VGIJEmSNB2sMiAn+UBVvTnJvzN4asUDVNX+nVYmSZIk9WCqEeRPN7/ftz4KkSRJkqaDVQbkqrqk+X3++itHkiRJ6tcoLwrZA3g3sADYYtheVX/QYV2SJElSL0Z5DvIngY8AdwPPABYDp3RZlCRJktSXUQLyllV1LpCq+nFVHQM8v9uyJEmSpH6M8hzku5JsAvwgyZHAMmCbbsuSJEmS+jHKCPKbgK2AvwEeDxwKvLzLoiRJkqS+TDmCnGQW8NKqOgqYAF6xXqqSJEmSerLKEeQkm1bVPcBT12M9kiRJUq+mGkG+ENgL+M8kZwKfA+4YbqyqL3RcmyRJkrTejfIlvS2AXwLPZPDK6TS/DciSJEmacaYKyP8jyVuA73N/MB6qTquSJEmSejJVQJ7F4HFuWck2A7IkSZJmpKkC8s1Vdex6q0SSJEmaBqZ6DvLKRo4lSZKkGW2qgPystT15kn2TXJvk+iRHr2T77knOTXJFkvEk81rb3pvkqiTXJPlQEgO7JEmSOrfKgFxVy9fmxM1LRk4A9gMWAIckWTBpt/cBi6tqT+BY4N3NsU8GngLsCTwaeALw52tTjyRJkjSKUV41/WDtDVxfVTdU1W+B04ADJu2zADivWV7S2l4MHi+3OfAQYDPgZx3WKkmSJAHdBuRdgRtb60ubtrbLgQOb5RcB2ybZqaq+wyAw39z8nF1V13RYqyRJkgSM9qKQLh0FHJ/kCOACYBlwT5JHAo8ChnOSv5HkaVX1zfbBSRYBiwDmzp3L+Pj4+qp72pqYmPA+CLAv6H72BQ3ZFzRkX5halwF5GbBba31e03afqrqJZgQ5yTbAQVV1a5JXA9+tqolm29eAJwHfnHT8icCJAAsXLqyxsbFuPskGZHx8HO+DwL6g+9kXNGRf0JB9YWpdTrG4CNgjySOSbA4cDJzZ3iHJnCTDGt4KnNQs/wT48ySbJtmMwRf0nGIhSZKkznUWkKvqbuBI4GwG4fb0qroqybFJ9m92GwOuTXIdMBd4V9N+BvBD4EoG85Qvr6p/76pWSZIkaajTOchVdRZw1qS2t7eWz2AQhicfdw/wmi5rkyRJklamyykWkiRJ0gbHgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLZ0G5CT7Jrk2yfVJjl7J9t2TnJvkiiTjSea1tj08yTlJrklydZL5XdYqSZIkQYcBOcks4ARgP2ABcEiSBZN2ex+wuKr2BI4F3t3athg4rqoeBewN3NJVrZIkSdJQlyPIewPXV9UNVfVb4DTggEn7LADOa5aXDLc3QXrTqvoGQFVNVNWdHdYqSZIkAd0G5F2BG1vrS5u2tsuBA5vlFwHbJtkJ+CPg1iRfSPKfSY5rRqQlSZKkTm3a8/WPAo5PcgRwAbAMuIdBXU8DHgf8BPgscATwifbBSRYBiwDmzp3L+Pj4eip7+pqYmPA+CLAv6H72BQ3ZFzRkX5halwF5GbBba31e03afqrqJZgQ5yTbAQVV1a5KlwGVVdUOz7UvAE5kUkKvqROBEgIULF9bY2FgnH2RDMj4+jvdBYF/Q/ewLGrIvaMi+MLUup1hcBOyR5BFJNgcOBs5s75BkTpJhDW8FTmodu0OSnZv1ZwJXd1irJEmSBHQYkKvqbuBI4GzgGuD0qroqybFJ9m92GwOuTXIdMBd4V3PsPQymX5yb5EogwMe6qlWSJEka6nQOclWdBZw1qe3treUzgDNWcew3gD27rE+SJEmazDfpSZIkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS2pqr5rWCeS/Bz4cd91TANzgF/0XYSmBfuChuwLGrIvaMi+MLB7Ve08uXHGBGQNJLm4qhb2XYf6Z1/QkH1BQ/YFDdkXpuYUC0mSJKnFgCxJkiS1GJBnnhP7LkDThn1BQ/YFDdkXNGRfmIJzkCVJkqQWR5AlSZKkFgOyJEmS1GJAniGSnJTkliTf77sW9SvJbkmWJLk6yVVJ3tR3TepHki2SXJjk8qYvvLPvmtSfJLOS/GeSr/Rdi/qV5EdJrkxyWZKL+65nOnIO8gyR5OnABLC4qh7ddz3qT5JdgF2q6tIk2wKXAC+sqqt7Lk3rWZIAW1fVRJLNgG8Bb6qq7/ZcmnqQ5C3AQmC7qnpB3/WoP0l+BCysKl8UsgqOIM8QVXUBsLzvOtS/qrq5qi5tln8FXAPs2m9V6kMNTDSrmzU/jopshJLMA54PfLzvWqQNgQFZmsGSzAceB3yv51LUk+af1S8DbgG+UVX2hY3TB4B/AO7tuQ5NDwWck+SSJIv6LmY6MiBLM1SSbYDPA2+uqtv7rkf9qKp7quqxwDxg7yROwdrIJHkBcEtVXdJ3LZo2nlpVewH7AW9opmmqxYAszUDNfNPPA6dW1Rf6rkf9q6pbgSXAvj2XovXvKcD+zbzT04BnJjml35LUp6pa1vy+BfgisHe/FU0/BmRphmm+mPUJ4Jqq+pe+61F/kuycZIdmeUvgOcB/9VqU1ruqemtVzauq+cDBwHlVdWjPZaknSbZuvsBNkq2B5wI+AWsSA/IMkeQzwHeAP06yNMkr+65JvXkKcBiDUaLLmp/n9V2UerELsCTJFcBFDOYg+4gvaeM2F/hWksuBC4GvVtXXe65p2vExb5IkSVKLI8iSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAl6UFKUkne31o/Kskx6+jcJyd58bo412qu85Ik1yRZMql9fhKfjSppo2RAlqQH7y7gwCRz+i6kLcmma7D7K4FXV9UzuqpnZdawRklarwzIkvTg3Q2cCPzt5A2TR4CTTDS/x5Kcn+TLSW5I8s9JXpbkwiRXJvnD1mmeneTiJNcleUFz/KwkxyW5KMkVSV7TOu83k5wJXL2Seg5pzv/9JO9p2t4OPBX4RJLjVvUhm9Hkbya5tPl5ctO+OMkLW/udmuSAUWts3uj11SSXN3W9dNQbL0ld8r/gJWntnABckeS9a3DMnwGPApYDNwAfr6q9k7wJeCPw5ma/+cDewB8yeCPeI4HDgduq6glJHgJ8O8k5zf57AY+uqv9uXyzJw4D3AI8HVgDnJHlhVR2b5JnAUVV18RT13gI8p6p+k2QP4DPAQgavNP9b4EtJtgeeDLycwaj0amtMchBwU1U9v6lz+zW4h5LUGUeQJWktVNXtwGLgb9bgsIuq6uaqugv4ITAMj1cyCMVDp1fVvVX1AwZB+k+A5wKHJ7kM+B6wE7BHs/+Fk8Nx4wnAeFX9vKruBk4Fnr4G9W4GfCzJlcDngAUAVXU+sEeSnYFDgM835x+1xiuB5yR5T5KnVdVta1CTJHXGEWRJWnsfAC4FPtlqu5tmECLJJsDmrW13tZbvba3fywP/Xq5J1ykgwBur6uz2hiRjwB0PpvgR/C3wMwYj35sAv2ltWwwcChwMvGJYzig1VtV1SfYCngf8U5Jzq+rYjj6DJI3MEWRJWktVtRw4ncHUgqEfMZjSALA/g1HYNfWSJJs085L/ALgWOBt4XZLNAJL8UZKtV3OeC4E/TzInySwGo73nr0Ed2wM3V9W9wGHArNa2k2mmhFTVcO7zSDU2Uz/urKpTgOMYTL+QpN45gixJ68b7gSNb6x8DvpzkcuDrPLjR3Z8wCLfbAa9t5gB/nME0jEuTBPg58MKpTlJVNyc5GljCYHT3q1X15TWo4/8An09yOJM+S1X9LMk1wJda+49a42OA45LcC/wOeN0a1CRJnUnV5H/BkyRpNEm2YjCXeC/nEEuaKZxiIUl6UJI8G7gG+LDhWNJM4giyJEmS1OIIsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJK0gUlyRJJvtdYnkvzBOjr3VUnG1sW5JGlDZUCWtFFoQuTw594kv26tv+xBnG88yau6qHVNVdU2VXUDQJKTk/zTWpzrT6tq/MEcm+RHk+7rOQ+2Dknq06Z9FyBJ60NVbTNcTvIj4FVV9R/9VTRj/eWDua9JZlXVPV0UJElryhFkSRu1JJskOTrJD5P8MsnpSXZstm2R5JSm/dYkFyWZm+RdwNOA45uR0uNXce7Dkvy4Of5tzQjrs5ttDxjpTTKWZGlrfVjTr5JcneRFU3yGSvLIJIuAlwH/0NT170n+PsnnJ+3/oSQfXMW52jUe09yPxU0dVyVZOOq9nUrz+T+S5KwkdwDPmDwqv5KpJJXktUl+0Px5nJAkzbZHJjk/yW1JfpHks+uiTkkbJwOypI3dG4EXAn8OPAxYAZzQbHs5sD2wG7AT8Frg11X1NuCbwJHN9IYjJ580yQLgI8BhzXl3AuatQV0/ZBDCtwfeCZySZJepDqiqE4FTgfc2df0lcAqwb5Idmro2BQ4GFo9Yx/7AacAOwJnASv9joOXUJD9Pck6SP1vNvn8NvAvYFvjWavYdegHwBGBP4K+Av2ja/xE4B5jN4D5/eMTzSdLvMSBL2ti9FnhbVS2tqruAY4AXN0HydwyC7SOr6p6quqSqbh/xvC8GvlJVFzTn/V/AvaMWVVWfq6qbqureqvos8ANg7zX4XMPz3AxcALykadoX+EVVXTLiKb5VVWc10x8+DUwVel8GzAd2B5YAZw+D+Sp8uaq+3XzG34xYzz9X1a1V9ZPmGo9t2n/XXPdhVfWbqho1cEvS7zEgS9rY7Q58sfkn+1uBa4B7gLkMAuHZwGlJbkry3iSbjXjehwE3Dleq6g7gl6MWleTwJJe16no0MGfU4yf5FHBos3wog881qp+2lu8Etmj+4+H3NGH311V1Z1W9G7iVwSj4qtw4xbZR6xnOLf8HIMCFzVSQ/+dBnFuSAAOyJN0I7FdVO7R+tqiqZVX1u6p6Z1UtAJ7M4J/3D2+Oq9Wc92YGUzMASLIVg9HooTuArVrrD23tuzvwMeBIYKeq2gH4PoMAuDorq+tLwJ5JHt18hlNHOM+6UExd8+RaV3lPVnuhqp9W1aur6mHAa4D/k+SRI1cqSS0GZEkbu48C72pCKUl2TnJAs/yMJI9JMgu4ncE/4w+nSfwMmOrZw2cAL0jy1CSbA8fywL9zLwOel2THJA8F3tzatjWD8Pjzpo5XMBhBHsXv1dVMXzgD+DfgwmZ6wjqV5OFJnpJk8+bLjX/PYMT722twmsuAA5Ns1YTbV67B9V+SZDjHewWD+zfylBZJajMgS9rYfZDBl8/OSfIr4LvAPs22hzIIlrczmHpxPvdPT/ggg7nKK5J8aPJJq+oq4A0MQunNDELb0tYunwYuB37E4Mtln20dezXwfuA7DALvYxg9aH4CWNBMzfhSq/1TzXnWZHrFmtiWwZcSVwDLGMx13q+qRp5WAvxv4LcMPvOnWLOR7icA30syweDP803DZ0NL0ppK1er+lVCStC70+fzlJA8H/gt46Bp80VCSNkqOIEvSDJdkE+AtwGmGY0laPd+kJ0kzWJKtGUxZ+DGDaQ+SpNVwioUkSZLU4hQLSZIkqWXGTLGYM2dOzZ8/v+8yenfHHXew9dZb912GpgH7gobsCxqyL2jIvjBwySWX/KKqdp7cPmMC8vz587n44ov7LqN34+PjjI2N9V2GpgH7gobsCxqyL2jIvjCQ5Mcra3eKhSRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS2dBuQk+ya5Nsn1SY5eyfbdk5yb5Iok40nmtbY9PMk5Sa5JcnWS+V3WKkmSJEGHATnJLOAEYD9gAXBIkgWTdnsfsLiq9gSOBd7d2rYYOK6qHgXsDdzSVa2SJEnSUJcjyHsD11fVDVX1W+A04IBJ+ywAzmuWlwy3N0F606r6BkBVTVTVnR3WKkmSJAHdvihkV+DG1vpSYJ9J+1wOHAh8EHgRsG2SnYA/Am5N8gXgEcB/AEdX1T3tg5MsAhYBzJ07l/Hx8Q4+xoZlYmLC+yDAvjCTPOMZz+i7BJYsWdJ3CcK+oHXH/4+YWt9v0jsKOD7JEcAFwDLgHgZ1PQ14HPAT4LPAEcAn2gdX1YnAiQALFy4s3wjjm3F0P/vCzFFVa3V8krU+h6YH+4LWFf8/YmpdTrFYBuzWWp/XtN2nqm6qqgOr6nHA25q2WxmMNl/WTM+4G/gSsFeHtUqSJElAtwH5ImCPJI9IsjlwMHBme4ckc5IMa3grcFLr2B2S7NysPxO4usNaJUmSJKDDgNyM/B4JnA1cA5xeVVclOTbJ/s1uY8C1Sa4D5gLvao69h8H0i3OTXAkE+FhXtUqSJElDnc5BrqqzgLMmtb29tXwGcMYqjv0GsGeX9UmSJEmT+SY9SZIkqcWALEmSJLUYkCVJkqQWA7IkSZLU0veLQiRpRttxxx1ZsWJF32WQpNfrz549m+XLl/dagySNyoAsSR1asWJF728umw5vzOo7oEvSmnCKhSRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1+BxkaZqaDs+N7fv5vdJM4ktjBnxpjDYEBmRpmlrbcJrEgCtNI740ZqDvgC6NwikWkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktTic5ClLhyzfd8VUO/YblrUwTG39V2BJElrxIAsdSDvvN0XAtC8rOSYXkuQJGmNOcVCkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJklp8DrIkdWg6vLBlDGC81xIG92EjZ18YsC9oQ2BAlqQO+dKYAV8aY18Ysi9oQ9DpFIsk+ya5Nsn1SY5eyfbdk5yb5Iok40nmtbbdk+Sy5ufMLuuUJEmShjobQU4yCzgBeA6wFLgoyZlVdXVrt/cBi6vqU0meCbwbOKzZ9uuqemxX9UmSJEkr0+UI8t7A9VV1Q1X9FjgNOGDSPguA85rlJSvZLkmSJK1XXc5B3hW4sbW+FNhn0j6XAwcCHwReBGybZKeq+iWwRZKLgbuBf66qL02+QJJFwCKAuXPnMj4+vq4/wwZnYmLC+zBN9P3nMF36wnSooW993wP7wvTR9z2wL2houvSF6SpdfWEgyYuBfavqVc36YcA+VXVka5+HAccDjwAuAA4CHl1VtybZtaqWJfkDBqPMz6qqH67qegsXLqyLL764k8+yIZkOX8BQ8yUUv4wzLe5D36bDPbAvTA/T4R7YFzQ0HfrCdJDkkqpaOLm9yxHkZcBurfV5Tdt9quomBiPIJNkGOKiqbm22LWt+35BkHHgcsMqALEmSJK0LXc5BvgjYI8kjkmwOHAw84GkUSeYkGdbwVuCkpn12kocM9wGeArS/3CdJkiR1orOAXFV3A0cCZwPXAKdX1VVJjk2yf7PbGHBtkuuAucC7mvZHARcnuZzBl/f+edLTLyRJkqROdPqikKo6CzhrUtvbW8tnAGes5Lj/H3hMl7V1Yccdd2TFihV9l9G72bNns3z58r7L6F2Svkvo3ezZs/suQZKkNeab9NahFStW9P7Fg+kw6d5gSO/9AKZHX5AkaUPU6Zv0JEmSpA2NAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLU4nOQ16F6x3ZwzPa91jAGMN5rCYP7IEmStIEyIK9Lx9zWdwW+HEKSJGktOcVCkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLWsNiAn+cskBmlJkiRtFEYJvi8FfpDkvUn+pOuCJEmSpD6tNiBX1aHA44AfAicn+U6SRUm27bw6SZIkaT0baepEVd0OnAGcBuwCvAi4NMkbO6xNkiRJWu9GmYO8f5IvAuPAZsDeVbUf8GfA33VbniRJkrR+bTrCPgcB/7uqLmg3VtWdSV7ZTVmSJM08SfouoXezZ8/uuwRptUYJyMcANw9XkmwJzK2qH1XVuV0VJknSTFJVfZdAkmlRhzTdjTIH+XPAva31e5o2SZIkacYZJSBvWlW/Ha40y5t3V5IkSZLUn1EC8s+T7D9cSXIA8IvuSpIkSZL6M8oc5NcCpyY5HghwI3B4p1VJkiRJPRnlRSE/rKonAguAR1XVk6vq+lFOnmTfJNcmuT7J0SvZvnuSc5NckWQ8ybxJ27dLsrQJ55IkSVLnRhlBJsnzgT8Fthg+oqaqjl3NMbOAE4DnAEuBi5KcWVVXt3Z7H7C4qj6V5JnAu4HDWtv/EXjA4+UkSZKkLo3yopCPAi8F3shgisVLgN1HOPfewPVVdUPzxb7TgAMm7bMAOK9ZXtLenuTxwFzgnBGuJUmSJK0To4wgP7mq9kxyRVW9M8n7ga+NcNyuDOYrDy0F9pm0z+XAgcAHGby+etskOwErgPcDhwLPXtUFkiwCFgHMnTuX8fHxEcqa2SYmJrwPAuwL04kvh4Btt93W/rgOPOMZz1jrc6xtf1yyZMla16C1ty76wtqayX1hlID8m+b3nUkeBvwS2GUdXf8o4PgkRzCYSrGMwXOWXw+cVVVLp/ofclWdCJwIsHDhwhobG1tHZW24xsfH8T4I7AvTxXR4KYMvh5g51vbP0b8XZo617Qv+vTC1UQLyvyfZATgOuBQo4GMjHLcM2K21Pq9pu09V3cRgBJkk2wAHVdWtSZ4EPC3J64FtgM2TTFTV733RT5IkSVqXpgzISTYBzq2qW4HPJ/kKsEVV3TbCuS8C9kjyCAbB+GDgryedfw6wvKruBd4KnARQVS9r7XMEsNBwLEmSpPVhyi/pNcH1hNb6XSOGY6rqbuBI4GzgGuD0qroqybGtF4+MAdcmuY7BF/LeteYfQZIkSVp3RplicW6Sg4Av1BpOVqmqs4CzJrW9vbV8BnDGas5xMnDymlxXkiRJerBGedX0a4DPAXcluT3Jr5Lc3nFdkiRJUi9WO4JcVduuj0IkSZKk6WC1ATnJ01fWXlW+4U6SJEkzzihzkP++tbwFgzfkXQI8s5OKJEkPsC5eNLK25/B5qZI2JqNMsfjL9nqS3YAPdFWQJOmBfDmEJK1fo3xJb7KlwKPWdSGSJEnSdDDKHOQPM3h7HgwC9WMZvFFPkiRJmnFGmYN8cWv5buAzVfXtjuqRJEmSejVKQD4D+E1V3QOQZFaSrarqzm5LkyRJkta/UeYgnwts2VrfEviPbsqRJEmS+jVKQN6iqiaGK83yVt2VJEmSJPVnlIB8R5K9hitJHg/8uruSJEmSpP6MMgf5zcDnktwEBHgo8NIui5IkSZqpdtxxR1asWNF3GevkJURrY/bs2SxfvrzXGlZllBeFXJTkT4A/bpqurarfdVuWJEnSzLRixYre3045HV4g1HdAn8pqp1gkeQOwdVV9v6q+D2yT5PXdlyZJkiStf6PMQX51Vd06XKmqFcCrO6tIkiRJ6tEoAXlWWmPgSWYBm3dXkiRJktSfUb6k93Xgs0n+tVl/TdMmSZIkzTijBOT/ySAUv65Z/wbw8c4qkiRJkno0ylMs7gU+0vxIkiRJM9pqA3KSPYB3AwuALYbtVfUHHdYlSZIk9WKUL+l9ksHo8d3AM4DFwCldFiVJkiT1ZZSAvGVVnQukqn5cVccAz++2LEmSJKkfo3xJ764kmwA/SHIksAzYptuyJEmSpH6MMoL8JmAr4G+AxwOHAi/vsihJkiSpL6M8xeKiZnECeEW35UiSJEn9GmUEWZIkSdpoGJAlSZKkltUG5CRPGaVNkiRJmglGGUH+8IhtkiRJ0gZvlV/SS/Ik4MnAzkne0tq0HTCr68IkSZKkPkz1FIvNGTzveFNg21b77cCLuyxKkiRJ6ssqA3JVnQ+cn+TkqvoxQPPCkG2q6vZRTp5kX+CDDEacP15V/zxp++7AScDOwHLg0Kpa2rR/kcEUkM2AD1fVR9f400mSJElraJQ5yO9Osl2SrYHvA1cn+fvVHZRkFnACsB+wADgkyYJJu70PWFxVewLHAu9u2m8GnlRVjwX2AY5O8rBRPpAkSZK0NkYJyAuaEeMXAl8DHgEcNsJxewPXV9UNVfVb4DTggMnnBs5rlpcMt1fVb6vqrqb9ISPWKUmSJK21UYLnZkk2YxCQz6yq3wE1wnG7Aje21pc2bW2XAwc2yy8Ctk2yE0CS3ZJc0ZzjPVV10wjXlCRJktbKal81Dfwr8CMGYfaCZn7wSHOQR3AUcHySI4ALgGXAPQBVdSOwZzO14ktJzqiqn7UPTrIIWAQwd+5cxsfH11FZG66JiQnvgwD7gu5nX9CQfWH66PvPYbr0helQw8qkapTB4EkHJZtW1d2r2edJwDFV9RfN+lsBqurdq9h/G+C/qmreSradBJxVVWes6noLFy6siy++eA0+xcw0Pj7O2NhY32VoGrAvaMi+oCH7wvSQhAeTv9al6dAXpsN9SHJJVS2c3D7Km/TmJvlEkq816wuAl49wzYuAPZI8IsnmwMHAmZPOPad5MgbAWxk80YIk85Js2SzPBp4KXDvCNSVJkqS1Msoc5JOBs4HhUySuA968uoOaEeYjm2OvAU6vqquSHJtk/2a3MeDaJNcBc4F3Ne2PAr6X5HLgfOB9VXXlKB9IkiRJWhtTvUlvOI1iTlWd3poicXeSe0Y5eVWdBZw1qe3treUzgN+bNlFV3wD2HO0jSJIkSevOVCPIFza/72ieLFEASZ4I3NZ1YZIkSVIfpnqKRZrfb2Ewd/gPk3ybwVvvfNW0JEmSZqSpAvLOSd7SLH+RwVSJAHcBzwau6Lg2SZIkab2bKiDPArbh/pHkoa26K0eSJEnq11QB+eaqOna9VSJJkiRNA1N9SW/yyLEkSZI0400VkJ+13qqQJEmSpolVBuSqWr4+C5EkSZKmg1HepCdJkiRtNAzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWjoNyEn2TXJtkuuTHL2S7bsnOTfJFUnGk8xr2h+b5DtJrmq2vbTLOiVJkqShzgJyklnACcB+wALgkCQLJu32PmBxVe0JHAu8u2m/Ezi8qv4U2Bf4QJIduqpVkiRJGupyBHlv4PqquqGqfgucBhwwaZ8FwHnN8pLh9qq6rqp+0CzfBNwC7NxhrZIkSRIAm3Z47l2BG1vrS4F9Ju1zOXAg8EHgRcC2SXaqql8Od0iyN7A58MPJF0iyCFgEMHfuXMbHx9dl/RukiYkJ74MA+4LuZ1/QkH1h+uj7z2G69IXpUMPKdBmQR3EUcHySI4ALgGXAPcONSXYBPg28vKrunXxwVZ0InAiwcOHCGhsbWw8lT2/j4+N4HwT2Bd3PvqAh+8L00fefw3TpC9OhhpXpMiAvA3Zrrc9r2u7TTJ84ECDJNsBBVXVrs74d8FXgbVX13Q7rlCRJku7T5Rzki4A9kjwiyebAwcCZ7R2SzEkyrOGtwElN++bAFxl8ge+MDmuUJEmSHqCzgFxVdwNHAmcD1wCnV9VVSY5Nsn+z2xhwbZLrgLnAu5r2vwKeDhyR5LLm57Fd1SpJkiQNdToHuarOAs6a1Pb21vIZwO+NEFfVKcApXdYmSZIkrYxv0pMkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqWXTvguQJEnamNQ7toNjtu+1hjGA8V5LGNyHacqALEmStB7lnbdTVb3WMD4+ztjYWK81JKGO6bWEVXKKhSRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUsmnfBUiSJG1skvRdQu9mz57ddwmr1OkIcpJ9k1yb5PokR69k++5Jzk1yRZLxJPNa276e5NYkX+myRkmSpPWpqnr/WbJkSe81LF++vO8/ilXqLCAnmQWcAOwHLAAOSbJg0m7vAxZX1Z7AscC7W9uOAw7rqj5JkiRpZbocQd4buL6qbqiq3wKnAQdM2mcBcF6zvKS9varOBX7VYX2SJEnS7+lyDvKuwI2t9aXAPpP2uRw4EPgg8CJg2yQ7VdUvR7lAkkXAIoC5c+cyPj6+tjVv8CYmJrwPAuwLup99QUP2BQ3ZF6bW95f0jgKOT3IEcAGwDLhn1IOr6kTgRICFCxfW2NhYByVuWMbHx/E+COwLup99QUP2BQ3ZF6bWZUBeBuzWWp/XtN2nqm5iMIJMkm2Ag6rq1g5rkiRJkqbU5Rzki4A9kjwiyebAwcCZ7R2SzEkyrOGtwEkd1iNJkiStVmcBuaruBo4EzgauAU6vqquSHJtk/2a3MeDaJNcBc4F3DY9P8k3gc8CzkixN8hdd1SpJkiQNdToHuarOAs6a1Pb21vIZwBmrOPZpXdYmSZIkrUyqqu8a1okkPwd+3Hcd08Ac4Bd9F6Fpwb6gIfuChuwLGrIvDOxeVTtPbpwxAVkDSS6uqoV916H+2Rc0ZF/QkH1BQ/aFqXX6qmlJkiRpQ2NAliRJkloMyDPPiX0XoGnDvqAh+4KG7Asasi9MwTnIkiRJUosjyJIkSVKLAVmSJElqMSDPEElOSnJLku/3XYv6lWS3JEuSXJ3kqiRv6rsm9SPJFkkuTHJ50xfe2XdN6k+SWUn+M8lX+q5F/UryoyRXJrksycV91zMdOQd5hkjydGACWFxVj+67HvUnyS7ALlV1aZJtgUuAF1bV1T2XpvUsSYCtq2oiyWbAt4A3VdV3ey5NPUjyFmAhsF1VvaDvetSfJD8CFlaVLwpZBUeQZ4iqugBY3ncd6l9V3VxVlzbLvwKuAXbttyr1oQYmmtXNmh9HRTZCSeYBzwc+3nct0obAgCzNYEnmA48DvtdzKepJ88/qlwG3AN+oKvvCxukDwD8A9/Zch6aHAs5JckmSRX0XMx0ZkKUZKsk2wOeBN1fV7X3Xo35U1T1V9VhgHrB3EqdgbWSSvAC4paou6bsWTRtPraq9gP2ANzTTNNViQJZmoGa+6eeBU6vqC33Xo/5V1a3AEmDfnkvR+vcUYP9m3ulpwDOTnNJvSepTVS1rft8CfBHYu9+Kph8DsjTDNF/M+gRwTVX9S9/1qD9Jdk6yQ7O8JfAc4L96LUrrXVW9tarmVdV84GDgvKo6tOey1JMkWzdf4CbJ1sBzAZ+ANYkBeYZI8hngO8AfJ1ma5JV916TePAU4jMEo0WXNz/P6Lkq92AVYkuQK4CIGc5B9xJe0cZsLfCvJ5cCFwFer6us91zTt+Jg3SZIkqcURZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJL0ICWpJO9vrR+V5Jh1dO6Tk7x4XZxrNdd5SZJrkiyZ1D4/ic9GlbRRMiBL0oN3F3Bgkjl9F9KWZNM12P2VwKur6hld1bMya1ijJK1XBmRJevDuBk4E/nbyhskjwEkmmt9jSc5P8uUkNyT55yQvS3JhkiuT/GHrNM9OcnGS65K8oDl+VpLjklyU5Iokr2md95tJzgSuXkk9hzTn/36S9zRtbweeCnwiyXGr+pDNaPI3k1za/Dy5aV+c5IWt/U5NcsCoNTZv9Ppqksubul466o2XpC75X/CStHZOAK5I8t41OObPgEcBy4EbgI9X1d5J3gS8EXhzs998YG/gDxm8Ee+RwOHAbVX1hCQPAb6d5Jxm/72AR1fVf7cvluRhwHuAxwMrgHOSvLCqjk3yTOCoqrp4inpvAZ5TVb9JsgfwGWAhg1ea/y3wpSTbA08GXs5gVHq1NSY5CLipqp7f1Ln9GtxDSeqMI8iStBaq6nZgMfA3a3DYRVV1c1XdBfwQGIbHKxmE4qHTq+reqvoBgyD9J8BzgcOTXAZ8D9gJ2KPZ/8LJ4bjxBGC8qn5eVXcDpwJPX4N6NwM+luRK4HPAAoCqOh/YI8nOwCHA55vzj1rjlcBzkrwnydOq6rY1qEmSOuMIsiStvQ8AlwKfbLXdTTMIkWQTYPPWtrtay/e21u/lgX8v16TrFBDgjVV1dntDkjHgjgdT/Aj+FvgZg5HvTYDftLYtBg4FDgZeMSxnlBqr6rokewHPA/4pyblVdWxHn0GSRuYIsiStpapaDpzOYGrB0I8YTGkA2J/BKOyaekmSTZp5yX8AXAucDbwuyWYASf4oydarOc+FwJ8nmZNkFoPR3vPXoI7tgZur6l7gMGBWa9vJNFNCqmo493mkGpupH3dW1SnAcQymX0hS7xxBlqR14/3Aka31jwFfTnI58HUe3OjuTxiE2+2A1zZzgD/OYBrGpUkC/Bx44VQnqaqbkxwNLGEwuvvVqvryGtTxf4DPJzmcSZ+lqn6W5BrgS639R63xMcBxSe4Ffge8bg1qkqTOpGryv+BJkjSaJFsxmEu8l3OIJc0UTrGQJD0oSZ4NXAN82HAsaSZxBFmSJElqcQRZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKklv8Ls0+PNy/ZFp0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x576 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axes = plt.subplots(2, 1, figsize=(10, 8))\n",
    "\n",
    "axes[0].boxplot(accs_train.T, showfliers=False)\n",
    "axes[1].boxplot(accs_test.T, showfliers=False)\n",
    "\n",
    "axes[0].set_xlabel(\"Number of layers\")\n",
    "axes[1].set_xlabel(\"Number of layers\")\n",
    "\n",
    "axes[0].set_ylabel(\"Train accuracy\")\n",
    "axes[1].set_ylabel(\"Test accuracy\")\n",
    "\n",
    "axes[0].set_title(\"Train quality in 5 runs\")\n",
    "axes[1].set_title(\"Test quality in 5 runs\")\n",
    "\n",
    "axes[0].grid(True)\n",
    "axes[1].grid(True)\n",
    "\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Дайте развёрнутый ответ на вопросы (в этой же ячейке):\n",
    "* Как изменяются качество на обучении и контроле и устойчивость процесса обучения при увеличении числа слоев?\n",
    "* Можно ли сказать, что логистическая регрессия (линейная модель) дает качество хуже, чем нелинейная модель?\n",
    "\n",
    "__Ответы:__\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Эксперименты c различными инициализациями весов (1 балл)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Как уже было сказано, начальная инициализация весов нейронной сети может сильно влиять на процесс ее обучения и, как следствие, на ее качество.\n",
    "\n",
    "В этом пункте вам предлагается попробовать обучить несколько нейронных сетей с различными инициализациями слоев.\n",
    "\n",
    "Для этого необходимо реализовать функцию, инициализирующую веса линейных слоёв нашей нейронной сети. Добавьте в функционал данного метода возможность инициализировать его веса с помощью инициализации Kaiming (используется, если в нейронной сети в качестве функций активации используется ReLU) и инициализации Xavier (используется, если в нейронной сети в качестве функций активации используется Tanh или Sigmoid):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:45:57.075786Z",
     "start_time": "2021-03-03T14:45:57.063819Z"
    },
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "def initialize_network(network, initialization):\n",
    "    for layer in network:\n",
    "        if isinstance(layer, Dense):\n",
    "            input_units, output_units = layer.weights.shape\n",
    "            if initialization == 'Kaiming':\n",
    "                ### your code here\n",
    "            elif initialization == 'Xavier':\n",
    "                ### your code here\n",
    "            else:\n",
    "                # Initialize weights with small random numbers from normal distribution.\n",
    "                # In this case `initialization` represents a standard deviation\n",
    "                # for normal distribution.\n",
    "                layer.weights = np.random.randn(input_units, output_units) * initialization\n",
    "            layer.biases = np.zeros_like(layer.biases)\n",
    "            \n",
    "            layer.params = [layer.weights, layer.biases]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь попробуйте для каждой из 3 инициализаций обучить нейронную сеть несколько раз. Попробуйте проделать данную операцию при зафиксированном числе слоев равным 3, 4 и 5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 слоя\n",
    "\n",
    "Зафиксируйте в сети число слоев равное трем."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Заполните матрицы `accs_train` и `accs_test`. В позиции `[i, j]` должна стоять величина доли правильных ответов сети при $j$-м запуске (все запуски идентичны) с инициализацией Kaiming при $i = 3$, с инициализацией Xavier при $i = 4$ и с инициализацией из нормального распределения с фиксированными параметрами при $0 \\leqslant i \\leqslant 2$ (попробуйте здесь 3 разных параметра для стандартного отклонения для нормального распределения, например: `1e-3`, `1e-2`, `1e-1`). Заметьте, что при большом числе слоев слишком низкое стандартное отклонение может не давать нейронной сети нормально обучиться."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:45:57.091744Z",
     "start_time": "2021-03-03T14:45:57.079777Z"
    }
   },
   "outputs": [],
   "source": [
    "init_vars = [1e-3, 1e-2, 1e-1, 'Kaiming', 'Xavier']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:45:57.107701Z",
     "start_time": "2021-03-03T14:45:57.094737Z"
    }
   },
   "outputs": [],
   "source": [
    "accs_train = np.zeros((5, 5))\n",
    "accs_test = np.zeros((5, 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:27.904407Z",
     "start_time": "2021-03-03T14:45:57.111690Z"
    }
   },
   "outputs": [],
   "source": [
    "### your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Построим боксплоты полученного качества (горизонтальная линия в каждом столбце — среднее, прямоугольник показывает разброс)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:28.632502Z",
     "start_time": "2021-03-03T14:46:27.907401Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAFgCAYAAACmDI9oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAp/UlEQVR4nO3dfbxdVX3v+8/XREQlSAQahVDAIz01RQqagtbabqm9hdbypG1RfKBXTT2Kt768WPF4DyKVQpVeLwq1oiJypIJGq9STFhRY4kOrBE94Fg0UJYFWhUTcoDyE3/1jzi3T7U72Ctkrc5P9eb9e67XnHHPOMcdca2Ttb8Yea65UFZIkSZIaj+m7AZIkSdJsYkCWJEmSOgzIkiRJUocBWZIkSeowIEuSJEkdBmRJkiSpw4AsSY9iSY5N8pXO+niSp81Q3dcnGZuJuiTp0cSALGnOaUPkxOOhJD/prB/zCOobJHnNKNq6uapqh6q6BSDJuUnetQV1/VpVDR7JsUlunfS8XvJI2yFJW9v8vhsgSVtbVe0wsZzkVuA1VfXF/lq0zfqjR/K8JplXVRtG0SBJGoYjyJLUSvKYJCckuTnJnUk+meTJ7bbtk3y8LV+f5Moki5KcAjwfOLMdKT1zI3W/Isl32+Pf3o6wvrDd9nMjvUnGkqzprE+06cdJbkhy5CauoZI8Pcky4BjgL9t2/VOStyT59KT935fkjI3U1W3jSe3zcV7bjuuTLB32ud2U9vo/kGRFknuAF0welZ9iKkkleV2S77Svx1lJ0m57epIvJflRkh8muXAm2ilp7jAgS9LD3ggcAfwOsBuwDjir3fYq4EnAHsDOwOuAn1TV24EvA8e10xuOm1xpkiXAB4BXtPXuDCzejHbdTBPCnwS8E/h4kqdu6oCqOhs4H3h3264/Aj4OHJJkp7Zd84GjgfOGbMdhwAXATsBFwJT/Geg4P8kPklyS5Nen2fdlwCnAAuAr0+w74UXAbwD7AX8C/H5b/lfAJcBCmuf5/UPWJ0mAAVmSul4HvL2q1lTVfcBJwEvaIPkATbB9elVtqKqrquruIet9CfD5qrqirfd/AA8N26iq+lRV3V5VD1XVhcB3gAM347om6rkDuAL447boEOCHVXXVkFV8papWtNMf/iewqdB7DLAXsCdwOXDxRDDfiM9V1Vfba/zpkO05rarWV9X32nPs35Y/0J53t6r6aVUNG7glCTAgS1LXnsA/tn+yXw/cCGwAFtEEwouBC5LcnuTdSR47ZL27AbdNrFTVPcCdwzYqySuTrOq0a19gl2GPn+RjwMvb5ZfTXNew/qOzfC+wffufh1/Qht2fVNW9VXUqsJ5mFHxjbtvEtmHbMzG3/C+BAN9op4L8n4+gbklzmAFZkh52G3BoVe3UeWxfVWur6oGqemdVLQF+k+bP+69sj6tp6r2DZmoGAEmeQDMaPeEe4Amd9ad09t0T+BBwHLBzVe0EXEcTAKczVbs+C+yXZN/2Gs4fop6ZUGy6zZPbutHnZNoTVf1HVb22qnYD/hz4uyRPH7qlkuY8A7IkPezvgVPaUEqSXZMc3i6/IMkzk8wD7qb5M/7ENIn/BDZ17+HlwIuS/FaS7YCT+fn331XAHyR5cpKnAG/qbHsiTXj8QduOP6MZQR7GL7Srnb6wHPgH4Bvt9IQZleSXkzwvyXbthxvfQjPi/dXNqGYVcFSSJ7Th9tWbcf4/TjIxx3sdzfM39JQWSTIgS9LDzqD58NklSX4M/BtwULvtKTTB8m6aqRdf4uHpCWfQzFVel+R9kyutquuBN9CE0jtoQtuazi7/E7gauJXmw2UXdo69Afhb4F9pAu8zGT5ofgRY0k7N+Gyn/GNtPZszvWJzLKD5UOI6YC3NXOdDq2roaSXAe4H7aa75Y2zeSPdvAF9PMk7zev7FxL2hJWkYqZruL4OSpJnW5/2Xk/wy8C3gKZvxQUNJmjMcQZakOSTJY4A3AxcYjiVpan6TniTNEUmeSDNl4bs00x4kSVNwioUkSZLU4RQLSZIkqWNOTLHYZZddaq+99uq7Gb265557eOITn9h3M9Qj+4DAfqCG/UBgPwC46qqrflhVu04unxMBea+99mLlypV9N6NXg8GAsbGxvpuhHtkHBPYDNewHAvsBQJLvTlXuFAtJkiSpw4AsSZIkdRiQJUmSpA4DsiRJktRhQJYkSZI6DMiSJElShwFZkiRJ6jAgS5IkSR0GZEmSJKnDgCxJkiR1GJAlSZKkDgOyJEmS1DG/7wZIkqTNk6TvJlBVfTdBGhkDsiRJjzJbGk6TGHClTXCKhSRJktRhQJYkSZI6DMiSJElSx0gDcpJDktyUZHWSE6bYvmeSS5Nck2SQZHFb/oIkqzqPnyY5ot22d5Kvt3VemGS7UV6DJEmS5paRBeQk84CzgEOBJcBLkyyZtNvpwHlVtR9wMnAqQFVdXlX7V9X+wMHAvcAl7TF/A7y3qp4OrANePaprkCRJ0twzyhHkA4HVVXVLVd0PXAAcPmmfJcBl7fLlU2wHeAnwz1V1b5r72hwMLG+3fQw4YqYbLkmSpLlrlLd52x24rbO+Bjho0j5XA0cBZwBHAguS7FxVd3b2ORr4f9vlnYH1VfVgp87dpzp5kmXAMoBFixYxGAwe+ZVsA8bHx+f8czDX2QcE9gM9zH4g3w82ru/7IB8PnJnkWOAKYC2wYWJjkqcCzwQu3tyKq+ps4GyApUuX1tjY2Aw099FrMBgw15+Duc4+ILAf6GH2A/l+sHGjDMhrgT0664vbsp+pqttpRpBJsgPw4qpa39nlT4B/rKoH2vU7gZ2SzG9HkX+hTkmSJGlLjHIO8pXAPu1dJ7ajmSpxUXeHJLskmWjD24BzJtXxUuATEyvVfO3P5TTzkgFeBXxuBG2XJEnSHDWygNyO8B5HMz3iRuCTVXV9kpOTHNbuNgbclOTbwCLglInjk+xFMwL9pUlVvxV4c5LVNHOSPzKqa5AkSdLcM9I5yFW1AlgxqezEzvJyHr4jxeRjb2WKD+BV1S00d8iQJEmSZpzfpCdJkiR1GJAlSZKkDgOyJEmS1GFAliRJkjoMyJIkSVKHAVmSJEnqMCBLkiRJHQZkSZIkqcOALEmSJHUYkCVJkqQOA7IkSZLUYUCWJEmSOgzIkiRJUocBWZIkSeowIEuSJEkdBmRJkiSpw4AsSZIkdRiQJUmSpA4DsiRJktRhQJYkSZI6DMiSJElShwFZkiRJ6jAgS5IkSR0GZEmSJKnDgCxJkiR1GJAlSZKkDgOyJEmS1GFAliRJkjpGGpCTHJLkpiSrk5wwxfY9k1ya5JokgySLO9t+OcklSW5MckOSvdryc5P8e5JV7WP/UV6DJEmS5paRBeQk84CzgEOBJcBLkyyZtNvpwHlVtR9wMnBqZ9t5wHuq6hnAgcD3O9veUlX7t49Vo7oGSZIkzT2jHEE+EFhdVbdU1f3ABcDhk/ZZAlzWLl8+sb0N0vOr6gsAVTVeVfeOsK2SJEkSAPNHWPfuwG2d9TXAQZP2uRo4CjgDOBJYkGRn4FeA9Uk+A+wNfBE4oao2tMedkuRE4NK2/L7JJ0+yDFgGsGjRIgaDwUxd16PS+Pj4nH8O5jr7gMB+oIfZD+T7wcaNMiAP43jgzCTHAlcAa4ENNO16PnAA8D3gQuBY4CPA24D/ALYDzgbeSjM94+dU1dntdpYuXVpjY2MjvZDZbjAYMNefg7nOPiCwH+hh9gP5frBxo5xisRbYo7O+uC37maq6vaqOqqoDgLe3ZetpRptXtdMzHgQ+Czyr3X5HNe4DPkozlUOSJEmaEaMMyFcC+yTZO8l2wNHARd0dkuySZKINbwPO6Ry7U5Jd2/WDgRvaY57a/gxwBHDdCK9BkiRJc8zIplhU1YNJjgMuBuYB51TV9UlOBlZW1UXAGHBqkqKZYvGG9tgNSY4HLm2D8FXAh9qqz2+Dc4BVwOtGdQ2SJI3Ck5/8ZNatW9drG5pfr/1YuHAhd911V2/nl6Yz0jnIVbUCWDGp7MTO8nJg+UaO/QKw3xTlB89wMyVJ2qrWrVtHVfV2/r7nnvYZzqVh+E16kiRJUocBWZIkSeowIEuSJEkdBmRJkiSpw4AsSZIkdRiQJUmSpA4DsiRJktRhQJYkSZI6DMiSJElShwFZkiRJ6jAgS5IkSR0GZEmSJKnDgCxJkiR1GJAlSZKkjvl9N0DTS9J3EwCoqr6bIEnbhHrHjnDSk3o7/xjAoLfTN9cvzWIG5EeBmQimSQy4kjRL5J139/qePBgMGBsb6+38SaiTeju9NC2nWEiSJEkdBmRJkiSpw4AsSZIkdRiQJUmSpA4DsiRJktRhQJYkSZI6DMiSJElShwFZkiRJ6jAgS5IkSR0GZEmSJKnDgCxJkiR1GJAlSZKkjpEG5CSHJLkpyeokJ0yxfc8klya5JskgyeLOtl9OckmSG5PckGSvtnzvJF9v67wwyXajvAZJkiTNLSMLyEnmAWcBhwJLgJcmWTJpt9OB86pqP+Bk4NTOtvOA91TVM4ADge+35X8DvLeqng6sA149qmuQJEnS3DPKEeQDgdVVdUtV3Q9cABw+aZ8lwGXt8uUT29sgPb+qvgBQVeNVdW+SAAcDy9tjPgYcMcJrkCRJ0hwzf4R17w7c1llfAxw0aZ+rgaOAM4AjgQVJdgZ+BVif5DPA3sAXgROAhcD6qnqwU+fuU508yTJgGcCiRYsYDAYzcEmPbj4H/TrssMP48Y9/3HczerVgwQIuuuiivpvRq7HB5HGCrXx+gEGvTQBgMPa5vpvQuz7fk8fHx3v/ndD3+fvm74TZ/TshVTWaipOXAIdU1Wva9VcAB1XVcZ19dgPOpAnBVwAvBvYFXgh8BDgA+B5wIbAC+Bzwb+30CpLsAfxzVe27qbYsXbq0Vq5cObMX+CiThFG91hpO36/BYDBgbGyst/ND/8/BbND3c2A/mB36fg767gd9X/9sMBueA/sBJLmqqpZOLh/lFIu1wB6d9cVt2c9U1e1VdVRVHQC8vS1bTzMyvKqdnvEg8FngWcCdwE5J5m+sTkmSJGlLjDIgXwns0951YjvgaODnxtGT7JJkog1vA87pHLtTkl3b9YOBG6r5b8blwEva8lfRjCpLkiRJM2JkAbkd+T0OuBi4EfhkVV2f5OQkh7W7jQE3Jfk2sAg4pT12A3A8cGmSa4EAH2qPeSvw5iSrgZ1ppmJIkiRJM2KUH9KjqlbQzB3ulp3YWV7Ow3ekmHzsF4D9pii/heYOGZIkSdKM85v0JEmSpA4DsiRJktRhQJYkSZI6DMiSJElShwFZkiRJ6jAgS5IkSR0GZEmSJKnDgCxJkiR1GJAlSZKkDgOyJEmS1DHSr5qW9LB6x45w0pN6O/8YwKC30wPtcyCS9N2EXi1cuLDvJki96/t3AvT/e2E2/06YNiAn+SPgf1XVQ1uhPdI2K++8m6rq7fyDwYCxsbHezg9NMKyTem1C7/rsAzA7+oGk/n8nQP/vB7P5d8IwUyz+FPhOkncn+dVRN0iSJEnq07QBuapeDhwA3Aycm+RfkyxLsmDkrZMkSZK2sqE+pFdVdwPLgQuApwJHAt9M8sYRtk2SJEna6qYNyEkOS/KPNNO4HwscWFWHAr8O/N+jbZ4kSZK0dQ1zF4sXA++tqiu6hVV1b5JXj6ZZkiRJUj+GCcgnAXdMrCR5PLCoqm6tqktH1TBJkiSpD8PMQf4U0L3F24a2TJIkSdrmDBOQ51fV/RMr7fJ2o2uSJEmS1J9hAvIPkhw2sZLkcOCHo2uSJEmS1J9h5iC/Djg/yZlAgNuAV460VZIkSVJPpg3IVXUz8JwkO7Tr4yNvlSRJktSTYUaQSfKHwK8B2ycBoKpOHmG7JEmSpF4M80Uhfw/8KfBGmikWfwzsOeJ2SZIkSb0Y5kN6v1lVrwTWVdU7gecCvzLaZkmSJEn9GCYg/7T9eW+S3YAHgKeOrkmSJElSf4aZg/xPSXYC3gN8EyjgQ6Ns1LbmyU9+MuvWreu7GUzMH+/DwoULueuuu3o7vyTNNn2+J/dt4cKFfTdhVpjLfQBmdz/YZEBO8hjg0qpaD3w6yeeB7avqR8NUnuQQ4AxgHvDhqjpt0vY9gXOAXYG7gJdX1Zp22wbg2nbX71XVYW35ucDvABNtOLaqVg3Tnr6sW7eOquq1DYPBgLGxsd7OP9ffBCSpq+/fCUl6b8NcNxuef/vBxm0yIFfVQ0nOAg5o1+8D7hum4iTzgLOA3wPWAFcmuaiqbujsdjpwXlV9LMnBwKnAK9ptP6mq/TdS/Vuqavkw7ZAkSZI2xzBzkC9N8uJs/hDggcDqqrql/XrqC4DDJ+2zBLisXb58iu2SJEnSVjXMHOQ/B94MPJjkpzS3equq2nGa43an+da9CWuAgybtczVwFM00jCOBBUl2rqo7ae65vBJ4EDitqj7bOe6UJCcClwIntCPbPyfJMmAZwKJFixgMBkNc6uj0ff7x8fHe29D3+WeDPp+D2dAHwH7Qt9nSD9Q/+4HAfrAxGdXckyQvAQ6pqte0668ADqqq4zr77AacCewNXAG8GNi3qtYn2b2q1iZ5Gs0o8+9W1c1Jngr8B7AdcDZw83RfWrJ06dJauXLlCK5yOLNhjs9smIPc93PQt76fg777APT/HGh29AP1z3+LAvsBQJKrqmrp5PJpR5CT/PZU5VV1xTSHrgX26Kwvbsu6ddxOM4JM+1XWL24/EEhVrW1/3pJkQDMP+uaquqM9/L4kHwWOn+4aJEmSpGENM8XiLZ3l7WnmFl8FHDzNcVcC+yTZmyYYHw28rLtDkl2Au6rqIeBtNHe0IMlC4N6quq/d53nAu9ttT62qO9o50UcA1w1xDZIkSdJQpg3IVfVH3fUkewD/3xDHPZjkOOBimtu8nVNV1yc5GVhZVRcBY8CpSYpmisUb2sOfAXwwyUM0HyQ8rXP3i/OT7EozF3oV8Lrp2iJJkiQNa5gR5MnW0ATYaVXVCmDFpLITO8vLgV+4XVtVfQ145kbqnG7kWpIkSXrEhpmD/H6ab8+DZjR3f5pv1JMkSZK2OcOMIHdv//Ag8Imq+uqI2iNJkiT1apiAvBz4aVVtgOYb8pI8oaruHW3TJEmSpK1vqG/SAx7fWX888MXRNEeSJEnq1zABefuqGp9YaZefMLomSZIkSf0ZJiDfk+RZEytJng38ZHRNkiRJkvozzBzkNwGfSnI7zb2HnwL86Sgbta2pd+wIJz2p1zaMAQz6O3+9Y8f+Ti5J0jao+c60fuvYVr+qepgvCrkyya8C/7UtuqmqHhhts7YteefdvXegwWDA2NhYb+dPQp3U2+klSdrmbGm26DsbzGbTTrFI8gbgiVV1XVVdB+yQ5PWjb5okSZK09Q0zB/m1VbV+YqWq1gGvHVmLJEmSpB4NE5DnpTNBJck8YLvRNUmSJEnqzzAf0vsX4MIkH2zX/7wtkyRJkrY5wwTkt9KE4v/Wrn8B+PDIWiRJkiT1aJi7WDwEfKB9SJIkSdu0aQNykn2AU4ElwPYT5VX1tBG2S5IkSerFMB/S+yjN6PGDwAuA84CPj7JRkiRJUl+GCciPr6pLgVTVd6vqJOAPR9ssSZIkqR/DfEjvviSPAb6T5DhgLbDDaJslSZIk9WOYEeS/AJ4A/F/As4GXA68aZaMkSZKkvgxzF4sr28Vx4M9G2xxJkiSpX8OMIEuSJElzhgFZkiRJ6pg2ICd53jBlkiRJ0rZgmLtYvB941hBlkqaRpO8m9GrhwoV9N0HaJszEe8mW1lFVW9wGabbaaEBO8lzgN4Fdk7y5s2lHYN6oGyZta/r+ZZKk9zZImhlb+m95MBgwNjY2M42RtkGbGkHejuZ+x/OBBZ3yu4GXjLJRkiRJUl82GpCr6kvAl5KcW1XfBWi/MGSHqrp7azVQkiRJ2pqGuYvFqUl2TPJE4DrghiRvGXG7JEmSpF4ME5CXtCPGRwD/DOwNvGKYypMckuSmJKuTnDDF9j2TXJrkmiSDJIs72zYkWdU+LuqU753k622dFybZbpi2SJIkScMYJiA/NsljaQLyRVX1ADDtpwOSzAPOAg4FlgAvTbJk0m6nA+dV1X7AycCpnW0/qar928dhnfK/Ad5bVU8H1gGvHuIaJEmSpKEME5A/CNwKPBG4IsmeNB/Um86BwOqquqWq7gcuAA6ftM8S4LJ2+fIptv+cNPekORhY3hZ9jCa4S5IkSTNi2vsgV9X7gPd1ir6b5AVD1L07cFtnfQ1w0KR9rgaOAs4AjgQWJNm5qu4Etk+yEngQOK2qPgvsDKyvqgc7de4+1cmTLAOWASxatIjBYDBEk0en7/OPj4/33oa+zy9fA82O9wL1z34gsB9syrQBOcki4K+B3arq0HaaxHOBj8zA+Y8HzkxyLHAFsBbY0G7bs6rWJnkacFmSa4EfDVtxVZ0NnA2wdOnS6vt+j32ffzbc87Lv88vXQLPjvUD9sx8I7AebMswUi3OBi4Hd2vVvA28a4ri1wB6d9cVt2c9U1e1VdVRVHQC8vS1b3/5c2/68BRgABwB3Ajslmb+xOiVJkqQtsdGA3Amhu1TVJ4GHANrpDRs2dlzHlcA+7V0ntgOOBi7q7pBkl/beygBvA85pyxcmedzEPsDzgBuq+eqgy3n4i0peBXxuiLZIkiRJQ9nUCPI32p/3JNmZ9s4VSZ7DEFMd2iB9HM3o843AJ6vq+iQnJ5m4K8UYcFOSbwOLgFPa8mcAK5NcTROIT6uqG9ptbwXenGQ1zZzkmZjqIUmSJAGbnoOc9uebaUZ+/0uSrwK7MuRXTVfVCmDFpLITO8vLefiOFN19vgY8cyN13kJzhwxJkiRpxm0qIO+a5M3t8j/SBN0A9wEvBK4ZcdskSZKkrW5TAXkesAMPjyRPeMLomiNJkiT1a1MB+Y6qOnmrtUSSJEmaBTb1Ib3JI8eSJEnSNm9TAfl3t1orJEmSpFliowG5qu7amg2RJEmSZoNhvklPkiRJmjMMyJIkSVKHAVmSJEnqMCBLkiRJHQZkSZIkqcOALEmSJHUYkCVJkqQOA7IkSZLUYUCWJEmSOgzIkiRJUocBWZIkSeowIEuSJEkd8/tugKThJJkVdVTVFtchSdJs5giy9ChRVVv0uPzyy7e4DsOxJGkuMCBLkiRJHQZkSZIkqcOALEmSJHUYkCVJkqQOA7IkSZLUYUCWJEmSOgzIkiRJUocBWZIkSeoYaUBOckiSm5KsTnLCFNv3THJpkmuSDJIsnrR9xyRrkpzZKRu0da5qH780ymuQJEnS3DKygJxkHnAWcCiwBHhpkiWTdjsdOK+q9gNOBk6dtP2vgCumqP6Yqtq/fXx/hpsuSZKkOWyUI8gHAqur6paquh+4ADh80j5LgMva5cu725M8G1gEXDLCNkqSJEk/Z/4I694duK2zvgY4aNI+VwNHAWcARwILkuwMrAP+Fng58MIp6v5okg3Ap4F3VVVN3iHJMmAZwKJFixgMBlt0MVuq7/OPj4/33oa+zz/XzYY+oP7ZDwT2AzXsBxs3yoA8jOOBM5McSzOVYi2wAXg9sKKq1iSZfMwxVbU2yQKagPwK4LzJO1XV2cDZAEuXLq2xsbFRXcNQ+j7/YDDovQ19n3+umw19QP2zHwjsB2rYDzZulAF5LbBHZ31xW/YzVXU7zQgySXYAXlxV65M8F3h+ktcDOwDbJRmvqhOqam177I+T/APNVI5fCMiSJEnSIzHKgHwlsE+SvWmC8dHAy7o7JNkFuKuqHgLeBpwDUFXHdPY5FlhaVSckmQ/sVFU/TPJY4EXAF0d4DZIkSZpjRhaQq+rBJMcBFwPzgHOq6vokJwMrq+oiYAw4NUnRTLF4wzTVPg64uA3H82jC8YdGdQ0zaYqpInPKwoUL+26CJEnSUEY6B7mqVgArJpWd2FleDiyfpo5zgXPb5XuAZ890O0dtis8QbnVJZkU7JEmSZju/SU+SJEnqMCBLkiRJHQZkSZIkqcOALEmSJHUYkCVJkqQOA7IkSZLUYUCWJEmSOgzIkiRJUocBWZIkSeowIEuSJEkdBmRJkiSpw4AsSZIkdRiQJUmSpA4DsiRJktRhQJYkSZI6DMiSJElShwFZkiRJ6jAgS5IkSR0GZEmSJKnDgCxJkiR1GJAlSZKkDgOyJEmS1GFAliRJkjoMyJIkSVKHAVmSJEnqMCBLkiRJHQZkSZIkqcOALEmSJHWMNCAnOSTJTUlWJzlhiu17Jrk0yTVJBkkWT9q+Y5I1Sc7slD07ybVtne9LklFegyRJkuaWkQXkJPOAs4BDgSXAS5MsmbTb6cB5VbUfcDJw6qTtfwVcMansA8BrgX3axyEz3HRJkiTNYaMcQT4QWF1Vt1TV/cAFwOGT9lkCXNYuX97dnuTZwCLgkk7ZU4Edq+rfqqqA84AjRnYFkiRJmnPmj7Du3YHbOutrgIMm7XM1cBRwBnAksCDJzsA64G+BlwMvnFTnmkl17j7VyZMsA5YBLFq0iMFg8EivY5vhczC3jY+P2wdkPxBgP1DDfrBxowzIwzgeODPJsTRTKdYCG4DXAyuqas0jnWJcVWcDZwMsXbq0xsbGZqK9j2o+B3PbYDCwD8h+IMB+oIb9YONGGZDXAnt01he3ZT9TVbfTjCCTZAfgxVW1PslzgecneT2wA7BdknGakebFm6pTkiRJ2hKjDMhXAvsk2ZsmxB4NvKy7Q5JdgLuq6iHgbcA5AFV1TGefY4GlVXVCu353kucAXwdeCbx/hNcgSZKkOWZkH9KrqgeB44CLgRuBT1bV9UlOTnJYu9sYcFOSb9N8IO+UIap+PfBhYDVwM/DPM912SZIkzV0jnYNcVSuAFZPKTuwsLweWT1PHucC5nfWVwL4z2U5JkiRpgt+kJ0mSJHUYkCVJkqQOA7IkSZLUYUCWJEmSOgzIkiRJUocBWZIkSeowIEuSJEkdBmRJkiSpw4AsSZIkdRiQJUmSpA4DsiRJktRhQJYkSZI6DMiSJElShwFZkiRJ6jAgS5IkSR0GZEmSJKnDgCxJkiR1GJAlSZKkDgOyJEmS1GFAliRJkjoMyJIkSVKHAVmSJEnqMCBLkiRJHQZkSZIkqcOALEmSJHUYkCVJkqQOA7IkSZLUMb/vBmh6SWZFPVU1I+2QJEmazUY6gpzkkCQ3JVmd5IQptu+Z5NIk1yQZJFncKf9mklVJrk/yus4xg7bOVe3jl0Z5DbNBVW3x4/LLL9/iOiRJkuaCkY0gJ5kHnAX8HrAGuDLJRVV1Q2e304HzqupjSQ4GTgVeAdwBPLeq7kuyA3Bde+zt7XHHVNXKUbVdkiRJc9coR5APBFZX1S1VdT9wAXD4pH2WAJe1y5dPbK+q+6vqvrb8cSNupyRJkvQzo5yDvDtwW2d9DXDQpH2uBo4CzgCOBBYk2bmq7kyyB/C/gKcDb+mMHgN8NMkG4NPAu2qKv/8nWQYsA1i0aBGDwWBmrupRanx8fM4/B3OdfUBgP1DDfiCwH2xK3x/SOx44M8mxwBXAWmADQFXdBuyXZDfgs0mWV9V/0kyvWJtkAU1AfgVw3uSKq+ps4GyApUuX1tjY2Fa4nNlrMBgw15+Duc4+ILAfqGE/ENgPNmWUUxfWAnt01he3ZT9TVbdX1VFVdQDw9rZs/eR9gOuA57fra9ufPwb+gWYqhyRJkjQjRhmQrwT2SbJ3ku2Ao4GLujsk2SXJRBveBpzTli9O8vh2eSHwW8BNSeYn2aUtfyzwIprwLEmSJM2IkQXkqnoQOA64GLgR+GRVXZ/k5CSHtbuN0QTfbwOLgFPa8mcAX09yNfAl4PSqupbmA3sXJ7kGWEUzIv2hUV2DJEmS5p6RzkGuqhXAikllJ3aWlwPLpzjuC8B+U5TfAzx75lsqSZIkNbx9miRJktRhQJYkSZI6DMiSJElSR6b4jo1tTpIfAN/tux092wX4Yd+NUK/sAwL7gRr2A4H9AGDPqtp1cuGcCMiCJCuramnf7VB/7AMC+4Ea9gOB/WBTnGIhSZIkdRiQJUmSpA4D8txxdt8NUO/sAwL7gRr2A4H9YKOcgyxJkiR1OIIsSZIkdRiQJUmSpA4DsiRJjwJJxjvLf5Dk20n23MT+r0vyys08x9e2pI3aupLskeTfkzy5XV/Yru+1mfXslmT5SBr5KOUcZEmSHgWSjFfVDkl+F/gg8PtVdXPf7VK/kvwl8PSqWpbkg8CtVXXqiM85v6oeHOU5+uYI8jYqySFJbkqyOskJm7vfJsrPSfL9JNeN+ho0c2agP/i6b2M2o0/42s8iSX4b+BDwoolwnOS1Sa5McnWSTyd5Qlt+UpLj2+VBkvcmWZnkxiS/keQzSb6T5F2d+sfbn2PtMcuTfCvJ+UnSbvuDtuyqJO9L8vmt/Tzo57wXeE6SNwG/BZyeZIcklyb5ZpJrkxwOkOS0JG+YOHCijyTZa+LfeJJ5Sd7T9qlrkvx5Wz6W5MtJLgJu2OpXubVVlY9t7AHMA24GngZsB1wNLBl2v00dD/w28Czgur6v08fW6Q++7tveY9g+4Ws/ux7AA8BdwH6TynfuLL8LeGO7fBJwfLs8AP6mXf4L4HbgqcDjgDUTdQDj7c8x4EfAYprBtH+lCV/bA7cBe7f7fQL4fN/PzVx/AL8PFPB77fp8YMd2eRdgNRDgAOBLneNuAPYA9pr4Nw4sA/6fdvlxwEpg77ZP3DPx2m/rD0eQZ5kkv57kiiQ3JHkoSSU5eTOrORBYXVW3VNX9wAXA4Zux30aPr6oraN6gtRXMkv7g6z6LbOU+4Ws/uzwAfA149aTyfduRvWuBY4Bf28jxF7U/rwWur6o7quo+4BaakDTZN6pqTVU9BKyiCVG/CtxSVf/e7vOJR3oxmlGHAncA+7brAf46yTXAF4HdgUVV9b+BX2rnHP86sK6qbptU1/8BvDLJKuDrwM7APu22b3Re+23a/L4boIcl2R64EHhlVX0jyV/R/G/9HZ19vgwsmOLw46vqi+3y7jT/w5+wBjhoimM2tt+wx2uEZlF/0CzRQ5/Q7PIQ8CfApUn+e1X9dVt+LnBEVV2d5Fiakb6p3Nep575O+UNMnQe6+2zYyD7qWZL9gd8DngN8JckFNCPKuwLPrqoHktxK814B8CngJcBTaN5PfqFKmr9CXDzpPGM0I8hzgp19dnkh8M2q+ka7fg1wSLV/5wCoquf30jL1wf6gyewTc1xV3ZvkD4EvJ/nPqvoIzX+I7kjyWJoR5LUjbMJNwNOS7FVVtwJ/OsJzaRrtvPAPAG+qqu8leQ9wOvAN4PttOH4B0L3byYU089h3AX5nimovBv5bksva43+F0fapWcmAPLvsS/OnrwnPAr7Z3WHI0aG1/PyfyxYzdefe2H7DHq/Rmi39QbPH1u4TmoWq6q4khwBXJPkB8D9o/hT+g/bnVK//TJ37J0leD/xLknuAK0d1Lg3ltcD3quoL7frfAX9G8xXSR7fTblYC35o4oKquT7IAWFtVd0xR54dpptN8sw3gPwCOGNkVzFLe5m0WSfJa4OCqemn7P7Z/An6zqu7czHrmA98Gfpfml96VwMuq6vph9qMZIdjo8Wnur/j5qtoXjcxs6Q++7rPH1u4Tnf33wtderSQ7VNV4G57OAr5TVe/tu13STPJDerPLJ4Ad2lutnA28dHN/8QFUc2/C42j+THIj8MlOyFmRZLdN7TfN8Z+g+TTzf02yJsnkD4to5syK/tDu5+s+O2zVPtGu+9prste2H+C6HngSzT2ZpW2KI8iSJElShyPIkiRJUocBWZIkSeowIEuSJEkdBmRJkiSpw4AsSZIkdRiQJaknScaH2OfDSZa0y/990ravDXuOJLslWf4I2rhT+8UQE+uPqB5JejTxNm+S1JMk41W1w6j2f6THTDp+L/ySEElzjCPIktSzJGNJBkmWJ/lWkvPbbymjLV+a5DTg8UlWJTm/3TYxOrxDkkuTfDPJtUkOn+Ice7VfMDIxKr2qffwgyTs2UcdpwH9p933PpHq2T/LRdv//neQFbfmxST6T5F+SfCfJu0f+JErSDJrfdwMkSQAcAPwacDvwVeB5wFcmNlbVCUmOq6r9pzj2p8CRVXV3kl2Af0tyUW3kT4RV9RqAJHsC/wKcu7E6gBOAfSfO244oT3hDU109M8mvApe0X4ENsH97TfcBNyV5f1XdtpnPiST1whFkSZodvlFVa6rqIWAVsNdmHBvgr5NcA3wR2B1YtMkDku2BTwFvrKrvPpI6gN8CPg5QVd8CvgtMBORLq+pHVfVT4AZgz824HknqlSPIkjQ73NdZ3sDmvT8fA+wKPLuqHkhyK7D9NMf8PfCZqvriFtSxKVtyPZLUK0eQJenR44Ekj52i/EnA99tg+wKmGa1N8gZgQVWdNkQdPwYWbKSqL9MEa9qpFb8M3DT01UjSLGVAlqRHj7OBayY+pNdxPrA0ybXAK4FvTVPP8cAzOx/Ue93G6qiqO4GvJrkuyXsm1fN3wGPaYy4Ejq2q+5CkRzlv8yZJkiR1OIIsSZIkdRiQJUmSpA4DsiRJktRhQJYkSZI6DMiSJElShwFZkiRJ6jAgS5IkSR3/P/I4J1gMmbRlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "init_vars_for_plot = [(x if isinstance(x, str) else fr\"$\\sigma = {x}$\") for x in init_vars]\n",
    "\n",
    "fig, ax = plt.subplots(1, 1, figsize=(10, 5))\n",
    "\n",
    "ax.boxplot(accs_test.T, labels=init_vars_for_plot, showfliers=False)\n",
    "\n",
    "ax.set_title(\"Test quality in 5 runs\")\n",
    "ax.set_xlabel(\"Initialization\")\n",
    "ax.set_ylabel(\"Test accuracy\")\n",
    "ax.grid(True)\n",
    "\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4 слоя\n",
    "\n",
    "Выполните тут тот же код, что и в предыдущем пункте, но только уже с 4 слоями в сети."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:45:57.091744Z",
     "start_time": "2021-03-03T14:45:57.079777Z"
    }
   },
   "outputs": [],
   "source": [
    "init_vars = [5e-3, 1e-2, 1e-1, 'Kaiming', 'Xavier']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:45:57.107701Z",
     "start_time": "2021-03-03T14:45:57.094737Z"
    }
   },
   "outputs": [],
   "source": [
    "accs_train = np.zeros((5, 5))\n",
    "accs_test = np.zeros((5, 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:27.904407Z",
     "start_time": "2021-03-03T14:45:57.111690Z"
    }
   },
   "outputs": [],
   "source": [
    "### your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Построим боксплоты полученного качества (горизонтальная линия в каждом столбце — среднее, прямоугольник показывает разброс)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:28.632502Z",
     "start_time": "2021-03-03T14:46:27.907401Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAFgCAYAAACmDI9oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmcElEQVR4nO3de7hddX3n8fdHIqIkSCA0ykWCo21NkaKm2NbaOWA7hY7lpm2leKFjpU7FqY+DFcYZRSrFVudxtFBbrBapjIDxUupQQTBHvLRCcMJdMFKUBDpeCEJAEcJ3/tjryI/jyclOcnbWSc779Tz7Oev6W9+1zy/JJ2v/9lqpKiRJkiQNPK7vAiRJkqTZxIAsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiStB1LckKSLzbz65M8fYbavjHJ2Ey0JUnbEwOypDmnC5ETr0eS/KCZP34L2htP8gejqHVzVdX8qroNIMm5Sd6xFW39XFWNb8m+SW6f9L5etqV1SNK2Nq/vAiRpW6uq+RPTSW4H/qCqLu+voh3Wb23J+5pkp6raMIqCJGkYXkGWpE6SxyU5Jck3knwvyUVJ9ujW7ZLkI93ye5JcnWRxkjOAFwJndVdKz9pI269I8s1u/7d0V1h/rVv3mCu9ScaSrGnmJ2q6L8lNSY6Z5hwqyTOSnAgcD/xJV9c/JnlTko9P2v59Sd67kbbaGk/r3o/zujpuTLJs2Pd2Ot35vz/JJUnuBw6dfFV+iqEkleS1Sb7e/T7OTpJu3TOSfD7J95N8N8mFM1GnpLnDgCxJj3o9cDTw74G9gXXA2d26VwFPBvYD9gReC/ygqt4CfAE4qRvecNLkRpMsBd4PvKJrd09g382o6xsMQviTgbcDH0ny1Ol2qKpzgPOBv+jq+i3gI8DhSXbv6poHvAw4b8g6jgQuAHYHLgam/M9A4/wk30lyWZKf38S2vwecASwAvriJbSe8GPgF4CDgd4Df6Jb/KXAZsJDB+/yXQ7YnSYABWZJarwXeUlVrqupB4DTgpV2QfIhBsH1GVW2oqmuq6t4h230p8OmqurJr938AjwxbVFV9rKrurKpHqupC4OvAIZtxXhPt3AVcCfx2t+hw4LtVdc2QTXyxqi7phj/8PTBd6D0eWALsD6wALp0I5hvxD1X1pe4cfzhkPe+sqnuq6lvdMQ7ulj/UHXfvqvphVQ0buCUJMCBLUmt/4JPdR/b3ADcDG4DFDALhpcAFSe5M8hdJHj9ku3sDd0zMVNX9wPeGLSrJK5Osauo6EFg07P6TfBh4eTf9cgbnNax/a6YfAHbp/vPwE7qw+4OqeqCqzgTuYXAVfGPumGbdsPVMjC3/EyDAVd1QkP+0BW1LmsMMyJL0qDuAI6pq9+a1S1WtraqHqurtVbUU+GUGH++/stuvNtHuXQyGZgCQ5EkMrkZPuB94UjP/lGbb/YEPACcBe1bV7sANDALgpkxV16eAg5Ic2J3D+UO0MxOK6WueXOtG35NNHqjq36rqNVW1N/CHwF8lecbQlUqa8wzIkvSovwbO6EIpSfZKclQ3fWiSZyfZCbiXwcf4E8Mk/h8w3b2HlwMvTvIrSXYGTuexf/+uAn4zyR5JngK8oVm3K4Pw+J2ujt9ncAV5GD9RVzd8YTnwv4GruuEJMyrJ05K8IMnO3Zcb38TgiveXNqOZVcCxSZ7UhdtXb8bxfzvJxBjvdQzev6GHtEiSAVmSHvVeBl8+uyzJfcC/AM/v1j2FQbC8l8HQi8/z6PCE9zIYq7wuyfsmN1pVNwKvYxBK72IQ2tY0m/w9cC1wO4Mvl13Y7HsT8D+Bf2YQeJ/N8EHzg8DSbmjGp5rlH+7a2ZzhFZtjAYMvJa4D1jIY63xEVQ09rAR4D/AjBuf8YTbvSvcvAF9Jsp7B7/OPJ+4NLUnDSNWmPhmUJM20Pu+/nORpwNeAp2zGFw0lac7wCrIkzSFJHge8EbjAcCxJU/NJepI0RyTZlcGQhW8yGPYgSZqCQywkSZKkxkiHWCQ5PMktSVYnOWWK9fsnuSLJdd1jRfftlh/a3fNz4vXDJEd36w5I8pWuzQu7b4RLkiRJM2JkV5C7WyHdCvw6g29rXw0c130je2KbjzF4utSHkxwG/H5VvWJSO3sAq4F9q+qBJBcBn6iqC5L8NXBtVb1/uloWLVpUS5YsmcnT2+7cf//97Lrrrn2XoR7ZBwT2Aw3YDwT2A4Brrrnmu1W11+TloxyDfAiweuLWOkkuAI4Cbmq2WcrgyyIweEzop6Zo56XAP3XhOMBhwO916z7M4FGw0wbkJUuWsHLlyi07ix3E+Pg4Y2NjfZehHtkHBPYDDdgPBPYDgCTfnGr5KIdY7MNjHx26plvWuhY4tps+BliQZM9J27wM+Gg3vSdwT1U9PE2bkiRJ0hbr+y4WJwNnJTkBuJLBDeU3TKxM8lQGN7O/dHMbTnIicCLA4sWLGR8fn4Fyt1/r16+f8+/BXGcfENgPNGA/ENgPpjPKgLwW2K+Z37db9mNVdSfdFeQk84GXVNU9zSa/A3yyqh7q5r8H7J5kXncV+SfabNo+BzgHYNmyZTXXP0LwYxTZBwT2Aw3YDwT2g+mMcojF1cAzu7tO7MxgqMTF7QZJFnU3rQc4FfjQpDaO49HhFdTgG4UrGIxLBngV8A8jqF2SJElz1MgCcneF9yQGwyNuBi6qqhuTnJ7kyG6zMeCWJLcCi4EzJvZPsoTBFejPT2r6zcAbk6xmMCb5g6M6B0mSJM09Ix2DXFWXAJdMWvbWZno5sHwj+97OFF/A6+6KcciMFipJkiR1RvqgEEmSJGl7Y0CWJEmSGgZkSZIkqWFAliRJkhoGZEmSJKlhQJYkSZIafT9qWpKkuee0J/d6+DGA8V5LgNO+33MB0sYZkCVJ2tZ6Doc+YliankMsJEmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpMa8vguQJEnS5kvSdwlUVd8ljIQBWZIkaTu0teE0yQ4bcLeWQywkSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqjDQgJzk8yS1JVic5ZYr1+ye5Isl1ScaT7Nuse1qSy5LcnOSmJEu65ecm+dckq7rXwaM8B0mSJM0tIwvISXYCzgaOAJYCxyVZOmmzdwPnVdVBwOnAmc2684B3VdWzgEOAbzfr3lRVB3evVaM6B0mSJM09o7yCfAiwuqpuq6ofARcAR03aZinwuW56xcT6LkjPq6rPAlTV+qp6YIS1SpIkSQDMG2Hb+wB3NPNrgOdP2uZa4FjgvcAxwIIkewI/DdyT5BPAAcDlwClVtaHb74wkbwWu6JY/OPngSU4ETgRYvHgx4+PjM3Ve26X169fP+fdgrrMPCOwHGrAfaIL9YGqjDMjDOBk4K8kJwJXAWmADg7peCDwH+BZwIXAC8EHgVODfgJ2Bc4A3Mxie8RhVdU63nmXLltXY2NhIT2S2Gx8fZ66/B3OdfUBgP9CA/UAT7AdTG+UQi7XAfs38vt2yH6uqO6vq2Kp6DvCWbtk9DK42r+qGZzwMfAp4brf+rhp4EPg7BkM5JEmSpBkxyoB8NfDMJAck2Rl4GXBxu0GSRUkmajgV+FCz7+5J9urmDwNu6vZ5avczwNHADSM8B0mSJM0xIwvI3ZXfk4BLgZuBi6rqxiSnJzmy22wMuCXJrcBi4Ixu3w0Mhl9ckeR6IMAHun3O75ZdDywC3jGqc5AkSdLcM9IxyFV1CXDJpGVvbaaXA8s3su9ngYOmWH7YDJcpSduNwYdn/aqqvkuQtnt77LEH69at67uMXv9OWbhwIXfffXdvx59O31/SkyRthq0Np0kMuNIssG7dut7/LPb9Zc3Z8B/+jfFR05IkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSY6QBOcnhSW5JsjrJKVOs3z/JFUmuSzKeZN9m3dOSXJbk5iQ3JVnSLT8gyVe6Ni9MsvMoz0GSJElzy8gCcpKdgLOBI4ClwHFJlk7a7N3AeVV1EHA6cGaz7jzgXVX1LOAQ4Nvd8j8H3lNVzwDWAa8e1TlIkiRp7hnlFeRDgNVVdVtV/Qi4ADhq0jZLgc910ysm1ndBel5VfRagqtZX1QNJAhwGLO/2+TBw9AjPQZIkSXPMvBG2vQ9wRzO/Bnj+pG2uBY4F3gscAyxIsifw08A9ST4BHABcDpwCLATuqaqHmzb3mergSU4ETgRYvHgx4+PjM3BK26/169fP+fdge3fooYf2XQIAK1as6LuE7dqRRx7Jfffd12sNg2sN/VmwYAEXX3xxrzXMdf6bMDv0/TuYDf2g7+NvzCgD8jBOBs5KcgJwJbAW2MCgrhcCzwG+BVwInAD8w7ANV9U5wDkAy5Ytq7GxsRkse/szPj7OXH8PtndVtVX7J9nqNrT17rvvvl5/D7Ph74Ikvdcw182GfiB6/x3Mhn7Q9/E3ZpRDLNYC+zXz+3bLfqyq7qyqY6vqOcBbumX3MLgyvKobnvEw8CngucD3gN2TzNtYm5IkSdLWGGVAvhp4ZnfXiZ2BlwGP+UwtyaIkEzWcCnyo2Xf3JHt184cBN9XgsssK4KXd8lexGVeVJUmSpE0ZWUDurvyeBFwK3AxcVFU3Jjk9yZHdZmPALUluBRYDZ3T7bmAw/OKKJNcDAT7Q7fNm4I1JVgN7Ah8c1TlIkiRp7hnpGOSqugS4ZNKytzbTy3n0jhST9/0scNAUy29jcIcMSZIkacb5JD1JkiSpYUCWJEmSGgZkSZIkqWFAliRJkhoGZEmSJKlhQJYkSZIamwzISX6reZiHJEmStEMbJvj+LvD1JH+R5GdHXZAkSZLUp00G5Kp6OfAc4BvAuUn+OcmJSRaMvDpJkiRpGxtq6ERV3cvgiXcXAE8FjgG+muT1I6xNkiRJ2uaGGYN8ZJJPAuPA44FDquoI4OeB/zra8iRJkqRta94Q27wEeE9VXdkurKoHkrx6NGVJkiRJ/RgmIJ8G3DUxk+SJwOKqur2qrhhVYZIkSVIfhhmD/DHgkWZ+Q7dMkiRJ2uEME5DnVdWPJma66Z1HV5IkSZLUn2EC8neSHDkxk+Qo4LujK0mSJEnqzzBjkF8LnJ/kLCDAHcArR1qVJEnSDqzethuc9uReaxiDwT3KelJv262/g2/CJgNyVX0D+MUk87v59SOvSpIkaUd22vf7roAkVFXfZcxKw1xBJsl/BH4O2CUJAFV1+gjrkiRJknoxzINC/hr4XeD1DIZY/Daw/4jrkiRJknoxzJf0frmqXgmsq6q3A78E/PRoy5IkSZL6MUxA/mH384EkewMPAU8dXUmSJElSf4YZg/yPSXYH3gV8FSjgA6MsSpIkSerLtAE5yeOAK6rqHuDjST4N7FJV/X/1UpIkSRqBaYdYVNUjwNnN/IOGY0mSJO3IhhmDfEWSl2Ti/m6SJEnSDmyYgPyHwMeAB5Pcm+S+JPeOuC5JkiSpF8M8SW/BtihEkiRJmg02GZCT/OpUy6vqypkvR5IkSerXMLd5e1MzvQtwCHANcNhIKpIkSZJ6NMwQi99q55PsB/yvURUkSZIk9WmYL+lNtgZ41kwXIkmSJM0Gw4xB/ksGT8+DQaA+mMET9SRJkqQdzjBjkFc20w8DH62qL42oHkmSJKlXwwTk5cAPq2oDQJKdkjypqh4YbWmSJEnStjfUk/SAJzbzTwQuH005kiRJUr+GCci7VNX6iZlu+kmjK0mSJEnqzzAB+f4kz52YSfI84AejK0mSJEnqzzBjkN8AfCzJnUCApwC/O8qidjR77LEH69at67uMXi1cuJC777677zJ6NRv6QZJej28/kCRtD4Z5UMjVSX4W+Jlu0S1V9dBoy9qxrFu3jqra9IYjND4+ztjYWG/H7zuYzQZ994O++wDYDyRJ24dNDrFI8jpg16q6oapuAOYn+aNhGk9yeJJbkqxOcsoU6/dPckWS65KMJ9m3WbchyarudXGz/Nwk/9qsO3ioM5UkSZKGMMwY5NdU1T0TM1W1DnjNpnZKshNwNnAEsBQ4LsnSSZu9Gzivqg4CTgfObNb9oKoO7l5HTtrvTc26VUOcgyRJkjSUYQLyTmk+F+2C785D7HcIsLqqbquqHwEXAEdN2mYp8LluesUU6yVJkqRtapiA/BngwiQvSvIi4KPdsk3ZB7ijmV/TLWtdCxzbTR8DLEiyZze/S5KVSf4lydGT9jujG5bxniRPGKIWSZIkaSjD3MXizcAfAv+5m/8s8LczdPyTgbOSnABcCawFNnTr9q+qtUmeDnwuyfVV9Q3gVODfGFzFPqer7/TJDSc5ETgRYPHixYyPj89QyVum7+OvX7++9xr6Pv5s0Od7MBv6ANgPwH4A9oO+zZZ+oP7ZD6aWUX2rPskvAadV1W9086cCVNWZG9l+PvC1qtp3inXnAp+uquWTlo8BJ1fVi6erZdmyZbVy5cotOIuZkcS7WMyC96Bvfb8HffcB6P89mA36fg/sB4LZ0Q/UP/8sQpJrqmrZ5OXD3MXimUmWJ7kpyW0TryGOeTXwzCQHJNkZeBlwcbtBkkVJJmo4FfhQt3zhxNCJJIuAFwA3dfNP7X4GOBq4YYhaJEmSpKEMMwb574D3Aw8DhwLnAR/Z1E5V9TBwEnApcDNwUVXdmOT0JBN3pRgDbklyK7AYOKNb/ixgZZJrGXx5751VdVO37vwk1wPXA4uAdwxxDpIkSdJQhhmD/MSquiJJquqbwGlJrgHeuqkdq+oS4JJJy97aTC8Hlk+x35eBZ2+kzcOGqFmSJEnaIsME5Ae7YRBfT3ISgy/SzR9tWZIkSVI/hhli8cfAk4D/AjwPeDnwqlEWJUmSJPVlk1eQq+rqbnI98PujLUeSJEnq1zBXkCVJkqQ5w4AsSZIkNYa5D/ILhlkmSZIk7QiGuYL8l0MukyRJkrZ7G/2SXveo6F8G9kryxmbVbsBOoy5MkiRJ6sN0d7HYmcH9jucBC5rl9wIvHWVRkiRJUl82GpCr6vPA55Oc2z1Bj+6BIfOr6t5tVaAkSZK0LQ0zBvnMJLsl2RW4AbgpyZtGXJckSZLUi2EeNb20qu5NcjzwT8ApwDXAu0Za2Q6k3rYbnPbkXmsYAxjv7/j1tt36O/gs0Xc/GINe+wDYDyRJ24dhAvLjkzweOBo4q6oeSlKjLWvHkrffS1W/b9n4+DhjY2O9HT8JdVpvh58V+u4HffcBsB9IkrYPwwyx+BvgdmBX4Mok+zP4op4kSZK0w9nkFeSqeh/wvmbRN5McOrqSJEmSpP4M8yS9xUk+mOSfuvmlwKtGXpkkSZLUg2GGWJwLXArs3c3fCrxhRPVIkiRJvdpoQE4yMfxiUVVdBDwCUFUPAxu2QW2SJEnSNjfdFeSrup/3J9kTKIAkvwh8f9SFSZIkSX2Y7kt66X6+EbgY+HdJvgTshY+aliRJ0g5quoC8V5I3dtOfBC5hEJofBH4NuG7EtUmSJEnb3HQBeSdgPo9eSZ7wpNGVI0mSJPVruoB8V1Wdvs0qkSRJkmaB6b6kN/nKsSRJkrTDmy4gv2ibVSFJkiTNEhsNyFV197YsRJIkSZoNhnmSniRJkjRnGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqTPeoaUnSDKu37QanPbm3448BjPd2eKB7DyRpFjMgS9I2lLffS1X1dvzx8XHGxsZ6Oz5AEuq0XkuQpGk5xEKSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqTGSANyksOT3JJkdZJTpli/f5IrklyXZDzJvs26DUlWda+Lm+UHJPlK1+aFSXYe5TlIkiRpbhlZQE6yE3A2cASwFDguydJJm70bOK+qDgJOB85s1v2gqg7uXkc2y/8ceE9VPQNYB7x6VOcgSZKkuWeUV5APAVZX1W1V9SPgAuCoSdssBT7XTa+YYv1jJAlwGLC8W/Rh4OiZKliSJEka5ZP09gHuaObXAM+ftM21wLHAe4FjgAVJ9qyq7wG7JFkJPAy8s6o+BewJ3FNVDzdt7jPVwZOcCJwIsHjxYsbHx2finLZY38dfv3597zX0ffzZoM/3YDb0AbAfgP0A7Ad9my39QP2zH0wto3rkaZKXAodX1R90868Anl9VJzXb7A2cBRwAXAm8BDiwqu5Jsk9VrU3ydAZXmV8EfB/4l254BUn2A/6pqg6crpZly5bVypUrZ/4kh5Sk10fLQv+Pl50N70Hf+n4P+u4D0P97MBv0/R7YDwSzox+of/5ZhCTXVNWyyctHeQV5LbBfM79vt+zHqupOBleQSTIfeElV3dOtW9v9vC3JOPAc4OPA7knmdVeRf6JNSZIkaWuMcgzy1cAzu7tO7Ay8DLi43SDJoiQTNZwKfKhbvjDJEya2AV4A3FSD/+asAF7a7fMq4B9GeA6SJEmaY0YWkLsrvCcBlwI3AxdV1Y1JTk8ycVeKMeCWJLcCi4EzuuXPAlYmuZZBIH5nVd3UrXsz8MYkqxmMSf7gqM5BkiRJc88oh1hQVZcAl0xa9tZmejmP3pGi3ebLwLM30uZtDO6QIUmSJM04n6QnSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1RnoXCz0qSd8l9GrhwoV9lyBJ0g5lJrLF1raxoz6Jz4C8DcyGzuPjJCVJ2rFs7b/rPnJ84xxiIUmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVJjXt8FSHNJkr5L6NXChQv7LmFWsB/YDyTNbgZkaRupql6Pn6T3GtR/PxgfH2dsbKzXGiRptnOIhSRJktQwIEuSJEkNA7IkSZLUMCBLkiRJDQOyJEmS1DAgS5IkSQ0DsiRJktQwIEuSJEkNA7IkSZLUMCBLkiRJDQOyJEmS1BhpQE5yeJJbkqxOcsoU6/dPckWS65KMJ9l30vrdkqxJclazbLxrc1X3+qlRnoMkSZLmlpEF5CQ7AWcDRwBLgeOSLJ202buB86rqIOB04MxJ6/8UuHKK5o+vqoO717dnuHRJkiTNYaO8gnwIsLqqbquqHwEXAEdN2mYp8LluekW7PsnzgMXAZSOsUZIkSXqMeSNsex/gjmZ+DfD8SdtcCxwLvBc4BliQZE9gHfA/gZcDvzZF23+XZAPwceAdVVWTN0hyInAiwOLFixkfH9+qk9kR+B7IPqD169fbD2Q/EGA/mM4oA/IwTgbOSnICg6EUa4ENwB8Bl1TVmiST9zm+qtYmWcAgIL8COG/yRlV1DnAOwLJly2psbGxU57Dd8D2QfUDj4+P2A9kPBNgPpjPKgLwW2K+Z37db9mNVdSeDK8gkmQ+8pKruSfJLwAuT/BEwH9g5yfqqOqWq1nb73pfkfzMYyvETAVmSJEnaEqMMyFcDz0xyAINg/DLg99oNkiwC7q6qR4BTgQ8BVNXxzTYnAMuq6pQk84Ddq+q7SR4PvBi4fITnIEmSpDlmZF/Sq6qHgZOAS4GbgYuq6sYkpyc5sttsDLglya0MvpB3xiaafQJwaZLrgFUMgvcHRlC+JEmS5qiRjkGuqkuASyYte2szvRxYvok2zgXO7abvB54303VKkiRJE/r+kp6GMMUXFXtpZ4qbhUiSJO1wfNT0dqCqtvq1YsWKrW5DkiRpLjAgS5IkSQ0DsiRJktQwIEuSJEkNA7IkSZLUMCBLkiRJDQOyJEmS1DAgS5IkSQ0DsiRJktQwIEuSJEkNA7IkSZLUMCBLkiRJDQOyJEmS1DAgS5IkSQ0DsiRJktQwIEuSJEkNA7IkSZLUMCBLkiRJDQOyJEmS1DAgS5IkSQ0DsiRJktSY13cBkoaTZFa0UVVb3YYkSbOZV5Cl7URVbdVrxYoVW92G4ViSNBcYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpEaqqu8aRi7Jd4Bv9l1HzxYB3+27CPXKPiCwH2jAfiCwHwDsX1V7TV44JwKyIMnKqlrWdx3qj31AYD/QgP1AYD+YjkMsJEmSpIYBWZIkSWoYkOeOc/ouQL2zDwjsBxqwHwjsBxvlGGRJkiSp4RVkSZIkqWFAliRJkhoGZEmStgNJ1jfTv5nk1iT7T7P9a5O8cjOP8eWtqVHbVpL9kvxrkj26+YXd/JLNbGfvJMtHUuR2yjHIkiRtB5Ksr6r5SV4E/A3wG1X1jb7rUr+S/AnwjKo6McnfALdX1ZkjPua8qnp4lMfom1eQd1BJDk9yS5LVSU7Z3O2mWX57kuuTrEqyctTnoZkxA/3hQ0m+neSGbVOxRm0z+oS/+1kkya8CHwBePBGOk7wmydVJrk3y8SRP6pafluTkbno8yXuSrExyc5JfSPKJJF9P8o6m/fXdz7Fun+VJvpbk/CTp1v1mt+yaJO9L8ult/T7oMd4D/GKSNwC/Arw7yfwkVyT5avdv9lEASd6Z5HUTO070kSRLJv6MJ9kpybu6PnVdkj/slo8l+UKSi4GbtvlZbmtV5WsHewE7Ad8Ang7sDFwLLB12u+n2B24HFvV9jr62XX/o1v0q8Fzghr7Px9e26xP+7mfXC3gIuBs4aNLyPZvpdwCv76ZPA07upseBP++m/xi4E3gq8ARgzUQbwPru5xjwfWBfBhfT/plB+NoFuAM4oNvuo8Cn+35v5voL+A2ggF/v5ucBu3XTi4DVQIDnAJ9v9rsJ2A9YMvFnHDgR+O/d9BOAlcABXZ+4f+J3v6O/vII8yyT5+SRXJrkpySNJKsnpm9nMIcDqqrqtqn4EXAActRnbDbu/RmyW9Aeq6koG/zCrZ9u4T/i7n10eAr4MvHrS8gO7K3vXA8cDP7eR/S/ufl4P3FhVd1XVg8BtDELSZFdV1ZqqegRYxSBE/SxwW1X9a7fNR7f0ZDSjjgDuAg7s5gP8WZLrgMuBfYDFVfV/gZ/qxhz/PLCuqu6Y1NZ/AF6ZZBXwFWBP4Jnduqua3/0ObV7fBehRSXYBLgReWVVXJflTBv9bf1uzzReABVPsfnJVXd5N78Pgf/gT1gDPn2KfjW033f4FXJakgL+pKm8yPiKzqD9oluihT2h2eQT4HeCKJP+tqv6sW34ucHRVXZvkBAZX+qbyYNPOg83yR5g6D7TbbNjINupZkoOBXwd+EfhikgsYXFHeC3heVT2U5HYGf1cAfAx4KfAUBn+f/ESTDD6FuHTSccYYXEGeE+zss8uvAV+tqqu6+euAw6v7nAOgql7YS2WP+pWqWpvkp4DPJvlad4VJM2976A/atuwTc1xVPZDkPwJfSPL/quqDDP5DdFeSxzO4grx2hCXcAjw9yZKquh343REeS5vQjQt/P/CGqvpWkncB7wauAr7dheNDgfZuJxcyGMe+CPj3UzR7KfCfk3yu2/+nGW2fmpUMyLPLgQw++prwXOCr7QZDXh1ay2M/LtuXqTv3xrbb6P5VNfHz20k+yeCjWgPyaMyW/qDZY1v3Cc1CVXV3ksOBK5N8B/gfDD4K/073c6rf/0wd+wdJ/gj4TJL7gatHdSwN5TXAt6rqs938XwG/z+AR0i/rht2sBL42sUNV3ZhkAbC2qu6aos2/ZTCc5qtdAP8OcPTIzmCW8jZvs0iS1wCHVdVx3f/Y/hH45ar63ma2Mw+4FXgRg3/0rgZ+r6puHGY7BlcIplp+O/C4qrovya7AZ4HTq+ozW3bGms5s6Q8T22VwX81PV9WBqBfbuk802y/B3706SeZX1fouPJ0NfL2q3tN3XdJM8kt6s8tHgfndrVbOAY7b3H/4AGpwb8KTGHxMcjNwURNyLkmy93TbTbP/Ygbjm65l8PHN/zEcj9Ss6A/ddh9l8C32n0myJsnkLwlp29imfaKb93evyV7TfYHrRuDJDO7JLO1QvIIsSZIkNbyCLEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIktSTJOuH2OZvkyztpv/bpHVfHvYYSfZOsnwLaty9ezDExPwWtSNJ2xNv8yZJPUmyvqrmj2r7Ld1n0v5L8CEhkuYYryBLUs+SjCUZT7I8ydeSnN89pYxu+bIk7wSemGRVkvO7dRNXh+cnuSLJV5Ncn+SoKY6xpHvAyMRV6VXd6ztJ3jZNG+8E/l237bsmtbNLkr/rtv+/SQ7tlp+Q5BNJPpPk60n+YuRvoiTNoHl9FyBJAuA5wM8BdwJfAl4AfHFiZVWdkuSkqjp4in1/CBxTVfcmWQT8S5KLayMfEVbVHwAk2R/4DHDuxtoATgEOnDhud0V5wusGzdWzk/wscFn3CGyAg7tzehC4JclfVtUdm/meSFIvvIIsSbPDVVW1pqoeAVYBSzZj3wB/luQ64HJgHwaPht/4DskuwMeA11fVN7ekDeBXgI8AVNXXgG8CEwH5iqr6flX9ELgJ2H8zzkeSeuUVZEmaHR5spjeweX8/Hw/sBTyvqh5Kcjuwyyb2+WvgE1V1+Va0MZ2tOR9J6pVXkCVp+/FQksdPsfzJwLe7YHsom7ham+R1wIKqeucQbdwHLNhIU19gEKzphlY8Dbhl6LORpFnKgCxJ249zgOsmvqTXOB9YluR64JXA1zbRzsnAs5sv6r12Y21U1feALyW5Icm7JrXzV8Djun0uBE6oqgeRpO2ct3mTJEmSGl5BliRJkhoGZEmSJKlhQJYkSZIaBmRJkiSpYUCWJEmSGgZkSZIkqWFAliRJkhr/H2YYgSHD8sLNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "init_vars_for_plot = [(x if isinstance(x, str) else fr\"$\\sigma = {x}$\") for x in init_vars]\n",
    "\n",
    "fig, ax = plt.subplots(1, 1, figsize=(10, 5))\n",
    "\n",
    "ax.boxplot(accs_test.T, labels=init_vars_for_plot, showfliers=False)\n",
    "\n",
    "ax.set_title(\"Test quality in 5 runs\")\n",
    "ax.set_xlabel(\"Initialization\")\n",
    "ax.set_ylabel(\"Test accuracy\")\n",
    "ax.grid(True)\n",
    "\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5 слоев\n",
    "\n",
    "Выполните тут тот же код, что и в предыдущем пункте, но только уже с 5 слоями в сети."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:45:57.091744Z",
     "start_time": "2021-03-03T14:45:57.079777Z"
    }
   },
   "outputs": [],
   "source": [
    "init_vars = [1e-2, 1e-1, 1e0, 'Kaiming', 'Xavier']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:45:57.107701Z",
     "start_time": "2021-03-03T14:45:57.094737Z"
    }
   },
   "outputs": [],
   "source": [
    "accs_train = np.zeros((5, 5))\n",
    "accs_test = np.zeros((5, 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:27.904407Z",
     "start_time": "2021-03-03T14:45:57.111690Z"
    }
   },
   "outputs": [],
   "source": [
    "### your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Построим боксплоты полученного качества (горизонтальная линия в каждом столбце — среднее, прямоугольник показывает разброс)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:28.632502Z",
     "start_time": "2021-03-03T14:46:27.907401Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAFgCAYAAACmDI9oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAf+0lEQVR4nO3deZhlVX3u8e8LyEUGsVtMK4O0RoxpcW6HaBKbaBIwEfTGqERj8BrQBIw+JibceKNIBr0arzcaHMhEjEamGNPxohi5lChRmUQUEEUC0kgiSgs2CDL88sfZFRdFV9Wp7tq1T1d/P89TT5+9zzrr/M6p1afeWrX23qkqJEmSJI3sMHQBkiRJ0iQxIEuSJEkNA7IkSZLUMCBLkiRJDQOyJEmS1DAgS5IkSQ0DsiRtA5IckeQzzfamJA9bpL4vTbJuMfqSpOXAgCxp2epC5PTX3Um+32y/eAv6m0ry633UulBVtXtVXQWQ5KQkf7QVfT2qqqa25LFJrp7xvn5iS+uQpEmx09AFSFJfqmr36dtJrgZ+vao+OVxFy9ZztuR9TbJjVd3VR0GStDWcQZa03UmyQ5Jjk3w9yXeSnJpkZXffLkk+0O3/bpLzk6xK8sfATwF/3s2U/vksff9qkmu6x7++m2F9VnffPWZ6k6xLsqHZnq7pe0kuS/K8OV5DJXl4kqOAFwO/29X1z0lel+QfZrR/Z5I/m6Wvtsbjuvfj/V0dlyZZO+57O5fu9b8nyRlJbgEOmjkrv5mlJJXklUm+1n0/TkiS7r6HJ/lUkpuSfDvJKYtRpyQZkCVtj14FPBd4BrA3sBE4obvv14A9gf2ABwCvBL5fVa8HPg0c0y1vOGZmp0nWAO8BfrXr9wHAvguo6+uMQviewJuADyR58FwPqKoTgQ8Cb+3qeg7wAeDgJPfv6toJeBHw/jHrOBQ4Gbg/sB7Y7C8DjQ8muSHJJ5I8dp62vwL8MbAH8Jl52k77ReBJwGOAFwA/3+3/Q+ATwApG7/O7xuxPkuZkQJa0PXol8Pqq2lBVtwPHAc/vguQdjILtw6vqrqq6sKpuHrPf5wMfrapzun7/ALh73KKq6rSq+mZV3V1VpwBfA568gNc13c/1wDnAL3e7Dga+XVUXjtnFZ6rqjG75w98Bc4XeFwOrgf2Bs4Ezp4P5LP6pqs7tXuNtY9bzlqr6blV9o3uOx3X77+ied++quq2qxg3ckjQnA7Kk7dH+wD92f7L/LnA5cBewilEgPBM4Ock3k7w1yX3G7Hdv4Nrpjaq6BfjOuEUleWmSi5u6DgT2GvfxM/wt8JLu9ksYva5x/Xtz+1Zgl+6Xh3vpwu73q+rWqnoz8F1Gs+CzuXaO+8atZ3pt+e8CAc7rloL8jy3oW5LuxYAsaXt0LXBIVd2/+dqlqq6rqjuq6k1VtQZ4GqM/77+0e1zN0+/1jJZmAJBkV0az0dNuAXZtth/UtN0f+AvgGOABVXV/4MuMAuB8NlfXR4DHJDmwew0fHKOfxVDMXfPMWmd9T+Z9oqp/r6ojq2pv4BXAu5M8fOxKJWkWBmRJ26P3An/chVKSPDDJYd3tg5I8OsmOwM2M/ow/vUziP4C5zj18OvCLSX4yyc7A8dzzc/Zi4NlJViZ5EPCa5r7dGIXHG7o6XsZoBnkc96qrW75wOvD3wHnd8oRFleQhSZ6eZOfu4MbXMZrxPncB3VwM/Pcku3bh9uULeP5fTjK9xnsjo/dv7CUtkjQbA7Kk7dGfMTr47BNJvgd8DnhKd9+DGAXLmxktvfgUP1ye8GeM1ipvTPLOmZ1W1aXA0YxC6fWMQtuGpsnfAV8ErmZ0cNkpzWMvA94OfJZR4H004wfNvwLWdEszPtLs/9uun4Usr1iIPRgdlLgRuI7RWudDqmrsZSXAO4AfMHrNf8vCZrqfBHw+ySZG389XT58bWpK2Rqrm+4uhJGlLDXn+5SQPAb4CPGgBBxpK0nbPGWRJWoaS7AC8FjjZcCxJC+OV9CRpmUmyG6MlC9cwWvYgSVoAl1hIkiRJDZdYSJIkSY1tbonFXnvtVatXrx66jMHdcsst7LbbbkOXoQnimFDL8aCW40Etx8MPXXjhhd+uqgfO3L/NBeTVq1dzwQUXDF3G4Kampli3bt3QZWiCOCbUcjyo5XhQy/HwQ0mu2dx+l1hIkiRJjd4CcpK/TvKtJF+e5f4keWeSK5NckuQJfdUiSZIkjavPGeSTmPv0QocAB3RfRzG6GpMkSZI0qN4CclWdA9w4R5PDgPfXyOeA+yd5cF/1SJIkSeMY8iC9fYBrm+0N3b7rZzZMchSjWWZWrVrF1NTUUtQ30TZt2uT7oHtwTKjleFDL8aCW42F+28RZLKrqROBEgLVr15ZHXnoEqu7NMaGW40Etx4Najof5DXkWi+uA/Zrtfbt9kiRJ0mCGDMjrgZd2Z7N4KnBTVd1reYUkSZK0lHpbYpHkQ8A6YK8kG4A3AvcBqKr3AmcAzwauBG4FXtZXLZIkSdK4egvIVXX4PPcXcHRfzy9JkiRtCa+kJ0mSJDUMyJIkSVJjmzjNmyRJ27MkQ5cAwGh1pLT8GZAlSZpwWxtMkxhupQUwIEuSJG1D/ItC/wzI0gTww05avlauXMnGjRuHLmPwz5kVK1Zw4403DlrDJJiU8bAYtnZMTfKYMCBLW8kPux+a5A87aSgbN24c/JfPSbi08NABfVLc+Ft3AfcbuowJcdfQBczKgCxtpUn44Qf+AJSkbUHedPPgPzMm4ecFdGvjjxu6is0zIEuSJC0hJxNGVqxYMXQJszIgD2RS/nMM/VusJC139cb7wXF7DlrDOoCpQUsYvQ+aiJ+7kzKDPMkMyAPxlD3LxyT88AN/AEqTKm+6eegSJsKKFSu48bihq5DGY0CWttZxN211F/5FQVq+FuP/lZ8R0tLyUtNbYOXKlSQZ9AsYvIaVK1cO/J1YPqpqq7/OPvvsre5D0mSahM8HPyO0PXEGeQtMwlkLJmH90KTMaEiSJC0mZ5AlSZKkhjPIW2ASDspaBx6QJUmS1AMD8hbwiOQRj0iWJEnLkQF5Cwy9/hgmYw2yJEnScuQaZEmSJKlhQJYkSZIaBmRJkiSpYUCWJEmSGgZkSZIkqWFAliRJkhoGZEmSJKlhQJYkSZIaBmRJkiSpYUCWJEmSGgZkSZIkqWFAliRJkhoGZEmSJKlhQJYkSZIaBmRJkiSpYUCWJEmSGgZkSZIkqWFAliRJkhoGZEmSJKmx09AFSJLuKcnQJQBQVUOXIEmDMCBL0oRZjGCaxIArSVvIJRaSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1eg3ISQ5OckWSK5Mcu5n7H5Lk7CRfSHJJkmf3WY8kSZI0n94CcpIdgROAQ4A1wOFJ1sxo9r+AU6vq8cCLgHf3VY8kSZI0jj5nkJ8MXFlVV1XVD4CTgcNmtCngft3tPYFv9liPJEmSNK+deux7H+DaZnsD8JQZbY4DPpHkVcBuwLM211GSo4CjAFatWsXU1NRi17rN2bRpk++D7sExoZkcD5rm54Najof59RmQx3E4cFJVvT3JTwB/l+TAqrq7bVRVJwInAqxdu7bWrVu39JVOmKmpKXwf1HJMaCbHg6b5+aCW42F+fS6xuA7Yr9net9vXejlwKkBVfRbYBdirx5okSZKkOfUZkM8HDkjy0CQ7MzoIb/2MNt8AngmQ5McZBeQbeqxJkiRJmlNvAbmq7gSOAc4ELmd0topLkxyf5NCu2W8DRyb5IvAh4Iiqqr5qkiRJkubT6xrkqjoDOGPGvjc0ty8Dnt5nDZIkSdJCeCU9SZIkqWFAliRJkhoGZEmSJKlhQJYkSZIaQ18oRJKWnZUrV7Jx48ahyyDJoM+/YsUKbrzxxkFrkKQtYUCWpEW2ceNGhj5j5SRcKWvogC5JW8olFpIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUmPegJzkOUkM0pIkSdoujBN8Xwh8Lclbkzyy74IkSZKkIc0bkKvqJcDjga8DJyX5bJKjkuzRe3WSJEnSEhtr6URV3QycDpwMPBh4HnBRklf1WJskSZK05MZZg3xokn8EpoD7AE+uqkOAxwK/3W95kiRJ0tLaaYw2vwS8o6rOaXdW1a1JXt5PWZIkSdIwxgnIxwHXT28kuS+wqqqurqqz+ipMkiRJGsI4a5BPA+5utu/q9kmSJEnLzjgBeaeq+sH0Rnd75/5KkiRJkoYzTkC+Icmh0xtJDgO+3V9JkiRJ0nDGCcivBH4/yTeSXAv8HvCKcTpPcnCSK5JcmeTYWdq8IMllSS5N8vfjly5JkiQtvnkP0quqrwNPTbJ7t71pnI6T7AicAPwssAE4P8n6qrqsaXMA8D+Bp1fVxiQ/sgWvQZIkSVo045zFgiS/ADwK2CUJAFV1/DwPezJwZVVd1fVxMnAYcFnT5kjghKra2PX5rQVVL0mSJC2ycS4U8l7ghcCrgAC/DOw/Rt/7ANc22xu6fa1HAI9Icm6SzyU5eKyqJUmSpJ6MM4P8tKp6TJJLqupNSd4OfGwRn/8AYB2wL3BOkkdX1XfbRkmOAo4CWLVqFVNTU4v09NuuTZs2+T7oHhwTk2Xo78WkjIdJqEGTMx40GRwP8xsnIN/W/Xtrkr2B7wAPHuNx1wH7Ndv7dvtaG4DPV9UdwL8l+SqjwHx+26iqTgROBFi7dm2tW7dujKdf3qampvB9UMsxMVmG/l5MyniYhBo0OeNBk8HxML9xzmLxz0nuD7wNuAi4GhjnbBPnAwckeWiSnYEXAetntPkIo9ljkuzFaMnFVWP0LUmSJPVizhnkJDsAZ3VLHv4hyUeBXarqpvk6rqo7kxwDnAnsCPx1VV2a5Hjggqpa3933c0kuY3SFvtdV1Xe27iVJ0rDqjfeD4/YctIZ1AFODljB6HyRpGzRnQK6qu5OcADy+274duH3czqvqDOCMGfve0Nwu4LXdlyQtC3nTzYw+3oYzCX9CTUIdN2gJkrRFxllicVaSX8r0+d0kSZKkZWycgPwK4DTg9iQ3J/lekpt7rkuSJEkaxDhX0ttjKQqRJEmSJsG8ATnJT29uf1Wds/jlSJIkScMa5zzIr2tu78LoEtIXAj/TS0WSJEnSgMZZYvGcdjvJfsD/7asgSZIkaUjjHKQ30wbgxxe7EEmSJGkSjLMG+V3A9Ak9dwAex+iKepIkSdKyM84a5Aua23cCH6qqc3uqR5IkSRrUOAH5dOC2qroLIMmOSXatqlv7LU2SJElaemNdSQ+4b7N9X+CT/ZQjSZIkDWucgLxLVW2a3uhu79pfSZIkSdJwxgnItyR5wvRGkicC3++vJEmSJGk446xBfg1wWpJvAgEeBLywz6IkSZKkoYxzoZDzkzwS+LFu1xVVdUe/ZUmSJEnDmHeJRZKjgd2q6stV9WVg9yS/2X9pkiRJ0tIbZw3ykVX13emNqtoIHNlbRZIkSdKAxgnIOybJ9EaSHYGd+ytJkiRJGs44B+l9HDglyfu67Vd0+yRJkqRlZ5yA/HuMQvFvdNv/AvxlbxVJkiRJAxrnLBZ3A+/pviRJkqRlbd6AnOQA4M3AGmCX6f1V9bAe65IkSZIGMc5Ben/DaPb4TuAg4P3AB/osSpIkSRrKOAH5vlV1FpCquqaqjgN+od+yJEmSpGGMc5De7Ul2AL6W5BjgOmD3fsuSJEmShjHODPKrgV2B3wKeCLwE+LU+i5IkSZKGMs5ZLM7vbm4CXtZvOZIkSdKwxplBliRJkrYbBmRJkiSpMW9ATvL0cfZJkiRJy8E4M8jvGnOfJEmStM2b9SC9JD8BPA14YJLXNnfdD9ix78IkSZKkIcx1FoudGZ3veCdgj2b/zcDz+yxKkiRJGsqsAbmqPgV8KslJVXUNQHfBkN2r6ualKlCSJElaSuOsQX5zkvsl2Q34MnBZktf1XJckSZI0iHEC8ppuxvi5wMeAhwK/2mdRkiRJ0lDGCcj3SXIfRgF5fVXdAVSvVUmSJEkDGScgvw+4GtgNOCfJ/owO1JMkSZKWnbnOYgFAVb0TeGez65okB/VXkiRJkjScca6ktyrJXyX5WLe9Bvi13iuTJEmSBjDOEouTgDOBvbvtrwKv6akeSZIkaVCzBuQk08sv9qqqU4G7AarqTuCuJahNkiRJWnJzzSCf1/17S5IH0J25IslTgZv6LkySJEkawlwH6aX797XAeuBHk5wLPBAvNS1JkqRlaq6A/MAkr+1u/yNwBqPQfDvwLOCSnmuTJEmSltxcAXlHYHd+OJM8bdf+ypEkSZKGNVdAvr6qjl+ySiRJkqQJMNdBejNnjiVJkqRlb66A/Mwlq0KSJEmaELMG5Kq6cSkLkSRJkibBOFfSkyRJkrYbBmRJkiSp0WtATnJwkiuSXJnk2Dna/VKSSrK2z3okSZKk+fQWkJPsCJwAHAKsAQ5PsmYz7fYAXg18vq9aJEmSpHH1OYP8ZODKqrqqqn4AnAwctpl2fwj8b+C2HmuRJEmSxjLXhUK21j7Atc32BuApbYMkTwD2q6r/l+R1s3WU5CjgKIBVq1YxNTW1+NVuYzZt2uT7oHtwTEyWob8XkzIeJqEGTc540GRwPMyvz4A8pyQ7AP8HOGK+tlV1InAiwNq1a2vdunW91rYtmJqawvdBLcfEZBn6ezEp42ESatDkjAdNBsfD/PpcYnEdsF+zvW+3b9oewIHAVJKrgacC6z1QT5IkSUPqMyCfDxyQ5KFJdgZeBKyfvrOqbqqqvapqdVWtBj4HHFpVF/RYkyRJkjSn3gJyVd0JHAOcCVwOnFpVlyY5PsmhfT2vJEmStDV6XYNcVWcAZ8zY94ZZ2q7rsxZJkiRpHF5JT5IkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpMZOQxcgSctRkqFLGNyKFSuGLkGStogBWZIWWVUNXQJJJqIOSdoWucRCkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJavQakJMcnOSKJFcmOXYz9782yWVJLklyVpL9+6xHkiRJmk9vATnJjsAJwCHAGuDwJGtmNPsCsLaqHgOcDry1r3okSZKkcfQ5g/xk4MqquqqqfgCcDBzWNqiqs6vq1m7zc8C+PdYjSZIkzWunHvveB7i22d4APGWO9i8HPra5O5IcBRwFsGrVKqamphapxG3Xpk2bfB90D44JzeR40DQ/H9RyPMyvz4A8tiQvAdYCz9jc/VV1InAiwNq1a2vdunVLV9yEmpqawvdBLceEZnI8aJqfD2o5HubXZ0C+Dtiv2d6323cPSZ4FvB54RlXd3mM9kiRJ0rz6XIN8PnBAkocm2Rl4EbC+bZDk8cD7gEOr6ls91iJJkiSNpbeAXFV3AscAZwKXA6dW1aVJjk9yaNfsbcDuwGlJLk6yfpbuJEmSpCXR6xrkqjoDOGPGvjc0t5/V5/NLkiRJC+WV9CRJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCVJkqSGAVmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWrsNHQBkqR7SjIR/VTVotQhSdsaZ5AlacJU1VZ/nX322VvdhyRtrwzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSI1U1dA0LkuQG4Jqh65gAewHfHroITRTHhFqOB7UcD2o5Hn5o/6p64Myd21xA1kiSC6pq7dB1aHI4JtRyPKjleFDL8TA/l1hIkiRJDQOyJEmS1DAgb7tOHLoATRzHhFqOB7UcD2o5HubhGmRJkiSp4QyyJEmS1DAgS5IkSQ0DsiRJEyjJpub2s5N8Ncn+c7R/ZZKXLvA5/nVratRwkuyX5N+SrOy2V3TbqxfYz95JTu+lyG2Ya5AlSZpASTZV1e5Jngm8D/j5qvr60HVpciT5XeDhVXVUkvcBV1fVm3t+zp2q6s4+n2MSOIO8DUpycJIrklyZ5NiFtkvy10m+leTLS1OxltICxofjYDsw7vd53HGjpZXkp4G/AH5xOhwnOTLJ+Um+mOQfkuza7T8uye90t6eSvCPJBUkuT/KkJB9O8rUkf9T0v6n7d133mNOTfCXJB5Oku+/Z3b4Lk7wzyUeX+n3QrN4BPDXJa4CfBP40ye5JzkpyUZIvJTkMIMlbkhw9/cDp8ZJk9fTnQ5Idk7ytG1+XJHlFt39dkk8nWQ9ctuSvcgAG5G1Mkh2BE4BDgDXA4UnWLLDdScDBS1KwltS446NzEo6D7cFJzPN9XuC40dL5b8BHgOdW1Vea/R+uqidV1WOBy4GXz/L4H3RXS3sv8E/A0cCBwBFJHrCZ9o8HXsNoDDwMeHqSXRjNXh9SVU8E7nVJXg2nqu4AXscoKL+m274NeF5VPQE4CHh798vOKcALmoe/oNvXejlwU1U9CXgScGSSh3b3PQF4dVU9orcXNEEMyEsoyWOTnJPksiR3J6kkxy+wmycDV1bVVVX1A+Bk4LCFtKuqc4Abt+KlqAdLPD4cBxNukcbDuN/nsceNltQdwL9y7wB8YDeb9yXgxcCjZnn8+u7fLwGXVtX1VXU7cBWw32ban1dVG6rqbuBiYDXwSOCqqvq3rs2HtvTFqDeHANcz+uUHIMCfJLkE+CSwD7Cqqr4A/Ei35vixwMaqunZGXz8HvDTJxcDngQcAB3T3ndeMg2Vvp6EL2F50v4WfAry0qs5L8ofALsAbmzafBvbYzMN/p6o+2d3eB2gH9AbgKZt5zLjtNAEGGB+aYIs4HsbluJlMdzOa5Tsrye9X1Z90+09iNKv8xSRHAOtmefztTT+3N/vvZvM//9s2d83SRhMkyeOAnwWeCnwmycnAzzOa6X9iVd2R5GpGnx8ApwHPBx7EvWePYRSuX1VVZ854nnXALYv/CiaXg3/pPAu4qKrO67YvAQ6u5ijJqvqpQSrTJHB8qOV4EABVdWuSXwA+neQ/quqvGP1idH2S+zCaQb6uxxKuAB6WZHVVXQ28sMfn0gJ0yybew2hpxTeSvA34U+A84FtdOD4IaM98cgqjNe17Ac/YTLdnAr+R5P93j38E/Y6viWVAXjoHMvoz17QnABe1DcacEbqOe/5pbF82P3jHbafJsNTjQ5NtscbDuBw3E6yqbkxyMHBOkhuAP2D05+8bun83Nw4W67m/n+Q3gY8nuQU4v6/n0oIdCXyjqv6l23438DJGl5F+UbcE5wLgv9avV9WlSfYArquq6zfT518yWlpzURfAbwCe29srmGCe5m2JJDkS+JmqOrz7jeyfgadV1XcW2M9OwFeBZzL6AXY+8CtVdelC2mV0nsSPVtWBaHBLPT6a9qtxHEycxRoPTX+rmeP7vNBxo+1Lkt2ralMXmE4AvlZV7xi6LqlPHqS3dD4E7N6dSuVE4PAt+WHXnXvwGEZ/BrkcOLUJvWck2XuMdh8CPgv8WJINSWY7AlpLZ0nHR7ftOJhcizIeYPbv87ifFxKjMxlcDFwK7MnorBbSsuYMsiRJktRwBlmSJElqGJAlSZKkhgFZkiRJahiQJUmSpIYBWZIkSWoYkCWpR0k2jdHmL5Os6W7//oz7/nXc50iyd5LTt6DG+3cXg5je3qJ+JGm58DRvktSjJJuqave+2m/pY2Y8fjVeMEaS/oszyJK0BJKsSzKV5PQkX0nywe7KZHT71yZ5C3DfJBcn+WB33/Ts8O5JzkpyUZIvJTlsM8+xuru4yPSs9MXd1w1J3jhHH28BfrRr+7YZ/eyS5G+69l9IclC3/4gkH07y8SRfS/LW3t9ESVoiOw1dgCRtRx4PPAr4JnAu8HTgM9N3VtWxSY6pqsdt5rG3Ac+rqpuT7AV8Lsn6muXPgFX16wBJ9gc+Dpw0Wx/AscCB08/bzShPO3rUXT06ySOBT3SXvwZ4XPeabgeuSPKuqrp2ge+JJE0cZ5AlaemcV1Ubqupu4GJg9QIeG+BPklwCfBLYB1g15wOSXYDTgFdV1TVb0gfwk8AHAKrqK8A1wHRAPquqbqqq24DLgP0X8HokaWI5gyxJS+f25vZdLOwz+MXAA4EnVtUdSa4GdpnnMe8FPlxVn9yKPuayNa9HkiaWM8iSNFnuSHKfzezfE/hWF2wPYp7Z2iRHA3tU1VvG6ON7wB6zdPVpRsGabmnFQ4Arxn41krQNMiBL0mQ5Ebhk+iC9xgeBtUm+BLwU+Mo8/fwO8OjmQL1XztZHVX0HODfJl5O8bUY/7wZ26B5zCnBEVd2OJC1jnuZNkiRJajiDLEmSJDUMyJIkSVLDgCxJkiQ1DMiSJElSw4AsSZIkNQzIkiRJUsOALEmSJDX+E1Csv7PVfkuOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "init_vars_for_plot = [(x if isinstance(x, str) else fr\"$\\sigma = {x}$\") for x in init_vars]\n",
    "\n",
    "fig, ax = plt.subplots(1, 1, figsize=(10, 5))\n",
    "\n",
    "ax.boxplot(accs_test.T, labels=init_vars_for_plot, showfliers=False)\n",
    "\n",
    "ax.set_title(\"Test quality in 5 runs\")\n",
    "ax.set_xlabel(\"Initialization\")\n",
    "ax.set_ylabel(\"Test accuracy\")\n",
    "ax.grid(True)\n",
    "\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Дайте развёрнутый ответ на вопросы (в этой же ячейке):\n",
    "* Как отличаются качество на обучении и контроле и устойчивость процесса обучения при различных инициализациях?\n",
    "* Какие инициализации помогают обучать более глубокие сети?\n",
    "\n",
    "__Ответы:__\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь сравним скорость обучения при различных инициализациях. Создайте два списка списков `accs_test_on_iterations`, `accs_train_on_iterations` в каждом из которых в позиции `[i]` (см. описание `i` в предыдущем пункте) будет лежать список из значений `accuracy` на тестовой и обучающей выборках соотвественно, полученных во время обучения модели. Количество слоев в сети зафиксируйте равным 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:45:57.091744Z",
     "start_time": "2021-03-03T14:45:57.079777Z"
    }
   },
   "outputs": [],
   "source": [
    "init_vars = [1e-3, 1e-2, 1e-1, 'Kaiming', 'Xavier']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:28.647453Z",
     "start_time": "2021-03-03T14:46:28.635495Z"
    }
   },
   "outputs": [],
   "source": [
    "accs_test_on_iterations = []\n",
    "accs_train_on_iterations = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:34.433662Z",
     "start_time": "2021-03-03T14:46:28.650417Z"
    }
   },
   "outputs": [],
   "source": [
    "### your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:34.875348Z",
     "start_time": "2021-03-03T14:46:34.436663Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAFgCAYAAACmDI9oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAB9wUlEQVR4nO3dd3xc1Zn/8c/RqGvUm9Vs2ZZ7ARvbdDDV4BBIpWwaaUCWkIRskiXZkBDSNr8km2xCGglJIBtMCCGEXgwYTDHGNsW9F/XeZqSRNDPn98cdyZLVRla3v+/XSy9p7px775mja+uZZ557jrHWIiIiIiIijojx7oCIiIiIyESiAFlEREREpBsFyCIiIiIi3ShAFhERERHpRgGyiIiIiEg3CpBFRERERLpRgCwiJyxjzPXGmFe6PfYYY2aM0LHnGGPeNsY0G2O+MBLHPOb4K40xJd0ebzfGrAz9bIwxfzLG1BtjNoa2fc4YUxl6jekj3Z/hMMaca4zZPRJtjTFTQ6/RFcax+h3DkTSS15WITAwKkEWkh9Af+86voDGmtdvjjxzH8dYZYz4zGn0dKmut21p7AMAY82djzPeGcbivAS9aaxOttb8YmR72z1q7wFq7LvTwHOASIN9au8IYEwX8D3Bp6DXWjnZ/ujPGHDLGXNzf89ba9dbaOeEc69i2xx7bWnsk9BoDQ+3nMWN4XPq6nrtfVyJyYlCALCI9hP7Yu621buAI8N5u2/463v2bQKYB249nR2NM5Aic+5C11ht6nA3EDqM/g2ZjRUROJgqQRSQsxpgIY8xtxpj9xphaY8yDxpi00HOxxpj/C21vMMa8aYzJNsZ8HzgXuCuUgb6rn2N/zBhzOLT/f3XPGh6b6e3jY/POPjUbY3YYY94/wGuwxpgiY8wNwEeAr4X69Zgx5qvGmH8c0/4Xxpj/7eM4LwAXdHtds40xycaY+4wx1aHX8k1jTESo/fXGmFeNMT8zxtQCd/RxzLjQa603xuwAlh/z/CFjzMXGmE8DfwDODJ17DdBZktAQ6hvGmLnGmOeMMXXGmN3GmKu7HevPxpjfGGOeNMZ4gQuMMbnGmH+E+n/QdCsbMcbcEfp93xca5+3GmGWh5/4CTAUeC/Xna328tmN/Z4eMMV8xxrxrjGk0xvzNGBN7bNu+jm2MKQz9HiNDbT5pjNkZ6tcBY8yNvX/zPccw9HODOfrJiDd0zEJjTKox5vHQONSHfs4P7dPn9dx5XYV+Huw6eMUY85PQsQ8aYy7v1r/rQ6+hOfTckD+xEZERYq3Vl770pa8+v4BDwMWhn78IbADygRjgd8Ca0HM3Ao8B8YALOA1ICj23DvjMAOeYD3iA80LH/R/A3+28fwa+1639SqCk2+MPA7k4b/ivAbxATui564FXurW1QFE/x80J7ZsSehwJVAGn9dPvHq8LuA/4F5AIFAJ7gE9364cfuCV03Lg+jvffwHogDSgAth3zOrv/Lo59XYWh1xYZepwAFAOfDJ1vCVADzO/22huBs0PjFg9sBr4FRAMzgAPAqlD7OwAfsDr0+/0hsKGvvvUzVsf+zg4BG0O/tzRgJ3DTAG0vHuC1vgeYCRjgfKAFWBrOsbpt/wHwMhAFpAMfDI1JIvB34JH+fu99XFeDXQcdwGdD4/g5oCzU9wSgCZjT7XpcMN7/B+hLXyfrlzLIIhKum4D/staWWGvbcIKmD4UyeR04gUWRtTZgrd1srW0K87gfAh631r4cOu7tQDDcTllr/26tLbPWBq21fwP2AiuG8Lo6j1OOEyR9OLTpMqDGWrt5sH2NU6JwLfB1a22ztfYQ8FPgY92alVlrf2mt9VtrW/s4zNXA9621ddbaYmA4dc1X4JRg/Cl0vreAf3D0tQH8y1r7qrU2CCwCMq21d1pr261TT/v70Gvq9Iq19knr1P7+BThlGP0D+EXo91aH8+bq1OM5iLX2CWvtfut4CXgWJ8sbFmPMNcC/AR+01nZYa2uttf+w1rZYa5uB7+ME3uEcK5zr4LC19vehcbwXJxDODj0XBBYaY+KsteXW2uMqmRGR4VOALCLhmgb8M/TRdANO1i+A88f9L8AzwAPGmDJjzP8zzo1j4cjFyXYCYJ262rBvMjPGfNw4s0l09mshkBHu/se4F/ho6OeP4ryucGTgZB8Pd9t2GMjr9riYgeUe0+Zwfw3DMA04vXNMQuPyEWBKP/2ZBuQe0/4bHA3cACq6/dwCxJrh1VIfezz38RzEGHO5MWZDqJSkASfLHdbv3xizBLgLeL+1tjq0Ld4Y87tQeUQTzpumFBNenXY410HX67bWtoR+dIeu+2tw3oiWG2OeMMbMDed1iMjIU4AsIuEqBi631qZ0+4q11paGMm/fsdbOB87CyWB+PLSfHeS45TglBYAToOBkozt5cT7u7jSlW9tpOJnOzwPp1toUnNIEE8br6atfjwCLjTELQ68h3JsSa3Cy6NO6bZsKlA5yvu56jENo/+NVDLx0zO/Kba39XD/9KQYOHtM+0Vq7OszzDfbahqPfYxtjYnAy4z8BskO//ycJ4/dvjMnC+X3fHMqwd/oPYA5wurU2Caf0h27HHOi1hnMd9Mta+4y19hKcrPIunGtbRMaBAmQRCddvge+HglKMMZnGmKtCP19gjFkUyrI14QQJnWUSlTg1rf15CLjCGHOOMSYauJOe/ze9Daw2xqQZY6YAX+r2XAJOwNKZ/fskTgY5HL36Za31hfpzP7DRWnsknAOFPi5/EGd8EkNj9GXg/8LsC6H9vx66SSwfp175eD0OzDbOzY9Roa/lxph5/bTfCDQbY/7TODcLuowxC40xy/tpf6zBfsfDMdCxo3Hq1qsBf+iGt0sHO2Ao8/0Q8H/W2gePeToRaMW54TEN+Ha4/RnOdWCcm1qvMsYkAG04dflhlxqJyMhSgCwi4fpf4FHgWWNMM84Ne6eHnpuCE3A04ZRevMTR8oT/xalVrjfG9KqrDdVZ3owTlJYD9UBJtyZ/Ad7BucHqWeBv3fbdgVPj+TpO4LIIeDXM13MPMD9UUvBIt+33ho4TbnlFp1twst0HgFdCr+ePQ9j/Ozgfxx/EeZ1DPX+XUO3spTj1sGU4H+v/CCeY7Kt9ACdjfmro/DU4M2Ukh3nKHwLfDI3lV46330M9duh1fgEnKK3HqSV+NIxj5uPUKX/J9Jz3eyrwcyAOZww2AE8fs++A1zPHfx1E4ATTZUAdTt3z5wbcQ0RGjbF2ND8ZExEZOmPMIZyZAtaOw7mn4ny8PWUINxqKiMgJRBlkEZGQ0Hy1XwYeUHAsInLyGu5qTiIiJ4RQ7WclTpnDZePcHRERGUcqsRARERER6UYlFiIiIiIi3YxaiYUx5o84d0VXWWt7TbtkjDE4dwOvxpkk/npr7ZbBjpuRkWELCwtHuLeD83q9JCQkjPl5TxQav+HTGA6Pxm94NH7Do/EbHo3f8Gj8+rd58+Yaa23msdtHswb5zzgrFN3Xz/OXA7NCX6cDv+HolFH9KiwsZNOmTSPUxfCtW7eOlStXjvl5TxQav+HTGA6Pxm94NH7Do/EbHo3f8Gj8+meM6XPV0lErsbDWvowzl2N/rgLus44NOEt55oxWf0REREREwjGeNch5OMubdiqh53r1IiIiIiJjblRnsTDGFAKP91OD/Djw39baV0KPnwf+01rbq37CGHMDcANAdnb2aQ888MCo9bk/Ho8Ht9s95uc9UWj8hk9jODwav+HR+A2Pxm94NH7Do/Hr3wUXXLDZWrvs2O3jOQ9yKVDQ7XF+aFsv1tq7gbsBli1bZsejjkb1O8Oj8Rs+jeHwaPyGR+M3PBq/4dH4DY/Gb+jGs8TiUeDjxnEG0GitLR/H/oiIiIiIjOo0b2uAlUCGMaYE+DYQBWCt/S3wJM4Ub/twpnn75Gj1RUREREQkXKMWIFtrrxvkeQvcPFrnFxERERE5HlpJT0RERESkGwXIIiIiIiLdKEAWEREREelmPKd5E5EwWGsJ1NYSUd9AR2Vln21MRASu9HRMxODvea21BOrqsH5/+J0whsiMjLCO39f5go2NRCQmYlyuvvvT0IBtbx/ScV1uNxEJCUPuj4iIyGAUIIuMs7YDB/FueB06Orq2BdvbaT94iLb9+2jft5+g10smzpQv/THx8cTMnEnMzJlEz5hBREx013O2o4O2Q4do37eftv37CTY3D7mfJjaWmBkziJlVRPT0GUTExfbb1vr9tB8+Qtu+fc75GhsxMTFEz5xBzMwioqdNw19ViWfvbtr37yeiyTvk/mR/85ukffQjQ96vLdDGi0deJNedy6KMRRhjhnwMERE5sSlAlknv6UNP87ddf8MfPJoRjY2M5Zy8c7h02qXkuHMAJ1O5t2Evzxx6hner3yXPncfMlJnOV/JMsuKzjjtYqvPV9Tj/saIjokmJTenqR/vBgzQ/8wxNTz9D2+7dfe7jysggpqiI5Pe9j+jp09l78CCz58zu3dBCS2sTEaWVdOzfj/fVV2l85JHex0tLg+kFlJ01k63xdZR11OAL+Hq1i4qIIiMug0hz9L8HVxAKGiPJqW4h6dWXcf3r0YEHBIhITiZmVhFJl19G9NRp+KuqaNqzg+rXXyLmsSa8sYYjGZaSmYbStAjaogc9JCkxKcxPX8CC9Pn4TimiqqWqz3ZtzQHSE9KId8d0beuwHfx151/549Y/UtXq7JeTkMMl0y7h0sJLWZyxWMGy9ORvg/0vwI5/QdAP894LRZdAdPx490wkbMGgpaEtSGVT7//vO7V1BNlf7WFPZTN7Kj0U17UQGGClZVeEYVpaPLOzE5mV7WZmppvoyP4/YWz3O8ffW+lhb1UzJfWt5KbEMSvL3XWMvJS4CfV/sAJkmXCaalsp39dI3uxU3Kkx/barba3l+298n+cOP0dhUiFTEqYcfc5Xy082/YSfbPoJizMWsyhzEa+WvsqhpkNEmAhmpcxiV90u/rH3H137JEYnMjPZCZinJU0jKiKq6zlXhItpidOYmXI0kN5Xt5eXNz/M7i1rCZSUETHAqu3RfpjnTWZWfQyJZY3Q0gpA3NKlZH/j67gvvBBXUlJXe+Ny9SofaHnxRdpWzGNfwz72N+x3vhr3c6DhAB7jIa4wjpmnzGRmynnMjM4j1hztvyfQwvM1r7G9djsA89PnsyjjXE5JmUlRShH57nwqWyq7jn246TAdwaMZbX/Qz2PNR7oC0ph2F66g81xSdBL5iflUt1RT3VoNgDVg4/xMT2lnZkobOQnNvFmxjbdz3obzYZ57DgtyTqUodRYrUmYyPWk6cVFx/Y5fm7+NV0pf4ZnDz/Cvsjfw29dhyz2wBeLbk5hWv4AMbz6prdmktuQQ53fz+ox/0jq3jJkpM5kSP4WHSx+m8Ugjp2Wfxp1n30mtr5ZnDz3L/bvu574d93Hj4hv5/JLP9/9LlMmv3Qs1e6B6N7TW99/OWih/G3Y/BW1NEJsCEZGw9e8QlQCzV0H+MjADlBzFZ8DiD4/0KxDpVzBoKW1oZU9lM3urnGB3b6WHfVUeWjsC8OLzYR0nKzGGwowE4iJ7l8R1avcHeXF3FX/fXDLkfma4o8lLjWfd7moe6rb/7z52GqsWTBlgz7Fl7ADvECaiZcuW2U2bNo35ebVMY/+stRQ3F7O3YS8HGg6wr2Ef5d5yul9bAU+AOy+5k6LUol77PnHwCZ4//DzXT/ssda9HsOu1coIBZ98pM5Ox0buIf+FeEgOtxEbGEuuKpT3WxT1zytlQ2M6/L7mZj87+OLExPVOQR5qO8OzhZ3nuwDPUHt7NBR0zObO9gGk1LmxFPZHBdvxBPz5/K60BHz6/D1/oe5vxU55mKEk3FGdCvTuGzMY28mtgel0kBTWQXdVObAdh8yRGcjAtQEkGtE7NonRRNs0pYaRNgfZgOwfqDuCzRzMAabFpXdnv/MR8yr3lXQFuTWtNr2MsjE7n0umXc8mCj5CfmN/9lxAKApK7fidtLX5i4iIxET3fzTe1N3Gg4QDFzcVkxWcxM2Um6bHpXe/6m9ub2d+wnwONznWwv2Y7++t2U+n3Mjsmg1VzPsylM1ZTmFwY/sAdo8HXwMu7X6N+V4C2PdF0lEYCBhMTJDI9gCstgCvNT11mMXvNNg40HKDMW0ZRTBHfWPkNlk9Z3us1rStex/y0+b2uz0mntcEJ6nY9Dt7qfpsFgrZXdigqIoKBkjeNjY0kJyePUEePQ3w6ZM51vrLmQkIm0NlhC80VTuBbvbN3AGwteCqh4YjTNhyxKTDvCpj/fphxvnOuw6/C9n/Czsegpfe/se46MhdS97GjAcnbG19n1cUXDOEFS3fd/wb7A0FqvT3vWWj2+dlX5QSEe6o8lDe09vhNx0RGMDPTzexsN7OyE5maFo+r2/9vcdEukmKj6Iu1lsO1LV2B595KJwM60JUU7YpgRmZCV4Z0ano8Ua7jnxehtT0QyvA6GdjiuhaC3Trg6whwoNrrBMIhWYkxXdnZjroy5s+d0+/xI12G6RlOf1Piw/u7BFDnbWdvZTOHa1vwB/sfkcgIw7T0eGZlJ5KWcPT49d52Z0yrmrlkXjZZSf2X7o0WY8xma+2yXtsVIIdHAXJv1lrWl67nN2//hm2127q25yTkkJ+Yj8scffe5tXIrHbaDG6d/ngsTV+Ota8fT7uGZQ8+wr2Ef6S25zK5eToQxzD83j7krctmzrYR3X9xJhC8NbIBY7ybiGp7C5a9hSr0lqxEaFl9A6ZwPUVkN2dOTmF4UTV5sDZFl+2nbu5+Gg5XU17TjiUzDm5CDNz6HFncOARNFXvAQswJbSaSp12vraG2hobyFithZVGcuxZuQQ2LTIbKq3yLJtx1vWhtRM2cyZe6FRGTNpSUyBdstm+SKjKBgVhJJ6TGdG3C5Eyj3lPPs4Wd5o/yNHhnawbiMi4jmCM6ffz4zmqooeuYOUtPnwIL3w4L3QcYsJwhoLoeqnXhL38S/6wmo3ApAZPI04usO0xJMoS7hHBpSziXY1gqeKvBWEujw0xg5l7qI+dR7k2jzWSJjXKRNiSc1J4HUKfFERh39fZoISEqPIy03gcS0WCeQ7gxCqndB+btOEFGyEYD2xByim8shMQfO+TIs/ThEDfwfobWW1uYO6sq91JV5qS/3dv3s8zpjl5abwMylWcxcmklaTkK/H891BDt49eVXJ/+/4dYGJ/ir2e1kQzsF/XDoFdj3PAQ7ICkfMorwBy0t7QFa2vzO93bne5s/2OvQcVEuCtLiyEyMJaKPYayrryctNXX0XttAOq+t2n3Oax2IKxrSZ0FCBhhDwFoaWjpodSURlTOflGmLicmZDwmZtHZYDtZ6OFDtwVqYkelmRoabuOgIiE4EVz8fsgYD0NaEryPII2+Xcs/6g1Q29/z4OkgEHnqWYuSlxDEr2wmYspNimTgfJkO6O5oL5mb1GySOBX8gyKHaFvZVNVPW0HM8t+7aiz8hk72VzRyo9tIe6H0Nd8pPjaMgtWcA7Gnzs7/KQ3Nb39ePMbCiMI0rFuewauEUMt0xvFvSyJNby3lyWznFda1dbfNS4ihIiyNygJuWW9r97Kvy0OQbws3QYZqSFMu0YwLuzgB3dnYis7LczMpKJDn+6O9SMUz/FCAPky6uo44NjHMTcvn4go+zOGMxM1JmkBDVszTgwNvVrH3gbdqaISLQ90c2ES5Dw4xD/CvxHqbm5HLlzCtZ98gv+fz9TXimzSFwzXfY+WY9wUCQ1EUu4grbaX8pSHVDFNFtDUxp2U1ddD6ehDwA4lsq8cWmEow4+k41PsGQlptIWoFTyrDj1XL87QFmLctm8QX5+LxOMFZf5qXyUBP1FS1gIDsrgvT4Vqqa46mpcd6dJ2fF0dLYTkdboPeL6SZzaiIzl2ZSdFoWyZlDr1sMBi37NlWy87Vy6mrqSU6IgYqtThAQEQltoZvtImOcwCHYrT/RbohPg/h0AkTRWOWlrbX/Pyqxka2kmQOkRR4hKTmIJ3oW9R051DW58Xr7/1MeGQWp0ZWksY80s5+0yGJSI0uIzJoOc1bB7NWQOg2ObIDX7oLSTQTjs2mMXUxdawZ1rRk0+NIIuuIhKg6i4gm6YmmsDeBrOfp6ouNcpOW4ScuJJy3XTcG8NNJy+5nForXeqR/tYli3aRsrL7ioZztPNex8FPY+C9EJkDnPyU5mzoWYxH5fM8EANByGqlC2smYPxKVC1jzInOMcJzapz13fLW3kgTcOke6v4JSYcmbaErLaDhNsb6G1I4CvPYDPHyAm0kVafBTu2CiMDUJjsfMGqD/JBTD/KvZnXcL/7Ehk8+EGKrrVHMZGRVCU5WZ2ViJF2W5S4o7+22j3B/j75hK2lzUxLT2emy8ooijLzb5KT1fWrKyqdsAM8sxMN6sX53DWzPRemTJvmx9vP4EJODndikZfV3Zub5WHptaebyBT4qOZmxnDqQm1zI4oJaKtkaomH5XNbVQ1+fC6konKmU/m1LkUTUmlpL6VJ7eW8/zOSrztR68jY6Ag1fm3WFzfwrF/Ao1xAqysxJ4BbHJcFEXZTuAxO9vN28UN/PrF/VQ0+VhemMqVp+TiGiBg2rRtF4GETPZUethf7aG9jzcp4y3aFcG5szJYvSiHeTlJHKhxMpb7qpqpamob/ADdxEW7QhlbZ7zyU+N7vPFq8nWE6lGPlgIcrBk48C1Ii+u6fvNT43F1e0McG+VkiIuy3CTE9P3GxlpLRZOPPZUeyhpae/zuyxtbeWpbBfuqPBgDGe4YqpvbiIwwnF2UwaULspmfk8Ss7ETc/Ry/r/NVN7c5Nb19XGtDEeUyzMhMoCgrkeS4ob+JUQzTPwXIw6SLyxG0Qe58/U7+sfcf5CbkcsPiG7hy5pVEuY7+g+2orML7yitE5uWxoyyJN58tJybJUpjgwW58hsSqg9hgDQltAcy82eTeeDNJ555JVEwkaw+v5bsbvkvurhq+/pAleupUiv5yP5FpaXgb23jruSNsf6kUf0eQ+ORoll6ST37dFlpefoHovDx8ubMp8+dQ540leUoiabkJpOYkkJYTT0x8z/9UWj3tvP1cMe+uK8HfLdCNT44mI89N4eIMZizJJCH5aB10Y3Ur+7dUUb6/kaT0WOfYuQmkZMUT4Tr6n3VbSwcH3q5h/5YqKg86GeqhBMudgfGbTxyiobKFlOx4OqyX1NaD0NEKeac5waS/zfkovbXBCZKj4p1ALzoBuv1OTIQhOSOuazxS01244o7W/EZEGKLjIqG50gkYdz8FVTu6ArL2YCzBwvPhrC9BwXKCAUtjRSP1Lz1M3a7d1NmZ1AUL8fr6ryPuT7SrjdS4WqKCTd2CWkuSq4q0yOLQ1xHio1sxmbOc4DNztvMGoGvA/FB/CKp2OR+xt9T2HlMTRUTWXCeATZsOxW84WVcbhNTpTtDbeGTI/SfaDelFTlDecHjIu7fYGPbZXJqsc024IgyxUS5a2wMErSU6MoL0hBhi0vKIzV1AyrRFRGbPg7iUHsfZVmv4+fP7WLuzkqTYSC6el82sUHAyKyuR/NQ4IvpKDYdYa3l+ZxU/f34P20qPfqrS+dE07R5S+8kgB4OwtbQRT5uflPgoLp2fTUp8dFfgU9rQ2ud+fen8aDrd3W0WFgtVzW0cqvH2+hg32hXB9IwELJYD1T2fT42PYtWCKbxncQ5TkmLZV+UEfHuqnDeWs7MSQxld51rqHrDVt7T3OH+dt71X5nJFYRpfungWZ85MH/TmomNLBLyDvLkea/uqPTy5tZyntpZT1nj0jZUxMC0tnpzkOIYyy2NTq/9o7esAOt+wzMpySh86SxLyU+OI6Damb7z+CpdeNPolKnsqm3ni3XL2VXk4f05m17U82SmG6Z8C5GHSxeUEx995/Ts8vPdhPrXwU3z+1M/3CIyttTT+8xEqf/hDOryt7JrzESqzVzCl/l1mHfgbUc0NxJ9xBjGf/Rg7pvg55c066u6+G39ZOdEzZ+IKZaf8QT/tO3YSM62Qaff+mci0tB798Da2UXW4mYJ5qT0+9j9erZ52infUkZjmBLyxCSP7EWNTbSv73yxj/zv1XcFyWm4CMXH9ZyE8DW001/pIz0tg+XumM+PUTEr++DEKSh6Dq/8C868c0T72q7UeqvfAkdfg9V+Dtwqmnw9LPgqv/MwJok/9KKz6PsSl0NbSQV15Cw2VXgL+/v9vMQaSMuJIy0kgPjn6aHDhazqakQ10y1gFA6EAOJSxberjxpCYpFD2NhQERyf02L/43VcoiPU6JSCNxU5Qu+D9MP99kL3A6VSbxylfqN4D/kGCuuQCyJzLpvp4/vTaYSIiDPPSXZwaV8kMSvG1eJwMZ1MbZQ2tHK5rISHaxTmzMjhjRjoxaQX4Umazrz2FfdUtJMVFMisrkbwUJ5D1tvl5YVcVT7xbzou7q7rKIiIjnI9Su2eR2vxBtpY2khQbyWfOncH1Zxce90fl1lpe3VdLa0eAWVluCkK1moP9H+jrCPDynmqe3FrO2p1VtAeCzMx0hz7udZPmHjjISE+IZlZ2ItPS4onsp1az3R/kUK2XvZUeIgzMyk6kMP1o+45AkEM1XvZWeUiKjeL0GWnDqvs8lj8Q5HBdC3srm0l3x7BsWmrYd91Plr8hwaDl7ZIGiutaKMpyZieIPc7/ZztvGttb1Ux5Y8+SifhoF0WZiczMSiA+evCM7GQZv4lK49c/BcjDdLJfXN2D488Xfpyrq6cTM20a0UVFRKam0lFZSfm3voX3pZcxy8/l3RkfpbrSz6KCBma2bKHq8CHmfPGLxC/veYOUbW+n4Z+P0Pzss9hu5QGRaelk/9c3egXHk07pFnjpR7Dnacg5hebCq9nfdg7Fhw3BAT5KdEVFMO/MHGacmunU9+58HP72ETj9Jrj8R2P4Arppb4HNf4JXfu4Eyom58N7/hdmXjkNfvBDofpOOcW4yHCBY6fFvuMPnZNz7aF/raePlvdXER0cyK8vNtPSEHrWMnTYdquPna/fyyr4a0hKiiY92UVLfO6iOi3IxK9vN6kU5fOyMaf1+/DsQX0eAfaEbWfZUOlMltXb0LFk4fXr6sALjwQzl/0B/6NruL9A9GZ3sf0OGS+M3PBq//vUXIGuaN+nl9bLXuX/n/eS4cyhKKWJG8gweO/AYD+99mM8tvIFV//MGFZv/2NXelZ6O9fkIBgJ4P/M93irNIFgf5LIbFjJzaRbwAfauW9crOAYw0dGkXnM1qddcPYavcAx0D4zjUmHFjVC6mcTX/4tTgVOz5kNCt3pO43I+9u+sYU0phIat8MYuJ2u64180JRaRdMmd4/SCcOZ+PfNmOO2TcGg9FJze62P+setLAjD4KnrWWl4/UMu9rx2isc7Hdruv66PchJijGeq2jiDr99bw5NZyXj9QS6Dbx/TRkRHMyEggMfbof5eetgA7y5vIcEfzX6vn8ZEzphIfHYm3zc/+aqfGNDkuqkdGeDhio1wszEtmYd44ziIxBAqMRWSyU4AsPeyo3cEXX/wicZFxbKzYSIu/peu5m065iQ+90Ebt5s1M+fa3iMovCK2Uto8Wn2F39nso3tdCTpGbCz8+j5Ssk3Ay/WAQXvwerP+pExhf9C1YccPRG74ajjiLDhxY1zMD6m936n7f+kvvY8ZnQP4ydmRcyxmR/c8LPWai4515YMdJVbOPNw7Ucfr0tH6nBLLW8vr+Wn6+di8bD9WR4Y7B+oNseKbvRVk6Tc9I4KbzZ3DZghyC1rKnspl9VZ5etZTpCS6+sXouHz1jWo+PhxNiIlmcn8Li/JQRea0iIjI+FCBLlwpvBZ9//vOkxKRw/3vuJz02nQpvBfsb9xNhIli0t4OSu28i8L5P81LFXIKlQWARpCyitsRD8IiPc66exeKV+b3mzz0ptHngnzc6c9Au/Tis+kHvmRBSpsJZtzhfffHWODWyDUecGtesec50VYBv3brR7f8YqGzy8dTWcl7YXU1re/+zGhgM+alxXTftFKTFs/FgLY+/W87GQ3VY68wqcMeV83nfqXlddaDHBsbZSTF858oFXLO8gA2vrmfZmeewt7KZ/dVefB09ZzZYUpDKvJzEHjWlpxSkjNpYiIjIxKUAWQDwdni5+fmbafW3ct/l95ER5wRlOe4cctw5dFRUcPA/30/EnIW8wRnYci+pOUc/4p66MJ3Tr5xxcmaNARqKYc11ULUdLvsRnH7jgPWw/UrIgIRzRr5/46jdH+Rvm4p59O1SNh2ux1ooynKTldh/NtwftLy2v5aH3yrtsX1WlpsvXDiL5YVp/GztHm792zs88W4533//IvZXeboC4ylJsdx51QKuXlbQ4wYjd0wkS6amsmTqOM3nKyIik4ICZMEf9POVl77C/ob9/PqiXzMrdVaP521HB6Vf/g9seztHLv0qLds8fPBry8ie3vc8ryedfWvhn58Dvw/+7e8w6+Lx7tGQ+QPBIdWNetv8xEe7Br2Df3tZI1/5+7vsLG9iTnYit148m9WLcijKcg+4X6fOuVIP13pZmJfM7OyjGfkzZ6bzp1cP8uNndnPuj16kPRDsNzAWEREZCgXIJzFrLS+XvMyv3/k1O2p38K0zv8VZeWf1alf7xz/RumUL9mv/w56NHpaumqbgGMDXCM/8l1M3nDEHrvmLc4PdJOLrCPCztXv4w/qDFGU6My28Z/EUirJ6L5JR3tjKk1sreOLdMrYcaSAh2kXRMXOXFmW5yUuJwx+0/OrFffzqxX2kJkRz98dO49IFU4bcv6TYKE6blspp03pnfF0Rhs+cO4ML52bxu5cOsCAvSYGxiIiMCAXIJ6FjA+N8dz4/OOcHvHfme3u19dfVUXv33cRcsIqX9ySRmhPJiiumj0OvR5G1zgIVgyx93MO+tfDoF5yFNM65Fc6/bWj7TwBvHannK39/h/3VXq5YnENVUxs/f34PP1u7h5mZCaQnHC2B8LT52VHuzOE8LyeJWy4sotnnZ09lMy/tqeahzUfnJY6PduGOiaSquY33L8nj2++dP6oT7c/IdPOjDy0eteOLiMjJRwHySeje7ffy080/Jd+dz51n3ckVM68gKqLvuVNrfv0bgj4f+xf+Gy3bvVx+0yJcUZN8CidfI7z1f1C5w1l1rXoPdLTA2V+AlV935scdaN9nvuHsnznXWbQj/7Sx6/tx8HUEeOydMlq6Lbd7oNrDXzYcZkpSLPd9agXnzc4EnJvont5WwbrdVfg6js7TnO6O5qur5nD5winMyOxdHtHQ0t5jydjShlY+fFr+cWWNRURExpsC5JNMR7CD+3bcx+k5p/Obi3/Tb2AM0H74MDUPPkz15beyZ6uHpZdNI7twkpdW1B2A+691VktLyHJKIk69zlmq+ZWfOVOtve/XzlLOx9r7HDz2xUmVNd58uJ6vPvQOB6q9vZ67dnkB33jPvB4LS2QnxfKJswr5xFmFQzpPSnw0ywvTWF44yRd2ERERQQHySWft4bVUt1Zzx1l3DBgcd7QFeO3Hj7Jn+bfp8LgpXJTOivdM8tKKg+vhwY85P3/iMZh+Xs/nF18Dj30B/nCxM3dx2syjz5VtgXfWOFnja/7SdwA9gfg6Avz02d384ZWD5CbH8edPLu8xN2+Uy5A4SiuuiYiITHYKkE8yf935V6YmTuWcvN5TiVlrqT7SzL7NVex8uRiffwHZKR7O/txScopSxr6zI2nTn+DJrzhB73VrIH1m7zazLoZ/f9258e6N3/Z8zrjgnC/DytsGLsEYIzWeNnZXNLO3spk9VR4O1Xjxd1v9rbS+ldKGVj5y+lS+vnoe7uNY3lhERORkpb+aJ5FtNdt4p/odvrbwG+x4uQx7NJ6iudbH/reqaKrxERFhyGg/zKIjz3Law7/H5R58Sd8JqWonbH8EdjziLL5RdDF86I8QO8ByvbHJcNVdsOr7EOi2kEVkDMSENzXZaKluCfK7l/bz5NZy3ilp7NqeHBfFjMwEYiKP1oYXZbn5fx9azNlFGePRVRERkUlNAfJJ5P6d9xMfGU/29oW89NqeHs9FRBjy56WybHUhmc27qf7ij5hyx7cnZ3C883F44XvODXgYmHYWvOd/YOknwBXmJT9QED2GgkHLk9vK+f36g7xT3ArsYnF+Ml+7bA6n5KcwK9tNpjtm0PmIRUREJHwKkE8SNa01PHXoKa6ecS2H/17HrOXZnHv10QVBIqNdRMU488ce+dQ3iczJIeWDHxyv7h6fljp46muw9e+QNR8u/zHMvxISJ99MCoGg5cmt5fzi+b3srfIwMzOBq+dEcctV51CQdpKuVigiIjJGFCCfJP6+5+/4g37O61jN275qFpyTS1xi77lp248cwfva62R84RZM1CS6iWvn4/D4rdBa50zVdu5/gGsS9b+bNw7U8s1HtrG3ysOsLDe/vG4JqxflsP7llxQci4iIjAEFyCeBjkAHf9/9d87OO5uat/0kpseSOyulz7YNf/87uFyTK3u8+yn420dgyiL42MPO90mopd3P/3t6N39+7RBT0+K7AmNXhMonRERExpIC5JPAg3sepLq1mv/KuYNd/6hn+epCTB9Bl21vp+Hhf+JeuZKo7Oxx6Olx2vEoxKXCZ16AyNFbse147a5o5udr95CaEM3s0JLM0zMTiHYdvaluV0UzX394K0fqWrj+rEK+dtkc4qP1z1NERGQ86C/wCSxog9z11l38fuvvOT3ndJIOF4A9yNwzc/ps3/zCCwRqa0m95uox7ukwWAv7X4AZKydscHzd7zfQEQhigCafv9+209Lj+dsNZ3D6jPSx66CIiIj0ogD5BNXS0cI3XvkGzx95ng/O+iDfWPENHvzOZvJmp5CUEdfnPg0PPkhkbg4JZ589xr0dhupd4KmAmReO6mmCQUvEEEsdOoPjKJfhH587h8L0eKqb29hT6eFQrZdgt3n2YqNcXLE4R1ljERGRCUB/jU9AVS1V3Pz8zeyp38PXln+Nj877KBX7G2msbmXZ6sI+92k/fBjva6+T+cUvYFyuse3wcOx/wfk+44JROXwwaPnxs7v5w/oDLM5P4T2Lcrh80RRykvt+k9GpMziOdkWw5oYzmJ7hTJeXlRRLVlIs58zS/MQiIiITlQLkE9Bv3vkNBxsPcteFd3Fu/rkA7Hy9nMgYFzOWZPa5T8NDD4HLRfIHJtHNeeAEyBmzIaVgxA/tbfPzpb+9zXM7Krl4XjYl9S3c+fgO7nx8B6dNS2X1ohxWHxMsH6lt4Ymt5fx+/YFewbGIiIhMDgqQTzD+oJ+1h9dy4dQLu4LjjrYA+zZXUbQ0k+jY3r/yrpvzLlhJVHbWGPd4GDp8cOhVOO0TQ9rN1xHgiXfLaekIdG2LijBMz0hgVnYiaQnRlNS38Jl7N7Gnspk73jufT5xViDGGA9UentxazhNbK/ju4zv4bihYXlaYyqv7athW2gTA0qkp/PTqUxUci4iITEIKkE8wGys20tDWwKrCVV3bdr1eTocv0OfNecHWVmp+89vQzXnXjGVXh694A/hbh1x//J3HtrNmY3G/z2e4o2n3B7HAnz+5gvNmH826z8h08/kLZ/H5C2f1CJZ/99IBTi1I4b9Wz+PyRVPIT9V8xSIiIpOVAuQTzLOHniU+Mp6zc50b7fZvqWL9g3vJKUomtyilq12wtZX6B/5G7R/+QKC2FvdFF5Fw1lnj1OvjtP8FiIiCaeHfVPjynmrWbCzm0+dM56bzZ3Zt93UEOFDjZW9lM3srPTT5OviPS+dQlOXu91jdg2VfR4DYqElUuy0iIiL9UoB8AukIdvD8kedZWbCS2MhY9m+p4pk/bCe7MIkrbj6la+7j5hdepPz22wnU1pJw1plk3Hwz8aedNs69Pw77X4CC0yGm/yC2u2ZfB7f9411mZibw1VVzegW0BWnxnD+77xrtwSg4FhEROXEoQD6BvFnxJg1tDVxaeGmP4Pi9Xzilq/a4ee1aSr50K7GzZ5P/y18Qv3TpOPf6OHmqoGIrXHh72Lv84MmdVDT5+MfnzlJAKyIiIv1SgHwC6SyvmNawoM/guOm55yi99cvELVhAwT1/wOUOL/M6IR1Y53wPs/64s7TixvNnsGRq6uj1S0RERCa9iMGbyGTQWV5xQc5FvLpmP6lT4k/c4Bic8oq4NMg5ZdCmDS3tXaUVt148eww6JyIiIpOZAuQTxJvlTnnFqcUX46lvY+W/zekKjr2vvXZiBcfdl5eOGLhU4oVdlaz6+ctUNbfxkw+fotIKERERGZRKLE4Qzx5+lpyOQurfMsw9I5uc0IwV1lqqfvo/ROXlnhjBMUDVDvBUDlhe0djSwZ2P7+AfW0qYk53IHz6+nEX5yWPYSREREZmsFCCfADqCHaw9vJb3l3yRyGgXZ36gqOu5lg0b8G3fzpTv3nliBMetDfDoLc70bkUX9Xq6c27iv2w4TI2nnVsuLOLzFxYRE6nMsYiIiIRHAfIJ4M3yN0kvn05sRTqnXzOD+KTorudq/3APrswMkq+8chx7OEJaG+D/PgDl78LV90JSLuBki/+y4RCPv1vOropmAFYUpilrLCIiIsdFAfIJ4PUjb3DWofeTnp/AwvNyu7b7du7E++qrZH75y0TExIxjD0dAj+D4Ppi7GoC1Oyr5xj+3UtXcxrJpqXzrivlcvmgKOclx49tfERERmbQUIJ8AWl5PIK09hfOvm0uE6+h9l7V/uIeIhARSr51kS0gfq3oPPHJTj+C4oaWdOx/bwcNvlTJ3SiJ/vH45C/OULRYREZHhU4A8yZXvbyR1z0zqZxwgp9tNa+0lJTQ9/TRpn/gErqSkcexhGEo3w8s/hdRCyJoLmXMhKh52Pwnb/+nclOeK7gqOd1U08bF7NlLvbecLF83i8xcUER2pCVlERERkZChAnsT87QFeuHcH3pgGos6u7/Fc3Z/vhYgI0j7+sXHqXZha6uBvHwNfE+x/Hvy+ns9PPRMu+xHMvwqScggGLf/5j61YC4/cfLayxiIiIjLiFCBPYhsePUBDVSsvzr+fj6V9oGu7v76ehoceIvmKK4iaMmUceziIYBD+eRN4q+HTz8KUxdBwGKp2QWudM41bUm6PXR7cVMw7xQ387JpTFByLiIjIqFCAPEmV72/kneeLyV4eTWnkHgoSCwAItrZSfvvtWJ+P9E9/apx7OYjXfwl7n4HVP4HcJc62tBnOVx/qve386OldrChM432n5o1hR0VERORkMqqFm8aYy4wxu40x+4wxt/Xx/FRjzIvGmLeMMe8aY1aPZn9OFP72AC/ct5PE1FiiznJKKwoSC+ioqODwRz6K5/kXyLrtP4kpKhrkSOPoyBuw9jsw/32w/DNh7fLjZ3fT5PNz5/sWYIwZ3f6JiIjISWvUAmRjjAv4FXA5MB+4zhgz/5hm3wQetNYuAa4Ffj1a/TmRbHz8IA2VLVz48bmUthVjMKQfqOPQh6+m/fBh8n/za9Kvv368u9m/ljp46JOQMhWu/AWEEey+U9zAmo1HuP6sQuZOmeA3HYqIiMikNpoZ5BXAPmvtAWttO/AAcNUxbSzQGe0kA2Wj2J8TQl25l3fWFjPvrBzy56ZR3FzMRcXJlF3/aUxMDIUPrCFx5crx7ubAXv6xs1T0h/8MsYPXEQeDlm/9axsZ7hi+dPGs0e+fiIiInNRGswY5Dyju9rgEOP2YNncAzxpjbgESgItHsT+TnrWWlx/YQ1SsizPfPxMA//ZdfPJvdcTMmU/B3b8jMi1tnHs5iJY62HwvLPwQ5J46aPMtR+r52XN7eKekkZ9fcyqJsVGj30cRERE5qRlr7egc2JgPAZdZaz8Tevwx4HRr7ee7tflyqA8/NcacCdwDLLTWBo851g3ADQDZ2dmnPfDAA6PS54F4PB7cbveYn7e7xiOWktcsOacZ0mYZImprifrB7ZjoaNq/fifBCTzfcef4TT38IDMO/pU3l/0vXndhv+33NQT4174OttYESIyC98yIZlVh5EldezwRrsHJTOM3PBq/4dH4DY/Gb3g0fv274IILNltrlx27fTQzyKVAQbfH+aFt3X0auAzAWvu6MSYWyACqujey1t4N3A2wbNkyu3IcSgjWrVvHeJy3U7vPz/1PbSBzagzv+/QyrNfDweuuo8Fv2f3d6/jI6ivHrW/hWLduHSvPPh02fhqKLmH5Fdf32/aJd8v53tNbSEuI5rbLZ/GxM6aREKMJV8b7GpzsNH7Do/EbHo3f8Gj8hkfjN3SjWYP8JjDLGDPdGBONcxPeo8e0OQJcBGCMmQfEAtWj2KdJa9MTh/A2tnPetbMxAT+lX7qVjoOH+OkHIsiYf+p4dy88b98PLTVwzpcGbPaHVw4wIzOB9V+7gJvOn6ngWERERMbUqAXI1lo/8HngGWAnzmwV240xdxpjOtOd/wF81hjzDrAGuN6OVs3HJFZX5uWd54uZd3YOU2YkU3fvvXhffZX6L17DtsKIrjmQJzQbgNd+CXmnwbSz+222s7yJt4408JHTlTUWERGR8TGqEYi19kngyWO2favbzzuA/qMlAWDruhIioiI4833OjXnNL7xI7MKF7D4rHzYzKQLkzOoNUH8QLvnOgNO6rdl4hOjICD64VAuBiIiIyPgY1YVCZGSU7WsgtyiZuMRogl4vre++S8KZZ1DcXExKTAqJ0Ynj3cWBWUtB8cPOCnlzr+i3WWt7gH9uKeU9i3JIiY8eww6KiIiIHKUAeYJr9bRTV+YlpygFgJbNm8HvJ/4MJ0CeDNljDq0nqXkfnHULRLj6bfbYu2U0t/m5bsXUMeyciIiISE8KkCe48n2NAOTOSgHA+/oGTFQU8UuXUtxcTH5i/jj2Lkxv/oGOyEQ45d8GbLZm4xGKstwsL0wdo46JiIiI9KYAeYIr29eAKzKC7GnOHMfeDRuIW7KEQEwkFd6KiZ9Bbq6EXU9QMeVCiIrtt1nnzXnXrZh6Us91LCIiIuNPAfIEV763gezpSbiiIvDX19O2cycJZ55BuaecgA1M/AD57b9C0E9Z7qUDNtPNeSIiIjJRKECewNp9fqqLPeQUJQPQ8sYbAF31xwD57glcYhEMwpZ7ofBcWuP772dLu18354mIiMiEoQB5Aqs40IgN2qP1xxs2EBEfT9zChV0B8oTOIB9cB/WH4LTrB2z26NvOzXn/drpuzhMREZHxpwB5Aivf14gxMGVGKIP8+gbily/HREVR0lxCjCuGzPjMce7lADb9CeLSYN57+20SDFruXn+ABblJLJumm/NERERk/ClAnsDK9jaQOTWR6NhIOsrLaT98mPgzzwBwZrBw5xNhJuivsLkSdj8Jp/4bRMb022ztzkoOVHu56fyZujlPREREJoQJGl1JoCNI5cGmrvmPva9vACDhzDMBKPZM8DmQ3/4/CPrhtE/228Ray29f2k9BWhyXL5wyhp0TERER6Z8C5Amq8nATAX+wW/3x67jS0oiZNQtrLSXNJRN3DuRgEDY7N+eRUdRvs02H69lypIHPnjuDSJcuRREREZkYFJVMUOX7GgDIKUrGWkvL6xtIOON0TEQEtb5aWv2tEzeDfOAFaDg86M15v123n7SEaD582gR9HSIiInJSUoA8QZXtbSA1J4E4dzTtBw7gr64m/oyj9ccwgWewWP8zcE8Z8Oa8PZXNPL+rik+cWUhcdP/LT4uIiIiMNQXIE1AwaCnf30huaP7jXvXHEzlAPrgeDr8C5355wJvz7n75AHFRLj5+5rQx7JyIiIjI4CLHuwPSW22Jhw5foKv+2LNuHVFTpxJd4ATEhxoPEWEiyHNPwFXn1v23kz1e+ok+n67ztrO1tJF/vV3KR06fRmqCFgYRERGRiUUB8gRUtrcBgJyiFALNzXjfeIO0j32s6/k3yt9gYfpColxR49TDfhx82ckeX/7/ICq2a/OeymZ+tLGV/1j/HLXedgDcMZF8+pzp49VTERERkX4pQJ6Aaks9xCVGkZgWS+Nja6Gjg8SLLwag3lfP1pqtfO6Uz41zL49hrZM9TszplT1+amsFu+qCXLM8m6IsN7OzE1mQm0S6u/8SDBEREZHxogB5AmqqaSU5Mw6A5rVrcWVmEHfqKQC8WvYqFsu5+eeOZxd7O7QeDr8Kl/+4R/YY4HCdl9RYw39/cPE4dU5EREQkfLpJbwJqrGklKTOOoM+HZ/16Ei+8CBPh/KpeKX2FtNg05qfPH+dedtMje/zxXk8fqW0hM06r5ImIiMjkoAB5ggn4g3jq20jKiMP72uvYlpau8oqgDfJa6WuclXvWxFpi+uBLTvb4nC/3yh4DHK5rISt+AvVXREREZACKWiaY5lofWEjOiKN57VoiEhNJOH0FANtrtlPfVs85eeeMcy+7CXTA01+H5II+s8ct7X6qm9vIilcGWURERCYH1SBPME01rQAkpkbjeeEF3Oefj4l2pkJbX7oeg+Hs3LPHs4s9bfw9VO2Aa/7aZ/b4SF0LgDLIIiIiMmkoaplgGqudADmqfC+BhgYSL76o67lXSl9hUeYiUmJTxql3x2iugBd/gH/GxWyMObPPJodrOwNkZZBFRERkclCAPME01bTiioog8NoLmOho3Oc6s1XU+erYVrNtYpVXPHs7NtDGN3wf5Zrfb6C6ua1XkyO1yiCLiIjI5KKoZYJpqvGRlB6L5/m1JJx1FhEJCQC8WupM73Ze3nnj3MOQQ6/A1gfZlP9xHjwQjbWws7ypV7PDdV6S46JIiFIGWURERCYHBcgTTGNNK+5YP/6ychIvubhre+f0bvPS541j70ICHfDEV2hLyOP6veewck4m0E+AXNvCtPT4se6hiIiIyHHTTXoTiLWWpppWUuOrICIC9wUXABAIBnit7DXOzTt37Kd389ZC9S6o3gnVu6Fqp/PYW803I28jMzWFX163hEv+52V2VTT32v1IXQuL8pKB3sGziIiIyESkAHkC8Xk76PAFiKrdTdzixUSmpQGwvXY7DW0NY1d/XP4OPHu7MzuFt/ro9uhEyJyDnbWKu4tz+VflKTz8yaUkxkYxLyexVwbZHwhSWt/KFYtzUIAsIiIik4UC5AmkqdoHQOTBrSRcc3Qp6bVH1hJhIjgr96zR70QwCI/cDM3lMOcyyJwHmXMhcw4k54Mx/H1TMT/c8C7fe98CFuYlAzAvJ4n1e2to8weIiXQBUNbgwx+0TEtLAO/od11ERERkJChAnkAaa5wZH+Jaqkk425nruLGtkQd3P8hFUy8am+ndtj4IlVvhg/fAog/12eTZ7ZXkp8bxkdOndm2bm5OEP2jZX+Vlfm4S4NygBzA1PR6fAmQRERGZJHST3gTSmUFOiPQRt3gRAPftuA9vh5ebTrlp9DvQ4YPnvws5p8KCD/TdJBDk9f01nDc7E2OOzkwxPycR6HmjXuccyLpJT0RERCYTBcgTSGNNK9F+D0mnn4aJjKSxrZG/7vwrl067lNmps0e/Axt/B00lcOl3IaLvS+OtIw142wOcNyujx/bC9ARiIiN6BMhH6lqIjowgO7H3CnsiIiIiE5UC5AmksaSeOG9lV3nFvdvvHbvscUsdvPxTmHUpTO9/ruX1e6uJMHDmzJ4BcqQrgtnZiT1msjhc62VqWjwREZoDWURERCYPBcgTSGOlh9jWWhLOPpsGXwP377qfS6ddyqzUWaN/8vU/hbYmuPiOgZvtreHUghSS46J6Pdc5k4W1FnBKLKamqbxCREREJhcFyBNEwB+kpc2FO6ad6IIC7ttxHy0dLWOTPa7ZBxvvhlM/AtkL+m3W0NLOuyUNnDsrs8/n505JotbbTnVzG9ZajtQpQBYREZHJR7NYTBBNFc2AIaUw82j2uHCUs8fNFfDq/8KmP4IrGi74xoDNX9tfS9DCebMz+nx+Xo4ze8XOimYiIgwt7QHdoCciIiKTjgLkCaL6zR0AZCyZxUN7H8Lb4eXGxTeOzslaG+ClHzmBcaADTrkOzvsPSM4bcLf1e6tJjInklPyUPp+f120mC3eMc2kpQBYREZHJRgHyBFHz9j4gj6xzlrD/4PPkJuSOXvb4if+A7f88GhinzRh0F2stL++p4ayidCJdfVfmpMRHk5Mcy87yJrKTYgCYmpYwol0XERERGW2qQZ4gGg5VE2H9JOamUeGtYErClNE5UekW2PYQnHMrvO9XYQXHAAdrvJQ2tPZbf9xpXk4Su8qbOVzbgjFQkBY3Er0WERERGTMKkCcAf309Hg+4YwOYCEOFt4LshOyRP5G18Ny3ID4dzv7ikHZdv7cGgPMGDZAT2V/tYW+Vh5yk2K5lp0VEREQmCwXIE0DLG2/QGpdB8hQ3QRuksqVydDLI+9bCofVw/m0QmzSkXdfvrWZaejxTB6kpnjvFWXL65T3Vg7YVERERmYgUIE8A3jc20hqXScr0TOp8dXQEO5gSP8IBcjDgZI/TZsBp1w9pV2d56VrOndX37BXddc5k0ezzM031xyIiIjIJKUCeAJp2HSTgiiE5M4FKbyXAyGeQ31kDVTvgom9BZPSQdu1cXvqcooHLKwAK0+OJiXQuK2WQRUREZDJSgDzObCBAQ0k9AEmZcVR4K4ARDpDbW+CF70PeaTD/fUPe/bF3yoh2RXDmzPRB20a6IpgzxZnuTVO8iYiIyGSkAHmctR85QotxA5CUEUtFywgHyEc2wP1XQ3MZXPJdMGZIu9d62nhwUzHvX5LX5/LSfZk3xSmzUImFiIiITEaDBsjGmPcaYxRIj5K2XbvwxTq1vUkZTgY5xhVDakzq8A58ZAPcdxX8cRVU74LVP4HCs4d8mHtfP0ybP8hnzwtvOjiAFdPTcMdEUpihDLKIiIhMPuEsFHIN8HNjzD+AP1prd41yn04qvp278CTm4U6NISra5UzxFp+NGWKmt4d3/w4PfwYSMuHS78GyT0H00LO5Le1+7nv9EJfMz6Yoyx32fh9YmseqhVO6VtMTERERmUwGjWCstR81xiQB1wF/NsZY4E/AGmtt82h38ETn27UTT+oqpkxzyhJGZJGQI69DbDJ88Z3jCow7/e3NYhpaOrjp/JlD2s8Yo+BYREREJq2wSiestU3AQ8ADQA7wfmCLMeaWUezbScGz+yAtkalkTnVubKtoGYEA2VMJibnDCo47AkH+sP4gywtTOW3aMMs9RERERCaRcGqQrzTG/BNYB0QBK6y1lwOnAP8xut07sflramhoc5ZizpqWSCAYoLqlmuz4Ya6i56mExOEd44l3yyltaOXG84aWPRYRERGZ7ML5HPyDwM+stS9332itbTHGfHp0unVy8O3aTXPiVAAypyVS3VpNwAaGn0FuroRpZx737tZafvvSfmZlublwbtbw+iIiIiIyyYRTYnEHsLHzgTEmzhhTCGCtfX50unVyaNu1kyb3VNwp0cS5o0dmDmRrwVMB7uPPIL+8t4ZdFc3ccN4MIiKGcbOgiIiIyCQUToD8dyDY7XEgtG1QxpjLjDG7jTH7jDG39dPmamPMDmPMdmPM/eEc90Th27kLT+oMsgqTAUZmDuTWegi0Q+LxH+OpreUkxkZy1al5x98PERERkUkqnBKLSGtte+cDa227MWbQtYqNMS7gV8AlQAnwpjHmUWvtjm5tZgFfB8621tYbY06qz/Obdx+gJX81mdOcG/RGZJlpT5XzfRgZ5Ff313DGjHSiIzX9tYiIiJx8womAqo0xV3Y+MMZcBdSEsd8KYJ+19kAowH4AuOqYNp8FfmWtrQew1laF1+3JL9jaSl2tk5jP6pzBwltBfGQ8iVGJx39gj5OFPt4McnFdC8V1rZwdxrLSIiIiIieicDLINwF/NcbcBRigGPh4GPvlhdp2KgFOP6bNbABjzKuAC7jDWvv0sQcyxtwA3ACQnZ3NunXrwjj9yPJ4PCN63siDB/G4CwDYU/IuB6oNW6u3kmSSeOmll477uNkV65gHvLHjEK2H/EPe/6XiDgCi6g+ybt3h4+7HsUZ6/E5GGsPh0fgNj8ZveDR+w6PxGx6N39CFs1DIfuAMY4w79NgzwuefBawE8oGXjTGLrLUNx/ThbuBugGXLltmVK1eOYBfCs27dOkbyvPUVlaxNLMCdHMnFq84D4HeP/44ZKTOGd55X34FdcPqFV0Js0pB3f3jNW2Ql1vJv77lgeKv5HWOkx+9kpDEcHo3f8Gj8hkfjNzwav+HR+A1dWMudGWPeAywAYjuDJmvtnYPsVgoUdHucH9rWXQnwhrW2AzhojNmDEzC/GU6/JjPfrp00J80nZ/rRRTjKveXMTps9vAM3V0JUPMQMvUzDWstr+2s5pyh9RINjERERkckknIVCfgtcA9yCU2LxYWBaGMd+E5hljJkeuqnvWuDRY9o8gpM9xhiTgVNycSDMvk9qnl0HaI3N6LpBrz3QTq2vlinxI7CKnjsbjiPA3VPpocbTxllFGcPrg4iIiMgkFs5NemdZaz8O1FtrvwOcSah2eCDWWj/weeAZYCfwoLV2uzHmzm43/T0D1BpjdgAvAl+11tYezwuZTGwgQHW5Dzh6g15lywjMYAGhVfSO7xiv7nPuvTxbAbKIiIicxMIpsfCFvrcYY3KBWiAnnINba58Enjxm27e6/WyBL4e+Thrth4/QFO1Mw9aZQe5cJCQ7YZjLTDdXQPb849r1tf01FKbHk5cSN7w+iIiIiExi4WSQHzPGpAA/BrYAh4CTakGPkda2ayfNiQW4E13EuZ0ppUdkFT0IlVgM/Rj+QJA3DtSpvEJEREROegNmkI0xEcDzoVkl/mGMeRyItdY2jkXnTlS+HTtoTpxGzoyUrm1dJRbDqUFub4G2Jkgcehb63dJGmtv8nKX5j0VEROQkN2AG2VobxFkNr/Nxm4Lj4WvcvJXWuEyypid3bavwVpAUnUR8VPzxH9jjBNnHk0F+LVR/fOYMBcgiIiJycgunxOJ5Y8wHjeb9GhFBr5eqI80AZE49OhVbhbdiZMor4LgyyK/uq2VeThLp7pjh9UFERERkkgsnQL4R+DvQZoxpMsY0G2OaRrlfJ6yWt96mOT4PgKypRxfyGJEAuTm0zLR7aAGyryPA5iP1Wl5aREREhPBW0hv6ihPSr5aNG2lJmEJcYhSx7qiu7RUtFZySecrwDn6cJRabD9fT7g9qejcRERERwgiQjTHn9bXdWvvyyHfnxNeycSP+9MtITIvt2tbqb6WxrXFkMsgRkRA/tEzwGwfriDCwfHra8M4vIiIicgIIZx7kr3b7ORZYAWwGLhyVHp3Agl4vrdu20XbhR0lPOVrrO3JTvFVBQhZEhFM5c9T20kaKsty4Y8JaeVxERETkhBZOicV7uz82xhQAPx+tDp3IWt56G/x+fMThHpUAueK4btDbVtbI2TNVXiEiIiIC4d2kd6wSYN5Id+Rk0LJxI4HoeNrbISG1jwB5OHMgAzRXDvkGvermNiqb2liQlzx4YxEREZGTQDg1yL8EbOhhBHAqzop6MkQtGzfCwuUAPTPILSO0zLSnAvKWDmmX7WXOtNYLcpMGaSkiIiJycgin6HRTt5/9wBpr7auj1J8TVmf9MdfeAiWQ0C1ArmqpIi02jWhX9PGfIOAHbw0kDi0Lvb3MmbFvvgJkERERESC8APkhwGetDQAYY1zGmHhrbcvodu3E0rLlLfD7CUydCyV+3KlHZ7GoaakhI26YNcDeasAOucRie1kj09LjSYqNGryxiIiIyEkgrJX0gLhuj+OAtaPTnRNXy8aNEBlJR2oOcEwGubWKzPjM4Z3AE1okZIgZ5G2lTSzMVf2xiIiISKdwAuRYa62n80Ho5/jR69KJqWXjRuIWLaLFa4mJjyQqxtX1XE1LDZlxwwyQmzsXCQk/g9zY2sGRuhYW5Km8QkRERKRTOAGy1xjTdeeXMeY0oHX0unTi6aw/jl+xAk+9r0f2OBAMUOurHX6A7Bn6MtM7QvXHC5RBFhEREekSTg3yl4C/G2PKAANMAa4ZzU6daFq2vAWBAPErluN9ua3HDBb1bfUEbGD4NcjHkUHWDBYiIiIivYWzUMibxpi5wJzQpt3W2o7R7daJpWXzJoiMJH7JEjyPbiE9z931XE1rDQBZ8VnDO4mnEuLSIDL8mTC2lTYyJSmWDHfM4I1FREREThKDllgYY24GEqy126y12wC3MebfR79rJw5/eQWRWZnYmFhamtp7TfEGDD+D7Kk8rineFqr+WERERKSHcGqQP2utbeh8YK2tBz47aj06Afkb6olMSaWlsd2Zia3bKnqdGeRhz2LRXDGk8orW9gD7qz2qPxYRERE5RjgBsssYYzofGGNcwDBWtDj5BBoacKWk4G1oA3pO8VbdUg2MUAZ5CAHyzoomglb1xyIiIiLHCidAfhr4mzHmImPMRcCa0DYJU6C+AVdqKp56J0DunkGubq0mKTqJGNcw6oCtDZVYDOEGvVLnBr2Fecogi4iIiHQXziwW/wncCHwu9Pg54A+j1qMT0EAZ5JrWmuHfoNdaD4F2cIdfg7yttInU+ChykmMHbywiIiJyEglnFosg8JvQlwyR9fsJNjU5GeSGNlyREcQmHF3WubqlemTKK2BoGeTyRhbmJdOtekZERERECG8Wi1nGmIeMMTuMMQc6v8aicyeCQKNTytCZQU5Iie4RlFa3Vo/AKnqdi4SEl0Fu9wfZXdGsG/RERERE+hBOicWfgG8DPwMuAD5JeLXLglNeAeBKTcGz04c79WhJg7WWmtYaMuJHKIPcz016v3/5AL9ff4AL52bxnsU5JMdF0RGwukFPREREpA/hBMhx1trnjTHGWnsYuMMYsxn41ij37YQQqK8HjmaQs6cfzdo2tjXSEewYuQxyHyUWGw7U8sOndjIz081j75TxwJvFRLmcDLZu0BMRERHpLZwAuc0YEwHsNcZ8HigF3IPsIyFdGeSUFLwN1T2Wma5udaZ4G/YcyDV7ID4dYhJ7bva08YU1b1GYnsA/bz6byAjDS3uqeXJrOd62ANPS4od3XhEREZETUDgB8heBeOALwHdxyiw+MZqdOpH4Qxlkf0wiAX9ln3MgDzuDXLwR8lf02BQMWm7929s0tnZw76dW4I5xftWrFkxh1YKhrbgnIiIicjIJZxaLN0M/enDqj2UIOjPIrTjZ2mPnQIZhBsgtdVC7F069rsfmX724j/V7a/jhBxYxL0e1xiIiIiLh0s12oyxQ34CJiaGl1Xmc0EeJxbCmeSvZ5HzvlkHedKiOn63dw1Wn5nLt8oLjP7aIiIjISUgB8igLNPRcRe/YRUISohKIjxpGLXDJRjAuyFvatenJrRVER0bw/fcv0jzHIiIiIkMUzjzIZ4ezTfoWqK/vmsHCGIhPju56rqqlamTqj7MXQHRC16byxlZyU+K66o5FREREJHzhZJB/GeY26UP3ZabjkqJxuY4OeU1rzfBmsAgGoHQzFPS8Qa+s0UductzxH1dERETkJNZvitEYcyZwFpBpjPlyt6eSANdod+xEEaivJ2beXDwNbT2meANnFotFmYuO/+BVO6Hd02sGi/KGVmbPHmZmWkREROQkNdBn8NE48x1HAt0n2G0CPjSanTqRdM8gJ2cezep2rqI3rBKLko3O94LlXZva/UGqPW3kpCiDLCIiInI8+g2QrbUvAS8ZY/4cWkGP0IIhbmtt01h1cDKzgQCBxkYiU1PxHGwjb3Zq13OeDg++gG94AXLxm84CIanTuzZVNvmwFnKTYwfYUURERET6E04N8g+NMUnGmARgG7DDGPPVUe7XCSHQ1ATWEkxMpb3VT0LK0Rv0uhYJGU4NcklogZBuM1WUN/oAlEEWEREROU7hBMjzQxnj9wFPAdOBj41mp04UgfoGANpincyxO/VoVnfYi4S01EHtvh7lFeDMYAHKIIuIiIgcr3AC5ChjTBROgPyotbYDsKPaqxNE5yp6bZFOCXefi4TEH+ciISWhBQ6PuUGvrEEZZBEREZHhCCdA/h1wCEgAXjbGTMO5UU8GEWioB7otM919kZCWGmAYGeTi3guEgJNBToyN1BzIIiIiIsdp0ADZWvsLa22etXa1dRwGLhiDvk16nSUWvqATGCek9swgx7picUe5j+/gJb0XCAEng6w5kEVERESOXzgr6WUbY+4xxjwVejwf+MSo9+wE0JVBbncRHRdJVPTR6aOrW6rJjM88vqWggwEo3dJrgRBwMsg5Kao/FhERETle4ZRY/Bl4BsgNPd4DfGmU+nNCCTQ0QFQUbW2WWHdUj+eqW6uPv7yiakefC4SAM4tFjjLIIiIiIset3wDZGNNZxJphrX0QCAJYa/1AYAz6Nun56+uJTEnB1+InNqFngFzTWkNG3HHeoHfgJef7MTNY+DoC1HnbNYOFiIiIyDAMlEEOLdOG1xiTTmjmCmPMGUDjaHfsRNC5il6bt4PYhJ43zVW3Vh/fHMgbfw/PfQtyl/ZYIAQ0B7KIiIjISBhoqoPO4tgvA48CM40xrwKZaKnpsATqG3ClpuLzdpCSHd+1vaWjBW+Hd2gZ5EAHPH0bvPkHmH0ZfPAPPRYIAShv0BzIIiIiIsM1UICcaYz5cujnfwJP4gTNbcDFwLuj3LdJL9DQQMzMmfi8fmK6lVh0zoGcFZ8V3oGayuCfN8HBl+CsL8DFd0CEq1ezMmWQRURERIZtoADZBbg5mknuFN9HW+lDoL4ek5JKe1XPGuTOZaYHzCA3lcPOR2H7I3DkdYiIhKt+DUs+0u8unRnkHGWQRURERI7bQAFyubX2zjHryQnGBoMEGhsJJqZDFT0C5JrWQRYJ2fIXePQWwELWfFj5dVj0IUifOeA5yxp9pCdEExvVO7ssIiIiIuEJpwZZjkOwuRkCAfzuVIAeN+l1llj0GyAXvwFxKfCpZyFzdtjn1BzIIiIiIsM30CwWFw334MaYy4wxu40x+4wxtw3Q7oPGGGuMWTbcc04UgYYGAPyxyUDPDHJ1azVREVEkxyT3vbO3BpLyhxQcA5Q3aA5kERERkeHqN0C21tYN58DGGBfwK+ByYD5wXWgVvmPbJQJfBN4Yzvkmmq4AOSYRoMdNemWeMqYkTOl/FT1vNSQMfY7kssZWzWAhIiIiMkzhrKR3vFYA+6y1B6y17cADwFV9tPsu8CPAN4p9GXP+emeZ6Q6Xk9GNPSZAznPn9b+ztwrcYc5wEeJp89Ps82sGCxEREZFhGqgGebjygOJuj0uA07s3MMYsBQqstU8YY77a34GMMTcANwBkZ2ezbt26ke/tIDwez5DOG7thA8nAvpIqIIFNb23AFe1kjA/WHWRx/OJ+j3duUyVldT72D+F8pZ4gAA2lB1m3rniQ1mNvqOMnvWkMh0fjNzwav+HR+A2Pxm94NH5DN5oB8oCMMRHA/wDXD9bWWns3cDfAsmXL7MqVK0e1b31Zt24dQzlv7cFDVAFTCmZSsb+SCy9ZiTGGlo4WPPd7WD57OSsX9XG8di+sa6Ng7hIKzgn/fC/tqYZXNnLRWUtZXpgW9n5jZajjJ71pDIdH4zc8Gr/h0fgNj8ZveDR+QzeaJRalQEG3x/mhbZ0SgYXAOmPMIeAM4NET5Ua9QEMDuFy0d0QQEx/ZVW9c5ikDIDcht+8dvc4MFyQMbRlqzYEsIiIiMjJGM0B+E5hljJlujIkGrsVZshoAa22jtTbDWltorS0ENgBXWms3jWKfxkygvh5XSgo+b89FQko9znuEvMR+apA9nQHy0GqQyxp9GAPZSQqQRURERIZj1AJka60f+DzwDLATeNBau90Yc6cx5srROu9EEWhowJWaQltLR485kEs8JQD936TXlUEe2iwW5Q2tZCXGEOUazfc8IiIiIie+Ua1BttY+CTx5zLZv9dN25Wj2ZawdzSB34E6J6dpe6ikl1hVLemx63zseb4lFo+ZAFhERERkJSjeOkkBDA5Gpqfi8Hb3mQM515w4wB3KV832IAXJZYyu5WkVPREREZNgUII8Sf0P/NcgDz4FcAzFJEBV+sGut1Sp6IiIiIiNEAfIosNYSaGiEpDT8bYEeNcilzYMFyENfRa+xtYPWjoBmsBAREREZAQqQR0HQ64WODgJuZz7izgxyU3sTzR3NYQTIQ5zBosFZhDBXq+iJiIiIDJsC5FEQaGgAwJ+QAtBVg1zaPMgUb+BM8zbUGSwaNQeyiIiIyEhRgDwKAvX1APhjEoGjGeSuOZAHzSAP9QY9ZZBFRERERooC5FHQmUHuiEoAhhAgBwPQUgvuoZVYbNhfS1yUiwx3zOCNRURERGRACpBHQWcGuSPCyejGhG7SK/WU4o5ykxSd1PeOLbWAHVIG+bX9NTyxtZwbz5+BK6KfqeNEREREJGwKkEdBVwbZOBnd7hnkPHfeAHMgD20VvY5AkG/9aztT0+K56fyZw+qziIiIiDgUII8CX00VRETQFoggwmWIinEBYU7xBmFnkP/06kH2VXn49nvnExvlGm63RURERAQFyKPi7V3raIqztHo7iE2IwhiDtZYyb9ngM1hAWNO8lTe28vO1e7l4XhYXzcseoZ6LiIiISOTgTWSoovYVcygTOmoriUmIB6DOV0ervzXMDPLgJRbff2IngaDl2+9dMBJdFhEREZEQZZBHWHn9EaZUtHNgClQ31Hetohf2FG8RkRCbMuA5XttXw+PvlvPvK4soSIsfqa6LiIiICMogj7itbzxOQRC8M6YQbGwjJssZ4jJPGRDmHMgR/b9vCQQt331iJ/mpcdx4/owR7buIiIhMfB0dHZSUlODz+cJqn5yczM6dO0e5VxNbbGws+fn5REVFhdVeAfIIq9jyGgXAORdfzzsPxNAS2QxAiacECCdAHri84pG3StlZ3sQvrluiG/NEREROQiUlJSQmJlJYWNj/zFjdNDc3k5iYOAY9m5istdTW1lJSUsL06dPD2kclFiPMv3MPvvhIzjrtfcT6EyjtKAacEovUmFTiowYoiRhkFT1fR4CfPrubxfnJXLEoZ6S7LiIiIpOAz+cjPT09rOBYwBhDenp62Bl3UIA8oiq8FWQfaaa9KJ84k0BkMIp9rbuw1g4+xRuEAuT+Z7C497VDlDX6uO3yuURoURAREZGTloLjoRnqeClAHkFvFr/O1GpIWryENm8HAFXBcvbU7xl8ijdrnWne+imxqPe2c9eL+7hgTiZnzQxvIRERERERGToFyCNo/+YXiAxC7rJz8YUC5LbIFtYeWUuZp4xcd27/O7d7wd/ab4nFr17ch7fNz22XzxuNrouIiIhIiALkEdT07lsAxC1chM/rB6AgI5d/7PkHHcEO8t35/e88wCp6xXUt3Pf6YT50Wj5zppy8RfYiIiIiY0EB8gip8FaQergOvzuWqLw8fB4ng7x02qlUtzrB78AzWNQ43929a5Dve/0QALdeMntE+ywiIiIy3p5++mnmzJlDUVER//3f/z3kdp/61KfIyspi4cKFI9YnBcgj5M2KN5lRbomcNwdjDG0tToB8zvQzutoMHCBXOd/7qEHeeLCOU6emkJMcN6J9FhERERlPgUCAm2++maeeeoodO3awZs0aduzYMaR2119/PU8//fSI9kvzII+QzcUb+GA1pF25HKCrBnnGlGnMSp3F3vq95LgHmJqtnxKL1vYA28uauOE8LQoiIiIiPX3nse3sKGsasE0gEMDlCn/thPm5SXz7vQsGbffOO+9wyy23UFNTw65dzqxdt99+O3feeWfY59q4cSNFRUXMmOHEOddeey3/+te/mD9/ftjtzjvvPA4dOhT2OcOhAHmElL7zOpFBiFvgpPd9Xj+RURFERrv42LyPsb50PTGumP4P0E+A/HZxA/6gZVlh6mh1XURERGRIfD4f11xzDffddx8rVqzg9ttvx+fz8Z3vfKerzbnnnktzc3OvfX/yk59w8cUXA1BaWkpBQUHXc/n5+bzxxhu99gm33UhRgDwCyj3luA9WAhC70HnH5fN2EJPgLGf4/lnv5/2z3j/wQTzVEJMMkT2D6M2H6wBYOlUBsoiIiPQUTqZ3NFbSW7t2LUuXLmXFihUALF68mKeffrrHfMPr168f0XOOJQXII2BT5SZmlltIdBOV59QZt3k7iE0Ib71voN9lpjcdrmdWlpuU+OiR6q6IiIjIsGzbto1FixZ1Pd6yZQtLly7t0SacDHJeXh7FxcVdz5WUlJCX1/uerXDbjRQFyCPgnep3WFYZQfyixV3vnHzeDmIThjC83upeM1gEg5bNh+u5YrGWlRYREZGJIz09nRdeeAGAPXv28PDDD/Paa6/1aBNOBnn58uXs3buXgwcPkpeXxwMPPMD9999/3O1GimaxGAEV9cXkVweJW3D0Yw6f1z/EDHJNrwzynqpmmn1+lk1LG6muioiIiAzbddddh8fjYeHChdxwww2sWbOG9PT0IR8nMjKSu+66i1WrVjFv3jyuvvpqFoTiqdWrV1NWVjZou+uuu44zzzyT3bt3k5+fzz333DPs16cM8giw+w/jClhiewTIR2uQw+Ktgmln9ti06VA9gG7QExERkQnF7Xbz2GOPjcixVq9ezerVq3ttf/LJJ8Nqt2bNmhHpR3fKIA+TtZbEAz1v0LPWDq0GOeCHlrpeM1hsPlxPhjuGqWnxI9pnEREREemfAuRhampvYu7+NjqS4rpu0OtoCxAMWGLCrUFurQNsrwB50+E6lk1L7XFHqIiIiIiMLgXIw1S++y2W77G0XnZ2jxv0gPAzyJ7OVfSOBsiVTT6K61pVXiEiIiIyxhQgD5P3vvsJRED8v13dta3N6weGECD3sUhIZ/3xadMUIIuIiIiMJQXIw+CvqSH2mdd4aZEhZ9q8ru1DziB7a5zv3aZ523S4jpjICBbkJo9Yf0VERERkcAqQh6Hu//4P4w/wzBkxpMUenYqtM0AOuwa5K4N8dJq3zYfrOaUghehI/YpERERExpKmeTtOQa+X+vvXcPiUKdip8USYo4Fs22AZ5LfXwHPfAqzzuN0LEVEQmwJAS7uf7WVN3HT+jFF8BSIiIiLSFwXIx6nhoYcINjXx4vm55CT0nBh70BKL3U+CDcD8q45uy14IoZv83i5uIBC0WiBEREREZBwoQD4OtqOD2j/fS/yyZWzOKOPMhAU9nm/1dBAV48LVX3lEzV4oOB2u+BkAzb4ONhyoY8+L+9hX5WHLEecGvaVTdYOeiIiInNiefvppvvjFLxIIBPjMZz7Dbbfd1me7T33qUzz++ONkZWWxbdu2Ue2TClyPQ9NTT+EvLyf509dT3VJNjjunx/O1JR5Sp/SzuEcwAHX7IWMWzb4Ofvn8Xs750Yt89r5N/PiZ3Ww4UMu09ARuv2I+yfFDWIlPREREZJIJBALcfPPNPPXUU+zYsYM1a9awY8eOPttef/31PP3002PSL2WQj4Nn/StEZmXhWTYHe8SSm5Db9VwwEKTyUBPzzs7te+eGwxBoZ21VEv/xoxdpbO3g4nlZfPqcGSzMSyIxVkGxiIiIhOmp26Bi64BN4gJ+cA0h5JuyCC7/70GbvfPOO9xyyy3U1NSwa9curLXcfvvt3HnnnWGfauPGjRQVFTFjhnPf1bXXXsu//vUv5s+f36vteeedx6FDh8I+9nAoQD4O/ooKogoKKPeWAzAlYUrXc7WlXvztQabMSOpzX2/pLhKA32yLYPncVL540WwW5WsqNxEREZk8fD4f11xzDffddx8rVqzg9ttvx+fz8Z3vfKerzbnnnktzc3OvfX/yk59w8cUXA1BaWkpBQUHXc/n5+bzxxhuj/wIGoQD5OHRUVhK3cCEV3goAchKOllhUHGgEYMr03kFvY0sHDz71PJ8FvnD1as5fMndM+isiIiInqDAyva3NzSQmJo7oadeuXcvSpUtZsWIFAIsXL+bpp5/uWlUYYP369SN6zrGkAHmIrLX4KyqIvPjiPjPIFQcbiU+KJjE9tsd+jS0dfOSeDXy0+SDtcakKjkVERGTS2rZtG4sWLep6vGXLFpYuXdqjTTgZ5Ly8PIqLi7ueKykpIS8vb5R6HT4FyEMUaGjAtrcTNSWbcu9B0mLTiI08GgxXHGhiyozkHu+gOoPjPRUeVuU2Ex07Zzy6LiIiIjIi0tPTeeGFFwDYs2cPDz/8MK+99lqPNuFkkJcvX87evXs5ePAgeXl5PPDAA9x///2j0ueh0CwWQ+SvcMoqIrOnUO4t75E9bmlqp6m6lexj6o9/vW4fu8qb+d3HTiO15RBkFI1ll0VERERG1HXXXYfH42HhwoXccMMNrFmzhvT09MF3PEZkZCR33XUXq1atYt68eVx99dUsWOBMn7t69WrKysp6nPPMM89k9+7d5Ofnc88994zY6+nVr1E78gmqIxQgR03JpuJgBYXJhV3PVR4M1R/P6Fl/fKDGy8xMNxdMiwZvFWTMHrP+ioiIiIw0t9vNY489NiLHWr16NatXr+61/cknn+zxeM2aNSNyvnAogzxE/spKAFzZ2ZR5y465Qa+JiAhD1tSehfDlja3kpMRC7T5nQ/qsMeuviIiIiAyNAuQh6qioAJeLlqRoWv2tPQLkyoONZBS4iYx29dinvMFHTnKcs4IeQIYCZBEREZGJSgHyEPkrKonMzKTCVwXQtYpe5wIh2ceUV/g6AtR628lNjoXavRARCamFY91tEREREQmTAuQh8ldVEpWdTbnHmeKtM4Pc3wIhFY0+p11KHNTsgdTp4NJqeSIiIiITlQLkIeqoqCRyypRecyD3t0BIWWMrgJNBrtmnG/REREREJrhRDZCNMZcZY3YbY/YZY27r4/kvG2N2GGPeNcY8b4yZNpr9GS5rLR0VFc4MFt4KoiOiSYtNA/pfIKS8IZRBToqGuv2a4k1ERERkghu1ANkY4wJ+BVwOzAeuM8bMP6bZW8Aya+1i4CHg/41Wf0ZC0OPBtrQQmRWawcKdQ4RxhrCvBULAmcECIJcqCLQrgywiIiIywY1mBnkFsM9ae8Ba2w48AFzVvYG19kVrbUvo4QYgfxT7M2xdi4RMye6xSEh/C4QAlDX6SEuIJqZhv7NBU7yJiIiITGijuVBIHlDc7XEJcPoA7T8NPNXXE8aYG4AbALKzs1m3bt0IdTF8Ho+Hzc8+Ryqwrbycw7GHmRc7j3Xr1tFUagEobzzAunUHe+y37YCPRJdl3xvPUgS8srsS/4Gx7/9483g84/J7O5FoDIdH4zc8Gr/h0fgNj8avp+TkZJqbm8NuHwgEhtR+rP37v/87Tz/9NJmZmbzxxhv9tnvuuef4z//8TwKBAJ/4xCf48pe/PKTz+Hy+sK+jCbGSnjHmo8Ay4Py+nrfW3g3cDbBs2TK7cuXKsetcyLp165ifnUU5sOTSi2l68RcsKVrCylNX8vo/91MacYRVV53Xaw7k/377ZWblxVOUEoT4dM655Mox7/tEsG7dOsbj93Yi0RgOj8ZveDR+w6PxGx6NX087d+4kMTFx8IYhzc3NQ2o/1j772c9y66238vGPf7zffgYCAb761a/y3HPPkZ+fz/Lly/nwhz/M/PnHVu/2LzY2liVLloTVdjQD5FKgoNvj/NC2HowxFwP/BZxvrW0bxf4MW0eFs4perdtisV1TvFUfaSItL6FXcAxQ1tDKiulpziIhKq8QERGREfSjjT9iV92uAdsEAgFcrt4xSn/mps3lP1f856Dt3nnnHW655RZqamrYtWsX1lpuv/127rzzzrDPBXDeeedx6NChAdts3LiRoqIiZsyYAcC1117Lv/71ryEFyEMxmgHym8AsY8x0nMD4WuDfujcwxiwBfgdcZq2tGsW+jAh/ZQWujAwq2msAZ4o3ay3VRzxMPzWjV3tvm58mn99ZRW/PXpi9aqy7LCIiIjLifD4f11xzDffddx8rVqzg9ttvx+fz8Z3vfKerzbnnnttnacdPfvITLr744iGdr7S0lIKCo3nX/Pz8AcsxhmvUAmRrrd8Y83ngGcAF/NFau90YcyewyVr7KPBjwA38PTT7wxFr7YStQeioCC0SEpoDOdedi6e+DZ+3g8yC3h8JdM5gMTW+HbxVWmJaRERERlQ4md7RKLFYu3YtS5cuZcWKFQAsXryYp59+usdsXuvXrx/Rc46lUa1BttY+CTx5zLZvdft5aG8fxpm/ooKoqVO7VtHLjs+mbJvzzihzau8Lryw0B3IhZc4GTfEmIiIiJ4Bt27axaNGirsdbtmxh6dKlPdqMZAY5Ly+P4uKjcz+UlJSQl5c3xF6Hb0LcpDdZdFRWEr98OUeaj5Aem05sZCzVxWUYA+l57l7tOzPIUzpCv1DVIIuIiMgJID09nRdeeAGAPXv28PDDD/Paa6/1aDOSGeTly5ezd+9eDh48SF5eHg888AD333//iB3/WFpqOkzG5yPY1ETklGzerHiTJVnOXZA1xR5SpiQQFdPXDXo+jIGUlkMQEQmpE3qhQBEREZGwXHfddXg8HhYuXMgNN9zAmjVrSE9PP+5jnXnmmezevZv8/HzuueceAFavXk1ZmfMpfGRkJHfddRerVq1i3rx5XH311SxYsGDEXs+xlEEOU0RDAwBNyVGUe8v51MJPAVB9pJm82Sl97lPe2EqmOwZX3T5InQ6uqDHqrYiIiMjocbvdPPbYYyNyrDVr1vS5/ckne1Tpsnr1alavXj0i5xyMMshhcoUC5J0uZ7KN03NOp6WpHW9DW5/1xwDljT5yUuKgdr9u0BMRERGZJBQghymivh6ATcGDZMVnUZhUSE1x6Aa9PmawAGcO5LykaCdATp85Zn0VERERkeOnADlMrvoGAF5u284ZOWdgjKE6FCBnFPS+Qc9aS3mjjzlxTRBo0w16IiIiIpOEapDDFNFQD8mJVAUbOD3ndACqj3hIyoglJr53bXFTq5+W9gCzI2udDelFY9ldERERETlOyiCHyVXfgDc1DoAVU5xJsauLm/stryhtCC0SYkOra6sGWURERGRSUIAcpoiGeqoSAhQmFTIlYQptrX6aqlvJ6PcGPSdAzmwvgZgkSMgcy+6KiIiIyHFSgBymiPoGDsU0dZVXdN2g10+AXNborKKX7D3klFd0W3pRREREZDJzu4/ef/Xkk08ye/ZsDh8+3G/73/72t9x3331DOsdZZ5113P0bLtUghyHY1obL46HSHcE5XfXHA89gUd7QSmSEIbrxAEw9c8z6KiIiIjJWnn/+eb7whS/wzDPPMG1a/wui3XTTTUM+9rEr840lBchh8FdWAlCXaLrqj2uKPSQkRxOfFN3nPuWNPgoSDaaxRDfoiYiIyKio+MEPaNu5a8A2/kCAOlfvFX/7EzNvLlO+8Y1B27388st89rOf5cknn2TmTGc629///vfcfffdtLe3U1RUxF/+8hfi4+O54447cLvdfOUrX2HlypUsWbKE9evX4/V6ue+++/jhD3/I1q1bueaaa/je974HOFlqj8fDunXruOOOO8jIyGDbtm2cdtpp/N///R/GGJ588km+/OUvk5CQwNlnn82BAwd4/PHHw36t/VGJRRg6KioASMidSnJMMhC6Qa+f8gpw5kBeklAHWMhQgCwiIiInjra2Nt73vvfxyCOPMHfu3K7tH/jAB3jzzTd55513mDdvXtey0ceKjo5m06ZN3HTTTVx11VX86le/Ytu2bfz5z3+mtra2V/u33nqLn//85+zYsYMDBw7w6quv4vP5uPHGG3nqqafYvHkz1dXVI/b6lEEOQ0tZMQDTZy0DoKM9QH25lxmn9n/jXXmjj0tTnFX3lEEWERGR0RBOpre5uZnExP6TescjKiqKs846i3vuuYf//d//7dq+bds2vvnNb9LQ0IDH42HVqlV97n/llVcCsGjRIhYsWEBOTg4AM2bMoLi4mPT09B7tV6xYQX5+PgCnnnoqhw4dwu12M2PGDKZPnw7Addddx9133z0ir08Z5DAc2fcWAAvnnQ9AbakHa/u/QS8YtFQ0+pgZ4WSeSdMqeiIiInLiiIiI4MEHH2Tjxo384Ac/6Np+/fXXc9ddd7F161a+/e1v4/P5+tw/Jiam6zidP3c+9vv9/bYHcLlcfbYZSQqQw1B1ZBfeGFhS6NxsV3Ok/xX0AGq97bQHguQHSiAxF2L6biciIiIyWcXHx/PEE0/w17/+tauUorm5mZycHDo6OvjrX/86quefM2cOBw4c4NChQwD87W9/G7Fjq8QiDDHz5rLP28SyqHgASvc0EJsQRWJabJ/tO+dAzmgrUf2xiIiInLDS0tJ4+umnOe+888jMzOS73/0up59+OpmZmZx++uk0NzeP2rnj4uL49a9/zWWXXUZCQgLLly8fsWMrQA7DhZ/7HuvWrQPA29DGgbeqWbQyH9PP3MZlDc7HCW7vIZjxgTHqpYiIiMjY8Hg8XT8XFBRw8ODBrsef+9znerW/4447un7ujKkAVq5cycqVK/t8rvMcx7a56667un6+4IIL2LVrF9Zabr75ZpYtW3Ycr6Y3lVgM0bb1pQStZeHKvH7blDe2kkoTkW0NkK4lpkVERERGw+9//3tOPfVUFixYQGNjIzfeeOOIHFcZ5CEIdATZ/nIphQvTScmK79r+1zcO85t1+7seN7Z2MD/SmTtZM1iIiIiIjI5bb72VW2+9dcSPqwB5CPZtrqS1uYPFFxR0bWttD/DTZ/eQlhDN4vzkru0fYAfsANI1g4WIiIjIZKIAOUzWWt55oYTUKfHkz0vt2v7Q5mLqvO389qOnsWJ62tEd1j4CEVGQ0v+yiyIiIiIy8agGOUyttVB9pLnHzXn+QJDfrz/I0qkpLC9M7blDzV5Imw4uvQcRERERmUwUIIepbo8lOtbFnDOmdG17alsFR+pauPH8mb1ntKjdrxv0RERERCYhBchh8Da00VgM887OJTrWyQhba/ndy/uZkZHAJfOye+4QDEDdAdUfi4iIyAmnuLiY6dOnU1dXB0B9fT3Tp0/vWrAjXGVlZXzoQx8ahR4OnwLkMGx7uRQsLOo2tdtr+2vZVtrEDefNICLimOxxYzEE2iBDGWQRERE5sRQUFPC5z32O2267DYDbbruNG264gcLCwiEdJzc3l4ceeijs9qO9vHR3KpANQ3JmHGmzIDnz6NRuv31pP5mJMbxvSR/zIdfsc75rijcREREZResf3ENNsWfANoFAAJfLFfYxMwrcnHv17AHb3HrrrZx22mn8/Oc/55VXXuGuu+7C4/Fw1VVXUV9fT0dHB9/73ve46qqruO222ygoKODmm28GnEVD3G43H/rQh7jiiivYtm0bgUCA2267jXXr1tHW1sbNN9/MjTfeyLp167j99ttJTU1l165d7NmzJ+zXMRwKkMOQnNtEQu4+9r0TDUBVcxt1+3by1bOmEVu9tfcOh9Y731WDLCIiIiegqKgofvzjH3PZZZfx7LPPEhUVhTGGf/7znyQlJVFTU8MZZ5zBlVdeyTXXXMOXvvSlrgD5wQcf5JlnniEQCHQd75577iE5OZk333yTtrY2zj77bC699FIAtmzZwrZt25g+ffqYvT4FyGEofuROrqx/HHY7j4uAJ2KAzaGvvsSnQ0LG2HRQRERETkqDZXoBmpubSUxMHPFzP/XUU+Tk5LBt2zYuueQSrLV84xvf4OWXXyYiIoLS0lIqKytZsmQJVVVVlJWVUV1dTWpqKgUFBT1qlp999lnefffdrpKLxsZG9u7dS3R0NCtWrBjT4BgUIIcl/YKb+debC5k29eicxlmJMeSmxPW/U9oMOHZmCxEREZETwNtvv81zzz3Hhg0bOOecc7j22mt55plnqK6uZvPmzURFRVFYWIjP5wPgwx/+MA899BAVFRVcc801vY5nreWXv/wlq1at6rF93bp1JCQkjMlr6k4BchhmLj6L4rp2Tl25cry7IiIiIjKurLV87nOf4+c//zlTp07lq1/9Kl/5yldYsWIFWVlZREVF8eKLL3L48OGufa655ho++9nPUlNTw0svvdTrmKtWreI3v/kNF154IVFRUezZs4e8vD7u8xojmsVCRERERML2+9//nqlTp3LJJZcA8O///u/s3LmTU089lU2bNrFo0SLuu+8+5s6d27XPggULaG5uJi8vj5ycnF7H/MxnPsP8+fNZunQpCxcu5MYbbxzTWSuOpQyyiIiIiITthhtu4IYbbuh67HK52LJlCwCvv/56v/tt3dpzYoPCwkK2bdsGQEREBD/4wQ/4wQ9+0KPNypUrWTkOn+ArgywiIiIi0o0CZBERERGRbhQgi4iIiEwy1trx7sKkMtTxUoAsIiIiMonExsZSW1urIDlM1lpqa2uJjY0Nex/dpCciIiIyieTn51NSUkJ1dXVY7X0+35CCwxNRbGws+fn5YbdXgCwiIiIyiURFRQ1pZbl169axZMmSUezRiUclFiIiIiIi3ShAFhERERHpRgGyiIiIiEg3ZrLdAWmMqQYOD9pw5GUANeNw3hOFxm/4NIbDo/EbHo3f8Gj8hkfjNzwav/5Ns9ZmHrtx0gXI48UYs8lau2y8+zFZafyGT2M4PBq/4dH4DY/Gb3g0fsOj8Rs6lViIiIiIiHSjAFlEREREpBsFyOG7e7w7MMlp/IZPYzg8Gr/h0fgNj8ZveDR+w6PxGyLVIIuIiIiIdKMMsoiIiIhINwqQRURERES6UYAcBmPMZcaY3caYfcaY28a7PxOdMabAGPOiMWaHMWa7MeaLoe1pxpjnjDF7Q99Tx7uvE5kxxmWMecsY83jo8XRjzBuh6/Bvxpjo8e7jRGWMSTHGPGSM2WWM2WmMOVPXX/iMMbeG/u1uM8asMcbE6vobmDHmj8aYKmPMtm7b+rzmjOMXobF81xizdPx6PjH0M34/Dv0bftcY809jTEq3574eGr/dxphV49LpCaSv8ev23H8YY6wxJiP0WNdfGBQgD8IY4wJ+BVwOzAeuM8bMH99eTXh+4D+stfOBM4CbQ2N2G/C8tXYW8HzosfTvi8DObo9/BPzMWlsE1AOfHpdeTQ7/CzxtrZ0LnIIzjrr+wmCMyQO+ACyz1i4EXMC16PobzJ+By47Z1t81dzkwK/R1A/CbMerjRPZneo/fc8BCa+1iYA/wdYDQ35NrgQWhfX4d+lt9MvszvccPY0wBcClwpNtmXX9hUIA8uBXAPmvtAWttO/AAcNU492lCs9aWW2u3hH5uxglO8nDG7d5Qs3uB941LBycBY0w+8B7gD6HHBrgQeCjUROPXD2NMMnAecA+AtbbdWtuArr+hiATijDGRQDxQjq6/AVlrXwbqjtnc3zV3FXCfdWwAUowxOWPS0Qmqr/Gz1j5rrfWHHm4A8kM/XwU8YK1ts9YeBPbh/K0+afVz/QH8DPga0H1GBl1/YVCAPLg8oLjb45LQNgmDMaYQWAK8AWRba8tDT1UA2ePVr0ng5zj/qQVDj9OBhm5/LHQd9m86UA38KVSi8gdjTAK6/sJirS0FfoKTcSoHGoHN6Po7Hv1dc/q7MnSfAp4K/azxC4Mx5iqg1Fr7zjFPafzCoABZRo0xxg38A/iStbap+3PWmV9Qcwz2wRhzBVBlrd083n2ZpCKBpcBvrLVLAC/HlFPo+utfqE72Kpw3GrlAAn18dCtDo2vu+Blj/gundO+v492XycIYEw98A/jWePdlslKAPLhSoKDb4/zQNhmAMSYKJzj+q7X24dDmys6PcULfq8arfxPc2cCVxphDOCU9F+LU1KaEPvIGXYcDKQFKrLVvhB4/hBMw6/oLz8XAQWtttbW2A3gY55rU9Td0/V1z+rsSJmPM9cAVwEfs0YUbNH6Dm4nzJved0N+SfGCLMWYKGr+wKEAe3JvArNAd3NE4NwY8Os59mtBC9bL3ADuttf/T7alHgU+Efv4E8K+x7ttkYK39urU231pbiHO9vWCt/QjwIvChUDONXz+stRVAsTFmTmjTRcAOdP2F6whwhjEmPvRvuXP8dP0NXX/X3KPAx0OzCZwBNHYrxZAQY8xlOKVmV1prW7o99ShwrTEmxhgzHedms43j0ceJylq71VqbZa0tDP0tKQGWhv5/1PUXBq2kFwZjzGqcmlAX8Edr7ffHt0cTmzHmHGA9sJWjNbTfwKlDfhCYChwGrrbW9nVTgYQYY1YCX7HWXmGMmYGTUU4D3gI+aq1tG8fuTVjGmFNxbnCMBg4An8RJCOj6C4Mx5jvANTgfa78FfAanRlHXXz+MMWuAlUAGUAl8G3iEPq650BuPu3BKV1qAT1prN41DtyeMfsbv60AMUBtqtsFae1Oo/X/h1CX7ccr4njr2mCeTvsbPWntPt+cP4cxMU6PrLzwKkEVEREREulGJhYiIiIhINwqQRURERES6UYAsIiIiItKNAmQRERERkW4UIIuIiIiIdKMAWURkhBhjAsaYt7t93Tb4XmEfu9AYs22kjncc519pjHl8vM4vIjKWIgdvIiIiYWq11p463p2YiIwxLmttYLz7ISISDmWQRURGmTHmkDHm/xljthpjNhpjikLbC40xLxhj3jXGPG+MmRranm2M+acx5p3Q11mhQ7mMMb83xmw3xjxrjInr41x/Nsb8whjzmjHmgDHmQ6HtPTLAxpi7Qsv4dvbvh6Gs9yZjzFJjzDPGmP3GmJu6HT7JGPOEMWa3Mea3xpiI0P6XGmNeN8ZsMcb83Rjj7nbcHxljtgAfHvmRFREZHQqQRURGTtwxJRbXdHuu0Vq7CGcFq5+Htv0SuNdauxj4K/CL0PZfAC9Za08BlgLbQ9tnAb+y1i4AGoAP9tOPHOAc4Argv8Ps+5FQ9ns98GecZaXPAL7Trc0K4BZgPjAT+IAxJgP4JnCxtXYpsAn4crd9aq21S621D4TZDxGRcacSCxGRkTNQicWabt9/Fvr5TOADoZ//Avy/0M8XAh8HCJUlNBpjUoGD1tq3Q202A4X9nOsRa20Q2GGMyQ6z74+Gvm8F3NbaZqDZGNNmjEkJPbfRWnsAupa2PQfw4QTMrzor2BINvN7tuH8L8/wiIhOGAmQRkbFh+/l5KNq6/RwAepVY9NHOhL776fmpYWw/+wSP2T/I0b8Vx/bbho7/nLX2un764u1nu4jIhKUSCxGRsXFNt++dGdbXgGtDP38Ep7wB4Hngc+Dc3GaMSR6B8x8G5htjYkIZ4YuO4xgrjDHTQ7XH1wCvABuAs7vVVScYY2aPQH9FRMaNMsgiIiMnzhjzdrfHT1trO6d6SzXGvIuTne3Mtt4C/MkY81WgGvhkaPsXgbuNMZ/GyRR/DigfTsestcXGmAeBbcBB4K3jOMybODXURcCLwD+ttcHQzX5rjDExoXbfBPYMp78iIuPJWHu8n/SJiEg4jDGHgGXW2prx7ouIiAxOJRYiIiIiIt0ogywiIiIi0o0yyCIiIiIi3ShAFhERERHpRgGyiIiIiEg3CpBFRERERLpRgCwiIiIi0s3/B0yjL4UptuuAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(10, 5))\n",
    "\n",
    "for idx, label in enumerate(init_vars_for_plot):\n",
    "    ax.plot(accs_test_on_iterations[idx], label=label)\n",
    "\n",
    "ax.set_title(\"Test quality for different initializations\")\n",
    "ax.set_xlabel(\"Epoch number\")\n",
    "ax.set_ylabel(\"Test accuracy\")\n",
    "ax.grid(True)\n",
    "ax.legend()\n",
    "\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:37:15.087902Z",
     "start_time": "2021-03-03T14:37:15.065996Z"
    }
   },
   "source": [
    "Дайте развёрнутый ответ на вопросы (в этой же ячейке):\n",
    "* Как меняется скорость обучения в зависимости от выбранной инициализации?\n",
    "\n",
    "__Ответы:__\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Эксперименты c различными функциями активации (1 балл)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Проверим теперь, с какой функцией активации нейронная сеть будет обучаться лучше.\n",
    "\n",
    "В этом пункте вам предлагается попробовать обучить несколько нейронных сетей с различными функциями активации.\n",
    "\n",
    "Для этого нам нужно реализовать еще 2 слоя: для функций активации `Tanh` и `Sigmoid`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:34.891308Z",
     "start_time": "2021-03-03T14:46:34.878379Z"
    }
   },
   "outputs": [],
   "source": [
    "class Tanh:\n",
    "    \"\"\"\n",
    "    tanh(y) = (e^y - e^(-y)) / (e^y + e^(-y))\n",
    "    Используйте функцию np.tanh для подсчета гиперболического тангенса.\n",
    "    Вы можете сами реализовать подсчет tanh, но тогда вам нужно устойчиво его вычислять.\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        self.params = [] # Tanh has no parameters\n",
    "\n",
    "    def forward(self, input):\n",
    "        \"\"\"\n",
    "        Apply elementwise Tanh to [batch, num_units] matrix\n",
    "        \"\"\"\n",
    "        ### your code here\n",
    "\n",
    "    def backward(self, grad_output):\n",
    "        \"\"\"\n",
    "        Compute gradient of loss w.r.t. Tanh input\n",
    "        grad_output shape: [batch, num_units]\n",
    "        output 1 shape: [batch, num_units]\n",
    "        output 2: []\n",
    "        \"\"\"\n",
    "        ### your code here\n",
    "\n",
    "    def __repr__(self):\n",
    "        return 'Tanh()'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:34.907337Z",
     "start_time": "2021-03-03T14:46:34.894300Z"
    }
   },
   "outputs": [],
   "source": [
    "from scipy.special import expit\n",
    "    \n",
    "class Sigmoid:\n",
    "    \"\"\"\n",
    "    sigmoid(y) = 1 / (1 + e^(-y))\n",
    "    Используйте функцию expit для подсчета сигмоиды.\n",
    "    Вы можете сами реализовать подсчет сигмоиды, но тогда вам нужно устойчиво ее вычислять.\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        self.params = [] # Sigmoid has no parameters\n",
    "\n",
    "    def forward(self, input):\n",
    "        \"\"\"\n",
    "        Apply elementwise Sigmoid to [batch, num_units] matrix\n",
    "        \"\"\"\n",
    "        ### your code here\n",
    "\n",
    "    def backward(self, grad_output):\n",
    "        \"\"\"\n",
    "        Compute gradient of loss w.r.t. Sigmoid input\n",
    "        grad_output shape: [batch, num_units]\n",
    "        output 1 shape: [batch, num_units]\n",
    "        output 2: []\n",
    "        \"\"\"\n",
    "        ### your code here\n",
    "\n",
    "    def __repr__(self):\n",
    "        return 'Sigmoid()'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь попробуйте для каждой из 3 функций активации обучить нейронную сеть несколько раз. Число слоев зафиксируйте равным 3. В случае `Tanh` и `Sigmoid` используйте инициализацию `Xavier`, а в случае `ReLU` используйте инициализацию `Kaiming`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Заполните матрицы `accs_train` и `accs_test`. В позиции `[i, j]` должна стоять величина доли правильных ответов сети при $j$-м запуске (все запуски идентичны) с функцией активации $ReLU$ при $i = 0$, с функцией активации $Tanh$ при $i = 1$ и с функцией активации $Sigmoid$ при $i = 2$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:34.922732Z",
     "start_time": "2021-03-03T14:46:34.910271Z"
    }
   },
   "outputs": [],
   "source": [
    "act_func_vars = ['Tanh', 'Sigmoid', 'ReLU']\n",
    "init_for_act_funcs = ['Xavier', 'Xavier', 'Kaiming']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:46:34.938935Z",
     "start_time": "2021-03-03T14:46:34.925724Z"
    }
   },
   "outputs": [],
   "source": [
    "accs_train = np.zeros((3, 5))\n",
    "accs_test = np.zeros((3, 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:47:04.514986Z",
     "start_time": "2021-03-03T14:46:34.941104Z"
    }
   },
   "outputs": [],
   "source": [
    "### your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Построим боксплоты полученного качества (горизонтальная линия в каждом столбце — среднее, прямоугольник показывает разброс)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-03T14:47:04.783320Z",
     "start_time": "2021-03-03T14:47:04.517701Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAFgCAYAAACmDI9oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAArLUlEQVR4nO3dfZgfdX3v/+fLYARNkHDTCASBVmyNiFAiaJV2oZ6r2KNy2wriDf2pKT+lPR4PVPh5apHKQSu9LAq1RUWk0gMa76iNAgJfqbYqQcO9YEQpCVgFgrigIOH9++M7i8O6Sb5JdvJddp+P6/peO/OZmc+8Z3cHXpn9zEyqCkmSJEl9Txp2AZIkSdJUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSU9gSY5N8tXW/GiSX5+kvm9MMjIZfUnSE4kBWdKM04TIsc+jSX7Wmj9mI/rrJXljF7VuqKqaU1W3ASQ5L8m7N6Gv51ZVb2O2TfKDcd/XSze2Dkna3LYYdgGStLlV1Zyx6SQ/AN5YVV8eXkXT1is25vuaZFZVremiIEkahFeQJamR5ElJTkryvST3JPlkkm2bZVsm+UTTfl+Sq5PMT3IacABwVnOl9Ky19P3aJLc327+jucL60mbZ4670JhlJsrI1P1bTT5PclOSwdRxDJXlWksXAMcBfNHX9S5ITk3x63PofSHLmWvpq13hK8/04v6njxiSLBv3erktz/B9KsjTJA8CB46/KTzCUpJIcl+S7zc/j7CRplj0ryVeS/CTJ3Ukumow6Jc0cBmRJ+qU/Aw4Ffg/YCVgNnN0sez3wdGAXYDvgOOBnVfUO4N+A45vhDceP7zTJQuBDwGubfrcDFmxAXd+jH8KfDrwL+ESSHde1QVWdA1wA/E1T1yuATwAHJ9mmqWsL4Cjg/AHreCVwIbANcDEw4T8GWi5I8uMklyZ5/nrWfTVwGjAX+Op61h3zcuAFwF7AHwN/0LT/NXApMI/+9/mDA/YnSYABWZLajgPeUVUrq+oh4BTgyCZI/oJ+sH1WVa2pqmuq6v4B+z0S+EJVXdX0+5fAo4MWVVWfqqo7q+rRqroI+C6w3wYc11g/dwFXAX/UNB0M3F1V1wzYxVeramkz/OGfgHWF3mOA3YBdgSuBS8aC+Vp8vqq+1hzjzwes5z1VdV9V/Wezj72b9l80+92pqn5eVYMGbkkCDMiS1LYr8NnmT/b3ATcDa4D59APhJcCFSe5M8jdJnjxgvzsBd4zNVNUDwD2DFpXkdUmWt+raE9h+0O3H+Tjwmmb6NfSPa1A/bE0/CGzZ/OPhVzRh92dV9WBVnQ7cR/8q+NrcsY5lg9YzNrb8L4AA32yGgvw/G9G3pBnMgCxJv3QH8LKq2qb12bKqVlXVL6rqXVW1EPgd+n/ef12zXa2n37voD80AIMlT6V+NHvMA8NTW/DNa6+4KfBg4HtiuqrYBbqAfANdnoro+B+yVZM/mGC4YoJ/JUKy75vG1rvV7st4dVf2wqt5UVTsBfwr8fZJnDVyppBnPgCxJv/QPwGlNKCXJDkkOaaYPTPK8JLOA++n/GX9smMR/Aet69vAS4OVJXpJkNnAqj//v73LgD5Nsm+QZwFtby55GPzz+uKnjT+hfQR7Er9TVDF9YAvwz8M1meMKkSvLMJC9OMru5ufFE+le8v7YB3SwHDk/y1CbcvmED9v9HScbGeK+m//0beEiLJBmQJemXzqR/89mlSX4KfB3Yv1n2DPrB8n76Qy++wi+HJ5xJf6zy6iQfGN9pVd0IvIV+KL2Lfmhb2Vrln4BrgR/Qv7nsota2NwF/C/wH/cD7PAYPmh8FFjZDMz7Xav9408+GDK/YEHPp35S4GlhFf6zzy6pq4GElwPuBh+kf88fZsCvdLwC+kWSU/s/zf4w9G1qSBpGq9f1lUJI02Yb5/OUkzwS+AzxjA240lKQZwyvIkjSDJHkS8DbgQsOxJE3MN+lJ0gyR5Gn0hyzcTn/YgyRpAg6xkCRJklocYiFJkiS1zIghFttvv33ttttuwy5DA3rggQd42tOeNuwypGnJ80vqjufXE88111xzd1XtML59RgTk3XbbjWXLlg27DA2o1+sxMjIy7DKkacnzS+qO59cTT5LbJ2p3iIUkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJatli2AVIkiRNhm233ZbVq1cPu4yhmjdvHvfee++wy3jCMyBLkqRpYfXq1VTV0Pbf6/UYGRkZ2v4Bkgx1/9OFQywkSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElq6TQgJzk4yS1JViQ5aYLluya5PMl1SXpJFrSWPTPJpUluTnJTkt2a9vOSfD/J8uazd5fHIEmSpJmls4CcZBZwNvAyYCFwdJKF41Y7Azi/qvYCTgVOby07H3hfVT0H2A/4UWvZiVW1d/NZ3tUxSJIkaebp8gryfsCKqrqtqh4GLgQOGbfOQuCKZvrKseVNkN6iqi4DqKrRqnqww1olSZIkoNuAvDNwR2t+ZdPWdi1weDN9GDA3yXbAs4H7knwmybeTvK+5Ij3mtGZYxvuTPKWrA5AkSdLMM+w36Z0AnJXkWOAqYBWwhn5dBwD7AP8JXAQcC3wUOBn4ITAbOAd4O/3hGY+TZDGwGGD+/Pn0er1OD0STZ3R01J+X1BHPL013w/z9nirn11So4YkuXb2SMcmLgFOq6g+a+ZMBqur0taw/B/hOVS1I8kLgvVX1e82y1wIvrKq3jNtmBDihql6+rloWLVpUy5Yt28Qj0uYyFV7VKU1Xnl+azpL4qukhfw+eaJJcU1WLxrd3OcTiamCPJLsnmQ0cBVw8rqjtk4zVcDJwbmvbbZLs0MwfBNzUbLNj8zXAocANHR6DJEmSZpjOhlhU1SNJjgcuAWYB51bVjUlOBZZV1cXACHB6kqI/xOItzbZrkpwAXN4E4WuADzddX9AE5wDLgeO6OgZJkvTEUX+1NZzy9KHtfwSgN7TdA833QJus0zHIVbUUWDqu7Z2t6SXAkrVsexmw1wTtB01ymZIkaRrIu+53iEVCnTLUEqYF36QnSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktXQakJMcnOSWJCuSnDTB8l2TXJ7kuiS9JAtay56Z5NIkNye5KcluTfvuSb7R9HlRktldHoMkSZJmls4CcpJZwNnAy4CFwNFJFo5b7Qzg/KraCzgVOL217HzgfVX1HGA/4EdN+3uB91fVs4DVwBu6OgZJkiTNPF1eQd4PWFFVt1XVw8CFwCHj1lkIXNFMXzm2vAnSW1TVZQBVNVpVDyYJcBCwpNnm48ChHR6DJEmSZpgtOux7Z+CO1vxKYP9x61wLHA6cCRwGzE2yHfBs4L4knwF2B74MnATMA+6rqkdafe480c6TLAYWA8yfP59erzcJh6TNYXR01J+X1BHPL013/WtpM9fcuXM9xydBlwF5ECcAZyU5FrgKWAWsoV/XAcA+wH8CFwHHAp8ftOOqOgc4B2DRokU1MjIyiWWrS71eD39eUjc8vzSdVdVQ9+/5NX10OcRiFbBLa35B0/aYqrqzqg6vqn2AdzRt99G/Mry8GZ7xCPA54LeBe4Btkmyxtj4lSZKkTdFlQL4a2KN56sRs4Cjg4vYKSbZPMlbDycC5rW23SbJDM38QcFP1/2l4JXBk0/56NuCqsiRJkrQ+nQXk5srv8cAlwM3AJ6vqxiSnJnlls9oIcEuSW4H5wGnNtmvoD7+4PMn1QIAPN9u8HXhbkhXAdsBHuzoGSZIkzTydjkGuqqXA0nFt72xNL+GXT6QYv+1lwF4TtN9G/wkZkiRJ0qTzTXqSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaOg3ISQ5OckuSFUlOmmD5rkkuT3Jdkl6SBa1la5Isbz4Xt9rPS/L91rK9uzwGSZIkzSxbdNVxklnA2cB/A1YCVye5uKpuaq12BnB+VX08yUHA6cBrm2U/q6q919L9iVW1pKPSJUmSNIN1eQV5P2BFVd1WVQ8DFwKHjFtnIXBFM33lBMslSZKkzaqzK8jAzsAdrfmVwP7j1rkWOBw4EzgMmJtku6q6B9gyyTLgEeA9VfW51nanJXkncDlwUlU9NH7nSRYDiwHmz59Pr9eblINS90ZHR/15SR3x/JK64/k1fXQZkAdxAnBWkmOBq4BVwJpm2a5VtSrJrwNXJLm+qr4HnAz8EJgNnAO8HTh1fMdVdU6znEWLFtXIyEjHh6LJ0uv18OcldcPzS+qO59f00eUQi1XALq35BU3bY6rqzqo6vKr2Ad7RtN3XfF3VfL0N6AH7NPN3Vd9DwMfoD+WQJEmSJkWXAflqYI8kuyeZDRwFXNxeIcn2ScZqOBk4t2mfl+QpY+sALwZuauZ3bL4GOBS4ocNjkCRJ0gzT2RCLqnokyfHAJcAs4NyqujHJqcCyqroYGAFOT1L0h1i8pdn8OcA/JnmUfoh/T+vpFxck2QEIsBw4rqtjkCRJ0szT6RjkqloKLB3X9s7W9BLgVx7XVlX/DjxvLX0eNMllSpIkSY/xTXqSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1DPtNepqCtt12W1avXj3sMoZm3rx53HvvvcMuQ5IkDYkBWb9i9erVVNXQ9j/sV3X230EjSZJmKodYSJIkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqWW9ATnJK5IYpCVJkjQjDBJ8XwV8N8nfJPmtrguSJEmShmm9AbmqXgPsA3wPOC/JfyRZnGRu59VJkiRJm9kWg6xUVfcnWQJsBbwVOAw4MckHquqDHdYnSWpJMuwSqKphlyBJnVpvQE7ySuBPgGcB5wP7VdWPkjwVuAkwIEvSZrKp4TSJAVeS1mOQK8hHAO+vqqvajVX1YJI3dFOWJEmSNByDBORTgLvGZpJsBcyvqh9U1eVdFSZJkiQNwyBPsfgU8Ghrfk3Ttl5JDk5yS5IVSU6aYPmuSS5Pcl2SXpIFrWVrkixvPhe32ndP8o2mz4uSzB6kFkmSJGkQgwTkLarq4bGZZnq9oTTJLOBs4GXAQuDoJAvHrXYGcH5V7QWcCpzeWvazqtq7+byy1f5e+kM+ngWsBhzmIUmSpEkzSED+cXOjHgBJDgHuHmC7/YAVVXVbE6ovBA4Zt85C4Ipm+soJlj9O+rdvHwQsaZo+Dhw6QC2SJEnSQAYZg3wccEGSs4AAdwCvG2C7nZt1x6wE9h+3zrXA4cCZ9B8dNzfJdlV1D7BlkmXAI8B7qupzwHbAfVX1SKvPnSfaeZLFwGKA+fPn0+v1BihZY4b5/RodHR36z2vY+5e65O+31I2p8P8vTY4M+rifJHMAqmp0wPWPBA6uqjc2868F9q+q41vr7AScBewOXEX/iRl7VtV9SXauqlVJfp3+VebfB34CfL0ZXkGSXYAvVtWe66pl0aJFtWzZsoGOU8N/DFSv12NkZGRo+x/28Utd8vdb6s6w//+lDZfkmqpaNL59oBeFJPnvwHPpX9UFoKpOXc9mq4BdWvMLmrbHVNWd9K8gjwXwI6rqvmbZqubrbUl69N/m92lgmyRbNFeRf6VPSZIkaVOsdwxykn8AXgX8Gf0hFn8E7DpA31cDezRPnZgNHAVc3F4hyfZJxmo4GTi3aZ+X5Clj6wAvBm6q/mWPK4Ejm21eD3x+gFokSZKkgQxyk97vVNXrgNVV9S7gRcCz17dRc4X3eOAS4Gbgk1V1Y5JTWzf9jQC3JLkVmA+c1rQ/B1iW5Fr6gfg9VXVTs+ztwNuSrKA/JvmjAxyDJEmSNJBBhlj8vPn6YDNm+B5gx0E6r6qlwNJxbe9sTS/hl0+kaK/z78Dz1tLnbfSfkCFJkiRNukEC8r8k2QZ4H/AtoIAPd1mUJEmSNCzrDMjN+ODLmxvnPp3kC8CWVfWTzVGchqP+ams45elD2/8IQG9ou+8fvyRJmrHWGZCr6tEkZ9N/ggRV9RDw0OYoTEN0ynD//eNjciRJ0jANcpPe5UmOyNjz3SRJkqRpbJCA/KfAp4CHktyf5KdJ7u+4LkmSJGko1nuTXlXN3RyFSJIkSVPBegNykt+dqL2qrpr8ciRJkqThGuQxbye2prek/wzia4CDOqlIkiRJGqJBhli8oj2fZBfg77oqSJIkSRqmQW7SG28l/VdBS5IkSdPOIGOQP0j/7XnQD9R703+jniRJkjTtDDIGeVlr+hHg/1bV1zqqR5IkSRqqQQLyEuDnVbUGIMmsJE+tqge7LU2SJEna/AZ6kx6wVWt+K+DL3ZQjSZIkDdcgAXnLqhodm2mmn9pdSZIkSdLwDBKQH0jy22MzSfYFftZdSZIkSdLwDDIG+a3Ap5LcCQR4BvCqLouSpOlq2223ZfXq1UOtIclQ9z9v3jzuvffeodYgSesyyItCrk7yW8BvNk23VNUvui1Lkqan1atXU1XrX7EjvV6PkZGRoe0fhh/QJWl91jvEIslbgKdV1Q1VdQMwJ8mbuy9NkiRJ2vwGGYP8pqq6b2ymqlYDb+qsIkmSJGmIBgnIs9L6e1iSWcDs7kqSJEmShmeQm/S+BFyU5B+b+T9t2iRJkqRpZ5CA/Hb6ofj/beYvAz7SWUWSJEnSEA3yFItHgQ81H0mSJGlaG+QpFnskWZLkpiS3jX0G6TzJwUluSbIiyUkTLN81yeVJrkvSS7Jg3PKtk6xMclarrdf0ubz5/NogtUiSJEmDGOQmvY/Rv3r8CHAgcD7wifVt1NzMdzbwMmAhcHSSheNWOwM4v6r2Ak4FTh+3/K+Bqybo/piq2rv5/GiAY5AkSZIGMkhA3qqqLgdSVbdX1SnAfx9gu/2AFVV1W1U9DFwIHDJunYXAFc30le3lzSut5wOXDrAvSZIkaVIMcpPeQ0meBHw3yfHAKmDOANvtDNzRml8J7D9unWuBw4EzgcOAuUm2A1YDfwu8BnjpBH1/LMka4NPAu2uC11IlWQwsBpg/fz69Xm+AkjUVjI6O+vPStDbM3++pcn5NhRqkyTZVzi9tukEC8v8Angr8Of0hDwcCr5+k/Z8AnJXkWPpDKVYBa4A3A0urauUEryQ9pqpWJZlLPyC/lv6wj8epqnOAcwAWLVpUw361qgY3FV6FK3VpmL/fU+X8mgo1SJNtqpxf2nSDPMXi6mZyFPiTDeh7FbBLa35B09bu+076V5BJMgc4oqruS/Ii4IDmldZzgNlJRqvqpKpa1Wz70yT/TH8ox68EZEmSJGljDHIFeWNdDeyRZHf6wfgo4NXtFZJsD9zbPEruZOBcgKo6prXOscCiqjopyRbANlV1d5InAy8HvtzhMUiSJGmGGeQmvY1SVY8AxwOXADcDn6yqG5OcmuSVzWojwC1JbqV/Q95p6+n2KcAlSa4DltMP3h/uoHxJkiTNUOu9gpzkxVX1tfW1TaSqlgJLx7W9szW9BFiynj7OA85rph8A9l3ffiVJkqSNNcgV5A8O2CZJkiQ94a31CnJzo9zvADskeVtr0dbArK4LkyRJkoZhXUMsZtN/gsQWwNxW+/3AkV0WJUmSJA3LWgNyVX0F+EqS86rqdoDmhSFzqur+zVWgJEmStDkNMgb59CRbJ3kacANwU5ITO65LkiRJGopBAvLC5orxocAXgd3pv71OkiRJmnYGCchPbl7KcShwcVX9AqhOq5IkSZKGZJCA/I/AD4CnAVcl2ZX+jXqSJEnStLPeF4VU1QeAD7Sabk9yYHclSZIkScOz3ivISeYn+WiSLzbzC4HXd16ZJEmSNASDDLE4D7gE2KmZvxV4a0f1SJIkSUO11oCcZGz4xfZV9UngUYCqegRYsxlqkyRJkja7dV1B/mbz9YEk29E8uSLJC4GfdF2YJEmSNAzrukkvzde3ARcDv5Hka8AO+KppSZIkTVPrCsg7JHlbM/1ZYCn90PwQ8FLguo5rkyRJkja7dQXkWcAcfnklecxTuytHkiRJGq51BeS7qurUzVaJJEmSNAWs6ya98VeOJUmSpGlvXQH59zdbFZIkSdIUsdaAXFX3bs5CJEmSpKlgkDfpSZIkSTOGAVmSJElqMSBLkiRJLQZkSZIkqaXTgJzk4CS3JFmR5KQJlu+a5PIk1yXpJVkwbvnWSVYmOavVtm+S65s+P5DEx9FJkiRp0nQWkJPMAs4GXgYsBI5OsnDcamcA51fVXsCpwOnjlv81cNW4tg8BbwL2aD4HT3LpkiRJmsG6vIK8H7Ciqm6rqoeBC4FDxq2zELiimb6yvTzJvsB84NJW247A1lX19aoq4Hzg0M6OQJIkSTPOul41val2Bu5oza8E9h+3zrXA4cCZwGHA3CTbAauBvwVeA7x0XJ8rx/W580Q7T7IYWAwwf/58er3exh6HNrPR0VF/XprWhvn7PVXOr6lQgzTZpsr5pU3XZUAexAnAWUmOpT+UYhWwBngzsLSqVm7sEOOqOgc4B2DRokU1MjIyGfVqM+j1evjz0nQ2zN/vqXJ+TYUapMk2Vc4vbbouA/IqYJfW/IKm7TFVdSf9K8gkmQMcUVX3JXkRcECSNwNzgNlJRulfaV6wrj4lSZKkTdFlQL4a2CPJ7vRD7FHAq9srJNkeuLeqHgVOBs4FqKpjWuscCyyqqpOa+fuTvBD4BvA64IMdHoMkSZJmmM5u0quqR4DjgUuAm4FPVtWNSU5N8spmtRHgliS30r8h77QBun4z8BFgBfA94IuTXbskSZJmrk7HIFfVUmDpuLZ3tqaXAEvW08d5wHmt+WXAnpNZpyRJkjTGN+lJkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktnb5JT5L0ePVXW8MpTx/a/kcAekPbPdB8DyRpCjMgS9JmlHfdT1UNbf+9Xo+RkZGh7R8gCXXKUEuQpHVyiIUkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS2dBuQkBye5JcmKJCdNsHzXJJcnuS5JL8mCVvu3kixPcmOS41rb9Jo+lzefX+vyGCRJkjSzbNFVx0lmAWcD/w1YCVyd5OKquqm12hnA+VX18SQHAacDrwXuAl5UVQ8lmQPc0Gx7Z7PdMVW1rKvaJUmSNHN1eQV5P2BFVd1WVQ8DFwKHjFtnIXBFM33l2PKqeriqHmran9JxnZIkSdJjOruCDOwM3NGaXwnsP26da4HDgTOBw4C5SbarqnuS7AL8K/As4MTW1WOAjyVZA3waeHdV1fidJ1kMLAaYP38+vV5vco5KnRsdHfXnpWltmL/fU+X8mgo1SJNtqpxf2nSZIFtOTsfJkcDBVfXGZv61wP5VdXxrnZ2As4DdgauAI4A9q+q+cet8DnhFVf1Xkp2ralWSufQD8ieq6vx11bJo0aJatswRGU8UvV6PkZGRYZchdSIJXf13dxBT4fwa9vdA6spUOL+0YZJcU1WLxrd3OXRhFbBLa35B0/aYqrqzqg6vqn2AdzRt941fB7gBOKCZX9V8/Snwz/SHckiSJEmTosuAfDWwR5Ldk8wGjgIubq+QZPskYzWcDJzbtC9IslUzPQ94CXBLki2SbN+0Pxl4Of3wLEmSJE2KzgJyVT0CHA9cAtwMfLKqbkxyapJXNquN0A++twLzgdOa9ucA30hyLfAV4Iyqup7+DXuXJLkOWE7/ivSHuzoGSZIkzTxd3qRHVS0Flo5re2dregmwZILtLgP2mqD9AWDfya9UkiRJ6vPxaZIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJklo6DchJDk5yS5IVSU6aYPmuSS5Pcl2SXpIFrfZvJVme5MYkx7W22TfJ9U2fH0iSLo9BkiRJM0tnATnJLOBs4GXAQuDoJAvHrXYGcH5V7QWcCpzetN8FvKiq9gb2B05KslOz7EPAm4A9ms/BXR2DJEmSZp4uryDvB6yoqtuq6mHgQuCQcessBK5opq8cW15VD1fVQ037U8bqTLIjsHVVfb2qCjgfOLTDY5AkSdIMs0WHfe8M3NGaX0n/anDbtcDhwJnAYcDcJNtV1T1JdgH+FXgWcGJV3ZlkUdNPu8+dJ9p5ksXAYoD58+fT6/U2/Yi0WYyOjvrz0rQ2zN/vqXJ+TYUapMk2Vc4vbbouA/IgTgDOSnIscBWwClgDUFV3AHs1Qys+l2TJhnRcVecA5wAsWrSoRkZGJrFsdanX6+HPS9PZMH+/p8r5NRVqkCbbVDm/tOm6DMirgF1a8wuatsdU1Z30ryCTZA5wRFXdN36dJDcABwBfa/pZa5+SJEnSpuhyDPLVwB5Jdk8yGzgKuLi9QpLtk4zVcDJwbtO+IMlWzfQ84CXALVV1F3B/khc2T694HfD5Do9BkiRJM0xnAbmqHgGOBy4BbgY+WVU3Jjk1ySub1UaAW5LcCswHTmvanwN8I8m1wFeAM6rq+mbZm4GPACuA7wFf7OoYJEmSNPN0Oga5qpYCS8e1vbM1vQT4lbHFVXUZsNda+lwG7Dm5lUqSJEl9vklPkiRJajEgS5IkSS3DfsybJM04/XuMZ6558+YNuwRJWicDsiRtRv2XgA5PkqHXIElTnUMsJEmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElq2WLYBUiSBpdk6H1U1SbXIElTmVeQJekJpKo26XPllVduch+SNN0ZkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWrpNCAnOTjJLUlWJDlpguW7Jrk8yXVJekkWNO17J/mPJDc2y17V2ua8JN9Psrz57N3lMUiSJGlm6SwgJ5kFnA28DFgIHJ1k4bjVzgDOr6q9gFOB05v2B4HXVdVzgYOBv0uyTWu7E6tq7+azvKtjkCRJ0szT5RXk/YAVVXVbVT0MXAgcMm6dhcAVzfSVY8ur6taq+m4zfSfwI2CHDmuVJEmSgG4D8s7AHa35lU1b27XA4c30YcDcJNu1V0iyHzAb+F6r+bRm6MX7kzxlcsuWJEnSTDbsV02fAJyV5FjgKmAVsGZsYZIdgX8CXl9VjzbNJwM/pB+azwHeTn94xuMkWQwsBpg/fz69Xq+zg9DkGh0d9ecldcTzS+qO59f00WVAXgXs0ppf0LQ9phk+cThAkjnAEVV1XzO/NfCvwDuq6uutbe5qJh9K8jH6IftXVNU59AM0ixYtqpGRkU0/Im0WvV4Pf15SNzy/pO54fk0fXQbkq4E9kuxOPxgfBby6vUKS7YF7m6vDJwPnNu2zgc/Sv4Fvybhtdqyqu5IEOBS4YX2FXHPNNXcnuX3TD0mbyfbA3cMuQpqmPL+k7nh+PfHsOlFjZwG5qh5JcjxwCTALOLeqbkxyKrCsqi4GRoDTkxT9IRZvaTb/Y+B3ge2a4RcAxzZPrLggyQ5AgOXAcQPU4g1+TyBJllXVomHXIU1Hnl9Sdzy/po9U1bBrkB7H/8BI3fH8krrj+TV9+CY9SZIkqcWArKnonGEXIE1jnl9Sdzy/pgmHWEiSJEktXkGWJEmSWgzIkiRJUosBWZ1Lsl2S5c3nh0lWteZnD9jHSJIvdF2rNFUleUeSG5Nc15w7+yf5SJKFHe93aZJtJmg/JcmEL2qSppMka5pz7oYk/zLR+TBu/V85N5Kcl+TIcW2jHZSrSTLsV01rBqiqe4C9of8fDmC0qs4YZk3SE0mSFwEvB367qh5qXrI0u6re2PW+q+oPu96HNMX9rKr2BkjycfrvbDhtqBWpc15B1lAkeVOSq5Ncm+TTSZ7atJ+X5ANJ/j3JbeP+xT0nyZIk30lyQfM2RWkm2BG4u6oeAqiqu6vqziS9JIsAkrwhya1Jvpnkw0nOatrPS/KhJF9vzqmRJOcmuTnJeWM7SHJ0kuubq2TvbbX/oAnkY1exb03yVeA3N+PxS1PFfwA7AyT5jSRfSnJNkn9L8ltDrk2TyICsYflMVb2gqp4P3Ay8obVsR+Al9K+YvafVvg/wVmAh8OvAizdPqdLQXQrs0oTTv0/ye+2FSXYC/hJ4If3zYvz/qOcBLwL+J3Ax8H7gucDzkuzdbP9e4CD6f+15QZJDx+1jX+CoZvkfAi+YxOOTprwks4Dfp38OQf+Rbn9WVfsCJwB/P6zaNPkcYqFh2TPJu4FtgDn0X0k+5nNV9ShwU5L5rfZvVtVKgCTLgd2Ar26WaqUhqqrRJqAeABwIXJTkpNYq+wFfqap7AZJ8Cnh2a/m/VFUluR74r6q6vlnvRvrn0a5Ar6p+3LRfAPwu8LlWHwcAn62qB5t1LkaaGbZq/p+zM/0LOpclmQP8DvCp1h8zn7KOPiZ6pq7P2Z3CDMgalvOAQ6vq2iTHAiOtZQ+1prOW9jX4+6sZpKrWAD2g1wTd12/A5mPnzqM8/jx6lP559IvJqFGapn5WVXs3QwEvoT8G+TzgvrGxyQO4h/5fcgBIsi1w9yTXqUnkEAsNy1zgriRPBo4ZdjHSVJbkN5Ps0WraG7i9NX818HtJ5iXZAjhiA3fxzWb77Zs/Ix8NfGXcOlcBhybZKslc4BUbuA/pCa3568mfA/8LeBD4fpI/Akjf89exeQ94VevJTccCV3ZXrTaVV+A0LH8JfAP4cfN17nDLkaa0OcAHm8dLPQKsABYDSwCqalWS/0M/6N4LfAf4yaCdV9VdzZCNK+n/1eZfq+rz49b5VpKLgGuBH9EP5dKMUlXfTnId/X9EHgN8KMn/Bp4MXEj//AD430ne2tpuQTNM6poka4DvAcdt1uK1QXzVtCRNA0nmNGOVtwA+C5xbVZ8ddl2S9ETkEAtJmh5OaW4kugH4Po+/wU6StAG8gixJkiS1eAVZkiRJajEgS5IkSS0GZEmSJKnFgCxJGynJoUkqyfhXO0+07lubFw2MzS9tHtu2ofvcJsmbW/M7JVmyof2spe8DktyYZHmSrSajz6bf/2/c/L9PVt+S1AVv0pOkjdQ8F3gn4Iqq+qv1rPsDYFFVbdLbs5LsBnyhqvbclH7W0vc/AF+tqk9Mcr+jVTVnMvuUpC55BVmSNkKSOcBLgDcAR7XaZyU5I8kNSa5L8mdJ/px+kL4yyZXNej9o3lz3niRvaW1/SpITksxJcnmSbyW5PskhzSrvAX6jucr7viS7Jbmh2XbLJB9r1v92kgOb9mOTfCbJl5J8N8nfTHA8bwT+GPjrJBckGUnyhdbys5rXwo/V/q5Wbb819j1p7f+6JEckeQ+wVVPvBc16o83XNMdwQ7PNq5r2kSS9JEuSfKepJ0jSZuKb9CRp4xwCfKmqbk1yT5J9q+oa+m+42w3Yu6oeSbJtVd2b5G3AgRNcQb4I+Dvg7Gb+j4E/AH4OHFZV9yfZHvh6kouBk4A9q2pveOyK8pi3AFVVz2tC66VJnt0s2xvYB3gIuCXJB6vqjrENq+ojSV5C/+r0kiQj6zn+u6vqt5vhHicAb6T/hsyfVNXzmtrmVdWnkxw/Vu84hzd1PR/YHrg6yVXNsn2A5wJ3Al8DXgx8dT01SdKk8AqyJG2co+m/Wpbm69HN9EuBf6yqRwCq6t51dVJV3wZ+rRlL/HxgdRNcA/yf5rW2XwZ2Buavp6aXAJ9o+v0OcDswFpAvr6qfVNXPgZuAXQc+0ol9pvl6Df1/EED/2MeCPlW1eoB6/29Vramq/wK+ArygWfbNqlpZVY8Cy1v7kKTOeQVZkjZQkm2Bg4DnJSlgFlBJTtzILj8FHAk8g/4VZYBjgB2AfavqF80Y5i03oeyHWtNrWP9//x/h8RdRxu97rL9B+toYG1qvJE0aryBL0oY7Evinqtq1qnarql3ov975AOAy4E+TbAGPhWmAnwJz19LfRfTHMR9JPywDPB34UROOD+SXV3zX1c+/0Q/WNEMrngncsnGHyO3AwiRPaZ628fsDbHMZ/WEeNDXMayZ/keTJa6n3Vc247R2A3wW+uZH1StKkMSBL0oY7GvjsuLZPN+0fAf4TuC7JtcCrm+XnAF8au0mvrapupB96V1XVXU3zBcCiJNcDrwO+06x7D/C15sa2943r6u+BJzXbXAQcW1UPsRGaYR6fBG5ovn57gM3eDcxrarsWOLBpP4f+9+OCcet/FrgOuBa4AviLqvrhxtQrSZPJx7xJkiRJLV5BliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqSW/x+PxklVG5/4uQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(10, 5))\n",
    "\n",
    "ax.boxplot(accs_test.T, labels=act_func_vars, showfliers=False)\n",
    "\n",
    "ax.set_title(\"Test quality in 5 runs\")\n",
    "ax.set_xlabel(\"Activation function\")\n",
    "ax.set_ylabel(\"Test accuracy\")\n",
    "ax.grid(True)\n",
    "\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Дайте развёрнутый ответ на вопросы (в этой же ячейке):\n",
    "* Как отличаются качество на обучении и контроле и устойчивость процесса обучения при различных функциях активации?\n",
    "\n",
    "__Ответы:__\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\* Несколько фрагментов кода в задании написаны на основе материалов [курса по глубинному обучению на ФКН НИУ ВШЭ](https://www.hse.ru/ba/ami/courses/205504078.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Бонусная часть"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Реализация метода оптимизации (1 балл)\n",
    "\n",
    "Реализуйте сами метод оптимизации (аналог функции `minimize`) для рассмотренной выше архитектуры. В качестве метода оптимизации используйте SGD + momentum. Продемонстрируйте правильную работу метода оптимизации, сравните его работу с LBFGS-B. Сделайте выводы."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropout (1 балл) \n",
    "\n",
    "Реализуйте слой Dropout. Сравните обучение сети из большого числа слоёв при использовании Dropout и без его использования (предварительно подберите адекватный параметр $p$). Сделайте выводы."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BatchNormalization (1 балл)\n",
    "\n",
    "Реализуйте слой `BatchNormalization`. Сравните обучение сети из большого числа слоёв при использовании `BatchNormalization` и без его использования. Сделайте выводы."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}