test.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CNTK 203: Reinforcement Learning Basics\n",
    "\n",
    "\n",
    "Reinforcement learning (RL) is an area of machine learning inspired by behaviorist psychology, concerned with how [software agents](https://en.wikipedia.org/wiki/Software_agent) ought to take [actions](https://en.wikipedia.org/wiki/Action_selection) in an environment so as to maximize some notion of cumulative reward. In machine learning, the environment is typically formulated as a [Markov decision process](https://en.wikipedia.org/wiki/Markov_decision_process) (MDP) as many reinforcement learning algorithms for this context utilize [dynamic programming](https://en.wikipedia.org/wiki/Dynamic_programming) techniques.\n",
    "\n",
    "In some machine learning settings, we do not have immediate access to labels, so we cannot rely on supervised learning techniques. If, however, there is something we can interact with and thereby get some feedback that tells us occasionally, whether our previous behavior was good or not, we can use RL to learn how to improve our behavior.\n",
    "\n",
    "Unlike in supervised learning, in RL, labeled correct input/output pairs are never presented and sub-optimal actions are never explicitly corrected. This mimics many of the online learning paradigms which involves finding a balance between exploration (of conditions or actions never learnt before) and exploitation (of already learnt conditions or actions from previous encounters). Multi-arm bandit problems is one of the category of RL algorithms where exploration vs. exploitation trade-off have been thoroughly studied. See figure below for [reference](http://www.simongrant.org/pubs/thesis/3/2.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://cntk.ai/jup/polecart.gif\" width=\"300\" height=\"300\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 1\n",
    "Image(url=\"https://cntk.ai/jup/polecart.gif\", width=300, height=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Problem**\n",
    "\n",
    "We will use the [CartPole](https://gym.openai.com/envs/CartPole-v0) environment from OpenAI's [gym](https://github.com/openai/gym) simulator to teach a cart to balance a pole. As described in the link above, in the CartPole example, a pole is attached by an un-actuated joint to a cart, which moves along a frictionless track. The system is controlled by applying a force of +1 or -1 to the cart. A reward of +1 is provided for every timestep that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center. See figure below for reference.\n",
    "\n",
    "**Goal**\n",
    "\n",
    "Our goal is to prevent the pole from falling over as the cart moves with the pole in upright position (perpendicular to the cart) as the starting state. More specifically if the pole is less than 15 degrees from vertical while the cart is within 2.4 units of the center we will collect reward. In this tutorial, we will train till we learn a set of actions (policies) that lead to an average reward of 200 or more over last 50 batches.\n",
    "\n",
    "In, RL terminology, the goal is to find _policies_ $a$, that maximize the _reward_ $r$ (feedback) through interaction with some environment (in this case the pole being balanced on the cart). So given a series of experiences $$s \\xrightarrow{a} r, s'$$ we then can learn how to choose action $a$ in a given state $s$ to maximize the accumulated reward $r$ over time:\n",
    "\n",
    "$$\n",
    "Q(s,a) = r_0 + \\gamma r_1 + \\gamma^2 r_2 + \\ldots = r_0 + \\gamma \\max_a Q^*(s',a)\n",
    "$$\n",
    "\n",
    "where $\\gamma \\in [0,1)$ is the discount factor that controls how much we should value reward that is further away. This is called the [*Bellmann*-equation](https://en.wikipedia.org/wiki/Bellman_equation).\n",
    "\n",
    "In this tutorial we will show how to model the state space, how to use the received reward to figure out which action yields the highest future reward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We present two different popular approaches here:\n",
    "\n",
    "**Deep Q-Networks**: DQNs have become famous in 2015 when they were successfully used to train how to play Atari just form raw pixels. We train neural network to learn the $Q(s,a)$ values (thus $Q$-Network ). From these $Q$ functions values we choose the best action.\n",
    "\n",
    "**Policy gradient**: This method directly estimates the policy (set of actions) in the network. The outcome is a learning of an ordered set of actions which leads to maximize reward by probabilistically choosing a subset of actions. In this tutorial, we learn the actions using a gradient descent approach to learn the policies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we focus how to implement RL in CNTK. We choose a straight forward shallow network. One can extend the approaches by replacing our shallow model with deeper networks that are introduced in other CNTK tutorials.\n",
    "\n",
    "Additionally, this tutorial is in its early stages and will be evolving in future updates.\n",
    "\n",
    "## Prerequisites\n",
    "Please run the following cell from the menu above or select the cell below and hit `Shift + Enter` to ensure the environment is ready. Verify that the following imports work in your notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "from __future__ import division\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import style\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "\n",
    "style.use('ggplot')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the following construct to install the OpenAI gym package if it is not installed. For users new to Jupyter environment, this construct can be used to install any python package. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import gym\n",
    "except:\n",
    "    !pip install gym\n",
    "    import gym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Select the notebook run mode**\n",
    "\n",
    "There are two run modes:\n",
    "- *Fast mode*: `isFast` is set to `True`. This is the default mode for the notebooks, which means we train for fewer iterations or train / test on limited data. This ensures functional correctness of the notebook though the models produced are far from what a completed training would produce.\n",
    "\n",
    "- *Slow mode*: We recommend the user to set this flag to `False` once the user has gained familiarity with the notebook content and wants to gain insight from running the notebooks for a longer period with different parameters for training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "isFast = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## CartPole: Data and Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the [CartPole](https://gym.openai.com/envs/CartPole-v0) environment from OpenAI's [gym](https://github.com/openai/gym) simulator to teach a cart to balance a pole. Please follow the links to get more details.\n",
    "\n",
    "In every time step, the agent\n",
    " * gets an observation $(x, \\dot{x}, \\theta, \\dot{\\theta})$, corresponding to *cart position*, *cart velocity*, *pole angle with the vertical*, *pole angular velocity*,\n",
    " * performs an action `LEFT` or `RIGHT`, and\n",
    " * receives\n",
    "  * a reward of +1 for having survived another time step, and\n",
    "  * a new state $(x', \\dot{x}', \\theta', \\dot{\\theta}')$\n",
    "\n",
    "The episode ends, if\n",
    " * the pole is more than 15 degrees from vertical and/or\n",
    " * the cart is moving more than 2.4 units from center.\n",
    "\n",
    "The task is considered done, if\n",
    " * the agent achieved and averaged reward of 200 over the last 50 episodes (if you manage to get a reward of 200 averaged over the last 100 episode you can consider submitting it to OpenAI).\n",
    "\n",
    "In fast mode these targets are relaxed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: DQN\n",
    "\n",
    "After a transition $(s,a,r,s')$, we are trying to move our value function $Q(s,a)$ closer to our target $r+\\gamma \\max_{a'}Q(s',a')$, where $\\gamma$ is a discount factor for future rewards and ranges in value between 0 and 1.\n",
    "\n",
    "DQNs\n",
    " * learn the _Q-function_ that maps observation (state, action) to a `score`\n",
    " * use memory replay (previously recorded $Q$ values corresponding to different $(s,a)$ to decorrelate experiences (sequence state transitions)\n",
    " * use a second network to stabilize learning (*not* part of this tutorial)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model: DQN\n",
    "\n",
    "$$\n",
    "l_1 = relu( x W_1 + b_1) \\\\\n",
    "Q(s,a) = l_1 W_2 + b_2 \\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start with a slightly modified version for Keras, https://github.com/jaara/AI-blog/blob/master/CartPole-basic.py, published by Jaromír Janisch in his [AI blog](https://jaromiru.com/2016/09/27/lets-make-a-dqn-theory/), and will then incrementally convert it to use CNTK.\n",
    "\n",
    "We use a simple two-layer densely connected network, for simpler illustrations. More advance networks can be substituted.\n",
    "\n",
    "**CNTK** concepts: The commented out code is meant to be an illustration of the similarity of concepts between CNTK API/abstractions against Keras. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy\n",
    "import math \n",
    "import os \n",
    "import random\n",
    "\n",
    "import cntk as C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the block below, we check if we are running this notebook in the CNTK internal test machines by looking for environment variables defined there. We then select the right target device (GPU vs CPU) to test this notebook. In other cases, we use CNTK's default policy to use the best available device (GPU, if available, else CPU)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Select the right target device when this notebook is being tested:\n",
    "if 'TEST_DEVICE' in os.environ:\n",
    "    if os.environ['TEST_DEVICE'] == 'cpu':\n",
    "        C.device.try_set_default_device(C.device.cpu())\n",
    "    else:\n",
    "        C.device.try_set_default_device(C.device.gpu(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "STATE_COUNT = 4 (corresponding to $(x, \\dot{x}, \\theta, \\dot{\\theta})$),\n",
    "\n",
    "ACTION_COUNT = 2 (corresponding to `LEFT` or `RIGHT`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'gym' is not defined",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-23a413fd8dbd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0menv\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mgym\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmake\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'CartPole-v0'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mSTATE_COUNT\u001b[0m  \u001b[1;33m=\u001b[0m \u001b[0menv\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mobservation_space\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mACTION_COUNT\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0menv\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0maction_space\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mn\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'gym' is not defined"
     ],
     "output_type": "error"
    }
   ],
   "source": [
    "env = gym.make('CartPole-v0')\n",
    "\n",
    "STATE_COUNT  = env.observation_space.shape[0]\n",
    "ACTION_COUNT = env.action_space.n\n",
    "\n",
    "STATE_COUNT, ACTION_COUNT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: in the cell below we highlight how one would do it in Keras. And a marked similarity with CNTK. While CNTK allows for more compact representation, we present a slightly verbose illustration for ease of learning.\n",
    "\n",
    "Additionally, you will note that, CNTK model doesn't need to be compiled explicitly and is implicitly done when data is processed during training.\n",
    "\n",
    "CNTK effectively uses available memory on the system between minibatch execution. Thus the learning rates are stated as **rates per sample** instead of **rates per minibatch** (as with other toolkits)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Targetted reward\n",
    "REWARD_TARGET = 30 if isFast else 200\n",
    "# Averaged over these these many episodes\n",
    "BATCH_SIZE_BASELINE = 20 if isFast else 50\n",
    "\n",
    "H = 64 # hidden layer size\n",
    "\n",
    "class Brain:\n",
    "    def __init__(self):\n",
    "        self.params = {}\n",
    "        self.model, self.trainer, self.loss = self._create()\n",
    "        # self.model.load_weights(\"cartpole-basic.h5\")\n",
    "\n",
    "    def _create(self):\n",
    "        observation = C.sequence.input_variable(STATE_COUNT, np.float32, name=\"s\")\n",
    "        q_target = C.sequence.input_variable(ACTION_COUNT, np.float32, name=\"q\")\n",
    "\n",
    "        # Following a style similar to Keras\n",
    "        l1 = C.layers.Dense(H, activation=C.relu)\n",
    "        l2 = C.layers.Dense(ACTION_COUNT)\n",
    "        unbound_model = C.layers.Sequential([l1, l2])\n",
    "        model = unbound_model(observation)\n",
    "\n",
    "        self.params = dict(W1=l1.W, b1=l1.b, W2=l2.W, b2=l2.b)\n",
    "\n",
    "        # loss='mse'\n",
    "        loss = C.reduce_mean(C.square(model - q_target), axis=0)\n",
    "        meas = C.reduce_mean(C.square(model - q_target), axis=0)\n",
    "\n",
    "        # optimizer\n",
    "        lr = 0.00025\n",
    "        lr_schedule = C.learning_rate_schedule(lr, C.UnitType.minibatch)\n",
    "        learner = C.sgd(model.parameters, lr_schedule, gradient_clipping_threshold_per_sample=10)\n",
    "        trainer = C.Trainer(model, (loss, meas), learner)\n",
    "\n",
    "        # CNTK: return trainer and loss as well\n",
    "        return model, trainer, loss\n",
    "\n",
    "    def train(self, x, y, epoch=1, verbose=0):\n",
    "        #self.model.fit(x, y, batch_size=64, nb_epoch=epoch, verbose=verbose)\n",
    "        arguments = dict(zip(self.loss.arguments, [x,y]))\n",
    "        updated, results =self.trainer.train_minibatch(arguments, outputs=[self.loss.output])\n",
    "\n",
    "    def predict(self, s):\n",
    "        return self.model.eval([s])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Memory` class stores the different states, actions and rewards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Memory:   # stored as ( s, a, r, s_ )\n",
    "    samples = []\n",
    "\n",
    "    def __init__(self, capacity):\n",
    "        self.capacity = capacity\n",
    "\n",
    "    def add(self, sample):\n",
    "        self.samples.append(sample)\n",
    "\n",
    "        if len(self.samples) > self.capacity:\n",
    "            self.samples.pop(0)\n",
    "\n",
    "    def sample(self, n):\n",
    "        n = min(n, len(self.samples))\n",
    "        return random.sample(self.samples, n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Agent` uses the `Brain` and `Memory` to replay the past actions to choose optimal set of actions that maximize the rewards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MEMORY_CAPACITY = 100000\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "GAMMA = 0.99 # discount factor\n",
    "\n",
    "MAX_EPSILON = 1\n",
    "MIN_EPSILON = 0.01 # stay a bit curious even when getting old\n",
    "LAMBDA = 0.0001    # speed of decay\n",
    "\n",
    "class Agent:\n",
    "    steps = 0\n",
    "    epsilon = MAX_EPSILON\n",
    "\n",
    "    def __init__(self):\n",
    "        self.brain = Brain()\n",
    "        self.memory = Memory(MEMORY_CAPACITY)\n",
    "\n",
    "    def act(self, s):\n",
    "        if random.random() < self.epsilon:\n",
    "            return random.randint(0, ACTION_COUNT-1)\n",
    "        else:\n",
    "            return numpy.argmax(self.brain.predict(s))\n",
    "\n",
    "    def observe(self, sample):  # in (s, a, r, s_) format\n",
    "        self.memory.add(sample)\n",
    "\n",
    "        # slowly decrease Epsilon based on our eperience\n",
    "        self.steps += 1\n",
    "        self.epsilon = MIN_EPSILON + (MAX_EPSILON - MIN_EPSILON) * math.exp(-LAMBDA * self.steps)\n",
    "\n",
    "    def replay(self):\n",
    "        batch = self.memory.sample(BATCH_SIZE)\n",
    "        batchLen = len(batch)\n",
    "\n",
    "        no_state = numpy.zeros(STATE_COUNT)\n",
    "\n",
    "\n",
    "        # CNTK: explicitly setting to float32\n",
    "        states = numpy.array([ o[0] for o in batch ], dtype=np.float32)\n",
    "        states_ = numpy.array([(no_state if o[3] is None else o[3]) for o in batch ], dtype=np.float32)\n",
    "\n",
    "        p = agent.brain.predict(states)\n",
    "        p_ = agent.brain.predict(states_)\n",
    "\n",
    "        # CNTK: explicitly setting to float32\n",
    "        x = numpy.zeros((batchLen, STATE_COUNT)).astype(np.float32)\n",
    "        y = numpy.zeros((batchLen, ACTION_COUNT)).astype(np.float32)\n",
    "\n",
    "        for i in range(batchLen):\n",
    "            s, a, r, s_ = batch[i]\n",
    "\n",
    "            # CNTK: [0] because of sequence dimension\n",
    "            t = p[0][i]\n",
    "            if s_ is None:\n",
    "                t[a] = r\n",
    "            else:\n",
    "                t[a] = r + GAMMA * numpy.amax(p_[0][i])\n",
    "\n",
    "            x[i] = s\n",
    "            y[i] = t\n",
    "\n",
    "        self.brain.train(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "As any learning experiences, we expect to see the initial state of actions to be wild exploratory and over the iterations the system learns the range of actions that yield longer runs and collect more rewards. The tutorial below implements the [epsilon-greedy](https://en.wikipedia.org/wiki/Reinforcement_learning) approach (a.k.a. $\\epsilon$-greedy). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def plot_weights(weights, figsize=(7,5)):\n",
    "    '''Heat map of weights to see which neurons play which role'''\n",
    "    sns.set(style=\"white\")\n",
    "    f, ax = plt.subplots(len(weights), figsize=figsize)\n",
    "    cmap = sns.diverging_palette(220, 10, as_cmap=True)\n",
    "    \n",
    "    for i, data in enumerate(weights):\n",
    "        axi = ax if len(weights)==1 else ax[i]\n",
    "        if isinstance(data, tuple):\n",
    "            w, title = data\n",
    "            axi.set_title(title)\n",
    "        else:\n",
    "            w = data\n",
    "                \n",
    "        sns.heatmap(w.asarray(), cmap=cmap, square=True, center=True, #annot=True,\n",
    "                    linewidths=.5, cbar_kws={\"shrink\": .25}, ax=axi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploration - exploitation trade-off\n",
    "\n",
    "Note that the initial $\\epsilon$ is set to 1 which implies we are entirely exploring but as steps increase we reduce exploration and start leveraging the learnt space to collect rewards (a.k.a. exploitation) as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x21b66c1ad30>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAERCAYAAACD9ivUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VOXdxvHvTPY9ZEPTWqn6qm21LiAIRQrILsimDyK4\ngFgRRQsqgiKi6ItWBUuVgoCgVi2PQEAEWRRRobKIWrVvW2ytS6UlG9n3ZN4/ZuhEqkAkh5PJ3J/r\n6lVnznDmN79kcp/nLM/x+Hw+REREmsLrdgEiIhJ6FB4iItJkCg8REWkyhYeIiDSZwkNERJpM4SEi\nIk0W6fQbGGM6AQ9Za3sc8vwg4B6gFlhqrV3sdC0iItI8HB15GGPuABYBMYc8HwnMAXoB3YFfGGMy\nnaxFRESaj9O7rf4GDP2G538EfGKtLbHW1gLbgG4O1yIiIs3E0fCw1uYAdd+wKBkobvS4FEhxshYR\nEWk+bh0wL8EfIAclAUUu1SIiIk3k+AHzAM8hj/8MnGaMSQUq8O+yeuQo1qOJuEREvptD/w4fk+MV\nHj4AY8xIIMFau9gYMxnYhP8DLbbW/utoVlTXrh2Rn39O+bXXUjxrFnjD82zj7Oxs9u3b53YZLYJ6\nEaReBKkXQdnZ2c2+Tk+Izarr+/f775M+ahRRf/4zFUOHUjR3LkRFuV3XcacvRpB6EaReBKkXQYHw\naNaRR8httje0bUv+ihXUtG9PfE4Oadddh6ey0u2yRETCSsiFB4AvNZWC3/+equ7diX39ddKuvBJP\ncfGR/6GIiDSLkAwPAF98PIVLl1J56aXE7NpFxuWX483Lc7ssEZGwELLhAUB0NAeeeILy0aOJ+tOf\nyBg6lIh//tPtqkREWr3QDg+AiAiKH3qI0ptvJvIf/yBj8GAi9+51uyoRkVYt9MMDwOOhdNo0iu+5\nh4h//5v0YcOIev99t6sSEWm1Wkd4BJSPH8+Bxx7DW1xM+uWXE/PGG26XJCLSKrWq8ACovOIKDixa\nhMfnI+3aa4lbscLtkkREWp1WFx4AVf36UfDii/gSEmhz660kzp8PoXUxpIhIi9YqwwOgpmNH8nNy\nqD/xRJIffJDkmTOhocHtskREWoVWGx4AdWecQd6aNdSefjqJixfT5qaboLra7bJEREJeqw4PgIbv\nfY/8nByqO3Yk7uWXSb/qKjylpW6XJSIS0lp9eEBgOpMXXqCyXz9itm8nY/hwvLm5bpclIhKywiI8\nAIiL48BTTwWvRh88mIi//93tqkREQlL4hAf852r0kttvJ/KLL8gYMoSoDz5wuyoRkZATXuEB4PFQ\nNmkSRQ8/jLeoiPTLLiNm82a3qxIRCSnhFx4BFaNHU7hkCfh8pI0dS/wzz7hdkohIyAjb8ACo7tOH\nghUraEhLI/Wuu0h+4AFdCyIichTCOjwAas87j/yXX6b21FNJ/O1vaTNhAlRVuV2WiEiLFvbhAVB/\n8snkr1lDdadOxK1dS/oVV+ApLHS7LBGRFkvhEeBr04aCF16gYvBgYnbvJvPSS4n47DO3yxIRaZEU\nHo3FxlL0xBPBG0sNGkTUnj1uVyUi0uIoPA7l9VI6bRpFDz2Et6iIDGOIXb/e7apERFoUhce3qLjq\nKgqXLcPn9dLmF78gYdEit0sSEWkxIp1cuTHGA8wHzgGqgHHW2k8bLb8KuB0oAp6x1j7tZD1NVX3x\nxRSsWkXaNdeQMnMmEZ9/TsnMmRDpaNtERFo8p0ceQ4AYa20XYBow5+ACY0w6cD/QDegOjDLG/MDh\nepqs9uyzyV+7ltozzyRx6VLSxozRrLwiEvacDo+uwAYAa+1OoEOjZacAH1hri621PmA3cKHD9Xwn\n9d/7HvmrV1PVowexW7b4J1X88ku3yxIRcY3T4ZEMFDd6XGeMOfienwA/McZkGmPigYuBBIfr+c58\nSUkULltG2XXXEfXXv5JxySVE7d7tdlkiIq5weud9CZDU6LHXWtsAYK0tMsZMBlYCBcAeIP9IK8zO\nznaizqO3eDGcfz4Rt9xC5ogR8PTTcOWVrpTiei9aEPUiSL0IUi+c43R4bAcGAiuMMRcCHx1cYIyJ\nAM631nYzxkQDm4C7jrTCffv2OVXr0RsyhJjUVNqMH4931ChK332X0ttuA4/nuJWQnZ3dMnrRAqgX\nQepFkHoR5ESIOr3bKgeoNsZsBx4DJhljRhpjxllr6wGMMe8BbwDzrLUhMydIdffu5L/8MnU/+AFJ\nc+f658SqrHS7LBGR48Lj8/ncrqEpfC1tS8JbUECb664jZvduas47j8Knn6YhK8vx99VWVZB6EaRe\nBKkXQYGRR7PuGtFFgseoIT2dguXLqRg+nOj33yfjkkuI/L//c7ssERFHKTyaQ0wMRb/+NSV33knk\nvn1kDBlCzKZNblclIuIYhUdz8Xgou+UWChcuhPp60saOJXHePAit3YIiIkdF4dHMqgYOpGD1ahpO\nOIHkhx+mzYQJeHQgXURaGYWHA2rPPpu8V1+l+oILiHv5ZdKHDCHiq6/cLktEpNkoPBzSkJlJwfLl\nlI8cSfTHH5MxYADRu3a5XZaISLNQeDgpJobiRx6heNYsvAcOkG4M8S+84HZVIiLHTOHhNI+H8rFj\nKXj+eXwJCaTecQfJ06dDba3blYmIfGcKj+Ok5qKLyFu/ntozziBx6VLSr7wST2HIXFAvIvI1Co/j\nqP7kk8l/+WUq+/Yl5g9/IPOSS4j8y1/cLktEpMkUHseZLzGRA4sXU3rrrUR+8QUZgwbpHukiEnIU\nHm7weimdMoXCBQvA5yPt+utJmj0b6uvdrkxE5KgoPFxUNWgQ+WvXUteuHUlPPEHa6NE6DiIiIUHh\n4bK6H/2IvPXrqerZk9i33iJzwAAiP/7Y7bJERA5L4dEC+FJSKHzmGUonTybyyy/JHDyYuJUr3S5L\nRORbKTxaCq+X0ttuo2DpUnxRUbS55RaS77lH14OISIuk8Ghhqvv0CV4P8vTTpI8YgTc31+2yRES+\nRuHRAtWfcgr5a9dSOXAgMTt3ktm/P1Hvvut2WSIi/6HwaKF8CQkcWLCA4unT8ebmknHZZcQ/95zu\nDyIiLYLCoyXzeCi/8UYKXniBhsREUqdOJXXSJN0fRERcp/AIATUXXUT+hg3UnHsu8S+9RMagQbB3\nr9tliUgYU3iEiPrvf5/8Vasov/Zaov78Z+jQgdi1a90uS0TClMIjlMTEUPzggxx48kloaCBt/HiS\nZ8yAmhq3KxORMKPwCEGVQ4bA7t3U/s//kLhkCRnDh+PVbW5F5DiKdHLlxhgPMB84B6gCxllrP220\nfBQwGagDllprFzhZT6vyox+Rv24dKVOnEr9qFZl9+1L0xBNUd+/udmUiEgacHnkMAWKstV2AacCc\nQ5Y/AvQEugK3GWNSHK6nVfElJFA0bx5Fs2fjLS8nbfRokh59VLPziojjnA6PrsAGAGvtTqDDIcv/\nCLQB4gKPdRFDU3k8VFx9NfmrV1P/ve+RNHcu6aNG4S0ocLsyEWnFnA6PZKC40eM6Y0zj9/wTsAf4\nCHjFWlvicD2tVu0555C3YQNVvXoR8/bbZPbpQ/SuXW6XJSKtlKPHPIASIKnRY6+1tgHAGHM2cAlw\nMlAOPG+MGW6tPex0stnZ2U7VGnL+qxfZ2bBxIzz8MBHTp5Nx2WVw//0wdSp4W/e5Efq9CFIvgtQL\n5zgdHtuBgcAKY8yF+EcYBxUDFUC1tdZnjMnFvwvrsPbt2+dIoaEmOzv723txzTVEn3kmbSZMIOLu\nu6nesIED8+bRkJV1fIs8Tg7bizCjXgSpF0FOhKjTm6M5QLUxZjvwGDDJGDPSGDPOWvsF8BSwzRjz\nFpACLHO4nrBR06kTeZs3U3XxxcHdWG+95XZZItJKeHyhNdGeT1sSfke9VeXzkfDUUyTPng11dZRN\nnEjpbbdBpNODzuNHW5hB6kWQehEUGHl4mnOdrXtHuPgnV7zhBvJzcqg/6SSS5s0j/fLL8epLJSLH\nQOERJmrPO4+8jRv99wjZtYus3r2J2bTJ7bJEJEQpPMKILzmZAwsWUPTQQ3gqK0kfM4bkmTM1N5aI\nNJnCI9x4PFRcdRV5a9dSe+qpJC5aRMbQoUR89pnblYlICFF4hKm6n/yE/FdfpeLyy4n+4AMy+/Qh\nbsUK3alQRI6KwiOM+RISKHr8cQ7MmwceD21uvZXUm2/GU6IL/UXk8BQeQuXw4eRt2kTN+ecTv3o1\nmb17E717t9tliUgLpvAQAOpPPpn8Vaso/eUvidi3j/Rhw0h67DGoq3O7NBFpgRQeEhQVRekdd1Cw\nYgX1J55I0pw5ZAwfTsQXX7hdmYi0MAoP+S8HpzapvPRSot99138wPSfH7bJEpAVReMg38qWkcGD+\nfA7MnQv19bS5+WZSJ07EU1rqdmki0gIoPOTbeTxUGuM/mH7uuf7b3fbpQ9S777pdmYi4TOEhR1T/\nwx+Sv3o1pRMnEvHll2QMHUrSr34FtbVulyYiLlF4yNGJiqJ06lQKXnqJ+uxskn79azIGDSJy7163\nKxMRFyg8pElqOncm77XXqDCG6I8+IrNfPxIWLYKGBrdLE5HjSOEhTeZLSqJo7lwKFy+mISGBlJkz\nSb/iCiK++srt0kTkOFF4yHdW1b8/eVu2UNWrFzHbt5PZqxdxK1dqfiyRMKDwkGPSkJlJ4bJlFD3y\nCNTV0eaWW2gzfjyewkK3SxMRByk85Nh5PFRceSV5mzdTfcEFxL3yClm9ehGzZYvblYmIQxQe0mzq\n27WjYOVKSqZNw1tYSPpVV5EydSqe8nK3SxORZqbwkOYVEUHZzTeT98or1J55JgnPPUdmr15Ev/OO\n25WJSDNSeIgj6s46i7x16yi96SYi/vlPMi67jOTp0/FUVLhdmog0A4WHOCc2ltK77iJ/zRpqTzuN\nxKVL/aOQHTvcrkxEjpHCQxxXe/755G3cSNmNN/qnNxk+nOQZMzQKEQlhkU6u3BjjAeYD5wBVwDhr\n7aeBZW2B3wM+wAOcC9xprX3KyZrEJbGxlEyfTmX//qROmkTikiXEvv46RXPmUNOpk9vViUgTOT3y\nGALEWGu7ANOAOQcXWGv3W2t7WGt7BpbtARY5XI+4rLZ9e/8oZPx4Ij7/nPThw0m+9148lZVulyYi\nTeB0eHQFNgBYa3cCHb7ldb8BxltrdWlyOIiLo+See8hfvZr6H/6QxMWLdd90kRDjdHgkA8WNHtcZ\nY772nsaYQcDH1tq/OVyLtDC1HTqQu2kTZTfcQMRnn5E+dKh/FKJjISItnqPHPIASIKnRY6+19tDp\nV0cDjx/tCrOzs5ujrlah1fRiwQK4+moYM4bExYtJfP11eOop6NXrqFfRanrRDNSLIPXCOU6Hx3Zg\nILDCGHMh8NE3vKaDtfaoryDbt29fc9UW0rKzs1tXL9q1g/XrSZo7l8QFC/D07k3FiBEUz5iBLzX1\nsP+01fXiGKgXQepFkBMh6vRuqxyg2hizHXgMmGSMGWmMGQdgjMng67u1JJzFxfmvC1m3jtqf/IT4\n5cvJ6tGD2HXr3K5MRA7h8YXW9Nk+bUn4tfqtqtpaEhcuJGnOHDzV1VQOGEDxAw/Q0Lbtf7201fei\nCdSLIPUiKDDy8DTnOnWRoLRMUVGU3XwzuZs2Ud2pE3Hr15PVowfxL76o+4WItAAKD2nR6k87jYIV\nKyiaPRvq60m9/XbSR4wg4rPP3C5NJKwpPKTl83qpuPpqct94I3jXwosvJmHBAqirc7s6kbCk8JCQ\n0ZCdTeGyZRTOn48vPp6UWbPIGDQI9uxxuzSRsKPwkNDi8VA1eDB5b75JxfDhRH/4IXTs6J9osazM\n7epEwobCQ0JSQ1oaRfPmkb98OZx6KolLlpD1858Tu2GD26WJhAWFh4S0mq5d4cMPKZ08GW9hIWnX\nXUebMWOI+Oort0sTadUUHhL6YmMpve02cjdvprpzZ+I2bSKze3cSFi7UAXURhyg8pNWoP+00Cl56\niQNz5+KLiSHl/vvJHDCAqA8+cLs0kVZH4SGti8dDpTHkvfUWFSNGEPWnP5ExcKD//uklJW5XJ9Jq\nKDykVWpIS6NozhzyX3qJulNOIXHpUv88WWvW6Ap1kWZwVOFhjDnzkMcpzpQj0rxqunQhb/NmSm6/\nHe+BA6RNmED6FVcQ+TfdPkbkWBztyONnAMaY0YHHFztTjogDYmIomzSJ3C1bqOrZk5ht28js1Yuk\n//1fPOXlblcnEpKONjzONcakA/GBxxr3S8ipb9eOwmefpfDpp6lv25akJ5/0XxvyyivalSXSREcb\nHkuAPcC1xpg7gdOdK0nEQR4PVX37krd1K6W33oq3oIC0G24g7coridCuLJGjdlThYa39AP+uq5eA\n/cCjThYl4jRfXBylU6aQ+/rrVHXvTuxbb5HVqxdJs2frHuoiR0E3gwpRutFN0DH3wucjdsMGku+9\nl8ivvqIuO5uS++6jqn9/8DTr/XMcp9+LIPUiSDeDEnGCx0NV//7+XVkTJxKRl0fa9deTNnq0dmWJ\nfAuFh0iALz6e0qlTyX3tNaq6dSN261ayLr6Y5Pvv1wWGIodQeIgcov600yh84QUKFy2i/sQTSVy4\nkKyLLvLfArehwe3yRFoEhYfIN/F4qBowgNytWymZMgVPeTmpt99OxiWXEL17t9vVibhO4SFyOLGx\nlN16K7lvv03FsGFEf/ghGUOGkHrzzXh1MFbCmMJD5Cg0nHgiRb/5DXmrV1NzzjnE5+SQ1a0biY8/\nDpWVbpcnctwpPESaoPaCC8h/5RUOzJmDLzGR5Ece8U+4uG6drlKXsBLp5MqNMR5gPnAOUAWMs9Z+\n2mj5BcBjgYf/BkZba2ucrEnkmHm9VI4YQdWAAST9+tckLF5M2i9+QXWXLhTfdx91P/6x2xWKOM7p\nkccQIMZa2wWYBsw5ZPlTwLXW2m7ABuBkh+sRaTa+pCRKpk/3X6Xeqxcxf/gDmX36kHL77Xj373e7\nPBFHOR0eXfGHAtbanUCHgwuMMacDBcBkY8xWIM1a+4nD9Yg0u/pTT6XwmWcoeP556s44g4QXXySr\na1cS587VVCfSajkdHslAcaPHdcaYg++ZAXQG5gG9gF7GmO4O1yPimOru3cnbuJGiX/0KX3w8yY8+\nStZFFxFnra4PkVbH0WMeQAmQ1Oix11p78FtUAPzNWrsXwBizAf/IZOvhVhiYo0VQLxprUb244w4Y\nPx4efpiIxx6jzaRJtHn2WXjsMejRw/G3b1G9cJl64Rynw2M7MBBYYYy5EPio0bJPgURjzCmBg+gX\nAYuPtEJNdOanSd+CWmwvJkzAO3gwyQ8/TPzKldCzJ5V9+lBy993Un3aaI2/ZYnvhAvUiyIkQdXRW\n3UZnW/008NQYoD2QYK1dHNhN9XBg2R+stZOOsErNqhugL0ZQKPQi6o9/JPn++4nZsQNfZCTlV19N\n2aRJNKSlNev7hEIvjhf1IsiJWXU1JXuI0hcjKGR64fMRu3EjyQ88QOQ//kFDcjKlt9xC+ZgxEBvb\nLG8RMr04DtSLIE3JLhLKPB6q+vUjd8sWiu+7D7xeUh54wH9QfflyqK93u0KRo6bwEDneoqMpHzeO\n/du3U3rTTUQUFtJm8mQye/cmZvNmXakuIUHhIeISX2oqpXfdxf6336Z85EgiP/mE9GuvJX34cKLe\nfdft8kQOS+Eh4rKG7GyKH32UvNdfp7JvX2J27iRz8GDajBtHpO5kKC2UwkOkhag7/XQOPP00+Tk5\n1HToQNyrr5LZowcpU6bg/de/3C5P5GsUHiItTE3HjuSvXk3h009Td+qpJDz/PFldu5I0ezae4uIj\nr0DkOFB4iLREHg9VffuS99prFD36KL7UVJKeeIK2XbqQ8Nvf4tE9RMRlCg+RliwykoqRI8ndto2S\nu+6Chgb/6b1duhC/bBlUV7tdoYQphYdICPDFxVF2003sf+cdSm+5BU9ZGal3301Wt27E/f73UFfn\ndokSZhQeIiHEl5pK6Z13kvvOO5Rdfz0ReXm0ue02/90M16zR7L1y3Cg8REJQQ0YGJTNnsn/bNspH\njybiiy9ImzABzj+fmE2bdKGhOE7hIRLCGrKzKX74YXLffJOK4cPhww9JHzOGjEGDiH77bbfLk1ZM\n4SHSCtS3a0fRvHnw8cdUDhhA9Pvvk3HFFaRffjlRu3e7XZ60QgoPkdbkxz/mwKJF5L36KlU9e/rv\nqz5kCGmjRmnKE2lWCg+RVqj2pz+l8LnnyM/JobprV2K3biVz8GCFiDQbhYdIK1bTsSMFy5eTv2qV\nQkSalcJDJAzUdOqkEJFmpfAQCSMKEWkuCg+RMKQQkWOl8BAJY98aIiNHEr1jh9vlSQum8BCR/w6R\nt94iY/hw0ocOJeaNN3TFuvwXhYeI/MfBEMlbs4aqiy8mZtcu0kePJmPAAGLXr9fcWfIfCg8R+S+1\nHTpQ+Oyz5G7cSOXAgUR99BFp119P5sUXE7dqlWbxFYWHiHy7urPO4sDCheRt3UrF5ZcT+fe/02bi\nRLK6dSP++ed1P5Ew5vE5uC/TGOMB5gPnAFXAOGvtp42W/xIYB+QGnrrBWvvJYVbp27dvn1PlhpTs\n7GzUCz/1IsjpXkR8+SWJ8+cTv3w5nupq6k84gbIbb6Ri1Ch8cXGOve93od+LoOzsbABPc67T6ZHH\nECDGWtsFmAbMOWR5e+Aqa23PwP8OFxwi4rL6k06iePZs9r/zDmU33ICnuJiUe+8lq1MnEn/zG91j\nPYw4HR5dgQ0A1tqdQIdDlrcHphlj3jbGTHW4FhFpJg1t21IyYwa5u3ZReuuteGprSX7oIdp27Ejy\nrFl4//Uvt0sUhzkdHslA402ROmNM4/d8ERgP9AC6GmMGOFyPiDSjhrQ0SqdMYf/OnRRPn44vIYHE\nBQto27kzqZMmEbl3r9slikMiHV5/CZDU6LHXWtv4XL9fW2tLAIwx64DzgPWHW2Fg352gXjSmXgS5\n0ovsbJg1C6ZPh+efx/PII8RbS7y1MGgQTJkCXbu6UJZ+L5zidHhsBwYCK4wxFwIfHVxgjEkGPjbG\nnAlUAj2BJUdaoQ6A+elgYJB6EdQietGvH/TpQ+zmzSQ++STRa9fC2rXUdOhA2YQJVPXuDV7nT/Rs\nEb1oIZwIUad/gjlAtTFmO/AYMMkYM9IYMy4w4pgGbAXeBD621m5wuB4ROR68Xqr69iX/5ZfJX72a\nqt69iX73XdLGjiWze3fiX3xRp/mGOEdP1XWATtUN0FZVkHoR1JJ7Ebl3L4kLFhC3ahWe2lrq27al\nfNw4ykeNwpeS0uzv15J7cbyF4qm6IiIA1J1+OkVz5vhP8x0/Hk95OckPPkjbCy4gecYMIj7/3O0S\npQkUHiJyXDWceCIl99zD/l27KLn7bnxJSSQuWUJW1660uf56onft0kSMIUDhISKu8KWkUDZhAvt3\n7ODAk09Se/bZxK1fT8bQoWQMHEjsmjVQW+t2mfItFB4i4q6oKCqHDCF/3TryV62isn9/ov74R9Im\nTCCrSxcSfvtbXbneAik8RKRl8Hio6dSJA4sXk7ttG2Vjx+ItKiLlgQdo26GD/7jIZ5+5XaUEKDxE\npMWpb9eOklmz2L97t//K9ZSU4HGRceOI3rlTx0VcpvAQkRbLl5pK+Y03sv+dd/zHRX76U+JefZWM\nYcPI7NPHf71IZaXbZYYlhYeItHyNj4vk5FA5cCCRf/0rqbffzgkdOpD04INE/POfblcZVhQeIhI6\nPB5qOnbkwMKF7N+xg9KJE/F5vSTNn09W587+XVrbt2uX1nGg8BCRkNSQnU3p1Kns372bA3PnUnvW\nWf5dWsaQ2asXLFyIp6LC7TJbLYWHiIS22FgqjSF//Xry1qyhYvBgIv/2Nxg/3n+W1n336ep1Byg8\nRKR18Hio7dCBovnz2b9zJ8yYgS86msSnniLrZz8j7ZpriNm6FRoajrgqOTKFh4i0Og0nnAD33cf+\nnTs5MG8eteeeS+xrr5E+ahRZXbuSOH8+3sJCt8sMaQoPEWm9YmKoHD6c/FdeIW/dOipGjCBi/37/\nhIzt25M6cSLRu3frAPt3oPAQkbBQe+65FM2Zw7/37KH4vvuoO+kk4letImPIEDJ79yZ+2TI8paVu\nlxkyFB4iElZ8qamUjxtH3ptvkm+t/5qRTz4h9e67adu+PSl33knkn/7kdpktnsJDRMKTx0PNz37m\nv2Zk925KpkyhITWVhN/9jqw+fci49FLiXnoJqqrcrrRFUniISNhryMqi7NZbyX3nHQqWLaOqZ0+i\n3nuPNr/8JSe0b0/yzJlE7t3rdpktisJDROSgiAiqe/em8LnnyH3nHUpvvhlfZCSJixaR1aMHGYMH\nE7d8uS4+ROEhIvKN6k86idJp09i/ezeFTz1FVY8eRO3ZQ5vJk2l73nmk3HknUX/8Y9ieqaXwEBE5\nnOhoqi65hMLf/Y7cHTsonTyZhuRkEn73OzIHDCCzb1//mVphdsMqhYeIyFGq//73Kb3tNnJ37KDg\nueeoHDDAP7vv3Xdzwvnnk3rLLUTv2BEWo5FItwsQEQk5ERFU9+xJdc+eeHNziV+xgvgXXiB+5Uri\nV66k7pRTKL/ySiovu4yGzEy3q3WERh4iIsegISuLsgkTyH37bfJXrKBi2DAivvrqP7fPbTNmDLGv\nvgo1NW6X2qwcHXkYYzzAfOAcoAoYZ6399BtetxAosNbe5WQ9IiKO8Xio6dyZms6dKZ41i/hVq4hb\nvpy4TZuI27SJ+jZtqBw2jApjqDvrLLerPWZOjzyGADHW2i7ANGDOoS8wxtwAhH4nRUQCfKmplI8d\nS/7GjeRu2kTZ9deD1+u/D3vfvmT27k3CokV48/PdLvU7czo8ugIbAKy1O4EOjRcaYzoDFwALHa5D\nRMQVdT/5CSUzZ7J/zx4Kli6lsn9/IvfuJWXmTNq2b0+bsWOJ3bABamvdLrVJnA6PZKDx+Wt1xhgv\ngDHmBOBe4GbA43AdIiLuioqiuk8fDixezP733vNPznjGGcRt3EjaddfRtn17ku+9N2Tm1XL6bKsS\nIKnRY6+19uCdWC4H0oH1wIlAnDHmL9baZw+3wuzsbEcKDUXqRZB6EaReBLXYXmRnw9lnw4wZ8MEH\nsGwZEc+3ti12AAAIgUlEQVQ/T+LixSQuXgznngtXXQUjR8KJJ7pd7Tfy+Bw8H9kYMwwYaK0da4y5\nELjHWnvJN7zuGuCMozhg7tu3b58TpYac7Oxs1As/9SJIvQgKuV7U1BC7ZQtx1hL7+ut46urweb1U\nX3QRlcOGUdW/P76EhO+06kCINuseHqdHHjlAb2PM9sDjMcaYkUCCtXaxw+8tIhI6oqOp6tePqn79\n8BYUELt2LfErVxL75pvEvvkmDVOnUtWvH5XDhlHdrRtEunuZnqMjDwdo5BEQcltVDlIvgtSLoNbS\ni4hPPyU+J4e4VauI/OwzAOozMqgcPJjKYcOoPecc8Bx+UOHEyEMXCYqItGD1p5zinxJl2zbyXn6Z\n8muvhfp6EpcsIfOSS8j8+c9JfPxxIr744rjWpfAQEQkFHg+17dtT/OCD7H//fQqWLaPy0kuJ/Oor\nkh95hLadO5M+ZAjxzzyDt7DQ8XI0t5WISKiJiqK6d2+qe/fGU1pK7Pr1xK9aRfT27cTs3o1vxgyq\nu3WjcsgQqvr2daQEhYeISAjzJSVROWIElSNG4P3Xv4hbs4a4NWuI3bKF2C1baIiNhcrKZn9fhYeI\nSCvRcOKJlI8fT/n48UT8/e/+IFm3zpHjEzrmISLSCtWfeiplkyeT9/rrjqxf4SEiIk2m8BARkSZT\neIiISJMpPEREpMkUHiIi0mQKDxERaTKFh4iINJnCQ0REmkzhISIiTabwEBGRJlN4iIhIkyk8RESk\nyRQeIiLSZAoPERFpMoWHiIg0mcJDRESaTOEhIiJN5uhtaI0xHmA+cA5QBYyz1n7aaPlw4E6gAXjB\nWjvPyXpERKR5OD3yGALEWGu7ANOAOQcXGGO8wP8CPYEuwARjTJrD9YiISDNwOjy6AhsArLU7gQ4H\nF1hrG4AfWWvLgIxALTUO1yMiIs3A6fBIBoobPa4LjDgAf4AYY4YCHwBbgXKH6xERkWbgdHiUAEmN\n3y8w4vgPa22OtTYbiAGudrgeERFpBo4eMAe2AwOBFcaYC4GPDi4wxiQBa4E+1toa/KOOhm9cSyPZ\n2dkOlRp61Isg9SJIvQhSL5zj8fl8jq280dlWPw08NQZoDyRYaxcbY8YB4/Af6/gQmGitda4gERFp\nFo6Gh4iItE66SFBERJpM4SEiIk2m8BARkSZTeIiISJM5fapuszjSHFmthTEmEngaaAdEAw8C/wcs\nw38a88fW2psCr70e+AVQCzxorV1njIkFfgdk4b/G5hprbcFx/hjNyhiTBbwL9ALqCdNeGGOmApcC\nUfi/C28Rhr0IfEeewf8dqQOuJwx/L4wxnYCHrLU9jDGncoyfP3ApxeOB12621t5/pBpCZeTxrXNk\ntTKjgXxrbTegH/AE/s96l7X254DXGDPYGNMWmAh0DrxutjEmCrgR+DDw758D7nHjQzSXwB+KBUBF\n4Kmw7IUx5udA58Dvf3fgB4RpL4ABQIS19mfALPzz44VVL4wxdwCL8F9YDc3z+X8LXGGtvQjoZIw5\n50h1hEp4fOscWa2MJfjDjMC/ZXW+tfbtwHOvAr2BjsA2a22dtbYE+AT/qOw/fQq8ttfxKtwhj+L/\npd4HeAjfXvQFPjbGrAZeBl4hfHuxF4gM7I1Iwb+lHG69+BswtNHj9sfw+S8OXLAdba39LPD8Ro6i\nL6ESHoedI6u1sNZWWGvLAz/Ml4C78f/RPKgUfy+S+Ho/yvB/kRo/f/C1IckYcy2Qa63dTLAHjX/m\nYdML/BOHtgcuw7/l+Dzh24sy4IfAX4CFwDzC7Dtirc3Bv2F50LF8/oPPlRyyjpQj1REqf4CPOEdW\na2GMOQnYAjxjrf09X5+yJQkowt+P5EOeP8DX+3TwtaFqDNDbGPMG/i2mZ4HMRsvDqRcFwMbAVuRe\n/Mf9Gn+5w6kXk4AN1tozCP5eRDdaHk69OOhY/0YcGqJH1ZdQCY/t+Pd1cugcWa1JYD/lRmCKtfaZ\nwNPvG2O6Bf67P/A2sBvoaoyJNsakAGcCHwN/INCnwP+/TYiy1v7cWtvDWtsD/6zLVwGvhmMvgG34\n91tjjMkGEoDXA8dCILx6UUhwy7kI/0k/74dpLw5671i+F9baUqDaGPPDwO7AvhxFX0IlPHLwf7jt\nwGP4tz5ao2lAKnCPMeYNY8wWYDpwf+CzRwErrLX78Q/XtwGv4T9YVoP/+MBZxpi38c8Zdp8bH8JB\ntxOGvbDWrsP/B3IXsAb/rqvbgPvCrRf4zwhqb4x5C/9nnArcRHj24qDm+F6MB14AdgDvWWt3H+lN\nNbeViIg0WaiMPEREpAVReIiISJMpPEREpMkUHiIi0mQKDxERaTKFh4iINJnCQ+Q7MMZsNMakuV2H\niFsUHiLfTW+3CxBxky4SFDkMY0wCsBQ4Df8cQu/hnxLjWvzT5AwAfPinzz8J/xW+v7fWPmSMORl4\nE9iKfx4mgInW2m3H8SOIOEIjD5HDGwokWmvPxz/NtQ94ILCsu7X2K/z3RVhirb0A6IR/QsfLAq/5\nAfCqtfY8/NPPLDfGRBzXTyDiAI08RA7DGNMO/+jhU2AzkGOt/bMxpgH/VOlV+Gcq/ZDg1NgJ+O/N\nsgjYY63NaLS+L4DB1tr3j9uHEHGARh4ihxG4Qc5p+O9YlwS8ZowZjn8EAv6bdoH/Tn/nBUYYnQOv\nh6/fdwH837l6R4sWOQ4UHiKHYYwZDyyz1m621k4DNgFn4Q+AqMB01jvwz2yKMSYV/y0EBgdWkWWM\n6RNYNgiooZXeUkDCi3ZbiRyGMSYeWIL/gHc58DlwHfAUcD7+kCgDngROxn/A/AVr7azAAfM/AyuB\ns/Hfi/0Ga63CQ0KewkPEIYHw+Nham3TEF4uEGO22EnGWts6kVdLIQ0REmkwjDxERaTKFh4iINJnC\nQ0REmkzhISIiTabwEBGRJlN4iIhIk/0/EsCvL/Kgj48AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x21b66bd43c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def epsilon(steps):\n",
    "    return MIN_EPSILON + (MAX_EPSILON - MIN_EPSILON) * np.exp(-LAMBDA * steps)\n",
    "plt.plot(range(10000), [epsilon(x) for x in range(10000)], 'r')\n",
    "plt.xlabel('step');plt.ylabel('$\\epsilon$')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to train our agent using **DQN**. Note this would take anywhere between 2-10 min and we stop whenever the learner hits the average reward of 200 over past 50 batches. One would get better results if they could train the learner until say one hits a reward of 200 or higher for say larger number of runs. This is left as an exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'isFast' is not defined",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-6f071f9600e9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mTOTAL_EPISODES\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m2000\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0misFast\u001b[0m \u001b[1;32melse\u001b[0m \u001b[1;36m3000\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mrun\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0magent\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m     \u001b[0ms\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0menv\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreset\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0mR\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'isFast' is not defined"
     ],
     "output_type": "error"
    }
   ],
   "source": [
    "TOTAL_EPISODES = 2000 if isFast else 3000\n",
    "\n",
    "def run(agent):\n",
    "    s = env.reset()\n",
    "    R = 0\n",
    "\n",
    "    while True:\n",
    "        # Uncomment the line below to visualize the cartpole\n",
    "        # env.render()\n",
    "\n",
    "        # CNTK: explicitly setting to float32\n",
    "        a = agent.act(s.astype(np.float32))\n",
    "\n",
    "        s_, r, done, info = env.step(a)\n",
    "\n",
    "        if done: # terminal state\n",
    "            s_ = None\n",
    "\n",
    "        agent.observe((s, a, r, s_))\n",
    "        agent.replay()\n",
    "\n",
    "        s = s_\n",
    "        R += r\n",
    "\n",
    "        if done:\n",
    "            return R\n",
    "\n",
    "agent = Agent()\n",
    "\n",
    "episode_number = 0\n",
    "reward_sum = 0\n",
    "while episode_number < TOTAL_EPISODES:\n",
    "    reward_sum += run(agent)\n",
    "    episode_number += 1\n",
    "    if episode_number % BATCH_SIZE_BASELINE == 0:\n",
    "        print('Episode: %d, Average reward for episode %f.' % (episode_number,\n",
    "                                                               reward_sum / BATCH_SIZE_BASELINE))\n",
    "        if episode_number%200==0:\n",
    "            plot_weights([(agent.brain.params['W1'], 'Episode %i $W_1$'%episode_number)], figsize=(14,5))\n",
    "        if reward_sum / BATCH_SIZE_BASELINE > REWARD_TARGET:\n",
    "            print('Task solved in %d episodes' % episode_number)\n",
    "            plot_weights([(agent.brain.params['W1'], 'Episode %i $W_1$'%episode_number)], figsize=(14,5))\n",
    "            break\n",
    "        reward_sum = 0\n",
    "agent.brain.model.save('dqn.mod')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you run it, you should see something like\n",
    "```\n",
    "Episode: 20, Average reward for episode 20.700000.\n",
    "Episode: 40, Average reward for episode 20.150000.\n",
    "Episode: 60, Average reward for episode 21.100000.\n",
    "...\n",
    "Episode: 960, Average reward for episode 20.150000.\n",
    "Episode: 980, Average reward for episode 26.700000.\n",
    "Episode: 1000, Average reward for episode 32.900000.\n",
    "Task solved in 1000 episodes\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task 1.1**\n",
    "Rewrite the model without using the layer lib.\n",
    "\n",
    "**Task 1.2**\n",
    "Play with different [learners](https://cntk.ai/pythondocs/cntk.learner.html#module-cntk.learner). Which one works better? Worse? Think about which parameters you would need to adapt when switching from one learner to the other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the DQN model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-05-16 20:21:24,188] Making new env: CartPole-v0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('CartPole-v0')\n",
    "\n",
    "num_episodes = 10  # number of episodes to run\n",
    "\n",
    "modelPath = 'dqn.mod'\n",
    "root = C.load_model(modelPath)\n",
    "\n",
    "for i_episode in range(num_episodes):\n",
    "    print(i_episode)\n",
    "    observation = env.reset()  # reset environment for new episode\n",
    "    done = False\n",
    "    while not done: \n",
    "        if not 'TEST_DEVICE' in os.environ:\n",
    "            env.render()\n",
    "        action = np.argmax(root.eval([observation.astype(np.float32)]))\n",
    "        observation, reward, done, info  = env.step(action)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Policy gradient (PG)\n",
    "**Goal:**\n",
    "$$\n",
    "\\text{maximize } E [R | \\pi_\\theta]\n",
    "$$\n",
    "\n",
    "**Approach:**\n",
    "1. Collect experience (sample a bunch of trajectories through $(s,a)$ space)\n",
    "2. Update the policy so that _good_ experiences become more probable\n",
    "\n",
    "**Difference to DQN:**\n",
    " * we don't consider single $(s,a,r,s')$ transitions, but rather use whole episodes for the gradient updates\n",
    " * our parameters directly model the policy (output is an action probability), whereas in DQN they model the value function (output is raw score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rewards\n",
    "\n",
    "Remember, we get +1 reward for every time step, in which we still were in the game.\n",
    "\n",
    "The problem: we normally do not know, which action led to a continuation of the game, and which was actually a bad one. Our simple heuristic: actions in the beginning of the episode are good, and those towards the end are likely bad (they led to losing the game after all)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def discount_rewards(r, gamma=0.999):\n",
    "    \"\"\"Take 1D float array of rewards and compute discounted reward \"\"\"\n",
    "    discounted_r = np.zeros_like(r)\n",
    "    running_add = 0\n",
    "    for t in reversed(range(0, r.size)):\n",
    "        running_add = running_add * gamma + r[t]\n",
    "        discounted_r[t] = running_add\n",
    "    return discounted_r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x21b79d49470>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATcAAACSCAYAAAAghI3bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADMVJREFUeJzt3X9M1PUfB/DnXSc/BATiC6tZAbOxfrDV1Bp9He5ykqdr\nLQRXciAu1gZkI6I6+aGETH60trKGeeTKdbCxNmnZP1JkWvJtJS2YtNHmwCWnIqAEQsj9eH//cCKZ\ninzu84bj3fOxuXFyfF6vO70nn8993q/7GIQQAkREijHOdwNERDIw3IhISQw3IlISw42IlMRwIyIl\nMdyISEl3FG6dnZ3IysoCAPzxxx/IyMhAZmYmKioqpDZHRKTVjOG2f/9+lJWVweVyAQCqq6vx+uuv\no6GhAV6vF62trdKbJCKarRnDLTY2FnV1dVO3f/vtN6xcuRIAsHr1avz444/yuiMi0sg00x1SUlLg\ndDqnbk8faAgJCcHo6Ohtf35iYgJdXV2Ijo7GXXfd5UOrRETXeTweDAwMIDExEUFBQf/4/ozhdiOj\n8frO3tjYGJYsWXLb+3d1dcFqtc62DBHRHWlsbJw6mpxu1uH2yCOP4MSJE3jiiSfw/fffIykp6bb3\nj46OnmogJiYGe5r/h0uX/5pt2RlFhgajYON//xa+RKSu8+fPw2q1TmXMjWYdbjabDTt27IDL5cKy\nZctgsVhue/9rh6L33HMP7r33XkwYgzHq1X9WP9gYjKVLl/LQl+hf5lav+TsKt6VLl6KpqQkAEBcX\nB4fDoV9nREQS8BiOiJTEcCMiJTHciEhJDDciUhLDjYiUxHAjIiUx3IhISbNexLuQCCHg9XqlbNto\nNMJgMEjZNhH5Tulw83q92OU4gsGRcV23+58li7Ezaw2nIYj8mNLhBgCDI+O4MKxvuBGR/+N7bkSk\nJIYbESmJ4UZESmK4EZGSGG5EpCRNZ0vdbjdsNhucTidMJhMqKysRHx+vd29ERJpp2nM7duwYvF4v\nmpqakJ+fj/fee0/vvoiIfKIp3OLi4uDxeCCEwOjoKBYtWqR3X0REPtF0WBoSEoK+vj5YLBYMDw/D\nbrfr3deCxHEvIv+hKdwOHDiA5ORkFBYWor+/H1u2bMFXX32FgIAAvftbUDjuReQ/NIVbeHg4TKar\nPxoWFga32y1tj2Wh4bgXkX/QFG7Z2dkoKSmB1WqF2+1GUVHRTa/4TEQ0XzSF2+LFi/H+++/r3QsR\nkW64iJeIlMRwIyIlMdyISEkMNyJSEsONiJTEcCMiJSl/DQWVcdyL6NYYbgsYx72Ibo3htsBx3Ivo\n5vieGxEpieFGREpiuBGRkhhuRKQkhhsRKUnz2dL6+nocOXIELpcLGRkZSEtL07MvIiKfaAq3n3/+\nGb/++iuampowPj6OTz75RO++iIh8oincjh8/joSEBOTn52NsbAxvvfWW3n0REflEU7hdunQJZ8+e\nhd1ux5kzZ5CXl4fDhw/r3Rv5GY570UKiKdwiIiKwbNkymEwmxMfHIzAwEBcvXsTdd9+td3/kRzju\nRQuJpnBbsWIFHA4Htm7div7+fkxMTCAyMlLv3sgPcdyLFgpN4WY2m9He3o709HQIIVBeXs5DCiLy\nK5qXgrzxxht69kFEpCsu4iUiJTHciEhJDDciUhLDjYiUxHAjIiUx3IhISbyGAvktjnuRLxhu5Lc4\n7kW+YLiRX+O4F2nF99yISEkMNyJSEsONiJTEcCMiJTHciEhJPoXb0NAQzGYzent79eqHiEgXmsPN\n7XajvLwcQUFBevZDRKQLzeFWW1uLzZs3IyYmRs9+iIh0oSncmpubERUVhVWrVkEIoXdPRPNCCAGP\nxyPlD18nc0/ThEJzczMMBgPa2trQ3d0Nm82Gjz76CFFRUXr3RzRnOO6lFk3h1tDQMPV1VlYWdu3a\nxWAjJXDcSx0+LwXhJysQkT/yeXD+s88+06MPIiJdcREvESmJ4UZESmK4EZGSGG5EpCSGGxEpieFG\nREriNRSI5gmv7iUXw41onnDcSy6GG9E84riXPHzPjYiUxHAjIiUx3IhISQw3IlISw42IlKTpbKnb\n7UZJSQmcTidcLhdyc3OxZs0avXsjItJMU7gdOnQIkZGReOedd/Dnn3/i+eefZ7gRkV/RFG7r16+H\nxWIBcHUhosnE5XJE5F80pVJwcDAA4PLlyygoKEBhYaGuTRGRvmSOegH+Oe6leZfr3Llz2LZtGzIz\nM7FhwwY9eyIincka9QL8d9xLU7gNDg4iJycHO3fuRFJSkt49EZEE/7ZRL01LQex2O0ZGRrB3715k\nZWVhy5YtmJyc1Ls3IiLNNO25lZaWorS0VO9eiIh0w0W8RKQkhhsRKYnhRkRKYrgRkZIYbkSkJIYb\nESmJ4UZESuLEOxHpbi5mWWfCcCMi3c3FLOtMGG5EJMV8z7LyPTciUhLDjYiUxHAjIiUx3IhISQw3\nIlKSprOlQgi8/fbb+P333xEQEIDdu3fj/vvv17s3IiLNNO25tba2YnJyEk1NTSgqKkJ1dbXefRER\n+URTuP3yyy9ITk4GADz22GPo6urStSkiIl9pOiy9fPkywsLCrm/EZILX673pSITH4wEAnD9/Hl6v\nF0HevxBmnNDY7q0FeQ1wOp1/60FWvZvVYj3W8+d6qrz2pte7cOECgOsZcyODEELMduM1NTV4/PHH\npy7MbDabcfTo0Zvet729HVardbYliIjuSGNjI1auXPmPv9e057Z8+XJ89913sFgs6OjoQEJCwi3v\nm5iYiMbGRkRHR/vddQ2JaOHyeDwYGBhAYmLiTb+vac9t+tlSAKiurkZ8fLxvnRIR6UhTuBER+Tsu\n4iUiJTHciEhJDDciUpJffljlfI13dXZ24t1334XD4ZBax+12o6SkBE6nEy6XC7m5uVizZuZPFtXK\n6/WirKwMvb29MBqNqKiowIMPPiitHgAMDQ0hLS0Nn3766ZycbNq4cSNCQ0MBAPfddx+qqqqk1aqv\nr8eRI0fgcrmQkZGBtLQ0abW++OILNDc3w2Aw4MqVK+ju7kZbW9vUY9Wb2+2GzWaD0+mEyWRCZWWl\n1H+/yclJFBcXo6+vD6GhoSgvL8cDDzygz8aFH/r666/F9u3bhRBCdHR0iLy8POk1P/74Y/Hss8+K\nF154QXqtgwcPiqqqKiGEEMPDw8JsNkut980334iSkhIhhBA//fST9OfT5XKJV155Raxbt0709PRI\nrSWEEFeuXBGpqanS6whx9fnLzc0VQggxNjYmPvzwwzmpK4QQFRUV4vPPP5dao7W1Vbz22mtCCCHa\n2trEq6++KrVeQ0OD2LFjhxBCiJ6eHvHSSy/ptm2/PCydj/Gu2NhY1NXVSa8DAOvXr0dBQQGAq3tV\nJpPcHei1a9eisrISAOB0OhEeHi61Xm1tLTZv3oyYmBipda7p7u7G+Pg4cnJysHXrVnR2dkqrdfz4\ncSQkJCA/Px95eXl4+umnpdWa7uTJkzh16hQ2bdoktU5cXBw8Hg+EEBgdHcWiRYuk1jt16hRWr14N\nAIiPj0dPT49u2/bLw9LZjHfpJSUlBU6nU9r2pwsODgZw9XEWFBSgsLBQek2j0Yjt27ejtbUVH3zw\ngbQ6zc3NiIqKwqpVq7Bv3z5pdaYLCgpCTk4ONm3ahNOnT+Pll19GS0uLlP8vly5dwtmzZ2G323Hm\nzBnk5eXh8OHDute5UX19PbZt2ya9TkhICPr6+mCxWDA8PAy73S613sMPP4yjR49i7dq16OjowIUL\nFyCEgMFg8HnbfrnnFhoairGxsanbsoNtPpw7dw7Z2dlITU3Fhg0b5qRmTU0NWlpaUFZWhokJ/Wf+\ngKvh1tbWhqysLHR3d8Nms2FoaEhKrWvi4uLw3HPPTX0dERGBgYEBKbUiIiKQnJwMk8mE+Ph4BAYG\n4uLFi1JqXTM6OorTp0/jySeflFoHAA4cOIDk5GS0tLTg0KFDsNlsmJyclFYvLS0NISEhsFqt+Pbb\nb/Hoo4/qEmyAn4bb8uXLcezYMQCYcbxLb2IO1jQPDg4iJycHb775JlJTU6XX+/LLL1FfXw8ACAwM\nhNFolPbLoqGhAQ6HAw6HAw899BBqa2sRFRUlpdY1Bw8eRE1NDQCgv78fY2NjiI6OllJrxYoV+OGH\nH6ZqTUxMIDIyUkqta06cOIGkpCSpNa4JDw+fOlkRFhYGt9st9fqjJ0+exFNPPYXGxkasW7dO1xOH\nfnlYmpKSgra2Nrz44osAMKefF6fXb43bsdvtGBkZwd69e1FXVweDwYD9+/cjICBASr1nnnkGxcXF\nyMzMhNvtRmlpqbRa083FcwkA6enpKC4uRkZGBoxGI6qqqqSFt9lsRnt7O9LT0yGEQHl5ufTH2dvb\nO2cfBpudnY2SkhJYrVa43W4UFRUhKChIWr3Y2Fjs2bMH+/btw5IlS7B7927dts3xKyJSkl8elhIR\n+YrhRkRKYrgRkZIYbkSkJIYbESmJ4UZESmK4EZGSGG5EpKT/A5cLN67q6saCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x21b79d29d30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "discounted_epr = discount_rewards(np.ones(10))\n",
    "f, ax = plt.subplots(1, figsize=(5,2))\n",
    "sns.barplot(list(range(10)), discounted_epr, color=\"steelblue\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We normalize the rewards so that they tank below zero towards the end. gamma controls how late the rewards tank."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x21b79f31668>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUAAAACSCAYAAAA0ERM6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEhdJREFUeJzt3WlMVOfbBvALOsOgc0DUakzUDoZKq+ISMUZrgHHh70aI\nCNOKOKCQGtziLgLWpQZB0ybaBiJWIw2aigpG0zRalZa0pFqKccNAoiCpVFvQWphRloHn/dAwr7hV\nz5yD4rl+n5h58Nw3ai5m5pznPm5CCAEiIg1yf9UNEBG9KgxAItIsBiARaRYDkIg0iwFIRJrFACQi\nzdLJ+UMOhwMpKSmoqalBS0sLEhMTMWnSJOd6YWEhsrKyoNPpEBkZCYvFoljDRERKkRWAJ06cQM+e\nPbFjxw78888/mDVrljMAHQ4HMjIyUFBQAIPBgOjoaEyePBm9evVStHEiIlfJegs8ffp0LF++HADQ\n1tYGne7/c/TGjRswmUyQJAl6vR6BgYEoKSlRplsiIgXJegXYrVs3AIDNZsPy5cuxcuVK55rNZoOX\nl5fzsdFoRENDw3OP19jYiKtXr6JPnz5466235LRERPSE1tZW1NbWIiAgAJ6enk+sywpAALh9+zaW\nLl2KefPmYcaMGc7nJUmCzWZzPrbb7fD29n7usa5evYqYmBi5rRARPdfBgwcxZsyYJ56XFYB1dXVI\nSEjAxo0bMW7cuA5rfn5+qK6uRn19PTw9PVFSUoKEhITnHq9Pnz7OJvv16yenJSKiJ9y5cwcxMTHO\njHmcrADMzs5GfX09srKykJmZCTc3N3z44Yd4+PAhLBYLkpOTER8fDyEELBYL+vbt+9zjtb/t7dev\nHwYMGCCnJSKiZ3rWR2uyAjA1NRWpqanPXDebzTCbzXIOTUTUaXghNBFpFgOQiDSLAUhEmsUAJCLN\nYgASkWYxAIlIs1wKwEuXLsFqtT7xfE5ODsLCwhAbG4vY2FjcvHnTlTJERKqQvRVu7969OH78OIxG\n4xNrZWVl2LFjB4YOHfrSxxVCoK2tTW5bz+Xu7g43NzdVjk1EXY/sADSZTMjMzMS6deueWCsrK0N2\ndjZqa2thNpuxcOHCFz5uW1sbPs0tRF39A7mtPdXb3t2x0TqJwxaIyEl2AIaGhqKmpuapazNnzkRM\nTAwkScKSJUtQVFSEkJCQFz52Xf0D/HVf2QAkInqcKidB4uLi4OPjA51Oh5CQEFy7dk2NMkRELnE5\nAIUQHR7bbDaEhYXh4cOHEELg3LlzGDZsmKtliIgUJ/stcLv2kwrffvutcxrMqlWrYLVaYTAYMH78\neAQHB7vcKBGR0lwKwP79++PQoUMAgLCwMOfz4eHhCA8Pd60zIiKV8UJoItIsBiARaRYDkIg0y+WT\nIF0dd54QaZfmA5A7T4i0S/MBCHDnCZFWqTINprCwEFFRUZgzZw6OHDniSgkiItUoPg3G4XAgIyMD\nBQUFMBgMiI6OxuTJk9GrVy+XmyUiUpLsV4Dt02Aed+PGDZhMJkiSBL1ej8DAQJSUlLjUJBGRGmQH\nYGho6FM/4LfZbPDy8nI+NhqNaGhokFuGiEg1il8HKEkSbDab87Hdboe3t7fSZYiIXKb4NBg/Pz9U\nV1ejvr4ezc3NKCkpwahRo1wtQ0SkOFWmwSQnJyM+Ph5CCFgsFvTt29flRomIlKbKNBiz2Qyz2exS\nY28q7jwhen3wQuhOxp0nRK8PBuArwJ0nRK8HToMhIs1iABKRZjEAiUizGIBEpFmyToIIIbB582ZU\nVFTAw8MDaWlpGDhwoHM9JycHR48edQ5A+PTTT+Hr66tIw0RESpEVgGfOnEFzczMOHTqES5cuIT09\nHVlZWc71srIy7NixA0OHDlWsUSIipckKwNLSUgQFBQEARo4ciatXr3ZYLysrQ3Z2Nmpra2E2m7Fw\n4ULXOyVZeOE10bPJCsDHJ77odDq0tbXB3f3fjxRnzpyJmJgYSJKEJUuWoKioCCEhIcp0TC+FF14T\nPZusAJQkCXa73fn40fADgLi4OEiSBAAICQnBtWvXGICvEC+8Jno6WWeBR48ejaKiIgDAxYsX4e/v\n71yz2WwICwvDw4cPIYTAuXPnMGzYMGW6JSJSkKxXgKGhoSguLsacOXMAAOnp6R2mwaxatQpWqxUG\ngwHjx49HcHCwok0TESlBVgC6ublhy5YtHZ4bNGiQ8+vw8HCEh4e71hkRkcp4ITQRaRYDkIg0iwFI\nRJrFACQizeJAVFKMmrtOAO48IeUxAEkxau06AbjzhNShyjSYwsJCZGVlQafTITIyEhaLRbGG6fXG\nXSfUlcj6DPDRaTCrV69Genq6c83hcCAjIwM5OTnIzc1FXl4e7t27p1jDRERKkRWAz5sGc+PGDZhM\nJkiSBL1ej8DAQJSUlCjTLRGRghSfBvP4mtFoRENDw0sd/23v7nLakn1M1lOunhq1nnXczj7p8ibV\ne9oJpTe93tMoPg1GkiTYbDbnmt1uh7e39wsf293dHRutk+S09ULHZj316qlZ62n1Ovuky5tS71kn\nlN70ek8jKwBHjx6NH374AdOmTXtiGoyfnx+qq6tRX18PT09PlJSUICEh4YWP7ebm1qln+liva9Zq\n19knXViva9d7nCrTYJKTkxEfHw8hBCwWC/r27ato00RESlBlGozZbIbZbHapMSIitfFCaOrSOvOk\nC715GIDUZXX2SRd68zAAqct6FSdd6M3CX3FEpFkMQCLSLAYgEWmWrM8Am5qasHbtWty9exeSJCEj\nIwM9e/bs8D1paWm4cOECjEYjACArK8t5r2AioteBrAD85ptv4O/vj6VLl+K7775DVlYWUlNTO3xP\nWVkZ9u3bBx8fH0UaJSJSmuxpMO33+g0ODsYvv/zSYV0IgerqamzcuBHR0dHIz893vVMiIoX95yvA\no0eP4uuvv+7w3Ntvv+18O2s0GjsMPwCABw8ewGq1YsGCBXA4HIiNjcXw4cM77BkmInrV/jMAo6Ki\nEBUV1eG5ZcuWOafB2O32DuOvAKBbt26wWq0wGAwwGAwYN24cysvLGYDU5XHnyZtF9jSYoqIiDB8+\nHEVFRRgzZkyH9aqqKqxcuRLHjx+Hw+FAaWkpZs+erUjDRK8Kd568eWQFYHR0NJKSkjB37lx4eHjg\n888/BwDk5OTAZDJh4sSJmDVrFiwWC/R6PSIiIuDn56do40SdjTtP3jyyAtDT0xO7du164vn58+c7\nv46Pj0d8fLzsxoiI1MbX3ESkWQxAItIsBiARaRYDkIg0iwFIRJrFgahEr7HOvoe01rgUgKdPn8bJ\nkyed1wE+6vDhw8jLy4Ner0diYiJvkkT0kjr7HtJaJDsA09LSUFxcjCFDhjyxVldXh9zcXBw7dgyN\njY2Ijo7GhAkToNfrXWqWSEt44bX6ZP8aGD16NDZv3vzUtcuXLyMwMBA6nQ6SJMHX1xcVFRVySxER\nqULWNJj09HRMnz4dv/7661P/jM1m6zAgoXv37mhoaHCxVSIiZcmaBvNfJEnqMCLLbrfD29v75bsj\nIlKRKp+EjhgxAqWlpWhubkZDQwMqKysxePBgNUoREcmm6GUwj06DsVqtmDt3LoQQWLVqFTw8PJQs\nRUTkMpcCcOzYsRg7dqzz8aPTYCwWCywWiyuHJyJSFS8GIiLN4k4QInLS2s4TBiARAdDmzhMGIBEB\n0ObOk9czlomIOgEDkIg0S7VpMGlpabhw4QKMRiMAICsry3kzdSKi14Eq02AAoKysDPv27YOPj4/s\n5oiI1KTKNBghBKqrq7Fx40ZER0cjPz9fbhkiItWoMg3mwYMHsFqtWLBgARwOB2JjYzF8+HD4+/sr\n0zURkQJUmQbTrVs3WK1WGAwGGAwGjBs3DuXl5QxAInqtqHIdYFVVFVauXInjx4/D4XCgtLQUs2fP\nVqMUEXVhr3rniWrTYGbNmgWLxQK9Xo+IiAj4+fkpWYqIurjXYeeJmxBCqNLBS7h16xYmT56Ms2fP\nYsCAAa+6HSJ6Q/xXtvBCaCLSLAYgEWkWA5CINIsBSESaxQAkIs1iABKRZsm6DtBms2HNmjWw2+1o\naWnB+vXrMWrUqA7fc/jwYeTl5UGv1yMxMRFms1mJfomIFCMrAPfv348PPvgAsbGxqKqqwurVq1FQ\nUOBcr6urQ25uLo4dO4bGxkZER0djwoQJ0Ov1ijVOROQqWQG4YMEC531+HQ4HDAZDh/XLly8jMDAQ\nOp0OkiTB19cXFRUVCAgIcL1jIiKFyJ4GExAQgNraWqxbtw6pqakd1m02G7y8vJyPu3fvjoaGhmfW\naG1tBQDcuXPnpZonInqe9kxpz5jHyZ4GU1FRgTVr1iApKQljxozpsCZJEmw2m/Ox3W6Ht7f3M2vU\n1tYCAGJiYv6rHSKil1ZbWwuTyfTE87LeAl+/fh0rVqzAzp078d577z2xPmLECOzcuRPNzc1oampC\nZWUlBg8e/MzjBQQE4ODBg+jTp4/m7kpFROppbW1FbW3tMz9+kzUMYfHixaioqED//v0hhIC3tzcy\nMzM7TIM5cuQI8vLyIITAokWLMGXKFJd/GCIiJb0W02CIiF4FXghNRJrFACQizWIAEpFmqXJPkM4g\nhMDmzZtRUVEBDw8PpKWlYeDAgarWvHTpEj777DPk5uaqWsfhcCAlJQU1NTVoaWlBYmIiJk1SZ3Q4\nALS1tWHDhg2oqqqCu7s7tmzZgnfffVe1eu3u3r2LyMhI7N+/H4MGDVK11uzZsyFJEgBgwIAB2LZt\nm6r19uzZg8LCQrS0tGDu3LmIjIxUrdaxY8dQUFAANzc3NDU1oby8HMXFxc6fV2kOhwNJSUmoqamB\nTqfD1q1bVf33a25uRnJyMm7dugVJkrBp0ya88847yhxcdFHff/+9WL9+vRBCiIsXL4pFixapWu+r\nr74SYWFh4qOPPlK1jhBC5Ofni23btgkhhLh//74wm82q1jt9+rRISUkRQghx/vx51f8uhRCipaVF\nLFmyREydOlVUVlaqWqupqUlERESoWuNR58+fF4mJiUIIIex2u/jyyy87rfaWLVvE4cOHVa1x5swZ\nsWLFCiGEEMXFxWLZsmWq1jtw4ID45JNPhBBCVFZWivj4eMWO3WXfApeWliIoKAgAMHLkSFy9elXV\neiaTCZmZmarWaDd9+nQsX74cwL+vznQ6dV+oT5kyBVu3bgUA1NTUoEePHqrWA4Dt27cjOjoaffv2\nVb1WeXk5Hjx4gISEBMyfPx+XLl1Std7PP/8Mf39/LF68GIsWLcLEiRNVrdfuypUruH79OiwWi6p1\nfH190draCiEEGhoaVN/jf/36dQQHBwMABg0ahMrKSsWO3WXfAj++3U6n06Gtre2F7wb1skJDQ1FT\nU6PKsR/XrVs3AP/+jMuXL8fKlStVr+nu7o7169fjzJkz+OKLL1StVVBQgN69e2PChAnYvXu3qrUA\nwNPTEwkJCbBYLLh58yY+/vhjnDp1SrX/K3///Tf++OMPZGdn4/fff8eiRYtw8uRJVWo9as+ePVi6\ndKnqdYxGI27duoVp06bh/v37yM7OVrXekCFD8OOPP2LKlCm4ePEi/vrrLwgh4Obm5vKxu+wrQEmS\nYLfbnY/VDL9X4fbt24iLi0NERARmzJjRKTUzMjJw6tQpbNiwAY2NjarVKSgoQHFxMaxWK8rLy5GU\nlIS7d++qVs/X1xfh4eHOr318fJzbL9Xg4+ODoKAg6HQ6DBo0CAaDAffu3VOtHgA0NDTg5s2bGDt2\nrKp1gH9vfxsUFIRTp07hxIkTSEpKQnNzs2r1IiMjYTQaERMTg7Nnz2LYsGGKhB/QhQNw9OjRKCoq\nAgBcvHgR/v7+nVJXdMJ143V1dUhISMDatWsRERGher3jx49jz549AACDwQB3d3dVf5kcOHAAubm5\nyM3Nxfvvv4/t27ejd+/eqtXLz89HRkYGAODPP/+E3W5Hnz59VKsXGBiIn376yVmvsbERPXv2VK0e\nAJSUlGDcuHGq1mjXo0cP5wkWLy8vOBwOtLW1qVbvypUrGD9+PA4ePIipU6cqerKzy74FDg0NRXFx\nMebMmQPg3wk1nUGp3zzPk52djfr6emRlZSEzMxNubm7Yu3evcwSZ0v73v/8hOTkZ8+bNg8PhQGpq\nqmq1HtcZf59RUVFITk7G3Llz4e7ujm3btqka8GazGb/99huioqIghMCmTZtU/zmrqqpUvwqiXVxc\nHFJSUhATEwOHw4HVq1fD09NTtXomkwm7du3C7t274e3tjbS0NMWOza1wRKRZXfYtMBGRqxiARKRZ\nDEAi0iwGIBFpFgOQiDSLAUhEmsUAJCLNYgASkWb9Hxw/+fqDvP9NAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x21b79d58be0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "discounted_epr_cent = discounted_epr - np.mean(discounted_epr)\n",
    "discounted_epr_norm = discounted_epr_cent/np.std(discounted_epr_cent)\n",
    "f, ax = plt.subplots(1, figsize=(5,2))\n",
    "sns.barplot(list(range(10)), discounted_epr_norm, color=\"steelblue\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x21b7f82d048>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUAAAADKCAYAAADUzJmgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGutJREFUeJzt3XtQVdf5PvDnHA83OSJqsLZqgDF1otIkA46jteiJSrxR\nGgI0XEQMTFKIOqhoEDAqcUB0TKaawlfQVhJk6hVHJ9NIVCqT8KuW0IEKDk4VdRQ1QRLLRbkczvr9\n4XAKohj32QuB/Xz+kr3JehdqHvdlrffohBACREQapH/eEyAiel4YgESkWQxAItIsBiARaRYDkIg0\niwFIRJplUPIfmc1mJCcno7a2Fu3t7YiNjcWcOXOs54uKipCVlQWDwYCgoCCEhISoNmEiIrUoCsAT\nJ05gxIgR2L59O/773//izTfftAag2WxGRkYGCgoK4ODggLCwMMydOxcjR45UdeJERLZSdAu8cOFC\nxMfHAwAsFgsMhv/l6JUrV+Du7g6j0Qg7Ozv4+PigtLRUndkSEalI0RWgk5MTAKCpqQnx8fFYvXq1\n9VxTUxOGDRtm/drZ2RmNjY29jtfS0oLKykq4ublhyJAhSqZERNRDR0cH6urq4OXlBUdHxx7nFQUg\nANy+fRsrVqzAkiVLsGjRIutxo9GIpqYm69fNzc1wcXHpdazKykpEREQonQoRUa/y8/MxderUHscV\nBeDdu3cRExODjRs3Yvr06d3OTZgwAdevX0dDQwMcHR1RWlqKmJiYXsdzc3OzTnL06NHY/PkZ1Dc8\nUDK1Jxrl4oTNS+dCr+9+12+xWLCz4P/hxyZ1640wOiH+rV9rqp4QAjK3lut0Ouh0Omnj0+Bz584d\nREREWDPmUYoCMDs7Gw0NDcjKykJmZiZ0Oh1+//vf48GDBwgJCUFSUhKio6MhhEBISAhGjx7d63id\nt71jxozBz3/+czi5jIS95b6SqT2Rk8tQjB07tsctdkdHB1r0Tmi0qPs/rpPeSXP1Ojo68FFeEe42\nqPtnBwAvuAzFxsg53eoJIWCxWFSv1Umv1zNwB4knPVpTFIApKSlISUl54nmTyQSTyaRkaBrg7jbc\nx/f31A/Ax7FYLH0auDT4KH4GSNQf9GXg8opz8GEAEv1EvOIcfBiARM+gL684ST7uBSYizWIAEpFm\n8RaYqJ+S+dKFL1weYgAS9VOyXrrwhcv/MACJ+jG+dJGLzwCJSLMYgESkWQxAItIsmwKwoqICkZGR\nPY7n5ubC398fS5cuxdKlS3Ht2jVbyhARSaH4JcjevXtx/PhxODs79zhXVVWF7du3Y/LkyTZNjohI\nJsVXgO7u7sjMzHzsuaqqKmRnZyM8PBw5OTmKJ0dEJJPiK0A/Pz/U1tY+9tzixYsREREBo9GI5cuX\no7i4GLNnz1Y8SSKST4sLr6WsA4yKioLRaAQAzJ49GxcvXmQAEvVzWlx4bXMAPtoCvampCf7+/vjy\nyy/h6OiIc+fOITg42NYyRNQHtLbw2uYA7Lys/eKLL6wt8desWYPIyEg4ODhgxowZmDVrls0TJSJS\nm00BOHbsWBw4cAAA4O/vbz0eEBCAgIAA22ZGRCQZF0ITkWYxAIlIsxiARKRZDEAi0iwGIBFpFgOQ\niDSLAUhEmsUAJCLNYgASkWYxAIlIsxiARKRZUlriFxUVITg4GKGhoTh8+LAtJYiIpFG9Jb7ZbEZG\nRgYKCgrg4OCAsLAwzJ07FyNHjrR5skREalK9Jf6VK1fg7u4Oo9EIOzs7+Pj4oLS01KZJEhHJoHpL\n/KamJgwbNsz6tbOzMxobG5WWIaJBqj+04Fe9Jb7RaERTU5P16+bmZri4uKhdhogGuP7Qgl/1lvgT\nJkzA9evX0dDQAEdHR5SWliImJsbWMkQ0CD3vFvxSWuInJSUhOjoaQgiEhIRg9OjRNk+UiEhtUlri\nm0wmmEwmmyZGRCQbF0ITkWYxAIlIsxiARKRZDEAi0iwGIBFpFgOQiDSLAUhEmsUAJCLNYgASkWYx\nAIlIsxiARKRZivYCCyGwefNmXLp0Cfb29khLS8P48eOt53Nzc3HkyBFrF+iPPvoIHh4eqkyYiEgt\nigLw9OnTaGtrw4EDB1BRUYGtW7ciKyvLer6qqgrbt2/H5MmTVZsoEZHaFAVgWVkZfH19AQCvvvoq\nKisru52vqqpCdnY26urqYDKZ8N5779k+UyIilSl6Bvho23uDwdCttfXixYuRmpqKzz//HGVlZSgu\nLrZ9pkREKlMUgEajEc3NzdavLRYL9Pr/DRUVFQVXV1cYDAbMnj0bFy9etH2mREQqUxSA3t7e1qu6\n8vJyTJw40XquqakJ/v7+ePDgAYQQOHfuHKZMmaLObImIVKToGaCfnx9KSkoQGhoKANi6dWu3lvhr\n1qxBZGQkHBwcMGPGDMyaNUvVSRMRqUFRAOp0OqSmpnY75unpaf11QEAAAgICbJsZEZFkXAhNRJrF\nACQizWIAEpFmMQCJSLMYgESkWQxAItIsBiARaRYDkIg0iwFIRJrFACQizWIAEpFmKQpAIQQ2bdqE\n0NBQLF26FDdu3Oh2vqioCMHBwQgNDcXhw4dVmSgRkdoUBWDXlvgJCQnYunWr9ZzZbEZGRgZyc3OR\nl5eHgwcP4ocfflBtwkREalEUgL21xL9y5Qrc3d1hNBphZ2cHHx8flJaWqjNbIiIVKWqH9aSW+Hq9\nvsc5Z2dnNDY29jpeR0cHAODOnTuwWCxwtDzAMH2Lkqk9kaNFh9ra2m6dqwGwnor1ZNVivcH3d6Wv\n6t25cwfA/zLmUYoCsLeW+EajEU1NTdZzzc3NcHFx6XW8uro6AEBERISS6fxkX/yf1OFZj/UGTL3B\n/LM9rl5dXR3c3d17fJ+iAPT29sbf//53LFiwoEdL/AkTJuD69etoaGiAo6MjSktLERMT0+t4Xl5e\nyM/Ph5ubG4YMGaJkSkREPXR0dKCurg5eXl6PPa8TQohnHbTrB6MDD1viV1VVWVvinz17Fn/6058g\nhEBwcDDCwsJs+ymIiCRQFIBERIMBF0ITkWYxAIlIsxiARKRZDEAi0ixFy2D6g65vou3t7ZGWlobx\n48dLrVlRUYEdO3YgLy9Pah2z2Yzk5GTU1taivb0dsbGxmDNnjrR6FosFGzZswNWrV6HX65GamoqX\nXnpJWr1O9fX1CAoKwr59+7p9rrQMb731FoxGIwBg3LhxSE9Pl1ovJycHRUVFaG9vR3h4OIKCgqTV\nOnbsGAoKCqDT6dDa2orq6mqUlJRYf161mc1mJCYmora2FgaDAVu2bJH659fW1oakpCTcvHkTRqMR\nmzZtwosvvqjO4GKA+uqrr8T69euFEEKUl5eLuLg4qfX27Nkj/P39xdtvvy21jhBCHD16VKSnpwsh\nhLh3754wmUxS6506dUokJycLIYQ4f/689N9LIYRob28Xy5cvF/Pnzxc1NTVSa7W2torAwECpNbo6\nf/68iI2NFUII0dzcLD799NM+q52amioOHToktcbp06fFqlWrhBBClJSUiJUrV0qtt3//fvHhhx8K\nIYSoqakR0dHRqo09YG+Be9uPLIO7uzsyMzOl1ui0cOFCxMfHA3h4dWYwyL1QnzdvHrZs2QIAqK2t\nxfDhw6XWA4Bt27YhLCwMo0ePll6ruroa9+/fR0xMDJYtW4aKigqp9b755htMnDgR77//PuLi4vD6\n669LrdfpwoULuHz5MkJCQqTW8fDwQEdHB4QQaGxshJ2dndR6ly9fxqxZswAAnp6eqKmpUW3sAXsL\n3Nt+ZBn8/PxQW1srZexHOTk5AXj4M8bHx2P16tXSa+r1eqxfvx6nT5/Grl27pNYqKCjAqFGjMHPm\nTOzevVtqLQBwdHRETEwMQkJCcO3aNbz77rsoLCyU9nflxx9/xK1bt5CdnY0bN24gLi4OJ0+elFKr\nq5ycHKxYsUJ6HWdnZ9y8eRMLFizAvXv3kJ2dLbXepEmTcPbsWcybNw/l5eX4/vvvIYSATqezeWyb\n/gZUVFQgMjKyx/G+6AfY237kweD27duIiopCYGAgFi1a1Cc1MzIyUFhYiA0bNqClRf0N/50KCgpQ\nUlKCyMhIVFdXIzExEfX19dLqeXh4ICAgwPprV1dX6/5zGVxdXeHr6wuDwQBPT084ODhIbwnX2NiI\na9euYdq0aVLrAEBubi58fX1RWFiIEydOIDExEW1tbdLqBQUFwdnZGREREThz5gymTJmiSvgBNgTg\n3r17sWHDBrS3t3c73lf9AL29vVFcXAwAPfYjyyT6YOPM3bt3ERMTg3Xr1iEwMFB6vePHjyMnJwcA\n4ODgAL1eL/Ufk/379yMvLw95eXl4+eWXsW3bNowaNUpavaNHjyIjIwMA8N1336G5uRlubm7S6vn4\n+ODrr7+21mtpacGIESOk1QOA0tJSTJ8+XWqNTsOHD7e+YBk2bBjMZjMsFou0ehcuXMCMGTOQn5+P\n+fPnq/qyU/EtcOczsQ8++KDb8a79AAFY+wHOnz/ftpk+ws/PDyUlJQgNDQWAbk1ZZVLrX57eZGdn\no6GhAVlZWcjMzIROp8PevXthb28vpd4bb7yBpKQkLFmyBGazGSkpKdJqPaovfj+Dg4ORlJSE8PBw\n6PV6pKenSw14k8mEb7/9FsHBwdbu6bJ/zqtXr0pfBdEpKioKycnJiIiIgNlsRkJCAhwdHaXVc3d3\nx86dO7F79264uLggLS1NtbFt2gtcW1uLhIQEHDhwwHqsrKwM+fn5+OSTTwAAu3btwi9+8QsEBwc/\ncZyWlhZUVlayGwwRqaprN5jHhbTqL0GU9AOsrKyU3guQiLQrPz8fU6dO7XHc5gB89AJSST/Azucx\n+fn5GDNmjK1TIiIC8LDLfERExBOf+docgJ3PNr744gtrP8CkpCRER0dDCIGQkJCnrvXqvO0dM2YM\nxo0bZ+uUiIi6edKjNZsCcOzYsdbnf/7+/tbjJpMJJpPJlqGJiKQbPAvniIiekaIrQPGURgS5ubk4\ncuQIRo4cCQD46KOP4OHh8ZPHlrWmSK/X98myCyIaGBQFYNcPRq+oqMDWrVuRlZVlPV9VVYXt27dj\n8uTJzzy2xWLBR3lFuNtwX8nUnugFl6HYGDmnx7OAvg7cwV6PaCBRFIBPa0RQVVWF7Oxs1NXVwWQy\n4b333num8e823Mf399QNwCfp68AdzPVkhi3AwCX1qf7B6ACwePFiREREwGg0Yvny5SguLsbs2bPV\nmbEEfRm4g7merLAFnhzwRLZQ/YPRgYdbZTq3ws2ePRsXL17s1wFI6unLcOcVJ9lK9Q9Gb2pqgr+/\nP7788ks4Ojri3LlzvW6DI1KKV5xkK0UB+LhGBF0XQq9ZswaRkZFwcHDAjBkzrM0MidTW148TaHBR\nFIA6nQ6pqandjnX9TICAgABr/zUiov5qwHaEJuprfOY4+DAAiX4iPnMcfKTsBCkqKkJWVhYMBgOC\ngoKkf0gLUV/hM8fBRdFe4K47QRISErp1Y+6rlvhERLZSFIC97QTp2hLfzs7O2hKfiKi/UX0nyKPn\nnJ2d0djY+Ezjv+AyVMm0FI/JeurVk1FLq/UG+77x/rBPXfWdIEpa4nel1+uxMXKOkmn9pLFZT149\nmbW0WG8w7xt/HvUeR/WdIEpa4nel0+n69E0Y6w3MWlqoBwzefePPq96jpOwEedaW+ET0eH39uERr\npOwEYUt8Itv19eMSLeJCaKJ+6nnccmuNogBsbW3FunXrUF9fD6PRiIyMDIwYMaLb96SlpeFf//oX\nnJ2dAQBZWVnWFllERP2BogD861//iokTJ2LFihX429/+hqysLKSkpHT7nqqqKvz5z3+Gq6urKhMl\nIlKb4oXQnS2uZs2ahX/84x/dzgshcP36dWzcuBFhYWE4evSo7TMlIlLZU68Ajxw5gs8++6zbsRde\neMF6O+vs7Nxt3R8A3L9/H5GRkXjnnXdgNpuxdOlS/OpXv+q2XIaI6Hl7agAGBwf36Oi8cuVK60Lo\n5ubmbjs/AMDJycnaENXBwQHTp09HdXU1A5CI+hVFt8De3t4oLi4GABQXF2Pq1Kndzl+9ehVhYWEQ\nQqC9vR1lZWWYMmWK7bMlIlKRopcgYWFhSExMRHh4OOzt7fHxxx8DePiB6O7u7nj99dfx5ptvIiQk\nBHZ2dggMDMSECRNUnTgRka0UBaCjoyN27tzZ4/iyZcusv46OjkZ0dLTiiRERycbl4ESkWQxAItIs\nmwLw1KlTSEhIeOy5Q4cOISgoCKGhoTh79qwtZYiIpFC8FzgtLQ0lJSWYNGlSj3N3795FXl4ejh07\nhpaWFoSFhWHmzJmws7OzabJERGpSHIDe3t7w8/PDwYMHe5z797//DR8fHxgMBhiNRnh4eODSpUvw\n8vKyabJEJJfW2m8p2gmydetWLFy4EP/85z8f+9882hZ/6NChz9wWn4j6lhbbbynaCfI0trbFJ6K+\np8X2W1Ji+ZVXXkFZWRna2trQ2NiImpoa/PKXv5RRiohIMVUbonbdCRIZGYnw8HAIIbBmzRrY29ur\nWYqIyGY2BeC0adMwbdo069ddd4KEhIQgJCTEluGJiKTqn08miYj6gE1XgKdOncLJkyetzRC6Ykt8\nIurvpCyEBtgSn4j6P8W3wN7e3ti8efNjz7ElPhENBFIWQrMlPhENBFIWQrMlPhENBFLeArMlPhEN\nBNIWQrMlPhH1d9IWQrMlPhH1d1wITUSaxQAkIs1SdAvc1NSEtWvXorm5Ge3t7Vi/fj1ee+21bt9z\n6NAhHDx4EHZ2doiNjYXJZFJjvkREqlEUgPv27cOvf/1rLF26FFevXkVCQgIKCgqs59kSn4gGAkUB\n+M4771jbW5nNZjg4OHQ7z5b4RDQQKN4J4uXlhbq6OnzwwQdISUnpdp4t8YloIFC8E+TSpUtYu3Yt\nEhMTMXXq1G7n2BKfiAYCRW+BL1++jFWrVmHHjh34zW9+0+M8W+IT0UCg6BngJ598gra2NqSlpUEI\nARcXF2RmZrIlPhENKIoCMCsr67HH2RKfiAYSLoQmIs2SthCaLfGJqL+TshAaYEt8Iur/pCyE7toS\nv66uDsHBwQgKCrJ9tkREKpKyEJot8YloIJCyEJot8YloIJCyEJot8YloIJC2EJot8YnoaV5wGfpc\nx9QJIYTqM3hGN2/exNy5c3HmzBmMGzfueU+HiPqAEAIWi0XK2Hq9Hjqd7qnZouqHIhER/VQ6nQ5D\nhgx5rnPgThAi0ixFV4APHjxAQkICGhoaYG9vj4yMDIwePbrb97AlPhH1d4quAA8dOgQvLy/s378f\nv/3tb7Fnz55u5ztb4h88eBB79+7Fxx9/jPb2dlUmTESkFkVXgFFRUeh8d3Lr1i0MHz6823m2xCei\ngcCmnSBRUVH4z3/+g7/85S/dzj9rS/yOjg4AwJ07d55p8kREvenMlM6MeZTinSAA8Nlnn6GmpgZ/\n+MMfcOrUKevxZ22JX1dXBwCIiIh42nSIiJ5ZXV0d3N3dexxXdAuck5ODn/3sZ/jd736HoUOH9niV\n/corr+CPf/wj2tra0Nra+tSW+F5eXsjPz4ebm9tzfy1ORINHR0cH6urqnvj4TdFC6Pr6eiQmJqK1\ntRVCCKxduxavvfZat50ghw8fxsGDByGEQFxcHObNm2fzD0NEpKZ+sROEiOh54EJoItIsBiARaRYD\nkIg0a8A2QxBCYPPmzbh06RLs7e2RlpaG8ePHS61ZUVGBHTt2IC8vT2ods9mM5ORk1NbWor29HbGx\nsZgzZ460ehaLBRs2bMDVq1eh1+uRmpqKl156SVq9TvX19QgKCsK+ffvg6ekptdZbb71l/VCucePG\nIT09XWq9nJwcFBUVob29HeHh4VI/EuLYsWMoKCiATqdDa2srqqurUVJSIu1DyMxmMxITE1FbWwuD\nwYAtW7ZI/fNra2tDUlISbt68CaPRiE2bNuHFF19UZ3AxQH311Vdi/fr1QgghysvLRVxcnNR6e/bs\nEf7+/uLtt9+WWkcIIY4ePSrS09OFEELcu3dPmEwmqfVOnTolkpOThRBCnD9/XvrvpRBCtLe3i+XL\nl4v58+eLmpoaqbVaW1tFYGCg1BpdnT9/XsTGxgohhGhubhaffvppn9VOTU0Vhw4dklrj9OnTYtWq\nVUIIIUpKSsTKlSul1tu/f7/48MMPhRBC1NTUiOjoaNXGHrC3wGVlZfD19QUAvPrqq6isrJRaz93d\nHZmZmVJrdFq4cCHi4+MBPLw6MxjkXqjPmzcPW7ZsAQDU1tb22Noow7Zt2xAWFtajiYYM1dXVuH//\nPmJiYrBs2TJUVFRIrffNN99g4sSJeP/99xEXF4fXX39dar1OFy5cwOXLlxESEiK1joeHBzo6OiCE\nQGNjI+zs7KTWu3z5MmbNmgUA8PT0RE1NjWpjD9hb4Ee32xkMBlgsFuj1cjLdz88PtbW1UsZ+lJOT\nE4CHP2N8fDxWr14tvaZer8f69etx+vRp7Nq1S2qtgoICjBo1CjNnzsTu3bul1gIAR0dHxMTEICQk\nBNeuXcO7776LwsJCaX9XfvzxR9y6dQvZ2dm4ceMG4uLicPLkSSm1usrJycGKFSuk13F2dsbNmzex\nYMEC3Lt3D9nZ2VLrTZo0CWfPnsW8efNQXl6O77//HkII6HQ6m8cesFeARqMRzc3N1q9lht/zcPv2\nbURFRSEwMBCLFi3qk5oZGRkoLCzEhg0b0NLSIq1OQUEBSkpKEBkZierqaiQmJqK+vl5aPQ8PDwQE\nBFh/7erqat1+KYOrqyt8fX1hMBjg6ekJBwcH/PDDD9LqAUBjYyOuXbuGadOmSa0DALm5ufD19UVh\nYSFOnDiBxMREtLW1SasXFBQEZ2dnRERE4MyZM5gyZYoq4QcM4AD09vZGcXExAKC8vLzPPnFO9MG6\n8bt37yImJgbr1q1DYGCg9HrHjx9HTk4OAMDBwQF6vV7qPyb79+9HXl4e8vLy8PLLL2Pbtm0YNWqU\ntHpHjx5FRkYGAOC7775Dc3Mz3NzcpNXz8fHB119/ba3X0tKCESNGSKsHAKWlpZg+fbrUGp2GDx9u\nfcEybNgwmM1maa3tgYe39jNmzEB+fj7mz5+v6svOAXsL7Ofnh5KSEoSGhgJ42KGmL6j1L09vsrOz\n0dDQgKysLGRmZkKn02Hv3r3WD6NX2xtvvIGkpCQsWbIEZrMZKSkp0mo9qi9+P4ODg5GUlITw8HDo\n9Xqkp6dLDXiTyYRvv/0WwcHBEEJg06ZN0n/Oq1evSl8F0SkqKgrJycmIiIiA2WxGQkICHB0dpdVz\nd3fHzp07sXv3bri4uCAtLU21sbkVjog0a8DeAhMR2YoBSESaxQAkIs1iABKRZjEAiUizGIBEpFkM\nQCLSLAYgEWnW/wfyBumzL4N5TQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x21b7f1a2320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "discounted_epr = discount_rewards(np.ones(10), gamma=0.5)\n",
    "discounted_epr_cent = discounted_epr - np.mean(discounted_epr)\n",
    "discounted_epr_norm = discounted_epr_cent/np.std(discounted_epr_cent)\n",
    "f, ax = plt.subplots(2, figsize=(5,3))\n",
    "sns.barplot(list(range(10)), discounted_epr, color=\"steelblue\", ax=ax[0])\n",
    "sns.barplot(list(range(10)), discounted_epr_norm, color=\"steelblue\", ax=ax[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model: Policy Gradient\n",
    "\n",
    "$$\n",
    "l_1 = relu( x W_1 + b_1) \\\\\n",
    "l_2 = l_1 W_2 + b_2 \\\\\n",
    "\\pi(a|s) = sigmoid(l_2)\n",
    "$$\n",
    "\n",
    "Note: in policy gradient approach, the output of the dense layer is mapped into to a 0-1 range via the sigmoid function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "TOTAL_EPISODES = 2000 if isFast else 10000\n",
    "\n",
    "D = 4  # input dimensionality\n",
    "H = 10 # number of hidden layer neurons\n",
    "\n",
    "observations = C.sequence.input_variable(STATE_COUNT, np.float32, name=\"obs\")\n",
    "\n",
    "W1 = C.parameter(shape=(STATE_COUNT, H), init=C.glorot_uniform(), name=\"W1\")\n",
    "b1 = C.parameter(shape=H, name=\"b1\")\n",
    "layer1 = C.relu(C.times(observations, W1) + b1)\n",
    "\n",
    "W2 = C.parameter(shape=(H, ACTION_COUNT), init=C.glorot_uniform(), name=\"W2\")\n",
    "b2 = C.parameter(shape=ACTION_COUNT, name=\"b2\")\n",
    "score = C.times(layer1, W2) + b2\n",
    "# Until here it was similar to DQN\n",
    "\n",
    "probability = C.sigmoid(score, name=\"prob\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the PG model\n",
    "\n",
    "**Policy Search**: The optimal policy search can be carried out with either gradient free approaches or by computing gradients over the policy space ($\\pi_\\theta$) which is parameterized by $\\theta$. In this tutorial, we use the classic forward (`loss.forward`) and back (`loss.backward`) propagation of errors over the parameterized space $\\theta$. In this case, $\\theta = \\{W_1, b_1, W_2, b_2\\}$, our model parameters.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 20. Average reward for episode 21.800000.\n",
      "Episode: 40. Average reward for episode 23.450000.\n",
      "Episode: 60. Average reward for episode 20.650000.\n",
      "Episode: 80. Average reward for episode 23.900000.\n",
      "Episode: 100. Average reward for episode 17.750000.\n",
      "Episode: 120. Average reward for episode 27.550000.\n",
      "Episode: 140. Average reward for episode 20.300000.\n",
      "Episode: 160. Average reward for episode 20.450000.\n",
      "Episode: 180. Average reward for episode 14.700000.\n",
      "Episode: 200. Average reward for episode 18.900000.\n",
      "Episode: 220. Average reward for episode 18.350000.\n",
      "Episode: 240. Average reward for episode 20.050000.\n",
      "Episode: 260. Average reward for episode 19.600000.\n",
      "Episode: 280. Average reward for episode 22.150000.\n",
      "Episode: 300. Average reward for episode 16.000000.\n",
      "Episode: 320. Average reward for episode 21.300000.\n",
      "Episode: 340. Average reward for episode 19.250000.\n",
      "Episode: 360. Average reward for episode 17.550000.\n",
      "Episode: 380. Average reward for episode 20.050000.\n",
      "Episode: 400. Average reward for episode 20.250000.\n",
      "Episode: 420. Average reward for episode 20.200000.\n",
      "Episode: 440. Average reward for episode 21.100000.\n",
      "Episode: 460. Average reward for episode 23.400000.\n",
      "Episode: 480. Average reward for episode 20.100000.\n",
      "Episode: 500. Average reward for episode 18.400000.\n",
      "Episode: 520. Average reward for episode 15.550000.\n",
      "Episode: 540. Average reward for episode 16.850000.\n",
      "Episode: 560. Average reward for episode 18.000000.\n",
      "Episode: 580. Average reward for episode 23.650000.\n",
      "Episode: 600. Average reward for episode 16.550000.\n",
      "Episode: 620. Average reward for episode 21.200000.\n",
      "Episode: 640. Average reward for episode 19.100000.\n",
      "Episode: 660. Average reward for episode 25.150000.\n",
      "Episode: 680. Average reward for episode 22.400000.\n",
      "Episode: 700. Average reward for episode 19.150000.\n",
      "Episode: 720. Average reward for episode 19.900000.\n",
      "Episode: 740. Average reward for episode 26.500000.\n",
      "Episode: 760. Average reward for episode 20.500000.\n",
      "Episode: 780. Average reward for episode 19.300000.\n",
      "Episode: 800. Average reward for episode 21.250000.\n",
      "Episode: 820. Average reward for episode 27.900000.\n",
      "Episode: 840. Average reward for episode 25.600000.\n",
      "Episode: 860. Average reward for episode 27.250000.\n",
      "Episode: 880. Average reward for episode 27.000000.\n",
      "Episode: 900. Average reward for episode 29.900000.\n",
      "Episode: 920. Average reward for episode 28.900000.\n",
      "Episode: 940. Average reward for episode 30.200000.\n",
      "Task solved in: 940 \n"
     ]
    }
   ],
   "source": [
    "input_y = C.sequence.input_variable(1, np.float32, name=\"input_y\")\n",
    "advantages = C.sequence.input_variable(1, np.float32, name=\"advt\")\n",
    "\n",
    "loss = -C.reduce_mean(C.log(C.square(input_y - probability) + 1e-4) * advantages, axis=0, name='loss')\n",
    "\n",
    "lr = 0.001\n",
    "lr_schedule = C.learning_rate_schedule(lr, C.UnitType.sample)\n",
    "sgd = C.sgd([W1, W2], lr_schedule)\n",
    "\n",
    "gradBuffer = dict((var.name, np.zeros(shape=var.shape)) for var in loss.parameters if var.name in ['W1', 'W2', 'b1', 'b2'])\n",
    "\n",
    "xs, hs, label, drs = [], [], [], []\n",
    "running_reward = None\n",
    "reward_sum = 0\n",
    "episode_number = 1\n",
    "\n",
    "observation = env.reset()\n",
    "\n",
    "while episode_number <= TOTAL_EPISODES:\n",
    "    x = np.reshape(observation, [1, STATE_COUNT]).astype(np.float32)\n",
    "\n",
    "    # Run the policy network and get an action to take.\n",
    "    prob = probability.eval(arguments={observations: x})[0][0][0]\n",
    "    action = 1 if np.random.uniform() < prob else 0\n",
    "\n",
    "    xs.append(x)  # observation\n",
    "    # grad that encourages the action that was taken to be taken\n",
    "\n",
    "    y = 1 if action == 0 else 0  # a \"fake label\"\n",
    "    label.append(y)\n",
    "\n",
    "    # step the environment and get new measurements\n",
    "    observation, reward, done, info = env.step(action)\n",
    "    reward_sum += float(reward)\n",
    "\n",
    "    # Record reward (has to be done after we call step() to get reward for previous action)\n",
    "    drs.append(float(reward))\n",
    "\n",
    "    if done:\n",
    "        # Stack together all inputs, hidden states, action gradients, and rewards for this episode\n",
    "        epx = np.vstack(xs)\n",
    "        epl = np.vstack(label).astype(np.float32)\n",
    "        epr = np.vstack(drs).astype(np.float32)\n",
    "        xs, label, drs = [], [], []  # reset array memory\n",
    "\n",
    "        # Compute the discounted reward backwards through time.\n",
    "        discounted_epr = discount_rewards(epr)\n",
    "        # Size the rewards to be unit normal (helps control the gradient estimator variance)\n",
    "        discounted_epr -= np.mean(discounted_epr)\n",
    "        discounted_epr /= np.std(discounted_epr)\n",
    "\n",
    "        # Forward pass\n",
    "        arguments = {observations: epx, input_y: epl, advantages: discounted_epr}\n",
    "        state, outputs_map = loss.forward(arguments, outputs=loss.outputs,\n",
    "                                          keep_for_backward=loss.outputs)\n",
    "\n",
    "        # Backward psas\n",
    "        root_gradients = {v: np.ones_like(o) for v, o in outputs_map.items()}\n",
    "        vargrads_map = loss.backward(state, root_gradients, variables=set([W1, W2]))\n",
    "\n",
    "        for var, grad in vargrads_map.items():\n",
    "            gradBuffer[var.name] += grad\n",
    "\n",
    "        # Wait for some batches to finish to reduce noise\n",
    "        if episode_number % BATCH_SIZE_BASELINE == 0:\n",
    "            grads = {W1: gradBuffer['W1'].astype(np.float32),\n",
    "                     W2: gradBuffer['W2'].astype(np.float32)}\n",
    "            updated = sgd.update(grads, BATCH_SIZE_BASELINE)\n",
    "\n",
    "            # reset the gradBuffer\n",
    "            gradBuffer = dict((var.name, np.zeros(shape=var.shape))\n",
    "                              for var in loss.parameters if var.name in ['W1', 'W2', 'b1', 'b2'])\n",
    "\n",
    "            print('Episode: %d. Average reward for episode %f.' % (episode_number, reward_sum / BATCH_SIZE_BASELINE))\n",
    "\n",
    "            if reward_sum / BATCH_SIZE_BASELINE > REWARD_TARGET:\n",
    "                print('Task solved in: %d ' % episode_number)\n",
    "                break\n",
    "\n",
    "            reward_sum = 0\n",
    "\n",
    "        observation = env.reset()  # reset env\n",
    "        episode_number += 1\n",
    "probability.save('pg.mod')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solutions**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((4, 10), (10,), (10, 2), (2,), (2,))"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "observation = C.sequence.input_variable(STATE_COUNT, np.float32, name=\"s\")\n",
    "\n",
    "W1 = C.parameter(shape=(STATE_COUNT, H), init=C.glorot_uniform(), name=\"W1\")\n",
    "b1 = C.parameter(shape=H, name=\"b1\")\n",
    "layer1 = C.relu(C.times(observation, W1) + b1)\n",
    "\n",
    "W2 = C.parameter(shape=(H, ACTION_COUNT), init=C.glorot_uniform(), name=\"W2\")\n",
    "b2 = C.parameter(shape=ACTION_COUNT, name=\"b2\")\n",
    "model = C.times(layer1, W2) + b2\n",
    "W1.shape, b1.shape, W2.shape, b2.shape, model.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'observation' is not defined",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-8-4b5375d1fbe3>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mz\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minput\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0mmodel\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcreate_model\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mobservation\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0mmodel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlayer1\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mW\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlayer1\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mb\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlayer2\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mW\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlayer2\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mb\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'observation' is not defined"
     ],
     "output_type": "error"
    }
   ],
   "source": [
    "# Correspoding layers implementation - Preferred solution\n",
    "def create_model(input):\n",
    "    with C.layers.default_options(init=C.glorot_uniform()):\n",
    "        z = C.layers.Sequential([C.layers.Dense(H, name=\"layer1\"),\n",
    "                                 C.layers.Dense(ACTION_COUNT, name=\"layer2\")])\n",
    "        return z(input)\n",
    "    \n",
    "model = create_model(observation)\n",
    "\n",
    "model.layer1.W.shape, model.layer1.b.shape, model.layer2.W.shape, model.layer2.b.shape, model.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 20, Average reward for episode 18.000000.\n",
      "Episode: 40, Average reward for episode 20.800000.\n",
      "Episode: 60, Average reward for episode 20.600000.\n",
      "Episode: 80, Average reward for episode 28.900000.\n",
      "Episode: 100, Average reward for episode 24.650000.\n",
      "Episode: 120, Average reward for episode 18.400000.\n",
      "Episode: 140, Average reward for episode 23.750000.\n",
      "Episode: 160, Average reward for episode 23.000000.\n",
      "Episode: 180, Average reward for episode 16.850000.\n",
      "Episode: 200, Average reward for episode 15.250000.\n",
      "Episode: 220, Average reward for episode 14.100000.\n",
      "Episode: 240, Average reward for episode 14.500000.\n",
      "Episode: 260, Average reward for episode 13.750000.\n",
      "Episode: 280, Average reward for episode 13.200000.\n",
      "Episode: 300, Average reward for episode 12.150000.\n",
      "Episode: 320, Average reward for episode 12.650000.\n",
      "Episode: 340, Average reward for episode 12.900000.\n",
      "Episode: 360, Average reward for episode 12.850000.\n",
      "Episode: 380, Average reward for episode 13.300000.\n",
      "Episode: 400, Average reward for episode 12.850000.\n",
      "Episode: 420, Average reward for episode 12.200000.\n",
      "Episode: 440, Average reward for episode 11.800000.\n",
      "Episode: 460, Average reward for episode 11.800000.\n",
      "Episode: 480, Average reward for episode 11.750000.\n",
      "Episode: 500, Average reward for episode 12.300000.\n",
      "Episode: 520, Average reward for episode 11.700000.\n",
      "Episode: 540, Average reward for episode 12.750000.\n",
      "Episode: 560, Average reward for episode 12.150000.\n",
      "Episode: 580, Average reward for episode 11.600000.\n",
      "Episode: 600, Average reward for episode 12.750000.\n",
      "Episode: 620, Average reward for episode 11.900000.\n",
      "Episode: 640, Average reward for episode 10.900000.\n",
      "Episode: 660, Average reward for episode 10.750000.\n",
      "Episode: 680, Average reward for episode 12.850000.\n",
      "Episode: 700, Average reward for episode 10.500000.\n",
      "Episode: 720, Average reward for episode 11.700000.\n",
      "Episode: 740, Average reward for episode 10.950000.\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-6f1a6bb65fe0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m    198\u001b[0m \u001b[0mreward_sum\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    199\u001b[0m \u001b[1;32mwhile\u001b[0m \u001b[0mepisode_number\u001b[0m \u001b[1;33m<\u001b[0m \u001b[0mTOTAL_EPISODES\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m--> 200\u001b[0;31m     \u001b[0mreward_sum\u001b[0m \u001b[1;33m+=\u001b[0m \u001b[0mrun\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0magent\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    201\u001b[0m     \u001b[0mepisode_number\u001b[0m \u001b[1;33m+=\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    202\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mepisode_number\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0mBATCH_SIZE_BASELINE\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-4-6f1a6bb65fe0>\u001b[0m in \u001b[0;36mrun\u001b[0;34m(agent)\u001b[0m\n\u001b[1;32m    185\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    186\u001b[0m         \u001b[0magent\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mobserve\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ms\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0ma\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mr\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0ms_\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m--> 187\u001b[0;31m         \u001b[0magent\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreplay\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    188\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    189\u001b[0m         \u001b[0ms\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0ms_\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-4-6f1a6bb65fe0>\u001b[0m in \u001b[0;36mreplay\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    122\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    123\u001b[0m         \u001b[1;31m# CNTK: explicitly setting to float32\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m--> 124\u001b[0;31m         \u001b[0mstates\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0marray\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m[\u001b[0m \u001b[0mo\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mo\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mbatch\u001b[0m \u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfloat32\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    125\u001b[0m         \u001b[0mstates_\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0marray\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mno_state\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mo\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m \u001b[1;32melse\u001b[0m \u001b[0mo\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mo\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mbatch\u001b[0m \u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfloat32\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    126\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ],
     "output_type": "error"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEFCAYAAADuT+DpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH3tJREFUeJzt3Xt8VPWd//HXmUsmE0K438a1IIJCRbAFFFQuKtIWa6u4\nfr3UbVerllbatXV1rVv9uT5c+9sulu62xVatSi+2/bqKq6iIl2otgnhXEESsIhUQuUgSkkwyl/3j\nTOIYQkJCTs7MnPfz8chjMuc2n28Iec/3nDnfr5PNZhERkeAK+V2AiIj4S0EgIhJwCgIRkYBTEIiI\nBJyCQEQk4CJ+F9BJ+oiTiEjXOPtbUWxBwJYtW7q0XyKR6PK+xUptDga1ORgOps2JRKLd9To1JCIS\ncAoCEZGAUxCIiAScgkBEJOAUBCIiAacgEBEJOM8/PmqMOQ74D2vtzFbLTweuA1LAHdba27yuRURE\n9uVpj8AYcxVwO1DeankUWAjMBmYAlxpjhnhVR3jTJrjsMpzaWq9eQkSkaHndI3gbmAv8ptXyscBG\na+1uAGPMX4DpwD0dHbCjGyPatHQpLFrEsBNOgPPP7/z+RaxLP68ipzYHg9rcfTwNAmvtvcaYEW2s\nqgL25D2vAfocyDG7cmddWZ8+DARqVq+mZubMTu9frHT3ZTCozcFQincWVwO98573Bj7y6sVSo0cD\nEHnrLa9eQkSkaPk11tA6YLQxpj9Qi3taaIFXL5YZMgSqqhQEIiJt6NEegTHmfGPMpdbaJuB7wKPA\nStxPDb3v2Qs7Dnz600TeeQeamjx7GRGRYuR5j8Ba+y4wJff93XnLHwQe9Pr1W4wdi7NqFZFNm0iN\nGtVjLysiUuiCc0PZ2LGArhOIiLSmIBARCTgFgYhIwAUnCEaMIBuLEdm40e9KREQKSnCCIBwmdfjh\nbo8gk/G7GhGRghGcIACaRo8mVF9POGB3JIqItCdQQaA7jEVE9hWsIMjdP6AgEBH5WLCCoLlHoAvG\nIiItghUEhx1GNhxWj0BEJE+ggoBYjPTw4UQ3bIBs1u9qREQKQrCCAGg68khCH31EaPt2v0sRESkI\ngQuC1JgxAETXr/e5EhGRwhC4IGjKBUFk3TqfKxERKQyBDQL1CEREXIELgvRhh5EtLyeiIBARAQIY\nBITDNI0eTfSttyCd9rsaERHfBS8IgNSRR+I0NBB+5x2/SxER8V0gg6ApNzeBrhOIiAQ0CPQRUhGR\njwUyCFo+QqogEBEJZhBkhgwh07cvUd1LICISzCDAcWgaO5bwpk04dXV+VyMi4qtgBgHu6SEnm9VI\npCISeIENgtSRRwK6TiAiEtggaBlq4o03fK5ERMRfgQ2C1NixZB2H6Nq1fpciIuKrwAZBtrKS9IgR\nbo9Ak9SISIAFNggAmsaNI7RnD+HNm/0uRUTEN4EPAoDomjU+VyIi4h8FAQoCEQk2BQEKAhEJtoiX\nBzfGhIBFwAQgCVxsrd2Yt/4rwBVAGrjDWnuLl/W0lhk4kPTQofrkkIgEmtc9gjOAcmvtVOBq4OZW\n6xcAs4ATgCuMMf08rmcfTUcdRXjbNkI7d/b0S4uIFARPewTAicAyAGvtKmPMpFbrXwP6ACnAATr8\nHGcikehyMW3uO3UqPPEEQ7duhaOP7vKxC9XB/LyKldocDGpz9/E6CKqAPXnP08aYiLU2lXu+BngR\n2AvcZ639qKMDbtmypUuFJBKJNvctHz6c/kD1009Tm7tmUCr21+ZSpjYHg9rc+X3b4/WpoWqgd/7r\nNYeAMWY8cBpwGDACGGyMOdvjevahC8YiEnReB8EKYA6AMWYK8Hreuj1APVBvrU0D24Eev0aQPvRQ\nMn36KAhEJLC8PjW0BDjVGPMs7jWAC40x5wOV1tpbjTG/BP5ijGkE3gbu8riefTkOTZ/+NGWrVuHs\n3Uu2V68eL0FExE+eBoG1NgPMa7V4fd76XwC/8LKGA9F09NHEVq4kunYtjcce63c5IiI9KtA3lDVr\nGj8egOirr/pciYhIz1MQAI3HHANA9JVXfK5ERKTnKQiA9IgRZPr2pUxBICIBpCAAcBwaJ0wg8u67\nOLt3+12NiEiPUhDkNOVOD5W99prPlYiI9CwFQU7LdYKXX/a5EhGRnqUgyGmaMAHQJ4dEJHgUBDmZ\nIUNIDxvmXjDWHMYiEiAKgjyNn/kM4e3bCW3d6ncpIiI9RkGQp/n0kD5GKiJBoiDI03LBWNcJRCRA\nFAR5moeaKNMnh0QkQBQEebJVVTSNGuX2CNJpv8sREekRCoJWmiZOJFRbS+TNN/0uRUSkRygIWmmc\nPBmAshde8LkSEZGeoSBopXHSJEBBICLBoSBoJXX44e5IpAoCEQkIBUFroRCNEycS2bSJ0Pbtflcj\nIuI5BUEbdHpIRIJEQdAGXTAWkSBRELSh6ZhjyIbDlD3/vN+liIh4TkHQhmw8TtO4cURffx0aGvwu\nR0TEUwqC/WicNAmnqUkzlolIyVMQ7EfLBWOdHhKREqcg2I+WC8arV/tciYiItxQE+5EZNozUiBGU\nPfecBqATkZKmIGhHcupUQjU1RNeu9bsUERHPKAja0Th1KgBlK1f6XImIiHcUBO1ITpkCQExBICIl\nTEHQjswhh+g6gYiUPAVBB5JTphCqrib6xht+lyIi4omIlwc3xoSARcAEIAlcbK3dmLd+MvBjwAG2\nARdYawvqVt7GqVPp9Yc/ULZyJU1HH+13OSIi3c7rHsEZQLm1dipwNXBz8wpjjAPcBlxorT0RWAYM\n97ieTtMFYxEpdV4HQfMfeKy1q4BJeeuOAHYC3zXGPA30t9YW3ETB6UMOITV8ODFdJxCREuXpqSGg\nCtiT9zxtjIlYa1PAQOB4YD6wEVhqjHnBWvtkewdMJBJdLqbL+558Mtx5J4kPP4TPfrbLr++Hg/l5\nFSu1ORjU5u7jdRBUA73znodyIQBub2CjtXYdgDFmGW6Pod0g2LJlS5cKSSQSXd43/pnP0O/OO9mz\nZAl7hw7t0jH8cDBtLlZqczCozZ3ftz1enxpaAcwBMMZMAV7PW/dXoNIYMyr3fBpQkLfwJqdNA6D8\n6ad9rkREpPt53SNYApxqjHkW95NBFxpjzgcqrbW3GmO+Dtydu3D8rLX2IY/r6ZLM4ME0jR3rDkBX\nXw/xuN8liYh0G0+DwFqbAea1Wrw+b/2TwLFe1tBdkjNmEF23jtjq1SRnzPC7HBGRbqMbyg5Q8x//\n2J//7HMlIiLdS0FwgJKTJ5ONxYjpOoGIlBgFwYGKx0kedxzRdesIbd/udzUiIt1GQdAJyenTAYg9\n84zPlYiIdB8FQSe0BIFOD4lICVEQdEJq7FjSgwa5PYJMxu9yRES6hYKgM0IhkjNnEt6+neiaNX5X\nIyLSLRQEndRwyikAxB5/3OdKRES6h4Kgk5IzZpCNRChXEIhIiVAQdFK2qorG446j7NVXCX3wgd/l\niIgcNAVBFzTMmgVA+ZPtDpQqIlIUFARd0BwEuk4gIqVAQdAF6ZEjSY0c6Y471FBQUyyLiHRah0Fg\njHnJGPPpVsv6GGMGeldW4WuYNYtQXR2xVav8LkVE5KAcSI9glLX2DQBjzCW5ZVXk5iIOqpbrBI89\n5nMlIiIH50CCIGmMKc99vxDAWrsZGOlZVUWg8dhjyfTtS/myZbrLWESK2oFMTPMYcKsx5i0gY4wZ\nCXxI0K8vRKM0zJ5NhbVEX3qJpkmT/K5IRKRLDuSP+TxgLzAIuBpYCtyLOx9xoNXPmQNA/OGHfa5E\nRKTrOuwRWGurgW82PzfG/A04CrjTw7qKQnL6dDKVlZQ//DDV114LjuN3SSIindbpOYuttQ8AD3hQ\nS/GJxWiYNYuK++8numYNTUcf7XdFIiKdFuzz/N2gIXd6qPyhh3yuRESkaxQEByl50klkysspf/hh\nyGb9LkdEpNMUBAcpW1FB8uSTib79NpENG/wuR0Sk0xQE3aDhtNMAiD+gSyciUnwUBN2gYfZsMhUV\nxJcs0ekhESk6CoJukK2ooOHznyeyaRPRl17yuxwRkU5REHST+jPPBHB7BSIiRURB0E2S06aRHjDA\nvU6QSvldjojIAVMQdJdolIbTTye8cyexZ57xuxoRkQOmIOhGdc2nh+67z+dKREQOnIKgGzVNnEhq\n+HDKly3D2bvX73JERA6IgqA7OQ71Z51FqK6O8qVL/a5GROSAdHrQuc4wxoSARcAEIAlcbK3d2MZ2\ntwK7rLVXe1lPT6g75xwqFy6k4ve/p/6cc/wuR0SkQ173CM4Ayq21U3HnMri59QbGmG8AJTNsZ/rv\n/o7k9OnEnn+eyFtv+V2OiEiHvA6CE8nNbWytXQV8YhovY8zxwHHALz2uo0fVnXsuABV/+IPPlYiI\ndMzJejgkgjHmduBea+0juefvASOttSljzDDgLuBMwABjDuDUUHGM35BMwiGHQDgMmzdDWZnfFYmI\n7HfmLE+vEQDVQO+85yFrbfPdVmcDA4GHgaFAhTFmvbX2rvYOuGXLli4VkkgkurxvV1SdeSaVt9/O\nrl//umXOgp7W020uBGpzMKjNnd+3PV6fGloBzAEwxkwBXm9eYa39b2vtRGvtTOD/A3d3FALFpO68\n8wCouPtunysREWmf10GwBGgwxjwLLAS+a4w53xhzqcev67vUmDE0TpxI7KmnCL/zjt/liIjsl6en\nhqy1GWBeq8Xr29juLi/r8Mveiy6i34sv0mvxYqqvv97vckRE2qQbyjxUP2cO6cGDqfjjH3WnsYgU\nLAWBl8rKqLvgAkLV1Rp/SEQKloLAY3svuIBsJEKvO+/U7GUiUpAUBB7LDBlC/WmnEX3zTcqefdbv\nckRE9qEg6AF7L7oIgMpbb/W5EhGRfSkIekDTpEk0TppE+eOPE1m/z4emRER8pSDoITWXXQZA5S23\n+FyJiMgnKQh6SHLWLJqOOIL4/fcTfv99v8sREWmhIOgpoRC18+bhpFL0uu02v6sREWmhIOhB9Wee\nSXroUCp+9zuc3bv9LkdEBFAQ9KyyMmovvZRQXR2V6hWISIFQEPSwuq9+lfSgQfS6/XacXbv8LkdE\nREHQ07LxOLXz5xPau5fKX5bUxGwiUqQUBD7Y+5WvkB46lF6/+hWhHTv8LkdEAk5B4Id4nJpvf5tQ\nfT2Vixb5XY2IBJyCwCd1551HKpGg1+LFhLZu9bscEQkwBYFfYjFqrrgCp6GBqh/9yO9qRCTAFAQ+\nqj/7bJrGjiV+zz1E1qzxuxwRCSgFgZ/CYaqvuw4nm6XPDTdovgIR8YWCwGfJ6dNpOPlkYitWEHvi\nCb/LEZEAUhAUgOof/IBsKETVDTdAY6Pf5YhIwCgICkDqyCOp++pXib79tm4yE5EepyAoENVXXUV6\n4EAqf/ITwps3+12OiASIgqBAZPv0ofq66wg1NNDn2mv9LkdEAkRBUEDq584lOXUq5Y89Rvmjj/pd\njogEhIKgkDgOe374Q7LRKH2uuQZnzx6/KxKRAFAQFJjU6NHU/NM/Ed62jT7XX+93OSISAAqCAlQ7\nfz6N48dTYS2xxx7zuxwRKXEKgkIUjfLRwoVky8roe9VVmtZSRDylIChQqTFjqLniCsLbt9P36qs1\n/ISIeEZBUMBq580jeeyxxJcupeK3v/W7HBEpUQqCQhaJsPtnPyPTty99rr+eyLp1flckIiUo4uXB\njTEhYBEwAUgCF1trN+atPw+4HEgBrwPfstZmvKyp2GQOOYTdCxcy4MIL6TdvHjseeYRsRYXfZYlI\nCfG6R3AGUG6tnQpcDdzcvMIYEwduBE6y1p4A9AG+6HE9RSk5eza1F19MdONG+lx5pa4XiEi38joI\nTgSWAVhrVwGT8tYlgeOttXW55xGgweN6ilb1NdfQOHEiFfffT+Utt/hdjoiUEE9PDQFVQP7tsWlj\nTMRam8qdAvoAwBjzbaAS6PBD84lEosvFHMy+BWHpUpg0iaqbbqLq+ONhzpwOdyn6NneB2hwManP3\n8ToIqoHeec9D1tpU85PcNYQfAUcAZ1lrOzznsWXLli4VkkgkurxvIYnedhsD584le+657Fi6lNSo\nUfvdtlTa3BlqczCozZ3ftz1enxpaAcwBMMZMwb0gnO+XQDlwRt4pImlH04QJfPSf/0mopob+X/kK\noW3b/C5JRIqc1z2CJcCpxphnAQe40BhzPu5poBeArwPPAE8aYwD+y1q7xOOail793LmEN22iasEC\nBvzDP7Dj3nvJVlX5XZaIFClPgyB3HWBeq8Xr877XfQxdVHv55YS3b6fXr39N/4suYufvfgexmN9l\niUgR0h/iYuU47LnxRurnzCG2ciX95s3TfMci0iUKgmIWDrP7pz8leeKJxJcvp983vwlNTX5XJSJF\nRkFQ7MrL2XXXXSRPOIH4smUKAxHpNAVBCcjG4+xavJjk8ccTf+QR+l9yCdTX+12WiBQJBUGJaAmD\nadMof+wxBpx/PmgeAxE5AAqCEpKtqGDn4sXUf+lLxFavhunTCW3d6ndZIlLgFASlJhZj989/Tu1F\nF8GaNQz64heJvvqq31WJSAFTEJSiUIjqG26AH/2I0AcfMHDuXOJLdJ+eiLRNQVCqHAeuvJJdixeT\njUbpN38+VTfeCKlUx/uKSKAoCEpc8pRT3MHpRo6k8pZbGHjWWYT/9je/yxKRAqIgCIDUqFF8+NBD\n1H/pS5S98AKDZs+m/KGH/C5LRAqEgiAgslVV7F60iI8WLIBkkv6XXkrfyy/H0UdMRQJPQRAkjkPd\neeexY9kyGo8+mop77mHwySdT/sgjflcmIj5SEARQavRodixdSvX3v09ozx76X3wx/S65hPD77/td\nmoj4QEEQVJEItfPn8+Hy5SQnTyb+8MMMmj6dyh//GEfDU4gEioIg4FKjRrHzvvvY/ZOfkK2qourm\nmxk0fTrxe++FdNrv8kSkBygIBEIh6s8+m+3PPEPN/PmEd+yg33e+w6BTTqH8wQchk/G7QhHxkIJA\nWmQrK6n5/vfZ/uc/s/e884j89a/0nzfP/bjp//6vbkYTKVEKAtlH+tBD2bNgAduffpq6uXOJrF9P\n/299i8EnnECv227Dqa31u0QR6UYKAtmv9GGH8dFPf+r2EL72NUI7dtDn+usZMnkyVdddR2T9+o4P\nIiIFT0EgHUqPHMmem25i+/PPU33llWRjMSp/9SsGn3IKA08/nYrf/x5n716/yxSRLlIQyAHL9O9P\n7eWX88Hzz7Pr9ttpOPlkoi+/TN9//meGjB9Pv298wx26Qh8/FSkqEb8LkCIUjdLwhS/Q8IUvEH7/\nfeJ//CPx++8nvnQp8aVLyfTqRcPs2TSceirJGTPI9u3rd8Ui0g4FgRyU9CGHUPu971H73e8SWbuW\n+IMPEn/gASqWLKFiyRKy4TCNkyaRPOUUGk46idSYMRBSR1SkkCgIpHs4Dqlx46gZN46aq68msnYt\n5Y8/TvkTT1C2ejWx556j6qabSPfrR+OUKTROmUJy6lRSY8cqGER8piCQ7pcLhdpx46i9/HJCu3YR\n+9OfiD39NGWrVhF/5BHiuYHuMn370njMMTRNmEDThAk0TphAZuhQnxsgEiwKAvFcpn9/6s86i/qz\nzgIgvHkzZc8+S2zVKsqee47yp56i/KmnWrZPDx1K4/jxpI48ktSYMTQdcQSpww+HWMynFoiUNgWB\n9Lj0oYdSf8451J9zDgDOrl2UvfYa0VdeIfrqq5S9+irx5cth+fKWfbLhMKnDDiN1xBGkRo4kPXw4\nqeHDSY8YQXrYMJ1eEjkICgLxXbZ/f5IzZ5KcObNlWWj7diJvvkl0wwb38c03iWzYQHTjxn33Lysj\ndeihpIcPh1GjqKyqIj1sGJmhQ0kPG0Z66FCyVVXuPM4isg8FgRSkzODBNA4eTOO0aR8vzGYJbdtG\nZNMmwps2EXn3XffxvfcIv/su0bffhiefpKqt48XjZIYOJTNgAOkBA8gMGECmf/9PPjZ/37s32d69\n1cuQwFAQSPFwHDLDhtE4bBhMmbLv6poahqXT7HztNUJbtxLeupXwtm2Et24ltG0b4Q8+IPree5Qd\nwPDaWcchW1nphkJVFZmqKrK9e3/ysaKCbDy+38dMPO4+zy0jFlOvRAqSgkBKRrZ3b0gkSLZ3A1sm\ng7NnD6GdOwnv2kVo585Pfu3eTaimBqemhlB1NU51NeGtW4ls2IDTDcNxZ8vKWr7I/z4aJRuLtf19\n8zaRCITDZMNhiETIRiJur6VfPyrr6qB5eTjsbpfbnkiEbCj0yX2at3Mc9ysUcr8ch2wbywiFPrk8\nb32by5uP02rb5tfLNv9AmoNxf4/7WxeJENq5c59tD/q4bTxmodP77NfBrveIp0FgjAkBi4AJQBK4\n2Fq7MW/96cB1QAq4w1p7m5f1iBAKke3Xj3S/fnRq2p1sFmfvXpzqakLNIVFfj1NX1/5j81ddHU5D\nA05jI05jIzQ24jQ14SST7nGbmtzlySRONttxPa20dTqs1AXpQ8ZZx4EFC+Dccz05vtc9gjOAcmvt\nVGPMFOBm4MsAxpgosBCYDOwFVhhjHrDWfuBxTSKdlztVlK2sxNNperJZSKc/Dovm4Egm3R5JKuWu\nT6chlcJJpxnYrx87tm1zl+W+nFTKXd/GPi3rc6fInEzGfd1Mxv3KZlu+nPxleev22Qfc3lYb25LJ\nuOGWf6zWbW7rET4OxVaP8fJy6uvr9z1We8c7gOPut7Z2tt3nWPtzMOtDIWJHHdX+/gfB6yA4EVgG\nYK1dZYyZlLduLLDRWrsbwBjzF2A6cI/HNYkUrtxpj2wkAhUVdPCnw5VI0Lhli9eVFZR4IsHugLU5\nkUiAR232OgiqgD15z9PGmIi1NtXGuhqgT0cHTCQSXS7mYPYtVmpzMKjNweBVm70Ogmqgd97zUC4E\n2lrXG/ioowNu6WIiJhKJLu9brNTmYFCbg+Fg2txRgHgdBCuA0wGbu0bwet66dcBoY0x/oBb3tNAC\nj+sREZFWvA6CJcCpxphnAQe40BhzPlBprb3VGPM94FHcCXLusNa+73E9IiLSiqdBYK3NAPNaLV6f\nt/5B4EEvaxARkfbpHnoRkYBTEIiIBJyCQEQk4JxsF25n91FRFSsiUkD2O5BRsQ06p6EbRUS6mU4N\niYgEnIJARCTgFAQiIgGnIBARCTgFgYhIwCkIREQCrtg+PtppHU2XWSxyM7rdAYwAYsCNwBvAXbj3\nV6wBLrPWZowxlwDfwJ0C9EZr7VJjTBz4LTAYd+6Hr1lrP8yNCvtfuW2XW2v/rUcb1gFjzGDgReBU\n3BrvorTb+33gS0AZ7u/t05Rwm3O/14txf6/TwCWU8L+zMeY44D+stTONMaPwqJ3GmP8HnJZbfrm1\ndnV7dQWhR9AyXSZwNe50mcXoAmCntXYa8HngZ8CPgR/kljnAl40xQ4HvACcAnwN+aIyJAd8EXs9t\n+2vgB7nj/gI4H3c2ueOMMZ/pwTa1K/dH4pdAfW5Rqbd3JnA8bltmAIdS4m0G5gARa+3xwA3Av1Oi\nbTbGXAXcDpTnFnnSTmPMZ3F/f44DzgV+3lFtQQiCT0yXCUxqf/OCdQ9wbe57BzfpJ+K+YwR4BJgF\nHAussNYmrbV7gI3AePJ+Ds3bGmOqgJi19m1rbRZ3SPBZPdGYA7QA9xe9eTaOUm/v53Dn7FiCOyrv\nUkq/zRuASK7nXgU0UbptfhuYm/fcq3aeiNs7yFpr38P9+Q5qr7AgBEGb02X6VUxXWWtrrbU1xpje\nwP/gviNwcr8A8PFUn/ubAjR/ef6y6ja29Z0x5h+BD621j+YtLtn25gzEfaNyNu7w7b/DndWvlNtc\ni3taaD1wG/DflOi/s7X2Xtyga+ZVOzs9DXAQgqC96TKLijHmUOBPwG+stXcDmbzVzVN97m8K0Pzl\nHW1bCC7CndToKeAY3O7w4Lz1pdZegJ3Ao9baRmvtm0ADn/wPXIpt/i5um4/AvY63GPf6SLNSbHMz\nr/7/drr9QQiCFbjnIWljusyiYYwZAiwH/sVae0du8cu588oAXwCeAVYD04wx5caYPsBY3AtRLT+H\n5m2ttdVAozHmcGOMg3tq4pkeaVAHrLXTrbUzrLUzgVeArwKPlGp7c/4CfN4Y4xhjEkAv4IkSb/Nu\nPn73uguIUsK/16141c4VwOeMMSFjzKdw3/zuaK+QIATBEqAhN13mQtx3IMXoGqAfcK0x5qncO+Uf\nAP9mjFmJ+y7qf6y123C7188ATwL/aq1tAG4BjjLG/AW4FGj+FEXzKYjVwMvW2ud6sE2ddQUl3F5r\n7VLgZdzaHgQuo8TbjPt/8rPGmOa2XIPb7lJuczNP/m2ttS/m9l8J3Iv782xXsQ1DLSIi3SwIPQIR\nEWmHgkBEJOAUBCIiAacgEBEJOAWBiEjAKQhEusAYs9wYM9DvOkS6g4JApGtO9bsAke6i+whE2mGM\nqQTuBEbjDgnwIu7w7f+Ie8fnnNzynwGfwr0z9g/W2puMMSNwBxV7Cnf4BAeYb60txLtcJcDUIxBp\n35lAb2vtMcDk3LJ/zz2eZK3dDPwGuMNaOxF39MhZxhiT2+ZTuGPpHIM7DPofc8NrixQM9QhE2mGM\nOQz4M+4Qwo8B91tr1xpjssAg3LkSqvnkGFaVgAVuBV6y1vbPO95m4IzcMAAiBUE9ApF2WGvfAUYB\nP8Qd3vdxY8zf520Sxj3lc7y19pjcO/8pwE259a1Hug3hzsQlUjAUBCLtMMZ8E/cawXJr7b/gTv4x\nDvePeTQ3AuQq4Hu57fvijv745dwhBhljPp9bdzruePRFOQKulC6dGhJphzGmF+5c0eOBvcB7wNdx\np9CciPsHvxb3YvFw3FEkf2+tvT53sXgdcB9ueNQD86y1r/RwM0TapSAQ8UguCNZYayv9rkWkPTo1\nJCIScOoRiIgEnHoEIiIBpyAQEQk4BYGISMApCEREAk5BICIScP8HVcvncR7qEu4AAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2aaf322b208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs4AAABWCAYAAADfRhArAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8TOf+B/DPZJkKEzRVGo3YWuGWWku1dg0VCYmRJkEW\norWEWIokEoRsltiJrXoRSlBFys8ay61qLLdc+0VRjdaalBnLJJnn98f8MnfmnOcsaVPk/r7v1yuv\nezPn4zln5jvnmaczJ/PVMMYYCCGEEEIIIbIcXvQBEEIIIYQQUh7QwpkQQgghhBAVaOFMCCGEEEKI\nCrRwJoQQQgghRAVaOBNCCCGEEKICLZwJIYQQQghRgRbOhBBCCCGEqEALZ0IIKQd69+6N3bt3W3+/\nePEivLy88M0331hvu3z5Mpo3b46HDx++iEMkhJD/ek4v+gAIIcSWl5cXGjRoAAcH+/+uX7x4MTw8\nPLj/5syZM1ixYgUWLFhQJsewa9curFu3DpmZmar/zbZt27By5UpoNBq4uLggPj4eTZo0AQAcPHgQ\ns2fPhslkgpeXF1JTU6HT6RS32XJ1dYXBYLD+vnr1atFtmZmZ8Pf3R+XKlf/oXSeEkJfe5Xbd7X5/\n+7vdEsmyRwtnQshLZ/Xq1XBzc1Odb9KkSZktmv+In376CbNmzcKWLVtQvXp1HDp0CCNHjsTBgwfx\n4MEDxMXFYf369ahTpw5mzZqF9PR0JCYmym4T0ul01kXygwcPkJOTgz59+uDRo0cAgIcPHyI7Oxub\nN29+nnedEEKePwfHF7frF7ZnQggppdzcXOj1egwfPhx+fn4IDAzE1atXkZubC19fXwCA0WhEdHQ0\nevfujYCAACQkJMBsNgMAsrKy4Ovri169emHQoEG4du2adez58+fjo48+Qt++fbF3717r7Tk5OQgM\nDIS/vz+Cg4Px448/io5Lq9UiOTkZ1atXBwA0btwY9+7dg8lkwnfffYcmTZqgTp06AICQkBBkZ2eD\nMSa7Tcj23eUNGzagZ8+eqFmzpvW2r7/+Gi1atED9+vX/5KNMCCEvN43W2e7neaJ3nAkhL53w8HC7\nSzU8PDywePFiAMD58+cRFxeHVq1aYf369Rg/fjxiYmKs2b1798JoNGLbtm0oLi7GlClTcPPmTdy6\ndQtffPEFsrKy4Obmhi1btiAqKgo7duzA/v37sWfPHmzduhUVKlRAVFQUAOD69euYO3cu1qxZg1df\nfRWXL1/GwIEDsWfPHlSsWNHu+EouI2GMIS0tDV26dIFWq8Vvv/2GN954w5p94403YDAYYDQaZbcJ\nL9coWTgXFhZiw4YNWL16NU6cOAGDwQCz2Yx169Zh0qRJePToEQYOHIirV68iKysLDRo0KMPKEELI\ni6dxfr6LZVu0cCaEvHTkLtVo2LAhWrVqBQDQ6/WYNm0aCgoKrNtbtmyJuXPnIjQ0FB988AHCw8NR\nu3ZtZGVlwcfHxzpunz59kJKSgl9++QVHjx6Ft7e3dbGq1+uRmZmJI0eO4M6dO4iIiLCOr9Fo8PPP\nP6Nhw4aiY3v8+DFiY2Px22+/4YsvvgAA67vdQg4ODrLbhFxdXZGfn4/du3ejYcOGqFu3Li5evIhH\njx7h4MGDcHR0RIcOHVBUVITly5dj5syZ3LEJIaS8o4UzIYSo5Ohof20bY8zutlq1amHv3r3Izc3F\nDz/8gIEDByIhIYF7+QNjDEVFRdBoNHbbS8Yzm81o27Yt5s2bZ93266+/Wi/JsHXr1i0MHToU9evX\nx5o1a1ChQgUAgLu7O06fPm3N3b59G1WqVEHFihVltwnpdDrcvHkTa9asQXR0tPU2g8GAtWvXIjQ0\nFBqNBs7OzqW6PpwQQsqb5315hi26xpkQUq5cvHgRFy9eBGC5ZrlFixZwdXW1bv/qq68QFxeHdu3a\nYfz48WjXrh0uX76Mdu3aYefOnXjw4AEAyzXBVatWRe3atdG+fXvs2rULDx8+hNlsxrZt2wAA77//\nPo4cOYKrV68CAA4dOoRevXrh2bNndsdUUFCAAQMGoFu3bpg7d6510QwA7dq1w+nTp3H9+nUAluuT\nu3btqrhNyNXVFcePH4fRaES7du0AWBbOFy5cwJkzZxAQEPBnHlZCCCk3HLRaux85hYWFGD9+PPr1\n64e+ffti//79dttzcnKg1+sRFBSEjRs3Ku6b3nEmhLx0hNc4A8DYsWNRoUIFVKtWDfPmzUNeXh7c\n3Nwwc+ZM5OXlWXP+/v44duwYfHx84OLigpo1ayIsLAxVqlRBREQEwsPDYTab4ebmhmXLlsHBwQEd\nO3bEpUuXoNfrUblyZTRs2BD5+fl4++23MW3aNIwdOxaMMTg5OWHJkiWid4TXr1+PX3/9FXv37rX7\nw8JVq1bhtddeQ1paGqKjo1FYWAhPT0/MmDEDAGS3Cbm6uuLOnTvW668By8L57t27CA8PR6VKlf70\n404IIeVBaS7V2L59O6pWrYpZs2ahoKAA/v7+1jcoCgsLkZaWhs2bN8PFxQUhISHo0qULqlWrJr1v\nxvv8khBCXkK5ublISkrCt99++6IP5aUXGxuLQYMG0R8HEkL+69z8NNru91orpL+O1Gg0gjEGnU6H\n/Px8u3edL168iFmzZmHlypUAgNTUVDRv3hw9evSQHI/ecSaEkP8yn376KS5cuIBr164hKCgIffr0\nedGHRAghZaY07ziXfBpnMBgQHR2N0aNHW7cZDAa7S/0qVapk11SKhxbOhJByo02bNvRusworVqx4\n0YdACCF/mdJ+q8avv/6KqKgo9OvXD35+ftbbdTodjEaj9Xej0Wi3kOahPw4khBBCCCHlRmkaoNy7\ndw+DBg3C+PHj0bdvX7tt9evXx40bN1BQUACTyYQTJ06gefPmsuPRO86EEEIIIaTc0Ch8k4atpUuX\n4uHDh8jIyEBGRgYAIDAwEE+ePEFQUBBiY2MRGRkJxhj0ej1q1Kghv2/640BCCCGEEFJe/DYlze73\nN6bGPbd90zvOhBBCCCGk3HiRDVD+8oXziC+3yG5fNMjy195zdxySzY3p2REzsw8o7m+CX2ekf3tQ\nMTfOtxMC566SzWwaEwEAiFu/QzaXFtITABC2eJ1sbk1Ufwz7YrPisS0Z3BdpW/fLZuL8Ld9B+Hnm\ndtnc7NBeqnPzdh5WPLbRPh0UH99xvp0AAJ8tl/8i8eWffQIAGLpik2xu6aeB6J6yVPHYdscPRXjG\nV7KZ1cP7AQCGr5SvQ0ak5TooNc9ftY+b0vMDsDxH1N4HpfNhgl9nAED03+Xvw4KBfTD/f/6heGyj\nerSHT9py2czOuM8AQPW5pWa8AQvXKh7b2pEDEDxvjWxmw+gwAEBC1v/I5pKDLF9DNH6t/Dkza0Av\n1edz0LzVspms0eEAgP4LMmVz66JDAah7fMes3qp4bHPD/TFpo/zjAQBJn/SA/6yVspmt4yMBAL4z\n5P8w8duYTwGoe16GzJd/PABg/ahQ1XO02uflhHXZsrmZ/f1UP24ztufIZmJ6dQEA1bmFu76TzY38\nuJ3ivAVY5i61rx9K8+/u+KEAgOQte2VzCX28y+ycASznjdrzWel8mBvuDwDoNfML2dz2CYMxatU3\nisc2PyIAi/cckc1EdfsQAJD6zT7Z3MSAjwBA8XxYPyoUH6cuUzy2XROHoOd0+bl3R+xniuO8SBpn\n9ZdqlDV6x5kQQgghhJQb5eIdZ7PZLOrkRQghhBBCyPOkcX5x7/vK7vnmzZtIS0vD2bNn4eTkBLPZ\njAYNGiAuLg5169Z9XsdICCGEEEIIgNJ/j3NZkl04x8fH4/PPP0fTpk2tt506dQpxcXHYsGHDX35w\nhBBCCCGE2CrN19GVOH36NNLT05GZaX+t+KpVq7Bp0ya4ubkBAKZOnYp69epJjiO7cDaZTHaLZgBo\n1qxZqQ+WEEIIIYSQslDaa5xXrFiB7du3w8XFRbTt7NmzmDFjBho3bqxqLNmFs5eXF+Li4tC+fXu4\nurrCaDTi0KFD8PLyKtUBE0IIIYQQUhZK+60anp6eWLhwISZMmCDadu7cOSxfvhx3795Fp06dMGTI\nENmxZBfOiYmJ2LdvH06ePAmDwQCdTofOnTvD29u7VAdMCCGEEEJIWdA4O5Yq3717d/zyyy/cbT17\n9kS/fv2g0+kwYsQIHDhwAJ07d5YcS3bhrNFo4O3tTQtlQgghhBDyUvgj1zjzMMYQHh4OV1dXAEDH\njh1x/vx52YUztdwmhBBCCCHlxqO99g3AXL2lF7olfvnlF4wdOxYbN/6nOdujR4/g6+uLnTt3omLF\nihg1ahT0ej06duwoOc5f/kV4nacult1+YEoUACB+w07ZXEqwDxI371bcX2Lf7pj29R7F3GR9N8R8\n9a1sZkY/XwDqux8OXir/TSNfDA1W3Q1JqbvSksGWznZqO2F9lJQhm9s3abjqbkhqOn4B6rvzqek2\npdRZErB0l1TqgDeqR3vL/yrc1/kRAQCAKZt2yeamBn6s2LEOsHStU+oOBVg6RKl9vqm9r2o6OCp1\nXgMs3dfUdFQDoNjtb+3IAQCg6nmutmNon9l/l81s+XwgAHUd6wB1c5LS8wOwPEfUPI8AwDt5iWxu\nb8IwAOo6jak9Z9Tm1D5uajvgqemSqNS9EbB0cFTbzVRtV041zyWlTElO7bHNVsh9/n85NXOXUjdW\nwNKRVe15unz/D7K5z7q+D0BdB2C186BSR0DA0hVQbdc9tZ1F1XQzVeouCFg6DCp1lR3t0wGA+tdx\nNR0yldY1gGVto6YD5cvsz77jnJ2djcePHyMoKAhjxoxBWFgYtFot2rZtK7toBqhzICGEEEIIKUc0\nTqVfvnp4eFjfbfbz87Pe7u/vD39/f9Xj0MKZEEIIIYSUG+Wi5TYhhBBCCCEvWln9ceAfQQtnQggh\nhBBSbry07ziHhoaisLDQ7jbGGDQaDbXcJoQQQgghz90fuca5rMjuedy4cUhISMDixYvh6Fi6L5sm\nhBBCCCGkrP2RSzVOnz6N9PR0ZGbaf6NPTk4OFi9eDCcnJ+j1enzyySey48gunJs2bYrevXvj0qVL\n1ASFEEIIIYS8cBrn0l2qsWLFCmzfvh0uLi52txcWFiItLQ2bN2+Gi4sLQkJC0KVLF1SrVk1yLAel\nnQ0ePJgWzYQQQggh5KXw1LWS3Y8ST09PLFy4UHT71atX4enpiSpVqkCr1aJly5Y4fvy47FjUOZAQ\nQgghhJQbjx49svu9pGW2HF7nwBMnTmDt2rWYN28eAGD+/PmoWbMmAgMDJcf5y6+uXrRbvjvNiO6W\n7jRJCt3jJvXxVuzMB1i684UtXqeYWxPVX3UXqZD58h2u1o8KBQBVXYIC0r9UPLZvxg3Cx6nLZDO7\nJg4BAMXxvhk3CIC6bl5qHze1x5a2db9sLs6/KwB1nbAGLVGu/ZfDgjEwY71s5u/DQwCoe74B6rpI\nKXVgAixdmJQ6pQGWbmlqu02p7eKnpmud2vtQ1p0ZfWeskM19G/Op4v0ELPdVTVctAKo7hqrp4vd5\npnJnu9mhvVQ/L9Xm1HSXVPu4qe2Ap/Z8Vnsf1HQuUzpPAcu5qvZ1Rqles0N7AQB6Tpc/73fEfqZY\nA8BSB7XHptTtb+mnlhdyNeeg2rlGaV79clgwAKDvHPnnyOaxlq6carpBqq3ppI3KnQOTPumhqvMs\noL6bqZr5V6kTL2DpxqumkyKg/vFVekySPumh+PoMWF6j1Xa0/G+n0+lgNBqtvxuNRsVFuOKlGoQQ\nQgghhLwsTA7Odj9/VP369XHjxg0UFBTAZDLhxIkTaN68uey/KfU7ziaTCdoX+MXThBBCCCHk/y9T\nUdGf+vfZ2dl4/PgxgoKCEBsbi8jISDDGoNfrUaNGDdl/K7lwzsnJQVJSEpycnDBmzBj4+PgAsPyx\n4Jo1a/7UARNCCCGEEPJHFBYXl/rfeHh4WK9v9vPzs97epUsXdOnSRfU4kgvnpUuXYuvWrTCbzRg1\nahSePXuGgIAA0N8SEkIIIYSQF6WwyPzC9i25cHZ2dkaVKlUAABkZGQgPD4e7uzs0Gs1zOzhCCCGE\nEEJsFRb/uUs1/gzJPw588803kZaWhsePH0On02HRokWYNm0afvrpp+d5fIQQQgghhFiZiortfp4n\nyYVzamoqvLy8rO8wu7u7Y82aNejRo8dzOzhCCCGEEEJsvciFMzVAIYQQQggh5cYPV362+/39tzyf\n277/8gYohBBCCCGElJXSvMtsNpuRmJiIS5cuQavVIjk5GbVr17ZuX7VqFTZt2gQ3NzcAwNSpU1Gv\nXj3J8f7yhbPaLmhqOo0pdf4BLN1/lLo+AZbOT2o7jantlhY4d5VsbtOYCNWdxlYfPiGbCe/QCgCw\nZO/3srlh3h8AAMau2SabmxPWG8HzlL9mcMPoMFX3EwD8Z62UzW0dHwlAXfdDtV2k1HQjA4DFe47I\n5qK6fQhAXbcppa5PgKXzk1JHNcDSVU1N9y1AfWdGpY6Qa6L6Y9gXmxWPbcngvqq7RnadJt9Za//k\n4QCAz5ZvlM0t/+wT1ed9r5lfyGa2TxgMAKpz077eI5ubrO+G+A07FY8tJdhHdVc4NV0eAXVzktrz\neWb2AcXcBL/OGL9Wfu6aNcDSdU/tfVDTzVTpdQGwvDYEzVstm8kaHQ4AqrulRf9dvivggoF9FDtL\nApbukmo7zyrVa8PoMADquvEOX6l8PmdE9kVClvy8mhxkuTRT7TmjpqOl0uMBWB4TtZ0v1XYAVtud\nT819UNtpVe0crbYLrJrXcaV5C7DMXZ0SF8lmDiaOUBznRSrNwnnfvn0wmUzIysrCqVOnMH36dCxZ\n8p/z9+zZs5gxYwYaN26sajx6x5kQQgghhJQbpflWjZMnT6J9e8sbnM2aNcPZs2fttp87dw7Lly/H\n3bt30alTJwwZMkR2vFItnJ8+fQoHBwfqHEgIIYQQQl6I0nyPs8FggE6ns/7u6OiIoqIiODlZlsA9\ne/ZEv379oNPpMGLECBw4cACdO3eWHE/yWzUA4MqVKxg+fDji4uLw/fffw8fHBz4+PjhwQPmjPUII\nIYQQQspaYXGx3Y8cnU4Ho9Fo/d1sNlsXzYwxhIeHw83NDVqtFh07dsT58+dlx5NdOE+ZMgURERFo\n3bo1oqOjsWnTJmzduhXLlslf50gIIYQQQshfwVRUZPcjp0WLFjh82PL3dqdOnUKDBg2s2wwGA3x9\nfWE0GsEYQ25uruK1zrKXapjNZrRu3RoAkJubi9dee83yj5zo0mhCCCGEEPL8leaPA729vXHkyBEE\nBweDMYbU1FRkZ2fj8ePHCAoKwpgxYxAWFgatVou2bduiY8eOsuPJroDr1q2L+Ph4JCUlYfr06QCA\n5cuXo1q1aqoPmBBCCCGEkLJSWIqFs4ODA6ZNm2Z3W/369a3/39/fH/7+/qrHk104JycnIycnBw4O\n/7mio0aNGggNDVW9A0IIIYQQQsqK0nXNfyXqHEgIIYQQQsoN4XfZl/TTeB7oYmVCCCGEEFJuvMh3\nnGnhTAghhBBCyg1TofoGKGWNFs6EEEIIIaTcKDKrb4BS1mS/x5kQQgghhJCXiamo2O5HjtlsxuTJ\nkxEUFITQ0FDcuHHDbntOTg70ej2CgoKwceNGxX3TO86EEEIIIaTcUGp6Ymvfvn0wmUzIysrCqVOn\nMH36dCxZsgQAUFhYiLS0NGzevBkuLi4ICQlBly5dZL92md5xJoQQQggh5UZhUbHdj5yTJ0+ifXvL\nt240a9YMZ8+etW67evUqPD09UaVKFWi1WrRs2RLHjx+XHY/ecSaEEEIIIeVGYbH6a5wNBgN0Op31\nd0dHRxQVFcHJyQkGgwGurq7WbZUqVYLBYJAd77ktnM1mMxITE3Hp0iVotVokJyejdu3a3Ozp06eR\nnp6OzMxM7vbCwkJMnDgReXl5MJlMGDZsGLp27SrKFRcXIyEhAdeuXYNGo8HUqVPtepTbun//Pvr0\n6YMvv/zSrqOMUEBAgLUAHh4eSEtLE2WWLVuGnJwcFBYWIiQkBIGBgaLMli1b8M033wAAnj17hgsX\nLuDIkSOoXLmy6L7GxsYiLy8PDg4OSEpK4h6fyWRCXFwcbt68CZ1Oh8mTJ6NOnTrW7baP6Y0bNxAb\nGwuNRoO3334bU6ZMsTa54T32qampqFu3LkJCQrjjXbhwAUlJSXB0dIRWq8WMGTOsH3PY5q5cuYJJ\nkyaBMYY6deogOTkZTk5O3H1mZ2dj7dq1yMrK4u7z/PnzGDJkiPU+hoSEwMfHR5S7f/8+EhIS8PDh\nQxQXF2PmzJnw9PQU5caMGYN79+4BAPLy8tC0aVPMnTtXdD+nTJkCR0dH1KlTBykpKdzH7dy5c5gy\nZQq0Wi0aNWqE+Ph4FBcXi56zb731lqgOvFzJc9u2DrxzoGbNmnZ1SElJwezZs+0ytWvXFtWAMSa5\nT9s68Pbp7u4uqoO3t7co16xZM7s6pKamYsmSJXaZb7/9VlSDmTNncu+nsA68x+2NN94Q1YExJpoT\nXnnlFVEdeLmSuaOkDp988okoU1xcLDoXXn31VVHOwcFBVAeNRiO5T9s68Oa1oqIiuzoEBQXhyJEj\ndpnXXntNdC68+eaborGWLFkiqkN6ejr3vgrrwHvcCgsLRXVwcHCwm3OdnJwk5yTe3Mybk2xzJpOJ\nOyfZZhhj3PlIap+8Ock29+zZM8k5yTZXtWpVyTnJNrdo0SLunCS8n1Jzkm3u6dOn3BoIX8+GDh3K\nrYPU656wDsJcWFiYqA6ffvqpXSYyMpJbB6l9CusgzIWGhnLrIMyNGzeOWwdh7unTp6I6XL9+XXQ/\neXUQjjVgwABuHYRrhtatW4vqsGLFCu66QlgD4ViNGzfmngvCXPPmzSXPh5dRaS7V0Ol0MBqN1t/N\nZrP1vgm3GY1Gu4U0F3tOdu/ezWJiYhhjjP34449s6NCh3Nzy5cuZr68vCwwMlBxr8+bNLDk5mTHG\nWH5+PuvYsSM3t3fvXhYbG8sYY+yHH36Q3KfJZGLDhw9n3bp1Y1euXJHc79OnT1nv3r0lt5fsZ8iQ\nIay4uJgZDAa2YMEC2TxjjCUmJrINGzZI3ofo6GjGGGPfffcdGzFiBDeXmZnJEhISGGOMXb16lQ0a\nNMi6TfiYDhkyhP3www+MMcYmTZrE9uzZw83dv3+fRUZGsq5du7KvvvpKcrz+/fuz8+fPM8YYW79+\nPUtNTeXmhg0bxo4dO8YYYywmJobt2bOHW+9z586xsLAwu9uEuY0bN7KVK1eKHgdhLiYmhu3YsYMx\nxtjRo0fZgQMHuLkSBQUFrFevXuz27duizPDhw9nBgwcZY4yNHTuW7d+/nztWQEAAO3nyJGOMsTlz\n5rCtW7dyn7O8OvByvDrwcsI6DBw4UJTh1UDqfBLWgZfj1YGXE9YhOTlZ8hy2rQFvLF4deDleHXhz\nAq8OvJywDrwM71zg5Xh1kJqvhHXg5YR14GV454LcHGlbB16OVwdejlcH4ZwrNScJc1JzkjDHq4Mw\nw6sBbyxeDXg5qTlJmJOak6Reh2zrIMxIzUnCHK8GvNczXh14OV4deDlhHaZNmybK8Oog9VorrAMv\nx6sDL8erg9xrfEkdfv75Z1GGVwfeWLw68NYMwjosWrRIlOHVgDcW71zg5aTOh5dVxOKv7H7k7Nq1\ny279GRkZad1mMpmYt7c3y8/PZ8+ePWMBAQHst99+kx3vuV3jLHeNiS1PT08sXLhQdqyPP/4Yo0aN\nAgAwxuDo6MjNffTRR0hKSgIA3Lp1S/RubokZM2YgODgY1atXl93vxYsX8eTJEwwaNAhhYWE4deqU\nKPPdd9+hQYMGiIqKwtChQ9GpUyfZMc+cOYMrV64gKCiIu71u3booLi6G2WyGwWCQ/C/AK1euoEOH\nDgCAevXq4erVq9Ztwsf03LlzaN26NQCgQ4cO+P7777k5o9GIkSNHonfv3nb7EubmzJmDRo0aAbC8\ny//KK69wcwsXLsR7770Hk8mEu3fvQqfTiTL5+fmYM2cOJk6cKLvPs2fP4uDBg+jfvz8mTpxo/WhF\nmPvnP/+J27dvIyIiAtnZ2db7LfU8W7hwIQYMGIDq1auLMo0aNUJBQQEYYzAajdZaCHO3b99GixYt\nAAAtWrTAyZMnuc9ZXh14OV4deDlhHby8vEQZXg14Y/HqwMvx6sDLCeswZMgQyXPYtga8sXh14OV4\ndeDNCbw68HLCOvAyvHOBl+PVgZfj1YGXE9bh/fffF2V454LcHGlbB16OVwdejlcH4ZwrNScJc1Jz\nkjDHq4Mww6sBbyypOUmYk5qThDmpOUnqdci2DsKM1JwkzPFqwHs949WBl+PVgZcT1sFgMIgyvDrw\nxuLVgZfj1YGX49VB7jW+pA4PHjwQZXh14I3FqwNvzSCsw759+0QZXg14Y/HOBV5O6nx4WZmKiux+\n5Hh7e0Or1SI4OBhpaWmIi4tDdnY2srKy4OzsjNjYWERGRiI4OBh6vR41atSQHe+5LZylrjER6t69\nu+LHA5UqVYJOp4PBYEB0dDRGjx4tmXVyckJMTAySkpLg5+cn2r5lyxa4ublZF/VyKlSogMjISKxc\nuRJTp07FuHHjRPchPz8fZ8+exfz5860ZJtPVfNmyZYiKipLcXrFiReTl5aFHjx6YNGkSQkNDublG\njRrhwIEDYIxZT9Di/+usI3xMGWPQaDQALI/lo0ePuLlatWqhadOmon0Jc7YvCGvXrkVERAQ35+jo\niLy8PPj6+iI/Px8NGza0yxQXFyM+Ph5xcXGoVKmS7D7fffddTJgwAevWrUOtWrWwePFibi4vLw+V\nK1fGqlWr4O7ujhUrVnBzgOWjzaNHj6JPnz7cTMlHcD169MD9+/fRpk0bycft2LFjAIADBw7gyZMn\n3Ocsrw68HK8OvJywDiUfidpmeDUQjjVq1ChuHXj75NWBlxPWYd26ddxzWFgD3li8Okg9bsI6AOI5\nQep8EOZ4dRBmpM4FYY5XB2HO19dX8nwQjsergzAjdS7w5khhHXg5qfOB97jZ1uHSpUuiOZdXA97c\nzKsBLyesw+uvvy7K8GogHMtsNnNrwNsnrwa8HK8OUq9DtnXgZXg1kHrchOcC7/WMVwdezt3dXVQH\nXs7Nzc1YDyzIAAAIFElEQVSuDnq9nrtPYR2EY40dOxaxsbGiOvD2+c4774jqwMvduHFDVAep13jb\nOvAyHh4eojrwcrw6SK0ZbOtgMBhEGQ8PD1ENeGO9/vrrdjWIiIjg5hwcHLhz0svKVFxs9yPHwcEB\n06ZNw4YNG5CVlYX69evDz8/P+oZlly5d8PXXX2PLli3o37+/8s5Vvy/+J6Wmplo/FmGMsfbt20tm\nb968KXupBmOM3bp1iwUEBLBNmzap2v+dO3dYp06dmNFotLu9X79+rH///mzAgAGsZcuWTK/Xszt3\n7nDHePbsGXvy5In1d71ez27dumWXmTVrlt3HRH5+fuzevXvc8X7//Xfm4+Mje9ypqaksPT2dMWa5\nz97e3uzp06eiXGFhIUtJSWHBwcFs5syZTK/X2223fUxtH/u9e/eyqVOncnMlFixYYPexKC+3Y8cO\n5uvry37++WfZXImNGzeyCRMm2GVOnz7NfHx82IABA1hgYCBr3ry59eN34Vi///679fbLly+zsLAw\nbu6DDz5gDx48YIxZPuYbPHiw5LGtXbuWZWRkSB7/+++/z/79739bs4mJidxcyaUyYWFhbMGCBSwl\nJYUxJn7OStVB6rktrAMvJ6yD3HliWwPbnFwdhONJ1UGY49WBd2y8GghzUnUQ5qTqUKJkTmjVqhW3\nDsJcydzBOx9sM1LnAm8sYR1sc02aNGGdO3eWPB9sx7P9aFF4PpRk2rRpI3kuCI+NVwfeeFLng23u\nwoULdnVo166daM5t1KiR9d+V1EBubratgVTOtg5K83xJDYQ5Ly8v1rVrV1ENlMYrqQEv17hxY1Ed\npMazrQMv884774hqwMvl5uaKzgXe61nDhg1FdZB73bOtg1TOtg5Kr6EldRDmpOqgNF5JHaTuq7AO\nUuPZ1oGXadCggagOvByvDrw1w7vvvmtXB39/f8l1hW0NpNYfwjlJaZ3Cm5NeNv6zVtr9PE/P7R3n\nFi1a4PDhwwCAU6dOSf6Rnhr37t3DoEGDMH78ePTt21cyt3XrVixbtgwA4OLiAo1GY/3DiRLr1q3D\n2rVrkZmZiUaNGmHGjBnW/0IT2rx5M6ZPnw7A8tGXwWAQZVu2bIl//OMfYIzh9u3bePLkCapWrcod\n7/jx42jbtq3sfa1cubL1QvUqVaqgqKjI+k6yrTNnzqBt27ZYv349Pv74Y9SqVUtyzL/97W/Izc0F\nABw+fBitWrWSPQYl27Ztsz6GcvsdOnQorl+/DsDyX9HCWrz77rvYsWMHMjMzMWfOHLz11luIj4/n\njhUZGYl//etfAICjR4/inXfe4eZatmyJQ4cOAbA83m+99Zbk8R09etR6uQtPlSpVrJ+aVK9eHQ8f\nPuTmDh06hPT0dKxevRoFBQX48MMPuc9ZXh3UPrd5OWEdeBleDYQ5qTrwxuPVgZcT1uHNN9/k3k9h\nDXhj8erAy/HqwJsTGjduLKqDmrmDl9mzZ4/oXODloqKiRHUQ5qpVq4adO3eK6sAbb8SIEXZ10Gq1\nosx7770nOhek7qewDrwcrw683IEDB+zqkJycLJpzO3ToIKqB2rmZl/v+++/t6sDLTJo0SVQDYW7H\njh3Yt2+fqAa88YYPHy46F3i5zp07i+ogdV9t68DLeHh4iGrAy507d050LvBezz788ENRHdS87gH8\n18djx47Z1YGXmTx5sqgOwlydOnWwa9cuUR1440VFRYnqwMt5e3uL6iB1X23rwMt4enqK6sDLlfwR\nuW0deGuGtm3b2tXhgw8+ULWu4I11+PBh0ZzEy8XHx8u+Pr9sSnOpRll7bn8y6e3tjSNHjiA4OBiM\nMaSmpv7hsZYuXYqHDx8iIyMDGRkZAGD9mMVWt27dEBcXh/79+6OoqAgTJ04UZUqjb9++iIuLQ0hI\nCDQaDVJTU0Uf93fu3BnHjx9H3759wRjD5MmTJa/BvnbtGjw8PGT3GRERgYkTJ6Jfv34oLCzEmDFj\nULFiRVGudu3amD9/PpYuXQpXV1ekpKRIjhkTE4NJkyZhzpw5qFevHrp3767i3vMVFxcjJSUF7u7u\nGDlyJADgvffeQ3R0tCj72WefITY2Fs7OznBxcUFycvIf3m9iYiKSkpLg7OyMatWqWa+rFIqJiUFC\nQgI2bNgAnU6H2bNnS4557do12YV/cnIyxowZAycnJzg7O0vus3bt2oiIiICLiwvatGmDjh07Ijk5\nWfScjY+PR3Jysl0d0tLSVD23hedAcXExLl++jJo1a1rr8PTpU9FYo0ePFtVA7fnEy8XGxiI1NdWu\nDvPmzRPlpk+fbleHmjVrcvcprAFvn7w68HIDBw4U1eHx48eiOaF+/fqi8+HZs2eKcwdvfpk4caLo\nXBg8eLAo5+bmJqqDTqdTNV/x9uvu7m53PqSmpiI5Odku06hRI9G54OzszN2nsA68fVatWlVUBzc3\nN1HOwcFBVAehspyTzGazqjmpLOcjoPzNSSXfxGT7evbqq6+K6lBcXKz4ugeIXx9TUlKs37xTUocW\nLVrg0aNHdmMBENWhatWqf2ifqampeOWVV0R10Gq1olyNGjVEdXBxceHu17YOvH2azWZRHapXry7K\nFRQUcM8F4ZrBw8PDrg5jx47F7NmzFdcVvPXH559/zj0XhLlKlSqV6fnwV1PqFvhX0jAmcwEuIYQQ\nQgghBAB1DiSEEEIIIUQVWjgTQgghhBCiAi2cCSGEEEIIUYEWzoQQQgghhKhAC2dCCCGEEEJUoIUz\nIYQQQgghKtDCmRBCCCGEEBVo4UwIIYQQQogK/wtCWbiAhCU0LwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2aaf37175f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs4AAABWCAYAAADfRhArAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XdYVNe6BvB3aAYdxBCjwSC2RPTERMUkaqJiCVERpAuo\nFMHEjiUWEFSUJgZ7xyRHBSMqsaE5toB6YhTLiR571KtGSWInOlgos+4fc5k7s/faBQ9Buff7PQ/P\nE2a/rr1nvtlrVoY982kYYwyEEEIIIYQQWRYv+gAIIYQQQgipCWjhTAghhBBCiAq0cCaEEEIIIUQF\nWjgTQgghhBCiAi2cCSGEEEIIUYEWzoQQQgghhKhAC2dCCCGEEEJUoIUzIYTUAN7e3ti9e7fx9wsX\nLsDFxQVbtmwx3nbp0iW0b98eDx8+fBGHSAgh/+fRwpkQ8lJxcXGBl5cXvL29zX5u3rwp+W9Onz6N\n6OjoKjuGXbt2ITQ09Ln+7b59++Dq6mp22/79++Hl5YXevXsjOjoaOp1O1TZTdnZ2ZtvWrFkjui0z\nMxM+Pj6oW7fucx07IYTUBJe69Db7qU5W1bo3QghRYc2aNXBwcFCdf/fdd7Fo0aK/8IjUuXbtGtLS\n0mDakPX+/fuIjY3F+vXr0bRpU3z55ZdIT09HQkKC7DYhrVZrXCTfv38feXl58PPzw6NHjwAADx8+\nRG5uLnJycqrlvhJCyAtjYfnidv3C9kwIIZVUUFAAf39/jBw5El5eXggMDMSVK1dQUFAAT09PAEBx\ncTGio6Ph7e0NX19fxMfHQ6/XAwA2bNgAT09P9O/fH5GRkbh69apx7IULF+KTTz5BQEAA9u7da7w9\nLy8PgYGB8PHxQXBwMH7++WfusT158gSTJk1CTEyM2e0//vgj3n33XTRt2hQAEBISgtzcXDDGZLcJ\nmb67nJ2djX79+qFRo0bG27777ju4urqiRYsWz/HIEkJIzaGxsTb7qU60cCaEvHTCw8PNLtMYNWqU\ncdu5c+cQGRmJ3Nxc+Pn5YdKkSWb/du/evSguLsa2bduM777euHEDhw8fxldffYW1a9di+/bt8PT0\nxKhRo8AYw759+7Bnzx5s3boV2dnZxsXotWvXMH/+fGRkZGDr1q1ITEzEmDFj8PjxY9ExT58+HUFB\nQXBxcTG7/Y8//sAbb7xh/P2NN96ATqdDcXGx7DahioVzaWkpsrOzERoaijp16kCn00Gv12PdunUI\nCwvDo0ePEBAQgPbt2+OXX355jkefEEJebhpra7Of6kSXahBCXjpyl2q0atUK77//PgDA398fs2bN\nQlFRkXF7hw4dMH/+fISGhuKjjz5CeHg4mjRpgg0bNsDDw8M4rp+fH5KTk3Hz5k0cPnwY7u7u0Gq1\nxnEzMzNx6NAh3L59GxEREcbxNRoNfv31V7Rq1cp427p162BlZYWAgADRtdgV73YLWVhYyG4TsrOz\nw4MHD7B79260atUKzZo1w4ULF/Do0SPs378flpaW6NatG8rKypCRkYE5c+ZwxyaEkJquuhfLpmjh\nTAipUSwtza9tY4yZ3da4cWPs3bsXBQUFOHLkCIYMGYL4+Hju5Q+MMZSVlUGj0ZhtrxhPr9ejc+fO\nWLBggXHb77//jgYNGpiNs2XLFjx9+hTe3t4oLS01/ndGRgYcHR1x6tQpY/bWrVuwt7dH7dq1ZbcJ\nabVa3LhxA2vXrjV+ELLiuuesrCyEhoZCo9HA2tq6UteHE0JITVPdl2eYoks1CCE1yoULF3DhwgUA\nhmuWXV1dYWdnZ9z+7bffIjY2Fl26dMGkSZPQpUsXXLp0CV26dMH333+P+/fvAzBcE1yvXj00adIE\nXbt2xa5du/Dw4UPo9Xps27YNANCpUyccOnQIV65cAQAcOHAA/fv3x7Nnz8yOKScnBzt27MC2bduQ\nkZGBV155Bdu2bUPDhg3RpUsXnDp1CteuXQNguD65V69eACC7TcjOzg7Hjh1DcXExunTpAsCwcD5/\n/jxOnz4NX1/fKnh0CSHk5WdhY2P2U53oHWdCyEsnPDxcdLnChAkT8Morr6B+/fpYsGABCgsL4eDg\ngDlz5qCwsNCY8/HxwdGjR+Hh4QFbW1s0atQIYWFhsLe3R0REBMLDw6HX6+Hg4ICVK1fCwsICbm5u\nuHjxIvz9/VG3bl20atUKDx48wNtvv41Zs2ZhwoQJYIzBysoKy5cv574jLOW1115DamoqoqOjUVpa\nCmdnZ6SlpSluE7Kzs8Pt27fNrvfWarW4c+cOwsPDUadOnco8xIQQUmNV5lKN0tJSTJ06FYWFhSgp\nKcGIESPM3qDIy8vD0qVLYWVlBX9/fwwYMEB+34z390tCCHkJFRQUIDExETt27HjRh/LSi4mJQWRk\nJFq2bPmiD4UQQqrUjc/Mv7e/8SrpryP97rvvcOHCBcTFxaGoqAg+Pj7Yv38/AMOi2sPDAzk5ObC1\ntUVISAhWrlyJ+vXrS45H7zgTQsj/MZ999hnOnz+Pq1evIigoCH5+fi/6kAghpMpU5h3nPn36oHdv\nQ5MU4Wdirly5AmdnZ9jb2wMwfLj82LFj6Nu3r+R4tHAmhNQYHTt2pHebVVi1atWLPgRCCPnLVGbh\nXHEZm06nQ3R0NMaNG2fcptPpzD4jU/EVn3Low4GEEEIIIaTGqGwDlN9//x1hYWHw9vaGl5eX8Xat\nVmv2vfnFxcVmC2keWjgTQgghhJAaQ2NjY/Yj5+7du4iMjMSkSZMQEBBgtq1Fixa4fv06ioqKUFJS\nguPHj6N9+/ay49GlGoQQQgghpMbQWKlfvq5YsQIPHz7EsmXLsGzZMgBAYGAgnjx5gqCgIMTExCAq\nKgqMMfj7+6Nhw4by+6Zv1SCEEEIIITXFreR0s98bxk2stn3/5e84T1i7TXb7vDBvAMCS3T/K5kb3\n7oL5Ow8o7m98Pzcs/Mc/FXNj+3bF4MVZspmsMYMBADM27ZLNzQzsAwAYuiJbNvfV8GCMX7NV8djm\nh/tg7o79spkvPLsDAGLX75TNpYb0U51TqgFgqIPS4zu2b1cAQPTfN8vmFg0xfNJ/7OotsrmFEb7w\n+fJrxWPbOikKn63cIJtZNSwIgPrnpZqc2sdN6dgqjk/tfVjw/UHZ3DiPbgCASVnbZXNfDu6PpXsO\nKR7bqE8/ht/cv8tmNn8xBABUn1u+6d/I5rZMjETkcvnzCgC+GRGM8GXfymbWjBwIAJj13R7Z3HT/\nTwEAcdnfy+aSgz1Un89hS9fJZtaOGgQAivf1mxHBANQ9vpPX5Soe25xBXkjcvFcxN83PHcEL1spm\nsseFAQAC5sk/R3ImGJ4jap6XQ5atVzy2v48MUT1Hq61D/IZ/yOaSgvqqftzUzuVKr2/j+7kBAJbv\n/Uk2N8L9I3yRKf/YAsDc0P6qXz+U5t+tk6IAAHNy82Vzk716KM73gGHOV6oVYKiX2vN5yrfyHypO\nG+gJAAhasEY2t2FcuOJzFzA8fzN+OCKb+bxXJwBAusJzZOL/PEeUzoe/jwxR/Vqp9jx9WWmsq7fp\niSm6VIMQQgghhNQYL7LltuqFs16vF3XyIoQQQgghpDpprF/c+76ye75x4wZSU1Nx5swZWFlZQa/X\no2XLloiNjUWzZs2q6xgJIYQQQggBULnvca5qsgvnuLg4fPHFF2jbtq3xtpMnTyI2NhbZ2crXHRJC\nCCGEEFKVlL6CjufUqVNIT09HZmam2e2rV6/Gpk2b4ODgAACYOXMmmjdvLjmO7MK5pKTEbNEMAO3a\ntav0wRJCCCGEEFIVKnuN86pVq7B9+3bY2tqKtp05cwZpaWlo06aNqrFkF84uLi6IjY1F165dYWdn\nh+LiYhw4cAAuLi6VOmBCCCGEEEKqQmW/VcPZ2RmLFy/G5MmTRdvOnj2LjIwM3LlzB927d8ewYcNk\nx5JdOCckJGDfvn04ceIEdDodtFotevToAXd390odMCGEEEIIIVVBY21ZqXzv3r1x8+ZN7rZ+/fph\n4MCB0Gq1GD16NPLz89GjRw/JsWQXzhqNBu7u7rRQJoQQQgghL4XnucaZhzGG8PBw2NnZAQDc3Nxw\n7tw52YUzdQ4khBBCCCE1xqO95s127NylF7oVbt68iQkTJmDjxo3/O86jR/D09MT333+P2rVrY+zY\nsfD394ebm5vkOH/5F+G5Jy2X3b43fgQAdd281HZqUurCAxg68UzbKN8dKnFAXwBQ7A42P9wHADDi\nqxzZ3PKhAYodvwBD1y+1XfdCFmbK5taPDQUA9JudIZvbGfO5Ypc8wNApT23XPbWPm5rOWot3KXfn\nG9Oni2IHvFGffmwYU6Gz1tzQ/gCAJIXnXLyfu+ruZkpdtQBDZy21j6/a+zr6G/nn0pJIPyTk7FY8\ntoSA3qo6qgFQ3f1QTddItR1D1Z4LajrWAVB8TBICequek5Ry0/wMf9XzTFslm9sx5TMAQNr2PNnc\nlP49VXeDVNv5Uu3jprYDnpouaErdGwFDB0el+WFMny4AgOGrNsnmVnwWCAAYtEj+ubQuOlT1XK72\n8VB7Pqupg9rufJ9nbJTNZHw+AACw5uBx2Vx4t/cBqOtmqjTfA4Y5X2lNABjWBWq77qntLKqmS6JS\n1z3A0HlPTUdkQH2nVTUdMpUyFblv9h+VzUR2/1BxnBfpP33HOTc3F48fP0ZQUBDGjx+PsLAw2NjY\noHPnzrKLZoA6BxJCCCGEkBpEY1X55auTk5Px3WYvLy/j7T4+PvDx8VE9Di2cCSGEEEJIjVEjWm4T\nQgghhBDyolXVhwOfBy2cCSGEEEJIjfHSvuMcGhqK0tJSs9sYY9BoNNRymxBCCCGEVLvnuca5qsju\neeLEiYiPj8fSpUthaVm5L5smhBBCCCGkqj3PpRqnTp1Ceno6MjPNvzUnLy8PS5cuhZWVFfz9/TFg\nwADZcWQXzm3btoW3tzcuXrxITVAIIYQQQsgLp7Gu3KUaq1atwvbt22Fra2t2e2lpKVJTU5GTkwNb\nW1uEhISgZ8+eqF+/vuRYFko7Gzp0KC2aCSGEEELIS+GpXR2zHyXOzs5YvHix6PYrV67A2dkZ9vb2\nsLGxQYcOHXDs2DHZsahzICGEEEIIqTEePXpk9ntFy2w5vM6Bx48fR1ZWFhYsWAAAWLhwIRo1aoTA\nwEDJcf7yq6szfjgiu/3zXp0AAKlbf5DNxfr0wsiv5TvzAcCyqADFrmWAoXOZ2u5majpcAVDVJSh4\nwVrFY8seF6aqexEAxfGyx4UBUNfNS+3j5pv+jWxmy8RIAOo6AgLqOmEpdfwCDF2/1HYGU9N5DYCq\n+7p870+KxzbC/SNELlf+QO03I4JVdxBT28VPTfdDtfehKrubAUDg/NWyuU3jIxTvJ2C4r2q6agFQ\n3TFUTRe/2PU7FY8tNaSf6g5tanNqOl+qfdyUOi4Chq6Las9ntfdB6Tk3wv0jxfMUMJyrK/cdls0M\n+6QzACjWKzWkHwAodobLmTBEdadVtcemtlusmnNQ7VxTlZ0UASju95sRwaprqrYrp9q5fOgK+WP7\nangwAHWPr9LcABjmB7UdLdU+vmo6LCu9ZgGG1y01XR7/P9BqtSguLjb+XlxcrLgIV7xUgxBCCCGE\nkJdFiYW12c/zatGiBa5fv46ioiKUlJTg+PHjaN++vey/qfQ7ziUlJbB5gV88TQghhBBC/v8qKSv7\nj/59bm4uHj9+jKCgIMTExCAqKgqMMfj7+6Nhw4ay/1Zy4ZyXl4fExERYWVlh/Pjx8PDwAGD4sODa\ntcqXGxBCCCGEEFLVSsvLK/1vnJycjNc3e3l5GW/v2bMnevbsqXocyYXzihUrsHXrVuj1eowdOxbP\nnj2Dr68v6LOEhBBCCCHkRSkt07+wfUsunK2trWFvbw8AWLZsGcLDw+Ho6AiNRlNtB0cIIYQQQoip\n0vL/7FKN/4TkhwPffPNNpKam4vHjx9BqtViyZAlmzZqF//qv/6rO4yOEEEIIIcSopKzc7Kc6SS6c\nU1JS4OLiYnyH2dHREWvXrkXfvn2r7eAIIYQQQggx9SIXztQAhRBCCCGE1BhHLv9q9nunt5wls3q9\nHgkJCbh48SJsbGyQlJSEJk2aGLevXr0amzZtgoODAwBg5syZaN68ueR4f3kDFEIIIYQQQqpKZd5l\n3rdvH0pKSrBhwwacPHkSs2fPxvLly43bz5w5g7S0NLRp00bVeC9N50A1nbDUdo9T29VHbQcxtfch\nbOk62dzaUYNUd/PacPikbCaoczsAwOoD8j3VI9w+AKCuY5ZSZyXA0F0pfNm3spk1IwcCUN/VUE33\nOLVdpNR0IwOAr/OPyuaienwIQF23KaWuT4Ch85Pa56+a7lsAkK7QyW1iJTq5KXUtAwydy9R2jeyT\nslI2t2vqMADquqWpfV6qfb6p6QoHqOtmmpCzW/HYEgJ6Y+zqLbKZhRG+AICULftkc1N9PwGgruue\n2o6ASl1FAUNnUbVd99R0qgSgqnPZ5HW5isc2Z5CX4n1dP9bQeU1t90OlroDzwrzRb3aG4rHtjPlc\ndYdatfdBzeuR2q6Gal8D1XayVarXnEFeiq+TQOVeK9U+vmq786m5D0o1AAx1UDtHq63DlG93yObS\nBnoqziGAYR7pnrBENrM/YbTiOC9SZRbOJ06cQNeuXQEA7dq1w5kzZ8y2nz17FhkZGbhz5w66d++O\nYcOGyY5H7zgTQgghhJAaozLfqqHT6aDVao2/W1paoqysDFZWhiVwv379MHDgQGi1WowePRr5+fno\n0aOH5HiVarn99OlTlJSUVOafEEIIIYQQUmVKy/RmP3K0Wi2Ki4uNv+v1euOimTGG8PBwODg4wMbG\nBm5ubjh37pzseLIL58uXL2PkyJGIjY3FTz/9BA8PD3h4eCA/P1/tfSOEEEIIIaTKlJaXm/3IcXV1\nxcGDhkvDTp48iZYtWxq36XQ6eHp6ori4GIwxFBQUKF7rLHupxowZMzB27FgUFhYiOjoau3fvRq1a\ntTB06FDZt7EJIYQQQgj5K5SUqb9Uw93dHYcOHUJwcDAYY0hJSUFubi4eP36MoKAgjB8/HmFhYbCx\nsUHnzp3h5uYmO57swlmv1+PDDw0fkiooKMBrr71m+EdWdGk0IYQQQgipfpX5cKCFhQVmzZpldluL\nFi2M/+3j4wMfHx/148ltbNasGeLi4qDX6zF79mwAQEZGBurXr696B4QQQgghhFSV0rJys5/qJPvW\ncVJSEvLy8mBh8b/r64YNGyI0NPQvPzBCCCGEEEKElK5r/itR50BCCCGEEFJjLPzHP81+H9u3a7Xt\nmy5WJoQQQgghNcaLfMeZFs6EEEIIIaTGKClV/60aVa1SDVAIIYQQQgh5kcr0erMfOXq9HtOnT0dQ\nUBBCQ0Nx/fp1s+15eXnw9/dHUFAQNm7cqLhveseZEEIIIYTUGJX5Orp9+/ahpKQEGzZswMmTJzF7\n9mwsX74cAFBaWorU1FTk5OTA1tYWISEh6Nmzp+y3x9E7zoQQQgghpMYoKSsz+5Fz4sQJdO1q+PBg\nu3btcObMGeO2K1euwNnZGfb29rCxsUGHDh1w7Ngx2fHoHWdCCCGEEFJjVOa7m3U6HbRarfF3S0tL\nlJWVwcrKCjqdDnZ2dsZtderUgU6nkx2PFs6EEEIIIaTGKC2Xv67ZlFarRXFxsfF3vV5v7IAt3FZc\nXGy2kOaptoWzXq9HQkICLl68CBsbGyQlJaFJkybc7KlTp5Ceno7MzEzu9tLSUkydOhWFhYUoKSnB\niBEj0KtXL1GuvLwc8fHxuHr1KjQaDWbOnImWLVtyx7x37x78/PzwzTffmLViFPL19TX+n4uTkxNS\nU1NFmZUrVyIvLw+lpaUICQlBYGCgKLN582Zs2bIFAPDs2TOcP38ehw4dQt26dUX3NSYmBoWFhbCw\nsEBiYiL3+EpKShAbG4sbN25Aq9Vi+vTpaNq0qXG76WN6/fp1xMTEQKPR4O2338aMGTOMTW54j31K\nSgqaNWuGkJAQ7njnz59HYmIiLC0tYWNjg7S0NOP1Qaa5y5cvY9q0aWCMoWnTpkhKSoKVlRV3n7m5\nucjKysKGDRu4+zx37hyGDRtmvI8hISHw8PAQ5e7du4f4+Hg8fPgQ5eXlmDNnDpydnUW58ePH4+7d\nuwCAwsJCtG3bFvPnzxfdzxkzZsDS0hJNmzZFcnIy93E7e/YsZsyYARsbG7Ru3RpxcXEoLy8XPWff\neustUR14uYrntmkdeOdAo0aNzOqQnJyMuXPnmmWaNGkiqgFjTHKfpnXg7dPR0VFUB3d3d1GuXbt2\nZnVISUnB8uXLzTI7duwQ1WDOnDnc+ymsA+9xe+ONN0R1YIyJ5oRatWqJ6sDLVcwdFXUYMGCAKFNe\nXi46F1599VVRzsLCQlQHjUYjuU/TOvDmtbKyMrM6BAUF4dChQ2aZ1157TXQuvPnmm6Kxli9fLqpD\neno6974K68B73EpLS0V1sLCwMJtzraysJOck3tzMm5NMcyUlJdw5yTTDGOPOR1L75M1Jprlnz55J\nzkmmuXr16knOSaa5JUuWcOck4f2UmpNMc0+fPuXWQPh6Nnz4cG4dpF73hHUQ5sLCwkR1+Oyzz8wy\nUVFR3DpI7VNYB2EuNDSUWwdhbuLEidw6CHNPnz4V1eHatWui+8mrg3CswYMHc+sgXDN8+OGHojqs\nWrWKu64Q1kA4Vps2bbjngjDXvn17yfPhZaR0eYYpV1dX5Ofnw8PDAydPnjRbB7Zo0QLXr19HUVER\nateujePHjyMqKkp+QFZNdu/ezaZMmcIYY+znn39mw4cP5+YyMjKYp6cnCwwMlBwrJyeHJSUlMcYY\ne/DgAXNzc+Pm9u7dy2JiYhhjjB05ckRynyUlJWzkyJHs008/ZZcvX5bc79OnT5m3t7fk9or9DBs2\njJWXlzOdTscWLVokm2eMsYSEBJadnS15H6KjoxljjP34449s9OjR3FxmZiaLj49njDF25coVFhkZ\nadwmfEyHDRvGjhw5whhjbNq0aWzPnj3c3L1791hUVBTr1asX+/bbbyXHGzRoEDt37hxjjLH169ez\nlJQUbm7EiBHs6NGjjDHGpkyZwvbs2cOt99mzZ1lYWJjZbcLcxo0b2ddffy16HIS5KVOmsJ07dzLG\nGDt8+DDLz8/n5ioUFRWx/v37s1u3bokyI0eOZPv372eMMTZhwgT2ww8/cMfy9fVlJ06cYIwxNm/e\nPLZ161buc5ZXB16OVwdeTliHIUOGiDK8GkidT8I68HK8OvBywjokJSVJnsOmNeCNxasDL8erA29O\n4NWBlxPWgZfhnQu8HK8OUvOVsA68nLAOvAzvXJCbI03rwMvx6sDL8eognHOl5iRhTmpOEuZ4dRBm\neDXgjcWrAS8nNScJc1JzktTrkGkdhBmpOUmY49WA93rGqwMvx6sDLyesw6xZs0QZXh2kXmuFdeDl\neHXg5Xh1kHuNr6jDr7/+Ksrw6sAbi1cH3ppBWIclS5aIMrwa8MbinQu8nNT58LKKWPqt2Y+c8vJy\nNm3aNBYUFMQGDBjALl++zLZv325cd/3www/Mz8+P+fr6sqysLMV9V9uHA+Uuzjbl7OyMxYsXy47V\np08fjB07FgDAGIOlpSU398knnyAxMREA8Ntvv4neza2QlpaG4OBgNGjQQHa/Fy5cwJMnTxAZGYmw\nsDCcPHlSlPnxxx/RsmVLjBo1CsOHD0f37t1lxzx9+jQuX76MoKAg7vZmzZqhvLwcer0eOp1O8v8A\nL1++jG7dugEAmjdvjitXrhi3CR/Ts2fP4sMPPwQAdOvWDT/99BM3V1xcjDFjxsDb29tsX8LcvHnz\n0Lp1awCGd/lr1arFzS1evBgffPABSkpKcOfOHWi1WlHmwYMHmDdvHqZOnSq7zzNnzmD//v0YNGgQ\npk6darwmSZj717/+hVu3biEiIgK5ubnG+y31PFu8eDEGDx6MBg0aiDKtW7dGUVERGGMoLi421kKY\nu3XrFlxdXQEY/k/3xIkT3Ocsrw68HK8OvJywDi4uLqIMrwa8sXh14OV4deDlhHUYNmyY5DlsWgPe\nWLw68HK8OvDmBF4deDlhHXgZ3rnAy/HqwMvx6sDLCevQqVMnUYZ3LsjNkaZ14OV4deDleHUQzrlS\nc5IwJzUnCXO8OggzvBrwxpKak4Q5qTlJmJOak6Reh0zrIMxIzUnCHK8GvNczXh14OV4deDlhHXQ6\nnSjDqwNvLF4deDleHXg5Xh3kXuMr6nD//n1RhlcH3li8OvDWDMI67Nu3T5Th1YA3Fu9c4OWkzoeX\nVWU+HGhhYYFZs2YhOzsbGzZsQIsWLeDl5WVcd/Xs2RPfffcdNm/ejEGDBinuu9oWzlIXZwv17t1b\n8c8DderUgVarhU6nQ3R0NMaNGyeZtbKywpQpU5CYmAgvLy/R9s2bN8PBwcG4qJfzyiuvICoqCl9/\n/TVmzpyJiRMniu7DgwcPcObMGSxcuNCYYTJdzVeuXIlRo0ZJbq9duzYKCwvRt29fTJs2DaGhodxc\n69atkZ+fD8aY8QQt/5/OOsLHlDEGjUYDwPBYPnr0iJtr3Lgx2rZtK9qXMGf6gpCVlYWIiAhuztLS\nEoWFhfD09MSDBw/QqlUrs0x5eTni4uIQGxuLOnXqyO7zvffew+TJk7Fu3To0btwYS5cu5eYKCwtR\nt25drF69Go6Ojli1ahU3Bxj+tHn48GH4+flxMxV/guvbty/u3buHjh07Sj5uR48eBQDk5+fjyZMn\n3Ocsrw68HK8OvJywDhV/EjXN8GogHGvs2LHcOvD2yasDLyesw7p167jnsLAGvLF4dZB63IR1AMRz\ngtT5IMzx6iDMSJ0LwhyvDsKcp6en5PkgHI9XB2FG6lzgzZHCOvByUucD73EzrcPFixdFcy6vBry5\nmVcDXk5Yh9dff12U4dVAOJZer+fWgLdPXg14OV4dpF6HTOvAy/BqIPW4Cc8F3usZrw68nKOjo6gO\nvJyDg4NZHfz9/bn7FNZBONaECRMQExMjqgNvn++8846oDrzc9evXRXWQeo03rQMv4+TkJKoDL8er\ng9SawbQOOp1OlHFychLVgDfW66+/blaDiIgIbs7CwoI7J72sSsrLzX6qVWXeGv9PpKSkGP8swhhj\nXbt2lcycwuoiAAAHeklEQVTeuHFD9lINxhj77bffmK+vL9u0aZOq/d++fZt1796dFRcXm90+cOBA\nNmjQIDZ48GDWoUMH5u/vz27fvs0d49mzZ+zJkyfG3/39/dlvv/1mlvnyyy/N/kzk5eXF7t69yx3v\nzz//ZB4eHrLHnZKSwtLT0xljhvvs7u7Onj59KsqVlpay5ORkFhwczObMmcP8/f3Ntps+pqaP/d69\ne9nMmTO5uQqLFi0y+7MoL7dz507m6enJfv31V9lchY0bN7LJkyebZU6dOsU8PDzY4MGDWWBgIGvf\nvr3xz+/Csf7880/j7ZcuXWJhYWHc3EcffcTu37/PGDP8mW/o0KGSx5aVlcWWLVsmefydOnViv/zy\nizGbkJDAzVVcKhMWFsYWLVrEkpOTGWPi56xUHaSe28I68HLCOsidJ6Y1MM3J1UE4nlQdhDleHXjH\nxquBMCdVB2FOqg4VKuaE999/n1sHYa5i7uCdD6YZqXOBN5awDqa5d999l/Xo0UPyfDAd748//jDe\nJjwfKjIdO3aUPBeEx8arA288qfPBNHf+/HmzOnTp0kU057Zu3dr47ypqIDc3m9ZAKmdaB6V5vqIG\nwpyLiwvr1auXqAZK41XUgJdr06aNqA5S45nWgZd55513RDXg5QoKCkTnAu/1rFWrVqI6yL3umdZB\nKmdaB6XX0Io6CHNSdVAar6IOUvdVWAep8UzrwMu0bNlSVAdejlcH3prhvffeM6uDj4+P5LrCtAZS\n6w/hnKS0TuHNSS8bny+/NvupTtX2jrOrqysOHjwIAKKLsyvr7t27iIyMxKRJkxAQECCZ27p1K1au\nXAkAsLW1hUajMX5wosK6deuQlZWFzMxMtG7dGmlpacb/QxPKycnB7NmzARj+9KXT6UTZDh064J//\n/CcYY7h16xaePHmCevXqccc7duwYOnfuLHtf69ata/yEp729PcrKyozvJJs6ffo0OnfujPXr16NP\nnz5o3Lix5Jh/+9vfUFBQAAA4ePAg3n//fdljULJt2zbjYyi33+HDh+PatWsADP8XLazFe++9h507\ndyIzMxPz5s3DW2+9hbi4OO5YUVFR+Pe//w0AOHz4MN555x1urkOHDjhw4AAAw+P91ltvSR7f4cOH\njZe78Njb2xv/atKgQQM8fPiQmztw4ADS09OxZs0aFBUV4eOPP+Y+Z3l1UPvc5uWEdeBleDUQ5qTq\nwBuPVwdeTliHN998k3s/hTXgjcWrAy/HqwNvTmjTpo2oDmrmDl5mz549onOBlxs1apSoDsJc/fr1\n8f3334vqwBtv9OjRZnWwsbERZT744APRuSB1P4V14OV4deDl8vPzzeqQlJQkmnO7desmqoHauZmX\n++mnn8zqwMtMmzZNVANhbufOndi3b5+oBrzxRo4cKToXeLkePXqI6iB1X03rwMs4OTmJasDLnT17\nVnQu8F7PPv74Y1Ed1LzuAfzXx6NHj5rVgZeZPn26qA7CXNOmTbFr1y5RHXjjjRo1SlQHXs7d3V1U\nB6n7aloHXsbZ2VlUB16u4kPkpnXgrRk6d+5sVoePPvpI1bqCN9bBgwdFcxIvFxcXJ/v6/LKpzKUa\nVa3aPjLp7u6OQ4cOITg4GIwxpKSkPPdYK1aswMOHD7Fs2TIsW7YMAIx/ZjH16aefIjY2FoMGDUJZ\nWRmmTp0qylRGQEAAYmNjERISAo1Gg5SUFNGf+3v06IFjx44hICAAjDFMnz5d8hrsq1evwsnJSXaf\nERERmDp1KgYOHIjS0lKMHz8etWvXFuWaNGmChQsXYsWKFbCzs0NycrLkmFOmTMG0adMwb948NG/e\nHL1791Zx7/nKy8uRnJwMR0dHjBkzBgDwwQcfIDo6WpT9/PPPERMTA2tra9ja2iIpKem595uQkIDE\nxERYW1ujfv36xusqhaZMmYL4+HhkZ2dDq9Vi7ty5kmNevXpVduGflJSE8ePHw8rKCtbW1pL7bNKk\nCSIiImBra4uOHTvCzc0NSUlJoudsXFwckpKSzOqQmpqq6rktPAfKy8tx6dIlNGrUyFiHp0+fisYa\nN26cqAZqzydeLiYmBikpKWZ1WLBggSg3e/Zsszo0atSIu09hDXj75NWBlxsyZIioDo8fPxbNCS1a\ntBCdD8+ePVOcO3jzy9SpU0XnwtChQ0U5BwcHUR20Wq2q+Yq3X0dHR7PzISUlBUlJSWaZ1q1bi84F\na2tr7j6FdeDts169eqI6ODg4iHIWFhaiOghV5Zyk1+tVzUlVOR8BNW9OqvgmJtPXs1dffVVUh/Ly\ncsXXPUD8+picnGz85p2KOri6uuLRo0dmYwEQ1aFevXrPtc+UlBTUqlVLVAcbGxtRrmHDhqI62Nra\ncvdrWgfePvV6vagODRo0EOWKioq454JwzeDk5GRWhwkTJmDu3LmK6wre+uOLL77gngvCXJ06dar0\nfPirVaZzYFXTMCZzAS4hhBBCCCEEALXcJoQQQgghRBVaOBNCCCGEEKICLZwJIYQQQghRgRbOhBBC\nCCGEqEALZ0IIIYQQQlSghTMhhBBCCCEq0MKZEEIIIYQQFWjhTAghhBBCiAr/DbtCsXtmOch9AAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2aaf31d5438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs4AAABWCAYAAADfRhArAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlcE2f+B/BPwqFoEEs9ikW8WtHV1qJtPariUaoiVgQV\nUTkEPFEUq3J4oVxKvQ9UbLsq0op1W5Xqei2o61E8dnW9q1atpVtvqsEjIXl+f2TJJjPPTIYWUfb3\nfb9eeb2E+fDMZL6ZyWMY5qtijDEQQgghhBBCZKlf9AYQQgghhBBSFdDEmRBCCCGEEAVo4kwIIYQQ\nQogCNHEmhBBCCCFEAZo4E0IIIYQQogBNnAkhhBBCCFGAJs6EEEIIIYQoQBNnQgipAvr374/du3eb\nv7548SI8PT3x7bffmr93+fJleHl54eHDhy9iEwkh5H+e/YveAEIIseTp6YnmzZtDrbb+f/3KlSvh\n7u7O/ZkzZ85g7dq1WLZsWYVsw65du5CTk4Ps7GzFP3Pp0iWkpKTg0aNHUKvVmDt3Llq3bg0A2L9/\nPxYuXAidTgdPT0+kpaVBo9HYXGbJ2dkZWq3W/PX69etF38vOzoa/vz9q1ar1e586IYS89C537mX1\n9ZuHdkskKx594kwIeemsX78e27Zts3pITZoB4K233qqwSfPv8eTJE0RGRiIqKgpbt27FuHHjMGXK\nFADA/fv3kZCQgOXLl2P37t1o2LAhFixYYHOZkEajMU+S79+/j/z8fAQEBODRo0cAgIcPHyIvLw/D\nhw+vhGdMCCEvkNrO+lGJ6BNnQkiVUVhYiIyMDNSvXx83b95E9erVMW/ePNy9exfJycn47rvvUFJS\ngoSEBNy4cQNqtRqtWrXC3LlzoVarkZubi+zsbKjVatSpUwczZ85EkyZNAABLly5FXl4eateujUaN\nGpnXmZ+fj1WrVkGv16N69eqIi4uDl5eX1XYdPnwYDRs2hLe3NwCgZ8+e5on+oUOH8NZbb6Fx48YA\ngODgYPTv3x+zZ8+WXaZSqazWYfnp8qZNm9C3b180aNAAt2/fBgD85S9/Qdu2bdGsWbOK3emEEPKS\nUTk6vLB10yfOhJCXTlhYGPr3729+REdHm5edP38eERERyMvLQ0BAAKZOnWr1s3v37kVJSQm2bduG\nLVu2AABu3ryJo0eP4rPPPsOGDRuwfft2+Pn5ITo6Gowx7Nu3D3v27MHWrVuxadMm8wT1+vXrWLx4\nMbKysrB161YkJydjwoQJePz4sdU6r127hrp16yIxMREBAQEYMWIEDAYDAODXX3/Fa6+9Zs6+9tpr\n0Gq1KCkpkV0mVDZx1uv12LRpE0JCQlCzZk1otVoYjUbk5OQgNDQUjx49wsCBA+Hl5YUffvjhD1aC\nEEJePioHB6tHZaJPnAkhL53169fD1dWVu6xFixZ49913AQCBgYGYO3cuiouLzcvbtWuHxYsXIyQk\nBJ06dUJYWBgaNWqE3Nxc+Pr6mscNCAhAamoqfv75Zxw9ehQ+Pj7ma4sDAwORnZ2Nw4cP4/bt2wgP\nDzePr1Kp8NNPP6FFixbm75WWluLAgQPYsGED2rRpg3379mHUqFEoKCiA0WjkPg+1Wi27TMjZ2RkP\nHjzA7t270aJFCzRp0gQXL17Eo0ePsH//ftjZ2aFr164oLS1FVlYWMjIyZPYwIYRUXZU9WbZEnzgT\nQqoUOzvr69kYY1bfa9iwIfbu3YtRo0ZBq9VixIgR2LVrFxhjorEYYygtLYVKpbJaXjae0WhEx44d\nra613rx5M958802rcerVq4emTZuiTZs2AIAPP/wQBoMBN2/ehJubG+7cuWPO3rp1Cy4uLqhRo4bs\nMqGya5w3bNiA0NBQq+9t3LgRISEhUKlUcHBwkPxPByGE/C9QOTpYPSoTTZwJIVXKxYsXcfHiRQBA\nbm4u2rZtC2dnZ/PyL7/8EgkJCejcuTOmTp2Kzp074/Lly+jcuTN27tyJ+/fvAzBdE1x2PXOXLl2w\na9cuPHz4EEajEdu2bQMAdOjQAYcPH8bVq1cBAAcOHMDHH3+MZ8+eWW1T165dUVRUhLNnzwIAjh8/\nDpVKBXd3d3Tu3BmnT5/G9evXAZiuT+7ZsycAyC4TcnZ2xvHjx1FSUoLOnTsDME2cL1y4gDNnzmDA\ngAF/dNcSQkiVoHZ0tHpUJrpUgxDy0gkLCxNdrjB58mRUr14dderUwZIlS1BUVARXV1dkZGSgqKjI\nnPP398exY8fg6+sLJycnNGjQAKGhoXBxcUF4eDjCwsJgNBrh6uqKNWvWQK1Ww9vbG5cuXUJgYCBq\n1aqFFi1a4MGDB3jzzTcxd+5cTJ48GYwx2NvbY9WqVaJPhOvWrYuVK1dizpw5ePLkCRwdHbF8+XJU\nq1YN1apVQ3p6OmJiYqDX6+Hh4YH58+cDAF599VXJZULOzs64ffu21fXeGo0Gd+7cQVhYGGrWrFlR\nu58QQl5qL/JSDRXj/f6SEEJeQoWFhea7ZxB58fHxiIiIQPPmzV/0phBCSIW6OTLG6uuGa6VvR6rX\n65GYmIiioiLodDqMHTvW6jd7+fn5WLlyJezt7REYGIjBgwfLrps+cSaEkP8xI0eOxIULF3Dt2jUE\nBQUhICDgRW8SIYRUmPJ84rx9+3bUrl0bn376KYqLi+Hv72+eOOv1eqSnp2PLli1wcnJCcHAwevTo\ngTp16kiORxNnQkiV0b59e/q0WYG1a9e+6E0ghJDnpjwT5969e6NXL1OnQeEfk1+9ehUeHh5wcXEB\nYLor0/Hjx9GnTx/J8WjiTAghhBBCqozy3Emj7O8/tFotYmJiMGnSJPMyrVZr9cflZffGl0N31SCE\nEEIIIVWGytHR6mHLv//9b4SGhqJ///7o16+f+fsajcaq4VRJSYnVRJqHJs6EEEIIIaTKUNnbWz3k\n3L17FxEREZg6dSoGDhxotaxZs2a4ceMGiouLodPpcOLECXh5ecmOR5dqEEIIIYSQKqM8l2qsXr0a\nDx8+RGZmJjIzMwEAgwYNwpMnTxAUFIT4+HhERkaCMYbAwEDUr19fft3P+3Z0cV/K/yHP/KF+AIA1\n+47K5kZ/2BHLdx2yub4JvTtj5Z7DNnPRH32AqNWbZDOfjRkCAEj+Zq9sbmaADwBg3OdbZHOZkQMx\nLSfP5rZlDOtn87lO6G1qgDD7612yuTmDeivO2aoBYKqDrf0b/dEHAIBPsrfL5haGfAwAmLpRPvfp\n8I8RtGS9zW3LnRSG8V98I5tZEWG6u4DS16WSnNL9Zuv1AZheI0qfw4rd8q+R8b1Mr5Hpm3bK5lKH\n+OKz/EKb2xbVoz2Cl2bLZr6aGGLKKjy2hizZIJvbNCkUYz+zvd9WRQ3EqKzNspmsUaZbDKV9u082\nlzjgQwBA0pbdsrmkgb0UH88j1+TKZtaODgIAm891VZTp0xIl+3dG7l9tbltKUB/M355vMxf3cQ+E\nZX4pm1k/bigAYNgy+ddITozpNaLkdTlm7dc2t231yEFIsXGOnvGfc7TS1+Xcv+yRzc0K/Ejxflv6\n17/LZib26QJA+fFs61iN6tEeCV/tsLlt6cF9Fb9/2Dr/5k4KAwAs3nFANhfb17vCjhnAdNykb/2b\nbCbB33T3hJmb5Y+H5MGmPwYLXZkjm9sQPczmaxcwvX6/2H9MNhPR7X0AwJKdB2Vzk3y7AoDN42H1\nyEE2z6mA6byq9Dh9Wd3OsL79XL1pMRLJikefOBNCCCGEkCqjsttsW1I8cTYajaJOXoQQQgghhFQm\nlcOL+9xXds03b95Eeno6zp49C3t7exiNRjRv3hwJCQlo0qRJZW0jIYQQQgghAF5sy23ZifP06dPx\nySefoE2bNubvnTp1CgkJCdi0Sf5aMUIIIYQQQiqaklvQPS+y117odDqrSTMAvPPOO891gwghhBBC\nCJGicnSweihx+vRphISI/+hx3bp16Nu3L0JCQhASEoIff/xRdhzZT5w9PT2RkJCALl26wNnZGSUl\nJThw4AA8PT0VbSQhhBBCCCEVSeVQvk+c165di+3bt8PJyUm07OzZs5g/fz5at26taCzZiXNSUhL2\n7duHkydPQqvVQqPRoHv37vDx8SnXBhNCCCGEEFIRVA525cp7eHhg+fLlmDZtmmjZuXPnkJWVhTt3\n7qBbt24YPXq07FiyE2eVSgUfHx+aKBNCCCGEkJdCea9x7tWrF37++Wfusr59+2Lo0KHQaDQYP348\nCgoK0L17d+l1P+8GKIQQQgghhFSUR3sLrL529pGe6Jb5+eefMXnyZGze/N9mWYwxaLVaODs7AwBy\ncnJQXFyM6OhoyXGe+43w+s7Lkl2+I34UACjq/mMrU5ZT2jlQaUdAW52OMob1AwBMXPetbG5p+ABE\nrLJ9N5Ivxg5R3NlOaTevgIV/ls1988kIm+ssW6/S/aE0p6Rj1qq9R2xu21ifTsj62/eymVE9OwCA\nzc5a6cF9TduYVyCbm9avu+LuZra6QwGmDlFKt01Jt01AWQdHpV3QlHRUA5QdC4Cy14jSjqEjMr+S\nzfx5XDAAZR3rAGUdBpXuN6XdzZQcpwCw8Lv9srlP/LrZ7FoGmDqXKe18qXS/Ke16qqTTo63ujYCp\ng6OSbnqA8telkg6DSjvbKe0cqLTLnJI62OrGCpg6ssb8Wb5L6bIRpi6lm78/LZsb3MF0EwEl53Jb\nmbKcreMPMB2DSvevkq57gLJuprY6qAKmLqpKz9G25gVfjDV1tFTSIdNWpiyXe/SUbCao48t9I4iK\nuquGVquFn58fdu7ciRo1aqCwsBCBgYGyP0OdAwkhhBBCSJWhsv9j09e8vDw8fvwYQUFBiI2NRWho\nKBwdHdGxY0d4e3vL/ixNnAkhhBBCSJXxe1puu7u7my/T6Nevn/n7/v7+8Pf3VzwOTZwJIYQQQkiV\n8SIboNDEmRBCCCGEVBm/5xPniiI7cQ4JCYFer7f6HmMMKpWKWm4TQgghhJBK90evcf4jZNc8ZcoU\nzJgxAytXroSdXfluNk0IIYQQQkhF+z2Xapw+fRoLFixAdrb1XVHy8/OxcuVK2NvbIzAwEIMHD5Yd\nR3bi3KZNG/Tv3x+XLl2iJiiEEEIIIeSFUzmU71INqZbber0e6enp2LJlC5ycnBAcHIwePXqgTp06\nkmOpba0sKiqKJs2EEEIIIeSl8NS5ptXDlrKW20JXr16Fh4cHXFxc4OjoiHbt2uH48eOyY1HnQEII\nIYQQUmU8evTI6uuyzn9yeJ0DT5w4gY0bN2LJkiUAgKVLl6JBgwYYNGiQ5DjP/epqpd2QlHTCil2/\n1eb6Fof5Y9znW2zmMiMHYvwX8l2TVkSYuiYp7TikpEuQrU5/gKnbX9CS9bKZ3ElhAJR3DlTSzUvp\nflPSWQmA4o5OSjph2er4BZi6finthLV4xwHZXGxf0w3QlTxXW13LAFPnsrGf2d6/q6IGKu5+qLSL\nn5Luh0qfg9JtU9plbvjyjbK5jROG23yegOm5KumqBSjfb4MWr5PNfR0bjtlf77K5bXMG9Vb8ulSa\nU9JdUul+s9VxETB1XVR6PCt9Dkq6/dl6XwBM7w2fF8i/z0R2N73P2KrXnEG9AQDDlsl3hsuJCVHc\naVXptinp8AlA0TGo9FyjtJOi0q6ctta7Kmqg4poq7cqp9FyuZNsAZe/jts4NgOn8YKvj7VifTgCU\ndw5U0oHU1nsWYHrfUtLl8f8DjUaDkpIS89clJSU2J+E2L9UghBBCCCHkZaFTO1g9fq9mzZrhxo0b\nKC4uhk6nw4kTJ+Dl5SX7M+X+xFmn08HxBd54mhBCCCGE/P+lKy39Qz9v2XI7Pj4ekZGRYIwhMDAQ\n9evXl/1ZyYlzfn4+kpOTYW9vj9jYWPj6mn61GhUVhQ0bbP8qgBBCCCGEkIqmNxjK/TNSLbd79OiB\nHj16KB5HcuK8evVqbN26FUajERMnTsSzZ88wYMAA0N8SEkIIIYSQF0Vfanxh65acODs4OMDFxQUA\nkJmZibCwMLi5uUGlUlXaxhFCCCGEEGJJb/hjl2r8EZJ/HPj6668jPT0djx8/hkajwYoVKzB37lz8\n+OOPlbl9hBBCCCGEmOlKDVaPyiQ5cU5LS4Onp6f5E2Y3Nzds2LABffr0qbSNI4QQQgghxNKLnDhT\nAxRCCCGEEFJlfH/lJ6uvO7zhIZk1Go1ISkrCpUuX4OjoiJSUFDRq1Mi8fN26dfj666/h6uoKAJgz\nZw6aNm0qOd5zb4BCCCGEEEJIRSnPp8z79u2DTqdDbm4uTp06hXnz5mHVqlXm5WfPnsX8+fPRunVr\nReM994nzugPyPb/Dvd8DAJtdmOYP9VPcPe7jjM9s5rZPi0KyjU5jM//TaUzpcxi5Jlc2t3Z0kM11\nlq035/A/ZDPDPmgLAPjqyD9lc8GdTDfyTtqyWzaXNLCX4s6BSjspKu02paQ7n9IuUkq6kQHAhr+f\nlM2FdmkHQFm3KaWd15S+fpV2uFLayU1Jh0ylXTmVdo30m79WNvdd3EgAyrqlKX1dKu2iqaQrHAAs\nsNHhbIpfN6R9u8/mtiUO+BDTcvJkMxnDTLdGUtJBFYCibnS2ujICps6MtrqKAqbOokq77inpVAlA\nUVe1Gbl/tbltKUF9FNdU6TGj5P0oYOGfbW7bN5+MUNxJUelzUPJ+pLSrodIumgMWfCGb+3ZKBADY\nrFdKUB+bXfIAU6c8pZ0vbZ0fMiNN50ul70dKunKuP3jC5raFdX1X8etNadfTmZvl92/y4D6K3yu7\nJa2QzexPGm9znBepPBPnkydPoksX075+5513cPbsWavl586dQ1ZWFu7cuYNu3bph9OjRsuPRJ86E\nEEIIIaTKKM9dNbRaLTQajflrOzs7lJaWwt7eNAXu27cvhg4dCo1Gg/Hjx6OgoADdu3eXHK9cLbef\nPn0KnU5Xnh8hhBBCCCGkwuhLjVYPORqNBiUlJeavjUajedLMGENYWBhcXV3h6OgIb29vnD9/XnY8\n2YnzlStXMG7cOCQkJODIkSPw9fWFr68vCgrkfxVHCCGEEELI86A3GKwectq2bYuDBw8CAE6dOoXm\nzZubl2m1Wvj5+aGkpASMMRQWFtq81ln2Uo3Zs2dj4sSJKCoqQkxMDHbv3o1q1aohKipK9mNsQggh\nhBBCngddqfJLNXx8fHD48GEMGTIEjDGkpaUhLy8Pjx8/RlBQEGJjYxEaGgpHR0d07NgR3t7esuPJ\nTpyNRiPef/99AEBhYSFeffVV0w/Z06XRhBBCCCGk8pXnjwPVajXmzp1r9b1mzZqZ/+3v7w9/f3/l\n48ktbNKkCaZPnw6j0Yh58+YBALKyslCnTh3FKyCEEEIIIaSi6EsNVo/KJPvRcUpKCvLz86FW/3d+\nXb9+fYSEhDz3DSOEEEIIIUTI1nXNzxN1DiSEEEIIIVWG8B7ZZffErgzluh0dIYQQQgghL1J57qph\nNBoxa9YsBAUFISQkBDdu3LBanp+fj8DAQAQFBWHz5s02101/5UcIIYQQQqoMnV75XTXkWm7r9Xqk\np6djy5YtcHJyQnBwMHr06CH7t3z0iTMhhBBCCKkySo1Gq4ccuZbbV69ehYeHB1xcXODo6Ih27drh\n+HH5tvb0iTMhhBBCCKkyynM7OrmW21qtFs7OzuZlNWvWhFarlR2PJs6EEEIIIaTKKE8DFLmW28Jl\nJSUlVhNpHrpUgxBCCCGEVBnluY+zXMvtZs2a4caNGyguLoZOp8OJEyfg5eUlOx594kwIIYQQQqoM\nvUH+umZLtlpux8fHIzIyEowxBAYGon79+rLjVdp9nI1GI5KSknDp0iU4OjoiJSUFjRo14mZPnz6N\nBQsWIDs7m7tcr9cjMTERRUVF0Ol0GDt2LHr27CnKGQwGzJgxA9euXYNKpcKcOXOs/qdh6d69ewgI\nCMAXX3xh1YpRaMCAAeZrZdzd3ZGeni7KrFmzBvn5+dDr9QgODsagQYNEmW+++QbffvstAODZs2e4\ncOECDh8+jFq1aomea3x8PIqKiqBWq5GcnMzdPp1Oh4SEBNy8eRMajQazZs1C48aNzcst9+mNGzcQ\nHx8PlUqFN998E7NnzzY3ueHt+7S0NDRp0gTBwcHc8S5cuIDk5GTY2dnB0dER8+fPN/9FqmXuypUr\nmDlzJhhjaNy4MVJSUmBvb89dZ15eHjZu3Ijc3FzuOs+fP4/Ro0ebn2NwcDB8fX1FuXv37mHGjBl4\n+PAhDAYDMjIy4OHhIcrFxsbi7t27AICioiK0adMGixcvFj3P2bNnw87ODo0bN0Zqaip3v507dw6z\nZ8+Go6MjWrZsienTp8NgMIhes2+88YaoDrxc2Wvbsg68Y6BBgwZWdUhNTcXChQutMo0aNRLVgDEm\nuU7LOvDW6ebmJqqDj4+PKPfOO+9Y1SEtLQ2rVq2yynz33XeiGmRkZHCfp7AOvP322muvierAGBOd\nE6pVqyaqAy9Xdu4oq8PgwYNFGYPBIDoWXnnlFVFOrVaL6qBSqSTXaVkH3nmttLTUqg5BQUE4fPiw\nVebVV18VHQuvv/66aKxVq1aJ6rBgwQLucxXWgbff9Hq9qA5qtdrqnGtvby95TuKdm3nnJMucTqfj\nnpMsM4wx7vlIap28c5Jl7tmzZ5LnJMtc7dq1Jc9JlrkVK1Zwz0nC5yl1TrLMPX36lFsD4fvZmDFj\nuHWQet8T1kGYCw0NFdVh5MiRVpnIyEhuHaTWKayDMBcSEsKtgzA3ZcoUbh2EuadPn4rqcP36ddHz\n5NVBONbw4cO5dRDOGd5//31RHdauXcudVwhrIByrdevW3GNBmPPy8pI8Hl5Go7KsbxuXNWpw5a2c\nVZLdu3ezuLg4xhhj//znP9mYMWO4uaysLObn58cGDRokOdaWLVtYSkoKY4yxBw8eMG9vb25u7969\nLD4+njHG2Pfffy+5Tp1Ox8aNG8c++ugjduXKFcn1Pn36lPXv319yedl6Ro8ezQwGA9NqtWzZsmWy\necYYS0pKYps2bZJ8DjExMYwxxg4dOsTGjx/PzWVnZ7MZM2Ywxhi7evUqi4iIMC8T7tPRo0ez77//\nnjHG2MyZM9mePXu4uXv37rHIyEjWs2dP9uWXX0qON2zYMHb+/HnGGGNfffUVS0tL4+bGjh3Ljh07\nxhhjLC4uju3Zs4db73PnzrHQ0FCr7wlzmzdvZp9//rloPwhzcXFxbMeOHYwxxo4ePcoKCgq4uTLF\nxcXs448/Zrdu3RJlxo0bx/bv388YY2zy5Mnsb3/7G3esAQMGsJMnTzLGGFu0aBHbunUr9zXLqwMv\nx6sDLyesw4gRI0QZXg2kjidhHXg5Xh14OWEdUlJSJI9hyxrwxuLVgZfj1YF3TuDVgZcT1oGX4R0L\nvByvDlLnK2EdeDlhHXgZ3rEgd460rAMvx6sDL8erg/CcK3VOEuakzknCHK8OwgyvBryxeDXg5aTO\nScKc1DlJ6n3Isg7CjNQ5SZjj1YD3fsarAy/HqwMvJ6zD3LlzRRleHaTea4V14OV4deDleHWQe48v\nq8NPP/0kyvDqwBuLVwfenEFYhxUrVogyvBrwxuIdC7yc1PHwsgpf+aXVozJV2jXOcrcDseTh4YHl\ny5fLjtW7d29MnDgRAMAYg52dHTf34YcfIjk5GQDwyy+/iD7NLTN//nwMGTIE9erVk13vxYsX8eTJ\nE0RERCA0NBSnTp0SZQ4dOoTmzZsjOjoaY8aMQbdu3WTHPHPmDK5cuYKgoCDu8iZNmsBgMMBoNEKr\n1Ur+D/DKlSvo2rUrAKBp06a4evWqeZlwn547dw7vv/8+AKBr1644cuQIN1dSUoIJEyagf//+VusS\n5hYtWoSWLVsCMH3KX61aNW5u+fLleO+996DT6XDnzh1oNBpR5sGDB1i0aBESExNl13n27Fns378f\nw4YNQ2JiovmvYIW5f/zjH7h16xbCw8ORl5dnft5Sr7Ply5dj+PDhqFevnijTsmVLFBcXgzGGkpIS\ncy2EuVu3bqFt27YATNdWnTx5kvua5dWBl+PVgZcT1sHT01OU4dWANxavDrwcrw68nLAOo0ePljyG\nLWvAG4tXB16OVwfeOYFXB15OWAdehncs8HK8OvByvDrwcsI6dOjQQZThHQty50jLOvByvDrwcrw6\nCM+5UuckYU7qnCTM8eogzPBqwBtL6pwkzEmdk4Q5qXOS1PuQZR2EGalzkjDHqwHv/YxXB16OVwde\nTlgHrVYryvDqwBuLVwdejlcHXo5XB7n3+LI63L9/X5Th1YE3Fq8OvDmDsA779u0TZXg14I3FOxZ4\nOanj4WWlKy21elSmSps4S90ORKhXr142fz1Qs2ZNaDQaaLVaxMTEYNKkSZJZe3t7xMXFITk5Gf36\n9RMt/+abb+Dq6mqe1MupXr06IiMj8fnnn2POnDmYMmWK6Dk8ePAAZ8+exdKlS80ZJnM1zJo1axAd\nHS25vEaNGigqKkKfPn0wc+ZMhISEcHMtW7ZEQUEBGGPmA9Twn246wn3KGINKpQJg2pePHj3i5ho2\nbIg2bdqI1iXMWb4hbNy4EeHh4dycnZ0dioqK4OfnhwcPHqBFixZWGYPBgOnTpyMhIQE1a9aUXefb\nb7+NadOmIScnBw0bNsTKlSu5uaKiItSqVQvr1q2Dm5sb1q5dy80Bpl9tHj16FAEBAdxM2a/g+vTp\ng3v37qF9+/aS++3YsWMAgIKCAjx58oT7muXVgZfj1YGXE9ah7FeilhleDYRjTZw4kVsH3jp5deDl\nhHXIycnhHsPCGvDG4tVBar8J6wCIzwlSx4Mwx6uDMCN1LAhzvDoIc35+fpLHg3A8Xh2EGaljgXeO\nFNaBl5M6Hnj7zbIOly5dEp1zeTXgnZt5NeDlhHWoW7euKMOrgXAso9HIrQFvnbwa8HK8Oki9D1nW\ngZfh1UBqvwmPBd77Ga8OvJybm5uoDrycq6urVR0CAwO56xTWQTjW5MmTER8fL6oDb52tWrUS1YGX\nu3HjhqgOUu/xlnXgZdzd3UV14OV4dZCaM1jWQavVijLu7u6iGvDGqlu3rlUNwsPDuTm1Ws09J72s\ndAaD1aNSVdZH22lpaeZfizDGWJcuXSSzN2/elL1UgzHGfvnlFzZgwAD29ddfK1r/7du3Wbdu3VhJ\nSYnV94dYQdrnAAAHPUlEQVQOHcqGDRvGhg8fztq1a8cCAwPZ7du3uWM8e/aMPXnyxPx1YGAg++WX\nX6wyn376qdWvifr168fu3r3LHe+3335jvr6+studlpbGFixYwBgzPWcfHx/29OlTUU6v17PU1FQ2\nZMgQlpGRwQIDA62WW+5Ty32/d+9eNmfOHG6uzLJly6x+LcrL7dixg/n5+bGffvpJNldm8+bNbNq0\naVaZ06dPM19fXzZ8+HA2aNAg5uXlZf71u3Cs3377zfz9y5cvs9DQUG6uU6dO7P79+4wx06/5oqKi\nJLdt48aNLDMzU3L7O3TowH744QdzNikpiZsru1QmNDSULVu2jKWmpjLGxK9ZqTpIvbaFdeDlhHWQ\nO04sa2CZk6uDcDypOghzvDrwto1XA2FOqg7CnFQdypSdE959911uHYS5snMH73iwzEgdC7yxhHWw\nzL311luse/fukseD5Xi//vqr+XvC46Es0759e8ljQbhtvDrwxpM6HixzFy5csKpD586dRefcli1b\nmn+urAZy52bLGkjlLOtg6zxfVgNhztPTk/Xs2VNUA1vjldWAl2vdurWoDlLjWdaBl2nVqpWoBrxc\nYWGh6FjgvZ+1aNFCVAe59z3LOkjlLOtg6z20rA7CnFQdbI1XVgep5yqsg9R4lnXgZZo3by6qAy/H\nqwNvzvD2229b1cHf319yXmFZA6n5h/CcZGuewjsnvWz8P/3c6lGZKu0TZ7nbgZTX3bt3ERERgalT\np2LgwIGSua1bt2LNmjUAACcnJ6hUKvMfTpTJycnBxo0bkZ2djZYtW2L+/Pnm/6EJbdmyBfPmzQNg\n+tWXVqsVZdu1a4e///3vYIzh1q1bePLkCWrXrs0d7/jx4+jYsaPsc61Vq5b5noIuLi4oLS01f5Js\n6cyZM+jYsSO++uor9O7dGw0bNpQc809/+hMKCwsBAAcPHsS7774ruw22bNu2zbwP5dY7ZswYXL9+\nHYDpf9HCWrz99tvYsWMHsrOzsWjRIrzxxhuYPn06d6zIyEj861//AgAcPXoUrVq14ubatWuHAwcO\nADDt7zfeeENy+44ePWq+3IXHxcXF/FuTevXq4eHDh9zcgQMHsGDBAqxfvx7FxcX44IMPuK9ZXh2U\nvrZ5OWEdeBleDYQ5qTrwxuPVgZcT1uH111/nPk9hDXhj8erAy/HqwDsntG7dWlQHJecOXmbPnj2i\nY4GXi46OFtVBmKtTpw527twpqgNvvPHjx1vVwdHRUZR57733RMeC1PMU1oGX49WBlysoKLCqQ0pK\niuic27VrV1ENlJ6bebkjR45Y1YGXmTlzpqgGwtyOHTuwb98+UQ14440bN050LPBy3bt3F9VB6rla\n1oGXcXd3F9WAlzt37pzoWOC9n33wwQeiOih53wP474/Hjh2zqgMvM2vWLFEdhLnGjRtj165dojrw\nxouOjhbVgZfz8fER1UHquVrWgZfx8PAQ1YGXK/sjcss68OYMHTt2tKpDp06dFM0reGMdPHhQdE7i\n5aZPny77/vyyeZGXalTan0zybgfye61evRoPHz5EZmYmMjMzAcD8axZLH330ERISEjBs2DCUlpYi\nMTFRlCmPgQMHIiEhAcHBwVCpVEhLSxP9ur979+44fvw4Bg4cCMYYZs2aJXkN9rVr1+Du7i67zvDw\ncCQmJmLo0KHQ6/WIjY1FjRo1RLlGjRph6dKlWL16NZydnZGamio5ZlxcHGbOnIlFixahadOm6NWr\nl4Jnz2cwGJCamgo3NzdMmDABAPDee+8hJiZGlB01ahTi4+Ph4OAAJycnpKSk/O71JiUlITk5GQ4O\nDqhTp475ukqhuLg4zJgxA5s2bYJGo8HChQslx7x27ZrsxD8lJQWxsbGwt7eHg4OD5DobNWqE8PBw\nODk5oX379vD29kZKSoroNTt9+nSkpKRY1SE9PV3Ra1t4DBgMBly+fBkNGjQw1+Hp06eisSZNmiSq\ngdLjiZeLj49HWlqaVR2WLFkiys2bN8+qDg0aNOCuU1gD3jp5deDlRowYIarD48ePReeEZs2aiY6H\nZ8+e2Tx38M4viYmJomMhKipKlHN1dRXVQaPRKDpf8dbr5uZmdTykpaUhJSXFKtOyZUvRseDg4MBd\np7AOvHXWrl1bVAdXV1dRTq1Wi+ogVJHnJKPRqOicVJHnI6DqnZPK7sRk+X72yiuviOpgMBhsvu8B\n4vfH1NRU8513yurQtm1bPHr0yGosAKI61K5d+3etMy0tDdWqVRPVwdHRUZSrX7++qA5OTk7c9VrW\ngbdOo9EoqkO9evVEueLiYu6xIJwzuLu7W9Vh8uTJWLhwoc15BW/+8cknn3CPBWGuZs2aFXo8PG/l\n6RxY0SrtdnSEEEIIIYRUZS/3Z/GEEEIIIYS8JGjiTAghhBBCiAI0cSaEEEIIIUQBmjgTQgghhBCi\nAE2cCSGEEEIIUYAmzoQQQgghhChAE2dCCCGEEEIUoIkzIYQQQgghCvwfwCrtevbAoLUAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2aaf347c6a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "from __future__ import division\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import style\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import math \n",
    "import os \n",
    "import random\n",
    "import numpy\n",
    "import cntk as C\n",
    "style.use('ggplot')\n",
    "%matplotlib inline\n",
    "try:\n",
    "    import gym\n",
    "except:\n",
    "    !pip install gym\n",
    "    import gym\n",
    "isFast = True\n",
    "env = gym.make('CartPole-v0')\n",
    "\n",
    "STATE_COUNT  = env.observation_space.shape[0]\n",
    "ACTION_COUNT = env.action_space.n\n",
    "\n",
    "STATE_COUNT, ACTION_COUNT\n",
    "# Targetted reward\n",
    "REWARD_TARGET = 30 if isFast else 200\n",
    "# Averaged over these these many episodes\n",
    "BATCH_SIZE_BASELINE = 20 if isFast else 50\n",
    "\n",
    "H = 64 # hidden layer size\n",
    "\n",
    "class Brain:\n",
    "    def __init__(self):\n",
    "        self.params = {}\n",
    "        self.model, self.trainer, self.loss = self._create()\n",
    "        # self.model.load_weights(\"cartpole-basic.h5\")\n",
    "\n",
    "    def _create(self):\n",
    "        observation = C.sequence.input_variable(STATE_COUNT, np.float32, name=\"s\")\n",
    "        q_target = C.sequence.input_variable(ACTION_COUNT, np.float32, name=\"q\")\n",
    "\n",
    "        # Following a style similar to Keras\n",
    "        l1 = C.layers.Dense(H, activation=C.relu)\n",
    "        l2 = C.layers.Dense(ACTION_COUNT)\n",
    "        unbound_model = C.layers.Sequential([l1, l2])\n",
    "        model = unbound_model(observation)\n",
    "\n",
    "        self.params = dict(W1=l1.W, b1=l1.b, W2=l2.W, b2=l2.b)\n",
    "\n",
    "        # loss='mse'\n",
    "        loss = C.reduce_mean(C.square(model - q_target), axis=0)\n",
    "        meas = C.reduce_mean(C.square(model - q_target), axis=0)\n",
    "\n",
    "        # optimizer\n",
    "        lr = 0.00025\n",
    "        lr_schedule = C.learning_rate_schedule(lr, C.UnitType.minibatch)\n",
    "        learner = C.sgd(model.parameters, lr_schedule, gradient_clipping_threshold_per_sample=10)\n",
    "        trainer = C.Trainer(model, (loss, meas), learner)\n",
    "\n",
    "        # CNTK: return trainer and loss as well\n",
    "        return model, trainer, loss\n",
    "\n",
    "    def train(self, x, y, epoch=1, verbose=0):\n",
    "        #self.model.fit(x, y, batch_size=64, nb_epoch=epoch, verbose=verbose)\n",
    "        arguments = dict(zip(self.loss.arguments, [x,y]))\n",
    "        updated, results =self.trainer.train_minibatch(arguments, outputs=[self.loss.output])\n",
    "\n",
    "    def predict(self, s):\n",
    "        return self.model.eval([s])\n",
    "MEMORY_CAPACITY = 100000\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "GAMMA = 0.99 # discount factor\n",
    "\n",
    "MAX_EPSILON = 1\n",
    "MIN_EPSILON = 0.01 # stay a bit curious even when getting old\n",
    "LAMBDA = 0.0001    # speed of decay\n",
    "class Memory:   # stored as ( s, a, r, s_ )\n",
    "    samples = []\n",
    "\n",
    "    def __init__(self, capacity):\n",
    "        self.capacity = capacity\n",
    "\n",
    "    def add(self, sample):\n",
    "        self.samples.append(sample)\n",
    "\n",
    "        if len(self.samples) > self.capacity:\n",
    "            self.samples.pop(0)\n",
    "\n",
    "    def sample(self, n):\n",
    "        n = min(n, len(self.samples))\n",
    "        return random.sample(self.samples, n)\n",
    "class Agent:\n",
    "    steps = 0\n",
    "    epsilon = MAX_EPSILON\n",
    "\n",
    "    def __init__(self):\n",
    "        self.brain = Brain()\n",
    "        self.memory = Memory(MEMORY_CAPACITY)\n",
    "\n",
    "    def act(self, s):\n",
    "        if random.random() < self.epsilon:\n",
    "            return random.randint(0, ACTION_COUNT-1)\n",
    "        else:\n",
    "            return numpy.argmax(self.brain.predict(s))\n",
    "\n",
    "    def observe(self, sample):  # in (s, a, r, s_) format\n",
    "        self.memory.add(sample)\n",
    "\n",
    "        # slowly decrease Epsilon based on our eperience\n",
    "        self.steps += 1\n",
    "        self.epsilon = MIN_EPSILON + (MAX_EPSILON - MIN_EPSILON) * math.exp(-LAMBDA * self.steps)\n",
    "\n",
    "    def replay(self):\n",
    "        batch = self.memory.sample(BATCH_SIZE)\n",
    "        batchLen = len(batch)\n",
    "\n",
    "        no_state = numpy.zeros(STATE_COUNT)\n",
    "\n",
    "\n",
    "        # CNTK: explicitly setting to float32\n",
    "        states = numpy.array([ o[0] for o in batch ], dtype=np.float32)\n",
    "        states_ = numpy.array([(no_state if o[3] is None else o[3]) for o in batch ], dtype=np.float32)\n",
    "\n",
    "        p = agent.brain.predict(states)\n",
    "        p_ = agent.brain.predict(states_)\n",
    "\n",
    "        # CNTK: explicitly setting to float32\n",
    "        x = numpy.zeros((batchLen, STATE_COUNT)).astype(np.float32)\n",
    "        y = numpy.zeros((batchLen, ACTION_COUNT)).astype(np.float32)\n",
    "\n",
    "        for i in range(batchLen):\n",
    "            s, a, r, s_ = batch[i]\n",
    "\n",
    "            # CNTK: [0] because of sequence dimension\n",
    "            t = p[0][i]\n",
    "            if s_ is None:\n",
    "                t[a] = r\n",
    "            else:\n",
    "                t[a] = r + GAMMA * numpy.amax(p_[0][i])\n",
    "\n",
    "            x[i] = s\n",
    "            y[i] = t\n",
    "\n",
    "        self.brain.train(x, y)    \n",
    "def plot_weights(weights, figsize=(7,5)):\n",
    "    '''Heat map of weights to see which neurons play which role'''\n",
    "    sns.set(style=\"white\")\n",
    "    f, ax = plt.subplots(len(weights), figsize=figsize)\n",
    "    cmap = sns.diverging_palette(220, 10, as_cmap=True)\n",
    "    \n",
    "    for i, data in enumerate(weights):\n",
    "        axi = ax if len(weights)==1 else ax[i]\n",
    "        if isinstance(data, tuple):\n",
    "            w, title = data\n",
    "            axi.set_title(title)\n",
    "        else:\n",
    "            w = data\n",
    "                \n",
    "        sns.heatmap(w.asarray(), cmap=cmap, square=True, center=True, #annot=True,\n",
    "                    linewidths=.5, cbar_kws={\"shrink\": .25}, ax=axi)\n",
    "def epsilon(steps):\n",
    "    return MIN_EPSILON + (MAX_EPSILON - MIN_EPSILON) * np.exp(-LAMBDA * steps)\n",
    "plt.plot(range(100000), [epsilon(x) for x in range(100000)], 'r')\n",
    "plt.xlabel('step');plt.ylabel('$\\epsilon$')\n",
    "TOTAL_EPISODES = 2000 if isFast else 3000\n",
    "\n",
    "def run(agent):\n",
    "    s = env.reset()\n",
    "    R = 0\n",
    "\n",
    "    while True:\n",
    "        # Uncomment the line below to visualize the cartpole\n",
    "        # env.render()\n",
    "\n",
    "        # CNTK: explicitly setting to float32\n",
    "        a = agent.act(s.astype(np.float32))\n",
    "\n",
    "        s_, r, done, info = env.step(a)\n",
    "\n",
    "        if done: # terminal state\n",
    "            s_ = None\n",
    "\n",
    "        agent.observe((s, a, r, s_))\n",
    "        agent.replay()\n",
    "\n",
    "        s = s_\n",
    "        R += r\n",
    "\n",
    "        if done:\n",
    "            return R\n",
    "\n",
    "agent = Agent()\n",
    "\n",
    "episode_number = 0\n",
    "reward_sum = 0\n",
    "while episode_number < TOTAL_EPISODES:\n",
    "    reward_sum += run(agent)\n",
    "    episode_number += 1\n",
    "    if episode_number % BATCH_SIZE_BASELINE == 0:\n",
    "        print('Episode: %d, Average reward for episode %f.' % (episode_number,\n",
    "                                                               reward_sum / BATCH_SIZE_BASELINE))\n",
    "        if episode_number%200==0:\n",
    "            plot_weights([(agent.brain.params['W1'], 'Episode %i $W_1$'%episode_number)], figsize=(14,5))\n",
    "        if reward_sum / BATCH_SIZE_BASELINE > REWARD_TARGET:\n",
    "            print('Task solved in %d episodes' % episode_number)\n",
    "            plot_weights([(agent.brain.params['W1'], 'Episode %i $W_1$'%episode_number)], figsize=(14,5))\n",
    "            break\n",
    "        reward_sum = 0\n",
    "agent.brain.model.save('dqn.mod')\n",
    "env = gym.make('CartPole-v0')\n",
    "\n",
    "num_episodes = 10  # number of episodes to run\n",
    "\n",
    "modelPath = 'dqn.mod'\n",
    "root = C.load_model(modelPath)\n",
    "\n",
    "for i_episode in range(num_episodes):\n",
    "    print(i_episode)\n",
    "    observation = env.reset()  # reset environment for new episode\n",
    "    done = False\n",
    "    while not done: \n",
    "        if not 'TEST_DEVICE' in os.environ:\n",
    "            env.render()\n",
    "        action = np.argmax(root.eval([observation.astype(np.float32)]))\n",
    "        observation, reward, done, info  = env.step(action)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import cntk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}