exp2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adapting Behavior via Intrinsic Reward: A Survey and Empirical Study\n",
    "## Experiment 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import ipywidgets as widgets\n",
    "from IPython.display import display, clear_output\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is adapted from [the Adapting Behavior via Intrinsic Reward: a Survey and Empirical Study Drifter Distractor notebook](http://jair.adaptingbehavior.com/). In this notebook, we implement experiment 2 and try to reproduce the results from [the research paper](https://arxiv.org/pdf/1906.07865.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment Two: Switched Drifter-Distractor Problem\n",
    "This section implements Experiment Two: Switched Drifter-Distractor Problem from the paper. See section 7 and 9 for a description."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_action_data_two(switch=50000, num_steps=150000):\n",
    "    num_steps_left = num_steps - switch\n",
    "    # For the distractors each step is a random number chosen from a Normal distribution mean 0.0, var 1.0\n",
    "    target1 = np.random.randn(switch)\n",
    "    target4 = np.random.randn(switch)\n",
    "\n",
    "    # Constant number randomly chosen between [-50, 50]\n",
    "    target3 = np.ones(switch) * np.random.uniform(-50, 50)\n",
    "\n",
    "    # At each step the drifter changes slightly by mean 0.0 and variance 0.1\n",
    "    random_drift = np.random.randn(switch) * 0.1\n",
    "    target2 = np.cumsum(random_drift)\n",
    "\n",
    "\n",
    "    # Switch\n",
    "\n",
    "    # At each step after switch, the drifter changes slightly\n",
    "    # by variance 0.1\n",
    "    random_drift1 = np.random.randn(num_steps_left) * 0.1\n",
    "    random_drift2 = np.random.randn(num_steps_left) * 0.1\n",
    "    drifter1 = np.cumsum(random_drift1) + target1[-1]\n",
    "    drifter2 = np.cumsum(random_drift2) + target3[-1]\n",
    "\n",
    "    # At each step after switch, for the distractors is a random number chosen\n",
    "    # from a Normal distribution mean of last value of the target before switch, var 1.0\n",
    "    distractor = np.random.randn(num_steps_left) + target2[-1]\n",
    "\n",
    "    # At each step after switch, the constant is 0\n",
    "    constant = np.zeros(num_steps_left) + target4[-1]\n",
    "\n",
    "    target1 = np.concatenate((target1, drifter1))\n",
    "    target2 = np.concatenate((target2, distractor))\n",
    "    target3 = np.concatenate((target3, drifter2))\n",
    "    target4 = np.concatenate((target4, constant))\n",
    "\n",
    "\n",
    "    return np.array([target1, target2, target3, target4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0xbfc5f10>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAABbQUlEQVR4nO2dd3hURReH30kj1IQO0kJvgST0gJTQm4B0BQQbYsGC0hQRQZSiSNMPURRQiiCCSC/SpYYWqrTQS2iBEELa/f64W7N9s5tNwrzPkyf3zsyde3Z29+zcKb8jFEVBIpFIJNkTL08bIJFIJBL3IZ28RCKRZGOkk5dIJJJsjHTyEolEko2RTl4ikUiyMT6eNsCQQoUKKUFBQZ42Q+Ii7ty5A0DBggU9bIlEkr2JjIy8rShKYXN5mcrJBwUFceDAAU+bIXERc+fOBWDAgAEetUMiye4IIS5aypPDNRKJRJKNkU5eIpFIsjHSyUskEkk2JlONyZsjKSmJK1eukJCQ4GlTJA5SvXp1AE6ePOlhS7Iv/v7+lCxZEl9fX0+bIsmkZHonf+XKFfLmzUtQUBBCCE+bI3GA27dvA1CoUCEPW5I9URSFO3fucOXKFcqWLetpcySZlEw/XJOQkEDBggWlg5dI0iCEoGDBgvIpV2KVTO/kAengJRILyO+GxBZZwslLJBKJxDmkk7fBnTt3CA0NJTQ0lGLFilGiRAndeWJiokvvdf/+fb7//nuL+a+88gpFihQhODjYrvq2bt1KQEAAYWFhVK5cmSZNmrBq1Spd/qxZs5g/f77V6//991/7X4ANpk6dSnx8fLrqWLp0KdWrV8fLy8tlG+cGDBhA2bJlCQkJoVKlSrz00ktcuXLFYvnXXnuNEydO6OypWrUqERERLmuvtm3bEhgYSMeOHdNdl0SCoiiZ5q927dpKWk6cOGGS5ik+++wzZfLkyXaVTUpKcrj+CxcuKNWrV7eYv23bNiUyMtJqGUO2bNmidOjQQXd+6NAhpUyZMsqmTZvsut7a67Xn9cXExCgxMTG68zJlyhid20NycrLR+YkTJ5RTp04pTZs2Vfbv3+9QXZbo37+/snTpUkVRFCU1NVWZMmWKUrFiReXJkyc27WnTpo2yY8cORVEc+3xoMdeOmzZtUlauXGn03lkjM31HJJ4BOKBY8KuyJ+8EP/74I3Xr1iUkJIRu3brpeqcDBgxg0KBB1K9fn2HDhnHu3DkaNGhAjRo1GDVqFHny5NHVMXnyZOrWrUvNmjX57LPPABgxYgTnzp0jNDSUoUOHmty3SZMmFChQwGm7Q0NDGT16NDNnzgRgzJgxfP311wBMnz6datWqUbNmTXr37k10dDSzZs3i22+/JTQ0lB07dpi8vn379hEeHk5YWBgNGzbk9OnTAKSkpPDRRx/RuHFjmjZtyowZM5g+fTrXrl0jIiKCiIgIABYtWkSNGjUIDg5m+PDhOjvz5MnDhx9+SEhICLt37zZ6DVWrVqVy5coOv/adO3fqtHSsIYTggw8+oFixYqxdu9asPc2aNePAgQOMHTuWnTt38uqrr9KjRw+T9oqJiaFbt27UrVuXunXrsmvXLl279+vXj0aNGtGvXz8TG1q0aEHevHkdfo0SiTky/RJKQz7/+zgnrj1waZ3VnsnHZ89Vd+iarl278vrrrwMwatQo5syZw+DBgwF1yee///6Lt7c3HTt25L333uOFF15g1qxZuus3bNjAmTNn2LdvH4qi0KlTJ7Zv386ECRM4duwYhw8fdsgebd2DBg2yWbZWrVpMnjzZJH3ChAlcuHCBHDlycP/+fQIDAxk0aBB58uTho48+AmDOnDlGr+/Bgwfs2LEDHx8fNm3axMcff8yyZcuYPXs20dHRbNmyBR8fH7y8vChQoABTpkxhy5YtFCpUiGvXrjF8+HAiIyPJnz8/rVu3ZsWKFXTp0oVHjx5Rv359vvnmG4fawRr37t2jXbt2rF271i7BtFq1anHq1Ck6d+5s0Z7Ro0fzzz//8PXXX1OnTh3GjBlj1F4vvvgiH3zwAc8++yyXLl2iTZs2uj0DJ06cYOfOneTMmdNlr1EiMUeWcvKZhWPHjjFq1Cju379PXFwcbdq00eX16NEDb29vAHbv3s2KFSsA9Quv/fJv2LCBDRs2EBYWBkBcXBxnzpyhdOnSTtljj3PXoliI6VuzZk369OlDly5d6NKli8XrDV9fbGws/fv358yZMwghSEpKAmDTpk0MGjQIHx/142Xu6WP//v00a9aMwoVV4bw+ffqwfft2unTpgre3N926dbP7NaVl4sSJ/Prrrybply5d4s0332TJkiU26zBsJ2ft2bRpk27sHuDBgwfExcUB0KlTJ+ngJRlClnLyjva43cWAAQNYsWIFISEhzJ07l61bt+rycufObfN6RVEYOXIkb7zxhlF6dHS0iy015dChQ1StWtUkffXq1Wzfvp2///6b8ePHExUVZfZ6w9f36aefEhERwfLly4mOjqZZs2YusdHf31/3Q+IMw4cPNxr+AThy5Aj9+vVj3LhxdtVx6NAhWrRokS57UlNT2bNnD/7+/iZ59nxOJBJXIMfkneDhw4cUL16cpKQkFixYYLFcgwYNWLZsGQCLFy/Wpbdp04aff/5Z16u7evUqt27dIm/evDx8+NBtdh89epRx48bx9ttvG6WnpqZy+fJlIiIimDhxIrGxscTFxdm0JzY2lhIlSgB6WWGAVq1a8cMPP5CcnAzA3bt3AYzqq1evHtu2beP27dukpKSwaNEimjZt6sqXa8TJkyf5448/bI7nK4rC9OnTuX79Om3btnXoHmnbq3Xr1syYMUN37ugwnETiClzm5IUQ3kKIQ0KIVZrzskKIvUKIs0KI34UQfq66l6cZN24c9evXp1GjRlSpUsViualTpzJlyhRq1qzJ2bNnCQgIANQv/4svvkh4eDg1atSge/fuPHz4kIIFC9KoUSOCg4PNTry+8MILhIeHc/r0aUqWLMmcOXMAdUzecMzfkB07duiWUL799ttMnz5d10PVkpKSQt++falRowZhYWG8++67BAYG8txzz7F8+XLdRGJahg0bxsiRIwkLC9M5dFCXGJYuXZqmTZvSrFkzFi5cCMDAgQNp27YtERERFC9enAkTJhAREUFISAi1a9emc+fONloeli9fTsmSJdm9ezcdOnQwGiqzRu/evalUqZLF/KFDh+qWUO7fv58tW7bg5+fYRzZte02fPp0DBw5Qs2ZNqlWrZvE9Skvjxo3p0aMHmzdvpmTJkqxfv94hOyQSQ4SlMVqHKxJiCFAHyKcoSkchxBLgT0VRFgshZgFHFEX5n7U66tSpo6Rd+3zy5EmzwwtZgfj4eHLmzIkQgsWLF7No0SL++usvT5uVYUjtmowhK39HJK5BCBGpKEodc3ku6ckLIUoCHYCfNOcCaA78oSkyD+jiintlJSIjIwkNDaVmzZp8//33Ll0tIpFIJPbgqonXqcAwQLu4tyBwX1EU7TP8FaCEuQuFEAOBgYDTq0syK40bN+bIkSOeNkMikTzFpLsnL4ToCNxSFCXSmesVRZmtKEodRVHqaJfTSSQSicQ1uKIn3wjoJIRoD/gD+YBpQKAQwkfTmy8JXHXBvSQSiUTiAOnuySuKMlJRlJKKogQBvYF/FEXpA2wBumuK9QeenhlHiUQiySS4c538cGCIEOIs6hj9HDfeSyKRSCRmcKmTVxRlq6IoHTXH5xVFqacoSgVFUXooivLElffKKDKL1LB2s1K1atWoXr0606ZNs1mfLanhtKxcuZIJEyYAEBMTQ/369QkLC2PHjh1WJZDdQXaUSZZIPIIleUpP/EmpYctSw9euXVMiIyMVRVGUBw8eKBUrVlSOHz9utT5HpIbT2rto0SLl1VdftWmXJVJTU5WbN286LC2sxdMyyWlxhUyyu8hM3xGJZ0BKDbsWT0gNFy9enFq1agHq9vmqVaty9apjc9lppYbT2jt37lzeeecdDh8+zLBhw/jrr78IDQ1l+PDhJnaZsz86OprKlSvz0ksvERwcbNO+2NhYI90fR2x3t0xycHAwNWvWdKlMskTiCbKUQBlrR8AN88JZTlOsBrSb4NAlnpYajo6O5tChQ9SvXx9In9Swob1a/ZnQ0FDGjh3LgQMHmDlzJtHR0Rw/flxnlyX7S5cuzZkzZ5g3bx4NGjTQ7Xi1RFJSEkOHDmXMmDF06NDBYdu1uEMm+fDhw/j4+HD37l2PyyRLJOkhazn5TIInpYbj4uLo1q0bU6dOJV++fED6pIYN7bUXa/aXKVOGBg0amFxz+vRps3K9Dx48oFu3bly7ds1mQJS0tmvJTjLJEomryVpO3sEet7vwlNRwUlIS3bp1o0+fPnTt2tUZ002khp2RvLVmv6X6KleuzLFjx4zSnjx5Qrdu3WjVqpVdEa+eBplkicTVyDF5J/CE1LCiKLz66qtUrVqVIUOGOGW3JalhW6S1y5L9jhITE0PXrl157733bJZ9mmWSJZL0IJ28E3hCanjXrl38+uuv/PPPP7olnGvWrAHSLzVsi7R2WbLfUUqWLMkrr7xiMd+TMsk1a9YkJCTEpTLJEokncJnUsCuQUsPZCyk1nDFk5e+IxDVYkxrOWmPyWYzIyEjeeecdFEUhMDCQn3/+2dMmSSSSpwzp5N2IlBqWSCSeRo7JSyQSSTZGOnmJRCLJxkgnL5FIJNkY6eQlEokkGyOdvA0yi9RwQkIC9erVIyQkhOrVq+tEwWyxdu1a6tSpQ7Vq1QgLC+PDDz90pcl8+eWX6bp+xYoVnDhxwiW2eHt7ExoaSvXq1QkJCeGbb74hNTXVbNlr167RvXt33fkLL7xAzZo1+fbbb10iKzxlyhSdaFqLFi24ePFiuuqTSJzGkjylJ/6k1LBlSd/U1FTl4cOHiqIoSmJiolKvXj1l9+7dVuuLiopSypUrp5w8eVJRFFX69vvvv3fYLmvkzp3bYl5MTIxNed7+/fsrS5cudbktN2/eVFq0aKGMHj3apFza9+b69etK+fLldeeukBX+559/lEePHimKoijff/+90rNnT4fqc4TM9B2ReAak1LBr8YTUsBBCd31SUhJJSUkIIazaOWnSJD755BPdrlxvb2/efPNNQNWZad68ua6neenSJd1rePfdd2nYsCHlypXjjz/+AOD69es0adKE0NBQgoOD2bFjByNGjODx48eEhobSp08fALp06ULt2rWpXr26UVCPPHny8MknnxASEkKDBg24efMm//77LytXrmTo0KGEhoZy7tw5q69HURRWrlxptYyWIkWKMHv2bGbOnImiKMydO5dOnTrRvHlzWrRoQXR0NMHBwYC6A/nq1auEhoby+eefm8gKb9iwgfDwcGrVqkWPHj10cg5BQUEMHz6cWrVqsXTpUqP7R0REkCtXLkCVt7hy5YpddkskriZLrZOfuG8ip+6ecmmdVQpUYXi94bYLGuApqeGUlBRq167N2bNnefvtt3VSw6NHj6ZOnTp06tTJqPyxY8csDs8MHjyY/v37079/f37++WfeffddnWLm9evX2blzJ6dOnaJTp050796dhQsX0qZNGz755BNSUlKIj4+ncePGzJw508jen3/+mQIFCvD48WNq1apFx44dKVSoEI8ePaJBgwaMHz+eYcOG8eOPPzJq1Cg6depEx44djYZOLKEoCkuWLGH//v2MGzfOZvly5cqRkpKi09U5ePAgR48epUCBAkZicCtXrqRjx4661/HLL7/oZIVv377NF198waZNm8idOzcTJ05kypQpjB49GlAlHw4ePGjVjjlz5tCuXTub9kok7iBLOfnMgqekhr29vTl8+DD379/n+eef59ixYwQHBzN27FiHX8Pu3bv5888/AejXrx/Dhg3T5XXp0gUvLy+qVavGzZs3Aahbty6vvPIKSUlJdOnShdDQULP1Tp8+neXLlwOqcNn58+epVKkSfn5+dOzYEYDatWuzceNGmzY2bNiQBw8eGKWlpKSwYMECwsPDad++vUOv2V61S0P27NnDiRMnaNSoEQCJiYmEh4fr8nv16mX1+t9++40DBw6wbds2h+4rkbiKLOXkHe1xuwtPSQ1rCQwMJCIignXr1umGHMxRvXp1IiMjCQkJsateLTly5DCyFaBJkyZs376d1atXM2DAAIYMGcJLL71kdN3WrVvZtGkTu3fvJleuXDRq1IgnT9TQvr6+vrrhJW9vbyNRMEuYi9H6xRdfEBUVRevWrW1ef/78eby9vSlSpAjgvKxyq1atWLRokdl8a3Vu2rSJ8ePHs23bNqM2lUgyEjkm7wSekBqOiYnh/v37ADx+/JiNGzdaVcAEGDp0KF9++SX//fcfoMryaoeNGjZsqLNpwYIFNG7c2GpdFy9epGjRorz++uu89tpruiEKX19fXcCN2NhY8ufPT65cuTh16hSRkZFW6wRTGWNrJCcn4+vry8KFC3VBPSwRExPDoEGDeOedd2zOXVizqUGDBuzatYuzZ88C8OjRI117WuPQoUO88cYbrFy5UvcjI5F4AunkncATUsPXr18nIiKCmjVrUrduXVq1aqUb/hg9erTZCcmaNWsydepUXnjhBapWrUpwcDDnz58HYMaMGfzyyy/UrFmTX3/9lWnTpll9zVu3biUkJISwsDB+//13nQb8wIEDdZGZ2rZtS3JyMlWrVmXEiBHUrl3bZlv27t2byZMnExYWZnPi1cfHh+HDh1sMyqGdBK5evTotW7akdevWdi81NcRQVrhw4cLMnTtXt8QyPDycU6dszwsNHTqUuLg4evToQWhoqMl8iUSSUUipYTcipYal1HBGkJW/I08rKakKXgKHnzItIaWGPYSUGpZIJGm5fDeexpO2ABA9wXYA+/QinbwbkVLDEokkLe8sOpSh95Nj8hKJRJKBXLv/OEPvJ528RCKRZBCDfo0k5uGTDL2ndPISiUSSQaw7fiPD7ymdvEQikXiIk9cf2C6UTqSTt0FmkRrWkpKSQlhYmG6NvDXmzp1L4cKFCQsLo2LFirRp08ZoF+no0aPZtGmTxetdKQMM6ZclBujTpw+VK1cmODhYJ7OQXpKSkhgxYgQVK1akVq1ahIeHs3bt2nTXq+Xw4cOsWbMmXXW4ou0k7iExOZWLdx7x3ZazWFuSfuxqrEnagF/2udM0wAVOXghRSgixRQhxQghxXAjxnia9gBBioxDijOZ//vSbm/EULFiQw4cPc/jwYQYNGsQHH3ygO/fz87N4nT3b9tNij5OfNm2aQ2uie/XqxaFDhzhz5gwjRoyga9eunDx5EoCxY8fSsmVLi9dac/LOvD5HHZWiKCZ68H369OHUqVNERUXx+PFjfvrpJ4ftSMunn37K9evXOXbsGAcPHmTFihV278K1B+nkszeVRq2l6eStTF5/migzjlxLxxk7TdIK5XG/3IUrevLJwIeKolQDGgBvCyGqASOAzYqiVAQ2a86zBZ6QGgZV4XL16tW89tprTtkdERHBwIEDmT17ts5erZTwiBEjdEEuPvroI7MywM2aNeP999+nTp06TJs2jb///pv69esTFhZGy5YtdWJmcXFxvPzyyzRp0oSmTZuybNkys7LEU6ZMITg4mODgYKZOnQqo+j2VK1fmpZdeIjg4mMuXLxu9hvbt2yOEQAhBvXr17JbwXb9+PY8fm65qiI+P58cff2TGjBk6fZmiRYvSs2dPABYtWkSNGjUIDg5m+HC9dpI56WSApUuXEhwcTEhICE2aNCExMZHRo0fz+++/Exoayu+//86+ffsIDw8nLCyMhg0bcvr0aUB98uratStt27alYsWKOtE4c20nyZw8STYfpCYtef3V1evViudzpzmAC9bJK4pyHbiuOX4ohDgJlAA6A800xeYBW4F0KYzd+PJLnpx0rdRwjqpVKPbxxw5d4ymp4ffff59JkyaZ9DItSQ2bo1atWvzwww9GaXfu3GH58uWcOnUKIQT3798nMDDQrAxwYmIi2l3J9+7dY8+ePQgh+Omnn5g0aRLffPMN48aNIyAggO3btwOqIFm3bt2MZIkjIyP55Zdf2Lt3L4qiUL9+fZo2bUr+/Pk5c+YM8+bNo0GDBhZfR1JSkl1yDFouXLjAlClTWLFiBTlz5tSlnz17ltKlS5Mvn+mX7dq1awwfPpzIyEjy589P69atWbFiBV26dLEonTx27FjWr19PiRIluH//Pn5+fowdO5YDBw4wc+ZMAB48eMCOHTvw8fFh06ZNfPzxxzqNo8OHD3Po0CFy5MhB5cqVGTx4MBMmTDCRdJZ4jvcWH+L0jYesfrcx3l7GO1avxyYA6lNoUoqCn4/5fvRnz1Xno6VHWBp5hXFdgvH3NS/V4QpcuhlKCBEEhAF7gaKaHwCAG0BRC9cMBAYCNqV2MwuekBpetWoVRYoUoXbt2kaql4BDUsPmxgwDAgLw9/fn1VdfpWPHjlbH+w2lda9cuUKvXr24fv06iYmJlC1bFlDVFw0F2fLnNx2p27lzJ88//7xOxbFr167s2LGDTp06UaZMGasOHuCtt96iSZMmZoXV3nvvPTZv3mySfvbsWT777DMmTZpktW4t+/fvp1mzZhQuXBhQh4q2b99Oly5dLEonN2rUiAEDBtCzZ0+6du1qtt7Y2Fj69+/PmTNnEEIYzSu0aNFCp3FUrVo1Ll68SKlSpeyyV+J+Yh8n8dfhawCU/9h0CK5Qbj8SklKo8uk6AP75sCnlCufh+DXjYZx6QXrJ6yqfrqNBuQIsHhiOO3CZkxdC5AGWAe8rivLAUJNBURRFCGF2RkJRlNnAbFC1a6zdw9Eet7vwhNTwrl27WLlyJWvWrCEhIYEHDx7Qt29ffvvtN4dsP3TokMmYvo+PD/v27WPz5s388ccfzJw5k3/++cfs9Yavb/DgwQwZMoROnTqxdetWxowZ45AtlrDVhp9//jkxMTEmTyRazPXuN27cyIgRIxgyZIhReoUKFbh06RIPHjww25u3hCXp5FmzZrF3715Wr15N7dq1zSpxfvrpp0RERLB8+XKio6Np1qyZLs9QktheSWZJxtFvzl6r+aP+Osb5mEe680dPUgCYte28Lu2NJuUomMd4Pm/P+bvExicRkMvXhdaquGR1jRDCF9XBL1AU5U9N8k0hRHFNfnHglivulRnwhNTwV199xZUrV4iOjmbx4sU0b97cYQe/bds2Zs+erRtq0hIXF0dsbCzt27fn22+/1Ukx2JIBjo2NpUSJEgDMmzdPl96qVSu+++473fm9e/cAY1nixo0bs2LFCuLj43n06BHLly+3KXcM8NNPP7F+/XoWLVqEl5f9H99z586xZs0aihUrZpSeK1cuXn31Vd577z3daqmYmBiWLl1KvXr12LZtG7dv3yYlJYVFixbRtGlTm/epX78+Y8eOpXDhwly+fNmkHQ3bbe7cuXbZb9h2Es9w+sZDjl6xPLEKGDl4QDec8/eRa7q0Ee2qkDuHaf86ZOwGF1hpiitW1whgDnBSUZQpBlkrgf6a4/5AtpFf9ITUsDUsSQ0Dugm/SpUq8eWXX7Js2TKTnvzDhw/p2LEjNWvW5Nlnn2XKFPVttCUDPGbMGHr06EHt2rWNlCZHjRrFvXv3aNy4Mc2aNWPLFlWMyVCWuFatWgwYMIB69epRv359XnvtNd3wlTUGDRrEzZs3CQ8PJzQ01O6hqkGDBlG0qNkRQ7744gsKFy5MtWrVCA4OpmPHjuTLl4/ixYszYcIEIiIiCAkJoXbt2nTu3NnqfYYOHaqbqG3YsCEhISFERERw4sQJ3cTrsGHDGDlyJGFhYXb31A3bTuIZ2kzd7vA1jxKTTZZOukp50l7SLTUshHgW2AFEAdqp5Y9Rx+WXAKWBi0BPRVHuWqtLSg1nL6TUcMaQlb8jWYmgEast5pUvnJtzaXrxltAqTx6/FkuH6TvN5jmKW6WGFUXZCVj6aWqR3vqzMlJqWCLJmpy+8RAFhSrF7JunWTW4MVVHr7NZbv4r9XTH1Z8J4JP2VRm/Rt23MrVXqFO22kJKDbsRKTUskWRNtEMzlnrWLzcK4pdd0QBEjWlNTj/7lkBWLJrH6Pz1JuW4/egJRy7f57mQZ5w32ApZwskripLh41gSSVYgM0V2yy5cuhNvs8xnz1WnaD5/6pTJT15/+1fEFMvnb5I2sp17h9oyvXaNv78/d+7ckR9miSQNiqJw584d/P1NHYfEeZpM3mKSlpJq6n8GNS1PHYP17r8P1O/tqGphJ6snOquZvidfsmRJrly5QkxMjKdNkTiIdomofO/ch7+/PyVLlvS0GdmeOTv169wndqthtkz9cgXZPbI5k9edZljbKjT4ynRDnifI9E7e19dXt5NSkrXQrgEfMGCAR+2QSOzF0ojBl2v0ciq96lremV48ICdTNBOoQ9tUZvL607q8ttWLWbjKvWT64RqJRCLJKN741XSH8m97LjpV1/NhJXTHu0Y0Z1a/2k7blR4yfU9eIpFIMooNJ24ancc9Sebbjf/pznvWsX9orFg+fzrUKM6gpuUpEZjT9gVuQvbkJRKJBNh19rZJ2tV7j+lTXz8883yY/U7ey0vwXZ9a1CgZ4BL7nEX25D1JzH9w9QCEvuhpSySSp5Z203bg4yXMBvy4ej+eFINx+vDyBTPSNJcgnbwn+a6u+r9GT/CWb4VE4gmsxVn9dfdFtpzO2qvD5HCNKzm1Gg784vh1T9wfzFcikThOVnfwIJ2845z7B1JTTNOvHYLFL8Kq9x2vc5t9QSwkEolribIgHbzynUYmabtGNHe3OW5BOnlHOLsZfn0exhYwzds727G64gx6CPF3nLcpNRVSZGAJicQZnptpGlz76JjWZmO1enKFTHqQTt4RfjMfzg2Ai7v0x2PsmE3f+Kn+OGqJ8zaNzQ/jCoKUfZBIXEI+f19CSgZ62gyXIZ28q7ifZsNEgpUIMkkJcGRR+u+5eZz++PE92+V3TYdDjkWTkkieRvx8vAgxWPr4dkR5D1qTPqSTd5b/1lvPX2clHu0PtsPc2cWOr/XHcXZEV9z4Kfz1tmvuLZFkYVJTFUb+edQkvW8D/Zr4v955Vnc8uHnFDLHLHUgn7ywLe8JDze64y/tN8w+b6TGnJMPP7eD2f6Z5SY/TZ8+xP9J3vSuZFgL/fOFpKyRPMUEjVluN5HTmVhyL9l02Sf+ii7H42MLX6zOlZwj+vvbpxWdGpJO3RPITmFEbbkRZLpOk0Z2e09K+Ou+eh0v/ms97ZLrbziG2T7aeH2818qJruRdt2x6JJAO4fNdUGz41VeGjpabBfHL4mLrDhuUL0bVW1lb5lE7ekMR4vbNd9QHcOQuznrVc3tyqmJ7z9ccz68HSl9WJ2BVvQdxN0/JaHO353r/kWPl9P+qPFUVdBhp7BdYMg4kalc+752FSOdjzP/i5rXOTuakGqxJSkhy/XiJJJ4ZKkqduPDTJrzt+k8nu1u1DI/g3iy6RtIV08ob8GAGTNRMshxfYLr/3B9O0ap31x7dPw/E/9fXN62havoDmfkcXO2brkzjHyqck6o93fquOzX9bHfb9AI81vfz9c9QfrnUj4NJuiHRiY5fhD9mBpzSmbdJjudrJg1y+qx/6XBZ5xShPURTuPEo0ShvRrgqlC+aiYJ4cGWJfRvP0OvkxAaZLHWM0mtH/jDctb+5LK9I03wjTMT6bPDHtadiF9sfDXh4bDNds/tx0dU9KEuyeaZx25YDjdt06rj++fth62cR49c8W9gw1Pb4Ht06Zpi/oCf/ONE035M45515rWi7uVof3xheD/T+paYnxsLC3OoRljoRY9XN4zMH301H+26CurhoTADumGOclJahPdfaiKHBylWee1FJN16/rsxQeJ6ZwOzoKUL+v647f4IWPJ8HdCwA8eKzuKcmN/odgwlozn5uMJCVJ/ey4iafXyVtju5kdqMlPTNPS9r797YvsDkChSvDsB1CxtWO2aTE35h1nZQu2rSWWkXNN0/wDHbFIdSS/dTNO2/4NJDxQnd2Ob+D8NjX9zjn4srj6B6rjiPoDTqzU2DNPdUhHfodJZVWnkpoK9y+r1yZremOHF6o/lP9rBN/XV8/jYtTVRooCZ9bDhk/U4alZjY1f5/Hl6hPRjFrwUwtju48vh0dphuPS7nSeWVcvY5GaCr+01Q/vrfkILmyHTWPgv7Uwo46+o5CSBBtGweP76hAZqE9XsVfVpx9LjuzhTfVHxNoPwq2TsHyQpq0uwem1qt0Le+j3Zmz+XH/fO+dgfmf1qS7xkXFdj+/D9q/VNh0ToN+Zvag3/N4HVryp/rBqnyp3fKN2kB7eMG/bvh/Vevb/pH+fAc5tUcX6QP2snN9qvg1uHFP3hfy3Xn2P75yDle+qu9BnNWbJxNd4ecwUav3dmt7e+hB+i/zGw/RQSIxn8f5L9PDeynH/VykvrgLQwOsEfFFUte3SXvX/uMLmv/MPrqlDusf+NO6gJD9R3z9D7l3Uf061bBqjttHd8/r23jha/ewcX26+3dKJyEyxU+vUqaMcOOCCHpU9aHvxI69CjjzqF/DzQAtlY1UnOTEIGrwNe77T5312X3/dGM043/Qw/ZfI4v01ZRNiYYJm2dYnN8HXznidWvvrvqbvNb64BCq1MS2bkqxumHKU/GXhvcP2l0/zZDT3sPoBHxDqZ1yuWA3rE9ptJ8K64epxidpw1TSQAwDP/wDL37BcT4VWcHajepyroH4OJbCM6b4GLfXegHzPwKbP1PMXl6o/DKcNVmrU6g+xl1Xn4ggdv4XyzdXVR1pyFjB+yjKkzVdqXqHKcGi++qNhjg5TYPUQ47T8ZeHeBcfsAyhcFQpVgPxB8O8Mx683pNlIuHkcTq6EOq+YDt+Nvqv+4I0vqp6Xi4DzBvFVB25VfzAW9YaWY1RHuvUrq7eckdyFwT4rdOe9E0ex2E8/3/VLjhd5+clCx15HlY5wahWEvAhHDK6t1BY6zYSVg9UfckO6/wJ/vOzYfUDvFxxECBGpKEods3nZ1sknPFCd3+bPodscqNFdn3f7LMzURGnp/B2E9VV7g19ZmEV/JxJ8csDUYPULVToc/heu5n18Xd8b1b5B9ux4NXwzteWHnFQdjD1or/n0jt6Bv7kbilYzLbvkJTjxl80q427k4N6ZXJR89h66eMOOfOjsdfISicQ8bnDy2XO4JukxTCilOniAZa/qH412f6d38AA7p8LNE5YdPKi9i6nB6vGRRZDXIFbjr887ZlvuwvCOhR+yaFMdDS0Jp0+T+lgzjrjjG32Gtw901ayc+V+4+bE9Oxw8wOVtBYi7mlM7nGmV2L9XEfPdd7YLSiQSj5JtnHzKgwcknNJMoIw3DZh7s2cVTlapCuvT7ES9c4bUGeHWK1/zke4w+Wwk2/dF6/Mu7zEpnuxdjAsbCpEUr2le31ykfHSNJy8dRvnoDEmp5nv6yqEFJKXoxyJTnzzh8fHjpMQ94kLnLlwdOhSAGxOnceOgOv7/ICEJZd1IfSW/tNUfb59s31OFGRLjvDm/vhBPLp/j8eHDpMbH8yQ5hTtx6jjltaFDuT1DP6HZc9ZuHousKeAkkWRnskWkCkVRONijD3kungWgWN1c5C2RwMXNBclZKJEClR9x91QetazGhwovUFLg8T1fLm4qTIlGd/HJkUp8jB+5iiSSs1AiQkDSI3Wnm2/uFO5fyMn1vfkpvKIPt2vkIf5WDh7dzEGppnfIU/wJ52PiENu3kLDAC/Dj7ErDHr/xk1SZZcu42E2dpHwmPCcBZR6T+t82zlavTsIzpXm+3rusXaH+uGxt1otmQNymzRwb3B/vs7kBSIrzYdHeQTx3wY+AsoHkLvqEa3vyU6jAdOLadSX5/dkIryIEBMUjvCDPMwmsf245nXc8B0BKoiA+xo+8JTQTTIrQpHtx60g+ntzz43wr/bLPjS36sjYxkJkph/Rpn0zg2d4d2Bd9F/8cj7m+P4D8FR+R8sSLa3tzUSREauVLJJ4kW4zJ7zp9kwKdm7ncnsI1HhATZf+KmeMFgqh+N9qpe/kXSCThrufGrvNXeMQ9zY+Hq1gee59cRRJ5/3kphSyR2IUckzdPwuq/3VKvIw4ecNrBAx518IDLHbyW+Fty0lUi8STZwsknLnGBbK9EIpFkQ9zu5IUQbYUQp4UQZ4UQI9xxj6D719xRrUQikWR53DrxKoTwBr4DWgFXgP1CiJWKopxw530lmYu4a9lTE0QicSW+eZJxxzfF3atr6gFnFUU5DyCEWAx0BqSTf4q4vN2J3bYSyVNGwaoPKeKGet3t5EsAhqpdV4D6hgWEEAOBgQClS5dGkv0IamVFU0cikQDg84Z9mxYdrtcttTqAoiizgdmgLqH0sDkSN5CzoA21Qm8/YylkgKDG4JsTzmxwj1Elaqs6NcsHuqd+eyhZF66YiSpWqLIqUw0gvCFfCSgRZnvncovRqqBY0mPs2rbsDG/ssD98Zb03VClrM1RJ+IVTo5up4nNZhWYfw9Yv7S8f1lf9HKfVV/LJCclmIsEFN0qffRZwt5O/CpQyOC+pSXMpcf45yZOQzvB5EreyP7USu1KDed9Ho6DY5ivImR+EgBo9VPnbguVVMbJWY8ErTbi1sQUhVbPe3scfev2mqhVq5ZFH31UFrC5sh/Uj4bnpqn5/u0mwdphapuXnqq5/pTbqsZc3+AfAol5qfsU2qobRpX8h4hMo01AVrnt8D6p1Uo9vn1Ed8DNhqnIjQPg7qvrjuc3Qd5mq1nh5r5pnuO457hbkyKfqIF07BCVqqQqOXt7q694xRa3jnX2qcJ1PTvBJswR151T19WsF3J6bBqUbQuFK6nnjD9X/igLzO6nt0W2OKu2hpVwzte0Amo6AbROg9yKo0l5VrBxbAIBjqUEEh7eDRzFqeMmavaFoMNQfBHVehYASqo23T8P3DdS03IVg20S13oiR0H4SjUbMpai4x0GlEj4kk5+HJJADchWAD07AqdWqgmtKoir2Vbqh+h68uBSilkDUUqjaSRU607Rp8sr38TloId6BVo8K4PAi9cc0ejuN/vDiOe/djGhTUX1PW3wGp/6GP15Ry1brAj3nqdpWB+ZAqXqwdACUbaK+nqBG0OhdtY1OrlRVOAEavQ9Vn1NFzPKVUOsRXpBbM0wZ0lsVCbx9GgJLQ468pjvRnVwfbw9u3QwlhPAB/gNaoDr3/cCLiqIcN1fe2c1Q37Z7mbYXTOUFJJ5leex9AEbu/5uaMy/wkFzUFaeoU6c+w7tZibhljnsX1Uhdi/tA26+gjkbhb0FPVU549D3wSrNYLOGBffLPyU/UyF0tRkP+MvbbpP2imvuC/thcddyOfHlTU1RH52uHPMSYAKjUDl60M9jMP+NVeeuaPXT3emP+PsLKFmFQ0/JGRVuPnEUOkohSyhE9oYOaGHsF8hQFb1/r97l3UZVtfnUjFCjLvUeJhI3baLbouS/b4+0lTDMSH8G1w6pTNSThASipXE/MwVcTv2C630z6Jo7kvpKbGCWQvc8/hvqWn8zeWXiQ6DuPWDXYzJNIYrz6ROmdpt+bEKt2BMyRmqIG1gl7yfTH2BbXDqsOf1JZ8MsLHzug528Gj6pQCiHaA1MBb+BnRVHMRORQcdbJf7fxFM0HOygUJnE7Wif/8fXrJkGVT4xtwxerT5KQmMKUXqHO3yTpsRqNKn+Q83U4y/Uj6g9PcDfTvKTHai89T2H33Dv2qiqfbK80tRm074nOkadJN5fn7D3MsWNYBKUK5HK6ziLc4xb5denptdUjnNsChSpCQPriyHp0x6uiKGsURamkKEp5aw4+PQxqUdkd1UpcwJ/lmwDQuGIho/Rqo9ezcO8l/jx0lZTUdHQ0fHN6xsEDFA8x7+BBtctdDh7UoZJ0OHhDTlzT6wulfS8SklJ49MQ1shSLBzagctG8uvOLd+L5futZkxB99mLo4Gf1rZVu+zxC+Yh0O3hbeHzi1RWYfeRzgN7tPiOkWB7e3ruAP1D13Ms+uM6FfMXpf2It3g5OYh0vEERA4iNKWojUlFiwCB9X7U7PWwep95/9w0y2tHFOB5ai8n39Yqa9Ravya9W2nA8ojoJg7V9DLV57udXzbG/xAt4//0C9Gyco9+C6Uf4/9WvRfO9BAJKeKYXvNf19ioTEkrNwIqe3PIN/mpBweapWAeDQpfsW730vPpFC2TS+ZmZl3TH9+7v1v1tUe0Yd1rp233huq8qn6wDHe8nTNp3h203/6c7fbFaeBuUKsv6DJrqeeN85e3X5bYKLkSeHqTu6fDeeEoE58bLwHc+SvfcMJls4eYBKB/bj5efHb2sOUmfEy+DtTZEPP8T3meKc2X+ML6K9+XLXbPDxgWR9z8QrTx7+ndCd3Dl8gI7UBx4mJHHh9iM658lB4s42JHz0HueLV6DRvFlcb90SAL+gIJY06kXSvr3kePs93g0J5PE/m1nkVw6/okXJUyAXFUvk4EyCFyNH/cKFfMX5pE9DetZR56G/u/2IYgFv8eHSI3Qtn5dahXyJW7qENvcq81vPKoSFlFeDgSgQU3YaT+7e51SpZ1kT84DPWpanww/7eL+cYLNfSfy2b6JT33Z0a6vq5EdevEeJwJzc2XuRVc0qUHX0OpP2+qDJYD6snZ8mrRvw+OBBqnbrRmvgbqspbDxxg3PXL1F22EDKtojBL28yVf1Xofx0m7vz5+MfXINL/fuTp8RjSjXWhxVcM2kxFfN6U7eIP9eSvZk+dBJvVFZ7s3FWeoP/3XhIoQrSyTtDaqpCuY/XsOC1+jSqUMj2BRoG/XZQdzxp3WnealaBUSui+G3PJbPlFUVBCPs6UzEPnxg5eICPWlt/2l649yIDmxjPDVy+G0/jSWqkKOnMnSfbOHnvPKqUcJ9O9bh6tA9F+rxAjvLqh6Z227ao0RM/QElMJP7gQS4NUCfu8rZqpXHwevL6+1KzZKB60rE1yeE7qVpQnSkPOHmC2GXLCHj+eYYocCeuF88EqhNlOfv04e00dlUFVsx538TeoEKqINh3L+ofM/OPGGa8S0yof0VeHgDAKIOsf798hjx+PjS88ZDnL91jYnh1XV7tMupj7IeaL9a+T1qAAvObLUekJPFhhxoYuv0cZfXL2Ark9qNX3dJAaeh0gtQlA/A6obaeOL+Zgq++iqIoFAmLJTBIH+PySeEafNJBH5WqLDCueyi+3rZHBG89NBNLU2IXL/6kPgn2+WlvuhyhtbFzgOuxCbrPuS0eJJgumTV82i6aLwc3Hxi/57fjEtNewn83nQxyLzEiWwiUGeLl5UWp0aN0Dj4tws+P3A0a4FtSHQcrPvZzm3X6FNTv2BRCENi9O8Lbmxw+3nZ/8J2iWmeobT5OZD5/X7y8BNWeycfpL9pRLMDy+GyRvP4UyefPR22r8GGHGg6Z4FXBIMD1oV/h0h7EoxgKVn6Edw79MFaOvr+bXGvo4NvXMA3koqVkfhlsxBF+3XORoBGrOX4tlj3nTePDPk5MYciSwySlpJKckmoUiMZZmk3eir2LNLxs9Pjz5zJdiXLlXrxJ2qvzrC/C8PVO3zDt00K2c/L2Un7jBqqcPIHwtbEkzJP0nA/PTfWsDdU6649PrYKf28DXFU3LBZSwWs13L9Zix7AIs3kHL90zmy4xz6crjgHQYbr5cJFVR6/jz4NXeXvBQSp8spaKnxgHmf77iG1Bv90jmxudJ6aksv74DbvsS7bxo/JuC9PPz/mYRzbr3XEmhoSkFN35/FfqWykt0fLUOnkhhN1jjE819qwztwMhBKUK5OLL502fJL5cc8ol98junL7xkMnr7W+rDSdu6o4v39X3lAcvOmSuuBHFA0yfrh4k2F5lk5SSSqtvt1st075GcZO0Uzcecj9eP2STmmaVz6YTN+k3Z59uIhggvLzURLKHp9bJSxzgvSPm09/eDy3HqNvX7eTF+lKfyFnaTN3Od1vOWczv+9Nei3naCcz0cPyq7Y1dL/7o2KbEz57Tz+OsOnqdQ5fuMX93NIlpngZem+/4/hmJSraZeJW4kQALjtk/AJ79IGNtecpITVVYe+yG1TkNLTvP3rY5VJKWAQ2DmPtvtFHarhHNzZY9bcdE6P5o+4bejo5pzZW7jyke4M/nf6vLDUZphqEAOoU8Y1c9EtvInrzENmnlArTkLZqxdjyFlPt4DW8vPMi3m87YVX7aZuvlfttzUXe8avCzRj1pUJcqltAsJhjbubpR3p7zd/l55wUAYh8nETRiNSevWw7U/segcMB0IxyoCweqPZOP/Ln1k7BlCup3v1qab9DSvIo7RHmzJ9LJSzKcXH7eJmlRV9wn0JQdmG7DeWuZ8c9Zi3lxT5KNesvBJQIQQnBqXFuz5fs1KMO4LsE0q6zfuTt21QmSUlIJ+VxVB203bQcADb/azGd/HTO6vk5QAf4YFM6PL5ndbW/CxTv6eYOr960LDub0Nf0MScwjnbzEPlqNVbfxu4AJ3WoCMKytfoPMczONe25BI1bbXLud3Vmy/7LtQkDNkhYEtNIQ/Nl6s+n+vt6827wCpdPoyAgh6NegDF/3MH7fw7/6x+h89dHrXItNYN7ui6SlTlAB/N3gkK/Y+BGQ6JFOXmIfjd6DWv3V47C+8Oltp6vqFPIMGz9owptNze9lMGSDncv23MHs7ec4euV+ht9Xux592LKjdpUv4YK9GkNaV2a7hSWuaTe03Y4z3sj09sKDpOXlRkF233vF25Z11Ee2q2I2fVp6RO2eMqSTl9iPVkipUGXbkrM2qFg0L0IIXn3WNGiE4fK5gb9GcuVePHcfme6IdCdxT5L5cs0pOs3claH3fZyYQtmRayw+xZz/sj3REzoY7W59PqwExfI5Jla2c7h5h26OgJyOv9efPVfddiENZTW7v81Rtbj5JbxBVq6RGCOdvMR+KrWBl/5Sg2S4iE/aVzVJK/fxGqPzZyduoZYFTXJ3EZ+oXxM+ws4etSv446B1RUZzQl25/Hy48SDBKO3UuLbMfbkuoztWMykPUDK/YxK/ec2Ih7kKa3Wfi4kD1KcVS716iXWkk5c4RrlmllfbOIGh07qVxlF5kvgn+p2Vi+0cG3cFn644ZrtQGmqVCTQ6z5PDB39fb5pVLsLLjYL4y8pwiL1Efd7G6rCKIaM6mP5wW8OSwiSoYmcr3m7EX+800q2oKZDbwQAdTznSyUsyDXcyeEjGGp6QWjB8enCEXH7GPeFjn7fRHQshCCkVmB6zdITaWc/zYdYlLsyx5I1ws+mdQp8htFQghfLkoEKRPIzqUJUNHzRxuP6nGenkJZmGC7et65e4O4qZIUOWGO/yvR+fSOxjGwHJ08H5mDie/+5fp6/fNKQplYvmzVBJ3vKFzY+LF3QiNkCt0oG6Y8O1+4ZBRoQQvNa4nIw94CDSyUsyDW8tOMi/5yyv2ik7cg1bTt1yux0fLTWVcQgdu5GQzzdw5PJ9t9yz+TfbLO4o3ajpuRquVwd13P24ptdeoUge1lvp4Q5sUk53/H5LMwJzdrJpSFP+HdGc9e83YfOHzZyuJy0+Bit4utfWR0qS+lLpR8oaSDIVL/6o11+pV7YA+y4YS+m+PHc/dcrk5483G5q9fn/0XXL6ehNcwr614+b4w0o4us7f7cqQ3rKvtyApRWH9+02oWDQvhz41jXvgyPrzhuULMnv7eSZ3r2nkRB2lQpE8Tl9ri9n9auPr40Vef19Ojm3LIyeHryTGyJ68xOM0KFfAbPrvAxuYTT9w0fJ4eY9Zu+k4Yye7z91xyhZzuuZpsRTzdOSfUWw5rT5pHIi+y3MzdjqsJaNl05CmRE/oQOVi6nBF/tx++Pk4/3VtVrkIWz5qRo86pTJt77h19WJEVFYnV3P6ecthGRchnbzE40zqZn4nrRCCVYOfpYUTOiUv/LiHoBGrTSRrbXHprm0nb6hpriX2cRKL9l3i5V/2A9B91m6irsay5bT5OL+2sBV4wxmsrUd3lpkvhgF6nZuhbayH+ZNkPNLJSzxO6YKW12wHlwhgzoC6dtVzz8zqnLSStbYwHC4C8+JaR83o7Gi1XACeJBsEttgd7dD9teTzz8TBbAzoWPMZoid04KXwIDYNacJbzWzvYpZkLNLJSzIlhqstQJW/Tdu5vfUggaARq/lll6qMOH7NSZN6/jx41Wkb3mhSjiGtKpmkLzt4haARqzl7S92ok3bVT7f/6VfJ2NMjN/xR0JLXP+tNl1UokjfTDgU9zUgnL8mU/PmW8cabEoE5ufCVfsJTURTqfbkZQKdHXqaA6RPB3UfOBQkvEZiTke2rElY6P9/2Mh5OWnX0OgAtp2wD1FU/hhy7qpfftRWkPDE5lcqj9NGOShXISfSEDlY3CEkkjiCdvCRLonWwWoJGrOZevOk6dksbgRRFYeK6UxaXRC4dpN+c83xYSYsrah4mWF87b01vHeC5GcbqmzuGmQ/YIZE4i3TykkxBkbzqSorFAxvYtUTxnJnAzz9rhm0MmW9G/hbg7qNE/rf1HJ2/My9AprXHkB5mlh7WGLPBJC0tPX/YbVbN8taDBLuiLUkk6SHrDfxJsiW7R7bgyr14yhR07QqQjQbBrEEd/zYcHrGEj7dp/ye4RABLrayht8S+C3fpNFO/vl5RFF6fH0lYmnmH7/vUcrhuicQWsicvyRR4ewmXOfgz49tx6NNWZvNGLIsySdNOfB6zEahau2bdEo0qFLTLvs0nb7Hp5E0mrz9tlO6OZZMSiXTykiyFPUqIvt5eBOYyXoI4btUJXvp5n9kgJD1n7QZg6QHrapMNyll34mM7B1vNDx2rDu1o5XPT0qKqjFsqcT1yuEaSpbBXCTHtUr45O03H67UcuRJL9O1HZsPXOUL5wnn47sVaVCmel8CcvtT+YpNR/n3NxPBXa08ZpVcumteq7oxEkh7S1ZMXQkwWQpwSQhwVQiwXQgQa5I0UQpwVQpwWQrSxUo1E4jIqFTXVVrEnSHizr7fqjme8EOb0/TvULE75wnkomCcHC1+rb9c1f75lXodHInEF6R2u2QgEK4pSE/gPGAkghKgG9AaqA22B74UQMry6xCWcHd9Od7z34xZEjmqpO//vpulQSNog4ba4eMe65LG9mBNUMDfu745A1xKJlnQ5eUVRNiiKolVr2gNo15h1BhYrivJEUZQLwFmgXnruJZFoMVz5UjSfv9MTloXymI8w1M0OlcYJXWsYnYebGa+/F28qs9Axzbr4Ee2q4C03PjnM6bunqTGvBodvHfa0KZkeV47JvwL8rjkuger0tVzRpEkkLuHHl+roglYYTrLaG9A6ekIHFEUx2a0KkNNKz3rfxy24FptAaKlAqj2TTxfou2NIcZOy7YKLk8vvKPGJprIFWnKkQ1nyaab7390B6Le2Hwf7HcTXK2to/XgCm58wIcQmIcQxM3+dDcp8AiQDCxw1QAgxUAhxQAhxICbGOcU+ydNHq2pFKVdYHX8XQrB7ZHOaVynC5g+b6srs/6Slpct11y170zTsXGAuyzFEi+Tz103+Gmqrv1ivtElZby/BibFt+Z+V9e+tqxezaqPENrV+lfsLrGGzJ68oitVvihBiANARaKHolZquAqUMipXUpJmrfzYwG6BOnToZF99Nkq0oHpCTn9OoVRY2s2sVjCdW0wYX+WOQ+Vij5sjl58OekS0okNvPqjBXuxrFaVWtqNHGrAblCrB4oP33kkicJb2ra9oCw4BOiqIYCnGvBHoLIXIIIcoCFYF96bmXROIqIgz06XP4eDOpW03deZ0g8wFMLFEswN+uYB6G9wCkg5dkGOkdEJwJ5AU2CiEOCyFmASiKchxYApwA1gFvK4pieWBSInET5qSCc/sZj7n3rFuKgrn93BraLn9uy0NAEok7SdfEq6IoFazkjQfGp6d+iSS99Kpbiikb/zNKMze0EmlBBkGSNYhLjCOPn/t+pLMycserJFtTNJ8/kaNaEpDTl+PXHnDmlnlJAUnWJnxROFH9TXWJJNLJS54CCmoCQoeUCrSoLy/JGqQqqYTMNx8TWGIeuUhXIskgtEs6R7Sr4mFLsi4PE6X+vqPInrxEkkEUzpvDroAoElMURUFB4b97xvMrbYPasi7adnyApxnZk5dIJJmemvNrEjI/hFfWv2KU/sWzX+iOa8yrwW8nftOdJ6aYyko8jUgnL5FIsiT7+uwjh3cOcvrk1KVN3D8RRVGYfXQ2tX+rzYZo2+EZszvSyUskkiyJ1rk/Tn5slD5uzzhmHJoBwIfbPsxwuzIb0slLJJJMx4B1A/h4x8cW8//p8Y/FvKX/LdUdlwso51K7siLSyUskBsjVG5mDyJuR/H3+b9r/2Z4a82qY5BfOVVh3/EWjL0zytZyPPe8W+7IS0slLJBqGbB1Cw0UNOXHnhKdNeap5lKQP2nL5oWnc3bENxxqdNykpQydaQy6hlEjAqLe46vwqgvIFkZiSSKB/oOeMekppsLCBxTxzu1pz+eZypzlZHunkJU8tR2KO0HdNX6oWqGqU/uuJX/n1xK+AeacicR8PEh84fI2flxR/s4YcrpE8lSiKQt81fQE4efekh62RaDl155TD11jT8nc3Nx/dRB9GI3MinbzkqeRJyhO7yqUqqW62RGJI9INop66L6h9l8akrPinebHp6OXnnJC3/aMmcY3PcUr+rkE5e8lShKApTDkyxe/30rfhbbrZIYkjZgLIW84rnNo2jaw/t/mxns8wPR37g5B3Hnug++/czAKYdnOaUXRmFdPKSp4rtV7bzy/Ff2H5lu13lj8QccbNFEkME6tDLT61/onxAeaO8b5t9a3c9/2v5P93x3YS7VsveS7jHzMMz6bmqJ/FJ8Xx3+Du7JBFcNcwXFRNFjXk13PZZk05e8tRw7PYx3vnnHYeuiX0S6yZrJOaIT1aHVnL75mZik4m69B9b/0j1QtVtXr/7hd3sfmE3z5Z4Fh9h37qSJr/rl2D+FPUTs47MYtaRWVaviY6Ntqtue3hxzYsAujkiVyOdvOSp4YXVLzh8TUJyghsskVhCO36eyycXFfNX1KU3KG55WaUhefzy6CJEda/U3eH7/xj1o9F/S/x28jer+ZkJ6eQl2Z7ElESzuyYNqV20Nv++8K9J+uQDk91lll3ceXznqdq1ee3RNUBd++4l0uee3glz7KktLdae4n4//Xu66s5IpJOXZHuO3zlus0ytIrXI65eXeW3nZYBF9tNsSTM6r+isOz986zCvrX/NRJQrO/Ag8QHfRqrj7j5e6lDL/j772dpzq1P1BeQI0B3fT7hvtkxSapLF6/fd2MeVh1ecundmQjp5SbZHq0hojVpFawEQWiQUHy8fRtUf5W6zrKIoitmnj35r+7H3xl4WnlzoAavcS6NFjXTHBfwLAODv40/BnAXTXbc5qQpFUaj1ay2L10zeP5l2f7bTSSt89u9ndF3Zld3XdpuU/fea6VOgPRhKOHSp0MWpOmwhnbwk23Pu/jmj82+afsPrNV6nXrF61C9eH4BiuYoB4CW8ONTvEL2q9MpwOw2Zf2K+1fypB6dmjCEeIr1DNWnJ6ZvTJM3Wapbrj64DcOPRDQD+PPMnZ+6dYeDGgSZl39j4hlN2GUo4pI165SqkrIEk25N2CV3roNa0DmoNQFxiHP9e+5cK+StYvP6V9a/wc5uf3WpjWr4+8LXRuaIoHt3Z6W5i4mPcUu/o8NGM3T1WtzTTkH5r+9lVh7+3v6vNAtTPniG/tXPPZK7syUuyPHce33FK8wTU1Rhah2+J/Tf2G53fTbjL1stb7b5HQnKCQ6t0zJWtOb+mzcnjrEzzpc3dUu/5++qk9aFbh6yWM1y981fnv4zydlzdYde93tj4Bi+tfclqmcSURFJSU4h9Ekv4onBduo/wwdfb1677OIp08pIsT7MlzYzGc63xe8f0r4po+ntTBv8z2O7Jz7oL6lJ3QV27yiYkJ9hdNrs6fVeKwrUs0xKAqgWNRejWR683Og/MEag7DgoIMsqzNN6+sftG2pdtb1TO2o/J8dvHqf1bbUJ/DeXZxc8a5SUryRavSy/SyUuyDb+fMu/Am5dSe4kbum2gWsFqTtVtTv/k4oOLTtVljbQO/rPwz1x+j6eRozFHdcfxSfF8tO0jo3zDdk47H2Bp7L5Y7mKMrDfSbhumH5pud1lXIp28JEtjKCD2xV7zEYL+uayGiiuexzHtk/Di+sfp+gvr89uJ33hhlX5D1eJTi/l89+dWBbDSK3Bmz/LPrI47n0i0PWvDFVZpV9oc7HdQt4HKUQL9A02kqrUoisKWS1tosbQFq8+vdnoFTnqRE6+SLI21dc4AB28edLrud8LeYfd1/XK5ifsnGuUvO7MMgEM3D7GiywqzdXy41flA0mFFwvjjvz+cvj4rMihkkEvrq1O0ju740oNLlM5XmtP3ThuV8fVSx8Ij+0biLbwB2PXCLqMhwLRywobj9mk1bE7fPU33v7vTq3Iv3aapETtGWLXzj+fc9z7LnrwkS2Ntvfi6C+vov66/03XXLFzTrnLnYs9ZzNt0aZPT9+9YrqPRmC+oW/W1G7bMrRgxJCE5gWkHp2WpjVNvh77t0voqBOpXTXVY3gGACfsmmC3r5+2Ht5fq5PP55aNorqK6vHtP7hmVLRdoOUB4979VOQV7d8VG9Y+icoHKdpV1BunkJVmaKZFTjM6P31aHN55b/hxDtw9Nd/329rDuPL4DqD2+C7EXmLBvgslQTf+1/W0KXxnSonQLkzH5z8I/023cUrAerGL83vH8FPUT9RbUs/uenkCrNvnls1+6vO7cvrmdvnZtt7W646a/N9Udpx2HN/whyYxIJy/JViw4uQBFUZwOPpEWe3tYzZY0A6DH3z3otKITC04u4ELsBaMyB28d5LvD33Ev4Z6ZGoyJ6h9FwZwFdfFLXw5+mYN9TYeelpxeYrGOFWdX6I53X9ttc2jLU2ifhJ4r/5zL6067t6DfGv3a+FZlWjG/neVNZ9phnLS8WPVFo/Ovm35ttpw91Cpiecetq3CJkxdCfCiEUIQQhTTnQggxXQhxVghxVAjh/lcikQDJqckuH56wtzeoKIrReK/hig5DDKVt7SGqfxRDag8xu4563J5xdtUxcONAvt7vvDPKyixov0B3fDjmsO54SrMphBUJc6iuAdUHmKSVDyxPs1LNnLLNWpAUV5FuJy+EKAW0Bi4ZJLcDKmr+BgL/M3OpROIyqhdUtcbPxp41u7Rx74t7na57z4t77OpxGToQgBvxN2xec/ruaWrMq8EXe75g19VddttkOKFoSI15NSyuVll4KvPp3VyNu+r2exTJVcRldb0V+pbZ9GF1h1m8ZkqzKRbz8uXIl26bbOGKnvy3wDAwGiDsDMxXVPYAgUII52J3SSR2oB0XPXPvjNn1yNphD2cxnIT9p8c/NC3Z1KTMpovGk6zfH/7eZr2Gk3SDNtm/sqRHpR5G5wnJCVyLu2b39ZmFtsvauv0exXIXM0nz8/Kz69ovGhkvy83pY6qBA1AqbymLm7halWnFkZeOENU/iqkRU43yIkpF2GVHekiXkxdCdAauKoqSdrdACeCywfkVTZq5OgYKIQ4IIQ7ExLhHv0KSPTFc1tYmqI3u2NJYanroX11dpZPTJyeFcxWmXzVT3RNbomKupF1Z47ilz//1PG2W6dtg7/W9ZoNmTN7vWX18T9GtYjej88RU2+H9ALM/5s6g3WDVonQLdvbeqUt3ly6O0b1tFRBCbBJCHDPz1xn4GBidHgMURZmtKEodRVHqFC5cOD1VSZ4yas7X967rFtPvFN1yeYvL71UoZyF29t6pG/aplL8SAIPDBtO1Yleb1//S5hdCC4fqzmPiY8zK39qLEEK3FT/2SSxX4ox1z1/b8BpPkp+YXDf/xHwWnVrE3YS7hP0axqm7p5y2wZW8V+s9t9Y/puEYp65zdpOUltdrvG6SZqhzX6VAlXTVbw82nbyiKC0VRQlO+wecB8oCR4QQ0UBJ4KAQohhwFShlUE1JTZpE4hLSLk/099H3iLS9+h9a/gDAr+1+dck9A3IE6FZr5PfPz5GXjvB6jdd5o6Ztmdk6xerwa3u9Hc2XNqfXKvNyxof7HbbLHq3melodFC1aqdy0fLn3S5r+3pTk1GR6/N3DbJmMpqB/+jXjHUErMW0LbfASezn60lE299isO7e2nh5MV/+4A6eHaxRFiVIUpYiiKEGKogShDsnUUhTlBrASeEmzyqYBEKsoivlPnCTb48xY8e3Ht7kep35kIm9GmgT+CJkfYvFarfhUQI4AovpHEVok1OH724OX8EIIQfHcrp1u0m7IsUXl/NaXdx64ecAV5rgNwyeZ2kVrZ+i9e1fubXfZUnlL2S6kQQhBkVxFaFJSXUHVonQLs+XK5CtD36ruCdydFnfJGqwB2gNngXjgZTfdR5LJWXdhHd8s+4YuFbowsMZAo7Xf1ohYok5I5fLJRXyyqg3To1IPs5NoltAOqbibtL2xqP5RRitc3DXueupe5hhqcRbDJ5nS+Upn2H3z58ivU6e0h68af0XfNX0dcsrftfjOav6q51fZXVd6cdlmKE2P/rbmWFEU5W1FUcorilJDUZTM3aWQuI2dV9VJphVnV9B+eXvqL6xPjXk1rAanNtRv1zp4gFZ/tDIr+LWx+0az9bhLn9scPkLtL81uNRuAknlK6vJWd12tOx4dnq4pLCNux9+2u2x6lpBmN+yZQzEkpHAIc1rPYUidIW6yyL1IgTKJR+i8orPFJWevrH/F4nUjdoygX1X9ypZ9ffZZXNaWkRx6yVhH3HBLvCGWAkprea3Ga3bf85e2v+iWYFoip09O9vXZZ3ed2Zm3Qt/i+8Pf07ea48Mk9YpnbmkIa0hZA0mWYt2Fdcw9Pld3bujgywVYn+TKDBiurDCkV+Ve+Hj5OLTKJK3kwvZe24nsG2mU9lXjr3THQfmCzNaTlQTM0sObIW8S1T+KQjkLedqUDEU6eYnHSE51PBqOgmIx1N+Exnp1wf7VnFefdCddKnQxSetWsRujGoziUD/rIerMsfuF3XzX4jvGNRpHfv/8+Hn7cfQlvZyC4cTfkufM69ysu7DO4fuml5N3TtouJHEJ0slL3Max28es5qcNZGwve67vMZtuGOLtwzrO67i7E8ONWpu6byIoXxCf1P/E6fry+OWhSckmRj8eQgh8vXxNdFG0Tz1pd3uO/td18wT2cOjWIXqu6qk7z6hVJk8rckxe4jYWn1psNT9FSTFJS0qxrZQ4KGSQTcnejFh/7AxCCFqWbkmbsm0omrsofz//t1vuc7Cf+WAphvMgnooRG/skVnccmCOQ4fWGe8SOpwXZk5dkKJu66/VdHiY+NMk3t+rmk/qfsL3Xdt35yrMrLda/tutao3HozMi3Ed/SNsj9mi22WNRhkUfue+ORXrjt/pP7HrHhaSJb9ORP3z3NynOWv/habAVZMCqr2F/WrvrsvLe993XHa3GkTmcpmrso9YvXZ+/1vXx/5HsmNZlklP/fvf8AeK7cc/x9Xu3lBhcKJr9/fl2Za4/UzVXmdLxL5i1JybwlTdIlppQPLK87/uHID7wRYnvnriOkKqlM2j+J7hW7UyG/PrCG4Xc1r29el95TYkq2cPJXHl6xOxamvY/xtkKrOVXW7mJ22uji1+LQa07HcEg+P1Ve1ZyQ2G8nfwOMw61p426mpXWZ1k7bIIEc3jl0xzMPz6R+8fou3R1849ENFpxcwIKTC3TDRHuu7yHqtn7IyNJSU4nryBZOvkWZFuwtIzd7ZDZKbjTfo34r5C02XtxI+DPhJnnare4342/q0iztXM2s4+5ZBa0yopb3t7zP1l5bXVa/4aa22CexBOQI4PUNesGutV3XWlxSKnEdckxe4nZ+bvMzoI9wH+gfCMCDJ+aXQgKkpKYQ1T+KqP5Rdmu5SBxnbMOxuuM7CXdcWveoXaN0x+ZE1Fx9P4l5pJOXuI2+1frSt1pf6harS1T/KJ0inzac3lf7LE+Qflz/Y5M0w6WGnzb41MXWPp2EFLEs9OZuahaqabuQJN1IJy9xG1UKVDGrl224S9WSnrm5IZr25drrjntW7mmSL3GccgHljATUUlJNl7WmZeK+iUTejKTGvBrUW6Df7q8oitXds4kpxoE65HBbxiCdvMSjWNIz1wbEMCSnt+c1arIj+/vqx87f3vy2Sf63kd/y5d4vAZh7bC6/nfyNAesGAMaSCN8c+IZ6C+pxLe4av5/63aSe2r9lrJywRCVbTLxKsjYzDs1gcNhgozRzvTxfb186le9Eh7IdMsq0p45d14yDiV+Pu87Px9Q5lQHVB/BN5DcWr513Yh6AURhCieeRPXmJx5l9dLbdZcc/O56GJRq60RrJF3u+0O2taL1Mv0zVmvO2Jh1tjuoFqztnnMRhpJOXZCrKB5SnVZlWnjbjqeb3078bxc+1xZOUJ3Re0dmhezgbc1XiONLJSzIFx28fByAhJSFT6MNL7GfjRfNBWwC+fPZLk7TnKzyfIQGsJSrSyUsyBb1X96bGvBpcjbtql0SFxLWkJ3LUyB0jLeZ1KGc8f7L7hd2MbTTWQmmJO5BOXuIRDAXHJJ4nl28ufmz9o1Ha5P2T01Xn+GfH4yW82NZrGwAru6wkj1+edNUpcRzp5CUeIb9/fqL6R0n9mUxEg+INjM7nn5hvtlxYkTCbdX3V+Cs6le8EQAH/AkT1jzLRt5dkDNLJSzyKuZ2tEs/Rq3Ivs+kVAvUqktMiptG0ZFOr9XQs19GldkmcRzp5iUcpmLOgSdrL1V/2gCUSgFENRplNX955Obl8cvF5w8/J75+fmS1msrnHZqMyzUo1M1K2lGQO5GYoicf5X8v/8eamN3XnH9T+wIPWSKZFTOO9LfqA4tre/d4+xpOzRXIVMTqf0XwG8UnxxCfHu99Iid1IJy/xOM+WeJYtPbeQyycXuXxzedqcp560yxvL5Ctj97W5fOV7mNmQTl6SKSiUs5CnTZBo8PPWB/puF9SObhW7edAaSXqRTl4ikRhhGLFrUtNJVkrqWdh+obvMkaQT6eQlEokRjkRr+qbpN5y5f4YahWu40SJJepBOXiKRmDCr5SwK5ypss1zroNa0Ru51yMxIJy+RSExoVKKRp02QuAi5Tl4ikUiyMel28kKIwUKIU0KI40KISQbpI4UQZ4UQp4UQMoqARCKReIB0DdcIISKAzkCIoihPhBBFNOnVgN5AdeAZYJMQopKiKLYDSEokEonEZaS3J/8mMEFRlCcAiqLc0qR3BhYrivJEUZQLwFmgnoU6JBKJROIm0uvkKwGNhRB7hRDbhBB1NeklgMsG5a5o0kwQQgwUQhwQQhyIiYlJpzkSiUQiMcTmcI0QYhNQzEzWJ5rrCwANgLrAEiFEOUcMUBRlNjAboE6dOooj10okEonEOkIbsNepi4VYB0xUFGWL5vwcqsN/DUBRlK806euBMYqi7LZRXwxw0UlzCgG3nbw2o5A2pp/Mbh9kfhszu32Q+W3MbPaVURTF7MaG9K6TXwFEAFuEEJUAP9QXvhJYKISYgjrxWhHYZ6syS0bagxDigKIodZy9PiOQNqafzG4fZH4bM7t9kPltzOz2GZJeJ/8z8LMQ4hiQCPRX1EeD40KIJcAJIBl4W66skUgkkownXU5eUZREoK+FvPHA+PTUL5FIJJL0kZ12vM72tAF2IG1MP5ndPsj8NmZ2+yDz25jZ7dORrolXiUQikWRuslNPXiKRSCRpkE5eIpFIsjHZwskLIdpqhNDOCiFGuPlepYQQW4QQJzSibO9p0gsIITYKIc5o/ufXpAshxHSNbUeFELUM6uqvKX9GCNHfIL22ECJKc810IYRwwk5vIcQhIcQqzXlZzc7ks0KI34UQfpr0HJrzs5r8IIM6zIrMuaK9hRCBQog/NOJ2J4UQ4ZmwDT/QvMfHhBCLhBD+nm5HIcTPQohbmhVt2jS3t5ule9hp32TN+3xUCLFcCBHobNs40/722GiQ96EQQhFCFPJUG7ocRVGy9B/gDZwDyqGu0z8CVHPj/YoDtTTHeYH/gGrAJGCEJn0E6iYxgPbAWkCgbhTbq0kvAJzX/M+vOc6vydunKSs017Zzws4hwEJgleZ8CdBbczwLeFNz/BYwS3PcG/hdc1xN05Y5gLKaNvZ2VXsD84DXNMd+QGBmakNUGY4LQE6D9hvg6XYEmgC1gGMGaW5vN0v3sNO+1oCP5niigX0Ot42j7W+vjZr0UsB61A2ZhTzVhi73WRlxE7e+AAgH1hucjwRGZuD9/wJaAaeB4pq04sBpzfEPwAsG5U9r8l8AfjBI/0GTVhw4ZZBuVM5Om0oCm4HmwCrNh+22wRdN12aaD3W45thHU06kbUdtOVe0NxCA6kBFmvTM1IZa/aUCmnZZBbTJDO0IBGHsRN3ebpbuYY99afKeBxaYe8222saZz7EjNgJ/ACFANHon75E2dOVfdhiusVsMzdVoHgnDgL1AUUVRrmuybgBFbdhnLf2KmXRHmAoMA1I15wWB+4qiJJupU2eHJj9WU95Rux2hLBAD/CLUIaWfhBC5yURtqCjKVeBr4BJwHbVdIslc7aglI9rN0j0c5RXU3q0z9jnzObYLIURn4KqiKEfSZGXGNnSI7ODkPYIQIg+wDHhfUZQHhnmK+lPtkbWpQoiOwC1FUSI9cX878UF9XP6foihhwCPUx1cdnmxDAM14aWfUH6RngNxAW0/ZYy8Z0W7O3kMI8QnqDvgFLjcqHQghcgEfA6Mz6p4Z+fnODk7+KupYmpaSmjS3IYTwRXXwCxRF+VOTfFMIUVyTXxzQautbss9aekkz6fbSCOgkhIgGFqMO2UwDAoUQ2h3OhnXq7NDkBwB3nLDbEa4AVxRF2as5/wPV6WeWNgRoCVxQFCVGUZQk4E/Uts1M7aglI9rN0j3sQggxAOgI9NE4OGfsu4Pj7W8P5VF/zI9ovjclgYNCiGJO2Oi2NnSajBgTcucfaq/wPOqbpJ2kqe7G+wlgPjA1TfpkjCdVJmmOO2A8cbNPk14AdVw6v+bvAlBAk5d24qa9k7Y2Qz/xuhTjCau3NMdvYzxhtURzXB3jSbHzqBNiLmlvYAdQWXM8RtN+maYNgfrAcSCXpo55wODM0I6Yjsm7vd0s3cNO+9qi6lgVTlPO4bZxtP3ttTFNXjT6MXmPtKEr/9x+gwx5EeoM+H+oM/KfuPlez6I+Zh0FDmv+2qOO/20GzgCbDN5wAXynsS0KqGNQ1yuoUbPOAi8bpNcBjmmumYmVCSQbtjZD7+TLaT58ZzVflByadH/N+VlNfjmD6z/R2HAag9UprmhvIBQ4oGnHFZovSqZqQ+Bz4JSmnl9RnZFH2xFYhDpHkIT6RPRqRrSbpXvYad9Z1PHrw5q/Wc62jTPtb4+NafKj0Tv5DG9DV/9JWQOJRCLJxmSHMXmJRCKRWEA6eYlEIsnGSCcvkUgk2Rjp5CUSiSQbI528RCKRZGOkk5dIJJJsjHTyEolEko35P21wBoB6JcjGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(generate_action_data_two().T)\n",
    "plt.axvline(x=50_000, alpha=0.5, color=\"black\")\n",
    "y_b, y_t = plt.ylim()\n",
    "plt.ylim(min(-50, y_b), max(50, y_t))\n",
    "plt.legend([\"Target 1: Distractor 1 → Drifter 1\",\n",
    "            \"Target 2: Drifter → Distractor\",\n",
    "            \"Target 3: Constant → Drifter 2\",\n",
    "            \"Target 4: Distractor 2 → Constant\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "The setup is the same as the original notebook except for the hyperparameters in the Choose Reward cell some minor changes in the last two cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Autostep():\n",
    "    def __init__(self, alpha=1.0, n=1.0, h=0.0, k=0.1):\n",
    "        self.alpha = alpha\n",
    "        self.n = n\n",
    "        self.h = h\n",
    "        self.k = k\n",
    "\n",
    "    def update(self, delta):\n",
    "        self.n = max(np.abs(delta * self.h), \n",
    "                     self.n + (1.0/10000.0) * self.alpha * (np.abs(delta * self.h) - self.n))\n",
    "\n",
    "        self.alpha = min(self.alpha * np.exp(self.k * delta * self.h / self.n), 0.5)\n",
    "        self.h = self.h * (1 - self.alpha) + (self.alpha * delta)\n",
    "        return self.alpha, self.h, self.n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MovingAverage:\n",
    "\n",
    "    def __init__(self, beta=0.001, one=0.0, method=\"window\"):\n",
    "        self.avg = 0.0\n",
    "\n",
    "        # for time based moving average\n",
    "        self.t = 0\n",
    "\n",
    "        # for windowed moving average\n",
    "        self.one = one\n",
    "        self.beta = beta\n",
    "\n",
    "        self.method = method\n",
    "\n",
    "    def update(self, x):\n",
    "\n",
    "        if self.method == \"window\":\n",
    "            self.one = ((1.0 - self.beta) * self.one) + (self.beta * 1.0)\n",
    "            rate = self.beta / self.one\n",
    "            self.avg = ((1.0 - rate) * self.avg) + (rate * x)\n",
    "\n",
    "        if self.method == \"time\":\n",
    "            self.t += 1\n",
    "            self.avg += (x - self.avg) / self.t\n",
    "\n",
    "        return self.avg\n",
    "\n",
    "class IncrementalVariance:\n",
    "\n",
    "    def __init__(self, beta=0.001, one=0.0, method=\"window\"):\n",
    "        self.sample_mean = MovingAverage(beta=beta, one=one, method=method)\n",
    "        self.moving_variance = MovingAverage(beta=beta, one=one, method=method)\n",
    "        self.type = method\n",
    "        self.differences = []\n",
    "\n",
    "\n",
    "    def update(self, x):\n",
    "\n",
    "        old_mean = self.sample_mean.avg\n",
    "\n",
    "        # update mean\n",
    "        self.sample_mean.update(x)\n",
    "\n",
    "        # update variance\n",
    "        var_sample = (x - old_mean) * (x - self.sample_mean.avg)\n",
    "\n",
    "        self.moving_variance.update(var_sample)\n",
    "\n",
    "        return self.moving_variance.avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    return np.exp(x) / np.sum(np.exp(x))\n",
    "\n",
    "def select_action(probs):\n",
    "    return np.random.choice(4, p=probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class IntrinsicReward:\n",
    "    def __init__(self):\n",
    "        self.prediction = 0.0\n",
    "        self.stepsize = 0.0\n",
    "        self.autostep_learner = None\n",
    "        self.is_introspective = None\n",
    "\n",
    "    def update_prediction(self, x):\n",
    "        delta = x - self.prediction\n",
    "\n",
    "        if self.is_introspective:\n",
    "            self.autostep_learner.update(delta)\n",
    "            stepsize = self.autostep_learner.alpha\n",
    "        else:\n",
    "            stepsize = self.stepsize\n",
    "\n",
    "        self.prediction += stepsize * delta\n",
    "\n",
    "        return delta\n",
    "\n",
    "class ErrorDerivative(IntrinsicReward):\n",
    "    def __init__(self, stepsize=0.03125, is_introspective=False, **kwargs):\n",
    "        self.stepsize = stepsize\n",
    "        self.is_introspective = is_introspective\n",
    "        self.eta = kwargs.get(\"eta\", 1000)\n",
    "        self.tau = kwargs.get(\"tau\", 100)\n",
    "        self.squared_errors = []\n",
    "        self.prediction = 0.0\n",
    "\n",
    "        if self.is_introspective:\n",
    "            self.autostep_learner = Autostep()\n",
    "\n",
    "    def update(self, x):\n",
    "\n",
    "        delta = self.update_prediction(x)\n",
    "\n",
    "        self.squared_errors.append(np.square(delta))\n",
    "        length = len(self.squared_errors)     \n",
    "\n",
    "        if length > (self.eta + self.tau + 1):\n",
    "            first = (1.0 / (self.eta + 1)) * np.sum(self.squared_errors[-(self.eta+self.tau+1):-self.tau])\n",
    "            second = (1.0 / (self.eta + 1)) * np.sum(self.squared_errors[-(self.eta+1):])\n",
    "            return np.abs(first - second)\n",
    "        else:\n",
    "            return 0.0\n",
    "\n",
    "class ExpectedError(IntrinsicReward):\n",
    "    def __init__(self, stepsize=0.0625, is_introspective=False, **kwargs):\n",
    "        self.stepsize = stepsize\n",
    "        self.is_introspective = is_introspective\n",
    "        self.one = 0.0\n",
    "        self.beta = kwargs.get(\"beta\", 0.01)\n",
    "        self.delta_bar = 0.0\n",
    "        self.prediction = 0.0\n",
    "\n",
    "        if self.is_introspective:\n",
    "            self.autostep_learner = Autostep()\n",
    "\n",
    "    def update(self, x):\n",
    "        delta = self.update_prediction(x)\n",
    "\n",
    "        self.one = (1.0 - self.beta) * self.one + self.beta * 1.0\n",
    "        rate = self.beta / self.one\n",
    "        self.delta_bar = (1.0 - rate) * self.delta_bar + rate * delta\n",
    "\n",
    "        return np.abs(self.delta_bar)\n",
    "\n",
    "class StepSizeChange(IntrinsicReward):\n",
    "    def __init__(self, stepsize=0.1, is_introspective=False, **kwargs):\n",
    "        super().__init__()\n",
    "        self.previous_stepsize = 0.0\n",
    "        self.is_introspective = is_introspective\n",
    "\n",
    "        if self.is_introspective:\n",
    "            self.autostep_learner = Autostep()\n",
    "\n",
    "    def update(self, x):\n",
    "        delta = self.update_prediction(x)\n",
    "\n",
    "        if self.is_introspective:\n",
    "            stepsize = self.autostep_learner.alpha\n",
    "        else:\n",
    "            stepsize = 0.0\n",
    "\n",
    "        stepsize_change = np.abs(self.previous_stepsize - stepsize)\n",
    "        self.previous_stepsize = stepsize\n",
    "\n",
    "        return stepsize_change\n",
    "\n",
    "class ErrorReduction(IntrinsicReward):\n",
    "    def __init__(self, stepsize=0.1, is_introspective=False, **kwargs):\n",
    "        super().__init__()\n",
    "        self.previous_error = 0.0\n",
    "\n",
    "        self.is_introspective = is_introspective\n",
    "\n",
    "        if self.is_introspective:\n",
    "            self.autostep_learner = Autostep()\n",
    "\n",
    "    def update(self, x):\n",
    "        delta = self.update_prediction(x)\n",
    "        im_reward = np.abs(self.previous_error) - np.abs(delta)\n",
    "        self.previous_error = delta\n",
    "\n",
    "        return im_reward\n",
    "\n",
    "class SquaredError(IntrinsicReward):\n",
    "    def __init__(self, stepsize=0.1, is_introspective=False, **kwargs):\n",
    "        super().__init__()\n",
    "        self.stepsize = stepsize\n",
    "        self.is_introspective = is_introspective\n",
    "\n",
    "        if self.is_introspective:\n",
    "            self.autostep_learner = Autostep()\n",
    "\n",
    "    def update(self, x):\n",
    "        delta = self.update_prediction(x)\n",
    "\n",
    "        return np.square(delta)\n",
    "\n",
    "class BayesianSurprise(IntrinsicReward):\n",
    "    def __init__(self, stepsize=0.1, is_introspective=False, **kwargs):\n",
    "        super().__init__()\n",
    "        self.stepsize = stepsize\n",
    "        self.is_introspective = is_introspective\n",
    "\n",
    "        self.previous_prediction = 0.0\n",
    "        self.previous_variance = 0.0\n",
    "        self.eta = 1.0\n",
    "        self.b = kwargs.get(\"beta\", 0.01)\n",
    "\n",
    "        if self.is_introspective:\n",
    "            self.autostep_learner = Autostep()\n",
    "\n",
    "    def update(self, x):\n",
    "        delta = self.update_prediction(x)\n",
    "\n",
    "        self.b = 0.01\n",
    "        self.eta = (1 - self.b) * self.eta + 1.0\n",
    "\n",
    "        welfords_var = (1 - self.b) * self.previous_variance + self.b * (x - self.previous_prediction) * \\\n",
    "            (x - self.prediction)\n",
    "        var = max(welfords_var / self.eta, 10**-2)\n",
    "\n",
    "        if var != 0.0 and self.previous_variance != 0.0:\n",
    "            first = 0.5 * np.log2(var / self.previous_variance)\n",
    "            second = self.previous_variance + np.square(self.previous_prediction - self.prediction) / (2 * var)\n",
    "            im_reward = first + second - 0.5\n",
    "        else:\n",
    "            im_reward = 0.0\n",
    "\n",
    "        self.previous_variance = np.copy(var)\n",
    "        self.previous_prediction = np.copy(self.prediction)\n",
    "\n",
    "        return im_reward\n",
    "\n",
    "class UDE(IntrinsicReward):\n",
    "    def __init__(self, stepsize=0.1, is_introspective=False, **kwargs):\n",
    "        super().__init__()\n",
    "        self.stepsize = stepsize\n",
    "        self.is_introspective = is_introspective\n",
    "\n",
    "        self.one = 0.0\n",
    "        self.beta = kwargs.get(\"beta\", 0.01)\n",
    "        self.delta_bar = 0.0\n",
    "\n",
    "        self.count = 0\n",
    "\n",
    "        self.delta_squared_avg = MovingAverage(method=\"time\")\n",
    "        self.delta_avg = MovingAverage(method=\"time\")\n",
    "\n",
    "        if self.is_introspective:\n",
    "            self.autostep_learner = Autostep()\n",
    "\n",
    "    def update(self, x):\n",
    "        self.count += 1\n",
    "        delta = self.update_prediction(x)\n",
    "\n",
    "        self.one = (1.0 - self.beta) * self.one + self.beta * 1.0\n",
    "        rate = self.beta / self.one\n",
    "        self.delta_bar = (1.0 - rate) * self.delta_bar + rate * delta\n",
    "\n",
    "        s_bar = self.delta_squared_avg.update(np.square(delta))\n",
    "        x_bar = self.delta_avg.update(delta)\n",
    "\n",
    "        if self.count > 30:\n",
    "            count = self.delta_avg.t\n",
    "            denom = np.sqrt((s_bar / count) - np.square(x_bar / count))\n",
    "            im_reward = np.abs(self.delta_bar / (denom + 0.00001))\n",
    "        else:\n",
    "            im_reward = 0.0\n",
    "\n",
    "        return im_reward\n",
    "\n",
    "class UncertaintyChange(IntrinsicReward):\n",
    "    def __init__(self, stepsize=0.1, is_introspective=False, **kwargs):\n",
    "        super().__init__()\n",
    "        self.stepsize = stepsize\n",
    "        self.is_introspective = is_introspective\n",
    "\n",
    "        self.beta = kwargs.get(\"beta_var\", 0.001)\n",
    "\n",
    "        self.variance = IncrementalVariance(beta=self.beta)\n",
    "        self.previous_variance = 0.0\n",
    "\n",
    "        if self.is_introspective:\n",
    "            self.autostep_learner = Autostep()\n",
    "\n",
    "    def update(self, x):\n",
    "        delta = self.update_prediction(x)\n",
    "\n",
    "        var = self.variance.update(self.prediction)\n",
    "        im_reward = np.abs(self.previous_variance - var)\n",
    "        self.previous_varaiance = var\n",
    "\n",
    "        return im_reward\n",
    "\n",
    "class VarianceOfPrediction(IntrinsicReward):\n",
    "    def __init__(self, stepsize=0.1, is_introspective=False, **kwargs):\n",
    "        super().__init__()\n",
    "        self.stepsize = stepsize\n",
    "        self.is_introspective = is_introspective\n",
    "\n",
    "        self.variance = IncrementalVariance()\n",
    "\n",
    "        if self.is_introspective:\n",
    "            self.autostep_learner = Autostep()\n",
    "\n",
    "    def update(self, x):\n",
    "        delta = self.update_prediction(x)\n",
    "\n",
    "        im_reward = self.variance.update(self.prediction)\n",
    "\n",
    "        return im_reward\n",
    "\n",
    "class WeightChange(IntrinsicReward):\n",
    "    def __init__(self, stepsize=0.1, is_introspective=False, **kwargs):\n",
    "        super().__init__()\n",
    "        self.stepsize = stepsize\n",
    "        self.is_introspective = is_introspective\n",
    "\n",
    "        if self.is_introspective:\n",
    "            self.autostep_learner = Autostep()\n",
    "\n",
    "    def update(self, x):\n",
    "        delta = self.update_prediction(x)\n",
    "\n",
    "        if self.is_introspective:\n",
    "            self.stepsize = self.autostep_learner.alpha\n",
    "\n",
    "        return np.abs(self.stepsize * delta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chose Reward\n",
    "Here we set the hyperparameters for each reward chosen at run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def choose_reward_exp2(name, is_introspective):\n",
    "    if name == \"Error Derivative\":\n",
    "        if is_introspective:\n",
    "            return (ErrorDerivative(stepsize=0.000001,\n",
    "                                    is_introspective=True),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.1,\n",
    "                     \"eta\": 1000,\n",
    "                     \"tau\": 1})\n",
    "        else:\n",
    "            return (ErrorDerivative(stepsize=0.000001,\n",
    "                                   is_introspective=False),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.1,\n",
    "                     \"eta\": 1000,\n",
    "                     \"tau\": 100})\n",
    "\n",
    "    elif name == \"Expected Error\":\n",
    "        if is_introspective:\n",
    "            return (ExpectedError(stepsize=0.0001,\n",
    "                                  is_introspective=True),\n",
    "                    0.007812,\n",
    "                    {\"beta_r\": 0.1,\n",
    "                     \"beta\": 0.001})\n",
    "        else:\n",
    "            return (ExpectedError(stepsize=0.01,\n",
    "                                  is_introspective=False),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.1,\n",
    "                     \"beta\": 0.1})\n",
    "\n",
    "    elif name == \"Step-size Change\":\n",
    "        if is_introspective:\n",
    "            return (StepSizeChange(stepsize=0.000001,\n",
    "                                  is_introspective=True),\n",
    "                    0.125,\n",
    "                    {\"beta_r\": 0.001})\n",
    "        else:\n",
    "            return (StepSizeChange(stepsize=0.000001,\n",
    "                                  is_introspective=False),\n",
    "                    0.25,\n",
    "                    {\"beta_r\": 0.001})\n",
    "\n",
    "    elif name == \"Error Reduction\":\n",
    "        if is_introspective:\n",
    "            return (ErrorReduction(stepsize=0.000001,\n",
    "                                   is_introspective=True),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.001})\n",
    "        else:\n",
    "            return (ErrorReduction(stepsize=0.000001,\n",
    "                                   is_introspective=False),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.01})\n",
    "\n",
    "    elif name == \"Squared Error\":\n",
    "        if is_introspective:\n",
    "            return (SquaredError(stepsize=0.000001,\n",
    "                                 is_introspective=True),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.1})\n",
    "        else:\n",
    "            return (SquaredError(stepsize=0.000001,\n",
    "                                 is_introspective=False),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.1})\n",
    "\n",
    "    elif name == \"Bayesian Surprise\":\n",
    "        if is_introspective:\n",
    "            return (BayesianSurprise(stepsize=0.001,\n",
    "                                     is_introspective=True),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.1,\n",
    "                     \"beta\": 0.01})\n",
    "        else:\n",
    "            return (BayesianSurprise(stepsize=0.1,\n",
    "                                     is_introspective=False),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.1,\n",
    "                     \"beta\": 0.01})\n",
    "\n",
    "    elif name == \"UDE\":\n",
    "        if is_introspective:\n",
    "            return (UDE(stepsize=0.0001,\n",
    "                        is_introspective=True),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.1,\n",
    "                     \"beta\": 0.01})\n",
    "        else:\n",
    "            return (UDE(stepsize=0.0001,\n",
    "                        is_introspective=False),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.1,\n",
    "                     \"beta\": 0.1})\n",
    "\n",
    "    elif name == \"Uncertainty Change\":\n",
    "        if is_introspective:\n",
    "            return (UncertaintyChange(stepsize=0.1,\n",
    "                                      is_introspective=True),\n",
    "                    0.015625,\n",
    "                    {\"beta_r\": 0.0001,\n",
    "                     \"beta_var\": 0.1})\n",
    "        else:\n",
    "            return (UncertaintyChange(stepsize=0.1,\n",
    "                                      is_introspective=False),\n",
    "                    0.25,\n",
    "                    {\"beta_r\": 0.01,\n",
    "                     \"beta_var\": 0.01})\n",
    "\n",
    "    elif name == \"Variance of Prediction\":\n",
    "        if is_introspective:\n",
    "            return (VarianceOfPrediction(stepsize=1.0,\n",
    "                                         is_introspective=True),\n",
    "                    0.007812,\n",
    "                    {\"beta_r\": 0.1})\n",
    "        else:\n",
    "            return (VarianceOfPrediction(stepsize=0.1,\n",
    "                                         is_introspective=False),\n",
    "                    0.015625,\n",
    "                    {\"beta_r\": 0.1})\n",
    "\n",
    "    elif name == \"Weight Change\":\n",
    "        if is_introspective:\n",
    "            return (WeightChange(stepsize=0.000001,\n",
    "                                 is_introspective=True),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.1})\n",
    "        else:\n",
    "            return (WeightChange(stepsize=0.000001,\n",
    "                                 is_introspective=False),\n",
    "                    0.003906,\n",
    "                    {\"beta_r\": 0.1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BehaviorAgent:\n",
    "    def __init__(self, behavior_stepsize, intrinsic_reward, is_introspective, *args, **kwargs):\n",
    "        self.intrinsic_rewards = [choose_reward_exp2(intrinsic_reward, is_introspective)[0] for _ in range(4)]\n",
    "        self.one = 0.0\n",
    "        self.r_bar = 0.0\n",
    "        self.beta = kwargs.get(\"beta_r\", 0.1)\n",
    "        self.behavior_stepsize = behavior_stepsize\n",
    "        self.action_values = np.zeros(4)\n",
    "        self.probs = np.ones(4) / 4\n",
    "\n",
    "    def calculate_probs(self):\n",
    "        self.probs = softmax(self.action_values)\n",
    "\n",
    "    def choose_action(self):\n",
    "        self.calculate_probs()\n",
    "        return select_action(self.probs)\n",
    "\n",
    "    def update(self, x, chosen_action):\n",
    "        # get intrinsic_reward\n",
    "        im_reward = self.intrinsic_rewards[chosen_action].update(x)\n",
    "\n",
    "        # update_average_reward\n",
    "        self.one = (1.0 - self.beta) * self.one + self.beta * 1.0\n",
    "        rate = self.beta / self.one\n",
    "        self.r_bar = (1.0 - rate) * self.r_bar + rate * im_reward\n",
    "\n",
    "        # Update the actions values based on the intrinsic reward\n",
    "        for action in range(4):\n",
    "            self.action_values[action] += self.behavior_stepsize * (im_reward - self.r_bar) * \\\n",
    "                ((action==chosen_action) * 1.0 - self.probs[action])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_experiment(is_introspective=True, record_stepsizes=False,\n",
    "                   intrinsic_reward=\"Error Derivative\"):\n",
    "    data = generate_action_data_two()\n",
    "    track_probs = []\n",
    "    track_stepsizes = []\n",
    "    \n",
    "    behavior_agent = BehaviorAgent(choose_reward_exp2(intrinsic_reward, is_introspective)[1], \n",
    "                                   intrinsic_reward,\n",
    "                                   is_introspective,\n",
    "                                   **choose_reward_exp2(intrinsic_reward, is_introspective)[2])\n",
    "    \n",
    "                                       \n",
    "    for i, step_data in enumerate(data.T):\n",
    "        chosen_action = behavior_agent.choose_action()\n",
    "\n",
    "        behavior_agent.update(step_data[chosen_action], chosen_action)\n",
    "\n",
    "        track_probs.append(np.copy(behavior_agent.probs))\n",
    "\n",
    "        if record_stepsizes:\n",
    "            track_stepsizes.append(np.copy(np.array([learner.autostep_learner.alpha \\\n",
    "                                                     for learner in behavior_agent.intrinsic_rewards])))\n",
    "\n",
    "    return np.array(track_probs), np.array(track_stepsizes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run():\n",
    "    output = widgets.Output()\n",
    "    button = widgets.Button(description=\"Run Experiment\")\n",
    "\n",
    "    choose_reward = widgets.Dropdown(\n",
    "        options=[\"Error Derivative\", \"Expected Error\", \"Step-size Change\", \n",
    "                \"Error Reduction\", \"Squared Error\", \"Bayesian Surprise\", \n",
    "                \"UDE\", \"Uncertainty Change\", \"Variance of Prediction\", \"Weight Change\"],\n",
    "        value=\"Weight Change\",\n",
    "        description=\"Choose Reward:\",\n",
    "        disabled=False,\n",
    "        style={'description_width': 'initial'},\n",
    "    )\n",
    "\n",
    "    choose_runs = widgets.IntText(description=\"Choose number of runs:\", value=2, style={'description_width': 'initial'})\n",
    "    choose_introspective = widgets.Checkbox(value=True, description=\"Use Introspective Learners?\")\n",
    "\n",
    "    def reset_displays():\n",
    "        clear_output()\n",
    "        display(choose_reward)\n",
    "        display(choose_runs)\n",
    "        display(choose_introspective)\n",
    "        display(button)\n",
    "\n",
    "    def on_button_click(b):\n",
    "        reset_displays()\n",
    "        reward = choose_reward.value\n",
    "        print(\"Running\")\n",
    "        runs = [run_experiment(intrinsic_reward=reward,\n",
    "                               is_introspective=choose_introspective.value)[0] for _ in range(choose_runs.value)]\n",
    "        action_probs = np.mean(runs, axis=0)\n",
    "        with output:\n",
    "            plt.figure(figsize=(4,5))\n",
    "            colors = [\"red\", \"black\", \"blue\", \"green\"]\n",
    "            for action, color in zip(action_probs.T, colors):\n",
    "                plt.plot(action, color=color)\n",
    "            plt.axvline(x=50_000, alpha=0.5, color=\"black\")\n",
    "            plt.legend([\"Target 1: Distractor 1 → Drifter 1\",\n",
    "            \"Target 2: Drifter → Distractor\",\n",
    "            \"Target 3: Constant → Drifter 2\",\n",
    "            \"Target 4: Distractor 2 → Constant\"])\n",
    "            plt.ylim([0.0, 1.0])\n",
    "\n",
    "    button.on_click(on_button_click)\n",
    "    reset_displays()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8ecf0f79d1354940a8c8e1439642b982",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Choose Reward:', index=6, options=('Error Derivative', 'Expected Error', 'Step-size Chan…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "90ff9565e5c24f2bacee0468792a931c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "IntText(value=2, description='Choose number of runs:', style=DescriptionStyle(description_width='initial'))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "951f98e38af042039142ef3aa7c4fd71",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Checkbox(value=True, description='Use Introspective Learners?')"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4fa09df25e4d4cffaef07a03126fc1fc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Button(description='Run Experiment', style=ButtonStyle())"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQ0AAAEzCAYAAAArYlbVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwFElEQVR4nO3dd3wVVdrA8d9JpwkYIvISUVREkpACgVAEiYggsIAgCoslK8qyKy5YKL4gUtQXRBERXcSWXVdB0QXjAqKRKoJCJCp1AYkQQIgQCKGlnfePW7xJbm5JbnJnzPP9fPLJ3KnPTO48OXNmzhyltUYIITwV4O8AhBDmIklDCOEVSRpCCK9I0hBCeEWShhDCK5I0hBBecZs0lFJvK6VOKKV2VDBdKaXmK6X2K6V+UEq1832YQgij8KSkkQr0cTH9dqCV9WcU8PeqhyWEMCq3SUNrvQE45WKWgcA/tcUWoJFSqpmvAhRCGIsv6jSaA4cdPmdbxwkhfoeCanJjSqlRWC5hqFevXvsbb7yxWrZz6dIlLl68yIULF7h06RIFBQUUFhZSUFBAcXFxldevlEIpVeqzN8POPnvCFntgYKDXy1Zme9VB4ijNl3EEBwdz3XXXuZ0vIyPjV611RGW344ukcQS4yuFzpHVcOVrrRcAigMTERL1t2zYfbN5izZo1fPbZZyxZsoTDh38r+DRr1ozIyEgiIyNp3rw5TZo0oX379oSHhxMSEkJwcLD9t+NPSEgIQUFBBAcHExgYSFBQEAEBAX79sqWmpgKQkpLitxiE+Smlfq7K8r5IGmnAGKXUEiAJOKO1PuaD9XqkqKiI6dOn88wzzwBw/fXXM3XqVHr16kXr1q2JiKh0QhVCOOE2aSilFgM9gCZKqWzgaSAYQGu9EFgJ9AX2A+eBP1VXsGWdOnWKXr168d1339GqVSs++ugjYmNja2rzQtRKbpOG1nq4m+kaeNhnEXno1KlThIeHA3D77bezcuXKmg5BiFrJtE+EPvnkkwCMHDlSEoYQNci0SWPRokUAvPnmm36ORIjaxZRJIz09HYBhw4b5ORIhah9TJo3Zs2cDMH/+fD9HIkTtY8qkkZGRASC3U4XwA9MljbNnz5Kbm8vkyZP9HYoQtZLpksZLL70EQEJCgp8jEaJ2Ml3SCAqyPFrSpUsXP0ciRO1kuqTx+uuvA3DllVf6ORIhaifTJY1Dhw4BxmmlKERtY7qkIYTwL1MlDVsXkuPGjfNvIELUYqZKGvn5+QA0by4vBhPCX0yVNH755RfA8g4NIYR/mCpp7Nhh6UWhSZMmfo5EiNrLVEnj2DHLC8FuueUWP0ciRO1lqqRx5swZwPLeTyGEf5gqaeTm5hIaGkqdOnX8HYoQtZapksaJEyekZasQfmaqpHHkyBGaNm3q7zCEqNVMlTQ2b97MVVdd5X5GIUS1MU3SOHfuHOfOnSMyMtLfoQhRq5kmaRw8eBBAKkGF8DPTJA2bjh07+jsEIWo10ySNgoICwNLJrRDCf0yXNEJCQvwciRC1m2mSRmFhISBJQwh/M03SkMsTIYzBNElDShpCGINpkoaUNIQwBtMkDSlpCGEMpkkaUtIQwhhMlzSkpCGEf5kmacjliRDGYJqkIZcnQhiDaZKGlDSEMAbTJA0paQhhDKZJGlLSEMIYTJM0pKQhhDGYKmkEBQVJb/FC+JlpkkZhYaFcmghhAKZJGgUFBXJpIoQBmCZpSElDCGMwTdKQkoYQxmCapCElDSGMwTRJQ0oaQhiDR0lDKdVHKbVXKbVfKTXJyfQWSqm1SqntSqkflFJ9fR1oQUGBlDSEMAC3SUMpFQi8CtwORAHDlVJRZWabAnyotU4AhgGv+TpQuTwRwhg8KWl0BPZrrX/SWhcAS4CBZebRwGXW4YbAUd+FaCGXJ0IYgydJozlw2OFztnWco2nAPUqpbGAl8IizFSmlRimltimltuXk5HgVaGFhoSQNIQzAVxWhw4FUrXUk0Bd4VylVbt1a60Va60StdWJERIRXG7h06RJhYWG+iVYIUWmeJI0jwFUOnyOt4xyNBD4E0FpvBsKAJr4I0ObixYuEhob6cpVCiErwJGlsBVoppVoqpUKwVHSmlZnnENATQCnVBkvS8O76ww0paQhhDEHuZtBaFymlxgCrgUDgba31TqXUDGCb1joNeBx4Qyn1KJZK0RSttfZloDt27KBRo0a+XKUQohLcJg0ArfVKLBWcjuOmOgzvArr6NrTyTpw4Ud2bEEK4YZonQps2bUr37t39HYYQtZ5pkobWmsDAQH+HIUStZ5qkUVxcLElDCAOQpCGE8IokDSGEVyRpCCG8IklDCOEVUyWNoCCPHisRQlQj0ySNoqIiKWkIYQCmSBolJSUAkjSEMABTJY2AAFOEK8TvminOQlvbN0kaQvifKc5CW0lD+nEVwv9MkTSkpCGEcZjiLJSShhDGYYqkISUNIYzDFGeh3D0RwjhMcRbK5YkQxmGKpCGXJ0IYhynOQilpCGEcpkgaUtIQwjhMcRZKSUMI4zBF0pCShhDGYYqzUEoaQhiHKZKGlDSEMA5TnIXycJcQxmGKs1AuT4QwDlMkDbk8EcI4THEWSklDCOMwRdKQkoYQxmGKs1BKGkIYhymShpQ0hDAOU5yFUtIQwjhMkTSkpCGEcZjiLJSHu4QwDlOchXJ5IoRxmCJpyOWJEMZhirNQShpCGIcpkoaUNIQwDlOchVLSEMI4TJE0pKQhhHGY4iyUkoYQxmGKpCElDSGMwxRnoZQ0hDAOUyUNKWkI4X8enYVKqT5Kqb1Kqf1KqUkVzHOXUmqXUmqnUup9XwYplydCGEeQuxmUUoHAq0AvIBvYqpRK01rvcpinFfAk0FVrnauUusKXQcrliRDG4cm/7o7Afq31T1rrAmAJMLDMPA8Br2qtcwG01id8GaSUNIQwDk/OwubAYYfP2dZxjm4AblBKbVJKbVFK9fFVgCAlDSGMxO3liRfraQX0ACKBDUqptlrr044zKaVGAaMAWrRo4fHKpaQhhHF4chYeAa5y+BxpHecoG0jTWhdqrQ8C/8WSRErRWi/SWidqrRMjIiI8DlJKGkIYhydJYyvQSinVUikVAgwD0srMsxxLKQOlVBMslys/+SpIKWkIYRxuz0KtdREwBlgN7AY+1FrvVErNUEoNsM62GjiplNoFrAXGa61P+ipIKWkIYRwe1WlorVcCK8uMm+owrIHHrD8+V1RUBEBQkK+qYIQQlWWK8n5xcTEgSUMIIzBF0rCVNAIDA/0ciRDCFElDShpCGIcpkoaUNIQwDlMkDSlpCGEcpkgaUtIQwjhMkTRsJQ1JGkL4nymShjynIYRxmCJpSElDCOMwRdKQkoYQxmGKpCElDSGMwxRJIzs7G5CShhBGYIqk0bhxY0BKGkIYgSmSho2UNITwP1MkDdv7NKSkIYT/mSJp2CpC5c1dQvifKc5CSRpCGIcpzsLi4mK5NBHCICRpCCG8YoqkUVJSIklDCIMwRdIoLi6W+gwhDMIUZ6JcnghhHJI0hBBekaQhhPCKKZKGVIQKYRymSBpSESqEcZjiTJTLEyGMQ5KGEMIrkjSEEF4xRdKQilAhjMMUSUMqQoUwDlOciXJ5IoRxSNIQQnhFkoYQwiumSBpSESqEcZgiaUhFqBDGYYozUS5PhDAOSRpCCK9I0hBCeMUUSUMqQoUwDlMkDakIFcI4THEmyuWJEMYhSUMI4RVJGkIIr0jSEEJ4xaOkoZTqo5Taq5Tar5Sa5GK+IUoprZRK9F2IlqQRFBTky1UKISrJbdJQSgUCrwK3A1HAcKVUlJP5GgBjgW98HWRRUZEkDSEMwpOSRkdgv9b6J611AbAEGOhkvpnAbOCiD+MDLElDLk+EMAZPkkZz4LDD52zrODulVDvgKq31Ch/GZiclDSGMo8oVoUqpAGAu8LgH845SSm1TSm3LycnxeBtSpyGEcXiSNI4AVzl8jrSOs2kAxADrlFJZQCcgzVllqNZ6kdY6UWudGBER4XGQUtIQwjg8SRpbgVZKqZZKqRBgGJBmm6i1PqO1bqK1vkZrfQ2wBRigtd7mqyClTkMI43CbNLTWRcAYYDWwG/hQa71TKTVDKTWgugMEuTwRwkg8OhO11iuBlWXGTa1g3h5VD6s0uTwRwjhM8USoJA0hjMM0SUPqNIQwBlMkDanTEMI4DJ80SkpK5M1dQhiI4ZPGxYuWp9Lr1Knj50iEEGCCpHHhwgUA6tat6+dIhBBggqRx/vx5QEoaQhiF4ZOGXJ4IYSymSRphYWF+jkQIAZI0hBBekqQhhPCKJA0hhFckaQghvCJJQwjhFdMkDbnlKoQxmCZpSElDCGMwfNKwPUYuSUMIYzB80rCVNEJDQ/0ciRACTJQ0pKQhhDGYImkEBwfL+zSEMAjDJ41Lly5JKUMIAzF80vjmm86cP7/R32EIIawM/+LNTZvuBKC4GOQKRQj/M3xJw+bMGX9HIIQAEyWNU6f8HYEQAgyeNAoLfxs+edJ/cQghfmPopGF9GBSAXbv8F4cQ4jeGThrWdwoDYH3GSwjhZ4ZOGo4lDbk8EcIYTJM0pKQhhDEYOmk4Xp5cuuS/OIQQvzF00nAsXUjSEMIYDJ005PJECOMxdNJIT//KPiwlDSGMwdBJIzNzFwRasoWUNIQwBkMnjc/ProGHkggIyZGShhAGYehWrh2uPszWK7+npNFxPvkkotz0wsJCsrOz7W/3+r2Ljo4GYPfu3X6ORJhBWFgYkZGRBAcH+3S9hk4aM+/oR5/dX0NontPp2dnZNGjQgGuuuQalVA1HV/N+/fVXAJo0aeLnSITRaa05efIk2dnZtGzZ0qfrNvTlyWWXXQFA3I2H6dq1/PSLFy8SHh5eKxKGEN5QShEeHl4tpXBDJ40GjZtaBoJyS7V4dSQJQwjnquvcMHTSqNfQWo9x7kiFScOfTp48SXx8PPHx8Vx55ZU0b97c/rmgoMCn2zp9+jRvv/12hdMfeOABrrjiCmJiYjxa37p162jYsCEJCQm0bt2a7t2785///Mc+feHChfzzn/90ufzXX3/t+Q64MW/ePM47PgJcCUuXLiU6OpqAgAC2bdvmk7hSUlJo2bIlcXFx3HDDDdx3331kZ2dXOP+DDz7ILmuT7KVLl9KmTRuSk5N9drz69OlDo0aN6N+/f5XXVWlaa7/8tG/fXrtzIveIZho6tttfdUxM+em7du1yu46a8vTTT+s5c+Z4NG9hYaHX6z948KC+8cYbdU5OjtPp69ev1xkZGTo6Otqj9a1du1b369fP/nn79u366quv1unp6R4t72p/K7N/V199dYX7VpGioqJSn3ft2qX37Nmjb775Zr1161avY3Dm/vvv10uXLtVaa11SUqLnzp2rW7VqpS9duuQ2nt69e+uNGzdqrb37ftg4O47p6ek6LS2t1N/OFWfnCLBNV+HcNXRJo369xgAcTFxhyJKGM2+88QYdOnQgLi6OIUOG2P97pqSkMHr0aJKSkpgwYQIHDhygU6dOtG3blilTplC/fn37OubMmUOHDh2IjY3l6aefBmDSpElkZWXRo0cPxo8fX2673bt35/LLL6903PHx8UydOpUFCxYAMG3aNF544QUA5s+fT1RUFLGxsQwbNoysrCwWLlzISy+9RHx8PBs3biy3f99++y2dO3cmISGBLl26sHfvXgCKi4t54okniImJITY2lldeeYX58+dz9OhRkpOTSU5OBmDx4sW0bduWmJgYJk6caI+zfv36PP7448TFxbF58+ZS+9CmTRtat27t9b5/9dVXnPSgGbVSikcffZQrr7ySVatWOY2nR48ebNu2jRkzZvDVV18xcuRIhg4dWu545eTkMGTIEDp06ECHDh3YtGmT/bjfe++9dO3alXvvvbdcDD179qRBgwZe76MvGfruSZ1gS6fPF+rm4La0P24cZGb6NoD4eJg3z6tFBg8ezEMPPQTAlClTeOutt3jkkUcAy92er7/+msDAQPr378/YsWMZPnw4CxcutC//+eefs2/fPr799lu01gwYMIANGzYwa9Ysvv/+e9atW+fV3RPbukePHu123nbt2jFnzpxy42fNmsXBgwcJDQ3l9OnTNGrUiNGjR1O/fn2eeOIJAN56661S+5eXl8fGjRsJCgoiPT2d//3f/+Xjjz9m0aJFZGVlkZmZSVBQEKdOneLyyy9n7ty5rF27liZNmnD06FEmTpxIRkYGjRs35rbbbmP58uUMGjSIc+fOkZSUxIsvvujxMXAnNzeX22+/nVWrVhEeHu7RcdqzZw8DBw6sMJ6pU6eyZs0aXnjhBRITE5k2bVqp4/XHP/6RRx99lJtuuolDhw7Ru3dv+630Xbt28dVXXxm203NDJw2AZucDaZCdzHmTlDR27NjBlClTOH36NPn5+fTu3ds+bejQofZOnzZv3szy5csByxfI9mX6/PPP+fzzz0lISAAgPz+fffv20aJFi0rF40mysLGUXMuLjY1lxIgRDBo0iEGDBlW4vOP+nTlzhvvvv599+/ahlKLQWlRMT09n9OjRBAVZvnrOSkdbt26lR48eRERY6rRGjBjBhg0bGDRoEIGBgQwZMsTjfSpr9uzZvPvuu+XGHzp0iL/85S98+OGHbtfheJwqG096erq97gMgLy+P/Px8AAYMGGDYhAEmSBoNi4MoCDrr/vLEyxJBdUlJSWH58uXExcWRmprKunXr7NPq1avndnmtNU8++SR//vOfS43PysrycaTlbd++nTZt2pQbv2LFCjZs2MCnn37Ks88+y48//uh0ecf9e+qpp0hOTmbZsmX2yypfCAsLq1JvexMnTix1uQPw/fffc++99zJz5kyP1rF9+3Z69uxZpXhKSkrYsmWL047APPme+JNHdRpKqT5Kqb1Kqf1KqUlOpj+mlNqllPpBKfWlUupqXwV4WUkIRZ4kDYM4e/YszZo1o7CwkPfee6/C+Tp16sTHH38MwJIlS+zje/fuzdtvv23/r3PkyBFOnDhBgwYN7OOqww8//MDMmTN5+OGHS40vKSnh8OHDJCcnM3v2bM6cOUN+fj4NGjTg7NmzFa7vzJkzNG/eHIDU1FT7+F69evH6669TVFQEwCnra+Yd19exY0fWr1/Pr7/+SnFxMYsXL+bmm2/25e6Wsnv3bj766CO39SFaa+bPn8+xY8fo06ePV9soe7xuu+02XnnlFfvnTF9fWlcjt0lDKRUIvArcDkQBw5VSUWVm2w4kaq1jgY+A530VYAMVSmFwvmmSxsyZM0lKSqJr167ceOONFc43b9485s6dS2xsLPv376dhw4aA5cv0xz/+kc6dO9O2bVvuvPNOzp49S3h4OB07dqRbt25OK0KHDx9O586d2bt3L5GRkbz11luApU7Dsc7E0caNG+23XB9++GHmz59v/w9qU1xczD333EPbtm1JSEjgb3/7G40aNeIPf/gDy5Yts1fslTVhwgSefPJJEhIS7AkCLLckW7RoQWxsLHFxcbz//vsAjBo1ij59+pCcnEyzZs2YNWsWycnJxMXF0b59ewYOHOjmyMOyZcuIjIxk8+bN9OvXr9SloSvDhg3jhhtuqHD6+PHj7bdct27dytq1awkJCfFo3TZlj9f8+fPZtm0bsbGxREVFVfg3Kqtbt24MHTqUL7/8ksjISFavXu1VHD7h7vYK0BlY7fD5SeBJF/MnAJvcrdeTW65aa33nUPTlD0fo0FDPbieZxblz53RJSYnWWuvFixfrAQMGuF0mJyfH69uSonarjluuntRpNAcOO3zOBpJczD8SWOVsglJqFDAK8Lhi79IVl3MqIocAk5Q0PJWRkcGYMWPQWtOoUSOXD24JYSQ+rQhVSt0DJAJOL0C11ouARQCJiYnOq+rLLhPRBDhFiS6mqCiQIMNX3XqmW7dufP/99/4OQwiveVIRegS4yuFzpHVcKUqpW4HJwACttc/efnFTA0tzcMLOyDs1hDAAT5LGVqCVUqqlUioEGAakOc6glEoAXseSME74MsBjYdZKtAZH5e1dQhiA26ShtS4CxgCrgd3Ah1rrnUqpGUqpAdbZ5gD1gaVKqUylVFoFq/PabS0sjxUTmidJQwgD8KiGQGu9ElhZZtxUh+FbfRyX3TVXtYUMIOFtLl7sUl2bEUJ4yNAN1gDa3GB9+05xiOFKGkZpGm97+CoqKoro6Ghefvllt+tz1zS+rLS0NGbNmgVATk4OSUlJJCQksHHjRl577bXK7VQl/R6b9ZtKVe7XVuXH0+c0tNa67qQgTe9xOiPD/T1of/Fn0/ijR4/qDOvBycvL061atdI7d+50uT5vmsaXjXfx4sV65MiR9rg8bY5vU1JSoouLi71axtHvsVl/dal1TeNt6hQFQ+hZHB4sNCx/NI1v1qwZ7dq1AyyPK7dp04YjR8rd4HKpbNP4svGmpqYyZswYMjMzmTBhAp988gnx8fFMnDiRAwcOEB8fb4/LWfxZWVm0bt2a++67j5iYGA4fPlxhLGB5DN2x3Y43sZuxWb+ZmOKphzpFwRDi+lHycePG+fz5/fj4eOaZrGl8VlYW27dvJynJ8vxdVZrGO8Zraz8SHx/PjBkz2LZtGwsWLCArK4udO3faj31F8bdo0YJ9+/bxj3/8g06dOrmNpbCwkPHjxzNt2jT69evndew2v/dm/f5gjqRRHAIh+aYoafizaXx+fj5Dhgxh3rx5XHbZZUDVmsY7xuspV/FfffXVThPG3r17nTYvz8vLY8iQIRw9etTtC4bKxm7ze2rWbxSmSBr1SkIgxPXlibclgurir6bxhYWFDBkyhBEjRjB48ODKhF6uaXxlmmi7ir+i9bVu3ZodO3aUGnfp0iWGDBlCr169PHojWW1o1m8UpqjTqFscbJqShj+axmutGTlyJG3atOGxxx6rVNwVNY13p2yT74ri91ZOTg6DBw9m7Nixbuetzc36/cEUSaOeDnFbp2EU/mgav2nTJt59913WrFljv+W7cqXlsZqqNo13Jzw8nK5duxITE8P48eMrjN9bkZGRPPDAAxVO/7016zcTVdG1YHVLTEzUnr5m/o7RbVle7xTLux/B8fjv3r3baZHUDM6fP0+dOnVQSrFkyRIWL17MJ5984nIZ6WFNeMvZOaKUytBaJ1Z2naao06hPKISepbjY35H4jjSNF2ZliqQREBwEoWcpLNTA76NHNWkaL8zKFHUav9SxVKrlFZz2byBCCHMkje4qFoBzF075ORIhhCmSRoOGlg5sLpz+1c+RCCFMkTTq1rG0y7hwwftbd0II3zJF0qgXanla72Jh9fX7URlGaRp/8eJFOnbsSFxcHNHR0fZGYu6sWrWKxMREoqKiSEhI4PHHH/dlyDz33HNVWn758uWleiGrisDAQOLj44mOjiYuLo4XX3yRkpISp/MePXqUO++80/55+PDhxMbG8tJLL/mkGfzcuXPtjeh69uzJzz//XKX11biqNJGtyo83TeM/eetFzTT0mAn/ctvs11/82TS+pKREnz17VmutdUFBge7YsaPevHmzy/X9+OOP+tprr9W7d+/WWluaar/22mtex+VKvXr1qrS8Y4/tvozl+PHjumfPnnrq1Knl5iv7tzl27Ji+7rrr7J990Qx+zZo1+ty5c1prrV977TV91113ebU+b9TapvH16lhKGgWFF/wciXv+aBqvlLIvX1hYSGFhIUq5vjX9/PPPM3nyZPtTq4GBgfzlL38BLO1EbrnlFvt/wkOHDtn34W9/+xtdunTh2muv5aOPPgLg2LFjdO/enfj4eGJiYti4cSOTJk3iwoULxMfHM2LECAAGDRpE+/btiY6OZtGiRfZY6tevz+TJk4mLi6NTp04cP36cr7/+mrS0NMaPH098fDwHDhxwuT9aa9LSPHvL5BVXXMGiRYtYsGABWmtSU1MZMGAAt9xyCz179iQrK4uYmBjA8oTukSNHiI+PZ/r06eWawX/++ed07tyZdu3aMXToUPvj89dccw0TJ06kXbt2LF26tNT2k5OTqVu3LmBpTpCdne1R3IZRlYxTlR9vShob/52qmYZOeeTVCrPo2LFa33yzb3/GjvU4RHtJ49dff7WPmzx5sp4/f77W2vJfs1+/fvb/Ov369dPvv/++1lrrv//97/b/hKtXr9YPPfSQ/UU1/fr10+vXr3dZ0tDa8t8sLi5O16tXT0+YMME+/qmnntKffPJJufkTEhJ0Zmam03X1799fp6amaq21fuutt/TAgQPt+3DnnXfq4uJivXPnTvt/4BdeeEE/88wz9jjy8vK01uVLGidPntRaa33+/HkdHR1tP1aATktL01prPX78eD1z5kz79jwtaRQXF+sRI0boKVOmOJ3urNTTsGFD/csvv+h33nlHN2/e3B6f44uFyr5kyLGkkZOTo7t166bz8/O11lrPmjVLT58+3T7f7Nmz3cb98MMP2/e3OvirsyS/q2PNygVFxu/DwF9N4wMDA8nMzOT06dPccccd7Nixg5iYGGbMmOH1PmzevJl///vfANx7771MmDDBPm3QoEEEBAQQFRXF8ePHAejQoQMPPPAAhYWFDBo0iPj4eKfrnT9/PsuWLQMsryjct28f4eHhhISE0L9/fwDat2/PF1984TbGLl26kJeXV2pccXEx7733Hp07d6Zv375e7bOnrWkdbdmyhV27dtG1q+WVlAUFBXTu3Nk+/e6773a5/L/+9S+2bdvG+vXrvdquv5kiaYTVrQNAUXHFlYsGaRnv917jGzVqRHJyMp999pm9iO1MdHQ0GRkZxMXFebRem9DQ0FKxAnTv3p0NGzawYsUKUlJSeOyxx7jvvvtKLbdu3TrS09PZvHkzdevWpUePHly0vvQ1ODjYfjkVGBhYqpFYRZy94/OZZ57hxx9/5LbbbnO7/E8//URgYCBXXHEFUPnXAPTq1YvFixc7ne5qnenp6Tz77LOsX7++1DE1A1PUadSpa63TKDZ+ScMfTeNzcnI4ffo0ABcuXOCLL75w2cIWLJ0aP/fcc/z3v/8FLM3Iba1hu3TpYo/pvffeo1u3bi7X9fPPP9O0aVMeeughHnzwQb777jvAkgxsL7A5c+YMjRs3pm7duuzZs4ctW7a4XCeUb3bvSlFREcHBwbz//vv2l+RUJCcnh9GjRzNmzBi3dT+uYurUqRObNm1i//79AJw7d85+PF3Zvn07f/7zn0lLS7MnLTMxRdIItScN397GrA7+aBp/7NgxkpOTiY2NpUOHDvTq1cte3J86darTCsLY2FjmzZvH8OHDadOmDTExMfz0008AvPLKK7zzzjvExsby7rvvun27+bp164iLiyMhIYEPPvjA/g6MUaNG2d+c1adPH4qKimjTpg2TJk3y6JV/w4YNY86cOSQkJLitCA0KCmLixIkVvuTGVikbHR3Nrbfeym233ebxrWlHjs3gIyIiSE1Ntd+S7dy5M3v27HG7jvHjx5Ofn8/QoUOJj49nwIABbpcxlKpUiFTlx5uK0BNZOzXT0MkjJrmt5DEL6TVe1IRaWxEaHGYpaVwww6u7PCRN44VZmSJphNRtAMCFIudP8JmRNI0XZmWKOo1gW9uTkt/RW3iEMClTJI2gwGAALhb/fkoaQpiVKZKGUoqAoiAuYYI3CwvxO2eKpAGgleZ8He9fhS+E8C3zJI3AYs7euMzfYZRilKbxNsXFxSQkJNif0XAlNTWViIgIEhISaNWqFb179y71lOXUqVNJT0+vcHlfNluHqjejB0tvZq1btyYmJsb+WHtVFRYWMmnSJFq1akW7du3o3Lkzq1atqvJ6bTIzM+3dTVSWL46dN0yTNC4/eSXqTKS/wyglPDyczMxMMjMzGT16NI8++qj9c0hISIXLefKYdFmnT5/mnXfecTnPyy+/7FWXDnfffTfbt29n3759TJo0icGDB7N7924AZsyYwa233lrhsq6SRmX2z9svvta63PswRowYwZ49e/jxxx+5cOECb775ptdxlPXUU09x7NgxduzYwXfffcfy5csr1Y9LRSRpVKP8+rnoRoep4L0phuGPpvFg6ax5xYoVPPjgg5WKOzk5mVGjRtmbrKekpNibvk+aNMn+0pgnnnjCabP1Hj16MG7cOBITE3n55Zf59NNPSUpKIiEhgVtvvdXeuC0/P58//elPtG3bltjYWD7++GOnzejnzp1LTEwMMTEx9i433fU837dvX5RSKKXo2LGjx03OV69ezYUL5V+7cP78ed544w1eeeUVe/uQpk2bctdddwGue4cv29QfYOnSpcTExBAXF0f37t0pKChg6tSpfPDBB8THx/PBBx9U2GN9amoqgwcPpk+fPrRq1creiNDZsatupnhOA+D6463Y1WIH585Bgwblp4/7bByZv2T6dJvxV8Yzr888r5bxV6/x48aN4/nnny/3X3Dq1KkkJiZ69Khyu3bteP3110uNO3nyJMuWLWPPnj0opew9rw8YMID+/fuXesNVQUEBtg6wcnNz2bJlC0op3nzzTZ5//nlefPFFZs6cScOGDe19qubm5jJkyBAWLFhg73k+IyODd955h2+++QatNUlJSdx88800btzYo57nCwsLPXr83ebgwYPMnTuX5cuXU6dOHfv4/fv306JFC3tn2o7c9Q7fqVMnnn32WSZMmMAbb7zBlClTmDFjBqtXr6Z58+acPn2akJAQZsyYwbZt21iwYAFAhT3Wg6VUsn37dkJDQ2ndujWPPPIIs2bNKnXsaoJpkkbTkqbsupDN2bPOk4ZR+KNp/H/+8x+uuOIK2rdvX6pVLeBV03ity/e217BhQ8LCwhg5ciT9+/d3WV/i2BQ8Ozubu+++m2PHjlFQUEDLli0BS+tOxwZ6jRs3Lreer776ijvuuMPeSnTw4MFs3LiRAQMGVNjzvKO//vWvdO/e3WlDu7Fjx/Lll1+WG79//36efvppnn/+eZfrtnHVO3xFTf27du1KSkoKd911V4WddFfUYz1Az5497W2UoqKi+Pnnn7nqqqs8iteXTJM06gaFQNBFKmjo6XWJoLr4o2n8pk2bSEtLY+XKlVy8eJG8vDzuuece/vWvf3kVu7Oe14OCgvj222/58ssv+eijj1iwYAFr1qxxurzj/j3yyCM89thjDBgwgHXr1jFt2jSvYqmIu2M4ffp0cnJyypWYbJyVPr744gsmTZpUrvPs66+/nkOHDpGXl+e0tFGRipr6L1y4kG+++YYVK1bQvn17MjIyyi3rqsd6xyb0nr5CoDqYpk6jfnAIBF/kTJ6xKzX80TT+//7v/8jOziYrK4slS5Zwyy23eJ0w1q9fz6JFi+yXVjb5+fmcOXOGvn378tJLL9kfffem5/V//OMf9vG9evXi1VdftX/Ozc0FSjej79atG8uXL+f8+fOcO3eOZcuWuW2eD/Dmm2+yevVqFi9eTECA51/tAwcOsHLlSq688spS4+vWrcvIkSMZO3as/W5YTk4OS5curVTv8AcOHCApKYkZM2YQERHB4cOHyx3Hinqsd8Xx2NUE0ySNRnUshaJ1a6v2Jujq5o+m8a5U1DQesFfA3XDDDTz33HN8/PHH5UoaZ8+epX///sTGxnLTTTcxd+5cwH2z9WnTpjF06FDat29fqg5mypQp5Obm2isE165dC5RuRt+uXTtSUlLo2LEjSUlJPPjgg/bLNVdGjx7N8ePH6dy5M/Hx8R5fmo0ePZqmTZs6nfbMM88QERFBVFQUMTEx9O/fn8suu6xSvcOPHz/eXnHapUsX4uLiSE5OZteuXfaK0Ip6rHfF8djVBFP0Gg8wa2J/nqy7gqT0Q2zZaLmOk17jhXCt1vYaD3BVq+vgCBzN/RWo+cofX5Om8cKsTJM0boy7BY7M53DSI5w69RVevgPWcKRpvDAr09RptEu0PmfQYhNuSvFCiGpkmqShlGJeHctzAB+u+Mw+3l91MkIYXXWdG6ZJGgD33z8bSgL4LO9LioogLCyMkydPSuIQogytNSdPniQsLMzn6zZNnQZAo6ZX0yYrjt1dX+Cvj93Lqy+2ITs7m5ycHH+HViNsz2nUlv0VVRMWFkZkpO8beXqUNJRSfYCXgUDgTa31rDLTQ4F/Au2Bk8DdWuss34Zq8d6dj9Duuwd4IzyOfl+cYGDfltWxGUOyPeyTkpLi1zhE7eb28kQpFQi8CtwORAHDlVJRZWYbCeRqra8HXgJm+zpQm4Q//Impv1racwxaE8etKTP5ePkGSkrkEkWImuBJSaMjsF9r/ROAUmoJMBBwfJnCQGCadfgjYIFSSulqqmyYPn8V4dPHMkG9y5ctp/Ll98B3wYTk/Q/NT9/A/+j/4bKQMBqE1qFBWB0uq1Ofy+rWo2HdBjRs2ICmlzemQZ36BAcFEhQMgUGK4OAAUAGgFAQEoAIsv1GgsH62tidQKsAyaG2GbRsGUAEB9nYHKsCyPmUbxjpfgEKhrMtjGea3yY69fgU4DOeeyydAKX45k1tumkLh8PG3uDyYZtuubZo9jlJxle6JzNfTHMc7W04YhydJozng+OKCbCCponm01kVKqTNAOPCrL4IsRyn+Nm0+j1yaw/bVq3jp03V8yhYKAy5yKPxHDl7mpAPhAuvPaeDnaomq+m23/BqXM8a/cfibVs6HAahoWpn5KjvNwELymnFpvuue6HyhRitClVKjgFHWj/lKqb0eLNaE6ko+3jFOHJ8YJA6/HQ97AbYJ6Fp+LH5TwE9N1CvKkziursp2PEkaRyj93HakdZyzebKVUkFAQywVoqVorRcBi7wJUCm1rSrPyfuKxGG8OIwQQ22Mw5PnNLYCrZRSLZVSIcAwoGyzyTTgfuvwncCa6qrPEEL4l9uShrWOYgywGsst17e11juVUjOwdCSbBrwFvKuU2g+cwpJYhBC/Qx7VaWitVwIry4yb6jB8ERjq29DsvLqcqUYSR2lGiMMIMUAti8Nv79MQQpiTqdqeCCEMQGtt2B+gD7AX2A9M8sH6rgLWYnkwbScw1jp+GpY7QJnWn74Oyzxp3f5eoLe72ICWwDfW8R8AIRXEkgX8aN3eNuu4y4EvgH3W342t4xUw37rOH4B2Duu53zr/PuB+h/Htrevfb11WOYmhtcM+ZwJ5wLiaOB7A28AJYIfDvNW+/2W2cQTIKRPDHGCPdTvLgEbW8dcAFxyOyUIvt+Vqfz5xcixq4m8Qav283zr9Go/OI38nBhcneCBwALgWCAG+B6KquM5mti8c0AD4L5ZH46cBTziZP8q63VDrgT9gjavC2IAPgWHW4YXAX1wkjSZlxj1v+2MDk4DZ1uG+wCrrl60T8I3Dl/In6+/G1mHbF/Nb67zKuuztHhzvX7Dcw6/24wF0B9qVOVGqff/LbGMhkFomhtuAIOvwbIcYrnGcr8xx8WRbrvZnl5NjURN/g79iTX5Ybl584Ml5ZOTLE/vj61rrAsD2+Hqlaa2Paa2/sw6fBXZjeZq1IgOBJVrrS1rrg1gycseKYlOWZ59vwfIoPcA/gEFehDjQukzZZQcC/9QWW4BGSqlmQG/gC631Ka11Lpb/Zn2s0y7TWm/Rlm/EPz2IoydwQGvt6nlZnx0PrfUGLHfaanr/HbcxHSj1mnOt9edaa9sbfbdgeS6pQl5sy9X+2E56T/jyO+kY30dAT+XB8/tGThrOHl93dYJ7RSl1DZCApVgGMEYp9YNS6m2llK0Hn4piqGh8OHDa4UvnKmYNfK6UyrA+KQvQVGt9zDr8C2B7Rba3cTS3Dpcd78owYLHD55o+HlAz+192GxEu4nkAS4nApqVSartSar1SypZsvNmWq/0p3X+CRXX/DUo1/wBszT9cMnLSqDZKqfrAx8A4rXUe8HfgOiAeOAa8WANh3KS1boel9fDDSqnujhOt/7Vq5NaW9aG9AcBS6yh/HI9SamL/XW1DKTUZKAJsndccA1porROAx4D3lVIe96BUif3x+9+gIkZOGp48vu41pVQwloTxntb63wBa6+Na62KtdQnwBpainqsYKhp/EkvROajM+HK01kesv09gqXDrCBy3Fndtxd4TlYzjCKWL1e6O3e3Ad1rr49aYavx4WNXE/pfdRrm2GkqpFKA/MMJ6smO9HDhpHc7AUn9wg5fbcrU/vzjGUEN/A/syrpp/lONJxYc/frA8ePYTlsoeW8VOdBXXqbBcc84rM76Zw/CjWK4ZAaIpXen0E5Zrzwpjw/Lf2rHS6a9O4qgHNHAY/hpLzfccSlecPW8d7kfpirNv9W8VgQexVAI2tg5frp1XzvV1cVyWAH+q6eNBmcrFmth/J9tYWCaGPlgqJiPKHKMIINA6fC2WE87bbVW4P06ORbX/DYCHKV0R+qFH55G/k4Obk7wvljscB4DJPljfTViKiD/gcCsLeBfLLbMfsLSjcfyDTbZufy8OdyAqis36hfoWSwXVUiDUSRzXWv+o32O59TvZOj4c+BLLLbp0hy+lwvIipAPWOBMd1vWAdVv7KX3iJwI7rMsswMktV+t89bD8d2noMK7ajweW+pNjQCGW6+yRNbH/Zbbxi/XHMYb9WK7zbd8P20k1xPq3ygS+A/7g5bZc7c9qJ8eiJv4GYdbP+63Tr/XkPJInQoUQXjFynYYQwoAkaQghvCJJQwjhFUkaQgivSNIQQnhFkoYQwiuSNIQQXpGkIYTwyv8DGbO8e3gPIrUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reproducibility\n",
    "\n",
    "The figure shown in [the problem section](#Experiment-Two:-Switched-Drifter-Distractor-Problem) follows the equivalent figure in the research paper.\n",
    "\n",
    "When it comes to the experiment results, the following match the figures shown in the research paper:\n",
    "- Weight Change\n",
    "- Bayesian Surprise\n",
    "- Squared Error\n",
    "- Error Reduction\n",
    "- Step-size change\n",
    "- Expected Error\n",
    "- Error Derivative\n",
    "\n",
    "The following were almost matching the research paper, however still differ.\n",
    "- Variance of Prediction\n",
    "- Uncertainty Change\n",
    "\n",
    "The following experiments results in this notebook did not show resemblance to the ones in the research paper:\n",
    "\n",
    "- UDE\n",
    "\n",
    "\n",
    "## Interpretation\n",
    "In our implementation of the second experiment, the first figure of the target data matches the one in the research paper. With a few exceptions, we were able to reproduce most of the results. \n",
    "\n",
    "As we played with different combinations of hyperparameters, we were able to reproduce most result and get very close with others. However, we ultimately could not recreate all the results. Variance of Prediction and Unvertainty Change were very similar to the paper, however the differed enough to seperate them from the other results. UDE differed completely. We did reach out to the authors of the paper to get their hyperparameters which greatly helped in getting closer results.\n",
    "\n",
    "We believe implementation of equations for each reward, along with our implementation of the actions and parameters is correct. However UDE was also differing in the original notebook. Our failing to reproduce certail results don't disprove the paper. To do that we would have to test a large combinations of hyperparameters, find ideal ones, and match those with the paper, instead of relying on the ones given by the researchers. However this was not possible for this project due to computational restrictions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}