3- Instance-wise feature importance.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Instance-wise feature selection with symbolic metamodels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we carry out the second experiment (Section 5.2) in our paper *\"Demystifying Black-box Models with Symbolic Metamodels\"* submitted to **NeurIPS 2019** by *Ahmed M. Alaa and Mihaela van der Schaar*. In this experiment, we evaluate the ability of symbolic metamodels to explain predictions in terms of instancewise feature importance (Section 4). To this end, we replicate the experiments in [1] with the following synthetic data sets: **XOR**, **Nonlinear additive features**, and **Feature switching**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Synthetic datasets\n",
    "\n",
    "The XOR, nonlinear additive features and feature switching datasets each has a 10-dimensional feature space. The description of each dataset is given below.\n",
    "\n",
    "- **2-dimensional XOR as binary classification**: The feature $x = (x_1,...,x_{10})$ is a 10-dimensional standard Gaussian $x \\sim \\mathcal{N}(0,1)$. The response $y$ is generated from $P(y = 1\\,|\\,x) \\propto \\exp(x_1\\cdot x_2)$.\n",
    "- **Nonlinear additive model**: The feature $x = (x_1,...,x_{10})$ is generated from a 10-dimensional standard Gaussian $x \\sim \\mathcal{N}(0,1)$, and the response $y$ is generated from $P(y = 1\\,|\\,x) \\propto \\exp\\left(−100 \\sin(2x_1) + 2|x_2| + x_3 + \\exp(-x_4)\\right)$.  \n",
    "- **Switch feature**: Generate $x_1$ from a mixture of two Gaussians centered at $\\pm 3$ respectively with equal probability. If $x_1$ is generated from the Gaussian centered at 3, the 2-5th dimensions are used to generate $y$ like the **orange skin** model explained below. Otherwise, the 6-9th dimensions are used to generate y from the nonlinear additive model. \n",
    "\n",
    "The **orange skin** model samples the feature  $x = (x_1,...,x_{10})$ is a 10-dimensional standard Gaussian $x \\sim \\mathcal{N}(0,1)$, and the response is generated from the functional form $P(y = 1 | x) \\propto \\exp(x_1 + x_2 + x_3 + x_4 − 4)$.   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The datasets above are commonly used in the feature selection literature [1]. The first dataset is designed specifically for feature interaction, whereas the third dataset is designed for instancewise feature selection. In all these datasets, ground-truth about feature importance and interactions are known."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can import all these synthetic datasets from the module **benchmarks.synthetic_datasets** in **pysymbolic** as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pysymbolic.benchmarks.synthetic_datasets import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next Section, we layout the experimental setup with more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experimental setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each of the three data sets above, we fit a 2-layer neural network $f(x)$ (with 200 hidden units) to predict the labels based on the 10 features. Since the underlying true features are known for each sample, we use the median feature importance ranking of each algorithm as its performance metric as in [2]. To explicitly fit the predictive model in each dataset, we use a Keras-based neural network model which can be imported from **algorithms.keras_predictive_models** in **pysymbolic** as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from pysymbolic.algorithms.keras_predictive_models import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will not explicitly fit the model to the datasets in this notebook as this will be done automatically inside the **get_instancewise_median_ranks** function as we will show later. In the following cell, we provide an example of how to generate a dataset and fit a model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x_train, y_train, x_test, y_test, datatype_val = create_data(\"XOR\", n = 1000)\n",
    "\n",
    "predictive_model   = get_predictive_model(x_train, y_train, model_type=\"keras\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following experiments, we will be explaning the predictions of the model **predictive_model** via instance-wise feature selection.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compare the performance of our symbolic metamodeling (SM) approach with the following benchmarks: \n",
    "\n",
    "- **DeepLIFT**: An activation propagation approach for feature importance that is specific to deep learning models [4]. \n",
    "- **SHAP**: A shapley value based approach to generlized additive models presented in [3].\n",
    "- **LIME**: Feature importance via local linear approximations [5].\n",
    "- **L2X**: (Learning to explain) An information-theoretic approach presented in [2].\n",
    "- **SR**: A symbolic regression approach [6]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of the benchmarks above are implemented in the module **algorithms.instancewise_feature_selection** in **pysymbolic**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the experiments\n",
    "\n",
    "As mentioned before, since the underlying true features are known for each sample, we use the median feature importance ranking of each algorithm as its performance metric as in [2]. In the following, we compute the median feature ranks for all the benchmarks above for all of the 3 datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Initialize dictionary of results for all methods\n",
    "\n",
    "from pysymbolic.algorithms.instancewise_feature_selection import *\n",
    "\n",
    "benchmark_dictionary = {'SM'      : {'XOR': [],'nonlinear_additive': [], 'switch': []},\n",
    "                        'SR'      : {'XOR': [],'nonlinear_additive': [], 'switch': []}, \n",
    "                        'LIME'    : {'XOR': [],'nonlinear_additive': [], 'switch': []},\n",
    "                        'DeepLIFT': {'XOR': [],'nonlinear_additive': [], 'switch': []},\n",
    "                        'L2X'     : {'XOR': [],'nonlinear_additive': [], 'switch': []},\n",
    "                        'SHAP'    : {'XOR': [],'nonlinear_additive': [], 'switch': []}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Now working on method: L2X for dataset: XOR ---\n",
      "WARNING:tensorflow:From C:\\Users\\ahmedmalaa\\Desktop\\Projects\\PySymbolic\\Symbolic Metamodeling\\pysymbolic\\algorithms\\instancewise_feature_selection.py:157: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
      "\n",
      "--- Now working on method: SHAP for dataset: XOR ---\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using 1000 background data samples could cause slower run times. Consider using shap.kmeans(data, K) to summarize the background as K weighted samples.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "61534c969f4140b9ac7df038cab98607",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=1000), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- Now working on method: LIME for dataset: XOR ---\n",
      "--- Now working on method: SR for dataset: XOR ---\n",
      "--- Now working on method: DeepLIFT for dataset: XOR ---\n",
      "--- Now working on method: SM for dataset: XOR ---\n",
      "---- Tuning the basis functions ----\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f07daa0d3f51474d888e0acb7af7e7a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=10), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "----  Optimizing the metamodel  ----\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0624c40cb6be435db109525aad3647c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=10), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- Now working on method: L2X for dataset: nonlinear_additive ---\n",
      "--- Now working on method: SHAP for dataset: nonlinear_additive ---\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using 1000 background data samples could cause slower run times. Consider using shap.kmeans(data, K) to summarize the background as K weighted samples.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f1a5ce45edde45b1a57605011d63cfdb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=1000), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- Now working on method: LIME for dataset: nonlinear_additive ---\n",
      "--- Now working on method: SR for dataset: nonlinear_additive ---\n",
      "--- Now working on method: DeepLIFT for dataset: nonlinear_additive ---\n",
      "--- Now working on method: SM for dataset: nonlinear_additive ---\n",
      "---- Tuning the basis functions ----\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "559910a7798a4b78a97945b4e75774d0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=10), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "----  Optimizing the metamodel  ----\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "023ed6cd8536487090b17de80765a34e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=10), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- Now working on method: L2X for dataset: switch ---\n",
      "--- Now working on method: SHAP for dataset: switch ---\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using 1000 background data samples could cause slower run times. Consider using shap.kmeans(data, K) to summarize the background as K weighted samples.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "95089e38af684e5cbf34276006f504cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=1000), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- Now working on method: LIME for dataset: switch ---\n",
      "--- Now working on method: SR for dataset: switch ---\n",
      "--- Now working on method: DeepLIFT for dataset: switch ---\n",
      "--- Now working on method: SM for dataset: switch ---\n",
      "---- Tuning the basis functions ----\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2a87f71279a84ecfae1400bdab3d85b2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=10), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "----  Optimizing the metamodel  ----\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d03c5553230040109b9bead1ccbc255d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=10), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "datasets              = ['XOR', 'nonlinear_additive', 'switch']\n",
    "methods               = ['L2X', 'SHAP', 'LIME', 'SR', 'DeepLIFT', 'SM'] \n",
    "num_samples           = 1000 \n",
    "\n",
    "num_selected_features = {'XOR': 2, \n",
    "                         'nonlinear_additive': 4, \n",
    "                         'switch': 5}\n",
    "\n",
    "for dataset in datasets:\n",
    "    \n",
    "    for method in methods:\n",
    "        \n",
    "        print('--- Now working on method: %s for dataset: %s ---' % (method, dataset))\n",
    "        \n",
    "        benchmark_dictionary[method][dataset] = get_instancewise_median_ranks(dataset_=dataset, \n",
    "                                                                              num_samples=num_samples, \n",
    "                                                                              num_selected_features=num_selected_features[dataset], \n",
    "                                                                              method_=method)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us compare the median ranks of all algorithms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3EAAADSCAYAAADkFEmRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dfZwddX33/9c74Z4EyQZMuckSLNZG9rKoW6tJxGwVohGpclE1rVZtNAbaWEEqxW29qVfkh/W2qSaCqVLRiCAgVWrCVTd6BRQM3mBitN4AiSYq5AZDuDGQz++Pmd2cbPZmdnfmzMzZ9/PxOI+cM2fOd95nztlP5jvzPTOKCMzMzMzMzKweJpQdwMzMzMzMzLJzJ87MzMzMzKxG3IkzMzMzMzOrEXfizMzMzMzMasSdODMzMzMzsxpxJ87MzMzMzKxG3IkzM7PKkjRDUkg6JH38X5JeV3aukZI0V9Ivhnj+05L+T3r/+ZJ+PMS87ZIekjSxiKxmVk2S3iHpk6N87b2SXpR3JiuPO3GWK0mT0kLxFw3TJkvaLOn89PEsSV+TtFvSg5L+U9LTG+afK2lfupGyW9KPJb2hjPdjZsNL/+Z/LenohmlvlLQ272VFxEsi4uq8262SiPh/EfG03sf9N74iYnNETIqIJ8pJaGajIWmOpNvTbZ8dkm6T9MdZXx8R74uIN6ZtHbCDy8Yfd+IsVxHxELAI+Kik49PJ7wfWR8T1kp4HrAG+BJwInAp8H7hN0lMamtoaEZOAY4CLgKskPQ0zq6pDgL8rO0QRvJFkZmMl6Rjgy8AyoA04CXgP8FiZuay+3Imz3EXEGuArwL9Kmgu8Evib9On3A/8RER+NiN0RsSMi/hH4FvDuAdqKiLgF2AE8oxn5zWxU/gW4RNKxAz2ZHoH/droH+tuSZjU8t1bSe9O90rslrZF03CDtrJXUuyf69ZLWSfqApJ2S7pH0koZ5nyRppaRtkn4p6f/0DkGU9PvpiIDtkh6Q9NnG7OnRr0sl3Q3sGagjJ+mjkrZI+q2kuyQ9v+G5I9Mhkjsl/RD4436vfaak76Tv91rgiIbn+oZeSvoM0A78Zzo64e2Ne+AlvVrS+n5tXyTp5vT+4en62ZweLV0h6ciB1q2ZFeoPACJiVUQ8ERGPRMSaiLhb0n2Sng0g6TXp3/fT08dvlHRTev/dkq5J2/tG+u+utDY8L53nTZI2pbXlh5Ke1ZDhDEl3p3X4WklHYLXlTpwV5SJgLnA9cElEbJN0FDALuG6A+b8AnNV/oqQJks4FjgN+WlxcMxuj9cBa4JL+T0hqI92xA0wFPgR8RdLUhtn+AngD8GTgsIHaGcSfAD8mqRHvB1ZKUvrc1cDjwGnAM4GzgTf2xgIuJxkRMBOYzsE7khYALwWOjYjHB1j2t4EzSPaqfw64rmGj6F3A76e3eUDf7/gkHQbcBHwmfe11wP8e6M1FxGuBzcDL0iGU7+83y83A0yQ9tWHaX6R5AK4g2Xg8I10PJwHvHGhZZlao/wGekHS1pJdImtLw3NdJtpkAzgR+Dryg4fHXB2jvzPTfY9Pa8E1Jf05Sx/6KZCTTucD2hte8EngxySioZwCvH+N7shK5E2eFiIidwEbgKOCGdHIbyXdu2wAv2UayEdbrREm7gEeAG4GLI+K7xSU2sxy8E1jSMJS610uBn0TEZyLi8YhYBfwIeFnDPJ+KiP+JiEdIduqckXGZ90XEVenvw64GTgCmSZoGvAR4a0TsiYjfAB8GXg0QET+NiFsj4rGIuJ+kY/mCfm3/a0RsSTMdJCKuiYjt6Xv6IHA40Dvs+5XA0nS0wRaSDmyv5wKHAh+JiL0RcT1Jh3DEIuJhkuHpCwDSztwfAjenndk3ARelOXYD7+tdB2bWPBHxW2AOEMBVwP2Sbk5r1dfZX3+eT7KDqffxCxi4EzeQNwLvj4hvpyOZfhoR9zU8/68RsTUidgD/SfY6axXkTpwVQtJrgBnA/yXZEwywE9hHspHV3wnAAw2Pt0bEsSR7kv4V+NPCwppZLiJiA8lvPv6h31MnAvf1m3YfyVGhXr9quP8wMCnjYvtel3ZoSF97CklHaZukXelOoU+QHOlD0pMlfT4dZvlb4BoO3JEEsGWoBUt6Wzps6cG0/Sc1tHFiv9c3vv8TgV9GRAzy/Eh9jrQTR3IU7qZ0XRxPsiPtroZ18NV0upk1WURsiojXR8TJQAdJLfgISSft+ZJ+D5gIXAvMljSDpK58L+MipgM/G+L50dZZqyB34ix3kp5Mssf7TcCbgVdKOjMi9gDfBP58gJe9Evjv/hMj4jHgUuB/SXp5canNLCfvIvnbb+ygbSXpVDVqB35ZYI4tJCcMOC4ijk1vx0TE6enzl5PsEX9GRBwDvIZkiGWjYBDp798uJaldU9KdTg82tLGNZIOqV3vD/W3ASQ3DPvs/39+gOVJrgOMknUHSmesdSvkAyWiG0xvWwZPSk0aZWYki4kfAp4GOiPgpSafqLcA30qPmvyI5Udy6iNg3UBMDTNtCMoTbxgF34qwI/0ayJ7gnIrYBbyc5u+ThJHvoXyfpLUouPTBFybWRnkdylqaDRMTvgA/i33GYVV66MXItycZIr1uAP5D0F+nJOF4FPJ3kqF1RObaRdG4+KOmY9Pe1vy+pd4jSZOAhkpMCnAT8/QgXMZnk93b3A4dIeifJyIFeXwAuS2vcycCShue+mb72Len6OA94zhDL+jXwlMGeTH+vdz3JyWXagFvT6ftIhm19ON25hqSTJM0b0Ts1szGT9Ifp0fuT08fTSXa6fCud5evA37J/6OTafo/7u59kdFNjbfgkyQmmnq3EaZL670CzFuFOnOUqPVo2h4YNooj4JPAL4J0RsY7kR/7nkeyNvo/khANzIuInQzT970C7pJcNMY+ZVcM/A33XjIuI7cA5wNtIfmT/duCciHhg4Jfn5q9ITpLyQ5Lh3Nezfzj3e4BnkRw9+wr7f7ub1Wrgv0hOVnAf8CgHDp98Tzr9HpLO5Gd6n0h3TJ1HclKBncCrhln+5cA/pkMiBzvhy+eAFwHX9TsJy6UkJ4X6Vjps9P+y/3d7ZtY8u0lOxHSHpD0knbcNJHURks7aZPafdbL/4wOkQ6aXklyiaZek50bEdem0z6XLu4lkx461IB04JN/MzMzMzMyqzEfizMzMzMzMasSdODMzMzMzsxpxJ87MzMzMzKxG3IkzMzMzMzOrEXfizMzMzMzMauSQsgMM5rjjjosZM2aUHcPMcnTXXXc9EBHHl51jLFybzFqPa5OZVdVg9amynbgZM2awfv36smOYWY4k3Vd2hrFybTJrPa5NZlZVg9UnD6c0MzMzMzOrEXfizMzMzMzMasSdOGsZq1atoqOjg4kTJ9LR0cGqVavKjlRLU6dORVLfberUqWVHshI1fhd6b1XT3t5+QL729vayIx1k3rx5TJgwAUlMmDCBefPmlR3JakDSRZI2StogaZWkI8rOZK2lDjXeBuZOnLWEVatW0d3dzbJly3j00UdZtmwZ3d3d7siN0NSpU9mxYwenn3469913H6effjo7duxwR26cavzP/Kabbhpwetna29vZsmULs2bNYuvWrcyaNYstW7ZUqiM3b9481qxZw+LFi9m1axeLFy9mzZo17sjZkCSdBLwF6IyIDmAi8OpyU1kraazlr3jFKwacbtVV2RObmI3E0qVLWblyJV1dXQB0dXWxcuVKlixZwoIFC0pOVx+9HbgNGzYAsGHDBjo6Oti4cWPJyaxMEdH3b9X+c+/twN12220A3HbbbcyePZvbb7+95GT73XrrrVxwwQV8/OMfB+j7d8WKFWXGsno4BDhS0l7gKGBryXmsBfXWeHAHrk5ashM3ki9g4xe3meqQsU42bdrEnDlzDpg2Z84cNm3aVFKi/er2Wd9yyy0HPT7llFNKSmNlazwC1/v45S9/eUlpBnb99dcf9PjEE08sKc3BIoLLL7/8gGmXX345y5cvLylRYqQba/7/srki4peSPgBsBh4B1kTEmsZ5JC0CFgFMmzaNtWvXjnp5vTtBs+jp6Rn1csbCGfP3nOc854DvzXOe8xzuvPPOMX2X8lCH9Vh2xpbsxA1UxCVVqrjXIWOdzJw5k3Xr1h3wB7Vu3TpmzpxZYqpE3T7r+fPn9x2J631s49fLX/7yA76rVevAAZx//vl9R+J6H1eJJC677LK+I3AAl112Wel7vAerQVWrT3WroXmRNAX4M+BUYBdwnaTXRMQ1vfNExJXAlQCdnZ0xd+7cUS+vDuvZGfN355130vi9ufPOOwEYy3cpD3VYj2Vn9G/irCV0d3ezcOFCenp62Lt3Lz09PSxcuJDu7u6yo9VKW1sbGzdupKOjg82bN/cNpWxrays7mpVIEl/60pdK73QMZPr06dx+++3Mnj2bbdu29Q2lnD59etnR+px11lksX76cCy+8kAcffJALL7yQ5cuXc9ZZZ5UdzartRcA9EXF/ROwFbgBmlZzJWpAkzjvvvErWeBtcSx6Js/Gn93dvS5YsYdOmTcycOZOlS5f693AjtH37dqZOncrGjRv7hlC2tbWxffv2kpNZGRp/A9d4BK5Ke0I3b95Me3s7t99+e98QyunTp7N58+aSk+23evVq5s2bx4oVK1i+fDmSOPvss1m9enXZ0azaNgPPlXQUyXDKFwK+mrflprHG33jjjQdMt+pzJ85axoIFC9xpy4E7bNaoDv+ZV6nDNhh32GykIuIOSdcD3wEeB75LOnTSLC91qPE2MHfizMzMzCooIt4FvKvsHGZWPf5NnGVShwtpL1myhCOOOAJJHHHEESxZsqTsSGZmZmZmuXMnzoZVhwtpL1myhBUrVvC+972PPXv28L73vY8VK1a4I2dmZmZmLcedOBtW44W0Dz300L4LaS9durTsaH2uuuoqrrjiCi6++GKOOuooLr74Yq644gquuuqqsqOZmZmZmeXKnTgbVpUvpN3rscceY/HixQdMW7x4MY899lhJiepFUuZbq5F0kaSNkjZIWiXpiLIzWeuZN28eEyZMQBITJkxg3rx5ZUcyM7MacyfOhtV7Ie1GVbmQdq/DDz+cFStWHDBtxYoVHH744SUlqpeIOOg21PRWIekk4C1AZ0R0ABOBV5ebylrNvHnzWLNmDYsXL2bXrl0sXryYNWvWuCNnZmaj5rNT2rB6L6S9cuVK5syZw7p161i4cGGlhlO+6U1v4tJLLwWSI3ArVqzg0ksvPejonNkADgGOlLQXOArYWnIeazG33norF1xwAR//+McB+v7tv+PJzMwsK3fibFh1uJD2smXLAHjHO97B2972Ng4//HAWL17cN91sIBHxS0kfILmo7iPAmohY038+SYuARQDTpk1j7dq1Tc3ZTF1dXSOav6enp6Akg6tDxkYRwfz58w/43syfP5/ly5dX9rtU1VyN6pDRzKwoqurwqM7Ozli/fn1u7Umq/FCwOmS0fNThsy4io6S7IqIz10bHQNIU4IvAq4BdwHXA9RFxzWCvybs21cV4/c7mYcKECSxevLjvCBzAhRdeyIoVK9i3b1+JyQZW1fXYKO+MVatNo1FEbRqP34Ui1CFjHdRhPTZz28m/iTOz8exFwD0RcX9E7AVuAGaVnMlazFlnncXy5cu58MILefDBB7nwwgtZvnw5Z511VtnRzMyspjyc0szGs83AcyUdRTKc8oXA+DvMZoVavXo18+bNY8WKFSxfvhxJnH322axevbrsaGZmVlM+EmeZTJ069YDTzE+dOrXsSAdpb28/IGN7e3vZkaziIuIO4HrgO8APSGrilaWGspa0evVq9u3bR0Swb98+d+DMzGxM3ImzYU2dOpUdO3Zw+umnc99993H66aezY8eOSnXk2tvb2bJlC7NmzWLr1q3MmjWLLVu2uCNnw4qId0XEH0ZER0S8NiJ8cUEzMzOrNHfibFi9HbgNGzbQ3t7Ohg0b+jpyVdHbgbvttts44YQTuO222/o6cmZmZmZmrcSdOMvklltuGfJxFVx//fVDPjYzMzMzawXuxFkm8+fPH/JxFZx//vlDPjYzMzMzawXuxNmw2tra2LhxIx0dHWzevJmOjg42btxIW1tb2dH6TJ8+ndtvv53Zs2ezbds2Zs+eze2338706dPLjmZmZmZmlit34mxY27dv7+vInXLKKX0duO3bt5cdrc/mzZv7OnInnnhiXwdu8+bNZUczG9SqVavo6Ohg4sSJdHR0sGrVqrIjWUF89lwzM8uTO3GWyfbt24mIvluVOnC9Nm/efEBGd+CsylatWkV3dzfLli3j0UcfZdmyZXR3d7sj14J89lwzM8tb0zpxki6StFHSBkmrJB3RrGWbmVXN0qVLWblyJV1dXRx66KF0dXWxcuVKli5dWnY0y5nPnmtmZnlrSidO0knAW4DOiOgAJgKvbsayzcyqaNOmTcyZM+eAaXPmzGHTpk0lJbIi+ey5ZmaWp2YOpzwEOFLSIcBRwNYmLtvMrFJmzpzJunXrDpi2bt06Zs6cWVIiK5LPnmsjJelpkr7XcPutpLeWncvMqqEpnbiI+CXwAWAzsA14MCLWNGPZZmZV1N3dzcKFC+np6WHv3r309PSwcOFCuru7y45mOfPZc200IuLHEXFGRJwBPBt4GLix5FhmVhGHNGMhkqYAfwacCuwCrpP0moi4pt98i4BFANOmTWPt2rW55si7vSJUKWNXV1fmeXt6egpMMrg6ZBxMlT7rwdQhY10tWLAAgCVLlrBp0yZmzpzJ0qVL+6Zb69i8eTPt7e19Z88FfPZcG6kXAj+LiPvKDmJm1aCIKH4h0p8DL46IhenjvwKeGxEXDvaazs7OWL9+fZ4ZaMZ7HQtnzIcz5qOIjJLuiojOXBttsrxrU12M1+/seFSH9Zh3xqrXJkn/DnwnIv5tsHmKqE3j8btQhDpkrIM6rMdmbjs15UgcyTDK50o6CniEZI/S+NsKMjMzMxsBSYcB5wKXDfBcoSOYoB4jMpxxZKo2iuncc89l9+7dmeaVNOw8kydP5uabbx5rrFFr1mfdlCNxAJLeA7wKeBz4LvDGiHhssPl9JK6anDEf4zVj1fd2Z+EjcdVVh4x1UIf1OJ6OxEn6M+BvIuLsoebzkbjqcsbmLruV3kvaZqlH4oiIdwHvatbyzMzMzFrAAmBV2SHMrFqaeYkBMzMzM8so/RnKWcANZWcxs2pxJ87Mxq2yr8M0adIkJPXdJk2a1KxFm1kNRMTDETE1Ih4sO4uZVYs7cWY2bpV5HaZJkyaxZ88eZsyYwU9/+lNmzJjBnj173JEzMzOzYTXtN3FmZhXX1Osw9Xbg7rnnHgDuueceTj31VO69995mLN7MzMxqzJ04M7PEqxnk5AFFncb7ve997wFtvfe97+W1r31tpU5F3aiquRpVJeNITuENzTmN90hUZT0OpQ4ZzcyKkukSA5KeHhE/HGD6vIhYXUQwX2KgmpwxH+M1YxGn8c6jPqXXYdoKnB4Rvx5q3rxqk6QDjsQBfUfiqvjdGK/f2bw5Yz7qcImBZm87+RID1eWMzV12K72XtM0B61PW38R9WdKp/Rp8GfDpHLKZmY1FHvXpJcB3huvA5enoo4/m3nvv5dRTT+VnP/tZXwfu6KOPblYEMyuWt53MrDBZO3F/D6yWdAKApPOATwDnFBXMzCyjPOpT06/D9NBDD/V15E477bS+DtxDDz3UzBhmVhxvO5lZYTL9Ji4ivijpGOBWSR8D/gl4cUTcXWg6M7NhjLU+NVyH6c0FxhyQO2xmrcvbTmZWpEE7cZL6H6W7GmgD3gmcDWyUNCEi9hWYz8zsIHnWp4h4GJiaf0ozG2+87WRmzTLUkbjHgf6/zFP67/fS+wFMLCCXmdlQXJ/MrIpcm8ysKYbqxJ06xHNmZmVyfTKzKnJtMrOmGLQT16wL3pqZjZTrk5lVkWuTmTVLphObSGoDLgHOACY1PhcRZxaQy6yW2tra2LlzZ6Z5JQ07z5QpU9ixY8dYY7U01yczqyLXJjMrUqZOHPA54HDgC8DDxcUxq7edO3fmfsFKG5brk5lVkWuTmRUmayduFnB8RDxWZBgzs1FwfTKzKnJtMrPCZL3Y993AyUUGMTMbJdcnM6si1yYzK0zWI3FfA74q6VPArxqfiIh/zz2VmVl2talPIx0em+fQXDNrutrUJjOrn6yduOcDvwDO6jc9ABciMytTberTYJ0ySe6wmbWe2tQmM6ufTJ24iOgqOoiZ2Wi4PplZFbk2mVmRsh6J66NkPFDfmKCI2JdrIjOzUXJ9MrMqcm0ys7xlOrGJpJMk3ShpO/A4sLfhZmZWGtcnM6si1yYzK1LWs1OuAH4HvBB4CHgWcDOwuKBcZmZZuT6ZWRW5NplZYUZynbj2iNgjKSLi+5IWArcDVxUXrzW0tbWxc+fOTPNmOXvdlClT2LFjx1hjmbUK1yczq6Ix1yZJxwKfBDpITojy1xHxzcISm1ltZO3EPUEyFABgl6Tjgd8CJxWSqsXs3Lkz1zPPjfQ05WYtzvXJzKooj9r0UeCrEXG+pMOAo3LOaGY1lXU45R3A/PT+auBa4AZgfRGhzMxGYEz1SdKxkq6X9CNJmyQ9r6CcZja+jLU2HQOcCawEiIjfRcSuAnKalUci0n/zuvW11+KyHol7Lfs7fG8F3gZMBj5SRCgzsxEYa33ynm4zK8JYa9NTgPuBT0n6I+Au4O8iYk/vDJIWAYsApk2bxtq1azM1fO6557J79+5M82YZ/TN58mRuvvnmTO1l5Yz5qHrGubm1dLCsfw9ZVHE9arhhfpImklyUclFEPDampY1AZ2dnrF+f34G+Mi+mm/eyW+m9FKGV1k8rvZe0zbsiojPH9sZUn9I93d8HnhIZ32zetSnN4b+rHDhjPsZjxqrVprSNTuBbwOyIuEPSR4HfRsQ/DTT/SGpTHf6vcsZqtpl7xiKPmFX4fY+kvcHq07DDKSPiCeBsYEzXNPGQJTPLWw71qXFP93clfVLS0bkFNLNxKadtp18Av4iIO9LH15Oc4dKsdUQkF1CMyO3W116Lyzqc8sPAeyS9KyJGe30TD1kysyKMpT4dQrJRtKRhT/c/AAfs6R7tkKWRKKLNvDljPpwxHzXIOKZtp4j4laQtkp4WET8muVTBD3NPaWa1NOxwSgBJW4DfIznT0v0kp7kFICLaM7y+9CFLrTQsrZXeSxFaaf200ntJ28x1yFLa5qjrk6TfA74VETPSx88H/iEiXjrYazycsrqcMR/jMWPValNDG2eQXGLgMODnwBsiYsBrFnk4ZXPbK6JNZ6xme0W0mcdwyqxH4l4zkmADGPbHuWZmozTq+uQ93WZWoLFuOxER3wNy7VyaWWvI1ImLiK/nsJxChixV8WwxA8l72EeZw0hqMISlpdZP3u3V5W8mqxzq0xLgs+kw758Dbxh7KjMb73KoTWZmg8o0nHLMCylwyJIPwTbXeBxmU+ayW+37WMSQpWbzcMrqcsZ8jMeM4602+f+qarbpjNVsr4g2m3J2yjxExK+ALZKelk7ykCUzMzMzM7NRyPqbuDx4yJKZmZmZmdkYNa0T5x/nmpmZmZmZjV2mTpykNuAS4AxgUuNzEXFmAbnMzDJxfTKzKnJtMrMiZT0S9zngcOALwMPFxTEzGzHXJzOrItcmMytM1k7cLOD4iHisyDBmZqPg+mRmVeTaZGaFyXp2yruBk4sMYlZ7EpH+m9etrz0biuuTmVWRa5OVy9slLS3rkbivAV+V9CngV41PRMS/557KzCw71ycrVVtbGzt37sw8vzJsAE2ZMoUdO3aMJdYB6pCxBbk2mVlhsnbing/8Ajir3/QAXIjMrEyuT1aqnTt3FnJh2TzVIWMLcm0ys8Jk6sRFRFfRQcxqLwJJuW4o5d1eK3J9MrMqcm2y0nm7pKWN+DpxSna99e1+i4h9uSYyMxsl1yczqyLXJjPLW6YTm0g6SdKNkrYDjwN7G25mZqVxfTKzKnJtMrMiZT075Qrgd8ALgYeAZwE3A4sLymVmlpXrk5lVkWuTmRVmJNeJa4+IPZIiIr4vaSFwO3BVcfHMzIbl+mRmVeTaZGaFyXok7gmSoQAAuyQdD+wBTioklZlZdq5PZlZFrk1mVpisnbg7gPnp/dXAtcANwPoiQpmZjYDrk5lVkWuTmRUm63DK17K/w/dW4BJgEvCRIkJlVsCV4/vay+v0qXXIaONHa34fx1SfJN0L7Cbdax4RnQVkNLPxp5rbTmbWErJeJ25Xw/1HgPcWlshK0dbWxs6dOzPNm+UCr1OmTGHHjh1jjXWAOmS05supPnVFxAP5pTKz8c7bTmZWpEE7cZK6I2Jpev+fB5svIt5ZRDBrrp07d+Z+Mci81SGjNYfrk5lVkWuTmTXLUEfiTm64P73oIKNShyvR1yGjjR+t833Msz4FsEZSAJ+IiCvH2J6NJwUMUYZKDFO20cl128nDvc1sMIN24iLigob7b2hOHDOz4eVcn2ZHxFZJTwZulfSjiPhG4wySFgGLAKZNm8batWvHuMiDFdHmcM4991x2796def4sR68nT57MzTffPJZYB6h6xrm5tDK4PL8XVf/eVv2zzqKgbScP9zazg2iwPeiSnpKlgYj4ea6JUp2dnbF+/fAncKrDUQVnrGabztj89iTdlcee5KLqk6R3Aw9FxAcGmydrbRrhcks5ut5q369S2ix6WHZOOSu/HgtobyRtVrU2pUfiOrN04kZSm1rpsyurvUL/9iv8vv1Z52wM9Wmo4ZQ/JRnRofTfvrb6PZ6YLaWZWW5yqU+SjgYmRMTu9P7ZwKC/YzE7SAFDlMHD5mss722nIYd7j2WUQN5HZqt+pDfv9ubm1tLB8n7feZ4DYPLkyYV81lXOODe3lg42lpxDDafsu4acpDcALwLeDdwHnAK8E/jvUS/ZzGyUcqxP04Ab0/88DgE+FxFfzTuvmY0PBWw7DTncO+3UXQnJkbi5c+dmbngk85bRXhFtFpGxCHnmHMFImdJ2HNUhY1HG8llnvU7ce4GnpqfIBfiJpDcD/wN8etRLNzMbu1HXp3RI0x8VG8/MxqkxbztFxNb0399IuhF4DvCNoV9lTdE6Jwqz4VT0s54w/Cx9883oN+0UPJTSzMrn+mRmVTSm2iTpaIBQKQkAABHhSURBVEmTe++TDPfekGdAM6uvrEfiPgx8TdKngC0kp819fTrdzBrkOa57ypQpubXVwlyfzKyKxlqbPNzbzAaVqRMXEf8i6QfAnwPPBLYBf+1iYnag8TyuuyyuT2ZWRWOtTR7ubWZDyXokjrToeKPIzCrH9cnMqsi1ycyKkuk3cZIOl7RU0s8lPZhOO1vS3xYbz8xsaK5PZlZFrk1mVqSsJzb5MNAB/CX7r3OyEbigiFBmZiPg+mRmVeTaZGaFyTqc8hXAaRGxR9I+gIj4paSTiotmZpaJ65PZUKSkB5HjSZeA/W36972DcW0ys8JkPRL3O/p1+CQdD2wfycIkTZT0XUlfHsnrzMyGkEt9yltbWxuSMt3SzMPe2traynxLNoSsn3XWm89M2xIqWZvMrDVk7cRdB1wt6VQASScA/wZ8foTL+ztg0whfY2Y2lLzqU6527txJROR627lzZ5lvyQYxks8w6/w7duwo+V1ZDipZm8ysNWTtxL0DuBf4AXAs8BNgK/CerAuSdDLwUuCTI4tohWscapPTLfehO3XIaGUZc30ya2kRKP03z1tfmzYY1yYzK0zW68T9Dngr8NZ0KMADMfKLXH0EeDswebAZJC0CFgFMmzaNtWvXZmo463xZ5d0e5HsB6MmTJ+eacW5uLR0sr5xzc2llYEV83lVeLlT7+zhSOdUnM7NcuTaZWZGG7MRJah/kqem9G4ERsXm4hUg6B/hNRNwlae5g80XElcCVAJ2dnTF37qCzHiDrfFnl3d54vgB03uuyCGVlLGu5rfJ9zKs+mZnlybXJzJphuCNx97L/tLgD7boPYGKG5cwGzpU0HzgCOEbSNRHxmqxBrUARuW+w594BqENGa7Z7yac+mZnl6V5cm8ysYMP9Ju5ukjHc/wicAhza73ZYloVExGURcXJEzABeDXzNHTgzG6Nc6pOZWc5cm8yscEN24iLiDOB8oA1YB9xC0gk7LCKeiIgnio9oZnYw1yczqyLXJjNrhmHPThkRGyLi74FTgQ8B5wDbJD1rNAuMiLURcc5oXmtm1ijv+mRmlgfXJjMrWtZLDAA8FXgB8Dzgu4AvWGRmVTGm+iRpoqTvSvpyEeHMbNzytpOZFWK4s1O2AQuA15FcGuAzwJk+q5KZlS3n+vR3wCbgmPwSmtl45G0nM2uG4c5OuRW4h6QAfSuddpqk03pniIivFZTNzGwoudQnSScDLwWWAhcXkLOaCrrgfV+bPvOrjV/edjKzwg3XifsVySUB3pTe+gvgKXmHGqk8L1w8ZcqU3Noys0LlVZ8+ArydZI/5gCQtAhYBTJs2LfPFzYu4CHpebc7NpZXB5fneq7we67TsOqzHOmTMoBbbTmZWb6rqdbI6Oztj/fr1ubVXh2uClZmxDtdgq0PGOiw7q4I+w7siojPXRsdA0jnA/Ii4UNJc4JLhTryUtTZV/m8g5yNwB8kpZ+XXY02WXYf1WGbGqtWm0RjJdlMrfXZltVdEm+OxNo1EK62fkbQ3WH0a7kicmVkrmw2cK2k+yZ7zYyRdMy6uYxlRi40aMzMzO9hIzk5pZtZSIuKyiDg5ImaQXMfpa+OiA2dmZma15k6cmZmZWUX5EihmNhAPpzQzAyJiLbC25BhmZv35EihmdhAfiTMzMzOroIZLoHyy7CxmVi0+EmdmZmZWTcNeAmXECrhGZO7Xh6xDRrOSuRNnZmZmVjHpJVB+ExF3pZdAGWieEV/DcsCGclKH61jmfd3APK9VPHny5KZc17Crq2vA6QO9l56enqLjjEiZ1/is2mftTpyZmZlZ9Qx7CZSIuBK4EpLrxM2dO7eUoL3KXn4WeWYcwXW+KnXplSplGamyvmNV/KzdibM+ee5hmDJlSm5tNapDRjMzs7GKiMuAywDSI3GX5HIJlAKuEZn7hmsdMpqVzJ04A6q5h6G/OmQ0MzMzMyuaO3FmZnkr4Ef5MA5/mO/1OH74sx6SL4FiZv25E2dmZpazoYZ+D/RcM0YP5DkcHTwk3cysTO7EmZmZ5axqQ7pHksdD0s3Mqs+dODOzvBXwo3wYhxvXXo/jhz9rM7MRmVB2ADMzMzMzM8vOnTgzMzMzM7MacSfOzMzMzMysRtyJMzMzMzMzqxF34szMzMzMzGrEnTgzMzMzM7MacSfOzMYtSUdIulPS9yVtlPSesjOZmZmZDcfXiTOz8ewx4E8j4iFJhwLrJP1XRHyr7GBmZmZmg3EnzszGrUiuAvxQ+vDQ9OYrA5uZmVmlNaUTJ2k68B/A7wH7gCsj4qPNWLaZ2VAkTQTuAk4DPhYRdwwwzyJgEcC0adNYu3Zt1rbzCwpMnjw587KzckYbSFnrp4jl+rM2s1bUrCNxjwNvi4jvSJoM3CXp1oj4YZOWb2Y2oIh4AjhD0rHAjZI6ImJDv3muBK4E6OzsjLlz52ZpN3MGSSOaPy/OaIPJ8h2vy3LLei9mZkVqyolNImJbRHwnvb8b2ASc1Ixlm5llERG7gLXAi0uOYmZmZjakpv8mTtIM4JlAbkOWsqrDkApnHJmurq4Bpw80/Kqnp6foOCNSpfU4mDpkHAtJxwN7I2KXpCOBFwFXlBzLzMzMbEhq5tATSZOArwNLI+KGoebt7OyM9evX57nsyg+zccbxow7rsYiMku6KiM5cGx0DSc8ArgYmkoxM+EJE/PNQr8m7NqU5xuX3IW91yFgHZa3HgmpOpjarVptGYyS1Ke91XeZnV1Z7dVl2K6nDemzmtlPTjsSlp+/+IvDZ4TpwZmbNEBF3k4wMMDMzM6uNpvwmTsnYtpXApoj4UDOWaWZmZmZm1oqa0okDZgOvBf5U0vfS2/wmLdvMzMzMzKxlNGU4ZUSsA/K90I+ZmZmZmdk41KwjcWZmZuPWqlWr6OjoYOLEiXR0dLBq1aqyI1nFSTpC0p2Svi9po6T3lJ3JWo+kg25WD+7EmZmZFWjVqlV0d3ezbNkyHn30UZYtW0Z3d7c7cjacx4A/jYg/As4AXizpuSVnshbS2GG76KKLBpxu1eVOnJmZWYGWLl3KypUr6erq4tBDD6Wrq4uVK1eydOnSsqNZhUXiofThoemt2udXt1qKCD70oQ9V/vT9dqCmX+y7GQbbgzDQ9DKvGZJ1ujPWWx3WYx0ytoKh9m5WZV07Y/42bdrEnDlzDpg2Z84cNm3aVFKiRBXXY95HAKZMmZJre80maSJwF3Aa8LGIuKPf84uARQDTpk1j7dq1I2k7t5yTJ08e0bKzqkPG/rq6ugacPtB76enpKTrOsM4777wD1st5553HDTfc0JR1NZQ6rMeyM7ZkJ67s/7CzcMbxow7rsQ4ZW0Ed1rMz5m/mzJmsW7fugP/w161bx8yZM0tMVb31WLU8VRARTwBnSDoWuFFSR0RsaHj+SuBKSC72PXfu3KztFpA2X3XIOJC65b7hhhv44he/2Pe4t05l/S4VpQ7rseyMHk5pZmZWoO7ubhYuXEhPTw979+6lp6eHhQsX0t3dXXY0q4mI2AWsBV5cchRrQZK4+OKL/Vu4mmnJI3FmZmZVsWDBAgCWLFnCpk2bmDlzJkuXLu2bbjYQSccDeyNil6QjgRcBV5Qcy1pIRPR13D784Q8fMN2qz504MzOzgi1YsMCdNhupE4Cr09/FTQC+EBFfLjmTtRh32OrLnTgzMzOziomIu4Fnlp3DzKrJv4kzMzMzMzOrEXfizMzMzMzMakRVHQsr6X7gvhybPA54IMf2iuCM+XDGfBSR8ZSIOD7nNpuqgNoE4/f7kDdnzMd4zOjaNLDx+F0ogjPmY7xmHLA+VbYTlzdJ6yOis+wcQ3HGfDhjPuqQsVXUYV07Yz6cMR91yNgK6rCenTEfzpiPZmb0cEozMzMzM7MacSfOzMzMzMysRsZTJ+7KsgNk4Iz5cMZ81CFjq6jDunbGfDhjPuqQsRXUYT07Yz6cMR9NyzhufhNnZmZmZmbWCsbTkTgzMzMzM7Paq1wnTtITkr4naaOk70u6WFLuOSWtldTZb9pcSV9O779e0v1plu9J+g9JH0vv/1DSIw3PnT/IMrrT93F3Ot+f9F+upBmSNvR73Ucl/bLxfffL80NJb8p5fTw0wLR3S7okvf9pSQ9LmtwvZ0g6Ln38RMM6+Z6kf8gz4zBZL07Xy92S/lvSKen0TkkbJB2WPv59ST+XdEwR2QbJO9j34Mfpd/zbks5oVp4R5NssSQ3z3TTQuh8vWqk2pe3Uoj65NhXL9ak1tFJ9cm0qRt3qk2vT8A4potExeiQizgCQ9GTgc8CTgHeVkOXaiPjb/hMlzQC+3JtzIJKeB5wDPCsiHkv/YA8bboFp8XkFsAU4E1jbP0+6XjZKujkifj2C9zNWPwX+DLgmzdkF/LLh+UeGWicF+y7QGREPS7oAeD/wqohYL+kbwCXA+4CPAd0R8dtmhBrme/CXab43AP8CnNWMTCPItwuYDayTdCxwQrPzVUxL1KZ0vlarT65No+D61FJaoj65NjVdJeuTa1M2lTsS1ygifgMsAv5WiYmS/iXtfd8t6c2980r6+4bp70mnzZD0I0lXp9Ovl3RUk+KfADwQEY+l7+WBiNia4XVdwAZgObBgoBnS9fIz4JScsma1CnhVen8ucBvweJMzDCgieiLi4fTht4CTG55+B/BGSW8HDo2IVU2MluV78E3gpCZmajRUvs8Dr07vnwfcUEK+Sqp5bYLWq0+uTaPj+tSCal6fXJuaqML1ybUpg0p34gAi4uckOZ8MLAQejIg/Bv4YeJOkUyWdDTwVeA5wBvBsSWemTTwNuDIingH8FrhwBIt/lfYf4n7DCKOvAaZL+h9JH5f0gobnPtvbLnBLv9ctIPmjvxE4R9Kh/RuW9BTgKSR7eJrpJ8DxkqaQ5Px8v+eP1IHDAl51cBNNsRD4r94HEbELuAK4nJF9/nkY6nvQ68XATU3O1WuofP8NnClpIklBuraUhBVV49oErVefXJtGx/WpRdW4Prk2ladK9cm1KYMqDqccSO/Y0rOBZ2j/OOonkRSgs9Pbd9Ppk9Lpm4EtEXFbOv0a4C3ABzIud8AhAVlExEOSng08n2QP0bXaP9b5LyNiPewfXpDePwyYD1wUEbsl3ZG+r6+kr3uVpDnAY8CbI2LHaLKN0Q0kX8o/Ad7c77myhwUg6TVAJ9D/D/4lwK+BpwM/blaeYb4Hn5V0NDAReFazMo0g3xPAOpK9iEdGxL3aP8zbErWrTdCy9cm1aYRcn1pe7eqTa1M5qlafXJuyqXwnLt1z8gTwG5KCtCQiVvebZx5weUR8ot/0GUD/ayg07ZoKEfEEybjstZJ+ALxumJe8mKS4/iD9wI8CHmZ/IRrThltOPg98B7g6IvZV6T9NSS8CuoEX9B7iTqefQ7Je5wE3SlrdMHygcEN8D/4S+D7w/5GMNz+vWZky5oPk874ReHfzk1VbnWsTtGR9cm0aBden1lTn+uTa1FxVrU+uTcOr9HBKSccDK4B/i4gAVgMX9B4ml/QHaW98NfDXkial009S8gNWgHYlP0CE5FD2uiZlf5qkpzZMOgO4b5iXLQDeGBEzImIGcCpwtpr7W5khRcRmkj/2j5edpZGkZwKfAM5Nx733Tj8S+CDwNxHxA+BLJPmblWvI70FE7AX+EXiupJnNypU1H/D/SIZSNPu3OpVW59qU5mi5+uTaNKpsrk8tqM71ybWpuapan1ybsqnikbgjlYx3PpTkx5+fAT6UPvdJYAbwHSW7Mu4HXh4Ra9IP8ZvpHo6HgNeQ7IXaBLxO0idIxiYvb1jWVyTtTe9/k6RHn5dJwDIlZ6Z5nGQM9iLg+oFmTovNPBoOtUfEHknrgJflmGswR0n6RcPjDw02Y/+9dg16P7teX42IIk6XO1DW+STr/Lr0O7A5Is4F/gm4KSJ+mM77buB7kj4dET8pIFt/w34PIuIRSR8kOQvUwiZkypwv3QDIOoSm1bVKbYJ61SfXpuK4PrWOVqlPrk3F1CaoV31ybcpAyXJak/afzraj5ChmZn1cm8ysqlyfzOqh0sMpzczMzMzM7EAtfSTOzMzMzMys1fhInJmZmZmZWY24E2dmZmZmZlYj7sSZmZmZmZnViDtxZmZmZmZmNeJOnJmZmZmZWY24E2dmZmZmZlYj/z+ou3dd22WOKQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(15, 3))\n",
    "\n",
    "baselines = ['DeepLIFT', 'SHAP', 'LIME', 'L2X', 'SR', 'SM']\n",
    "\n",
    "XOR_results                = [benchmark_dictionary[baseline]['XOR'] for baseline in baselines]\n",
    "nonlinear_additive_results = [benchmark_dictionary[baseline]['nonlinear_additive'] for baseline in baselines]\n",
    "switch_results             = [benchmark_dictionary[baseline]['switch'] for baseline in baselines]\n",
    "\n",
    "axes[0].boxplot(XOR_results, medianprops = dict(linewidth=3, color='red'))\n",
    "axes[0].set_title('XOR', fontsize=12)\n",
    "\n",
    "axes[1].boxplot(nonlinear_additive_results, medianprops = dict(linewidth=3, color='red'))\n",
    "axes[1].set_title('Nonlinear additive', fontsize=12)\n",
    "\n",
    "axes[2].boxplot(switch_results, medianprops = dict(linewidth=3, color='red'))\n",
    "axes[2].set_title('Switch', fontsize=12)\n",
    "\n",
    "for ax in axes:\n",
    "    ax.yaxis.grid(True)\n",
    "    ax.set_xticks([y+1 for y in range(len(XOR_results))])\n",
    "    ax.set_ylabel('Median rank', fontsize=12)\n",
    "\n",
    "plt.setp(axes, xticks=[y+1 for y in range(len(XOR_results))],\n",
    "         xticklabels=baselines)\n",
    "plt.show() \n",
    "\n",
    "fig.savefig('boxplot_comparison.pdf', dpi=200,  bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "[1] Jianbo Chen, Mitchell Stern, Martin J Wainwright, and Michael I Jordan. Kernel feature selection via conditional covariance minimization. *In Advances in Neural Information Processing Systems (NeurIPS)*, pages 6946–6955, 2017.\n",
    "\n",
    "[2] Jianbo Chen, Le Song, Martin J Wainwright, and Michael I Jordan. Learning to explain: An information-theoretic perspective on model interpretation. *International Conference on Machine Learning (ICML)*, 2018.\n",
    "\n",
    "[3] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. *In Advances in Neural Information Processing Systems (NeurIPS)*, pages 4765–4774, 2017.\n",
    "\n",
    "[4] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. *In International Conference on Machine Learning (ICML)*, pages 3145–3153, 2017.\n",
    "\n",
    "[5] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Explaining the predictions of any classifier. *In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining*, pages 1135–1144. ACM, 2016.\n",
    "\n",
    "[6] Yiqun Wang, Nicholas Wagner, and James M Rondinelli. *Symbolic regression in materials science*. arXiv preprint arXiv:1901.04136, 2019.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tensorflow",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}