examples/quickstart_autogluon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FairPredictor Autogluon Examples\n",
    "This file contains demo code for an extended version of the example in Readme.md (additionally handling more fairness over multiple groups), and enforcing a range of fairness definition on COMPAS.\n",
    "\n",
    "FairPredictor is a postprocessing approach for enforcing fairness, with support for a wide range of performance metrics and fairness criteria, and support for inferred attributes, i.e., it does not require access to protected attributes at test time. \n",
    "Under the hood, FairPredictor works by adjusting the decision boundary for each group individually. Where groups are not available, it makes use of inferred group membership to adjust decision boundaries.\n",
    "\n",
    "The key idea underlying this toolkit is that for a wide range of use cases, the most suitable classifier should do more than maximize some form of accuracy.\n",
    "We offer a general toolkit that allows different measures to be optimized and additional constraints to be imposed by tuning the behavior of a binary predictor on validation data.\n",
    "\n",
    "For example, classifiers can be tuned to maximize performance for a wide range of metrics such as:\n",
    "\n",
    "* Accuracy\n",
    "* Balanced Accuracy\n",
    "* F1 score\n",
    "* MCC\n",
    "* Custom utility functions\n",
    "\n",
    "While also approximately satisfying a wide range of group constraints such as:\n",
    "\n",
    "* Demographic Parity (The idea that positive decisions should occur at the same rates for all protected groups, for example for men at the same rate as for women)\n",
    "* Equal Opportunity (The recall should be the same for all protected groups)\n",
    "* Minimum recall constraints (The recall should be above a particular level for all groups)\n",
    "* Minimum Precision constraints (The precision should be above a particular level for all groups)\n",
    "* Custom Fairness Metrics\n",
    "\n",
    "The full set of constraints and objectives can be seen in Readme.md "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:18:59.770976Z",
     "iopub.status.busy": "2024-06-17T19:18:59.770525Z",
     "iopub.status.idle": "2024-06-17T19:19:14.708127Z",
     "shell.execute_reply": "2024-06-17T19:19:14.706485Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/miniconda3/envs/ag/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No path specified. Models will be saved in: \"AutogluonModels/ag-20240617_191908\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No presets specified! To achieve strong results with AutoGluon, it is recommended to use the available presets.\n",
      "\tRecommended Presets (For more details refer to https://auto.gluon.ai/stable/tutorials/tabular/tabular-essentials.html#presets):\n",
      "\tpresets='best_quality'   : Maximize accuracy. Default time_limit=3600.\n",
      "\tpresets='high_quality'   : Strong accuracy with fast inference speed. Default time_limit=3600.\n",
      "\tpresets='good_quality'   : Good accuracy with very fast inference speed. Default time_limit=3600.\n",
      "\tpresets='medium_quality' : Fast training time, ideal for initial prototyping.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Beginning AutoGluon training ... Time limit = 5s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will save models to \"AutogluonModels/ag-20240617_191908\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "=================== System Info ===================\n",
      "AutoGluon Version:  1.1.0\n",
      "Python Version:     3.10.13\n",
      "Operating System:   Darwin\n",
      "Platform Machine:   arm64\n",
      "Platform Version:   Darwin Kernel Version 23.5.0: Wed May  1 20:14:38 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6020\n",
      "CPU Count:          10\n",
      "Memory Avail:       6.56 GB / 16.00 GB (41.0%)\n",
      "Disk Space Avail:   360.82 GB / 460.43 GB (78.4%)\n",
      "===================================================\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Rows:    39073\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Columns: 14\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Label Column:       class\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t2 unique label values:  [' <=50K', ' >50K']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tIf 'binary' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Problem Type:       binary\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Preprocessing data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Selected class <--> label mapping:  class 1 =  >50K, class 0 =  <=50K\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tNote: For your binary classification, AutoGluon arbitrarily selected which label-value represents positive ( >50K) vs negative ( <=50K) class.\n",
      "\tTo explicitly set the positive_class, either rename classes to 1 and 0, or specify positive_class in Predictor init.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Feature Generators to preprocess the data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting AutoMLPipelineFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tAvailable Memory:                    6735.36 MB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Original)  Memory Usage: 21.86 MB (0.3% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 1 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting AsTypeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tNote: Converting 1 features to boolean dtype as they only contain 2 unique values.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 2 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting FillNaFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 3 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting IdentityFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting CategoryFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tFitting CategoryMemoryMinimizeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 4 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropUniqueFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 5 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropDuplicatesFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in original data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])    : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('object', []) : 8 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in processed data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('category', [])  : 7 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])       : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', ['bool']) : 1 | ['sex']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.1s = Fit runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t14 features in original data used to generate 14 features in processed data.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Processed) Memory Usage: 2.09 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Data preprocessing and feature engineering runtime = 0.17s ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTo change this, specify the eval_metric parameter of Predictor()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Automatically generating train/validation split with holdout_frac=0.0639828014229775, Train Rows: 36573, Val Rows: 2500\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "User-specified model hyperparameters to be fit:\n",
      "{\n",
      "\t'NN_TORCH': {},\n",
      "\t'GBM': [{'extra_trees': True, 'ag_args': {'name_suffix': 'XT'}}, {}, 'GBMLarge'],\n",
      "\t'CAT': {},\n",
      "\t'XGB': {},\n",
      "\t'FASTAI': {},\n",
      "\t'RF': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'XT': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'KNN': [{'weights': 'uniform', 'ag_args': {'name_suffix': 'Unif'}}, {'weights': 'distance', 'ag_args': {'name_suffix': 'Dist'}}],\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting 13 L1 models ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsUnif ... Training model for up to 4.83s of the 4.82s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.7752\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t1.42s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.04s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsDist ... Training model for up to 3.36s of the 3.36s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.766\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.03s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.02s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: LightGBMXT ... Training model for up to 3.32s of the 3.32s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/miniconda3/envs/ag/lib/python3.10/site-packages/dask/dataframe/__init__.py:31: FutureWarning: \n",
      "Dask dataframe query planning is disabled because dask-expr is not installed.\n",
      "\n",
      "You can install it with `pip install dask[dataframe]` or `conda install dask`.\n",
      "This will raise in a future version.\n",
      "\n",
      "  warnings.warn(msg, FutureWarning)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tRan out of time, early stopping on iteration 156. Best iteration is:\n",
      "\t[145]\tvalid_set's binary_error: 0.1284\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.8716\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t4.17s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.01s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: WeightedEnsemble_L2 ... Training model for up to 4.83s of the -1.03s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tEnsemble Weights: {'LightGBMXT': 0.842, 'KNeighborsUnif': 0.105, 'KNeighborsDist': 0.053}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.8744\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.02s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon training complete, total runtime = 6.13s ... Best model: \"WeightedEnsemble_L2\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/ag-20240617_191908\")\n"
     ]
    }
   ],
   "source": [
    "# Load and train a baseline classifier\n",
    "\n",
    "from autogluon.tabular import TabularDataset, TabularPredictor\n",
    "from oxonfair import FairPredictor \n",
    "from oxonfair.utils import group_metrics as gm\n",
    "train_data = TabularDataset('https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv')\n",
    "test_data = TabularDataset('https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv')\n",
    "predictor = TabularPredictor(label='class').fit(train_data=train_data,time_limit=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:14.715124Z",
     "iopub.status.busy": "2024-06-17T19:19:14.714539Z",
     "iopub.status.idle": "2024-06-17T19:19:15.155238Z",
     "shell.execute_reply": "2024-06-17T19:19:15.154020Z"
    }
   },
   "outputs": [],
   "source": [
    "# Modify predictor to enforce fairness over the train_data with respect to groups given by the column 'sex'\n",
    "fpredictor = FairPredictor(predictor,train_data,'sex')\n",
    "# Maximize accuracy while enforcing that the demographic parity (the difference in positive decision rates between men and women is at most 0.02)\n",
    "fpredictor.fit(gm.accuracy,gm.demographic_parity,0.02)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:15.159470Z",
     "iopub.status.busy": "2024-06-17T19:19:15.158888Z",
     "iopub.status.idle": "2024-06-17T19:19:15.313776Z",
     "shell.execute_reply": "2024-06-17T19:19:15.312793Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        <=50K\n",
       "1        <=50K\n",
       "2         >50K\n",
       "3        <=50K\n",
       "4        <=50K\n",
       "         ...  \n",
       "9764     <=50K\n",
       "9765     <=50K\n",
       "9766     <=50K\n",
       "9767     <=50K\n",
       "9768     <=50K\n",
       "Length: 9769, dtype: object"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate on test data\n",
    "fpredictor.predict(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:15.317613Z",
     "iopub.status.busy": "2024-06-17T19:19:15.317317Z",
     "iopub.status.idle": "2024-06-17T19:19:15.632565Z",
     "shell.execute_reply": "2024-06-17T19:19:15.632003Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>original</th>\n",
       "      <th>updated</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Accuracy</th>\n",
       "      <td>0.868973</td>\n",
       "      <td>0.842563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Balanced Accuracy</th>\n",
       "      <td>0.776058</td>\n",
       "      <td>0.723973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>F1 score</th>\n",
       "      <td>0.684574</td>\n",
       "      <td>0.600312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MCC</th>\n",
       "      <td>0.613887</td>\n",
       "      <td>0.524332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Precision</th>\n",
       "      <td>0.798276</td>\n",
       "      <td>0.754902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Recall</th>\n",
       "      <td>0.599223</td>\n",
       "      <td>0.498274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ROC AUC</th>\n",
       "      <td>0.919433</td>\n",
       "      <td>0.817906</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   original   updated\n",
       "Accuracy           0.868973  0.842563\n",
       "Balanced Accuracy  0.776058  0.723973\n",
       "F1 score           0.684574  0.600312\n",
       "MCC                0.613887  0.524332\n",
       "Precision          0.798276  0.754902\n",
       "Recall             0.599223  0.498274\n",
       "ROC AUC            0.919433  0.817906"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate a range of performance measures, and compare against original classifier on test data\n",
    "fpredictor.evaluate(test_data, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:15.637027Z",
     "iopub.status.busy": "2024-06-17T19:19:15.636462Z",
     "iopub.status.idle": "2024-06-17T19:19:16.033572Z",
     "shell.execute_reply": "2024-06-17T19:19:16.031149Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>original</th>\n",
       "      <th>updated</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Statistical Parity</th>\n",
       "      <td>0.155761</td>\n",
       "      <td>0.013698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predictive Parity</th>\n",
       "      <td>0.020717</td>\n",
       "      <td>0.354613</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Equal Opportunity</th>\n",
       "      <td>0.065251</td>\n",
       "      <td>0.371301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in False Negative Rate</th>\n",
       "      <td>0.065251</td>\n",
       "      <td>0.371301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Equalized Odds</th>\n",
       "      <td>0.056320</td>\n",
       "      <td>0.216631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Conditional Use Accuracy</th>\n",
       "      <td>0.060920</td>\n",
       "      <td>0.265098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Accuracy</th>\n",
       "      <td>0.098982</td>\n",
       "      <td>0.088835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Treatment Equality</th>\n",
       "      <td>0.051315</td>\n",
       "      <td>3.802436</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 original   updated\n",
       "Statistical Parity                               0.155761  0.013698\n",
       "Predictive Parity                                0.020717  0.354613\n",
       "Equal Opportunity                                0.065251  0.371301\n",
       "Average Group Difference in False Negative Rate  0.065251  0.371301\n",
       "Equalized Odds                                   0.056320  0.216631\n",
       "Conditional Use Accuracy                         0.060920  0.265098\n",
       "Average Group Difference in Accuracy             0.098982  0.088835\n",
       "Treatment Equality                               0.051315  3.802436"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate against a range of standard fairness definitions and compare against original classifier on test data\n",
    "fpredictor.evaluate_fairness(test_data, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:16.040489Z",
     "iopub.status.busy": "2024-06-17T19:19:16.039575Z",
     "iopub.status.idle": "2024-06-17T19:19:16.634654Z",
     "shell.execute_reply": "2024-06-17T19:19:16.633981Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Balanced Accuracy</th>\n",
       "      <th>F1 score</th>\n",
       "      <th>MCC</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>Positive Count</th>\n",
       "      <th>Negative Count</th>\n",
       "      <th>Positive Label Rate</th>\n",
       "      <th>Positive Prediction Rate</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Groups</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">original</th>\n",
       "      <th>Overall</th>\n",
       "      <td>0.868973</td>\n",
       "      <td>0.776058</td>\n",
       "      <td>0.684574</td>\n",
       "      <td>0.613887</td>\n",
       "      <td>0.798276</td>\n",
       "      <td>0.599223</td>\n",
       "      <td>0.919433</td>\n",
       "      <td>2318.0</td>\n",
       "      <td>7451.0</td>\n",
       "      <td>0.237281</td>\n",
       "      <td>0.178114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Female</th>\n",
       "      <td>0.934631</td>\n",
       "      <td>0.762758</td>\n",
       "      <td>0.641068</td>\n",
       "      <td>0.618349</td>\n",
       "      <td>0.780488</td>\n",
       "      <td>0.543909</td>\n",
       "      <td>0.930589</td>\n",
       "      <td>353.0</td>\n",
       "      <td>2936.0</td>\n",
       "      <td>0.107327</td>\n",
       "      <td>0.074795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Male</th>\n",
       "      <td>0.835648</td>\n",
       "      <td>0.771690</td>\n",
       "      <td>0.692108</td>\n",
       "      <td>0.593010</td>\n",
       "      <td>0.801205</td>\n",
       "      <td>0.609160</td>\n",
       "      <td>0.900545</td>\n",
       "      <td>1965.0</td>\n",
       "      <td>4515.0</td>\n",
       "      <td>0.303241</td>\n",
       "      <td>0.230556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Maximum difference</th>\n",
       "      <td>0.098982</td>\n",
       "      <td>0.008931</td>\n",
       "      <td>0.051039</td>\n",
       "      <td>0.025339</td>\n",
       "      <td>0.020717</td>\n",
       "      <td>0.065251</td>\n",
       "      <td>0.030044</td>\n",
       "      <td>1612.0</td>\n",
       "      <td>1579.0</td>\n",
       "      <td>0.195913</td>\n",
       "      <td>0.155761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">updated</th>\n",
       "      <th>Overall</th>\n",
       "      <td>0.842563</td>\n",
       "      <td>0.723973</td>\n",
       "      <td>0.600312</td>\n",
       "      <td>0.524332</td>\n",
       "      <td>0.754902</td>\n",
       "      <td>0.498274</td>\n",
       "      <td>0.817906</td>\n",
       "      <td>2318.0</td>\n",
       "      <td>7451.0</td>\n",
       "      <td>0.237281</td>\n",
       "      <td>0.156618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Female</th>\n",
       "      <td>0.901490</td>\n",
       "      <td>0.862578</td>\n",
       "      <td>0.639198</td>\n",
       "      <td>0.603680</td>\n",
       "      <td>0.526606</td>\n",
       "      <td>0.813031</td>\n",
       "      <td>0.930589</td>\n",
       "      <td>353.0</td>\n",
       "      <td>2936.0</td>\n",
       "      <td>0.107327</td>\n",
       "      <td>0.165704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Male</th>\n",
       "      <td>0.812654</td>\n",
       "      <td>0.707908</td>\n",
       "      <td>0.588475</td>\n",
       "      <td>0.532366</td>\n",
       "      <td>0.881218</td>\n",
       "      <td>0.441730</td>\n",
       "      <td>0.900545</td>\n",
       "      <td>1965.0</td>\n",
       "      <td>4515.0</td>\n",
       "      <td>0.303241</td>\n",
       "      <td>0.152006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Maximum difference</th>\n",
       "      <td>0.088835</td>\n",
       "      <td>0.154670</td>\n",
       "      <td>0.050724</td>\n",
       "      <td>0.071314</td>\n",
       "      <td>0.354613</td>\n",
       "      <td>0.371301</td>\n",
       "      <td>0.030044</td>\n",
       "      <td>1612.0</td>\n",
       "      <td>1579.0</td>\n",
       "      <td>0.195913</td>\n",
       "      <td>0.013698</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             Accuracy  Balanced Accuracy  F1 score       MCC  \\\n",
       "         Groups                                                                \n",
       "original Overall             0.868973           0.776058  0.684574  0.613887   \n",
       "          Female             0.934631           0.762758  0.641068  0.618349   \n",
       "          Male               0.835648           0.771690  0.692108  0.593010   \n",
       "         Maximum difference  0.098982           0.008931  0.051039  0.025339   \n",
       "updated  Overall             0.842563           0.723973  0.600312  0.524332   \n",
       "          Female             0.901490           0.862578  0.639198  0.603680   \n",
       "          Male               0.812654           0.707908  0.588475  0.532366   \n",
       "         Maximum difference  0.088835           0.154670  0.050724  0.071314   \n",
       "\n",
       "                             Precision    Recall   ROC AUC  Positive Count  \\\n",
       "         Groups                                                              \n",
       "original Overall              0.798276  0.599223  0.919433          2318.0   \n",
       "          Female              0.780488  0.543909  0.930589           353.0   \n",
       "          Male                0.801205  0.609160  0.900545          1965.0   \n",
       "         Maximum difference   0.020717  0.065251  0.030044          1612.0   \n",
       "updated  Overall              0.754902  0.498274  0.817906          2318.0   \n",
       "          Female              0.526606  0.813031  0.930589           353.0   \n",
       "          Male                0.881218  0.441730  0.900545          1965.0   \n",
       "         Maximum difference   0.354613  0.371301  0.030044          1612.0   \n",
       "\n",
       "                             Negative Count  Positive Label Rate  \\\n",
       "         Groups                                                    \n",
       "original Overall                     7451.0             0.237281   \n",
       "          Female                     2936.0             0.107327   \n",
       "          Male                       4515.0             0.303241   \n",
       "         Maximum difference          1579.0             0.195913   \n",
       "updated  Overall                     7451.0             0.237281   \n",
       "          Female                     2936.0             0.107327   \n",
       "          Male                       4515.0             0.303241   \n",
       "         Maximum difference          1579.0             0.195913   \n",
       "\n",
       "                             Positive Prediction Rate  \n",
       "         Groups                                        \n",
       "original Overall                             0.178114  \n",
       "          Female                             0.074795  \n",
       "          Male                               0.230556  \n",
       "         Maximum difference                  0.155761  \n",
       "updated  Overall                             0.156618  \n",
       "          Female                             0.165704  \n",
       "          Male                               0.152006  \n",
       "         Maximum difference                  0.013698  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate a range of performance measures per group, and compare against original classifier on test data\n",
    "fpredictor.evaluate_groups(test_data, verbose=True, return_original=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:16.639839Z",
     "iopub.status.busy": "2024-06-17T19:19:16.639439Z",
     "iopub.status.idle": "2024-06-17T19:20:39.414319Z",
     "shell.execute_reply": "2024-06-17T19:20:39.408483Z"
    }
   },
   "outputs": [],
   "source": [
    "#We can repeat the same analysis using the feature 'race' instead of 'sex'\n",
    "fpredictor = FairPredictor(predictor,train_data, 'race')\n",
    "# Maximize accuracy while enforcing that the demographic parity (the difference in positive decision rates between men and women is at most 0.02)\n",
    "fpredictor.fit(gm.accuracy, gm.demographic_parity, .02)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:20:39.435818Z",
     "iopub.status.busy": "2024-06-17T19:20:39.435388Z",
     "iopub.status.idle": "2024-06-17T19:20:39.820251Z",
     "shell.execute_reply": "2024-06-17T19:20:39.819926Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>original</th>\n",
       "      <th>updated</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Statistical Parity</th>\n",
       "      <td>0.089690</td>\n",
       "      <td>0.043099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predictive Parity</th>\n",
       "      <td>0.132235</td>\n",
       "      <td>0.241701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Equal Opportunity</th>\n",
       "      <td>0.153663</td>\n",
       "      <td>0.160266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in False Negative Rate</th>\n",
       "      <td>0.153663</td>\n",
       "      <td>0.160266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Equalized Odds</th>\n",
       "      <td>0.092069</td>\n",
       "      <td>0.104162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Conditional Use Accuracy</th>\n",
       "      <td>0.082708</td>\n",
       "      <td>0.152206</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Accuracy</th>\n",
       "      <td>0.043393</td>\n",
       "      <td>0.022143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Treatment Equality</th>\n",
       "      <td>0.222273</td>\n",
       "      <td>2.197229</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 original   updated\n",
       "Statistical Parity                               0.089690  0.043099\n",
       "Predictive Parity                                0.132235  0.241701\n",
       "Equal Opportunity                                0.153663  0.160266\n",
       "Average Group Difference in False Negative Rate  0.153663  0.160266\n",
       "Equalized Odds                                   0.092069  0.104162\n",
       "Conditional Use Accuracy                         0.082708  0.152206\n",
       "Average Group Difference in Accuracy             0.043393  0.022143\n",
       "Treatment Equality                               0.222273  2.197229"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Unlike the previous case, we find that demographic parity is still high on test data, although it is improved.\n",
    "fpredictor.evaluate_fairness(test_data, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:20:39.824487Z",
     "iopub.status.busy": "2024-06-17T19:20:39.824220Z",
     "iopub.status.idle": "2024-06-17T19:20:42.456476Z",
     "shell.execute_reply": "2024-06-17T19:20:42.456153Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAHFCAIAAACVbdGXAABXYklEQVR4nO3deVgTV9sH4JMESNhXgaAIuAJFrIIoIFoXcCsubb/ihmLBpdUKLq3ygoKK4lKtVgsqRXFBpVr3UhRFrQtWxV0oKqKgBlFQQJAtme+PqdM0BAhLgMHffb3Xe2XOOTPzzJjm4Zw5M8OhKIoAAACwDbe5AwAAAKgPJDAAAGAlJDAAAGAlJDAAAGAlJDAAAGAlJDAAAGAlJDAAAGAlJDAAAGAlJDAAAGAlJDAAQgiJiYnhVDF//vxG31FJSUloaOjZs2er7v3x48eNtZezZ89WPZwvvviisbavuEY/NACGSnMHANCCbN++3dramlk0MzNr9F2UlJQsWbKEEPLJJ58whSNGjEhOThYKhY27rxUrVgwYMIBZNDQ0bNztAzQvJDCAf9nZ2Tk6OtbQoKKigsPhqKg08n84bdq0adOmTf3WLSkp0dDQkFvVuXPnPn36NCAugBYNQ4gANaHH4nbt2jVv3ry2bdvy+fyHDx8SQrZt29a9e3eBQGBgYDBmzJi0tDRmFR8fHy0trYcPHw4fPlxLS8vc3HzevHllZWWEkMePH9OJasmSJfSwno+PD5E3znbq1KlBgwbp6OhoaGi4urqePn2aqQoNDeVwONevX//iiy/09fU7duxY14O6cOHCoEGDtLW1NTQ0XFxcfv/9d5mNSzeWic3S0vLTTz9NSEjo2bOnurq6tbX1tm3bpNtfvnzZ1dVVIBCYmZkFBgZWVFTUNTwABSGBAfxLLBZXSmHKAwMDs7KyNm/efOzYMWNj4/DwcF9f348++ujgwYMbNmy4ffu2s7PzgwcPmPYVFRUjR44cNGjQkSNHvvrqqx9//HHVqlWEEKFQmJCQQAjx9fVNTk5OTk5etGhR1TB2797t4eGho6OzY8eOX3/91cDAYMiQIdI5jBDy2WefderUaf/+/Zs3b67ucCQSSdXDOXfu3MCBAwsKCqKjo/fu3autre3p6RkXF6f4Wbp169a8efPmzJlz5MgRe3t7X1/fP//8k65KTU0dNGjQmzdvYmJiNm/efOPGjbCwMMW3DFA3FABQ1Pbt26v+11FRUXHmzBlCSL9+/ZiWr1+/VldXHz58OFOSlZXF5/PHjx9PL06ePJkQ8uuvvzINhg8f3rVrV/rzy5cvCSEhISFV956ZmUlRVHFxsYGBgaenJ1MrFou7d+/u5OREL4aEhBBCFi9eXMPh0GHLePDgAUVRffr0MTY2LioqoltWVlba2dm1a9dOIpEwG68uNoqiLCwsBALBkydP6MV3794ZGBhMnz6dXvTy8lJXV8/JyWE2Tl9TZFYHaETogQH8a+fOnVelMNe6Pv/8c6ZNcnLyu3fv6KE/mrm5+cCBA6V7SBwOx9PTk1m0t7d/8uSJgjFcunQpPz9/8uTJTM9JIpEMHTr06tWrxcXFTDPpkKqzatUq6cMxNzcvLi7+66+/vvjiCy0tLboNj8fz9vZ++vRpenq6ghF+/PHH7du3pz8LBIIuXbowR3fmzJlBgwaZmJgwG/fy8lJwswB1hUkcAP+ysbGRO4lDen5gXl6eTAkhxMzMLDExkVnU0NAQCATMIp/PLy0tVTCGFy9eEELkTnnPz8/X1NSsGlJ1OnToIHM4L1++pCiqavDk/XEpQmY2I5/Pf/fuHf05Ly/P1NRUulZmEaARIYEB1E56XgP98y0SiaQbPH/+3MjIqFH2RW9n48aNVScQMj0bmZAUp6+vz+VyqwbP7JfOu2VlZXw+n6599eqV4ts3NDTMycmRLpFZBGhEGEIEqBtnZ2d1dfXdu3czJU+fPk1KSho0aJAiq9OJgemyVOXq6qqnp5eamupYhZqaWgOD19TU7N2798GDB5kAJBLJ7t2727Vr16VLF0KIpaUlIeT27dvMKseOHVN8+wMGDDh9+jTdiSSEiMXiOk0PAagT9MAA6kZPT2/RokX/+9//Jk2aNG7cuLy8vCVLlggEAnr6Q620tbUtLCyOHDkyaNAgAwMDIyMjOmcwtLS0Nm7cOHny5Pz8/C+++MLY2Pjly5e3bt16+fJlZGRkw+MPDw93d3cfMGDA/Pnz1dTUIiIi7t69u3fvXrpLN3z4cAMDA19f36VLl6qoqMTExGRnZyu+8eDg4KNHjw4cOHDx4sUaGho///yz9HU7gMaFHhhAnQUGBv7yyy+3bt0aPXr0rFmzPvroo0uXLnXu3FnB1aOjozU0NEaOHNmrV6/Q0NCqDSZOnHjmzJm3b99Onz598ODB/v7+169fV7CHV6v+/fsnJSVpamr6+PiMHTu2oKDg6NGjzFQLHR2dhIQEbW3tiRMnzpgxw87OLigoSPGN29nZnTp1SkdHZ/LkydOmTbO3t5d7nwBAo+BQFNXcMQAAANQZemAAAMBKSGAAAMBKSGAAAMBKSGAAAMBKSGAAAMBKSGAAAMBKre1GZolE8vz5c21t7fo9aAcAAFoIiqKKiorMzMy4XPl9rdaWwJ4/f25ubt7cUQAAQOPIzs5u166d3KrWlsC0tbUJIdnZ2To6Os0dCwAA1F9hYaG5uTn9qy5Xa0tg9Mihjo4OEhgAQCtQw/UgTOIAAABWQgIDAABWQgIDAABWQgIDAABWQgIDAABWQgIDAABWQgIDAABWQgIDAABWQgIDAABWam1P4gAAdhFLqCuZ+blFpcbaAicrAx639sdwK28VRZrV2qa6BvUIWxlaSBiNAgkMoKUQS6jLGXnJj14RwnHuaNing2Gtv311WkXuRuqx00Y8tIS7oiXHUkUFpXQzoa4gxNN2qJ2whk0pbxVFmtXaproG9QhbGVpIGI2FQ1FUc8fQmAoLC3V1dQsKCvAsRKiH6n5nmZ9yI00+4ZBXb8uks8LljLyLGS+fvyltq6/u0tGIXquGxFBeKdlxKfPq49eaarzPerZz6WTE43IS7ooWHrzzpqSCCUZPQ3XlZ91q+O2r0yojuwuP3hLJbIQQUted1uOsVhcnIeTr3delf4DoExQ5sWd1O0q4K1LSKoo0q7VNdQ2m9bPa+mdmncJWhnqcveZV6+85Ehi0UDJ//osl1K7kx0/ySywMNLydLdVUuNLNcgre5ReXG2jxTXX+07e49ODVbzeelpSLe1kaTHb5Zy2Z1NLL0iDlyevcotLHr0q2X8x8807O76z0TzmDzgpx155K/zTTa3k5tpMpZxJDeHyqzG+ZphpvYp/2W/7MlHseplfz2zetn5Xiq1TFIaSGBtXttB6/dAl3RTN2X5dbpaehKnPq6B2Z6gouLBgod2iu76qkqv8QDV9FkWa1tiGEyG1ACOFyiKTK6a4hbGWox9lrqIp3JPln4jyTqKrXbwNIYNCcmFQhoYi+hpqR9j8JprpsxKyy+6/H5x+8elsmpgs11HjvKsTMV5XLIVPdrAKH28r0EmhM32Lur7dKysVMOYdDprlZ9WivL9Mb4HBIk/1H4G5rnJiaW6dV5P72kRozUHWrNHCn9filE0so15WncwrL6hrA3ql9nDsayhQmZ+SNi7qsjFUUaVZrG0JIDQ3qFLYy1OPsNdStOHJoGvksith/Wb8N1Pp7jmtgoJAnhU+KK4olEuru88K8t2WF7yp01FUNtfhOFkILHYvqLm8s/O2OdIeGJpONlsen0dmIyBtroknnIUKIhCJb/sx89Kr4VGpu1R9qUUGp3D/5KYps+TOTENleS1P+CVfX7EVItamohqgbmL2q2wJFiKig9EpmvuK/dFcy8+uRvQghuUVy+jFyCxtlFUWa1WPviqj3io27I6WEce/QP/9f3wRWKyQw+A+5V26eFD759NCn8le4ScRZ35cUG9BLm848ZIbdqhs4kpuNCCE92utXt4pc9UgG0HB1+qWr98+isbZAwcJGWUWRZvXYuyLqvWLj7qjRwnibS5J/JpJKQgjJOE0IIQ9PkRNBhBDCVSHOs4hWm8bZESEECax1q2EWmdxEJdMBYrJRe2FxDXspFb+TXnxTUjFj93UN1brdYrj1z0xj7ed1WgWaRZ1+6erxs0gPVDpZGVStcrIyEOoKcgpKqcZeRZFmirSR24AQwuUQipLtNNcQtjLU4+zVR1EOuRxJxGWEw/3nyqlETC5HEEpCeHxi93njJjDcyNxKiCVUckbekZvPkjPyxBKKEJJwV9R3VdK4qMv++26Oi7rcd1VSwl0R3TjhrsghLHFC9F+bzmRsOvNwwi9/OYQlhsenzth9XWb4js5G0Rce1TWekgpJndpThLwoqs9YU6vH5RC5V5xquAxV3SoN3CmHEGEdf+mcrAxMdfjV1eppqHL+eyD05xBPW7mX2XhcDn11s9FXUaRZrW2qa8AhZKqbVZ3CVoZ6nL36ENqT6eeIURdCCKHE//6/URcy/RwR2jfOXt7DJI6Wq7xSIj3TgcflyPSZCCF0ByvzZfHOy0/yi8vpFenZcdXNIiPVD+5Vhyt4pmm1sbra4sxvJaVt63hwoCh6QiCRuuil4CxEUuN1MkVmIVbdaePOQtw8sSf57wxP3AembE0UxrvXZHUHQr3/K5bDJd8/Iur6dd0MZiGy1fLfU3+5kMn843AIUVPhllX+263RUOOpqXCrzneoAYcQEx0+RdW5r4MEVvPPveJqmIU4vZ+VzMx7fQ3V8DreB1bDKgreB1brTutx1DXESfAkjibXFGH8/TvZN/4/JWP3Euvhdd0MEliLIPONIYRUd4srberOqy1qhkITJDAOIcba/DplVndbY7mzEBsR/TtL6ngfmL6G6pdV7gNjfrLl3AfG5639v+5D7YQf4JM4oHU6NJ3c2kfaOxP3ZeRkMMm+TLqPI2M213UzSGDNT+YvWT0N1fJKifRMPOYWV3rx+M1ns/bdbPo4a1CPBFbXLsv0flY1zELkq3DLxZImuA/MVIfv1ctcLKGa/kkcdTlbAC3b5c2kspS4fEu4PCIRk0sbiYqA9JlR180ggSlFzbP7pKsSU3Nknt1Snc0Te9J/g3+89ERRqbj2FZoQT/BMo44JbHr1V2iquyuZyBtr0lTjTevXYdbAzk3wJI5W8GxTgNYECazxxd8WBR+5Kz1jgrk2INMhMNXhl1ZKFLxMJdQVXFgw8Epmfj1u5ldQva+BjXMRHH8dUF2t9H1g5L8XUUKP3mPuY9Xi8/7PoZ3HR0JFnsSBsSYAQAJrZOHxqVU7Fpz3s/sU7GxVZ+/UPrlFpf6NN37IkTeLjFQ/C7GGeQS7rl374dTtotJKpkqgwvm8p/kUF+vqnsRBWsxVawBgIySwxhR/+/k3e27IrTLV4RPCySls0ONYNoz92Fhb0Fg9sKluVsdvy846Y3qK1c0Kq/mSPvpGANBkkMAajVhC9VqemF9ch2nrdbV3ah8nK4PqHmhdJ9P7WQUOt63rkzgauFMAgEaEh/k2miuZ+UrNXvQzDui75eUORapwOZVSz1jVVOOpyrsPzEBTNWyU3XB7M0IIj8up7rmrPC7HtbORa2ejRjwEAICmhASmqFOpOUrdPvM0l6F2wsiJPaUng2gLeOGjuw2zN6vuSRxV53YrNVQAgJYAQ4gKEUsox7DE19XPJ9TXUOWr8F4UynmOZ62kn0ogvUdMfwCADxmGEBvHlcz8GrIXIWSys6W1UPvr3ddlJv5Vl8/8B3WqerestBpG/wAAgCCBKajW1xpZtdGsOvRn+v6REC3hIZ4AAK0MEphCan2tEd1gqJ3Q3da06tCf3EIAAGgIJDCFOFjo1zAeKP2SJLlDfxgPBABodHihpUJSnryuYXbG2F7t0akCAGhiSGAKqfkamKWRRpNFAgAANKUnsIiICCsrK4FA4ODgcP78ebltYmNju3fvrqGhIRQKp0yZkpeXx1StX7++a9eu6urq5ubmc+bMKS1t6CMq6qfma2C1XiEDAIBGp9wEFhcXFxAQEBQUdOPGDTc3t2HDhmVlZcm0uXDhwqRJk3x9fe/du7d///6rV6/6+fnRVbGxsQsXLgwJCUlLS4uOjo6LiwsMDFRqwNVxsjIQ6gqqjhJy/nsBDAAAmoxyE9i6det8fX39/PxsbGzWr19vbm4eGRkp0+by5cuWlpazZ8+2srLq27fv9OnTr127RlclJye7urqOHz/e0tLSw8Nj3LhxTFUTo5/wRN4/051Gf2aeoAEAAE1JiQmsvLw8JSXFw8ODKfHw8Lh06ZJMMxcXl6dPn8bHx1MU9eLFiwMHDowYMYKu6tu3b0pKypUrVwghjx49io+PZ6qklZWVFUpR0uHQt3mZ6v47WmiqK4ic2BN3dAEANAslTqN/9eqVWCw2MTFhSkxMTHJyZJ8o6OLiEhsb6+XlVVpaWllZOXLkyI0b/3n579ixY1++fNm3b1+KoiorK7/++uuFCxdW3VF4ePiSJUuUdyCM6m7zAgCApqf0SRwczr8/8RRFSS/SUlNTZ8+evXjx4pSUlISEhMzMzBkzZtBVZ8+eXb58eURExPXr1w8ePHj8+PFly5ZV3UVgYGDBe9nZ2co7FvL+jq5RH7d17oj3jwAANCcl9sCMjIx4PJ50lys3N1e6Q0YLDw93dXX97rvvCCH29vaamppubm5hYWFCoXDRokXe3t70nI5u3boVFxdPmzYtKCiIy/1P3uXz+Xw+X3kHAgAALZASe2BqamoODg6JiYlMSWJioouLi0yzkpIS6YTE4/EIIfQz8qtWURTVyh6fDwAA9aPcR0nNnTvX29vb0dHR2dl569atWVlZ9PBgYGDgs2fPdu7cSQjx9PScOnVqZGTkkCFDRCJRQECAk5OTmZkZXbVu3boePXr07t374cOHixYtGjlyJJ3hAADgA6fcBObl5ZWXl7d06VKRSGRnZxcfH29hYUEIEYlEzA1hPj4+RUVFmzZtmjdvnp6e3sCBA1etWkVXBQcHczic4ODgZ8+etWnTxtPTc/ny5UoNGAAA2AIvtAQAgJao1t9zPAsRAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYCQkMAABYSekJLCIiwsrKSiAQODg4nD9/Xm6b2NjY7t27a2hoCIXCKVOm5OXlMVVv3ryZOXOmUCgUCAQ2Njbx8fHKDhgAAFhBuQksLi4uICAgKCjoxo0bbm5uw4YNy8rKkmlz4cKFSZMm+fr63rt3b//+/VevXvXz86OrysvL3d3dHz9+fODAgfT09KioqLZt2yo1YAAAYAsORVHK23rv3r179uwZGRlJL9rY2IwePTo8PFy6zQ8//BAZGZmRkUEvbty4cfXq1dnZ2YSQzZs3r1mz5u+//1ZVVVVwj4WFhbq6ugUFBTo6Oo13HAAA0NRq/T1XYg+svLw8JSXFw8ODKfHw8Lh06ZJMMxcXl6dPn8bHx1MU9eLFiwMHDowYMYKuOnr0qLOz88yZM01MTOzs7FasWCEWi6vuqKysrFCK8o4IAABaDiUmsFevXonFYhMTE6bExMQkJydHppmLi0tsbKyXl5eampqpqament7GjRvpqkePHh04cEAsFsfHxwcHB69du3b58uVVdxQeHq77nrm5ufKOCAAAWg6lT+LgcDjMZ4qipBdpqamps2fPXrx4cUpKSkJCQmZm5owZM+gqiURibGy8detWBweHsWPHBgUFMaOR0gIDAwveo8ceAQCg1VNR3qaNjIx4PJ50lys3N1e6Q0YLDw93dXX97rvvCCH29vaamppubm5hYWFCoVAoFKqqqvJ4PLqljY1NTk5OeXm5mpqa9Bb4fD6fz1fegQAAQAukxB6Ympqag4NDYmIiU5KYmOji4iLTrKSkhMv9Nww6XdFTS1xdXR8+fCiRSOiq+/fvC4VCmewFAAAfJuUOIc6dO/eXX37Ztm1bWlranDlzsrKy6OHBwMDASZMm0W08PT0PHjwYGRn56NGjixcvzp4928nJyczMjBDy9ddf5+Xl+fv7379///fff1+xYsXMmTOVGjAAALCFEocQCSFeXl55eXlLly4ViUR2dnbx8fEWFhaEEJFIxNwQ5uPjU1RUtGnTpnnz5unp6Q0cOHDVqlV0lbm5+cmTJ+fMmWNvb9+2bVt/f/8FCxYoNWAAAGAL5d4H1vRwHxgAQOvQnPeBAQAAKA8SGAAAsBISGAAAsBISGAAAsBISGAAAsBISGAAAsBISGAAAsBISGAAAsBISGAAAsBISGAAAsBISGAAAsBISGAAAsJJyn0bfmogl1JXM/NyiUmNtgZOVAY8r+2ppAABoSkhgCkm4K1pyLFVUUEovCnUFIZ62Q+2EzRsVAMCHDEOItUu4K/p693UmexFCcgpKv959PeGuqBmjAgD4wCGB1UIsoZYcS5V5Zxq9uORYqljSqt6mBgDAIkhgtbiSmS/d92JQhIgKSq9k5jd9SAAAQJDAapVbJCd7KVgLAADKgwRWC2NtQb1rAQBAeZDAauFkZSDUFVSdMs8hRKgrcLIyaIaYAAAACaxWPC4nxNOWECKdw+jPIZ62uBsMAKC5IIHVbqidMHJiT1Pdf0cLTXUFkRN74j4wAIBmhBuZFTLUTuhua4oncQAAtBxIYIricTnOHQ2bOwoAAPgHhhABAICVkMAAAICVkMAAAICVkMAAAICVkMAAAICVkMAAAICVkMAAAICVkMAAAICVkMAAAICV8CSOWoglFJ4gBQDQAiGB1SThrmjJsVTmjcxCXUGIpy2e4QsA0BJgCLFaCXdFX+++zmQvQkhOQenXu68n3BU1Y1QAAEBDApNPLKGWHEul/ltILy45liqWUHLWAQCAJoQEJt+VzHzpvheDIkRUUHolM7/pQwIAAGlIYPLlFsnJXgrWAgBAE0ACk89YW1DvWgAAaAJKT2ARERFWVlYCgcDBweH8+fNy28TGxnbv3l1DQ0MoFE6ZMiUvL0+mwb59+zgczujRo5UdLcPJykCoK6g6ZZ5DiFBX4GRl0GSRAACAXMpNYHFxcQEBAUFBQTdu3HBzcxs2bFhWVpZMmwsXLkyaNMnX1/fevXv79++/evWqn5+fdIMnT57Mnz/fzc1NqaHK4HE5IZ62hBDpHEZ/DvG0xd1gAADNTrkJbN26db6+vn5+fjY2NuvXrzc3N4+MjJRpc/nyZUtLy9mzZ1tZWfXt23f69OnXrl1jasVi8YQJE5YsWdKhQwelhlrVUDth5MSeprr/jhaa6goiJ/bEfWAAAC2BEm9kLi8vT0lJWbhwIVPi4eFx6dIlmWYuLi5BQUHx8fHDhg3Lzc09cODAiBEjmNqlS5e2adPG19e3uuFHpRpqJ3S3NcWTOAAAWiAlJrBXr16JxWITExOmxMTEJCcnR6aZi4tLbGysl5dXaWlpZWXlyJEjN27cSFddvHgxOjr65s2bNe+orKysrKyM/lxYWNhoB0AIIYTH5Th3NGzcbQIAQMMpfRIHh/Nvl4WiKOlFWmpq6uzZsxcvXpySkpKQkJCZmTljxgxCSFFR0cSJE6OiooyMjGreRXh4uO575ubmjX4IAADQAnEoSlkPlSgvL9fQ0Ni/f/+YMWPoEn9//5s3b547d066mbe3d2lp6f79++nFCxcuuLm5PX/+/MWLFz169ODxeHS5RCIhhHC53PT09I4dO0pvQaYHZm5uXlBQoKOjo6TjAgCAJlBYWKirq1vD77kShxDV1NQcHBwSExOZBJaYmDhq1CiZZiUlJSoq/4ZBZyyKoqytre/cucOUBwcHFxUVbdiwoWofi8/n8/l8pRwDAAC0VMp9Gv3cuXO9vb0dHR2dnZ23bt2alZVFDw8GBgY+e/Zs586dhBBPT8+pU6dGRkYOGTJEJBIFBAQ4OTmZmZkRQuzs7JhN6enpyZQAAMCHTLkJzMvLKy8vb+nSpSKRyM7OLj4+3sLCghAiEomYG8J8fHyKioo2bdo0b948PT29gQMHrlq1SqlRAQBAK6DEa2DNotYxUwAAYIVaf8/xLEQAAGAlJDAAAGAlJDAAAGAlhRKYpaXl0qVLqz6HFwAAoLkolMDmzZt35MiRDh06uLu779u3j7lrGAAAoLkolMC+/fbblJSUlJQUW1vb2bNnC4XCWbNmXb9+XdnBAQAAVKfO0+grKioiIiIWLFhQUVFhZ2fn7+8/ZcqUqk84bC6YRg8A0Do05qOkKioqDh06tH379sTExD59+vj6+j5//jwoKOjUqVN79uxppIABAAAUolACu379+vbt2/fu3cvj8by9vX/88Udra2u6ysPDo1+/fsqMEAAAQA6FElivXr3c3d0jIyNHjx6tqqoqXWVrazt27FjlxAYAAFAthRLYo0eP6GcYVqWpqbl9+/ZGDQkAAKB2Cs1CzM3N/euvv6RL/vrrr2vXriknJAAAgNoplMBmzpyZnZ0tXfLs2bOZM2cqJyQAAIDaKZTAUlNTe/bsKV3So0eP1NRU5YQEAABQO4USGJ/Pf/HihXSJSCSSfo0yAABAE1Mogbm7uwcGBhYUFNCLb968+d///ufu7q7MwAAAAGqiUC9q7dq1/fr1s7Cw6NGjByHk5s2bJiYmu3btUnJsAAAA1VIogbVt2/b27duxsbG3bt1SV1efMmXKuHHjZG4IAwAAaEqKXsfS1NScNm2aUkMBAABQXB0mYqSmpmZlZZWXlzMlI0eOVEJIAAAAtVP0SRxjxoy5c+cOh/PP0+vpx8+LxWLlRgcAAFANhWYh+vv7W1lZvXjxQkND4969e3/++aejo+PZs2eVHBsAAEC1FOqBJScnJyUltWnThsvlcrncvn37hoeHz549+8aNG8qODwAAQC6FemBisVhLS4sQYmRk9Pz5c0KIhYVFenq6ckMDAAConkI9MDs7u9u3b3fo0KF3796rV69WU1PbunVrhw4dlB0cAABAdRRKYMHBwcXFxYSQsLCwTz/91M3NzdDQMC4uTsmxAQAAVOufWYV1kp+fr6+vT09EbGkKCwt1dXULCgp0dHSaOxYAAKi/Wn/Pa78GVllZqaKicvfuXabEwMCgZWYvAAD4cNSewFRUVCwsLHDLFwAAtCgKzUIMDg4ODAzMz89XdjQAAAAKUmgSx08//fTw4UMzMzMLCwtNTU2m/Pr160oLDAAAoCYKJbDRo0crOQwAAIC6qc8sxJYMsxABAFqHRpiFCAAA0AIpNITI5XLlzpvH1EQAAGguCiWwQ4cOMZ8rKipu3LixY8eOJUuWKC0qAACAWtTzGtiePXvi4uKOHDnS6AE1EK6BAQC0Dsq6Bta7d+9Tp041IDAAAIAGqU8Ce/fu3caNG9u1a9fo0QAAAChIoWtg0o/upSiqqKhIQ0Nj9+7dygwMAACgJgolsB9//JFJYFwut02bNr1799bX11dmYAAAADVRaAjRx8dn8nve3t5Dhw5VPHtFRERYWVkJBAIHB4fz58/LbRMbG9u9e3cNDQ2hUDhlypS8vDy6PCoqys3NTV9fX19ff/DgwVeuXFFwpwAA0OoplMC2b9++f/9+6ZL9+/fv2LGj1hXj4uICAgKCgoJu3Ljh5uY2bNiwrKwsmTYXLlyYNGmSr6/vvXv39u/ff/XqVT8/P7rq7Nmz48aNO3PmTHJycvv27T08PJ49e6bYcQEAQGtHKaBLly5JSUnSJWfPnu3SpUutKzo5Oc2YMYNZtLa2XrhwoUybNWvWdOjQgVn86aef2rVrV3VTlZWV2traO3bsqHmPBQUFhJCCgoJaYwMAgJas1t9zhXpgT548sbKyki6xsLCo2peSUV5enpKS4uHhwZR4eHhcunRJppmLi8vTp0/j4+Mpinrx4sWBAwdGjBhRdWslJSUVFRUGBgZVq8rKygqlKHJEAADAdgolMGNj49u3b0uX3Lp1y9DQsOa1Xr16JRaLTUxMmBITE5OcnByZZi4uLrGxsV5eXmpqaqampnp6ehs3bqy6tYULF7Zt23bw4MFVq8LDw3XfMzc3V+SIAACA7RRKYGPHjp09e/aZM2fEYrFYLE5KSvL39x87dqwi60o/RJGiqKrPVExNTZ09e/bixYtTUlISEhIyMzNnzJgh02b16tV79+49ePCgQCCouovAwMCC97KzsxWJCgAA2E6hafRhYWFPnjwZNGiQiooKIUQikUyaNGnFihU1r2VkZMTj8aS7XLm5udIdMlp4eLirq+t3331HCLG3t9fU1HRzcwsLCxMKhXSDH374YcWKFadOnbK3t5e7Iz6fz+fzFTkQAABoNRTqgampqcXFxaWnp8fGxh48eDAjI2Pbtm1qamq1ruXg4JCYmMiUJCYmuri4yDQrKSnhcv8Ng8fjEUKo909oXLNmzbJlyxISEhwdHRUJFQAAPhAK9cBonTt37ty5c522PnfuXG9vb0dHR2dn561bt2ZlZdHDg4GBgc+ePdu5cychxNPTc+rUqZGRkUOGDBGJRAEBAU5OTmZmZoSQ1atXL1q0aM+ePZaWlnRPTktLS0tLq04xAABA66TIXMbPP/88PDxcumT16tVffPGFIuv+/PPPFhYWampqPXv2PHfuHF04efLk/v37M21++uknW1tbdXV1oVA4YcKEp0+f0uUWFhYy0YaEhNS8O0yjBwBoHWr9PVfodSpt2rRJSkrq1q0bU3Lnzp3Bgwe/ePGiUZNpI8DrVAAAWofGeZ3K27dvZa54qaqq4o4rAABoRgolMDs7u7i4OOmSffv22draKickAACA2ik0iWPRokWff/55RkbGwIEDCSGnT5/es2fPgQMHlBwbAABAtRRKYCNHjjx8+PCKFSsOHDigrq7evXv3pKQkXGQCAIBmpNAkDmlv3ryJjY2Njo6+deuWWCxWUlj1hkkcAACtQ+NM4qAlJSVNnDjRzMxs06ZNw4cPv3btWiMFCQAAUGe1DyE+ffo0JiZm27ZtxcXFX375ZUVFxW+//YYZHAAA0Lxq6YENHz7c1tY2NTV148aNz58/l/uceAAAgKZXSw/s5MmTs2fP/vrrr+v6ECkAAAClqqUHdv78+aKiIkdHx969e2/atOnly5dNExYAAEDNaklgzs7OUVFRIpFo+vTp+/bta9u2rUQiSUxMLCoqapr4AAAA5KrbNPr09PTo6Ohdu3a9efPG3d396NGjyousfjCNHgCgdWjMafSEkK5du65evfrp06d79+5tjPAAAADqqc43Mrdw6IEBALQOjdwDAwAAaCGQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWUnsAiIiKsrKwEAoGDg8P58+fltomNje3evbuGhoZQKJwyZUpeXh5T9dtvv9na2vL5fFtb20OHDik7WgAAYAvlJrC4uLiAgICgoKAbN264ubkNGzYsKytLps2FCxcmTZrk6+t77969/fv3X7161c/Pj65KTk728vLy9va+deuWt7f3l19++ddffyk1YAAAYAsORVHK23rv3r179uwZGRlJL9rY2IwePTo8PFy6zQ8//BAZGZmRkUEvbty4cfXq1dnZ2YQQLy+vwsLCP/74g64aOnSovr7+3r17a9hjYWGhrq5uQUGBjo5O4x8PAAA0lVp/z5XYAysvL09JSfHw8GBKPDw8Ll26JNPMxcXl6dOn8fHxFEW9ePHiwIEDI0aMoKuSk5OlVx8yZEjV1QkhZWVlhVKUcCgAANDiKDGBvXr1SiwWm5iYMCUmJiY5OTkyzVxcXGJjY728vNTU1ExNTfX09DZu3EhX5eTk1Lo6ISQ8PFz3PXNzcyUcCgAAtDhKn8TB4XCYzxRFSS/SUlNTZ8+evXjx4pSUlISEhMzMzBkzZii+OiEkMDCw4D167BEAAFo9FeVt2sjIiMfjSfeZcnNzpXtUtPDwcFdX1++++44QYm9vr6mp6ebmFhYWJhQKTU1Na12dEMLn8/l8vnIOAgAAWigl9sDU1NQcHBwSExOZksTERBcXF5lmJSUlXO6/YfB4PEIIPbXE2dlZevWTJ09WXR0AAD5MSuyBEULmzp3r7e3t6Ojo7Oy8devWrKwsengwMDDw2bNnO3fuJIR4enpOnTo1MjJyyJAhIpEoICDAycnJzMyMEOLv79+vX79Vq1aNGjXqyJEjp06dunDhglIDBgAAtlBuAvPy8srLy1u6dKlIJLKzs4uPj7ewsCCEiEQi5oYwHx+foqKiTZs2zZs3T09Pb+DAgatWraKrXFxc9u3bFxwcvGjRoo4dO8bFxfXu3VupAQMAAFso9z6wpof7wAAAWofmvA8MAABAeZDAAACAlZDAAACAlZDAAACAlZDAAACAlZDAAACAlZR7HxjbiSXUlcz83KJSY22Bk5UBjyvnSYwAANAskMCqlXBXtORYqqiglF4U6gpCPG2H2gmbNyoAAKBhCFG+hLuir3dfZ7IXISSnoPTr3dcT7oqaMSoAAGAggckhllBLjqXKPKGEXlxyLFUsaVXPLgEAYCkkMDmuZOZL970YFCGigtIrmflNHxIAAMhAApMjt0hO9lKwFgAAmgYSmBzG2oJ61wIAQNNAApPDycpAqCuoOmWeQ4hQV+BkZdAMMQEAwH8hgcnB43JCPG0JIdI5jP4c4mmLu8EAAFoCJDD5htoJIyf2NNX9d7TQVFcQObEn7gMDAGghcCNztYbaCd1tTfEkDgCAlgkJrCY8Lse5o2FzRwEAAHJgCBEAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFgJCQwAAFhJpbkDAIAPl1gsrqioaO4ooPmpqalxuXXuUCGBAUAzoCgqJyfnzZs3zR0ItAhcLtfKykpNTa1OayGBAUAzoLOXsbGxhoYGh8Np7nCgOUkkkufPn4tEovbt29fpy6D0BBYREbFmzRqRSPTRRx+tX7/ezc1NpoGPj8+OHTukS2xtbe/du0d/Xr9+fWRkZFZWlpGR0RdffBEeHi4QCJQdMwAolVgsprOXoaFhc8cCLUKbNm2eP39eWVmpqqqq+FrKncQRFxcXEBAQFBR048YNNze3YcOGZWVlybTZsGGD6L3s7GwDA4P/+7//o6tiY2MXLlwYEhKSlpYWHR0dFxcXGBio1IABoAnQ1700NDSaOxBoKejBQ7FYXKe1lJvA1q1b5+vr6+fnZ2Njs379enNz88jISJk2urq6pu9du3bt9evXU6ZMoauSk5NdXV3Hjx9vaWnp4eExbty4a9euKTVgAGgyGDkERv2+DEpMYOXl5SkpKR4eHkyJh4fHpUuXalglOjp68ODBFhYW9GLfvn1TUlKuXLlCCHn06FF8fPyIESOqrlVWVlYopVEPAgAAWiglJrBXr16JxWITExOmxMTEJCcnp7r2IpHojz/+8PPzY0rGjh27bNmyvn37qqqqduzYccCAAQsXLqy6Ynh4uO575ubmjXsUAAANFxoa+vHHHzd3FK2N0m9klu4YUhRVQz8xJiZGT09v9OjRTMnZs2eXL18eERFx/fr1gwcPHj9+fNmyZVVXDAwMLHgvOzu7UcMHgFbOx8eH818PHz5s+GY5HM7hw4eZxfnz558+fbrhmwVpSpyFaGRkxOPxpLtcubm50h0yaRRFbdu2zdvbW/o+gEWLFnl7e9N9sm7duhUXF0+bNi0oKEjmfjc+n8/n85VzEADQgogl1JXM/NyiUmNtgZOVAY/bOFfRhg4dun37dmaxTZs2zOfy8vK63pwkl5aWlpaWVp1WqaioqNOUvA+QEntgampqDg4OiYmJTEliYqKLi4vcxufOnXv48KGvr690YUlJiXSu4vF4FEVRFKWkgAGgJUu4K+q7Kmlc1GX/fTfHRV3uuyop4a6oUbbM5/NNpQwaNGjWrFlz5841MjJyd3cnhJw7d87JyYnP5wuFwoULF1ZWVtIrfvLJJ7Nnz/7+++8NDAxMTU1DQ0PpcktLS0LImDFjOBwO/VlmCHH79u02NjYCgcDa2joiIoIufPz4MYfD+fXXXz/55BOBQLB79+5GObpWTLlDiHPnzv3ll1+2bduWlpY2Z86crKysGTNmEEICAwMnTZok3TI6Orp37952dnbShZ6enpGRkfv27cvMzExMTFy0aNHIkSN5PJ5SYwaAFijhrujr3ddFBaVMSU5B6de7rzdWDpOxY8cOFRWVixcvbtmy5dmzZ8OHD+/Vq9etW7ciIyOjo6PDwsKkW2pqav7111+rV69eunQp/Sf71atXCSHbt28XiUT0Z2lRUVFBQUHLly9PS0tbsWLFokWLpO+FXbBgwezZs9PS0oYMGaKMQ2tNlHsjs5eXV15e3tKlS0UikZ2dXXx8PD3DUCQSSd8QVlBQ8Ntvv23YsEFm9eDgYA6HExwc/OzZszZt2nh6ei5fvlypAQNACySWUEuOpcqMvVCEcAhZcizV3da0gWOJx48fZ8b3hg0bRgjp1KnT6tWr6ZKgoCBzc/NNmzZxOBxra+vnz58vWLBg8eLF9PiQvb19SEgIIaRz586bNm06ffq0u7s7PQipp6dnampadXfLli1bu3btZ599RgixsrJKTU3dsmXL5MmT6dqAgAC6Cmql9CdxfPPNN998841MYUxMjPSirq5uSUlJ1XVVVFRCQkLoLwcAfLCuZOZL970YFCGigtIrmfnOHRv0RI8BAwYwt6hqamqOGzfO0dGRqU1LS3N2dmYmoLm6ur59+/bp06ft27cnhNjb2zMthUJhbm5uzft6+fJldna2r6/v1KlT6ZLKykpdXV2mgfSuoWZ4FiIAtHS5RXKyl4K1itDU1OzUqZNMCfNZZvo0fRmeKZGeZ8HhcCQSSc37ohtERUX17t2bKZS+MiK9a6gZEhgAtHTG2jU9AbXm2oaztbX97bffmDR26dIlbW3ttm3b1ryWqqqq3AcjmZiYtG3b9tGjRxMmTFBKuB8SvNASAFo6JysDoa6g6mUuDiFCXYGTlYFS9/7NN99kZ2d/++23f//995EjR0JCQubOnVvry6ssLS1Pnz6dk5Pz+vVrmarQ0NDw8PANGzbcv3//zp0727dvX7dundLCb82QwACgpeNxOSGetoQQ6RxGfw7xtG2su8Gq07Zt2/j4+CtXrnTv3n3GjBm+vr7BwcG1rrV27drExERzc/MePXrIVPn5+f3yyy8xMTHdunXr379/TEyMlZWVcmJv5Tit7LaqwsJCXV3dgoICHR2d5o4FAOQrLS3NzMy0srKq09uREu6KlhxLZWZzCHUFIZ62Q+2EyokRmpTcr0Stv+e4BgYA7DDUTuhua6qMJ3EASyGBAQBr8LicBs6Yh9YE18AAAICVkMAAAICVkMAAAICVkMAAAICVkMAAAICVkMAAAICVkMAAAOqAfu3kzZs3a24WExOjp6en7L00Fg6Hc/jw4abfbwMhgQHAh45+v4mZmZmampqFhYW/v39eXl51jc3NzekXHNa8TS8vr/v37zd2pEqnyNH5+PiMHj26qSKqCRIYAHzQHj165OjoeP/+/b179z58+HDz5s2nT592dnbOz8+v2ri8vJzH45mamqqo1PIUCHV1dWNjY+WErJCKiop6rKXg0dVDeXl5o28TCQwAWKXiHfnzB1LxrrG2N3PmTDU1tZMnT/bv3799+/bDhg07derUs2fPgoKC6AaWlpZhYWE+Pj66urpTp06VGWQ7evRo586d1dXVBwwYsGPHDg6H8+bNG/LfIcTQ0NCPP/54165dlpaWurq6Y8eOLSoqoqsSEhL69u2rp6dnaGj46aefZmRk1BqwpaXlsmXLxo8fr6WlZWZmtnHjRqaKw+Fs3rx51KhRmpqaYWFhhJBjx445ODgIBIIOHTosWbKksrKSbvngwYN+/foJBAJbW9vExERmCzJHd+/evREjRujo6Ghra7u5uWVkZISGhu7YsePIkSMcDofD4Zw9e5YQcufOnYEDB6qrqxsaGk6bNu3t27f06nRfLTw83MzMrEuXLnX+t6kNEhgAsErqUZK0jKQda5SN5efnnzhx4ptvvlFXV2cKTU1NJ0yYEBcXxzzrfM2aNXZ2dikpKYsWLZJe/fHjx1988cXo0aNv3rw5ffp0JudVlZGRcfjw4ePHjx8/fvzcuXMrV66ky4uLi+fOnXv16tXTp09zudwxY8bU+kpMOh57e/vr168HBgbOmTNHOgOFhISMGjXqzp07X3311YkTJyZOnDh79uzU1NQtW7bExMQsX76cECKRSD777DMej3f58uXNmzcvWLBA7l6ePXtGJ7mkpKSUlJSvvvqqsrJy/vz5X3755dChQ0UikUgkcnFxKSkpGTp0qL6+/tWrV/fv33/q1KlZs2YxGzl9+nRaWlpiYuLx48drPa46o1qXgoICQkhBQUFzBwIA1Xr37l1qauq7d+/qs3KsFxWiQ+0Z2yiRXL58mRBy6NAhmXL6BV0vXrygKMrCwmL06NFMVWZmJiHkxo0bFEUtWLDAzs6OqaIT2OvXrymK2r59u66uLl0eEhKioaFRWFhIL3733Xe9e/euGkxubi4h5M6dOzJ7kWFhYTF06FBm0cvLa9iwYfRnQkhAQABT5ebmtmLFCmZx165dQqGQoqgTJ07weLzs7Gy6/I8//mBOgvR+AwMDraysysvLZQKYPHnyqFGjmMWtW7fq6+u/ffuWXvz999+5XG5OTg7d0sTEpKysrOpRyJD7laj19xwP8wWAFu9tLkn+mUgqCSEk4zQhhDw8RU4EEUIIV4U4zyJabRp3h3QyoF/BTAhxdHSU2yw9Pb1Xr17MopOTU3UbtLS01NbWpj8LhUI6VxFCMjIyFi1adPny5VevXtF9r6ysrFpniDg7O0t/Xr9+PbMoHWpKSsrVq1fpXhchRCwWl5aWlpSUpKWltW/fvl27dlW3Ju3mzZtubm6qqqo1B5OWlta9e3dNTU160dXVVSKRpKenm5iYEEK6deumpqZW8xbqDQkMAFq8ohxyOZKIywiH+8+bLCVicjmCUBLC4xO7z+udwDp16sThcFJTU2Wm1f3999/6+vpGRkb0IvPrLIOiKCbJkfdpTy7pNMDhcJhxQk9PT3Nz86ioKDMzM4lEYmdnV4/JDtIxSIcqkUiWLFny2WefSTcWCAQycUqvLk16WLUGMidBZpvVnbpGgWtgANDiCe3J9HPEqAshhFDif//fqAuZfo4I7eu9YUNDQ3d394iIiHfv/p0VkpOTExsb6+XlVd0vO8Pa2vrq1avM4rVr1+q097y8vLS0tODg4EGDBtnY2Lx+/VrBFemRT+aztbW13GY9e/ZMT0/v9F9cLtfW1jYrK+v58+d0s+TkZLmr29vbnz9/vupsRjU1NbFYzCza2trevHmzuLiYXrx48SKXy1XGlI2qkMAAgA2MbYjvSdlC35PE2KaBG960aVNZWdmQIUP+/PPP7OzshIQEd3f3tm3bMiNvNZg+ffrff/+9YMGC+/fv//rrrzExMaT6Dk1V+vr6hoaGW7duffjwYVJS0ty5cxVc8eLFi6tXr75///7PP/+8f/9+f39/uc0WL168c+fO0NDQe/fupaWlxcXFBQcHE0IGDx7ctWvXSZMm3bp16/z589XNPZk1a1ZhYeHYsWOvXbv24MGDXbt2paenE0IsLS1v376dnp7+6tWrioqKCRMmCASCyZMn371798yZM99++623tzc9fqhsSGAAwBJPLhFKaoYeJSFP5Hcd6qRz587Xrl3r2LGjl5dXx44dp02bNmDAgOTkZAMDg1rXtbKyOnDgwMGDB+3t7SMjI+lMwOfzFdw1l8vdt29fSkqKnZ3dnDlz1qxZo+CK8+bNS0lJ6dGjx7Jly9auXTtkyBC5zYYMGXL8+PHExMRevXr16dNn3bp1FhYW9H4PHTpUVlbm5OTk5+dXXao2NDRMSkp6+/Zt//79HRwcoqKi6IHQqVOndu3a1dHRsU2bNhcvXtTQ0Dhx4kR+fn6vXr2++OKLQYMGbdq0ScEDaSBODYO2bFRYWKirq1tQUKCjo9PcsQCAfKWlpZmZmVZWVgKBoA6rHZpObu0j7Z2J+zJyMphkXybdx5Exm5UWZp0tX7588+bN2dnZSt2LpaVlQEBAQECAUvfSxOR+JWr9PcckDgBgCWEP0saGuHxLuDwyJZ5c2khU6pL/lCMiIqJXr16GhoYXL15cs2aN9C1QoGxIYADAEn1m/PuZyyN9A5otEikPHjwICwvLz89v3779vHnzAgMDmzuiDwgSGABA/f34448//vhjU+7x8ePHTbm7lgyTOAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAgJWQwAAAGgH99mElbVzmRcnKxuFwDh8+3PT7rSskMAD4oH3yyScyj2U6fPiw4g/krZMWng+qMjc3F4lENb+fTKmZu2ZIYAAArV/Vt6IogsfjmZqaqqg0/iMv6vHas6qQwACABZ4UPknNS636vyeFT5S639DQ0I8//njLli3m5uYaGhr/93//9+bNG7pKLBbPnTtXT0/P0NDw+++/l34wekJCQt++femqTz/9NCMjgy63srIihPTo0YPD4XzyySd04fbt221sbAQCgbW1dUREBLORK1eu9OjRQyAQODo63rhxo7oILS0tly1bNn78eC0tLTMzs40bNzJVHA5n8+bNo0aN0tTUDAsLI4QcO3bMwcFBIBB06NBhyZIllZWVdMsHDx7069dPIBDY2tomJiYyW5DpMt67d2/EiBE6Ojra2tpubm4ZGRmhoaE7duw4cuQIh8PhcDhnz54lhNy5c2fgwIHq6uqGhobTpk17+/YtvTrdVwsPDzczM2uUF4bhUVIA0NI9KXzy6aFPq6s9Pua4hY6F8vb+8OHDX3/99dixY4WFhb6+vjNnzoyNjSWErF27dtu2bdHR0ba2tmvXrj106NDAgQPpVYqLi+fOndutW7fi4uLFixePGTPm5s2bXC73ypUrTk5Op06d+uijj9TU1AghUVFRISEhmzZt6tGjx40bN6ZOnaqpqTl58uTi4uJPP/104MCBu3fvzszMrO6NX7Q1a9b873//Cw0NPXHixJw5c6ytrd3d3emqkJCQ8PDwH3/8kcfjnThxYuLEiT/99BOde6ZNm0Y3kEgkn332mZGR0eXLlwsLC6t7zv2zZ8/69ev3ySefJCUl6ejoXLx4sbKycv78+WlpaYWFhdu3byeEGBgYlJSUDB06tE+fPlevXs3NzfXz85s1axb9pjRCyOnTp3V0dBITExvnRShU61JQUEAIKSgoqPcWyirEv/yZsejwnV/+zCirEDdibABAe/fuXWpq6rt37xRsf+/VPbsYu+r+d+/VvYYE079/f39/f+mSQ4cOMb+NISEhPB4vOzubXvzjjz+4XK5IJKIoSigUrly5ki6vqKho167dqFGjqm4/NzeXEHLnzh2KojIzMwkhN27cYGrNzc337NnDLC5btszZ2ZmiqC1bthgYGBQXF9PlkZGRMisyLCwshg4dyix6eXkNGzaM/kwICQgIYKrc3NxWrFjBLO7atUsoFFIUdeLECZljJIQcOnRIJuDAwEArK6vy8nKZACZPnix94Fu3btXX13/79i29+Pvvv3O53JycHLqliYlJWVlZ1aOQ+5Wo9fdc6UOIERER9CteHBwczp8/X7WBj48P578++ugjpvbNmzczZ84UCoUCgcDGxiY+Pl6p0YbHp1ov+mPZ72k7k58s+z3NetEf4fGpSt0jALRw7du3b9euHf3Z2dlZIpGkp6cXFBSIRCJnZ2e6XEVFxdHRkVklIyNj/PjxHTp00NHRoYcNs7Kyqm755cuX2dnZvr6+Wu+FhYXR441paWndu3fX0NBg9ltDhNK1zs7OaWlpzKJ0VCkpKUuXLmX2NXXqVJFIVFJSkpaWJnOMcvdy8+ZNNzc3+p2WNaAj19TUpBddXV3pM0YvduvWje56NgrlDiHGxcUFBARERES4urpu2bJl2LBhqamp7du3l26zYcOGlStX0p8rKyu7d+/+f//3f/RieXm5u7u7sbHxgQMH2rVrl52dra2trbxow+NTt/yZKV0ioQhdEjjcVnn7BYBmpKOjQ/+lz3jz5k11b1CkZyfWOkfR09PT3Nw8KirKzMxMIpHY2dnJnbMgkUgIIVFRUb1792YKeTweIYRqwAibdHhMIqF3t2TJks8++0y6sUAgkNlXdUenrq6uyN4piqq6BaZEOp6GU24PbN26db6+vn5+fjY2NuvXrzc3N6c7wtJ0dXVN37t27drr16+nTJlCV23bti0/P//w4cOurq4WFhZ9+/bt3r27kkItr5REnc+UWxV1PrO8UiK3CgDYztra+tq1a9IlV69e7dq1K7OYlZX1/Plz+nNycjKXy+3SpYuurq5QKLx8+TJdXllZmZKSQn/Oy8tLS0sLDg4eNGiQjY3N69evmU3RnQ+xWEwvmpiYtG3b9tGjR52k0D02W1vbW7duvXv3jm7J7Egu6drLly9bW1vLbdazZ8/09PRO/8Xlcm1tbWWOUe7q9vb258+frzqbUU1NjTkiOvKbN28WFxfTixcvXqTPWA3x15sSE1h5eXlKSoqHhwdT4uHhcenSpRpWiY6OHjx4sIXFP9djjx496uzsPHPmTBMTEzs7uxUrVkifJkZZWVmhlPpFuyv5saSav3gkFNmV/Lh+mwWAFu6bb77JyMiYOXPmrVu37t+///PPP0dHR3/33XdMA4FAMHny5Fu3bp0/f3727NlffvmlqakpIcTf33/lypWHDh36+++/v/nmG2Z2or6+vqGh4datWx8+fJiUlDR37lxmU8bGxurq6gkJCS9evKC7faGhoeHh4Rs2bLh///6dO3e2b9++bt06Qsj48eO5XK6vr29qamp8fPwPP/xQwyFcvHhx9erVdPD79++vbsbH4sWLd+7cGRoaeu/evbS0tLi4uODgYELI4MGDu3btOmnSJPoYg4KC5K4+a9aswsLCsWPHXrt27cGDB7t27aIHBi0tLW/fvp2env7q1auKiooJEybQZ+zu3btnzpz59ttvvb29TUxM6vBPojAlJrBXr16JxWLpuE1MTHJycqprLxKJ/vjjDz8/P6bk0aNHBw4cEIvF8fHxwcHBa9euXb58edUVw8PDdd8zNzevX7RP8kvqXQsA7GVpaXn+/PmMjAwPD49evXrFxMTExMQwFzIIIZ06dfrss8+GDx/u4eFhZ2fHzHSfN2/epEmTfHx8nJ2dtbW1x4wZQ5dzudx9+/alpKTY2dnNmTNnzZo1zKZUVFR++umnLVu2mJmZjRo1ihDi5+f3yy+/xMTEdOvWrX///jExMXQPTEtL69ixY6mpqT169AgKClq1alUNhzBv3ryUlJQePXosW7Zs7dq1Q4YMkdtsyJAhx48fT0xM7NWrV58+fdatW0f3Frhc7qFDh8rKypycnPz8/OT+zBJCDA0Nk5KS3r59279/fwcHh6ioKPp62NSpU7t27ero6NimTZuLFy9qaGicOHEiPz+/V69eX3zxxaBBgzZt2qTwv0bdcBoy0lqz58+ft23b9tKlS8wlweXLl+/atevvv/+W2z48PHzt2rXPnz9nLvF16dKltLQ0MzOTHhRet27dmjVrRCKRzIplZWVlZWX058LCQnNz84KCguqGsKsTff7Rst/TqqtdNMLG161DnTYIANWh/7ump3cp0j41L9XruFd1tXGfxtkaKusqdWho6OHDh1vyszMsLS0DAgKqm/vOFnK/EoWFhbq6ujX8niuxB2ZkZMTj8aS7XLm5udV1JCmK2rZtm7e3t/QEFaFQ2KVLFzp7EUJsbGxycnKqXgvl8/k6UuoXrbezJbea67JcDvF2tqzfZgGg4TRVa7ryX3MttGJKnIWopqbm4OCQmJjI9KwTExPpXnNV586de/jwoa+vr3Shq6vrnj17JBIJl8slhNy/f18oFDbiFMz/RKvCnepmJTMLkTbVzUpNBY8sAWg2FjoWx8ccL64orlqlqaqp1LuYoSVT4hAiISQuLs7b23vz5s3Ozs5bt26Nioq6d++ehYVFYGDgs2fPdu7cybT09vZ+8OCBzEyb7OxsW1tbHx+fb7/99sGDB1999dXs2bOru8BIq7XLWbPw+NSo85nMbA4uh0x1s8IceoDGVdchRGj16jeEqNz7wLy8vPLy8pYuXUo/zzg+Pp6+ZigSiaRv6ysoKPjtt982bNggs7q5ufnJkyfnzJljb2/ftm1bf3//BQsWKDXgwOG28zysdyU/fpJfYmGg4e1sib4XAEDLpNweWNNrYA8MAJoAemAgo8VN4gAAqEEr++sZGqJ+XwYkMABoavT9QyUluL0S/kFPL2fmnCsIr1MBgKbG4/H09PTox7RraGgo6fXHwBYSieTly5caGhp1fXMmEhgANAP6aUx0DgPgcrnt27ev658ySGAA0Aw4HI5QKDQ2Nq7fq+6hlVFTU6Pv960TJDAAaDY8Hq+ulz0AGJjEAQAArIQEBgAArIQEBgAArNTaroHRd8PV+7WWAADQQtC/5DXc49zaElhRUREhpN6vtQQAgBalqKhIV1dXblVrexaiRCJ5/vy5trZ2Q26NpN+KmZ2djQcqyoXzUyucoprh/NQM54dGUVRRUZGZmVl1M+xbWw+My+W2a9euUTbVkNdjfghwfmqFU1QznJ+a4fwQQqrre9EwiQMAAFgJCQwAAFgJCUwOPp8fEhLC5/ObO5AWCuenVjhFNcP5qRnOj4Ja2yQOAAD4QKAHBgAArIQEBgAArIQEBgAArIQEBgAArPTBJbCIiAgrKyuBQODg4HD+/Hm5bc6dO+fg4CAQCDp06LB582bpqt9++83W1pbP59va2h46dKhJQm5qDTlFUVFRbm5u+vr6+vr6gwcPvnLlSlNF3XQa+BWi7du3j8PhjB49WrmxNocGnp83b97MnDlTKBQKBAIbG5v4+PgmibpJNfAUrV+/vmvXrurq6ubm5nPmzCktLW2SqFsk6kOyb98+VVXVqKio1NRUf39/TU3NJ0+eyLR59OiRhoaGv79/ampqVFSUqqrqgQMH6KpLly7xeLwVK1akpaWtWLFCRUXl8uXLTX4QytXAUzR+/Piff/75xo0baWlpU6ZM0dXVffr0aZMfhBI18PzQHj9+3LZtWzc3t1GjRjVd6E2igeenrKzM0dFx+PDhFy5cePz48fnz52/evNnkB6FcDTxFu3fv5vP5sbGxmZmZJ06cEAqFAQEBTX4QLcWHlcCcnJxmzJjBLFpbWy9cuFCmzffff29tbc0sTp8+vU+fPvTnL7/8cujQoUzVkCFDxo4dq8x4m0EDT5G0yspKbW3tHTt2KCnUZtHw81NZWenq6vrLL79Mnjy59SWwBp6fyMjIDh06lJeXN0GozaWBp2jmzJkDBw5kqubOndu3b19lxtuifUBDiOXl5SkpKR4eHkyJh4fHpUuXZJolJydLtxkyZMi1a9cqKirkVlVdndUafoqklZSUVFRUGBgYKC/gJtYo52fp0qVt2rTx9fVtgoCbWMPPz9GjR52dnWfOnGliYmJnZ7dixQqxWNw0wTeNhp+ivn37pqSk0IPzjx49io+PHzFiRJPE3hK1tof51uDVq1disdjExIQpMTExycnJkWmWk5Mj06aysvLVq1dCobBqVdXVWa3hp0i62cKFC9u2bTt48GClxtyUGn5+Ll68GB0dffPmzaYJuIk1/Pw8evQoKSlpwoQJ8fHxDx48mDlzZmVl5eLFi5voAJSv4ado7NixL1++pHtdlZWVX3/99cKFC5so+pbnA0pgNOnXrFAUJfetKzJtpEsUWZ3tGniKaKtXr967d+/Zs2cFAoHSIm0e9T4/RUVFEydOjIqKMjIyaoI4m0tDvj8SicTY2Hjr1q08Hs/BweH58+dr1qxpTQmM1pBTdPbs2eXLl0dERPTu3fvhw4f+/v5CoXDRokXKj7ol+oASmJGREY/Hk/5jJzc3V/rPHJqpqalMGxUVFUNDQ7lVVVdntYafItoPP/ywYsWKU6dO2dvbKzvmptTA83Pv3r3Hjx97enrS5RKJhBCioqKSnp7esWNH5YevdA3//giFQlVVVR6PR1fZ2Njk5OSUl5erqakpP/ym0PBTtGjRIm9vbz8/P0JIt27diouLp02bFhQUVN0bs1q3D+iY1dTUHBwcEhMTmZLExEQXFxeZZs7OztJtTp486ejoqKqqKreq6uqs1vBTRAhZs2bNsmXLEhISHB0dmyDmptTA82NtbX3nzp2b740cOXLAgAE3b95sNS8Qb/j3x9XV9eHDh3RqJ4Tcv39fKBS2muxFGuMUlZSUSOcqHo9HT2dQcuAtVdPPG2lG9ATW6Ojo1NTUgIAATU3Nx48fUxS1cOFCb29vug09gXXOnDmpqanR0dHSE1gvXrzI4/FWrlyZlpa2cuXKVjyNvt6naNWqVWpqagcOHBC9V1RU1GwHowQNPD/SWuUsxAaen6ysLC0trVmzZqWnpx8/ftzY2DgsLKzZDkY5GniKQkJCtLW19+7d++jRo5MnT3bs2PHLL79stoNpbh9WAqMo6ueff7awsFBTU+vZs+e5c+fowsmTJ/fv359pc/bs2R49eqipqVlaWkZGRkqvvn///q5du9J/Tf/2229NGXmTacgpsrCwkPkLKSQkpGnDV7oGfoUYrTKBUQ0+P5cuXerduzefz+/QocPy5csrKyubMvim0ZBTVFFRERoa2rFjR4FAYG5u/s0337x+/bppw29B8DoVAABgpQ/oGhgAALQmSGAAAMBKSGAAAMBKSGAAAMBKSGAAAMBKSGAAAMBKSGAAAMBKSGAArUFMTIyenp7cKh8fnyZ+9XNoaOjHH3/clHuEDxMSGHxAfHx8OBwOh8NRVVU1MTFxd3fftm0b8+S91mrDhg0xMTGKt5c+Sx06dJg/f35xcXGd9jh//vzTp08zW2vi9AkfDiQw+LAMHTpUJBI9fvz4jz/+GDBggL+//6efflpZWdnccf2jvLy80bepq6tbXeesOvRZevToUVhYWERExPz58xVckaKoyspKLS0t6bcTACgJEhh8WPh8vqmpadu2bXv27Pm///3vyJEjf/zxB9NBKSgomDZtmrGxsY6OzsCBA2/dukWX02Ni27Zta9++vZaW1tdffy0Wi1evXm1qampsbLx8+XJm+1lZWaNGjdLS0tLR0fnyyy9fvHjBVIWFhRkbG2tra/v5+S1cuJAZZKP7KOHh4WZmZl26dCGE7N6929HRUVtb29TUdPz48bm5uXTLs2fPcjic33//vXv37gKBoHfv3nfu3JE+uhMnTtjY2GhpadEZSHr79GeJRLJq1apOnTrx+fz27dtLR171LJmbm48fP37ChAmHDx+uNaoTJ044Ojry+fzz588zQ4ihoaE7duw4cuQI3aU7e/bswIEDZ82axewoLy+Pz+cnJSXV5d8Q4B9IYPBBGzhwYPfu3Q8ePEgIoShqxIgROTk58fHxKSkpPXv2HDRoUH5+Pt0yIyPjjz/+SEhI2Lt377Zt20aMGPH06dNz586tWrUqODj48uXL9BZGjx6dn59/7ty5xMTEjIwMLy8vevXY2Njly5evWrUqJSWlffv2kZGR0mGcPn06LS0tMTHx+PHjhJDy8vJly5bdunXr8OHDmZmZPj4+0o2/++67H3744erVq8bGxiNHjqTfNE8IKSkp+eGHH3bt2vXnn39mZWXJ7TYFBgauWrVq0aJFqampe/bsUeSFdurq6vQuao7q+++/Dw8PT0tLk34J3Pz587/88ks6m4pEIhcXFz8/vz179pSVlTGnxczMbMCAAbWGASBH8z5LGKApyX0AvJeXl42NDUVRp0+f1tHRKS0tZao6duy4ZcsWiqJCQkI0NDQKCwvp8iFDhlhaWorFYnqxa9eu4eHhFEWdPHmSx+NlZWXR5ffu3SOEXLlyhaKo3r17z5w5k9myq6tr9+7dmahMTEzKysrkxnzlyhVCCP1WmjNnzhBC9u3bR1fl5eWpq6vHxcVRFLV9+3ZCyMOHD+mqn3/+2cTEROaoCwsL+Xx+VFSU4mfpr7/+MjQ0rPrCjqpRHT58mKkNCQmRPjrpc15aWmpgYEDHTFHUxx9/HBoaWnM8ANVBDww+dNT7d7qnpKS8ffvW0NBQ673MzMyMjAy6maWlpba2Nv3ZxMTE1taWea+giYkJPZ6WlpZmbm7OvKDS1tZWT08vLS2NEJKenu7k5MTsVPozIaRbt27Sr228cePGqFGjLCwstLW1P/nkE0JIVlYWU+vs7Ex/MDAw6Nq1K719QoiGhgbzZmehUMgM8THS0tLKysoGDRpU6zk5fvy4lpaWQCBwdnbu16/fxo0ba41KwfeX8vn8iRMnbtu2jRBy8+bNW7duyfTkABSn0twBADSztLQ0KysrQohEIhEKhWfPnpWuZaY/MK+cJoTQM/SkF+mpjEwuZEiXSFdR/32NkaamJvO5uLjYw8PDw8Nj9+7dbdq0ycrKGjJkSA2TO5jNyoREVXlTkrq6enUbkTFgwIDIyEhVVVUzMzN6s7VGJX0INfPz8/v444+fPn26bdu2QYMGVX2HHICC0AODD1pSUtKdO3c+//xzQkjPnj1zcnJUVFQ6STEyMlJ8a7a2tllZWdnZ2fRiampqQUGBjY0NIaRr1670sBvt2rVr1W3k77//fvXq1cqVK93c3Kytrat2pOjrbYSQ169f379/39raWsHwOnfurK6uzkxwr4GmpmanTp0sLCyYpFhrVNVRU1MTi8XSJd26dXN0dIyKitqzZ89XX32l4HYAqkICgw9LWVlZTk7Os2fPrl+/vmLFilGjRn366aeTJk0ihAwePNjZ2Xn06NEnTpx4/PjxpUuXgoODa8g0VQ0ePNje3n7ChAnXr1+/cuXKpEmT+vfvT4+tffvtt9HR0Tt27Hjw4EFYWNjt27dl+mqM9u3bq6mpbdy48dGjR0ePHl22bJlMg6VLl54+ffru3bs+Pj5GRkaK32UlEAgWLFjw/fff79y5MyMj4/Lly9HR0QquW2tU1bG0tLx9+3Z6evqrV6+Y+SZ+fn4rV64Ui8VjxoxRcDsAVSGBwYclISFBKBRaWloOHTr0zJkzP/3005EjR3g8HiGEw+HEx8f369fvq6++6tKly9ixYx8/fqzIPD0Gh8M5fPiwvr5+v379Bg8e3KFDh7i4OLpqwoQJgYGB8+fP79mzJz2FTyAQyN1ImzZtYmJi9u/fb2tru3Llyh9++EGmwcqVK/39/R0cHEQi0dGjR6UvntVq0aJF8+bNW7x4sY2NjZeXl+IdqVqjqs7UqVO7du3q6OjYpk2bixcv0oXjxo1TUVEZP358dScBQBFyBsoBQNnc3d1NTU137dpVp7XOnj07YMCA169f1/XG5JYmOzvb0tLy6tWrPXv2bO5YgMUwiQOgKZSUlGzevHnIkCE8Hm/v3r2nTp1KTExs7qCaQUVFhUgkWrhwYZ8+fZC9oIGQwACaAj0+GRYWVlZW1rVr199++23w4MHNHVQzuHjx4oABA7p06XLgwIHmjgVYD0OIAADASpjEAQAArIQEBgAArIQEBgAArIQEBgAArIQEBgAArIQEBgAArIQEBgAArIQEBgAArIQEBgAArPT/Lhz3rPTxfhUAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#To debug this, we can look at the Pareto Frontier of solutions considered, on the training set\n",
    "fpredictor.plot_frontier() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:20:42.460837Z",
     "iopub.status.busy": "2024-06-17T19:20:42.460694Z",
     "iopub.status.idle": "2024-06-17T19:20:43.130390Z",
     "shell.execute_reply": "2024-06-17T19:20:43.130014Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAHFCAIAAACVbdGXAABVy0lEQVR4nO3dd1gU59oG8Hd3KQtIR2BRBNQoEMQCgoBoRAEbts8jxkoCtmAES6LEAlasMR4NqAhi7FFjDaJYg6hRUWxLUAEFdbGAAoK03fn+mGSyZ1lgKQsM3r/rXOfa6c9McO+dmXfe4VAURQAAANiG29QFAAAA1AUCDAAAWAkBBgAArIQAAwAAVkKAAQAAKyHAAACAlRBgAADASggwAABgJQQYAACwEgIMgBBCYmNjOZXMmzevwTdUXFwcFhZ26dKlylt/+vRpQ23l0qVLlXdn9OjRDbV+xTX4rgEwVJq6AIBmZOfOndbW1sygmZlZg2+iuLh46dKlhJAvvviCGTlkyJBr164JBIKG3daqVav69evHDBoaGjbs+gGaFgIM4F92dnaOjo7VzFBeXs7hcFRUGvgfTuvWrVu3bl23ZYuLizU1NeVO+uyzz3r16lWPugCaNVxCBKgOfS1u9+7dc+fObdOmjbq6+pMnTwghMTExXbt25fP5BgYGI0eOTE1NZRbx8/Nr1arVkydPBg8e3KpVK3Nz87lz55aWlhJCnj59SgfV0qVL6ct6fn5+RN51tnPnzvXv319HR0dTU9PNze38+fPMpLCwMA6Hc/v27dGjR+vr63fo0KG2O3XlypX+/ftra2tramq6urr+/vvvMiuXnlmmNktLy6FDh8bHx/fo0UNDQ8Pa2jomJkZ6/uvXr7u5ufH5fDMzs5CQkPLy8tqWB6AgBBjAv8RicYUUZnxISEhWVtbWrVtPnjxpbGwcHh7u7+//+eef//bbb5s2bbp3756Li8vjx4+Z+cvLy4cNG9a/f//jx49//fXXGzduXLNmDSFEIBDEx8cTQvz9/a9du3bt2rXFixdXLmPPnj1eXl46Ojq7du369ddfDQwMvL29pTOMEDJq1KiOHTseOnRo69atVe2ORCKpvDuXL1/28PDIz8+Pjo7ev3+/tra2j4/PwYMHFT9Kd+/enTt37uzZs48fP25vb+/v7//HH3/Qk4RCYf/+/d+/fx8bG7t169Y7d+6sWLFC8TUD1A4FABS1c+fOyv86ysvLL168SAjp06cPM+e7d+80NDQGDx7MjMnKylJXVx83bhw9OHnyZELIr7/+yswwePDgzp0705/fvHlDCAkNDa289czMTIqiioqKDAwMfHx8mKlisbhr165OTk70YGhoKCFkyZIl1ewOXbaMx48fUxTVq1cvY2PjwsJCes6Kigo7O7u2bdtKJBJm5VXVRlGUhYUFn89/9uwZPfjx40cDA4Np06bRg76+vhoaGjk5OczK6XuKzOIADQhnYAD/+uWXX25KYe51/d///R8zz7Vr1z5+/Ehf+qOZm5t7eHhInyFxOBwfHx9m0N7e/tmzZwrWcPXq1by8vMmTJzNnThKJZODAgTdv3iwqKmJmky6pKmvWrJHeHXNz86Kioj///HP06NGtWrWi5+HxeBMnTnz+/HlaWpqCFXbr1q1du3b0Zz6f36lTJ2bvLl682L9/fxMTE2blvr6+Cq4WoLbQiAPgXzY2NnIbcUi3D8zNzZUZQwgxMzNLSEhgBjU1Nfl8PjOorq5eUlKiYA2vXr0ihMht8p6Xl6elpVW5pKq0b99eZnfevHlDUVTl4sk/+6UImdaM6urqHz9+pD/n5uaamppKT5UZBGhACDCAmkm3a6C/vkUikfQML1++NDIyapBt0evZvHlz5QaEzJmNTEmK09fX53K5lYtntkvnbmlpqbq6Oj317du3iq/f0NAwJydHeozMIEADwiVEgNpxcXHR0NDYs2cPM+b58+cXLlzo37+/IovTwcCcslTm5uamp6cnFAodK1FTU6tn8VpaWs7Ozr/99htTgEQi2bNnT9u2bTt16kQIsbS0JITcu3ePWeTkyZOKr79fv37nz5+nTyIJIWKxuFbNQwBqBWdgALWjp6e3ePHiH374YdKkSV9++WVubu7SpUv5fD7d/KFG2traFhYWx48f79+/v4GBgZGREZ0ZjFatWm3evHny5Ml5eXmjR482NjZ+8+bN3bt337x5ExkZWf/6w8PDPT09+/XrN2/ePDU1tYiIiAcPHuzfv58+pRs8eLCBgYG/v/+yZctUVFRiY2Ozs7MVX/miRYtOnDjh4eGxZMkSTU3Nn3/+Wfq+HUDDwhkYQK2FhITs2LHj7t27I0aMmDlz5ueff3716tXPPvtMwcWjo6M1NTWHDRvWs2fPsLCwyjNMmDDh4sWLHz58mDZt2oABA4KCgm7fvq3gGV6N+vbte+HCBS0tLT8/v7Fjx+bn5584cYJpaqGjoxMfH6+trT1hwoTp06fb2dktXLhQ8ZXb2dmdO3dOR0dn8uTJU6dOtbe3l/ucAECD4FAU1dQ1AAAA1BrOwAAAgJUQYAAAwEoIMAAAYCUEGAAAsBICDAAAWAkBBgAArNTSHmSWSCQvX77U1tauW0c7AADQTFAUVVhYaGZmxuXKP9dqaQH28uVLc3Pzpq4CAAAaRnZ2dtu2beVOamkBpq2tTQjJzs7W0dFp6loAAKDuCgoKzM3N6W91uVpagNFXDnV0dBBgAAAtQDX3g9CIAwAAWAkBBgAArIQAAwAAVkKAAQAAKyHAAACAlRBgAADASggwAABgJQQYAACwEgIMAABYqaX1xAEAn6ayCsnua0+f5RVbGGhOdLFUU/n317lYQt3IzHtdWGKszXeyMuBxW05P3y141xSBAAMARTXbr8vwOGFUYqaE+ntwZVzqFHerkMG2hJD4B6KlJ4Wi/BJ6kkCXH+pjO9BO0FSl1oH8w17+8dGx1VMfOz8t+Hu32bhr9cShKKrmudijoKBAV1c3Pz8ffSECVKMOUdRskyA8Trjtj8zK46f1sereTn/GntvS33H0fkZO6NEcKldEVYf97u/but78Pqjsm+OS3vQk1u1ajWr8PkeAAdSd8s5IlHquU9soKquQ/PDbvcO3X0iPbCZfl2UVEuvFpyXyvsY4hJjo8HMKSiqPN9XlX5nv0XzOIKsS/0AkN4B/Hte91dGJfahbZ8UOU8vnSk9ly64posbvc1xCBFBI5USpHAOLh9joa6nXIXXolefkf8wrKjNopZ6VW7z/RhbzzavNV+nRTr/PZ0Yyt3YY1dz+qazyd2JOfsmMPberiqLwOOH2PzIrBwRFCIeQpSeFnramTfh1ufvaU7npRQihCKmcXvR4UX7Jjcw8lw6Gyi2ufsQSaulJofTOGZF8f5U4VSLOPbpvgCSFcMgX3LsLVfYQQioIb0fF4Fyiy4pdaygIMICaVc6qYV0FMl/rovySb/bdYQYVv8Ims/LKCksqLj96c/nRG+lbO4yqbv/IPYcrq5D8cPSBzBc+Pbjgt/va6qq9OhhKp1FVV+eYBZv86/JZXnHdFnxdWOUBbyZuZObJ/FUYc959zTutzqkQSzgU4RBCuETyNe80j0OVUionxS65lC5pHrvWOLdLEWAANZB7ylLN1zozTzWnNdWsvBoSitDbZTKscsDQ82S8LXrwokDmIiEhJOS3+++Ky+Wu/H1x+fjoP6Vzt6xCEpVYw26Spv66tDDQrNuCxtr8hq2kwVU+sELKcmjZqkjVn6w4IhWOhBCiwpGIKc4TidmM8uDH1N+vLW7yXWu026V4DgygOpUv45B/TlmqR8+z9KRQXNUVripWXqPtf2SWVUgIIWUVku1V5GiC8LX0j/ec/JLpe25P33O7qvSSnnPGntvxD0Sk2qtz0pr263Kii2VVP+45hJjq8CtP5BAi0OU7WRkoubT6kntgH1NtR5WFSe8Uh5BRZWF0ejWHXaN/k8n8+TF/VA0LAQZQncqXcRTHXGFr2JVThEz55ebxlBdLj8teDKxmEcVXTv7J3RqvzjWHr0s1Fe4Udyu5k6b2sQobZkv+afhAoz+H+tg2/2YOTlYGAl05AdyL+xeX83frO4oiXA7lxE0jzWPXqvnBV/2PubpBgAFUp/7Xx3YkpjOfxRLqWnru8ZQX19JzxRKqziu//Oht0IGUvTez61mbXEzuKnJ1rjkkQchg22l9rKSr4HLItD5WIYNtB9oJIif0MNX991TGVJff5C0nFcTjcugLvzIB7M27SQjhtHO51u/APa41IWQg7wZpHrtW1W+yGn/M1Q3ugQFUp/7Xx87/9SY8Thgy2LbyjYGxPc3rXaCyvC4smehiuTIutaofzc3nOTBCSMhg27le1nKbYg60E3jamjbP569rRAew9J+NqS7f1tqdtB5AXL914fLEbp7P4tZ2Lufu79arOexa9b/JGvx2KQIM4F+Vm069Kyqt/2qjEjO7mOl+eyBFpiXIxnOP9TRV39d0X6pJGGvz6atzcpur+Nib/jS2R5N/XUpTU+H6u7eXO4nH5bC3Wbm8AO7PTOWpqFgM+6EJy5NR/Q++Br9digAD+Jvc57qW/55a/zVLKBJy7H7lGwP01z+nNveoGoeehip9Z4tu7ijdTJ/LIZWb8oNSsSiA6ft2OfklMn/S9BPWDX67FD1xABBSRZcHjfNvY/aATv+98EgsaZSNKWZ0jzbrx3RjBmv1oDR84uh/SkTqn0+dO21BTxwANRNLqLATD+vQVp7GUX3L4cm50kiJ1alyoxoXL/hY9mjFYNfV514VlCm8TeVy6/g/ZVdzdQ5Ahtz7dkq6XYoAAyBjtl3NKajjvS6O6ttWHddXNfXDk3k1Zlh00tOeVgZ//uCZX1z+deyN9DdF7z/W4q4YhxBjbbWiMvGHUnH1s03tY3U85aUie2qs09wf8oXmrNEaziDA4FM35Zebyc/e13lxuede0lOpckII4XAIoao8q6N7FNTVVD3yjRshJP6BKOyEUG4/frLrJ4QQMs7ZcuO5R9XMxrQY/H6gDf218rqgdGVc1bf3WtSNBWgCjXPfDgEG9VL/Hs9qe39FZouEkPoU8LFMnCB8XduapSm4vervNcv0KEj/gA0+cPvkvZzKM2uqcovL/75jRl+cKa2o7gbazH4dZnt2po8M87VyPOVFNYu8bYi2lwDKhgCDuquqx7OPZeJlpx5cz8hT43FHdm/zde/2PC5HbsxU8x5CRbaop6lKCGGaoRtoqY3oZuZpa6p4sK2KE9Z2r/U1VSmpjaor3KKha1udu88Lqpoq84gMj8vZPM7BTE9YuQUgcxbF7Nq19NxqtuvWsXXl3W/k5s4AyoAAgzqq6q0cXdrq3JP6ml4dn7Y6Pk1LjVdU9vcdGibnquqIlkh1Vlv9FmWeoMorKotJehqT9FQm2Kp55PZpbi36Mu9sqh3m87lMOq67eO6xYotbGbWqJsDkZkZVz+fKXJypQ9vlRm7uDKAMaA4LdVFVj2cUIffkfUcz6UX+yblTKS+r6uk8KvHvzmpr3GJV3heXS2dbNX2JWhrWoi9zJ0t9lw6GPC6HvhA3vFsblw6GbfQ1FFz8/3q0ldu1XfU9CtItAJcNt/N3b1/V9dWq+hwiVXf1VIdFAJobBBjURT27uKUImXfkblV9FEkosvva04bdIqmiL9EfavNArtyZv3ZTqH25nqaqa0cj5WVGHTr9Y3U/gQAElxChburfp1lJeXXtDp7lFcs01qjnFqt69aKGGs/T1liRdhyetsYaarzK49VVFfoVuHpUFx6Xo9RHZOrQdpnV/QQCIMCgLpR9k7+4tKL3mgsN3u+t3BSMmtRzyi83q88wT1vjqEk95U7SUtWqZkFKrG6qox427HMmopSaGXVou8yibooAZKArKagLsYTqveZC5SYADYLDkW10TvfqpKepml9cXp8t7p/Sq6ov649l4lVxwqe5xZaGmvSlQulBuedejGcFz4rKi0rLJTFJGS/fl5jp8f1crB6/+VBUwrM16oDTGoC6qfH7HAEGdSS3x7MG+WPSVOMVl8l2KsEhRFdTNb+4nNRpK3TjuivzPZAlAGxR4/c5GnFAHcltAmDftl6/G7gcMtReUDm9CCH0o1fBAzpJb1FfU5VuMV+92jaUkHntpILFA0Ajwz0wqDu5t3Pk3k+Sfg5MLi9bE2crg4kulqcfiE7dk9PenWZppHllvkflnjjOCXOOprzIK/q76bzMc2C1aihR1dPZiiwLAI0JlxCh4cntieN6Rm7g3tuVu6mVubh3LT33y6jrVa25mptYDdLFlNyXqpA6vQkCAOoJ98CgGVHkRUFVNQ9phJtY9KYrP22G+2cATaLp74FFRERYWVnx+XwHB4fExES58+zdu7dr166ampoCgeCrr77Kzf23Y7f3798HBgYKBAI+n29jYxMXF6fsgkF5FHlytgl7iKjqWWnmGTLlbRoA6kC598AOHjwYHBwcERHh5ua2bdu2QYMGCYXCdu3aSc9z5cqVSZMmbdy40cfH58WLF9OnTw8ICDh69CghpKyszNPT09jY+PDhw23bts3OztbW1lZqwaBsijwF1ZgvxJNW/bPS9X92GwAalnID7Mcff/T39w8ICCCE/PTTT2fOnImMjAwPD5ee5/r165aWlrNmzSKEWFlZTZs2be3atfSkmJiYvLy8q1evqqqqEkIsLCyUWi00DkWenG2SHiLQQTsAuyjxEmJZWVlycrKXlxczxsvL6+rVqzKzubq6Pn/+PC4ujqKoV69eHT58eMiQIfSkEydOuLi4BAYGmpiY2NnZrVq1SiyuriUbtCTSHeY2zs0nuoP22na2CwBNRYkB9vbtW7FYbGJiwowxMTHJyZF9QZ+rq+vevXt9fX3V1NRMTU319PQ2b95MT8rIyDh8+LBYLI6Li1u0aNGGDRtWrlxZeUOlpaUFUpS3R9CyoYN2AHZReiMODufff/YURUkP0oRC4axZs5YsWZKcnBwfH5+ZmTl9+nR6kkQiMTY23r59u4ODw9ixYxcuXBgZGVl5E+Hh4br/MDdvgB7z4JM10E4wtY+V9B8ph0Om9rFCG3qAZkiJAWZkZMTj8aRPuV6/fi19QkYLDw93c3P77rvv7O3tvb29IyIiYmJiRCIRIUQgEHTq1InH+7sbOhsbm5ycnLKyMpk1hISE5P8jOztbeXsELV78A9H2PzKlO9+QUGT7H5ly3yUGAE1LiQGmpqbm4OCQkJDAjElISHB1dZWZrbi4mMv9tww6ruin09zc3J48eSKR/P3ejUePHgkEAjU1NZk1qKur60hRxr7Ap6Cad2bKfZcYADQt5V5CnDNnzo4dO2JiYlJTU2fPnp2VlUVfHgwJCZk0aRI9j4+Pz2+//RYZGZmRkZGUlDRr1iwnJyczMzNCyIwZM3Jzc4OCgh49evT777+vWrUqMDBQqQXDpwzPgQGwi3Kb0fv6+ubm5i5btkwkEtnZ2cXFxdFN4UUiUVZWFj2Pn59fYWHhli1b5s6dq6en5+HhsWbNGnqSubn52bNnZ8+ebW9v36ZNm6CgoPnz5yu1YPiU4TkwAHZBV1IAf6tzN4wAoAxN35UUAFvgOTAAdkGAAfwNz4EBsAsCDOBfinQ3DADNBF5oCfA/mqQbRgCoAwQYgCxFuhsGgCaHS4gAAMBKCDAAAGAlBBgAALASAgwAAFgJAQYAAKyEAAMAAFZCgAEAACshwAAAgJUQYAAAwEoIMAAAYCUEGAAAsBICDAAAWAkBBgAArIQAAwAAVkKAAQAAKyHAAACAlRBgAADASggwAABgJQQYAACwEgIMAABYCQEGAACshAADAABWQoABAAArIcAAAICVEGAAAMBKCDAAAGAlBBgAALASAgwAAFgJAQYAAKyEAAMAAFZCgAEAACshwAAAgJUQYAAAwEoIMAAAYCUEGAAAsBICDAAAWAkBBgAArIQAAwAAVkKAAQAAKyHAAACAlRBgAADASggwAABgJQQYAACwEgIMAABYCQEGAACshAADAABWQoABAAArKT3AIiIirKys+Hy+g4NDYmKi3Hn27t3btWtXTU1NgUDw1Vdf5ebmysxw4MABDoczYsQIZVcLAABsodwAO3jwYHBw8MKFC+/cuePu7j5o0KCsrCyZea5cuTJp0iR/f/+HDx8eOnTo5s2bAQEB0jM8e/Zs3rx57u7uSi0VAADYRbkB9uOPP/r7+wcEBNjY2Pz000/m5uaRkZEy81y/ft3S0nLWrFlWVla9e/eeNm3arVu3mKlisXj8+PFLly5t3769UksFAAB2UWKAlZWVJScne3l5MWO8vLyuXr0qM5urq+vz58/j4uIoinr16tXhw4eHDBnCTF22bFnr1q39/f2VVycAALCRivJW/fbtW7FYbGJiwowxMTHJycmRmc3V1XXv3r2+vr4lJSUVFRXDhg3bvHkzPSkpKSk6OjolJaX6DZWWlpaWltKfCwoKGmwHAACgGVN6Iw4Oh8N8pihKepAmFApnzZq1ZMmS5OTk+Pj4zMzM6dOnE0IKCwsnTJgQFRVlZGRU/SbCw8N1/2Fubt7guwAAAM0Qh6IoJa26rKxMU1Pz0KFDI0eOpMcEBQWlpKRcvnxZeraJEyeWlJQcOnSIHrxy5Yq7u/vLly9fvXrVvXt3Ho9Hj5dIJIQQLpeblpbWoUMH6TXInIGZm5vn5+fr6Ogoab8AAKARFBQU6OrqVvN9rsRLiGpqag4ODgkJCUyAJSQkDB8+XGa24uJiFZV/y6ATi6Ioa2vr+/fvM+MXLVpUWFi4adOmyudY6urq6urqStkHAABorpQYYISQOXPmTJw40dHR0cXFZfv27VlZWfTlwZCQkBcvXvzyyy+EEB8fnylTpkRGRnp7e4tEouDgYCcnJzMzM0KInZ0dsyo9PT2ZMQAA8ClTboD5+vrm5uYuW7ZMJBLZ2dnFxcVZWFgQQkQiEfNAmJ+fX2Fh4ZYtW+bOnaunp+fh4bFmzRqlVgUAAC2AEu+BNYkar5kCAAAr1Ph9jr4QAQCAlRBgAADASggwAABgJQQYAACwEgIMAABYCQEGAACshAADAABWQoABAAArIcAAAICVEGAAAMBKCDAAAGAlBBgAALCScnujh5ZBLKFuZOa9Liwx1uY7WRnwuLKv1QYAaHwIMKhB/APR0pNCUX4JPSjQ5Yf62A60EzRtVQAAuIQI1Yl/IJqx5zaTXoSQnPySGXtuxz8QNWFVAAAEAQbVEEuopSeFMu+LoweXnhSKJS3qTXIAwDoIMKjSjcw86XMvBkWIKL/kRmZe45cEAMBAgEGVXhfKSS8FpwIAKBsCDKpkrM2v81QAAGVDgEGVnKwMBLr8yk3mOYQIdPlOVgZNUBMAwD8QYFAlHpcT6mNLCJHOMPpzqI8tngYDgKaFAIPqDLQTRE7oYar779VCU11+5IQeeA4MAJocHmSGGgy0E3jamqInDgBobhBgUDMel+PSwbCpqwAA+B+4hAgAAKyEAAMAAFZCgAEAACshwAAAgJUQYAAAwEoIMAAAYCU0owfWwIuhAUAaAgzYAS+GBgAZuIQILIAXQwNAZQgwaO7wYmgAkAsBBs0dXgwNAHIhwKC5w4uhAUAuBBg0d3gxNADIhQCD5g4vhgYAuRBg0NzhxdAAIBcCDFgAL4YGgMoUepDZ0tLy66+/9vPza9eunbILApALL4YGABkKnYHNnTv3+PHj7du39/T0PHDgQGlpqbLLAqiMfjH08G5tXDoYIr0AQKEA+/bbb5OTk5OTk21tbWfNmiUQCGbOnHn79m1lFwcAAFAVDkXVriOD8vLyiIiI+fPnl5eX29nZBQUFffXVVxxOc/k5XFBQoKurm5+fr6Oj09S1AABA3dX4fV6LznzLy8uPHj26c+fOhISEXr16+fv7v3z5cuHChefOndu3b18DFQwAAKAQhQLs9u3bO3fu3L9/P4/Hmzhx4saNG62trelJXl5effr0UWaFAAAAcigUYD179vT09IyMjBwxYoSqqqr0JFtb27FjxyqnNgAAgCopFGAZGRkWFhZyJ2lpae3cubNBSwIAAKiZQq0QX79+/eeff0qP+fPPP2/duqWckgAAAGqmUIAFBgZmZ2dLj3nx4kVgYKBySgIAAKiZQgEmFAp79OghPaZ79+5CoVA5JQEAANRMoQBTV1d/9eqV9BiRSKSiUosm+AAAAA1LoQDz9PQMCQnJz8+nB9+/f//DDz94enoqsmxERISVlRWfz3dwcEhMTJQ7z969e7t27aqpqSkQCL766qvc3Fx6fFRUlLu7u76+vr6+/oABA27cuKHIFgEA4FOgUIBt2LAhOzvbwsKiX79+/fr1s7KyysnJ2bBhQ40LHjx4MDg4eOHChXfu3HF3dx80aFBWVpbMPFeuXJk0aZK/v//Dhw8PHTp08+bNgIAAetKlS5e+/PLLixcvXrt2rV27dl5eXi9evKjtHgIAQIukaFdSRUVFe/fuvXv3roaGhr29/ZdffinzQJhczs7OPXr0iIyMpAdtbGxGjBgRHh4uPc/69esjIyPT09Ppwc2bN69du1amzQghRCwW6+vrb9myZdKkSdVsEV1JAQC0DA3WlZSWltbUqVNrte2ysrLk5OQFCxYwY7y8vK5evSozm6ur68KFC+Pi4gYNGvT69evDhw8PGTKk8tqKi4vLy8sNDPD6XQAAIKRWfSEKhcKsrKyysjJmzLBhw6qZ/+3bt2Kx2MTEhBljYmKSk5MjM5urq+vevXt9fX1LSkoqKiqGDRu2efPmymtbsGBBmzZtBgwYUHlSaWkp84aXgoICxfcIAADYS9GeOEaOHHn//n0O5+9LjnT382KxuMZlpTuqpyiqcr/1QqFw1qxZS5Ys8fb2FolE33333fTp06Ojo6XnWbt27f79+y9dusTn80kl4eHhS5cuVWRHAACgxVCoEUdQUJCVldWrV680NTUfPnz4xx9/ODo6Xrp0qfqljIyMeDye9CnX69evpU/IaOHh4W5ubt999529vb23t3dERERMTIxIJGJmWL9+/apVq86ePWtvby93Q3QLSVrlm2cAANAiKRRg165dW7ZsWevWrblcLpfL7d27d3h4+KxZs6pfSk1NzcHBISEhgRmTkJDg6uoqM1txcTGX+28ZPB6PEMI0LVm3bt3y5cvj4+MdHR2r2pC6urqOFEX2CAAA2E6hABOLxa1atSKEGBkZvXz5khBiYWGRlpZW44Jz5szZsWNHTExMamrq7Nmzs7Kypk+fTggJCQlhGhP6+Pj89ttvkZGRGRkZSUlJs2bNcnJyMjMzI4SsXbt20aJFMTExlpaWOTk5OTk5Hz58qPOuAgBAS6LQPTA7O7t79+61b9/e2dl57dq1ampq27dvb9++fY0L+vr65ubmLlu2TCQS2dnZxcXF0b3ai0Qi5oEwPz+/wsLCLVu2zJ07V09Pz8PDY82aNfSkiIiIsrKy0aNHMysMDQ0NCwur7U4CAEDLo9BzYGfOnCkqKho1alRGRsbQoUP/+usvQ0PDgwcPenh4NEKJtYLnwAAAWoYav88VfZBZWl5enr6+fuX2hM0BAgwAoGWo8fu85ntgFRUVKioqDx48YMYYGBg0z/QCAIBPR80BpqKiYmFhocgjXwAAAI1GoVaIixYtCgkJycvLU3Y1AAAAClKoFeJ///vfJ0+emJmZWVhYaGlpMeNv376ttMIAAACqo1CAjRgxQsllAAAA1E5dWiE2Z2iFCADQMjRAK0QAAIBmSKFLiFwuV267eTRNBACApqJQgB09epT5XF5efufOnV27duENJgAA0ITqeA9s3759Bw8ePH78eIMXVE+4BwYA0DIo6x6Ys7PzuXPn6lEYAABAvdQlwD5+/Lh58+a2bds2eDUAAAAKUugemHTXvRRFFRYWampq7tmzR5mFAQAAVEehANu4cSMTYFwut3Xr1s7Ozvr6+sosDAAAoDoKBZifn5+SywAAAKgdhe6B7dy589ChQ9JjDh06tGvXLuWUBAAAUDOFAmz16tVGRkbSY4yNjVetWqWckgAAAGqmUIA9e/bMyspKeoyFhUVWVpZySgIAAKiZQgFmbGx879496TF37941NDRUTkkAAAA1UyjAxo4dO2vWrIsXL4rFYrFYfOHChaCgoLFjxyq7OAAAgKoo1ApxxYoVz54969+/v4qKCiFEIpFMmjQJ98AAAKAJ1aIvxMePH6ekpGhoaHTp0sXCwkKpZdUZ+kIEAGgZavw+V+gMjPbZZ5999tlnDVQYAABAvSh0D2z06NGrV6+WHrNu3br//Oc/yikJAACgZgoF2OXLl4cMGSI9ZuDAgX/88YdySgIAAKiZQgH24cMHNTU16TGqqqoFBQXKKQkAAKBmCgWYnZ3dwYMHpcccOHDA1tZWOSUBAADUTKFGHIsXL/6///u/9PR0Dw8PQsj58+f37dt3+PBhJdcGAABQJYUCbNiwYceOHVu1atXhw4c1NDS6du164cIFtFMHAIAmVIvnwGjv37/fu3dvdHT03bt3xWKxksqqMzwHBgDQMtT4fa7QPTDahQsXJkyYYGZmtmXLlsGDB9+6dauBigQAAKi1mi8hPn/+PDY2NiYmpqioaMyYMeXl5UeOHEELDgAAaFo1nIENHjzY1tZWKBRu3rz55cuXmzdvbpyyAAAAqlfDGdjZs2dnzZo1Y8YMdCIFAADNSg1nYImJiYWFhY6Ojs7Ozlu2bHnz5k3jlAUAAFC9GgLMxcUlKipKJBJNmzbtwIEDbdq0kUgkCQkJhYWFjVMfAACAXLVrRp+WlhYdHb179+737997enqeOHFCeZXVDZrRAwC0DA3ZjJ4Q0rlz57Vr1z5//nz//v0NUR4AAEAd1fpB5mYOZ2AAAC1DA5+BAQAANBMIMAAAYCUEGAAAsBICDAAAWAkBBgAArIQAAwAAVkKAAQAAKyHAAACAlRBgAADASggwAABgJQQYAACwEgIMAABYSekBFhERYWVlxefzHRwcEhMT5c6zd+/erl27ampqCgSCr776Kjc3l5l05MgRW1tbdXV1W1vbo0ePKrtaAABgC+UG2MGDB4ODgxcuXHjnzh13d/dBgwZlZWXJzHPlypVJkyb5+/s/fPjw0KFDN2/eDAgIoCddu3bN19d34sSJd+/enThx4pgxY/7880+lFgwAAGyh3NepODs79+jRIzIykh60sbEZMWJEeHi49Dzr16+PjIxMT0+nBzdv3rx27drs7GxCiK+vb0FBwenTp+lJAwcO1NfXr/5VZHidCgBAy9CUr1MpKytLTk728vJixnh5eV29elVmNldX1+fPn8fFxVEU9erVq8OHDw8ZMoSedO3aNenFvb29Ky8OAACfJiUG2Nu3b8VisYmJCTPGxMQkJydHZjZXV9e9e/f6+vqqqamZmprq6elt3ryZnpSTk1Pj4oSQ0tLSAilK2BUAAGh2lN6Ig8PhMJ8pipIepAmFwlmzZi1ZsiQ5OTk+Pj4zM3P69OmKL04ICQ8P1/2Hubl5Q+8BAAA0R0oMMCMjIx6PJ33O9Pr1a+kzKlp4eLibm9t3331nb2/v7e0dERERExMjEokIIaampjUuTggJCQnJ/wd98wwAAFo8JQaYmpqag4NDQkICMyYhIcHV1VVmtuLiYi733zJ4PB4hhG5a4uLiIr342bNnKy9OCFFXV9eR0rB7AQAAzZOKUtc+Z86ciRMnOjo6uri4bN++PSsri748GBIS8uLFi19++YUQ4uPjM2XKlMjISG9vb5FIFBwc7OTkZGZmRggJCgrq06fPmjVrhg8ffvz48XPnzl25ckWpBQMAAFsoN8B8fX1zc3OXLVsmEons7Ozi4uIsLCwIISKRiHkgzM/Pr7CwcMuWLXPnztXT0/Pw8FizZg09ydXV9cCBA4sWLVq8eHGHDh0OHjzo7Oys1IIBAIAtlPscWOPDc2AAAC1DUz4HBgAAoDwIMAAAYCUEGAAAsBICDAAAWAkBBgAArIQAAwAAVkKAAQAAKyHAAACAlRBgAADASggwAABgJQQYAACwEgIMAABYCQEGAACshAADAABWUu77wEBBYgl1IzPvdWGJsTbfycqAx+U0dUUAAM0dAqzpxT8QLT0pFOWX0IMCXX6oj+1AO0HTVgUA0MzhEmITi38gmrHnNpNehJCc/JIZe27HPxA1YVUAAM0fAqwpiSXU0pNCmVdi04NLTwrFkhb1smwAgIaFAGtKNzLzpM+9GBQhovySG5l5jV8SAABbIMCa0utCOeml4FQAgE8cAqwpGWvz6zwVAOAThwBrSk5WBgJdfuUm8xxCBLp8JyuDJqgJAIAlEGBNicflhPrYEkKkM4z+HOpji6fBAACqgQBrYgPtBJETepjq/nu10FSXHzmhB54DAwCoHh5kbnoD7QSetqboiQMAoFYQYM0Cj8tx6WDY1FUAALAJLiECAAArIcAAAICVEGAAAMBKCDAAAGAlBBgAALASAgwAAFgJAQYAAKyEAAMAAFZCgAEAACshwAAAgJUQYAAAwEoIMAAAYCUEGAAAsBICDAAAWAkBBgAArIQAAwAAVkKAAQAAKyHAAACAlRBgAADASggwAABgJQQYAACwEgIMAABYCQEGAACshAADAABWQoABAAArIcAAAICVlB5gERERVlZWfD7fwcEhMTGx8gx+fn6c//X5558zU3/66afOnTtraGiYm5vPnj27pKRE2QUDAAArqCh17QcPHgwODo6IiHBzc9u2bdugQYOEQmG7du2k59m0adPq1avpzxUVFV27dv3Pf/5DD+7du3fBggUxMTGurq6PHj3y8/MjhGzcuFGpNQNAoxGLxeXl5U1dBTQ9NTU1LrfWJ1QciqKUUQ3N2dm5R48ekZGR9KCNjc2IESPCw8Ormv/YsWOjRo3KzMy0sLAghMycOTM1NfX8+fP01Llz5964cUPuaRyjoKBAV1c3Pz9fR0en4fYDABoYRVE5OTnv379v6kKgWeByuVZWVmpqatIja/w+V+IZWFlZWXJy8oIFC5gxXl5eV69erWaR6OjoAQMG0OlFCOndu/eePXtu3Ljh5OSUkZERFxc3efJk5RUMAI2GTi9jY2NNTU0Oh9PU5UBTkkgkL1++FIlE7dq1q9UfgxID7O3bt2Kx2MTEhBljYmKSk5NT1fwikej06dP79u1jxowdO/bNmze9e/emKKqiomLGjBnSccgoLS0tLS2lPxcUFDTcHgCAUojFYjq9DA0Nm7oWaBZat2798uXLiooKVVVVxZdSeiMO6TilKKqadI2NjdXT0xsxYgQz5tKlSytXroyIiLh9+/Zvv/126tSp5cuXV14wPDxc9x/m5uYNWj4ANDz6vpempmZTFwLNBX3xUCwW12opJZ6BGRkZ8Xg86VOu169fS5+QSaMoKiYmZuLEidLXQBcvXjxx4sSAgABCSJcuXYqKiqZOnbpw4UKZe30hISFz5syhPxcUFCDDAFgBVw6BUbc/BiWegampqTk4OCQkJDBjEhISXF1d5c58+fLlJ0+e+Pv7S48sLi6Wzioej0dRVOVWJ+rq6jpSGm4PAACg+VLuJcQ5c+bs2LEjJiYmNTV19uzZWVlZ06dPJ4SEhIRMmjRJes7o6GhnZ2c7OzvpkT4+PpGRkQcOHMjMzExISFi8ePGwYcN4PJ5SawYAaHBhYWHdunVr6ipaGuU+B+br65ubm7ts2TKRSGRnZxcXF0e3MBSJRFlZWcxs+fn5R44c2bRpk8ziixYt4nA4ixYtevHiRevWrX18fFauXKnUggHgU+Pn57dr1y7pMY8fP+7YsWM9V8vhcI4ePcrc1J83b963335bz3WCDOU+B9b48BwYQPNXUlKSmZlJ99FTqwXFEupGZt7rwhJjbb6TlQGP2wB30fz8/F69erVz505mTOvWrZkrPWVlZTIPJylIJsDqoLy8vFZN8lhN7p9Ejd/n6AsRANgh/oGo95oLX0ZdDzqQ8mXU9d5rLsQ/EDXImtXV1U2l9O/ff+bMmXPmzDEyMvL09CSEXL582cnJSV1dXSAQLFiwoKKigl7wiy++mDVr1vfff29gYGBqahoWFkaPt7S0JISMHDmSw+HQn2UuIe7cudPGxobP51tbW0dERNAjnz59yuFwfv311y+++ILP5+/Zs6dB9q4FQ4ABAAvEPxDN2HNblP9vb6g5+SUz9txuqAyTsWvXLhUVlaSkpG3btr148WLw4ME9e/a8e/duZGRkdHT0ihUrpOfU0tL6888/165du2zZMrrZ2s2bNwkhO3fuFIlE9GdpUVFRCxcuXLlyZWpq6qpVqxYvXix9DXP+/PmzZs1KTU319vZWxq61JMq9BwYAUH9iCbX0pFDmbgdFCIeQpSeFnram9byWeOrUqVatWtGfBw0aRAjp2LHj2rVr6TELFy40NzffsmULh8OxtrZ++fLl/PnzlyxZQreRtre3Dw0NJYR89tlnW7ZsOX/+vKenZ+vWrQkhenp6pqamlTe3fPnyDRs2jBo1ihBiZWUlFAq3bdvGdDMUHBxMT4IaIcAAoLm7kZknfe7FoAgR5ZfcyMxz6VCvHj369evHdNmqpaX15ZdfOjo6MlNTU1NdXFyYB5Xc3Nw+fPjw/Plzul9ye3t7Zk6BQPD69evqt/XmzZvs7Gx/f/8pU6bQYyoqKnR1dZkZpDcN1UOAAUBz97qwuvcoVT9VEVpaWjLNDrW0tJjPMl0I0Q3fmDHS7Sw4HI5EIql+W/QMUVFRzs7OzEjpp4OkNw3VQ4ABQHNnrF1dY8Xqp9afra3tkSNHmBi7evWqtrZ2mzZtql9KVVVVbsdIJiYmbdq0ycjIGD9+vFLK/ZSgEQcANHdOVgYCXX7l21wcQgS6fCcrA6Vu/ZtvvsnOzv7222//+uuv48ePh4aGzpkzp8aXV1laWp4/fz4nJ+fdu3cyk8LCwsLDwzdt2vTo0aP79+/v3Lnzxx9/VFr5LRkCDACaOx6XE+pjSwiRzjD6c6iPbYM8DVaNNm3axMXF3bhxo2vXrtOnT/f391+0aFGNS23YsCEhIcHc3Lx79+4ykwICAnbs2BEbG9ulS5e+ffvGxsZaWVkpp/YWDg8yA0Bjq9uDzPEPREtPCpnWHAJdfqiP7UA7gXJqhEZVtweZcQ8MANhhoJ3A09a0wXviAPZCgAEAa/C4nHq2mIeWBPfAAACAlRBgAADASggwAABgJQQYAACwEgIMAABYCQEGAACshAADAKgF+rWTKSkp1c8WGxurp6en7K00FA6Hc+zYscbfbj0hwADgU0e/38TMzExNTc3CwiIoKCg3N7eqmc3NzUUikZ2dXfXr9PX1ffToUUNXqnSK7J2fn9+IESMaq6LqIMAA4JOWkZHh6Oj46NGj/fv3P3nyZOvWrefPn3dxccnLy6s8c1lZGY/HMzU1VVGpoRcIDQ0NY2Nj5ZSskPLy8jospeDe1UFZWVmDrxMBBgCsUv6R/LGelH9sqPUFBgaqqamdPXu2b9++7dq1GzRo0Llz5168eLFw4UJ6BktLyxUrVvj5+enq6k6ZMkXmItuJEyc+++wzDQ2Nfv367dq1i8PhvH//nvzvJcSwsLBu3brt3r3b0tJSV1d37NixhYWF9KT4+PjevXvr6ekZGhoOHTo0PT29xoItLS2XL18+bty4Vq1amZmZbd68mZnE4XC2bt06fPhwLS2tFStWEEJOnjzp4ODA5/Pbt2+/dOnSiooKes7Hjx/36dOHz+fb2tomJCQwa5DZu4cPHw4ZMkRHR0dbW9vd3T09PT0sLGzXrl3Hjx/ncDgcDufSpUuEkPv373t4eGhoaBgaGk6dOvXDhw/04vS5Wnh4uJmZWadOnWr936YmCDAAYBXhCXJhOUk92SAry8vLO3PmzDfffKOhocGMNDU1HT9+/MGDB5m+ztetW2dnZ5ecnLx48WLpxZ8+fTp69OgRI0akpKRMmzaNybzK0tPTjx07durUqVOnTl2+fHn16tX0+KKiojlz5ty8efP8+fNcLnfkyJE1vhKTrsfe3v727dshISGzZ8+WTqDQ0NDhw4ffv3//66+/PnPmzIQJE2bNmiUUCrdt2xYbG7ty5UpCiEQiGTVqFI/Hu379+tatW+fPny93Ky9evKBD7sKFC8nJyV9//XVFRcW8efPGjBkzcOBAkUgkEolcXV2Li4sHDhyor69/8+bNQ4cOnTt3bubMmcxKzp8/n5qampCQcOrUqRr3q9aoliU/P58Qkp+f39SFAECVPn78KBQKP378WJeF9/pSoTrUvrENUsn169cJIUePHpUZT7+g69WrVxRFWVhYjBgxgpmUmZlJCLlz5w5FUfPnz7ezs2Mm0QH27t07iqJ27typq6tLjw8NDdXU1CwoKKAHv/vuO2dn58rFvH79mhBy//59ma3IsLCwGDhwIDPo6+s7aNAg+jMhJDg4mJnk7u6+atUqZnD37t0CgYCiqDNnzvB4vOzsbHr86dOnmYMgvd2QkBArK6uysjKZAiZPnjx8+HBmcPv27fr6+h8+fKAHf//9dy6Xm5OTQ89pYmJSWlpaeS9kyP2TqPH7HJ35AkCz9+E1ufYzkVQQQkj6eUIIeXKOnFlICCFcFeIyk7Rq3bAbpMOAfgUzIcTR0VHubGlpaT179mQGnZycqlqhpaWltrY2/VkgENBZRQhJT09fvHjx9evX3759S597ZWVl1dhCxMXFRfrzTz/9xAxKl5qcnHzz5k36rIsQIhaLS0pKiouLU1NT27Vr17Zt28prk5aSkuLu7q6qqlp9MampqV27dtXS0qIH3dzcJBJJWlqaiYkJIaRLly5qamrVr6HOEGAA0OwV5pDrkURcSjjcv99kKRGT6xGEkhCeOrH7vzoHWMeOHTkcjlAolGlW99dff+nr6xsZGdGDzLezDIqimJAj/8SeXNIxwOFwmOuEPj4+5ubmUVFRZmZmEonEzs6uDo0dpGuQLlUikSxdunTUqFHSM/P5fJk6pReXJn1ZtRoyB0FmnVUdugaBe2AA0OwJ7Mm0y8SoEyGEUOJ//9+oE5l2mQjs67xiQ0NDT0/PiIiIjx//bRWSk5Ozd+9eX1/fqr7ZGdbW1jdv3mQGb926Vaut5+bmpqamLlq0qH///jY2Nu/evVNwQfrKJ/PZ2tpa7mw9evRIS0vr+L+4XK6trW1WVtbLly/p2a5duyZ3cXt7+8TExMqtGdXU1MRiMTNoa2ubkpJSVFREDyYlJXG5XGU02agMAQYAbGBsQ/zPyo70P0uMbeq54i1btpSWlnp7e//xxx/Z2dnx8fGenp5t2rRhrrxVY9q0aX/99df8+fMfPXr066+/xsbGkqpPaCrT19c3NDTcvn37kydPLly4MGfOHAUXTEpKWrt27aNHj37++edDhw4FBQXJnW3JkiW//PJLWFjYw4cPU1NTDx48uGjRIkLIgAEDOnfuPGnSpLt37yYmJlbV9mTmzJkFBQVjx469devW48ePd+/enZaWRgixtLS8d+9eWlra27dvy8vLx48fz+fzJ0+e/ODBg4sXL3777bcTJ06krx8qGwIMAFji2VVCSbXQoyTkmfxTh1r57LPPbt261aFDB19f3w4dOkydOrVfv37Xrl0zMDCocVkrK6vDhw//9ttv9vb2kZGRdBKoq6sruGkul3vgwIHk5GQ7O7vZs2evW7dOwQXnzp2bnJzcvXv35cuXb9iwwdvbW+5s3t7ep06dSkhI6NmzZ69evX788UcLCwt6u0ePHi0tLXVycgoICKgqqg0NDS9cuPDhw4e+ffs6ODhERUXRF0KnTJnSuXNnR0fH1q1bJyUlaWpqnjlzJi8vr2fPnqNHj+7fv/+WLVsU3JF64lRz0ZaNCgoKdHV18/PzdXR0mroWAJCvpKQkMzPTysqKz+fXYrGj08jdA6SdC/FcTs4uItnXSdcvycitSiuz1lauXLl169bs7GylbsXS0jI4ODg4OFipW2lkcv8kavw+RyMOAGAJQXfS2oa4fku4PPJVHLm6majUJv+UIyIiomfPnoaGhklJSevWrZN+BAqUDQEGACzRa/q/n7k80ju4ySqR8vjx4xUrVuTl5bVr127u3LkhISFNXdEnBAEGAFB3Gzdu3LhxY2Nu8enTp425ueYMjTgAAICVEGAAAMBKCDAAAGAlBBgAALASAgwAAFgJAQYAAKyEAAMAaAD024eVtHKZFyUrG4fDOXbsWONvt7YQYADwSfviiy9kumU6duyY4h3y1kozz4PKzM3NRSJR9e8nU2pyVw8BBgDQ8lV+K4oieDyeqampikrDd3lRh9eeVYYAAwAWeFbwTJgrrPy/ZwXPlLrdsLCwbt26bdu2zdzcXFNT8z//+c/79+/pSWKxeM6cOXp6eoaGht9//710x+jx8fG9e/emJw0dOjQ9PZ0eb2VlRQjp3r07h8P54osv6JE7d+60sbHh8/nW1tYRERHMSm7cuNG9e3c+n+/o6Hjnzp2qKrS0tFy+fPm4ceNatWplZma2efNmZhKHw9m6devw4cO1tLRWrFhBCDl58qSDgwOfz2/fvv3SpUsrKiroOR8/ftynTx8+n29ra5uQkMCsQeaU8eHDh0OGDNHR0dHW1nZ3d09PTw8LC9u1a9fx48c5HA6Hw7l06RIh5P79+x4eHhoaGoaGhlOnTv3w4QO9OH2uFh4ebmZm1iAvDENXUgDQ3D0reDb06NCqpp4aecpCx0J5W3/y5Mmvv/568uTJgoICf3//wMDAvXv3EkI2bNgQExMTHR1ta2u7YcOGo0ePenh40IsUFRXNmTOnS5cuRUVFS5YsGTlyZEpKCpfLvXHjhpOT07lz5z7//HM1NTVCSFRUVGho6JYtW7p3737nzp0pU6ZoaWlNnjy5qKho6NChHh4ee/bsyczMrOqNX7R169b98MMPYWFhZ86cmT17trW1taenJz0pNDQ0PDx848aNPB7vzJkzEyZM+O9//0tnz9SpU+kZJBLJqFGjjIyMrl+/XlBQUFU/9y9evOjTp88XX3xx4cIFHR2dpKSkioqKefPmpaamFhQU7Ny5kxBiYGBQXFw8cODAXr163bx58/Xr1wEBATNnzqTflEYIOX/+vI6OTkJCQsO8CIVqWfLz8wkh+fn5dV5Dabl4xx/pi4/d3/FHemm5uAFrAwDax48fhULhx48fFZz/4duHdrF2Vf3v4duH9Smmb9++QUFB0mOOHj3KfDeGhobyeLzs7Gx68PTp01wuVyQSURQlEAhWr15Njy8vL2/btu3w4cMrr//169eEkPv371MUlZmZSQi5c+cOM9Xc3Hzfvn3M4PLly11cXCiK2rZtm4GBQVFRET0+MjJSZkGGhYXFwIEDmUFfX99BgwbRnwkhwcHBzCR3d/dVq1Yxg7t37xYIBBRFnTlzRmYfCSFHjx6VKTgkJMTKyqqsrEymgMmTJ0vv+Pbt2/X19T98+EAP/v7771wuNycnh57TxMSktLS08l7I/ZOo8fscZ2D/IzxOGJWYKfnnl8HKuNQp7lYhg22btCgAaErt2rVr27Yt/dnFxUUikaSlpWloaIhEIhcXF3q8ioqKo6Mj9c9ZRXp6+uLFi69fv/727VuJREIIycrKqtwU4s2bN9nZ2f7+/lOmTKHHVFRU6OrqEkJSU1O7du2qqanJbLeaCqWnuri4/PTTT8ygo6Mj8zk5OfnmzZvM6yvFYnFJSUlxcXFqaqrMPsrdSkpKiru7O/1Oy2rQlWtpadGDbm5u9BGj39HcpUsX+tSzQSDA/hUeJ9z2R6b0GAlF6DHIMICWSkdHh/6lz3j//n1Vb1CkWyfW2EbRx8fH3Nw8KirKzMxMIpHY2dnJbbNAZ1tUVJSzszMzksfjEUKoelxhky6PCRJ6c0uXLh01apT0zHw+X2ZbVe2dhoaGIlunKKryGpgx0vXUHxpx/K2sQhKVmCl3UlRiZlmFRO4kAGA7a2vrW7duSY+5efNm586dmcGsrKyXL1/Sn69du8blcjt16qSrqysQCK5fv06Pr6ioSE5Opj/n5uampqYuWrSof//+NjY27969Y1ZFn3yIxWJ60MTEpE2bNhkZGR2l0A09bG1t7969+/HjR3pOZkNySU+9fv26tbW13Nl69OiRlpbW8X9xuVxbW1uZfZS7uL29fWJiYuXWjGpqaswe0ZWnpKQUFRXRg0lJSfQRq6b+OkOA/W33taeSKn7xSCiy+9rTRq0GABrLN998k56eHhgYePfu3UePHv3888/R0dHfffcdMwOfz588efLdu3cTExNnzZo1ZswYU1NTQkhQUNDq1auPHj36119/ffPNN0zrRH19fUNDw+3btz958uTChQtz5sxhVmVsbKyhoREfH//q1Sv6tC8sLCw8PHzTpk2PHj26f//+zp07f/zxR0LIuHHjuFyuv7+/UCiMi4tbv359NbuQlJS0du1auvhDhw5V1eJjyZIlv/zyS1hY2MOHD1NTUw8ePLho0SJCyIABAzp37jxp0iR6HxcuXCh38ZkzZxYUFIwdO/bWrVuPHz/evXt3WloaIcTS0vLevXtpaWlv374tLy8fP348fcQePHhw8eLFb7/9duLEifT1wwaHAPvbs7ziOk8FAPaytLRMTExMT0/38vLq2bNnbGxsbGzsf/7zH2aGjh07jho1avDgwV5eXnZ2dkxL97lz506aNMnPz8/FxUVbW3vkyJH0eC6Xe+DAgeTkZDs7u9mzZ69bt45ZlYqKyn//+99t27aZmZkNHz6cEBIQELBjx47Y2NguXbr07ds3NjaWPgNr1arVyZMnhUJh9+7dFy5cuGbNmmp2Ye7cucnJyd27d1++fPmGDRu8vb3lzubt7X3q1KmEhISePXv26tXrxx9/tLCwoAs+evRoaWmpk5NTQEAAc5NMhqGh4YULFz58+NC3b18HB4eoqCj6ftiUKVM6d+7s6OjYunXrpKQkTU3NM2fO5OXl9ezZc/To0f3799+yZYvC/zVqh1OfK63NUEFBga6ubn5+flWXsKsSnZix/PfUqqYuHmLj796+3tUBACGElJSUZGZmWllZ8fl8ReYX5gp9T/lWNfXg0IO2hsq6Sx0WFnbs2LHm3HeGpaVlcHBwVW3f2ULun0SN3+c4A/vbRBdLbhX3ZbkcMtHFslGrAQApWqrV3fmvfiq0YGiF+Dc1Fe4UdyuZVoi0Ke5WaipIeoAmY6FjcWrkqaLyosqTtFS1lPoUMzRnuIT4P2SeA+NyCJ4DA2hwtb2ECC1eM72EGBERQdfk4OCQmJhYeQY/Pz/O//r888+Zqe/fvw8MDBQIBHw+38bGJi4uTqnVhgy2/Wv5oMVDbCa5WCweYvPX8kFILwCA5km5lxAPHjwYHBwcERHh5ua2bdu2QYMGCYXCdu3aSc+zadOm1atX058rKiq6du3KtP8pKyvz9PQ0NjY+fPhw27Zts7OztbW1lVowIURNhYv2GgAAzZ9yA+zHH3/09/cPCAgghPz0009nzpyJjIwMDw+XnkdXV5fuOoUQcuzYsXfv3n311Vf0YExMTF5e3tWrV+nGmnSLTwBoGVrY/Quoj7r9MSjxEmJZWVlycrKXlxczxsvL6+rVq9UsEh0dPWDAACaoTpw44eLiEhgYaGJiYmdnt2rVKunnvRmlpaUFUhp2LwCgwdE/SYuL8Xgl/I3uaovuRktxSjwDe/v2rVgsln4A28TEJCcnp6r5RSLR6dOn9+3bx4zJyMi4cOHC+PHj4+LiHj9+HBgYWFFRsWTJEpkFw8PDly5d2uD1A4CS8Hg8PT09upt2TU1NJb3+GNhCIpG8efNGU1Oztm/OVHozeuk/TbmdPDJiY2P19PSkX00tkUiMjY23b9/O4/EcHBxevny5bt26ygEWEhLCdNZSUFBgbm7ekDsAAEpA98ZEZxgAl8tt165dbX/KKDHAjIyMeDye9CnX69evq+oRi6KomJiYiRMnSve0LxAIVFVVmZNKGxubnJycsrIymd741dXV1dXVlbAHAKAsHA5HIBAYGxvX7VX30MKoqalxubW+paXEAFNTU3NwcEhISGC6CEtISKC7/6rs8uXLT5488ff3lx7p5ua2b98+iURC79ijR48EAkEDvksGAJoWj8er7W0PAIZynwObM2fOjh07YmJiUlNTZ8+enZWVNX36dEJISEjIpEmTpOeMjo52dnaWeeHbjBkzcnNzg4KCHj169Pvvv69atSowMFCpBQMAAFso9x6Yr69vbm7usmXLRCKRnZ1dXFwc3cJQJBJlZWUxs+Xn5x85cmTTpk0yi5ubm589e3b27Nn29vZt2rQJCgqaP3++UgsGAAC2QFdSAADQHNX4fd7SOvOl8xhPgwEAsB39TV7NWVZLC7DCwkJCCFrSAwC0DIWFhUxvTTJa2iVEiUTy8uVLbW3tuj0aST9Glp2djSuQOBQMHAoGDgUDh4KhvENBUVRhYaGZmVlVLexb2hkYl8tt27ZtPVeio6ODP0oaDgUDh4KBQ8HAoWAo6VBUde5Fw3saAQCAlRBgAADASgiw/6Gurh4aGoqOqQgOhRQcCgYOBQOHgtGEh6KlNeIAAIBPBM7AAACAlRBgAADASggwAABgJQQYAACw0icUYBEREVZWVnw+38HBITExUe48ly9fdnBw4PP57du337p1q/SkI0eO2Nraqqur29raHj16tFFKVpb6HIqoqCh3d3d9fX19ff0BAwbcuHGjsapWinr+VdAOHDjA4XCkXybORvU8FO/fvw8MDBQIBHw+38bGJi4urlGqVop6Hoqffvqpc+fOGhoa5ubms2fPLikpaZSqlaLGQyESicaNG9e5c2culxscHCwzVelfm9Sn4cCBA6qqqlFRUUKhMCgoSEtL69mzZzLzZGRkaGpqBgUFCYXCqKgoVVXVw4cP05OuXr3K4/FWrVqVmpq6atUqFRWV69evN/pONIx6Hopx48b9/PPPd+7cSU1N/eqrr3R1dZ8/f97oO9Ew6nkoaE+fPm3Tpo27u/vw4cMbr/SGVs9DUVpa6ujoOHjw4CtXrjx9+jQxMTElJaXRd6Jh1PNQ7NmzR11dfe/evZmZmWfOnBEIBMHBwY2+Ew1DkUORmZk5a9asXbt2devWLSgoSHpSI3xtfioB5uTkNH36dGbQ2tp6wYIFMvN8//331tbWzOC0adN69epFfx4zZszAgQOZSd7e3mPHjlVmvUpUz0MhraKiQltbe9euXUoqVdnqfygqKirc3Nx27NgxefJkVgdYPQ9FZGRk+/bty8rKGqFUZavnoQgMDPTw8GAmzZkzp3fv3sqsV4kUORSMvn37ygRYI3xtfhKXEMvKypKTk728vJgxXl5eV69elZnt2rVr0vN4e3vfunWrvLxc7qTKi7NC/Q+FtOLi4vLycgMDA+UVrDwNciiWLVvWunVrf3//RihYeep/KE6cOOHi4hIYGGhiYmJnZ7dq1SqxWNw4xTes+h+K3r17Jycn05fWMzIy4uLihgwZ0ii1NzAFD0U1GuFrs6V15ivX27dvxWKxiYkJM8bExCQnJ0dmtpycHJl5Kioq3r59KxAIKk+qvDgr1P9QSM+2YMGCNm3aDBgwQKk1K0n9D0VSUlJ0dHRKSkrjFKw89T8UGRkZFy5cGD9+fFxc3OPHjwMDAysqKpYsWdJIO9Bw6n8oxo4d++bNG/qsq6KiYsaMGQsWLGik6huUgoeiGo3wtflJBBhN+gUrFEXJfd+KzDzSYxRZnC3qeShoa9eu3b9//6VLl/h8vtIqVbo6H4rCwsIJEyZERUUZGRk1Qp2NoD5/FRKJxNjYePv27Twez8HB4eXLl+vWrWNjgNHqcyguXbq0cuXKiIgIZ2fnJ0+eBAUFCQSCxYsXK79qpajn956yvzY/iQAzMjLi8XjS4f/69WvpnwY0U1NTmXlUVFQMDQ3lTqq8OCvU/1DQ1q9fv2rVqnPnztnb2yu7ZiWp56F4+PDh06dPfXx86PESiYQQoqKikpaW1qFDB+WX35Dq/1chEAhUVVV5PB49ycbGJicnp6ysTE1NTfnlN6T6H4rFixdPnDgxICCAENKlS5eioqKpU6cuXLiwqjdaNVsKHopqNMLXJsuOad2oqak5ODgkJCQwYxISElxdXWVmc3FxkZ7n7Nmzjo6OqqqqcidVXpwV6n8oCCHr1q1bvnx5fHy8o6NjI9SsJPU8FNbW1vfv30/5x7Bhw/r165eSksLGt4HX/6/Czc3tyZMndIoTQh49eiQQCFiXXqQhDkVxcbF0VvF4PLq5gZILb3gKHopqNMbXZsO2CWm26Pag0dHRQqEwODhYS0vr6dOnFEUtWLBg4sSJ9Dx009jZs2cLhcLo6GjpprFJSUk8Hm/16tWpqamrV69uAc3o63wo1qxZo6amdvjwYdE/CgsLm2xn6qeeh0Ia21sh1vNQZGVltWrVaubMmWlpaadOnTI2Nl6xYkWT7Uz91PNQhIaGamtr79+/PyMj4+zZsx06dBgzZkyT7Uz9KHIoKIq6c+fOnTt3HBwcxo0bd+fOnYcPH9LjG+Fr81MJMIqifv75ZwsLCzU1tR49ely+fJkeOXny5L59+zLzXLp0qXv37mpqapaWlpGRkdKLHzp0qHPnzvRP7yNHjjRm5Q2uPofCwsJC5jdQaGho45bfkOr5V8Fge4BR9T4UV69edXZ2VldXb9++/cqVKysqKhqz+IZVn0NRXl4eFhbWoUMHPp9vbm7+zTffvHv3rnHLb0iKHAqZLwQLCwtmkrK/NvE6FQAAYKVP4h4YAAC0PAgwAABgJQQYAACwEgIMAABYCQEGAACshAADAABWQoABAAArIcAAWoLY2Fg9PT25k/z8/Br5bdFhYWHdunVrzC3CpwkBBp8QPz8/DofD4XBUVVVNTEw8PT1jYmKYHvxaqk2bNsXGxio+v/RRat++/bx584qKimq1xXnz5p0/f55ZWyPHJ3w6EGDwaRk4cKBIJHr69Onp06f79esXFBQ0dOjQioqKpq7rb2VlZQ2+Tl1d3apOzqpCH6WMjIwVK1ZERETMmzdPwQUpiqqoqGjVqpX0uwsAlAQBBp8WdXV1U1PTNm3a9OjR44cffjh+/Pjp06eZE5T8/PypU6caGxvr6Oh4eHjcvXuXHk9fE4uJiWnXrl2rVq1mzJghFovXrl1rampqbGy8cuVKZv1ZWVnDhw9v1aqVjo7OmDFjXr16xUxasWKFsbGxtrZ2QEDAggULmIts9DlKeHi4mZlZp06dCCF79uxxdHTU1tY2NTUdN27c69ev6TkvXbrE4XB+//33rl278vl8Z2fn+/fvS+/dmTNnbGxsWrVqRSeQ9PrpzxKJZM2aNR07dlRXV2/Xrp105ZWPkrm5+bhx48aPH3/s2LEaqzpz5oyjo6O6unpiYiJzCTEsLGzXrl3Hjx+nT+kuXbrk4eExc+ZMZkO5ubnq6uoXLlyozX9DgL8hwOCT5uHh0bVr199++40QQlHUkCFDcnJy4uLikpOTe/To0b9//7y8PHrO9PT006dPx8fH79+/PyYmZsiQIc+fP798+fKaNWsWLVp0/fp1eg0jRozIy8u7fPlyQkJCenq6r68vvfjevXtXrly5Zs2a5OTkdu3aRUZGSpdx/vz51NTUhISEU6dOEULKysqWL19+9+7dY8eOZWZm+vn5Sc/83XffrV+//ubNm8bGxsOGDaPfZE8IKS4uXr9+/e7du//444+srCy5p00hISFr1qxZvHixUCjct2+fIu9n0tDQoDdRfVXff/99eHh4amqq9Cvi5s2bN2bMGDpNRSKRq6trQEDAvn37SktLmcNiZmbWr1+/GssAkKPBuwcGaLbk9hnv6+trY2NDUdT58+d1dHRKSkqYSR06dNi2bRtFUaGhoZqamgUFBfR4b29vS0tLsVhMD3bu3Dk8PJyiqLNnz/J4vKysLHr8w4cPCSE3btygKMrZ2TkwMJBZs5ubW9euXZmqTExMSktL5dZ848YNQgj9zpqLFy8SQg4cOEBPys3N1dDQOHjwIEVRO3fuJIQ8efKEnvTzzz+bmJjI7HVBQYG6unpUVJTiR+nPP/80NDSs/EKQylUdO3aMmRoaGiq9d9LHvKSkxMDAgK6Zoqhu3bqFhYVVXw9AVXAGBp866p83nScnJ3/48MHQ0LDVPzIzM9PT0+nZLC0ttbW16c8mJia2trbMewtNTEzo62mpqanm5ubMOy1tbW319PRSU1MJIWlpaU5OTsxGpT8TQrp06SL9+sc7d+4MHz7cwsJCW1v7iy++IIRkZWUxU11cXOgPBgYGnTt3ptdPCNHU1GReBi0QCJhLfIzU1NTS0tL+/fvXeExOnTrVqlUrPp/v4uLSp0+fzZs311iVgm83VVdXnzBhQkxMDCEkJSXl7t27MmdyAIpTaeoCAJpYamqqlZUVIUQikQgEgkuXLklPZZo/MC+kJoTQLfSkB+mmjEwWMqTHSE+i/vc1RlpaWsznoqIiLy8vLy+vPXv2tG7dOisry9vbu5rGHcxqZUqiKr0pSUNDo6qVyOjXr19kZKSqqqqZmRm92hqrkt6F6gUEBHTr1u358+cxMTH9+/ev/IY5AAXhDAw+aRcuXLh///7//d//EUJ69OiRk5OjoqLSUYqRkZHia7O1tc3KysrOzqYHhUJhfn6+jY0NIaRz5870ZTfarVu3qlrJX3/99fbt29WrV7u7u1tbW1c+kaLvtxFC3r179+jRI2trawXL++yzzzQ0NJgG7tXQ0tLq2LGjhYUFE4o1VlUVNTU1sVgsPaZLly6Ojo5RUVH79u37+uuvFVwPQGUIMPi0lJaW5uTkvHjx4vbt26tWrRo+fPjQoUMnTZpECBkwYICLi8uIESPOnDnz9OnTq1evLlq0qJqkqWzAgAH29vbjx4+/ffv2jRs3Jk2a1LdvX/ra2rfffhsdHb1r167Hjx+vWLHi3r17MudqjHbt2qmpqW3evDkjI+PEiRPLly+XmWHZsmXnz59/8OCBn5+fkZGR4k9Z8fn8+fPnf//997/88kt6evr169ejo6MVXLbGqqpiaWl57969tLS0t2/fMu1NAgICVq9eLRaLR44cqeB6ACpDgMGnJT4+XiAQWFpaDhw48OLFi//973+PHz/O4/EIIRwOJy4urk+fPl9//XWnTp3Gjh379OlTRdrpMTgczrFjx/T19fv06TNgwID27dsfPHiQnjR+/PiQkJB58+b16NGDbsLH5/PlrqR169axsbGHDh2ytbVdvXr1+vXrZWZYvXp1UFCQg4ODSCQ6ceKE9M2zGi1evHju3LlLliyxsbHx9fVV/ESqxqqqMmXKlM6dOzs6OrZu3TopKYke+eWXX6qoqIwbN66qgwCgCDkXygFA2Tw9PU1NTXfv3l2rpS5dutSvX793797V9sHk5iY7O9vS0vLmzZs9evRo6lqAxdCIA6AxFBcXb9261dvbm8fj7d+//9y5cwkJCU1dVBMoLy8XiUQLFizo1asX0gvqCQEG0Bjo65MrVqwoLS3t3LnzkSNHBgwY0NRFNYGkpKR+/fp16tTp8OHDTV0LsB4uIQIAACuhEQcAALASAgwAAFgJAQYAAKyEAAMAAFZCgAEAACshwAAAgJUQYAAAwEoIMAAAYCUEGAAAsNL/AzK1N6E9FxcGAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#and on the test set\n",
    "fpredictor.plot_frontier(test_data) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:20:43.133407Z",
     "iopub.status.busy": "2024-06-17T19:20:43.133279Z",
     "iopub.status.idle": "2024-06-17T19:20:43.491964Z",
     "shell.execute_reply": "2024-06-17T19:20:43.491628Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Balanced Accuracy</th>\n",
       "      <th>F1 score</th>\n",
       "      <th>MCC</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>Positive Count</th>\n",
       "      <th>Negative Count</th>\n",
       "      <th>Positive Label Rate</th>\n",
       "      <th>Positive Prediction Rate</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Groups</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"7\" valign=\"top\">original</th>\n",
       "      <th>Overall</th>\n",
       "      <td>0.868973</td>\n",
       "      <td>0.776058</td>\n",
       "      <td>0.684574</td>\n",
       "      <td>0.613887</td>\n",
       "      <td>0.798276</td>\n",
       "      <td>0.599223</td>\n",
       "      <td>0.919433</td>\n",
       "      <td>2318.0</td>\n",
       "      <td>7451.0</td>\n",
       "      <td>0.237281</td>\n",
       "      <td>0.178114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Amer-Indian-Eskimo</th>\n",
       "      <td>0.934959</td>\n",
       "      <td>0.636364</td>\n",
       "      <td>0.428571</td>\n",
       "      <td>0.504525</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.272727</td>\n",
       "      <td>0.928571</td>\n",
       "      <td>11.0</td>\n",
       "      <td>112.0</td>\n",
       "      <td>0.089431</td>\n",
       "      <td>0.024390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Asian-Pac-Islander</th>\n",
       "      <td>0.859756</td>\n",
       "      <td>0.764319</td>\n",
       "      <td>0.661765</td>\n",
       "      <td>0.583363</td>\n",
       "      <td>0.762712</td>\n",
       "      <td>0.584416</td>\n",
       "      <td>0.900864</td>\n",
       "      <td>77.0</td>\n",
       "      <td>251.0</td>\n",
       "      <td>0.234756</td>\n",
       "      <td>0.179878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Black</th>\n",
       "      <td>0.928171</td>\n",
       "      <td>0.737251</td>\n",
       "      <td>0.619048</td>\n",
       "      <td>0.610779</td>\n",
       "      <td>0.852459</td>\n",
       "      <td>0.485981</td>\n",
       "      <td>0.940647</td>\n",
       "      <td>107.0</td>\n",
       "      <td>784.0</td>\n",
       "      <td>0.120090</td>\n",
       "      <td>0.068462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Other</th>\n",
       "      <td>0.913580</td>\n",
       "      <td>0.735915</td>\n",
       "      <td>0.588235</td>\n",
       "      <td>0.552396</td>\n",
       "      <td>0.714286</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.863380</td>\n",
       "      <td>10.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>0.123457</td>\n",
       "      <td>0.086420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>White</th>\n",
       "      <td>0.861610</td>\n",
       "      <td>0.777682</td>\n",
       "      <td>0.689766</td>\n",
       "      <td>0.612016</td>\n",
       "      <td>0.797516</td>\n",
       "      <td>0.607667</td>\n",
       "      <td>0.916398</td>\n",
       "      <td>2113.0</td>\n",
       "      <td>6233.0</td>\n",
       "      <td>0.253175</td>\n",
       "      <td>0.192907</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Maximum difference</th>\n",
       "      <td>0.075203</td>\n",
       "      <td>0.141319</td>\n",
       "      <td>0.261195</td>\n",
       "      <td>0.107491</td>\n",
       "      <td>0.285714</td>\n",
       "      <td>0.334940</td>\n",
       "      <td>0.077267</td>\n",
       "      <td>2103.0</td>\n",
       "      <td>6162.0</td>\n",
       "      <td>0.163744</td>\n",
       "      <td>0.168517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"7\" valign=\"top\">updated</th>\n",
       "      <th>Overall</th>\n",
       "      <td>0.864981</td>\n",
       "      <td>0.780276</td>\n",
       "      <td>0.685128</td>\n",
       "      <td>0.606017</td>\n",
       "      <td>0.766970</td>\n",
       "      <td>0.619068</td>\n",
       "      <td>0.902293</td>\n",
       "      <td>2318.0</td>\n",
       "      <td>7451.0</td>\n",
       "      <td>0.237281</td>\n",
       "      <td>0.191524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Amer-Indian-Eskimo</th>\n",
       "      <td>0.878049</td>\n",
       "      <td>0.769075</td>\n",
       "      <td>0.482759</td>\n",
       "      <td>0.434487</td>\n",
       "      <td>0.388889</td>\n",
       "      <td>0.636364</td>\n",
       "      <td>0.928571</td>\n",
       "      <td>11.0</td>\n",
       "      <td>112.0</td>\n",
       "      <td>0.089431</td>\n",
       "      <td>0.146341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Asian-Pac-Islander</th>\n",
       "      <td>0.850610</td>\n",
       "      <td>0.713329</td>\n",
       "      <td>0.588235</td>\n",
       "      <td>0.541195</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>0.454545</td>\n",
       "      <td>0.900864</td>\n",
       "      <td>77.0</td>\n",
       "      <td>251.0</td>\n",
       "      <td>0.234756</td>\n",
       "      <td>0.128049</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Black</th>\n",
       "      <td>0.897868</td>\n",
       "      <td>0.824945</td>\n",
       "      <td>0.631579</td>\n",
       "      <td>0.580505</td>\n",
       "      <td>0.557143</td>\n",
       "      <td>0.728972</td>\n",
       "      <td>0.940647</td>\n",
       "      <td>107.0</td>\n",
       "      <td>784.0</td>\n",
       "      <td>0.120090</td>\n",
       "      <td>0.157127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Other</th>\n",
       "      <td>0.864198</td>\n",
       "      <td>0.836620</td>\n",
       "      <td>0.592593</td>\n",
       "      <td>0.543856</td>\n",
       "      <td>0.470588</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.863380</td>\n",
       "      <td>10.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>0.123457</td>\n",
       "      <td>0.209877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>White</th>\n",
       "      <td>0.861850</td>\n",
       "      <td>0.781440</td>\n",
       "      <td>0.693921</td>\n",
       "      <td>0.614000</td>\n",
       "      <td>0.790206</td>\n",
       "      <td>0.618552</td>\n",
       "      <td>0.916398</td>\n",
       "      <td>2113.0</td>\n",
       "      <td>6233.0</td>\n",
       "      <td>0.253175</td>\n",
       "      <td>0.198179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Maximum difference</th>\n",
       "      <td>0.047258</td>\n",
       "      <td>0.123291</td>\n",
       "      <td>0.211162</td>\n",
       "      <td>0.179513</td>\n",
       "      <td>0.444444</td>\n",
       "      <td>0.345455</td>\n",
       "      <td>0.077267</td>\n",
       "      <td>2103.0</td>\n",
       "      <td>6162.0</td>\n",
       "      <td>0.163744</td>\n",
       "      <td>0.081828</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              Accuracy  Balanced Accuracy  F1 score       MCC  \\\n",
       "         Groups                                                                 \n",
       "original Overall              0.868973           0.776058  0.684574  0.613887   \n",
       "          Amer-Indian-Eskimo  0.934959           0.636364  0.428571  0.504525   \n",
       "          Asian-Pac-Islander  0.859756           0.764319  0.661765  0.583363   \n",
       "          Black               0.928171           0.737251  0.619048  0.610779   \n",
       "          Other               0.913580           0.735915  0.588235  0.552396   \n",
       "          White               0.861610           0.777682  0.689766  0.612016   \n",
       "         Maximum difference   0.075203           0.141319  0.261195  0.107491   \n",
       "updated  Overall              0.864981           0.780276  0.685128  0.606017   \n",
       "          Amer-Indian-Eskimo  0.878049           0.769075  0.482759  0.434487   \n",
       "          Asian-Pac-Islander  0.850610           0.713329  0.588235  0.541195   \n",
       "          Black               0.897868           0.824945  0.631579  0.580505   \n",
       "          Other               0.864198           0.836620  0.592593  0.543856   \n",
       "          White               0.861850           0.781440  0.693921  0.614000   \n",
       "         Maximum difference   0.047258           0.123291  0.211162  0.179513   \n",
       "\n",
       "                              Precision    Recall   ROC AUC  Positive Count  \\\n",
       "         Groups                                                               \n",
       "original Overall               0.798276  0.599223  0.919433          2318.0   \n",
       "          Amer-Indian-Eskimo   1.000000  0.272727  0.928571            11.0   \n",
       "          Asian-Pac-Islander   0.762712  0.584416  0.900864            77.0   \n",
       "          Black                0.852459  0.485981  0.940647           107.0   \n",
       "          Other                0.714286  0.500000  0.863380            10.0   \n",
       "          White                0.797516  0.607667  0.916398          2113.0   \n",
       "         Maximum difference    0.285714  0.334940  0.077267          2103.0   \n",
       "updated  Overall               0.766970  0.619068  0.902293          2318.0   \n",
       "          Amer-Indian-Eskimo   0.388889  0.636364  0.928571            11.0   \n",
       "          Asian-Pac-Islander   0.833333  0.454545  0.900864            77.0   \n",
       "          Black                0.557143  0.728972  0.940647           107.0   \n",
       "          Other                0.470588  0.800000  0.863380            10.0   \n",
       "          White                0.790206  0.618552  0.916398          2113.0   \n",
       "         Maximum difference    0.444444  0.345455  0.077267          2103.0   \n",
       "\n",
       "                              Negative Count  Positive Label Rate  \\\n",
       "         Groups                                                     \n",
       "original Overall                      7451.0             0.237281   \n",
       "          Amer-Indian-Eskimo           112.0             0.089431   \n",
       "          Asian-Pac-Islander           251.0             0.234756   \n",
       "          Black                        784.0             0.120090   \n",
       "          Other                         71.0             0.123457   \n",
       "          White                       6233.0             0.253175   \n",
       "         Maximum difference           6162.0             0.163744   \n",
       "updated  Overall                      7451.0             0.237281   \n",
       "          Amer-Indian-Eskimo           112.0             0.089431   \n",
       "          Asian-Pac-Islander           251.0             0.234756   \n",
       "          Black                        784.0             0.120090   \n",
       "          Other                         71.0             0.123457   \n",
       "          White                       6233.0             0.253175   \n",
       "         Maximum difference           6162.0             0.163744   \n",
       "\n",
       "                              Positive Prediction Rate  \n",
       "         Groups                                         \n",
       "original Overall                              0.178114  \n",
       "          Amer-Indian-Eskimo                  0.024390  \n",
       "          Asian-Pac-Islander                  0.179878  \n",
       "          Black                               0.068462  \n",
       "          Other                               0.086420  \n",
       "          White                               0.192907  \n",
       "         Maximum difference                   0.168517  \n",
       "updated  Overall                              0.191524  \n",
       "          Amer-Indian-Eskimo                  0.146341  \n",
       "          Asian-Pac-Islander                  0.128049  \n",
       "          Black                               0.157127  \n",
       "          Other                               0.209877  \n",
       "          White                               0.198179  \n",
       "         Maximum difference                   0.081828  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#We find that the demographic parity is very different on test data for a range of solutions found.\n",
    "#By looking at the per group decomposition, we can find out why.\n",
    "fpredictor.evaluate_groups(test_data, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:20:43.494315Z",
     "iopub.status.busy": "2024-06-17T19:20:43.494162Z",
     "iopub.status.idle": "2024-06-17T19:20:44.705811Z",
     "shell.execute_reply": "2024-06-17T19:20:44.705529Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Balanced Accuracy</th>\n",
       "      <th>F1 score</th>\n",
       "      <th>MCC</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>Positive Count</th>\n",
       "      <th>Negative Count</th>\n",
       "      <th>Positive Label Rate</th>\n",
       "      <th>Positive Prediction Rate</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Groups</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"7\" valign=\"top\">original</th>\n",
       "      <th>Overall</th>\n",
       "      <td>0.888951</td>\n",
       "      <td>0.812828</td>\n",
       "      <td>0.742171</td>\n",
       "      <td>0.679654</td>\n",
       "      <td>0.837131</td>\n",
       "      <td>0.666560</td>\n",
       "      <td>0.951884</td>\n",
       "      <td>9369.0</td>\n",
       "      <td>29704.0</td>\n",
       "      <td>0.239782</td>\n",
       "      <td>0.190925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Amer-Indian-Eskimo</th>\n",
       "      <td>0.936599</td>\n",
       "      <td>0.798567</td>\n",
       "      <td>0.710526</td>\n",
       "      <td>0.686736</td>\n",
       "      <td>0.843750</td>\n",
       "      <td>0.613636</td>\n",
       "      <td>0.957846</td>\n",
       "      <td>44.0</td>\n",
       "      <td>303.0</td>\n",
       "      <td>0.126801</td>\n",
       "      <td>0.092219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Asian-Pac-Islander</th>\n",
       "      <td>0.874895</td>\n",
       "      <td>0.827344</td>\n",
       "      <td>0.762360</td>\n",
       "      <td>0.680040</td>\n",
       "      <td>0.810169</td>\n",
       "      <td>0.719880</td>\n",
       "      <td>0.947550</td>\n",
       "      <td>332.0</td>\n",
       "      <td>859.0</td>\n",
       "      <td>0.278757</td>\n",
       "      <td>0.247691</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Black</th>\n",
       "      <td>0.935424</td>\n",
       "      <td>0.781025</td>\n",
       "      <td>0.683871</td>\n",
       "      <td>0.663316</td>\n",
       "      <td>0.838608</td>\n",
       "      <td>0.577342</td>\n",
       "      <td>0.973068</td>\n",
       "      <td>459.0</td>\n",
       "      <td>3335.0</td>\n",
       "      <td>0.120980</td>\n",
       "      <td>0.083289</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Other</th>\n",
       "      <td>0.926154</td>\n",
       "      <td>0.742982</td>\n",
       "      <td>0.625000</td>\n",
       "      <td>0.610476</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.980877</td>\n",
       "      <td>40.0</td>\n",
       "      <td>285.0</td>\n",
       "      <td>0.123077</td>\n",
       "      <td>0.073846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>White</th>\n",
       "      <td>0.883319</td>\n",
       "      <td>0.813129</td>\n",
       "      <td>0.744947</td>\n",
       "      <td>0.677550</td>\n",
       "      <td>0.838216</td>\n",
       "      <td>0.670356</td>\n",
       "      <td>0.948752</td>\n",
       "      <td>8494.0</td>\n",
       "      <td>24922.0</td>\n",
       "      <td>0.254190</td>\n",
       "      <td>0.203286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Maximum difference</th>\n",
       "      <td>0.061704</td>\n",
       "      <td>0.084361</td>\n",
       "      <td>0.137360</td>\n",
       "      <td>0.076260</td>\n",
       "      <td>0.033581</td>\n",
       "      <td>0.219880</td>\n",
       "      <td>0.033327</td>\n",
       "      <td>8454.0</td>\n",
       "      <td>24637.0</td>\n",
       "      <td>0.157777</td>\n",
       "      <td>0.173845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"7\" valign=\"top\">updated</th>\n",
       "      <th>Overall</th>\n",
       "      <td>0.888260</td>\n",
       "      <td>0.820083</td>\n",
       "      <td>0.747309</td>\n",
       "      <td>0.680235</td>\n",
       "      <td>0.816285</td>\n",
       "      <td>0.689081</td>\n",
       "      <td>0.938464</td>\n",
       "      <td>9369.0</td>\n",
       "      <td>29704.0</td>\n",
       "      <td>0.239782</td>\n",
       "      <td>0.202416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Amer-Indian-Eskimo</th>\n",
       "      <td>0.907781</td>\n",
       "      <td>0.879200</td>\n",
       "      <td>0.698113</td>\n",
       "      <td>0.658761</td>\n",
       "      <td>0.596774</td>\n",
       "      <td>0.840909</td>\n",
       "      <td>0.957846</td>\n",
       "      <td>44.0</td>\n",
       "      <td>303.0</td>\n",
       "      <td>0.126801</td>\n",
       "      <td>0.178674</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Asian-Pac-Islander</th>\n",
       "      <td>0.865659</td>\n",
       "      <td>0.777515</td>\n",
       "      <td>0.705882</td>\n",
       "      <td>0.650614</td>\n",
       "      <td>0.905660</td>\n",
       "      <td>0.578313</td>\n",
       "      <td>0.947550</td>\n",
       "      <td>332.0</td>\n",
       "      <td>859.0</td>\n",
       "      <td>0.278757</td>\n",
       "      <td>0.178002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Black</th>\n",
       "      <td>0.927517</td>\n",
       "      <td>0.898649</td>\n",
       "      <td>0.741784</td>\n",
       "      <td>0.709708</td>\n",
       "      <td>0.651815</td>\n",
       "      <td>0.860566</td>\n",
       "      <td>0.973068</td>\n",
       "      <td>459.0</td>\n",
       "      <td>3335.0</td>\n",
       "      <td>0.120980</td>\n",
       "      <td>0.159726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Other</th>\n",
       "      <td>0.938462</td>\n",
       "      <td>0.954167</td>\n",
       "      <td>0.795918</td>\n",
       "      <td>0.779342</td>\n",
       "      <td>0.672414</td>\n",
       "      <td>0.975000</td>\n",
       "      <td>0.980877</td>\n",
       "      <td>40.0</td>\n",
       "      <td>285.0</td>\n",
       "      <td>0.123077</td>\n",
       "      <td>0.178462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>White</th>\n",
       "      <td>0.883918</td>\n",
       "      <td>0.817372</td>\n",
       "      <td>0.749176</td>\n",
       "      <td>0.680184</td>\n",
       "      <td>0.831014</td>\n",
       "      <td>0.682011</td>\n",
       "      <td>0.948752</td>\n",
       "      <td>8494.0</td>\n",
       "      <td>24922.0</td>\n",
       "      <td>0.254190</td>\n",
       "      <td>0.208613</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Maximum difference</th>\n",
       "      <td>0.072802</td>\n",
       "      <td>0.176651</td>\n",
       "      <td>0.097805</td>\n",
       "      <td>0.128728</td>\n",
       "      <td>0.308886</td>\n",
       "      <td>0.396687</td>\n",
       "      <td>0.033327</td>\n",
       "      <td>8454.0</td>\n",
       "      <td>24637.0</td>\n",
       "      <td>0.157777</td>\n",
       "      <td>0.048887</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              Accuracy  Balanced Accuracy  F1 score       MCC  \\\n",
       "         Groups                                                                 \n",
       "original Overall              0.888951           0.812828  0.742171  0.679654   \n",
       "          Amer-Indian-Eskimo  0.936599           0.798567  0.710526  0.686736   \n",
       "          Asian-Pac-Islander  0.874895           0.827344  0.762360  0.680040   \n",
       "          Black               0.935424           0.781025  0.683871  0.663316   \n",
       "          Other               0.926154           0.742982  0.625000  0.610476   \n",
       "          White               0.883319           0.813129  0.744947  0.677550   \n",
       "         Maximum difference   0.061704           0.084361  0.137360  0.076260   \n",
       "updated  Overall              0.888260           0.820083  0.747309  0.680235   \n",
       "          Amer-Indian-Eskimo  0.907781           0.879200  0.698113  0.658761   \n",
       "          Asian-Pac-Islander  0.865659           0.777515  0.705882  0.650614   \n",
       "          Black               0.927517           0.898649  0.741784  0.709708   \n",
       "          Other               0.938462           0.954167  0.795918  0.779342   \n",
       "          White               0.883918           0.817372  0.749176  0.680184   \n",
       "         Maximum difference   0.072802           0.176651  0.097805  0.128728   \n",
       "\n",
       "                              Precision    Recall   ROC AUC  Positive Count  \\\n",
       "         Groups                                                               \n",
       "original Overall               0.837131  0.666560  0.951884          9369.0   \n",
       "          Amer-Indian-Eskimo   0.843750  0.613636  0.957846            44.0   \n",
       "          Asian-Pac-Islander   0.810169  0.719880  0.947550           332.0   \n",
       "          Black                0.838608  0.577342  0.973068           459.0   \n",
       "          Other                0.833333  0.500000  0.980877            40.0   \n",
       "          White                0.838216  0.670356  0.948752          8494.0   \n",
       "         Maximum difference    0.033581  0.219880  0.033327          8454.0   \n",
       "updated  Overall               0.816285  0.689081  0.938464          9369.0   \n",
       "          Amer-Indian-Eskimo   0.596774  0.840909  0.957846            44.0   \n",
       "          Asian-Pac-Islander   0.905660  0.578313  0.947550           332.0   \n",
       "          Black                0.651815  0.860566  0.973068           459.0   \n",
       "          Other                0.672414  0.975000  0.980877            40.0   \n",
       "          White                0.831014  0.682011  0.948752          8494.0   \n",
       "         Maximum difference    0.308886  0.396687  0.033327          8454.0   \n",
       "\n",
       "                              Negative Count  Positive Label Rate  \\\n",
       "         Groups                                                     \n",
       "original Overall                     29704.0             0.239782   \n",
       "          Amer-Indian-Eskimo           303.0             0.126801   \n",
       "          Asian-Pac-Islander           859.0             0.278757   \n",
       "          Black                       3335.0             0.120980   \n",
       "          Other                        285.0             0.123077   \n",
       "          White                      24922.0             0.254190   \n",
       "         Maximum difference          24637.0             0.157777   \n",
       "updated  Overall                     29704.0             0.239782   \n",
       "          Amer-Indian-Eskimo           303.0             0.126801   \n",
       "          Asian-Pac-Islander           859.0             0.278757   \n",
       "          Black                       3335.0             0.120980   \n",
       "          Other                        285.0             0.123077   \n",
       "          White                      24922.0             0.254190   \n",
       "         Maximum difference          24637.0             0.157777   \n",
       "\n",
       "                              Positive Prediction Rate  \n",
       "         Groups                                         \n",
       "original Overall                              0.190925  \n",
       "          Amer-Indian-Eskimo                  0.092219  \n",
       "          Asian-Pac-Islander                  0.247691  \n",
       "          Black                               0.083289  \n",
       "          Other                               0.073846  \n",
       "          White                               0.203286  \n",
       "         Maximum difference                   0.173845  \n",
       "updated  Overall                              0.202416  \n",
       "          Amer-Indian-Eskimo                  0.178674  \n",
       "          Asian-Pac-Islander                  0.178002  \n",
       "          Black                               0.159726  \n",
       "          Other                               0.178462  \n",
       "          White                               0.208613  \n",
       "         Maximum difference                   0.048887  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#The groups with the smallest Positive Prediction Rate (American-Indian-Eskimo, and Asian-Pacific-Islander) \n",
    "# only have around 100-300 samples making it impossible to accurately evaluate demographic parity, while the\n",
    "# group labelled 'Other' has even less data.\n",
    "# Moreover, on training data we find that there are less than 350 people identified as American-Indian-Eskimo, \n",
    "# meaning that it is not possible to predict if the positive prediction rate will hold on new data. \n",
    "fpredictor.evaluate_groups(verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fairness on COMPAS using Inferred Attributes\n",
    "\n",
    "We demonstrate how to enforce a wide range of fairness definitions on the COMPAS dataset. This dataset records paroles caught violating the terms of parole. As it measures who was caught, it is strongly influenced by policing and environmental biases, and should not be confused with a measurement of who actually violated their terms of parole. See [this paper](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/92cc227532d17e56e07902b254dfad10-Paper-round1.pdf) for a discussion of its limitations and caveats. \n",
    "We use it because it is a standard fairness dataset that captures such strong differences in outcome between people identified as African-American and everyone else, that classifiers trained on this dataset violate most definitions of fairness.\n",
    "\n",
    "As many of the ethnic groups are too small for reliable statistical estimation, we only consider differences is in outcomes between African-Americans vs. everyone else (labeled as other).\n",
    "We load and preprocess the COMPAS dataset, splitting it into three roughly equal partitions of train, validation, and test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:20:44.708055Z",
     "iopub.status.busy": "2024-06-17T19:20:44.707943Z",
     "iopub.status.idle": "2024-06-17T19:20:45.349115Z",
     "shell.execute_reply": "2024-06-17T19:20:45.348796Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from oxonfair import inferred_attribute_builder\n",
    "all_data = pd.read_csv('https://github.com/propublica/compas-analysis/raw/master/compas-scores-two-years.csv')\n",
    "condensed_data=all_data[['sex','race','age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count', 'age_cat', 'c_charge_degree','two_year_recid']].copy()\n",
    "condensed_data.replace({'Caucasian':'Other', 'Hispanic':'Other', 'Native American':'Other', 'Asian':'Other'},inplace=True)\n",
    "train=condensed_data.sample(frac=0.3, random_state=0)\n",
    "val_and_test=condensed_data.drop(train.index)\n",
    "val=val_and_test.sample(frac=0.5, random_state=0)\n",
    "test=val_and_test.drop(val.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:20:45.351727Z",
     "iopub.status.busy": "2024-06-17T19:20:45.351607Z",
     "iopub.status.idle": "2024-06-17T19:20:55.582861Z",
     "shell.execute_reply": "2024-06-17T19:20:55.582379Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No path specified. Models will be saved in: \"AutogluonModels/ag-20240617_192045\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No presets specified! To achieve strong results with AutoGluon, it is recommended to use the available presets.\n",
      "\tRecommended Presets (For more details refer to https://auto.gluon.ai/stable/tutorials/tabular/tabular-essentials.html#presets):\n",
      "\tpresets='best_quality'   : Maximize accuracy. Default time_limit=3600.\n",
      "\tpresets='high_quality'   : Strong accuracy with fast inference speed. Default time_limit=3600.\n",
      "\tpresets='good_quality'   : Good accuracy with very fast inference speed. Default time_limit=3600.\n",
      "\tpresets='medium_quality' : Fast training time, ideal for initial prototyping.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Beginning AutoGluon training ... Time limit = 5s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will save models to \"AutogluonModels/ag-20240617_192045\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "=================== System Info ===================\n",
      "AutoGluon Version:  1.1.0\n",
      "Python Version:     3.10.13\n",
      "Operating System:   Darwin\n",
      "Platform Machine:   arm64\n",
      "Platform Version:   Darwin Kernel Version 23.5.0: Wed May  1 20:14:38 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6020\n",
      "CPU Count:          10\n",
      "Memory Avail:       9.80 GB / 16.00 GB (61.3%)\n",
      "Disk Space Avail:   357.78 GB / 460.43 GB (77.7%)\n",
      "===================================================\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Rows:    2164\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Columns: 8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Label Column:       two_year_recid\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t2 unique label values:  [0, 1]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tIf 'binary' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Problem Type:       binary\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Preprocessing data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Selected class <--> label mapping:  class 1 = 1, class 0 = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Feature Generators to preprocess the data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting AutoMLPipelineFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tAvailable Memory:                    10034.31 MB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Original)  Memory Usage: 0.47 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 1 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting AsTypeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tNote: Converting 2 features to boolean dtype as they only contain 2 unique values.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 2 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting FillNaFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 3 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting IdentityFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting CategoryFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tFitting CategoryMemoryMinimizeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 4 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropUniqueFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 5 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropDuplicatesFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in original data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])    : 5 | ['age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('object', []) : 3 | ['sex', 'age_cat', 'c_charge_degree']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in processed data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('category', [])  : 1 | ['age_cat']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])       : 5 | ['age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', ['bool']) : 2 | ['sex', 'c_charge_degree']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s = Fit runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t8 features in original data used to generate 8 features in processed data.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Processed) Memory Usage: 0.09 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Data preprocessing and feature engineering runtime = 0.03s ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTo change this, specify the eval_metric parameter of Predictor()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 1731, Val Rows: 433\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "User-specified model hyperparameters to be fit:\n",
      "{\n",
      "\t'NN_TORCH': {},\n",
      "\t'GBM': [{'extra_trees': True, 'ag_args': {'name_suffix': 'XT'}}, {}, 'GBMLarge'],\n",
      "\t'CAT': {},\n",
      "\t'XGB': {},\n",
      "\t'FASTAI': {},\n",
      "\t'RF': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'XT': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'KNN': [{'weights': 'uniform', 'ag_args': {'name_suffix': 'Unif'}}, {'weights': 'distance', 'ag_args': {'name_suffix': 'Dist'}}],\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting 13 L1 models ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsUnif ... Training model for up to 4.97s of the 4.97s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6351\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.01s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsDist ... Training model for up to 4.94s of the 4.94s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6259\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.01s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: LightGBMXT ... Training model for up to 4.92s of the 4.92s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6882\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t1.56s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: LightGBM ... Training model for up to 3.35s of the 3.35s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6767\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t2.09s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: RandomForestGini ... Training model for up to 1.26s of the 1.26s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6074\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.46s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.04s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: RandomForestEntr ... Training model for up to 0.74s of the 0.74s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6097\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.35s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.04s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: CatBoost ... Training model for up to 0.33s of the 0.33s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTime limit exceeded... Skipping CatBoost.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: WeightedEnsemble_L2 ... Training model for up to 4.97s of the -0.06s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tEnsemble Weights: {'LightGBM': 0.667, 'LightGBMXT': 0.333}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6928\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.02s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon training complete, total runtime = 5.11s ... Best model: \"WeightedEnsemble_L2\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/ag-20240617_192045\")\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No path specified. Models will be saved in: \"AutogluonModels/ag-20240617_192050\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No presets specified! To achieve strong results with AutoGluon, it is recommended to use the available presets.\n",
      "\tRecommended Presets (For more details refer to https://auto.gluon.ai/stable/tutorials/tabular/tabular-essentials.html#presets):\n",
      "\tpresets='best_quality'   : Maximize accuracy. Default time_limit=3600.\n",
      "\tpresets='high_quality'   : Strong accuracy with fast inference speed. Default time_limit=3600.\n",
      "\tpresets='good_quality'   : Good accuracy with very fast inference speed. Default time_limit=3600.\n",
      "\tpresets='medium_quality' : Fast training time, ideal for initial prototyping.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Beginning AutoGluon training ... Time limit = 5s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will save models to \"AutogluonModels/ag-20240617_192050\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "=================== System Info ===================\n",
      "AutoGluon Version:  1.1.0\n",
      "Python Version:     3.10.13\n",
      "Operating System:   Darwin\n",
      "Platform Machine:   arm64\n",
      "Platform Version:   Darwin Kernel Version 23.5.0: Wed May  1 20:14:38 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6020\n",
      "CPU Count:          10\n",
      "Memory Avail:       10.55 GB / 16.00 GB (65.9%)\n",
      "Disk Space Avail:   358.74 GB / 460.43 GB (77.9%)\n",
      "===================================================\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Rows:    2164\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Columns: 8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Label Column:       race\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t2 unique label values:  ['African-American', 'Other']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tIf 'binary' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Problem Type:       binary\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Preprocessing data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Selected class <--> label mapping:  class 1 = Other, class 0 = African-American\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tNote: For your binary classification, AutoGluon arbitrarily selected which label-value represents positive (Other) vs negative (African-American) class.\n",
      "\tTo explicitly set the positive_class, either rename classes to 1 and 0, or specify positive_class in Predictor init.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Feature Generators to preprocess the data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting AutoMLPipelineFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tAvailable Memory:                    10803.08 MB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Original)  Memory Usage: 0.47 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 1 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting AsTypeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tNote: Converting 2 features to boolean dtype as they only contain 2 unique values.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 2 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting FillNaFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 3 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting IdentityFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting CategoryFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tFitting CategoryMemoryMinimizeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 4 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropUniqueFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 5 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropDuplicatesFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in original data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])    : 5 | ['age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('object', []) : 3 | ['sex', 'age_cat', 'c_charge_degree']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in processed data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('category', [])  : 1 | ['age_cat']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])       : 5 | ['age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', ['bool']) : 2 | ['sex', 'c_charge_degree']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s = Fit runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t8 features in original data used to generate 8 features in processed data.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Processed) Memory Usage: 0.09 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Data preprocessing and feature engineering runtime = 0.03s ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTo change this, specify the eval_metric parameter of Predictor()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 1731, Val Rows: 433\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "User-specified model hyperparameters to be fit:\n",
      "{\n",
      "\t'NN_TORCH': {},\n",
      "\t'GBM': [{'extra_trees': True, 'ag_args': {'name_suffix': 'XT'}}, {}, 'GBMLarge'],\n",
      "\t'CAT': {},\n",
      "\t'XGB': {},\n",
      "\t'FASTAI': {},\n",
      "\t'RF': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'XT': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'KNN': [{'weights': 'uniform', 'ag_args': {'name_suffix': 'Unif'}}, {'weights': 'distance', 'ag_args': {'name_suffix': 'Dist'}}],\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting 13 L1 models ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsUnif ... Training model for up to 4.97s of the 4.97s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6051\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.02s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsDist ... Training model for up to 4.94s of the 4.94s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.5889\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.01s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: LightGBMXT ... Training model for up to 4.92s of the 4.92s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6559\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t3.28s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: LightGBM ... Training model for up to 1.63s of the 1.63s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tRan out of time, early stopping on iteration 134. Best iteration is:\n",
      "\t[4]\tvalid_set's binary_error: 0.364896\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6351\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t1.63s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: WeightedEnsemble_L2 ... Training model for up to 4.97s of the -0.04s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tEnsemble Weights: {'LightGBMXT': 0.8, 'KNeighborsUnif': 0.2}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6582\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.02s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon training complete, total runtime = 5.08s ... Best model: \"WeightedEnsemble_L2\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/ag-20240617_192050\")\n"
     ]
    }
   ],
   "source": [
    "# To enforce fairness constraints without access to protected attributes at test time, we train two classifiers to infer the 2-year recidivism rate, and ethnicity.\n",
    "  \n",
    "predictor2, protected = inferred_attribute_builder(train, 'two_year_recid', 'race', time_limit=5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:20:55.586135Z",
     "iopub.status.busy": "2024-06-17T19:20:55.585848Z",
     "iopub.status.idle": "2024-06-17T19:20:55.712300Z",
     "shell.execute_reply": "2024-06-17T19:20:55.711572Z"
    }
   },
   "outputs": [],
   "source": [
    "# From these a single predictor that maximizes acuracy while reducing the demographic parity violation to less than 2.5% can be trained by running:\n",
    "\n",
    "fpredictor=FairPredictor(predictor2, val, 'race', inferred_groups=protected)\n",
    "fpredictor.fit(gm.accuracy, gm.demographic_parity, 0.025)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:20:55.716187Z",
     "iopub.status.busy": "2024-06-17T19:20:55.715980Z",
     "iopub.status.idle": "2024-06-17T19:20:55.719871Z",
     "shell.execute_reply": "2024-06-17T19:20:55.719294Z"
    }
   },
   "outputs": [],
   "source": [
    "#However, instead we will show how a family of fairness measures can be individually optimized. \n",
    "# The following code plots a table showing the change in accuracy and the fairness measure on a held-out test set as we decrease\n",
    "# the fairness measure to less than 0.025 for all measures except for disparate impact which we raise to above 0.975.\n",
    "# We define a helper function for evaluation:\n",
    "\n",
    "def evaluate(fpredictor,use_metrics):\n",
    "    \"Print a table showing the accuracy drop that comes with enforcing fairness\"\n",
    "    extra_metrics= {**use_metrics, 'accuracy':gm.accuracy}\n",
    "    collect=pd.DataFrame(columns=['Measure (original)', 'Measure (updated)', 'Accuracy (original)', 'Accuracy (updated)'])\n",
    "    for d in use_metrics.items():\n",
    "        if d[1].greater_is_better is False:\n",
    "            fpredictor.fit(gm.accuracy,d[1], 0.025)\n",
    "        else:\n",
    "            fpredictor.fit(gm.accuracy,d[1], 1-0.025)\n",
    "        tmp=fpredictor.evaluate_fairness(test,metrics=extra_metrics)\n",
    "        collect.loc[d[1].name]=np.concatenate((np.asarray(tmp.loc[d[1].name]), np.asarray(tmp.loc[gm.accuracy.name])), 0)\n",
    "    return collect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:20:55.722720Z",
     "iopub.status.busy": "2024-06-17T19:20:55.722572Z",
     "iopub.status.idle": "2024-06-17T19:21:00.856229Z",
     "shell.execute_reply": "2024-06-17T19:21:00.855874Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No path specified. Models will be saved in: \"AutogluonModels/ag-20240617_192055\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No presets specified! To achieve strong results with AutoGluon, it is recommended to use the available presets.\n",
      "\tRecommended Presets (For more details refer to https://auto.gluon.ai/stable/tutorials/tabular/tabular-essentials.html#presets):\n",
      "\tpresets='best_quality'   : Maximize accuracy. Default time_limit=3600.\n",
      "\tpresets='high_quality'   : Strong accuracy with fast inference speed. Default time_limit=3600.\n",
      "\tpresets='good_quality'   : Good accuracy with very fast inference speed. Default time_limit=3600.\n",
      "\tpresets='medium_quality' : Fast training time, ideal for initial prototyping.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Beginning AutoGluon training ... Time limit = 5s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will save models to \"AutogluonModels/ag-20240617_192055\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "=================== System Info ===================\n",
      "AutoGluon Version:  1.1.0\n",
      "Python Version:     3.10.13\n",
      "Operating System:   Darwin\n",
      "Platform Machine:   arm64\n",
      "Platform Version:   Darwin Kernel Version 23.5.0: Wed May  1 20:14:38 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6020\n",
      "CPU Count:          10\n",
      "Memory Avail:       7.88 GB / 16.00 GB (49.2%)\n",
      "Disk Space Avail:   358.74 GB / 460.43 GB (77.9%)\n",
      "===================================================\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Rows:    2164\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Columns: 9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Label Column:       two_year_recid\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t2 unique label values:  [0, 1]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tIf 'binary' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Problem Type:       binary\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Preprocessing data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Selected class <--> label mapping:  class 1 = 1, class 0 = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Feature Generators to preprocess the data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting AutoMLPipelineFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tAvailable Memory:                    8069.29 MB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Original)  Memory Usage: 0.61 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 1 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting AsTypeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tNote: Converting 3 features to boolean dtype as they only contain 2 unique values.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 2 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting FillNaFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 3 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting IdentityFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting CategoryFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tFitting CategoryMemoryMinimizeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 4 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropUniqueFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 5 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropDuplicatesFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in original data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])    : 5 | ['age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('object', []) : 4 | ['sex', 'race', 'age_cat', 'c_charge_degree']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in processed data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('category', [])  : 1 | ['age_cat']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])       : 5 | ['age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', ['bool']) : 3 | ['sex', 'race', 'c_charge_degree']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s = Fit runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t9 features in original data used to generate 9 features in processed data.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Processed) Memory Usage: 0.09 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Data preprocessing and feature engineering runtime = 0.04s ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTo change this, specify the eval_metric parameter of Predictor()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 1731, Val Rows: 433\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "User-specified model hyperparameters to be fit:\n",
      "{\n",
      "\t'NN_TORCH': {},\n",
      "\t'GBM': [{'extra_trees': True, 'ag_args': {'name_suffix': 'XT'}}, {}, 'GBMLarge'],\n",
      "\t'CAT': {},\n",
      "\t'XGB': {},\n",
      "\t'FASTAI': {},\n",
      "\t'RF': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'XT': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'KNN': [{'weights': 'uniform', 'ag_args': {'name_suffix': 'Unif'}}, {'weights': 'distance', 'ag_args': {'name_suffix': 'Dist'}}],\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting 13 L1 models ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsUnif ... Training model for up to 4.96s of the 4.96s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6351\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.02s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsDist ... Training model for up to 4.94s of the 4.94s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6259\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.02s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: LightGBMXT ... Training model for up to 4.91s of the 4.91s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6975\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t2.65s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: LightGBM ... Training model for up to 2.24s of the 2.24s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tRan out of time, early stopping on iteration 249. Best iteration is:\n",
      "\t[42]\tvalid_set's binary_error: 0.325635\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6744\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t2.25s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: WeightedEnsemble_L2 ... Training model for up to 4.96s of the -0.08s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tEnsemble Weights: {'LightGBMXT': 1.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6975\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.02s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon training complete, total runtime = 5.12s ... Best model: \"WeightedEnsemble_L2\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/ag-20240617_192055\")\n"
     ]
    }
   ],
   "source": [
    "#We can now contrast the behavior of a fair classifier that relies on access to the protected\n",
    "# attribtute at test time with one that infers it.\n",
    "\n",
    "# we first create a classifer using the protected attribute\n",
    "predictor=TabularPredictor(label='two_year_recid').fit(train_data=train, time_limit=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:21:00.858380Z",
     "iopub.status.busy": "2024-06-17T19:21:00.858264Z",
     "iopub.status.idle": "2024-06-17T19:21:01.258334Z",
     "shell.execute_reply": "2024-06-17T19:21:01.257694Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Measure (original)</th>\n",
       "      <th>Measure (updated)</th>\n",
       "      <th>Accuracy (original)</th>\n",
       "      <th>Accuracy (updated)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Demographic Parity</th>\n",
       "      <td>0.336580</td>\n",
       "      <td>0.010250</td>\n",
       "      <td>0.661386</td>\n",
       "      <td>0.646337</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Disparate Impact</th>\n",
       "      <td>0.424223</td>\n",
       "      <td>0.985274</td>\n",
       "      <td>0.661386</td>\n",
       "      <td>0.638812</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Conditional Acceptance Rate</th>\n",
       "      <td>0.644987</td>\n",
       "      <td>0.055467</td>\n",
       "      <td>0.661386</td>\n",
       "      <td>0.670891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Conditional Rejectance Rate</th>\n",
       "      <td>0.354906</td>\n",
       "      <td>0.003563</td>\n",
       "      <td>0.661386</td>\n",
       "      <td>0.666535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Accuracy</th>\n",
       "      <td>0.048422</td>\n",
       "      <td>0.036576</td>\n",
       "      <td>0.661386</td>\n",
       "      <td>0.662574</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Recall</th>\n",
       "      <td>0.306877</td>\n",
       "      <td>0.065208</td>\n",
       "      <td>0.661386</td>\n",
       "      <td>0.640396</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Acceptance Rate</th>\n",
       "      <td>0.000623</td>\n",
       "      <td>0.017213</td>\n",
       "      <td>0.661386</td>\n",
       "      <td>0.666139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Specificity</th>\n",
       "      <td>0.296718</td>\n",
       "      <td>0.054075</td>\n",
       "      <td>0.661386</td>\n",
       "      <td>0.671287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Rejection Rate</th>\n",
       "      <td>0.049978</td>\n",
       "      <td>0.072272</td>\n",
       "      <td>0.661386</td>\n",
       "      <td>0.667723</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Treatment Equality</th>\n",
       "      <td>1.117626</td>\n",
       "      <td>0.047186</td>\n",
       "      <td>0.661386</td>\n",
       "      <td>0.672079</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    Measure (original)  \\\n",
       "Demographic Parity                                            0.336580   \n",
       "Disparate Impact                                              0.424223   \n",
       "Average Group Difference in Conditional Accepta...            0.644987   \n",
       "Average Group Difference in Conditional Rejecta...            0.354906   \n",
       "Average Group Difference in Accuracy                          0.048422   \n",
       "Average Group Difference in Recall                            0.306877   \n",
       "Average Group Difference in Acceptance Rate                   0.000623   \n",
       "Average Group Difference in Specificity                       0.296718   \n",
       "Average Group Difference in Rejection Rate                    0.049978   \n",
       "Treatment Equality                                            1.117626   \n",
       "\n",
       "                                                    Measure (updated)  \\\n",
       "Demographic Parity                                           0.010250   \n",
       "Disparate Impact                                             0.985274   \n",
       "Average Group Difference in Conditional Accepta...           0.055467   \n",
       "Average Group Difference in Conditional Rejecta...           0.003563   \n",
       "Average Group Difference in Accuracy                         0.036576   \n",
       "Average Group Difference in Recall                           0.065208   \n",
       "Average Group Difference in Acceptance Rate                  0.017213   \n",
       "Average Group Difference in Specificity                      0.054075   \n",
       "Average Group Difference in Rejection Rate                   0.072272   \n",
       "Treatment Equality                                           0.047186   \n",
       "\n",
       "                                                    Accuracy (original)  \\\n",
       "Demographic Parity                                             0.661386   \n",
       "Disparate Impact                                               0.661386   \n",
       "Average Group Difference in Conditional Accepta...             0.661386   \n",
       "Average Group Difference in Conditional Rejecta...             0.661386   \n",
       "Average Group Difference in Accuracy                           0.661386   \n",
       "Average Group Difference in Recall                             0.661386   \n",
       "Average Group Difference in Acceptance Rate                    0.661386   \n",
       "Average Group Difference in Specificity                        0.661386   \n",
       "Average Group Difference in Rejection Rate                     0.661386   \n",
       "Treatment Equality                                             0.661386   \n",
       "\n",
       "                                                    Accuracy (updated)  \n",
       "Demographic Parity                                            0.646337  \n",
       "Disparate Impact                                              0.638812  \n",
       "Average Group Difference in Conditional Accepta...            0.670891  \n",
       "Average Group Difference in Conditional Rejecta...            0.666535  \n",
       "Average Group Difference in Accuracy                          0.662574  \n",
       "Average Group Difference in Recall                            0.640396  \n",
       "Average Group Difference in Acceptance Rate                   0.666139  \n",
       "Average Group Difference in Specificity                       0.671287  \n",
       "Average Group Difference in Rejection Rate                    0.667723  \n",
       "Treatment Equality                                            0.672079  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Create a fair predictor object, using the attribute 'race' on validation data\n",
    "fpredictor = FairPredictor(predictor, val, 'race')\n",
    "#and then evaluate it\n",
    "evaluate(fpredictor, gm.clarify_metrics)\n",
    "#Note that Class Imbalance is a property of the dataset and can not be altered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:21:01.260845Z",
     "iopub.status.busy": "2024-06-17T19:21:01.260654Z",
     "iopub.status.idle": "2024-06-17T19:21:02.672497Z",
     "shell.execute_reply": "2024-06-17T19:21:02.671929Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Measure (original)</th>\n",
       "      <th>Measure (updated)</th>\n",
       "      <th>Accuracy (original)</th>\n",
       "      <th>Accuracy (updated)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Demographic Parity</th>\n",
       "      <td>0.234362</td>\n",
       "      <td>0.035235</td>\n",
       "      <td>0.671287</td>\n",
       "      <td>0.554851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Disparate Impact</th>\n",
       "      <td>0.561681</td>\n",
       "      <td>0.988790</td>\n",
       "      <td>0.671287</td>\n",
       "      <td>0.542970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Conditional Acceptance Rate</th>\n",
       "      <td>0.300350</td>\n",
       "      <td>0.008952</td>\n",
       "      <td>0.671287</td>\n",
       "      <td>0.622574</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Conditional Rejectance Rate</th>\n",
       "      <td>0.165646</td>\n",
       "      <td>0.017900</td>\n",
       "      <td>0.671287</td>\n",
       "      <td>0.653861</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Accuracy</th>\n",
       "      <td>0.043198</td>\n",
       "      <td>0.042238</td>\n",
       "      <td>0.671287</td>\n",
       "      <td>0.666139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Recall</th>\n",
       "      <td>0.190611</td>\n",
       "      <td>0.018044</td>\n",
       "      <td>0.671287</td>\n",
       "      <td>0.557624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Acceptance Rate</th>\n",
       "      <td>0.033289</td>\n",
       "      <td>0.207826</td>\n",
       "      <td>0.671287</td>\n",
       "      <td>0.588515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Specificity</th>\n",
       "      <td>0.197344</td>\n",
       "      <td>0.076504</td>\n",
       "      <td>0.671287</td>\n",
       "      <td>0.651485</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Average Group Difference in Rejection Rate</th>\n",
       "      <td>0.074006</td>\n",
       "      <td>0.077252</td>\n",
       "      <td>0.671287</td>\n",
       "      <td>0.666535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Treatment Equality</th>\n",
       "      <td>0.561487</td>\n",
       "      <td>0.069292</td>\n",
       "      <td>0.671287</td>\n",
       "      <td>0.659010</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    Measure (original)  \\\n",
       "Demographic Parity                                            0.234362   \n",
       "Disparate Impact                                              0.561681   \n",
       "Average Group Difference in Conditional Accepta...            0.300350   \n",
       "Average Group Difference in Conditional Rejecta...            0.165646   \n",
       "Average Group Difference in Accuracy                          0.043198   \n",
       "Average Group Difference in Recall                            0.190611   \n",
       "Average Group Difference in Acceptance Rate                   0.033289   \n",
       "Average Group Difference in Specificity                       0.197344   \n",
       "Average Group Difference in Rejection Rate                    0.074006   \n",
       "Treatment Equality                                            0.561487   \n",
       "\n",
       "                                                    Measure (updated)  \\\n",
       "Demographic Parity                                           0.035235   \n",
       "Disparate Impact                                             0.988790   \n",
       "Average Group Difference in Conditional Accepta...           0.008952   \n",
       "Average Group Difference in Conditional Rejecta...           0.017900   \n",
       "Average Group Difference in Accuracy                         0.042238   \n",
       "Average Group Difference in Recall                           0.018044   \n",
       "Average Group Difference in Acceptance Rate                  0.207826   \n",
       "Average Group Difference in Specificity                      0.076504   \n",
       "Average Group Difference in Rejection Rate                   0.077252   \n",
       "Treatment Equality                                           0.069292   \n",
       "\n",
       "                                                    Accuracy (original)  \\\n",
       "Demographic Parity                                             0.671287   \n",
       "Disparate Impact                                               0.671287   \n",
       "Average Group Difference in Conditional Accepta...             0.671287   \n",
       "Average Group Difference in Conditional Rejecta...             0.671287   \n",
       "Average Group Difference in Accuracy                           0.671287   \n",
       "Average Group Difference in Recall                             0.671287   \n",
       "Average Group Difference in Acceptance Rate                    0.671287   \n",
       "Average Group Difference in Specificity                        0.671287   \n",
       "Average Group Difference in Rejection Rate                     0.671287   \n",
       "Treatment Equality                                             0.671287   \n",
       "\n",
       "                                                    Accuracy (updated)  \n",
       "Demographic Parity                                            0.554851  \n",
       "Disparate Impact                                              0.542970  \n",
       "Average Group Difference in Conditional Accepta...            0.622574  \n",
       "Average Group Difference in Conditional Rejecta...            0.653861  \n",
       "Average Group Difference in Accuracy                          0.666139  \n",
       "Average Group Difference in Recall                            0.557624  \n",
       "Average Group Difference in Acceptance Rate                   0.588515  \n",
       "Average Group Difference in Specificity                       0.651485  \n",
       "Average Group Difference in Rejection Rate                    0.666535  \n",
       "Treatment Equality                                            0.659010  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " # Now using infered attributes\n",
    "fpredictor2 = FairPredictor(predictor2, val, 'race', inferred_groups=protected)\n",
    "evaluate(fpredictor2, gm.clarify_metrics)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.15 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.1.-1"
  },
  "vscode": {
   "interpreter": {
    "hash": "397704579725e15f5c7cb49fe5f0341eb7531c82d19f2c29d197e8b64ab5776b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}