examples/compas_autogluon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fairness on COMPAS using Inferred Attributes and autogluon\n",
    "\n",
    "\n",
    "We demonstrate how to enforce a wide range of fairness definitions on the COMPAS dataset. This dataset records paroles caught violating the terms of parole. As it measures who was caught, it is strongly influenced by policing and environmental biases, and should not be confused with a measurement of who actually violated their terms of parole. See [this paper](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/92cc227532d17e56e07902b254dfad10-Paper-round1.pdf) for a discussion of its limitations and caveats.\n",
    "\n",
    "We use it because it is a standard fairness dataset that captures such strong differences in outcome between people identified as African-American and everyone else, that classifiers trained on this dataset violate most definitions of fairness.\n",
    "\n",
    "As many of the ethnic groups are too small for reliable statistical estimation, we only consider differences is in outcomes between African-Americans vs. everyone else (labeled as other).\n",
    "We load and preprocess the COMPAS dataset, splitting it into three roughly equal partitions of train, validation, and test:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:18:59.770600Z",
     "iopub.status.busy": "2024-06-17T19:18:59.770368Z",
     "iopub.status.idle": "2024-06-17T19:19:04.537414Z",
     "shell.execute_reply": "2024-06-17T19:19:04.537064Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/miniconda3/envs/ag/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from autogluon.tabular import TabularPredictor\n",
    "from oxonfair import FairPredictor, inferred_attribute_builder\n",
    "from oxonfair.utils import group_metrics as gm\n",
    "all_data = pd.read_csv('https://github.com/propublica/compas-analysis/raw/master/compas-scores-two-years.csv')\n",
    "condensed_data=all_data[['sex','race','age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count', 'age_cat', 'c_charge_degree','two_year_recid']].copy()\n",
    "condensed_data.replace({'Caucasian':'Other', 'Hispanic':'Other', 'Native American':'Other', 'Asian':'Other'},inplace=True)\n",
    "train=condensed_data.sample(frac=0.3, random_state=0)\n",
    "val_and_test=condensed_data.drop(train.index)\n",
    "val=val_and_test.sample(frac=0.5, random_state=0)\n",
    "test=val_and_test.drop(val.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To enforce fairness constraints without access to protected attributes at test time, we train two classifiers to infer the 2-year recidivism rate, and ethnicity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:04.539129Z",
     "iopub.status.busy": "2024-06-17T19:19:04.539027Z",
     "iopub.status.idle": "2024-06-17T19:19:15.151112Z",
     "shell.execute_reply": "2024-06-17T19:19:15.150017Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No path specified. Models will be saved in: \"AutogluonModels/ag-20240617_191904\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No presets specified! To achieve strong results with AutoGluon, it is recommended to use the available presets.\n",
      "\tRecommended Presets (For more details refer to https://auto.gluon.ai/stable/tutorials/tabular/tabular-essentials.html#presets):\n",
      "\tpresets='best_quality'   : Maximize accuracy. Default time_limit=3600.\n",
      "\tpresets='high_quality'   : Strong accuracy with fast inference speed. Default time_limit=3600.\n",
      "\tpresets='good_quality'   : Good accuracy with very fast inference speed. Default time_limit=3600.\n",
      "\tpresets='medium_quality' : Fast training time, ideal for initial prototyping.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Beginning AutoGluon training ... Time limit = 5s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will save models to \"AutogluonModels/ag-20240617_191904\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "=================== System Info ===================\n",
      "AutoGluon Version:  1.1.0\n",
      "Python Version:     3.10.13\n",
      "Operating System:   Darwin\n",
      "Platform Machine:   arm64\n",
      "Platform Version:   Darwin Kernel Version 23.5.0: Wed May  1 20:14:38 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6020\n",
      "CPU Count:          10\n",
      "Memory Avail:       6.44 GB / 16.00 GB (40.2%)\n",
      "Disk Space Avail:   360.82 GB / 460.43 GB (78.4%)\n",
      "===================================================\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Rows:    2164\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Columns: 8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Label Column:       two_year_recid\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t2 unique label values:  [0, 1]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tIf 'binary' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Problem Type:       binary\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Preprocessing data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Selected class <--> label mapping:  class 1 = 1, class 0 = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Feature Generators to preprocess the data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting AutoMLPipelineFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tAvailable Memory:                    6611.62 MB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Original)  Memory Usage: 0.47 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 1 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting AsTypeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tNote: Converting 2 features to boolean dtype as they only contain 2 unique values.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 2 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting FillNaFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 3 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting IdentityFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting CategoryFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tFitting CategoryMemoryMinimizeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 4 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropUniqueFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 5 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropDuplicatesFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in original data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])    : 5 | ['age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('object', []) : 3 | ['sex', 'age_cat', 'c_charge_degree']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in processed data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('category', [])  : 1 | ['age_cat']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])       : 5 | ['age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', ['bool']) : 2 | ['sex', 'c_charge_degree']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s = Fit runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t8 features in original data used to generate 8 features in processed data.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Processed) Memory Usage: 0.09 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Data preprocessing and feature engineering runtime = 0.03s ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTo change this, specify the eval_metric parameter of Predictor()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 1731, Val Rows: 433\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "User-specified model hyperparameters to be fit:\n",
      "{\n",
      "\t'NN_TORCH': {},\n",
      "\t'GBM': [{'extra_trees': True, 'ag_args': {'name_suffix': 'XT'}}, {}, 'GBMLarge'],\n",
      "\t'CAT': {},\n",
      "\t'XGB': {},\n",
      "\t'FASTAI': {},\n",
      "\t'RF': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'XT': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'KNN': [{'weights': 'uniform', 'ag_args': {'name_suffix': 'Unif'}}, {'weights': 'distance', 'ag_args': {'name_suffix': 'Dist'}}],\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting 13 L1 models ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsUnif ... Training model for up to 4.97s of the 4.97s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6351\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t1.43s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.03s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsDist ... Training model for up to 3.5s of the 3.5s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6259\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.01s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: LightGBMXT ... Training model for up to 3.48s of the 3.48s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/miniconda3/envs/ag/lib/python3.10/site-packages/dask/dataframe/__init__.py:31: FutureWarning: \n",
      "Dask dataframe query planning is disabled because dask-expr is not installed.\n",
      "\n",
      "You can install it with `pip install dask[dataframe]` or `conda install dask`.\n",
      "This will raise in a future version.\n",
      "\n",
      "  warnings.warn(msg, FutureWarning)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6882\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t2.71s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: LightGBM ... Training model for up to 0.76s of the 0.76s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tRan out of time, early stopping on iteration 69. Best iteration is:\n",
      "\t[27]\tvalid_set's binary_error: 0.323326\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6767\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.76s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: WeightedEnsemble_L2 ... Training model for up to 4.97s of the -0.03s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tEnsemble Weights: {'LightGBM': 0.667, 'LightGBMXT': 0.333}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6928\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.02s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon training complete, total runtime = 5.07s ... Best model: \"WeightedEnsemble_L2\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/ag-20240617_191904\")\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No path specified. Models will be saved in: \"AutogluonModels/ag-20240617_191909\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No presets specified! To achieve strong results with AutoGluon, it is recommended to use the available presets.\n",
      "\tRecommended Presets (For more details refer to https://auto.gluon.ai/stable/tutorials/tabular/tabular-essentials.html#presets):\n",
      "\tpresets='best_quality'   : Maximize accuracy. Default time_limit=3600.\n",
      "\tpresets='high_quality'   : Strong accuracy with fast inference speed. Default time_limit=3600.\n",
      "\tpresets='good_quality'   : Good accuracy with very fast inference speed. Default time_limit=3600.\n",
      "\tpresets='medium_quality' : Fast training time, ideal for initial prototyping.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Beginning AutoGluon training ... Time limit = 5s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will save models to \"AutogluonModels/ag-20240617_191909\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "=================== System Info ===================\n",
      "AutoGluon Version:  1.1.0\n",
      "Python Version:     3.10.13\n",
      "Operating System:   Darwin\n",
      "Platform Machine:   arm64\n",
      "Platform Version:   Darwin Kernel Version 23.5.0: Wed May  1 20:14:38 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6020\n",
      "CPU Count:          10\n",
      "Memory Avail:       6.47 GB / 16.00 GB (40.4%)\n",
      "Disk Space Avail:   360.82 GB / 460.43 GB (78.4%)\n",
      "===================================================\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Rows:    2164\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Columns: 8\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Label Column:       race\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t2 unique label values:  ['African-American', 'Other']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tIf 'binary' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Problem Type:       binary\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Preprocessing data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Selected class <--> label mapping:  class 1 = Other, class 0 = African-American\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tNote: For your binary classification, AutoGluon arbitrarily selected which label-value represents positive (Other) vs negative (African-American) class.\n",
      "\tTo explicitly set the positive_class, either rename classes to 1 and 0, or specify positive_class in Predictor init.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Feature Generators to preprocess the data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting AutoMLPipelineFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tAvailable Memory:                    6626.29 MB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Original)  Memory Usage: 0.47 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 1 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting AsTypeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tNote: Converting 2 features to boolean dtype as they only contain 2 unique values.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 2 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting FillNaFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 3 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting IdentityFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting CategoryFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tFitting CategoryMemoryMinimizeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 4 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropUniqueFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 5 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropDuplicatesFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in original data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])    : 5 | ['age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('object', []) : 3 | ['sex', 'age_cat', 'c_charge_degree']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in processed data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('category', [])  : 1 | ['age_cat']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])       : 5 | ['age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', ['bool']) : 2 | ['sex', 'c_charge_degree']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s = Fit runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t8 features in original data used to generate 8 features in processed data.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Processed) Memory Usage: 0.09 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Data preprocessing and feature engineering runtime = 0.04s ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTo change this, specify the eval_metric parameter of Predictor()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 1731, Val Rows: 433\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "User-specified model hyperparameters to be fit:\n",
      "{\n",
      "\t'NN_TORCH': {},\n",
      "\t'GBM': [{'extra_trees': True, 'ag_args': {'name_suffix': 'XT'}}, {}, 'GBMLarge'],\n",
      "\t'CAT': {},\n",
      "\t'XGB': {},\n",
      "\t'FASTAI': {},\n",
      "\t'RF': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'XT': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'KNN': [{'weights': 'uniform', 'ag_args': {'name_suffix': 'Unif'}}, {'weights': 'distance', 'ag_args': {'name_suffix': 'Dist'}}],\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting 13 L1 models ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsUnif ... Training model for up to 4.96s of the 4.96s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6051\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.02s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsDist ... Training model for up to 4.94s of the 4.94s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.5889\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.01s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: LightGBMXT ... Training model for up to 4.92s of the 4.92s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tRan out of time, early stopping on iteration 551. Best iteration is:\n",
      "\t[360]\tvalid_set's binary_error: 0.344111\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6559\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t4.95s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.01s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: WeightedEnsemble_L2 ... Training model for up to 4.96s of the -0.24s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tEnsemble Weights: {'LightGBMXT': 0.8, 'KNeighborsUnif': 0.2}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6582\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.02s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon training complete, total runtime = 5.41s ... Best model: \"WeightedEnsemble_L2\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/ag-20240617_191909\")\n"
     ]
    }
   ],
   "source": [
    "predictor2, protected = inferred_attribute_builder(train, 'two_year_recid', 'race', time_limit=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From these a single predictor that maximizes accuracy while reducing the demographic parity violation to less than 2.5% can be trained by running:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:15.155733Z",
     "iopub.status.busy": "2024-06-17T19:19:15.155103Z",
     "iopub.status.idle": "2024-06-17T19:19:15.367919Z",
     "shell.execute_reply": "2024-06-17T19:19:15.366933Z"
    }
   },
   "outputs": [],
   "source": [
    "fpredictor=FairPredictor(predictor2, val, 'race', inferred_groups=protected)\n",
    "fpredictor.fit(gm.accuracy, gm.demographic_parity, 0.025)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will show how a family of fairness measures can be individually optimized. First, we consider the measures of Sagemaker Clarify. \n",
    "\n",
    "The following code plots a table showing the change in accuracy and the fairness measure on a held-out test set.\n",
    "We decrease each fairness measure to less than 0.025 (on validation) for all measures except for disparate impact which we raise to above 0.975.\n",
    "We define a helper function for evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:15.372820Z",
     "iopub.status.busy": "2024-06-17T19:19:15.372419Z",
     "iopub.status.idle": "2024-06-17T19:19:15.400090Z",
     "shell.execute_reply": "2024-06-17T19:19:15.398771Z"
    }
   },
   "outputs": [],
   "source": [
    "def evaluate(fpredictor,use_metrics):\n",
    "    \"Print a table showing the accuracy drop that comes with enforcing fairness\"\n",
    "    extra_metrics = {**use_metrics, 'accuracy':gm.accuracy}\n",
    "    collect=pd.DataFrame(columns=['Measure (original)', 'Measure (updated)', 'Accuracy (original)', 'Accuracy (updated)'])\n",
    "    for d in use_metrics.items():\n",
    "        if d[1].greater_is_better is False:\n",
    "            fpredictor.fit(gm.accuracy, d[1], 0.025)\n",
    "        else:\n",
    "            fpredictor.fit(gm.accuracy, d[1], 1-0.025)\n",
    "        tmp=fpredictor.evaluate_fairness(test, metrics=extra_metrics,verbose=False)\n",
    "        collect.loc[d[1].name]=np.concatenate((np.asarray(tmp.loc[d[0]]), np.asarray(tmp.loc['accuracy'])), 0)\n",
    "    print(collect.to_markdown())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now contrast the behavior of a fair classifier that relies on access to the protected attribute at test time with one that infers it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:15.406078Z",
     "iopub.status.busy": "2024-06-17T19:19:15.405749Z",
     "iopub.status.idle": "2024-06-17T19:19:21.212793Z",
     "shell.execute_reply": "2024-06-17T19:19:21.211712Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No path specified. Models will be saved in: \"AutogluonModels/ag-20240617_191915\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No presets specified! To achieve strong results with AutoGluon, it is recommended to use the available presets.\n",
      "\tRecommended Presets (For more details refer to https://auto.gluon.ai/stable/tutorials/tabular/tabular-essentials.html#presets):\n",
      "\tpresets='best_quality'   : Maximize accuracy. Default time_limit=3600.\n",
      "\tpresets='high_quality'   : Strong accuracy with fast inference speed. Default time_limit=3600.\n",
      "\tpresets='good_quality'   : Good accuracy with very fast inference speed. Default time_limit=3600.\n",
      "\tpresets='medium_quality' : Fast training time, ideal for initial prototyping.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Beginning AutoGluon training ... Time limit = 5s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will save models to \"AutogluonModels/ag-20240617_191915\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "=================== System Info ===================\n",
      "AutoGluon Version:  1.1.0\n",
      "Python Version:     3.10.13\n",
      "Operating System:   Darwin\n",
      "Platform Machine:   arm64\n",
      "Platform Version:   Darwin Kernel Version 23.5.0: Wed May  1 20:14:38 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6020\n",
      "CPU Count:          10\n",
      "Memory Avail:       5.34 GB / 16.00 GB (33.4%)\n",
      "Disk Space Avail:   360.79 GB / 460.43 GB (78.4%)\n",
      "===================================================\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Rows:    2164\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Train Data Columns: 9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Label Column:       two_year_recid\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t2 unique label values:  [0, 1]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tIf 'binary' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Problem Type:       binary\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Preprocessing data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Selected class <--> label mapping:  class 1 = 1, class 0 = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Feature Generators to preprocess the data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting AutoMLPipelineFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tAvailable Memory:                    5469.42 MB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Original)  Memory Usage: 0.61 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 1 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting AsTypeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tNote: Converting 3 features to boolean dtype as they only contain 2 unique values.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 2 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting FillNaFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 3 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting IdentityFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting CategoryFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t\tFitting CategoryMemoryMinimizeFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 4 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropUniqueFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tStage 5 Generators:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\tFitting DropDuplicatesFeatureGenerator...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in original data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])    : 5 | ['age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('object', []) : 4 | ['sex', 'race', 'age_cat', 'c_charge_degree']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTypes of features in processed data (raw dtype, special dtypes):\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('category', [])  : 1 | ['age_cat']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', [])       : 5 | ['age', 'juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t('int', ['bool']) : 3 | ['sex', 'race', 'c_charge_degree']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.1s = Fit runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t9 features in original data used to generate 9 features in processed data.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTrain Data (Processed) Memory Usage: 0.09 MB (0.0% of available memory)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Data preprocessing and feature engineering runtime = 0.15s ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTo change this, specify the eval_metric parameter of Predictor()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 1731, Val Rows: 433\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "User-specified model hyperparameters to be fit:\n",
      "{\n",
      "\t'NN_TORCH': {},\n",
      "\t'GBM': [{'extra_trees': True, 'ag_args': {'name_suffix': 'XT'}}, {}, 'GBMLarge'],\n",
      "\t'CAT': {},\n",
      "\t'XGB': {},\n",
      "\t'FASTAI': {},\n",
      "\t'RF': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'XT': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],\n",
      "\t'KNN': [{'weights': 'uniform', 'ag_args': {'name_suffix': 'Unif'}}, {'weights': 'distance', 'ag_args': {'name_suffix': 'Dist'}}],\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting 13 L1 models ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsUnif ... Training model for up to 4.85s of the 4.85s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6351\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.01s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.03s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: KNeighborsDist ... Training model for up to 4.8s of the 4.79s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6259\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.02s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: LightGBMXT ... Training model for up to 4.73s of the 4.73s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tRan out of time, early stopping on iteration 343. Best iteration is:\n",
      "\t[81]\tvalid_set's binary_error: 0.30254\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6975\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t4.75s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.01s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: WeightedEnsemble_L2 ... Training model for up to 4.85s of the -0.08s of remaining time.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tEnsemble Weights: {'LightGBMXT': 1.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6975\t = Validation score   (accuracy)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.01s\t = Training   runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.0s\t = Validation runtime\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "AutoGluon training complete, total runtime = 5.15s ... Best model: \"WeightedEnsemble_L2\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/ag-20240617_191915\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|                                                         |   Measure (original) |   Measure (updated) |   Accuracy (original) |   Accuracy (updated) |\n",
      "|:--------------------------------------------------------|---------------------:|--------------------:|----------------------:|---------------------:|\n",
      "| Demographic Parity                                      |          0.33658     |          0.0102499  |              0.661386 |             0.646337 |\n",
      "| Disparate Impact                                        |          0.424223    |          0.985274   |              0.661386 |             0.638812 |\n",
      "| Average Group Difference in Conditional Acceptance Rate |          0.644987    |          0.055467   |              0.661386 |             0.670891 |\n",
      "| Average Group Difference in Conditional Rejectance Rate |          0.354906    |          0.00356274 |              0.661386 |             0.666535 |\n",
      "| Average Group Difference in Accuracy                    |          0.0484221   |          0.0365764  |              0.661386 |             0.662574 |\n",
      "| Average Group Difference in Recall                      |          0.306877    |          0.0652084  |              0.661386 |             0.640396 |\n",
      "| Average Group Difference in Acceptance Rate             |          0.000623377 |          0.0172127  |              0.661386 |             0.666139 |\n",
      "| Average Group Difference in Specificity                 |          0.296718    |          0.0540748  |              0.661386 |             0.671287 |\n",
      "| Average Group Difference in Rejection Rate              |          0.0499777   |          0.0722716  |              0.661386 |             0.667723 |\n",
      "| Treatment Equality                                      |          1.11763     |          0.0471855  |              0.661386 |             0.672079 |\n"
     ]
    }
   ],
   "source": [
    "# we first create a classifier using the protected attribute\n",
    "predictor=TabularPredictor(label='two_year_recid').fit(train_data=train,time_limit=5)\n",
    "fpredictor = FairPredictor(predictor, val, 'race', )\n",
    "evaluate(fpredictor, gm.clarify_metrics)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In contrast, even though the base classifiers have similar accuracy, when using inferred attributes (N.B. the base classifier is not directly trained to maximize accuracy, which is why it can have higher accuracy when it doesn't use race), we see a much greater drop in accuracy as fairness is enforced. This is consistent with [Lipton et al.](https://arxiv.org/pdf/1711.07076.pdf)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:21.224375Z",
     "iopub.status.busy": "2024-06-17T19:19:21.221254Z",
     "iopub.status.idle": "2024-06-17T19:19:23.079442Z",
     "shell.execute_reply": "2024-06-17T19:19:23.077623Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|                                                         |   Measure (original) |   Measure (updated) |   Accuracy (original) |   Accuracy (updated) |\n",
      "|:--------------------------------------------------------|---------------------:|--------------------:|----------------------:|---------------------:|\n",
      "| Demographic Parity                                      |            0.234362  |          0.0352353  |              0.671287 |             0.554851 |\n",
      "| Disparate Impact                                        |            0.561681  |          0.98879    |              0.671287 |             0.54297  |\n",
      "| Average Group Difference in Conditional Acceptance Rate |            0.30035   |          0.00895243 |              0.671287 |             0.622574 |\n",
      "| Average Group Difference in Conditional Rejectance Rate |            0.165646  |          0.0179003  |              0.671287 |             0.653861 |\n",
      "| Average Group Difference in Accuracy                    |            0.0431977 |          0.0422382  |              0.671287 |             0.666139 |\n",
      "| Average Group Difference in Recall                      |            0.190611  |          0.0180442  |              0.671287 |             0.557624 |\n",
      "| Average Group Difference in Acceptance Rate             |            0.0332893 |          0.207826   |              0.671287 |             0.588515 |\n",
      "| Average Group Difference in Specificity                 |            0.197344  |          0.076504   |              0.671287 |             0.651485 |\n",
      "| Average Group Difference in Rejection Rate              |            0.0740064 |          0.0772521  |              0.671287 |             0.666535 |\n",
      "| Treatment Equality                                      |            0.561487  |          0.0692925  |              0.671287 |             0.65901  |\n"
     ]
    }
   ],
   "source": [
    "# Now using the inferred attributes\n",
    "fpredictor2 = FairPredictor(predictor2, val, 'race', inferred_groups=protected)\n",
    "evaluate(fpredictor2, gm.clarify_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar results can be obtained using the metrics of Verma and Rubin, by running"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:23.104726Z",
     "iopub.status.busy": "2024-06-17T19:19:23.104155Z",
     "iopub.status.idle": "2024-06-17T19:19:23.481769Z",
     "shell.execute_reply": "2024-06-17T19:19:23.480274Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|                                                 |   Measure (original) |   Measure (updated) |   Accuracy (original) |   Accuracy (updated) |\n",
      "|:------------------------------------------------|---------------------:|--------------------:|----------------------:|---------------------:|\n",
      "| Statistical Parity                              |          0.33658     |           0.0102499 |              0.661386 |             0.646337 |\n",
      "| Predictive Parity                               |          0.000623377 |           0.0172127 |              0.661386 |             0.666139 |\n",
      "| Equal Opportunity                               |          0.306877    |           0.0652084 |              0.661386 |             0.640396 |\n",
      "| Average Group Difference in False Negative Rate |          0.306877    |           0.0652084 |              0.661386 |             0.640396 |\n",
      "| Equalized Odds                                  |          0.301797    |           0.0154928 |              0.661386 |             0.665347 |\n",
      "| Conditional Use Accuracy                        |          0.0253005   |           0.0353927 |              0.661386 |             0.665347 |\n",
      "| Average Group Difference in Accuracy            |          0.0484221   |           0.0365764 |              0.661386 |             0.662574 |\n",
      "| Treatment Equality                              |          1.11763     |           0.0471855 |              0.661386 |             0.672079 |\n"
     ]
    }
   ],
   "source": [
    "evaluate(fpredictor, gm.verma_metrics)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And again for infered attributes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-17T19:19:23.485490Z",
     "iopub.status.busy": "2024-06-17T19:19:23.485061Z",
     "iopub.status.idle": "2024-06-17T19:19:25.030825Z",
     "shell.execute_reply": "2024-06-17T19:19:25.028459Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No solutions satisfy the constraint found, selecting the closest solution.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|                                                 |   Measure (original) |   Measure (updated) |   Accuracy (original) |   Accuracy (updated) |\n",
      "|:------------------------------------------------|---------------------:|--------------------:|----------------------:|---------------------:|\n",
      "| Statistical Parity                              |            0.234362  |           0.0352353 |              0.671287 |             0.554851 |\n",
      "| Predictive Parity                               |            0.0332893 |           0.207826  |              0.671287 |             0.588515 |\n",
      "| Equal Opportunity                               |            0.190611  |           0.0180442 |              0.671287 |             0.557624 |\n",
      "| Average Group Difference in False Negative Rate |            0.190611  |           0.0180442 |              0.671287 |             0.557624 |\n",
      "| Equalized Odds                                  |            0.193978  |           0.0333313 |              0.671287 |             0.56396  |\n",
      "| Conditional Use Accuracy                        |            0.0536478 |           0.0568714 |              0.671287 |             0.666931 |\n",
      "| Average Group Difference in Accuracy            |            0.0431977 |           0.0422382 |              0.671287 |             0.666139 |\n",
      "| Treatment Equality                              |            0.561487  |           0.0692925 |              0.671287 |             0.65901  |\n"
     ]
    }
   ],
   "source": [
    "evaluate(fpredictor2, gm.verma_metrics)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ag",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.1.-1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}