Skip to content

Commit

Permalink
Figure placement
Browse files Browse the repository at this point in the history
  • Loading branch information
profvjreddi committed Nov 17, 2024
1 parent 863156e commit af5e625
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions contents/core/responsible_ai/responsible_ai.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -197,10 +197,10 @@ To evaluate and address these issues, fairness in machine learning is typically

This concept is illustrated in @fig-fairness-example, which visualizes the performance of a machine learning model predicting loan repayment for two subgroups, Subgroup A (blue) and Subgroup B (red). Each individual in the dataset is represented by a symbol: plusses (+) indicate individuals who will repay their loans (true positives), while circles (O) indicate individuals who will default on their loans (true negatives). The model's objective is to correctly classify these individuals into repayers and defaulters.

To evaluate performance, two dotted lines are shown, representing the thresholds at which the model achieves acceptable accuracy for each subgroup. For Subgroup A, the threshold needs to be set at 81.25% accuracy (the second dotted line) to correctly classify all repayers (plusses). However, using this same threshold for Subgroup B would result in misclassifications, as some repayers in Subgroup B would incorrectly fall below this threshold and be classified as defaulters. For Subgroup B, a lower threshold of 75% accuracy (the first dotted line) is necessary to correctly classify its repayers. However, applying this lower threshold to Subgroup A would result in misclassifications for that group. This illustrates how the model performs unequally across the two subgroups, with each requiring a different threshold to maximize their true positive rates.

![Illustrates the trade-off in setting classification thresholds for two subgroups (A and B) in a loan repayment model. Plusses (+) represent true positives (repayers), and circles (O) represent true negatives (defaulters). Different thresholds (75% for B and 81.25% for A) maximize subgroup accuracy but reveal fairness challenges.](images/png/fairness_cartoon.png){#fig-fairness-example}

To evaluate performance, two dotted lines are shown, representing the thresholds at which the model achieves acceptable accuracy for each subgroup. For Subgroup A, the threshold needs to be set at 81.25% accuracy (the second dotted line) to correctly classify all repayers (plusses). However, using this same threshold for Subgroup B would result in misclassifications, as some repayers in Subgroup B would incorrectly fall below this threshold and be classified as defaulters. For Subgroup B, a lower threshold of 75% accuracy (the first dotted line) is necessary to correctly classify its repayers. However, applying this lower threshold to Subgroup A would result in misclassifications for that group. This illustrates how the model performs unequally across the two subgroups, with each requiring a different threshold to maximize their true positive rates.

The disparity in required thresholds highlights the challenge of achieving fairness in model predictions. If positive classifications lead to loan approvals, individuals in Subgroup B would be disadvantaged unless the threshold is adjusted specifically for their subgroup. However, adjusting thresholds introduces trade-offs between group-level accuracy and fairness, demonstrating the inherent tension in optimizing for these objectives in machine learning systems.

Thus, the fairness literature has proposed three main _fairness metrics_ for quantifying how fair a model performs over a dataset [@hardt2016equality]. Given a model $h$ and a dataset $D$ consisting of $(x, y, s)$ samples, where $x$ is the data features, $y$ is the label, and $s$ is the subgroup attribute, and we assume there are simply two subgroups $a$ and $b$, we can define the following:
Expand Down

0 comments on commit af5e625

Please sign in to comment.