Skip to content

Commit

Permalink
fixing figure and clarifying figure explanation
Browse files Browse the repository at this point in the history
  • Loading branch information
jasonjabbour committed Nov 17, 2024
1 parent ae2d91c commit 6c095a2
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 12 deletions.
Binary file modified contents/core/responsible_ai/images/png/fairness_cartoon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
23 changes: 11 additions & 12 deletions contents/core/responsible_ai/responsible_ai.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ With TinyML, accountability mechanisms must be traced across long, complex suppl

### Governance

Organizations institute internal governance for cloud ML, such as ethics boards, audits, and model risk management. But external governance also oversees cloud ML, like regulations on bias and transparency such as the [AI Bill of Rights](https://www.whitehouse.gov/ostp/ai-bill-of-rights/), [General Data Protection Regulation (GDPR)](https://gdpr-info.eu/), and [California Consumer Protection Act (CCPA)](https://oag.ca.gov/privacy/ccpa). Third-party auditing supports cloud ML governance.
Organizations institute internal governance for cloud ML, such as ethics boards, audits, and model risk management. External governance also plays a significant role in ensuring accountability and fairness. We have already introduced the [General Data Protection Regulation (GDPR)](https://gdpr-info.eu/), which sets stringent requirements for data protection and transparency. However, it is not the only framework guiding responsible AI practices. The [AI Bill of Rights](https://www.whitehouse.gov/ostp/ai-bill-of-rights/) establishes principles for ethical AI use in the United States, and the [California Consumer Protection Act (CCPA)](https://oag.ca.gov/privacy/ccpa) focuses on safeguarding consumer data privacy within California. Third-party audits further bolster cloud ML governance by providing external oversight.

Edge ML is more decentralized, requiring responsible self-governance by developers and companies deploying models locally. Industry associations coordinate governance across edge ML vendors, and open software helps align incentives for ethical edge ML.

Expand Down Expand Up @@ -186,33 +186,32 @@ Extreme decentralization and complexity make external governance infeasible with
| | like GDPR or CCPA are feasible | developers and stakeholders | and cryptographic assurances |
+------------------------+--------------------------------------+------------------------------------+--------------------------------+


: Comparison of key principles in Cloud ML, Edge ML, and TinyML. {#tbl-ml-principles-comparison .striped .hover}


## Technical Aspects

### Detecting and Mitigating Bias

A large body of work has demonstrated that machine learning models can exhibit bias, from underperforming people of a certain identity to making decisions that limit groups' access to important resources [@buolamwini2018genderShades].
Machine learning models, like any complex system, can sometimes exhibit biases in their predictions. These biases may manifest in underperformance for specific groups or in decisions that inadvertently restrict access to certain opportunities or resources [@buolamwini2018genderShades]. Understanding and addressing these biases is critical, especially as machine learning systems are increasingly used in sensitive domains like lending, healthcare, and criminal justice.

Ensuring fair and equitable treatment for all groups affected by machine learning systems is crucial as these models increasingly impact people's lives in areas like lending, healthcare, and criminal justice. We typically evaluate model fairness by considering "subgroup attributes" unrelated to the prediction task that capture identities like race, gender, or religion. For example, in a loan default prediction model, subgroups could include race, gender, or religion. When models are trained naively to maximize accuracy, they often ignore subgroup performance. However, this can negatively impact marginalized communities.
To evaluate and address these issues, fairness in machine learning is typically assessed by analyzing "subgroup attributes," which are characteristics unrelated to the prediction task, such as geographic location, age group, income level, race, gender, or religion. For example, in a loan default prediction model, subgroups could include race, gender, or religion. When models are trained with the sole objective of maximizing accuracy, they may overlook performance differences across these subgroups, potentially resulting in biased or inconsistent outcomes.

To illustrate, imagine a model predicting loan repayment where the plusses (+'s) represent repayment and the circles (O's) represent default, as shown in @fig-fairness-example. The optimal accuracy would be correctly classifying all of Group A while misclassifying some of Group B's creditworthy applicants as defaults. If positive classifications allow access loans, Group A would receive many more loans---which would naturally result in a biased outcome.
This concept is illustrated in @fig-fairness-example, which visualizes the performance of a machine learning model predicting loan repayment for two subgroups, Subgroup A (blue) and Subgroup B (red). Each individual in the dataset is represented by a symbol: plusses (+) indicate individuals who will repay their loans (true positives), while circles (O) indicate individuals who will default on their loans (true negatives). The model’s objective is to correctly classify these individuals into repayers and defaulters.

![Fairness and accuracy.](images/png/fairness_cartoon.png){#fig-fairness-example}
To evaluate performance, two dotted lines are shown, representing the thresholds at which the model achieves acceptable accuracy for each subgroup. For Subgroup A, the threshold needs to be set at 81.25% accuracy (the second dotted line) to correctly classify all repayers (plusses). However, using this same threshold for Subgroup B would result in misclassifications, as some repayers in Subgroup B would incorrectly fall below this threshold and be classified as defaulters. For Subgroup B, a lower threshold of 75% accuracy (the first dotted line) is necessary to correctly classify its repayers. However, applying this lower threshold to Subgroup A would result in misclassifications for that group. This illustrates how the model performs unequally across the two subgroups, with each requiring a different threshold to maximize their true positive rates.

Alternatively, correcting the biases against Group B would likely increase "false positives" and reduce accuracy for Group A. Or, we could train separate models focused on maximizing true positives for each group. However, this would require explicitly using sensitive attributes like race in the decision process.
![Illustrates the trade-off in setting classification thresholds for two subgroups (A and B) in a loan repayment model. Plusses (+) represent true positives (repayers), and circles (O) represent true negatives (defaulters). Different thresholds (75% for B and 81.25% for A) maximize subgroup accuracy but reveal fairness challenges.](images/png/fairness_cartoon.png){#fig-fairness-example}

As we see, there are inherent tensions around priorities like accuracy versus subgroup fairness and whether to explicitly account for protected classes. Reasonable people can disagree on the appropriate tradeoffs. Constraints around costs and implementation options further complicate matters. Overall, ensuring the fair and ethical use of machine learning involves navigating these complex challenges.
The disparity in required thresholds highlights the challenge of achieving fairness in model predictions. If positive classifications lead to loan approvals, individuals in Subgroup B would be disadvantaged unless the threshold is adjusted specifically for their subgroup. However, adjusting thresholds introduces trade-offs between group-level accuracy and fairness, demonstrating the inherent tension in optimizing for these objectives in machine learning systems.

Thus, the fairness literature has proposed three main _fairness metrics_ for quantifying how fair a model performs over a dataset [@hardt2016equality]. Given a model h and a dataset D consisting of (x,y,s) samples, where x is the data features, y is the label, and s is the subgroup attribute, and we assume there are simply two subgroups a and b, we can define the following.
Thus, the fairness literature has proposed three main _fairness metrics_ for quantifying how fair a model performs over a dataset [@hardt2016equality]. Given a model $h$ and a dataset $D$ consisting of $(x, y, s)$ samples, where $x$ is the data features, $y$ is the label, and $s$ is the subgroup attribute, and we assume there are simply two subgroups $a$ and $b$, we can define the following:

1. **Demographic Parity** asks how accurate a model is for each subgroup. In other words, P(h(X) = Y S = a) = P(h(X) = Y S = b)
1. **Demographic Parity** asks how accurate a model is for each subgroup. In other words, $P(h(X) = Y \mid S = a) = P(h(X) = Y \mid S = b)$.

2. **Equalized Odds** asks how precise a model is on positive and negative samples for each subgroup. P(h(X) = y S = a, Y = y) = P(h(X) = y S = b, Y = y)
2. **Equalized Odds** asks how precise a model is on positive and negative samples for each subgroup. $P(h(X) = y \mid S = a, Y = y) = P(h(X) = y \mid S = b, Y = y)$.

3. **Equality of Opportunity** is a special case of equalized odds that only asks how precise a model is on positive samples. This is relevant in cases such as resource allocation, where we care about how positive (i.e., resource-allocated) labels are distributed across groups. For example, we care that an equal proportion of loans are given to both men and women. P(h(X) = 1 S = a, Y = 1) = P(h(X) = 1 S = b, Y = 1)
3. **Equality of Opportunity** is a special case of equalized odds that only asks how precise a model is on positive samples. This is relevant in cases such as resource allocation, where we care about how positive (i.e., resource-allocated) labels are distributed across groups. For example, we care that an equal proportion of loans are given to both men and women. $P(h(X) = 1 \mid S = a, Y = 1) = P(h(X) = 1 \mid S = b, Y = 1)$.

Note: These definitions often take a narrow view when considering binary comparisons between two subgroups. Another thread of fair machine learning research focusing on _multicalibration_ and _multiaccuracy_ considers the interactions between an arbitrary number of identities, acknowledging the inherent intersectionality of individual identities in the real world [@hebert2018multicalibration].

Expand Down

0 comments on commit 6c095a2

Please sign in to comment.