Skip to content

Commit

Permalink
case study naming
Browse files Browse the repository at this point in the history
  • Loading branch information
jasonjabbour committed Nov 17, 2024
1 parent 2a9e1cd commit 4098c9b
Showing 1 changed file with 10 additions and 13 deletions.
23 changes: 10 additions & 13 deletions contents/core/privacy_security/privacy_security.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ While the specifics of machine learning hardware security can be distinct, the e

In 2010, something unexpected was found on a computer in Iran - a very complicated computer virus that experts had never seen before. [Stuxnet](https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/200661/Cyber-Reports-2017-04.pdf) was a malicious computer worm that targeted supervisory control and data acquisition (SCADA) systems and was designed to damage Iran's nuclear program [@farwell2011stuxnet]. Stuxnet was using four "[zero-day exploits](https://en.wikipedia.org/wiki/Zero-day_(computing))" - attacks that take advantage of secret weaknesses in software that no one knows about yet. This made Stuxnet very sneaky and hard to detect.

But Stuxnet wasn't designed to steal information or spy on people. Its goal was physical destruction - to sabotage centrifuges at Iran's Natanz nuclear plant! So how did the virus get onto computers at the Natanz plant, which was supposed to be disconnected from the outside world for security? Experts think someone inserted a USB stick containing Stuxnet into the internal Natanz network. This allowed the virus to "jump" from an outside system onto the isolated nuclear control systems and wreak havoc.
But Stuxnet wasn't designed to steal information or spy on people. Its goal was physical destruction - to sabotage centrifuges at Iran's Natanz nuclear plant! So, how did the virus get onto computers at the Natanz plant, which was supposed to be disconnected from the outside world for security? Experts think someone inserted a USB stick containing Stuxnet into the internal Natanz network. This allowed the virus to "jump" from an outside system onto the isolated nuclear control systems and wreak havoc.

Stuxnet was incredibly advanced malware built by national governments to cross from the digital realm into real-world infrastructure. It specifically targeted important industrial machines, where embedded machine learning is highly applicable in a way never done before. The virus provided a wake-up call about how sophisticated cyberattacks could now physically destroy equipment and facilities.

Expand Down Expand Up @@ -106,7 +106,7 @@ Although the devices were not ML-based, the incident is a stark reminder of what

### Implications

These historical breaches demonstrate the cascading effects of hardware vulnerabilities in embedded systems. Each incident offers a precedent for understanding the risks and designing better security protocols. For instance, the Mirai botnet highlights the immense destructive potential when threat actors can gain control over networked devices with weak security, a situation becoming increasingly common with ML systems. Many current ML devices function as "edge" devices meant to collect and process data locally before sending it to the cloud. Much like the cameras and DVRs compromised by Mirai, edge ML devices often rely on embedded hardware like ARM processors and run lightweight O.S. like Linux. Securing the device credentials is critical.
These historical breaches demonstrate the cascading effects of hardware vulnerabilities in embedded systems. Each incident offers a precedent for understanding the risks and designing better security protocols. For instance, the Mirai botnet highlights the immense destructive potential when threat actors can gain control over networked devices with weak security, a situation becoming increasingly common with ML systems. Many current ML devices function as "edge" devices meant to collect and process data locally before sending it to the cloud. Much like the cameras and DVRs compromised by Mirai, edge ML devices often rely on embedded hardware like ARM processors and run lightweight OS like Linux. Securing the device credentials is critical.

Similarly, the Jeep Cherokee hack was a watershed moment for the automotive industry. It exposed serious vulnerabilities in the growing network-connected vehicle systems and their lack of isolation from core drive systems like brakes and steering. In response, auto manufacturers invested heavily in new cybersecurity measures, though gaps likely remain.

Expand All @@ -124,10 +124,7 @@ The lesson is the importance of designing for security from the outset and havin

## Security Threats to ML Models

ML models face security risks that can undermine their integrity, performance, and trustworthiness if not adequately addressed. While there are several different threats, the primary threats include:
Model theft, where adversaries steal the proprietary model parameters and the sensitive data they contain.
Data poisoning, which compromises models through data tampering.
Adversarial attacks deceive the model to make incorrect or unwanted predictions.
ML models face security risks that can undermine their integrity, performance, and trustworthiness if not adequately addressed. Among these, three primary threats stand out: model theft, where adversaries steal proprietary model parameters and the sensitive data they contain; data poisoning, which compromises models by tampering with training data; and adversarial attacks, designed to deceive models into making incorrect or unwanted predictions. We will discuss each of these threats in detail and provide case study examples to illustrate their real-world implications.

### Model Theft

Expand Down Expand Up @@ -168,13 +165,13 @@ Instead of extracting exact numerical values of the model's parameters, these at

* **Prediction Consistency:** The attacker tries to align their model's prediction patterns with the target model's. This involves matching prediction outputs (both positive and negative) on the same set of inputs and ensuring distributional consistency across different classes. For instance, consider a natural language processing (NLP) model that generates sentiment analysis for movie reviews (labels reviews as positive, neutral, or negative). The attacker will try to fine-tune their model to match the prediction of the original models on the same set of movie reviews. This includes ensuring that the model makes the same mistakes (mispredictions) that the targeted model makes.

#### Case Study
#### Case Study: Tesla's IP Theft Case

In 2018, Tesla filed a [lawsuit](https://storage.courtlistener.com/recap/gov.uscourts.nvd.131251/gov.uscourts.nvd.131251.1.0_1.pdf) against self-driving car startup [Zoox](https://zoox.com/), alleging former employees stole confidential data and trade secrets related to Tesla's autonomous driving assistance system.

Tesla claimed that several of its former employees took over 10 G.B. of proprietary data, including ML models and source code, before joining Zoox. This allegedly included one of Tesla's crucial image recognition models for identifying objects.
Tesla claimed that several of its former employees took over 10 GB of proprietary data, including ML models and source code, before joining Zoox. This allegedly included one of Tesla's crucial image recognition models for identifying objects.

The theft of this sensitive proprietary model could help Zoox shortcut years of ML development and duplicate Tesla's capabilities. Tesla argued this theft of I.P. caused significant financial and competitive harm. There were also concerns it could allow model inversion attacks to infer private details about Tesla's testing data.
The theft of this sensitive proprietary model could help Zoox shortcut years of ML development and duplicate Tesla's capabilities. Tesla argued this theft of IP caused significant financial and competitive harm. There were also concerns it could allow model inversion attacks to infer private details about Tesla's testing data.

The Zoox employees denied stealing any proprietary information. However, the case highlights the significant risks of model theft—enabling the cloning of commercial models, causing economic impacts, and opening the door for further data privacy violations.

Expand Down Expand Up @@ -208,7 +205,7 @@ There are six main categories of data poisoning [@oprea2022poisoning]:

* **Knowledge:** While targeted attacks require a high degree of familiarity with the data, subpopulation attacks require less intimate knowledge to be effective.

#### Case Study 1
#### Case Study: Poisoning Content Moderation Systems

In 2017, researchers demonstrated a data poisoning attack against a popular toxicity classification model called Perspective [@hosseini2017deceiving]. This ML model detects toxic comments online.

Expand All @@ -218,9 +215,9 @@ After retraining on the poisoned data, the model's false negative rate increased

This case highlights how data poisoning can degrade model accuracy and reliability. For social media platforms, a poisoning attack that impairs toxicity detection could lead to the proliferation of harmful content and distrust of ML moderation systems. The example demonstrates why securing training data integrity and monitoring for poisoning is critical across application domains.

#### Case Study 2
#### Case Study: Protecting Art Through Data Poisoning

Interestingly enough, data poisoning attacks are not always malicious [@shan2023prompt]. Nightshade, a tool developed by a team led by Professor Ben Zhao at the University of Chicago, utilizes data poisoning to help artists protect their art against scraping and copyright violations by generative A.I. models. Artists can use the tool to modify their images subtly before uploading them online.
Interestingly enough, data poisoning attacks are not always malicious [@shan2023prompt]. Nightshade, a tool developed by a team led by Professor Ben Zhao at the University of Chicago, utilizes data poisoning to help artists protect their art against scraping and copyright violations by generative AI models. Artists can use the tool to modify their images subtly before uploading them online.

While these changes are imperceptible to the human eye, they can significantly degrade the performance of generative AI models when integrated into the training data. Generative models can be manipulated to produce unrealistic or nonsensical outputs. For example, with just 300 corrupted images, the University of Chicago researchers could deceive the latest Stable Diffusion model into generating images of canines resembling felines or bovines when prompted for automobiles.

Expand Down Expand Up @@ -254,7 +251,7 @@ The landscape of machine learning models is complex and broad, especially given

* **Transfer Learning Adversarial Attacks** exploit the knowledge transferred from a pre-trained model to a target model, creating adversarial examples that can deceive both models. These attacks pose a growing concern, particularly when adversaries have knowledge of the feature extractor but lack access to the classification head (the part or layer responsible for making the final classifications). Referred to as “headless attacks," these transferable adversarial strategies leverage the expressive capabilities of feature extractors to craft perturbations while oblivious to the label space or training data. The existence of such attacks underscores the importance of developing robust defenses for transfer learning applications, especially since pre-trained models are commonly used [@ahmed2020headless].

#### Case Study
#### Case Study: Tricking Traffic Sign Detection Models

In 2017, researchers conducted experiments by placing small black and white stickers on stop signs [@eykholt2018robust]. When viewed by a normal human eye, the stickers did not obscure the sign or prevent interpretability. However, when images of the stickers stop signs were fed into standard traffic sign classification ML models, they were misclassified as speed limit signs over 85% of the time.

Expand Down

0 comments on commit 4098c9b

Please sign in to comment.