diff --git a/privacy_security.qmd b/privacy_security.qmd index be4ccf2d..3b1c099a 100644 --- a/privacy_security.qmd +++ b/privacy_security.qmd @@ -92,10 +92,10 @@ using ML. Here are detailed explorations of past breaches: ### Stuxnet In 2010, something unexpected was found on a computer in Iran - a very -complicated computer virus that experts had never seen before. Stuxnet +complicated computer virus that experts had never seen before. [Stuxnet](https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/200661/Cyber-Reports-2017-04.pdf) was a malicious computer worm that targeted supervisory control and data acquisition (SCADA) systems and was designed to damage Iran's nuclear -program [?]. Stuxnet was using four "zero-day exploits" - attacks +program [@farwell2011stuxnet]. Stuxnet was using four "[zero-day exploits](https://en.wikipedia.org/wiki/Zero-day_(computing))" - attacks that take advantage of secret weaknesses in software that no one knows about yet. This made Stuxnet very sneaky and hard to detect. @@ -128,7 +128,7 @@ and physical worlds with devastating effects. ### Jeep Cherokee Hack The Jeep Cherokee hack was a groundbreaking event demonstrating the -risks inherent in increasingly connected automobiles [?]. In a +risks inherent in increasingly connected automobiles [@miller2019lessons]. In a controlled demonstration, security researchers remotely exploited a vulnerability in the Uconnect entertainment system, which had a cellular connection to the internet. They were able to control the vehicle's @@ -136,8 +136,9 @@ engine, transmission, and brakes, alarming the automotive industry into recognizing the severe safety implications of cyber vulnerabilities in vehicles. -[Hackers Remotely Kill a Jeep on a Highway \| -WIRED](https://www.youtube.com/watch?v=MK0SrxBC1xs&ab_channel=WIRED) +{{< video https://www.youtube.com/watch?v=MK0SrxBC1xs&ab_channel=WIRED + title="Hackers Remotely Kill a Jeep on a Highway" +>}} While this wasn't an attack on an ML system per se, the reliance of modern vehicles on embedded systems for safety-critical functions has @@ -230,22 +231,17 @@ to security rather than leaving it as an afterthought. Rapid response and dissemination of best practices will be key as threats continue evolving. -## Security Threats to ML Models (Lead: Elias) +## Security Threats to ML Models ML models face security risks that can undermine their integrity, -performance, and trustworthiness if not properly addressed. While there -are several different threats, the key threats include 1) model theft, -where adversaries steal the proprietary model parameters and the -sensitive data they contain; 2) data poisoning, which compromises models -through data tampering; and 3) adversarial attacks, which deceive the -model to make incorrect or unwanted predictions +performance, and trustworthiness if not properly addressed. While there are several different threats, the key threats include: 1) model theft, where adversaries steal the proprietary model parameters and the sensitive data they contain; 2) data poisoning, which compromises models through data tampering; and 3) adversarial attacks, which deceive the model to make incorrect or unwanted predictions. ### Model Theft Model theft occurs when an attacker gains unauthorized access to a deployed ML model. The concern here is the theft of the model's structure and trained parameters and the proprietary data it contains -[?]. Model theft is a real and growing threat, as demonstrated by +[@ateniese2015hacking]. Model theft is a real and growing threat, as demonstrated by cases like ex-Google engineer Anthony Levandowski, who [allegedly stole Waymo's self-driving car designs](https://www.nytimes.com/2017/02/23/technology/google-self-driving-waymo-uber-otto-lawsuit.html) @@ -283,7 +279,7 @@ One historical example of such a vulnerability being explored was the research on inversion attacks against the U.S. Netflix Prize dataset, where researchers demonstrated that it was possible to learn about an individual's movie preferences, which could lead to privacy breaches -[?]. +[@narayanan2006break]. Model theft implies that it could lead to economic losses, undermine competitive advantage, and violate user privacy. There's also the risk @@ -298,7 +294,7 @@ desired asset: exact model properties and approximate model behavior. In these attacks, the objective is to extract information about concrete metrics, such as the learned parameters of a network, the fine-tuned -hyperparameters, and the model's internal layer architecture. +hyperparameters, and the model's internal layer architecture [@oliynyk2023know]. - **Learned Parameters:** adversaries aim to steal the learnedknowledge (weights and biases) of a model in order to replicateit. Parameter theft is generally used in conjunction with otherattacks, such as architecture theft, which lacks parameterknowledge. @@ -311,7 +307,7 @@ hyperparameters, and the model's internal layer architecture. Instead of focusing on extracting exact numerical values of the model's parameters, these attacks aim at reproducing the model's behavior (predictions and effectiveness), decision-making, and high-level -characteristics. These techniques aim at achieving similar outcomes +characteristics [@oliynyk2023know]. These techniques aim at achieving similar outcomes while allowing for internal deviations in parameters and architecture. Types of approximate behavior theft include achieving the same level of effectiveness and obtaining prediction consistency. @@ -322,7 +318,7 @@ effectiveness and obtaining prediction consistency. #### Case Study -In 2019, Tesla filed a lawsuit against self-driving car startup Zoox, +In 2018, Tesla filed a [lawsuit](https://storage.courtlistener.com/recap/gov.uscourts.nvd.131251/gov.uscourts.nvd.131251.1.0_1.pdf) against self-driving car startup [Zoox](https://zoox.com/), alleging former employees stole confidential data and trade secrets related to Tesla's autonomous driving assistance system. @@ -345,7 +341,7 @@ door for further data privacy violations. ### Data Poisoning Data poisoning is an attack where the training data is tampered with, -leading to a compromised model [?]. Attackers can modify existing +leading to a compromised model [@biggio2012poisoning]. Attackers can modify existing training examples, insert new malicious data points, or influence the data collection process. The poisoned data is labeled in such a way as to skew the model's learned behavior. This can be particularly damaging @@ -374,24 +370,24 @@ incorrect predictions or cause it to behave unpredictably. In critical applications like healthcare, such alterations can lead to significant trust and safety issues. -There are four main categories of data poisoning: +There are four main categories of data poisoning [@oprea2022poisoning]: -1) **Availability Attacks**: these attacks aim to compromise theoverall functionality of a model. They cause it to misclassify themajority of testing samples, rendering the model unusable forpractical applications. An example is label flipping, where labelsof a specific, targeted class are replaced with labels from adifferent one. +* **Availability Attacks**: these attacks aim to compromise theoverall functionality of a model. They cause it to misclassify themajority of testing samples, rendering the model unusable forpractical applications. An example is label flipping, where labelsof a specific, targeted class are replaced with labels from adifferent one. -2) **Targeted Attacks:** in contrast to availability attacks, targetedattacks aim to compromise a small number of the testing samples.So the effect is localized to a limited number of classes, whilethe model maintains the same original level of accuracy on themajority of the classes. The targeted nature of the attackrequires the attacker to possess knowledge of the model's classes.It also makes detecting these attacks more challenging. +* **Targeted Attacks:** in contrast to availability attacks, targetedattacks aim to compromise a small number of the testing samples.So the effect is localized to a limited number of classes, whilethe model maintains the same original level of accuracy on themajority of the classes. The targeted nature of the attackrequires the attacker to possess knowledge of the model's classes.It also makes detecting these attacks more challenging. -3) **Backdoor Attacks:** in these attacks, an adversary targetsspecific patterns in the data. The attacker introduces a backdoor(a malicious, hidden trigger or pattern) into the training data.For example, manipulating certain features in structured data ormanipulating a pattern of pixels at a fixed position. This causesthe model to associate the malicious pattern with specific labels.As a result, when the model encounters test samples that containthe malicious pattern, it makes false predictions. +* **Backdoor Attacks:** in these attacks, an adversary targetsspecific patterns in the data. The attacker introduces a backdoor(a malicious, hidden trigger or pattern) into the training data.For example, manipulating certain features in structured data ormanipulating a pattern of pixels at a fixed position. This causesthe model to associate the malicious pattern with specific labels.As a result, when the model encounters test samples that containthe malicious pattern, it makes false predictions. -4) **Subpopulation Attacks:** here attackers selectively choose tocompromise a subset of the testing samples, while maintainingaccuracy on the rest of the samples. You can think of theseattacks as a combination of availability and targeted attacks:performing availability attacks (performance degradation) withinthe scope of a targeted subset. Although subpopulation attacks mayseem very similar to targeted attacks, the two have cleardifferences: +* **Subpopulation Attacks:** here attackers selectively choose tocompromise a subset of the testing samples, while maintainingaccuracy on the rest of the samples. You can think of theseattacks as a combination of availability and targeted attacks:performing availability attacks (performance degradation) withinthe scope of a targeted subset. Although subpopulation attacks mayseem very similar to targeted attacks, the two have cleardifferences: -- **Scope:** while targeted attacks target a selected set of samples,subpopulation attacks target a general subpopulation with similarfeature representations. For example, in a targeted attack, anactor inserts manipulated images of a 'speed bump' warning sign(with carefully crafted perturbation or patterns), which causes anautonomous car to fail to recognize such sign and slow down. Onthe other hand, manipulating all samples of people with a Britishaccent so that a speech recognition model would misclassify aBritish person's speech is an example of a subpopulation attack. + - **Scope:** while targeted attacks target a selected set of samples,subpopulation attacks target a general subpopulation with similarfeature representations. For example, in a targeted attack, anactor inserts manipulated images of a 'speed bump' warning sign(with carefully crafted perturbation or patterns), which causes anautonomous car to fail to recognize such sign and slow down. Onthe other hand, manipulating all samples of people with a Britishaccent so that a speech recognition model would misclassify aBritish person's speech is an example of a subpopulation attack. -- **Knowledge:** while targeted attacks require a high degree offamiliarity with the data, subpopulation attacks require lessintimate knowledge in order to be effective. + - **Knowledge:** while targeted attacks require a high degree offamiliarity with the data, subpopulation attacks require lessintimate knowledge in order to be effective. #### Case Study 1 In 2017, researchers demonstrated a data poisoning attack against a -popular toxicity classification model called Perspective. This ML model +popular toxicity classification model called Perspective [@hosseini2017deceiving]. This ML model is used to detect toxic comments online. The researchers added synthetically generated toxic comments with slight @@ -414,7 +410,7 @@ poisoning is critical across application domains. #### Case Study 2 -Interestingly enough, data poisoning attacks are not always malicious. +Interestingly enough, data poisoning attacks are not always malicious [@shan2023prompt]. Nightshade, a tool developed by a team led by Professor Ben Zhao at the University of Chicago, utilizes data poisoning to help artists protect their art against scraping and copyright violations by generative AI @@ -452,13 +448,13 @@ of poisoning: Adversarial attacks are methods that aim to trick models into making incorrect predictions by providing it with specially crafted, deceptive -inputs (called adversarial examples). By adding slight perturbations to +inputs (called adversarial examples) [@parrish2023adversarial]. By adding slight perturbations to input data, adversaries can "hack" a model's pattern recognition and deceive it. These are sophisticated techniques where slight, often imperceptible alterations to input data can trick an ML model into making a wrong prediction. -In text-to-image models like DALLE or Stable Diffusion, one can generate +In text-to-image models like DALLE [@ramesh2021zero] or Stable Diffusion [@Rombach22cvpr], one can generate prompts that lead to unsafe images. For example, by altering the pixel values of an image, attackers can deceive a facial recognition system into identifying a face as a different person. @@ -471,11 +467,11 @@ the model's perceptions. Adversarial attacks fall under different scenarios: -- Whitebox Attacks: the attacker possess full knowledge of the targetmodel's internal workings, including the training data,parameters, and architecture. This comrehensive access createsfavorable conditions for an attacker to exploit the model'svulnerabilities. The attacker can take advantage of specific andsubtle weaknesses to craft effective adversarial examples. +- **Whitebox Attacks:** the attacker possess full knowledge of the targetmodel's internal workings, including the training data,parameters, and architecture. This comrehensive access createsfavorable conditions for an attacker to exploit the model'svulnerabilities. The attacker can take advantage of specific andsubtle weaknesses to craft effective adversarial examples. -- Blackbox Attacks: in contrast to whitebox attacks, in blackboxattacks, the attacker has little to no knowlede of the targetmodel. To carry out the attack, the adversarial actor needs tomake careful observations of the model's output behavior. +- **Blackbox Attacks:** in contrast to whitebox attacks, in blackboxattacks, the attacker has little to no knowlede of the targetmodel. To carry out the attack, the adversarial actor needs tomake careful observations of the model's output behavior. -- Greybox Attacks: these fall in between blackbox and whiteboxattacks. The attacker has only partial knowledge about the targetmodel's internal design. For example, the attacker could haveknowledge about training data but not the architecture orparameters. In the real-world, practical attacks fall under bothblackbox and greybox scenarios. +- **Greybox Attacks:** these fall in between blackbox and whiteboxattacks. The attacker has only partial knowledge about the targetmodel's internal design. For example, the attacker could haveknowledge about training data but not the architecture orparameters. In the real-world, practical attacks fall under bothblackbox and greybox scenarios. The landscape of machine learning models is both complex and broad, especially given their relatively recent integration into commercial @@ -486,14 +482,14 @@ strategically exploiting different aspects of different models. Below, we highlight a subset of these methods, showcasing the multifaceted nature of adversarial attacks on machine learning models: -- Generative Adversarial Networks (GANs) are deep learning models thatconsist of two networks competing against each other: a generatorand a discriminator. The generator tries to synthesize realisticdata, while the discriminator evaluates whether they are real orfake [.]. GANs can be used to craft adversarial examples. Thegenerator network is trained to produce inputs that aremisclassified by the target model. These GAN-generated images canthen be used to attack a target classifier or detection model. Thegenerator and the target model are engaged in a competitiveprocess, with the generator continually improving its ability tocreate deceptive examples, and the target model enhancing itsresistance to such examples. GANs provide a powerful framework forcrafting complex and diverse adversarial inputs, illustrating theadaptability of generative models in the adversarial landscape[.]. +* **Generative Adversarial Networks (GANs)** are deep learning models that consist of two networks competing against each other: a generatorand and a discriminator [@goodfellow2020generative]. The generator tries to synthesize realisticdata, while the discriminator evaluates whether they are real or fake. GANs can be used to craft adversarial examples. The generator network is trained to produce inputs that aremisclassified by the target model. These GAN-generated images canthen be used to attack a target classifier or detection model. The generator and the target model are engaged in a competitiveprocess, with the generator continually improving its ability tocreate deceptive examples, and the target model enhancing itsresistance to such examples. GANs provide a powerful framework forcrafting complex and diverse adversarial inputs, illustrating the adaptability of generative models in the adversarial landscape[.]. -- Transfer Learning Adversarial Attacks exploit the knowledgetransferred from a pre-trained model to a target model, enablingthe creation of adversarial examples that can deceive both models.These attacks pose a growing concern, particularly whenadversaries have knowledge of the feature extractor but lackaccess to the classification head (the part or layer that isresponsible for making the final classifications). Referred to as"headless attacks," these transferable adversarial strategiesleverage the expressive capabilities of feature extractors tocraft perturbations while being oblivious to the label space ortraining data. The existence of such attacks underscores theimportance of developing robust defenses for transfer learningapplications, especially since pre-trained models are commonlyused [.]. +* **Transfer Learning Adversarial Attacks** exploit the knowledgetransferred from a pre-trained model to a target model, enablingthe creation of adversarial examples that can deceive both models.These attacks pose a growing concern, particularly whenadversaries have knowledge of the feature extractor but lackaccess to the classification head (the part or layer that isresponsible for making the final classifications). Referred to as"headless attacks," these transferable adversarial strategiesleverage the expressive capabilities of feature extractors tocraft perturbations while being oblivious to the label space ortraining data. The existence of such attacks underscores theimportance of developing robust defenses for transfer learningapplications, especially since pre-trained models are commonlyused [@Abdelkader_2020]. #### Case Study -In 2018, researchers conducted experiments by placing small black and -white stickers on stop signs. When viewed by a normal human eye, the +In 2017, researchers conducted experiments by placing small black and +white stickers on stop signs [@eykholt2018robust]. When viewed by a normal human eye, the stickers did not obscure the sign or prevent interpretability. However, when images of the stickers stop signs were fed into standard traffic sign classification ML models, they were misclassified as speed limit @@ -514,7 +510,7 @@ self-driving cars. The attack's simplicity shows how even minor changes imperceptible to humans can lead models astray. Developers need robust defenses against such threats. -## Security Threats to ML Hardware (Lead: Elias) +## Security Threats to ML Hardware Discussing the threats to embedded ML hardware security in a structured order is useful for a clear and in-depth understanding of the potential @@ -542,15 +538,15 @@ nature of hardware security and the need for vigilance at every stage. Here's an overview table summarizing the topics: -| Order | Threat Type | Description | Relevance to Embedded ML Hardware Security | -|-------|---------------------|---------------------------------------------------------------------------------------------------|--------------------------------------------| -| 1 | Hardware Bugs | Intrinsic flaws in hardware designs that can compromise system integrity. | Foundation of hardware vulnerability. | -| 2 | Physical Attacks | Direct exploitation of hardware through physical access or manipulation. | Basic and overt threat model. | -| 3 | Fault-injection Attacks | Induction of faults to cause errors in hardware operation, leading to potential system compromise. | Systematic manipulation leading to failure. | -| 4 | Side-Channel Attacks | Exploitation of leaked information from hardware operation to extract sensitive data. | Indirect attack via environmental observation. | -| 5 | Leaky Interfaces | Vulnerabilities arising from interfaces that expose data unintentionally. | Data exposure through communication channels. | -| 6 | Counterfeit Hardware | Use of unauthorized hardware components that may have security flaws. | Compounded vulnerability issues. | -| 7 | Supply Chain Risks | Risks introduced through the lifecycle of hardware, from production to deployment. | Cumulative and multifaceted security challenges. | +| Threat Type | Description | Relevance to Embedded ML Hardware Security | +| ----------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------ | +| Hardware Bugs | Intrinsic flaws in hardware designs that can compromise system integrity. | Foundation of hardware vulnerability. | +| Physical Attacks | Direct exploitation of hardware through physical access or manipulation. | Basic and overt threat model. | +| Fault-injection Attacks | Induction of faults to cause errors in hardware operation, leading to potential system compromise. | Systematic manipulation leading to failure. | +| Side-Channel Attacks | Exploitation of leaked information from hardware operation to extract sensitive data. | Indirect attack via environmental observation. | +| Leaky Interfaces | Vulnerabilities arising from interfaces that expose data unintentionally. | Data exposure through communication channels. | +| Counterfeit Hardware | Use of unauthorized hardware components that may have security flaws. | Compounded vulnerability issues. | +| Supply Chain Risks | Risks introduced through the lifecycle of hardware, from production to deployment. | Cumulative and multifaceted security challenges. | ### Hardware Bugs @@ -565,7 +561,7 @@ processors. These bugs allow attackers to bypass the hardware barrier that separates applications, allowing a malicious program to read the memory of other programs and the operating system. -Meltdown and Spectre work by taking advantage of optimizations in modern +Meltdown [@Lipp2018meltdown] and Spectre [@Kocher2018spectre] work by taking advantage of optimizations in modern CPUs that allow them to speculatively execute instructions out of order before validity checks have completed. This reveals data that should be inaccessible, which the attack captures through side channels like @@ -592,10 +588,10 @@ healthcare industry. Here, ML systems routinely process highly sensitive patient data to help diagnose, plan treatment, and forecast outcomes. A bug in the system's hardware could lead to the unauthorized disclosure of personal health information, violating patient privacy and -contravening strict regulatory standards like the Health Insurance -Portability and Accountability Act (HIPAA). +contravening strict regulatory standards like the [Health Insurance +Portability and Accountability Act (HIPAA)](https://www.cdc.gov/phlp/publications/topic/hipaa.html) -The Meltdown and Spectre vulnerabilities are stark reminders that +The [Meltdown and Spectre](https://meltdownattack.com/) vulnerabilities are stark reminders that hardware security is not just about preventing unauthorized physical access, but also about ensuring that the hardware's architecture does not become a conduit for data exposure. Similar hardware design flaws @@ -657,12 +653,12 @@ outputs. This manipulation compromises the integrity of ML operations and can serve as a vector for further exploitation, such as system reverse engineering or security protocol bypass. Fault injection involves intentionally disrupting normal computations in a system -through external interference [?]. By precisely triggering +through external interference [@joye2012fault]. By precisely triggering computational errors, adversaries can alter program execution in ways that degrade reliability or leak sensitive information. Various physical tampering techniques can be used for fault injection. -Low voltage [?], power spikes [?], clock glitches [?], +Low voltage [@barenghi2010low], power spikes [@hutter2009contact], clock glitches [@amiel2006fault], electromagnetic pulses [?], temperate increase [?] and laser strikes [?] are common hardware attack vectors. They are precisely timed to induce faults like flipped bits or skipped instructions during key @@ -712,7 +708,7 @@ consequences. Mitigating fault injection risks necessitates a multilayer approach. Physical hardening through tamper-proof enclosures and design obfuscation helps reduce access. Lightweight anomaly detection can -identify unusual sensor inputs or erroneous model outputs [?]. +identify unusual sensor inputs or erroneous model outputs [@hsiao2023mavfi]. Error-correcting memories minimize disruption, while data encryption safeguards information. Emerging model watermarking techniques trace stolen parameters. @@ -736,8 +732,8 @@ level of security is expected. The fundamental premise of a side-channel attack is that a device's operation can inadvertently leak information. Such leaks can come from -various sources, including the electrical power a device consumes, the -electromagnetic fields it emits, the time it takes to process certain +various sources, including the electrical power a device consumes [@kocher1999differential], the +electromagnetic fields it emits [@gandolfi2001electromagnetic], the time it takes to process certain operations or even the sounds it produces. Each channel can indirectly glimpse the system's internal processes, revealing information that can compromise security. @@ -750,7 +746,7 @@ performing encryption to figure out the cryptographic key. With sophisticated statistical methods, small variations in power usage during the encryption process can be correlated with the data being processed, eventually revealing the key. Some differential analysis -attack techniques are Differential Power Analysis (DPA) [.], +attack techniques are Differential Power Analysis (DPA) [@Kocher2011Intro], Differential Electromagnetic Analysis (DEMA), and Correlation Power Analysis (CPA). @@ -796,8 +792,8 @@ After the device processes the fourth byte, it determines that there is a mismatch between the secret key and the attempted input. We notice a change in the pattern at the transition point between the fourth and fifth bytes: the voltage has gone up (the current has gone down) because -the device has stopped processing the rest of the input.\ -\ +the device has stopped processing the rest of the input. + ![](images/security_privacy/image16.png) Here is another chart of a completely wrong password. After the device @@ -811,6 +807,12 @@ encryption process and the secret key itself through analyzing different inputs and try to 'eavesdrop' on the operations that the device is performing on each byte of the input. +For additional details, please see the following video: + +{{< video https://www.youtube.com/watch?v=2iDLfuEBcs8&ab_channel=ColinO%27Flynn + title="ECED4406 - 0x501 Power Analysis Attacks" +>}} + Another example is an ML system for speech recognition, which processes voice commands to perform actions. By measuring the time it takes for the system to respond to commands or the power used during processing, @@ -830,7 +832,7 @@ of the time until an ingenious observation changed the game. MI5 agent Peter Wright proposed using a microphone to capture the subtle acoustic signatures emitted from the embassy's rotor cipher machine -during encryption. The distinct mechanical clicks of the rotors as +during encryption [@Burnet1989Spycatcher]. The distinct mechanical clicks of the rotors as operators configured them daily leaked critical information about the initial settings. This simple side channel of sound enabled MI5 to reduce the complexity of deciphering messages dramatically. This early @@ -843,8 +845,8 @@ adversaries to extract secrets indirectly through careful signal analysis. Today, acoustic cryptanalysis has evolved into attacks like keyboard -eavesdropping [?]. Electrical side channels range from power analysis -on cryptographic hardware [?] to voltage fluctuations [?] on machine +eavesdropping [@Asonov2004Keyboard]. Electrical side channels range from power analysis +on cryptographic hardware [@gnad2017voltage] to voltage fluctuations [@zhao2018fpga] on machine learning accelerators. Timing, electromagnetic emission, and even heat footprints can likewise be exploited. New and unexpected side channels often emerge as computing becomes more interconnected and miniaturized. @@ -871,13 +873,13 @@ shielding of the transmitted data. Here are some real-world examples of leaky interface issues causing security problems in IoT and embedded devices: -- Baby Monitors: Many WiFi-enabled baby monitors have been found tohave unsecured interfaces for remote access. This allowedattackers to gain live audio and video feeds from people's homes,representing a major privacy violation. +- **Baby Monitors:** Many WiFi-enabled baby monitors have been found to have unsecured interfaces for remote access. This allowed attackers to gain live audio and video feeds from people's homes, representing a major [privacy violation](https://www.fox19.com/story/25310628/hacked-baby-monitor/). -- Pacemakers: Interface vulnerabilities were discovered in somepacemakers that could allow attackers to manipulate cardiacfunctions if exploited. This presents a potential life-threateningscenario. +- **Pacemakers:** Interface vulnerabilities were discovered in some [pacemakers](https://www.fda.gov/medical-devices/medical-device-recalls/abbott-formally-known-st-jude-medical-recalls-assuritytm-and-enduritytm-pacemakers-potential) that could allow attackers to manipulate cardiacfunctions if exploited. This presents a potential life-threateningscenario. -- Smart Lightbulbs: A researcher found he could access unencrypteddata from smart lightbulbs via a debug interface, including WiFicredentials, allowing him to gain access to the connected network[?]. +- **Smart Lightbulbs:** A researcher found he could access unencrypteddata from smart lightbulbs via a debug interface, including WiFicredentials, allowing him to gain access to the connected network [@dhanjani2015abusing]. -- Smart Cars: The OBD-II diagnostic port has been shown to provide anattack vector into automotive systems if left unsecured.Researchers were able to take control of brakes and othercomponents through it [?]. +- **Smart Cars:** The OBD-II diagnostic port has been shown to provide anattack vector into automotive systems if left unsecured.Researchers were able to take control of brakes and othercomponents through it [@miller2015remote]. While the above are not directly connected with ML, consider the example of a smart home system with an embedded ML component that controls home @@ -1000,7 +1002,7 @@ pervasive risks of convoluted, opaque global supply chains. #### Case Study -In 2018, Bloomberg Businessweek published an alarming story that got +In 2018, Bloomberg Businessweek published an alarming [story](https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies) that got much attention in the tech world. The article claimed that tiny spy chips had been secretly planted on server hardware by Supermicro. Reporters said Chinese state hackers working with Supermicro could sneak @@ -1024,7 +1026,7 @@ tech companies realizing it. Companies relying too much on single manufacturers or distributors creates risk. For instance, due to the overreliance on [TSMC](https://www.tsmc.com/english) for semiconductor manufacturing, the US has invested 50 billion dollars into -the CHIPS Act. +the [CHIPS Act](https://www.whitehouse.gov/briefing-room/statements-releases/2022/08/09/fact-sheet-chips-and-science-act-will-lower-costs-create-jobs-strengthen-supply-chains-and-counter-china/). As ML moves into more critical systems, verifying hardware integrity from design through production and delivery is crucial. The reported @@ -1032,7 +1034,7 @@ Supermicro backdoor demonstrated that for ML security, we cannot take global supply chains and manufacturing for granted. We must inspect and validate hardware at every link in the chain. -## Embedded ML Hardware Security (Lead: Elizabeth) +## Embedded ML Hardware Security ### Trusted Execution Environments @@ -1106,18 +1108,7 @@ provide hardware-based security for sensitive applications: - **[Apple SecureEnclave](https://support.apple.com/guide/security/secure-enclave-sec59b0b31ff/web):**TEE for biometric data and key management on iPhones and iPads.Facilitates mobile payments. -![](images/security_privacy/image1.png) - -Figure: System on chip showing secure enclave isolated from the main -processor to provide an extra layer of security. The secure enclave has -a boot ROM to establish a hardware root of trust, an AES engine for -efficient and secure cryptographic operations, and protected memory. The -secure enclave has a mechanism to store inromation securely on attached -storage seperate from the NAND flash storage used by the application -processor and operating system. This design keeps sensitive user data -secure even when the Application Processor kernel becomes compromised. -Credit: -[https://support.apple.com/guide/security/secure-enclave-sec59b0b31ff/web](https://support.apple.com/guide/security/secure-enclave-sec59b0b31ff/web) +![System on chip showing secure enclave isolated from the main processor to provide an extra layer of security. The secure enclave has a boot ROM to establish a hardware root of trust, an AES engine for efficient and secure cryptographic operations, and protected memory. The secure enclave has a mechanism to store inromation securely on attached storage seperate from the NAND flash storage used by the application processor and operating system. This design keeps sensitive user data secure even when the Application Processor kernel becomes compromised. Credit: [Apple](https://support.apple.com/guide/security/secure-enclave-sec59b0b31ff/web.](images/security_privacy/image1.png) #### Trade-Offs @@ -1217,9 +1208,7 @@ TEE can securely operate, enabling secure operations such as cryptographic key management, secure processing, and sensitive data handling. -![](images/security_privacy/image4.png) - -Figure: The Secure Boot flow of a trusted embedded system. Source: [?] +![The Secure Boot flow of a trusted embedded system. Source: [@Rashmi2018Secure].](images/security_privacy/image4.png) #### Case Study: Apple's Face ID @@ -1435,7 +1424,7 @@ challenge-response mechanism can be used to generate keys securely and identifiers tied to the specific hardware, perform device authentication, or securely store secrets. For example, a key derived from a PUF will only work on that device and cannot be cloned or -extracted even with physical access or full reverse engineering. [?] +extracted even with physical access or full reverse engineering [@Gao2020Physical]. #### Benefits @@ -1493,30 +1482,9 @@ that utilize ML for processing sensor data can employ PUFs to secure communication between devices and prevent the execution of ML models on counterfeit hardware. -![](images/security_privacy/image2.png) - -Figure: PUF basics. a) A PUF exploits intrinsic random variation at the -microscale or nanoscale. Such random variation resulting from -uncontrollable fabrication processes can be conceptually thought as a -unique physical 'fingerprint' of a hardware device. b) Optical PUF. -Under illumination from a given angle/polarization, complex interference -occurs within an inhomogeneous transparent plastic token. Then, a two -dimensional (2D) speckle pattern is recorded using a charge-coupled -device camera. The angle/polarization is treated as the challenge while -the 2D speckle pattern is the response. c) APUF. Consisting of multiple -stages (1 to k), the challenge bits (C) select two theoretically -identical but practically unequal delay paths at each stage. At the end -of the APUF, an arbiter judges whether the top path is faster or not, -and reacts with a '1' or '0' response (r; ref. 10). A challenge bit of -'0' means two signals pass a given stage in parallel, whilst '1' means -two signals cross over. d) SRAM PUF. The mismatch of threshold voltage -Vth of the transistors determines the response124. For example, if the -is slightly smaller than, at power-up, the transistor M1 starts -conducting before M2, thus, the logic state at point A = '1'. This in -turn prevents M2 switching on. As a result, the SRAM power-up state -prefers to be '1' (point A = '1', point B = '0'), which is the response, -while the address of the memory cell is the challenge. WL, word line; -BL, bit line. Source: [?] +![PUF basics. a) A PUF exploits intrinsic random variation at the microscale or nanoscale. Such random variation resulting from uncontrollable fabrication processes can be conceptually thought as a unique physical 'fingerprint' of a hardware device. b) Optical PUF. Under illumination from a given angle/polarization, complex interference occurs within an inhomogeneous transparent plastic token. Then, a two dimensional (2D) speckle pattern is recorded using a charge-coupled device camera. The angle/polarization is treated as the challenge while the 2D speckle pattern is the response. c) APUF. Consisting of multiple stages (1 to k), the challenge bits (C) select two theoretically identical but practically unequal delay paths at each stage. At the end of the APUF, an arbiter judges whether the top path is faster or not, and reacts with a '1' or '0' response (r; ref. 10). A challenge bit of '0' means two signals pass a given stage in parallel, whilst '1' means two signals cross over. d) SRAM PUF. The mismatch of threshold voltage Vth of the transistors determines the response124. For example, if the is slightly smaller than, at power-up, the transistor M1 starts conducting before M2, thus, the logic state at point A = '1'. This in turn prevents M2 switching on. As a result, the SRAM power-up state prefers to be '1' (point A = '1', point B = '0'), which is the response, while the address of the memory cell is the challenge. WL, word line; BL, bit line. Source: [@Gao2020Physical].](images/security_privacy/image2.png) + + #### Challenges @@ -1703,7 +1671,7 @@ synthetic data generation allow for deriving useful insights from less raw user data. Performing data flow mapping and impact assessments help identify opportunities to minimize raw data usage. -Methodologies like Privacy by Design consider such minimization early in +Methodologies like Privacy by Design [@cavoukian2009privacy] consider such minimization early in system architecture. Regulations like GDPR also mandate data minimization principles. With a multilayered approach across legal, technical, and process realms, data minimization limits risks in @@ -1711,7 +1679,7 @@ embedded ML products. #### Case Study - Performance Based Data Minimization -Performance based data minimization focuses on expanding upon the third +Performance based data minimization [@Biega2020Oper] focuses on expanding upon the third category of data minimization mentioned above, namely *limitation*. It specifically defines the robustness of model results on a given dataset by certain performance metrics, such that data should not be @@ -1797,9 +1765,9 @@ While ChatGPT has instituted protections to prevent people from accessing private and ethically questionable information, several individuals have successfully been able to bypass these protections through prompt injection attacks, and other security attacks. As -demonstrated below, users have been able to bypass ChatGPT protections +demonstrated below, users have been able to bypass ChatGPT protections to mimic the tone of a "deceased grandmother" to learn how to bypass a -web application firewall. +web application firewall [@Gupta2023ChatGPT]. ![](images/security_privacy/image6.png) @@ -1878,7 +1846,7 @@ of modern ML systems. #### Core Idea Differential Privacy (DP) is a framework for quantifying and managing -the privacy of individuals in a dataset. It provides a mathematical +the privacy of individuals in a dataset [@Dwork2006Theory]. It provides a mathematical guarantee that the privacy of individuals in the dataset will not be compromised, regardless of any additional knowledge an attacker may possess. The core idea of differential privacy is that the outcome of @@ -1927,7 +1895,7 @@ between privacy and accuracy. To illustrate the trade-off of privacy and accuracy in (ϵ, 𝛿)-differential privacy, the following graphs show the results on accuracy for different noise levels on the MNIST dataset, a large -dataset of handwritten digits. An increasing delta value relaxes the +dataset of handwritten digits [@abadi2016deep]. An increasing delta value relaxes the privacy guarantee, so the noise level can be reduced. Since the data will retain many of its original characteristics, accuracy simultaneously increases with drawbacks on privacy preservation. This @@ -2000,9 +1968,9 @@ research area. #### Case Study -Apple's implementation of differential privacy in iOS and MacOS -provides a prominent real-world example of how differential privacy can -be deployed at large scale. Apple wanted to collect aggregated usage +[Apple's implementation of differential privacy](https://machinelearning.apple.com/research/learning-with-privacy-at-scale#DMNS06) in iOS and MacOS +provides a prominent real-world example of [how differential privacy can +be deployed at large scale](https://docs-assets.developer.apple.com/ml-research/papers/learning-with-privacy-at-scale.pdf). Apple wanted to collect aggregated usage statistics across their ecosystem to improve products and services, but aimed to do so without compromising individual user privacy. @@ -2059,7 +2027,7 @@ transferred, or compiled together. This privacy-preserving approach allows jointly developing ML models without centralizing the potentially sensitive training data in one place. -The figure below illustrates the FL lifecycle. The training data always +The figure below from [@MAL-083] illustrates the FL lifecycle. The training data always remains on the client data, the model repeatedly is sent back and forth between individual devices and server for local updates and compiling the global model, respectively. @@ -2101,7 +2069,7 @@ patient data with each other. Here's how Federated Learning can help. There are several system performance-related aspects of FL in machine learning systems. It would be wise to understand these trade-offs -because there is no "free lunch" for preserving privacy through FL. +because there is no "free lunch" for preserving privacy through FL [@Li2020Federated]. **Communication Overhead and Network Constraints:** In FL, one of the most significant challenges is managing the communication overhead. This @@ -2233,13 +2201,13 @@ explicitly unlearn data from a model. One type of approach that researchers have adopted includes adjusting the model loss function to explicitly treat the losses of the "forget set" (data to be unlearned) and the "retain set" (remaining data that should still be remembered) -differently. +differently [@tarun2023deep; @khan2021knowledgeadaptation]. #### Case Study Some researchers demonstrate a real life example of machine unlearning approaches applied to SOTA machine learning models through training an -LLM, Llama2-7b, to unlearn any references to Harry Potter. Though this +LLM, Llama2-7b, to unlearn any references to Harry Potter [@eldan2023whos]. Though this model took 184K GPU-hours to pretrain, it only took 1 GPU hour of fine tuning to erase the model's ability to generate or recall Harry Potter-related content, without noticeably compromising the accuracy of diff --git a/references.bib b/references.bib index 659a4af8..d820b3d8 100644 --- a/references.bib +++ b/references.bib @@ -1,3 +1,50 @@ +@article{oprea2022poisoning, + title={Poisoning Attacks Against Machine Learning: Can Machine Learning Be Trustworthy?}, + author={Oprea, Alina and Singhal, Anoop and Vassilev, Apostol}, + journal={Computer}, + volume={55}, + number={11}, + pages={94--99}, + year={2022}, + publisher={IEEE} +} + +@article{goodfellow2020generative, + title={Generative adversarial networks}, + author={Goodfellow, Ian and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua}, + journal={Communications of the ACM}, + volume={63}, + number={11}, + pages={139--144}, + year={2020}, + publisher={ACM New York, NY, USA} +} + + +@conference{Rombach22cvpr, +title = {High-Resolution Image Synthesis with Latent Diffusion Models}, +author = {Robin Rombach and Andreas Blattmann and Dominik Lorenz and Patrick Esser and Björn Ommer}, +url = {https://github.com/CompVis/latent-diffusionhttps://arxiv.org/abs/2112.10752}, +year = {2022}, +booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, +} + + +@inproceedings{ramesh2021zero, + title={Zero-shot text-to-image generation}, + author={Ramesh, Aditya and Pavlov, Mikhail and Goh, Gabriel and Gray, Scott and Voss, Chelsea and Radford, Alec and Chen, Mark and Sutskever, Ilya}, + booktitle={International Conference on Machine Learning}, + pages={8821--8831}, + year={2021}, + organization={PMLR} +} + +@article{shan2023prompt, + title={Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models}, + author={Shan, Shawn and Ding, Wenxin and Passananti, Josephine and Zheng, Haitao and Zhao, Ben Y}, + journal={arXiv preprint arXiv:2310.13828}, + year={2023} +} @article{soufleri2023synthetic, author = {Efstathia Soufleri and Gobinda Saha and Kaushik Roy},