diff --git a/contents/robust_ai/robust_ai.qmd b/contents/robust_ai/robust_ai.qmd index 0dd8aa9b..f78f1fbe 100644 --- a/contents/robust_ai/robust_ai.qmd +++ b/contents/robust_ai/robust_ai.qmd @@ -62,9 +62,9 @@ Here are some real-world examples of cases where faults in hardware or software In February 2017, Amazon Web Services (AWS) experienced [a significant outage](https://aws.amazon.com/message/41926/) due to human error during maintenance. An engineer inadvertently entered an incorrect command, causing many servers to be taken offline. This outage disrupted many AWS services, including Amazon's AI-powered assistant, Alexa. As a result, Alexa-powered devices, such as Amazon Echo and third-party products using Alexa Voice Service, could not respond to user requests for several hours. This incident highlights the potential impact of human errors on cloud-based ML systems and the need for robust maintenance procedures and failsafe mechanisms. -In another example [@dixit2021silent], Facebook encountered a silent data corruption issue within its distributed querying infrastructure, as shown in [@fig-sdc-example](#8owvod923jax). Facebook's infrastructure includes a querying system that fetches and executes SQL and SQL-like queries across multiple datasets using frameworks like Presto, Hive, and Spark. One of the applications that utilized this querying infrastructure was a compression application to reduce the footprint of data stores. In this compression application, files were compressed when not being read and decompressed when a read request was made. Before decompression, the file size was checked to ensure it was greater than zero, indicating a valid compressed file with contents. +In another example [@dixit2021silent], Facebook encountered a silent data corruption issue within its distributed querying infrastructure, as shown in @fig-sdc-example. Facebook's infrastructure includes a querying system that fetches and executes SQL and SQL-like queries across multiple datasets using frameworks like Presto, Hive, and Spark. One of the applications that utilized this querying infrastructure was a compression application to reduce the footprint of data stores. In this compression application, files were compressed when not being read and decompressed when a read request was made. Before decompression, the file size was checked to ensure it was greater than zero, indicating a valid compressed file with contents. -![Silent data corruption in database applications (Source: [Facebook](https://arxiv.org/pdf/2102.11245))](./images/png/sdc_example.png){#fig-sdc-example} +![Silent data corruption in database applications. Credit [Facebook.](https://arxiv.org/pdf/2102.11245)](./images/png/sdc_example.png){#fig-sdc-example} However, in one instance, when the file size was being computed for a valid non-zero-sized file, the decompression algorithm invoked a power function from the Scala library. Unexpectedly, the Scala function returned a zero size value for the file despite having a known non-zero decompressed size. As a result, the decompression was not performed, and the file was not written to the output database. This issue manifested sporadically, with some occurrences of the same file size computation returning the correct non-zero value. @@ -78,7 +78,7 @@ Regarding examples of faults and errors in edge ML systems, one area that has ga In May 2016, a fatal accident occurred when a Tesla Model S operating on Autopilot crashed into a white semi-trailer truck crossing the highway. The Autopilot system, which relied on computer vision and machine learning algorithms, failed to recognize the white trailer against a bright sky background. The driver, who was reportedly watching a movie when the crash, did not intervene in time, and the vehicle collided with the trailer at full speed. This incident raised concerns about the limitations of AI-based perception systems and the need for robust failsafe mechanisms in autonomous vehicles. It also highlighted the importance of driver awareness and the need for clear guidelines on using semi-autonomous driving features, as shown in [@fig-tesla-example](#tckwqf2ctxw). -![Tesla in the fatal California crash was on Autopilot (Source: [BBC News](https://www.bbc.com/news/world-us-canada-43604440))](./images/png/tesla_example.jpg){#fig-tesla-example} +![Tesla in the fatal California crash was on Autopilot. Credit: [BBC News.](https://www.bbc.com/news/world-us-canada-43604440)](./images/png/tesla_example.jpg){#fig-tesla-example} In March 2018, an Uber self-driving test vehicle struck and killed a pedestrian crossing the street in Tempe, Arizona. The incident was caused by a software flaw in the vehicle's object recognition system, which failed to identify the pedestrians appropriately to avoid them as obstacles. The safety driver, who was supposed to monitor the vehicle's operation and intervene if necessary, was found distracted during the crash. [This incident](https://money.cnn.com/2018/03/19/technology/uber-autonomous-car-fatal-crash/index.html?iid=EL) led to widespread scrutiny of Uber's self-driving program and raised questions about the readiness of autonomous vehicle technology for public roads. It also emphasized the need for rigorous testing, validation, and safety measures in developing and deploying AI-based self-driving systems. @@ -90,7 +90,7 @@ Embedded systems, which often operate in resource-constrained environments and s Let's consider a few examples, starting with outer space exploration. NASA's Mars Polar Lander mission in 1999 suffered [a catastrophic failure](https://spaceref.com/uncategorized/nasa-reveals-probable-cause-of-mars-polar-lander-and-deep-space-2-mission-failures/) due to a software error in the touchdown detection system ([@fig-nasa-example](#e3z8hq3qpwn4)). The spacecraft's onboard software mistakenly interpreted the noise from the deployment of its landing legs as a sign that it had touched down on the Martian surface. As a result, the spacecraft prematurely shut down its engines, causing it to crash into the surface. This incident highlights the critical importance of robust software design and extensive testing in embedded systems, especially those operating in remote and unforgiving environments. As AI capabilities are integrated into future space missions, ensuring these systems' reliability and fault tolerance will be paramount to mission success. -![NASA's Failed Mars Polar Lander mission in 1999 cost over \$200M (Source: [SlashGear](https://www.slashgear.com/1094840/nasas-failed-mars-missions-that-cost-over-200-million/))](./images/png/nasa_example.png){#fig-nasa-example} +![NASA's Failed Mars Polar Lander mission in 1999 cost over \$200M. Credit: [SlashGear.](https://www.slashgear.com/1094840/nasas-failed-mars-missions-that-cost-over-200-million/)](./images/png/nasa_example.png){#fig-nasa-example} Back on earth, in 2015, a Boeing 787 Dreamliner experienced a complete electrical shutdown during a flight due to a software bug in its generator control units. The bug caused the generator control units to enter a failsafe mode, cutting power to the aircraft's electrical systems and forcing an emergency landing. [This incident](https://www.engineering.com/story/vzrxw) underscores the potential for software faults to have severe consequences in complex embedded systems like aircraft. As AI technologies are increasingly applied in aviation, such as in autonomous flight systems and predictive maintenance, ensuring the robustness and reliability of these systems will be critical to passenger safety. @@ -126,7 +126,7 @@ Transient faults are characterized by their short duration and non-permanent nat Transient faults can be attributed to various external factors. One common cause is cosmic rays, high-energy particles originating from outer space. When these particles strike sensitive areas of the hardware, such as memory cells or transistors, they can induce charge disturbances that alter the stored or transmitted data. This is illustrated in [@fig-transient-fault](#9jd0z5evi3fa). Another cause of transient faults is [electromagnetic interference (EMI)](https://www.trentonsystems.com/en-us/resource-hub/blog/what-is-electromagnetic-interference) from nearby devices or power fluctuations. EMI can couple with the circuits and cause voltage spikes or glitches that temporarily disrupt the normal operation of the hardware. -![Mechanism of Hardware Transient Fault Occurrence (Source: [NTT](https://group.ntt/en/newsrelease/2018/11/22/181122a.html))](./images/png/transient_fault.png){#fig-transient-fault} +![Mechanism of Hardware Transient Fault Occurrence. Credit: [NTT.](https://group.ntt/en/newsrelease/2018/11/22/181122a.html)](./images/png/transient_fault.png){#fig-transient-fault} #### Mechanisms of Transient Faults @@ -158,7 +158,7 @@ The FDIV bug occurred due to an error in the lookup table used by the division u Although the error was small, it could compound over many division operations, leading to significant inaccuracies in mathematical calculations. The impact of the FDIV bug was significant, especially for applications that relied heavily on precise floating-point division, such as scientific simulations, financial calculations, and computer-aided design. The bug led to incorrect results, which could have severe consequences in fields like finance or engineering. -![Intel Pentium processor with the FDIV permanent fault. The triangular regions are where erroneous calculations occurred. (Source: [Byte Magazine](https://www.halfhill.com/byte/1995-3_truth.html))](./images/png/permanent_fault.png){#fig-permanent-fault} +![Intel Pentium processor with the FDIV permanent fault. Credit: [Byte Magazine.](https://www.halfhill.com/byte/1995-3_truth.html)](./images/png/permanent_fault.png){#fig-permanent-fault} The Intel FDIV bug is a cautionary tale for the potential impact of permanent faults on ML systems. In the context of ML, permanent faults in hardware components can lead to incorrect computations, affecting the accuracy and reliability of the models. For example, if an ML system relies on a processor with a faulty floating-point unit, similar to the Intel FDIV bug, it could introduce errors in the calculations performed during training or inference. @@ -176,7 +176,7 @@ On the other hand, [wear-out mechanisms](https://semiengineering.com/what-causes Permanent faults can manifest through various mechanisms, depending on the nature and location of the fault. Stuck-at faults [@seong2010safer] are common permanent faults where a signal or memory cell remains fixed at a particular value (either 0 or 1) regardless of the inputs, as illustrated in [@fig-stuck-fault](#ahtmh1s1mxgf). -![Stuck-at Fault Model in Digital Circuits (Source: [Accendo Reliability](https://accendoreliability.com/digital-circuits-stuck-fault-model/))](./images/png/stuck_fault.png){#fig-stuck-fault} +![Stuck-at Fault Model in Digital Circuits. Credit: [Accendo Reliability.](https://accendoreliability.com/digital-circuits-stuck-fault-model/)](./images/png/stuck_fault.png){#fig-stuck-fault} Stuck-at faults can occur in logic gates, memory cells, or interconnects, causing incorrect computations or data corruption. Another mechanism is device failures, where a component, such as a transistor or a memory cell, completely ceases to function. This can be due to manufacturing defects or severe wear-out. Bridging faults occur when two or more signal lines are unintentionally connected, causing short circuits or incorrect logic behavior. @@ -194,7 +194,7 @@ Designing ML systems with fault tolerance in mind is crucial to ensure their rel Intermittent faults are hardware faults that occur sporadically and unpredictably in a system. An example is illustrated in [@fig-intermittent-fault](#kix.1c0l0udn3cp7), where cracks in the material can introduce increased resistance in circuitry. These faults are particularly challenging to detect and diagnose because they appear and disappear intermittently, making it difficult to reproduce and isolate the root cause. Intermittent faults can lead to system instability, data corruption, and performance degradation. -![Increased resistance due to an intermittent fault -- crack between copper bump and package solder (Source: [Constantinescu](https://ieeexplore.ieee.org/document/4925824))](./images/png/intermittent_fault.png){#fig-intermittent-fault} +![Increased resistance due to an intermittent fault. Credit: [Constantinescu.](https://ieeexplore.ieee.org/document/4925824)](./images/png/intermittent_fault.png){#fig-intermittent-fault} #### Definition and Characteristics @@ -210,7 +210,7 @@ Manufacturing defects or process variations can also introduce intermittent faul Environmental factors, such as temperature fluctuations, humidity, or vibrations, can trigger intermittent faults by altering the electrical characteristics of the components. Loose or degraded connections, such as those in connectors or printed circuit boards, can cause intermittent faults. -![Residue induced intermittent fault in a DRAM chip (Source: [Hynix Semiconductor](https://ieeexplore.ieee.org/document/4925824))](./images/png/intermittent_fault_dram.png){#fig-intermittent-fault-dram} +![Residue induced intermittent fault in a DRAM chip. Credit: [Hynix Semiconductor.](https://ieeexplore.ieee.org/document/4925824)](./images/png/intermittent_fault_dram.png){#fig-intermittent-fault-dram} #### Mechanisms of Intermittent Faults @@ -244,7 +244,7 @@ During the BIST process, predefined test patterns are applied to the processor's **Error detection codes:** Error detection codes are widely used to detect data storage and transmission errors [@hamming1950error]. These codes add redundant bits to the original data, allowing the detection of bit errors. Example: Parity checks are a simple form of error detection code shown in [@fig-parity](#kix.2vxlbeehnemj). In a single-bit parity scheme, an extra bit is appended to each data word, making the number of 1s in the word even (even parity) or odd (odd parity). -![Parity bit example (Source: [Computer Hope](https://www.computerhope.com/jargon/p/paritybi.htm))](./images/png/parity.png){#fig-parity} +![Parity bit example. Credit: [Computer Hope.](https://www.computerhope.com/jargon/p/paritybi.htm)](./images/png/parity.png){#fig-parity} When reading the data, the parity is checked, and if it doesn't match the expected value, an error is detected. More advanced error detection codes, such as cyclic redundancy checks (CRC), calculate a checksum based on the data and append it to the message. The checksum is recalculated at the receiving end and compared with the transmitted checksum to detect errors. Error-correcting code (ECC) memory modules, commonly used in servers and critical systems, employ advanced error detection and correction codes to detect and correct single-bit or multi-bit errors in memory. @@ -254,7 +254,7 @@ In a TMR system, three identical instances of a hardware component, such as a pr Tesla's self-driving computers employ a redundant hardware architecture to ensure the safety and reliability of critical functions, such as perception, decision-making, and vehicle control, as shown in [@fig-tesla-dmr](#kix.nsc1yczcug9r). One key component of this architecture is using dual modular redundancy (DMR) in the car's onboard computer systems. -![Tesla full self-driving computer with dual redundant SoCs (Source: [Tesla](https://old.hotchips.org/hc31/HC31_2.3_Tesla_Hotchips_ppt_Final_0817.pdf))](./images/png/tesla_dmr.png){#fig-tesla-dmr} +![Tesla full self-driving computer with dual redundant SoCs. Credit: [Tesla.](https://old.hotchips.org/hc31/HC31_2.3_Tesla_Hotchips_ppt_Final_0817.pdf)](./images/png/tesla_dmr.png){#fig-tesla-dmr} In Tesla's DMR implementation, two identical hardware units, often called "redundant computers" or "redundant control units," perform the same computations in parallel [@bannon2019computer]. Each unit independently processes sensor data, executes perception and decision-making algorithms, and generates control commands for the vehicle's actuators (e.g., steering, acceleration, and braking). @@ -272,7 +272,7 @@ The use of DMR in Tesla's self-driving computer highlights the importance of har **Watchdog timers:** Watchdog timers are hardware components that monitor the execution of critical tasks or processes [@pont2002using]. They are commonly used to detect and recover from software or hardware faults that cause a system to become unresponsive or stuck in an infinite loop. In an embedded system, a watchdog timer can be configured to monitor the execution of the main control loop, as illustrated in [@fig-watchdog](#3l259jcz0lli). The software periodically resets the watchdog timer to indicate that it functions correctly. Suppose the software fails to reset the timer within a specified time limit (timeout period). In that case, the watchdog timer assumes that the system has encountered a fault and triggers a predefined recovery action, such as resetting the system or switching to a backup component. Watchdog timers are widely used in automotive electronics, industrial control systems, and other safety-critical applications to ensure the timely detection and recovery from faults. -![Watchdog timer example in detecting MCU faults (Source: [Ablic](https://www.ablic.com/en/semicon/products/automotive/automotive-watchdog-timer/intro/))](./images/png/watchdog.png){#fig-watchdog} +![Watchdog timer example in detecting MCU faults. Credit: [Ablic.](https://www.ablic.com/en/semicon/products/automotive/automotive-watchdog-timer/intro/)](./images/png/watchdog.png){#fig-watchdog} ##### Software-level fault detection @@ -280,21 +280,21 @@ Software-level fault detection techniques rely on software algorithms and monito **Runtime monitoring and anomaly detection:** Runtime monitoring involves continuously observing the behavior of the system and its components during execution [@francalanza2017foundation]. It helps detect anomalies, errors, or unexpected behavior that may indicate the presence of faults. For example, consider an ML-based image classification system deployed in a self-driving car. Runtime monitoring can be implemented to track the classification model's performance and behavior [@mahmoud2021issre]. -Anomaly detection algorithms can be applied to the model's predictions or intermediate layer activations, such as statistical outlier detection or machine learning-based approaches (e.g., One-Class SVM or Autoencoders) [@chandola2009anomaly]. [@fig-ad](#a0u8fu59ui0r) shows example of anomaly detection. Suppose the monitoring system detects a significant deviation from the expected patterns, such as a sudden drop in classification accuracy or out-of-distribution samples. In that case, it can raise an alert indicating a potential fault in the model or the input data pipeline. This early detection allows for timely intervention and fault mitigation strategies to be applied. +Anomaly detection algorithms can be applied to the model's predictions or intermediate layer activations, such as statistical outlier detection or machine learning-based approaches (e.g., One-Class SVM or Autoencoders) [@chandola2009anomaly]. [@fig-ad](#a0u8fu59ui0r) shows examples of anomaly detection: (a) Fully supervised anomaly detection, (b) normal-only anomaly detection, (c, d, e) semi-supervised anomaly detection, (f) unsupervised anomaly detection. Suppose the monitoring system detects a significant deviation from the expected patterns, such as a sudden drop in classification accuracy or out-of-distribution samples. In that case, it can raise an alert indicating a potential fault in the model or the input data pipeline. This early detection allows for timely intervention and fault mitigation strategies to be applied. -![Examples of anomaly detection. (a) Fully supervised anomaly detection, (b) normal-only anomaly detection, (c, d, e) semi-supervised anomaly detection, (f) unsupervised anomaly detection (Source: [Google](https://www.google.com/url?sa=i&url=http%3A%2F%2Fresearch.google%2Fblog%2Funsupervised-and-semi-supervised-anomaly-detection-with-data-centric-ml%2F&psig=AOvVaw1p9owe13lxfZogUHTZnxrj&ust=1714877457779000&source=images&cd=vfe&opi=89978449&ved=0CBIQjRxqFwoTCIjMmMP-8oUDFQAAAAAdAAAAABAE))](./images/png/ad.png){#fig-ad} +![Examples of anomaly detection. Credit: [Google.](https://www.google.com/url?sa=i&url=http%3A%2F%2Fresearch.google%2Fblog%2Funsupervised-and-semi-supervised-anomaly-detection-with-data-centric-ml%2F&psig=AOvVaw1p9owe13lxfZogUHTZnxrj&ust=1714877457779000&source=images&cd=vfe&opi=89978449&ved=0CBIQjRxqFwoTCIjMmMP-8oUDFQAAAAAdAAAAABAE)](./images/png/ad.png){#fig-ad} **Consistency checks and data validation:** Consistency checks and data validation techniques ensure data integrity and correctness at different processing stages in an ML system [@lindholm2019data]. These checks help detect data corruption, inconsistencies, or errors that may propagate and affect the system's behavior. Example: In a distributed ML system where multiple nodes collaborate to train a model, consistency checks can be implemented to validate the integrity of the shared model parameters. Each node can compute a checksum or hash of the model parameters before and after the training iteration, as shown in @fig-ad. Any inconsistencies or data corruption can be detected by comparing the checksums across nodes. Additionally, range checks can be applied to the input data and model outputs to ensure they fall within expected bounds. For instance, if an autonomous vehicle's perception system detects an object with unrealistic dimensions or velocities, it can indicate a fault in the sensor data or the perception algorithms [@wan2023vpp]. **Heartbeat and timeout mechanisms:** Heartbeat mechanisms and timeouts are commonly used to detect faults in distributed systems and ensure the liveness and responsiveness of components [@kawazoe1997heartbeat]. These are quite similar to the watchdog timers found in hardware. For example, in a distributed ML system, where multiple nodes collaborate to perform tasks such as data preprocessing, model training, or inference, heartbeat mechanisms can be implemented to monitor the health and availability of each node. Each node periodically sends a heartbeat message to a central coordinator or its peer nodes, indicating its status and availability. Suppose a node fails to send a heartbeat within a specified timeout period, as shown in [@fig-heartbeat](#ojufkz2g56e). In that case, it is considered faulty, and appropriate actions can be taken, such as redistributing the workload or initiating a failover mechanism. Timeouts can also be used to detect and handle hanging or unresponsive components. For example, if a data loading process exceeds a predefined timeout threshold, it may indicate a fault in the data pipeline, and the system can take corrective measures. -![Heartbeat messages in distributed systems (Source: [GeeksforGeeks](https://www.geeksforgeeks.org/what-are-heartbeat-messages/))](./images/png/heartbeat.png){#fig-heartbeat} +![Heartbeat messages in distributed systems. Credit: [GeeksforGeeks](https://www.geeksforgeeks.org/what-are-heartbeat-messages/)](./images/png/heartbeat.png){#fig-heartbeat} **Software-implemented fault tolerance (SIFT) techniques:** SIFT techniques introduce redundancy and fault detection mechanisms at the software level to enhance the reliability and fault tolerance of the system [@reis2005swift]. Example: N-version programming is a SIFT technique where multiple functionally equivalent software component versions are developed independently by different teams. This can be applied to critical components such as the model inference engine in an ML system. Multiple versions of the inference engine can be executed in parallel, and their outputs can be compared for consistency. It is considered the correct result if most versions produce the same output. If there is a discrepancy, it indicates a potential fault in one or more versions, and appropriate error-handling mechanisms can be triggered. Another example is using software-based error correction codes, such as Reed-Solomon codes [@plank1997tutorial], to detect and correct errors in data storage or transmission, as shown in [@fig-Reed-Solomon](#kjmtegsny44z). These codes add redundancy to the data, enabling detecting and correcting certain errors and enhancing the system's fault tolerance. -![n-bits representation of the Reed-Solomon codes (Source: [GeeksforGeeks](https://www.geeksforgeeks.org/what-is-reed-solomon-code/))](./images/png/Reed-Solomon.png){#fig-Reed-Solomon} +![n-bits representation of the Reed-Solomon codes. Credit: [GeeksforGeeks.](https://www.geeksforgeeks.org/what-is-reed-solomon-code/)](./images/png/Reed-Solomon.png){#fig-Reed-Solomon} :::{#exr-ad .callout-exercise collapse="true"} @@ -329,7 +329,7 @@ In this Colab, play the role of an AI fault detective! You'll build an autoencod Adversarial attacks aim to trick models into making incorrect predictions by providing them with specially crafted, deceptive inputs (called adversarial examples) [@parrish2023adversarial]. By adding slight perturbations to input data, adversaries can \"hack\" a model's pattern recognition and deceive it. These are sophisticated techniques where slight, often imperceptible alterations to input data can trick an ML model into making a wrong prediction, as shown in [@fig-adversarial-attack-noise-example]. -![A small adversarial noise added to the original image can make the neural network classify the image as a Guacamole instead of an Egyptian cat (Source: [Sutanto](https://www.mdpi.com/2079-9292/10/1/52))](./images/png/adversarial_attack_detection.png){#fig-adversarial-attack-noise-example} +![Adversarial noise attack. Credit: [Sutanto.](https://www.mdpi.com/2079-9292/10/1/52)](./images/png/adversarial_attack_detection.png){#fig-adversarial-attack-noise-example} One can generate prompts that lead to unsafe images in text-to-image models like DALLE [@ramesh2021zero] or Stable Diffusion [@rombach2022highresolution]. For example, by altering the pixel values of an image, attackers can deceive a facial recognition system into identifying a face as a different person. @@ -351,7 +351,7 @@ The landscape of machine learning models is complex and broad, especially given #### Mechanisms of Adversarial Attacks -![Gradient-Based Attacks (Source: [Ivezic](https://defence.ai/ai-security/gradient-based-attacks/))](./images/png/gradient_attack.png){#fig-gradient-attack} +![Gradient-Based Attacks. Credit: [Ivezic.](https://defence.ai/ai-security/gradient-based-attacks/)](./images/png/gradient_attack.png){#fig-gradient-attack} **Gradient-based Attacks** @@ -404,9 +404,9 @@ One striking example of the impact of adversarial attacks was demonstrated by re This demonstration shed light on the alarming potential of simple adversarial stickers to trick ML systems into misreading critical road signs. The implications of such attacks in the real world are significant, particularly in the context of autonomous vehicles. If deployed on actual roads, these adversarial stickers could cause self-driving cars to misinterpret stop signs as speed limits, leading to dangerous situations, as shown in [@fig-graffiti]. Researchers warned that this could result in rolling stops or unintended acceleration into intersections, endangering public safety. -![Adversarial example generation applied to GoogLeNet (Szegedy et al., 2014a) on ImageNet (Source: [Goodfellow](https://arxiv.org/abs/1412.6572))](./images/png/adversarial_googlenet.png){#fig-adversarial-googlenet} +![Adversarial example generation applied to GoogLeNet (Szegedy et al., 2014a) on ImageNet. Credit: [Goodfellow.](https://arxiv.org/abs/1412.6572)](./images/png/adversarial_googlenet.png){#fig-adversarial-googlenet} -![Graffiti on a stop sign tricked a self-driving car into thinking it was a 45 mph speed limit sign (Source: [Eykholt](https://arxiv.org/abs/1707.08945))](./images/png/graffiti.png){#fig-graffiti} +![Graffiti on a stop sign. Credit: [Eykholt.](https://arxiv.org/abs/1707.08945)](./images/png/graffiti.png){#fig-graffiti} The case study of the adversarial stickers on stop signs provides a concrete illustration of how adversarial examples exploit how ML models recognize patterns. By subtly manipulating the input data in ways that are invisible to humans, attackers can induce incorrect predictions and create serious risks, especially in safety-critical applications like autonomous vehicles. The attack's simplicity highlights the vulnerability of ML models to even minor changes in the input, emphasizing the need for robust defenses against such threats. @@ -439,7 +439,7 @@ Think you can outsmart an AI? In this Colab, learn how to trick image classifica Data poisoning is an attack where the training data is tampered with, leading to a compromised model [@biggio2012poisoning], as shown in [@fig-poisoning-example]. Attackers can modify existing training examples, insert new malicious data points, or influence the data collection process. The poisoned data is labeled in such a way as to skew the model's learned behavior. This can be particularly damaging in applications where ML models make automated decisions based on learned patterns. Beyond training sets, poisoning tests, and validation data can allow adversaries to boost reported model performance artificially. -![NightShade's poisoning effects on Stable Diffusion (Source: [TOMÉ](https://telefonicatech.com/en/blog/attacks-on-artificial-intelligence-iii-data-poisoning))](./images/png/poisoning_example.png){#fig-poisoning-example} +![NightShade's poisoning effects on Stable Diffusion. Credit: [TOMÉ.](https://telefonicatech.com/en/blog/attacks-on-artificial-intelligence-iii-data-poisoning)](./images/png/poisoning_example.png){#fig-poisoning-example} The process usually involves the following steps: @@ -484,7 +484,7 @@ Data poisoning attacks can be carried out through various mechanisms, exploiting Each of these mechanisms presents unique challenges and requires different mitigation strategies. For example, detecting label manipulation may involve analyzing the distribution of labels and identifying anomalies [@zhou2018learning], while preventing feature manipulation may require secure data preprocessing and anomaly detection techniques [@carta2020local]. Defending against insider threats may involve strict access control policies and monitoring of data access patterns. Moreover, the effectiveness of data poisoning attacks often depends on the attacker's knowledge of the ML system, including the model architecture, training algorithms, and data distribution. Attackers may use adversarial machine learning or data synthesis techniques to craft samples that are more likely to bypass detection and achieve their malicious objectives. -![Garbage In -- Garbage Out (Source: [Information Matters](https://informationmatters.net/data-poisoning-ai/))](./images/png/distribution_shift_example.png){#fig-distribution-shift-example} +![Garbage In -- Garbage Out. Credit: [Information Matters.](https://informationmatters.net/data-poisoning-ai/)](./images/png/distribution_shift_example.png){#fig-distribution-shift-example} **Modifying training data labels:** One of the most straightforward mechanisms of data poisoning is modifying the training data labels. In this approach, the attacker selectively changes the labels of a subset of the training samples to mislead the model's learning process as shown in [@fig-distribution-shift-example]. For example, in a binary classification task, the attacker might flip the labels of some positive samples to negative, or vice versa. By introducing such label noise, the attacker aims to degrade the model's performance or cause it to make incorrect predictions for specific target instances. @@ -496,7 +496,7 @@ Each of these mechanisms presents unique challenges and requires different mitig **Manipulating data at the source (e.g., sensor data):** In some cases, attackers can manipulate the data at its source, such as sensor data or input devices. By tampering with the sensors or manipulating the environment in which data is collected, attackers can introduce poisoned samples or bias the data distribution. For instance, in a self-driving car scenario, an attacker might manipulate the sensors or the environment to feed misleading information into the training data, compromising the model's ability to make safe and reliable decisions. -![Data Poisoning Attack (Source: [Sikandar](https://www.researchgate.net/publication/366883200_A_Detailed_Survey_on_Federated_Learning_Attacks_and_Defenses))](./images/png/poisoning_attack_example.png){#fig-poisoning-attack-example} +![Data Poisoning Attack. Credit: [Sikandar.](https://www.researchgate.net/publication/366883200_A_Detailed_Survey_on_Federated_Learning_Attacks_and_Defenses)](./images/png/poisoning_attack_example.png){#fig-poisoning-attack-example} **Poisoning data in online learning scenarios:** Data poisoning attacks can also target ML systems that employ online learning, where the model is continuously updated with new data in real time. In such scenarios, an attacker can gradually inject poisoned samples over time, slowly manipulating the model's behavior. Online learning systems are particularly vulnerable to data poisoning because they adapt to new data without extensive validation, making it easier for attackers to introduce malicious samples, as shown in [@fig-poisoning-attack-example]. @@ -534,7 +534,7 @@ This case highlights how data poisoning can degrade model accuracy and reliabili ##### Case Study 2 -![Samples of dirty-label poison data regarding mismatched text/image pairs (Source: [Shan](https://arxiv.org/pdf/2310.13828))](./images/png/dirty_label_example.png){#fig-dirty-label-example} +![Samples of dirty-label poison data regarding mismatched text/image pairs. Credit: [Shan.](https://arxiv.org/pdf/2310.13828)](./images/png/dirty_label_example.png){#fig-dirty-label-example} Interestingly enough, data poisoning attacks are not always malicious [@shan2023prompt]. Nightshade, a tool developed by a team led by Professor Ben Zhao at the University of Chicago, utilizes data poisoning to help artists protect their art against scraping and copyright violations by generative AI models. Artists can use the tool to make subtle modifications to their images before uploading them online, as shown in [@fig-dirty-label-example]. @@ -546,7 +546,7 @@ On the other hand, this tool can be used maliciously and can affect legitimate a [@fig-poisoning] demonstrates the effects of different levels of data poisoning (50 samples, 100 samples, and 300 samples of poisoned images) on generating images in different categories. Notice how the images start deforming and deviating from the desired category. For example, after 300 poison samples, a car prompt generates a cow. -![Data poisoning (Source: @shan2023prompt)](images/png/image14.png){#fig-poisoning} +![Data poisoning. Credit: @shan2023prompt.](images/png/image14.png){#fig-poisoning} :::{#exr-pa .callout-exercise collapse="true"} @@ -561,9 +561,9 @@ Get ready to explore the dark side of AI security! In this Colab, you'll learn a #### Definition and Characteristics -Distribution shift refers to the phenomenon where the data distribution encountered by an ML model during deployment (inference) differs from the distribution it was trained on, as shown in [@fig-distribution-shift]. This is not so much an attack as it is that the model's robustness will vary over time. In other words, the data's statistical properties, patterns, or underlying assumptions can change between the training and test phases. +Distribution shift refers to the phenomenon where the data distribution encountered by an ML model during deployment (inference) differs from the distribution it was trained on, as shown in [@fig-distribution-shift]. The curly brackets enclose the distribution shift between the environments. Here, z stands for the spurious feature, and y stands for label class. This is not so much an attack as it is that the model's robustness will vary over time. In other words, the data's statistical properties, patterns, or underlying assumptions can change between the training and test phases. -![The curly brackets enclose the distribution shift between the environments. Here, z stands for the spurious feature, and y stands for label class (Source: [Xin](https://www.researchgate.net/publication/366423741_On_the_Connection_between_Invariant_Learning_and_Adversarial_Training_for_Out-of-Distribution_Generalization))](./images/png/distribution_shift.png){#fig-distribution-shift} +![Distribution shift. Credit: [Xin.](https://www.researchgate.net/publication/366423741_On_the_Connection_between_Invariant_Learning_and_Adversarial_Training_for_Out-of-Distribution_Generalization)](./images/png/distribution_shift.png){#fig-distribution-shift} The key characteristics of distribution shift include: @@ -575,14 +575,14 @@ The key characteristics of distribution shift include: **Unrepresentative training data:** The training data may only partially capture the variability and diversity of the real-world data encountered during deployment. Unrepresentative training data can lead to biased or skewed models that perform poorly on real-world data. Suppose the training data needs to capture the variability and diversity of the real-world data adequately. In that case, the model may learn patterns specific to the training set but needs to generalize better to new, unseen data. This can result in poor performance, biased predictions, and limited model applicability. For instance, if a facial recognition model is trained primarily on images of individuals from a specific demographic group, it may struggle to accurately recognize faces from other demographic groups when deployed in a real-world setting. Ensuring that the training data is representative and diverse is crucial for building models that can generalize well to real-world scenarios. -![Concept drift refers to a change in data patterns and relationships over time (Source: [Evidently AI](https://www.evidentlyai.com/ml-in-production/concept-drift))](./images/png/drift_over_time.png){#fig-drift-over-time} - Distribution shift can manifest in various forms, such as: **Covariate shift:** The distribution of the input features (covariates) changes while the conditional distribution of the target variable given the input remains the same. Covariate shift matters because it can impact the model's ability to make accurate predictions when the input features (covariates) differ between the training and test data. Even if the relationship between the input features and the target variable remains the same, a change in the distribution of the input features can affect the model's performance. For example, consider a model trained to predict housing prices based on features like square footage, number of bedrooms, and location. Suppose the distribution of these features in the test data significantly differs from the training data (e.g., the test data contains houses with much larger square footage). In that case, the model's predictions may become less accurate. Addressing covariate shifts is important to ensure the model's robustness and reliability when applied to new data. **Concept drift:** The relationship between the input features and the target variable changes over time, altering the underlying concept the model is trying to learn, as shown in [@fig-drift-over-time]. Concept drift is important because it indicates changes in the fundamental relationship between the input features and the target variable over time. When the underlying concept that the model is trying to learn shifts, its performance can deteriorate if not adapted to the new concept. For instance, in a customer churn prediction model, the factors influencing customer churn may evolve due to market conditions, competitor offerings, or customer preferences. If the model is not updated to capture these changes, its predictions may become less accurate and irrelevant. Detecting and adapting to concept drift is crucial to maintaining the model's effectiveness and alignment with evolving real-world concepts. +![Concept drift. Credit: [Evidently AI.](https://www.evidentlyai.com/ml-in-production/concept-drift)](./images/png/drift_over_time.png){#fig-drift-over-time} + **Domain generalization:** The model must generalize to unseen domains or distributions not present during training. Domain generalization is important because it enables ML models to be applied to new, unseen domains without requiring extensive retraining or adaptation. In real-world scenarios, training data that covers all possible domains or distributions that the model may encounter is often infeasible. Domain generalization techniques aim to learn domain-invariant features or models that can generalize well to new domains. For example, consider a model trained to classify images of animals. If the model can learn features invariant to different backgrounds, lighting conditions, or poses, it can generalize well to classify animals in new, unseen environments. Domain generalization is crucial for building models that can be deployed in diverse and evolving real-world settings. The presence of a distribution shift can significantly impact the performance and reliability of ML models, as the models may need help generalizing well to the new data distribution. Detecting and adapting to distribution shifts is crucial to ensure ML systems' robustness and practical utility in real-world scenarios. @@ -591,7 +591,7 @@ The presence of a distribution shift can significantly impact the performance an The mechanisms of distribution shift, such as changes in data sources, temporal evolution, domain-specific variations, selection bias, feedback loops, and adversarial manipulations, are important to understand because they help identify the underlying causes of distribution shift. By understanding these mechanisms, practitioners can develop targeted strategies to mitigate their impact and improve the model's robustness. Here are some common mechanisms: -![Temporal evolution (Source: [Białek](https://www.nannyml.com/blog/types-of-data-shift))](./images/png/temporal_evoltion.png){#fig-temporal-evoltion} +![Temporal evolution. Credit: [Białek.](https://www.nannyml.com/blog/types-of-data-shift)](./images/png/temporal_evoltion.png){#fig-temporal-evoltion} **Changes in data sources:** Distribution shifts can occur when the data sources used for training and inference differ. For example, if a model is trained on data from one sensor but deployed on data from another sensor with different characteristics, it can lead to a distribution shift. @@ -673,13 +673,13 @@ Recall that data poisoning is an attack that targets the integrity of the traini ##### Anomaly Detection Techniques for Identifying Poisoned Data -![Malicious data injection (Source: [Li](https://www.mdpi.com/2227-7390/12/2/247))](./images/png/adversarial_attack_injection.png){#fig-adversarial-attack-injection} +![Malicious data injection. Credit: [Li.](https://www.mdpi.com/2227-7390/12/2/247)](./images/png/adversarial_attack_injection.png){#fig-adversarial-attack-injection} Statistical outlier detection methods identify data points that deviate significantly from most data. These methods assume that poisoned data instances are likely to be statistical outliers. Techniques such as the [Z-score method](https://ubalt.pressbooks.pub/mathstatsguides/chapter/z-score-basics/), [Tukey's method](https://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm), or the [Mahalanobis] [distance](https://www.statisticshowto.com/mahalanobis-distance/) can be used to measure the deviation of each data point from the central tendency of the dataset. Data points that exceed a predefined threshold are flagged as potential outliers and considered suspicious for data poisoning. Clustering-based methods group similar data points together based on their features or attributes. The assumption is that poisoned data instances may form distinct clusters or lie far away from the normal data clusters. By applying clustering algorithms like [K-means](https://www.oreilly.com/library/view/data-algorithms/9781491906170/ch12.html), [DBSCAN](https://www.oreilly.com/library/view/machine-learning-algorithms/9781789347999/50efb27d-abbe-4855-ad81-a5357050161f.xhtml), or [hierarchical clustering](https://www.oreilly.com/library/view/cluster-analysis-5th/9780470978443/chapter04.html), anomalous clusters or data points that do not belong to any cluster can be identified. These anomalous instances are then treated as potentially poisoned data. -![Autoencoder (Source: [Dertat](https://towardsdatascience.com/applied-deep-learning-part-3-autoencoders-1c083af4d798))](./images/png/autoencoder.png){#fig-autoencoder} +![Autoencoder. Credit: [Dertat.](https://towardsdatascience.com/applied-deep-learning-part-3-autoencoders-1c083af4d798)](./images/png/autoencoder.png){#fig-autoencoder} Autoencoders are neural networks trained to reconstruct the input data from a compressed representation, as shown in [@fig-autoencoder]. They can be used for anomaly detection by learning the normal patterns in the data and identifying instances that deviate from them. During training, the autoencoder is trained on clean, unpoisoned data. At inference time, the reconstruction error for each data point is computed. Data points with high reconstruction errors are considered abnormal and potentially poisoned, as they do not conform to the learned normal patterns. @@ -727,7 +727,7 @@ In addition, domain classifiers are trained to distinguish between different dom ##### Mitigation Techniques for Distribution Shifts -![Transfer learning (Source: [Bhavsar](https://medium.com/modern-nlp/transfer-learning-in-nlp-f5035cc3f62f))](./images/png/transfer_learning.png){#fig-transfer-learning} +![Transfer learning. Credit: [Bhavsar.](https://medium.com/modern-nlp/transfer-learning-in-nlp-f5035cc3f62f)](./images/png/transfer_learning.png){#fig-transfer-learning} Transfer learning leverages knowledge gained from one domain to improve performance in another, as shown in [@fig-transfer-learning]. By using pre-trained models or transferring learned features from a source domain to a target domain, transfer learning can help mitigate the impact of distribution shifts. The pre-trained model can be fine-tuned on a small amount of labeled data from the target domain, allowing it to adapt to the new distribution. Transfer learning is particularly effective when the source and target domains share similar characteristics or when labeled data in the target domain is scarce. @@ -769,7 +769,7 @@ Machine learning frameworks, such as TensorFlow, PyTorch, and sci-kit-learn, pro **Memory Leaks and Resource Management Issues:** Improper memory management, such as failing to release memory or close file handles, can lead to memory leaks and resource exhaustion over time. This issue is compounded by inefficient memory usage, where creating unnecessary copies of large tensors or not leveraging memory-efficient data structures can cause excessive memory consumption and degrade system performance. Additionally, failing to manage GPU memory properly can result in out-of-memory errors or suboptimal utilization of GPU resources, further exacerbating the problem as shown in [@fig-gpu-out-of-memory](#nt13lz9kgr7t). -![Example of GPU out-of-the-memory and suboptimal utilization issues](./images/png/gpu_out_of_memory.png){#fig-gpu-out-of-memory} +![Example of GPU out-of-the-memory and suboptimal utilization issues.](./images/png/gpu_out_of_memory.png){#fig-gpu-out-of-memory} **Synchronization and Concurrency Problems:** Incorrect synchronization between threads or processes can lead to race conditions, deadlocks, or inconsistent behavior in multi-threaded or distributed ML systems. This issue is often tied to improper handling of [asynchronous operations](https://odsc.medium.com/optimizing-ml-serving-with-asynchronous-architectures-1071fc1be8e2), such as non-blocking I/O or parallel data loading, which can cause synchronization issues and impact the correctness of the ML pipeline. Moreover, proper coordination and communication between distributed nodes in a cluster can result in consistency or stale data during training or inference, compromising the reliability of the ML system. @@ -803,7 +803,7 @@ Detecting and mitigating software faults in machine learning frameworks is essen **Thorough Testing and Validation:** Comprehensive unit testing of individual components and modules can verify their correctness and identify potential faults early in development. Integration testing validates the interaction and compatibility between different components of the ML framework, ensuring seamless integration. Systematic testing of edge cases, boundary conditions, and exceptional scenarios helps uncover hidden faults and vulnerabilities. [Continuous testing and regression testing](https://u-tor.com/topic/regression-vs-integration) as shown in [@fig-regression-testing](#gaprh7zcofc9) detect faults introduced by code changes or updates to the ML framework. -![Automated regression testing (Source: [UTOR](https://u-tor.com/topic/regression-vs-integration))](./images/png/regression_testing.png){#fig-regression-testing} +![Automated regression testing. Credit: [UTOR.](https://u-tor.com/topic/regression-vs-integration)](./images/png/regression_testing.png){#fig-regression-testing} **Static Code Analysis and Linting:** Utilizing static code analysis tools automatically identifies potential coding issues, such as syntax errors, undefined variables, or security vulnerabilities. Enforcing coding standards and best practices through linting tools maintains code quality and reduces the likelihood of common programming mistakes. Conducting regular code reviews allows manual inspection of the codebase, identification of potential faults, and ensures adherence to coding guidelines and design principles. @@ -817,7 +817,7 @@ Detecting and mitigating software faults in machine learning frameworks is essen **Automated Testing and Continuous Integration/Continuous Deployment (CI/CD):** Implement automated testing frameworks and scripts, execute comprehensive test suites, and catch faults early in development. Integrating automated testing into the CI/CD pipeline, as shown in [@fig-CI-CD-procedure](#f14k3aj3u8av), ensures that code changes are thoroughly tested before being merged or deployed to production. Utilizing continuous monitoring and automated alerting systems detects and notifies developers and operators about potential faults or anomalies in real-time. -![Continuous Integration/Continuous Deployment (CI/CD) procedure (Source: [geeksforgeeks](https://www.geeksforgeeks.org/ci-cd-continuous-integration-and-continuous-delivery/))](./images/png/CI_CD_procedure.png){#fig-CI-CD-procedure} +![Continuous Integration/Continuous Deployment (CI/CD) procedure. Credit: [geeksforgeeks.](https://www.geeksforgeeks.org/ci-cd-continuous-integration-and-continuous-delivery/)](./images/png/CI_CD_procedure.png){#fig-CI-CD-procedure} Adopting a proactive and systematic approach to fault detection and mitigation can significantly improve ML systems' robustness, reliability, and maintainability. By investing in comprehensive testing, monitoring, and fault-tolerant design practices, organizations can minimize the impact of software faults and ensure their ML systems' smooth operation in production environments. @@ -856,7 +856,7 @@ Furthermore, implementing an error model is also an important consideration, par Recent research has shown that certain characteristics of error models may exhibit similar behaviors across different levels of abstraction [@sangchoolie2017one] [@papadimitriou2021demystifying]. For example, single-bit errors are generally more problematic than multi-bit errors, regardless of whether they are modeled at the hardware or software level. However, other characteristics, such as error masking [@mohanram2003partial] as shown in [@fig-error-masking](#kncu0umx706t), may not always be accurately captured by software-level models, as they can hide underlying system effects. Masking occurs when -![Example of error masking in microarchitectural components [@ko2021characterizing]](./images/png/error_masking.png){#fig-error-masking} +![Example of error masking in microarchitectural components. Credit: @ko2021characterizing](./images/png/error_masking.png){#fig-error-masking} Some tools, such as Fidelity [@he2020fidelity], aim to bridge the gap between hardware-level and software-level error models by mapping patterns between the two levels of abstraction. This allows for more accurate modeling of hardware faults in software-based tools, essential for developing robust and reliable ML systems. Lower-level tools typically represent more accurate error propagation characteristics but must be faster in simulating many errors due to the complex nature of hardware system designs. On the other hand, higher-level tools, such as those implemented in ML frameworks like PyTorch or TensorFlow, which we will discuss soon in the later sections, are often faster and more efficient for evaluating the robustness of ML systems. @@ -864,9 +864,9 @@ In the following subsections, we will discuss various hardware-based and softwar ### Hardware-based Fault Injection -An error injection tool is a tool that allows the user to implement a particular error model, such as a transient single-bit flip during inference @fig-hardware-errors. Most error injection tools are software-based, as software-level tools are faster for ML robustness studies. However, hardware-based fault injection methods are still important for grounding the higher-level error models, as they are considered the most accurate way to study the impact of faults on ML systems by directly manipulating the hardware to introduce faults. These methods allow researchers to observe the system's behavior under real-world fault conditions. Both software-based and hardware-based error injection tools are described in this section in more detail. +An error injection tool is a tool that allows the user to implement a particular error model, such as a transient single-bit flip during inference. In the inference pipeline in @fig-hardware-errors, faults (indicated by thunder bolt shapes) are injected into hardware components, such as DRAM and SRAM, in order manipulate the output. In this case, the vision model classifies the red light as a green light, which could lead to disasters in self-driving cars. Most error injection tools are software-based, as software-level tools are faster for ML robustness studies. However, hardware-based fault injection methods are still important for grounding the higher-level error models, as they are considered the most accurate way to study the impact of faults on ML systems by directly manipulating the hardware to introduce faults. These methods allow researchers to observe the system's behavior under real-world fault conditions. Both software-based and hardware-based error injection tools are described in this section in more detail. -![Hardware errors can occur due to a variety of reasons and at different times and/or locations in a system, which can be explored when studying the impact of hardware-based errors on systems [@ahmadilivani2024systematic]](./images/png/hardware_errors.png){#fig-hardware-errors} +![Hardware fault injection. Credit: @ahmadilivani2024systematic.](./images/png/hardware_errors.png){#fig-hardware-errors} #### Methods @@ -878,7 +878,7 @@ Two of the most common hardware-based fault injection methods are FPGA-based fau ![](./images/png/image15.png) -![Radiation test setup for semiconductor components [@lee2022design] (Source: [JD Instrument](https://jdinstruments.net/tester-capabilities-radiation-test/))](./images/png/image14.png){#fig-beam-testing} +![Radiation test setup for semiconductor components [@lee2022design]. Credit: [JD Instrument.](https://jdinstruments.net/tester-capabilities-radiation-test/)](./images/png/image14.png){#fig-beam-testing} #### Limitations @@ -914,7 +914,7 @@ Software-based fault injection tools also have some limitations compared to hard **Fidelity:** Software-based tools may provide a different level of Fidelity than hardware-based methods in terms of representing real-world fault conditions. The accuracy of the results obtained from software-based fault injection experiments may depend on how closely the software model approximates the actual hardware behavior. -![Comparison of techniques at layers of abstraction (Source: [MAVFI](https://ieeexplore.ieee.org/abstract/document/10315202))](./images/png/mavfi.jpg){#fig-mavfi} +![Comparison of techniques at layers of abstraction. Credit: [MAVFI.](https://ieeexplore.ieee.org/abstract/document/10315202)](./images/png/mavfi.jpg){#fig-mavfi} ##### Types of Fault Injection Tools @@ -922,10 +922,10 @@ Software-based fault injection tools can be categorized based on their target fr Ares [@reagen2018ares], a fault injection tool initially developed for the Keras framework in 2018, emerged as one of the first tools to study the impact of hardware faults on deep neural networks (DNNs) in the context of the rising popularity of ML frameworks in the mid-to-late 2010s. The tool was validated against a DNN accelerator implemented in silicon, demonstrating its effectiveness in modeling hardware faults. Ares provides a comprehensive study on the impact of hardware faults in both weights and activation values, characterizing the effects of single-bit flips and bit-error rates (BER) on hardware structures. Later, the Ares framework was extended to support the PyTorch ecosystem, enabling researchers to investigate hardware faults in a more modern setting and further extending its utility in the field. -![Hardware bitflips in ML workloads can cause phantom objects and misclassifications, which can erroneously be used downstream by larger systems, such as in autonomous driving. Shown above is a correct and faulty version of the same image using the PyTorchFI injection framework.](./images/png/phantom_objects.png){#fig-phantom-objects} - PyTorchFI [@mahmoud2020pytorchfi], a fault injection tool specifically designed for the PyTorch framework, was developed in 2020 in collaboration with Nvidia Research. It enables the injection of faults into the weights, activations, and gradients of PyTorch models, supporting a wide range of fault models. By leveraging the GPU acceleration capabilities of PyTorch, PyTorchFI provides a fast and efficient implementation for conducting fault injection experiments on large-scale ML systems, as shown in [@fig-phantom-objects](#txkz61sj1mj4). The tool's speed and ease of use have led to widespread adoption in the community, resulting in multiple developer-led projects, such as PyTorchALFI by Intel Labs, which focuses on safety in automotive environments. Follow-up PyTorch-centric tools for fault injection include Dr. DNA by Meta [@ma2024dr] (which further facilitates the Pythonic programming model for ease of use), and the GoldenEye framework [@mahmoud2022dsn], which incorporates novel numerical datatypes (such as AdaptivFloat [@tambe2020algorithm] and [BlockFloat](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format) in the context of hardware bit flips. +![Hardware bitflips in ML workloads can cause phantom objects and misclassifications.](./images/png/phantom_objects.png){#fig-phantom-objects} + TensorFI [@chen2020tensorfi], or the TensorFlow Fault Injector, is a fault injection tool developed specifically for the TensorFlow framework. Analogous to Ares and PyTorchFI, TensorFI is considered the state-of-the-art tool for ML robustness studies in the TensorFlow ecosystem. It allows researchers to inject faults into the computational graph of TensorFlow models and study their impact on the model's performance, supporting a wide range of fault models. One of the key benefits of TensorFI is its ability to evaluate the resilience of various ML models, not just DNNs. Further advancements, such as BinFi [@chen2019sc], provide a mechanism to speed up error injection experiments by focusing on the \"important\" bits in the system, accelerating the process of ML robustness analysis and prioritizing the critical components of a model. NVBitFI [@tsai2021nvbitfi], a general-purpose fault injection tool developed by Nvidia for their GPU platforms, operates at a lower level compared to framework-specific tools like Ares, PyTorchFI, and TensorFlow. While these tools focus on various deep learning platforms to implement and perform robustness analysis, NVBitFI targets the underlying hardware assembly code for fault injection. This allows researchers to inject faults into any application running on Nvidia GPUs, making it a versatile tool for studying the resilience of ML systems and other GPU-accelerated applications. By enabling users to inject errors at the architectural level, NVBitFI provides a more general-purpose fault model that is not restricted to just ML models. As Nvidia's GPU systems are commonly used in many ML-based systems, NVBitFI is a valuable tool for comprehensive fault injection analysis across various applications. @@ -943,7 +943,7 @@ MAVFI [@hsiao2023mavfi] is a fault injection tool designed for the robotics doma The development of software-based fault injection tools has greatly expanded the capabilities of researchers and practitioners to study the resilience of ML systems to hardware faults. By leveraging the speed, flexibility, and accessibility of these tools, the ML community can develop more robust and reliable systems that can operate safely and effectively in the presence of hardware faults. ### Bridging the Gap between Hardware and Software Error Models -![Hardware errors may manifest themselves in different ways at the software level, as classified by Bolchini et al. [@bolchini2022fast]](./images/png/hardware_errors_Bolchini.png){#fig-hardware-errors-bolchini} +![Hardware errors may manifest themselves in different ways at the software level, as classified by Bolchini et al. Credit: @bolchini2022fast.](./images/png/hardware_errors_Bolchini.png){#fig-hardware-errors-bolchini} While software-based fault injection tools offer many advantages in speed, flexibility, and accessibility, they may not always accurately capture the full range of effects that hardware faults can have on the system. This is because software-based tools operate at a higher level of abstraction than hardware-based methods and may miss some of the low-level hardware interactions and error propagation mechanisms that can impact the behavior of the ML system, as shown in [@fig-hardware-errors-bolchini].