Built site for gh-pages

.nojekyll

@@ -1 +1 @@
-eec41acf
+585ab0ee

dl_primer.html

@@ -84,6 +84,8 @@
 }</script>
 <style>html{ scroll-behavior: smooth; }</style>
 
+  <script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
+  <script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml-full.js" type="text/javascript"></script>
 
 </head>
 
@@ -438,7 +440,7 @@ <h3 data-number="3.1.1" class="anchored" data-anchor-id="definition-and-importan
 <section id="brief-history-of-deep-learning" class="level3" data-number="3.1.2">
 <h3 data-number="3.1.2" class="anchored" data-anchor-id="brief-history-of-deep-learning"><span class="header-section-number">3.1.2</span> Brief History of Deep Learning</h3>
 <p>The concept of deep learning has its roots in the early artificial neural networks. It has witnessed several waves of popularity, starting with the introduction of the Perceptron in the 1950s <span class="citation" data-cites="rosenblatt1957perceptron">(<a href="references.html#ref-rosenblatt1957perceptron" role="doc-biblioref">Rosenblatt 1957</a>)</span>, followed by the development of backpropagation algorithms in the 1980s <span class="citation" data-cites="rumelhart1986learning">(<a href="references.html#ref-rumelhart1986learning" role="doc-biblioref">Rumelhart, Hinton, and Williams 1986</a>)</span>.</p>
-<p>The term “deep learning” emerged in the 2000s, marked by breakthroughs in computational power and data availability. Key milestones include the successful training of deep networks by <a href="https://amturing.acm.org/award_winners/hinton_4791679.cfm">Geoffrey Hinton</a>, one of the god fathers of AI, and the resurgence of neural networks as a potent tool for data analysis and modeling.</p>
+<p>The term <em>deep learning</em> emerged in the 2000s, marked by breakthroughs in computational power and data availability. Key milestones include the successful training of deep networks such as AlexNet <span class="citation" data-cites="krizhevsky2012imagenet">(<a href="references.html#ref-krizhevsky2012imagenet" role="doc-biblioref">Krizhevsky, Sutskever, and Hinton 2012</a>)</span> by <a href="https://amturing.acm.org/award_winners/hinton_4791679.cfm">Geoffrey Hinton</a>, one of the god fathers of AI, and the resurgence of neural networks as a potent tool for data analysis and modeling.</p>
 <p>In recent years, deep learning has witnessed exponential growth, becoming a transformative force across various industries. <a href="#fig-trends">Figure&nbsp;<span>3.1</span></a> shows that we are currently in the third era of deep learning. From 1952 to 2010, computational growth followed an 18-month doubling pattern. This dramatically accelerated to a 6-month cycle from 2010 to 2022. At the same time, we witnessed the advent of major-scale models between 2015 and 2022; these appeared 2 to 3 orders of magnitude faster and followed a 10-month doubling cycle.</p>
 <div id="fig-trends" class="quarto-figure quarto-figure-center anchored">
 <figure class="figure">
@@ -587,17 +589,17 @@ <h4 data-number="3.2.6.3" class="anchored" data-anchor-id="recurrent-neural-netw
 </section>
 <section id="generative-adversarial-networks-gans" class="level4" data-number="3.2.6.4">
 <h4 data-number="3.2.6.4" class="anchored" data-anchor-id="generative-adversarial-networks-gans"><span class="header-section-number">3.2.6.4</span> Generative Adversarial Networks (GANs)</h4>
-<p>GANs consist of two networks, a generator and a discriminator, that are trained simultaneously through adversarial training. The generator produces data that tries to mimic the real data distribution, while the discriminator aims to distinguish between real and generated data. GANs are widely used in image generation, style transfer, and data augmentation.</p>
+<p>GANs consist of two networks, a generator and a discriminator, that are trained simultaneously through adversarial training <span class="citation" data-cites="goodfellow2020generative">(<a href="references.html#ref-goodfellow2020generative" role="doc-biblioref">Goodfellow et al. 2020</a>)</span>. The generator produces data that tries to mimic the real data distribution, while the discriminator aims to distinguish between real and generated data. GANs are widely used in image generation, style transfer, and data augmentation.</p>
 <p>In embedded contexts, GANs could be used for on-device data augmentation to enhance the training of models directly on the embedded device, facilitating continual learning and adaptation to new data without the need for cloud computing resources.</p>
 </section>
 <section id="autoencoders" class="level4" data-number="3.2.6.5">
 <h4 data-number="3.2.6.5" class="anchored" data-anchor-id="autoencoders"><span class="header-section-number">3.2.6.5</span> Autoencoders</h4>
-<p>Autoencoders are neural networks used for data compression and noise reduction. They are structured to encode input data into a lower-dimensional representation and then decode it back to the original form. Variations like Variational Autoencoders (VAEs) introduce probabilistic layers that allow for generative properties, finding applications in image generation and anomaly detection.</p>
+<p>Autoencoders are neural networks used for data compression and noise reduction <span class="citation" data-cites="bank2023autoencoders">(<a href="references.html#ref-bank2023autoencoders" role="doc-biblioref">Bank, Koenigstein, and Giryes 2023</a>)</span>. They are structured to encode input data into a lower-dimensional representation and then decode it back to the original form. Variations like Variational Autoencoders (VAEs) introduce probabilistic layers that allow for generative properties, finding applications in image generation and anomaly detection.</p>
 <p>Implementing autoencoders can assist in efficient data transmission and storage, enhancing the overall performance of embedded systems with limited computational and memory resources.</p>
 </section>
 <section id="transformer-networks" class="level4" data-number="3.2.6.6">
 <h4 data-number="3.2.6.6" class="anchored" data-anchor-id="transformer-networks"><span class="header-section-number">3.2.6.6</span> Transformer Networks</h4>
-<p>Transformer networks have emerged as a powerful architecture, especially in the field of natural language processing. These networks use self-attention mechanisms to weigh the influence of different input words on each output word, facilitating parallel computation and capturing complex patterns in data. Transformer networks have led to state-of-the-art results in tasks such as language translation, summarization, and text generation.</p>
+<p>Transformer networks have emerged as a powerful architecture, especially in the field of natural language processing <span class="citation" data-cites="vaswani2017attention">(<a href="references.html#ref-vaswani2017attention" role="doc-biblioref">Vaswani et al. 2017</a>)</span>. These networks use self-attention mechanisms to weigh the influence of different input words on each output word, facilitating parallel computation and capturing complex patterns in data. Transformer networks have led to state-of-the-art results in tasks such as language translation, summarization, and text generation.</p>
 <p>These networks can be optimized to perform language-related tasks directly on-device. For instance, transformers can be utilized in embedded systems for real-time translation services or voice-assisted interfaces, where latency and computational efficiency are critical factors. Techniques such as model distillation which we will discss later on can be employed to deploy these networks on embedded devices with constrained resources.</p>
 <p>Each of these architectures serves specific purposes and excel in different domains, offering a rich toolkit for tackling diverse problems in the realm of embedded AI systems. Understanding the nuances of these architectures is vital in designing effective and efficient deep learning models for various applications.</p>
 </section>
@@ -608,23 +610,23 @@ <h2 data-number="3.3" class="anchored" data-anchor-id="libraries-and-frameworks"
 <p>In the world of deep learning, the availability of robust libraries and frameworks has been a cornerstone in facilitating the development, training, and deployment of models, particularly in embedded AI systems where efficiency and optimization are key. These libraries and frameworks are often equipped with pre-defined functions and tools that allow for rapid prototyping and deployment. This section sheds light on popular libraries and frameworks, emphasizing their utility in embedded AI scenarios.</p>
 <section id="tensorflow" class="level3" data-number="3.3.1">
 <h3 data-number="3.3.1" class="anchored" data-anchor-id="tensorflow"><span class="header-section-number">3.3.1</span> TensorFlow</h3>
-<p><a href="https://www.tensorflow.org/">TensorFlow</a>, developed by Google, stands as one of the premier frameworks for developing deep learning models. Its ability to work seamlessly with embedded systems comes from <a href="https://www.tensorflow.org/lite">TensorFlow Lite</a>, a lightweight solution designed to run on mobile and embedded devices. TensorFlow Lite enables the execution of optimized models on a variety of platforms, making it easier to integrate AI functionalities in embedded systems. For TinyML we will be dealing with <a href="https://www.tensorflow.org/lite/microcontrollers">TensorFlow Lite for Microcontrollers</a>.</p>
+<p><a href="https://www.tensorflow.org/">TensorFlow</a>, developed by Google <span class="citation" data-cites="abadi2016tensorflow">(<a href="references.html#ref-abadi2016tensorflow" role="doc-biblioref">Abadi et al. 2016</a>)</span>, stands as one of the premier frameworks for developing deep learning models. Its ability to work seamlessly with embedded systems comes from <a href="https://www.tensorflow.org/lite">TensorFlow Lite</a>, a lightweight solution designed to run on mobile and embedded devices. TensorFlow Lite enables the execution of optimized models on a variety of platforms, making it easier to integrate AI functionalities in embedded systems. For TinyML we will be dealing with <a href="https://www.tensorflow.org/lite/microcontrollers">TensorFlow Lite for Microcontrollers</a>.</p>
 </section>
 <section id="pytorch" class="level3" data-number="3.3.2">
 <h3 data-number="3.3.2" class="anchored" data-anchor-id="pytorch"><span class="header-section-number">3.3.2</span> PyTorch</h3>
-<p><a href="https://pytorch.org/">PyTorch</a>, an open-source library developed by Facebook, is praised for its dynamic computation graph and ease of use. For embedded AI, PyTorch can be a suitable choice for research and prototyping, offering a seamless transition from research to production with the use of the TorchScript scripting language. PyTorch Mobile further facilitates the deployment of models on mobile and embedded devices, offering tools and workflows to optimize performance.</p>
+<p><a href="https://pytorch.org/">PyTorch</a>, an open-source library developed by Facebook <span class="citation" data-cites="paszke2019pytorch">(<a href="references.html#ref-paszke2019pytorch" role="doc-biblioref">Paszke et al. 2019</a>)</span>, is praised for its dynamic computation graph and ease of use. For embedded AI, PyTorch can be a suitable choice for research and prototyping, offering a seamless transition from research to production with the use of the TorchScript scripting language. PyTorch Mobile further facilitates the deployment of models on mobile and embedded devices, offering tools and workflows to optimize performance.</p>
 </section>
 <section id="onnx-runtime" class="level3" data-number="3.3.3">
 <h3 data-number="3.3.3" class="anchored" data-anchor-id="onnx-runtime"><span class="header-section-number">3.3.3</span> ONNX Runtime</h3>
 <p>The <a href="https://onnx.ai/">Open Neural Network Exchange (ONNX)</a> Runtime is a cross-platform, high-performance engine for running machine learning models. It is not particularly developed for embedded AI systems, though it supports a wide range of hardware accelerators and is capable of optimizing computations to improve performance in resource-constrained environments.</p>
 </section>
 <section id="keras" class="level3" data-number="3.3.4">
 <h3 data-number="3.3.4" class="anchored" data-anchor-id="keras"><span class="header-section-number">3.3.4</span> Keras</h3>
-<p>Keras serves as a high-level neural networks API, capable of running on top of TensorFlow, and other frameworks like Theano, or CNTK. For developers venturing into embedded AI, Keras offers a simplified interface for building and training models. Its ease of use and modularity can be especially beneficial in the rapid development and deployment of models in embedded systems, facilitating the integration of AI capabilities with minimal complexity.</p>
+<p>Keras <span class="citation" data-cites="chollet2015">(<a href="references.html#ref-chollet2015" role="doc-biblioref">Chollet 2015</a>)</span> serves as a high-level neural networks API, capable of running on top of TensorFlow, and other frameworks like Theano, or CNTK. For developers venturing into embedded AI, Keras offers a simplified interface for building and training models. Its ease of use and modularity can be especially beneficial in the rapid development and deployment of models in embedded systems, facilitating the integration of AI capabilities with minimal complexity.</p>
 </section>
 <section id="tvm" class="level3" data-number="3.3.5">
 <h3 data-number="3.3.5" class="anchored" data-anchor-id="tvm"><span class="header-section-number">3.3.5</span> TVM</h3>
-<p>TVM is an open-source machine learning compiler stack that aims to enable efficient deployment of deep learning models on a variety of platforms. Particularly in embedded AI, TVM and µTVM (Micro TVM) can be crucial in optimizing and streamlining models to suit the restricted computational and memory resources, thus making deep learning more accessible and feasible on embedded devices.</p>
+<p>TVM is an open-source machine learning compiler stack that aims to enable efficient deployment of deep learning models on a variety of platforms <span class="citation" data-cites="chen2018tvm">(<a href="references.html#ref-chen2018tvm" role="doc-biblioref">Chen et al. 2018</a>)</span>. Particularly in embedded AI, TVM and µTVM (Micro TVM) can be crucial in optimizing and streamlining models to suit the restricted computational and memory resources, thus making deep learning more accessible and feasible on embedded devices.</p>
 <p>These libraries and frameworks are pivotal in leveraging the capabilities of deep learning in embedded AI systems, offering a range of tools and functionalities that enable the development of intelligent and optimized solutions. Selecting the appropriate library or framework, however, is a crucial step in the development pipeline, aligning with the specific requirements and constraints of embedded systems.</p>
 </section>
 </section>
@@ -690,15 +692,39 @@ <h3 data-number="3.4.8" class="anchored" data-anchor-id="scalability"><span clas
 
 
 <div id="refs" class="references csl-bib-body hanging-indent" role="list" style="display: none">
+<div id="ref-abadi2016tensorflow" class="csl-entry" role="listitem">
+Abadi, Martı́n, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. <span>“<span class="math inline">\(\{\)</span>TensorFlow<span class="math inline">\(\}\)</span>: A System for <span class="math inline">\(\{\)</span>Large-Scale<span class="math inline">\(\}\)</span> Machine Learning.”</span> In <em>12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)</em>, 265–83.
+</div>
+<div id="ref-bank2023autoencoders" class="csl-entry" role="listitem">
+Bank, Dor, Noam Koenigstein, and Raja Giryes. 2023. <span>“Autoencoders.”</span> <em>Machine Learning for Data Science Handbook: Data Mining and Knowledge Discovery Handbook</em>, 353–74.
+</div>
+<div id="ref-chen2018tvm" class="csl-entry" role="listitem">
+Chen, Tianqi, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, et al. 2018. <span>“<span class="math inline">\(\{\)</span>TVM<span class="math inline">\(\}\)</span>: An Automated <span class="math inline">\(\{\)</span>End-to-End<span class="math inline">\(\}\)</span> Optimizing Compiler for Deep Learning.”</span> In <em>13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)</em>, 578–94.
+</div>
+<div id="ref-chollet2015" class="csl-entry" role="listitem">
+Chollet, François. 2015. <span>“Keras.”</span> <em>GitHub Repository</em>. <a href="https://github.com/fchollet/keras" class="uri">https://github.com/fchollet/keras</a>; GitHub.
+</div>
+<div id="ref-goodfellow2020generative" class="csl-entry" role="listitem">
+Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. <span>“Generative Adversarial Networks.”</span> <em>Communications of the ACM</em> 63 (11): 139–44.
+</div>
 <div id="ref-jouppi2017datacenter" class="csl-entry" role="listitem">
 Jouppi, Norman P, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, et al. 2017. <span>“In-Datacenter Performance Analysis of a Tensor Processing Unit.”</span> In <em>Proceedings of the 44th Annual International Symposium on Computer Architecture</em>, 1–12.
 </div>
+<div id="ref-krizhevsky2012imagenet" class="csl-entry" role="listitem">
+Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E Hinton. 2012. <span>“Imagenet Classification with Deep Convolutional Neural Networks.”</span> <em>Advances in Neural Information Processing Systems</em> 25.
+</div>
+<div id="ref-paszke2019pytorch" class="csl-entry" role="listitem">
+Paszke, Adam, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, et al. 2019. <span>“Pytorch: An Imperative Style, High-Performance Deep Learning Library.”</span> <em>Advances in Neural Information Processing Systems</em> 32.
+</div>
 <div id="ref-rosenblatt1957perceptron" class="csl-entry" role="listitem">
 Rosenblatt, Frank. 1957. <em>The Perceptron, a Perceiving and Recognizing Automaton Project Para</em>. Cornell Aeronautical Laboratory.
 </div>
 <div id="ref-rumelhart1986learning" class="csl-entry" role="listitem">
 Rumelhart, David E, Geoffrey E Hinton, and Ronald J Williams. 1986. <span>“Learning Representations by Back-Propagating Errors.”</span> <em>Nature</em> 323 (6088): 533–36.
 </div>
+<div id="ref-vaswani2017attention" class="csl-entry" role="listitem">
+Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. <span>“Attention Is All You Need.”</span> <em>Advances in Neural Information Processing Systems</em> 30.
+</div>
 </div>
 </section>
 </section>