ml_audio_classifier_example_for_pico.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "ml_audio_classifier_example_for_pico.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w7gY4ELwlmOk"
      },
      "source": [
        "\n",
        "<img alt=\"Arm Software Dev\" src=\"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/4gKgSUNDX1BST0ZJTEUAAQEAAAKQbGNtcwQwAABtbnRyUkdCIFhZWiAAAAAAAAAAAAAAAABhY3NwQVBQTAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA9tYAAQAAAADTLWxjbXMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAtkZXNjAAABCAAAADhjcHJ0AAABQAAAAE53dHB0AAABkAAAABRjaGFkAAABpAAAACxyWFlaAAAB0AAAABRiWFlaAAAB5AAAABRnWFlaAAAB+AAAABRyVFJDAAACDAAAACBnVFJDAAACLAAAACBiVFJDAAACTAAAACBjaHJtAAACbAAAACRtbHVjAAAAAAAAAAEAAAAMZW5VUwAAABwAAAAcAHMAUgBHAEIAIABiAHUAaQBsAHQALQBpAG4AAG1sdWMAAAAAAAAAAQAAAAxlblVTAAAAMgAAABwATgBvACAAYwBvAHAAeQByAGkAZwBoAHQALAAgAHUAcwBlACAAZgByAGUAZQBsAHkAAAAAWFlaIAAAAAAAAPbWAAEAAAAA0y1zZjMyAAAAAAABDEoAAAXj///zKgAAB5sAAP2H///7ov///aMAAAPYAADAlFhZWiAAAAAAAABvlAAAOO4AAAOQWFlaIAAAAAAAACSdAAAPgwAAtr5YWVogAAAAAAAAYqUAALeQAAAY3nBhcmEAAAAAAAMAAAACZmYAAPKnAAANWQAAE9AAAApbcGFyYQAAAAAAAwAAAAJmZgAA8qcAAA1ZAAAT0AAACltwYXJhAAAAAAADAAAAAmZmAADypwAADVkAABPQAAAKW2Nocm0AAAAAAAMAAAAAo9cAAFR7AABMzQAAmZoAACZmAAAPXP/bAEMABQMEBAQDBQQEBAUFBQYHDAgHBwcHDwsLCQwRDxISEQ8RERMWHBcTFBoVEREYIRgaHR0fHx8TFyIkIh4kHB4fHv/bAEMBBQUFBwYHDggIDh4UERQeHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHv/CABEIAZABkAMBIgACEQEDEQH/xAAcAAEAAwEBAQEBAAAAAAAAAAAABQYHBAMBAgj/xAAaAQEBAAMBAQAAAAAAAAAAAAAAAQIEBQMG/9oADAMBAAIQAxAAAAHZQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgKACAoAIACgAAAAAAAAAAAAAARuc+Dq/NRsveola/P0G55+Wg5HreI62/NwkjeNnQhpKbren0q1+YjSd7lZ7c+Ko2S8ddP35+vhf8L2XW35AaPZAAAAAAAAAAAAAcfZyXDFNUyvU+r85YxyfpskmYWa7Hy+g4ht+IeG3f7jULfq9FSLvSLjTtoxjZ/fSr2VavlHvqbfzd3Dy/oMZ1rJda6nAnByfpQAAAAAAAAAAAAAMl5tVzrq/N22Jp8wV658k5lhcMQ2/Ldbeslurlj195SbtV7KDs+Z6X7akBlOu0H21dW4O/j53cxbWqHoXQ4ksOb9AAAAAAAAAAAAAAAACAoAAAAAAAAAAAAAAAAAAAg0lfDIZKzWlDvkoKBSOWs7NcfqPkJkAAAAAAAAAAAAAAAAAAol750q1xyL7ZrHpHdEy6WO9Nx1lkmlrleyfz5olxrOy/wA+aYXdkmt45efplOpHookoWdk3yzWmLXRbqzCOTYFQo67Ox+0l3EoAAAAAAAAAzNNMiPTpMftlTu2WPRbKJe5lns5VLnZnOyY3skY1suNbKZTdO3C6ktBn8fTv1CuWuXOdWx7YSm8/l61A6VmunGdxszOFcrW3VMuIxyAAAAAAAAARskMh/OwLjBzX6S5pw6ysz/QCXPdCDPNDBRrZ6HTSLuPP0FocDrSzLbXZxRbn7o58x1YZJdrKAUAAAAAABxdsamV+nbqeUoF9ptZjXBMgAAAAKJKRcpcbOJkAAAAAAAAAAAAAAjZKNTPNVyrVbGO7Fkpp9ekMoJuyyuYVqHn1ZjGy0Kz58ajm/wC+ojNDhKXVllIuUiVz6M1Yz7Q/HMDSavKVavlotuQmw1jvyGJ6wTWbVs+d2zJSdus3mUeOgwVpOsTIAAAABGyUcmd6r/P8/ljqOQ9GpL+Mp2vHS3c8dDpred6T4TLINZzTX0yKfms9suHrWZZfaUjJOM92mgQda3i3XZjr5unODdM55oWLtQdtySrTzR0QmrZbq+UrsXj7ZdL1ePN13HSxjmAAAAAB+fv0AAcPb9AHl6g8vUcvUHz6Dn6B4e4VLx6PG42Pt+pkBxdf6D59AHF2gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB//xAAxEAACAgIBAQUHAwQDAAAAAAADBAIFAQYANRATFBYzERIVIDA0QDZGUCEiMZAlJmD/2gAIAQEAAQUC/wBZjzYlAt2rbGe8JxK0aWk5ZsTZ11k5y8K+5gsLZqKsillnWpMTLsZzAGGybiV2wZZlAxoSSu/Yu3YtMyiUsc1FtPvfz7ZrLTlYlN0w6hGEbClhmHNV9bhvWq0pOn+CJd2grFNfavT4jTLwDe141o8p6oUwXFWCC3K8nepfmu591TmvD9yt7Lwfd2eq+vw/ratjHhOza/TQx7XebDj/AIrgY+4Gx+w5RdL/ADW4++rzXi+/W9l2TvbPVfX4f1tW+z7Nr9Ou+/5sHSuY5Y/YcoulfnXSuVnK5wiZhXSUo2N3jMOar6/Detq/2fZtXp133/Ng6VzH+LH7DlH0r85tYTQXKhoOfDse1KpaPlqnawegSYVLwtS/klCsZZbsv1TtQTq3ht8uAkOh8IsOxyGSKfCLDlUKYEP/AHxSiDgTapZfNfXmESa/buuvfw9274BCvrnbojusnELU3mmB/KTGCbfwrqgi/wANusZZrNTZCSq5GMY4+X9482T9R9uZRxL+CYEM4XqF1MqWxtrzRcXdCyYa4C2FpcteXbKPK+3drmnYzOj4dv4rVV9sA9gu2F+or7AUKF1la45f2kh3sJYnDb35Lra2I0K23uWmmoa9Zyw25aog1ApS1u5MHCyNO3uRK5zTUg8294Y9PbIQ1u3k9j8exr1Xh0si12wbqbMUNVXiGo5uwI5T1gmSUv7x5sn6j5uCuQtCsoZpFkCuoai331b/AFu9jt5ZDVaQKMj824MSVGldL3j7pCOII7yWXK2+8En5pnypP/2H8Xa2X1W65wTqrBhrhqfesdl29eRqvUXoFS5uTo58oV5K1P7x5sn6j5aK4cQ79iCtUpFSub76psNPT7lKxDlhHUWoqvc3FwcE9K6XvH3Sn2u7LykDWmhNVufcjhK7XZsvxbBQLy06G1UL8EuW5VNcGuDnGM4sdcn33gtllin16K5OfB2/MHLenbat+wlR7dk5sdVJ5gcIjhy7oItkglssMeWz5V11EyCWy1TNgcEcwAcQzhY151Y863YW8UtSGuh/BNtAUGuYTAfz3SSEmO/tyc8wWws1+zAJKMoyj9Tdemat0H8+06bo3r5xjOLihXZhrFgRRz6m69M1boP59p03RvX7NwDgFooXvlbm6BX58yWOcU98F2d04VFOjscWS3HL0kLUpIBE1s8slhszQyIthdBuvTNW6C84BIBNmZIQOzngVNkLYLlySKGNlFhHOyWHspr0D0uXFwCu55ksJYqNgC2Tmdk7t/GyuzLbWUK5TzM9jlY8J9b6tp03RvX7N3njLgJ+EpKXwrFji0rMR2LCUW1sxsKqiJOsvbZrCSGmqZKzuzUsYoCVKKj7VO6DUTZDbbr0zVug2852uwJKATC+ku6HXikr7zbui6jWiNjOMZxtKEUWUXImqq3IHbaFnVwjs0a+RKdjxVYBeLWzYhD3d3GTvELCtfSWXCsP6tp03RvX480FMCQy3d5cxzKp1qvVsJ+Wa7hqCoDxMEFVt0T9krizlZAq1cJo7tHPj1terTL+Wa7lZVVoHd16Zq3QU8+F2zsFnxO27d0XTixnVc3csfDVYCeVNcSVfY8s13D0FQCKCok1an9W8LAZYWetCzHULA0y/Vs+nVVgxXSlsVqTAay2tTVyQEV849uLBBynejtX9mIWGwOAFEIXl4tKatXyzacv674ipX2jtPlrZTmHrFSVeW69M1boWzU82sp7G0rB2/bdhrNRJLG3dFrPiCIc7X/Yoq7evwjGELeuarHh7Vnu55sNgaTBBZapxnzbzZgPLvk2kmV9PQLCf1vZj5sqKyn/AI7YQjDPYQcCRCuAPylAE3BBELHNu6Lp3R8qK5n2zUVnPGMYx8mFVok/hdrjKVNqMZRqP9NX/8QALBEAAQMCBQMCBgMAAAAAAAAAAQACAwQRBRAhMUESEzAUQCAyMzRwcSNQUf/aAAgBAwEBPwH8b1VZLUS9uLZegqoR1tOqop5H1IuVi8jmBvSVHT1NU3qvopI3wURBOqpm1FTdjXaLuT0UhbdOo6l7O8SsKqnSAsdx55PlKwi3f1ygt67T/VjWzVQfbtWI/bOWC/I5Yv8AXCkH8R/Swf6x/XsKuilgk7kS9bVzDoAVFTSsqQXBYtE+QN6RdUTS2BoKr2l0DgFhMTo2HqFlikEj5gWi6eLsKwqGSOUlwt/T39zfK6ughldXV1dXV1fy8ILlBBbLZBBcrlc+W2VvjsrZWVvEPBz7EbIq+Zz5W+XK2RORWyHgBW6K0yGfK2W65RKK0yK509nz+IP/xAArEQACAQMCBAUEAwAAAAAAAAAAAQIDBBExQRASITAUIDIzQAUTInA0UGH/2gAIAQIBAT8B/W9GhClDnmeJoz/FouKUY0nhFjFSbyTq0aLwRlGpcf4VXSo9Wjlp3Ecir0Yy5Ei9oqGJR78dS+z9vhU/jdT6fqy591lp7qL/ANSLH2yPqL/218ChcQqQ5Jnh6FP8my4rQlSeGWM4xzllw06jaLZqNRNl7OMpLDLOpCNN5ZH1F7UjKCw/6fHkfxEY4YMDGNGDBgwYMGO2uG4zYYxdTUZLU2Hobd3m4Z8+TPDJntMx59vgIlqR1EjXgjHDY0NRaHRiXBI1H2GjQR1Exmx1NjU0FoJCZ1EI26/D2/UH/8QAShAAAQIDBAQICggDBwUAAAAAAQIDAAQREhMhMRAiQVEUMmFxcnOBsQUgI0JQUoKRocEwM0BikrLR8BVDoiQ0NVODhJBgY6Ph8f/aAAgBAQAGPwL/AIzLxzsG+DRd2ncmOOr3xxy4j1VQpTL7iWzkIdDrql0GFdCwJhecFu2SsnjndFVOLJ5TC6urLSRkd8NXLhRUmtIQpb7ikg4jfGLhSnYlMWkOrB54VwkVcSMKedGs4Up9VOEVS4sHnhLE0q0DgF7vQClV1BgjmiyNVCeMqKFq3ykwVylUq9TYdD3RGhfSMWa2UJ4xizRdfWtYxdINca1hjnMUgGYTeOHPHAQl5gUQTQjdoD8yLVripgvS6bBRmN+hpw5lOP255Q2IOhB2rJJ0ugZHWh7ojQvpGHD9/SxzmGB/3Boc5CO/QhA2CkP9We7Qz29/251A2oI0JTtQSDpdIyGrD3RGhfSMOdP5aWOcwx1g0O9nfpf6s92hnt7/ALerDya9ZMW04pPGTviqlKQdxEFuUBqfPOh7ojQvpGHOn8tLHOYY6waHezv0v9We7Qz29/28tOjDuglCb1G9Ofuj6h38JgFabpG9WfuhQYaq3sJUIdL6LNRhjoUQzgT6whaXk2SVVz0tBlFqyccYaWpqiUrBOsNDjTQqs0oO2Pqf6hodQnNSCBH1P9QhtpwUWM/+v7TrqGxvUqkWWpllw7krB8fg7CAt6mNckxczCEBNgqqE09EKfAqvio54VMvPURXFasfcIvJZ+9UPNpQ9kLZmErUG8nD3eNZWKi/0XTsw2hfqlWPodtQyS7j7jCGUkXjVbSe3PRRIA5vG/wBfQPY8QJJFTly+g1NOpCkKwIi/kFqWBlZNFiLqebvKYHYoRey67Q2jaIW86aIQKmCzJ2m29yTSg5TFtM2i30jHBfCVtSNtrjJ/WHUy66LWjUVWODXn9pt0tWtvPC1TMxbSWyB5UnGLmYctPYa1qsTCJ2YKrxFEkOE0hUnOPLVaNjWVWitDN2apljjyk5wFpNQRUQiXZWpLjhqSDkIS5MOLWt3W1jWg2RwPwdaCa2ap4y4vFTSEucqzWHJCaUuqsULtY57DC1OuKWbzNRrsEMBp5xuqDxVUjhCnglvJAUqLycWpa91quOwCFWXClrbjRAi/YfK6Z3ajWCxMUv0itfWH2iy82K7FjMRwUnArulcu6GmR/MXj2Q2scZ3WUdDUzTXSuz2GGLXm1T8Y/wBxoHsaG59rC1goj1hlH8QONEVI+9u98Tk+a2kGo+8c1RdKOsxh2RvZr/QP38YmFIwo2aU2RMPEayAAO3/5oU4RrNEEe+kOdae4RL9A98MIGQbT3RLM11cVGES6JCtnNVvP4R/h/wD5P/UNPttXaXHeLur9mbLT60sqFQBhiIS+0rPMeqYU68sIQnaY4RQ0vC4eQbPlF4gVLSrXZAk1EB1rIbxoRJNqtFKrS6d0MNLFFUqe2P8AcaB7GhyXOZGrzwvwea2bypTyw1Lcmtz7Ympdo0C0lPsmDNLGu9l0YfYGa0ECHZd42L3DH1ho4GFVccIJG4Q51p7hEv0D3w10B3QxMjJBKVdsNoFLxoWVCKmgAgyjbSqV1Fjb9mLDwwOR3HfBVJO2tykLsmBwt3AbXHLVIsN1UpXGWdsUOIMX3g5wIxrYJpTmMXZdcs774QH5xQccGISMho4dVu6vbWeOgTTRbsauZ3aUTYT5H6w9IfuuiWcbGNqw50YShAspSKAaDMSyg26eMk5Ki6S6uzvvhDq3XQ5NryxwGO+FNP2bRXXVMNLYsUSmhtGEIOaUgQpp1NpChQiL3wc/XdrWVCLuYcIR953D4QSDbeVxl/L0GHJhwNpJpWA8yu2hWR9APOo4yUEiPJhK+ZuseUbR7TdICJtu5PrDEQFJIIORH0rfWjuMS/tfmPoCZ6pXdEz0RFCKwpyWQll/k4qo/h8zUIUqgteYr6VvrR3GJf2vzH0BM9UruiZ6I0ofbwLibXaIae9dAVF0Beveru54tiUau99lUBl1Ny8cscFRwhpoOUOtU5QXCkIWlVFJB0GSl5dLutYra2wpxxQShIqTFiSlrQ2Fe3sgCak005KpPxgPMKqn4iG+tHcYl/a/MYL0wqyNg2mKSsmmmytSfhFmblABts4H4wHmF2kn4QqZSgLIIFDAdW1V8k0bSrvi84G3d76GLlxN09sFcFaLBF48ckD5xbRJt2OiYDD6Ll08XHA6HGXmAG0KULQOOEJDco3ZUaJGPfCXHU1dVkgHbAWqUauzyHvgPtYbFJOw/TTPVK7omeiNLCNobr8YQtQxaYBp2QuY8JPIAGtRXnGLIm2AOeG5jwc6jW4wQeKYQXRg81rQqUdyWbtXyP73w7MbQNXnhyfcxsYJ5zn++WGZROStdXyhBM0zfrFVknHmgsvTTBByNcRC5W0FIcBy3jbDfWjuMS/tfmMcFSrUSu7TybzAaYbCR3wW30A7lbUwqSWdVai2rn2GHOknvhU4+gLCTZQDlzxQjCG5qV8mlZ2eaqETij/LtK7M4U/4ReSlHHNo58kBKZthKRkAYRMyLzRUo66UH4ww+cynHnGEKZXxTMKr7zATYFE5CmUS71PJ0Ke2EyaylFU2C2rD3RdsNpbTyfTTPVK7omeiNBefVRI+MF1Y1LVpfIndE0B/lGHkTBXVIBTZPv8AlGb/AOKE3rziLRsirgxMIYbrYRlWG55HQX8olJdANqmuN68v3zw1LjzRid5hlWwt/OG3Qp6i0g8aM3/xRfSry1Oskgi3WkN9aO4xL+1+Ywb3V8soe/LSFN4i/rhuEOdJPfF2OM2s10MM+cV2ouqay2V0HPWkOMzJWCE1TZNIzf8AxQFPPOoBNAVLEJYZKigZVMK61z56Ch1KVJOwwXJFRQr/AC1ZGFSLyioBNUV2cn00z1Su6FqZbQq2KG2DFENNJP3WzAdmytCfWc+QgMsp5ztVFDHC5O0Wq4KAy5DGtJ63IvCAtYsMp2+akfrCGkcVAoIcl1ZLFOaFreT/AHY09rRZTQOoxQflBlZhgqQMkqwpzRdSkvdrVtraMGdmqhxQ1U7ueG+tHcYl/a/MY4XKiroGsn1ouZxguFOFSbKu2ODyjBbt4GmsowZiY+uUKAeqIc6Se+B4RlU22ibKx+sYSet08I4RMVDXnK2U3CAhIoAKARw6RCrutQR5nJzR5STqvkXhCRZsMp/Cn9TCGG+KgUhXWufPQnwgytxTYxG2wf0gpRLBLpHGtYQudeSU2k2UV28v0+XjW1SzJVvsDxFFKQCo1Vy6bLiErG5QrHkWG2+imni+Vabc6SaxRppDY+6mmhzpJ74/1DFsyzJXvsCviW1yzKlbygRQCg8W8EuyF+tYFfQzgSCTaTlzxRSSDeHP/hr/AP/EACwQAQABAgMIAQQDAQEAAAAAAAERACExQVEQYXGBkaGx8MEgQFDRMOHxkGD/2gAIAQEAAT8h/wCTkmv1ya/TJrtk1/Aolu2PF0RLlvHVxpRln4qADnKz0cqiPA4UWo7YKFhenCjowYE76Ea+yykMClbViqtJ18zSSwxwpRSkM1qNgFnwaUvc7qA/dGtQh1Cb0MdG6kjk5If3UAozGVdgcrFb6PvmxQPksXlzqUOI0uG+is1muWjiATJngb6RGEhK9ZrThXvNaTlAkzDdxrcm9GHap4L6Rdn0r3WhQKgJWokAmS4BFPEvHkWtZUq2OZYA1ajPaZ1BU0rSz5s/vlxws6OwY3yFB2DaBSFBzJe817TWta9ZrRId2L0NvqtKBzB8tFEq5k6NhkIDBwNtXZ+T758cEczYU35Sk87RTSoORD3mvaa1k16zWvV3NvqtK9LrRXf+PZgOFe11bOx8n38LQVUw3nKoW4RMlB8ymvisDREYjgVevaa04Ne81r3dzb7rSvS60YV3fj2YFel1bO18n385FYJitSodXh6HSoIMmlS8s8I5KRE+MZW40EUmWXPdThQvVKcbjVluog2g02y1qNg80J9AYAPGisxjQGAc63X230YFFPJBOKlbr7b6hlVhM5v4GD/z27KKHet/nWdn64mtJ5zCxi0gZQIUkb9/4iHhEBwl6vKnA2F2XShGW9XO6saBult3ub9S0DGR3VBpWErbIuw/D3UCPOFBvQM66eS+yJ1mYEH1Gz9/h9BqWDW/B+DF6MrOnC2n/QeXSplS5I57WifIouiVcCwVFDPMFSadN7G9YoaYGMXeDkqSooEbpZmu5P8APmpx4vsHg3oHvXcWF6K/TYHe5plRkpGMAwx1w57ImBoHAelqP0MhmNWFLpI7974abDWP0C/XnSNLaxTc5FCnC8JDvaOFgkoQOKUCjsEqN5QGhCJK+6jMTyAGLWCerURPuvEernTsSYg91ti9ac4skwOGdYZxBbla/cI3N2c/4o+uUJhreGrTF3gH7Sg0xtZeDtsHAurqD8nen5K4jQUdqNl7/DYdrOZBiOniretGrLUMK/RBlQrfEpc9x8nKsgoZ8v20umTYGCLUcxT3K6dg7kn53CO/baA8N2BjooAKFG1bB89at/RkZuLsTvQQL5BbE8/tje+sKa4pjl1oNDDHvmDRxAlohRypH+FC0jqOV+HlUtkJteZ5TsNO3JmIO5q+CR6KmO9Gy9/hsPQu05HB60xII5yNo8dKiZKY+avRuk5dbg8Yt1qPgTjBh1fioGb1axarNYA2QjZ6uyE4Aslm/MNoDz3+iiBnoEh7d6s0bn7YPBKKTIlXApvKwaSBddD7ZzTeDHKFQlWEjm//AGrvj9BAWn+KWM/UpmYCEc6NKlhUtDDKrK+Bmnu7m1q67P1abhGyNJ6N1ZbbbSPjusdWwFGIaYp5X60FUQWAGBsue/Be13NCxTYsA6zUmsF7wkriWJplmk3CIP1S6FVhz4VGLII3FGFmlnVrQfEpwa1T6ErlKaiSSEIto0PwcEIhDjyo1UqHOGPJ+AZQu9qFJIgY4anolf5FJlTO6uZ3om7SiRPx3i/0GqvT6tRJDRKPkLgY3CZcalmgPFocHz+O8X+g1V6/V2yHEJNaJ8Va6LVxJqVmGYsHG+KOg0HWayuo3cJ13VFqmGCDn1irpw0I0fdNipoIavn6Pip4wHgFRWYUqeVRduYtHVUl+sjiaJt8X6VFLroFLXMiaOEKDb3EuWKjedetEyalQCCC7FGxDWADNZUO0rBR9ZoHI8SXAdd2w6QSURG9ZUOInFljnNTODCJPpudl6xalXRbfFRMbZJXTJ2oAtrj2a+hrUCVUDDO66KmYuzDS/m9Bqr1+rtI7dLn/AEpL/kWpho3olWzviigSQBaFPl4sGYhNJ+KlCCCaperbN7WvUcl4w1dipDLjOeM6URqC4uQ89qgKTYifAoLSal1GhwSNDdBynb4vCvLgwh+p6FBCZdi71XOrbRwrmo1OAAMoenPYhCuC8yzhTkiiES1C1k1sF7af1Vg5Jd4+Q0kYrZpzbh+CgrRAIBURAGeZUd5Md8x3KKOXTqCR2oQ7NlIYRV8CJmRKe/xT8gdxs3jOgozkceOv83oNVev1dgvcIZvQ31BHizgsPhRJ3+Gg/wAwEtfRT/H/AKozIezcYUt7CJZaQu3wMnN8dKSTA9JxpBwtmzLr1oizAjwX7pu2xTmcK/w/6qzAh0iIiRv2+L+IqPHLubTvtgc2Z22IPWIA8bj7psDKvIaAR81jF79N5RySHdXvlwr/AA/6oBDiIVywp0hqpm7O3+2VdORoXJFkncDlzmkt8a9i34L9v5vcaqX1FIsOCVIuM2PdaQhcbcHrpUbIxfE1aBAkcRqIkkzAXL0p4343YfCsNzrOS1pN3FmZYKtfrlqyetR/BkJwnS702DLJTu/NS55cU42lPk2mEcCMaJl4K3Di71BYyaiyethfNNcGCb6If1XAXp88xFBoWtQPGuP92xByMCPJbR818h/qrFDyzApgIG0AVk0OZU4ooUBQXYk8IqXlVoGHWaJXrEuLvqRwxszD1LiC3avmoGKMQOsRR9yBN5N+z+fcOlAGB9OfDZs9YoAQAG2WUqDFAT0DbusSB3rMy0/DYg4lFthUGaF5VuvgQ7bUPV4VnfyfeRQAQEG3eAGnrFExBgBsgmYNuZGId5H4bAboEtMpZIjT/jX/AP/aAAwDAQACAAMAAAAQ888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888888188588158888888888888884Oy5c+3pk/B88888888888888Aq8gooUDDq98888888888888881iDSdol5F/8888888888888888d88888888888888888888bb/88i0888888888888888888rewz4R37byR0LwU88888888887v70lD/azH/wBBbtPPPPPPPPPPPHS3/wB/+P8Ac0cvbe88888888/I/wDPPPPPKfPPPPPPPPPPPPPPP/786B/1+Kdz+aN7qR2/PPPPPO+8HNUB65IdGBvjM6JT/PPPPPLPPLPPPLPLLHB/PDPLPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP/xAAnEQEAAgIBAwMEAwEAAAAAAAABABEhMUEQUbEwQGEgkaHBcHGB0f/aAAgBAwEBPxD+H7HrZ0uCMs9hqWKcqK5+V7ThK4Fv/jLb8rZbWu0usLvTXaFdDBb4iIKgudWmCBoUy2tfEdsvuN6Y1tYurbrxEztyF3Xr2xbpmXypr7n66f1dv3Pz39Q1/V+54vkgyfJ4n455gAHd4n57yewoOpdlbP8AO05TvII/fRF0gLmmtd5eHS9C9oGdIaYParWDLsgSyU2JxEYijQvMYRunxFFiuROT2NHtVqWchBsvqWt+0SyoUw9M9ELRxZYNwszap8CFmo5VGkMqfTWi5hIbTSEc5sxbSsGDFzXpNpmEyW+pWBUrADUCoFRBiX0RAErNxDAHosCl3BRp+s9gddLWK1FTNxaSJuiKrRLRqEWrEtHMVwI3lOxFTNxOKjYlu/1uoITK+IM3L4ESyoeWIDmFLghA2pi2xFqLCiCql8I8RUQr+Mx//8QAKREBAAICAAQFAwUAAAAAAAAAAQARITEQQWHBMEBRcbFwoeEgUJHR8P/aAAgBAgEBPxD6b1pzvPKXdJ9TH4lVGA7SuQ67xHTLuiG2v4dI0Z3KiV9f79p7ZmsXFgVe/H136zFrV54X6lHafB7z72fL+Gfbzf7vwRWXr3n3HbyFgc6zz/MsaHV/1wOO0MXnlEaxrb7xVLLikUfiMhOOUJCG+b0iAL6wMJb5PT9mC5UwsSmuNAryipuNskqY7Y0h6w4JRGgTS51salwxtYXjhZ4YtqZGOlzaOtzkmpBSpeZFmuJ0SyRKaPEtFuXiruLcVYKQab4DItKi2VBEW48EiBqohLPLtuIA2wDCoFj6wFKwAWxBLrgoGZQMQGTDRUs3AHFQM3ClUp6frNx1uYHWPCQOZlDcYYIK4RoZeBKWQKZguEG2WWQOZnNBbG7fTN//xAAsEAEAAQMDAwMDBAMBAAAAAAABEQAhMUFRYRBxgZGh8ECxwSAwUPGQ0eFg/9oACAEBAAE/EP8AE2oa1xPWhHWpN+sm9SUlo9aEcI1JUm9KGUobA6KGUqbR6/wLuVHRbg/Lgp83bs44bubhxUphMyzfeprWvNvJde3DVxWw2YyRwzUzbsQlSlKGmlH1vFoCgpMIQkEAdGZZ0pIRSqPK0PkY2+AXWArEaUez5MQBE+taWd2Vl8i1SuiyJtJi65fahR9mOfvRMlmVNYlFkWl2m0l1llZ8TZi/lNCUJI8PI1EAFCjLA9RsS3NV0wuF/rkIulGtWJiD9y/ninTQCCYuAaqH0WncCQtbdhA8BV8044GiZTZKjxTByIRIRrbT/sVNW0NPQ/uUAck0Q4HKLdl0iu0Feu3iiPRR8KrTGwDxV7cdDImQASrUIHgAuwiY3c8Vfe65AUC3hh8xvQEmb08guyriCFXITEOrhVsFJlvlYTNtmhBazU6xNbxj3D9bMU7SORojVFpqH4Uozd0BpQoAwblQfAbqcdlfBbqEwvjcIvu1p0+R2UZYgo9tYHagwl4IxfZehwA40AI9qImkn5VaUlZZ+tiJoNJNDVUPvU2io8Uqm4S9gHjo4odSpOL2FPgN1PsV8Fur32k6fO7K+b21h1i+2dDmn18hJEdaTnABAFnynG0U6MEcgLF9E0f91AnFzB4ZD8xQ1oNqDnWnliNqZSrK1uP7Fa7aitj+5V5c6XgoZrG0/CoWH4msHbrFsTgpxzVhP18lrNi1tpE0anSyQTHnKfKmKrem9IpHsc0HJee8FLfhCCASshlZ0KfKbEtEuTFXstek5AkMikoWZ8UmFMpMjWauETRqxaxRAMicVODhvAqwSbUmKmax3BIXQYGiKrmCdCrVXeEiBLYu0mTUythxMJYuKNk/geA9P/OZ6f31QJBX9WpOR/XG7x9tpCgoMxJAmZipV+S6ABUJdp/DlFlhAk8LwBBrZrUvlZgAmIhBJqBptQqtWYAm4Q8GO+lH4FP3vCrDk1iZ0nT9IomhSJCB4sUGiCQgYwUHTbO0As3/AIc/UGGgAvljzTDmkQtEGVAvvJ0EDahZFVYN1Xu9PH6PvaK95Rp1QG6CCBLDWC9OOvj9Hip62+nYM0GQfhG46INQs6qme0G26WlyEznzZAI2Qd2hD2hsBOqH2dFKMm4/YwG6sAaqFTANJHoxHv4mNhadyiAyeNbvFCAKtlcea5bYdEqSFRLANwuMlGa/CB6qbyPYYLQtEN9KQe7DYWe9rdqHRTP/AJmFEqTam4C4UZimyGRmFNJc5jkgSPjxRpthnSAkTw1JkYZ3suWC2lEM5yWcsSYk+BUn8w4CoobNohglYsDSntCBG/ae9DzCaDQoMqCI3J0uUCfOQkRKbXaDOUuWtETUc9B/5hahm4s5pMq1/plMgQEpbLFWb2gfJG69yveChEhiFLrOImxLxT5mBwAgWFgKTGRkLP1EYo2IrRNRykdqYCxTSxD3gcgpq0jDc27mPU+KKHLrFwewDyu/QUAQS6iF4CO6gSFOpsg8QPFfdVrXvKChSiGhNPdR8tA5dxu6cJgHCNTplJiT2HPdrLm9km8L0BSOLOWQO0NPzo987dRBZiJtR0nYlDaNmwnZd+if0UbAgdkmnDbqIQPMEgjFytiUcjJ8DUQywg26VJ5ndgA0rn+umBPA5qhAkGRbAfTMvqKDAwhJQt5GKBFnBLSLIj6kJZpYAVNtAyroF2oCB5LyF70Gl0zLIVjiU2FSSSmIVQG6kIaA9JX6BSGpRrcTS1XovfKE3IAe1fdVrXvKCpdxclJuwnhaHKGoxBcTIso3FKAy+b7clQ4CknkSbsBwyaNlZYRUuEdlx5CiwiVUE1dxMVezTsIJNiQO4GvScA5xtV2LAOF/QIQQWlzguEnalncUsky4RDEeUBfeTSmvUHADKrgpPiQe5AWbTDeSJj6ZzrYggiNyXhFGzROLiDaQoHaRUTsMSoc5B5J3qf0gSGMEaEsHOrTqQDSBsibUrvVgTSJXL4GI3qLcIhE1ll1EJyaluIgs2gB36HbTz+2T56FgNMXCwltv1IwZgLTAcFXNFNKC8AxVVv8AYe5ois1wNAcAHRx0WFrJhL6sIuzKivqMgrALCbQVPmCHSI8SwMRfXI8QySYeULyqkj3w5ZEQqkUTUkkBj0pl1A7D8IwiXEEqUuepvmYICNZJ2qVMJYqbKvJcp/J9rrGHcB3YJwB9R4rP7J8QugsULHQfSrTilQkKyDZDx+35q59IFA5iQSkmtyiUCJnB3jFGkRmBPpRZUbPn4RD/AGaJW8olcRLI7/va/e/wQfNUiLkSPhoCyJBkYFk7IvmaSTeKuYjOEsm4bSz+7r97/BA8VcdJGXkxaG1jzl1qSAlBomj3pI9m1pw3YkuAXGBGlrwblAen7UCBkJIbJCLd4XFTk9UxRMHY801CFiCTevCT5W1aUMevRSAQDAiPdTB1IwDL/wAp5KsC24UQ8zwU4BpGLK6Cl2t3KnBeEgedM/8AEk666DQgwD3BP2yxQLkvcsUh2vG7TjCmC9xnwk70saotYGcgNvOGiV5mpFKTvWB9ZwoaSGWCFY80WTaZGPS9qAbPxi0aHueFooNELeHS4g6WV21pPs2RjWSD2oUfFQ6TdsAyO8sdDb7clwjCJQMwTT1MsVkASE3KGsfGSBvg3tGpa9E6ywxHIST48UXOcAmBVGS8jqbY+gPFXHQoCL7DB99L0XuhTMtr2p0TJMlkM5F1OxigRiKACADEcUiSSYhKEbmlpW9BasoSzjtDMbUtk6lq6Q7qRwqDgkfxHtLLwNJgDESmmTcUf8U1diNxHwJF3NlRckNlAxdMEaotJKDbyy2mS3fDJUksOpnENkd49NfvaEOrlcLwaspyBQBBA2Q5S7+KkXzAEzui8Wwxeae+aCktgOUjtX47ZUCMhUIKLMSBpM6hCYvBFDkTakMqs8I4Y4JcDCsEUCAjAsMPTSMlCwowXCccFCHEUNgAsFKfIS5CGMIxrDmaZEI7KyvlHzXK8z+Vw2eaAhmCJtBgiCIxTxKiTZjtDHdVvi+1gBpNzBghYq/1i1lESsrll+hPFQ+PGaLY9VH5YBaca1hVIT7oA3uxZqQY1A4b+KnON6UhMqYfd0rqpQqSgoFzYadIlwUVboE52oCFYowSz6ay4ovY4ryMINbCPDSnbHRPwlJjiKagbfKhPZ60aySw7D9/Tu6YsDGpBLLycdNfvaAgMNEkQ9hmnZ6aUtYzjpK2aI2vjtlBka+YfiMp5dG4UaG6snumOztUqycgukvqG80fXrCJCcqc2d+nedAlFKJOdmiDAyaZXAIl26KNGVWF+O1iByXri7vuGMZpai3SiJG7CROLNf3iphK1xviUEEpHI1mi9sA79g3tSbUgC9u09oA70mCQXGDCv4waUUAyAkR0abFSamEg8JbNsODgR5CO8LQ4lplSIFiGUnmg7sTBgkjgGCLurTuwpGXJDwB8Uqy2CMkL8O4dBlSsCqXHQgX0QakEFNcsrCi86l5G7JgAi71tA7me1JUYd7y+ziWb4BEWgTWUAR0NJEIB4BeDLRqRFyECrAYAWgKlzA7y3o0MLMS1gXTkJ2SiJ41SSyilpwTGAjVr47ZTkB4V8MBcLkHDJabpMMjZlD2vjzUdgFJLuCzL6xMqrcyzkwCA9ClSEt5Q3JSgtoYb5HYwPawySdpaLl60jMKWgYy6ATU0gyMplXKq+aIgsiLV1ojXZE6E0kEziUOSWz1N+yIuTuC95pDnIAQMTcLAdZf3kEhxX9arCB2P0IJCSVel2Tu8pUZMCACA62KoxN0N2B46vlTKR4CVLaGb/wCg6AQCclAEABsdAJRMByc4NI0X/UQOnx2yrwPzKLnJk/LeUqJCCwBAdbm+m6/dKhSREEBwHQOAneOiCQklX5TYk94S/hmA9BqWaFGpGZ6iLof8Nf8A/9k=\" width=200></img>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m174xspUl7LC"
      },
      "source": [
        "# ML Audio Classifier Example for Pico\n",
        "\n",
        "```\n",
        "Copyright (c) 2021 Arm Limited and Contributors. All rights reserved.\n",
        "\n",
        "SPDX-License-Identifier: Apache-2.0\n",
        "```\n",
        "\n",
        "Authors: [Sandeep Mistry](https://twitter.com/sandeepmistry), [Henri Woodcock](https://twitter.com/henriwoodcock) from the [Arm Software Developers team](https://twitter.com/armsoftwaredev)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_h5zXsutmdCS"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "This tutorial will guide you through how to train a TensorFlow based audio classification Machine Learning (ML) model to detect a fire alarm sound. We’ll show you how to use [TensorFlow Lite for Microcontrollers](https://www.tensorflow.org/lite/microcontrollers) with Arm [CMSIS-NN](https://arm-software.github.io/CMSIS_5/NN/html/index.html) accelerated kernels to deploy the ML model to an [Arm Cortex-M0+](https://developer.arm.com/ip-products/processors/cortex-m/cortex-m0-plus) based microcontroller (MCU) board for local on-device ML interferencing. Arm’s [CMSIS-DSP](https://arm-software.github.io/CMSIS_5/DSP/html/index.html) library, which provides optimized Digital Signal Processing (DSP) function implementations for [Arm Cortex-M](https://developer.arm.com/ip-products/processors/cortex-m) processors, will also be used to extract features from the real-time audio data prior to inference.\n",
        "\n",
        "While this guide focuses on detecting a fire alarm sound, it can be adapted for other sound classification tasks. You may also need to adapt the feature extraction stages and/or adjust ML model architecture for your use case.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6w60BKbnzzrP"
      },
      "source": [
        "## What you need to to get started\n",
        "\n",
        "### Development Environment\n",
        "\n",
        " * [Google Chrome](https://www.google.com/intl/en_ca/chrome/)\n",
        " * [Google Colab](https://colab.research.google.com/notebooks/)\n",
        " * A [Google Account](https://www.google.com/account/about/)\n",
        "\n",
        "### Hardware\n",
        "\n",
        "You’ll need one of the following development boards that are based on [Raspberry Pi’s RP2040 MCU chip](https://www.raspberrypi.org/products/rp2040/) that was released early in 2021.\n",
        "\n",
        "\n",
        "#### SparkFun RP2040 MicroMod and MicroMod ML Carrier\n",
        "\n",
        "This is recommended for people who are new to electronics and microcontrollers. While it does cost a bit more than the option below, it is easier to assemble and does not require a soldering iron, knowing how to solder and how to wire up breadboards.\n",
        "\n",
        " * [SparkFun MicroMod RP2040 Processor](https://www.sparkfun.com/products/17720)\n",
        "  * For the brains of the operation! Contains Raspberry Pi’s RP2040 MCU and 16MB of flash storage\n",
        " * [SparkFun MicroMod Machine Learning Carrier Board](https://www.sparkfun.com/products/16400)\n",
        "Enables USB connectivity, and provides a built-in microphone, IMU and camera connector\n",
        " * A USB-C cable to connect the board to your computer\n",
        " * A Phillips screwdriver\n",
        "\n",
        "#### Raspberry Pi Pico and PDM microphone board\n",
        "\n",
        "This option is slightly lower in cost, however it requires a soldering iron and knowledge of how to wire a breadboard with electronic components.\n",
        "\n",
        " * [Raspberry Pi Pico](https://www.raspberrypi.org/products/raspberry-pi-pico/)\n",
        " * [Adafruit PDM MEMS Microphone Breakout](https://www.adafruit.com/product/3492)\n",
        " * Half size or full size breadboard\n",
        " * Jumper wires\n",
        " * A USB-B micro cable to connect the board to your computer\n",
        " * Soldering iron\n",
        "\n",
        "#### More information\n",
        "\n",
        "Both of the options above will allow you to collect real-time 16 kHz audio from a digital microphone and process the audio signal in real-time on the development board’s Arm Cortex-M0+ processor which operates at 125 MHz. The application running on the Arm Cortex-M0+ will have a Digital Signal Processing (DSP) stage to extract features from the audio signal, the extracted features will then be fed into a neural network to perform a classification task to determine if a fire alarm sound is present in the board’s environment.\n",
        "\n",
        "### Hardware Setup\n",
        "\n",
        "#### SparkFun MicroMod RP2040\n",
        "\n",
        "For assembly, remove the screw on the carrier board, at an angle, slide in the MicroMod RP2040 Processor board into the socket and secure it in place with the screw. See the [MicroMod Machine Learning Carrier Board Hookup Guide](https://learn.sparkfun.com/tutorials/micromod-machine-learning-carrier-board-hookup-guide?_ga=2.90268890.1509654996.1628608170-268367655.1627493370#hardware-hookup) for more details.\n",
        "\n",
        "\n",
        "#### Raspberry Pi Pico\n",
        "\n",
        "Follow the instructions from the [Hardware Setup section of the \"Create a USB Microphone with the Raspberry Pi Pico\"](https://www.hackster.io/sandeep-mistry/create-a-usb-microphone-with-the-raspberry-pi-pico-cc9bd5#toc-hardware-setup-5) guide for assembly instructions.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ojuc2yoIrA8G"
      },
      "source": [
        "## Install dependencies"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2t4hpGoHoUR6"
      },
      "source": [
        "### Python Libraries\n",
        "\n",
        "Let's start by installing the Python library dependencies:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B6CQatks4z4p"
      },
      "source": [
        "!pip install librosa matplotlib pandas \"tensorflow==2.8.*\" \"tensorflow-io==0.24.*\" \"tensorflow-model-optimization==0.7.2\"\n",
        "\n",
        "!pip install git+https://github.com/ARM-software/CMSIS_5.git@5.8.0#egg=CMSISDSP\\&subdirectory=CMSIS/DSP/PythonWrapper"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3QMTfgPBoeRB"
      },
      "source": [
        "### Command line tools\n",
        "\n",
        "Now let's install the command line tools we will need to build applications for the Raspberry Pi RP2040:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lb3DjP6lXyUf"
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "tf.keras.utils.get_file('cmake-3.21.0-linux-x86_64.tar.gz',\n",
        "                        'https://github.com/Kitware/CMake/releases/download/v3.21.0/cmake-3.21.0-linux-x86_64.tar.gz',\n",
        "                        cache_dir='./',\n",
        "                        cache_subdir='tools',\n",
        "                        extract=True)\n",
        "\n",
        "tf.keras.utils.get_file('gcc-arm-none-eabi-10-2020-q4-major-x86_64-linux.tar.bz2',\n",
        "                        'https://developer.arm.com/-/media/Files/downloads/gnu-rm/10-2020q4/gcc-arm-none-eabi-10-2020-q4-major-x86_64-linux.tar.bz2',\n",
        "                        cache_dir='./',\n",
        "                        cache_subdir='tools',\n",
        "                        extract=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TV4uI68XX_J0"
      },
      "source": [
        "!apt-get install -y xxd"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EPi1R_THovN9"
      },
      "source": [
        "Now add the downloaded and extracted tools to the `PATH` environmental variable, so we can use them later on without specifying the full path to them:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9evc84YJYz2P"
      },
      "source": [
        "import os\n",
        "\n",
        "os.environ['PATH'] = f\"{os.getcwd()}/tools/cmake-3.21.0-linux-x86_64/bin:{os.environ['PATH']}\"\n",
        "os.environ['PATH'] = f\"{os.getcwd()}/tools/gcc-arm-none-eabi-10-2020-q4-major/bin:{os.environ['PATH']}\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MT78ms3GpQIv"
      },
      "source": [
        "### Raspberry Pi Pico SDK\n",
        "\n",
        "We can use `git` to get the `v1.2.0` of the [Raspberry Pi Pico SDK](https://github.com/raspberrypi/pico-sdk)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9P02F36WY5uZ"
      },
      "source": [
        "%%shell\n",
        "git clone --branch 1.2.0 https://github.com/raspberrypi/pico-sdk.git\n",
        "cd pico-sdk\n",
        "git submodule init\n",
        "git submodule update"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iE_gX86hpk8h"
      },
      "source": [
        "Set the `PICO_SDK_PATH` environment variable to specify the location of the `pico-sdk`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yXCHQtSWY_dS"
      },
      "source": [
        "os.environ['PICO_SDK_PATH'] = f\"{os.getcwd()}/pico-sdk\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YzxhKhMQpzqr"
      },
      "source": [
        "**You will need to change the code cell below** to select the board you will be using for the remainder of the tutorial.\n",
        "\n",
        "By default the `PICO_BOARD` environment variable is set to `sparkfun_micromod` for the SparkFun RP2040 MicroMod. Set the value to `pico` if you are using a Raspberry Pi Pico board."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5WVJZJFUTMEC"
      },
      "source": [
        "# for SparkFun MicroMod\n",
        "os.environ['PICO_BOARD'] = 'sparkfun_micromod'\n",
        "\n",
        "# for Raspberry Pi Pico (uncomment next line)\n",
        "# os.environ['PICO_BOARD'] = 'pico'\n",
        "\n",
        "print(f\"PICO_BOARD env. var. set to '{os.environ['PICO_BOARD']}'\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n1-rsj4vchBC"
      },
      "source": [
        "### Project Files\n",
        "\n",
        "The source code for the inference application and Python utilities for Google Colab can also be cloned using `git`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-jDNNonWcyAL"
      },
      "source": [
        "%%shell\n",
        "git clone --recurse-submodules https://github.com/ArmDeveloperEcosystem/ml-audio-classifier-example-for-pico.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yWTsVPrgdL9R"
      },
      "source": [
        "For convenience we can create symbolic links for the project files that we've cloned to the root Google Colab folder:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_1JjfZ15eZX9"
      },
      "source": [
        "%%shell\n",
        "ln -s ml-audio-classifier-example-for-pico/colab_utils colab_utils\n",
        "ln -s ml-audio-classifier-example-for-pico/inference-app inference-app"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HMuv2599z8hy"
      },
      "source": [
        "## Baseline model\n",
        "\n",
        "We will start by training a generic sound classifier model with TensorFlow using the [ESC-50: Dataset for Environmental Sound Classification](https://github.com/karolpiczak/ESC-50). This will allow us to create a more generic model that is trained on a broader dataset, and then use Transfer Learning later on to fine tune it for our specific audio classification task.\n",
        "\n",
        "This model will be trained on the ESC-50 dataset, which contains 50 types of sounds; each sound category has 40 audio files that are 5 seconds each in length. Each audio file will be split into 1 second soundbites, and any soundbites that contain pure silence will be discarded.\n",
        "\n",
        "### Prepare dataset\n",
        "\n",
        "#### Download and extract\n",
        "\n",
        "The ESC-50 dataset will be downloaded and extracted to the `datasets` folder using the [`tf.keras.utils.get_file(...)`](https://www.tensorflow.org/api_docs/python/tf/keras/utils/get_file) function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EVM5_CQdoEAZ"
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "tf.keras.utils.get_file('esc-50.zip',\n",
        "                        'https://github.com/karoldvl/ESC-50/archive/master.zip',\n",
        "                        cache_dir='./',\n",
        "                        cache_subdir='datasets',\n",
        "                        extract=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tH8NUBY90nlH"
      },
      "source": [
        "#### Load dataset metadata\n",
        "\n",
        "Now we will use the [pandas](https://pandas.pydata.org/) library to read the `datasets/ESC-50-master/meta/esc50.csv` file which contains the metadata for the audio files in the ESC-50 dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UJnoX0I70r2j"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "esc50_csv = './datasets/ESC-50-master/meta/esc50.csv'\n",
        "base_data_path = './datasets/ESC-50-master/audio/'\n",
        "\n",
        "df = pd.read_csv(esc50_csv)\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5SMgChs01Jry"
      },
      "source": [
        "Then add new column with the `fullpath` of the wave files:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mioGTM5M1TQN"
      },
      "source": [
        "from os import path\n",
        "\n",
        "base_data_path = './datasets/ESC-50-master/audio/'\n",
        "\n",
        "df['fullpath'] = df['filename'].map(lambda x: path.join(base_data_path, x))\n",
        "\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LqnpuM-G2jDr"
      },
      "source": [
        "#### Load wave file data\n",
        "\n",
        "We can then define a new function named `load_wav` to load audio samples from a wave file using TensorFlow's [`tf.io.read_file(...)`](https://www.tensorflow.org/api_docs/python/tf/io/read_file) and[`tf.audio.decode_wav(...)`](https://www.tensorflow.org/api_docs/python/tf/audio/decode_wav) API's. The [`tfio.audio.resample(...)`](https://www.tensorflow.org/io/api_docs/python/tfio/audio/resample) API will be used to resample the audio samples at the specified sampling rate.\n",
        "\n",
        "[librosa](https://librosa.org/)'s [`load(...)`](https://librosa.org/doc/main/generated/librosa.load.html) API will be used as a fallback if TensorFlow is unable to decode the wave file.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "70RwxZs12iPZ"
      },
      "source": [
        "import tensorflow_io as tfio\n",
        "import librosa\n",
        "\n",
        "def load_wav(filename, desired_sample_rate, desired_channels):\n",
        "  try:\n",
        "    file_contents = tf.io.read_file(filename)\n",
        "    wav, sample_rate = tf.audio.decode_wav(file_contents, desired_channels=desired_channels)\n",
        "    wav = tf.squeeze(wav, axis=-1)\n",
        "  except:\n",
        "    # fallback to librosa if the wav file can be read with TF\n",
        "    filename = tf.cast(filename, tf.string)\n",
        "    wav, sample_rate = librosa.load(filename.numpy().decode('utf-8'), sr=None, mono=(desired_channels == 1))\n",
        "  \n",
        "  wav = tfio.audio.resample(wav, rate_in=tf.cast(sample_rate, dtype=tf.int64), rate_out=tf.cast(desired_sample_rate, dtype=tf.int64))\n",
        "\n",
        "  return wav"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P_2AYbxi5oIp"
      },
      "source": [
        "Now let's load the first wave file, which is a sound of a dog barking, from the pandas `DataFrame`, and plot it overtime using `matplotlib`. The [`IPython.display.Audio(...)`](https://ipython.org/ipython-doc/3/api/generated/IPython.display.html#IPython.display.Audio) API can be used to playback the audio samples inside the notebook.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YgLhGyi75bvI"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from IPython import display\n",
        "\n",
        "sample_rate = 16000\n",
        "channels = 1\n",
        "\n",
        "test_wav_file_path = df['fullpath'][0]\n",
        "test_wav_data = load_wav(test_wav_file_path, sample_rate, channels)\n",
        "\n",
        "plt.plot(test_wav_data)\n",
        "plt.show()\n",
        "\n",
        "display.Audio(test_wav_data, rate=sample_rate)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FLVRrhvt7E9T"
      },
      "source": [
        "If we zoom in and only plot samples `32000` to `48000`, we can get a closer plot of the audio samples in the wave file in the 2 to 3 second span:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gvKn45Pl6X4B"
      },
      "source": [
        "_ = plt.plot(test_wav_data[32000:48000])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sYg6G3ivfOPq"
      },
      "source": [
        "We can then use the [`tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) TensorFlow API to create a pipeline that loads all wave file data from the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hDr615fNfLdp"
      },
      "source": [
        "fullpaths = df['fullpath']\n",
        "targets = df['target']\n",
        "folds = df['fold']\n",
        "\n",
        "fullpaths_ds = tf.data.Dataset.from_tensor_slices((fullpaths, targets, folds))\n",
        "fullpaths_ds.element_spec"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v2IJbEHPf2ok"
      },
      "source": [
        "Map each `fullpath` value to wave file samples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u-A6jYUz6lfI"
      },
      "source": [
        "def load_wav_for_map(fullpath, label, fold):\n",
        "  wav = tf.py_function(load_wav, [fullpath, sample_rate, channels], tf.float32)\n",
        "\n",
        "  return wav, label, fold\n",
        "\n",
        "wav_ds = fullpaths_ds.map(load_wav_for_map)\n",
        "wav_ds.element_spec"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V2RD4zXVRAJ2"
      },
      "source": [
        "#### Split Wave file data\n",
        "\n",
        "We would like to train the model on 1 secound soundbites, so we must split up the 5 seconds of audio per item in the ESC-50 dataset to slices of 16000 samples. We will also stride the original audio samples `4000` samples at a time, and filter out any sound chunks that contain pure silence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UnDQ_PDERIJl"
      },
      "source": [
        "@tf.function\n",
        "def split_wav(wav, width, stride):\n",
        "  return tf.map_fn(fn=lambda t: wav[t * stride:t * stride + width], elems=tf.range((tf.shape(wav)[0] - width) // stride), fn_output_signature=tf.float32)\n",
        "\n",
        "@tf.function\n",
        "def wav_not_empty(wav):\n",
        "  return tf.experimental.numpy.any(wav)\n",
        "\n",
        "def split_wav_for_flat_map(wav, label, fold):\n",
        "  wavs = split_wav(wav, width=16000, stride=4000)\n",
        "  labels = tf.repeat(label, tf.shape(wavs)[0])\n",
        "  folds = tf.repeat(fold, tf.shape(wavs)[0])\n",
        "\n",
        "  return tf.data.Dataset.from_tensor_slices((wavs, labels, folds))\n",
        "\n",
        "split_wav_ds = wav_ds.flat_map(split_wav_for_flat_map)\n",
        "split_wav_ds = split_wav_ds.filter(lambda x, y, z: wav_not_empty(x))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pyiCY4I-9AFS"
      },
      "source": [
        "Let's plot the first 5 soundbites over time using `matplotlib`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qeRO3Z4khchs"
      },
      "source": [
        "for wav, _, _ in split_wav_ds.take(5):\n",
        "  _ = plt.plot(wav)\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XdV6yAYYhsBw"
      },
      "source": [
        "#### Create Spectrograms\n",
        "\n",
        "Rather than passing in the time series data directly into our TensorFlow model, we will transform the audio data into an audio spectrogram representation. This will create a 2D representation of the audio signal’s frequency content over time.\n",
        "\n",
        "The input audio signal we will use will have a sampling rate of 16kHz, this means one second of audio will contain 16,000 samples. Using TensorFlow’s [`tf.signal.stft(...)`](https://www.tensorflow.org/api_docs/python/tf/signal/stft) function we can transform a 1 second audio signal into a 2D tensor representation. We will choose a frame length of 256 and a frame step of 128, so the output of this feature extraction stage will be a Tensor that has a shape of `(124, 129)`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PzOXIaNkh9jW"
      },
      "source": [
        "@tf.function\n",
        "def create_spectrogram(samples):\n",
        "  return tf.abs(\n",
        "      tf.signal.stft(samples, frame_length=256, frame_step=128)\n",
        "  )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ts4rVQvG9kgL"
      },
      "source": [
        "Let's take the same 2 - 3 second interval of the first dog barking wave file and create it's spectrogram representation:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NuGcCinfilFs"
      },
      "source": [
        "spectrogram = create_spectrogram(test_wav_data[32000:48000])\n",
        "\n",
        "spectrogram.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mFq1Mpq_-Ecc"
      },
      "source": [
        "We can then create `plot_spectrogram` function to plot the spectrogram representation using `matplotlib`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hrj38Jdiig3H"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "def plot_spectrogram(spectrogram, vmax=None):\n",
        "  transposed_spectrogram = tf.transpose(spectrogram)\n",
        "\n",
        "  fig = plt.figure(figsize=(8,6))\n",
        "  height = transposed_spectrogram.shape[0]\n",
        "  X = np.arange(transposed_spectrogram.shape[1])\n",
        "  Y = np.arange(height * int(sample_rate / 256), step=int(sample_rate / 256))\n",
        "\n",
        "  im = plt.pcolormesh(X, Y, tf.transpose(spectrogram), vmax=vmax)\n",
        "\n",
        "  fig.colorbar(im)\n",
        "  plt.show()\n",
        "\n",
        "\n",
        "plot_spectrogram(spectrogram)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "THC4DDLg-SyA"
      },
      "source": [
        "Then we can map each split wave item to a spectrogram:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xw-KTABdvSpG"
      },
      "source": [
        "def create_spectrogram_for_map(samples, label, fold):\n",
        "  return create_spectrogram(samples), label, fold\n",
        "\n",
        "spectrograms_ds = split_wav_ds.map(create_spectrogram_for_map)\n",
        "spectrograms_ds.element_spec"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nMFP7b-l-kOb"
      },
      "source": [
        "Let's plot the first 5 spectrograms in the dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cc3fqcs6lMal"
      },
      "source": [
        "for s, _, _ in spectrograms_ds.take(5):\n",
        "  plot_spectrogram(s)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OrCmiQVyQ_NB"
      },
      "source": [
        "### Split Dataset\n",
        "\n",
        "Before we start training the ML classifier model, we must split the dataset up in three parts: training, validation, and test.\n",
        "\n",
        "We will use the same technique in TensorFlow's [Transfer learning with YAMNet for environmental sound classification](https://www.tensorflow.org/tutorials/audio/transfer_learning_audio#split_the_data) guide, and use the `fold` column of the ESC-50 dataset to determine the split.\n",
        "\n",
        "Before splitting the dataset, let's set a random seed for reproducibility:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T3bR-BHBRCYi"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "\n",
        "# Set seed for experiment reproducibility\n",
        "random_seed = 42\n",
        "tf.random.set_seed(random_seed)\n",
        "np.random.seed(random_seed)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kq3kDVG9_nhW"
      },
      "source": [
        "Entries with a `fold` value of less than 4 will used for training, the ones with a `value` will be used for validation, and finally the remaining items with be used for testing.\n",
        "\n",
        "The `fold` column will be removed as it is no longer needed, and the dimensions of the spectrogram shape will be expanded from `(124, 129)` to `(124, 129, 1)`. The training items will also be shuffled."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SyclwrPnMEIh"
      },
      "source": [
        "cached_ds = spectrograms_ds.cache()\n",
        "\n",
        "train_ds = cached_ds.filter(lambda spectrogram, label, fold: fold < 4)\n",
        "val_ds = cached_ds.filter(lambda spectrogram, label, fold: fold == 4)\n",
        "test_ds = cached_ds.filter(lambda spectrogram, label, fold: fold > 4)\n",
        "\n",
        "# remove the folds column as it's no longer needed\n",
        "remove_fold_column = lambda spectrogram, label, fold: (tf.expand_dims(spectrogram, axis=-1), label)\n",
        "\n",
        "train_ds = train_ds.map(remove_fold_column)\n",
        "val_ds = val_ds.map(remove_fold_column)\n",
        "test_ds = test_ds.map(remove_fold_column)\n",
        "\n",
        "train_ds = train_ds.cache().shuffle(1000, seed=random_seed).batch(32).prefetch(tf.data.AUTOTUNE)\n",
        "val_ds = val_ds.cache().batch(32).prefetch(tf.data.AUTOTUNE)\n",
        "test_ds = test_ds.cache().batch(32).prefetch(tf.data.AUTOTUNE)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LIdHUP8mRF-9"
      },
      "source": [
        "### Train Model\n",
        "\n",
        "Now that we have the features extracted from the audio signal, we can create a model using TensorFlow’s Keras  API. The model will consist of 8 layers:\n",
        "\n",
        " 1. An input layer.\n",
        " 1. A preprocessing layer, that will resize the input tensor from 124x129x1 to 32x32x1.\n",
        " 1. A normalization layer, that will scale the input values between -1 and 1\n",
        " 1. A 2D convolution layer with: 8 filters, a kernel size of 8x8, and stride of 2x2, and ReLU activation function.\n",
        " 1. A 2D max pooling layer with size of 2x2\n",
        " 1. A flatten layer to flatten the 2D data to 1D\n",
        " 1. A dropout layer, that will help reduce overfitting during training\n",
        " 1. A dense layer with 50 outputs and a softmax activation function, which outputs the likelihood of the sound category (between 0 and 1).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pbTAU5yZA43R"
      },
      "source": [
        "Before we build the model using [Tensflow's Keras API's](https://www.tensorflow.org/api_docs/python/tf/keras), we will create normalization layer and feed in all the spectrogram dataset items."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fFLHe9y-iGmj"
      },
      "source": [
        "for spectrogram, _, _ in cached_ds.take(1):\n",
        "    input_shape = tf.expand_dims(spectrogram, axis=-1).shape\n",
        "    print('Input shape:', input_shape)\n",
        "  \n",
        "norm_layer = tf.keras.layers.experimental.preprocessing.Normalization()\n",
        "norm_layer.adapt(cached_ds.map(lambda x, y, z: tf.reshape(x, input_shape)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W3N5rys9EF8B"
      },
      "source": [
        "Define a sequential 8 layer model as described above:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XyO5ilzPPePi"
      },
      "source": [
        "baseline_model = tf.keras.models.Sequential([\n",
        "  tf.keras.layers.Input(shape=input_shape),\n",
        "  tf.keras.layers.experimental.preprocessing.Resizing(32, 32, interpolation=\"nearest\"), \n",
        "  norm_layer,\n",
        "  tf.keras.layers.Conv2D(8, kernel_size=(8,8), strides=(2, 2), activation=\"relu\"),\n",
        "  tf.keras.layers.MaxPool2D(pool_size=(2,2)),\n",
        "  tf.keras.layers.Flatten(),\n",
        "  tf.keras.layers.Dropout(0.25),\n",
        "  tf.keras.layers.Dense(50, activation='softmax')\n",
        "])\n",
        "\n",
        "baseline_model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N4PgoMF8ENpu"
      },
      "source": [
        "Compile the model with `accuracy` metrics, an Adam optimizer and a sparse categorical crossentropy loss function. As well as define early stopping and dynamic learning rate scheduler callbacks for training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RyTlP0G1QHD6"
      },
      "source": [
        "METRICS = [\n",
        "      \"accuracy\",\n",
        "]\n",
        "\n",
        "baseline_model.compile(\n",
        "    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),\n",
        "    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),\n",
        "    metrics=METRICS,\n",
        ")\n",
        "\n",
        "def scheduler(epoch, lr):\n",
        "  if epoch < 100:\n",
        "    return lr\n",
        "  else:\n",
        "    return lr * tf.math.exp(-0.1)\n",
        "\n",
        "callbacks = [\n",
        "    tf.keras.callbacks.EarlyStopping(verbose=1, patience=25), \n",
        "    tf.keras.callbacks.LearningRateScheduler(scheduler)\n",
        "]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FBUof7CcErKE"
      },
      "source": [
        "Train the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lqs-O9o8QV58"
      },
      "source": [
        "EPOCHS = 250\n",
        "history = baseline_model.fit(\n",
        "    train_ds, \n",
        "    validation_data=val_ds,  \n",
        "    epochs=EPOCHS,\n",
        "    callbacks=callbacks,\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AjnfP2hCFB9z"
      },
      "source": [
        "Evaluate the loss and accuracy of the test dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sWpXhy1eQmgH"
      },
      "source": [
        "baseline_model.evaluate(test_ds)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cnDIdyN38UVh"
      },
      "source": [
        "The baseline model has a relatively low accuracy ~24%, however in the next steps we will use it as a starting point to fine tune a more accurate model for our use case."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TkW5RFLVFIQO"
      },
      "source": [
        "Save the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZmZNjn-OR-rF"
      },
      "source": [
        "baseline_model.save(\"baseline_model\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CalvtW18FOra"
      },
      "source": [
        "Create a zip file of the saved model, for download purposes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "00l_RwHWSY-Y"
      },
      "source": [
        "!zip -r baseline_model.zip baseline_model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EopW6v06SsRu"
      },
      "source": [
        "## Transfer Learning\n",
        "\n",
        "Now we will use Transfer Learning and change the classification head of the model to train a binary classification model for fire alarm sounds.\n",
        "\n",
        "Transfer Learning is the process of retraining a model that has been developed for a task to complete a new similar task. The idea is that the model has learned transferable \"skills\" and the weights and biases can be used in other models as a starting point.\n",
        "\n",
        "Transfer learning is very common in computer vision. Big data companies spend weeks training models on ImageNet, this is not possible for most people and so people reuse the models built in these research companies to complete their own tasks. A model designed to recognise 1000 different objects in a image can be adapted to recognise other or similar objects.\n",
        "\n",
        "As humans we use transfer learning too. The skills you developed to learn to walk could also be used to learn to run later on.\n",
        "\n",
        "In a neural network, the first few layers of a model start to perform a \"feature extraction\" such as finding shapes, edges and colours. The layers later on are used as classifiers; they take the extracted features and classify them.\n",
        "\n",
        "You can find more information and visualizations about this here https://yosinski.com/deepvis.\n",
        "\n",
        "Because of this, we can assume the first few layers have learned quite general feature extraction techniques that can be applied to all similar tasks and so we can freeze all these layers. The classifier layer will need to be trained based on the new classes.\n",
        "\n",
        "To do this, we break the process into two steps:\n",
        "Freeze the \"backbone\" of the model and train the head with a fairly high learning rate. We slowly reduce the learning rate.\n",
        "Unfreeze the \"backbone\" and fine-tune the model with a low learning rate.\n",
        "\n",
        "\n",
        "### Dataset\n",
        "\n",
        "We have collected 10 fire alarm clips from [freesound.org](https://freesound.org/) and [BigSoundBank.com](https://bigsoundbank.com/).  Background noise clips from the [SpeechCommands](https://www.tensorflow.org/datasets/catalog/speech_commands) dataset, will be used for non-fire alarm sounds. This dataset is small and represents the sort of data you might expect to see in the real world. Data augmentation techniques will be used to supplement the training data we’ve collected."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "APrXeNMkZmyT"
      },
      "source": [
        "### Download datasets\n",
        "\n",
        "We've created an archive with the following wave files for you:\n",
        "\n",
        " * https://freesound.org/people/rayprice/sounds/155006/ ([CC BY 3.0 license](https://creativecommons.org/licenses/by/3.0/))\n",
        "\n",
        " * https://freesound.org/people/deleted_user_2104797/sounds/164686/ ([CC0 1.0 license](https://creativecommons.org/publicdomain/zero/1.0/))\n",
        "\n",
        " * https://freesound.org/people/AdamWeeden/sounds/255180/ ([CC BY 3.0 license](https://creativecommons.org/licenses/by/3.0/))\n",
        "\n",
        "* https://freesound.org/people/MoonlightShadow/sounds/325367/([CC0 1.0 license](https://creativecommons.org/publicdomain/zero/1.0/))\n",
        "\n",
        "* https://freesound.org/people/SpliceSound/sounds/369847/ ([CC0 1.0 license](https://creativecommons.org/publicdomain/zero/1.0/))\n",
        "\n",
        "* https://freesound.org/people/SpliceSound/sounds/369848/ ([CC0 1.0 license](https://creativecommons.org/publicdomain/zero/1.0/))\n",
        "\n",
        "* https://bigsoundbank.com/detail-0800-smoke-detector-alarm.html ([free of charge and royalty free.](https://bigsoundbank.com/droit.html))\n",
        "\n",
        "* https://bigsoundbank.com/detail-1151-smoke-detector-alarm-2.html ([free of charge and royalty free.](https://bigsoundbank.com/droit.html))\n",
        "\n",
        "* https://bigsoundbank.com/detail-1153-smoke-detector-alarm-3.html ([free of charge and royalty free.](https://bigsoundbank.com/droit.html))\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h-FUUQXMZaJQ"
      },
      "source": [
        "tf.keras.utils.get_file('fire_alarms.tar.gz',\n",
        "                        'https://github.com/ArmDeveloperEcosystem/ml-audio-classifier-example-for-pico/archive/refs/heads/fire_alarms.tar.gz',\n",
        "                        cache_dir='./',\n",
        "                        cache_subdir='datasets',\n",
        "                        extract=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CpS5FBOJaK0z"
      },
      "source": [
        "# Since we only need the files in the _background_noise_ folder of the dataset\n",
        "# use the curl command to download the archive file and then manually extract\n",
        "# using the tar command, instead of using tf.keras.utils.get_file(...) \n",
        "# in Python\n",
        "\n",
        "!mkdir -p datasets/speech_commands\n",
        "!curl -L -o datasets/speech_commands_v0.02.tar.gz http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz\n",
        "!tar --wildcards --directory datasets/speech_commands -xzvf datasets/speech_commands_v0.02.tar.gz './_background_noise_/*'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2NoayTieb_WR"
      },
      "source": [
        "### Load dataset\n",
        "\n",
        "Instead of using a pandas DataFrame to load the dataset, we will load the fire alarm files and background noise files separately. The `label` and `fold` values will be mapped manually."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "owe9kEqBcWR0"
      },
      "source": [
        "fire_alarm_files_ds = tf.data.Dataset.list_files(\"datasets/ml-audio-classifier-example-for-pico-fire_alarms/*.wav\", shuffle=False)\n",
        "fire_alarm_files_ds = fire_alarm_files_ds.map(lambda x: (x, 1, -1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "StJH-68kbYvF"
      },
      "source": [
        "background_noise_files_ds = tf.data.Dataset.list_files(\"datasets/speech_commands/_background_noise_/*.wav\", shuffle=False)\n",
        "background_noise_files_ds = background_noise_files_ds.map(lambda x: (x, 0, -1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TMOJWhoFcm-5"
      },
      "source": [
        "fire_alarm_wav_ds = fire_alarm_files_ds.map(load_wav_for_map)\n",
        "fire_alarm_wav_ds = fire_alarm_wav_ds.cache()\n",
        "\n",
        "background_noise_wav_ds = background_noise_files_ds.map(load_wav_for_map)\n",
        "background_noise_wav_ds = background_noise_wav_ds.cache()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l29lUf44JB1r"
      },
      "source": [
        "Let's plot and listen to the first fire alarm file:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2TpRuruRTwpi"
      },
      "source": [
        "for wav_data, _, _ in fire_alarm_wav_ds.take(1):\n",
        "  plt.plot(wav_data)\n",
        "  plt.ylim([-1, 1])\n",
        "  plt.show()\n",
        "\n",
        "  display.display(display.Audio(wav_data, rate=sample_rate))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3i6TmVUUJeuh"
      },
      "source": [
        "Then do the same for the first background noise file:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CIxUgvNaZWWN"
      },
      "source": [
        "for wav_data, _, _ in background_noise_wav_ds.take(1):\n",
        "  plt.plot(wav_data)\n",
        "  plt.ylim([-1, 1])\n",
        "  plt.show()\n",
        "\n",
        "  display.display(display.Audio(wav_data, rate=sample_rate))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KNoTz4FpJv8i"
      },
      "source": [
        "Then split the audio samples into 1 second soundbites:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JYJT95GgpX4p"
      },
      "source": [
        "split_fire_alarm_wav_ds = fire_alarm_wav_ds.flat_map(split_wav_for_flat_map)\n",
        "split_fire_alarm_wav_ds = split_fire_alarm_wav_ds.filter(lambda x, y, z: wav_not_empty(x))\n",
        "\n",
        "split_background_noise_wav_ds = background_noise_wav_ds.flat_map(split_wav_for_flat_map)\n",
        "split_background_noise_wav_ds = split_background_noise_wav_ds.filter(lambda x, y, z: wav_not_empty(x))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LICTjFR3J8uO"
      },
      "source": [
        "TensorFlow Lite for Microcontroller (TFLu) provides a subset of TensorFlow operations, so we are unable to use the `tf.signal.sft(...)` API we’ve used for feature extraction of the baseline model on our MCU. However, we can leverage Arm’s CMSIS-DSP library to generate spectrograms on the MCU. CMSIS-DSP contains support for both floating-point and fixed-point DSP operations which are optimized for Arm Cortex-M processors, including the Arm Cortex-M0+ that we will be deploying the ML model to. The Arm Cortex-M0+ does not contain a floating-point unit (FPU) so it would be better to leverage a 16-bit fixed-point DSP based feature extraction pipeline on the board.\n",
        "\n",
        "We can leverage CMSIS-DSP’s Python Wrapper in the notebook to perform the same operations on our training pipeline using 16-bit fixed-point math. At a high level we can replicate the TensorFlow SFT API with the following CMSIS-DSP based operations:\n",
        "\n",
        " 1. Manually creating a Hanning Window of length 256 using the Hanning Window formula along with CMSIS-DSP’s `arm_cos_f32` API.\n",
        " 1. Creating a CMSIS-DSP `arm_rfft_instance_q15` instance and initializing it using CMSIS-DSP’s `arm_rfft_init_q15` API. \n",
        " 1. Looping through the audio data 256 samples at a time, with a stride of 128 (this matches the parameters we’ve passed into the TF sft API)\n",
        "  1. Multiplying the 256 samples by the Hanning Window, using CMSIS-DSP’s `arm_mult_q15` API\n",
        "  1. Calculating the FFT of the output of the previous step, using CMSIS-DSP’s `arm_rfft_q15` API\n",
        "  1. Calculating the magnitude of the previous step, using CMSIS-DSP’s `arm_cmplx_mag_q15` API\n",
        " 1. Each audio soundbites’s FFT magnitude represents the one column of the spectrogram.\n",
        " 1. Since our baseline model expects a floating-point input, instead of the 16-bit quantized value we were using, the CMSIS-DSP `arm_q15_to_float` API can be used to convert the spectrogram data from a 16-bit fixed-point value to a floating-point value for training.\n",
        "\n",
        "For an in-depth description of how to create audio spectrograms using fixed-point operations with CMSIS-DSP, please see the [Towards Data Science “Fixed-point DSP for Data Scientists” guide](https://towardsdatascience.com/fixed-point-dsp-for-data-scientists-d773a4271f7f).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cdhrliYiAfKE"
      },
      "source": [
        "import cmsisdsp\n",
        "from numpy import pi as PI\n",
        "\n",
        "window_size = 256\n",
        "step_size = 128\n",
        "\n",
        "hanning_window_f32 = np.zeros(window_size)\n",
        "for i in range(window_size):\n",
        "  hanning_window_f32[i] = 0.5 * (1 - cmsisdsp.arm_cos_f32(2 * PI * i / window_size ))\n",
        "\n",
        "hanning_window_q15 = cmsisdsp.arm_float_to_q15(hanning_window_f32)\n",
        "\n",
        "rfftq15 = cmsisdsp.arm_rfft_instance_q15()\n",
        "status = cmsisdsp.arm_rfft_init_q15(rfftq15, window_size, 0, 1)\n",
        "\n",
        "def get_arm_spectrogram(waveform):\n",
        "  \n",
        "  num_frames = int(1 + (len(waveform) - window_size) // step_size)\n",
        "  fft_size = int(window_size // 2 + 1)\n",
        "\n",
        "  # Convert the audio to q15\n",
        "  waveform_q15 = cmsisdsp.arm_float_to_q15(waveform)\n",
        "\n",
        "  # Create empty spectrogram array\n",
        "  spectrogram_q15 = np.empty((num_frames, fft_size), dtype = np.int16)\n",
        "\n",
        "  start_index = 0\n",
        "\n",
        "  for index in range(num_frames):\n",
        "    # Take the window from the waveform.\n",
        "    window = waveform_q15[start_index:start_index + window_size]\n",
        "\n",
        "    # Apply the Hanning Window.\n",
        "    window = cmsisdsp.arm_mult_q15(window, hanning_window_q15)\n",
        "\n",
        "    # Calculate the FFT, shift by 7 according to docs\n",
        "    window = cmsisdsp.arm_rfft_q15(rfftq15, window)\n",
        "\n",
        "    # Take the absolute value of the FFT and add to the Spectrogram.\n",
        "    spectrogram_q15[index] = cmsisdsp.arm_cmplx_mag_q15(window)[:fft_size]\n",
        "\n",
        "    # Increase the start index of the window by the overlap amount.\n",
        "    start_index += step_size\n",
        "\n",
        "  # Convert to numpy output ready for keras\n",
        "  return cmsisdsp.arm_q15_to_float(spectrogram_q15).reshape(num_frames,fft_size) * 512"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9hHnibH7LCz1"
      },
      "source": [
        "Let's create a spectrogram representation for all of the fire alarm soundbites, and plot the first spectrogram."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BRDqTa1CZfEA"
      },
      "source": [
        "@tf.function\n",
        "def create_arm_spectrogram_for_map(wav, label, fold):\n",
        "  spectrogram = tf.py_function(get_arm_spectrogram, [wav], tf.float32)\n",
        "\n",
        "  return spectrogram, label, fold\n",
        "\n",
        "fire_alarm_spectrograms_ds = split_fire_alarm_wav_ds.map(create_arm_spectrogram_for_map)\n",
        "fire_alarm_spectrograms_ds = fire_alarm_spectrograms_ds.cache()\n",
        "\n",
        "for spectrogram, _, _ in fire_alarm_spectrograms_ds.take(1):\n",
        "  plot_spectrogram(spectrogram)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RZ-fpDdkLyyy"
      },
      "source": [
        "The do the same for the background noise soundbites:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "haGn0RT9Z1FI"
      },
      "source": [
        "background_noise_spectrograms_ds = split_background_noise_wav_ds.map(create_arm_spectrogram_for_map)\n",
        "background_noise_spectrograms_ds = background_noise_spectrograms_ds.cache()\n",
        "\n",
        "for spectrogram, _, _ in background_noise_spectrograms_ds.take(1):\n",
        "  plot_spectrogram(spectrogram)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wi-trTZtL6FG"
      },
      "source": [
        "Now let's calculate the lengths of each dataset to see how balanced they are:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ouhaiuLzMFiN"
      },
      "source": [
        "def calculate_ds_len(ds):\n",
        "  count = 0\n",
        "  for _, _, _ in ds:\n",
        "    count += 1\n",
        "  \n",
        "  return count\n",
        "\n",
        "num_fire_alarm_spectrograms = calculate_ds_len(fire_alarm_spectrograms_ds)\n",
        "num_background_noise_spectrograms = calculate_ds_len(background_noise_spectrograms_ds)\n",
        "\n",
        "print(f\"num_fire_alarm_spectrograms = {num_fire_alarm_spectrograms}\")\n",
        "print(f\"num_background_noise_spectrograms = {num_background_noise_spectrograms}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EigYdlAXMpkP"
      },
      "source": [
        "We can see there a more background noise samples than fire alarm samples. In the next section we will use data augmentation to balance this."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sxYKzLPCmIrO"
      },
      "source": [
        "### Data Augmentation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iGZI32bKMI9Z"
      },
      "source": [
        "Data augmentation is a set of techniques used to increase the size of a dataset. This is done by slightly modifying samples from the dataset or by creating synthetic data. In this situation we are using audio and we will create a few functions to augment the different samples. We will use three techniques:\n",
        "\n",
        " * adding white noise to the audio samples\n",
        " * adding random silence to the audio\n",
        " * mixing two audio samples together\n",
        "\n",
        "As well as increasing the size of the dataset, data augmentation also helps to reduce overfitting by training the model on different (not perfect) data samples. For example, on a microcontroller you are unlikely to have perfect high quality audio, and so a technique like adding white noise can help the model work in situations where your microphone might every so often have noise in there.\n",
        "\n",
        "First let's plot the time representation of the first fire alarm soundbite over time along with it's spectrogram representation, so we can compare against the augmented versions.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sjWJd9Zdo4NH"
      },
      "source": [
        "for wav, _, _ in split_fire_alarm_wav_ds.take(1):\n",
        "  test_fire_alarm_wav = wav\n",
        "\n",
        "plt.plot(test_fire_alarm_wav)\n",
        "plt.ylim([-1, 1])\n",
        "plt.show()\n",
        "\n",
        "plot_spectrogram(get_arm_spectrogram(test_fire_alarm_wav), vmax=25)\n",
        "\n",
        "display.display(display.Audio(test_fire_alarm_wav, rate=sample_rate))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MDSEeTsLo5aN"
      },
      "source": [
        "#### White Noise"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61lkrtSuNIS1"
      },
      "source": [
        "TensorFlow's [`tf.random.uniform(...)`](https://www.tensorflow.org/api_docs/python/tf/random/uniform) API can be used generate a Tensor of equal shape to the original audio. This Tensor can then be multiplied by a random scalar, and then added to the original audio samples. The [`tf.clip_by_value(...)`](https://www.tensorflow.org/api_docs/python/tf/clip_by_value) API will also be used to ensure the audio remains in the range of -1.0 to 1.0."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mXV0un4cmRTY"
      },
      "source": [
        "def add_white_noise(audio):\n",
        "  #generate noise and the scalar multiplier\n",
        "  noise = tf.random.uniform(shape=tf.shape(audio), minval=-1, maxval=1)\n",
        "  noise_scalar = tf.random.uniform(shape=[1], minval=0, maxval=0.2)\n",
        "\n",
        "  # add them to the original audio\n",
        "  audio_with_noise = audio + (noise * noise_scalar)\n",
        "  \n",
        "  # final clip the values to ensure they are still between -1 and 1\n",
        "  audio_with_noise = tf.clip_by_value(audio_with_noise, clip_value_min=-1, clip_value_max=1)\n",
        "  \n",
        "  return audio_with_noise"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6z-wGwCmNx4x"
      },
      "source": [
        "Let's apply the white noise to the fire alarm sound and then plot it to compare. We can also listen to the difference."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FsffnIb7paHj"
      },
      "source": [
        "test_fire_alarm_with_white_noise_wav = add_white_noise(test_fire_alarm_wav)\n",
        "\n",
        "plt.plot(test_fire_alarm_with_white_noise_wav)\n",
        "plt.ylim([-1, 1])\n",
        "plt.show()\n",
        "\n",
        "plot_spectrogram(get_arm_spectrogram(test_fire_alarm_with_white_noise_wav), vmax=25)\n",
        "\n",
        "display.display(display.Audio(test_fire_alarm_with_white_noise_wav, rate=sample_rate))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KCE-90pJpoio"
      },
      "source": [
        "#### Random Silence\n",
        "\n",
        "TensorFlow's [`tf.random.categorical(...)`](https://www.tensorflow.org/api_docs/python/tf/random/categorical) API can be used generate a Tensor of equal shape to the original audio containing mask of `True` or `False`. This mask can then be casted to a float type of 1.0 or 0.0, so that it can be multiplied by the original audio single to create random periods of silence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rSNkZPqPnW_k"
      },
      "source": [
        "def add_random_silence(audio):\n",
        "  audio_mask = tf.random.categorical(tf.math.log([[0.2, 0.8]]), num_samples=tf.shape(audio)[0])\n",
        "  audio_mask = tf.cast(audio_mask, dtype=tf.float32)\n",
        "  audio_mask = tf.squeeze(audio_mask, axis=0)\n",
        "\n",
        "  # multiply the audio input by the mask\n",
        "  augmented_audio = audio * audio_mask\n",
        "  \n",
        "  return augmented_audio"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BQ2p-MiwOzm9"
      },
      "source": [
        "Let's apply the random silence to the fire alarm sound and then plot it to compare. We can also listen to the difference."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wl94M9Ryp9zJ"
      },
      "source": [
        "test_fire_alarm_with_random_silence_wav = add_random_silence(test_fire_alarm_wav)\n",
        "\n",
        "plt.plot(test_fire_alarm_with_random_silence_wav)\n",
        "plt.ylim([-1, 1])\n",
        "plt.show()\n",
        "\n",
        "plot_spectrogram(get_arm_spectrogram(test_fire_alarm_with_random_silence_wav), vmax=25)\n",
        "\n",
        "display.display(display.Audio(test_fire_alarm_with_random_silence_wav, rate=sample_rate))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oSfVXJhcs1sn"
      },
      "source": [
        "#### Audio Mixups\n",
        "\n",
        "We can combine a fire alarm soundbite with a background noise soundbite to create a mixed up version of the two.\n",
        "\n",
        "Let's select the first background noise soundbite do see how this can be done."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2mfaofOktV19"
      },
      "source": [
        "for wav, _, _ in split_background_noise_wav_ds.take(1):\n",
        "  test_background_noise_wav = wav\n",
        "\n",
        "plt.plot(test_background_noise_wav)\n",
        "plt.ylim([-1, 1])\n",
        "plt.show()\n",
        "\n",
        "plot_spectrogram(get_arm_spectrogram(test_background_noise_wav))\n",
        "\n",
        "display.display(display.Audio(test_background_noise_wav, rate=sample_rate))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SmiYtvxwPXd0"
      },
      "source": [
        "We will multiply the background noise soundbite with a random scalar before adding it to the original fire alarm soundbite. Then ensure the mixed up value is between the range of -1.0 and 1.0."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vdzr8Hy9szgl"
      },
      "source": [
        "def add_audio_mixup(audio, mixup_audio):\n",
        "  # randomly generate a scalar\n",
        "  noise_scalar = tf.random.uniform(shape=[1], minval=0, maxval=1)\n",
        "\n",
        "  # add the background noise to the audio\n",
        "  augmented_audio = audio + (mixup_audio * noise_scalar)\n",
        "  \n",
        "  #final clip the values so they are stil between -1 and 1\n",
        "  augmented_audio = tf.clip_by_value(augmented_audio, clip_value_min=-1, clip_value_max=1)\n",
        "  \n",
        "  return augmented_audio"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0FvodQe0Pu8w"
      },
      "source": [
        "Let's apply the audio mixup to the fire alarm sound and then plot it to compare. We can also listen to the difference."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "82af5IyPuqQU"
      },
      "source": [
        "test_fire_alarm_with_mixup_wav = add_audio_mixup(test_fire_alarm_wav, test_background_noise_wav)\n",
        "\n",
        "plt.plot(test_fire_alarm_with_mixup_wav)\n",
        "plt.ylim([-1, 1])\n",
        "plt.show()\n",
        "\n",
        "plot_spectrogram(get_arm_spectrogram(test_fire_alarm_with_mixup_wav), vmax=25)\n",
        "\n",
        "display.display(display.Audio(test_fire_alarm_with_mixup_wav, rate=sample_rate))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ntHK8oSHvl3N"
      },
      "source": [
        "### Create Augmented Dataset\n",
        "\n",
        "We can now combine all three augmententation techniques to balance our dataset.\n",
        "\n",
        "First let's calculate how many augmented files we need to generate:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gFrYCTPYL0Vs"
      },
      "source": [
        "num_augmented_fire_alarm_spectrograms = num_background_noise_spectrograms - num_fire_alarm_spectrograms\n",
        "\n",
        "print(f'num_augmented_fire_alarm_spectrograms = {num_augmented_fire_alarm_spectrograms}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2RkWzf_qQSlG"
      },
      "source": [
        "Then we can divide by 3, to calculate how many augmented soundbites per technique to generate:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KpJwqgWbNLCk"
      },
      "source": [
        "num_white_noise_fire_alarm_spectrograms = num_augmented_fire_alarm_spectrograms // 3\n",
        "num_random_silence_fire_alarm_spectrograms = num_augmented_fire_alarm_spectrograms // 3\n",
        "num_audio_mixup_fire_alarm_spectrograms = num_augmented_fire_alarm_spectrograms // 3\n",
        "\n",
        "print(f'num_white_noise_fire_alarm_spectrograms = {num_white_noise_fire_alarm_spectrograms}')\n",
        "print(f'num_random_silence_fire_alarm_spectrograms = {num_random_silence_fire_alarm_spectrograms}')\n",
        "print(f'num_audio_mixup_fire_alarm_spectrograms = {num_audio_mixup_fire_alarm_spectrograms}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y12lDIeNQe7e"
      },
      "source": [
        "Select and shuffle the number of soundbites required:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c0_hV-Y8vrFx"
      },
      "source": [
        "split_fire_alarm_wav_ds = split_fire_alarm_wav_ds.cache()\n",
        "preaugmented_split_fire_alarm_wav = split_fire_alarm_wav_ds.shuffle(num_augmented_fire_alarm_spectrograms, seed=random_seed).take(num_augmented_fire_alarm_spectrograms)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VQKWXZSuQk2T"
      },
      "source": [
        "Create the white noise augmented soundbites:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mBu718tfNxLY"
      },
      "source": [
        "def add_white_noise_for_map(wav, label, fold):\n",
        "  return add_white_noise(wav), label, fold\n",
        "\n",
        "white_noise_fire_alarm_wav_ds = preaugmented_split_fire_alarm_wav.take(num_white_noise_fire_alarm_spectrograms)\n",
        "white_noise_fire_alarm_wav_ds = white_noise_fire_alarm_wav_ds.map(add_white_noise_for_map)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tK2pAnK-QqIq"
      },
      "source": [
        "Create the random noise augmented soundbites:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0oKTJN6uPVIT"
      },
      "source": [
        "def add_random_silence_for_map(wav, label, fold):\n",
        "  return add_random_silence(wav), label, fold\n",
        "\n",
        "random_silence_fire_alarm_wav_ds = preaugmented_split_fire_alarm_wav.skip(num_white_noise_fire_alarm_spectrograms)\n",
        "random_silence_fire_alarm_wav_ds = random_silence_fire_alarm_wav_ds.take(num_random_silence_fire_alarm_spectrograms)\n",
        "random_silence_fire_alarm_wav_ds = random_silence_fire_alarm_wav_ds.map(add_random_silence_for_map)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FmYVy-VwQx_J"
      },
      "source": [
        "Create the audio mixup augmented soundbites:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DQ0jfAJVQre7"
      },
      "source": [
        "audio_mixup_background_noise_ds = split_background_noise_wav_ds.shuffle(num_audio_mixup_fire_alarm_spectrograms).take(num_audio_mixup_fire_alarm_spectrograms)\n",
        "audio_mixup_background_noise_iter = iter(audio_mixup_background_noise_ds.map(lambda x, y, z: x))\n",
        "\n",
        "def add_audio_mixup_for_map(wav, label, fold):\n",
        "  return add_audio_mixup(wav, next(audio_mixup_background_noise_iter)), label, fold\n",
        "\n",
        "audio_mixup_split_fire_alarm_wav_ds = preaugmented_split_fire_alarm_wav.skip(num_white_noise_fire_alarm_spectrograms + num_random_silence_fire_alarm_spectrograms)\n",
        "audio_mixup_split_fire_alarm_wav_ds = audio_mixup_split_fire_alarm_wav_ds.take(num_audio_mixup_fire_alarm_spectrograms)\n",
        "audio_mixup_split_fire_alarm_wav_ds = audio_mixup_split_fire_alarm_wav_ds.map(add_audio_mixup_for_map)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cx-yL-pdQ5KF"
      },
      "source": [
        "Combine all the augmented soundbites together and map them to their spectrogram representations:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N85bl1Xx1C4l"
      },
      "source": [
        "augment_split_fire_alarm_wav_ds = tf.data.Dataset.concatenate(white_noise_fire_alarm_wav_ds, random_silence_fire_alarm_wav_ds)\n",
        "augment_split_fire_alarm_wav_ds = tf.data.Dataset.concatenate(augment_split_fire_alarm_wav_ds, audio_mixup_split_fire_alarm_wav_ds)\n",
        "\n",
        "augment_fire_alarm_spectrograms_ds = augment_split_fire_alarm_wav_ds.map(create_arm_spectrogram_for_map)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vD6zn_eWu-YS"
      },
      "source": [
        "### Split Dataset\n",
        "\n",
        "Now combine the spectrogram datasets, and split them into training, validation, and test sets. Instead of using the `fold` value to split them, we will shuffle all the items, and then split by percentage."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UmQxp4EtFiVn"
      },
      "source": [
        "full_ds = tf.data.Dataset.concatenate(fire_alarm_spectrograms_ds, background_noise_spectrograms_ds)\n",
        "full_ds = tf.data.Dataset.concatenate(full_ds, augment_fire_alarm_spectrograms_ds)\n",
        "full_ds = full_ds.cache()\n",
        "\n",
        "full_ds_size = calculate_ds_len(full_ds)\n",
        "\n",
        "print(f'full_ds_size = {full_ds_size}')\n",
        "\n",
        "full_ds = full_ds.shuffle(full_ds_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3CltJjc1Dgjl"
      },
      "source": [
        "train_ds_size = int(0.60 * full_ds_size)\n",
        "val_ds_size = int(0.20 * full_ds_size)\n",
        "test_ds_size = int(0.20 * full_ds_size)\n",
        "\n",
        "train_ds = full_ds.take(train_ds_size)\n",
        "\n",
        "remaining_ds = full_ds.skip(train_ds_size) \n",
        "val_ds = remaining_ds.take(val_ds_size)\n",
        "test_ds = remaining_ds.skip(val_ds_size)\n",
        "\n",
        "# remove the folds column as it's no longer needed\n",
        "remove_fold_column = lambda spectrogram, label, fold: (tf.expand_dims(spectrogram, axis=-1), label)\n",
        "\n",
        "train_ds = train_ds.map(remove_fold_column)\n",
        "val_ds = val_ds.map(remove_fold_column)\n",
        "test_ds = test_ds.map(remove_fold_column)\n",
        "\n",
        "train_ds = train_ds.cache().shuffle(1000).batch(32).prefetch(tf.data.AUTOTUNE)\n",
        "val_ds = val_ds.cache().batch(32).prefetch(tf.data.AUTOTUNE)\n",
        "test_ds = test_ds.cache().batch(32).prefetch(tf.data.AUTOTUNE)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ffwTQt_vFAx"
      },
      "source": [
        "### Replace Baseline Model Classification Head and Train Model\n",
        "\n",
        "The model we previously trained on the ESC-50 dataset, predicted the presence of 50 sound types, and which resulted in the final dense layer of the model having 50 outputs. The new model we would like to create is a binary classifier, and needs to have a single output value.\n",
        "\n",
        "We will load the baseline model, and swap out the final dense layer to match our needs:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tqY5I0QeGKAV"
      },
      "source": [
        "# we need a new head with one neuron.\n",
        "model_body = tf.keras.Model(inputs=baseline_model.input, outputs=baseline_model.layers[-2].output)\n",
        "\n",
        "classifier_head = tf.keras.layers.Dense(1, activation=\"sigmoid\")(model_body.output)\n",
        "\n",
        "fine_tune_model = tf.keras.Model(model_body.input, classifier_head)\n",
        "\n",
        "fine_tune_model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w8Sla0atSL1G"
      },
      "source": [
        "To freeze a layer in TensorFlow we can set `layer.trainable = False`. Let's loop through all the layers and do this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lbcW4ANCGXUa"
      },
      "source": [
        "for layer in fine_tune_model.layers:\n",
        "  layer.trainable = False"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MY9pZMYjSSHV"
      },
      "source": [
        "and now unfreeze the last layer (the head):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HMBdchlqSToY"
      },
      "source": [
        "fine_tune_model.layers[-1].trainable = True"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WuE1ONbpSWil"
      },
      "source": [
        "Then we can `compile` the model, this time with using a binary crossentropy loss function as this model contains a single output."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CUkVzymnGOOV"
      },
      "source": [
        "METRICS = [\n",
        "      \"accuracy\",\n",
        "]\n",
        "\n",
        "fine_tune_model.compile(\n",
        "    optimizer=tf.keras.optimizers.Adam(learning_rate=0.01),\n",
        "    loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n",
        "    metrics=METRICS,\n",
        ")\n",
        "\n",
        "def scheduler(epoch, lr):\n",
        "  if epoch < 10:\n",
        "    return lr\n",
        "  else:\n",
        "    return lr * tf.math.exp(-0.1)\n",
        "\n",
        "callbacks = [\n",
        "    tf.keras.callbacks.EarlyStopping(verbose=1, patience=5), \n",
        "    tf.keras.callbacks.LearningRateScheduler(scheduler)\n",
        "]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YblAlGy4SpRQ"
      },
      "source": [
        "Kick off training:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GoEcHNBcGqnT"
      },
      "source": [
        "EPOCHS = 25\n",
        "\n",
        "history_1 = fine_tune_model.fit(\n",
        "    train_ds, \n",
        "    validation_data=val_ds,  \n",
        "    epochs=EPOCHS,\n",
        "    callbacks=callbacks,\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KCWZ_SUkS95R"
      },
      "source": [
        "Now unfreeze all the layers, and train for a few more epochs:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CiJYDSOxe6uQ"
      },
      "source": [
        "for layer in fine_tune_model.layers:\n",
        "  layer.trainable = True\n",
        "\n",
        "fine_tune_model.compile(\n",
        "    optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001),\n",
        "    loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n",
        "    metrics=METRICS,\n",
        ")\n",
        "\n",
        "def scheduler(epoch, lr):\n",
        "  return lr * tf.math.exp(-0.1)\n",
        "\n",
        "callbacks = [\n",
        "    tf.keras.callbacks.EarlyStopping(verbose=1, patience=5), \n",
        "    tf.keras.callbacks.LearningRateScheduler(scheduler)\n",
        "]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gRq1N0qXIBSs"
      },
      "source": [
        "EPOCHS = 10\n",
        "\n",
        "history_2 = fine_tune_model.fit(\n",
        "    train_ds, \n",
        "    validation_data=val_ds,  \n",
        "    epochs=EPOCHS,\n",
        "    callbacks=callbacks,\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kOewhILQcemp"
      },
      "source": [
        "fine_tune_model.save(\"fine_tuned_model\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BXENjCJJzyrO"
      },
      "source": [
        "## Training with your own audio (optional)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AIIuAagAjoVq"
      },
      "source": [
        "We now have an ML model which can classify the presence of fire alarm sound. However this model was trained on publicly available sound recordings which might not match the sound characteristics of the hardware microphone we will use for inferencing.\n",
        "\n",
        "The Raspberry Pi RP2040 MCU has a native USB feature that allows it to act like a USB microphone. We can flash an application to the board to enable it to act like a USB microphone to our PC. Then we can extend Google Colab’s capabilities with the [Web Audio API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API) on a modern Web browser like Google Chrome to collect live data samples all from within Google Colab!\n",
        "\n",
        "**If you don't have a fire alarm handy to record from, you can skip to the [next section](#scrollTo=Model_Optimization).**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R-ajrgSMaBvI"
      },
      "source": [
        "### Record your own audio\n",
        "\n",
        "#### Software Setup\n",
        "\n",
        "Now we can use the USB microphone example from the [Microphone Library for Pico](https://github.com/ArmDeveloperEcosystem/microphone-library-for-pico). The example application can be compiled using `cmake` and `make`. Then we can flash the example application to the board over USB by putting the board into “boot ROM mode” which will allow us to upload an application to the board.\n",
        "\n",
        "Let's use `git` to clone the library source code and accompanying examples:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GBsW1c5xaFwy"
      },
      "source": [
        "%%shell\n",
        "git clone https://github.com/ArmDeveloperEcosystem/microphone-library-for-pico.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sQ9vcp6ApjXN"
      },
      "source": [
        "Now let's change to the libraries directory folder, and create a `build` folder to run `cmake` on:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0R5fJC5HaMom"
      },
      "source": [
        "%%shell\n",
        "cd microphone-library-for-pico\n",
        "mkdir -p build\n",
        "cd build\n",
        "cmake .. -DPICO_BOARD=${PICO_BOARD}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cr0Txtlip6iw"
      },
      "source": [
        "Then we can run `make` to compile the example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q7ohkHZnp_Ns"
      },
      "source": [
        "%%shell\n",
        "cd microphone-library-for-pico/build\n",
        "\n",
        "make -j usb_microphone"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m9asx3qzuIvR"
      },
      "source": [
        "#### Flashing the board\n",
        "\n",
        "If you are using a [WebUSB API](https://wicg.github.io/webusb/) enabled browser like Google Chrome, you can directly flash the image onto the board from within Google Collab! (Otherwise, you can manually download the .uf2 file to your computer and then drag it onto the USB disk for the RP2040 board.)\n",
        "\n",
        "**Note for Windows**: If you are using Windows you must install WinUSB drivers in order to use WebUSB, you can do so by following the instructions found [here](https://github.com/ArmDeveloperEcosystem/ml-audio-classifier-example-for-pico/blob/main/windows.md).\n",
        "\n",
        "**Note for Linux**: If you are using Linux you must configure udev in order to use WebUSB, you can do so by following the instructions found [here](https://github.com/ArmDeveloperEcosystem/ml-audio-classifier-example-for-pico/blob/main/linux.md).\n",
        "\n",
        "First you must place the board in USB Boot ROM mode, as follows:\n",
        "\n",
        " * SparkFun MicroMod\n",
        "  * Plug the USB-C cable into the board and your PC to power the board\n",
        "  * While holding down the BOOT button on the board, tap the RESET button\n",
        " * Raspberry Pi Pico\n",
        "  * Plug the USB Micro cable into your PC, but do NOT plug in the Pico side.\n",
        "  * While holding down the white BOOTSEL button, plug in the micro USB cable to the Pico\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hmtVMRLrvfvw"
      },
      "source": [
        "Run the code cell below and then click the \"Flash\" button to upload the USB microphone example application to the board over USB."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vhy9ddOXAZnY"
      },
      "source": [
        "from colab_utils.pico import flash_pico\n",
        "\n",
        "flash_pico('microphone-library-for-pico/build/examples/usb_microphone/usb_microphone.bin')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p1GJDEYIwI0r"
      },
      "source": [
        "Now you can record audio by running the cells below, select the \"MicNode\" item from the drop down, and then click the \"Starting Recording\" button to start capturing audio. You must click the \"Stop Recording\" button to stop recording."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wlpeIDxIxSIw"
      },
      "source": [
        "Record the your own fire alarm sounds:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Pgq9iZ7ybmt"
      },
      "source": [
        "from colab_utils.audio import record_wav_file\n",
        "\n",
        "os.makedirs('datasets/custom/fire_alarm', exist_ok=True)\n",
        "\n",
        "record_wav_file('datasets/custom/fire_alarm')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r8nXgrdTyIfB"
      },
      "source": [
        "Record the your own background noise sounds:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0AH4-FQ3zSHV"
      },
      "source": [
        "os.makedirs('datasets/custom/background_noise', exist_ok=True)\n",
        "\n",
        "record_wav_file('datasets/custom/background_noise')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NoQuE4jOzKoX"
      },
      "source": [
        "We can zip up the recorded wave files to download and use again:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cxsdP2uY0IpD"
      },
      "source": [
        "!zip -r custom.zip datasets/custom"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YqOLRYDT33Y2"
      },
      "source": [
        "### Load dataset\n",
        "\n",
        "We can load and transform the custom recorded dataset using the same pipeline we used before."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B8D_30220vlN"
      },
      "source": [
        "custom_fire_alarm_ds = tf.data.Dataset.list_files(\"datasets/custom/fire_alarm/*.wav\", shuffle=False)\n",
        "custom_fire_alarm_ds = custom_fire_alarm_ds.map(lambda x: (x, 1, -1))\n",
        "custom_fire_alarm_ds = custom_fire_alarm_ds.map(load_wav_for_map)\n",
        "custom_fire_alarm_ds = custom_fire_alarm_ds.flat_map(split_wav_for_flat_map)\n",
        "custom_fire_alarm_ds = custom_fire_alarm_ds.map(create_arm_spectrogram_for_map)\n",
        "\n",
        "custom_background_noise_ds = tf.data.Dataset.list_files(\"datasets/custom/background_noise/*.wav\", shuffle=False)\n",
        "custom_background_noise_ds = custom_background_noise_ds.map(lambda x: (x, 0, -1))\n",
        "custom_background_noise_ds = custom_background_noise_ds.map(load_wav_for_map)\n",
        "custom_background_noise_ds = custom_background_noise_ds.flat_map(split_wav_for_flat_map)\n",
        "custom_background_noise_ds = custom_background_noise_ds.map(create_arm_spectrogram_for_map)\n",
        "\n",
        "custom_ds = tf.data.Dataset.concatenate(custom_fire_alarm_ds, custom_background_noise_ds)\n",
        "custom_ds = custom_ds.map(lambda x, y, z: (tf.expand_dims(x, axis=-1), y, z))\n",
        "custom_ds_len = calculate_ds_len(custom_ds)\n",
        "\n",
        "print(f'{custom_ds_len}')\n",
        "\n",
        "custom_ds = custom_ds.map(lambda x, y,z: (x, y))\n",
        "\n",
        "custom_ds = custom_ds.shuffle(custom_ds_len).cache()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3H8L-1jp0MX2"
      },
      "source": [
        "Evaluate dataset performance before training:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sJDj7dD1PGfN"
      },
      "source": [
        "fine_tune_model.evaluate(custom_ds.batch(1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DphMcfEf3_hR"
      },
      "source": [
        "### Fine tune model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hp8R30Uc5FRc"
      },
      "source": [
        "EPOCHS = 25\n",
        "\n",
        "for layer in fine_tune_model.layers:\n",
        "  layer.trainable = False\n",
        "\n",
        "fine_tune_model.layers[-1].trainable = True\n",
        "\n",
        "fine_tune_model.compile(\n",
        "    optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001),\n",
        "    loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n",
        "    metrics=METRICS,\n",
        ")\n",
        "\n",
        "history3 = fine_tune_model.fit(\n",
        "    custom_ds.take(int(custom_ds_len * 0.8)).batch(1), \n",
        "    validation_data=custom_ds.skip(int(custom_ds_len * 0.8)).batch(1),  \n",
        "    epochs=EPOCHS,\n",
        "    # callbacks=callbacks,\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sx0iguJePXHN"
      },
      "source": [
        "fine_tune_model.evaluate(custom_ds.batch(1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "alhP_RWTPkNl"
      },
      "source": [
        "fine_tune_model.save('fine_tuned_model')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8HW-PdaT5Fx0"
      },
      "source": [
        "## Model Optimization\n",
        "\n",
        "To optimize the model to run on the Arm Cortex-M0+ processor, we will use a process called model quantization. Model quantization converts the model’s weights and biases from 32-bit floating-point values to 8-bit values. The [pico-tflmicro](https://github.com/raspberrypi/pico-tflmicro) library, which is a port of TFLu for the RP2040’s Pico SDK contains Arm’s CMSIS-NN library, which supports optimized kernel operations for quantized 8-bit weights on Arm Cortex-M processors.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2HRVOG9-5JsE"
      },
      "source": [
        "### Quantization Aware Training\n",
        "\n",
        "We can use [TensorFlow’s Quantization Aware Training (QAT)](https://www.tensorflow.org/model_optimization/guide/quantization/training) feature to easily convert the floating-point model to quantized."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XyWDswSD5PKF"
      },
      "source": [
        "final_model = tf.keras.models.load_model(\"fine_tuned_model\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2uWWv-Wd5Uvx"
      },
      "source": [
        "import tensorflow_model_optimization as tfmot\n",
        "\n",
        "def apply_qat_to_dense_and_cnn(layer):\n",
        "  if isinstance(layer, (tf.keras.layers.Dense, tf.keras.layers.Conv2D)):\n",
        "    return tfmot.quantization.keras.quantize_annotate_layer(layer)\n",
        "  return layer\n",
        "\n",
        "annotated_model = tf.keras.models.clone_model(\n",
        "    fine_tune_model,\n",
        "    clone_function=apply_qat_to_dense_and_cnn,\n",
        ")\n",
        "\n",
        "quant_aware_model = tfmot.quantization.keras.quantize_apply(annotated_model)\n",
        "quant_aware_model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jmHPDwpI5g29"
      },
      "source": [
        "quant_aware_model.compile(\n",
        "    optimizer=tf.keras.optimizers.Adam(),\n",
        "    loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n",
        "    metrics=METRICS,\n",
        ")\n",
        "\n",
        "EPOCHS=1\n",
        "quant_aware_history = quant_aware_model.fit(\n",
        "    train_ds, \n",
        "    validation_data=val_ds,  \n",
        "    epochs=EPOCHS\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rINV6dEU5sqp"
      },
      "source": [
        "### Saving model in TFLite format\n",
        "\n",
        "We will now use the [tf.lite.TFLiteConverter.from_keras_model(...)](https://www.tensorflow.org/api_docs/python/tf/lite/TFLiteConverter#from_keras_model) API to convert the quantized Keras model to TF Lite format, and then save it to disk as a `.tflite` file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rRuuJi145r8r"
      },
      "source": [
        "converter = tf.lite.TFLiteConverter.from_keras_model(quant_aware_model)\n",
        "converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
        "\n",
        "def representative_data_gen():\n",
        "  for input_value, output_value in train_ds.unbatch().batch(1).take(100):\n",
        "    # Model has only one input so each data point has one element.\n",
        "    yield [input_value]\n",
        "    \n",
        "converter.representative_dataset = representative_data_gen\n",
        "# Ensure that if any ops can't be quantized, the converter throws an error\n",
        "converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]\n",
        "# Set the input and output tensors to uint8 (APIs added in r2.3)\n",
        "converter.inference_input_type = tf.int8\n",
        "converter.inference_output_type = tf.int8\n",
        "tflite_model_quant = converter.convert()\n",
        "\n",
        "with open(\"tflite_model.tflite\", \"wb\") as f:\n",
        "  f.write(tflite_model_quant)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "StbdFxpk6D_8"
      },
      "source": [
        "### Test TF Lite model\n",
        "\n",
        "Since TensorFlow also supports loading TF Lite models using [`tensorflow.lite`](https://www.tensorflow.org/api_docs/python/tf/lite), we can also verify the functionality of the quantized model and compare its accuracy with the regular unquantized model inside the notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qZkEUTBr6Hd-"
      },
      "source": [
        "import tensorflow.lite as tflite\n",
        "\n",
        "# Load the interpreter and allocate tensors\n",
        "interpreter = tflite.Interpreter(\"tflite_model.tflite\")\n",
        "interpreter.allocate_tensors()\n",
        "\n",
        "# Load input and output details\n",
        "input_details = interpreter.get_input_details()[0]\n",
        "output_details = interpreter.get_output_details()[0]\n",
        "\n",
        "# Set quantization values\n",
        "input_scale, input_zero_point = input_details[\"quantization\"]\n",
        "output_scale, output_zero_point = output_details[\"quantization\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yLw9Yy7w6bwy"
      },
      "source": [
        "# Calculate the number of correct predictions\n",
        "correct = 0\n",
        "test_ds_len = 0\n",
        "\n",
        "# Loop through entire test set\n",
        "for x, y in test_ds.unbatch():\n",
        "  # original shape is [124, 129, 1] expand to [1, 124, 129, 1]\n",
        "  x = tf.expand_dims(x, 0).numpy()\n",
        "  \n",
        "  # quantize the input value\n",
        "  if (input_scale, input_zero_point) != (0, 0):\n",
        "    x = x / input_scale + input_zero_point\n",
        "  x = x.astype(input_details['dtype'])\n",
        "\n",
        "  # add the input tensor to interpreter\n",
        "  interpreter.set_tensor(input_details[\"index\"], x)\n",
        "  \n",
        "  #run the model\n",
        "  interpreter.invoke()\n",
        "\n",
        "  # Get output data from model and convert to fp32\n",
        "  output_data = interpreter.get_tensor(output_details[\"index\"])\n",
        "  output_data = output_data.astype(np.float32)\n",
        "\n",
        "  # Dequantize the output\n",
        "  if (output_scale, output_zero_point) != (0.0, 0):\n",
        "    output_data = (output_data - output_zero_point) * output_scale\n",
        "\n",
        "  # convert output to category\n",
        "  if output_data[0][0] >= 0.5:\n",
        "    category = 1\n",
        "  else:\n",
        "    category = 0\n",
        "  \n",
        "  # add 1 if category = y\n",
        "  correct += 1 if category == y.numpy() else 0\n",
        "\n",
        "  test_ds_len += 1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R38f75_p61se"
      },
      "source": [
        "accuracy = correct / test_ds_len\n",
        "print(f\"Accuracy for quantized model is {accuracy*100:.2f}% (to 2 D.P) on test set.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "exRvsfKz7JPR"
      },
      "source": [
        "## Deploy on Device"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WcWf2CCK_dst"
      },
      "source": [
        "#### Convert `.tflite` to `.h` file\n",
        "\n",
        "The RP2040 MCU on the boards we are deploying to, does not have a built-in file system, which means we cannot use the .tflite file directly on the board. However, we can use the Linux `xxd` command to convert the .tflite file to a .h file which can then be compiled in the inference application in the next step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PWBsWuIN7MHx"
      },
      "source": [
        "%%shell\n",
        "echo \"alignas(8) const unsigned char tflite_model[] = {\" > tflite_model.h\n",
        "cat tflite_model.tflite | xxd -i                        >> tflite_model.h\n",
        "echo \"};\"                                               >> tflite_model.h"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x14EFzyN_mk_"
      },
      "source": [
        "#### Inference Application\n",
        "\n",
        "We now have a model that is ready to be deployed to the device! We’ve created an application template for inference which can be compiled with the .h file that we’ve generated for the model.\n",
        "\n",
        "The C++ application uses the `pico-sdk` as the base, along with the `CMSIS-DSP`, `pico-tflmicro`, and `Microphone Libary for Pico` libraries. It’s general structure is as follows:\n",
        " 1. Initialization\n",
        "  1. Configure the board's built-in LED for output. The application will map the brightness of the LED to the output of the model. (0.0 LED off, 1.0 LED on with full brightness)\n",
        "  1. Setup the TF Lite library and TF Lite model for inference\n",
        "  1. Setup the CMSIS-DSP based DSP pipeline\n",
        "  1. Setup and start the microphone for real-time audio\n",
        " 1. Inference loop\n",
        "  1. Wait for 128 * 4 = 512 new audio samples from the microphone\n",
        "  1. Shift the spectrogram array over by 4 columns\n",
        "  1. Shift the audio input buffer over by 128 * 4 = 512 samples and copy in the new samples\n",
        "  1. Calculate 4 new spectrogram columns for the updated input buffer\n",
        "  1. Perform inference on the spectrogram data\n",
        "  1. Map the inference output value to the on-board LED’s brightness and output the status to the USB port\n",
        "\n",
        "In-order to run in real-time each cycle of the inference loop must take under (512 / 16000) = 0.032 seconds or 32 milliseconds. The model we’ve trained and converted takes 24 ms for inference, which gives us ~8 ms for the other operations in the loop.\n",
        "\n",
        "128 was used above to match the stride of 128 used in the training pipeline for the spectrogram. We used a shift of 4 in the spectrogram to fit within the real-time constraints we had.\n",
        "\n",
        "The source code for the inference application can be found on GitHub: https://github.com/ArmDeveloperEcosystem/ml-audio-classifier-example-for-pico/tree/main/inference-app\n",
        "\n",
        "**Note:** We have already cloned this project in the setup steps from earlier.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mbJTbrhI2Lm7"
      },
      "source": [
        "Now we can copy the updated `tflite_model.h` file over:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ieVMgugZ-k5L"
      },
      "source": [
        "!cp tflite_model.h inference-app/src/tflite_model.h"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0NehBUZd_ydi"
      },
      "source": [
        "#### Compile Inference Application\n",
        "\n",
        "Once again we can use `cmake` to setup project before compiling it:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bPswlG9u_AfN"
      },
      "source": [
        "%%shell\n",
        "cd inference-app\n",
        "mkdir -p build\n",
        "cd build\n",
        "cmake .. -DPICO_BOARD=${PICO_BOARD}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9m89TzMy2r01"
      },
      "source": [
        "Then use `make` to compile it:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sm9D9Bsv2mre"
      },
      "source": [
        "%%shell\n",
        "cd inference-app/build\n",
        "\n",
        "make -j"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8PIvWvzJABRb"
      },
      "source": [
        "#### Flash inferencing application to board\n",
        "\n",
        "You’ll need to put the board into USB boot ROM mode again to load the new application to it. If you are using a WebUSB API enabled browser like Google Chrome, you can directly flash the image onto the board from within Google Collab! Otherwise, you can manually download the .uf2 file to your computer and then drag it onto the USB disk for the RP2040 board.\n",
        "\n",
        "**Note for Windows**: If you are using Windows you must install WinUSB drivers in order to use WebUSB, you can do so by following the instructions found [here](https://github.com/ArmDeveloperEcosystem/ml-audio-classifier-example-for-pico/blob/main/windows.md).\n",
        "\n",
        "**Note for Linux**: If you are using Linux you must configure udev in order to use WebUSB, you can do so by following the instructions found [here](https://github.com/ArmDeveloperEcosystem/ml-audio-classifier-example-for-pico/blob/main/linux.md).\n",
        "\n",
        " * SparkFun MicroMod\n",
        "  * Plug the USB-C cable into the board and your PC to power the board\n",
        "  * While holding down the BOOT button on the board, tap the RESET button\n",
        "\n",
        " * Raspberry Pi Pico\n",
        "  * Plug the USB Micro cable into your PC, but do NOT plug in the Pico side.\n",
        "  * While holding down the white BOOTSEL button, plug in the micro USB cable to the Pico\n",
        "\n",
        "\n",
        "Then run the code cell below and click the \"flash\" button."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CsWtAuAiAFYc"
      },
      "source": [
        "from colab_utils.pico import flash_pico\n",
        "\n",
        "flash_pico('inference-app/build/pico_inference_app.bin')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0qFp9s1I0P7G"
      },
      "source": [
        "### Monitoring the Inference on the board\n",
        "\n",
        "\n",
        "Now that the inference application is running on the board you can observe it in action in two ways:\n",
        "\n",
        " 1. Visually by observing the brightness of the LED on the board. It should remain off or dim when no fire alarm sound is present - and be on when a fire alarm sound is present.\n",
        "\n",
        " 1. Connecting to the board’s USB serial port to view output from the inference application. If you are using a [Web Serial API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Serial_API) enabled browser like Google Chrome, this can be done directly from Google Colab!\n",
        "\n",
        "#### Test Audio\n",
        "\n",
        "Use the code cell below to playback the fire alarms sounds used during training from your computer. You may need to adjust the speaker volume from your computer.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4gqmp3nPtp6P"
      },
      "source": [
        "for wav, _, _ in fire_alarm_wav_ds:\n",
        "  display.display(display.Audio(wav, rate=sample_rate))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w2ufVilt4BKU"
      },
      "source": [
        "#### Serial Monitor\n",
        "\n",
        "Run the code cell below and then click the \"Connect Port\" button to view the serial output from the board:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RAvIj39m3fUa"
      },
      "source": [
        "from colab_utils.serial_monitor import run_serial_monitor\n",
        "\n",
        "run_serial_monitor()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YoQzVwGA41Eq"
      },
      "source": [
        "## Improving the model\n",
        "\n",
        "You now have the first version of the model deployed to the board, and it is performing inference on live 16,000 kHz audio data!\n",
        "\n",
        "Test out various sounds to see if the model has the expected output. Maybe the fire alarm sound is being falsely detected (false positive) or not detected when it should be (false negative). \n",
        "\n",
        "If this occurs, you can record more new audio data for the scenario(s) by flashing the USB microphone application firmware to the board, recording the data for training, re-training the model and converting to TF lite format, and re-compiling + flashing the inference application to the board.\n",
        "\n",
        "Supervised machine learning models can generally only be as good as the training data they are trained with, so additional training data for these scenarios might help. You can also try to experiment with changing the model architecture or feature extraction process - but keep in mind that your model must be small enough and fast enough to run on the RP2040 MCU!\n",
        "\n",
        "## Conclusion\n",
        "\n",
        "This guide covered an end-to-end flow of how to train a custom audio classifier model to run locally on a development board that uses an Arm Cortex-M0+ processor. Google Colab was used to train the model using Transfer Learning techniques along with a smaller dataset and data augmentation techniques. We also collected our own data from the microphone that is used at inference time by loading an USB microphone application to the board, and extending Colab’s features with the Web Audio API and custom JavaScript\n",
        "\n",
        "The training side of the project combined Google’s Colab service and Chrome browser, with the open source TensorFlow library. The inference application captured audio data from a digital microphone, used Arm’s CMSIS-DSP library for the feature extraction stage, then used TensorFlow Lite for Microcontrollers with Arm CMSIS-NN accelerated kernels to perform inference with a 8-bit quantized model that classified a real-time 16 kHz audio input on an Arm Cortex-M0+ processor.\n",
        "\n",
        "The Web Audio API, Web USB API, and Web Serial API features of Google Chrome were used to extend Google Colab’s functionality to interact with the development board. This allowed us to experiment with and develop our application entirely with a web browser and deploy it to a constrained development board for on-device inference.\n",
        "\n",
        "Since the ML processing was performed on the development boards RP2040 MCU, privacy was preserved as no raw audio data left the device at inference time.\n",
        "\n"
      ]
    }
  ]
}