04-ModelosAsociativos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modelos Asociativos y Aprendizaje Hebbiano\n",
    "===\n",
    "\n",
    "**Juan David Velásquez Henao**  \n",
    "jdvelasq@unal.edu.co   \n",
    "Universidad Nacional de Colombia, Sede Medellín  \n",
    "Facultad de Minas  \n",
    "Medellín, Colombia\n",
    "\n",
    "---\n",
    "\n",
    "Haga click [aquí](https://github.com/jdvelasq/deep-neural-nets/tree/master/) para acceder al repositorio online.\n",
    "\n",
    "Haga click [aquí](http://nbviewer.jupyter.org/github/jdvelasq/deep-neural-nets/tree/master/) para explorar el repositorio usando `nbviewer`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Definición del problema"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Se contruyó un sistema de transmisión que permite enviar información entre dos puntos diferentes. Por simplicidad se supondrá que solo es de interés transmitir tres patrones diferentes, los cuales son presentados en la figura de abajo. El problema radica en que en el punto de recepción, el patrón recibido puede llegar incompleto o contaminado con ruido. Se desea construir un sistema que permita recuperar el patrón original a partir del patrón recibido.  \n",
    "\n",
    "![alt](images/ModelosAsociativos-01.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Metodología de Solución"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Este problema puede ser solucionado mediante una memoria lineal asociativa, la cual relaciona un patrón de actividad neuronal de entrada  con un patrón de actividad neuronal de salida. Existen dos fases:\n",
    "\n",
    "* Aprendizaje: Un patrón clave es presentado como un estímulo. La memoria lo transforma a un patrón almacenado o memorizado.\n",
    "\n",
    "\n",
    "![alt](images/ModelosAsociativos-02.png)\n",
    "\n",
    "* Uso: Se presenta una versión del patrón de entrada incompleta o con ruido y la memoria devuelve el patrón almacenado más parecido.\n",
    "\n",
    "![alt](images/ModelosAsociativos-03.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Las características de la memoria asociativa son las siguientes:\n",
    "\n",
    "\n",
    "* La memoria es distribuida.\n",
    "\n",
    "\n",
    "* Los patrones de estímulo y las respuestas son vectores de datos.\n",
    "\n",
    "\n",
    "* La información se almacena en las neuronas a través de un patrón de actividad neural a través de un gran número de neuronas.\n",
    "\n",
    "\n",
    "* La información contenida en el estímulo determina su localización de almacenamiento en la memoria y la dirección para su recuperación.\n",
    "\n",
    "\n",
    "* La memoria tiene un alto grado de resistencia al ruido.\n",
    "\n",
    "\n",
    "* Las interacciones entre patrones son almacenadas en la memoria.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conceptualmente se reconocen dos tipos de asociación:\n",
    "\n",
    "* Autoasociación: El estímulo es asociado consigo mismo en la memoria.\n",
    "\n",
    "![alt](images/ModelosAsociativos-04.png)\n",
    "\n",
    "* Heteroasociación: Vectores arbitrarios de estímulos son asociados con vectores arbitrarios de respuesta.\n",
    "\n",
    "![alt](images/ModelosAsociativos-02.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hay dos tipos de memorias:\n",
    "\n",
    "* Memoria asociativa lineal: la neurona actúa como un combinador lineal: \n",
    "$$\\mathbf{b}=\\mathbf{Ma}$$\n",
    "Donde $\\mathbf{M}$ es la matriz de memoria, $\\mathbf{a}$ el estímulo, y $\\mathbf{b}$ la respuesta\n",
    "\n",
    "\n",
    "* Memoria asociativa no lineal: \n",
    "$$\\mathbf{b}=\\Phi(\\mathbf{M},\\mathbf{a})$$\n",
    "en la que $\\Phi(.)$ es una función no lineal.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solución mediante una memoria asociativa lineal\n",
    "\n",
    "Una memoria asociativa lineal puede representarse como la siguiente red neuronal:\n",
    "\n",
    "![alt](images/ModelosAsociativos-05.png)\n",
    "\n",
    "donde:\n",
    "\n",
    "* El $k$-ésimo estímulo o patrón clave es $\\mathbf{a}_k = [a_{k1}, ..., a_{km}]^T$.\n",
    "\n",
    "\n",
    "* El $k$-ésimo vector respuesta o patrón memorizado es $\\mathbf{b}_k = [b_{k1}, ..., b_{kn}]$.\n",
    "\n",
    "\n",
    "* La red asocia $\\mathbf{a}_k$ con $\\mathbf{b}_k$, esto es $\\mathbf{a}_k \\, \\to \\, \\mathbf{b}_k$ como:\n",
    "\n",
    "$$\\mathbf{b}_k = \\mathbf{M} \\mathbf{a}_k$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Para $Q$ patrones a asociar, la matriz $\\mathbf{M}$ se calcula como:\n",
    "\n",
    "$$\\mathbf{M} = \\sum_{k=1}^Q \\mathbf{b}_k \\mathbf{a}_k^T \n",
    "= [\\mathbf{b}_1 ... \\mathbf{b}_Q] \n",
    " \\begin{bmatrix} \\mathbf{a}_1^T  \\\\ \\vdots \\\\ \\mathbf{a}_Q^T\\end{bmatrix} \n",
    "= \\mathbf{B} \\mathbf{A}^T\n",
    "$$\n",
    "\n",
    "donde $\\mathbf{A}$ es la matriz clave y $\\mathbf{B}$ es la matriz memorizada."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Propagación de la señal.** Cuando se presenta un estímulo $\\mathbf{a}_j$ la respuesta será:\n",
    "\n",
    "$$\\mathbf{b} = \n",
    "\\mathbf{M} \\mathbf{a_j} = \n",
    "\\sum_{k=1}^Q \\mathbf{b}_k \\mathbf{a}_k^T \\mathbf{a}_j =\n",
    "\\sum_{k=1}^Q (\\mathbf{a}_k^T \\mathbf{a}_j) \\mathbf{b}_k  =\n",
    "(\\mathbf{a}_j^T \\mathbf{a}_j) \\mathbf{b}_j + \n",
    "\\sum_{\\substack{k=1\\\\k \\neq j}}^Q (\\mathbf{a}_k^T \\mathbf{a}_j) \\mathbf{b}_k\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ya que la red neuronal reproduce perfectamente la salida deseada $\\mathbf{b}_j$, entonces:\n",
    "\n",
    "* Cada patrón de entrada debe estar normalizado $\\mathbf{a}_j^T \\mathbf{a}_j = 1$.\n",
    "\n",
    "\n",
    "* Los patrones de entrada deben ser ortogonales:\n",
    "$$\n",
    "\\mathbf{a}_k^T \\mathbf{a}_j = \n",
    "\\begin{cases}\n",
    "1, & k = j \\\\\n",
    "0, & k \\ne j\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "Y $Q$ debe ser menor que la dimensión de las claves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Ejercicio.--** Construya la matriz $\\mathbf{M}$ para los siguientes patrones de entrenamiento (realice los cálculos usando `numpy`):\n",
    "\n",
    "* $\\mathbf{a}_1 = [+0.5, -0.5, +0.5, -0.5]^T$, $\\mathbf{b}_1 = [+1, -1]^T$.\n",
    "\n",
    "\n",
    "* $\\mathbf{a}_2 = [+0.5, +0.5, -0.5, -0.5]^T$, $\\mathbf{b}_2 = [+1, +1]^T$.\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  0.  0. -1.]\n",
      " [ 0.  1. -1.  0.]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np \n",
    "a = np.array([[0.5, -0.5, 0.5, -0.5], [0.5, 0.5, -0.5, -0.5]])\n",
    "b = np.array([[1, 1], [-1, 1]])\n",
    "print(np.dot(b,a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Método de la pseudoinversa.** En este caso, los vectores de es:\n",
    "\n",
    "$$\\mathbf{b} = \n",
    "\\mathbf{M} \\mathbf{a_j} = \n",
    "(\\mathbf{a}_j^T \\mathbf{a}_j) \\mathbf{b}_j + \n",
    "\\sum_{\\substack{k=1\\\\k \\neq j}}^Q (\\mathbf{a}_k^T \\mathbf{a}_j) \\mathbf{b}_k = \n",
    "\\mathbf{b}_j + \n",
    "\\sum_{\\substack{k=1\\\\k \\neq j}}^Q (\\mathbf{a}_k^T \\mathbf{a}_j) \\mathbf{b}_k =\n",
    "\\mathbf{b}_j+ \\mathbf{v}_j\n",
    "$$\n",
    "\n",
    "donde $\\mathbf{b}_j$ es la respuesta deseada y $\\mathbf{v}_j$ es un vector de ruido, con:\n",
    "\n",
    "$$\\mathbf{v}_j \n",
    "= \\sum_{\\substack{k=1\\\\k \\neq j}}^Q (\\mathbf{a}_k^T \\mathbf{a}_j) \\mathbf{b}_k \n",
    "= \\sum_{\\substack{k=1\\\\k \\neq j}}^Q \\cos (\\mathbf{a}_k, \\mathbf{a_j}) \\mathbf{b}_k \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Se puede minimizar el error cuando los patrones no son ortogonales aplicando la regla de la pseudoinversa.\n",
    "\n",
    "* Se tienen $Q$ patrones de entrenamiento, con $Q>m$.\n",
    "\n",
    "\n",
    "* Es posible definir una medida del error, tal como la suma de errores al cuadrado:\n",
    "$$\n",
    "J(\\mathbf{M}) \n",
    "= \\sum_{k=1}^Q \\| \\mathbf{b}_k - \\mathbf{Ma}_k) \\|^2\n",
    "= \\| \\mathbf{B} - \\mathbf{MA} \\|^2\n",
    "= \\| \\mathbf{E} \\|^2\n",
    "$$\n",
    "\n",
    "\n",
    "* Derivando e igualando a cero:\n",
    "\n",
    "$$ \\mathbf{B} - \\mathbf{MA} = 0 $$\n",
    "\n",
    "$$ \\mathbf{MA} = \\mathbf{B} $$\n",
    "\n",
    "$$ \\mathbf{M} = \\mathbf{B} \\mathbf{A} ^{-1}$$\n",
    "\n",
    "\n",
    "* Ya que $\\mathbf{A}$ es una matriz rectangular, su inversa exacta no existe. Se recurre a la definición de psuedoinversa. \n",
    "\n",
    "$$ \\mathbf{M} = \\mathbf{B} \\mathbf{A} ^{+}$$\n",
    "\n",
    "$$ \\mathbf{A}^{+} = (\\mathbf{A}^T \\mathbf{A})^{-1} \\mathbf{A}^T $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solución al problema propuesto.** Para el problema propuesto en este tutorial, se tienen tres  patrones de entrenamento, los cuales serán aprendidos por una red neuronal asociativa NO lineal. Este tipo de red neuronal se construye usando una memoria asociativa lineal en la que se cambia la función de activación. La arquitectura del modelo es la siguiente:\n",
    "\n",
    "![alt](images/ModelosAsociativos-06.png)\n",
    "\n",
    "Las figuras son representadas mediante patrones binarios, $\\{-1, +1\\}$, donde $-1$ representa el blanco y $+1$ representa el negro."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solucion usando `numpy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_digits(x, y=None):\n",
    "    \"\"\"\n",
    "    Genera el gráfico de patrones por parejas.\n",
    "    \n",
    "    x: lista con los patrones iniciales\n",
    "    y: lista con los patrones limpios\n",
    "    \"\"\"\n",
    "    N = len(x)\n",
    "    if y is None:\n",
    "        fig, axes = plt.subplots(1, N, figsize=(2.9, 1.2))\n",
    "        for i, data in enumerate(x):\n",
    "            axes[i].pcolor(data, edgecolors='grey', linewidths=1, cmap=plt.get_cmap('Greys'))\n",
    "            axes[i].set(xticks=[], yticks=[])\n",
    "    else:\n",
    "        if N == 1:\n",
    "            fig, axes = plt.subplots(N, 2, figsize=(2.0, 1.2))\n",
    "            axes[0].pcolor(x[0], edgecolors='grey', linewidths=1, cmap=plt.get_cmap('Greys'))\n",
    "            axes[1].pcolor(y[0], edgecolors='grey', linewidths=1, cmap=plt.get_cmap('Greys'))\n",
    "            axes[0].set(xticks=[], yticks=[])\n",
    "            axes[1].set(xticks=[], yticks=[])\n",
    "        else:\n",
    "            fig, axes = plt.subplots(N, 2, figsize=(2, 3))\n",
    "            for i, (data_x, data_y) in enumerate(zip(x,y)):\n",
    "                axes[i][0].pcolor(data_x, edgecolors='grey', linewidths=1, cmap=plt.get_cmap('Greys'))\n",
    "                axes[i][1].pcolor(data_y, edgecolors='grey', linewidths=1, cmap=plt.get_cmap('Greys'))\n",
    "                axes[i][0].set(xticks=[], yticks=[])\n",
    "                axes[i][1].set(xticks=[], yticks=[])\n",
    "    fig.tight_layout()\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMkAAABPCAYAAABBL+HAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAACGhJREFUeJztnUtMU2sXht+Wlt0bUChQbkIUmygm6ODHCRoxYWCc6cREnRyiIxIHEjUGjSNMHOjUxMShzpw40MQw8ZoY+QfekEZEUBQKDYXSdve+/wExOZP9rb3zN4jnvE/S0Vqs79tvvzfarr12HYZhgBBijvN3b4CQrQ5NQogATUKIAE1CiABNQogATUKIAE1CiABNQogATUKIAE1CiIDLTnJDQ4OxY8cO03g2m0U8HkcsFlPWCYVCyOVySKVSyryWlhYsLy+jVCqZ5vh8PjQ2NqKpqUlZa21tDdPT03HDMNSJFthsHdra2tDc3IyqqirTnHQ6jUwms6k6OJ1OQ3Vbk6ZpCAaDf/55MAzD8qu3t9dQMTU1ZYyMjBgAlK+hoSHj4MGDYt7ly5eNuro6ZU5fX59x+/Zt5b4MwzAePnxoAJiwc71bRYexsTEjkUgo13z9+vWm6+B2u5X7jkQixvnz5//488D/bhEiQJMQIkCTECJAkxAiQJMQIkCTECJgq0+SzWbx9u1b0/ji4iLcbjf27t2rrOPz+dDa2irmAUBPTw8ymYxpvL29HZlMRrkvAIjH4+JaVvkdOnz69Ak+n880Pj8/D13XN1UHTdPQ09NjGg+Hw/+I8+AwbMy4t7W1GX/99Zdp3OVyoVwuIxAIKOuk02k4HA7lmw4AyWQSfr9f2UTLZrNwOBzI5XLKWpqm4dq1a/81DOM/ykQLdHZ2GpcuXTKN5/N5rKysiDqkUilks1nk83llXigUQjAYFHXI5/Oora1V1tJ1HRcuXKiIDu3t7ca5c+dM4/l8HtlsVtyT1fOwuroKXdehOrNutxv19fUIBoPKWrZ0qHTz6ObNm2Ij5+7du8bTp0/FvOvXr1tqog0PD4uNqBMnTmxqM/HWrVvi9d25c+ePbiZWSger52F0dJTNREK2IjQJIQI0CSECNAkhAjQJIQI0CSECNAkhAraaic3NzcaxY8dM45qmoampCa2trco6y8vL0DRNbDLFYjGEQiG4XOY3BqRSKcTjcSwvLytr1dfX48aNGxVponV1dRlXr141jWezWSQSCVGHWCyGlZUVcSKvs7PTkg7pdBrhcFhZK5FI4OLFi1tKB6vnYXZ2FolEAsVi0TTH7/cjFApVVAdbt6Xkcjl8/PjRNB4KhdDS0oLdu3cr6ySTSYRCIezcuVOZt7i4iO7ubmUndm5uDgsLC8p9AUBvb68ybofq6mrlNcZiMayvr4s6rK6uIh6P48uXL8q8jo4OSzr8+PFDXHNyclIZt0OldLB6HmZnZxGNRpHNZk1zOjo6LJ1BOzrYMomu63j58qVpPBKJ4NChQ+jv71fWiUaj2L59u5j37Nkz9PX1KW8xcLvdePXqlXJfwIZ4lcLn8yn3Ho1GMTc3J17f5OQkvn79Ku796NGjlnRIJpPimisrK8q4HSqlg9Xz8PjxY0xMTGBtbc00p6+vDwMDAxXVgZ9JCBGgSQgRoEkIEaBJCBGgSQgRoEkIEbD1FbDT6URNTY1p3Ov1olwuY319XVmnVCohl8uJecDG1Jo0kSftC4CyEWcX6Rp1Xbesg6Zp4t4BazpYWVOagrRDJXWwch4Mw0AgEEC5XDbN8Xg8KJVKFdXBVse9tbXVGBoaUuZ4vV44nep/oMrlspgDAA6HQzmqCWwIJ410/qo1NjZWsbHV4eFh5Z5+ramiWCyKY8fARj9CqmV1TcMwcOXKlU3VQcLqedB13VI9j8cj1rOlg5XxxV8vju9yfNewqcM/4ZnI/ExCiABNQogATUKIAE1CiABNQogATUKIAE1CiICtNrTX68X+/ftN442Njcjlcnjx4oWyTjKZxLdv38Q8AJiYmIDH4zGNf//+HV6vVxyyaWhoENeySiaTUe49Ho9b1qGrq0vZQf6FFR3S6bS45sLCgriWVazoUF1dLb43NTU1W1oHWybRNE05FqlpGgqFAqamppR1MpkMYrGYcgwT2OhIT09PK28pWV9fF/cFQHx4tR3y+bzyGnVdt6SDrusIh8PKNx0ACoWCJR2y2ay4ppVbgaxiRQeXyyW+N16vF3v27BHNFIvFfo8Odjqs7Liz427Y1MFKx/3MmTMVPQ/suBOyydAkhAjQJIQI0CSECNAkhAjQJIQI0CSECNga321paTFOnTplGq+qqkIgEBBnttPpNFwuFzRNU+ZlMhlxFDOXyyGTySCdTitreb3eio3vbtu2zRgZGTGNFwoF6LouPgA6mUxC13VxhDcUCsHj8Shn3HO5HPL5vKh9JpPB6OjopumQSqXExl0gEIDP56vYeSgUCmLzWNd1yzrY6rgHg0GcPXvWND4/P4/x8XHcv39fWWdwcBCLi4v48OGDMu/48eN48OCB8ne7I5EIDh8+jJMnTyprvXnzRhm3g9/vx5EjR0zj8/PzmJiYUOYAwJMnT/Du3TtRh9OnT2NgYAB+v980JxqNYmZmRlxzs3UYHx/Ho0ePlHUGBwcxOTlZsfMQiURw7949Za0DBw4o43/H9m0pu3btMo07HA4Ui0VEo1Flnf7+fiwtLYl5ADAzM6N8QHJtbS18Pp9yXwDw+fNncS2rWNHh/fv34p6eP39uSQen04nu7m7lA7OTySQWFha2nA6/4zx0dnaKtfbt2yeu9Qt+JiFEgCYhRIAmIUSAJiFEgCYhRIAmIUTA1lfAxWIRP3/+NI0nEglUVVWJv7ZaXV2N2tpaMQ8AwuGw8gc16+rqUCgUlPsCoPxu3S5WdCiVSuKeCoWCZR2WlpaU17C2trYldfgd58Hlcom1pGnQv2Or4+5wOJYBzFn+g61Hl2EYTf9vEeqwwb9FB1smIeTfCD+TECJAkxAiQJMQIkCTECJAkxAiQJMQIkCTECJAkxAiQJMQIvA/X9wjRLoL0HIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 208.8x86.4 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "##\n",
    "## Se representan los tres patrones como matrices\n",
    "## Note que las matrices estan invertidas para\n",
    "## efectos de graficacion\n",
    "## -1: blanco,  +1: negro\n",
    "##\n",
    "\n",
    "d0 = np.array([[-1, +1, +1, +1, -1],\n",
    "               [+1, -1, -1, -1, +1],\n",
    "               [+1, -1, -1, -1, +1],\n",
    "               [+1, -1, -1, -1, +1],\n",
    "               [+1, -1, -1, -1, +1],\n",
    "               [-1, +1, +1, +1, -1]])\n",
    "\n",
    "d1 = np.array([[-1, -1, +1, -1, -1],\n",
    "               [-1, -1, +1, -1, -1],\n",
    "               [-1, -1, +1, -1, -1],\n",
    "               [-1, -1, +1, -1, -1],\n",
    "               [-1, -1, +1, -1, -1],\n",
    "               [-1, +1, +1, -1, -1]])\n",
    "\n",
    "d2 = np.array([[-1, +1, +1, +1, +1],\n",
    "               [-1, +1, -1, -1, -1],\n",
    "               [-1, +1, +1, -1, -1],\n",
    "               [-1, -1, -1, +1, -1],\n",
    "               [-1, -1, -1, +1, -1],\n",
    "               [+1, +1, +1, +1, -1]])\n",
    "\n",
    "plot_digits(x=[d0, d1, d2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIgAAADQCAYAAADcZynIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFG9JREFUeJztnWuMlOXZx//PM+eZ3ZndWdhlBnYBdYHFyIbSxla6SUNqP1hJQ2JcRdRoPFWrCT2ERtmW0lq/iI39aNwawwcbTE20omg8EVYiCIgics4eYI/MsjszO8fncL8ftsML7/vOdT1PmPjScv2S+XT9ue/7uee/OveV674eTSkFQaiG/v+9AOHqRgwikIhBBBIxiEAiBhFIxCACiRhEIBGDCCRiEIHE60Yci8XU/Pnzq8Zt20Ymk0E6nSbHCQQC8Pl8mJmZIXV1dXUwDAOlUonUNTY2IhwOw+ulH2d6ehqjo6MppdRcUsgQiUTUwoULSU0ul0Mul0OhUCB19fX1KBaLMAyD1EWjUWSzWVCZb6/Xi/r6ekSjUXKscrmMM2fOONsHpZTjz4oVKxRFLpdTPT09CgD56erqUg8//DCre/DBB1VXVxere/bZZ9XAwAC5NqWU2rJliwJwwM0z/1+ftrY2dq7XXntN3X777ezaN27cqDo6OljdM888o8LhMKlpb29X27ZtY9d24MABx/sg/4sRSMQgAokYRCARgwgkYhCBxNUx1zRNTExMVI0Xi0UAwNy59OkpEonA6/WyOp/Ph7q6OlYHzB5hQ6EQq6sFSilyHwCgVCohGAyya9c0DbFYzNEzzpkzhzw2NzQ0wLZtdm3ZbJad6+L6lIuKsgULFqgnnniialwphXw+z45z8Qil0/8Bc6oLBAJsDgSYzdNs3rz5oFLqu6yYYP78+eoXv/gFO1cul2PXblkWNE1jdbZtQ9M0aJpWVaOUQjgcZscCgKefftrZPrg5//t8PvIcHgqF1NatW9lz+O7du1Vvby+r6+3tVbt372Z1PT09qq2tjc0l1CoP0tLSws7V3d2tdu7cya79xRdfVN988w2r+9Of/qRyuRypOX78uPrVr37Frm3VqlWSBxFqgxhEIBGDCCRiEIFEDCKQiEEEEld5kObmZrVu3brqg2kaEokEqJoRAMhkMigWi2hubiZ158+fRyAQYOsb+vv7kclkUC6XSV0ikcAf/vCHK86DJJNJtXbtWlITj8cRj8fR2NhI6lKpFCKRCJvkGxsbQ3NzM5njKBaLmJiYwPj4ODlWJBLBX/7yF0f74CqTWiqVcPTo0apxn8+HRCKBjo4OcpwzZ84glUqxukwmg6amJtxwww2kbnBwEP39/ZieniZ1yWSSjDtFKUXuAwB0dnZi0aJF7DP29fUhmUxi3rx5pG50dBTt7e3w+/1VNePj4xgZGWHXxv0BX4orgxQKBXz66adV46FQCGvWrMHq1avJcSzLgmmarO7EiRNYvHgxq3vvvfdw5MgRDA0Nkbpbb72VjDtFKUXuAwAsWLAAra2t7NoPHjyI5cuXs0b65JNPcPPNNyMcDlfVnDhxAnv37mXXtmrVKjJ+KfIbRCARgwgkYhCBRAwikIhBBBJXpxi/34/FixdXjQcCASilcOLECXKcVCqFQqHA6gqFAlKpFKuzbRttbW0IBoOkrlZomoYlS5aQmvr6ekxPT7NrN00Tw8PDjmo4Tp8+jUAgUDU+MjICr9fLrq2lpYWdq4KrRNm8efPUPffcQ2oaGhrYxFapVIJt22xyKJ/Pw+v1kmd/AJicnEShUIBt26QuGAziz3/+8xUnyhKJhFq/fj2p8fv9aGhoYE1bKBTg8/nYgqd8Po9QKEQWDJmmiQsXLlys7KuGrut4/vnnpWDof36kYEgKhoQaIwYRSMQgAokYRCARgwgkYhCBxFUeJJFIqHvvvZfURCIRRCIRUlMul2FZFpsHKRaL0HWdzYNMTU3Btm1YlkXqwuFwzQqGNmzYQGoCgQCCwSCZ2AJm90LXdTYPUiwWEQgE2DxIsVhkL695vV4899xztS8YymQy2LFjR9V4IBDA+vXrcccdd5DjHD58GMPDw/jpT39K6nbu3IlkMomVK1eSut7eXnzwwQdsJdUDDzxAxp2ilCL3AQDWrFmDW2+9FT/4wQ9I3T/+8Q+sWrUKixYtInXbt2/H2rVrycTb2bNn8fbbb7Nr4zKtl+LKIIZhYHBwsGo8FApB13X2YYeGhnDhwgVWFwqF0NTUxOo0TcPY2BhbMFQrlFLkPgCzmc9YLMau3efzoaWlhdXpuo62tjayYKhUKsGyLHZtc+bMIeOXzetYKVyTiEEEEjGIQCIGEUjEIAKJq1NMKBTCd77znapxn88HpRQ+++wzcpyzZ88im82yumw2i3PnzrE6y7Jw4403IpFIkLpaoWkabr75ZlITj8cxNjbGrt0wDJw8eZJtPgwABw4cIHNCExMT8Pl87NpaW1tx8OBBdj7AZaKstbVV/eY3v6kat20b6XQaU1NT5DjhcBiBQIDVxWIxmKaJXC5H6uLxOEKhEJuUSqVS+OMf/1iTDkObNm0iNdPT0zAMg2331NjYiHw+z3aTjsfjmJ6eJoui/H7/xdQARbFYxKZNm2pfMCSdlqXTsiBchhhEIBGDCCRiEIFEDCKQiEEEEld5kGQyqR599FFSY9u2o1tiSimy+MXNWF6vF6ZpsrpgMIjf/va3NcmDPPLII1cyxEUMw4DP52N1pmk66iataRqcfKdbtmxxtA+uDKJp2nkAdLHB1c9CdYWvJLuW9sGVQYRrD/kNIpCIQQQSMYhAIgYRSMQgAokYRCARgwgkYhCBRAwikIhBBBIxiEAiBhFIxCACiRhEIBGDCCSurl7W1dUp7taWYRhsS+zKO+g5na7r8Hg8bFWZZVnQdZ2tUDNNEyMjI6krLRiKRCKKa8JiWRYsy2Kruypr5nQej4etKFNKwbIsVmfbNs6dO+dsH9zcKPu2b9Y99NBDjlpxb9u27T/+Zp3TVtwvvPACuza5WSfUDDGIQCIGEUjEIAKJGEQgEYMIJK7uxbS1tamnn366aty2bUxMTCCVSpHjhMNh+P1+tsNQPB5HLBZDXV0dqZuamkIkEmFbdo+Pjzu+UUYxf/581dPTw64pnU5jZmaG1NXX16NUKrEdhhKJBOLxOJkTKpVKyGQymDuXTm8UCgX88pe/rH0r7mAwSL7Wu1wu44MPPsD9999PjnPy5EkcOnQIn3/+OalbvXo1li5diqVLl5K6Dz/8EEuWLGFbL73zzjtk3Cm6rrOvN9+/fz+am5uxYsUKdk1Hjx7F2NgYqVu7di06OzvJP4KxsTF8/fXX7Nq4TsyX4rqJ3fe+972q8Xw+j48++ojUALMO3r17N2uQzs5OtLW1sePt2bMHHR0dWLhwIamrpUG4NZ05cwbRaJTV7dq1C6dPn8axY8dI3c9+9jOsXLmSbMV94sQJnD59mp3TyX3ni1rHSuGaRAwikIhBBBIxiEAiBhFIXJ1iSqUSjh49SsaVUqQGmG0ZHQqFsHz5clIXDocxMTHBjmdZFgYGBticQ62wbZtdUzabhWmarM40TbS2trK1LEopHD9+nOwmPTIyAsMw2DlHRkbI+KW4Mkgul8NHH31UNW7bNrxeL6kBZltBRyIR/OQnPyF1oVAI/f39GB0dJXWmaeLQoUNsoYyTFk5OUEqxz5jL5aDrOpsM1HUdN910E/vHopTCnj17yCOqYRiwLItdm2EYZPx/Tez046RgaOvWrWzByu7du6+JgqGdO3eyuq1bt0rBkPDvixhEIBGDCCRiEIFEDCKQiEEEElcFQwsWLFBPPvlk1bht2zAMg301mGEYMAwD5XKZ1Pn9fvh8PrZVtWVZAGYvF1GUSiX8/ve/r0nB0FNPPUVqyuUyNE1j157NZqGUYi+RRSIReL1eMqFmWRYMwyBf3w7M5o02b95c+4KhWCyG++67r2q8UChg+/btpAYADh48iHfffRdvvPEGqVu3bh1uu+02tgBm+/bt+PGPf8y+1PCll14i407xeDzsM+7atQt1dXX44Q9/yK7pvffeQ39/P6l7/PHH0d3dTX75AwMD6Ovrw4YNG8ixuNqTS3FlEK/XS34J+Xweuq6zX1Q0GoVhGGwVlWEYiEaj7HherxdNTU3f6lsvublCoRAikQir03Ud09PT7F7ouo6WlhayYCiTybDfEeAu1S6/QQQSMYhAIgYRSMQgAokYRCBxdYrJ5/Po6+urGi+Xy1BKkRpg9l5GXV0dVq9eTeqi0SiGhobY8UzTxJEjR3D27FlSVyts22bXNDk5iVwux+oMw8CyZcsQj8dJnVIK+/btI/MqqVQKpVKJnXN4eJiMX4org5TLZRw/frxqvJKwojTA7HEsFAqho6OD1AUCAYyPj6NYLLLrGhgYYBNEXNWWU9S/qrsostkscrkcq9M0Da2treylL8uycOrUKbJgqFAowDAMds58Pk/GL8NJ0UjlIwVDUjAkCJchBhFIxCACiRhEIBGDCCSujrlKKfJORSXG3buoNL7l7qlUmu1y46l/NZB1dd/jCuHmsm3b0doBZ01yK3NS41Ua9zrZf6e4KhiaP3++euyxx6rGlVLQdZ3tGlz5QjmdpmmONs7j8Th6aE3T0NPTU5OCIWofgP/unMzlXkqlkqN+HYFAgC0qUko5zvX87ne/c7YPTs7ClY/kQSQPIgiXIQYRSMQgAokYRCARgwgkYhCBxFWibGZmBv/85z+rxk3ThG3bpAYARkdHEY/H0d3dTeqamppw7NgxpNNpUmeaJvr6+vDVV1+RulphWRb7jGNjY5icnGR1hmHglltuYRvuKqWwa9cusmAonU6jUCiwc3KdsC/FVaJM07TzAJy36b06Waiu8JVk19I+uDKIcO0hv0EEEjGIQCIGEUjEIAKJGEQgEYMIJGIQgUQMIpCIQQQSMYhAIgYRSMQgAokYRCARgwgkYhCBRAwikLjttKyoa4KapqGpqYktaQuFQggEApieniZ1DQ0NKJfLbMukuXPnIhaLsX3Rz58/j9HR0dSVVpSFw2FlmiapiUajsG2bfdFiPB7HzMwM27e+qakJFy5cIK+r+v1+NDQ0sO2sisUiTp486WwfnFy/q3x8Ph95PTAUCqmenh72GmFXV5ejq5cPPvig6urqYnXPPvvst3r1sqWlhV1Td3e3uv3221ndxo0bHV29fOaZZ1Q4HCY17e3tatu2bXL1Uvj2EIMIJGIQgUQMIpCIQQQSV8fcQCBAvkLc7/dD0zR0dnaS4yQSCYRCIVYXDoeRSCRYHQCcPn2aPTbXCifPGIvFEAgEWJ2u61i0aBH8fj8774033kgeh1taWmAYBr788ktynHPnzrFzVXBlkGg0irvvvrtq3LIszMzMkBpg9hxuGAarm5mZQTAYxLx580idYRg4cuQImwfh3mnnFE3T2LXncjkAIP+ggNmW3d///vfZ9/yl02nccsstZB7E4/HAMAy2VzuXc7kMN+f/Wrag6u3tZXUvvfTSVZkHqWULqhdffFF98803rG7z5s2SBxGuPsQgAokYRCARgwgkYhCBRAwikLhuxf34449XjSs1+w56Lt9Q6WPOtdk2TROWZbEtqMPhMGzbZttQ27btvAU1QTKZVE888QSpqbQG5/ai0o+eW3uxWHTcupzbVzf74CpR1tjYCMog+XweL7/8MqkBgP3792NgYAB33nknqXvttdfw5ptvYt++faTu17/+NdatW4dkMknq/vrXv5Jxp/h8PvYZ33rrLdTV1WHNmjWk7pVXXkFXVxduuOEGUvfCCy/gb3/7G1k8dd111+Huu+/GQw89RI515MgRMn4prgzi8XjQ2NhYNR4IBKDrOqkBZivKfD4fq/P5fCgUCmwTO13XUVdXx45XS7i5AoEAQqEQq/N6vY7Xnk6nSYPMzMyw3xEARCIRdq4K8htEIBGDCCRiEIFEDCKQiEEEElenmHw+j08++aRqvFwuQylFagBgcHAQ6XSa1aXTaVx33XWO6jgOHz6M/v5+VlcLLMti155KpZDNZllduVzG0aNHMT4+TuqUUli9ejX5Prp4PI5SqcTOOTo6SsYvxZVByuUyBgYGqsYria1XX32VHKehoQHhcJjVJZNJ3HTTTfjRj35E6s6dO4fh4WFMTU2RulpC7QMwe+QcGhrC3r17SV0ymcTg4CB7wWrx4sXo7u4m/1hyuRwuXLjAri2bzZLxy3BSNFL5OCkYquXFqX/3d9bV8uKUvLNOuCoRgwgkYhCBRAwikIhBBBJXx1zbtlEoFKrGK7FgMEiO4/P5oGkaq9N1HaZpknMCsyexcrnM6moJN5dlWfB4POwzArMXzpzoisUiWTdiGAaUUuza3NyLcV0w9POf/7xqvPJF1Qpd19nLUMBsGUKlQIdC0zRs3rz5iguGuH0AZv+YqKRWhYqROILBIFswpJRiC48q9PT0ONsHN+f/WuZBXn75Zfa83tvb+x+fB9myZYuji1OSBxGuSsQgAokYRCARgwgkYhCBRAwikLhKlGUyGbz++utV46ZpQtd13HXXXeQ4lWa7O3bsIHWTk5MoFAoYGxsjdYZh4OOPP0Y4HCZ1tcKyLHIfAGB4eBjXX389uxcejweffvope1dFKYU333yTzJlks1nk83l2X93UzbgyCJf9NE0TSim2+CUajWJqaorthLNs2TJks1kMDw+TuhUrVuCrr75CJpMhdd/97hXlxy6Dy3xWbsxxe5HL5dDX14fJyUlSt3LlSuzYsQNUh+dYLIZFixZh+/bt5Fhz5zpvNO3KIPX19Vi7dm3VeD6fxxdffIG3336bHKerqwvLli1jdc3NzTh16hT27NlD6jo6OtDX14ehoSFSVyuDeDwech8A4O9//zs+/vhj9hnb29uxf/9+HDt2jNR1dnbi/fffJy9Otbe3o6WlhZ1z1apVZPxS5DeIQCIGEUjEIAKJGEQgEYMIJK5OMaZpYmRkpGq8WCwCmO2kTFFfXw+fz8fq/H4/otEoqwOAOXPmOKq/qAVKKXIfgNm9qHSKptA0DY2NjY6ecd68eWQxUDweh8fjYceKxWLsXBfXp1wUDGmadh7AoON/cHWyUF3hG6eupX1wZRDh2kN+gwgkYhCBRAwikIhBBBIxiEAiBhFIxCACiRhEIBGDCCT/BVjjAK0+bvCXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 144x216 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class ALM:\n",
    "    def __init__(self):\n",
    "        self.M = None\n",
    "        \n",
    "    def fit(self, x):\n",
    "        ##\n",
    "        ## Recibe una lista de matrices con los patrones\n",
    "        ##\n",
    "        x = [e.flatten() for e in x]            ## matrix a vector\n",
    "        x = (e.reshape(len(e), 1) for e in x)   ## vector-fila a vector-columna\n",
    "        A = np.hstack(x)                        ## matriz A = [x0 x1 x2]\n",
    "        An = []\n",
    "        for i in A:\n",
    "            xb = []\n",
    "            for j in i:         \n",
    "                if j < 0:\n",
    "                    xb.append(-1)\n",
    "                else: \n",
    "                    xb.append(1)\n",
    "            An.append(xb)\n",
    "        self.M = A @ np.transpose(An)          ## B * inv(At * A)\n",
    "        \n",
    "    def predict(self, x):\n",
    "        ##\n",
    "        ## x es una lista de matrices\n",
    "        ##\n",
    "        \n",
    "        m, n = x[0].shape\n",
    "        x = [e.flatten() for e in x]            ## matrix a vector\n",
    "        x = (e.reshape(len(e), 1) for e in x)   ## vector-fila a vector-columna\n",
    "        x = [self.M @ e for e in x]\n",
    "        \n",
    "        x = [e.reshape(m,n) for e in x]\n",
    "        xt = []\n",
    "        for i in x:\n",
    "            xa = []\n",
    "            for j in i:\n",
    "                xb = []\n",
    "                for k in j:\n",
    "                    if k < 0:\n",
    "                        xb.append(-1)\n",
    "                    else: \n",
    "                        xb.append(1)\n",
    "                xa.append(xb)\n",
    "            xt.append(xa)\n",
    "        return np.array(xt)\n",
    "\n",
    "alm = ALM()\n",
    "alm.fit([d0, d1, d2])\n",
    "n = alm.predict([d0, d1, d2])\n",
    "plot_digits(x=[d0, d1, d2], y=n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIgAAABPCAYAAAAwV41eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAB6tJREFUeJztnc1rE2sbxq+JmaSTREsTTJv0U7FgVdpFD3Uh3SiKiF0IgggutNA/QLqU4AdW3BTxD+haxI0ICuLGSi0oFip+NWiprdq0Na12miYzaWaes+jbg+/Lm/vOcMI5B879g1ndV5+558nVNrnyzDOaUgqCUA7f392A8M9GDCKQiEEEEjGIQCIGEUjEIAKJGEQgEYMIJGIQgUQMIpD4vYij0ajavXt32brrulhYWEAmkyHHiUQiqKmpQTabJXWxWAy2bSOXy5G6ZDKJWCyGQCBA6ubn55HJZLJKqZ2kkCESiai9e/eSmpWVFaysrGB1dZXUxeNxmKYJy7JIXUNDA5aWluC6bllNMBjEzp07UV9fT46Vz+fx4cOHyuZBKVXx0dnZqSjW19dVKpVSAMijt7dXDQwMsLr+/n7V29vL6oaGhtTnz5/J3pRS6sqVKwrAKy/X/P+OlpYW9lx37txRJ0+eZHu/ePGi6ujoYHWXLl1SoVCI1LS3t6vh4WG2t1evXlU8D/IvRiARgwgkYhCBRAwikIhBBBIxiEDiKQexLAuvX78uW7dtG5qmoaurixwnkUjAMAxWFwqFkEgkWB0AfPr0CT9//mR11cB1XXIeAGB1dRWxWIzt3efzoa2tjc1wAGD//v0oFotl6/X19djY2GB7+/r1K3uuLTTlYU1qMplUFy5cKD+YpsHn8yEcDpPjWJaFjY0NbN++ndStr69D0zSEQiFWBwCO45A6wzCQSqUmlFK/kUKGxsZGdf78eVITDAZRKpXYuVhbW0MgEEAwGCR1pmkiEonA5yv/R79YLEIpBdu2ybF0XcfVq1crmwcvAZGu62RQYxiGunbtGhvUjI6OqpGREVY3MjKiRkdHWV0qlVItLS1s2FStoKy+vp4915kzZ9TDhw/Z3m/fvq3ev3/P6q5fv67W19dJzdTUlBocHGR76+7ulqBMqA5iEIFEDCKQiEEEEjGIQCIGEUjEIAKJp6AsHo+rU6dOlR9M05BIJNDY2EiOs7WCKh6Pk7rv378jGAxix44dpG5mZgamaZIpI7CZ4FYcEBEkk0nV19dHaqLRKKLRKOrq6khdNptFOByGYRikbmFhAfF4nAzKLMvC0tISFhcXybHC4TBu3bpV0Tx4itpt28a7d+/K1nVdRyKRQEdHBznO9PQ0stksqzNNE7FYDHv27CF1s7OzmJmZYaP2ZDJJ1itFKUXOAwB0dXWhra2NvcaxsTEkk0k0NDSQukwmg/b2djKSX1xcxPz8PNsb9wv8K54MUigU8Pz587J1wzBw+PBhHDp0iBzHcRyUSiVWl06nsWvXLlb3+PFjvHnzBnNzc6Tu6NGjZL1SlFLkPABAU1MTmpub2d4nJiawb98+1khPnz7FwYMHya8d0uk0xsfH2d66u7vJ+q/IexCBRAwikIhBBBIxiEAiBhFIxCACiaePuT6fj1wFVlNTA2BzlRSFbdtwHIfVOY4D27ZZnVIKoVCIXaFWTbhz6bqOjY0NtnfXdVEoFFgdAORyOXLVXKFQgKZpbG9br1MleEpSE4mE6u/vJzWGYZBpH7D5ggKbySuF67rsWMDmvaaVXIemaRgaGvrTSWol86BpGgzDYK+xUjRNY69RKQXLssj7d7e4ceOGLDn830OWHMqSQ6HKiEEEEjGIQCIGEUjEIAKJGEQgEYMIJJ6SVMMw0NPTU7au6zqUUhgbGyPHmZ2dhWmarM40TczNzbE6x3HQ2dmJ5uZmUlctNE1jFwLFYjFkMhm292KxiKmpKSwvL7PnffHiBXRdL1vPZrMIBAJsb01NTZiYmGDPB3g0SG1tLc6dO1e27jgOstkspqamyHG21qRyunw+j8XFRXYHwGAwiCNHjrARMrf7ohe4FWCRSATLy8vsNdq2jbm5OdYgruvi48ePZLJcKBTg9/vZ3rib4f8LLwliJbsc/h1J6vDw8F+6y6EkqYLwH8QgAokYRCARgwgkYhCBRAwikIhBBBJPSw6bm5vV4OBg2brrulhbW0NtbS05jmVZKJVKiEQipC6Xy8Hv97MBWKFQgK7r8Pvp3M80TVy+fLkqN2+fPXuW1BiGgVAoxN6UbVkWtm3bRiakwOZOjqFQiFzCWCqVkMvl2MenBAIB3Lx5s/o3b4fDYRw/frxs3bZt3L9/n9QAwNu3b5HJZNh7ZZ88eYJEIoEDBw6QugcPHqCnp4fdLeDu3btkvVL8fj8GBgZIzbNnzzA2NoaXL1+Sur6+PkxOTuLLly+k7vTp07h37x65xWUymcSxY8fAmXd6epqs/4ongwSDQVAP0snn8/D5fKQGAJaWlmCaJqsbHx9HPB5ndY8ePUJraytaW1tJXbXQNI3taXJyEj9+/EA6nSZ1J06cwLdv31gdsLlZcD6fL1t3XReBQIDtbWtf2UqQ9yACiRhEIBGDCCRiEIFEDCKQiEEEEk8fc0ulEubn58vWt1Z+URoAf+xIyOmKxSLW1tZYneu6WF5eZsOmaqGUYnuyLOuP591QaJqGuro6VgdsPju3UCiUrUejUTiOw/bm5bk6npJUTdO+A5it+Af+mbSqP/lg5X/TPHgyiPDvQ96DCCRiEIFEDCKQiEEEEjGIQCIGEUjEIAKJGEQgEYMIJL8D1seh6AymcJEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 144x86.4 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "##\n",
    "## Se ejecuta el modelo sobre patrones incompletos\n",
    "## o contaminados con ruido\n",
    "##\n",
    "n0 = np.array([[-1, -1, -1, -1, -1],\n",
    "               [-1, -1, -1, -1, -1],\n",
    "               [+1, -1, -1, -1, +1],\n",
    "               [+1, -1, -1, -1, +1],\n",
    "               [+1, -1, -1, -1, +1],\n",
    "               [-1, +1, +1, +1, -1]])\n",
    "\n",
    "plot_digits(x=[n0], y=alm.predict([n0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIgAAADQCAYAAADcZynIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFG9JREFUeJztnWuMlOXZx//PM+eZ3ZndWdhlBnYBdYHFyIbSxla6SUNqP1hJQ2JcRdRoPFWrCT2ERtmW0lq/iI39aNwawwcbTE20omg8EVYiCIgics4eYI/MsjszO8fncL8ftsML7/vOdT1PmPjScv2S+XT9ue/7uee/OveV674eTSkFQaiG/v+9AOHqRgwikIhBBBIxiEAiBhFIxCACiRhEIBGDCCRiEIHE60Yci8XU/Pnzq8Zt20Ymk0E6nSbHCQQC8Pl8mJmZIXV1dXUwDAOlUonUNTY2IhwOw+ulH2d6ehqjo6MppdRcUsgQiUTUwoULSU0ul0Mul0OhUCB19fX1KBaLMAyD1EWjUWSzWVCZb6/Xi/r6ekSjUXKscrmMM2fOONsHpZTjz4oVKxRFLpdTPT09CgD56erqUg8//DCre/DBB1VXVxere/bZZ9XAwAC5NqWU2rJliwJwwM0z/1+ftrY2dq7XXntN3X777ezaN27cqDo6OljdM888o8LhMKlpb29X27ZtY9d24MABx/sg/4sRSMQgAokYRCARgwgkYhCBxNUx1zRNTExMVI0Xi0UAwNy59OkpEonA6/WyOp/Ph7q6OlYHzB5hQ6EQq6sFSilyHwCgVCohGAyya9c0DbFYzNEzzpkzhzw2NzQ0wLZtdm3ZbJad6+L6lIuKsgULFqgnnniialwphXw+z45z8Qil0/8Bc6oLBAJsDgSYzdNs3rz5oFLqu6yYYP78+eoXv/gFO1cul2PXblkWNE1jdbZtQ9M0aJpWVaOUQjgcZscCgKefftrZPrg5//t8PvIcHgqF1NatW9lz+O7du1Vvby+r6+3tVbt372Z1PT09qq2tjc0l1CoP0tLSws7V3d2tdu7cya79xRdfVN988w2r+9Of/qRyuRypOX78uPrVr37Frm3VqlWSBxFqgxhEIBGDCCRiEIFEDCKQiEEEEld5kObmZrVu3brqg2kaEokEqJoRAMhkMigWi2hubiZ158+fRyAQYOsb+vv7kclkUC6XSV0ikcAf/vCHK86DJJNJtXbtWlITj8cRj8fR2NhI6lKpFCKRCJvkGxsbQ3NzM5njKBaLmJiYwPj4ODlWJBLBX/7yF0f74CqTWiqVcPTo0apxn8+HRCKBjo4OcpwzZ84glUqxukwmg6amJtxwww2kbnBwEP39/ZieniZ1yWSSjDtFKUXuAwB0dnZi0aJF7DP29fUhmUxi3rx5pG50dBTt7e3w+/1VNePj4xgZGWHXxv0BX4orgxQKBXz66adV46FQCGvWrMHq1avJcSzLgmmarO7EiRNYvHgxq3vvvfdw5MgRDA0Nkbpbb72VjDtFKUXuAwAsWLAAra2t7NoPHjyI5cuXs0b65JNPcPPNNyMcDlfVnDhxAnv37mXXtmrVKjJ+KfIbRCARgwgkYhCBRAwikIhBBBJXpxi/34/FixdXjQcCASilcOLECXKcVCqFQqHA6gqFAlKpFKuzbRttbW0IBoOkrlZomoYlS5aQmvr6ekxPT7NrN00Tw8PDjmo4Tp8+jUAgUDU+MjICr9fLrq2lpYWdq4KrRNm8efPUPffcQ2oaGhrYxFapVIJt22xyKJ/Pw+v1kmd/AJicnEShUIBt26QuGAziz3/+8xUnyhKJhFq/fj2p8fv9aGhoYE1bKBTg8/nYgqd8Po9QKEQWDJmmiQsXLlys7KuGrut4/vnnpWDof36kYEgKhoQaIwYRSMQgAokYRCARgwgkYhCBxFUeJJFIqHvvvZfURCIRRCIRUlMul2FZFpsHKRaL0HWdzYNMTU3Btm1YlkXqwuFwzQqGNmzYQGoCgQCCwSCZ2AJm90LXdTYPUiwWEQgE2DxIsVhkL695vV4899xztS8YymQy2LFjR9V4IBDA+vXrcccdd5DjHD58GMPDw/jpT39K6nbu3IlkMomVK1eSut7eXnzwwQdsJdUDDzxAxp2ilCL3AQDWrFmDW2+9FT/4wQ9I3T/+8Q+sWrUKixYtInXbt2/H2rVrycTb2bNn8fbbb7Nr4zKtl+LKIIZhYHBwsGo8FApB13X2YYeGhnDhwgVWFwqF0NTUxOo0TcPY2BhbMFQrlFLkPgCzmc9YLMau3efzoaWlhdXpuo62tjayYKhUKsGyLHZtc+bMIeOXzetYKVyTiEEEEjGIQCIGEUjEIAKJq1NMKBTCd77znapxn88HpRQ+++wzcpyzZ88im82yumw2i3PnzrE6y7Jw4403IpFIkLpaoWkabr75ZlITj8cxNjbGrt0wDJw8eZJtPgwABw4cIHNCExMT8Pl87NpaW1tx8OBBdj7AZaKstbVV/eY3v6kat20b6XQaU1NT5DjhcBiBQIDVxWIxmKaJXC5H6uLxOEKhEJuUSqVS+OMf/1iTDkObNm0iNdPT0zAMg2331NjYiHw+z3aTjsfjmJ6eJoui/H7/xdQARbFYxKZNm2pfMCSdlqXTsiBchhhEIBGDCCRiEIFEDCKQiEEEEld5kGQyqR599FFSY9u2o1tiSimy+MXNWF6vF6ZpsrpgMIjf/va3NcmDPPLII1cyxEUMw4DP52N1pmk66iataRqcfKdbtmxxtA+uDKJp2nkAdLHB1c9CdYWvJLuW9sGVQYRrD/kNIpCIQQQSMYhAIgYRSMQgAokYRCARgwgkYhCBRAwikIhBBBIxiEAiBhFIxCACiRhEIBGDCCSurl7W1dUp7taWYRhsS+zKO+g5na7r8Hg8bFWZZVnQdZ2tUDNNEyMjI6krLRiKRCKKa8JiWRYsy2Kruypr5nQej4etKFNKwbIsVmfbNs6dO+dsH9zcKPu2b9Y99NBDjlpxb9u27T/+Zp3TVtwvvPACuza5WSfUDDGIQCIGEUjEIAKJGEQgEYMIJK7uxbS1tamnn366aty2bUxMTCCVSpHjhMNh+P1+tsNQPB5HLBZDXV0dqZuamkIkEmFbdo+Pjzu+UUYxf/581dPTw64pnU5jZmaG1NXX16NUKrEdhhKJBOLxOJkTKpVKyGQymDuXTm8UCgX88pe/rH0r7mAwSL7Wu1wu44MPPsD9999PjnPy5EkcOnQIn3/+OalbvXo1li5diqVLl5K6Dz/8EEuWLGFbL73zzjtk3Cm6rrOvN9+/fz+am5uxYsUKdk1Hjx7F2NgYqVu7di06OzvJP4KxsTF8/fXX7Nq4TsyX4rqJ3fe+972q8Xw+j48++ojUALMO3r17N2uQzs5OtLW1sePt2bMHHR0dWLhwIamrpUG4NZ05cwbRaJTV7dq1C6dPn8axY8dI3c9+9jOsXLmSbMV94sQJnD59mp3TyX3ni1rHSuGaRAwikIhBBBIxiEAiBhFIXJ1iSqUSjh49SsaVUqQGmG0ZHQqFsHz5clIXDocxMTHBjmdZFgYGBticQ62wbZtdUzabhWmarM40TbS2trK1LEopHD9+nOwmPTIyAsMw2DlHRkbI+KW4Mkgul8NHH31UNW7bNrxeL6kBZltBRyIR/OQnPyF1oVAI/f39GB0dJXWmaeLQoUNsoYyTFk5OUEqxz5jL5aDrOpsM1HUdN910E/vHopTCnj17yCOqYRiwLItdm2EYZPx/Tez046RgaOvWrWzByu7du6+JgqGdO3eyuq1bt0rBkPDvixhEIBGDCCRiEIFEDCKQiEEEElcFQwsWLFBPPvlk1bht2zAMg301mGEYMAwD5XKZ1Pn9fvh8PrZVtWVZAGYvF1GUSiX8/ve/r0nB0FNPPUVqyuUyNE1j157NZqGUYi+RRSIReL1eMqFmWRYMwyBf3w7M5o02b95c+4KhWCyG++67r2q8UChg+/btpAYADh48iHfffRdvvPEGqVu3bh1uu+02tgBm+/bt+PGPf8y+1PCll14i407xeDzsM+7atQt1dXX44Q9/yK7pvffeQ39/P6l7/PHH0d3dTX75AwMD6Ovrw4YNG8ixuNqTS3FlEK/XS34J+Xweuq6zX1Q0GoVhGGwVlWEYiEaj7HherxdNTU3f6lsvublCoRAikQir03Ud09PT7F7ouo6WlhayYCiTybDfEeAu1S6/QQQSMYhAIgYRSMQgAokYRCBxdYrJ5/Po6+urGi+Xy1BKkRpg9l5GXV0dVq9eTeqi0SiGhobY8UzTxJEjR3D27FlSVyts22bXNDk5iVwux+oMw8CyZcsQj8dJnVIK+/btI/MqqVQKpVKJnXN4eJiMX4org5TLZRw/frxqvJKwojTA7HEsFAqho6OD1AUCAYyPj6NYLLLrGhgYYBNEXNWWU9S/qrsostkscrkcq9M0Da2treylL8uycOrUKbJgqFAowDAMds58Pk/GL8NJ0UjlIwVDUjAkCJchBhFIxCACiRhEIBGDCCSujrlKKfJORSXG3buoNL7l7qlUmu1y46l/NZB1dd/jCuHmsm3b0doBZ01yK3NS41Ua9zrZf6e4KhiaP3++euyxx6rGlVLQdZ3tGlz5QjmdpmmONs7j8Th6aE3T0NPTU5OCIWofgP/unMzlXkqlkqN+HYFAgC0qUko5zvX87ne/c7YPTs7ClY/kQSQPIgiXIQYRSMQgAokYRCARgwgkYhCBxFWibGZmBv/85z+rxk3ThG3bpAYARkdHEY/H0d3dTeqamppw7NgxpNNpUmeaJvr6+vDVV1+RulphWRb7jGNjY5icnGR1hmHglltuYRvuKqWwa9cusmAonU6jUCiwc3KdsC/FVaJM07TzAJy36b06Waiu8JVk19I+uDKIcO0hv0EEEjGIQCIGEUjEIAKJGEQgEYMIJGIQgUQMIpCIQQQSMYhAIgYRSMQgAokYRCARgwgkYhCBRAwikLjttKyoa4KapqGpqYktaQuFQggEApieniZ1DQ0NKJfLbMukuXPnIhaLsX3Rz58/j9HR0dSVVpSFw2FlmiapiUajsG2bfdFiPB7HzMwM27e+qakJFy5cIK+r+v1+NDQ0sO2sisUiTp486WwfnFy/q3x8Ph95PTAUCqmenh72GmFXV5ejq5cPPvig6urqYnXPPvvst3r1sqWlhV1Td3e3uv3221ndxo0bHV29fOaZZ1Q4HCY17e3tatu2bXL1Uvj2EIMIJGIQgUQMIpCIQQQSV8fcQCBAvkLc7/dD0zR0dnaS4yQSCYRCIVYXDoeRSCRYHQCcPn2aPTbXCifPGIvFEAgEWJ2u61i0aBH8fj8774033kgeh1taWmAYBr788ktynHPnzrFzVXBlkGg0irvvvrtq3LIszMzMkBpg9hxuGAarm5mZQTAYxLx580idYRg4cuQImwfh3mnnFE3T2LXncjkAIP+ggNmW3d///vfZ9/yl02nccsstZB7E4/HAMAy2VzuXc7kMN+f/Wrag6u3tZXUvvfTSVZkHqWULqhdffFF98803rG7z5s2SBxGuPsQgAokYRCARgwgkYhCBRAwikLhuxf34449XjSs1+w56Lt9Q6WPOtdk2TROWZbEtqMPhMGzbZttQ27btvAU1QTKZVE888QSpqbQG5/ai0o+eW3uxWHTcupzbVzf74CpR1tjYCMog+XweL7/8MqkBgP3792NgYAB33nknqXvttdfw5ptvYt++faTu17/+NdatW4dkMknq/vrXv5Jxp/h8PvYZ33rrLdTV1WHNmjWk7pVXXkFXVxduuOEGUvfCCy/gb3/7G1k8dd111+Huu+/GQw89RI515MgRMn4prgzi8XjQ2NhYNR4IBKDrOqkBZivKfD4fq/P5fCgUCmwTO13XUVdXx45XS7i5AoEAQqEQq/N6vY7Xnk6nSYPMzMyw3xEARCIRdq4K8htEIBGDCCRiEIFEDCKQiEEEElenmHw+j08++aRqvFwuQylFagBgcHAQ6XSa1aXTaVx33XWO6jgOHz6M/v5+VlcLLMti155KpZDNZllduVzG0aNHMT4+TuqUUli9ejX5Prp4PI5SqcTOOTo6SsYvxZVByuUyBgYGqsYria1XX32VHKehoQHhcJjVJZNJ3HTTTfjRj35E6s6dO4fh4WFMTU2RulpC7QMwe+QcGhrC3r17SV0ymcTg4CB7wWrx4sXo7u4m/1hyuRwuXLjAri2bzZLxy3BSNFL5OCkYquXFqX/3d9bV8uKUvLNOuCoRgwgkYhCBRAwikIhBBBJXx1zbtlEoFKrGK7FgMEiO4/P5oGkaq9N1HaZpknMCsyexcrnM6moJN5dlWfB4POwzArMXzpzoisUiWTdiGAaUUuza3NyLcV0w9POf/7xqvPJF1Qpd19nLUMBsGUKlQIdC0zRs3rz5iguGuH0AZv+YqKRWhYqROILBIFswpJRiC48q9PT0ONsHN+f/WuZBXn75Zfa83tvb+x+fB9myZYuji1OSBxGuSsQgAokYRCARgwgkYhCBRAwikLhKlGUyGbz++utV46ZpQtd13HXXXeQ4lWa7O3bsIHWTk5MoFAoYGxsjdYZh4OOPP0Y4HCZ1tcKyLHIfAGB4eBjXX389uxcejweffvope1dFKYU333yTzJlks1nk83l2X93UzbgyCJf9NE0TSim2+CUajWJqaorthLNs2TJks1kMDw+TuhUrVuCrr75CJpMhdd/97hXlxy6Dy3xWbsxxe5HL5dDX14fJyUlSt3LlSuzYsQNUh+dYLIZFixZh+/bt5Fhz5zpvNO3KIPX19Vi7dm3VeD6fxxdffIG3336bHKerqwvLli1jdc3NzTh16hT27NlD6jo6OtDX14ehoSFSVyuDeDwech8A4O9//zs+/vhj9hnb29uxf/9+HDt2jNR1dnbi/fffJy9Otbe3o6WlhZ1z1apVZPxS5DeIQCIGEUjEIAKJGEQgEYMIJK5OMaZpYmRkpGq8WCwCmO2kTFFfXw+fz8fq/H4/otEoqwOAOXPmOKq/qAVKKXIfgNm9qHSKptA0DY2NjY6ecd68eWQxUDweh8fjYceKxWLsXBfXp1wUDGmadh7AoON/cHWyUF3hG6eupX1wZRDh2kN+gwgkYhCBRAwikIhBBBIxiEAiBhFIxCACiRhEIBGDCCT/BVjjAK0+bvCXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 144x216 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "##\n",
    "## Ya que se requiere que la salida sea {-1, +1}\n",
    "## se modifica el codigo anterior agregando la\n",
    "## función de activación de paso duro\n",
    "##\n",
    "class ANLM:\n",
    "    def __init__(self):\n",
    "        self.M = None\n",
    "        \n",
    "    def fit(self, x):\n",
    "        x = [e.flatten() for e in x]            ## matrix a vector\n",
    "        x = (e.reshape(len(e), 1) for e in x)   ## vector-fila a vector-columna\n",
    "        A = np.hstack(x)                        ## matriz A = [x0 x1 x2]\n",
    "        self.M = A @ np.linalg.pinv(A)          ## B * inv(At * A)\n",
    "        \n",
    "    def predict(self, x):\n",
    "        m, n = x[0].shape\n",
    "        x = [e.flatten() for e in x]            ## matrix a vector\n",
    "        x = (e.reshape(len(e), 1) for e in x)   ## vector-fila a vector-columna\n",
    "        x = [self.M @ e for e in x]\n",
    "        ##\n",
    "        ## ----- función de activación -------\n",
    "        ##\n",
    "        x = [np.sign(e) for e in x]\n",
    "        ##\n",
    "        ## ------------------------------------\n",
    "        ##\n",
    "        x = [e.reshape(m,n) for e in x]\n",
    "        return x\n",
    "\n",
    "anlm = ANLM()\n",
    "anlm.fit([d0, d1, d2])\n",
    "n = anlm.predict([d0, d1, d2])\n",
    "plot_digits(x=[d0, d1, d2], y=n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIgAAABPCAYAAAAwV41eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAABzFJREFUeJztnc1rE1sYxp+ZpB9J02oTTYkxjYqBBsRurnYh4kIEBf8BcasrN4oIghZrcav/QdfiTgTdKEJLFQJVlFbSooXE5lOTtkk6yWQmmXMXJReFm/fMcHO5Xnx/MKv36TnvnDzpx9NzMooQAgzTDfW/boD5tWGDMCRsEIaEDcKQsEEYEjYIQ8IGYUjYIAwJG4QhYYMwJG4nYr/fL44cOdK1blkWCoUC8vk8OY7P58Pg4CBKpRKp27dvHwKBAHw+H6krFosYHR1Ff38/qcvlcsjn8yUhxH5SKMHn84mJiQlSs7m5ic3NTVQqFVIXDAZRrVah6zqpO3DgAMbGxqCq3d/Tuq6jUqlgbGyMHKteryOZTNpbByGE7ev48eOCQtM0MT09LQCQ1+nTp8XVq1eluitXroj5+XlyTiGEePjwoUilUlLdzMyMALDk5J7/7hofH5fO9fjxY3Hx4kXpPd64cUPE43Gp7sGDB0LTNHLO1dVV8ejRI2lvS0tLtteBf8QwJGwQhoQNwpCwQRgSNghDwgZhSBzlILqu4+PHj13rzWYTiqJgcnKSHCcUCsHj8Uh1Q0NDKBQK5JwA0G638eXLF2xvb5O6XmFZlrSnSqWCQCAgvUdVVXHo0CFphiOEwMrKCgYGBrpqCoUCDMOQ9pbJZMj6jzgySKPRwOLiYte6ZVkQQuDSpUvkOLquQ9d1nDlzhtR5vV6k02l8//6d1LVaLSwvL6Ovr4/UuVwusm4XIQS5DgBQq9UwPj6OeDxO6ra3txGPxxGLxUidZVlIJBJkUGYYBtrttrQ3wzDI+k84CYjsBGWzs7PSoGZ+fv63CMqeP38u1c3OznJQxvx/YYMwJGwQhoQNwpCwQRgSNghDwgZhSBTh4PB2NBoV09PTXeuWZSGfzyMcDpPjVKtVlMtl6Y6y/fv3w+/3Y2RkhNSVSiUMDw+TKSMAZLNZzMzMvBNC/EEKJYTDYXH//n1pT263G3v37iV1qVQKtVpNuqMsEokgGAxKd5RtbW0hFAqRY2mahuvXr9taB0dJan9/P5kMGoaBfD4vTQ/X19eRy+Xw6dMnUjc1NYVAIICjR4+SuoWFBUQiEQQCAVKXzWbJul1UVZXe4/v376Xr1ekpk8mgWCySuoMHDyIWi5GRfLFYRK1Wk875r0XtXq8Xp06d6lqv1+tYXFwkNQD+ioPfvHlD6uLxOA4fPiwdL5FIYHJyEtFolNS9evWKrNtFVVVpTxsbGxgZGZHqXr9+jWQyiWQySeouXLiAqakpeL3erpq1tTWk02npnO/evSPrP8K/gzAkbBCGhA3CkLBBGBI2CEPCBmFIHP2Za1kWarVa13q9XgcAUgPsbk10uVwYHh4mdS6XC81mUzqeEAL1el2q6yWyuVqtFkzTtNW7x+ORrgUA7OzsoN1ud603Gg3pawRAGsr9iKMkNRwOi2vXrnWtd8ZSFIUcRwiBRqMhnU9VVQwODkp1iqLAzn0IIXD37t2eJKnUOgC7byYq9ezQeVFleL1eW+sKyNcfAO7cuWNvHexsO+tcvdxyODc3J9XNzc3xlkPecsj8yrBBGBI2CEPCBmFI2CAMCRuEIWGDMCSOktTOhqBuGIZh69xqOp1GtVqV6qrVKr5+/SrVdc7mbmxskLpeYVmWtKdyuQxN06Q60zQxMTEBv99P6oQQSCQS5PnjUqmEZrMpndPJzjpHBjEMA6urq13rnRiY0gD461P9ZLp6vY5isSiNhg3DQCqVkqaudhJGOwghpL3XajVomibVKYpia7tku93G58+fyXS20WjANE1b62obO2la5+IklZNUhvkJNghDwgZhSNggDAkbhCFhgzAkbBCGxNGWw0gkIm7evNm13tkPuWfPHnIcXdfRarWkj/nY2dmB2+2WBmCNRgN9fX1wu+ncr1qt4t69ez3Zcnjr1i1So2kaFEUhj0oCu+mnaZowTZPU+f1+eL1eMigzTRONRkN62L3ZbOL27du9P7w9NDSE8+fPkxM/ffqU1ADAysoK8vk8zp07R+pevnyJUCiEY8eOkbpnz57h5MmTCAaDpO7Jkydk3S4ul0t6jwsLC/B4PDhx4oS0p0QiIf03weXLl3H27FnyEwwymQyWlpakva2vr5P1H3FkkIGBAVAP0qnX61BVldQAwLdv31CtVqW6t2/fIhgMSnUvXrxANBqVHt7uFYqiSHv68OEDRkZGpDpVVZHNZrG2tibVxWIx8juSoihYXl6WzqlpGln/aV7bSua3hA3CkLBBGBI2CEPCBmFI2CAMiaM/c1utFnK5XNd6Z+cXpQF2AyvDMKQ6wzBQq9WkOsuyUC6XpY8D6RVCCGlPuq7D5XLZ6n10dFT6yYTA7vNgqNBwa2sL7XZbOqeT5+o4SlIVRfkOIG37C35NouIfPlj5d1oHRwZhfj/4dxCGhA3CkLBBGBI2CEPCBmFI2CAMCRuEIWGDMCRsEIbkT3IOr0u2ES1vAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 144x86.4 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "##\n",
    "## Se ejecuta el modelo sobre patrones incompletos\n",
    "## o contaminados con ruido\n",
    "##\n",
    "n0 = np.array([[-1, -1, -1, -1, -1],\n",
    "               [-1, -1, -1, -1, -1],\n",
    "               [-1, -1, -1, -1, -1],\n",
    "               [-1, -1, +1, -1, -1],\n",
    "               [-1, -1, +1, -1, -1],\n",
    "               [-1, +1, +1, -1, -1]])\n",
    "\n",
    "plot_digits(x=[n0], y=anlm.predict([n0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Ejercicio.--** Verifique si las siguientes reconstrucciones son correctas.\n",
    "\n",
    "![alt](images/ModelosAsociativos-07.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Ejercicio.--** Implemente el algoritmo anterior usando las operaciones elementales de `TensorFlow` y con compare los resultados obtenidos usando `numpy`.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solución mediante una memoria asociativa bidireccional (BAM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Es una memoria asociativa lineal donde la señal se propaga de una capa a otra (retropropagación) hasta estabilizarse (la salida de la red se hace estable). El modelo esquemático es el siguiente:\n",
    "\n",
    "![alt](images/ModelosAsociativos-08.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A continuación se ejemplifica el proceso de propagación de una señal a través de la BAM. El $k$-ésimo patrón, $\\mathbf{a}_k(1)$, es patrón $\\mathbf{a}_k$ en la iteración 1; entonces, la propagación de la señal se realiza de la siguiente manera:\n",
    "\n",
    "* $\\mathbf{b}_k(1) = \\text{sign}(\\mathbf{M} \\, \\mathbf{a}_k(1))$\n",
    "\n",
    "\n",
    "* $\\mathbf{a}_k(2) = \\text{sign}(\\mathbf{M}^T \\, \\mathbf{b}_k(1))$\n",
    "\n",
    "\n",
    "* Y así sucesivamente hasta que $\\mathbf{b}_k(t+1) = \\mathbf{b}_k(t)$ y $\\mathbf{a}_k(t+1) = \\mathbf{a}_k(t)$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La matriz $\\mathbf{M}$ se calcula como:\n",
    "\n",
    "$$\\mathbf{M} = \\sum_{k=1}^Q \\textbf{b}_k \\mathbf{a}_k^T$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Ejercicio.--** Implemente una BAM usando `TensorFlow`. Verifique la reconstrucción de los patrones incompletos y contaminados con ruído.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solución mediante una red discreta de Hopfield\n",
    "\n",
    "Para una matriz lineal auto-asociativa, los coeficientes se calculan como:\n",
    "\n",
    "$$\\mathbf{M} = \\sum_{k=1}^Q \\mathbf{a}_k \\mathbf{a}_K^T$$\n",
    "\n",
    "La red discreta de Hopfield se obtiene al unir los nodos de entrada y salida en una sola capa:\n",
    "\n",
    "![alt](images/ModelosAsociativos-09.png)\n",
    "\n",
    "Este tipo de red neuronal tiene las siguientes características:\n",
    "\n",
    "* La función de activación de las neuronas es $f(u)=\\text{sign}⁡(u)$.\n",
    "\n",
    "\n",
    "* Los elementos de los vectores a asociar son bipolares.\n",
    "\n",
    "\n",
    "* Los pesos de las conexiones se calculan como:\n",
    "\n",
    "$$m_{i,j} = \\frac{1}{P} \\sum_{p=1}^P (a_{p,i} * a_{p,j})$$\n",
    "\n",
    "\n",
    "* \tLos nodos toman valores $\\{-1,+1\\}$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La propagación de la señal puede ser sincrónica o asincrónica.\n",
    "\n",
    "* Prograpación asincrónica: cuando un patrón es presentado, el valor de un solo nodo es actualizado en cada instante de tiempo hasta que la red se estabiliza.\n",
    "$$\n",
    "x_{p,i}(t+1)=\n",
    "\\begin{cases}\n",
    "x_{p,i}(t),         & \\text{if } i = k \\\\\n",
    "\\text{sign} \n",
    "\\left(  \n",
    "\\sum_i^n w_{i,j} \\cdot  x_{p,j}(t) + I_{p,i}\n",
    "\\right), & \\text{if } i \\ne k\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "  - $x_{p,i}$ representa el estado actual de la neurona $i$ cuando se presentó el patrón $p$\n",
    "  \n",
    "  - $I_{p,i}$ es la entrada externa a la neurona $i$ para el patrón $p$ en el instante actual.\n",
    "\n",
    "\n",
    "* Propagación sincrónica:\n",
    "$$\n",
    "x_{p,i}(t+1)=\n",
    "\\text{sign} \n",
    "\\left(  \n",
    "\\sum_i^n w_{i,j} \\cdot  x_{p,j}(t) + I_{p,i}\n",
    "\\right)\n",
    "$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Ejercicio.--** Calcule los pesos de los nodos de una red discreta de Hoptfield para los patrones $[(+1,+1,+1),(-1,-1,-1),(+1,-1,-1)]$\n",
    "\n",
    "---\n",
    "\n",
    "**Ejercicio.--** Realice la propagación sincrónica de los siguientes patrones suponiendo que los pesos se actualizan secuencialmente.\n",
    "\n",
    "* $(+1,+1,-1)$\n",
    "\n",
    "\n",
    "* $(-1,+1,+1)$\n",
    "\n",
    "\n",
    "* $(0,0,+1)$\n",
    "\n",
    "---\n",
    "\n",
    "**Ejercicio.--** Implemente el algoritmo en `TensorFlow`.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Brain-state-in-a-Box"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Es similar a una memoria discreta de Hopfield con las siguientes variaciones:\n",
    "\n",
    "\n",
    "* La función de activación de las neuronas es $f(u)=\\min⁡(1,\\max⁡(-1,u))$.\n",
    "\n",
    "\n",
    "* Todos los nodos son actualizados simultáneamente\n",
    "\n",
    "\n",
    "* Los nodos toman valores en el intervalo $[-1,+1]$.\n",
    "\n",
    "\t\n",
    "* Los patrones a asociar son bipolares.\n",
    "    \n",
    "\n",
    "* Los pesos de las conexiones se calculan como:\n",
    "\n",
    "\n",
    "$$m_{i,j}=\\frac{1}{P} \\sum_{p=1}^P \\left( a_{p,i} \\cdot a_{p,j} \\right) $$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**Ejercicio.--** Calcule los pesos de los nodos para los patrones $[(+1,+1,+1),(-1,-1,-1),(+1,-1,-1)]$\n",
    "\n",
    "---\n",
    "\n",
    "**Ejercicio.--** Realice la propagación del patrón de entrada $(0.5, 0.6, 0.1)$ hasta que la red se estabilice.\n",
    "\n",
    "---\n",
    "**Ejercicio.--** Implemente el algoritmo en `TensorFlow`.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modelos Asociativos y Aprendizaje Hebbiano\n",
    "===\n",
    "\n",
    "**Juan David Velásquez Henao**  \n",
    "jdvelasq@unal.edu.co   \n",
    "Universidad Nacional de Colombia, Sede Medellín  \n",
    "Facultad de Minas  \n",
    "Medellín, Colombia\n",
    "\n",
    "---\n",
    "\n",
    "Haga click [aquí](https://github.com/jdvelasq/deep-neural-nets/tree/master/) para acceder al repositorio online.\n",
    "\n",
    "Haga click [aquí](http://nbviewer.jupyter.org/github/jdvelasq/deep-neural-nets/tree/master/) para explorar el repositorio usando `nbviewer`. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}