Merge pull request #1021 from Shantnu-singh/Shantnu-singh-learning--r…

…ules

Added learning rules in ANN

Learning Rules In ANN/Learning-Rules.md

@@ -0,0 +1,106 @@
+# Learning Rules in Artificial Neural Networks (ANN)
+
+## Introduction
+
+Learning rules are essential components of Artificial Neural Networks (ANNs) that govern how the network updates its weights and biases. This document focuses on two fundamental learning rules: Hebbian Learning and Adaline (Adaptive Linear Neuron) Learning.
+
+## 1. Hebbian Learning
+
+Hebbian Learning, proposed by Donald Hebb in 1949, is one of the earliest and simplest learning rules in neural networks. It is based on the principle that neurons that fire together, wire together.
+
+### Basic Principle
+
+The strength of a connection between two neurons increases if both neurons are activated simultaneously.
+
+### Mathematical Formulation
+
+For neurons $i$ and $j$ with activation values $x_i$ and $x_j$, the weight update $\Delta w_{ij}$ is given by:
+
+$$\Delta w_{ij} = \eta x_i x_j$$
+
+Where:
+- $\Delta w_{ij}$ is the change in weight between neurons $i$ and $j$
+- $\eta$ is the learning rate
+- $x_i$ is the output of the presynaptic neuron
+- $x_j$ is the output of the postsynaptic neuron
+
+### Variations
+
+1. **Oja's Rule**: A modification of Hebbian learning that includes weight normalization:
+
+   $$\Delta w_{ij} = \eta(x_i x_j - \alpha y_j^2 w_{ij})$$
+
+   Where $y_j$ is the output of neuron $j$ and $\alpha$ is a forgetting factor.
+
+2. **Generalized Hebbian Algorithm (GHA)**: Extends Oja's rule to multiple outputs:
+
+   $$\Delta W = \eta(xy^T - \text{lower}(Wy^Ty))$$
+
+   Where $\text{lower}()$ denotes the lower triangular part of a matrix.
+
+## 2. Adaline Learning (Widrow-Hoff Learning Rule)
+
+Adaline (Adaptive Linear Neuron) Learning, developed by Bernard Widrow and Marcian Hoff in 1960, is a single-layer neural network that uses linear activation functions.
+
+### Basic Principle
+
+Adaline learning aims to minimize the mean squared error between the desired output and the actual output of the neuron.
+
+### Mathematical Formulation
+
+For an input vector $\mathbf{x}$ and desired output $d$, the weight update is given by:
+
+$$ \Delta \mathbf{w} = \eta(d - y)\mathbf{x} $$
+
+Where:
+- $\Delta \mathbf{w}$ is the change in weight vector
+- $\eta$ is the learning rate
+- $d$ is the desired output
+- $y = \mathbf{w}^T\mathbf{x}$ is the actual output
+- $\mathbf{x}$ is the input vector
+
+### Learning Process
+
+1. Initialize weights randomly
+2. For each training example:
+3. 
+   a. Calculate the output:
+
+    $y = \mathbf{w}^T\mathbf{x}$
+
+   b. Update weights:
+
+  $$w_{new} = w_{old} + \eta(d - y)x$$
+
+5. Repeat step 2 until convergence or a maximum number of epochs is reached
+
+### Comparison with Perceptron Learning
+
+While similar to the perceptron learning rule, Adaline uses the actual output value for weight updates, not just the sign of the output. This allows for more precise weight adjustments.
+
+## Conclusion
+
+Both Hebbian and Adaline learning rules play crucial roles in the development of neural network theory:
+
+- Hebbian Learning provides a biological inspiration for neural learning and is fundamental in unsupervised learning scenarios.
+- Adaline Learning introduces the concept of minimizing error, which is a cornerstone of many modern learning algorithms, including backpropagation in deep neural networks.
+
+Understanding these basic learning rules provides insight into more complex learning algorithms used in deep learning and helps in appreciating the historical development of neural network theory.
+
+
+## How to Use This Repository
+
+- Clone this repository to your local machine.
+
+```bash
+  git clone https://github.com/Niketkumardheeryan/ML-CaPsule/Learning Rules In ANN
+```
+- For Python implementations and visualizations:
+
+1. Ensure you have Jupyter Notebook installed 
+
+```bash
+  pip install jupyter
+```
+2. Navigate to the project directory in your terminal.
+3. Open learning_rules.ipynb.

Learning Rules In ANN/learning_rules.ipynb

@@ -0,0 +1,117 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<h3>Implementation of different activation fucntion in learning rules</h3>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 49,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Trained Weights for hebb network are :  [1.5488135  1.71518937]\n",
+      "Trained bias for hebb network are :  [1.60276338]\n",
+      "Trained Weights for hebb network are :  [0.53143536 0.57216559]\n",
+      "Trained bias for hebb network are :  [0.51141955]\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "\n",
+    "# Problem --> OR- Gate\n",
+    "input = np.array([[1, 1] ,[1,-1] , [-1,1] ,[-1,-1] ])\n",
+    "output = np.array([1 ,1 , 1 ,-1 ])\n",
+    "np.random.seed(0)\n",
+    "weight = np.random.rand(2)\n",
+    "bias = np.random.rand(1)\n",
+    "learning_rate = 0.1\n",
+    "\n",
+    "# Hebbian Learning rule\n",
+    "def hebbNetwork(input , output , b , w ,epoches, learning_rate = .1):\n",
+    "    for epoch in range(epoches):\n",
+    "        for i in range(len(input)):\n",
+    "            cal_output = (np.dot(input[i] , w) + b)\n",
+    "            # print(w)\n",
+    "            if cal_output != output[i]:\n",
+    "                w = w + (learning_rate*output[i]*input[i])\n",
+    "                b = b + (learning_rate* output[i])\n",
+    "    return w , b\n",
+    "\n",
+    "# Aderline Learning rule\n",
+    "def AderlineNetwork(input , output , b , w ,epoches, learning_rate = .1):\n",
+    "        for epoch in range(epoches):\n",
+    "            for i in range(len(input)):\n",
+    "                cal_output = (np.dot(input[i] , w) + b)\n",
+    "                error = (output[i] - cal_output)**2\n",
+    "                # print(error)\n",
+    "                if cal_output != output[i]:\n",
+    "                    w = w + (learning_rate*(output[i]- cal_output)*input[i])\n",
+    "                    b = b + (learning_rate*(output[i]- cal_output))\n",
+    "        return w , b\n",
+    "\n",
+    "\n",
+    "wh , bh = hebbNetwork(input , output , bias , weight , 5)\n",
+    "\n",
+    "print(\"Trained Weights for hebb network are : \" , wh)\n",
+    "print(\"Trained bias for hebb network are : \" , bh)\n",
+    "\n",
+    "wa , ba = AderlineNetwork(input , output , bias , weight , 5)\n",
+    "\n",
+    "print(\"Trained Weights for hebb network are : \" , wa)\n",
+    "print(\"Trained bias for hebb network are : \" , ba)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 50,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[4.86676625]\n",
+      "[1.43638751]\n",
+      "[1.76913924]\n",
+      "[-1.66123949]\n"
+     ]
+    }
+   ],
+   "source": [
+    "for i in range(len(input)):\n",
+    "    cal_output = (np.dot(input[i] , wh) + bh)\n",
+    "    print(cal_output)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}