class_10_neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fcf70380",
   "metadata": {},
   "source": [
    "# Neural networks basics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "spread-happiness",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import matplotlib.ticker as ticker\n",
    "from IPython.display import Markdown, display, HTML\n",
    "\n",
    "# Fix the dying kernel problem (only a problem in some installations - you can remove it, if it works without it)\n",
    "import os\n",
    "os.environ['KMP_DUPLICATE_LIB_OK'] = 'True'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regular-niger",
   "metadata": {},
   "source": [
    "**Task 1.** Write a method for performing feed forward for a network with 2 input neurons, 3 hidden neurons with ReLU as the activation function, and 1 output neuron with no activation function. The interface of the method should be as follows:\n",
    "\n",
    "    feed_forward_1(x, w1, w2)\n",
    "    \n",
    "where x is a numpy array of size 1x2, w1 is a numpy array of size 3x2 of weights from the input layer to the hidden layer, w2 is a numpy array of size 1x3. The result should be a single number.\n",
    "\n",
    "As a helper function you can write the relu function:\n",
    "\n",
    "    relu(x)\n",
    "   \n",
    "which takes a numpy array as input and applies the ReLU function element-wise.\n",
    "\n",
    "Test it on 2-3 examples, calculate the result also with pen and paper by yourself (it's very important to do it at least once in your life!) and compare your results with the function results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "agreed-single",
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu(x):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n",
    "\n",
    "def feed_forward_1(x, w1, w2):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n",
    "\n",
    "# Test\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "another-catch",
   "metadata": {},
   "source": [
    "**Task 2.** Write a method for performing feed forward for a network with 2 input neurons, 3 hidden neurons with sigmoid activation function, and 1 output neuron with sigmoid activation function. The interface of the method should be as follows:\n",
    "\n",
    "    feed_forward_2(x, w1, w2)\n",
    "    \n",
    "where x is a numpy array of size 1x2, w1 is a numpy array of size 3x2 of weights from the input layer to the hidden layer, w2 is a numpy array of size 1x3. The result should be a single number.\n",
    "\n",
    "As a helper function you can write the sigmoid function:\n",
    "\n",
    "    sigmoid(x)\n",
    "   \n",
    "which takes a numpy array as input and applies the logistic function element-wise.\n",
    "\n",
    "Test it on 2-3 examples. Try to estimate the result by hand and compare both results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "forbidden-journalism",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n",
    "\n",
    "def feed_forward_2(x, w1, w2):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n",
    "\n",
    "# Test\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acute-amber",
   "metadata": {},
   "source": [
    "**Task 3.** Write a method for performing feed forward for a network with 2 input neurons, 3 hidden neurons with relu activation, and 3 output neurons with no activation function but softmax applied to them. The interface of the method should be as follows:\n",
    "\n",
    "    feed_forward_3(x, w1, w2)\n",
    "    \n",
    "where x is a numpy array of size 1x2, w1 is a numpy array of size 3x2 of weights from the input layer to the hidden layer, w2 is a numpy array of size 3x3. The result should be a numpy array of size 1x3.\n",
    "\n",
    "As a helper function you can write the softmax function:\n",
    "\n",
    "    softmax(x)\n",
    "   \n",
    "which takes a numpy array as input, applies softmax to it and returns a numpy array of the same size.\n",
    "\n",
    "Test it on 2-3 examples. Try to estimate the result by hand and compare both results. Verify that the sum of output neurons is equal to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "falling-holder",
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n",
    "\n",
    "def feed_forward_3(x, w1, w2):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n",
    "\n",
    "# Test\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94fe2269",
   "metadata": {},
   "source": [
    "**Task 4.** Write a method for calculating the squared error:\n",
    "\n",
    "    se(y_bar, y)\n",
    "    \n",
    "where y_bar is a numpy array of predicted results and y is a numpy array of real values to be predicted.\n",
    "\n",
    "Test it on the 2-3 examples you created for feed_forward_1 and find the SE with respect to real values chosen by hand. Experiment with several values to get a grip of how SE works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b7d7940",
   "metadata": {},
   "outputs": [],
   "source": [
    "def se(y_bar, y):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n",
    "    \n",
    "# Test\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0082d383",
   "metadata": {},
   "source": [
    "**Task 5.** Write a method for calculating the cross-entropy loss:\n",
    "\n",
    "    crossentropy(y_bar, y)\n",
    "    \n",
    "where y_bar is a numpy array of predicted results and y is a numpy array of real values to be predicted (y must contain one value of 1 and 0 on all other positions - just like in one-hot encoding).\n",
    "\n",
    "Test it on the 2-3 examples you created for feed_forward_3 and find the cross-entropy with respect to vectors of real values chosen by hand. Experiment with several target vectors to get a grip of how cross-entropy works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47526b2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def crossentropy(y_bar, y):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n",
    "    \n",
    "# Test\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3718602",
   "metadata": {},
   "source": [
    "**Task 6.** Write a method which calculates the error term for network 2:\n",
    "\n",
    "    error_term(x, w1, w2, y)\n",
    "    \n",
    "where x is a numpy array of size 1x2, w1 is a numpy array of size 3x2 of weights from the input layer to the hidden layer, w2 is a numpy array of size 1x3, y is a float.\n",
    "\n",
    "The solution is extremely simple - it's just the difference between y and the result of feed_forward_2. This task is solely to bring this notion to your attention.\n",
    "\n",
    "Test it on the 2-3 examples you created for feed_forward_2 and several values of y (remember that the values of y should logically be in the interval [0-1], because you used the sigmoid activation which sends the network's results into this interval, technically turning them into probabilities)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47d98db2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def error_term(x, w1, w2, y):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n",
    "\n",
    "# Test\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3896f79",
   "metadata": {},
   "source": [
    "**Task 7.** Write a method which calculates the derivative of the sigmoid function on every element of a numpy array:\n",
    "\n",
    "    sigmoid_derivative(x)\n",
    "    \n",
    "where x is a numpy array. The result should have the same size as the input.\n",
    "\n",
    "Use the formula:\n",
    "\n",
    "<center>\n",
    "$$\n",
    "    \\sigma'(x) = \\sigma(x) (1 - \\sigma(x))\n",
    "$$\n",
    "</center>\n",
    "\n",
    "Verify this formula by differentiating the sigmoid function by hand.\n",
    "\n",
    "Test the sigmoid_derivative method on a numpy array with several values (e.g. -3, -2, -1, 0, 1, 2, 3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98020df9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid_derivative(x):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n",
    "\n",
    "# Test\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f301009",
   "metadata": {},
   "source": [
    "**Task 8.** Write a method for performing the backpropagation step for network 2. The method should have the following interface:\n",
    "\n",
    "    backpropagate(x, w1, w2, y, alpha)\n",
    "    \n",
    "where x is a numpy array of size 1x2, w1 is a numpy array of size 3x2 of weights from the input layer to the hidden layer, w2 is a numpy array of size 1x3, y is a float, alpha is the learning rate. The method should return a tuple with updated matrices w1 and w2.\n",
    "\n",
    "Test it on 2-3 examples. Test several values of the learning rate alpha on the same input to see how it affects the update step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8838f3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def backpropagate(x, w1, w2, y, alpha):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n",
    "    return w1, w2\n",
    "\n",
    "# Test\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce4f63a3",
   "metadata": {},
   "source": [
    "**Task 9.** Write a method for performing the backpropagation step for network 2 but without the activation function on the output neuron."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec9a8c86",
   "metadata": {},
   "outputs": [],
   "source": [
    "def backpropagate_2(x, w1, w2, y, alpha):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n",
    "    return w1, w2\n",
    "\n",
    "# Test\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}