measures_of_dispersion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Measures of Dispersion"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Measures of dispersion are used to measure how \"spread out\" the data is. Here we will talk about `Range`, `Standard Deviation` and `Variance`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a video that will help you understand the concept better:\n",
    "    \n",
    "    https://www.youtube.com/watch?v=E4HAYd0QnRc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Range"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "It is the difference between two extreme observations of the data set. If `X max` and `X min` are the two extreme observations then `Range` = `X max` – `X min`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics as stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "99"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lst = [12, 23, 43, 19, 1, 33, 76 , 100]\n",
    "\n",
    "lst_range = max(lst) - min(lst)\n",
    "lst_range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variance "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Variance describes spread (from the mean) in the data. It tells us how scattered the data is, around the mean. \n",
    "\n",
    "This quantity is always positive \n",
    "\n",
    "Larger the magnitude, larger is the spread and vice-versa"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Here we will see how to calculate the variance manually:\n",
    "    \n",
    "Step 1: Compute the mean\n",
    "Step 2: Subtract the mean from each individual observation. This is called deviation \n",
    "Step 3: Take the square of these deviations. These are called squared deviations \n",
    "Step 4: Add all the squared deviations. This is called sum of squares \n",
    "Step 5: Divide the sum of squares by (n-1) where `n` is the number of observations. This is called variance \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using statistics library to compute variance "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "lst = [12, 23, 43, 19, 1, 33, 76 , 100, 12, 54, 56, 98, 32, 54, 87]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1027.952380952381"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import statistics as stats\n",
    "stats.variance(lst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standard Deviation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Here we will see how to calculate the variance manually:\n",
    "    \n",
    "Step 1: Compute the mean\n",
    "Step 2: Subtract the mean from each individual observation. This is called deviation \n",
    "Step 3: Take the square of these deviations. These are called squared deviations \n",
    "Step 4: Add all the squared deviations. This is called sum of squares \n",
    "Step 5: Divide the sum of squares by (n-1) where `n` is the number of observations. This is called variance \n",
    "Step 6: Take the square root of variance. This is called standard deviation \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using statistics library to compute standard deviation "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "33.738225459304424"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import statistics as stats\n",
    "stats.stdev(lst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Difference between standard deviation and variance "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Even though both variance and standard deviation are measures of spread/dispersion, there is a difference in the units of the two things. Unit of variance is squared of the unit of the original data while unit of standard deviation is same as the unit of the original data. Therefore for practical purposes, sometimes people prefer to use standard deviation instead of variance. \n",
    "\n",
    "Also since variance is square of standard deviation, if the value of standard deviation is large then the magnitude of variance becomes larger. Sometimes it is prefereable to work with numbers of lesser magnitudes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Population and Sample"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "When we talked about descriptive statistics, we only talked about the data as a whole ie we considered all the \n",
    "data that was given to us. This is called \"Population\" data. But we only considered dummy cases where we \n",
    "used some data in a list. When statisticians have to work with data, they have to deal with a large number of \n",
    "values in order to be able to make some reliable estimates. But in real world it is not possible to collect such a huge number of data points. Therefore we work with a smaller set or a subet of data. This subset is also called as a sample. This sample is assumed to come from a large population and is taken randomly, hence the sample drawn is also called random sample. \n",
    "\n",
    "Whenever we are working with a sample , the statistics described earlier are called sample statitics. \n",
    "For eg instead of `mean` we use the term `sample mean`, instead of `variance` we used the term `sample variance` and instead of `standard deviation` we use `sample standard deviation`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "We will talk about this in more detail during the bootcamp. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "1. What is the range for the data set\n",
    "    points = [-4,  17,  25,  19,  6,  11,  7]\n",
    "    \n",
    "2. You are given the following dataset:\n",
    "    \n",
    "    data = [2,4,7,1,6,8]\n",
    "    \n",
    "    How does the standard deviation change when 6 is replaced with 12 ? Does it increase, or decrease, or it remains the same?\n",
    "    \n",
    "\n",
    "3. Is is possile to have a dataset with 0 standard deviation or variance? If yes, can you think of any dataset with 6 data points that have 0 standard deviation?\n",
    "\n",
    "\n",
    "4. We know that standard deviation is a measure of spread in the dataset. What is meant by deviation here?\n",
    "\n",
    "\n",
    "5. Given the dataset in the list:\n",
    "    data = [23, 12, 34, 65, 34, 81]\n",
    "    What is the deviation from the mean for each of the points in the list. Write a for loop and print each of the values \n",
    "    \n",
    "    \n",
    "6. How is standard deviation different than variance ?   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "points = [-4,  17,  25,  19,  6,  11,  7]\n",
    "max(points)-min(points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.131182235954578"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = [2,4,7,1,12,8]\n",
    "import statistics as stats\n",
    "stats.stdev(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the standart deviation increase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = [1,1,1,1,1,1]\n",
    "stats.stdev(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# yes it is possible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The deviation is used to determine whether something deviates from a normal value, whether a value drifts slightly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deviation for item:  23 is:  -18.5\n",
      "Deviation for item:  12 is:  -29.5\n",
      "Deviation for item:  34 is:  -7.5\n",
      "Deviation for item:  65 is:  23.5\n",
      "Deviation for item:  34 is:  -7.5\n",
      "Deviation for item:  81 is:  39.5\n"
     ]
    }
   ],
   "source": [
    "data = [23, 12, 34, 65, 34, 81]\n",
    "for item in data:\n",
    "    print(\"Deviation for item: \", item, \"is: \", item - stats.mean(data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "#To calculate the standard deviation, the arithmetic mean and variance must first be determined. \n",
    "#The variance indicates the mean square deviation of a data set from the arithmetic mean.\n",
    "#The difference between variance and standard deviation is that the standard deviation measures the \n",
    "#average distance from the mean and the variance measures the squared average distance from the mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}