DS101-Basic-Statistics/DS101-L4-Common-Descriptive-Statistics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "61bef591-bf77-43a3-8d42-714b58415dcd",
   "metadata": {},
   "source": [
    "# Basic Statistic : Lesson Four Common Descriptive Statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "84971424-6257-4af9-86d6-b722d3794994",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/231927377\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x2a282ea5670>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Science Software\n",
    "VimeoVideo('231927377', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08691ebb-12ee-4455-806a-595dc95f9af5",
   "metadata": {},
   "source": [
    "### For more examples, and concepts presented in a different light: check out this walkthrough\n",
    "[DS102 Basic Stats Lesson 4 Examples](Examples/Nolan/BasicStatsL4Guide.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "104ea1b7-84d1-40ce-8e8b-cacdddb53c23",
   "metadata": {},
   "source": [
    "### Table of Contents <a class=\"anchor\" id=\"DS101L4_toc\"></a>\n",
    "* [Table of Contents](#DS101L4_toc)\n",
    "    * [Page 1 - Overview and Objectives](#DS101L4_page_1)\n",
    "    * [Page 2 - Population Parameters and Sample Statistics](#DS101L4_page_2)\n",
    "    * [Page 3 - Probability Rules 1 & 2](#DS101L4_page_3)\n",
    "    * [Page 4 - Measures of Central Tendency](#DS101L4_page_4)\n",
    "    * [Page 5 - Measures of Central Tendency Activity Solution](#DS101L4_page_5)\n",
    "    * [Page 6 - Measures of Distribution ](#DS101L4_page_6)\n",
    "    * [Page 7 - Measures of Distribution Activity](#DS101L4_page_7)\n",
    "    * [Page 8 - Measures of Distribution Activity Solution](#DS101L4_page_8)\n",
    "    * [Page 9 - Confidence Interval](#DS101L4_page_9)\n",
    "    * [Page 10 - Measures of Frequency](#DS101L4_page_10)\n",
    "    * [Page 11 - Key Terms](#DS101L4_page_11)\n",
    "    * [Page 12 - Lesson 4 Practice Hands On](#DS101L4_page_12)\n",
    "    * [Page 13 - Lesson 4 Practice Hands On Solution](#DS101L4_page_13)\n",
    "    * [Page 14 - Common Descriptive Statistics - Skills Assessment](#DS101L4_page_14)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb4e65fe-34ee-44c5-8bda-7a1f07c507d8",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 1- Overview and Objectives<a class=\"anchor\" id=\"DS101L4_page_1\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "419bc8a6-b7be-440c-a36d-ff3c78684f44",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "This lesson is an introduction to descriptive statistics that are used frequently. \n",
    "\n",
    "Descriptive statistics are derived from the opportunity to sample a population over and over again.\n",
    "\n",
    "When this lesson is completed successfully, you will be able to calculate descriptive statistics.\n",
    "\n",
    "A quiz will assess concept mastery at the end of this lesson."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87a83a87-138e-40b7-818a-8e7d7bb849a4",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 2 - Population Parameters and Sample Statistics<a class=\"anchor\" id=\"DS101L4_page_2\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e3c07b97-eeda-4b79-ba04-c552d71e8be7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/315975449\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x2a282ebbf10>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Science Software\n",
    "VimeoVideo('315975449', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d0f8b27-058b-4fa5-960e-13e8b9595c33",
   "metadata": {},
   "source": [
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO101-L04-pg3tutorial.zip)**.\n",
    "\n",
    "# Population Parameters and Sample Statistics\n",
    "\n",
    "Did you notice the alliteration there? When you are describing a sample, you use the term sample statistics. When you are describing a population, use the term population parameters.  However, the ways to describe a population or a sample are usually the same - for instance, the mean.  You calculate the mean the same, whether it is a population parameter or a sample statistic.  Just the name changes.\n",
    "\n",
    "Statistics is the branch of mathematics that uses samples to estimate population parameters.  \n",
    "\n",
    "One of the secrets of data that usually doesn't get talked about is this: It is pretty much impossible to know the true value of population parameters. Does that sound reasonable? If you think about it, how could you possibly know the true mean weight of all 15-year-olds? First of all, the value is constantly changing. People lose and gain weight every day, and people turn from 14 to 15 and from 15 to 16 every moment of every day. Even if you could freeze everyone's age and body weight for a certain amount of time to take all the measurements, it would take too much time. Nobody would want to do it. You can learn a lot about a population by looking at a small, manageable subset of the population.\n",
    "\n",
    "---\n",
    "\n",
    "## Descriptive Statistics vs. Inferential Statistics\n",
    "\n",
    "* **Descriptive Statistics** are used when you just want to describe your data.  \n",
    "\n",
    "* **Inferential Statistics** are used when you are trying to make conclusions (i.e. inferences) about your data.\n",
    "\n",
    "---\n",
    "\n",
    "## Descriptive Statistics\n",
    "\n",
    "In the upcoming pages, you will learn about some basic descriptive statistics, including:\n",
    "\n",
    "* Measures of Central Tendency\n",
    "    * Mean\n",
    "    * Median\n",
    "    * Mode\n",
    "\n",
    "* Measures of Distribution\n",
    "    * Range\n",
    "    * Variance\n",
    "    * Standard Deviation\n",
    "    * Inter-Quartile Range\n",
    "\n",
    "* Measures of Frequency\n",
    "    * Count\n",
    "    * Percents\n",
    "    * Proportions\n",
    "\n",
    "---\n",
    "\n",
    "## Workshop on Spreadsheet Software\n",
    "\n",
    "The remainder of this lesson assumes a basic knowledge of either MS Excel or Google Sheets.  If you've never used a spreadsheet program before or would like a refresher, you can check out the video below, which covers: \n",
    "\n",
    "* How to open Excel and/or Google Sheets\n",
    "* Scrolling through data\n",
    "* Columns versus rows\n",
    "* Typing in and completing formulas\n",
    "* Highlighting data\n",
    "* Creating equations\n",
    "* How to download Google Sheets files\n",
    "* How to save Excel files \n",
    "\n",
    "**[Basic Excel and Google Sheets Workshop](https://vimeo.com/353693628)**\n",
    "\n",
    "---\n",
    "\n",
    "## Review\n",
    "\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded.\n",
    "\n",
    "```c-lms\n",
    "start-activity: Descriptive Statistics Quiz \n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0766bba0-b74a-4eb7-9439-330dff67ce3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from DS_Students import *\n",
    "    from ipynb.fs.full.DS101Questions import *\n",
    "except:\n",
    "    !pip install DS_Students\n",
    "    from DS_Students import *\n",
    "    from ipynb.fs.full.DS101Questions import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "af3b63bb-2f8a-4d7f-90df-3856c2a791da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0624a33cad104213ad87e4f0b0e32337",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Population', 0), ('Sample…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0635f66d227a41c693d6db5ce76b4697",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Population', 0), ('Sample…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3aac532d08d34bf1a45256957a4060cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('When drawing conclusions …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bd2ed1087bc54a2fbcd8b6850610a3f7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('When drawing conclusions …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L4P2Q1, L4P2Q2, L4P2Q3, L4P2Q4)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a671dce6-0c0f-488e-bafb-fb26ac82d7e8",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 3 - Measures of Central Tendency<a class=\"anchor\" id=\"DS101L4_page_3\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8e7d1700-27e2-4ccf-9367-2884ca865779",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/315975353\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x2a282e97ac0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Science Software\n",
    "VimeoVideo('315975353', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b6d7f71-c214-4b64-a1c5-8c7a87b104c1",
   "metadata": {},
   "source": [
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO101-L04-pg4tutorial.zip)**.\n",
    "\n",
    "# Measures of Central Tendency\n",
    "\n",
    "Measures of Central Tendency are meant to help you determine where the majority of your data points lie.  They show the estimated middle of your data.  There are three measures of central tendency: Mean, Median, and Mode.  However, most statistics will be focused on testing the mean.  The median and mode are unsung heroes.\n",
    "\n",
    "---\n",
    "\n",
    "## Mean\n",
    "\n",
    "Mean is also referred to as the average. To calculate the mean of a sample, add up all the values and divide the total by the size of the sample.  For example, to calculate the mean for a list of these numbers:\n",
    "\n",
    "```text\n",
    "5\n",
    "7\n",
    "9\n",
    "4\n",
    "3\n",
    "```\n",
    "\n",
    "First, add them all together:\n",
    "\n",
    "```text\n",
    "5 + 7 + 9 + 4 +3 = 28\n",
    "```\n",
    "\n",
    "Then divide the sum by the number of numbers: \n",
    "\n",
    "```text\n",
    "28 / 5 = 5.6\n",
    "```\n",
    "\n",
    "This yields a mean of **5.6**.  \n",
    "\n",
    "You will notice that doing this manually is relatively easy when you have a small number of numbers, but if you had a long list, this would get tedious quickly.  Luckily, there are many programs that quickly and easily calculate the mean for you if you just provide it with the data.\n",
    "\n",
    "Try an example with **[this dataset](https://repo.exeterlms.com/documents/V2/DataScience/Basic-Statistics/Ages.zip)**.  In MS Excel, you can use the ```=average()``` formula to get the mean of the data just by highlighting it.\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Don't have MS Excel? Check out  <a href='https://www.libreoffice.org/download/download/'>Libre Office</a>, which is free but nearly identical to MS Excel and will serve you better than other options like Google Sheets. </p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Calculating Means for Different Variable Types\n",
    "\n",
    "The method of calculating the mean for continuous quantitative variables is discussed above. Other types of variables will be discussed below.\n",
    "\n",
    "---\n",
    "\n",
    "### Categorical Variables\n",
    "\n",
    "The mean for a categorical variable doesn't exist. If your categorical variable is \"preferred type of music\" and your only choices are Country, Jazz, Classic Rock, and Hip-Hop, how would you express the average of that? You might determine which musical style is the most popular among the group of people you surveyed, but that isn't the same as the average.\n",
    "\n",
    "Even if your categorical variable looks like a number, and you can calculate the average value, that doesn't mean you should; it probably won't make any sense.\n",
    "\n",
    "---\n",
    "\n",
    "### Discrete Quantitative Variables\n",
    "\n",
    "To calculate the mean for a discrete quantitative variable, you need to take into account the probability that each response will occur. In the case of a six-sided die, you assume that each side will occur with a probability of 1/6. The general approach to calculating the mean is to take each possible response, and multiply it by the probability that response will occur, and then add them all up.\n",
    "\n",
    "For a single die, you have the following table:\n",
    "\n",
    "![A chart with headings roll, probability, and roll times probability. Row one, one, one sixth, one sixth. Row two, two, one sixth, two sixths. Row three, three, one sixth, three sixths. Row four, four, one sixth, four sixths. Row five, five, one sixth, five sixths. Row six, six, one sixth, six sixths. Total, twenty one sixths equals three point five.](Media/L03-02.png)\n",
    "\n",
    "As you can see, the total of each possible outcome multiplied by the probability of the outcome is 21/6, which equals 3.5. It is interesting to note that the \"average\" roll for a six-sided die is actually not a possible outcome at all. This is very common with means for discrete random variables.\n",
    "\n",
    "The example above is theoretical because the mean is based on theoretical probabilities. Suppose you calculated the mean again, but this time you will use real data instead of a theoretical probability for each roll.\n",
    "\n",
    "If you rolled a single die 6,000 times, it is reasonable to expect that each number will appear about 1,000 times, but also extremely unlikely that each number will appear exactly 1,000 times. Look at some simulated data:\n",
    "\n",
    "![A chart with headings roll, occurrences, probability, rolls times probability. Row one, one, one thousand sixteen, zero point one six nine three, zero point one six nine three. Row two, two, one thousand twelve, zero point one six eight seven, zero point three three seven three. Row three, three, nine hundred fifty eight, zero point one five nine seven, zero point four seven nine zero. Row four, four, nine hundred ninety nine, zero point one six six five, zero point six six six zero. Row five, five, one thousand two, zero point one six seven zero, zero point eight three five zero. Row six, one thousand thirteen, zero point one six eight eight, one point zero one three zero. Total, three point four nine nine seven.](Media/L03-03.png)\n",
    "\n",
    "The probability column in this table is calculated by taking the number of occurrences of each roll, and dividing it by the total rolls, or 6,000. As you can see, the experimental mean is very close to the theoretical mean: 3.5 vs. 3.4997, which gives you confidence that the theoretical mean is correctly calculated.\n",
    "\n",
    "---\n",
    "\n",
    "## Median\n",
    "\n",
    "The median indicates the exact middle value. To calculate the median, you sort all of the values from smallest to largest and then choose the middle value. If there are an even number of values, pick the two in the middle, calculate the \"middle\" of those two by doing a quick mean of those two values, and that will be your median.\n",
    "\n",
    "Time to take a few numbers to practice with:\n",
    "\n",
    "```text\n",
    "5\n",
    "7\n",
    "9\n",
    "4\n",
    "3\n",
    "```\n",
    "\n",
    "If you were doing this by hand, the first thing you would need to do is sort them all from smallest to largest:\n",
    "\n",
    "```text\n",
    "3\n",
    "4\n",
    "5\n",
    "7\n",
    "9\n",
    "```\n",
    "\n",
    "Then, you'll want to find the middle.  Count the number of numbers you have, divide by two, and then count the whole number down.\n",
    "\n",
    "```text\n",
    "5 / 2 = 2.5\n",
    "```\n",
    "\n",
    "So count two down:\n",
    "\n",
    "```text\n",
    "3\n",
    "4\n",
    "```\n",
    "\n",
    "And the next number is your median! Which is`5`!\n",
    "\n",
    "You can easily do this in MS Excel by using the function `=median()`.  \n",
    "\n",
    "<div class=\"panel panel-warning\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Try It!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Try it with the Ages dataset you downloaded above.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Mode\n",
    "\n",
    "The mode is the most frequently observed value in your sample. The mode is supposed to indicate relative popularity of a particular value, but it is easily manipulated and doesn't have much analytical value.\n",
    "\n",
    "If you have this set of numbers:\n",
    "\n",
    "```text\n",
    "7\n",
    "7\n",
    "10\n",
    "3\n",
    "5\n",
    "```\n",
    "\n",
    "The mode would be 7, because it occurs twice and all the other numbers only occur once.\n",
    "\n",
    "The MS Excel function for finding the mode is `=mode()`.  Feel free to try it with your ages dataset as well.  \n",
    "\n",
    "---\n",
    "\n",
    "## Review\n",
    "\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded.\n",
    "\n",
    "```c-lms\n",
    "start-activity: Mean Quiz\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c96c054f-53bf-4914-8fb7-59aeceed4bd0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6087b71914354c118159af5d41d9312f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Add up all the responses …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1c9b24a87c07497494123eb595e3d1bf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('3', 0), ('18', 1), ('6.1'…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L4P3Q1, L4P3Q2)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e577b59-5f8c-4f96-9eec-dd9069f5d4ed",
   "metadata": {},
   "source": [
    "A carpenter wants to make sure he has all his boards cut correctly. He measures the same board three different times and then takes the average. Given the following measurements, what is the average board length?\n",
    "\n",
    "**6**\n",
    "\n",
    "**6.1**\n",
    "\n",
    "**5.9**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5ac9bbad-03d6-43cc-aff9-ff4d63588298",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "028d8cf447ef4afaa0ff9fc4e50bab85",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('The height of fashion', 0…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a45fbed385aa456795741114230f2c88",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('The weighted addition of …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L4P3Q3, L4P3Q4)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44c0c2ba-dbd9-46f3-b195-843295207d01",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 4 - Measures of Central Tendency Activity<a class=\"anchor\" id=\"DS101L4_page_4\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "### Directions\n",
    "\n",
    "# Measures of Central Tendency Activity\n",
    "\n",
    "In MS Excel or a similar program such as Libre Office or Google Sheets, please use **[this data](https://repo.exeterlms.com/documents/V2/DataScience/Basic-Statistics/Heights.zip)** to explore measures of central tendency.\n",
    "\n",
    "Please determine the following:\n",
    "\n",
    "- Mean\n",
    "\n",
    "- Median\n",
    "\n",
    "- Mode\n",
    "\n",
    "And then attach your spreadsheet with your answers.\n",
    "\n",
    "This activity will not count towards your grade, but the best way to become a data scientist is to practice!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee3b706a-455f-4284-9898-684fe920b503",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 5 - Measures of Central Tendency Solution<a class=\"anchor\" id=\"DS101L4_page_5\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "    \n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "Here are the answers: \n",
    "\n",
    "* Mean: 69.2\n",
    "\n",
    "* Median: 70\n",
    "\n",
    "* Mode: 65\n",
    "\n",
    "The way this was done in MS Excel can be found **[here](https://repo.exeterlms.com/documents/V2/DataScience/Basic-Statistics/Heights_Answer.zip).**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0363336-1025-456e-bba6-59b6eec66f67",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 6 - Measures of Distribution<a class=\"anchor\" id=\"DS101L4_page_6\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "493c8f54-faae-48f5-8127-6f65ce5f47d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/315975378\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x2a284e9c550>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Science Software\n",
    "VimeoVideo('315975378', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44a0a0ad-f15f-4b3f-a27f-f3cffcb8e7bd",
   "metadata": {},
   "source": [
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO101-L04-pg7tutorial.zip)**.\n",
    "\n",
    "# Measures of Distribution\n",
    "\n",
    "## Minimum\n",
    "\n",
    "Often referred to as the `min`, it is simply the smallest value in the sample.\n",
    "\n",
    "In the data below, the minimum is 3.\n",
    "\n",
    "```text\n",
    "3\n",
    "4\n",
    "5\n",
    "7\n",
    "9\n",
    "```\n",
    "\n",
    "In MS Excel, you can find the minimum value by simply using the formula `=min()`.\n",
    "\n",
    "---\n",
    "\n",
    "## Maximum\n",
    "\n",
    "Often referred to as the `max`, it is simply the largest value in the sample.\n",
    "\n",
    "In the data shown above, the maximum is 9.\n",
    "\n",
    "In MS Excel, you can find the maximum value by using the formula `=max()`.\n",
    "\n",
    "---\n",
    "\n",
    "## Range\n",
    "\n",
    "If you take the maximum and subtract the minimum, you will have the range.\n",
    "\n",
    "Given this data:\n",
    "\n",
    "```text\n",
    "3\n",
    "4\n",
    "5\n",
    "7\n",
    "9\n",
    "```\n",
    "\n",
    "The range is:\n",
    "\n",
    "```text\n",
    "9 - 3 = 6\n",
    "```\n",
    "\n",
    "You use range a lot in your every day life. For instance, how long is your commute? Are you thinking something like \"it is 25 to 40 minutes, depending on traffic\"? You have just identified the min and max for your commute, so your range is 15 minutes (max - min).\n",
    "\n",
    "A range is expressed as a single value, max - min. Even though you might say your commute \"ranges\" from 25 to 40 minutes, proper use of the statistical term \"range\" is that your range is 15 minutes.\n",
    "\n",
    "You can do this in MS Excel by first calculating the min, then calculating the max, and finally subtracting min from max.  \n",
    "\n",
    "---\n",
    "\n",
    "## Quartiles\n",
    "\n",
    "\"Quart\" means *four*, and thus *quartiles* are when you break your data into four equal parts.  To get quartiles in MS Excel, you will use the formula =quartiles() and then after highlighting your data, you will also need to provide the number for the quartile you want.  If you use a 1, the first quartile will be provided. A 2 will provide the second quartile, and a 3 will provide the third quartile.  \n",
    "\n",
    "It may help to visualize quartiles - the way this is typically done is with a box plot (also known as a box-and-whiskers plot).  In the graph below, the first line on the left, labeled as \"lower whisker\" marks the boundary of the first quartile, from the left side of the box to the middle line is the second quartile, from the middle line to the right side of the box is the third quartile, and the right side of the box to the rightmost line, labeled \"upper whisker\" is the fourth quartile.\n",
    "\n",
    "![A box and whisker plot showing the outliers, lower whisker, lower quartile, median, upper quartile, and upper whisker. The scale runs from negative fifteen to five, in increments of five. Below the scale, text reads, Fun fact, the breakdown into quartiles is a little arbitrary. People have been known to break their data into quintiles open parentheses five groups close parentheses and even percentiles open parentheses one hundred groups close parentheses.](Media/DescriptiveStats3.png)\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Fun Fact!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>The breakdown into quartiles is a little arbitrary.  People have been known to break their data into quintiles (five groups) and even percentiles (100 groups. </p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Inter-Quartile Range\n",
    "\n",
    "Inter-quartile range is almost always abbreviated as `IQR`. It is very similar to the range, but instead of max - min, it is 3rd quartile - 1st quartile. Referring back to the graph above, this would be just the area inside the box.  IQR is a useful sample statistic because it tends to ignore the effect of outliers (very large or very small data that don't mesh well with the rest) in your sample.\n",
    "\n",
    "The range, as calculated above, emphasizes the outliers, because it takes the extreme maximum value and subtracts the extreme minimum value. If there are huge outliers, they have heavy influence and cause the range to be somewhat misleading as to the spread of the sample.\n",
    "\n",
    "The IQR only looks at the \"middle half\" of the sample. If you order your sample from smallest to largest, the smallest quarter of values are below the 1st quartile, and the largest quarter of values are above the 3rd quartile.\n",
    "\n",
    "Therefore, half the values in your sample don't influence the IQR, and they are the most extreme half of the values on both the low side and the high side.\n",
    "\n",
    "As with the normal range, MS Excel won't do all the work for you, so you'll have to subtract the first quartile from the third in Excel once you've gotten them via formula.\n",
    "\n",
    "---\n",
    "\n",
    "## Percentiles\n",
    "\n",
    "A percentile is a way of expressing where a value fits within a large sample. Have you ever heard someone say that their baby is in the 25th percentile for weight? Have you ever taken a standardized test, and they tell you that your score is 840 (which is meaningless), and your percentile is 82nd? What does 82nd percentile even mean?\n",
    "\n",
    "It means that of all the people that have taken the standardized test, your score is better than about 82% of the people that took it, and about 18% of those who took it scored better than you. To calculate a percentile, order the sample values from smallest to largest.\n",
    "\n",
    "Then, if you want the 82nd percentile (for instance), find the point at which 82% of your sample values are as small or smaller, and 18% are as large or larger. Percentiles work best if your sample size is larger than 100.\n",
    "\n",
    "Some specialized percentiles have unique names. You have already learned about one: Median.  Median is the same thing as the 50th percentile.\n",
    "\n",
    "---\n",
    "\n",
    "## Standard Deviation\n",
    "\n",
    "Standard deviation may be abbreviated as `stdev`, `SD`, or even just `s`. Standard deviation is a powerful statistic, but it is not very intuitive. In layman's terms, standard deviation measures the spread of values in a sample.  The bigger the standard deviation is, the more spread out the values in the sample are. Here is the equation for standard deviation, where you are finding the difference between a value (*x*) and the mean (*<span>x&#773;</span>*) , squaring it, and then summing it (the funny capital E shape means to sum).  Then you'll divide by the total sample size (*n*) minus 1, and take the square root of the whole kit and kaboodle!\n",
    "\n",
    "![The equation to find the standard deviation.](Media/L04-01.png)\n",
    "\n",
    "The good news is that you don't need to understand the equation, just familiarize yourself with the concept. Rest assured that practically every spreadsheet or programming language can calculate the standard deviation. In fact, try calculating the standard deviation with your Ages dataset right now! Use `=stdev.s()`.\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>The standard deviation will <b>NEVER</b> be negative. If you have calculated a standard deviation and it is negative, you made a mistake somewhere.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Variance\n",
    "\n",
    "Sometimes you will see this abbreviated as `var`. This statistic, just like the standard deviation, is powerful but not intuitive. The best way to think of variance is similar to how you think about standard deviation: the bigger the variance, the more spread out the sample data are.\n",
    "\n",
    "To calculate the variance by hand, you take the standard deviation, and square it.\n",
    "\n",
    "![The equation to find the variance of a sample](Media/DescriptiveStats4.png)\n",
    "\n",
    "You'll notice that this formula is incredibly similar to the last, but it's missing a square root because the variance is the square of the standard deviation.  You'll also notice that variance has been denoted as `S squared` - again, the square of the standard deviation!\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>The word \"variance\" has a specific statistical meaning. \"Variation\" or \"variance\" are words often used just to indicate change, but this doesn't work for statistics! </p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "<div class=\"panel panel-warning\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Try It!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>To calculate variance in MS Excel, use the formula <code>=var.s()</code>. Go ahead and try it with the Ages dataset!</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Review\n",
    "\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded.\n",
    "\n",
    "```c-lms\n",
    "start-activity: Measures of Distribution Quiz\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b21affc7-88b1-4614-9e53-d45ae33c84b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d4f0b1f174c54062b51e864cf40ecbc2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('By subtracting the minimu…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3627f51b5d984dc59b156c32da818bf6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('It looks at the average c…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "97ee62245cc24ed3bb862573b97e27d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Standard deviation is the…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "883b101499c84027be7f33f3b430992f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('It looks at the average c…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "78c5b73234ba4167af3ea2c8a851b4e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Standard deviation is the…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L4P6Q1, L4P6Q2, L4P6Q3, L4P6Q4, L4P6Q5)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e89129b-21bf-4782-aa87-6bb1a30a61eb",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 7 - Measures of Distribution Activity<a class=\"anchor\" id=\"DS101L4_page_7\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "### Directions\n",
    "\n",
    "# Measures of Distribution Activity\n",
    "\n",
    "In MS-Excel or a similar program such as Libre Office or Google Sheets, you will use the **[heights dataset](https://repo.exeterlms.com/documents/V2/DataScience/Basic-Statistics/Heights.zip)** from the last activity to explore measures of distribution.\n",
    "\n",
    "Please determine the following:\n",
    "\n",
    "- Min\n",
    "\n",
    "- Max\n",
    "\n",
    "- Range\n",
    "\n",
    "- Quartiles\n",
    "\n",
    "- Inter-quartile range\n",
    "\n",
    "- Standard Deviation\n",
    "\n",
    "- Variance\n",
    "\n",
    "And then attach your spreadsheet with your answers.\n",
    "\n",
    "This activity will not count towards your grade, but the best way to become a data scientist is to practice!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36242d9e-da08-41ed-b247-1b6235a7a115",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 8 - Measures of Distribution Activity Solution<a class=\"anchor\" id=\"DS101L4_page_8\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Measures of Distribution Activity Solution\n",
    "\n",
    "Here are the answers: \n",
    "\n",
    "* Minimum: 60\n",
    "\n",
    "* Maximum: 78\n",
    "\n",
    "* Range: 18\n",
    "\n",
    "* First Quartile: 65\n",
    "\n",
    "* Third Quartile: 74\n",
    "\n",
    "* Inter-Quartile Range: 9\n",
    "\n",
    "* Standard Deviation: 5.25\n",
    "\n",
    "* Variance: 27.61\n",
    "\n",
    "The way this was done in MS Excel can be found **[here](https://repo.exeterlms.com/documents/V2/DataScience/Basic-Statistics/Heights_Answer2.zip)**.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c295bd9-348b-4195-bb0f-cb1696f8f6ee",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 9 - Confidence Intervals<a class=\"anchor\" id=\"DS101L4_page_9\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "    \n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "47bcf936-842c-4514-a3cd-dbd5bb16eb27",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/315975318\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x2a284ead7c0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Science Software\n",
    "VimeoVideo('315975318', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6313933e-adb0-4db5-9c0b-8dc989038209",
   "metadata": {},
   "source": [
    "\n",
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO101-L04-pg10tutorial.zip)**.\n",
    "\n",
    "# Margin of Error and Confidence Intervals\n",
    "\n",
    "Have you ever paid attention to political polling data? Usually, somewhere in the polling results, there will be something that says \"+/- 3.2 percentage points\". The \"plus or minus\" part is called the *margin of error*. Stated another way, the margin of error is the amount that your estimate can be wrong by. The +/- part of the margin of error is integral - because you can estimate too high or too low.  When you add the margin of error to the mean, the result is the *upper confidence limit.* When you subtract the margin of error from the mean, the result is the *lower confidence limit.* All together, the spread from the lower confidence limit to the upper confidence limit is called the *confidence interval.* \n",
    "\n",
    "![On the left, a vertical line represents the lower confidence limit. On the right, a vertical line represents the upper confidence limit. Equidistant between them is a vertical line that represents the mean. Between the lower confident limit and the mean is the lower margin of error. Between the mean and the upper confidence limit is the upper margin of error. The lower and upper margin of error together is the confidence interval.](Media/CI.png)\n",
    "\n",
    "When you sample, you'll draw conclusions about the population by looking at a very small subset of the population. There is a risk doing this, because you are probably not representing the population exactly perfectly. The sample will have some error associated with it.\n",
    "\n",
    "For this reason, when you use sample statistics to form judgments about population parameters, you typically report both the estimate of the parameter (for instance the mean) and a confidence interval around that estimate. The *confidence interval* tells you that the true mean falls somewhere within the range you have provided. \n",
    "\n",
    "---\n",
    "\n",
    "## An Example \n",
    "\n",
    "For example, if you collect data from a sample of ceramic flooring tiles, you might report that the width is 11.96 inches, plus or minus 0.22 inches. The 0.22 inches is the margin of error.  If you take the mean, 11.96, and add and subtract 0.22 from it, you are left with 11.74 inches, which is your lower confidence limit: \n",
    "\n",
    "```text\n",
    "11.96 -.22 = 11.74 = lower limit of the confidence interval\n",
    "```\n",
    "\n",
    "And 12.18 inches, which is your upper confidence limit: \n",
    "\n",
    "```text\n",
    "11.96 + .22 = 12.18 = upper limit of the confidence interval\n",
    "```\n",
    "\n",
    "Altogether, you could say that the confidence interval is `11.74 - 12.18`. \n",
    "\n",
    "Having a confidence interval implies that you estimate the population parameter is 11.96 inches, but you are confident that the interval from 11.74 inches to 12.18 inches contains the true mean width of all tiles manufactured.\n",
    "\n",
    "---\n",
    "\n",
    "## Calculating Margin of Error\n",
    "\n",
    "Up above, you were already provided a margin of error.  But, you can calculate it, given information about your sample: \n",
    "\n",
    "* *x*-bar = 24.1\n",
    "* *s* = 0.96\n",
    "* *n* = 42\n",
    "\n",
    "There is a *critical value*, or CV, that helps you determine the margin of error. In this case, the CV is 2.58. The critical value is determine by how confident you want to be.  If you want to be 99% confident in your estimate, then your CV will always be 2.58.\n",
    "\n",
    "Here is the formula for margin of error: \n",
    "\n",
    "![The equation to find the margin of error](Media/L08-03.png)\n",
    "\n",
    "For your example, you can plug in the above numbers, and the margin of error is:\n",
    "\n",
    "![Margin of error equals two point zero two times zero point nine six divided by square root of forty two equals zero point two nine nine](Media/L08-04.png)\n",
    "\n",
    "Once you have the margin of error calculated, the confidence interval is easily constructed. To find the confidence interval, simply take the mean minus (`-`) the margin of error for the lower bound of the confidence interval, and the mean plus (`+`) the margin of error for the upper bound of the confidence interval.\n",
    "\n",
    "In this case, the estimate (*x*-bar) is 24.1, and the margin of error is 0.3, which means that the confidence interval is 23.8 to 24.4.\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Check out this Margin of Error and Confidence Interval Calculator!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>There is <a href=\"https://www.woz-students.com/margin-of-error\"> a website here </a> that will calculate margin of error and confidence interval for you! </p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## What Does the Confidence Interval Mean? \n",
    "\n",
    "The confidence interval is a declaration of how good your estimate is. Going back to the example of toddler weights, your summary would be something like \"The estimated mean weight is 24.1 lbs, and the 95% confidence interval (this is often abbreviated as `CI`) is 23.8 to 24.4 lbs\".\n",
    "\n",
    "In general terms, the confidence interval really means that:\n",
    "\n",
    "> There is a 95% chance the confidence interval includes the true mean.\n",
    "\n",
    "You can choose to have confidence intervals at different levels.  The most common are 90%, 95%, and 99%.  The higher the percentage, the wider the confidence interval and the more certain you are about where the true mean falls. \n",
    "\n",
    "![A vertical line showing the mean of three different confidence levels, ninety percent, ninety five percent, and ninety nine percent.](Media/CIranges.png)\n",
    "\n",
    "You can see that there is a lot more places the true mean could be in a 90% CI as opposed to a 99% CI.\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>You may want to watch this <a href=\"https://vimeo.com/458358286\"> recorded live workshop </a> that goes over the content on this page about confidence intervals. </p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Review\n",
    "\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded.\n",
    "\n",
    "```c-lms\n",
    "start-activity: Confidence Interval Quiz\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2d301848-c889-4164-8fbf-51ee82c45f3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a4f4b38503334f919291aaf9301abfc1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('The variance', 0), ('The …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L4P9Q1)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cba89ee-e16c-4b6a-8adb-d08395194afb",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 10 - Measures of Frequency<a class=\"anchor\" id=\"DS101L4_page_10\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "efaf97f0-8a4d-4808-a6d4-af6a031f9575",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/315975438\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x2a284ead160>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Data Science Software\n",
    "VimeoVideo('315975438', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a39259c0-3455-4d3a-be7d-2c44acce5913",
   "metadata": {},
   "source": [
    "The transcript for the above topic tutorial video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO101-L04-pg11tutorial.zip)**.\n",
    "\n",
    "# Measures of Frequency\n",
    "\n",
    "## Counts and Percentages\n",
    "\n",
    "When a sample is collected, the size of the sample is expressed at the count. The sample count is part of many statistics, so in spite of its simplicity, it is important! \n",
    "\n",
    "Count is only a sample statistic, not a population parameter.\n",
    "\n",
    "---\n",
    "\n",
    "### An Election Example\n",
    "\n",
    "Suppose it is an election year, and a polling firm is trying to get a feel for the electorate's preferences. They devise a survey asking which candidate you prefer in an election: Democrat, Republican, Independent, Other, or Don't Know/Undecided.\n",
    "\n",
    "![A hand placing a ballot into a box at a polling place on election day.](Media/L03-12.png)\n",
    "\n",
    "They survey a total of 570 people and want to report their findings. Clearly, the thing that is being measured is a preference, and it is a categorical variable.\n",
    "\n",
    "This **IS NOT** how they will report their results:\n",
    "\n",
    "> The first person you asked wanted the Republican, the second wanted the Democrat, the third wanted the Democrat, etc.\n",
    "\n",
    "Instead, results will typically reported like this:\n",
    "\n",
    "> Polling suggests that the Democrat is favored by 38.8%, the Republican by 27.5%, the Independent by 11.7%, somebody else by 2.3%, and 19.7% are undecided.\n",
    "\n",
    "Better yet, the results will be tabulated:\n",
    "\n",
    "![Democrat, thirty eight point eight percent. Republican, twenty seven point five percent. Independent, eleven point seven percent. Other, two point three percent. Undecided, nineteen point seven percent.](Media/L03-01.png)\n",
    "\n",
    "This is typically called getting frequency counts, and as you've seen above, frequencies (how often something happened) often gets converted to percentages.\n",
    "\n",
    "To report on categorical variables, you will either:\n",
    "\n",
    "* Count up the number of \"successes,\" such as hits, wins, completions, Democrats, or whatever you deem a \"success\" to be, and report that number.\n",
    "\n",
    "* Count up the number of \"successes\" and divide it by the number of \"opportunities\" to report a success percentage.\n",
    "\n",
    "---\n",
    "\n",
    "### A Sports Example\n",
    "\n",
    "Consider a quarterback. Each time he throws the ball, it is a completion, or it is not. Completion (Y/N) is a categorical variable where the possible values are \"yes\" or \"no\" as answers to the question of whether the pass resulted in a completion.\n",
    "\n",
    "Post-game analysis never reports a quarterback's completion sequence was \"yes, yes, no, yes, yes, yes, no, no...\" Instead, they simply say the quarterback was 26 for 38, or report a completion percentage such as 68.4%.\n",
    "\n",
    "---\n",
    "\n",
    "## Proportion\n",
    "\n",
    "Proportion is a very common statistic used to summarize categorical data. The calculation is done by taking the number of successes and dividing it by the number of possibilities. This is similar to probability, but proportion is calculated after the sample is collected, whereas a probability is usually theoretical, based on past experience or knowledge.\n",
    "\n",
    "For example, if you are rolling a die, there is a 1/6 probability (0.167) that any one roll will result in a 3. On the other hand, if you roll a die 600 times, and there are 94 times that a 3 is rolled, then the proportion of 3 is 94/600, or 0.157.\n",
    "\n",
    "It is common to report a baseball player's batting average, expressed as a proportion, rather than identify the sequence of base hits versus outs.\n",
    "\n",
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "Often it is difficult, if not impossible, to measure an entire population, so sampling is imperative.  There are many different ways to sample, and each has its merits for particular situations.  Unfortunately, the least rigorous method, convenience sampling, is the most often used method.\n",
    "\n",
    "Once a sample has been collected, it can be used to describe or infer qualities about the population. Qualities are known as statistics in a sample, and parameters in a population.  A wide variety of sample statistics can be used to describe data, including measures of central tendency, measures of distribution, and measures of frequency. You were able to calculate most of these measures both by hand and using MS Excel formulas.\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Want a little extra information on using spreadsheets to calculate measures of central tendency and distribution? Then please watch <a href=\"https://vimeo.com/402778034\"> this recorded live workshop! </a></p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## Review\n",
    "\n",
    "Below is a quiz to review the recently covered material. Quizzes are _not_ graded.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c2080221-9b6f-4a24-b8de-98e9a4360bac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5b0e56dba7e148a3b1f1f1f7e0cd9b6c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Using means', 0), ('Conve…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e936c0924a5e4e4b90f79d2680349442",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=((\"Probability uses frequenc…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L4P10Q1, L4P10Q2)\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd6049aa-7170-4b4a-9a65-8ed642cd90b8",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 11 - Key Terms<a class=\"anchor\" id=\"DS101L4_page_11\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "\n",
    "# Key Terms\n",
    "\n",
    "Below is a list and short description of the important keywords learned in this lesson. Please read through and go back and review any concepts you do not fully understand. Great Work!\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Statistic</td>\n",
    "        <td>Information about a sample.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Parameter</td>\n",
    "        <td>Information about a statistic.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Mean</td>\n",
    "      <td>The sum of all numbers divided by the number of numbers.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Median</td>\n",
    "      <td>The exact middle of the dataset.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Mode</td>\n",
    "      <td>The number that most frequently occurs.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Range</td>\n",
    "      <td>The distance between the highest and the lowest number in a dataset.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Quartiles</td>\n",
    "      <td>When the data is broken up into four equal parts (quarters).</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Inter-Quartile Range</td>\n",
    "      <td>The distance between the third and first quartile.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Percentile</td>\n",
    "      <td>When the data is broken up into 100 equal parts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Standard Deviation</td>\n",
    "      <td>A measure of the spread of values in a dataset.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Variance</td>\n",
    "      <td>The square of standard deviation.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Confidence Interval</td>\n",
    "      <td>An estimate of the amount of error around the mean.</td>\n",
    "    </tr>\n",
    "      <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Critical Value (CV)</td>\n",
    "      <td>Something used to calculate the margin of error.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Count</td>\n",
    "      <td>How often something occurs in the data.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td style=\"font-weight: bold;\" nowrap>Proportion</td>\n",
    "      <td>How often something occurs in the data in relation to the whole dataset.</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "---\n",
    "\n",
    "## Key MS Excel Formulas\n",
    "\n",
    "* `=average()` Calculates mean\n",
    "* `=median()` Calculates the median\n",
    "* `=mode()` Calculates the mode\n",
    "* `=min()` Calculates the smallest value\n",
    "* `=max()` Calculates the largest value\n",
    "* `=stdev.s()` Calculates the standard deviation\n",
    "* `=var.s()` Calculates the variance\n",
    "* `=quartile()` Calculates the values for quartiles\n",
    "\n",
    "```c-lms\n",
    "topic: Lesson 4 Practice Hands On\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a850fd2-9d83-4258-a4b7-94ae2da99e17",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 12 - Lesson 4 Practice Hands On<a class=\"anchor\" id=\"DS101L4_page_12\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "This activity will not be graded, but the best way to become a data scientist is to practice!\n",
    "\n",
    "#### Caution!\n",
    "Do not submit your project until you have completed all requirements, as you will not be able to resubmit.\n",
    "\n",
    "## Part I\n",
    "Use the **[following dataset](https://repo.exeterlms.com/documents/V2/DataScience/Basic-Statistics/L4ExamQ9-12.zip)** to find in MS Excel or equivalent the:\n",
    "\n",
    "- Mean\n",
    "- Median\n",
    "- 1st quartile\n",
    "- 3rd quartile\n",
    "- Interquartile range\n",
    "\n",
    "## Part II\n",
    "For the **[following dataset](https://repo.exeterlms.com/documents/V2/DataScience/Basic-Statistics/L4ExamQ17-18.zip)**, find the:\n",
    "\n",
    "- Standard deviation\n",
    "- Variance\n",
    "- Minimum value\n",
    "- Maximum value\n",
    "- Range"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf12d0f1-d40f-4cd6-be20-291209a419c9",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 13 - Lesson 4 Practice Hands On Solutions<a class=\"anchor\" id=\"DS101L4_page_13\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "    \n",
    "\n",
    "\n",
    "## Part I\n",
    "\n",
    "- Mean:** 218.3\n",
    "- Median:** 218.5\n",
    "- 1st Quartile:** 207.25\n",
    "- 3rd Quartile:** 228.5\n",
    "- Interquartile Range:** 21.3\n",
    "\n",
    "---\n",
    "\n",
    "## Part II\n",
    "\n",
    "- Standard Deviation:** 19.6\n",
    "- Variance:** 385.3\n",
    "- Min:** 654\n",
    "- Max:** 721\n",
    "Range:** 67\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb6e34e8-b849-40e7-9330-2d7cfa02093c",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 14 - Your Path through Data Science<a class=\"anchor\" id=\"DS101L4_page_14\"></a>\n",
    "\n",
    "[Back to Top](#DS101L4_toc)\n",
    "    \n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "    \n",
    "Quiz on LMS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcc0bf74-7c5d-448a-bce8-9d4681ec2b0d",
   "metadata": {},
   "source": [
    "---\n",
    "## Common Descriptive Statistics\n",
    "### Lesson 4 Exam\n",
    "---\n",
    "This exam ***will be graded***, but please feel free to refer to your notes or the curriculum since you don't have to memorize everything in the real world - as long as you know where to find information!\n",
    "Which of the following formulas will yield the mean in a spreadsheet program like MS Excel or Google Sheets?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3e2019d1-48dc-4756-a110-9745e89dfabf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "84cdd002a962495aae1d7a24fd1b8b4d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('.mean()', 0), ('.average(…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d3595f1f5dce42b58d20ccff1c7dccbf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Mean', 0), ('Median', 1),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4ce383cbd1c4e1987aac6f163696f1e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('.min()', 0), ('.max()', 1…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9713c06cd3644336a5dcff2b39004047",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Range', 0), ('Inter-quart…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0df8cdb8d94a4b4092e81c1baf541f3a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('Coefficient of variation'…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ed37733a55d146f080da558d9f169f82",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(), RadioButtons(layout=Layout(width='max-content'), options=(('It provides a range for w…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    display(L4P14Q1, L4P14Q2, L4P14Q3, L4P14Q4, L4P14Q5, L4P14Q6)\n",
    "except:\n",
    "    pass"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}