A_Data_Mining_Project_Final_Version_A_a.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working With Large Datasets: PLACES LOCAL Data for Better Health.\n",
    "\n",
    "<img src=\"places_data_logo.png\">\n",
    "\n",
    "\n",
    "# Purpose\n",
    "The purpose of this project is to accomplish two main tasks. \n",
    "1) Test Whether or not States with more PLACES locations have different means that those with fewer locations, and how they differ exactly.\n",
    "2) Sucessfully work with very large data sets: Cleaning, organizinng and performing analyisis on them. \n",
    "\n",
    "image credit: https://www.cdc.gov/places/index.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import statistics\n",
    "import scipy.stats as stats\n",
    "from distfit import distfit\n",
    "import statsmodels.api as sm \n",
    "import pylab\n",
    "from numpy import cov\n",
    "from itertools import permutations \n",
    "import itertools\n",
    "from scipy.stats import spearmanr\n",
    "from random import sample \n",
    "import random \n",
    "from collections import Counter\n",
    "from tabulate import tabulate\n",
    "import requests\n",
    "import zipfile\n",
    "import io\n",
    "import os  \n",
    "from scipy.stats import iqr\n",
    "import scikit_posthocs as sp\n",
    "pd.set_option('display.max_colwidth', None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Background, Loading, Inspection\n",
    "The CDC and Robert Wood Johnson foundation have been partnering for the last few years to provide a huge repository of data and metrics on public health*. \\\n",
    "This data is known as the PLACES dataset.The purpose of such data is to help the CDC stay ahead of emerging disease trends. \\\n",
    "There are hundreds of cities and many disease metrics that are tracked,\\\n",
    "meaning a vast amount of information can be mined and gained from such a repository.\\\n",
    "My project hopes to do two things: First would be to navigate such a large dataset successfully,\\\n",
    "second to answer the question: For the States with more PLACES data locations, are disease metrics generally different. And how do they differ? \n",
    "\n",
    "## Data loading\n",
    "There are many ways to acess the files that are needed. Either downloading from github, \\\n",
    "or if the files are too big  we can use the shortcuts themselves. And download right from the source. \\\n",
    "The files can be found below. The next few cells load in the data and put it into the current working directory. \n",
    "Note: The 2020 release covers 2017-2018, 2021 release 2019-2020 and 2022 release 2020-2021.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "https://catalog.data.gov/dataset/places-local-data-for-better-health-place-data-2022-release/resource/4bfea8ab-534b-4f2b-9cb1-d0b951709c2a \n",
    "\n",
    "https://catalog.data.gov/dataset/places-local-data-for-better-health-place-data-2021-release-06a9b/resource/f9bb4b0d-7db2-432e-bf3e-58bd400a6ffc \n",
    "\n",
    "https://catalog.data.gov/dataset/places-local-data-for-better-health-place-data-2020-release-670b7/resource/b95272f6-c03d-487b-81fa-460d4b9bcb1f \n",
    "\n",
    "\n",
    "\n",
    "https://www.cdc.gov/places/index.html*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "current_directory = os.getcwd()\n",
    "\n",
    "print('Downloading started')\n",
    "url = 'https://github.com/oohtmeel1/Data_Mining_Final/raw/main/s2020.zip'\n",
    "\n",
    "# Downloading the file by sending the request to the URL\n",
    "req = requests.get(url)\n",
    "\n",
    "# Check if the request was successful (status code 200)\n",
    "if req.status_code == 200:\n",
    "    # Create a BytesIO object from the content\n",
    "    zip_content = io.BytesIO(req.content)\n",
    "\n",
    "    # Get the current working directory\n",
    "    current_directory = os.getcwd()\n",
    "    print(f'Current Working Directory: {current_directory}')\n",
    "\n",
    "    # Specify the directory where you want to extract the contents\n",
    "    extraction_path = 'zipped_files_now'  \n",
    "    zip_ref = zipfile.ZipFile(zip_content, 'r')\n",
    "\n",
    "    # Create the extraction directory if it doesn't exist\n",
    "    os.makedirs(extraction_path, exist_ok=True)\n",
    "\n",
    "    # Join the extraction path and the filename\n",
    "    extraction_file_path = os.path.join(extraction_path, 's2020.zip')\n",
    "\n",
    "    # Extract the contents\n",
    "    zip_ref.extractall(extraction_file_path)\n",
    "    zip_ref.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "extraction_path = 'zipped_files_now'\n",
    "# Specify the filename \n",
    "csv_filename = 's2020.csv'  # Replace with the actual CSV file name\n",
    "\n",
    "# Join the extraction path and the filename\n",
    "csv_file_path = os.path.join(extraction_path, csv_filename)\n",
    "\n",
    "\n",
    "    # Read the CSV file into a DataFrame\n",
    "s2020 = pd.read_csv(csv_file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "url = 'https://data.cdc.gov/api/views/q8ig-wwk9/rows.csv?accessType=DOWNLOAD'\n",
    "\n",
    "# Downloading the file by sending the request to the URL\n",
    "req = requests.get(url)\n",
    "\n",
    "# Check if the request was successful \n",
    "if req.status_code == 200:\n",
    "    # Create a BytesIO object from the content\n",
    "    s2021 = pd.read_csv(io.StringIO(req.text))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import pandas as pd\n",
    "\n",
    "url = ' https://data.cdc.gov/api/views/epbn-9bv3/rows.csv?accessType=DOWNLOAD'\n",
    "\n",
    "# Downloading the file by sending the request to the URL\n",
    "req = requests.get(url)\n",
    "\n",
    "# Check if the request was successful \n",
    "if req.status_code == 200:\n",
    "    # Create a BytesIO object from the content\n",
    "    s2022 = pd.read_csv(io.StringIO(req.text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s2021.info()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s2020.info()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s2022.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Some Basic Data Cleaning\n",
    "All NA values are dropped. And all rows and columns have been inspected to ensure correctness.\\\n",
    "The only null values remaining are data value footnotes(Which are NaN) but are of no importance.\\\n",
    "Otherwise the Data Value Column does not containy any null values. And looking at the info shows \\\n",
    "there are no very large values.Which is a good sign that there are no random digits present."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop NA values\n",
    "s2020['Measure'] = s2020['Measure'].str.replace(\" \", \"_\") # Strip whitespace and replace with _\n",
    "s2021['Measure'] = s2021['Measure'].str.replace(\" \", \"_\") # Strip whitespace and replace with _\n",
    "s2022['Measure'] = s2022['Measure'].str.replace(\" \", \"_\") # Strip whitespace and replace with _\n",
    "s2020= s2020[s2020['Data_Value'].notna()]\n",
    "s2021= s2021[s2021['Data_Value'].notna()]\n",
    "s2022= s2022[s2022['Data_Value'].notna()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.where(pd.isnull(s2020['Data_Value'])) # Returns where the dataframe is null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.where(pd.isnull(s2021['Data_Value']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.where(pd.isnull(s2022['Data_Value']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s2020.loc[s2020['Data_Value'].idxmax()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s2021.loc[s2021['Data_Value'].idxmax()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s2022.loc[s2022['Data_Value'].idxmax()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Some basic visualizations\n",
    "How are the locations distributed for each state?\n",
    "The number of unique locations are counted and plotted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "city_counts1 = s2020.groupby('StateAbbr')['LocationName'].nunique()\n",
    "city_counts1 = city_counts1.sort_values(ascending=False)\n",
    "\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(city_counts1,palette=\"viridis\")\n",
    "\n",
    "ax.set(xlabel=\"State\",ylabel =\"Number of locations \",title=\"Locations per State 2017-2018\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "city_counts2 = s2021.groupby('StateAbbr')['LocationName'].nunique()\n",
    "city_counts2 = city_counts2.sort_values(ascending=False)\n",
    "\n",
    "\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(city_counts2,palette=\"viridis\")\n",
    "\n",
    "ax.set(xlabel=\"State\",ylabel =\"Number of locations \",title=\"Locations per State 2018-2019\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "city_counts3 = s2022.groupby('StateAbbr')['LocationName'].nunique()\n",
    "city_counts3 = city_counts3.sort_values(ascending=False)\n",
    "\n",
    "\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(city_counts3,palette=\"viridis\")\n",
    "\n",
    "ax.set(xlabel=\"State\",ylabel =\"Number of locations \",title=\"Locations per State 2019-2020\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "measure_counts = s2020.groupby(['StateAbbr', 'Measure']).size().reset_index(name='Count')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What about how many data points per state?\n",
    "The below plots answer the obvious. Do places with more locations measured overall \\\n",
    " have a higher number of Data points obtained. Yes, the answer is yes. They also look fairly consistent year to year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 6))\n",
    "sns.barplot(x='StateAbbr', y='Count', data=measure_counts, dodge=True,legend=False)\n",
    "plt.xlabel('State Abbreviation')\n",
    "plt.ylabel('Count')\n",
    "plt.title('Count of Measures in Each State 2017-2018')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "measure_counts1 = s2021.groupby(['StateAbbr', 'Measure']).size().reset_index(name='Count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 6))\n",
    "sns.barplot(x='StateAbbr', y='Count', data=measure_counts1, dodge=True,legend=False)\n",
    "plt.xlabel('State Abbreviation')\n",
    "plt.ylabel('Count')\n",
    "plt.title('Count of Measures in Each State 2018-2019')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "measure_counts3 = s2022.groupby(['StateAbbr', 'Measure']).size().reset_index(name='Count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 6))\n",
    "sns.barplot(x='StateAbbr', y='Count', data=measure_counts3, dodge=True,legend=False)\n",
    "plt.xlabel('State Abbreviation')\n",
    "plt.ylabel('Count')\n",
    "plt.title('Count of Measures in Each State 2019-2020')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How are values divided up as far as category goes?\n",
    "There are 3. PREVENT, HLTHOUT, abd UNHBEH.\n",
    "* PREVENT - preventative\n",
    "* HLTHOUT - health outcomes\n",
    "* UNHBEH - unhealthy behavior. \\\n",
    "HLTHOUT always has the most, PREVENT seems to be second and UNHBEH usually has the smallest number of entries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "measure_counts5 = s2020.groupby(['StateAbbr', 'CategoryID']).size().reset_index(name='Count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "measure_counts6 = s2021.groupby(['StateAbbr', 'CategoryID']).size().reset_index(name='Count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "measure_counts7 = s2022.groupby(['StateAbbr', 'CategoryID']).size().reset_index(name='Count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 6))\n",
    "sns.barplot(x='StateAbbr', y='Count', data=measure_counts5, hue ='CategoryID',dodge=True,legend=True)\n",
    "plt.xlabel('State Abbreviation')\n",
    "plt.ylabel('Count')\n",
    "plt.title('Count of Measures in Each State 2017-2018')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 6))\n",
    "sns.barplot(x='StateAbbr', y='Count', data=measure_counts6, hue ='CategoryID',dodge=True,legend=True)\n",
    "plt.xlabel('State Abbreviation')\n",
    "plt.ylabel('Count')\n",
    "plt.title('Count of Measures in Each State 2018-2019')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 6))\n",
    "sns.barplot(x='StateAbbr', y='Count', data=measure_counts7, hue ='CategoryID',dodge=True,legend=True)\n",
    "plt.xlabel('State Abbreviation')\n",
    "plt.ylabel('Count')\n",
    "plt.title('Count of Measures in Each State 2019-2020')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code to split large dataframes into smaller chunks\n",
    "It is really cumbersome trying to work with these dataframes as they are. So I am going to split them up into smaller bits.\\\n",
    "Working with such a large amounts of data can be a little bit unwieldy at first. \n",
    "So while it is not recommended for normal use, in this case \\\n",
    "I used the exec and eval functions to split the dataframes into smaller chunks\\\n",
    "Storing those in a list of dataframes. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#Getting measures by themselves and creating just a ton of df\n",
    "sa=s2020['StateAbbr'].unique()  # Getting all names of all sates\n",
    "n = len(sa)\n",
    "new_list_of_titles = []\n",
    "dfs = []\n",
    "for i in sa:\n",
    "\tif i not in new_list_of_titles:\n",
    "\t\tnew_list_of_titles.append(i)\n",
    "print(new_list_of_titles)\n",
    "\n",
    "\n",
    "grouped = s2020.groupby(s2020['StateAbbr']) # Group by year\n",
    "for i in new_list_of_titles:\n",
    "\texec(f'{i}=grouped.get_group(\"{i}\")')  # Using f string here to execute code. Getting the groups of the grouped well.groups. and executing based on state. So I make 51 dataframes this way one for each\n",
    "dataframes_dict = {}\n",
    "\n",
    "# Create DataFrames and store them in the dictionary \n",
    "for i in new_list_of_titles:\n",
    "\tdataframes_dict[i] = eval(i)\n",
    "# put all of them into a list so I can maybe iterateover them easier Use this for the iteration\n",
    "#eval(f'dataframes_list_{year} = list(dataframes_dict.values())') # no one sees me using exec. Nope. \n",
    "dataframes_list2020 = list(dataframes_dict.values())\n",
    "\t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframes_list2020[0].info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\t\n",
    "sa=s2021['StateAbbr'].unique()  # Getting all names of all sates\n",
    "n = len(sa)\n",
    "new_list_of_titles = []\n",
    "dfs = []\n",
    "for i in sa:\n",
    "\tif i not in new_list_of_titles:\n",
    "\t\tnew_list_of_titles.append(i)\n",
    "print(new_list_of_titles)\n",
    "\n",
    "\n",
    "grouped = s2021.groupby(s2021['StateAbbr']) # Group by year\n",
    "for i in new_list_of_titles:\n",
    "\texec(f'{i}=grouped.get_group(\"{i}\")')  # Using f string here to execute code. Getting the groups of the grouped well.groups. and executing based on state. So I make 51 dataframes this way one for each\n",
    "dataframes_dict = {}\n",
    "\n",
    "# Create DataFrames and store them in the dictionary \n",
    "for i in new_list_of_titles:\n",
    "\tdataframes_dict[i] = eval(i)\n",
    "# put all of them into a list so I can maybe iterateover them easier Use this for the iteration\n",
    "#eval(f'dataframes_list_{year} = list(dataframes_dict.values())') # no one sees me using exec. Nope. \n",
    "dataframes_list2021 = list(dataframes_dict.values())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Getting measures by themselves and creating just a ton of df\n",
    "sa=s2022['StateAbbr'].unique()  # Getting all names of all sates\n",
    "n = len(sa)\n",
    "new_list_of_titles = []\n",
    "dfs = []\n",
    "for i in sa:\n",
    "\tif i not in new_list_of_titles:\n",
    "\t\tnew_list_of_titles.append(i)\n",
    "print(new_list_of_titles)\n",
    "\n",
    "\n",
    "grouped = s2022.groupby(s2022['StateAbbr']) # Group by year\n",
    "for i in new_list_of_titles:\n",
    "\texec(f'{i}=grouped.get_group(\"{i}\")')  # Using f string here to execute code. Getting the groups of the grouped well.groups. and executing based on state. So I make 51 dataframes this way one for each\n",
    "dataframes_dict = {}\n",
    "\n",
    "# Create DataFrames and store them in the dictionary \n",
    "for i in new_list_of_titles:\n",
    "\tdataframes_dict[i] = eval(i)\n",
    "# put all of them into a list so I can maybe iterateover them easier Use this for the iteration\n",
    "#eval(f'dataframes_list_{year} = list(dataframes_dict.values())') # no one sees me using exec. Nope. \n",
    "dataframes_list2022 = list(dataframes_dict.values())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The new lists of dataframes can use indexing to access where each State is. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframes_list2020[0].head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframes_list2021[0].head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframes_list2022[0]['Measure'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to deal with outliers with large amounts of data\n",
    "There are many ways to approach this problem. The biggest factor is time\\\n",
    "larger datasets require more time and care since it is easier to miss values that may not belong or outliers.\\\n",
    "While histograms and QQ plots are still useful, it might be better to use some python libraries and functions to help \\\n",
    "and to save time. \n",
    "The percentile functions will be used to perform this. A summary is given below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below runs the summary statistics for all unique measures in the entire dataframe by State.\n",
    "Those summary statistics include: count, mean, standard deviation(std), min,\\\n",
    "25%(25th percentile of the data),50%(50th percentile of the data),\\\n",
    "75%(75th percentile of the data),and the max value found in the dataset.\n",
    "`quartiles are a special type of percentile by the way denoting the 25th,50th and 75th percentile` \n",
    "\n",
    "_An example of my dataframe output:_\n",
    "|       |   Data_Value |\n",
    "|:------|-------------:|\n",
    "| count |   2388       |\n",
    "| mean  |     22.4544  |\n",
    "| std   |      4.24708 |\n",
    "| min   |     11       |\n",
    "| 25%   |     20       |\n",
    "| 50%   |     23       |\n",
    "| 75%   |     25       |\n",
    "| max   |     32       |\n",
    "\n",
    "\\\n",
    "Visually represented the data would look like:\n",
    "\n",
    "\n",
    "!['Boxplot'](Boxplot_example.png)\n",
    "\n",
    "\n",
    "* 25th quartile being Q1\n",
    "* 50th quartile being the median\n",
    "* 75th quartile being Q3\n",
    "\n",
    "In order to remove outliers we can use the ``` IQR ``` method. (Interquartile range method)\\\n",
    "Where we set up a 'fence' outside of Q1 and Q3. Anything outside of this fence is considered an outlier.\\\n",
    "The formula for calculating the 'fence' is : \\\n",
    "$\n",
    "((1.5 * IQR) - Q1)$ \\\n",
    "OR \\\n",
    "$\n",
    "((1.5 * IQR) + Q3)$\n",
    "\n",
    "* $IQR$ is the interquartile range. Which is obtained by subtracting Q1-Q3.\n",
    "\n",
    "\n",
    "https://online.stat.psu.edu/stat200/lesson/3/3.2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abc=dataframes_list2020[50]\n",
    "len(abc.loc[abc['Measure']=='Chronic_obstructive_pulmonary_disease_among_adults_aged_>=18_years','Data_Value'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting rid of outliers 2020"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_sum=[]\n",
    "y=[]\n",
    "n = len(dataframes_list2020)\n",
    "#n=2\n",
    "new_list_of_measures=[]\n",
    "print_count = 0\n",
    "for j in range(n):\n",
    "\tmatches_2020 = dataframes_list2020[j] # matches 2020 is first dataframe\n",
    "\tsubseta=(matches_2020['StateAbbr'].unique()) # Just a list of the state names in that df\n",
    "\t#print(subseta)\n",
    "\tmatching_indices1 = [index for index, df in enumerate(dataframes_list2020) if (df['StateAbbr'].isin(subseta).any())]# No matter where they are, this could should find the matching states. \n",
    "\tfor index in matching_indices1:\n",
    "\t\tmatches_2020 = dataframes_list2020[index]\n",
    "\t\tsubset1=(matches_2020['Measure'].unique()) \n",
    "\t\tintersection_list = set(list(subset1))\n",
    "\t\tfor i in intersection_list:\n",
    "\t\t\tlis1 = matches_2020.loc[matches_2020['Measure']==i,'Data_Value'] # select all of the values in the dataframe that match\n",
    "\t\t\t#print(len(lis1))\n",
    "\t\t\tmy_array = np.array(lis1)\n",
    "\t\t\tq1,q3 = np.percentile(my_array, (25,75))\n",
    "\t\t\tiqr1 = q3-q1\t\t\n",
    "\t\t\t#print(iqr)\n",
    "\t\t\tmin1 = q1-(1.5*iqr1) # Subtract from Q1\n",
    "\t\t\t#(print(min1))\n",
    "\t\t\tmax1 = (1.5*iqr1)+q3 # Add to Q3\n",
    "\t\t\tx_sum.append([subseta,min1,max1,i,q1,q3])\n",
    "\t\t\t#print(\"Total prints:\", print_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "List_of_min_max=pd.DataFrame(x_sum) # Put those in a dataframe\n",
    "List_of_min_max['StateAbbr'] = List_of_min_max[0].astype(str)\n",
    "List_of_min_max['StateAbbr'] = List_of_min_max['StateAbbr'].str.replace(\"['\", '').str.replace(\"']\", '')\n",
    "List_of_min_max[\"StateAbbr\"] = List_of_min_max[\"StateAbbr\"].str.strip()\n",
    "List_of_min_max['min1'] = List_of_min_max[1].astype(float)\n",
    "List_of_min_max['max1'] = List_of_min_max[2].astype(float)\n",
    "List_of_min_max['Measure'] = List_of_min_max[3].astype(str)\n",
    "List_of_min_max['Measure'] = List_of_min_max['Measure'].str.strip()\n",
    "List_of_min_max['Q1'] = List_of_min_max[4].astype(float)\n",
    "List_of_min_max['Q3'] = List_of_min_max[5].astype(float)\n",
    "List_of_min_max= List_of_min_max.drop([0,1,2,3,4,5],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "okee=s2020[s2020['StateAbbr']=='NC']\n",
    "abc2=pd.DataFrame(okee.loc[okee['Measure']=='Older_adult_women_aged_>=65_years_who_are_up_to_date_on_a_core_set_of_clinical_preventive_services:_Flu_shot_past_year,_PPV_shot_ever,_Colorectal_cancer_screening,_and_Mammogram_past_2_years','Data_Value'])\n",
    "abc2.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experimentdf=dataframes_list2020[0] # Manually Checking My data\n",
    "experimentdf1=dataframes_list2020[1] \n",
    "experimentdf2=dataframes_list2020[2] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experimentdf.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experimentdf1.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_names=s2020.columns.values.tolist()\n",
    "dataframes_list2020_1 =[] #<- New Dataframe list\n",
    "n = len(dataframes_list2020)\n",
    "\n",
    "result_df = pd.DataFrame()\n",
    "for j in range(n):\n",
    "\n",
    "\tmatches_2020 = dataframes_list2020[j] # <- Change this matches 2020 is first dataframe\n",
    "\tmatches_2020['Measure'] = matches_2020['Measure'].str.strip()\n",
    "\tmatches_2020['StateAbbr']=matches_2020['StateAbbr'].str.strip()\n",
    "\tmatches_2020['Data_Value']=matches_2020['Data_Value'].astype(int)\n",
    "\t#print(matches_2020['StateAbbr'].unique())\n",
    "\txbar = len(matches_2020['Measure'].unique()) # match all of the unique values\n",
    "\tlong_dataframe_list=[]\n",
    "\tfor k in range(xbar):\n",
    "\t\n",
    "\t\tfirst_row = List_of_min_max.iloc[k] # Matches the same index in my new df because they should be the same\n",
    "\t\t#print(first_row)\n",
    "\t\tstate_abbr = first_row[0]\n",
    "\t\t#print(state_abbr)\n",
    "\t\tmeasure = first_row[3]\n",
    "\t\t#print(measure)\n",
    "\t\tmaxes1 = first_row[2].astype(int)\n",
    "\t\t#print(maxes1)\n",
    "\t\tmins1 = first_row[1].astype(int)\n",
    "\t\t#print(mins1)\n",
    "\t\tcondition = matches_2020[matches_2020['Measure']== measure]\n",
    "\t\tfiltering_now = condition[condition['Data_Value'] >= mins1]\n",
    "\t\tfiltering_now1 = filtering_now[filtering_now['Data_Value'] <= maxes1]\n",
    "\t\t#result_df = pd.concat([result_df,filtering_now1])\n",
    "\t\tresult_df1= filtering_now1.values.tolist()\n",
    "\t\tlong_dataframe_list.extend(result_df1)\n",
    "\t\t\n",
    "\tresults_test=pd.DataFrame(long_dataframe_list,columns=column_names)\n",
    "\tdataframes_list2020_1.append(results_test) # Remember to change this after debugging\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting rid of outliers for 2021"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_sum1=[]\n",
    "y=[]\n",
    "n = len(dataframes_list2021)\n",
    "\n",
    "new_list_of_measures=[]\n",
    "print_count = 0\n",
    "for j in range(n):\n",
    "\tmatches_2021 = dataframes_list2021[j] # matches 2020 is first dataframe\n",
    "\tsubseta=(matches_2021['StateAbbr'].unique()) # Just a list of the state names in that df\n",
    "\t#print(subseta)\n",
    "\tmatching_indices1 = [index for index, df in enumerate(dataframes_list2021) if (df['StateAbbr'].isin(subseta).any())]# No matter where they are, this could should find the matching states. \n",
    "\tfor index in matching_indices1:\n",
    "\t\tmatches_2021 = dataframes_list2021[index]\n",
    "\t\tsubset1=(matches_2021['Measure'].unique()) \n",
    "\t\tintersection_list = set(list(subset1))\n",
    "\t\tfor i in intersection_list:\n",
    "\t\t\tlis1 = matches_2021.loc[matches_2021['Measure']==i,'Data_Value'] # select all of the values in the dataframe that match\n",
    "\t\t\t#print(len(lis1))\n",
    "\t\t\tmy_array = np.array(lis1)\n",
    "\t\t\tq1,q3 = np.percentile(my_array, (25,75))\n",
    "\t\t\tiqr1 = q3-q1\t\t\n",
    "\t\t\t#print(iqr)\n",
    "\t\t\tmin1 = q1-(1.5*iqr1) # Subtract from Q1\n",
    "\t\t\t#(print(min1))\n",
    "\t\t\tmax1 = (1.5*iqr1)+q3 # Add to Q3\n",
    "\t\t\tx_sum1.append([subseta,min1,max1,i,q1,q3])\n",
    "\t\t\t#print(\"Total prints:\", print_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "List_of_min_max=pd.DataFrame(x_sum1) # Put those in a dataframe\n",
    "List_of_min_max['StateAbbr'] = List_of_min_max[0].astype(str)\n",
    "List_of_min_max['StateAbbr'] = List_of_min_max['StateAbbr'].str.replace(\"['\", '').str.replace(\"']\", '')\n",
    "List_of_min_max[\"StateAbbr\"] = List_of_min_max[\"StateAbbr\"].str.strip()\n",
    "List_of_min_max['min1'] = List_of_min_max[1].astype(float)\n",
    "List_of_min_max['max1'] = List_of_min_max[2].astype(float)\n",
    "List_of_min_max['Measure'] = List_of_min_max[3].astype(str)\n",
    "List_of_min_max['Measure'] = List_of_min_max['Measure'].str.strip()\n",
    "List_of_min_max['Q1'] = List_of_min_max[4].astype(float)\n",
    "List_of_min_max['Q3'] = List_of_min_max[5].astype(float)\n",
    "List_of_min_max= List_of_min_max.drop([0,1,2,3,4,5],axis=1)\n",
    "List_of_min_max_2021 =List_of_min_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_names=s2021.columns.values.tolist()\n",
    "dataframes_list2021_1 =[] #<- New Dataframe list\n",
    "n = len(dataframes_list2021)\n",
    "\n",
    "result_df = pd.DataFrame()\n",
    "for j in range(n):\n",
    "\n",
    "\tmatches_2021 = dataframes_list2021[j] # <- Change this matches 2020 is first dataframe\n",
    "\tmatches_2021['Measure'] = matches_2021['Measure'].str.strip()\n",
    "\tmatches_2021['StateAbbr']=matches_2021['StateAbbr'].str.strip()\n",
    "\tmatches_2021['Data_Value']=matches_2021['Data_Value'].astype(int)\n",
    "\t#print(matches_2020['StateAbbr'].unique())\n",
    "\txbar = len(matches_2021['Measure'].unique()) # match all of the unique values\n",
    "\tlong_dataframe_list=[]\n",
    "\tfor k in range(xbar):\n",
    "\t\n",
    "\t\tfirst_row = List_of_min_max_2021.iloc[k] # Matches the same index in my new df because they should be the same\n",
    "\t\t#print(first_row)\n",
    "\t\tstate_abbr = first_row[0]\n",
    "\t\t#print(state_abbr)\n",
    "\t\tmeasure = first_row[3]\n",
    "\t\t#print(measure)\n",
    "\t\tmaxes1 = first_row[2].astype(int)\n",
    "\t\t#print(maxes1)\n",
    "\t\tmins1 = first_row[1].astype(int)\n",
    "\t\t#print(mins1)\n",
    "\t\tcondition = matches_2021[matches_2021['Measure']== measure]\n",
    "\t\tfiltering_now = condition[condition['Data_Value'] >= mins1]\n",
    "\t\tfiltering_now1 = filtering_now[filtering_now['Data_Value'] <= maxes1]\n",
    "\t\t#result_df = pd.concat([result_df,filtering_now1])\n",
    "\t\tresult_df1= filtering_now1.values.tolist()\n",
    "\t\tlong_dataframe_list.extend(result_df1)\n",
    "\t\t\n",
    "\tresults_test=pd.DataFrame(long_dataframe_list,columns=column_names)\n",
    "\tdataframes_list2021_1.append(results_test) # Remember to change this after debugging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframes_list2021_1[0]['Measure'].nunique()\n",
    "dataframes_list2021_1[0].loc[dataframes_list2021_1[0]['Measure']=='Current_smoking_among_adults_aged_>=18_years','Data_Value'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframes_list2021[0].loc[dataframes_list2021[0]['Measure']=='Current_smoking_among_adults_aged_>=18_years','Data_Value'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting rid of outliers 2022"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_sum2=[]\n",
    "y=[]\n",
    "n = len(dataframes_list2022)\n",
    "\n",
    "new_list_of_measures=[]\n",
    "print_count = 0\n",
    "for j in range(n):\n",
    "\tmatches_2022 = dataframes_list2022[j] # matches 2020 is first dataframe\n",
    "\tsubseta=(matches_2022['StateAbbr'].unique()) # Just a list of the state names in that df\n",
    "\t#print(subseta)\n",
    "\tmatching_indices1 = [index for index, df in enumerate(dataframes_list2022) if (df['StateAbbr'].isin(subseta).any())]# No matter where they are, this could should find the matching states. \n",
    "\tfor index in matching_indices1:\n",
    "\t\tmatches_2022 = dataframes_list2022[index]\n",
    "\t\tsubset1=(matches_2022['Measure'].unique()) \n",
    "\t\tintersection_list = set(list(subset1))\n",
    "\t\tfor i in intersection_list:\n",
    "\t\t\tlis1 = matches_2022.loc[matches_2022['Measure']==i,'Data_Value'] # select all of the values in the dataframe that match\n",
    "\t\t\t#print(len(lis1))\n",
    "\t\t\tmy_array = np.array(lis1)\n",
    "\t\t\tq1,q3 = np.percentile(my_array, (25,75))\n",
    "\t\t\tiqr1 = q3-q1\t\t\n",
    "\t\t\t#print(iqr)\n",
    "\t\t\tmin1 = q1-(1.5*iqr1) # Subtract from Q1\n",
    "\t\t\t#(print(min1))\n",
    "\t\t\tmax1 = (1.5*iqr1)+q3 # Add to Q3\n",
    "\t\t\tx_sum2.append([subseta,min1,max1,i,q1,q3])\n",
    "\t\t\t#print(\"Total prints:\", print_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "List_of_min_max=pd.DataFrame(x_sum2) # Put those in a dataframe\n",
    "List_of_min_max['StateAbbr'] = List_of_min_max[0].astype(str)\n",
    "List_of_min_max['StateAbbr'] = List_of_min_max['StateAbbr'].str.replace(\"['\", '').str.replace(\"']\", '')\n",
    "List_of_min_max[\"StateAbbr\"] = List_of_min_max[\"StateAbbr\"].str.strip()\n",
    "List_of_min_max['min1'] = List_of_min_max[1].astype(float)\n",
    "List_of_min_max['max1'] = List_of_min_max[2].astype(float)\n",
    "List_of_min_max['Measure'] = List_of_min_max[3].astype(str)\n",
    "List_of_min_max['Measure'] = List_of_min_max['Measure'].str.strip()\n",
    "List_of_min_max['Q1'] = List_of_min_max[4].astype(float)\n",
    "List_of_min_max['Q3'] = List_of_min_max[5].astype(float)\n",
    "List_of_min_max= List_of_min_max.drop([0,1,2,3,4,5],axis=1)\n",
    "List_of_min_max_2022 =List_of_min_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_names=s2022.columns.values.tolist()\n",
    "dataframes_list2022_1 =[] #<- New Dataframe list\n",
    "n = len(dataframes_list2022)\n",
    "\n",
    "result_df = pd.DataFrame()\n",
    "for j in range(n):\n",
    "\n",
    "\tmatches_2022 = dataframes_list2022[j] # <- Change this matches 2020 is first dataframe\n",
    "\tmatches_2022['Measure'] = matches_2022['Measure'].str.strip()\n",
    "\tmatches_2022['StateAbbr']=matches_2022['StateAbbr'].str.strip()\n",
    "\tmatches_2022['Data_Value']=matches_2022['Data_Value'].astype(int)\n",
    "\t#print(matches_2020['StateAbbr'].unique())\n",
    "\txbar = len(matches_2022['Measure'].unique()) # match all of the unique values\n",
    "\tlong_dataframe_list=[]\n",
    "\tfor k in range(xbar):\n",
    "\t\n",
    "\t\tfirst_row = List_of_min_max_2022.iloc[k] # Matches the same index in my new df because they should be the same\n",
    "\t\t#print(first_row)\n",
    "\t\tstate_abbr = first_row[0]\n",
    "\t\t#print(state_abbr)\n",
    "\t\tmeasure = first_row[3]\n",
    "\t\t#print(measure)\n",
    "\t\tmaxes1 = first_row[2].astype(int)\n",
    "\t\t#print(maxes1)\n",
    "\t\tmins1 = first_row[1].astype(int)\n",
    "\t\t#print(mins1)\n",
    "\t\tcondition = matches_2022[matches_2022['Measure']== measure]\n",
    "\t\tfiltering_now = condition[condition['Data_Value'] >= mins1]\n",
    "\t\tfiltering_now1 = filtering_now[filtering_now['Data_Value'] <= maxes1]\n",
    "\t\t#result_df = pd.concat([result_df,filtering_now1])\n",
    "\t\tresult_df1= filtering_now1.values.tolist()\n",
    "\t\tlong_dataframe_list.extend(result_df1)\n",
    "\t\t\n",
    "\tresults_test=pd.DataFrame(long_dataframe_list,columns=column_names)\n",
    "\tdataframes_list2022_1.append(results_test) # Remember to change this after debugging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframes_list2022_1[0]['Measure'].nunique()\n",
    "dataframes_list2022_1[0].loc[dataframes_list2022_1[0]['Measure']=='Current_smoking_among_adults_aged_>=18_years','Data_Value'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframes_list2022[0].loc[dataframes_list2022[0]['Measure']=='Current_smoking_among_adults_aged_>=18_years','Data_Value'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframes_list2022_1[50].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All Cells seem normal. The values update accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframes_list2020 = dataframes_list2020_1\n",
    "dataframes_list2021 = dataframes_list2021_1\n",
    "dataframes_list2022 = dataframes_list2022_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Okay a note on Density Plots and why I am not using them. \n",
    "Density plots can be very useful for visualizing distributions of numeric data but with huge amounts of data can be overwhelming. \n",
    "\n",
    "The below code visualizes just a few. \n",
    "In order to plot all of them would require a huge amount of computing power. So it does not make sense to do this for so much data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lmn = list(s2020['Measure'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lmn[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "n = 5\n",
    "lmn = list(s2020['Measure'].unique())\n",
    "\n",
    "plt.figure(figsize=(10, 6))\n",
    "abc_index = 0  # Initialize index for lmn\n",
    "for j in range(n):\n",
    "    dataframe_needed = dataframes_list2020_1[j]\n",
    "    abc = lmn[abc_index]\n",
    "    data_time = dataframe_needed.loc[dataframe_needed['Measure'] == abc, 'Data_Value']\n",
    "    sns.kdeplot(data=data_time, fill=True, label=f'DataFrame {j + 1} - {dataframe_needed[\"StateAbbr\"].unique()}')\n",
    "\n",
    "    if j == n:\n",
    "        # Increment index for abc if it's the last dataframe\n",
    "        abc_index += 1\n",
    "\n",
    "plt.legend(title='DataFrame')\n",
    "plt.title(f'{abc}')\n",
    "plt.xlabel('Data Value')\n",
    "plt.show()\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "n = 10\n",
    "lmn = list(s2020['Measure'].unique())\n",
    "\n",
    "plt.figure(figsize=(10, 6))\n",
    "abc_index = 0  # Initialize index for lmn\n",
    "for j in range(5,n):\n",
    "    dataframe_needed = dataframes_list2020_1[j]\n",
    "    abc = lmn[abc_index]\n",
    "    data_time = dataframe_needed.loc[dataframe_needed['Measure'] == abc, 'Data_Value']\n",
    "    sns.kdeplot(data=data_time, fill=True, label=f'DataFrame {j + 1} - {dataframe_needed[\"StateAbbr\"].unique()}')\n",
    "\n",
    "    if j == n:\n",
    "        # Increment index for abc if it's the last dataframe\n",
    "        abc_index += 1\n",
    "\n",
    "plt.legend(title='DataFrame')\n",
    "plt.title(f'{abc}')\n",
    "plt.xlabel('Data Value')\n",
    "plt.show()\n",
    "n = 15\n",
    "lmn = list(s2020['Measure'].unique())\n",
    "\n",
    "plt.figure(figsize=(10, 6))\n",
    "abc_index = 0  # Initialize index for lmn\n",
    "for j in range(10,n):\n",
    "    dataframe_needed = dataframes_list2020_1[j]\n",
    "    abc = lmn[abc_index]\n",
    "    data_time = dataframe_needed.loc[dataframe_needed['Measure'] == abc, 'Data_Value']\n",
    "    sns.kdeplot(data=data_time, fill=True, label=f'DataFrame {j + 1} - {dataframe_needed[\"StateAbbr\"].unique()}')\n",
    "\n",
    "    if j == n:\n",
    "        # Increment index for abc if it's the last dataframe\n",
    "        abc_index += 1\n",
    "\n",
    "plt.legend(title='DataFrame')\n",
    "plt.title(f'{abc}')\n",
    "plt.xlabel('Data Value')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# So what to do when visualization is unrealistic but we need to check out the distributions of the data so the right type of test can be applied?\n",
    "\n",
    "Some python functions help us out here. \n",
    "distfit is a great one for example. It can determine what the best distribution fit would be for data \\\n",
    "This was already performed elsewhere and the results are loaded in below.\n",
    "\n",
    "https://pypi.org/project/distfit/\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "normality_2021 = pd.read_csv('2021_data_df')\n",
    "normality_2022 = pd.read_csv('2022_data_df')\n",
    "normality_2020 = pd.read_csv('2020_data_df')\n",
    "print(('normal',len(normality_2020[normality_2020['Best_Fit_Distribution']=='norm'])),\n",
    "('not_normal',len(normality_2020[normality_2020['Best_Fit_Distribution']!='norm'])),\n",
    "('normal',len(normality_2021[normality_2021['Best_Fit_Distribution']=='norm'])),\n",
    "('not_normal',len(normality_2021[normality_2021['Best_Fit_Distribution']!='norm'])),\n",
    "('normal',len(normality_2022[normality_2022['Best_Fit_Distribution']=='norm'])),\n",
    "('not_normal',len(normality_2022[normality_2022['Best_Fit_Distribution']!='norm'])),sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Since the data are overwhelmingly non-normally distributed.\n",
    "\n",
    "We can use the Spearman's rank-order correlation.\n",
    "Which is the non-parametric version of the pearson product moment correlation test. \\\n",
    "It helps show the strength of the relationship between two variables. \n",
    "With a -1 or a +1 denoting perfect positive or negative relationship. \\\n",
    "So what is needed for this analysis are all of the values that are as close to 0 as possible.\n",
    "\n",
    "\n",
    "## The formula for the Spearman's rank order correlation:\n",
    "\n",
    "$ r_s = \\rho R(X),R(Y)= \\frac{cov(R(X),R(Y))}{\\sigma_{R(X)} \\sigma_{R(Y)}} $\n",
    "\n",
    "\n",
    "$cov(R(X),R(Y)) = {\\text{The covariance of the rank variables}}$ \n",
    "\n",
    "and p denotes the pearson correlation coefficient.\n",
    "\n",
    "And once all of the values are obtained, the data can be filtered leaving only uncorrelated pairs of variables.\n",
    "\n",
    "https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "def find_my_state(states,dataframe_list):\n",
    "\tstate = []\n",
    "\tx =[]\n",
    "\ty=[]\n",
    "\tn=len(dataframe_list)\n",
    "\tfor state in states:\n",
    "\t\tstate = set(state['StateAbbr'].unique())  # Get the unique state name\n",
    "\t\tfor i in range(n):\n",
    "\t\t\tmatches_2020 = dataframe_list[i]\n",
    "\t\t\tsubseta=set(matches_2020['StateAbbr'].unique())\n",
    "\t\t\tif state == subseta:\n",
    "\t\t\t\tmatches_texas = dataframe_list[i]\n",
    "\t\t\t\tsubset1 = (matches_texas['Measure'].unique())\n",
    "\t\t\t\tsubsetb=(matches_texas['StateAbbr'].unique())\n",
    "\t\t\telse:\n",
    "\t\t\t\ti +=1\n",
    "\t\tfor i in range(n):\n",
    "\t\t\tnot_texas = dataframe_list[i] # Otherwise iterate over everything\n",
    "\t\t\tsubsetc = (not_texas['StateAbbr'].unique())\n",
    "\t\t\tsubset2 = (not_texas['Measure'].unique())\n",
    "\t\t\t\n",
    "\t\t\tintersection_of_measures = set(subset1).intersection(set(subset2)) # now the intersection of all 2\t\n",
    "\t\t\tintersection_list = list(intersection_of_measures)\n",
    "\n",
    "\t\t\tfor i in intersection_list:\n",
    "\t\t\t\ty.append([i])\n",
    "\t\t\t\tlis1 = matches_texas.loc[matches_texas['Measure']==i,'Data_Value'] # select all of the values in the dataframe that match\n",
    "\t\t\t\tlis2 = not_texas.loc[not_texas['Measure']==i,'Data_Value']\n",
    "\t\t\t\tif len(lis1) != len(lis2):\n",
    "\t\t\t\t\tmin_len = (min(len(lis1), len(lis2)))\n",
    "\t\t\t\t\t\n",
    "\t\t\t\t\tif len(lis1) == min_len:\n",
    "\t\t\t\t\n",
    "\t\t\t\t\t\tlis2= random.sample(list(lis2), min_len) # Random sample does not use replacement and also makes sure the shorter list length is used\n",
    "\t\t\t\t\t\t\n",
    "\t\t\t\t\tif len(lis2) == min_len:\n",
    "\t\t\t\t\t\tlis1= random.sample(list(lis1), min_len)\n",
    "\t\t\t\t\telse:\n",
    "\t\t\t\t\t\tcontinue\n",
    "\n",
    "\t\t\t\t\tx.append([subsetb,subsetc,stats.spearmanr(lis1, lis2),i])\n",
    "\treturn x\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below code takes a list of the dataframes, and the function. And outputs the filtered p values and such."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_of_states = [TX,CA,PA]\n",
    "yay=find_my_state(list_of_states,dataframes_list2020)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_of_states = [TX,CA,PA]\n",
    "yay1=find_my_state(list_of_states,dataframes_list2021)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_of_states = [TX,CA,PA]\n",
    "yay2=find_my_state(list_of_states,dataframes_list2022)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This code makes the needed df for all specific variables. And cleans up. And filters. \n",
    "df1=pd.DataFrame()\n",
    "def now_make_dataframes(xm):\n",
    "\tdf1 = pd.DataFrame(xm)\n",
    "\tdf1['state'] = df1[0]+df1[1]\n",
    "\tdf1['state'] = df1['state'].astype(str)\n",
    "\tdf1['result_string'] = df1[2].astype(str) # gotta change the kurkwalis to string\n",
    "\tdf1['Corr'] = df1['result_string'].str.extract(r'statistic=([0-9.-]+)').astype(float) # Then just regex it away\n",
    "\tdf1['p-value'] = df1['result_string'].str.extract(r'pvalue=([0-9.eE+-]+)').astype(float)\n",
    "\tdf1['State1'] = df1[0].astype(str).str.extractall(r'\\'([A-Z]{2})\\'').groupby(level=0).apply(lambda x: ','.join(x[0]))\n",
    "\tdf1['State2'] = df1[1].astype(str).str.extractall(r'\\'([A-Z]{2})\\'').groupby(level=0).apply(lambda x: ','.join(x[0]))\n",
    "\tdf1['p-value'] = df1['p-value'].astype(float).apply(lambda x: '{:.10f}'.format(x) if pd.notna(x) else '')\n",
    "\tdf1['p-value'] = pd.to_numeric(df1['p-value'], errors='coerce') # Convert P value to numeric\n",
    "\tdf1['Measure'] = df1[3]\n",
    "\tdf1= df1.drop(['result_string',0,1,2,3],axis=1)\n",
    "\tempty_rows = df1[df1.isna().any(axis=1)]\n",
    "\tdf1 = df1[~df1.index.isin(empty_rows.index)] # Use boolean indexing to get rid of all of the empty values \n",
    "\tfiltered_df = df1[df1['Corr'] < 0.5]\n",
    "\tfiltered_df1 = filtered_df[filtered_df[\"Corr\"]>-0.5]\n",
    "\treturn filtered_df1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df1=now_make_dataframes(yay)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df1.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df2=now_make_dataframes(yay1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df2.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df3=now_make_dataframes(yay2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(filtered_df1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(filtered_df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(filtered_df3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking below if there are any duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "duplicates10 = filtered_df1[filtered_df1.duplicated(['Measure','state'])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "duplicates10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "duplicates10 = filtered_df2[filtered_df2.duplicated(['Measure','state'])]\n",
    "duplicates10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "duplicates10 = filtered_df3[filtered_df3.duplicated(['Measure','state'])]\n",
    "duplicates10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No duplicates is a good sign. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# So the above value can be used to select all of the State combinations that pass correlation test and visualize them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped = filtered_df1.groupby('State2') # So group each by state\n",
    "fig, axes = plt.subplots(nrows=11, ncols=5, figsize=(50, 55))  # Adjust figsize as needed\n",
    "\n",
    "for i, (state, group) in enumerate(grouped):\n",
    "    ax = axes[i // 5, i % 5]  # Calculate the correct subplot based on row and column ( a 10 by 5 basically with extra steps)\n",
    "    ax.hist(group['Corr'], bins=20, edgecolor='k', alpha=0.7)\n",
    "    ax.set_title(f'Distribution of Corr Values for {state}')\n",
    "    ax.set_xlabel('Corr Value')\n",
    "    ax.set_ylabel('Frequency')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped = filtered_df2.groupby('State2') # So group each by state\n",
    "fig, axes = plt.subplots(nrows=11, ncols=5, figsize=(50, 55))  # Adjust figsize as needed\n",
    "\n",
    "for i, (state, group) in enumerate(grouped):\n",
    "    ax = axes[i // 5, i % 5]  # Calculate the correct subplot based on row and column ( a 10 by 5 basically with extra steps)\n",
    "    ax.hist(group['Corr'], bins=20, edgecolor='k', alpha=0.7)\n",
    "    ax.set_title(f'Distribution of Corr Values for {state}')\n",
    "    ax.set_xlabel('Corr Value')\n",
    "    ax.set_ylabel('Frequency')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped = filtered_df3.groupby('State2') # So group each by state\n",
    "fig, axes = plt.subplots(nrows=11, ncols=5, figsize=(50, 55))  # Adjust figsize as needed\n",
    "\n",
    "for i, (state, group) in enumerate(grouped):\n",
    "    ax = axes[i // 5, i % 5]  # Calculate the correct subplot based on row and column ( a 10 by 5 basically with extra steps)\n",
    "    ax.hist(group['Corr'], bins=20, edgecolor='k', alpha=0.7)\n",
    "    ax.set_title(f'Distribution of Corr Values for {state}')\n",
    "    ax.set_xlabel('Corr Value')\n",
    "    ax.set_ylabel('Frequency')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# So given the original question: Do states with more places data have different metrics in general?\n",
    "\n",
    "\n",
    "The null hypothesis is:\n",
    "$H_0$ :$\\mu_1-\\mu_2 =0$ \n",
    "\n",
    "The alternative hypothesis is:\n",
    "$H_0$ :$\\mu_1-\\mu_2 \\neq0 $ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Not sure why I have this barplot any more the values are the same\n",
    "years = ['2020', '2021', '2022']\n",
    "values = [28329, 28329, 28329]\n",
    "\n",
    "\n",
    "plt.bar(years, values, color='green')\n",
    "\n",
    "plt.xlabel('Year')\n",
    "plt.ylabel('Values')\n",
    "plt.title('Sum of all cities in PLACES Dataset per year')\n",
    "\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "city_counts1 = s2020.groupby('StateAbbr')['LocationName'].nunique()\n",
    "city_counts1 = city_counts1.sort_values(ascending=False)\n",
    "city_counts2 = s2021.groupby('StateAbbr')['LocationName'].nunique()\n",
    "city_counts2 = city_counts2.sort_values(ascending=False)\n",
    "city_counts3 = s2022.groupby('StateAbbr')['LocationName'].nunique()\n",
    "city_counts3 = city_counts3.sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "city_counts1['CA']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "city_counts1['PA']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "city_counts1['TX']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plots for Locations per year do not change very much between each data set.\n",
    "They are visualized below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(city_counts1,palette=\"dark:#5A9_r\")\n",
    "\n",
    "\n",
    "ax.set(xlabel=\"State\",ylabel =\"Number of locations \",title=\"Locations per State 2017-2018\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(city_counts2,palette=\"dark:#5A9_r\")\n",
    "\n",
    "ax.set(xlabel=\"State\",ylabel =\"Number of locations \",title=\"Locations per State 2019-2020\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(city_counts3,palette=\"dark:#5A9_r\")\n",
    "\n",
    "ax.set(xlabel=\"State\",ylabel =\"Number of locations \",title=\"Locations per State 2020-2021\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kruskal-Wallis H Test \n",
    "\n",
    "The Kruskal-Wallis H-test tests whether the null hypothesis that the population medians or means of all of groups are equal. It is a non-parametric version of ANOVA. The test works on 2 or more independent samples, which may have different sizes. Note that rejecting the null hypothesis does not indicate which of the groups differs. Post hoc comparisons between groups are required to determine which groups are different.*\n",
    "\n",
    "\n",
    "The test statistic is calculated using the below formula:\n",
    "\n",
    "$ H = \\frac{12}{n(n+1)} \\sum_{i=1}^{k} \\frac{R^2}{n_i} -3(n-1)$\n",
    "\n",
    "\n",
    "Where _ni_ is the sample size for each of groups (1,2,3...k) from 1 to k. Then the rank is computed (Ri), \\\n",
    "The resulting statistic is similiar to a chi-squared distribution with k-1 degrees of freedom.\\\n",
    "If the p value is less than alpha which is 0.05, we reject the null hypothesis. That the means/medians of the states\\\n",
    "with more places data differ from other states. **\n",
    "\n",
    "\n",
    "https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kruskal.html * \\\n",
    "https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/kruskwal.htm **\n",
    "\n",
    "\n",
    "For the code below. I take each state index, match the same states in the other dataframes and then run all stats kruskal tests on the variables. This test can also take in arrays of different lengths so I commented out the random sampling to make them the same. \n",
    "I need to see if the medians of the groups are rejected more for the cities with more places data than others. Over years. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df1.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_kruskal(states,dataframe_list):\n",
    "\tstate = []\n",
    "\tx =[]\n",
    "\ty=[]\n",
    "\tn=len(dataframe_list)\n",
    "\tfor state in states:\n",
    "\t\tstate = set(state['StateAbbr'].unique())  # Get the unique state name\n",
    "\t\tfor i in range(n):\n",
    "\t\t\tmatches_2020 = dataframe_list[i]\n",
    "\t\t\tsubseta=set(matches_2020['StateAbbr'].unique())\n",
    "\t\t\tif state == subseta:\n",
    "\t\t\t\tmatches_texas = dataframe_list[i]\n",
    "\t\t\t\tsubset1 = (matches_texas['Measure'].unique())\n",
    "\t\t\t\tsubsetb=(matches_texas['StateAbbr'].unique())\n",
    "\t\t\telse:\n",
    "\t\t\t\ti +=1\n",
    "\t\tfor i in range(n):\n",
    "\t\t\tnot_texas = dataframe_list[i] # Otherwise iterate over everything\n",
    "\t\t\tsubsetc = (not_texas['StateAbbr'].unique())\n",
    "\t\t\tsubset2 = (not_texas['Measure'].unique())\n",
    "\t\t\t\n",
    "\t\t\tintersection_of_measures = set(subset1).intersection(set(subset2)) # now the intersection of all 2\t\n",
    "\t\t\tintersection_list = list(intersection_of_measures)\n",
    "\n",
    "\t\t\tfor i in intersection_list:\n",
    "\t\t\t\ty.append([i])\n",
    "\t\t\t\tlis1 = matches_texas.loc[matches_texas['Measure']==i,'Data_Value'] # select all of the values in the dataframe that match\n",
    "\t\t\t\tlis2 = not_texas.loc[not_texas['Measure']==i,'Data_Value']\n",
    "\t\t\t\t#if len(lis1) != len(lis2):\n",
    "\t\t\t\t\t#min_len = (min(len(lis1), len(lis2)))\n",
    "\t\t\t\t\t\n",
    "\t\t\t\t\t#if len(lis1) == min_len:\n",
    "\t\t\t\t\n",
    "\t\t\t\t\t\t#lis2= random.sample(list(lis2), min_len) # Random sample does not use replacement and also makes sure the shorter list length is used\n",
    "\t\t\t\t\t\t\n",
    "\t\t\t\t\t#if len(lis2) == min_len:\n",
    "\t\t\t\t\t\t#lis1= random.sample(list(lis1), min_len)\n",
    "\t\t\t\t\t#else:\n",
    "\t\t\t\t\t\t#continue\n",
    "\n",
    "\t\t\t\tx.append([subsetb,subsetc,stats.stats.kruskal(lis1, lis2),i])\n",
    "\treturn x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "states_list_1 = [TX,PA,CA]\n",
    "yay5=run_kruskal(states_list_1,dataframes_list2020)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "states_list_2 = [TX,PA,CA]\n",
    "yay6=run_kruskal(states_list_2,dataframes_list2021)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "states_list_3 = [TX,PA,CA]\n",
    "yay7=run_kruskal(states_list_3,dataframes_list2022)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yay5[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This code makes the needed df for all specific variables. And cleans up. And filters. \n",
    "df1=pd.DataFrame()\n",
    "def now_make_dataframes(xm):\n",
    "\tdf1 = pd.DataFrame(xm)\n",
    "\tdf1['state'] = df1[0]+df1[1]\n",
    "\tdf1['state'] = df1['state'].astype(str)\n",
    "\tdf1['result_string'] = df1[2].astype(str) # gotta change the kurkwalis to string\n",
    "\tdf1['Statistic'] = df1['result_string'].str.extract(r'statistic=([0-9.-]+)').astype(float) # Then just regex it away\n",
    "\tdf1['p-value'] = df1['result_string'].str.extract(r'pvalue=([0-9.eE+-]+)').astype(float)\n",
    "\tdf1['State1'] = df1[0].astype(str).str.extractall(r'\\'([A-Z]{2})\\'').groupby(level=0).apply(lambda x: ','.join(x[0]))\n",
    "\tdf1['State2'] = df1[1].astype(str).str.extractall(r'\\'([A-Z]{2})\\'').groupby(level=0).apply(lambda x: ','.join(x[0]))\n",
    "\tdf1['p-value'] = df1['p-value'].astype(float).apply(lambda x: '{:.10f}'.format(x) if pd.notna(x) else '')\n",
    "\tdf1['p-value'] = pd.to_numeric(df1['p-value'], errors='coerce') # Convert P value to numeric\n",
    "\tdf1['Measure'] = df1[3]\n",
    "\tdf1= df1.drop(['result_string',0,1,2,3],axis=1)\n",
    "\tempty_rows = df1[df1.isna().any(axis=1)]\n",
    "\tdf1 = df1[~df1.index.isin(empty_rows.index)] # Use boolean indexing to get rid of all of the empty values \n",
    "\treturn df1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df_kruskal_2020=now_make_dataframes(yay5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df_kruskal_2021=now_make_dataframes(yay6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df_kruskal_2022=now_make_dataframes(yay7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shorted_df1 = filtered_df1.drop(columns=['Corr','p-value','State1','State2'],axis=1)\n",
    "shorted_df2 = filtered_df2.drop(columns=['Corr','p-value','State1','State2'],axis=1)\n",
    "shorted_df3 = filtered_df3.drop(columns=['Corr','p-value','State1','State2'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(shorted_df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_2020_kruskal = filtered_df_kruskal_2020.merge(shorted_df1, how='inner', on=['Measure','state'])\n",
    "len(result_2020_kruskal)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_2021_kruskal = filtered_df_kruskal_2021.merge(shorted_df2, how='inner', on=['Measure','state'])\n",
    "len(result_2021_kruskal)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_2022_kruskal = filtered_df_kruskal_2022.merge(shorted_df3, how='inner', on=['Measure','state'])\n",
    "len(result_2022_kruskal)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_df3.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "So to see if Texas for example differed more than any other state, I need to sum all of the times texas occurs and then sum all of the other states. The above filtered DF is where there is a difference from the median. And if texas occurs more, then.. it occurs more. It does not answer the question on if the data is higher or lower. Just that a difference exists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(filtered_df1[filtered_df1['p-value']<0.05])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(filtered_df1[filtered_df1['p-value']>0.05])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(filtered_df2[filtered_df2['p-value']<0.05])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(filtered_df2[filtered_df2['p-value']>0.05])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(filtered_df3[filtered_df3['p-value']<0.05])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(filtered_df3[filtered_df3['p-value']>0.05])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many no values but enough yes values to warrant post hoc analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Start of the Dunn test \n",
    "\n",
    "The Dunn test will tell us which values differ based on pairs. \\\n",
    "It is a post hoc analysis often performed after and\\\n",
    "using the mean rankings of the outcome of the Kruskal-Wallis test.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_of_measures_k=list(s2022['Measure'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#filtered_df_kruskal_2020.head(1)\n",
    "states=[CA,PA,TX]\n",
    "n=len(dataframes_list2022)\n",
    "x=[]\n",
    "b=[]\n",
    "r=[]\n",
    "for k in list_of_measures_k:\n",
    "\tresult1 = s2020[s2020['Measure'] == k]\n",
    "\tfor state in states:\n",
    "\t\tstate = set(state['StateAbbr'].unique())  # Get the unique state name\n",
    "\t\tfor i in range(n):\n",
    "\t\t\tmatches_2020 = dataframes_list2020[i]\n",
    "\t\t\tsubseta=set(matches_2020['StateAbbr'].unique())\n",
    "\t\t\tif state == subseta:\n",
    "\t\t\t\tmatches_texas = dataframes_list2020[i]\n",
    "\t\t\t\tsubset1 = (matches_texas['Measure'].unique())\n",
    "\t\t\t\tsubsetb=(matches_texas['StateAbbr'].unique())\n",
    "\t\t\telse:\n",
    "\t\t\t\ti +=1\n",
    "\t\tfor i in range(n):\n",
    "\t\t\tnot_texas = dataframes_list2020[i] # Otherwise iterate over everything\n",
    "\t\t\tsubsetc = (not_texas['StateAbbr'].unique())\n",
    "\t\t\tsubset2 = (not_texas['Measure'].unique())\n",
    "\t\t\t\n",
    "\t\t\tintersection_of_measures = set(subset1).intersection(set(subset2)) # now the intersection of all 2\t\n",
    "\t\t\tintersection_list = list(intersection_of_measures)\n",
    "\t\t\tfor i in intersection_list:\n",
    "\t\t\t\ty.append([i])\n",
    "\t\t\t\tlis1 = matches_texas.loc[matches_texas['Measure']==i,'Data_Value'] # select all of the values in the dataframe that match\n",
    "\t\t\t\tlis2 = not_texas.loc[not_texas['Measure']==i,'Data_Value']\n",
    "\t\t\t\tif len(lis1) != len(lis2):\n",
    "\t\t\t\t\tmin_len = (min(len(lis1), len(lis2)))\n",
    "\t\t\t\t\t\n",
    "\t\t\t\t\tif len(lis1) == min_len:\n",
    "\t\t\t\t\n",
    "\t\t\t\t\t\tlis2= random.sample(list(lis2), min_len) # Random sample does not use replacement and also makes sure the shorter list length is used\n",
    "\t\t\t\t\t\t\n",
    "\t\t\t\t\tif len(lis2) == min_len:\n",
    "\t\t\t\t\t\tlis1= random.sample(list(lis1), min_len)\n",
    "\t\t\t\t\telse:\n",
    "\t\t\t\t\t\tcontinue\n",
    "\t\t\tdata=[lis1,lis2]\n",
    "\t\t\tr.append(subsetc)\n",
    "\t\t\tz=sp.posthoc_dunn(data,p_adjust = 'bonferroni')\n",
    "\t\t\tb.append(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "okay=[]\n",
    "for i in range(len(b)):\n",
    "    okay.append(b[i][2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "woo=pd.DataFrame(okay)\n",
    "woo['states']=r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results of my testing are below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(woo[woo[1]<0.05])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(woo[woo[1]>0.05])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Updated Hypothesis\n",
    "\n",
    "So using the above code and visualizing it. It is easy to see that overall States with more PLACES locations\\\n",
    "differ from the mean/median when using pairwise post-hoc analysis. Which is not the most clear conclusion.\\\n",
    "Given these results, it might be a good idea to perform some other testing. And the hypothesis will now change slightly.\n",
    "Maybe a more specific type of question will yield better answers.\n",
    "Do the states with more PLACES locations have higher means in terms of Measure results than States with fewer datapoints?\n",
    "\n",
    "So the new hypothesis is \\\n",
    "$H_0$ :$\\mu_1-\\mu_2 =0$ \\\n",
    "No difference in the means. \n",
    "\n",
    "Or\n",
    "\n",
    "$H_1$ :$\\mu_1-\\mu_2 >0 $ \\\n",
    "Meaning that the states with more PLACES data have higher metrics. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# It seems a parametric set of tests will be required after all. \n",
    "But how can parametric tests be used on this data?\n",
    "With large enough sample sizes (> 30 or 40), the violation of the normality assumption should not cause major problems * \\\n",
    "And each sample size is a few hundred. So all measures and States satisfy this rule. \\\n",
    "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3693611/*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Levene's Test\n",
    "Going to use the Levene test to see if samples come from equal variance distributions.\n",
    "If the resulting p-value of the Levene's tests are less than some significance level (typically 0.05),\\\n",
    "the obtained differences in sample variances are unlikely to have occurred based on random sampling from a \n",
    "population with equal variances. Thus, the null hypothesis of equal variances is rejected and it is concluded that there is a difference \\\n",
    "between variances in the population. \n",
    "The scipy stats Levene test library was used to perform the Levene Test.\n",
    "\n",
    "\n",
    "To discern whether:\\\n",
    "The null hypothesis that there is no difference in variances is true: \\\n",
    "$ H_0: \\sigma_1^{2} = \\sigma_2^{2} = \\sigma_k^{2}$\n",
    "\n",
    "or\n",
    "\n",
    "That variances are not equal (rejecting the null hypothesis)\n",
    "\n",
    "$H_0:  \\sigma_1^{2} \\neq \\sigma_2^{2} \\neq \\sigma_k^{2} $\n",
    "\n",
    "\n",
    "The test statistic is written as:\n",
    "\n",
    "$ W = \\frac{(N-k)}{(k-1)}\\frac{\\sum_{i=1}^k N_i (Z_i-Z)^2}{\\sum_{i=1}^k \\sum_{j=1}^{N_i}(Z_{ij}-Z_i)^2}$\n",
    "\n",
    "* $k$ being the number of groups to which the sampled cases belong\n",
    "* $N_i$ is the number of cases in the ith group\n",
    "* $N$ is the total number of cases in all groups\n",
    "* $Y_{ij}$ is the variable from the jth case from the ith group\n",
    "* $Z_{ij}$ is either the mean or median of each group\n",
    "* $Z$ is each individual value\n",
    "\n",
    "https://en.wikipedia.org/wiki/Levene%27s_test\n",
    "\n",
    "https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.levene.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_levenes(states,dataframe_list):\n",
    "\tstate = []\n",
    "\tx =[]\n",
    "\ty=[]\n",
    "\tn=len(dataframe_list)\n",
    "\tfor state in states:\n",
    "\t\tstate = set(state['StateAbbr'].unique())  # Get the unique state name\n",
    "\t\tfor i in range(n):\n",
    "\t\t\tmatches_2020 = dataframe_list[i]\n",
    "\t\t\tsubseta=set(matches_2020['StateAbbr'].unique())\n",
    "\t\t\tif state == subseta:\n",
    "\t\t\t\tmatches_texas = dataframe_list[i]\n",
    "\t\t\t\tsubset1 = (matches_texas['Measure'].unique())\n",
    "\t\t\t\tsubsetb=(matches_texas['StateAbbr'].unique())\n",
    "\t\t\telse:\n",
    "\t\t\t\ti +=1\n",
    "\t\tfor i in range(n):\n",
    "\t\t\tnot_texas = dataframe_list[i] # Otherwise iterate over everything\n",
    "\t\t\tsubsetc = (not_texas['StateAbbr'].unique())\n",
    "\t\t\tsubset2 = (not_texas['Measure'].unique())\n",
    "\t\t\t\n",
    "\t\t\tintersection_of_measures = set(subset1).intersection(set(subset2)) # now the intersection of all 2\t\n",
    "\t\t\tintersection_list = list(intersection_of_measures)\n",
    "\n",
    "\t\t\tfor i in intersection_list:\n",
    "\t\t\t\ty.append([i])\n",
    "\t\t\t\tlis1 = matches_texas.loc[matches_texas['Measure']==i,'Data_Value'] # select all of the values in the dataframe that match\n",
    "\t\t\t\tlis2 = not_texas.loc[not_texas['Measure']==i,'Data_Value']\n",
    "\t\t\t\t#if len(lis1) != len(lis2):\n",
    "\t\t\t\t\t#min_len = (min(len(lis1), len(lis2)))\n",
    "\t\t\t\t\t\n",
    "\t\t\t\t\t#if len(lis1) == min_len:\n",
    "\t\t\t\t\n",
    "\t\t\t\t\t\t#lis2= random.sample(list(lis2), min_len) # Random sample does not use replacement and also makes sure the shorter list length is used\n",
    "\t\t\t\t\t\t\n",
    "\t\t\t\t\t#if len(lis2) == min_len:\n",
    "\t\t\t\t\t\t#lis1= random.sample(list(lis1), min_len)\n",
    "\t\t\t\t\t#else:\n",
    "\t\t\t\t\t\t#continue\n",
    "\n",
    "\t\t\t\tx.append([subsetb,subsetc,stats.levene(lis1, lis2),i])\n",
    "\treturn x\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_for_levene = [TX,PA,CA]\n",
    "trying_2020 = run_levenes(list_for_levene, dataframes_list2020)\n",
    "trying_2021 = run_levenes(list_for_levene, dataframes_list2021)\n",
    "trying_2022 = run_levenes(list_for_levene, dataframes_list2022)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1=pd.DataFrame()\n",
    "def now_make_dataframes(xm):\n",
    "\tdf1 = pd.DataFrame(xm)\n",
    "\tdf1['state'] = df1[0]+df1[1]\n",
    "\tdf1['state'] = df1['state'].astype(str)\n",
    "\tdf1['result_string'] = df1[2].astype(str) # gotta change the kurkwalis to string\n",
    "\tdf1['Statistic'] = df1['result_string'].str.extract(r'statistic=([0-9.-]+)').astype(float) # Then just regex it away\n",
    "\tdf1['p-value'] = df1['result_string'].str.extract(r'pvalue=([0-9.eE+-]+)').astype(float)\n",
    "\tdf1['State1'] = df1[0].astype(str).str.extractall(r'\\'([A-Z]{2})\\'').groupby(level=0).apply(lambda x: ','.join(x[0]))\n",
    "\tdf1['State2'] = df1[1].astype(str).str.extractall(r'\\'([A-Z]{2})\\'').groupby(level=0).apply(lambda x: ','.join(x[0]))\n",
    "\tdf1['p-value'] = df1['p-value'].astype(float).apply(lambda x: '{:.10f}'.format(x) if pd.notna(x) else '')\n",
    "\tdf1['p-value'] = pd.to_numeric(df1['p-value'], errors='coerce') # Convert P value to numeric\n",
    "\tdf1['Measure'] = df1[3]\n",
    "\tdf1= df1.drop(['result_string',0,1,2,3],axis=1)\n",
    "\tempty_rows = df1[df1.isna().any(axis=1)]\n",
    "\tdf1 = df1[~df1.index.isin(empty_rows.index)] # Use boolean indexing to get rid of all of the empty values \n",
    "\treturn df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "levene_attempt2020 = now_make_dataframes(trying_2020)\n",
    "levene_attempt2021 = now_make_dataframes(trying_2021)\n",
    "levene_attempt2022 = now_make_dataframes(trying_2022)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "levene_attempt2021.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shorted_df1 = filtered_df1.drop(columns=['Corr','p-value','State1','State2'],axis=1)\n",
    "shorted_df2 = filtered_df2.drop(columns=['Corr','p-value','State1','State2'],axis=1)\n",
    "shorted_df3 = filtered_df3.drop(columns=['Corr','p-value','State1','State2'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_2020_levene = levene_attempt2020.merge(shorted_df1, how='right', on=['Measure','state'])\n",
    "result_2021_levene = levene_attempt2021.merge(shorted_df2, how='right', on=['Measure','state'])\n",
    "result_2022_levene = levene_attempt2022.merge(shorted_df3, how='right', on=['Measure','state'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(result_2020_levene)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(result_2021_levene)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(result_2022_levene)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# So we have the results of the Levene test for all years and the top states. \n",
    "Which will tell us if the values come from equal or unequal variances.\\\n",
    "So now we can run T_tests on the samples.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T test starts Below\n",
    "A T-test compares averages of two groups, determening whether differences between them are due to chance or not. \\\n",
    "We can apply this here due to the fact that given a large enough sample size normality can be assumed. \\\n",
    "The scipy.stats library will be used to perform the test in question.\n",
    "\n",
    "$H_0$ :$\\mu_1-\\mu_2 =0$ \\\n",
    "No difference in the means. \n",
    "\n",
    "Or\n",
    "\n",
    "$H_1$ :$\\mu_1-\\mu_2 >0 $ \n",
    "\n",
    "Whether the averages for PA,TX,CA PLACES data are higher overall than other States.\n",
    "\n",
    "\n",
    "https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html\n",
    "\n",
    "https://en.wikipedia.org/wiki/Student%27s_t-test#Independent_two-sample_t-test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T-Test for equal variances\n",
    "\n",
    "Using the Levene results above we can filter out the samples that have an alpha >0.05 and \\\n",
    "perform the version of the test where sample sizes may vary but variances are similiar/equal. \n",
    "\n",
    "The t-statistic is below: \n",
    "\n",
    "\n",
    "$ t= \\frac{\\bar{X_1}-\\bar{X_2}}{s_p * \\sqrt(\\frac{1}{n_1} + \\frac{1}{n_2})}$\n",
    "\n",
    "* $\\bar{X_1}$ is the mean of sample 1\n",
    "* $\\bar{X_2}$ is the mean of sample 2\n",
    "* $n_1$ is the sample size of sample 1\n",
    "* $n_2$ is the sample size of sample 2\n",
    "* $s_p$ is the pooled variance calculated by:\n",
    "$s_p = \\sqrt\\frac{(n_1 -1)s_{X_1}^2+(n_2 -1)s_{X_2}^2}{n_1+n_2-2}$\n",
    "* $s_{X_1}^2$ is the variance for sample 1\n",
    "* $s_{X_2}^2$ is the variance for sample 2\n",
    "* ${n_1+n_2-2}$ calculates the degrees of freedom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def running_t_equal_var(levene_results,dataframe):\n",
    "\tx=[]\n",
    "\ty=[]\n",
    "\tdefg=[]\n",
    "\tprints=[]\n",
    "\tcurrent=[]\n",
    "\tdefg=levene_results[levene_results['p-value'] >0.05 ]\n",
    "\tprints=defg[['Measure','State1','State2']]\n",
    "\tn = len(prints)\n",
    "\tcurrent = prints.values.tolist()\n",
    "\tfor i in range(n):  # For i in range n, set the current list parts to separate variables.\n",
    "\t\tcurrent1 = current[i]\n",
    "\t\tmeasure=(current1[0])\n",
    "\n",
    "\t\tstate1=(current1[1]) # first second, third element\n",
    "\t\tstate2=(current1[2].strip()) # Need to strip random whitesapce\n",
    "\n",
    "\n",
    "\t\tcondition1 = dataframe.loc[dataframe['StateAbbr']==state1]\n",
    "\t\t#print(len(condition1))\n",
    "\t\tlis1 = condition1.loc[condition1['Measure']==measure,'Data_Value'] # For a dataframe .loc seems to be the simplest for performing this work\n",
    "\t\tcondition2 = dataframe.loc[dataframe['StateAbbr']==state2]\n",
    "\t\t#print(len(condition2))\n",
    "\t\tlis2 = condition2.loc[condition2['Measure']==measure,'Data_Value']\n",
    "\t\ty.append([state1,state2])\n",
    "\t\tif len(lis1) != len(lis2):\n",
    "\t\t\tmin_len = (min(len(lis1), len(lis2)))\n",
    "\t\t\t\n",
    "\t\t\tif len(lis1) == min_len:\n",
    "\n",
    "\t\t\t\tlis2= random.sample(list(lis2), min_len) # Random sample does not use replacement\n",
    "\t\t\t\t\n",
    "\t\t\tif len(lis2) == min_len:\n",
    "\t\t\t\tlis1= random.sample(list(lis1), min_len)\n",
    "\t\telse:\n",
    "\t\t\tcontinue\n",
    "\t\t\t#print(len(lis1),len(lis2))\n",
    "\t\t#print(stats.ttest_ind(lis1, lis2,equal_var=True,alternative='greater'),i)\n",
    "\t\tx.append([state1,state2,measure,stats.ttest_ind(lis1, lis2,equal_var=True,alternative='greater'),i]) # Make sure to set the alternative to greater\n",
    "\t\t#print(\"Total prints:\", print_count)\n",
    "\t\t#result1 = list(list(t) for t in zip(x, y)) # now just zip all of the lists together\n",
    "\treturn x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "defg=result_2022_levene[result_2022_levene['p-value'] >0.05 ]\n",
    "len(defg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "defg=result_2022_levene[result_2022_levene['p-value'] <0.05 ]\n",
    "len(defg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trying_t_test_2020=running_t_equal_var(result_2020_levene,s2020)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "trying_t_test_2021=running_t_equal_var(result_2021_levene,s2021)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trying_t_test_2022=running_t_equal_var(result_2022_levene,s2022)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1=pd.DataFrame()\n",
    "def now_make_dataframes1(xm):\n",
    "\tdf1=pd.DataFrame(xm)\n",
    "\tdf1['state'] = df1[0]+df1[1]\n",
    "\tdf1['state'] = df1['state'].astype(str)\n",
    "\tdf1['result_string'] = df1[3].astype(str) # gotta change the kurkwalis to string\n",
    "\tdf1['Statistic'] = df1['result_string'].str.extract(r'statistic=([0-9.-]+)').astype(float) # Then just regex it away\n",
    "\tdf1['p-value'] = df1['result_string'].str.extract(r'pvalue=([0-9.eE+-]+)').astype(float)\n",
    "\tdf1['State1'] = df1[0].astype(str)\n",
    "\tdf1['State2'] = df1[1].astype(str)\n",
    "\tdf1['p-value'] = df1['p-value'].astype(float).apply(lambda x: '{:.10f}'.format(x) if pd.notna(x) else '')\n",
    "\tdf1['p-value'] = pd.to_numeric(df1['p-value'], errors='coerce') # Convert P value to numeric\n",
    "\tdf1['Measure'] = df1[2].astype(str)\n",
    "\tdf1['Measure'] = df1['Measure'].str.strip()\n",
    "\tdf1= df1.drop(['result_string',0,1,2,3,4],axis=1)\n",
    "\tempty_rows = df1[df1.isna().any(axis=1)]\n",
    "\tdf1 = df1[~df1.index.isin(empty_rows.index)]\n",
    "\treturn df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_test_equal_var_2020=now_make_dataframes1(trying_t_test_2020)\n",
    "t_test_equal_var_2021=now_make_dataframes1(trying_t_test_2021)\n",
    "t_test_equal_var_2022=now_make_dataframes1(trying_t_test_2022)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(t_test_equal_var_2020)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T-test  for unequal variances\n",
    "\n",
    "When variances are unequal, Welch's T-Test can be used.\n",
    "\n",
    "$ t= \\frac{\\bar{X_1}-\\bar{X_2}}{s_{\\bar\\Delta}}$\n",
    "\n",
    "with \n",
    "$s_{\\bar\\Delta} =\\sqrt{\\frac{s_1^2}{n_1}+\\frac{s_2^2}{n_2}}$\n",
    "\n",
    "\n",
    "*$s_{\\bar\\Delta}$ being the difference between the means and is not a pooled variance in this case\n",
    "* $n_1$ is the sample size of sample 1\n",
    "* $n_2$ is the sample size of sample 2\n",
    "\n",
    "and the degrees of freedom are calculated using:\n",
    "\n",
    "$d.f.= \\frac{(\\frac{s_{1}^2}{n_1} + \\frac{s_{2}^2}{n_2})^2}{\\frac{(s_{_1}^2/n_1)^2}{n_1-1}+\\frac{(s_{_2}^2/n_2)^2}{n_2-1}}$\n",
    "\n",
    "* $s_{1}^2$ is the unbiased estimator of the variance of sample 1 \n",
    "* $s_{2}^2$ is the unbiased estimator of the variance of sample 2\n",
    "\n",
    "https://en.wikipedia.org/wiki/Student%27s_t-test#Unequal_variances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def running_t_unequal_var(levene_results,dataframe):\n",
    "\tx=[]\n",
    "\ty=[]\n",
    "\tdefg=[]\n",
    "\tprints=[]\n",
    "\tdefg=levene_results[levene_results['p-value'] <0.05 ]\n",
    "\tprints=defg[['Measure','State1','State2']]\n",
    "\tn = len(prints)\n",
    "\tcurrent = prints.values.tolist()\n",
    "\tfor i in range(n):  # For i in range n, set the current list parts to separate variables.\n",
    "\t\tcurrent1 = current[i]\n",
    "\t\tmeasure=(current1[0])\n",
    "\n",
    "\t\tstate1=(current1[1]) # first second, third element\n",
    "\t\tstate2=(current1[2].strip()) # Need to strip random whitesapce\n",
    "\n",
    "\n",
    "\t\tcondition1 = dataframe.loc[dataframe['StateAbbr']==state1]\n",
    "\t\t#print(len(condition1))\n",
    "\t\tlis1 = condition1.loc[condition1['Measure']==measure,'Data_Value'] # For a dataframe .loc seems to be the simplest for performing this work\n",
    "\t\tcondition2 = dataframe.loc[dataframe['StateAbbr']==state2]\n",
    "\t\t#print(len(condition2))\n",
    "\t\tlis2 = condition2.loc[condition2['Measure']==measure,'Data_Value']\n",
    "\t\ty.append([state1,state2])\n",
    "\t\tif len(lis1) != len(lis2):\n",
    "\t\t\tmin_len = (min(len(lis1), len(lis2)))\n",
    "\t\t\t\n",
    "\t\t\tif len(lis1) == min_len:\n",
    "\n",
    "\t\t\t\tlis2= random.sample(list(lis2), min_len) # Random sample does not use replacement\n",
    "\t\t\t\t\n",
    "\t\t\tif len(lis2) == min_len:\n",
    "\t\t\t\tlis1= random.sample(list(lis1), min_len)\n",
    "\t\telse:\n",
    "\t\t\tcontinue\n",
    "\t\t\t#print(len(lis1),len(lis2))\n",
    "\t\t#print(stats.ttest_ind(lis1, lis2,equal_var=True,alternative='greater'),i)\n",
    "\t\tx.append([state1,state2,measure,stats.ttest_ind(lis1, lis2,equal_var=False,alternative='greater'),i]) # Make sure to set the alternative to greater\n",
    "\t\t#print(\"Total prints:\", print_count)\n",
    "\t\t#result1 = list(list(t) for t in zip(x, y)) # now just zip all of the lists together\n",
    "\treturn x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trying_t_test_2020_u=running_t_unequal_var(result_2020_levene,s2020)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trying_t_test_2021_u=running_t_unequal_var(result_2021_levene,s2021)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "trying_t_test_2022_u=running_t_unequal_var(result_2022_levene,s2022)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_test_unequal_var_2020=now_make_dataframes1(trying_t_test_2020_u)\n",
    "t_test_unequal_var_2021=now_make_dataframes1(trying_t_test_2021_u)\n",
    "t_test_unequal_var_2022=now_make_dataframes1(trying_t_test_2022_u)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(t_test_unequal_var_2020)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualzations for results-equal variances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "texas=t_test_equal_var_2020[t_test_equal_var_2020['State1']=='TX']\n",
    "ca=t_test_equal_var_2020[t_test_equal_var_2020['State1']=='CA']\n",
    "pa=t_test_equal_var_2020[t_test_equal_var_2020['State1']=='PA']\n",
    "texas1=t_test_equal_var_2021[t_test_equal_var_2021['State1']=='TX']\n",
    "ca1=t_test_equal_var_2021[t_test_equal_var_2021['State1']=='CA']\n",
    "pa1=t_test_equal_var_2021[t_test_equal_var_2021['State1']=='PA']\n",
    "texas2=t_test_equal_var_2022[t_test_equal_var_2022['State1']=='TX']\n",
    "ca2=t_test_equal_var_2022[t_test_equal_var_2022['State1']=='CA']\n",
    "pa2=t_test_equal_var_2022[t_test_equal_var_2022['State1']=='PA']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=texas['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = texas[texas['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] <= 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['TX',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS Texas',\n",
    "       title='Resulting P values'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=texas1['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = texas1[texas1['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] <= 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['TX',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS Texas',\n",
    "       title='count of p_values that are significant per state VS Texas'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=texas2['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = texas2[texas2['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] <= 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['TX',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS Texas',\n",
    "       title='Resulting P values'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=ca['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = ca[ca['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] <= 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['CA',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS California',\n",
    "       title='Resulting P values'\n",
    "       )\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=ca1['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = ca1[ca1['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] <= 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['CA',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS California',\n",
    "       title='Resulting P values'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=ca2['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = ca2[ca2['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] <= 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['CA',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS California',\n",
    "       title='Resulting P values'\n",
    "       )\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=pa['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = pa[pa['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] <= 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['PA',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS California',\n",
    "       title='Resulting P values'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=pa1['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = pa1[pa1['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] <= 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['PA',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS California',\n",
    "       title='Resulting P values'\n",
    "       )\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=pa2['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = pa2[pa2['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] <= 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['PA',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS California',\n",
    "       title='Resulting P values'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# The below is for unequal var\n",
    "visualize the p values for texas in front of (or behind) every other state\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_test_unequal_var_2020"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "texas=t_test_unequal_var_2020[t_test_unequal_var_2020['State1']=='TX']\n",
    "ca=t_test_unequal_var_2020[t_test_unequal_var_2020['State1']=='CA']\n",
    "pa=t_test_unequal_var_2020[t_test_unequal_var_2020['State1']=='PA']\n",
    "texas3=t_test_unequal_var_2021[t_test_unequal_var_2021['State1']=='TX']\n",
    "ca3=t_test_unequal_var_2021[t_test_unequal_var_2021['State1']=='CA']\n",
    "pa3=t_test_unequal_var_2021[t_test_unequal_var_2021['State1']=='PA']\n",
    "texas4=t_test_unequal_var_2022[t_test_unequal_var_2022['State1']=='TX']\n",
    "ca4=t_test_unequal_var_2022[t_test_unequal_var_2022['State1']=='CA']\n",
    "pa4=t_test_unequal_var_2022[t_test_unequal_var_2022['State1']=='PA']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=texas['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = texas[texas['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] < 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['TX',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df10.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS TX',\n",
    "       title='Resulting P values'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=texas3['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = texas3[texas3['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] < 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['TX',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS TX',\n",
    "       title='Resulting P values'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=texas4['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = texas4[texas4['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] < 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['TX',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS TX',\n",
    "       title='Resulting P values'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=ca['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = ca[ca['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] < 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['CA',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS CA',\n",
    "       title='Resulting P values'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=ca3['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = ca3[ca3['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] < 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['CA',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS CA',\n",
    "       title='Resulting P values'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=ca4['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = ca4[ca4['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] < 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['CA',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS CA',\n",
    "       title='Resulting P values'\n",
    "       )\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=pa['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = pa[pa['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] < 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['PA',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS PA',\n",
    "       title='Resulting P values'\n",
    "       )\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=pa3['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = pa3[pa3['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] < 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['PA',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS PA',\n",
    "       title='Resulting P values'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prints=pa4['State2']  # get the state all by itself\n",
    "prints=prints.unique() \n",
    "lists1=[]\n",
    "for i in prints:   \n",
    "\tabc = pa4[pa4['State2']==i]\n",
    "\tabcd = abc[abc['p-value'] < 0.05]\n",
    "\tdefg = abc[abc['p-value'] > 0.05]\n",
    "\tlists1.append(['PA',len(abcd),i,len(defg)])\n",
    "\tset_of_tuples = {tuple(inner_list) for inner_list in lists1}\n",
    "df10 = pd.DataFrame(set_of_tuples)\n",
    "df10['combined'] = df10[0]+'/'+df10[2]\n",
    "X_1 = range(len(set_of_tuples))\n",
    "x_val= [x[0] for x in set_of_tuples]\n",
    "y_val = [x[1] for x in set_of_tuples]\n",
    "x_val1 = [x[2] for x in set_of_tuples]\n",
    "y_val1 =[x[3] for x in set_of_tuples]\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax = sns.barplot(data=(y_val),alpha=0.9)\n",
    "ax = sns.barplot(data=y_val1, alpha=0.7)\n",
    "ax.set_xticklabels(df10['combined'], rotation='vertical', fontsize=10)\n",
    "ax.set(xlabel='States',\n",
    "       ylabel='count of p_values that are significant per state VS PA',\n",
    "       title='Resulting P values'\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The results above are nice to look at but with that many, it can be a little difficult to interpret"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_results(dataframe):\n",
    "\tequal_var_different = dataframe[dataframe['p-value'] <= 0.05]\n",
    "\tequal_var_same = dataframe[dataframe['p-value'] > 0.05]\n",
    "\ttexas_a=equal_var_same[equal_var_same['State1']=='TX']\n",
    "\tca_a=equal_var_same[equal_var_same['State1']=='CA']\n",
    "\tpa_a=equal_var_same[equal_var_same['State1']=='PA']\n",
    "\ttexas_b=equal_var_different[equal_var_different['State1']=='TX']\n",
    "\tca_b=equal_var_different[equal_var_different['State1']=='CA']\n",
    "\tpa_b=equal_var_different[equal_var_different['State1']=='PA']\n",
    "\tprint('Texas equal variance, no differences',len(texas_a), 'differences',len(texas_b))\n",
    "\tprint('California equal variance, no differences',len(ca_a),'differences',len(ca_b))\n",
    "\tprint('Pennsylvania equal variance, no differences',len(pa_a),'differences',len(pa_b))\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_results1(dataframe):\n",
    "\tequal_var_different = dataframe[dataframe['p-value'] <= 0.050]\n",
    "\tequal_var_same = dataframe[dataframe['p-value'] > 0.050]\n",
    "\ttexas_a=equal_var_same[equal_var_same['State1']=='TX']\n",
    "\tca_a=equal_var_same[equal_var_same['State1']=='CA']\n",
    "\tpa_a=equal_var_same[equal_var_same['State1']=='PA']\n",
    "\ttexas_b=equal_var_different[equal_var_different['State1']=='TX']\n",
    "\tca_b=equal_var_different[equal_var_different['State1']=='CA']\n",
    "\tpa_b=equal_var_different[equal_var_different['State1']=='PA']\n",
    "\tprint('Texas unequal variance, no differences',len(texas_a), 'differences',len(texas_b))\n",
    "\tprint('California unequal variance, no differences',len(ca_a),'differences',len(ca_b))\n",
    "\tprint('Pennsylvania unequal variance, no differences',len(pa_a),'differences',len(pa_b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Please look here for the results and conclusion section for equal and unequal variance totals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_results(t_test_equal_var_2020)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_results1(t_test_unequal_var_2020)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_results(t_test_equal_var_2021)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_results1(t_test_unequal_var_2021)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_results(t_test_equal_var_2022)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_results1(t_test_unequal_var_2022)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Proportions of each measure are in the dataframes below. \n",
    "So the totals of each measure were taken (accept or reject) and divided by the total number of entries to get a sort of proportion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "california_equal1=pd.read_csv('California_equal_var_2020-2022.csv',index_col=[0])\n",
    "california_equal1.style\n",
    "california_unequal1=pd.read_csv('California_unequal_var_2020-2022.csv',index_col=[0])\n",
    "texas_equal1=pd.read_csv('Texas_equal_var_2020-2022.csv',index_col=[0])\n",
    "texas_unequal1=pd.read_csv('Texas_unequal_var_2020-2022.csv',index_col=[0])\n",
    "pa_equal1=pd.read_csv('Pennsylvania_equal_var_2020-2022.csv',index_col=[0])\n",
    "pa_unequal1=pd.read_csv('Pennsylvania_unequal_var_2020-2022.csv',index_col=[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.nansum(california_equal1['p-value_<0.05_2017-2018'])/29*100)\n",
    "print(np.nansum(california_unequal1['p-value_<0.05_2017-2018'])/29*100)\n",
    "print(np.nansum(texas_equal1['p-value_<0.05_2017-2018'])/29*100)\n",
    "print(np.nansum(texas_unequal1['p-value_<0.05_2017-2018'])/29*100)\n",
    "print(np.nansum(pa_equal1['p-value_<0.05_2017-2018'])/29*100)\n",
    "print(np.nansum(pa_unequal1['p-value_<0.05_2017-2018'])/29*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.nansum(california_equal1['p-value_<0.05_2018-2019'])/29*100)\n",
    "print(np.nansum(california_equal1['p-value_>0.05_2018-2019'])/29*100)\n",
    "print(np.nansum(california_unequal1['p-value_<0.05_2018-2019'])/29*100)\n",
    "print(np.nansum(california_unequal1['p-value_>0.05_2018-2019'])/29*100)\n",
    "print(np.nansum(texas_equal1['p-value_<0.05_2018-2019'])/29*100)\n",
    "print(np.nansum(texas_equal1['p-value_>0.05_2018-2019'])/29*100)\n",
    "print(np.nansum(texas_unequal1['p-value_<0.05_2018-2019'])/29*100)\n",
    "print(np.nansum(texas_unequal1['p-value_>0.05_2018-2019'])/29*100)\n",
    "print(np.nansum(pa_equal1['p-value_<0.05_2018-2019'])/29*100)\n",
    "print(np.nansum(pa_equal1['p-value_>0.05_2018-2019'])/29*100)\n",
    "print(np.nansum(pa_unequal1['p-value_<0.05_2018-2019'])/29*100)\n",
    "print(np.nansum(pa_unequal1['p-value_>0.05_2018-2019'])/29*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.nansum(california_equal1['p-value_>0.05_2019-2020'])/29*100)\n",
    "print(np.nansum(california_unequal1['p-value_>0.05_2019-2020'])/29*100)\n",
    "print(np.nansum(texas_equal1['p-value_>0.05_2019-2020'])/29*100)\n",
    "print(np.nansum(texas_unequal1['p-value_>0.05_2019-2020'])/29*100)\n",
    "print(np.nansum(pa_equal1['p-value_>0.05_2019-2020'])/29*100)\n",
    "print(np.nansum(pa_unequal1['p-value_>0.05_2019-2020'])/29*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pa_equal1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pqr=texas_equal1.loc[:, texas_equal1.columns != 'Texas_Measure_equal_var']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def style_negative(v, props=''):\n",
    "    return props if v < 0 else None\n",
    "s2 = pqr.style.map(style_negative, props='color:red;')\\\n",
    "              .map(lambda v: 'opacity: 20%;' if (v < 0.0) else None)\n",
    "s2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def highlight_max(s, props=''):\n",
    "    return np.where(s >0.499, props, '')\n",
    "s2.apply(highlight_max, props='color:white;background-color:darkblue', axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results, conclusion, discussion\n",
    "\n",
    "The overall project felt like a success in terms of working with the very large \n",
    "amounts of data present. Data was compiled, cleaned, visualized and analyzed in a manner that \\\n",
    "was straightforward and simple to follow. \n",
    "\n",
    "A general discussion of non-parametric results:\n",
    "\n",
    "The first question, do the States with more PLACES data have different metrics over all\\\n",
    "had a straightforward and simple conclusion.That States with more PLACES locations did differ from the mean/median. And due to the \n",
    "non parametric nature of the tests that were involved, a directional difference from the means/medians could not be calculated.\n",
    "So in order to answer more questions, normality had to be assumed.\n",
    "\\\n",
    "\n",
    "A general discussion of parametric results:\n",
    "Do States with more PLACES data have higher metrics for measures that the States with fewer locations? \\\n",
    "For some tests, and States there were clear differences. In the table below are all of the proportions \\\n",
    "of p-values rejecting(p<0.05) or failing to reject (p>0.05) the null hypothesis for CA,PA,TX for all 3 years separated by measure. \n",
    "\n",
    "For California, both equal and unequal variance tests, the `proportions` of measures rejecting the null are higher \n",
    "in general,with more than 70% of values falling into the reject category.\\\n",
    "But when the above results for the `total` counts of whether or not the null hypothesis (no difference in means) is rejected is inspected.\\\n",
    "More than 50% of the time there is failiure to reject.\n",
    "\n",
    "Texas was the exact same, as far as the `proportions` of the measures. More than 60% of the time the null hypothesis was rejected. \\\n",
    "But when the values are simply summed, the sheer number of tests where there was failiure to reject led to a conclusion of failiure to reject. \n",
    "\n",
    "Pennsylvania was a little bit different than Texas and California in terms of results. for the `proportions`. \\\n",
    "For equal  and unequal variances, failiure to reject was a little lower than for California and Texas. It then \n",
    "leveled off and ended up at about 50%. \n",
    "But for the `total` counts the numbers were very similiar, at about 50%.\n",
    "\n",
    "So the conclusion to this long, drawn out process is:\n",
    "Overall the null hypothesis should be rejected:\n",
    "There are many times where this is shown to be true, no matter how the data is looked at.\n",
    "\n",
    "Improvements for the future:\n",
    "I can probably go into even more detail and optimize the code even better for faster run times. \n",
    "And show more visualizations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from ipywidgets import interact, Dropdown, Output\n",
    "from IPython.display import display\n",
    "pd.reset_option('^display.', silent=True)\n",
    "df_ls = [california_equal1,california_unequal1,texas_equal1,texas_unequal1,pa_equal1,pa_unequal1]\n",
    "dropdown = Dropdown(options=list(range(len(df_ls))), description='Select DataFrame')\n",
    "#output = Output(layout={'overflow_x': 'auto'})\n",
    "def display_dataframe(selected_index):\n",
    "    selected_df = df_ls[selected_index]\n",
    "    display(selected_df)\n",
    "interact(display_dataframe, selected_index=dropdown)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}