39-ANOVA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ANOVA\n",
    "\n",
    "## Introduction\n",
    "\n",
    "One-way ANOVA compares the means of three or more groups, assuming that all values are sampled from Gaussian populations. It is called one-way ANOVA, because the groups are categorized by one scheme (e.g. treatment). Two-way ANOVA is used when the data are categorized by two schemes (e.g. treatment and gender).\n",
    "\n",
    "![flowchart for selecting the right ANOVA function](https://pingouin-stats.org/_images/flowchart_one_way_ANOVA.svg)\n",
    "\n",
    "The main problem with computing several t tests is multiple comparisons. As you include more groups in the study, you increase the chance of obtaining one or more P values less than 0.05 just by chance. If the null hypothesis were true, there would be a 5% chance that each particular t test would yield a P value less than 0.05, but with three comparisons, the chance that any one (or more) of the values would be less than 0.05 would be far higher that 5%\n",
    "\n",
    "One-way ANOVA can be seen as a regression model with a **single categorical predictor**. This predictor usually has **2+ categories**. In fact ANOVA compares the fit of several models to the data, and this can be done with regression techniques. The answers will be fundamentally identical, but they will look very different.\n",
    "\n",
    "Comparing three or more means with one-way ANOVA can be viewed as comparing the fit of the data to two different models:\n",
    "\n",
    "- null hypothesis; all populations share the same mean, and nothing but random sampling causes any differences between sample means\n",
    "- alternative hypothesis; all populations do not share the same means, at least one population has a mean different than the rest.\n",
    "\n",
    "### Interpreting a P value\n",
    "\n",
    "One-way ANOVA compares all the groups at once. The null hypothesis is that all groups were sampled from populations with identical means, i.e. there is no difference between means: $ \\overline{x_1} = \\overline{x_2} = ... = \\overline{x_i}$.\n",
    "\n",
    "The P value answers the question\n",
    ">if the null hypothesis were true, what is the chance of randomly picking samples with mean values as different (or more different) than those observed in the study?\n",
    "\n",
    "With a P value of 0.004, there is only 0.4% chance that the different means would be as far apart as actually observed or even further apart.\n",
    "\n",
    "### How it works\n",
    "\n",
    "The classic ANOVA is very powerful when the groups are **normally distributed** and have **equal variances** (homoscedasticity). It does not require all the samples to have the same number of values, though the two assumptions (Gaussian population and equal variance) matter much more when the sample size varies a lot between groups. When the groups have unequal variances, it is best to use the _Welch ANOVA_ that better controls for type I error.\n",
    "\n",
    "The main idea of ANOVA is to partition the variance (sums of squares) into several components. For example, in one-way ANOVA:\n",
    "\n",
    "$$ \\text{SS}_\\text{total} = \\text{SS}_\\text{effect} + \\text{SS}_\\text{error} $$\n",
    "$$ \\text{SS}_\\text{total} = \\sum_i \\sum_j (Y_{ij} - \\overline{Y})^2 $$\n",
    "$$ \\text{SS}_\\text{effect} = \\sum_i n_i (\\overline{Y_i} - \\overline{Y})^2 $$\n",
    "$$ \\text{SS}_\\text{error} = \\sum_i \\sum_j (Y_{ij} - \\overline{Y}_i)^2 $$\n",
    "\n",
    "where:\n",
    "\n",
    "- $ i=1,...,r $\n",
    "- $ j=1,...,n_i $\n",
    "- $ r $ the number of groups\n",
    "- $ n_i $ the number of observations for the $ i $ th group\n",
    "- $ N $ the total number of observations (in all groups)\n",
    "\n",
    "The **total variability** among all the values, ignoring which value came from which group, by **summing the squares of the difference of each value from the grand mean**. Some of the variation comes from **differences among the group means** because the group means are not all the same. **Sum the squares of each group mean from the grand mean and weight by sample size** to get the sum of squares resulting from treatment. The rest of the variation comes from **variability within each group**, quantified by **summing the squares of the differences of each value from its group mean**. This is also called the residual sum of squares or the error sum of squares.\n",
    "\n",
    "The F-statistics is then defined as:\n",
    "$$ F^* = \\frac{S_\\text{effect}^2}{S_\\text{error}^2} = \\frac{\\text{MS}_{\\text{effect}}}{\\text{MS}_{\\text{error}}} = \\frac{\\text{SS}_{\\text{effect}} / (r - 1)}{\\text{SS}_{\\text{error}} / (N - r)} $$\n",
    "\n",
    "and the P-value can be calculated using a F-distribution with $ r - 1, N - r $ degrees of freedom.\n",
    "\n",
    "The default effect size reported is the *partial eta-square*, which, **for one-way ANOVA is the same as eta-square and generalized eta-square**:\n",
    "$$ \\eta_p^2 = \\frac{\\text{SS}_{\\text{effect}}}{\\text{SS}_{\\text{effect}} + \\text{SS}_{\\text{error}}} $$\n",
    "$$ \\eta^2 = \\frac{\\text{SS}_{\\text{effect}}}{\\text{SS}_{\\text{total}}} $$\n",
    "\n",
    "It is interpreted in the same way as $R^2$. For example, an $\\eta^2$ of 0.053, only 5.3% of the total variability is the result of differences within the groups.\n",
    "\n",
    "Of note, the concept of one- and two-tailed P values does not apply in ANOVA. Because the means of the groups can be in many different orders, the P values has many tails."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Partitioning the sum of squares"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from scipy import stats\n",
    "import pingouin as pg\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the analysis of the variance: _comparing the ratio of systematic variance to unsystematic variance_ using sum of squares (SS), between the groups **SSB** (i.e. effect or $\\text{SS}_\\text{effect}$) and the variance within the group **SSW** (i.e. error, residual or $\\text{SS}_\\text{error}$), with $ \\text{SSW} + \\text{SSB} = \\text{SST} $.\n",
    "\n",
    "When F increase, then the variance between the group (SSB) is bigger than within the group (SSW).\n",
    "\n",
    "The group means are represented as deviations from the grand mean by grouping their coefficients under a single term $ y_{ij} = \\mu_{grand} + \\tau_j + \\epsilon_{ij} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One example of one-way ANOVA comparing 3 groups:  \n",
    "\n",
    "| Hypothesis  | Scatter from | SS    | Variation | $\\eta^2$ |\n",
    "| ----------- | ------------ | -----:| ---------:| -------- |\n",
    "| Null        | Grand mean   | 17.38 |  100.0%   |          |\n",
    "| Alternative | Group mean   | 16.45 |   94.7%   |          |\n",
    "| Difference  |              |  0.93 |    5.3%   |  0.053   |\n",
    "\n",
    "Of all the variation, 94.7% is the result of variation within the groups, leaving 5.3% of the total variation as the result of differences between the group means. *The sum of squares resulting from the treatment and the sum of squares within the groups always add up to the total sum of squares.*\n",
    "\n",
    "| Source variation | SS    | DF  | MS    | F ratio | P value |\n",
    "| ----------------:| -----:| ---:| ----- | ------  | ------- |\n",
    "|   Between groups |  0.93 | 2   | 0.460 |  5.69   |  0.004  |\n",
    "| + Within groups  | 16.45 | 202 | 0.081 |         |         |\n",
    "| =          Total | 17.38 | 204 |       |         |         |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the total, there are 205 values and only one parameter (grand mean) estimated, so $ \\text{DF} = 204$. For the SSW, 3 groups/parameters were fit (the mean of each group), so $ \\text{DF} = 205 - 3 = 202 $. If the null hypothesis were true, F would be likely to have a value close to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P values computed from the F ratio distribution: 0.0040\n"
     ]
    }
   ],
   "source": [
    "dfn, dfd = 2, 202 # DF at numerator is for 'within', at the denominator for 'within'\n",
    "F = .460 / .081\n",
    "\n",
    "print(f\"P values computed from the F ratio distribution: {1 - stats.f(dfn, dfd).cdf(F):.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The low P value means that the differences among group means would be very unlikely if in fact all the population means were equal.  \n",
    "The low R² means that the differences among group means are only a tiny fraction of the overall variability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practical example of one-way ANOVA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Subject</th>\n",
       "      <th>Hair color</th>\n",
       "      <th>Pain threshold</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Light Blond</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Light Blond</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Light Blond</td>\n",
       "      <td>71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Light Blond</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Light Blond</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Subject   Hair color  Pain threshold\n",
       "0        1  Light Blond              62\n",
       "1        2  Light Blond              60\n",
       "2        3  Light Blond              71\n",
       "3        4  Light Blond              55\n",
       "4        5  Light Blond              48"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 0. Load an example dataset comparing pain threshold as a function of hair color\n",
    "data = pg.read_dataset('anova')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsZklEQVR4nO3deXwV5d338c8vIYEEAmHfIiDEBYtCK7ailIKir4ILVm2tFcUFsVXUqtVqq89tW/pYrcttAOsOuPSmuCGFx4WiqHdVIMgiSIWwgwJhC4FAIMn1/DETOITk5CQ5a873/Xrldc7MXDPzy2TyO9e5Zua6zDmHiIgkj5RYByAiItGlxC8ikmSU+EVEkowSv4hIklHiFxFJMk1iHUAo2rVr53r06BHrMEREEsrChQu3O+faV52fEIm/R48e5OfnxzoMEZGEYmbrq5uvph4RkSSjxC8ikmSU+EVEkkxCtPHHo9LSUl577TVef/11iouLOemkkxgzZgz9+vWLdWgiIkFZIvTV079/fxdPF3fXrl3L+eefT0FBwTHLxo4dS15eHmYWg8hERI4ws4XOuf5V56upp47KysoYNmxYtUkfYMKECTz22GNRjkpEJHRK/HX09ttv8/XXXwct88QTT1BWVhaliERE6kZt/EBeXl6NNfiq3n///VrLfPPNN/z85z+nS5cudYojNzeX2267rU7riIjUlWr8dXTo0KGQyqnGLyLxSjV+qFMtu2XLlvz5z38OWiYlJYW8vDy6devW0NBERMIuYjV+MzvJzBYH/Owxs1+bWRszm21mq/zX1pGKIRJuvPFGUlKCH7Zhw4Yp6YtI3IpY4nfOfe2c6+ec6wecDpQAbwH3AnOccycAc/zphNG9e3fGjRtX4/L27dvz+OOPRzEiEZG6iVYb/7nAaufcemAEMMWfPwW4JEoxhM19993HlClT6N279+F5KSkp9OrVi08//ZQTTzwxhtGJiAQXlQe4zOxF4Avn3AQz2+2cy/bnG7CrcrrKOmOAMQDdunU7ff36ajuZi7lly5ZRXFzMCy+8QGZmJnl5ebEOSUQEiOEDXGaWDlwMvFZ1mfM+dar95HHOPeuc6++c69++/THdSceNPn36MGDAADIzM2MdiohISKLR1DMMr7a/1Z/eamadAfzXbVGIQUREfNFI/FcC/xMwPQMY5b8fBbwdhRhERMQX0cRvZs2B84A3A2b/BTjPzFYBQ/1pERGJkog+wOWc2we0rTJvB95dPiIiEgPqskFEJMko8YuIJBklfhGRJKPELyKSZJT4RUSSjBK/iEiSUeIXEUkySvwiIklGiV9EJMko8YuIJBklfhGRJKPELyKSZJT4RUSSjBK/iEiSUeIXEUkySvwiIklGiV9EJMko8YuIJBklfhGRJKPELyKSZJT4RUSSjBK/iEiSUeKXuLJlyxaWL1/Ozp07Yx2KSKOlxC9x4eOPP2bo0KF06dKFPn360LFjRy6//HKWLVsW69BEGh0lfom56dOnc+655zJnzhyccwCUlZXxxhtvcPbZZ7Nw4cIYRyjSuCjxS0wdOHCAG264gbKysmqX79mzh9GjR0c5KpHGTYlfYmratGm1tucvXryYefPmRSkikcavSawDaAwqKirYuHEj+/fvZ+7cufzoRz/CzGIdVszk5eVRUFAQUtlPPvkkpHL33Xcfffr0qVMcubm53HbbbXVaRyQZqMbfQC+//DI9e/Zk+vTpvPfeewwZMoTc3FymTZsW69ASQpMmodU9UlNTIxyJSPKwyotp8ax///4uPz8/1mEc49lnn+Wmm26qdpmZ8dJLLzFy5MgoR5VYPv/8cwYMGBC0TFpaGhs3bqRjx45RikqkcTCzhc65/lXnq8ZfTyUlJdxzzz01LnfOcdddd3Hw4MEoRpV4zjzzTM4666ygZUaOHKmkLxJGSvz19Nprr1FUVBS0zLZt25gxY0aUIkpcb7zxBqeeemq1y4YOHcqECROiHJFI45bwF3frciExnEK9y+SJJ57g448/jnA0RyTiBc1OnTqRn5/Pa6+9xssvv8yiRYto0aIFTz75JMOHDyclRfUTkXBK+MRfUFDAoi+/oiKzTVT3u604tCacrXtKWbh6S4Sj8aSUJG43B+np6Vx11VVcddVVhz+4LrzwwhhHJdI4JXziB6jIbMOBU6KbJFoctxO+/hIqymssY6lpNB90HQcys6ISU7OvZkZlPyKS2PQdup7Ss9rQts/AoGXa9xtCkyglfRGRUCnxN8Bx515Dm1OquyPFaHvaj+j6oyujHpOISG0aRVNPrKSkNqHH8Jvo+IOL2Ln8fzm0dxdlB/bhKsop21fEt59Np91pg0nPiu71BxGRYJT4wyCjbRdan3wmq998jEN7dx+eX7R6MVs+/yfHnXMV7b87NHYBiogEiGhTj5llm9nrZvYfM1thZgPMrI2ZzTazVf5r60jGEA1lB/ZR8PqjRyX9w1wFG+e8zO7Vi6Iel4hIdSLdxv8k8K5z7mSgL7ACuBeY45w7AZjjTye0HV9+TFlJ8Ie5ts6fFaVoRESCi1hTj5m1AgYB1wI45w4CB81sBDDYLzYFmAv8tr772bRpEyklRTG9lXHPkg9qLbNv8ypSFr1OetNmEYsjpWQHmzZV36+9iEilSNb4jwcKgUlmtsjMnjez5kBH59y3fpktQLWdsJjZGDPLN7P8wsLCCIbZcOXloSXbihoGGxERiaZIXtxtAnwPuNU5N8/MnqRKs45zzplZtd2DOueeBZ4Fr3fOmnaSk5PD1tImUX+AK1B6wTpKihcELZOS3oyKfpdyIC09YnE0+2omOTmdIrZ9EWkcIlnj3wRscs5VdmrzOt4HwVYz6wzgv26LYAxR0a7vkFrLtDnlbFIimPRFREIVscTvnNsCbDSzk/xZ5wJfATOAUf68UcDbkYohWlp2/04ND3J50lu1p/OAEVGMSESkZpG+j/9W4FUzSwfWANfhfdhMM7MbgPXAzyIcQ1R0H3YjTVt3onDRnMN3+FhqE7JP6E/O4CtJa94qxhGKiHgimvidc4uBY0Z/wav9NypmKXQeMIJO37+AfVvW4srLaNauK2mZLWMdmojIUfTkbphZahNadD0h1mGIiNRInbSJiCQZJX4RkSSjph6Juo0bN7Ju3Tqys7NrHGtXRCJHNX6JmiVLljBs2DC6d+/OoEGDOO200+jTpw9Tp06NdWgiSUU1fomKhQsXMmTIEIqLi4+av3z5cq688koKCwu59dZbYxSdSHKpMfGb2feCreic+yL84UhjNXbs2GOSfqC7776bn//857Rv3z6KUYkkp2A1/sf812Z49+IvAQw4DcgHBkQ2NGksli5dyueffx60TGlpKZMnT+buu++OUlQiyavGxO+cGwJgZm8C33POfelP9wEejEp0EnV5eXkUFBSEdZsrV64MqdykSZPYuHEjq1atAuC2224Laxz1kZubGxdxiIRTKG38J1UmfQDn3DIz6x3BmCSGCgoKWLnsC7q1KA/fRnftCqmY7d/JgXULSD/k3XNwYF3wHk8jbcPe1JjuXyRSQkn8S83seeAVf/oqYGnkQpJY69ainPv77w3b9vYfTOX7S1PYc6AiaLkHBmdw1vHh229DjctvEesQRCIilNs5rwOWA7f7P1/580RCkpGewrU/aBu0TL+uGZx1vBKtSDTUWuN3zh0AnvB/ROrljsEd2LLnENMW7T5m2SmdmvHcld2iH5RIkgp2O+eXQI0jXznnTotIRHFu35a17Pp6PhWlJTTN7kib7wwkrbl64KxNSorx10tyuOb7bZm6cBcbdx+kZbNULurTiqEnZZGaYrEOUSRpBKvxx24swzhUXlrC2n8+xZ51Xx41/5t/v0Hnsy6l0w8uiFFkieXULhmc2iUj1mGIJLVgt3Our3xvZh2BM/zJ+c65hB8usa7WvD2e4g1fHTPflZfxzSfTaNIsM6QhGEVEYq3Wi7tm9jNgPvBTvNGy5pnZ5ZEOLJ7s3bSy2qQfaMvn/8RVBL9rRUQkHoRyO+fvgTMqa/lm1h74F97g6XEhpWQnzb6aGbHtf7Os9t4pDhbv4NCHT9Gyc4+IxVGblJKdQKeY7V9EEkMoiT+lStPODuKoV8/c3NyI72N7QSpbQiiXRhmn94pl4u0UleMhIoktlMT/rpm9B/yPP30F8P8iF1LdRONx+t/97nc89NBDtZY76aSTyMvLi3g8IiINUWvN3Tl3N/AsXudspwHPOud+G+nA4sn111+PWfDbDTt06KCeJUUkIYTUZOOce8M5d6f/81akg4o3tXXUlZ6eztlnnx3FiERE6i+Uu3ouNbNVZlZkZnvMrNjM9kQjuHjyxBNP8Kc//YnWrVsfNb9Pnz68++675OTkxCgyEZG6CaWN/xHgIufcikgHE8/MjPvvv5+77rqL2bNns2fPHnJzcznzzDMBeOutpPsiJCIJKpTEvzXZk36gjIwMLr744liHISJSb8H66rnUf5tvZv8ApgOllcudc29GNjQREYmEYDX+iwLelwDnB0w7QIlfRCQBBeurR33uS8QdKnds2XOItFSjU8u0WIcjkhRqbeM3s0eAccB+4F28e/nvcM69EnRFkSD2H6xg4ieFTP1iF4V7ywDo3bEZ1w9oy8++27qWtUWkIUK5j/9859wevG6a1wG5wN2RDEoat/0HK7hyylrGf1x4OOkDrNh6gLunb+ZP734bw+hEGr9QEn/lt4ILgNecc0URjEeSwNP/LmTRpv01Ln/+sx0sWL8vihGJJJdQEv9MM/sPcDowx++d80Bkw5LGqrzC8feFu2ot9/KCnVGIRiQ5hTLm7r1+O3+Rc67czEqAEZEPTWJh06ZN7CtOZVx+ZAY+37f/INuKy2ot99Ha0ojFEKr1xak037QppjFI9G3YsIHFixeTlpbGD3/4Q1q0iO15GAmhdNmQCdwM/M2f1QXoH8mgpPFKCXFs3ZSUuOn5W5LE+vXrufjiizn++OMZMWIEw4cPp0uXLtx1110cPHgw1uGFVShP7k4CFgJn+dObgdeAyI18IjGTk5PDgbJvub//3ojtY+myDJZ+U3MbP8CVfTO5J4IxhGJcfguaqQ+mpPDNN98wcOBANlX5hldcXMzjjz/OypUrefvtt0lJSWH79u384Q9/4MEHH6Rt27YxirhhQqlW9XLOPQIcAnDOlQChVdtEqnHjWcH/WTLSjKv6t4lSNCIwbty4Y5J+oJkzZzJr1iwApkyZwtKlS5kyZUq0wgu7UBL/QTPLwHtaFzPrRUDXDSJ1dfGp2dw6qPqxCzLSjL9d0Y2u2elRjkqS1YEDB3jlldofS3ruuefYvn0777zzDs453nnnHXbs2BGFCMMvlKae/8J7cOs4M3sVOBu4NpJBSeP3m3M7ct7JWby8YCdffrOftFRjcG4WV/VvQ+dWeoK3vrZs2cLrr7/Ozp076dGjB5dffjmZmZmxDiuubd26leLi4lrLFRQUMGXKFJxzAFRUVDBlyhTuvPPOSIcYdkETv5mlAK2BS4Ez8Zp4bnfObY9CbNLI9e2aSd+uSkrhUFZWxp133snTTz/NoUOHDs+//fbbeeihh/jlL38Zw+jiW1ZWFmZ2OKHXpGXLlsyePfvw8T106BDvv/9+Qib+oE09zrkK4B7n3A7n3Czn3My6JH0zW2dmX5rZYjPL9+e1MbPZ/uAus81Mz+eLNNAtt9zC+PHjj0r6ALt37+ZXv/oVkyZNilFk8a9NmzYMHTq01nJXXHEF5513Hmlp3jfStLQ0zj///FrWik+htPH/y8x+Y2bH+Um7jZnV5crbEOdcP+dc5S2g9wJznHMnAHP8aRGppzVr1vDcc88FLfPAAw9QVlb78xPJ6r777iM1NbXG5V26dOG6665j1KhRh8ffTklJYdSoUdEKMaxCaeO/wn+9JWCeA3rWc58jgMH++ynAXCCpBm8XCVVeXh4FBQVBy8ybN6/WZorNmzdz2WWX0b1793rHUtvY04lsyJAhvPTSS4wePZr9+4++1fj4449n5syZZGdnAzBs2DBmzJjBsGHDEvZ2zlCe3D2+Adt3wPtm5oBnnHPPAh2dc5W9cG0BOla3opmNAcYAdOvWrQEhiDRuJSUlYS2XrH7xi18wbNgwJk+ezMKFC0lPT2f48OFccsklNGlyJFWOGjWKdevWJWxtH0Kr8WNmZwE9Ass7514KYdWBzrnNZtYBmO33+XOYc875HwrH8D8kngXo379/8OqMSCMVSg37j3/8I8uWLau13N13352wbdLR0rp1a+64446gZdq1a8f48eOjFFFkhNJlw8vAo8BA4Az/J6QuG5xzm/3XbcBbwPeBrWbW2d92Z2BbvSIXEQCuvvrqWru4aNGiBeeee26UIpJ4F8rF3f7A2c65m51zt/o/tVZDzKy5mWVVvscbunEZMAOo/I40Cni7fqGLCHht0DfddFPQMmeeeWbQi5eSXEJp6lkGdALqOjpGR+At/wp4E+Dvzrl3zWwBMM3MbgDWAz+r43ZFpIrx48eTkZHBxIkTKS098mB9mzZt6NevH717945hdBJvakz8ZvZPvIuzWcBXZjafgK4anHMXB9uwc24N0Lea+TuARvmds6KigjfffJPnn3+e1atXk52dzRVXXMH1119/+I4AkUhITU3lscce47777uONN944/OTuT37yE+65555YhydxJliN/9GoRdEIlJWVMWvWLCZOnHjU/Pnz5/P4448ze/Zs1bok4tq1a1drs49IjW38zrmPnHMfAcMr3wfOi16IieHf//43GzZsqHbZ5s2bueiiiygvL49yVCIixwrl4u551cwbFu5AEllRURErVqwIWmb16tXMmDEjShGJiNQsWBv/r/BG3uppZksDFmUB/450YNEUytORwaxdu/aYPlKq88ADD/Dhhx/WuLwxPxkpIvEjWBv/34F3gIc4uj+dYuecRsIOEGoTjpp6RCQe1Jj4nXNFQBFwZfTCiY2G1rLXrFlDbm5urf2l3HLLLYwdO7ZB+xIRaSiNaB0GPXv2ZNiw4Jc9mjdvzjXXXBOliEREaqbEHyZPPfUUXbt2rXZZkyZNmDRpEi1btoxyVCIix1LiD5Pu3bszb948br75ZrKysgCvv+7hw4fzwQcf8NOf/jTGEYqIeGrtssHMLgUeBjrgDb1oeB1rqvpaRdeuXZk4cSKPP/44hYWFZGVl0apVq1iHJSJylFD66nkEuMg5F/xGdTmsadOm5OTkxDoMEZFqhdLUs1VJX0Sk8Qilxp9vZv8ApnN0J21vRiooERGJnFASf0ugBK8//UoOUOIXEUlAoYy5e100AhERkegI1lfPPc65R8xsPF4N/yihjMIlIiLxJ1iNv/KCbn40AhERkegI1lfPP/3XKdELR0REIi2UB7jaA78FTgGaVc53zp0TwbhERCRCQrmP/1W8Zp/jgT8A64AFEYxJREQiKJTE39Y59wJwyB968XpAtX0RkQQVyn38lUNLfWtmFwDfAG0iF5KIiERSKIl/nJm1Au4CxuM90HVHRKMSaSQaOqxnOKxatQpo+IBD4aDhReNDsPv4mwG/BHKBrsALzrkh0QpMktvSzfv5+8KdrNt5kJZNU7igTyuG9W5JepPE6km8oKCARcsXQXYMg6jwXhZtXhTDIIDdsd29HBGsxj8Fr5nnE2AY3l09t0cjKEleFRWO+/75DVO/2HXU/Pf+U0yvdum8fHUPumanxyi6esqGisEVsY4i5lLmJtaHdmMW7C9xinNupHPuGeBy4IdRikmS2PiPC49J+pVWbz/Ida+up6Ii+NjGIhJcsMRfeVEX51xZFGKRJHfgUAWT5u0IWubrbaV8WLA3ShGJNE7Bmnr6mtke/70BGf60RuBq5DbsTWVcfouo73dzYRG7SsprLfeXj0v4bE/niMezYW8qJ0Z8LyLRF6zLhtRoBiLxITc3N2b7too1HOkiKoiM1jTrcUbE4zmR2B6PUJQfLKdwfiFbP9tK6c5SmmQ2od3p7eg0sBPpLRPsWohETSi3c0oSieWtdgUFBcyaNavWcldffTX3339/FCKKb2UlZSx/ajn7Nuw7Mm9fGZve3cTWf2/llFtOoXmX5jGMUOKVLrNL3MjNzeWcc4I/FJ6WlsYNN9wQpYji25ppa45K+oEOFR/iP8/9B6cL4VINJX6JK+PHj6dZs2Y1Ln/44Yfp3Dny7fvx7mDRQXYsDn4hvHRHKbuWVX+HlCQ3JX6JK6eccgqXXXYZPXv2JDX1yGWm3r178+KLL3LHHXpoHKCooCik2vzulburne/KHQeLDlK2XzfsJSMlfok7bdq04YILLmDJkiVceeWVtGzZkhUrVnDDDTcwbNgw5s6dG+sQYy/U58GqfDaUlZSxfsZ6FjywgPwH8pl/73yWT1jO7hW7wx2hxDFd3JW4tG/fPkaMGMHq1asPz3PO8e677zJ79mwmT57MyJEjYxhhaDZt2gRF4X9qteXe0O6mbrm/5eF9Hyo9xLKPl1FSXHKkgIOilUUUrSyiV79edO4ZwWa03bDJbYrc9iVkqvFLXPr444+PSvqBysvLGT16NNu3b49yVPEjo0UGrTu2DlomrWka7bq2Ozy9btm6o5N+FauXrKa0pDRsMUr8Uo1f4s6+fftYs2ZN0DKlpaW8+OKL3HPPPVGKqn5ycnIotMKI9NXTs29Plj25jNKdxybrlPQUThxzIpwAFVRQVlJG4czC4Bt08K37lm6Du4U9VvC+9eR0zYnItqVulPgl7BraFfGSJUuoqKg9UT733HNeU0oNGnsXwE1bN+XUu07l2w+/ZevnWynbW0ZKWgptv9eWrud0JbNz5uGyJVtKqDhY+zHdu0HdYSQDJX6JOxkZGSGVS0lRS2V6VjrdL+5O94u7U15aTkpaCpZix5Sz1GPnVae6daXxUeKXsGtoLbukpIQuXbpQVFQUtNzvf/97rrnmmgbtqzFJbVpzLyvNc5qTlpXGoeJDNZYByO6dHeaoJB5FvMpkZqlmtsjMZvrTx5vZPDMrMLN/mJk6FJGjZGZmMmbMmKBlunTpwhVXXBGliBJfSmoKnX7YKWiZJs2b0OH7HaIUkcRSNL4r387RPW89DDzhnMsFdgF6/l6OMW7cOC666KJql7Vv356ZM2fStGnTKEeV2HLOz6Ht99pWuyw1I5WTbzyZ1GbqmzEZRLSpx8xygAuAPwN3mpkB5wC/8ItMAR4E/hbJOCTxpKenM336dGbMmMEzzzzDqlWryMrK4qc//Sk33ngj7du3j3WIodsd49GnKq/XtoCTu5/Mrqa72LJ2CyXFJaQ0SaFdl3Z06tGJ9A3psCGCcezGG8S1Efjiiy/46KOPABg0aBCnn356jCOqm0i38f83cA+Q5U+3BXYHDOyyiRpOBTMbA4wB6NYtMreXSXxLSUnhkksu4ZJLLol1KPUWD906Vw62fkLXE7wZOUD/GATSNT6OR0OsXbuWkSNH8umnnx41f8CAAbzyyiv07NkzRpHVjTkXmd77zOxCYLhz7mYzGwz8BrgW+Nxv5sHMjgPecc71Cbat/v37u/z8/IjEKdLYVV5sz8vLi3Ekia2wsJD+/fuzYUP1X4uOO+448vPz6dAhfq6TmNlC59wxH/OR/P55NnCxma0DpuI18TwJZJtZ5TeNHGBzBGMQEQmLCRMm1Jj0ATZu3MiECROiGFH9RSzxO+fuc87lOOd6AD8HPnDOXQV8iDd4O8Ao4O1IxSAiEi5TpkyptczkyZMjH0gYxOI+/t8CU81sHLAIeCEGMYhIgmnoE+HgdZq3f//+eq0brLYfuP1hw4aFtL2MjAxycurfhUVDnkyPSuJ3zs0F5vrv1wDfj8Z+RaTxKCgo4D+LFxP8aYTgSoH6jkCQnpZG6cGDtZYp21f9qGjHxLJvH7vr2dHglnqtdYSe3BWRhNEJuIHYdCvxXMeOvLpxY9Ayl3XoyJgoxPdC1YEW6kidnYiIhODSLl1om15zRwNt0tK5rGuXKEZUf0r8IiIhaJvelCdOPY1ezZsfs6xX8+Y8cdpptE1PjKfJ1dQjkuD27t3Lyy+/zD/+8Q92795Nr169GD16ND/+8Y/xHpaXcOmWmckL3zudJUW7WeJ3Iti3VSv6tsqObWB1pMQvksAKCgo477zzWLdu3eF5S5Ys4c0332TEiBFMmzYtdsE1Yn1bZSdcsg+kph6RBFVRUcGFF154VNIP9Pbbb3PvvfdGNyhJCEr8Iglq1qxZfP3110HLPPfccxys5RZEST5q6hGJY8EeWvrggw9qXX/v3r0sWLCADh06NHiAnMY+lGUyUY1fJEGVl5eHVK5JkyYhD2cpyUE1fpE4FqyG/eSTT/LrX/866Ppmxquvvprw3SFLeKnGL5KgRo0aRWZmZtAyQ4cOVdKXYyjxiySo7OxsnnnmGVJSqv837tixI0899VSUo5JEoMQvksBGjhzJe++9x5AhQw7Py8jI4LrrruPzzz9XbT/Mdh86xJp9+9hxsDTWoTSI2vhFEtzQoUMZOnQo27Zto6ioiM6dO9OiRYtYh9WorN63l0nr1/Ppjh1UAAZ8LzubUd26c1qrVrEOr86U+EUaiQ4dOsTVsH+NxYriPdz55ZfsD7iLygELd3vdNvyh9ymc3bZt7AKsBzX1iIgE8cjKlUcl/UBlzvHIypUcrKiIclQNo8QvIlKDpUVFrC0pCVqmqOwQH20vjFJE4aGmHhFJCJs2baKYhg9CUhdf79sbUrnp+/axIYpxfQvs3bSp3uurxi8iUoOabpWtKjXEcvFCNX4RSQg5OTns3r49qkMvbm/dhivMKHfBa/O3tGnLSVGM6wUc2Q0YqD2xPqZERKKoXdOmDG3fPmiZvi1bcVJWVpQiCg8lfhGRIH6dewLfy86udlmv5s15sHfv6AYUBmrqEREJIiM1lUf7nMrnO3fy7tatbCstJTstjfM6dGBQu3akJVj7Pijxi4jUKsWMs9q25awEe1CrJon3USUiIg2ixC8ikmSU+EVEkoza+EVEwmDV3r3M2vItWw6UkpXWhHPbd+D7rVuTYtG7vz9USvwiIg1Q7hx/XbWSd7duPWr+7G3b6J2VxUPf6UN2WlqMoquemnpERBrghXXrjkn6lVYUF/NfK76KckS1U+IXEamnkvJypn/7TdAyS4qKWL5nT5QiCo2aekQkYWwhur1zVmeH/9oW2LBrJyU19NUfaOL2Qk5vGb5uHbYA2Q1YX4lfRBJCvIwfXLhqFQDZJ5zAtytWwIoVta6T0ro12f36hS2GbBp2PJT4RSQh3HbbbbEOATgSR15eHgsWLOBf//pXreuMHj2a22+/PdKhhUxt/CIi9XTGGWfw3e9+N2iZjIwMrrnmmihFFBolfhGRBpg4cSKZmZk1Ln/00Udp3bp1FCOqnRK/iEgdHTx4kOLiYgAGDBjA3LlzOeecc44q853vfIepU6dy8803xyLEoNTGLyISosmTJzN16lQKCwt55pln6NevH2PHjuX6669nzpw5rFmzhvXr19OmTRv69u0b63BrZK6WIcXiQf/+/V1+fn6swxCRJHbjjTfy/PPPV7ts1KhRTJo0CYuz7hnMbKFzrn/V+RFr6jGzZmY238yWmNlyM/uDP/94M5tnZgVm9g8zS49UDCIi4fD666/XmPQBpkyZwtSpU6MYUcNEso2/FDjHOdcX6Af82MzOBB4GnnDO5QK7gBsiGIOISIM99dRTtZaZOHFiFCIJj4i18TuvDWmvP5nm/zjgHOAX/vwpwIPA3yIVh4hIpby8PAoKCuq83ieffFJrmc8++6xOzxrk5ubG7NmEiN7VY2apZrYY2AbMBlYDu51zZX6RTUDXGtYdY2b5ZpZfWFgYyTBFRIIKpe0+3tr3g4noXT3OuXKgn5llA28BJ9dh3WeBZ8G7uBuRAEUkqdS3hr1582befPPNoGWGDx9OXl5evbYfbVG5j985txv4EBgAZJtZ5QdODrA5GjGIiNRXKN0txFOXDLWJ5F097f2aPmaWAZwHrMD7ALjcLzYKeDtSMYiIhMOgQYN47LHHamzO+ctf/sK5554b5ajqL5JNPZ2BKWaWivcBM805N9PMvgKmmtk4YBHwQgRjEBEJizvvvJOBAwcyYcIEPvroI5xzDBo0iLFjx3LmmWfGOrw60QNcIiKNVNQf4BIRkfikxC8ikmSU+EVEkowSv4hIkkmIi7tmVgisj3UcIWgHbI91EI2EjmV46XiGV6Icz+7OufZVZyZE4k8UZpZf3RV0qTsdy/DS8QyvRD+eauoREUkySvwiIklGiT+8no11AI2IjmV46XiGV0IfT7Xxi4gkGdX4RUSSjBK/iEiSadSJ38z2VjPvl2Z2TS3rXWtmE2pY9rsg660zsy/NbLH/OiJYLPVhZoPNbGY4tlXP/Zf7v99yM1tiZneZWZ3OIzPrYWbLQiiz39/XEjP71MxO8peF7RiY2YNm9ptwbKuabcfN+RcpZpZtZjcHTPcws18EW6cB+4r5uRdJZtbPzIYHTA82s7Misa9Gnfir45x72jn3UgM2UeM/nm+Ic64f3pgDiTEcT93sd871c859B2+MhWHAf4W6csAgPKFY7e+rL974zLUd+7gX6/PPPOH8v88Gbg6Y7sGRMbXDLa7OvTpuLxT9gOEB04MBJf5wCKzhmdkZZrbU/2T/a5WaQBcze9fMVpnZI375vwAZfvlXa9lVS2BXNfu3yn35tbIr/PmDzWyumb1uZv8xs1fNH/XBzH7sz/sCuDQMhyEsnHPbgDHAWP/36mFmn5jZF/7PWXD4d/vEzGYAXwVuw8x6mtkiMzujlt3VdDzbmNl0/+/4uZmd5s9/0Mxe9I/pGjO7LWCd35vZSjP7XyDiNbkq8Ub9/PP/Ll+b2UvAMuC4wG8jZna5mU323082szy/lrvGzC4PKHe3mS3wY/6DP/svQK/K38Gf/qE/fYd5427/NWC9mxpw+A6L1bln3rexGWb2ATDHqnz7NLMJZnat/36dmf3Bj+dLMzvZn9/cPzfn+/sfYWbpwB+BK/xj91vgl8Ad/vQPzRvc6g3/WC4ws7MbcgAb7Q+wt5p5DwK/8d8vAwb47/8CLPPfXwusAVoBzfC6iziupm0GbHsd8KW/3RLgwqqxAJfhDTyfCnQENuANWjMYKMIbjjIF+AwY6O9/I3ACYMA0YGacHdPd/u+SCTTz550A5PvvBwP7gOP96R7+MToJbzCevtVsswewH1gMrAa+BboFbG+m/3488F/++3OAxQF/50+BpniP1+8A0oDT/b9RJt4/dEHl+dBYzz//WFYAZ1YXG963g8n++8nAa/45eApQ4M8/H+8WRvOXzQQGVf4tA7Z1+G/jT48B7vffNwXyK8+DBD33rgU2AW1q+H0nANcG/D1u9d/fDDzvv/+/wEj/fTawEmjub3tCdeeKP/13YKD/vhuwor7nZkQHW49n5g0LmeWc+8yf9XfgwoAic5xzRX7Zr4DueAm4NkOcc9vNrBdejWCucy6wrXcg8D/OG4h+q5l9BJwB7AHmO+c2+ftcjHcC7gXWOudW+fNfwftnikdpwAQz6weUAycGLJvvnFsbMN0eb9jNS51zR9XEAqx2XrMF5n0zehb4cZUyA/E+THHOfWBmbc2spb9slnOuFCg1s214CeKHwFvOuRJ/uzPq9Zs2UDTPP3/+eufc5yGGN905VwF8ZWYd/Xnn+z+L/OkWeAl2Qy3bOh84LeCbQyt/vbU1r1Iv0Tz3ZjvndoYYV+UI7Qs58m39fOBiO3JtqRleIq/NUOAUOzL8Y0sza1Elv4QkaRN/CEoD3pdTx2PlnFttZlvxak3zo7HPWDCznnixbsNrb90K9MWrFR4IKLqvyqpFeEljIFW+gtdgBjCpjuEl3PEMEM7zbxvHHv/AB3iaBdm3Bbw+5Jx7JrCgmfWoJRTDq/W+F0rcdRHDcy9we2Uc3WRe07EM/BsacJlz7uvAgmb2g1riSMH71naglnK1Sro2/krOud1AccDB/nmIqx4ys7TaCplZB+B4ju1V9BO8drxUM2uP93U52AfDf4Aefg0O4MoQ44w4P/6n8b6eOrza3Ld+bfFqvOasmhwEfgJcY6HdBTIQ72t3VZ8AV/nxDAa2O+f2BNnOx8AlZpZhZlnARSHsO+xieP5V2mpmvc270PuTEPb7HnC9mbXwt9/V30cxkBVQrur0e8CvKmM2sxPNrHkI+wsqTs498I7vKWbW1P8WF8qI6+8Bt5odvob3XX9+bcfyfeDWygn/2029JFINqD4yzWxTwPTjVZbfADxnZhXAR3g1gdo8Cyw1sy+cc1dVs/xDMyvH++p5r3Nua5XlbwEDgCV4ta57nHNbKi/8VOWcO2BmY4BZZlaCl+iyqisbJRl+M1QaXm3nZY4c16eAN8y7XfFdjq1pHcU5t8/MLgRmm9le51zVZpde/r4M7591dDWbeRB40cyW4rVrj6pln1+Y2T/wjv82YEGw8g0UF+dfDbXye/Ha6Qvx2t1bBNupc+59M+sNfObnq7147dSrzezf5l2Yfgfv7pdyM1uCd73gSbwmyy/8RFcIXBLC71mdeDv3cM5tNLNpeNcN1nKkKSyYPwH/jfd3TPHXuxD4ELjX3+9DwD+B1827LfdW4DZgon+uN8GrxPwyhP0dI6m7bAhsHzOze4HOzrnbYxyWJAmdfxIrjb3GX5sLzOw+vOOwHu+quki06PyTmEjqGr+ISDJK2ou7IiLJSolfRCTJKPGLiCQZJX5p9KxKL5kWpPfLgDIX+3fahDuWuWaWsIN0S+OQ7Hf1iFTLv6/7mO4czKyJc64sWnGYWarfvYdI2KjGL0nNzC4ys3l+L4n/quybJvBbgXk9Vj5tZvOAR6qsn2pmj5rX2+pSM7vVn3+uv80vzeuJsWk1+77SX77MzB4OmL/XzB7zH4IaEMnfX5KTavySDCqf+KzUhiO1+f/F6//Emdlo4B7grmq2kQOcVU3tewzek6n9nHNl5nUT3QzvqdVznXMrzesO+Vd4T2sCYGZdgIfxegvdBbxvZpc456bj9dQ4zzlXXRwiDaYavySDygE8+vk9Lv6fgGU5wHtm9iVwN/CdGrbxWg1NLkOBZyqbf/xeG0/C61F1pV9mCl6fTIHOAOY65wr9dV8NKFMOvFGn31CkDpT4JdmNx+vo61TgJo7tXbFS0L5fwuyA2vUlkpT4Jdm1Ajb774N28FaD2cBN5g/DZ2ZtgK/xelTN9ctcjdcJW6D5wI/MrJ2ZpeL1ulq1jEhEKPFLsnsQeM3MFgLb67H+83h9uy/1L8b+wu8v/Tp/u1/ijX71dOBKzrlv8XrI/BCvp9CFzrm36/1biNSB+uoREUkyqvGLiCQZJX4RkSSjxC8ikmSU+EVEkowSv4hIklHiFxFJMkr8IiJJ5v8D7EGGf4bxGNMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.boxplot(x='Hair color', y='Pain threshold', data=data)\n",
    "sns.stripplot(x='Hair color', y='Pain threshold', data=data, color='black', size=10);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using pingouin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>W</th>\n",
       "      <th>pval</th>\n",
       "      <th>normal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Light Blond</th>\n",
       "      <td>0.991032</td>\n",
       "      <td>0.983182</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dark Blond</th>\n",
       "      <td>0.939790</td>\n",
       "      <td>0.664456</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Light Brunette</th>\n",
       "      <td>0.930607</td>\n",
       "      <td>0.597973</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dark Brunette</th>\n",
       "      <td>0.883214</td>\n",
       "      <td>0.324129</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       W      pval  normal\n",
       "Light Blond     0.991032  0.983182    True\n",
       "Dark Blond      0.939790  0.664456    True\n",
       "Light Brunette  0.930607  0.597973    True\n",
       "Dark Brunette   0.883214  0.324129    True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1. This is a between subject design, so the first step is to test for equality of variances\n",
    "pg.normality(data=data, dv='Pain threshold', group='Hair color')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>W</th>\n",
       "      <th>pval</th>\n",
       "      <th>equal_var</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>levene</th>\n",
       "      <td>0.392743</td>\n",
       "      <td>0.760016</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               W      pval  equal_var\n",
       "levene  0.392743  0.760016       True"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 3. Next step is to test for equality of variances\n",
    "pg.homoscedasticity(data=data, dv='Pain threshold', group='Hair color')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Source</th>\n",
       "      <th>SS</th>\n",
       "      <th>DF</th>\n",
       "      <th>MS</th>\n",
       "      <th>F</th>\n",
       "      <th>p-unc</th>\n",
       "      <th>np2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Hair color</td>\n",
       "      <td>1360.726</td>\n",
       "      <td>3</td>\n",
       "      <td>453.575</td>\n",
       "      <td>6.791</td>\n",
       "      <td>0.004</td>\n",
       "      <td>0.576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Within</td>\n",
       "      <td>1001.800</td>\n",
       "      <td>15</td>\n",
       "      <td>66.787</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Source        SS  DF       MS      F  p-unc    np2\n",
       "0  Hair color  1360.726   3  453.575  6.791  0.004  0.576\n",
       "1      Within  1001.800  15   66.787    NaN    NaN    NaN"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 3. If the groups have equal variances, we can use a regular one-way ANOVA\n",
    "aov = pg.anova(\n",
    "    data=data,\n",
    "    dv='Pain threshold',\n",
    "    between='Hair color',\n",
    "    detailed=True\n",
    ")\n",
    "aov.round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Source</th>\n",
       "      <th>ddof1</th>\n",
       "      <th>ddof2</th>\n",
       "      <th>F</th>\n",
       "      <th>p-unc</th>\n",
       "      <th>n2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Hair color</td>\n",
       "      <td>3</td>\n",
       "      <td>15</td>\n",
       "      <td>6.791407</td>\n",
       "      <td>0.004114</td>\n",
       "      <td>0.575962</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Source  ddof1  ddof2         F     p-unc        n2\n",
       "0  Hair color      3     15  6.791407  0.004114  0.575962"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Same but using a standard eta-squared instead of a partial eta-squared effect size. Also\n",
    "# note how here we're using the anova function directly as a method (= built-in function)\n",
    "# of our pandas dataframe. In that case, we don't have to specify data anymore.\n",
    "data.anova(\n",
    "    dv='Pain threshold',\n",
    "    between='Hair color',\n",
    "    detailed=False,\n",
    "    effsize='n2',\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>mean(A)</th>\n",
       "      <th>mean(B)</th>\n",
       "      <th>diff</th>\n",
       "      <th>se</th>\n",
       "      <th>T</th>\n",
       "      <th>p-tukey</th>\n",
       "      <th>hedges</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Dark Blond</td>\n",
       "      <td>Dark Brunette</td>\n",
       "      <td>51.2</td>\n",
       "      <td>37.4</td>\n",
       "      <td>13.8</td>\n",
       "      <td>5.168623</td>\n",
       "      <td>2.669957</td>\n",
       "      <td>0.074168</td>\n",
       "      <td>1.525213</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dark Blond</td>\n",
       "      <td>Light Blond</td>\n",
       "      <td>51.2</td>\n",
       "      <td>59.2</td>\n",
       "      <td>-8.0</td>\n",
       "      <td>5.168623</td>\n",
       "      <td>-1.547801</td>\n",
       "      <td>0.436903</td>\n",
       "      <td>-0.884182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Dark Blond</td>\n",
       "      <td>Light Brunette</td>\n",
       "      <td>51.2</td>\n",
       "      <td>42.5</td>\n",
       "      <td>8.7</td>\n",
       "      <td>5.482153</td>\n",
       "      <td>1.586968</td>\n",
       "      <td>0.416008</td>\n",
       "      <td>0.946285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Dark Brunette</td>\n",
       "      <td>Light Blond</td>\n",
       "      <td>37.4</td>\n",
       "      <td>59.2</td>\n",
       "      <td>-21.8</td>\n",
       "      <td>5.168623</td>\n",
       "      <td>-4.217758</td>\n",
       "      <td>0.003713</td>\n",
       "      <td>-2.409395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Dark Brunette</td>\n",
       "      <td>Light Brunette</td>\n",
       "      <td>37.4</td>\n",
       "      <td>42.5</td>\n",
       "      <td>-5.1</td>\n",
       "      <td>5.482153</td>\n",
       "      <td>-0.930291</td>\n",
       "      <td>0.769703</td>\n",
       "      <td>-0.554719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Light Blond</td>\n",
       "      <td>Light Brunette</td>\n",
       "      <td>59.2</td>\n",
       "      <td>42.5</td>\n",
       "      <td>16.7</td>\n",
       "      <td>5.482153</td>\n",
       "      <td>3.046249</td>\n",
       "      <td>0.036653</td>\n",
       "      <td>1.816432</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               A               B  mean(A)  mean(B)  diff        se         T  \\\n",
       "0     Dark Blond   Dark Brunette     51.2     37.4  13.8  5.168623  2.669957   \n",
       "1     Dark Blond     Light Blond     51.2     59.2  -8.0  5.168623 -1.547801   \n",
       "2     Dark Blond  Light Brunette     51.2     42.5   8.7  5.482153  1.586968   \n",
       "3  Dark Brunette     Light Blond     37.4     59.2 -21.8  5.168623 -4.217758   \n",
       "4  Dark Brunette  Light Brunette     37.4     42.5  -5.1  5.482153 -0.930291   \n",
       "5    Light Blond  Light Brunette     59.2     42.5  16.7  5.482153  3.046249   \n",
       "\n",
       "    p-tukey    hedges  \n",
       "0  0.074168  1.525213  \n",
       "1  0.436903 -0.884182  \n",
       "2  0.416008  0.946285  \n",
       "3  0.003713 -2.409395  \n",
       "4  0.769703 -0.554719  \n",
       "5  0.036653  1.816432  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 4. If there is a main effect, we can proceed to post-hoc Tukey test\n",
    "pg.pairwise_tukey(data=data, dv='Pain threshold', between='Hair color')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If all groups have the same number of values, then all CIs will have identical lengths. If the sample size are unequal, then the SE depends on the sample size. The CI will be wider when sample size is small and narrower when the sample size is larger (see later section on multiple comparisons)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using scipy.stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F statistic = 6.791 with P value from the F distribution = 0.00411\n"
     ]
    }
   ],
   "source": [
    "# The one-way ANOVA tests the null hypothesis that two or more groups have the same population mean.\n",
    "F, p = stats.f_oneway(\n",
    "    data[data['Hair color'] == 'Dark Blond']['Pain threshold'],\n",
    "    data[data['Hair color'] == 'Dark Brunette']['Pain threshold'],\n",
    "    data[data['Hair color'] == 'Light Blond']['Pain threshold'],\n",
    "    data[data['Hair color'] == 'Light Brunette']['Pain threshold'],\n",
    ")\n",
    "\n",
    "print(f\"F statistic = {F:.3f} with P value from the F distribution = {p:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using statsmodels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                             OLS Regression Results                            \n",
      "===============================================================================\n",
      "Dep. Variable:     Q('Pain threshold')   R-squared:                       0.576\n",
      "Model:                             OLS   Adj. R-squared:                  0.491\n",
      "Method:                  Least Squares   F-statistic:                     6.791\n",
      "Date:                 Tue, 14 Sep 2021   Prob (F-statistic):            0.00411\n",
      "Time:                         13:33:32   Log-Likelihood:                -64.628\n",
      "No. Observations:                   19   AIC:                             137.3\n",
      "Df Residuals:                       15   BIC:                             141.0\n",
      "Df Model:                            3                                         \n",
      "Covariance Type:             nonrobust                                         \n",
      "=====================================================================================================\n",
      "                                        coef    std err          t      P>|t|      [0.025      0.975]\n",
      "-----------------------------------------------------------------------------------------------------\n",
      "Intercept                            51.2000      3.655     14.009      0.000      43.410      58.990\n",
      "Q('Hair color')[T.Dark Brunette]    -13.8000      5.169     -2.670      0.017     -24.817      -2.783\n",
      "Q('Hair color')[T.Light Blond]        8.0000      5.169      1.548      0.143      -3.017      19.017\n",
      "Q('Hair color')[T.Light Brunette]    -8.7000      5.482     -1.587      0.133     -20.385       2.985\n",
      "==============================================================================\n",
      "Omnibus:                        1.137   Durbin-Watson:                   2.021\n",
      "Prob(Omnibus):                  0.566   Jarque-Bera (JB):                0.979\n",
      "Skew:                           0.358   Prob(JB):                        0.613\n",
      "Kurtosis:                       2.150   Cond. No.                         4.70\n",
      "==============================================================================\n",
      "\n",
      "Notes:\n",
      "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\WIECKOWS\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\scipy\\stats\\stats.py:1541: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=19\n",
      "  warnings.warn(\"kurtosistest only valid for n>=20 ... continuing \"\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.sandbox.stats as sandbox_stats\n",
    "import statsmodels.api as sm\n",
    "from statsmodels.formula.api import ols\n",
    "\n",
    "model = ols(\n",
    "    formula=\"Q('Pain threshold') ~ Q('Hair color')\",\n",
    "    data=data\n",
    ").fit() # we use OLS regression\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>df</th>\n",
       "      <th>sum_sq</th>\n",
       "      <th>mean_sq</th>\n",
       "      <th>F</th>\n",
       "      <th>PR(&gt;F)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Q('Hair color')</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1360.7263</td>\n",
       "      <td>453.5754</td>\n",
       "      <td>6.7914</td>\n",
       "      <td>0.0041</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Residual</th>\n",
       "      <td>15.0</td>\n",
       "      <td>1001.8000</td>\n",
       "      <td>66.7867</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   df     sum_sq   mean_sq       F  PR(>F)\n",
       "Q('Hair color')   3.0  1360.7263  453.5754  6.7914  0.0041\n",
       "Residual         15.0  1001.8000   66.7867     NaN     NaN"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anova_table = sm.stats.anova_lm(model)\n",
    "anova_table.round(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sum_sq</th>\n",
       "      <th>df</th>\n",
       "      <th>F</th>\n",
       "      <th>PR(&gt;F)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Q('Hair color')</th>\n",
       "      <td>1360.726</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6.791</td>\n",
       "      <td>0.004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Residual</th>\n",
       "      <td>1001.800</td>\n",
       "      <td>15.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   sum_sq    df      F  PR(>F)\n",
       "Q('Hair color')  1360.726   3.0  6.791   0.004\n",
       "Residual         1001.800  15.0    NaN     NaN"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sm.stats.anova_lm(model, typ='II').round(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using R\n",
    "\n",
    "A pipeline for one-way ANOVA analysis in R can be the following:\n",
    "\n",
    "```R\n",
    "# data import\n",
    "data <- read.csv2(\"PlantGrowth.csv\", sep = ',', dec='.')\n",
    "data <- data[-c(1)] # drop the 'X' (index) column\n",
    "\n",
    "# EDA\n",
    "head(data)\n",
    "str(data)\n",
    "levels(data$group) # check group encoding\n",
    "summary(data)\n",
    "\n",
    "# visualization\n",
    "boxplot(weight~group, data)\n",
    "\n",
    "# setup ANOVA as a linear regression model\n",
    "fit = lm(weight~group, data)\n",
    "\n",
    "plot(fit) # plots the following:\n",
    "# 1. Residuals vs Fitted\n",
    "# 2. Normal Q-Q\n",
    "# 3. Scale-Location\n",
    "# 4. Constant Leverage\n",
    "\n",
    "# ANOVA table\n",
    "aov = anova(fit)\n",
    "aov\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing manually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 4 different conditions and 19 total values\n",
      "This leads to 3 and 15 degrees of freedom for the effect (SSB) and error (SSW) Mean Squares (MS) respectively\n",
      "With SST = 2362.5 and SSW = 1001.8, we obtain SSB = 1360.7, and eta-squared = 0.575962\n",
      "MS for SSB = 453.6 and for SSW = 66.8\n",
      "Finally, with a F ratio = 6.791407, the associated P value = 0.004114\n"
     ]
    }
   ],
   "source": [
    "# parameters of the analysis\n",
    "r = data['Hair color'].nunique() # number of groups/conditions\n",
    "N = len(data)                    # number of total values\n",
    "DF_between = r - 1               # degrees of freedom for the effect Mean Square\n",
    "DF_within  = N - r               # degrees of freedom for the error  Mean Square\n",
    "\n",
    "print(f\"There are {r} different conditions and {N} total values\")\n",
    "print(f\"This leads to {DF_between} and {DF_within} degrees of freedom for the effect (SSB) and error (SSW) Mean Squares (MS) respectively\")\n",
    "\n",
    "\n",
    "# sums of squares\n",
    "SST = (\n",
    "    (\n",
    "        data['Pain threshold'] -\n",
    "        data['Pain threshold'].mean()\n",
    "    )**2\n",
    ").sum()\n",
    "\n",
    "SSW = (\n",
    "    data.groupby('Hair color')['Pain threshold']\n",
    "    .transform(lambda x: (x - x.mean())**2) \n",
    ").sum()\n",
    "\n",
    "SSB = SST - SSW\n",
    "\n",
    "print(f\"With SST = {SST:.1f} and SSW = {SSW:.1f}, we obtain SSB = {SSB:.1f}, and eta-squared = {SSB/SST:.6f}\")\n",
    "\n",
    "\n",
    "# MS\n",
    "MS_between = SSB / DF_between\n",
    "MS_within  = SSW / DF_within\n",
    "\n",
    "print(f\"MS for SSB = {MS_between:.1f} and for SSW = {MS_within:.1f}\")\n",
    "\n",
    "\n",
    "# F ratio and associated P value\n",
    "Fratio = MS_between / MS_within\n",
    "Pvalue = stats.f.sf(Fratio, DF_between, DF_within) # using here the `survival function 1 - cdf()`\n",
    "\n",
    "print(f\"Finally, with a F ratio = {Fratio:.6f}, the associated P value = {Pvalue:.6f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SSB computed from scratch = 1360.7\n"
     ]
    }
   ],
   "source": [
    "# we could also compute SSB from scratch\n",
    "SSB = (\n",
    "    data.groupby('Hair color')['Pain threshold']\n",
    "    .apply(lambda x: x.count() * (x.mean() - data['Pain threshold'].mean())**2) \n",
    ").sum()\n",
    "\n",
    "print(f\"SSB computed from scratch = {SSB:.1f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ANOVA table\n",
    "\n",
    "| Hypothesis  | Scatter from | SS     | Variation | $\\eta^2$ |\n",
    "| ----------- | ------------ | ------:| ---------:| -------- |\n",
    "| Null        | Grand mean   | 2362.5 |  100.0%   |          |\n",
    "| Alternative | Group mean   | 1001.8 |   42.4%   |          |\n",
    "| Difference  |              | 1360.7 |   57.6%   |  0.5760  |\n",
    "\n",
    "Of all the variation, 42.4% is the result of variation within the groups, leaving 57.6% of the total variation as the result of differences between the group means.\n",
    "\n",
    "| Source variation | SS     | DF  | MS    | F ratio | P value |\n",
    "| ----------------:| ------:| ---:| ----- | ------  | ------- |\n",
    "|  Between groups  | 1360.7 | 3   | 453.6 |  6.791  |  0.0041 |\n",
    "| + Within groups  | 1001.8 | 15  |  66.8 |         |         |\n",
    "| =         Total  | 2362.5 | 18  |       |         |         |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualization of the F-distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEMCAYAAAArnKpYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA5y0lEQVR4nO3deXwU9f348dd7N/dBEpLIfd8oior3hfWoWopHvauV1qsetXi1av1Z69F6ttqviuCFRxWsqAXFCwRvBayolUtAblGBECAByfH+/TGTsIQck2R3Z3fyfj4en0d2Z2dn3pt8Mu+dz8zn8xFVxRhjjAEI+R2AMcaYxGFJwRhjTC1LCsYYY2pZUjDGGFPLkoIxxphalhSMMcbUsqTgIxFREenrPn5YRP5flLbbXUS2iEjYfT5TRC6Ixrbd7b0mIudFa3sR280UkSkiUioi/4729qNJHE+ISImIzHKXXSIi37m/+0K/Y4wnEblZRJ7xOw7TepYUWklElonIVvdAUFM6N3c7qvpbVb3V4/6ObmJbK1Q1R1WrmhtHPfvb5Z9dVY9X1Sdbu+16nAp0AApV9bTWbkxEhotIdcTfZZWIPC8i+9VZT0WkLGK9jR42fyhwDNBVVfcXkVTg78Cx7u9+fQtj7unGk9KC9x4pIl+KyEYRWS8iL4lIl5bEEU0x/jv4SkROF5EPRaRcRGbW83rdz/SoD2E2iyWF6Pi5eyCoKWv8CqQlB5ME0gNYpKqVzX1jI597jarmALnAgcAC4D0ROarOentF/P3yPca6TFXL3OcdgAzgq+bGHkXzgJ+68XcGvgbG+BhPpFj9HXZStx64Z3Sej3PNXR/YANwH3NHIOpGfKWpn7DGjqlZaUYBlwNEe170W+BZYA/wGUKCv+9p44Db3cRHwCrARp9K9h5PAnwaqga3AFuAPQE93O+cDK4B3I5aluNubCfwNmAVsAv4DtHdfGw6squ8zAccB24EKd3+fR2zvAvdxCLgRWA58DzwF5Lmv1cRxnhvbOuBPDfxu/lJnX+d73Hbt565nm7t8Nnf5A8CciOe1f4c663UGJgE/AN8AV7jLzwe2AVVurM8BZe52tgBvu+sNBN5y/4YLgdMjtp0J3Ot+tlLgfXfZiojtbAEOamG9THf/5vMaWec6YAmwGSehnBzx2ig3pnuAEvfzHx/xei/gHfe9b7m/02ca2E9M/g7uazcDLwDP4NTtC3Dq5+3ABzj/K32Bg4HZ7u96NnBwxDZ2Wb8Fv+8LgJn1LK/3MyVy8T2AZC94TAo4B9jvgD2AbOBZGk4KfwMeBlLdchgg9e2PHQfHp9ztZlJ/Ulgdse9JNf/A9f3DRu7D/ad7ps7rM9mRFH4DLAZ6AznAi8DTdWJ7xI1rL+BHYFADv6Od9uVx27Wfu57t7fLZ3OU/wUmu2e7zXf5xcRLSp8BNQJobw1Kcb+LgHjTr+TvU/M6zgZXAr4EUYG+cpDjYff1B9/fYBQjjHLTS627HXfdQnC8IDZVDI9bt7i6rxkmwoxqpk6fhHHBDwBk4ia1TxOerAC5047sE58tMTT38CKe5LB04HCc5NDcpROPvcLMb50nuupnu73UFsLv7u++Ak9jOdZ+f5T4vjKjPkeunAg818vv+op7P0lhSWAOsxam/Pf0+ZjV5rPI7gGQvOAfQLREV5uUG1nscuCPieX8aTgq34Hybr+9b0zLqTwq961kWmRQi9z0Y51t5uL5/WJqXFKYDl0a8NsD9J02JiKNrxOuzgDMb+B3ttC+P2+5d37bc9Xf5bO7yge57u7jPFedbZs3f8J/AAcCKOu+7HnjCfTyKxpPCGcB7dd4/FvgzzsFrK06zQt3YdtpOK+ple+CPwIHNeM9c4MSIz7c44rUsN66OOImnEvdg7r7+bN16Eqe/w83UOUt06+ctEc/PBWbVWecj3IRZd/0W/r4bSgqH4ySzfJwzo/+19m8b65LM7c+J5CRVndbEOp1xvvHUWN7IunfjVPY3RQRgnKo21mYJzrdSr68vx/k2VNTEe7zozM6fZTk7vp3VWBvxuBznW3+0tt3U565PF5wD0MaIZfuo6uKaJyJyOtC5zsXOME5Tnhc9gAPqvD8FpwmwCOf6w5LmBu6Vqm4QkSeBz0Wki9ZznUZEfgVchZOIwPm7RNaJ2r+bqpa7dbFmnRLdcT0FnL9Nt2aGGa2/Q311IHJZ3XpUE2/kRfiW1KMmqeq77sPtIvJ7nKQ3CPgyFvuLBksK8fMtO//TdG9oRVXdDFwNXC0iewBvi8hsVZ2O809U79ua2H/dfVfgNGeU4XwLBMC9jbW4Gdtdg3MAjNx2JU5TWdcm3tsUL9tuKr76nAz8t85Bra6VwDeq2q8F2695/zuqekzdF9wLmduAPsDndV7e5fOIyGHAa43s63hVrS9ZpQC7Ae1wrmtEbrMHTrPeUcBHqlolInMBaWQ/Nb4FCkQkO+J32L2+2JsQrb9DffuNXFa3HoET7+sNbUNEHgbOaWB/y1V190biaYzi7XfsG7v7KH6eB0aJyGARycJpRqiXiIwQkb7ifDUrxbmgWe2+/B1Ou2pznROx71uAF9S5ZXURkCEiP3Nvq7wRp524xndAz0buyHgOuFJEeolIDvBXYGJ930xbIGrbdu8q6SIif8Y51b+hibfMAjaLyB/F6T8RFpE96t5G2YhXgP4icq6IpLplPxEZpKrVOM2JfxeRzu62DxKRdJyLqdVE/I1V9T3d+e62uuU99zOeIiIDRCQkIsU4bf6fqeqGXcMjG+cA9YP73l/jXHNqkqouB+YAfxGRNBE5FPi5l/f68HcAmIrztzhbRFJE5AycJtRXGnqDOreIN/T7rk0IbjwZOAk4JCIZ7v8RIrK7iAx118nBubFgNTC/GbHHnSWFOFHV13BuXXsb5+Lp242s3g+YhnOt4iPgIVWd4b72N+BGce5Fv6YZITyNc91iLU7TxRVuXKXApcCjOBW2DFgV8b6aTmTrReS/9Wz3cXfb7+LcGbIN+F0z4mpMNLbdWURq7uSZDQwBhqvqm429yU2YI4Ch7r7X4fyO8rzs1D3bOxY4kx0XGu9kR8K9BqcJYTbOt/g7gZCqluPeCeP+jQ/0/Emd5pDXcS76fomTXE5uIL55OAepj3AS/xCcu2+8OhunvX8Dzhecp5pY35e/g7uN9e42rgbW49y1N0JV13ndRiPOxbk+NAbnhpCtOGdg4DRzTsRpMlqK00w3QlUrorDfmKm5k8AYY4yxMwVjjDE7WFIwxhhTy5KCMcaYWpYUjDHG1LKkYIwxplZSd14rKirSnj17+h2GSRYLFzo/BwzwNw4TOAvXOnVrQMfkqFuffvrpOlUtru+1pE4KPXv2ZM6cOX6HYZLF8OHOz5kz/YzCBNDwu4cDMPPamb7G4ZWINDjMTtySgogcB9yPM27Jo3XH8hGRfwBHuk+zgN20BeOpG9OgG2/0OwITUDf+LDh1Ky5JwR1P50GcmapWAbNFZLLbqxIAVb0yYv3f4Qw1HDVffvklY8eOZd999+Xkk08mPz8/mps3yeDoRiesM6bFjh4cnLoVrwvN++MMw7tUVbcDE4ATG1n/LJxxb6LiscceY9iwYTz08EP85je/od+QfqxcHZNBEU0imzvXKcZE2dwVc5m7Yq7fYURFvJJCF3YemnYVOw9bW8sdvbEXjY8N5NmaNWu47LLLyBqaxdDXh9JvTD82bNjAsKOGsXHjxmjswiSL0aOdYkyUjZ44mtETR/sdRlQk4i2pZ7JjBM9diMhFIjJHROb88MMPTW7s7rvvpqKygh439kDaC7n759L7nt58//X3nHrNqdGO3Rhjklq8ksJqdh7Pv6u7rD5n0kjTkaqOU9VhqjqsuLjeO6pqfffdd4wdO5bCEwoJdwnXLm93QDuKTili+vjpzPxsptfPYIwxgRevpDAb6OeOi5+Gc+CfXHclERkIFOAM59tqzz77LFu3bqXzqM67vNb5ks6Es8Oc97vzorErY4wJhLgkBXdSlMuBN3AmmHheVb8SkVtEZGTEqmcCEzRK43lPmzaN7J7ZhHuGd3ktJT+Fjr/uyIoPVjDx3YnR2J0xxiS9pJ5PYdiwYdpQ57WKigoK2heQfXw2Xa+rf1bIqi1VfHnCl/Q+vDdfT/06lqGaRPDhh87Pgw/2Nw4TOB8udurWwX2To26JyKeqOqy+15K6R3NjZs2aRdmWMrod0PBc4uGcMMWnFrP46cV8MP8DDhl0SBwjNHFnycDESLIkAy8S8e6jqJg+fToIpO2T1uh6xWcVIyL88R9/jFNkxjcffrjjbMGYKPpw8Ye1ZwvJLrBnCtOnTyd3UC4peY1/xLTiNPKOyOOTFz6h9J+l5GV4nvrVJJsb3DnibewjE2U3vOTUrWQZ+6gxgTxTqKqqYtasWeTsneNp/cKTCqksqeSOCXc0vbIxxgRYIJPCsmXL2LZtGzl9vSWFdge2I7VDKuMfGx/bwIwxJsEFMinMnz8fgJSe3lrHJCwUjihk7Qdrmf3N7FiGZowxCS2QSWHePGfw1dReqZ7fU3BcASjc9cxdsQrLGGMSXiAvNM+fP5/0onRScr1/vMzemWT0zeCtF99Cb1REJIYRGl/cd5/fEZiAuu+M+/wOIWoCeaYwf/580nulN/t9BccWUDq3lOmLpscgKuO7oUOdYkyUDe0+lKHdh/odRlQELimoKvPmzyOzV2az31twbAEAf3/679EOyySCadOcYkyUTZs3jWnzglG3Atd8tGbNGjZv2sxuvXdr9nszumWQNSiL919+n+pbqwlJ4HJm23bbbc5Pm4HNRNltrzp1KwgzsAXuqFdz51Faz8Z7Mjek4NgCNn+1mVf/92o0wzLGmKQQ2KTg9XbUuvKPyQfggWceiFZIxhiTNAKXFJYvX044PUxKUcuSQnqndLL3zOaDlz+gWqujHJ0xxiS2wCWFNWvWkFac1qpbSguOKaBsURmvfm5NSMaYtiVwF5pXr15NSnHrPlbekXmsuncV414Yx8+H/jxKkRnfjR3rdwQmoMaeG5y6FbyksGY1qX2892SuT3qndDL7Z/L+1PfRW60jW2AMGOB3BCagBnQMTt0KVPORqrJ69WoyijNava28I/LY+PlG5qysf2Y3k4SmTHGKMVE25fMpTPk8GHUrUElh48aNbNu6jYwOrU8K+UfkQzU8OOnB1gdmEsO99zrFmCi79817uffNYNStQCWFNWvWALT4zqNImQMzSe2QyhuvvNHqbRljTLIIVFJYvXo1AKHi1n8sESH/iHy+//B7lm1c1urtGWNMMohbUhCR40RkoYgsFpHrGljndBGZJyJficizzd1HzZlCuCjcymgdeUfkUb2tmjFTxkRle8YYk+jikhREJAw8CBwPDAbOEpHBddbpB1wPHKKquwOjm7ufmjOF1OLW3X1UI2ffHELZIV7+z8tR2Z4xxiS6eN2Suj+wWFWXAojIBOBEYF7EOhcCD6pqCYCqft/cnaxevZq0vDRCGdHJdaHUEHmH5rH07aVs2b6FnDRv03uaBPX0035HYALq6fODU7fi1XzUBVgZ8XyVuyxSf6C/iHwgIh+LyHHN3cmaNWuidpZQI++IPCpLKnlixhNR3a7xQbduTjEmyrq170a39sGoW4l0oTkF6AcMB84CHhGR/LorichFIjJHROb88MMPO722evVqwsXRuZ5Qo93B7SAMz77c7EscJtFMnOgUY6Js4uyJTJwdjLoVr6SwGohMo13dZZFWAZNVtUJVvwEW4SSJnajqOFUdpqrDiouLd97AmlWkFzd/xrXGpOSmkLN3Dl+89YUNkJfsxoxxijFRNmbmGMbMDEbdildSmA30E5FeIpIGnAlMrrPOyzhnCYhIEU5z0lKvO6iqquL7td+TsVvrO67VlXdYHuVLynlr3ltR37YxxiSSuCQFVa0ELgfeAOYDz6vqVyJyi4iMdFd7A1gvIvOAGcC1qrre6z5KSkqorq4mraBlk+s0Ju/wPAAeffHRqG/bGGMSSdwGxFPVqcDUOstuiniswFVuabaSkhIAwu2ie00BIKN7Buk90nnntXfg/0V988YYkzAS6UJzq9QkBcmNzYimeYfnsW72OpasWxKT7RtjTCIIXFLQXI3J9vMOz0MrlYcnPxyT7Zs4eOEFpxgTZS/89gVe+G0w6lbgkkIsmo8AcvbMIdwuzH+m/Ccm2zdxUFTkFGOirCi3iKLcYNStwCWFlNzYXCaRFKHdwe34ZuY3lFeUx2QfJsbGj3eKMVE2/oPxjP9gvN9hREVgksKGDRuA2J0pgNOEVLmxkqdnBKdLe5tiScHEyPgPxzP+w/F+hxEVgUkKJSUlhDPChNJj95Fqejc/8/IzMduHMcb4KVBJIaVdbO+wrend/Nmbn+HcQWuMMcESqKQQzo1d01GNvMPyKFtSxrsL3o35vowxJt4ClRRCubH/ODW9mx958ZGY78sYY+Itbj2aY21DyQbCebE/U6jp3Txt6jT4U8x3Z6Jp6tSm1zGmBaZeEZy6FZgzhQ0lG0hrF/1xj+qTd3ge38/6nlUbVsVlfyZKsrKcYkyUZaVnkZUejLoVmKRQsqGE1LzoTrDTEOvdnKQeesgpxkTZQzMe4qEZwahbgUgKFRUVlG0pIzU3Pkmhpnfzy1Nejsv+TJQ8/7xTjImy5+c8z/NzglG3ApEUNm7cCMS241qkmt7NC99eyPbK7XHZpzHGxEMgkkKsxz2qT03v5udmPhe3fRpjTKwFKinEatjs+rQ7uB0SFp56+am47dMYY2ItUEmB3PjtMyU3hey9s5n1xqz47dQYY2IsEP0U4jEYXn3yDstj9T9W88mCTzhg4AFx3bdpgZkz/Y7ABNTMa2f6HULUBOpMIdZjH9VV07v54Rft1lRjTDAEKinE+0yhpnfzm6++Gdf9mha65x6nGBNl97xxD/e8EYy65SkpiEh7EfmriEwVkXcjS6wD9KKkpIRwZphQavxzXN7heXw761vWlqyN+75NM73yilOMibJXvniFV74IRt3yehR9FjgImAI8Vqf4bvPmzaRk+3N5xHo3G2OCxGtSOBg4TlXHqOqTkcXrjkTkOBFZKCKLReS6el4fJSI/iMhct1zgddvl5eVIRvxuR41U07t50pRJvuzfGGOiyevX6y+ArsCSluxERMLAg8AxwCpgtohMVtV5dVadqKqXN3f7ZWVlhDL9uTxS07t5wfQFVFRWkJoSn6E2jDEmFrweSd8GXheRG0TkN5HF4/v3Bxar6lJV3Q5MAE5sScD1KS8vR9L9OVOAHb2bJ7wzwbcYjAeZmU4xJsoy0zLJTAtG3fJ6pnAYzjf8Y+osV+BxD+/vAqyMeL4KqO/G/l+IyOHAIuBKVV1ZdwURuQi4CKB79+6Ac6bgV/MR7Ojd/ORLT3LuUef6Fodpwmuv+R2BCajXfh+cuuUpKajqkbEOBOci9nOq+qOIXAw8CfyknljGAeMAhg0bpgBl5WWEc+J7O2qkmt7Nn7z+iW8xGGNMNHhuiBeRAhH5lYhc7/4saMZ+VgPdIp53dZfVUtX1qvqj+/RRYF+vGy8rLyOc4V9SAKd385YlW/howUe+xmEaceutTjEmym595VZufSUYdctrP4WDcC4y/xbYE7gYWOIu92I20E9EeolIGnAmMLnOPjpFPB0JzPe4bcrKykjJ8nfEjtrezZPs1tSENX26U4yJsunzpzN9fjDqltcj6X3ApapaeyVVRM4A/gns19SbVbVSRC4H3gDCwOOq+pWI3ALMUdXJwBUiMhKoBDYAo7x+iPLycjIz/L3IU9u7eeqbNnezMSZpeU0K/YG60wq9AHj+WqyqU4GpdZbdFPH4euB6r9uLVF5WTk5GTkveGlV5h+fx3XPfsWrDKrq27+p3OMYY02xeryl8jdPkE+k0WthvIZqqqqrY/uN2wpn+XlOAHb2bH/pPMOZqNca0PV6TwmjgARH5WEQmisgnwEPAFTGLzKOtW7cCEMrwf2y/mt7NL015ye9QTH0KC51iTJQV5hRSmBOMuuX1ltQPRaQP8DOgM87to1NVdUMsg/OirKwMwLcezZFqejcvnrGY8opyslKz/A7JRJpkQ5GY2Jh0SXDqlucjqaqWqOozqnqX+9P3hADORWbA185rkWp6N4+fMd7vUIwxptkaTAoi8nrE4/fqDpmdKENn154pJEDzETi9mwnDMy8+43copq7rr3eKMVF2/YvXc/2LwahbjTUfRc5I/2isA2mpmjMFMvyNo0ZKbgq5w3L5bOpnVFZXkhIKxIynwfCRdSw0sfHRkuDUrQaPWKr6bMTTBaq6yxgOIrJ/TKJqhtqkkO5vHJHyj8pn5V9XMmnWJM448Ay/wzHGGM+8trm81cDy1xtYHjc1zUeSmRjXFADyh+dDCMZNGOd3KMYY0yyNJgURCblzIYgrFFH64fQ+9lXNmUKiXFMASC1MJWfvHD6e8jGq6nc4xhjjWVNH0kpgO5DlPq6IKPNw+ir4KpFuSY2Uf1Q+5UvLeeW/wZi3NRC6dnWKMVHWtaArXQuCUbeaugraCxDgHeDwiOUK/KCqW2MVmFeJeKYAkP+TfFbdtYoxE8bw831/7nc4BuAZuyPMxMYzFwSnbjWaFFR1OYCIDACqVLWi5jURSRWR9Ijhrn2RqEkhrTiN7L2yeX/K++hdikjiXPMwxpiGeD2Svsmu8xvsizPqqa8SrZ9CpIKjC9i8cDPT/jfN71AMwOjRTjEmykZPGM3oCaP9DiMqvB5J9wTq3pI6C9gruuE0X3l5OaG0EBJOvG/i+UfmA/DAsw/4G4hxzJ3rFGOibO7KucxdOdfvMKLCa1LYCHSos6wDUBbVaFqgrKwsIUZIrU9apzSyds9i5uSZfodijDGeeE0Kk4BnRWQPEckSkSE4PZ7rzrEQd+Xl5QnZdFSj4OgCNs3bxDsL3vE7FGOMaZLXo+mfcKbHnAVsBj4GFgI3xCguzxI9KeT/JB+A+5+9399AjDHGA69DZ28DLnOn1CwC1mmC9MoqKytLuD4KkdK7ppM5IJMZ/5kBt/gdTRvXv7/fEZiA6t8hOHXL82htIpIHDABy3OcAqOrbMYnMo/LyciQ98S4yRyo4uoA1D67hva/f47B+h/kdTts1zoYdMbEx7lfBqVuevmKLyChgDc7kOo9FFN9HT91StiWhxj2qT/7R+QDc+/S9/gZijDFN8Nrucjtwqqp2UNVeEaV3LIPzoqy8jJTMxB6eOqN7BlmDs5gxaYaNheSniy5yijFRdtFTF3HRU8GoW16TQgpOB7YWE5HjRGShiCwWkesaWe8XIqIiMszLdsvLywlnJOYtqZEKjnPuQnrt89f8DqXtWrTIKcZE2aLvFrHou2DULa9J4U7gRhFp0RVdd6TVB4HjgcHAWSIyuJ71coHfs2tHuQYlcj+FSAXHFoDAP578h9+hGGNMg7we5K8EbgQ2i8iKyOLx/fsDi1V1qapuByYAJ9az3q04CWibx+0mzZlCWnEaOfvm8MFLH1BVXeV3OMYYUy+vSeEc4GjgBODcOsWLLsDKiOer3GW1RGQfoJuqvtrYhkTkIhGZIyJzfvjhB7aWbU2KMwWA9se1Z+vyrUz4YILfoRhjTL289lOIaXdct1nq78AoD7GMA8YB7LvvvrpixYqkOFMAd5rOO1bywFMP8MvDful3OG3P0KF+R2ACami3oX6HEDWekoKINNjtSlVv8rCJ1UC3iOdd3WU1coE9gJlu/4eOwGQRGamqcxraaHV1NZB4E+w0JKVdCu0Oacd/J/+XH8f8SHpKAk0s3Rbcd5/fEZiAuu/M+/wOIWq8Hk271Sn7AdcAfTy+fzbQT0R6iUgacCYwueZFVS1V1SJV7amqPXGG0Wg0IUBEUkjgYS7qKvhpAdu/386Y18f4HYoxxuzC09FUVX9dpxwPnILHOZpVtRK4HGf+hfnA86r6lYjcIiIjWxp8TVKQ1MTuvBYp//B8Qtkhxo0PTg/IpHHOOU4xJsrOefQcznk0GHWrNb2+3gQmel1ZVacCU+ssq7fpSVWHe9ym8yA5LikATlNXwdEFLHxtIWtL19Ixr6PfIbUdq1b5HYEJqFUlwalbXoe56F2n7AHcxs53FPkniZICQPsR7akur+Zv//qb36EYY8xOvDbGLwa+dn8uxmnzPww4L0ZxeZKMZwoAOUNzSOuSxsRnPJ9oGWNMXDSYFESkdqpNVQ2patj9GVLVHFU9TFU/jU+Y9atNCslznRkACQmFIwr57uPv+GSx587bxhgTc40dTt+reSAiX8chlpZLsjMFgPY/aw8Ktz16m9+htB0HHeQUY6LsoD4HcVCfYNQtaWjUTncIi0uBecAXwBBgl9t8VHVpLANszKBBg3TBggX0u78fuYfm+hVGiy26cBHV66vZtHQT4VASZjZjTFISkU9Vtd5BRxs7U/g9cB/OtJuZwBJ2XFOoKb6eQSRr81GN9iPaU768nPEzxvsdijHGAI0cTlX1JVXtq6qpQHnE9YTIkhhfbxN7OoUGFRxdQCgjxP2P2PzNcfGLXzjFmCj7xZhf8IsxwahbXg+nhTGNooWS/UwhnB2m4KcFfDX5K1aXrKZLQZem32Rabv16vyMwAbV+S3DqltcezdtjHUhL1CQFCSdPj+a6ik4uonprNTc/frPfoRhjTLJ+x64jMRqxWiRrjywy+mbw7yf+bVN1GmN8l9RJofZMIZS8ZwoiQtHJRZR+VcqLH73odzjGmDbO6zAXJ4pIwl3ODULzEUD7E9oj6cLfHrJhL2LqqKOcYkyUHTXoKI4aFIy61WA/hZ1WEvkc6IwzAN7TqpoQ3XD79u2rS5YsYeBzA8nqn+V3OK2y7KZllM4sZfWa1XRo18HvcIwxAdbSfgq1VHUvnOk4twKTRGShiNwoIj2jF2bzBaH5qEbRyUVUlVVx8xM3+x2KMaYN83xNQVU/V9VrcSbZuQw4DVgiIu+KyC/dKTXjKijNRwDZQ7PJ6J3Bc2Ofq50nwkTZ8cc7xZgoO/7+4zn+/mDUrWYdyEWkD3ATMAbIcB8/gjOBzgtRj86rJL77qIaIUHx6MaXzS3l6xtN+hxNMW7c6xZgo27p9K1u3B6Nueb3QfJmIfAzMAjoA56rqAFW9XVWfBo4Cjo1hnPUK0pkCOIPkhXPC3PHPO/wOxRjTRnk9UzgeuBforKqXqurHkS+qajnO9JxxFaRrCgDhrDCFIwtZOHUh81bO8zscY0wb5DUpzFTVf6vqj5ELReSqmseq+mZUI/MgWSfZaUzRaUVopfKH+//gdyjGmDbIa1Kody5l4MZoBdIaQWk+AsjonkG7Q9rx1pNvUbatzO9wgmXECKcYE2Uj9hzBiD2DUbca7ZAmIj+pWU9EjmTn+RR6A5tjFZgXQWs+qlF8RjFLrljCLU/fwp0X3ul3OMFxzTV+R2AC6pqfBqduNXWm8Jhb0oHHI54/CvwG+J3XHYnIcW7/hsUicl09r/9WRL4Ukbki8r6IDG5qm7XNRwnX17p12h3UjvTu6Tzyf4/Y7anGmLhqNCmoai9V7QX8q+axW3qr6sGqOtnLTkQkDDyIc8F6MHBWPQf9Z1V1iKoOBe4C/u71QwTtTEFCwm5n70bJlyWMf2u83+EEx/DhTjEmyobfPZzhdw/3O4yo8Nqj+Vet3M/+wGJVXeoOwz0BOLHOPjZFPM0Gmhx/I6jNRwCFIwpJyU/h1rtu9TsUY0wb0mBSEJH5EY9XisiK+orH/XQBVkY8X+Uuq7vPy0RkCc6ZwhUNxHWRiMwRkTlbtmxxFgas+QgglBmi6PQilr29jBlfzPA7HGNMG9HY4fTCiMfnxDoQAFV9EHhQRM7GubPpvHrWGQeMA+jSpYuWlpYG8kwBoPj0Yr578juu+ds1fPrcp36HY4xpAxpMCqr6fsTjd1q5n9U4YybV6Ooua8gEnKE0GpXs03E2JbUglcIRhXw26TMWrFzAwG4D/Q7JGBNwXoe5uEpEhrqPD3Sbjr4RkYM87mc20E9EeolIGnAmsNNFahHpF/H0Z8DXTW1UVZGwIBLMMwWA3c7ZDa1Ufn/37/0OJfmdfrpTjImy04edzunDglG3vM6nsBLYQ1VLRWQG8B+cPgoXqeoBnnYkcgJwH07/48dV9XYRuQWYo6qTReR+nOG5K4AS4HJV/aqxbXbq1Em/X/89Qz8e6iWEpLX02qVsnrWZFctX0Ll9Z7/DMcYkucbmU/CaFDapajsRyQWWA8WqWiUiG1U1P7rhetexY0ddt2kde72/l18hxEX5/HIWnLOAk/9wMi/eaVN2tlh5ufMzK7knZDKJp/xHp25lpSdH3Wr1JDvAShE5GKfZ5103IbQDqqIVZEuoamAvMkfKGpRFu0PbMWXsFNaVrvM7nOR1wglOMSbKTvjnCZzwz2DULa9J4Vqc+RL+BNTcOD8CZyhtXwVp3KPGdLqgE5WllVxxT7136hpjTFR47bw2VVU7q2pPVa25N/LfwMjYheYprkCNkNqY7CHZ5B6YywsPvkDJ5hK/wzHGBJTnmzlFJE9E9heRn7gD5R3mFt+0leajGp0u6ERFSQWj7xvtdyjGmIDyekvqKGANMIUdg+LVDIznq7bSfASQs3cOOcNymHD/BDtbMMbEhNcBIm4HTlXV12IZTHPV9FNoSzpf0plF5y/ikjsuYcLtE/wOJ7mMGuV3BCagRh08yu8QosbrLanf4UzF6evdRnUVFhZqeVY5g//T5CjbgbJ49GLK55az/JvldC60fgvGmOaJxi2pdwI3ikhCDSjR1q4p1Oh8WWcqt1Ry4c0XNr2y2WHdOqcYE2XrNq9j3eZg1K3m9GjuCGwH1ke+pqrdYxNa09q3b6/b8rcx6IVBfoXgm2/+9A2lM0tZsGgB/br1a/oNZsdcCjNn+hmFCaCauRRmXjvT1zi8auxMwes1hbiMktpcbfGaQo3Ov+3Mxmkb+fWNv+b9J99v+g3GGOOBp6QQhVFSY0JVAztCalPSu6VTeFIhHz77IR9d8xEHDfE6NqExxjTM6y2p6SJyu4gsFZFSd9mxInJ5bMPzEFsbPVMA6HRxJyRd+NXo1k6MZ4wxDq/fs/8B7AH8kh3TZH4FXBKLoLxqSz2a65PaPpWO53dk8duLeWLKE36HY4wJAK/XFE4G+qpqmYhUA6jqahHZZUrNeGrLzUc1djtrN9ZNWsdVV1/FOcedQ2pqqt8hJa5LfP0OYwLskuHBqVteD6nbqZNARKSYOnci+aEtNx8BhNJCdPl9FzZ+vZE/PvBHv8NJbGec4RRjouyM/c7gjP2CUbe8JoV/A0+KSC8AEekEPIAzbaZv7EzBkf+TfHL2yeGBWx9g7fq1foeTuFaudIoxUbZyw0pWbghG3fJ6SL0B+Ab4EsjHmSpzDfCX2ITljaoiKW37TAFAROh6TVcqSis4/cpgTAkYE+ee6xRjouzcx87l3MeCUbe8Dp29XVWvVNUcoAOQ6z7fHtvwmoyrTfZork/WgCyKzyjmvWfe49X3X/U7HGNMkvJ6S+pgEblYRK4HTgESoguxYkkhUuffdia1KJVRF4+isrLS73CMMUmo0aQgjsdxmo1uwJlU50/AFyLyhIj4e0RWrPkoQjgnTNeru7Ju3jquvOdKv8MxxiShps4ULgKGAweqag9VPcgd6+ggnAl2Lo5xfI2y5qNd5R+dT7uD2jHmtjEsWr7I73CMMUmmqaRwLnCFqs6OXOg+H+2+7omIHCciC0VksYhcV8/rV4nIPBH5QkSmi0iPprap2IXmukSEbtd1Q6uUEb8egZcBD9uMq692ijFRdvWxV3P1scGoW42OkioiG4Aeqrq5ntdygRWqWtDkTkTCwCLgGGAVMBs4S1XnRaxzJPCJqpaLyCXAcFVt9Mbf9Ix0zR2eS4+/Npk/2pzvn/ueVfes4i8P/4WbLr7J73CMMQmkNfMphOtLCADucq+3tO4PLFbVpe4dSxOAE+tsb4aqlrtPPwa6NrlVxZqPGlB8RjE5e+dw67W3snTVUr/DSQwLFzrFmChbuHYhC9cGo241NcxFqvsNvqEjr9dhMroAkT07VgEHNLL++UCTU39a81HDJCR0v6k7C85awM9G/Yx5b83D7/sCfHexewnM5lMwUXbx007dSpb5FBrT1EH9e+DxJl6PKhE5BxgGHNHA6xfhXAAnFA7ZmUIjMrpn0OnSTiz4+wJuHnMzf7nU176Gxpgk0GhSUNWeUdrPaqBbxPOu7rKdiMjROLe8HqGqPzYQ0zhgHEBqaqqGUmyci8bsdtZulL5byu3X3M5JR57E3oP29jskY0wCi9cRdTbQT0R6iUgacCYwOXIFEdkbGAuMVFVPZyCKjX3UFAkJPW/piaQJx59+PNu3+9oJ3RiT4OJySFXVSuBy4A1gPvC8qn4lIreIyEh3tbuBHODfIjJXRCY3sLmIDduFZi/SOqTR/cbufPe/7zjn2oScWdUYkyAavSU10YXDYe14Vkc6XtXR71CSworbV7DupXX8a8q/OPtnZ/sdTvxNm+b8PPpof+MwgTNtnlO3jh6cHHWrsVtSkzophMIh7fzLznQY3cHvUJJC1dYqFp63kOoN1cz7bB59evTxOyRjjA9a008hsalzB5LxJpwZpvddvanaXsURJx3R9q4vzJ3rFGOibO6KucxdMdfvMKIiqY+obX2O5pbI6JlB9z93Z/Xc1Zxy6Sl+hxNfo0c7xZgoGz1xNKMnjvY7jKhI6qQANh1nSxQcVcBu5+zGq4+9yl2P3OV3OMaYBJL8ScHuPmqRLpd3IWdYDtdddh2vvdNk53FjTBuR/EnBzhRaRFKF3nf1Jq1jGiefcjJff/O13yEZYxJA0icFu6bQcil5KfS5rw+VFZUcctwhlJaW+h2SMcZnXge0S1h2ptA6GT0z6HVnLxb/bjEHjzyYz976jLS0NL/Dio2//tXvCExA/fXk4NStpO6nICLa++re5J+d73coSW/95PUs/8tyjvjFEbz9/NuEQsl/EmmMqV9w+ymANR9FSeHIQjpf3pl3Jr3D2ZefHcwZ2z780CnGRNmHiz/kw8XBqFvWfGRqdRjVgYr1FUwcM5EOHTpw/5/v9zuk6LrhBuenzadgouyGl5y61RbmU0h8yX+ukzBEhK5XdaVyYyX/vPmfZKZlcsf1d/gdljEmjpI+KdiZQnRJSOh5c0+0UrnzhjtJS03jlmtu8TssY0ycJH1SsGsK0ScpQq9be/FN1Tfceu2thMNh/nzln/0OyxgTB0nf+GI9mmNDUoWef+1J/vB8br7qZv5w6x/8DskYEwd2pmAaFEoN0euOXiy7eRl333Q3G0s3MvbusYgkaSK+7z6/IzABdd8Z9/kdQtQkf1JI+nOdxCapznSe4ewwj9z7CCUbS5gwdgLhcBJm46FD/Y7ABNTQ7kP9DiFqkv6QaheaY0/CQrfru9HhvA688NgLHDbiMMrKyvwOq/mmTdsx+5oxUTRt3rTa2deSXdKfKWg4gJ2sEpCI0OWKLqR1TOOjuz9iz0P25IPXP6BjxySaCvW225yfNh2nibLbXnXqVrJMx9mY5D9TsAvNcVV8ejG97+3NsoXLGDRsEHP+O8fvkIwxUZT0ScEuNMdf/uH59H+0P+WV5Rx4yIE8/q/H/Q7JGBMlcUsKInKciCwUkcUicl09rx8uIv8VkUoROdXrdjVkzUd+yBqUxYCnB5A1MIvzzzmfC6+8kIqKCr/DMsa0UlySgoiEgQeB44HBwFkiMrjOaiuAUcCzzdq4nSn4JrUwlT4P96HotCIeve9R9jx4T5YtW+Z3WMaYVojXheb9gcWquhRARCYAJwLzalZQ1WXua9XN2rIlBV+FUkN0v647Ofvk8PXtXzNoz0E89shjnH3G2X6HtquxY/2OwATU2HODU7fi1XzUBVgZ8XyVu6zV7JbUxND+2PYMfHYgKT1S+OWZv+TMX59JeXm532HtbMAApxgTZQM6DmBAx2DUraS70CwiF4nIHBGZA3b3USJJ75JOv0f70XFURyY+OZE+Q/ow852Zfoe1w5QpTjEmyqZ8PoUpnwejbsUrKawGukU87+ouazZVHaeqw2pnDbLmo4QiqULn33Wm30P9KPmxhCOHH8l5F5/Hpk2b/A4N7r3XKcZE2b1v3su9bwajbsUrKcwG+olILxFJA84EJkdjw9Z8lJhy989l4MSBdPhlB5565Cl6DerFlFeD8U3KmCCLS1JQ1UrgcuANYD7wvKp+JSK3iMhIABHZT0RWAacBY0XkKy/btuajxBXODNPlqi4MeGIAWzO2MnLESI458RiWLl3qd2jGmAbE7ZqCqk5V1f6q2kdVb3eX3aSqk93Hs1W1q6pmq2qhqu7uacNJP1BH8GUPyWbAvwbQ6dJOvP3m2/Qf1J+rr7uazZs3+x2aMaaOpLvQXJedKSSHUFqITud3YvBLgyk4poC/3/l3uvfrzmNPPEZVVZXf4RljXKKavD2CRUR3f3l30rul+x2KaaayL8tYfc9qtvxvC937d+euW+7itNNOIxSK4feUle5d0d26Nb6eMc20coNTt7q1T466JSKf1t6sU/e1pE8Kr+xOeidLCslIq5WNMzay9uG1bF26lb579OWe2+5h5MiRyTuRjzFJoLGkYM1HxjcSEgqOKmDghIH0vK0nq0pXcdJJJzFwr4H861//orKyMro7nDjRKcZE2cTZE5k4Oxh1K+nPFIa8MYTUolS/QzFRoJXKhqkb+P6p79n6zVY6dOvAH6/6IxdecCE5OTmt38Hw4c7PmTNbvy1jIgy/ezgAM6+d6WscXgX6TME6rwWHpAiFIwsZ+PxAev+jN1uLtnLVlVfRsVtHrrjyCr7++mu/QzQm8JI+KVjzUfBISMg/PJ++j/ZlwPgBZOyXwQMPPED//v059CeHMmnSJBum25gYSf6kYD2aAy17SDY97ujBHq/uQZdLuzB7/mxOPfVUOnTtwO+v/D2fffYZydwEakyiSfqkYM1HbUNqUSodzu/A7pN3p/c/ehPaI8T/PfB/7LPPPvTfoz933nknK1as8DtMY5Je0l9oHvrRUEJpyZ/bTPNVbqxk41sbKXm1hM1fOr2jh+wzhLNOPYtTTjmFAXWHyV63zvlZVBTnSE3Qrdvs1K2i3OSoW4Hup7D3J3sjKdaE1NZtW7mNzW9vpuTtErb8bwsAfQb24cxfnMmJJ57IvvvuG9uOccYkkWAnhTl7W0cns5Pt321n08xNlM4opfTTUqiGvPZ53N6nH3sO2ZPel11Al959Ib/Y71BNQIz/YDwAow4Z5WscXgU6Kezz6T5+h2ESWGVJJZs+3kT5x+U8//oPaKVyJLB7Nhzdu4jDDzyAw0aeRvFhP4O85Dj1N4knSP0UbIxRE2gpBSm0P7497Y9vT/aacrI3bOfu1O28uR7GfrWO+798FR55lQFZcGjXPI7Ydy8OPfYEeg4fgXQfBNbkZNqY5E4K1mpkmkMgLSPENd3gmp7wYzV8ugneK4H3NsKkpaU8tuhdeO5dilOvY1hBCsP6dGW/ffdhvyN/SscDjoROfSxRmECzpGDarPQQHJzvlD8C1Qr/2wIfbITZm2D2pkre+GgZ1R8tgwdepEs67JUXZkj3TgwZNJAh+x3AwMOOJq3vUMjJ9/GTGBM9yZ0UjImikMCeuU65xF1WVgWfbXKSxJxN8OWWKt76dBUVc1bB09NIkdvpnwVDCjPYo3tnBvTvT7899qLffgeT3W8vKO5mZxYmqST1heZQSkj3nrW332GYJJGxtZrD55Xwt38vb9V2tlfDonL4cjP8r8z5+eUWWLZt5/U6p0P/bKFfcR79u3aiT5/e9Ow/mB57DKWg3xCkY0/Iym1VLCYxlP9YDkBWepbPkXhjF5qNAbZlhqhIb/239rQQ7JHjlEhlVbC43EkYi8rg63JYVK68tHQj6xZuhOnzgVcByAlDjwzokZNCj8I8enTcjZ7du9K9Zx869+1Px76DSO/cC9p3chKH3Xad0JIlGXiR3EnB/k9MM5z2/A/0/7Y8ZtvPDsNeuU6pq6QClpTD8m1u2er+LK/ko/XrKZm3HpgPvFX7noIU6JQOnTJDdGqXRaeCPDoWFdKpYwc6delKx67dKerWk4KuPQm37+j0u8gpsOYqHzw04yEALj3yUp8jab3kTgrGNMPR00ooKKsEH2ZMLEiFYXlOqc/mSidJrNgK326Hb390y3ZY+2M176/ZwrfLtvBj9epd3ivu9otSoSgNCjNTKcrJpKhdDoV57SgqKKB9YXvy2heSV7gb+bt1IK+4E3m7dSK1oAiy8yCrnVPCNphYSzw/53nAkoL/7EzBBERuSv1NUpFUYWPljoSxdjusr4B1NT/dsmJzBf/dUMG6ik38WL2m0f1mhiAvBfJTnZ95aWHyMlLJz0wnLzuT3KxMcrKyyM7OJjsnh+ycXLJzc8nObUdOXj7ZeQVuySe7oJBwVi5kZEF6TcmEtExLNkkkbklBRI4D7scZ1/RRVb2jzuvpwFPAvsB64AxVXRav+IxJdCLOGUFBKgz2MBGdqnOdY30FbKiA0konqZQ2VrZXsbysitLKbZRWlrK1unkxpoec6yXZbskMQUYIMlKEjJSwU1JTnJKW6pY052d6Ghnp6WRkZOz4mZFJRmYG6RkZpKZnkJaeQWpGpvM4I4PU9ExSMzJJy6xZlkVqRgbh9ExISXVKOHXnx+EU93HKjueWtGrFJSmISBh4EDgGWAXMFpHJqjovYrXzgRJV7SsiZwJ3Amc0vuEYBWxMAIhATopTemS2bBuV1VBe7SSXLZVQ5j6ufV61c9lS5/m2ardUKZsqKtlWXcm2qh93LHdLZZRvghScGwJSxSk1j2uXhSDN/ZkqkCIQDgnhkJASEsIScp+HnOc1j8MhwqEdpeb5kgEbEIEbD+rpLg8TDocIh8LO8xTnZzgcIiWc4rwWDiMihMJhQqEwoVBoRwmHkcjnobC7Xs3jECIhd5nstDwUinxvuHZ7oXCIkPuexsTrTGF/YLGqLgUQkQnAiUBkUjgRuNl9/ALwgIiINnLPrFhWMCamUkLQLgTtUoD02O2nshp+VNhWxS4Jw0kqUKFO2V6988+Katju/tzp9XqWNfT+KoXtlUoVSpVCpVZRpbiPqX1c5cZaxc6vlXcBBf728XKaeXKVcOKVFLoAKyOerwIOaGgdVa0UkVKgEFgXuZKIXARcBCAZogvOWWCZwXhyBFC5sZoUCc7tgwbntEDwdcqwjKWwbcW2DdVUf+NfFM3So6EXku5Cs6qOA8b5HYdJTiIy58dvf6y3044xrSEicxrqEJZM4pVbV7PzjYBd3WX1riMiKUAezgVnY4wxcRKvpDAb6CcivUQkDTgTmFxnncnAee7jU4G3G7ueYIwxJvri0nzkXiO4HHgD55bUx1X1KxG5BZijqpOBx4CnRWQxsAEncRgTbdb0aGIlEHUrqQfEM8YYE102SIoxxphalhSMMcbUsqRgjDGmliUFEzwiFyPyLSJz3fKM3yEZkyzsQrMJHpEHgM9QfczvUEzwyYVyMc4QPd+5i/6nj+g5/kXUOknXo9kYD/YEnvA7CNNmDAFu1EeC8SXEmo9MEO0OPOE2HU3z/C6RgtiFZAJsT2Cu30FEiyUFEywi3YC1qO6J6lBUj25ifUHkVvfZP2Ienwmi3YEn5EKZKxc240tIgrLmIxM0Q4CvdlrizOdxN87oxsuB54GXgJdxJnZKxZkEaiAi16J6dzwDNslLLnS+hOgjuqffsUSLJQUTNHtSNynAJcB/UH0HwE0Az6H6T/fxXJwh2p9B9YE4xmqS365fQpKcNR+ZoBnCzpM3gTPF6wcRz4cCb0U8nouTTD6PbWgmgOr7EpLU7EzBBIvqL+tZ+jIwFpES4K9AP2Ch+1o/YBHQH7gAkXWozo9HqCYQhuDUr8CwfgrGGGNqWfORMcaYWpYUjDHG1LKkYIwxppYlBWOMMbUsKRhjjKllScEYY0wtSwrGGGNqWVIwxhhTy5KCMcaYWv8fKOnIPpq/mj4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "F_crit = stats.f(DF_between, DF_within).ppf(.95)\n",
    "\n",
    "x = np.linspace(0, 18, num=500)\n",
    "rvf = stats.f(DF_between, DF_within)\n",
    "plt.plot(x, rvf.pdf(x), color='black')\n",
    "hx = rvf.pdf(x)\n",
    "\n",
    "plt.margins(y = 0)\n",
    "plt.ylim((0, .72))\n",
    "plt.xlim((0, Fratio+1))\n",
    "\n",
    "# Shade the probability p-value\n",
    "plt.fill_between(x[x <= Fratio], hx[x <= Fratio], edgecolor = None, color = 'limegreen')\n",
    "# plot the observed F ratio\n",
    "plt.axvline(x = Fratio, color = \"darkgreen\", linestyle = \"--\")\n",
    "# Shade the probability alpha\n",
    "plt.fill_between(\n",
    "    x[x >= F_crit], hx[x >= F_crit], \n",
    "    edgecolor = \"orangered\", linestyle = \"-\", linewidth = 2, facecolor = 'orangered')\n",
    "\n",
    "# plot the critical F value\n",
    "plt.axvline(x = F_crit, color = \"red\", linestyle = \"--\")\n",
    "\n",
    "ax=plt.gca()\n",
    "ax.set_xticks([F_crit, Fratio])\n",
    "ax.set_xticklabels([\"$F_{crit}$\", \"$F$\"], fontdict={'size': 10})\n",
    "ax.get_xticklabels()[0].set_color(\"red\")\n",
    "ax.get_xticklabels()[1].set_color(\"darkgreen\")\n",
    "plt.ylabel('Density function', fontdict={'size': 12})\n",
    "plt.title(f\"F distribution for DFeffect={DF_between} and DFerror={DF_within}\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Effect size\n",
    "\n",
    "Omega squared ($\\omega^2$) is another measure of the effect size, or the degree of association for a population. It is an estimate of **how much variance in the response variables are accounted for by the explanatory variables**. Omega squared is widely viewed as a lesser biased alternative to eta-squared, especially when sample sizes are small. Zero indicates no effect.\n",
    "$$ \\omega^2 = \\frac{\\sigma_Y^2 - \\sigma_{Y \\mid X}^2}{\\sigma_Y^2}\n",
    "= \\frac{df_\\text{effect} \\times (\\text{MS}_\\text{effect} - \\text{MS}_\\text{error})}{df_\\text{effect} \\times \\text{MS}_\\text{effect} + (N - df_\\text{effect}) \\times \\text{MS}_\\text{error}}\n",
    "= \\frac{df_\\text{effect} \\times (\\text{MS}_\\text{effect} - \\text{MS}_\\text{error})}{\\text{SS}_\\text{total} + \\text{MS}_\\text{error}}\n",
    "= \\frac{\\text{SS}_\\text{effect} - (r - 1) \\text{MS}_\\text{error}}{\\text{SS}_\\text{total} + \\text{MS}_\\text{error}}\n",
    "= \\frac{(r-1) \\times (F - 1)}{(r-1) \\times (F - 1) + N} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "omega squared = 0.47765205\n",
      "omega squared = 0.47765205\n",
      "omega squared = 0.47765205\n",
      "omega squared = 0.47765205\n"
     ]
    }
   ],
   "source": [
    "print(f\"omega squared = {DF_between * (MS_between - MS_within) / (DF_between * MS_between + (N - DF_between) * MS_within):.8f}\")\n",
    "print(f\"omega squared = {DF_between * (MS_between - MS_within) / (SST + MS_within):.8f}\")\n",
    "print(f\"omega squared = {(SSB - DF_between * MS_within) / (SST + MS_within):.8f}\")\n",
    "print(f\"omega squared = {(r-1) * (Fratio-1) / ((r-1) * (Fratio-1) + N):.8f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ANOVA reconstruction from summary data\n",
    "\n",
    "One-way ANOVA (but not repeated-measures ANOVA) can be [computed without raw data](https://www.youtube.com/watch?v=PPQRZsBSpI8), so long as you know the mean, sample size and SD (or SEM) of each group.\n",
    "\n",
    "When we look at the very first equations of $\\text{SS}_\\text{effect}$, $\\text{SS}_\\text{error}$ and $\\text{SS}_\\text{total}$, we see that we need the individual data points $ Y_{ij} $ in particular for computing $\\text{SS}_\\text{total}$. So we leave this equation apart.\n",
    "\n",
    "For $\\text{SS}_\\text{effect} = \\sum_i n_i (\\overline{Y_i} - \\overline{Y})^2 $, it's simply the sum of the squared difference between the group mean $ \\overline{Y_i} $ and the grand mean $ \\overline{Y} $ multiplied by the number of values in each group.\n",
    "\n",
    "In the example below, the **grand mean** $ = (41 + 38 + 14 + 37) / 4 = 32.5 $. _Beware this is only true when the sample sizes accross the groups are equal, otherwise we should compute the proper sample mean._\n",
    "\n",
    "| group | mean | $n_i$ | calculation  |\n",
    "| -----:| ---- | ----- |:------------:|\n",
    "|   A   |  41  |   8   | 8*(41-32.5)² |\n",
    "|   B   |  38  |   8   | 8*(38-32.5)² |\n",
    "|   C   |  14  |   8   | 8*(14-32.5)² |\n",
    "|   D   |  37  |   8   | 8*(37-32.5)² |\n",
    "|  sum  |      |       |     3720     |\n",
    "\n",
    "For $ \\text{SS}_\\text{error} = \\sum_i \\sum_j (Y_{ij} - \\overline{Y}_i)^2 $, we don't have access to the individual data points $ Y_{ij} $, but we have access to the standard deviation for each group. We know that the variance is the average of the sum of squares, with $ \\sigma_i^2 = \\frac{\\text{SS}_\\text{error}}{n_i - 1} $. Therefore $ \\text{SS}_\\text{error} $ is the sum of the squared standard deviation multiplied by the number or values minus 1 in each group as shown below:\n",
    "\n",
    "| group | SD  | $n_i$ | calculation |\n",
    "| -----:| --- | ----- |:-----------:|\n",
    "|   A   | 3.5 |   8   | (8-1)*3.5²  |\n",
    "|   B   | 4.6 |   8   | (8-1)*4.6²  |\n",
    "|   C   | 3.8 |   8   | (8-1)*3.8²  |\n",
    "|   D   | 4.9 |   8   | (8-1)*4.9²  |\n",
    "|  sum  |     |       |   503.02    |\n",
    "\n",
    "Let's compute the SSs using the Hair color dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "grand mean = 47.8421052631579\n",
      "Hair color        count     mean     std    SSeffect    SSerror\n",
      "--------------  -------  -------  ------  ----------  ---------\n",
      "Dark Blond        5.000   51.200   9.284      56.377    344.800\n",
      "Dark Brunette     5.000   37.400   8.325     545.188    277.200\n",
      "Light Blond       5.000   59.200   8.526     645.009    290.800\n",
      "Light Brunette    4.000   42.500   5.447     114.152     89.000\n",
      "sum              19.000  190.300  31.582    1360.726   1001.800\n",
      "\n",
      "The values to be used in the ANOVA table are SSeffect = 1360.7, with DFeffect = 3, and SSerror = 1001.8 with DFerror = 15\n"
     ]
    }
   ],
   "source": [
    "grand_mean = data['Pain threshold'].mean()\n",
    "print(f\"grand mean = {grand_mean}\")\n",
    "\n",
    "# getting the count, mean and std for each group\n",
    "calc_table = data.groupby('Hair color').describe()['Pain threshold'].loc[:, ['count', 'mean', 'std']]\n",
    "\n",
    "calc_table['SSeffect'] = calc_table['count'] * (calc_table['mean'] - grand_mean)**2\n",
    "calc_table['SSerror']  = (calc_table['count'] - 1) * calc_table['std']**2\n",
    "calc_table.loc['sum'] = calc_table.sum(axis=0) # adding a new line as sum margin\n",
    "\n",
    "pg.print_table(calc_table.reset_index(), tablefmt='simple')\n",
    "\n",
    "print(f\"The values to be used in the ANOVA table are SSeffect = {calc_table.loc['sum','SSeffect']:.1f}, \\\n",
    "with DFeffect = {DF_between}, and SSerror = {calc_table.loc['sum','SSerror']:.1f} with DFerror = {DF_within}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Repeated-measures one-way ANOVA\n",
    "\n",
    "The difference between ordinary and repeated-measures ANOVA is the same as the difference between the unpaired and paired t tests. Use repeated-measures ANOVA to analyze data collected in three kinds if experiments:\n",
    "\n",
    "* measurements are made repeatedly for each subject (e.g. before, during and after)\n",
    "* subjects are recruited as matched sets or _blocks_, which are matched for variables such as age, postal code, diagnosis etc.\n",
    "* experiment is run several times, each time with several treatment handled in parallel\n",
    "\n",
    "More generally, you should use a repeated-measures test **whenever the value of one subject in the first group is expected to be closer to a particular value in the other group than to a random subject in the other group**.\n",
    "\n",
    "In one-way repeated-measures ANOVA, the total variance (sums of squares) is divided into three components $ \\text{SS}_{\\text{total}} = \\text{SS}_{\\text{effect}} + (\\text{SS}_{\\text{subjects}} + \\text{SS}_{\\text{error}}) $, with $ \\text{SS}_\\text{subjects} = r \\sum (\\overline Y_s - \\overline Y)^2 $ and $ \\overline Y_s $ the mean of the $ s $ th subject.\n",
    "\n",
    "The F statistic is defined as\n",
    "\n",
    "$$ F^* = \\frac{\\text{MS}_{\\text{effect}}}{\\text{MS}_{\\text{error}}} = \\frac{\\frac{\\text{SS}_{\\text{effect}}}{r-1}}{\\frac{\\text{SS}_{\\text{error}}}{(n - 1)(r - 1)}} $$\n",
    "\n",
    "and the P value can be calculated using a F-distribution with $ v_{\\text{effect}} = r - 1 $ and $ v_{\\text{error}} = (n - 1)(r - 1) $ degrees of freedom."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In one-way repeated-measures ANOVA, there is only one factor (treatment of timepoint) to study. The variation is divided into three components including within-subjects ($\\text{SS}_\\text{subjects}$), variation among the treatments or time points ($\\text{SS}_\\text{B}$), and residual variance ($\\text{SS}_\\text{E}$), as follows:\n",
    "\n",
    "| Source of variation  | Degree of freedom                 | Sum of squares       | Mean square       | F value     |\n",
    "| -------------------- | --------------------------------- | -------------------- | ----------------- | ----------- |\n",
    "| Subjects             | $\\text{DF}_\\text{S} = n - 1$      | $\\text{SS}_\\text{S}$ |                   |             |\n",
    "| Treatment/Time point | $\\text{DF}_\\text{B} = k - 1$      | $\\text{SS}_\\text{B}$ | $MS_B=SS_B/DF_B $ | $MS_B/MS_E$ |\n",
    "| Residuals or error   | $\\text{DF}_\\text{E} = (n-1)(k-1)$ | $\\text{SS}_\\text{E}$ | $MS_E=SS_E/DF_E $ |             |\n",
    "| Total                | $\\text{DF}_\\text{T} = nk-1$       | $\\text{SS}_\\text{T}$ |                   |             |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Subject</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Region</th>\n",
       "      <th>Education</th>\n",
       "      <th>DesireToKill</th>\n",
       "      <th>Disgustingness</th>\n",
       "      <th>Frighteningness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "      <td>North</td>\n",
       "      <td>some</td>\n",
       "      <td>10.0</td>\n",
       "      <td>High</td>\n",
       "      <td>High</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "      <td>North</td>\n",
       "      <td>some</td>\n",
       "      <td>9.0</td>\n",
       "      <td>High</td>\n",
       "      <td>Low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "      <td>North</td>\n",
       "      <td>some</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Low</td>\n",
       "      <td>High</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "      <td>North</td>\n",
       "      <td>some</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Low</td>\n",
       "      <td>Low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>Female</td>\n",
       "      <td>North</td>\n",
       "      <td>advance</td>\n",
       "      <td>10.0</td>\n",
       "      <td>High</td>\n",
       "      <td>High</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Subject  Gender Region Education  DesireToKill Disgustingness  \\\n",
       "0        1  Female  North      some          10.0           High   \n",
       "1        1  Female  North      some           9.0           High   \n",
       "2        1  Female  North      some           6.0            Low   \n",
       "3        1  Female  North      some           6.0            Low   \n",
       "4        2  Female  North   advance          10.0           High   \n",
       "\n",
       "  Frighteningness  \n",
       "0            High  \n",
       "1             Low  \n",
       "2            High  \n",
       "3             Low  \n",
       "4            High  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_rm = pg.read_dataset('rm_anova')\n",
    "data_rm.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Source</th>\n",
       "      <th>SS</th>\n",
       "      <th>DF</th>\n",
       "      <th>MS</th>\n",
       "      <th>F</th>\n",
       "      <th>p-unc</th>\n",
       "      <th>ng2</th>\n",
       "      <th>eps</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Disgustingness</td>\n",
       "      <td>27.485215</td>\n",
       "      <td>1</td>\n",
       "      <td>27.485215</td>\n",
       "      <td>12.043878</td>\n",
       "      <td>0.000793</td>\n",
       "      <td>0.025784</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Error</td>\n",
       "      <td>209.952285</td>\n",
       "      <td>92</td>\n",
       "      <td>2.282090</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Source          SS  DF         MS          F     p-unc       ng2  \\\n",
       "0  Disgustingness   27.485215   1  27.485215  12.043878  0.000793  0.025784   \n",
       "1           Error  209.952285  92   2.282090        NaN       NaN       NaN   \n",
       "\n",
       "   eps  \n",
       "0  1.0  \n",
       "1  NaN  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.rm_anova(\n",
    "    dv='DesireToKill',\n",
    "    within='Disgustingness',\n",
    "    subject='Subject',\n",
    "    data=data_rm,\n",
    "    detailed=True,\n",
    "    effsize=\"ng2\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assumption of sphericity\n",
    "\n",
    "The assumption of sphericity can be tested using Mauchly's test of sphericity. The violation of the assumption of sphericity can lead to an increase in type II error (loss of statistical power) and the F value is not valid. If the P value exceeds 0.05, the data met the assumption of sphericity, and variances of differences of independent variables are equal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Does the data have the sphericity property: True.\n",
      "The Mauchly W statistic equals nan, with a chi² of nan and 1 degrees of freedom.\n",
      "The P value equals 1.0000\n"
     ]
    }
   ],
   "source": [
    "spher, W, chi2_spher, dof_spher, pval_spher = pg.sphericity(\n",
    "    dv='DesireToKill',\n",
    "    within='Disgustingness',\n",
    "    subject='Subject',\n",
    "    data=data_rm,\n",
    ")\n",
    "\n",
    "print(f\"Does the data have the sphericity property: {spher}.\\n\\\n",
    "The Mauchly W statistic equals {W:.3f}, with a chi² of {chi2_spher:.2f} and {dof_spher} degrees of freedom.\\n\\\n",
    "The P value equals {pval_spher:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Source</th>\n",
       "      <th>ddof1</th>\n",
       "      <th>ddof2</th>\n",
       "      <th>F</th>\n",
       "      <th>p-unc</th>\n",
       "      <th>ng2</th>\n",
       "      <th>eps</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Disgustingness</td>\n",
       "      <td>1</td>\n",
       "      <td>92</td>\n",
       "      <td>12.043878</td>\n",
       "      <td>0.000793</td>\n",
       "      <td>0.025784</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Source  ddof1  ddof2          F     p-unc       ng2  eps\n",
       "0  Disgustingness      1     92  12.043878  0.000793  0.025784  1.0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.rm_anova(\n",
    "    dv='DesireToKill',\n",
    "    within='Disgustingness',\n",
    "    subject='Subject',\n",
    "    data=data_rm,\n",
    "    detailed=False,\n",
    "    effsize=\"ng2\",\n",
    "    correction='auto' # the default for one-way design is to compute Mauchly's test of\n",
    "                      # sphericity to determine whether the p-values needs to be corrected\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mixed-design (split-plot) ANOVA\n",
    "\n",
    "In statistics, a mixed-design analysis of variance model, also known as a split-plot ANOVA, is used to **test for differences between two or more independent groups whilst subjecting participants to repeated measures**. Thus, in a mixed-design ANOVA model, one factor (a fixed effects factor) is a between-subjects variable and the other (a random effects factor) is a within-subjects variable. Thus, overall, the model is a type of mixed-effects model.\n",
    "\n",
    "A repeated measures design is used when multiple independent variables or measures exist in a data set, but all participants have been measured on each variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Scores  Time    Group      Subject\n",
      "--------  ------  -------  ---------\n",
      "   5.971  August  Control          0\n",
      "   4.309  August  Control          1\n",
      "   6.933  August  Control          2\n",
      "   5.187  August  Control          3\n",
      "   4.779  August  Control          4\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEGCAYAAABvtY4XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeQElEQVR4nO3dfXRV9b3n8fc3DxAeDSAVFCF4WwEJIJCiNgIC1VqupQadVakPKNNhCi4od1pbb+/4cHHs1F7a4UqxS+xY9QporxLraMdiEUEYa+Uxgjz0KgkFRAMkCgiBkO/8cU7SACE5gbOzz9n5vNZicbLPPr/9PdnwOb/z23v/trk7IiISPRlhFyAiIsFQwIuIRJQCXkQkohTwIiIRpYAXEYmorLALqO/888/3vLy8sMsQEUkba9eu3efu3Rt6LqUCPi8vjzVr1oRdhohI2jCzsjM9pyEaEZGIUsCLiESUAl5EJKJSagy+IcePH2fXrl0cPXo07FIiKScnh169epGdnR12KSKSZCkf8Lt27aJTp07k5eVhZmGXEynuzv79+9m1axd9+/YNuxwRSbKUH6I5evQo3bp1U7gHwMzo1q2bvh2JRFTKBzygcA+Qfrci0ZXyQzQiIi3lmWeeoazsjKeVn2Tv3r0A9OjRI6H1+/Tpwx133HHWtZ2NtOjBh+Xjjz/m29/+NpdccgnDhw/nqquuori4OOyyRCQFVFVVUVVVFXYZjVIP/gzcnRtvvJHJkyezaNEiAMrKynj55ZdPWq+6upqsLP0aRaKgOT3shx56CID77rsvqHLOmXrwZ/DGG2/Qpk0bvvvd79Yt69OnDzNmzOCpp55iwoQJjB07lnHjxnHgwAFuvPFGBg8ezJVXXklJSQkADz74IHPmzKl7fX5+PqWlpZSWltK/f39uvfVWBgwYwM0338znn3/e4u9RRKJNAX8GmzdvZtiwYWd8ft26dbzwwgusWLGCBx54gKFDh1JSUsJPfvKThHoB27ZtY/r06WzZsoXOnTvz2GOPJbN8EREFfKLuvvtuhgwZwpe//GUArr32Wrp27QrAqlWruP322wEYO3Ys+/fv57PPPmu0vYsvvpjCwkIAbrvtNlatWhVg9SLSGingz2DgwIGsW7eu7uf58+ezbNkyysvLAejQoUOTbWRlZVFTU1P3c/3zzU89PVGnK4pIsingz2Ds2LEcPXqUX/3qV3XLzjROPnLkSBYuXAjAm2++yfnnn0/nzp3Jy8ur+5BYt24dO3bsqHvNzp07efvttwFYtGgRV199dVBvRURaKQX8GZgZL730EitWrKBv376MGDGCyZMn88gjj5y27oMPPsjatWsZPHgw9957L08//TQAN910EwcOHGDgwIH88pe/5NJLL617Tb9+/Zg/fz4DBgygoqKCadOmtdh7E5HWQef3NaJnz54899xzDT5355131j3u2rUrL7300mnrtGvXjqVLl562vLS0lKysLJ599tlklSoichr14EVEIko9+BDk5eWxadOmsMuQAETtUndJbwp4kZCk+mXukv4U8CJJFLVL3SW9BToGb2b/YGabzWyTmS02s5wgtyciIn8TWMCb2UXATKDA3fOBTOCWoLYnIiInC3qIJgtoZ2bHgfbAnnNt8KGf/k8OfFZ5rs3U6do5l/vu/cdG19m7dy+zZs3i3XffJTc3lwsuuIC5c+eedF57IubOncvUqVNp3759s17XsWNHDh061KzXtEYVFRXMmzePmTNnkpubG3Y5IqELLODdfbeZzQF2AkeApe5+2knhZjYVmArQu3fvJts98FklGSMHJq3OA29tbvR5d6eoqIjJkyfXnRO/ceNGPv7447MK+Ntuu63BgD9x4gSZmZnNak9OVlxczLZt21iyZAlTpkwJuxyR0AU5RNMF+CbQF7gQ6GBmt526nrsvcPcCdy/o3r17UOWcteXLl5OdnX3StMFDhgzh6quv5p577iE/P59Bgwbx/PPPA7GpCq655hpuvvnmuimB3Z1HH32UPXv2MGbMGMaMGQPEeubf//73GTJkCG+//Ta/+MUvyM/PJz8/n7lz54bxdtNWRUUFK1aswN1ZuXIllZWVYZckErogD7J+Fdjh7uXufhxYAnwlwO0FYtOmTQwfPvy05UuWLGHDhg1s3LiRP/7xj9xzzz189NFHAKxfv565c+fy/vvv8+GHH7J69WpmzpzJhRdeyPLly1m+fDkAhw8f5oorrmDjxo20a9eO3/zmN7zzzjv86U9/4oknnmD9+vUt+l7TWXFxMe4OQE1NDUuWLAm5IpHwBRnwO4Erzay9xaZKHAdsCXB7LWrVqlVMmjSJzMxMLrjgAkaPHs27774LwIgRI+jVqxcZGRlcfvnllJaWNthGZmYmN910U117RUVFdOjQgY4dOzJx4kTeeuutlno7aW/16tVUV1cDsbtsrV69OuSKRMIXWMC7+zvAC8A64L34thYEtb2gDBw4kLVr1zbrNW3btq17nJmZWRc8p8rJydG4e5IUFhbW3ToxKyurbq59kdYs0PPg3f0Bd+/v7vnufru7p92le2PHjqWqqooFC/722VRSUkJubi7PP/88J06coLy8nJUrVzJixIhG2+rUqRMHDx5s8LmRI0fy0ksv8fnnn3P48GGKi4sZOXJkUt9LlBUVFdXNqZ+RkcHEiRNDrkgkfGl3JWvXzrlNnvnS3PYaY2YUFxcza9YsHnnkEXJycsjLy2Pu3LkcOnSIIUOGYGb87Gc/o0ePHmzduvWMbU2dOpXrr7++biy+vmHDhnHnnXfWfUh85zvfYejQoef8/lqLLl26MHr0aJYtW8aoUaN0mqQIYLUHplJBQUGBr1mz5qRlW7ZsYcCAASFV1DpE5XecbufBa6qC9JYq+8/M1rp7QUPPpV0PXuRMunTpwv333x92GSIpQ/PBi4hElAJeRCSiFPAiIhGlgBcRiSgdZBWRQOj2heFLu4D/+U8e4mDlgaS11ym3K9//ceOnOZkZt956K88++ywQuxS+Z8+eXHHFFbzyyisJb+uaa65hzpw5FBQUMH78eBYtWgTAokWLmD59eqOvraysPGm9PXv2MHPmTF544YWEty+SqnT7wmCkXcAfrDzA9L7ZSWvvsR1Nf1h06NCBTZs2ceTIEdq1a8frr7/ORRdddE7b/f3vfw9AaWkpjz32WEIBX3+9Cy+8UOEuKU23LwyfxuATNH78eF599VUAFi9ezKRJk+qeO3z4MFOmTGHEiBEMHTqU3/3udwAcOXKEW265hQEDBlBUVMSRI0fqXpOXl8e+ffu49957+eCDD7j88su55557OHToEOPGjWPYsGEMGjSorq1T1ystLSU/Px+Ao0ePctdddzFo0CCGDh1ad5XsU089xcSJE7n++uv50pe+xA9/+MMW+V2JSGpIux58WG655RZmz57NDTfcQElJCVOmTKmb7fHhhx9m7NixPPnkk1RWVjJixAi++tWv8vjjj9O+fXu2bNlCSUkJw4YNO63dn/70p2zatIkNGzYAseGf4uJiOnfuzL59+7jyyiuZMGHCaevVn6Fy/vz5mBnvvfceW7du5brrrmP79u0AbNiwgfXr19O2bVv69evHjBkzuPjiiwP9XYlIalDAJ2jw4MGUlpayePFixo8ff9JzS5cu5eWXX2bOnDlArEe9c+dOVq5cycyZM+teP3jw4Ca34+78+Mc/ZuXKlWRkZLB7924+/vjjRl+zatUqZsyYAUD//v3p06dPXcCPGzeO8847D4DLLruMsrIyBbxIK6GAb4YJEybwgx/8gDfffJP9+/fXLXd3XnzxRfr163fO21i4cCHl5eWsXbuW7Oxs8vLyOHr06Fm3l+jUxSISPRqDb4YpU6bwwAMPMGjQoJOWf+1rX2PevHl1dxSqvRPTqFGj6s6U2bRpEyUlJae1eeoUwp9++ilf+MIXyM7OZvny5XWnmTU11fDChQsB2L59Ozt37kzKh42IpLe068F3yu2a0JkvzWkvUb169aobcqnvvvvuY9asWQwePJiamhr69u3LK6+8wrRp07jrrrsYMGAAAwYMaPDWf926daOwsJD8/Hy+/vWv86Mf/YhvfOMbDBo0iIKCAvr379/genfffXddG9OnT2fatGkMGjSIrKwsnnrqqZN67iLSOmm6YNHvOCQ6NfBv0vF3kSo1NzZdsIZoREQiSgEvIhJRaRHwqTSMFDX63YpEV8oHfE5ODvv371cQBcDd2b9/Pzk5OWGXIiIBCOwsGjPrBzxfb9ElwP3uPrc57fTq1Ytdu3ZRXl6ezPIkLicnh169eoVdhogEILCAd/dtwOUAZpYJ7AaKm9tOdnY2ffv2TW5xkjY05azI2Wup8+DHAR+4e2L/U0XOgqacFTlZSwX8LcDihp4ws6nAVIDevXu3UDmSLjTlrMjZCzzgzawNMAH4x4aed/cFwAKIXeiU7O3rK75I69acDGiO2jZrOxbJlKxsaYke/NeBde7e+JSIKUBf8UWip6ysjB3bt3JRx+TdKAggOz5x37E9HyS13d2HjietrZYI+EmcYXimJaTjV3x96xBJros6ZjNj8BfCLiMh80o+SVpbgZ4Hb2YdgGuBJUFupzWrqqrSNw8RaVCgPXh3Pwx0C3IbUZSO3zpEJPWk/JWsIiJydhTwIiIRpYAXEYkoBbyISEQp4EVEIkoBLyISUWl3022RltaaL3WX9KaAF2lCa77UXdKbAl4kAa31UndJbxqDFxGJKAW8iEhEKeBFRCIqLcfgdVaDiEjT0jLgy8rK2PoffyG7a6ektlvNCQA+OLA3qe0eP3Awqe2JSOL27t3LkUPH0+bg8+5Dx2m3NzkZlJYBD5DdtRPdrrsi7DISsn/pO2GXICKtUNoGvIhIInr06MGxmsNpdZprmwTv0NYUHWQVEYko9eClxaXbQfKysjJ6Jvci1rSVbvsOYmPwXZPeanpQwEuLS7eD5MeOfA7ZbZLaZrpKt313/MBB2mW3gVb6Aa2Al1Ck00Hyjxb+gd2t9CyMhqTTvtu/9B042HpvSq8xeBGRiAq0B29mucCvgXzAgSnu/naQ2xRJNsvM5KL2Ga3yLAxJb0EP0fwr8Jq732xmbYD2AW8vZe3duzeQA0i6+lZEziSwgDez84BRwJ0A7n4MOBbU9lJdVVWV5hQXkRYVZA++L1AO/MbMhgBrge+5++H6K5nZVGAqQO/evQMsJ3yaU1xEWlKQB1mzgGHAr9x9KHAYuPfUldx9gbsXuHtB9+7dAyxHRKR1CTLgdwG73L12IpYXiAW+iIi0gMAC3t33An81s37xReOA94PanoiInCzos2hmAAvjZ9B8CNwV8PZERCQu0IB39w1AQZDbEBGRhulKVhGRiNJcNCKSsL1793L80MG0uYnN8QMHqXEgu3X2ZVvnuxYRaQUS6sGb2d8RO+WxysyuAQYDz7h7ZXCliUiq6dGjB4cPkFazSWYdrAKqwy4lFIn24F8ETpjZF4EFwMXAosCqEhGRc5boGHyNu1ebWREwz93nmdn6IAtrjMYBRaQ5gpjPf9+R2LeC89sl91Dm7kPH6ZukthKt7LiZTQImA9+IL2ul90iRc5VuH9BeXc2+I/pwTldt27alR58+SW/3eHwm1zYXJrftvsRmck2GRAP+LuC7wMPuvsPM+gL/lpQKzoLGAUUkUT169OC+++5Leru1U3QH0XayJBTw7v6+mf0I6B3/eQfwSJCFSXSl2wf03uf+yPnt1IOX9JPoWTTfAOYAbYC+ZnY5MNvdJwRYW6QcO3aM3VUn0mYa3qDv6ykiwUu0W/IgMAKohLopCC4JpCIREUmKhA+yuvunZlZ/WU0A9URWmzZt6JldnVY3/NB9PUXSW6IBv9nMvg1kmtmXgJnA/wuuLBEROVeJDtHMAAYCVcQucPoUmBVQTSIikgRN9uDNLBN41d3HAP8UfEkiqae1Xigj6a3Jf1nufsLMaszsPHf/tCWKEkkllp1JBm2SfkFLOlwoI+kt0a7DIeA9M3ud2M2zAXD3mYFUJZF3/EDyr2StPvg5AFmd2ie1XT9+gj5fvCTpF7Skw4Uykt4SDfgl8T8i5yyo3mXZwViPuE/XJJ/907WHesSSlhK9kvXp+H1VL40v2ubux4MrS6LsjjvuCKRd9YhFTpbolazXAE8DpYABF5vZZHdfGVhlIpKS0ml47fiBg5Dsb3RpJNEhmp8D17n7NgAzuxRYDAwPqjARST0aXksviQZ8dm24A7j7djNrcrpgMysFDgIngGp3LzirKkUkJWh4Lb0kGvBrzOzXwLPxn28F1iT42jHuvq/ZlYmIyDlJNOCnAXcTm6IA4C3gsUAqijBdLCMiLSnRVMgC/tXdfwF1V7e2TeB1Diw1Mwced/cFp65gZlOBqQC9e/dOsJz005rvKiMi4Ug04JcBXyV2wRNAO2Ap8JUmXne1u+82sy8Ar5vZ1lPPvImH/gKAgoICT7jyNNOa7yojIuFINOBz3L023HH3Q2bW5PlM7r47/vcnZlZMbE75pJxaqVO1REQal2jAHzazYe6+DsDMCoAjjb3AzDoAGe5+MP74OmD2OVUbp1O1RESalmjAzwL+3cz2xH/uCXyriddcABTHbxKSBSxy99fOpshT6VQtEZGmNRrwZvZl4K/u/q6Z9Qf+KzAReA3Y0dhr3f1DYEiyChURkeZp6oYfjwPH4o+vAn4MzAcqiB8YFRGR1NTUEE2mux+IP/4WsMDdXwReNLMNgVYmIiLnpKkefKaZ1X4IjAPeqPdccq+sERGRpGoqpBcDK8xsH7GzZt4CMLMvErsvq4iIpKhGA97dHzazZcTOmlnq7rUXImUQuxG3iIikqETuyfqnBpZtD6YcERFJlqbG4EVEJE0p4EVEIkoBLyISUQp4EZGIUsCLiESUAl5EJKIU8CIiEaXpBkQkEM888wxl8VtKNqV2vdopu5vSp0+fwKYNjxIFvIiErm3bRG7xLM2lgBeRQKRjDztq3zoU8CIiZyEdvnUo4EVE4tLxW0djdBaNiEhEKeBFRCJKAS8iElGBj8GbWSawBtjt7jcEvT2Jlqid1SDSklriIOv3gC1A5xbYlrRi6XBWg0hLCjTgzawX8PfAw8B/C3JbEk3qYYucvaDH4OcCPwRqzrSCmU01szVmtqa8vDzgckREWo/AAt7MbgA+cfe1ja3n7gvcvcDdC7p37x5UOSIirU6QPfhCYIKZlQLPAWPN7NkAtyciIvUEFvDu/o/u3svd84BbgDfc/bagticiIifTefAiIhHVInPRuPubwJstsS0REYlRD15EJKIU8CIiEaXpgkWSSFMrSCpRwKcghUTroKkVJGgK+DSnkEgt+vCUVKKAT0EKCRFJBh1kFRGJKAW8REZFRQWzZ8+msrIy7FJEUoICXiKjuLiYbdu2sWTJkrBLEUkJCniJhIqKClasWIG7s3LlSvXiRVDAS0QUFxfj7gDU1NSoFy+CAl4iYvXq1VRXVwNQXV3N6tWrQ65IJHwKeImEwsJCsrJiZ/1mZWVRWFgYckUi4VPASyQUFRVhZgBkZGQwceLEkCsSCZ8CXiKhS5cujB49GjNj1KhR5Obmhl2SSOgifyWr5nVpPYqKiti1a5d67yJxkQ/45tC8LumtS5cu3H///WGXIZIyIh/w6mGLpL6KigrmzZvHzJkzNbyWRBqDF5HQ6SrkYCjgRSRUugo5OAp4EQmVrkIOTmABb2Y5ZvZnM9toZpvN7J+D2paIpC9dhRycIHvwVcBYdx8CXA5cb2ZXBrg9EUlDugo5OIEFvMcciv+YHf/jQW1PRNKTrkIOTqBj8GaWaWYbgE+A1939nQbWmWpma8xsTXl5eZDliEgK0lXIwQk04N39hLtfDvQCRphZfgPrLHD3Ancv6N69e5DliEiKKioqol+/fuq9J1mLXOjk7pVmthy4HtjUEtsUkfShq5CDEeRZNN3NLDf+uB1wLbA1qO2JiMjJguzB9wSeNrNMYh8kv3X3VwLcnoiI1BNYwLt7CTA0qPZFRKRxupJVRCSiFPAiIhGlgBcRiSgFvIhIRCngRUQiSgEvIhJRCngRkYhSwIuIRJQCXkQkohTwIiIRpYAXEYkoBbyISEQp4EVEIkoBLyISUQp4EZGIUsCLiESUAl5EJKIU8CIiEaWAFxGJKAW8iEhEKeBFRCIqsIA3s4vNbLmZvW9mm83se0FtS0REThdkD74a+L67XwZcCdxtZpcFuD0RSVMVFRXMnj2bysrKsEuJlMAC3t0/cvd18ccHgS3ARUFtT0TSV3FxMdu2bWPJkiVhlxIpLTIGb2Z5wFDgnQaem2pma8xsTXl5eUuUIyIppKKighUrVuDurFy5Ur34JAo84M2sI/AiMMvdPzv1eXdf4O4F7l7QvXv3oMsRkRRTXFyMuwNQU1OjXnwSBRrwZpZNLNwXurv2moicZvXq1VRXVwNQXV3N6tWrQ64oOoI8i8aA/w1scfdfBLUdEUlvhYWFZGVlAZCVlUVhYWHIFUVHkD34QuB2YKyZbYj/GR/g9kQkDRUVFRHrD0JGRgYTJ04MuaLoyAqqYXdfBVhQ7YtINHTp0oXRo0ezbNkyRo0aRW5ubtglRUZgAS8ikqiioiJ27dql3nuSWe3R61RQUFDga9asCbsMEZG0YWZr3b2goec0F42ISEQp4EVEIkoBLyISUQp4EZGISqmDrGZWDpSFXUdAzgf2hV2EnDXtv/QW5f3Xx90bnOclpQI+ysxszZmOdEvq0/5Lb611/2mIRkQkohTwIiIRpYBvOQvCLkDOifZfemuV+09j8CIiEaUevIhIRCngRUQiSgHfDGZ2o5m5mfVvwW3OMrP2LbW9KDKzQ2HXIMHQvm2cAr55JgGr4n+3lFmAAj7NmJmm4pbQKeATFL95+NXAfwZuiS+7xsxeqbfOL83szvjj8Wa21czWmtmjteuZ2YNm9oN6r9lkZnlm1sHMXjWzjfFl3zKzmcCFwHIzW95y7zZ6zKyjmS0zs3Vm9p6ZfTO+PM/MtpjZE2a22cyWmlm7+HNvmllB/PH5ZlZa7zVvxdtaZ2ZfiS+/Jr78ZeB9M5ttZrPq1fCwmX2vhd965DXx/7DUzP653n7vH1/ewcyeNLM/m9n62n8PUaOAT9w3gdfcfTuw38yGn2lFM8sBHge+7u7DgQYvIz7F9cAedx/i7vnxbT0K7AHGuPuYc38LrdpRoMjdhwFjgJ9b7X3i4EvAfHcfCFQCNzXR1ifAtfG2vgU8Wu+5YcD33P1S4EngDgAzyyDWMXg2OW9HmmFffF/9CqjtXP0T8Ia7jyD27+FfzKxDWAUGRQGfuEnAc/HHz9H4ME1/4EN33xH/eXEC7b8HXGtmj5jZSHf/9OxLlQYY8BMzKwH+CFwEXBB/boe7b4g/XgvkNdFWNvCEmb0H/DtwWb3n/ly73929lFhnYChwHbDe3fef+1uRZloS/7v+vr0OuNfMNgBvAjlA75YuLGgaJ0yAmXUFxgKDzMyBTMCB33Hyh2ROAs1VN/Qad99uZsOA8cD/MLNl7j47GfULALcS+yY13N2Px4dbavdXVb31TgDt4o/r76v6+/YfgI+BIfHnj9Z77vAp2/01cCfQg1iPXpKvwf9T9dTu3xP8LfMMuMndtwVcW6jUg0/MzcC/uXsfd89z94uBHcR+f5eZWVszywXGxdffBlxiZnnxn79Vr61SYl/jiQd63/jjC4HP3f1Z4F9q1wEOAp0Cel+tyXnAJ/FwHwP0SeA1pUDtUNzNp7T1kbvXALcT+8A/k2Jiw29fBv7Q3KIlIWU0/P+wMX8AZtQO08W/ZUWOevCJmQQ8csqyF4mNqf4W2EQs8NcDuPsRM5sOvGZmh4F3T3ndHWa2GXgH2B5fPojYOGANcByYFl++IN7OHo3DN1/8bJYqYCHwf+LDKmuArQm8fA7wWzObCrxab/ljwItmdgfwGqf32uu4+7H4AfJKdz9xlm9DGlC7b939r2Z22v/DJjwEzAVK4sdHdgA3BFVrWDRVQUDMrKO7H4r3EOYDf3H3/xV2Xa2NmQ0BnogfTAtj+xnAOuA/uftfwqghqsLet+lAQzTB+S/xAzibiX2lfzzcclofM/susQPc/z2k7V8G/AewTOGeXGHv23ShHryISESpBy8iElEKeBGRiFLAi4hElAJeWiUz62ZmG+J/9prZ7vjjQ2b2WNj1iSSDDrJKq2dmDwKH3H1O2LWIJJN68CL11J+ZMD7z59PxGSLLzGyimf0sPivha2aWHV9vuJmtsNjMoX8ws57hvguRGAW8SOP+jtg8RBOIzQS53N0HAUeAv4+H/Dzg5vjMoU8CD4dVrEh9mqpApHH/Nz5/zXvE5px5Lb78PWIzE/YD8oHX49OaZAIfhVCnyGkU8CKNqwJw9xozO+5/O2hVQ+z/jwGb3f2qsAoUORMN0Yicm21AdzO7CsDMss1sYMg1iQAKeJFz4u7HiE0l/IiZbQQ2AF8JtSiROJ0mKSISUerBi4hElAJeRCSiFPAiIhGlgBcRiSgFvIhIRCngRUQiSgEvIhJR/x862Vs24b1lCQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_mixed = pg.read_dataset('mixed_anova')\n",
    "\n",
    "pg.print_table(data_mixed.head())\n",
    "\n",
    "sns.boxplot(\n",
    "    x='Time',\n",
    "    y='Scores',\n",
    "    data=data_mixed,\n",
    "    hue='Group',\n",
    "    palette='Set2',\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Source</th>\n",
       "      <th>SS</th>\n",
       "      <th>DF1</th>\n",
       "      <th>DF2</th>\n",
       "      <th>MS</th>\n",
       "      <th>F</th>\n",
       "      <th>p-unc</th>\n",
       "      <th>np2</th>\n",
       "      <th>eps</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Group</td>\n",
       "      <td>5.459963</td>\n",
       "      <td>1</td>\n",
       "      <td>58</td>\n",
       "      <td>5.459963</td>\n",
       "      <td>5.051709</td>\n",
       "      <td>0.028420</td>\n",
       "      <td>0.080120</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Time</td>\n",
       "      <td>7.628428</td>\n",
       "      <td>2</td>\n",
       "      <td>116</td>\n",
       "      <td>3.814214</td>\n",
       "      <td>4.027394</td>\n",
       "      <td>0.020369</td>\n",
       "      <td>0.064929</td>\n",
       "      <td>0.998751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Interaction</td>\n",
       "      <td>5.167192</td>\n",
       "      <td>2</td>\n",
       "      <td>116</td>\n",
       "      <td>2.583596</td>\n",
       "      <td>2.727996</td>\n",
       "      <td>0.069545</td>\n",
       "      <td>0.044922</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Source        SS  DF1  DF2        MS         F     p-unc       np2  \\\n",
       "0        Group  5.459963    1   58  5.459963  5.051709  0.028420  0.080120   \n",
       "1         Time  7.628428    2  116  3.814214  4.027394  0.020369  0.064929   \n",
       "2  Interaction  5.167192    2  116  2.583596  2.727996  0.069545  0.044922   \n",
       "\n",
       "        eps  \n",
       "0       NaN  \n",
       "1  0.998751  \n",
       "2       NaN  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.mixed_anova(\n",
    "    dv='Scores',\n",
    "    between='Group',\n",
    "    within='Time',\n",
    "    subject='Subject',\n",
    "    data=data_mixed,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repeated measures two-way ANOVA\n",
    "\n",
    "In two-way repeated measures ANOVA, there are two within-subjects factors (treatments or time) to study. In the example below:\n",
    "\n",
    "- the same animals were first given inactive treatment and the response measured; then they were given the active treatment and the response measured again\n",
    "- the same animals are first measured for a response after a short duration and then later after a long duration\n",
    "- each animal was first given the inactive treatment and measured for a short and then a long duration; then the same animal was given the active treatment and measured for a short and then a long duration. This is a **repeated measures in both factors** design."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "plants    nutrient_level    year    time      num_leaves\n",
      "--------  ----------------  ------  ------  ------------\n",
      "P1        Low               Year1   Week1              4\n",
      "P2        Low               Year1   Week1              3\n",
      "P3        Low               Year1   Week1              6\n",
      "P4        Low               Year1   Week1              5\n",
      "P5        Low               Year1   Week1              5\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEGCAYAAACNaZVuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcqElEQVR4nO3df3RV5Z3v8fc3ISSgYSCIBsviwGjH4g8UTEVrpVqt0NY6VfRal53UIkPvslo62vbWeqOtGUu7rFOr93a56EidqKtesXrbOlMH9Q7ShUIFQUsrGn9FwR5/JALhR35/7x/nBAI5CdknZ++dZH9ea7HIefbZz/PdT5JvnvOcfZ7H3B0REUmWorgDEBGR6Cn5i4gkkJK/iEgCKfmLiCSQkr+ISAKNijuAgTriiCN82rRpcYchIjKsbNiw4QN3n3Rw+bBJ/tOmTWP9+vVxhyEiMqyYWUOuck37iIgkkJK/iEgCKfmLiCTQsJnzz6W9vZ2tW7fS0tISdyiRKisrY8qUKZSUlMQdiogMU8M6+W/dupXy8nKmTZuGmcUdTiTcncbGRrZu3cr06dPjDkdEhqlQp33MbLmZvWdmm3uU3WZmW8zsRTN71MzG51t/S0sLEydOTEziBzAzJk6cmLhXOyJSWGHP+d8LzD+o7AngRHefCbwC3DCYBpKU+Lsl8ZpFpLBCnfZx99VmNu2gspU9Hq4FLgkzBhGRbnV1dTQ09L7tPZ1O09raGri+0tJSKisrcx5LpVJUV1cHrjMqcc/5LwT+T18HzWwxsBhg6tSpUcUkIiNUQ0MDW16tp6Si/IDyjtY9eHtn4Ppa6GR3U7pXeXtTc94xRiW25G9mNwIdwAN9PcfdlwHLAKqqqobtrjOdnZ0UFxfHHYaIACUV5Uw8f06obTSuXBdq/YUQy33+ZnYlcAFwhQ+xrcRuuukm7rjjjn2Pb7zxRn72s59x22238fGPf5yZM2dy88037zv+xS9+kVNPPZUTTjiBZcuW7Ss//PDDuf766zn55JN59tlno7wEEZFDijz5m9l84DvAhe6+J+r2D2XhwoXU1dUB0NXVxYMPPkhlZSX19fX88Y9/ZNOmTWzYsIHVq1cDsHz5cjZs2MD69eu58847aWxsBGD37t3MmTOHF154gU9+8pOxXY+ISC6hTvuY2a+As4EjzGwrcDOZu3tKgSeyd62sdff/HmYcQUybNo2JEyeyceNG3n33XWbNmsVzzz3HypUrmTVrFgC7du2ivr6euXPncuedd/Loo48C8Pbbb1NfX8/EiRMpLi5mwYIFcV6KiEifwr7b5/IcxfeE2WYhLFq0iHvvvZd0Os3ChQt56qmnuOGGG/ja1752wPNWrVrFk08+ybPPPsvYsWM5++yz991/X1ZWpnl+ERmytLZPDhdddBGPP/44zz33HPPmzWPevHksX76cXbt2AbBt2zbee+89duzYwYQJExg7dixbtmxh7dq1MUcuIjIwcd/qOSSNHj2ac845h/Hjx1NcXMz555/PSy+9xBlnnAFk3sy9//77mT9/PnfffTczZszguOOO4/TTT485chGRgVHyz6Grq4u1a9eyYsWKfWVLlixhyZIlvZ77+9//Pmcd3a8SRESGIk37HOQvf/kLxx57LOeeey4f/ehH4w5HRCQUGvkf5Pjjj+f111+POwwRkVBp5C8ikkBK/iIiCaTkLyKSQEr+IiIJNKLe8K390VKadm4vWH0V48ZT892+95pxd8466yxuvPFGPvvZzwKwYsUK7rnnHh5//PG82124cCGPPfYYRx55JJs3bz70CSIiAY2o5N+0cztFZ51QuPr+8Od+j5sZd999N5deeinnnHMOHR0dfO9738s78Xd0dDBq1CiuvPJKrrnmmiG9EYSIDG+a9hmkE088kS984Qv8+Mc/5pZbbuHLX/4yt956K6eddhqzZs3iN7/5DQBvvvkmZ511FrNnz2b27Nk888wzQGZ9oLPOOosLL7yQ448/HoC5c+dSUVER2zWJyMg3okb+cbn55puZPXs2o0eP5oILLuDTn/40y5cvZ/v27Zx22mmcd955HHnkkTzxxBOUlZVRX1/P5Zdfzvr16wF4/vnn2bx5M9OnT4/5SkQkKZT8C+Cwww7jsssu4/DDD+ehhx7id7/7HT/5yU8AaGlp4a233uLoo4/mmmuuYdOmTRQXF/PKK6/sO/+0005T4heRSCn5F0hRURFFRUW4O7/+9a857rjjDjj+/e9/n6OOOooXXniBrq4uysrK9h077LDDog5XRBJOc/4FNm/ePO666y66d6fcuHEjADt27GDy5MkUFRVx33330dkZfLNoEZFCGVEj/4px4w95h07Q+oKqqanhm9/8JjNnzqSrq4vp06fz2GOPcfXVV7NgwQLq6uqYP39+v6P9yy+/nFWrVvHBBx8wZcoUfvCDH3DVVVcN4kpERA5kQ2z/9D5VVVV59xuk3V566SVmzJgRU0TxSvK1i+SrtraW15rSTDx/TqjtNK5cxzEVldTU1ITazkCY2QZ3rzq4XNM+IiIJpOQvIpJASv4iIgmk5C8ikkBK/iIiCaTkLyKSQCPqPv/bf1hL8/amgtVXPr6C67/X961aYSzp/Pbbb1NdXc27776LmbF48WKWLFmSV10iIn0ZUcm/eXsTV08vKVh9P3+j/z8kYS3pfPvttzN79myam5s59dRT+cxnPrNvxU8RkULQtM8gFXpJ58mTJzN79mwAysvLmTFjBtu2bYvt+kRkZBpRI/+4hLWk85tvvsnGjRuZMyfcTyOKSPIo+RdAGEs679q1iwULFnDHHXcwbty4SK9HREY+Jf8CKeSSzu3t7SxYsIArrriCiy++OJL4RSRZNOdfYINd0tndueqqq5gxYwbXXXddZHGLSLKMqJF/+fiKQ96hE7S+oAa7pPOaNWu47777OOmkkzjllFMA+OEPf8jnPve5wVyKiMgBtKTzMJXkaxfJl5Z03k/TPiIiCaTkLyKSQKEmfzNbbmbvmdnmHmUVZvaEmdVn/58wmDaGy7RVISXxmkWksMIe+d8LzD+o7LvAU+7+UeCp7OO8lJWV0djYmKhk6O40NjYecKuoiEhQob/ha2bTgMfc/cTs45eBs939r2Y2GVjl7sf1VwfkfsO3vb2drVu30tLSEkLkQ1dZWRlTpkyhpKRw6xjJyFVXV0dDQ0Ov8nQ6TWtra+D6SktLqayszHkslUpRXV0duM6ofP3rX+fD7R9iowZ+o6Nnb8u24uKBn9PeAcDYsWN7HWtra6Orq2vAdXUrKipi9OjROY9NmjSJpUuX5jzW1xu+cdzqeZS7/zX7dRo4qq8nmtliYDHA1KlTex0vKSnp9clYETlQQ0MDW16tp6Si/IDyjtY9eHvuz5v0p4VOdjele5W3NzXnHWNUxo0bF/gPXktnZnBZVpI78eayJ5v8u1r39jrmXU4+Y273Lrpae3+/WjudnTt3Bq4v1vv83d3NrM9ucPdlwDLIjPwjC0xkhCmpKI/k9sahrq/RcX9qa2sBAt22WVtbS9s7r3HtzCMDtxfUXS++x+g+Xon1J467fd7NTveQ/f+9GGIQEUm0OJL/b4GvZL/+CvCbGGIQEUm0sG/1/BXwLHCcmW01s6uAHwGfMbN64LzsYxERiVCoc/7ufnkfh84Ns10REemfPuErIpJASv4iIgmk5C8ikkBK/iIiCaTkLyKSQEr+IiIJpOQvIpJASv4iIgmk5C8ikkBK/iIiCaTkLyKSQEr+IiIJpOQvIpJASv4iIgmk5C8ikkCx7uErErW6ujoaGhpyHkunM5uSV+bYDzWVSlFdXR1qbBKf/n4uusu79/Ltqb+fi2272rnrxd671H6wt4PWzuBbkpcWG0eM6Z2yt+1qZ3rg2pT8RfZpbW2NOwQZgkpLSwOfk0ql+jxWlE5TlMfPWlFpac6N2qcfor2+KPlLovQ3eu8e2dXU1EQVjgwRhX5VNxxeJWrOX0QkgZT8RUQSSMlfRCSBlPxFRBIor+RvZhPMbGahgxERkWgMOPmb2SozG2dmFcDzwC/M7F/CC01ERMISZOT/N+6+E7gYqHP3OcB54YQlIiJhCpL8R5nZZOC/AY+FFI+IiEQgSPK/BfhP4DV3f87M/haoDycsEREJ04A/4evuK4AVPR6/DiwIIygREQlXkDd8/87MnjKzzdnHM83sf4YXmoiIhCXItM8vgBuAdgB3fxH4UhhBiYhIuIIk/7Hu/seDyjoKGYyIiEQjSPL/wMyOARzAzC4B/hpKVCIiEqogSzp/HVgGfMzMtgFvAFeEEpWIiIQqSPJvcPfzzOwwoMjdm8MKSkREwhVk2ucNM1sGnA7sCikeERGJQJDk/zHgSTLTP2+Y2f8ys0/m27CZ/ZOZ/dnMNpvZr8ysLN+6REQkmAEnf3ff4+4PufvFwCxgHPB0Po2a2UeAbwBV7n4iUIxuGxURiUygPXzN7FPAZcB8YD2ZdX4G0/YYM2sHxgLvDKIukQPU1dXR0NAQ6Jzu53fv5TtQqVRqSO/Zmk6nadv+IekHnxzwOd7ZCYAVFw/8nPYO6hubc/ZfOp2mNY9Ny0tLS6nMsWk5DP1+H+oGnPzN7E1gI/AQ8G13351vo+6+zcx+ArwF7AVWuvvKHG0uBhYDTJ06Nd/mJIEaGhrY8mo9JRXlAz6ng0zCe60pPeBz2puG/n0P48aNC5x4WzpbACgrGR3gnC68s4O2d17rdaxrbwddnR4oBoCujhba3umdarbtag9clxwoyMh/ZnZJ50EzswnA3wPTge3ACjP7srvf3/N57r6MzO2lVFVVBf/JkUQrqShn4vlzQm2jceW6UOsvhKVLlwY+p3v0XlNTE+ictnde49qZRwZuL6i7Xnwv9DZGuiDJv83Mvg6cAOx7c9bdF+bR7nnAG+7+PoCZPQJ8Ari/37NERKQggtztcx9QCcwj80bvFCDf17xvAaeb2VgzM+Bc4KU86xIRkYCCJP9j3b0G2O3u/wZ8HsjrNbW7rwMeJrMd5J+ycSzLpy4REQkuyLRP9zss283sRCAN5D255+43Azfne76IiOQvSPJfln2jtgb4LXA4cFMoUYmISKiC7OT1r9kvnwb+NpxwREQkCkF28jrKzO4xs99nHx9vZleFF5qIiIQlyBu+95LZwP3o7ONXgG8WOB4REYlAkOR/hLs/BHQBuHsHZD8SKSIiw0qQ5L/bzCayfyev04EdoUQlIiKhCnK3z3Vk7vI5xszWAJOAS0KJSkREQhXkbp/ns6t6HgcY8LK7a3UlEZFh6JDJ38wu7uPQ35kZ7v5IgWMSEZGQDWTk/4V+jjmg5C8iMswcMvm7+1cHUpGZfSW75o+IiAxxQe72OZQlBaxLRERCVMjkbwWsS0REQlTI5K+dtkREhgmN/EVEEijIh7wOZU0B65I81NXV0dDQ0Ks8nU4H3sAboLS0lMrKypzHUqkU1dXVgeuMSjqdpn1Xc+h77LY3NZNuC7WJYSOdTrN9RxvffWZbr2PtXU5XHnMDRQYlRb3Hla2dzviidD5hStaAk7+ZjQeqgWk9z3P3b2T/v6bAsUlADQ0NbHm1npKK8gPKO1r34O3Bl2FqoZPdTb1/wdqb8t29U0aycePG9TnIsLY2rKsrcJ1WVETR6NG9ysdk25P8BRn5/wewlsy2i8G/ixKJkopyJp6f1+6aAxb2aLoQKisr2d1EJH1RWZH71VHSLF26NO4QJIAgyb/M3a8LLRIREYlMkDd87zOzfzSzyWZW0f0vtMhERCQ0QUb+bcBtwI3sv63T0ZaOIiLDTpDkfz1wrLt/EFYwIiISjSDTPq8Ce8IKREREohNk5L8b2GRm/wXsu5+r+1ZPEREZPoIk//+b/SciIsNckJ28tFyziMgIEeQTvm+QY/E2d9fdPiIiw0yQaZ+qHl+XAZcCus9fRGQYGvDdPu7e2OPfNne/A/h8eKGJiEhYgkz7zO7xsIjMK4FCrgoqIiIRCZK8b2f/nH8H8CaZqR8RERlmgiT/zwILOHBJ5y8BtxQ4JhERCVnQ+/y3A88DLWEEIyIi0QiS/Ke4+/zQIhERkcgEWdvnGTM7KbRIREQkMkGS/yeBDWb2spm9aGZ/MrMX823YzMab2cNmtsXMXjKzM/KtS0REggn6hm8h/Qx43N0vMbPRwNgC1y8iIn0IsrZPQ6EaNbO/AeYCV2brbiOzWUzB1NXV0dCQO+R0OrMpeWVl7r1XU6kU1dXVhQwnEul0mvZdzaHvsdve1Ey6oN+tcLQ39e6Ljub8NrO3kmJGlfcen3T3xaJFi3od27t3L+69VkQ5dFtmjBkzJuexuXPnFuxns7/fke7y2traXseG6++HHCiuD2lNB94HfmlmJwMbgCXuvrvnk8xsMbAYYOrUqQVrvLW19dBPkmEtlUrlLE+3pWkl+Pe/tLQ090btFZXs2bOH999/P3CdQ1lpaWncIUjILJ+RyaAbNasC1gJnuvs6M/sZsNPda/o6p6qqytevX1+Q9rtHMzU1fTY3LNXW1vJaU5qJ588JtZ3Gles4pqJyxPWfyEhkZhvcverg8iBv+BbSVmCru3e/Jn8YmN3P80VEpIBiSf7ungbeNrPjskXnAn+JIxYRkSSKc2G2a4EHsnf6vA58NcZYREQSJbbk7+6bOHCPABERiUhcc/4iIhIjJX8RkQRS8hcRSSAlfxGRBFLyFxFJICV/EZEEUvIXEUkgJX8RkQRS8hcRSSAlfxGRBFLyFxFJICV/EZEEUvKXyHz44YfccsstbN++Pe5QYqe+kLgp+UtkHn30UV5++WUeeeSRuEOJnfpC4qbkL5H48MMPefrpp3F3Vq9enegRr/pChoI4N3MpiBtuuCHw5tktLS0ALFq0KNB5kyZNYunSpb3K6+rqaGhoyHlOOp0GoLKy9+bfqVSK6urqQDEcSntTM40r1x36iVkdzXsAGFU+NlAb5NrMvB+PPvoo3ftFd3V18cgjj7Bw4cJAdYwU6gsZCoZ98t+5cyd79u7BRg38UpzML97e9raBn9PRwc6dOwPH19raGvicfKVSqcDnNDRn/milgiTzisrAba1Zs4aOjg4AOjo6WLNmTWITnvpChoJhn/wrKyvZPRomnj8n1HYaV66jso8E2d/ovba2FoCamppQ4hpoHH2JKr4zzzyTVatW0dHRwahRozjzzDNDbW8oU1/IUKA5f4nERRddhJkBUFRUxMUXXxxzRPFRX8hQoOQvkZgwYQKf+tSnMDPmzp3L+PHj4w4pNuoLGQqG/bSPDB8XXXQRW7du1UgX9YXET8lfIjNhwgRuuummuMMYEtQXEjdN+4iIJJCSv4hIAin5i4gkkJK/iEgCKfmLiCSQkr+ISAIp+YuIJJCSv4hIAin5i4gkkJK/iEgCKfmLiCSQkr+ISAIp+YuIJFCsyd/Mis1so5k9FmccIiJJE/fIfwnwUswxiIgkTmzr+ZvZFODzwK3AdYOpq72pmcaV6wb8/I7mPQCMKh8bqI102/49bweqoSGzQXrQ81KpVF578uZSV1e3L46D9RdffzHU1dWxevXqXuV79+7F3QPHaGaMGTMm57G5c+cWrC9EJCPOzVzuAL4DlPf1BDNbDCwGmDp1as7npFKpwA03NGcSXqqPDdlzqqgknU7zxitb+MjhJQM+raSjA4C2d14b8DnbdrUPPK5BKi0tjawtERk6LJ9R2qAbNbsA+Jy7X21mZwPfcvcL+junqqrK169fX5D2u0e5NTU1gc9re+c1rp15ZEHi6MtdL77H6KOPCRyfiMjBzGyDu1cdXB7XnP+ZwIVm9ibwIPBpM7s/plhERBInluTv7je4+xR3nwZ8Cfh/7v7lOGIREUmiuO/2ERGRGMT5hi8A7r4KWBVzGCIiiaKRv4hIAin5i4gkkJK/iEgCKfmLiCSQkr+ISAIp+YuIJJCSv4hIAin5i4gkkJK/iEgCKfmLiCSQkr+ISAIp+YuIJJCSv4hIAin5i4gkkJK/iEgCxb6ef1jq6upoaGjIeay7vHsv34OlUimqq6t7lafTabbvaOO7z2w7oLy9y+nKYyvkIoOSIutV3trpjC9KB69QRGSARmzy709paWle540bN47W1tZe5dbWhnV1Ba7PioooGj26V/mYbFsiImEx9zyGrDGoqqry9evXxx2GiMiwYmYb3L3q4HLN+YuIJJCSv4hIAin5i4gkkJK/iEgCKfmLiCSQkr+ISAIp+YuIJJCSv4hIAg2bD3mZ2ftA7vUaonME8EHMMQwV6ov91Bf7qS/2Gyp9kXL3SQcXDpvkPxSY2fpcn5RLIvXFfuqL/dQX+w31vtC0j4hIAin5i4gkkJJ/MMviDmAIUV/sp77YT32x35DuC835i4gkkEb+IiIJpOQvIpJAiUr+ZvZTM/tmj8f/aWb/2uPx7WZ2XcA67zWzS3KUX2Nmr5qZm9kRgwo8BBH3xQNm9rKZbTaz5WZWMqjgCyzivrjHzF4wsxfN7GEzO3xQwRdYlH3R4/idZrYrr4BDFPHPxb1m9oaZbcr+O2UwsQ9EopI/sAb4BICZFZH5EMYJPY5/AnimgG2dR/wfTOtLlH3xAPAx4CQyu1QuKlC9hRJlX/yTu5/s7jOBt4BrClRvoUTZF5hZFTChUPUVWKR9AXzb3U/J/ttUwHpzSlryfwY4I/v1CcBmoNnMJphZKTADcDN72sw2ZP/STwYws2PM7PFs+R/M7GMHV25mtdm/4MXuvtHd34zouvIRZV/8h2cBfwSmRHOJAxZlX+zMlhmZP4RD7Y6LyPrCzIqB24DvRHNpgUXWF1FdUE+JSv7u/g7QYWZTyfzVfhZYR+YbXAW8BPwUuMTdTwWWA7dmT18GXJst/xbw8551m9ltwCTgq+7eGcHlDEocfWGZ6Z5/AB4P8dICi7ovzOyXQJrMq6G7wr26YCLui2uA37r7X0O/sDzE8Dtya3Y68KfZPy6hGhV2A0PQM2S+kZ8A/gX4SPbrHcA24HzgiczAjGLgr5aZl/0EsCJbDtDzm1MDrHP3xVFcQAFF3Rc/B1a7+x8KfymDFllfuPtXs6O9u4DLgF+GdE35Cr0vzOxo4FLg7JCvZbCi+rm4gcyAYDSZPxz/A7glnEvKSGLy757HO4nMy7i3geuBncAq4CPufkbPE8xsHLDd3U/po87ngFPNrMLdm0KKOwyR9YWZ3UxmpPO1Al9DoUT6c+HunWb2IJkpj6GW/KPoi1nAscCr2QQ51sxedfdjC385gxLJz0WPVz+t2VeG3yrwdfSSqGmfrGeAC4Amd+/Mdv54Mi/lfgVMMrMzIDNNYWYnZOdp3zCzS7PlZmYn96jzceBHwL+bWXmE1zJYkfSFmS0C5gGXu3tXRNcWVOh9kT1+bPdzgQuBLRFdXxCh94W7/7u7V7r7NHefBuwZgokfovsd6X6vwIAvkvlDE6okJv8/kXnXfu1BZTvc/T3gEuDHZvYCsInsu/3AFcBV2fI/A3/fs1J3XwH8AvitmY0xs2+Y2VYyb26+aD1uERtCIukL4G7gKOBZy9zGdlN4l5S30PsCGAv8m5n9KVv3ZEJ+aZ+nqH4uhoOo+uKBHj8XRwD/HNoVZWl5BxGRBEriyF9EJPGU/EVEEkjJX0QkgZT8RUQSSMlfRCSBlPxFcjCz8WZ2dfbro83s4bhjEikk3eopkoOZTQMec/cT445FJAxJXN5BZCB+BBxjZpuAemCGu59oZleS+QTmYcBHgZ+QWY/lH4BW4HPu3mRmxwD/m8ySFnuAf3T3ofhpXkkoTfuI5PZd4LXs+izfPujYicDFwMfJrOK4x91nkVn1sTr7nH5XdRSJm0b+IsH9l7s3k1nbfQfwu2z5n4CZA1jVUSR2Sv4iwbX2+Lqrx+MuMr9TRfS/qqNI7DTtI5JbM5DXCq0DWNVRJHZK/iI5uHsjsMbMNpPZajCofld1FImbbvUUEUkgjfxFRBJIyV9EJIGU/EVEEkjJX0QkgZT8RUQSSMlfRCSBlPxFRBLo/wNyxKALehm/VAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# we use another dataset to illustrate the concept\n",
    "data_rm2 = pd.read_csv(\"https://reneshbedre.github.io/assets/posts/anova/plants_leaves_two_within.csv\")\n",
    "pg.print_table(data_rm2.head())\n",
    "\n",
    "sns.boxplot(\n",
    "    x='time',\n",
    "    y='num_leaves',\n",
    "    data=data_rm2,\n",
    "    hue='year',\n",
    "    palette='Set2',\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Source</th>\n",
       "      <th>SS</th>\n",
       "      <th>ddof1</th>\n",
       "      <th>ddof2</th>\n",
       "      <th>MS</th>\n",
       "      <th>F</th>\n",
       "      <th>p-unc</th>\n",
       "      <th>p-GG-corr</th>\n",
       "      <th>np2</th>\n",
       "      <th>eps</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>time</td>\n",
       "      <td>226.88</td>\n",
       "      <td>4</td>\n",
       "      <td>16</td>\n",
       "      <td>56.72</td>\n",
       "      <td>158.6573</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.9754</td>\n",
       "      <td>0.4212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>year</td>\n",
       "      <td>18.00</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>18.00</td>\n",
       "      <td>3.6000</td>\n",
       "      <td>0.1306</td>\n",
       "      <td>0.1306</td>\n",
       "      <td>0.4737</td>\n",
       "      <td>1.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>time * year</td>\n",
       "      <td>0.80</td>\n",
       "      <td>4</td>\n",
       "      <td>16</td>\n",
       "      <td>0.20</td>\n",
       "      <td>1.4545</td>\n",
       "      <td>0.2621</td>\n",
       "      <td>0.2917</td>\n",
       "      <td>0.2667</td>\n",
       "      <td>0.4416</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Source      SS  ddof1  ddof2     MS         F   p-unc  p-GG-corr  \\\n",
       "0         time  226.88      4     16  56.72  158.6573  0.0000     0.0000   \n",
       "1         year   18.00      1      4  18.00    3.6000  0.1306     0.1306   \n",
       "2  time * year    0.80      4     16   0.20    1.4545  0.2621     0.2917   \n",
       "\n",
       "      np2     eps  \n",
       "0  0.9754  0.4212  \n",
       "1  0.4737  1.0000  \n",
       "2  0.2667  0.4416  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.rm_anova(\n",
    "    dv='num_leaves',\n",
    "    within=['time', 'year'],\n",
    "    subject='plants',\n",
    "    data=data_rm2,\n",
    "    detailed=True,\n",
    ").round(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We reject the null hypothesis in favor of the alternate hypothesis for time, but not for year and time + year interaction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repeated-measures ANOVA with statsmodels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                 Anova\n",
      "========================================\n",
      "          F Value  Num DF  Den DF Pr > F\n",
      "----------------------------------------\n",
      "time      158.6573 4.0000 16.0000 0.0000\n",
      "year        3.6000 1.0000  4.0000 0.1306\n",
      "time:year   1.4545 4.0000 16.0000 0.2621\n",
      "========================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from statsmodels.stats.anova import AnovaRM\n",
    "\n",
    "res_rmANOVA = AnovaRM(\n",
    "    data=data_rm2,\n",
    "    depvar='num_leaves',\n",
    "    subject='plants',\n",
    "    within=['time', 'year'],\n",
    ")\n",
    "print(res_rmANOVA.fit())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Two-way ANOVA (also called _two-factor ANOVA_)\n",
    "\n",
    "The data are divided in two ways because each data point is either from an animal given either an inactive or active treatment (one factor) given for either a short or long duration (second factor). If male and female animals were both included you'd need three-way ANOVA, so that each value would be categorized in three ways, i.e. treatment, duration and gender.\n",
    "\n",
    "Two-way ANOVA simultaneously tests 3 null hypotheses and so computes 3 P values:\n",
    "\n",
    "1. There is no **interaction between the two factors** (treatment and duration), i.e. H0 is that the difference in effects between durations is the same in animals given active or inactive treatment, or that the treatment causes the same change in response if given for short or long duration.\n",
    "2. The population means are identical for animals given placebo and active treatment (treatment, i.e. **first factor**), pooling short and long duration.\n",
    "3. The population means are identical for animals given a treatment for short vs. long duration (duration, i.e. **second factor**).\n",
    "\n",
    "\n",
    "### How two-way ANOVA works\n",
    "\n",
    "The bottom row of the ANOVA table quantifies the total variation among all the values, paying no attention to which row or column they are in. It is the **sum-of-squares of the squared difference between each value and the grand mean**. The rest of the table breaks this variability into four components: interaction, differences between rows (on the average), differences between columns (on the average) and variation amoing replicates in the same row and column. Each MS is divided by the residual MS to compute an F ratio, and this is used to obtain a P value.\n",
    "\n",
    "| Source of variation | Sum squares | DF |  MS   |\n",
    "| -------------------:|:----------- | -- | ----- |\n",
    "|   Interaction       |    12896    |  1 | 12896 |\n",
    "| \\+ Between rows     |     5764    |  1 |  5764 |\n",
    "| \\+ Between columns  |     3710    |  1 |  3710 |\n",
    "| \\+ Among replicates |      928    |  8 |   116 |\n",
    "| \\= Total            |    23298    | 11 |       |\n",
    "\n",
    "![Partionining of the sum of squares](https://www.marsja.se/wp-content/uploads/2016/03/two-way-anova-sum-of-squares.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Ss  Blend    Crop      Yield\n",
      "----  -------  ------  -------\n",
      "   1  Blend X  Wheat       123\n",
      "   2  Blend X  Wheat       156\n",
      "   3  Blend X  Wheat       112\n",
      "   4  Blend X  Wheat       100\n",
      "   5  Blend X  Corn        128\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAt1ElEQVR4nO3dd3yV9fn/8deVsIeADBkBwghEGQkxoCAojgpOquDE1ar4rdhv+2urlS5tLdW62n7LV1u+grNiFVFxgVpFRVQ2gcgmQcKQpWGPJJ/fH/edkxNysiBnJOf9fDzOg5zPfZ/kCoFc57qvz+dzm3MOERERgIRoByAiIrFDSUFERAKUFEREJEBJQUREApQUREQkoF60AzgRbdq0ccnJydEOQ0SkVlm0aNFO51zbUMdqdVJITk5m4cKF0Q5DRKRWMbON5R3T5SMREQlQUhARkQAlBRERCajVPYVQjh49Sl5eHocOHYp2KDGtUaNGJCUlUb9+/WiHIiIxpM4lhby8PJo3b05ycjJmFu1wYpJzjl27dpGXl0e3bt2iHY6IxJA6d/no0KFDtG7dWgmhAmZG69atVU2JSBl1LikASghVoL8jEQmlTiYFEZHqSL73bZLvfTvaYcQEJYVqSkxMJD09nbS0NDIyMpg3bx4Aubm59O3bt0a+xpw5c7j00ktr5HOJiFRHnWs0h1vjxo1ZunQpALNnz2bChAl8/PHH0Q1KRKSGqFI4AXv27KFVq1ZlxgsLC7n77rsZOHAg/fv355///CfgVQDDhw9nzJgxpKamMnbsWIrvfDdr1ixSU1PJyMhgxowZEf0+RESKqVKopoMHD5Kens6hQ4fYunUrH374YZlzpkyZQosWLViwYAGHDx/mrLPO4sILLwRgyZIlZGdn07FjR8466yw+++wzMjMzuf322/nwww/p2bMn11xzTaS/LQmX+1v4f+ZHNw6RKgpbpWBmU81su5mtCBpLN7MvzGypmS00s0H+uJnZ/5jZOjPLMrOMcMV1ooovH61atYpZs2Zx0003cex9rt977z2ee+450tPTOeOMM9i1axdr164FYNCgQSQlJZGQkEB6ejq5ubmsWrWKbt26kZKSgplxww03RONbExEJa6XwDDAJeC5o7GHg9865d83sYv/5cOAiIMV/nAE86f8Z0wYPHszOnTvZsWNHqXHnHH//+98ZMWJEqfE5c+bQsGHDwPPExEQKCgoiEquISFWErVJwzn0C7D52GDjJ/7gFsMX/eBTwnPN8AbQ0sw7hiq2mrFq1isLCQlq3bl1qfMSIETz55JMcPXoUgDVr1rB///5yP09qaiq5ubmsX78egGnTpoUvaBGRCkS6p/BTYLaZPYqXkIb4452ATUHn5fljW4/9BGY2DhgH0KVLl3DGGlJxTwG8iuDZZ58lMTGx1Dm33XYbubm5ZGRk4Jyjbdu2vP766+V+zkaNGjF58mQuueQSmjRpwrBhw9i7d28YvwsRkdDs2OvhNfrJzZKBt5xzff3n/wN87Jx71cyuBsY55y4ws7eAh5xzc/3z/gP80jlX4R10MjMz3bE32Vm5ciWnnnpqGL6bukd/VxGgRnOtULxwLfehS6IcSWSY2SLnXGaoY5GeknozUDzf8hVgkP/xZqBz0HlJ/piIiERQpJPCFuAc/+PzgLX+xzOBm/xZSGcC+c65MpeOREQkvMLWUzCzaXgzi9qYWR5wH3A78Dczqwccwu8NAO8AFwPrgAPAD8IVl4iIlC9sScE5d105h04Pca4DxocrFhERqRptcyEiIgFKCiIiEqCkEAbR3F57xowZnH/++YHnc+fOJT09XSunRaRK4jopFBQW8fKCTYx+ch5DHvwPo5+cx8sLNlFYdGJrN4r3R1q2bBkPPvggEyZMqKGIK3fllVfSsGFDXnzxRY4ePcqdd97JE088Qb162vtQRCoXt78pCgqLuOvFJczK3hYY25J/iEUbv+XDVduZdP0A6iWeeM6saHvte++9lzlz5nD48GHGjx/PHXfcwZw5c7j//vtp06YNK1as4PTTT+eFF17AzJg1axY//elPadKkCUOHDi33a06aNIkLLriA7OxsBg4cyJAhQ8o9V0QkWNwmhRmLN5dKCMFmZW9jxpLNXJ3ZOeTxykR7e+3u3btzzTXXMGnSpMB+SiIiVRG3l4/+vXBThcdfXlDx8YpEe3vtwsJC3n//fZo1a8bGjRuP+/sQkfgTt0lh63cHKzy+pZLjVVXZ9tpLly5l6dKl5OTkBCqFE91e+4knnqBfv35MmTKF8ePHl0lIIiLliduk0KFl4wqPd6zkeFVFenvtbdu28fjjj/Pwww8zcuRIOnXqxFNPPVUj34uI1H1x21O4JrMzizZ+W+7xqwceXz8Boru99s9+9jPuuece2rZtC8Bf//pXhg0bxujRozn55JOP+3sSkfgQ1q2zw+1Ets4uLHKM/9fikM3mkX3a879jM0hMsBqLNRZp6+wI0NbZtYK2zi4Rt5VCYoIx6foBzFiymZcXbGLLdwfp2LIxVw/szOiMpDqfEEREQonbpABQLzGBqzM7H/fUUxGRuiZuG80iIsW7GhSrqV0NarO4rhREJH6F2tVg0cZva3xXg9om/r5jERGqtqtBPApbUjCzqWa23cxWBI3928yW+o9cM1sadGyCma0zs9VmNiJccYmIQHh3NajNwlkpPAOMDB5wzl3jnEt3zqUDrwIzAMzsNOBaoI//mifMrPTE/lokWltnb9++neTkZLZtK3n3M378eB588MEa+ZoidcnXu8pfLAo1t6tBbRPO23F+YmbJoY6ZmQFXA+f5Q6OAl5xzh4EcM1sHDAI+D1d8ABQWwLJpsOR5yN8MLTrBgBsh/XpIOP6cVLz3EcDs2bOZMGECH3/8cQ0FXb527dpx77338otf/IIXXniBxYsX8+mnn7Jo0aKwf22R2qKgsIh/fLyenfuOVHheTe1qUNtEq6cwDPjGObfWf94JCK7V8vyxMsxsnJktNLOFx+4nVC2FBTD9Fph5F2z6EvbkeX/OvAteudk7XgMq2jr77rvvZuDAgfTv359//vOfgFcBDB8+nDFjxpCamsrYsWMDexfNmjWL1NRUMjIymDFjRsivN27cONavX89HH33E+PHjmTRpEvXr16+R70Wktlu9bS9XPDGPR99bQ2Xzi05kV4PaLFqzj64DQm/eUwnn3GRgMngrmo87gmXTYOWboY+tfBOyXoIB5e9EWpFobp2dkJDAk08+yXnnncfll1/O2WeffVzfg0hdUlBYxD8/2cDfPljLkcIiABrVTyC5dVNWbSu7XczIPu0ZnZEU6TBjQsSTgpnVA64ETg8a3gwEp+Ukfyx8ljxf8fHFzx93Ugi+fPT5559z0003sWLFilLnvPfee2RlZTF9+nQA8vPzWbt2LQ0aNAhsnQ0Ets5u1qxZYOtsgBtuuIHJkyeH/Prp6en07duXO++887jiF6lL1n6zl1+8soxleSVbjWR2bcUjV6XRuVVjZizZzD3TswLj8b6rQTQqhQuAVc65vKCxmcCLZvY40BFIAeaHNYr8SnJOfl7Fx6uosq2zR4woPdFqzpw5J7x1NngVQ0KCZhxL/CooLOL/Ps3hL++vCVQHDeslcPeI3vzgrG6BX/pXZ3YOJIXpP9JdCsM5JXUaXqO4t5nlmdmt/qFrOebSkXMuG3gZ+AqYBYx3zhWGKzbAaypXeLxmSsdIb50tIrBu+15G/+Nz/jxrVSAhnN61Fe/+ZBi3Deset1VAVYRz9tF15YzfUs74RGBiuOIpY8CNXmO5PBk3HvenjubW2RI98bbTZiwqLHI89ekGHnt/DUcKSqqDX1zYmx8O7aZkUAVxu3U2RYXeLKNQzeZTL4Ornj2haam1gbbOrlkhk4K2zo6Y9Tv28YtXlrHk6+8CYwO6tOTRq9Lo0bZZha+Nt4SurbNDSUiEMc94s4wWP+/1EFokeRVC2nV1PiGI1BWFRY6pc3N49L3VHPargwb1Evj593rpUtFxiN+kAJBYz5thdJyzjEQkujbs2Mfd07NK3UUxrXNLHruqPz3bNY9iZLVXnUwKzjm8RdNSntp82VCksMjx9Gc5PDI7qDpITOD/fa8Xtw/rFpe7m9aUOpcUGjVqxK5du2jdurUSQzmcc+zatYtGjRpFOxSRasvZuZ+7X1nGwuDqIKkFj16VRsopqg5OVJ1LCklJSeTl5ZVZFyClNWrUKLBATqQ2KCpyPDMvl4dnr+LQ0ZLq4CcXpHDH2d1VHdSQOpcU6tevT7du3aIdhojUoNyd+7lnehbzc3cHxvp18qqD3u1VHdSkOpcURKTuKCpyPPd5Ln+etZqDR731rPUTjZ+cn8Id5/SgvqqDGqekUBtorrvEoa93HeDu6cv4MqekOujb6SQevSqN1PYnRTGyuk1JQURiSlGR44UvN/LQu6s4cKSkOvjxeSn8aLiqg3BTUhCRmLFpt1cdfLGhpDo4rcNJPHZ1Gqd2UHUQCUoKIhJ1RUWOf83/mgffWRmoDuolGHed15Px5/ZUdRBBSgoiElWbdh/gl69mMW/9rsBYavvmPHZ1Gn06tohiZPFJSUFEosI5x4vzv+ZPb69kf1B1cOe5Pbnr3J40qKfqIBqUFEQk4vK+PcC9ry5n7rqdgbHU9s159Ko0+nZSdRBNSgoiEjHOOV5asImJb69k32HvjoKJCcadw3vw4/NSVB3EACUFEYmILd8d5JevZvHp2pLqoNcpzXjsqnT6Jak6iBXhvB3nVDPbbmYrjhn/sZmtMrNsM3s4aHyCma0zs9VmNqLsZxSR2sg5x78XfM2Iv3wSSAiJCcb4c3vw5o+HKiHEmHBWCs8Ak4DnigfM7FxgFJDmnDtsZu388dPw7t3cB+gIfGBmvcJ+n2YRCaut+Qe599XlfLymZIPKlHbNePSqNNI6t4xeYFKucN6j+RMzSz5m+EfAQ865w/452/3xUcBL/niOma0DBgGfhys+EQkf5xyvLMrjgbe+Yu8hr3eQYHDHOT34yfkpNKqvOxvGqkj3FHoBw8xsInAI+IVzbgHQCfgi6Lw8f6wMMxsHjAPo0qVLeKMVkWrbln+ICTOy+Gh1SXXQo21THrs6nXRVBzEv0kmhHnAycCYwEHjZzLpX5xM45yYDkwEyMzN1+zCRGOGc49XFm/n9m9mlqoPbz+7O/7ugl6qDWiLSSSEPmOG8e0HON7MioA2wGegcdF6SPyYitcA3ew4xYcZyPly1PTDWvW1THr0qjYwuraIYmVRXpJPC68C5wEdm1gtoAOwEZgIvmtnjeI3mFGB+hGMTkWpyzvHaks3cPzObPX51YAa3D+vOz75Xe6qD3IcuiXYIMSNsScHMpgHDgTZmlgfcB0wFpvrTVI8AN/tVQ7aZvQx8BRQA4zXzSCS2bd9ziF+9tpwPVgZVB22a8shV/Tm968lRjExORDhnH11XzqEbyjl/IjAxXPGISM1wzvHG0i3cNzOb/INHAa86uPWsbvxiRO9aUx1IaFrRLCJVtn3vIX792gre/+qbwFhy6yY8elUamcmqDuoCJQURqZRzjpnLvOrguwMl1cEPhnTj7hG9adxA1UFdoaQgIhXasfcwv3l9ObOzS6qDrq2b8MiYNAZ1U3VQ1ygpiEhIzjneytrK795Ywbd+dQBwy5Bk7hnZmyYN9OujLtJPVUTK2LnvML99fQXvrtgWGOtychMeHtOfM7u3jmJkEm5KCiJSyttZW/ntGyvYvf9IYOzmwV355UWpqg7igH7CIgLArn2H+d0b2by9fGtgLKlVYx4Zk8bgHqoO4oWSgkg4FBbAsmklz6dcCANuhPTrISH2Zuq8u3wrv3l9BbuCqoMbz+zKvRel0rShfk3EE/20RWpaYQFMvwVWvlkytulL77F2Nox5BhJj47/e7v1H+N0bK3grq6Q66NSyMY+M6c+Qnm2iGFmMuN+/AdD9+dGNI4Ji41+mSF2ybFrphBBs5ZuQ9RIMCLmwP6JmrdjGb15fzs59JdXB2DO6MOHiU2mm6iBu6ScvUoMacgTm/b3ikz6cCE1aQ9IgaBr5a/Xf7j/CfTOzmblsS2CsU8vG/Hl0f4amqDqId0oKIidq/y5YO5t/1J/KsIQs2Hm44vP3boFp13ofn9wDOg/yH2dA29Sw9hzey97Gr15bwc59JTFeN6gLv7o4leaN6oft60rtoaQgcjx2roPVb8Pqd71egSti5PH8Lt+93nsUN6UbNIekzJJE0SkTGrc84XC/O3CE+2dm8/rSkuqgY4tGPDS6P2f3anvCn1/qDiUFkaooKoS8BbDKTwS71oY8bW1RJ1J6nQbr3i//c531U+8X/ab5XkI5sKvk2JG9sOEj7wGAedVDcDXRuqe38VAVffDVN0x4bTk79pZUB9cO7MyvLjmVk1QdyDGUFETKc2Q/rP8IVr8Da2aV/uVdzBKgy2DofRHD32xMrutA7vUj4ZWbQzebT70Mzv9dySUi52D3Bi9B5M33/vwmGyi+06yDHSu9x+JnvaHGJ0PSwKBq4nRo0LTMl8o/cJTfv5nNjCUlNzHs4FcH56g6kHIoKYgE27vNqwRWvwsb5kBhiP5A/abQ83zofTGkXBhoFufOfNs7npDoTTvNegneGO+NdT4TMm6EtOtK9wzMoHUP75Hu34Lk0B7YvCgoUSyAw0FTIg/u9qa2rp3tf45EaN/Xa1x3PgM6D+TDbY2Y8NoKvtlTEv/VmUn85tLTVB1IhZQUJL45B9u/8qqB1e96v4xDad4Bel/kJYLkYVC/UcWfN7GeN+20OCncOrvqMTU6CXqc6z0Aiopg5xp/rYOfKHauCfoeCmHrMu+x4P8A6Otacn9RCosTU8hp3IcbrxzFOad1DvHFREoL5+04pwKXAtudc339sfuB24Ed/mm/cs694x+bANwKFAL/7Zyrxv8ikWooPAob5/kVwTvw3cbQ553Sz08EF0HHAdW6jl+jEhKgXar3OP1mb+zAbq/HUdyX2LwYju4PvKSdfcdFiQu4KHGBd4PbV38P89K8SiJpoPfnSR2i8/1ITAtnpfAMMAl47pjxvzjnHg0eMLPTgGuBPkBH4AMz66X7NEuNOZQP6z7wEsHa97znx0qoB8lDvWqg90XQskvk46yqJidDrxHQawR7Dh1l4swsViz5goyENWQkrGVQ4lo6UXLvZAqPeEkkb0HJWIvOJc3rpIHQvh8k6tJSvAvnPZo/MbPkKp4+CnjJOXcYyDGzdcAg4PNwxSdx4LuvYfUsrxrInQtFR8ue07AFpHwPUi+GnhdAoxaRj/MEfLxmB/e+msXW/ENAMtmFyexP+wHnXdoHCnf7PYkvvb7EliWleyT5m7zHile95/UaQ6eMoEQRncV1El3R6CncZWY3AQuBnzvnvgU6AV8EnZPnj5VhZuOAcQBdusTwOzmJPOdg69KSy0Lbloc+r2UXvxq4GLoOqZXvjvccOsrEt1by74WbAmNtmzfkwSv6ccFpp/gjp3iznU69zHtacNj7Oyneh2nTfNhbsucRBQdh42feo1iEF9dJ9EU6KTwJPIA33+4B4DHgh9X5BM65ycBkgMzMTFfJ6VLXFRyGnE9LGsV7t4Q+r2OGlwRSL4Z2p0WvP1ADPvGrgy35hwJjVwzoxH2XnUbLJg3Kf2G9ht7CuKRMGDzeS6L5eSVTYTd96SWNooKS15S7uM6b5VRTi+skdkQ0KTjnAjd5NbP/A97yn24GgqdGJPljImUd2O31BVa9Des/hCP7yp6T2BC6n+Mlgl4j60RTde+ho/zpnZVMm19SHbRp1pA/XdGXC/u0r/4nNIOWnb1H39He2JED3mWmTV/6jezwL66T2BLRpGBmHZxzxfXqFcAK/+OZwItm9jheozkFmB/J2CTG7Vpfclno68/BFZU9p/HJXgJIvRi6nwsNm0U+zjCZu3Ynv3w1i83fHQyMjUrvyP2X9aFV0wqqg+pq0ASSz/IeUHpxXXGiqKHFdRKbwjkldRowHGhjZnnAfcBwM0vH+xeVC9wB4JzLNrOXga/wJtCN18yjOFdU6K0ZKN5WYufq0Oed3MNLAr0v8X4B1bHr3fsOF/Cnd1by4pdfB8baNGvAH7/fj5F9j6M6qK7KFtdt+hLyFlZzcd0gr6+jaiImhXP20XUhhqdUcP5EYGK44pFa4MgBbxXx6rdhzWzYvyPESQZdzixZSNYmJdJRRsy8dTu5e3rp6uCytI78/vI+nFyT1UF1hVxct9pPEn6iCN4bKsTiOpqd4iWH4kTRIa3yBYESEeUmBTPbS0mNWIZz7qSwRCTxZd/2oG0lPoKCQ2XPqd8Eepzn9wdGQNO6vef//sMFPPTuKp7/omRRXeumDfjj9/tyUb8Y7I0kJEC7U71HmcV1/iynzYvg6IGS1+z7xtsbqnh/qMQGXmLQ4rqoKzcpOOeaA5jZA8BW4HnAgLGAflpyfJyDHatKZgvlLSTke49mp5RUA93OhvqNIx5qNHy+fhf3vLqMTbtLqoNL+nfgD5f3oXWzhlGMrJqCFtcB3i1Kt2eXVBKb5pdeSa7FdTGjKpePLnfOpQU9f9LMlgG/C1NMUtcUFnjN4eJG8bc5oc9r16ckEXQc4L0DjRMHjhTw53dX8eznJb8oT27agAdG9eWS/nXgPVhiPa8S6JAGg273xvZ+E7S4bj5sWarFdTGgKklhv5mNBV7Ce0t3HbC/4pdI3Du8t2RbiTWz4dB3Zc+xRG+WS/G00ZO7RTzMWPDFhl3cMz2Lr3eXXF65uF97/jCqL21qU3VQXc1DLK7bmlU6UVRpcZ2/ZkKL62pEVZLC9cDf/IcDPvPHRErLzyvpD+R+6l0SOFbDk7ztJHpfDCkXQONWkY8zRhw4UsDDs1bzzLzcwFirJvV54Pt9ubR/x+gFFi31Gvq/3AeWXlwXvGZia5bXuC4WWFz3ovdci+tOWKVJwTmXi7c3kUhpzsG2rJLLQluXhT6vReeS3Ua7DoV6UZw5EyPm5+zm7unL2LirpDoY2ac9D3y/L22b1+HqoDqCF9f1G+ONBS+uK95GXIvralRFs4/+TsWzj/47LBHFseR7vZu05D50SZQjqUDBEa8KKK4I9uSFPq9Desluo+376T+h7+CRQh6ZvZqn5+Xg/P9dLZvU5w+j+nJZ/w6Y/p4qVtniuk3zvftjVHlx3Rlen+LYxXWFBSVbewBMuRAG3Ajp19f5y1MVVQoLIxaFxLYDu/3+wDuw9gPv3dixEht4s4R6XwS9LoIWIfczjGsLc3dz9/QscnaWtOQuPO0U/nhFX9o11xz941Lu4rqF3s6wx7O4rtPp8P5vYdVbJa8p3kRw7WzvrnqJdff+ZBVNSX02+LmZNXHOHSjvfKljdueUXBbaOK/0ddxijVtByggvEfQ8Hxo2j3yctcQf3/qKKZ+VVActGtfnD6P6cHlaR1UHNa3RSd66lh7nec8Di+u+LEkUlS2uK8/KN73brA64IXzxR1ml6c7MBuOtRG4GdDGzNOAO59yd4Q5OIqioCLYs9pLAqne8cjuUVt0g9RLv0lDnM+r0O6aa9NTckmm4F5x6Cn+6oi/tTlJ1EBGlFtfd4o1VtriuIoufj++kAPwVGIG3aR3OuWVmdnY4g5IIOXoQNnzsbSuxehbs3x7iJPOuv/a+yEsGbXqpP1AFh44W8vj7a/hV0NhJjerx+1F9+H56J1UH0RZqcd03K7xE8d5vvemv5ckvp49WR1TpbZ5zbtMx/4i1WV1ttW8HrJnlXRpa/2Hof/z1Gnv72hRvK9GsXeTjrEUKCouYsbhkp/cRf/2E3fuOsGPfYX7lFwPnp7bjT1f24xRVB7EpsR50TPcey1/xKojytEiKVFRRUZWksMnMhgDOzOoDPwHKubYgMcc52LnWrwbe9UrlUJPKmrb1t52+BLqd483ykEoVFBZx14tLmJW9LTC2elvZRvxTN2eqOqgtBtxYcVLIuDFysURBVZLCf+EtXOuEd+Ob94Dx4QxKTlBhgfePunh/od3rQ5/XNtVfP3CJN+MijraVqCkzFm8ulRDKo4RQi6Rf780yKt6sL9ipl0FaqA2g646qLF7bibcJnsSyw3u9y0HF20oc3F32HEuALkO8+w/0GulN45Pjln/wKH/7z9rKT5TaJSHRm3aa9RK84b//7XymVyGkXRe/6xTM7B7n3MPlLWLT4rUIqGwBzZ4tJYvIcj4Ova1Eg2bedNHel0DK97wGm5yQjbv28/Rnuby8cBMHjqi9Vicl1vNmGBUnhVtnRzeeCKqoUnjQvxuaFrFFQ2EBTL+ldAlbvIBm3t+9raS3Lg392pM6lWwrkTzM21NGTohzjoUbv+WpTzfw3lffBNYbiNQ1FSWFa/H6B88CDzvnjlbnE5vZVOBSYLtzru8xx34OPAq0dc7tNO+C69+Ai4EDwC3OucXV+Xp1zrJpoa9pQuhbU7bvX7KtRIc0TRutIUcLi3hn+Vamzs1hWV5+qWOtmzYgo0sr3l/5TZSiE6l5Fa1ofsXM3gV+Cyw0s+eBoqDjj1fyuZ8BJgHPBQ+aWWfgQuDroOGLgBT/cQbwpP9n/FryfCUnWNC00ZHepmFSY/IPHuWl+V/zzLxctuaXvhtcSrtm3DasG6PSO1E/MYHx/1ocstk8sk97KKfHLxKrKms0H8G7d0JDoDlBSaEyzrlPzCw5xKG/APcAbwSNjQKec8454Asza2lmHZxzW0O8Pj7kb674ePMOcONrkYkljlTULzi7V1tuG9qNYSltSs0mmnT9AGYs2cw907MAyOzaiqsHdmZ0RhL8IaLhi5ywihrNI4HH8VYyZ9TEvkdmNgrY7K+KDj7UCdgU9DzPHyuTFMxsHDAOoEuXLicaUuxq0an8HUgBWtbh7z3CnHMsyP2WKXPL9gsa1EvgivRO/HBoN3q3D723U73EBK7O7BxICtN/NCQSYYuERUWVwq+Bq5xz2TXxhcysCfArvEtHx805NxmYDJCZmVl3231xvoAmEor7BVPm5pAVol9w4+Cu3HBm17p99zORY1TUUxhWw1+rB9ANKK4SkoDFZjYIb1Fc8EXxJH8sfsX5Appwyj9wlGkLvubZEP2CXqc047ah3bk8vSON6tft+egioURsi0vn3HIgsImOmeUCmf7so5nAXWb2El6DOT+u+wkQ9wtowuF4+gUi8SZsScHMpgHDgTZmlgfc55ybUs7p7+BNR12HNyX1B+GKq1aJ4wU0NaW4X/DUpxt4f2XZfsGVA7x+Qa9TdC8IEQhjUnDOVXh9wzmXHPSxQ/spSQ2qqF/QplkDbjhT/QKRUHSHFKlT1C8QOTFKClIn5O7cz9Of5fDKorwy/YJzerXltmHdGNpT/QKRyigpSK2lfoFIzVNSkFqnuF/w1Kc5LN9ctl9w45nJjD2zi/oFIsdBSUFqjfwDR3lxvtcv2LandL+g9ynNuXVYNy5PU79A5EQoKUjMK+4XvLwwj4NH1S8QCSclBYlJzjnm5+zmqbk5fBCiXzA6oxM/PKsbKeoXiNQoJQWJKeoXiESXkoLEBPULRGKDkoJEVU7x+oIQ/YLhvdty61D1CySK7s+v/Jw6RklBIk79ApHYpaQgEXO0sIi3s7by1NwNrNi8p9SxNs0acNPgZMae0YXW6heIRI2SgoTddweO8OL8r3lu3sYy/YLU9s354VD1C0RihZKChE1l/YLbhnbnrJ6t1S8QiSFKClKjnHN8mbObpz7N4T+rSvcLGtZL4MqMJG4dmkzPduoXiMQiJQWpEUcK/PUF6heI1GrhvPPaVOBSYLtzrq8/9gAwCigCtgO3OOe2mHf94G94d1874I8vDldsUnOK+wXPzsvlmz2HSx1Lbd+cW4d24/L0jjSsp36BSG0QzkrhGWAS8FzQ2CPOud8CmNl/A78D/gu4CEjxH2cAT/p/SozK2bmfqXNzmL6obL/g3N5tuVX9ApFaKZy34/zEzJKPGQu+rtAUKL7iPAp4zr8t5xdm1tLMOjjntoYrPqk+9QtE6r6I9xTMbCJwE5APnOsPdwI2BZ2W54+VSQpmNg4YB9ClS5ewxiqeIwVFvL18C1Pm5oToFzTkpsFd1S8QqSMinhScc78Gfm1mE4C7gPuq+frJwGSAzMxMV8npcgK+O3CEf335Nc99rn6BSLyI5uyjfwHv4CWFzUDnoGNJ/phEwYYd+3j6s9xy+wW3DevOkB7qF4jURRFNCmaW4pxb6z8dBazyP54J3GVmL+E1mPPVT4gs5xxfbNjNlLnqF4jEs3BOSZ0GDAfamFkeXkVwsZn1xpuSuhFv5hF4FcPFwDq8Kak/CFdcUtqRgiLeyvL6BdlbyvYLbh7clbFnduXkpg2iFKGIRFI4Zx9dF2J4SjnnOmB8uGKRsor7Bc/Oy2X7XvULRMSjFc1xRv0CEamIkkIcKOkXbOA/q7aX6ReMPj2JH57VjZ7tmkUvSBGJCUoKdVhxv+CpT3P4amvpfkHb5g256Uz1C0SkNCWFOqiyfsFtw7pzWVoH9QtEpAwlhTpkw459TP3M24/o0NGiUsfOS23HrUO7qV8gIhVSUqjlnHN8vmEXU+fm8MHK7aWOqV8gItWlpFBLVdYvuHlwV64/Q/2CqLs/P9oRiFSLkkIt8+3+kvsXqF8gIjVNSaEW+c3ry8vtF9w2tBuD1S8QkROkpBADCgqLmLG4ZP+/0U/O45rMzlyZ0Yn5ubsZ4o+/8MXXgXMa1U9gdEYSP1C/QERqkJJClBUUFnHXi0uYlb0tMLZo47cs2vgtD7z9FXsPFZDbqOR89QtEJJyUFKJsxuLNpRJCsL2HCko9f+yqNC5Vv0BEwkhJIcr+vXBThcdbNK7PvLHrGdyjNaPVLxCRMFNSiLKt3x2s8HjTBokM6dkmQtGISLxLiHYA8a5Dy8YVHu9YyXERkZqkpBBl12R2rvD41QMrPi4iUpN0+SjKRp+exIertodsNo/s057RGUlRiEqOR+5Dl0Q7BJETFrZKwcymmtl2M1sRNPaIma0ysywze83MWgYdm2Bm68xstZmNCFdcsSYxwZh0/QAeHtM/MJbZtRUPj+nP/47NIDFBzWURiZxwXj56Bhh5zNj7QF/nXH9gDTABwMxOA64F+vivecLM4mbeZb3EBK4Ouow0/UdDuDqzsxKCiERc2JKCc+4TYPcxY+8554on338BFF8bGQW85Jw77JzLAdYBg8IVm4iIhBbNRvMPgXf9jzsBwRP28/yxMsxsnJktNLOFO3bsCHOIIiLxJSpJwcx+DRQA/6rua51zk51zmc65zLZt29Z8cCIicSzis4/M7BbgUuB85wK3kN8MBM+9TPLHREQkgiJaKZjZSOAe4HLn3IGgQzOBa82soZl1A1KA+ZGMTUREwlgpmNk0YDjQxszygPvwZhs1BN739/3/wjn3X865bDN7GfgK77LSeOdcYbhiExGR0MKWFJxz14UYnlLB+ROBieGKR0REKqdtLkREJEBJQUREApQUREQkQElBREQClBRERCRASUFERAKUFEREJEBJQUREApQUREQkQElBREQClBRERCRASUFERAKUFEREJEBJQUREApQUREQkQElBREQClBRERCQgbEnBzKaa2XYzWxE0dpWZZZtZkZllHnP+BDNbZ2arzWxEuOISEZHyhbNSeAYYeczYCuBK4JPgQTM7DbgW6OO/5gkzSwxjbCIiEkLYkoJz7hNg9zFjK51zq0OcPgp4yTl32DmXA6wDBoUrNhERCS1WegqdgE1Bz/P8sTLMbJyZLTSzhTt27IhIcCIi8SJWkkKVOecmO+cynXOZbdu2jXY4IiJ1Sqwkhc1A56DnSf6YiIhEUKwkhZnAtWbW0My6ASnA/CjHJCISd+qF6xOb2TRgONDGzPKA+/Aaz38H2gJvm9lS59wI51y2mb0MfAUUAOOdc4Xhik1EREILW1Jwzl1XzqHXyjl/IjAxXPGIiEjlYuXykYiIxAAlBRERCVBSEBGRACUFEREJUFIQEZEAJQUREQkI25RUqb7chy6JdggiEudUKYiISICSgoiIBCgpiIhIgJKCiIgEKCmIiEiAkoKIiAQoKYiISICSgoiIBJhzLtoxHDcz2wFsjHYcYdQG2BntIOS46edXe9X1n11X51zIm9zX6qRQ15nZQudcZrTjkOOjn1/tFc8/O10+EhGRACUFEREJUFKIbZOjHYCcEP38aq+4/dmppyAiIgGqFEREJEBJQUREApQUwsjM/mJmPw16PtvMngp6/piZ/czM3qqhr/d9MzutJj6XhGZm7c3sJTNbb2aLzOwdM+sV7bik+szs12aWbWZZZrbUzM6IdkyxQEkhvD4DhgCYWQLegpg+QceHAA1q8Ot9H1BSCBMzM+A1YI5zrodz7nRgAnBKFV6ruxzGEDMbDFwKZDjn+gMXAJuiG1VsUFIIr3nAYP/jPsAKYK+ZtTKzhsCpwGKgmZlNN7NVZvYv/5cPZna6mX3svyOdbWYd/PHbzWyBmS0zs1fNrImZDQEuBx7x3/X0iPh3W/edCxx1zv2jeMA5twyYa2aPmNkKM1tuZtcAmNlwM/vUzGYCX/nP54T6WUvEdQB2OucOAzjndjrntpjZ+Wa2xP85TjWzhmZ2npm9XvxCM/uemb0WrcDDTUkhjJxzW4ACM+uCVxV8DnyJlygygeXAEWAA8FO8d/ndgbPMrD7wd2CM/450KjDR/9QznHMDnXNpwErgVufcPGAmcLdzLt05tz5C32Y86QssCjF+JZAOpOG943ykOIEDGcBPnHPFl5jK/KzDGK+U7z2gs5mtMbMnzOwcM2sEPANc45zrh3cP+x8BHwGpZla8LcQP8P4/1klKCuE3Dy8hFCeFz4Oef+afM985l+ecKwKWAslAb7xfQu+b2VLgN0CSf35f/x3ocmAspS9JSeQNBaY55wqdc98AHwMD/WPznXM5QeeG+llLhDnn9gGnA+OAHcC/gTuAHOfcGv+0Z4GznTdv/3ngBjNrifem7t2IBx0hus4ZfsV9hX54l482AT8H9gBP++ccDjq/EO/nYkC2c24wZT0DfN85t8zMbgGGhyNwKSMbGFPN1+w/5nmon7VEgXOuEJgDzPHfYI2v4PSngTeBQ8ArzrmC8EcYHaoUwm8eXkNrt/9OcjfQEu/dxrwKXrcaaOs3xDCz+mZWXBE0B7b6l5jGBr1mr39MwuNDoKGZjSseMLP+wHfANWaW6F9iOBuYH50QpSrMrLeZpQQNpQPrgWQz6+mP3YhX9RVfCt6CV7E/TR2mpBB+y/FmHX1xzFi+c67crXmdc0fw3pX+2cyW4V1qGOIf/i1eb+IzYFXQy14C7vYbZWo01zD/MsIVwAX+lNRs4EHgRSALWIaXOO5xzm2LXqRSBc2AZ83sKzPLwuvx3IvXL3jFrxyKgH8EveZfwCbn3MqIRxtB2uZCRKQKzGwSsMQ5NyXasYSTkoKISCXMbBFef+h7xdNY6yolBRERCVBPQUREApQUREQkQElBREQClBREqkk7pUpdpqQgUg1V2SlVO6JKbaakIFI95e2UmnjMjqiNzOxpf7fNJWZ2LoCZ3WJmb/i7pa41s/ui9H2IhKR3NCLVU95OqeDtiNrXOZdjZj/HWwTdz8xSgfeCLjEN8j/PAWCBmb3tnFsY9shFqkCVgkjNCd4RdSjwAoBzbhWwEShOCu8753Y55w4CM/xzRWKCkoJI9WTjbbkcyrE7opbn2BWjWkEqMUNJQaR6ytspddgx532Kv4Otf9moC97OtwDfM7OTzawx3i1UP0MkRigpiFRDBTulHrsr6hNAgr/b5r+BW4L2zJkPvIq3s+qr6idILNHeRyIR5N8UKdM5d1e0YxEJRZWCiIgEqFIQEZEAVQoiIhKgpCAiIgFKCiIiEqCkICIiAUoKIiIS8P8BoY2RqR0MTAgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# two-way ANOVA with balanced design (equal n for all level combinations)\n",
    "# but works similar with unbalanced design (thus requires statsmodels)\n",
    "data2 = pg.read_dataset('anova2')\n",
    "pg.print_table(data2.head())\n",
    "sns.pointplot(\n",
    "    data=data2,\n",
    "    x='Crop',\n",
    "    y='Yield',\n",
    "    hue='Blend',\n",
    "    dodge=True,\n",
    "    errwidth=2,\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Source</th>\n",
       "      <th>SS</th>\n",
       "      <th>DF</th>\n",
       "      <th>MS</th>\n",
       "      <th>F</th>\n",
       "      <th>p-unc</th>\n",
       "      <th>np2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Crop</td>\n",
       "      <td>2736.583</td>\n",
       "      <td>2</td>\n",
       "      <td>1368.292</td>\n",
       "      <td>2.525</td>\n",
       "      <td>0.108</td>\n",
       "      <td>0.219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Blend</td>\n",
       "      <td>2.042</td>\n",
       "      <td>1</td>\n",
       "      <td>2.042</td>\n",
       "      <td>0.004</td>\n",
       "      <td>0.952</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Crop * Blend</td>\n",
       "      <td>2360.083</td>\n",
       "      <td>2</td>\n",
       "      <td>1180.042</td>\n",
       "      <td>2.178</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Residual</td>\n",
       "      <td>9753.250</td>\n",
       "      <td>18</td>\n",
       "      <td>541.847</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Source        SS  DF        MS      F  p-unc    np2\n",
       "0          Crop  2736.583   2  1368.292  2.525  0.108  0.219\n",
       "1         Blend     2.042   1     2.042  0.004  0.952  0.000\n",
       "2  Crop * Blend  2360.083   2  1180.042  2.178  0.142  0.195\n",
       "3      Residual  9753.250  18   541.847    NaN    NaN    NaN"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2.anova(\n",
    "    dv=\"Yield\",\n",
    "    between=[\"Crop\", \"Blend\"],\n",
    "    detailed=True,\n",
    ").round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "grand mean for the two-way ANOVA = 145.54166666666666\n"
     ]
    }
   ],
   "source": [
    "grand_mean2 = data2['Yield'].mean()\n",
    "print(f\"grand mean for the two-way ANOVA = {grand_mean2}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Crop</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Corn</th>\n",
       "      <td>147.750</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Soy</th>\n",
       "      <td>157.375</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wheat</th>\n",
       "      <td>131.500</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          mean  count\n",
       "Crop                 \n",
       "Corn   147.750    8.0\n",
       "Soy    157.375    8.0\n",
       "Wheat  131.500    8.0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2_crop = data2.groupby('Crop').describe().loc[:, ('Yield', ['mean', 'count'])].droplevel(level=0, axis=1)\n",
    "data2_crop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first calculate the sum of squares for the 1st factor 'Crop'. For each Crop, we calculate the **difference of its group mean to the grand mean and square it, and add it up for all the individual values**.\n",
    "For example, the Yield mean in the Corn, Soy and Wheat Corn groups are respectively 147.75, 157.375 and 131.5, and the Yield grand mean equals 145.5, and there are 8 values un each groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS for factor 1 (Crop) = 2736.583\n",
      "Degree of freedom for Crop factor: DF_crop = 2\n"
     ]
    }
   ],
   "source": [
    "SS_crop = 8 * (147.75 - grand_mean2)**2 + 8 * (157.375 - grand_mean2)**2 + 8 * (131.5 - grand_mean2)**2\n",
    "print(f\"SS for factor 1 (Crop) = {SS_crop:.3f}\") # corresponds to the SS on the 1st line of the ANOVA table\n",
    "\n",
    "DF_crop = data2['Crop'].nunique() - 1\n",
    "print(f\"Degree of freedom for Crop factor: DF_crop = {DF_crop}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>SS_crop</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Crop</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Corn</th>\n",
       "      <td>147.750</td>\n",
       "      <td>8.0</td>\n",
       "      <td>39.013889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Soy</th>\n",
       "      <td>157.375</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1120.222222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wheat</th>\n",
       "      <td>131.500</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1577.347222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>436.625</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2736.583333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          mean  count      SS_crop\n",
       "Crop                              \n",
       "Corn   147.750    8.0    39.013889\n",
       "Soy    157.375    8.0  1120.222222\n",
       "Wheat  131.500    8.0  1577.347222\n",
       "sum    436.625   24.0  2736.583333"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we can run that directly in the data2_crop dataframe\n",
    "data2_crop['SS_crop'] = data2_crop['count'] * (data2_crop['mean'] - grand_mean2)**2\n",
    "data2_crop.loc['sum'] = data2_crop.sum(axis=0)\n",
    "data2_crop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>SS_blend</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Blend</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Blend X</th>\n",
       "      <td>145.250000</td>\n",
       "      <td>12.0</td>\n",
       "      <td>1.020833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Blend Y</th>\n",
       "      <td>145.833333</td>\n",
       "      <td>12.0</td>\n",
       "      <td>1.020833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>291.083333</td>\n",
       "      <td>24.0</td>\n",
       "      <td>2.041667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               mean  count  SS_blend\n",
       "Blend                               \n",
       "Blend X  145.250000   12.0  1.020833\n",
       "Blend Y  145.833333   12.0  1.020833\n",
       "sum      291.083333   24.0  2.041667"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For the sum of squares for the 2nd factor 'Blend', we use a similar approach.\n",
    "data2_blend = data2.groupby('Blend').describe().loc[:, ('Yield', ['mean', 'count'])].droplevel(level=0, axis=1)\n",
    "data2_blend['SS_blend'] = data2_blend['count'] * (data2_blend['mean'] - grand_mean2)**2\n",
    "data2_blend.loc['sum'] = data2_blend.sum(axis=0)\n",
    "data2_blend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS for factor 2 (Blend) = 2.042\n",
      "Degree of freedom for Blend factor: DF_blend = 1\n"
     ]
    }
   ],
   "source": [
    "SS_blend = 12 * (145.25 - grand_mean2)**2 + 12 * (145.833333 - grand_mean2)**2\n",
    "print(f\"SS for factor 2 (Blend) = {SS_blend:.3f}\") # corresponds to the SS on the 2nd line of the ANOVA table\n",
    "\n",
    "DF_blend = data2['Blend'].nunique() - 1\n",
    "print(f\"Degree of freedom for Blend factor: DF_blend = {DF_blend}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The approach for residuals/error is similar to the SSW computed for one-way ANOVA, here using the 'Crop' and 'Blend' subgroups and calculating the sum of squares for each individual value. The degree of freedom for the within sum of squares equals **the number of factor 'Crop' times the number of factors 'Blend' times the number of replicates in each combination of factor minus 1**. Of note, _the degrees of freedom for the interaction 'Crop*Blend' is equal to the product of the degrees of freedom for the individual factors._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS for residuals/within = 9753.250\n",
      "Degree of freedom for within: DF_within = 18\n"
     ]
    }
   ],
   "source": [
    "SSW_2 = (\n",
    "    data2.groupby(['Crop', 'Blend'])['Yield']\n",
    "    .apply(lambda x: (x - x.mean())**2)\n",
    ").sum()\n",
    "\n",
    "print(f\"SS for residuals/within = {SSW_2:.3f}\") # corresponds to the SS on the last line of the ANOVA table\n",
    "\n",
    "replicates = data2.groupby(['Crop', 'Blend']).count().iloc[0,0]\n",
    "DF_W = (DF_crop + 1) * (DF_blend + 1) * (replicates - 1)\n",
    "print(f\"Degree of freedom for within: DF_within = {DF_W}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we calculate the total SS, as the squared difference of each value to the grand mean. It's exactly the same as for one-way ANOVA. _The degrees of freedom total is equal to the total number of values minus 1._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SST = 14851.958\n",
      "Degree of freedom Total DF_total = 23\n"
     ]
    }
   ],
   "source": [
    "SST_2 = (\n",
    "    (\n",
    "        data2['Yield'] -\n",
    "        data2['Yield'].mean() # or grand_mean2\n",
    "    )**2\n",
    ").sum()\n",
    "\n",
    "print(f\"SST = {SST_2:.3f}\") # corresponds to the sum of all the SS in the ANOVA table\n",
    "\n",
    "DF_T = len(data2) - 1\n",
    "print(f\"Degree of freedom Total DF_total = {DF_T}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then complete the ANOVA table and compute the F ratio for each factor and interaction as in one-way ANOVA, first by completing the values for the interaction or 'both factors' row, knowing that the SS sum up together as the degrees of freedom:\n",
    "\n",
    "| Source of variation | Sum squares | DF |  MS    | F ratio | P value |\n",
    "|:-------------------:|:----------- | -- | ------ | ------- | ------- |\n",
    "|    Factor 1 (Crop)  |   2736.58   |  2 | 1368.3 |  2.525  |  0.108  | \n",
    "| \\+ Factor 2 (Blend) |      2.04   |  1 |    2.0 |  0.004  |  0.952  |\n",
    "| \\+ both factors     |   2360.08   |  2 | 1180.0 |  2.178  |  0.142  |\n",
    "| \\+ Within (error)   |   9753.25   | 18 |  541.8 |         |         |\n",
    "| \\= Total            |  14851.96   | 23 |        |         |         |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SS for interaction SS_B = 2360.083 with 2 degrees of freedom\n"
     ]
    }
   ],
   "source": [
    "SSB_2 = SST_2 - (SS_blend + SS_crop + SSW_2)\n",
    "DF_B  = DF_T  - (DF_blend + DF_crop + DF_W )\n",
    "\n",
    "print(f\"SS for interaction SS_B = {SSB_2:.3f} with {DF_B} degrees of freedom\") # corresponds to the SS Crop*Blend in the ANOVA table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*For factor Within, MS = 541.85\n",
      "*For factor Crop, P value = 0.108 with F ratio = 2.525  and MS = 1368.29\n",
      "*For factor Blend, P value = 0.952 with F ratio = 0.004 and MS = 2.04\n",
      "*For factor Crop*Blend, P value = 0.142 with F ratio = 2.178 and MS = 1180.04\n"
     ]
    }
   ],
   "source": [
    "MS_crop = SS_crop / DF_crop\n",
    "MS_blend= SS_blend/ DF_blend\n",
    "MS_W2   = SSW_2   / DF_W\n",
    "MS_B2   = SSB_2   / DF_B\n",
    "\n",
    "F_crop  = MS_crop / MS_W2\n",
    "F_blend = MS_blend/ MS_W2\n",
    "F_B2    = MS_B2   / MS_W2\n",
    "\n",
    "Pvalue_crop = stats.f.sf(F_crop, DF_crop, DF_W)\n",
    "Pvalue_blend= stats.f.sf(F_blend,DF_blend,DF_W)\n",
    "Pvalue_B2   = stats.f.sf(F_B2,   DF_B    ,DF_W)\n",
    "\n",
    "print(f\"*For factor Within, MS = {MS_W2:.2f}\")\n",
    "\n",
    "print(f\"*For factor Crop, P value = {Pvalue_crop:.3f} with F ratio = {F_crop:.3f}  and MS = {MS_crop:.2f}\")\n",
    "print(f\"*For factor Blend, P value = {Pvalue_blend:.3f} with F ratio = {F_blend:.3f} and MS = {MS_blend:.2f}\")\n",
    "print(f\"*For factor Crop*Blend, P value = {Pvalue_B2:.3f} with F ratio = {F_B2:.3f} and MS = {MS_B2:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ANCOVA\n",
    "\n",
    "Analysis of covariance (ANCOVA) is a general linear model which blends ANOVA and regression. It's a model that is equivalent to multiple linear regression when **at least one independent variable is categorical and at least one is continuous**.\n",
    "\n",
    "ANCOVA evaluates whether the means of a dependent variable (dv) are equal across levels of a categorical independent variable (between) often called a treatment, while statistically controlling for the effects of other continuous variables that are not of primary interest, known as covariates or nuisance variables (covar)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Scores</th>\n",
       "      <th>Income</th>\n",
       "      <th>BMI</th>\n",
       "      <th>Method</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12</td>\n",
       "      <td>17.5</td>\n",
       "      <td>20</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>39</td>\n",
       "      <td>104.6</td>\n",
       "      <td>20</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>36</td>\n",
       "      <td>64.7</td>\n",
       "      <td>24</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>17</td>\n",
       "      <td>47.0</td>\n",
       "      <td>19</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>25</td>\n",
       "      <td>22.0</td>\n",
       "      <td>21</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Scores  Income  BMI Method\n",
       "0      12    17.5   20      A\n",
       "1      39   104.6   20      A\n",
       "2      36    64.7   24      A\n",
       "3      17    47.0   19      A\n",
       "4      25    22.0   21      A"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_ancova = pg.read_dataset('ancova')\n",
    "data_ancova.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAEGCAYAAAC0FJuBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAACbNElEQVR4nOyddXgU9/aH31mJu0OQkIQEd2nRIC1QikMpbalSd7/tvdV7297+6u7tpQLF3V1b3BIgCRY0LhvZ3azM74/JhgQiu8lKZN7n4dlkduQkbL5z5sjnCKIoIiMjIyMjIyMj43gUrjZARkZGRkZGRqa5IDteMjIyMjIyMjJOQna8ZGRkZGRkZGSchOx4ycjIyMjIyMg4CdnxkpGRkZGRkZFxEipXG2ANISEhYlRUlKvNkJGRcRIHDhzIFkUx1NV22AN5/ZKRaX7UtIY1CscrKiqK/fv3u9oMGRkZJyEIQpqrbbAX8volI9P8qGkNk1ONMjIyMjIyMjJOQna8ZGRkZGRkZGSchOx4ycjIyMjIyMg4iUZR41UVBoOBixcvotPpXG2KVXh4eNCqVSvUarWrTZGRkZGRkWlwNLb7OtTt3t5oHa+LFy/i6+tLVFQUgiC42pwaEUWRnJwcLl68SLt27VxtjoyMjIyMTIOjMd3Xoe73doemGgVBOCcIwjFBEA4LgrC/bFuQIAgbBEFILXsNrMu5dTodwcHBjeI/RxAEgoODG5UXLyPT3HHk+iUjI3M9jem+DnW/tzujxmuYKIo9RFHsU/b9P4BNoii2BzaVfV8nGst/DjQuW2VkZMpx2PolIyNzPY3tXlkXe11RXD8BmF329WxgogtskJFxPGe2wpWjrrZCxr40vvXLWOpqC2RkZCrgaMdLBNYLgnBAEISHyraFi6J4pezrdCC8qgMFQXhIEIT9giDsz8rKsupigiBw1113lX9vNBoJDQ3l1ltvrfG4w4cPs3r16vLv33zzTT788EOrrlkV9T1epglgNsOC+2DZY662RKbuOHX9cghmE6QfBYPWdTbIyNSDpnhfd7TjNUgUxV7AGOBxQRCGVHxTFEURaXG7DlEUvxdFsY8oin1CQ62bHOLt7U1iYiJarbTIbNiwgcjIyFqPu/Y/SEam3mQeB20upB+DK0dcbY1M3XDq+uUQirOhJMd115eRqSdN8b7uUMdLFMVLZa+ZwBKgH5AhCEILgLLXTHte85ZbbmHVqlUAzJ07lxkzZpS/V1xczP3330+/fv3o2bMny5Yto7S0lNdff5158+bRo0cP5s2bB8Dx48dJSEggOjqazz//vPwcH3/8MV26dKFLly58+umn5dvfeecd4uLiGDRoEMnJyfb8kWQaI+d2Sq8KFRz6w7W2yNQJV6xfdkUUIfe0FPWSkWnENLX7usMcL0EQvAVB8LV8DdwMJALLgXvKdrsHWGbP695+++38+eef6HQ6jh49Sv/+/cvfe+eddxg+fDh79+5ly5YtvPjiixgMBt5++22mT5/O4cOHmT59OgAnT55k3bp17N27l7feeguDwcCBAwf45Zdf2LNnD3///Tc//PADhw4d4sCBA/z555/lHva+ffvs+SPJNEbO7YCAttBxPBybD0a9qy2SsQFXrV92RZsHpUWglLUDZRo3Te2+7kgdr3BgSVnFvwqYI4riWkEQ9gHzBUF4AEgDbrPnRbt168a5c+eYO3cut9xyS6X31q9fz/Lly8vztDqdjvPnz1d5nrFjx+Lu7o67uzthYWFkZGSwc+dOJk2ahLe3NwCTJ09mx44dmM1mJk2ahJeXFwDjx4+3548k09gwmyFtF8SPhS6TIWkxJK+GzpNcbZmM9bhk/bIreWdB7QkGWcZGpnHT1O7rDnO8RFE8A3SvYnsOMMJR1wXpF/TCCy+wdetWcnKu1jeIosiiRYuIj4+vtP+ePXuuO4e7u3v510qlEqPR6DiDZZoWmcelaEPUIIhOAL9IKd0oO16NBleuX3ZBp4GSXPAOkR0vmSZBU7qvN8lZjffffz9vvPEGXbt2rbR91KhRfPHFF0g1sXDo0CEAfH19KSwsrPW8gwcPZunSpZSUlFBcXMySJUsYPHgwQ4YMYenSpWi1WgoLC1mxYoX9fyiZxoOlvitqICiU0OMOOL0JNJdda5dM86HggpxilGlSNKX7epN0vFq1asVTTz113fbXXnsNg8FAt27d6Ny5M6+99hoAw4YN4/jx45WK8KqiV69e3HvvvfTr14/+/fsza9YsevbsSa9evZg+fTrdu3dnzJgx9O3b12E/m0wjwFLfFdBG+r7HHSCa4chc19ol0zwoLYGCy+Du52pLZGTsRlO6rwsWL7Eh06dPH3H//v2Vtp04cYKOHTu6yKK60RhtlrERsxk+iJbquyZ+dXX7L7dA4RV48iA0MmVmVyAIwoEKavGNmqrWL4eSlQoF58ErSPq+OFtKe6s9nWeDjEwdaKz3yKrsrmkNa5IRLxkZl1GxvqsiPe+C3DNw/m/X2CXTPDCWQn4aePi72hIZGZlqkB0vGRl7UrG+qyKdJoCbDxz63fk2yTQfitKlV4XStXbIyMhUi+x4ycjYk2vruyy4eUtdjUlLQF/kGttkmjZmkxRV9ZBru2RkGjKy4yUjYy8s+l3tBlf9fs+7wFAMx5c61SyZZkJxFpgMcjejjEwDR3a8ZGTsRXl9VzWOV+v+EBwrjxCSsT+iCDmnwd3H1ZbIyMjUgux4ycjYi3M7pNe2A6t+XxCgx51wfrd0k5SRsRfaPCgtBpWHqy2RkZGpBdnxsgNLly5FEAROnjzpalNkXMm5nRAYBQGtq9+n+wwQFHBYjnrJ2JHcM+Dm5WorZGSaDI68r8uOlx2YO3cugwYNYu5cWSCz2WI2S47XtTIS1+LXAmJHwuG5UjG0jEx90RVI44HcvF1tiYxMk8GR93XZ8aonRUVF7Ny5k59++ok///zT1ebIuIrMJNDlV1/fVZGed0HhZTi9xeFmyTQD8s6Dyr32/WRkZKzC0fd1hw3JdjZvrUji+GWNXc/ZqaUfb4zrXOM+y5YtY/To0cTFxREcHMyBAwfo3bu3Xe2QaQRY9Luqq++qSNwY8AyCQ79B+5GOtUumaVNaIml3eQW72hIZGbvTVO/rcsSrnsydO5fbb78dgNtvv11ONzZXrKnvsqByg27TIXm1lCKSkakrBRdBUMpjqGRk7Iij7+tNJuJVmwfrCHJzc9m8eTPHjh1DEARMJhOCIPDBBx8gyAth88FS39XxVuuP6Xkn7PkGji2A/g87zjaZpotRL81k9AhwtSUyMg6hqd7X5YhXPVi4cCEzZ84kLS2Nc+fOceHCBdq1a8eOHTtcbZqMM7GlvstCRFdo0V0eISRTdwrTJf0ueTyQjIzdcMZ9XXa86sHcuXOZNGlSpW1TpkyR042NnaIs2PejdFOzBlvquyrS4y5IPwpXjtp2nIyMyShJSHgGuNoSGWswmyWttdyzcPmQ1FiTtBQO/A92fgp/fSXV68m4HGfc15tMqtEVbNlyfVfaU0895QJLZOxGSS78Ol5SoY8eBsExtR9jS31XRbpOhfX/lDS9WnSrk7kyzZTiLMn5UshLuNMwGaXItq4AtPmgyyt7zS97LajwdYVXXQHoNEAtD3KZJ2DClw78AWSswRn3dfmvVkbGgk4Dv0+WnC4As7H2Y+pS32XBKwg6jIWj8+Cmt2VJABnrMJulyQcevq62pPFh1FvvLF27rbSW4fYqD6nezsNfikT6toCwjtI2z4DK71Xc5hkA2z+AnZ9A7AjoPKm6K8g0EWTHS0YGpHErc26D9GOSuvyRudalGutS31WRnndB0hJIXgOdJ9btHDLNC22eNGzdPczVljgfUQSDtmpnyRpHyqit+fxq78oOUUAbiOhWtbPk4V95m7oe45qG/RPObIMVT0NkH9uj5zKNCtnxkpEx6uHPO+HCHpjyozTS58hcak0NQN3ruyxEDwO/SKnIXna8ZKwh9zS4NeJh2KII+sK6RZ10BWAqrfn87n5lDlGZYxTSvopoU+D1kSgPf0nqxRUo1TD1J/h2MCx+CO5dKTdNNGFkx0umeWMywIL74MwWmPAVdJkCx5dJ74nm2o+va32XBYVSirDt/Bg0l8GvZd3OI9M80OaDtgB8Qlxrh9kM+mocoyojURXfKwCxhnFZguKqI2Rxjvwia442VXSgGqvDEhQNYz+CJQ/Djo9g6EuutkjGQciOl0zzxWyCpY9C8ioY84GU9gOgTKultlRjeX3XuPrZ0eMO2PEhHPkTBj9Xv3PJNG3yz9svKmMyVHaGrisWz6/GkSoAfS3F4grVNY5RkORYVOcsVdzm5guKZtpw3206nNoIW/8L7YZCm/6utkjGAciOl0zzRBRh5TOSgOmIN6D/Q1ffs4jk1RbxykisX32XheAYKVV56HcY9KysQi5TNaXFknaXtw3RLtEsfa72/Qj6ItuKxZXulR2i8mLxaqJNFSNRbt7y57guCIIU9bqwBxbPgkd2Sr9TmSaF7HjVE6VSSdeuXRFFEaVSyZdffsmAAQNcbZZMTYgirH0FDv4Kg1+4PsokWJ62a4l4Weq7oupY31WRHnfCssekBbfNDfU/n0zTo+CiVAtki0OTcRyOL5XS4b4try8Wr67Lrr7F4jJ1x8MfpvwEP4+Glc+V1Z3KTqwzcfR9XXa86omnpyeHDx8GYN26dbzyyits27bNtUbJ1MyWd6RxPf0fheH/qmIHK1ON53ZCYDvwb1V/mzpNgNUvSoOzZcdL5lqMeii4IBWF20LaLmmW4/3rwDfCMbbJ2J/W/SDhFdjyH4gdCT1muNqiZoWj7+vNNJHuGDQaDYGBNi6MMs5l5yeSZk7PmTD6vaqfJK1JNZrN0k0tapB97HL3gS6TJDVrfS0pIJnmh+aK9CrYuGSn7YLwTnK6qjEy+DmpBGH1C5Jum4xLcMR9velEvNb8Q9JgsicRXWHMf2vcRavV0qNHD3Q6HVeuXGHz5s32tUHGfuz9ATa+KXUujvus+vC9NalGe9V3VaTHXVI9zvFl0hBtGRmQFNPzztruPOVfkIrx+9zvGLtkHItCCZO/h28GwKJZUtTSVXIXrqKJ3tfliFc9sYQkT548ydq1a7n77rsRrZ3xJ+M8Dv0hPTnG3wKTvqul5dyKVKM967sstLkBgmKkEUIyMhaKs6QpCraOB0rbJb1G9rW/TTLOwb8VjPscLh+Ere+62ppmg6Pv600n4lWLB+sMbrzxRrKzs8nKyiIsrBmqSjdUkpbA8icksdKpv0gFyjVhiXjV5njZq76r/LqCFOna9LaUWrBmTqRM08YyHsi9DuOB0nZJnyGfUPvbJeM8Ok+E03dLw7Sjh0H0UFdb5Dya6H1djnjZkZMnT2IymQgODna1KQ2e2Umz2Zi20fEXSlknhelb94fb/7CuU6s8A1mN42Xv+q6KdJ8hOX6H59j/3DKND20uGEtsn+OpzYeMJGjrgM9oI0JTqnG1CfZh9H8hOFYSVy3OcbU1zQpH3NebTsTLRVhywQCiKDJ79myUykaqnOxEfjv+G4IgMKz1MJSOUpo+sw3mzYTwLnDHPElbyCpqKa53RH2XBb+WEDNCcryGvdp4Vbhl6o8oStEuqz+3FTj/l/T5bdt8pW1MZhNnC87SJbiL49YYZ+HmLY0U+mEELH9SeoiUJSYchqPv67LjVU9MphpGX8hUS4mhhEJDITsv7WRoaweEzi/shbkzpFTLzCW2FSbXlmp0RH1XRXreBQvukcYYxY50zDVkGj66MoV4WwRTLaTtAu9QCImDkuYZIdEateiMOlebYT9adIeRb8L6f8L+n6HvA662qMni6Pu6nGqUcTqiKFJsLAZgfsp8+1/gyhH4fSr4hsPMpeAVZNvxtclJOKK+qyLxYyS9pkO/O+b8Mo2DvHO2pxhB0vy6uF+SImjGUZESQwmltQ3Ubmzc8BjEDId1r0LmCVdbI1NHZMdLxunoTXrMohlfN192XNzB5aLL9jt55kn4bRJ4+MHdyyXny2YsN6sqIl6OrO+yoHKXZradXAUluY67jkzDRV8ExZng5mP7sZcOgFHXrNOMAPml+Yi1TZ9obCgUMPFb6XOx8AEwNKGIXjNCdrxknE6xQYp2TY2biiAILExZaJ8T556BXydIbfd3L4OA1nU7T02pRkfWd1Wkx51gKoVjdvrdyDQuCi6CwsbxQBbSdoHaC1r2sLtZjYl8Xb6rTXAMvuEw8RvITIKNb7jaGpk6IDteMk6nxFACQIx/DIMjB7M4dTEGs6F+Jy24CLMnSM7KzKX1k2KoKdV4bof06qj6Lgstukkz9Q7L6cZmh0EnjQfy8LP9WNEMabulLl5lMxPbrECpqZRScykKW5X+GwtxN0P/R2DPt1LntkyjwuGfSkEQlIIgHBIEYWXZ9+0EQdgjCMIpQRDmCYLQfFeHZkqJUXK8vNXe3BZ/Gzm6HLac31L3ExZlSpEuXT7MXCyNSKkPNSnXO7q+qyI975Lq1eyt3CxjNS5ZvzRXpM9gXZyGzBOgzXP8g0EDR2vU1jrjvtEz8i2pY3vpY1CY7mprZGzAGY8DTwMVqwDfBz4RRTEWyAPk1oxmhiXV6KX2YmDLgbTwbsGClAV1O1lJLvw6ETSX4Y750LKnHSysJuJlNjm+vqsiXadJUYtDspK9C3Hu+mUylI0HqkO0C8qGYiukiFczpqi0CEHRxBsL1B4w5ScoLYKlj0r1pzKNAoc6XoIgtALGAj+WfS8AwwFL4cpsYKIjbXA06enp3H777cTExNC7d29uueUWUlJSXG1Wg6bc8VJ5oVQomRo3lb+v/E2aJs22E+k08PsUyEmF2+dA2xvtY6BQzcigjESpxb/dEPtcpza8gqQRR0fngbGJdWc1AlyyfhVlgmiyfTyQhXO7oEWPuindNyHydHl4KK0QS27shHWAUe/C6c3w99eutqZJ4ch7u6MjXp8CLwEWVzwYyBdF0Vj2/UUgsqoDBUF4SBCE/YIg7M/KynKwmXVDFEUmTZpEQkICp0+f5sCBA7z33ntkZGS42rQGTcVUI8Ck2EmoBJVtRfalJTD3dkg/Crf9CjHD7GdgdalGi35XWyemcXrOlKIgmcedd00ZC5/i7PWrKAPcvOpmbcFFyE9r9mlGs2hGU6rBrbnUuPW5HzrcChvfhMuHXW1Nk8DR93aHOV6CINwKZIqieKAux4ui+L0oin1EUewTGtowZ41t2bIFtVrNI488Ur6te/fuDB7s4I63Ro6luN7ieIV6hTKszTCWnlqK3qSv/QRGPcy7SyoinvSdpHtlV6qJeJ3bCUHR4F/lvdYxxAyDF5KbfYeas3Ht+lXHFFn5g0HzlpHQGXWYRXPTLay/FkGA8V9IQruLHoDSYldb1Ohx9L3dkcr1A4HxgiDcAngAfsBnQIAgCKqyp8ZWwCV7XOz9ve9zMvekPU5VToegDrzc7+Vq309MTKR37952vWZzoGKq0cK0uGlsSNvAhrQN3Bp9a/UHm4yw8H44vQnGfwldp9rfwKrkJCz1XZ0m2P96NaFQ1m1kjEx9cer6ZRfSdkNQDPi2cLUlLsUSUW9WeAXB5O9h9nhY+w/JEWsCuOK+Do6/tzvskUAUxVdEUWwlimIUcDuwWRTFO4EtgOVueQ+wzFE2yDRMrk01AvRv0Z82vm1YkFxDkb3ZLBWRnlwJo9+HXjMdY2C5fmqFYlVLfZej9buaKCazSLHeWPuODYRGt37p8qXPaDNPMwIU6AtQKZvhNLx2Q2DQs3DwV0ha6mprZGrAFZ/Ol4E/BUH4D3AI+MkuJ63Fg3UEnTt3ZuFCWeDSVooNxagVatRKdfk2haBgWtw0PjrwEafyThEbGFv5IFGEVc/Csfkw4nW44REcRxXK9a6o72oiHEjL443liXSM8OODad1dbU59ccj6VW/S5KHYFvJ1+c2jsL4qhr0KZ7bCiqcgsnfdRaQbCK64r4Pj7+1OSYKLorhVFMVby74+I4piP1EUY0VRnCaKohVFPQ2T4cOHo9fr+f7778u3HT16lB07drjQqoZPsaEYL/X1BcQTYiegVqivl5YQRVj3TzjwPxj0HAx+3rEGVpVqdEV9VyMns1DHc/MPM+Wb3WQXljI4rmHWatZGo1i/0nZLNT4h8U67ZIam4Y2rMZgNaI3aSg91zQqlGqb+JJVGLH5IepWxGUff25tJ9aFjEASBJUuWsHHjRmJiYujcuTOvvPIKERERrjatQaM1avFWXV+3FOgRyM1RN7Pi9IryAnwAtr4Hf38F/R6Wol2O5lrlemfrdzVyDCYzP+44w/APt7HiyGUeTYhh0/NDGd+9patNa5oY9XBxn9OHYmdqGobPWRGtUYtQ1+aEpkJQNIz9CM7vhh0fu9qaRomj7+3NMBFuX1q2bMn8+fNdbUajorqIF0hF9qvOrGLduXVMaj8Jdn0G296XVNxH/9dJN5ZrUo1yfZfV7EzN5s0VSZzKLGJYfCivj+tMuxC5OcChlA/FltPgxYbiOjeFNim6TYdTG6WH1uih0Lqfqy1qdDjy3i47XjJOpybHq1dYL2L8Y5ifPJ9Jedmw4XXoPBnGfQ4KJwVoy1ONZREvub6rVi7mlfCflSdYm5RO22AvfrqnDyM6hrvarOZB2m55KHYZBbqC5qPfVROCIEW9LuyRJCYe2Qke/q62SqYMOdUo43RKjCVVphpBCvFOi59GYk4ixze+AnFjpDZphdJ5Bl6rXC/Xd1WLzmDi040pjPhoG9tSsnjh5jjWPTNEdrqcRflQ7H7Neig2SKKXefo8PFTNtLD+Wjz8pZFCBZdg5XPX6xLKuAzZ8ZJxOiWGkkpSEtcyzqDEw2xmQWQcTPufVDDqTCoq15tN0hgWub6rEqIosi4pnZEfb+PTjamM7BTOpueH8sTw9nionegkN3cyT4A2V47GAnqTHpNoaj7CqdbQuh8kvAKJC+HIn662RqYM+RMq43RqSjWSsh6/pU8wBh9WqYwUia7QfqpQXJ9+DPRyfVdFTmUWcffPe3n4twN4uSmZ82B/vrqjFy0DPF1tWvMjbbc8FLsMrVHrahMaJoOfgzYDYPULkHPa1dbIIDteMi6gxFhSSbW+nLM7YP5MCO/EtJs/RWvUsurMKucbWDHVKNd3lVOkN/Lu6hOM/nQ7hy/k8/qtnVj11GAGxIS42rTmS9ouaNEdPPxcbYnL0ZRqUApytPU6FMqr5RqLZkmzX2Vciux4yVTPydUOaUcuNhRfn2q8sA/mTIfAKLhrCV1a3kDHoI7MS5mH6OzahIo6XnJ9F6IosuTQRYZ/uJXvt59hcq9ItryQwP2D2qFWykuIyyi4CHnn5IeCMvL0ebir3F1tRsMkoLXUoHT5IGx5x9XWNHvkVbOeKJVKevToQefOnenevTsfffQRZrO59gMbA1vfhT3f2vWUpaZSjGZj5VTjlaPwxxTwCYO7l4F3cHmRfWpeKkeyjtjVhtopi3iZjVIqpxnXdyVeKmDat3/x7LwjtPD3YMljA/i/qd0J8ZFvcC4nbbf0ao3jpWjaDewms4kifRFuiubdYFAjnSdCr7th56dwZpurrWnQOPq+3rT/Gp2Ap6cnhw8fBiAzM5M77rgDjUbDW2+95VrD6kv+eam+yd2+LcgWYdTyiFdWMvw2Cdx84Z7l4HtVoO6Wdrfw0f6PWJCygB5hPexqR41YUo3NuL4rr7iUD9cnM3fveQK93Hh/Slem9W6NQiGLJDUY0nZJ0Vi/GoZimwyg8nR+g4qT0Rq1iIgIThSQbZSM/q80XmrJw/Dobmm4tsx1OPq+Lke87EhYWBjff/89X375pfPTY/Ymea30aueC1WJjMYBU45V7Fn6dIKX27lkOAW0q7eut9ubW6FtZd24dBfoCu9pRI5bF++x26bUZRbxMZpHf/05j2Edb+XPfBe6+MYrNLyQwvW8b2elqSOjypQeD2qJdRh14BjjDIpdSYiiRFeutwc1bGilUnA3Ln5QlJqzAEff1JhPxSn/3XfQnTtr1nO4dOxDx6qs2HRMdHY3JZCIzM5Pw8EasZZRcVtRuKpUkFeyko1VsKHO8DDrJ6TLq4N5VEBxT5f7T4qYxL3key08vZ2anmVZfx2gyozea8Xavy0e8bAHPOAZBMeDXPEbd7D+XyxvLk0i6rOGG6CDeHN+ZDhFy0XaD5PzfUtdtVC2Ol8kAnoHOscmF5JfmN+r5jMV6I15uSudE7Fp0h5Fvwvp/wv6foe8Djr9mHWmq93U54iVzPdp8qajckg402C/qVZ5q3PoBlOTCXYshvHO1+8cHxdMttBvzk+fb9LTxyuJj9H1nI99vP43BZGNuvqIOUDOIdmVqdDw77zBTv/2L3OJSvpjRk7kP3iA7XQ2Zc7vBKwRC4mreTxTBrRrpliZEvi6/UQqnGkxm3lyeROc31jHq0+18u+006QVOGD5+w2MQMxzWvQqZ9nVsZGqnyUS8bPVgHcWZM2dQKpWEhYW52pS6c2qjVFjeaaIkvGfUgbuPXU5dUpwBgHdRJtw5HyJ71XrMbXG38a9d/2J/xn76RvStdf9TmYUsPHiRFn4evLv6JIsOXOKdSV3oE2VlPUPFp84mXN9VajTzv91n+WxjKgaTyOPDYnh8WCxebk1mWWiaGPVwcS+0v7nyQ0KViNI4oSZMqakUvVmPt6JxzQTNKdLz+JyD/H0ml0k9IzmfW8J/15zk/bUnGRgTwqSekYzuElHHqH0tKBQw8Vv4ZoA0UmjWJlA3PMe1qd7X5YiXHcnKyuKRRx7hiSeeaNxFnsmrwTsU2pU5HfaKeOkLKd70NgBeo96FtgOsOmxU1Ch83XxZkLzAqv0/3ZiKl1rJyqcG893M3hTqDEz99i9eWniE3OJSK85Q0fFqmq3621OyGP3Zdt5dfZL+0cGsf3YIL47qIDtdjYHLB6WHodo+m2ajVFTfxCUWtEYtgti41tvESwWM/3IXh87n88n07nwyvQeLHh3A1hcSeGp4e9Jyi3l+wRH6/Gcjz847zI7ULExmO9dj+YbDxG8gIxE2vmHfczchHHFfl1fZeqLVaunRowcGgwGVSsXMmTN57rnnXG1W3TGWQuoG6DTevqnG0hKYczsl+WkQGohX9DCrD/VQeTAhZgJ/Jv9JjjaHYM/gavc9ma5h1bErPJYQQ5C3G6M6RzAoNoTPN6Xy086zrD+ewStjOtTcoWeJIjTB+q4LuSX8e+Vx1h/PICrYi5/v7cPwDo24FrE5cm43qD2hZc+a9zOWgkeAU0xyJUWlRQiNqPFj2eFLvLzoKEFebix8ZABdW13tHI8K8ebZm+J4ZmR79qflsfjgJVYevcySQ5cI93NnYo9IJvdqRXyEr32MibsZ+j8iyQbFDIe4UfY5byPH0fd12fGqJyaTydUm2Je0naDXQPxYqXgX6t/ZaNRLivRpuyge/BBcXFPjrMaqmBY/jd9P/M7SU0t5oGv1xaCfbUzF203Fg4Ojy7d5u6t45ZaOTO7ViteWJvLyomPM33+R/0zsQscWVdQxWZ5qmlB9l85g4putp/l222kUgsCLo+KZNbgd7ipZ6btRIZolGYlWVgzFNurAv5Vz7HIhefo8PJQNL012LUaTmffXnuSHHWfp1y6Ir+/sVa0eniAI9I0Kom9UEG+M68Tmk5ksPniRn3ae5bvtZ+jUwo/JvSIZ36MlYb71/NlHviXV9C59TJKY8JUfxBx9X5dTjTKVSV4j6f5EJ1zN+RvqUexpMko1BKc2wrjPKAmTioGrHBlUA9H+0fQJ78OClAWYxaqL5Y9f1rAmMZ37B7UjwOv6m1J8hC/zHr6BD6Z242x2Mbd+sZP/rDxOkf6aeZCeQVLnT7fbbLKxISKKImsTrzDio218timVm8qGWT8+LFZ2uhojWSelodjWpMBFM7jZpzazoWIWzWj0Gtxqc0JdTF5xKff+so8fdpzlnhvb8ses/laLEHuoldzStQU/3tOXPa+O4M1xnVApBf6z6gQ3vreZe3/Zy/Ijl9EZ6ugsqD1gyk9QWgRLHwFnC4Brrjj3eg0A2fGSuYooSmOCYoZJnVCqsqHHZZ2INmM2w7LH4cQKGPUe9L6HEkMJSkGJu9L2upPb4m/jUtEl/rr8V5Xvf7oxBV8PFQ8MalftOQRBYFqf1mx6bii39WnFjzvPMvKjbaw5duVq16TaAx7e3ugjXqcyC5n5014e+f0gPu4q5j54A1/Kw6wbN+csQ7FvsG5/ddP+v9YZdZhFM4pamwxcx4krGsZ/tZO9Z3P5vyndeGtClzqP2gr2cefege1Y/sQgNj43hIeHRJOSXshTcw/R5z8beWnhEf46nYPZ1nqwsA4w6l04vRn+/rpOttWZonTnXq8B0KhTjaLYeJSKG4WgavpR0FyEhH9I31siXsY6RLxEEVY/D0f/hOH/ghsfAyQdLy+1V53+30a0GUGQRxDzk+czMLLyE3/ipQLWH8/g2ZFx+HvWrucT6O3Ge5O7MbV3a/61NJFH/zhIQnwob4/vQpvgxt0FVqgz8PmmVH7ZdQ4vNyVvjuvEXTe0RSXPVWz8pO2EiG61D8UWzSAom7zjpbWzwLO9WXX0Ci8sOIKfp4p5D99Azzb201SLDfPlpdEdeOHmeP4+m8OSg5dYdfQK8/dfJDLAk4k9WzKpZytiw6yMeva5X3K8Nr4pNVa16G43W22hMd3XoW739ka7Ent4eJCTk9MoHBpRFMnJycHDo4HXISSvAQSIGy19b2lDt7W4XhRhw2uSON+gZ2HwC+VvFRuKbU4zWnBTujExdiLbLm4jo0yWwsKnG1Pw91Rz36Aom87Zu20gK54YyL/GdmTf2Vxu+mQbn29KRW9sfLV7ZrPIogMXGf7RNn7ceZapvVux5YUE7h3YTna6mgKaS9JQbGvSjEY9ePhWlkZpghToC1ApG178wGQWeX/tSR6fc5BOLf1Y8eQguzpdFVEoBAbEhPDBtO7s/9dNfHZ7D2LDfPhm62lGfryNCV/tYvbuc7V3dAsCjP8CvENg4QNQWuwQe2uiMd3Xoe739ob3ibWSVq1acfHiRbKyslxtilV4eHjQqlUDL3Q9uQpa9wOfUOl7VR0jXtv+D3Z/Af0eghFvVFr8S4wlNhfWV2Rq3FR+TvyZxacW82j3RwE4ciGfjScyeXFUPH4etqtXq5QKZg2O5tZuLfn3yuN8vCGFpYcu8e+JXRgYG1JnW51J4qUCXl+WyMHz+XRvHcCPd/ehe+sAV5slY0/O2TAU26QH70asJWgl+fr8OpUtOJKCEgNP/XmIbSlZ3NG/DW+O64ybyjkPPp5uSib0iGRCj0gyNTqWH7nMooOXeGN5Ev9eeZyE+DCm9IpkeMewqms8vYJg0nfSVJG1/5AcMSfS2O7rULd7e6N1vNRqNe3aVV/LI2Mj+RekVOPICkNA1XWo8dr9BWx9F3rcCaPfv+6Ju8RQP8ertW9rBrYcyKKURTzY9UFUChWfbEwh0EvNPQOi6nxegAh/D766sxfTkjN5Y3kSd/64h/HdW/KvsR0J82uY0crc4lI+WJfMn/vOE+ztxv9N7cbUXq3kuYpNkbSdENjOOokTkxE87DvgvqFhMBsoMZQQ2IBGIqVkFPLQr/u5lK/lnUlduLN/W5fZEubnwazB0cwaHM2JKxqWHLrE0kOX2HgiAz8PFbd2b8mUXpH0ahNYObUXPRQGPQM7P4GYEdB5otNsbi739UbreMnYmZSyodgdxl7dVu54WRnx2vcTrP8XdJ4kPSkprn/Kq0+q0cK0uGk8s/UZdlzcgZ/Yg63JWbw8ugM+dlJ4TogPY90zwXy99TTfbj3NlpOZvDAqnrtuaIuygTg0JrPInD1pfLg+hSK9kfsGtOPpke2tqm+TaYToCqSh2D3usP6YBqhEbk8aWn3X2sR0np9/GE83qZHF6kkZTqBjCz86tvDj5dEd2HUqm8UHL7Lk4CXm7DlP22AvJvWMZFLPSNoGlz0UD/snnNkGK56CVn2ahSyJM5EdLxmJk6sgOBZC2l/dZulqtGaBOzIPVj0P7UfBpO+rHapdYiwh0KN+T6hDWg8hzDOM+SnzKUrzItjbjbtvtO+TpYdayXM3xTGpZySvL0vkjeVJLDwgaX+5OoW396w0zPrEFQ03Rgfz1oTOxIXbSVBRpmFyfo9UMN/Wik7b8u7cxt0kUhvFhuJKQyZchdks8ummVD7flEr31gF8d1dvIvwbptOrVAgMiQtlSFwoRXojaxPTWXLoIp9tSuXTjan0aRvIpF6R3Nq1Jf5TfoTvhsDih+CeFdWu6TK2I1fcykhP0+d2Qvwtlbcr1VLrem0RrxMrYOmjUifMbbNBVb2mTrGhuF6pRgC1Qs3kuMnsurSLnedSeHhotGPmmQHtQrz59f5+fDGjJxkaHRO/3sW/lh6jQGtwyPVqIkOj4+k/D3Hbd39RUFLKV3f0Ys6D/WWnqzmQthO8giG0lqHYAKZSSb+rid8oC3QFLq/v0ugMPPTbfj7flMq03q2Y99ANDdbpuhYfdxVTe7fij1k3sOvl4bw0Op4CrYF/Lkmk7zsbeWxtPok9XpMEe3d87GpzmxRyxEumbCi2oXKaEaT6LJVn7V2NG96AsE5w+9xa29dLDCX1TjUCTGk/hW8Pf4d/2EFm3mBD+qUOCILAuO4tGRofysfrU/j1r3OsTUznn2M7MrFHpMNbn0uNZn7edZYvNknDrJ8YFstjw2LkuYrNBaMeLuyF9jdZMRS7bH+fpq0+Looi+fp8vFwY1TudVcSDv+7nfE4Jb0/ozMwb2jYqGYSKtAzw5LGEWB4dGkPiJQ2LD11k+eHLrC5uxdeegxi15T3OePcitvfwRvszNiTklVtGEk31CoFWfa9/T+1Ze6qxOBt6zAD32vVi6tvVaOFchhpDUQe8AvejUpoBxz/d+3moeXN8Z6b2bsU/lyby7LwjzNt3gf9M7EJsmGOiTluTM3l7xXHOZBczsmMYr93a6Wodhkzz4PIhqbPYmjQjSA9RTbywXm/SYzAbULooqrfpRAbP/HkYN5WC32f154bo6ufHNiYEQaBrK3+6tvLn1Vs6siM1i1X7/Oh66n48lj/MhK2fMap3PBN7RhIpCzHXGTnV2NwxGaSh2HGjq05NqD1rTjWazdJsRysWeqPZiN6kr/dTqiiKfLwhBW/9IEpM+Wy+sLle57OVLpH+LH50AO9M6sLxyxrGfLaD/1t7Em2p/bS/zueUMGv2fu79ZR8i8Mt9ffnxnr6y09UcSdslSbu07GHd/iLS5IkmjNaodUnkxWwW+WJTKrN+3U/bEC+WPzmoyThd16JWKhjeIZyPZg4hcOavRCpzecn4PR+sO8nA/27m9u//Yv7+CxTqnF920diRHa/mTtou0BdAh1uqfl/lUbOchF4DiFY5XsUGSZCvvhGvv07nsPdsLo/3v4WW3i1ZkLygXuerC0qFwJ3927L5hQTGdW/J11tPc9Mn29h0IqP2g2tAW2ri4/XJjPxkG7tPZ/Py6A6sfWYww+KbviaTTBWIZkjbLenrqaytZxKbfGG9plSDUnButKtIb+SxPw7y0YYUJvaIZOEjA5pN1McndgCKhFcYpNvKgXE5PHdTHBkaPS8tPErfdzby1NxDbEnOxGhy8pzHRoqcamzunFwtOVfRw6p+X+1Rs4CqrkB6tcLxsrR/16fGyxLtivDz4I7+7TCdmMrnhz7nXME5ovyj6nzeuhLi487Ht/Xgtj7S6KEHZu/n5k7hvDG+s02LsiiKrElM551VJ7iUr2V895a8ekvHRlOoK+MgspKhJMeGNKNR+ntWNm1ZkXx9Pu5WO6L151x2MQ/9tp/TWcX8a2xHHhjUrvnVOg1+Dk5vJnjbqzz18HaeHD6UQxfyWXLwEiuOXmb5kcuE+LgzsUdLJvWKpFMLv+b3O7ISOeLVnBFFSF4tOV3VpSbUXjUX19vgeNkj4rUjNZv9aXk8PjwWD7WSSe0noRJULExZWOdz2oMbooNZ/dRgXhodz/bULEZ+tI3vtp3GYMUTYGpGIXf+uIfH/jiIr4eKeQ/dwOczespOl4wUkRYU0MbKodhGHXgGONQkV2MymyjUF+KmqL572p5sTc5k/Jc7ySzU8+v9/Zg1OLp5OhQKJUwukwpaNAvBbKRXm0D+PbELe18dyXcze9O7bQCz/zrH2M93MuazHXy37TQZmjrM+m3iyI5XcyYjEQouVJ9mBOnp2U4RL4vjVdcaL1EU+WRjCpEBntzWRxL0C/EMYXib4Sw9vRS9SV+n89oLN5WCxxJi2fDsUAbGBvPempOM/XwHe8/mVrm/Rmfg7RXHGf3ZDhIvFfD2hM6sfHIQ/ZtozYhMHTi3CyK61j4U24KxFOqpk9fQ0Zmk9cjRzo8oinyz9TT3/W8fkYFerHhiUKMZIeYwAlrDuM/h8kHY8m75ZjeVglGdI/huZh/2vjqSf0/sgqebkvfWnOTG9zYx86c9LDl0kZJSowuNbzhY5XgJghAjCIJ72dcJgiA8JQhCgEMtk3E8J1dTaSh2Vag9a67xssHxKjFK56lrqnFrShaHzufz+LDYSnPGbou/jQJ9AevPra/Tee1N6yAvfrynLz/c3YdivYnbvvuLFxYcIadIcgzNZpEF+y8w/MOt/LL7LLf1kYZZ331jlDzM2gE02vVLcxnyzlqfZgSpJqyJF9aX2DLCrK7XKDXyxNxDvL/2JGO7tmDRozfSOqhp/16tpvNE6HW3NFLo7Pbr3g70dmPmDW1Z8thAtryQwBPDYjmbXcyz847Q5z8beW7+YXadysZkbhyDsB2BtTVei4A+giDEAt8Dy4A5QA2hEpkGT/IqSULCp4bC7dq6Gp2UahRFkU82pNAq0JOpvSuPr+gX0Y+2fm1ZkLKAcTHjbD63o7ipUzgDY4P5fNMpftxxho0nMnhkaAyrk85w9HwpPdsE8PO9fenWKsDVpjZ1Guf6lbZLeo0aYNtxTiisL9AXIIquuXHm6/NRO7CG7UJuCQ/+up/kjEL+MaYDDw9ppqnFmhj9X0j7CxY/DI/ukoZrV0G7EG+euzmeZ0bGsT8tj8UHL7Lq2BUWH7xEhJ8HE3tGMrW1gVgrxo82Jax9vDaLomgEJgFfiKL4ItDCcWbJOJyCS3DlSM1pRpAEVO2UaswsyQTA1812zatNJzI5erGAp4a3x01V+WMrCALT4qZxKPMQqXmpNp/bkXi5qfjHmA6sfnowceG+fLhzHme9XuD1Ca1Y9MgA2elyDo1z/Tq3CwKjwC/Suv3NJlCobOh+rBsZxRkMnz+cLZdXOPQ61ZGvy8dD5Zj6x12nshn35U4u52v53339eGRojOx0VYWbN0z9CYqzYPmTV8dUVYNCIdCvXRD/ndKNff8cyZd39KRzSz9+2HGGsXOzuJTfsOZuOhprHS+DIAgzgHuAlWXbmnbbTFMnebX0Gj+25v3UtchJWBwv99prUFaeXkm0fzSRPlbeSMqw1Ha1DfZiUq+qjx0fMx43hRsLUpwvLWENceG+/Plgfzp0OAiCmQHx7igayMDtZkCjWr8y9fmg00D6UdvSjEa9VFjvYEdhT/oeSs2lrEybg1l0rnxAqakUnVmHSmHfhnxRFPlxxxlm/rSHMF93lj8xiKFxoXa9RpOjRXcY+SacXAkHfrH6MA+1klu7teSne/uy8slB6E2w+WSm4+xsgFjreN0H3Ai8I4riWUEQ2gG/1XSAIAgegiDsFQThiCAISYIgvFW2vZ0gCHsEQTglCMI8QRCc05oiU5nk1RAUU3kodlWorEg1uvvVOhcuJS+Fo9lHmdJ+is1PkOuPZ5B0WcOTw9ujrqYGKtAjkJujbmbF6RVOqQGpC8dyjpFWlAyAwSSLDjqRRrV+ZZUWwIW/pXqtqIHWH2jSO6Wwfu+VvQCkay+y89JOh1+vIlqjVhKItSM6g4ln5x3mP6tOcHOnCBY/NpCoEFmo2CpueAxihsPaVyHzpM2Hd4jwpY2fkm3JsuN1HaIoHgdeBg6WfX9WFMX3azlMDwwXRbE70AMYLQjCDcD7wCeiKMYCecADdbRdpq7oNHB2h5RmrM0JsowMqi6UrCuwKs24OHUxaoXa5hoss1mq7WoX4s3EHjUXAtwWfxtFhiLWnltr0zWcxZwTc8q/LjWXutCS5kWjXL/O7Sobih1v/TFmM7g7fmD6vvR9JLRKINA9hD9O/OHw61WkqLQIhTXzKq3kUr6Wqd/uZtmRy7xwcxxf39kLH3dZ3tJqFAqY+K2Uelz0QM0P6VUgCAIJUe7sPp2D3mi/yR8NHWu7GscBh4G1Zd/3EARheU3HiBJFZd+qy/6JwHDAIro0G5hos9Uy9cMyFLu2NCNIqUbRLI0WqgorHC+9Sc+K0ysY0WYEgTY+ka9NSudkeiFPj2hfa8dfj9AexAbEMj95vk3XcAbZ2mzWp62nU3AnQEqZyDiHxrZ+CSYDXNwLbQZYNxS7HLHWIfX15WLhRS4XX+bGljdyU+Qkdl/ezZn8Mw69ZkXy9Hl2E079+0wO47/YSVp2CT/e3YcnhreX0/91wTccJn4jyRNtfNPmwxPaelBSamLf2Tz729ZAsfav+k2gH5APIIriYSC6toMEQVAKgnAYyAQ2AKeB/LJCV4CLQJVFO4IgPCQIwn5BEPZnZWVZaaaMVSSvlp6mW/erfV9V2UJeXfrOCsdrY9pGNKUapsRNsclMs1nk040pxIR6M6577W0vliL7pJwkkrKTbLqWo1mQvACj2cjdne4GwGCWU41O5E0a0frlnZUiiRbbkmYURclJc1DRuYV96fsAqZN4eKRUV+msqJcoimj0GtyV9XO8RFFk9u5z3PnjHvy91Cx9YiAjOobbycpmStzN0P8R2PMNpNgm63NjKzfcVAq2NqN0o9XF9aIoFlyzrdaqSlEUTaIo9gBaIS18Haw1TBTF70VR7COKYp/QULnI0W6YDJC6vvqh2NdieYKurrPRCsdrUeoiIn0i6RdhhaNXgVXHrpCSUcTTI+NQWvkkOi5mHJ4qzwZVZG8wGZifMp9BkYOIDYgF5IiXk2lU65fv5SNlQ7F7Wn+QSS+lGRWO1YHbl76PII8gYgJi8HMLZGz0WFacWUGB/tpfr/3RGrWYRXO9Uo06g4kXFx7ljeVJDIsPZenjA4kJ9bGjlc2YkW9BeBdY+igUWj+z1lOtoH+7ILamNJ8Ai7Wf4CRBEO4AlIIgtBcE4Qtgt7UXEUUxH9iCVOAaIAiCJYneCrhkg70y9SVtt+QsxVspYWRxvKobG1SL45WmSWNf+j6mtJ9i04JpKot2xYX7MLar9Z3/vm6+jGk3htVnV1NYWmj1cY5kQ9oGsrXZ3NHhjnL9IdnxciqNZ/0SzZLj1cqWodiUjQqqWkvJXoiiyN70vfQJ71PeIHNnxzvRGrUsSl3k0GvD1VmvdeVKgZbp3/3FwgMXeXpEe76f2Qc/jwbb3Nr4UHvAlJ+gtAiWPiLVHFpJQnwYpzKLuJjXMBuj7I21d8Ingc5IBadzgALgmZoOEAQh1KIOLQiCJ3ATcAJpAZtatts9SGKGMs4iuWwodkw1Q7GvxZK6qGPEa3HqYpSCkgmxE2wyc8WRy5zOKuYZG6JdFm6Luw2tUcuqM6tsOs5RzDk5h7Z+bRkYObB8vpxcXO9UGs/6lZ2CWldgW5oRwGS0fqxQHblQeIGMkoxKkev4oHj6RvRl7sm5GM2OHQejKdWgUtat8H3/uVzGfbGLU5lFfDezN8/eFCfXczmCsA4w6l04vVlKO1qJRbpja3LziHrV6ngJgqAEVomi+E9RFPuW/fuXKIq1tS+0ALYIgnAU2AdsEEVxJVJ30XOCIJwCgoGf6vkzyFhL+VDsBKkLxRrUNdR4mc2g11TreBnMBpadWsbgVoMJ86pBHf8ajCYzn29KpUOEL6M7R1h9nIXOIZ3pFNyJ+SnzXaaubSEpJ4kjWUe4Pf52FIICN2WZ4yVHvJxCo1u/0nYjIlg/FLscweGF9XvTJRmJvi36Vtp+V8e7SC9OZ/P5zQ69fr4uv071XXP2nGfGD3/j465k6eMDGVWHNUXGBvrcDx1uhQ1vSCLdVhAT6k2rQE/Z8bIgiqIJMAuCULtmQOXjjoqi2FMUxW6iKHYRRfHtsu1nRFHsJ4pirCiK00RRdO1k4+ZERhLkn7c+zQgVHK8q7lN6DSBW63htv7CdHF0OU9rbVlS/7PBlzmRL0a66PpVOi5tGal4qR7Ks+8N3FHNPzMVT5Vke8bNEvOTieufQ6NavtF2UhMRaJdFyjcUOHxW0N30vIZ4htPNrV2n70FZDifSJdGiRvdFspNhYXP7gYg0Go8hPm7W8tuw4A2NDWPbEINqHO15uo9kjCDD+C/AOgYUPQGmxFYcIJMSHsvt0drOQlbA21VgEHBME4SdBED63/HOkYTIOINmKodjXYulqrKq+opZxQQtTFxLmGcagSOvVtw0mM59vTqVzSz9Gda57p9Et7W7BW+3tUmmJXF0ua86uYXzM+PIxSXLEyyU0jvWrKAvyz1PYsrttx5lKpQi2Nc0ydUQURfal76NvRN/rBJCVCiV3dLiDg5kHScpxTDexrfVduUVGXpmXzpakUh4dGs1P9/TF31Ou53IaXkEw6TvIOQVrX7HqkIS4MEpKTew/1/RlJax1vBYDrwHbgQMV/sk0Jk6uglZ9JN0Va1GX1XhVVVxfg+OVXpzOrku7mNh+ok3jPZYcvERaTgnPjoyr14w0L7UXt0bfyrpz65zScVUVi1MXU2ouZUaHGeXb5OJ6l9A41i+fUJi5hLx2NtZ3GfXg6VjF+nOac2Rrs+kb0bfK9ye1n4SXyos/jjsm6mXLNIqTl3U889sVzmSV8uRoL14YZXudqIwdiB4Kg56Bg7MhaWmtuw+IDcZN2TxkJaxVrp8NzOXqgjWnbJtMY6HgElw5bFuaEa6mL6pKNdbgeC1JXYKIyKTYSVZfyhLt6tbKnxEdra8Jq45pcdMoNZey7JTz+zeMZiPzkufRv0V/YgJiyrerBBUCglxc70Qa1frl5oPZ1lotUyl4BDjEHAsV9buqwtfNlwmxE1hzbg3Z2my7Xz9Pl2dVfdf6Y4W8/Gc6KqXAR3e0oH97OcrlUob9E1r2ghVPQcHFGnf1clPRr11Qs6jzsla5PgFIBb4CvgZSBEEY4jizZOxOyhrp1VbHq7yr0fqIl8lsYsmpJdzY4kZa+bay+lILD1zkYp623tEuC/FB8fQI7cGClAVOL7LfcmEL6cXp3NHhjkrbBUHATekmz2p0Ik1+/RIdr1i/N30vYV5htPFtU+0+d3S4A6PZaPf0viiK5OtrLqw3mkS+2ZjDZ2tz6NrKg09ntqBdmDwG2OUo1TDlRzCbYPFD0msNJMSHkppZxKX8+kmHNHSsTTV+BNwsiuJQURSHAKOATxxnlozdObkagqJtm/0GNRfXV+N4/XXlL64UX2Fy3GSrL6M3mvhy8yl6tA4gId5+grnT4qdxTnOu/IndWcw5MYdIn0iGthp63XtuCjc54uVcGs36VWAopsRk27w7wKGF9Zb6rn4R/Wp8IIryj2JIqyHMS55n11S63qTHYDagrKaGLb/YxKvz01l5qJDJff14a2o4fp6Oq3eTsZHgGLjlQ0jbBTs+rnFXy9rf1NON1jpealEUky3fiKKYgjS7TKYxoNPA2e1StMvWSJIl4lVVjUU1jteilEUEugcyvPVwqy8zf/9FLuVree4m+0S7LNzc9mb83PyYn+K8Ivvk3GT2Z+xnevz0Km8WaqVarvFyLo1i/crWZpOw/Um2ZNvQiWs2Sn+jKsdFd07nnyZXl2vV5Ik7O95Jri7XroPqayqsT03X8/Rvl0lNL+XFsSE8kBAk13M1RLrfDl2nwdb34MLeaneLCfUhMqDpy0pY63jtFwThR0EQEsr+/QDsd6RhMnbk9Kayodg2phmh5pFBFsfL/apwY7Y2m60XtjIuZpzVrd86g4mvt5yiT9tABrcPsd3GGvBQeTAhdgKb0jY5pPakKv5M/hMPpQeT21cd8XNTusmOl3NpFOtXiGcIMd6R7M1Prn1nC0ZdHaQnbKNcv6uawvqK3NjiRmL8Y/j9+O92S+9rSjVVPsBsSizixTlXEAT44I4IEjrJo38aLIIAYz8C/0hY9MDVe8d1u5XJSpzKptRovfJ9Y8Nax+tR4DjwVNm/42XbZBoDJ1dL40Ra97f9WIUSlG7VdzW6+1VqY19+ejlG0WiTdte8fRe4UqDjWTtHuyxMi5uGUTSy9NRSu5/7Wgr0Baw6s4qx0WPxd6/6hiinGp1Oo1m/hof2IqXoIjmlGusOcEJH4770fbT0bmlVvaYgCNzR8Q5O5J7gYOZBu1w/X5+PR4Xh3yazyPebc/l4TTYdWnrw2cyWxIbXb3C2jBPw8JdGChVcglXPS7WJVZAQH0ZxqYn953KdbKDzsNbxUgGfiaI4WRTFycDngJxEbwyYDJC6TtLuquO4DVSe1TteFZ62RVFkcepieoX1Ijog2qpT6wwmvtpyin7tghgQE1w3+2qhnX87+kX0Y2HKQsyiY5+ilp5aitaorSQhcS1ycb3TaTTr14iw3ojAVmudFhHrp1DUAbNoZn/GfvpE9LH6mHEx4/Bz87OLoKrJbKJQX1guPFxQYuK1BRksO6BhQm9f/jMtHH+vBvlfKVMVrftBwj/g2AI4Oq/KXQbElMlKNOGh2dY6XpuAim0znsBG+5sjY3fO/1U2FHtM3c+h9qy+q7FCG/v+jP2kadKqTbFVxZw958ks1Nu9tutapsVP41LRJXZftno2ss2YzCbmnpxLr7BexAdV38SgVqjliJdzaTTrV5xPa0Ld/NmcZa3MmGMV61PzUsnX51tV32XBU+XJlLgpbDq/ictFl+t1fV1Zo4EgCJzO0PPMb5c5fknHs2NCeGh4MCqlXM/V6Bj8PLQZIEW9cs9c97a3u4q+7QKbdIG9tY6XhyiKRZZvyr527HwKGftwcjUo3SHG+kL361B7VN/VWCHitTh1MT5qH26Outmq02pLTXy99TQDYoK5Idox0S4LI1qPIMgjyKFK9jsu7eBS0SXu6HhHjfvJNV5Op9GsX4Ig0Dcgnr9ykiiuTa3dbAJBBSrHpdlq0++qjhnxMxAQ+PPkn/W6vtaoRRAEtp0o4sU56ZjM8H8zWjCyi1zP1WhRKGHy99LrollSg8g1JMSFkZJRxOUmKithreNVLAhCL8s3giD0AZrmb6QpUXEotns9FipVdRGv/HLHq0BfwIa0DYyNHounyjpNod//TiO7SM+zN8XVuJ+poABTUe3zvmpCrVQzKXYS2y5uI704vV7nqo45J+YQ5hXG8DY1O7my4+V0GtX61TcgDoNoZGfOsZp3NOnB09/2TmUb2Ju+l1Y+rWjh08Km41r4tGBEmxEsTF1ok+r8teSU5PHnbj3/tzKb2HA3Pru7JXEtmlY9lyiKGLOd0/jTYAhoDeM+h0sHYP8v1719VVaiaaYbrXW8ngEWCIKwQxCEHcCfwBMOs0rGPmQeh/w06FCHbsaKqD1qrfFaeWYlepPe6qL6klIj3247zeD2IfSNCqp2P93Jk5weNZrUAQO48PgTFKxYWWcnbErclPI6NHtzpuAMf135i+nx01EralYqcFO4yUOyncszNKL1K96nFYFqXzZl1tJ4adRLTTMOwmQ2sT9jP/1a2BbtsnBXp7soLC1k5ZmVdTo+v6SUl+adY8UBLWN7+PLO9AgCvZtOPZdZryd/8RLOTppM6qDBaI/YICPSFOg8EXrdDYfnQHrlh4zYMIusRNNMN9boeAmC0FcQhAhRFPcBHYB5gAFYC5x1gn0y9eHkaunVlqHYVaH2qjHVKIoii1IX0TGoIx2DO1p1yl//SiOnuJRnRlYf7dIdP875e+5F8PAg4Lbb0B07xuUXX5ScsCeeoGDlKpucsNa+rRkQOYBFqYswVhHerg9zT8xFrVBb5XjKOl7OobGuXwpBQUJoT3ZkH8FQ0+fUZKpfJLsWkvOSKSwttEpGoip6hPagU3Anfj/xu81NLSfTNYz7cieJF0t5alQwj90UjLqJ1HMZs7PJ+uJLTg0fwZVXX5X+H1UqCjc2yLJDxzL6v5DwCoR3qbRZEASGxoey+3ROk5SVqC3i9R1guUPcCLyKNHYjD/jegXbJ2IPk1RDZB3wj6ncelcf1qUazCfQa8PAnMTuR1LxUpsZNtep0RXoj3207TUJ8KL3bVt0Kr01KIu2++xG8vWj7269E/OufxG7dQts/fidg+nR0R45y+YUXSB04kItPPknBqlWYi2t3wm6Lu43Mkky2X9xula1W/TylRSw/vZwx7cYQ7Fl7rZosJ+E0Gu36NSKsN0VGLXtzj1e/k4BDRwVZ6rv6hld2vPKKS5nw5U6enX+Yh37dzxvLEvl222mWHb7EnjM5nM8pQW80IQgCd3W8i7MFZ/nr8l9WX3f1sStM/no3OoOJf032ZlQ3X7v+XK5Cd+IEl195lVPDhpP91Vd4du1Km19+pt3yZXj17EnR9h2uNtH5uHlD3M1VpssT4kIp0hs5kJbnAsMcS236AkpRFC1iGtOB70VRXAQsEgThsEMtk6kfmitw+SCMeL3+51J7QuGVytv0ZTpDHv4sSl2Ep8qTW9pZl9KcvfsceSWGaqNd2mOJnH/gAZQ+PrT5dTZurST9IEGhwKt3b7x69yb8lX+gPXgQzdp1FK5bR+GGjQju7vgMGYLv6FH4JiSg8L6+zX5IqyGEeYUxP2V+rbVY1rLs9DJKjCXXzWWsDrnGy2k02vXrhqDOeCrd2Zx1kIEh3a7fQRRBUEj1lw5iX/o+2vq1Jdw7vNL2r7ac4tilArq1CuBcTjF/ncmhUHd9ZC7Y241wf2/U/n68vvUbJrcMJ8LPgxYBHrTw9yTCzwNPt6upQ5NZ5OMNyXy15TQ92wTwxsRIisVLDvv5nIFoMlG0dSu5s3+lZO9eBE9PAqZNI3DmXbi3a1e+n/eQwWR99DGGjAzU4eE1nLH5MCA2BLVSYGtKJjc6SGrIVdTqeAmCoBJF0QiMAB6y4dgmR15xKX/uu8CU3pGE+XrUfoArSS5LM9ZFrf5a1FXoeJUpD5eoPVmTvIab296Mj1vtaQ+NzsD3288wokMYPVoHXPe+9uhRzj8wC6WfH21mz8atVWSV5xEUCrz69MGrTx/CX33lGidsQ7kTFnTfvXj1Kq+rRqVQMaX9FL498i0XCi/Q2re19b+HKjCLZv48+SfdQrvROaSzVcfIOl5Oo9GuX+5KNwYFd2NL5kH+2eFuFMI1yQmTHtx8QGFtma5tGM1GDmQcYHS7ymUKF/NK+PWvNKb0asXdN0bRtZVU41mkN5JeoCO9QMeVAq30qpG+L9AOIdNzJR9u2YlYWnkOa4CXWnLG/D3Q6KToxvQ+rXl7YmdO5Z/E3dQ4C+lNRcUULF5M7u+/Yzh/HlXLFoS9+CIBU6eg9L9eWNlnyBCyPvqY4h07CJhqXeagqePjrqJvVBDbkrN4ZYx1JSyNhdoWn7nANkEQspG6gHYACIIQC1St+d+EWXnsCu+vPcnXW07x9Mj23DMgCrXSMQtfvUleA4HtILRD/c+l8rh+ZFCZ47Wm+BwlxhKr04z/23WOAq2hyk5G7ZEjktMVGEjb2f9D3bKlVees5IRViIRp1q/DL3PsdftPbj+Z745+x6KURTzT+xmrrlEdf13+i3Oac7w3+D2rj5F1vJxGo16/RoT1ZkPmPo4VnKF7QGzlN4168LXu76MunMw9SZGh6DoZiU82pIIAz94UR07R1c+wj7uK2DAfYsOuf/jK1rbj5oXruG/0RWbGTbvqmJU7ajrSNVoKtAb+PbELd/VvA0id0r7u9U8zGjMzUUbY1pVZV0ovXiTv9z/IX7gQc1ERnj17Evbcs/iOHImgqv526x4Xhyo8nKIdO2XHqwIJ8aG8u/okVwq0tPB3XHTX2dToeImi+I4gCJuAFsB68erwLQXwpKONa2hotFKUomfbQP6z6gR/7rvAG+M6Mbh9aC1HOhl9IZzdBv0esk+reQ0Rr8U5B4n2j6Z7aPdaT1OgNfDDjjPc1CmcLpGVn/pKDh3iwqwHUQYHS05Xi7otlIJSiVffvnj17Uv4q69UOZYiwjuCoa2GsuTUEh7v8ThqZd3nJc85OYdgj2BGtR1l9TFyqtE5NPb1a3BId1SCkk1ZB653vEwG8Axw2LWrms+YnF7I4kMXeXBwNC0DPCs5XjUR4hnCmHZjWHl2Oc/0eYp2IbXPYy0xlGAWzddH+uqAMSsLdwc6XqIooj1wgNzZv1K4aRMoFPiNGkXQPXfj2a2KNHEVCIKA9+BBFK5bj2g01uikNScS4sN4d/VJtiVncXu/Nq42x27U+qkWRfFvURSXiKJYXGFbiiiK9hnE1Ygo0BrwUCuYfV9ffrqnDwaTmZk/7eXh3/ZzIbfuWjV259QmMJXWT62+Iqoq5CR0BaSo1RwtPMeU9lOsUp3/aedZCnVGnhnZvtL2koMHufDALFQhIbT9dXadna5rEZTKahew2+JvI1eXy6YLm+p8/guaC+y4uINp8dNsct7k4nrn0ZjXLz+1N/2COrI5c3/VA6cdWFi/N30v0f7RhHhedZI+WHcSH3cVjyXE2Hy+uzrehdaoZUnqEqv219YmHtsAEEtLKVi+nHNTp5F210xK9u4leNYsYjduIPKjD612uiz4DB6CubAQ7eHDjjG4EdI+zIcW/h5NTs+rgebJGiYarQE/DzWCIDCiYzjrnhnCi6Pi2Z6SzciPt/HxhhS0pSZXmymlGT0DofUN9jmf2kuqKTFXaOvVFbDY1we1QsW4mHG1niK/pJSfd55lTJcIOre8Gu0q2b+f87MeRBUWRptfZ6OOqGcHppUMaDmASJ9IFiQvqPM55ibPRSkomRY3zabj3JRuGM1Gh8+NlGn8DA/tTVpJBmeKqxi946DCeoPZwMGMg5WiXfvO5bLxRCaPDI0hwMvN5nN2DO5Ir7BezD05F5O59jVSU6pBqWyYml3GvDyyv/mG1BEjuPzSy5i1WiLefJPYrVsIe+7ZOq9h3gNuBJWqeXY3VoMgCCTEh7LrVDYGU9NZL2XHywY0OgP+nlcjGx5qJY8Pi2XzC0MZ1TmCzzelMvLjbaw+duW6J9Qfj/3Ik5udkN0wGes/FPta1GWNBBXqvPQlOazw8WJE5BACPaqWhKjIjzvOUlxqrNTJWLJvH+cfehh1eLjkdDmxm0chKJgaN5W96Xs5U3D9vLDaKDGUsDR1KTe1vYkwrzCbjnVTSjcuWURVpjaGhUqNIZVmN5pKpTZ8e/19X8PxnONojdpyx0sURf675iRhvu7cP7BdLUdXz12d7uJS0SW2Xtha6775unw8lA2rgUmXksKV117jVMIwsj77HI8OHWn9ww9Er1xB4O3TUXjWzxFW+vri1aMHRTtkx6siQ+PCKGxishKy42UDBVoDfp7Xp5Ra+Hvy+YyezHvoBnw9VDz2x0Hu/HEPKRmF5fusP7eerRe2kqvLve54u3L+L9Dm2S/NCFefrCs4XhvzktAolUyJv73Ww3OLS/ll11lu6dqC+AipWLZ4z17J6WrRQkovhtnmvNiDibETUQkqFqYstPnYlWdWUmgorHUuY1VYlO3lOi+Zqgh1uxoRDvMIpJt/DJsyKzheRn2l4fT2ply/q8zx2ngikwNpeTwzMq6S/IOtDGs9jJbeLfn9xO817mc0Gyk2Fpc/oLgS0WymaNs2zt//AGfHT6BgxUr8J04keuUK2vzwPT6DByHYsbPUe8gQ9CdOYMhsmortdWFgbDAqhdCk0o2y42UDGq0RP4/qnzL7Rwez8slB/HtCZ5Iuaxjz2Q7eWpFEZlERqXmpAOxPr2UMSH1JXlM2FHuE/c5pqSWpMHNtkSaZSKOZfi3713r499vPUGIw8cwIqbar+O+/ufDww6gjW9J29v9QhbqmOSHEM4QRbUew7NQydNd2bdaAKIrMPTmXjkEdrWoquBbLDUV2vGSqIsw9oNL3w0N7k6Q5S7ouR9pgLHVsYf2VvcQGxBLkEYTJLPJ/a08SHeLNbX1a1eu8KoWKGR1msD9jPydzT1a7X0Oo7zKXlJA7Zw5nbhnLhYcfQX/qFKHPPkvsls20eOtN3GNjaz9JHfAZMhiA4h07HXL+xoivh5o+UYFNanyQ7HjZQIG2cqqxKlRKBTNvjGLLCwlM79ua/+0+x6iv5mIUJYFBy9OkQxBFSF4F0UPtO0qk3PGSnJM0TRr7DLlMKVXU2nWUXaRn9u5zjO/ekvbhvhTv3s2Fhx/BrXVr2s6ejcqKDidHclvcbWhKNWxI22D1MfvS93Eq/xR3dLzDqqaCa7FEvORUo4w1DA/rDcDmzAr9AGovh1zLYDJwOOtwuYzEooMXSc0s4sVR8ajsIJ0zqf0kPFWe/H68+qhXfYZqWzCLZjalbeLtv95mR771fRSGK1fI/PBDUhOGkfH2v1H4+tLyww+J3bSRkIcfQhVYe1lFfXCPj0cVFianG68hIT6Mk+mFpBdY/4DckJEdLxvQ6KpONVZFkLcb707qyoonBhEUlAGA2tSSHRf+dpyBmScg75x904wgdTVC+digxamLUQIThNo1dr7ffga90cRTI9pTtHMXFx59DLeoKNrM/h+qYNerEfeN6EuUXxTzk+dbfcyck3MIcA9gdFTdZmDKES8ZW2jn3YJo75YV6rxEhzlex7KPoTVq6RfRD53BxCcbUujeOoDRXezT9OLv7s/4mPGsPruaHG1Olfvk6fJwV9ZdOPWC5gJv7X6LH479QJ4uj1fPfsbuy7trPEZ7+DCXnnuOUyNvIufnX/AeMIC2c+YQNX8e/reORVDXXXLGFiyyEsW7dyMa7TtPtjGTEC9lRbalNI2ol+x4WYkoiuVdjbbQJdKffh2K8VEFIhb14lLJOZ6cv52sQr39jbSo1cfZ2fEqj3hpMZgNLDu1jMGiB2HuNT/9ZRbq+PWvc0zsEUn4yUNcfOwx3Nq1o83/fkEVFGRfG+uIIAhMjZvK4azDJOcm17r/5aLLbLmwhSntp+Chqlvxr5tCdrxkbGN4aG/2552kQJ8PSjdQOab+aW/6XgQE+kT04de/znGlQMfLo+PrFNmtjjs63oHBbGBBStUdxfn6/Do5XnqTnjkn5vCPHf/gctFlHun+CP8e+G/aerTk6c1PcyDjQKX9RYOBglWrODt9Oudun0HRjp0E3XMPsRvW0+rTT/Dq1dOuP7e1+AweglmjQXvkiNOv3VCJD/clwq/pyErIjpeVFOmNmEVqTTVWxfGc4/SO6Ma3U6cDsP70LoZ/uJUfd5yxb4ts8mqI7A1+dhYLrOB4bb+wnRxdDlNKBfC4fvRFRb7degaDSeQxrwwuPvY4brExtPnlZ4eH621lQswE3BRu1d4IKjIveR4A0+On1/l6Fs0vWctLxlpGhPXGJJrZnnnQ4YX18UHxYPbiqy2nGRoXyoAY+5YDRPtHM7DlQOYlz7tudJbepMdoNqJU2FbEfyjjEC9sfYHlp5czuNVgPhr2EQmtE/Bx8+Hz2H8Q4R3B45seJyk7CVN+Ptk//MCpm27m8vMvYM4vIPy1f9F+6xbCX3oRdWTVY8qchfeAG0GplGUlKmCRldiZ2jRkJWTHy0o0ZUNg/Txta+EuNhRztuAsnUM60zuiC95qb8bfoKV3lKR+P+azHexItYMXr7kClw7YP80IlboaF6YuJMwzjEHFJTU6XhkaHb/vSeNpr3SMr76Ie1wcbX9ueE4XQIBHAKOiRrHyzMoa60t0Rh2LUhcxvPVwWvjU3bmVI14yttLJL4ow90Cpu9HLMX9DepOeI1lH6BPeh++2naZAa+Cl0fEOudZdne4iW5vNurR1lbZrrxVqroVcXS6fHPiE9/e9j5vSjTdufINHuj+Cn5tf+T5Ban9+vPlH4jTebH/mTlKGDiXro49xaxdFq2+/IXrNaoLuvBOFt7ddfrb6ovTzw7NnD4p2bHe1KQ2KhPhQCvVGDjYBWQnZ8bKSghLpyczWiNfxnOOIiHQJ7oJKoaJXWC+SCw7zy712Vr9PWSO9xl8/m7DelOl4pRens+vSLia2n4hKV1Cj4/XN1tP0uXiMm/78BPf4eNr8/BPKgAD722Ynbou/jWJDMWvOrql2nzVn11CgL6iThERFZB0vGVtRCAqGh/VmV/5JtArH6HcdzTqK3qQnzr8HP+86y4QeLSuJHduTAS0HEOUXxe/Hf6+keagp1aCwQp7BLJpZe3Ytz299noMZB5keP533h7xPx+BrhimLItojR9A/+xqvfHqZgYf17OyswP2Pr2n7yy/4JiTYVQ7CXvgMHoL++AmMWU0jtWYPBsaGSLISKY3/d9LwPnENFI1OuknaWuOVmJ0IQJeQLgD0i+jH2YKzZGuz7at+n7wGAqMgzAFT3MtSjUsy9iAiMil6Aug11TpeVwq0nF68ilf3/YpHp06S0+XvmAXcXnQP7U77wPbMT6m6yF4UReacnENsQCx9wvvU61pycb1MXRge2gud2cBfOUcdcv596ftQCAp2JfpjMos8f5Njol0gOZJ3dryTpJwkjmRdrWWypr7rbMFZ/rXzX/wv6X+0D2zPB0M/YFL7SaiudUgPJeH/4Ftk/edddMdPEPLUk3iv+IPfJ/jx0Jl3uFR0yRE/ml2wyEoUybIS5fh6qOndNrBJ1HnJjpeVWAZkW9vVaCExO5FIn8hydXeLKOH+DEnPyxb1+2rRF8GZbRB/i32GYl+LyhMTsDj3MDe2uJFWbmXdjNU4Xsu/+IOX9sxG3aEjbX76EaWfX5X7NSQEQeC2uNs4nnOcpOyk694/lHmIk7kn6ywhURE51ShTF/oExOGr9GTzJcfcjPem76WdbxxLDuRyZ/+2tAl2TOekhfEx4/FV+5YLqppFMxq9plrHS2vUMjtpNq/ueJUcXQ5P9XyKV/q9QoT3NR2XxVqEj35A8cQbCLpSgp54jNjNmwh97DFi2vXi+5u+R2vUMmvdLDKKMxz6M9YV9w4dUIWGyunGa0iID+PEFQ0ZmsYtKyE7XlZSoK1bqjEpJ4nOwZ3Lv+8Q1AEftQ970/dW2s8a9ftqOb1JmqUYf4tNtlmN2oO/PD1INxQyOW4y6Aqk7VU4XmcXLWfgH5+Q1zqW2Nm/oPStXXKioTA2eiyeKs8qo15zT87F182Xse3qn8qVi+tl6oLabGRoaA+2XdyG0WxfqQGdUcfRrKPoCqPwUCl4YrhjBEIr4qX2YnL7yWxM20h6cTpaoxYR8boHG1EU2XtlL89vfZ61Z9cysu1IPk74mAGRA65/CNp7BOHu52DJesTpt5L/4xv4DB2Kwu1qF2h8UDzfjvyWXF0uD2540PHTROqAJCsxmOJdsqxERcplJRp51Et2vKykvLjehlRjri6XS0WXytOMAEqFkt7hvatVsK9O/d7i+FVJ8hqp06nNjVbbZhMqTxb5+hCocGd46+HVOl6a1aspee0VkoPaEvPLTyh97Cji6gR83Xy5pd0trDm7Bk2ppnx7ZkkmG9M2Mil2El520E+SI14ydcJYyojIoeTr8zmUeciupz6cdRiD2UBqWgQPDokmxKfuOlq2MKPjDERE/jz5J1qj9jpHKqskiw/3f8jHBz7GR+3D2wPf5oGuD+CtvqYQvqgY4b/foHj23+DuhvjtfxCfuhc8qv45uoZ25asRX3Gl6AoPb3iYAn2Bg37CuuMzeJAkK3HUManlxkiHiDJZiUau5yU7XlZicXx8ahgZdC2WlFXFiBdI6cZzmnNkllT94alK/X74h1uZt+88ZvM16UeTEVLWQtwohw3NzTYUsdXLk3FebaX6pCocr4KVq7j0woscD4wi6em3iIx0rSJ9XZkWPw2tUcvK0yvLt81Pno9JNHF7h9rnUlqDXFwvUyfMJga0HoK70p1N5zfZ9dR7r+wFUYG/EMeswdF2PXdNRPpEMrz1cBamLiSjKKN8qoPRbGTF6RW8sO0FErMTubPjnbw7+F3aB7a//iS7DyDc9Sys2oJ41yTEXz6ALrXXp/WJ6MOnwz7ldP5pHtv4GMWGYnv/ePXCe8AAUCgo2i6nGy0IgsDQuFB2pGZjbMSyErLjZSUarQFfDxVKhfX1PYk5iQgIdAruVGm7pc7r2nTjtVRUv28X4s3Li44x8etdHDpfoZ32wt9lQ7EdlGYElp9ZjlEQmOJRpm9zjeNVsGIFl196ifQ28fx78IM8NKqrw2xxNJ2DO9M5uDMLUhYgiiKlplIWpCxgSKshtPZtbZdrlDteJtnxkrEFES/PIG5seSObz2+2vgbUCjad+wuTLpKnhnXBx90xD3DVcWfHOynQF7D23Frcle6k5qXy6o5X+ePEH3QJ6cKHQz9kXMy464vnNYUIb3+O4sX3wNcH8ft3ER+9E9ytF5cdGDmQD4Z+QFJOEk9seqJBzIm0oPT3x7NHD4plPa9KJMSHUqgzcvB8vqtNqTOy42UlGp3tqvVJ2UlE+Ufh41Y55RYfGI+vm6/VA7O7RPqz4JEb+XR6D9ILdEz6ejcvLDgiqd8nr5GUrGPtOBS7AqIosjh1Mb1KTURT9vNXcLwKli3j8sv/QNGjF493m8mUge2J8K+bontD4bb42ziVf4rDWYdZd24dubpc7uhQPwmJivi5+fHpsE8ZGDnQbueUaeKIotQ4o/JkeOvhXCm+woncE3Y5dZG+mDOaE3iZ47ijf1u7nNMWeof3pkNQB9anred/Sf/j9V2vU2Qo4rnez/FCnxcI9Qq9/qCtfyPc+Sxs3IV43zTEn9+HjnWrSxvRZgTvDnqXAxkHeHbrsw2qBMBnyGB0x4/LshIVGNhekpVozOODHOZ4CYLQWhCELYIgHBcEIUkQhKfLtgcJgrBBEITUsteGp6hZBRorBmRXRBRFErMT6RLc5br3LHVetUW8KiIIAhN7RrL5hQQeGRrDssOXGP7hFgoOL8Pcbgi4O6aIfX/GftI0aUwuFcAibljmeOWv38nlf7yCV79+/HTLk5jdPXg0IcYhdjiT0VGj8VH7MD95PnNPziXKL4obWt5gt/O7Kd0Y0WYErXxb2e2cMvalwa1fplJw8wWFgoTWCSgEBZvPb7bLqb/duwkEE7d3GYabyvnP4oIgcGfHO7lcfJnN5zczut1oPkr4iH4t+l1fPJ9XgPDaxyj++SGEBCL+9D7irOlQz1mKt0TfwpsD3mTXpV28vP1luzcv1BXvwWWyEjt3udiShoOfh5pejVxWwpF/ZUbgeVEUOwE3AI8LgtAJ+AewSRTF9sCmsu8bPBqt0SbV+oySDHJ0OXQO6Vzl+/0i+nGh8ALpxek22eHjruIfYzqw7pkhjGupwV97gc8vtreP+n0VLEpdhI/ah5vN7pUcr/wzXlx58x28b7wB478/YGFSFnf1b0uYb+OOdoHUbXVr9K2sObuGY9nHmNFhBgpBDg43MxrW+mXUgafk4wV6BNIrrJdd6rxKjWb+PLYFRAWP9B9Z7/PVlbHtxjIpdhJvD3ybezrfg6dlWoYFUYSNO6Varh17MT80A/GH96B9lN1smNx+Mi/3fZmN5zfyr13/wiy6vobIo2NHlCEhFMuyEpVIiA8l6bKGzMLGKSvhsLuJKIpXRFE8WPZ1IXACiAQmALPLdpsNTHSUDfbE1lTjtcKp19Ivoh8giRbWhehQH97peAGA7UJf+6jfX0OBvoAN5zaUySx4SYs/kLf1KFf2BuA9YACtvv6aL3ZdwF2l5OGhjT/aZWFa/DRMoglvtTcTYie42hwZJ9Pg1i+TsVIzy4g2IziVf4rzmvP1Ou2cPWmUKJKJ9uuIj7vrRuaolWrGx4ynnX+769/MzkN45f9QvPEptAxD/PkDuGcKqOxfi3ZXp7t4qudTrDqzin///W+71tHVBUGhwGfQIIpkWYlKJMSFAY1XVsIpj/GCIEQBPYE9QLgoilfK3koHwqs55iFBEPYLgrA/qwHktwtsTDUm5SShElR0COpQ5fvtA9vj7+5fZ8cLQEheDS17Mee5ifZRv7+GlWdWUmouZUr7KdLYIIOWvHnzSZ93BO/WAq2+/oozGgPLjlzm7gFtCfV1Tgu6M4gLjGN8zHhmdZ11feu6TLOiwaxf6qtRoGFthgHUK91YpDfy+ZZElB6XGBE1oN7m1RdfN9/Ks1JFEVZvRbjrGdhzBPPjMxG/fQei7dPkUh0PdnuQB7s+yMKUhXyw/wOXO18+QwZjLihAe+yYS+1oSHRs4UuYr3ujHR/kcMdLEAQfYBHwjCiKmorvidInuspPtSiK34ui2EcUxT6hoVUUVzoZjdZgk2p9YnYi7QPbV6vCrBAU9AnvY1OdVyUK0+HSfoi/xT7q99cgiiKLUhfRMaijNP9M7Une3kzS33gD7/Z+tBofiMLdnc82ncJLreThIU0n2mXhnUHvMKvrLFebIeNCGs76JUIFDblIn0g6BnWsV7rxxx1nKBBTQDCXd1q7kij/KERRlOqrMrIRXngXxTtfQrvWiLM/hDsmgFLpFFue7Pkkd3a8k9+O/8ZXh79yyjWrwyIrUbxD7m60UC4rkZLVKGUlHOp4CYKgRlq0/hBFcXHZ5gxBEFqUvd8CaPCtCQaTmeJSk9WpRlEUJcX6auq7LPSN6MuloktcLrpsu1Epa6XXDldlJOqlfn8NidmJpOalMjVuKgC5h0pI35iHT0ICrcYHofAJIDm9kJVHL3PPgCiCvK1v4ZaRaQw0mPXLZACV53U6fcPaDONI1hGytdk2nzK7SM8P288Q1zYDlUJFj7AedjK27rgr3YkNiEW/aIVUy3XkBOZnH0D86m1o09KptgiCwEt9X2Jy+8l8d/Q7fjr2k1OvXxFlQACe3btTJMtKVCIhPgyNzsjhC/muNsVmHNnVKAA/ASdEUfy4wlvLgXvKvr4HWOYoG+xFYZlqvb+VxfXnC89TWFpYZUdjRSxPmXVKN55cDQFtIazTdW/VSf3+GhalLsJT5ckt7W4h97ffyVh7GZ92KiI//wyFURqQ/dmmFLzdVDzoRMFFGRln0KDWrwqF9RUZ0WYEIiJbLmyx+ZRfbj6FzmjGw+8c3UK6XV/M7gJKL15E+/hL+H76B6b4KMRfP4KpY0DhmsYWhaDg9RteZ0y7MXx68FPmnJjjEjugTFYiMRFjTo7LbGhoDGofglIhNMruRkd+ogcCM4HhgiAcLvt3C/Bf4CZBEFKBkWXfN2hsHZBdW2G9hdiAWALcA6xKN1588klyfv5F+kZfBGe21jgU2yb1+2soNhSz+uxqxgcMQfPWe2S88w6+nUNoNUKUZp7pCsgXPVl9LJ37B0YRKEe7ZJoeDWf9MpaC1/WOV/uA9rTyaWVzndf5nBL+2JPGpN7BnNEk069FP3tZWmdKL17k7ISJ6I4lEvzGP9F88BzGiGBXm4VSoeSdQe8wrPUw3tv7HktSl7jEDu/BQwAo3umYAemNEX9PNb3aBDTK8UEOkygWRXEnUJ3Mu2PUPh2ErQOyE7MT8VB6EBNQc92TQlDQN6KvVUKqxX/vAUVZfcOZLdJQ7A61q9Vb1O/v6NeGN5cn8fKiY/yx5zxvje9MzzZVSxCtS1nJTTuLmbxnMwWlBoLuu4+w+IsIqaulHXQFHMgw4+uh4oFBcrRLpunRsNYvsVJhvQVBEBjRZgRzTs6hqLToOqHm6vhoQzJKhcCgLgWs+9tc3mHtSrK//gbRaCR6xXLc2rTBXJxOSl4KwZ71d75U9ayxUyvUfDj0Q57c/CRv/vUmnipPRrcbXW+7bMGjU0eUwcEUbd+B/wS5y9pCQnwYH6xLJrNQ16ikjGRxIivQ6GyLeCXlJNEhqMP1Iy6qoG9EXy4XX+Zi4cVq9xGNRsyFhZgLy2p7T662eSh2jer3luuIIoUbNxL+0HvctcWMT7/+RK9YTvjLLyF4eINBB2YT6DUcy4EHBrXD36t+woUyMjJWUM1w9uFthmMwG9h5ybpISNLlApYdvsz9A9uRojmMm8KNbqHd7GmpzZSeO0fBsmUE3n47bm3aABDmFUaAewBFpUX1Pr8qLKze53BTuvHpsE/pEdqDV3a8wtYLW+t9TluwyEoU79yJaKp/x3pTYWic5FRvT7G9ztGVyI6XFVgiXtYU1xvNRk7knKi1sN5C3/Da67xMGk3Za+HVodjtbwalbU5P1er3W/lxxxmKTpzk/H33c/GJJykW9CS/Np02336De7syXR2VBxi1oJdsKVX5cv+gKjR3ZGRk7IfZKI0EU1XdHd09tDtBHkFWdzf+39pk/D3VPDw0hn3p++gR1qPazmtnkf3NNwhqNcEPXu0gVggKYgNiKTWXYjI3DEfDU+XJVyO+okNQB57b+hx/Xf7Lqdf3HjIYU0EBOllWopzOLf0I9XVna3LjSjfKjpcVaLSW4vraHZ3T+afRmXR0DrbO8YoJiCHII6hmxytfGtFjKtTAxb2gzbUqzVgdFdXv+0d6kffuf0ibPJmipOMcndmPVx/0YOiUpysfpPYCs5GTZ84C0CsuyubZlTIyMjZi1FcSTr0WpULJsNbD2HFpR60zBnefzmZbShZPDIsFRQnJucn0iehjb4ttQn/mLAUrVhJ4xx2oQkIqveel9iLaL5oCfYGLrLseHzcfvr3pW9r5t+PpLU9zMOOg8649cCAoFC7rbjTr9RRu2eJyXbOKlMtKpGY3KlkJ2fGygqupxtpTh0k5SUDthfUWBEEo1/Oq7gNtys8HwKwphJOryoZi13+8R3SoD//lJLee/YutHYcyfcjT/LdVMv1bJhDocU39l1rKny/dfgCAgV2bnm6XjEyDw6ivsqOxIsPbDKfYUMyeK3uq3UcURd5fm0xLfw9m3tiW/Rn7ERFdXt+V/fXXCB4eBM96oMr3W/i0uF5Y1cX4u/vz3U3fEe4VzuObHicpO8kp11UGBODZrRtFLtLzyvrscy4++liD0xNLiA+lQGvgyMV8V5tiNbLjZQUFWgMqhYCnunbxvsTsRHzUPrT1a2v1+ftF9COjJKPaOi+L42UqLEQ8uQqiBtttKLZm7Ro8unfjgXlfMmKYFiPFbNrb7nr1e5XkeF26cA4AT58gu1xfRkamBkSx1r/1/i3646XyYvOF6rsb1yamc+RCPs/eFIeHWsm+9H14KD3oGtLV3hZbjf7UKTSrVhF0552ogqpeTxSCgvaB7dEatQ1idqKFEM8Qfrj5B/zd/Xl448Ok5KU45breFlmJ3FynXM9CaVoaub/9BkDenLlOvXZtDI4NRSHQqGQlZMfLCjRl44KEaqQbKpKYnUjn4M42DVW26HlVJythKigLtRuNiFnn6pVmrEjpuXPoj5/Ab/QYPNRKCtW7iPBqyU3tBl6vfl/WVRXlXibGWkP6Q0ZGxl5U3dFYEXelO4NbDWbL+S1V1kMZTWY+WJdMXLgPk3u1AqS1pkdYD9yUrpOCyfrqKxSengTdf1+N+3mrvYnyi6JA13BSjgAR3hH8cPMPuCvdeWj9Q5wrOOfwa/oMHgKi6HRZiYwPPkBQq/GfOoWibdsovXjJqdevCX8vNb3aBMqOV1OjwMpxQXqTntT8VKsL6y20829HsEcw+zKqrvOyRLwATKUCxI2x6fzVoVm7DgC/0aNI06SxL30ft8VP5Ys7el+nfv/XeSnUP7J12VOn7HjJyDgW0QyCsjzaXBPDWw8nR5fDsezrC6/n77/ImexiXhzVAaVCIFeXS2peqkvTjLrkFArXriPw7pmoAmtOpYI0IslT7YnWqHWCddbT2rc1P9z8AyIis9bP4lKRYx0Sj86dymUlnEXx339TtHETIQ89ROjjj4MgkD9vntOubw0J8aEcu1RQqUu/ISM7Xlag0RmtcrxSclMwmo1W13dZEASBvhF92XdlX5V1XqaC/PKvzf6dwD/SpvNXh2bNGjx79kTdogWLUxejFJRMiJU0Yq5Vv/9pTzoAnX110sGy4yUj41iMevDwq1YkuSKDWw1GpVBd192oLTXx6cYU+rQNZGRHSVbBohvoyvmM2V99hcLbm+B777Vqf6VCSfvA9pQYShpUyhEg2j+a72/6nhJjCbPWzSKzxHEddpKsxECnyUqIJhMZ7/0XdcuWBN17D+oWLfAZPoz8hQsxl9bczOFMEuKlz/b2RjI0W3a8rECjNeDnUXthfWJOmWJ9LaOCqqJvRF8ytZmkadKue69SxKtt/YvqAfRnzqBPTsZvzGgMZgPLTi1jcKvBhHld1bypqH4/opukr6MqyQAEcPezix0yMjLVYNSDp3W1lL5uvvSP6M+m85sqPbxpdAa6RPrz8pgO5aUSe9P34qnytDkyby90J05QuH49QffcgzIgwOrj/Nz8aOPbBo1eU/vOTiY+KJ5vR35Lri6XB9c/SK7OcTVY3oOHYMrPR5eY6LBrWMhfvBh9cjJhL76AwkOKvAbdcQemvDwK1661z0V8Iup9ik4t/AjxcWer7Hg1HTRWphoTsxMJ8ggiwtv2D5Il7F9VurG8xgswtx5u87mrQrN2LQgCvqNGse3CNnJ0OUxpP6XKfYO83ZgxoIP0TWG65HS5aH6ajEyzwWy0qYlmeJvhXCi8wKn8U+Xbwv08+PnevvSNuurA7U/fT6+wXqgVrpGDyfryKxS+vgTdc7fNx7bybYWb0g2dUecAy+pHt9BufDXiKy4XXebhDQ+jKXWMg+g9cIBTZCVMRUVkffoZnr164Tv6qlK/1w034BYVZb8ie78W9T6FQmGRlcjCVMtIvIaAfPe0Ao3OYJWGV1J2El1CulhVhH8tbf3aEuoZyr4rVThe+fnlT4YmTaHN566KwjVr8OzdC3V4OItSFxHmGcagyEHVH2CpMynMkNOMMjJOQai1sL4iw1oPQ0CocXZjtjab0wWnXZZm1CYmUbRpE0H33YvSz/aouUqhIi4wjuLS4galJ2WhT0QfPh32KafyT/HoxkcpNhTb/RqqwEA8u3Z1uKxEznffYcrJIfyVVyrd0wSFgsAZt6M9fBjd8eMOtcEWEuJDyS8xcPhCvqtNqRXZ8aoFURTRaI21ioUWG4o5U3CmTmlGqFDnlXF9nZcpvwB12SgNU2H9n6L0qanoU0/hN3oMV4qusOvSLia2n1jziCPLDUBfIDteMjKORhSlSZE2OF6hXqF0C+1Wo4q9pb7LVYX12V9+icLfn6C7bY92WfB39yfSN7JBphwBBkYO5MOhH5KUncSTm590SHTOe/BgdMeOOUxWovTCBXL/Nxv/CRPw7Hr9Pc1/4kQEDw/y5v7pkOvXhcHtQ1AIsK0RqNjLjlct6AxmSk3mWiNex3OOIyLWq26ib0RfsrXZnNWcrbTdVJCPWyupDdxsh4iXZu06Kc14800sPbUUEZFJsZNqPqhiZ5XseMnIOBZTKai9QVG7dmBFhrcZzoncE1wpulLl+3vT9+Kt9qZjcEd7WGkT2qNHKdq6leD77kPpY91A7+po49sGhUJRq1q/qxjRZgTvDHqH/en7eXbrsxhMBrue32fIYElWYtcuu57XQuYHH4JKRehzz1b5vtLfH79bx1KwcmX5SDtXE+DlRs82gWxrBHVesuNVC9aq1lvUi23taKyI5SnU8lRqwZRfgCo0FMHdvd4RL1EU0axZg1ffvihCgll8ajE3triRVr6taj6w4pBe2fGSkXEsVijWV8WINiMAqhVT3Ze+j97hvWuObjuIrC+/RBkQQOBdd9X7XGqlmriAOApLCxtkyhFgbPRY3rjxDXZe2snLO17GaDba7dweXbqgDAx0SJ1X8d69FK5fT/CDs1CHh1e7X+CMGYhaLQVLl9ndhrqSEBfK0UsF5BQ1bFkJ2fGqBWsHZCfmJNLSuyVBHnVXdG/t25owr7BKQqrm0lLEkhKUAf4o/HzrHfHSp6RSeuYMfmNG89eVv0gvTmdy3OTaD1TLES8ZGadhKq2T49XWry0x/jFV1nlllmRyTnPOJWnGkkOHKN6+g+BZD6D08bbLOYM8gwj3CndYEbs9mBI3hZf7vsyGtA28tus1u0lhCAoF3oMGSbISZvvJa4gmExn//S+qFi0Ivq9mYVvPzp3x6N6NvLlzG4zzmxAfhijC9tSGHfWSHa9a0JQ5XrWlGhOzE+vdni0IAv0i+rEv/Wqdl0VKQhkQgNLXD1Nh/Rwvzdo1oFDge9NNLEpZRKB7IMOt6ZRUVag1kR0vGRnHY0N9V0WGtxnOgYwD5OvyK223PNC5YjB29hdfogwKIvCOO+x63nb+7VCgsHsqz57c1ekunur5FCvPrOQ/f//Hbk6Kz5DBmPLy7CorUbB0KfrjJwh7/nkUnrV//gJnzKD07FlK9lQ/J9SZdG7pR4iPW4NXsW9SjldKRiFPzj2EzmA/YbmrqcbqHa88XR6Xii7RObj+ujj9IvqRq8vlTMEZ4FrHyxdzHfLpebo8fk78maOZRyhcvQav/v3I94atF7YyLmacdWNDlCqwtJ/LjpeMjOOpo+M1os0ITKKJbRe3Vdq+P30/vm6+dAjsYA/rrKbkwAGKd+8meNYsFF5etR9gA25KN2IDYht01AvgwW4PMqvrLBakLODD/R/axfnyHjQIBMFu6UZTUTGZn3yKZ/fu+I21biyd35gxKAMCGsz8RoVCYEj7ULanZGE0NSyh3Yo0Ocdr5dHLPDHnkN1+6VdTjdXXRCTl1L++y4LladTydGou0/BS+vuj8LMt4pVVksWH+z5k1KJRfHLgE5av+4LStDT8Ro9h+enlGEVjtdpdVWK5EciOl4yM4zAbpQizsm46W52COxHuFX5dd+Pe9L30Du+N0saC/fqS9cWXKENCCJxxu0POH+wZTIhHCIV6+0jtOIqnej7FnR3v5Nfjv/L1ka/rfT5VYCAeXbtStGO7HayDnO+/x5SdTfirr1gtiaRwdydg6hQKN23CkJ5uFzvqy7juLckrMfDDjrO17+wimpTjdWu3lrw9vjMbT2Tw0qKjmO0gpKbRSgWRNaUaLYX1nYI71ft6rXxa0cK7BfvSJT0vYx0iXleKrvDO3+8wetFofjvxGyPajKBDUAci/j4FSiU+N41kcepieoX1Ijog2nrjLJ2NsuMlI+M4jLo61XdZEASB4W2G89flv8pnG6YXp3Oh8ILT67uK9+yl5O+/CXnoQatSV3VBEASiA6IxiSa7FrDbG0EQeKnvS0xuP5lvj3zLz4k/1/ucPoMHozt6DGNeXr3OU3rxIrn/+x9+48fh2b27TccGTJ8OZjP58xfUywZ7kRAfyqjO4XyyMYUzWUWuNqdKmpTjBTDzxiieuymOxQcv8Z9VJ+od0i2PeNXgeCXmJBLlF4Wvm/Uq09Vh0fPan74fs2i+JuLlW2PE67zmPG/sfoNbFt/CwtSFjIsZx8qJK3lv8HvEB8QRdzAb7xtu4HDpadI0aUxub0VRfUXUsuMlI+NwjKX1/hsb0WYEOpOO3Zd2A1cj6M50vERRJPuLL1CFhhJw220OvZaHyoPYgFgKdAW17+xCFIKC1294nTFRY/jkwCfMPVm/FN1VWYnd9TpP5ocfgUJB2HPP2XysW+vWeA8ZTN6C+YgG19faCYLAvyd0wUOl4GU7BWDsTZNzvACeHB7LfQOj+HnXWb7acqr2A6rBkJmJRmvAy02JWln9r8qiWG8v+kb0JU+fx6n8U9cV15sLr2+fPpV3ipe3v8y4peNYdWYVt8XfxprJa3hzwJu09msNQEyGgtA8E96jb2ZR6iJ81D7cHHWzbYZZJCVkx0tGxiEY8wok8VT3+nX+9QrvhZ+bX7msxN4rewlwD6B9YHt7mGkVJXv2ULJ/P8EPP1w+58+RhHmFEeQZRFFpw4xyWFAqlLwz+B2GtR7Gu3veZemppXU+l0eXLigDAiiuR7qx5MABCteuJXjWLNQRdZubGDhjBqasbAo3VS/e60zC/Dx47dZO7DuXx+97rp9/7GqapOMlCAKvje3EpJ6RfLg+hd/+tv0Xn/fnPE6PGo146UKNUhIZxRlkabPs7niBpLljKigAtRrBywuFny+iwYColzRKjucc59ktzzJp+SS2XNjCPZ3uYe2UtbzS/5Xr5kXGHLyCUQHZfWPYcG4DY6PH4qmyMfQvpxplZByGqaiI03c/R9obP1C4a1+9ZALUCjUJrRPYemErBrOBfen76BPeB4XgnCVfFEWyPv8CVUQEAdOmOuWagiAQExCDwWTAZLZfg5UjUCvUfDj0Qwa0HMAbu99g7bm6DZwWlEq8Bw2iaEfdZCVEs5mMd99DFRFB8AP318kGkFKe6sjIBlNkDzC1dyuGxIXy/pqTXMwrcbU5lWiSjhdI3Q3/N7UbIzuG8fqyRJYfuWzT8T7DEhCUSvot+QE/j+qLURNzpFZee3Q0Woj0iSTSJ1JyvPLzUQb4IwgCSl9pttnRM3/x6MZHmb5yOnuu7OHhbg+zfsp6nuvzHCGeIdedTxRFgnaf5GiUwC8XFlBqLrWtqN6CXFwvI+MwBEEg5K5JlKZnc/GJpzk9Zgy5v/+Bubhu8/6Gtx6OplTDitMruFx82akyEsW7dqM9eJCQhx9C4e7utOt6qjyJDogmX5/vtGvWFTelG58O+5QeoT14ZfsrbL2wtU7n8RkyGFNuLrok2+cmFixbji4pibDnn6tXDZ6gVBJw+3RK9u5Ff6ruWSZ7IggC707qggi8uiSxwWiNQRN2vADUSgVf3tGLvlFBPDfvMFttmOGkDg8n7PnnaHMuiaHnrh9cbSEpOwmVoKJDkH1btPtG9GV/xn6Mefko/f0RRZEzpgwA/rHmSZKyk3i619Osm7qOJ3o+QYBHQLXn0h07hjI9h786Cqw5u4aOQR3rNjJEdrxkZByGwtub4Om3EvvH50R+8jGqwCAy/vMfUhOGkfH+/1F68ZJN5xsQOQAPpQefH/wccF59V3ltV8sW+E+pwwNePYnwjsDf3d8hA6rtjafKk69GfEWHoA48v/V5/rr8l83nKJeVsDHdaC4uJuvjj/Ho1g2/sWNtvu61BEyZgqBWN6j5ja0CvXh5dAe2p2Sx6KBtfz+OpEk7XgAeaiU/3tOHuHBfHvn9AAfSrB8qGjB9OmcjYhizfR7GnJwq90nMTiQ2MBYPlX1rGPpG9KVAX0BRzhWKPAVmrpnJJ8nfAfBguztYO2Uts7rOsqqgX7NmLahU7IsTMItmpsbVMfSv8gQEcPer2/EyMjK1IviG4jdmDFF/ziVq3p/4DB5M7q+/cvrmm7n41NOUHDxo1dO7p8qTAS0HkKPLIcgjiNiAWCdYD8U7dqA9coSQRx5B4WaFRqCdUQgK2ge0R2/SQ8MJclSLj5sP3970LW392/L0lqc5lHnIpuNVQUF4dOlCsY16Xtk//ogxK4vwV/6BoKi/K6AKCsLvljEULF1a5yitI5h5Q1v6RgXy75XHySy0/8DyutDkHS+Qxv3Mvr8fEX4e3PfLPk6mWye2JygU/HLjDNwMOjLefe+690VRJCknya5pRgt9w6U6r4uXTnJYf5qskiym95ZGOIwKGYSX2johQtFsRrN2Ld6DBmH0csdT5ckt7awTx7sOtYfkdNnhj1RGRqYKFGpwuzpA2rN7dyI//ojYTRsJfuABivfsIe2OOzk37TYKVqxELK15SPTwNtJUij7hfazWZqoPltoudWQkAZMmOfx61eGl9iLKL8opP7M98Hf35/ubvifcK5zHNj5Wrg1pLT6DB6M9erS8Gas2DJcukfvzL/iNHYtXz551sLhqAmfMwFxcTMGKlXY7Z31RKAT+O6UbWoOJ15fa9nt1FM3mDhrq685vD/THy03FzJ/2cj7HumK7ZPcQTgybjGbVKoq2Vw7lXii8gKZUY9fCegstfFrQKbgTvjqIa9ublZNXcnNXSf7BZMO8Ru2RIxivXMF/zGg6BXdiUuwkfCos7DYREg/h9dcqk5GRqQavIHC//u9THRFB2PPP0X7LZiLefANzcTGXX3yRUyNvIvvb76rVcUponYCfm1/58GxHU7RlK7rEREIeexRBXTcBWHvR0rslrX1bO62hoL6EeIbww80/4O/uz8MbHiY1L9XqY32GDAazmaJdu6zaP/OjjwEIe952+Yia8OjeHfdOHcmbM6dB1VTFhPrw7Mg41ials+bYFVeb03wcL4DWQV789kA/DCYzd/20h0xNzWFHs1mkUG/k0pjbcIuJ4cqbb1YKoSZmS4X1jnC8AObcMocAnZLoNt1QK9QofaW0ornQ+vEYhWvXIri54TNiBLPHzOblfi/X3aCEl+H+unXfyMjIWIF/JKiqL0ZXeHkRePvtRK9aSevvv8O9fXuyPv2UUwnDuPLa69cVNvu7+7N9+nbGtBvjaMulaNeXX6Bu0wb/8eMdfr3aUCqUtPFr02iiXiDVp/1w8w+4K9x5cP2DpGms68j36NpVkpWwIt1YcvAQmtWrCX7gftQtW9bX5EoIgkDgjBnoU1LQHjxo13PXlwcHt6NLpB+vLUsiv6TmSLGjaVaOF0D7cF9+ubcv2UV67v55LwUl1Qu+FeqNiCL4+HnR4t9vY7x8hazPPy9/PzEnEXelOzEBMQ6xVdCXIpaWovQPAEDhJ9VWWRvxktKM6/AePBiljw8KQdFonv5kZGSqR1Ao8BkyhDY//Uj0iuX4T5hAwfLlnLl1HOcfmEXR9u3l8gJKhdIpzkfRpk3oj59oENGuxkxr39b8MOoHRERmrZ/F5aLaO/IFpRLvgQMp2lmzrIRoNpPx3nuowsIIfuABe5pdjv/YsSh8fRuUtASASqng/6Z0J7+klLdX2t4Bak+a5V24Z5tAvpvZm9NZRdw/ex8lpVWPmdBUUK336tWLgBm3k/vb72iPHQOkjsb4oHjUCscsMlfFU6UuQoW7O4Kbm9URL+2hQxgzMvAbPdoh9snIyLge9/btafH2W8Ru3ULoM8+gT0nhwkMPc+bWceT9+SfmEsdrGIlmM1lffIlb27b433qrw6/X1In2j+b7m76n2FDMrPWzyCypvSPfZ8hgTDk56I6fqHYfzYoV6I4dI/S5Z1F410+ktzoUXl74T5qIZv16jNnZDrlGXenU0o9HE2JYfPCSTSoH9qZZOl4Ag9uH8tntPTl0Po8XFxytcp+rA7IlxyrsuedQhYRw5V+vYdBrOZF7gi7BjkkzQgXHqyziBVLUy1RgneOlWbMWwd0dn2HDHGCdjIxMQ0IVGEjIIw8Tu2kjLT/4PxSenqS/+Rapw4aT+dFHDh1iXLhhI/rkZEKeeBxBpXLYdZoT8UHxfDvyW3K0OTy0/iFydTV35HsPGgRQrYq9uaSEzI8+xqNLF4enggNvnwEGA/kLFzn0OnXhieGxxIb58M8liRTpXTPbs9k6XgC3dG3BI0NjWJ14hYwq6r00OsnxsgzIVvr6EvH6a+iTkznz7adojVqH1XcBkmo90rggC+5RUWiTEms9VjSZ0Kxbi8+QISh9HPNkIyMj0/AQ3NzwHzeOqIULaDvnD7xvuIGcn37m1IiRXHruebRHjtjlOmF+Ui2aaDaT/eUXuEVH43dLHTumZaqkW2g3vhzxJReLLvLIhkfQlFb/0K0KDsajSxeKyuq8DBmVIzo5P/2MMTOT8FdfsYt8RE24R7fDe8CN5M2bh2hqWFME3FVK3p/SjcsFWt5fc9IlNjRrxwtgcq9WiCKsrqLT4Wqq8eoTnO/IkfjedBPGH+cQkSvSOcT+UhIWKs5ptOA9aCD64ycw5tb89FNy4ACmrGz8xshpRhmZ5oggCHj16kWrzz4lZv16gu6+m6Lt2zk3/XbO3T4DzZo1iMa6P/GH+0nahYVr16JPPUXI448hKKuf8iFTN/pG9OXTYZ+Smp/KYxsfo8RQferYZ8hgtEeOYMrPx5iVVb7dcOUKOT/9hN8tY/Dq1csZZhMwYwbGK1co2rbNKdezhd5tA7lvQDt++zuNPWeq1uh0JM3e8YoN86FDhC8rj1bleEmL0rWzGsP/9S+MKoFH1wm09W3rMNtM+ZaI11WleO+BAwEo3l2zwnHh2rUIHh74JCQ4zD4ZGZnGgVurSMJffonYrVsJ/+c/MebmcunZ5zh1083k/PhjeXTdVkSTiawvv8K9faxcS+pABkUO4sMhH5KYnciTm59EZ6y6I997sCQrUbx7d6XtmR9/AmYzYc8/7wxzAfAdNgxVeHiDK7K38MKoONoEefGPxcfQGZwblWv2jhfAuO4tOZCWx6V8baXt5alGr8qOlzo8jHVjw+l4zkjhkqUOs+tqjddVx8ujUycU/v4U16DXIhqNaNatxychAYWXdUKrMjIyTR+ljzdBM+8iZs1qWn39NW5t2pD54UekJgwj/e230Z85a9P5NKvXUHrmDCGPPyFHuxzMiLYj+M+g/7AvfR/PbX0Og+n6jnzPbt1Q+PuXpxsBtIcPo1mxgqD770MdGek0ewWVioDpt1G8cyeladbJYjgTLzcV/53clbPZxXyyMcWp15YdL+DWbi0AWHW0cttugdaAIICPW+Vi0VJTKX/GZZEbH0HG/33gsM4NU34+gocHCo+r44gEpRLvG2+kePfuagXqSvbvx5STIz+BysjIVImgVOI7fBhtZ/+PdkuX4DdmDPkLFnLmlls4//DDNa4vFkSjkeyvvsI9Ph7fm29ykuXNm1ujb+X1G19nx6UdvLzjZYzmyqliQanEZ+CAclkJURRJf+89lKEhhDz4oNPtDZg6FVSqBjW/sSIDYkOY0a81P2w/w9GL+U67rux4AW2Dveka6X9dulGjNeDnoUahqKyBk5KXgkE0oX/hfsSSEjLefdchdpkKCirVd1nwHjgAY0YGpadPV3mcZs1aBC8vfIYOcYhdMjIyTQePDh1o+e47xG7ZTMgTT6BLTOL8/Q9wdvx48hYswKyrOq1VsHIlpefOSZ2M8hgxpzE1biov9X2JDWkbeH3X65jFyrpd3oOHYMrOpvTcOTQrV6E7cpSwZxwnH1ET6rAwfG8aSf6SJZi12toPcAGv3NKRUF93Xlp4lFJj9Rpo9sRhfy2CIPwsCEKmIAiJFbYFCYKwQRCE1LLXQEdd31Zu7daCoxcLSMu5qkxfoDVUKqy3YFGs79BzBMGPPoJm9RoKt261u02m/PxKaUYLPgMGAFSZbhSNRgrXr8c3IQGFp6fdbZKRaS40tjWsvqhCQgh94nFit2ymxXvvgVJF+muvc2rYcDI/+wxD5tUuOdFoJPvrb3Dv1BHfkSNdaHXzZGanmTzZ80lWnFnBO3+/Uyk66TNYkpUo+ftvMj/6CPdOHfGfNNFFlpbNbywoQLN6jctsqAk/DzXvTOzKyfRCvt1WdTDD3jjyMeV/wLW5rn8Am0RRbA9sKvu+QTC2LN1YMeql0RmvK6wHyfEK8giihXcLQmbNwr19LOlvvY2pyL4T2auLeKkjI3GLiqLomgJKgOI9ezDl5eErdzPKyNSX/9GI1jB7oXBzI2DSRNotWUyb2bPx7NWLnG+/k+QoXnoJbWISBcuWYzh/ntAnnmxUI3maEg92fZAHujzA/JT5fLT/o3LnSxUSgkenThQsXYoxPZ2IVxwvH1ETXn374t4+lry5DbPIHmBkp3DGd2/JF5tTScmwfhZyXXHY/4YoituBazUPJgCzy76eDUx01PVtpVWgFz3bBFR2vLSGcg2viiTlJNE5uDOCICC4uRHx9tsY09PJ+vRTu9kjiiKmnJwqHS+QuhtL9u7DXFp55lTh2rUovLzwGSKnGWVk6kNjW8PsjSAIePfvR+uvviRm7RoCb7+doo2bODd1KulvvYVHly74DEtwtZnNFkEQeLrX09zR4Q5mH5/NN0e+KX/Pu2xotu+oUXj17etCKyU7A2bMQJeYWD71pSHyxrhO+HqoeWnhUUxmxw74drYbHC6KosWzSQfCq9tREISHBEHYLwjC/qwKeiSO5NZuLTlxRcPprCKgLNV4TcSrxFDCmYIzlYRTvXr2JHDGDPL++APt4cN2sSX3518oTUvDs3v3Kt/3HjgAUatFe/BQ+TbRYKBw/QZ8RoxA4V79oF0ZGZk6Y9Ua5or1y5G4tW1LxD9fJXbbVsL+8TLucXGEv/ySHO1yMYIg8HK/l5kUO4lvjnzDL4m/AOA/fjzucXGEvfiiiy2U8B8/HoWXV4OVlgAI9nHnjXGdOHwhn1922dbdaysuiz+KUly0WrdSFMXvRVHsI4pin9DQUKfYNLZrCwQBVh6R1lWN7voar+M5xzGL5usU60OfexZVeDhXXnsdsbR+k881a9eS+cEH+I4ZTdC991S5j1e/fqBSVdJrKf77b0wFBbJoqoyME6hpDXPF+uUMlL6+BN97L+0WLnB5JEVGQiEoeOPGNxgTNYaPD3zMnyf/xD06mhbvvotbK+fJR9SE0scHvwnj0axejTEvz9XmVMv47i0Z2TGMD9cnV6r3tjfOdrwyBEFoAVD26roplVUQ4e9B37ZBrCyTlSioItWYlJMEQKfgTpW2K318iHj9dfSpqeT8/HOdbSg5eJDLL72MZ8+etPzvf6vNzSt9fPDs0b1Sgb1mzVoUPj7lM7tkZGTsToNew2SaJ0qFkv9v7+6jpajvO46/P/cBLs9cAQFzUSwRU0UFA4mASQAx8eloaDWA5ogn9tRqkqMmxvhQiWlD4lMbo1VTG+uJLcEYNIo5jRJFW+sTIoqA0vqEcBFB5cGLPHO//WPm1gXuI1xmduXzOmfP3Zmd3f3MzO7vfnfmNzPTvjSNMQPGMO35aTz0xkN5R9pN9aTJxJYtrN+H577cW5L4ydePorKsjCvuX9jiKVX2VNaF1yygYRPOFKDoPh2nHdOf11dvYNGK9WzeVr/brsbFHyymX5d+9O7Ue7fndhs3lm4nncQHt9/Blrfbvqly69Kl1F70bSr796fm9tta3F3YZdQoNr/6KtvXriW2bqXuscfodsIJlHXo0Ob3NrNWKfo2zPZPlWWV3PSVmxjZfyRTn5nKY2ufyzvSTqoOH0yn4Z9n7b33EvXZnLZhT/TrUcXVp/45z771ITPmLt8n77EvTycxA3gWOFxSraTzgeuAEyW9DoxPh4vKyUP6UyaY/vwyYPez1i/6cBFDejV9Yex+V1+Fqqp4b+qP2vTh2r5mDcv++gKQGHDnP1NR3fJR6l1Hj4YINj77LBueeYb6jz6i+yknt/o9zaxppdqG2f6rY3lHbh57M0P7DOWat27lpdUvtfykDFVPnsy2ZcuavfJKMZg4YgCjBvXip//xGivXt//5x/blUY2TI6J/RFRGRE1E3BURH0bECRFxWESMj4jmr/Scgz7dOjJyUC9mvbwC2Pk6jRHBlCOmMOGwCU0+v6JPHw78wWVsfOEF1t1/f6ves37zZmov+jbbV62i5vbb6HBI667/WDVkCGXdu7Ph6aep++MjlHXvTpeRI1v1XDNrXqm2YbZ/61zZmdtOuI1z+p66W5eYvHU/8UTKe/cu6k72kOxyvO4vjmZHfXD17xe1+y5Hn264EacdfRAfb00umlnYuV4SEz83kS/XNH+qhp5nnknnESNYfcONO510sDFRX8+7l/+QTQsWcNANN9B52LBW51R5OV2OO46Pn/pv6h5/nG7jxyPvZjQz26917dCV79ScTcfy4jq6XR060POsM9nw5JNsrV2Rd5xmHdyrM5d97XDmLFnNrAXvtvyENnDh1YiTjuxHRXqZoMbO49USSfT7ux8TW7awalrzlxNafeNN1M2ezYGXX073r321ze/VZfRotq9eTf2GDXQ/2bsZzcyseFV/4xsgse6++/KO0qLzRg3k2IN7cu2sxXywYUu7va4Lr0ZUd+nA6M8mnecbO3N9a3Q89FB6X3QRdY8+St2cOY1Os2b6dNbcfTfVZ5/d5GkjWtJldHL5oPIePehy3Bf36DXMzMyyUNm/P13HjWXdzJm7nQC82JSXiev/8mg+3rKDa2ctbrfXdeHVhMlfOJgenSrp26Nqj1+j1/nfouPgwenlhDbs9FjdnCdYNe2ndB07lr5XX7XHJyLsUFND1VFH0WPCBFS5Z0WimZl9ulQU8fnjqidPZseaNdQ9OjvvKC06rG83vjvus8xZspratRvb5TW1r85T0Z6GDx8e8+bNy/x96+uDsrK9OzPzpgULWDppMtWTJ9Nv6jXJuIWLeOfcc+k4aBCH3PNryjp33qv3aFiHPou0fVpIejEihuedoz3k1X6ZFauor+etk0+hvFcvBv5met5xWrRtRz2rPtpMTXXr/1c314Z5i1cz9rboAuh0zDFUf/ObrJ0xg43zX2Jr7QqWX3ghFdXVDLjj9r0uuiApuFx0mZlZKVBZGT0nT2LT/PlsXrIk7zgtqiwva1PR1RIXXhnoc/HFVPTrx8prrmH5BRcQW7cy4F/uLOpNwWZmZvtKzwkTUFUVa2fcm3eUzLnwykB51y70+9FUtr75JluXLaPmllvoOGhQ3rHMzMxyUd6jB91PPYX1Dz/Mjrq6vONkqqLlSaw9dBszhr5XXkGHgQN99KGZme33qs8+m/q6DdTX1VHerVvecTLjwitDB0zZs1NGmJmZfdp0OvJIam75Rd4xMuddjWZmZmYZceFlZmZmlhEXXmZmZmYZceFlZmZmlhEXXmZmZmYZceFlZmZmlhEXXmZmZmYZceFlZmZmlhFFRN4ZWiTpfeCdHN66N/BBDu/b3jwfxcXz0bJDIuJTcTHTPWi/SuXz4ZztpxQygnO2RZNtWEkUXnmRNC8ihuedY295PoqL58OaUyrL1TnbTylkBOdsL97VaGZmZpYRF15mZmZmGXHh1bw78w7QTjwfxcXzYc0pleXqnO2nFDKCc7YL9/EyMzMzy4i3eJmZmZllxIWXmZmZWUZceKUkDZD0hKRXJS2WdHE6/gBJf5L0evq3Ou+srSGpXNJLkv6QDh8q6XlJb0j6raQOeWdsiaSekmZKWiLpNUkjS3F9SLo0/UwtkjRDUlUprA9J/ypptaRFBeMaXf5K3JLOzyuSjs0veeloYhkPlfScpJclzZP0hZwzlkTb2EzOG9M25BVJv5fUsxhzFjz+fUkhqXdeGdMcTeaU9N10mS6WdEMx5iy279FOIsK3pJ9bf+DY9H434H+BI4AbgCvS8VcA1+edtZXz8z3gN8Af0uH7gEnp/V8CF+adsRXz8Gvgr9L7HYCepbY+gM8AbwOdCtbDeaWwPoAvA8cCiwrGNbr8gVOAPwICjgOezzt/KdyaWMazgZMLluuTOWcsibaxmZxfBSrS8dcXa850eADwKMkJd3sXY05gLPAY0DF97MAizVlU36PCm7d4pSJiZUTMT+/XAa+R/NM8g6QAIP379VwCtoGkGuBU4FfpsIBxwMx0kqKfD0k9SP4p3QUQEVsjYh0luD6ACqCTpAqgM7CSElgfEfFfwJpdRje1/M8A7onEc0BPSf0zCVrCmljGAXRP7/cA3s001C5KpW1sKmdEzI6I7elkzwE1eWWEZpcnwM+By0k+A7lqJueFwHURsSV9bHV+KZvNWVTfo0IuvBohaSAwDHge6BsRK9OH3gP65pWrDW4m+fLWp8O9gHUFjU8tn3zRi9WhwPvA3eku019J6kKJrY+IWAHcBCwjKbjWAy9SeuujQVPL/zPA8oLpSmmeis0lwI2SlpN8dq7MN84nSqVt3CVnoW+RbJktCoU5JZ0BrIiIBfmm2t0uy3Mw8KW0q8R/ShqRa7gCu+S8hCL9Hrnw2oWkrsD9wCUR8VHhY5Fss8z9l0hzJJ0GrI6IF/POspcqSHbB3BERw4CPSXZn/L8SWR/VJFsGDgUOAroAJ+Uaqp2UwvIvURcCl0bEAOBS0q2+eSuVtrGpnJKuBrYD0/PKVqgwJ0muq4CpeWZqTCPLswI4gKRLwQ+A+9K9KrlqJGdRfo/AhddOJFWSrLjpEfFAOnpVwy6T9G+um1VbYTRwuqSlwL0ku7R+QbLrpyKdpgZYkU+8VqsFaiOi4RfrTJJCrNTWx3jg7Yh4PyK2AQ+QrKNSWx8Nmlr+K0j6pzQopXkqNlNIPicAvwNy7xRcKm1jEzmRdB5wGnBOWiTmqpGcg0h+nC1I2+4aYL6kfvmlbHJ51gIPpN0K5pLsWcn7QIDGchbd96iBC69UWrHfBbwWEf9Y8NAskhVI+vehrLO1RURcGRE1ETEQmATMiYhzgCeAM9PJSmE+3gOWSzo8HXUC8Coltj5IdjEeJ6lz+hlrmI+SWh8Fmlr+s4Bz06MbjwPWF+yGsrZ5F/hKen8c8HqOWUqmbWwqp6STSLpenB4RG/PKV5Bnt5wRsTAiDoyIgWnbXUvSYfy9YsqZepCkgz2SBpMc+PRB5gFTzeQsqu/RTvLu3V8sN+B4kk3lrwAvp7dTSPpHPU6y0h4DDsg7axvmaQyfHNX4Z8Bc4A2S6r9j3vlakX8oMC9dJw8C1aW4PoAfA0uARcC/AR1LYX0AM0j6pW0j+UdwflPLn+RoxtuAN4GFwPC885fCrYllfDxJP8AFJH1VPp9zxpJoG5vJ+QZJ/8OGcb8sxpy7TLOU/I9qbGp5dgD+PW3P5gPjijRnUX2PCm++ZJCZmZlZRryr0czMzCwjLrzMzMzMMuLCy8zMzCwjLrzMzMzMMuLCy8zMzCwjLrxsn5C0Ie8MZmb7kqQdkl6WtEDSfEmj0vEDJYWknxRM21vSNkn/lA5fK+myvLJbflx4mZmZ7ZlNETE0Io4huRbgzwoeexs4tWD4LGBxluGsOLnwsn1K0hhJT0qaKWmJpOkN1/WSNELSM+mvxbmSukmqknS3pIXpxbEbzpB8nqQHJf1J0lJJ35H0vXSa5yQdkE43SNIjkl6U9JSkz+U5/2a23+gOrC0Y3gi8Jml4OjwRuC/zVFZ0KlqexGyvDQOOJLmEw9PAaElzgd8CEyPiBUndgU3AxSTX3D0qLZpmp5elABiSvlYVydmofxgRwyT9HDgXuBm4E/ibiHhd0heB20kuF2Fm1t46SXqZpE3qz+5tzb3AJEmrgB0kbeBBmSa0ouPCy7IwNyJqAdJGaiCwHlgZES8ARHI1eSQdD9yajlsi6R2gofB6IiLqgDpJ64GH0/ELgaPTq9OPAn6XblSD5PI8Zmb7wqaIGAogaSRwj6QhBY8/Avw9sIrkh6aZCy/LxJaC+zvY889d4evUFwzXp69ZBqxraAjNzLISEc9K6g30KRi3VdKLwPeBI4DT88pnxcN9vCwv/wP0lzQCIO3fVQE8BZyTjhsMHJxO26J0q9nbks5Kny9Jx+yL8GZmhdKuEeXAh7s89A8k3SLWZJ/KipG3eFku0l+CE4FbJXUi6d81nqRP1h2SFgLbgfMiYkvBrsOWnJM+/2+BSpI+FgvafQbMzD7p4wUgYEpE7ChsryJiMT6a0QooIvLOYGZmZrZf8K5GMzMzs4y48DIzMzPLiAsvMzMzs4y48DIzMzPLiAsvMzMzs4y48DIzMzPLiAsvMzMzs4z8H7p4ryIn9LeIAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 720x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,4))\n",
    "plt.subplot(121)\n",
    "sns.lineplot(\n",
    "    x='Income',\n",
    "    y='Scores',\n",
    "    data=data_ancova,\n",
    "    hue='Method',\n",
    ")\n",
    "\n",
    "plt.subplot(122)\n",
    "sns.lineplot(\n",
    "    x='BMI',\n",
    "    y='Scores',\n",
    "    data=data_ancova,\n",
    "    hue='Method',\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUQAAAEYCAYAAAAkpo9KAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABGhklEQVR4nO3dd3xU15bg+9+upFAKVQpVQkjCAZwz0QZnME7XOSecwLagu2f6zeu+M3PvdE+7+82dz/TrvrcvYAw2Ns4452tj44gDGJxwxoEghKoUqpSlCmfPH+cIC5CgSlKpStL6fj58kIo6p5aKYnHO3muvrbTWCCGEAFu6AxBCiEwhCVEIISySEIUQwiIJUQghLJIQhRDC4kh3AIk499xz9WuvvZbuMIQQ/VPpDmAojIgrxIaGhnSHIIQYA0ZEQhRCiOEgCVEIISySEIUQwiIJUQghLJIQhRDCIglRCCEsKa1DVEptBVqBOBDTWk9RShUBq4GDgK3AVVrrUCrjEGKgtDYIR+toj4VwO7x4nGUoJdcRo9VwFGafqbXuXUj4W2Ct1voPSqnfWt///TDEIURStDb4sX0DbwSWEtMRHMrFHH81E93TJCmOUun4W70YWGV9vQq4JA0xCHFA4Wjd7mQIENMR3ggsJRytS3NkIlVSnRA1sEYptUkptcB6zK+13mV9XQf4+zpQKbVAKbVRKbWxvr4+xWEKsa/2WGh3MuwR0xHaY+H0BCRSLtW3zLO01juVUj7gDaXUd73/UGutlVJ9tuzWWi8HlgNMmTJF2nqLYed2eHEo1x5J0aFcuB2e9AUlUiqlV4ha653W70HgOWAaEFBKjQOwfg+mMgYhBsrjLGOOvxqHcgHsHkP0OMvSHJlIlZRdISql3IBNa91qfX0O8E/Ai8A84A/W7y+kKgYhBkMpGxPd0yiprKI9Fsbt8Mgs8yiXyltmP/CcUqrndR7TWr+mlPoEeFIpdRuwDbgqhTEIMShK2fC6yvG6ytMdihgGKUuIWuufgeP7eLwRODtVryuEEAMl1/5CCGGRhCiEEBZJiEIIYZGEKIQQFkmIQghhkYQohBAWSYhCCGGRhCiEEBZJiEIIYZGEKIQQFkmIQghhkYQohBAWSYhCCGGRhCiEEBZJiEIIYZGEKIQQluHYl1mItJGN5kUyJCGKUUs2mhfJkk+FGLVko3mRLEmIYtSSjeZFsiQhilGrZ6P53mSjebE/khDFqCUbzYtkyaSKGLVko3mRLEmIYlSTjeZFMuS/SiGEsEhCFEIIiyREIYSwSEIUQgiLJEQhhLBIQhRCCIskRCGEsEgdohgyhtbURaKEojG8TgdlLic2pdIdlhAJk4QohoShNRta2llaEyCiNS6lqK7wM63ALUlRjBhyyyyGRF0kujsZAkS0ZmlNgLpINM2RiYHQWtO2aQM6Hk93KMNKEqIYEqFobHcy7BHRmnA0lqaIMoOh46xveoYPGh5PdygJi7U0U/cf/0rdn/+N0CsvpjucYSW3zGJIeJ0OXErtkRRdSuFxjt2PWDhSx5rAEuq6twCKg90nUp5zRLrD2q/2zzcRXLmceEszAJFdO9Fao8bIsMfY/bSKIVXmclJd4d9nDLHM5Ux3aMNOa83XLW/xfsNDRHU3ACcUnosv65A0R9Y/o6uLhicepuWdtQDYcnIpvfEW8k6eNWaSIUhCFEPEphTTCtxUTawkHI3hGaOzzB2xMGuDK/ilYxMAbnsRc/x3UZV7bJoj61/XT1sILF9CNGBurZBzxFH45lfjLC5Jc2TDTxKiGDI2pSjPclGe5Trwk0ehn9s3sja4nM54CwCH5Z3CGaW3km3PS3NkfdOxGE0vPUfopefAMMDhoPjya/DMPR9lG5vTC5IQhRikiNHF+w0P8XXLWwC4bLmcWXobh+fPTHNk/YvsqiWwfAndv/wEgKuiCv8dC8mqnJDmyNJLEqIQg7Cr8wfWBJfQHA0AUJFzNHN8d5HvzMzbTa01LW+/QcMTj6AjEVAKz9wLKLrsKmyusXll31vKE6JSyg5sBHZqrS9USh0MPAEUA5uAG7Xea2s0ITJcXMfY0PQMG0PPo9HYlZOTi67hRM95e2xRoLVBOFpHeyyE2+FN6xYGsXCI4Mp76fjycwAcRcX45leTe+TRaYknEw3HFeLfAN8CBdb3/xv4d631E0qpZcBtwD3DEIcQQyIUqeX1wGKC3T8DUOKqYq7/ryjOqtzjeVob/Ni+Yffe0D2bXE10Txv2pNi2aQPBB1ZgtLUCkHfyLEpvuAW72z2scWS6lCZEpVQFcAHwL8DfKnP+/izgOuspq4B/RBKiGAG01nzZvIYPGh+19ntWnOS5kBnFV+FQ+5YXhaN1u5MhmHtCvxFYSkll1bDt8WJ0dlL/2Cpa338HAJvbTelNt5E//ZRhef2RJtVXiH8E/g7It74vBsJa657lCzXA+L4OVEotABYAVFVVpTZKIQ6gPRbizeAytnV8AUC+o4Q5/moqco7a7zGxvUaDYjpCeyw8LAmxc8v3BJYvIVYfBCDn6GPx334XDm9Ryl97pEpZQlRKXQgEtdablFJnJHu81no5sBxgypQp+gBPFyJlfmxbz1vBFXQZbQAckX8ap5fcTJY9d7/HuR1eHMq1R1J0KBduhyeV4ZrlNM8/ZS670xrldFJ81XUUnj13zJbTJCqVV4gzgYuUUucD2ZhjiH8CPEoph3WVWAHsTGEMQgxYt9HBe/UP8m3rewBk2/I403c7k/JmJHS8x1nGHH/1PmOIHmdZymKO7KwhsHwx3du2ApA14SD8CxbhGl+R8Dm0YRAN1BELh3B4vDj9ZWMmkaYsIWqt/yvwXwGsK8T/orW+Xin1FHAF5kzzPOCFVMUgxEDt7PyWNwJLaYnVA1CVcxyz/XeS50j8dlMpGxPd0yiprKI9Fsbt8KRsllkbBs1rX6fxycfQ0SgohfeCiyi65EqUI/F/5towaN+0gcCKpehIBOVy4Z9fjXvytDGRFNNRh/j3wBNKqX8GPgPuT0MMQvQppqOsb3yKTeGXwCqnmVV8PccVzh3Qml6lbHhd5SkdM4yFmgjcdw+dX28GwFHqwz+/mpzDkm8kEQ3U7U6GADoSIbBiKZUVVbjGDc9EUDoNS0LUWr8DvGN9/TMwbTheV4hkNHbv4PXAEhoiWwEozTqYuf5FFLn6nPfLCK0bPqJ+1X0Y7e0A5J96BqXX3YQtZ//jm/2JhUO7k2EPHYkQC4clIQoxFmht8Hnza3zY+DhxHUWhmOy9mOlFV2BXmflPJN7eTsMjD9D60ToAbHn5+G6eT96UwV1rODxelMu1R1JULhcOj2dQ5x0pMvNvW4hh0hpr5M3APezo/AqAAoePc/wLKc85PM2R9a/j268JrlhKrKkRgNzjTsB36x04PN5Bn9vpL8M/v3qfMUSnP3UTQZlEEqIYs35o/ZC36++n2zBvN4/KP4PTSufhsuWkObK+GZEITc+uJvz6q2Y5jctFydU3UHDWnCHrWahsNtyTp1FZUUUsHMbh8cgssxCjWVe8jXfrH+D7tg8AyLblc7ZvAYfmTU1zZP3r3rGNwL1LiNRsByDr4EPxL1iYknE9ZbPhGlc+JsYM9yYJUYwpOzq+4o3gPbTFzNvNg3JP5GzfHSkvlh4obRiEX3+FxmdWQywGNhveCy+h6KLLkiqnEYmRd1SMCTEjwoeNT/B586sAOFQWp5bcyDEFZ2dsi/xoYwPBFUvp/O4bAJw+P/4Fi8ieOCnNkY1ekhDHqExqS5VqDd3beD2wmMbIDgD8WYdyjn/hsDVYSJbWmraPPqD+4ZUYnR0AFJx+FiXX3oQtOzvN0Y1ukhDHoExqS5VKhjb4LPwKHzWuxiCGwsZU76VMLbo0c8tp2tqof+h+2jZ8BIC9oBDfLQtwnzg5zZGNDZn5qRAplQltqVKtJVrPG4Gl7Oz6FjDXFZ/jX0hZdubebnZ8vZnAffcQDzUB4D5xMqW3LMBRUJjmyMYOSYhjULrbUqWS1prvWt/n3YYHiBidABxbMJtZJTfgtGXm7aYRidD41OM0v/EXAFRWFiXXzaPgtDMzdnxztJKEOAalqy1VqnXGW3m7/n5+bPsYgFx7IWf77uBg90lpjqx/3dt+MctpamsAyDp0kllOM0YKoTONJMQxKB1tqQbrQC2ptnV8wZuBZbTHQwAc4p7KWb755NoL+jtlWmnDIPTqizQ99xTE42C3U3TRZXgvvARlt6c7vDFLEuIYNJxtqYbC/lpSxYjyQeNjfNn8OgBOlc1ppTdxVH7m3m5G64MEli+ha8v3ADjLyvHfsZDsgw9Nc2RCEuIYNRxtqYZKfy2psscp3oqvJhStBWBc9mGc419IodOfznD7pbWmdd271D/6ILqrC4DCs86h+OrrsWVlpTk6AZIQxQiwd0sqreDHqRG2dP4JrQxs2JledAWTvRdjy9Cr3HhrC8EHV9C+6RMA7IUefLfdifu4E9IbmNiDJESR8Xq3pOrwwBcXQ6gSwMDrLGeufxG+7EPSHGX/2r/8jOD99xJvDgPgnjwN3823Y8/PzPHNsUwSosh4Tn8Zvvl3senjxXx9Vpy4dXd5XMFcZpZch9PW/+1mOlfkGN1dNKx+lJa33gBAZedQesPN5M88LWPHN8c6SYgi43XqNt4d/wE/nxcHIFcVMMdfzYS8E/Z7XDpX5HT9/COB5UuI1u0CIPuww/HPX4iz1JfS1xWDIwlRZLSt7Z/xZnAZHfFmACa6p3Om73Zy7PkHODI9K3J0PE7o5edpeuEZMAyw2ym+7Co85/1mzPQUHMkkIYqMFDW6WNfwCJtb3gTAZcvh9JJbOCL/1IRvN4d7RU6kbheB5Uvo/vlHM+byCvx3LCJrwkFD/loiNSQhioxT17WFNYElhKN1AIzPPpI5/moKnKVJnWe4VuRorWl5dy0Njz2MjnQDUHjOeRRfcS02l2tIX0ukliREkTEMHWdD07N8EnoOjVlOc3Lx1ZzouXBA5TTDsSIn1hwmuHI5HV98CoDdW4T/9rvIPfrYIXsNMXwkIYqMEI7s4vXAYgLdPwFQ5Kpgrv+vKM2aMOBzpnpFTvtnGwmuXE68tQWAvGknU3rTbdjz8obk/GL4SUIUaaW15quWN3m/4RFiuhtQnOg5n5OLrsZhG/ztZipW5BhdXTQ89hAt770FgC0nl9KbbiX/5FlD9hoiPSQhirRpj4VZG7yXrR2fAZDnKGKOr5rK3GPSHFn/On/8gcC9i4nVBwHIOfJofLffhbO4JM2RiaEgCVGkxU9tn7A2uJwuoxWAw/NmckbprWTZ3WmOrG86FqPphWcIvfy8uQWow0nxlddQOOc8KacZRSQhin6lYpVHxOjkvfpVfNP6DgBZNjdnlt7GYfmnDD7gFInU7jTLabb+DICrsgr/HX9FVkVlmiMTQ00SouhTKlZ51HZ+z5rAElpi5u1mZc4xzPbfRb6jeChDHzJaa5rXrqHxyUfN5hJK4Tn3QoovuwrldKY7PJECkhBFn4ZylUdcx1jf9DSbQi+g0diVk1OKr+WEwnMztgdjLNRE8P576fjqCwAcxSX451eTc8RRaY5MpJIkRNGnoVrl0RTZyeuBxdR3/wJAiesg5voXUpyVubebbZ+sJ/jgCoz2NgDyTzmVkhtuwZ6bm+bIRKpJQhR9GuwqD60Nvmxew7rGR4nrKKCY7PkNM4qvytgtQI3ODuofeZDWD94DwOZ245s3n7xpM9IcmRgumfnJFGk3mFUebbEm3gwsY3vnlwDkO0o4x7+Q8TlHpjrsAev8/lsCK5YSa6gHIOeY4/DfdicOb1GaIxPDSRKi6NNAV3lsafuYt4Ir6DbaATgy/zROK7mZLHtm3m7qaJTG554i/JeXzHIap5Piq6+n8Oy50rNwCCilNPCo1voG63sHsAtYr7W+cD/HnQCUa61ftb7/R6BNa/2vA4wjoeMlIYp+JbPKozvewbsND/Jdq3m7mW3L4yzffCbmTU91mAPWvXOHuQXo9q0AZE04GP8di3CVj09vYKNLO3CMUipHa90JzAF2JnDcCcAU4NUUxrYPSYhi0Go6v+GNwFJaYw0AVOUezxzfnbgd3jRH1jdtGDS/8RqNTz2OjkVBKbwXXEzRJVegHIn/k6jtjrC1s5tTPAfuzTjGvQpcADwNXAs8DpwKoJRyA38GjgGcwD8CfwH+CchRSs0C/pd1nqOUUu8AVcAftdb/YZ3jb4Fbrefcp7X+o/X4fwfmAUFgB7DpQIFKQhQDFtNRPm58kk/DLwMah3Ixs/h6jis8J2NvN2NNjQTuu4fOb74CwFHqw79gITmTDk/4HFpr3mhq4ZG6Bgw0ldkuKrNl17z9eAL4H0qpl4HjgJVYCRH478BbWutblVIeYAPwJvA/gCla60Ww+5b3COBMIB/4Xil1j3W+W4DpgALWK6XeBWzANZhXmg7gUyQhilRp6N7OmsBiGiLbAfA6D+LwwvmU5lSiMT+Zmab14w+pf+h+jA5zfDP/1DMovW4etpychM/R1B1h2dadfBk1tzPwOux0xI2UxDtaaK2/VEodhHl1uPct8DnARUqp/2J9n415BdiXV7TW3UC3UioI+IFZwHNa63YApdSzmMnWZj3eYT3+YiKxJpQQlVKHAjVa626l1BmYWfkhrXU4kePF6KG1wWfhv/Bh4+MYxFAoxuVewEuhabzSqnGpHVRX+JlW4MaWIVeJ8fY26h9eSdvHHwJgy8/Hd/MC8iZPTeo868OtrNixi7ZeE0ttsTihaAxD64z5eTPUi8C/AmcAvZcmKeByrfX3vZ+slOpr8Lm719dxUnBBl+gygWeAuFJqIrAcqAQeG+pgRGZrjTbwXO2/sK7xYQxiFDr9nOH7HS+GTqZbmx+liNYsrQlQF4mmOVpTxzdfseP3f7c7GeYefyJVd/+fpJJhR9zgnpoA/14T2CMZAkSBpTuDGfPzZrCVwP/UWm/e6/HXgb9S1hiLUupE6/FWzFvjA3kfuEQplWuNR15qPfae9XiOUiof+E0iQSaaYQ2tdUwpdSnwZ631n5VSnyV4rBgFvm9dx9v1K4kYHQAcXXAWp5bcxJYOg4iu3eO5Ea0JR2OUZ6Wvfb4RidD0zGrCr78CgHJlUXLNDRScOTup8c1v2ztZWhOgPhoDYEI8yjb7nuuYM+HnzXRa6xrgP/r4o7uBPwJfKrOm6xfgQuBt4LdKqc/5dVKlr/N+qpR6EHPsEcxJlc8AlFKrgS8wJ1U+SSTORBNiVCl1LeaMTU+m3e/qdqVUNmaWzrJe52mt9T8opQ7GHGQtxhzkvFHrvdaIiYzRFW/jnfqV/NBmXmHl2As427eAQ9xTAPA6I7iUIqL17mNcSuFxpm94unv7NgLLFxOp2QFA1iET8S9YiKtsXMLniBmap4KNvNgQRgNOpbg6x8Gh23fwv0qrMurnzWRa633ah2ut3wHesb7uBO7o4zlNQL+X8VrrY3p9/W/Av/XxnH8B/iWZeBP9W7wFuBP4F631L1ZSe/gAx3QDZ2mt25RSTmCdUuovwN8C/661fkIptQy4DbgnmaDF8NjRsZk1gXtojzcBcFDuScz2LSC31/K9MpeT6go/S2sCRLTGpRTVFX7KXMPfDUYbBuHXXqbxmdUQj4PNRtFFl+H9zaUouz3h8+zo6mZJTYCtXeb/0wdnZ7Gowk+5y0FbayPzXZoVEfXrzzvel5afVww9pXv9T7ffJyqVA1TtPfiZ4LG5wDrgLuAVoMy6BT8Z+Eet9dz9HT9lyhS9cePGZF9WDFDMiPBh4+N83vwXABwqi9NKbuTogrP7vN00tKYuEiUcjeFxOihzOYd9giHaUE9gxVK6vv8WAKe/DP8di8g+ZGLC5zC05rXGZh4PNBLVGgVcVOLlSl8RDpv582jDoDsYJGiz0WJ3UeTOoSzLJRMqmVlYkLREZ5l/gzlD5AIOtpbV/JPW+qIDHGfHvC2eCCwBfgLCWuuY9ZQaQJYFZJBg9y+sCSyhKVIDgD9rInP9C/G4+r/dtClFeZYrLWNoWmtaP3yfhkcewOjsBKDgzNmUXHMDtqzshM/TGI2xrCbA5nbzHD6ng+oKP0e49yzJUTYb2WVl/daFiJEt0VvmfwSm8et9/+dKqUMOdJDWOg6cYBVcPodZWJkQpdQCYAFAVZV8/FLN0Aafhl/i48YnMYijsDGt6HKmei/BphK/3RxO8bZW6lfdT9snHwNgLyjEd+sduE84KanzfNjcyv0762k3zHrCMzz53DSulFx7ZvZqFKmT8KSK1rp5r9ulhKtRtdZhpdTbwMmARynlsK4SK+hnXaPWejlmiQ9TpkxJ7L5eDEhLNMiawFJqu74DwOMcx1z/IvzZh6Y5sv51fPUFgfuWEQ+HAHCfOAXfLQuwFxQkfI72eJwHautZ12z1PbTbmD/ex7QC2UZ0rEo0IX6tlLoOsCulJgF/DXy4vwOUUqWYiTRsjT/OAf435nT6FZgzzfOAFwYavBgcrTXftb7HO/UPEtXmreKxBXOYVXI9Tlvit5vDyYhEaHzyMZrffA0AlZ1N6XXzyD/1jKTKab5u62DpziCNVjnNiXm53DHeJ7PFY1yif/t/hbnmsBuzIPt14J8PcMw4YJU1jmgDntRav6yU+gZ4Qin1z8BnwP0DilwMSme8lbeCK/ip3SzfyrUXMtt3Jwe5TzzAkenTtfUXAvf+megus+4xe+Jh+BcsxOnzJ3yOiGGwOtDEq41mOY1LKW4oK2FOUUHGrr8WyVFKXYI5RHek1vq7pI490CyzldDe1FqfOeAIB0lmmYfW1vbPeTO4jI54GIBD3VM5yzefHHvit5vDSRsGoVdeoOn5p81yGrudokuuwHvBxUltAbqtq5slOwJs7zbLaQ7NyWJhhV8KqodGxvxvYhVkl2M2jfiHZI494BWi1jqulDKUUoVa6+aBBinSL2p080Hjo3zZvAYAp8rh9NJ5HJl/esZeHUWDAbOcZotZ7eUcV26W0xx0wDm93QyteaUhzOpgIzFt3q5cWurlUl8Rjgz9uceCeEdHVWTHtrtj4VC5w1u0y1VR9Tt7bu72wZxTKZWH2fDhTOAlYGgToqUN2KyUegOz4SMAWuu/TubFRPoEun5iTWAJoah5u1mefThz/AspdPrSHFnftNa0vvc29Y8/hO7qAqBw9rkUX3UdNlfiV3T1kSj37AzyjVVOU+ZysrDCz6TczBwjHSviHR1V7RvXv1n/yAOTdCSCcrkoveGWGe4p02cPMileDLymtf5BKdWolJqstT5g268eiSbEZ61fYoQxdJyNoRfY0PQMBnFs2JledCWTvRdhy9AtQOMtLQQfXEH7p+byU7vHi++2O3Efe3zC59Bas665jZW19XRa5TSzvQXcMK6E7CRus0VqRHZsu7snGQLoSIT6Rx6Y5PSX3Z1z+JHzBnHqa4E/WV8/YX0/tAlRa71KKeUCDrMe+l5rLe09Mlw4WseawBLqurYA4HWOZ27ZInxZB6c5sv61f/4pwZX3Em8xR2fyps6gdN5t2PMS70rdFotzX209H7eY5TSFdjsLxvuYXOBOScwiebFwqLwnGfbQkQixcDi5Tb97UUoVAWcBx1p7udgBrZT6f3WCS/ISXalyBrAK2Io5eFqplJqntX5vAHGLFNNa83XL27zfsIqoNlvInVB4HqcUX4vDlpkTCEZ3Fw2PP0LLO28CYMvJoeSGW8g/5dSkxje/bOvgnpoAoZjZwHVyvpsF40spTGJrAJF6Dm/RLuVy0TspKpcLh8dTu5/DDuQK4GGt9e5mEVb37FMxG80cOK4EX+j/B87pWceslDoMc1+EyUmFK1KuI9bM2vrl/NJu3iW47UXM8d9JVe5xaY6sf10/bSGwfAnRQB0A2YcfiX9+Nc6S0oTPETEMHgs08lqjeWWZZVPMKyvlTG9+xk4YjWWuiqrfld5wy4y9xhC3uCon/H4Qp70Ws9a5t2esx4c0ITp7N3WwBiylvUeG+aV9E28Gl9MZN5PCpLyTObP0NrLtmbnyQsdiNL30HKGXngPDALud4suvwXPuBUmV0/zS2cXimgA7u81RnEk52Sys8FOWJR/RTGXPzd3unjJ9ttNfdncsHC53eDy1rsoJvx/MhEpfpYE9G1ElKtGEuFEpdR/wiPX99YAUBmaIiNHFuoaH+aplLQAuWw5nlN7K4XmzMvbqKFK3i8DyxXT//BMAropK/AsWkVU1IeFzGFrzYkOIpwJNxDEHjC73FXFxqRd7hv7c4lf23Nztg5xAGXKJJsS7gIWYS/bAbNG9NCURiaTs6trCmsASmqPm7eb4nKM4x1dNvrMkzZH1TWtNyztraXj8YXTEHN/0zL2AosuvTqqcJhCJsrQmwPcdZklOucvJwko/h+ZIOY0YuEQTogP4k9WZtmf1iuy7mEZxHeOTpmf5JPQcGo0NB6cUX8OJnvNRGVpOE2sOE1x5Lx1fmLtPOIqK8N1eTe5RxxzgyF9prXk33MqDu+rpMsyJw7lFhVxXVkyWlNOIQUo0Ia4FZmMWaAPkAGuAU1IRlNi/UKSWNYElBLrN281iVxVz/YsoycrcNmltmz4h+OByjNZWAPJmzKT0xluwuxMf32yJxVlRG+STFnNtgNdh587xPo7Pl3IaMTQSTYjZWuueZIi1LUBuimIS/dBas7nlDdY1PEJMRwDFSZ4LmFF8NY4MneMyOjupf2wVre+/A4At103pTbeSP2NmUuf5rLWdZTuDNFvlNNMK3Mwv95HvyMxejWJkSjQhtiulTtJafwqglJoCdKYuLLG39liIN4P3sq3jcwDyHMXM8VVTmXt0egPbj84t3xNYvoRYfRCAnCOPxj+/GkdRMdrQtNRF6QzFyPE6KChzomz7ToR0GQaP1DXwZlOLeQ6b4pZxpZzqkXIaMfQSTYj/CXhKKdVTNDkOuDolEYl9/NS2gbXBFXQZ5u3m4fmzOKPkFrLsmXmrqGMxmp5/mtArL4DWKIeT4iuvoXDOeSibDW1otm1o54OlAeIRjd2lmFntZ8I09x5J8ccOs5ymZ8/jI3OzuavCj082dBL9UErFgc2YC0jiwCKt9X57t/a234SolJoK7NBaf6KUOgJzu8DLgNcw908VKdRtdPBe/Sq+bX0XgCybmzNLb+Ow/Mwduo3U7iRw72K6t5kfD1fVQfjvWEjW+Mrdz2mpi+5OhgDxiOaDpQG8VZUUlruIa83z9SGeCTZhAHYFV/uKubDEI5s5iQPp1FqfAKCUmou5p/PpiR58oCvEezEnU8Bs///fMJvFnoDZ3v+K5GIViart/I41gSW0xOoBqMw5ljn+u8hzFKU5sr5prWle+zqNqx9FR6OgFJ7zLqT4sqtRey2b6wzFdifDHvGIpjMco6MYltQE+LHTLMmpyHKxqMLPQTlS1DDadMc7qhoi2+5uj4XK8xxFu4pdVb/Lsg+u/ddeCoBQMgccKCHarQ2jwbxFXq61fgZ4Rin1efLxiQMxtMFHjavZFH4R0NiVk5nF13F84dzMLacJNRG4fxmdX30JgKO4BP+CheQcfmSfz8/xOrC71B5J0eaCT7I7efrHMN3WOvzziwu5xl+MS8ppRp3ueEfVj+3r33y3/oFJMR3BoVycXnrLjInu6bMHmRRzrNyUjTm0d1YyBx8wIfbaEOpsrF3wEjx2VNCGQTRQRywcwuHx4vSXJbWsLFkKRUssAGhKsw7iHP8iil0VKXu9wWrb8DHBVSsw2s1SmPyZp1F6w83YcvovQigoczKz2r/7tjnmge13ZvFdh/mfeZHDwV0VPo7N+/UciU7CiJGhIbLt7p5kCBDTEd6tf2CSx1l29/icQa1e6X3LfDLwkFLqmKHqdvM48K5SqgFzVvl964UmAqO+e7Y2DNo3bSCwYik9C9D986txT56WsqSolOLM0tspcU3gJO9vsKvM/H8n3tFBwyMP0Prh+wDY3Hn4bp5P3tTpBzxW2RQTprnxVlXySaiN1YRptbrynFKYx63lpeTZfy2nSXQSRowc7bFQeU8y7BHTEdpjA2//tTet9UdKqRKgFAgmcsx+/7Vprf9FKbUW89JzTa8sa8McSxzVooG63ckQzH5tgRVLqayowjVuyP7e9pFtz2Nq0aUpO/9gdX7/rVlO09gAQO6xx+O77U4cHm/C5+jSmtU6zFvaLKdx22zcWl7KTM++fQ8PNAkjRp48R9Euh3LROyk6lAu3Y1Dtv/ZgTQTbgcZEj0lkT5WP+3jsh+RCG5li4RD9NLFMaULMVDoapfG5Jwn/5WWznMbloviq6yk8+5ykagJ/6Ohk8Y4AQWsL0KPdOdw13kdJP+U0+5uEkYQ4MhW7qn53euktM/YaQ9xS4hpU+y/4dQwRzNKbeVrreKIHZ+b9WIZweLz008Ryv8cZWlMXiRKKxvA6HZS5nCO+XKS7ZgeBexcT2bENgKyDDsG/YCGu8vEJnyOmNc8Em3i+PoQGnEpxjb+Y84oL9/v+9DUJY3cpcjzy8R2psuy52ye6p8/2OMvubo+Fy90OT22Ja8LvBzvLrLUe1NKlA25DmgnStQ3pQMYQDa3Z0NLO0poAEa1xKUV1hZ9pBe4RmRS1YRBe8xcan34cYjFQCu9vLqXoosv2KafZn53dEZbsCPBzlzlWOCHbLKepzD5wOY2MIY4Io+IvQv6L3Q9ls+GePI3Kiipi4TAOj+eAs8x1VluqiPUfTURrltYEqJpYOeL2/402NhC87x46v/0aAEepD/8di8iZeNgBjvyV1po1Tc08WtdIRGsUcGGJh6t8xTgTTGa9J2E6QlFyvU6ZZRYpIQnxAJTNhmtcecJjhqFobHcy7BHRmnA0NqISYuvHH1C/6n6Mzg4ACk4/i5Jrb8KWnXi/wVA0xrKdQb5oM89R4nRQXeHnKHdO8gEpaNjSxU/vtzL7t+WSDEVKSEIcYl6nA5dSeyRFl1J4nCPjrY63t1H/0Era1pvLP+35BfhuXYD7xClJnWd9cxsraoO0xc0tQE/z5HPzuBJy7ckP8XS1xll/X5Bt681ax69fDnHsJZm5YkeMbCPjX+kIUuZyUl3h32cMsWwENCTo+GYzgRX3EA+Zi5Nyjz8J360LcBR6Ej9HPM6Duxp4L2z1PbTbuL3cx4zCge3rsvOLdj5cFqQzZE4UVk51M+mswgGdS4gDkYQ4xGxKMa3ATdXESsLRGJ4RMMtsRCI0Pv04zWv+AoByZVFy3Y0UnH52UuU037Z3sqQmQINVTnNcXg53jvdTNICr41i3wabHGvn+dbP+35GtmDqvlIlnSNsvkTqSEFPAphTlWa4RMWbYvW2rWU5TWwNA1iETzXKasnEJnyNqaJ4KNvJSQ3h3Oc31ZcXMLSocUPJq/LmLdYsDNNeabb9KD89mVrWffH/mX2WL9FNKlQF/BKYCYSAA/KdE6qclIY5R2jAI/+UlGp99EuJxsNkouvhyvBdegkpinG9HVzeLawJs6zJrNQ/JzmJhpZ/xA/jPwIhrvnoxxBdPN6HjoOxwwhVFHH2xF5tMoogEKPN/4OeAVVrra6zHjgf8gCREsa9ofZDAiqV0/fAdAM6ycfgXLCL7kEMTPoehNa81NvN4oJGoVU5zSamXy0uLcAwgebXWRVm3NED9D+YueoXjncxa5Kf4YNlFb7TqiMertnVF7g7FYuVFDseuqmzX73Lt9sG2/zoTiGqtl/U8oLX+ItGDJSGOIVprWj98n/qHH0B3mTtAFJw1h5Krr8eWlXjiaYhEWbYzyFft5jl8TgcLK/0cnpt8OY3Wmh/fbuWTVfXEus2Z+SPOLeSk64pxuKTt12jVEY9XrW9pf/OB2vpJPZOPt5SXzphe4J49yKR4DLBpoAdLQhwj4q0tBFfdR/vGDQDYCz34br0D9/EnJnWeD8KtrKytp90wy2nO9BZwU1kJOfbkk1dnc4yPV9SzY6NZTpPjtTPzTj/lx8v+ZaPdtq7I3T3JEMxa3Qdq6yeVuZx3H+nOSdvm9ZIQx4D2Lz8neP8y4s1hANyTp+G7+Xbs+QUJn6MtHmdlbT0fNpubL+bbbSwY72NqwcDKaWo+befDe4N0NZvlNBNm5DHj9lKy8mQXvbEgFIuV97mAIRYbbNeUrxlEJ39JiKOY0d1N45OP0rx2DQAqO4fS6+eRP+v0pGZ/v2rrYGlNkKaYWU5zUn4uC8b78CSxlrlHtMtg0yMN/PCm2fbLmWNj2i0lHHKqlNOMJUUOx64+FzA4HINt//UW8P8ppRZorZcDKKWOAwq11u8f6GBJiKNU1y8/Ebh3CdE68/OVfdjh+OcvxFnqS/gcEcPgiUAjrzaatYBZSnHjuBLO9hYMKHnVb+li3ZIArXVmOY3/yGxmVvvJK5VymrGmKtv1u1vKS2fsNYa4ZUK2a1Dtv7TWWil1KfBHpdTfA13AVsydQw9IEuIoo+NxQq+8QNMLz5jlNHY7RZdeiff8i5Lq8r210yynqek2y2km5mSxsMLPuIGU08Q0Xz7XxObnQmgDbA444epijrrAM2LLaUZji7fhlGu3b59e4J5d5nLeHY7Fyj0OR+2EbNfvh2CWGa11LXDVQI6VhDiKRIN1BJYvoevHLQA4y8dTdscisiYcnPA5DK15uSHM6mAjcW22Rr/MV8SlpV7sA/gH31Ib4f0lARp/Mtt+eSpdnLrIj3fCyN1Fb7S1eEuXXLt9ezonUPoiCXEU0FrT8t7bNDy2Ct1tJp7COedSfOV12FyJX9EFrdZl33WYtYBlLieLKvxMzE2+FlBrzQ9vtrDx4QazsauCo873cOLVRdhHeDnNaGrxJvYkCXGEi7U0U//Acto/M0uv7B4v/tvvIveY4xI+h9aa98OtPLCrnk7D/Ec+u6iAG8pKyB7AZlqd4Rgf3htk52dm26/cYgezqn2UHT06ymlGS4s3sS9JiCNY++ebCK5cTrzFnPTImzaD0ptux56XeClMayzOitogG1rMWsBCh507xvs4Kd89oJi2f9LGR8uDdLeadYoHz8xj+q2luNyjp5xmpLd4E/1L2d+gUqoSeAhzDaHG3OT+T0qpImA1cBDm7M9VWutQquIYjYyuLhoef4iWd98CwJaTS+mNt5J38sykZn+/aG1n2c4goZhZCzg138388T4KHMknr2inwSer6vnxHbPtl8ttY/ptpRx8yr676A23oZ4AGckt3sT+pWxPFaXUOGCc1vpTpVQ+5nKaS4CbgSat9R+UUr8FvFrrv9/fudK1p0om6vzxB4LLlxANBgDIOeIofPOrcRaXJHyObsPg0bpG1jSZV5bZNsW8caWc4RlYLWDw+07WLQnQFjTrFMcdm8PMu/zkFqX/iilVEyA9SXaktHgbBqPih0/ZJ1ZrvQvYZX3dqpT6FhgPXAycYT1tFfAOsN+EKEDHYjS9+Cyhl54DrcHhoPjya/DMPT+pcpqfO7tYvCNAbcSsBTw8N5vqCj/+AVzdxGOaL59u4qsXQmgNNqdi8rXFHHFuYca0+E/VBMhIavE2liil4sBmwAnEMO9S/11rbSRy/LD8F66UOgg4EVgP+K1kCVCHeUvd1zELgAUAVVVVwxBl5orU1RK4dwndv/wEgKuiCv8dC8mqnJDwOeJa82J9iKeDTcQxd+++0l/ERSXeAV3ZhHdGWLc4QNMv5qy29yAXpy7046nMrHIamQAZczq11icAKKV8wGNAAfAPiRyc8oSolMoDnsFs0NjS+5bMqirv857dWnazHMxb5lTHmYm01rS8/QYNTzxi7g2tFJ65F1B8+dUoZ+JXdHXdUZbuDPCDVU4zPssspzk4ZwDlNIbmuzXNfPpoI/GoWU5zzEUejr+yGLsjM64Ke5MJkMwV6YhXhbZF7u4Mxcpzihy7vFWu37lyB1+Y3UNrHbQurD5RSv2jTmB8MKWfCqWUEzMZPqq1ftZ6OKCUGqe13mWNMwZTGcNIow1NS12U9h2NxN56kMj3Zis3R1ExvvnV5B55dOLn0pq3Q608VFdPl1VOc25RIdeVFeOybrN7Xq8zFCPH69jv9p4dTTE+uCfArs1m26+8Ugczq/34jxzALnrDRCZAMlOkI161fX37m+sfqJ/Us9f29FtKZ1RNd88e4qT4s1LKDvgwO2fvVypnmRVwP/Ct1vrfev3Ri8A84A/W7y+kKoaRpmdD9m+WvsXB6nmcyqzjyzt5FqU33ILdnXgpTEsszvKdQTa2muU0Xoeduyr8HJf3ay1gMhvAb/24jY9XBIm0m0Mxh56ez9R5pbhyM7vIeiTucTMWhLZF7u5JhgDxiGb9A/WT8sucd/uPHJ3tv2YCNwKblVKfW4/9N8xE+KRS6jZgGwNcczgahbe2UHvPvRxm+xSAqM5hq76YU35zPnZ34uNdn7a2c29NkOa4WU4zoyCP28pLyd+rnKalLro7GYL5ofxgaQBvVSWF5ebrRTribFjZwM/rzHKarHwbM273MWH6wNp+pYNMgGSezlCsvOdz1yMe0XSGB93+aw9KqUOAOAneiaZylnkd/U/Fn52q1x2pOn/4jqYli/GpBgDC8UP5MXo5EQroDMd2J6j96TIMHtnVwJshs7VWjs3GreWlzCrM67OcpjMUo58PJYXlLuq+6eSDpQHaG8xymvLjcznlTh+5Xhl/E4OTU+TYZXepPT5/dpcixzPo9l+7KaVKgWXA4kTGD0FWqqSdjsVofO4pwq++CFpjaAdbY3Opi08HbD0fkgOeZ0tHF0tqAtRZ5TRHWuU0pfsZK8vxOujrQ+nKs7Pp0Qa+fjkM2nxs8g0lHD5nYG2/hNibt8r1u+m3lM7Yawxxi3fC4Np/ATnWHWlP2c3DwL/t94heJCGmUWRnDYHli+nethUAV9VBxGfcSv0Tdoj/OqZXUNZ/UotpzXPBJp6rD2EADgVX+4u5oNiDTan9TpoUlDmZWe3fYwzxhKuK+GBxgNB2s+1X8aFZzFroT+gKVYhEuXLt26umu2fnlznv7gzHynM8jlrvBNfvBzuhorUe1BrRlK1UGUqjbaWKNgya175O45OPoaNRUArvBRdRdMmVYLObCSwcI8ez/1nf2u4IS2oC/NRp1gJWZrlYVOlnQnaW9ToHnjTpSZgdoSi1X3Ty7athjBgoBcde6uW4y4qwZWA5jcg4o+JDIleIwywWaiJw3z10fr0ZAEepD//8anIOO2L3cwrLXfu9ItNa82aohUd2NdBtbQF6frGHq/1Fu8tpILFJE2VT2F2Kzc+GqfvaLKfJ9zuZtdBH6WGZW04jRCpIQhxGrRs+on7VfRjtZilM/qlnUHrdTdhyEm+LFY7GuHdnkM/azJKcYqeD6vE+js7b9xwHmjTRWvPLB22sX1lPtMMsp5l0VgFTbirBmZ3Z5TRCpIIkxGEQb2+n4ZEHaP1oHQC2vHx8N88nb8q0pM7zSUsby3cGaY2byWtWYR63lJfitvc9bNLfpEmOx0F3W5z199ez9SNzF73sAjsn3+GjcvLA2n4JMRpIQkyxjm+/JrhiKbGmRgByjzsB36134PB4+3y+1gbhaB3tsRBuhxePs4wuA1btquedsFkL6LbZuG18KacU7r+1Vl+TJjOr/bTXR3njn3fS0WTWKVaclMuUSzSO+FYiu7w4/WVJNYwQYrSQhJgiRiRC07OrCb/+KmiNcrkoueYGCs6c02/pitYGP7Zv4I3AUmI6gkO5OKzwr3mxsYRg1KwFPMadw10VfooTWIurbIoJ09x4qyrpDMdw5dr48Z1WvnvNbPvlyFJMubGYcQXfEfzXpehIBOVy4Z9fjXvyNEmKYsyRhJgC3Tu2Ebh3CZEas4Ig6+BD8d+xEFfZ/ovww9G63cnQ0HZ+ip7Ge3UFQAynUlzrL+bc4sKklp0pm6Kw3EU8onl/cYDmGrOcpmSSWU6TrRvY8Q9mMgTQkQiBFUuprKjCNW5IFw0IkfEkIQ4hbRiEX3+FxmdWQywGNhve31xK0W8uRSWwqXt7LERMR+gwStkSvYJ2PR6AcpfBf66aQGV28q21DEPz9UthvniyESMOygbHX17EMZd4sdkVHd+GdifD3T9HJEIsHJaEKMYcSYhDJNrYQHDFUjq/+wYAp78M//yFZE+clPA5cuweAvFZ/BI9GwMnYFDp+Ii/mzCb0qzkk2FbMMq6pQGC35ltvwrKncxa6Kfk0F/bfjk8XpTLtUdSVC4XDo8n6dcTYqSThDhIWmvaPvqA+odXYnSapTAFZ5xNyTU3YstOvN9gUzTGslr4KXouAFkqxBHOF7ii/EJKXOOSjunn91rZ8GA90U5zhvnwcwqZfH0xjqw9xwXNxF1NYMWeY4hOf1lSrynEaCAJcRDibW3UP3QfbRs+BsBeUIjv1gW4T5ic1Hk+bm5jRW2QdqucZkaBnQuL3BS7qvE4y1Aq8cmNrpY4H98XZPsGs9Yxx2PnlDt9jD+h73IaZbPhnjyNyooqYuEwDo9HZpnFmCUJcYA6vvqSwH33EA+bGwa6T5xM6S0LcBQUJn6OeJyVtQ2sazbLafLtNuaX+5hWOLDWWjs/b+fDZUE6w2Y5TdU0NzPm+8jO3//yTmWz4RpXLmOGYsyThJgkIxKh8anHaH7jNQBUVhal180j/7Qzk+oE8017J0trAjRY5TTH5+Vy53gf3gG0to91G2x6tJHv15jlNM4cxdR5pRx6+sB20RNirJKEmISurb8QWL6YaO1OALInTsK/YCFOX+LjbVFD82SwkZcbwmjM/T1uKCthTtHAWms1/NTFusUBWnaZbb98R2Qzs9pPvk9a5AuRrFGRELVhEA3UEQuHcHiGfqWFNgxCr75I03NPQTwOdjtFF1+O94KLUf0sm+vL9q5uFtcE2N5lzugempPFwgr/gDo5G3HNV8+H+OKZJrQBNjsccrmHgtk5tGZp3FpLm3whkjTiE6I2DNo3bdhnlnSoVlpE64MEli+ha8v3ADjLyvHfsZDsgw9N+ByG1rzaGOaJQCMxc6M6Li31cpmvCMcAklZLXYR1SwI0bDHbfhWOd1J4cwHLbE1EtoeHbDN2IcaaEZ8Qo4G63ckQhm6lhdaa1nXvUv/og+gus46v8Oy5FF91HbYkagIbIlHu2Rnk63aztZbfZW4BOil3AFuAas2WtS1sfLiBWLdZTnPkeYWUXZbPf9++c8g3YxdirBnxCTEWHvqVFvHWFoIP3kf7pg0A2D1efLfdifvY4xM+h9aaD5rbWFlbT4dhltOc7S3gxrISsu3JX7l2Nsf4aHmQmk1mrWNukZ1T7vJTfmwuX7d1yGbsQgyBEZ8Qh3qlRfsXnxG8fxnxFnPG1j1lOr6bb8eet//OMr21xeLcv6uej5rN1lqFdjsLxvuYXDCw1lo7Nrbz0fIgXS1mOc1BJ+cx/bZSsvLM8UvZjF2IoTHi/8UM1UoLo7ubhicepuXtNwFQWdmU3HAzBbNOT2r2d3NbB/fUBGiKmclrcn4uC8b7KExgLfPeol0GGx9qYMtb5i56zlwb028t5ZBZeyZn2YxdiKEx4hPiUKy06Pr5JwL3LiYa2AVAszGBnzuvZKrzCAo0Ce0WETEMHg808pdG88oyy6a4qayEs7wDK6ep32KW07QGzHKasqNzmHmXD3fJvklONmMXYmiM+IQIA19poeNxQi8/T9MLz4BhYGg722OzqY3PBGz77D/Sn62dZjlNTbd52z4pJ5uFFX7KspK/QjNimi+fbWLzcyG0BpsDTrymmKPO9/S72RTIZuxCDIVRkRAHIlK3i8DyJXT//CMAtpLxfL7zEjr0r40Ueu8/0hdDa15qCPNksJG4Bhtwua+IS0q92AdwddZca5bTNP5kltN4q1zMWuTHW5V8pxshRPLGXELUWtPy7loaHnsYHbHq+M45D+fMy+n+fQD62H+kL8FIlKU1Ab7rMEtyxlnlNIcOsJzm+zXNbHq00dz/RMHRF3o44api7E657RViuIyphBhrDhNcuZyOLz4FwO4twn/7XeQefSza0H3uP7L3JvFaa94Nt7JqVz2dhpk8zykq5PqyYrIGUAje0RTjw3sD1H5h1ilmF9s5tdrPuKMT34mvh6E1dZEooWgMb5LjiH3t5ZJMlx0hRoMxkxDbP9tIcOVy4q3mjG3ejFMovfFW7G6zs8ze+4/0tUl8SyzOfbVBNrSYrbU8Djt3jPdxYv7Aymm2rW/joxVBIm1mnWLNsbDl3DhVFQb+JJfeGVqzoaV9n5nmRFar9LWXyxx/NRPd0yQpijFl1CdEo7OThscfouW9twGw5bopvelW8mfM3Oe5PfuP9DVm+HlrO8t2Bglb5TTTCtzcXu6jwJH4WuYekQ6DTx6s56f3zLZf0WzYfD7UHWX++UBWmdRZt/ADWa3Sey8XgJiO8EZgKSWVVXhd0hJMjB2jOiF2/vgDgXsXE6sPApBz5NH4br8LZ3FJwufoNgwerWtkTZNZTpNjU9w8rpTTPANrrRX4tpMPlgZoqzfbfhUcncVzZ3fTXfDrcwayyiQUjQ14tUrPXi69xXSE9lhYEqIYU0ZlQtSxGE0vPEPo5efNLUAdToqvvIbCOecl3fDhk5b23cnwiNxsqiv8+AZQ8ByPab54qpGvXgyDBrtTcdL1xRScnsOzP9fAIFeZDGa1itvhxaFceyRFh3LhdniSikGIkW7UJcRI7U6znGbrzwC4KqvwL1hEVmXV7uck0y5sZmEeG1vaOTgni9+UeAZU7Bze0c37SwKEtpoJp+igLGb9lR/PeBeG1vusMvmb8X5yGzV1oQ5yvPuOZfZlMKtVPM4y5vir9xlD9DhlXxUxtii9121WJpoyZYreuHHjfp+jtaZ57Roan3zUXNesFJ7zLqT40qtQzl+TwkDahWmtB3R7rA3Nt6818+njjRhRjVJwzMVejruiCLvj1/P1zA6HozGKHHa6Po/wwdLgHrPdE6a5D5gUe58n2dUqv84yh3E7PDLLLJI1KurDRsUVoo7F2PWn/0PH5i8AcBSX4J9fTc4RR+3z3IG0CxtIMmxvjPHhsgC7NpvlNHk+B7MW+vEdnrPPc3uvMmmujbDWSoZgFocnumJmMKtVlLLhdZXLmKEY00ZFQlQOBw5vMQD5M0+j5Pqbsef2XceXinZhe/vlw1bW319PpN0sp5l4Rj5T55XizDnwFVdnKLY7GfY40IoZIcTQGBUJEaDkupvIPeEk8k6ast/npXJj9kh7nPUr6/nlA7PtV1a+jZMX+KiamvguejleB3aX2iMp7m/FjBBi6IyaQSJbdvYBkyH82i5MucyrraHamL3u6w5e/Lsdu5Ph+BNzuej/VCWVDAEKypzMrPZjd5m36f2tmBFCDL0xd9kx1BuzxyMGn61u4ptXwgA4shSTbyjhsNkDa/uVyIoZIURqjLmECEO3MXtoWzfvLw4Q3mHefhcfmsWpC/0UDHKsb38rZoQQqTMmE+JgGYbm21fCfLa6ESMGygbHXVbEsZd4sTnkSk6IkUoSYpLa6qN8sDRA4Fuz7Vd+mZNZC/2UTkq+7ZcQIrOkLCEqpVYCFwJBrfUx1mNFwGrgIGArcJXWOpSqGIaS1ppf1rWxfmU90U6znOaw2QVMvqEEZ/bg5qa0oWmpi9IZiiW8MkUIMfRSeYX4ILAYeKjXY78F1mqt/6CU+q31/d+nMIYh0d0W5+P76tn2sTmDnF1o55QFPiomD6ztV2/a0Gzb0L5PH8ZEVqYIIYZWyhKi1vo9pdRBez18MXCG9fUq4B0yPCHWftHBB8sCdIbMtl+VU9zMmF9KTuHQvHUtddHdyRCSW5kihBhawz2G6Nda77K+rgP8/T1RKbUAWABQVVXV39NSJhYx+PSxRr57zex048hSTJ1XysQzB9b2qz+yMkWIzJG2SRWttVZK9dtZQmu9HFgOZnOHYQsMaPyli3V/DtBca24BWnpYNrMW+sn3D31xtKxMESJzDPdKlYBSahyA9XtwmF9/vwxDs/m5Jl79XQ3NtVGUHY6/qohjf1vKdneU2u4IxhB3B5KVKUJkjuG+DHkRmAf8wfr9hWF+/f36/vVmPlvdBEBBuZOZC/38VBLjv/1Sk/Q+JYmSlSlCZI5Ult08jjmBUqKUqgH+ATMRPqmUug3YBlyVqtcfiMPOLmDL2y2UHZXDSdcWEyTG0h8Htk9JMmRlihCZIZWzzNf280dnp+o1B8vusnH+3RU4ssyRhFDbwPcpEUKMPKOm281Q6UmG8Os+Jb0NZL8TIcTIIAmxF20YRHbV0vHt10R21eJ32Kmu8O9OisnsUzL0sWmaayPUfd1Bc20EbWT+1g9CjDRyqWPpb6+VqSdN5Q8TKwe0T8nQxSarWYQYDnKFaOlvr5V4MEB5louj8nIpz3INezKE/leztNRFhz0WIUYzSYiW/e21km77W80ihBg6khAtPXut9DZUe60MVs9qlt5kNYsQQ08SomWo91oZykkQWc0ixPCQSwzLUO61MtSTILKaRYjhIQmxl6HaayUVLb1kNYsQqSe3zCkgkyBCjEySEFNAJkGEGJkkIaaATIIIMTLJJUsKyCSIECOTJMQUkUkQIUYeuWUWQgiLJEQhhLBIQhRCCIskRCGEsEhCFEIIiyREIYSwKD3E+wynglKqHnOXvuFSAjQM4+sNFYl7eEncv2rQWp87xOccdiMiIQ43pdRGrfWUdMeRLIl7eEnco4/cMgshhEUSohBCWCQh9m15ugMYIIl7eEnco4yMIQohhEWuEIUQwiIJUQghLGM6ISqlKpVSbyulvlFKfa2U+hvr8SKl1BtKqS3W7950x9oXpZRdKfWZUupl6/uDlVLrlVI/KqVWK6UyrveYUsqjlHpaKfWdUupbpdTJI+H9Vkr9Z+sz8pVS6nGlVHamvt9KqZVKqaBS6qtej/X5HivTf1g/w5dKqZPSF3n6jemECMSA/0drfRQwA1iolDoK+C2wVms9CVhrfZ+J/gb4ttf3/xv4d631RCAE3JaWqPbvT8BrWusjgOMx48/o91spNR74a2CK1voYwA5cQ+a+3w8CexdJ9/cenwdMsn4tAO4Zphgzk9Zaflm/gBeAOcD3wDjrsXHA9+mOrY9YKzA/2GcBLwMKc/WBw/rzk4HX0x3nXjEXAr9gTeb1ejyj329gPLADKMJsqvwyMDeT32/gIOCrA73HwL3AtX09byz+GutXiLsppQ4CTgTWA36t9S7rj+oAf7ri2o8/An8HGNb3xUBYa92ztV8N5j/kTHIwUA88YN3q36eUcpPh77fWeifwr8B2YBfQDGwi89/v3vp7j3uSfY9M/zlSShIioJTKA54B/pPWuqX3n2nzv82Mqk1SSl0IBLXWm9IdS5IcwEnAPVrrE4F29ro9ztD32wtcjJnQywE3+96SjhiZ+B5nijGfEJVSTsxk+KjW+lnr4YBSapz15+OAYLri68dM4CKl1FbgCczb5j8BHqVUzz45FcDO9ITXrxqgRmu93vr+acwEmenv92zgF611vdY6CjyL+XeQ6e93b/29xzuByl7Py/SfI6XGdEJUSingfuBbrfW/9fqjF4F51tfzMMcWM4bW+r9qrSu01gdhDu6/pbW+HngbuMJ6WibGXQfsUEodbj10NvANGf5+Y94qz1BK5VqfmZ64M/r93kt/7/GLwE3WbPMMoLnXrfXYk+5BzHT+AmZh3jp8CXxu/TofczxuLbAFeBMoSnes+/kZzgBetr4+BNgA/Ag8BWSlO74+4j0B2Gi9588D3pHwfgP/E/gO+Ap4GMjK1PcbeBxzrDOKeVV+W3/vMeZk3BLgJ2Az5kx62n+GdP2SpXtCCGEZ07fMQgjRmyREIYSwSEIUQgiLJEQhhLBIQhRCCIskRJEUpVRbumMQIlUkIQohhEUSohgQpdQZSql3evU2fNRaxYFSaqpS6kOl1BdKqQ1KqXyrf+ADSqnNVmOHM63n3qyUet7q0bdVKbVIKfW31nM+VkoVWc87VCn1mlJqk1LqfaXUEen8+cXo5DjwU4To14nA0UAt8AEwUym1AVgNXK21/kQpVQB0YvZu1FrrY61ktkYpdZh1nmOsc2Vjrvr4e631iUqpfwduwuzssxy4U2u9RSk1HViKuYZbiCEjCVEMxgatdQ2AUupzzB58zcAurfUnANrqHqSUmgX82XrsO6XUNqAnIb6ttW4FWpVSzcBL1uObgeOsbkSnAE9ZF6FgLp0TYkhJQhSD0d3r6zgD/zz1Po/R63vDOqcNs/fgCQM8vxAJkTFEMdS+B8YppaYCWOOHDuB94HrrscOAKuu5B2RdZf6ilLrSOl4ppY5PRfBibJOEKIaU1joCXA38WSn1BfAG5tjgUsCmlNqMOcZ4s9a6u/8z7eN64DbrnF9jNmwVYkhJtxshhLDIFaIQQlgkIQohhEUSohBCWCQhCiGERRKiEEJYJCEKIYRFEqIQQlj+LxcauPnnj0n/AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 333.375x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pg.plot_rm_corr(\n",
    "    data=data_ancova,\n",
    "    x='Income',\n",
    "    y='Scores',\n",
    "    subject='Method',\n",
    "    legend=True,\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Source</th>\n",
       "      <th>SS</th>\n",
       "      <th>DF</th>\n",
       "      <th>F</th>\n",
       "      <th>p-unc</th>\n",
       "      <th>np2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Method</td>\n",
       "      <td>552.284043</td>\n",
       "      <td>3</td>\n",
       "      <td>3.232550</td>\n",
       "      <td>0.036113</td>\n",
       "      <td>0.244288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Income</td>\n",
       "      <td>1573.952434</td>\n",
       "      <td>1</td>\n",
       "      <td>27.637304</td>\n",
       "      <td>0.000011</td>\n",
       "      <td>0.479504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BMI</td>\n",
       "      <td>60.013656</td>\n",
       "      <td>1</td>\n",
       "      <td>1.053790</td>\n",
       "      <td>0.312842</td>\n",
       "      <td>0.033934</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Residual</td>\n",
       "      <td>1708.508657</td>\n",
       "      <td>30</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Source           SS  DF          F     p-unc       np2\n",
       "0    Method   552.284043   3   3.232550  0.036113  0.244288\n",
       "1    Income  1573.952434   1  27.637304  0.000011  0.479504\n",
       "2       BMI    60.013656   1   1.053790  0.312842  0.033934\n",
       "3  Residual  1708.508657  30        NaN       NaN       NaN"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluating the scores with different method and income or BMI as a covariate\n",
    "pg.ancova(data=data_ancova, dv='Scores', covar=['Income', 'BMI'], between='Method')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multiple comparison tests after ANOVA\n",
    "\n",
    "Depending on the scientific goals, ANOVA tests the overall null hypothesis that all the data come from the groups that have identical means. If that is your experimental question, i.e. do the data provide convincing evidence that the means are not all identical, then ANOVA is exactly what you want. If the experimental questions are more focused and can be answered by multiple comparisons tests, you can safely ignore the overall ANOVA results and jump right to the results of multiple comparisons. Note that the multiple comparisons calculations all use the mean square result from the ANOVA table. Consequently, even if you don't care about the value of F and the P value, the multiple comparisons tests still require that the ANOVA table be computed.\n",
    "\n",
    "The goal is to compare every mean with every other mean. To avoid getting fooled by bogus statistically significant conclusions, the significance level is refined to apply to an entire family of comparisons, rather than to each individual comparisons. Using the new definition of statistical significance will reduce the chance of obtaining false reports of statistically significance (type I errors) but at the cost of reducing the power to detect real differences (type II errors).  \n",
    "\n",
    "To choose a multiple comparison test, you must articulate the goals of the study:\n",
    "\n",
    "- compare each group mean to every other group mean (Tukey's/Tukey-Kramer)\n",
    "- each group mean to a control group mean (Dunnett)\n",
    "- only compare a small set of pairs of group means (Bonferroni or Holm)\n",
    "- or compare _constrasts_, e.g. the mean of groups A and B with the mean of groups C, D and E (Scheffe)\n",
    "\n",
    "Correcting for multiple comparisons reduces the risk of a Type I error but at the cost of increasing the risk of Type II error. [Rothman](https://pubmed.ncbi.nlm.nih.gov/2081237/) argues that this trade-off is not worth it and recommends that researchers not correct for multiple comparisons. Instead it is essential to report all the comparisons so that people reading the research can informally correct for multiple comparisons. This recommendation is sensible but not the mainstream.\n",
    "\n",
    "When one or a gew comparisons are clearly defined in advance as being critical, some statisticians advocate not using any correction for multiple comparisons. This approach is called _planned comparisons_. The statistical principles of this approach are fairly straightforward and do not generate much controversy, however that doesn't make it easy to decide what to do in a particular situation. Reasonable statisticians disagree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tukey\n",
    "\n",
    "The results of Tukey (or more generally Tuker-Kramer test which allows for unequal sample size) include both CIs and conclusions about statistical significance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        Multiple Comparison of Means - Tukey HSD, FWER=0.05         \n",
      "====================================================================\n",
      "    group1        group2     meandiff p-adj   lower    upper  reject\n",
      "--------------------------------------------------------------------\n",
      "   Dark Blond  Dark Brunette    -13.8 0.0742 -28.6977  1.0977  False\n",
      "   Dark Blond    Light Blond      8.0 0.4369  -6.8977 22.8977  False\n",
      "   Dark Blond Light Brunette     -8.7  0.416 -24.5014  7.1014  False\n",
      "Dark Brunette    Light Blond     21.8 0.0037   6.9023 36.6977   True\n",
      "Dark Brunette Light Brunette      5.1 0.7697 -10.7014 20.9014  False\n",
      "  Light Blond Light Brunette    -16.7 0.0367 -32.5014 -0.8986   True\n",
      "--------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "mult_comp = sm.stats.multicomp.pairwise_tukeyhsd(\n",
    "    data['Pain threshold'],\n",
    "    data['Hair color'],\n",
    ")\n",
    "# Tukey's Honestly Significant Difference\n",
    "print(mult_comp.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are multiple comparisons CIs, so the 95% confidence level applies to the entire family of comparisons, rather than to each individual interval. There is a 95% change that all three of these CIs include the true population value, leaving a 5% chance that any one or more of the intervals does not include the population value.  \n",
    "If a 95% CI for the difference between two means include zero (H0) then the difference is not statistically significant (P>0.05).  \n",
    "The 5% significance level is a **familywise significance level**, meaning that it applies to the entire set of comparisons. If the null hypothesis is true, there is a 5% chance that one or more of the comparisaons will be statistically significant. The _yes/no_ conclusions apply to each comparison individually. Said differently, there is a 95% chance that none of the comparisons will be declared statistically significant.\n",
    "\n",
    "In summary, for a family of comparisons, if all null hypotheses were true, there is a 5% chance that random sampling would lead to one or more incorrect conclusions that there is a real difference between the mean of two groups.\n",
    "\n",
    "In the previous multiple comparison table, the CI of the difference between the mean of `Dark Brunette` and `Light Blond` does not include zero, so that the difference is statistically significant.  \n",
    "\n",
    "It really doesn't make sense to think about whether one particular comparison is statistically significant (accounting for the rest). Instead, think that the method divides the family of comparisons into two piles, those that are deemed to be statistically significant and those that are not. **Think about the set of differences deemed to be statistically significant, not about individual comparisons.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[29.79538132],\n",
       "       [29.79538132],\n",
       "       [31.60277427],\n",
       "       [29.79538132],\n",
       "       [31.60277427],\n",
       "       [31.60277427]])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.diff(mult_comp.confint) # same CI for all three comparisons (see end of this part)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When computing most multiple comparisons tests, the SD for difference between 2 means is not computed from the SDs of those two groups but rather from the pooled SD of all the groups. Given the assumptions of the analysis, there is a 95% chance that all of these CIs include the true population value, leaving a 5% chance that _any one or more_ of the intervals does not include the population value.  \n",
    "\n",
    "The margin of error of the CI is computed by multiplying the SE of the difference by a critical value that depends on the choice of test, the number of df, the degree of confidence desired and the number of comparisons:\n",
    "$$\n",
    "\\bar{y_i} - \\bar{y_j} \\pm t_{1-\\alpha/2}\\sqrt{\\frac{\\mathrm{SSE}}{n-k}\\left(\\frac{1}{n_i}+\\frac{1}{n_j}\\right)}\n",
    "$$\n",
    "\n",
    "with $$ t = \\frac{\\bar{x_i} - \\bar{x_j}}{\\sqrt{2 . \\mathrm{MS_w} / n}} $$\n",
    "\n",
    "and $$\\mathrm{MSE} = \\frac{\\mathrm{SSE}}{n-k} = s_p^2$$\n",
    "\n",
    "with $s_p^2$ the pooled variance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\WIECKOWS\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\statsmodels\\sandbox\\stats\\multicomp.py:775: UserWarning: FixedFormatter should only be used together with FixedLocator\n",
      "  ax1.set_yticklabels(np.insert(self.groupsunique.astype(str), 0, ''))\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApsAAAGDCAYAAABk/9sPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAy3klEQVR4nO3dfbxmdV3v/9fbERIcGFRwchIYRRAEEWwMJTOU8nibN1lamFLaaGXHOpqinscvPOXJ7sxMzaASM1TSUsHKm1RIRUcHBgdHEM7oIDQ6gODAyMjY8Pn9sdbWi+2+m2HWta5Z+/V8POZx7XX/+V7rs+f67O93rXWlqpAkSZK6cLe+A5AkSdJwWWxKkiSpMxabkiRJ6ozFpiRJkjpjsSlJkqTOWGxKkiSpMxab0iKVpJI8aI7lG5KcssB9bUryM3sqtr1Fkn9P8vy+4xAkWdnm9N3b6QuTvHAPH+O0JB/dg/t7UZI33sV9zPl7vJv7/HySY/fkPrW4WWxKe5m2sNuR5OBp89e1Hzwrd2Of5yT5w9F5VXVsVV1416Jd0LEPTPLGJF9Psi3Jxnb64Pm37ldVPbGq3tF3HPD9vNjevoc3J/nXJIcucNtTklzXdYx7QhtrJXnlXdjHmUm+175X305ycZJHzbddVZ1bVY/f3eNOi2Ff4H8Df5rkp9pYtiX5Ttu+bSP/DtsTx9wFfwb8nzEfUwNmsSntnb4G/NLURJKHAvv3F87uaT9wPw4cCzwBOBB4FPAt4Cd6DG1OaUzi/59PraqlwP2ALcBf9RxPF54P3AQ87y7u57z2vToE+DTwL0myuzub6lHdBU8Drqyq/6qqT1XV0jaeqR7Fg6bmVdXXdzeu3XQ+8NgkPzrm42qgJvE/S0nzeyd3/rB9PvAPoytMH0ZMcnqST0/fUZLVwGnAK9pelAva+d8fGm97gt6X5Lwktya5NMnDZgosyd2SnNH2UH4ryT8lufcs7XgecBjwjKr6clXdUVXXV9UfVNW/tfs7pm3Lt9uh/Z8bOdY5Sd7aDmdvS/KZJD/a9ozenOTKJCeOrL8pyauSfLld/vYk92iX3SvJh5Lc0C77UJL7T3s/X5fkM8BtwANH3+MkD0pyUZKtSW5Mct7Iticn+UK77AtJTp623z9oY781yUenenWT3CPJP7bv47fbbZfP8l5+X1V9F3gf8JCR4/xIkj9re5C3JHlbkv2S3BP4d2DFSE/airaXdCqO1yT57yQHttN/kHb4d7b9jhz3KUkuyw96EI+fdj5enmR9+96cN3U+ZtLG+izgt4Ajk6ya771YwHv1PeAdwI8C9xnJ3VvbPHnGyPHv9DuUpgfyt5JcDVydxl8kuT7JLUkuT3LcLId+InDRfPFlgb/H7bJHJ7k27eUvSX4tyRVtPn8kyeHt/Lck+fNp256f5Hfb9+S7wCXA/5gvPmkhLDalvdPngAPTFGJLgOcA/7g7O6qqs4BzgT9pe1GeOsuqTwPeC9wbeBfwgST7zLDebwNPB34aWAHcDLxlln3+DPDhqto208J2/xcAHwXu2+773CQPHlntF2mGIw8Gbgc+C1zaTr8PeMO03Z5G8yF6BHBUuy00/x++HTicpgDeDrx52ra/AqwGDgCumbbsD9o47wXcn7ZXMU2h/a/Am4D7tPH8a5L7jGz7y8Cvtm3cF3h5O//5wDLg0HbbF7dxzSnJ/sCzafJkyuvb9p4APAj4MeD/q6rv0BQ+m0d60jYDX6A5h7Sv1wA/OTJ90Vz7beM4Efh74EVt/H8DnJ/kR0bi+kWaXu0HAMcDp8/RtGcC22jy8CM0789d0sZyOnBtVd0IbAR+iuZ9fy3wj0nuN8cung6cRFPYPx54DM37sYymbd+aZbuHAl+5q/FPSfIE4N3Az1fVhUmeBrya5j07BPhUuxya4vqX0vbOt39U/AzN7/WUK4AZ/6CUdpXFprT3murd/FmaD4b/6vh4l1TV+9qeoDcA9wAeOcN6LwZeU1XXVdXtwJnAszLzMON9gG/MccxHAkuB11fVjqr6BPAhRi4hAN5fVZe0vTHvB75bVf9QVTuB84ATp+3zzVV1bVXdBLxual9V9a2q+uequq2qbm2X/fS0bc+pqg1V9d/t+zDqezSF6oqq+m5VTfU+PRm4uqre2W73buBKYLSof3tVXVVV24F/oincpvZ5H+BBVbWzbectc7xfH0jybWArTV78KTTD/jRF8u9W1U1t+/4vzR8ps7kI+On2vB1PUyz/dNvz+AjgPxew39XA31TVmjb+d9D8QTCaN2+qqs3t+bhgpO0zeT7N8PdOmsLoObP8wbMQv9i+V9cCPw48A6Cq3tvGc0dVnQdczdyXdPxR2/btNOfrAOBoIFV1RVXNlt8HAbfuZuzT/QJNIf/Eqvp8O+/FbWxXVNV/05yXE5Ic3q6zFTi1Xfc5wIVVtWVkn7e2MUp3mcWmtPd6J02P2OlMG0LvyLVTP1TVHcB1ND2X0x0OvL8dNv02TSG8E5hp+PdbNNcXzmYFTY/THSPzrqHpPZsy+gG5fYbppbO1o93XCmh6A5P8TZJrktwC/CdwUNtzPNO2070CCPD5NMP9vzbShum9oNPb8M2Rn28bifmdND1470myOcmfzFNcPb2qDqL5Q+AlwEVprrs7hOaa3ktGzsuH2/mzuQg4BXg4cDnwMZri+5HA/6uqby1gv4cDL5ta1i4/lDvnzWxtv5M0Nzs9lqYXHuCDbTufPEcb5vJPVXVQVd23qh5XVZe0x3neyLD/t4HjaHrJZzP6e/EJmt7wtwDXJzlr6tKDGdxMU5juCb9D054vjcw7HPjLkXbcRJOfU3n3DuC57c/Ppcm1UQcA395D8WmRs9iU9lJVdQ3NjUJPAv5lhlW+w51vGprrYv9awCG/f2dzO/x2f2DzDOtdS9PDctDIv3tU1Uw9r/8B/I/2WryZbAYOzZ1vxjmMu9aLO3qH9mH8oA0vAx4MnFRVB9IMh0LzAT1l1vepqr5ZVb9eVStoho3fmuaRNJtpPvhHLagNVfW9qnptVT0EOBl4Cgu4MabtRfwXmiL/0cCNNIX3sSPnZFl7Q8ps7bqY5v14BnBRVX25jftJ/GAIfb79Xgu8blou7N/27u6qX6H5zLogyTeBr9IUm3vs0VPtNY1n0xTq92kL9y9x5xyY7k7vXVW9qap+nGZY/Sjg92bZbn27fD4L+T3+BeDpSV46Mu9a4EXT3vv9quridvk/Ak9Lc+31McAHpu3zGOCLC4hPmpfFprR3ewHwuPa6u+kuA57Z9tg9qF13NluAB85zrB9P8sx2WPV3aIZDPzfDem8DXjdyM8Ih7fVjM3knzYfiPyc5Os3NRfdJ8uokTwLW0PR2vSLJPu2ND08F3jNPrHP5rST3b6+lfA3NUDs0PTnbgW+3y35/V3aa5BfygxuKbqYpQu4A/g04KskvJ7l7kmfTFCIfWsA+H5vkoW3v6i00w7R3zLPZ1N3yT6O5fvSKtmf4bOAvkty3XefHkkzdALKF5uaYZVP7qKrbaG4S+S1+UFxeTDM8e1G7znz7PRt4cZKT2pjumeTJSXanR+/5NNdQnjDy7+eBJ027/vWuuCfNebsBIMmv0vRsLkiSR7Rt3YemSPwus5+vf+OHL9OYyWXM/3u8mWZI/KVJfqOd9zbgVWmfl5lkWZJfmNqgqq6juS73ncA/t5cBTLXjHjSXFnxsAfFJ87LYlPZiVbWxqtbOsvgvgB00hcQ7+MHw40z+DnhIO+T2gVnW+SDNTSc30/QyPXOG6xYB/pLm0SkfTXIrTUF60izx305zY8KVNB9stwCfpxm2XFNVO2iKyyfS9KK9FXheVV05R1vm8y6aG3m+SnMzyNTzRd8I7Nce53M0w8G74hHAmiTbaNr/0qr6ajvc/BSantNv0Qy3P6W9GWU+P0pzk9MtNJcjXMQPD3eOuqA9/i0015w+v6o2tMteCfw/4HPtZQL/QdNzSft+vhv4apsDU8PcFwH70JyTqekDaC4xYAH7XQv8Os3Q8s3teqcvoN13kuSRNL3Db2l7kKf+nd/u85fm3sPCtL23f05zk9kWmpt4PrMLuziQpsC+meZSiW/RXjc7gwuAo0fe69ks6Pe4mscjnQqckeSFVfV+4I9pLsG4haaH9onTNnsHTRun59RTaa7hnGnkQtplqVrI6JmkxSzJmTQ3qTx3vnUnWZJNwAur6j/6jkVK89ixh1TV7/R0/MfQDKcfXiPFQJI1wAumXQMq7bZdfQitJEnaA6p57Fgv2qH+lwJ/W9N6napqxpEIaXc5jC5J0iKS5BiaO83vR3P5iNQph9ElSZLUGXs2JUmS1BmLTUmSJHXGG4QmQHtH4mqAe97znj9+9NFH9xyRJEnS/C655JIbq2qubyPzms1Js2rVqlq7drbHJkqSJE2OJJdU1aq51nEYXZI0UTZt2sSmTZv6DkMam6HnvMWmJGmibN68mc2b/fIaLR5Dz3mLTUmSJHXGYlOSJEmdsdiUJElSZyw2JUmS1BmfsylJmignn3xy3yFIYzX0nLdnU5IkSZ2x2JQkTZSNGzeycePGvsOQxmboOW+xKUmaKFu2bGHLli19hyGNzdBz3mJTkiRJnbHYlCRJUmcsNiVJktQZH30kSZooS5Ys6TsEaayGnvMWm5KkiXLSSSf1HYI0VkPPeYfRJUmS1BmLTUnSRLnqqqu46qqr+g5DGpuh57zFpiRpotx4443ceOONfYchjc3Qc95iU5IkSZ2x2JQkSVJnLDYlSZLUGR99JEmaKPvuu2/fIUhjNfSct9iUJE2UVatW9R2CNFZDz3mH0SVJktQZi01J0kS54ooruOKKK/oOQxqboee8w+iSpIly88039x2CNFZDz3l7NiVJktQZi01JkiR1xmJTkiRJnfGaTUnSRNlvv/36DkEaq6HnvMWmJGminHjiiX2HII3V0HPeYXRJkiR1xmJTkjRRNmzYwIYNG/oOQxqboee8w+iSpImydevWvkOQxmroOW/PpiRJkjpjsSlJkqTOWGxKkiSpM16zKUmaKEuXLu07BGmshp7zFpuSpIly/PHH9x2CNFZDz3mH0SVJktQZi01J0kRZv34969ev7zsMaWyGnvMOo0uSJsq2bdv6DkEaq6HnvD2bkiRJ6ozFpiRJkjpjsSlJkqTOeM2mJGmiLFu2rO8QpLEaes5bbEqSJsqxxx7bdwjSWA095x1GlyRJUmcsNiVJE2XdunWsW7eu7zCksRl6zjuMLkmaKNu3b+87BGmshp7z9mxKkiSpMxabkiRJ6ozFpiRJkjrjNZuSpIlyr3vdq+8QpLEaes5bbEqSJsoxxxzTdwjSWA095x1GlyRJUmcsNiVJE2Xt2rWsXbu27zCksRl6zjuMLkmaKDt27Og7BGmshp7z9mxKkiSpM50Vm0m2zTDvxUmeN892pyd58yzLXj3HdpuSXJ7ksvb1abse9a5JclCS3xyZXpnkl7s+riRJ0t5irD2bVfW2qvqHu7CLWYvN1mOr6gTgWcCbpi9MY0+2+SDgN0emVwIWm9K4nHsurFwJd7tb83ruuX1HJEmaZqzFZpIzk7y8/fkRSda3PZF/muRLI6uuSPLhJFcn+ZN2/dcD+7Xrz/eJciBwc7vdyiRfSfIPwJeAQ0d7XZM8K8k57c/nJHlTkouTfDXJs0bW+70kX2hjfm07+/XAEVNtaKd/qp3+3SRL2rZNbfeiu/D2SRp17rmwejVccw1UNa+rV1twDsDBBx/MwQcf3HcY0tgMPef7vEHo7cCvV9Vn20Jy1AnAicDtwFeS/FVVnZHkJW3P5Ww+mSTAA4FfHJl/JPD8qvocQLPKrO4HPBo4GjgfeF+Sx7f7+AkgwPlJHgOcARw3FVOSU4CXV9VT2unVwNaqekSSHwE+k+SjVfW1uQLQBDnllL4j0Gw+9zm4/fY7z7vtNnjBC+Dss/uJSbO78MIFr3rUUUd1F4c0gYae873cIJTkIOCAqvpsO+td01b5eFVtrarvAl8GDl/grh9bVccBDwXenGRpO/+aqUJzAT5QVXdU1ZeB5e28x7f/1gGX0hSiRy5gX48HnpfkMmANcJ+ZtkuyOsnaJGtvuOGGBYYpLXLTC8355kuSejGpjz4a/bTYyS7GWVUbk2wBHgJcD3xn+iojP99jjmNn5PWPqupvRldMsnKeUAL8dlV9ZJ54zwLOAli1alXNta7GbBd6YzRmK1c2Q+fTHX64520vt2bNGgBOOumkniORxmPoOd9Lz2ZVfRu4NcnUu/qcBW76vST7zLdSkvsCDwBm+CQCYEuSY9qbhZ6xgON+BPi1qZ7SJD/WHuNW4ICR9aZPfwT4jamYkxyV5J4LOJ6k+bzudbD//neet//+zXzt1Xbu3MnOnTv7DkMam6HnfJc9m/snuW5k+g3Tlr8AODvJHcBFwNYF7PMsYH2SS6vqtBmWfzLJTmAf4Iyq2jJL7+MZwIeAG4C1wNIZ1vm+qvpokmOAz7bXe24Dntv2oH6mvbnp32nult+Z5IvAOcBf0tyhfml7LekNwNMX0E5J8zmt/S/gNa+Br38dDjusKTRPm+m/BklSX1LVz6htkqVVta39+QzgflX10l6CmSCrVq2qIX9llSTN5+KLLwbg5JNP7jkSaTz25pxPcklVrZprnT6v2Xxykle1MVwDnN5jLJIkSepAb8VmVZ0HnNfX8SVJk2n58uXzryQNyNBzflLvRpckLVJHHHFE3yFIYzX0nO/lbnRJkiQtDhabkqSJcvHFF3//hglpMRh6zltsSpIkqTMWm5IkSeqMxaYkSZI6Y7EpSZKkzvjoI0nSRFmxYkXfIUhjNfSct9iUJE2UlStX9h2CNFZDz3mH0SVJE2Xnzp3s3Lmz7zCksRl6zltsSpImypo1a1izZk3fYUhjM/Sct9iUJElSZyw2JUmS1BmLTUmSJHXGYlOSJEmd8dFHkqSJcuihh/YdgjRWQ895i01J0kQZ+gevNN3Qc95hdEnSRNmxYwc7duzoOwxpbIae8xabkqSJsnbtWtauXdt3GNLYDD3nLTYlSZLUGYtNSZIkdcZiU5IkSZ2x2JQkSVJnfPSRJGmirFy5su8QpLEaes5bbEqSJsqKFSv6DkEaq6HnvMPokqSJsn37drZv3953GNLYDD3nLTYlSRNl3bp1rFu3ru8wpLEZes5bbEqSJKkzFpuSJEnqjMWmJEmSOmOxKUmSpM746CNJ0kQ54ogj+g5BGquh57zFpiRpoixfvrzvEKSxGnrOO4wuSZoo27ZtY9u2bX2HIY3N0HPeYlOSNFHWr1/P+vXr+w5DGpuh57zFpiRJkjpjsSlJkqTOWGxKkiSpMxabkiRJ6oyPPpIkTZQjjzyy7xCksRp6zltsSpImyiGHHNJ3CNJYDT3nHUaXJE2UrVu3snXr1r7DkMZm6DlvsSlJmigbNmxgw4YNfYchjc3Qc95iU5IkSZ2x2JQkSVJnLDYlSZLUGYtNSZIkdcZHH0mSJsrRRx/ddwjSWA095y02JUkT5d73vnffIUhjNfScdxhdkjRRbrrpJm666aa+w5DGZug5b7EpSZooV155JVdeeWXfYUhjM/Sct9iUJElSZyw2JUmS1BmLTUmSJHXGYlOSJEmdmchiM8m2Gea9OMnz5tnu9CRvnmXZq+fYblOSy5Nc1r4+ba5YdkeSU5J8aE/sS5KG7Nhjj+XYY4/tO4xF5dxzYeVKuNvdmtdzz+07osVl6Dm/1zxns6redhd38Wrg/86x/LFVdWOSBwMfBT54F48nSdoNy5Yt6zuEReXcc2H1arjttmb6mmuaaYDTTusvrsVk6Dm/1xSbSc4EtlXVnyV5BPB3wB3Ax4AnVtVx7aorknwYOAJ4f1W9Isnrgf2SXAZsqKq5fn0OBG6e4fgB/gR4IlDAH1bVeUlOAc4EbgSOAy4BnltVleQJwBuB24BP737rJWnxuOGGG3jZy+DrXz+k71AWhc99Dm6//c7zbrsNXvACOPvsfmJabA477Ab+/M/hkEOGmfMTOYy+AG8HXlRVJwA7py07AXg28FDg2UkOraozgO1VdcIcheYnk3wJuAj43zMsf2a774cBPwP8aZL7tctOBH4HeAjwQOAnk9wDOBt4KvDjwI/O1pgkq5OsTbL2hhtumKvdkjR4V199NY961NV9h7FoTC8055uvPe9Rj7qaq68ebs7vNT2bU5IcBBxQVZ9tZ70LeMrIKh+vqq3tul8GDgeuXcCup4bRjwA+nuTCqhq9XvPRwLuraiewJclFwCOAW4DPV9V17TEvA1YC24CvVdXV7fx/BFbPdOCqOgs4C2DVqlW1gFgladAe9jD4jd/oO4rFYeXKZuh8usMPhwsvHHc0i9PFF/cdQbf21p7NuYz+LbaTXSyoq2ojsIWml3Isx5QkqS+vex3sv/+d5+2/fzNf2hP2umKzqr4N3JrkpHbWcxa46feS7DPfSknuCzwAmP533qdohuWXJDkEeAzw+Tl2dSWwsu0pBfilBcYpSdLYnHYanHVW05OZNK9nneXNQdpzJrUHbv8k141Mv2Ha8hcAZye5g+Yay60L2OdZwPokl85y3eYnk+wE9gHOqKot05a/H3gU8EWaG4ReUVXfTHL0TAerqu8mWQ38a5LbaIrVAxYQpyRJY3XaaRaX6k6q9r5LBJMsnbqeMskZwP2q6qU9h7VHrFq1qtauXdt3GJLUm23bmsvlly5d2nMk0njszTmf5JKqWjXXOpPaszmfJyd5FU381wCn9xuOJGlP2Rs/cKW7Yug5v1cWm1V1HnBe33FIkva8LVuaq5iWL1/ecyTSeAw95/fKYlOSNFwbN24EhvvBK0039Jyf8270JHdLcvK4gpEkSdKwzFlsVtUdwFvGFIskSZIGZiHP2fx4kp9vvxtckiRJWrCFFJsvAt4L7EhyS5Jbk9zScVySJEkagHlvEKoqH0QuSRqbE088se8QpLEaes4v6G70JD9H8/WMABdW1Ye6C0mStJjtt99+fYcgjdXQc37eYfQkrwdeCny5/ffSJH/UdWCSpMVp8+bNbN68ue8wpLEZes4vpGfzScAJ7Z3pJHkHsA54VZeBSZIWp02bNgGwYsWKfgORxmToOb+QG4QADhr5eVkHcUiSJGmAFtKz+UfAuiSfBEJz7eYZnUYlSZKkQVjI3ejvTnIh8Ih21iur6pudRiVJkqRBmLXYTPLwabOua19XJFlRVZd2F5YkSZKGYK6ezT+fY1kBj9vDsUiSxKpVq/oOQRqroef8rMVmVT12nIFIkgSw77779h2CNFZDz/l5r9lMsg/wG4w81B34m6r6XodxSZIWqWuvvRaAQw89tOdIpPEYes4v5G70vwb2Ad7aTv9KO++FXQUlSVq8hv7BK0039JxfSLH5iKp62Mj0J5J8sauAJEmSNBwLeaj7ziRHTE0keSCws7uQJEmSNBQL6dn8PeCTSb5K81D3w4Ff7TQqSZIkDcJCHur+8SRHAg9uZ32lqm7vNixJkiQNwULuRv8t4NyqWt9O3yvJC6rqrfNsKknSLjvppJP6DkEaq6Hn/EKu2fz1qvr21ERV3Qz8emcRSZIWtSVLlrBkyZK+w5DGZug5v5Bic0mSTE0kWQIM++mjkqTebNq0iU2bNvUdhjQ2Q8/5hRSbHwbOS3JqklOBd7fzJEna4zZv3szmzZv7DkMam6Hn/ELuRn8lsJrmW4QAPgb8bWcRSZIkaTAWcjf6HcDb2n+SJEnSgi1kGF2SJEnaLRabkiRJ6sycw+jtned/XFUvH1M8kqRF7uSTT+47BGmshp7zc/ZsVtVO4NFjikWSJEkDs5C70dclOR94L/CdqZlV9S+dRSVJWrQ2btwIwBFHHNFzJNJ4DD3nF1Js3gP4FvC4kXkFWGxKkva4LVu2AMP94JWmG3rOL+TRR786jkAkSZI0PLMWm0leUVV/kuSvaHoy76Sq/menkUmSJGmvN1fP5hXt69pxBCJJkqThmbXYrKoL2td3jC8cSdJit2TJkr5DkMZq6Dk/7zWbSQ6h+X70h9DcLARAVT1u1o0kSdpNJ510Ut8hSGM19JxfyDcInUszpP4A4LXAJuALHcYkSZKkgVhIsXmfqvo74HtVdVFV/Rp3fgySJEl7zFVXXcVVV13VdxjS2Aw95xdSbH6vff1GkicnORG4d4cxSZIWsRtvvJEbb7yx7zCksRl6zi/koe5/mGQZ8DLgr4ADgd/tNCpJkiQNwkIe6v6h9setwGO7DUeSJElDMtdD3Wd8mPsUH+ouSZKk+czVszn6MPfXAr/fcSySJLHvvvv2HYI0VkPP+VTN2nn5g5WSdVV14hjiWfRWrVpVa9f6pU2SJGnyJbmkqlbNtc5C7kaHOYbTJUmSpNkstNiUJGksrrjiCq644oq+w5DGZug5P9cNQrfygx7N/ZPcMrUIqKo6sOvgJEmLz80339x3CNJYDT3nZy02q+qAcQYiSZKk4XEYXZIkSZ2x2JQkSVJnFvJ1lZIkjc1+++3XdwjSWA095y02JUkT5cQTfayzFpeh53ynw+hJdia5LMmGJF9M8rIku3TMJCuTfGkB62xvj/XFJBcnefBdi35BsZ2Q5Ekj06ckObnr40oAnHsurFwJd7tb83ruuX1HJEnSD+n6ms3tVXVCVR0L/CzwRHbhay+T7ErP68b2WA8D3gG8+i7ubyFOAJ40Mn0KYLGp7p17LqxeDddcA1XN6+rVFpwahA0bNrBhw4a+w5DGZug5P7Zh9Kq6Pslq4AtJzgQOB94J3LNd5SVVdXGSU4A/AG4GjgYeP7WPJA8E/hlYXVVfmONwB7bbk+R04JnAUmBJkt8HXl5VT2mXvxlYW1XnJNlEU6g+FdgH+IWqujLJPYG/Ao5r558J/Dvwf4D9kjwaeDfwYmBnkucCvw1cCbwNOKyN63eq6jO79MbtTU45pe8IFo/PfQ5uv/3O8267DV7wAjj77H5iWmwuvLDvCAZr69atfYcgjdXQc36s12xW1VeTLAHuC1wP/GxVfTfJkTTF2tR3az4cOK6qvpZkJUA7LP4e4PSq+uIMuz8iyWXAAcD+wEkjyx4OHF9VN7XF7FxurKqHJ/lN4OXAC4HXAJ+oql9LchDweeA/gP8PWFVVL2lj3A/YVlV/1k6/C/iLqvp0ksOAjwDHTD9gW4SvBjjssMOmL5Z+2PRCc775kiT1pM8bhPYB3pzkBGAncNTIss9X1ddGpg8BPgg8s6q+PMv+NlbVCQBJng2cBTyhXfaxqrppgXH9S/t6CU2PKDS9qz+X5OXt9D34QW/lXH4GeEiSqekDkyytqm2jK1XVWW28rFq1au/9Hnp7esZn5cpm6Hy6ww/3PEiSJspYn7PZDoPvpOnV/F1gC/Awmh7NfUdW/c60TbcCXwcevcBDnQ88Zpb9/Td3bvc9pm071TW0kx8U4wF+vr0m9ISqOqyqFvIlpncDHjmy3Y9NLzSl3fK618H++9953v77N/MlSZogYys2kxxCc/3im6uqgGXAN6rqDuBXgCVzbL4DeAbwvCS/vIDDPRrYOMuya2h6G3+kHRI/dQH7+wjw22m7KJNMPaPgVpphe2aZ/ijNtZu0252wgGNJ8zvtNDjrrKYnM2lezzqrmS/t5ZYuXcrSpUv7DkMam6HnfNfD6Pu111HuQ9Oj+E7gDe2ytwL/nOR5wIf54d7MO6mq7yR5CvCxJNuq6vxpq0xdsxma4vSFs+zn2iT/BHwJ+BqwbgHt+APgjcD69tFNXwOeAnwSOKM97h8BFwDvS/I0miLzfwJvSbKe5r3+T5qbiKS77rTTLC41SMcff3zfIUhjNfScT9PJqEmxatWqWrt2bd9hSJIkzSvJJVW1aq51/G50SdJEWb9+PevXr+87DGlshp7zfl2lJGmibNvmfZRaXIae8/ZsSpIkqTMWm5IkSeqMxaYkSZI64zWbkqSJsmzZsr5DkMZq6DlvsSlJmijHHnts3yFIYzX0nHcYXZIkSZ2x2JQkTZR169axbt1CvtxNGoah57zD6JKkibJ9+/a+Q5DGaug5b8+mJEmSOmOxKUmSpM5YbEqSJKkzXrMpSZoo97rXvfoOQRqroee8xaYkaaIcc8wxfYcgjdXQc95hdEmSJHXGYlOSNFHWrl3L2rVr+w5DGpuh57zD6JKkibJjx46+Q5DGaug5b8+mJEmSOmOxKUmSpM5YbEqSJKkzXrMpSZooBx98cN8hSGM19Jy32JQkTZSjjjqq7xCksRp6zjuMLkmSpM5YbEqSJsqaNWtYs2ZN32FIYzP0nHcYXZI0UXbu3Nl3CNJYDT3n7dmUJElSZyw2JUmS1BmLTUmSJHXGazYlSRNl+fLlfYcgjdXQc95iU5I0UY444oi+Q5DGaug57zC6JEmSOmOxKUmaKBdffDEXX3xx32FIYzP0nLfYlCRJUmcsNiVJktQZi01JkiR1xmJTkiRJnfHRR5KkibJixYq+Q5DGaug5b7EpSZooK1eu7DsEaayGnvMOo0uSJsrOnTvZuXNn32FIYzP0nLfYlCRNlDVr1rBmzZq+w5DGZug5b7EpSZKkzlhsSpIkqTMWm5IkSeqMxaYkSZI646OPJEkT5dBDD+07BGmshp7zFpuSpIky9A9eabqh57zD6JKkibJjxw527NjRdxjS2Aw95y02JUkTZe3ataxdu7bvMKSxGXrOW2xKkiSpMxabkiRJ6ozFpiRJkjpjsSlJkqTO+OgjSdJEWblyZd8hSGM19Jy32JQkTZQVK1b0HYI0VkPPeYfRJUkTZfv27Wzfvr3vMKSxGXrOT3SxmWRnksuSbEjyxSQvS7JLMSdZmeRLC1hne3usLya5OMmD22WnJPnQXWnHyHHOTPLyPbEvSRqqdevWsW7dur7DkMZm6Dk/6cPo26vqBIAk9wXeBRwI/P5CNk6yK+3bOHKsFwGvBp6/K8FKkuZ3+eWX8/GPf5ytW7eybNkyTj31VB760If2HZakjkx0z+aoqroeWA28JI2VST6V5NL238nw/Z7ITyU5H/jy6D6SPDDJuiSPmOdwBwI3T5+Z5N5JPpBkfZLPJTm+nX9mkr9PcmGSryb5nyPbvCbJVUk+DTz4rr0LkrR3u/zyy7ngggvYunUrAFu3buWCCy7g8ssv7zkySV2Z9J7NO6mqryZZAtwXuB742ar6bpIjgXcDq9pVHw4cV1VfS7ISoB0Wfw9welV9cYbdH5HkMuAAYH/gpBnWeS2wrqqenuRxwD8AJ7TLjgYe227/lSR/DRwPPKdd5+7ApcAlu/0GSNrjzjnnnL5DWFSuu+46du7cead53/ve9/jgBz/IJZc0/z0uW7YM8Nx05fTTT+87BC0ye1WxOc0+wJuTnADsBI4aWfb5qvrayPQhwAeBZ1bVnXo7R4wOoz8bOAt4wrR1Hg38PEBVfSLJfZIc2C7716q6Hbg9yfXAcuCngPdX1W3tfs+f6cBJVtP02nLYYYfN125J2mtNLzTnmy9p77dXFZtJHkhTWF5Pc93mFuBhNJcDfHdk1e9M23Qr8HWaYnG2YnPU+cDbdzG820d+3skuvLdVdRZNccuqVatqF48r6S6wl2e83vjGN35/CH3UsmXLvn8utmzZAsDy5cvHGZrUmyOOOKLvEDq111yzmeQQ4G3Am6uqgGXAN6rqDuBXgCVzbL4DeAbwvCS/vIDDPRrYOMP8TwGntfGcAtxYVbfMsZ//BJ6eZL8kBwBPXcCxJWmwTj31VPbZZ587zdtnn3049dRTvz+9fPlyC00tKkPP+Unv2dyvvY5yH+C/gXcCb2iXvRX45yTPAz7MD/dm3klVfSfJU4CPJdlWVdOHtKeu2QxNcfrCGXZzJvD3SdYDtzHP3epVdWmS84Av0vTGfmGu9SVp6KbuOp/rbvRt27YBsHTp0l5ilMZt6DmfppNQk2LVqlW1du3avsOQpN5cfPHFAJx88sk9RyKNx96c80kuqapVc62z1wyjS5Ikae9jsSlJkqTOWGxKkiSpMxabkiRJ6syk340uSVpkjjzyyL5DkMZq6DlvsSlJmiiHHHJI3yFIYzX0nHcYXZI0UbZu3TrjtwxJQzX0nLfYlCRNlA0bNrBhw4a+w5DGZug5b7EpSZKkzlhsSpIkqTMWm5IkSeqMxaYkSZI646OPJEkT5eijj+47BGmshp7zFpuSpIly73vfu+8QpLEaes47jC5Jmig33XQTN910U99hSGMz9Jy32JQkTZQrr7ySK6+8su8wpLEZes5bbEqSJKkzFpuSJEnqjMWmJEmSOmOxKUmSpM746CNJ0kQ59thj+w5BGquh57zFpiRpoixbtqzvEKSxGnrOO4wuSZooN9xwAzfccEPfYUhjM/Sct2dTkjRRrr76agAOOeSQniORxmPoOW/PpiRJkjpjsSlJkqTOWGxKkiSpMxabkiRJ6ow3CEmSJsrxxx/fdwjSWA095y02JUkTZenSpX2HII3V0HPeYXRJ0kTZsmULW7Zs6TsMaWyGnvP2bEqSJsrGjRsBWL58ec+RSOMx9Jy3Z1OSJEmdsdiUJElSZyw2JUmS1BmLTUmSJHXGG4QkSRPlxBNP7DsEaayGnvMWm5KkibLffvv1HYI0VkPPeYfRJUkTZfPmzWzevLnvMKSxGXrO27MpSZoomzZtAmDFihX9BiKNydBz3p5NSZIkdcZiU5IkSZ2x2JQkSVJnLDYlSZLUGW8QkiRNlFWrVvUdgjRWQ895i01J0kTZd999+w5BGquh57zD6JKkiXLttddy7bXX9h2GNDZDz3mLTUnSRBn6B6803dBz3mJTkiRJnbHYlCRJUmcsNiVJktQZi01JkiR1xkcfSZImykknndR3CNJYDT3nLTYlSRNlyZIlfYcgjdXQc95hdEnSRNm0aRObNm3qOwxpbIae8xabkqSJsnnzZjZv3tx3GNLYDD3nLTYlSZLUGYtNSZIkdcZiU5IkSZ2x2JQkSVJnfPTRBEiyGljdTm5L8pU+47kLDgZu7DuIHi3m9tv2xWsxt9+2L16Luf3T2374fBukqroLR4tKkrVVtarvOPqymNtv2xdn22Fxt9+2L862w+Ju/+603WF0SZIkdcZiU5IkSZ2x2NSedFbfAfRsMbffti9ei7n9tn3xWszt3+W2e82mJEmSOmPPpiRJkjpjsandkuQeST6f5ItJNiR5bTv/AUnWJPl/Sc5Lsm/fse5pc7T9nCRfS3JZ+++EnkPtTJIlSdYl+VA7PfjzPmWGti+m874pyeVtO9e28+6d5GNJrm5f79V3nF2Ype1nJvmvkXP/pL7j7EqSg5K8L8mVSa5I8qhFdO5navuiOPdJHjzSxsuS3JLkd3b13FtsanfdDjyuqh4GnAA8IckjgT8G/qKqHgTcDLygvxA7M1vbAX6vqk5o/13WV4Bj8FLgipHpxXDep0xvOyye8w7w2LadU48+OQP4eFUdCXy8nR6q6W2HJu+nzv2/9RZZ9/4S+HBVHQ08jOZ3YLGc+5naDovg3FfVV6baCPw4cBvwfnbx3FtsardUY1s7uU/7r4DHAe9r578DePr4o+vWHG1fFJLcH3gy8LftdFgE5x1+uO0C4Gk05xwGfO4XsyTLgMcAfwdQVTuq6tssgnM/R9sXo1OBjVV1Dbt47i02tdva4cTLgOuBjwEbgW9X1X+3q1wH/FhP4XVqeturak276HVJ1if5iyQ/0l+EnXoj8Argjnb6PiyS884Pt33KYjjv0PxR9dEkl7TffAawvKq+0f78TWB5P6F1bqa2A7ykPfd/P9RhZOABwA3A29tLSP42yT1ZHOd+trbD4jj3o54DvLv9eZfOvcWmdltV7Wy71u8P/ARwdL8Rjc/0tic5DngVzXvwCODewCv7i7AbSZ4CXF9Vl/Qdy7jN0fbBn/cRj66qhwNPBH4ryWNGF1bzeJOh9vLP1Pa/Bo6guZzmG8Cf9xdep+4OPBz466o6EfgO04ZNB3zuZ2v7Yjn3ALTX4f8c8N7pyxZy7i02dZe1QwqfBB4FHJTk7u2i+wP/1Vdc4zDS9idU1TfaIfbbgbfTFOBD85PAzyXZBLyHZvj8L1kc5/2H2p7kHxfJeQegqv6rfb2e5rqtnwC2JLkfQPt6fX8RdmemtlfVlvYPzzuAsxnuub8OuG5kBOd9NAXYYjj3M7Z9EZ37KU8ELq2qLe30Lp17i03tliSHJDmo/Xk/4GdpLpr+JPCsdrXnAx/sJcAOzdL2K0d+8UJz/cqX+oqxK1X1qqq6f1WtpBlS+URVncYiOO+ztP25i+G8AyS5Z5IDpn4GHk/T1vNpzjkM9NzP1vapc996BgM991X1TeDaJA9uZ50KfJlFcO5na/tiOfcjfokfDKHDLp77u8+1UJrD/YB3JFlC80fLP1XVh5J8GXhPkj8E1tFeVD0ws7X9E0kOAQJcBry4xxjH7ZUM/7zP5txFct6XA+9vamruDryrqj6c5AvAPyV5AXAN8Is9xtiV2dr+zjSPuipgE/Ci3iLs3m/T5Pq+wFeBX6X9/2/g5x5mbvubFsu5b//A+lnu3MbXswvn3m8QkiRJUmccRpckSVJnLDYlSZLUGYtNSZIkdcZiU5IkSZ2x2JQkSVJnLDYlqUNJdia5LMmXkrw3yf5zrPtzSc6YbfkM6x+U5DdHpk9J8qG7GvMMxzknybPmX/P7669MMuNzB5NcmGTVnotO0qSz2JSkbm2vqhOq6jhgB3M8h7Oqzq+q1+/Cvg8CfnO+laZrnxErSWNhsSlJ4/Mp4EFJnppkTZJ1Sf4jyXKAJKcneXP78zlJ3pTk4iRfnaVn8fXAEW3P6Z+285YmeV+SK5Oc236zEUk2JfnjJJcCv5Dk8Uk+m+TStsd1abve65N8Ocn6JH82cqzHTI8ljT9te20vT/Ls6QEm2S/Je5JckeT9wH576L2UtJfwG4QkaQza745/IvBh4NPAI6uqkrwQeAXwshk2ux/waOBomq+He9+05WcAx1XVCe0xTgFOBI4FNgOfoflO90+363+rqh6e5GDgX4CfqarvJHkl8L+SvIXmq/eObmM7aJ5YngmcADwMOBj4QpL/nBbjbwC3VdUxSY4HLp33zZI0KBabktSt/ZJc1v78KZqv8nwwcF77/cr7Al+bZdsPVNUdNN/FvHyBx/t8VV0H0B53JT8oNs9rXx8JPAT4TNvxuS/wWWAr8F3g79prP0ev/5wplkcD766qncCWJBcBjwDWj2z3GOBNAFW1PsnoMkmLgMWmJHVr+1TP45QkfwW8oarOb3sjz5xl29tHN1vg8Ua32cmd/5//zsi+PlZVvzR94yQ/AZwKPAt4CfC4uxCLJHnNpiT1YBnwX+3Pz78L+7kVOGA3tvsc8JNJHgSQ5J5Jjmqv21xWVf8G/C7N8PhcPgU8O8mSJIfQ9GJ+fto6/wn8cnuc44DjdyNeSXsxezYlafzOBN6b5GbgE8ADdmcnVfWtJJ9pHzP078C/LnC7G5KcDrw7yY+0s/83TfH6wST3oOm9/F/z7Or9wKOALwIFvKKqvplk5cg6fw28PckVwBXAJQtqnKTBSFX1HYMkSZIGymF0SZIkdcZiU5IkSZ2x2JQkSVJnLDYlSZLUGYtNSZIkdcZiU5IkSZ2x2JQkSVJnLDYlSZLUmf8f5n/Wrj9QDxMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = mult_comp.plot_simultaneous(\n",
    "    comparison_name='Light Blond',\n",
    "    xlabel='Pain threshold',\n",
    "    ylabel='Hair color',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dunnett's test\n",
    "\n",
    "If one of the groups in the study is considered the control group, then we should use Dunnett's test as the post-hoc test following the ANOVA. The Dunnett's test **compares the mean of each group to the mean of a control group** without comparing the other groups among themselves. The test makes fewer comparisons than Tukey's method, so it generates narrower CIs and has more power to detect differences.\n",
    "\n",
    "Unfortunately, Dunnett's test is not yet implemented in Python but [only in R](https://rdrr.io/cran/DescTools/man/DunnettTest.html), see also [PMCMRplus](https://rdrr.io/cran/PMCMRplus/). See also more recently another solution using Scipy and SAS description [here](https://github.com/maximtrp/scikit-posthocs/issues/24#issuecomment-1210047534).\n",
    "\n",
    "Yet we can perform the Dunnett's test manually. The Dunnett's test statistic is computed as $$ D = t_\\mathrm{Dunnett}\\sqrt{2\\frac{\\mathrm{MSW}}{n}} $$\n",
    "\n",
    "with $\\mathrm{MSW}$ the mean square of the within group and n the size of the group samples.\n",
    "\n",
    "Using the example of Pain Threshold in Hair colors, with $\\mathrm{MS} = 66.8$, we can look into the [Dunnett's table](https://www.statology.org/dunnetts-table/) to find the critical value for $n=5$, $N=4$ (number of groups including the control group) and $\\alpha=0.05$, $t_\\mathrm{Dunnett} = 3.29$. Thus $D = 65.86$.\n",
    "\n",
    "We now compare the differences in group means to the D value, for each group and the control group, here `Light Blond`. When the absolute difference between onoe group and the control group is higher than the D value, then we can say it reached the significance level of 0.05."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hair color</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Dark Blond</th>\n",
       "      <td>5.0</td>\n",
       "      <td>51.2</td>\n",
       "      <td>9.284396</td>\n",
       "      <td>41.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>63.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dark Brunette</th>\n",
       "      <td>5.0</td>\n",
       "      <td>37.4</td>\n",
       "      <td>8.324662</td>\n",
       "      <td>30.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>51.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Light Blond</th>\n",
       "      <td>5.0</td>\n",
       "      <td>59.2</td>\n",
       "      <td>8.526429</td>\n",
       "      <td>48.0</td>\n",
       "      <td>55.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>71.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Light Brunette</th>\n",
       "      <td>4.0</td>\n",
       "      <td>42.5</td>\n",
       "      <td>5.446712</td>\n",
       "      <td>37.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>41.5</td>\n",
       "      <td>44.0</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                count  mean       std   min   25%   50%   75%   max\n",
       "Hair color                                                         \n",
       "Dark Blond        5.0  51.2  9.284396  41.0  43.0  52.0  57.0  63.0\n",
       "Dark Brunette     5.0  37.4  8.324662  30.0  32.0  35.0  39.0  51.0\n",
       "Light Blond       5.0  59.2  8.526429  48.0  55.0  60.0  62.0  71.0\n",
       "Light Brunette    4.0  42.5  5.446712  37.0  40.0  41.5  44.0  50.0"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.groupby('Hair color')['Pain threshold'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The D critical value computed for that example = 17.01\n"
     ]
    }
   ],
   "source": [
    "D = 3.29 * (2*66.8/5)**.5\n",
    "print(f\"The D critical value computed for that example = {D:.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is the absolute difference between `Dark Blond` and `Light Blond` (control) higher than the D critical value: False\n",
      "Is the absolute difference between `Dark Brunette` and `Light Blond` (control) higher than the D critical value: True\n"
     ]
    }
   ],
   "source": [
    "print(f\"Is the absolute difference between `Dark Blond` and `Light Blond` (control) higher than the D critical value: {abs(51.2 - 59.2) > D}\")\n",
    "print(f\"Is the absolute difference between `Dark Brunette` and `Light Blond` (control) higher than the D critical value: {abs(37.4 - 59.2) > D}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonferroni's test\n",
    "\n",
    "Although Bonferroni's approach (see chapter on P values) can be used to compare all paris of means, it should not be used in this way because Tukey's test has more power. Similarly it should not be used to compare each group against a control because Dunnett's test has more power for that purpose.  \n",
    "\n",
    "Bonferroni's multiple comparisons test should be used **when the experiment design required comparing only selected pairs of means**. By making a limited set of comparisons, you get noarrower CIs and more statistical power to detect differences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.19363526672361778 0.0034822355519029003\n"
     ]
    }
   ],
   "source": [
    "_, pval1 = stats.ttest_ind(\n",
    "    data[data['Hair color'] == 'Light Blond']['Pain threshold'],\n",
    "    data[data['Hair color'] == 'Dark Blond']['Pain threshold'],\n",
    ")\n",
    "\n",
    "_, pval2 = stats.ttest_ind(\n",
    "    data[data['Hair color'] == 'Light Blond']['Pain threshold'],\n",
    "    data[data['Hair color'] == 'Dark Brunette']['Pain threshold'],\n",
    ")\n",
    "\n",
    "print(pval1, pval2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False,  True])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_val_adj = .05 / 2 # 2 comparisons performed\n",
    "pvals = np.array([pval1, pval2])\n",
    "\n",
    "pvals <= p_val_adj"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The individual t-tests that are conducted have to have a p-value less than 0.025 in order to be considered significant, which is the case for the second comparison only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([False,  True]),\n",
       " array([0.38727053, 0.00696447]),\n",
       " 0.025320565519103666,\n",
       " 0.025)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from statsmodels.sandbox.stats.multicomp import multipletests\n",
    "multipletests(pvals, alpha=.05, method='bonferroni')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementing in pingouin\n",
    "\n",
    "The simplest way to adjust your P values is to use the conservative Bonferroni correction method which multiplies the raw P values by the number of tests m (i.e. length of the vector P_values), with $\\widetilde {p}_{{(i)}}= n \\cdot p_{{(i)}}$.\n",
    "\n",
    "Using the `method` argument set to 'bonf', we get a vector of same length but with adjusted P values. This adjustment approach corrects according to the family-wise error rate of at least one false positive (FamilywiseErrorRate (FWER)=Probability (FalsePositive≥1)).\n",
    "\n",
    "Less conservative methods are also included such as Sidak $$\\widetilde {p}_{{(i)}}= 1 - (1 - p_{{(i)}})^{n}$$\n",
    "Holm $$\\widetilde {p}_{{(i)}}=\\max _{{j\\leq i}}\\left\\{(n-j+1)p_{{(j)}}\\right\\}_{{1}}$$\n",
    "Benjamini & Hochberg $$P_{(k)}\\leq \\frac {k}{n}\\alpha$$\n",
    "and Benjamini & Yekutieli $$P_{(k)} \\leq \\frac{k}{n \\cdot c(n)} \\alpha$$, respectively.\n",
    "\n",
    "The first three methods are designed to give strong control of the family-wise error rate. Note that the Holm's method is usually preferred. The 'fdr_bh' and 'fdr_by' methods control the false discovery rate, i.e. the expected proportion of false discoveries amongst the rejected hypotheses. The false discovery rate is a less stringent condition than the family-wise error rate, so these methods are more powerful than the others.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([False,  True]), array([0.38727053, 0.00696447]))"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.multicomp(\n",
    "    pvals, # from last comparisons\n",
    "    method='bonf',\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Contrast</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>Paired</th>\n",
       "      <th>Parametric</th>\n",
       "      <th>T</th>\n",
       "      <th>dof</th>\n",
       "      <th>alternative</th>\n",
       "      <th>p-unc</th>\n",
       "      <th>p-corr</th>\n",
       "      <th>p-adjust</th>\n",
       "      <th>BF10</th>\n",
       "      <th>hedges</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Hair color</td>\n",
       "      <td>Dark Blond</td>\n",
       "      <td>Dark Brunette</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>2.474565</td>\n",
       "      <td>8.0</td>\n",
       "      <td>two-sided</td>\n",
       "      <td>0.038436</td>\n",
       "      <td>0.115308</td>\n",
       "      <td>bonf</td>\n",
       "      <td>2.183</td>\n",
       "      <td>1.413596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Hair color</td>\n",
       "      <td>Dark Blond</td>\n",
       "      <td>Light Blond</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>-1.419100</td>\n",
       "      <td>8.0</td>\n",
       "      <td>two-sided</td>\n",
       "      <td>0.193635</td>\n",
       "      <td>0.580906</td>\n",
       "      <td>bonf</td>\n",
       "      <td>0.875</td>\n",
       "      <td>-0.810661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hair color</td>\n",
       "      <td>Dark Brunette</td>\n",
       "      <td>Light Blond</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>-4.090697</td>\n",
       "      <td>8.0</td>\n",
       "      <td>two-sided</td>\n",
       "      <td>0.003482</td>\n",
       "      <td>0.010447</td>\n",
       "      <td>bonf</td>\n",
       "      <td>10.877</td>\n",
       "      <td>-2.336811</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Contrast              A              B  Paired  Parametric         T  \\\n",
       "0  Hair color     Dark Blond  Dark Brunette   False        True  2.474565   \n",
       "1  Hair color     Dark Blond    Light Blond   False        True -1.419100   \n",
       "2  Hair color  Dark Brunette    Light Blond   False        True -4.090697   \n",
       "\n",
       "   dof alternative     p-unc    p-corr p-adjust    BF10    hedges  \n",
       "0  8.0   two-sided  0.038436  0.115308     bonf   2.183  1.413596  \n",
       "1  8.0   two-sided  0.193635  0.580906     bonf   0.875 -0.810661  \n",
       "2  8.0   two-sided  0.003482  0.010447     bonf  10.877 -2.336811  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx = data['Hair color'] == 'Light Brunette' # we keep only 3 groups\n",
    "\n",
    "pg.pairwise_ttests(\n",
    "    dv='Pain threshold',\n",
    "    between='Hair color',\n",
    "    data=data[~idx],\n",
    "    padjust='bonf',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### allpairtest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Test Multiple Comparison ttest_ind \n",
       "FWER=0.05 method=bonf\n",
       "alphacSidak=0.02, alphacBonf=0.017</caption>\n",
       "<tr>\n",
       "    <th>group1</th>       <th>group2</th>      <th>stat</th>   <th>pval</th>  <th>pval_corr</th> <th>reject</th>\n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Light Blond</td>  <td>Dark Blond</td>   <td>1.4191</td> <td>0.1936</td>  <td>0.5809</td>    <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Light Blond</td> <td>Dark Brunette</td> <td>4.0907</td> <td>0.0035</td>  <td>0.0104</td>    <td>True</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dark Blond</td>  <td>Dark Brunette</td> <td>2.4746</td> <td>0.0384</td>  <td>0.1153</td>    <td>False</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.table.SimpleTable'>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mult_comp2 = sm.stats.multicomp.MultiComparison(\n",
    "    data=data[~idx]['Pain threshold'],\n",
    "    groups=data[~idx]['Hair color'],\n",
    "    group_order=['Light Blond', 'Dark Blond', 'Dark Brunette']\n",
    ")\n",
    "\n",
    "mult_comp2.allpairtest(stats.ttest_ind, method='bonf')[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE - The pairwise.ttest for independent samples in R, as well as the Tukey HSD test in both R and statsmoldels, and JASP posthoc tests, use the **joint variance across all samples**, while the pairwise ttest calculates the joint variance estimate for each pair of sample separately.\n",
    "\n",
    "Pingouin and `stats.ttest_ind` don't pool the standard error for each factor but rather calculate each pairwise T-test completely independent of others:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```Rscipt\n",
    "pairwise.t.test(data$weight, data$group, p.adj = \"bonferroni\")\n",
    "\n",
    "data:  data$weight and data$group \n",
    "\n",
    "     ctrl  trt1 \n",
    "trt1 0.583 -    \n",
    "trt2 0.263 0.013\n",
    "\n",
    "P value adjustment method: bonferroni\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NEW - scikit-posthocs\n",
    "\n",
    "[scikit-posthocs]() is a Python package which provides post hoc tests for pairwise multiple comparisons that are usually performed in statistical data analysis to assess the differences between group levels if a statistically significant result of ANOVA test has been obtained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                Light Blond  Dark Blond  Light Brunette  Dark Brunette\n",
      "Light Blond        1.000000    0.104878        0.855039       0.800231\n",
      "Dark Blond         0.104878    1.000000        0.004474       1.000000\n",
      "Light Brunette     0.855039    0.004474        1.000000       0.048994\n",
      "Dark Brunette      0.800231    1.000000        0.048994       1.000000\n"
     ]
    }
   ],
   "source": [
    "# pip install scikit-posthocs\n",
    "import scikit_posthocs as sp\n",
    "print(\n",
    "    sp.posthoc_ttest(\n",
    "        data,\n",
    "        val_col='Pain threshold',\n",
    "        group_col='Hair color',\n",
    "        pool_sd=True,\n",
    "        p_adjust='b'\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualization of the results of multiple comparisons tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAE2CAYAAAByaKVZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAkLElEQVR4nO3deZgeVZn+8e9N2CFhM3EQhQRkNRMiCTjK8gMdIiIiIAoiApcocVgcYVAC6DStE8UNZBERhQBu4DAqQRHiQljVEEISCBAJISwuhEXCHk14fn/Uaai8vN1d3V3d1fX2/bmuvrrq1Kmq5+1AP33OqTpHEYGZmVl3Vqs6ADMzqwcnDDMzK8QJw8zMCnHCMDOzQlavOoBW1d7e7qcJzAZAW1ubqo5hqHALox84WZhZK3ILox+1tbVVHUKp2tvbAbhp40kVR1K+PZ6aAcDpS2dVHEm5po7aBWi9zwWvfjYbOG5hmJlZIU4YZmZWiBOGmZkV4oRhZmaFOGGYmVkhThhmZlaIE4aZmRXihGFmZoU4YZiZWSFOGGZmVogThpmZFeKEYWZmhThhmJlZIU4YZmZWiBOGmZkV4oRhZmaFOGGYmVkhThhmZlaIE4aZmRXihGFmZoU4YZiZWSEDkjAkrZQ0V9ICSfMk/ZekHt1b0mhJdxeo82K61zxJt0naNh3bU9Iv+vI5cvc5Q9LJZVzLzKwuVh+g+7wYEeMBJI0CfgSMANqKnCypJ3E+kLvXZOA04MieBGtmZq814F1SEbEUOAY4XpnRkm6WNCd9vQNeaRHcLGk6cE/+GpK2lHSnpJ27ud0I4O+NhZI2lvRzSfMl/UHSuFR+hqRLJM2UtFjSp3LnnC7pT5JuAbbt20/BzKx+BqqFsYqIWCxpGDAKWArsHREvSdoa+DEwMVXdCRgbEQ9KGg2QupiuAI6KiHlNLr+VpLnAcGBd4G1N6rQDd0bEAZLeCVwOjE/HtgP2SucvlPRtYBxwaKqzOjAHuKPXPwAzsxqqJGE0WAM4X9J4YCWwTe7YrIh4MLc/ErgaOCgiVml15OS7pA4BLgL2aaizG/ABgIj4naRNJI1Ix34ZEcuB5ZKWAq8Hdgd+FhEvpOtO79UnNTOrsUoShqQtyZLDUrJxjMeAHcm6yF7KVX2+4dRlwMNkv/A7Sxh504FpPQxveW57JT34GUk6Bjhm0003ZcKECT28rZnZ4DbgYxiSRgIXAudHRAAbAH+NiJeBjwLDujj9H8CBwBGSDitwu92AB5qU3wx8JMWzJ/BERDzTxXVuAg6QtI6k4cD7mlWKiIsiYuLkyZOZOHFisypmZrU1UC2MddK4whrACuD7wFnp2AXA/0k6AriO17YqVhERz0vaD/i1pOciorF7qGMMQ2QJ5uNNLnMGcImk+cALdPMUVUTMkXQlMI+sVXR7V/XNzFrRgCSMiOi01RAR95MNKnc4JZXPBGbm6i0Bxqbtp4HXPCGV6qzTyX1euV5EPAUc0KTOGQ37Y3PbU4GpnX0OM7NW5ze9zcysECcMMzMrxAnDzMwKccIwM7NCnDDMzKwQJwwzMyvECcPMzApxwjAzs0KcMMzMrBAnDDMzK8QJw8zMCnHCMDOzQpwwzMysECcMMzMrxAnDzMwKccIwM7NCnDDMzKwQJwwzMyvECcPMzApxwjAzs0KcMMzMrBAnDDMzK0QRUXUMLae9vd0/VLMB0tbWpqpjGCrcwjAzs0JWrzqAVnbTxpOqDqFUezw1A4C2traKIylfe3s7AGf+/PGKIynXlANGAnD60lkVR1K+qaN2qTqEIcctDDMzK8QJw8zMCnHCMDOzQpwwzMysECcMMzMrxAnDzMwKccIwM7NCnDDMzKwQJwwzMyvECcPMzArx1CBmNiR9879Pi2XD1urrZR5qa2sbXUI4teCEYWZD0rJha/V5jq2po3bZoqRwasFdUmZmVogThpmZFeKEYWZmhThhmJlZIU4YZmZWiBOGmVmJJIWkb+T2T5Z0RtreVtJMSXMl3SvposoC7QUnDDOzci0HDpL0uibHzgXOjojxEbE9cN7AhtY3ThhmZuVaAVwEnNjk2KbAox07EXHXQAVVBicMM7PyfQv4iKQNGsrPBn4n6VeSTpS04cCH1nuFEoaklanPbYGkeZL+S1KPko2k0ZLuLlDnxXSveZJuk7RtT+7TG5LGS9o3t7+npHf0933NrDVFxDPA5cCnGsqnAdsD/wvsCfxBUp/nJxkoRX/pv5j63N4C7A28B2grehNJPZmC5IF0rx2By4DT+ni9IsYD++b29wScMMysL74JHA2sly+MiL9ExCUR8X6y7quxFcTWKz3ukoqIpcAxwPHKjJZ0s6Q56esd8Mpf6TdLmg7ck7+GpC0l3Slp525uNwL4ezrnKEnTJf0O+G26/i9y1zxf0lFpe4mk9hTPXZK2S+XrSbpE0qx0//dLWhP4AnBIatmcAnwSODHt7y5ppKT/k3R7+tq1pz83MxtaIuIp4CdkSQMASftIWiNt/wuwCfDnaiLsuV79pR4RiyUNA0YBS4G9I+IlSVsDPwYmpqo7AWMj4kFJoyF7rAy4AjgqIuY1ufxWkuYCw4F1gbflju0EjIuIpyTt2U2YT0TETpKOBU4GPg6cDvwuIj6W+g5nAb8B/huYGBHHpxjXAZ6LiK+n/R+RPdlwi6TNgevJmpVmZl35BnB8bn8ScI6kl9L+ZyLibwMfVu+U0bWzBnC+pPHASmCb3LFZEfFgbn8kcDVwUESs0urIeSAixgNIOoTsaYN90rFfp6xdxE/T9zuAg9L2JGB/SSen/bWBzQtc69+BHSR17I+QtH5EPJevJOkY4JhNN92UCRMmwKRJBUM1s1YREevnth8j+8O3Y/8k4KQq4ipDrxKGpC3JksNSsrGMx4Adybq4XspVfb7h1GXAw8BuNHRTdWI6MK2T661g1S61tRvOXZ6+r+TVzyngAxGxMF9RUr4V08xqwL9FxEtdVYqIi4CL2tvbA+Cmbi5qZlYnPR7DkDQSuBA4PyIC2AD4a0S8DHwUGNbF6f8ADgSOkHRYgdvtBjzQybGHyP7qXyt1L72rwPWuB05QaipIemsqf5asC4xO9mcAJ3TspNaUmdmQUjRhrNPxWC1Zn/8MoD0duwA4UtI8YDte26pYRUQ8D+xHNqi8f5MqW3U8Vgt8iWzsodl1HiEbULo7fb+zwOf4IlkX2vz0Wb6Yym8gSz5zUzfYNcCBHYPeZI/GTZQ0X9I9ZIPiZmZDSqEuqYjotNUQEfcD43JFp6TymcDMXL0lpMfHIuJp4DVPSKU663Ryn0uBSxvKPgt8tknd0bnt2WSPyRIRLwKTm9R/qkk84xr2D2kWl5nZUOE3vc3MrBCv6W1mQ9btJ/6wbxf44TnlBFITbmGYmVkhThhmZlaIE4aZmRXihGFmZoU4YZiZWSFOGGZmVogThpmZFeKEYWY2yEk6VdIiSQslvbuTOmMk/THVuzKt9dOxltDjaaqjuZKaTrdUhBOGmdkgIWnjJmU7AIcCbyFb6uGCtB5Ro6+QrdvzZrKF547OHbsyrWQ6PiK+19v4nDDMzEqSViC9T9IPJd0r6SpJ63ZzzghJkyXNIlvsrdH7gSsiYnlaX2gRsEvDNQS8E7gqFV0GHNDXz9PICcPMrFzbAhdExPbAM8CxzSpJ2k3SpWSLvI0BDo+I05pU3Qx4JLf/aCrL2wR4OiJWdFLnA2m27askvamnH6iDE4aZWbkeiYhb0/YPyNb1WYWkc8mWUZgBbBcRUyLiT/0UzzXA6IgYB/yarPXRK04YZmblim72Ac4CvkW2Yuk0SXt1LOzWxJ+BfKvgjaks70lgQ0mrN9aJiCcjomMF0u8BEwp9iiacMMzMyrW5pLen7cOAWxorRMSSiPgcsANwBdmKnvdJ+kiT600HDk2ri44BtgZmNVwvyBaCOzgVHQlcDSBp01zV/YF7e/vBnDDMzMq1EDhO0r3ARsC3O6sYESsj4tqIOAjYnWzp6cY6C8hWFb0HuA44LiJWAki6VtIbUtVTgJMkLSIb07g4lX9K0oK0iumngKN6+8G8HoaZWblWRMThPT0pIpYCSzs5NhWY2qR839z2YhqenkrlpwKn9jSeZtzCMDOzQtzCMDMrSUQsAcZWHUd/cQvDzMwKcQujH+3x1IyqQ+gX7e3tVYfQb6YcMLLqEPrF1FGv6do26zG3MMzMrBC3MPrR6UtndV+pRjr+Sj3z549XHEn5OloWbW1tFUdSro7W4E0bT6o4kvKV0YL/3K96/UpCFkOfI6gXtzDMzKwQJwwzMyvECcPMzApxwjAzs0KcMMzMrBAnDDMzK8QJw8xskJN0qqRFkhZKencndcZI+mOqd6WkNVP5HpLmSFoh6eBm5xblhGFmNkhI2rhJ2Q7AocBbgH2ACyQNa3L6V4CzI+LNwN+Bo1P5w2RTmv+or/E5YZiZlUTSaEn3SfqhpHvTGtrrdnPOCEmTJc0CTm5S5f3AFRGxPCIeBBbRMI15Wq3vncBVqegy4AB4ZbGm+cDLffpwOGGYmZVtW+CCiNgeeAY4tlklSbtJuhS4AxgDHB4RpzWpuhnwSG7/0VSWtwnwdESs6KJOnzlhmJmV65GIuDVt/wDYrbGCpHOBa4AZwHYRMSUi/jSAMfaK55IyMytXdLMPcBZZ66MN2EfSNGBmWpu70Z+BN+X235jK8p4ENpS0emplNKvTZ25hmJmVa3NJb0/bhwG3NFZI4wqfA3YArgBOAO6T9JEm15sOHCppLUljgK2BVWY2TYnmBqDjKagjgavL+DB5ThhmZuVaCBwn6V5gI+DbnVWMiJURcW1EHATsDjzUpM4C4CfAPcB1wHERsRJA0rWS3pCqngKcJGkR2ZjGxanOzpIeBT4IfEfSgt5+MHdJmZmVa0VEHN7TkyJiKbC0k2NTgalNyvfNbS+m4empVH47WRdVn7mFYWZmhbiFYWZWkohYAoytOo7+4haGmZkV4oRhZmaFOGGYmVkhThhmZlZIvw56S3ouItZvKPsk8EJEXN7FeUcBEyPi+CbHTouIL3Vy3hLgWWAlMAz4XERc3VksvSFpT+DkiNivr9cys2rddskP+3T+HgeMLCmSehjwp6Qi4sI+XuI0oGnCSPaKiCckbUs2T0vpbzuamQ1FA94lJekMSSen7Z0lzZc0V9LXJN2dq/oGSddJul/SV1P9M4F1Uv3u/jQYQTYnfOP91XEvSXdJOiSV7ylpZpqOuGN6YqVj+6SyOcBBJfwYzMxqp+r3MKYBn4iI36dkkDceeCuwHFgo6byImCLp+IgY38U1b0i/6LcEPtTk+EHp2jsCrwNul3RTOvZWskVK/gLcCuwqaTbwXbK55hcBV/b4U5qZtYDKBr0lbQgMj4jfp6LG1aB+GxHLIuIlsjlUtih46b0iYizwr8D5khrHLXYDfpzmcHkMuBHYOR2bFRGPRsTLwFxgNLAd8GBE3J8m+PpBF5/pGEmzv/Od7zB79uyC4ZqZ1cNgfkpqeW57JT1sDUXEA8BjZLNBDtQ9L4qIiZMnT2bixIk9OdXMbNCrLGFExNPAs5LelooOLXjqPyWt0V0lSaPIVrFqnP3xZuAQScMkjQT2oGGq4Ab3AaMlbZX2P1wwTjOzltLfCWNdSY/mvk5qOH408F1Jc4H1gGUFrnkRML+LQe8b0vVuAKakbqe8nwHzgXnA74DPRsTfOrtZ6hI7BvhlGvRuOpukmVl/kXSqpEWSFkp6dyd1xkj6Y6p3paQ1U/laaX9ROj46lW8i6QZJz0k6v0gc/TroHRHdJaQFETEOQNIUYHY671Lg0tx19sttn0I273uz+43uIpb10/cAPpO+8sdnAjNz+8fntq8jG8swM+s3kjaOiKcaynYg64F5C/AG4DeStulYEyPnK8DZEXGFpAvJ/iD/dvr+94h4s6RDU71DgJeAz5NNllhowsSqxzDemx6RvZts8ZD/qTgeM7NekzQ691j+vekx/XW7OWeEpMmSZgEnN6nyfuCKiFgeEQ+SPa25S8M1RPYk51Wp6DLggNz5l6Xtq4B3SVJEPB8Rt5AljkIqTRgRcWVEjI+IsRHx3oh4vMp4zMxKsC1wQURsT7Zu97HNKknaTdKlwB1k462HR8RpTapuBjyS2380leVtAjyd1vNurPPK+en4slS/x6puYZiZtZpHIuLWtP0Dskf5VyHpXOAastkotouIKRHxpwGMsVecMMzMyhXd7AOcBXwLaAOmSdqrY2aJJv4MvCm3/8ZUlvcksKGk1ZvUeeX8dHyDVL/HnDDMzMq1uaS3p+3DgFsaK0TEkoj4HNl7YlcAJwD3SfpIk+tNBw5NTzuNAbam4VWA9DDPDcDBqehIXp1Hb3raJx3/XarfY04YZmblWggcJ+leYCOyJ5WaSjNOXBsRB5E9+NP43hgRsQD4CdmMF9cBx3U8ISXpWklvSFVPAU6StIhsjOLiVH4xsEkqPwmY0nHtNMP3WcBR6dWHLl90rnouKTOzVrMiIg7v6UkRsZRO3vOKiKnA1Cbl++a2F9Pw9FQqfwn4YCfXHd2TGN3CMDOzQtzCMDMrSUQsoeBLcHXkFoaZmRXihGFmZoU4YZiZWSEewzCzoeqhKQeMLLowW6fXKCWSmnDCMLMhqa2tbXTVMdSNu6TMzKwQJwwzMyvECcPMzApxwjAzs0KcMMzMrBAnDDMzK8QJw8zMCnHCMDOzQtTLhZesC+3t7f6hmg2Qtra2zpY2tZK5hWFmZoV4apB+dPrSWd1XqpGpo7LFvFrtc8Grn+2mjSdVHEm59nhqBgBtbW0VR1K+9vb2qkMYctzCMDOzQpwwzMysECcMMzMrxAnDzMwKccIwM7NCnDDMzKwQJwwzMyvECcPMzApxwjAzs0KcMMzMrBAnDDMzK8QJw8zMCnHCMDOzQpwwzMysECcMMzMrxAnDzMwKccIwM7NCnDDMzKwQJwwzMyvECcPMzArpNmFIeq5J2SclHdHNeUdJOr+TY6d1cd4SSXdJmpu+v7+7GPtK0oaSjs3tj5Z0WH/f18ysTnrVwoiICyPi8j7ct9OEkewVEeOBg4FzGw8qU2braEPg2Nz+aMAJw8wsp1e/dCWdIenktL2zpPmpRfA1SXfnqr5B0nWS7pf01VT/TGCdVP+H3dxqBPD3dN5oSQslXQ7cDbwp3/qRdLCkS9P2pZLOlXSbpMWSDs7V+4yk21PM7an4TGCrjs+Q9ndP+ydKGpY+W8d5k3vzczMzq7PVS7jGNOATEfH7lAzyxgNvBZYDCyWdFxFTJB2fWhCduUGSgC2BD+XKtwaOjIg/AGRVOrUpsBuwHTAduErSpHSNXQAB0yXtAUwBxnbEJGlP4OSI2C/tHwMsi4idJa0F3CppRkQ82FUAZmatpE8JQ9KGwPCI+H0q+hGwX67KbyNiWap7D7AF8EiBS+8VEU9I2gr4raSZqfyhjmRRwM8j4mXgHkmvT2WT0tedaX99sgTycDfXmgSMy7VUNkjnOWGY2ZBRRgujK8tz2yt7er+IeEDSY8AOwFLg+cYque21u7i3ct+/HBHfyVeUNLqbUAScEBHXd1kpa4kcs+mmmzJhwgTYvJurmpnVSJ8GjiPiaeBZSW9LRYcWPPWfktborpKkUcAY4KFOqjwmafs0AH5ggfteD3xM0vrp+pulezwLDM/Va9y/HviPjpglbSNpvcaLR8RFETFx8uTJTJw4sUA4Zmb1UeQv/nUlPZrbP6vh+NHAdyW9DNwILCtwzYuA+ZLmRMRHmhy/QdJKYA1gSkQ81kkrYArwC+BxYDZZF1OnImKGpO2B36fxj+eAw1NL5tY0YP8rsqe4VkqaB1wKnEP25NScNLbyOHBAgc9pZtYyuk0YEdFdK2RBRIwDkDSF7Bc3EXEp2S/bjuvsl9s+BTilk/uN7qR8CTC2oewq4KomdY9q2F8/t30OWQJoPKfxMdp3NuyfRvePA5uZtawyxjDeK+nUdK2HgKNKuKaZmQ0yfU4YEXElcGUJsZiZ2SDmuaTMzKwQJwwzMyvECcPMzApxwjAzs0KcMMzMrBAnDDMzK8QJw8zMCnHCMDOzQpwwzMysECcMMzMrxAnDzMwKccIwM7NCnDDMzKwQJwwzMyvECcPMzApxwjAzs0KcMMzMrBAnDDMzK8QJw8zMCnHCMDOzQhQRVcfQctrb2/1DNRsgbW1tqjqGocItjBYwe/bsqkPoN6362fy5rI7cwmgBkmZHxMSq4+gPrfrZ/LmsjtzCMDOzQpwwzMysECeM1nBR1QH0o1b9bP5cVjsewzAzs0LcwjAzs0KcMMzMrBAnDBuUJK1bdQxmtqrVqw7AipN0UlfHI+KsgYqlv0h6B/A9YH1gc0k7ApMj4thqI+sdSdcAnQ4URsT+AxhOv5C0BbB1RPxG0jrA6hHxbNVxWfmcMOplePq+LbAzMD3tvw+YVUlE5TsbeDfps0XEPEl7VBtSn3w9fT8I+BfgB2n/w8BjlURUIkmfAI4BNga2At4IXAi8q8q4rH84YdRIRLQDSLoJ2KnjrzhJZwC/rDC0UkXEI9Iq0wOtrCqWvoqIGwEkfaPhDehrJLXCPBrHAbsAfwSIiPsljao2JOsvHsOop9cD/8jt/yOVtYJHUrdUSFpD0snAvVUHVYL1JG3ZsSNpDLBehfGUZXlEvPLfoqTV6aILzurNLYx6uhyYJelnaf8A4LLqwinVJ4FzgM2APwMzgFqOXzQ4EZgpaTEgYAtgcrUhleJGSacB60jam+zf6pqKY7J+4hf3akrSBGC3tHtTRNxZZTxlkbRrRNzaXVkdSVoL2C7t3hcRy6uMpwySVgOOBiaRJcLrI+K71UZl/cUJo6YkDSPrhnqllRgRD1cXUTkkzYmInborq6PU1TaaVf/NLq8soBJI+s+IOKe7MmsN7pKqIUknAG1kT9msJPvLLoBxVcbVF5LeDrwDGNnw+PAIYFg1UZVH0vfJniKay6uD+EHWvVhnR5J1IeYd1aTMWoATRj39J7BtRDxZdSAlWpPs3YvVefXxYYBngIMriahcE4EdokWa9JI+DBwGjJE0PXdoOPBUNVFZf3PCqKdHgGVVB1Gm9PjpjZJejIiv5o9J+iBwfzWRleZusvcw/lp1ICW5jeyzvA74Rq78WWB+JRFZv/MYRg1Jupjs5b1fAq8MnLbIm94tOYYh6QZgPNkLlvl/s1q/6S3pKxFxSndl1hrcwqinh9PXmumr9iS9B9gX2EzSublDI4AV1URVqjOqDqCf7A00Jof3NCmzFuCEUUO5N77XT/vPVRtRKf4CzAb2B+7IlT9L9g5DrUXEjZJeTzalC8CsiFhaZUx9Iek/yN652EpSvgtqOFD7R6CtOXdJ1ZCkscD3yebvAXgCOCIiFlQXVTkkrUH2h8zmEbGw6njKIulDwNeAmWRPte0OfCYirqoyrt6StAGwEfBlYEru0LMR4UHvFuWEUUOSbgNOj4gb0v6ewJci4h1VxlUGSe8jm7BvzYgYI2k88IUW6OufB+zd0aqQNBL4TUTsWG1kfSdpN7LZaqdJeh0wPCIerDouK5/nkqqn9TqSBUBEzKQ15iWCrK9/F+BpgIiYC4ypLpzSrNbQBfUkLfD/n6Q2svGKU1PRmrw6I6+1GI9h1NNiSZ8n65YCOBxYXGE8ZfpnRCxrmK22FZrB10m6Hvhx2j8EuLbCeMpyIPBWYA5ARPxF0vCuT7G6csKop48B7cBP0/7NqawVLJB0GDBM0tbAp8ie+a+1iPiMpA8Au6aiiyLiZ12dUxP/iIiQFACSWqWla014DMMGlbQ06+nkJrMDvhgRL1UamDWVpp/fmuzx2i+T/eHyo4g4r9LArF84YdTIUFjus9VIepbm/2YCIiJGDHBIpUvTmudnq/11xSFZP3HCqBFJ/6+r4x2ru9WZpG2Ak3ntrK7vrComM8s4Ydigkh4/vZDs5b1XlmaNiDs6PalGUpfbDsCSiHii6nj6StJBwFeAUWQtjJZpOdlr1f6xvqFE0taSpkk6S9IbJf1K0nOS5kma2P0VamFFRHw7ImZFxB0dX1UH1VuS9pe0RNIcSfsCC4DzgbslHVlxeGX4KrB/RGwQESMiYriTRetyC6NGJN1Ctn7CCLLpMj5Nthzm7sD/RMTbqouuHJLOAJYCP2PVSfpq+fZwajF9ENgAuAEYFxGLJY0CfhsR/1ppgH0k6daI2LX7mtYKnDBqRNLciBifthdFxJubHaszSc3eEI6I2HLAgymBpDsj4q1p+658gsgfqytJ55BN2/5zVk3wP+3sHKsvv4dRLy/ntp/p4lhtRUQrvNWdt5qkjci6f19O2x1vJbZCl/AI4AWyp6Q6BK++I2QtxC2MGpH0ArCI7BfOVmmbtL9lRNT+pSlJRzQrr+va15KWkCVzNTlc25aTDU1uYdTL9lUHMAB2zm2vDbyLbNqJWiaMiBhddQz9SdI0mrxnEhGtMvOA5Thh1EhEPFR1DP0tIk7I70vaELiimmisgF/kttcmm1vqLxXFYv3MXVI2qKX1Me6OiG2rjsW6J2k14JZWmGrfXsstDBtUGqY/WY3sJbefVBeR9dDWZC/xWQtywqghSRMaX2aTtF9E/KKzc2rk67ntFcBDEfFoVcGURdL3I+Kj3ZXVTZO5sv6G1/NuWe6SqiFJc8iWZL077X8Y+HTdX9yTNIxsFbq9qo6lbJLmRMROuf1hwF0RsUOFYfWJskVL3hQRD1cdiw2MVngOfCg6GLhc0naSPgEcy6rPwddSRKwke1dhg6pjKYukU9Nf4eMkPZO+niV7m/3qisPrk8j+2vxl1XHYwHELo6bSrK4/Bx4GDoyIF6uNqBySriZbwe3XwPMd5RHxqcqCKoGkL0fEqd3XrBdJlwHnR8TtVcdi/c8Jo0Yk3cWq/cWjgGWkKRkiYlwVcZWpswn5IuKygY6lbJI2A7Zg1Wnbb6ouor6TdB/wZuAhsgTfMVtt7f9btNdywqgRSVt0dXwovKdRV5LOBA4F7uHVaduj7otedfbfpP9bbE1OGDWTBksXRMR2VcdSprR+9+nAU8BZwHfJZuF9ADg6ImZXGF6fSVpINlPt8m4r11BunY+HIuLxquOx/uFB75pJA8MLJW1edSwlmwbcRvaW8B+BS4DXka2+960K4yrLYmCNqoMoSxfrfNzVIut8WBNuYdSQpJvIBoZnserAcG27N1p16nZJ55GNO20G7Aj8llWnAa/lYH6rr/NhzfnFvXr6fNUB9INWnbq9oyvtDmB6lYGU7OWI+BNka5hExGKAiFgqaUW1oVl/ccKooYi4seoY+sF2kuaTpm5P26T92k4B3gpPd3Wi1df5sCbcJVVDkv4NOI9suvM1gWHA83VeS7nVnwBr8kg0ZI9EzyZbXvfJgY+q97zOx9DkFkY9nU/2iOb/AhOBI4BtKo2oj+qeEAr4FdnjtD9K+4cC65LNvXQp8L5qwuqdVl/nw5pzC6OGJM2OiImS5ne8INUK60O3ssa5pPJljWt9mw1WbmHU0wuS1gTmSvoq8FfcbzzYDZO0S0TMApC0M1lXImSz8poNev4lU08fJfu3O57ssdo3AR+oNKKSSJrQpGy/KmIp2ceBiyU9mPr/LwY+IWk94MuVRmZWkLukakrSSIBWe6u2Vadu79AxE29ELKs6ljK06jof1py7pGokrT/QRtayWC0VrQDOi4gvVBpceQ4GrpJ0GNnUIEdQ46nbJR0eET+QdFJDOQARcVYlgZXnLfmdNHXNa1qJ1hrcJVUvJwK7AjtHxMYRsRHwNmBXSSdWG1o50gtghwI/Jetmm1Tzv8bXS9+HN/lav6qg+qqV1/mwzrlLqkYk3QnsHRFPNJSPBGbU+SmpoTB1eyNJn46Ib1YdR1+06jof1pwTRo1Iujsixvb0WB20+ot7zUh6OCJqP4lkK67zYc15DKNe/tHLY4NeRDzUqlO3d6HZW9K10tk6H4ATRgtywqiXHSU1TswH2S+etQc6mLJFxEpJCyVtHhEPVx3PAGiF5v2BwLatus6HrcoJo0YiYlj3tWpvI2CBpJaYuj0NBDdLDALWGeBw+kPHOh9OGEOAE4YNNi01dXtEDK86hv6QW+fjBbIZB1pinQ/rmge9zazHultVr4WndR/SnDBsUGnFqdvNWoW7pGywabmp21tZq63zYV1zwrBBJyIWSRoWESuBaemFRb8cNji11Dof1jUnDBtsPHV7vfx7wzofd+XW+Ti8sqisX/h/RBtsWnbq9hY1TNIuHTte56O1edDbBp1Wnbq9FaUEcQnZRIoCniFb+2MB8N6I+EmF4VnJnDBsUGg2dTvZX6itNHV7y2q1dT6sOY9h2GCRn7r9QQBJWwLflnRiRJxdaXS2iiGwzoc14YRhg8VHaZi6PSIWp4HTGYATxuCSX+ejkbstWpQThg0WazSu8wHZOIakNaoIyDoXEd9J39sbj0n69IAHZAPCT0nZYNGyU7cPQSd1X8XqyC0MGyxaeur2Iab263xYc04YNigMkanbhwqPYbQoJwwz67EhsM6HNeH3MMzMrBAPepuZWSFOGGZmVogThpmZFeKEYWZmhThhmJlZIf8f11XJ1fSKxSIAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pc = sp.posthoc_conover(\n",
    "    data,\n",
    "    val_col='Pain threshold',\n",
    "    group_col='Hair color',\n",
    ")\n",
    "\n",
    "cmap = ['1', '#fb6a4a',  '#08306b',  '#4292c6', '#c6dbef']\n",
    "\n",
    "# significance plots\n",
    "heatmap_args = {\n",
    "    'cmap': cmap,\n",
    "    'linewidths': 0.25,\n",
    "    'linecolor': '0.5',\n",
    "    'clip_on': False,\n",
    "    'square': True,\n",
    "    'cbar_ax_bbox': [0.80, 0.35, 0.04, 0.3]\n",
    "}\n",
    "\n",
    "sp.sign_plot(pc, **heatmap_args);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>**</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    0\n",
       "0   -\n",
       "1  **"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sp.sign_table(pvals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Holm adjustment\n",
    "\n",
    "The Holm multiple comparisons test is a powerful and versatile multiple comparisons test. It can be used to compare all pairs of means, compare each group mean to a control mean, or compare preselected pairs of means. The method starts with a set of P values and then determines which of these P values are low enough for the corresponding comparison to be declared statistically significant. It doesn't adjust the P values themselves but simply determines the conclusion for each one. The threshold for determining significance depends on the rank of the P values. **The threshold for the smallest P value is much smaller than the threshold for the largest P value.**\n",
    "\n",
    "The Holm adjustment sequentially compares the lowest p-value with a Type I error rate that is reduced for each consecutive test. In our case, this means that our first p-value is tested at the .05/3 level (.017), second at the .05/2 level (.025), and third at the .05/1 level (.05). This method is generally considered superior to the Bonferroni adjustment.\n",
    "\n",
    "The Holm's test can only be used to decide which comparisons are statistically significant and which are not, but cannot compute a set of CI.\n",
    "\n",
    "The Holm-Sidak test (different from the Sidak test related to Bonferroni test) is a modified version of Holm's test that is slightly more powerful, especially when there are many comparisons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Test Multiple Comparison ttest_ind \n",
       "FWER=0.05 method=h\n",
       "alphacSidak=0.02, alphacBonf=0.017</caption>\n",
       "<tr>\n",
       "    <th>group1</th>       <th>group2</th>      <th>stat</th>   <th>pval</th>  <th>pval_corr</th> <th>reject</th>\n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Light Blond</td>  <td>Dark Blond</td>   <td>1.4191</td> <td>0.1936</td>  <td>0.1936</td>    <td>False</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Light Blond</td> <td>Dark Brunette</td> <td>4.0907</td> <td>0.0035</td>  <td>0.0104</td>    <td>True</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dark Blond</td>  <td>Dark Brunette</td> <td>2.4746</td> <td>0.0384</td>  <td>0.0769</td>    <td>False</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.table.SimpleTable'>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mult_comp2.allpairtest(stats.ttest_ind, method='h')[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([False,  True, False]), array([0.1936, 0.0105, 0.0768]))"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.multicomp(\n",
    "    [0.1936, 0.0035, 0.0384], # from last comparisons\n",
    "    method='holm',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benjamini and Hochberg (a.k.a. BH step-up procedure)\n",
    "\n",
    "A philosophically different and more powerful adjustment method is that proposed by Benjamini and Hochberg. This method, rather than controlling the false positive rate (FWER) as in the Bonferroni method, controls the false discovery rate (FalseDiscoveryRate (FDR) = Expected (FalsePositive/ (FalsePositive + TruePositive))). In other words, **FDR is the expected proportion of false positives among all positives which rejected the null hypothesis and not among all the tests undertaken**. In the FDR method, P values are ranked in an ascending array and multiplied by m/k where k is the position of a P value in the sorted vector and m is the number of independent tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Contrast</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>Paired</th>\n",
       "      <th>Parametric</th>\n",
       "      <th>T</th>\n",
       "      <th>dof</th>\n",
       "      <th>alternative</th>\n",
       "      <th>p-unc</th>\n",
       "      <th>p-corr</th>\n",
       "      <th>p-adjust</th>\n",
       "      <th>BF10</th>\n",
       "      <th>hedges</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Hair color</td>\n",
       "      <td>Dark Blond</td>\n",
       "      <td>Dark Brunette</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>2.474565</td>\n",
       "      <td>8.0</td>\n",
       "      <td>two-sided</td>\n",
       "      <td>0.038436</td>\n",
       "      <td>0.057654</td>\n",
       "      <td>fdr_bh</td>\n",
       "      <td>2.183</td>\n",
       "      <td>1.413596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Hair color</td>\n",
       "      <td>Dark Blond</td>\n",
       "      <td>Light Blond</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>-1.419100</td>\n",
       "      <td>8.0</td>\n",
       "      <td>two-sided</td>\n",
       "      <td>0.193635</td>\n",
       "      <td>0.193635</td>\n",
       "      <td>fdr_bh</td>\n",
       "      <td>0.875</td>\n",
       "      <td>-0.810661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hair color</td>\n",
       "      <td>Dark Brunette</td>\n",
       "      <td>Light Blond</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>-4.090697</td>\n",
       "      <td>8.0</td>\n",
       "      <td>two-sided</td>\n",
       "      <td>0.003482</td>\n",
       "      <td>0.010447</td>\n",
       "      <td>fdr_bh</td>\n",
       "      <td>10.877</td>\n",
       "      <td>-2.336811</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Contrast              A              B  Paired  Parametric         T  \\\n",
       "0  Hair color     Dark Blond  Dark Brunette   False        True  2.474565   \n",
       "1  Hair color     Dark Blond    Light Blond   False        True -1.419100   \n",
       "2  Hair color  Dark Brunette    Light Blond   False        True -4.090697   \n",
       "\n",
       "   dof alternative     p-unc    p-corr p-adjust    BF10    hedges  \n",
       "0  8.0   two-sided  0.038436  0.057654   fdr_bh   2.183  1.413596  \n",
       "1  8.0   two-sided  0.193635  0.193635   fdr_bh   0.875 -0.810661  \n",
       "2  8.0   two-sided  0.003482  0.010447   fdr_bh  10.877 -2.336811  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.pairwise_ttests(\n",
    "    dv='Pain threshold',\n",
    "    between='Hair color',\n",
    "    data=data[~idx],\n",
    "    padjust='fdr_bh',\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([False,  True, False]), array([0.1936, 0.0105, 0.0576]))"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pg.multicomp(\n",
    "    [0.1936, 0.0035, 0.0384],\n",
    "    method='fdr_bh',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scheffe\n",
    "\n",
    "Scheffe's multiple comparisons test can make more elaborate comparisons than the other tests. To allow for the huge number of possible comparisons, the CIs generated by Scheffe's method are wider than those generated by other methods. Accordingly, the test has less statistical power to detect differences than do other multiple comparisons tests. The precise comparisons do not need to be defined as part of the experimental design.\n",
    "\n",
    "The method is not implemented in Python and should be use with [ScheffeTest in R](https://rdrr.io/cran/DescTools/man/ScheffeTest.html) (see below)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Post-hoc analysis with R\n",
    "\n",
    "A pipeline for post-hoc analysis upon ANOVA in R can be the following:\n",
    "\n",
    "```R\n",
    "# Post-hoc analysis\n",
    "library(\"lsmeans\") # needed for ANOVA analysis and post-hoc tests\n",
    "\n",
    "pairwise.t.test(data$weight, data$group, p.adj = \"none\") # using pooled SD\n",
    "pairwise.t.test(data$weight, data$group, p.adj = \"none\", pool.sd = FALSE) # no adjustment\n",
    "pairwise.t.test(data$weight, data$group, p.adj = \"bonferroni\")\n",
    "pairwise.t.test(data$weight, data$group, p.adj = \"holm\")\n",
    "\n",
    "summary(glht(fit, linfct=mcp(group=\"Dunnett\")))\n",
    "\n",
    "lsmeans(fit, pairwise ~ group) # results of Tukey's test with P values\n",
    "# see also TukeyHSD()\n",
    "\n",
    "# Scheffe's test\n",
    "# some special contrasts\n",
    "y <- c(7,33,26,27,21,6,14,19,6,11,11,18,14,18,19,14,9,12,6,24,7,10,1,10,42,\n",
    "  25,8,28,30,22,17,32,28,6,1,15,9,15,2,37,13,18,23,1,3,4,6,2)\n",
    "group <- factor(c(1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,4,4,4,4,\n",
    "  4,4,4,4,5,5,5,5,5,5,5,5,6,6,6,6,6,6,6,6))\n",
    "\n",
    "r.aov <- aov(y ~ group)\n",
    "\n",
    "ScheffeTest(\n",
    "  r.aov, contrasts=matrix(\n",
    "    c(1,-0.5,-0.5,0,0,0,0,0,0,1,-0.5,-0.5),\n",
    "    ncol=2))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.7 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "48fad77966b7472878a315f3b299a8a5fde8f3976a84f77b0641baae73a364c0"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}