ExpAn-Intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ExpAn: Experiment Analysis\n",
    "\n",
    "ExpAn is a Python library for the statistical analysis of randomised controlled trials (A/B tests). \n",
    "\n",
    "The functions are standalone and can be imported and used from within other projects, and from the command line.\n",
    "\n",
    "The library is Open Source, published under the MIT license here:\n",
    "\n",
    "[github.com/zalando/expan](https://github.com/zalando/expan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assumptions used in analysis \n",
    "\n",
    "1. Sample-size estimation:\n",
    "  * Treatment does not affect variance\n",
    "  * Variance in treatment and control is identical\n",
    "  * Mean of delta is normally distributed\n",
    "2. Welch t-test:\n",
    "  * Mean of means is t-distributed (or normally distributed) \n",
    "3. In general:\n",
    "  * Sample represents underlying population\n",
    "  * Entities are independent\n",
    "\n",
    "## Main user stories\n",
    "\n",
    "As a Data Scientist I want to perform all the basic analysis routines that are typical of a the analysis of an A/B Test (a.k.a. Between-Subject Randomised Control Trial) while retaining access to the raw data so I can perform very also custom analyses in order to answer the questions of stakeholders with little effort.\n",
    "\n",
    "As an analyst from a different department, I want to be able to bring my own data, and easily be able to use this library to perform analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "# Installation\n",
    "\n",
    "To install the library:\n",
    "\n",
    "    $ pip install expan\n",
    "\n",
    "For more information, start with the [README.rst](https://github.com/zalando/expan/blob/master/README.rst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ExpAn Architecture\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## `data.loaders` seperate details of data from the library\n",
    "\n",
    "Loaders read the raw data (e.g. **`csv_fetcher.py`**) and construct an `Experiment` object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## `core.experiment` provides the analysis functionality\n",
    "\n",
    "**`Experiment`** class provides methods for the analysis of experiment data. Currently we support only the **`deltaKPI`** computation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## `core.statistics` contains underlying statistical functions\n",
    "\n",
    "**`Statistics`** class provides methods for statistical computations such as: **`delta`** - computes the difference of means between the samples (x-y) with the confidence intervals, **`bootstrap`** - confidence intervals boostrapping, **`chi-square`** - chi-square homogeneity test on categorical data. \n",
    "\n",
    "The class is used by higher-level `experiment` module, and can be used directly from CLI, by passing in `Array`s."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## `core.binning` implements categorical and numerical binnings\n",
    "\n",
    "The class keeps binning separate from the data.\n",
    "\n",
    "**`Binning`** class has two subclasses `NumericalBinning` and `CategoricalBinning`. `NumericalBinning` groups data into numerical bins defined by numerical intervals. `CategoricalBinning` bins data into categories. This methods provides binning implementations which can be applied to unseed data as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## `core.utils` contains supplied utility methods shared by other classes\n",
    "\n",
    "Currently it supports methods for generating random data for performing an experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `core.version` constructs versioning of the package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Details of Components\n",
    "\n",
    "In this section we will go deep into the details of the components and present some examples of the usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `data.loaders`\n",
    "\n",
    "Data loaders can be written as needed to handle different formats (CSV, Parquet, HDF5, etc) and different internal structures, so long as they return an `ExperimentData` object.\n",
    "\n",
    "Currently, only a simply CSV loader (`data.csv_fetcher`) has been implemented.\n",
    "\n",
    "We'll bypass this and work with synthesized data for now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ExpAn core init: v0.6.1\n"
     ]
    }
   ],
   "source": [
    "import sys, os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from expan.core.util import generate_random_data\n",
    "from os.path import dirname, join, realpath\n",
    "sys.path.insert(0, join(os.getcwd(), 'tests'))\n",
    "\n",
    "np.random.seed(0)\n",
    "data,metadata = generate_random_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>entity</th>\n",
       "      <th>variant</th>\n",
       "      <th>normal_same</th>\n",
       "      <th>normal_shifted</th>\n",
       "      <th>feature</th>\n",
       "      <th>normal_shifted_by_feature</th>\n",
       "      <th>treatment_start_time</th>\n",
       "      <th>normal_unequal_variance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>A</td>\n",
       "      <td>-1.487862</td>\n",
       "      <td>-0.616148</td>\n",
       "      <td>non</td>\n",
       "      <td>-1.088533</td>\n",
       "      <td>7</td>\n",
       "      <td>0.003991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>B</td>\n",
       "      <td>-1.125186</td>\n",
       "      <td>1.783682</td>\n",
       "      <td>has</td>\n",
       "      <td>1.167307</td>\n",
       "      <td>3</td>\n",
       "      <td>-3.565511</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>B</td>\n",
       "      <td>0.388819</td>\n",
       "      <td>1.007539</td>\n",
       "      <td>non</td>\n",
       "      <td>-1.055948</td>\n",
       "      <td>1</td>\n",
       "      <td>6.704536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>A</td>\n",
       "      <td>-1.173873</td>\n",
       "      <td>-0.889252</td>\n",
       "      <td>non</td>\n",
       "      <td>-0.152459</td>\n",
       "      <td>4</td>\n",
       "      <td>1.209668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>A</td>\n",
       "      <td>1.112634</td>\n",
       "      <td>0.434377</td>\n",
       "      <td>has</td>\n",
       "      <td>0.175988</td>\n",
       "      <td>4</td>\n",
       "      <td>0.148207</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   entity variant  normal_same  normal_shifted feature  \\\n",
       "0       0       A    -1.487862       -0.616148     non   \n",
       "1       1       B    -1.125186        1.783682     has   \n",
       "2       2       B     0.388819        1.007539     non   \n",
       "3       3       A    -1.173873       -0.889252     non   \n",
       "4       4       A     1.112634        0.434377     has   \n",
       "\n",
       "   normal_shifted_by_feature  treatment_start_time  normal_unequal_variance  \n",
       "0                  -1.088533                     7                 0.003991  \n",
       "1                   1.167307                     3                -3.565511  \n",
       "2                  -1.055948                     1                 6.704536  \n",
       "3                  -0.152459                     4                 1.209668  \n",
       "4                   0.175988                     4                 0.148207  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'experiment': 'random_data_generation',\n",
       " 'primary_KPI': 'normal_shifted',\n",
       " 'source': 'simulated'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## core.experiment.Experiment\n",
    "\n",
    "This class provides methods for the analysis of experiment data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Constructing `Experiment` \n",
    "\n",
    "The `Experiment` class has the following parameters to construct an experiment:\n",
    "\n",
    "| Parameter | Description |\n",
    "|---|---|\n",
    "| **control_variant_name** | Indicates which of the variants is to be considered as a baseline (a.k.a. control). |\n",
    "| **data** | A data you want to run experiment for. An example of the data structure see above. |\n",
    "| **metadata** | Specifies an experiment name as the mandatory and data source as the optional fields. |\n",
    "| **report_kpi_names** | A list of strings specifying desired kpis to analyse (empty list by default). |\n",
    "| **derived_kpis** | A dictionary structured **{'name': ' `<`name_of_the_kpi`>`, 'formula': `<`formula_to_compute_kpi`>`}** (empty list by default) or a list of such dictionaries if more than 1 derived_kpi is wanted. **`<`name_of_the_kpi`>`**: name of the kpi. **`<`formula_to_compute_kpi`>`**: formula to calculate the desired kpi.|\n",
    "    \n",
    "**NOTE 1**. You should be careful specifying the correct structure to the derived_kpis dictionary including keys **'name'** and **'formula'**. Otherwise, construction of `Experiment` object will raise an exception.\n",
    "\n",
    "**NOTE 2**. Specify the derived kpi name in the **report_kpi_names** if you want to see the results for it too.\n",
    "\n",
    "**NOTE 3**. **data** must contain a column **entity**, a column **variant** and one column each for the kpis you defined.\n",
    "\n",
    "**NOTE 4**. Fields in **metadata** see below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```metadata``` should contain the following fields. Optional fields are wrapped in brackets.\n",
    "\n",
    "| Field | Description |\n",
    "|---|---|\n",
    "|**`experiment`**| Name of the experiment, as known to stakeholders. It can be anything meaningful to you. |\n",
    "|**`[sources]`**| Names of the data sources used in the preparation of this data.|\n",
    "|**`[experiment_id]`**| This uniquely identifies the experiment. Could be a concatenation of the experiment name and the experiment start timestamp. |\n",
    "|**`[retrieval_time]`**| Time that data was fetched from original sources... perhaps this should be a list with entry per source? |\n",
    "|**`[primary_KPI]`**| Overall Evaluation Criteria. |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import expan\n",
    "\n",
    "exp = expan.experiment.Experiment(control_variant_name='A', \n",
    "                                  data=data, \n",
    "                                  metadata=metadata, \n",
    "                                  report_kpi_names=['derived_kpi_one'],\n",
    "                                  derived_kpis=[{'name':'derived_kpi_one','formula':'normal_same/normal_shifted'}])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment \"random_data_generation\" with 1 derived kpis, 1 report kpis, 10000 entities and 2 variants: *A*, B\n"
     ]
    }
   ],
   "source": [
    "print(exp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The wrong input structure (e.g. missing derived_kpis dictionary keys or incorrect kpi keys) will raise an exception."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'Dictionary should have key \"formula\"'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-482dfafa7973>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m                                   \u001b[0mmetadata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmetadata\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m                                   \u001b[0mreport_kpi_names\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'normal_shifted'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'normal_same'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m                                   derived_kpis=[{'name':'derived_kpi_1'}])\n\u001b[0m",
      "\u001b[0;32m/Users/ddedik/Documents/Zalando/new_expan/expan/expan/core/experiment.pyc\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, control_variant_name, data, metadata, report_kpi_names, derived_kpis)\u001b[0m\n\u001b[1;32m     52\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Derived kpis should be an array of dictionaries'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;34m'formula'\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 54\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Dictionary should have key \"formula\"'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     55\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;34m'name'\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     56\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Dictionary should have key \"name\"'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'Dictionary should have key \"formula\"'"
     ]
    }
   ],
   "source": [
    "exp = expan.experiment.Experiment(control_variant_name='A',\n",
    "                                  data=data, \n",
    "                                  metadata=metadata,\n",
    "                                  report_kpi_names=['normal_shifted', 'normal_same'],\n",
    "                                  derived_kpis=[{'name':'derived_kpi_1'}])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our data we have two variants and one them is a baseline or control:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variants: set(['A', 'B'])\n",
      "Control or baseline variant: A\n"
     ]
    }
   ],
   "source": [
    "print('Variants: {}'.format(exp.variant_names))\n",
    "print('Control or baseline variant: {}'.format(exp.control_variant_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Now we can start analysing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Let's start with a single DeltaKPI of orders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "import json\n",
    "\n",
    "warnings.simplefilter('once', UserWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"kpis\": [\n",
      "    {\n",
      "      \"variants\": [\n",
      "        {\n",
      "          \"delta_statistics\": {\n",
      "            \"treatment_mean\": -4.572524000045541, \n",
      "            \"control_sample_size\": 6108, \n",
      "            \"treatment_sample_size\": 6108, \n",
      "            \"delta\": 0.0, \n",
      "            \"confidence_interval\": [\n",
      "              {\n",
      "                \"percentile\": 2.5, \n",
      "                \"value\": -6.445256794169719\n",
      "              }, \n",
      "              {\n",
      "                \"percentile\": 97.5, \n",
      "                \"value\": 6.445256794169717\n",
      "              }\n",
      "            ], \n",
      "            \"control_mean\": -4.572524000045541\n",
      "          }, \n",
      "          \"name\": \"A\"\n",
      "        }, \n",
      "        {\n",
      "          \"delta_statistics\": {\n",
      "            \"treatment_mean\": -0.007948584804651233, \n",
      "            \"control_sample_size\": 6108, \n",
      "            \"treatment_sample_size\": 3892, \n",
      "            \"delta\": 4.564575415240889, \n",
      "            \"confidence_interval\": [\n",
      "              {\n",
      "                \"percentile\": 2.5, \n",
      "                \"value\": -1.1450816040987393\n",
      "              }, \n",
      "              {\n",
      "                \"percentile\": 97.5, \n",
      "                \"value\": 10.274232434580517\n",
      "              }\n",
      "            ], \n",
      "            \"control_mean\": -4.572524000045541\n",
      "          }, \n",
      "          \"name\": \"B\"\n",
      "        }\n",
      "      ], \n",
      "      \"name\": \"derived_kpi_one\"\n",
      "    }\n",
      "  ], \n",
      "  \"control_variant\": \"A\", \n",
      "  \"errors\": [], \n",
      "  \"expan_version\": \"0.6.1\", \n",
      "  \"warnings\": [\n",
      "    \"kpi: derived_kpi_one, variant: B: Sample variances differ too much to assume that population variances are equal.\"\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "res_delta = exp.delta()\n",
    "print(json.dumps(res_delta, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently **`deltaKPI`** supports 4 methods to compute `delta`: `fixed_horizon` (default), `group_sequential`, `bayes_factor` and `bayes_precision`. All methods requires different additional parameters.\n",
    "\n",
    "**`fixed_horizon`** is a default method which has default settings/parameters:\n",
    "\n",
    "| Parameter | Description |\n",
    "|---|---|\n",
    "|`assume_normal=True`| Specifies whether normal distribution assumptions can be made.|\n",
    "|`percentiles=[2.5, 97.5]`| A list of percentile values for confidence bounds.|\n",
    "|`min_observations=20`| Minimum number of observations needed.|\n",
    "|`nruns=10000`| Only used if assume normal is false.|\n",
    "|`relative=False`| If relative==True, then the values will be returned as distances below and above the mean, respectively, rather than the absolute values.|\n",
    "\n",
    "**`group_sequential`**:\n",
    "\n",
    "| Parameter | Description |\n",
    "|---|---|\n",
    "|`spending_function='obrien_fleming`| Currently we support only Obrient-Fleming alpha spending function for the frequentist early stopping decision.|\n",
    "|`estimated_sample_size=None`| Sample size to be achieved towards the end of experiment.|\n",
    "|`alpha=0.05`| Type-I error rate.|\n",
    "|`cap=8`| Upper bound of the adapted z-score.|\n",
    "\n",
    "\n",
    "**`bayes_factor`**:\n",
    "\n",
    "| Parameter | Description |\n",
    "|---|---|\n",
    "|`distribution='normal'`| The name of the KPI distribution model, which assumes a Stan model file with the same name exists.`*`\n",
    "|`num_iters=25000`| Number of iterations of bayes sampling.\n",
    "\n",
    "**`bayes_precision`**:\n",
    "\n",
    "| Parameter | Description |\n",
    "|---|---|\n",
    "|`distribution='normal'`| The name of the KPI distribution model, which assumes a Stan model file with the same name exists.`*`\n",
    "|`posterior_width=0.08`| The stopping criterion, threshold of the posterior width.\n",
    "|`num_iters=25000`| Number of iterations of bayes sampling.\n",
    "\n",
    "`*` currently we support **`normal`** and **`poisson`** models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to change any of the default values, just pass them as parameters to delta:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "delta_freq = exp.delta(method='fixed_horizon', assume_normal=True, percentiles=[2.5, 99.5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "delta_g_s = exp.delta(method='group_sequential', estimated_sample_size=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "delta_bayes_factor = exp.delta(method='bayes_factor', distribution='normal')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"kpis\": [\n",
      "    {\n",
      "      \"variants\": [\n",
      "        {\n",
      "          \"delta_statistics\": {\n",
      "            \"treatment_sample_size\": 6108, \n",
      "            \"control_sample_size\": 6108, \n",
      "            \"treatment_mean\": -4.572524000045541, \n",
      "            \"delta\": 0.0, \n",
      "            \"number_of_iterations\": 25000, \n",
      "            \"confidence_interval\": [\n",
      "              {\n",
      "                \"percentile\": 2.5, \n",
      "                \"value\": -8.396765530428699\n",
      "              }, \n",
      "              {\n",
      "                \"percentile\": 97.5, \n",
      "                \"value\": 3.5794735964894677\n",
      "              }\n",
      "            ], \n",
      "            \"stop\": true, \n",
      "            \"control_mean\": -4.572524000045541\n",
      "          }, \n",
      "          \"name\": \"A\"\n",
      "        }, \n",
      "        {\n",
      "          \"delta_statistics\": {\n",
      "            \"treatment_sample_size\": 3892, \n",
      "            \"control_sample_size\": 6108, \n",
      "            \"treatment_mean\": -0.007948584804651233, \n",
      "            \"delta\": 4.564575415240889, \n",
      "            \"number_of_iterations\": 25000, \n",
      "            \"confidence_interval\": [\n",
      "              {\n",
      "                \"percentile\": 2.5, \n",
      "                \"value\": -2.8506067127900847\n",
      "              }, \n",
      "              {\n",
      "                \"percentile\": 97.5, \n",
      "                \"value\": 8.497896742163277\n",
      "              }\n",
      "            ], \n",
      "            \"stop\": true, \n",
      "            \"control_mean\": -4.572524000045541\n",
      "          }, \n",
      "          \"name\": \"B\"\n",
      "        }\n",
      "      ], \n",
      "      \"name\": \"derived_kpi_one\"\n",
      "    }\n",
      "  ], \n",
      "  \"control_variant\": \"A\", \n",
      "  \"errors\": [], \n",
      "  \"expan_version\": \"0.6.1\", \n",
      "  \"warnings\": []\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(delta_bayes_factor, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting the output\n",
    "\n",
    "| Metric | Description |\n",
    "|---|---|\n",
    "|**`treatment_mean`**| the mean of the treatment group |\n",
    "|**`control_mean`**| the mean of the control group |\n",
    "|**`control_sample_size`**| the sample size for the control group |\n",
    "|**`treatment_sample_size`**| the sample size for the treatment group |\n",
    "|**`delta`**| the difference between the treatment_mean and control_mean |\n",
    "|**`confidence_interval`**| the confidence interval: **`percentile`** - lower percentile and upper percentile; **`value`** - value for each percentile |\n",
    "|**`number_of_iterations`**| number of iterations used for bayes sampling for **`bayes_factor`** and **`bayes_precision`** methods.|\n",
    "|**`stop`**| when **`True`** means the early stopping was done|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Using Bootstrapping:\n",
    "\n",
    "We implement boostrapping for data which is not normally distributed.\n",
    "\n",
    "We switch the flag 'assume_normal' to False for the `delta` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"kpis\": [\n",
      "    {\n",
      "      \"variants\": [\n",
      "        {\n",
      "          \"delta_statistics\": {\n",
      "            \"treatment_mean\": -4.572524000045541, \n",
      "            \"control_sample_size\": 6108, \n",
      "            \"treatment_sample_size\": 6108, \n",
      "            \"delta\": 0.0, \n",
      "            \"confidence_interval\": [\n",
      "              {\n",
      "                \"percentile\": 2.5, \n",
      "                \"value\": -6.451719231491042\n",
      "              }, \n",
      "              {\n",
      "                \"percentile\": 97.5, \n",
      "                \"value\": 6.412118814781721\n",
      "              }\n",
      "            ], \n",
      "            \"control_mean\": -4.572524000045541\n",
      "          }, \n",
      "          \"name\": \"A\"\n",
      "        }, \n",
      "        {\n",
      "          \"delta_statistics\": {\n",
      "            \"treatment_mean\": -0.007948584804651233, \n",
      "            \"control_sample_size\": 6108, \n",
      "            \"treatment_sample_size\": 3892, \n",
      "            \"delta\": 4.564575415240889, \n",
      "            \"confidence_interval\": [\n",
      "              {\n",
      "                \"percentile\": 2.5, \n",
      "                \"value\": 0.06194124312058608\n",
      "              }, \n",
      "              {\n",
      "                \"percentile\": 97.5, \n",
      "                \"value\": 9.032079819101977\n",
      "              }\n",
      "            ], \n",
      "            \"control_mean\": -4.572524000045541\n",
      "          }, \n",
      "          \"name\": \"B\"\n",
      "        }\n",
      "      ], \n",
      "      \"name\": \"derived_kpi_one\"\n",
      "    }\n",
      "  ], \n",
      "  \"control_variant\": \"A\", \n",
      "  \"errors\": [], \n",
      "  \"expan_version\": \"0.6.1\", \n",
      "  \"warnings\": []\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "res_delta = exp.delta(assume_normal=False)\n",
    "print(json.dumps(res_delta, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You may not notice here: bootstrapping takes considerably longer time than assuming the normality before running experiment. If we do not have an explicit reason to use it, it is almost always better to leave it off."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## core.binning\n",
    "\n",
    "Defines a Binning class that represents a particular binning of a data, such that the same binning can then be applied to unseen data.\n",
    "\n",
    "Numerical and Categorical Binnings are defined.\n",
    "\n",
    "Tries to handle skewed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = exp.data[exp.data.variant == 'A']\n",
    "b = exp.data[exp.data.variant == 'B']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Now we create the binning\n",
    "\n",
    "This simply determines the thresholds appropriate for creating the requested number of bins..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NumericalBinning with 8 bins:\n",
      " 0: [0.0,1.0)\n",
      " 1: [1.0,2.0)\n",
      " 2: [2.0,3.0)\n",
      " 3: [3.0,4.0)\n",
      " 4: [4.0,5.0)\n",
      " 5: [5.0,6.0)\n",
      " 6: [6.0,8.0)\n",
      " 7: [8.0,9.0]\n"
     ]
    }
   ],
   "source": [
    "import expan.core.binning as binning\n",
    "\n",
    "bins = binning.create_binning(a.loc[:,'treatment_start_time'])\n",
    "\n",
    "print(bins)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### We can *apply* this binning to the same data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[6.0,8.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[4.0,5.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[4.0,5.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[3.0,4.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[2.0,3.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[8.0,9.0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[0.0,1.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[0.0,1.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[6.0,8.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>[4.0,5.0)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           0\n",
       "0  [6.0,8.0)\n",
       "1  [4.0,5.0)\n",
       "2  [4.0,5.0)\n",
       "3  [3.0,4.0)\n",
       "4  [2.0,3.0)\n",
       "5  [8.0,9.0]\n",
       "6  [0.0,1.0)\n",
       "7  [0.0,1.0)\n",
       "8  [6.0,8.0)\n",
       "9  [4.0,5.0)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_bins = bins.label(a.treatment_start_time)\n",
    "\n",
    "pd.DataFrame(a_bins).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### And we can *apply* it to different data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[3.0,4.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[1.0,2.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[4.0,5.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[8.0,9.0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[8.0,9.0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[5.0,6.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[6.0,8.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[5.0,6.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[8.0,9.0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>[8.0,9.0]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           0\n",
       "0  [3.0,4.0)\n",
       "1  [1.0,2.0)\n",
       "2  [4.0,5.0)\n",
       "3  [8.0,9.0]\n",
       "4  [8.0,9.0]\n",
       "5  [5.0,6.0)\n",
       "6  [6.0,8.0)\n",
       "7  [5.0,6.0)\n",
       "8  [8.0,9.0]\n",
       "9  [8.0,9.0]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b_bins = bins.label(b.treatment_start_time)\n",
    "\n",
    "pd.DataFrame(b_bins).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note that there is a hidden 'catch-all' bin...\n",
    "\n",
    "This is implemented as the last entry in the arrays, making indexing very easy: an unknown bin is always -1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.,  2.,  3.,  4.,  5.,  6.,  8.,  9.])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bins.uppers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  1.,   2.,   3.,   4.,   5.,   6.,   8.,   9.,  nan])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bins._uppers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Bin labels can be arbitrarily formatted:\n",
    "\n",
    "Without running the binning algorithm on the data again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NumericalBinning with 8 bins:\n",
      " 0: [0.0,1.0)\n",
      " 1: [1.0,2.0)\n",
      " 2: [2.0,3.0)\n",
      " 3: [3.0,4.0)\n",
      " 4: [4.0,5.0)\n",
      " 5: [5.0,6.0)\n",
      " 6: [6.0,8.0)\n",
      " 7: [8.0,9.0]\n"
     ]
    }
   ],
   "source": [
    "print(bins)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NumericalBinning with 8 bins:\n",
      " 0: 0.0<=x<1.0\n",
      " 1: 1.0<=x<2.0\n",
      " 2: 2.0<=x<3.0\n",
      " 3: 3.0<=x<4.0\n",
      " 4: 4.0<=x<5.0\n",
      " 5: 5.0<=x<6.0\n",
      " 6: 6.0<=x<8.0\n",
      " 7: 8.0<=x<=9.0\n"
     ]
    }
   ],
   "source": [
    "print(bins.__str__('{conditions}'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NumericalBinning with 8 bins:\n",
      " 0: A\n",
      " 1: B\n",
      " 2: C\n",
      " 3: D\n",
      " 4: E\n",
      " 5: F\n",
      " 6: G\n",
      " 7: H\n"
     ]
    }
   ],
   "source": [
    "print(bins.__str__('{iter.uppercase}'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NumericalBinning with 8 bins:\n",
      " 0: A: From 0.00 \t To 1.00\n",
      " 1: B: From 1.00 \t To 2.00\n",
      " 2: C: From 2.00 \t To 3.00\n",
      " 3: D: From 3.00 \t To 4.00\n",
      " 4: E: From 4.00 \t To 5.00\n",
      " 5: F: From 5.00 \t To 6.00\n",
      " 6: G: From 6.00 \t To 8.00\n",
      " 7: H: From 8.00 \t To 9.00\n"
     ]
    }
   ],
   "source": [
    "print(bins.__str__('{iter.uppercase}: From {lo:.2f} \\t To {up:.2f}'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## core.statistics\n",
    "\n",
    "Here the underlying statistical functions are implemented. These are used by the higher-level `experiment` module, and can indeed be used directly by passing in NumPy `Array`s.\n",
    "\n",
    "The more interesting functions are:\n",
    "\n",
    "### `bootstrap`\n",
    "\n",
    "Bootstraps the Confidence Intervals for a particular function comparing two samples. NaNs are ignored (discarded before calculation).\n",
    "\n",
    "This function, as well as others such as `normal_sample_difference`, and `delta`, take as input a list of percentiles, and return the values corresponding to those percentiles. This implementation is very general, allowing us to use the same functions for one-sided as well as two-sided tests, as well as more exactly recreating an output distribution (e.g. if we want to graphically depict more than 95% confidence intervals).\n",
    "\n",
    "### `delta`\n",
    "\n",
    "Uses either bootstrap or standard normal assumptions to compute the difference between two arrays."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## core.utils\n",
    "\n",
    "A `commons` module: contains supplied utility methods shared by other classes.\n",
    "\n",
    "Currently it supports methods for generating random data for performing an experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# That's it! Try it out for yourself:\n",
    "\n",
    "\n",
    "[github.com/zalando/expan](https://github.com/zalando/expan)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}