quantile-transformation.ipynb

{
  "cells": [
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Quantile Transformation\nCopyright (c) Microsoft Corporation. All rights reserved.\nLicensed under the MIT License.\n\nDataPrep has the ability to perform quantile transformation to a numeric column. This transformation can transform the data into a normal or uniform distribution. Values bigger than the learnt boundaries will simply be clipped to the learnt boundaries when applying quantile transformation.\n\nLet's load a sample of the median income of california households in different suburbs from the 1990 census data. From the data profile, we can see that the minimum value and maximum value is 0.9946 and 15 respectively."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "!pip install azureml",
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Requirement already satisfied: azureml in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (0.2.7)\nRequirement already satisfied: python-dateutil in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (from azureml) (2.7.3)\nRequirement already satisfied: requests in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (from azureml) (2.19.1)\nRequirement already satisfied: pandas in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (from azureml) (0.22.0)\nRequirement already satisfied: six>=1.5 in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (from python-dateutil->azureml) (1.11.0)\nRequirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (from requests->azureml) (3.0.4)\nRequirement already satisfied: certifi>=2017.4.17 in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (from requests->azureml) (2018.8.24)\nRequirement already satisfied: urllib3<1.24,>=1.21.1 in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (from requests->azureml) (1.23)\nRequirement already satisfied: idna<2.8,>=2.5 in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (from requests->azureml) (2.7)\nRequirement already satisfied: pytz>=2011k in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (from pandas->azureml) (2018.5)\nRequirement already satisfied: numpy>=1.9.0 in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (from pandas->azureml) (1.14.5)\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "import azureml.dataprep as dprep\n\ndf = dprep.read_csv(path='./data/median_income.csv').set_column_types(type_conversions={\n    'median_income': dprep.TypeConverter(dprep.FieldType.DECIMAL)\n})\ndf.get_profile()",
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 2,
          "data": {
            "text/html": "<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Type</th>\n      <th>Min</th>\n      <th>Max</th>\n      <th>Count</th>\n      <th>Missing Count</th>\n      <th>Error Count</th>\n      <th>Lower Quartile</th>\n      <th>Median</th>\n      <th>Upper Quartile</th>\n      <th>Standard Deviation</th>\n      <th>Mean</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>median_income</th>\n      <td>FieldType.DECIMAL</td>\n      <td>0.9946</td>\n      <td>15.0</td>\n      <td>250.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>2.6907</td>\n      <td>3.6307</td>\n      <td>4.77335</td>\n      <td>2.026679</td>\n      <td>4.007843</td>\n    </tr>\n  </tbody>\n</table>",
            "text/plain": "ColumnProfile\n    name: median_income\n    type: FieldType.DECIMAL\n\n    min: 0.9946\n    max: 15.0\n    count: 250.0\n    missing_count: 0.0\n    error_count: 0.0\n\n    lower_quartile: 2.6907\n    median: 3.6307\n    upper_quartile: 4.773350000000001\n    std: 2.026679472255346\n    mean: 4.007842799999996"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Let's now apply quantile transformation to `median_income` and see how that affects the data. We will apply quantile transformation twice, one that maps the data to a Uniform(0, 1) distribution, one that maps it to a Normal(0, 1) distribution.\n\nFrom the data profile, we can see that the min and max of the uniform median income is strictly between 0 and 1 and the mean and standard deviation of the normal median income is close to 1 and 0 respectively.\n\n*note: for normal distribution, we will clip the values at the ends as the 0th percentile and the 100th percentile are -Inf and Inf respectively.*"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "df = df.quantile_transform(source_column='median_income', new_column='median_income_uniform', quantiles_count=5)\ndf = df.quantile_transform(source_column='median_income', new_column='median_income_normal', \n                           quantiles_count=5, output_distribution=\"Normal\")\ndf.get_profile()",
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 3,
          "data": {
            "text/html": "<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Type</th>\n      <th>Min</th>\n      <th>Max</th>\n      <th>Count</th>\n      <th>Missing Count</th>\n      <th>Error Count</th>\n      <th>Lower Quartile</th>\n      <th>Median</th>\n      <th>Upper Quartile</th>\n      <th>Standard Deviation</th>\n      <th>Mean</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>median_income</th>\n      <td>FieldType.DECIMAL</td>\n      <td>0.994600</td>\n      <td>15.000000</td>\n      <td>250.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>2.690700</td>\n      <td>3.630700</td>\n      <td>4.773350</td>\n      <td>2.026679</td>\n      <td>4.007843</td>\n    </tr>\n    <tr>\n      <th>median_income_normal</th>\n      <td>FieldType.DECIMAL</td>\n      <td>-7.941345</td>\n      <td>7.941444</td>\n      <td>250.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>-0.671590</td>\n      <td>-0.000337</td>\n      <td>0.667810</td>\n      <td>1.021506</td>\n      <td>-0.060922</td>\n    </tr>\n    <tr>\n      <th>median_income_uniform</th>\n      <td>FieldType.DECIMAL</td>\n      <td>0.000000</td>\n      <td>1.000000</td>\n      <td>250.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.250934</td>\n      <td>0.499866</td>\n      <td>0.747861</td>\n      <td>0.252830</td>\n      <td>0.484762</td>\n    </tr>\n  </tbody>\n</table>",
            "text/plain": "ColumnProfile\n    name: median_income\n    type: FieldType.DECIMAL\n\n    min: 0.9946\n    max: 15.0\n    count: 250.0\n    missing_count: 0.0\n    error_count: 0.0\n\n    lower_quartile: 2.6907\n    median: 3.6307\n    upper_quartile: 4.773350000000001\n    std: 2.026679472255346\n    mean: 4.007842799999996\n\nColumnProfile\n    name: median_income_normal\n    type: FieldType.DECIMAL\n\n    min: -7.941345326170997\n    max: 7.94144448741598\n    count: 250.0\n    missing_count: 0.0\n    error_count: 0.0\n\n    lower_quartile: -0.6715898847385642\n    median: -0.00033696356609359737\n    upper_quartile: 0.6678101623094225\n    std: 1.021505801777812\n    mean: -0.06092218967843191\n\nColumnProfile\n    name: median_income_uniform\n    type: FieldType.DECIMAL\n\n    min: 0.0\n    max: 1.0\n    count: 250.0\n    missing_count: 0.0\n    error_count: 0.0\n\n    lower_quartile: 0.25093366375866033\n    median: 0.4998655717951272\n    upper_quartile: 0.7478610044020887\n    std: 0.25283034846216024\n    mean: 0.4847624122367444"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Let's now save the dataflow which we will later load in the operationalization notebook."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from tempfile import mkdtemp\nfrom os import path\n\ntmp_dir = mkdtemp()\npackage_path = path.join(tmp_dir, 'quantile_transform.dprep')\npackage = dprep.Package(arg=df)\npackage.save(package_path)\nprint('Package saved to: \"{}\"'.format(package_path))",
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Package saved to: \"/tmp/tmp29cvg68a/quantile_transform.dprep\"\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "",
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python36",
      "display_name": "Python 3.6",
      "language": "python"
    },
    "language_info": {
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "name": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6",
      "file_extension": ".py",
      "codemirror_mode": {
        "version": 3,
        "name": "ipython"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}