Decision trees.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decision trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decision trees are a supervised machine learning method for classification\n",
    "\n",
    "+ Supervised - use labeled data for training\n",
    "+ Classification - predict the category for a given input\n",
    "\n",
    "To explore decision trees, we'll use the Python scikit-learn machine learning (sklearn) package and the famous iris dataset that was described in R.A. Fisher's classic 1936 paper “The Use of Multiple Measurements in Taxonomic Problems”. The goal is straightforward - given four measurements of a flower (sepal width, sepal length, petal width and petal length), predict which of three species (setosa, versicolor, virginica) the flower belongs to.\n",
    "\n",
    "We'll dive right in first and then go back and examine the steps in more detail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the data set and sklearn tree package, load the data, create the decision tree classifier and train the classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import tree\n",
    "from sklearn.datasets import load_iris\n",
    "iris = load_iris()\n",
    "clf = tree.DecisionTreeClassifier()\n",
    "clf = clf.fit(iris.data, iris.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import graphviz, create a DOT representation of the decision tree, render using Graphviz and display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import graphviz \n",
    "dot_data = tree.export_graphviz(clf, out_file=None, \n",
    "                         feature_names=iris.feature_names,  \n",
    "                         class_names=iris.target_names,  \n",
    "                         filled=True, rounded=True,  \n",
    "                         special_characters=True)  \n",
    "graph = graphviz.Source(dot_data)  \n",
    "graph "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We start by importing sklearn's tree module. Since the entire sklearn package is very large, we imported just the functionality we needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then import and load a data set. In this case, we used one of the data sets that is provided with sklearn, but more generally you may need to read in and process one or more files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "iris = load_iris()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the type and content of iris."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "type(iris)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sklearn.utils.Bunch objects aggregate multiple arrays and strings. This is not necessary, but is a convenient way to package everything needed for a machine learning project. Let's now look into the contents of iris"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "iris"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're primarily concerned with the data and target arrays. The data array contains the measurements (sepal length, sepal width, petal length and petal width), with one row for each sample. The target array contains the data lables (0 = setosa, 1 = versicolor, 3 = virginica)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "iris.data[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "iris.target[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next two lines create a new classifier object and perform the fitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf = tree.DecisionTreeClassifier()\n",
    "clf = clf.fit(iris.data, iris.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, we're done with the hard work. We've trained the decision tree and can use it to predict the class from  input data. Let's try it with a few flower measurements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf.predict([[ 5. ,  3.6,  1.4,  0.2],\n",
    "             [ 5.9,  3. ,  5.1,  1.8],\n",
    "             [ 6.7,  3. ,  5. ,  1.7]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above example, we see that the predicted classes for the three flowers are 0, 2, and 1. We can get a little more detail with the predict_proba method. For each row of input measurements, it gives the probability that item will belong to a given class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf.predict_proba([[ 5. ,  3.6,  1.4,  0.2],\n",
    "                   [ 5.9,  3. ,  5.1,  1.8],\n",
    "                   [ 6.7,  3. ,  5. ,  1.7]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes we may want to inspect the decision tree. This is a two=step process. First we create a representation of the graph using the DOT language (http://www.graphviz.org/content/dot-language) and then we render an image of the graph using the graphviz Source method. To take a little of the mystery out of the process, we show a simple example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "simple_dot_data = 'digraph Tree {\\n0; \\n1; \\n2; 0 -> 1; 0 -> 2}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import graphviz\n",
    "graph = graphviz.Source(simple_dot_data)  \n",
    "graph "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we can generate the DOT description of a graph by hand, this is tedious and error prone for all but the simplest graphs. Fortunately, the sklearn tree module contains a method to automatically generate the graph from the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dot_data = tree.export_graphviz(clf, out_file=None)\n",
    "graph = graphviz.Source(dot_data)  \n",
    "graph "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we only need to pass the classifier to the Source method, the resulting graph is much more useful if we label the features (sepal length, ...) and classes (setosa, ...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dot_data = tree.export_graphviz(clf, out_file=None, \n",
    "                         feature_names=iris.feature_names,  \n",
    "                         class_names=iris.target_names,  \n",
    "                         filled=True, rounded=True,  \n",
    "                         special_characters=True)  \n",
    "graph = graphviz.Source(dot_data)  \n",
    "graph "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's worth taking a quick look at the DOT representation for this more complex graph. Makes you fully appreciate that sklearn provides a method for generating this from the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dot_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}