Training Part 2 Clean.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**\\*Remember to activate the *far_nlp* environment before starting jupyter notebook from command line and running this notebook:**\n",
    "\n",
    "**OS X, Linux:** `$ source activate far_nlp`\n",
    "\n",
    "**Windows:** `$ activate far_nlp`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify the right environment is enabled by checking the python path\n",
    "import sys\n",
    "print(sys.executable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Classifying Text Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Imagine the following scenario: Amazon is wanting to do a better job promoting new, potentially helpful, product reviews so they can be more visible to customers. The algorithm for displaying reviews is based on the current helpfulness rating for each review and new reviews get pushed to the bottom of the list because they have no rating.\n",
    "\n",
    "Our task is to use information from past reviews that have already been rated to predict the helpfulness of new reviews. We'll focus on seeing if we can use the review text to make a prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Load in the Amazon Reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.preprocessing import Binarizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The data we're using for this is from the Amazon product data curated by Julian McAuley [http://jmcauley.ucsd.edu/data/amazon/](http://jmcauley.ucsd.edu/data/amazon/). We're looking at a random sample of the reviews in the Electronics category. See the link for the full data source and a further explanation of the fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# First load in the reviews data set\n",
    "amazon_reviews = pd.read_csv('data/amazon_electronics_reviews_subset.csv', header=0, index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Try it Out:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Preview a sample of the data set by using either `.head()` or `.sample()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "amazon_reviews.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Operationalize the Y Variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The *helpful* column gives us how many reviews were voted helpful and the total number of votes received for that review. For example, if a particular review was voted helpful 6 times and unhelpful 4 times, we would end up with a rating of [6, 10].\n",
    "\n",
    "We'll have to do some work place the reviews into categories for helpful/not helpful. We are only going to consider reviews with 10 or more total votes because it's too easy for a review with only a few votes to have a high rating. Then we'll divide the number of helpful votes by the total number of votes received. Lastly, if the average for a review is greater than 60% we'll assign the review to the helpful (1) category. Otherwsie the review will be assigned to the unhelpful (0) category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# This funciton will created our helpfulness average\n",
    "def helpful_transformer(help_string):\n",
    "    stripped = help_string.strip().replace(\"[\", \"\").replace(\"]\", \"\")\n",
    "    split = stripped.split(\",\")\n",
    "    split[0] = split[0].strip()\n",
    "    split[1] = split[1].strip()\n",
    "    if int(split[1]) == 0 or int(split[1]) < 10:\n",
    "        return 0\n",
    "    else:\n",
    "        helpful_avg = int(split[0]) / int(split[1]) \n",
    "    return helpful_avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Here we apply the function above to the full dataframe\n",
    "amazon_reviews['helpful_avg'] = amazon_reviews['helpful'].apply(helpful_transformer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# How many reviews are greater than 60% (helpful)?\n",
    "print(amazon_reviews.loc[amazon_reviews['helpful_avg'] > 0.60].shape)\n",
    "\n",
    "# How many reviews are 60% or below (unhelpful)?\n",
    "print(amazon_reviews.loc[amazon_reviews['helpful_avg'] <= 0.60].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# We'll use binarizer to make a binary is helpful column\n",
    "helpful_binzrizer = Binarizer(copy=True, threshold=0.60)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Apply the Binarizer to the data frame\n",
    "amazon_reviews['is_helpful'] = helpful_binzrizer.fit_transform(amazon_reviews['helpful_avg'].values.reshape(-1,1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Try it Out:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Load a sample of the new data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Your Code Here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Let's view the resulting data frame\n",
    "amazon_reviews.sample(10, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Construct a Balanced Data Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Right now our data set is very unbalanced, we have 189,279 unhelpful reviews and only 10,721 helpful reviews. Modeling on a more balanced data set will help our algorithm better predict between each class. We'll combine our helpful reviews with a random sample of the same number of unhelpful reviews to construct a balanced data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "helpful_reviews = amazon_reviews.loc[amazon_reviews['is_helpful'] == 1].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# View the shape and sample of the helpful reviews\n",
    "print(helpful_reviews.shape)\n",
    "helpful_reviews.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "not_helpful = amazon_reviews.loc[amazon_reviews['is_helpful'] == 0].sample(n=10721, random_state=17).copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# View the shape and sample of the unhelpful reviews\n",
    "print(not_helpful.shape)\n",
    "not_helpful.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Join the two subsets to make a ballanced sample\n",
    "reviews_ballanced_sample = pd.concat([helpful_reviews, not_helpful], axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Try it Out:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Preview the new ballanced data set. Make sure ther eare intances of each class in the preview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Your Code here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "# Preview our ballanced data set\n",
    "print(reviews_ballanced_sample.shape)\n",
    "reviews_ballanced_sample.sample(5, random_state=17)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Manually Vectorize the Text and Predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Basic stuff just in case it isn't loaded already\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# For splitting our data in to training and test sets\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# For our vectorizing and modeling pipeline\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "from sklearn.pipeline import Pipeline #, make_pipeline, make_union\n",
    "from pprint import pprint\n",
    "from time import time\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "# import logging\n",
    "\n",
    "# For vectorizing and tokenizing our text\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "import spacy                        \n",
    "nlp = spacy.load('en') \n",
    "\n",
    "# Classifiers\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "# Metrics\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix, classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Take a sample of the data set for quicker processing.\n",
    "sample = reviews_ballanced_sample.sample(n=1000, random_state=17)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Confirm the shape of the sample\n",
    "sample.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Split out X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Setup training and final test data\n",
    "# If you want to run on the full set just replace\n",
    "# Sample with the full data set\n",
    "\n",
    "X = sample\n",
    "y = sample['is_helpful']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.33, random_state=17)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Try it Out:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Preview the length/shape of X_train, y_train, X_test and y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "print(\"x train:\", X_train.shape)\n",
    "print(\"y train:\", len(y_train))\n",
    "print(\"x test:\", X_test.shape)\n",
    "print(\"y test:\", len(y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Try it Out: — Extract the Review Column from X_train and X_test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Returns the review text column from the data frame\n",
    "def review_extractor(dataframe):\n",
    "    return dataframe['reviewText'].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Use the *review_extractor* to pull out the *reviewText* column from the data frame for X_train and X_test. You can accomplish this by passing in the whole data frame to the funciton. Set the result to a new variables called *review_text_train* and *review_text_text*. Confirm that the variables are a list, see how long each list is and display a few items from each list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "# Pull out the review column from X_train\n",
    "review_text_train = review_extractor(X_train)\n",
    "review_text_test = review_extractor(X_test)\n",
    "\n",
    "# Display the first two reviews and the length of the array\n",
    "print(\"Training Data:\")\n",
    "print(\"type: \", type(review_text_train))\n",
    "print(\"length: \", len(review_text_train))\n",
    "print(review_text_train[0:2])\n",
    "\n",
    "print(\"\\n=======\")\n",
    "\n",
    "print(\"\\nTest Data:\")\n",
    "print(\"type: \", type(review_text_test))\n",
    "print(\"length: \", len(review_text_test))\n",
    "print(review_text_test[0:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Try it Out — Tokenize the Text:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Vectorize the X_train and X_test using either CountVectorizer or TfidfVectorizer and the spaCy tokenizer below. Pass in *review_text_train* and *review_text_test* from the previous step. Set the results to new variables *vec_text_train*, and *vec_text_test*. When vectorizing the training text use `.fit_transoform()`, then use the same instance of your vectorizer with only `.transform()` on the test text. Verify that the resulting sparce matrices are the same width.\n",
    "\n",
    "**What happens to the shape of *vec_text_train*, and *vec_text_test* if you use `.fit_transform()` on both the train and test text? Why is this a bad idea?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Custom tokenizer using SpaCy\n",
    "def spacy_tokenizer(doc_as_string):\n",
    "    spacy_doc = nlp(doc_as_string)\n",
    "\n",
    "    tokens = []\n",
    "    for tok in spacy_doc:\n",
    "        if tok.like_email == True:\n",
    "            tokens.append('email')\n",
    "        elif tok.like_url:\n",
    "            tokens.append('URL')\n",
    "        elif tok.lemma_ == \"-PRON-\":\n",
    "            tokens.append(tok.lower_)\n",
    "        elif tok.is_alpha == True:\n",
    "            tokens.append(tok.lemma_)\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "TFIDF = TfidfVectorizer(tokenizer=spacy_tokenizer)\n",
    "vec_text_train = TFIDF.fit_transform(review_text_train)\n",
    "vec_text_test = TFIDF.transform(review_text_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "vec_text_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "vec_text_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "# 2nd time using fit_transform for both\n",
    "TFIDF = TfidfVectorizer(tokenizer=spacy_tokenizer)\n",
    "vec_text_train_2 = TFIDF.fit_transform(review_text_train)\n",
    "vec_text_test_2 = TFIDF.fit_transform(review_text_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "vec_text_train_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "vec_text_test_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Train the Model and Predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We'll use a multinomial Naive Bayes as our baseline model. See the docs for more: [MultinomialNB](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Fit the classifier\n",
    "clf = MultinomialNB()\n",
    "clf.fit(vec_text_train, y_train)\n",
    "\n",
    "# Generate Predictions\n",
    "y_predictions = clf.predict(vec_text_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Try it Out — View the Predictions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Print or display the y_predicitons to see the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "y_predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Evaluate the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We can accuracy score to get a broad idea of how well out classifier is doing. We pass in the correct label, and then the predicted label and the percentage of accurate predictions is returned. **How well did our classifier do?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "score = accuracy_score(y_test, y_predictions)\n",
    "print(score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Try it Out:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Go back through the steps above and make adjustments to the vectorization and the paramters of multinomial Naive Bayes to see if you can improve the accuracy score. **What is the top score you can acheive?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Model Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Thankfully there is a better way. We can use scikit learn pipelines to efficiently go through the steps above and iterate over differen parameters.\n",
    "\n",
    "Pipeline docs: [scikit learn Pipeline](http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Returns the review text column from the data frame\n",
    "def review_extractor(dataframe):\n",
    "    return dataframe['reviewText'].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Setup training and final test data\n",
    "# If you want to run on the full set just replace\n",
    "# Sample with the full data set\n",
    "\n",
    "X = sample\n",
    "y = sample['is_helpful']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.33, random_state=17)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Returns the review text column from the data frame\n",
    "def review_extractor(dataframe):\n",
    "    return dataframe['reviewText'].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Custom tokenizer using SpaCy\n",
    "def spacy_tokenizer(doc_as_string):\n",
    "    spacy_doc = nlp(doc_as_string)\n",
    "\n",
    "    tokens = []\n",
    "    for tok in spacy_doc:\n",
    "        if tok.like_email == True:\n",
    "            tokens.append('email')\n",
    "        elif tok.like_url:\n",
    "            tokens.append('URL')\n",
    "        elif tok.lemma_ == \"-PRON-\":\n",
    "            tokens.append(tok.lower_)\n",
    "        elif tok.is_alpha == True:\n",
    "            tokens.append(tok.lemma_)\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "pipeline = Pipeline([\n",
    "    ('extractor', FunctionTransformer(review_extractor, validate=False)),\n",
    "    ('vect', TfidfVectorizer()),\n",
    "    ('clf',MultinomialNB())\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# uncommenting more parameters will give better exploring power but will\n",
    "# increase processing time\n",
    "parameters = {\n",
    "    'vect__max_features': (None, 5000, 10000, 50000),\n",
    "    'vect__ngram_range': ((1, 1), (1, 2), (1,3)),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "if __name__ == \"__main__\":\n",
    "    # multiprocessing requires the fork to happen in a __main__ protected\n",
    "    # block\n",
    "\n",
    "    # find the best parameters for both the feature extraction and the\n",
    "    # classifier\n",
    "    grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, cv=5)\n",
    "\n",
    "    print(\"Performing grid search...\")\n",
    "    print(\"pipeline:\", [name for name, _ in pipeline.steps])\n",
    "    print(\"parameters:\")\n",
    "    pprint(parameters)\n",
    "    t0 = time()\n",
    "    grid_search.fit(X, y)\n",
    "    print(\"done in %0.3fs\" % (time() - t0))\n",
    "    print()\n",
    "\n",
    "    print(\"Best score: %0.3f\" % grid_search.best_score_)\n",
    "    print(\"Best parameters set:\")\n",
    "    best_parameters = grid_search.best_estimator_.get_params()\n",
    "    for param_name in sorted(parameters.keys()):\n",
    "        print(\"\\t%s: %r\" % (param_name, best_parameters[param_name]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Model Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Generate Predicitons\n",
    "\n",
    "# Get the best performing model from the GridSearch above\n",
    "best_model = grid_search.best_estimator_\n",
    "\n",
    "# Fit and predict\n",
    "best_model.fit(X_train, y_train)\n",
    "y_predictions = best_model.predict(X_test)\n",
    "\n",
    "# Score our Predictions\n",
    "score = accuracy_score(y_test, y_predictions)\n",
    "print(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# What did our predicitons look like?\n",
    "pd.Series(y_predictions).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# What did the true class realy look like?\n",
    "pd.Series(y_test).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# A confusion Matrix can help us better understand how our classifier is doing.\n",
    "from utilities import plot_confusion_matrix\n",
    "\n",
    "conf_matrix = confusion_matrix(y_test, y_predictions)\n",
    "labels = ['Unhelpful Review - 0', 'Helpful Review - 1']\n",
    "np.set_printoptions(precision=2)\n",
    "\n",
    "plt.figure(figsize=(5,5), dpi=150)\n",
    "plt.grid(False)\n",
    "plot_confusion_matrix(conf_matrix, labels )\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Precision and Recall can also help us understand how our predicitons are doing.\n",
    "\n",
    "See the link below for a great explanation of Precision and Recall. This site is a great general resource for outher Data Science topics as well.\n",
    "https://chrisalbon.com/machine-learning/precision_recall_and_F1_scores.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(y_test, y_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Try it Out:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Iterate through the steps above to try and improve the model. You can adjust parameters in how the documents are vectorized and parameters for the classifier. You can also try using a different classifier like LinearSVC instead of MultinomialNB by substituting it in the last step of the pipeline above. You can use `pipeline.get_params().keys()` to get a listing of all the parameters available in the current pipeline. This is demonstrated in the cell below.\n",
    "\n",
    "**What is the best overal classification accuracy that you can acheive?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "pipeline.get_params().keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Clustering Text Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The data set for the next exercise contains reviews for the SanDisk Ultra 64GB MicroSDXC Memory Card. Imagine we work for SanDisk and our task is to sort through the negative reviews to find any discernible patterns. What can we learn from these negative reviews that will give insight on how we can improve our product?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "\n",
    "import spacy\n",
    "nlp = spacy.load('en')  \n",
    "\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "from bokeh.plotting import figure, show, output_notebook, ColumnDataSource\n",
    "from bokeh.palettes import viridis\n",
    "from bokeh.models import HoverTool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# import sandisk review\n",
    "sandisk_reviews = pd.read_csv(\"data/sandisk_sd_card_reviews.csv\", index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Try it Out:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Preview the data set by using `.head()`. Notice the asin number is the same for every row, this is because the asin represents the product and all of these reviews are for the same product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "sandisk_reviews.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Process  the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Let's drop any rows where there are missing values in \n",
    "# the overal and reviewText columns\n",
    "\n",
    "print(sandisk_reviews.shape)\n",
    "sandisk_reviews.dropna(axis=0, how='any', subset=['overall', 'reviewText'], inplace=True)\n",
    "\n",
    "# See the shape after the drop\n",
    "sandisk_reviews.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "sandisk_reviews.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "So it looks like rating and review text are not missing for any of these columns. So we're good to go."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Make a Subset of the Reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "sandisk_bad_reviews = sandisk_reviews.loc[sandisk_reviews['overall'] <= 2].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "sandisk_bad_reviews.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# We're going to make another column of a trimmed version of the review \n",
    "# We'll end up using this later on when we graph our data.\n",
    "def review_trimmer(text):\n",
    "    words = text.split()\n",
    "    if len(words) > 200:\n",
    "        words = words[0:75]\n",
    "    words = ' '.join(words)\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "sandisk_bad_reviews['reviewPreview'] = sandisk_bad_reviews['reviewText'].apply(review_trimmer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "sandisk_bad_reviews.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Vectorize the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Use the same pipeline, `review_extractor` function and `spacy_tokenizer` from the classification example above to vectorize all the review text in the `sandisk_bad_reviews` data frame. However, remove the last step from the pipeline (the classifier) so the pipeline ends with a vectorizer, not a classifier. Use `fit_transform()` on the pipeline and assign the results to the variable tfidf_docs. Note that we don't need to split the data into training and test sets for this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Your Code here\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### Hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "# Custom Functions to extract the review text from the DF\n",
    "def review_extractor(dataframe):\n",
    "    return dataframe['reviewText'].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "# Custom tokenizer using SpaCy\n",
    "def spacy_tokenizer(doc_as_string):\n",
    "    spacy_doc = nlp(doc_as_string)\n",
    "\n",
    "    tokens = []\n",
    "    for tok in spacy_doc:\n",
    "        if tok.like_email == True:\n",
    "            tokens.append('email')\n",
    "        elif tok.like_url:\n",
    "            tokens.append('URL')\n",
    "        elif tok.lemma_ == \"-PRON-\":\n",
    "            tokens.append(tok.lower_)\n",
    "        elif tok.is_alpha == True:\n",
    "            tokens.append(tok.lemma_)\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "tfidf_pipeline = Pipeline([\n",
    "    ('extractor', FunctionTransformer(review_extractor, validate=False)),\n",
    "    ('vect', TfidfVectorizer(tokenizer=spacy_tokenizer, ngram_range=(1,1), use_idf=False, min_df=1)),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "editable": false,
    "hidden": true,
    "run_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "tfidf_docs = tfidf_pipeline.fit_transform(sandisk_bad_reviews)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Cluster, Reduce and Plot the Documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Instantiate KMeans\n",
    "kmeans = KMeans(n_clusters=7, n_jobs=-1, max_iter=700, n_init=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Predict with Kmeans to generate unsupervised classes\n",
    "cluster_labels = kmeans.fit_predict(tfidf_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Reduce the Term Matrix to 2 dimensions for Plotting\n",
    "TSVD = TruncatedSVD(n_components=2, random_state=17)\n",
    "reduced_docs = TSVD.fit_transform(tfidf_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Notice how everything gets reduced to 2 columns\n",
    "print(reduced_docs.shape)\n",
    "reduced_docs[0:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# plot our classes\n",
    "n_categores = pd.Series(cluster_labels).nunique()\n",
    "color_swatches = viridis(n_categores)\n",
    "categories =  np.unique(cluster_labels).tolist()\n",
    "colormap = dict(zip(categories, color_swatches))\n",
    "cat_colors = [colormap[i] for i in cluster_labels]\n",
    "\n",
    "\n",
    "source = ColumnDataSource(data=dict(\n",
    "    x = reduced_docs[:,0],\n",
    "    y = reduced_docs[:,1],\n",
    "    desc = sandisk_bad_reviews['reviewPreview'].tolist(),\n",
    "    color=cat_colors,\n",
    "    label = cluster_labels\n",
    "))\n",
    "\n",
    "\n",
    "hover = HoverTool(tooltips=[\n",
    "    (\"index\", \"$index\"),\n",
    "    (\"desc\", \"@desc\")\n",
    "])\n",
    "\n",
    "p = figure(title = \"Documents by Truncated SVD Values\", width=700, height=700, tools=[hover, 'pan', 'box_zoom', 'reset', 'zoom_in','zoom_out'])\n",
    "p.xaxis.axis_label = 'TSVD Dimension 1'\n",
    "p.yaxis.axis_label = 'TSVD Dimension 2'\n",
    "\n",
    "p.circle('x','y', size=8, fill_alpha=0.2, \n",
    "        color='color',\n",
    "        legend='label',\n",
    "         source=source,)\n",
    "\n",
    "output_notebook(notebook_type='jupyter')\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Iterate through the steps above and adjust parameters to see if you can find any natural clusters or trends in the reviews that would shed light on ways we might be able to improve our product."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "source": [
    "\n",
    "# End of Part 2"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}