Chatbot.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1ZpkEp7u4yJf",
    "outputId": "4bcd7ba1-7774-410e-8176-82834374d309"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s06854YE4KVn"
   },
   "source": [
    "# Intent Recognition In Chatbot System\n",
    "\n",
    "In this project, I will be building machine learning models that can recognize intents of user inputs in chatbot systems. A chatbot is a program that conducts a conversation via auditory or textual methods. In practice chatbots are used in many different purposes including customer services or information acquisition. Today, most chatbots are accessed via virtual assistants such as Google Assistant, Apple Siri, Amazon Alexa, via messaging apps such as Facebook Messenger, or via individual organization’ apps and websites.\n",
    "Many chatbot systems that contains the intent recognition module which is trying to figuring out what the users want when interacting with the chatbot. For example, an user might say “find restaurants near me”, “find Vietnamese restaurants near me” which all have the “find_restaurant” intent. Usually, a goal-oriented chatbot has a pre-defined set of intents (goal-oriented chatbot is a type of chatbot that helps users do a specific task like set an alarm, find restaurnts, order pizza, …). For example, an virtual assistants might contains the following intents: set_alarm, find_restaurants, open_app, find_video, … So, the purpose of this project is trying to find the intent of a user input using this dataset https://github.com/snipsco/nlu-benchmark/tree/master/2017-06-custom-intent-engines. This problem can be treated as a classification problem which classifies a user input text into a category which is an intent. \n",
    "So, in this project you will do the following:\n",
    "<br>\n",
    "- Collecting the data\n",
    "- Preprocessing the data\n",
    "- Understanding the data\n",
    "- Represent user texts as vectors using TfidfVectorizer\n",
    "- Train, evaluate baseline Logistic Regression model\n",
    "- Train, evaluate cross-validation Logistic Regression model\n",
    "- Learning about word embeddings\n",
    "- Represent user texts as vectors using word embedding with spaCy\n",
    "- Train, evaluate Feedforward Neural Network model\n",
    "- Error analysis\n",
    "- Improving the models\n",
    "- Deploy the models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kMFpwSec4KVx"
   },
   "source": [
    "## Collecting and preprocessing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fr0Ze8nY4KVy"
   },
   "source": [
    "I will use this data set https://github.com/snipsco/nlu-benchmark/tree/master/2017-06-custom-intent-engines. The data is originally collected as a benchmark for chatbot creating frameworks. Those frameworks usually contain intents classification and other features, here I only focus on the intent classification feature.\n",
    "<br>\n",
    "I have already included the data in the **data** folder and only keep the files that I need for this project.\n",
    "<br>\n",
    "Each folder includes data for the intent and inside each of them contains two JSON files, one for training and one for testing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hZRauB0j4KVz"
   },
   "source": [
    "### Import some libs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "6xm3z5Xs4KVz"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import sys\n",
    "import os\n",
    "import json\n",
    "import pandas as pd\n",
    "from sklearn.utils import shuffle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VnJODNHY4KVz"
   },
   "source": [
    "### Read json files\n",
    "Loading the JSON data into **train_df** and **test_df** dataframes. The dataframes have two columns which are **text** (the user input text), **intent** (the intent of the user input text)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "qhZ10Xj_4KV0"
   },
   "outputs": [],
   "source": [
    "def get_json_examples(intent_name, json_data):\n",
    "    examples = []\n",
    "    for e in json_data:\n",
    "        parts = []\n",
    "        for p in e[\"data\"]:\n",
    "            parts.append(p[\"text\"])\n",
    "        examples.append({\n",
    "            \"intent\": intent_name,\n",
    "            \"text\": \"\".join(parts)\n",
    "        })\n",
    "    return examples\n",
    "\n",
    "train_json_examples = []\n",
    "test_json_examples = []\n",
    "\n",
    "for intent_name in next(os.walk(\"/content/drive/MyDrive/MLP305x_asm1_longdhFX06754@funix.edu.vn/data\"))[1]:\n",
    "    train_file = f\"/content/drive/MyDrive/MLP305x_asm1_longdhFX06754@funix.edu.vn/data/{intent_name}/train_{intent_name}_full.json\"\n",
    "    test_file = f\"/content/drive/MyDrive/MLP305x_asm1_longdhFX06754@funix.edu.vn/data/{intent_name}/validate_{intent_name}.json\"\n",
    "\n",
    "    with open(train_file, \"r\", encoding=\"utf8\") as f:\n",
    "        train_json = json.load(f)\n",
    "        train_json_examples += get_json_examples(intent_name, train_json[intent_name])\n",
    "        \n",
    "    with open(test_file, \"r\", encoding=\"utf8\") as f:\n",
    "        test_json = json.load(f)\n",
    "        test_json_examples += get_json_examples(intent_name, test_json[intent_name])\n",
    "        \n",
    "train_df = pd.DataFrame.from_records(train_json_examples)\n",
    "test_df = pd.DataFrame.from_records(test_json_examples)\n",
    "\n",
    "train_df = shuffle(train_df, random_state=10)\n",
    "test_df = shuffle(test_df, random_state=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lPWu9CPN4KV0"
   },
   "source": [
    "### Plot bar graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8AvqctG64KV0"
   },
   "source": [
    "Plot a bar graph to count the number of texts for each intent in the **train_df**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YogQvaFIFYAu",
    "outputId": "94cb3476-9e22-42af-f413-29e3fb5c60db"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PlayMusic               2000\n",
       "GetWeather              2000\n",
       "BookRestaurant          1973\n",
       "SearchScreeningEvent    1959\n",
       "RateBook                1956\n",
       "SearchCreativeWork      1954\n",
       "AddToPlaylist           1942\n",
       "Name: intent, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['intent'].value_counts()\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "yLLaFM9XFDjs"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 467
    },
    "id": "8ln5dpp04KV0",
    "outputId": "e433295c-1709-4303-96a7-44341ed027e4"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlwAAAHCCAYAAAA6mBceAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd5hcZd3/8fdXQiAiUgNSDUpQQQQ1UgwhAekWqhQBQSnyICJYEAQFQcAfqAgiKCjtQeEBFUEJIsWgiKhBkCoaIZpESuidUL6/P+57w7Bskk12z87u5v26rrl25syZM/ecnfI5dzuRmUiSJKk5r2t3ASRJkgY7A5ckSVLDDFySJEkNM3BJkiQ1zMAlSZLUMAOXJElSwwxckiRJDTNwacCKiKMi4vx2l2NuRcQ5EfH1Nj13RMTZEfFoRPy5HWXoroj4ekQ8FBH3t7ssszI3/8uIWDkinoqIBZou17yIiCsiYo9e2taIiMiIGNIb2+sNdd+/pZe3OSEi9m653e/fs2ofA5f6rfoF2XF5OSKebbm9a7vLN0BtAGwKrJiZ63S+MyL2jIjre+OJerKtiFgZ+Dywema+qTfK026Z+Z/MfENmvjSndXs7sHTnf5GZW2bmufO4/ckRscm8lW6O2x4XEVN7up267+/pjTJ1ZTC+Z9W7DFzqt+oX5Bsy8w3Af4APtyz7cbvL1x/MQ23Jm4HJmfl0E+XpRSsDD2fmg3P7wP5Uq6K+0U/+5/P8ntX8wcClgW5oRJwXEU9GxB0RMarjjohYPiJ+FhHTI+LeiDhwVhupTUPfi4jL67b+FBFvrfe9prahtSmh1h78ISJOiojHIuKeiHh/XT4lIh7soqlm6Yi4qj7XdRHx5pZtv73e90hE3B0RO3Yq5+kRMT4ingY26uK1LB8Rl9XHT4qIferyvYAfAuvXWsKvdXrcO4Dvt9z/WF2+UER8MyL+ExEPRMT3I2JYvW98RHyrZRsXRsRZs9nWVhFxZ33d0yLiC12UfxPgKmD5+thz6vKP1P/xY3X/v6PlMZMj4ksRcSvwdFc/wHPYrx+MiJsj4on6Pzuq02M3iIgb6nNPiYg9W+5eoqv3TRfP/6r3UX0Nx9T3zpMR8ZuIWLqu/rv697G6D9avj/lkRNwVpUn4yk7vm4yI/SLin7Wc34uiy/9FF+Xr/J6+vv7fH43y+dlyFo/7X0rY+GXd/iEtd+9a3zcPRcThLY95XUQcGhH/ioiHI+KiiFiyi20vAlzBK++Fp+r7+6iI+GlEnB8RTwB7RsQ6EfHH+trvi4hTI2Jop/2zar0+y8/77ETEphHx94h4PCJOBaIu7/I9K71KZnrx0u8vwGRgk07LjgKeA7YCFgCOB26s970OuAn4KjAUeAtwD7D5LLZ/DvAwsA4wBPgxcGG9bwSQwJCW9ScAe9frewIvAp+o5fg6pUbue8BCwGbAk8AbWp7rSWDDev/JwPX1vkWAKXVbQ4B3Aw9Rmik6Hvs4MLq+xoW7eC2/A04DFgbWBqYDG7eU9frZ7OfX3A+cBFwGLAksCvwSOL7e9ybgQWBjYNe6jxedzbbuA8bU60sA75lFOcYBU1turwY8TWkOXRA4BJgEDG15f9wCrAQM62J7c9qv44A16z59F/AAsE297831/7VLfe6lgLXn9L7pogwjaHkfUd5D/6qvbVi9/Y3ZvOe2rq/5HfW5jgBuaLk/gV8Bi1MC0HRgi+7832fxnn4B2Ifynv4f4L9AdOfz2VL+M+trWwt4HnhHvf+zwI3AipTPwA+AC7rzXmj57L8AbFP/Z8OA9wLr1X0zArgLOKjT/ll1bv9vLY9fur4Pdqjvg4Mpn/u9Z1VOL15aL9ZwaaC7PjPHZ+kX87+UL3aA9wHDM/PozJyRpe/GmcDOs9nWJZn558x8kfIFvPZclOPezDy7luP/KD/8R2fm85n5G2AGsGrL+pdn5u8y83ngcErtw0rAhyhNfmdn5ouZeTPwM+CjLY+9NDP/kJkvZ+ZzrYWo2xgNfCkzn8vMWyi1Wh+fi9fSur0A9gUOzsxHMvNJ4DjqfszM+yk/xudSguPH6zqz8gKwekS8MTMfzcy/drMoO1H22VWZ+QLwTcqP7Ptb1jklM6dk5rNdPH62+zUzJ2TmbXWf3gpcAIytj/0YcHVmXpCZL2Tmw3W/dujJ++bszPxHLfNFc3jsfpSge1d9ruOAtVtruSiB7bHM/A/w27ksS2f/zswz63v6XGA5YNm53MbXMvPZzPwb8Dde+XzuBxyemVPrZ+AoYIeuaiZn44+Z+Yv6P3s2M2/KzBvr/3cyJcSNnc3j5/b/thVwR2b+tL4HvwPYOV7dZuDSQNf6hfcMsHD90n4zpXr/sY4L8GVm/4PReVtvmItyPNBy/VmAzOy8rHV7UzquZOZTwCPA8rXc63Yq966UmqTXPLYLywMdwajDv4EV5uK1tBoOvB64qaU8v67LO/ySUgtyd2bOqZP89pQfrn9HaUpdv5vlWJ7yOgDIzJcp+6H1dc1uv8x2v0bEuhHx2yjNz49TAkFH895KlJqoWenJ+2ZuHvtm4OSW8j9CadJq3Qc9Kcssy5aZz9Src7u9WZXnzcAlLa/lLuAl5i7Qver/HRGrRcSvIuL+2sx4HK/8D+embLOyPK/+3GbnMkizY+DSYDWFUuu0eMtl0czcah621dHB/PUty3o6CmmljisR8QZKc91/KeW+rlO535CZ/9Py2JzNdv8LLBkRi7YsWxmY1s1ydd72Q5SwuEZLeRbLMpChw7GUH8zlImKX2ZUzM/+SmVsDywC/oNTqdMd/KT/SwMyat5V49eua3X6Z0379CaXZdKXMXIzS5ylaHjvH/j29rKvXMgX4VKfXMCwzb5jH7fWmud3+FGDLTq9l4czs6n06q213Xn468HdgZGa+kXKAFa951Ly7j1d/bqP1tjQnBi4NVn8GnozSkXpYRCwQEe+MiPfN7YYyczrlh323up1P0vMf4K2idMQeChxD6Xs2hdIHZ7WI2D0iFqyX90VLB/E5lHUKcANwfEQsHBHvAvYCujtf2QPAih2djWtN0pnASRGxDEBErBARm9frG1L6RX0c2AP4bkSs0NW2ImJoROwaEYvVJpkngJe7Wa6LgA9GxAciYkHK8Pvn62vtjjnt10UpNYPPRcQ6lGbEDj8GNomIHSNiSEQsFRE9aarrjumUfdM6b9T3gcMiYg2AiFgsIj7a1YO78Kr/RQMe4NVlnZPvA8d2NIdGxPCI2Ho2214qIhabwzYXpbynnoqIt1OaunvT5cAaEbFdrUU/kJ4feGk+YuDSoFT7nXyI0i/jXkpNzQ+BOX1pz8o+wBcpHW3XoPs/9LPyE+BISrPQe4HdAGpT4GaUPlL/pTR7/D9Kx+Lu2oXSafi/wCXAkZl5dTcfey1wB3B/RDxUl32J0ln7xtpUczXwtoh4I3AecEBmTsvM3wM/As6uR/9dbWt3YHLdzn6UZr05ysy7Kfvou5T/5Ycp04TM6Obj57Rf9weOjognKQMtLmp57H8ozaCfp/y/buGVvkiNqE14xwJ/qM1u62XmJbXMF9b9dzvQ5cjBLnT1v+hNxwNH1LK+ZuRpF06m1Cj+pu7zG4F1u1oxM/9O6VN3T93+8rPY5hcoQflJykHC/83la5itzHyI0ufvG5TvgZHAH3rzOTS4RWmGliRJUlOs4ZIkSWpYf5idV5KktouIMZSJVl+j00ARaa7ZpChJktQwmxQlSZIa1u+bFJdeeukcMWJEu4shSZI0RzfddNNDmTm88/J+H7hGjBjBxIkT210MSZKkOYqIf3e13CZFSZKkhhm4etnNN9/M6NGj2XDDDdl444255557eO6559h1110ZM2YMu+66K889V843PHnyZDbeeGNGjx7NcccdN3Mbv/71r1l//fVZf/31ufLKK9v1UhrnvpIkzS/6/SjFUaNG5UBqUrz//vtZZJFFWHTRRRk/fjwXXHABo0ePZvr06XzlK1/h6KOPZplllmG//fZj55135tOf/jRjxoxhk0024dRTT2XkyJG8+93v5ne/+x0AY8eO5a9//SsLLLBAm19Z73NfSZIGm4i4KTNHdV5uDVcve9Ob3sSii5bzBi+00EIMGTKE6667jg996EMAfPjDH+a6664D4JZbbmHMmDEAfPCDH+S6665j0qRJrLLKKiy++OIsvvjijBgxgkmTJrXnxTTMfSVJml8YuBry9NNPc8QRR/DFL36Rhx9+mCWWWAKAxRdfnEceeQSAl19+5by9Hctb1+28/mDlvpIkDXYGrga88MIL7LTTTnzpS19i9dVXZ8kll+Sxxx4D4PHHH2fJJZcE4HWve2X3dyxvXbfz+oOR+6p7Nt98c4YPH87Xv/51AJ555hl22GEHxo0bx7bbbjtzPxxyyCGMGzeOcePGsdxyy/Hd734XgN13351x48YxatQoTjrppLa9DkmaXxm4etnLL7/MbrvtxjbbbMM222wDlL5F48ePB2D8+PGMHTsWgLXWWosbbrgBgCuuuIINN9yQkSNHcu+99/LEE0/wxBNPcO+997Lqqqu258U0zH3VfT/60Y848cQTZ94+44wzGDVqFBMmTGDnnXeeed8JJ5zAhAkTmDBhAksvvTTbb7/9zMdPmDCBG2+8kdNOO40nn3yyLa9DkuZX/X4eroHm5z//OZdffjkPPPAA559/PmuuuSYnnHACn/zkJxkzZgwrrrgiZ599NgDHH388e+21FzNmzGDLLbfkHe94x8zlm2+++czrg7UTuPuq+1ZcccVX3f7HP/7BdtttB8A666zDySef/Kr7//rXv7Lsssuy/PLLAzB06FAAnnvuOVZeeWVe//rX90GpJUkzZWa/vrz3ve9NSZlnn312HnPMMZmZedppp+XnP//5zMw89dRT821ve9ur1j344IPzrLPOetWyHXbYIYcPH55f/epX+6bAbbDZZpvl0ksvPXM/Pf3007n99tvn2LFjc5tttslHH300MzOPPPLIfPvb355jx47NsWPH5osvvpiZmZdeemmus846ucEGG+T555/fttfRF3q6r6ZPn5477rhjbrTRRrnpppu27XU0raf7KTNzxowZueqqq87chgY3YGJ2kWdsUpQGoL322ovnnnuOjTbaiGnTps2syQJ46aWXuPTSS2c2J3a4+OKLmTx5Mpdffjl33nlnXxe5T3S36RXg8MMPn9n8usACC/Dyyy/zxS9+kauvvpprr72WU089lccff7wdL6NP9GRfARx00EF89atf5dprr+U3v/lNn5e/r/R0PwH84Ac/4O1vf3ufllv9zxwDV0SsFBG/jYg7I+KOiPhsXb5kRFwVEf+sf5eoyyMiTomISRFxa0S8p2Vbe9T1/xkRezT3sqTBbejQoZx66qn89re/ZcSIEeywww4z77vmmmsYNWoUb3zjG4FSiz1jxgwAFl54YYYNG8awYcPaUu6mddX0OmpUmQ5nnXXW4be//e3M+0444QQ22GADTjnlFAAeeughhg8fzqKLLsqCCy7Iaqutxp/+9Ke+K3wf68m+eumll7j99tv51re+xdixYznttNP6ruB9rCf7CeCpp57iiiuueM0B0GDU3cE9HfbYYw822WQTAKZOncrYsWMZM2YMo0ePHpSn9OtODdeLwOczc3VgPeDTEbE6cChwTWaOBK6ptwG2BEbWy77A6VACGnAksC6wDnBkR0iTNHv77LMPJ554Iueccw7bbLMNd955J+PGjeMDH/gAd911F/vuu+/Mdc8//3x22223mbdffPFFNttsM8aNG8cGG2zA9ttvzyqrrNKOl9Hn1lxzTX79618DZRBGx7Qhn/nMZ/jb3/7GVVddxWWXXcbvf/97hg8fzkMPPcS0adN44oknuP766+eraUbmZl89+OCD3HbbbXz2s5/lqquu4ic/+Ql33XVXO4vfZ+ZmPwGceOKJHHTQQW0rb1+am9rAW2+99VUBbNFFF+Xiiy/m97//PWeeeSYHH3xwn5a9L8wxcGXmfZn513r9SeAuYAVga+Dcutq5wDb1+tbAebUp80Zg8YhYDtgcuCozH8nMR4GrgC169dVIg9SZZ57JHXfcwaRJk/jFL37B6quvzoQJE7jmmms46aSTGDLklfEv5513Hh/+8Idn3l5wwQVnNnPccMMN882XP8y66XWppZYiIhg2bBjbbbcdEydOJCI444wz2H333dl1111Zc801X9VUO9jNzb5aYoklWH755VlrrbUYOnQo48aN47bbbmvzK+gbc7OfHnjgAW6++WY23XTTNpe6b8xNbeAxxxzDl7/85Zm3F1tsMZZZZhnglYmwB5u5ekURMQJ4N/AnYNnMvK/edT+wbL2+AjCl5WFT67JZLe/qefal1I6x8sorz00R58mIQy9v/Dn6k8nf+OA8P3Z+2lc92U/qHzqaXqEcbXf8IDz22GMsvvjiZCYTJkxgzz33BGDDDTfk2muv5cknn2TbbbdlvfXWa1fR+9zc7KuFF16Yt7zlLUyZMoWVVlqJm266aeao2cFubvbTbbfdxvTp09liiy2YNm0azz//PGuttdarDogGs47awE022eRVtYETJkxgtdVWY9lll33NY1566SUOPPBADj/88L4ubuO6Hbgi4g3Az4CDMvOJiJh5X2ZmRPTaSRkz8wzgDCjnUuyt7Uoa3PbZZx9uuOEGnn/+eSZOnMhxxx3H/vvvzwILLMC73vWumU0aBx10EHfffTeZybhx49hqq62AMnHsX/7yF4YMGcLxxx8/czqNwain++rkk09mt91244UXXmDjjTfmPe95z+yebsDq6X7q6KN0zjnnMHXq1PkmbEGpDfzc5z7HRhttxPrrrz+zNvAb3/gGF1544Wv6dAF86lOfYsstt5y53waTbp28OiIWBH4FXJmZ367L7gbGZeZ9tclwQma+LSJ+UK9f0LpexyUzP1WXv2q9WemLk1fPT7U2YA1Xd1nDJUlzpyNYHnHEEa9afsYZZ/Diiy+y++67M2bMGN70pjfx7LPPcscdd3DwwQdz+OGH84UvfIFhw4ZxzDHHtKn0vWNWJ6+eYw1XlKqsHwF3dYSt6jJgD+Ab9e+lLcsPiIgLKR3kH6+h7ErguJaO8psBh83rC5IkSf1Hd2oDhwwZwi233ALA5MmT2XvvvTn88MOZOHEiJ598MqNHj2bcuHEMHz6ciy++uM2vqHd1p0lxNLA7cFtE3FKXfZkStC6KiL2AfwM71vvGA1sBk4BngE8AZOYjEXEM8Je63tGZOf8MAZIkaRA788wzX7NswoQJs1x/xIgRXH311QCMGjWKF154oami9QtzDFyZeT0Qs7j7A12sn8CnZ7Gts4Cz5qaA0kA0PzW9gs2vkjQnzjQvSZLUsME30YWkAWV+qg10wEr3WWuqwcbAJUkaVAyn6o8MXJIkzafmp3Da7mBqHy5JkqSGGbgkSZIaZuCSJElqmIFLkiSpYQYuSZKkhhm4JEmSGmbgkiRJapiBS5IkqWEGLkmSpIYZuCRJkhpm4JIkSWqYgUuSJKlhBi5JkqSGGbgkSZIaZuCSJElqmIFLkiSpYQYuSZKkhhm4JEmSGmbgkiRJapiBS5IkqWEGLkmSpIYZuCRJkhpm4JIkSWqYgUuSJKlhBi5JkqSGGbgkSZIaNsfAFRFnRcSDEXF7y7L/i4hb6mVyRNxSl4+IiGdb7vt+y2PeGxG3RcSkiDglIqKZlyRJktS/DOnGOucApwLndSzIzJ06rkfEt4DHW9b/V2au3cV2Tgf2Af4EjAe2AK6Y+yJLkiQNLHOs4crM3wGPdHVfraXaEbhgdtuIiOWAN2bmjZmZlPC2zdwXV5IkaeDpaR+uMcADmfnPlmWrRMTNEXFdRIypy1YAprasM7Uu61JE7BsREyNi4vTp03tYREmSpPbqaeDahVfXbt0HrJyZ7wY+B/wkIt44txvNzDMyc1Rmjho+fHgPiyhJktRe3enD1aWIGAJsB7y3Y1lmPg88X6/fFBH/AlYDpgErtjx8xbpMkiRp0OtJDdcmwN8zc2ZTYUQMj4gF6vW3ACOBezLzPuCJiFiv9vv6OHBpD55bkiRpwOjOtBAXAH8E3hYRUyNir3rXzry2s/yGwK11moifAvtlZkeH+/2BHwKTgH/hCEVJkjSfmGOTYmbuMovle3ax7GfAz2ax/kTgnXNZPkmSpAHPmeYlSZIaZuCSJElqmIFLkiSpYQYuSZKkhhm4JEmSGmbgkiRJapiBS5IkqWEGLkmSpIYZuCRJkhpm4JIkSWqYgUuSJKlhBi5JkqSGGbgkSZIaZuCSJElqmIFLkiSpYQYuSZKkhhm4JEmSGmbgkiRJapiBS5IkqWEGLkmSpIYZuCRJkhpm4JIkSWqYgUuSJKlhBi5JkqSGGbgkSZIaZuCSJElqmIFLkiSpYQYuSZKkhhm4JEmSGjbHwBURZ0XEgxFxe8uyoyJiWkTcUi9btdx3WERMioi7I2LzluVb1GWTIuLQ3n8pkiRJ/VN3arjOAbboYvlJmbl2vYwHiIjVgZ2BNepjTouIBSJiAeB7wJbA6sAudV1JkqRBb8icVsjM30XEiG5ub2vgwsx8Hrg3IiYB69T7JmXmPQARcWFd9865LrEkSdIA05M+XAdExK21yXGJumwFYErLOlPrslktlyRJGvTmNXCdDrwVWBu4D/hWr5UIiIh9I2JiREycPn16b25akiSpz81T4MrMBzLzpcx8GTiTV5oNpwErtay6Yl02q+Wz2v4ZmTkqM0cNHz58XoooSZLUb8xT4IqI5Vpubgt0jGC8DNg5IhaKiFWAkcCfgb8AIyNilYgYSulYf9m8F1uSJGngmGOn+Yi4ABgHLB0RU4EjgXERsTaQwGTgUwCZeUdEXETpDP8i8OnMfKlu5wDgSmAB4KzMvKPXX40kSVI/1J1Rirt0sfhHs1n/WODYLpaPB8bPVekkSZIGAWealyRJapiBS5IkqWEGLkmSpIYZuCRJkhpm4JIkSWqYgUuSJKlhBi5JkqSGGbgkSZIaZuCSJElqmIFLkiSpYQYuSZKkhhm4JEmSGmbgkiRJapiBS5IkqWEGLkmSpIYZuCRJkhpm4JIkSWqYgUuSJKlhBi5JkqSGGbgkSZIaZuCSJElqmIFLkiSpYQYuSZKkhhm4JEmSGmbgkiRJapiBS5IkqWEGLkmSpIYZuCRJkhpm4JIkSWqYgUuSJKlhcwxcEXFWRDwYEbe3LDsxIv4eEbdGxCURsXhdPiIino2IW+rl+y2PeW9E3BYRkyLilIiIZl6SJElS/9KdGq5zgC06LbsKeGdmvgv4B3BYy33/ysy162W/luWnA/sAI+ul8zYlSZIGpTkGrsz8HfBIp2W/ycwX680bgRVnt42IWA54Y2bemJkJnAdsM29FliRJGlh6ow/XJ4ErWm6vEhE3R8R1ETGmLlsBmNqyztS6TJIkadAb0pMHR8ThwIvAj+ui+4CVM/PhiHgv8IuIWGMetrsvsC/Ayiuv3JMiSpIktd0813BFxJ7Ah4BdazMhmfl8Zj5cr98E/AtYDZjGq5sdV6zLupSZZ2TmqMwcNXz48HktoiRJUr8wT4ErIrYADgE+kpnPtCwfHhEL1OtvoXSOvycz7wOeiIj16ujEjwOX9rj0kiRJA8AcmxQj4gJgHLB0REwFjqSMSlwIuKrO7nBjHZG4IXB0RLwAvAzsl5kdHe73p4x4HEbp89Xa70uSJGnQmmPgysxdulj8o1ms+zPgZ7O4byLwzrkqnSRJ0iDgTPOSJEkNM3BJkiQ1zMAlSZLUMAOXJElSwwxckiRJDTNwSZIkNczAJUmS1DADlyRJUsMMXJIkSQ0zcEmSJDXMwCVJktQwA5ckSVLDDFySJEkNM3BJkiQ1zMAlSZLUMAOXJElSwwxckiRJDTNwSZIkNczAJUmS1DADlyRJUsMMXJIkSQ0zcEmSJDXMwCVJktQwA5ckSVLDDFySJEkNM3BJkiQ1zMAlSZLUMAOXJElSwwxckiRJDTNwSZIkNaxbgSsizoqIByPi9pZlS0bEVRHxz/p3ibo8IuKUiJgUEbdGxHtaHrNHXf+fEbFH778cSZKk/qe7NVznAFt0WnYocE1mjgSuqbcBtgRG1su+wOlQAhpwJLAusA5wZEdIkyRJGsy6Fbgy83fAI50Wbw2cW6+fC2zTsvy8LG4EFo+I5YDNgasy85HMfBS4iteGOEmSpEGnJ324ls3M++r1+4Fl6/UVgCkt602ty2a1XJIkaVDrlU7zmZlA9sa2ACJi34iYGBETp0+f3lublSRJaoueBK4HalMh9e+Ddfk0YKWW9Vasy2a1/DUy84zMHJWZo4YPH96DIkqSJLVfTwLXZUDHSMM9gEtbln+8jlZcD3i8Nj1eCWwWEUvUzvKb1WWSJEmD2pDurBQRFwDjgKUjYipltOE3gIsiYi/g38COdfXxwFbAJOAZ4BMAmflIRBwD/KWud3Rmdu6IL0mSNOh0K3Bl5i6zuOsDXaybwKdnsZ2zgLO6XTpJkqRBwJnmJUmSGmbgkiRJapiBS5IkqWEGLkmSpIYZuCRJkhpm4JIkSWqYgUuSJKlhBi5JkqSGGbgkSZIaZuCSJElqmIFLkiSpYQYuSZKkhhm4JEmSGmbgkiRJapiBS5IkqWEGLkmSpIYZuCRJkhpm4JIkSWqYgUuSJKlhBi5JkqSGGbgkSZIaZuCSJElqmIFLkiSpYQYuSZKkhhm4JEmSGmbgkiRJapiBS5IkqWEGLkmSpIYZuCRJkhpm4JIkSWrYPAeuiHhbRNzScnkiIg6KiKMiYlrL8q1aHnNYREyKiLsjYvPeeQmSJEn925B5fWBm3g2sDRARCwDTgEuATwAnZeY3W9ePiNWBnYE1gOWBqyNitcx8aV7LIEmSNBD0VpPiB4B/Zea/Z7PO1sCFmfl8Zt4LTALW6aXnlyRJ6rd6K3DtDFzQcvuAiLg1Is6KiCXqshWAKS3rTK3LJEmSBrUeB66IGAp8BLi4LjodeCulufE+4FvzsM19I2JiREycPn16T4soSZLUVr1Rw7Ul8NfMfAAgMx/IzJcy82XgTF5pNpwGrNTyuBXrstfIzDMyc1Rmjho+fHgvFFGSJKl9eiNw7UJLc2JELNdy37bA7fX6ZcDOEbFQRKwCjAT+3MXFYBsAACAASURBVAvPL0mS1K/N8yhFgIhYBNgU+FTL4hMiYm0ggckd92XmHRFxEXAn8CLwaUcoSpKk+UGPAldmPg0s1WnZ7rNZ/1jg2J48pyRJ0kDjTPOSJEkNM3BJkiQ1zMAlSZLUMAOXJElSwwxckiRJDTNwSZIkNczAJUmS1DADlyRJUsMMXJIkSQ0zcEmSJDXMwCVJktQwA5ckSVLDDFySJEkNM3BJkiQ1zMAlSZLUMAOXJElSwwxckiRJDTNwSZIkNczAJUmS1DADlyRJUsMMXJIkSQ0zcEmSJDXMwCVJktQwA5ckSVLDDFySJEkNM3BJkiQ1zMAlSZLUMAOXJElSwwxckiRJDTNwSZIkNazHgSsiJkfEbRFxS0RMrMuWjIirIuKf9e8SdXlExCkRMSkibo2I9/T0+SVJkvq73qrh2igz187MUfX2ocA1mTkSuKbeBtgSGFkv+wKn99LzS5Ik9VtNNSluDZxbr58LbNOy/LwsbgQWj4jlGiqDJElSv9AbgSuB30TETRGxb122bGbeV6/fDyxbr68ATGl57NS67FUiYt+ImBgRE6dPn94LRZQkSWqfIb2wjQ0yc1pELANcFRF/b70zMzMicm42mJlnAGcAjBo1aq4eK0mS1N/0uIYrM6fVvw8ClwDrAA90NBXWvw/W1acBK7U8fMW6TJIkadDqUeCKiEUiYtGO68BmwO3AZcAedbU9gEvr9cuAj9fRiusBj7c0PUqSJA1KPW1SXBa4JCI6tvWTzPx1RPwFuCgi9gL+DexY1x8PbAVMAp4BPtHD55ckSer3ehS4MvMeYK0ulj8MfKCL5Ql8uifPKUmSNNA407wkSVLDDFySJEkNM3BJkiQ1zMAlSZLUMAOXJElSwwxckiRJDTNwSZIkNczAJUmS1DADlyRJUsMMXJIkSQ0zcEmSJDXMwCVJktQwA5ckSVLDDFySJEkNM3BJkiQ1zMAlSZLUMAOXJElSwwxckiRJDTNwSZIkNczAJUmS1DADlyRJUsMMXJIkSQ0zcEmSJDXMwCVJktQwA5ckSVLDDFySJEkNM3BJkiQ1zMAlSZLUMAOXJElSwwxckiRJDZvnwBURK0XEbyPizoi4IyI+W5cfFRHTIuKWetmq5TGHRcSkiLg7IjbvjRcgSZLU3w3pwWNfBD6fmX+NiEWBmyLiqnrfSZn5zdaVI2J1YGdgDWB54OqIWC0zX+pBGSRJkvq9ea7hysz7MvOv9fqTwF3ACrN5yNbAhZn5fGbeC0wC1pnX55ckSRooeqUPV0SMAN4N/KkuOiAibo2IsyJiibpsBWBKy8OmMouAFhH7RsTEiJg4ffr03iiiJElS2/Q4cEXEG4CfAQdl5hPA6cBbgbWB+4Bvze02M/OMzByVmaOGDx/e0yJKkiS1VY8CV0QsSAlbP87MnwNk5gOZ+VJmvgycySvNhtOAlVoevmJdJkmSNKj1ZJRiAD8C7srMb7csX65ltW2B2+v1y4CdI2KhiFgFGAn8eV6fX5IkaaDoySjF0cDuwG0RcUtd9mVgl4hYG0hgMvApgMy8IyIuAu6kjHD8tCMUJUnS/GCeA1dmXg9EF3eNn81jjgWOndfnlCRJGoicaV6SJKlhBi5JkqSGGbgkSZIaZuCSJElqmIFLkiSpYQYuSZKkhhm4JEmSGmbgkiRJapiBS5IkqWEGLkmSpIYZuCRJkhpm4JIkSWqYgUuSJKlhBi5JkqSGGbgkSZIaZuCSJElqmIFLkiSpYQYuSZKkhhm4JEmSGmbgkiRJapiBS5IkqWEGLkmSpIYZuCRJkhpm4JIkSWqYgUuSJKlhBi5JkqSGGbgkSZIaZuCSJElqmIFLkiSpYX0euCJii4i4OyImRcShff38kiRJfa1PA1dELAB8D9gSWB3YJSJW78sySJIk9bW+ruFaB5iUmfdk5gzgQmDrPi6DJElSn+rrwLUCMKXl9tS6TJIkadCKzOy7J4vYAdgiM/eut3cH1s3MAzqtty+wb735NuDuPitk31oaeKjdhRgA3E/d577qPvdV97ifus991T2DfT+9OTOHd144pI8LMQ1YqeX2inXZq2TmGcAZfVWodomIiZk5qt3l6O/cT93nvuo+91X3uJ+6z33VPfPrfurrJsW/ACMjYpWIGArsDFzWx2WQJEnqU31aw5WZL0bEAcCVwALAWZl5R1+WQZIkqa/1dZMimTkeGN/Xz9tPDfpm017ifuo+91X3ua+6x/3Ufe6r7pkv91OfdpqXJEmaH3lqH0mSpIYZuCRJkhpm4NKgFxExu9uSJDXNwDWAdASFiFiq3WUZKCIisnZUjIjNI2JY2nFxrkXEwi3X+3ywzWAVEQtFxCL1+vDBcDAwGF7DQDM/HlTWczN3XF+wnWXpLgPXANERHCJiM+CHEbFcu8s0ELSErV2BQ4A3tLdEA09EvB7YOSLWjohDgFPaXabBoP5gvA/YPSJ2A44ClmhroXpBy2du/XaXZX7Rss/fFxELDvaDyohYCFg/It4cERsAHxwIB4IGrgGgJWxtAHwPODkz72t3uQaKiBgN7AKcmJnTW4+MNGeZ+QzwKPBLYA/gpPaWaHDIzJeAB4CtgROBqzPzkYgYkN/LdULrVev1VYGvzQ81Le0UEStHxKL1+nuAL7e5SH1lYWBV4LvAT4D/1Hk++/X7bUB+sOcX9Qts+ZajldWB0zNzQkQM6e9vrnbpYr+8ifIB3TYihtcfOs1Bp/14JXAd8DiwdK31mtW6moOO/ZWZ/6SErj8C76yf95fbWrh5EBFLAAcCu0XE8sCzwIx6oLhQe0s3+ESxKHAu8D+1duchyoFRRw0QAzW8z06tgHgcuBVYF/gD5f1Gf6/ZG3T/jEHmg8AKLe3Tz1GqTpfLzBfrl9nYiHh/G8vYr3Tqs7VORKwBXAIcCcwAdoyIJdtZxoGg035cGxiSmbsB3wCOA8bV+8ZFxML9/YuuP2mpsR5Z+2N+AjgIWBL4bF1nxYH0uc7MR4FfAYsAuwOjKD+IZObzbSzaYBWZ+STlffMBYB9gZeDxzHxhsO7zls/OGGAhymufCHwiIsbVdZbs6BfZ3zjxaT8XEcsAVwPbUo5eDqMcEV8CvBE4EzgoM69vWyH7oXoKqV2AWyjhYB1gS+D9wH+BH9UfCXXSKWztT+n7NpFSw/UD4MPA/sDtwK7AezPz320q7oAUEVtR9uUfKQe+H6XUYO8NvBVYA/hYZv6pbYXshtb3Sr29AbAN8GZgDPBr4PWUz9xTmXlEWwo6iHT6fI6mvFd2ptRwLV//3g8k8ARw6GA6IIqIbYCvAl/KzKsiYhXKQctQ4ClgI2CPzJzaxmJ2ycDVD7Wk+EUz88mI+DawPuVL+c2UH7wPAE9T+nNd0sbi9jv1S+grlL4xnwa2zcwx9b4dKOHreAPX7EXEdsCmlNrB9wOjgUco/Y3WAtYEbsjMf7StkANQRLyNUjNxPnAT8ENgyczcKiIWo7xvp2bmtW0s5hx1+uHfAHgG+A+lpu4LwDDgeuAfQABTahOqekHd58dn5pj6njqFErD+j/LbsCjwp4F+MFRrgYdm5n11sNjPgR0zc0pEvIPyfrud8vu4OfDjzPxF+0o8awaufqYlbH2E8gY6ODMfiogTgLHARzPzP/WN90K9LwbTEczc6uIo+63AxpSjvdHAhzJzRkR8JDMvi4hFMvPpdpV3IKjNrhOABzJz07psS0rQfxb4TmY+3L4SDjx1sMZiwKXA88BeHT+GEXEesAowtqMP10D5XNda0P+h1NZtCmwHvADsSalx+FFmTmlbAQehiPggZfDK9zLz5LrsncA3KecqPnUg9gXsLMp0NAcAFwPTKCHyt5SWndWBpSk1qp/MzB9HxND6Xd8vPzv24epnatjahDJE/NTMfKguP4TSjHhNRLwlM+9rua/fvbH62OvglY6ilC/7g4GPZOZm9QO4G3BwRCxl2Hqtzp3eM/MRSvPWKhFxWF12BfD7NhRvQGvZty/X/Xog5T27aUdfk8z8ODCVMk0EdVm//1xHxGqUkasfysx9gWOAiyih/CJgwXpdPdDFoJTxwB3ARzo6xmfm7ZSa/Y0ZBNOL1Jqt1wHfAl6idG14HvgasAEwPjN3onQdWTMihmTmDOi/nx1ruPqRltqtoyhHhhcBGwIfAf6WmcdGxP8DfmmfrTLnTGb+pV7/HLAZZcTKhZTOu1dRPqzDKV9Cu9cvJbXo1DS0C2Wusn9n5m8i4t3A6cAvMvMbdR1rCLup5TO9ObAT8G/gcsoPyLcpo8x+mplPtbGY3dZFbfJilKlqvgzcl5kvRMTRwIKZeVhEvD7LtCLqBVHmYXw98PfM/HtE/JwSQnZtqRldaKB3mq81W4cBi1MC1luB/SjN0z/IzMfqepsA3wE+n5lXtqm43WYNV/+yZv37Z8qR7mWUKQ3+CCwbEUtm5pcMWzMdHxE3RMT7KEc8ZwEvU/ocPUMJq09QOpDuaNjqWkvYOoBS+/IY8KuI2DMzb6Z80X2ihloMW91Xw9ZmlJGd5wPvAA6v+/UoyuCDXWIADN/vFMzfFBHLZBmevyBlZGKHRzquGLZ6JiKGt1z/PHA0sD1wVEQckZnbUUbr/aKlFmxG35e0d2Xmc5RBOs8An6P00foesAKwX0S8JSKWpoyaPmwghC2whqtfaDkKvhO4KTN3j4hlgddn5r0RsRbly/qjmfn39pa2/Tp98V9I6Vf0qcz8ee2/tQVlfpYfZeZ1bSxqv9ZpP44AzqH0h9iV0v9mKUpfrVMiYk3gycyc3I6yDmQR8VngN8BKlCa3HVv6br2f0tR4YxuLOFci4kBKbd0M4E+Uztr/S6m9e54yKOXjmXlH2wo5CETEWyjThHwPeJDS4vHRzHy8dpI/DLg0My+JiF8A+2fmf9tX4t4REa/LzJdr7em7KH0Cn6YctKxOaUJ8gvK+e11mPtxf+2x11u+PqgazeGXG8455ttYG3hUR52TmAzVsbUH5oB1m2IIaRFeq1zfOzJ0pX/pfA8jMf1HmA7oF+FhELNJF/4f5Xqew9UlgCCVsjaF8qb+P0kz0nYj4aGbeZtiaOxGxbpQJKV+mjBz7CrB1Zv47IraKiM9l5g0DLGxtSqnN2pEyqOf9wGcoU65cRPks7mTY6hUvAiOBzWsT2orA2+t9k4HbgHcDZOY2gyFsVRnlTAXnUL7Hz6U0o34ZuJPyPluGMrL3Yei/fbY6M3C1QUQsFRGLZ+ZLUSaV/FhErFA7/L0XWDcizq6rvwjsnZm/aluB+5cVgB9HxHeAb9U+Ih8CHoiICQC19uCnlHlanh4oH8a+1BK2tqfUZj1bv9QXBibV1WYAF1Dm4NLc+xSlo+95lGbtOzLz/igTNJ5EaSbp16Kc7eKNUU8fQ+kA/5fMnFYH7WwMfAjYLDN/nZnnZOakWW5QcxQRb41yRoz/UA4kj4yIdSnNZ1+JiHfWPlrPAytGxNCB0CTdXVlMokz0fWBm3gL8glIx8TVK0DysHlwPKIPmnzRQRMSKlC/ao6IMvR9OmXdn4yin9XgR2AHYIyJOzsyrM9ORYVVm/pUyd9H/AMd19BHJzE2AZyPib/X2fzo6VuoVrbV9Nex/BvhNZk6rX9r3A0NqZ9xjgK9k5r3tKe3A0kVN6v9RjsIfp/SNWzUiLqf8cH4uM3/T12WcG7V2/ZeUzv2Hxiuzd78jIkYC1O+r8ZQDQ/VQrdn5GfDDiFgly8S3n6VMjXA3cAVwZUR8qy4/MTNn5CCYAgJKn7V45bRh36Z0ayAzf0c5vdgQYMXMfKJNRewR+3D1sTr64gZgCvB3SifIdYG9KLMyX0KZj2cP4MrMvKpNRe03uhgZ9V7K/Fr7U/otXNty39nA12z+mr0o576bwSuTmh6SmTfU0LA2pSnjlnRS0zmKiJUpweqWiFiHMknj9bVW6HLgksw8qa47nNLv5IH+3O8kyjxPRwBfotQs7ELpJ/lSROwLfJLS5LMYpYlxG2u2ekdEnEGZwPMGSg3zMErfpdsy86cRsTFl6pupg+VgqHavWYoSNv9ImUD3rHr965l5cV1viRzAE1YbuPpYRAyl9OVYitJ08yZKNem6wMcpH67RlNN6/K4/fyn3hU59jbalzC/zh8y8OyJ2pwSGHSkddVfOzC+3r7QDQ+0AfzWwCXAXZVbwVYDzMvMP7SzbQBMRb6f8MHw9M8dHxD7A4cDJwI3AdEpwOWqgHATUfpJ/An6emZ+LiJWAmykzfC9GqRVdC3gP8BbgJPuX9kwdtPKGzLy9BvWDKLU5U4FVKU239wF7ZpnLbVBoGTC2YJYpRVagdBv5DqUZcQvKaaH2H6i1Wq0MXH2gjpwbTZk/69GIWB04g5LmF6f0mzmG0jFwVQbYqKW+0DIy6ipK5+6TMvPcKBOadgTVz9T2frXoKrRHxMGUE97uQOmAewClZuukrHObafbqSLFzgLMy88xaOzgMGEE5n9tWlA7zSwJfzcxr2lTUuRIRwyifsQ8Df6nXL6M0He5AmU1+q8x8an4/IOwNUSb4PI7y/f/rLDOm70Y5F+LllIPz/0cZrffjzNx9lhsbQFrC1gcpLTpTKa06V9aKiTGUEejbAttl5l1tLG7vyEwvDV4oRynXU754x1Mm53w75Y20E+Uo8QTKENcV2l3e/ngB1qMc7QyhHF3fTqlq37PevwSweLvL2d8vlPNxLt9y+wDKRIIjKSdCPwhYrt3lHAgXyiipKcD/1NsLUgZq7NyyzjDgeErt0FbtLnM3XlO0XF+EMgrxX8AFLcuHU6aoWbrd5R1MF8oJqLejTK3xJUpgvwTYsN6/GGUAxlvbXdZeft1b1s/HepTT9UymTOLaus4XKRUUr2t3eXt6sdN8w7J0Kt2DMk/NapTq0iMofbf2AP5Gqel6jnKeqPleF52P/0wZ8fVByrD6d1KGBx8ZER8DHk87yL9Gx36MiAVq38GjgUOiTFr5usw8ldJP5HeUH9KTM/O+9pV44MjMBymjN0fXJqCLgP9m5oUt6zybmYdRzo84vj9PT9Kp6X4pyo/bxcChwEJRzkAA5UBxdcrJqNUDre+HzLwjM39OaTocQ+lX+RLwjYhYI8vAixNzAI7MaxURy0bEcS2vfQ3KvH/DKRURRwFfi4iPtjxsOqUlaMC/5wxcfeMp4KuUkPAuSgf564DHgbdlGYlyXNoPovMX/+YRsRHw9sx8gPKh7OgkOonS3HF1DpIROr2pU1PPQllmbt4RWBb4fP0Lpfb1Shg4c9m0W5S5tcjMbSm1rvdQToV0YMs6a0fE6LreE/Vvv92/LZ+5L1BqE66MiE8A1wA/BraLiP+l1DDvlpnT21bYQaJln+9RQ8huwMPAbpTa0ymUmp+PxCtzNg50i1P6/X27hq6TgEcps8kfmJnnUL7bT46I5Wrz9gLA0Zn5UpvK3Gvsw9Ww2gnwC5SOtMtQ5uS5NjOPqu3UmZkvtLOM/UVHx8l6/QuUGq2bKEdBR1JOGfJTSugaCWyfmXe3qbgDQh1RNpoSTn9F+UL/ASXsJ+UL/YPWbM2dOi3C/Zn5ZET8EFiaMmHsCzVonQIcnGU4+4AQETtQ5vzbIsrM5c9n5k61dnQHSpPWrpl5W1sLOohExKcoozyPpQTd4zPztDpFy4KUJukzBvrBeEQsAyxHCVOrAvtSphI5iFJzdQ7lvLeLU7rafDdrn61aGz8oDqqt4Wre85Tq4Z2zjFL6OPD+iPhelvlTDFuUE1FTPnxExGrABpm5EWX/zaAEr6mUDpS/pgxDN2zNRsuX+emUDvJHU87ltzcwgTLqaXfDVve0NNGuC1wMnBMRQzNzb8r79Cd1yP53KJ3k+3XYqgd8rWYAp0XEoZSBPB2ds5fKzPMpn0nDVi+pQXZNSt+tpSjTBP2ghq1hmfl8Zn5uEIStd1Bq0U8HzqZUPPyAMkjgpBqmplAC/f9S5gW8K6rBErbAwNWYiFgmysldH6J0+hsXZTb5yZSTAa9R34gqgjIZ7BaUYPVkRJxMGX6+Y61O3hR4MDN/kM758xqtzQ5R5tlaDfgI5ewFj1M6QH8OGJWZ/5eZx6SnYOm2zMyI2JwyyOUsSs3rSRExLDN3onyf/go4MjMvb2NR56j2O9usNttsHRFbUWoXvgy8j1LrOaOOZv1ODZYDflh+fxER76E0lf0HuJRy4LNp/Z47gHKQPuDVmuBfUmaGfz+lU/y+lHD5HWDRiDg6Mw+vy8dm5i86ukT052b4eWHgakCdt+arwHm1D9KjwJOUkSZk5j3AFjkYhrn2UEQsFhELZeafKbUwn6FUOd9H6Tz6qcx8PiL2pnSoXGSWG5uP1Y7O69Xre1OauL5GOZr8cGZuSAkJawFbRsQb2lXWgagebA+l9L88NzNPoYz6HAGcXZvDtwfWyczxbSxqdy1EqVW5hBIgr8jM8ygHO9OBjSJif+ATlDnEZrStpINMlIlyD6T0Sb2dMmDqtHrfTpQa6H+3rYC9a11K14XHATLzEMp307L1YO87lDMwnJLlNGz31vUGVdDqMKTdBRgsWuYUicycEhFfpFQVH0hpDvsYsEJEfCwzn6mdmOdrtWnmXEo1+vmUkV7vpnxIr6O8P/83Im6k1NTsVEeH6bUC2C8ijgDeTJnP54l6hDmyhoV3UY4sT8rMp9pY1gGn/gDMiIjbgDdGxCJZ5tQ7gDKK9ghKzdbtHU2P/fFHo6Xm4KGIeJrSxHwZpSPzvyhdHr5Kmb7mTZSuEHe2rcCDQKcBLGTmf+rAi0Myc//a0rFDfS8tSpn0erCc4eGXlO+mIyLiWGClevvJev8dlFNd9bvPShPsNN8LWsLWRyiTHS5EOXHygxHxZkp/iEMo80UdlmWW9Pl+wsCIWJDSl+hpygdub2AcJajuTTlR7jaUviU35wAfEt2ETqM6twG+D5yTmYdGxAJZTsXyA0oz0YKUL3P74XRDy+d69bpoMqVWa3/gRMqB1Fspk1KuARyRLdNC9DddfedEOXffBynB6+zM/FNEjMzMf7YOYtG8iYhls4ywJiLeSRkxfFOtYT6L0lF8ImUevKWARzPz4bYVuJd0fPfU64tQOsJ/htJsvWaWSXPnu/eXgauX1D4Qx1DeWOdRJj3craOPTA0XpwGTM/PYthW0H4iIxYBn6oiuD1GO6pahzHR+KWU/3k2p0RrwQ4Gb0ilsfYBy5PgwZf/9gTK66eHaOXdpYIY1hN3TMTKq9tk6hzI9wgqU2de3oxxYDaUErQ9Tzn13f9ZzvvVntSZlNcoosRPr3/0or+8pYGVgnyxzP2ke1FrOpSmdxHemdIX4CqW7xG+BH1JaPyZn5tntKmdvq9/tT9UDvXdROsZPp0ydsidlzq0v1y4k8x37cM2jiFg1IjpG1QVlQsB9KZMCzqCcdPMXUc5bR03ytwCrdzE6aL5Q+8GsQjmP3z4RsQZl0s1NKOed+wrlA3o75UdtzzYVdUBoCVtfoEyb8Z/MvJkyDclY4BMRcSTwG2C6YWvOImIhgBq21qDMhL19Zu5GOb/gNZSDgn0pfZ82pfTj2g+4tR1lnp2O5s2W25+lfLZ+SKn1vIQSBk6hTB2yOmXOI8NWz0SWucp2pnSAf19mHgAcTPm++zSlNv//1WAy4EU5r+gpwOvrgcollMm9r6W85y6nVEacUCso5jsGrp75ZkQcVn/4DqMk+S8Au2Tm/pQjx5MjYpFawwXwjfm1A2rtO3Iv5ah6WeDrlC+db1Fq/xbLzJ9Q+sNcSTkS1GxExFjKl9nYzPxHHf00lHIkuQylJuPAzHy+jcUcECJiOPDJjoEcwDcpAxGeBcjMQyk/Hn8GFs1yvtOXKIMTdsn+OU3JzIO7KCMTl6c0028OPEg5+Pkp5VQ9P6JMt3J7Owo6WETE4vnKVAZvpPwOnBER29eBUh+lnObtD8ATwIA/S0aUqXz+f3tnHqZ1Xa7xz424QGRGbqSWmrmUHcWjGJULHhVxL5Fw4yiCS2ZuWUmae+pJMjdMwVLLJdHEzKgkFyo0lcQ1RSULNY4eQVNAY7nPH8934m0AG+Sd+b0z83yuay6v+c3g9Z253nl/z+957ue+ryWybmcTheUw20cTW5dHEaPE64iiq1M+/OVIcRmRtDawarm5rUO8+Y6yfW6ZVf+gfMwiXmRjbE+q7sSNgcKmYGvbd0nqQ7TbIfKzziYEuqsCZxcxcocxu6snzXU4krYkgm8fAnoQWpwNCb3E3UDXLLZaRtEzmbhhrEJ0qq8gOlujXOKjJF0A/Nz2b8vnPRpxCUHSrsDRRGf9Mdu3SVqN6MhdRsTIrEYsqMwkHn7md3Zt6fKg8NA6nHjYeY0oxHco+t6LgZNt31Lz/T1tz6zmtPVBEeI+gYhY27xcu4bocN1pe77CgHkgUeiv4Ii863TkluIyUFqm1wF/l/Qy8Wa8LfBwuQ9+W9Jkoo26HXCE7UkpkAdiVLirpBFEZ3V327NLh+Y0Qse1M5Er+Xs6ydbKstBMs7UNYZ3xBnA10dG6kBDgfokwq1xAdGCSFmD7uaJ3O5MouEYST+ejgYWSrrI90/bX4V82/hqx2NqN+DmuI27+e0h61PY0SfMJjWQ3YmT6E2B0ZxMwtwblIXG0pBeIUW3vcv1nkgyMVHia3VD+yaxqTlofikTkJ8R7+HBJP3F40j1NLJhMISwuHgM+R2R0dspiC7LD1WLKptL1hHHks0RbfivbQ0uF/ztiLXyUwhNpbaep5L8g6TQix+9Hto+tub4asDWxyXmM7WcrOmK7QNKJxGbZ08RD0zcdBrtIOoT4PQ5s0BFXw7GEruEmhD3CCsSoeyFwA6GFO7/RbxiSegL/RwS93yFpXSI65grbD5TR4hhgDtFx2Nlp/bBc1Gy0Ni1bnEyMDicS9g8Ly/cNIOJ6trP95rv8LxseRVzPEOBJ2+OLXvARYupzJOEm343wGduG8HMbV9V5G4EsuFqIpM8BE213KZ9vQrSID7f9ksLv6HHgHNvnVHjUhmEJN7IPE8ab+wKv2+kViQAAEWZJREFU2R5Rrve0PTPHiP8eSZ8mxq67SLqW0OgcSIxjP06IVo9IHc6yUUS8g1gU5r2A2CJbQHS4FhAPUQ9VdshlQNIehKi/r8OP7U7CePkRYvw8ieKHZHtGdSdt/zTrPG9NWDs8Xz7/LfCE7aMlHUR0oF9o72P+Ip85jjAwPa7Z1x4FJpWfeRtgU+DZUux36mlPFlzLQGnTj7K9oaTPE/qIgcQa7MIiHPyI7QmVHrTBkHQUId5+kZjr9yLGXlOBPwF9iOy5uZUdskEpLftVbT9aPu9LFKyvECPYfR1O/J8hOl4Lm7RGybtT05XYlPAvu5vQwa1O6OLmEN3CecBpbmdmxaWbcgmRPboREY68JpHo8Efgq864nrpRtoX3Iqw1phIj/pmEkHwGEbG1R0foJiqMW7cjft5ZwHdtz675elNhObCiIzYkWXAtI+VJ+Gbi5ra97TlFKCkvMnrr3FV8TadK0mHAUEJI+hCxgfh9IrT1BML9/ICO8CZUb8pr7QJCz/YE8UQ5nyhaexAxMgsVESx7Er5l7XpM0dZoUdrBKUVU/gnid7kJ8bufDfRor+NZSTsTo9BeXmTA2QXo2TSGTpYfSfsRneX+ki4lnPp/TnQZZxLWEM/bfrHCY9YFLTJU3oXQjm4GjAMurdUzlk7XYYRpdae9H9aSBdd7QNJOwHW21y2fd210XUdbUUavGxNbUQ9LGkkYR/YGDiae8ObVfH+739JpDcqG2RWEwebzxDLBU7ZPKRs/WxDaoqlEMXtwjhGXHUXg9+OET9kO5domhIHxxsRNdE6FR1xuSqdrJLCj04utLixBLrEtYQvUn4ghO5Eo5KcB57qDpTuUjvqNRLe0L9CTKCy/097/XlqT9OF6D9i+Gxgm6RVJH8xiKygj10uJLsyq5fJfiIDSg23v6nCXHyHpUIAsthZHUjfCSPcZYG55fR0FrF46MDcQRexLRKfrwCy2WkYR9iJpy/Jw0APYnMhHvB6gdLNuJsbc7f7mYXs84RP4y9LdSpaDZpqtT0jqDjxuexrxYHmOw2/rPkIw/nJ1p60PktaTtKeKMTBhLXKr7V8TSwC/IgqvrxZ9V7IEssO1HJSRzxzb91Z9lqop9g5XAwfZ/kPN9VMJ7cKVxMbO7sRYcbDtp6s4ayNToyvanNhEXI/oTjQ5hIv4PXYFjrf9t8oO205RZE6OIOxHNibCmqcQaQcv2d63wuO1GmpQv7D2iiJpZHfiwehNQvd3PPF3ejPweUIuMa2yQ9YJhY/YWUSixc8IO6TRRAf4/vI9txH6tbNy03zJpA/XcmD7F5CarUJvYoZfW2x9m9igm0/8gR5PBHsfksXWUlmB+H29RryxfQEYSxT2H1GE3m4F7EN0Z5JlQNJaxMp6P+K12ZcoshaUDdApCjPZRzva33QWW8tHs85WP2JhZUdJ4wi3+NnA5cTfbl9gaAcptuTwEduI0N32IN6TRhPxYT2JYPc1ia5wFltLIQuuOtDR3piXhZo3oY8RJpxN1wcQIbgDgR8D04mNKeUYcXEkbWx7qsOV+TOE9cN/SbqVMFDsKWkDRzTSxPKRLDtdiG3ZYwitzcG2ZxRx+b22P1Xp6ZKGpabY+jAhmRhXNrBXAY4snen1bV8haYw7iJFs+bl2J5IJXiEkIm8RyxivE12vN4hNxQ6lVas3Oc9PlouaYnMcsK0iyw8i6mGo7YeBHwHv2J6VxdbiKIwoR0i6rFx6kRDgUjqBNxJauLPKqDFpITWarTWKQH5G+TiCGMk+V5ZgLiKsE5JkqUgaQowOpwGHAofZ3q1YsxxLaJi6dZRiC/4ZZ3cqcKHtQURs2NeBLR3ZiP2IgPdbm/7ekiWTHa6kXjxAaGIGl63NBwEkHUDoHA6p8nANzmxi2eBoRU7fDwjbEQBsP17EznMJB/GkhZSn872BYUQU0oNEzEh34FRJE4lNq5NzzJ00pzwMvV2WfYYRetTzbT8taSywVim05hIWCEPau59g2dLtDfze9vTSAX4GeF+xhLhJkV5wraR/ALc1PXh35mlPS0jRfFI3FGHehxOt50eIN6GBhNYhfbaasYTV8i2Ip8ctgHWJJ+kNCU3IFGCC25n5ZtWUjusYoGmDtiuh3VqXyHZbEZhqe2JqMZNaJPUn4nkmlCLjUmIU3dv2o5I+RqQ7DCY60te2923h0qG6kEhZuIUYIZ5OdLheJoqrP0vqTXSFT7I9uarztjey4ErqSrE02ArYhbAtuDdFlIvTTIC7GbCC7SfKG9kRxO/vOKIw6Ab8MjswLaNm03Nlwl5jI+Ap4Bxii/b5Gj1ckiyGpD2BM4ht1sm2XyvXLyc69lu4xqVfHSiWTGFoejrw38B3iYe99YGPEtrRNQmX+S/bvqeiY7ZLsuBKkgpRxIHsT2wnTgFuAv5O6ENm2z65utO1P2qKrf7EDeN7hGP8hwjT3enFFmIQsdKem3vJv1BE8TcT3Zs/lK6PvCg9YxTRxf+0O2iMVtm8nGz7bEVayOmEp9idwBpExN0DVZ6xPZKi+SSpiDJC3Af4nO2tidXqAYQgdzTQvdgYJC2kFFufJYqtUUVL+CjwC6C3pO0JP6Ebs9hKlsI7hFbyGUkrOVjYJAi3/SXC1PQ3HU0kXvPznAusVN6jTiK6w8cQ24gjs9h6b6RoPknaiCVohGYDCwhz02nAdwjH5sG2L5N0ou13Kjhqu6UsF/QjdDXnl8tjgO2JImwu8E3bd6RmK1kK3YG1gA/afr28plyK+XWArW0fKWmtjvb6qfl5/gL0Ab5KbPOOAZB0vztA+kJVZMGVJG1AM83WB4hA6r8CfwT6SJpv+6+SfkZkJJLFVsuoGSNuC8wjRoi9gDGS9irauKdtj5L0Ptuzs9hKaqnVYJWx80Tgekn7236pWIosIDrQ20ga7xIG3hGx/YoiJeQSwoC56XeUxdZykCPFJGkDaoqtk4hQ2/uIN+8XCVfq8yWdD3wJuLuqc7Y3mmm2biJMKOcT2573ADdJWtsl79T27PLfLLaSf1Kjz9qsjNXOAn4D3K7IL11L0mBirHaJ7X9Ud9o2YwrwJLBdR1oKqJIUzSdJGyFpV0ILsTOx6bQd4V32GLApsAGxdv1cZYdsJ0hauakDWHRuNwNn2L6npghbkeh29QV26CQ3yWQZKFvB/2l7jKRjiJD4PxMaygeBoUB/wprlA8CJth+r6rxtjaRtgBVtT6r6LB2BHCkmSSsh6UPAQtuzyqU1gWfKOvlNkmYRW3R7276lqnO2NyRtCgyVdE3xd3sDeJ5FZrErExtVqwOnAB/NYitpTtFmrUOYNW9EWB/sRPi0DQDeb/s8RQKECeuWN5b2/+uI2H6o6jN0JHKkmCStgCJ7bDxwpSLEG8ILqklrhO1fAZMIgW7thlCyFIpn2Y3EksELTZeJG+cJALbfVgRRXwB0sz21gqMmDUzpgi4kxoYXApsTk+ZXbV8MTAZ2kDQU6Gr7rc5WbCX1JztcSVJnJO1GGCaeS2z7nFTGW48QRcJASf2ITL8dCY+b1BX9G4qp7sWEhuaH5ZoIMfN+wO8l9SK6XfsCZ3ZUn6TkvdNsYeIjwP3ASsDZkobbHm37akmrAJ8iHNeTZLnJgitJ6oiknoTn0362b5fUh9BsXQK8SVgV9AN2IGJB9rb9YlXnbWd0JbYQbweQ9BVCn7U5EZbemzCMXRk4zvZ9uY2YNKdmgeUrwEHAnsBdROE+XBKl6Lpc0gdqHeWTZHnIgitJ6ojtmZL2As6RNI3ocl0FXA3cCqxueyhwa63wO/n32H5T0lPAQ5JeJ8Ko7yciWMYCh9se3ezfZLGVAItZs/QnivO9bb9art1NWLJ8TdI829fkGDGpJ1lwJUmdsX2npAXECHGE7fMByhjxdklrlDf5FHK3AEkr2F4AYPtkSZMIQfwtwNu250r6YaWHTBqaZsXW+sRSxU9tvyipR9FozZX0AHAeobdMkrqSovkkaQVs/5JYJz9M0mrl8v5EEPXb5Xuy+/IuFINYbC+QtL2kE8o22bgy8plVbpJbAYcR2q0kWYyaYms40RFdDxgmqZdLxJOkQwlN5V055k9ag+xwJUkrYfsuSccDv1ME3g4mApPfrPhoDY+k7sBlku4h3PgvJ2wfdiQsNcYTmXfDgC8TQcNpGJssFUWOZn/itfIXSRsC9yoC5DcAjgAG5YNQ0lpkwZUkrYjt8SUW5KdAb9tPVn2mdsJC4DZgEOGLNMT2I5KGEQsHELmT04Cjs9hKmlNjgNuF6CwPIRZVPi3pr7bPKl54A8rXBxVftyRpFdJpPknaAEndM4esZdTcKN9HhE5fCtxo+7Ty9aFEp+vXwPXZkUia00yz1d32HEmrAqcRxfzNticv6fuTpLXIgitJkoahptjaDXg/YR67EzAcuMP2VeX7hgMP2H68utMmjY6kowgblj8SJqdPE5vDcwgt4B8qPF7SyUjRfJIkDUNNEPVFwKwiaP418H2gv6Rjy/eNzmIreTdKsXUg4SQ/ADgT+CxhSrwmMEDSytWdMOlspIYrSZKGoTjHHwJ80/YESV1KVM9E4v1quKTbgek5AkqWhqQ1COuQvQjt1kLgDuArhMHpsURWYvrgJW1GFlxJkjQEknYEViWyEZs2OVcG5gKrAXcDk5qMKpOkieYaLNuvlgzTjwN72N5J0pqE2elBwP22/7ea0yadlSy4kiSpHEn/QRhOfh5YF7hE0i7FmLIvMIpwBZ9e5TmTxqSZz9Z6wHzgGsI6ZG1JKxHjxBeAr9meXc1Jk85MFlxJklSKpM2ArwFP2p4BjCqZlBMljSX0N6dmsZU0p3b7t2Qj7g2cBYwk6rCzJU0G7iO6pwdmhzSpiiy4kiSpmlnAG8AGkvrYftD2OSVm5S1grO2Hc3U/qUXS7sCukkYCLxOdrf7AccCrwP+UWKjDJa1DxEC9Vt2Jk85O2kIkSdKm1Fg/bEkImF8i4o7OI/RaY2s9kpKkOZL2JOwdTrc9rixb3ECMo18Fvmh7XtlUnGF7XIXHTRIgbSGSJGljSrG1F/ADQsR8MbAlcAqwEjCk5CMmyWJIWhs4CRhWiq1VSufzGqAX8ONSbB1KdLvSPiRpCLLgSpKkTZG0OfANQps1nSi2jgP6AN8i3pfSlT9ZGu8A84C3Ja0CfKNkbh4DzASukjS6fL6f7Qw1TxqCHCkmSdKmSPoE0cnqSYibhxJeSdsCZwATbC+s7IBJQ1PGhycCuwKfBCYAvwOeAvYFphI5nF1SIJ80EimaT5KkVanRbG1I6Gum235T0onAt0so9WeBVYBXsthK3o3yWroSmEQI5W9vMjCVdATwWIrjk0YkC64kSVqVcoMcAJwN/BzoK2kwMTr8nqSuwPHAENtTKjxq0k4okU/3lw8AJO0PfIp4nSVJw5EFV5IkdUfShwmX+BeAjYmNsoHAFwg/pH/YvlCSCe3WCbYnVXTcpB0jqRfwRSLg/Iup2UoalSy4kiSpK5I2BW4l9FivEjl2PyJiVgYBB9ieXRzkLwKwvTB9tpL3yOvAs8A+tp+r+jBJsjRSNJ8kSd2QtD4xNrzI9tXl2mrAw8CKwCdtvyVpe+Bk4EjbL1d03CRJkjYjO1xJktSTfsBvbF8tqQth+bA+cD2Rk7iPpHmE59YZWWwlSdJZyIIrSZJ6Mg0YJqk/oavpBmwB/IrwTzoA+BMwwvb4HCMmSdJZyIIrSZJ68hAwFrgAeI5wkX+C6HItAL5VEzacxVaSJJ2G1HAlSVJ3JPW0PbPm8x2JTcX9gb9loZUkSWcjO1xJktSdpmJL0orALkQw9YjUbCVJ0lnJLMUkSVqFUmz1IWJYTrV9Z8VHSpIkqYwcKSZJ0mqUoutDtmekZitJks5MFlxJkiRJkiStTI4UkyRJkiRJWpksuJIkSZIkSVqZLLiSJEmSJElamSy4kiRJkiRJWpksuJIkSZIkSVqZLLiSJEmSJElamSy4kiRJkiRJWpn/BwdXfgQHBmeWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize = (10,6))\n",
    "ax.bar(train_df['intent'].value_counts().index,train_df['intent'].value_counts()\n",
    ")\n",
    "ax.set_title('The number of texts for each intent in the train_df')\n",
    "plt.xticks(rotation = 45)\n",
    "for bar in ax.patches: \n",
    "    \n",
    "  # Using Matplotlib's annotate function and \n",
    "  # passing the coordinates where the annotation shall be done \n",
    "  # x-coordinate: bar.get_x() + bar.get_width() / 2 \n",
    "  # y-coordinate: bar.get_height() \n",
    "  # free space to be left to make graph pleasing: (0, 8) \n",
    "  # ha and va stand for the horizontal and vertical alignment \n",
    "  ax.annotate(bar.get_height(),  \n",
    "                   (bar.get_x() + bar.get_width() / 2,  \n",
    "                    bar.get_height()), ha='center', va='center', \n",
    "                   size=9, xytext=(0, 8), \n",
    "                   textcoords='offset points') \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N1SrH6si4KV1"
   },
   "source": [
    "Plot a bar graph to count the number of texts for each intent in the **test_df**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "49FMqrDcG6us",
    "outputId": "1d18ebba-8bb8-4269-8c88-930d9eb9ad46"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RateBook                100\n",
       "GetWeather              100\n",
       "SearchScreeningEvent    100\n",
       "SearchCreativeWork      100\n",
       "BookRestaurant          100\n",
       "PlayMusic               100\n",
       "AddToPlaylist           100\n",
       "Name: intent, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df['intent'].value_counts()\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 467
    },
    "id": "iUw2t8op4KV1",
    "outputId": "3ae1c21f-1bb2-4c6d-fb7f-df57710f9a38",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAHCCAYAAAAtuofXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd5hlRbWG8XfBkHMYkAwKBsCAlyAiYgBRQERFBElKUlFUMIKBoChGgumKoIAJEUVQUMQEKorCBRVFFJEoYSQniev+sarl0PbM9EzvmdN9+v09Tz/TfVJX7znh21WrqiIzkSRJ0tjN0+8GSJIkDQqDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhpwoiIQyLiq/1ux6yKiBMi4kN9+t0REV+OiNsi4rf9aMNoRcSHIuJfEXFjv9syPbPyfxkRq0bE3REx75xu1+yIiB9ExO4dPdbqEZERMaWLxxsPIuLnEbFXz8/j/vmp8cFgpXGjfQgNfT0SEff1/Lxzv9s3QT0H2AJYOTM3HH5lRLw2In7ZxS8ay2NFxKrA24G1M/NxXbSn3zLzmsxcNDMfntltuw4mo/m/yMyXZOaJs/n4V0XE5rPXupk+9vMi4rqOHusx4WgMjzNwz0/NOQYrjRvtQ2jRzFwUuAZ4ac9lX+t3+8aD2ej9WA24KjPvmRPt6dCqwC2ZefOs3nGQekk0bs3281OTj8FKE838EXFSRNwVEX+KiPWHroiIFSPi2xExLSL+ERFvmd6DtCGdz0bEme2xLoiIJ7Tr/qv3oPfMt/UG/CoijoyI2yPiyoh4drv82oi4eYQhlmUj4pz2u86NiNV6HvvJ7bpbI+LyiNhhWDs/HxFnRcQ9wPNH+FtWjIgz2v2viIi92+V7AscBG7dev0OH3e8pwP/2XH97u3yBiPhERFwTETdFxP9GxELturMi4pM9j3FyRHxpBo+1VUT8uf3d10fEO0Zo/+bAOcCK7b4ntMu3bf/Ht7fj/5Se+1wVEe+OiD8A94wUrmZyXLeOiIsj4s72f3bIsPs+JyLOb7/72oh4bc/VS430vBnh9z/medT+hg+2585dEfGjiFi23fy89u/t7Rhs3O6zR0RcFjWUe/aw501GxBsi4m+tnZ+NMuL/xQjtG/6c/mX7f78t6vXzkunc7ytU0Phee/x39Vy9c3ve/Csi3ttzn3ki4j0R8feIuCUiTomIpUd47EWAH/Doc+Hu9vye7v0jYsGI+Gq7/PaI+F1ELB8RhwObAp9pj/OZkf6ent+9RUT8JSLuaLeNdvmIz09pujLTL7/G3RdwFbD5sMsOAf4NbAXMC3wE+E27bh7gIuADwPzA44ErgS2n8/gnALcAGwJTgK8BJ7frVgcSmNJz+58De7XvXws8BLyuteNDVA/bZ4EFgBcBdwGL9vyuu4DntuuPBn7ZrlsEuLY91hRgPeBf1JDD0H3vADZpf+OCI/wt5wGfAxYEngFMA17Q09ZfzuA4/9f1wJHAGcDSwGLA94CPtOseB9wMvADYuR3jxWbwWDcAm7bvlwKeOZ12PA+4rufnJwL3UMOY8wHvAq4A5u95flwCrAIsNMLjzey4Pg94ajumTwNuArZr163W/r92ar97GeAZM3vejNCG1el5HlHPob+3v22h9vMRM3jOvaz9zU9pv+t9wPk91yfwfWBJKuhMA148mv/36TynHwT2pp7TbwT+CcRoXp897f9i+9ueDtwPPKVd/1bgN8DK1GvgC8A3RvNcmNn9gddTz9GFW9v/B1h8+N84k2OxbPs/3779n+9Pvcb3ml6b/PJrel/2WGmi+WVmnpVVt/IV6g0cYANgamYelpkPZOaV1Jv8jjN4rNMy87eZ+RD1AfmMWWjHPzLzy60d36Q+4A/LzPsz80fAA8CaPbc/MzPPy8z7gfdSvQmrANtQQ3VfzsyHMvNi4NvAq3rue3pm/iozH8nMf/c2oj3GJsC7M/PfmXkJ1Uu12yz8Lb2PF8A+wP6ZeWtm3gV8mHYcM/NG6kP3RCog7tZuMz0PAmtHxOKZeVtm/t8om/Jq6pidk5kPAp+gPrCf3XObYzLz2sy8b4T7z/C4ZubPM/OP7Zj+AfgGsFm772uAH2fmNzLzwcy8pR3XIWN53nw5M//a2nzKTO77BirQXtZ+14eBZ/T2WlHB7PbMvAb42Sy2ZbirM/OL7Tl9IrACsPwsPsahmXlfZv4e+D2Pvj7fALw3M69rr4FDgO1H6mmcjhnd/0Eq/K6ZmQ9n5kWZeecstnsr4E+ZeWp7vh0FWKSu2WKw0kTT+2Z3L7Bge3Ndjeqqv33oCziIGX8wDH+sRWehHTf1fH8fQGYOv6z38a4d+iYz7wZuBVZs7d5oWLt3pnqG/uu+I1gRGApAQ64GVpqFv6XXVOrM/6Ke9vywXT7ke1TPwOWZObNi9VdSH1pXRw2BbjzKdqxI/R0AZOYj1HHo/btmdFxmeFwjYqOI+FnUsPEd1Af30LDcKlTP0vSM5XkzK/ddDTi6p/23UsNTvcdgLG2Zbtsy89727aw+3vTasxpwWs/fchnwMKMPbjO6/1eAs4GTI+KfEfGxiJhvFtu9Io99jSYzfn5J02XRpwbFtVQv0lodPNZQoffCwNCZ71hnAq0y9E1ELEoNs/2Tave5mbnFDO6bM7jun8DSEbFYT7haFbh+lO0a/tj/okLhOpk5vcc4nPpgWyMidsrMb0yvnZn5O+Bl7YPuzVQvzSrDbzeCf1JDdcB/etJW4bF/14yOy8yO69eBzwAvycx/R8RRPBqsrqWG+uamkf6Wa4HDc/Ymbszo2HRhVh//WmCPzPzVbD72zO5/KHBoRKwOnAVcDhw/C+28gce+RoPRPU+l/2KPlQbFb4G7ogqaF4qIeSNi3YjYYFYfKDOnUR/gu7TH2QMYsUB5FmwVVRA9P/BBqjbsWqpG5okRsWtEzNe+NoieQu2ZtPVa4HzgI62I92nAnsBo1/u6CVi5tWuoZ+iLwJERsRxARKwUEVu2759L1S3tBuwOfDoiVhrpsSJi/ojYOSKWaMMrdwKPjLJdpwBbR8QLWyh7O1Wzc/4o7z+z47oY1dP374jYkBr+G/I1YPOI2CEipkTEMhExliG20ZhGHZvH91z2v8CBEbEOQEQsERGvGunOI3jM/8UccBOPbevM/C9w+NAwZkRMjYiXzeCxl4mIJUZz/4h4fkQ8NWrG7J3U0OAjPY81mnaeCawTEa9oPeBvYewnU5qkDFYaCK0uZBuqxuQfVM/LccASM7rfDOwNvJMqVF6H0X+gT8/XgYOp4Zz/AXYBaL1ML6JqmP5JDaV8lCrQHa2dqOLhfwKnAQdn5o9Hed+fAn8CboyIf7XL3k0VTf8mIu4Efgw8KSIWB04C3pyZ12fmL6hegS+3M/yRHmtX4Kr2OG+ghuNmKjMvp47Rp6n/y5dSy288MMr7z+y47gscFhF3URMeTum57zXU8OXbqf+vS3i0VmiOaENvhwO/asNdz8rM01qbT27H71JgxJl6Ixjp/6JLHwHe19r6XzM9R3A0NSHiR+2Y/wbYaKQbZuZfqJq3K9vjrziT+z8OOJUKVZcB51LDg0O/d/uomY7HTK9xmfkvqv7uCOo1vxYwmt416b9EDSVLkiRprOyxkiRJ6ojF65KkgRcRm1KLj/6XrN0epE44FChJktQRhwIlSZI6Mi6GApdddtlcffXV+90MSZKkmbrooov+lZlTR7puXASr1VdfnQsvvLDfzZAkSZqpiLh6etc5FChJktQRg1UHttxyS6ZOncqHPvQhADKT/fbbj0033ZRtttmGW2+9FYBbb72VbbbZhk033ZT99tuPyTZxwOM0eh6r0fE4jZ7HavQ8VqPjcRqZwaoDxx9/PB//+Mf/8/PZZ5/Nvffeyy9+8Qt22GEHPvaxjwHwsY99jFe/+tX84he/4J577uHss8/uV5P7wuM0eh6r0fE4jZ7HavQ8VqPjcRqZwaoDK6+88mN+Pvfcc9lmm20AeOlLX8q55547w8snC4/T6HmsRsfjNHoeq9HzWI2Ox2lkBqs54JZbbmGppZYCYMkll+S2224Dqjt0ySWX/M/lQ92kk5XHafQ8VqPjcRo9j9XoeaxGx+NUDFZzwNJLL83tt98OwB133PGfJ9pSSy3FHXfc8Z/Ll1566b61cTzwOI2ex2p0PE6j57EaPY/V6HicisFqDthss80466yzADjrrLPYbLPNZnj5ZOVxGj2P1eh4nEbPYzV6HqvR8TiVcbGlzfrrr58TeR2rvffem/PPP5/777+fddddl+985zvst99+/OEPf2DxxRfnpJNOYpllluGWW25ht91248477+RpT3san/70p5lnnsmTbT1Oo+exGh2P0+h5rEbPYzU6k/k4RcRFmbn+iNcZrCRJkkZvRsFqYkdGSZKkcWSmwSoivhQRN0fEpT2XLR0R50TE39q/S7XLIyKOiYgrIuIPEfHMOdl4SZKk8WQ0PVYnAC8edtl7gJ9k5lrAT9rPAC8B1mpf+wCf76aZkiRJ499Mg1VmngcMX3TiZcCJ7fsTge16Lj8py2+AJSNiha4aK0mSNJ5Nmc37LZ+ZN7TvbwSWb9+vBFzbc7vr2mU3MExE7EP1arHqqqvOZjNGb/X3nDnHf8d4ctURW8/2fT1WozeZjpXHafQ8VqM3u8fK4zR6Hqu5a8zF61nTCmd5amFmHpuZ62fm+lOnTh1rMyRJkvpudoPVTUNDfO3fm9vl1wOr9Nxu5XaZJEnSwJvdYHUGsHv7fnfg9J7Ld2uzA58F3NEzZChJkjTQZlpjFRHfAJ4HLBsR1wEHA0cAp0TEnsDVwA7t5mcBWwFXAPcCr5sDbZYkSRqXZhqsMnOn6Vz1whFum8CbxtooSZKkiciV1yVJkjpisJIkSeqIwUqSJKkjBitJkqSOGKwkSZI6YrCSJEnqiMFKkiSpIwYrSZKkjhisJEmSOmKwkiRJ6ojBSpIkqSMGK0mSpI4YrCRJkjpisJIkSeqIwUqSJKkjBitJkqSOGKwkSZI6YrCSJEnqiMFKkiSpIwYrSZKkjhisJEmSOmKwkiRJ6ojBSpIkqSMGK0mSpI4YrCRJkjpisJIkSeqIwUqSJKkjBitJkqSOGKwkSZI6YrCSJEnqiMFKkiSpIwYrSZKkjhisJEmSOmKwkiRJ6ojBSpIkqSMGK0mSpI4YrCRJkjpisJIkSeqIwUqSJKkjBitJkqSOGKwkSZI6YrCSJEnqiMFKkiSpIwYrSZKkjhisJEmSOmKwkiRJ6ojBSpIkqSMGK0mSpI4YrCRJkjpisJIkSeqIwUqSJKkjBitJkqSOGKwkSZI6MqZgFRH7R8SfIuLSiPhGRCwYEWtExAURcUVEfDMi5u+qsZIkSePZbAeriFgJeAuwfmauC8wL7Ah8FDgyM9cEbgP27KKhkiRJ491YhwKnAAtFxBRgYeAG4AXAqe36E4Htxvg7JEmSJoTZDlaZeT3wCeAaKlDdAVwE3J6ZD7WbXQesNNZGSpIkTQRjGQpcCngZsAawIrAI8OJZuP8+EXFhRFw4bdq02W2GJEnSuDGWocDNgX9k5rTMfBD4DrAJsGQbGgRYGbh+pDtn5rGZuX5mrj916tQxNEOSJGl8GEuwugZ4VkQsHBEBvBD4M/AzYPt2m92B08fWREmSpIlhLDVWF1BF6v8H/LE91rHAu4EDIuIKYBng+A7aKUmSNO5NmflNpi8zDwYOHnbxlcCGY3lcSZKkiciV1yVJkjpisJIkSeqIwUqSJKkjBitJkqSOGKwkSZI6YrCSJEnqiMFKkiSpIwYrSZKkjhisJEmSOmKwkiRJ6ojBSpIkqSMGK0mSpI4YrCRJkjpisJIkSeqIwUqSJKkjBitJkqSOGKwkSZI6YrCSJEnqiMFKkiSpIwYrSZKkjhisJEmSOmKwkiRJ6ojBSpIkqSMGK0mSpI4YrCRJkjpisJIkSeqIwUqSJKkjBitJkqSOGKwkSZI6YrCSJEnqiMFKkiSpIwYrSZKkjhisJEmSOmKwkiRJ6ojBSpIkqSMGK0mSpI4YrCRJkjpisJIkSeqIwUqSJKkjBitJkqSOGKwkSZI6YrCSJEnqiMFKkiSpIwYrSZKkjhisJEmSOmKwkiRJ6ojBSpIkqSMGK0mSpI4YrCRJkjpisJIkSeqIwUqSJKkjBitJkqSOGKwkSZI6MqZgFRFLRsSpEfGXiLgsIjaOiKUj4pyI+Fv7d6muGitJkjSejbXH6mjgh5n5ZODpwGXAe4CfZOZawE/az5IkSQNvtoNVRCwBPBc4HiAzH8jM24GXASe2m50IbDfWRkqSJE0EY+mxWgOYBnw5Ii6OiOMiYhFg+cy8od3mRmD5sTZSkiRpIhhLsJoCPBP4fGauB9zDsGG/zEwgR7pzROwTERdGxIXTpk0bQzMkSZLGh7EEq+uA6zLzgvbzqVTQuikiVgBo/9480p0z89jMXD8z1586deoYmiFJkjQ+zHawyswbgWsj4kntohcCfwbOAHZvl+0OnD6mFkqSJE0QU8Z4//2Ar0XE/MCVwOuosHZKROwJXA3sMMbfIUmSNCGMKVhl5iXA+iNc9cKxPK4kSdJE5MrrkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0Zc7CKiHkj4uKI+H77eY2IuCAiroiIb0bE/GNvpiRJ0vjXRY/VW4HLen7+KHBkZq4J3Abs2cHvkCRJGvfGFKwiYmVga+C49nMALwBObTc5EdhuLL9DkiRpohhrj9VRwLuAR9rPywC3Z+ZD7efrgJXG+DskSZImhNkOVhGxDXBzZl40m/ffJyIujIgLp02bNrvNkCRJGjfG0mO1CbBtRFwFnEwNAR4NLBkRU9ptVgauH+nOmXlsZq6fmetPnTp1DM2QJEkaH2Y7WGXmgZm5cmauDuwI/DQzdwZ+BmzfbrY7cPqYWylJkjQBzIl1rN4NHBARV1A1V8fPgd8hSZI07kyZ+U1mLjN/Dvy8fX8lsGEXjytJkjSRuPK6JElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkdmO1hFxCoR8bOI+HNE/Cki3touXzoizomIv7V/l+quuZIkSePXWHqsHgLenplrA88C3hQRawPvAX6SmWsBP2k/S5IkDbzZDlaZeUNm/l/7/i7gMmAl4GXAie1mJwLbjbWRkiRJE0EnNVYRsTqwHnABsHxm3tCuuhFYvovfIUmSNN6NOVhFxKLAt4G3ZeadvddlZgI5nfvtExEXRsSF06ZNG2szJEmS+m5MwSoi5qNC1dcy8zvt4psiYoV2/QrAzSPdNzOPzcz1M3P9qVOnjqUZkiRJ48JYZgUGcDxwWWZ+queqM4Dd2/e7A6fPfvMkSZImjiljuO8mwK7AHyPiknbZQcARwCkRsSdwNbDD2JooSZI0Mcx2sMrMXwIxnatfOLuPK0mSNFG58rokSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSRwxWkiRJHTFYSZIkdcRgJUmS1BGDlSRJUkcMVpIkSR0xWEmSJHXEYCVJktQRg5UkSVJHDFaSJEkdMVhJkiR1xGAlSZLUEYOVJElSR+ZIsIqIF0fE5RFxRUS8Z078DkmSpPGm82AVEfMCnwVeAqwN7BQRa3f9eyRJksabOdFjtSFwRWZemZkPACcDL5sDv0eSJGlcmRPBaiXg2p6fr2uXSZIkDbTIzG4fMGJ74MWZuVf7eVdgo8x887Db7QPs0358EnB5pw0ZP5YF/tXvRkwQHqvR8TiNnsdq9DxWo+NxGr1BPlarZebUka6YMgd+2fXAKj0/r9wue4zMPBY4dg78/nElIi7MzPX73Y6JwGM1Oh6n0fNYjZ7HanQ8TqM3WY/VnBgK/B2wVkSsERHzAzsCZ8yB3yNJkjSudN5jlZkPRcSbgbOBeYEvZeafuv49kiRJ482cGAokM88CzpoTjz0BDfxwZ4c8VqPjcRo9j9XoeaxGx+M0epPyWHVevC5JkjRZuaWNJElSRwxWkiRJHTFYaSBERMzoZ0mS5gaD1ThjIJh1ERHZigUjYsuIWCgtHpzjImKBiFikfT/V5+6jDPqPiogFe76fIxOmxrOh//uIWKbfbZlo2t7DQ9/P18+2zAqD1TjTExA27ndbJoqeY7Yz8C5g0f62aPC1N7wNgF0jYhfgEGCpvjZqHOl5Tm4QEfNN1qAfEQsDO0bEMyLiXcAx/W7T3DR00hcRLwKOi4gV+t2miSIiFgA2jojVIuI5wNYTJZgbrMaJtqDqmu37NYFDJ/NZ7qyKiE2AnYCPZ+a03jMddS8zHwZuojZY/zjw48y8NSIm9XtKRKwaEYu1758JHNTnJvVVZt4L3AZ8D9gdOLK/LZp7ekLVc4DPAkdn5g39btcEsiCwJvBp4OvANW2dzHH/uTip3wTHi4hYCngLsEtErAjcBzzQXpQL9Ld149MIL67HUS/El0fE1PbBrzlg6Nhn5t+ocPVrYN2IWDEzH+lr4/okymLAicAb25n1v6hQMXT2zWQJnsNen2cD5wJ3AMu2Xqzp3XbCayfJK/b0Uq4NfD4zfx4RUwbt750TWii9A/gDsBHwK+pzkYnQ+zspXuTjXWbeBnwfWATYFVifekKRmff3sWnj0rCaqg0jYh3gNOBg4AFgh4hYup9tHFQ9Z+FrtZqR1wFvA5YG3tpus3JEPLuf7eyDyMy7qGPxQmBvYFXgjsx8cDK9joe9Pp8BTMnMXYAjgA8Dz2vXPS8iFpwIH5SzaGtgpZ6aoH9Tw1grZOZD7fWz2SR8jYxKz3vMpsAC1OvpQuB1EfG8dpulh+o7xyMXCO2j3jeg9vNzgO2A1YBNgR8CCwP/BO7OzPf1paHjVNs6aSfgEurNekPgJcCzqWN2fAut6lBEbAV8geqpmgd4FXVWvhfwBGAd4DWZeUHfGjkXDQsSm1B//45Uj9WK7d8bgQTuBN4zgGEC+K9jsS9V83gh1WP1BeClwL7ApcDOwP9k5tV9au4cExHLAT8GXk71Wh5I9e6eBiwOfBF4W2b+sm+NHMciYjvgA8C7M/OciFiDOombH7gbeD6we2Ze18dmTpfBqk+GvQE9B7gXuIY6838HsBDwS+CvQADXtqEX8Z8PsPdTNT5vAl6emZu267anQtZHDFbdiognUb0yXwUuAo4Dls7MrSJiCer/47rM/Gkfm9kX7XX8kczctB2nY6gg9U3gHmAx4IJBDBLDRcQrgC2oXuRnA5sAt1L1eE8Hngqcn5l/7VsjO9bT07JYZt4VEZ8CNqZOPFajQuULqefC0Zl5Wh+bO6603u/5M/OGVuD/HWCHzLw2Ip5CfS5eSh3LLYGvZeZ3+9fiGTNY9Vk7q3sjdfa/BfAK4EHgtVQyPz4zr+1bA8eJEXr3ngC8gOoR2ATYJjMfiIhtM/OMiFgkM+/pV3sHTZsMsARwOnA/sOdQQIiIk4A1gM2GaqyG/38NuojYmirM/mxmHt0uWxf4BLVv6mcmS/1ZG4b/OXBTZm7RLnsJFSruA47KzFv618Lu9YSqbakP//0z818R8TFgM+BVmXlNCw0Ptusm1WtkeqKW43gz8C3geuoE5GdUr97awLLUSM4emfm1iJi/vdeP2+NnjVUfRcQTqZky22TmPsAHgVOoN59TgPna92rP1Z5i/geB/YFtM/NF7YW2C7B/RCxjqOpGT6HtI5l5KzXJYh5gi6Eah8zcDbiOWn6Bdtm4fMPryggFyGcBfwK2HSpQz8xLqV7VFzDAS1EMPxbtebIXsEZEHNgu+wHwiz40b65ooWpzatmRz2Tmv9rl76KG/34SEY/PzBt6rhvo18hotJ6qeYBPAg9TQ8f3A4cCzwHOysxXUyUfT42IKZn5AIzv42eP1Vw0Qq/LEtQ03IOAGzLzwYg4DJgvMw+MiIXbdOVJKyI2yMzfte8PAF5EzRA5mSr2P4d6UU6lPsB2bR9oGqOes/AtgVcDVwNnUm+An6JmwJ2amXf3sZl9FbU+0cLAXzLzLxHxHeqDYeee3rsFBrV4fVhJw4E57dkAACAASURBVE7UGnJXZ+aPImI94PPAdzPziHabgetJ7nmdHEKNMpwCPBfYFvh9Zh4eER8FvmdN1aNaT9WBwJJUkHoC8Aaq/OULmXl7u93mwFHA2zPz7D41d5bYYzWXDHsDelxELJc1nXQ+aibgkFuHvpnsoar5SEScHxEbUGcwXwIeoWo37qXewO6kioN3MFR1p31YvIiayfVV4CnAezPzYurMfF9gp5gkSwhArTDf8/3bgcOAVwKHRMT7MvMV1Eym7/b05Dww91s6d/S8p72Z6s28Hfh+RLy2PU/eQM3mOqDdfqBCVfPU9u9vqV7bM6jlX34NLB8RS2fmuw1Vj5WZ/6YmNdwLHEDVUH0WWAl4Q0Q8PiKWpWaTHjhRQhXYYzXXRcRbqLP/B4ALqALXr1C9AfdTRde7Zeaf+tbIcWBYED2Zqs94fWZ+p9VXvZha3+T4zDy3j00daBHxVuBHwCrUUPUOPbVVz6aGCH/TxybONRHxeGpJic8CN1M9E6/KzDtasfqBwOmZeVpEfBfYNzP/2b8WzznDXp+rAydQdTA7U/Why1C1VMdExFOBuzLzqn60dU7p6an6M3BRZu4aEcsDC2fmPyLi6dQJyasy8y/9be34EhHzZOYjbdTmaVRt8T3USdza1NDfndTn4zyZect4rqkabtKcaY4HEbEF1Tu1A1Xg+GxgP2qJgFOooPVqQ1UsT32QExEvyMwdqWNzKEBm/p1a9+sS4DURscgINS8ag4jYKGqRy0eoWW3vB16WmVdHxFYRcUBmnj9ZQlXzELAWsGUbplgZeHK77irgj8B6AJm53SQJVXsAU6hQtSkVIjagyhuOiohXZeYfBylUxaO7OgytU/UM4GkRcUJm3tRC1Yup9/QDDVUjyqgdRk6g3sdPpIbUDwL+TB275agZx7fA+K6pGs5gNQdFrcC7eLQtLqhC9N9l5vWtgPEFwDbAizLzh5l5QmZe0bcGjx8rAV+LiKOAT7Zas22AmyLi5wCt1+RUap2TeybSi26CeD1VSHoSNcz6p8y8MWqBviOpbvtJISKeELWa/zVUuD84IjaihijeHxHrthqq+4GVI2L+QR4e7QlVr6R6p+5rQXNBYOj96wHgG9QaVgMhIpaJiCUz8+GohU9fExErtWLq/wE2iogvt5s/BOyVmd/vW4PHsSxXUIunviUzLwG+S4XVQ6mTlAPbSfSEM7Av/n5rZyzfo4p83xOPrhL7lIhYCyAzH6JmEz3Un1aOT5n5f9QaSW8EPjxUa5aZmwP3RcTv28/XDBU4amxG6PH7JnW2eAdVO7NmRJxJhYkDMvNHc7uN/dDOqr9NbaC7Rtaip2+lpodfDvwAODsiPtku/3hmPpADuLRC73OkBYv9gB9l5vUtSN4ITGkF/B8E3p+Z/+hPa7sVEStTJxOHRC0nMZVas+0FUdvXPARsD+weEUdn5o8zc2BnQY5FREyNR7c1+hQ1bExmnkdtfzQFWDkz7+xTE8fMGqs5IGpNm/cB76YS+E5UfdDDEbEPsAfVBboENTS43WTvqRo+fh4R/0OtT7UvVavy057rvgwcOkjDC/0SEatSAeqSiNiQWqTvl62X9UzgtMw8st12KlXvcNNEqncYq4g4llqU8HyqF2Yhqg7kj5l5akS8gFr+47pBCRIzErW36QM8uvjnuzLz/Ba8nkENl16Sg7X454LU//+1wF+oSQsbAXtSO2ScRq3ltjtwdmae06emjlttCHUZ6kTl19SC2F9q338oM7/VbrdUTvCFnQ1WHWv1QRcA38nMAyJiFeBiaiXZJaizvKcDzwQeDxw52cfgh9VsvJxa8+dXmXl5ROxKvYHvQBX2r5qZB/WvtYMjIp5MvbF9KDPPioi9gfcCRwO/AaZRJwiHTLYQ2wqyF83MS1vIfBt1Jn0dsCY1jH8D8NqsdZsmhVaI/mNgc+AyapeINYCTMvNX/WzbnBQR81N1hstQw52Po4asNgJ2o8L2JtRWTudNphOPmekp8p8va0mhlahyj6Oo4b8XU1uQ7TuRe6l6Gaw6FhELUYWcLwV+174/gxry255aXX2rzLzbF99j9cyYPIc6bkdm5olRC38OvXnt18bjNQZtFtsJwJcy84utt2EhYHVqH66tqML1pYEPZOZP+tTUuS5q0cIPU8W0P8xa7XkXaq+/M6kP149SM5m+lpm7TvfBJriR3qMiYn9qk+ntqaL9N1M9VUdmW3NuEETNPt6EWn/qtohYGziW6nFZkqop+yD1PFmTSTRDdrR6QtXWVG/edVSP3tktrG5Kzfh+OfCKzLysj83tTmb61cEXLaS27xehZv39HfhGz+VTqem3y/a7vePtC3gWdfYyherVu5Qadnltu34pYMl+t3MQvqjZNtcCb2w/z0dNBNix5zYLAR+helu36neb+3CM1qGC09XUkP7zqeGe57brl6CK+5/Q77bOpeOxMbBiz89vphZyXIvaVPhtwAr9bmeHf+8Uaq/WR6iT4hdRM0BfSJ38PRP4GLUcwEr9bu94/qJmvV/c3uO/SIXxnYfd5p1UaJ2n3+3t4svi9Q4MG8pahnpyfAt4D7BA1IrEUC/KtalNlSe1EYqlf0vNRNuamta/LjXt9uCIeA1wR1qo3onMvJmarbVJG+Y6BfhnZp7cc5v7MvNAav+/sybDcha9f2Nm/ikzv0MN+W1K1RI9DBwREetkFfV/PCforKWZGToWETFvqy86DHhX1OLG82TmZ6iao/OoE8ajM/OG/rW4W1nF6LtTaww+kRq6eh91HHYHfk/1XP2b2ttOTUQsHxEf7nk9rUOtbzaVCqeHAIdGxKt67jaN6gUciPcZg1UHekLVO6jUfXZEvA74CfA14BUR8RWqJ2aXzJzWt8aOA8OC6JYR8XzgyZl5E/XiGyoAvoIaTv1xDuAsq36IWpuKzHw5dVZ+JbUFyVt6bvOMiNik3e7O9u/AD1n3PCd3bx8MuwC3ALtQPXzXUmfd28ajaxkNnGHDfwtkrZC9A7A88Pb2L1SPztkwsM+Pu4EPUCd4T6MK1c8F7gCelDVD9MM5yWtkR7AkVT/8qRaujgRuo1ZXf0tmnkC9tx8dESu08pl5gcMy8+E+tblT1lh1JCK2p9YteXHUqsv3Z+ar29ne9tSwwc6Z+ce+NrTPhgoY2/fvoHqoLqLOag6mtvQ5lQpXawGvzMzL+9TcgdSW+7gxM++KiOOo3eNflVVYugk1vLF/1vTnSSUiXk/N1D2cOkn6SGZ+ri0nMB81PHrsZPgwbTOYN6FObr5PhcwvUMEiqZC59SD1VA1pBdbvoCZzLEet5/bTzDyk1Qbl0PuYSkQsB6xAhaY1gX2opYTeRvVEnUDt67okNZz66Ww1Va0XdGBOnu2xmk3txdXrAeBzEfEeqqhxqKB1mcz8KvAcQ1VsQL3IiIgnUsfk+dTCig9QAes6qpDxh9QyFIaqDvQM7WwEfAs4ISLmz8y9qOP/9bZswFFUsfpkDFULUvu+vYIqUP8L8IUWqhbKzPsz84BJEqqGAubnqUL1w6i9IvcCfk7NiNx1EENVcz81/Ltj1ozY3YBnR8Rns9YpM1T1iIinUL2Xnwe+TIXRL1CF/Ue20HQt1cHwFWr9s8uiGaRQBQar2dLqUl7UujFfFhFbUSn8IGoTzq0z84E2e+ao9gE2ENNIxyioBfZeTAWouyLiaGr5iR1aN/AWwM2Z+YWc5Gt7dSkzMyK2pApuv0T1EB4ZEQtl5qup94LvAwdn5pl9bGpfRMQzqeGIa4DTqdCwRXtOvpn6kB1YvUObUetUPRHYllpR/A5qIs4BwPqZ+c3M/GAO4NZbEbFcRCyXtTPGO4HnRa2ufhW1ofQ6LUSoaT3g36NWSn82VZy+D3VichSwWEQclpnvbZdvlpnfHRpyHsRhZIPV7FmAOqM9jfqg+kFmnkSFhWnA8yNiX+B11BpAA7u7/WhExBIRsUBm/pY6+92P6iq+gSoMfn1m3h8Re1GFjYtM98E0y9pJ4fxUjciJmXkMNctrdeDLbXj2lcCGmXlWH5vaF1GLpL6Fqu+7lCpI/ly77tVUL83VfWvgHNYm3Dyrfb8XNTR8KNXr8NLMfC4Vxp8OvCQiFu1XW+ekqDUHPwCc1Oo+bwPuomaAkplXAi/OQVkSoDsbUUPDdwBk5ruo59DyLXwfRe3ccEzW9mP/aLcbuEA1ZEq/GzCR9CTsf0XEPVTX+BlUod7fqe7iD1BTcx9HdSP/uW8NHgfa0NOJ1JDKV6kZaOtRL8ZzqefgVyLiN9QZ8qvbrDV1pL2BPRARfwQWj4hFstbleTM1G/N9VE/VpUNDhoP8pjesOJvMvKYV9b8rM/dtPRLbt+OzGLXo48CsIj6CAN4QEe8DVqPW7rqz9USs1UL506geiCMz8+4+trVTQ8+F9u+1EfFOaij4LVRpwmuAlSLiNZl5byvk12N9j3oOvS8iDgdWaT/f1a7/E7UV1sC+pwxn8fooDX8zbpetSRVfPwX4cmZeEBFrZebfeou0J7OImI+qybiHemHtBTyPevPai9qYejuqxuriHNDp63NbzwfG2u2iq6heqn2Bj1MfGk+gFrpcB3hf9iy3MKgiYvms2adExLrUrLeLWi/Ml6ji2guptZmWAW7LzFv61uA5qPc9LSK2A/4XOCEz3xMR82ZtwfUFqrxhPipgDkydaM9rZFtqQdwFqE3db46I1aha2XdRa+gdmLUThIs6N0PPkfb9IlRB+n5UWcxTsxbBnpSfgwarWdTOYp9IzXb4ePv3DdQ6J3cDqwJ7Z61zM2lFxBLAvW2m2TbUmf9y1ArNp1MrFl9O9VANxBTb8WJohk2rqTqBWvZjJWo3gFdQHyLzU4HqpdQ+eDdm26trELWeuGWpwtodqeHm91ND0j8DjqN6Ka7KzC/3q51zy7BQ9UKqh+EW6nX5K2rm4y2toH9Z4IFB7Elu9bEfpELBSdTCuLsM1Y+1E8PPUc+Lw/vW0HGkvbff3YL306gC9WnU0i2vpdasOqiVfkxK1ljNwNCwSM/Pb6U+mI6jzuJOo96gj6GmJK9NrcUxaUNVq+dZg9pPbO+IWIdaRHBzav+591MvxEupY/naPjV14ETEAgAtVK1DrXj8yszchdq/8idUqN2Hqg3cgqqzegPwh360eS6KrPXjdqQK0TfIzDcD+1PPzTdRPakfbR8WA60nVL2DWubkmsy8mFpiYDPgdRFxMPAjYNqghKqIWDMihmYmB7Vo8z7Ue/cD1IbA343aE5HW23IJsHb890zwSSdqf9FjgIXbidtp1IKpP6Xez8+kAurHWmidlAxWM/afF1LUTMAVqWGrLYGbqfBwKrVFzfHU8gCX9qOh40WrQfsH1Zu3PPAh6gPrk9SZ3xKZ+XWqrudsqrdAYxQRU4E9hiYKAJ+gCpLvA8jM91Bvfr8FFsva0+xhqkh5pxzgZS0iYsl8dDr34lQv87ER8cpWiPwqatuSXwF3ApNihf+I2Iz6MNwsM//aZkbOT/U4LEf1zL8lM+/vYzPnhE9ExIEtXB5I9ba8g3od7Es9P46OiEVajxXAEU5CiidS9bLnUKUd+1NrN76Rmjn7BmoI8CQqXA1EGJ8dDgVOR0S8CHgjdbbyh8w8LSKWpM7wP0NtdbEkVYB9KxUeHprM4+9R07TXz8xzImJDaggBan+oD1IF/YsDH2zF0wO1KFw/tXq/pN7wFqTOvj9P9VR9Ltt2QBHxUeD7mfmL9vOig1SMPFzUGlR7UkHhFurDc7NWV3M08M7MPLXn9ktn5q39ae2cNbw+KCKeQW02/TtgUapW9PFUncxPgSmDEqoi4nHA4i1ArkSdYHwuMw9v9UFfal+3UQHhuMw8v38tHl+iNm3/MbW12LrtshOoHqszM/OhqAVlt6c6HubN2hZoUnJW4Aii1lk6lErdywFbR8TvM/PKiHiIqg1aiBpq+SbwxclYoDeChan1vQ6iekO3ysx72pnx+6k6q82pPbZ+xSSaJTKnZeYVrR7mUCpYfZI6i/wi8EhEHJuZt2bmu+ExM1wHNlRBDYsCX4yIq6hh+/Xa5WdERAKfjFpn7uvtLrf1p6Vz1rCaqg2opU7uAI6neqg+QRXt70stavww1aM54bXhq5OAOyPin9QJx0bAhe2wfDgiLqKGtDYF9snM8y1UL62045vUe/jeEfHNrLXv/kJNiLmEWo7kD8BzqL1yJ22oAnus/ktELA38i9oI+HsRsTK1vcXnM/M3bUjwOOBeKplvnpN8SYVeEfF+aj+xr2Tmfj2XLwmsT82yeVNm/q1PTRwoI/RCPIla9mNeauj1EeDrVK3MEZPlDa9nxtdQIf87qSG/86hlFR5pt3sJtU3Nppl51wweciBExAHUTOa/UCfW781aDJOI2JV6fW4/KEPDbVbs16jFTf9GlXI8MzP3aK+VX1JLjXwuaj2vx+UALnw6u6K2qdkN+FNm/qDVpV1M9fi9nlpdfSFq7bcNqHUbv9uv9o4XBqsRRMTWVHHvxlnruZxJLRJ3MdVtfj5tnY7MvLF/Le2/ET7YV6QWEtwOuCUzD2qXL52Ztzr8171WJLoDj26K+zA1w+1hqsfqYeoD43d9a+RcNKx3Zn1qyYS/t59/AVyamW+MiJ2pXpqrBmXIa0Yi4lnUMPwWEXEiVU/1Gmp4fi2qKHmfQaoTjYjnAOdl5jzt5ydRQ8B7Zub1UWt1/RH4UGZ+qI9NHXfaEOlbqYU+3zrsut8D57fX0QbAk4G/tc6HSd/TZ7CajnYmewy1Z92a1Iasy1Erh/8f8I50m5r/iIg3UMWu11Hj7itQwwp/BS4DNqT2oLuvb40cID09Mk+m1h/6KVUnsyxVN3Mv1fvwIPD+nIQLG7YZby+llkH5KzXcdStVfHsjtV3L1oPa49yGcBbPzN+3nzemTnhupobkt8va8eDZVA/WI0O1eIOklXZ8LjMfHxEvp2pnt6eWDHikFWWvmpk/7mtDx5moRXM3pV5DtwGfysx7eq4fOinZvk9NHLcMVjMQEZtTQygr5KOLCs4DLD3UfT5Z9fY8RcTrgD2oIuHfUTP+/pfa0HZ/atXmnQb1A6xf4tFV7Q9skyvWBrYBnkQt/HkPsOigDOvMioh4JdX7smVEfJraDeH7VE/0rdSSC3/PzOv62Mw5pvVifpSqY7yU6nl4iDrpWZTavuiRqK23tqHWkxvYodB2PE6hAuRzM/Pe9l4e+egil5O+p2VIPLpA7BZUDd5TgO8Cn+6ty2w9V6+jFnf22DUGq5loPVefBJ6XA7KWy1i17vUnUrMlL4yIT1ILUa4H7EL1AjzYc/uBnWnVT1Eb5/6RWmdos3bZk6jFDp9IBYt7+9jEuWaEIemNqGn0W1JbJR1AhdArgcNzgFYQH67NaP48tRDs36nJIn/OzAPbzK2nU7V3f6VOhnYZpOG/6YmIFwAnZebK7ecpk6XmcHa0nsxvUKM0GwNLUyclH58s7yuzy3WsZiIzf0CtdfLDdoYzqbVu9U9TZ7+Lt4uvpjba3CUzX5S12vpBEfFaAENVN1rhKBHxjBZuFwXWpfb/+xpA6506hRp2nRRvfsNqqtaOiIWBP2ZtmrseVT9zGbU0yr+Bf/avtXNWRCxELXp5OXBfCw5vAJZtPZpfp06CrqeeP6+ZDKEKIDN/CuwVETdHxFKGqseKiFUiYptoCw1TSwt9OzN/RE3wOJsKWO9o9VeaDnusRikGfL2f0YhaNuF4YOfMvKDn8vdR9SpfoGZdbUUNB+6YmX/pR1sHVdSebgdRy1U8kdr0+xJqVfvrM3O7Pjavr6JW1N6KChV3UbVmb6MWwTwFeDk1JH1l3xo5B/XU3a1LzfxbheptH9oxIqjX5xTgbZl5Q98a20dtWPDezPx5v9synkSt7XYYtRL/GdSSFF+ker5/3W5zGlWzeJgzu6fPdaxGabKHqmY9aoy9N1R9mJpZ9BD1QnwbtZnproaqbkXE8tQU5+dTx3xjKkw93GZ8XRK16OPvJ0O9w7CequdTxdjPi4jvUqun3wN8lloYdGNgj0ENVc281OvwFuqD8RXAt6gQsWrURtPPBF5G9VZNSpl5FlhT1asdizOiFhren3p+fIsKVq+LWoboKmoC1wcMVTNmsNJM9bwBPYFaVHDo8pdQm05vD3wVuJaaSRkO/80R81CzLt9E1Q3tkpk3tkkWP8/Mp/a1dXNZT6hakRqW/m6bnbog8PrWe7N6Zn4+Io7LAV3ENyKemJl/zVr9+tnUkgovjIhvU4uiLh0Ra2RtNXVe+5r0DFWPaq+VragdRW6mSjvupiZv3U71Yt1BzQwc2PrErkz6miHNXM8b0HeBjaL2FIPa4mCPzLwQ+Apwf2beZqjqRk9N1dRWqH5j+9qHGsq5ohXkHkktCTLpRMRu1JDfldSG3q/LzBe3ZQT2o+pBFhrgULUYcFBEfKZddB1VtE/rMf4GVQN5WBsilP5L1JY/7wM+kZk7UNsavRt4Rtbef8+nNnT/9tD7kqbPHivNit9QtT07thk1vwWIiJ2o2pZd+9m4QdPOIrcF9qK2IPkttW3EwsD7IuI8asbOOyfLsGsLEv9uEyT2omr7jsjMv0TEt4DlW6C6j5oGvlsO9tpp91CTSd4YtQ/kl6glBQDIzD+2STf3UTtKSEOzh9cDfpWZ17ae78uBRdpSCydH7TpyYkQ8AJw2dIJtT9/MWbyuWRK1gemeVJfxxdQb9vZUfYvrVHWo9QweBwzNxJxC1VatTO3JNR/w18w8bzLUi0TEltS2ND9ub/yfpoZF18vM30fEE6gVxHekem1OHNQZbyMsL/F0qpfh6dTz48PUhsq3UJMbfpyTcJFY/bfW4/QJaneGU6mhv4OpHqt/UiHqHxGxHtUb/vbMvKhf7Z2IDFaaZW1K9zOBLahp2z+3mLEbPTO7FqCmza8J/Bn4EDUb8+899TKTRkRsAxxCzYi8KDNvaZd/luotfXr27IQQA7x10rCi/acA82bmpe2DcB/qdflWKmAtBPxwsvRoanSiFv48mNp4+lNU+F4dWI2qwVuOWnX9zZn5sz41c8IyWEnjRE+o2pJ6wzuKWj17GWrR1Wvbcgs7UFOgJ8VM1Vacfgp15nxBO+OOfHTl/89RPajPygHckmV6orbseRU1G/AS4GTgTqrW7J7MfGf/Wqfxrs2evSgzPxi1e8bB1DpvZwJTqW2AftPPNk5UFq9L40QLVZtQoepzrYbt98BZwHoR8VxqnZlvTJZQ1dxP1QddHhHzZ3lkqIg2M/elFv/8yWQprG1Dfy8DnpOZ61NT4V9CFfF/EVi4Lc8hPUbPa+RwYP72XHo71Sv+Jmr23ycNVbPP4nVpnGhFxs+naoSOaBcfBzyXClv3Ae/NzO9NhpqqHgsDywNLZebt7ThlC6IrAetn5usjYvlBPSYj/H/fAzxMLQJ6JfBxamXsHTPzMxFxQGbe34emapzreR5dDWwIvIOaZXwcQET8OifJrg1zisFK6qOe4b+NgAepob8VgOMi4qWtduYvmfm5iFgkM++ZDKGqt0aqDYGeB3wtIl6Vmde35ScepnppNoiIH2TbKH3QDKupWoLaWPka4P+ADSPiocy8JiLOoPYAxFClmcnMm6N2zTiGWlB26HVnqBojhwKlPhlWU3UytbDlQ9Tsrp8BJ0fE47LtaZaZ97R/BzpUAfTUTz2lDV0cBvwEOD1qz7vlI2JHaujimMx8oH+tnbN6QtXbqY2kz6UC5XXUivJHRMQRwL7AT/vVTk1IlwB/AjYd5Akfc5vF69JcFhELDPUotDqYU4BDMvNnPWFrPqr3amNgs0EODr3azLb/yczjIuJN1AbC/6Dqhn4L7AFsSS0jsARwQGb+oV/tnVsi4kVUDczm1CzITak15f4APBlYg5omf0XfGqkJKSI2AObLzPP73ZZB4VCgNBdFxJOBPSLihLbu1x3A33l0UccFqJk5ywIHAqtNolA1D7AStQDtmtT07xdQa3e9BFgsMz/SVhlPapmBO6b3eBNZRCwDPJKZt7WLlgMub0tKnBwRt1GzRrfNzFP71U5NfJn5u363YdA4FCjNJW3NoW9QxcZXDV1MhYn9ATLz31EbKn8UWCgz/9qHps51rafuEWq47xPAutQo2LTMPBq4CNgsIvYApmTm3QMcqrYCfgB8IWqTc6i1zIZq8cjMs4HzqaL+3plekvrMHitpLmiLqh5N1QN9uV0WVAH2K4FfRcQKVO/VdsChk2VNpmHF+KsCvwbmBz4YEXtn5hcz8/iIWBB4KrVa9ECKiBdTi6AeTs3aensbFr6YCuPbR8TzqT0jn0etPTQp6u6kicJgJc0dU6hZf6cDRMRbqPqpdanNrNejFnZcAHhrZp47GWb/wWOKs98C7AxsA5xDhc69I4IWrj4bEUv0rrA+SCJiaWrNsldm5ukRsSFVU3UMcBe1BMfzgc2orXu2zczr+tVeSSMzWElzQWbeFRF/Bn4XEbdTmyr/mtqm5VvAnpn5xWH3GehQNWwZgS2pYLltZk5rl/2UWj7gXRHxYGaeMKjDfwCZeWtEvBT4UERcSfVaHQscD3wbWDYz9wC+3TsBQtL4YrCS5qConeIfBsjMd0bE+VRh+qnA/7d396F+1nUYx9/XSJ1WMgzFZcIUispK64+VCLIEXavMMrep0fDhzIoyH0KxpWYaWZGsRynFEswCh7ZhstKVJTKttKKyokSsla1GrvBhq7Vd/fH5nvbjlzoP59bb7b5esD/OOb8/vjtw4OL+fu7rs9n2Jklf6/WQPRgLVXOogf2bbP9J0gvaDNUmSXcDl1MzRrs827dI2kpd/S2z/UmAdv23StK+LXgO4oWGiJ1RhtcjngGtyBHbWyUdKemc9qbbynattbEFh9cBp1KzVYMxEqqWUk/tDgQmJM12W9cj6RRqjui2IV152f4OVSlxqqRZ7dsLqYXKm9tndumnmRE7szyxiuiYpL2AL0q6nWrH/hJVpzCPelV+NbX/bgL4ALVceHDFWeeNHgAABSpJREFUjqrdh/Op//8fJB0M/EC1XPgg4Axg0RBDhO3bJJ0N3KlaMn0itXj7kZ6PFhE7kGAV0b1twLeARVQH0xLbP5M0QQ0eQ+11ewB431BC1Uj56Qzq6csSagj7DZL+aPvS1s+0oP18Uev6GiTbq9vqnpuA19q+r+8zRcSOpXk9okMj4eH51PLkLwDftH1R+/lp1JOrW4Hrh/I0Zmymai/bj0vaG7iICqI32L73iT4/dJO/r77PERFPT4JVREdGQtWbgBdSJY9HAUuBm21f1T63FLjb9i/7O20/JL2Xqgz4KVUG+lvq7bfHqfmzH/V4vIiIacvwekRHRhYqLwc2tiHsW4EvA/Mlndk+d/WAQ9XJVLP6AuBjwBFUIeZ+wAJJe/R3woiI6cuMVURHWpP6u4GP2F7TtsVvlnQH9be2VNIqYN3Qrrkk7UvVTBxLzVZtA24GPkgVgZ5J7QJMN1NE7NQSrCI6IGkesDe1+2/yza09gE3ALOD7wNrJ8std3fiMlO0Nbe/dS4G32D5K0n5UKei7gLts/7Wf00ZEdCfBKmKaJL2GKrF8B/AS4POSjm5ll4cDV1KN4uv6POezaayn6kDgP8C1VM3E/pJ2p64BHwTOt/1YPyeNiOhWglXENEh6BXA+cJ/t9cCVbefbHZJWULNEFw4lVI2+wdZ2/70NuBS4gspbl0m6F/gh9YTv5KE8xYuIYUiwipiejcA/gYMkzbX9Y9sfb6tYHgVW2L5nCPUBkt4MHCPpCuAh6knVfOAsYAPw6bbi53RJB1Arff7e34kjIrqXuoWIKRipVDiMGrr+M7Vm5HJqnmrFaB/TUEh6K1Wb8FHbK9sg/zeoq9ENwGLbW9qbgettr+zxuBERz5jULURMQQtVxwJfpQavPwccBnwY2B1Y0vb/DYak/YEPARMtVM1sT+euBWYDX2+h6hTq6dXgqiYiYjgSrCKmQNKrgAuo2al1VKg6C5gLXEz9TQ2tJftfwBZgs6SZwAVtT+L7gYeBqyRd3b5+p+1BLZyOiGHJVWDEFEh6JfVkah9qIPs0qpfp9cAlwBrb23o7YA/atd+5wDHAIcAa4E7g18Dbgd9RuxNnZFA9InZ1GV6PeAojM1UHU7NC62w/Iulc4BNtufIRwEzgb0MLVfC/69GvAGupgfVVk0Wfks4AfpEh9YgYigSriKfQQsMC4DLg28Dhkk6krvw+K+l5wNnAEts/7/GovWrre+5q/wCQtBB4NfW7i4gYhASriDGSXky1pj8IvIx62+0E4Hiqe+nftj8jydRs1Tm21/Z03OccSbOBxdTy6cWZqYqIIUmwihgh6eXAjdS81AZqp9111CqWRcBJth9rjerLAWxvG0JP1RT8A/g9cJzt+/s+TETEsynD6xGNpDnUdd9y29e0780C7gF2Aw6x/aikI4HzgPfYfqin40ZExHNQnlhFbPdG4Hu2r5E0g6pSmANcT+0BPE7SFqqz6pKEqoiIGJdgFbHdA8CEpPnUjNCewKHAd6muppOA3wDLbK/O9V9ERIxLsIrY7ifACuBTwP1Uq/qvqKdWW4GLRxYMJ1RFRMT/yYxVxBhJ+9h+eOTredSbgQuBvyRQRUTEk8kTq4gxk6FK0m7A0dSC5WWZqYqIiB3JrsCIJ9BC1VxqVcuFtm/p+UgREbETyFVgxJNo4epFttdnpioiIp6OBKuIiIiIjuQqMCIiIqIjCVYRERERHUmwioiIiOhIglVERERERxKsIiIiIjqSYBURERHRkQSriIiIiI78F63t7Qn/WnyKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize = (10,6))\n",
    "ax.bar(test_df['intent'].value_counts().index,test_df['intent'].value_counts()\n",
    ")\n",
    "ax.set_title('The number of texts for each intent in the test_df')\n",
    "plt.xticks(rotation = 45)\n",
    "for bar in ax.patches: \n",
    "    \n",
    "  # Using Matplotlib's annotate function and \n",
    "  # passing the coordinates where the annotation shall be done \n",
    "  # x-coordinate: bar.get_x() + bar.get_width() / 2 \n",
    "  # y-coordinate: bar.get_height() \n",
    "  # free space to be left to make graph pleasing: (0, 8) \n",
    "  # ha and va stand for the horizontal and vertical alignment \n",
    "  ax.annotate(bar.get_height(),  \n",
    "                   (bar.get_x() + bar.get_width() / 2,  \n",
    "                    bar.get_height()), ha='center', va='center', \n",
    "                   size=9, xytext=(0, 8), \n",
    "                   textcoords='offset points') \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wfWE6Z2r4KV2"
   },
   "source": [
    "\n",
    "**The train set is balanced**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4F69Wm4g4KV2"
   },
   "source": [
    "### Look at the data\n",
    "let's look at a few examples in the data to get a sense of what the data is about."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BueUj78r4KV2"
   },
   "source": [
    "Print the first 30 examples in the **train_df**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 935
    },
    "id": "I_PfiWCV4KV2",
    "outputId": "3d81ce97-177c-483d-ac2d-d2bee8bcabb7",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>intent</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12353</th>\n",
       "      <td>BookRestaurant</td>\n",
       "      <td>book any popular puglia serving oyster bar for...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10554</th>\n",
       "      <td>AddToPlaylist</td>\n",
       "      <td>Add louis nelson delisle to the pulse of ameri...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12414</th>\n",
       "      <td>BookRestaurant</td>\n",
       "      <td>Book a restaurant for one person at 7 AM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11545</th>\n",
       "      <td>AddToPlaylist</td>\n",
       "      <td>Add this artist to my garage jams</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8313</th>\n",
       "      <td>SearchScreeningEvent</td>\n",
       "      <td>Find the schedule for The Tale of Sweeney Todd...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2999</th>\n",
       "      <td>GetWeather</td>\n",
       "      <td>Is there snowfall nearby AMerican Samoa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9118</th>\n",
       "      <td>SearchScreeningEvent</td>\n",
       "      <td>Give me a list of films at AMC Theatres</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1231</th>\n",
       "      <td>SearchCreativeWork</td>\n",
       "      <td>Find The American Journal of Economics and Soc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1574</th>\n",
       "      <td>SearchCreativeWork</td>\n",
       "      <td>play The Rest of My Life</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4361</th>\n",
       "      <td>PlayMusic</td>\n",
       "      <td>Play the red room sessions from chris cunningham</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8724</th>\n",
       "      <td>SearchScreeningEvent</td>\n",
       "      <td>Where can I find movie schedules</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4697</th>\n",
       "      <td>PlayMusic</td>\n",
       "      <td>Play the good sort of 1992 theme music by Layz...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5733</th>\n",
       "      <td>PlayMusic</td>\n",
       "      <td>I want to hear a Silvia Night ep from the sixties</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13385</th>\n",
       "      <td>BookRestaurant</td>\n",
       "      <td>I would like to book a donut serving delicates...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>SearchCreativeWork</td>\n",
       "      <td>Locate me the novel entitled Isthmus.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5149</th>\n",
       "      <td>PlayMusic</td>\n",
       "      <td>Play play Lodge by Alkabli on lastfm .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9006</th>\n",
       "      <td>SearchScreeningEvent</td>\n",
       "      <td>Is Brother at United Paramount Theatres</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8300</th>\n",
       "      <td>SearchScreeningEvent</td>\n",
       "      <td>What's the movie schedule for animated movies ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7005</th>\n",
       "      <td>RateBook</td>\n",
       "      <td>Rate this textbook one stars out of 6\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1369</th>\n",
       "      <td>SearchCreativeWork</td>\n",
       "      <td>show creativity of video game Labyrinth</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8236</th>\n",
       "      <td>SearchScreeningEvent</td>\n",
       "      <td>Can i get the movie schedules for Speakeasy Th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6048</th>\n",
       "      <td>RateBook</td>\n",
       "      <td>Rate this textbook a zero</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5454</th>\n",
       "      <td>PlayMusic</td>\n",
       "      <td>Play the song Long Live Love</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12614</th>\n",
       "      <td>BookRestaurant</td>\n",
       "      <td>Book a restaurant for eight people in six year...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5310</th>\n",
       "      <td>PlayMusic</td>\n",
       "      <td>I want to listen to the album going back to th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5527</th>\n",
       "      <td>PlayMusic</td>\n",
       "      <td>I want to hear some theme music by Edsel Dope</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6263</th>\n",
       "      <td>RateBook</td>\n",
       "      <td>give 0 to The Sky Village</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5747</th>\n",
       "      <td>PlayMusic</td>\n",
       "      <td>Play a song from 1950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>237</th>\n",
       "      <td>SearchCreativeWork</td>\n",
       "      <td>find Face the Truth, a TV show</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12150</th>\n",
       "      <td>BookRestaurant</td>\n",
       "      <td>I need a table for 4 at a restaurant around 0 AM.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     intent                                               text\n",
       "12353        BookRestaurant  book any popular puglia serving oyster bar for...\n",
       "10554         AddToPlaylist  Add louis nelson delisle to the pulse of ameri...\n",
       "12414        BookRestaurant           Book a restaurant for one person at 7 AM\n",
       "11545         AddToPlaylist                  Add this artist to my garage jams\n",
       "8313   SearchScreeningEvent  Find the schedule for The Tale of Sweeney Todd...\n",
       "2999             GetWeather            Is there snowfall nearby AMerican Samoa\n",
       "9118   SearchScreeningEvent            Give me a list of films at AMC Theatres\n",
       "1231     SearchCreativeWork  Find The American Journal of Economics and Soc...\n",
       "1574     SearchCreativeWork                           play The Rest of My Life\n",
       "4361              PlayMusic   Play the red room sessions from chris cunningham\n",
       "8724   SearchScreeningEvent                   Where can I find movie schedules\n",
       "4697              PlayMusic  Play the good sort of 1992 theme music by Layz...\n",
       "5733              PlayMusic  I want to hear a Silvia Night ep from the sixties\n",
       "13385        BookRestaurant  I would like to book a donut serving delicates...\n",
       "99       SearchCreativeWork              Locate me the novel entitled Isthmus.\n",
       "5149              PlayMusic             Play play Lodge by Alkabli on lastfm .\n",
       "9006   SearchScreeningEvent            Is Brother at United Paramount Theatres\n",
       "8300   SearchScreeningEvent  What's the movie schedule for animated movies ...\n",
       "7005               RateBook            Rate this textbook one stars out of 6\\n\n",
       "1369     SearchCreativeWork            show creativity of video game Labyrinth\n",
       "8236   SearchScreeningEvent  Can i get the movie schedules for Speakeasy Th...\n",
       "6048               RateBook                          Rate this textbook a zero\n",
       "5454              PlayMusic                       Play the song Long Live Love\n",
       "12614        BookRestaurant  Book a restaurant for eight people in six year...\n",
       "5310              PlayMusic  I want to listen to the album going back to th...\n",
       "5527              PlayMusic      I want to hear some theme music by Edsel Dope\n",
       "6263               RateBook                          give 0 to The Sky Village\n",
       "5747              PlayMusic                              Play a song from 1950\n",
       "237      SearchCreativeWork                     find Face the Truth, a TV show\n",
       "12150        BookRestaurant  I need a table for 4 at a restaurant around 0 AM."
      ]
     },
     "execution_count": 13,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PacRh3FI4KV2"
   },
   "source": [
    "Print the first 20 examples in the **test_df**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 639
    },
    "id": "9wLCB8nx4KV3",
    "outputId": "ecfc1b87-fee4-4ee9-f203-8b1212a732ba",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>intent</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>SearchCreativeWork</td>\n",
       "      <td>Can you search the Daring Youth saga?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>641</th>\n",
       "      <td>BookRestaurant</td>\n",
       "      <td>Book a reservation for a bar with a spa nearby ID</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>GetWeather</td>\n",
       "      <td>what is the overcast forecast for the current ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>302</th>\n",
       "      <td>RateBook</td>\n",
       "      <td>rate A Twist in the Tale zero out of 6 points</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>541</th>\n",
       "      <td>AddToPlaylist</td>\n",
       "      <td>Add a song to Indie Hipster</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>518</th>\n",
       "      <td>AddToPlaylist</td>\n",
       "      <td>Add porter wagoner to the The Sleep Machine Wa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351</th>\n",
       "      <td>RateBook</td>\n",
       "      <td>Give zero stars to Rhialto the Marvellous</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>191</th>\n",
       "      <td>GetWeather</td>\n",
       "      <td>Tell me the weather forecast here</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>GetWeather</td>\n",
       "      <td>what is the forecast in North Carolina for Edg...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173</th>\n",
       "      <td>GetWeather</td>\n",
       "      <td>Need weather for Parc national Tolhuaca to see...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>578</th>\n",
       "      <td>AddToPlaylist</td>\n",
       "      <td>Add jack white to my playlist This Is Shakira</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>566</th>\n",
       "      <td>AddToPlaylist</td>\n",
       "      <td>add Muzika za decu to my Crash Course playlist</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>524</th>\n",
       "      <td>AddToPlaylist</td>\n",
       "      <td>Put Vandemataram Srinivas's track onto HipHop ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>531</th>\n",
       "      <td>AddToPlaylist</td>\n",
       "      <td>Add the album to the The Sweet Suite playlist.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>RateBook</td>\n",
       "      <td>Give this novel a 1.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>SearchCreativeWork</td>\n",
       "      <td>Please get me The Henderson Kids saga.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>440</th>\n",
       "      <td>SearchScreeningEvent</td>\n",
       "      <td>Is American Primitive showing in Santikos Thea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240</th>\n",
       "      <td>PlayMusic</td>\n",
       "      <td>Play some sixties songs on Google Music</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>GetWeather</td>\n",
       "      <td>What's the weather in Benton City?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>SearchCreativeWork</td>\n",
       "      <td>Please look up and find me Monty Python Live a...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   intent                                               text\n",
       "27     SearchCreativeWork              Can you search the Daring Youth saga?\n",
       "641        BookRestaurant  Book a reservation for a bar with a spa nearby ID\n",
       "152            GetWeather  what is the overcast forecast for the current ...\n",
       "302              RateBook      rate A Twist in the Tale zero out of 6 points\n",
       "541         AddToPlaylist                        Add a song to Indie Hipster\n",
       "518         AddToPlaylist  Add porter wagoner to the The Sleep Machine Wa...\n",
       "351              RateBook          Give zero stars to Rhialto the Marvellous\n",
       "191            GetWeather                  Tell me the weather forecast here\n",
       "171            GetWeather  what is the forecast in North Carolina for Edg...\n",
       "173            GetWeather  Need weather for Parc national Tolhuaca to see...\n",
       "578         AddToPlaylist      Add jack white to my playlist This Is Shakira\n",
       "566         AddToPlaylist     add Muzika za decu to my Crash Course playlist\n",
       "524         AddToPlaylist  Put Vandemataram Srinivas's track onto HipHop ...\n",
       "531         AddToPlaylist     Add the album to the The Sweet Suite playlist.\n",
       "390              RateBook                               Give this novel a 1.\n",
       "64     SearchCreativeWork             Please get me The Henderson Kids saga.\n",
       "440  SearchScreeningEvent  Is American Primitive showing in Santikos Thea...\n",
       "240             PlayMusic            Play some sixties songs on Google Music\n",
       "143            GetWeather                 What's the weather in Benton City?\n",
       "52     SearchCreativeWork  Please look up and find me Monty Python Live a..."
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LwL2hj5B4KV3"
   },
   "source": [
    "### Preprocess the data\n",
    "Lower case the texts and remove punctuations from the texts.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "FUZvqLTo4KV3"
   },
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "translator = str.maketrans('', '', string.punctuation)\n",
    "\n",
    "def pre_process(text):\n",
    "    \"\"\"\n",
    "        :param text: A user input text\n",
    "        :return: The preprocessed text\n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "    text = text.translate(translator)\n",
    "    text = text.strip().lower()\n",
    "    return text\n",
    "    \n",
    "\n",
    "train_df[\"text\"] = train_df[\"text\"].apply(pre_process)\n",
    "test_df[\"text\"] = test_df[\"text\"].apply(pre_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-BHwkgjj4KV3"
   },
   "source": [
    "Let's have a quick look at texts again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 195
    },
    "id": "4WSXVH9q4KV4",
    "outputId": "96a170e6-0102-456c-f071-c12505303d36"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>intent</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12353</th>\n",
       "      <td>BookRestaurant</td>\n",
       "      <td>book any popular puglia serving oyster bar for...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10554</th>\n",
       "      <td>AddToPlaylist</td>\n",
       "      <td>add louis nelson delisle to the pulse of ameri...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12414</th>\n",
       "      <td>BookRestaurant</td>\n",
       "      <td>book a restaurant for one person at 7 am</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11545</th>\n",
       "      <td>AddToPlaylist</td>\n",
       "      <td>add this artist to my garage jams</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8313</th>\n",
       "      <td>SearchScreeningEvent</td>\n",
       "      <td>find the schedule for the tale of sweeney todd...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     intent                                               text\n",
       "12353        BookRestaurant  book any popular puglia serving oyster bar for...\n",
       "10554         AddToPlaylist  add louis nelson delisle to the pulse of ameri...\n",
       "12414        BookRestaurant           book a restaurant for one person at 7 am\n",
       "11545         AddToPlaylist                  add this artist to my garage jams\n",
       "8313   SearchScreeningEvent  find the schedule for the tale of sweeney todd..."
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 195
    },
    "id": "sQ0iywxO4KV4",
    "outputId": "ae7eda10-b878-4606-83a9-5339e9145b61",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>intent</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>SearchCreativeWork</td>\n",
       "      <td>can you search the daring youth saga</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>641</th>\n",
       "      <td>BookRestaurant</td>\n",
       "      <td>book a reservation for a bar with a spa nearby id</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>GetWeather</td>\n",
       "      <td>what is the overcast forecast for the current ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>302</th>\n",
       "      <td>RateBook</td>\n",
       "      <td>rate a twist in the tale zero out of 6 points</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>541</th>\n",
       "      <td>AddToPlaylist</td>\n",
       "      <td>add a song to indie hipster</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 intent                                               text\n",
       "27   SearchCreativeWork               can you search the daring youth saga\n",
       "641      BookRestaurant  book a reservation for a bar with a spa nearby id\n",
       "152          GetWeather  what is the overcast forecast for the current ...\n",
       "302            RateBook      rate a twist in the tale zero out of 6 points\n",
       "541       AddToPlaylist                        add a song to indie hipster"
      ]
     },
     "execution_count": 17,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YLogQ9lb4KV4"
   },
   "source": [
    "## Choosing the model\n",
    "There is no right way to choose the perfect model for a machine learning problem. I have to try them out to see what is the best fit for this problem. Because this is a text classification problem, so let's use our experience with text classification problems. First, I will try out a simple **Logistic Regression** model with **tfidf** as a baseline and then use **cross-validation** to see if I can improve the model. After that, I will try **Feed Forward Neural Network** with **word embedding** to see what happens. **Word embedding** is a way to represent texts as vectors for training with **Neural Network** more easily."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "25WBlLMz4KV4"
   },
   "source": [
    "## Logistic Regression with tfidf\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PB03vn7e4KV5"
   },
   "source": [
    "### Import libs that you need here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "W0TsR6_-4KV5",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4SSAsvxT4KV5"
   },
   "source": [
    "### Train tfidf vectorizer and transform texts to vectors\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "bOa_rrcX4KV5"
   },
   "outputs": [],
   "source": [
    "# Create the vectorizer\n",
    "tfidf = TfidfVectorizer(ngram_range=(1, 2))\n",
    "\n",
    "# fit and transform the train data\n",
    "tfidf_train = tfidf.fit_transform(train_df['text'])\n",
    "# transform the test data\n",
    "tfidf_test = tfidf.transform(test_df['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "n2VCFarLPWLH",
    "outputId": "e10e34b6-448a-4b14-9363-785de25736b7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<13784x48769 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 210340 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 20,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QN_Bl3LS4KV5"
   },
   "source": [
    "### Create and train the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2wIp1rSy4KV6",
    "outputId": "f7f3fb30-55ab-48be-c4bb-3bf3d4590368",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    }
   ],
   "source": [
    "# Create and train the model here\n",
    "logistic = LogisticRegression(multi_class=\"multinomial\", solver = \"lbfgs\", max_iter=100 )\n",
    "logis_model = logistic.fit(tfidf_train, train_df['intent'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ITIH0UsS4KV6"
   },
   "source": [
    "### Evaluate the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3sODY7qd4KV6",
    "outputId": "79c7a7b7-41c4-4125-ed6d-992abd9566f6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classificaton report for test data\n",
      "\n",
      "                      precision    recall  f1-score   support\n",
      "\n",
      "       AddToPlaylist       1.00      1.00      1.00      1942\n",
      "      BookRestaurant       0.99      1.00      1.00      1973\n",
      "          GetWeather       1.00      0.99      1.00      2000\n",
      "           PlayMusic       0.98      1.00      0.99      2000\n",
      "            RateBook       1.00      1.00      1.00      1956\n",
      "  SearchCreativeWork       0.98      0.98      0.98      1954\n",
      "SearchScreeningEvent       1.00      0.98      0.99      1959\n",
      "\n",
      "            accuracy                           0.99     13784\n",
      "           macro avg       0.99      0.99      0.99     13784\n",
      "        weighted avg       0.99      0.99      0.99     13784\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model\n",
    "from sklearn.metrics import classification_report\n",
    "X_train = tfidf_train\n",
    "X_test = tfidf_test\n",
    "y_true_train = train_df['intent']\n",
    "y_true_test = test_df['intent']\n",
    "y_pred_train_logis = logis_model.predict(X_train)\n",
    "\n",
    "print('Classificaton report for test data')\n",
    "print('')\n",
    "print(classification_report(y_true_train, y_pred_train_logis))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pmMFBHW4TMwv",
    "outputId": "d93345e5-6ea6-4c6e-a7f4-8a6b3d2943d9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classificaton report for training data\n",
      "\n",
      "                      precision    recall  f1-score   support\n",
      "\n",
      "       AddToPlaylist       0.98      1.00      0.99       100\n",
      "      BookRestaurant       0.98      1.00      0.99       100\n",
      "          GetWeather       0.99      0.96      0.97       100\n",
      "           PlayMusic       1.00      0.97      0.98       100\n",
      "            RateBook       1.00      1.00      1.00       100\n",
      "  SearchCreativeWork       0.93      0.99      0.96       100\n",
      "SearchScreeningEvent       0.97      0.93      0.95       100\n",
      "\n",
      "            accuracy                           0.98       700\n",
      "           macro avg       0.98      0.98      0.98       700\n",
      "        weighted avg       0.98      0.98      0.98       700\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred_test_logis = logis_model.predict(X_test)\r\n",
    "print('Classificaton report for training data')\r\n",
    "print('')\r\n",
    "print(classification_report(y_true_test, y_pred_test_logis))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zRrLDIs_4KV6"
   },
   "source": [
    "The f1 score is about **0.98**. It is really high though :))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ilyA0QX_4KV6"
   },
   "source": [
    "## Cross-validation with Logistic Regression\n",
    "Let's tune some parameters using cross-validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xZbhbtvg4KV6"
   },
   "source": [
    "### Create and train the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "collapsed": true,
    "id": "P0yS_vRy4KV7",
    "jupyter": {
     "outputs_hidden": true
    },
    "outputId": "345f1adb-7ecb-4483-a561-e17150670582",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score=nan,\n",
       "             estimator=LogisticRegression(C=1.0, class_weight=None, dual=False,\n",
       "                                          fit_intercept=True,\n",
       "                                          intercept_scaling=1, l1_ratio=None,\n",
       "                                          max_iter=100,\n",
       "                                          multi_class='multinomial',\n",
       "                                          n_jobs=None, penalty='l2',\n",
       "                                          random_state=None, solver='lbfgs',\n",
       "                                          tol=0.0001, verbose=0,\n",
       "                                          warm_start=False),\n",
       "             iid='deprecated', n_jobs=None,\n",
       "             param_grid={'C': [0.1, 1, 10],\n",
       "                         'solver': ('newton-cg', 'lbfgs', 'sag')},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=0)"
      ]
     },
     "execution_count": 24,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "# Create the model\n",
    "logistic_2 = LogisticRegression(multi_class=\"multinomial\", max_iter=100 )\n",
    "# Create tuning parameters\n",
    "parameters = {'solver':('newton-cg', 'lbfgs','sag'), 'C':[0.1,1, 10]}\n",
    "# Create GridSearchCV\n",
    "grid_search = GridSearchCV(logistic_2, parameters)\n",
    "# Train the model\n",
    "grid_search.fit(X_train, y_true_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Hu1dYJH64KV7"
   },
   "source": [
    "### Evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Hj2Evzfy4KV7",
    "outputId": "ce3f6041-d035-4e63-a6ab-b837230dbe49"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 10, 'solver': 'newton-cg'}"
      ]
     },
     "execution_count": 25,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate the model\n",
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HRL5jvqCZfee",
    "outputId": "df80d2ac-f8c2-479d-b437-d25b10dbfd89"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classificaton report for test data with GridSearchCV\n",
      "\n",
      "                      precision    recall  f1-score   support\n",
      "\n",
      "       AddToPlaylist       0.98      1.00      0.99       100\n",
      "      BookRestaurant       0.99      1.00      1.00       100\n",
      "          GetWeather       0.99      0.97      0.98       100\n",
      "           PlayMusic       1.00      0.97      0.98       100\n",
      "            RateBook       1.00      1.00      1.00       100\n",
      "  SearchCreativeWork       0.93      1.00      0.97       100\n",
      "SearchScreeningEvent       0.98      0.93      0.95       100\n",
      "\n",
      "            accuracy                           0.98       700\n",
      "           macro avg       0.98      0.98      0.98       700\n",
      "        weighted avg       0.98      0.98      0.98       700\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred_test_logis2 = grid_search.predict(X_test)\r\n",
    "print('Classificaton report for test data with GridSearchCV')\r\n",
    "print('')\r\n",
    "print(classification_report(y_true_test, y_pred_test_logis2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YighPWa64KV7"
   },
   "source": [
    "The f1 score is about **0.98**. My baseline Logistic Regression is pretty good though so, with that, we can say that our dataset is well separated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ci4zrrDi4KV7"
   },
   "source": [
    "## Word embedding and Neural Network\n",
    "In this section I will use **word embedding** to represent user input texts as vectors then I will use **Neural Network** to train the classification model. **Word embeddings** are vector representations of a particular word. But, why do we need them and can we just use **tfidf** to represent texts as vectors? Well, one disadvantage of **tfidf** or **bag of word** is that the feature dimension could be really large and sparse (sparse in the context of vector mean vectors with a lot of zero values). For this problem, the feature dimension size is **48769** (for ngram_range=(1,2)), meanwhile, we only have **13784** examples and the dimension size increases when the vocabulary size increases. That is very inefficient when training with **Neural Network** in terms of computation. When using **word embedding**, the vector size is fixed no matter the vocabulary size. Another advantage of **word embedding** is that it is capable of capturing the context of a word in a document, semantic and syntactic similarity, relation with other words. I will use a pre-trained **word embedding** model to represent our words as vectors and then use those vectors to feed into **Neural Network** model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hvZpv7Np4KV7"
   },
   "source": [
    "### Using spaCy for word embedding\n",
    "I will use spaCy (https://spacy.io/) which is a python library for solving **Natural Language Process (NLP)** tasks more easily. The **en_core_web_lg** model from spaCy (https://spacy.io/models/en#en_core_web_lg) contains many English models for different tasks. It includes a pre-trained **Glove** word embedding model which has 685k keys, 685k unique vectors (300 dimensions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YKQBJ-nc4KV7",
    "outputId": "bad020a5-96a5-4929-fae5-c7114373664b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting en_core_web_lg==2.2.5\n",
      "\u001b[?25l  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-2.2.5/en_core_web_lg-2.2.5.tar.gz (827.9MB)\n",
      "\u001b[K     |████████████████████████████████| 827.9MB 1.2MB/s \n",
      "\u001b[?25hRequirement already satisfied: spacy>=2.2.2 in /usr/local/lib/python3.6/dist-packages (from en_core_web_lg==2.2.5) (2.2.4)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (2.0.5)\n",
      "Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (7.4.0)\n",
      "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (0.4.1)\n",
      "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (1.0.5)\n",
      "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (1.1.3)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (1.0.5)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (4.41.1)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (1.19.5)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (0.8.2)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (53.0.0)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (3.0.5)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (2.23.0)\n",
      "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.2.2->en_core_web_lg==2.2.5) (1.0.0)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_lg==2.2.5) (1.24.3)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_lg==2.2.5) (3.0.4)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_lg==2.2.5) (2020.12.5)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->en_core_web_lg==2.2.5) (2.10)\n",
      "Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_lg==2.2.5) (3.4.0)\n",
      "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_lg==2.2.5) (3.7.4.3)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy>=2.2.2->en_core_web_lg==2.2.5) (3.4.0)\n",
      "Building wheels for collected packages: en-core-web-lg\n",
      "  Building wheel for en-core-web-lg (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for en-core-web-lg: filename=en_core_web_lg-2.2.5-cp36-none-any.whl size=829180945 sha256=9387ddbe0d072053506c156e2bdfc465a115a211160177515893304086bdd4a9\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-_6gnm4gz/wheels/2a/c1/a6/fc7a877b1efca9bc6a089d6f506f16d3868408f9ff89f8dbfc\n",
      "Successfully built en-core-web-lg\n",
      "Installing collected packages: en-core-web-lg\n",
      "Successfully installed en-core-web-lg-2.2.5\n",
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the model via spacy.load('en_core_web_lg')\n"
     ]
    }
   ],
   "source": [
    "# # On Windows, remember to run Anaconda prompt with Administrator to make sure this command works probably\n",
    "# !python -m spacy download en_core_web_lg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O3t4UTrL4KV8"
   },
   "source": [
    "### Import spaCy model\n",
    "Run the cell below to load the **en_core_web_lg** model from spaCy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KfPmixBt4KV8"
   },
   "source": [
    "**NOTE:** Having trouble loading the **en_core_web_lg** model on Google colab? Please check this question [Unable to load the spacy model 'en_core_web_lg' on Google colab](https://stackoverflow.com/questions/56927602/unable-to-load-the-spacy-model-en-core-web-lg-on-google-colab) on stackoverflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "QF1mHzLF4KV8"
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_lg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hDljvsKB4KV8"
   },
   "source": [
    "### Word embedding examples\n",
    "The example below shows how to get the embedding vector for the word **cat**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "h4WjaVNr4KV8",
    "outputId": "77e3c0cb-0bd6-47f1-9bd6-3542e8d6dfaa",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.15067  , -0.024468 , -0.23368  , -0.23378  , -0.18382  ,\n",
       "        0.32711  , -0.22084  , -0.28777  ,  0.12759  ,  1.1656   ,\n",
       "       -0.64163  , -0.098455 , -0.62397  ,  0.010431 , -0.25653  ,\n",
       "        0.31799  ,  0.037779 ,  1.1904   , -0.17714  , -0.2595   ,\n",
       "       -0.31461  ,  0.038825 , -0.15713  , -0.13484  ,  0.36936  ,\n",
       "       -0.30562  , -0.40619  , -0.38965  ,  0.3686   ,  0.013963 ,\n",
       "       -0.6895   ,  0.004066 , -0.1367   ,  0.32564  ,  0.24688  ,\n",
       "       -0.14011  ,  0.53889  , -0.80441  , -0.1777   , -0.12922  ,\n",
       "        0.16303  ,  0.14917  , -0.068429 , -0.33922  ,  0.18495  ,\n",
       "       -0.082544 , -0.46892  ,  0.39581  , -0.13742  , -0.35132  ,\n",
       "        0.22223  , -0.144    , -0.048287 ,  0.3379   , -0.31916  ,\n",
       "        0.20526  ,  0.098624 , -0.23877  ,  0.045338 ,  0.43941  ,\n",
       "        0.030385 , -0.013821 , -0.093273 , -0.18178  ,  0.19438  ,\n",
       "       -0.3782   ,  0.70144  ,  0.16236  ,  0.0059111,  0.024898 ,\n",
       "       -0.13613  , -0.11425  , -0.31598  , -0.14209  ,  0.028194 ,\n",
       "        0.5419   , -0.42413  , -0.599    ,  0.24976  , -0.27003  ,\n",
       "        0.14964  ,  0.29287  , -0.31281  ,  0.16543  , -0.21045  ,\n",
       "       -0.4408   ,  1.2174   ,  0.51236  ,  0.56209  ,  0.14131  ,\n",
       "        0.092514 ,  0.71396  , -0.021051 , -0.33704  , -0.20275  ,\n",
       "       -0.36181  ,  0.22055  , -0.25665  ,  0.28425  , -0.16968  ,\n",
       "        0.058029 ,  0.61182  ,  0.31576  , -0.079185 ,  0.35538  ,\n",
       "       -0.51236  ,  0.4235   , -0.30033  , -0.22376  ,  0.15223  ,\n",
       "       -0.048292 ,  0.23532  ,  0.46507  , -0.67579  , -0.32905  ,\n",
       "        0.08446  , -0.22123  , -0.045333 ,  0.34463  , -0.1455   ,\n",
       "       -0.18047  , -0.17887  ,  0.96879  , -1.0028   , -0.47343  ,\n",
       "        0.28542  ,  0.56382  , -0.33211  , -0.38275  , -0.2749   ,\n",
       "       -0.22955  , -0.24265  , -0.37689  ,  0.24822  ,  0.36941  ,\n",
       "        0.14651  , -0.37864  ,  0.31134  , -0.28449  ,  0.36948  ,\n",
       "       -2.8174   , -0.38319  , -0.022373 ,  0.56376  ,  0.40131  ,\n",
       "       -0.42131  , -0.11311  , -0.17317  ,  0.1411   , -0.13194  ,\n",
       "        0.18494  ,  0.097692 , -0.097341 , -0.23987  ,  0.16631  ,\n",
       "       -0.28556  ,  0.0038654,  0.53292  , -0.32367  , -0.38744  ,\n",
       "        0.27011  , -0.34181  , -0.27702  , -0.67279  , -0.10771  ,\n",
       "       -0.062189 , -0.24783  , -0.070884 , -0.20898  ,  0.062404 ,\n",
       "        0.022372 ,  0.13408  ,  0.1305   , -0.19546  , -0.46849  ,\n",
       "        0.77731  , -0.043978 ,  0.3827   , -0.23376  ,  1.0457   ,\n",
       "       -0.14371  , -0.3565   , -0.080713 , -0.31047  , -0.57822  ,\n",
       "       -0.28067  , -0.069678 ,  0.068929 , -0.16227  , -0.63934  ,\n",
       "       -0.62149  ,  0.11222  , -0.16969  , -0.54637  ,  0.49661  ,\n",
       "        0.46565  ,  0.088294 , -0.48496  ,  0.69263  , -0.068977 ,\n",
       "       -0.53709  ,  0.20802  , -0.42987  , -0.11921  ,  0.1174   ,\n",
       "       -0.18443  ,  0.43797  , -0.1236   ,  0.3607   , -0.19608  ,\n",
       "       -0.35366  ,  0.18808  , -0.5061   ,  0.14455  , -0.024368 ,\n",
       "       -0.10772  , -0.0115   ,  0.58634  , -0.054461 ,  0.0076487,\n",
       "       -0.056297 ,  0.27193  ,  0.23096  , -0.29296  , -0.24325  ,\n",
       "        0.10317  , -0.10014  ,  0.7089   ,  0.17402  , -0.0037509,\n",
       "       -0.46304  ,  0.11806  , -0.16457  , -0.38609  ,  0.14524  ,\n",
       "        0.098122 , -0.12352  , -0.1047   ,  0.39047  , -0.3063   ,\n",
       "       -0.65375  , -0.0044248, -0.033876 ,  0.037114 , -0.27472  ,\n",
       "        0.0053147,  0.30737  ,  0.12528  , -0.19527  , -0.16461  ,\n",
       "        0.087518 , -0.051107 , -0.16323  ,  0.521    ,  0.10822  ,\n",
       "       -0.060379 , -0.71735  , -0.064327 ,  0.37043  , -0.41054  ,\n",
       "       -0.2728   , -0.30217  ,  0.015771 , -0.43056  ,  0.35647  ,\n",
       "        0.17188  , -0.54598  , -0.21541  , -0.044889 , -0.10597  ,\n",
       "       -0.54391  ,  0.53908  ,  0.070938 ,  0.097839 ,  0.097908 ,\n",
       "        0.17805  ,  0.18995  ,  0.49962  , -0.18529  ,  0.051234 ,\n",
       "        0.019574 ,  0.24805  ,  0.3144   , -0.29304  ,  0.54235  ,\n",
       "        0.46672  ,  0.26017  , -0.44705  ,  0.28287  , -0.033345 ,\n",
       "       -0.33181  , -0.10902  , -0.023324 ,  0.2106   , -0.29633  ,\n",
       "        0.81506  ,  0.038524 ,  0.46004  ,  0.17187  , -0.29804  ],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 2,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp(\"cat\").vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WRhi7g284KV8"
   },
   "source": [
    "A nice property of word embedding is that similar words (words appear in the same contexts) are usually grouped together. For example, the words **good** and **great** are often used in the same context (Have a **great** day, Have a **good** day) so, their embedding vectors are closed to each other than let's say **good** and **cat**. We can use cosine similarity to check the similarity between two vectors, spaCy already has the similarity function so, let's check it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XxWWgSD-4KV8",
    "outputId": "a0697ae8-3721-4929-d1d2-8501ea044846"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.841670781054402\n",
      "0.2433478967733492\n"
     ]
    }
   ],
   "source": [
    "print (nlp(\"great\").similarity(nlp(\"good\")))\n",
    "print (nlp(\"great\").similarity(nlp(\"cat\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xqwqFXsE4KV8"
   },
   "source": [
    "The similarity between **great** and **good** is **0.84**. Meanwhile the similarity between **great** and **cat** is only **0.24**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pB3SHbhO4KV9"
   },
   "source": [
    "### Use word embbedding to represent texts as vectors\n",
    "We now can represent a single word as vector using word embedding but, how do we represent a sentence or paragraph as a vector so we can feed into our **Feedforward Neural Network**? \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "id": "Z5dJG-f-4KV9"
   },
   "outputs": [],
   "source": [
    "def average_vectors_text(text):\n",
    "    \"\"\"\n",
    "        :param text: A user input text\n",
    "        :return: The average vector of word embedding vectors in the text input\n",
    "    \"\"\"\n",
    "    word_list = text.split()\n",
    "    embedding_list = [nlp(i).vector for i in word_list]\n",
    "    average_vector = np.mean(embedding_list, axis = 0)\n",
    "    return average_vector\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Poqtxr9cKlCs",
    "outputId": "aa39fdbe-23e0-468f-ecd4-19caf4e2d4d4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['find', 'the', 'soundtrack', 'titled', 'this', 'side', 'of', 'paradise']"
      ]
     },
     "execution_count": 28,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['text'][0].split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OOweo1uV4KV9"
   },
   "source": [
    "Apply the **average_vectors_text** function to the **text** columns of the **train_df** and **test_df** data frames to get **X_train_spacy** and **X_test_spacy**. The feature dimension size is now only 300.\n",
    "<br>\n",
    "**NOTE**: It might takes 15 - 20 minutes to run "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "id": "rw9SwyzH4KV9"
   },
   "outputs": [],
   "source": [
    "X_train_spacy = train_df[\"text\"].apply(average_vectors_text)\n",
    "X_test_spacy = test_df[\"text\"].apply(average_vectors_text)\n",
    "\n",
    "X_train_spacy = X_train_spacy.tolist()\n",
    "X_test_spacy = X_test_spacy.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "25rUiNW3Q7IQ",
    "outputId": "1011b559-b1e9-4996-9365-924090223a09"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "300"
      ]
     },
     "execution_count": 43,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X_train_spacy[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1Db3Qoop4KV9"
   },
   "source": [
    "### Create and train the model\n",
    "Now create a **MLPClassifier** and train it on the **X_train_spacy** and **y_train**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "id": "_DcZu06MRU08"
   },
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "id": "4lI_imtm4KV9"
   },
   "outputs": [],
   "source": [
    "# Create and train the neural network model\n",
    "mlp_clf = MLPClassifier(solver='adam', alpha=1e-2, activation='relu', hidden_layer_sizes=(128))\n",
    "nn_model = mlp_clf.fit(X_train_spacy, y_true_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dNgh2TUc4KV-"
   },
   "source": [
    "### Evaluate the model\n",
    "Evaluate on the model on the **X_test_spacy** and **y_test**.\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pd6al0XE4KV-",
    "outputId": "136e8454-2097-4c6e-de02-c552340f5d32",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classificaton report for test data with Neural Network\n",
      "\n",
      "                      precision    recall  f1-score   support\n",
      "\n",
      "       AddToPlaylist       0.97      1.00      0.99       100\n",
      "      BookRestaurant       0.99      0.99      0.99       100\n",
      "          GetWeather       0.99      1.00      1.00       100\n",
      "           PlayMusic       0.99      0.97      0.98       100\n",
      "            RateBook       1.00      1.00      1.00       100\n",
      "  SearchCreativeWork       0.95      0.98      0.97       100\n",
      "SearchScreeningEvent       0.99      0.94      0.96       100\n",
      "\n",
      "            accuracy                           0.98       700\n",
      "           macro avg       0.98      0.98      0.98       700\n",
      "        weighted avg       0.98      0.98      0.98       700\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model\n",
    "y_pred_test_nn = nn_model.predict(X_test_spacy)\n",
    "print('Classificaton report for test data with Neural Network')\n",
    "print('')\n",
    "print(classification_report(y_true_test, y_pred_test_nn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "od-enFNq4KV-"
   },
   "source": [
    "The f1-score is about **0.98**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6aI2uv804KV-"
   },
   "source": [
    "## Error analysis\n",
    "Let's look at some of the examples that the models make mistakes to see if we can improve the models.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "id": "gYMJ1MJ54KV-"
   },
   "outputs": [],
   "source": [
    "def print_mistake_samples(y, y_pred, df):\n",
    "    \"\"\" \n",
    "        :param y: The true labels\n",
    "        :param y_pred: The predicted labels\n",
    "        :param df: The dataframe (train_df/test_df)\n",
    "        \n",
    "        :return: None\n",
    "    \"\"\"\n",
    "    wrong_indices = y != y_pred \n",
    "    wrong_df = df.loc[wrong_indices, :].copy()\n",
    "    wrong_df[\"pred\"] = y_pred[wrong_indices]\n",
    "    \n",
    "    print (f\"Number of mistakes: {wrong_df.shape[0]}\")\n",
    "    print ('text, true, predict')\n",
    "    for i, row in wrong_df.iterrows():\n",
    "        print(f\"{row['text']}, {row['intent']}, {row['pred']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zRoCjhAw4KV-"
   },
   "source": [
    "Print the mistakes by the simple Logistic Regression model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "r1zcHJEX4KV-",
    "outputId": "4ef90b16-f4a3-4e51-ff6c-d024af9da291"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of mistakes: 15\n",
      "text, true, predict\n",
      "i want to see shattered image, SearchScreeningEvent, SearchCreativeWork\n",
      "find the movie splendor in the grass, SearchCreativeWork, SearchScreeningEvent\n",
      "humidity not far from colorado city on november the 7th 2024, GetWeather, BookRestaurant\n",
      "where is belgium located, GetWeather, SearchScreeningEvent\n",
      "live in lajoseph meyer please, PlayMusic, SearchCreativeWork\n",
      "overcast on state holiday in pawling nature reserve  and neighboring places, GetWeather, BookRestaurant\n",
      "i want to see jla adventures trapped in time, SearchScreeningEvent, SearchCreativeWork\n",
      "i want to see fear chamber, SearchScreeningEvent, SearchCreativeWork\n",
      "i want to see medal for the general, SearchScreeningEvent, SearchCreativeWork\n",
      "i want to see those kids from town when will it be showing, SearchScreeningEvent, GetWeather\n",
      "i want to see outcast, SearchScreeningEvent, SearchCreativeWork\n",
      "when is sunrise for ar, GetWeather, SearchScreeningEvent\n",
      "put what color is your sky by alana davis on the stereo, PlayMusic, AddToPlaylist\n",
      "where can i see the prime ministers the pioneers, SearchScreeningEvent, SearchCreativeWork\n",
      "can you put on like a hurricane by paul landers, PlayMusic, AddToPlaylist\n"
     ]
    }
   ],
   "source": [
    "print_mistake_samples(y_true_test, y_pred_test_logis, test_df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cdep9c5E4KV_"
   },
   "source": [
    "Print the mistakes by the cross-validation Logistic Regression model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Yj6ndgwW4KV_",
    "outputId": "7a86bc3e-065f-4b24-ffaf-0cbef49f180a",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of mistakes: 13\n",
      "text, true, predict\n",
      "i want to see shattered image, SearchScreeningEvent, SearchCreativeWork\n",
      "humidity not far from colorado city on november the 7th 2024, GetWeather, BookRestaurant\n",
      "where is belgium located, GetWeather, SearchScreeningEvent\n",
      "live in lajoseph meyer please, PlayMusic, SearchCreativeWork\n",
      "i want to see jla adventures trapped in time, SearchScreeningEvent, SearchCreativeWork\n",
      "i want to see fear chamber, SearchScreeningEvent, SearchCreativeWork\n",
      "i want to see medal for the general, SearchScreeningEvent, SearchCreativeWork\n",
      "i want to see those kids from town when will it be showing, SearchScreeningEvent, GetWeather\n",
      "i want to see outcast, SearchScreeningEvent, SearchCreativeWork\n",
      "when is sunrise for ar, GetWeather, SearchScreeningEvent\n",
      "put what color is your sky by alana davis on the stereo, PlayMusic, AddToPlaylist\n",
      "where can i see the prime ministers the pioneers, SearchScreeningEvent, SearchCreativeWork\n",
      "can you put on like a hurricane by paul landers, PlayMusic, AddToPlaylist\n"
     ]
    }
   ],
   "source": [
    "print_mistake_samples(y_true_test, y_pred_test_logis2, test_df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Defp7Z8b4KV_"
   },
   "source": [
    "Print the mistakes by the Neural Network model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aRG4kJIr4KV_",
    "outputId": "a32ed590-9000-47b6-ff0a-d2ad78592247",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of mistakes: 12\n",
      "text, true, predict\n",
      "i want to book oregon electric station in north city, BookRestaurant, GetWeather\n",
      "i need to find the movie theatre showing the crooked web closest to me, SearchScreeningEvent, SearchCreativeWork\n",
      "i want to see shattered image, SearchScreeningEvent, SearchCreativeWork\n",
      "will i think i do be playing at 7 pm, SearchScreeningEvent, PlayMusic\n",
      "play my black sabbath the dio years playlist, PlayMusic, AddToPlaylist\n",
      "im looking for welcome to the canteen, SearchCreativeWork, BookRestaurant\n",
      "live in lajoseph meyer please, PlayMusic, AddToPlaylist\n",
      "i want to see medal for the general, SearchScreeningEvent, SearchCreativeWork\n",
      "i want to see outcast, SearchScreeningEvent, SearchCreativeWork\n",
      "find me the soundtrack live at the greek theatre, SearchCreativeWork, SearchScreeningEvent\n",
      "show me the schedule for rat rod rockers, SearchScreeningEvent, SearchCreativeWork\n",
      "can you put on like a hurricane by paul landers, PlayMusic, AddToPlaylist\n"
     ]
    }
   ],
   "source": [
    "print_mistake_samples(y_true_test, y_pred_test_nn, test_df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lUbmOmBn4KV_"
   },
   "source": [
    "**Here are the few examples that all the models make mistakes on:**\n",
    "* \"where is belgium located\": The correct label is **GetWeather** but, as we can see it is a question about finding where **belgium** is. It is not a \"GetWeather\" command that is kind of similar in the train set. I think the label is wrong for this sample.\n",
    "* \"can you put on like a hurricane by paul landers\": The correct label is **PlayMusic**. Here, the models are confused with the key word **put on**. In the train set many examples of the intent **AddToPlaylist** includes the keyword **put on** so, this example is really hard for machine learning models.\n",
    "* \"live in lajoseph meyer please\": The correct label is **PlayMusic**. Because of the word **please** is added at the end of the sentence so it becomes a \"PlayMusic\" command so, this example is clearly a really hard example for our models to figure it out.\n",
    "Similarly, you can analyze by yourself to debug more about the models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TN3wsXAR4KV_"
   },
   "source": [
    "## Improving the models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w1hwoWSm4KV_"
   },
   "source": [
    "I have achieved a pretty high score (around 0.98 f1) for this problem just using a simple **Logistic Regression** model and after analyzing errors, I saw that the models make mistakes on incorrectly labeled data and ambiguous samples so, for now, it is pretty hard to increase the score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dy0ZIRXN4KWA"
   },
   "source": [
    "## Deploy the models\n",
    "Let's assume we are working on a team that is building a virtual assistant and our job is building the intent recognition module. The input is a list of user input texts and the output is a list of corresponding intents.\n",
    "\n",
    "**NOTE**: If predicted intents have probabilities lower than the **threshold**, return **Fallback** intent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "id": "EfSJyv9l4KWA"
   },
   "outputs": [],
   "source": [
    "def recognize_intents(texts, model, model_type, threshold, vectorizer=None):\n",
    "    \"\"\"\n",
    "        Recognize intents of user input texts\n",
    "        \n",
    "        :param text: A list of user input texts\n",
    "        :param model: A sklearn model\n",
    "        :param model_type: Either \"lg\" or \"nn\"\n",
    "        :param threshold: The predicted intents must have confidences equal or greater than this threshold\n",
    "        :param vectorizer: Vectorizer for transforming texts into vectors (only use when model_type == \"lg\")\n",
    "        \n",
    "        :return: A list of intents\n",
    "    \"\"\"\n",
    "    # Preprocess texts using the pre_process function\n",
    "    processed_texts = [pre_process(text) for text in texts]\n",
    "    \n",
    "    # Transforms texts into vectors corresponding to the model_type\n",
    "    if model_type == \"lg\": # use tfidf vectorizer here\n",
    "      vectors = vectorizer.transform(processed_texts)\n",
    "    elif model_type == \"nn\": # user word embedding here\n",
    "      vectors = [average_vectors_text(text) for text in processed_texts]\n",
    "        \n",
    "    else:\n",
    "        raise Exception(f\"Not found {model_type} model_type\")\n",
    "    \n",
    "    # Predict intents and check the probability threshold\n",
    "\n",
    "    # If all intent probabilities below the threshold,\n",
    "    # return the intent as \"Fallback\" for that user input text\n",
    "    list_intents = [model.classes_[idx] for idx in np.argmax(model.predict_proba(vectors), axis =1)]\n",
    "    intents_proba = np.max(model.predict_proba(vectors), axis =1).tolist()\n",
    "    for proba, intent in zip(intents_proba,list_intents):\n",
    "      if proba < threshold:\n",
    "        list_intents[intents_proba.index(proba)] = 'Fallback'\n",
    "\n",
    "    return list_intents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WYRyIH8k4KWA"
   },
   "source": [
    "Let's try it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "APccE24gepDJ",
    "outputId": "b03adc3f-d648-415b-ff57-4cfb753b42c8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['AddToPlaylist', 'BookRestaurant', 'GetWeather', 'PlayMusic',\n",
       "       'RateBook', 'SearchCreativeWork', 'SearchScreeningEvent'],\n",
       "      dtype='<U20')"
      ]
     },
     "execution_count": 104,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nn_model.classes_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "id": "ciIAv7mW4KWA"
   },
   "outputs": [],
   "source": [
    "input_texts = [\n",
    "    \"Play happy new year\",\n",
    "    \"What is the weather in Ha Noi?\",\n",
    "    \"Will it rain tomorow?\",\n",
    "    \"Add despacito to my favourite playlist.\",\n",
    "    \"What is your name?\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gCQEFalN4KWA"
   },
   "source": [
    "Try simple Logistic Regression with **0.7** threshold and tfidf vectorizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WRU6Ogrl4KWA",
    "outputId": "1e58076d-83b4-4c42-c0c8-653bd0702d73"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['PlayMusic', 'GetWeather', 'GetWeather', 'AddToPlaylist', 'Fallback']"
      ]
     },
     "execution_count": 107,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recognize_intents(input_texts, model=logis_model, model_type='lg', threshold=0.7, vectorizer=tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Yb_-6WpX4KWA"
   },
   "source": [
    "Try your Neural Network with 0.7 threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7THRA71s4KWB",
    "outputId": "41cee3b1-056c-4c69-e5c9-28bf295a1f6b",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['PlayMusic',\n",
       " 'GetWeather',\n",
       " 'GetWeather',\n",
       " 'AddToPlaylist',\n",
       " 'SearchCreativeWork']"
      ]
     },
     "execution_count": 108,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recognize_intents(input_texts, model=nn_model, model_type='nn', threshold=0.7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WmRV49s54KWB"
   },
   "source": [
    "Try Neural Network with 0.8 threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wZh62ARO4KWB",
    "outputId": "e6b12f42-c626-4a49-bee6-1c38b98cf7e4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['PlayMusic',\n",
       " 'GetWeather',\n",
       " 'GetWeather',\n",
       " 'AddToPlaylist',\n",
       " 'SearchCreativeWork']"
      ]
     },
     "execution_count": 109,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recognize_intents(input_texts, model=nn_model, model_type='nn', threshold=0.8)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uoJ1bkPH4KWB"
   },
   "source": [
    "Try Neural Network with 0.95 threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LDzm_pIp4KWC",
    "outputId": "58ebbebc-0832-472c-f263-03e5b2d591ed"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Fallback', 'GetWeather', 'GetWeather', 'AddToPlaylist', 'SearchCreativeWork']"
      ]
     },
     "execution_count": 110,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recognize_intents(input_texts, model=nn_model, model_type='nn', threshold=0.95)\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UOzuJ7SmnTe_"
   },
   "source": [
    "# Improve model\r\n",
    "\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mz1OIDaqIa7a"
   },
   "source": [
    "## Try to improve neural network model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "collapsed": true,
    "id": "qt9piYmNnmY8",
    "jupyter": {
     "outputs_hidden": true
    },
    "outputId": "fd9a8319-561a-4cb1-a763-a70fcca7649f",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:470: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score=nan,\n",
       "             estimator=MLPClassifier(activation='relu', alpha=0.0001,\n",
       "                                     batch_size='auto', beta_1=0.9,\n",
       "                                     beta_2=0.999, early_stopping=False,\n",
       "                                     epsilon=1e-08, hidden_layer_sizes=(100,),\n",
       "                                     learning_rate='constant',\n",
       "                                     learning_rate_init=0.001, max_fun=15000,\n",
       "                                     max_iter=200, momentum=0.9,\n",
       "                                     n_iter_no_change=10,\n",
       "                                     nesterovs_momentum=True, power_t=0.5,\n",
       "                                     random_state=None, shuffle=True,\n",
       "                                     solver='adam', tol=0.0001,\n",
       "                                     validation_fraction=0.1, verbose=False,\n",
       "                                     warm_start=False),\n",
       "             iid='deprecated', n_jobs=None,\n",
       "             param_grid={'activation': ('relu', 'tanh'),\n",
       "                         'alpha': [0.001, 0.01, 0.1],\n",
       "                         'hidden_layer_sizes': [(128,), (128, 64)],\n",
       "                         'solver': ('lbfgs', 'sgd', 'adam')},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=0)"
      ]
     },
     "execution_count": 101,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create the model\r\n",
    "mlp_clf_2 = MLPClassifier()\r\n",
    "# Create tuning parameters\r\n",
    "parameters_nn = {'solver':('lbfgs', 'sgd', 'adam'), 'alpha':[1e-3,1e-2, 1e-1],\r\n",
    "              'activation':('relu','tanh'), 'hidden_layer_sizes':[(128,), (128,64)]}\r\n",
    "# Create GridSearchCV\r\n",
    "grid_search_nn = GridSearchCV(mlp_clf_2, parameters_nn)\r\n",
    "# Train the model\r\n",
    "grid_search_nn.fit(X_train_spacy, y_true_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "z_YtILnSIWAv",
    "outputId": "8123eab8-e6cc-414a-bab8-9732e0c1e7d2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'activation': 'relu',\n",
       " 'alpha': 0.01,\n",
       " 'hidden_layer_sizes': (128, 64),\n",
       " 'solver': 'adam'}"
      ]
     },
     "execution_count": 111,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search_nn.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4qXhj47aIXsq",
    "outputId": "f6d7b01b-2641-40a1-e478-0f8409ae585d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classificaton report for test data with GridSearchCV for Neural Network\n",
      "\n",
      "                      precision    recall  f1-score   support\n",
      "\n",
      "       AddToPlaylist       0.97      1.00      0.99       100\n",
      "      BookRestaurant       0.99      0.99      0.99       100\n",
      "          GetWeather       0.98      1.00      0.99       100\n",
      "           PlayMusic       0.99      0.96      0.97       100\n",
      "            RateBook       1.00      1.00      1.00       100\n",
      "  SearchCreativeWork       0.95      0.97      0.96       100\n",
      "SearchScreeningEvent       0.98      0.94      0.96       100\n",
      "\n",
      "            accuracy                           0.98       700\n",
      "           macro avg       0.98      0.98      0.98       700\n",
      "        weighted avg       0.98      0.98      0.98       700\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred_test_nn2 = grid_search_nn.predict(X_test_spacy)\r\n",
    "print('Classificaton report for test data with GridSearchCV for Neural Network')\r\n",
    "print('')\r\n",
    "print(classification_report(y_true_test, y_pred_test_nn2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YrCvHLq3IZHQ"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Assignment 1.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}