24. Dropout Regularization (exporting the model).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overfitting and underfitting are common phenomena in the field of machine learning and the techniques used to tackle overfitting problem is called **regularization**. In deep learning, dropout regularization is used to randomly drop neurons from hidden layers and this helps with generalization. In the code below I have implemented an **artificial neural network** for binary classification problem and observed how using dropout layer can increase the performance of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import warnings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data Set Information**:\n",
    "\n",
    "The file \"sonar.mines\" contains 111 patterns obtained by bouncing sonar signals off a metal cylinder at various angles and under various conditions. The file \"sonar.rocks\" contains 97 patterns obtained from rocks under similar conditions. The transmitted sonar signal is a frequency-modulated chirp, rising in frequency. The data set contains signals obtained from a variety of different aspect angles, spanning 90 degrees for the cylinder and 180 degrees for the rock. \n",
    "\n",
    "Each pattern is a set of 60 numbers in the range 0.0 to 1.0. Each number represents the energy within a particular frequency band, integrated over a certain period of time. The integration aperture for higher frequencies occur later in time, since these frequencies are transmitted later during the chirp. \n",
    "\n",
    "The label associated with each record contains the letter \"R\" if the object is a rock and \"M\" if it is a mine (metal cylinder). The numbers in the labels are in increasing order of aspect angle, but they do not encode the angle directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>51</th>\n",
       "      <th>52</th>\n",
       "      <th>53</th>\n",
       "      <th>54</th>\n",
       "      <th>55</th>\n",
       "      <th>56</th>\n",
       "      <th>57</th>\n",
       "      <th>58</th>\n",
       "      <th>59</th>\n",
       "      <th>60</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0200</td>\n",
       "      <td>0.0371</td>\n",
       "      <td>0.0428</td>\n",
       "      <td>0.0207</td>\n",
       "      <td>0.0954</td>\n",
       "      <td>0.0986</td>\n",
       "      <td>0.1539</td>\n",
       "      <td>0.1601</td>\n",
       "      <td>0.3109</td>\n",
       "      <td>0.2111</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0027</td>\n",
       "      <td>0.0065</td>\n",
       "      <td>0.0159</td>\n",
       "      <td>0.0072</td>\n",
       "      <td>0.0167</td>\n",
       "      <td>0.0180</td>\n",
       "      <td>0.0084</td>\n",
       "      <td>0.0090</td>\n",
       "      <td>0.0032</td>\n",
       "      <td>R</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0453</td>\n",
       "      <td>0.0523</td>\n",
       "      <td>0.0843</td>\n",
       "      <td>0.0689</td>\n",
       "      <td>0.1183</td>\n",
       "      <td>0.2583</td>\n",
       "      <td>0.2156</td>\n",
       "      <td>0.3481</td>\n",
       "      <td>0.3337</td>\n",
       "      <td>0.2872</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0084</td>\n",
       "      <td>0.0089</td>\n",
       "      <td>0.0048</td>\n",
       "      <td>0.0094</td>\n",
       "      <td>0.0191</td>\n",
       "      <td>0.0140</td>\n",
       "      <td>0.0049</td>\n",
       "      <td>0.0052</td>\n",
       "      <td>0.0044</td>\n",
       "      <td>R</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0262</td>\n",
       "      <td>0.0582</td>\n",
       "      <td>0.1099</td>\n",
       "      <td>0.1083</td>\n",
       "      <td>0.0974</td>\n",
       "      <td>0.2280</td>\n",
       "      <td>0.2431</td>\n",
       "      <td>0.3771</td>\n",
       "      <td>0.5598</td>\n",
       "      <td>0.6194</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0232</td>\n",
       "      <td>0.0166</td>\n",
       "      <td>0.0095</td>\n",
       "      <td>0.0180</td>\n",
       "      <td>0.0244</td>\n",
       "      <td>0.0316</td>\n",
       "      <td>0.0164</td>\n",
       "      <td>0.0095</td>\n",
       "      <td>0.0078</td>\n",
       "      <td>R</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0100</td>\n",
       "      <td>0.0171</td>\n",
       "      <td>0.0623</td>\n",
       "      <td>0.0205</td>\n",
       "      <td>0.0205</td>\n",
       "      <td>0.0368</td>\n",
       "      <td>0.1098</td>\n",
       "      <td>0.1276</td>\n",
       "      <td>0.0598</td>\n",
       "      <td>0.1264</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0121</td>\n",
       "      <td>0.0036</td>\n",
       "      <td>0.0150</td>\n",
       "      <td>0.0085</td>\n",
       "      <td>0.0073</td>\n",
       "      <td>0.0050</td>\n",
       "      <td>0.0044</td>\n",
       "      <td>0.0040</td>\n",
       "      <td>0.0117</td>\n",
       "      <td>R</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0762</td>\n",
       "      <td>0.0666</td>\n",
       "      <td>0.0481</td>\n",
       "      <td>0.0394</td>\n",
       "      <td>0.0590</td>\n",
       "      <td>0.0649</td>\n",
       "      <td>0.1209</td>\n",
       "      <td>0.2467</td>\n",
       "      <td>0.3564</td>\n",
       "      <td>0.4459</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0031</td>\n",
       "      <td>0.0054</td>\n",
       "      <td>0.0105</td>\n",
       "      <td>0.0110</td>\n",
       "      <td>0.0015</td>\n",
       "      <td>0.0072</td>\n",
       "      <td>0.0048</td>\n",
       "      <td>0.0107</td>\n",
       "      <td>0.0094</td>\n",
       "      <td>R</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 61 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       0       1       2       3       4       5       6       7       8   \\\n",
       "0  0.0200  0.0371  0.0428  0.0207  0.0954  0.0986  0.1539  0.1601  0.3109   \n",
       "1  0.0453  0.0523  0.0843  0.0689  0.1183  0.2583  0.2156  0.3481  0.3337   \n",
       "2  0.0262  0.0582  0.1099  0.1083  0.0974  0.2280  0.2431  0.3771  0.5598   \n",
       "3  0.0100  0.0171  0.0623  0.0205  0.0205  0.0368  0.1098  0.1276  0.0598   \n",
       "4  0.0762  0.0666  0.0481  0.0394  0.0590  0.0649  0.1209  0.2467  0.3564   \n",
       "\n",
       "       9   ...      51      52      53      54      55      56      57  \\\n",
       "0  0.2111  ...  0.0027  0.0065  0.0159  0.0072  0.0167  0.0180  0.0084   \n",
       "1  0.2872  ...  0.0084  0.0089  0.0048  0.0094  0.0191  0.0140  0.0049   \n",
       "2  0.6194  ...  0.0232  0.0166  0.0095  0.0180  0.0244  0.0316  0.0164   \n",
       "3  0.1264  ...  0.0121  0.0036  0.0150  0.0085  0.0073  0.0050  0.0044   \n",
       "4  0.4459  ...  0.0031  0.0054  0.0105  0.0110  0.0015  0.0072  0.0048   \n",
       "\n",
       "       58      59  60  \n",
       "0  0.0090  0.0032   R  \n",
       "1  0.0052  0.0044   R  \n",
       "2  0.0095  0.0078   R  \n",
       "3  0.0040  0.0117   R  \n",
       "4  0.0107  0.0094   R  \n",
       "\n",
       "[5 rows x 61 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=pd.read_csv(\"./0 resources/sonar.csv\",header=None)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "M    111\n",
       "R     97\n",
       "Name: 60, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[60].value_counts()   # M stands for Mines and R stands for Rocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "X=df.drop(60,axis=1)\n",
    "#OHE   -> because Deep Learning Model won't work with string data\n",
    "y=pd.get_dummies(df[60],drop_first=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "R\n",
       "0    111\n",
       "1     97\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.value_counts()  # R->1 & M->0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((139, 60), (139, 1))"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape , y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neural networks are defined in Keras as a **sequence of layers**. The container for these layers is the Sequential class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Building ANN model\n",
    "#input_shape-> builds the input layer of same number of neurons as the input features, 50 is the hidden layer\n",
    "\n",
    "model1= keras.Sequential([\n",
    "    \n",
    "    keras.layers.Dense(50, input_dim=60,activation=\"relu\"),  #input_shape=(60,)\n",
    "    keras.layers.Dense(30,activation=\"relu\"),\n",
    "    keras.layers.Dense(15,activation=\"relu\"),\n",
    "    keras.layers.Dense(1, activation='sigmoid')\n",
    "    \n",
    "])\n",
    "\n",
    "model1.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- A metric is a function that is used to judge the performance of your model. Metric functions are similar to loss functions, except that the **results from evaluating a metric are not used when training the model**. Note that you may use any loss function as a metric.\n",
    "\n",
    "\n",
    "- For the negative input values, the result is zero, that means the neuron does not get activated. Since only a certain number of neurons are activated, the **ReLU function is far more computationally efficient** when compared to the sigmoid and tanh function. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "batch_size: Number of samples per gradient update. \n",
    "\n",
    "\n",
    "Mini-Batch Gradient Descent. In each iteration, 8 samples are feeded calculate the loss and do back propagation(update weights)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/200\n",
      "18/18 [==============================] - 1s 11ms/step - loss: 0.6859 - accuracy: 0.6132\n",
      "Epoch 2/200\n",
      "18/18 [==============================] - 0s 13ms/step - loss: 0.6640 - accuracy: 0.6499 0s - loss: 0.6596 - accuracy: 0.\n",
      "Epoch 3/200\n",
      "18/18 [==============================] - 0s 11ms/step - loss: 0.6367 - accuracy: 0.7153\n",
      "Epoch 4/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 0.6208 - accuracy: 0.7407\n",
      "Epoch 5/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.5993 - accuracy: 0.6916\n",
      "Epoch 6/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.5665 - accuracy: 0.7152\n",
      "Epoch 7/200\n",
      "18/18 [==============================] - 0s 12ms/step - loss: 0.4992 - accuracy: 0.7750\n",
      "Epoch 8/200\n",
      "18/18 [==============================] - 0s 13ms/step - loss: 0.4981 - accuracy: 0.7981\n",
      "Epoch 9/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.4693 - accuracy: 0.8125\n",
      "Epoch 10/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.4062 - accuracy: 0.8579\n",
      "Epoch 11/200\n",
      "18/18 [==============================] - 0s 12ms/step - loss: 0.4137 - accuracy: 0.8211\n",
      "Epoch 12/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.3609 - accuracy: 0.9010\n",
      "Epoch 13/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.4175 - accuracy: 0.7981\n",
      "Epoch 14/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.4216 - accuracy: 0.8232\n",
      "Epoch 15/200\n",
      "18/18 [==============================] - 0s 11ms/step - loss: 0.4030 - accuracy: 0.8678\n",
      "Epoch 16/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 0.3911 - accuracy: 0.8135\n",
      "Epoch 17/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 0.3462 - accuracy: 0.8326\n",
      "Epoch 18/200\n",
      "18/18 [==============================] - 0s 11ms/step - loss: 0.3009 - accuracy: 0.8911\n",
      "Epoch 19/200\n",
      "18/18 [==============================] - 0s 15ms/step - loss: 0.3022 - accuracy: 0.8932\n",
      "Epoch 20/200\n",
      "18/18 [==============================] - 0s 17ms/step - loss: 0.2895 - accuracy: 0.8674\n",
      "Epoch 21/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.2923 - accuracy: 0.8571\n",
      "Epoch 22/200\n",
      "18/18 [==============================] - 0s 28ms/step - loss: 0.3012 - accuracy: 0.8870 0s - loss: 0.3363 - accuracy: \n",
      "Epoch 23/200\n",
      "18/18 [==============================] - 0s 13ms/step - loss: 0.2494 - accuracy: 0.8915\n",
      "Epoch 24/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.2546 - accuracy: 0.8852\n",
      "Epoch 25/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.2229 - accuracy: 0.8981\n",
      "Epoch 26/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.2554 - accuracy: 0.8964\n",
      "Epoch 27/200\n",
      "18/18 [==============================] - ETA: 0s - loss: 0.1958 - accuracy: 0.94 - 0s 7ms/step - loss: 0.1983 - accuracy: 0.9454\n",
      "Epoch 28/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.2338 - accuracy: 0.8906\n",
      "Epoch 29/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.1711 - accuracy: 0.9574\n",
      "Epoch 30/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.2041 - accuracy: 0.9242\n",
      "Epoch 31/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.2368 - accuracy: 0.8949\n",
      "Epoch 32/200\n",
      "18/18 [==============================] - 0s 12ms/step - loss: 0.1821 - accuracy: 0.9426\n",
      "Epoch 33/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.1746 - accuracy: 0.9551\n",
      "Epoch 34/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.1623 - accuracy: 0.9406\n",
      "Epoch 35/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.1633 - accuracy: 0.9604\n",
      "Epoch 36/200\n",
      "18/18 [==============================] - 0s 11ms/step - loss: 0.1330 - accuracy: 0.9637\n",
      "Epoch 37/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.1533 - accuracy: 0.9610\n",
      "Epoch 38/200\n",
      "18/18 [==============================] - 0s 12ms/step - loss: 0.1228 - accuracy: 0.9860\n",
      "Epoch 39/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.1134 - accuracy: 0.9758\n",
      "Epoch 40/200\n",
      "18/18 [==============================] - ETA: 0s - loss: 0.2005 - accuracy: 0.9037 ETA: 0s - loss: 0.2724 - accuracy:  - 0s 11ms/step - loss: 0.1980 - accuracy: 0.9061\n",
      "Epoch 41/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.1842 - accuracy: 0.9240\n",
      "Epoch 42/200\n",
      "18/18 [==============================] - 0s 12ms/step - loss: 0.1705 - accuracy: 0.9480\n",
      "Epoch 43/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.1078 - accuracy: 0.9651\n",
      "Epoch 44/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 0.1370 - accuracy: 0.9452\n",
      "Epoch 45/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.0891 - accuracy: 0.9782\n",
      "Epoch 46/200\n",
      "18/18 [==============================] - 0s 14ms/step - loss: 0.0999 - accuracy: 0.9623\n",
      "Epoch 47/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0823 - accuracy: 0.9837\n",
      "Epoch 48/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0657 - accuracy: 0.9905\n",
      "Epoch 49/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0505 - accuracy: 0.9975\n",
      "Epoch 50/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 0.0800 - accuracy: 0.9766\n",
      "Epoch 51/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0688 - accuracy: 1.0000\n",
      "Epoch 52/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0474 - accuracy: 1.0000\n",
      "Epoch 53/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0497 - accuracy: 1.0000\n",
      "Epoch 54/200\n",
      "18/18 [==============================] - 0s 12ms/step - loss: 0.0441 - accuracy: 0.9960\n",
      "Epoch 55/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0574 - accuracy: 0.9832\n",
      "Epoch 56/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 0.0390 - accuracy: 1.0000\n",
      "Epoch 57/200\n",
      "18/18 [==============================] - 0s 15ms/step - loss: 0.0373 - accuracy: 1.0000\n",
      "Epoch 58/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0274 - accuracy: 1.0000\n",
      "Epoch 59/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0336 - accuracy: 1.0000\n",
      "Epoch 60/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0315 - accuracy: 1.0000\n",
      "Epoch 61/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0356 - accuracy: 0.9865\n",
      "Epoch 62/200\n",
      "18/18 [==============================] - 0s 12ms/step - loss: 0.0225 - accuracy: 1.0000\n",
      "Epoch 63/200\n",
      "18/18 [==============================] - 0s 12ms/step - loss: 0.0213 - accuracy: 1.0000\n",
      "Epoch 64/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.0168 - accuracy: 1.0000\n",
      "Epoch 65/200\n",
      "18/18 [==============================] - 0s 11ms/step - loss: 0.0155 - accuracy: 1.0000\n",
      "Epoch 66/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0325 - accuracy: 1.0000: 0s - loss: 0.0347 - accuracy: 1.00\n",
      "Epoch 67/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 0.0165 - accuracy: 1.0000\n",
      "Epoch 68/200\n",
      "18/18 [==============================] - 0s 14ms/step - loss: 0.0154 - accuracy: 1.0000\n",
      "Epoch 69/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.0140 - accuracy: 1.0000\n",
      "Epoch 70/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0158 - accuracy: 1.0000\n",
      "Epoch 71/200\n",
      "18/18 [==============================] - 0s 15ms/step - loss: 0.0138 - accuracy: 1.0000\n",
      "Epoch 72/200\n",
      "18/18 [==============================] - 0s 16ms/step - loss: 0.0138 - accuracy: 1.0000\n",
      "Epoch 73/200\n",
      "18/18 [==============================] - 0s 11ms/step - loss: 0.0105 - accuracy: 1.0000\n",
      "Epoch 74/200\n",
      "18/18 [==============================] - 0s 16ms/step - loss: 0.0103 - accuracy: 1.0000\n",
      "Epoch 75/200\n",
      "18/18 [==============================] - 0s 12ms/step - loss: 0.0104 - accuracy: 1.0000\n",
      "Epoch 76/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0093 - accuracy: 1.0000\n",
      "Epoch 77/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0097 - accuracy: 1.0000\n",
      "Epoch 78/200\n",
      "18/18 [==============================] - 0s 14ms/step - loss: 0.0089 - accuracy: 1.0000\n",
      "Epoch 79/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0090 - accuracy: 1.0000\n",
      "Epoch 80/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0071 - accuracy: 1.0000\n",
      "Epoch 81/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 1.0000\n",
      "Epoch 82/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.0060 - accuracy: 1.0000\n",
      "Epoch 83/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0085 - accuracy: 1.0000\n",
      "Epoch 84/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.0087 - accuracy: 1.0000\n",
      "Epoch 85/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 0.0060 - accuracy: 1.0000\n",
      "Epoch 86/200\n",
      "18/18 [==============================] - 0s 11ms/step - loss: 0.0061 - accuracy: 1.0000\n",
      "Epoch 87/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0058 - accuracy: 1.0000\n",
      "Epoch 88/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0049 - accuracy: 1.0000\n",
      "Epoch 89/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.0047 - accuracy: 1.0000\n",
      "Epoch 90/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0126 - accuracy: 1.0000\n",
      "Epoch 91/200\n",
      "18/18 [==============================] - 0s 13ms/step - loss: 0.0047 - accuracy: 1.0000\n",
      "Epoch 92/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0052 - accuracy: 1.0000\n",
      "Epoch 93/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 0.0039 - accuracy: 1.0000\n",
      "Epoch 94/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 0.0049 - accuracy: 1.0000\n",
      "Epoch 95/200\n",
      "18/18 [==============================] - 0s 13ms/step - loss: 0.0038 - accuracy: 1.0000\n",
      "Epoch 96/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 0.0036 - accuracy: 1.0000\n",
      "Epoch 97/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0042 - accuracy: 1.0000\n",
      "Epoch 98/200\n",
      "18/18 [==============================] - 0s 20ms/step - loss: 0.0036 - accuracy: 1.0000\n",
      "Epoch 99/200\n",
      "18/18 [==============================] - 0s 11ms/step - loss: 0.0032 - accuracy: 1.0000\n",
      "Epoch 100/200\n",
      "18/18 [==============================] - 0s 13ms/step - loss: 0.0026 - accuracy: 1.0000\n",
      "Epoch 101/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.0036 - accuracy: 1.0000\n",
      "Epoch 102/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.0035 - accuracy: 1.0000\n",
      "Epoch 103/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.0031 - accuracy: 1.0000\n",
      "Epoch 104/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.0027 - accuracy: 1.0000\n",
      "Epoch 105/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.0027 - accuracy: 1.0000\n",
      "Epoch 106/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0023 - accuracy: 1.0000\n",
      "Epoch 107/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0029 - accuracy: 1.0000\n",
      "Epoch 108/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.0023 - accuracy: 1.0000\n",
      "Epoch 109/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0018 - accuracy: 1.0000\n",
      "Epoch 110/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0022 - accuracy: 1.0000\n",
      "Epoch 111/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.0023 - accuracy: 1.0000\n",
      "Epoch 112/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0023 - accuracy: 1.0000\n",
      "Epoch 113/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0021 - accuracy: 1.0000\n",
      "Epoch 114/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.0017 - accuracy: 1.0000\n",
      "Epoch 115/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0023 - accuracy: 1.0000\n",
      "Epoch 116/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0019 - accuracy: 1.0000\n",
      "Epoch 117/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0017 - accuracy: 1.0000\n",
      "Epoch 118/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.0019 - accuracy: 1.0000\n",
      "Epoch 119/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 0.0017 - accuracy: 1.0000\n",
      "Epoch 120/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0014 - accuracy: 1.0000\n",
      "Epoch 121/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0014 - accuracy: 1.0000\n",
      "Epoch 122/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0013 - accuracy: 1.0000\n",
      "Epoch 123/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0016 - accuracy: 1.0000\n",
      "Epoch 124/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0016 - accuracy: 1.0000\n",
      "Epoch 125/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.0017 - accuracy: 1.0000\n",
      "Epoch 126/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.0014 - accuracy: 1.0000\n",
      "Epoch 127/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.0016 - accuracy: 1.0000\n",
      "Epoch 128/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0012 - accuracy: 1.0000\n",
      "Epoch 129/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0013 - accuracy: 1.0000 ETA: 0s - loss: 0.0013 - accuracy: 1.0000  \n",
      "Epoch 130/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0013 - accuracy: 1.0000\n",
      "Epoch 131/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 9.5456e-04 - accuracy: 1.0000\n",
      "Epoch 132/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.0010 - accuracy: 1.0000\n",
      "Epoch 133/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 8.9822e-04 - accuracy: 1.0000\n",
      "Epoch 134/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0012 - accuracy: 1.0000\n",
      "Epoch 135/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.0014 - accuracy: 1.0000\n",
      "Epoch 136/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.0011 - accuracy: 1.0000\n",
      "Epoch 137/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 9.4713e-04 - accuracy: 1.0000\n",
      "Epoch 138/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 9.3684e-04 - accuracy: 1.0000\n",
      "Epoch 139/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 9.0917e-04 - accuracy: 1.0000\n",
      "Epoch 140/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 9.1110e-04 - accuracy: 1.0000\n",
      "Epoch 141/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.0011 - accuracy: 1.0000\n",
      "Epoch 142/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 7.2389e-04 - accuracy: 1.0000\n",
      "Epoch 143/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 9.8670e-04 - accuracy: 1.0000\n",
      "Epoch 144/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 9.4770e-04 - accuracy: 1.0000\n",
      "Epoch 145/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 8.6826e-04 - accuracy: 1.0000\n",
      "Epoch 146/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 7.2152e-04 - accuracy: 1.0000\n",
      "Epoch 147/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 8.6832e-04 - accuracy: 1.0000\n",
      "Epoch 148/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 6.3205e-04 - accuracy: 1.0000\n",
      "Epoch 149/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 8.0901e-04 - accuracy: 1.0000\n",
      "Epoch 150/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 7.6703e-04 - accuracy: 1.0000\n",
      "Epoch 151/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 6.8574e-04 - accuracy: 1.0000\n",
      "Epoch 152/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 6.0604e-04 - accuracy: 1.0000\n",
      "Epoch 153/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 7.7462e-04 - accuracy: 1.0000\n",
      "Epoch 154/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 6.0541e-04 - accuracy: 1.0000\n",
      "Epoch 155/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 6.5899e-04 - accuracy: 1.0000\n",
      "Epoch 156/200\n",
      "18/18 [==============================] - 0s 18ms/step - loss: 6.3346e-04 - accuracy: 1.0000\n",
      "Epoch 157/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 5.0238e-04 - accuracy: 1.0000\n",
      "Epoch 158/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 6.3427e-04 - accuracy: 1.0000\n",
      "Epoch 159/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18/18 [==============================] - 0s 7ms/step - loss: 6.1293e-04 - accuracy: 1.0000\n",
      "Epoch 160/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 6.3845e-04 - accuracy: 1.0000\n",
      "Epoch 161/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 6.4050e-04 - accuracy: 1.0000\n",
      "Epoch 162/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 6.7679e-04 - accuracy: 1.0000\n",
      "Epoch 163/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 6.5725e-04 - accuracy: 1.0000\n",
      "Epoch 164/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 5.4786e-04 - accuracy: 1.0000\n",
      "Epoch 165/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 5.2956e-04 - accuracy: 1.0000\n",
      "Epoch 166/200\n",
      "18/18 [==============================] - ETA: 0s - loss: 7.3554e-04 - accuracy: 1.00 - 0s 9ms/step - loss: 6.7608e-04 - accuracy: 1.0000\n",
      "Epoch 167/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 6.0484e-04 - accuracy: 1.0000\n",
      "Epoch 168/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 4.4707e-04 - accuracy: 1.0000\n",
      "Epoch 169/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 4.9437e-04 - accuracy: 1.0000\n",
      "Epoch 170/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 4.3738e-04 - accuracy: 1.0000\n",
      "Epoch 171/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 5.0545e-04 - accuracy: 1.0000\n",
      "Epoch 172/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 4.8167e-04 - accuracy: 1.0000\n",
      "Epoch 173/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 4.9960e-04 - accuracy: 1.0000\n",
      "Epoch 174/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 5.2700e-04 - accuracy: 1.0000\n",
      "Epoch 175/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 4.3119e-04 - accuracy: 1.0000\n",
      "Epoch 176/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 3.8467e-04 - accuracy: 1.0000\n",
      "Epoch 177/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 4.8603e-04 - accuracy: 1.0000: 0s - loss: 5.1299e-04 - accuracy: 1.00\n",
      "Epoch 178/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 3.7501e-04 - accuracy: 1.0000\n",
      "Epoch 179/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 3.7137e-04 - accuracy: 1.0000\n",
      "Epoch 180/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 4.9931e-04 - accuracy: 1.0000\n",
      "Epoch 181/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 3.7403e-04 - accuracy: 1.0000\n",
      "Epoch 182/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 4.7409e-04 - accuracy: 1.0000\n",
      "Epoch 183/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 4.7823e-04 - accuracy: 1.0000\n",
      "Epoch 184/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 4.0252e-04 - accuracy: 1.0000\n",
      "Epoch 185/200\n",
      "18/18 [==============================] - 0s 10ms/step - loss: 3.0879e-04 - accuracy: 1.0000\n",
      "Epoch 186/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 3.5172e-04 - accuracy: 1.0000\n",
      "Epoch 187/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 4.1460e-04 - accuracy: 1.0000\n",
      "Epoch 188/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 3.1892e-04 - accuracy: 1.0000\n",
      "Epoch 189/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 3.6363e-04 - accuracy: 1.0000\n",
      "Epoch 190/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 3.0819e-04 - accuracy: 1.0000\n",
      "Epoch 191/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 4.6086e-04 - accuracy: 1.0000\n",
      "Epoch 192/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 3.2271e-04 - accuracy: 1.0000\n",
      "Epoch 193/200\n",
      "18/18 [==============================] - 0s 11ms/step - loss: 3.4398e-04 - accuracy: 1.0000\n",
      "Epoch 194/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 2.8787e-04 - accuracy: 1.0000\n",
      "Epoch 195/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 3.2195e-04 - accuracy: 1.0000\n",
      "Epoch 196/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 3.3782e-04 - accuracy: 1.0000\n",
      "Epoch 197/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 2.6009e-04 - accuracy: 1.0000\n",
      "Epoch 198/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 3.2479e-04 - accuracy: 1.0000\n",
      "Epoch 199/200\n",
      "18/18 [==============================] - 0s 14ms/step - loss: 3.1303e-04 - accuracy: 1.0000\n",
      "Epoch 200/200\n",
      "18/18 [==============================] - 0s 12ms/step - loss: 3.1350e-04 - accuracy: 1.0000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f8dcecd6310>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model1.fit(X_train,y_train,epochs=200,batch_size=8)   #returns a history object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model got the training-set accuracy of 100% but testing it against the test data , we are getting the accuracy of 85% only, this is due to Overfitting, I will introduce the **drop-out layer to drop neurons in between the hidden layers**, this will lead to neurons not depending on any one input, as it may be dropped at random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3/3 [==============================] - 0s 2ms/step - loss: 0.6206 - accuracy: 0.8551\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.6206290125846863, 0.8550724387168884]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model1.evaluate(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [0],\n",
       "       [1],\n",
       "       [1],\n",
       "       [0],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [0],\n",
       "       [1],\n",
       "       [1],\n",
       "       [0],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [0],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [0],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [1],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [1],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [1],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [1],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [0]], dtype=int32)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model1.predict_classes(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report,confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[34  4]\n",
      " [ 6 25]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.85      0.89      0.87        38\n",
      "           1       0.86      0.81      0.83        31\n",
      "\n",
      "    accuracy                           0.86        69\n",
      "   macro avg       0.86      0.85      0.85        69\n",
      "weighted avg       0.86      0.86      0.85        69\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred=model1.predict_classes(X_test)\n",
    "\n",
    "print(confusion_matrix(y_test,y_pred))\n",
    "print(classification_report(y_test,y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Introducing Dropout layers in between Hidden layers**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/200\n",
      "18/18 [==============================] - 1s 2ms/step - loss: 0.7478 - accuracy: 0.5305\n",
      "Epoch 2/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.7014 - accuracy: 0.5381\n",
      "Epoch 3/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.7005 - accuracy: 0.4907\n",
      "Epoch 4/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.6653 - accuracy: 0.5427\n",
      "Epoch 5/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.7583 - accuracy: 0.4389\n",
      "Epoch 6/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.7170 - accuracy: 0.4688\n",
      "Epoch 7/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.6881 - accuracy: 0.5271\n",
      "Epoch 8/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.6915 - accuracy: 0.5052\n",
      "Epoch 9/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.7001 - accuracy: 0.4938\n",
      "Epoch 10/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.6824 - accuracy: 0.6375\n",
      "Epoch 11/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.6800 - accuracy: 0.5824\n",
      "Epoch 12/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.7004 - accuracy: 0.4763\n",
      "Epoch 13/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.6978 - accuracy: 0.4927\n",
      "Epoch 14/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.6753 - accuracy: 0.5938\n",
      "Epoch 15/200\n",
      "18/18 [==============================] - 0s 9ms/step - loss: 0.6694 - accuracy: 0.6091\n",
      "Epoch 16/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.6757 - accuracy: 0.5910\n",
      "Epoch 17/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.6661 - accuracy: 0.6238\n",
      "Epoch 18/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.6676 - accuracy: 0.5867\n",
      "Epoch 19/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.6650 - accuracy: 0.5624\n",
      "Epoch 20/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.6764 - accuracy: 0.6295\n",
      "Epoch 21/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.6672 - accuracy: 0.6074\n",
      "Epoch 22/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.6433 - accuracy: 0.6167\n",
      "Epoch 23/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.6865 - accuracy: 0.5427\n",
      "Epoch 24/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.6476 - accuracy: 0.6803\n",
      "Epoch 25/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.6573 - accuracy: 0.6084\n",
      "Epoch 26/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.6678 - accuracy: 0.5586\n",
      "Epoch 27/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.6733 - accuracy: 0.6234\n",
      "Epoch 28/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.6144 - accuracy: 0.7235\n",
      "Epoch 29/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.6631 - accuracy: 0.6222\n",
      "Epoch 30/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.6752 - accuracy: 0.5498\n",
      "Epoch 31/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.6137 - accuracy: 0.7184\n",
      "Epoch 32/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.6339 - accuracy: 0.6374\n",
      "Epoch 33/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.5957 - accuracy: 0.6822\n",
      "Epoch 34/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.6203 - accuracy: 0.6542\n",
      "Epoch 35/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.5835 - accuracy: 0.6464\n",
      "Epoch 36/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.5740 - accuracy: 0.6608\n",
      "Epoch 37/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.6026 - accuracy: 0.6588: 0s - loss: 0.6138 - accuracy: 0.64\n",
      "Epoch 38/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.6324 - accuracy: 0.6526\n",
      "Epoch 39/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.6242 - accuracy: 0.6083\n",
      "Epoch 40/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.6005 - accuracy: 0.6404\n",
      "Epoch 41/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.5817 - accuracy: 0.7571\n",
      "Epoch 42/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.5851 - accuracy: 0.7181\n",
      "Epoch 43/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.5861 - accuracy: 0.6856\n",
      "Epoch 44/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.6103 - accuracy: 0.6867\n",
      "Epoch 45/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.5836 - accuracy: 0.5905\n",
      "Epoch 46/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5105 - accuracy: 0.7836\n",
      "Epoch 47/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5445 - accuracy: 0.7371\n",
      "Epoch 48/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5736 - accuracy: 0.6733\n",
      "Epoch 49/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5954 - accuracy: 0.6892\n",
      "Epoch 50/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5120 - accuracy: 0.7846\n",
      "Epoch 51/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4605 - accuracy: 0.7947\n",
      "Epoch 52/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5139 - accuracy: 0.6902\n",
      "Epoch 53/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5727 - accuracy: 0.7123\n",
      "Epoch 54/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.5785 - accuracy: 0.6746\n",
      "Epoch 55/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5988 - accuracy: 0.6945\n",
      "Epoch 56/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.5546 - accuracy: 0.7844\n",
      "Epoch 57/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5011 - accuracy: 0.7448\n",
      "Epoch 58/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4751 - accuracy: 0.7841\n",
      "Epoch 59/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5413 - accuracy: 0.7849\n",
      "Epoch 60/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5492 - accuracy: 0.7715\n",
      "Epoch 61/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4619 - accuracy: 0.7752\n",
      "Epoch 62/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4695 - accuracy: 0.8639\n",
      "Epoch 63/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4355 - accuracy: 0.8313\n",
      "Epoch 64/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4436 - accuracy: 0.7816\n",
      "Epoch 65/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4230 - accuracy: 0.8027\n",
      "Epoch 66/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5087 - accuracy: 0.7168\n",
      "Epoch 67/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4238 - accuracy: 0.8256\n",
      "Epoch 68/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.4895 - accuracy: 0.7852\n",
      "Epoch 69/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.4528 - accuracy: 0.8445\n",
      "Epoch 70/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4875 - accuracy: 0.7816\n",
      "Epoch 71/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5224 - accuracy: 0.7542\n",
      "Epoch 72/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.4531 - accuracy: 0.7531\n",
      "Epoch 73/200\n",
      "18/18 [==============================] - 0s 8ms/step - loss: 0.4234 - accuracy: 0.7698\n",
      "Epoch 74/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.3983 - accuracy: 0.8483\n",
      "Epoch 75/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.4003 - accuracy: 0.7953\n",
      "Epoch 76/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.4501 - accuracy: 0.7581\n",
      "Epoch 77/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.3723 - accuracy: 0.8236\n",
      "Epoch 78/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3651 - accuracy: 0.8401\n",
      "Epoch 79/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3825 - accuracy: 0.8023\n",
      "Epoch 80/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.4660 - accuracy: 0.7979\n",
      "Epoch 81/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4048 - accuracy: 0.8648\n",
      "Epoch 82/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.3698 - accuracy: 0.8709\n",
      "Epoch 83/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.3903 - accuracy: 0.7851\n",
      "Epoch 84/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.3632 - accuracy: 0.8521\n",
      "Epoch 85/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.4108 - accuracy: 0.7870\n",
      "Epoch 86/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.5332 - accuracy: 0.7003\n",
      "Epoch 87/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3535 - accuracy: 0.8691\n",
      "Epoch 88/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3970 - accuracy: 0.7960\n",
      "Epoch 89/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.3772 - accuracy: 0.8447\n",
      "Epoch 90/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.4155 - accuracy: 0.8768\n",
      "Epoch 91/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3435 - accuracy: 0.8573\n",
      "Epoch 92/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.4002 - accuracy: 0.8197\n",
      "Epoch 93/200\n",
      "18/18 [==============================] - 0s 11ms/step - loss: 0.3137 - accuracy: 0.8877\n",
      "Epoch 94/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.3706 - accuracy: 0.8606\n",
      "Epoch 95/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4217 - accuracy: 0.7848\n",
      "Epoch 96/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4260 - accuracy: 0.8274\n",
      "Epoch 97/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3879 - accuracy: 0.8719\n",
      "Epoch 98/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3147 - accuracy: 0.8831\n",
      "Epoch 99/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4016 - accuracy: 0.8444\n",
      "Epoch 100/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.4509 - accuracy: 0.8179\n",
      "Epoch 101/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2618 - accuracy: 0.9087\n",
      "Epoch 102/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3370 - accuracy: 0.8771\n",
      "Epoch 103/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3317 - accuracy: 0.8857\n",
      "Epoch 104/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2438 - accuracy: 0.9082\n",
      "Epoch 105/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3296 - accuracy: 0.8736\n",
      "Epoch 106/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2647 - accuracy: 0.9157\n",
      "Epoch 107/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2949 - accuracy: 0.8584\n",
      "Epoch 108/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2924 - accuracy: 0.8904\n",
      "Epoch 109/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3073 - accuracy: 0.9004\n",
      "Epoch 110/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2629 - accuracy: 0.8967\n",
      "Epoch 111/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2518 - accuracy: 0.8930\n",
      "Epoch 112/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2529 - accuracy: 0.9072\n",
      "Epoch 113/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3644 - accuracy: 0.9072\n",
      "Epoch 114/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3650 - accuracy: 0.8177\n",
      "Epoch 115/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3229 - accuracy: 0.8753\n",
      "Epoch 116/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.3108 - accuracy: 0.8703\n",
      "Epoch 117/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2856 - accuracy: 0.8945\n",
      "Epoch 118/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2234 - accuracy: 0.9309\n",
      "Epoch 119/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.3401 - accuracy: 0.8527\n",
      "Epoch 120/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.1854 - accuracy: 0.9261\n",
      "Epoch 121/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.2172 - accuracy: 0.9636\n",
      "Epoch 122/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2010 - accuracy: 0.9336\n",
      "Epoch 123/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2929 - accuracy: 0.9036\n",
      "Epoch 124/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2738 - accuracy: 0.8938\n",
      "Epoch 125/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1876 - accuracy: 0.9156\n",
      "Epoch 126/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2181 - accuracy: 0.8957\n",
      "Epoch 127/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2171 - accuracy: 0.9364\n",
      "Epoch 128/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1853 - accuracy: 0.9336\n",
      "Epoch 129/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2779 - accuracy: 0.9022\n",
      "Epoch 130/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2183 - accuracy: 0.9155\n",
      "Epoch 131/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2672 - accuracy: 0.8892\n",
      "Epoch 132/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2440 - accuracy: 0.8891\n",
      "Epoch 133/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2276 - accuracy: 0.9103\n",
      "Epoch 134/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2222 - accuracy: 0.8913\n",
      "Epoch 135/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2711 - accuracy: 0.8537\n",
      "Epoch 136/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.3108 - accuracy: 0.8621\n",
      "Epoch 137/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.2206 - accuracy: 0.9030\n",
      "Epoch 138/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2561 - accuracy: 0.8894\n",
      "Epoch 139/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2459 - accuracy: 0.9051\n",
      "Epoch 140/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2126 - accuracy: 0.9386\n",
      "Epoch 141/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2060 - accuracy: 0.9332\n",
      "Epoch 142/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2669 - accuracy: 0.8735\n",
      "Epoch 143/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.2123 - accuracy: 0.9174\n",
      "Epoch 144/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2034 - accuracy: 0.9460\n",
      "Epoch 145/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.4506 - accuracy: 0.7798\n",
      "Epoch 146/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1992 - accuracy: 0.9300\n",
      "Epoch 147/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1454 - accuracy: 0.9570\n",
      "Epoch 148/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1816 - accuracy: 0.9387\n",
      "Epoch 149/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.1582 - accuracy: 0.9270\n",
      "Epoch 150/200\n",
      "18/18 [==============================] - 0s 6ms/step - loss: 0.1934 - accuracy: 0.9168\n",
      "Epoch 151/200\n",
      "18/18 [==============================] - 0s 12ms/step - loss: 0.1820 - accuracy: 0.9549\n",
      "Epoch 152/200\n",
      "18/18 [==============================] - 0s 7ms/step - loss: 0.2058 - accuracy: 0.9158\n",
      "Epoch 153/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1432 - accuracy: 0.9748\n",
      "Epoch 154/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.1420 - accuracy: 0.9761\n",
      "Epoch 155/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.1718 - accuracy: 0.9509\n",
      "Epoch 156/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2552 - accuracy: 0.8850\n",
      "Epoch 157/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1296 - accuracy: 0.9532\n",
      "Epoch 158/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2144 - accuracy: 0.9266\n",
      "Epoch 159/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.1859 - accuracy: 0.9080\n",
      "Epoch 160/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2143 - accuracy: 0.9085\n",
      "Epoch 161/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1804 - accuracy: 0.9207\n",
      "Epoch 162/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1950 - accuracy: 0.9091\n",
      "Epoch 163/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1634 - accuracy: 0.9182\n",
      "Epoch 164/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1747 - accuracy: 0.9173\n",
      "Epoch 165/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1942 - accuracy: 0.9199\n",
      "Epoch 166/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1068 - accuracy: 0.9677\n",
      "Epoch 167/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1791 - accuracy: 0.9108\n",
      "Epoch 168/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1474 - accuracy: 0.9276\n",
      "Epoch 169/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.1275 - accuracy: 0.9418\n",
      "Epoch 170/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.1356 - accuracy: 0.9509\n",
      "Epoch 171/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2066 - accuracy: 0.9195\n",
      "Epoch 172/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2373 - accuracy: 0.8733\n",
      "Epoch 173/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1004 - accuracy: 0.9636\n",
      "Epoch 174/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1457 - accuracy: 0.9196\n",
      "Epoch 175/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1196 - accuracy: 0.9645\n",
      "Epoch 176/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1234 - accuracy: 0.9557\n",
      "Epoch 177/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1961 - accuracy: 0.9178\n",
      "Epoch 178/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1077 - accuracy: 0.9730\n",
      "Epoch 179/200\n",
      "18/18 [==============================] - 0s 4ms/step - loss: 0.1607 - accuracy: 0.9725\n",
      "Epoch 180/200\n",
      "18/18 [==============================] - 0s 5ms/step - loss: 0.1339 - accuracy: 0.9246\n",
      "Epoch 181/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1798 - accuracy: 0.8997\n",
      "Epoch 182/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1218 - accuracy: 0.9439\n",
      "Epoch 183/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1658 - accuracy: 0.9486\n",
      "Epoch 184/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1704 - accuracy: 0.9075\n",
      "Epoch 185/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1540 - accuracy: 0.9518\n",
      "Epoch 186/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.0914 - accuracy: 0.9668\n",
      "Epoch 187/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1358 - accuracy: 0.9172\n",
      "Epoch 188/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.2204 - accuracy: 0.8916\n",
      "Epoch 189/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1832 - accuracy: 0.8968\n",
      "Epoch 190/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.2006 - accuracy: 0.8984\n",
      "Epoch 191/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1796 - accuracy: 0.9076\n",
      "Epoch 192/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1621 - accuracy: 0.9538\n",
      "Epoch 193/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1862 - accuracy: 0.8910\n",
      "Epoch 194/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1091 - accuracy: 0.9635\n",
      "Epoch 195/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1461 - accuracy: 0.9126\n",
      "Epoch 196/200\n",
      "18/18 [==============================] - 0s 3ms/step - loss: 0.1999 - accuracy: 0.9466\n",
      "Epoch 197/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1791 - accuracy: 0.9448\n",
      "Epoch 198/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1544 - accuracy: 0.9141\n",
      "Epoch 199/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1386 - accuracy: 0.9670\n",
      "Epoch 200/200\n",
      "18/18 [==============================] - 0s 2ms/step - loss: 0.1418 - accuracy: 0.9292\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f8dd0037c90>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2=keras.Sequential([\n",
    "    keras.layers.Dense(50, input_dim=60,activation=\"relu\"),\n",
    "    keras.layers.Dropout(0.5),  #drops 50% data,  Dropout layers are introduced between hidden layers\n",
    "    keras.layers.Dense(30,activation=\"relu\"),\n",
    "    keras.layers.Dropout(0.5),  #drops 50% data\n",
    "    keras.layers.Dense(15,activation=\"relu\"),\n",
    "    keras.layers.Dropout(0.5),  #drops 50% data\n",
    "    keras.layers.Dense(1, activation='sigmoid')\n",
    "])\n",
    "\n",
    "model2.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n",
    "\n",
    "model2.fit(X_train,y_train,epochs=200,batch_size=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3/3 [==============================] - 0s 3ms/step - loss: 0.4661 - accuracy: 0.8696\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.46607494354248047, 0.8695651888847351]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.evaluate(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[37  1]\n",
      " [ 8 23]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.82      0.97      0.89        38\n",
      "           1       0.96      0.74      0.84        31\n",
      "\n",
      "    accuracy                           0.87        69\n",
      "   macro avg       0.89      0.86      0.86        69\n",
      "weighted avg       0.88      0.87      0.87        69\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred=model2.predict_classes(X_test)\n",
    "\n",
    "print(confusion_matrix(y_test,y_pred))\n",
    "print(classification_report(y_test,y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Improved f1 score and Accuracy**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_4 (Dense)              (None, 50)                3050      \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 30)                1530      \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 30)                0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 15)                465       \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 15)                0         \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 1)                 16        \n",
      "=================================================================\n",
      "Total params: 5,061\n",
      "Trainable params: 5,061\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model2.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "model2.save(\"./0 resources/DropoutModel.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import load_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_model=load_model(\"./0 resources/DropoutModel.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_4 (Dense)              (None, 50)                3050      \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 30)                1530      \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 30)                0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 15)                465       \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 15)                0         \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 1)                 16        \n",
      "=================================================================\n",
      "Total params: 5,061\n",
      "Trainable params: 5,061\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "new_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reference: https://machinelearningmastery.com/5-step-life-cycle-neural-network-models-keras/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}