3.Modelling_and_Hyperparameter_Tuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "# 3. Modelling and Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.1 Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\candy\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:144: FutureWarning: The sklearn.metrics.classification module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.metrics. Anything that cannot be imported from sklearn.metrics is now part of the private API.\n",
      "  warnings.warn(message, FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split, StratifiedKFold\n",
    "\n",
    "import pickle\n",
    "\n",
    "from Classification import Classification\n",
    "from Ensemble import Ensemble"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "I have created .py files; Classifiction.py and Ensemble.py with classes, that contain functions to simplify the modelling process, and to neaten up the modelling notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "sns.set_context('poster')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "x_train = pd.read_csv('Data/3.x_train_data.csv')\n",
    "y_train = pd.read_csv('Data/3.y_train_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7524, 138)\n",
      "(7524, 1)\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape)\n",
    "print(y_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.2 Train and Validation Split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "I did another data split into Train and Validation data in preparation for using GridSearch Cross Validation. I also chose Stratified 5-fold has a my choice for cross validating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "x_train, x_val, y_train, y_val = train_test_split(x_train,y_train['score'],test_size=.25,random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "skf = StratifiedKFold(n_splits=5,random_state=42,shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.3 Decision Tree (Baseline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.3.1 1st Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'min_samples_leaf':[3,5,10,15,30,50,100],\n",
    "          'max_depth':[3,4,5,6,7,8,9]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "dec_tree_1 = Classification('Decision Tree',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Decision Tree</td>\n",
       "      <td>0.432926</td>\n",
       "      <td>0.353535</td>\n",
       "      <td>0.07939</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0  Decision Tree        0.432926             0.353535              0.07939"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'max_depth': 9, 'min_samples_leaf': 5} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.531381</td>\n",
       "      <td>0.256169</td>\n",
       "      <td>0.290323</td>\n",
       "      <td>0.349057</td>\n",
       "      <td>0.495702</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.564767</td>\n",
       "      <td>0.095745</td>\n",
       "      <td>0.310056</td>\n",
       "      <td>0.455263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.409677</td>\n",
       "      <td>0.352466</td>\n",
       "      <td>0.144000</td>\n",
       "      <td>0.328402</td>\n",
       "      <td>0.474623</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.531381  0.256169  0.290323  0.349057  0.495702\n",
       "recall     0.333333  0.564767  0.095745  0.310056  0.455263\n",
       "f1-score   0.409677  0.352466  0.144000  0.328402  0.474623"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dec_tree_1.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.3.2 2nd Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'min_samples_leaf':[3,4,5],\n",
    "          'max_depth':[7,8,9,10]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "dec_tree_2 = Classification('Decision Tree',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Decision Tree</td>\n",
       "      <td>0.432926</td>\n",
       "      <td>0.353535</td>\n",
       "      <td>0.07939</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0  Decision Tree        0.432926             0.353535              0.07939"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'max_depth': 9, 'min_samples_leaf': 5} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.531381</td>\n",
       "      <td>0.256169</td>\n",
       "      <td>0.290323</td>\n",
       "      <td>0.349057</td>\n",
       "      <td>0.495702</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.564767</td>\n",
       "      <td>0.095745</td>\n",
       "      <td>0.310056</td>\n",
       "      <td>0.455263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.409677</td>\n",
       "      <td>0.352466</td>\n",
       "      <td>0.144000</td>\n",
       "      <td>0.328402</td>\n",
       "      <td>0.474623</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.531381  0.256169  0.290323  0.349057  0.495702\n",
       "recall     0.333333  0.564767  0.095745  0.310056  0.455263\n",
       "f1-score   0.409677  0.352466  0.144000  0.328402  0.474623"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dec_tree_2.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.4 Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.4.1 1st Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'min_samples_leaf':[3,5,10,15,30,50,100],\n",
    "          'max_depth':[3,5,7,9,11,13,15]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "ran_for_1 = Classification('Random Forest',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Random Forest</td>\n",
       "      <td>0.575403</td>\n",
       "      <td>0.454014</td>\n",
       "      <td>0.121389</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0  Random Forest        0.575403             0.454014             0.121389"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'max_depth': 15, 'min_samples_leaf': 10} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.485769</td>\n",
       "      <td>0.408451</td>\n",
       "      <td>0.412000</td>\n",
       "      <td>0.381963</td>\n",
       "      <td>0.530474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.671916</td>\n",
       "      <td>0.300518</td>\n",
       "      <td>0.273936</td>\n",
       "      <td>0.402235</td>\n",
       "      <td>0.618421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.563877</td>\n",
       "      <td>0.346269</td>\n",
       "      <td>0.329073</td>\n",
       "      <td>0.391837</td>\n",
       "      <td>0.571081</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.485769  0.408451  0.412000  0.381963  0.530474\n",
       "recall     0.671916  0.300518  0.273936  0.402235  0.618421\n",
       "f1-score   0.563877  0.346269  0.329073  0.391837  0.571081"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ran_for_1.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.4.2 2nd Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'min_samples_leaf':[7,8,9,10,11,12,13,14],\n",
    "          'max_depth':[13,14,15,16,17,18]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "ran_for_2 = Classification('Random Forest',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Random Forest</td>\n",
       "      <td>0.586745</td>\n",
       "      <td>0.45933</td>\n",
       "      <td>0.127414</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0  Random Forest        0.586745              0.45933             0.127414"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'max_depth': 18, 'min_samples_leaf': 11} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.504854</td>\n",
       "      <td>0.404000</td>\n",
       "      <td>0.434028</td>\n",
       "      <td>0.393701</td>\n",
       "      <td>0.510067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.682415</td>\n",
       "      <td>0.261658</td>\n",
       "      <td>0.332447</td>\n",
       "      <td>0.418994</td>\n",
       "      <td>0.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.580357</td>\n",
       "      <td>0.317610</td>\n",
       "      <td>0.376506</td>\n",
       "      <td>0.405954</td>\n",
       "      <td>0.551391</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.504854  0.404000  0.434028  0.393701  0.510067\n",
       "recall     0.682415  0.261658  0.332447  0.418994  0.600000\n",
       "f1-score   0.580357  0.317610  0.376506  0.405954  0.551391"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ran_for_2.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.4.3 3rd Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'min_samples_leaf':[7,8,9,10,11,12,13,14],\n",
    "          'max_depth':[17,18,19,20,21,22]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "ran_for_3 = Classification('Random Forest',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Random Forest</td>\n",
       "      <td>0.647351</td>\n",
       "      <td>0.457735</td>\n",
       "      <td>0.189615</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0  Random Forest        0.647351             0.457735             0.189615"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'max_depth': 22, 'min_samples_leaf': 7} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.493952</td>\n",
       "      <td>0.390728</td>\n",
       "      <td>0.423358</td>\n",
       "      <td>0.383632</td>\n",
       "      <td>0.555024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.643045</td>\n",
       "      <td>0.305699</td>\n",
       "      <td>0.308511</td>\n",
       "      <td>0.418994</td>\n",
       "      <td>0.610526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.558723</td>\n",
       "      <td>0.343023</td>\n",
       "      <td>0.356923</td>\n",
       "      <td>0.400534</td>\n",
       "      <td>0.581454</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.493952  0.390728  0.423358  0.383632  0.555024\n",
       "recall     0.643045  0.305699  0.308511  0.418994  0.610526\n",
       "f1-score   0.558723  0.343023  0.356923  0.400534  0.581454"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ran_for_3.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.5 Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.5.1 1st Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'penalty':['l1','l2'],\n",
    "          'C':[0.01,0.05,0.1,0.5,1,5,10]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "log_reg_1 = Classification('Logistic Regression',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>0.554847</td>\n",
       "      <td>0.501329</td>\n",
       "      <td>0.053518</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Model Name  Train Accuracy  Validation Accuracy  \\\n",
       "0  Logistic Regression        0.554847             0.501329   \n",
       "\n",
       "   Accuracy Difference  \n",
       "0             0.053518  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'C': 0.1, 'penalty': 'l2'} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.547619</td>\n",
       "      <td>0.427762</td>\n",
       "      <td>0.508532</td>\n",
       "      <td>0.402439</td>\n",
       "      <td>0.612346</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.603675</td>\n",
       "      <td>0.391192</td>\n",
       "      <td>0.396277</td>\n",
       "      <td>0.460894</td>\n",
       "      <td>0.652632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.574282</td>\n",
       "      <td>0.408660</td>\n",
       "      <td>0.445441</td>\n",
       "      <td>0.429688</td>\n",
       "      <td>0.631847</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.547619  0.427762  0.508532  0.402439  0.612346\n",
       "recall     0.603675  0.391192  0.396277  0.460894  0.652632\n",
       "f1-score   0.574282  0.408660  0.445441  0.429688  0.631847"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_reg_1.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.5.2 2nd Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'penalty':['l1','l2'],\n",
    "          'C':[0.3,0.4,0.5,0.6,0,7]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "log_reg_2 = Classification('Logistic Regression',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>0.562112</td>\n",
       "      <td>0.509304</td>\n",
       "      <td>0.052809</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Model Name  Train Accuracy  Validation Accuracy  \\\n",
       "0  Logistic Regression        0.562112             0.509304   \n",
       "\n",
       "   Accuracy Difference  \n",
       "0             0.052809  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'C': 0.3, 'penalty': 'l2'} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.570025</td>\n",
       "      <td>0.410468</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.431525</td>\n",
       "      <td>0.617073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.608924</td>\n",
       "      <td>0.386010</td>\n",
       "      <td>0.417553</td>\n",
       "      <td>0.466480</td>\n",
       "      <td>0.665789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.588832</td>\n",
       "      <td>0.397864</td>\n",
       "      <td>0.455072</td>\n",
       "      <td>0.448322</td>\n",
       "      <td>0.640506</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.570025  0.410468  0.500000  0.431525  0.617073\n",
       "recall     0.608924  0.386010  0.417553  0.466480  0.665789\n",
       "f1-score   0.588832  0.397864  0.455072  0.448322  0.640506"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_reg_2.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.5.3 3rd Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'penalty':['l2'],\n",
    "          'C':[0.25,0.26,0.27,0.28,0.29,0.30,0.31,0.32]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "log_reg_3 = Classification('Logistic Regression',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>0.56229</td>\n",
       "      <td>0.50824</td>\n",
       "      <td>0.054049</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Model Name  Train Accuracy  Validation Accuracy  \\\n",
       "0  Logistic Regression         0.56229              0.50824   \n",
       "\n",
       "   Accuracy Difference  \n",
       "0             0.054049  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'C': 0.29, 'penalty': 'l2'} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.566176</td>\n",
       "      <td>0.410468</td>\n",
       "      <td>0.501597</td>\n",
       "      <td>0.430412</td>\n",
       "      <td>0.616137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.606299</td>\n",
       "      <td>0.386010</td>\n",
       "      <td>0.417553</td>\n",
       "      <td>0.466480</td>\n",
       "      <td>0.663158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.585551</td>\n",
       "      <td>0.397864</td>\n",
       "      <td>0.455733</td>\n",
       "      <td>0.447721</td>\n",
       "      <td>0.638783</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.566176  0.410468  0.501597  0.430412  0.616137\n",
       "recall     0.606299  0.386010  0.417553  0.466480  0.663158\n",
       "f1-score   0.585551  0.397864  0.455733  0.447721  0.638783"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_reg_3.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.6 Support Vector Machines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.6.1 1st Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'kernel':['poly'],\n",
    "          'degree':[2,3]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "svm_1 = Classification('SVM',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>SVM</td>\n",
       "      <td>0.810385</td>\n",
       "      <td>0.498139</td>\n",
       "      <td>0.312245</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0        SVM        0.810385             0.498139             0.312245"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'degree': 2, 'kernel': 'poly'} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.531513</td>\n",
       "      <td>0.411765</td>\n",
       "      <td>0.457227</td>\n",
       "      <td>0.430303</td>\n",
       "      <td>0.643646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.664042</td>\n",
       "      <td>0.398964</td>\n",
       "      <td>0.412234</td>\n",
       "      <td>0.396648</td>\n",
       "      <td>0.613158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.590432</td>\n",
       "      <td>0.405263</td>\n",
       "      <td>0.433566</td>\n",
       "      <td>0.412791</td>\n",
       "      <td>0.628032</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.531513  0.411765  0.457227  0.430303  0.643646\n",
       "recall     0.664042  0.398964  0.412234  0.396648  0.613158\n",
       "f1-score   0.590432  0.405263  0.433566  0.412791  0.628032"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm_1.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.6.2 2nd Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'C':[0.2,0.3,0.4],\n",
    "          'kernel':['linear'],\n",
    "          'gamma':['scale','auto']}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "svm_2 = Classification('SVM',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>SVM</td>\n",
       "      <td>0.559631</td>\n",
       "      <td>0.515152</td>\n",
       "      <td>0.04448</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0        SVM        0.559631             0.515152              0.04448"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'C': 0.2, 'gamma': 'scale', 'kernel': 'linear'} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.561033</td>\n",
       "      <td>0.426702</td>\n",
       "      <td>0.482456</td>\n",
       "      <td>0.448468</td>\n",
       "      <td>0.647849</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.627297</td>\n",
       "      <td>0.422280</td>\n",
       "      <td>0.438830</td>\n",
       "      <td>0.449721</td>\n",
       "      <td>0.634211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.592317</td>\n",
       "      <td>0.424479</td>\n",
       "      <td>0.459610</td>\n",
       "      <td>0.449093</td>\n",
       "      <td>0.640957</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.561033  0.426702  0.482456  0.448468  0.647849\n",
       "recall     0.627297  0.422280  0.438830  0.449721  0.634211\n",
       "f1-score   0.592317  0.424479  0.459610  0.449093  0.640957"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm_2.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.6.3 3rd Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'C':[0.18,0.19,0.2,0.21],\n",
    "          'kernel':['linear'],\n",
    "          'gamma':['scale']}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "svm_3 = Classification('SVM',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>SVM</td>\n",
       "      <td>0.559986</td>\n",
       "      <td>0.51143</td>\n",
       "      <td>0.048556</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0        SVM        0.559986              0.51143             0.048556"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'C': 0.19, 'gamma': 'scale', 'kernel': 'linear'} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.558411</td>\n",
       "      <td>0.422164</td>\n",
       "      <td>0.482558</td>\n",
       "      <td>0.438202</td>\n",
       "      <td>0.644385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.627297</td>\n",
       "      <td>0.414508</td>\n",
       "      <td>0.441489</td>\n",
       "      <td>0.435754</td>\n",
       "      <td>0.634211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.590853</td>\n",
       "      <td>0.418301</td>\n",
       "      <td>0.461111</td>\n",
       "      <td>0.436975</td>\n",
       "      <td>0.639257</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.558411  0.422164  0.482558  0.438202  0.644385\n",
       "recall     0.627297  0.414508  0.441489  0.435754  0.634211\n",
       "f1-score   0.590853  0.418301  0.461111  0.436975  0.639257"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm_3.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.7 Guassian Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.7.1 1st Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'var_smoothing':[1e-09,1e-06,1e-03,1e-01,1e2]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "gnb_1 = Classification('Naive Bayes',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Naive Bayes</td>\n",
       "      <td>0.498671</td>\n",
       "      <td>0.484848</td>\n",
       "      <td>0.013822</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0  Naive Bayes        0.498671             0.484848             0.013822"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'var_smoothing': 0.1} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.508681</td>\n",
       "      <td>0.517857</td>\n",
       "      <td>0.550296</td>\n",
       "      <td>0.389333</td>\n",
       "      <td>0.494098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.769029</td>\n",
       "      <td>0.225389</td>\n",
       "      <td>0.247340</td>\n",
       "      <td>0.407821</td>\n",
       "      <td>0.771053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.612330</td>\n",
       "      <td>0.314079</td>\n",
       "      <td>0.341284</td>\n",
       "      <td>0.398363</td>\n",
       "      <td>0.602261</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.508681  0.517857  0.550296  0.389333  0.494098\n",
       "recall     0.769029  0.225389  0.247340  0.407821  0.771053\n",
       "f1-score   0.612330  0.314079  0.341284  0.398363  0.602261"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gnb_1.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.7.2 2nd Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'var_smoothing':[1e-02,1e-01,1]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "gnb_2 = Classification('Naive Bayes',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Naive Bayes</td>\n",
       "      <td>0.498671</td>\n",
       "      <td>0.484848</td>\n",
       "      <td>0.013822</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0  Naive Bayes        0.498671             0.484848             0.013822"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'var_smoothing': 0.1} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.508681</td>\n",
       "      <td>0.517857</td>\n",
       "      <td>0.550296</td>\n",
       "      <td>0.389333</td>\n",
       "      <td>0.494098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.769029</td>\n",
       "      <td>0.225389</td>\n",
       "      <td>0.247340</td>\n",
       "      <td>0.407821</td>\n",
       "      <td>0.771053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.612330</td>\n",
       "      <td>0.314079</td>\n",
       "      <td>0.341284</td>\n",
       "      <td>0.398363</td>\n",
       "      <td>0.602261</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.508681  0.517857  0.550296  0.389333  0.494098\n",
       "recall     0.769029  0.225389  0.247340  0.407821  0.771053\n",
       "f1-score   0.612330  0.314079  0.341284  0.398363  0.602261"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gnb_2.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.8 KNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.8.1 1st Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'n_neighbors':[5,10,50,100,200,300]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "knn_1 = Classification('KNN',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>KNN</td>\n",
       "      <td>0.50824</td>\n",
       "      <td>0.49176</td>\n",
       "      <td>0.016481</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0        KNN         0.50824              0.49176             0.016481"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'n_neighbors': 300} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.503953</td>\n",
       "      <td>0.435976</td>\n",
       "      <td>0.523810</td>\n",
       "      <td>0.410053</td>\n",
       "      <td>0.568750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.669291</td>\n",
       "      <td>0.370466</td>\n",
       "      <td>0.263298</td>\n",
       "      <td>0.432961</td>\n",
       "      <td>0.718421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.574972</td>\n",
       "      <td>0.400560</td>\n",
       "      <td>0.350442</td>\n",
       "      <td>0.421196</td>\n",
       "      <td>0.634884</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.503953  0.435976  0.523810  0.410053  0.568750\n",
       "recall     0.669291  0.370466  0.263298  0.432961  0.718421\n",
       "f1-score   0.574972  0.400560  0.350442  0.421196  0.634884"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_1.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.8.2 2nd Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'n_neighbors':[250,300,350]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "knn_2 = Classification('KNN',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>KNN</td>\n",
       "      <td>0.507</td>\n",
       "      <td>0.501329</td>\n",
       "      <td>0.005671</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0        KNN           0.507             0.501329             0.005671"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'n_neighbors': 350} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.511765</td>\n",
       "      <td>0.446108</td>\n",
       "      <td>0.546875</td>\n",
       "      <td>0.408108</td>\n",
       "      <td>0.583158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.685039</td>\n",
       "      <td>0.386010</td>\n",
       "      <td>0.279255</td>\n",
       "      <td>0.421788</td>\n",
       "      <td>0.728947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.585859</td>\n",
       "      <td>0.413889</td>\n",
       "      <td>0.369718</td>\n",
       "      <td>0.414835</td>\n",
       "      <td>0.647953</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.511765  0.446108  0.546875  0.408108  0.583158\n",
       "recall     0.685039  0.386010  0.279255  0.421788  0.728947\n",
       "f1-score   0.585859  0.413889  0.369718  0.414835  0.647953"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_2.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.8.3 3rd Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'n_neighbors':[340,345,350,355,360]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "knn_3 = Classification('KNN',x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>KNN</td>\n",
       "      <td>0.507</td>\n",
       "      <td>0.501329</td>\n",
       "      <td>0.005671</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0        KNN           0.507             0.501329             0.005671"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'n_neighbors': 350} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.511765</td>\n",
       "      <td>0.446108</td>\n",
       "      <td>0.546875</td>\n",
       "      <td>0.408108</td>\n",
       "      <td>0.583158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.685039</td>\n",
       "      <td>0.386010</td>\n",
       "      <td>0.279255</td>\n",
       "      <td>0.421788</td>\n",
       "      <td>0.728947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.585859</td>\n",
       "      <td>0.413889</td>\n",
       "      <td>0.369718</td>\n",
       "      <td>0.414835</td>\n",
       "      <td>0.647953</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.511765  0.446108  0.546875  0.408108  0.583158\n",
       "recall     0.685039  0.386010  0.279255  0.421788  0.728947\n",
       "f1-score   0.585859  0.413889  0.369718  0.414835  0.647953"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_3.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.9 Adaboost (log_reg_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.9.1 1st Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'learning_rate':[0.1,1,10]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "adaboost_1 = Ensemble('AdaBoost',log_reg_2.best_model,x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>AdaBoost</td>\n",
       "      <td>0.513734</td>\n",
       "      <td>0.503987</td>\n",
       "      <td>0.009747</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0   AdaBoost        0.513734             0.503987             0.009747"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'learning_rate': 0.1} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.655914</td>\n",
       "      <td>0.411633</td>\n",
       "      <td>0.440501</td>\n",
       "      <td>0.423398</td>\n",
       "      <td>0.687697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.480315</td>\n",
       "      <td>0.476684</td>\n",
       "      <td>0.561170</td>\n",
       "      <td>0.424581</td>\n",
       "      <td>0.573684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.554545</td>\n",
       "      <td>0.441777</td>\n",
       "      <td>0.493567</td>\n",
       "      <td>0.423989</td>\n",
       "      <td>0.625538</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.655914  0.411633  0.440501  0.423398  0.687697\n",
       "recall     0.480315  0.476684  0.561170  0.424581  0.573684\n",
       "f1-score   0.554545  0.441777  0.493567  0.423989  0.625538"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adaboost_1.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.9.2 2nd Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'learning_rate':[0.01,0.05,0.1]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "adaboost_2 = Ensemble('AdaBoost',log_reg_2.best_model,x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>AdaBoost</td>\n",
       "      <td>0.521531</td>\n",
       "      <td>0.499203</td>\n",
       "      <td>0.022329</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0   AdaBoost        0.521531             0.499203             0.022329"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'learning_rate': 0.01} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.626298</td>\n",
       "      <td>0.440443</td>\n",
       "      <td>0.486726</td>\n",
       "      <td>0.391221</td>\n",
       "      <td>0.622283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.475066</td>\n",
       "      <td>0.411917</td>\n",
       "      <td>0.438830</td>\n",
       "      <td>0.572626</td>\n",
       "      <td>0.602632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.540299</td>\n",
       "      <td>0.425703</td>\n",
       "      <td>0.461538</td>\n",
       "      <td>0.464853</td>\n",
       "      <td>0.612299</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.626298  0.440443  0.486726  0.391221  0.622283\n",
       "recall     0.475066  0.411917  0.438830  0.572626  0.602632\n",
       "f1-score   0.540299  0.425703  0.461538  0.464853  0.612299"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adaboost_2.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.10 XGBoost (log_reg_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.10.1 1st Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'eta':[0.001,0.005,0.1,0.5],\n",
    "          'min_child_weight':[1,5,10]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "xgboost_1 = Ensemble('XGBoost',log_reg_2.best_model,x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>XGBoost</td>\n",
       "      <td>0.853624</td>\n",
       "      <td>0.484848</td>\n",
       "      <td>0.368775</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0    XGBoost        0.853624             0.484848             0.368775"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'eta': 0.001, 'min_child_weight': 5} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.541353</td>\n",
       "      <td>0.387097</td>\n",
       "      <td>0.441520</td>\n",
       "      <td>0.440341</td>\n",
       "      <td>0.607792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.566929</td>\n",
       "      <td>0.404145</td>\n",
       "      <td>0.401596</td>\n",
       "      <td>0.432961</td>\n",
       "      <td>0.615789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.553846</td>\n",
       "      <td>0.395437</td>\n",
       "      <td>0.420613</td>\n",
       "      <td>0.436620</td>\n",
       "      <td>0.611765</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.541353  0.387097  0.441520  0.440341  0.607792\n",
       "recall     0.566929  0.404145  0.401596  0.432961  0.615789\n",
       "f1-score   0.553846  0.395437  0.420613  0.436620  0.611765"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgboost_1.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### 3.10.2 2nd Attempt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'eta':[0.0001,0.0005,0.001],\n",
    "          'min_child_weight':[5]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "xgboost_2 = Ensemble('XGBoost',log_reg_1.best_model,x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>XGBoost</td>\n",
       "      <td>0.853624</td>\n",
       "      <td>0.484848</td>\n",
       "      <td>0.368775</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0    XGBoost        0.853624             0.484848             0.368775"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'eta': 0.0001, 'min_child_weight': 5} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.541353</td>\n",
       "      <td>0.387097</td>\n",
       "      <td>0.441520</td>\n",
       "      <td>0.440341</td>\n",
       "      <td>0.607792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.566929</td>\n",
       "      <td>0.404145</td>\n",
       "      <td>0.401596</td>\n",
       "      <td>0.432961</td>\n",
       "      <td>0.615789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.553846</td>\n",
       "      <td>0.395437</td>\n",
       "      <td>0.420613</td>\n",
       "      <td>0.436620</td>\n",
       "      <td>0.611765</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.541353  0.387097  0.441520  0.440341  0.607792\n",
       "recall     0.566929  0.404145  0.401596  0.432961  0.615789\n",
       "f1-score   0.553846  0.395437  0.420613  0.436620  0.611765"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgboost_2.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.11 Voting (Adaboost(log_reg_2)/Logistic Regression)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "params = {'voting':['hard','soft']}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "adaboost_best = ('ada', adaboost_1.best_model)\n",
    "log_reg_best = ('lr2', log_reg_2.best_model)\n",
    "\n",
    "estimators = [adaboost_best,log_reg_best]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "voting = Ensemble('Voting',estimators,x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Voting</td>\n",
       "      <td>0.557859</td>\n",
       "      <td>0.515683</td>\n",
       "      <td>0.042176</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0     Voting        0.557859             0.515683             0.042176"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best hyperparameters are:  {'voting': 'soft'} \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.592493</td>\n",
       "      <td>0.427441</td>\n",
       "      <td>0.491379</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.650919</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.580052</td>\n",
       "      <td>0.419689</td>\n",
       "      <td>0.454787</td>\n",
       "      <td>0.469274</td>\n",
       "      <td>0.652632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.586207</td>\n",
       "      <td>0.423529</td>\n",
       "      <td>0.472376</td>\n",
       "      <td>0.443272</td>\n",
       "      <td>0.651774</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.592493  0.427441  0.491379  0.420000  0.650919\n",
       "recall     0.580052  0.419689  0.454787  0.469274  0.652632\n",
       "f1-score   0.586207  0.423529  0.472376  0.443272  0.651774"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "voting.get_scores(params,skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.11 Stacking (Adaboost(log_reg_2)/Logistic Regression)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "adaboost_best = ('ada', adaboost_2.best_model)\n",
    "log_reg_best = ('lr2', log_reg_2.best_model)\n",
    "\n",
    "estimators = [adaboost_best,log_reg_best]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "stacking = Ensemble('Stacking',estimators,x_train,x_val,y_train,y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Stacking</td>\n",
       "      <td>0.55467</td>\n",
       "      <td>0.519405</td>\n",
       "      <td>0.035265</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Model Name  Train Accuracy  Validation Accuracy  Accuracy Difference\n",
       "0   Stacking         0.55467             0.519405             0.035265"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>precision</td>\n",
       "      <td>0.594937</td>\n",
       "      <td>0.422652</td>\n",
       "      <td>0.481771</td>\n",
       "      <td>0.452830</td>\n",
       "      <td>0.639566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>recall</td>\n",
       "      <td>0.616798</td>\n",
       "      <td>0.396373</td>\n",
       "      <td>0.492021</td>\n",
       "      <td>0.469274</td>\n",
       "      <td>0.621053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>f1-score</td>\n",
       "      <td>0.605670</td>\n",
       "      <td>0.409091</td>\n",
       "      <td>0.486842</td>\n",
       "      <td>0.460905</td>\n",
       "      <td>0.630174</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  1         2         3         4         5\n",
       "precision  0.594937  0.422652  0.481771  0.452830  0.639566\n",
       "recall     0.616798  0.396373  0.492021  0.469274  0.621053\n",
       "f1-score   0.605670  0.409091  0.486842  0.460905  0.630174"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacking.get_scores({},skf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.12 All Models Compared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "For the majority of models I created, I applied hyperparameter tuning, where I started with a broad range of hyperparameters, and tuned for optimal train accuracy and validation accuracy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "all_models = pd.concat([dec_tree_1.scores_table,\n",
    "                        ran_for_3.scores_table,\n",
    "                        log_reg_2.scores_table,\n",
    "                        svm_3.scores_table,\n",
    "                        gnb_1.scores_table,\n",
    "                        knn_3.scores_table,\n",
    "                        adaboost_1.scores_table,\n",
    "                        xgboost_1.scores_table,\n",
    "                        voting.scores_table,\n",
    "                        stacking.scores_table],\n",
    "                        axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model Name</th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Validation Accuracy</th>\n",
       "      <th>Accuracy Difference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Decision Tree</td>\n",
       "      <td>0.432926</td>\n",
       "      <td>0.353535</td>\n",
       "      <td>0.079390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Random Forest</td>\n",
       "      <td>0.647351</td>\n",
       "      <td>0.457735</td>\n",
       "      <td>0.189615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>0.562112</td>\n",
       "      <td>0.509304</td>\n",
       "      <td>0.052809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>SVM</td>\n",
       "      <td>0.559986</td>\n",
       "      <td>0.511430</td>\n",
       "      <td>0.048556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Naive Bayes</td>\n",
       "      <td>0.498671</td>\n",
       "      <td>0.484848</td>\n",
       "      <td>0.013822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>KNN</td>\n",
       "      <td>0.507000</td>\n",
       "      <td>0.501329</td>\n",
       "      <td>0.005671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>AdaBoost</td>\n",
       "      <td>0.513734</td>\n",
       "      <td>0.503987</td>\n",
       "      <td>0.009747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>XGBoost</td>\n",
       "      <td>0.853624</td>\n",
       "      <td>0.484848</td>\n",
       "      <td>0.368775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Voting</td>\n",
       "      <td>0.557859</td>\n",
       "      <td>0.515683</td>\n",
       "      <td>0.042176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Stacking</td>\n",
       "      <td>0.554670</td>\n",
       "      <td>0.519405</td>\n",
       "      <td>0.035265</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Model Name  Train Accuracy  Validation Accuracy  \\\n",
       "0        Decision Tree        0.432926             0.353535   \n",
       "0        Random Forest        0.647351             0.457735   \n",
       "0  Logistic Regression        0.562112             0.509304   \n",
       "0                  SVM        0.559986             0.511430   \n",
       "0          Naive Bayes        0.498671             0.484848   \n",
       "0                  KNN        0.507000             0.501329   \n",
       "0             AdaBoost        0.513734             0.503987   \n",
       "0              XGBoost        0.853624             0.484848   \n",
       "0               Voting        0.557859             0.515683   \n",
       "0             Stacking        0.554670             0.519405   \n",
       "\n",
       "   Accuracy Difference  \n",
       "0             0.079390  \n",
       "0             0.189615  \n",
       "0             0.052809  \n",
       "0             0.048556  \n",
       "0             0.013822  \n",
       "0             0.005671  \n",
       "0             0.009747  \n",
       "0             0.368775  \n",
       "0             0.042176  \n",
       "0             0.035265  "
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "all_models.to_csv('Data/4.all_models.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "Initially, I thought the validation accuracy was low for most of the models I created, but when considering these models were attempting to classify for 5 different classes, 0.45 and greater seems very reasonable (where 0.2 = randomly guessing correctly)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "## 3.13 Saving Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "I have saved all the models using the pickle library's dump function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "for model in [dec_tree_1,ran_for_3,log_reg_2,svm_3,gnb_1,knn_3,adaboost_1,xgboost_1,voting,stacking]:\n",
    "    pickle.dump(model, open(f'Models/{model.model_type}.pkl', 'wb'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc-autonumbering": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}