model_evaluation_guide.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "handled-batch",
   "metadata": {},
   "source": [
    "# Diabetes - Model Evaluation "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sound-sight",
   "metadata": {},
   "source": [
    "In this exercise you will train your skills in evaluating a classification model. \n",
    "\n",
    "You will focus on a binary problem to identify patients with diabetes. In this task, you have to create a baseline model and compare it with a logistic regression model.\n",
    "\n",
    "To do so, you will have to choose the best metric for the task. Then, improve the model score with feature selection. \n",
    "\n",
    "Be sure that you use the hold out and cross validation method to evaluate generalization. At the end, use the learning curves to evaluate your best model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "equal-protocol",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports \n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import cross_validate\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, \\\n",
    "    confusion_matrix, classification_report\n",
    "\n",
    "from sklearn.metrics import roc_auc_score, plot_roc_curve, roc_curve\n",
    "from sklearn.metrics import average_precision_score\n",
    "from sklearn.metrics import precision_recall_curve,plot_precision_recall_curve\n",
    "\n",
    "from sklearn.model_selection import learning_curve\n",
    "\n",
    "from statsmodels.stats.outliers_influence import variance_inflation_factor as vif\n",
    "\n",
    "from sklearn.inspection import permutation_importance\n",
    "#import warnings\n",
    "#warnings.simplefilter(action=\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sexual-lexington",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Exploratory data analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "friendly-necessity",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Download the diabetes dataset from kaggle in your local\n",
    "\n",
    "https://www.kaggle.com/mathchi/diabetes-data-set/download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "matched-entity",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "diabetes_df = pd.read_csv(\"data/diabetes.csv\")\n",
    "diabetes_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cooked-newcastle",
   "metadata": {
    "hidden": true
   },
   "source": [
    "How many features do we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "muslim-zimbabwe",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "diabetes_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "authentic-litigation",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "polar-variance",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Is it a cleaned data set? how much data is missing?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "difficult-rogers",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "adolescent-tyler",
   "metadata": {
    "hidden": true
   },
   "source": [
    "How many patients are considered with diabetes ? do we have equal number of healthy and sick patients? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "generic-finnish",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "diabetes_df['???'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "arranged-dakota",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ignored-triumph",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "nuclear-runner",
   "metadata": {
    "hidden": true
   },
   "source": [
    "What can you say about outliers? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "opponent-yahoo",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "columns = list(diabetes_df.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "virgin-possession",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.boxplot(diabetes_df, vert=0,)\n",
    "plt.yticks([1, 2, 3, 4,5,6,7,8,9], columns,  rotation = 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "complete-keeping",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Bonus: find a best way to plot individually the boxplots \n",
    "# any difference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "settled-plasma",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "sufficient-thousand",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's observe the distribution of the dataset. Use a pairplot using a hue with Outcome."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "raised-decision",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "sns.pairplot(data=diabetes_df,hue=\"Outcome\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aware-walnut",
   "metadata": {
    "hidden": true
   },
   "source": [
    "is there a feature that let you identify easily the patients with diabetes?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "moving-allen",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "brilliant-pathology",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Compare your analysis with a pairgrid. what can you say? are you still ok with the features you selected previously?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "magnetic-desktop",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "g = sns.PairGrid(data = diabetes_df,corner = True)\n",
    "g.map_lower(sns.kdeplot, hue = None, levels = 4, color = \".2\")\n",
    "g.map_lower(sns.scatterplot, marker = \"+\")\n",
    "g.map_diag(sns.histplot, element = 'step', linewidth=0,kde=True)\n",
    "g.add_legend(frameon=True)\n",
    "g.legend.set_bbox_to_anchor((.61,.6))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "anonymous-galaxy",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eastern-superintendent",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "accessory-wellington",
   "metadata": {},
   "source": [
    "# Building a baseline model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "specialized-darkness",
   "metadata": {},
   "source": [
    "Let's start building a dummy model using DummyClassifier with strategy \"most_frequent\". And calculate the score. \n",
    "\n",
    "To do so, use all features in X and Outcome as target (y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "pending-beaver",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define X, y\n",
    "y = diabetes_df[\"??\"]\n",
    "X = diabetes_df.drop(\"??\", axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "exterior-neighbor",
   "metadata": {},
   "outputs": [],
   "source": [
    "# instantiate Dummy classifier\n",
    "dummy_clf = DummyClassifier(strategy=\"most_frequent\")\n",
    "\n",
    "# fit the modem\n",
    "dummy_clf.fit(X, y)\n",
    "\n",
    "# calculate the score\n",
    "dummy_clf.score(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adolescent-scottish",
   "metadata": {},
   "source": [
    "What are you evaluating? what is the performance metric used by score? Is this a good score? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "infrared-cradle",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "significant-commissioner",
   "metadata": {},
   "source": [
    "Now let's use the hold out method with the same dummy model. split the date in 70% train and 30% test. Use random_state = 1 for the split\n",
    "\n",
    "Fit the dummy model with the train and score with the test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "wrong-benjamin",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splitting the data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, \n",
    "                                                    test_size=.3, \n",
    "                                                    random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "humanitarian-chrome",
   "metadata": {},
   "outputs": [],
   "source": [
    "dummy_clf = DummyClassifier(strategy=\"most_frequent\")\n",
    "\n",
    "# fit the modem\n",
    "dummy_clf.fit(X_train, y_train)\n",
    "\n",
    "# calculate the score\n",
    "dummy_clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "diverse-effect",
   "metadata": {},
   "source": [
    "Is the score better or worst? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "inappropriate-speaking",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "addressed-working",
   "metadata": {},
   "source": [
    "Try to apply the holdout method 2 times more with different random state.\n",
    "\n",
    "Do you see a difference with the score ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "equal-lincoln",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splitting the data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, \n",
    "                                                    test_size=.3, \n",
    "                                                    random_state=??)\n",
    "\n",
    "dummy_clf = DummyClassifier(strategy=\"most_frequent\")\n",
    "\n",
    "# fit the modem\n",
    "dummy_clf.fit(X_train, y_train)\n",
    "\n",
    "# calculate the score\n",
    "dummy_clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "heard-retro",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splitting the data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, \n",
    "                                                    test_size=.3, \n",
    "                                                    random_state=??)\n",
    "\n",
    "dummy_clf = DummyClassifier(strategy=\"most_frequent\")\n",
    "\n",
    "# fit the modem\n",
    "dummy_clf.fit(X_train, y_train)\n",
    "\n",
    "# calculate the score\n",
    "dummy_clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "paperback-perfume",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "distinct-oakland",
   "metadata": {},
   "source": [
    "What is your conclusion? Can we trust the hold out method? why? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "medical-affair",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "worst-aruba",
   "metadata": {},
   "source": [
    "Let's try to use the cross validate method with dummy model. \n",
    "\n",
    "Remember, cross validate don't need to use the train test method. Is going to do it for you several times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "superior-underwear",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instanciate model\n",
    "dummy_clf = DummyClassifier(strategy=\"most_frequent\")\n",
    "\n",
    "# 5-Fold Cross validate model\n",
    "cv_results = cross_validate(dummy_clf, X, y, cv=5)\n",
    "\n",
    "# obtain the mean of scores\n",
    "cv_results['test_score'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "filled-seeking",
   "metadata": {},
   "source": [
    "How many splits did you performed? what represents the parameter cv ?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "failing-finland",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "small-satisfaction",
   "metadata": {},
   "source": [
    "try to see the information in cv_results. What can you say about? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "digital-immune",
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "boxed-roommate",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "indirect-smith",
   "metadata": {},
   "source": [
    "Can you trust more this score ? why ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "solid-exercise",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "clinical-enough",
   "metadata": {},
   "source": [
    "Congratulations! Now you have your first baseline model :) \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "biological-print",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# First Iteration: ML modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "surprising-melissa",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This time let's perform a logistic regression using all features and compare the result with the dummy model. \n",
    "\n",
    "Use the cross validation method "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "nuclear-words",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "log_model = LogisticRegression(max_iter=1000)\n",
    "\n",
    "# 5-Fold Cross validate model\n",
    "cv_results = cross_validate(log_model, X, y, cv=5)\n",
    "\n",
    "# obtain the mean of scores\n",
    "cv_results['test_score'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exclusive-atlanta",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Did you improved your baseline score? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "reserved-detail",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "southern-aggregate",
   "metadata": {
    "hidden": true
   },
   "source": [
    "What do you think is going to happend if you increase cv. Is it going to increase the performance ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "accompanied-tooth",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "fossil-assistant",
   "metadata": {
    "hidden": true
   },
   "source": [
    "let's find the optimal value for cv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "surprised-coalition",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "K = []\n",
    "total_time = []\n",
    "score = []\n",
    "\n",
    "for k in range(2,40):\n",
    "    cross_val_results = cross_validate(log_model, X, y, cv=k)\n",
    "    total_time.append(sum(cross_val_results['fit_time'])+sum(cross_val_results['score_time']))\n",
    "    K.append(k)\n",
    "    score.append(cross_val_results['test_score'].mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "finished-advice",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "plt.plot(K, total_time, label = 'Total time')\n",
    "plt.ylabel('Total time', fontsize = 14)\n",
    "plt.xlabel('K', fontsize = 14)\n",
    "plt.title('K vs Computational time', fontsize = 18, y = 1.03)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "alert-dividend",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "plt.plot(K, score, label = 'Score')\n",
    "plt.ylabel('Score', fontsize = 14)\n",
    "plt.xlabel('K', fontsize = 14)\n",
    "plt.title('K vs Score', fontsize = 18, y = 1.03)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "painful-mayor",
   "metadata": {
    "hidden": true
   },
   "source": [
    "What is your conclusion? Increasing the cv increases the score? which is the optimal value for the k-fold (cv) considering computational time and scoring?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "express-invasion",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "suitable-village",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Congratulations! You improved your score by changing the model. Let's improve this :) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "determined-partner",
   "metadata": {},
   "source": [
    "# Performance evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "proof-gibraltar",
   "metadata": {},
   "source": [
    "In this chapter, we will have to explore the different performance metrics to evaluate our classification task.\n",
    "\n",
    "To do so, run the following cells. Try to understand what is happening at each step. \n",
    "\n",
    "In this chapter we will use the hold out method and cross validation depending what we are trying to achieve. Pay attention on which method is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "induced-preliminary",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splitting the data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, \n",
    "                                                    test_size=.3, \n",
    "                                                    random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "protective-deposit",
   "metadata": {},
   "outputs": [],
   "source": [
    "log_model = LogisticRegression()\n",
    "\n",
    "log_model.fit(X_train, y_train)\n",
    "\n",
    "y_pred = log_model.predict(X_test)\n",
    "\n",
    "y_prob = log_model.predict_proba(X_test)\n",
    "\n",
    "log_model.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "illegal-airport",
   "metadata": {},
   "source": [
    "What is the utility of using predict_proba instead of predict ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "weird-panel",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "balanced-affiliate",
   "metadata": {},
   "source": [
    "## Classification metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ranking-outside",
   "metadata": {},
   "source": [
    "Explore the following metrics. What is the difference between accuracy, recall, precision and f1-score? \n",
    "\n",
    "... don't forget to see the imports of libraries to understand where this metrics come from"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bound-somewhere",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Accuracy\n",
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "geographic-mounting",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Precision\n",
    "precision_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "significant-matter",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recall\n",
    "recall_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "three-airline",
   "metadata": {},
   "outputs": [],
   "source": [
    "# F1\n",
    "f1_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "induced-devon",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "educational-share",
   "metadata": {},
   "source": [
    "## Classification report"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "danish-sewing",
   "metadata": {},
   "source": [
    "You can have access to all the metrics in one single line using classification report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "indian-being",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incorrect-partnership",
   "metadata": {},
   "source": [
    "To have access to this data, you have to transform the report into a dictionnary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "connected-version",
   "metadata": {},
   "outputs": [],
   "source": [
    "report =  classification_report(y_test,y_pred,output_dict=True)\n",
    "report"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "equal-acting",
   "metadata": {},
   "source": [
    "How can you collect the f1-score to patients having diabetes? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "reliable-creek",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "sapphire-thermal",
   "metadata": {},
   "source": [
    "Among these metrics. Which metric measures the ratio of correct predictions? What is the ratio?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "comprehensive-complex",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "dominican-gnome",
   "metadata": {},
   "source": [
    "Among these metrics.  Which metric can flag the patients with risk of diabetes? what is the value? is this good or bad? what happends if this is wrong? is it critical?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "african-lighting",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "engaged-scale",
   "metadata": {},
   "source": [
    "Among these metrics. Which metric can measure the pourcentage of patients that can develop diabetes? what happends if this is wrong? is it critical?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "brazilian-benjamin",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "reverse-printer",
   "metadata": {},
   "source": [
    "which metric can flag as many patients with diabetes while limiting false alarms? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "alleged-syndrome",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ordinary-grass",
   "metadata": {},
   "source": [
    "Which is the best metric to measure for this problem (detecting patients with diabetes)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "buried-latin",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "operational-guess",
   "metadata": {},
   "source": [
    "## Using cross validation to evaluate different metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adapted-michael",
   "metadata": {},
   "source": [
    "You can change the metric when performing cross validation. To do so, we use the scoring parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "demonstrated-frederick",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 5-Fold Cross validate model\n",
    "cv_results = cross_validate(LogisticRegression(max_iter=1000), \n",
    "                            X, y, \n",
    "                            cv=5,\n",
    "                            scoring=['accuracy','recall','precision','f1'])\n",
    "cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exotic-permit",
   "metadata": {},
   "source": [
    "Change the following code with the metric you choose before  to obtain the mean score of the different splits?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "certain-drama",
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_results['??????'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "strategic-calcium",
   "metadata": {},
   "source": [
    "There is another method called cross_val_score. this is different from the method cross_validate\n",
    "\n",
    "- cross_val_score: calculate score for each CV split, only one metric \n",
    "\n",
    "- cross_validate: calculate one or more scores and timings for each CV split\n",
    "\n",
    "Run the following cells and write your conclusions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "altered-backing",
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_scores = cross_val_score(LogisticRegression(max_iter=1000), \n",
    "                            X, y, \n",
    "                            cv=5,\n",
    "                            scoring='recall')\n",
    "cv_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "interior-malawi",
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_scores.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "connected-techno",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "undefined-highway",
   "metadata": {},
   "source": [
    "Let's use cross validation to calculate the classification report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "satisfactory-tribe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get an average prediction for the dataset\n",
    "y_pred_cv = cross_val_predict(LogisticRegression(max_iter=1000), \n",
    "                              X, y, cv=5) \n",
    "y_pred_cv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "happy-surgery",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y,y_pred_cv)) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sporting-motel",
   "metadata": {},
   "source": [
    "How different is this result compared to the classification report obtained with the hold out method? which one is better to evaluate your model ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "realistic-outreach",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "labeled-eagle",
   "metadata": {},
   "source": [
    "There is also a way to get the probabilities of predicting each class with cross val predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cardiovascular-demand",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_prob_cv = cross_val_predict(LogisticRegression(max_iter=1000), \n",
    "                              X, y, cv=5, method='predict_proba')\n",
    "y_prob_cv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "green-objective",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_prob_cv.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "under-porcelain",
   "metadata": {},
   "source": [
    "This is very important to evaluate the precision-recall tradeoff and the ROC-AUC curve"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "australian-climb",
   "metadata": {},
   "source": [
    "## confusion matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "extensive-bulgarian",
   "metadata": {},
   "source": [
    "Let's analyse another performance tool. The confusion matrix is useful to evaluate classification models. How do you read this matrix ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "spoken-growing",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Confusion Matrix\n",
    "def plot_confusion_matrix(y, y_pred):\n",
    "     acc = round(accuracy_score(y, y_pred), 2)\n",
    "     cm = confusion_matrix(y, y_pred)\n",
    "     sns.heatmap(cm, annot=True, fmt=\".0f\")\n",
    "     plt.xlabel('y_pred')\n",
    "     plt.ylabel('y')\n",
    "     plt.title('Accuracy Score: {0}'.format(acc), size=10)\n",
    "     plt.show()\n",
    "\n",
    "plot_confusion_matrix(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "worldwide-submission",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "afraid-airfare",
   "metadata": {},
   "source": [
    "What is the difference between of True Positive, True Negative, False Positive and False Negative? What are their values? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hawaiian-albuquerque",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "equal-manhattan",
   "metadata": {},
   "source": [
    "Which of these flags a missing detection? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "enclosed-messenger",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "adaptive-genesis",
   "metadata": {},
   "source": [
    "Which of these flags a false alarm? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "competitive-model",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "hollywood-genre",
   "metadata": {},
   "source": [
    "We can use the confusion matrix to calculate accuracy, recall, precision and f1-score. They can be calculated using the values of TP, FN, FP, TN. \n",
    "\n",
    "Try to calculate by hand using the cells ... You should find the same values as before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ancient-consolidation",
   "metadata": {},
   "outputs": [],
   "source": [
    "# accuracy\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bottom-alarm",
   "metadata": {},
   "outputs": [],
   "source": [
    "# recall of patients with diabetes\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "incredible-breeding",
   "metadata": {},
   "outputs": [],
   "source": [
    "# precision\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "competitive-studio",
   "metadata": {},
   "outputs": [],
   "source": [
    "# f1-score\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "secure-waterproof",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "brief-maple",
   "metadata": {},
   "source": [
    "## ROC-AUC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "running-rainbow",
   "metadata": {},
   "source": [
    "Let's see how to use the ROC AUC curve. Run the following cells and answer to the following questions.\n",
    "\n",
    "we will use the hold out method in this section"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "certified-invite",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splitting the data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, \n",
    "                                                    test_size=.3, \n",
    "                                                    random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "distant-monitor",
   "metadata": {},
   "outputs": [],
   "source": [
    "log_model = LogisticRegression()\n",
    "\n",
    "log_model.fit(X_train, y_train)\n",
    "\n",
    "y_pred = log_model.predict(X_test)\n",
    "\n",
    "y_prob = log_model.predict_proba(X_test)\n",
    "\n",
    "log_model.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "gorgeous-inflation",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ROC Curve\n",
    "plot_roc_curve(log_model, X_test, y_test)\n",
    "plt.title('ROC Curve')\n",
    "plt.plot([0, 1], [0, 1], 'r--')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "mighty-revelation",
   "metadata": {},
   "outputs": [],
   "source": [
    "# AUC to have diabetes\n",
    "roc_auc_score(y_test, y_prob[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "respiratory-withdrawal",
   "metadata": {},
   "source": [
    "As you can see, the figure is plotted using the TPR and FPR. What are these two values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "precious-colonial",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "allied-debate",
   "metadata": {},
   "source": [
    "What is the difference between the ROC and the AUC? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "small-friendship",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "native-foundation",
   "metadata": {},
   "source": [
    "How do you calculate the ROC? don't try to code, just theoretically speaking write your answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "confused-diana",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "attractive-conflict",
   "metadata": {},
   "source": [
    "How do you calculate the AUC? don't try to code, just theoretically speaking write your answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "accurate-manor",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "split-ranch",
   "metadata": {},
   "source": [
    "The ROC-AUC is used most of the time to compare models. Let's compare the dummy model and the logistic regression using the ROC curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "spiritual-studio",
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit logistic model, get the probabilities, calculate the roc curve\n",
    "log_model = LogisticRegression()\n",
    "log_model.fit(X_train, y_train)\n",
    "y_pred_log = log_model.predict_proba(X_test)[:,1]\n",
    "fpr1 , tpr1, thresholds1 = roc_curve(y_test, y_pred_log)\n",
    "AUC_log = roc_auc_score(y_test, y_pred_log)\n",
    "print(\"AUC logistic: \", AUC_log)\n",
    "\n",
    "\n",
    "# fit the dummy model, get the probabilities, calculate the roc curve\n",
    "dummy_clf = DummyClassifier(strategy=\"most_frequent\")\n",
    "dummy_clf.fit(X_train, y_train)\n",
    "y_pred_dummy = dummy_clf.predict_proba(X_test)[:,1]\n",
    "fpr2 , tpr2, thresholds2 = roc_curve(y_test, y_pred_dummy)\n",
    "AUC_dummy = roc_auc_score(y_test, y_pred_dummy)\n",
    "print(\"AUC dummy: \", AUC_dummy)\n",
    "\n",
    "\n",
    "plt.plot(fpr1, tpr1, label= \"Logistic\")\n",
    "plt.plot(fpr2, tpr2, label= \"Dummy\")\n",
    "plt.legend()\n",
    "plt.xlabel(\"FPR\")\n",
    "plt.ylabel(\"TPR\")\n",
    "plt.title('Receiver Operating Characteristic')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pending-adult",
   "metadata": {},
   "source": [
    "What does it mean to have a AUC of 0.5 ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "operating-valley",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "varying-volume",
   "metadata": {},
   "source": [
    "What does it mean to have an AUC higher to 0.5? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tribal-cholesterol",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "fewer-entrance",
   "metadata": {},
   "source": [
    "What does it mean to have an AUC of 1?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "iraqi-tennessee",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "skilled-census",
   "metadata": {},
   "source": [
    "What does it mean to have an AUC lower to 0.5?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "explicit-binary",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "sized-memorial",
   "metadata": {},
   "source": [
    "What does it mean to have an AUC of 0?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "apart-tension",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "variable-happening",
   "metadata": {},
   "source": [
    "According to you which one is better? dummy or logistic?  why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "attended-brass",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "great-conservative",
   "metadata": {},
   "source": [
    "Let's try to compare the Logistic model with another model... let's say the KNN model. We will see more about this model soon ;)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "reasonable-evans",
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit logistic model, get the probabilities, calculate the roc curve\n",
    "log_model = LogisticRegression()\n",
    "log_model.fit(X_train, y_train)\n",
    "y_pred_log = log_model.predict_proba(X_test)[:,1]\n",
    "fpr1 , tpr1, thresholds1 = roc_curve(y_test, y_pred_log)\n",
    "AUC_log = roc_auc_score(y_test, y_pred_log)\n",
    "print(\"AUC logistic: \", AUC_log)\n",
    "\n",
    "# fit the KNN model, get the probabilities, calculate the roc curve\n",
    "knn_clf = KNeighborsClassifier()\n",
    "knn_clf.fit(X_train, y_train)\n",
    "y_pred_knn = knn_clf.predict_proba(X_test)[:,1]\n",
    "fpr2 , tpr2, thresholds2 = roc_curve(y_test, y_pred_knn)\n",
    "AUC_knn = roc_auc_score(y_test, y_pred_knn)\n",
    "print(\"AUC KNN: \", AUC_knn)\n",
    "\n",
    "\n",
    "plt.plot([0,1],[0,1], 'k--')\n",
    "plt.plot(fpr1, tpr1, label= \"Logistic\")\n",
    "plt.plot(fpr2, tpr2, label= \"KNN\")\n",
    "plt.legend()\n",
    "plt.xlabel(\"FPR\")\n",
    "plt.ylabel(\"TPR\")\n",
    "plt.title('Receiver Operating Characteristic')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "intended-drain",
   "metadata": {},
   "source": [
    "Which one is the best classifier model in terms of AUC? knn or logistic? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "alternate-cookbook",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "random-hostel",
   "metadata": {},
   "source": [
    "## Precision-recall curve"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "funky-concept",
   "metadata": {},
   "source": [
    "Let's evaluate the tradeoff between the precision and the recall using the precision-recall curve. Run the following cells and answer to the following questions.\n",
    "\n",
    "we will use the cross val method in this section"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sunrise-importance",
   "metadata": {},
   "outputs": [],
   "source": [
    "# getting all values in one variables\n",
    "y_prob_cv = cross_val_predict(LogisticRegression(max_iter=1000), \n",
    "                              X, y, cv=5, method='predict_proba')\n",
    "y_prob_cv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "thorough-austin",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_prob_cv.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "further-likelihood",
   "metadata": {},
   "outputs": [],
   "source": [
    "# split proba values in two variables\n",
    "y_pred_prob_cv_0, y_pred_prob_cv_1 = cross_val_predict(LogisticRegression(max_iter=1000), \n",
    "                              X, y, cv=5, method='predict_proba').T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "crude-sphere",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_prob_cv_0.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "conditional-breast",
   "metadata": {},
   "source": [
    "To evaluate the precision/recall tradeoff, we need to get value of precision and recall depending of the threshold. To do so, we need to  use probabilities for class 1 (diabetes).\n",
    "\n",
    "Then, we can plot it manually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "honey-tobacco",
   "metadata": {},
   "outputs": [],
   "source": [
    "precision, recall, thresholds = precision_recall_curve(diabetes_df['Outcome'], \n",
    "                                                       y_pred_prob_cv_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "collective-central",
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating a dataframe with recall, precision and threshold\n",
    "p_r_df = pd.DataFrame({\"threshold\" : thresholds,\n",
    "                       \"precision\" : precision[:-1], \n",
    "                       \"recall\" : recall[:-1], \n",
    "                       })\n",
    "p_r_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "rising-bedroom",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(p_r_df['recall'],p_r_df['precision'])\n",
    "plt.ylabel('precision')\n",
    "plt.xlabel('recall')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "attractive-fluid",
   "metadata": {},
   "source": [
    "How do you interpret this curve? what is the impact of the threshold on the recall and precision?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "related-worse",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "breeding-montgomery",
   "metadata": {},
   "source": [
    "We can also calculate the curve using the plot_precision_recall_curve. Note AP stands for average precision. This method is more suited using the hold out method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "organizational-reader",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the plot_precision_recall_curve takes a model trained as parameter\n",
    "# in order to compare with what we did previously with crossval\n",
    "# we plot use the whole data set (X,y)\n",
    "# but keep in mind that we normally plot it with X_test\n",
    "disp = plot_precision_recall_curve(log_model, \n",
    "                                   X, y)\n",
    "\n",
    "disp.ax_.set_title('Precision-Recall curve')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hungry-beijing",
   "metadata": {},
   "source": [
    "AP is the area under the precision-recall curve. Read more about the Average Precision here,\n",
    "\n",
    "https://en.wikipedia.org/w/index.php?title=Information_retrieval&oldid=793358396#Average_precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "operational-spanish",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "higher-disclosure",
   "metadata": {},
   "source": [
    "Since, we are trying to diagnostic people with diabetes, it would be better to always evaluate the recall. \n",
    "\n",
    "Which is the threshold that let us predict patients with a recall of 90% guarantee ? \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "minus-trick",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_threshold = p_r_df[p_r_df['recall'] >= 0.9]['threshold'].max()\n",
    "new_threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cultural-desire",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "communist-airport",
   "metadata": {},
   "source": [
    "Let's try to plot this point in the precision recall curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "surface-northwest",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the value of recall at this threshold \n",
    "recall_thr = ???"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dense-strength",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the value of precision at this threshold \n",
    "precision_thr = ???"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "mighty-oxide",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(p_r_df['recall'],p_r_df['precision'])\n",
    "plt.plot(recall_thr, precision_thr, '-ro' )\n",
    "plt.ylabel('precision')\n",
    "plt.xlabel('recall')\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "studied-calculator",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "driven-blocking",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Building your own custom predict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stupid-announcement",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Now that we have our own threshold, we need to use it for future prediction and scoring\n",
    "\n",
    "Let's create a function custom_predict that takes as parameters the X, the model fitted and the threshold.\n",
    "\n",
    "Then, it calculates the probabilities and gives the prediction based on the threshold\n",
    "\n",
    "Remember, cross validate does not train the model (it just evaluate it). So, we can have two solutions: \n",
    "1- use the hold out method to train the model and then, use our custom prediction to evaluate probabilities\n",
    "2- use cross val to calculate probabilities and use the custom prediction "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sunset-limit",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Define custom predict function for hold out method\n",
    "def custom_predict1(model, X, custom_threshold):\n",
    "    # Get probability of each sample being classified as 0 or 1\n",
    "    probs = model.predict_proba(X) \n",
    "    # Only keep probabilities of class [1]\n",
    "    diabetes_probs = probs[:, 1]\n",
    "    # return the prediction given the threshold\n",
    "    return (diabetes_probs > custom_threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "proved-senator",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, \n",
    "                                                    test_size=.3, \n",
    "                                                    random_state=1)\n",
    "log_model = LogisticRegression()\n",
    "log_model.fit(X_train, y_train)\n",
    "\n",
    "y_pred_1 = custom_predict1(log_model, X_test, new_threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dressed-advertising",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "plot_confusion_matrix(y_test, y_pred_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "desirable-honduras",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Define custom predict function for cross val\n",
    "def custom_predict2(model_probs, X, custom_threshold):\n",
    "    # Only keep probabilities of class [1]\n",
    "    diabetes_probs = model_probs[:, 1]\n",
    "    # return the prediction given the threshold\n",
    "    return (diabetes_probs > custom_threshold).astype('int64')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "straight-spray",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "y_prob_cv = cross_val_predict(LogisticRegression(max_iter=1000), \n",
    "                              X, y, cv=5, method='predict_proba')\n",
    "\n",
    "y_pred_2 = custom_predict2(y_prob_cv, X, new_threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "signal-venue",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "plot_confusion_matrix(y, y_pred_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "handmade-washington",
   "metadata": {
    "hidden": true
   },
   "source": [
    "What is your conclusion with the changes ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "individual-scheme",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "flush-example",
   "metadata": {},
   "source": [
    "# Learning curves"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "assumed-value",
   "metadata": {},
   "source": [
    "Now, let's evaluate our model capacity to generalize. To do so, we will evaluate the overfitting and underfitting with the learning curves. \n",
    "\n",
    "We will use the module learning_curve from scikit learn. Read the documentation to understand what is happening"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "common-browse",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get train scores, train sizes, and validation scores using `learning_curve`, r2 score\n",
    "train_sizes, train_scores, test_scores = learning_curve(estimator = LogisticRegression(max_iter=1000),\n",
    "                                          X = X,\n",
    "                                          y = y,\n",
    "                                          train_sizes = [5,10,50,100,200,300,400],\n",
    "                                          cv = 5,\n",
    "                                          scoring='recall',\n",
    "                                          shuffle = True,\n",
    "                                          random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bridal-edinburgh",
   "metadata": {},
   "source": [
    "According to you, is it necessary to perform the holdout or cross val to see the learning curves? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "american-craps",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "olive-knitting",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take the mean of cross-validated train scores and validation scores\n",
    "train_scores_mean = np.mean(train_scores, axis=1)\n",
    "test_scores_mean = np.mean(test_scores, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "collaborative-switch",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the learning curves!\n",
    "plt.plot(train_sizes, train_scores_mean, label = 'Training score')\n",
    "plt.plot(train_sizes, test_scores_mean, label = 'Test score')\n",
    "plt.ylabel('Recall', fontsize = 14)\n",
    "plt.xlabel('Training set size', fontsize = 14)\n",
    "plt.title('Learning curves - log model', fontsize = 18, y = 1.03)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "special-boost",
   "metadata": {},
   "source": [
    "What can you say of the curves?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bound-underwear",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "peripheral-formula",
   "metadata": {},
   "source": [
    "How is the biais and the variance? high or low?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "unable-guarantee",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "alpine-tourism",
   "metadata": {},
   "source": [
    "Is the model overfitting? underfitting?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hourly-watershed",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "expanded-webster",
   "metadata": {},
   "source": [
    "Note, that we are not using the the threshold from the previous section. In order to do that, you should create your own algorithm to calculate the learning curves. Is not hard, but we will not do it this time :) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "leading-preserve",
   "metadata": {},
   "source": [
    "# Second Iteration: Feature Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "formal-execution",
   "metadata": {},
   "source": [
    "Let's recap what you have achieved so far:\n",
    "\n",
    "- You build a baseline model with a dummy score (65% accuracy) and using all features\n",
    "- You beat the dummy model with a logistic regression during your first ML iteration using all features. This become your new baseline model (77% accuracy)\n",
    "- You noticed that the default score (accuracy) is not the best to detect diabetes. So you decided to change the performance metric and the threshold. Your new baseline score (57% recall with a threshold at 0.5)\n",
    "- You evaluated if your model overfit/underfit and now you have an idea if the logistic regression is a good model for the problem\n",
    "\n",
    "Now, let's do a second iteration by performing feature selection. Our goal is to find the best features that increases the recall"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "appointed-drinking",
   "metadata": {},
   "source": [
    "## Univariate feature selection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "secure-science",
   "metadata": {},
   "source": [
    "Let's evaluate the Pearson correlation of the features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "offshore-acting",
   "metadata": {},
   "outputs": [],
   "source": [
    "corr = diabetes_df.corr() # Pearson Correlation\n",
    "\n",
    "# Heatmap\n",
    "sns.heatmap(corr, \n",
    "        xticklabels=corr.columns,\n",
    "        yticklabels=corr.columns,\n",
    "        cmap= \"YlGnBu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "endless-trial",
   "metadata": {},
   "outputs": [],
   "source": [
    "# another way to visualize it\n",
    "diabetes_df.corr().style.background_gradient(cmap='coolwarm')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "technical-guidance",
   "metadata": {},
   "source": [
    "According to this graph, which are the features correlated with the target Outcome?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "exposed-truth",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "center-backing",
   "metadata": {},
   "source": [
    "Let's visualize the pairs of correlation numerically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "committed-certification",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Unstack correlation matrix \n",
    "corr_df = corr.unstack().reset_index() \n",
    "corr_df\n",
    "\n",
    "# rename columns\n",
    "corr_df.columns = ['feature_1','feature_2', 'correlation'] \n",
    "\n",
    "# sort by correlation\n",
    "corr_df.sort_values(by=\"correlation\",ascending=False, inplace=True) \n",
    "\n",
    "# Remove self correlation\n",
    "corr_df = corr_df[corr_df['feature_1'] != corr_df['feature_2']] \n",
    "corr_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chief-netherlands",
   "metadata": {},
   "source": [
    "Now let's evaluate which are the features correlated with the target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "contrary-anchor",
   "metadata": {},
   "outputs": [],
   "source": [
    "diabetes_corr_df = corr_df[corr_df['feature_1'] == 'Outcome']\n",
    "diabetes_corr_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "convinced-scotland",
   "metadata": {},
   "source": [
    "Which is the feature that explains the most the risk of diabetes ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "focal-manner",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "simplified-binding",
   "metadata": {},
   "source": [
    "Try different features to train logistic regression with cross validation by adding new features (starting with the one with highest correlation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "romance-jewelry",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_eval = X[['???']]\n",
    "\n",
    "cv_results = cross_validate(LogisticRegression(max_iter=1000), \n",
    "                            X_eval, y, \n",
    "                            cv=5,\n",
    "                            scoring=['recall'])\n",
    "cv_results['test_recall'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "streaming-header",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "silent-concentrate",
   "metadata": {},
   "source": [
    "Which is the best combination you found ? Did you beat the baseline score? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "increasing-input",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "mounted-margin",
   "metadata": {},
   "source": [
    "## Collinearity investigation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "noticed-pollution",
   "metadata": {},
   "source": [
    "Let's evaluate which features have a risk to show data leakage. \n",
    "\n",
    "First, let's see which features present a correlation of 0.9 or higher"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "decimal-turtle",
   "metadata": {},
   "outputs": [],
   "source": [
    "correlated_features = len(corr_df[(corr_df['correlation'] >= 0.9) | (corr_df['correlation'] <= -0.9)])\n",
    "\n",
    "correlated_features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stainless-territory",
   "metadata": {},
   "source": [
    "How many features seems to have high correlation?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "verbal-pixel",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "controlled-certification",
   "metadata": {},
   "source": [
    "Now, we will evaluate the VIF factor to the whole dataframe. To do so, we will use the module variance_inflation_factor from stats_models \n",
    "\n",
    "You can read this article to understand more about it : https://en.wikipedia.org/wiki/Variance_inflation_factor\n",
    "\n",
    "${\\displaystyle \\mathrm {VIF} _{i}={\\frac {1}{1-R_{i}^{2}}}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hazardous-winning",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute VIF factor for feature index 0\n",
    "vif(diabetes_df.values, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "public-butler",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute the VIF factor to all the features \n",
    "# store results in a dataframe\n",
    "vif_df = pd.DataFrame()\n",
    "vif_df[\"vif_index\"] = [vif(diabetes_df.values, i) for i in range(diabetes_df.shape[1])]\n",
    "vif_df[\"features\"] = diabetes_df.columns\n",
    "vif_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afraid-elizabeth",
   "metadata": {},
   "source": [
    "What can you say about the results?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "beginning-traffic",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "numeric-billion",
   "metadata": {},
   "source": [
    "there is a big possibility that there is some collinearity between Glucose, BloodPressure and BMI, age. So, we should keep attention to these pairs of features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dedicated-advocate",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "extreme-basement",
   "metadata": {},
   "source": [
    "Let's verify your hypothesis by removing Outcome. The VIF will give you the collinearity between features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "physical-internet",
   "metadata": {},
   "outputs": [],
   "source": [
    "vif_df = pd.DataFrame()\n",
    "vif_df[\"vif_index\"] = [vif(X.values, i) for i in range(X.shape[1])]\n",
    "vif_df[\"features\"] = X.columns\n",
    "vif_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "little-pastor",
   "metadata": {},
   "source": [
    "What are your conclusions?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "arctic-cause",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "royal-pasta",
   "metadata": {},
   "source": [
    "Try new cross_validate training to observe which feature has the most of information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "excellent-harris",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "beautiful-hampshire",
   "metadata": {},
   "source": [
    "What is your conclusion?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "humanitarian-gasoline",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "increasing-roulette",
   "metadata": {},
   "source": [
    "## Feature Permutation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prospective-practitioner",
   "metadata": {},
   "source": [
    "To finish this exercise, let's perform feature permutation technique. To do so, you will use the hold out method to train your model. then use the test set to use the module permutation_importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "subtle-cleanup",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, \n",
    "                                                    test_size=.3, \n",
    "                                                    random_state=1)\n",
    "log_model = LogisticRegression()\n",
    "log_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "organizational-curve",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform Permutation\n",
    "permutation_score = permutation_importance(log_model, \n",
    "                                           X_test, y_test,\n",
    "                                           scoring='recall',\n",
    "                                           random_state=3,\n",
    "                                           n_repeats=100) \n",
    "\n",
    "# Unstack results\n",
    "importance_df = pd.DataFrame(np.vstack((X.columns,\n",
    "                                        permutation_score.importances_mean)).T) \n",
    "\n",
    "importance_df.columns=['feature','feature importance']\n",
    "\n",
    "# Order by importance\n",
    "importance_df.sort_values(by=\"feature importance\", ascending = False) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "specified-september",
   "metadata": {},
   "source": [
    "What are you conclusion about the results? Does it confirms your previous choice ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ultimate-germany",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "demonstrated-marking",
   "metadata": {},
   "source": [
    "Try to explain with your own words how feature importance is calculated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "choice-market",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "technical-tucson",
   "metadata": {},
   "source": [
    "We can also visualize the feature importance by using the boxplot. However, it is important to put more attention to the median"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ongoing-consortium",
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_idx = permutation_score.importances_mean.argsort()  \n",
    "\n",
    "fig, ax = plt.subplots() \n",
    "ax.boxplot(permutation_score.importances[sorted_idx].T, vert=False, labels=X_test.columns[sorted_idx]) \n",
    "ax.set_title(\"Permutation Importances (test set)\") \n",
    "fig.tight_layout() \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "italian-reduction",
   "metadata": {},
   "source": [
    "How do you interpret this figure?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "selected-expansion",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "express-service",
   "metadata": {},
   "source": [
    "If you have to select only 3 features which would be your best feature selection? What is the score? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "female-nomination",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "handed-joint",
   "metadata": {},
   "source": [
    "It is better or worst compared to the previous one? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "academic-grammar",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "qualified-brave",
   "metadata": {},
   "source": [
    "🏁 Congratulations !! You have improved your model :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}