2018-05-04-ipywidgets-for-learning-logistic-sigmoid-and-bayes-classifiers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `ipywidgets.interact` for logistic sigmoid and Bayes classifiers\n",
    "\n",
    "This notebook shows how I used `ipywidgets.interact` to better understand equations in machine learning, including the logistic sigmoid and Bayes classifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from ipywidgets import interact"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Sigmoid\n",
    "\n",
    "The sigmoid function shows up a lot in machine learning. For one input dimension, one form I frequently see is\n",
    "\n",
    "$$\\sigma(x) = \\frac{1}{1 + \\exp(-v^{\\top}x + b)}$$\n",
    "\n",
    "This code allows one to vary $v$ and $b$ for two sigmoid functions. The code produces something like:\n",
    "\n",
    "![sigmoid example](images/2018-05-04-sigmoid.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a97904098819464da2d62ef02565c67c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>interactive</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another notebook frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "interactive(children=(FloatSlider(value=-1.0, description='vv1', max=10.0, min=-12.0, step=0.25), IntSlider(value=0, description='bb1', max=10, min=-10), IntSlider(value=1, description='vv2', max=12, min=-10), IntSlider(value=0, description='bb2', max=10, min=-10), Output()), _dom_classes=('widget-interact',))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.plot_logistic_sigmoid>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def logistic_sigmoid(xx, vv, b):\n",
    "    return 1 / (1 + np.exp(-(np.dot(vv, xx) + b)))\n",
    "\n",
    "plt.clf()\n",
    "grid_size = 0.01\n",
    "x_grid = np.arange(-5, 5, grid_size)\n",
    "\n",
    "\n",
    "def plot_logistic_sigmoid(vv1, bb1, vv2, bb2):\n",
    "    plt.plot(x_grid, logistic_sigmoid(x_grid, vv=vv1, b=bb1), '-b')\n",
    "    plt.plot(x_grid, logistic_sigmoid(x_grid, vv=vv2, b=bb2), '-r')\n",
    "    plt.axis([-5, 5, -0.5, 1.5])\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "interact(\n",
    "    plot_logistic_sigmoid, \n",
    "    vv1=(-12, 10, .25), \n",
    "    bb1=(-10, 10), \n",
    "    vv2=(-10, 12), \n",
    "    bb2=(-10, 10)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bayes Classifiers\n",
    "\n",
    "(These notes are from the machine learning course I took last semester. The relevant notes are here: [MLPR](http://www.inf.ed.ac.uk/teaching/courses/mlpr/2017/notes/w3a_intro_classification.html))\n",
    "\n",
    "Bayes classifiers are a method of classifying used in machine learning. \n",
    "\n",
    "For each class $k$, the Bayes classifier has a prior, $P(y = k)$, and a distribution over features, $P(\\textbf x \\mid y = k)$, such as a Gaussian $\\mathcal N (\\textbf x; \\hat\\mu_k, \\Sigma_k)$. \n",
    "The probability of the decision is $P(y=k \\mid \\textbf x) = \\frac{\\mathcal N(\\textbf x; \\hat \\mu_k, \\Sigma_k)}{\\sum_k'\\mathcal N(\\textbf x; \\hat \\mu_k', \\Sigma_k')}$. ([Naive Bayes](https://en.wikipedia.org/wiki/Naive_Bayes_classifier) is an example of a classifier that assumes features are independent, i.e. $\\Sigma_k$ is diagonal.)\n",
    "\n",
    "I wanted to see what shapes the decision surface could take on. \n",
    "If the covariances of the classes are equal, the decision boundary is linear.\n",
    "Even with one input dimension, the decision boundaries do interesting things. For example, if one class's variance is larger than the other, then there are two decision boundaries! And the sigmoid function makes another appearance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One input dimension\n",
    "\n",
    "This code allows you to adjust the parameters of a Bayes classifier with one input dimension and two classes. It produces something like\n",
    "\n",
    "![Bayes classifer with one dimension](images/2018-05-04-bayes-1.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "668a7957ab6c49c98436e825195f47ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>interactive</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another notebook frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "interactive(children=(IntSlider(value=5, description='zoom', max=10, min=1), FloatSlider(value=0.505, description='pi_1', max=1.0, min=0.01), IntSlider(value=3, description='mu_1', max=12, min=-5), IntSlider(value=-4, description='mu_2', max=5, min=-12), FloatSlider(value=1.0, description='var_1', max=2.0), FloatSlider(value=1.0, description='var_2', max=2.0), Output()), _dom_classes=('widget-interact',))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.draw>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from ipywidgets import interact\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import multivariate_normal\n",
    "\n",
    "plt.clf()\n",
    "grid_size = 0.1\n",
    "x_grid = np.arange(-5, 5, grid_size)\n",
    "\n",
    "def draw(zoom, pi_1, mu_1, mu_2, var_1, var_2):\n",
    "\n",
    "    Z1 = multivariate_normal(mu_1, var_1)\n",
    "    Z2 = multivariate_normal(mu_2, var_2)\n",
    "    \n",
    "    # Decision boundary!\n",
    "    Z_num = pi_1 * Z1.pdf(x_grid)\n",
    "    z_den = ((1 - pi_1) * Z2.pdf(x_grid) + pi_1 * Z1.pdf(x_grid)) + 1e-300\n",
    "    Z = Z_num / z_den\n",
    "\n",
    "    plt.figure()\n",
    "    plt.plot(x_grid, Z, c='k')\n",
    "    plt.plot(x_grid, Z1.pdf(x_grid))\n",
    "    plt.plot(x_grid, Z2.pdf(x_grid))\n",
    "\n",
    "    # set the axis based on the zoom\n",
    "    plt.axis([-5, 5, -0.05, 1.05])\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "interact(\n",
    "    draw, \n",
    "    zoom=(1, 10),  # zoom into image\n",
    "    pi_1=(0.01, 1),  # prior on first class\n",
    "    mu_1=(-5, 12),  # mean of first class\n",
    "    mu_2=(-12, 5),   # mean of second class\n",
    "    var_1=(0, 2.),  # variance of first class\n",
    "    var_2=(0, 2.),  # variance of second class\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Two input dimensions\n",
    "\n",
    "This code produces something like\n",
    "\n",
    "![Bayes classifier with 2 input dimensions](images/2018-05-04-bayes-2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48d361e08db541c888ac92f36daf3c84",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>interactive</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another notebook frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "interactive(children=(IntSlider(value=5, description='zoom', max=10, min=1), FloatSlider(value=0.505, description='pi_1', max=1.0, min=0.01), IntSlider(value=3, description='m_1', max=12, min=-5), IntSlider(value=-4, description='m_2', max=5, min=-12), IntSlider(value=3, description='stretch', max=5, min=1), IntSlider(value=3, description='stretch_2', max=5, min=1), FloatSlider(value=0.5, description='cov_1', max=1.0), FloatSlider(value=0.5, description='cov_2', max=1.0), Output()), _dom_classes=('widget-interact',))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.draw>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from ipywidgets import interact\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import multivariate_normal\n",
    "\n",
    "\n",
    "delta = 0.2\n",
    "x = np.arange(-60, 60, delta)\n",
    "y = np.arange(-40, 40, delta)\n",
    "X, Y = np.meshgrid(x, y)\n",
    "pos = np.empty(X.shape + (2,))\n",
    "pos[:, :, 0] = X\n",
    "pos[:, :, 1] = Y\n",
    "\n",
    "\n",
    "def draw(zoom, pi_1, m_1, m_2, stretch, stretch_2, cov_1, cov_2):\n",
    "    Sigma_1 = np.array([[stretch, cov_1], [cov_1, 1.0]])\n",
    "    mu_1 = np.array([m_1, 0.0])\n",
    "\n",
    "    Sigma_2 = np.array([[stretch_2, cov_2], [cov_2, stretch_2]])\n",
    "    mu_2 = np.array([m_2, 0.0])\n",
    "\n",
    "    Z1 = multivariate_normal(mu_1, Sigma_1)\n",
    "    Z2 = multivariate_normal(mu_2, Sigma_2)\n",
    "    \n",
    "    # compute the decision boundary!\n",
    "    Z_num = pi_1 * Z1.pdf(pos)\n",
    "    z_den = ((1 - pi_1) * Z2.pdf(pos) + pi_1 * Z1.pdf(pos)) + 1e-300  # add an offset to avoid divide by 0\n",
    "    Z = Z_num / z_den - 0.5\n",
    "\n",
    "    plt.figure()\n",
    "    plt.contour(X, Y, Z, 10, colors='k')    \n",
    "    plt.contour(X, Y, pi_1 * Z1.pdf(pos), 5)\n",
    "    plt.contour(X, Y, (1 - pi_1) * Z2.pdf(pos), 5)\n",
    "\n",
    "    # set the axis based on the zoom\n",
    "    plt.axis([int(i / (zoom * 4)) for i in [-150, 150, -100, 100]])\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "interact(\n",
    "    draw, \n",
    "    zoom=(1, 10), \n",
    "    pi_1=(0.01, 1),  # prior on the first class\n",
    "    m_1=(-5, 12),  # x dimension of the first class's mu\n",
    "    m_2=(-12, 5),  # x dimension of the second class's mu\n",
    "    stretch=(1, 5),  # x dimension variance\n",
    "    stretch_2=(1, 5),   # x and y dimension variance\n",
    "    cov_1=(0, 1.0),  # covariance between two dimension's for first class\n",
    "    cov_2=(0, 1.0),  # covariance between two dimension's for second class\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}