udlbook/Notebooks/Chap02/2_1_Supervised_Learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap02/2_1_Supervised_Learning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Notebook 2.1 Supervised Learning\n",
        "\n",
        "The purpose of this notebook is to explore the linear regression model discussed in Chapter 2 of the book.\n",
        "\n",
        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and write code to complete the functions. There are also questions interspersed in the text.\n",
        "\n",
        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
      ],
      "metadata": {
        "id": "sfB2oX2RNvuF"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uoYl2Gn3Nr52"
      },
      "outputs": [],
      "source": [
        "# Math library\n",
        "import numpy as np\n",
        "# Plotting library\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create some input / output data\n",
        "x = np.array([0.03, 0.19, 0.34, 0.46, 0.78, 0.81, 1.08, 1.18, 1.39, 1.60, 1.65, 1.90])\n",
        "y = np.array([0.67, 0.85, 1.05, 1.0, 1.40, 1.5, 1.3, 1.54, 1.55, 1.68, 1.73, 1.6 ])\n",
        "\n",
        "print(x)\n",
        "print(y)"
      ],
      "metadata": {
        "id": "MUbTD4znORtd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Define 1D linear regression model\n",
        "def f(x, phi0, phi1):\n",
        "  # TODO :  Replace this line with the linear regression model (eq 2.4)\n",
        "  y = x\n",
        "\n",
        "  return y"
      ],
      "metadata": {
        "id": "lw2dCRHwSW9a"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Function to help plot the data\n",
        "def plot(x, y, phi0, phi1):\n",
        "    fig,ax = plt.subplots()\n",
        "    ax.scatter(x,y)\n",
        "    plt.xlim([0,2.0])\n",
        "    plt.ylim([0,2.0])\n",
        "    ax.set_xlabel('Input, $x$')\n",
        "    ax.set_ylabel('Output, $y$')\n",
        "    # Draw line\n",
        "    x_line = np.arange(0,2,0.01)\n",
        "    y_line = f(x_line, phi0, phi1)\n",
        "    plt.plot(x_line, y_line,'b-',lw=2)\n",
        "\n",
        "    plt.show()"
      ],
      "metadata": {
        "id": "VT4F3xxSOt8C"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Set the intercept and slope as in figure 2.2b\n",
        "phi0 = 0.4 ; phi1 = 0.2\n",
        "# Plot the data and the model\n",
        "plot(x,y,phi0,phi1)"
      ],
      "metadata": {
        "id": "AkdZdmhHWuVR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Function to calculate the loss\n",
        "def compute_loss(x,y,phi0,phi1):\n",
        "\n",
        "  # TODO Replace this line with the loss calculation (equation 2.5)\n",
        "  loss = 0\n",
        "\n",
        "\n",
        "  return loss"
      ],
      "metadata": {
        "id": "1-GW218wX44b"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Compute the loss for our current model\n",
        "loss = compute_loss(x,y,phi0,phi1)\n",
        "print(f'Your Loss = {loss:3.2f}, Ground truth =7.07')"
      ],
      "metadata": {
        "id": "Hgw7_GzBZ8tX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Set the intercept and slope as in figure 2.2c\n",
        "phi0 = 1.60 ; phi1 =-0.8\n",
        "# Plot the data and the model\n",
        "plot(x,y,phi0,phi1)\n",
        "loss = compute_loss(x,y,phi0,phi1)\n",
        "print(f'Your Loss = {loss:3.2f}, Ground truth =10.28')"
      ],
      "metadata": {
        "id": "_vZS28-FahGP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO -- Change the parameters manually to fit the model\n",
        "# First fix phi1 and try changing phi0 until you can't make the loss go down any more\n",
        "# Then fix phi0 and try changing phi1 until you can't make the loss go down any more\n",
        "# Repeat this process until you find a set of parameters that fit the model as in figure 2.2d\n",
        "# You can either do this by hand, or if you want to get fancy, write code to descent automatically in this way\n",
        "# Start at these values:\n",
        "phi0 = 1.60 ; phi1 =-0.8\n",
        "\n",
        "plot(x,y,phi0,phi1)\n",
        "print(f'Your Loss = {compute_loss(x,y,phi0,phi1):3.2f}')"
      ],
      "metadata": {
        "id": "VzpnzdW5d9vj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Visualizing the loss function\n",
        "\n",
        "The above process is equivalent to descending coordinate wise on the loss function<br>\n",
        "\n",
        "Now let's plot that function"
      ],
      "metadata": {
        "id": "MNC4qEZognEe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Make a 2D grid of possible phi0 and phi1 values\n",
        "phi0_mesh, phi1_mesh = np.meshgrid(np.arange(0.0,2.0,0.02), np.arange(-1.0,1.0,0.02))\n",
        "\n",
        "# Make a 2D array for the losses\n",
        "all_losses = np.zeros_like(phi1_mesh)\n",
        "# Run through each 2D combination of phi0, phi1 and compute loss\n",
        "for indices,temp in np.ndenumerate(phi1_mesh):\n",
        "    all_losses[indices] = compute_loss(x,y, phi0_mesh[indices], phi1_mesh[indices])\n"
      ],
      "metadata": {
        "id": "ATrU8sqqg2hJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Plot the loss function as a heatmap\n",
        "fig = plt.figure()\n",
        "ax = plt.axes()\n",
        "fig.set_size_inches(7,7)\n",
        "levels = 256\n",
        "ax.contourf(phi0_mesh, phi1_mesh, all_losses ,levels)\n",
        "levels = 40\n",
        "ax.contour(phi0_mesh, phi1_mesh, all_losses ,levels, colors=['#80808080'])\n",
        "ax.set_ylim([1,-1])\n",
        "ax.set_xlabel(r'Intercept, $\\phi_0$')\n",
        "ax.set_ylabel(r'Slope, $\\phi_1$')\n",
        "\n",
        "# Plot the position of your best fitting line on the loss function\n",
        "# It should be close to the minimum\n",
        "ax.plot(phi0,phi1,'ro')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "6OXAjx5xfQkl"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}