{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "provenance": [], "authorship_tag": "ABX9TyMZw5ysoZN18RhdkayKdi07", "include_colab_link": true }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "view-in-github", "colab_type": "text" }, "source": [ "\"Open" ] }, { "cell_type": "markdown", "source": [ "" ], "metadata": { "id": "lTJK0o1TUMnq" } }, { "cell_type": "markdown", "source": [ "# Loss functions\n", "\n", "The purpose of this Python notebook is to compute the loss and subsequently plot the loss function for the 1D linear regression model with a least squares loss.\n", "\n", "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and write code to complete the functions.\n", "\n", "You can save a local copy of this notebook in your Google account and work through it in Colab (recommended) or you can download the notebook and run it locally using Jupyter notebook or similar. If you are using CoLab, we recommend that *turn off* AI autocomplete (under cog icon in top-right corner), which will give you the answers and defeat the purpose of the exercise.\n", "\n", "A fully working version of this notebook with the complete answers can be found [here](https://colab.research.google.com/github/udlbook/udlbook/blob/main/Trees/LinearRegression_LossFunction_Answers.ipynb#scrollTo=niGXQTgJjclv).\n", "\n", "Contact me at iclimbtreesmail@gmail.com if you find any mistakes or have any suggestions." ], "metadata": { "id": "uORlKyPv02ge" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "bbF6SE_F0tU8" }, "outputs": [], "source": [ "# Math library\n", "import numpy as np\n", "# Plotting library\n", "import matplotlib.pyplot as plt\n", "from matplotlib import cm\n", "from matplotlib.colors import ListedColormap" ] }, { "cell_type": "code", "source": [ "# Create the same input / output data as used in the unit\n", "x = np.array([0.03, 0.19, 0.34, 0.46, 0.78, 0.81, 1.08, 1.18, 1.39, 1.60, 1.65, 1.90])\n", "y = np.array([0.67, 0.85, 1.05, 1.0, 1.40, 1.5, 1.3, 1.54, 1.55, 1.68, 1.73, 1.6 ])" ], "metadata": { "id": "UQ1qYLOeYf61" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "# Define the model\n", "\n", "The linear regression model is defined as:\n", "\n", "$$\\textrm{f}[x,\\boldsymbol\\phi] = \\phi_0+\\phi_1 x$$\n", "\n", "where $\\phi_0$ is the y-intercept and $\\phi_1$ is the slope." ], "metadata": { "id": "cO0m5OaCd17h" } }, { "cell_type": "code", "source": [ "# Define 1D linear regression model\n", "def f(x, phi0, phi1):\n", " # TODO -- define the linear regression model\n", " # REPLACE THIS CODE\n", " y = x\n", "\n", " return y" ], "metadata": { "id": "aD2g9L9hd5Bd" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "# Function to help plot the data\n", "def plot(x, y, f, phi0=None, phi1=None):\n", " fig,ax = plt.subplots()\n", " ax.scatter(x,y)\n", " plt.xlim([0,2.0])\n", " plt.ylim([0,2.0])\n", " ax.set_xlabel('Input, $x$')\n", " ax.set_ylabel('Output, $y$')\n", " ax.set_aspect('equal')\n", " # Draw line if parameters passed\n", " x_line = np.arange(0,2,0.01)\n", " if(phi0 is not None and phi1 is not None):\n", " y_line = f(x_line, phi0, phi1)\n", " plt.plot(x_line, y_line,'r-',lw=2)\n", " plt.show()" ], "metadata": { "id": "785wp16FYjt5" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "plot(x,y, f, phi0 = 1.5, phi1=-0.2)" ], "metadata": { "id": "sONMDaj_eJ-6" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "## Compute the least squares loss\n", "\n", "The least squares loss is defined as the sum of the squared deviations of the model output $\\textrm{f}[x_i,\\boldsymbol\\phi]$ and the true output target $y_i$:\n", "\n", " \\begin{align} L[\\boldsymbol\\phi] & = \\sum_{i=1}^{I} \\bigl(\\textrm{f}[x_{i}, \\boldsymbol\\phi]-y_{i}\\bigr)^{2} \\\\ &= \\sum_{i=1}^{I} \\bigl(\\phi_{0}+\\phi_{1}x_i-y_{i}\\bigr)^{2} \\tag{1.2}\\end{align}" ], "metadata": { "id": "xoe2kvdreYLh" } }, { "cell_type": "code", "source": [ "# Function to calculate the loss\n", "def compute_loss(x,y,phi0,phi1):\n", "\n", " # TODO Replace this line with the loss calculation\n", " loss = 0\n", "\n", "\n", " return loss" ], "metadata": { "id": "PEenxRo3ePCL" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "# Let's check that this is correct\n", "loss = compute_loss(x,y, phi0 = 0.4 , phi1 = 0.2)\n", "print(f'Your Loss = {loss:3.2f}, Ground truth =7.07')" ], "metadata": { "id": "EbTeNI3fhgGw" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "## Draw the loss function" ], "metadata": { "id": "9A32qcIAiU5x" } }, { "cell_type": "code", "source": [ "# Helper code to do the drawing\n", "def draw_loss_function(compute_loss, x_in, y_in):\n", " # Define pretty colormap\n", " my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n", " my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n", " r = np.floor(my_colormap_vals_dec/(256*256))\n", " g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n", " b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n", " my_colormap = ListedColormap(np.vstack((r,g,b)).transpose()/255.0)\n", "\n", " # Make grid of intercept/slope values to plot\n", " intercepts_mesh, slopes_mesh = np.meshgrid(np.arange(0.0,2.0,0.02), np.arange(-1.0,1.0,0.002))\n", " loss_mesh = np.zeros_like(slopes_mesh)\n", "\n", " # Compute loss for every set of parameters\n", " for idslope, slope in np.ndenumerate(slopes_mesh):\n", " loss_mesh[idslope] = compute_loss(x_in, y_in, intercepts_mesh[idslope], slope)\n", "\n", " fig,ax = plt.subplots()\n", " fig.set_size_inches(6,6)\n", " ax.contourf(intercepts_mesh,slopes_mesh,loss_mesh,256,cmap=my_colormap)\n", " ax.contour(intercepts_mesh,slopes_mesh,loss_mesh,40,colors=['#80808080'])\n", "\n", " ax.set_ylim([1,-1])\n", " ax.set_xlabel('Intercept, $\\phi_0$')\n", " ax.set_ylabel('Slope, $\\phi_1$')\n", " ax.set_aspect('equal')\n", " plt.show()" ], "metadata": { "id": "zlUXJ-5zhlb9" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "draw_loss_function(compute_loss, x, y )" ], "metadata": { "id": "sUqE8Ll6iULa" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "TODO -- experiment by changing the input x and y values (keeping them in the range $x\\in[0,2], y\\in[0,2]$)\n", "Does the loss function always have a single minimum?\n", "Think about why this might be." ], "metadata": { "id": "rpJ3ZnCDjPp0" } }, { "cell_type": "code", "source": [], "metadata": { "id": "niGXQTgJjclv" }, "execution_count": null, "outputs": [] } ] }