Created using Colaboratory

CM20315_Intro.ipynb

@@ -0,0 +1,298 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyMTtWwAtwIZJoIsfGehxMMC",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/CM20315_Intro.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "\n",
+        "# **CM20315 Background Mathematics**\n",
+        "\n",
+        "The purpose of this Python notebook is to make sure you can use CoLab and to familiarize yourself with some of the background mathematical concepts that you are going to need to understand deep learning. <br><br> It's not meant to be difficult and it may be that you know some or all of this information already.<br><br> Maths is *NOT* a spectator sport.  You won't learn it by just listening to lectures or reading books.  It really helps to interact with it and explore yourself. <br><br> Work through the cells below, running each cell in turn.  In various places you will see the words **\"TO DO\"**. Follow the instructions at these places.  If you can't figure out what is going on or how to complete the section, feel free to ask your fellow students or the TAs in the practical session or via Moodle.  This does not count toward your final marks, but complete=ing it will help you do well in your coursework and exam and reduce your stress levels later on.\n"
+      ],
+      "metadata": {
+        "id": "s5zzKSOusPOB"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "aUAjBbqzivMY"
+      },
+      "outputs": [],
+      "source": [
+        "# Imports math library\n",
+        "import numpy as np\n",
+        "# Imports plotting library\n",
+        "import matplotlib.pyplot as plt"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**Linear functions**<br> In this course, we will be using the term *linear equation* to mean a weighted sum of inputs plus an offset. If there is just one input $x$, then this is a straight line:\n",
+        "\n",
+        "\\begin{equation}y=\\beta+\\omega x,\\end{equation} <br>\n",
+        "\n",
+        "where $\\beta$ is the y-intercept of the linear and $\\omega$ is the slope of the line. When there are two inputs $x_{1}$ and $x_{2}$, then this becomes:\n",
+        "\n",
+        "\\begin{equation}y=\\beta+\\omega_1 x_1 + \\omega_2 x_2.\\end{equation} <br><br>\n",
+        "\n",
+        "Any other functions are by definition **non-linear**.\n",
+        "\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "WV2Dl6owme2d"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Define a linear function with just one input, x\n",
+        "def linear_function_1D(x,beta,omega):\n",
+        "  # TODO -- replace the code lin below with formula for 1D linear equation\n",
+        "  y = x\n",
+        "  return y"
+      ],
+      "metadata": {
+        "id": "WeFK4AvTotd8"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Plot the 1D linear function \n",
+        "\n",
+        "# Define an array of x values from 0 to 10 with increments of 0.1\n",
+        "# https://numpy.org/doc/stable/reference/generated/numpy.arange.html\n",
+        "x = np.arange(0.0,10.0, 0.01) \n",
+        "# Compute y using the function you filled in above\n",
+        "beta = 0.0; omega = 1.0\n",
+        "y = linear_function_1D(x,beta,omega)\n",
+        "\n",
+        "# Plot this function\n",
+        "fig, ax = plt.subplots()\n",
+        "ax.plot(x,y,'r-')\n",
+        "ax.set_ylim([0,10]);ax.set_xlim([0,10])\n",
+        "ax.set_xlabel('x'); ax.set_ylabel('y')\n",
+        "plt.show\n",
+        "\n",
+        "# TODO -- experiment with changing the values of beta and omega\n",
+        "# to understand what they do.  Try to make a line\n",
+        "# that crosses the y-axis at y=10 and the x-axis at x=5"
+      ],
+      "metadata": {
+        "id": "eimhJ8_jpmEp"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's investigate a 2D linear function"
+      ],
+      "metadata": {
+        "id": "AedfvD9dxShZ"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Code to draw 2D function -- read it so you know what is going on, but you don't have to change it\n",
+        "def draw_2D_function(x1_mesh, x2_mesh, y):\n",
+        "    fig, ax = plt.subplots()\n",
+        "    fig.set_size_inches(7,7)\n",
+        "    ax.contourf(x1_mesh, x2_mesh, y, levels=256 ,cmap = 'hot', vmin=-10,vmax=10.0)\n",
+        "    ax.set_xlabel('x1');ax.set_ylabel('x2')\n",
+        "    levels = np.arange(-10,10,1.0)\n",
+        "    ax.contour(x1_mesh, x2_mesh, y, levels, cmap='winter')\n",
+        "    plt.show()"
+      ],
+      "metadata": {
+        "id": "57Gvkk-Ir_7b"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Define a linear function with two inputs, x1 and x2\n",
+        "def linear_function_2D(x1,x2,beta,omega1,omega2):\n",
+        "  # TODO -- replace the code line below with formula for 2D linear equation\n",
+        "  y = x1\n",
+        "  return y"
+      ],
+      "metadata": {
+        "id": "YxeNhrXMzkZR"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Plot the 2D function\n",
+        "\n",
+        "# Make 2D array of x and y points\n",
+        "x1 = np.arange(0.0, 10.0, 0.1)\n",
+        "x2 = np.arange(0.0, 10.0, 0.1)\n",
+        "x1,x2 = np.meshgrid(x1,x2)  # https://www.geeksforgeeks.org/numpy-meshgrid-function/\n",
+        "\n",
+        "# Compute the 2D function for given values of omega1, omega2\n",
+        "beta = 0.0; omega1 = 1.0; omega2 = -0.5\n",
+        "y  = linear_function_2D(x1,x2,beta, omega1, omega2)\n",
+        "\n",
+        "# Draw the function.  \n",
+        "# Color represents y value (brighter = higher value)\n",
+        "# Black = -10 or less, White = +10 or more\n",
+        "# 0 = mid orange\n",
+        "# Lines are conoturs where value is equal\n",
+        "draw_2D_function(x1,x2,y)\n",
+        "\n",
+        "# TODO\n",
+        "# Predict what this plot will look like if you set omega_1 to zero\n",
+        "# Change the code and see if you are right.\n",
+        "\n",
+        "# TODO\n",
+        "# Predict what this plot will look like if you set omega_2 to zero\n",
+        "# Change the code and see if you are right.\n",
+        "\n",
+        "#TODO \n",
+        "# Predict what this plot will look like if you set beta to -5\n",
+        "# Change the code and see if you are correct\n"
+      ],
+      "metadata": {
+        "id": "rn_UBRDBysmR"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Often we will want to compute many linear functions at the same time.  For example, we might have three inputs, $x_1$, $x_2$, and $x_3$ and want to compute two linear functions giving $y_1$ and $y_2$. Of course, we could do this by just running each equation separately,<br><br>\n",
+        "\n",
+        "\\begin{eqnarray}y_1 &=& \\beta_1 + \\omega_{11} x_1 + \\omega_{12} x_2 + \\omega_{13} x_3\\\\\n",
+        "y_2 &=& \\beta_2 + \\omega_{21} x_1 + \\omega_{22} x_2 + \\omega_{23} x_3.\n",
+        "\\end{eqnarray}<br>\n",
+        "\n",
+        "However, we can write it more compactly with vectors and matrices:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\begin{bmatrix} y_1\\\\ y_2 \\end{bmatrix} = \\begin{bmatrix}\\beta_{1}\\\\\\beta_{2}\\end{bmatrix}+ \\begin{bmatrix}\\omega_{11}&\\omega_{12}&\\omega_{13}\\\\\\omega_{21}&\\omega_{22}&\\omega_{23}\\end{bmatrix}\\begin{bmatrix}x_{1}\\\\x_{2}\\\\x_{3}\\end{bmatrix},\n",
+        "\\end{equation}<br>\n",
+        "or \n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\mathbf{y} = \\boldsymbol\\beta +\\boldsymbol\\Omega\\mathbf{x}.\n",
+        "\\end{equation}\n",
+        "\n",
+        "for short.  Here, lowercase bold symbols are used for vectors.  Upper case bold symbols are used for matrices.\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "i8tLwpls476R"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Define a linear function with three inputs, x1, x2, and x_3\n",
+        "def linear_function_3D(x1,x2,x3,beta,omega1,omega2,omega3):\n",
+        "  # TODO -- replace the code below with formula for a single 3D linear equation\n",
+        "  y = x1\n",
+        "  return y"
+      ],
+      "metadata": {
+        "id": "MjHXMavh9IUz"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's compute two linear equations, using both the individual equations and the vector / matrix form and check they give the same answer"
+      ],
+      "metadata": {
+        "id": "fGzVJQ6N-mHJ"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Define the parameters\n",
+        "beta1 = 0.5; beta2 = 0.2\n",
+        "omega11 =  -1.0 ; omega12 = 0.4; omega13 = -0.3\n",
+        "omega21 =  0.1  ; omega22 = 0.1; omega23 = 1.2\n",
+        "\n",
+        "# Define the inputs\n",
+        "x1 = 4 ; x2 =-1; x3 = 2\n",
+        "\n",
+        "# Compute using the individual equations\n",
+        "y1 = linear_function_3D(x1,x2,x3,beta1,omega11,omega12,omega13)\n",
+        "y2 = linear_function_3D(x1,x2,x3,beta2,omega21,omega22,omega23)\n",
+        "print(\"Individual equations\")\n",
+        "print('y1 = %3.3f\\ny2 = %3.3f'%((y1,y2)))\n",
+        "\n",
+        "# Define vectors and matrices\n",
+        "beta_vec = np.array([[beta1],[beta2]])\n",
+        "omega_mat = np.array([[omega11,omega12,omega13],[omega21,omega22,omega23]])\n",
+        "x_vec = np.array([[x1], [x2], [x3]])\n",
+        "\n",
+        "# Compute with vector/matrix form\n",
+        "y_vec = beta_vec+np.matmul(omega_mat, x_vec)\n",
+        "print(\"Matrix/vector form\")\n",
+        "print('y1= %3.3f\\ny2 = %3.3f'%((y_vec[0],y_vec[1])))\n"
+      ],
+      "metadata": {
+        "id": "Swd_bFIE9p2n"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "(Optional, but highly recommended) Think that you have this down?  Here's some stuff to think about:<br>\n",
+        "\n",
+        "1.  A single linear equation with three inputs (i.e. **linear_function_3D()**) associates a value y with each point in a 3D space ($x_1$,$x_2$,$x_3$).  Is it possible to visualize this?   What value is at position (0,0,0)?\n",
+        "\n",
+        "2.  Write code to compute three linear equations with two inputs ($x_1$, $x_2$) using both the individual equations and the matrix form (you can make up any values for the inputs $\\beta_{i}$ and the slopes $\\omega_{ij}$."
+      ],
+      "metadata": {
+        "id": "3LGRoTMLU8ZU"
+      }
+    }
+  ]
+}

index.html

@@ -0,0 +1,51 @@
+<h1>Understanding Deep Learning</h1>
+by Simon J.D. Prince
+<br>
+To be published by MIT Press.
+
+
+<h2>Key links</h2>
+<ul>
+	<li> <a href="https://github.com/udlbook/udlbook/releases/download/v0.2.0/UnderstandingDeepLearning_01_10_22_C.pdf">Draft PDF Chapters 2-13</a>  2022-10-01. CC-BY-NC-ND license
+	<li> Draft PDF Chapters 1,14-19 (coming Jan 2nd, 2023)
+	<li> Jupyter notebooks (coming Spring 2023)
+	<li> Report errata via <a href="https://github.com/udlbook/udlbook/issues">github</a> or contact me directly at udlbookmail@gmail.com
+	<li> Follow me on <a href="https://twitter.com/SimonPrinceAI">Twitter</a> or <a href="https://www.linkedin.com/in/simon-prince-615bb9165/">LinkedIn</a> for updates.
+</ul>
+
+<h2>Table of contents</h2>
+<ul> 
+	<li>  Chapter 1 - Introduction
+	<li>  Chapter 2 - Supervised learning <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap2PDF.zip">PDF Figures</a> / <a href="https://github.com/udlbook/udlbook/raw/main/Slides/UDLChap2.pptx">PowerPoint Figures</a> 
+	<li>  Chapter 3 - Shallow neural networks <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap3PDF.zip">PDF Figures</a> / <a href="https://github.com/udlbook/udlbook/raw/main/Slides/UDLChap3.pptx">PowerPoint Figures</a> 
+	<li>  Chapter 4 - Deep neural networks <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap4PDF.zip">PDF Figures</a> / <a href="https://github.com/udlbook/udlbook/raw/main/Slides/UDLChap4.pptx">PowerPoint Figures</a> 
+	<li>  Chapter 5 - Loss functions <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap5PDF.zip">PDF Figures</a> / <a href="https://github.com/udlbook/udlbook/raw/main/Slides/UDLChap5.pptx">PowerPoint Figures</a> 
+	<li>  Chapter 6 - Training models <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap6PDF.zip">PDF Figures</a> / <a href="https://github.com/udlbook/udlbook/raw/main/Slides/UDLChap6.pptx">PowerPoint Figures</a> 
+	<li>  Chapter 7 - Gradients and initialization <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap7PDF.zip">PDF Figures</a> / <a href="https://github.com/udlbook/udlbook/raw/main/Slides/UDLChap7.pptx">PowerPoint Figures</a> 
+	<li>  Chapter 8 - Measuring performance <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap8PDF.zip">PDF Figures</a> / <a href="https://github.com/udlbook/udlbook/raw/main/Slides/UDLChap8.pptx">PowerPoint Figures</a> 
+	<li>  Chapter 9 - Regularization <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap9PDF.zip">PDF Figures</a> / <a href="https://github.com/udlbook/udlbook/raw/main/Slides/UDLChap9.pptx">PowerPoint Figures</a> 
+	<li>  Chapter 10 - Convolutional nets  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap10PDF.zip">PDF Figures</a> / <a href="https://github.com/udlbook/udlbook/raw/main/Slides/UDLChap10.pptx">PowerPoint Figures</a> 
+	<li>  Chapter 11 - Residual networks <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap11PDF.zip">PDF Figures</a> / <a href="https://github.com/udlbook/udlbook/raw/main/Slides/UDLChap11.pptx">PowerPoint Figures</a> 
+	<li>  Chapter 12 - Transformers  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap12PDF.zip">PDF Figures</a> / <a href="https://github.com/udlbook/udlbook/raw/main/Slides/UDLChap12.pptx">PowerPoint Figures</a> 
+	<li>  Chapter 13 - Graph neural networks  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap13PDF.zip">PDF Figures</a> / <a href="https://github.com/udlbook/udlbook/raw/main/Slides/UDLChap13.pptx">PowerPoint Figures</a> 
+	<li>  Chapter 14 - Variational auto-encoders
+	<li>  Chapter 15 - Normalizing flows
+	<li>  Chapter 16 - Generative adversarial networks
+	<li>  Chapter 17 - Diffusion models
+	<li>  Chapter 18 - Deep reinforcement learning
+	<li>  Chapter 19 - Why does deep learning work?
+</ul>
+
+<br>
+Citation:
+<pre><code>
+ @book{prince2022understanding,
+ author = "Simon J.D. Prince",
+ title = "Understanding Deep Learning",
+ publisher = "MIT Press",
+ year = 2022,
+ url = "https://udlbook.github.io/udlbook/"
+}
+</code></pre>
+	
+