Created using Colab

commit.

.editorconfig

@@ -0,0 +1,10 @@
+root = true
+
+[*.{js,jsx,ts,tsx,md,mdx,json,cjs,mjs,css}]
+indent_style = space
+indent_size = 4
+end_of_line = lf
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = true
+max_line_length = 100

.eslintrc.cjs

@@ -0,0 +1,18 @@
+module.exports = {
+    root: true,
+    env: { browser: true, es2020: true, node: true },
+    extends: [
+        "eslint:recommended",
+        "plugin:react/recommended",
+        "plugin:react/jsx-runtime",
+        "plugin:react-hooks/recommended",
+    ],
+    ignorePatterns: ["build", ".eslintrc.cjs"],
+    parserOptions: { ecmaVersion: "latest", sourceType: "module" },
+    settings: { react: { version: "18.2" } },
+    plugins: ["react-refresh"],
+    rules: {
+        "react/jsx-no-target-blank": "off",
+        "react-refresh/only-export-components": ["warn", { allowConstantExport: true }],
+    },
+};

.gitignore

@@ -0,0 +1,30 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# production
+/dist
+
+# ENV
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# IDE
+.idea
+.vscode
+
+# macOS
+.DS_Store

.prettierignore

@@ -0,0 +1,7 @@
+# ignore these directories when formatting the repo
+/Blogs
+/CM20315
+/CM20315_2023
+/Notebooks
+/PDFFigures
+/Slides

.prettierrc.cjs

@@ -0,0 +1,14 @@
+/** @type {import("prettier").Config} */
+const prettierConfig = {
+    trailingComma: "all",
+    tabWidth: 4,
+    useTabs: false,
+    semi: true,
+    singleQuote: false,
+    bracketSpacing: true,
+    printWidth: 100,
+    endOfLine: "lf",
+    plugins: [require.resolve("prettier-plugin-organize-imports")],
+};
+
+module.exports = prettierConfig;

Blogs/BorealisGradientFlow.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyO6cFY1oR4CmbHL2QywgTXm",
+      "authorship_tag": "ABX9TyP9fLqBQPgcYJB1KXs3Scp/",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -31,7 +31,7 @@
       "source": [
         "# Gradient flow\n",
         "\n",
-        "This notebook replicates some of the results in the the Borealis AI blog on gradient flow.  \n"
+        "This notebook replicates some of the results in the Borealis AI [blog](https://www.borealisai.com/research-blogs/gradient-flow/) on gradient flow.  \n"
       ],
       "metadata": {
         "id": "ucrRRJ4dq8_d"

Blogs/BorealisODENumerical.ipynb

@@ -0,0 +1,432 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Blogs/BorealisODENumerical.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "JXsO7ce7oqeq"
+      },
+      "source": [
+        "# Numerical methods for ODEs\n",
+        "\n",
+        "This blog contains code that accompanies the RBC Borealis blog on numerical methods for ODEs. Contact udlbookmail@gmail.com if you find any problems."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "AnvAKtP_oqes"
+      },
+      "source": [
+        "Import relevant libraries"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "UF-gJyZggyrl"
+      },
+      "outputs": [],
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "szWLVrSSoqet"
+      },
+      "source": [
+        "Define the ODE that we will be experimenting with."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "NkrGZLL6iM3P"
+      },
+      "outputs": [],
+      "source": [
+        "# The ODE that we will experiment with\n",
+        "def ode_lin_homog(t,x):\n",
+        "  return 0.5 * x ;\n",
+        "\n",
+        "# The derivative of the ODE function with respect to x (needed for Taylor's method)\n",
+        "def ode_lin_homog_deriv_x(t,x):\n",
+        "  return 0.5 ;\n",
+        "\n",
+        "# The derivative of the ODE function with respect to t (needed for Taylor's method)\n",
+        "def ode_lin_homog_deriv_t(t,x):\n",
+        "  return 0.0 ;\n",
+        "\n",
+        "# The closed form solution (so we can measure the error)\n",
+        "def ode_lin_homog_soln(t,C=0.5):\n",
+        "  return C * np.exp(0.5 * t) ;"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "In1C9wZkoqet"
+      },
+      "source": [
+        "This is a generic method that runs the numerical methods. It takes the initial conditions ($t_0$, $x_0$), the final time $t_1$ and the step size $h$.  It also takes the ODE function itself and its derivatives (only used for Taylor's method).  Finally, the parameter \"step_function\" is the method used to update (e.g., Euler's methods, Runge-Kutte 4-step)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "VZfZDJAfmyrf"
+      },
+      "outputs": [],
+      "source": [
+        "def run_numerical(x_0, t_0, t_1, h, ode_func, ode_func_deriv_x, ode_func_deriv_t, ode_soln, step_function):\n",
+        "  x = [x_0]\n",
+        "  t = [t_0]\n",
+        "  while (t[-1] <= t_1):\n",
+        "    x = x+[step_function(x[-1],t[-1],h, ode_func, ode_func_deriv_x, ode_func_deriv_t)]\n",
+        "    t = t + [t[-1]+h]\n",
+        "\n",
+        "  # Returns x,y plot plus total numerical error at last point.\n",
+        "  return t, x, np.abs(ode_soln(t[-1])-x[-1])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Vfkc3-_7oqet"
+      },
+      "source": [
+        "Run the numerical method with step sizes of 2.0, 1.0, 0.5, 0.25, 0.125, 0.0675 and plot the results"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1tyGbMZhoqeu"
+      },
+      "outputs": [],
+      "source": [
+        "def run_and_plot(ode, ode_deriv_x, ode_deriv_t, ode_solution, step_function):\n",
+        "    # Specify the grid of points to draw the ODE\n",
+        "    t = np.arange(0.04, 4.0, 0.2)\n",
+        "    x = np.arange(0.04, 4.0, 0.2)\n",
+        "    T, X = np.meshgrid(t,x)\n",
+        "\n",
+        "    # ODE equation at these grid points (used to draw quiver-plot)\n",
+        "    dx = ode(T,X)\n",
+        "    dt = np.ones(dx.shape)\n",
+        "\n",
+        "    # The ground truth solution\n",
+        "    t2= np.arange(0,10,0.1)\n",
+        "    x2 = ode_solution(t2)\n",
+        "\n",
+        "    #####################################x_0, t_0, t_1, h #################################################\n",
+        "    t_sim1,x_sim1,error1 = run_numerical(0.5, 0.0, 4.0, 2.0000, ode, ode_deriv_x, ode_deriv_t, ode_solution, step_function)\n",
+        "    t_sim2,x_sim2,error2 = run_numerical(0.5, 0.0, 4.0, 1.0000, ode, ode_deriv_x, ode_deriv_t, ode_solution, step_function)\n",
+        "    t_sim3,x_sim3,error3 = run_numerical(0.5, 0.0, 4.0, 0.5000, ode, ode_deriv_x, ode_deriv_t, ode_solution, step_function)\n",
+        "    t_sim4,x_sim4,error4 = run_numerical(0.5, 0.0, 4.0, 0.2500, ode, ode_deriv_x, ode_deriv_t, ode_solution, step_function)\n",
+        "    t_sim5,x_sim5,error5 = run_numerical(0.5, 0.0, 4.0, 0.1250, ode, ode_deriv_x, ode_deriv_t, ode_solution, step_function)\n",
+        "    t_sim6,x_sim6,error6 = run_numerical(0.5, 0.0, 4.0, 0.0675, ode, ode_deriv_x, ode_deriv_t, ode_solution, step_function)\n",
+        "\n",
+        "    # Plot the ODE and ground truth solution\n",
+        "    fig,ax = plt.subplots()\n",
+        "    ax.quiver(T,X,dt,dx, scale=35.0)\n",
+        "    ax.plot(t2,x2,'r-')\n",
+        "\n",
+        "    # Plot the numerical approximations\n",
+        "    ax.plot(t_sim1,x_sim1,'.-',markeredgecolor='#773c23ff',markerfacecolor='#d18362', color='#d18362', markersize=10)\n",
+        "    ax.plot(t_sim2,x_sim2,'.-',markeredgecolor='#773c23ff',markerfacecolor='#d18362', color='#d18362', markersize=10)\n",
+        "    ax.plot(t_sim3,x_sim3,'.-',markeredgecolor='#773c23ff',markerfacecolor='#d18362', color='#d18362', markersize=10)\n",
+        "    ax.plot(t_sim4,x_sim4,'.-',markeredgecolor='#773c23ff',markerfacecolor='#d18362', color='#d18362', markersize=10)\n",
+        "    ax.plot(t_sim5,x_sim5,'.-',markeredgecolor='#773c23ff',markerfacecolor='#d18362', color='#d18362', markersize=10)\n",
+        "    ax.plot(t_sim6,x_sim6,'.-',markeredgecolor='#773c23ff',markerfacecolor='#d18362', color='#d18362', markersize=10)\n",
+        "\n",
+        "    ax.set_aspect('equal')\n",
+        "    ax.set_xlim(0,4)\n",
+        "    ax.set_ylim(0,4)\n",
+        "\n",
+        "    plt.show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "JYrq8QIwvOIy"
+      },
+      "source": [
+        "# Euler Method\n",
+        "\n",
+        "Define the Euler method and set up functions for plotting."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "N73xMnCukVVX"
+      },
+      "outputs": [],
+      "source": [
+        "def euler_step(x_0, t_0, h, ode_func, ode_func_deriv_x=None, ode_func_deriv_t=None):\n",
+        "  return x_0 + h * ode_func(t_0, x_0) ;"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4B1_PGEcsZ9H"
+      },
+      "outputs": [],
+      "source": [
+        "run_and_plot(ode_lin_homog, None, None, ode_lin_homog_soln, euler_step)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FfwNihtkvJeX"
+      },
+      "source": [
+        "# Heun's Method"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "srHfNDcDxI1o"
+      },
+      "outputs": [],
+      "source": [
+        "def heun_step(x_0, t_0, h, ode_func, ode_func_deriv_x=None, ode_func_deriv_t=None):\n",
+        "  f_x0_t0 = ode_func(t_0, x_0)\n",
+        "  return x_0 + h/2 * ( f_x0_t0 + ode_func(t_0+h, x_0+h*f_x0_t0)) ;"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "WOApHz9xoqev"
+      },
+      "outputs": [],
+      "source": [
+        "run_and_plot(ode_lin_homog, None, None, ode_lin_homog_soln, heun_step)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0XSzzFDIvRhm"
+      },
+      "source": [
+        "# Modified Euler method"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "fSXprgVJ5Yep"
+      },
+      "outputs": [],
+      "source": [
+        "def modified_euler_step(x_0, t_0, h, ode_func, ode_func_deriv_x=None, ode_func_deriv_t=None):\n",
+        "  f_x0_t0 = ode_func(t_0, x_0)\n",
+        "  return x_0 + h * ode_func(t_0+h/2, x_0+ h * f_x0_t0/2) ;"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "8LKSrCD2oqev"
+      },
+      "outputs": [],
+      "source": [
+        "run_and_plot(ode_lin_homog, None, None, ode_lin_homog_soln, modified_euler_step)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "yp8ZBpwooqev"
+      },
+      "source": [
+        "# Second order Taylor's method"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "NtBBgzWLoqev"
+      },
+      "outputs": [],
+      "source": [
+        "def taylor_2nd_order(x_0, t_0, h, ode_func, ode_func_deriv_x, ode_func_deriv_t):\n",
+        "    f1 = ode_func(t_0, x_0)\n",
+        "    return x_0 + h * ode_func(t_0, x_0) + (h*h/2) * (ode_func_deriv_x(t_0,x_0) * ode_func(t_0, x_0) + ode_func_deriv_t(t_0, x_0))"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ioeeIohUoqev"
+      },
+      "outputs": [],
+      "source": [
+        "run_and_plot(ode_lin_homog, ode_lin_homog_deriv_x, ode_lin_homog_deriv_t, ode_lin_homog_soln, taylor_2nd_order)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WcuhV5lL1zAJ"
+      },
+      "source": [
+        "# Fourth Order Runge Kutta"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "0NZN81Bpwu56"
+      },
+      "outputs": [],
+      "source": [
+        "def runge_kutta_4_step(x_0, t_0, h, ode_func, ode_func_deriv_x=None, ode_func_deriv_t=None):\n",
+        "    f1 = ode_func(t_0, x_0)\n",
+        "    f2 = ode_func(t_0+h/2,x_0+f1 * h/2)\n",
+        "    f3 = ode_func(t_0+h/2,x_0+f2 * h/2)\n",
+        "    f4 = ode_func(t_0+h, x_0+ f3*h)\n",
+        "    return x_0 + (h/6) * (f1 + 2*f2 + 2*f3+f4)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "K-OxE9E6oqew"
+      },
+      "outputs": [],
+      "source": [
+        "run_and_plot(ode_lin_homog, None, None, ode_lin_homog_soln, runge_kutta_4_step)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "7JifxBhhoqew"
+      },
+      "source": [
+        "# Plot the error as a function of step size"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZoEpmlCfsi9P"
+      },
+      "outputs": [],
+      "source": [
+        "# Run systematically with a number of different step sizes and store errors for each\n",
+        "def get_errors(ode, ode_deriv_x, ode_deriv_t, ode_solution, step_function):\n",
+        "    # Choose the step size h to divide the plotting interval into 1,2,4,8... segments.\n",
+        "    # The plots in the article add a few more smaller step sizes, but this takes a while to compute.\n",
+        "    # Add them back in if you want the full plot.\n",
+        "    all_h = (1./np.array([1,2,4,8,16,32,64,128,256,512,1024,2048,4096])).tolist()\n",
+        "    all_err = []\n",
+        "\n",
+        "    for i in range(len(all_h)):\n",
+        "        t_sim,x_sim,err = run_numerical(0.5, 0.0, 4.0, all_h[i], ode, ode_deriv_x, ode_deriv_t, ode_solution, step_function)\n",
+        "        all_err = all_err + [err]\n",
+        "\n",
+        "    return all_h, all_err"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "X0O0KK47xF28"
+      },
+      "outputs": [],
+      "source": [
+        "# Plot the errors\n",
+        "all_h, all_err_euler = get_errors(ode_lin_homog, ode_lin_homog_deriv_x, ode_lin_homog_deriv_t, ode_lin_homog_soln, euler_step)\n",
+        "all_h, all_err_heun = get_errors(ode_lin_homog, ode_lin_homog_deriv_x, ode_lin_homog_deriv_t, ode_lin_homog_soln, heun_step)\n",
+        "all_h, all_err_mod_euler = get_errors(ode_lin_homog, ode_lin_homog_deriv_x, ode_lin_homog_deriv_t, ode_lin_homog_soln, modified_euler_step)\n",
+        "all_h, all_err_taylor = get_errors(ode_lin_homog, ode_lin_homog_deriv_x, ode_lin_homog_deriv_t, ode_lin_homog_soln, taylor_2nd_order)\n",
+        "all_h, all_err_rk = get_errors(ode_lin_homog, ode_lin_homog_deriv_x, ode_lin_homog_deriv_t, ode_lin_homog_soln, runge_kutta_4_step)\n",
+        "\n",
+        "\n",
+        "fig, ax = plt.subplots()\n",
+        "ax.loglog(all_h, all_err_euler,'ro-')\n",
+        "ax.loglog(all_h, all_err_heun,'bo-')\n",
+        "ax.loglog(all_h, all_err_mod_euler,'go-')\n",
+        "ax.loglog(all_h, all_err_taylor,'co-')\n",
+        "ax.loglog(all_h, all_err_rk,'mo-')\n",
+        "ax.set_ylim(1e-13,1e1)\n",
+        "ax.set_xlim(1e-6,1e1)\n",
+        "ax.set_aspect(0.5)\n",
+        "ax.set_xlabel('Step size, $h$')\n",
+        "ax.set_ylabel('Error')\n",
+        "plt.show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "BttOqpeo9MsJ"
+      },
+      "source": [
+        "Note that for this ODE, the Heun, Modified Euler and Taylor methods provide EXACTLY the same updates, and so the error curves for all three are identical (subject to difference is numerical rounding errors).  This is not in general the case, although the general trend would be the same for each."
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3 (ipykernel)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.9.10"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

CM20315/CM20315_Coursework_IV.ipynb

@@ -128,7 +128,7 @@
         "\n",
         "In part (b) of the practical we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius.  You will find that the volume decreases to almost nothing in high dimensions.  All of the volume is in the corners of the unit hypercube (which always has volume 1). Double weird.\n",
         "\n",
-        "Note that you you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
+        "Note that you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
       ],
       "metadata": {
         "id": "b2FYKV1SL4Z7"

CM20315/CM20315_Loss_II.ipynb

@@ -199,7 +199,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "The left is model output and the right is the model output after the sigmoid has been applied, so it now lies in the range [0,1] and represents the probability, that y=1.  The black dots show the training data.  We'll compute the the likelihood and the negative log likelihood."
+        "The left is model output and the right is the model output after the sigmoid has been applied, so it now lies in the range [0,1] and represents the probability, that y=1.  The black dots show the training data.  We'll compute the likelihood and the negative log likelihood."
       ],
       "metadata": {
         "id": "MvVX6tl9AEXF"

CM20315/CM20315_Loss_III.ipynb

@@ -218,7 +218,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue)   The dots at the bottom show the training data with the same color scheme.  So we want the red curve to be high where there are red dots, the green curve to be high where there are green dotsmand the blue curve to be high where there are blue dots  We'll compute the the likelihood and the negative log likelihood."
+        "The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue)   The dots at the bottom show the training data with the same color scheme.  So we want the red curve to be high where there are red dots, the green curve to be high where there are green dotsmand the blue curve to be high where there are blue dots  We'll compute the likelihood and the negative log likelihood."
       ],
       "metadata": {
         "id": "MvVX6tl9AEXF"

CM20315_2023/CM20315_Coursework_IV.ipynb

@@ -128,7 +128,7 @@
         "\n",
         "In part (b) of the practical we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius.  You will find that the volume decreases to almost nothing in high dimensions.  All of the volume is in the corners of the unit hypercube (which always has volume 1). Double weird.\n",
         "\n",
-        "Note that you you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
+        "Note that you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
       ],
       "metadata": {
         "id": "b2FYKV1SL4Z7"

CM20315_2023/CM20315_Coursework_V_2023.ipynb

@@ -214,7 +214,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Compute the derivative of the the loss with respect to the function output f_val\n",
+        "# Compute the derivative of the loss with respect to the function output f_val\n",
         "def dl_df(f_val,y):\n",
         "  # Compute sigmoid of network output\n",
         "  sig_f_val = sig(f_val)\n",

Notebooks/Chap01/1_1_BackgroundMathematics.ipynb

@@ -1,18 +1,16 @@
 {
   "cells": [
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "colab_type": "text",
+        "id": "view-in-github",
-        "id": "view-in-github"
+        "colab_type": "text"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap01/1_1_BackgroundMathematics.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "s5zzKSOusPOB"
@@ -21,7 +19,7 @@
         "\n",
         "# **Notebook 1.1 -- Background Mathematics**\n",
         "\n",
-        "The purpose of this Python notebook is to make sure you can use CoLab and to familiarize yourself with some of the background mathematical concepts that you are going to need to understand deep learning. <br><br> It's not meant to be difficult and it may be that you know some or all of this information already.<br><br> Math is *NOT* a spectator sport.  You won't learn it by just listening to lectures or reading books.  It really helps to interact with it and explore yourself. <br><br> Work through the cells below, running each cell in turn.  In various places you will see the words **\"TO DO\"**. Follow the instructions at these places and write code to complete the functions.  There are also questions interspersed in the text.\n",
+        "The purpose of this Python notebook is to make sure you can use CoLab and to familiarize yourself with some of the background mathematical concepts that you are going to need to understand deep learning. <br><br> It's not meant to be difficult and it may be that you know some or all of this information already.<br><br> Math is *NOT* a spectator sport.  You won't learn it by just listening to lectures or reading books.  It really helps to interact with it and explore yourself. <br><br> Work through the cells below, running each cell in turn.  In various places you will see the words **\"TODO\"**. Follow the instructions at these places and write code to complete the functions.  There are also questions interspersed in the text.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]
@@ -41,7 +39,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "WV2Dl6owme2d"
@@ -49,11 +46,11 @@
       "source": [
         "**Linear functions**<br> We will be using the term *linear equation* to mean a weighted sum of inputs plus an offset. If there is just one input $x$, then this is a straight line:\n",
         "\n",
-        "\\begin{equation}y=\\beta+\\omega x,\\end{equation} \n",
+        "\\begin{equation}y=\\beta+\\omega x,\\end{equation}\n",
         "\n",
         "where $\\beta$ is the y-intercept of the linear and $\\omega$ is the slope of the line. When there are two inputs $x_{1}$ and $x_{2}$, then this becomes:\n",
         "\n",
-        "\\begin{equation}y=\\beta+\\omega_1 x_1 + \\omega_2 x_2.\\end{equation} \n",
+        "\\begin{equation}y=\\beta+\\omega_1 x_1 + \\omega_2 x_2.\\end{equation}\n",
         "\n",
         "Any other functions are by definition **non-linear**.\n",
         "\n",
@@ -99,7 +96,7 @@
         "ax.plot(x,y,'r-')\n",
         "ax.set_ylim([0,10]);ax.set_xlim([0,10])\n",
         "ax.set_xlabel('x'); ax.set_ylabel('y')\n",
-        "plt.show\n",
+        "plt.show()\n",
         "\n",
         "# TODO -- experiment with changing the values of beta and omega\n",
         "# to understand what they do.  Try to make a line\n",
@@ -107,7 +104,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "AedfvD9dxShZ"
@@ -192,7 +188,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "i8tLwpls476R"
@@ -236,7 +231,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "fGzVJQ6N-mHJ"
@@ -275,11 +269,10 @@
         "# Compute with vector/matrix form\n",
         "y_vec = beta_vec+np.matmul(omega_mat, x_vec)\n",
         "print(\"Matrix/vector form\")\n",
-        "print('y1= %3.3f\\ny2 = %3.3f'%((y_vec[0],y_vec[1])))\n"
+        "print('y1= %3.3f\\ny2 = %3.3f'%((y_vec[0][0],y_vec[1][0])))\n"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "3LGRoTMLU8ZU"
@@ -293,7 +286,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "7Y5zdKtKZAB2"
@@ -325,11 +317,10 @@
         "ax.plot(x,y,'r-')\n",
         "ax.set_ylim([0,100]);ax.set_xlim([-5,5])\n",
         "ax.set_xlabel('x'); ax.set_ylabel('exp[x]')\n",
-        "plt.show"
+        "plt.show()"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "XyrT8257IWCu"
@@ -341,11 +332,10 @@
         "2. What is $\\exp[1]$?\n",
         "3. What is $\\exp[-\\infty]$?\n",
         "4. What is $\\exp[+\\infty]$?\n",
-        "5. A function is convex if we can draw a straight line between any two points on the function, and this line always lies above the function. Similarly, a function is concave if a straight line between any two points always lies below the function.  Is the exponential function convex or concave or neither?\n"
+        "5. A function is convex if we can draw a straight line between any two points on the function, and the line lies above the function everywhere between these two points. Similarly, a function is concave if a straight line between any two points lies below the function everywhere between these two points.  Is the exponential function convex or concave or neither?\n"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "R6A4e5IxIWCu"
@@ -373,11 +363,10 @@
         "ax.plot(x,y,'r-')\n",
         "ax.set_ylim([-5,5]);ax.set_xlim([0,5])\n",
         "ax.set_xlabel('x'); ax.set_ylabel('$\\log[x]$')\n",
-        "plt.show"
+        "plt.show()"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "yYWrL5AXIWCv"
@@ -397,8 +386,8 @@
   ],
   "metadata": {
     "colab": {
-      "include_colab_link": true,
+      "provenance": [],
-      "provenance": []
+      "include_colab_link": true
     },
     "kernelspec": {
       "display_name": "Python 3 (ipykernel)",

Notebooks/Chap02/2_1_Supervised_Learning.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOmndC0N7dFV7W3Mh5ljOLl",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +32,7 @@
         "\n",
         "The purpose of this notebook is to explore the linear regression model discussed in Chapter 2 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and write code to complete the functions. There are also questions interspersed in the text.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and write code to complete the functions. There are also questions interspersed in the text.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -175,7 +174,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# TO DO -- Change the parameters manually to fit the model\n",
+        "# TODO -- Change the parameters manually to fit the model\n",
         "# First fix phi1 and try changing phi0 until you can't make the loss go down any more\n",
         "# Then fix phi0 and try changing phi1 until you can't make the loss go down any more\n",
         "# Repeat this process until you find a set of parameters that fit the model as in figure 2.2d\n",
@@ -197,7 +196,7 @@
       "source": [
         "# Visualizing the loss function\n",
         "\n",
-        "The above process is equivalent to to descending coordinate wise on the loss function<br>\n",
+        "The above process is equivalent to descending coordinate wise on the loss function<br>\n",
         "\n",
         "Now let's plot that function"
       ],
@@ -235,8 +234,8 @@
         "levels = 40\n",
         "ax.contour(phi0_mesh, phi1_mesh, all_losses ,levels, colors=['#80808080'])\n",
         "ax.set_ylim([1,-1])\n",
-        "ax.set_xlabel('Intercept, $\\phi_0$')\n",
+        "ax.set_xlabel(r'Intercept, $\\phi_0$')\n",
-        "ax.set_ylabel('Slope, $\\phi_1$')\n",
+        "ax.set_ylabel(r'Slope, $\\phi_1$')\n",
         "\n",
         "# Plot the position of your best fitting line on the loss function\n",
         "# It should be close to the minimum\n",

Notebooks/Chap03/3_2_Shallow_Networks_II.ipynb

@@ -32,7 +32,7 @@
         "\n",
         "The purpose of this notebook is to gain some familiarity with shallow neural networks with 2D inputs.  It works through an example similar to figure 3.8 and experiments with different activation functions. <br><br>\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and write code to complete the functions. There are also questions interspersed in the text.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and write code to complete the functions. There are also questions interspersed in the text.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ],

Notebooks/Chap03/3_3_Shallow_Network_Regions.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyNioITtfAcfxEfM3UOfQyb9",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +32,7 @@
         "\n",
         "The purpose of this notebook is to compute the maximum possible number of linear regions as seen in figure 3.9 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and write code to complete the functions. There are also questions interspersed in the text.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and write code to complete the functions. There are also questions interspersed in the text.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -62,7 +61,7 @@
       "source": [
         "The number of regions $N$ created by a shallow neural network with $D_i$ inputs and $D$ hidden units is given by Zaslavsky's formula:\n",
         "\n",
-        "\\begin{equation}N = \\sum_{j=0}^{D_{i}}\\binom{D}{j}=\\sum_{j=0}^{D_{i}} \\frac{D!}{(D-j)!j!} \\end{equation} <br>\n",
+        "\\begin{equation}N = \\sum_{j=0}^{D_{i}}\\binom{D}{j}=\\sum_{j=0}^{D_{i}} \\frac{D!}{(D-j)!j!} \\end{equation} \n",
         "\n"
       ],
       "metadata": {
@@ -221,7 +220,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Now let's plot the graph from figure 3.9a (takes ~1min)\n",
+        "# Now let's plot the graph from figure 3.9b (takes ~1min)\n",
         "dims = np.array([1,5,10,50,100])\n",
         "regions = np.zeros((dims.shape[0], 200))\n",
         "params = np.zeros((dims.shape[0], 200))\n",

Notebooks/Chap03/3_4_Activation_Functions.ipynb

@@ -22,7 +22,7 @@
         "\n",
         "The purpose of this practical is to experiment with different activation functions. <br>\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and write code to complete the functions. There are also questions interspersed in the text.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and write code to complete the functions. There are also questions interspersed in the text.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]

Notebooks/Chap04/4_1_Composing_Networks.ipynb

@@ -28,11 +28,11 @@
     {
       "cell_type": "markdown",
       "source": [
-        "#Notebook 4.1 -- Composing networks\n",
+        "# Notebook 4.1 -- Composing networks\n",
         "\n",
         "The purpose of this notebook is to understand what happens when we feed one neural network into another. It works through an example similar to 4.1 and varies both networks\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions"
       ],
@@ -134,7 +134,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Let's define two networks.  We'll put the prefixes n1_ and n2_ before all the variables to make it clear which network is which.  We'll just consider the inputs and outputs over the range [-1,1].  If you set the \"plot_all\" flat to True,  you can see the details of how they were created."
+        "Let's define two networks.  We'll put the prefixes n1_ and n2_ before all the variables to make it clear which network is which.  We'll just consider the inputs and outputs over the range [-1,1]."
       ],
       "metadata": {
         "id": "LxBJCObC-NTY"
@@ -343,7 +343,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# TO DO\n",
+        "# TODO\n",
         "# How many linear regions would there be if we ran N copies of the first network, feeding the result of the first\n",
         "# into the second, the second into the third and so on, and then passed the result into the original second\n",
         "# network (blue curve above)\n",

Notebooks/Chap04/4_2_Clipping_functions.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyPkFrjmRAUf0fxN07RC4xMI",
+      "authorship_tag": "ABX9TyPZzptvvf7OPZai8erQ/0xT",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -29,11 +29,11 @@
     {
       "cell_type": "markdown",
       "source": [
-        "#Notebook 4.2 -- Clipping functions\n",
+        "# Notebook 4.2 -- Clipping functions\n",
         "\n",
         "The purpose of this notebook is to understand how a neural network with two hidden layers build more complicated functions by clipping and recombining the representations at the intermediate hidden variables.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions"
       ],
@@ -127,26 +127,26 @@
         "    fig, ax = plt.subplots(3,3)\n",
         "    fig.set_size_inches(8.5, 8.5)\n",
         "    fig.tight_layout(pad=3.0)\n",
-        "    ax[0,0].plot(x,layer2_pre_1,'r-'); ax[0,0].set_ylabel('$\\psi_{10}+\\psi_{11}h_{1}+\\psi_{12}h_{2}+\\psi_{13}h_3$')\n",
+        "    ax[0,0].plot(x,layer2_pre_1,'r-'); ax[0,0].set_ylabel(r'$\\psi_{10}+\\psi_{11}h_{1}+\\psi_{12}h_{2}+\\psi_{13}h_3$')\n",
-        "    ax[0,1].plot(x,layer2_pre_2,'b-'); ax[0,1].set_ylabel('$\\psi_{20}+\\psi_{21}h_{1}+\\psi_{22}h_{2}+\\psi_{23}h_3$')\n",
+        "    ax[0,1].plot(x,layer2_pre_2,'b-'); ax[0,1].set_ylabel(r'$\\psi_{20}+\\psi_{21}h_{1}+\\psi_{22}h_{2}+\\psi_{23}h_3$')\n",
-        "    ax[0,2].plot(x,layer2_pre_3,'g-'); ax[0,2].set_ylabel('$\\psi_{30}+\\psi_{31}h_{1}+\\psi_{32}h_{2}+\\psi_{33}h_3$')\n",
+        "    ax[0,2].plot(x,layer2_pre_3,'g-'); ax[0,2].set_ylabel(r'$\\psi_{30}+\\psi_{31}h_{1}+\\psi_{32}h_{2}+\\psi_{33}h_3$')\n",
-        "    ax[1,0].plot(x,h1_prime,'r-'); ax[1,0].set_ylabel(\"$h_{1}^{'}$\")\n",
+        "    ax[1,0].plot(x,h1_prime,'r-'); ax[1,0].set_ylabel(r\"$h_{1}^{'}$\")\n",
-        "    ax[1,1].plot(x,h2_prime,'b-'); ax[1,1].set_ylabel(\"$h_{2}^{'}$\")\n",
+        "    ax[1,1].plot(x,h2_prime,'b-'); ax[1,1].set_ylabel(r\"$h_{2}^{'}$\")\n",
-        "    ax[1,2].plot(x,h3_prime,'g-'); ax[1,2].set_ylabel(\"$h_{3}^{'}$\")\n",
+        "    ax[1,2].plot(x,h3_prime,'g-'); ax[1,2].set_ylabel(r\"$h_{3}^{'}$\")\n",
-        "    ax[2,0].plot(x,phi1_h1_prime,'r-'); ax[2,0].set_ylabel(\"$\\phi_1 h_{1}^{'}$\")\n",
+        "    ax[2,0].plot(x,phi1_h1_prime,'r-'); ax[2,0].set_ylabel(r\"$\\phi_1 h_{1}^{'}$\")\n",
-        "    ax[2,1].plot(x,phi2_h2_prime,'b-'); ax[2,1].set_ylabel(\"$\\phi_2 h_{2}^{'}$\")\n",
+        "    ax[2,1].plot(x,phi2_h2_prime,'b-'); ax[2,1].set_ylabel(r\"$\\phi_2 h_{2}^{'}$\")\n",
-        "    ax[2,2].plot(x,phi3_h3_prime,'g-'); ax[2,2].set_ylabel(\"$\\phi_3 h_{3}^{'}$\")\n",
+        "    ax[2,2].plot(x,phi3_h3_prime,'g-'); ax[2,2].set_ylabel(r\"$\\phi_3 h_{3}^{'}$\")\n",
         "\n",
         "    for plot_y in range(3):\n",
         "      for plot_x in range(3):\n",
         "        ax[plot_y,plot_x].set_xlim([0,1]);ax[plot_x,plot_y].set_ylim([-1,1])\n",
         "        ax[plot_y,plot_x].set_aspect(0.5)\n",
-        "      ax[2,plot_y].set_xlabel('Input, $x$');\n",
+        "      ax[2,plot_y].set_xlabel(r'Input, $x$');\n",
         "    plt.show()\n",
         "\n",
         "    fig, ax = plt.subplots()\n",
         "    ax.plot(x,y)\n",
-        "    ax.set_xlabel('Input, $x$'); ax.set_ylabel('Output, $y$')\n",
+        "    ax.set_xlabel(r'Input, $x$'); ax.set_ylabel(r'Output, $y$')\n",
         "    ax.set_xlim([0,1]);ax.set_ylim([-1,1])\n",
         "    ax.set_aspect(0.5)\n",
         "    plt.show()"
@@ -169,7 +169,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Define parameters (note first dimension of theta and phi is padded to make indices match\n",
+        "# Define parameters (note first dimension of theta and psi is padded to make indices match\n",
         "# notation in book)\n",
         "theta = np.zeros([4,2])\n",
         "psi = np.zeros([4,4])\n",

Notebooks/Chap04/4_3_Deep_Networks.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyO2DaD75p+LGi7WgvTzjrk1",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -31,9 +30,9 @@
       "source": [
         "# **Notebook 4.3 Deep neural networks**\n",
         "\n",
-        "This network investigates converting neural networks to matrix form.\n",
+        "This notebook investigates converting neural networks to matrix form.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -118,7 +117,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Let's define a network.  We'll just consider the inputs and outputs over the range [-1,1].  If you set the \"plot_all\" flat to True,  you can see the details of how it was created."
+        "Let's define a network.  We'll just consider the inputs and outputs over the range [-1,1]."
       ],
       "metadata": {
         "id": "LxBJCObC-NTY"
@@ -150,7 +149,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Now we'll define the same neural network, but this time, we will  use matrix form.  When you get this right, it will draw the same plot as above."
+        "Now we'll define the same neural network, but this time, we will  use matrix form as in equation 4.15.  When you get this right, it will draw the same plot as above."
       ],
       "metadata": {
         "id": "XCJqo_AjfAra"
@@ -176,8 +175,8 @@
         "n1_in_mat = np.reshape(n1_in,(n_dim_in,n_data))\n",
         "\n",
         "# This runs the network for ALL of the inputs, x at once so we can draw graph\n",
-        "h1 = ReLU(np.matmul(beta_0,np.ones((1,n_data))) + np.matmul(Omega_0,n1_in_mat))\n",
+        "h1 = ReLU(beta_0 + np.matmul(Omega_0,n1_in_mat))\n",
-        "n1_out = np.matmul(beta_1,np.ones((1,n_data))) + np.matmul(Omega_1,h1)\n",
+        "n1_out = beta_1 + np.matmul(Omega_1,h1)\n",
         "\n",
         "# Draw the network and check that it looks the same as the non-matrix case\n",
         "plot_neural(n1_in, n1_out)"
@@ -247,9 +246,9 @@
         "n1_in_mat = np.reshape(n1_in,(n_dim_in,n_data))\n",
         "\n",
         "# This runs the network for ALL of the inputs, x at once so we can draw graph (hence extra np.ones term)\n",
-        "h1 = ReLU(np.matmul(beta_0,np.ones((1,n_data))) + np.matmul(Omega_0,n1_in_mat))\n",
+        "h1 = ReLU(beta_0 + np.matmul(Omega_0,n1_in_mat))\n",
-        "h2 = ReLU(np.matmul(beta_1,np.ones((1,n_data))) + np.matmul(Omega_1,h1))\n",
+        "h2 = ReLU(beta_1 + np.matmul(Omega_1,h1))\n",
-        "n1_out = np.matmul(beta_2,np.ones((1,n_data))) + np.matmul(Omega_2,h2)\n",
+        "n1_out = beta_2 + np.matmul(Omega_2,h2)\n",
         "\n",
         "# Draw the network and check that it looks the same as the non-matrix version\n",
         "plot_neural(n1_in, n1_out)"
@@ -291,10 +290,10 @@
         "\n",
         "\n",
         "# If you set the parameters to the correct sizes, the following code will run\n",
-        "h1 = ReLU(np.matmul(beta_0,np.ones((1,n_data))) + np.matmul(Omega_0,x));\n",
+        "h1 = ReLU(beta_0 + np.matmul(Omega_0,x));\n",
-        "h2 = ReLU(np.matmul(beta_1,np.ones((1,n_data))) + np.matmul(Omega_1,h1));\n",
+        "h2 = ReLU(beta_1 + np.matmul(Omega_1,h1));\n",
-        "h3 = ReLU(np.matmul(beta_2,np.ones((1,n_data))) + np.matmul(Omega_2,h2));\n",
+        "h3 = ReLU(beta_2 + np.matmul(Omega_2,h2));\n",
-        "y = np.matmul(beta_3,np.ones((1,n_data))) + np.matmul(Omega_3,h3)\n",
+        "y = beta_3 + np.matmul(Omega_3,h3)\n",
         "\n",
         "if h1.shape[0] is not D_1 or h1.shape[1] is not n_data:\n",
         "  print(\"h1 is wrong shape\")\n",

Notebooks/Chap05/5_1_Least_Squares_Loss.ipynb

@@ -32,7 +32,7 @@
         "\n",
         "This notebook investigates the least squares loss and the equivalence of maximum likelihood and minimum negative log likelihood.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -118,7 +118,7 @@
         "  ax.plot(x_model,y_model)\n",
         "  if sigma_model is not None:\n",
         "    ax.fill_between(x_model, y_model-2*sigma_model, y_model+2*sigma_model, color='lightgray')\n",
-        "  ax.set_xlabel('Input, $x$'); ax.set_ylabel('Output, $y$')\n",
+        "  ax.set_xlabel(r'Input, $x$'); ax.set_ylabel(r'Output, $y$')\n",
         "  ax.set_xlim([0,1]);ax.set_ylim([-1,1])\n",
         "  ax.set_aspect(0.5)\n",
         "  if title is not None:\n",
@@ -222,7 +222,7 @@
         "gauss_prob = normal_distribution(y_gauss, mu, sigma)\n",
         "fig, ax = plt.subplots()\n",
         "ax.plot(y_gauss, gauss_prob)\n",
-        "ax.set_xlabel('Input, $y$'); ax.set_ylabel('Probability $Pr(y)$')\n",
+        "ax.set_xlabel(r'Input, $y$'); ax.set_ylabel(r'Probability $Pr(y)$')\n",
         "ax.set_xlim([-5,5]);ax.set_ylim([0,1.0])\n",
         "plt.show()\n",
         "\n",

Notebooks/Chap05/5_2_Binary_Cross_Entropy_Loss.ipynb

@@ -32,7 +32,7 @@
         "\n",
         "This notebook investigates the binary cross-entropy loss.  It follows from applying the formula in section 5.2 to a loss function based on the Bernoulli distribution.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -119,12 +119,12 @@
         "  fig.set_size_inches(7.0, 3.5)\n",
         "  fig.tight_layout(pad=3.0)\n",
         "  ax[0].plot(x_model,out_model)\n",
-        "  ax[0].set_xlabel('Input, $x$'); ax[0].set_ylabel('Model output')\n",
+        "  ax[0].set_xlabel(r'Input, $x$'); ax[0].set_ylabel(r'Model output')\n",
         "  ax[0].set_xlim([0,1]);ax[0].set_ylim([-4,4])\n",
         "  if title is not None:\n",
         "    ax[0].set_title(title)\n",
         "  ax[1].plot(x_model,lambda_model)\n",
-        "  ax[1].set_xlabel('Input, $x$'); ax[1].set_ylabel('$\\lambda$ or Pr(y=1|x)')\n",
+        "  ax[1].set_xlabel(r'Input, $x$'); ax[1].set_ylabel(r'$\\lambda$ or Pr(y=1|x)')\n",
         "  ax[1].set_xlim([0,1]);ax[1].set_ylim([-0.05,1.05])\n",
         "  if title is not None:\n",
         "    ax[1].set_title(title)\n",

Notebooks/Chap05/5_3_Multiclass_Cross_entropy_Loss.ipynb

@@ -20,7 +20,7 @@
         "\n",
         "This notebook investigates the multi-class cross-entropy loss.  It follows from applying the formula in section 5.2 to a loss function based on the Categorical distribution.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]
@@ -211,7 +211,7 @@
         "id": "MvVX6tl9AEXF"
       },
       "source": [
-        "The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue).  The dots at the bottom show the training data with the same color scheme.  So we want the red curve to be high where there are red dots, the green curve to be high where there are green dots, and the blue curve to be high where there are blue dots  We'll compute the the likelihood and the negative log likelihood."
+        "The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue).  The dots at the bottom show the training data with the same color scheme.  So we want the red curve to be high where there are red dots, the green curve to be high where there are green dots, and the blue curve to be high where there are blue dots  We'll compute the likelihood and the negative log likelihood."
       ]
     },
     {
@@ -236,11 +236,10 @@
       },
       "outputs": [],
       "source": [
-        "# Let's double check we get the right answer before proceeding\n",
+        "# Here are three examples\n",
-        "print(\"Correct answer = %3.3f, Your answer = %3.3f\"%(0.2,categorical_distribution(np.array([[0]]),np.array([[0.2],[0.5],[0.3]]))))\n",
+        "print(categorical_distribution(np.array([[0]]),np.array([[0.2],[0.5],[0.3]])))\n",
-        "print(\"Correct answer = %3.3f, Your answer = %3.3f\"%(0.5,categorical_distribution(np.array([[1]]),np.array([[0.2],[0.5],[0.3]]))))\n",
+        "print(categorical_distribution(np.array([[1]]),np.array([[0.2],[0.5],[0.3]])))\n",
-        "print(\"Correct answer = %3.3f, Your answer = %3.3f\"%(0.3,categorical_distribution(np.array([[2]]),np.array([[0.2],[0.5],[0.3]]))))\n",
+        "print(categorical_distribution(np.array([[2]]),np.array([[0.2],[0.5],[0.3]])))"
-        "\n"
       ]
     },
     {

Notebooks/Chap06/6_1_Line_Search.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyN4E9Vtuk6t2BhZ0Ajv5SW3",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +32,7 @@
         "\n",
         "This notebook investigates how to find the minimum of a 1D function using line search as described in Figure 6.10.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ],
@@ -67,7 +66,7 @@
         "  fig,ax = plt.subplots()\n",
         "  ax.plot(phi_plot,loss_function(phi_plot),'r-')\n",
         "  ax.set_xlim(0,1); ax.set_ylim(0,1)\n",
-        "  ax.set_xlabel('$\\phi$'); ax.set_ylabel('$L[\\phi]$')\n",
+        "  ax.set_xlabel(r'$\\phi$'); ax.set_ylabel(r'$L[\\phi]$')\n",
         "  if a is not None and b is not None and c is not None and d is not None:\n",
         "      plt.axvspan(a, d, facecolor='k', alpha=0.2)\n",
         "      ax.plot([a,a],[0,1],'b-')\n",
@@ -131,7 +130,8 @@
         "\n",
         "        print('Iter %d, a=%3.3f, b=%3.3f, c=%3.3f, d=%3.3f'%(n_iter, a,b,c,d))\n",
         "\n",
-        "        # Rule #1 If the HEIGHT at point A is less than the HEIGHT at points B, C, and D then halve values of B, C, and D\n",
+        "        # Rule #1 If the HEIGHT at point A is less than the HEIGHT at points B, C, and D then move them to they are half\n",
+        "        # as far from A as they start\n",
         "        # i.e. bring them closer to the original point\n",
         "        # TODO REPLACE THE BLOCK OF CODE BELOW WITH THIS RULE\n",
         "        if (0):\n",

Notebooks/Chap06/6_2_Gradient_Descent.ipynb

@@ -1,18 +1,16 @@
 {
   "cells": [
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "colab_type": "text",
+        "id": "view-in-github",
-        "id": "view-in-github"
+        "colab_type": "text"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap06/6_2_Gradient_Descent.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "el8l05WQEO46"
@@ -22,7 +20,7 @@
         "\n",
         "This notebook recreates the gradient descent algorithm as shown in figure 6.1.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n"
@@ -111,7 +109,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "QU5mdGvpTtEG"
@@ -140,7 +137,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "eB5DQvU5hYNx"
@@ -162,7 +158,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "F3trnavPiHpH"
@@ -218,7 +213,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "s9Duf05WqqSC"
@@ -252,7 +246,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "RS1nEcYVuEAM"
@@ -265,7 +258,7 @@
         "\\frac{\\partial L}{\\partial \\phi_{1}}&\\approx & \\frac{L[\\phi_0, \\phi_1+\\delta]-L[\\phi_0, \\phi_1]}{\\delta}\n",
         "\\end{align}\n",
         "\n",
-        "We can't do this when there are many parameters;  for a million parameters, we would have to evaluate the loss function two million times, and usually computing the gradients directly is much more efficient."
+        "We can't do this when there are many parameters;  for a million parameters, we would have to evaluate the loss function one million plus one times, and usually computing the gradients directly is much more efficient."
       ]
     },
     {
@@ -290,7 +283,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "5EIjMM9Fw2eT"
@@ -333,11 +325,11 @@
         "          print('Iter %d, a=%3.3f, b=%3.3f, c=%3.3f, d=%3.3f'%(n_iter, a,b,c,d))\n",
         "          print('a %f, b%f, c%f, d%f'%(lossa,lossb,lossc,lossd))\n",
         "\n",
-        "        # Rule #1 If point A is less than points B, C, and D then halve points B,C, and D\n",
+        "        # Rule #1 If point A is less than points B, C, and D then halve distance from A to points B,C, and D\n",
         "        if np.argmin((lossa,lossb,lossc,lossd))==0:\n",
-        "          b = b/2\n",
+        "          b = a+ (b-a)/2\n",
-        "          c = c/2\n",
+        "          c = a+ (c-a)/2\n",
-        "          d = d/2\n",
+        "          d = a+ (d-a)/2\n",
         "          continue;\n",
         "\n",
         "        # Rule #2 If point b is less than point c then\n",
@@ -412,8 +404,8 @@
   ],
   "metadata": {
     "colab": {
-      "include_colab_link": true,
+      "provenance": [],
-      "provenance": []
+      "include_colab_link": true
     },
     "kernelspec": {
       "display_name": "Python 3",

Notebooks/Chap06/6_3_Stochastic_Gradient_Descent.ipynb

@@ -1,18 +1,16 @@
 {
   "cells": [
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "colab_type": "text",
+        "id": "view-in-github",
-        "id": "view-in-github"
+        "colab_type": "text"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap06/6_3_Stochastic_Gradient_Descent.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "el8l05WQEO46"
@@ -22,7 +20,7 @@
         "\n",
         "This notebook investigates gradient descent and stochastic gradient descent and recreates figure 6.5 from the book\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n",
@@ -122,7 +120,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "QU5mdGvpTtEG"
@@ -150,7 +147,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "eB5DQvU5hYNx"
@@ -172,7 +168,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "F3trnavPiHpH"
@@ -228,7 +223,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "s9Duf05WqqSC"
@@ -279,7 +273,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "RS1nEcYVuEAM"
@@ -316,7 +309,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "5EIjMM9Fw2eT"
@@ -359,11 +351,11 @@
         "          print('Iter %d, a=%3.3f, b=%3.3f, c=%3.3f, d=%3.3f'%(n_iter, a,b,c,d))\n",
         "          print('a %f, b%f, c%f, d%f'%(lossa,lossb,lossc,lossd))\n",
         "\n",
-        "        # Rule #1 If point A is less than points B, C, and D then halve points B,C, and D\n",
+        "        # Rule #1 If point A is less than points B, C, and D then change B,C,D so they are half their current distance from A\n",
         "        if np.argmin((lossa,lossb,lossc,lossd))==0:\n",
-        "          b = b/2\n",
+        "          b = a+ (b-a)/2\n",
-        "          c = c/2\n",
+        "          c = a+ (c-a)/2\n",
-        "          d = d/2\n",
+        "          d = a+ (d-a)/2\n",
         "          continue;\n",
         "\n",
         "        # Rule #2 If point b is less than point c then\n",
@@ -577,9 +569,8 @@
   ],
   "metadata": {
     "colab": {
-      "authorship_tag": "ABX9TyNk5FN4qlw3pk8BwDVWw1jN",
+      "provenance": [],
-      "include_colab_link": true,
+      "include_colab_link": true
-      "provenance": []
     },
     "kernelspec": {
       "display_name": "Python 3",

Notebooks/Chap06/6_4_Momentum.ipynb

@@ -32,7 +32,7 @@
         "\n",
         "This notebook investigates the use of momentum as illustrated in figure 6.7 from the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n",

Notebooks/Chap06/6_5_Adam.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyNFsCOnucz1nQt7PBEnKeTV",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +32,7 @@
         "\n",
         "This notebook investigates the Adam algorithm as illustrated in figure 6.9 from the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -109,8 +108,8 @@
         "    ax.contour(phi0mesh, phi1mesh, loss_function, 20, colors=['#80808080'])\n",
         "    ax.plot(opt_path[0,:], opt_path[1,:],'-', color='#a0d9d3ff')\n",
         "    ax.plot(opt_path[0,:], opt_path[1,:],'.', color='#a0d9d3ff',markersize=10)\n",
-        "    ax.set_xlabel(\"$\\phi_{0}$\")\n",
+        "    ax.set_xlabel(r\"$\\phi_{0}$\")\n",
-        "    ax.set_ylabel(\"$\\phi_{1}$\")\n",
+        "    ax.set_ylabel(r\"$\\phi_{1}$\")\n",
         "    plt.show()"
       ],
       "metadata": {
@@ -169,7 +168,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Because the function changes much faster in $\\phi_1$ than in $\\phi_0$, there is no great step size to choose.  If we set the step size so that it makes sensible progress in the $\\phi_1$ direction, then it takes many iterations to converge.  If we set the step size so that we make sensible progress in the $\\phi_{0}$ direction, then the path oscillates in the $\\phi_1$ direction.  \n",
+        "Because the function changes much faster in $\\phi_1$ than in $\\phi_0$, there is no great step size to choose.  If we set the step size so that it makes sensible progress in the $\\phi_1$ direction, then it takes many iterations to converge.  If we set the step size so that we make sensible progress in the $\\phi_0$ direction, then the path oscillates in the $\\phi_1$ direction.  \n",
         "\n",
         "This motivates Adam.  At the core of Adam is the idea that we should just determine which way is downhill along each axis (i.e. left/right for $\\phi_0$ or up/down for $\\phi_1$) and move a fixed distance in that direction."
       ],
@@ -186,11 +185,11 @@
         "    for c_step in range(n_steps):\n",
         "        # Measure the gradient as in equation 6.13 (first line)\n",
         "        m = get_loss_gradient(grad_path[0,c_step], grad_path[1,c_step]);\n",
-        "        # TO DO -- compute the squared gradient as in equation 6.13 (second line)\n",
+        "        # TODO -- compute the squared gradient as in equation 6.13 (second line)\n",
         "        # Replace this line:\n",
         "        v = np.ones_like(grad_path[:,0])\n",
         "\n",
-        "        # TO DO -- apply the update rule (equation 6.14)\n",
+        "        # TODO -- apply the update rule (equation 6.14)\n",
         "        # Replace this line:\n",
         "        grad_path[:,c_step+1] = grad_path[:,c_step]\n",
         "\n",
@@ -222,7 +221,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "This moves towards the minimum at a sensible speed, but we never actually converge -- the solution just bounces back and forth between the last two points.  To make it converge, we add momentum to both the estimates of the gradient and the pointwise squared gradient.  We also modify the statistics by a factor that depends on the time to make sure the progress is now slow to start with."
+        "This moves towards the minimum at a sensible speed, but we never actually converge -- the solution just bounces back and forth between the last two points.  To make it converge, we add momentum to both the estimates of the gradient and the pointwise squared gradient.  We also modify the statistics by a factor that depends on the time to make sure the progress is not slow to start with."
       ],
       "metadata": {
         "id": "_6KoKBJdGGI4"
@@ -255,7 +254,7 @@
         "        v_tilde = v\n",
         "\n",
         "\n",
-        "        # TO DO -- apply the update rule (equation 6.17)\n",
+        "        # TODO -- apply the update rule (equation 6.17)\n",
         "        # Replace this line:\n",
         "        grad_path[:,c_step+1] = grad_path[:,c_step]\n",
         "\n",

Notebooks/Chap07/7_1_Backpropagation_in_Toy_Model.ipynb

@@ -22,7 +22,7 @@
         "\n",
         "This notebook computes the derivatives of the toy function discussed in section 7.3 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]
@@ -131,7 +131,7 @@
       "source": [
         "beta0 = 1.0; beta1 = 2.0; beta2 = -3.0; beta3 = 0.4\n",
         "omega0 = 0.1; omega1 = -0.4; omega2 = 2.0; omega3 = 3.0\n",
-        "x = 2.3; y =2.0\n",
+        "x = 2.3; y = 2.0\n",
         "l_i_func = loss(x,y,beta0,beta1,beta2,beta3,omega0,omega1,omega2,omega3)\n",
         "print('l_i=%3.3f'%l_i_func)"
       ]
@@ -279,7 +279,7 @@
             "f2: true value = 7.137, your value = 0.000\n",
             "h3: true value = 0.657, your value = 0.000\n",
             "f3: true value = 2.372, your value = 0.000\n",
-            "like original = 0.139, like from forward pass = 0.000\n"
+            "l_i original = 0.139, l_i from forward pass = 0.000\n"
           ]
         }
       ],
@@ -292,7 +292,7 @@
         "print(\"f2: true value = %3.3f, your value = %3.3f\"%(7.137, f2))\n",
         "print(\"h3: true value = %3.3f, your value = %3.3f\"%(0.657, h3))\n",
         "print(\"f3: true value = %3.3f, your value = %3.3f\"%(2.372, f3))\n",
-        "print(\"like original = %3.3f, like from forward pass = %3.3f\"%(l_i_func, l_i))\n"
+        "print(\"l_i original = %3.3f, l_i from forward pass = %3.3f\"%(l_i_func, l_i))\n"
       ]
     },
     {

Notebooks/Chap07/7_2_Backpropagation.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyM2kkHLr00J4Jeypw41sTkQ",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +32,7 @@
         "\n",
         "This notebook runs the backpropagation algorithm on a deep neural network as described in section 7.4 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -68,7 +67,7 @@
         "# Set seed so we always get the same random numbers\n",
         "np.random.seed(0)\n",
         "\n",
-        "# Number of layers\n",
+        "# Number of hidden layers\n",
         "K = 5\n",
         "# Number of neurons per layer\n",
         "D = 6\n",
@@ -115,9 +114,9 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Now let's run our random network.  The weight matrices $\\boldsymbol\\Omega_{1\\ldots K}$ are the entries of the list \"all_weights\" and the biases $\\boldsymbol\\beta_{1\\ldots k}$ are the entries of the list \"all_biases\"\n",
+        "Now let's run our random network.  The weight matrices $\\boldsymbol\\Omega_{0\\ldots K}$ are the entries of the list \"all_weights\" and the biases $\\boldsymbol\\beta_{0\\ldots K}$ are the entries of the list \"all_biases\"\n",
         "\n",
-        "We know that we will need the activations $\\mathbf{f}_{0\\ldots K}$ and the activations $\\mathbf{h}_{1\\ldots K}$ for the forward pass of backpropagation, so we'll store and return these as well.\n"
+        "We know that we will need the preactivations $\\mathbf{f}_{0\\ldots K}$ and the activations $\\mathbf{h}_{1\\ldots K}$ for the forward pass of backpropagation, so we'll store and return these as well.\n"
       ],
       "metadata": {
         "id": "5irtyxnLJSGX"
@@ -132,7 +131,7 @@
         "  K = len(all_weights) -1\n",
         "\n",
         "  # We'll store the pre-activations at each layer in a list \"all_f\"\n",
-        "  # and the activations in a second list[all_h].\n",
+        "  # and the activations in a second list \"all_h\".\n",
         "  all_f = [None] * (K+1)\n",
         "  all_h = [None] * (K+1)\n",
         "\n",
@@ -142,14 +141,14 @@
         "\n",
         "  # Run through the layers, calculating all_f[0...K-1] and all_h[1...K]\n",
         "  for layer in range(K):\n",
-        "      # Update preactivations and activations at this layer according to eqn 7.16\n",
+        "      # Update preactivations and activations at this layer according to eqn 7.17\n",
-        "      # Remmember to use np.matmul for matrrix multiplications\n",
+        "      # Remember to use np.matmul for matrix multiplications\n",
         "      # TODO -- Replace the lines below\n",
         "      all_f[layer] = all_h[layer]\n",
         "      all_h[layer+1] = all_f[layer]\n",
         "\n",
         "  # Compute the output from the last hidden layer\n",
-        "  # TO DO -- Replace the line below\n",
+        "  # TODO -- Replace the line below\n",
         "  all_f[K] = np.zeros_like(all_biases[-1])\n",
         "\n",
         "  # Retrieve the output\n",
@@ -166,7 +165,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Define in input\n",
+        "# Define input\n",
         "net_input = np.ones((D_i,1)) * 1.2\n",
         "# Compute network output\n",
         "net_output, all_f, all_h = compute_network_output(net_input,all_weights, all_biases)\n",
@@ -230,8 +229,8 @@
         "# We'll need the indicator function\n",
         "def indicator_function(x):\n",
         "  x_in = np.array(x)\n",
-        "  x_in[x_in>=0] = 1\n",
+        "  x_in[x_in>0] = 1\n",
-        "  x_in[x_in<0] = 0\n",
+        "  x_in[x_in<=0] = 0\n",
         "  return x_in\n",
         "\n",
         "# Main backward pass routine\n",
@@ -249,23 +248,23 @@
         "\n",
         "  # Now work backwards through the network\n",
         "  for layer in range(K,-1,-1):\n",
-        "    # TODO Calculate the derivatives of the loss with respect to the biases at layer this from all_dl_df[layer]. (eq 7.21)\n",
+        "    # TODO Calculate the derivatives of the loss with respect to the biases at layer from all_dl_df[layer]. (eq 7.22)\n",
         "    # NOTE!  To take a copy of matrix X, use Z=np.array(X)\n",
         "    # REPLACE THIS LINE\n",
         "    all_dl_dbiases[layer] = np.zeros_like(all_biases[layer])\n",
         "\n",
-        "    # TODO Calculate the derivatives of the loss with respect to the weights at layer from all_dl_df[layer] and all_h[layer] (eq 7.22)\n",
+        "    # TODO Calculate the derivatives of the loss with respect to the weights at layer from all_dl_df[layer] and all_h[layer] (eq 7.23)\n",
         "    # Don't forget to use np.matmul\n",
         "    # REPLACE THIS LINE\n",
         "    all_dl_dweights[layer] = np.zeros_like(all_weights[layer])\n",
         "\n",
-        "    # TODO: calculate the derivatives of the loss with respect to the activations from weight and derivatives of next preactivations (second part of last line of eq 7.24)\n",
+        "    # TODO: calculate the derivatives of the loss with respect to the activations from weight and derivatives of next preactivations (second part of last line of eq 7.25)\n",
         "    # REPLACE THIS LINE\n",
         "    all_dl_dh[layer] = np.zeros_like(all_h[layer])\n",
         "\n",
         "\n",
         "    if layer > 0:\n",
-        "      # TODO Calculate the derivatives of the loss with respect to the pre-activation f (use deriv of ReLu function, first part of last line of eq. 7.24)\n",
+        "      # TODO Calculate the derivatives of the loss with respect to the pre-activation f (use derivative of ReLu function, first part of last line of eq. 7.25)\n",
         "      # REPLACE THIS LINE\n",
         "      all_dl_df[layer-1] = np.zeros_like(all_f[layer-1])\n",
         "\n",
@@ -300,7 +299,7 @@
         "delta_fd = 0.000001\n",
         "\n",
         "# Test the dervatives of the bias vectors\n",
-        "for layer in range(K):\n",
+        "for layer in range(K+1):\n",
         "  dl_dbias  = np.zeros_like(all_dl_dbiases[layer])\n",
         "  # For every element in the bias\n",
         "  for row in range(all_biases[layer].shape[0]):\n",
@@ -324,7 +323,7 @@
         "\n",
         "\n",
         "# Test the derivatives of the weights matrices\n",
-        "for layer in range(K):\n",
+        "for layer in range(K+1):\n",
         "  dl_dweight  = np.zeros_like(all_dl_dweights[layer])\n",
         "  # For every element in the bias\n",
         "  for row in range(all_weights[layer].shape[0]):\n",

Notebooks/Chap07/7_3_Initialization.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyNHLXFpiSnUzAbzhtOk+bxu",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +32,7 @@
         "\n",
         "This notebook explores weight initialization in deep neural networks as described in section 7.5 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -117,10 +116,10 @@
         "def compute_network_output(net_input, all_weights, all_biases):\n",
         "\n",
         "  # Retrieve number of layers\n",
-        "  K = len(all_weights) -1\n",
+        "  K = len(all_weights)-1\n",
         "\n",
         "  # We'll store the pre-activations at each layer in a list \"all_f\"\n",
-        "  # and the activations in a second list[all_h].\n",
+        "  # and the activations in a second list \"all_h\".\n",
         "  all_f = [None] * (K+1)\n",
         "  all_h = [None] * (K+1)\n",
         "\n",
@@ -151,7 +150,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Now let's investigate how this the size of the outputs vary as we change the initialization variance:\n"
+        "Now let's investigate how the size of the outputs vary as we change the initialization variance:\n"
       ],
       "metadata": {
         "id": "bIUrcXnOqChl"
@@ -164,7 +163,7 @@
         "K = 5\n",
         "# Number of neurons per layer\n",
         "D = 8\n",
-        "  # Input layer\n",
+        "# Input layer\n",
         "D_i = 1\n",
         "# Output layer\n",
         "D_o = 1\n",
@@ -177,7 +176,7 @@
         "data_in = np.random.normal(size=(1,n_data))\n",
         "net_output, all_f, all_h = compute_network_output(data_in, all_weights, all_biases)\n",
         "\n",
-        "for layer in range(K):\n",
+        "for layer in range(1,K+1):\n",
         "  print(\"Layer %d, std of hidden units = %3.3f\"%(layer, np.std(all_h[layer])))"
       ],
       "metadata": {
@@ -192,11 +191,11 @@
         "# You can see that the values of the hidden units are increasing on average (the variance is across all hidden units at the layer\n",
         "# and the 1000 training examples\n",
         "\n",
-        "# TO DO\n",
+        "# TODO\n",
         "# Change this to 50 layers with 80 hidden units per layer\n",
         "\n",
-        "# TO DO\n",
+        "# TODO\n",
-        "# Now experiment with sigma_sq_omega to try to stop the variance of the forward computation explode"
+        "# Now experiment with sigma_sq_omega to try to stop the variance of the forward computation exploding"
       ],
       "metadata": {
         "id": "VL_SO4tar3DC"
@@ -249,6 +248,9 @@
         "\n",
         "# Main backward pass routine\n",
         "def backward_pass(all_weights, all_biases, all_f, all_h, y):\n",
+        "  # Retrieve number of layers\n",
+        "  K = len(all_weights) - 1\n",
+        "\n",
         "  # We'll store the derivatives dl_dweights and dl_dbiases in lists as well\n",
         "  all_dl_dweights = [None] * (K+1)\n",
         "  all_dl_dbiases = [None] * (K+1)\n",
@@ -297,7 +299,7 @@
         "K = 5\n",
         "# Number of neurons per layer\n",
         "D = 8\n",
-        "  # Input layer\n",
+        "# Input layer\n",
         "D_i = 1\n",
         "# Output layer\n",
         "D_o = 1\n",
@@ -335,13 +337,13 @@
     {
       "cell_type": "code",
       "source": [
-        "# You can see that the values of the hidden units are increasing on average (the variance is across all hidden units at the layer\n",
+        "# You can see that the gradients of the hidden units are increasing on average (the standard deviation is across all hidden units at the layer\n",
-        "# and the 1000 training examples\n",
+        "# and the 100 training examples\n",
         "\n",
-        "# TO DO\n",
+        "# TODO\n",
         "# Change this to 50 layers with 80 hidden units per layer\n",
         "\n",
-        "# TO DO\n",
+        "# TODO\n",
         "# Now experiment with sigma_sq_omega to try to stop the variance of the gradients exploding\n"
       ],
       "metadata": {

Notebooks/Chap08/8_1_MNIST_1D_Performance.ipynb

@@ -1,28 +1,10 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "gpuType": "T4",
-      "authorship_tag": "ABX9TyOuKMUcKfOIhIL2qTX9jJCy",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    },
-    "accelerator": "GPU"
-  },
   "cells": [
     {
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap08/8_1_MNIST_1D_Performance.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
@@ -30,33 +12,38 @@
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "L6chybAVFJW2"
+      },
       "source": [
         "# **Notebook 8.1: MNIST_1D_Performance**\n",
         "\n",
         "This notebook runs a simple neural network on the MNIST1D dataset as in figure 8.2a. It uses code from https://github.com/greydanus/mnist1d to generate the data.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "L6chybAVFJW2"
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
-        "!git clone https://github.com/greydanus/mnist1d"
-      ],
       "metadata": {
         "id": "ifVjS4cTOqKz"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "# Run this if you're in a Colab to install MNIST 1D repository\n",
+        "%pip install git+https://github.com/greydanus/mnist1d"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "qyE7G1StPIqO"
+      },
+      "outputs": [],
       "source": [
         "import torch, torch.nn as nn\n",
         "from torch.utils.data import TensorDataset, DataLoader\n",
@@ -64,47 +51,47 @@
         "import numpy as np\n",
         "import matplotlib.pyplot as plt\n",
         "import mnist1d"
-      ],
+      ]
-      "metadata": {
-        "id": "qyE7G1StPIqO"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "Let's generate a training and test dataset using the MNIST1D code.  The dataset gets saved as a .pkl file so it doesn't have to be regenerated each time."
-      ],
       "metadata": {
         "id": "F7LNq72SP6jO"
-      }
+      },
+      "source": [
+        "Let's generate a training and test dataset using the MNIST1D code.  The dataset gets saved as a .pkl file so it doesn't have to be regenerated each time."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "YLxf7dJfPaqw"
+      },
+      "outputs": [],
       "source": [
         "args = mnist1d.data.get_dataset_args()\n",
-        "data = mnist1d.data.get_dataset(args, path='./sample_data/mnist1d_data.pkl', download=False, regenerate=False)\n",
+        "data = mnist1d.data.get_dataset(args, path='./mnist1d_data.pkl', download=False, regenerate=False)\n",
         "\n",
         "# The training and test input and outputs are in\n",
         "# data['x'], data['y'], data['x_test'], and data['y_test']\n",
         "print(\"Examples in training set: {}\".format(len(data['y'])))\n",
         "print(\"Examples in test set: {}\".format(len(data['y_test'])))\n",
         "print(\"Length of each example: {}\".format(data['x'].shape[-1]))"
-      ],
+      ]
-      "metadata": {
-        "id": "YLxf7dJfPaqw"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "FxaB5vc0uevl"
+      },
+      "outputs": [],
       "source": [
         "D_i = 40    # Input dimensions\n",
         "D_k = 100   # Hidden dimensions\n",
         "D_o = 10    # Output dimensions\n",
-        "# TO DO:\n",
+        "# TODO:\n",
         "# Define a model with two hidden layers of size 100\n",
         "# And ReLU activations between them\n",
         "# Replace this line (see Figure 7.8 of book for help):\n",
@@ -112,7 +99,7 @@
         "\n",
         "\n",
         "def weights_init(layer_in):\n",
-        "  # TO DO:\n",
+        "  # TODO:\n",
         "  # Initialize the parameters with He initialization\n",
         "  # Replace this line (see figure 7.8 of book for help)\n",
         "  print(\"Initializing layer\")\n",
@@ -120,15 +107,15 @@
         "\n",
         "# Call the function you just defined\n",
         "model.apply(weights_init)\n"
-      ],
+      ]
-      "metadata": {
-        "id": "FxaB5vc0uevl"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "_rX6N3VyyQTY"
+      },
+      "outputs": [],
       "source": [
         "# choose cross entropy loss function (equation 5.24)\n",
         "loss_function = torch.nn.CrossEntropyLoss()\n",
@@ -136,11 +123,10 @@
         "optimizer = torch.optim.SGD(model.parameters(), lr = 0.05, momentum=0.9)\n",
         "# object that decreases learning rate by half every 10 epochs\n",
         "scheduler = StepLR(optimizer, step_size=10, gamma=0.5)\n",
-        "# create 100 dummy data points and store in data loader class\n",
         "x_train = torch.tensor(data['x'].astype('float32'))\n",
-        "y_train = torch.tensor(data['y'].transpose().astype('long'))\n",
+        "y_train = torch.tensor(data['y'].transpose().astype('int64'))\n",
         "x_test= torch.tensor(data['x_test'].astype('float32'))\n",
-        "y_test = torch.tensor(data['y_test'].astype('long'))\n",
+        "y_test = torch.tensor(data['y_test'].astype('int64'))\n",
         "\n",
         "# load the data into a class that creates the batches\n",
         "data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=100, shuffle=True, worker_init_fn=np.random.seed(1))\n",
@@ -185,15 +171,15 @@
         "\n",
         "  # tell scheduler to consider updating learning rate\n",
         "  scheduler.step()"
-      ],
+      ]
-      "metadata": {
-        "id": "_rX6N3VyyQTY"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "yI-l6kA_EH9G"
+      },
+      "outputs": [],
       "source": [
         "# Plot the results\n",
         "fig, ax = plt.subplots()\n",
@@ -214,25 +200,38 @@
         "ax.set_title('Train loss %3.2f, Test loss %3.2f'%(losses_train[-1],losses_test[-1]))\n",
         "ax.legend()\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "yI-l6kA_EH9G"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "q-yT6re6GZS4"
+      },
       "source": [
-        "**TO DO**\n",
+        "**TODO**\n",
         "\n",
         "Play with the model -- try changing the number of layers, hidden units, learning rate, batch size, momentum or anything else you like.  See if you can improve the test results.\n",
         "\n",
         "Is it a good idea to optimize the hyperparameters in this way?  Will the final result be a good estimate of the true test performance?"
-      ],
+      ]
-      "metadata": {
-        "id": "q-yT6re6GZS4"
-      }
     }
-  ]
+  ],
+  "metadata": {
+    "accelerator": "GPU",
+    "colab": {
+      "authorship_tag": "ABX9TyOuKMUcKfOIhIL2qTX9jJCy",
+      "gpuType": "T4",
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap08/8_2_Bias_Variance_Trade_Off.ipynb

@@ -32,7 +32,7 @@
         "\n",
         "This notebook investigates the bias-variance trade-off for the toy model used throughout chapter 8 and reproduces the bias/variance trade off curves seen in figure 8.9.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -92,7 +92,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Draw the fitted function, together win uncertainty used to generate points\n",
+        "# Draw the fitted function, together with uncertainty used to generate points\n",
         "def plot_function(x_func, y_func, x_data=None,y_data=None, x_model = None, y_model =None, sigma_func = None, sigma_model=None):\n",
         "\n",
         "    fig,ax = plt.subplots()\n",
@@ -203,7 +203,7 @@
         "# Closed form solution\n",
         "beta, omega = fit_model_closed_form(x_data,y_data,n_hidden=3)\n",
         "\n",
-        "# Get prediction for model across graph grange\n",
+        "# Get prediction for model across graph range\n",
         "x_model = np.linspace(0,1,100);\n",
         "y_model = network(x_model, beta, omega)\n",
         "\n",
@@ -268,7 +268,7 @@
         "mean_model, std_model = get_model_mean_variance(n_data, n_datasets, n_hidden, sigma_func) ;\n",
         "\n",
         "# Plot the results\n",
-        "plot_function(x_func, y_func, x_data,y_data, x_model, mean_model, sigma_model=std_model)"
+        "plot_function(x_func, y_func, x_model=x_model, y_model=mean_model, sigma_model=std_model)"
       ],
       "metadata": {
         "id": "Wxk64t2SoX9c"
@@ -302,7 +302,7 @@
         "sigma_func = 0.3\n",
         "n_hidden = 5\n",
         "\n",
-        "# Set random seed so that get same result every time\n",
+        "# Set random seed so that we get the same result every time\n",
         "np.random.seed(1)\n",
         "\n",
         "for c_hidden in range(len(hidden_variables)):\n",

Notebooks/Chap08/8_3_Double_Descent.ipynb

@@ -5,7 +5,6 @@
     "colab": {
       "provenance": [],
       "gpuType": "T4",
-      "authorship_tag": "ABX9TyN/KUpEObCKnHZ/4Onp5sHG",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -37,7 +36,7 @@
         "\n",
         "It uses the MNIST-1D database which can be found at https://github.com/greydanus/mnist1d\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -48,8 +47,8 @@
     {
       "cell_type": "code",
       "source": [
-        "# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
+        "# Run this if you're in a Colab to install MNIST 1D repository\n",
-        "!git clone https://github.com/greydanus/mnist1d"
+        "!pip install git+https://github.com/greydanus/mnist1d"
       ],
       "metadata": {
         "id": "fn9BP5N5TguP"
@@ -100,7 +99,7 @@
         "# data['x'], data['y'], data['x_test'], and data['y_test']\n",
         "print(\"Examples in training set: {}\".format(len(data['y'])))\n",
         "print(\"Examples in test set: {}\".format(len(data['y_test'])))\n",
-        "print(\"Length of each example: {}\".format(data['x'].shape[-1]))"
+        "print(\"Dimensionality of each example: {}\".format(data['x'].shape[-1]))"
       ],
       "metadata": {
         "id": "PW2gyXL5UkLU"
@@ -124,7 +123,7 @@
         "  D_k = n_hidden   # Hidden dimensions\n",
         "  D_o = 10    # Output dimensions\n",
         "\n",
-        "  # Define a model with two hidden layers of size 100\n",
+        "  # Define a model with two hidden layers\n",
         "  # And ReLU activations between them\n",
         "  model = nn.Sequential(\n",
         "  nn.Linear(D_i, D_k),\n",
@@ -148,7 +147,7 @@
     {
       "cell_type": "code",
       "source": [
-        "def fit_model(model, data):\n",
+        "def fit_model(model, data, n_epoch):\n",
         "\n",
         "  # choose cross entropy loss function (equation 5.24)\n",
         "  loss_function = torch.nn.CrossEntropyLoss()\n",
@@ -157,7 +156,6 @@
         "  optimizer = torch.optim.SGD(model.parameters(), lr = 0.01, momentum=0.9)\n",
         "\n",
         "\n",
-        "  # create 100 dummy data points and store in data loader class\n",
         "  x_train = torch.tensor(data['x'].astype('float32'))\n",
         "  y_train = torch.tensor(data['y'].transpose().astype('long'))\n",
         "  x_test= torch.tensor(data['x_test'].astype('float32'))\n",
@@ -166,9 +164,6 @@
         "  # load the data into a class that creates the batches\n",
         "  data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=100, shuffle=True, worker_init_fn=np.random.seed(1))\n",
         "\n",
-        "  # loop over the dataset n_epoch times\n",
-        "  n_epoch = 1000\n",
-        "\n",
         "  for epoch in range(n_epoch):\n",
         "    # loop over batches\n",
         "    for i, batch in enumerate(data_loader):\n",
@@ -205,12 +200,24 @@
       "execution_count": null,
       "outputs": []
     },
+    {
+      "cell_type": "code",
+      "source": [
+        "def count_parameters(model):\n",
+        "    return sum(p.numel() for p in model.parameters() if p.requires_grad)"
+      ],
+      "metadata": {
+        "id": "AQNCmFNV6JpV"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
     {
       "cell_type": "markdown",
       "source": [
         "The following code produces the double descent curve by training the model with different numbers of hidden units and plotting the test error.\n",
         "\n",
-        "TO DO:\n",
+        "TODO:\n",
         "\n",
         "*Before* you run the code, and considering that there are 4000 training examples predict:<br>\n",
         "\n",
@@ -228,19 +235,27 @@
         "# This code will take a while (~30 mins on GPU) to run!  Go and make a cup of coffee!\n",
         "\n",
         "hidden_variables = np.array([2,4,6,8,10,14,18,22,26,30,35,40,45,50,55,60,70,80,90,100,120,140,160,180,200,250,300,400]) ;\n",
+        "\n",
         "errors_train_all = np.zeros_like(hidden_variables)\n",
         "errors_test_all = np.zeros_like(hidden_variables)\n",
+        "total_weights_all = np.zeros_like(hidden_variables)\n",
+        "\n",
+        "# loop over the dataset n_epoch times\n",
+        "n_epoch = 1000\n",
         "\n",
         "# For each hidden variable size\n",
         "for c_hidden in range(len(hidden_variables)):\n",
         "    print(f'Training model with {hidden_variables[c_hidden]:3d} hidden variables')\n",
         "    # Get a model\n",
         "    model = get_model(hidden_variables[c_hidden]) ;\n",
+        "    # Count and store number of weights\n",
+        "    total_weights_all[c_hidden] = count_parameters(model)\n",
         "    # Train the model\n",
-        "    errors_train, errors_test = fit_model(model, data)\n",
+        "    errors_train, errors_test = fit_model(model, data, n_epoch)\n",
         "    # Store the results\n",
         "    errors_train_all[c_hidden] = errors_train\n",
-        "    errors_test_all[c_hidden]= errors_test"
+        "    errors_test_all[c_hidden]= errors_test\n",
+        "\n"
       ],
       "metadata": {
         "id": "K4OmBZGHWXpk"
@@ -251,12 +266,29 @@
     {
       "cell_type": "code",
       "source": [
+        "import matplotlib.pyplot as plt\n",
+        "import numpy as np\n",
+        "\n",
+        "# Assuming data['y'] is available and contains the training examples\n",
+        "num_training_examples = len(data['y'])\n",
+        "\n",
+        "# Find the index where total_weights_all is closest to num_training_examples\n",
+        "closest_index = np.argmin(np.abs(np.array(total_weights_all) - num_training_examples))\n",
+        "\n",
+        "# Get the corresponding value of hidden variables\n",
+        "hidden_variable_at_num_training_examples = hidden_variables[closest_index]\n",
+        "\n",
         "# Plot the results\n",
         "fig, ax = plt.subplots()\n",
-        "ax.plot(hidden_variables, errors_train_all,'r-',label='train')\n",
+        "ax.plot(hidden_variables, errors_train_all, 'r-', label='train')\n",
-        "ax.plot(hidden_variables, errors_test_all,'b-',label='test')\n",
+        "ax.plot(hidden_variables, errors_test_all, 'b-', label='test')\n",
-        "ax.set_ylim(0,100);\n",
+        "\n",
-        "ax.set_xlabel('No hidden variables'); ax.set_ylabel('Error')\n",
+        "# Add a vertical line at the point where total weights equal the number of training examples\n",
+        "ax.axvline(x=hidden_variable_at_num_training_examples, color='g', linestyle='--', label='N(weights) = N(train)')\n",
+        "\n",
+        "ax.set_ylim(0, 100)\n",
+        "ax.set_xlabel('No. hidden variables')\n",
+        "ax.set_ylabel('Error')\n",
         "ax.legend()\n",
         "plt.show()\n"
       ],
@@ -265,6 +297,24 @@
       },
       "execution_count": null,
       "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [],
+      "metadata": {
+        "id": "KT4X8_hE5NFb"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [],
+      "metadata": {
+        "id": "iGKZSfVF2r4z"
+      },
+      "execution_count": null,
+      "outputs": []
     }
   ]
 }

Notebooks/Chap08/8_4_High_Dimensional_Spaces.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates the strange properties of high-dimensional spaces as discussed in the notes at the end of chapter 8.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -134,7 +134,7 @@
       "source": [
         "# Volume of a hypersphere\n",
         "\n",
-        "In the second part of this notebook we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius.  Note that you you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
+        "In the second part of this notebook we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius.  Note that you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
       ],
       "metadata": {
         "id": "b2FYKV1SL4Z7"
@@ -224,7 +224,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "You should see see that by the time we get to 300 dimensions most of the volume is in the outer 1 percent. <br><br>\n",
+        "You should see that by the time we get to 300 dimensions most of the volume is in the outer 1 percent. <br><br>\n",
         "\n",
         "The conclusion of all of this is that in high dimensions you should be sceptical of your intuitions about how things work.  I have tried to visualize many things in one or two dimensions in the book, but you should also be sceptical about these visualizations!"
       ],

Notebooks/Chap09/9_1_L2_Regularization.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates adding L2 regularization to the loss function for the Gabor model as in figure 9.1.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ],
@@ -178,7 +178,7 @@
         "\n",
         "def draw_loss_function(compute_loss, data,  model, my_colormap, phi_iters = None):\n",
         "\n",
-        "  # Make grid of intercept/slope values to plot\n",
+        "  # Make grid of offset/frequency values to plot\n",
         "  offsets_mesh, freqs_mesh = np.meshgrid(np.arange(-10,10.0,0.1), np.arange(2.5,22.5,0.1))\n",
         "  loss_mesh = np.zeros_like(freqs_mesh)\n",
         "  # Compute loss for every set of parameters\n",
@@ -304,7 +304,7 @@
         "for c_step in range (n_steps):\n",
         "  # Do gradient descent step\n",
         "  phi_all[:,c_step+1:c_step+2] = gradient_descent_step(phi_all[:,c_step:c_step+1],data, model)\n",
-        "  # Measure loss and draw model every 4th step\n",
+        "  # Measure loss and draw model every 8th step\n",
         "  if c_step % 8 == 0:\n",
         "    loss =  compute_loss(data[0,:], data[1,:], model, phi_all[:,c_step+1:c_step+2])\n",
         "    draw_model(data,model,phi_all[:,c_step+1], \"Iteration %d, loss = %f\"%(c_step+1,loss))\n",
@@ -369,7 +369,7 @@
         "# Code to draw the regularization function\n",
         "def draw_reg_function():\n",
         "\n",
-        "  # Make grid of intercept/slope values to plot\n",
+        "  # Make grid of offset/frequency values to plot\n",
         "  offsets_mesh, freqs_mesh = np.meshgrid(np.arange(-10,10.0,0.1), np.arange(2.5,22.5,0.1))\n",
         "  loss_mesh = np.zeros_like(freqs_mesh)\n",
         "  # Compute loss for every set of parameters\n",
@@ -399,7 +399,7 @@
         "# Code to draw loss function with regularization\n",
         "def draw_loss_function_reg(data,  model, lambda_, my_colormap, phi_iters = None):\n",
         "\n",
-        "  # Make grid of intercept/slope values to plot\n",
+        "  # Make grid of offset/frequency values to plot\n",
         "  offsets_mesh, freqs_mesh = np.meshgrid(np.arange(-10,10.0,0.1), np.arange(2.5,22.5,0.1))\n",
         "  loss_mesh = np.zeros_like(freqs_mesh)\n",
         "  # Compute loss for every set of parameters\n",
@@ -512,7 +512,7 @@
         "for c_step in range (n_steps):\n",
         "  # Do gradient descent step\n",
         "  phi_all[:,c_step+1:c_step+2] = gradient_descent_step2(phi_all[:,c_step:c_step+1],lambda_, data, model)\n",
-        "  # Measure loss and draw model every 4th step\n",
+        "  # Measure loss and draw model every 8th step\n",
         "  if c_step % 8 == 0:\n",
         "    loss =  compute_loss2(data[0,:], data[1,:], model, phi_all[:,c_step+1:c_step+2], lambda_)\n",
         "    draw_model(data,model,phi_all[:,c_step+1], \"Iteration %d, loss = %f\"%(c_step+1,loss))\n",
@@ -528,7 +528,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "You should see that the gradient descent algorithm now finds the correct minimum.  By applying a tiny bit of domain knowledge (the parameter phi0 tends to be near zero and the parameters phi1 tends to be near 12.5), we get a better solution.  However, the cost is that this solution is slightly biased towards this prior knowledge."
+        "You should see that the gradient descent algorithm now finds the correct minimum.  By applying a tiny bit of domain knowledge (the parameter phi0 tends to be near zero and the parameter phi1 tends to be near 12.5), we get a better solution.  However, the cost is that this solution is slightly biased towards this prior knowledge."
       ],
       "metadata": {
         "id": "wrszSLrqZG4k"

Notebooks/Chap09/9_2_Implicit_Regularization.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOR3WOJwfTlMD8eOLsPfPrz",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +32,7 @@
         "\n",
         "This notebook investigates how the finite step sizes in gradient descent cause the trajectory to deviate and how this can be explained by adding an implicit regularization term.  It recreates figure 9.3 from the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ],
@@ -140,7 +139,7 @@
         "    fig.set_size_inches(7,7)\n",
         "    ax.contourf(phi0mesh, phi1mesh, loss_function, 256, cmap=my_colormap);\n",
         "    ax.contour(phi0mesh, phi1mesh, loss_function, 20, colors=['#80808080'])\n",
-        "    ax.set_xlabel('$\\phi_{0}$'); ax.set_ylabel('$\\phi_{1}$')\n",
+        "    ax.set_xlabel(r'$\\phi_{0}$'); ax.set_ylabel(r'$\\phi_{1}$')\n",
         "\n",
         "    if grad_path_typical_lr is not None:\n",
         "        ax.plot(grad_path_typical_lr[0,:], grad_path_typical_lr[1,:],'ro-')\n",

Notebooks/Chap09/9_3_Ensembling.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates how ensembling can improve the performance of models. We'll work with the simplified neural network model (figure 8.4 of book) which we can fit in closed form, and so we can eliminate any errors due to not finding the global maximum.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ],
@@ -52,7 +52,7 @@
         "# import libraries\n",
         "import numpy as np\n",
         "import matplotlib.pyplot as plt\n",
-        "# Define seed so get same results each time\n",
+        "# Define seed to get same results each time\n",
         "np.random.seed(1)"
       ]
     },
@@ -80,7 +80,7 @@
         "    for i in range(n_data):\n",
         "        x[i] = np.random.uniform(i/n_data, (i+1)/n_data, 1)\n",
         "\n",
-        "    # y value from running through functoin and adding noise\n",
+        "    # y value from running through function and adding noise\n",
         "    y = np.ones(n_data)\n",
         "    for i in range(n_data):\n",
         "        y[i] = true_function(x[i])\n",
@@ -96,7 +96,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Draw the fitted function, together win uncertainty used to generate points\n",
+        "# Draw the fitted function, together with uncertainty used to generate points\n",
         "def plot_function(x_func, y_func, x_data=None,y_data=None, x_model = None, y_model =None, sigma_func = None, sigma_model=None):\n",
         "\n",
         "    fig,ax = plt.subplots()\n",
@@ -137,7 +137,7 @@
         "n_data = 15\n",
         "x_data,y_data = generate_data(n_data, sigma_func)\n",
         "\n",
-        "# Plot the functinon, data and uncertainty\n",
+        "# Plot the function, data and uncertainty\n",
         "plot_function(x_func, y_func, x_data, y_data, sigma_func=sigma_func)"
       ],
       "metadata": {
@@ -216,7 +216,7 @@
         "# Closed form solution\n",
         "beta, omega = fit_model_closed_form(x_data,y_data,n_hidden=14)\n",
         "\n",
-        "# Get prediction for model across graph grange\n",
+        "# Get prediction for model across graph range\n",
         "x_model = np.linspace(0,1,100);\n",
         "y_model = network(x_model, beta, omega)\n",
         "\n",
@@ -297,7 +297,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Plot the median of the results\n",
+        "# Plot the mean of the results\n",
         "# TODO -- find the mean prediction\n",
         "# Replace this line\n",
         "y_model_mean = all_y_model[0,:]\n",

Notebooks/Chap09/9_4_Bayesian_Approach.ipynb

@@ -1,18 +1,16 @@
 {
   "cells": [
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "colab_type": "text",
+        "id": "view-in-github",
-        "id": "view-in-github"
+        "colab_type": "text"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap09/9_4_Bayesian_Approach.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "el8l05WQEO46"
@@ -22,7 +20,7 @@
         "\n",
         "This notebook investigates the Bayesian approach to model fitting and reproduces figure 9.11 from the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ]
@@ -38,7 +36,7 @@
         "# import libraries\n",
         "import numpy as np\n",
         "import matplotlib.pyplot as plt\n",
-        "# Define seed so get same results each time\n",
+        "# Define seed to get same results each time\n",
         "np.random.seed(1)"
       ]
     },
@@ -87,7 +85,7 @@
       },
       "outputs": [],
       "source": [
-        "# Draw the fitted function, together win uncertainty used to generate points\n",
+        "# Draw the fitted function, together with uncertainty used to generate points\n",
         "def plot_function(x_func, y_func, x_data=None,y_data=None, x_model = None, y_model =None, sigma_func = None, sigma_model=None):\n",
         "\n",
         "    fig,ax = plt.subplots()\n",
@@ -159,7 +157,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "i8T_QduzeBmM"
@@ -195,7 +192,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "JojV6ueRk49G"
@@ -211,7 +207,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "YX0O_Ciwp4W1"
@@ -225,7 +220,7 @@
         " &\\propto&\\text{Norm}_{\\boldsymbol\\phi}\\biggl[\\frac{1}{\\sigma^2}\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\biggr].\n",
         "\\end{align}\n",
         "\n",
-        "In fact, since this already a normal distribution, the constant of proportionality must be one and we can write\n",
+        "In fact, since this is already a normal distribution, the constant of proportionality must be one and we can write\n",
         "\n",
         "\\begin{align}\n",
         " Pr(\\boldsymbol\\phi|\\{\\mathbf{x}_{i},\\mathbf{y}_{i}\\}) &=& \\text{Norm}_{\\boldsymbol\\phi}\\biggl[\\frac{1}{\\sigma^2}\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\biggr].\n",
@@ -277,7 +272,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "GjPnlG4q0UFK"
@@ -334,7 +328,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "GiNg5EroUiUb"
@@ -343,17 +336,16 @@
         "Now we need to perform inference for a new data points $\\mathbf{x}^*$ with corresponding hidden values $\\mathbf{h}^*$.  Instead of having a single estimate of the parameters, we have a distribution over the possible parameters.  So we marginalize (integrate) over this distribution to account for all possible values:\n",
         "\n",
         "\\begin{align}\n",
-        "Pr(y^*|\\mathbf{x}^*)  &=& \\int Pr(y^{*}|\\mathbf{x}^*,\\boldsymbol\\phi)Pr(\\boldsymbol\\phi|\\{\\mathbf{x}_{i},\\mathbf{y}_{i}\\}) d\\boldsymbol\\phi\\\\\n",
+        "Pr(y^*|\\mathbf{x}^*)  &= \\int Pr(y^{*}|\\mathbf{x}^*,\\boldsymbol\\phi)Pr(\\boldsymbol\\phi|\\{\\mathbf{x}_{i},\\mathbf{y}_{i}\\}) d\\boldsymbol\\phi\\\\\n",
-        "&=& \\int \\text{Norm}_{y^*}\\bigl[[\\mathbf{h}^{*T},1]\\boldsymbol\\phi,\\sigma^2\\bigr]\\cdot\\text{Norm}_{\\boldsymbol\\phi}\\biggl[\\frac{1}{\\sigma^2}\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\biggr]d\\boldsymbol\\phi\\\\\n",
+        "&= \\int \\text{Norm}_{y^*}\\bigl[[\\mathbf{h}^{*T},1]\\boldsymbol\\phi,\\sigma^2\\bigr]\\cdot\\text{Norm}_{\\boldsymbol\\phi}\\biggl[\\frac{1}{\\sigma^2}\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\biggr]d\\boldsymbol\\phi\\\\\n",
-        "&=& \\text{Norm}_{y^*}\\biggl[\\frac{1}{\\sigma^2} [\\mathbf{h}^{*T},1]\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},  [\\mathbf{h}^{*T},1]\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\n",
+        "&= \\text{Norm}_{y^*}\\biggl[\\frac{1}{\\sigma^2} [\\mathbf{h}^{*T},1]\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},  [\\mathbf{h}^{*T},1]\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\n",
-        "[\\mathbf{h}^*;1]\\biggr]\n",
+        "[\\mathbf{h}^*;1]\\biggr],\n",
         "\\end{align}\n",
         "\n",
+        "where the notation $[\\mathbf{h}^{*T},1]$ is a row vector containing $\\mathbf{h}^{T}$ with a one appended to the end and $[\\mathbf{h};1 ]$ is a column vector containing $\\mathbf{h}$ with a one appended to the end.\n",
         "\n",
         "\n",
-        "\n",
+        "To compute this, we reformulated the integrand using the relations from appendices C.3.3 and C.3.4 as the product of a normal distribution in $\\boldsymbol\\phi$ and a constant with respect\n",
-        "To compute this, we reformulated the integrand using the relations from appendices\n",
-        "C.3.3 and C.3.4 as the product of a normal distribution in $\\boldsymbol\\phi$ and a constant with respect\n",
         "to $\\boldsymbol\\phi$. The integral of the normal distribution must be one, and so the final result is just the constant. This constant is itself a normal distribution in $y^*$. <br>\n",
         "\n",
         "If you feel so inclined you can work through the math of this yourself.\n",
@@ -404,7 +396,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "8Hcbe_16sK0F"
@@ -419,9 +410,8 @@
   ],
   "metadata": {
     "colab": {
-      "authorship_tag": "ABX9TyMB8B4269DVmrcLoCWrhzKF",
+      "provenance": [],
-      "include_colab_link": true,
+      "include_colab_link": true
-      "provenance": []
     },
     "kernelspec": {
       "display_name": "Python 3",

Notebooks/Chap09/9_5_Augmentation.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyM38ZVBK4/xaHk5Ys5lF6dN",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +32,7 @@
         "\n",
         "This notebook investigates data augmentation for the MNIST-1D model.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ],
@@ -44,8 +43,8 @@
     {
       "cell_type": "code",
       "source": [
-        "# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
+        "# Run this if you're in a Colab to install MNIST 1D repository\n",
-        "!git clone https://github.com/greydanus/mnist1d"
+        "!pip install git+https://github.com/greydanus/mnist1d"
       ],
       "metadata": {
         "id": "syvgxgRr3myY"
@@ -95,7 +94,7 @@
         "D_k = 200   # Hidden dimensions\n",
         "D_o = 10    # Output dimensions\n",
         "\n",
-        "# Define a model with two hidden layers of size 100\n",
+        "# Define a model with two hidden layers of size 200\n",
         "# And ReLU activations between them\n",
         "model = nn.Sequential(\n",
         "nn.Linear(D_i, D_k),\n",
@@ -108,10 +107,7 @@
         "  # Initialize the parameters with He initialization\n",
         "  if isinstance(layer_in, nn.Linear):\n",
         "    nn.init.kaiming_uniform_(layer_in.weight)\n",
-        "    layer_in.bias.data.fill_(0.0)\n",
+        "    layer_in.bias.data.fill_(0.0)\n"
-        "\n",
-        "# Call the function you just defined\n",
-        "model.apply(weights_init)"
       ],
       "metadata": {
         "id": "JfIFWFIL33eF"
@@ -186,7 +182,7 @@
         "ax.plot(errors_test,'b-',label='test')\n",
         "ax.set_ylim(0,100); ax.set_xlim(0,n_epoch)\n",
         "ax.set_xlabel('Epoch'); ax.set_ylabel('Error')\n",
-        "ax.set_title('TrainError %3.2f, Test Error %3.2f'%(errors_train[-1],errors_test[-1]))\n",
+        "ax.set_title('Train Error %3.2f, Test Error %3.2f'%(errors_train[-1],errors_test[-1]))\n",
         "ax.legend()\n",
         "plt.show()"
       ],
@@ -233,7 +229,7 @@
       "cell_type": "code",
       "source": [
         "n_data_orig = data['x'].shape[0]\n",
-        "# We'll double the amount o fdata\n",
+        "# We'll double the amount of data\n",
         "n_data_augment = n_data_orig+4000\n",
         "augmented_x = np.zeros((n_data_augment, D_i))\n",
         "augmented_y = np.zeros(n_data_augment)\n",

Notebooks/Chap10/10_1_1D_Convolution.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates 1D convolutional layers.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ],

Notebooks/Chap10/10_2_Convolution_for_MNIST_1D.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyNJodaaCLMRWL9vTl8B/iLI",
+      "authorship_tag": "ABX9TyNb46PJB/CC1pcHGfjpUUZg",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates a 1D convolutional network for MNIST-1D as in figure 10.7 and 10.8a.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n"
@@ -45,8 +45,8 @@
     {
       "cell_type": "code",
       "source": [
-        "# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
+        "# Run this if you're in a Colab to install MNIST 1D repository\n",
-        "!git clone https://github.com/greydanus/mnist1d"
+        "!pip install git+https://github.com/greydanus/mnist1d"
       ],
       "metadata": {
         "id": "D5yLObtZCi9J"

Notebooks/Chap10/10_3_2D_Convolution.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates the 2D convolution operation.  It asks you to hand code the convolution so we can be sure that we are computing the same thing as in PyTorch.  The next notebook uses the convolutional layers in PyTorch directly.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],

Notebooks/Chap10/10_4_Downsampling_and_Upsampling.ipynb

@@ -31,9 +31,9 @@
       "source": [
         "# **Notebook 10.4: Downsampling and Upsampling**\n",
         "\n",
-        "This notebook investigates the down sampling and downsampling methods discussed in section 10.4 of the book.\n",
+        "This notebook investigates the upsampling and downsampling methods discussed in section 10.4 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ],
@@ -73,7 +73,7 @@
       "source": [
         "def subsample(x_in):\n",
         "  x_out = np.zeros(( int(np.ceil(x_in.shape[0]/2)), int(np.ceil(x_in.shape[1]/2)) ))\n",
-        "  # TO DO -- write the subsampling routine\n",
+        "  # TODO -- write the subsampling routine\n",
         "  # Replace this line\n",
         "  x_out = x_out\n",
         "\n",
@@ -159,7 +159,7 @@
         "# Now let's try max-pooling\n",
         "def maxpool(x_in):\n",
         "  x_out = np.zeros(( int(np.floor(x_in.shape[0]/2)), int(np.floor(x_in.shape[1]/2)) ))\n",
-        "  # TO DO -- write the maxpool routine\n",
+        "  # TODO -- write the maxpool routine\n",
         "  # Replace this line\n",
         "  x_out = x_out\n",
         "\n",
@@ -230,7 +230,7 @@
         "# Finally, let's try mean pooling\n",
         "def meanpool(x_in):\n",
         "  x_out = np.zeros(( int(np.floor(x_in.shape[0]/2)), int(np.floor(x_in.shape[1]/2)) ))\n",
-        "  # TO DO -- write the meanpool routine\n",
+        "  # TODO -- write the meanpool routine\n",
         "  # Replace this line\n",
         "  x_out = x_out\n",
         "\n",
@@ -301,7 +301,7 @@
       "cell_type": "code",
       "source": [
         "# Define 2 by 2 original patch\n",
-        "orig_2_2 = np.array([[2, 4], [4,8]])\n",
+        "orig_2_2 = np.array([[6, 8], [8,4]])\n",
         "print(orig_2_2)"
       ],
       "metadata": {
@@ -316,7 +316,7 @@
         "# Let's first use the duplication method\n",
         "def duplicate(x_in):\n",
         "  x_out = np.zeros(( x_in.shape[0]*2, x_in.shape[1]*2 ))\n",
-        "  # TO DO -- write the duplication routine\n",
+        "  # TODO -- write the duplication routine\n",
         "  # Replace this line\n",
         "  x_out = x_out\n",
         "\n",
@@ -388,7 +388,7 @@
         "# The input x_high_res is the original high res image, from which you can deduce the position of the maximum index\n",
         "def max_unpool(x_in, x_high_res):\n",
         "  x_out = np.zeros(( x_in.shape[0]*2, x_in.shape[1]*2 ))\n",
-        "  # TO DO -- write the subsampling routine\n",
+        "  # TODO -- write the subsampling routine\n",
         "  # Replace this line\n",
         "  x_out = x_out\n",
         "\n",
@@ -460,7 +460,7 @@
         "  x_out = np.zeros(( x_in.shape[0]*2, x_in.shape[1]*2 ))\n",
         "  x_in_pad = np.zeros((x_in.shape[0]+1, x_in.shape[1]+1))\n",
         "  x_in_pad[0:x_in.shape[0],0:x_in.shape[1]] = x_in\n",
-        "  # TO DO -- write the duplication routine\n",
+        "  # TODO -- write the duplication routine\n",
         "  # Replace this line\n",
         "  x_out = x_out\n",
         "\n",

Notebooks/Chap10/10_5_Convolution_For_MNIST.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyNAcc98STMeyQgh9SbVHWG+",
+      "authorship_tag": "ABX9TyORZF8xy4X1yf4oRhRq8Rtm",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -35,7 +35,7 @@
         "\n",
         "The code is adapted from https://nextjournal.com/gkoehler/pytorch-mnist\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ],
@@ -65,10 +65,19 @@
       "source": [
         "# Run this once to load the train and test data straight into a dataloader class\n",
         "# that will provide the batches\n",
+        "\n",
+        "# (It may complain that some files are missing because the files seem to have been\n",
+        "# reorganized on the underlying website, but it still seems to work). If everything is working\n",
+        "# properly, then the whole notebook should run to the end without further problems\n",
+        "# even before you make changes.\n",
         "batch_size_train = 64\n",
         "batch_size_test = 1000\n",
+        "\n",
+        "# TODO Change this directory to point towards an existing directory\n",
+        "myDir = '/files/'\n",
+        "\n",
         "train_loader = torch.utils.data.DataLoader(\n",
-        "  torchvision.datasets.MNIST('/files/', train=True, download=True,\n",
+        "  torchvision.datasets.MNIST(myDir, train=True, download=True,\n",
         "                             transform=torchvision.transforms.Compose([\n",
         "                               torchvision.transforms.ToTensor(),\n",
         "                               torchvision.transforms.Normalize(\n",
@@ -77,7 +86,7 @@
         "  batch_size=batch_size_train, shuffle=True)\n",
         "\n",
         "test_loader = torch.utils.data.DataLoader(\n",
-        "  torchvision.datasets.MNIST('/files/', train=False, download=True,\n",
+        "  torchvision.datasets.MNIST(myDir, train=False, download=True,\n",
         "                             transform=torchvision.transforms.Compose([\n",
         "                               torchvision.transforms.ToTensor(),\n",
         "                               torchvision.transforms.Normalize(\n",

Notebooks/Chap11/11_1_Shattered_Gradients.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyMrF4rB2hTKq7XzLuYsURdL",
+      "authorship_tag": "ABX9TyP3VmRg51U+7NCfSYjRRrgv",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates the phenomenon of shattered gradients as discussed in section 11.1.1.  It replicates some of the experiments in [Balduzzi et al. (2017)](https://arxiv.org/abs/1702.08591).\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -235,7 +235,7 @@
         "# Finite difference calculation\n",
         "dydx_fd = (y2-y1)/delta\n",
         "\n",
-        "print(\"Gradient calculation=%f, Finite difference gradient=%f\"%(dydx,dydx_fd))\n"
+        "print(\"Gradient calculation=%f, Finite difference gradient=%f\"%(dydx.squeeze(),dydx_fd.squeeze()))\n"
       ],
       "metadata": {
         "id": "KJpQPVd36Haq"
@@ -267,8 +267,8 @@
         "  fig,ax = plt.subplots()\n",
         "  ax.plot(np.squeeze(x_in), np.squeeze(dydx), 'b-')\n",
         "  ax.set_xlim(-2,2)\n",
-        "  ax.set_xlabel('Input, $x$')\n",
+        "  ax.set_xlabel(r'Input, $x$')\n",
-        "  ax.set_ylabel('Gradient, $dy/dx$')\n",
+        "  ax.set_ylabel(r'Gradient, $dy/dx$')\n",
         "  ax.set_title('No layers = %d'%(K))\n",
         "  plt.show()"
       ],

Notebooks/Chap11/11_2_Residual_Networks.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyMXS3SPB4cS/4qxix0lH/Hq",
+      "authorship_tag": "ABX9TyNIY8tswL9e48d5D53aSmHO",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +33,7 @@
         "\n",
         "This notebook adapts the networks for MNIST1D to use residual connections.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n"
@@ -45,8 +45,8 @@
     {
       "cell_type": "code",
       "source": [
-        "# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
+        "# Run this if you're in a Colab to install MNIST 1D repository\n",
-        "!git clone https://github.com/greydanus/mnist1d"
+        "!pip install git+https://github.com/greydanus/mnist1d"
       ],
       "metadata": {
         "id": "D5yLObtZCi9J"

Notebooks/Chap11/11_3_Batch_Normalization.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyPVeAd3eDpEOCFh8CVyr1zz",
+      "authorship_tag": "ABX9TyPx2mM2zTHmDJeKeiE1RymT",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates the use of batch normalization in residual networks.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n"
@@ -45,8 +45,8 @@
     {
       "cell_type": "code",
       "source": [
-        "# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
+        "# Run this if you're in a Colab to install MNIST 1D repository\n",
-        "!git clone https://github.com/greydanus/mnist1d"
+        "!pip install git+https://github.com/greydanus/mnist1d"
       ],
       "metadata": {
         "id": "D5yLObtZCi9J"

Notebooks/Chap12/12_1_Self_Attention.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook builds a self-attention mechanism from scratch, as discussed in section 12.2 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n"

Notebooks/Chap12/12_2_Multihead_Self_Attention.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyMSk8qTqDYqFnRJVZKlsue0",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -29,11 +28,11 @@
     {
       "cell_type": "markdown",
       "source": [
-        "# **Notebook 12.1: Multhead Self-Attention**\n",
+        "# **Notebook 12.2: Multihead Self-Attention**\n",
         "\n",
         "This notebook builds a multihead self-attention mechanism as in figure 12.6\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n"
@@ -147,9 +146,7 @@
         "  exp_values = np.exp(data_in) ;\n",
         "  # Sum over columns\n",
         "  denom = np.sum(exp_values, axis = 0);\n",
-        "  # Replicate denominator to N rows\n",
+        "  # Compute softmax (numpy broadcasts denominator to all rows automatically)\n",
-        "  denom = np.matmul(np.ones((data_in.shape[0],1)), denom[np.newaxis,:])\n",
-        "  # Compute softmax\n",
         "  softmax = exp_values / denom\n",
         "  # return the answer\n",
         "  return softmax"

Notebooks/Chap12/12_3_Tokenization.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook builds set of tokens from a text string as in figure 12.8 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "I adapted this code from *SOMEWHERE*.  If anyone recognizes it, can you let me know and I will give the proper attribution or rewrite if the license is not permissive.\n",
         "\n",

Notebooks/Chap12/12_4_Decoding_Strategies.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This practical investigates neural decoding from transformer models.  \n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],

Notebooks/Chap13/13_1_Graph_Representation.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates representing graphs with matrices as illustrated in figure 13.4 from the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n"

Notebooks/Chap13/13_2_Graph_Classification.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOMSGUFWT+YN0fwYHpMmHJM",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +32,7 @@
         "\n",
         "This notebook investigates representing graphs with matrices as illustrated in figure 13.4 from the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -99,7 +98,7 @@
         "\n",
         "# TODO -- Define node matrix\n",
         "# There will be 9 nodes and 118 possible chemical elements\n",
-        "# so we'll define a 9x118 matrix.  Each column represents one\n",
+        "# so we'll define a 118x9 matrix.  Each column represents one\n",
         "# node and is a one-hot vector (i.e. all zeros, except a single one at the\n",
         "# chemical number of the element).\n",
         "# Chemical numbers:  Hydrogen-->1, Carbon-->6,  Oxygen-->8\n",

Notebooks/Chap13/13_3_Neighborhood_Sampling.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates neighborhood sampling of graphs as in figure 13.10 from the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],

Notebooks/Chap13/13_4_Graph_Attention_Networks.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook builds a graph attention mechanism from scratch, as discussed in section 13.8.6 of the book and illustrated in figure 13.12c\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n"
@@ -109,7 +109,7 @@
         "# Choose random values for the parameters\n",
         "omega = np.random.normal(size=(D,D))\n",
         "beta = np.random.normal(size=(D,1))\n",
-        "phi = np.random.normal(size=(1,2*D))"
+        "phi = np.random.normal(size=(2*D,1))"
       ],
       "metadata": {
         "id": "79TSK7oLMobe"

Notebooks/Chap15/15_1_GAN_Toy_Example.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyM0StKV3FIZ3MZqfflqC0Rv",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +32,7 @@
         "\n",
         "This notebook investigates the GAN toy example as illustrated in figure 15.1 in the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -339,7 +338,7 @@
         "    print(\"Initial generator loss = \", compute_generator_loss(z, theta, phi0, phi1))\n",
         "    for iter in range(n_iter):\n",
         "      # Get gradient\n",
-        "      dl_dtheta = compute_generator_gradient(x_real, x_syn, phi0, phi1)\n",
+        "      dl_dtheta = compute_generator_gradient(z, theta, phi0, phi1)\n",
         "      # Take a gradient step (uphill, since we are trying to make synthesized data less well classified by discriminator)\n",
         "      theta = theta + alpha * dl_dtheta ;\n",
         "\n",

Notebooks/Chap15/15_2_Wasserstein_Distance.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyNyLnpoXgKN+RGCuTUszCAZ",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +32,7 @@
         "\n",
         "This notebook investigates the GAN toy example as illustrated in figure 15.1 in the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -87,6 +86,7 @@
       "cell_type": "code",
       "source": [
         "# TODO Define the distance matrix from figure 15.8d\n",
+        "# The index should be normalized before being used in the distance calculation.\n",
         "# Replace this line\n",
         "dist_mat = np.zeros((10,10))\n",
         "\n",
@@ -129,7 +129,7 @@
     {
       "cell_type": "code",
       "source": [
-        "draw_2D_heatmap(dist_mat,'Distance $|i-j|$', my_colormap)"
+        "draw_2D_heatmap(dist_mat,r'Distance $|i-j|$', my_colormap)"
       ],
       "metadata": {
         "id": "G0HFPBXyHT6V"
@@ -153,9 +153,9 @@
       "cell_type": "code",
       "source": [
         "# TODO:  Now construct the matrix A that has the initial distribution constraints\n",
-        "# so that Ap=b where p is the transport plan P vectorized rows first so p = np.flatten(P)\n",
+        "# so that A @ TPFlat=b where TPFlat is the transport plan TP vectorized rows first so TPFlat = np.flatten(TP)\n",
         "# Replace this line:\n",
-        "A = np.zeros((20,100))\n"
+        "A = np.zeros((20,100))"
       ],
       "metadata": {
         "id": "7KrybL96IuNW"
@@ -197,8 +197,8 @@
     {
       "cell_type": "code",
       "source": [
-        "P = np.array(opt.x).reshape(10,10)\n",
+        "TP = np.array(opt.x).reshape(10,10)\n",
-        "draw_2D_heatmap(P,'Transport plan $\\mathbf{P}$', my_colormap)"
+        "draw_2D_heatmap(TP,r'Transport plan $\\mathbf{P}$', my_colormap)"
       ],
       "metadata": {
         "id": "nZGfkrbRV_D0"
@@ -218,8 +218,9 @@
     {
       "cell_type": "code",
       "source": [
-        "was = np.sum(P * dist_mat)\n",
+        "was = np.sum(TP * dist_mat)\n",
-        "print(\"Wasserstein distance = \", was)"
+        "print(\"Your Wasserstein distance = \", was)\n",
+        "print(\"Correct answer =  0.15148578811369506\")"
       ],
       "metadata": {
         "id": "yiQ_8j-Raq3c"

Notebooks/Chap16/16_1_1D_Normalizing_Flows.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates a 1D normalizing flows example similar to that illustrated in figures 16.1 to 16.3 in the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],

Notebooks/Chap16/16_2_Autoregressive_Flows.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates a 1D normalizing flows example similar to that illustrated in figure 16.7 in the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],

Notebooks/Chap16/16_3_Contraction_Mappings.ipynb

@@ -22,7 +22,7 @@
         "\n",
         "This notebook investigates a 1D normalizing flows example similar to that illustrated in figure 16.9 in the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]

Notebooks/Chap17/17_1_Latent_Variable_Models.ipynb

@@ -22,7 +22,7 @@
         "\n",
         "This notebook investigates a non-linear latent variable model similar to that in figures 17.2 and 17.3 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]
@@ -55,7 +55,7 @@
         "Pr(z) = \\text{Norm}_{z}[0,1]\n",
         "\\end{equation}\n",
         "\n",
-        "As in figure 17.2, we'll assume that the output is two dimensional, we we need to define a function that maps from the 1D latent variable to two dimensions.  Usually, we would use a neural network, but in this case, we'll just define an arbitrary relationship.\n",
+        "As in figure 17.2, we'll assume that the output is two dimensional, we need to define a function that maps from the 1D latent variable to two dimensions.  Usually, we would use a neural network, but in this case, we'll just define an arbitrary relationship.\n",
         "\n",
         "\\begin{align}\n",
         "x_{1} &=& 0.5\\cdot\\exp\\Bigl[\\sin\\bigl[2+ 3.675 z \\bigr]\\Bigr]\\\\\n",

Notebooks/Chap17/17_2_Reparameterization_Trick.ipynb

@@ -1,18 +1,16 @@
 {
   "cells": [
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "colab_type": "text",
+        "id": "view-in-github",
-        "id": "view-in-github"
+        "colab_type": "text"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap17/17_2_Reparameterization_Trick.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "t9vk9Elugvmi"
@@ -22,7 +20,7 @@
         "\n",
         "This notebook investigates the reparameterization trick as described in section 17.7 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]
@@ -40,7 +38,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "paLz5RukZP1J"
@@ -114,7 +111,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "r5Hl2QkimWx9"
@@ -139,13 +135,12 @@
         "\n",
         "fig,ax = plt.subplots()\n",
         "ax.plot(phi_vals, expected_vals,'r-')\n",
-        "ax.set_xlabel('Parameter $\\phi$')\n",
+        "ax.set_xlabel(r'Parameter $\\phi$')\n",
-        "ax.set_ylabel('$\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
+        "ax.set_ylabel(r'$\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
         "plt.show()"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "zTCykVeWqj_O"
@@ -253,13 +248,12 @@
         "\n",
         "fig,ax = plt.subplots()\n",
         "ax.plot(phi_vals, deriv_vals,'r-')\n",
-        "ax.set_xlabel('Parameter $\\phi$')\n",
+        "ax.set_xlabel(r'Parameter $\\phi$')\n",
-        "ax.set_ylabel('$\\partial/\\partial\\phi\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
+        "ax.set_ylabel(r'$\\partial/\\partial\\phi\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
         "plt.show()"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "ASu4yKSwAEYI"
@@ -269,7 +263,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "xoFR1wifc8-b"
@@ -366,13 +359,12 @@
         "\n",
         "fig,ax = plt.subplots()\n",
         "ax.plot(phi_vals, deriv_vals,'r-')\n",
-        "ax.set_xlabel('Parameter $\\phi$')\n",
+        "ax.set_xlabel(r'Parameter $\\phi$')\n",
-        "ax.set_ylabel('$\\partial/\\partial\\phi\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
+        "ax.set_ylabel(r'$\\partial/\\partial\\phi\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
         "plt.show()"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "1TWBiUC7bQSw"
@@ -403,7 +395,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "d-0tntSYdKPR"
@@ -415,9 +406,8 @@
   ],
   "metadata": {
     "colab": {
-      "authorship_tag": "ABX9TyOxO2/0DTH4n4zhC97qbagY",
+      "provenance": [],
-      "include_colab_link": true,
+      "include_colab_link": true
-      "provenance": []
     },
     "kernelspec": {
       "display_name": "Python 3",

Notebooks/Chap17/17_3_Importance_Sampling.ipynb

@@ -1,18 +1,16 @@
 {
   "cells": [
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "colab_type": "text",
+        "id": "view-in-github",
-        "id": "view-in-github"
+        "colab_type": "text"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap17/17_3_Importance_Sampling.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "t9vk9Elugvmi"
@@ -22,7 +20,7 @@
         "\n",
         "This notebook investigates importance sampling as described in section 17.8.1 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]
@@ -40,7 +38,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "f7a6xqKjkmvT"
@@ -61,7 +58,7 @@
         "by drawing $I$ samples $y_i$ and using the formula:\n",
         "\n",
         "\\begin{equation}\n",
-        "\\mathbb{E}_{y}\\Bigl[\\exp\\bigl[- (y-1)^4\\bigr]\\Bigr] \\approx \\frac{1}{I} \\sum_{i=1}^I \\exp\\bigl[-(y-1)^4 \\bigr]\n",
+        "\\mathbb{E}_{y}\\Bigl[\\exp\\bigl[- (y-1)^4\\bigr]\\Bigr] \\approx \\frac{1}{I} \\sum_{i=1}^I \\exp\\bigl[-(y_i-1)^4 \\bigr]\n",
         "\\end{equation}"
       ]
     },
@@ -126,7 +123,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "Jr4UPcqmnXCS"
@@ -166,8 +162,8 @@
         "mean_all = np.zeros_like(n_sample_all)\n",
         "variance_all = np.zeros_like(n_sample_all)\n",
         "for i in range(len(n_sample_all)):\n",
-        "  print(\"Computing mean and variance for expectation with %d samples\"%(n_sample_all[i]))\n",
+        "  mean_all[i],variance_all[i] = compute_mean_variance(n_sample_all[i])\n",
-        "  mean_all[i],variance_all[i] = compute_mean_variance(n_sample_all[i])"
+        "  print(\"No samples: \", n_sample_all[i], \", Mean: \", mean_all[i], \", Variance: \", variance_all[i])"
       ]
     },
     {
@@ -189,7 +185,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "XTUpxFlSuOl7"
@@ -199,7 +194,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "6hxsl3Pxo1TT"
@@ -234,7 +228,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "G9Xxo0OJsIqD"
@@ -283,7 +276,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "2sVDqP0BvxqM"
@@ -313,8 +305,8 @@
         "mean_all2 = np.zeros_like(n_sample_all)\n",
         "variance_all2 = np.zeros_like(n_sample_all)\n",
         "for i in range(len(n_sample_all)):\n",
-        "  print(\"Computing variance for expectation with %d samples\"%(n_sample_all[i]))\n",
+        "  mean_all2[i], variance_all2[i] = compute_mean_variance2(n_sample_all[i])\n",
-        "  mean_all2[i], variance_all2[i] = compute_mean_variance2(n_sample_all[i])"
+        "  print(\"No samples: \", n_sample_all[i], \", Mean: \", mean_all2[i], \", Variance: \", variance_all2[i])"
       ]
     },
     {
@@ -348,7 +340,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "EtBP6NeLwZqz"
@@ -360,7 +351,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "_wuF-NoQu1--"
@@ -387,7 +377,7 @@
         "def compute_expectation2b(n_samples):\n",
         "  # TODO -- complete this function\n",
         "  # 1. Draw n_samples from auxiliary distribution\n",
-        "  # 2. Compute f[y] for those samples\n",
+        "  # 2. Compute f2[y] for those samples\n",
         "  # 3. Scale the results by pr_y / q_y\n",
         "  # 4. Compute the mean of these weighted samples\n",
         "  # Replace this line\n",
@@ -432,8 +422,8 @@
         "mean_all2b = np.zeros_like(n_sample_all)\n",
         "variance_all2b = np.zeros_like(n_sample_all)\n",
         "for i in range(len(n_sample_all)):\n",
-        "  print(\"Computing variance for expectation with %d samples\"%(n_sample_all[i]))\n",
+        "  mean_all2b[i], variance_all2b[i] = compute_mean_variance2b(n_sample_all[i])\n",
-        "  mean_all2b[i], variance_all2b[i] = compute_mean_variance2b(n_sample_all[i])"
+        "  print(\"No samples: \", n_sample_all[i], \", Mean: \", mean_all2b[i], \", Variance: \", variance_all2b[i])"
       ]
     },
     {
@@ -478,7 +468,6 @@
       ]
     },
     {
-      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "y8rgge9MNiOc"
@@ -490,9 +479,8 @@
   ],
   "metadata": {
     "colab": {
-      "authorship_tag": "ABX9TyNecz9/CDOggPSmy1LjT/Dv",
+      "provenance": [],
-      "include_colab_link": true,
+      "include_colab_link": true
-      "provenance": []
     },
     "kernelspec": {
       "display_name": "Python 3",

Notebooks/Chap18/18_1_Diffusion_Encoder.ipynb

@@ -3,8 +3,8 @@
     {
       "cell_type": "markdown",
       "metadata": {
-        "colab_type": "text",
+        "id": "view-in-github",
-        "id": "view-in-github"
+        "colab_type": "text"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_1_Diffusion_Encoder.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
@@ -20,7 +20,7 @@
         "\n",
         "This notebook investigates the diffusion encoder as described in section 18.2 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]
@@ -405,11 +405,11 @@
         "\n",
         "    # TODO Write this function\n",
         "    # 1. For each x (value in x_plot_vals):\n",
-        "      # 2. Compute the mean and variance of the diffusion kernel at time t\n",
+        "    # 2. Compute the mean and variance of the diffusion kernel at time t\n",
-        "      # 3. Compute pdf of this Gaussian at every x_plot_val\n",
+        "    # 3. Compute pdf of this Gaussian at every x_plot_val\n",
-        "      # 4. Weight Gaussian by probability at position x and by 0.01 to componensate for bin size\n",
+        "    # 4. Weight Gaussian by probability at position x and by 0.01 to componensate for bin size\n",
-        "      # 5. Accumulate weighted Gaussian in marginal at time t.\n",
+        "    # 5. Accumulate weighted Gaussian in marginal at time t.\n",
-        "    # 6. Multiply result by 0.01 to compensate for bin size\n",
+        "\n",
         "    # Replace this line:\n",
         "    marginal_at_time_t = marginal_at_time_t\n",
         "\n",
@@ -454,9 +454,8 @@
   ],
   "metadata": {
     "colab": {
-      "authorship_tag": "ABX9TyMpC8kgLnXx0XQBtwNAQ4jJ",
+      "provenance": [],
-      "include_colab_link": true,
+      "include_colab_link": true
-      "provenance": []
     },
     "kernelspec": {
       "display_name": "Python 3",

Notebooks/Chap18/18_2_1D_Diffusion_Model.ipynb

@@ -22,7 +22,7 @@
         "\n",
         "This notebook investigates the diffusion encoder as described in section 18.3 and 18.4 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]

Notebooks/Chap18/18_3_Reparameterized_Model.ipynb

@@ -22,7 +22,7 @@
         "\n",
         "This notebook investigates the reparameterized model as described in section 18.5 of the book and implements algorithms 18.1 and 18.2.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]

Notebooks/Chap18/18_4_Families_of_Diffusion_Models.ipynb

@@ -22,7 +22,7 @@
         "\n",
         "This notebook investigates the reparameterized model as described in section 18.5 of the book and computers the results shown in figure 18.10c-f.  These models are based on the paper \"Denoising diffusion implicit models\" which can be found [here](https://arxiv.org/pdf/2010.02502.pdf).\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]

Notebooks/Chap19/19_1_Markov_Decision_Processes.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates Markov decision processes as described in section 19.1 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],

Notebooks/Chap19/19_2_Dynamic_Programming.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOlD6kmCxX3SKKuh3oJikKA",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +32,7 @@
         "\n",
         "This notebook investigates the dynamic programming approach to tabular reinforcement learning as described in figure 19.10 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -393,7 +392,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Update the state values for the current policy, by making the values at at adjacent\n",
+        "# Update the state values for the current policy, by making the values at adjacent\n",
         "# states compatible with the Bellman equation (equation 19.11)\n",
         "def policy_evaluation(policy, state_values, rewards,  transition_probabilities_given_action, gamma):\n",
         "\n",
@@ -406,6 +405,10 @@
         "            state_values_new[state] = 3.0\n",
         "            break\n",
         "\n",
+        "        # TODO -- Write this function (from equation 19.11, but bear in mind policy is deterministic here)\n",
+        "        # Replace this line\n",
+        "        state_values_new[state] = 0\n",
+        "\n",
         "    return state_values_new\n",
         "\n",
         "# Greedily choose the action that maximizes the value for each state.\n",

Notebooks/Chap19/19_3_Monte_Carlo_Methods.ipynb

@@ -1,20 +1,4 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyMWjsdr5SDwyzcDftnehlNo",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
       "cell_type": "markdown",
@@ -28,6 +12,9 @@
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      },
       "source": [
         "# **Notebook 19.3: Monte-Carlo methods**\n",
         "\n",
@@ -35,44 +22,51 @@
         "\n",
         "NOTE!  There is a mistake in Figure 19.11 in the first printing of the book, so check the errata to avoid becoming confused.  Apologies!\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
-        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
-      ],
+        "\n",
-      "metadata": {
+        "Thanks to [Akshil Patel](https://www.akshilpatel.com) and [Jessica Nicholson](https://jessicanicholson1.github.io) for their help in preparing this notebook."
-        "id": "t9vk9Elugvmi"
+      ]
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "import numpy as np\n",
-        "import matplotlib.pyplot as plt\n",
-        "from PIL import Image"
-      ],
       "metadata": {
         "id": "OLComQyvCIJ7"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "from PIL import Image\n",
+        "\n",
+        "from IPython.display import clear_output\n",
+        "from time import sleep"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZsvrUszPLyEG"
+      },
+      "outputs": [],
       "source": [
         "# Get local copies of components of images\n",
         "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Empty.png\n",
         "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Hole.png\n",
         "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Fish.png\n",
         "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Penguin.png"
-      ],
+      ]
-      "metadata": {
-        "id": "ZsvrUszPLyEG"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Gq1HfJsHN3SB"
+      },
+      "outputs": [],
       "source": [
         "# Ugly class that takes care of drawing pictures like in the book.\n",
         "# You can totally ignore this code!\n",
@@ -257,205 +251,281 @@
         "\n",
         "\n",
         "    plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "Gq1HfJsHN3SB"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "eBQ7lTpJQBSe"
+      },
+      "outputs": [],
       "source": [
         "# We're going to work on the problem depicted in figure 19.10a\n",
         "n_rows = 4; n_cols = 4\n",
         "layout = np.zeros(n_rows * n_cols)\n",
         "reward_structure = np.zeros(n_rows * n_cols)\n",
-        "layout[9] = 1 ; reward_structure[9] = -2\n",
+        "layout[9] = 1 ; reward_structure[9] = -2    # Hole\n",
-        "layout[10] = 1; reward_structure[10] = -2\n",
+        "layout[10] = 1; reward_structure[10] = -2   # Hole\n",
-        "layout[14] = 1; reward_structure[14] = -2\n",
+        "layout[14] = 1; reward_structure[14] = -2   # Hole\n",
-        "layout[15] = 2; reward_structure[15] = 3\n",
+        "layout[15] = 2; reward_structure[15] = 3    # Fish\n",
         "initial_state = 0\n",
         "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
         "mdp_drawer.draw(layout, state = initial_state, rewards=reward_structure, draw_state_index = True)"
-      ],
+      ]
-      "metadata": {
-        "id": "eBQ7lTpJQBSe"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "For clarity, the black numbers are the state number and the red numbers are the reward for being in that state.  Note that the states are indexed from 0 rather than 1 as in the book to make the code neater."
-      ],
       "metadata": {
         "id": "6Vku6v_se2IG"
-      }
+      },
+      "source": [
+        "For clarity, the black numbers are the state number and the red numbers are the reward for being in that state.  Note that the states are indexed from 0 rather than 1 as in the book to make the code neater."
+      ]
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "Fhc6DzZNOjiC"
+      },
       "source": [
         "Now let's define the state transition function $Pr(s_{t+1}|s_{t},a)$ in full where $a$ is the actions.  Here $a=0$ means try to go upward, $a=1$, right, $a=2$ down and $a=3$ right.  However, the ice is slippery, so we don't always go the direction we want to.\n",
         "\n",
         "Note that as for the states, we've indexed the actions from zero (unlike in the book) so they map to the indices of arrays better"
-      ],
+      ]
-      "metadata": {
-        "id": "Fhc6DzZNOjiC"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "l7rT78BbOgTi"
+      },
+      "outputs": [],
       "source": [
         "transition_probabilities_given_action0 = np.array(\\\n",
-        "[[0.00 , 0.33, 0.00, 0.00,  0.50, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        "[[0.90, 0.05, 0.00, 0.00,  0.85, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.50 , 0.00, 0.33, 0.00,  0.00, 0.50, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.05, 0.85, 0.05, 0.00,  0.00, 0.85, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.33, 0.00, 0.50,  0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.05, 0.85, 0.05,  0.00, 0.00, 0.85, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.33, 0.00,  0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.05, 0.90,  0.00, 0.00, 0.00, 0.85,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.50 , 0.00, 0.00, 0.00,  0.00, 0.17, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.05, 0.00, 0.00, 0.00,  0.05, 0.05, 0.00, 0.00,  0.85, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.34, 0.00, 0.00,  0.25, 0.00, 0.17, 0.00,   0.00, 0.50, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.05, 0.00, 0.00,  0.05, 0.00, 0.05, 0.00,  0.00, 0.85, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.34, 0.00,  0.00, 0.17, 0.00, 0.25,   0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.05, 0.00,  0.00, 0.05, 0.00, 0.05,  0.00, 0.00, 0.85, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.50,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.05, 0.05,  0.00, 0.00, 0.00, 0.85,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.17, 0.00, 0.00,   0.75, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.05, 0.05, 0.00, 0.00,  0.85, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.16, 0.00, 0.00,   0.25, 0.00, 0.17, 0.00,   0.00, 0.50, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.05, 0.00, 0.00,  0.05, 0.00, 0.05, 0.00,  0.00, 0.85, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.16, 0.00,   0.00, 0.17, 0.00, 0.25,   0.00, 0.00, 0.50, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.05, 0.00,  0.00, 0.05, 0.00, 0.05,  0.00, 0.00, 0.85, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.75 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.05, 0.05,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.10, 0.05, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.16, 0.00, 0.00,   0.25, 0.00, 0.25, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.05, 0.00, 0.00,  0.05, 0.05, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.16, 0.00,   0.00, 0.25, 0.00, 0.25 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.05, 0.00,  0.00, 0.05, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.25, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.05, 0.00]])\n",
-        "])\n",
+        "\n",
         "\n",
         "transition_probabilities_given_action1 = np.array(\\\n",
-        "[[0.00 , 0.25, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        "[[0.10, 0.05, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.75 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.85, 0.05, 0.05, 0.00,  0.00, 0.05, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.50, 0.00, 0.50,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.85, 0.05, 0.05,  0.00, 0.00, 0.05, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.50, 0.00,  0.00, 0.00, 0.00, 0.33,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.85, 0.90,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.25 , 0.00, 0.00, 0.00,  0.00, 0.17, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.05, 0.00, 0.00, 0.00,  0.05, 0.05, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.25, 0.00, 0.00,  0.50, 0.00, 0.17, 0.00,   0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.05, 0.00, 0.00,  0.85, 0.00, 0.05, 0.00,  0.00, 0.05, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.25, 0.00,  0.00, 0.50, 0.00, 0.33,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.05, 0.00,  0.00, 0.85, 0.00, 0.05,  0.00, 0.00, 0.05, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.50,  0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.33,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.85, 0.85,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.17, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.05, 0.05, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.16, 0.00, 0.00,   0.50, 0.00, 0.17, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.05, 0.00, 0.00,  0.85, 0.00, 0.05, 0.00,  0.00, 0.05, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.16, 0.00,   0.00, 0.50, 0.00, 0.33,   0.00, 0.00, 0.25, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.05, 0.00,  0.00, 0.85, 0.00, 0.05,  0.00, 0.00, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.34,   0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.50 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.85, 0.85,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.10, 0.05, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.16, 0.00, 0.00,   0.75, 0.00, 0.25, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.05, 0.00, 0.00,  0.85, 0.05, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.16, 0.00,   0.00, 0.50, 0.00, 0.50 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.05, 0.00,  0.00, 0.85, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.34,   0.00, 0.00, 0.50, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.85, 0.00]])\n",
-        "])\n",
+        "\n",
         "\n",
         "transition_probabilities_given_action2 = np.array(\\\n",
-        "[[0.00 , 0.25, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        "[[0.10, 0.05, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.25 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.05, 0.05, 0.05, 0.00,  0.00, 0.05, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.25, 0.00, 0.25,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.05, 0.05, 0.05,  0.00, 0.00, 0.05, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.25, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.05, 0.10,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.75 , 0.00, 0.00, 0.00,  0.00, 0.17, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.85, 0.00, 0.00, 0.00,  0.05, 0.05, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.50, 0.00, 0.00,  0.25, 0.00, 0.17, 0.00,   0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.85, 0.00, 0.00,  0.05, 0.00, 0.05, 0.00,  0.00, 0.05, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.50, 0.00,  0.00, 0.16, 0.00, 0.25,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.85, 0.00,  0.00, 0.05, 0.00, 0.05,  0.00, 0.00, 0.05, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.75,  0.00, 0.00, 0.16, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.85,  0.00, 0.00, 0.05, 0.05,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.50, 0.00, 0.00, 0.00,   0.00, 0.17, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.85, 0.00, 0.00, 0.00,  0.05, 0.05, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.50, 0.00, 0.00,   0.25, 0.00, 0.17, 0.00,   0.00, 0.33, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.85, 0.00, 0.00,  0.05, 0.00, 0.05, 0.00,  0.00, 0.05, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.50, 0.00,   0.00, 0.16, 0.00, 0.25,   0.00, 0.00, 0.33, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.85, 0.00,  0.00, 0.05, 0.00, 0.05,  0.00, 0.00, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.16, 0.00,   0.00, 0.00, 0.00, 0.50 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.85,  0.00, 0.00, 0.05, 0.05,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00,   0.00, 0.33, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.85, 0.00, 0.00, 0.00,  0.90, 0.05, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.50, 0.00, 0.00,   0.50, 0.00, 0.33, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.85, 0.00, 0.00,  0.05, 0.85, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.50, 0.00,   0.00, 0.34, 0.00, 0.50 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.85, 0.00,  0.00, 0.05, 0.85, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.34, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.85,  0.00, 0.00, 0.05, 0.00]])\n",
-        "])\n",
         "\n",
         "transition_probabilities_given_action3 = np.array(\\\n",
-        "[[0.00 , 0.25, 0.00, 0.00,  0.33, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        "[[0.90, 0.85, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.50 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.05, 0.05, 0.85, 0.00,  0.00, 0.05, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.50, 0.00, 0.75,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.05, 0.05, 0.85,  0.00, 0.00, 0.05, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.50, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.05, 0.10,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.50 , 0.00, 0.00, 0.00,  0.00, 0.50, 0.00, 0.00,   0.33, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.05, 0.00, 0.00, 0.00,  0.85, 0.85, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.25, 0.00, 0.00,  0.33, 0.00, 0.50, 0.00,   0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.05, 0.00, 0.00,  0.05, 0.00, 0.85, 0.00,  0.00, 0.05, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.50,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.05, 0.00,  0.00, 0.05, 0.00, 0.85,  0.00, 0.00, 0.05, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.25,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.05, 0.05,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.34, 0.00, 0.00, 0.00,   0.00, 0.50, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.85, 0.85, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.16, 0.00, 0.00,   0.33, 0.00, 0.50, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.05, 0.00, 0.00,  0.05, 0.00, 0.85, 0.00,  0.00, 0.05, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.16, 0.00,   0.00, 0.17, 0.00, 0.50,   0.00, 0.00, 0.25, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.05, 0.00,  0.00, 0.05, 0.00, 0.85,  0.00, 0.00, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.25 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.05, 0.05,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.34, 0.00, 0.00, 0.00,   0.00, 0.50, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.90, 0.85, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.16, 0.00, 0.00,   0.50, 0.00, 0.50, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.05, 0.00, 0.00,  0.05, 0.05, 0.85, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.16, 0.00,   0.00, 0.25, 0.00, 0.75 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.05, 0.00,  0.00, 0.05, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.25, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.05, 0.00]])\n",
-        "])\n",
+        "\n",
+        "\n",
         "\n",
         "# Store all of these in a three dimension array\n",
         "# Pr(s_{t+1}=2|s_{t}=1, a_{t}=3] is stored at position [2,1,3]\n",
         "transition_probabilities_given_action = np.concatenate((np.expand_dims(transition_probabilities_given_action0,2),\n",
         "                                                        np.expand_dims(transition_probabilities_given_action1,2),\n",
         "                                                        np.expand_dims(transition_probabilities_given_action2,2),\n",
-        "                                                        np.expand_dims(transition_probabilities_given_action3,2)),axis=2)"
+        "                                                        np.expand_dims(transition_probabilities_given_action3,2)),axis=2)\n",
-      ],
+        "\n",
+        "print('Grid Size:', len(transition_probabilities_given_action[0]))\n",
+        "print()\n",
+        "print('Transition Probabilities for when next state = 2:')\n",
+        "print(transition_probabilities_given_action[2])\n",
+        "print()\n",
+        "print('Transitions Probabilities for when next state = 2 and current state = 1')\n",
+        "print(transition_probabilities_given_action[2][1])\n",
+        "print()\n",
+        "print('Transitions Probabilities for  when next state = 2 and current state = 1 and action = 3 (Left):')\n",
+        "print(transition_probabilities_given_action[2][1][3])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
       "metadata": {
-        "id": "l7rT78BbOgTi"
+        "id": "BHWjp6Qq4tBF"
       },
-      "execution_count": null,
+      "source": [
-      "outputs": []
+        "## Implementation Details\n",
+        "\n",
+        "We provide the following methods:\n",
+        "\n",
+        "- **`markov_decision_process_step_stochastic`** - this function selects an action based on the stochastic policy for the current state, updates the state based on the transition probabilities associated with the chosen action, and returns the new state, the reward obtained for the new state, the chosen action, and whether the episode terminates.\n",
+        "\n",
+        "- **`get_one_episode`** - this function simulates an episode of agent-environment interaction. It returns the states, rewards, and actions seen in that episode, which we can then use to update the agent.\n",
+        "\n",
+        "- **`calculate_returns`** - this function calls on the **`calculate_return`** function that computes the discounted sum of rewards from a specific step, in a sequence of rewards.\n",
+        "\n",
+        "You have to implement the following methods:\n",
+        "\n",
+        "- **`deterministic_policy_to_epsilon_greedy`** - given a deterministic policy, where one action is chosen per state, this function computes the $\\epsilon$-greedy version of that policy, where each of the four actions has some nonzero probability of being selected per state. In each state, the probability of selecting each of the actions should sum to 1.\n",
+        "\n",
+        "- **`update_policy_mc`** - this function updates the action-value function using the Monte Carlo method.  We use the rollout trajectories collected using `get_one_episode` to calculate the returns. Then update the action values towards the Monte Carlo estimate of the return for each state."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "akjrncMF-FkU"
+      },
+      "outputs": [],
       "source": [
         "# This takes a single step from an MDP\n",
-        "def markov_decision_process_step_stochastic(state, transition_probabilities_given_action, reward_structure, stochastic_policy):\n",
+        "def markov_decision_process_step_stochastic(state, transition_probabilities_given_action, reward_structure, terminal_states, stochastic_policy):\n",
         "  # Pick action\n",
         "  action = np.random.choice(a=np.arange(0,4,1),p=stochastic_policy[:,state])\n",
+        "\n",
         "  # Update the state\n",
         "  new_state = np.random.choice(a=np.arange(0,transition_probabilities_given_action.shape[0]),p = transition_probabilities_given_action[:,state,action])\n",
         "  # Return the reward\n",
         "  reward = reward_structure[new_state]\n",
+        "  is_terminal = new_state in [terminal_states]\n",
         "\n",
-        "  return new_state, reward, action"
+        "  return new_state, reward, action, is_terminal"
-      ],
+      ]
-      "metadata": {
-        "id": "akjrncMF-FkU"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "# Run one episode and return actions, rewards, returns\n",
-        "def get_one_episode(initial_state, transition_probabilities_given_action, reward_structure, stochastic_policy):\n",
-        "\n",
-        "  max_steps = 1000\n",
-        "  states = np.zeros(max_steps, dtype='uint8') ;\n",
-        "  rewards = np.zeros(max_steps) ;\n",
-        "  actions = np.zeros(max_steps, dtype='uint8') ;\n",
-        "\n",
-        "  t = 0\n",
-        "  states[t] = initial_state\n",
-        "  # While haven't reached maximum number of steps\n",
-        "  while t< max_steps:\n",
-        "    # Keep stepping through MDP\n",
-        "    states[t+1],rewards[t+1],actions[t] = markov_decision_process_step_stochastic(states[t], transition_probabilities_given_action, reward_structure, stochastic_policy)\n",
-        "    # If we reach te:rminal state then quit\n",
-        "    if states[t]==15:\n",
-        "      break;\n",
-        "    t+=1\n",
-        "\n",
-        "  states = states[:t+1]\n",
-        "  rewards = rewards[:t+1]\n",
-        "  actions = actions[:t+1]\n",
-        "\n",
-        "  return states, rewards, actions"
-      ],
       "metadata": {
         "id": "bFYvF9nAloIA"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "# Run one episode and return actions, rewards, returns\n",
+        "def get_one_episode(initial_state, transition_probabilities_given_action, reward_structure, terminal_states, stochastic_policy):\n",
+        "\n",
+        "    states  = []\n",
+        "    rewards = []\n",
+        "    actions = []\n",
+        "\n",
+        "    states.append(initial_state)\n",
+        "    state = initial_state\n",
+        "\n",
+        "    is_terminal = False\n",
+        "    # While we haven't reached a terminal state\n",
+        "    while not is_terminal:\n",
+        "        # Keep stepping through MDP\n",
+        "        state, reward, action, is_terminal = markov_decision_process_step_stochastic(state,\n",
+        "                                                                                     transition_probabilities_given_action,\n",
+        "                                                                                     reward_structure,\n",
+        "                                                                                     terminal_states,\n",
+        "                                                                                     stochastic_policy)\n",
+        "        states.append(state)\n",
+        "        rewards.append(reward)\n",
+        "        actions.append(action)\n",
+        "\n",
+        "    states  = np.array(states, dtype=\"uint8\")\n",
+        "    rewards = np.array(rewards)\n",
+        "    actions = np.array(actions, dtype=\"uint8\")\n",
+        "\n",
+        "    # If the episode was terminated early, we need to compute the return differently using r_{t+1} + gamma*V(s_{t+1})\n",
+        "    return states, rewards, actions"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "qJhOrIId4tBF"
+      },
+      "outputs": [],
+      "source": [
+        "def visualize_one_episode(states, actions):\n",
+        "    # Define actions for visualization\n",
+        "    acts = ['up', 'right', 'down', 'left']\n",
+        "\n",
+        "    # Iterate over the states and actions\n",
+        "    for i in range(len(states)):\n",
+        "\n",
+        "        if i == 0:\n",
+        "            print('Starting State:', states[i])\n",
+        "\n",
+        "        elif i == len(states)-1:\n",
+        "            print('Episode Done:', states[i])\n",
+        "\n",
+        "        else:\n",
+        "            print('State', states[i-1])\n",
+        "            a = actions[i]\n",
+        "            print('Action:', acts[a])\n",
+        "            print('Next State:', states[i])\n",
+        "\n",
+        "        # Visualize the current state using the MDP drawer\n",
+        "        mdp_drawer.draw(layout, state=states[i], rewards=reward_structure, draw_state_index=True)\n",
+        "        clear_output(True)\n",
+        "\n",
+        "        # Pause for a short duration to allow observation\n",
+        "        sleep(1.5)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "_AKwdtQQHzIK"
+      },
+      "outputs": [],
       "source": [
         "# Convert deterministic policy (1x16) to an epsilon greedy stochastic policy (4x16)\n",
-        "def deterministic_policy_to_epsilon_greedy(policy, epsilon=0.1):\n",
+        "def deterministic_policy_to_epsilon_greedy(policy, epsilon=0.2):\n",
         "  # TODO -- write this function\n",
         "  # You should wind up with a 4x16 matrix, with epsilon/3 in every position except the real policy\n",
         "  # The columns should sum to one\n",
@@ -464,27 +534,27 @@
         "\n",
         "\n",
         "  return stochastic_policy"
-      ],
+      ]
-      "metadata": {
-        "id": "_AKwdtQQHzIK"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "Let's try generating an episode"
-      ],
       "metadata": {
         "id": "OhVXw2Favo-w"
-      }
+      },
+      "source": [
+        "Let's try generating an episode"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "5zQ1Oh9Zvnwt"
+      },
+      "outputs": [],
       "source": [
         "# Set seed so random numbers always the same\n",
-        "np.random.seed(0)\n",
+        "np.random.seed(6)\n",
         "# Print in compact form\n",
         "np.set_printoptions(precision=3)\n",
         "\n",
@@ -494,32 +564,55 @@
         "# Convert deterministic policy to stochastic\n",
         "stochastic_policy = deterministic_policy_to_epsilon_greedy(policy)\n",
         "\n",
-        "print(\"Initial policy:\")\n",
+        "print(\"Initial Penguin Policy:\")\n",
         "print(policy)\n",
+        "print()\n",
+        "print('Stochastic Penguin Policy:')\n",
+        "print(stochastic_policy)\n",
+        "print()\n",
         "\n",
         "initial_state = 5\n",
-        "states, rewards, actions = get_one_episode(initial_state,transition_probabilities_given_action, reward_structure, stochastic_policy)"
+        "terminal_states=[15]\n",
-      ],
+        "states, rewards, actions = get_one_episode(initial_state,transition_probabilities_given_action, reward_structure, terminal_states, stochastic_policy)\n",
-      "metadata": {
+        "\n",
-        "id": "5zQ1Oh9Zvnwt"
+        "print('Initial Penguin Position:')\n",
-      },
+        "mdp_drawer.draw(layout, state = initial_state, rewards=reward_structure, draw_state_index = True)\n",
-      "execution_count": null,
+        "\n",
-      "outputs": []
+        "print('Total steps to termination:', len(states))\n",
-    },
+        "print('Final Reward:', np.sum(rewards))"
-    {
+      ]
-      "cell_type": "markdown",
-      "source": [
-        "We'll need to calculate the returns (discounted cumulative reward) for each state action pair"
-      ],
-      "metadata": {
-        "id": "nl6rtNffwhcU"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "KJH-UGKk4tBF"
+      },
+      "outputs": [],
+      "source": [
+        "#this visualizes the complete episode\n",
+        "visualize_one_episode(states, actions)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nl6rtNffwhcU"
+      },
+      "source": [
+        "We'll need to calculate the returns (discounted cumulative reward) for each state action pair"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "FxrItqGPLTq7"
+      },
+      "outputs": [],
       "source": [
         "def calculate_returns(rewards, gamma):\n",
-        "  returns = np.zeros_like(rewards)\n",
+        "  returns = np.zeros(len(rewards))\n",
         "  for c_return in range(len(returns)):\n",
         "    returns[c_return] = calculate_return(rewards[c_return:], gamma)\n",
         "  return returns\n",
@@ -529,26 +622,26 @@
         "  for i in range(len(rewards)):\n",
         "      return_val += rewards[i] * np.power(gamma, i)\n",
         "  return return_val"
-      ],
+      ]
-      "metadata": {
-        "id": "FxrItqGPLTq7"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "This routine does the main work of the Monte Carlo method.  We repeatedly rollout episods, calculate the returns.  Then we figure out the average return for each state action pair, and choose the next policy as the action that has greatest state action value at each state."
-      ],
       "metadata": {
         "id": "DX1KfHRhzUOU"
-      }
+      },
+      "source": [
+        "This routine does the main work of the on-policy Monte Carlo method.  We repeatedly rollout episods, calculate the returns.  Then we figure out the average return for each state action pair, and choose the next policy as the action that has greatest state action value at each state."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "hCghcKlOJXSM"
+      },
+      "outputs": [],
       "source": [
-        "def update_policy_mc(initial_state, transition_probabilities_given_action, reward_structure, stochastic_policy, gamma, n_rollouts=1):\n",
+        "def update_policy_mc(initial_state, transition_probabilities_given_action, reward_structure, terminal_states, stochastic_policy, gamma, n_rollouts=1):\n",
         "  # Create two matrices to store total returns for each action/state pair and the\n",
         "  # number of times we observed that action/state pair\n",
         "  n_state = transition_probabilities_given_action.shape[0]\n",
@@ -574,18 +667,18 @@
         "  state_action_values = state_action_returns_total/( state_action_count+0.00001)\n",
         "  policy = np.argmax(state_action_values, axis=0).astype(int)\n",
         "  return policy, state_action_values\n"
-      ],
+      ]
-      "metadata": {
-        "id": "hCghcKlOJXSM"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "8jWhDlkaKj7Q"
+      },
+      "outputs": [],
       "source": [
         "# Set seed so random numbers always the same\n",
-        "np.random.seed(3)\n",
+        "np.random.seed(0)\n",
         "# Print in compact form\n",
         "np.set_printoptions(precision=3)\n",
         "\n",
@@ -597,32 +690,60 @@
         "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
         "mdp_drawer.draw(layout, policy = policy, rewards = reward_structure)\n",
         "\n",
-        "\n",
+        "terminal_states = [15]\n",
-        "n_policy_update = 5\n",
+        "# Track all the policies so we can visualize them later\n",
+        "all_policies = []\n",
+        "n_policy_update = 2000\n",
         "for c_policy_update in range(n_policy_update):\n",
-        "  # Convert policy to stochastic\n",
+        "    # Convert policy to stochastic\n",
-        "  stochastic_policy = deterministic_policy_to_epsilon_greedy(policy)\n",
+        "    stochastic_policy = deterministic_policy_to_epsilon_greedy(policy)\n",
-        "  # Update policy by Monte Carlo method\n",
+        "    # Update policy by Monte Carlo method\n",
-        "  policy, state_action_values = update_policy_mc(initial_state, transition_probabilities_given_action, reward_structure, stochastic_policy, gamma, n_rollouts=1000)\n",
+        "    policy, state_action_values = update_policy_mc(initial_state, transition_probabilities_given_action, reward_structure, terminal_states, stochastic_policy, gamma, n_rollouts=100)\n",
-        "  print(\"Updated policy\")\n",
+        "    all_policies.append(policy)\n",
-        "  print(policy)\n",
+        "\n",
-        "  mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "    # Print out 10 snapshots of progress\n",
-        "  mdp_drawer.draw(layout, policy = policy, rewards = reward_structure, state_action_values=state_action_values)\n"
+        "    if (c_policy_update % (n_policy_update//10) == 0) or c_policy_update == n_policy_update - 1:\n",
-      ],
+        "        print(\"Updated policy\")\n",
-      "metadata": {
+        "        print(policy)\n",
-        "id": "8jWhDlkaKj7Q"
+        "        mdp_drawer = DrawMDP(n_rows, n_cols)\n",
-      },
+        "        mdp_drawer.draw(layout, policy = policy, rewards = reward_structure, state_action_values=state_action_values)\n",
-      "execution_count": null,
+        "\n",
-      "outputs": []
+        "\n"
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "You can see that the results are quite noisy, but there is a definite improvement from the initial policy."
-      ],
       "metadata": {
         "id": "j7Ny47kTEMzH"
-      }
+      },
+      "source": [
+        "You can see a definite improvement to the policy"
+      ]
     }
-  ]
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3 (ipykernel)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.10.12"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap19/19_4_Temporal_Difference_Methods.ipynb

@@ -1,20 +1,4 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyNEAhORON7DFN1dZMhDK/PO",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
       "cell_type": "markdown",
@@ -28,49 +12,59 @@
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      },
       "source": [
         "# **Notebook 19.4: Temporal difference methods**\n",
         "\n",
         "This notebook investigates temporal difference methods for  tabular reinforcement learning as described in section 19.3.3 of the book\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
-        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
-      ],
+        "\n",
-      "metadata": {
+        "Thanks to [Akshil Patel](https://www.akshilpatel.com) and [Jessica Nicholson](https://jessicanicholson1.github.io) for their help in preparing this notebook."
-        "id": "t9vk9Elugvmi"
+      ]
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "import numpy as np\n",
-        "import matplotlib.pyplot as plt\n",
-        "from PIL import Image"
-      ],
       "metadata": {
         "id": "OLComQyvCIJ7"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "from PIL import Image\n",
+        "from IPython.display import clear_output\n",
+        "from time import sleep\n",
+        "from copy import deepcopy"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZsvrUszPLyEG"
+      },
+      "outputs": [],
       "source": [
         "# Get local copies of components of images\n",
         "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Empty.png\n",
         "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Hole.png\n",
         "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Fish.png\n",
         "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Penguin.png"
-      ],
+      ]
-      "metadata": {
-        "id": "ZsvrUszPLyEG"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Gq1HfJsHN3SB"
+      },
+      "outputs": [],
       "source": [
         "# Ugly class that takes care of drawing pictures like in the book.\n",
         "# You can totally ignore this code!\n",
@@ -253,269 +247,516 @@
         "          self.draw_text(\"%2.2f\"%(state_action_values[3, c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'lc','black')\n",
         "\n",
         "    plt.show()"
-      ],
+      ]
+    },
+    {
+      "cell_type": "markdown",
       "metadata": {
-        "id": "Gq1HfJsHN3SB"
+        "id": "JU8gX59o76xM"
       },
-      "execution_count": null,
+      "source": [
-      "outputs": []
+        "# Penguin Ice Environment\n",
+        "\n",
+        "In this implementation we have designed an icy gridworld that a penguin has to traverse to reach the fish found in the bottom right corner.\n",
+        "\n",
+        "## Environment Description\n",
+        "\n",
+        "Consider having to cross an icy surface to reach the yummy fish. In order to achieve this task as quickly as possible, the penguin needs to waddle along as fast as it can whilst simultaneously avoiding falling into the holes.\n",
+        "\n",
+        "In this icy environment the penguin is at one of the discrete cells in the gridworld. The agent starts each episode on a randomly chosen cell. The environment state dynamics are captured by the transition probabilities $Pr(s_{t+1} |s_t, a_t)$ where $s_t$ is the current state, $a_t$ is the action chosen, and $s_{t+1}$ is the next state at decision stage t. At each decision stage, the penguin can move in one of four directions: $a=0$ means try to go upward, $a=1$, right, $a=2$ down and $a=3$ left.\n",
+        "\n",
+        "However, the ice is slippery, so we don't always go the direction we want to: every time the agent chooses an action, with 0.25 probability, the environment changes the action taken to a differenct action, which is uniformly sampled from the other available actions.\n",
+        "\n",
+        "The rewards are deterministic; the penguin will receive a reward of +3 if it reaches the fish, -2 if it slips into a hole and 0 otherwise.\n",
+        "\n",
+        "Note that as for the states, we've indexed the actions from zero (unlike in the book) so they map to the indices of arrays better"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "eBQ7lTpJQBSe"
+      },
+      "outputs": [],
       "source": [
         "# We're going to work on the problem depicted in figure 19.10a\n",
         "n_rows = 4; n_cols = 4\n",
         "layout = np.zeros(n_rows * n_cols)\n",
         "reward_structure = np.zeros(n_rows * n_cols)\n",
-        "layout[9] = 1 ; reward_structure[9] = -2\n",
+        "layout[9] = 1 ; reward_structure[9] = -2    # Hole\n",
-        "layout[10] = 1; reward_structure[10] = -2\n",
+        "layout[10] = 1; reward_structure[10] = -2   # Hole\n",
-        "layout[14] = 1; reward_structure[14] = -2\n",
+        "layout[14] = 1; reward_structure[14] = -2   # Hole\n",
-        "layout[15] = 2; reward_structure[15] = 3\n",
+        "layout[15] = 2; reward_structure[15] = 3    # Fish\n",
         "initial_state = 0\n",
         "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
         "mdp_drawer.draw(layout, state = initial_state, rewards=reward_structure, draw_state_index = True)"
-      ],
+      ]
-      "metadata": {
-        "id": "eBQ7lTpJQBSe"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "For clarity, the black numbers are the state number and the red numbers are the reward for being in that state.  Note that the states are indexed from 0 rather than 1 as in the book to make the code neater."
-      ],
       "metadata": {
         "id": "6Vku6v_se2IG"
-      }
+      },
+      "source": [
+        "For clarity, the black numbers are the state number and the red numbers are the reward for being in that state.  Note that the states are indexed from 0 rather than 1 as in the book to make the code neater."
+      ]
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "Fhc6DzZNOjiC"
+      },
       "source": [
         "Now let's define the state transition function $Pr(s_{t+1}|s_{t},a)$ in full where $a$ is the actions.  Here $a=0$ means try to go upward, $a=1$, right, $a=2$ down and $a=3$ right.  However, the ice is slippery, so we don't always go the direction we want to.\n",
         "\n",
         "Note that as for the states, we've indexed the actions from zero (unlike in the book) so they map to the indices of arrays better"
-      ],
+      ]
-      "metadata": {
-        "id": "Fhc6DzZNOjiC"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "wROjgnqh76xN"
+      },
+      "outputs": [],
       "source": [
         "transition_probabilities_given_action0 = np.array(\\\n",
-        "[[0.00 , 0.33, 0.00, 0.00,  0.50, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        "[[0.90, 0.05, 0.00, 0.00,  0.85, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.50 , 0.00, 0.33, 0.00,  0.00, 0.50, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.05, 0.85, 0.05, 0.00,  0.00, 0.85, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.33, 0.00, 0.50,  0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.05, 0.85, 0.05,  0.00, 0.00, 0.85, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.33, 0.00,  0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.05, 0.90,  0.00, 0.00, 0.00, 0.85,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.50 , 0.00, 0.00, 0.00,  0.00, 0.17, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.05, 0.00, 0.00, 0.00,  0.05, 0.05, 0.00, 0.00,  0.85, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.34, 0.00, 0.00,  0.25, 0.00, 0.17, 0.00,   0.00, 0.50, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.05, 0.00, 0.00,  0.05, 0.00, 0.05, 0.00,  0.00, 0.85, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.34, 0.00,  0.00, 0.17, 0.00, 0.25,   0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.05, 0.00,  0.00, 0.05, 0.00, 0.05,  0.00, 0.00, 0.85, 0.00,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.50,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.05, 0.05,  0.00, 0.00, 0.00, 0.85,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.17, 0.00, 0.00,   0.75, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.05, 0.05, 0.00, 0.00,  0.85, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.16, 0.00, 0.00,   0.25, 0.00, 0.17, 0.00,   0.00, 0.50, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.05, 0.00, 0.00,  0.05, 0.00, 0.05, 0.00,  0.00, 0.85, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.16, 0.00,   0.00, 0.17, 0.00, 0.25,   0.00, 0.00, 0.50, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.05, 0.00,  0.00, 0.05, 0.00, 0.05,  0.00, 0.00, 0.85, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.75 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.05, 0.05,  0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.05, 0.00, 0.00, 0.00,  0.10, 0.05, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.16, 0.00, 0.00,   0.25, 0.00, 0.25, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.05, 0.00, 0.00,  0.05, 0.05, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.16, 0.00,   0.00, 0.25, 0.00, 0.25 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.05, 0.00,  0.00, 0.05, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.25, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.05,  0.00, 0.00, 0.05, 0.00]])\n",
-        "])\n",
+        "\n",
         "\n",
         "transition_probabilities_given_action1 = np.array(\\\n",
-        "[[0.00 , 0.25, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        "[[0.10, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.75 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.85, 0.05, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.50, 0.00, 0.50,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.85, 0.05, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.50, 0.00,  0.00, 0.00, 0.00, 0.33,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.85, 0.90, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.25 , 0.00, 0.00, 0.00,  0.00, 0.17, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.05, 0.00, 0.00, 0.00, 0.05, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.25, 0.00, 0.00,  0.50, 0.00, 0.17, 0.00,   0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.05, 0.00, 0.00, 0.85, 0.00, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.25, 0.00,  0.00, 0.50, 0.00, 0.33,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.05, 0.00, 0.00, 0.85, 0.00, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.50,  0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.33,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.85, 0.85, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.17, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.05, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.16, 0.00, 0.00,   0.50, 0.00, 0.17, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.85, 0.00, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.16, 0.00,   0.00, 0.50, 0.00, 0.33,   0.00, 0.00, 0.25, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.85, 0.00, 0.05, 0.00, 0.00, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.34,   0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.50 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.85, 0.85, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.10, 0.05, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.16, 0.00, 0.00,   0.75, 0.00, 0.25, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.85, 0.05, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.16, 0.00,   0.00, 0.50, 0.00, 0.50 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.85, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.34,   0.00, 0.00, 0.50, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.85, 0.00]])\n",
-        "])\n",
+        "\n",
         "\n",
         "transition_probabilities_given_action2 = np.array(\\\n",
-        "[[0.00 , 0.25, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        "[[0.10, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.25 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.05, 0.05, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.25, 0.00, 0.25,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.05, 0.05, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.25, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.05, 0.10, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.75 , 0.00, 0.00, 0.00,  0.00, 0.17, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.85, 0.00, 0.00, 0.00, 0.05, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.50, 0.00, 0.00,  0.25, 0.00, 0.17, 0.00,   0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.85, 0.00, 0.00, 0.05, 0.00, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.50, 0.00,  0.00, 0.16, 0.00, 0.25,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.85, 0.00, 0.00, 0.05, 0.00, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.75,  0.00, 0.00, 0.16, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.85, 0.00, 0.00, 0.05, 0.05, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.50, 0.00, 0.00, 0.00,   0.00, 0.17, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.85, 0.00, 0.00, 0.00, 0.05, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.50, 0.00, 0.00,   0.25, 0.00, 0.17, 0.00,   0.00, 0.33, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.85, 0.00, 0.00, 0.05, 0.00, 0.05, 0.00, 0.00, 0.05, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.50, 0.00,   0.00, 0.16, 0.00, 0.25,   0.00, 0.00, 0.33, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.85, 0.00, 0.00, 0.05, 0.00, 0.05, 0.00, 0.00, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.16, 0.00,   0.00, 0.00, 0.00, 0.50 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.85, 0.00, 0.00, 0.05, 0.05, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00,   0.00, 0.33, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.85, 0.00, 0.00, 0.00, 0.90, 0.05, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.50, 0.00, 0.00,   0.50, 0.00, 0.33, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.85, 0.00, 0.00, 0.05, 0.85, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.50, 0.00,   0.00, 0.34, 0.00, 0.50 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.85, 0.00, 0.00, 0.05, 0.85, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.34, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.85, 0.00, 0.00, 0.05, 0.00]])\n",
-        "])\n",
         "\n",
         "transition_probabilities_given_action3 = np.array(\\\n",
-        "[[0.00 , 0.25, 0.00, 0.00,  0.33, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        "[[0.90, 0.85, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.50 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.05, 0.05, 0.85, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.50, 0.00, 0.75,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.05, 0.05, 0.85, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.50, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.05, 0.10, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.50 , 0.00, 0.00, 0.00,  0.00, 0.50, 0.00, 0.00,   0.33, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.05, 0.00, 0.00, 0.00, 0.85, 0.85, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.25, 0.00, 0.00,  0.33, 0.00, 0.50, 0.00,   0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.05, 0.00, 0.00, 0.05, 0.00, 0.85, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.50,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.05, 0.00, 0.00, 0.05, 0.00, 0.85, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.25,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.05, 0.05, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.34, 0.00, 0.00, 0.00,   0.00, 0.50, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.85, 0.85, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.16, 0.00, 0.00,   0.33, 0.00, 0.50, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.05, 0.00, 0.85, 0.00, 0.00, 0.05, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.16, 0.00,   0.00, 0.17, 0.00, 0.50,   0.00, 0.00, 0.25, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.05, 0.00, 0.85, 0.00, 0.00, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.25 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.05, 0.05, 0.00, 0.00, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.34, 0.00, 0.00, 0.00,   0.00, 0.50, 0.00, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.00, 0.90, 0.85, 0.00, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.16, 0.00, 0.00,   0.50, 0.00, 0.50, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.05, 0.05, 0.85, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.16, 0.00,   0.00, 0.25, 0.00, 0.75 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.05, 0.05, 0.00],\n",
-        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.25, 0.00 ],\n",
+        " [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.05, 0.00, 0.00, 0.05, 0.00]])\n",
-        "])\n",
+        "\n",
+        "\n",
         "\n",
         "# Store all of these in a three dimension array\n",
         "# Pr(s_{t+1}=2|s_{t}=1, a_{t}=3] is stored at position [2,1,3]\n",
         "transition_probabilities_given_action = np.concatenate((np.expand_dims(transition_probabilities_given_action0,2),\n",
         "                                                        np.expand_dims(transition_probabilities_given_action1,2),\n",
         "                                                        np.expand_dims(transition_probabilities_given_action2,2),\n",
-        "                                                        np.expand_dims(transition_probabilities_given_action3,2)),axis=2)"
+        "                                                        np.expand_dims(transition_probabilities_given_action3,2)),axis=2)\n",
-      ],
+        "\n",
+        "print('Grid Size:', len(transition_probabilities_given_action[0]))\n",
+        "print()\n",
+        "print('Transition Probabilities for when next state = 2:')\n",
+        "print(transition_probabilities_given_action[2])\n",
+        "print()\n",
+        "print('Transitions Probabilities for when next state = 2 and current state = 1')\n",
+        "print(transition_probabilities_given_action[2][1])\n",
+        "print()\n",
+        "print('Transitions Probabilities for  when next state = 2 and current state = 1 and action = 3 (Left):')\n",
+        "print(transition_probabilities_given_action[2][1][3])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
       "metadata": {
-        "id": "l7rT78BbOgTi"
+        "id": "eblSQ6xZ76xN"
       },
-      "execution_count": null,
+      "source": [
-      "outputs": []
+        "## Implementation Details\n",
+        "\n",
+        "We provide the following methods:\n",
+        "- **`markov_decision_process_step`** - this function simulates $Pr(s_{t+1} | s_{t}, a_{t})$. It randomly selects an action, updates the state based on the transition probabilities associated with the chosen action, and returns the new state, the reward obtained for leaving the current state, and the chosen action. The randomness in action selection and state transitions reflects a random exploration process and the stochastic nature of the MDP, respectively.\n",
+        "\n",
+        "- **`get_policy`** - this function computes a policy that acts greedily with respect to the state-action values. The policy is computed for all states and the action that maximizes the state-action value is chosen for each state. When there are multiple optimal actions, one is chosen at random.\n",
+        "\n",
+        "\n",
+        "You have to implement the following method:\n",
+        "\n",
+        "- **`q_learning_step`** - this function implements a single step of the Q-learning algorithm for reinforcement learning as shown below. The update follows the Q-learning formula and is controlled by parameters such as the learning rate (alpha) and the discount factor $(\\gamma)$. The function returns the updated state-action values matrix.\n",
+        "\n",
+        "$Q(s, a) \\leftarrow (1 - \\alpha) \\cdot Q(s, a) + \\alpha \\cdot \\left(r + \\gamma \\cdot \\max_{a'} Q(s', a')\\right)$"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cKLn4Iam76xN"
+      },
+      "outputs": [],
       "source": [
-        "def q_learning_step(state_action_values, reward, state, new_state, action, gamma, alpha = 0.1):\n",
+        "def get_policy(state_action_values):\n",
+        "    policy = np.zeros(state_action_values.shape[1]) # One action for each state\n",
+        "    for state in range(state_action_values.shape[1]):\n",
+        "        # Break ties for maximising actions randomly\n",
+        "        policy[state] = np.random.choice(np.flatnonzero(state_action_values[:, state] == max(state_action_values[:, state])))\n",
+        "    return policy"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "akjrncMF-FkU"
+      },
+      "outputs": [],
+      "source": [
+        "def markov_decision_process_step(state, transition_probabilities_given_action, reward_structure, terminal_states, action=None):\n",
+        "  # Pick action\n",
+        "  if action is None:\n",
+        "    action = np.random.randint(4)\n",
+        "  # Update the state\n",
+        "  new_state = np.random.choice(a=range(transition_probabilities_given_action.shape[0]), p = transition_probabilities_given_action[:, state,action])\n",
+        "\n",
+        "  # Return the reward -- here the reward is for arriving at the state\n",
+        "  reward = reward_structure[new_state]\n",
+        "  is_terminal = new_state in [terminal_states]\n",
+        "\n",
+        "  return new_state, reward, action, is_terminal"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "5pO6-9ACWhiV"
+      },
+      "outputs": [],
+      "source": [
+        "def q_learning_step(state_action_values, reward, state, new_state, action, is_terminal, gamma, alpha = 0.1):\n",
         "  # TODO -- write this function\n",
         "  # Replace this line\n",
         "  state_action_values_after = np.copy(state_action_values)\n",
         "\n",
         "  return state_action_values_after"
-      ],
+      ]
-      "metadata": {
-        "id": "5pO6-9ACWhiV"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
-      "cell_type": "code",
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "u4OHTTk176xO"
+      },
       "source": [
-        "# This takes a single step from an MDP which just has a completely random policy\n",
+        "Lets run this for a single Q-learning step"
-        "def markov_decision_process_step(state, transition_probabilities_given_action, reward_structure):\n",
+      ]
-        "  # Pick action\n",
-        "  action = np.random.randint(4)\n",
-        "  # Update the state\n",
-        "  new_state = np.random.choice(a=np.arange(0,transition_probabilities_given_action.shape[0]),p = transition_probabilities_given_action[:,state,action])\n",
-        "  # Return the reward -- here the reward is for leaving the state\n",
-        "  reward = reward_structure[state]\n",
-        "\n",
-        "  return new_state, reward, action"
-      ],
-      "metadata": {
-        "id": "akjrncMF-FkU"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Fu5_VjvbSwfJ"
+      },
+      "outputs": [],
       "source": [
         "# Initialize the state-action values to random numbers\n",
         "np.random.seed(0)\n",
         "n_state = transition_probabilities_given_action.shape[0]\n",
         "n_action = transition_probabilities_given_action.shape[2]\n",
+        "terminal_states=[15]\n",
         "state_action_values = np.random.normal(size=(n_action, n_state))\n",
+        "# Hard code value of termination state of finding fish to 0\n",
+        "state_action_values[:, terminal_states] = 0\n",
         "gamma = 0.9\n",
         "\n",
-        "policy = np.argmax(state_action_values, axis=0).astype(int)\n",
+        "policy = get_policy(state_action_values)\n",
         "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
         "mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values, rewards = reward_structure)\n",
         "\n",
         "# Now let's simulate a single Q-learning step\n",
         "initial_state = 9\n",
-        "print(\"Initial state = \", initial_state)\n",
+        "print(\"Initial state =\",initial_state)\n",
-        "new_state, reward, action = markov_decision_process_step(initial_state, transition_probabilities_given_action, reward_structure)\n",
+        "new_state, reward, action, is_terminal = markov_decision_process_step(initial_state, transition_probabilities_given_action, reward_structure, terminal_states)\n",
-        "print(\"Action = \", action)\n",
+        "print(\"Action =\",action)\n",
-        "print(\"New state = \", new_state)\n",
+        "print(\"New state =\",new_state)\n",
-        "print(\"Reward = \", reward)\n",
+        "print(\"Reward =\", reward)\n",
         "\n",
-        "state_action_values_after = q_learning_step(state_action_values, reward, initial_state, new_state, action, gamma)\n",
+        "state_action_values_after = q_learning_step(state_action_values, reward, initial_state, new_state, action, is_terminal, gamma)\n",
         "print(\"Your value:\",state_action_values_after[action, initial_state])\n",
-        "print(\"True value:  0.27650262412468796\")\n",
+        "print(\"True value: 0.3024718977397814\")\n",
         "\n",
-        "policy = np.argmax(state_action_values, axis=0).astype(int)\n",
+        "policy = get_policy(state_action_values)\n",
         "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
         "mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values_after, rewards = reward_structure)\n"
-      ],
+      ]
-      "metadata": {
-        "id": "Fu5_VjvbSwfJ"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "Now let's run this for a while and watch the policy improve"
-      ],
       "metadata": {
         "id": "Ogh0qucmb68J"
-      }
+      },
+      "source": [
+        "Now let's run this for a while (20000) steps and watch the policy improve"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "N6gFYifh76xO"
+      },
+      "outputs": [],
       "source": [
         "# Initialize the state-action values to random numbers\n",
         "np.random.seed(0)\n",
-        "n_state = transition_probabilities_given_action.shape[0]\n",
+        "n_state  = transition_probabilities_given_action.shape[0]\n",
         "n_action = transition_probabilities_given_action.shape[2]\n",
         "state_action_values = np.random.normal(size=(n_action, n_state))\n",
-        "# Hard code termination state of finding fish\n",
+        "\n",
-        "state_action_values[:,n_state-1] = 3.0\n",
+        "# Hard code value of termination state of finding fish to 0\n",
+        "terminal_states = [15]\n",
+        "state_action_values[:, terminal_states] = 0\n",
         "gamma = 0.9\n",
         "\n",
         "# Draw the initial setup\n",
-        "policy = np.argmax(state_action_values, axis=0).astype(int)\n",
+        "print('Initial Policy:')\n",
+        "policy = get_policy(state_action_values)\n",
         "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
         "mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values, rewards = reward_structure)\n",
         "\n",
-        "\n",
+        "state = np.random.randint(n_state-1)\n",
-        "state= np.random.randint(n_state-1)\n",
         "\n",
         "# Run for a number of iterations\n",
-        "for c_iter in range(10000):\n",
+        "for c_iter in range(20000):\n",
-        "  new_state, reward, action = markov_decision_process_step(state, transition_probabilities_given_action, reward_structure)\n",
+        "  new_state, reward, action, is_terminal = markov_decision_process_step(state, transition_probabilities_given_action, reward_structure, terminal_states)\n",
-        "  state_action_values_after = q_learning_step(state_action_values, reward, state, new_state, action, gamma)\n",
+        "  state_action_values_after = q_learning_step(state_action_values, reward, state, new_state, action, is_terminal, gamma)\n",
+        "\n",
         "  # If in termination state, reset state randomly\n",
-        "  if new_state==15:\n",
+        "  if is_terminal:\n",
-        "    state= np.random.randint(n_state-1)\n",
+        "    state = np.random.randint(n_state-1)\n",
         "  else:\n",
         "    state = new_state\n",
-        "  # Update the policy\n",
-        "  state_action_values = np.copy(state_action_values_after)\n",
-        "  policy = np.argmax(state_action_values, axis=0).astype(int)\n",
         "\n",
+        "  # Update the policy\n",
+        "  state_action_values = deepcopy(state_action_values_after)\n",
+        "  policy = get_policy(state_action_values_after)\n",
+        "\n",
+        "print('Final Optimal Policy:')\n",
         "# Draw the final situation\n",
         "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
         "mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values, rewards = reward_structure)"
-      ],
+      ]
+    },
+    {
+      "cell_type": "markdown",
       "metadata": {
-        "id": "qQFhwVqPcCFH"
+        "id": "djPTKuDk76xO"
       },
+      "source": [
+        "Finally, lets run this for a **single** episode and visualize the penguin's actions"
+      ]
+    },
+    {
+      "cell_type": "code",
       "execution_count": null,
-      "outputs": []
+      "metadata": {
+        "id": "pWObQf2h76xO"
+      },
+      "outputs": [],
+      "source": [
+        "def get_one_episode(n_state, state_action_values, terminal_states, gamma):\n",
+        "\n",
+        "    state = np.random.randint(n_state-1)\n",
+        "\n",
+        "    # Create lists to store all the states seen and actions taken throughout the single episode\n",
+        "    all_states  = []\n",
+        "    all_actions = []\n",
+        "\n",
+        "    # Initalize episode termination flag\n",
+        "    done  = False\n",
+        "    # Initialize counter for steps in the episode\n",
+        "    steps = 0\n",
+        "\n",
+        "    all_states.append(state)\n",
+        "\n",
+        "    while not done:\n",
+        "      steps += 1\n",
+        "\n",
+        "      new_state, reward, action, is_terminal = markov_decision_process_step(state, transition_probabilities_given_action, reward_structure, terminal_states)\n",
+        "      all_states.append(new_state)\n",
+        "      all_actions.append(action)\n",
+        "\n",
+        "      state_action_values_after = q_learning_step(state_action_values, reward, state, new_state, action, is_terminal, gamma)\n",
+        "\n",
+        "      # If in termination state, reset state randomly\n",
+        "      if is_terminal:\n",
+        "        state = np.random.randint(n_state-1)\n",
+        "        print(f'Episode Terminated at {steps} Steps')\n",
+        "        # Set episode termination flag\n",
+        "        done = True\n",
+        "      else:\n",
+        "        state = new_state\n",
+        "\n",
+        "      # Update the policy\n",
+        "      state_action_values = deepcopy(state_action_values_after)\n",
+        "      policy = get_policy(state_action_values_after)\n",
+        "\n",
+        "    return all_states, all_actions, policy, state_action_values\n",
+        ""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "P7cbCGT176xO"
+      },
+      "outputs": [],
+      "source": [
+        "def visualize_one_episode(states, actions):\n",
+        "    # Define actions for visualization\n",
+        "    acts = ['up', 'right', 'down', 'left']\n",
+        "\n",
+        "    # Iterate over the states and actions\n",
+        "    for i in range(len(states)):\n",
+        "\n",
+        "        if i == 0:\n",
+        "            print('Starting State:', states[i])\n",
+        "\n",
+        "        elif i == len(states)-1:\n",
+        "            print('Episode Done:', states[i])\n",
+        "\n",
+        "        else:\n",
+        "            print('State', states[i-1])\n",
+        "            a = actions[i]\n",
+        "            print('Action:', acts[a])\n",
+        "            print('Next State:', states[i])\n",
+        "\n",
+        "        # Visualize the current state using the MDP drawer\n",
+        "        mdp_drawer.draw(layout, state=states[i], rewards=reward_structure, draw_state_index=True)\n",
+        "        clear_output(True)\n",
+        "\n",
+        "        # Pause for a short duration to allow observation\n",
+        "        sleep(1.5)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cr98F8PT76xP"
+      },
+      "outputs": [],
+      "source": [
+        "# Initialize the state-action values to random numbers\n",
+        "np.random.seed(2)\n",
+        "n_state  = transition_probabilities_given_action.shape[0]\n",
+        "n_action = transition_probabilities_given_action.shape[2]\n",
+        "state_action_values = np.random.normal(size=(n_action, n_state))\n",
+        "\n",
+        "# Hard code value of termination state of finding fish to 0\n",
+        "terminal_states = [15]\n",
+        "state_action_values[:, terminal_states] = 0\n",
+        "gamma = 0.9\n",
+        "\n",
+        "# Draw the initial setup\n",
+        "print('Initial Policy:')\n",
+        "policy = get_policy(state_action_values)\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values, rewards = reward_structure)\n",
+        "\n",
+        "states, actions, policy, state_action_values = get_one_episode(n_state, state_action_values, terminal_states, gamma)\n",
+        "\n",
+        "print()\n",
+        "print('Final Optimal Policy:')\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values, rewards = reward_structure)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "5zBu1g3776xP"
+      },
+      "outputs": [],
+      "source": [
+        "visualize_one_episode(states, actions)"
+      ]
     }
-  ]
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3 (ipykernel)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.10.12"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap19/19_5_Control_Variates.ipynb

@@ -34,7 +34,7 @@
         "This notebook investigates the method of control variates as described in figure 19.16\n",
         "\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],

Notebooks/Chap20/20_1_Random_Data.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyPkSYbEjOcEmLt8tU6HxNuR",
+      "authorship_tag": "ABX9TyNgBRvfIlngVobKuLE6leM+",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates training the network with random data, as illustrated in figure 20.1.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n"
@@ -45,8 +45,8 @@
     {
       "cell_type": "code",
       "source": [
-        "# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
+        "# Run this if you're in a Colab to install MNIST 1D repository\n",
-        "!git clone https://github.com/greydanus/mnist1d"
+        "!pip install git+https://github.com/greydanus/mnist1d"
       ],
       "metadata": {
         "id": "D5yLObtZCi9J"

Notebooks/Chap20/20_2_Full_Batch_Gradient_Descent.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOo4vm4MXcIvAzVlMCaLikH",
+      "authorship_tag": "ABX9TyO6xuszaG4nNAcWy/3juLkn",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates training a network with full batch gradient descent as in figure 20.2.  There is also a version (notebook takes a long time to run), but this didn't speed it up much for me.  If you run out of CoLab time,  you'll need to download the Python file and run locally.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ],
@@ -44,8 +44,8 @@
     {
       "cell_type": "code",
       "source": [
-        "# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
+        "# Run this if you're in a Colab to install MNIST 1D repository\n",
-        "!git clone https://github.com/greydanus/mnist1d"
+        "!pip install git+https://github.com/greydanus/mnist1d"
       ],
       "metadata": {
         "id": "D5yLObtZCi9J"

Notebooks/Chap20/20_2_Full_Batch_Gradient_Descent_GPU.ipynb

@@ -5,7 +5,7 @@
     "colab": {
       "provenance": [],
       "gpuType": "T4",
-      "authorship_tag": "ABX9TyMjPBfDONmjqTSyEQDP2gjY",
+      "authorship_tag": "ABX9TyOG/5A+P053/x1IfFg52z4V",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -35,7 +35,7 @@
         "\n",
         "This notebook investigates training a network with full batch gradient descent as in figure 20.2.  This is the GPU version (notebook takes a long time to run).  If you are using Colab then you need to go change the runtime type to GPU on the Runtime menu.  Even then, you may run out of time.  If that's the case, you'll need to download the Python file and run locally.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n"
@@ -47,8 +47,8 @@
     {
       "cell_type": "code",
       "source": [
-        "# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
+        "# Run this if you're in a Colab to install MNIST 1D repository\n",
-        "!git clone https://github.com/greydanus/mnist1d"
+        "!pip install git+https://github.com/greydanus/mnist1d"
       ],
       "metadata": {
         "id": "D5yLObtZCi9J"

Notebooks/Chap20/20_3_Lottery_Tickets.ipynb

@@ -32,7 +32,7 @@
         "\n",
         "This notebook investigates the phenomenon of lottery tickets as discussed in section 20.2.7.  This notebook is highly derivative of the MNIST-1D code hosted by Sam Greydanus at https://github.com/greydanus/mnist1d.   \n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
       ]
@@ -43,7 +43,8 @@
         "id": "Sg2i1QmhKW5d"
       },
       "source": [
-        "# Run this if you're in a Colab\n",
+        "# Run this if you're in a Colab to install MNIST 1D repository\n",
+        "!pip install git+https://github.com/greydanus/mnist1d\n",
         "!git clone https://github.com/greydanus/mnist1d"
       ],
       "execution_count": null,
@@ -95,6 +96,12 @@
         "id": "I-vm_gh5xTJs"
       },
       "source": [
+        "from mnist1d.data import get_dataset, get_dataset_args\n",
+        "from mnist1d.utils import set_seed, to_pickle, from_pickle\n",
+        "\n",
+        "import sys ; sys.path.append('./mnist1d/notebooks')\n",
+        "from train import get_model_args, train_model\n",
+        "\n",
         "args = mnist1d.get_dataset_args()\n",
         "data = mnist1d.get_dataset(args=args)  # by default, this will download a pre-made dataset from the GitHub repo\n",
         "\n",
@@ -210,7 +217,7 @@
         "  # we would return [1,1,0,0,1]\n",
         "  # Remember that these are torch tensors and not numpy arrays\n",
         "  # Replace this function:\n",
-        "  mask = torch.ones_like(scores)\n",
+        "  mask = torch.ones_like(absolute_weights)\n",
         "\n",
         "\n",
         "  return mask"
@@ -237,7 +244,6 @@
         "def find_lottery_ticket(model, dataset, args, sparsity_schedule, criteria_fn=None, **kwargs):\n",
         "\n",
         "  criteria_fn = lambda init_params, final_params: final_params.abs()\n",
-        "\n",
         "  init_params = model.get_layer_vecs()\n",
         "  stats = {'train_losses':[], 'test_losses':[], 'train_accs':[], 'test_accs':[]}\n",
         "  models = []\n",
@@ -253,7 +259,7 @@
         "    model.set_layer_masks(masks)\n",
         "\n",
         "    # training process\n",
-        "    results = mnist1d.train_model(dataset, model, args)\n",
+        "    results = train_model(dataset, model, args)\n",
         "    model = results['checkpoints'][-1]\n",
         "\n",
         "    # store stats\n",
@@ -291,7 +297,8 @@
       },
       "source": [
         "# train settings\n",
-        "model_args = mnist1d.get_model_args()\n",
+        "from train import get_model_args, train_model\n",
+        "model_args = get_model_args()\n",
         "model_args.total_steps = 1501\n",
         "model_args.hidden_size = 500\n",
         "model_args.print_every = 5000 # print never\n",

Notebooks/Chap20/20_4_Adversarial_Attacks.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook builds uses the network for classification of MNIST from Notebook 10.5.  The code is adapted from https://nextjournal.com/gkoehler/pytorch-mnist, and uses the fast gradient sign attack of [Goodfellow et al. (2015)](https://arxiv.org/abs/1412.6572).  Having trained, the network, we search for adversarial examples -- inputs which look very similar to class A, but are mistakenly classified as class B.  We do this by starting with a correctly classified example and perturbing it according to the gradients of the network so that the output changes.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ],

Notebooks/Chap21/21_1_Bias_Mitigation.ipynb

@@ -22,7 +22,7 @@
         "\n",
         "This notebook investigates a post-processing method for bias mitigation (see figure 21.2 in the book). It based on this [blog](https://www.borealisai.com/research-blogs/tutorial1-bias-and-fairness-ai/) that I wrote for Borealis AI in 2019, which itself was derived from [this blog](https://research.google.com/bigpicture/attacking-discrimination-in-ml/) by Wattenberg, Viégas, and Hardt.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ]
@@ -137,7 +137,7 @@
         "id": "CfZ-srQtmff2"
       },
       "source": [
-        "Why might the distributions for blue and yellow populations be different? It could be that the behaviour of the populations is identical, but the credit rating algorithm is biased; it may favor one population over another or simply be more noisy for one group. Alternatively, it could be that that the populations genuinely behave differently. In practice, the differences in blue and yellow distributions are probably attributable to a combination of these factors.\n",
+        "Why might the distributions for blue and yellow populations be different? It could be that the behaviour of the populations is identical, but the credit rating algorithm is biased; it may favor one population over another or simply be more noisy for one group. Alternatively, it could be that the populations genuinely behave differently. In practice, the differences in blue and yellow distributions are probably attributable to a combination of these factors.\n",
         "\n",
         "Let’s assume that we can’t retrain the credit score prediction algorithm; our job is to adjudicate whether each individual is refused the loan ($\\hat{y}=0$)\n",
         " or granted it ($\\hat{y}=1$). Since we only have the credit score\n",
@@ -328,7 +328,7 @@
       },
       "outputs": [],
       "source": [
-        "# TO DO -- try to change the two thresholds so the overall probability of getting the loan is 0.6 for each group\n",
+        "# TODO -- try to change the two thresholds so the overall probability of getting the loan is 0.6 for each group\n",
         "# Change the values in these lines\n",
         "tau0 = 0.3\n",
         "tau1 = -0.1\n",
@@ -382,7 +382,7 @@
       "source": [
         "# Equal opportunity:\n",
         "\n",
-        "The thresholds are chosen so that so that the true positive rate is is the same for both population. Of the people who pay back the loan, the same proportion are offered credit in each group. In terms of the two ROC curves, it means choosing thresholds so that the vertical position on each curve is the same without regard for the horizontal position."
+        "The thresholds are chosen so that so that the true positive rate is the same for both population. Of the people who pay back the loan, the same proportion are offered credit in each group. In terms of the two ROC curves, it means choosing thresholds so that the vertical position on each curve is the same without regard for the horizontal position."
       ]
     },
     {
@@ -393,7 +393,7 @@
       },
       "outputs": [],
       "source": [
-        "# TO DO -- try to change the two thresholds so the true positive are 0.8 for each group\n",
+        "# TODO --try to change the two thresholds so the true positive are 0.8 for each group\n",
         "# Change the values in these lines so that both points on the curves have a height of 0.8\n",
         "tau0 = -0.1\n",
         "tau1 = -0.7\n",

Notebooks/Chap21/21_2_Explainability.ipynb

@@ -33,7 +33,7 @@
         "\n",
         "This notebook investigates the LIME explainability method as depicted in figure 21.3 of the book.\n",
         "\n",
-        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TODO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
       ],

Notebooks/LICENSE (MIT)

@@ -0,0 +1,7 @@
+Copyright 2023 Simon Prince
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.