Created using Colaboratory

Fix typos in notebooks

CM20315_2023/CM20315_Coursework_I.ipynb

@@ -0,0 +1,280 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyPNASgWoh4kBvxFP0xkN/I4",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/CM20315_2023/CM20315_Coursework_I.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Coursework I -- Model hyperparameters\n",
+        "\n",
+        "The goal of the coursework is to modify a simple bit of numpy code that trains a network and measures the performance on a validation set for the MNIST 1D dataset.\n",
+        "\n",
+        "In this coursework, you need to modify the **model hyperparameters** (only) to improve the performance over the current attempt.  This could mean the number of layers, the number of hidden units per layer, or the type of activation function, or any combination of the three.\n",
+        "\n",
+        "The only constraint is that you MUST use a fully connected network (no convolutional networks for now if you have read ahead in the book).\n",
+        "\n",
+        "You must improve the performance by at least 2% to get full marks.\n",
+        "\n",
+        "You will need to upload three things to Moodle:\n",
+        "1.   The image that this notebook saves (click the folder icon on the left on colab to download it)\n",
+        "2.   The lines of code you changed\n",
+        "3.   The whole notebook as a .ipynb file.  You can do this on the File menu\n",
+        "\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import os\n",
+        "import torch, torch.nn as nn\n",
+        "from torch.utils.data import TensorDataset, DataLoader\n",
+        "from torch.optim.lr_scheduler import StepLR\n",
+        "import matplotlib.pyplot as plt\n",
+        "import random\n",
+        "import gdown"
+      ],
+      "metadata": {
+        "id": "YrXWAH7sUWvU"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "\n",
+        "# Run this once to copy the train and validation data to your CoLab environment\n",
+        "# or download from my github to your local machine if you are doing this locally\n",
+        "if not os.path.exists('./Data.zip'):\n",
+        "  !gdown 1HtnCrncY6dFCYqzgPf1HtPVAerTpwFRm\n",
+        "  !unzip Data.zip"
+      ],
+      "metadata": {
+        "id": "wScBGXXFVadm"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Load in the data\n",
+        "train_data_x = np.load('train_data_x.npy')\n",
+        "val_data_y = np.load('val_data_y.npy')\n",
+        "train_data_y = np.load('train_data_y.npy')\n",
+        "val_data_x = np.load('val_data_x.npy')\n",
+        "# Print out sizes\n",
+        "print(\"Train data: %d examples (columns), each of which has %d dimensions (rows)\"%((train_data_x.shape[1],train_data_x.shape[0])))\n",
+        "print(\"Validation data: %d examples (columns), each of which has %d dimensions (rows)\"%((val_data_x.shape[1],val_data_x.shape[0])))"
+      ],
+      "metadata": {
+        "id": "8bKADvLHbiV5"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Define the network"
+      ],
+      "metadata": {
+        "id": "_sFvRDGrl4qe"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# YOU SHOULD ONLY CHANGE THIS CELL!\n",
+        "\n",
+        "# There are 40 input dimensions and 10 output dimensions for this data\n",
+        "# The inputs correspond to the 40 offsets in the MNIST1D template.\n",
+        "D_i = 40\n",
+        "# The outputs correspond to the 10 digits\n",
+        "D_o = 10\n",
+        "\n",
+        "# Number of hidden units in layers 1 and 2\n",
+        "D_1 = 100\n",
+        "D_2 = 100\n",
+        "\n",
+        "# create model with two hidden layers\n",
+        "model = nn.Sequential(\n",
+        "nn.Linear(D_i, D_1),\n",
+        "nn.ReLU(),\n",
+        "nn.Linear(D_1, D_2),\n",
+        "nn.ReLU(),\n",
+        "nn.Linear(D_2, D_o))"
+      ],
+      "metadata": {
+        "id": "FslroPJJffrh"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# He initialization of weights\n",
+        "def weights_init(layer_in):\n",
+        "  if isinstance(layer_in, nn.Linear):\n",
+        "    nn.init.kaiming_uniform_(layer_in.weight)\n",
+        "    layer_in.bias.data.fill_(0.0)"
+      ],
+      "metadata": {
+        "id": "YgLaex1pfhqz"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# You need all this stuff to ensure that PyTorch is deterministic\n",
+        "def set_seed(seed):\n",
+        "    torch.manual_seed(seed)\n",
+        "    torch.cuda.manual_seed_all(seed)\n",
+        "    torch.backends.cudnn.deterministic = True\n",
+        "    torch.backends.cudnn.benchmark = False\n",
+        "    np.random.seed(seed)\n",
+        "    random.seed(seed)\n",
+        "    os.environ['PYTHONHASHSEED'] = str(seed)"
+      ],
+      "metadata": {
+        "id": "zXRmxCQNnL_M"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set seed so always get same result (do not change)\n",
+        "set_seed(1)\n",
+        "\n",
+        "# choose cross entropy loss function (equation 5.24 in the loss notes)\n",
+        "loss_function = nn.CrossEntropyLoss()\n",
+        "# construct SGD optimizer and initialize learning rate and momentum\n",
+        "optimizer = torch.optim.SGD(model.parameters(), lr = 0.05, momentum=0.9)\n",
+        "# object that decreases learning rate by half every 10 epochs\n",
+        "scheduler = StepLR(optimizer, step_size=10, gamma=0.5)\n",
+        "# create 100 dummy data points and store in data loader class\n",
+        "x_train = torch.tensor(train_data_x.transpose().astype('float32'))\n",
+        "y_train = torch.tensor(train_data_y.astype('long'))\n",
+        "x_val= torch.tensor(val_data_x.transpose().astype('float32'))\n",
+        "y_val = torch.tensor(val_data_y.astype('long'))\n",
+        "\n",
+        "# load the data into a class that creates the batches\n",
+        "data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=100, shuffle=True, worker_init_fn=np.random.seed(1))\n",
+        "\n",
+        "# Initialize model weights\n",
+        "model.apply(weights_init)\n",
+        "\n",
+        "# loop over the dataset n_epoch times\n",
+        "n_epoch = 50\n",
+        "# store the loss and the % correct at each epoch\n",
+        "losses_train = np.zeros((n_epoch))\n",
+        "errors_train = np.zeros((n_epoch))\n",
+        "losses_val = np.zeros((n_epoch))\n",
+        "errors_val = np.zeros((n_epoch))\n",
+        "\n",
+        "for epoch in range(n_epoch):\n",
+        "  # loop over batches\n",
+        "  for i, data in enumerate(data_loader):\n",
+        "    # retrieve inputs and labels for this batch\n",
+        "    x_batch, y_batch = data\n",
+        "    # zero the parameter gradients\n",
+        "    optimizer.zero_grad()\n",
+        "    # forward pass -- calculate model output\n",
+        "    pred = model(x_batch)\n",
+        "    # compute the lss\n",
+        "    loss = loss_function(pred, y_batch)\n",
+        "    # backward pass\n",
+        "    loss.backward()\n",
+        "    # SGD update\n",
+        "    optimizer.step()\n",
+        "\n",
+        "  # Run whole dataset to get statistics -- normally wouldn't do this\n",
+        "  pred_train = model(x_train)\n",
+        "  pred_val = model(x_val)\n",
+        "  _, predicted_train_class = torch.max(pred_train.data, 1)\n",
+        "  _, predicted_val_class = torch.max(pred_val.data, 1)\n",
+        "  errors_train[epoch] = 100 - 100 * (predicted_train_class == y_train).float().sum() / len(y_train)\n",
+        "  errors_val[epoch]= 100 - 100 * (predicted_val_class == y_val).float().sum() / len(y_val)\n",
+        "  losses_train[epoch] = loss_function(pred_train, y_train).item()\n",
+        "  losses_val[epoch]= loss_function(pred_val, y_val).item()\n",
+        "  print(f'Epoch {epoch:5d}, train loss {losses_train[epoch]:.6f}, train error {errors_train[epoch]:3.2f},  val loss {losses_val[epoch]:.6f}, percent error {errors_val[epoch]:3.2f}')\n",
+        "\n",
+        "  # tell scheduler to consider updating learning rate\n",
+        "  scheduler.step()\n",
+        "\n",
+        "# Plot the results\n",
+        "fig, ax = plt.subplots()\n",
+        "ax.plot(errors_train,'r-',label='train')\n",
+        "ax.plot(errors_val,'b-',label='validation')\n",
+        "ax.set_ylim(0,100); ax.set_xlim(0,n_epoch)\n",
+        "ax.set_xlabel('Epoch'); ax.set_ylabel('Error')\n",
+        "ax.set_title('Part I: Validation Result %3.2f'%(errors_val[-1]))\n",
+        "ax.legend()\n",
+        "ax.plot([0,n_epoch],[37.45, 37.45],'k:') # Original results. You should be better than this!\n",
+        "plt.savefig('Coursework_I_Results.png',format='png')\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "NYw8I_3mmX5c"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Leave this all commented for now\n",
+        "# We'll see how well you did on the test data after the coursework is submitted\n",
+        "\n",
+        "# # I haven't given you this yet, leave commented\n",
+        "# test_data_x = np.load('test_data_x.npy')\n",
+        "# test_data_y = np.load('test_data_y.npy')\n",
+        "# x_test = torch.tensor(test_data_x.transpose().astype('float32'))\n",
+        "# y_test = torch.tensor(test_data_y.astype('long'))\n",
+        "# pred_test = model(x_test)\n",
+        "# _, predicted_test_class = torch.max(pred_test.data, 1)\n",
+        "# errors_test = 100 - 100 * (predicted_test_class == y_test).float().sum() / len(y_test)\n",
+        "# print(\"Test error = %3.3f\"%(errors_test))"
+      ],
+      "metadata": {
+        "id": "O7nBz-R84QdJ"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

CM20315_2023/CM20315_Coursework_II.ipynb

@@ -0,0 +1,276 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyM+iKos5DEoHUxL8+9oxA2A",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/CM20315_2023/CM20315_Coursework_II.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Coursework II -- Training hyperparameters\n",
+        "\n",
+        "The goal of the coursework is to modify a simple bit of numpy code that trains a network and measures the performance on a validation set for the MNist 1D dataset.\n",
+        "\n",
+        "In this coursework, you need to modify the **training hyperparameters** (only) to improve the performance over the current attempt. This could mean the training algorithm, learning rate, learning rate schedule, momentum term, initialization etc.  \n",
+        "\n",
+        "You must improve the performance by at least 2% to get full marks.\n",
+        "\n",
+        "You will need to upload three things to Moodle:\n",
+        "1.   The image that this notebook saves (click the folder icon on the left on colab to download it)\n",
+        "2.   The lines of code you changed\n",
+        "3.   The whole notebook as a .ipynb file.  You can do this on the File menu"
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import os\n",
+        "import torch, torch.nn as nn\n",
+        "from torch.utils.data import TensorDataset, DataLoader\n",
+        "from torch.optim.lr_scheduler import StepLR\n",
+        "import matplotlib.pyplot as plt\n",
+        "import random\n",
+        "import gdown"
+      ],
+      "metadata": {
+        "id": "YrXWAH7sUWvU"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Run this once to copy the train and validation data to your CoLab environment\n",
+        "if not os.path.exists('./Data.zip'):\n",
+        "  !gdown 1HtnCrncY6dFCYqzgPf1HtPVAerTpwFRm\n",
+        "  !unzip Data.zip"
+      ],
+      "metadata": {
+        "id": "wScBGXXFVadm"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Load in the data\n",
+        "train_data_x = np.load('train_data_x.npy',allow_pickle=True)\n",
+        "train_data_y = np.load('train_data_y.npy',allow_pickle=True)\n",
+        "val_data_x = np.load('val_data_x.npy',allow_pickle=True)\n",
+        "val_data_y = np.load('val_data_y.npy',allow_pickle=True)\n",
+        "# Print out sizes\n",
+        "print(\"Train data: %d examples (columns), each of which has %d dimensions (rows)\"%((train_data_x.shape[1],train_data_x.shape[0])))\n",
+        "print(\"Validation data: %d examples (columns), each of which has %d dimensions (rows)\"%((val_data_x.shape[1],val_data_x.shape[0])))"
+      ],
+      "metadata": {
+        "id": "8bKADvLHbiV5"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Define the network"
+      ],
+      "metadata": {
+        "id": "_sFvRDGrl4qe"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# YOU SHOULD NOT CHANGE THIS CELL!\n",
+        "\n",
+        "# There are 40 input dimensions and 10 output dimensions for this data\n",
+        "# The inputs correspond to the 40 offsets in the MNIST1D template.\n",
+        "D_i = 40\n",
+        "# The outputs correspond to the 10 digits\n",
+        "D_o = 10\n",
+        "\n",
+        "# Number of hidden units in layers 1 and 2\n",
+        "D_1 = 100\n",
+        "D_2 = 100\n",
+        "\n",
+        "# create model with two hidden layers\n",
+        "model = nn.Sequential(\n",
+        "nn.Linear(D_i, D_1),\n",
+        "nn.ReLU(),\n",
+        "nn.Linear(D_1, D_2),\n",
+        "nn.ReLU(),\n",
+        "nn.Linear(D_2, D_o))"
+      ],
+      "metadata": {
+        "id": "FslroPJJffrh"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# He initialization of weights\n",
+        "def weights_init(layer_in):\n",
+        "  if isinstance(layer_in, nn.Linear):\n",
+        "    nn.init.kaiming_uniform_(layer_in.weight)\n",
+        "    layer_in.bias.data.fill_(0.0)"
+      ],
+      "metadata": {
+        "id": "YgLaex1pfhqz"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# You need all this stuff to ensure that PyTorch is deterministic\n",
+        "def set_seed(seed):\n",
+        "    torch.manual_seed(seed)\n",
+        "    torch.cuda.manual_seed_all(seed)\n",
+        "    torch.backends.cudnn.deterministic = True\n",
+        "    torch.backends.cudnn.benchmark = False\n",
+        "    np.random.seed(seed)\n",
+        "    random.seed(seed)\n",
+        "    os.environ['PYTHONHASHSEED'] = str(seed)"
+      ],
+      "metadata": {
+        "id": "zXRmxCQNnL_M"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set seed so always get same result (do not change)\n",
+        "set_seed(1)\n",
+        "\n",
+        "# choose cross entropy loss function (equation 5.24 in the loss notes)\n",
+        "loss_function = nn.CrossEntropyLoss()\n",
+        "# construct SGD optimizer and initialize learning rate and momentum\n",
+        "optimizer = torch.optim.SGD(model.parameters(), lr = 0.05, momentum=0.9)\n",
+        "# object that decreases learning rate by half every 10 epochs\n",
+        "scheduler = StepLR(optimizer, step_size=10, gamma=0.5)\n",
+        "# create 100 dummy data points and store in data loader class\n",
+        "x_train = torch.tensor(train_data_x.transpose().astype('float32'))\n",
+        "print(x_train.shape)\n",
+        "y_train = torch.tensor(train_data_y.astype('long'))\n",
+        "print(y_train.shape)\n",
+        "x_val= torch.tensor(val_data_x.transpose().astype('float32'))\n",
+        "y_val = torch.tensor(val_data_y.astype('long'))\n",
+        "\n",
+        "# load the data into a class that creates the batches\n",
+        "data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=100, shuffle=True, worker_init_fn=np.random.seed(1))\n",
+        "\n",
+        "# Initialize model weights\n",
+        "model.apply(weights_init)\n",
+        "\n",
+        "# loop over the dataset n_epoch times\n",
+        "n_epoch = 50\n",
+        "# store the loss and the % correct at each epoch\n",
+        "losses_train = np.zeros((n_epoch))\n",
+        "errors_train = np.zeros((n_epoch))\n",
+        "losses_val = np.zeros((n_epoch))\n",
+        "errors_val = np.zeros((n_epoch))\n",
+        "\n",
+        "for epoch in range(n_epoch):\n",
+        "  # loop over batches\n",
+        "  for i, data in enumerate(data_loader):\n",
+        "    # retrieve inputs and labels for this batch\n",
+        "    x_batch, y_batch = data\n",
+        "    # zero the parameter gradients\n",
+        "    optimizer.zero_grad()\n",
+        "    # forward pass -- calculate model output\n",
+        "    pred = model(x_batch)\n",
+        "    # compute the lss\n",
+        "    loss = loss_function(pred, y_batch)\n",
+        "    # backward pass\n",
+        "    loss.backward()\n",
+        "    # SGD update\n",
+        "    optimizer.step()\n",
+        "\n",
+        "  # Run whole dataset to get statistics -- normally wouldn't do this\n",
+        "  pred_train = model(x_train)\n",
+        "  pred_val = model(x_val)\n",
+        "  _, predicted_train_class = torch.max(pred_train.data, 1)\n",
+        "  _, predicted_val_class = torch.max(pred_val.data, 1)\n",
+        "  errors_train[epoch] = 100 - 100 * (predicted_train_class == y_train).float().sum() / len(y_train)\n",
+        "  errors_val[epoch]= 100 - 100 * (predicted_val_class == y_val).float().sum() / len(y_val)\n",
+        "  losses_train[epoch] = loss_function(pred_train, y_train).item()\n",
+        "  losses_val[epoch]= loss_function(pred_val, y_val).item()\n",
+        "  print(f'Epoch {epoch:5d}, train loss {losses_train[epoch]:.6f}, train error {errors_train[epoch]:3.2f},  val loss {losses_val[epoch]:.6f}, percent error {errors_val[epoch]:3.2f}')\n",
+        "\n",
+        "  # tell scheduler to consider updating learning rate\n",
+        "  scheduler.step()\n",
+        "\n",
+        "# Plot the results\n",
+        "fig, ax = plt.subplots()\n",
+        "ax.plot(errors_train,'r-',label='train')\n",
+        "ax.plot(errors_val,'b-',label='validation')\n",
+        "ax.set_ylim(0,100); ax.set_xlim(0,n_epoch)\n",
+        "ax.set_xlabel('Epoch'); ax.set_ylabel('Error')\n",
+        "ax.set_title('Part II: Validation Result %3.2f'%(errors_val[-1]))\n",
+        "ax.legend()\n",
+        "ax.plot([0,n_epoch],[37.45, 37.45],'k:') # Original results. You should be better than this!\n",
+        "plt.savefig('Coursework_II_Results.png',format='png')\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "NYw8I_3mmX5c"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Leave this all commented for now\n",
+        "# We'll see how well you did on the test data after the coursework is submitted\n",
+        "\n",
+        "# # I haven't given you this yet, leave commented\n",
+        "# test_data_x = np.load('test_data_x.npy')\n",
+        "# test_data_y = np.load('test_data_y.npy')\n",
+        "# x_test = torch.tensor(test_data_x.transpose().astype('float32'))\n",
+        "# y_test = torch.tensor(test_data_y.astype('long'))\n",
+        "# pred_test = model(x_test)\n",
+        "# _, predicted_test_class = torch.max(pred_test.data, 1)\n",
+        "# errors_test = 100 - 100 * (predicted_test_class == y_test).float().sum() / len(y_test)\n",
+        "# print(\"Test error = %3.3f\"%(errors_test))"
+      ],
+      "metadata": {
+        "id": "O7nBz-R84QdJ"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

CM20315_2023/CM20315_Coursework_III.ipynb

@@ -0,0 +1,275 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyNDH1z3I76jjglu3o0LSlZc",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/CM20315_2023/CM20315_Coursework_III.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Coursework III -- Regularization\n",
+        "\n",
+        "The goal of the coursework is to modify a simple bit of numpy code that trains a network and measures the performance on a validation set for the MNist 1D dataset.\n",
+        "\n",
+        "In this coursework, you need add **regularization** of some kind to improve the performance.  Anything from chapter 9 of the book or anything else you can find is fine *except* early stopping.  You must not change the model hyperparameters or the training algorithm.\n",
+        "\n",
+        "You must improve the performance by at least 2% to get full marks.\n",
+        "\n",
+        "You will need to upload three things to Moodle:\n",
+        "1.   The image that this notebook saves (click the folder icon on the left on colab to download it)\n",
+        "2.   The lines of code you changed\n",
+        "3.   The whole notebook as a .ipynb file.  You can do this on the File menu\n",
+        "\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import os\n",
+        "import torch, torch.nn as nn\n",
+        "from torch.utils.data import TensorDataset, DataLoader\n",
+        "from torch.optim.lr_scheduler import StepLR\n",
+        "import matplotlib.pyplot as plt\n",
+        "import random\n",
+        "import gdown"
+      ],
+      "metadata": {
+        "id": "YrXWAH7sUWvU"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Run this once to copy the train and validation data to your CoLab environment\n",
+        "if not os.path.exists('./Data.zip'):\n",
+        "  !gdown 1HtnCrncY6dFCYqzgPf1HtPVAerTpwFRm\n",
+        "  !unzip Data.zip"
+      ],
+      "metadata": {
+        "id": "wScBGXXFVadm"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Load in the data\n",
+        "train_data_x = np.load('train_data_x.npy')\n",
+        "train_data_y = np.load('train_data_y.npy')\n",
+        "val_data_x = np.load('val_data_x.npy')\n",
+        "val_data_y = np.load('val_data_y.npy')\n",
+        "# Print out sizes\n",
+        "print(\"Train data: %d examples (columns), each of which has %d dimensions (rows)\"%((train_data_x.shape[1],train_data_x.shape[0])))\n",
+        "print(\"Validation data: %d examples (columns), each of which has %d dimensions (rows)\"%((val_data_x.shape[1],val_data_x.shape[0])))"
+      ],
+      "metadata": {
+        "id": "8bKADvLHbiV5"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Define the network"
+      ],
+      "metadata": {
+        "id": "_sFvRDGrl4qe"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# There are 40 input dimensions and 10 output dimensions for this data\n",
+        "# The inputs correspond to the 40 offsets in the MNIST1D template.\n",
+        "D_i = 40\n",
+        "# The outputs correspond to the 10 digits\n",
+        "D_o = 10\n",
+        "\n",
+        "# Number of hidden units in layers 1 and 2\n",
+        "D_1 = 100\n",
+        "D_2 = 100\n",
+        "\n",
+        "# create model with two hidden layers\n",
+        "model = nn.Sequential(\n",
+        "nn.Linear(D_i, D_1),\n",
+        "nn.ReLU(),\n",
+        "nn.Linear(D_1, D_2),\n",
+        "nn.ReLU(),\n",
+        "nn.Linear(D_2, D_o))"
+      ],
+      "metadata": {
+        "id": "FslroPJJffrh"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# He initialization of weights\n",
+        "def weights_init(layer_in):\n",
+        "  if isinstance(layer_in, nn.Linear):\n",
+        "    nn.init.kaiming_uniform_(layer_in.weight)\n",
+        "    layer_in.bias.data.fill_(0.0)"
+      ],
+      "metadata": {
+        "id": "YgLaex1pfhqz"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# You need all this stuff to ensure that PyTorch is deterministic\n",
+        "def set_seed(seed):\n",
+        "    torch.manual_seed(seed)\n",
+        "    torch.cuda.manual_seed_all(seed)\n",
+        "    torch.backends.cudnn.deterministic = True\n",
+        "    torch.backends.cudnn.benchmark = False\n",
+        "    np.random.seed(seed)\n",
+        "    random.seed(seed)\n",
+        "    os.environ['PYTHONHASHSEED'] = str(seed)"
+      ],
+      "metadata": {
+        "id": "zXRmxCQNnL_M"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set seed so always get same result (do not change)\n",
+        "set_seed(1)\n",
+        "\n",
+        "# choose cross entropy loss function (equation 5.24 in the loss notes)\n",
+        "loss_function = nn.CrossEntropyLoss()\n",
+        "# construct SGD optimizer and initialize learning rate and momentum\n",
+        "optimizer = torch.optim.SGD(model.parameters(), lr = 0.05, momentum=0.9)\n",
+        "# object that decreases learning rate by half every 10 epochs\n",
+        "scheduler = StepLR(optimizer, step_size=10, gamma=0.5)\n",
+        "# create 100 dummy data points and store in data loader class\n",
+        "x_train = torch.tensor(train_data_x.transpose().astype('float32'))\n",
+        "y_train = torch.tensor(train_data_y.astype('long'))\n",
+        "x_val= torch.tensor(val_data_x.transpose().astype('float32'))\n",
+        "y_val = torch.tensor(val_data_y.astype('long'))\n",
+        "\n",
+        "# load the data into a class that creates the batches\n",
+        "data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=100, shuffle=True, worker_init_fn=np.random.seed(1))\n",
+        "\n",
+        "# Initialize model weights\n",
+        "model.apply(weights_init)\n",
+        "\n",
+        "# loop over the dataset n_epoch times\n",
+        "n_epoch = 50\n",
+        "# store the loss and the % correct at each epoch\n",
+        "losses_train = np.zeros((n_epoch))\n",
+        "errors_train = np.zeros((n_epoch))\n",
+        "losses_val = np.zeros((n_epoch))\n",
+        "errors_val = np.zeros((n_epoch))\n",
+        "\n",
+        "for epoch in range(n_epoch):\n",
+        "  # loop over batches\n",
+        "  for i, data in enumerate(data_loader):\n",
+        "    # retrieve inputs and labels for this batch\n",
+        "    x_batch, y_batch = data\n",
+        "    # zero the parameter gradients\n",
+        "    optimizer.zero_grad()\n",
+        "    # forward pass -- calculate model output\n",
+        "    pred = model(x_batch)\n",
+        "    # compute the lss\n",
+        "    loss = loss_function(pred, y_batch)\n",
+        "    # backward pass\n",
+        "    loss.backward()\n",
+        "    # SGD update\n",
+        "    optimizer.step()\n",
+        "\n",
+        "  # Run whole dataset to get statistics -- normally wouldn't do this\n",
+        "  pred_train = model(x_train)\n",
+        "  pred_val = model(x_val)\n",
+        "  _, predicted_train_class = torch.max(pred_train.data, 1)\n",
+        "  _, predicted_val_class = torch.max(pred_val.data, 1)\n",
+        "  errors_train[epoch] = 100 - 100 * (predicted_train_class == y_train).float().sum() / len(y_train)\n",
+        "  errors_val[epoch]= 100 - 100 * (predicted_val_class == y_val).float().sum() / len(y_val)\n",
+        "  losses_train[epoch] = loss_function(pred_train, y_train).item()\n",
+        "  losses_val[epoch]= loss_function(pred_val, y_val).item()\n",
+        "  print(f'Epoch {epoch:5d}, train loss {losses_train[epoch]:.6f}, train error {errors_train[epoch]:3.2f},  val loss {losses_val[epoch]:.6f}, percent error {errors_val[epoch]:3.2f}')\n",
+        "\n",
+        "  # tell scheduler to consider updating learning rate\n",
+        "  scheduler.step()\n",
+        "\n",
+        "# Plot the results\n",
+        "fig, ax = plt.subplots()\n",
+        "ax.plot(errors_train,'r-',label='train')\n",
+        "ax.plot(errors_val,'b-',label='validation')\n",
+        "ax.set_ylim(0,100); ax.set_xlim(0,n_epoch)\n",
+        "ax.set_xlabel('Epoch'); ax.set_ylabel('Error')\n",
+        "ax.set_title('Part III: Validation Result %3.2f'%(errors_val[-1]))\n",
+        "ax.legend()\n",
+        "ax.plot([0,n_epoch],[37.45, 37.45],'k:') # Original results. You should be better than this!\n",
+        "plt.savefig('Coursework_III_Results.png',format='png')\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "NYw8I_3mmX5c"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Leave this all commented for now\n",
+        "# We'll see how well you did on the test data after the coursework is submitted\n",
+        "\n",
+        "\n",
+        "# # I haven't given you this yet, leave commented\n",
+        "# test_data_x = np.load('test_data_x.npy')\n",
+        "# test_data_y = np.load('test_data_y.npy')\n",
+        "# x_test = torch.tensor(test_data_x.transpose().astype('float32'))\n",
+        "# y_test = torch.tensor(test_data_y.astype('long'))\n",
+        "# pred_test = model(x_test)\n",
+        "# _, predicted_test_class = torch.max(pred_test.data, 1)\n",
+        "# errors_test = 100 - 100 * (predicted_test_class == y_test).float().sum() / len(y_test)\n",
+        "# print(\"Test error = %3.3f\"%(errors_test))"
+      ],
+      "metadata": {
+        "id": "O7nBz-R84QdJ"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

CM20315_2023/CM20315_Coursework_IV.ipynb

@@ -0,0 +1,212 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyNNnZyVCX9glFJGIC8BwtVT",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/CM20315_2023/CM20315_Coursework_IV.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Coursework IV\n",
+        "\n",
+        "This coursework explores the geometry of high dimensional spaces.  It doesn't behave how you would expect and all your intuitions are wrong!  You will write code and it will give you three numerical answers that you need to type into Moodle."
+      ],
+      "metadata": {
+        "id": "EjLK-kA1KnYX"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4ESMmnkYEVAb"
+      },
+      "outputs": [],
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "import scipy.special as sci"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Part (a)\n",
+        "\n",
+        "In part (a) of the practical, we investigate how close random points are in 2D, 100D, and 1000D.   In each case, we generate 1000 points and calculate the Euclidean distance between each pair.  You should find that in 1000D, the furthest two points are only slightly further apart than the nearest points.  Weird!"
+      ],
+      "metadata": {
+        "id": "MonbPEitLNgN"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Fix the random seed so we all have the same random numbers\n",
+        "np.random.seed(0)\n",
+        "n_data = 1000\n",
+        "# Create 1000 data examples (columns) each with 2 dimensions (rows)\n",
+        "n_dim = 2\n",
+        "x_2D = np.random.normal(size=(n_dim,n_data))\n",
+        "# Create 1000 data examples (columns) each with 100 dimensions (rows)\n",
+        "n_dim = 100\n",
+        "x_100D = np.random.normal(size=(n_dim,n_data))\n",
+        "# Create 1000 data examples (columns) each with 1000 dimensions (rows)\n",
+        "n_dim = 1000\n",
+        "x_1000D = np.random.normal(size=(n_dim,n_data))\n",
+        "\n",
+        "# These values should be the same, otherwise your answer will be wrong\n",
+        "# Get in touch if they are not!\n",
+        "print('Sum of your data is %3.3f, Should be %3.3f'%(np.sum(x_1000D),1036.321))"
+      ],
+      "metadata": {
+        "id": "vZSHVmcWEk14"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def distance_ratio(x):\n",
+        "  # TODO -- replace the two lines below to calculate the largest and smallest Euclidean distance between\n",
+        "  # the data points in the columns of x.  DO NOT include the distance between the data point\n",
+        "  # and itself (which is obviously zero)\n",
+        "  smallest_dist = 1.0\n",
+        "  largest_dist = 1.0\n",
+        "\n",
+        "  # Calculate the ratio and return\n",
+        "  dist_ratio = largest_dist / smallest_dist\n",
+        "  return dist_ratio"
+      ],
+      "metadata": {
+        "id": "PhVmnUs8ErD9"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "print('Ratio of largest to smallest distance 2D: %3.3f'%(distance_ratio(x_2D)))\n",
+        "print('Ratio of largest to smallest distance 100D: %3.3f'%(distance_ratio(x_100D)))\n",
+        "print('Ratio of largest to smallest distance 1000D: %3.3f'%(distance_ratio(x_1000D)))\n",
+        "print('**Note down the last of these three numbers, you will need to submit it for your coursework**')"
+      ],
+      "metadata": {
+        "id": "0NdPxfn5GQuJ"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Part (b)\n",
+        "\n",
+        "In part (b) of the practical we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius.  You will find that the volume decreases to almost nothing in high dimensions.  All of the volume is in the corners of the unit hypercube (which always has volume 1). Double weird.\n",
+        "\n",
+        "Note that you you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
+      ],
+      "metadata": {
+        "id": "b2FYKV1SL4Z7"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def volume_of_hypersphere(diameter, dimensions):\n",
+        "  # Formula given in Problem 8.7 of the notes\n",
+        "  # You will need sci.special.gamma()\n",
+        "  # Check out:    https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.gamma.html\n",
+        "  # Also use this value for pi\n",
+        "  pi = np.pi\n",
+        "  # TODO replace this code with formula for the volume of a hypersphere\n",
+        "  volume = 1.0\n",
+        "\n",
+        "  return volume\n"
+      ],
+      "metadata": {
+        "id": "CZoNhD8XJaHR"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "diameter = 1.0\n",
+        "for c_dim in range(1,11):\n",
+        "  print(\"Volume of unit diameter hypersphere in %d dimensions is %3.3f\"%(c_dim, volume_of_hypersphere(diameter, c_dim)))\n",
+        "print('**Note down the last of these ten numbers, you will need to submit it for your coursework**')"
+      ],
+      "metadata": {
+        "id": "fNTBlg_GPEUh"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Part (c)\n",
+        "\n",
+        "In part (c) of the coursework, you will calculate what proportion of the volume of a hypersphere is in the outer 1% of the radius/diameter.  Calculate the volume of a hypersphere and then the volume of a hypersphere with 0.99 of the radius and then figure out the proportion (a number between 0 and 1).  You'll see that by the time we get to 300 dimensions most of the volume is in the outer 1 percent.  Extremely weird!"
+      ],
+      "metadata": {
+        "id": "GdyMeOBmoXyF"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def get_prop_of_volume_in_outer_1_percent(dimension):\n",
+        "  # TODO -- replace this line\n",
+        "  proportion = 1.0\n",
+        "\n",
+        "  return proportion"
+      ],
+      "metadata": {
+        "id": "8_CxZ2AIpQ8w"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# While we're here, let's look at how much of the volume is in the outer 1% of the radius\n",
+        "for c_dim in [1,2,10,20,50,100,150,200,250,300]:\n",
+        "  print('Proportion of volume in outer 1 percent of radius in %d dimensions =%3.3f'%(c_dim, get_prop_of_volume_in_outer_1_percent(c_dim)))\n",
+        "print('**Note down the last of these ten numbers, you will need to submit it for your coursework**')"
+      ],
+      "metadata": {
+        "id": "LtMDIn2qPVfJ"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

CM20315_2023/CM20315_Coursework_V_2023.ipynb

@@ -0,0 +1,525 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyN7KaQQ63bf52r+b5as0MkK",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/CM20315_2023/CM20315_Coursework_V_2023.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Coursework V: Backpropagation in Toy Model**\n",
+        "\n",
+        "This notebook computes the derivatives of a toy function similar (but different from) that in section 7.3 of the book.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.  At various points, you will get an answer that you need to copy into Moodle to be marked.\n",
+        "\n",
+        "Post to the content forum if you find any mistakes or need to clarify something."
+      ],
+      "metadata": {
+        "id": "pOZ6Djz0dhoy"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Problem setting\n",
+        "\n",
+        "We're going to investigate how to take the derivatives of functions where one operation is composed with another, which is composed with a third and so on.  For example, consider the model:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "     \\mbox{f}[x,\\boldsymbol\\phi] = \\beta_3+\\omega_3\\cdot\\mbox{PReLU}\\Bigl[\\gamma, \\beta_2+\\omega_2\\cdot\\mbox{PReLU}\\bigl[\\gamma, \\beta_1+\\omega_1\\cdot\\mbox{PReLU}[\\gamma, \\beta_0+\\omega_0x]\\bigr]\\Bigr],\n",
+        "\\end{equation}\n",
+        "\n",
+        "with parameters $\\boldsymbol\\phi=\\{\\beta_0,\\omega_0,\\beta_1,\\omega_1,\\beta_2,\\omega_2,\\beta_3,\\omega_3\\}$, where\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\mbox{PReLU}[\\gamma, z] = \\begin{cases} \\gamma\\cdot z & \\quad z \\leq0 \\\\ z & \\quad z> 0\\end{cases}.\n",
+        "\\end{equation}\n",
+        "\n",
+        "Suppose that we have a binary cross-entropy loss function (equation 5.20 from the book):\n",
+        "\n",
+        "\\begin{equation*}\n",
+        "\\ell_i  = -(1-y_{i})\\log\\Bigl[1-\\mbox{sig}[\\mbox{f}[\\mathbf{x}_i,\\boldsymbol\\phi]]\\Bigr] - y_{i}\\log\\Bigl[\\mbox{sig}[\\mbox{f}[\\mathbf{x}_i,\\boldsymbol\\phi]]\\Bigr].\n",
+        "\\end{equation*}\n",
+        "\n",
+        "Assume that we know the current values of $\\beta_{0},\\beta_{1},\\beta_{2},\\beta_{3},\\omega_{0},\\omega_{1},\\omega_{2},\\omega_{3}$, $\\gamma$, $x_i$ and $y_i$. We want to know how $\\ell_i$ changes when we make a small change to $\\beta_{0},\\beta_{1},\\beta_{2},\\beta_{3},\\omega_{0},\\omega_{1},\\omega_{2}$, or $\\omega_{3}$.  In other words, we want to compute the eight derivatives:\n",
+        "\n",
+        "\\begin{eqnarray*}\n",
+        "\\frac{\\partial \\ell_i}{\\partial \\beta_{0}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\beta_{1}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\beta_{2}}, \\quad \\frac{\\partial \\ell_i }{\\partial \\beta_{3}},  \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{0}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{1}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{2}},  \\quad\\mbox{and} \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{3}}.\n",
+        "\\end{eqnarray*}"
+      ],
+      "metadata": {
+        "id": "1DmMo2w63CmT"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# import library\n",
+        "import numpy as np"
+      ],
+      "metadata": {
+        "id": "RIPaoVN834Lj"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's first define the original function and the loss term:"
+      ],
+      "metadata": {
+        "id": "32-ufWhc3v2c"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "AakK_qen3BpU"
+      },
+      "outputs": [],
+      "source": [
+        "# Defines the activation function\n",
+        "def paramReLU(gamma,x):\n",
+        "  if x > 0:\n",
+        "    return x\n",
+        "  else:\n",
+        "    return x * gamma\n",
+        "\n",
+        "# Defines the main function\n",
+        "def fn(x, beta0, beta1, beta2, beta3, omega0, omega1, omega2, omega3, gamma):\n",
+        "  return beta3+omega3 * paramReLU(gamma, beta2 + omega2 * paramReLU(gamma, beta1 + omega1 * paramReLU(gamma, beta0 + omega0 * x)))\n",
+        "\n",
+        "# Logistic sigmoid\n",
+        "def sig(z):\n",
+        "  return 1./(1+np.exp(-z))\n",
+        "\n",
+        "# The loss function (equation 5.20 from book)\n",
+        "def loss(f,y):\n",
+        "  sig_net_out = sig(f)\n",
+        "  l = -(1-y) * np.log(1-sig_net_out) - y * np.log(sig_net_out)\n",
+        "  return l"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now we'll choose some values for the betas and the omegas and x and compute the output of the function:"
+      ],
+      "metadata": {
+        "id": "y7tf0ZMt5OXt"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "beta0 = 1.0; beta1 = -2.0; beta2 = -3.0; beta3 = 0.4\n",
+        "omega0 = 0.1; omega1 = -0.4; omega2 = 2.0; omega3 = -3.0\n",
+        "gamma = 0.2\n",
+        "x = 2.3; y =1.0\n",
+        "f_val = fn(x,beta0,beta1,beta2,beta3,omega0,omega1,omega2,omega3, gamma)\n",
+        "l_i_func = loss(f_val, y)\n",
+        "print('Loss full function = %3.3f'%l_i_func)"
+      ],
+      "metadata": {
+        "id": "pwvOcCxr41X_"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Forward pass\n",
+        "\n",
+        "We compute a series of intermediate values $f_0, h_0, f_1, h_1, f_2, h_2, f_3$, and finally the loss $\\ell$"
+      ],
+      "metadata": {
+        "id": "W6ZP62T5fU64"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "x = 2.3; y =1.0\n",
+        "gamma = 0.2\n",
+        "# Compute all the f_k and h_k terms\n",
+        "# I've done the first two for you\n",
+        "f0 = beta0+omega0 * x\n",
+        "h1 = paramReLU(gamma, f0)\n",
+        "\n",
+        "\n",
+        "# TODO:  Replace the code below\n",
+        "f1 = 0\n",
+        "h2 = 0\n",
+        "f2 = 0\n",
+        "h3 = 0\n",
+        "f3 = 0\n",
+        "\n",
+        "\n",
+        "# Compute the loss and print\n",
+        "# The answer should be the same as when we computed the full function above\n",
+        "l_i = loss(f3, y)\n",
+        "print(\"Loss forward pass = %3.3f\"%(l_i))\n"
+      ],
+      "metadata": {
+        "id": "z-BckTpMf5PL"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Backward pass:  Derivative of loss function with respect to function output\n",
+        "\n",
+        "Now, we'll compute the derivative $\\frac{dl}{df_3}$ of the loss function with respect to the network output $f_3$.  In other words, we are asking how does the loss change as we make a small change in the network output.\n",
+        "\n",
+        "Since the loss it itself a function of $\\mbox{sig}[f_3]$ we'll compute this using the chain rule:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\frac{dl}{df_3} = \\frac{d\\mbox{sig}[f_3]}{df_3}\\cdot \\frac{dl}{d\\mbox{sig}[f_3]}\n",
+        "\\end{equation}\n",
+        "\n",
+        "Your job is to compute the two quantities on the right hand side.\n"
+      ],
+      "metadata": {
+        "id": "TbFbxz64Xz4I"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Compute the derivative of the the loss with respect to the function output f_val\n",
+        "def dl_df(f_val,y):\n",
+        "  # Compute sigmoid of network output\n",
+        "  sig_f_val = sig(f_val)\n",
+        "  # Compute the derivative of loss with respect to network output using chain rule\n",
+        "  dl_df_val = dsig_df(f_val) * dl_dsigf(sig_f_val, y)\n",
+        "  # Return the derivative\n",
+        "  return dl_df_val"
+      ],
+      "metadata": {
+        "id": "ZWKAq6HC90qV"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# MOODLE ANSWER # Notebook V 1a: Copy this code when you have finished it.\n",
+        "\n",
+        "# Compute the derivative of the logistic sigmoid function with respect to its input (as a closed form solution)\n",
+        "def dsig_df(f_val):\n",
+        "  # TODO Write this function\n",
+        "  # Replace this line:\n",
+        "  return 1\n",
+        "\n",
+        "# Compute the derivative of the loss with respect to the logistic sigmoid (as a closed form solution)\n",
+        "def dl_dsigf(sig_f_val, y):\n",
+        "  # TODO Write this function\n",
+        "  # Replace this line:\n",
+        "  return 1"
+      ],
+      "metadata": {
+        "id": "lIngYAgPq-5I"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's run that for some f_val, y.  Check previous practicals to see how you can check whether your answer is correct."
+      ],
+      "metadata": {
+        "id": "Q-j-i8khXzbK"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "y = 0.0\n",
+        "dl_df3 = dl_df(f3,y)\n",
+        "print(\"Moodle Answer Notebook V 1b: dldh3=%3.3f\"%(dl_df3))\n",
+        "\n",
+        "y= 1.0\n",
+        "dl_df3 = dl_df(f3,y)\n",
+        "print(\"Moodle Answer Notebook V 1c: dldh3=%3.3f\"%(dl_df3))"
+      ],
+      "metadata": {
+        "id": "Z7Lb5BibY50H"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Backward pass:  Derivative of activation function with respect to preactivations\n",
+        "\n",
+        "Write a function to compute the derivative $\\frac{\\partial h}{\\partial f}$ of the activation function (parametric ReLU) with respect to its input.\n"
+      ],
+      "metadata": {
+        "id": "BA7QaOzejzZw"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# MOODLE ANSWER Notebook V 2a: Copy this code when you have finished it.\n",
+        "\n",
+        "def dh_df(gamma, f_val):\n",
+        "  # TODO:  Write this function\n",
+        "  # Replace this line:\n",
+        "  return 1\n"
+      ],
+      "metadata": {
+        "id": "bBPfPg04j-Qw"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's run that for some values of f_val.  Check previous practicals to see how you can check whether your answer is correct."
+      ],
+      "metadata": {
+        "id": "QRNCM0EGk9-w"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "f_val_test = 0.6\n",
+        "dh_df_val = dh_df(gamma, f_val_test)\n",
+        "print(\"Moodle Answer Notebook V 2b: dhdf=%3.3f\"%(dh_df_val))\n",
+        "\n",
+        "f_val_test = -0.4\n",
+        "dh_df_val = dh_df(gamma, f_val_test)\n",
+        "print(\"Moodle Answer Notebook V 2c: dhdf=%3.3f\"%(dh_df_val))"
+      ],
+      "metadata": {
+        "id": "bql8VZIGk8Wy"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        " # Backward pass:  Compute the derivatives of $l_i$ with respect to the intermediate quantities but in reverse order:\n",
+        "\n",
+        "\\begin{eqnarray}\n",
+        "\\frac{\\partial \\ell_i}{\\partial h_3}, \\quad \\frac{\\partial \\ell_i}{\\partial f_2}, \\quad\n",
+        "\\frac{\\partial \\ell_i}{\\partial h_2}, \\quad \\frac{\\partial \\ell_i}{\\partial f_1}, \\quad\\frac{\\partial \\ell_i}{\\partial h_1},  \\quad\\mbox{and} \\quad \\frac{\\partial \\ell_i}{\\partial f_0}.\n",
+        "\\end{eqnarray}\n",
+        "\n",
+        "The first of these derivatives can be calculated using the chain rule:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\frac{\\partial \\ell_i}{\\partial h_{3}} =\\frac{\\partial f_{3}}{\\partial h_{3}} \\frac{\\partial \\ell_i}{\\partial f_{3}} .\n",
+        "\\end{equation}\n",
+        "\n",
+        "The left-hand side asks how $\\ell_i$ changes when $h_{3}$ changes.  The right-hand side says we can decompose this into (i) how $\\ell_i$ changes when $f_{3}$ changes and how $f_{3}$ changes when $h_{3}$ changes.  So you get a chain of events happening:  $h_{3}$ changes $f_{3}$, which changes $\\ell_i$, and the derivatives represent the effects of this chain.  Notice that we computed the first of these derivatives already.  The second term is the derivative of $\\beta_{3} + \\omega_{3}h_{3}$ with respect to $h_3$ which is simply $\\omega_3$.  \n",
+        "\n",
+        "We can continue in this way, computing the derivatives of the output with respect to these intermediate quantities:\n",
+        "\n",
+        "\\begin{eqnarray}\n",
+        "\\frac{\\partial \\ell_i}{\\partial f_{2}} &=& \\frac{\\partial h_{3}}{\\partial f_{2}}\\left(\n",
+        "\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}} \\right)\n",
+        "\\nonumber \\\\\n",
+        "\\frac{\\partial \\ell_i}{\\partial h_{2}} &=& \\frac{\\partial f_{2}}{\\partial h_{2}}\\left(\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}}\\right)\\nonumber \\\\\n",
+        "\\frac{\\partial \\ell_i}{\\partial f_{1}} &=& \\frac{\\partial h_{2}}{\\partial f_{1}}\\left( \\frac{\\partial f_{2}}{\\partial h_{2}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}} \\right)\\nonumber \\\\\n",
+        "\\frac{\\partial \\ell_i}{\\partial h_{1}} &=& \\frac{\\partial f_{1}}{\\partial h_{1}}\\left(\\frac{\\partial h_{2}}{\\partial f_{1}} \\frac{\\partial f_{2}}{\\partial h_{2}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}} \\right)\\nonumber \\\\\n",
+        "\\frac{\\partial \\ell_i}{\\partial f_{0}} &=& \\frac{\\partial h_{1}}{\\partial f_{0}}\\left(\\frac{\\partial f_{1}}{\\partial h_{1}}\\frac{\\partial h_{2}}{\\partial f_{1}} \\frac{\\partial f_{2}}{\\partial h_{2}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}} \\right).\n",
+        "\\end{eqnarray}\n",
+        "\n",
+        "In each case, we have already computed all of the terms except the last one in the previous step, and the last term is simple to evaluate.  This is called the **backward pass**."
+      ],
+      "metadata": {
+        "id": "jay8NYWdFHuZ"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "x = 2.3; y =1.0\n",
+        "dldf3 = dl_df(f3,y)"
+      ],
+      "metadata": {
+        "id": "RSC_2CIfKF1b"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# MOODLE ANSWER Notebook V 3a: Copy this code when you have finished it.\n",
+        "# TODO -- Compute the derivatives of the output with respect\n",
+        "# to the intermediate computations h_k and f_k (i.e, run the backward pass)\n",
+        "# I've done the first two for you.  You replace the code below:\n",
+        "# Replace the code below\n",
+        "dldh3 = 1\n",
+        "dldf2 = 1\n",
+        "dldh2 = 1\n",
+        "dldf1 = 1\n",
+        "dldh1 = 1\n",
+        "dldf0 = 1"
+      ],
+      "metadata": {
+        "id": "gCQJeI--Egdl"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Finally, we consider how the loss~$\\ell_{i}$ changes when we change the parameters $\\beta_{\\bullet}$ and $\\omega_{\\bullet}$. Once more, we apply the chain rule:\n",
+        "\n",
+        "\n",
+        "\n",
+        "\n",
+        "\\begin{eqnarray}\n",
+        "\\frac{\\partial \\ell_i}{\\partial \\beta_{k}} &=& \\frac{\\partial f_{k}}{\\partial \\beta_{k}}\\frac{\\partial \\ell_i}{\\partial f_{k}}\\nonumber \\\\\n",
+        "\\frac{\\partial \\ell_i}{\\partial \\omega_{k}} &=& \\frac{\\partial f_{k}}{\\partial \\omega_{k}}\\frac{\\partial \\ell_i}{\\partial f_{k}}.\n",
+        "\\end{eqnarray}\n",
+        "\n",
+        "\\noindent In each case, the second term on the right-hand side was computed in step 2. When $k>0$, we have~$f_{k}=\\beta_{k}+\\omega_k \\cdot h_{k}$, so:\n",
+        "\n",
+        "\\begin{eqnarray}\n",
+        "\\frac{\\partial f_{k}}{\\partial \\beta_{k}} = 1 \\quad\\quad\\mbox{and}\\quad \\quad \\frac{\\partial f_{k}}{\\partial \\omega_{k}} &=& h_{k}.\n",
+        "\\end{eqnarray}"
+      ],
+      "metadata": {
+        "id": "FlzlThQPGpkU"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# MOODLE ANSWER Notebook V 3b: Copy this code when you have finished it.\n",
+        "# TODO -- Calculate the final derivatives with respect to the beta and omega terms\n",
+        "# Replace these terms\n",
+        "dldbeta3 = 1\n",
+        "dldomega3 = 1\n",
+        "dldbeta2 = 1\n",
+        "dldomega2 = 1\n",
+        "dldbeta1 = 1\n",
+        "dldomega1 = 1\n",
+        "dldbeta0 = 1\n",
+        "dldomega0 = 1"
+      ],
+      "metadata": {
+        "id": "1I2BhqZhGMK6"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Print the last two values out (enter these into Moodle).  Again, think about how you can test whether these are correct.\n",
+        "print('Moodle Answer Notebook V 3c: dldbeta0=%3.3f'%(dldbeta0))\n",
+        "print('Moodle Answer Notebook V 3d: dldOmega0=%3.3f'%(dldomega0))"
+      ],
+      "metadata": {
+        "id": "38eiOn2aHgHI"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Compute the derivatives of  $\\ell_i$  with respect to the parmeter $\\gamma$ of the parametric ReLU function.  \n",
+        "\n",
+        "In other words, compute:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\frac{d\\ell_i}{d\\gamma}\n",
+        "\\end{equation}\n",
+        "\n",
+        "Along the way, we will need to compute derivatives\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\frac{dh_k(\\gamma,f_{k-1})}{d\\gamma}\n",
+        "\\end{equation}\n",
+        "\n",
+        "This is quite difficult and not worth many marks, so don't spend too much time on it if you are confused!"
+      ],
+      "metadata": {
+        "id": "lhD5AoUHx3DS"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Computes how an activation changes with a small change in gamma assuming preactivations are f\n",
+        "# MOODLE ANSWER # Notebook V 4a: Copy this code when you have finished it.\n",
+        "def dhdgamma(gamma, f):\n",
+        "  # TODO -- Write this function\n",
+        "  # Replace this line\n",
+        "  return 1"
+      ],
+      "metadata": {
+        "id": "yC-9MTQevliP"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Compute how the loss changes with gamma\n",
+        "# Replace this line:\n",
+        "# MOODLE ANSWER # Notebook V 4b: Copy this code when you have finished it.\n",
+        "dldgamma = 1"
+      ],
+      "metadata": {
+        "id": "DiNQrveoLuHR"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "print(\"Moodle Answer Notebook V 4c: dldgamma = %3.3f\"%(dldgamma))"
+      ],
+      "metadata": {
+        "id": "YHxmAEnxzy3O"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

Notebooks/Chap02/2_1_Supervised_Learning.ipynb

@@ -31,7 +31,7 @@
       "source": [
         "# Notebook 2.1 Supervised Learning\n",
         "\n",
-        "The purpose of this notebook is to explore the linear regression model dicussed in Chapter 2 of the book.\n",
+        "The purpose of this notebook is to explore the linear regression model discussed in Chapter 2 of the book.\n",
         "\n",
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and write code to complete the functions. There are also questions interspersed in the text.\n",
         "\n",

Notebooks/Chap03/3_2_Shallow_Networks_II.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyPD+qTkgmZCe+VessXM/kIU",
+      "authorship_tag": "ABX9TyNk2dAhwwRxGpfVSC3b2Owv",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -182,7 +182,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Now we'll extend this model to have two outputs $y_1$ and $y_2$, each of which can be visualized with a separate heatmap.  You will now have sets of parameters $\\phi_{10}, \\phi_{11},\\phi_{12}$ and $\\phi_{2}, \\phi_{21},\\phi_{22}$ that correspond to each of these outputs."
+        "Now we'll extend this model to have two outputs $y_1$ and $y_2$, each of which can be visualized with a separate heatmap.  You will now have sets of parameters $\\phi_{10}, \\phi_{11},\\phi_{12}$ and $\\phi_{20}, \\phi_{21},\\phi_{22}$ that correspond to each of these outputs."
       ],
       "metadata": {
         "id": "Xl6LcrUyM7Lh"

Notebooks/Chap03/3_3_Shallow_Network_Regions.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyMdflMfWi9hu9ZEg/80HCd8",
+      "authorship_tag": "ABX9TyNioITtfAcfxEfM3UOfQyb9",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -79,7 +79,7 @@
       "source": [
         "def number_regions(Di, D):\n",
         "  # TODO -- implement Zaslavsky's formula\n",
-        "  # You will need to use math.factorial() https://www.geeksforgeeks.org/factorial-in-python/\n",
+        "  # You can use math.com() https://www.w3schools.com/python/ref_math_comb.asp\n",
         "  # Replace this code\n",
         "  N = 1;\n",
         "\n",
@@ -191,7 +191,7 @@
       "cell_type": "code",
       "source": [
         "# Now let's compute and plot the number of regions as a function of the number of parameters as in figure 3.9b\n",
-        "# First let's write a function that computes the number of parameters as a function of the input dimension and number of hidden layers (assuming just one output)\n",
+        "# First let's write a function that computes the number of parameters as a function of the input dimension and number of hidden units (assuming just one output)\n",
         "\n",
         "def number_parameters(D_i, D):\n",
         "  # TODO -- replace this code with the proper calculation\n",

Notebooks/Chap04/4_3_Deep_Networks.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyMbJGN6f2+yKzzsVep/wi5U",
+      "authorship_tag": "ABX9TyO2DaD75p+LGi7WgvTzjrk1",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -232,7 +232,7 @@
         "beta_2 = np.zeros((1,1))\n",
         "Omega_2 = np.zeros((1,3))\n",
         "\n",
-        "# TODO Fill in the values of the beta and Omega matrices for with the n1_theta, n1_phi, n2_theta, and n2_phi parameters\n",
+        "# TODO Fill in the values of the beta and Omega matrices for the n1_theta, n1_phi, n2_theta, and n2_phi parameters\n",
         "# that define the composition of the two networks above (see eqn 4.5 for Omega1 and beta1 albeit in different notation)\n",
         "# !!! NOTE THAT MATRICES ARE CONVENTIONALLY INDEXED WITH a_11 IN THE TOP LEFT CORNER, BUT NDARRAYS START AT [0,0] SO EVERYTHING IS OFFSET\n",
         "# To get you started I've filled in a few:\n",
@@ -274,7 +274,7 @@
       "cell_type": "code",
       "source": [
         "# define sizes\n",
-        "D_i=4; D_1=5; D_2=2; D_3=1; D_o=1\n",
+        "D_i=4; D_1=5; D_2=2; D_3=4; D_o=1\n",
         "# We'll choose the inputs and parameters of this network randomly using np.random.normal\n",
         "# For example, we'll set the input using\n",
         "n_data = 4;\n",

Notebooks/Chap05/5_1_Least_Squares_Loss.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyNkBMOVt5gO7Awn9JMn4N8Z",
+      "authorship_tag": "ABX9TyOJeBMhN9fXO8UepZ4+Pbg6",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -66,7 +66,7 @@
         "  return activation\n",
         "\n",
         "# Define a shallow neural network\n",
-        "def shallow_nn(x, beta_0, omega_0, beta_1, omaga_1):\n",
+        "def shallow_nn(x, beta_0, omega_0, beta_1, omega_1):\n",
         "    # Make sure that input data is (1 x n_data) array\n",
         "    n_data = x.size\n",
         "    x = np.reshape(x,(1,n_data))\n",
@@ -139,7 +139,7 @@
       "source": [
         "# Univariate regression\n",
         "\n",
-        "We'll investigate a simple univarite regression situation with a single input $x$ and a single output $y$ as pictured in figures 5.4 and 5.5b."
+        "We'll investigate a simple univariate regression situation with a single input $x$ and a single output $y$ as pictured in figures 5.4 and 5.5b."
       ],
       "metadata": {
         "id": "PsgLZwsPxauP"
@@ -306,8 +306,9 @@
       "source": [
         "# Return the negative log likelihood of the data under the model\n",
         "def compute_negative_log_likelihood(y_train, mu, sigma):\n",
-        "  # TODO -- compute the likelihood of the data -- don't use the likelihood function above -- compute the negative sum of the log probabilities\n",
-        "  # Bottom line of equation 5.3 in the notes\n",
+        "  # TODO -- compute the negative log likelihood of the data without using a product\n",
+        "  # In other words, compute minus one times the sum of the log probabilities\n",
+        "  # Equation 5.4 in the notes\n",
         "  # You will need np.sum(), np.log()\n",
         "  # Replace the line below\n",
         "  nll = 0\n",
@@ -352,7 +353,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Return the squared distance between the predicted\n",
+        "# Return the squared distance between the observed data (y_train) and the prediction of the model (y_pred)\n",
         "def compute_sum_of_squares(y_train, y_pred):\n",
         "  # TODO -- compute the sum of squared distances between the training data and the model prediction\n",
         "  # Eqn 5.10 in the notes.  Make sure that you understand this, and ask questions if you don't\n",
@@ -372,9 +373,9 @@
       "source": [
         "# Let's test this again\n",
         "beta_0, omega_0, beta_1, omega_1 = get_parameters()\n",
-        "# Use our neural network to predict the mean of the Gaussian\n",
-        "y_pred = shallow_nn(x_train, beta_0, omega_0, beta_1, omega_1)\n",
-        "# Compute the log likelihood\n",
+        "# Use our neural network to predict the mean of the Gaussian, which is out best prediction of y\n",
+        "y_pred = mu_pred = shallow_nn(x_train, beta_0, omega_0, beta_1, omega_1)\n",
+        "# Compute the sum of squares\n",
         "sum_of_squares = compute_sum_of_squares(y_train, y_pred)\n",
         "# Let's double check we get the right answer before proceeding\n",
         "print(\"Correct answer = %9.9f, Your answer = %9.9f\"%(2.020992572,sum_of_squares))"
@@ -554,7 +555,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Obviously, to fit the full neural model we would vary all of the 10 parameters of the network in the $\\boldsymbol\\beta_{0},\\boldsymbol\\omega_{0},\\boldsymbol\\beta_{1},\\boldsymbol\\omega_{1}$ (and maybe $\\sigma$) until we find the combination that have the maximum likelihood / minimum negative log likelihood / least squares.<br><br>\n",
+        "Obviously, to fit the full neural model we would vary all of the 10 parameters of the network in $\\boldsymbol\\beta_{0},\\boldsymbol\\omega_{0},\\boldsymbol\\beta_{1},\\boldsymbol\\omega_{1}$ (and maybe $\\sigma$) until we find the combination that have the maximum likelihood / minimum negative log likelihood / least squares.<br><br>\n",
         "\n",
         "Here we just varied one at a time as it is easier to see what is going on.  This is known as **coordinate descent**.\n"
       ],

Notebooks/Chap05/5_3_Multiclass_Cross_entropy_Loss.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOVTohDBtmCCzSEqLJ4J9R/",
+      "authorship_tag": "ABX9TyPNAZtbS+8jYc+tZqhDHNev",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -68,7 +68,7 @@
         "  return activation\n",
         "\n",
         "# Define a shallow neural network\n",
-        "def shallow_nn(x, beta_0, omega_0, beta_1, omaga_1):\n",
+        "def shallow_nn(x, beta_0, omega_0, beta_1, omega_1):\n",
         "    # Make sure that input data is (1 x n_data) array\n",
         "    n_data = x.size\n",
         "    x = np.reshape(x,(1,n_data))\n",

Notebooks/Chap06/6_1_Line_Search.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOfxeJ15PMkIi4geDTRCz3c",
+      "authorship_tag": "ABX9TyN4E9Vtuk6t2BhZ0Ajv5SW3",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -131,14 +131,15 @@
         "\n",
         "        print('Iter %d, a=%3.3f, b=%3.3f, c=%3.3f, d=%3.3f'%(n_iter, a,b,c,d))\n",
         "\n",
-        "        # Rule #1 If point A is less than points B, C, and D then halve values of B, C, and D\n",
+        "        # Rule #1 If the HEIGHT at point A is less the HEIGHT at points B, C, and D then halve values of B, C, and D\n",
+        "        # i.e. bring them closer to the original point\n",
         "        # i.e. bring them closer to the original point\n",
         "        # TODO REPLACE THE BLOCK OF CODE BELOW WITH THIS RULE\n",
         "        if (0):\n",
         "          continue;\n",
         "\n",
         "\n",
-        "        # Rule #2 If point b is less than point c then\n",
+        "        # Rule #2 If the HEIGHT at point b is less than the HEIGHT at point c then\n",
         "        #                     then point d becomes point c, and\n",
         "        #                     point b becomes 1/3 between a and new d\n",
         "        #                     point c becomes 2/3 between a and new d\n",
@@ -146,7 +147,7 @@
         "        if (0):\n",
         "          continue;\n",
         "\n",
-        "        # Rule #3 If point c is less than point b then\n",
+        "        # Rule #3 If the HEIGHT at point c is less than the HEIGHT at point b then\n",
         "        #                     then point a becomes point b, and\n",
         "        #                     point b becomes 1/3 between new a and d\n",
         "        #                     point c becomes 2/3 between new a and d\n",

Notebooks/Chap06/6_2_Gradient_Descent.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyM/FIXDTd6tZYs6WRzK00hB",
+      "authorship_tag": "ABX9TyN2N4cCnlIobOZXEjcwAvZ5",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -31,7 +31,7 @@
       "source": [
         "# **Notebook 6.2 Gradient descent**\n",
         "\n",
-        "This notebook recreates the gradient descent algorithm as shon in figure 6.1.\n",
+        "This notebook recreates the gradient descent algorithm as shown in figure 6.1.\n",
         "\n",
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
@@ -301,7 +301,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Now we are ready to perform gradient descent.  We'll need to use our line search routine from part I, which I've reproduced here plus the helper function loss_function_1D that converts from a 2D problem to a 1D problem"
+        "Now we are ready to perform gradient descent.  We'll need to use our line search routine from noteboo 6.1, which I've reproduced here plus the helper function loss_function_1D that converts from a 2D problem to a 1D problem"
       ],
       "metadata": {
         "id": "5EIjMM9Fw2eT"
@@ -375,9 +375,9 @@
       "source": [
         "def gradient_descent_step(phi, data,  model):\n",
         "  # TODO -- update Phi with the gradient descent step (equation 6.3)\n",
-        "  # 1. Compute the gradient\n",
-        "  # 2. Find the best step size alpha (use negative gradient as going downhill)\n",
-        "  # 3. Update the parameters phi\n",
+        "  # 1. Compute the gradient (you wrote this function above)\n",
+        "  # 2. Find the best step size alpha using line search function (above) -- use negative gradient as going downhill\n",
+        "  # 3. Update the parameters phi based on the gradient and the step size alpha.\n",
         "\n",
         "  return phi"
       ],

Notebooks/Chap06/6_3_Stochastic_Gradient_Descent.ipynb

@@ -123,7 +123,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Initialize the parmaeters and draw the model\n",
+        "# Initialize the parameters and draw the model\n",
         "phi = np.zeros((2,1))\n",
         "phi[0] =  -5     # Horizontal offset\n",
         "phi[1] =  25     # Frequency\n",

Notebooks/Chap06/6_4_Momentum.ipynb

@@ -123,7 +123,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Initialize the parmaeters and draw the model\n",
+        "# Initialize the parameters and draw the model\n",
         "phi = np.zeros((2,1))\n",
         "phi[0] =  -5     # Horizontal offset\n",
         "phi[1] =  25     # Frequency\n",

Notebooks/Chap06/6_5_Adam.ipynb

@@ -248,7 +248,7 @@
         "        # Replace this line:\n",
         "        v = v\n",
         "\n",
-        "        # TODO -- Modify the statistics according to euation 6.16\n",
+        "        # TODO -- Modify the statistics according to equation 6.16\n",
         "        # You will need the function np.power\n",
         "        # Replace these lines\n",
         "        m_tilde = m\n",

Notebooks/Chap07/7_1_Backpropagation_in_Toy_Model.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyP5wHK5E7/el+vxU947K3q8",
+      "authorship_tag": "ABX9TyOjXmTmoff61y15VqEB5sDW",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -83,13 +83,13 @@
       "metadata": {
         "id": "RIPaoVN834Lj"
       },
-      "execution_count": null,
+      "execution_count": 1,
       "outputs": []
     },
     {
       "cell_type": "markdown",
       "source": [
-        "Let's first define the original function for $y$ and the likelihood term:"
+        "Let's first define the original function for $y$ and the loss term:"
       ],
       "metadata": {
         "id": "32-ufWhc3v2c"
@@ -97,7 +97,7 @@
     },
     {
       "cell_type": "code",
-      "execution_count": null,
+      "execution_count": 2,
       "metadata": {
         "id": "AakK_qen3BpU"
       },
@@ -106,7 +106,7 @@
         "def fn(x, beta0, beta1, beta2, beta3, omega0, omega1, omega2, omega3):\n",
         "  return beta3+omega3 * np.cos(beta2 + omega2 * np.exp(beta1 + omega1 * np.sin(beta0 + omega0 * x)))\n",
         "\n",
-        "def likelihood(x, y, beta0, beta1, beta2, beta3, omega0, omega1, omega2, omega3):\n",
+        "def loss(x, y, beta0, beta1, beta2, beta3, omega0, omega1, omega2, omega3):\n",
         "  diff = fn(x, beta0, beta1, beta2, beta3, omega0, omega1, omega2, omega3) - y\n",
         "  return diff * diff"
       ]
@@ -126,14 +126,26 @@
         "beta0 = 1.0; beta1 = 2.0; beta2 = -3.0; beta3 = 0.4\n",
         "omega0 = 0.1; omega1 = -0.4; omega2 = 2.0; omega3 = 3.0\n",
         "x = 2.3; y =2.0\n",
-        "l_i_func = likelihood(x,y,beta0,beta1,beta2,beta3,omega0,omega1,omega2,omega3)\n",
+        "l_i_func = loss(x,y,beta0,beta1,beta2,beta3,omega0,omega1,omega2,omega3)\n",
         "print('l_i=%3.3f'%l_i_func)"
       ],
       "metadata": {
-        "id": "pwvOcCxr41X_"
+        "id": "pwvOcCxr41X_",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
         },
-      "execution_count": null,
-      "outputs": []
+        "outputId": "9541922c-dfc4-4b2e-dfa3-3298812155ce"
+      },
+      "execution_count": 3,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "l_i=0.139\n"
+          ]
+        }
+      ]
     },
     {
       "cell_type": "markdown",
@@ -163,7 +175,7 @@
       "metadata": {
         "id": "7t22hALp5zkq"
       },
-      "execution_count": null,
+      "execution_count": 4,
       "outputs": []
     },
     {
@@ -178,15 +190,27 @@
     {
       "cell_type": "code",
       "source": [
-        "dldomega0_fd = (likelihood(x,y,beta0,beta1,beta2,beta3,omega0+0.00001,omega1,omega2,omega3)-likelihood(x,y,beta0,beta1,beta2,beta3,omega0,omega1,omega2,omega3))/0.00001\n",
+        "dldomega0_fd = (loss(x,y,beta0,beta1,beta2,beta3,omega0+0.00001,omega1,omega2,omega3)-loss(x,y,beta0,beta1,beta2,beta3,omega0,omega1,omega2,omega3))/0.00001\n",
         "\n",
         "print('dydomega0: Function value = %3.3f, Finite difference value = %3.3f'%(dldomega0_func,dldomega0_fd))"
       ],
       "metadata": {
-        "id": "1O3XmXMx-HlZ"
+        "id": "1O3XmXMx-HlZ",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
         },
-      "execution_count": null,
-      "outputs": []
+        "outputId": "389ed78e-9d8d-4e8b-9e6b-5f20c21407e8"
+      },
+      "execution_count": 5,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "dydomega0: Function value = 5.246, Finite difference value = 5.246\n"
+          ]
+        }
+      ]
     },
     {
       "cell_type": "markdown",
@@ -237,7 +261,7 @@
       "metadata": {
         "id": "ZWKAq6HC90qV"
       },
-      "execution_count": null,
+      "execution_count": 6,
       "outputs": []
     },
     {
@@ -254,15 +278,34 @@
         "print(\"like original = %3.3f, like from forward pass = %3.3f\"%(l_i_func, l_i))\n"
       ],
       "metadata": {
-        "id": "ibxXw7TUW4Sx"
+        "id": "ibxXw7TUW4Sx",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
         },
-      "execution_count": null,
-      "outputs": []
+        "outputId": "4575e3eb-2b16-4e0b-c84e-9c22b443c3ce"
+      },
+      "execution_count": 7,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "f0: true value = 1.230, your value = 0.000\n",
+            "h1: true value = 0.942, your value = 0.000\n",
+            "f1: true value = 1.623, your value = 0.000\n",
+            "h2: true value = 5.068, your value = 0.000\n",
+            "f2: true value = 7.137, your value = 0.000\n",
+            "h3: true value = 0.657, your value = 0.000\n",
+            "f3: true value = 2.372, your value = 0.000\n",
+            "like original = 0.139, like from forward pass = 0.000\n"
+          ]
+        }
+      ]
     },
     {
       "cell_type": "markdown",
       "source": [
-        "**Step 2:** Compute the derivatives of $y$ with respect to the intermediate quantities that we just calculated, but in reverse order:\n",
+        "**Step 2:** Compute the derivatives of $\\ell_i$ with respect to the intermediate quantities that we just calculated, but in reverse order:\n",
         "\n",
         "\\begin{eqnarray}\n",
         "\\quad \\frac{\\partial \\ell_i}{\\partial f_3}, \\quad \\frac{\\partial \\ell_i}{\\partial h_3}, \\quad \\frac{\\partial \\ell_i}{\\partial f_2}, \\quad\n",
@@ -281,7 +324,7 @@
         "\\frac{\\partial \\ell_i}{\\partial h_{3}} =\\frac{\\partial f_{3}}{\\partial h_{3}} \\frac{\\partial \\ell_i}{\\partial f_{3}} .\n",
         "\\end{equation}\n",
         "\n",
-        "The left-hand side asks how $\\ell_i$ changes when $h_{3}$ changes.  The right-hand side says we can decompose this into (i) how $ell_i$ changes when $f_{3}$ changes and how $f_{3}$ changes when $h_{3}$ changes.  So you get a chain of events happening:  $h_{3}$ changes $f_{3}$, which changes $\\ell_i$, and the derivatives represent the effects of this chain.  Notice that we computed the first of these derivatives already and is  $2 (f_3-y)$. We calculated $f_{3}$ in step 1.  The second term is the derivative of $\\beta_{3} + \\omega_{3}h_{3}$ with respect to $h_3$ which is simply $\\omega_3$.  \n",
+        "The left-hand side asks how $\\ell_i$ changes when $h_{3}$ changes.  The right-hand side says we can decompose this into (i) how $\\ell_i$ changes when $f_{3}$ changes and how $f_{3}$ changes when $h_{3}$ changes.  So you get a chain of events happening:  $h_{3}$ changes $f_{3}$, which changes $\\ell_i$, and the derivatives represent the effects of this chain.  Notice that we computed the first of these derivatives already and is  $2 (f_3-y)$. We calculated $f_{3}$ in step 1.  The second term is the derivative of $\\beta_{3} + \\omega_{3}h_{3}$ with respect to $h_3$ which is simply $\\omega_3$.  \n",
         "\n",
         "We can continue in this way, computing the derivatives of the output with respect to these intermediate quantities:\n",
         "\n",
@@ -319,7 +362,7 @@
       "metadata": {
         "id": "gCQJeI--Egdl"
       },
-      "execution_count": null,
+      "execution_count": 8,
       "outputs": []
     },
     {
@@ -335,10 +378,28 @@
         "print(\"dldf0: true value = %3.3f, your value = %3.3f\"%(2.281, dldf0))"
       ],
       "metadata": {
-        "id": "dS1OrLtlaFr7"
+        "id": "dS1OrLtlaFr7",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
         },
-      "execution_count": null,
-      "outputs": []
+        "outputId": "414f0862-ae36-4a0e-b68f-4758835b0e23"
+      },
+      "execution_count": 9,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "dldf3: true value = 0.745, your value = -4.000\n",
+            "dldh3: true value = 2.234, your value = -12.000\n",
+            "dldf2: true value = -1.683, your value = 1.000\n",
+            "dldh2: true value = -3.366, your value = 1.000\n",
+            "dldf1: true value = -17.060, your value = 1.000\n",
+            "dldh1: true value = 6.824, your value = 1.000\n",
+            "dldf0: true value = 2.281, your value = 1.000\n"
+          ]
+        }
+      ]
     },
     {
       "cell_type": "markdown",
@@ -380,7 +441,7 @@
       "metadata": {
         "id": "1I2BhqZhGMK6"
       },
-      "execution_count": null,
+      "execution_count": 10,
       "outputs": []
     },
     {
@@ -397,10 +458,29 @@
         "print('dldomega0: Your value = %3.3f, Function value = %3.3f, Finite difference value = %3.3f'%(dldomega0, dldomega0_func, dldomega0_fd))"
       ],
       "metadata": {
-        "id": "38eiOn2aHgHI"
+        "id": "38eiOn2aHgHI",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
         },
-      "execution_count": null,
-      "outputs": []
+        "outputId": "1a67a636-e832-471e-e771-54824363158a"
+      },
+      "execution_count": 11,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "dldbeta3: Your value = 1.000, True value = 0.745\n",
+            "dldomega3: Your value = 1.000, True value = 0.489\n",
+            "dldbeta2: Your value = 1.000, True value = -1.683\n",
+            "dldomega2: Your value = 1.000, True value = -8.530\n",
+            "dldbeta1: Your value = 1.000, True value = -17.060\n",
+            "dldomega1: Your value = 1.000, True value = -16.079\n",
+            "dldbeta0: Your value = 1.000, True value = 2.281\n",
+            "dldomega0: Your value = 1.000, Function value = 5.246, Finite difference value = 5.246\n"
+          ]
+        }
+      ]
     },
     {
       "cell_type": "markdown",

Notebooks/Chap07/7_2_Backpropagation.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyN2nPVR0imZntgj4Oasyvmo",
+      "authorship_tag": "ABX9TyOlKB4TrCJnt91TnHOrfRSJ",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -244,28 +244,28 @@
         "  all_dl_dh = [None] * (K+1)\n",
         "  # Again for convenience we'll stick with the convention that all_h[0] is the net input and all_f[k] in the net output\n",
         "\n",
-        "  # Compute derivatives of net output with respect to loss\n",
+        "  # Compute derivatives of the loss with respect to the network output\n",
         "  all_dl_df[K] = np.array(d_loss_d_output(all_f[K],y))\n",
         "\n",
         "  # Now work backwards through the network\n",
         "  for layer in range(K,-1,-1):\n",
-        "    # TODO Calculate the derivatives of biases at layer this from all_dl_df[layer]. (eq 7.21)\n",
+        "    # TODO Calculate the derivatives of the loss with respect to the biases at layer this from all_dl_df[layer]. (eq 7.21)\n",
         "    # NOTE!  To take a copy of matrix X, use Z=np.array(X)\n",
         "    # REPLACE THIS LINE\n",
         "    all_dl_dbiases[layer] = np.zeros_like(all_biases[layer])\n",
         "\n",
-        "    # TODO Calculate the derivatives of weight at layer from all_dl_df[K] and all_h[K] (eq 7.22)\n",
+        "    # TODO Calculate the derivatives of the loss with respect to the weights at layer from all_dl_df[layer] and all_h[layer] (eq 7.22)\n",
         "    # Don't forget to use np.matmul\n",
         "    # REPLACE THIS LINE\n",
         "    all_dl_dweights[layer] = np.zeros_like(all_weights[layer])\n",
         "\n",
-        "    # TODO: calculate the derivatives of activations from weight and derivatives of next preactivations (eq 7.20)\n",
+        "    # TODO: calculate the derivatives of the loss with respect to the activations from weight and derivatives of next preactivations (second part of last line of eq 7.24)\n",
         "    # REPLACE THIS LINE\n",
         "    all_dl_dh[layer] = np.zeros_like(all_h[layer])\n",
         "\n",
         "\n",
         "    if layer > 0:\n",
-        "      # TODO Calculate the derivatives of the pre-activation f with respect to activation h (deriv of ReLu function)\n",
+        "      # TODO Calculate the derivatives of the loss with respect to the pre-activation f (use deriv of ReLu function, first part of last line of eq. 7.24)\n",
         "      # REPLACE THIS LINE\n",
         "      all_dl_df[layer-1] = np.zeros_like(all_f[layer-1])\n",
         "\n",

Notebooks/Chap08/8_2_Bias_Variance_Trade_Off.ipynb

@@ -77,7 +77,7 @@
         "    for i in range(n_data):\n",
         "        x[i] = np.random.uniform(i/n_data, (i+1)/n_data, 1)\n",
         "\n",
-        "    # y value from running through functoin and adding noise\n",
+        "    # y value from running through function and adding noise\n",
         "    y = np.ones(n_data)\n",
         "    for i in range(n_data):\n",
         "        y[i] = true_function(x[i])\n",
@@ -229,7 +229,7 @@
         "  y_model_all = np.zeros((n_datasets, x_model.shape[0]))\n",
         "\n",
         "  for c_dataset in range(n_datasets):\n",
-        "    # TODO -- Generate n_data x,y, pairs with standard divation sigma_func\n",
+        "    # TODO -- Generate n_data x,y, pairs with standard deviation sigma_func\n",
         "    # Replace this line\n",
         "    x_data,y_data = np.zeros([1,n_data]),np.zeros([1,n_data])\n",
         "\n",
@@ -316,7 +316,7 @@
         "\n",
         "  # Compute variance -- average of the model variance (average squared deviation of fitted models around mean fitted model)\n",
         "  variance[c_hidden] = 0\n",
-        "  # Compute bias (average squared deviaton of mean fitted model around true function)\n",
+        "  # Compute bias (average squared deviation of mean fitted model around true function)\n",
         "  bias[c_hidden] = 0\n",
         "\n",
         "# Plot the results\n",

Notebooks/Chap09/9_1_L2_Regularization.ipynb

@@ -120,7 +120,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Initialize the parmaeters and draw the model\n",
+        "# Initialize the parameters and draw the model\n",
         "phi = np.zeros((2,1))\n",
         "phi[0] =  -5     # Horizontal offset\n",
         "phi[1] =  25     # Frequency\n",
@@ -341,7 +341,7 @@
       "source": [
         "# Computes the regularization term\n",
         "def compute_reg_term(phi0,phi1):\n",
-        "  # TODO compute the regularization term (term in large brackets in the above equstion)\n",
+        "  # TODO compute the regularization term (term in large brackets in the above equation)\n",
         "  # Replace this line\n",
         "  reg_term = 0.0\n",
         "\n",

Notebooks/Chap09/9_4_Bayesian_Approach.ipynb

@@ -80,7 +80,7 @@
         "    for i in range(n_data):\n",
         "        x[i] = np.random.uniform(i/n_data, (i+1)/n_data, 1)\n",
         "\n",
-        "    # y value from running through functoin and adding noise\n",
+        "    # y value from running through function and adding noise\n",
         "    y = np.ones(n_data)\n",
         "    for i in range(n_data):\n",
         "        y[i] = true_function(x[i])\n",
@@ -137,7 +137,7 @@
         "n_data = 15\n",
         "x_data,y_data = generate_data(n_data, sigma_func)\n",
         "\n",
-        "# Plot the functinon, data and uncertainty\n",
+        "# Plot the function, data and uncertainty\n",
         "plot_function(x_func, y_func, x_data, y_data, sigma_func=sigma_func)"
       ],
       "metadata": {
@@ -357,7 +357,7 @@
         "\n",
         "To compute this, we reformulated the integrand using the relations from appendices\n",
         "C.3.3 and C.3.4 as the product of a normal distribution in $\\boldsymbol\\phi$ and a constant with respect\n",
-        "to $\\boldsymbol\\phi$. The integral of the normal distribution must be one, and so the finnal result is just the constant. This constant is itself a normal distribution in $y^*$. <br>\n",
+        "to $\\boldsymbol\\phi$. The integral of the normal distribution must be one, and so the final result is just the constant. This constant is itself a normal distribution in $y^*$. <br>\n",
         "\n",
         "If you feel so inclined you can work through the math of this yourself."
       ],

Notebooks/Chap10/10_1_1D_Convolution.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyPHUNRkJMI5LujaxIXNV60m",
+      "authorship_tag": "ABX9TyPTidpnPhn4O5QF011gt0cz",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -85,10 +85,10 @@
       "cell_type": "code",
       "source": [
         "# Now let's define a zero-padded convolution operation\n",
-        "# with a convolution kernel size of 3, a stride of 1, and a dilation of 0\n",
+        "# with a convolution kernel size of 3, a stride of 1, and a dilation of 1\n",
         "# as in figure 10.2a-c.  Write it yourself, don't call a library routine!\n",
         "# Don't forget that Python arrays are indexed from zero, not from 1 as in the book figures\n",
-        "def conv_3_1_0_zp(x_in, omega):\n",
+        "def conv_3_1_1_zp(x_in, omega):\n",
         "    x_out = np.zeros_like(x_in)\n",
         "    # TODO -- write this function\n",
         "    # replace this line\n",
@@ -119,7 +119,7 @@
       "source": [
         "\n",
         "omega = [0.33,0.33,0.33]\n",
-        "h = conv_3_1_0_zp(x, omega)\n",
+        "h = conv_3_1_1_zp(x, omega)\n",
         "\n",
         "# Check that you have computed this correctly\n",
         "print(f\"Sum of output is {np.sum(h):3.3}, should be 71.1\")\n",
@@ -155,7 +155,7 @@
       "source": [
         "\n",
         "omega = [-0.5,0,0.5]\n",
-        "h2 = conv_3_1_0_zp(x, omega)\n",
+        "h2 = conv_3_1_1_zp(x, omega)\n",
         "\n",
         "# Draw the signal\n",
         "fig,ax = plt.subplots()\n",
@@ -187,9 +187,9 @@
       "cell_type": "code",
       "source": [
         "# Now let's define a zero-padded convolution operation\n",
-        "# with a convolution kernel size of 3, a stride of 2, and a dilation of 0\n",
-        "# as in figure 10.2a-c.  Write it yourself, don't call a library routine!\n",
-        "def conv_3_2_0_zp(x_in, omega):\n",
+        "# with a convolution kernel size of 3, a stride of 2, and a dilation of 1\n",
+        "# as in figure 10.3a-b.  Write it yourself, don't call a library routine!\n",
+        "def conv_3_2_1_zp(x_in, omega):\n",
         "    x_out = np.zeros(int(np.ceil(len(x_in)/2)))\n",
         "    # TODO -- write this function\n",
         "    # replace this line\n",
@@ -209,7 +209,7 @@
       "cell_type": "code",
       "source": [
         "omega = [0.33,0.33,0.33]\n",
-        "h3 = conv_3_2_0_zp(x, omega)\n",
+        "h3 = conv_3_2_1_zp(x, omega)\n",
         "\n",
         "# If you have done this right, the output length should be six and it should\n",
         "# contain every other value from the original convolution with stride 1\n",
@@ -226,9 +226,9 @@
       "cell_type": "code",
       "source": [
         "# Now let's define a zero-padded convolution operation\n",
-        "# with a convolution kernel size of 5, a stride of 1, and a dilation of 0\n",
-        "# as in figure 10.2a-c.  Write it yourself, don't call a library routine!\n",
-        "def conv_5_1_0_zp(x_in, omega):\n",
+        "# with a convolution kernel size of 5, a stride of 1, and a dilation of 1\n",
+        "# as in figure 10.3c.  Write it yourself, don't call a library routine!\n",
+        "def conv_5_1_1_zp(x_in, omega):\n",
         "    x_out = np.zeros_like(x_in)\n",
         "    # TODO -- write this function\n",
         "    # replace this line\n",
@@ -249,7 +249,7 @@
       "source": [
         "\n",
         "omega2 = [0.2, 0.2, 0.2, 0.2, 0.2]\n",
-        "h4 = conv_5_1_0_zp(x, omega2)\n",
+        "h4 = conv_5_1_1_zp(x, omega2)\n",
         "\n",
         "# Check that you have computed this correctly\n",
         "print(f\"Sum of output is {np.sum(h4):3.3}, should be 69.6\")\n",
@@ -273,10 +273,10 @@
       "cell_type": "code",
       "source": [
         "# Finally let's define a zero-padded convolution operation\n",
-        "# with a convolution kernel size of 3, a stride of 1, and a dilation of 1\n",
-        "# as in figure 10.2a-c.  Write it yourself, don't call a library routine!\n",
+        "# with a convolution kernel size of 3, a stride of 1, and a dilation of 2\n",
+        "# as in figure 10.3d.  Write it yourself, don't call a library routine!\n",
         "# Don't forget that Python arrays are indexed from zero, not from 1 as in the book figures\n",
-        "def conv_3_1_1_zp(x_in, omega):\n",
+        "def conv_3_1_2_zp(x_in, omega):\n",
         "    x_out = np.zeros_like(x_in)\n",
         "    # TODO -- write this function\n",
         "    # replace this line\n",
@@ -295,7 +295,7 @@
       "cell_type": "code",
       "source": [
         "omega = [0.33,0.33,0.33]\n",
-        "h5 = conv_3_1_1_zp(x, omega)\n",
+        "h5 = conv_3_1_2_zp(x, omega)\n",
         "\n",
         "# Check that you have computed this correctly\n",
         "print(f\"Sum of output is {np.sum(h5):3.3}, should be 68.3\")\n",
@@ -328,7 +328,7 @@
       "cell_type": "code",
       "source": [
         "# Compute matrix in figure 10.4 d\n",
-        "def get_conv_mat_3_1_0_zp(n_out, omega):\n",
+        "def get_conv_mat_3_1_1_zp(n_out, omega):\n",
         "  omega_mat = np.zeros((n_out,n_out))\n",
         "  # TODO Fill in this matix\n",
         "  # Replace this line:\n",
@@ -349,11 +349,11 @@
       "source": [
         "# Run original convolution\n",
         "omega = np.array([-1.0,0.5,-0.2])\n",
-        "h6 = conv_3_1_0_zp(x, omega)\n",
+        "h6 = conv_3_1_1_zp(x, omega)\n",
         "print(h6)\n",
         "\n",
         "# If you have done this right, you should get the same answer\n",
-        "omega_mat = get_conv_mat_3_1_0_zp(len(x), omega)\n",
+        "omega_mat = get_conv_mat_3_1_1_zp(len(x), omega)\n",
         "h7 = np.matmul(omega_mat, x)\n",
         "print(h7)\n"
       ],

Notebooks/Chap10/10_2_Convolution_for_MNIST_1D.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOgDisWDe/zHpfTGCH8AZ3i",
+      "authorship_tag": "ABX9TyN1v/yg9PtdSVOWlYJ7bgkz",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -128,11 +128,11 @@
         "\n",
         "\n",
         "# TODO Create a model with the following layers\n",
-        "# 1. Convolutional layer, (input=length 40 and 1 channel, kernel size 3x3, stride 2, padding=\"valid\", 15 output channels )\n",
+        "# 1. Convolutional layer, (input=length 40 and 1 channel, kernel size 3, stride 2, padding=\"valid\", 15 output channels )\n",
         "# 2. ReLU\n",
-        "# 3. Convolutional layer, (input=length 19 and 15 channels, kernel size 3x3, stride 2, padding=\"valid\", 15 output channels )\n",
+        "# 3. Convolutional layer, (input=length 19 and 15 channels, kernel size 3, stride 2, padding=\"valid\", 15 output channels )\n",
         "# 4. ReLU\n",
-        "# 5. Convolutional layer, (input=length 9 and 15 channels, kernel size 3x3, stride 2, padding=\"valid\", 15 output channels)\n",
+        "# 5. Convolutional layer, (input=length 9 and 15 channels, kernel size 3, stride 2, padding=\"valid\", 15 output channels)\n",
         "# 6. ReLU\n",
         "# 7. Flatten (converts 4x15) to length 60\n",
         "# 8. Linear layer (input size = 60, output size = 10)\n",

Notebooks/Chap11/11_3_Batch_Normalization.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOoGS+lY+EhGthebSO4smpj",
+      "authorship_tag": "ABX9TyOZaNcBrdZ9yCHhjLOwSi69",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -205,7 +205,8 @@
         "    self.linear3 = nn.Linear(hidden_size, hidden_size)\n",
         "    self.linear4 = nn.Linear(hidden_size, hidden_size)\n",
         "    self.linear5 = nn.Linear(hidden_size, hidden_size)\n",
-        "    self.linear6 = nn.Linear(hidden_size, output_size)\n",
+        "    self.linear6 = nn.Linear(hidden_size, hidden_size)\n",
+        "    self.linear7 = nn.Linear(hidden_size, output_size)\n",
         "\n",
         "  def count_params(self):\n",
         "    return sum([p.view(-1).shape[0] for p in self.parameters()])\n",
@@ -220,11 +221,11 @@
         "    print_variance(\"After second residual connection\",res2)\n",
         "    res3 = res2 + self.linear4(res2.relu())\n",
         "    print_variance(\"After third residual connection\",res3)\n",
-        "    res4 = res3 + self.linear4(res3.relu())\n",
+        "    res4 = res3 + self.linear5(res3.relu())\n",
         "    print_variance(\"After fourth residual connection\",res4)\n",
-        "    res5 = res4 + self.linear4(res4.relu())\n",
+        "    res5 = res4 + self.linear6(res4.relu())\n",
         "    print_variance(\"After fifth residual connection\",res5)\n",
-        "    return self.linear6(res5)"
+        "    return self.linear7(res5)"
       ],
       "metadata": {
         "id": "FslroPJJffrh"
@@ -266,13 +267,14 @@
         "# Use the torch function nn.BatchNorm1d\n",
         "class ResidualNetworkWithBatchNorm(torch.nn.Module):\n",
         "  def __init__(self, input_size, output_size, hidden_size=100):\n",
-        "    super(ResidualNetworkWithBatchNorm, self).__init__()\n",
+        "    super(ResidualNetwork, self).__init__()\n",
         "    self.linear1 = nn.Linear(input_size, hidden_size)\n",
         "    self.linear2 = nn.Linear(hidden_size, hidden_size)\n",
         "    self.linear3 = nn.Linear(hidden_size, hidden_size)\n",
         "    self.linear4 = nn.Linear(hidden_size, hidden_size)\n",
         "    self.linear5 = nn.Linear(hidden_size, hidden_size)\n",
-        "    self.linear6 = nn.Linear(hidden_size, output_size)\n",
+        "    self.linear6 = nn.Linear(hidden_size, hidden_size)\n",
+        "    self.linear7 = nn.Linear(hidden_size, output_size)\n",
         "\n",
         "  def count_params(self):\n",
         "    return sum([p.view(-1).shape[0] for p in self.parameters()])\n",
@@ -287,11 +289,11 @@
         "    print_variance(\"After second residual connection\",res2)\n",
         "    res3 = res2 + self.linear4(res2.relu())\n",
         "    print_variance(\"After third residual connection\",res3)\n",
-        "    res4 = res3 + self.linear4(res3.relu())\n",
+        "    res4 = res3 + self.linear5(res3.relu())\n",
         "    print_variance(\"After fourth residual connection\",res4)\n",
-        "    res5 = res4 + self.linear4(res4.relu())\n",
+        "    res5 = res4 + self.linear6(res4.relu())\n",
         "    print_variance(\"After fifth residual connection\",res5)\n",
-        "    return self.linear6(res5)"
+        "    return self.linear7(res5)"
       ],
       "metadata": {
         "id": "5JvMmaRITKGd"

Notebooks/Chap12/12_1_Self_Attention.ipynb

@@ -153,7 +153,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "We'll need a softmax function (equation 12.5) -- here, it will take a list of arbirtrary numbers and return a list where the elements are non-negative and sum to one\n"
+        "We'll need a softmax function (equation 12.5) -- here, it will take a list of arbitrary numbers and return a list where the elements are non-negative and sum to one\n"
       ],
       "metadata": {
         "id": "Se7DK6PGPSUk"
@@ -364,7 +364,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "TODO -- Investigate whether the self-attention mechanism is covariant with respect to permulation.\n",
+        "TODO -- Investigate whether the self-attention mechanism is covariant with respect to permutation.\n",
         "If it is, when we permute the columns of the input matrix $\\mathbf{X}$, the columns of the output matrix $\\mathbf{X}'$ will also be permuted.\n"
       ],
       "metadata": {

Notebooks/Chap12/12_2_Multihead_Self_Attention.ipynb

@@ -31,7 +31,7 @@
       "source": [
         "# **Notebook 12.1: Multhead Self-Attention**\n",
         "\n",
-        "This notebook builds a multihead self-attentionm mechanism as in figure 12.6\n",
+        "This notebook builds a multihead self-attention mechanism as in figure 12.6\n",
         "\n",
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",

Notebooks/Chap15/15_1_GAN_Toy_Example.ipynb

@@ -31,7 +31,7 @@
       "source": [
         "# **Notebook 15.1: GAN Toy example**\n",
         "\n",
-        "This notebook investigates the GAN toy example as illustred in figure 15.1 in the book.\n",
+        "This notebook investigates the GAN toy example as illustrated in figure 15.1 in the book.\n",
         "\n",
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
@@ -101,7 +101,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Now, we define our disriminator.  This is a simple logistic regression model (1D linear model passed through sigmoid) that returns the probability that the data is real"
+        "Now, we define our discriminator.  This is a simple logistic regression model (1D linear model passed through sigmoid) that returns the probability that the data is real"
       ],
       "metadata": {
         "id": "Xrzd8aehYAYR"
@@ -387,7 +387,7 @@
         "print(\"Final parameters (phi0,phi1)\", phi0, phi1)\n",
         "for c_gan_iter in range(5):\n",
         "\n",
-        "  # Run generator to product syntehsized data\n",
+        "  # Run generator to product synthesized data\n",
         "  x_syn = generator(z, theta)\n",
         "  draw_data_model(x_real, x_syn, phi0, phi1)\n",
         "\n",

Notebooks/Chap15/15_2_Wasserstein_Distance.ipynb

@@ -29,9 +29,9 @@
     {
       "cell_type": "markdown",
       "source": [
-        "# **Notebook 15.2: Wassserstein Distance**\n",
+        "# **Notebook 15.2: Wasserstein Distance**\n",
         "\n",
-        "This notebook investigates the GAN toy example as illustred in figure 15.1 in the book.\n",
+        "This notebook investigates the GAN toy example as illustrated in figure 15.1 in the book.\n",
         "\n",
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",

Notebooks/Chap16/16_2_Autoregressive_Flows.ipynb

@@ -65,7 +65,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# First let's make the 1D piecewise linear mapping as illustated in figure 16.5\n",
+        "# First let's make the 1D piecewise linear mapping as illustrated in figure 16.5\n",
         "def g(h, phi):\n",
         "  # TODO -- write this function (equation 16.12)\n",
         "  # Note: If you have the first printing of the book, there is a mistake in equation 16.12\n",
@@ -156,7 +156,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Now let's define an autogressive flow.  Let's switch to looking at figure 16.7.# We'll assume that our piecewise function will use five parameters phi1,phi2,phi3,phi4,phi5"
+        "Now let's define an autoregressive flow.  Let's switch to looking at figure 16.7.# We'll assume that our piecewise function will use five parameters phi1,phi2,phi3,phi4,phi5"
       ],
       "metadata": {
         "id": "t8XPxipfd7hz"
@@ -175,7 +175,7 @@
         "  x = x/ np.sum(x) ;\n",
         "  return x\n",
         "\n",
-        "# Return value of phi that doesn't depend on any of the iputs\n",
+        "# Return value of phi that doesn't depend on any of the inputs\n",
         "def get_phi():\n",
         "  return np.array([0.2, 0.1, 0.4, 0.05, 0.25])\n",
         "\n",

Notebooks/Chap17/17_1_Latent_Variable_Models.ipynb

@@ -0,0 +1,396 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyMBYNsjj1iTgHUYhAXqUYJd",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap17/17_1_Latent_Variable_Models.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 17.1: Latent variable models**\n",
+        "\n",
+        "This notebook investigates a non-linear latent variable model similar to that in figures 17.2 and 17.3 of the book.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "import scipy\n",
+        "from matplotlib.colors import ListedColormap\n",
+        "from matplotlib import cm"
+      ],
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "We'll assume that our base distribution over the latent variables is a 1D standard normal so that\n",
+        "\n",
+        "\\begin{equation}\n",
+        "Pr(z) = \\mbox{Norm}_{z}[0,1]\n",
+        "\\end{equation}\n",
+        "\n",
+        "As in figure 17.2, we'll assume that the output is two dimensional, we we need to define a function that maps from the 1D latent variable to two dimensions.  Usually, we would use a neural network, but in this case, we'll just define an arbitrary relationship.\n",
+        "\n",
+        "\\begin{eqnarray}\n",
+        "x_{1} &=& 0.5\\cdot\\exp\\Bigl[\\sin\\bigl[2+ 3.675 z \\bigr]\\Bigr]\\\\\n",
+        "x_{2} &=& \\sin\\bigl[2+ 2.85 z \\bigr]\n",
+        "\\end{eqnarray}"
+      ],
+      "metadata": {
+        "id": "IyVn-Gi-p7wf"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# The function that maps z to x1 and x2\n",
+        "def f(z):\n",
+        "  x_1 = np.exp(np.sin(2+z*3.675)) * 0.5\n",
+        "  x_2 = np.cos(2+z*2.85)\n",
+        "  return x_1, x_2"
+      ],
+      "metadata": {
+        "id": "ZIfQwhd-AV6L"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's plot the 3D relation between the two observed variables $x_{1}$ and $x_{2}$ and the latent variables $z$ as in figure 17.2 of the book.  We'll use the opacity to represent the prior probability $Pr(z)$."
+      ],
+      "metadata": {
+        "id": "KB9FU34onW1j"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def draw_3d_projection(z,pr_z, x1,x2):\n",
+        "  alpha = pr_z / np.max(pr_z)\n",
+        "  ax = plt.axes(projection='3d')\n",
+        "  fig = plt.gcf()\n",
+        "  fig.set_size_inches(18.5, 10.5)\n",
+        "  for i in range(len(z)-1):\n",
+        "    ax.plot([z[i],z[i+1]],[x1[i],x1[i+1]],[x2[i],x2[i+1]],'r-', alpha=pr_z[i])\n",
+        "  ax.set_xlabel('$z$',)\n",
+        "  ax.set_ylabel('$x_1$')\n",
+        "  ax.set_zlabel('$x_2$')\n",
+        "  ax.set_xlim(-3,3)\n",
+        "  ax.set_ylim(0,2)\n",
+        "  ax.set_zlim(-1,1)\n",
+        "  ax.set_box_aspect((3,1,1))\n",
+        "  plt.show()"
+      ],
+      "metadata": {
+        "id": "lW08xqAgnP4q"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Compute the prior\n",
+        "def get_prior(z):\n",
+        "  return scipy.stats.multivariate_normal.pdf(z)"
+      ],
+      "metadata": {
+        "id": "9DUTauMi6tPk"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Define the latent variable values\n",
+        "z = np.arange(-3.0,3.0,0.01)\n",
+        "# Find the probability distribution over z\n",
+        "pr_z = get_prior(z)\n",
+        "# Compute x1 and x2 for each z\n",
+        "x1,x2 = f(z)\n",
+        "# Plot the function\n",
+        "draw_3d_projection(z,pr_z, x1,x2)"
+      ],
+      "metadata": {
+        "id": "PAzHq461VqvF"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "The likelihood is defined as:\n",
+        "\\begin{eqnarray}\n",
+        " Pr(x_1,x_2|z) &=&  \\mbox{Norm}_{[x_1,x_2]}\\Bigl[\\mathbf{f}[z],\\sigma^{2}\\mathbf{I}\\Bigr]\n",
+        "\\end{eqnarray}\n",
+        "\n",
+        "so we will also need to define the noise level $\\sigma^2$"
+      ],
+      "metadata": {
+        "id": "sQg2gKR5zMrF"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "sigma_sq = 0.04"
+      ],
+      "metadata": {
+        "id": "In_Vg4_0nva3"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Draws a heatmap to represent a probability distribution, possibly with samples overlaed\n",
+        "def plot_heatmap(x1_mesh,x2_mesh,y_mesh, x1_samples=None, x2_samples=None, title=None):\n",
+        "  # Define pretty colormap\n",
+        "  my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
+        "  my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
+        "  r = np.floor(my_colormap_vals_dec/(256*256))\n",
+        "  g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
+        "  b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
+        "  my_colormap = ListedColormap(np.vstack((r,g,b)).transpose()/255.0)\n",
+        "\n",
+        "  fig,ax = plt.subplots()\n",
+        "  fig.set_size_inches(8,8)\n",
+        "  ax.contourf(x1_mesh,x2_mesh,y_mesh,256,cmap=my_colormap)\n",
+        "  ax.contour(x1_mesh,x2_mesh,y_mesh,8,colors=['#80808080'])\n",
+        "  if title is not None:\n",
+        "    ax.set_title(title);\n",
+        "  if x1_samples is not None:\n",
+        "    ax.plot(x1_samples, x2_samples, 'c.')\n",
+        "  ax.set_xlim([-0.5,2.5])\n",
+        "  ax.set_ylim([-1.5,1.5])\n",
+        "  ax.set_xlabel('$x_1$'); ax.set_ylabel('$x_2$')\n",
+        "  plt.show()\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "6P6d-AgAqxXZ"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Returns the likelihood\n",
+        "def get_likelihood(x1_mesh, x2_mesh, z_val):\n",
+        "  # Find the corresponding x1 and x2 values\n",
+        "  x1,x2 = f(z_val)\n",
+        "\n",
+        "  # Calculate the probability for a mesh of x1,x2 values.\n",
+        "  mn = scipy.stats.multivariate_normal([x1, x2], [[sigma_sq, 0], [0, sigma_sq]])\n",
+        "  pr_x1_x2_given_z_val = mn.pdf(np.dstack((x1_mesh, x2_mesh)))\n",
+        "  return pr_x1_x2_given_z_val"
+      ],
+      "metadata": {
+        "id": "diYKb7_ZgjlJ"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's plot the likelihood $Pr(x_1,x_2|z)$ as in fig 17.3b in the book."
+      ],
+      "metadata": {
+        "id": "0X4NwixzqxtZ"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Choose some z value\n",
+        "z_val = 1.8\n",
+        "\n",
+        "# Compute the conditional distribution on a grid\n",
+        "x1_mesh, x2_mesh = np.meshgrid(np.arange(-0.5,2.5,0.01), np.arange(-1.5,1.5,0.01))\n",
+        "pr_x1_x2_given_z_val = get_likelihood(x1_mesh,x2_mesh, z_val)\n",
+        "\n",
+        "# Plot the result\n",
+        "plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2_given_z_val, title=\"Conditional distribution $Pr(x1,x2|z)$\")\n",
+        "\n",
+        "# TODO -- Experiment with different values of z and make sure that you understand the what is happening."
+      ],
+      "metadata": {
+        "id": "hWfqK-Oz5_DT"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "The data density is found by marginalizing over the latent variables $z$:\n",
+        "\n",
+        "\\begin{eqnarray}\n",
+        " Pr(x_1,x_2) &=& \\int Pr(x_1,x_2, z) dz \\nonumber \\\\\n",
+        " &=& \\int Pr(x_1,x_2 | z) \\cdot Pr(z)dz\\nonumber \\\\\n",
+        " &=& \\int \\mbox{Norm}_{[x_1,x_2]}\\Bigl[\\mathbf{f}[z],\\sigma^{2}\\mathbf{I}\\Bigr]\\cdot \\mbox{Norm}_{z}\\left[\\mathbf{0},\\mathbf{I}\\right]dz.\n",
+        "\\end{eqnarray}"
+      ],
+      "metadata": {
+        "id": "25xqXnmFo-PH"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# TODO Compute the data density\n",
+        "# We can't integrate this function in closed form\n",
+        "# So let's approximate it as a sum over the z values (z = np.arange(-3,3,0.01))\n",
+        "# You will need the functions get_likelihood() and get_prior()\n",
+        "# To make this a valid probability distribution, you need to divide\n",
+        "# By the z-increment (0.01)\n",
+        "# Replace this line\n",
+        "pr_x1_x2 = np.zeros_like(x1_mesh)\n",
+        "\n",
+        "\n",
+        "# Plot the result\n",
+        "plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2, title=\"Data density $Pr(x1,x2)$\")\n"
+      ],
+      "metadata": {
+        "id": "H0Ijce9VzeCO"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's draw some samples from the model"
+      ],
+      "metadata": {
+        "id": "W264N7By_h9y"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def draw_samples(n_sample):\n",
+        "  # TODO Write this routine to draw n_sample samples from the model\n",
+        "  # First draw a random value of z from the prior (a standard normal distribution)\n",
+        "  # Then draw a sample from Pr(x1,x2|z)\n",
+        "  # Replace this line\n",
+        "  x1_samples=0; x2_samples = 0;\n",
+        "\n",
+        "  return x1_samples, x2_samples"
+      ],
+      "metadata": {
+        "id": "Li3mK_I48k0k"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's plot those samples on top of the heat map."
+      ],
+      "metadata": {
+        "id": "D7N7oqLe-eJO"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "x1_samples, x2_samples = draw_samples(500)\n",
+        "# Plot the result\n",
+        "plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2, x1_samples, x2_samples, title=\"Data density $Pr(x1,x2)$\")\n"
+      ],
+      "metadata": {
+        "id": "XRmWv99B-BWO"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Return the posterior distribution\n",
+        "def get_posterior(x1,x2):\n",
+        "  z = np.arange(-3,3, 0.01)\n",
+        "  # TODO -- write this function\n",
+        "  # Again, we can't integrate, but we can sum\n",
+        "  # We don't know the constant in the denominator of equation 17.19, but we can just normalize\n",
+        "  # by the sum of the numerators for all values of z\n",
+        "  # Replace this line:\n",
+        "  pr_z_given_x1_x2 = np.ones_like(z)\n",
+        "\n",
+        "\n",
+        "  return z, pr_z_given_x1_x2"
+      ],
+      "metadata": {
+        "id": "PwOjzPD5_1OF"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "x1 = 0.9; x2 = -0.9\n",
+        "z, pr_z_given_x1_x2 = get_posterior(x1,x2)\n",
+        "\n",
+        "\n",
+        "fig, ax = plt.subplots()\n",
+        "ax.plot(z, pr_z_given_x1_x2, 'r-')\n",
+        "ax.set_xlabel(\"Latent variable $z$\")\n",
+        "ax.set_ylabel(\"Posterior probability $Pr(z|x_{1},x_{2})$\")\n",
+        "ax.set_xlim([-3,3])\n",
+        "ax.set_ylim([0,1.5 * np.max(pr_z_given_x1_x2)])\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "PKFUY42K-Tp7"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

Notebooks/Chap17/17_2_Reparameterization_Trick.ipynb

@@ -0,0 +1,423 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyOxO2/0DTH4n4zhC97qbagY",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap17/17_2_Reparameterization_Trick.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 17.2: Reparameterization trick**\n",
+        "\n",
+        "This notebook investigates the reparameterization trick as described in section 17.7 of the book.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt"
+      ],
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "The reparameterization trick computes the derivative of an expectation of a function $\\mbox{f}[x]$:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\bigl[\\mbox{f}[x]\\bigr],\n",
+        "\\end{equation}\n",
+        "\n",
+        "with respect to the parameters $\\boldsymbol\\phi$ of the distribution $Pr(x|\\boldsymbol\\phi)$ that the expectation is over.\n",
+        "\n",
+        "Let's consider a simple concrete example, where:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "Pr(x|\\phi) = \\mbox{Norm}_{x}\\Bigl[\\mu, \\sigma^2\\Bigr]=\\mbox{Norm}_{x}\\Bigl[\\phi^3,(\\exp[\\phi])^2\\Bigr]\n",
+        "\\end{equation}\n",
+        "\n",
+        "and\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\mbox{f}[x] = x^2+\\sin[x]\n",
+        "\\end{equation}"
+      ],
+      "metadata": {
+        "id": "paLz5RukZP1J"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Let's approximate this expectation for a particular value of phi\n",
+        "def compute_expectation(phi, n_samples):\n",
+        "  # TODO complete this function\n",
+        "  # 1. Compute the mean of the normal distribution, mu\n",
+        "  # 2. Compute the standard deviation of the normal distribution, sigma\n",
+        "  # 3. Draw n_samples samples using np.random.normal(mu, sigma, size=(n_samples, 1))\n",
+        "  # 4. Compute f[x] for each of these samples\n",
+        "  # 4. Approximate the expectation by taking the average of the values of f[x]\n",
+        "  # Replace this line\n",
+        "  expected_f_given_phi = 0\n",
+        "\n",
+        "\n",
+        "  return expected_f_given_phi"
+      ],
+      "metadata": {
+        "id": "FdEbMnDBY0i9"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set the seed so the random numbers are all the same\n",
+        "np.random.seed(0)\n",
+        "\n",
+        "# Compute the expectation for two values of phi\n",
+        "phi1 = 0.5\n",
+        "n_samples = 10000000\n",
+        "expected_f_given_phi1 = compute_expectation(phi1, n_samples)\n",
+        "print(\"Your value: \", expected_f_given_phi1, \", True value:  2.7650801613563116\")\n",
+        "\n",
+        "phi2 = -0.1\n",
+        "n_samples = 10000000\n",
+        "expected_f_given_phi2 = compute_expectation(phi2, n_samples)\n",
+        "print(\"Your value: \", expected_f_given_phi2, \", True value:  0.8176793102849222\")"
+      ],
+      "metadata": {
+        "id": "FTh7LJ0llNJZ"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Le't plot this expectation as a function of phi"
+      ],
+      "metadata": {
+        "id": "r5Hl2QkimWx9"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "phi_vals = np.arange(-1.5,1.5, 0.05)\n",
+        "expected_vals = np.zeros_like(phi_vals)\n",
+        "n_samples = 1000000\n",
+        "for i in range(len(phi_vals)):\n",
+        "  expected_vals[i] = compute_expectation(phi_vals[i], n_samples)\n",
+        "\n",
+        "fig,ax = plt.subplots()\n",
+        "ax.plot(phi_vals, expected_vals,'r-')\n",
+        "ax.set_xlabel('Parameter $\\phi$')\n",
+        "ax.set_ylabel('$\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "05XxVLJxmkER"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "It's this curve that we want to find the derivative of (so for example, we could run gradient descent and find the minimum.\n",
+        "\n",
+        "This is tricky though -- if you look at the computation that you performed, then there is a sampling step in the procedure (step 3).  How do we compute the derivative of this?\n",
+        "\n",
+        "The answer is the reparameterization trick.  We note that:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\mbox{Norm}_{x}\\Bigl[\\mu, \\sigma^2\\Bigr]=\\mbox{Norm}_{x}\\Bigl[0, 1\\Bigr] \\times \\sigma + \\mu\n",
+        "\\end{equation}\n",
+        "\n",
+        "and so:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\mbox{Norm}_{x}\\Bigl[\\phi^3,(\\exp[\\phi])^2\\Bigr]  = \\mbox{Norm}_{x}\\Bigl[0, 1\\Bigr] \\times \\exp[\\phi]+ \\phi^3\n",
+        "\\end{equation}\n",
+        "\n",
+        "So, if we draw a sample $\\epsilon^*$ from $\\mbox{Norm}_{\\epsilon}[0, 1]$, then we can compute a sample $x^*$ as:\n",
+        "\n",
+        "\\begin{eqnarray*}\n",
+        "x^* &=& \\epsilon^* \\times \\sigma + \\mu \\\\\n",
+        "&=& \\epsilon^* \\times \\exp[\\phi]+ \\phi^3\n",
+        "\\end{eqnarray*}"
+      ],
+      "metadata": {
+        "id": "zTCykVeWqj_O"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def compute_df_dx_star(x_star):\n",
+        "  # TODO Compute this derivative (function defined at the top)\n",
+        "  # Replace this line:\n",
+        "  deriv = 0;\n",
+        "\n",
+        "\n",
+        "\n",
+        "  return deriv\n",
+        "\n",
+        "def compute_dx_star_dphi(epsilon_star, phi):\n",
+        "  # TODO Compute this derivative\n",
+        "    # Replace this line:\n",
+        "  deriv = 0;\n",
+        "\n",
+        "\n",
+        "\n",
+        "  return deriv\n",
+        "\n",
+        "def compute_derivative_of_expectation(phi, n_samples):\n",
+        "  # Generate the random values of epsilon\n",
+        "  epsilon_star= np.random.normal(size=(n_samples,1))\n",
+        "  # TODO -- write\n",
+        "  # 1. Compute dx*/dphi using the function defined above\n",
+        "  # 2. Compute x*\n",
+        "  # 3. Compute df/dx* using the function you wrote above\n",
+        "  # 4. Compute df/dphi = df/x* * dx*dphi\n",
+        "  # 5. Average the samples of df/dphi to get the expectation.\n",
+        "  # Replace this line:\n",
+        "  df_dphi = 0\n",
+        "\n",
+        "\n",
+        "\n",
+        "  return df_dphi"
+      ],
+      "metadata": {
+        "id": "w13HVpi9q8nF"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set the seed so the random numbers are all the same\n",
+        "np.random.seed(0)\n",
+        "\n",
+        "# Compute the expectation for two values of phi\n",
+        "phi1 = 0.5\n",
+        "n_samples = 10000000\n",
+        "\n",
+        "deriv = compute_derivative_of_expectation(phi1, n_samples)\n",
+        "print(\"Your value: \", deriv, \", True value:  5.726338035051403\")"
+      ],
+      "metadata": {
+        "id": "ntQT4An79kAl"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "phi_vals = np.arange(-1.5,1.5, 0.05)\n",
+        "deriv_vals = np.zeros_like(phi_vals)\n",
+        "n_samples = 1000000\n",
+        "for i in range(len(phi_vals)):\n",
+        "  deriv_vals[i] = compute_derivative_of_expectation(phi_vals[i], n_samples)\n",
+        "\n",
+        "fig,ax = plt.subplots()\n",
+        "ax.plot(phi_vals, deriv_vals,'r-')\n",
+        "ax.set_xlabel('Parameter $\\phi$')\n",
+        "ax.set_ylabel('$\\partial/\\partial\\phi\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "t0Jqd_IN_lMU"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "This should look plausibly like the derivative of the function we plotted above!"
+      ],
+      "metadata": {
+        "id": "ASu4yKSwAEYI"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "The reparameterization trick computes the derivative of an expectation of a function $\\mbox{f}[x]$:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\bigl[\\mbox{f}[x]\\bigr],\n",
+        "\\end{equation}\n",
+        "\n",
+        "with respect to the parameters $\\boldsymbol\\phi$ of the distribution $Pr(x|\\boldsymbol\\phi)$ that the expectation is over. This derivative can also be computed as:\n",
+        "\n",
+        "\\begin{eqnarray}\n",
+        "\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\bigl[\\mbox{f}[x]\\bigr] &=& \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\left[\\mbox{f}[x]\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\log\\bigl[ Pr(x|\\boldsymbol\\phi)\\bigr]\\right]\\nonumber \\\\\n",
+        "&\\approx & \\frac{1}{I}\\sum_{i=1}^{I}\\mbox{f}[x_i]\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\log\\bigl[ Pr(x_i|\\boldsymbol\\phi)\\bigr].\n",
+        "\\end{eqnarray}\n",
+        "\n",
+        "This method is known as the REINFORCE algorithm or score function estimator.  Problem 17.5 asks you to prove this relation.  Let's use this method to compute the gradient and compare.\n",
+        "\n",
+        "Recall that the expression for a univariate Gaussian is:\n",
+        "\n",
+        "\\begin{equation}\n",
+        " Pr(x|\\mu,\\sigma^2) = \\frac{1}{\\sqrt{2\\pi\\sigma^{2}}}\\exp\\left[-\\frac{(x-\\mu)^{2}}{2\\sigma^{2}}\\right].\n",
+        "\\end{equation}\n"
+      ],
+      "metadata": {
+        "id": "xoFR1wifc8-b"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def d_log_pr_x_given_phi(x,phi):\n",
+        "  # TODO -- fill in this function\n",
+        "  # Compute the derivative of log[Pr(x|phi)]\n",
+        "  # Replace this line:\n",
+        "  deriv =0;\n",
+        "\n",
+        "\n",
+        "  return deriv\n",
+        "\n",
+        "\n",
+        "def compute_derivative_of_expectation_score_function(phi, n_samples):\n",
+        "  # TODO -- Compute this function\n",
+        "  # 1. Calculate mu from phi\n",
+        "  # 2. Calculate sigma from phi\n",
+        "  # 3. Generate n_sample random samples of x using np.random.normal\n",
+        "  # 4. Calculate f[x] for all of the samples\n",
+        "  # 5. Multiply f[x] by d_log_pr_x_given_phi\n",
+        "  # 6. Take the average of the samples\n",
+        "  # Replace this line:\n",
+        "  deriv = 0\n",
+        "\n",
+        "\n",
+        "\n",
+        "  return deriv"
+      ],
+      "metadata": {
+        "id": "4TUaxiWvASla"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set the seed so the random numbers are all the same\n",
+        "np.random.seed(0)\n",
+        "\n",
+        "# Compute the expectation for two values of phi\n",
+        "phi1 = 0.5\n",
+        "n_samples = 100000000\n",
+        "\n",
+        "deriv = compute_derivative_of_expectation_score_function(phi1, n_samples)\n",
+        "print(\"Your value: \", deriv, \", True value:  5.724609927313369\")"
+      ],
+      "metadata": {
+        "id": "0RSN32Rna_C_"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "phi_vals = np.arange(-1.5,1.5, 0.05)\n",
+        "deriv_vals = np.zeros_like(phi_vals)\n",
+        "n_samples = 1000000\n",
+        "for i in range(len(phi_vals)):\n",
+        "  deriv_vals[i] = compute_derivative_of_expectation_score_function(phi_vals[i], n_samples)\n",
+        "\n",
+        "fig,ax = plt.subplots()\n",
+        "ax.plot(phi_vals, deriv_vals,'r-')\n",
+        "ax.set_xlabel('Parameter $\\phi$')\n",
+        "ax.set_ylabel('$\\partial/\\partial\\phi\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "EM_i5zoyElHR"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "This should look the same as the derivative that we computed with the reparameterization trick.  So, is there any advantage to one way or the other?  Let's compare the variances of the estimates\n"
+      ],
+      "metadata": {
+        "id": "1TWBiUC7bQSw"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "n_estimate = 100\n",
+        "n_sample = 1000\n",
+        "phi = 0.3\n",
+        "reparam_estimates = np.zeros((n_estimate,1))\n",
+        "score_function_estimates = np.zeros((n_estimate,1))\n",
+        "for i in range(n_estimate):\n",
+        "  reparam_estimates[i]= compute_derivative_of_expectation(phi, n_samples)\n",
+        "  score_function_estimates[i] = compute_derivative_of_expectation_score_function(phi, n_samples)\n",
+        "\n",
+        "print(\"Variance of reparameterization estimator\", np.var(reparam_estimates))\n",
+        "print(\"Variance of score function estimator\", np.var(score_function_estimates))"
+      ],
+      "metadata": {
+        "id": "vV_Jx5bCbQGs"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "The variance of the reparameterization estimator should be quite a bit lower than the score function estimator which is why it is preferred in this situation."
+      ],
+      "metadata": {
+        "id": "d-0tntSYdKPR"
+      }
+    }
+  ]
+}

Notebooks/Chap17/17_3_Importance_Sampling.ipynb

@@ -0,0 +1,496 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyMvae+1cigwg2Htl6vt1Who",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap17/17_3_Importance_Sampling.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 17.3: Importance sampling**\n",
+        "\n",
+        "This notebook investigates importance sampling as described in section 17.8.1 of the book.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt"
+      ],
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's approximate the expectation\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\mathbb{E}_{y}\\Bigl[\\exp\\bigl[- (y-1)^4\\bigr]\\Bigr] = \\int \\exp\\bigl[- (y-1)^4\\bigr] Pr(y) dy,\n",
+        "\\end{equation}\n",
+        "\n",
+        "where\n",
+        "\n",
+        "\\begin{equation}\n",
+        "Pr(y)=\\mbox{Norm}_y[0,1]\n",
+        "\\end{equation}\n",
+        "\n",
+        "by drawing $I$ samples $y_i$ and using the formula:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\mathbb{E}_{y}\\Bigl[\\exp\\bigl[- (y-1)^4\\bigr]\\Bigr] \\approx \\frac{1}{I} \\sum_{i=1}^I \\exp\\bigl[-(y-1)^4 \\bigr]\n",
+        "\\end{equation}"
+      ],
+      "metadata": {
+        "id": "f7a6xqKjkmvT"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def f(y):\n",
+        "  return np.exp(-(y-1) *(y-1) *(y-1) * (y-1))\n",
+        "\n",
+        "\n",
+        "def pr_y(y):\n",
+        "  return (1/np.sqrt(2*np.pi)) * np.exp(-0.5 * y * y)\n",
+        "\n",
+        "fig,ax = plt.subplots()\n",
+        "y = np.arange(-10,10,0.01)\n",
+        "ax.plot(y, f(y), 'r-', label='f$[y]$');\n",
+        "ax.plot(y, pr_y(y),'b-',label='$Pr(y)$')\n",
+        "ax.set_xlabel(\"$y$\")\n",
+        "ax.legend()\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "VjkzRr8o2ksg"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def compute_expectation(n_samples):\n",
+        "  # TODO -- compute this expectation\n",
+        "  # 1. Generate samples y_i using np.random.normal\n",
+        "  # 2. Approximate the expectation of f[y]\n",
+        "  # Replace this line\n",
+        "  expectation = 0\n",
+        "\n",
+        "\n",
+        "  return expectation"
+      ],
+      "metadata": {
+        "id": "LGAKHjUJnWmy"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set the seed so the random numbers are all the same\n",
+        "np.random.seed(0)\n",
+        "\n",
+        "# Compute the expectation  with a very large number of samples (good estimate)\n",
+        "n_samples = 100000000\n",
+        "expected_f= compute_expectation(n_samples)\n",
+        "print(\"Your value: \", expected_f, \", True value:  0.43160702267383166\")"
+      ],
+      "metadata": {
+        "id": "nmvixMqgodIP"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's investigate how the variance of this approximation decreases as we increase the number of samples $N$.\n",
+        "\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "Jr4UPcqmnXCS"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def compute_mean_variance(n_sample):\n",
+        "  n_estimate = 10000\n",
+        "  estimates = np.zeros((n_estimate,1))\n",
+        "  for i in range(n_estimate):\n",
+        "    estimates[i] = compute_expectation(n_sample.astype(int))\n",
+        "  return np.mean(estimates), np.var(estimates)"
+      ],
+      "metadata": {
+        "id": "yrDp1ILUo08j"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Compute the mean and variance for 1,2,... 20 samples\n",
+        "n_sample_all = np.array([1.,2,3,4,5,6,7,8,9,10,15,20,25,30,45,50,60,70,80,90,100,150,200,250,300,350,400,450,500])\n",
+        "mean_all = np.zeros_like(n_sample_all)\n",
+        "variance_all = np.zeros_like(n_sample_all)\n",
+        "for i in range(len(n_sample_all)):\n",
+        "  print(\"Computing mean and variance for expectation with %d samples\"%(n_sample_all[i]))\n",
+        "  mean_all[i],variance_all[i] = compute_mean_variance(n_sample_all[i])"
+      ],
+      "metadata": {
+        "id": "BcUVsodtqdey"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "fig,ax = plt.subplots()\n",
+        "ax.semilogx(n_sample_all, mean_all,'r-',label='mean estimate')\n",
+        "ax.fill_between(n_sample_all, mean_all-2*np.sqrt(variance_all), mean_all+2*np.sqrt(variance_all))\n",
+        "ax.set_xlabel(\"Number of samples\")\n",
+        "ax.set_ylabel(\"Mean of estimate\")\n",
+        "ax.plot([0,500],[0.43160702267383166,0.43160702267383166],'k--',label='true value')\n",
+        "ax.legend()\n",
+        "plt.show()\n"
+      ],
+      "metadata": {
+        "id": "feXmyk0krpUi"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "As you might expect, the more samples that we use to compute the approximate estimate, the lower the variance of the estimate."
+      ],
+      "metadata": {
+        "id": "XTUpxFlSuOl7"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        " Now consider the function\n",
+        " \\begin{equation}\n",
+        " \\mbox{f}[y]= 20.446\\exp\\left[-(y-3)^4\\right],\n",
+        " \\end{equation}\n",
+        "\n",
+        "which decreases rapidly as we move away from the position $y=4$."
+      ],
+      "metadata": {
+        "id": "6hxsl3Pxo1TT"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def f2(y):\n",
+        "  return 20.446*np.exp(- (y-3) *(y-3) *(y-3) * (y-3))\n",
+        "\n",
+        "fig,ax = plt.subplots()\n",
+        "y = np.arange(-10,10,0.01)\n",
+        "ax.plot(y, f2(y), 'r-', label='f$[y]$');\n",
+        "ax.plot(y, pr_y(y),'b-',label='$Pr(y)$')\n",
+        "ax.set_xlabel(\"$y$\")\n",
+        "ax.legend()\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "znydVPW7sL4P"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's again, compute the expectation:\n",
+        "\n",
+        "\\begin{eqnarray}\n",
+        "\\mathbb{E}_{y}\\left[\\mbox{f}[y]\\right] &=& \\int \\mbox{f}[y] Pr(y) dy\\\\\n",
+        "&\\approx& \\frac{1}{I} \\mbox{f}[y]\n",
+        "\\end{eqnarray}\n",
+        "\n",
+        "where $Pr(y)=\\mbox{Norm}_y[0,1]$ by approximating with samples $y_{i}$.\n"
+      ],
+      "metadata": {
+        "id": "G9Xxo0OJsIqD"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def compute_expectation2(n_samples):\n",
+        "  y = np.random.normal(size=(n_samples,1))\n",
+        "  expectation = np.mean(f2(y))\n",
+        "\n",
+        "  return expectation"
+      ],
+      "metadata": {
+        "id": "l8ZtmkA2vH4y"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set the seed so the random numbers are all the same\n",
+        "np.random.seed(0)\n",
+        "\n",
+        "# Compute the expectation with a very large number of samples (good estimate)\n",
+        "n_samples = 100000000\n",
+        "expected_f2= compute_expectation2(n_samples)\n",
+        "print(\"Expected value: \", expected_f2)"
+      ],
+      "metadata": {
+        "id": "dfUQyJ-svZ6F"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "I deliberately chose this function, because it's expectation is roughly the same as for the previous function.\n",
+        "\n",
+        "Again, let's look at the mean and the  variance of the estimates"
+      ],
+      "metadata": {
+        "id": "2sVDqP0BvxqM"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def compute_mean_variance2(n_sample):\n",
+        "  n_estimate = 10000\n",
+        "  estimates = np.zeros((n_estimate,1))\n",
+        "  for i in range(n_estimate):\n",
+        "    estimates[i] = compute_expectation2(n_sample.astype(int))\n",
+        "  return np.mean(estimates), np.var(estimates)\n",
+        "\n",
+        "# Compute the variance for 1,2,... 20 samples\n",
+        "mean_all2 = np.zeros_like(n_sample_all)\n",
+        "variance_all2 = np.zeros_like(n_sample_all)\n",
+        "for i in range(len(n_sample_all)):\n",
+        "  print(\"Computing variance for expectation with %d samples\"%(n_sample_all[i]))\n",
+        "  mean_all2[i], variance_all2[i] = compute_mean_variance2(n_sample_all[i])"
+      ],
+      "metadata": {
+        "id": "mHnILRkOv0Ir"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "fig,ax1 = plt.subplots()\n",
+        "ax1.semilogx(n_sample_all, mean_all,'r-',label='mean estimate')\n",
+        "ax1.fill_between(n_sample_all, mean_all-2*np.sqrt(variance_all), mean_all+2*np.sqrt(variance_all))\n",
+        "ax1.set_xlabel(\"Number of samples\")\n",
+        "ax1.set_ylabel(\"Mean of estimate\")\n",
+        "ax1.plot([1,500],[0.43160702267383166,0.43160702267383166],'k--',label='true value')\n",
+        "ax1.set_ylim(-5,6)\n",
+        "ax1.set_title(\"First function\")\n",
+        "ax1.legend()\n",
+        "\n",
+        "fig2,ax2 = plt.subplots()\n",
+        "ax2.semilogx(n_sample_all, mean_all2,'r-',label='mean estimate')\n",
+        "ax2.fill_between(n_sample_all, mean_all2-2*np.sqrt(variance_all2), mean_all2+2*np.sqrt(variance_all2))\n",
+        "ax2.set_xlabel(\"Number of samples\")\n",
+        "ax2.set_ylabel(\"Mean of estimate\")\n",
+        "ax2.plot([0,500],[0.43160428638892556,0.43160428638892556],'k--',label='true value')\n",
+        "ax2.set_ylim(-5,6)\n",
+        "ax2.set_title(\"Second function\")\n",
+        "ax2.legend()\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "FkCX-hxxAnsw"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "You can see that the variance of the estimate of the second function is considerably worse than the estimate of the variance of estimate of the first function\n",
+        "\n",
+        "TODO:  Think about why this is."
+      ],
+      "metadata": {
+        "id": "EtBP6NeLwZqz"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        " Now let's repeat this experiment with the second function, but this time use importance sampling with auxiliary distribution:\n",
+        "\n",
+        " \\begin{equation}\n",
+        "   q(y)=\\mbox{Norm}_{y}[3,1]\n",
+        " \\end{equation}\n"
+      ],
+      "metadata": {
+        "id": "_wuF-NoQu1--"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def q_y(y):\n",
+        "  return (1/np.sqrt(2*np.pi)) * np.exp(-0.5 * (y-3) * (y-3))\n",
+        "\n",
+        "def compute_expectation2b(n_samples):\n",
+        "  # TODO -- complete this function\n",
+        "  # 1. Draw n_samples from auxiliary distribution\n",
+        "  # 2. Compute f[y] for those samples\n",
+        "  # 3. Scale the results by pr_y / q_y\n",
+        "  # 4. Compute the mean of these weighted samples\n",
+        "  # Replace this line\n",
+        "  expectation = 0\n",
+        "\n",
+        "  return expectation"
+      ],
+      "metadata": {
+        "id": "jPm0AVYVIDnn"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set the seed so the random numbers are all the same\n",
+        "np.random.seed(0)\n",
+        "\n",
+        "# Compute the expectation with a very large number of samples (good estimate)\n",
+        "n_samples = 100000000\n",
+        "expected_f2= compute_expectation2b(n_samples)\n",
+        "print(\"Your value: \", expected_f2,\", True value:  0.43163734204459125 \")"
+      ],
+      "metadata": {
+        "id": "No2ByVvOM2yQ"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def compute_mean_variance2b(n_sample):\n",
+        "  n_estimate = 10000\n",
+        "  estimates = np.zeros((n_estimate,1))\n",
+        "  for i in range(n_estimate):\n",
+        "    estimates[i] = compute_expectation2b(n_sample.astype(int))\n",
+        "  return np.mean(estimates), np.var(estimates)\n",
+        "\n",
+        "# Compute the variance for 1,2,... 20 samples\n",
+        "mean_all2b = np.zeros_like(n_sample_all)\n",
+        "variance_all2b = np.zeros_like(n_sample_all)\n",
+        "for i in range(len(n_sample_all)):\n",
+        "  print(\"Computing variance for expectation with %d samples\"%(n_sample_all[i]))\n",
+        "  mean_all2b[i], variance_all2b[i] = compute_mean_variance2b(n_sample_all[i])"
+      ],
+      "metadata": {
+        "id": "6v8Jc7z4M3Mk"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "fig,ax1 = plt.subplots()\n",
+        "ax1.semilogx(n_sample_all, mean_all,'r-',label='mean estimate')\n",
+        "ax1.fill_between(n_sample_all, mean_all-2*np.sqrt(variance_all), mean_all+2*np.sqrt(variance_all))\n",
+        "ax1.set_xlabel(\"Number of samples\")\n",
+        "ax1.set_ylabel(\"Mean of estimate\")\n",
+        "ax1.plot([1,500],[0.43160702267383166,0.43160702267383166],'k--',label='true value')\n",
+        "ax1.set_ylim(-5,6)\n",
+        "ax1.set_title(\"First function\")\n",
+        "ax1.legend()\n",
+        "\n",
+        "fig2,ax2 = plt.subplots()\n",
+        "ax2.semilogx(n_sample_all, mean_all2,'r-',label='mean estimate')\n",
+        "ax2.fill_between(n_sample_all, mean_all2-2*np.sqrt(variance_all2), mean_all2+2*np.sqrt(variance_all2))\n",
+        "ax2.set_xlabel(\"Number of samples\")\n",
+        "ax2.set_ylabel(\"Mean of estimate\")\n",
+        "ax2.plot([0,500],[0.43160428638892556,0.43160428638892556],'k--',label='true value')\n",
+        "ax2.set_ylim(-5,6)\n",
+        "ax2.set_title(\"Second function\")\n",
+        "ax2.legend()\n",
+        "plt.show()\n",
+        "\n",
+        "fig2,ax2 = plt.subplots()\n",
+        "ax2.semilogx(n_sample_all, mean_all2b,'r-',label='mean estimate')\n",
+        "ax2.fill_between(n_sample_all, mean_all2b-2*np.sqrt(variance_all2b), mean_all2b+2*np.sqrt(variance_all2b))\n",
+        "ax2.set_xlabel(\"Number of samples\")\n",
+        "ax2.set_ylabel(\"Mean of estimate\")\n",
+        "ax2.plot([0,500],[ 0.43163734204459125, 0.43163734204459125],'k--',label='true value')\n",
+        "ax2.set_ylim(-5,6)\n",
+        "ax2.set_title(\"Second function with importance sampling\")\n",
+        "ax2.legend()\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "C0beD4sNNM3L"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "You can see that the importance sampling technique has reduced the amount of variance for any given number of samples."
+      ],
+      "metadata": {
+        "id": "y8rgge9MNiOc"
+      }
+    }
+  ]
+}

Notebooks/Chap18/18_1_Diffusion_Encoder.ipynb

@@ -0,0 +1,471 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyMpC8kgLnXx0XQBtwNAQ4jJ",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_1_Diffusion_Encoder.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 18.1: Diffusion Encoder**\n",
+        "\n",
+        "This notebook investigates the diffusion encoder as described in section 18.2 of the book.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "from matplotlib.colors import ListedColormap\n",
+        "from operator import itemgetter"
+      ],
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#Create pretty colormap as in book\n",
+        "my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
+        "my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
+        "r = np.floor(my_colormap_vals_dec/(256*256))\n",
+        "g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
+        "b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
+        "my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
+        "my_colormap = ListedColormap(my_colormap_vals)"
+      ],
+      "metadata": {
+        "id": "4PM8bf6lO0VE"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Probability distribution for normal\n",
+        "def norm_pdf(x, mu, sigma):\n",
+        "    return np.exp(-0.5 * (x-mu) * (x-mu) / (sigma * sigma)) / np.sqrt(2*np.pi*sigma*sigma)"
+      ],
+      "metadata": {
+        "id": "ONGRaQscfIOo"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# True distribution is a mixture of four Gaussians\n",
+        "class TrueDataDistribution:\n",
+        "  # Constructor initializes parameters\n",
+        "  def __init__(self):\n",
+        "    self.mu = [1.5, -0.216, 0.45, -1.875]\n",
+        "    self.sigma = [0.3, 0.15, 0.525, 0.075]\n",
+        "    self.w = [0.2, 0.3, 0.35, 0.15]\n",
+        "\n",
+        "  # Return PDF\n",
+        "  def pdf(self, x):\n",
+        "    return(self.w[0] *norm_pdf(x,self.mu[0],self.sigma[0]) +  self.w[1] *norm_pdf(x,self.mu[1],self.sigma[1]) + self.w[2] *norm_pdf(x,self.mu[2],self.sigma[2]) + self.w[3] *norm_pdf(x,self.mu[3],self.sigma[3]))\n",
+        "\n",
+        "  # Draw samples\n",
+        "  def sample(self, n):\n",
+        "    hidden = np.random.choice(4, n, p=self.w)\n",
+        "    epsilon = np.random.normal(size=(n))\n",
+        "    mu_list = list(itemgetter(*hidden)(self.mu))\n",
+        "    sigma_list = list(itemgetter(*hidden)(self.sigma))\n",
+        "    return mu_list + sigma_list * epsilon"
+      ],
+      "metadata": {
+        "id": "gZvG0MKhfY8Y"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Define ground truth probability distribution that we will model\n",
+        "true_dist = TrueDataDistribution()\n",
+        "# Let's visualize this\n",
+        "x_vals = np.arange(-3,3,0.01)\n",
+        "pr_x_true = true_dist.pdf(x_vals)\n",
+        "fig,ax = plt.subplots()\n",
+        "ax.plot(x_vals, pr_x_true, 'r-')\n",
+        "ax.set_xlabel(\"$x$\")\n",
+        "ax.set_ylabel(\"$Pr(x)$\")\n",
+        "ax.set_ylim(0,1.0)\n",
+        "ax.set_xlim(-3,3)\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "qXmej3TUuQyp"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's first implement the forward process"
+      ],
+      "metadata": {
+        "id": "XHdtfRP47YLy"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Do one step of diffusion (equation 18.1)\n",
+        "def diffuse_one_step(z_t_minus_1, beta_t):\n",
+        "  # TODO -- Implement this function\n",
+        "  # Use np.random.normal to generate the samples epsilon\n",
+        "  # Replace this line\n",
+        "  z_t = np.zeros_like(z_t_minus_1)\n",
+        "\n",
+        "  return z_t"
+      ],
+      "metadata": {
+        "id": "hkApJ2VJlQuk"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's run the diffusion process for a whole bunch of samples"
+      ],
+      "metadata": {
+        "id": "ECAUfHNi9NVW"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Generate some samples\n",
+        "n_sample = 10000\n",
+        "np.random.seed(6)\n",
+        "x = true_dist.sample(n_sample)\n",
+        "\n",
+        "# Number of time steps\n",
+        "T = 100\n",
+        "# Noise schedule has same value at every time step\n",
+        "beta = 0.01511\n",
+        "\n",
+        "# We'll store the diffused samples in an array\n",
+        "samples = np.zeros((T+1, n_sample))\n",
+        "samples[0,:] = x\n",
+        "\n",
+        "for t in range(T):\n",
+        "  samples[t+1,:] = diffuse_one_step(samples[t,:], beta)"
+      ],
+      "metadata": {
+        "id": "M-TY5w9Q8LYW"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's, plot the evolution of a few paths as in figure 18.2"
+      ],
+      "metadata": {
+        "id": "jYrAW6tN-gJ4"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "fig, ax = plt.subplots()\n",
+        "t_vals = np.arange(0,101,1)\n",
+        "ax.plot(samples[:,0],t_vals,'r-')\n",
+        "ax.plot(samples[:,1],t_vals,'g-')\n",
+        "ax.plot(samples[:,2],t_vals,'b-')\n",
+        "ax.plot(samples[:,3],t_vals,'c-')\n",
+        "ax.plot(samples[:,4],t_vals,'m-')\n",
+        "ax.set_xlim([-3,3])\n",
+        "ax.set_ylim([101, 0])\n",
+        "ax.set_xlabel('value')\n",
+        "ax.set_ylabel('z_{t}')\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "4XU6CDZC_kFo"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Notice that the samples have a tendency to move toward the center.  Now let's look at the histogram of the samples at each stage"
+      ],
+      "metadata": {
+        "id": "SGTYGGevAktz"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def draw_hist(z_t,title=''):\n",
+        "  fig, ax = plt.subplots()\n",
+        "  fig.set_size_inches(8,2.5)\n",
+        "  plt.hist(z_t , bins=np.arange(-3,3, 0.1), density = True)\n",
+        "  ax.set_xlim([-3,3])\n",
+        "  ax.set_ylim([0,1.0])\n",
+        "  ax.set_title('title')\n",
+        "  plt.show()"
+      ],
+      "metadata": {
+        "id": "bn5E5NzL-evM"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "draw_hist(samples[0,:],'Original data')\n",
+        "draw_hist(samples[5,:],'Time step 5')\n",
+        "draw_hist(samples[10,:],'Time step 10')\n",
+        "draw_hist(samples[20,:],'Time step 20')\n",
+        "draw_hist(samples[40,:],'Time step 40')\n",
+        "draw_hist(samples[80,:],'Time step 80')\n",
+        "draw_hist(samples[100,:],'Time step 100')"
+      ],
+      "metadata": {
+        "id": "pn_XD-EhBlwk"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "You can clearly see that as the diffusion process continues, the data becomes more Gaussian."
+      ],
+      "metadata": {
+        "id": "skuLfGl5Czf4"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's investigate the diffusion kernel as in figure 18.3 of the book.\n"
+      ],
+      "metadata": {
+        "id": "s37CBSzzK7wh"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def diffusion_kernel(x, t, beta):\n",
+        "    # TODO -- write this function\n",
+        "    # Replace this line:\n",
+        "    dk_mean = 0.0 ; dk_std = 1.0\n",
+        "\n",
+        "    return dk_mean, dk_std"
+      ],
+      "metadata": {
+        "id": "vL62Iym0LEtY"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def draw_prob_dist(x_plot_vals, prob_dist, title=''):\n",
+        "  fig, ax = plt.subplots()\n",
+        "  fig.set_size_inches(8,2.5)\n",
+        "  ax.plot(x_plot_vals, prob_dist, 'b-')\n",
+        "  ax.set_xlim([-3,3])\n",
+        "  ax.set_ylim([0,1.0])\n",
+        "  ax.set_title(title)\n",
+        "  plt.show()\n",
+        "\n",
+        "def compute_and_plot_diffusion_kernels(x, T, beta, my_colormap):\n",
+        "    x_plot_vals = np.arange(-3,3,0.01)\n",
+        "    diffusion_kernels = np.zeros((T+1,len(x_plot_vals)))\n",
+        "    dk_mean_all = np.ones((T+1,1))*x\n",
+        "    dk_std_all = np.zeros((T+1,1))\n",
+        "    for t in range(T):\n",
+        "      dk_mean_all[t+1], dk_std_all[t+1] = diffusion_kernel(x,t+1,beta)\n",
+        "      diffusion_kernels[t+1,:] =  norm_pdf(x_plot_vals, dk_mean_all[t+1], dk_std_all[t+1])\n",
+        "\n",
+        "    samples = np.ones((T+1, 5))\n",
+        "    samples[0,:] = x\n",
+        "\n",
+        "    for t in range(T):\n",
+        "      samples[t+1,:] = diffuse_one_step(samples[t,:], beta)\n",
+        "\n",
+        "    fig, ax = plt.subplots()\n",
+        "    fig.set_size_inches(6,6)\n",
+        "\n",
+        "    # Plot the image containing all the kernels\n",
+        "    plt.imshow(diffusion_kernels, cmap=my_colormap, interpolation='nearest')\n",
+        "\n",
+        "    # Plot +/- 2 standard deviations\n",
+        "    ax.plot((dk_mean_all -2 * dk_std_all)/0.01 + len(x_plot_vals)/2, np.arange(0,101,1),'y--')\n",
+        "    ax.plot((dk_mean_all +2 * dk_std_all)/0.01 + len(x_plot_vals)/2, np.arange(0,101,1),'y--')\n",
+        "\n",
+        "    # Plot a few trajectories\n",
+        "    ax.plot(samples[:,0]/0.01 + + len(x_plot_vals)/2, np.arange(0,101,1), 'r-')\n",
+        "    ax.plot(samples[:,1]/0.01 + + len(x_plot_vals)/2, np.arange(0,101,1), 'g-')\n",
+        "    ax.plot(samples[:,2]/0.01 + + len(x_plot_vals)/2, np.arange(0,101,1), 'b-')\n",
+        "    ax.plot(samples[:,3]/0.01 + + len(x_plot_vals)/2, np.arange(0,101,1), 'c-')\n",
+        "    ax.plot(samples[:,4]/0.01 + + len(x_plot_vals)/2, np.arange(0,101,1), 'm-')\n",
+        "\n",
+        "    # Tidy up and plot\n",
+        "    ax.set_ylabel(\"$Pr(z_{t}|x)$\")\n",
+        "    ax.get_xaxis().set_visible(False)\n",
+        "    ax.set_xlim([0,601])\n",
+        "    ax.set_aspect(601/T)\n",
+        "    plt.show()\n",
+        "\n",
+        "\n",
+        "    draw_prob_dist(x_plot_vals, diffusion_kernels[20,:],'$q(z_{20}|x)$')\n",
+        "    draw_prob_dist(x_plot_vals, diffusion_kernels[40,:],'$q(z_{40}|x)$')\n",
+        "    draw_prob_dist(x_plot_vals, diffusion_kernels[80,:],'$q(z_{80}|x)$')"
+      ],
+      "metadata": {
+        "id": "KtP1KF8wMh8o"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "x = -2\n",
+        "compute_and_plot_diffusion_kernels(x, T, beta, my_colormap)"
+      ],
+      "metadata": {
+        "id": "g8TcI5wtRQsx"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "TODO -- Run this for different version of $x$ and check that you understand how the graphs change"
+      ],
+      "metadata": {
+        "id": "-RuN2lR28-hK"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Finally, let's estimate the marginal distributions empirically and visualize them as in figure 18.4 of the book. This is only tractable because the data is in one dimension and we know the original distribution.\n",
+        "\n",
+        "The marginal distribution at time t is the sum of the diffusion kernels for each position x, weighted by the probability of seeing that value of x in the true distribution."
+      ],
+      "metadata": {
+        "id": "n-x6Whz2J_zy"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def diffusion_marginal(x_plot_vals, pr_x_true, t, beta):\n",
+        "    # If time is zero then marginal is just original distribution\n",
+        "    if t == 0:\n",
+        "        return pr_x_true\n",
+        "\n",
+        "    # The thing we are computing\n",
+        "    marginal_at_time_t = np.zeros_like(pr_x_true);\n",
+        "\n",
+        "\n",
+        "    # TODO Write ths function\n",
+        "    # 1. For each x (value in x_plot_vals):\n",
+        "      # 2. Compute the mean and variance of the diffusion kernel at time t\n",
+        "      # 3. Compute pdf of this Gaussian at every x_plot_val\n",
+        "      # 4. Weight Gaussian by probability at position x and by 0.01 to compensate for bin size\n",
+        "      # 5. Accumulate weighted Gaussian in marginal at time t.\n",
+        "    # 6. Multiply result by 0.01 to compensate for bin size\n",
+        "    # Replace this line:\n",
+        "    marginal_at_time_t = marginal_at_time_t\n",
+        "\n",
+        "\n",
+        "\n",
+        "    return marginal_at_time_t"
+      ],
+      "metadata": {
+        "id": "YzN5duYpg7C-"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "x_plot_vals = np.arange(-3,3,0.01)\n",
+        "marginal_distributions = np.zeros((T+1,len(x_plot_vals)))\n",
+        "\n",
+        "for t in range(T+1):\n",
+        "  marginal_distributions[t,:] = diffusion_marginal(x_plot_vals, pr_x_true , t, beta)\n",
+        "\n",
+        "fig, ax = plt.subplots()\n",
+        "fig.set_size_inches(6,6)\n",
+        "\n",
+        "# Plot the image containing all the kernels\n",
+        "plt.imshow(marginal_distributions, cmap=my_colormap, interpolation='nearest')\n",
+        "\n",
+        "# Tidy up and plot\n",
+        "ax.set_ylabel(\"$Pr(z_{t})$\")\n",
+        "ax.get_xaxis().set_visible(False)\n",
+        "ax.set_xlim([0,601])\n",
+        "ax.set_aspect(601/T)\n",
+        "plt.show()\n",
+        "\n",
+        "\n",
+        "draw_prob_dist(x_plot_vals, marginal_distributions[0,:],'$q(z_{0})$')\n",
+        "draw_prob_dist(x_plot_vals, marginal_distributions[20,:],'$q(z_{20})$')\n",
+        "draw_prob_dist(x_plot_vals, marginal_distributions[60,:],'$q(z_{60})$')"
+      ],
+      "metadata": {
+        "id": "OgEU9sxjRaeO"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

Notebooks/Chap18/18_2_1D_Diffusion_Model.ipynb

@@ -0,0 +1,380 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyM4DdZDGoP1xGst+Nn+rwvt",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_2_1D_Diffusion_Model.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 18.2: 1D Diffusion Model**\n",
+        "\n",
+        "This notebook investigates the diffusion encoder as described in section 18.3 and 18.4 of the book.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "from matplotlib.colors import ListedColormap\n",
+        "from operator import itemgetter\n",
+        "from scipy import stats\n",
+        "from IPython.display import display, clear_output"
+      ],
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#Create pretty colormap as in book\n",
+        "my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
+        "my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
+        "r = np.floor(my_colormap_vals_dec/(256*256))\n",
+        "g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
+        "b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
+        "my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
+        "my_colormap = ListedColormap(my_colormap_vals)"
+      ],
+      "metadata": {
+        "id": "4PM8bf6lO0VE"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Probability distribution for normal\n",
+        "def norm_pdf(x, mu, sigma):\n",
+        "    return np.exp(-0.5 * (x-mu) * (x-mu) / (sigma * sigma)) / np.sqrt(2*np.pi*sigma*sigma)"
+      ],
+      "metadata": {
+        "id": "ONGRaQscfIOo"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# True distribution is a mixture of four Gaussians\n",
+        "class TrueDataDistribution:\n",
+        "  # Constructor initializes parameters\n",
+        "  def __init__(self):\n",
+        "    self.mu = [1.5, -0.216, 0.45, -1.875]\n",
+        "    self.sigma = [0.3, 0.15, 0.525, 0.075]\n",
+        "    self.w = [0.2, 0.3, 0.35, 0.15]\n",
+        "\n",
+        "  # Return PDF\n",
+        "  def pdf(self, x):\n",
+        "    return(self.w[0] *norm_pdf(x,self.mu[0],self.sigma[0]) +  self.w[1] *norm_pdf(x,self.mu[1],self.sigma[1]) + self.w[2] *norm_pdf(x,self.mu[2],self.sigma[2]) + self.w[3] *norm_pdf(x,self.mu[3],self.sigma[3]))\n",
+        "\n",
+        "  # Draw samples\n",
+        "  def sample(self, n):\n",
+        "    hidden = np.random.choice(4, n, p=self.w)\n",
+        "    epsilon = np.random.normal(size=(n))\n",
+        "    mu_list = list(itemgetter(*hidden)(self.mu))\n",
+        "    sigma_list = list(itemgetter(*hidden)(self.sigma))\n",
+        "    return mu_list + sigma_list * epsilon"
+      ],
+      "metadata": {
+        "id": "gZvG0MKhfY8Y"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Define ground truth probability distribution that we will model\n",
+        "true_dist = TrueDataDistribution()\n",
+        "# Let's visualize this\n",
+        "x_vals = np.arange(-3,3,0.01)\n",
+        "pr_x_true = true_dist.pdf(x_vals)\n",
+        "fig,ax = plt.subplots()\n",
+        "fig.set_size_inches(8,2.5)\n",
+        "ax.plot(x_vals, pr_x_true, 'r-')\n",
+        "ax.set_xlabel(\"$x$\")\n",
+        "ax.set_ylabel(\"$Pr(x)$\")\n",
+        "ax.set_ylim(0,1.0)\n",
+        "ax.set_xlim(-3,3)\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "iJu_uBiaeUVv"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "To train the model to describe this distribution, we'll need to generate pairs of samples drawn from $Pr(z_t|x)$ (diffusion kernel) and $q(z_{t-1}|z_{t},x)$ (equation 18.15).\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "DRHUG_41i4t_"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# The diffusion kernel returns the parameters of Pr(z_{t}|x)\n",
+        "def diffusion_kernel(x, t, beta):\n",
+        "    alpha = np.power(1-beta,t)\n",
+        "    dk_mean = x * np.sqrt(alpha)\n",
+        "    dk_std = np.sqrt(1-alpha)\n",
+        "    return dk_mean, dk_std\n",
+        "\n",
+        "# Compute mean and variance q(z_{t-1}|z_{t},x)\n",
+        "def conditional_diffusion_distribution(x,z_t,t,beta):\n",
+        "    # TODO -- Implement this function\n",
+        "    # Replace this line\n",
+        "    cd_mean = 0; cd_std = 1\n",
+        "\n",
+        "    return cd_mean, cd_std\n",
+        "\n",
+        "def get_data_pairs(x_train,t,beta):\n",
+        "    # Find diffusion kernel for every x_train and draw samples\n",
+        "    dk_mean, dk_std = diffusion_kernel(x_train, t, beta)\n",
+        "    z_t = np.random.normal(size=x_train.shape) * dk_std + dk_mean\n",
+        "    # Find conditional diffusion distribution for each x_train, z pair and draw samples\n",
+        "    cd_mean, cd_std  = conditional_diffusion_distribution(x_train,z_t,t,beta)\n",
+        "    if t == 1:\n",
+        "      z_tminus1 = x_train\n",
+        "    else:\n",
+        "      z_tminus1 = np.random.normal(size=x_train.shape) * cd_std + cd_mean\n",
+        "\n",
+        "    return z_t, z_tminus1"
+      ],
+      "metadata": {
+        "id": "x6B8t72Ukscd"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "We also need models $\\mbox{f}_t[z_{t},\\phi_{t}]$ that map from $z_{t}$ to the mean of the distribution at time $z_{t-1}$.  We're just going to use a very hacky non-parametric model (basically a lookup table) that tells you the result based on the (quantized) input."
+      ],
+      "metadata": {
+        "id": "aSG_4uA8_zZ-"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# This code is really ugly!  Don't look too closely at it!\n",
+        "# All you need to know is that it is a model that trains from pairs zt, zt_minus1\n",
+        "# And can then predict zt\n",
+        "class NonParametricModel():\n",
+        "   # Constructor initializes parameters\n",
+        "  def __init__(self):\n",
+        "\n",
+        "    self.inc = 0.01\n",
+        "    self.max_val = 3.0\n",
+        "    self.model = []\n",
+        "\n",
+        "  # Learns a model that predicts z_t_minus1 given z_t\n",
+        "  def train(self, zt, zt_minus1):\n",
+        "      zt = np.clip(zt,-self.max_val,self.max_val)\n",
+        "      zt_minus1 = np.clip(zt_minus1,-self.max_val,self.max_val)\n",
+        "      bins = np.arange(-self.max_val,self.max_val+self.inc,self.inc)\n",
+        "      numerator, *_ = stats.binned_statistic(zt, zt_minus1-zt, statistic='sum',bins=bins)\n",
+        "      denominator, *_ = stats.binned_statistic(zt, zt_minus1-zt, statistic='count',bins=bins)\n",
+        "      self.model = numerator / (denominator + 1)\n",
+        "\n",
+        "  def predict(self, zt):\n",
+        "      bin_index = np.floor((zt+self.max_val)/self.inc)\n",
+        "      bin_index = np.clip(bin_index,0, len(self.model)-1).astype('uint32')\n",
+        "      return zt + self.model[bin_index]"
+      ],
+      "metadata": {
+        "id": "ZHViC0pL_yy5"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Sample data from distribution (this would usually be our collected training set)\n",
+        "n_sample = 100000\n",
+        "x_train = true_dist.sample(n_sample)\n",
+        "\n",
+        "# Define model parameters\n",
+        "T = 100\n",
+        "beta = 0.01511\n",
+        "\n",
+        "all_models = []\n",
+        "for t in range(0,T):\n",
+        "    clear_output(wait=True)\n",
+        "    display(\"Training timestep %d\"%(t))\n",
+        "    zt,zt_minus1 = get_data_pairs(x_train,t+1,beta)\n",
+        "    all_models.append(NonParametricModel())\n",
+        "    # The model at index t maps data from z_{t+1} to z_{t}\n",
+        "    all_models[t].train(zt,zt_minus1)"
+      ],
+      "metadata": {
+        "id": "CzVFybWoBygu"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now that we've learned the model, let's draw some samples from it.  We start at $z_{100}$ and use the model to predict $z_{99}$, then $z_{98}$ and so on until finally we get to $z_{1}$ and then $x$ (represented as $z_{0}$ here). We'll store all of the intermediate stages as well, so we can plot the trajectories.  See equations 18.16."
+      ],
+      "metadata": {
+        "id": "ZPc9SEvtl14U"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def sample(model, T, sigma_t, n_samples):\n",
+        "    # Create the output array\n",
+        "    # Each row represents a time step, first row will be sampled data\n",
+        "    # Each column represents a different sample\n",
+        "    samples = np.zeros((T+1,n_samples))\n",
+        "\n",
+        "    # TODO -- Initialize the samples z_{T} at samples[T,:] from standard normal distribution\n",
+        "    # Replace this line\n",
+        "    samples[T,:] = np.zeros((1,n_samples))\n",
+        "\n",
+        "\n",
+        "    # For t=100...99..98... ...0\n",
+        "    for t in range(T,0,-1):\n",
+        "        clear_output(wait=True)\n",
+        "        display(\"Predicting z_{%d} from z_{%d}\"%(t-1,t))\n",
+        "        # TODO Predict samples[t-1,:] from samples[t,:] using the appropriate model\n",
+        "        # Replace this line:\n",
+        "        samples[t-1,:] = np.zeros((1,n_samples))\n",
+        "\n",
+        "\n",
+        "        # If not the last time step\n",
+        "        if t>0:\n",
+        "            # TODO Add noise to the samples at z_t-1 we just generated with mean zero, standard deviation sigma_t\n",
+        "            # Replace this line\n",
+        "            samples[t-1,:] = samples[t-1,:]\n",
+        "\n",
+        "    return samples"
+      ],
+      "metadata": {
+        "id": "A-ZMFOvACIOw"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's run the diffusion process for a whole bunch of samples"
+      ],
+      "metadata": {
+        "id": "ECAUfHNi9NVW"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "sigma_t=0.12288\n",
+        "n_samples = 100000\n",
+        "samples = sample(all_models, T, sigma_t, n_samples)\n",
+        "\n",
+        "\n",
+        "# Plot the data\n",
+        "sampled_data = samples[0,:]\n",
+        "bins = np.arange(-3,3.05,0.05)\n",
+        "\n",
+        "fig,ax = plt.subplots()\n",
+        "fig.set_size_inches(8,2.5)\n",
+        "ax.set_xlim([-3,3])\n",
+        "plt.hist(sampled_data, bins=bins, density =True)\n",
+        "ax.set_ylim(0, 0.8)\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "M-TY5w9Q8LYW"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's, plot the evolution of a few of the paths as in figure 18.7 (paths are from bottom to top now)."
+      ],
+      "metadata": {
+        "id": "jYrAW6tN-gJ4"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "fig, ax = plt.subplots()\n",
+        "t_vals = np.arange(0,101,1)\n",
+        "ax.plot(samples[:,0],t_vals,'r-')\n",
+        "ax.plot(samples[:,1],t_vals,'g-')\n",
+        "ax.plot(samples[:,2],t_vals,'b-')\n",
+        "ax.plot(samples[:,3],t_vals,'c-')\n",
+        "ax.plot(samples[:,4],t_vals,'m-')\n",
+        "ax.set_xlim([-3,3])\n",
+        "ax.set_ylim([101, 0])\n",
+        "ax.set_xlabel('value')\n",
+        "ax.set_ylabel('z_{t}')\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "4XU6CDZC_kFo"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Notice that the samples have a tendency to move from positions that are near the center at time 100 to positions that are high in the true probability distribution at time 0"
+      ],
+      "metadata": {
+        "id": "SGTYGGevAktz"
+      }
+    }
+  ]
+}

Notebooks/Chap18/18_3_Reparameterized_Model.ipynb

@@ -0,0 +1,362 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyNd+D0/IVWXtU2GKsofyk2d",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_3_Reparameterized_Model.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 18.3: 1D Reparameterized model**\n",
+        "\n",
+        "This notebook investigates the reparameterized model as described in section 18.5 of the book and implements algorithms 18.1 and 18.2.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "from matplotlib.colors import ListedColormap\n",
+        "from operator import itemgetter\n",
+        "from scipy import stats\n",
+        "from IPython.display import display, clear_output"
+      ],
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#Create pretty colormap as in book\n",
+        "my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
+        "my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
+        "r = np.floor(my_colormap_vals_dec/(256*256))\n",
+        "g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
+        "b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
+        "my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
+        "my_colormap = ListedColormap(my_colormap_vals)"
+      ],
+      "metadata": {
+        "id": "4PM8bf6lO0VE"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Probability distribution for normal\n",
+        "def norm_pdf(x, mu, sigma):\n",
+        "    return np.exp(-0.5 * (x-mu) * (x-mu) / (sigma * sigma)) / np.sqrt(2*np.pi*sigma*sigma)"
+      ],
+      "metadata": {
+        "id": "ONGRaQscfIOo"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# True distribution is a mixture of four Gaussians\n",
+        "class TrueDataDistribution:\n",
+        "  # Constructor initializes parameters\n",
+        "  def __init__(self):\n",
+        "    self.mu = [1.5, -0.216, 0.45, -1.875]\n",
+        "    self.sigma = [0.3, 0.15, 0.525, 0.075]\n",
+        "    self.w = [0.2, 0.3, 0.35, 0.15]\n",
+        "\n",
+        "  # Return PDF\n",
+        "  def pdf(self, x):\n",
+        "    return(self.w[0] *norm_pdf(x,self.mu[0],self.sigma[0]) +  self.w[1] *norm_pdf(x,self.mu[1],self.sigma[1]) + self.w[2] *norm_pdf(x,self.mu[2],self.sigma[2]) + self.w[3] *norm_pdf(x,self.mu[3],self.sigma[3]))\n",
+        "\n",
+        "  # Draw samples\n",
+        "  def sample(self, n):\n",
+        "    hidden = np.random.choice(4, n, p=self.w)\n",
+        "    epsilon = np.random.normal(size=(n))\n",
+        "    mu_list = list(itemgetter(*hidden)(self.mu))\n",
+        "    sigma_list = list(itemgetter(*hidden)(self.sigma))\n",
+        "    return mu_list + sigma_list * epsilon"
+      ],
+      "metadata": {
+        "id": "gZvG0MKhfY8Y"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Define ground truth probability distribution that we will model\n",
+        "true_dist = TrueDataDistribution()\n",
+        "# Let's visualize this\n",
+        "x_vals = np.arange(-3,3,0.01)\n",
+        "pr_x_true = true_dist.pdf(x_vals)\n",
+        "fig,ax = plt.subplots()\n",
+        "fig.set_size_inches(8,2.5)\n",
+        "ax.plot(x_vals, pr_x_true, 'r-')\n",
+        "ax.set_xlabel(\"$x$\")\n",
+        "ax.set_ylabel(\"$Pr(x)$\")\n",
+        "ax.set_ylim(0,1.0)\n",
+        "ax.set_xlim(-3,3)\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "iJu_uBiaeUVv"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "To train the model to describe this distribution, we'll need to generate pairs of samples drawn from $Pr(z_t|x)$ (diffusion kernel) and $q(z_{t-1}|z_{t},x)$ (equation 18.15).\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "DRHUG_41i4t_"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Return z_t (the argument of g_{t}[] in the loss function in algorithm 18.1) and epsilon\n",
+        "def get_data_pairs(x_train,t,beta):\n",
+        "    # TODO -- write this function\n",
+        "    # Replace these lines\n",
+        "    epsilon = np.ones_like(x_train)\n",
+        "    z_t = np.ones_like(x_train)\n",
+        "\n",
+        "    return z_t, epsilon"
+      ],
+      "metadata": {
+        "id": "x6B8t72Ukscd"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "We also need models $\\mbox{g}_t[z_{t},\\phi_{t}]$ that map from $z_{t}$ to the noise $\\epsilon$ that was added.  We're just going to use a very hacky non-parametric model (basically a lookup table) that tells you the result based on the (quantized) input."
+      ],
+      "metadata": {
+        "id": "aSG_4uA8_zZ-"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# This code is really ugly!  Don't look too closely at it!\n",
+        "# All you need to know is that it is a model that trains from pairs zt, zt_minus1\n",
+        "# And can then predict zt\n",
+        "class NonParametricModel():\n",
+        "   # Constructor initializes parameters\n",
+        "  def __init__(self):\n",
+        "\n",
+        "    self.inc = 0.01\n",
+        "    self.max_val = 3.0\n",
+        "    self.model = []\n",
+        "\n",
+        "  # Learns a model that predicts epsilon given z_t\n",
+        "  def train(self, zt, epsilon):\n",
+        "      zt = np.clip(zt,-self.max_val,self.max_val)\n",
+        "      epsilon = np.clip(epsilon,-self.max_val,self.max_val)\n",
+        "      bins = np.arange(-self.max_val,self.max_val+self.inc,self.inc)\n",
+        "      numerator, *_ = stats.binned_statistic(zt, epsilon, statistic='sum',bins=bins)\n",
+        "      denominator, *_ = stats.binned_statistic(zt, epsilon, statistic='count',bins=bins)\n",
+        "      self.model = numerator / (denominator + 1)\n",
+        "\n",
+        "  def predict(self, zt):\n",
+        "      bin_index = np.floor((zt+self.max_val)/self.inc)\n",
+        "      bin_index = np.clip(bin_index,0, len(self.model)-1).astype('uint32')\n",
+        "      return self.model[bin_index]"
+      ],
+      "metadata": {
+        "id": "ZHViC0pL_yy5"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Sample data from distribution (this would usually be our collected training set)\n",
+        "n_sample = 100000\n",
+        "x_train = true_dist.sample(n_sample)\n",
+        "\n",
+        "# Define model parameters\n",
+        "T = 100\n",
+        "beta = 0.01511\n",
+        "\n",
+        "all_models = []\n",
+        "for t in range(0,T):\n",
+        "    clear_output(wait=True)\n",
+        "    display(\"Training timestep %d\"%(t))\n",
+        "    zt,epsilon= get_data_pairs(x_train,t,beta)\n",
+        "    all_models.append(NonParametricModel())\n",
+        "    # The model at index t maps data from z_{t+1} to epsilon\n",
+        "    all_models[t].train(zt,epsilon)"
+      ],
+      "metadata": {
+        "id": "CzVFybWoBygu"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now that we've learned the model, let's draw some samples from it.  We start at $z_{100}$ and use the model to predict $z_{99}$, then $z_{98}$ and so on until finally we get to $z_{1}$ and then $x$ (represented as $z_{0}$ here). We'll store all of the intermediate stages as well, so we can plot the trajectories.  See algorithm 18.2"
+      ],
+      "metadata": {
+        "id": "ZPc9SEvtl14U"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def sample(model, T, sigma_t, n_samples):\n",
+        "    # Create the output array\n",
+        "    # Each row represents a time step, first row will be sampled data\n",
+        "    # Each column represents a different sample\n",
+        "    samples = np.zeros((T+1,n_samples))\n",
+        "\n",
+        "    # TODO -- Initialize the samples z_{T} at samples[T,:] from standard normal distribution\n",
+        "    # Replace this line\n",
+        "    samples[T,:] = np.zeros((1,n_samples))\n",
+        "\n",
+        "\n",
+        "\n",
+        "    # For t=100...99..98... ...0\n",
+        "    for t in range(T,0,-1):\n",
+        "        clear_output(wait=True)\n",
+        "        display(\"Predicting z_{%d} from z_{%d}\"%(t-1,t))\n",
+        "        # TODO Predict samples[t-1,:] from samples[t,:] using the appropriate model\n",
+        "        # Replace this line:\n",
+        "        samples[t-1,:] = np.zeros((1,n_samples))\n",
+        "\n",
+        "\n",
+        "        # If not the last time step\n",
+        "        if t>0:\n",
+        "            # TODO Add noise to the samples at z_t-1 we just generated with mean zero, standard deviation sigma_t\n",
+        "            # Replace this line\n",
+        "            samples[t-1,:] = samples[t-1,:]\n",
+        "\n",
+        "    return samples"
+      ],
+      "metadata": {
+        "id": "A-ZMFOvACIOw"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's run the diffusion process for a whole bunch of samples"
+      ],
+      "metadata": {
+        "id": "ECAUfHNi9NVW"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "sigma_t=0.12288\n",
+        "n_samples = 100000\n",
+        "samples = sample(all_models, T, sigma_t, n_samples)\n",
+        "\n",
+        "\n",
+        "# Plot the data\n",
+        "sampled_data = samples[0,:]\n",
+        "bins = np.arange(-3,3.05,0.05)\n",
+        "\n",
+        "fig,ax = plt.subplots()\n",
+        "fig.set_size_inches(8,2.5)\n",
+        "ax.set_xlim([-3,3])\n",
+        "plt.hist(sampled_data, bins=bins, density =True)\n",
+        "ax.set_ylim(0, 0.8)\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "M-TY5w9Q8LYW"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's, plot the evolution of a few of the paths as in figure 18.7 (paths are from bottom to top now)."
+      ],
+      "metadata": {
+        "id": "jYrAW6tN-gJ4"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "fig, ax = plt.subplots()\n",
+        "t_vals = np.arange(0,101,1)\n",
+        "ax.plot(samples[:,0],t_vals,'r-')\n",
+        "ax.plot(samples[:,1],t_vals,'g-')\n",
+        "ax.plot(samples[:,2],t_vals,'b-')\n",
+        "ax.plot(samples[:,3],t_vals,'c-')\n",
+        "ax.plot(samples[:,4],t_vals,'m-')\n",
+        "ax.set_xlim([-3,3])\n",
+        "ax.set_ylim([101, 0])\n",
+        "ax.set_xlabel('value')\n",
+        "ax.set_ylabel('z_{t}')\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "4XU6CDZC_kFo"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Notice that the samples have a tendency to move from positions that are near the center at time 100 to positions that are high in the true probability distribution at time 0"
+      ],
+      "metadata": {
+        "id": "SGTYGGevAktz"
+      }
+    }
+  ]
+}

Notebooks/Chap18/18_4_Families_of_Diffusion_Models.ipynb

@@ -0,0 +1,484 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyNFSvISBXo/Z1l+onknF2Gw",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_4_Families_of_Diffusion_Models.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 18.4: Families of diffusion models**\n",
+        "\n",
+        "This notebook investigates the reparameterized model as described in section 18.5 of the book and computers the results shown in figure 18.10c-f.  These models are based on the paper \"Denoising diffusion implicit models\" which can be found [here](https://arxiv.org/pdf/2010.02502.pdf).\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "from matplotlib.colors import ListedColormap\n",
+        "from operator import itemgetter\n",
+        "from scipy import stats\n",
+        "from IPython.display import display, clear_output"
+      ],
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#Create pretty colormap as in book\n",
+        "my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
+        "my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
+        "r = np.floor(my_colormap_vals_dec/(256*256))\n",
+        "g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
+        "b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
+        "my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
+        "my_colormap = ListedColormap(my_colormap_vals)"
+      ],
+      "metadata": {
+        "id": "4PM8bf6lO0VE"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Probability distribution for normal\n",
+        "def norm_pdf(x, mu, sigma):\n",
+        "    return np.exp(-0.5 * (x-mu) * (x-mu) / (sigma * sigma)) / np.sqrt(2*np.pi*sigma*sigma)"
+      ],
+      "metadata": {
+        "id": "ONGRaQscfIOo"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# True distribution is a mixture of four Gaussians\n",
+        "class TrueDataDistribution:\n",
+        "  # Constructor initializes parameters\n",
+        "  def __init__(self):\n",
+        "    self.mu = [1.5, -0.216, 0.45, -1.875]\n",
+        "    self.sigma = [0.3, 0.15, 0.525, 0.075]\n",
+        "    self.w = [0.2, 0.3, 0.35, 0.15]\n",
+        "\n",
+        "  # Return PDF\n",
+        "  def pdf(self, x):\n",
+        "    return(self.w[0] *norm_pdf(x,self.mu[0],self.sigma[0]) +  self.w[1] *norm_pdf(x,self.mu[1],self.sigma[1]) + self.w[2] *norm_pdf(x,self.mu[2],self.sigma[2]) + self.w[3] *norm_pdf(x,self.mu[3],self.sigma[3]))\n",
+        "\n",
+        "  # Draw samples\n",
+        "  def sample(self, n):\n",
+        "    hidden = np.random.choice(4, n, p=self.w)\n",
+        "    epsilon = np.random.normal(size=(n))\n",
+        "    mu_list = list(itemgetter(*hidden)(self.mu))\n",
+        "    sigma_list = list(itemgetter(*hidden)(self.sigma))\n",
+        "    return mu_list + sigma_list * epsilon"
+      ],
+      "metadata": {
+        "id": "gZvG0MKhfY8Y"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Define ground truth probability distribution that we will model\n",
+        "true_dist = TrueDataDistribution()\n",
+        "# Let's visualize this\n",
+        "x_vals = np.arange(-3,3,0.01)\n",
+        "pr_x_true = true_dist.pdf(x_vals)\n",
+        "fig,ax = plt.subplots()\n",
+        "fig.set_size_inches(8,2.5)\n",
+        "ax.plot(x_vals, pr_x_true, 'r-')\n",
+        "ax.set_xlabel(\"$x$\")\n",
+        "ax.set_ylabel(\"$Pr(x)$\")\n",
+        "ax.set_ylim(0,1.0)\n",
+        "ax.set_xlim(-3,3)\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "iJu_uBiaeUVv"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "To train the model to describe this distribution, we'll need to generate pairs of samples drawn from $Pr(z_t|x)$ (diffusion kernel) and $q(z_{t-1}|z_{t},x)$ (equation 18.15).\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "DRHUG_41i4t_"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Return z_t (the argument of g_{t}[] in the loss function in algorithm 18.1) and epsilon\n",
+        "def get_data_pairs(x_train,t,beta):\n",
+        "\n",
+        "    epsilon = np.random.standard_normal(x_train.shape)\n",
+        "    alpha_t = np.power(1-beta,t)\n",
+        "    z_t = x_train * np.sqrt(alpha_t) + np.sqrt(1-alpha_t) * epsilon\n",
+        "\n",
+        "    return z_t, epsilon"
+      ],
+      "metadata": {
+        "id": "x6B8t72Ukscd"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "We also need models $\\mbox{g}_t[z_{t},\\phi_{t}]$ that map from $z_{t}$ to the noise $\\epsilon$ that was added.  We're just going to use a very hacky non-parametric model (basically a lookup table) that tells you the result based on the (quantized) input."
+      ],
+      "metadata": {
+        "id": "aSG_4uA8_zZ-"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# This code is really ugly!  Don't look too closely at it!\n",
+        "# All you need to know is that it is a model that trains from pairs zt, zt_minus1\n",
+        "# And can then predict zt\n",
+        "class NonParametricModel():\n",
+        "   # Constructor initializes parameters\n",
+        "  def __init__(self):\n",
+        "\n",
+        "    self.inc = 0.01\n",
+        "    self.max_val = 3.0\n",
+        "    self.model = []\n",
+        "\n",
+        "  # Learns a model that predicts epsilon given z_t\n",
+        "  def train(self, zt, epsilon):\n",
+        "      zt = np.clip(zt,-self.max_val,self.max_val)\n",
+        "      epsilon = np.clip(epsilon,-self.max_val,self.max_val)\n",
+        "      bins = np.arange(-self.max_val,self.max_val+self.inc,self.inc)\n",
+        "      numerator, *_ = stats.binned_statistic(zt, epsilon, statistic='sum',bins=bins)\n",
+        "      denominator, *_ = stats.binned_statistic(zt, epsilon, statistic='count',bins=bins)\n",
+        "      self.model = numerator / (denominator + 1)\n",
+        "\n",
+        "  def predict(self, zt):\n",
+        "      bin_index = np.floor((zt+self.max_val)/self.inc)\n",
+        "      bin_index = np.clip(bin_index,0, len(self.model)-1).astype('uint32')\n",
+        "      return self.model[bin_index]"
+      ],
+      "metadata": {
+        "id": "ZHViC0pL_yy5"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Sample data from distribution (this would usually be our collected training set)\n",
+        "n_sample = 100000\n",
+        "x_train = true_dist.sample(n_sample)\n",
+        "\n",
+        "# Define model parameters\n",
+        "T = 100\n",
+        "beta = 0.01511\n",
+        "\n",
+        "all_models = []\n",
+        "for t in range(0,T):\n",
+        "    clear_output(wait=True)\n",
+        "    display(\"Training timestep %d\"%(t))\n",
+        "    zt,epsilon= get_data_pairs(x_train,t,beta)\n",
+        "    all_models.append(NonParametricModel())\n",
+        "    # The model at index t maps data from z_{t+1} to epsilon\n",
+        "    all_models[t].train(zt,epsilon)"
+      ],
+      "metadata": {
+        "id": "CzVFybWoBygu"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now that we've learned the model, let's draw some samples from it.  We start at $z_{100}$ and use the model to predict $z_{99}$, then $z_{98}$ and so on until finally we get to $z_{1}$ and then $x$ (represented as $z_{0}$ here). We'll store all of the intermediate stages as well, so we can plot the trajectories.\n",
+        "\n",
+        "This is the same model we learned last time.  The whole point of this is that it is compatible with any forward process with the same diffusion kernel.\n",
+        "\n",
+        "One such model is the denoising diffusion implicit model, which has a sampling step:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "\\mathbf{z}_{t-1} = \\sqrt{\\alpha_{t-1}}\\left(\\frac{\\mathbf{z}_{t}-\\sqrt{1-\\alpha_{t}}\\mbox{g}_t[\\mathbf{z}_{t},\\boldsymbol\\phi]}{\\sqrt{\\alpha_{t}}}\\right) + \\sqrt{1-\\alpha_{t-1}-\\sigma^2}\\mbox{g}_t[\\mathbf{z}_{t},\\boldsymbol\\phi]+\\sigma\\epsilon\n",
+        "\\end{equation}\n",
+        "\n",
+        "(see equation 12 of the denoising [diffusion implicit models paper ](https://arxiv.org/pdf/2010.02502.pdf).\n"
+      ],
+      "metadata": {
+        "id": "ZPc9SEvtl14U"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def sample_ddim(model, T, sigma_t, n_samples):\n",
+        "    # Create the output array\n",
+        "    # Each row represents a time step, first row will be sampled data\n",
+        "    # Each column represents a different sample\n",
+        "    samples = np.zeros((T+1,n_samples))\n",
+        "    samples[T,:] = np.random.standard_normal(n_samples)\n",
+        "\n",
+        "    # For t=100...99..98... ...0\n",
+        "    for t in range(T,0,-1):\n",
+        "        clear_output(wait=True)\n",
+        "        display(\"Predicting z_{%d} from z_{%d}\"%(t-1,t))\n",
+        "\n",
+        "        alpha_t = np.power(1-beta,t+1)\n",
+        "        alpha_t_minus1 = np.power(1-beta,t)\n",
+        "\n",
+        "        # TODO -- implement the DDIM sampling step\n",
+        "        # Note the final noise term is already added in the \"if\" statement below\n",
+        "        # Replace this line:\n",
+        "        samples[t-1,:] = samples[t-1,:]\n",
+        "\n",
+        "        # If not the last time step\n",
+        "        if t>0:\n",
+        "            samples[t-1,:] = samples[t-1,:]+ np.random.standard_normal(n_samples) * sigma_t\n",
+        "    return samples"
+      ],
+      "metadata": {
+        "id": "A-ZMFOvACIOw"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's run the diffusion process for a whole bunch of samples"
+      ],
+      "metadata": {
+        "id": "ECAUfHNi9NVW"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Now we'll set the noise to a MUCH smaller level\n",
+        "sigma_t=0.001\n",
+        "n_samples = 100000\n",
+        "samples_low_noise = sample_ddim(all_models, T, sigma_t, n_samples)\n",
+        "\n",
+        "\n",
+        "# Plot the data\n",
+        "sampled_data = samples_low_noise[0,:]\n",
+        "bins = np.arange(-3,3.05,0.05)\n",
+        "\n",
+        "fig,ax = plt.subplots()\n",
+        "fig.set_size_inches(8,2.5)\n",
+        "ax.set_xlim([-3,3])\n",
+        "plt.hist(sampled_data, bins=bins, density =True)\n",
+        "ax.set_ylim(0, 0.8)\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "M-TY5w9Q8LYW"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's, plot the evolution of a few of the paths as in figure 18.7 (paths are from bottom to top now)."
+      ],
+      "metadata": {
+        "id": "jYrAW6tN-gJ4"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "fig, ax = plt.subplots()\n",
+        "t_vals = np.arange(0,101,1)\n",
+        "ax.plot(samples_low_noise[:,0],t_vals,'r-')\n",
+        "ax.plot(samples_low_noise[:,1],t_vals,'g-')\n",
+        "ax.plot(samples_low_noise[:,2],t_vals,'b-')\n",
+        "ax.plot(samples_low_noise[:,3],t_vals,'c-')\n",
+        "ax.plot(samples_low_noise[:,4],t_vals,'m-')\n",
+        "ax.set_xlim([-3,3])\n",
+        "ax.set_ylim([101, 0])\n",
+        "ax.set_xlabel('value')\n",
+        "ax.set_ylabel('z_{t}')\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "4XU6CDZC_kFo"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "The samples have a tendency to move from positions that are near the center at time 100 to positions that are high in the true probability distribution at time 0"
+      ],
+      "metadata": {
+        "id": "SGTYGGevAktz"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Let's now sample from the accelerated model, that requires fewer models.  Again, we don't need to learn anything new -- this is just the reverse process that corresponds to a different forward process that is compatible with the same diffusion kernel.\n",
+        "\n",
+        "There's nothing to do here except read the code.  It uses the same DDIM model as you just implemented in the previous step, but it jumps timesteps five at a time."
+      ],
+      "metadata": {
+        "id": "Z-LZp_fMXxRt"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def sample_accelerated(model, T, sigma_t, n_steps, n_samples):\n",
+        "    # Create the output array\n",
+        "    # Each row represents a sample (i.e. fewer than the time steps), first row will be sampled data\n",
+        "    # Each column represents a different sample\n",
+        "    samples = np.zeros((n_steps+1,n_samples))\n",
+        "    samples[n_steps,:] = np.random.standard_normal(n_samples)\n",
+        "\n",
+        "    # For each sampling step\n",
+        "    for c_step in range(n_steps,0,-1):\n",
+        "      # Find the corresponding time step and previous time step\n",
+        "      t= int(T * c_step/n_steps)\n",
+        "      tminus1 = int(T * (c_step-1)/n_steps)\n",
+        "      display(\"Predicting z_{%d} from z_{%d}\"%(tminus1,t))\n",
+        "\n",
+        "      alpha_t = np.power(1-beta,t+1)\n",
+        "      alpha_t_minus1 = np.power(1-beta,tminus1+1)\n",
+        "      epsilon_est = all_models[t-1].predict(samples[c_step,:])\n",
+        "\n",
+        "      samples[c_step-1,:]=np.sqrt(alpha_t_minus1)*(samples[c_step,:]-np.sqrt(1-alpha_t) * epsilon_est)/np.sqrt(alpha_t) \\\n",
+        "                                            + np.sqrt(1-alpha_t_minus1 - sigma_t*sigma_t) * epsilon_est\n",
+        "       # If not the last time step\n",
+        "      if t>0:\n",
+        "            samples[c_step-1,:] = samples[c_step-1,:]+ np.random.standard_normal(n_samples) * sigma_t\n",
+        "    return samples"
+      ],
+      "metadata": {
+        "id": "3Z0erjGbYj1u"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's draw a bunch of samples from the model"
+      ],
+      "metadata": {
+        "id": "D3Sm_WYrcuED"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "sigma_t=0.11\n",
+        "n_samples = 100000\n",
+        "n_steps = 20 # i.e. sample 5 times as fast as before -- should be a divisor of 100\n",
+        "samples_accelerated = sample_accelerated(all_models, T, sigma_t, n_steps, n_samples)\n",
+        "\n",
+        "\n",
+        "# Plot the data\n",
+        "sampled_data = samples_accelerated[0,:]\n",
+        "bins = np.arange(-3,3.05,0.05)\n",
+        "\n",
+        "fig,ax = plt.subplots()\n",
+        "fig.set_size_inches(8,2.5)\n",
+        "ax.set_xlim([-3,3])\n",
+        "plt.hist(sampled_data, bins=bins, density =True)\n",
+        "ax.set_ylim(0, 0.9)\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "UB45c7VMcGy-"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "fig, ax = plt.subplots()\n",
+        "step_increment = 100/ n_steps\n",
+        "t_vals = np.arange(0,101,5)\n",
+        "\n",
+        "for i in range(len(t_vals)-1):\n",
+        "  ax.plot( (samples_accelerated[i,0],samples_accelerated[i+1,0]), (t_vals[i], t_vals[i+1]),'r.-')\n",
+        "  ax.plot( (samples_accelerated[i,1],samples_accelerated[i+1,1]), (t_vals[i], t_vals[i+1]),'g.-')\n",
+        "  ax.plot( (samples_accelerated[i,2],samples_accelerated[i+1,2]), (t_vals[i], t_vals[i+1]),'b.-')\n",
+        "  ax.plot( (samples_accelerated[i,3],samples_accelerated[i+1,3]), (t_vals[i], t_vals[i+1]),'c.-')\n",
+        "  ax.plot( (samples_accelerated[i,4],samples_accelerated[i+1,4]), (t_vals[i], t_vals[i+1]),'m.-')\n",
+        "\n",
+        "ax.set_xlim([-3,3])\n",
+        "ax.set_ylim([101, 0])\n",
+        "ax.set_xlabel('value')\n",
+        "ax.set_ylabel('z_{t}')\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "Luv-6w84c_qO"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [],
+      "metadata": {
+        "id": "LSJi72f0kw_e"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

Notebooks/Chap19/19_1_Markov_Decision_Processes.ipynb

@@ -0,0 +1,736 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyPg3umHnqmIXX6jGe809Nxf",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap19/19_1_Markov_Decision_Processes.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 19.1: Markov Decision Processes**\n",
+        "\n",
+        "This notebook investigates Markov decision processes as described in section 19.1 of the book.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "from PIL import Image"
+      ],
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Get local copies of components of images\n",
+        "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Empty.png\n",
+        "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Hole.png\n",
+        "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Fish.png\n",
+        "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Penguin.png"
+      ],
+      "metadata": {
+        "id": "ZsvrUszPLyEG"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Ugly class that takes care of drawing pictures like in the book.\n",
+        "# You can totally ignore this code!\n",
+        "class DrawMDP:\n",
+        "  # Constructor initializes parameters\n",
+        "  def __init__(self, n_row, n_col):\n",
+        "    self.empty_image = np.asarray(Image.open('Empty.png'))\n",
+        "    self.hole_image = np.asarray(Image.open('Hole.png'))\n",
+        "    self.fish_image = np.asarray(Image.open('Fish.png'))\n",
+        "    self.penguin_image = np.asarray(Image.open('Penguin.png'))\n",
+        "    self.fig,self.ax = plt.subplots()\n",
+        "    self.n_row = n_row\n",
+        "    self.n_col = n_col\n",
+        "\n",
+        "    my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
+        "    my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
+        "    r = np.floor(my_colormap_vals_dec/(256*256))\n",
+        "    g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
+        "    b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
+        "    self.colormap = np.vstack((r,g,b)).transpose()/255.0\n",
+        "\n",
+        "\n",
+        "  def draw_text(self, text, row, col, position, color):\n",
+        "    if position == 'bc':\n",
+        "      self.ax.text( 83*col+41,83 * (row+1) -10, text, horizontalalignment=\"center\", color=color, fontweight='bold')\n",
+        "    if position == 'tl':\n",
+        "      self.ax.text( 83*col+5,83 * row +5, text, verticalalignment = 'top', horizontalalignment=\"left\", color=color, fontweight='bold')\n",
+        "\n",
+        "  # Draws a set of states\n",
+        "  def draw_path(self, path, color1, color2):\n",
+        "    for i in range(len(path)-1):\n",
+        "      row_start = np.floor(path[i]/self.n_col)\n",
+        "      row_end = np.floor(path[i+1]/self.n_col)\n",
+        "      col_start = path[i] - row_start * self.n_col\n",
+        "      col_end = path[i+1] - row_end * self.n_col\n",
+        "\n",
+        "      color_index = int(np.floor(255 * i/(len(path)-1.)))\n",
+        "      self.ax.plot([col_start * 83+41 + i, col_end * 83+41 + i ],[row_start * 83+41 +  i, row_end * 83+41 + i ], color=(self.colormap[color_index,0],self.colormap[color_index,1],self.colormap[color_index,2]))\n",
+        "\n",
+        "\n",
+        "  # Draw deterministic policy\n",
+        "  def draw_deterministic_policy(self,i, action):\n",
+        "      row = np.floor(i/self.n_col)\n",
+        "      col = i - row * self.n_col\n",
+        "      center_x = 83 * col + 41\n",
+        "      center_y = 83 * row + 41\n",
+        "      arrow_base_width = 10\n",
+        "      arrow_height = 15\n",
+        "      # Draw arrow pointing upward\n",
+        "      if action ==0:\n",
+        "          triangle_indices = np.array([[center_x, center_y-arrow_height/2],\n",
+        "                              [center_x - arrow_base_width/2, center_y+arrow_height/2],\n",
+        "                              [center_x + arrow_base_width/2, center_y+arrow_height/2]])\n",
+        "      # Draw arrow pointing right\n",
+        "      if action ==1:\n",
+        "          triangle_indices = np.array([[center_x + arrow_height/2, center_y],\n",
+        "                              [center_x - arrow_height/2, center_y-arrow_base_width/2],\n",
+        "                              [center_x - arrow_height/2, center_y+arrow_base_width/2]])\n",
+        "      # Draw arrow pointing downward\n",
+        "      if action ==2:\n",
+        "          triangle_indices = np.array([[center_x, center_y+arrow_height/2],\n",
+        "                              [center_x - arrow_base_width/2, center_y-arrow_height/2],\n",
+        "                              [center_x + arrow_base_width/2, center_y-arrow_height/2]])\n",
+        "      # Draw arrow pointing left\n",
+        "      if action ==3:\n",
+        "          triangle_indices = np.array([[center_x - arrow_height/2, center_y],\n",
+        "                              [center_x + arrow_height/2, center_y-arrow_base_width/2],\n",
+        "                              [center_x + arrow_height/2, center_y+arrow_base_width/2]])\n",
+        "      self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
+        "\n",
+        "  # Draw stochastic policy\n",
+        "  def draw_stochastic_policy(self,i, action_probs):\n",
+        "      row = np.floor(i/self.n_col)\n",
+        "      col = i - row * self.n_col\n",
+        "      offset = 20\n",
+        "      # Draw arrow pointing upward\n",
+        "      center_x = 83 * col + 41\n",
+        "      center_y = 83 * row + 41 - offset\n",
+        "      arrow_base_width = 15 * action_probs[0]\n",
+        "      arrow_height = 20 * action_probs[0]\n",
+        "      triangle_indices = np.array([[center_x, center_y-arrow_height/2],\n",
+        "                          [center_x - arrow_base_width/2, center_y+arrow_height/2],\n",
+        "                          [center_x + arrow_base_width/2, center_y+arrow_height/2]])\n",
+        "      self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
+        "\n",
+        "      # Draw arrow pointing right\n",
+        "      center_x = 83 * col + 41 + offset\n",
+        "      center_y = 83 * row + 41\n",
+        "      arrow_base_width = 15 * action_probs[1]\n",
+        "      arrow_height = 20 * action_probs[1]\n",
+        "      triangle_indices = np.array([[center_x + arrow_height/2, center_y],\n",
+        "                          [center_x - arrow_height/2, center_y-arrow_base_width/2],\n",
+        "                          [center_x - arrow_height/2, center_y+arrow_base_width/2]])\n",
+        "      self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
+        "\n",
+        "      # Draw arrow pointing downward\n",
+        "      center_x = 83 * col + 41\n",
+        "      center_y = 83 * row + 41 +offset\n",
+        "      arrow_base_width = 15 * action_probs[2]\n",
+        "      arrow_height = 20 * action_probs[2]\n",
+        "      triangle_indices = np.array([[center_x, center_y+arrow_height/2],\n",
+        "                          [center_x - arrow_base_width/2, center_y-arrow_height/2],\n",
+        "                          [center_x + arrow_base_width/2, center_y-arrow_height/2]])\n",
+        "      self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
+        "\n",
+        "      # Draw arrow pointing left\n",
+        "      center_x = 83 * col + 41 -offset\n",
+        "      center_y = 83 * row + 41\n",
+        "      arrow_base_width = 15 * action_probs[3]\n",
+        "      arrow_height = 20 * action_probs[3]\n",
+        "      triangle_indices = np.array([[center_x - arrow_height/2, center_y],\n",
+        "                          [center_x + arrow_height/2, center_y-arrow_base_width/2],\n",
+        "                          [center_x + arrow_height/2, center_y+arrow_base_width/2]])\n",
+        "      self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
+        "\n",
+        "\n",
+        "\n",
+        "\n",
+        "  def draw(self, layout, state, draw_state_index= False, rewards=None, policy=None, state_values=None, action_values=None,path1=None, path2 = None):\n",
+        "    # Construct the image\n",
+        "    image_out = np.zeros((self.n_row * 83, self.n_col * 83, 4),dtype='uint8')\n",
+        "    for c_row in range (self.n_row):\n",
+        "      for c_col in range(self.n_col):\n",
+        "        if layout[c_row * self.n_col + c_col]==0:\n",
+        "          image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.empty_image\n",
+        "        elif layout[c_row * self.n_col + c_col]==1:\n",
+        "          image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.hole_image\n",
+        "        else:\n",
+        "          image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.fish_image\n",
+        "        if state == c_row * self.n_col + c_col:\n",
+        "          image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.penguin_image\n",
+        "\n",
+        "    # Draw the image\n",
+        "    plt.imshow(image_out)\n",
+        "    self.ax.get_xaxis().set_visible(False)\n",
+        "    self.ax.get_yaxis().set_visible(False)\n",
+        "    self.ax.spines['top'].set_visible(False)\n",
+        "    self.ax.spines['right'].set_visible(False)\n",
+        "    self.ax.spines['bottom'].set_visible(False)\n",
+        "    self.ax.spines['left'].set_visible(False)\n",
+        "\n",
+        "    if draw_state_index:\n",
+        "      for c_cell in range(layout.size):\n",
+        "          self.draw_text(\"%d\"%(c_cell), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'tl','k')\n",
+        "\n",
+        "    # Draw the policy as triangles\n",
+        "    if policy is not None:\n",
+        "        # If the policy is deterministic\n",
+        "        if len(policy) == len(layout):\n",
+        "          for i in range(len(layout)):\n",
+        "            self.draw_deterministic_policy(i, policy[i])\n",
+        "        # Else it is stochastic\n",
+        "        else:\n",
+        "          for i in range(len(layout)):\n",
+        "            self.draw_stochastic_policy(i,policy[:,i])\n",
+        "\n",
+        "\n",
+        "    if path1 is not None:\n",
+        "      # self.draw_path(path1, np.array([0.81, 0.51, 0.38]), np.array([1.0, 0.2, 0.5]))\n",
+        "      self.draw_path(path1, np.array([1.0, 0.0, 0.0]), np.array([0.0, 1.0, 1.0]))\n",
+        "\n",
+        "\n",
+        "    plt.show()"
+      ],
+      "metadata": {
+        "id": "Gq1HfJsHN3SB"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Let's draw the initial situation with the penguin in top right\n",
+        "n_rows = 4; n_cols = 4\n",
+        "layout = np.zeros(n_rows * n_cols)\n",
+        "initial_state = 0\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, state = initial_state, draw_state_index = True)"
+      ],
+      "metadata": {
+        "id": "eBQ7lTpJQBSe"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Note that the states are indexed from 0 rather than 1 as in the book to make\n",
+        "the code neater."
+      ],
+      "metadata": {
+        "id": "P7P40UyMunKb"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Define the state probabilities\n",
+        "transition_probabilities = np.array( \\\n",
+        "[[0.00 , 0.33, 0.00, 0.00,  0.33, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.50 , 0.00, 0.33, 0.00,  0.00, 0.25, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.33, 0.00, 0.50,  0.00, 0.00, 0.25, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.33, 0.00,  0.00, 0.00, 0.00, 0.33,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.50 , 0.00, 0.00, 0.00,  0.00, 0.25, 0.00, 0.00,   0.33, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.34, 0.00, 0.00,  0.33, 0.00, 0.25, 0.00,   0.00, 0.25, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.34, 0.00,  0.00, 0.25, 0.00, 0.33,   0.00, 0.00, 0.25, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.50,  0.00, 0.00, 0.25, 0.00,   0.00, 0.00, 0.00, 0.33,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.34, 0.00, 0.00, 0.00,   0.00, 0.25, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.25, 0.00, 0.00,   0.33, 0.00, 0.25, 0.00,   0.00, 0.33, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.25, 0.00,   0.00, 0.25, 0.00, 0.33,   0.00, 0.00, 0.33, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.34,   0.00, 0.00, 0.25, 0.00,   0.00, 0.00, 0.00, 0.50 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.34, 0.00, 0.00, 0.00,   0.00, 0.33, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.25, 0.00, 0.00,   0.50, 0.00, 0.33, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.25, 0.00,   0.00, 0.34, 0.00, 0.50 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.34,   0.00, 0.00, 0.34, 0.00 ],\n",
+        "])\n",
+        "initial_state = 0"
+      ],
+      "metadata": {
+        "id": "wgFcIi4YQJWI"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Define a step from the Markov process"
+      ],
+      "metadata": {
+        "id": "axllRDDuDDLS"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def markov_process_step(state, transition_probabilities):\n",
+        "  # TODO -- update the state according to the appropriate transition probabilities\n",
+        "  # One way to do this is to use np.random.choice\n",
+        "  # Replace this line:\n",
+        "  new_state = 0\n",
+        "\n",
+        "\n",
+        "  return new_state"
+      ],
+      "metadata": {
+        "id": "FrSZrS67sdbN"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Run the Markov process for 10 steps and visualise the results"
+      ],
+      "metadata": {
+        "id": "uTj7rN6LDFXd"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "np.random.seed(0)\n",
+        "T = 10\n",
+        "states = np.zeros(T, dtype='uint8')\n",
+        "states[0] = 0\n",
+        "for t in range(T-1):\n",
+        "  states[t+1] = markov_process_step(states[t], transition_probabilities)\n",
+        "\n",
+        "\n",
+        "\n",
+        "print(\"Your States:\", states)\n",
+        "print(\"True States: [ 0  4  8  9 10  9 10  9 13 14]\")\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, state = states[0], path1=states, draw_state_index = True)"
+      ],
+      "metadata": {
+        "id": "lRIdjagCwP62"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Define a Markov one step of a reward process."
+      ],
+      "metadata": {
+        "id": "QLyjyBjjDMin"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def markov_reward_process_step(state, transition_probabilities, reward_structure):\n",
+        "\n",
+        "    # TODO -- write this function\n",
+        "    # Update the state.  Return a reward of +1 if the Penguin lands on the fish\n",
+        "    # or zero otherwise.\n",
+        "    # Replace this line\n",
+        "    new_state = 0; reward = 0\n",
+        "\n",
+        "\n",
+        "    return new_state, reward"
+      ],
+      "metadata": {
+        "id": "YPHSJRKx-pgO"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Run the Markov reward process for 10 steps and visualise the results"
+      ],
+      "metadata": {
+        "id": "AIz8QEiRFoCm"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set up the reward structure so it matches figure 19.2\n",
+        "reward_structure = np.zeros((16,1))\n",
+        "reward_structure[3] = 1; reward_structure[8] = 1; reward_structure[10] = 1\n",
+        "\n",
+        "# Initialize random numbers\n",
+        "np.random.seed(0)\n",
+        "T = 10\n",
+        "# Set up the states, so the fish are in the same positions as figure 19.2\n",
+        "states = np.zeros(T, dtype='uint8')\n",
+        "rewards = np.zeros(T, dtype='uint8')\n",
+        "\n",
+        "states[0] = 0\n",
+        "for t in range(T-1):\n",
+        "  states[t+1],rewards[t+1] = markov_reward_process_step(states[t], transition_probabilities, reward_structure)\n",
+        "\n",
+        "print(\"Your States:\", states)\n",
+        "print(\"Your Rewards:\", rewards)\n",
+        "print(\"True Rewards: [0 0 1 0 1 0 1 0 0 0]\")\n",
+        "\n",
+        "\n",
+        "# Draw the figure\n",
+        "layout = np.zeros(n_rows * n_cols)\n",
+        "layout[3] = 2; layout[8] = 2 ; layout[10] = 2\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, state = states[0], path1=states, draw_state_index = True)"
+      ],
+      "metadata": {
+        "id": "0p1gCpGoFn4M"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's calculate the return -- the sum of discounted future rewards"
+      ],
+      "metadata": {
+        "id": "lyz47NWrITfj"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def calculate_return(rewards, gamma):\n",
+        "  # TODO -- you write this function\n",
+        "  # It should compute one return for the start of the sequence (i.e. G_1)\n",
+        "  # Replace this line\n",
+        "  return_val = 0.0\n",
+        "\n",
+        "\n",
+        "  return return_val"
+      ],
+      "metadata": {
+        "id": "4fEuBRPnFm_N"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "gamma = 0.9\n",
+        "for t in range(len(states)):\n",
+        "  print(\"Return at time %d = %3.3f\"%(t, calculate_return(rewards[t:],gamma)))\n",
+        "\n",
+        "# Reality check!\n",
+        "print(\"True return at time 0: 1.998\")"
+      ],
+      "metadata": {
+        "id": "o19lQgM3JrOz"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's define the state transition function $Pr(s_{t+1}|s_{t},a)$ in full where $a$ is the actions.  Here $a=0$ means try to go upward, $a=1$, right, $a=2$ down and $a=3$ right.  However, the ice is slippery, so we don't always go the direction we want to.\n",
+        "\n",
+        "Note that as for the states, we've indexed the actions from zero (unlike in the book, so they map to the indices of arrays better)"
+      ],
+      "metadata": {
+        "id": "Fhc6DzZNOjiC"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "transition_probabilities_given_action1 = np.array(\\\n",
+        "[[0.00 , 0.33, 0.00, 0.00,  0.50, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.50 , 0.00, 0.33, 0.00,  0.00, 0.50, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.33, 0.00, 0.50,  0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.33, 0.00,  0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.50 , 0.00, 0.00, 0.00,  0.00, 0.17, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.34, 0.00, 0.00,  0.25, 0.00, 0.17, 0.00,   0.00, 0.50, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.34, 0.00,  0.00, 0.17, 0.00, 0.25,   0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.50,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.17, 0.00, 0.00,   0.75, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.16, 0.00, 0.00,   0.25, 0.00, 0.17, 0.00,   0.00, 0.50, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.16, 0.00,   0.00, 0.17, 0.00, 0.25,   0.00, 0.00, 0.50, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.75 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.16, 0.00, 0.00,   0.25, 0.00, 0.25, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.16, 0.00,   0.00, 0.25, 0.00, 0.25 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.25, 0.00 ],\n",
+        "])\n",
+        "\n",
+        "transition_probabilities_given_action2 = np.array(\\\n",
+        "[[0.00 , 0.25, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.75 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.50, 0.00, 0.50,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.50, 0.00,  0.00, 0.00, 0.00, 0.33,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.25 , 0.00, 0.00, 0.00,  0.00, 0.17, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.25, 0.00, 0.00,  0.50, 0.00, 0.17, 0.00,   0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.25, 0.00,  0.00, 0.50, 0.00, 0.33,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.50,  0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.33,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.17, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.16, 0.00, 0.00,   0.50, 0.00, 0.17, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.16, 0.00,   0.00, 0.50, 0.00, 0.33,   0.00, 0.00, 0.25, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.34,   0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.50 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.16, 0.00, 0.00,   0.75, 0.00, 0.25, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.16, 0.00,   0.00, 0.50, 0.00, 0.50 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.34,   0.00, 0.00, 0.50, 0.00 ],\n",
+        "])\n",
+        "\n",
+        "transition_probabilities_given_action3 = np.array(\\\n",
+        "[[0.00 , 0.25, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.25 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.25, 0.00, 0.25,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.25, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.75 , 0.00, 0.00, 0.00,  0.00, 0.17, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.50, 0.00, 0.00,  0.25, 0.00, 0.17, 0.00,   0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.50, 0.00,  0.00, 0.16, 0.00, 0.25,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.75,  0.00, 0.00, 0.16, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.50, 0.00, 0.00, 0.00,   0.00, 0.17, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.50, 0.00, 0.00,   0.25, 0.00, 0.17, 0.00,   0.00, 0.33, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.50, 0.00,   0.00, 0.16, 0.00, 0.25,   0.00, 0.00, 0.33, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.16, 0.00,   0.00, 0.00, 0.00, 0.50 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00,   0.00, 0.33, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.50, 0.00, 0.00,   0.50, 0.00, 0.33, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.50, 0.00,   0.00, 0.34, 0.00, 0.50 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.34, 0.00 ],\n",
+        "])\n",
+        "\n",
+        "transition_probabilities_given_action4 = np.array(\\\n",
+        "[[0.00 , 0.25, 0.00, 0.00,  0.33, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.50 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.50, 0.00, 0.75,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.50, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.50 , 0.00, 0.00, 0.00,  0.00, 0.50, 0.00, 0.00,   0.33, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.25, 0.00, 0.00,  0.33, 0.00, 0.50, 0.00,   0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.50,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.25,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.34, 0.00, 0.00, 0.00,   0.00, 0.50, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.16, 0.00, 0.00,   0.33, 0.00, 0.50, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.16, 0.00,   0.00, 0.17, 0.00, 0.50,   0.00, 0.00, 0.25, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.25 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.34, 0.00, 0.00, 0.00,   0.00, 0.50, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.16, 0.00, 0.00,   0.50, 0.00, 0.50, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.16, 0.00,   0.00, 0.25, 0.00, 0.75 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.25, 0.00 ],\n",
+        "])\n",
+        "\n",
+        "# Store all of these in a three dimension array\n",
+        "# Pr(s_{t+1}=2|s_{t}=1, a_{t}=3] is stored at position [2,1,3]\n",
+        "transition_probabilities_given_action = np.concatenate((np.expand_dims(transition_probabilities_given_action1,2),\n",
+        "                                                        np.expand_dims(transition_probabilities_given_action2,2),\n",
+        "                                                        np.expand_dims(transition_probabilities_given_action3,2),\n",
+        "                                                        np.expand_dims(transition_probabilities_given_action4,2)),axis=2)"
+      ],
+      "metadata": {
+        "id": "l7rT78BbOgTi"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Now we need a policy.  Let's start with the deterministic policy in figure 19.5a:\n",
+        "policy = [2,2,1,1, 2,1,1,1, 1,1,0,2, 1,0,1,1]\n",
+        "\n",
+        "# Let's draw the policy first\n",
+        "layout = np.zeros(n_rows * n_cols)\n",
+        "layout[15] = 2\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, state = states[0], policy = policy, draw_state_index = True)"
+      ],
+      "metadata": {
+        "id": "8jWhDlkaKj7Q"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def markov_decision_process_step_deterministic(state, transition_probabilities_given_action, reward_structure, policy):\n",
+        "  # TODO -- complete this function.\n",
+        "  # For each state, theres is a corresponding action.\n",
+        "  # Draw the next state based on the current state and that action\n",
+        "  # and calculate the reward\n",
+        "  # Replace this line:\n",
+        "  new_state = 0; reward = 0;\n",
+        "\n",
+        "  return new_state, reward\n"
+      ],
+      "metadata": {
+        "id": "dueNbS2SUVUK"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set up the reward structure so it matches figure 19.2\n",
+        "reward_structure = np.zeros((16,1))\n",
+        "reward_structure[15] = 1\n",
+        "\n",
+        "# Initialize random number seed\n",
+        "np.random.seed(3)\n",
+        "T = 10\n",
+        "# Set up the states, so the fish are in the same positions as figure 19.5\n",
+        "states = np.zeros(T, dtype='uint8')\n",
+        "rewards = np.zeros(T, dtype='uint8')\n",
+        "\n",
+        "states[0] = 0\n",
+        "for t in range(T-1):\n",
+        "  states[t+1],rewards[t+1] = markov_decision_process_step_deterministic(states[t], transition_probabilities_given_action, reward_structure, policy)\n",
+        "\n",
+        "print(\"Your States:\", states)\n",
+        "print(\"True States: [ 0  4  8  9 13 14 15 11  7  3]\")\n",
+        "print(\"Your Rewards:\", rewards)\n",
+        "print(\"True Rewards: [0 0 0 0 0 0 1 0 0 0]\")\n",
+        "\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, state = states[0], path1=states, policy = policy, draw_state_index = True)"
+      ],
+      "metadata": {
+        "id": "4Du5aUfd2Lci"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "You can see that the Penguin usually follows the policy, (heads in the direction of the cyan arrows (when it can).  But sometimes, the penguin \"slips\" to a different neighboring state\n",
+        "\n",
+        "Now let's investigate a stochastic policy"
+      ],
+      "metadata": {
+        "id": "bLEd8xug33b-"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "np.random.seed(0)\n",
+        "# Let's now choose a random policy.  We'll generate a set of random numbers and pass\n",
+        "# them through a softmax function\n",
+        "stochastic_policy = np.random.normal(size=(4,n_rows*n_cols))\n",
+        "stochastic_policy = np.exp(stochastic_policy) / (np.ones((4,1))@ np.expand_dims(np.sum(np.exp(stochastic_policy), axis=0),0))\n",
+        "np.set_printoptions(precision=2)\n",
+        "print(stochastic_policy)\n",
+        "\n",
+        "# Let's draw the policy first\n",
+        "layout = np.zeros(n_rows * n_cols)\n",
+        "layout[15] = 2\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, state = states[0], path1=states, policy = stochastic_policy, draw_state_index = True)"
+      ],
+      "metadata": {
+        "id": "o7T0b3tyilDc"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def markov_decision_process_step_stochastic(state, transition_probabilities_given_action, reward_structure, stochastic_policy):\n",
+        "  # TODO -- complete this function.\n",
+        "  # For each state, theres is a corresponding distribution over actions\n",
+        "  # Draw a sample from that distribution to get the action\n",
+        "  # Draw the next state based on the current state and that action\n",
+        "  # and calculate the reward\n",
+        "  # Replace this line:\n",
+        "  new_state = 0; reward = 0;action = 0\n",
+        "\n",
+        "\n",
+        "\n",
+        "  return new_state, reward, action"
+      ],
+      "metadata": {
+        "id": "T68mTZSe6A3w"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set up the reward structure so it matches figure 19.2\n",
+        "reward_structure = np.zeros((16,1))\n",
+        "reward_structure[15] = 1\n",
+        "\n",
+        "# Initialize random number seed\n",
+        "np.random.seed(0)\n",
+        "T = 10\n",
+        "# Set up the states, so the fish are in the same positions as figure 19.5\n",
+        "states = np.zeros(T, dtype='uint8')\n",
+        "rewards = np.zeros(T, dtype='uint8')\n",
+        "actions = np.zeros(T-1, dtype='uint8')\n",
+        "\n",
+        "states[0] = 0\n",
+        "for t in range(T-1):\n",
+        "  states[t+1],rewards[t+1],actions[t] = markov_decision_process_step_stochastic(states[t], transition_probabilities_given_action, reward_structure, stochastic_policy)\n",
+        "\n",
+        "print(\"Actions\", actions)\n",
+        "print(\"Your States:\", states)\n",
+        "print(\"Your Rewards:\", rewards)\n",
+        "\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, state = states[0], path1=states, policy = stochastic_policy, draw_state_index = True)"
+      ],
+      "metadata": {
+        "id": "hMRVYX2HtqMg"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

Notebooks/Chap19/19_4_Temporal_Difference_Methods.ipynb

@@ -0,0 +1,521 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyNEAhORON7DFN1dZMhDK/PO",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap19/19_4_Temporal_Difference_Methods.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 19.4: Temporal difference methods**\n",
+        "\n",
+        "This notebook investigates temporal difference methods for  tabular reinforcement learning as described in section 19.3.3 of the book\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "from PIL import Image"
+      ],
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Get local copies of components of images\n",
+        "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Empty.png\n",
+        "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Hole.png\n",
+        "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Fish.png\n",
+        "!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Penguin.png"
+      ],
+      "metadata": {
+        "id": "ZsvrUszPLyEG"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Ugly class that takes care of drawing pictures like in the book.\n",
+        "# You can totally ignore this code!\n",
+        "class DrawMDP:\n",
+        "  # Constructor initializes parameters\n",
+        "  def __init__(self, n_row, n_col):\n",
+        "    self.empty_image = np.asarray(Image.open('Empty.png'))\n",
+        "    self.hole_image = np.asarray(Image.open('Hole.png'))\n",
+        "    self.fish_image = np.asarray(Image.open('Fish.png'))\n",
+        "    self.penguin_image = np.asarray(Image.open('Penguin.png'))\n",
+        "    self.fig,self.ax = plt.subplots()\n",
+        "    self.n_row = n_row\n",
+        "    self.n_col = n_col\n",
+        "\n",
+        "    my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
+        "    my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
+        "    r = np.floor(my_colormap_vals_dec/(256*256))\n",
+        "    g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
+        "    b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
+        "    self.colormap = np.vstack((r,g,b)).transpose()/255.0\n",
+        "\n",
+        "\n",
+        "  def draw_text(self, text, row, col, position, color):\n",
+        "    if position == 'bc':\n",
+        "      self.ax.text( 83*col+41,83 * (row+1) -5, text, horizontalalignment=\"center\", color=color, fontweight='bold')\n",
+        "    if position == 'tc':\n",
+        "      self.ax.text( 83*col+41,83 * (row) +10, text, horizontalalignment=\"center\", color=color, fontweight='bold')\n",
+        "    if position == 'lc':\n",
+        "      self.ax.text( 83*col+2,83 * (row) +41, text, verticalalignment=\"center\", color=color, fontweight='bold', rotation=90)\n",
+        "    if position == 'rc':\n",
+        "      self.ax.text( 83*(col+1)-5,83 * (row) +41, text, horizontalalignment=\"right\", verticalalignment=\"center\", color=color, fontweight='bold', rotation=-90)\n",
+        "    if position == 'tl':\n",
+        "      self.ax.text( 83*col+5,83 * row +5, text, verticalalignment = 'top', horizontalalignment=\"left\", color=color, fontweight='bold')\n",
+        "    if position == 'tr':\n",
+        "      self.ax.text( 83*(col+1)-5, 83 * row +5, text, verticalalignment = 'top', horizontalalignment=\"right\", color=color, fontweight='bold')\n",
+        "\n",
+        "  # Draws a set of states\n",
+        "  def draw_path(self, path, color1, color2):\n",
+        "    for i in range(len(path)-1):\n",
+        "      row_start = np.floor(path[i]/self.n_col)\n",
+        "      row_end = np.floor(path[i+1]/self.n_col)\n",
+        "      col_start = path[i] - row_start * self.n_col\n",
+        "      col_end = path[i+1] - row_end * self.n_col\n",
+        "\n",
+        "      color_index = int(np.floor(255 * i/(len(path)-1.)))\n",
+        "      self.ax.plot([col_start * 83+41 + i, col_end * 83+41 + i ],[row_start * 83+41 +  i, row_end * 83+41 + i ], color=(self.colormap[color_index,0],self.colormap[color_index,1],self.colormap[color_index,2]))\n",
+        "\n",
+        "\n",
+        "  # Draw deterministic policy\n",
+        "  def draw_deterministic_policy(self,i, action):\n",
+        "      row = np.floor(i/self.n_col)\n",
+        "      col = i - row * self.n_col\n",
+        "      center_x = 83 * col + 41\n",
+        "      center_y = 83 * row + 41\n",
+        "      arrow_base_width = 10\n",
+        "      arrow_height = 15\n",
+        "      # Draw arrow pointing upward\n",
+        "      if action ==0:\n",
+        "          triangle_indices = np.array([[center_x, center_y-arrow_height/2],\n",
+        "                              [center_x - arrow_base_width/2, center_y+arrow_height/2],\n",
+        "                              [center_x + arrow_base_width/2, center_y+arrow_height/2]])\n",
+        "      # Draw arrow pointing right\n",
+        "      if action ==1:\n",
+        "          triangle_indices = np.array([[center_x + arrow_height/2, center_y],\n",
+        "                              [center_x - arrow_height/2, center_y-arrow_base_width/2],\n",
+        "                              [center_x - arrow_height/2, center_y+arrow_base_width/2]])\n",
+        "      # Draw arrow pointing downward\n",
+        "      if action ==2:\n",
+        "          triangle_indices = np.array([[center_x, center_y+arrow_height/2],\n",
+        "                              [center_x - arrow_base_width/2, center_y-arrow_height/2],\n",
+        "                              [center_x + arrow_base_width/2, center_y-arrow_height/2]])\n",
+        "      # Draw arrow pointing left\n",
+        "      if action ==3:\n",
+        "          triangle_indices = np.array([[center_x - arrow_height/2, center_y],\n",
+        "                              [center_x + arrow_height/2, center_y-arrow_base_width/2],\n",
+        "                              [center_x + arrow_height/2, center_y+arrow_base_width/2]])\n",
+        "      self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
+        "\n",
+        "  # Draw stochastic policy\n",
+        "  def draw_stochastic_policy(self,i, action_probs):\n",
+        "      row = np.floor(i/self.n_col)\n",
+        "      col = i - row * self.n_col\n",
+        "      offset = 20\n",
+        "      # Draw arrow pointing upward\n",
+        "      center_x = 83 * col + 41\n",
+        "      center_y = 83 * row + 41 - offset\n",
+        "      arrow_base_width = 15 * action_probs[0]\n",
+        "      arrow_height = 20 * action_probs[0]\n",
+        "      triangle_indices = np.array([[center_x, center_y-arrow_height/2],\n",
+        "                          [center_x - arrow_base_width/2, center_y+arrow_height/2],\n",
+        "                          [center_x + arrow_base_width/2, center_y+arrow_height/2]])\n",
+        "      self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
+        "\n",
+        "      # Draw arrow pointing right\n",
+        "      center_x = 83 * col + 41 + offset\n",
+        "      center_y = 83 * row + 41\n",
+        "      arrow_base_width = 15 * action_probs[1]\n",
+        "      arrow_height = 20 * action_probs[1]\n",
+        "      triangle_indices = np.array([[center_x + arrow_height/2, center_y],\n",
+        "                          [center_x - arrow_height/2, center_y-arrow_base_width/2],\n",
+        "                          [center_x - arrow_height/2, center_y+arrow_base_width/2]])\n",
+        "      self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
+        "\n",
+        "      # Draw arrow pointing downward\n",
+        "      center_x = 83 * col + 41\n",
+        "      center_y = 83 * row + 41 +offset\n",
+        "      arrow_base_width = 15 * action_probs[2]\n",
+        "      arrow_height = 20 * action_probs[2]\n",
+        "      triangle_indices = np.array([[center_x, center_y+arrow_height/2],\n",
+        "                          [center_x - arrow_base_width/2, center_y-arrow_height/2],\n",
+        "                          [center_x + arrow_base_width/2, center_y-arrow_height/2]])\n",
+        "      self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
+        "\n",
+        "      # Draw arrow pointing left\n",
+        "      center_x = 83 * col + 41 -offset\n",
+        "      center_y = 83 * row + 41\n",
+        "      arrow_base_width = 15 * action_probs[3]\n",
+        "      arrow_height = 20 * action_probs[3]\n",
+        "      triangle_indices = np.array([[center_x - arrow_height/2, center_y],\n",
+        "                          [center_x + arrow_height/2, center_y-arrow_base_width/2],\n",
+        "                          [center_x + arrow_height/2, center_y+arrow_base_width/2]])\n",
+        "      self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
+        "\n",
+        "\n",
+        "  def draw(self, layout, state=None, draw_state_index= False, rewards=None, policy=None, state_values=None, state_action_values=None,path1=None, path2 = None):\n",
+        "    # Construct the image\n",
+        "    image_out = np.zeros((self.n_row * 83, self.n_col * 83, 4),dtype='uint8')\n",
+        "    for c_row in range (self.n_row):\n",
+        "      for c_col in range(self.n_col):\n",
+        "        if layout[c_row * self.n_col + c_col]==0:\n",
+        "          image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.empty_image\n",
+        "        elif layout[c_row * self.n_col + c_col]==1:\n",
+        "          image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.hole_image\n",
+        "        else:\n",
+        "          image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.fish_image\n",
+        "        if state is not None and state == c_row * self.n_col + c_col:\n",
+        "          image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.penguin_image\n",
+        "\n",
+        "    # Draw the image\n",
+        "    plt.imshow(image_out)\n",
+        "    self.ax.get_xaxis().set_visible(False)\n",
+        "    self.ax.get_yaxis().set_visible(False)\n",
+        "    self.ax.spines['top'].set_visible(False)\n",
+        "    self.ax.spines['right'].set_visible(False)\n",
+        "    self.ax.spines['bottom'].set_visible(False)\n",
+        "    self.ax.spines['left'].set_visible(False)\n",
+        "\n",
+        "    if draw_state_index:\n",
+        "      for c_cell in range(layout.size):\n",
+        "          self.draw_text(\"%d\"%(c_cell), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'tl','k')\n",
+        "\n",
+        "    # Draw the policy as triangles\n",
+        "    if policy is not None:\n",
+        "        # If the policy is deterministic\n",
+        "        if len(policy) == len(layout):\n",
+        "          for i in range(len(layout)):\n",
+        "            self.draw_deterministic_policy(i, policy[i])\n",
+        "        # Else it is stochastic\n",
+        "        else:\n",
+        "          for i in range(len(layout)):\n",
+        "            self.draw_stochastic_policy(i,policy[:,i])\n",
+        "\n",
+        "\n",
+        "    if path1 is not None:\n",
+        "      self.draw_path(path1, np.array([1.0, 0.0, 0.0]), np.array([0.0, 1.0, 1.0]))\n",
+        "\n",
+        "    if rewards is not None:\n",
+        "        for c_cell in range(layout.size):\n",
+        "          self.draw_text(\"%d\"%(rewards[c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'tr','r')\n",
+        "\n",
+        "    if state_values is not None:\n",
+        "        for c_cell in range(layout.size):\n",
+        "          self.draw_text(\"%2.2f\"%(state_values[c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'bc','black')\n",
+        "\n",
+        "    if state_action_values is not None:\n",
+        "        for c_cell in range(layout.size):\n",
+        "          self.draw_text(\"%2.2f\"%(state_action_values[0, c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'tc','black')\n",
+        "          self.draw_text(\"%2.2f\"%(state_action_values[1, c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'rc','black')\n",
+        "          self.draw_text(\"%2.2f\"%(state_action_values[2, c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'bc','black')\n",
+        "          self.draw_text(\"%2.2f\"%(state_action_values[3, c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'lc','black')\n",
+        "\n",
+        "    plt.show()"
+      ],
+      "metadata": {
+        "id": "Gq1HfJsHN3SB"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# We're going to work on the problem depicted in figure 19.10a\n",
+        "n_rows = 4; n_cols = 4\n",
+        "layout = np.zeros(n_rows * n_cols)\n",
+        "reward_structure = np.zeros(n_rows * n_cols)\n",
+        "layout[9] = 1 ; reward_structure[9] = -2\n",
+        "layout[10] = 1; reward_structure[10] = -2\n",
+        "layout[14] = 1; reward_structure[14] = -2\n",
+        "layout[15] = 2; reward_structure[15] = 3\n",
+        "initial_state = 0\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, state = initial_state, rewards=reward_structure, draw_state_index = True)"
+      ],
+      "metadata": {
+        "id": "eBQ7lTpJQBSe"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "For clarity, the black numbers are the state number and the red numbers are the reward for being in that state.  Note that the states are indexed from 0 rather than 1 as in the book to make the code neater."
+      ],
+      "metadata": {
+        "id": "6Vku6v_se2IG"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's define the state transition function $Pr(s_{t+1}|s_{t},a)$ in full where $a$ is the actions.  Here $a=0$ means try to go upward, $a=1$, right, $a=2$ down and $a=3$ right.  However, the ice is slippery, so we don't always go the direction we want to.\n",
+        "\n",
+        "Note that as for the states, we've indexed the actions from zero (unlike in the book) so they map to the indices of arrays better"
+      ],
+      "metadata": {
+        "id": "Fhc6DzZNOjiC"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "transition_probabilities_given_action0 = np.array(\\\n",
+        "[[0.00 , 0.33, 0.00, 0.00,  0.50, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.50 , 0.00, 0.33, 0.00,  0.00, 0.50, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.33, 0.00, 0.50,  0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.33, 0.00,  0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.50 , 0.00, 0.00, 0.00,  0.00, 0.17, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.34, 0.00, 0.00,  0.25, 0.00, 0.17, 0.00,   0.00, 0.50, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.34, 0.00,  0.00, 0.17, 0.00, 0.25,   0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.50,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.17, 0.00, 0.00,   0.75, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.16, 0.00, 0.00,   0.25, 0.00, 0.17, 0.00,   0.00, 0.50, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.16, 0.00,   0.00, 0.17, 0.00, 0.25,   0.00, 0.00, 0.50, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.75 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.16, 0.00, 0.00,   0.25, 0.00, 0.25, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.16, 0.00,   0.00, 0.25, 0.00, 0.25 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.25, 0.00 ],\n",
+        "])\n",
+        "\n",
+        "transition_probabilities_given_action1 = np.array(\\\n",
+        "[[0.00 , 0.25, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.75 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.50, 0.00, 0.50,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.50, 0.00,  0.00, 0.00, 0.00, 0.33,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.25 , 0.00, 0.00, 0.00,  0.00, 0.17, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.25, 0.00, 0.00,  0.50, 0.00, 0.17, 0.00,   0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.25, 0.00,  0.00, 0.50, 0.00, 0.33,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.50,  0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.33,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.17, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.16, 0.00, 0.00,   0.50, 0.00, 0.17, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.16, 0.00,   0.00, 0.50, 0.00, 0.33,   0.00, 0.00, 0.25, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.34,   0.00, 0.00, 0.50, 0.00,   0.00, 0.00, 0.00, 0.50 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.16, 0.00, 0.00,   0.75, 0.00, 0.25, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.16, 0.00,   0.00, 0.50, 0.00, 0.50 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.34,   0.00, 0.00, 0.50, 0.00 ],\n",
+        "])\n",
+        "\n",
+        "transition_probabilities_given_action2 = np.array(\\\n",
+        "[[0.00 , 0.25, 0.00, 0.00,  0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.25 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.25, 0.00, 0.25,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.25, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.75 , 0.00, 0.00, 0.00,  0.00, 0.17, 0.00, 0.00,   0.25, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.50, 0.00, 0.00,  0.25, 0.00, 0.17, 0.00,   0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.50, 0.00,  0.00, 0.16, 0.00, 0.25,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.75,  0.00, 0.00, 0.16, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.50, 0.00, 0.00, 0.00,   0.00, 0.17, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.50, 0.00, 0.00,   0.25, 0.00, 0.17, 0.00,   0.00, 0.33, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.50, 0.00,   0.00, 0.16, 0.00, 0.25,   0.00, 0.00, 0.33, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.16, 0.00,   0.00, 0.00, 0.00, 0.50 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00,   0.00, 0.33, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.50, 0.00, 0.00,   0.50, 0.00, 0.33, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.50, 0.00,   0.00, 0.34, 0.00, 0.50 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.50,   0.00, 0.00, 0.34, 0.00 ],\n",
+        "])\n",
+        "\n",
+        "transition_probabilities_given_action3 = np.array(\\\n",
+        "[[0.00 , 0.25, 0.00, 0.00,  0.33, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.50 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.50, 0.00, 0.75,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.50, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.50 , 0.00, 0.00, 0.00,  0.00, 0.50, 0.00, 0.00,   0.33, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.25, 0.00, 0.00,  0.33, 0.00, 0.50, 0.00,   0.00, 0.17, 0.00, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.25, 0.00,  0.00, 0.17, 0.00, 0.50,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.25,  0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.34, 0.00, 0.00, 0.00,   0.00, 0.50, 0.00, 0.00,   0.50, 0.00, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.16, 0.00, 0.00,   0.33, 0.00, 0.50, 0.00,   0.00, 0.25, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.16, 0.00,   0.00, 0.17, 0.00, 0.50,   0.00, 0.00, 0.25, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.17, 0.00,   0.00, 0.00, 0.00, 0.25 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.34, 0.00, 0.00, 0.00,   0.00, 0.50, 0.00, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.16, 0.00, 0.00,   0.50, 0.00, 0.50, 0.00 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.16, 0.00,   0.00, 0.25, 0.00, 0.75 ],\n",
+        " [0.00 , 0.00, 0.00, 0.00,  0.00, 0.00, 0.00, 0.00,   0.00, 0.00, 0.00, 0.25,   0.00, 0.00, 0.25, 0.00 ],\n",
+        "])\n",
+        "\n",
+        "# Store all of these in a three dimension array\n",
+        "# Pr(s_{t+1}=2|s_{t}=1, a_{t}=3] is stored at position [2,1,3]\n",
+        "transition_probabilities_given_action = np.concatenate((np.expand_dims(transition_probabilities_given_action0,2),\n",
+        "                                                        np.expand_dims(transition_probabilities_given_action1,2),\n",
+        "                                                        np.expand_dims(transition_probabilities_given_action2,2),\n",
+        "                                                        np.expand_dims(transition_probabilities_given_action3,2)),axis=2)"
+      ],
+      "metadata": {
+        "id": "l7rT78BbOgTi"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def q_learning_step(state_action_values, reward, state, new_state, action, gamma, alpha = 0.1):\n",
+        "  # TODO -- write this function\n",
+        "  # Replace this line\n",
+        "  state_action_values_after = np.copy(state_action_values)\n",
+        "\n",
+        "  return state_action_values_after"
+      ],
+      "metadata": {
+        "id": "5pO6-9ACWhiV"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# This takes a single step from an MDP which just has a completely random policy\n",
+        "def markov_decision_process_step(state, transition_probabilities_given_action, reward_structure):\n",
+        "  # Pick action\n",
+        "  action = np.random.randint(4)\n",
+        "  # Update the state\n",
+        "  new_state = np.random.choice(a=np.arange(0,transition_probabilities_given_action.shape[0]),p = transition_probabilities_given_action[:,state,action])\n",
+        "  # Return the reward -- here the reward is for leaving the state\n",
+        "  reward = reward_structure[state]\n",
+        "\n",
+        "  return new_state, reward, action"
+      ],
+      "metadata": {
+        "id": "akjrncMF-FkU"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Initialize the state-action values to random numbers\n",
+        "np.random.seed(0)\n",
+        "n_state = transition_probabilities_given_action.shape[0]\n",
+        "n_action = transition_probabilities_given_action.shape[2]\n",
+        "state_action_values = np.random.normal(size=(n_action, n_state))\n",
+        "gamma = 0.9\n",
+        "\n",
+        "policy = np.argmax(state_action_values, axis=0).astype(int)\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values, rewards = reward_structure)\n",
+        "\n",
+        "# Now let's simulate a single Q-learning step\n",
+        "initial_state = 9\n",
+        "print(\"Initial state = \", initial_state)\n",
+        "new_state, reward, action = markov_decision_process_step(initial_state, transition_probabilities_given_action, reward_structure)\n",
+        "print(\"Action = \", action)\n",
+        "print(\"New state = \", new_state)\n",
+        "print(\"Reward = \", reward)\n",
+        "\n",
+        "state_action_values_after = q_learning_step(state_action_values, reward, initial_state, new_state, action, gamma)\n",
+        "print(\"Your value:\",state_action_values_after[action, initial_state])\n",
+        "print(\"True value:  0.27650262412468796\")\n",
+        "\n",
+        "policy = np.argmax(state_action_values, axis=0).astype(int)\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values_after, rewards = reward_structure)\n"
+      ],
+      "metadata": {
+        "id": "Fu5_VjvbSwfJ"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's run this for a while and watch the policy improve"
+      ],
+      "metadata": {
+        "id": "Ogh0qucmb68J"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Initialize the state-action values to random numbers\n",
+        "np.random.seed(0)\n",
+        "n_state = transition_probabilities_given_action.shape[0]\n",
+        "n_action = transition_probabilities_given_action.shape[2]\n",
+        "state_action_values = np.random.normal(size=(n_action, n_state))\n",
+        "# Hard code termination state of finding fish\n",
+        "state_action_values[:,n_state-1] = 3.0\n",
+        "gamma = 0.9\n",
+        "\n",
+        "# Draw the initial setup\n",
+        "policy = np.argmax(state_action_values, axis=0).astype(int)\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values, rewards = reward_structure)\n",
+        "\n",
+        "\n",
+        "state= np.random.randint(n_state-1)\n",
+        "\n",
+        "# Run for a number of iterations\n",
+        "for c_iter in range(10000):\n",
+        "  new_state, reward, action = markov_decision_process_step(state, transition_probabilities_given_action, reward_structure)\n",
+        "  state_action_values_after = q_learning_step(state_action_values, reward, state, new_state, action, gamma)\n",
+        "  # If in termination state, reset state randomly\n",
+        "  if new_state==15:\n",
+        "    state= np.random.randint(n_state-1)\n",
+        "  else:\n",
+        "    state = new_state\n",
+        "  # Update the policy\n",
+        "  state_action_values = np.copy(state_action_values_after)\n",
+        "  policy = np.argmax(state_action_values, axis=0).astype(int)\n",
+        "\n",
+        "# Draw the final situation\n",
+        "mdp_drawer = DrawMDP(n_rows, n_cols)\n",
+        "mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values, rewards = reward_structure)"
+      ],
+      "metadata": {
+        "id": "qQFhwVqPcCFH"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

Notebooks/Chap19/19_5_Control_Variates.ipynb

@@ -0,0 +1,170 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyO6CLgMIO5bUVAMkzPT3z4y",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap19/19_5_Control_Variates.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 19.5: Control variates**\n",
+        "\n",
+        "This notebook investigates the method of control variates as described in figure 19.16\n",
+        "\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt"
+      ],
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Generate from our two variables, $a$ and $b$.  We are interested in estimating the mean of $a$, but we can use $b$$ to improve our estimates if it is correlated"
+      ],
+      "metadata": {
+        "id": "uwmhcAZBzTRO"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Sample from two variables with mean zero, standard deviation one, and a given correlation coefficient\n",
+        "def get_samples(n_samples, correlation_coeff=0.8):\n",
+        "  a = np.random.normal(size=(1,n_samples))\n",
+        "  temp = np.random.normal(size=(1, n_samples))\n",
+        "  b = correlation_coeff * a + np.sqrt(1-correlation_coeff * correlation_coeff) * temp\n",
+        "  return a, b"
+      ],
+      "metadata": {
+        "id": "bC8MBXPawQJ3"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "N = 10000000\n",
+        "a,b, = get_samples(N)\n",
+        "\n",
+        "# Verify that these two variables have zero mean and unit standard deviation\n",
+        "print(\"Mean of a = %3.3f,  Std of a = %3.3f\"%(np.mean(a),np.std(a)))\n",
+        "print(\"Mean of b = %3.3f,  Std of b = %3.3f\"%(np.mean(b),np.std(b)))"
+      ],
+      "metadata": {
+        "id": "1cT66nbRyW34"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's samples $N=10$ examples from $a$ and estimate their mean $\\mathbb{E}[a]$.  We'll do this 1000000 times and then compute the variance of those estimates."
+      ],
+      "metadata": {
+        "id": "PWoYRpjS0Nlf"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "n_estimate = 1000000\n",
+        "\n",
+        "N = 5\n",
+        "\n",
+        "# TODO -- sample N examples of variable $a$\n",
+        "# Compute the mean of each\n",
+        "# Compute the mean and variance of these estimates of the mean\n",
+        "# Replace this line\n",
+        "mean_of_estimator_1 = -1; std_of_estimator_1 = -1\n",
+        "\n",
+        "print(\"Standard estimator mean = %3.3f, Standard estimator variance = %3.3f\"%(mean_of_estimator_1, std_of_estimator_1))"
+      ],
+      "metadata": {
+        "id": "n6Uem2aYzBp7"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's estimate the mean $\\mathbf{E}[a]$ of $a$ by computing $a-b$ where $b$ is correlated with $a$"
+      ],
+      "metadata": {
+        "id": "F-af86z13TFc"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "n_estimate = 1000000\n",
+        "\n",
+        "N = 5\n",
+        "\n",
+        "# TODO -- sample N examples of variables $a$ and $b$\n",
+        "# Compute $c=a-b$ for each and then compute the mean of $c$\n",
+        "# Compute the mean and variance of these estimates of the mean of $c$\n",
+        "# Replace this line\n",
+        "mean_of_estimator_2 = -1; std_of_estimator_2 = -1\n",
+        "\n",
+        "print(\"Control variate estimator mean = %3.3f, Control variate estimator variance = %3.3f\"%(mean_of_estimator_2, std_of_estimator_2))"
+      ],
+      "metadata": {
+        "id": "MrEVDggY0IGU"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Note that they both have a very similar mean, but the second estimator has a lower variance.   \n",
+        "\n",
+        "TODO -- Experiment with different samples sizes $N$ and correlation coefficients."
+      ],
+      "metadata": {
+        "id": "Jklzkca14ofS"
+      }
+    }
+  ]
+}

Notebooks/Chap20/20_1_Random_Data.ipynb

@@ -0,0 +1,305 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyPkSYbEjOcEmLt8tU6HxNuR",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap20/20_1_Random_Data.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 20.1: Random Data**\n",
+        "\n",
+        "This notebook investigates training the network with random data, as illustrated in figure 20.1.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
+        "!git clone https://github.com/greydanus/mnist1d"
+      ],
+      "metadata": {
+        "id": "D5yLObtZCi9J"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import os\n",
+        "import torch, torch.nn as nn\n",
+        "from torch.utils.data import TensorDataset, DataLoader\n",
+        "from torch.optim.lr_scheduler import StepLR\n",
+        "import matplotlib.pyplot as plt\n",
+        "import mnist1d\n",
+        "import random\n",
+        "from IPython.display import display, clear_output"
+      ],
+      "metadata": {
+        "id": "YrXWAH7sUWvU"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "args = mnist1d.data.get_dataset_args()\n",
+        "data = mnist1d.data.get_dataset(args, path='./mnist1d_data.pkl', download=False, regenerate=False)\n",
+        "\n",
+        "# The training and test input and outputs are in\n",
+        "# data['x'], data['y'], data['x_test'], and data['y_test']\n",
+        "print(\"Examples in training set: {}\".format(len(data['y'])))\n",
+        "print(\"Examples in test set: {}\".format(len(data['y_test'])))\n",
+        "print(\"Length of each example: {}\".format(data['x'].shape[-1]))"
+      ],
+      "metadata": {
+        "id": "twI72ZCrCt5z"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Define the network"
+      ],
+      "metadata": {
+        "id": "_sFvRDGrl4qe"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "D_i = 40    # Input dimensions\n",
+        "D_k = 300   # Hidden dimensions\n",
+        "D_o = 10    # Output dimensions\n",
+        "\n",
+        "model = nn.Sequential(\n",
+        "nn.Linear(D_i, D_k),\n",
+        "nn.ReLU(),\n",
+        "nn.Linear(D_k, D_k),\n",
+        "nn.ReLU(),\n",
+        "nn.Linear(D_k, D_k),\n",
+        "nn.ReLU(),\n",
+        "nn.Linear(D_k, D_k),\n",
+        "nn.ReLU(),\n",
+        "nn.Linear(D_k, D_o))"
+      ],
+      "metadata": {
+        "id": "FslroPJJffrh"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# He initialization of weights\n",
+        "def weights_init(layer_in):\n",
+        "  if isinstance(layer_in, nn.Linear):\n",
+        "    nn.init.kaiming_uniform_(layer_in.weight)\n",
+        "    layer_in.bias.data.fill_(0.0)"
+      ],
+      "metadata": {
+        "id": "YgLaex1pfhqz"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def train_model(train_data_x, train_data_y, n_epoch):\n",
+        "  # choose cross entropy loss function (equation 5.24 in the loss notes)\n",
+        "  loss_function = nn.CrossEntropyLoss()\n",
+        "  # construct SGD optimizer and initialize learning rate and momentum\n",
+        "  optimizer = torch.optim.SGD(model.parameters(), lr = 0.02, momentum=0.9)\n",
+        "  # object that decreases learning rate by half every 20 epochs\n",
+        "  scheduler = StepLR(optimizer, step_size=20, gamma=0.5)\n",
+        "  # create 100 dummy data points and store in data loader class\n",
+        "  x_train = torch.tensor(train_data_x.transpose().astype('float32'))\n",
+        "  y_train = torch.tensor(train_data_y.astype('long'))\n",
+        "\n",
+        "  # load the data into a class that creates the batches\n",
+        "  data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=100, shuffle=True, worker_init_fn=np.random.seed(1))\n",
+        "\n",
+        "  # Initialize model weights\n",
+        "  model.apply(weights_init)\n",
+        "\n",
+        "  # store the loss and the % correct at each epoch\n",
+        "  losses_train = np.zeros((n_epoch))\n",
+        "\n",
+        "  for epoch in range(n_epoch):\n",
+        "    # loop over batches\n",
+        "    for i, data in enumerate(data_loader):\n",
+        "      # retrieve inputs and labels for this batch\n",
+        "      x_batch, y_batch = data\n",
+        "      # zero the parameter gradients\n",
+        "      optimizer.zero_grad()\n",
+        "      # forward pass -- calculate model output\n",
+        "      pred = model(x_batch)\n",
+        "      # compute the loss\n",
+        "      loss = loss_function(pred, y_batch)\n",
+        "      # backward pass\n",
+        "      loss.backward()\n",
+        "      # SGD update\n",
+        "      optimizer.step()\n",
+        "\n",
+        "    # Run whole dataset to get statistics -- normally wouldn't do this\n",
+        "    pred_train = model(x_train)\n",
+        "    _, predicted_train_class = torch.max(pred_train.data, 1)\n",
+        "    losses_train[epoch] = loss_function(pred_train, y_train).item()\n",
+        "    if epoch % 5 == 0:\n",
+        "        clear_output(wait=True)\n",
+        "        display(\"Epoch %d, train loss %3.3f\"%(epoch, losses_train[epoch]))\n",
+        "\n",
+        "    # tell scheduler to consider updating learning rate\n",
+        "    scheduler.step()\n",
+        "\n",
+        "  return losses_train"
+      ],
+      "metadata": {
+        "id": "NYw8I_3mmX5c"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Load in the data\n",
+        "train_data_x = data['x'].transpose()\n",
+        "train_data_y = data['y']\n",
+        "# Print out sizes\n",
+        "print(\"Train data: %d examples (columns), each of which has %d dimensions (rows)\"%((train_data_x.shape[1],train_data_x.shape[0])))"
+      ],
+      "metadata": {
+        "id": "4FE3HQ_vedXO"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Compute loss for proper data  and plot\n",
+        "n_epoch = 60\n",
+        "loss_true_labels = train_model(train_data_x, train_data_y, n_epoch)\n",
+        "# Plot the results\n",
+        "fig, ax = plt.subplots()\n",
+        "ax.plot(loss_true_labels,'r-',label='true_labels')\n",
+        "# ax.set_ylim(0,0.7); ax.set_xlim(0,n_epoch)\n",
+        "ax.set_xlabel('Epoch'); ax.set_ylabel('Loss')\n",
+        "ax.legend()\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "b56wdODqemF1"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# TODO -- Randomize the input data (train_data_x), but retain overall mean and variance\n",
+        "# Replace this line\n",
+        "train_data_x_randomized = np.copy(train_data_x)"
+      ],
+      "metadata": {
+        "id": "SbPCiiUKgTLw"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Compute loss for true labels and plot\n",
+        "n_epoch = 60\n",
+        "loss_randomized_data = train_model(train_data_x_randomized, train_data_y, n_epoch)\n",
+        "# Plot the results\n",
+        "fig, ax = plt.subplots()\n",
+        "ax.plot(loss_true_labels,'r-',label='true_labels')\n",
+        "ax.plot(loss_randomized_data,'b-',label='random_data')\n",
+        "# ax.set_ylim(0,0.7); ax.set_xlim(0,n_epoch)\n",
+        "ax.set_xlabel('Epoch'); ax.set_ylabel('Loss')\n",
+        "ax.legend()\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "y7CcCJvvjLnn"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# TODO -- Permute the labels\n",
+        "# Replace this line:\n",
+        "train_data_y_permuted = np.copy(train_data_y)"
+      ],
+      "metadata": {
+        "id": "ojaMTrzKj_74"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Compute loss for true labels and plot\n",
+        "n_epoch = 60\n",
+        "loss_permuted_labels = train_model(train_data_x, train_data_y_permuted, n_epoch)\n",
+        "# Plot the results\n",
+        "fig, ax = plt.subplots()\n",
+        "ax.plot(loss_true_labels,'r-',label='true_labels')\n",
+        "ax.plot(loss_randomized_data,'b-',label='random_data')\n",
+        "ax.plot(loss_permuted_labels,'g-',label='random_labels')\n",
+        "# ax.set_ylim(0,0.7); ax.set_xlim(0,n_epoch)\n",
+        "ax.set_xlabel('Epoch'); ax.set_ylabel('Loss')\n",
+        "ax.legend()\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "LaYCSjyMo9LQ"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

Notebooks/Chap20/20_2_Full_Batch_Gradient_Descent.ipynb

@@ -0,0 +1,296 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyOo4vm4MXcIvAzVlMCaLikH",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap20/20_2_Full_Batch_Gradient_Descent.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 20.2: Full Batch Gradient Descent**\n",
+        "\n",
+        "This notebook investigates training a network with full batch gradient descent as in figure 20.2.  There is also a version (notebook takes a long time to run), but this didn't speed it up much for me.  If you run out of CoLab time,  you'll need to download the Python file and run locally.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
+        "!git clone https://github.com/greydanus/mnist1d"
+      ],
+      "metadata": {
+        "id": "D5yLObtZCi9J"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import os\n",
+        "import torch, torch.nn as nn\n",
+        "from torch.utils.data import TensorDataset, DataLoader\n",
+        "from torch.optim.lr_scheduler import StepLR\n",
+        "import matplotlib.pyplot as plt\n",
+        "import mnist1d\n",
+        "import random\n",
+        "from IPython.display import display, clear_output"
+      ],
+      "metadata": {
+        "id": "YrXWAH7sUWvU"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "args = mnist1d.data.get_dataset_args()\n",
+        "data = mnist1d.data.get_dataset(args, path='./mnist1d_data.pkl', download=False, regenerate=False)\n",
+        "\n",
+        "# The training and test input and outputs are in\n",
+        "# data['x'], data['y']\n",
+        "print(\"Examples in training set: {}\".format(len(data['y'])))\n",
+        "print(\"Length of each example: {}\".format(data['x'].shape[-1]))"
+      ],
+      "metadata": {
+        "id": "twI72ZCrCt5z"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Define the network"
+      ],
+      "metadata": {
+        "id": "_sFvRDGrl4qe"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Data is length forty, and there are 10 classes\n",
+        "D_i = 40\n",
+        "D_o = 10\n",
+        "\n",
+        "# create model with one hidden layer and 298 hidden units\n",
+        "model_1_layer = nn.Sequential(\n",
+        "nn.Linear(D_i, 298),\n",
+        "nn.ReLU(),\n",
+        "nn.Linear(298, D_o))\n",
+        "\n",
+        "\n",
+        "# TODO -- create model with three hidden layers and 100 hidden units per layer\n",
+        "# Replace this line\n",
+        "model_2_layer = nn.Sequential(nn.Linear(D_i, D_o))\n",
+        "\n",
+        "\n",
+        "\n",
+        "# TODO -- Create model with three hidden layers and 75 hidden units per layer\n",
+        "# Replace this line\n",
+        "model_3_layer = nn.Sequential(nn.Linear(D_i, D_o))\n",
+        "\n",
+        "\n",
+        "\n",
+        "# TODO create model with four hidden layers and 63 hidden units per layer\n",
+        "# Replace this line\n",
+        "model_4_layer = nn.Sequential(nn.Linear(D_i, D_o))\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "FslroPJJffrh"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# He initialization of weights\n",
+        "def weights_init(layer_in):\n",
+        "  if isinstance(layer_in, nn.Linear):\n",
+        "    nn.init.kaiming_uniform_(layer_in.weight)\n",
+        "    layer_in.bias.data.fill_(0.0)"
+      ],
+      "metadata": {
+        "id": "YgLaex1pfhqz"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def train_model(model, train_data_x, train_data_y, n_epoch):\n",
+        "  print(\"This is going to take a long time!\")\n",
+        "  # choose cross entropy loss function (equation 5.24 in the loss notes)\n",
+        "  loss_function = nn.CrossEntropyLoss()\n",
+        "  # construct SGD optimizer and initialize learning rate to small value and momentum to 0\n",
+        "  optimizer = torch.optim.SGD(model.parameters(), lr = 0.0025, momentum=0.0)\n",
+        "  # create 100 dummy data points and store in data loader class\n",
+        "  x_train = torch.tensor(train_data_x.transpose().astype('float32'))\n",
+        "  y_train = torch.tensor(train_data_y.astype('long'))\n",
+        "\n",
+        "  # load the data into a class that creates the batches -- full batch as there are 4000 examples\n",
+        "  data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=4000, shuffle=False, worker_init_fn=np.random.seed(1))\n",
+        "\n",
+        "  # Initialize model weights\n",
+        "  model.apply(weights_init)\n",
+        "\n",
+        "  # store the errors percentage at each point\n",
+        "  errors_train = np.zeros((n_epoch))\n",
+        "\n",
+        "  for epoch in range(n_epoch):\n",
+        "    # loop over batches\n",
+        "    for i, data in enumerate(data_loader):\n",
+        "      # retrieve inputs and labels for this batch\n",
+        "      x_batch, y_batch = data\n",
+        "      # zero the parameter gradients\n",
+        "      optimizer.zero_grad()\n",
+        "      # forward pass -- calculate model output\n",
+        "      pred = model(x_batch)\n",
+        "      # compute the loss\n",
+        "      loss = loss_function(pred, y_batch)\n",
+        "      # Store the errors\n",
+        "      _, predicted_train_class = torch.max(pred.data, 1)\n",
+        "      errors_train[epoch] = 100 - 100 * (predicted_train_class == y_train).float().sum() / len(y_train)\n",
+        "      # backward pass\n",
+        "      loss.backward()\n",
+        "      # SGD update\n",
+        "      optimizer.step()\n",
+        "\n",
+        "      if epoch % 10 == 0:\n",
+        "        clear_output(wait=True)\n",
+        "        display(\"Epoch %d, errors_train %3.3f\"%(epoch, errors_train[epoch]))\n",
+        "\n",
+        "  return errors_train"
+      ],
+      "metadata": {
+        "id": "NYw8I_3mmX5c"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Load in the data\n",
+        "train_data_x = data['x'].transpose()\n",
+        "train_data_y = data['y']\n",
+        "# Print out sizes\n",
+        "print(\"Train data: %d examples (columns), each of which has %d dimensions (rows)\"%((train_data_x.shape[1],train_data_x.shape[0])))"
+      ],
+      "metadata": {
+        "id": "4FE3HQ_vedXO"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Train the models\n",
+        "errors_four_layers = train_model(model_4_layer, train_data_x, train_data_y, n_epoch=200000)"
+      ],
+      "metadata": {
+        "id": "b56wdODqemF1"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "errors_three_layers = train_model(model_3_layer, train_data_x, train_data_y, n_epoch=200000)\n"
+      ],
+      "metadata": {
+        "id": "hqY-MJVPnCBV"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "errors_two_layers = train_model(model_2_layer, train_data_x, train_data_y, n_epoch=200000)\n"
+      ],
+      "metadata": {
+        "id": "T61jfpNGnDGj"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "errors_one_layer = train_model(model_1_layer, train_data_x, train_data_y, n_epoch=500000)"
+      ],
+      "metadata": {
+        "id": "HO8ZFgYqnEQe"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Plot the results\n",
+        "fig, ax = plt.subplots()\n",
+        "ax.plot(errors_one_layer,'r-',label='one layer')\n",
+        "ax.plot(errors_two_layers,'g-',label='two layers')\n",
+        "ax.plot(errors_three_layers,'b-',label='three layers')\n",
+        "ax.plot(errors_four_layers,'m-',label='four layers')\n",
+        "ax.set_ylim(0,100)\n",
+        "ax.set_xlabel('Epoch'); ax.set_ylabel('Percent error')\n",
+        "ax.legend()\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "pYL0YMI5oNSR"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [],
+      "metadata": {
+        "id": "wJerga3M7eDw"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

Notebooks/Chap20/20_2_Full_Batch_Gradient_Descent_GPU.ipynb

@@ -0,0 +1,303 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "gpuType": "T4",
+      "authorship_tag": "ABX9TyMjPBfDONmjqTSyEQDP2gjY",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    },
+    "accelerator": "GPU"
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap20/20_2_Full_Batch_Gradient_Descent_GPU.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 20.2: Full Batch Gradient Descent**\n",
+        "\n",
+        "This notebook investigates training a network with full batch gradient descent as in figure 20.2.  This is the GPU version (notebook takes a long time to run).  If you are using Colab then you need to go change the runtime type to GPU on the Runtime menu.  Even then, you may run out of time.  If that's the case, you'll need to download the Python file and run locally.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
+        "!git clone https://github.com/greydanus/mnist1d"
+      ],
+      "metadata": {
+        "id": "D5yLObtZCi9J"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import os\n",
+        "import torch, torch.nn as nn\n",
+        "from torch.utils.data import TensorDataset, DataLoader\n",
+        "from torch.optim.lr_scheduler import StepLR\n",
+        "import matplotlib.pyplot as plt\n",
+        "import mnist1d\n",
+        "import random\n",
+        "from IPython.display import display, clear_output\n",
+        "\n",
+        "\n",
+        "# Try attaching to GPU\n",
+        "DEVICE = str(torch.device('cuda' if torch.cuda.is_available() else 'cpu'))\n",
+        "print('Using:', DEVICE)"
+      ],
+      "metadata": {
+        "id": "YrXWAH7sUWvU"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "args = mnist1d.data.get_dataset_args()\n",
+        "data = mnist1d.data.get_dataset(args, path='./mnist1d_data.pkl', download=False, regenerate=False)\n",
+        "\n",
+        "# The training and test input and outputs are in\n",
+        "# data['x'], data['y']\n",
+        "print(\"Examples in training set: {}\".format(len(data['y'])))\n",
+        "print(\"Length of each example: {}\".format(data['x'].shape[-1]))"
+      ],
+      "metadata": {
+        "id": "twI72ZCrCt5z"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Define the network"
+      ],
+      "metadata": {
+        "id": "_sFvRDGrl4qe"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Data is length forty, and there are 10 classes\n",
+        "D_i = 40\n",
+        "D_o = 10\n",
+        "\n",
+        "# create model with one hidden layer and 298 hidden units\n",
+        "model_1_layer = nn.Sequential(\n",
+        "nn.Linear(D_i, 298),\n",
+        "nn.ReLU(),\n",
+        "nn.Linear(298, D_o))\n",
+        "\n",
+        "\n",
+        "# TODO -- create model with three hidden layers and 100 hidden units per layer\n",
+        "# Replace this line\n",
+        "model_2_layer = nn.Sequential(nn.Linear(D_i, D_o))\n",
+        "\n",
+        "# TODO -- Create model with three hidden layers and 75 hidden units per layer\n",
+        "# Replace this line\n",
+        "model_3_layer = nn.Sequential(nn.Linear(D_i, D_o))\n",
+        "\n",
+        "# TODO create model with four hidden layers and 63 hidden units per layer\n",
+        "# Replace this line\n",
+        "model_4_layer = nn.Sequential(nn.Linear(D_i, D_o))\n"
+      ],
+      "metadata": {
+        "id": "FslroPJJffrh"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# He initialization of weights\n",
+        "def weights_init(layer_in):\n",
+        "  if isinstance(layer_in, nn.Linear):\n",
+        "    nn.init.kaiming_uniform_(layer_in.weight)\n",
+        "    layer_in.bias.data.fill_(0.0)"
+      ],
+      "metadata": {
+        "id": "YgLaex1pfhqz"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def train_model(model, train_data_x, train_data_y, n_epoch, DEVICE):\n",
+        "  print(\"This is going to take a long time!\")\n",
+        "  # choose cross entropy loss function (equation 5.24 in the loss notes)\n",
+        "  loss_function = nn.CrossEntropyLoss()\n",
+        "  # construct SGD optimizer and initialize learning rate to small value and momentum to 0\n",
+        "  optimizer = torch.optim.SGD(model.parameters(), lr = 0.0025, momentum=0.0)\n",
+        "  # create 100 dummy data points and store in data loader class\n",
+        "  x_train = torch.tensor(train_data_x.transpose(), dtype=torch.float32, device=DEVICE)\n",
+        "  y_train = torch.tensor(train_data_y, dtype=torch.long, device=DEVICE)\n",
+        "\n",
+        "  # load the data into a class that creates the batches -- full batch as there are 4000 examples\n",
+        "  data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=4000, shuffle=False, worker_init_fn=np.random.seed(1))\n",
+        "\n",
+        "  # Initialize model weights\n",
+        "  model.apply(weights_init)\n",
+        "\n",
+        "  # store the errors percentage at each point\n",
+        "  errors_train = np.zeros((n_epoch))\n",
+        "\n",
+        "  for epoch in range(n_epoch):\n",
+        "    # loop over batches\n",
+        "    for i, data in enumerate(data_loader):\n",
+        "      # retrieve inputs and labels for this batch\n",
+        "      x_batch, y_batch = data\n",
+        "      # zero the parameter gradients\n",
+        "      optimizer.zero_grad()\n",
+        "      # forward pass -- calculate model output\n",
+        "      pred = model(x_batch)\n",
+        "      # compute the loss\n",
+        "      loss = loss_function(pred, y_batch)\n",
+        "      # Store the errors\n",
+        "      _, predicted_train_class = torch.max(pred.data, 1)\n",
+        "      errors_train[epoch] = 100 - 100 * (predicted_train_class == y_train).float().sum() / len(y_train)\n",
+        "      # backward pass\n",
+        "      loss.backward()\n",
+        "      # SGD update\n",
+        "      optimizer.step()\n",
+        "\n",
+        "      if epoch % 10 == 0:\n",
+        "        clear_output(wait=True)\n",
+        "        display(\"Epoch %d, errors_train %3.3f\"%(epoch, errors_train[epoch]))\n",
+        "\n",
+        "  return errors_train"
+      ],
+      "metadata": {
+        "id": "NYw8I_3mmX5c"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Load in the data\n",
+        "train_data_x = data['x'].transpose()\n",
+        "train_data_y = data['y']\n",
+        "# Print out sizes\n",
+        "print(\"Train data: %d examples (columns), each of which has %d dimensions (rows)\"%((train_data_x.shape[1],train_data_x.shape[0])))"
+      ],
+      "metadata": {
+        "id": "4FE3HQ_vedXO"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Train the four models\n",
+        "model_4_layer = model_4_layer.to(DEVICE)\n",
+        "errors_four_layers = train_model(model_4_layer, train_data_x, train_data_y, n_epoch=200000, DEVICE=DEVICE)\n"
+      ],
+      "metadata": {
+        "id": "b56wdODqemF1"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "model_3_layer = model_3_layer.to(DEVICE)\n",
+        "errors_three_layers = train_model(model_3_layer, train_data_x, train_data_y, n_epoch=200000, DEVICE=DEVICE)\n"
+      ],
+      "metadata": {
+        "id": "63WsEgDCmbB4"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "model_2_layer = model_2_layer.to(DEVICE)\n",
+        "errors_two_layers = train_model(model_2_layer, train_data_x, train_data_y, n_epoch=200000, DEVICE=DEVICE)\n"
+      ],
+      "metadata": {
+        "id": "3TfS5DaZmdCN"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "model_1_layer = model_1_layer.to(DEVICE)\n",
+        "errors_one_layer = train_model(model_1_layer, train_data_x, train_data_y, n_epoch=500000, DEVICE=DEVICE)"
+      ],
+      "metadata": {
+        "id": "3f9Z6Mh4meeA"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Plot the results\n",
+        "fig, ax = plt.subplots()\n",
+        "ax.plot(errors_one_layer,'r-',label='one layer')\n",
+        "ax.plot(errors_two_layers,'g-',label='two layers')\n",
+        "ax.plot(errors_three_layers,'b-',label='three layers')\n",
+        "ax.plot(errors_four_layers,'m-',label='four layers')\n",
+        "ax.set_ylim(0,100)\n",
+        "ax.set_xlabel('Epoch'); ax.set_ylabel('Percent error')\n",
+        "ax.legend()\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "pYL0YMI5oNSR"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [],
+      "metadata": {
+        "id": "iJem05Y03mZB"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

Notebooks/Chap20/20_3_Lottery_Tickets.ipynb

@@ -0,0 +1,377 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap20/20_3_Lottery_Tickets.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dKUcDM76bHx3"
+      },
+      "source": [
+        "# **Notebook 20.3: Lottery tickets**\n",
+        "\n",
+        "This notebook investigates the phenomenon of lottery tickets as discussed in section 20.2.7.  This notebook is highly derivative of the MNIST-1D code hosted by Sam Greydanus at https://github.com/greydanus/mnist1d.   \n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "Sg2i1QmhKW5d"
+      },
+      "source": [
+        "# Run this if you're in a Colab\n",
+        "!git clone https://github.com/greydanus/mnist1d"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Lottery tickets\n",
+        "\n",
+        "Lottery tickets were first identified by [Frankle and Carbin (2018)](https://arxiv.org/abs/1803.03635).  They noted that after training a network, they could set the smaller weights to zero and clamp them there and retrain to get a network that was sparser (had fewer parameters) but could actually perform better.  So within the neural network there lie smaller sub-networks which are superior.  If we knew what these were, we could train them from scratch, but there is currently no way of finding out."
+      ],
+      "metadata": {
+        "id": "97g8gY5XdcKR"
+      }
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "KaQo7QhvXvid"
+      },
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "\n",
+        "import torch\n",
+        "import torch.nn as nn\n",
+        "import torch.nn.functional as F\n",
+        "import torch.optim as optim\n",
+        "\n",
+        "import mnist1d\n",
+        "import copy"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nre26wEOfZsM"
+      },
+      "source": [
+        "## Get the MNIST1D dataset"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "I-vm_gh5xTJs"
+      },
+      "source": [
+        "args = mnist1d.get_dataset_args()\n",
+        "data = mnist1d.get_dataset(args=args)  # by default, this will download a pre-made dataset from the GitHub repo\n",
+        "\n",
+        "print(\"Examples in training set: {}\".format(len(data['y'])))\n",
+        "print(\"Examples in test set: {}\".format(len(data['y_test'])))\n",
+        "print(\"Length of each input: {}\".format(data['x'].shape[-1]))\n",
+        "print(\"Number of classes: {}\".format(len(data['templates']['y'])))"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "O2vy0FKjfDwr"
+      },
+      "source": [
+        "## Make an MLP that can be masked\n",
+        "These parameter-wise binary masks are how we will represent sparsity in this project. There's not a great PyTorch API for this yet, so here's a temporary solution."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "uBx5gNW-mqH_"
+      },
+      "source": [
+        "# Class to represent linear layer where some of the weights are forced to zero.\n",
+        "class SparseLinear(torch.nn.Module):\n",
+        "  def __init__(self, x_size, y_size):\n",
+        "    super(SparseLinear, self).__init__()\n",
+        "    self.linear = torch.nn.Linear(x_size, y_size)\n",
+        "    param_vec = torch.cat([p.flatten() for p in self.parameters()])\n",
+        "    self.mask = torch.ones_like(param_vec)\n",
+        "\n",
+        "  def forward(self, x, apply_mask=True):\n",
+        "    if apply_mask:\n",
+        "      self.apply_mask()\n",
+        "    return self.linear(x)\n",
+        "\n",
+        "  def update_mask(self, new_mask):\n",
+        "    self.mask = new_mask\n",
+        "    self.apply_mask()\n",
+        "\n",
+        "  def apply_mask(self):\n",
+        "    self.vec2param(self.param2vec())\n",
+        "\n",
+        "  def param2vec(self):\n",
+        "    vec = torch.cat([p.flatten() for p in self.parameters()])\n",
+        "    return self.mask * vec\n",
+        "\n",
+        "  def vec2param(self, vec):\n",
+        "    pointer = 0\n",
+        "    for param in self.parameters():\n",
+        "      param_len = np.cumprod(param.shape)[-1]\n",
+        "      new_param = vec[pointer:pointer+param_len].reshape(param.shape)\n",
+        "      param.data = new_param.data\n",
+        "      pointer += param_len\n",
+        "\n",
+        "# A two layer residual network where the linear layers are sparse\n",
+        "class SparseMLP(torch.nn.Module):\n",
+        "  def __init__(self, input_size, output_size, hidden_size=100):\n",
+        "    super(SparseMLP, self).__init__()\n",
+        "    self.linear1 = SparseLinear(input_size, hidden_size)\n",
+        "    self.linear2 = SparseLinear(hidden_size, hidden_size)\n",
+        "    self.linear3 = SparseLinear(hidden_size, output_size)\n",
+        "    self.layers = [self.linear1, self.linear2, self.linear3]\n",
+        "\n",
+        "  def forward(self, x):\n",
+        "    h = torch.relu(self.linear1(x))\n",
+        "    h = h + torch.relu(self.linear2(h))\n",
+        "    h = self.linear3(h)\n",
+        "    return h\n",
+        "\n",
+        "  def get_layer_masks(self):\n",
+        "    return [l.mask for l in self.layers]\n",
+        "\n",
+        "  def set_layer_masks(self, new_masks):\n",
+        "    for i, l in enumerate(self.layers):\n",
+        "      l.update_mask(new_masks[i])\n",
+        "\n",
+        "  def get_layer_vecs(self):\n",
+        "    return [l.param2vec() for l in self.layers]\n",
+        "\n",
+        "  def set_layer_vecs(self, vecs):\n",
+        "    for i, l in enumerate(self.layers):\n",
+        "      l.vec2param(vecs[i])"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2hwmH2vIbHin"
+      },
+      "source": [
+        "Now we need a routine that takes the weights from the model and returns a mask that identifies the positions with the lowest magnitude.  These will be the weights that we mask."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "Md2F9WDgYSqT"
+      },
+      "source": [
+        "# absolute weights -- absolute values of all the weights from the model in a long vector\n",
+        "# percent_sparse: how much to sparsify the model\n",
+        "def get_mask(absolute_weights, percent_sparse):\n",
+        "  # TODO -- Write a function that returns a mask that has a zero\n",
+        "  # everywhere for the lowest \"percent_sparse\" of the absolute weights.\n",
+        "  # E.g. if absolute_weights contains [5,6,0,1,7] and we want percent_sparse of 40%,\n",
+        "  # we would return [1,1,0,0,1]\n",
+        "  # Remember that these are torch tensors and not numpy arrays\n",
+        "  # Replace this function:\n",
+        "  mask = torch.ones_like(scores)\n",
+        "\n",
+        "\n",
+        "  return mask"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "z0McGMV-a3Xo"
+      },
+      "source": [
+        "## The prune-and-retrain cycle\n",
+        "This is the core method for finding a lottery ticket. We train a model for a fixed number of epochs, prune it, and then re-train and re-prune. We repeat this cycle until we achieve the desired level of sparsity."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "5idcbyA3Ylz_"
+      },
+      "source": [
+        "def find_lottery_ticket(model, dataset, args, sparsity_schedule, criteria_fn=None, **kwargs):\n",
+        "\n",
+        "  criteria_fn = lambda init_params, final_params: final_params.abs()\n",
+        "\n",
+        "  init_params = model.get_layer_vecs()\n",
+        "  stats = {'train_losses':[], 'test_losses':[], 'train_accs':[], 'test_accs':[]}\n",
+        "  models = []\n",
+        "  for i, percent_sparse in enumerate(sparsity_schedule):\n",
+        "\n",
+        "    # layer-wise pruning, where pruning heuristic is determined by criteria_fn\n",
+        "    final_params = model.get_layer_vecs()\n",
+        "    scores = [criteria_fn(ip, fp) for ip, fp in zip(init_params, final_params)]\n",
+        "    masks = [get_mask(s, percent_sparse) for s in scores]\n",
+        "\n",
+        "    # update model with mask and init parameters\n",
+        "    model.set_layer_vecs(init_params)\n",
+        "    model.set_layer_masks(masks)\n",
+        "\n",
+        "    # training process\n",
+        "    results = mnist1d.train_model(dataset, model, args)\n",
+        "    model = results['checkpoints'][-1]\n",
+        "\n",
+        "    # store stats\n",
+        "    stats['train_losses'].append(results['train_losses'])\n",
+        "    stats['test_losses'].append(results['test_losses'])\n",
+        "    stats['train_accs'].append(results['train_acc'])\n",
+        "    stats['test_accs'].append(results['test_acc'])\n",
+        "\n",
+        "    # print progress\n",
+        "    if (i+1) % 1 == 0:\n",
+        "      print('\\tretrain #{}, sparsity {:.2f}, final_train_loss {:.3e}, max_acc {:.1f}, last_acc {:.1f}, mean_acc {:.1f}'\n",
+        "            .format(i+1, percent_sparse, results['train_losses'][-1], np.max(results['test_acc']),\n",
+        "            results['test_acc'][-1], np.mean(results['test_acc']) ))\n",
+        "      models.append(copy.deepcopy(model))\n",
+        "\n",
+        "  stats = {k: np.stack(v) for k, v in stats.items()}\n",
+        "  return models, stats"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "m4lokvdD4DKI"
+      },
+      "source": [
+        "## Choose hyperparameters"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "OUe7-b-7Yl2c"
+      },
+      "source": [
+        "# train settings\n",
+        "model_args = mnist1d.get_model_args()\n",
+        "model_args.total_steps = 1501\n",
+        "model_args.hidden_size = 500\n",
+        "model_args.print_every = 5000 # print never\n",
+        "model_args.eval_every = 100\n",
+        "model_args.learning_rate = 2e-2\n",
+        "model_args.device = str('cpu')"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "hVgDM5rI4J65"
+      },
+      "source": [
+        "Find the lottery ticket by repeatedly training and then pruning weights based on their magnitudes. We'll remove 1% of the weights each time. This is going to take half an hour or so.  Go and have lunch or whatever."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# sparsity settings - we will train 100 models with progressively increasing sparsity\n",
+        "num_retrains = 100\n",
+        "sparsity_schedule = np.linspace(0,1.,num_retrains)\n",
+        "\n",
+        "print(\"Magnitude pruning\")\n",
+        "mnist1d.set_seed(model_args.seed)\n",
+        "model = SparseMLP(model_args.input_size, model_args.output_size, hidden_size=model_args.hidden_size)\n",
+        "\n",
+        "criteria_fn = lambda init_params, final_params: final_params.abs()\n",
+        "lott_models, lott_stats = find_lottery_ticket(model, data, model_args, sparsity_schedule, criteria_fn=criteria_fn, prune_print_every=1)"
+      ],
+      "metadata": {
+        "id": "M25YpCuS1Gn0"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "test_losses = lott_stats['test_losses'][:,-1]\n",
+        "test_accs = lott_stats['test_accs'][:,-1]\n",
+        "\n",
+        "fig,ax = plt.subplots()\n",
+        "ax.plot(sparsity_schedule, test_losses,'r-')\n",
+        "ax.plot((sparsity_schedule[0], sparsity_schedule[-1]),(test_losses[0], test_losses[0]),'k--',label='dense')\n",
+        "ax.set_xlabel('Sparsity')\n",
+        "ax.set_ylabel('Loss')\n",
+        "ax.set_xlim(0,1)\n",
+        "ax.legend()\n",
+        "plt.show()\n",
+        "\n",
+        "fig,ax = plt.subplots()\n",
+        "ax.plot(sparsity_schedule, 100-test_accs,'r-')\n",
+        "ax.plot((sparsity_schedule[0], sparsity_schedule[-1]),(100-test_accs[0], 100-test_accs[0]),'k--',label='dense')\n",
+        "ax.set_xlabel('Sparsity')\n",
+        "ax.set_ylabel('Percent Error')\n",
+        "ax.set_xlim(0,1)\n",
+        "ax.set_ylim(0,100)\n",
+        "ax.legend()\n",
+        "plt.show()\n"
+      ],
+      "metadata": {
+        "id": "TCs-kt6-3xHB"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "You should see that the test loss decreases and the errors decrease with more as the network gets sparser.  The dashed line represents the original dense (unpruned) network.  We have identified a simpler network that was \"inside\" the original network for which the results are superior.  Of course if we make it too sparse, then it gets worse again.\n",
+        "\n",
+        "This phenomenon is explored much further in the original notebook by Sam Greydanus which can be found [here](https://github.com/greydanus/mnist1d)."
+      ],
+      "metadata": {
+        "id": "CEj5_ZEHcRpw"
+      }
+    }
+  ]
+}

Notebooks/Chap20/20_4_Adversarial_Attacks.ipynb

@@ -0,0 +1,386 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyP9amtzXsNWqkmiPUQgxzKV",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap20/20_4_Adversarial_Attacks.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 20.4: Adversarial attacks**\n",
+        "\n",
+        "This notebook builds uses the network for classification of MNIST from Notebook 10.5.  The code is adapted from https://nextjournal.com/gkoehler/pytorch-mnist, and uses the fast gradient sign attack of [Goodfellow et al. (2015)](https://arxiv.org/abs/1412.6572).  Having trained, the network, we search for adversarial examples -- inputs which look very similar to class A, but are mistakenly classified as class B.  We do this by starting with a correctly classified example and perturbing it according to the gradients of the network so that the output changes.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import torch\n",
+        "import torchvision\n",
+        "import torch.nn as nn\n",
+        "import torch.nn.functional as F\n",
+        "import torch.optim as optim\n",
+        "import matplotlib.pyplot as plt\n",
+        "import random"
+      ],
+      "metadata": {
+        "id": "YrXWAH7sUWvU"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Run this once to load the train and test data straight into a dataloader class\n",
+        "# that will provide the batches\n",
+        "batch_size_train = 64\n",
+        "batch_size_test = 1000\n",
+        "train_loader = torch.utils.data.DataLoader(\n",
+        "  torchvision.datasets.MNIST('/files/', train=True, download=True,\n",
+        "                             transform=torchvision.transforms.Compose([\n",
+        "                               torchvision.transforms.ToTensor(),\n",
+        "                               torchvision.transforms.Normalize(\n",
+        "                                 (0.1307,), (0.3081,))\n",
+        "                             ])),\n",
+        "  batch_size=batch_size_train, shuffle=True)\n",
+        "\n",
+        "test_loader = torch.utils.data.DataLoader(\n",
+        "  torchvision.datasets.MNIST('/files/', train=False, download=True,\n",
+        "                             transform=torchvision.transforms.Compose([\n",
+        "                               torchvision.transforms.ToTensor(),\n",
+        "                               torchvision.transforms.Normalize(\n",
+        "                                 (0.1307,), (0.3081,))\n",
+        "                             ])),\n",
+        "  batch_size=batch_size_test, shuffle=True)"
+      ],
+      "metadata": {
+        "id": "wScBGXXFVadm"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Let's draw some of the training data\n",
+        "examples = enumerate(test_loader)\n",
+        "batch_idx, (example_data, example_targets) = next(examples)\n",
+        "\n",
+        "fig = plt.figure()\n",
+        "for i in range(6):\n",
+        "  plt.subplot(2,3,i+1)\n",
+        "  plt.tight_layout()\n",
+        "  plt.imshow(example_data[i][0], cmap='gray', interpolation='none')\n",
+        "  plt.title(\"Ground Truth: {}\".format(example_targets[i]))\n",
+        "  plt.xticks([])\n",
+        "  plt.yticks([])\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "8bKADvLHbiV5"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Define the network.  This is a more typical way to define a network than the sequential structure.  We define a class for the network, and define the parameters in the constructor.  Then we use a function called forward to actually run the network.  It's easy to see how you might use residual connections in this format."
+      ],
+      "metadata": {
+        "id": "_sFvRDGrl4qe"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "from os import X_OK\n",
+        "\n",
+        "class Net(nn.Module):\n",
+        "    def __init__(self):\n",
+        "        super(Net, self).__init__()\n",
+        "        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)\n",
+        "        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)\n",
+        "        self.drop = nn.Dropout2d()\n",
+        "        self.fc1 = nn.Linear(320, 50)\n",
+        "        self.fc2 = nn.Linear(50, 10)\n",
+        "\n",
+        "    def forward(self, x):\n",
+        "        x = self.conv1(x)\n",
+        "        x = F.max_pool2d(x,2)\n",
+        "        x = F.relu(x)\n",
+        "        x = self.conv2(x)\n",
+        "        x = self.drop(x)\n",
+        "        x = F.max_pool2d(x,2)\n",
+        "        x = F.relu(x)\n",
+        "        x = x.flatten(1)\n",
+        "        x = F.relu(self.fc1(x))\n",
+        "        x = self.fc2(x)\n",
+        "        x = F.log_softmax(x)\n",
+        "        return x"
+      ],
+      "metadata": {
+        "id": "EQkvw2KOPVl7"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# He initialization of weights\n",
+        "def weights_init(layer_in):\n",
+        "  if isinstance(layer_in, nn.Linear):\n",
+        "    nn.init.kaiming_uniform_(layer_in.weight)\n",
+        "    layer_in.bias.data.fill_(0.0)"
+      ],
+      "metadata": {
+        "id": "qWZtkCZcU_dg"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Create network\n",
+        "model = Net()\n",
+        "# Initialize model weights\n",
+        "model.apply(weights_init)\n",
+        "# Define optimizer\n",
+        "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)"
+      ],
+      "metadata": {
+        "id": "FslroPJJffrh"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Main training routine\n",
+        "def train(epoch):\n",
+        "  model.train()\n",
+        "  # Get each\n",
+        "  for batch_idx, (data, target) in enumerate(train_loader):\n",
+        "    optimizer.zero_grad()\n",
+        "    output = model(data)\n",
+        "    loss = F.nll_loss(output, target)\n",
+        "    loss.backward()\n",
+        "    optimizer.step()\n",
+        "    # Store results\n",
+        "    if batch_idx % 10 == 0:\n",
+        "      print('Train Epoch: {} [{}/{}]\\tLoss: {:.6f}'.format(\n",
+        "        epoch, batch_idx * len(data), len(train_loader.dataset), loss.item()))"
+      ],
+      "metadata": {
+        "id": "xKQd9PzkQ766"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Run on test data\n",
+        "def test():\n",
+        "  model.eval()\n",
+        "  test_loss = 0\n",
+        "  correct = 0\n",
+        "  with torch.no_grad():\n",
+        "    for data, target in test_loader:\n",
+        "      output = model(data)\n",
+        "      test_loss += F.nll_loss(output, target, size_average=False).item()\n",
+        "      pred = output.data.max(1, keepdim=True)[1]\n",
+        "      correct += pred.eq(target.data.view_as(pred)).sum()\n",
+        "  test_loss /= len(test_loader.dataset)\n",
+        "  print('\\nTest set: Avg. loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n",
+        "    test_loss, correct, len(test_loader.dataset),\n",
+        "    100. * correct / len(test_loader.dataset)))"
+      ],
+      "metadata": {
+        "id": "Byn-f7qWRLxX"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Get initial performance\n",
+        "test()\n",
+        "# Train for three epochs\n",
+        "n_epochs = 3\n",
+        "for epoch in range(1, n_epochs + 1):\n",
+        "  train(epoch)\n",
+        "  test()"
+      ],
+      "metadata": {
+        "id": "YgLaex1pfhqz"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Run network on data we got before and show predictions\n",
+        "output = model(example_data)\n",
+        "\n",
+        "fig = plt.figure()\n",
+        "for i in range(6):\n",
+        "  plt.subplot(2,3,i+1)\n",
+        "  plt.tight_layout()\n",
+        "  plt.imshow(example_data[i][0], cmap='gray', interpolation='none')\n",
+        "  plt.title(\"Prediction: {}\".format(\n",
+        "    output.data.max(1, keepdim=True)[1][i].item()))\n",
+        "  plt.xticks([])\n",
+        "  plt.yticks([])\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "o7fRUAy9Se1B"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "This is the code that does the adversarial attack. It is adapted from [here](https://pytorch.org/tutorials/beginner/fgsm_tutorial.html).   It is an example of the fast gradient sign method (FGSM), which modifies the data by\n",
+        "\n",
+        "\n",
+        "\n",
+        "*   Calculating the derivative $\\partial L/\\partial \\mathbf{x}$ of the loss $L$ with respect to the input data $\\mathbf{x}$.\n",
+        "*   Finds the sign of the gradient at each point (making a tensor the same size as $\\mathbf{x}$ with a one where it was positive and minus one where it was negative.  \n",
+        "*   Multiplying this vector by $\\epsilon$ and adding it back to the original data\n",
+        "\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "EabuoMdP32Hd"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# FGSM attack code.\n",
+        "def fgsm_attack(x, epsilon, dLdx):\n",
+        "    # TODO -- write this function\n",
+        "    # Get the sign of the gradient\n",
+        "    # Add epsilon times the size of gradient to x\n",
+        "    # Replace this line\n",
+        "    x_modified = torch.zeros_like(x)\n",
+        "\n",
+        "    # Return the perturbed image\n",
+        "    return x_modified"
+      ],
+      "metadata": {
+        "id": "gAX7tnld46q1"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "no_examples = 3\n",
+        "epsilon = 0.5\n",
+        "for i in range(no_examples):\n",
+        "  # Reset gradients\n",
+        "  optimizer.zero_grad()\n",
+        "\n",
+        "  # Get the i'th data example\n",
+        "  x = example_data[i,:,:,:]\n",
+        "  # Add an extra dimension back to the beginning\n",
+        "  x= x[None, :,:,:]\n",
+        "  x.requires_grad = True\n",
+        "  # Get the i'th target\n",
+        "  y = torch.ones(1, dtype=torch.long) * example_targets[i]\n",
+        "\n",
+        "  # Run the model\n",
+        "  output = model(x)\n",
+        "  # Compute the loss\n",
+        "  loss = F.nll_loss(output, y)\n",
+        "  # Back propagate\n",
+        "  loss.backward()\n",
+        "\n",
+        "  # Collect ``datagrad``\n",
+        "  dLdx = x.grad.data\n",
+        "\n",
+        "  # Call FGSM Attack\n",
+        "  x_prime = fgsm_attack(x, epsilon, dLdx)\n",
+        "\n",
+        "  # Re-classify the perturbed image\n",
+        "  output_prime = model(x_prime)\n",
+        "\n",
+        "  x = x.detach().numpy()\n",
+        "  fig = plt.figure()\n",
+        "  plt.subplot(1,2,1)\n",
+        "  plt.tight_layout()\n",
+        "  plt.imshow(x[0][0], cmap='gray', interpolation='none')\n",
+        "  plt.title(\"Original Prediction: {}\".format(\n",
+        "    output.data.max(1, keepdim=True)[1][0].item()))\n",
+        "  plt.xticks([])\n",
+        "  plt.yticks([])\n",
+        "\n",
+        "  plt.subplot(1,2,2)\n",
+        "  plt.tight_layout()\n",
+        "  plt.imshow(x_prime[0][0].detach().numpy(), cmap='gray', interpolation='none')\n",
+        "  plt.title(\"Perturbed Prediction: {}\".format(\n",
+        "    output_prime.data.max(1, keepdim=True)[1][0].item()))\n",
+        "  plt.xticks([])\n",
+        "  plt.yticks([])\n",
+        "\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "AuNTYWboufbm"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Although we have only added a small amount of noise, the model is fooled into thinking that these images come from a different class.\n",
+        "\n",
+        "TODO -- Modify the attack so that it iteratively perturbs the data. i.e., so we take a small step epsilon, then re-calculate the gradient and take another small step according to the new gradient signs."
+      ],
+      "metadata": {
+        "id": "vFXWK826HPQ8"
+      }
+    }
+  ]
+}

Notebooks/Chap21/21_1_Bias_Mitigation.ipynb

@@ -0,0 +1,441 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyNQPfTDV6PFG7Ctcl+XVNlz",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap21/21_1_Bias_Mitigation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 21.1: Bias mitigation**\n",
+        "\n",
+        "This notebook investigates a post-processing method for bias mitigation (see figure 21.2 in the book). It based on this [blog](https://www.borealisai.com/research-blogs/tutorial1-bias-and-fairness-ai/) that I wrote for Borealis AI in 2019, which itself was derirved from [this blog](https://research.google.com/bigpicture/attacking-discrimination-in-ml/) by Wattenberg, Viégas, and Hardt.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt"
+      ],
+      "metadata": {
+        "id": "yC_LpiJqZXEL"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Worked example: loans\n",
+        "\n",
+        "Consider the example of an algorithm $c=\\mbox{f}[\\mathbf{x},\\boldsymbol\\phi]$ that predicts credit rating scores $c$ for loan decisions.  There are two pools of loan applicants identified by the variable $p\\in\\{0,1\\}$ that we’ll describe as the blue and yellow populations. We assume that we are given historical data, so we know both the credit rating and whether the applicant actually defaulted on the loan ($y=0$) or\n",
+        " repaid it ($y=1$).\n",
+        "\n",
+        "We can now think of four groups of data corresponding to (i) the blue and yellow populations and (ii) whether they did or did not repay the loan. For each of these four groups we have a distribution of credit ratings (figure 1). In an ideal world, the two distributions for the yellow population would be exactly the same as those for the blue population. However, as figure 1 shows, this is clearly not the case here."
+      ],
+      "metadata": {
+        "id": "2FYo1dWGZXgg"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Class that can describe interesting curve shapes based on the input parameters\n",
+        "# Details dont' matter\n",
+        "class FreqCurve:\n",
+        "  def __init__(self, weight, mean1, mean2, sigma1, sigma2, prop):\n",
+        "    self.mean1 = mean1\n",
+        "    self.mean2 = mean2\n",
+        "    self.sigma1 = sigma1\n",
+        "    self.sigma2 = sigma2\n",
+        "    self.prop = prop\n",
+        "    self.weight = weight\n",
+        "\n",
+        "  def freq(self, x):\n",
+        "    return self.weight * self.prop * np.exp(-0.5 * (x-self.mean1) * (x-self.mean1) / (self.sigma1 * self.sigma1)) \\\n",
+        "                * 1.0 / np.sqrt(2*np.pi*self.sigma1*self.sigma1) \\\n",
+        "          + self.weight * (1-self.prop) * np.exp(-0.5 * (x-self.mean2) * (x-self.mean2) / (self.sigma2 * self.sigma2)) \\\n",
+        "                * 1.0 / np.sqrt(2*np.pi*self.sigma2*self.sigma2)\n"
+      ],
+      "metadata": {
+        "id": "O_0gGH9hZcjo"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "credit_scores = np.arange(-4,4,0.01)\n",
+        "freq_y0_p0 = FreqCurve(800, -1.5, -2.5, 0.8, 0.6, 0.6).freq(credit_scores)\n",
+        "freq_y1_p0 = FreqCurve(500, 0.1, 0.7, 1.5, 0.8, 0.4 ).freq(credit_scores)\n",
+        "freq_y0_p1 = FreqCurve(400, 0.2, -0.1, 0.8, 0.6, 0.3).freq(credit_scores)\n",
+        "freq_y1_p1 = FreqCurve(650, 0.6, 1.6, 1.2, 0.7, 0.6 ).freq(credit_scores)\n"
+      ],
+      "metadata": {
+        "id": "Bkp7vffBbrNW"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "\n",
+        "fig = plt.figure\n",
+        "ax = plt.subplot(2,2,1)\n",
+        "plt.tight_layout()\n",
+        "ax.plot(credit_scores, freq_y0_p0, 'b--', label='y=0 (defaulted)')\n",
+        "ax.plot(credit_scores, freq_y1_p0, 'b-', label='y=1 (repaid)')\n",
+        "ax.set_xlim(-4,4)\n",
+        "ax.set_ylim(0,500)\n",
+        "ax.set_xlabel('Credit score, $c$')\n",
+        "ax.set_ylabel('Frequency')\n",
+        "ax.set_title('Population p=0')\n",
+        "ax.legend()\n",
+        "\n",
+        "ax = plt.subplot(2,2,2)\n",
+        "plt.tight_layout()\n",
+        "ax.plot(credit_scores, freq_y0_p1, 'y--', label='y=0 (defaulted)')\n",
+        "ax.plot(credit_scores, freq_y1_p1, 'y-', label='y=1 (repaid)')\n",
+        "ax.set_xlim(-4,4)\n",
+        "ax.set_ylim(0,500)\n",
+        "ax.set_xlabel('Credit score, $c$')\n",
+        "ax.set_ylabel('Frequency')\n",
+        "ax.set_title('Population p=1')\n",
+        "ax.legend()\n",
+        "\n",
+        "plt.show()"
+      ],
+      "metadata": {
+        "id": "Jf7uqyRyhVdS"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Why might the distributions for blue and yellow populations be different? It could be that the behaviour of the populations is identical, but the credit rating algorithm is biased; it may favor one population over another or simply be more noisy for one group. Alternatively, it could be that that the populations genuinely behave differently. In practice, the differences in blue and yellow distributions are probably attributable to a combination of these factors.\n",
+        "\n",
+        "Let’s assume that we can’t retrain the credit score prediction algorithm; our job is to adjudicate whether each individual is refused the loan ($\\hat{y}=0$)\n",
+        " or granted it ($\\hat{y}=1$). Since we only have the credit score\n",
+        " to go on, the best we can do is to assign different thresholds $\\tau_{1}$\n",
+        " and $\\tau_{2}$\n",
+        " for the blue and yellow populations so that the loan is granted if the credit score $c$ generated by the model exceeds $\\tau_0$ for the blue population and $\\tau_1$ for the yellow population."
+      ],
+      "metadata": {
+        "id": "CfZ-srQtmff2"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's investiate how to set these thresholds to fulfil different criteria."
+      ],
+      "metadata": {
+        "id": "569oU1OtoFz8"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Blindness to protected attribute\n",
+        "\n",
+        "We'll first do the simplest possible thing.  We'll choose the same threshold for both blue and yellow populations so that $\\tau_0$ = $\\tau_1$.  Basically, we'll ingore what we know about the group membership.  Let's see what the ramifications of that."
+      ],
+      "metadata": {
+        "id": "bE7yPyuWoSUy"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Set the thresholds\n",
+        "tau0 = tau1 = 0.0"
+      ],
+      "metadata": {
+        "id": "WIG8I-LvoFBY"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def compute_probability_get_loan(credit_scores, frequencies, threshold):\n",
+        "  # TODO - Write this function\n",
+        "  # Return the probability that somemone from this group loan based on the frequencies of each\n",
+        "  # credit score for this group\n",
+        "  # Replace this line:\n",
+        "  prob = 0.5\n",
+        "\n",
+        "\n",
+        "  return prob"
+      ],
+      "metadata": {
+        "id": "2EvkCvVBiCBn"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "First let's see what the overall probability of getting the loan is for the yellow and blue populations."
+      ],
+      "metadata": {
+        "id": "AGT40q6_qfpv"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "pr_get_loan_p0 = compute_probability_get_loan(credit_scores, freq_y0_p0+freq_y1_p0, tau0)\n",
+        "pr_get_loan_p1 = compute_probability_get_loan(credit_scores, freq_y0_p1+freq_y1_p1, tau1)\n",
+        "print(\"Probability blue group gets loan = %3.3f\"%(pr_get_loan_p0))\n",
+        "print(\"Probability yellow group gets loan = %3.3f\"%(pr_get_loan_p1))"
+      ],
+      "metadata": {
+        "id": "4nI-PR_wqWj6"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Now let's plot a receiver operating characteristic (ROC) curve.  This shows the rate of true positives $Pr(\\hat{y}=1|y=1)$ (people who got loan and paid it back) and false alarms $Pr(\\hat{y}=1|y=0)$ (people who got the loan but didn't pay it back) for all possible thresholds."
+      ],
+      "metadata": {
+        "id": "G2pEa6h6sIyu"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def plot_roc(credit_scores, freq_y0_p0, freq_y1_p0, freq_y0_p1, freq_y1_p1, tau0, tau1):\n",
+        "  true_positives_p0 = np.zeros_like(credit_scores)\n",
+        "  false_alarms_p0 = np.zeros_like(credit_scores)\n",
+        "  true_positives_p1 = np.zeros_like(credit_scores)\n",
+        "  false_alarms_p1 = np.zeros_like(credit_scores)\n",
+        "  for i in range(len(credit_scores)):\n",
+        "    true_positives_p0[i] = compute_probability_get_loan(credit_scores, freq_y1_p0, credit_scores[i])\n",
+        "    true_positives_p1[i] = compute_probability_get_loan(credit_scores, freq_y1_p1, credit_scores[i])\n",
+        "    false_alarms_p0[i] = compute_probability_get_loan(credit_scores, freq_y0_p0, credit_scores[i])\n",
+        "    false_alarms_p1[i] = compute_probability_get_loan(credit_scores, freq_y0_p1, credit_scores[i])\n",
+        "\n",
+        "  true_positives_p0_tau0 = compute_probability_get_loan(credit_scores, freq_y1_p0, tau0)\n",
+        "  true_positives_p1_tau1 = compute_probability_get_loan(credit_scores, freq_y1_p1, tau1)\n",
+        "  false_alarms_p0_tau0 = compute_probability_get_loan(credit_scores, freq_y0_p0, tau0)\n",
+        "  false_alarms_p1_tau1 = compute_probability_get_loan(credit_scores, freq_y0_p1, tau1)\n",
+        "\n",
+        "  fig, ax = plt.subplots()\n",
+        "  ax.plot(false_alarms_p0, true_positives_p0, 'b-')\n",
+        "  ax.plot(false_alarms_p1, true_positives_p1, 'y-')\n",
+        "  ax.plot(false_alarms_p0_tau0, true_positives_p0_tau0,'bo')\n",
+        "  ax.plot(false_alarms_p1_tau1, true_positives_p1_tau1,'yo')\n",
+        "  ax.set_xlim(0,1)\n",
+        "  ax.set_ylim(0,1)\n",
+        "  ax.set_xlabel('False alarms $Pr(\\hat{y}=1|y=0)$')\n",
+        "  ax.set_ylabel('True positives $Pr(\\hat{y}=1|y=1)$')\n",
+        "  ax.set_aspect('equal')\n",
+        "\n",
+        "  plt.show()"
+      ],
+      "metadata": {
+        "id": "2C7kNt3hqwiu"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "plot_roc(credit_scores, freq_y0_p0, freq_y1_p0, freq_y0_p1, freq_y1_p1, tau0, tau1)"
+      ],
+      "metadata": {
+        "id": "h3OOQeTsv8uS"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "On this plot, the true positive and false alarm rate for the particular thresholds ($\\tau_0=\\tau_{1}=0$) that we chose are indicated by the circles.\n",
+        "\n",
+        "This criterion is clearly not great.  The blue and yellow groups get given loans at different rates overall, and (for this threshold), the false alarms and true positives are also different, so it's not even fair when we consider whether the loans really were paid back.  \n",
+        "\n",
+        "TODO -- investigate setting a different threshols $\\tau_{0}=\\tau_{1}$.  Is it possible to make the overall rates that loans are given the same?  Is it possible to make the false alarm rates the same?  Is it possible to make the true positive rates the same?"
+      ],
+      "metadata": {
+        "id": "UCObTsa57uuC"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Equality of odds\n",
+        "\n",
+        "This definition of fairness proposes that the false positive and true positive rates should be the same for both populations. This also sounds reasonable, but the ROC curve shows that it is not possible for this example. There is no combination of thresholds that can achieve this because the ROC curves do not intersect. Even if they did, we would be stuck giving loans based on the particular false positive and true positive rates at the intersection which might not be desirable."
+      ],
+      "metadata": {
+        "id": "Yhrxv5AQ-PWA"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Demographic parity\n",
+        "\n",
+        "The thresholds can be chosen so that the same proportion of each group are classified as $\\hat{y}=1$ and given loans. We make an equal number of loans to each group despite the different tendencies of each to repay. This has the disadvantage that the true positive and false positive rates might be completely different in different populations. From the perspective of the lender, it is desirable to give loans in proportion to people’s ability to pay them back. From the perspective of an individual in a more reliable group, it may seem unfair that the other group gets offered the same number of loans despite the fact they are less reliable."
+      ],
+      "metadata": {
+        "id": "l6yb8vjX-gdi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# TO DO -- try to change the two thresholds so the overall probability of getting the loan is 0.6 for each group\n",
+        "# Change the values in these lines\n",
+        "tau0 = 0.3\n",
+        "tau1 = -0.1\n",
+        "\n",
+        "\n",
+        "\n",
+        "# Compute overall probability of getting loan\n",
+        "pr_get_loan_p0 = compute_probability_get_loan(credit_scores, freq_y0_p0+freq_y1_p0, tau0)\n",
+        "pr_get_loan_p1 = compute_probability_get_loan(credit_scores, freq_y0_p1+freq_y1_p1, tau1)\n",
+        "print(\"Probability blue group gets loan = %3.3f\"%(pr_get_loan_p0))\n",
+        "print(\"Probability yellow group gets loan = %3.3f\"%(pr_get_loan_p1))"
+      ],
+      "metadata": {
+        "id": "syjZ2fn5wC9-"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "This is good, because now both groups get roughly the same amount of loans.  But hold on... let's look at the ROC curve:"
+      ],
+      "metadata": {
+        "id": "5QrtvZZlHCJy"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "plot_roc(credit_scores, freq_y0_p0, freq_y1_p0, freq_y0_p1, freq_y1_p1, tau0, tau1)"
+      ],
+      "metadata": {
+        "id": "VApyl_58GUQb"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "The blue dot is waaay above the yellow dot. The proportion of people who are given a load and do pay it back from the blue population is much higher than that from the yellow population.  From another perspective, that's unfair... it seems like the yellow population are 'allowed' to default more often than the blue. This leads to another possibility."
+      ],
+      "metadata": {
+        "id": "_GgX_b6yIE4W"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Equal opportunity:\n",
+        "\n",
+        "The thresholds are chosen so that so that the true positive rate is is the same for both population. Of the people who pay back the loan, the same proportion are offered credit in each group. In terms of the two ROC curves, it means choosing thresholds so that the vertical position on each curve is the same without regard for the horizontal position."
+      ],
+      "metadata": {
+        "id": "WDnaqetXHhlv"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# TO DO -- try to change the two thresholds so the true positive are 0.8 for each group\n",
+        "# Change the values in these lines so that both points on the curves have a height of 0.8\n",
+        "tau0 = -0.1\n",
+        "tau1 = -0.7\n",
+        "\n",
+        "\n",
+        "plot_roc(credit_scores, freq_y0_p0, freq_y1_p0, freq_y0_p1, freq_y1_p1, tau0, tau1)"
+      ],
+      "metadata": {
+        "id": "zEN6HGJ7HJAZ"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "This seems fair -- people who are given loans default at the same rate (20%) for both groups.  But hold on... let's look at the overall loan rate between the two populations:"
+      ],
+      "metadata": {
+        "id": "JsyW0pBGJ24b"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Compute overall probability of getting loan\n",
+        "pr_get_loan_p0 = compute_probability_get_loan(credit_scores, freq_y0_p0+freq_y1_p0, tau0)\n",
+        "pr_get_loan_p1 = compute_probability_get_loan(credit_scores, freq_y0_p1+freq_y1_p1, tau1)\n",
+        "print(\"Probability blue group gets loan = %3.3f\"%(pr_get_loan_p0))\n",
+        "print(\"Probability yellow group gets loan = %3.3f\"%(pr_get_loan_p1))"
+      ],
+      "metadata": {
+        "id": "2a5PXHeNJDjg"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "The conclusion from all this is that (i) definitions of fairness are quite subtle and (ii) it's not possible to satisfy them all simultaneously."
+      ],
+      "metadata": {
+        "id": "tZTM7N6jKC7q"
+      }
+    }
+  ]
+}

Notebooks/Chap21/21_2_Explainability.ipynb

@@ -0,0 +1,412 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyOLMPuSWpvv8BfyPV36RuJP",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap21/21_2_Explainability.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# **Notebook 21.2: Explainability**\n",
+        "\n",
+        "This notebook investigates the LIME explainability method as depicted in figure 21.3 of the book.\n",
+        "\n",
+        "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
+        "\n",
+        "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
+      ],
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "import numpy.matlib\n",
+        "from matplotlib.colors import ListedColormap"
+      ],
+      "metadata": {
+        "id": "yC_LpiJqZXEL"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "First we'll build a black box model for predicting a credit score.  This simulates a neural network.  It takes four inputs $x1,x2,x3,x4$ in a column vector and it returns a value $y$.  Let's assume that if the output $y$ is greater than 0 then you get the loan, and if the output is less than 0 then you don't get the zone."
+      ],
+      "metadata": {
+        "id": "WM6mq9KNit3j"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Details of this not important -- a hacky thing that takes four inputs and returns\n",
+        "# a scalar output\n",
+        "class BlackBoxModel:\n",
+        "  def __init__(self):\n",
+        "    self.n_dim = 4\n",
+        "    self.n_points = 10\n",
+        "    self.means = np.random.uniform(size=(self.n_dim, self.n_points))\n",
+        "    self.stds = np.random.uniform(size=(self.n_dim,self.n_points))+0.1\n",
+        "    self.values = np.random.normal(size=(self.n_points))/10\n",
+        "    self.values = self.values - np.mean(self.values)\n",
+        "\n",
+        "\n",
+        "  def intensity(self, x, mean, std, value):\n",
+        "\n",
+        "    dist = (x-np.matlib.repmat(mean,1,x.shape[1])) / np.matlib.repmat(std,1,x.shape[1])\n",
+        "    out = value * np.exp(-np.sum(dist*dist,axis=0))\n",
+        "    out = out[None,:]\n",
+        "    return out\n",
+        "\n",
+        "\n",
+        "  def get_output(self,x):\n",
+        "    y = np.zeros((1,x.shape[1]))\n",
+        "    t_vals = np.arange(0, self.n_points-1, 0.01)\n",
+        "    for t in t_vals:\n",
+        "      i = np.floor(t)\n",
+        "      prop = t-i\n",
+        "      i = int(i)\n",
+        "      y = y+ prop  * self.intensity(x, self.means[:,[i]], self.stds[:,[i]], self.values[i])\n",
+        "      y = y+ (1-prop) * self.intensity(x,self.means[:,[i+1]], self.stds[:,[i+1]], self.values[i+1])\n",
+        "      y = np.clip(y,-10,10)\n",
+        "    return y"
+      ],
+      "metadata": {
+        "id": "rt4FS42dIa9_"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Code to draw 2D slide through the four dimensional function\n",
+        "# Again, you don't really need to read this.\n",
+        "def draw_2D_slice(model, dim1, dim2, first_other_dim_value = 0.5, second_other_dim_value = 0.6):\n",
+        "\n",
+        "    #Create pretty colormap as in book\n",
+        "    my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
+        "    my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
+        "    r = np.floor(my_colormap_vals_dec/(256*256))\n",
+        "    g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
+        "    b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
+        "    my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
+        "    my_colormap = ListedColormap(my_colormap_vals)\n",
+        "\n",
+        "    x1_vals = np.arange(0.0, 1.0, 0.01)\n",
+        "    x2_vals = np.arange(0.0, 1.0, 0.01)\n",
+        "    x1_mesh, x2_mesh = np.meshgrid(x1_vals,x2_vals)\n",
+        "    n_vals = x1_mesh.shape[0]\n",
+        "\n",
+        "    x1 = np.reshape(x1_mesh,(1,n_vals*n_vals))\n",
+        "    x2 = np.reshape(x2_mesh,(1,n_vals*n_vals))\n",
+        "\n",
+        "    x = np.ones((4,n_vals*n_vals))\n",
+        "    x[dim1,:] = x1\n",
+        "    x[dim2,:] = x2\n",
+        "    if((dim1==0 and dim2 ==1) or (dim1==1 and dim2 ==0)):\n",
+        "      x[2,:] = x[2,:] * first_other_dim_value\n",
+        "      x[3,:] = x[3,:] * second_other_dim_value\n",
+        "      message = \"$x_{2}$ = %3.3f, $x_3$=%3.3f\"%(first_other_dim_value, second_other_dim_value)\n",
+        "    if((dim1==0 and dim2 ==2) or (dim1==2 and dim2 ==0)):\n",
+        "      x[1,:] = x[1,:] * first_other_dim_value\n",
+        "      x[3,:] = x[3,:] * second_other_dim_value\n",
+        "      message = \"$x_{1}$ = %3.3f, $x_3$=%3.3f\"%(first_other_dim_value, second_other_dim_value)\n",
+        "    if((dim1==0 and dim2 ==3) or (dim1==3 and dim2 ==0)):\n",
+        "      x[1,:] = x[1,:] * first_other_dim_value\n",
+        "      x[2,:] = x[2,:] * second_other_dim_value\n",
+        "      message = \"$x_{1}$ = %3.3f, $x_2$=%3.3f\"%(first_other_dim_value, second_other_dim_value)\n",
+        "    if((dim1==1 and dim2 ==2) or (dim1==2 and dim2 ==1)):\n",
+        "      x[0,:] = x[0,:] * first_other_dim_value\n",
+        "      x[3,:] = x[3,:] * second_other_dim_value\n",
+        "      message = \"$x_{0}$ = %3.3f, $x_3$=%3.3f\"%(first_other_dim_value, second_other_dim_value)\n",
+        "    if((dim1==1 and dim2 ==3) or (dim1==3 and dim2 ==1)):\n",
+        "      x[0,:] = x[0,:] * first_other_dim_value\n",
+        "      x[2,:] = x[2,:] * second_other_dim_value\n",
+        "      message = \"$x_{0}$ = %3.3f, $x_2$=%3.3f\"%(first_other_dim_value, second_other_dim_value)\n",
+        "    if((dim1==2 and dim2 ==3) or (dim1==3 and dim2 ==2)):\n",
+        "      x[0,:] = x[0,:] * first_other_dim_value\n",
+        "      x[1,:] = x[1,:] * second_other_dim_value\n",
+        "      message = \"$x_{0}$ = %3.3f, $x_1$=%3.3f\"%(first_other_dim_value, second_other_dim_value)\n",
+        "\n",
+        "    y = model.get_output(x)\n",
+        "    y[0,0] = -10; y[0,1]=10 # Hack the first two values so we see whole range of colormap\n",
+        "    y_mesh = np.reshape(y,(n_vals, n_vals))\n",
+        "\n",
+        "\n",
+        "    fig, ax = plt.subplots()\n",
+        "    fig.set_size_inches(7,7)\n",
+        "    pos = ax.contourf(x1_mesh, x2_mesh, y_mesh, levels=256 ,cmap = my_colormap, vmin=-10,vmax=10.0)\n",
+        "    ax.set_xlabel('Dimension x%d'%dim1);ax.set_ylabel('Dimension x%d'%dim2)\n",
+        "    ax.set_title(message)\n",
+        "    levels = np.array([0])\n",
+        "    ax.contour(x1_mesh, x2_mesh, y_mesh, levels, cmap=my_colormap)\n",
+        "    cb = fig.colorbar(pos)\n",
+        "    cb.set_ticks((-10,-5,0,5,10))\n",
+        "    plt.show()"
+      ],
+      "metadata": {
+        "id": "g0sosSU4RdU3"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Create an instance of our model"
+      ],
+      "metadata": {
+        "id": "RlHjBpcyjcw4"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "np.random.seed(3)\n",
+        "model = BlackBoxModel()"
+      ],
+      "metadata": {
+        "id": "-JXgQD4oT3J1"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "The four inputs to the model might represent the four inputs measures of our debt, age, income etc, and the output represents the credit score.\n",
+        "\n",
+        "As a responsible model owner, we want to understand our model and make sure that it is doing something sensible.  \n",
+        "\n",
+        "Unfortunately, the model describes a four dimensional function, which makes it really hard to understand (and imagine, that there could easily be hundreds of input in a real model).\n",
+        "\n",
+        "One thing that we can do it look at the effect of two of the inputs at one time.  For example, we can look at how inputs 0 and 1 change when we fix dimension 2 to 0.2 and dimension 3 to 0.9.   The black line represents the decision boundary (where the model predicts a credit score of zero).  If we are on the wrong side of this boundary, then our loan is refused."
+      ],
+      "metadata": {
+        "id": "6LxuB6p3k-VM"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "draw_2D_slice(model,0,1,0.2,0.9)"
+      ],
+      "metadata": {
+        "id": "NblKr3W0dBJJ"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Similarly, we could look at how inputs 1 and 3 change the input when we set input 0 to 0.3 and input 2 to 0.2:"
+      ],
+      "metadata": {
+        "id": "Fp2GeFn5mIRW"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "draw_2D_slice(model,1,3,0.3,0.2)"
+      ],
+      "metadata": {
+        "id": "VzIe0py5d5Bk"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "This tells us something -- it might be good for a reality check if we had some expectations about what effect each input would have, but it's still hard to ensure that the model does something sensible everywhere, especially for models where there are thousands of inputs.  Unfortunately, there are basically no good solutions to this problem at the time of writing."
+      ],
+      "metadata": {
+        "id": "A9XUV9B6m7v0"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "However, let's view this from the perspective of a customer.  We can assume that the four inputs have some particular values, and see what the output is."
+      ],
+      "metadata": {
+        "id": "1w968kJQjjUm"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "x = np.array([[0.3],[0.8],[0.6],[0.3]])\n",
+        "y = model.get_output(x)\n",
+        "print(\"Your credit score is %3.3f\"%(y))\n",
+        "print(\"Sorry,  your loan is refused\")"
+      ],
+      "metadata": {
+        "id": "Nr71IahkjfV3"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Well, that is bad news.  Why was our loan refused?  We'd like to understand what we could do to improve our credit score.   One way to do this is through individual conditional expectation or ICE plots ([Goldstein et al. 2015](https://arxiv.org/abs/1309.6392)). These take shows how the model output would change as we vary a single feature. Essentially, they answer the question: what if the $k^{th}$ feature had taken another value?"
+      ],
+      "metadata": {
+        "id": "mafmi3dSkTuf"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def ice_plot(model, x, k):\n",
+        "  # Get output for the input\n",
+        "  y = model.get_output(x)\n",
+        "\n",
+        "  # Possible values of the k'th dimension of the input\n",
+        "  x_k_all = np.arange(0,1,0.001)\n",
+        "  # TODO write code that varies the k'th dimension of x and runs the model on the result to create a series of outputs y\n",
+        "  # Replace this line\n",
+        "  y_all = np.zeros_like(x_k_all)\n",
+        "\n",
+        "\n",
+        "\n",
+        "  fig, ax = plt.subplots()\n",
+        "  ax.plot(x_k_all, np.squeeze(y_all), 'r-')\n",
+        "  ax.plot(x[k],y,'ro') ;\n",
+        "  ax.plot([0,1.0],[0.0,0.0],'k--')\n",
+        "  ax.set_xlabel('Dimension x%d'%(k))\n",
+        "  ax.set_ylabel('Credit score')\n",
+        "  ax.set_xlim(0,1)\n",
+        "  ax.set_ylim([-10,10])\n",
+        "\n",
+        "  plt.show()\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "v2nNsvW-m2fb"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "ice_plot(model, x, 0)\n",
+        "ice_plot(model, x, 1)\n",
+        "ice_plot(model, x, 2)\n",
+        "ice_plot(model, x, 3)"
+      ],
+      "metadata": {
+        "id": "5I91gzfSnL9N"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "We can learn something from this.  For example, decreasing the value of $x_{3}$ would be the most effective way to increase our credit score.  However, this might be impossible if it is a variable we can't control like our age.  Perhaps decreasing $x_0$ and $x_{1}$ might improve it further.  Well... perhaps, but we don't know what is going on in our model;  it might also make things much worse.\n",
+        "\n",
+        "Local interpretable model-agnostic explanations or LIME ([Ribeiro et al. 2016](https://arxiv.org/abs/1602.04938)) approximate the main machine learning model locally around a given input using a simpler model that is easier to understand.  \n",
+        "\n",
+        "The principle is simple.  First, we sample some points $\\mathbf{x}_{i}$ close to  the input $\\mathbf{x}$ that we are interested in.  Then we find the outputs $\\mathbf{y}_i$ that correspond to those inputs.  Now we have a training set, and we can train any other kind of model that explains this small area of the input space. This can be a model that is much more interpretable and easier to understand such as a linear model or a tree."
+      ],
+      "metadata": {
+        "id": "RmjuAR7HojtR"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# TODO -- Choose 100 points where each element of x is perturbed by noise sampled from a uniform distribution\n",
+        "# that takes values between [-0.05 and 0.05].  Then run these points through the model.\n",
+        "# Replace these lines\n",
+        "x_lime_train = np.matlib.repmat(x, 1, 100)\n",
+        "y_lime_train = np.ones((1,100))\n",
+        "\n",
+        "# BEGIN_ANSWER\n",
+        "x_lime_train = x_lime_train + np.random.uniform(low=-0.05,high=0.05,size=x_lime_train.shape)\n",
+        "y_lime_train = model.get_output(x_lime_train)\n",
+        "# END_ANSWER"
+      ],
+      "metadata": {
+        "id": "-GprSftsnS0M"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "We'll train a linear model:\n",
+        "\n",
+        "\\begin{equation}\n",
+        "y = \\beta_0 + \\boldsymbol\\phi^{T}\\mathbf{x}\n",
+        "\\end{equation}"
+      ],
+      "metadata": {
+        "id": "yTFDYbqGqmcA"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# TODO -- train this model using a least squares loss\n",
+        "# to find values for the offset \\beta_0 and the four slopes in \\phi\n",
+        "# One way to do this is with sklearn.linear_model\n",
+        "# Replace this line\n",
+        "beta = 0; phi = np.zeros((1,4))\n",
+        "\n",
+        "print(phi)"
+      ],
+      "metadata": {
+        "id": "5e4VPh40qlEl"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "This model is easilly intepretable.  The k'th coeffeicient tells us the how much (and in which direction) changing the value of the k'th input will change the output.  This is only valid in the vicinity of the input $x$.\n",
+        "\n",
+        "Note that a more sophisticated version of LIME would weight the training points according to how close they are to the original data point of interest."
+      ],
+      "metadata": {
+        "id": "hsZHWuVWtzIK"
+      }
+    }
+  ]
+}

index.html

@@ -1,24 +1,31 @@
-<h1>Understanding Deep Learning</h1>
-by Simon J.D. Prince
-<br>
-To be published by MIT Press Dec 5th 2023.<br>
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <title>udlbook</title>
+    <link rel="stylesheet" href="style.css">
+</head>
 
-<img src="https://raw.githubusercontent.com/udlbook/udlbook/main/UDLCoverSmall.jpg" alt="front cover">
-
-<h2> Download draft PDF </h2>
-
-<a href="https://github.com/udlbook/udlbook/releases/download/v1.1.3/UnderstandingDeepLearning_01_10_23_C.pdf">Draft PDF Chapters 1-21</a><br> 2023-10-01. CC-BY-NC-ND license
-<br>
+<body>
+<div id="head">
+    <div>
+        <h1 style="margin: 0; font-size: 36px">Understanding Deep Learning</h1>
+        by Simon J.D. Prince
+        <br>To be published by MIT Press Dec 5th 2023.<br>
+        <ul>
+            <li>
+                <p style="font-size: larger; margin-bottom: 0">Download draft PDF Chapters 1-21 <a
+                        href="https://github.com/udlbook/udlbook/releases/download/v1.15/UnderstandingDeepLearning_23_10_23_C.pdf">here</a>
+                </p>2023-23-23. CC-BY-NC-ND license<br>
                 <img src="https://img.shields.io/github/downloads/udlbook/udlbook/total" alt="download stats shield">
-<br>
-<ul>
-	<li> Appendices and notebooks coming soon
-	<li> Report errata via <a href="https://github.com/udlbook/udlbook/issues">github</a> or contact me directly at udlbookmail@gmail.com
-	<li> Follow me on <a href="https://twitter.com/SimonPrinceAI">Twitter</a> or <a href="https://www.linkedin.com/in/simon-prince-615bb9165/">LinkedIn</a> for updates.
-</ul>
-
-<h2>Table of contents</h2>
-<ul> 
+            </li>
+            <li> Report errata via <a href="https://github.com/udlbook/udlbook/issues">github</a>
+                or contact me directly at udlbookmail@gmail.com
+            <li> Follow me on <a href="https://twitter.com/SimonPrinceAI">Twitter</a> or <a
+                    href="https://www.linkedin.com/in/simon-prince-615bb9165/">LinkedIn</a> for updates.
+        </ul>
+        <h2>Table of contents</h2>
+        <ul>
             <li> Chapter 1 - Introduction
             <li> Chapter 2 - Supervised learning
             <li> Chapter 3 - Shallow neural networks
@@ -40,122 +47,331 @@ To be published by MIT Press Dec 5th 2023.<br>
             <li> Chapter 19 - Deep reinforcement learning
             <li> Chapter 20 - Why does deep learning work?
             <li> Chapter 21 - Deep learning and ethics
-</ul>
+        </ul>
+    </div>
+    <div id="cover">
+        <img src="https://raw.githubusercontent.com/udlbook/udlbook/main/UDLCoverSmall.jpg"
+             alt="front cover">
+    </div>
+</div>
+<div id="body">
+    <h2>Resources for instructors </h2>
+    <p>Instructor answer booklet available with proof of credentials via <a
+            href="https://mitpress.mit.edu/9780262048644/understanding-deep-learning"> MIT Press</a>.</p>
+    <p>Request an exam/desk copy via <a href="https://mitpress.ublish.com/request?cri=15055">MIT Press</a>.</p>
+    <p>Figures in PDF (vector) / SVG (vector) / Powerpoint (images):
+    <ul>
+        <li> Chapter 1 - Introduction: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap1PDF.zip">PDF
+            Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1udnl5pUOAc8DcAQ7HQwyzP9pwL95ynnv">
+            SVG
+            Figures</a> / <a
+                href="https://docs.google.com/presentation/d/1IjTqIUvWCJc71b5vEJYte-Dwujcp7rvG/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+            Figures</a>
+        <li> Chapter 2 - Supervised learning: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap2PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1VSxcU5y1qNFlmd3Lb3uOWyzILuOj1Dla"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1Br7R01ROtRWPlNhC_KOommeHAWMBpWtz/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+                Figures</a>
+        <li> Chapter 3 - Shallow neural networks: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap3PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=19kZFWlXhzN82Zx02ByMmSZOO4T41fmqI"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1e9M3jB5I9qZ4dCBY90Q3Hwft_i068QVQ/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+                Figures</a>
+        <li> Chapter 4 - Deep neural networks: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap4PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1ojr0ebsOhzvS04ItAflX2cVmYqHQHZUa"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1LTSsmY4mMrJbqXVvoTOCkQwHrRKoYnJj/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+                Figures</a>
+        <li> Chapter 5 - Loss functions: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap5PDF.zip">PDF
+            Figures</a> / <a href="https://drive.google.com/uc?export=download&id=17MJO7fiMpFZVqKeqXTbQ36AMpmR4GizZ">
+            SVG
+            Figures</a> / <a
+                href="https://docs.google.com/presentation/d/1gcpC_3z9oRp87eMkoco-kdLD-MM54Puk/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+            Figures</a>
+        <li> Chapter 6 - Training models: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap6PDF.zip">PDF
+            Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1VPdhFRnCr9_idTrX0UdHKGAw2shUuwhK">
+            SVG
+            Figures</a> / <a
+                href="https://docs.google.com/presentation/d/1AKoeggAFBl9yLC7X5tushAGzCCxmB7EY/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+            Figures</a>
+        <li> Chapter 7 - Gradients and initialization: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap7PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1TTl4gvrTvNbegnml4CoGoKOOd6O8-PGs"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/11zhB6PI-Dp6Ogmr4IcI6fbvbqNqLyYcz/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+                Figures</a>
+        <li> Chapter 8 - Measuring performance: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap8PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=19eQOnygd_l0DzgtJxXuYnWa4z7QKJrJx"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1SHRmJscDLUuQrG7tmysnScb3ZUAqVMZo/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+                Figures</a>
+        <li> Chapter 9 - Regularization: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap9PDF.zip">PDF
+            Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1LprgnUGL7xAM9-jlGZC9LhMPeefjY0r0">
+            SVG
+            Figures</a> / <a
+                href="https://docs.google.com/presentation/d/1VwIfvjpdfTny6sEfu4ZETwCnw6m8Eg-5/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+            Figures</a>
+        <li> Chapter 10 - Convolutional networks: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap10PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1-Wb3VzaSvVeRzoUzJbI2JjZE0uwqupM9"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1MtfKBC4Y9hWwGqeP6DVwUNbi1j5ncQCg/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+                Figures</a>
+        <li> Chapter 11 - Residual networks: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap11PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1Mr58jzEVseUAfNYbGWCQyDtEDwvfHRi1"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1saY8Faz0KTKAAifUrbkQdLA2qkyEjOPI/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+                Figures</a>
+        <li> Chapter 12 - Transformers: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap12PDF.zip">PDF
+            Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1txzOVNf8-jH4UfJ6SLnrtOfPd1Q3ebzd">
+            SVG
+            Figures</a> / <a
+                href="https://docs.google.com/presentation/d/1GVNvYWa0WJA6oKg89qZre-UZEhABfm0l/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+            Figures</a>
+        <li> Chapter 13 - Graph neural networks: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap13PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1lQIV6nRp6LVfaMgpGFhuwEXG-lTEaAwe"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1YwF3U82c1mQ74c1WqHVTzLZ0j7GgKaWP/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+                Figures</a>
+        <li> Chapter 14 - Unsupervised learning: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap14PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1aMbI6iCuUvOywqk5pBOmppJu1L1anqsM"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1A-lBGv3NHl4L32NvfFgy1EKeSwY-0UeB/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">
+                PowerPoint Figures</a>
+        <li> Chapter 15 - Generative adversarial networks: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap15PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1EErnlZCOlXc3HK7m83T2Jh_0NzIUHvtL"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/10Ernk41ShOTf4IYkMD-l4dJfKATkXH4w/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+                Figures</a>
+        <li> Chapter 16 - Normalizing flows: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap16PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1B9bxtmdugwtg-b7Y4AdQKAIEVWxjx8l3"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1nLLzqb9pdfF_h6i1HUDSyp7kSMIkSUUA/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+                Figures</a>
+        <li> Chapter 17 - Variational autoencoders: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap17PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1SNtNIY7khlHQYMtaOH-FosSH3kWwL4b7"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1lQE4Bu7-LgvV2VlJOt_4dQT-kusYl7Vo/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+                Figures</a>
+        <li> Chapter 18 - Diffusion models: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap18PDF.zip">PDF Figures</a> / <a
+                href="https://docs.google.com/presentation/d/1x_ufIBtVPzWUvRieKMkpw5SdRjXWwdfR/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">
+            PowerPoint Figures</a>
+        <li> Chapter 19 - Deep reinforcement learning: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap19PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1a5WUoF7jeSgwC_PVdckJi1Gny46fCqh0"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1TnYmVbFNhmMFetbjyfXGmkxp1EHauMqr/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">
+                PowerPoint Figures </a>
+        <li> Chapter 20 - Why does deep learning work?: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap20PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1M2d0DHEgddAQoIedKSDTTt7m1ZdmBLQ3"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1coxF4IsrCzDTLrNjRagHvqB_FBy10miA/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">
+                PowerPoint Figures</a>
+        <li> Chapter 21 - Deep learning and ethics: <a
+                href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap21PDF.zip">PDF Figures</a> / <a
+                href="https://drive.google.com/uc?export=download&id=1jixmFfwmZkW_UVYzcxmDcMsdFFtnZ0bU"> SVG Figures</a>/
+            <a
+                    href="https://docs.google.com/presentation/d/1EtfzanZYILvi9_-Idm28zD94I_6OrN9R/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
+                Figures</a>
+        <li> Appendices - <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLAppendixPDF.zip">PDF
+            Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1k2j7hMN40ISPSg9skFYWFL3oZT7r8v-l">
+            SVG
+            Figures</a> / <a
+                href="https://docs.google.com/presentation/d/1_2cJHRnsoQQHst0rwZssv-XH4o5SEHks/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">Powerpoint
+            Figures</a>
+    </ul>
+
+    Instructions for editing figures / equations can be found <a
+        href="https://drive.google.com/file/d/1T_MXXVR4AfyMnlEFI-UVDh--FXI5deAp/view?usp=sharing">here</a>.
+
+    <h2>Resources for students</h2>
+
+    <p>Answers to selected questions: <a
+            href="https://github.com/udlbook/udlbook/raw/main/UDL_Answer_Booklet_Students.pdf">PDF</a>
+    </p>
+    <p>Python notebooks: (Early ones more thoroughly tested than later ones!)</p>
+
+    <ul>
+        <li> Notebook 1.1 - Background mathematics: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap01/1_1_BackgroundMathematics.ipynb">ipynb/colab</a>
+        </li>
+        <li> Notebook 2.1 - Supervised learning: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap02/2_1_Supervised_Learning.ipynb">ipynb/colab</a>
+        </li>
+        <li> Notebook 3.1 - Shallow networks I: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_1_Shallow_Networks_I.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 3.2 - Shallow networks II: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_2_Shallow_Networks_II.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 3.3 - Shallow network regions: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_3_Shallow_Network_Regions.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 3.4 - Activation functions: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_4_Activation_Functions.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 4.1 - Composing networks: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap04/4_1_Composing_Networks.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 4.2 - Clipping functions: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap04/4_2_Clipping_functions.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 4.3 - Deep networks: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap04/4_3_Deep_Networks.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 5.1 - Least squares loss: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap05/5_1_Least_Squares_Loss.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 5.2 - Binary cross-entropy loss: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap05/5_2_Binary_Cross_Entropy_Loss.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 5.3 - Multiclass cross-entropy loss: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap05/5_3_Multiclass_Cross_entropy_Loss.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 6.1 - Line search: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_1_Line_Search.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 6.2 - Gradient descent: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_2_Gradient_Descent.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 6.3 - Stochastic gradient descent: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_3_Stochastic_Gradient_Descent.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 6.4 - Momentum: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_4_Momentum.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 6.5 - Adam: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_5_Adam.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 7.1 - Backpropagation in toy model: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap07/7_1_Backpropagation_in_Toy_Model.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 7.2 - Backpropagation: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap07/7_2_Backpropagation.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 7.3 - Initialization: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap07/7_3_Initialization.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 8.1 - MNIST-1D performance: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_1_MNIST_1D_Performance.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 8.2 - Bias-variance trade-off: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_2_Bias_Variance_Trade_Off.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 8.3 - Double descent: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_3_Double_Descent.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 8.4 - High-dimensional spaces: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_4_High_Dimensional_Spaces.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 9.1 - L2 regularization: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_1_L2_Regularization.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 9.2 - Implicit regularization: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_2_Implicit_Regularization.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 9.3 - Ensembling: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_3_Ensembling.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 9.4 - Bayesian approach: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_4_Bayesian_Approach.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 9.5 - Augmentation <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_5_Augmentation.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 10.1 - 1D convolution: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_1_1D_Convolution.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 10.2 - Convolution for MNIST-1D: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_2_Convolution_for_MNIST_1D.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 10.3 - 2D convolution: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_3_2D_Convolution.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 10.4 - Downsampling & upsampling: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_4_Downsampling_and_Upsampling.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 10.5 - Convolution for MNIST: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_5_Convolution_For_MNIST.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 11.1 - Shattered gradients: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap11/11_1_Shattered_Gradients.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 11.2 - Residual networks: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap11/11_2_Residual_Networks.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 11.3 - Batch normalization: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap11/11_3_Batch_Normalization.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 12.1 - Self-attention: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_1_Self_Attention.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 12.2 - Multi-head self-attention: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_2_Multihead_Self_Attention.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 12.3 - Tokenization: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_3_Tokenization.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 12.4 - Decoding strategies: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_4_Decoding_Strategies.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 13.1 - Encoding graphs: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap13/13_1_Graph_Representation.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 13.2 - Graph classification : <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap13/13_2_Graph_Classification.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 13.3 - Neighborhood sampling: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap13/13_3_Neighborhood_Sampling.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 13.4 - Graph attention: <a
+                href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap13/13_4_Graph_Attention_Networks.ipynb">ipynb/colab </a>
+        </li>
+        <li> Notebook 15.1 - GAN toy example: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap15/15_1_GAN_Toy_Example.ipynb">ipynb/colab </a></li>
+        <li> Notebook 15.2 - Wasserstein distance: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap15/15_2_Wasserstein_Distance.ipynb">ipynb/colab </a></li>
+        <li> Notebook 16.1 - 1D normalizing flows: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap16/16_1_1D_Normalizing_Flows.ipynb">ipynb/colab </a></li>
+        <li> Notebook 16.2 - Autoregressive flows: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap16/16_2_Autoregressive_Flows.ipynb">ipynb/colab </a></li>
+        <li> Notebook 16.3 - Contraction mappings: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap16/16_3_Contraction_Mappings.ipynb">ipynb/colab </a></li>
+        <li> Notebook 17.1 - Latent variable models: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap17/17_1_Latent_Variable_Models.ipynb">ipynb/colab </a></li>
+        <li> Notebook 17.2 - Reparameterization trick: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap17/17_2_Reparameterization_Trick.ipynb">ipynb/colab </a></li>
+        <li> Notebook 17.3 - Importance sampling: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap17/17_3_Importance_Sampling.ipynb">ipynb/colab </a></li>
+        <li> Notebook 18.1 - Diffusion encoder: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap18/18_1_Diffusion_Encoder.ipynb">ipynb/colab </a></li>
+        <li> Notebook 18.2 - 1D diffusion model: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap18/18_2_1D_Diffusion_Model.ipynb">ipynb/colab </a></li>
+        <li> Notebook 18.3 - Reparameterized model: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap18/18_3_Reparameterized_Model.ipynb">ipynb/colab </a></li>
+        <li> Notebook 18.4 - Families of diffusion models: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap18/18_4_Families_of_Diffusion_Models.ipynb">ipynb/colab </a></li>
+        <li> Notebook 19.1 - Markov decision processes: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap19/19_1_Markov_Decision_Processes.ipynb">ipynb/colab </a></li>
+        <li> Notebook 19.2 - Dynamic programming: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap19/19_2_Dynamic_Programming.ipynb">ipynb/colab </a></li>
+        <li> Notebook 19.3 - Monte-Carlo methods: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap19/19_3_Monte_Carlo_Methods.ipynb">ipynb/colab </a></li>
+        <li> Notebook 19.4 - Temporal difference methods: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap19/19_4_Temporal_Difference_Methods.ipynb">ipynb/colab </a></li>
+        <li> Notebook 19.5 - Control variates: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap19/19_5_Control_Variates.ipynb">ipynb/colab </a></li>
+        <li> Notebook 20.1 - Random data: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap20/20_1_Random_Data.ipynb">ipynb/colab </a></li>
+        <li> Notebook 20.2 - Full-batch gradient descent: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap20/20_2_Full_Batch_Gradient_Descent.ipynb">ipynb/colab </a></li>
+        <li> Notebook 20.3 - Lottery tickets: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap20/20_3_Lottery_Tickets.ipynb">ipynb/colab </a></li>
+        <li> Notebook 20.4 - Adversarial attacks: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap20/20_4_Adversarial_Attacks.ipynb">ipynb/colab </a></li>
+        <li> Notebook 21.1 - Bias mitigation: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap21/21_1_Bias_Mitigation.ipynb">ipynb/colab </a></li>
+        <li> Notebook 21.2 - Explainability: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap21/21_2_Explainability.ipynb">ipynb/colab </a></li>
+    </ul>
 
 
-<h2>Resources for instructors </h2>
-
-<p></p>Instructor answer booklet available with proof of credentials via <a href="https://mitpress.mit.edu/9780262048644/understanding-deep-learning"/> MIT Press</a></p>
-
-<p></p>Figures in PDF (vector) / SVG (vector) / Powerpoint (images):
-
-<ul> 																	           
-	<li>  Chapter 1 - Introduction: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap1PDF.zip">PDF Figures</a>  / <a href="https://drive.google.com/uc?export=download&id=1udnl5pUOAc8DcAQ7HQwyzP9pwL95ynnv"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1IjTqIUvWCJc71b5vEJYte-Dwujcp7rvG/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
-	<li>  Chapter 2 - Supervised learning: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap2PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1VSxcU5y1qNFlmd3Lb3uOWyzILuOj1Dla"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1Br7R01ROtRWPlNhC_KOommeHAWMBpWtz/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a> 
-	<li>  Chapter 3 - Shallow neural networks:  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap3PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=19kZFWlXhzN82Zx02ByMmSZOO4T41fmqI"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1e9M3jB5I9qZ4dCBY90Q3Hwft_i068QVQ/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a> 
-	<li>  Chapter 4 - Deep neural networks:  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap4PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1ojr0ebsOhzvS04ItAflX2cVmYqHQHZUa"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1LTSsmY4mMrJbqXVvoTOCkQwHrRKoYnJj/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a> 
-	<li>  Chapter 5 - Loss functions:  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap5PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=17MJO7fiMpFZVqKeqXTbQ36AMpmR4GizZ"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1gcpC_3z9oRp87eMkoco-kdLD-MM54Puk/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a> 
-	<li>  Chapter 6 - Training models:  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap6PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1VPdhFRnCr9_idTrX0UdHKGAw2shUuwhK"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1AKoeggAFBl9yLC7X5tushAGzCCxmB7EY/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a> 
-	<li>  Chapter 7 - Gradients and initialization:  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap7PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1TTl4gvrTvNbegnml4CoGoKOOd6O8-PGs"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/11zhB6PI-Dp6Ogmr4IcI6fbvbqNqLyYcz/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a> 
-	<li>  Chapter 8 - Measuring performance:  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap8PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=19eQOnygd_l0DzgtJxXuYnWa4z7QKJrJx"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1SHRmJscDLUuQrG7tmysnScb3ZUAqVMZo/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a> 
-	<li>  Chapter 9 - Regularization: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap9PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1LprgnUGL7xAM9-jlGZC9LhMPeefjY0r0"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1VwIfvjpdfTny6sEfu4ZETwCnw6m8Eg-5/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a> 
-	<li>  Chapter 10 - Convolutional networks: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap10PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1-Wb3VzaSvVeRzoUzJbI2JjZE0uwqupM9"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1MtfKBC4Y9hWwGqeP6DVwUNbi1j5ncQCg/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a> 
-	<li>  Chapter 11 - Residual networks: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap11PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1Mr58jzEVseUAfNYbGWCQyDtEDwvfHRi1"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1saY8Faz0KTKAAifUrbkQdLA2qkyEjOPI/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a> 
-	<li>  Chapter 12 - Transformers:  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap12PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1txzOVNf8-jH4UfJ6SLnrtOfPd1Q3ebzd"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1GVNvYWa0WJA6oKg89qZre-UZEhABfm0l/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a> 
-	<li>  Chapter 13 - Graph neural networks:  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap13PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1lQIV6nRp6LVfaMgpGFhuwEXG-lTEaAwe"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1YwF3U82c1mQ74c1WqHVTzLZ0j7GgKaWP/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a> 
-	<li>  Chapter 14 - Unsupervised learning:  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap14PDF.zip">PDF Figures</a>  / <a href="https://drive.google.com/uc?export=download&id=1aMbI6iCuUvOywqk5pBOmppJu1L1anqsM"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1A-lBGv3NHl4L32NvfFgy1EKeSwY-0UeB/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true"> Powerpoint Figures</a>
-	<li>  Chapter 15 - Generative adversarial networks:  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap15PDF.zip">PDF Figures</a>  / <a href="https://drive.google.com/uc?export=download&id=1EErnlZCOlXc3HK7m83T2Jh_0NzIUHvtL"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/10Ernk41ShOTf4IYkMD-l4dJfKATkXH4w/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
-	<li>  Chapter 16 - Normalizing flows:   <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap16PDF.zip">PDF Figures</a>  / <a href="https://drive.google.com/uc?export=download&id=1B9bxtmdugwtg-b7Y4AdQKAIEVWxjx8l3"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1nLLzqb9pdfF_h6i1HUDSyp7kSMIkSUUA/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
-	<li>  Chapter 17 - Variational autoencoders:  <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap17PDF.zip">PDF Figures</a>  / <a href="https://drive.google.com/uc?export=download&id=1SNtNIY7khlHQYMtaOH-FosSH3kWwL4b7"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1lQE4Bu7-LgvV2VlJOt_4dQT-kusYl7Vo/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
-	<li>  Chapter 18 - Diffusion models: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap18PDF.zip">PDF Figures</a>  / <a href="https://docs.google.com/presentation/d/1x_ufIBtVPzWUvRieKMkpw5SdRjXWwdfR/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true"> PowerPoint Figures</a>
-	<li>  Chapter 19 - Deep reinforcement learning: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap19PDF.zip">PDF Figures</a>  / <a href="https://drive.google.com/uc?export=download&id=1a5WUoF7jeSgwC_PVdckJi1Gny46fCqh0"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1TnYmVbFNhmMFetbjyfXGmkxp1EHauMqr/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true"> PowerPoint Figures </a> 
-	<li>  Chapter 20 - Why does deep learning work?: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap20PDF.zip">PDF Figures</a>  / <a href="https://drive.google.com/uc?export=download&id=1M2d0DHEgddAQoIedKSDTTt7m1ZdmBLQ3"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1coxF4IsrCzDTLrNjRagHvqB_FBy10miA/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true"> PowerPoint Figures</a>
-	<li>  Chapter 21 - Deep learning and ethics: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap21PDF.zip">PDF Figures</a>  / <a href="https://drive.google.com/uc?export=download&id=1jixmFfwmZkW_UVYzcxmDcMsdFFtnZ0bU"> SVG Figures</a>/ <a href="https://docs.google.com/presentation/d/1EtfzanZYILvi9_-Idm28zD94I_6OrN9R/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
-	<li>  Appendices - <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLAppendixPDF.zip">PDF Figures</a>  / <a href="https://drive.google.com/uc?export=download&id=1k2j7hMN40ISPSg9skFYWFL3oZT7r8v-l"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1_2cJHRnsoQQHst0rwZssv-XH4o5SEHks/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">Powerpoint Figures</a>
-</ul>
-
-Instructions for editing figures / equations can be found <a href="https://drive.google.com/file/d/1T_MXXVR4AfyMnlEFI-UVDh--FXI5deAp/view?usp=sharing">here</a>.</p>
-
-<h2>Resources for students</h2>
-
-<p>Answers to selected questions: <a href="https://github.com/udlbook/udlbook/raw/main/UDL_Answer_Booklet_Students.pdf">PDF</a></p>
-<p>Python notebooks:</p>
-
-<ul> 
-	<li> Notebook 1.1 - Background mathematics: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap01/1_1_BackgroundMathematics.ipynb">ipynb/colab</a>
-	<li> Notebook 2.1 - Supervised learning: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap02/2_1_Supervised_Learning.ipynb"">ipynb/colab</a>
-	<li> Notebook 3.1 - Shallow networks I: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_1_Shallow_Networks_I.ipynb">ipynb/colab </a>
-	<li> Notebook 3.2 - Shallow networks II: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_2_Shallow_Networks_II.ipynb">ipynb/colab </a>
-	<li> Notebook 3.3 - Shallow network regions: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_3_Shallow_Network_Regions.ipynb">ipynb/colab </a>
-	<li> Notebook 3.4 - Activation functions: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_4_Activation_Functions.ipynb">ipynb/colab </a>
-	<li> Notebook 4.1 - Composing networks: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap04/4_1_Composing_Networks.ipynb">ipynb/colab </a>
-	<li> Notebook 4.2 - Clipping functions: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap04/4_2_Clipping_functions.ipynb">ipynb/colab </a>
-	<li> Notebook 4.3 - Deep networks: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap04/4_3_Deep_Networks.ipynb">ipynb/colab </a>
-	<li> Notebook 5.1 - Least squares loss: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap05/5_1_Least_Squares_Loss.ipynb">ipynb/colab </a>
-	<li> Notebook 5.2 - Binary cross-entropy loss: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap05/5_2_Binary_Cross_Entropy_Loss.ipynb">ipynb/colab </a>
-	<li> Notebook 5.3 - Multiclass cross-entropy loss: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap05/5_3_Multiclass_Cross_entropy_Loss.ipynb">ipynb/colab </a>
-	<li> Notebook 6.1 - Line search: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_1_Line_Search.ipynb">ipynb/colab </a>
-	<li> Notebook 6.2 - Gradient descent: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_2_Gradient_Descent.ipynb">ipynb/colab </a>
-	<li> Notebook 6.3 - Stochastic gradient descent: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_3_Stochastic_Gradient_Descent.ipynb">ipynb/colab </a>
-	<li> Notebook 6.4 - Momentum: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_4_Momentum.ipynb">ipynb/colab </a>
-	<li> Notebook 6.5 - Adam: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_5_Adam.ipynb">ipynb/colab </a>
-	<li> Notebook 7.1 - Backpropagtion in toy model: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap07/7_1_Backpropagation_in_Toy_Model.ipynb"">ipynb/colab </a>
-	<li> Notebook 7.2 - Backpropagation: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap07/7_2_Backpropagation.ipynb">ipynb/colab </a>
-	<li> Notebook 7.3 - Initialization: <a href=""https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap07/7_3_Initialization.ipynb">ipynb/colab </a>
-	<li> Notebook 8.1 - MNIST-1D performance: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_1_MNIST_1D_Performance.ipynb"">ipynb/colab </a>
-	<li> Notebook 8.2 - Bias-variance trade-off: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_2_Bias_Variance_Trade_Off.ipynb">ipynb/colab </a>
-	<li> Notebook 8.3 - Double descent: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_3_Double_Descent.ipynb">ipynb/colab </a>
-	<li> Notebook 8.4 - High-dimensional spaces: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_4_High_Dimensional_Spaces.ipynb">ipynb/colab </a>
-	<li> Notebook 9.1 - L2 regularization: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_1_L2_Regularization.ipynb">ipynb/colab </a>
-	<li> Notebook 9.2 - Implicit regularization: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_2_Implicit_Regularization.ipynb">ipynb/colab </a>
-	<li> Notebook 9.3 - Ensembling: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_3_Ensembling.ipynb">ipynb/colab </a>
-	<li> Notebook 9.4 - Bayesian approach: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_4_Bayesian_Approach.ipynb">ipynb/colab </a>
-	<li> Notebook 9.5 - Augmentation <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_5_Augmentation.ipynb">ipynb/colab </a>
-	<li> Notebook 10.1 - 1D convolution: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_1_1D_Convolution.ipynb">ipynb/colab </a>
-	<li> Notebook 10.2 - Convolution for MNIST-1D: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_2_Convolution_for_MNIST_1D.ipynb">ipynb/colab </a>
-	<li> Notebook 10.3 - 2D convolution: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_3_2D_Convolution.ipynb">ipynb/colab </a>
-	<li> Notebook 10.4 - Downsampling & upsampling: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_4_Downsampling_and_Upsampling.ipynb">ipynb/colab </a>
-	<li> Notebook 10.5 - Convolution for MNIST: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_5_Convolution_For_MNIST.ipynb">ipynb/colab </a>
-	<li> Notebook 11.1 - Shattered gradients:  <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap11/11_1_Shattered_Gradients.ipynb">ipynb/colab </a>
-	<li> Notebook 11.2 - Residual networks: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap11/11_2_Residual_Networks.ipynb">ipynb/colab </a>
-	<li> Notebook 11.3 - Batch normalization: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap11/11_3_Batch_Normalization.ipynb">ipynb/colab </a>
-	<li> Notebook 12.1 - Self-attention: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_1_Self_Attention.ipynb">ipynb/colab </a>
-	<li> Notebook 12.2 - Multi-head self-attention: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_2_Multihead_Self_Attention.ipynb">ipynb/colab </a>
-	<li> Notebook 12.3 - Tokenization: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_3_Tokenization.ipynb">ipynb/colab </a>
-	<li> Notebook 12.4 - Decoding strategies: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_4_Decoding_Strategies.ipynb">ipynb/colab </a>
-	<li> Notebook 13.1 - Encoding graphs: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap13/13_1_Graph_Representation.ipynb">ipynb/colab </a>
-	<li> Notebook 13.2 - Graph classification : <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap13/13_2_Graph_Classification.ipynb">ipynb/colab </a>
-	<li> Notebook 13.3 - Neighborhood sampling: (coming soon)
-	<li> Notebook 13.4 - Graph attention: (coming soon)
-	<li> Notebook 15.1 - GAN toy example: (coming soon)
-	<li> Notebook 15.2 - Wasserstein distance: (coming soon)
-	<li> Notebook 16.1 - 1D normalizing flows: (coming soon)
-	<li> Notebook 16.2 - Autoregressive flows: (coming soon)
-	<li> Notebook 16.3 - Contraction mappings: (coming soon)
-	<li> Notebook 17.1 - Latent variable models: (coming soon)
-	<li> Notebook 17.2 - Reparameterization trick: (coming soon)
-	<li> Notebook 17.3 - Importance sampling: (coming soon)
-	<li> Notebook 18.1 - Diffusion encoder: (coming soon)
-	<li> Notebook 18.2 - 1D diffusion model: (coming soon)
-	<li> Notebook 18.3 - Reparameterized model: (coming soon)
-	<li> Notebook 18.4 - Families of diffusion models: (coming soon)
-	<li> Notebook 19.1 - Markov decision processes: (coming soon)
-	<li> Notebook 19.2 - Dynamic programming: (coming soon)
-	<li> Notebook 19.3 - Monte-Carlo methods: (coming soon)
-	<li> Notebook 19.4 - Temporal difference methods: (coming soon)
-	<li> Notebook 19.5 - Control variates: (coming soon)
-	<li> Notebook 20.1 - Random data: (coming soon)
-	<li> Notebook 20.2 - Full-batch gradient descent: (coming soon)
-	<li> Notebook 20.3 - Lottery tickets: (coming soon)
-	<li> Notebook 20.4 - Adversarial attacks: (coming soon)
-	<li> Notebook 21.1 - Bias mitigation: (coming soon)
-	<li> Notebook 21.2 - Explainability: (coming soon)
-</ul>
-
-
-<br>
-<h2>Citation:</h2>
-<pre><code>
+    <br>
+    <h2>Citation</h2>
+    <pre><code>
  @book{prince2023understanding,
  author = "Simon J.D. Prince",
  title = "Understanding Deep Learning",
@@ -163,4 +379,6 @@ Instructions for editing figures / equations can be found <a href="https://drive
  year = 2023,
  url = "http://udlbook.com"
 }
-</code></pre>
+    </code></pre>
+</div>
+</body>

style.css

@@ -0,0 +1,23 @@
+body {
+    font-size: 17px;
+    margin: 2% 10%;
+}
+
+#head {
+    display: flex;
+    flex-direction: row;
+    flex-wrap: wrap-reverse;
+    justify-content: space-between;
+    width: 100%;
+}
+
+#cover {
+    justify-content: center;
+    display: flex;
+    width: 30%;
+}
+
+#cover img {
+    width: 100%;
+    height: min-content;
+}