Update index.html

Typo fixes in Notebooks/Chap03/3_3_Shallow_Network_Regions.ipynb

CM20315/CM20315_Convolution_II.ipynb

@@ -105,7 +105,7 @@
       "cell_type": "code",
       "source": [
         "\n",
-        "# TODO Create a model with the folowing layers\n",
+        "# TODO Create a model with the following layers\n",
         "# 1. Convolutional layer, (input=length 40 and 1 channel, kernel size 3x3, stride 2, padding=\"valid\", 15 output channels ) \n",
         "# 2. ReLU\n",
         "# 3. Convolutional layer, (input=length 19 and 15 channels, kernel size 3x3, stride 2, padding=\"valid\", 15 output channels )\n",
@@ -120,7 +120,7 @@
         "# https://pytorch.org/docs/1.13/generated/torch.nn.Linear.html?highlight=linear#torch.nn.Linear\n",
         "\n",
         "# Replace the following function which just runs a standard fully connected network\n",
-        "# The flatten at the beginning is becuase we are passing in the data in a slightly different format.\n",
+        "# The flatten at the beginning is because we are passing in the data in a slightly different format.\n",
         "model = nn.Sequential(\n",
         "nn.Flatten(),\n",
         "nn.Linear(40, 100),\n",

CM20315/CM20315_Convolution_III.ipynb

@@ -148,7 +148,7 @@
         "# 8. A flattening operation\n",
         "# 9. A fully connected layer mapping from (whatever dimensions we are at-- find out using .shape) to 50 \n",
         "# 10. A ReLU\n",
-        "# 11. A fully connected layer mappiing from 50 to 10 dimensions\n",
+        "# 11. A fully connected layer mapping from 50 to 10 dimensions\n",
         "# 12. A softmax function.\n",
         "\n",
         "# Replace this class which implements a minimal network (which still does okay)\n",

CM20315/CM20315_Gradients_II.ipynb

@@ -32,7 +32,7 @@
       "source": [
         "# Gradients II: Backpropagation algorithm\n",
         "\n",
-        "In this practical, we'll investigate the backpropagation algoritithm.  This computes the gradients of the loss with respect to all of the parameters (weights and biases) in the network.  We'll use these gradients when we run stochastic gradient descent."
+        "In this practical, we'll investigate the backpropagation algorithm.  This computes the gradients of the loss with respect to all of the parameters (weights and biases) in the network.  We'll use these gradients when we run stochastic gradient descent."
       ],
       "metadata": {
         "id": "L6chybAVFJW2"
@@ -53,7 +53,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "First let's define a neural network.  We'll just choose the weights and biaes randomly for now"
+        "First let's define a neural network.  We'll just choose the weights and biases randomly for now"
       ],
       "metadata": {
         "id": "nnUoI0m6GyjC"
@@ -178,7 +178,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Now let's define a loss function.  We'll just use the least squaures loss function. We'll also write a function to compute dloss_doutpu"
+        "Now let's define a loss function.  We'll just use the least squares loss function. We'll also write a function to compute dloss_doutpu"
       ],
       "metadata": {
         "id": "SxVTKp3IcoBF"

CM20315/CM20315_Gradients_III.ipynb

@@ -53,7 +53,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "First let's define a neural network.  We'll just choose the weights and biaes randomly for now"
+        "First let's define a neural network.  We'll just choose the weights and biases randomly for now"
       ],
       "metadata": {
         "id": "nnUoI0m6GyjC"
@@ -204,7 +204,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Now let's define a loss function.  We'll just use the least squaures loss function. We'll also write a function to compute dloss_doutput\n"
+        "Now let's define a loss function.  We'll just use the least squares loss function. We'll also write a function to compute dloss_doutput\n"
       ],
       "metadata": {
         "id": "SxVTKp3IcoBF"

CM20315/CM20315_Intro.ipynb

@@ -176,7 +176,7 @@
         "# Color represents y value (brighter = higher value)\n",
         "# Black = -10 or less, White = +10 or more\n",
         "# 0 = mid orange\n",
-        "# Lines are conoturs where value is equal\n",
+        "# Lines are contours where value is equal\n",
         "draw_2D_function(x1,x2,y)\n",
         "\n",
         "# TODO\n",

CM20315/CM20315_Intro_Answers.ipynb

@@ -215,7 +215,7 @@
         "# Color represents y value (brighter = higher value)\n",
         "# Black = -10 or less, White = +10 or more\n",
         "# 0 = mid orange\n",
-        "# Lines are conoturs where value is equal\n",
+        "# Lines are contours where value is equal\n",
         "draw_2D_function(x1,x2,y)\n",
         "\n",
         "# TODO\n",

CM20315/CM20315_Loss.ipynb

@@ -36,7 +36,7 @@
         "\n",
         "We'll compute loss functions for maximum likelihood, minimum negative log likelihood, and least squares and show that they all imply that we should use the same parameter values\n",
         "\n",
-        "In part II, we'll investigate binary classification (where the output data is 0 or 1).  This will be based on the Bernouilli distribution\n",
+        "In part II, we'll investigate binary classification (where the output data is 0 or 1).  This will be based on the Bernoulli distribution\n",
         "\n",
         "In part III we'll investigate multiclass classification (where the output data is 0,1, or, 2).  This will be based on the categorical distribution."
       ],
@@ -178,7 +178,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "The blue line i sthe mean prediction of the model and the gray area represents plus/minus two standardard deviations.  This model fits okay, but could be improved. Let's compute the loss.  We'll compute the  the least squares error, the likelihood, the negative log likelihood."
+        "The blue line is the mean prediction of the model and the gray area represents plus/minus two standard deviations.  This model fits okay, but could be improved. Let's compute the loss.  We'll compute the  the least squares error, the likelihood, the negative log likelihood."
       ],
       "metadata": {
         "id": "MvVX6tl9AEXF"
@@ -276,7 +276,7 @@
         "beta_0, omega_0, beta_1, omega_1 = get_parameters()\n",
         "# Use our neural network to predict the mean of the Gaussian\n",
         "mu_pred = shallow_nn(x_train, beta_0, omega_0, beta_1, omega_1)\n",
-        "# Set the standard devation to something reasonable\n",
+        "# Set the standard deviation to something reasonable\n",
         "sigma = 0.2\n",
         "# Compute the likelihood\n",
         "likelihood = compute_likelihood(y_train, mu_pred, sigma)\n",
@@ -292,7 +292,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "You can see that this gives a very small answer, even for this small 1D dataset, and with the model fitting quite well.  This is because it is the product of sveral probabilities, which are all quite small themselves.\n",
+        "You can see that this gives a very small answer, even for this small 1D dataset, and with the model fitting quite well.  This is because it is the product of several probabilities, which are all quite small themselves.\n",
         "This will get out of hand pretty quickly with real datasets -- the likelihood will get so small that we can't represent it with normal finite-precision math\n",
         "\n",
         "This is why we use negative log likelihood"
@@ -326,7 +326,7 @@
         "beta_0, omega_0, beta_1, omega_1 = get_parameters()\n",
         "# Use our neural network to predict the mean of the Gaussian\n",
         "mu_pred = shallow_nn(x_train, beta_0, omega_0, beta_1, omega_1)\n",
-        "# Set the standard devation to something reasonable\n",
+        "# Set the standard deviation to something reasonable\n",
         "sigma = 0.2\n",
         "# Compute the log likelihood\n",
         "nll = compute_negative_log_likelihood(y_train, mu_pred, sigma)\n",
@@ -397,7 +397,7 @@
       "source": [
         "# Define a range of values for the parameter\n",
         "beta_1_vals = np.arange(0,1.0,0.01)\n",
-        "# Create some arrays to store the likelihoods, negative log likehoos and sum of squares\n",
+        "# Create some arrays to store the likelihoods, negative log likelihoods and sum of squares\n",
         "likelihoods = np.zeros_like(beta_1_vals)\n",
         "nlls = np.zeros_like(beta_1_vals)\n",
         "sum_squares = np.zeros_like(beta_1_vals)\n",
@@ -482,7 +482,7 @@
       "source": [
         "# Define a range of values for the parameter\n",
         "sigma_vals = np.arange(0.1,0.5,0.005)\n",
-        "# Create some arrays to store the likelihoods, negative log likehoos and sum of squares\n",
+        "# Create some arrays to store the likelihoods, negative log likelihoods and sum of squares\n",
         "likelihoods = np.zeros_like(sigma_vals)\n",
         "nlls = np.zeros_like(sigma_vals)\n",
         "sum_squares = np.zeros_like(sigma_vals)\n",

CM20315/CM20315_Loss_II.ipynb

@@ -34,7 +34,7 @@
         "\n",
         "This practical investigates loss functions.  In part I we investigated univariate regression (where the output data $y$ is continuous.  Our formulation was based on the normal/Gaussian distribution.\n",
         "\n",
-        "In this notebook, we investigate binary classification (where the output data is 0 or 1).  This will be based on the Bernouilli distribution\n",
+        "In this notebook, we investigate binary classification (where the output data is 0 or 1).  This will be based on the Bernoulli distribution\n",
         "\n",
         "In part III we'll investigate multiclass classification (where the outputs data can take multiple values 1,... K.\n",
         "\n",
@@ -199,7 +199,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "The left is model output and the right is the model output after the sigmoid has been applied, so it now lies in the range [0,1] and represents the probabiilty, that y=1.  The black dots show the training data.  We'll compute the the likelihood and the negative log likelihood."
+        "The left is model output and the right is the model output after the sigmoid has been applied, so it now lies in the range [0,1] and represents the probability, that y=1.  The black dots show the training data.  We'll compute the the likelihood and the negative log likelihood."
       ],
       "metadata": {
         "id": "MvVX6tl9AEXF"
@@ -210,7 +210,7 @@
       "source": [
         "# Return probability under Bernoulli distribution for input x\n",
         "def bernoulli_distribution(y, lambda_param):\n",
-        "    # TODO-- write in the equation for the Bernoullid distribution \n",
+        "    # TODO-- write in the equation for the Bernoulli distribution \n",
         "    # Equation 5.17 from the notes (you will need np.power)\n",
         "    # Replace the line below\n",
         "    prob = np.zeros_like(y)\n",
@@ -249,7 +249,7 @@
       "source": [
         "# Return the likelihood of all of the data under the model\n",
         "def compute_likelihood(y_train, lambda_param):\n",
-        "  # TODO -- compute the likelihood of the data -- the product of the Bernoullis probabilities for each data point\n",
+        "  # TODO -- compute the likelihood of the data -- the product of the Bernoulli's probabilities for each data point\n",
         "  # Top line of equation 5.3 in the notes\n",
         "  # You will need np.prod() and the bernoulli_distribution function you used above\n",
         "  # Replace the line below\n",
@@ -284,7 +284,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "You can see that this gives a very small answer, even for this small 1D dataset, and with the model fitting quite well.  This is because it is the product of sveral probabilities, which are all quite small themselves.\n",
+        "You can see that this gives a very small answer, even for this small 1D dataset, and with the model fitting quite well.  This is because it is the product of several probabilities, which are all quite small themselves.\n",
         "This will get out of hand pretty quickly with real datasets -- the likelihood will get so small that we can't represent it with normal finite-precision math\n",
         "\n",
         "This is why we use negative log likelihood"
@@ -317,7 +317,7 @@
         "beta_0, omega_0, beta_1, omega_1 = get_parameters()\n",
         "# Use our neural network to predict the mean of the Gaussian\n",
         "model_out = shallow_nn(x_train, beta_0, omega_0, beta_1, omega_1)\n",
-        "# Set the standard devation to something reasonable\n",
+        "# Set the standard deviation to something reasonable\n",
         "lambda_train = sigmoid(model_out)\n",
         "# Compute the log likelihood\n",
         "nll = compute_negative_log_likelihood(y_train, lambda_train)\n",
@@ -362,7 +362,7 @@
       "source": [
         "# Define a range of values for the parameter\n",
         "beta_1_vals = np.arange(-2,6.0,0.1)\n",
-        "# Create some arrays to store the likelihoods, negative log likehoods\n",
+        "# Create some arrays to store the likelihoods, negative log likelihoods\n",
         "likelihoods = np.zeros_like(beta_1_vals)\n",
         "nlls = np.zeros_like(beta_1_vals)\n",
         "\n",

CM20315/CM20315_Loss_III.ipynb

@@ -33,7 +33,7 @@
         "# Loss functions part III\n",
         "\n",
         "This practical investigates loss functions.  In part I we investigated univariate regression (where the output data $y$ is continuous.  Our formulation was based on the normal/Gaussian distribution.\n",
-        "In part II we investigated binary classification (where the output data is 0 or 1).  This will be based on the Bernouilli distribution.<br><br>\n",
+        "In part II we investigated binary classification (where the output data is 0 or 1).  This will be based on the Bernoulli distribution.<br><br>\n",
         "\n",
         "Now we'll investigate multiclass classification (where the outputs data can take multiple values 1,... K, which is based on the categorical distribution\n",
         "\n",
@@ -218,7 +218,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probabiilty, that y=0 (red), 1 (green) and 2 (blue)   The dots at the bottom show the training data with the same color scheme.  So we want the red curve to be high where there are red dots, the green curve to be high where there are green dotsmand the blue curve to be high where there are blue dots  We'll compute the the likelihood and the negative log likelihood."
+        "The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue)   The dots at the bottom show the training data with the same color scheme.  So we want the red curve to be high where there are red dots, the green curve to be high where there are green dotsmand the blue curve to be high where there are blue dots  We'll compute the the likelihood and the negative log likelihood."
       ],
       "metadata": {
         "id": "MvVX6tl9AEXF"
@@ -228,7 +228,7 @@
       "cell_type": "code",
       "source": [
         "# Return probability under Bernoulli distribution for input x\n",
-        "# Complicated code to commpute it but just take value from row k of lambda param where y =k, \n",
+        "# Complicated code to compute it but just take value from row k of lambda param where y =k, \n",
         "def categorical_distribution(y, lambda_param):\n",
         "    prob = np.zeros_like(y)\n",
         "    for row_index in range(lambda_param.shape[0]):\n",
@@ -305,7 +305,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "You can see that this gives a very small answer, even for this small 1D dataset, and with the model fitting quite well.  This is because it is the product of sveral probabilities, which are all quite small themselves.\n",
+        "You can see that this gives a very small answer, even for this small 1D dataset, and with the model fitting quite well.  This is because it is the product of several probabilities, which are all quite small themselves.\n",
         "This will get out of hand pretty quickly with real datasets -- the likelihood will get so small that we can't represent it with normal finite-precision math\n",
         "\n",
         "This is why we use negative log likelihood"
@@ -338,7 +338,7 @@
         "beta_0, omega_0, beta_1, omega_1 = get_parameters()\n",
         "# Use our neural network to predict the mean of the Gaussian\n",
         "model_out = shallow_nn(x_train, beta_0, omega_0, beta_1, omega_1)\n",
-        "# Set the standard devation to something reasonable\n",
+        "# Set the standard deviation to something reasonable\n",
         "lambda_train = softmax(model_out)\n",
         "# Compute the log likelihood\n",
         "nll = compute_negative_log_likelihood(y_train, lambda_train)\n",
@@ -365,7 +365,7 @@
       "source": [
         "# Define a range of values for the parameter\n",
         "beta_1_vals = np.arange(-2,6.0,0.1)\n",
-        "# Create some arrays to store the likelihoods, negative log likehoods\n",
+        "# Create some arrays to store the likelihoods, negative log likelihoods\n",
         "likelihoods = np.zeros_like(beta_1_vals)\n",
         "nlls = np.zeros_like(beta_1_vals)\n",
         "\n",

CM20315/CM20315_Shallow.ipynb

@@ -233,7 +233,7 @@
         "# TODO\n",
         "# 1. Predict what effect changing phi_0 will have on the network.  \n",
         "#   Answer:\n",
-        "# 2. Predict what effect multplying phi_1, phi_2, phi_3 by 0.5 would have.  Check if you are correct\n",
+        "# 2. Predict what effect multiplying phi_1, phi_2, phi_3 by 0.5 would have.  Check if you are correct\n",
         "#   Answer:\n",
         "# 3. Predict what effect multiplying phi_1 by -1 will have.  Check if you are correct.\n",
         "#   Answer:\n",
@@ -500,7 +500,7 @@
         "print(\"Loss = %3.3f\"%(loss))\n",
         "\n",
         "# TODO.  Manipulate the parameters (by hand!) to make the function \n",
-        "# fit the data better and try to reduct the loss to as small a number \n",
+        "# fit the data better and try to reduce the loss to as small a number \n",
         "# as possible.  The best that I could do was 0.181\n",
         "# Tip... start by manipulating phi_0.\n",
         "# It's not that easy, so don't spend too much time on this!"

CM20315/CM20315_Training_I.ipynb

@@ -108,7 +108,7 @@
       "source": [
         "def line_search(loss_function, thresh=.0001, max_iter = 10, draw_flag = False):\n",
         "\n",
-        "    # Initialize four points along the rnage we are going to search\n",
+        "    # Initialize four points along the range we are going to search\n",
         "    a = 0\n",
         "    b = 0.33\n",
         "    c = 0.66\n",
@@ -139,7 +139,7 @@
         "        # Rule #2 If point b is less than point c then\n",
         "        #                     then point d becomes point c, and\n",
         "        #                     point b becomes 1/3 between a and new d\n",
-        "        #                     point c beocome 2/3 between a and new d \n",
+        "        #                     point c becomes 2/3 between a and new d \n",
         "        # TODO REPLACE THE BLOCK OF CODE BELOW WITH THIS RULE\n",
         "        if (0):\n",
         "          continue;\n",
@@ -147,7 +147,7 @@
         "        # Rule #3 If point c is less than point b then\n",
         "        #                     then point a becomes point b, and\n",
         "        #                     point b becomes 1/3 between new a and d\n",
-        "        #                     point c beocome 2/3 between new a and d \n",
+        "        #                     point c becomes 2/3 between new a and d \n",
         "        # TODO REPLACE THE BLOCK OF CODE BELOW WITH THIS RULE\n",
         "        if(0):\n",
         "          continue\n",

CM20315/CM20315_Training_II.ipynb

@@ -114,7 +114,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Initialize the parmaeters and draw the model\n",
+        "# Initialize the parameters and draw the model\n",
         "phi = np.zeros((2,1))\n",
         "phi[0] = 0.6      # Intercept\n",
         "phi[1] = -0.2      # Slope\n",
@@ -314,7 +314,7 @@
         "  return compute_loss(data[0,:], data[1,:], model, phi_start+ gradient * dist_prop)\n",
         "\n",
         "def line_search(data, model, phi, gradient, thresh=.00001, max_dist = 0.1, max_iter = 15, verbose=False):\n",
-        "    # Initialize four points along the rnage we are going to search\n",
+        "    # Initialize four points along the range we are going to search\n",
         "    a = 0\n",
         "    b = 0.33 * max_dist\n",
         "    c = 0.66 * max_dist\n",
@@ -345,7 +345,7 @@
         "        # Rule #2 If point b is less than point c then\n",
         "        #                     then point d becomes point c, and\n",
         "        #                     point b becomes 1/3 between a and new d\n",
-        "        #                     point c beocome 2/3 between a and new d \n",
+        "        #                     point c becomes 2/3 between a and new d \n",
         "        if lossb < lossc:\n",
         "          d = c\n",
         "          b = a+ (d-a)/3\n",
@@ -355,7 +355,7 @@
         "        # Rule #2 If point c is less than point b then\n",
         "        #                     then point a becomes point b, and\n",
         "        #                     point b becomes 1/3 between new a and d\n",
-        "        #                     point c beocome 2/3 between new a and d \n",
+        "        #                     point c becomes 2/3 between new a and d \n",
         "        a = b\n",
         "        b = a+ (d-a)/3\n",
         "        c = a+ 2*(d-a)/3\n",

CM20315/CM20315_Training_III.ipynb

@@ -340,7 +340,7 @@
         "  return compute_loss(data[0,:], data[1,:], model, phi_start+ gradient * dist_prop)\n",
         "\n",
         "def line_search(data, model, phi, gradient, thresh=.00001, max_dist = 0.1, max_iter = 15, verbose=False):\n",
-        "    # Initialize four points along the rnage we are going to search\n",
+        "    # Initialize four points along the range we are going to search\n",
         "    a = 0\n",
         "    b = 0.33 * max_dist\n",
         "    c = 0.66 * max_dist\n",
@@ -371,7 +371,7 @@
         "        # Rule #2 If point b is less than point c then\n",
         "        #                     then point d becomes point c, and\n",
         "        #                     point b becomes 1/3 between a and new d\n",
-        "        #                     point c beocome 2/3 between a and new d \n",
+        "        #                     point c becomes 2/3 between a and new d \n",
         "        if lossb < lossc:\n",
         "          d = c\n",
         "          b = a+ (d-a)/3\n",
@@ -381,7 +381,7 @@
         "        # Rule #2 If point c is less than point b then\n",
         "        #                     then point a becomes point b, and\n",
         "        #                     point b becomes 1/3 between new a and d\n",
-        "        #                     point c beocome 2/3 between new a and d \n",
+        "        #                     point c becomes 2/3 between new a and d \n",
         "        a = b\n",
         "        b = a+ (d-a)/3\n",
         "        c = a+ 2*(d-a)/3\n",

CM20315/CM20315_Transformers.ipynb

@@ -175,7 +175,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# TODO Modify the code below by changeing the number of tokens generated and the initial sentence\n",
+        "# TODO Modify the code below by changing the number of tokens generated and the initial sentence\n",
         "# to get a feel for how well this works.  Since I didn't reset the seed, it will give a different\n",
         "# answer every time that you run it.\n",
         "\n",
@@ -253,7 +253,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# TODO Modify the code below by changeing the number of tokens generated and the initial sentence\n",
+        "# TODO Modify the code below by changing the number of tokens generated and the initial sentence\n",
         "# to get a feel for how well this works.  \n",
         "\n",
         "# TODO Experiment with changing this line:\n",
@@ -471,7 +471,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# This routine reutnrs the k'th most likely next token.\n",
+        "# This routine returns the k'th most likely next token.\n",
         "# If k =0 then it returns the most likely token, if k=1 it returns the next most likely and so on\n",
         "# We will need this for beam search\n",
         "def get_kth_most_likely_token(input_tokens, model, tokenizer, k):\n",

CM20315_2023/CM20315_Coursework_IV.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyNNnZyVCX9glFJGIC8BwtVT",
+      "authorship_tag": "ABX9TyMrWYwQrwgJvDza1vhYK9WQ",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -139,7 +139,7 @@
       "source": [
         "def volume_of_hypersphere(diameter, dimensions):\n",
         "  # Formula given in Problem 8.7 of the notes\n",
-        "  # You will need sci.special.gamma()\n",
+        "  # You will need sci.gamma()\n",
         "  # Check out:    https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.gamma.html\n",
         "  # Also use this value for pi\n",
         "  pi = np.pi\n",

Notebooks/Chap01/1_1_BackgroundMathematics.ipynb

@@ -1,16 +1,18 @@
 {
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap01/1_1_BackgroundMathematics.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "s5zzKSOusPOB"
@@ -39,6 +41,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "WV2Dl6owme2d"
@@ -46,11 +49,11 @@
       "source": [
         "**Linear functions**<br> We will be using the term *linear equation* to mean a weighted sum of inputs plus an offset. If there is just one input $x$, then this is a straight line:\n",
         "\n",
-        "\\begin{equation}y=\\beta+\\omega x,\\end{equation} <br>\n",
+        "\\begin{equation}y=\\beta+\\omega x,\\end{equation} \n",
         "\n",
         "where $\\beta$ is the y-intercept of the linear and $\\omega$ is the slope of the line. When there are two inputs $x_{1}$ and $x_{2}$, then this becomes:\n",
         "\n",
-        "\\begin{equation}y=\\beta+\\omega_1 x_1 + \\omega_2 x_2.\\end{equation} <br><br>\n",
+        "\\begin{equation}y=\\beta+\\omega_1 x_1 + \\omega_2 x_2.\\end{equation} \n",
         "\n",
         "Any other functions are by definition **non-linear**.\n",
         "\n",
@@ -83,7 +86,7 @@
       "source": [
         "# Plot the 1D linear function\n",
         "\n",
-        "# Define an array of x values from 0 to 10 with increments of 0.1\n",
+        "# Define an array of x values from 0 to 10 with increments of 0.01\n",
         "# https://numpy.org/doc/stable/reference/generated/numpy.arange.html\n",
         "x = np.arange(0.0,10.0, 0.01)\n",
         "# Compute y using the function you filled in above\n",
@@ -104,6 +107,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "AedfvD9dxShZ"
@@ -171,7 +175,7 @@
         "# Color represents y value (brighter = higher value)\n",
         "# Black = -10 or less, White = +10 or more\n",
         "# 0 = mid orange\n",
-        "# Lines are conoturs where value is equal\n",
+        "# Lines are contours where value is equal\n",
         "draw_2D_function(x1,x2,y)\n",
         "\n",
         "# TODO\n",
@@ -188,6 +192,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "i8tLwpls476R"
@@ -195,15 +200,15 @@
       "source": [
         "Often we will want to compute many linear functions at the same time.  For example, we might have three inputs, $x_1$, $x_2$, and $x_3$ and want to compute two linear functions giving $y_1$ and $y_2$. Of course, we could do this by just running each equation separately,<br><br>\n",
         "\n",
-        "\\begin{eqnarray}y_1 &=& \\beta_1 + \\omega_{11} x_1 + \\omega_{12} x_2 + \\omega_{13} x_3\\\\\n",
+        "\\begin{align}y_1 &=& \\beta_1 + \\omega_{11} x_1 + \\omega_{12} x_2 + \\omega_{13} x_3\\\\\n",
         "y_2 &=& \\beta_2 + \\omega_{21} x_1 + \\omega_{22} x_2 + \\omega_{23} x_3.\n",
-        "\\end{eqnarray}<br>\n",
+        "\\end{align}\n",
         "\n",
         "However, we can write it more compactly with vectors and matrices:\n",
         "\n",
         "\\begin{equation}\n",
         "\\begin{bmatrix} y_1\\\\ y_2 \\end{bmatrix} = \\begin{bmatrix}\\beta_{1}\\\\\\beta_{2}\\end{bmatrix}+ \\begin{bmatrix}\\omega_{11}&\\omega_{12}&\\omega_{13}\\\\\\omega_{21}&\\omega_{22}&\\omega_{23}\\end{bmatrix}\\begin{bmatrix}x_{1}\\\\x_{2}\\\\x_{3}\\end{bmatrix},\n",
-        "\\end{equation}<br>\n",
+        "\\end{equation}\n",
         "or\n",
         "\n",
         "\\begin{equation}\n",
@@ -231,6 +236,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "fGzVJQ6N-mHJ"
@@ -273,6 +279,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "3LGRoTMLU8ZU"
@@ -286,6 +293,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "7Y5zdKtKZAB2"
@@ -295,7 +303,7 @@
         "\n",
         "Throughout the book, we'll be using some special functions (see Appendix B.1.3).  The most important of these are the logarithm and exponential functions.  Let's investigate their properties.\n",
         "\n",
-        "We'll start with the exponential function $y=\\mbox{exp}[x]=e^x$ which maps the real line $[-\\infty,+\\infty]$ to non-negative numbers $[0,+\\infty]$."
+        "We'll start with the exponential function $y=\\exp[x]=e^x$ which maps the real line $[-\\infty,+\\infty]$ to non-negative numbers $[0,+\\infty]$."
       ]
     },
     {
@@ -308,7 +316,7 @@
       "source": [
         "# Draw the exponential function\n",
         "\n",
-        "# Define an array of x values from -5 to 5 with increments of 0.1\n",
+        "# Define an array of x values from -5 to 5 with increments of 0.01\n",
         "x = np.arange(-5.0,5.0, 0.01)\n",
         "y = np.exp(x) ;\n",
         "\n",
@@ -321,6 +329,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "XyrT8257IWCu"
@@ -328,14 +337,15 @@
       "source": [
         "# Questions\n",
         "\n",
-        "1. What is $\\mbox{exp}[0]$?  \n",
+        "1. What is $\\exp[0]$?  \n",
-        "2. What is $\\mbox{exp}[1]$?\n",
+        "2. What is $\\exp[1]$?\n",
-        "3. What is $\\mbox{exp}[-\\infty]$?\n",
+        "3. What is $\\exp[-\\infty]$?\n",
-        "4. What is $\\mbox{exp}[+\\infty]$?\n",
+        "4. What is $\\exp[+\\infty]$?\n",
         "5. A function is convex if we can draw a straight line between any two points on the function, and this line always lies above the function. Similarly, a function is concave if a straight line between any two points always lies below the function.  Is the exponential function convex or concave or neither?\n"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "R6A4e5IxIWCu"
@@ -354,7 +364,7 @@
       "source": [
         "# Draw the logarithm function\n",
         "\n",
-        "# Define an array of x values from -5 to 5 with increments of 0.1\n",
+        "# Define an array of x values from -5 to 5 with increments of 0.01\n",
         "x = np.arange(0.01,5.0, 0.01)\n",
         "y = np.log(x) ;\n",
         "\n",
@@ -367,6 +377,7 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
         "id": "yYWrL5AXIWCv"
@@ -374,20 +385,20 @@
       "source": [
         "# Questions\n",
         "\n",
-        "1. What is $\\mbox{log}[0]$?  \n",
+        "1. What is $\\log[0]$?  \n",
-        "2. What is $\\mbox{log}[1]$?\n",
+        "2. What is $\\log[1]$?\n",
-        "3. What is $\\mbox{log}[e]$?\n",
+        "3. What is $\\log[e]$?\n",
-        "4. What is $\\mbox{log}[\\exp[3]]$?\n",
+        "4. What is $\\log[\\exp[3]]$?\n",
-        "5. What is $\\mbox{exp}[\\log[4]]$?\n",
+        "5. What is $\\exp[\\log[4]]$?\n",
-        "6. What is $\\mbox{log}[-1]$?\n",
+        "6. What is $\\log[-1]$?\n",
         "7. Is the logarithm function concave or convex?\n"
       ]
     }
   ],
   "metadata": {
     "colab": {
-      "provenance": [],
+      "include_colab_link": true,
-      "include_colab_link": true
+      "provenance": []
     },
     "kernelspec": {
       "display_name": "Python 3 (ipykernel)",

Notebooks/Chap02/2_1_Supervised_Learning.ipynb

@@ -213,7 +213,7 @@
         "\n",
         "# Make a 2D array for the losses\n",
         "all_losses = np.zeros_like(phi1_mesh)\n",
-        "# Run throught each 2D combination of phi0, phi1 and compute loss\n",
+        "# Run through each 2D combination of phi0, phi1 and compute loss\n",
         "for indices,temp in np.ndenumerate(phi1_mesh):\n",
         "    all_losses[indices] = compute_loss(x,y, phi0_mesh[indices], phi1_mesh[indices])\n"
       ],

Notebooks/Chap03/3_2_Shallow_Networks_II.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyNk2dAhwwRxGpfVSC3b2Owv",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -182,7 +181,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Now we'll extend this model to have two outputs $y_1$ and $y_2$, each of which can be visualized with a separate heatmap.  You will now have sets of parameters $\\phi_{10}, \\phi_{11},\\phi_{12}$ and $\\phi_{20}, \\phi_{21},\\phi_{22}$ that correspond to each of these outputs."
+        "Now we'll extend this model to have two outputs $y_1$ and $y_2$, each of which can be visualized with a separate heatmap.  You will now have sets of parameters $\\phi_{10}, \\phi_{11}, \\phi_{12}, \\phi_{13}$ and $\\phi_{20}, \\phi_{21}, \\phi_{22}, \\phi_{23}$ that correspond to each of these outputs."
       ],
       "metadata": {
         "id": "Xl6LcrUyM7Lh"

Notebooks/Chap03/3_3_Shallow_Network_Regions.ipynb

@@ -48,7 +48,7 @@
         "import numpy as np\n",
         "# Imports plotting library\n",
         "import matplotlib.pyplot as plt\n",
-        "# Imports math libray\n",
+        "# Imports math library\n",
         "import math"
       ],
       "metadata": {
@@ -79,7 +79,7 @@
       "source": [
         "def number_regions(Di, D):\n",
         "  # TODO -- implement Zaslavsky's formula\n",
-        "  # You can use math.com() https://www.w3schools.com/python/ref_math_comb.asp\n",
+        "  # You can use math.comb() https://www.w3schools.com/python/ref_math_comb.asp\n",
         "  # Replace this code\n",
         "  N = 1;\n",
         "\n",
@@ -102,7 +102,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Calculate the number of regions for 10D input (Di=2) and 50 hidden units (D=50)\n",
+        "# Calculate the number of regions for 10D input (Di=10) and 50 hidden units (D=50)\n",
         "N = number_regions(10, 50)\n",
         "print(f\"Di=10, D=50, Number of regions = {int(N)}, True value = 13432735556\")"
       ],
@@ -126,7 +126,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Show that calculation fails when $D_i < D$\n",
+        "# Depending on how you implemented it, the calculation may fail when $D_i > D$ (not to worry...)\n",
         "try:\n",
         "  N = number_regions(10, 8)\n",
         "  print(f\"Di=10, D=8, Number of regions = {int(N)}, True value = 256\")\n",

Notebooks/Chap03/3_4_Activation_Functions.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyPmra+JD+dm2M3gCqx3bMak",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap03/3_4_Activation_Functions.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "Mn0F56yY8ohX"
+      },
       "source": [
         "# **Notebook 3.4 -- Activation functions**\n",
         "\n",
@@ -36,10 +25,7 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and write code to complete the functions. There are also questions interspersed in the text.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "Mn0F56yY8ohX"
-      }
     },
     {
       "cell_type": "code",
@@ -57,6 +43,11 @@
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "AeHzflFt9Tgn"
+      },
+      "outputs": [],
       "source": [
         "# Plot the shallow neural network.  We'll assume input in is range [0,1] and output [-1,1]\n",
         "# If the plot_all flag is set to true, then we'll plot all the intermediate stages as in Figure 3.3\n",
@@ -94,15 +85,15 @@
         "    for i in range(len(x_data)):\n",
         "      ax.plot(x_data[i], y_data[i],)\n",
         "  plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "AeHzflFt9Tgn"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7qeIUrh19AkH"
+      },
+      "outputs": [],
       "source": [
         "# Define a shallow neural network with, one input, one output, and three hidden units\n",
         "def shallow_1_1_3(x, activation_fn, phi_0,phi_1,phi_2,phi_3, theta_10, theta_11, theta_20, theta_21, theta_30, theta_31):\n",
@@ -123,38 +114,39 @@
         "\n",
         "  # Return everything we have calculated\n",
         "  return y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3"
-      ],
+      ]
-      "metadata": {
-        "id": "7qeIUrh19AkH"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cwTp__Fk9YUx"
+      },
+      "outputs": [],
       "source": [
         "# Define the Rectified Linear Unit (ReLU) function\n",
         "def ReLU(preactivation):\n",
         "  activation = preactivation.clip(0.0)\n",
         "  return activation"
-      ],
+      ]
-      "metadata": {
-        "id": "cwTp__Fk9YUx"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "First, let's run the network with a ReLU functions"
-      ],
       "metadata": {
         "id": "INQkRzyn9kVC"
-      }
+      },
+      "source": [
+        "First, let's run the network with a ReLU functions"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "jT9QuKou9i0_"
+      },
+      "outputs": [],
       "source": [
         "# Now lets define some parameters and run the neural network\n",
         "theta_10 =  0.3 ; theta_11 = -1.0\n",
@@ -170,15 +162,14 @@
         "    shallow_1_1_3(x, ReLU, phi_0,phi_1,phi_2,phi_3, theta_10, theta_11, theta_20, theta_21, theta_30, theta_31)\n",
         "# And then plot it\n",
         "plot_neural(x, y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3, plot_all=True)"
-      ],
+      ]
-      "metadata": {
-        "id": "jT9QuKou9i0_"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "-I8N7r1o9HYf"
+      },
       "source": [
         "# Sigmoid activation function\n",
         "\n",
@@ -189,13 +180,15 @@
         "\\end{equation}\n",
         "\n",
         "(Note that the factor of 10 is not standard -- but it allow us to plot on the same axes as the ReLU examples)"
-      ],
+      ]
-      "metadata": {
-        "id": "-I8N7r1o9HYf"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "hgkioNyr975Y"
+      },
+      "outputs": [],
       "source": [
         "# Define the sigmoid function\n",
         "def sigmoid(preactivation):\n",
@@ -204,15 +197,15 @@
         "  activation = np.zeros_like(preactivation);\n",
         "\n",
         "  return activation"
-      ],
+      ]
-      "metadata": {
-        "id": "hgkioNyr975Y"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "94HIXKJH97ve"
+      },
+      "outputs": [],
       "source": [
         "# Make an array of inputs\n",
         "z = np.arange(-1,1,0.01)\n",
@@ -223,25 +216,26 @@
         "ax.plot(z,sig_z,'r-')\n",
         "ax.set_xlim([-1,1]);ax.set_ylim([0,1])\n",
         "ax.set_xlabel('z'); ax.set_ylabel('sig[z]')\n",
-        "plt.show"
+        "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "94HIXKJH97ve"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Let's see what happens when we use this activation function in a neural network"
-      ],
       "metadata": {
         "id": "p3zQNXhj-J-o"
-      }
+      },
+      "source": [
+        "Let's see what happens when we use this activation function in a neural network"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "C1dASr9L-GNt"
+      },
+      "outputs": [],
       "source": [
         "theta_10 =  0.3 ; theta_11 = -1.0\n",
         "theta_20 = -1.0  ; theta_21 = 2.0\n",
@@ -256,39 +250,41 @@
         "    shallow_1_1_3(x, sigmoid, phi_0,phi_1,phi_2,phi_3, theta_10, theta_11, theta_20, theta_21, theta_30, theta_31)\n",
         "# And then plot it\n",
         "plot_neural(x, y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3, plot_all=True)"
-      ],
+      ]
-      "metadata": {
-        "id": "C1dASr9L-GNt"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "You probably notice that this gives nice smooth curves.  So why don't we use this?  Aha... it's not obvious right now, but we will get to it when we learn to fit models."
-      ],
       "metadata": {
         "id": "Uuam_DewA9fH"
-      }
+      },
+      "source": [
+        "You probably notice that this gives nice smooth curves.  So why don't we use this?  Aha... it's not obvious right now, but we will get to it when we learn to fit models."
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "C9WKkcMUABze"
+      },
       "source": [
         "# Heaviside activation function\n",
         "\n",
         "The Heaviside function is defined as:\n",
         "\n",
         "\\begin{equation}\n",
-        "\\mbox{heaviside}[z] = \\begin{cases} 0 & \\quad z <0 \\\\ 1 & \\quad z\\geq 0\\end{cases}\n",
+        "\\text{heaviside}[z] = \\begin{cases} 0 & \\quad z <0 \\\\ 1 & \\quad z\\geq 0\\end{cases}\n",
         "\\end{equation}"
-      ],
+      ]
-      "metadata": {
-        "id": "C9WKkcMUABze"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "-1qFkdOL-NPc"
+      },
+      "outputs": [],
       "source": [
         "# Define the heaviside function\n",
         "def heaviside(preactivation):\n",
@@ -299,15 +295,15 @@
         "\n",
         "\n",
         "  return activation"
-      ],
+      ]
-      "metadata": {
-        "id": "-1qFkdOL-NPc"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mSPyp7iA-44H"
+      },
+      "outputs": [],
       "source": [
         "# Make an array of inputs\n",
         "z = np.arange(-1,1,0.01)\n",
@@ -318,16 +314,16 @@
         "ax.plot(z,heav_z,'r-')\n",
         "ax.set_xlim([-1,1]);ax.set_ylim([-2,2])\n",
         "ax.set_xlabel('z'); ax.set_ylabel('heaviside[z]')\n",
-        "plt.show"
+        "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "mSPyp7iA-44H"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "t99K2lSl--Mq"
+      },
+      "outputs": [],
       "source": [
         "theta_10 =  0.3 ; theta_11 = -1.0\n",
         "theta_20 = -1.0  ; theta_21 = 2.0\n",
@@ -342,39 +338,41 @@
         "    shallow_1_1_3(x, heaviside, phi_0,phi_1,phi_2,phi_3, theta_10, theta_11, theta_20, theta_21, theta_30, theta_31)\n",
         "# And then plot it\n",
         "plot_neural(x, y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3, plot_all=True)"
-      ],
+      ]
-      "metadata": {
-        "id": "t99K2lSl--Mq"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "This can approximate any function, but the output is discontinuous, and there are also reasons not to use it that we will discover when we learn more about model fitting."
-      ],
       "metadata": {
         "id": "T65MRtM-BCQA"
-      }
+      },
+      "source": [
+        "This can approximate any function, but the output is discontinuous, and there are also reasons not to use it that we will discover when we learn more about model fitting."
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "RkB-XZMLBTaR"
+      },
       "source": [
         "# Linear activation functions\n",
         "\n",
         "Neural networks don't work if the activation function is linear.  For example, consider what would happen if the activation function was:\n",
         "\n",
         "\\begin{equation}\n",
-        "\\mbox{lin}[z] = a + bz\n",
+        "\\text{lin}[z] = a + bz\n",
         "\\end{equation}"
-      ],
+      ]
-      "metadata": {
-        "id": "RkB-XZMLBTaR"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Q59v3saj_jq1"
+      },
+      "outputs": [],
       "source": [
         "# Define the linear activation function\n",
         "def lin(preactivation):\n",
@@ -384,15 +382,15 @@
         "  activation = a+b * preactivation\n",
         "  # Return\n",
         "  return activation"
-      ],
+      ]
-      "metadata": {
-        "id": "Q59v3saj_jq1"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "IwodsBr0BkDn"
+      },
+      "outputs": [],
       "source": [
         "# TODO\n",
         "# 1. The linear activation function above just returns the input: (0+1*z) = z\n",
@@ -415,12 +413,23 @@
         "    shallow_1_1_3(x, lin, phi_0,phi_1,phi_2,phi_3, theta_10, theta_11, theta_20, theta_21, theta_30, theta_31)\n",
         "# And then plot it\n",
         "plot_neural(x, y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3, plot_all=True)"
+      ]
+    }
   ],
   "metadata": {
-        "id": "IwodsBr0BkDn"
+    "colab": {
+      "authorship_tag": "ABX9TyOmxhh3ymYWX+1HdZ91I6zU",
+      "include_colab_link": true,
+      "provenance": []
     },
-      "execution_count": null,
+    "kernelspec": {
-      "outputs": []
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
     }
-  ]
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap04/4_1_Composing_Networks.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyPEQEGetZqWnLRNn99Q2aaT",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -220,7 +219,7 @@
       "source": [
         "# TODO\n",
         "# Take a piece of paper and draw what you think will happen when we feed the\n",
-        "# output of the first network into the second one now that we have changed it.  Draw the relationship between\n",
+        "# output of the first network into the modified second network.  Draw the relationship between\n",
         "# the input of the first network and the output of the second one."
       ],
       "metadata": {
@@ -261,7 +260,7 @@
       "source": [
         "# TODO\n",
         "# Take a piece of paper and draw what you think will happen when we feed the\n",
-        "# output of the first network now we have changed it into the original second network.  Draw the relationship between\n",
+        "# output of the modified first network into the original second network.  Draw the relationship between\n",
         "# the input of the first network and the output of the second one."
       ],
       "metadata": {
@@ -302,7 +301,7 @@
       "source": [
         "# TODO\n",
         "# Take a piece of paper and draw what you think will happen when we feed the\n",
-        "# output of the first network into the original second network.  Draw the relationship between\n",
+        "# output of the first network into the a copy of itself.  Draw the relationship between\n",
         "# the input of the first network and the output of the second one."
       ],
       "metadata": {
@@ -350,7 +349,7 @@
         "# network (blue curve above)\n",
         "\n",
         "# Take away conclusion:  with very few parameters, we can make A LOT of linear regions, but\n",
-        "# they depend on one another in complex ways that quickly become to difficult to understand intuitively."
+        "# they depend on one another in complex ways that quickly become too difficult to understand intuitively."
       ],
       "metadata": {
         "id": "HqzePCLOVQK7"

Notebooks/Chap04/4_3_Deep_Networks.ipynb

@@ -101,7 +101,6 @@
       "cell_type": "code",
       "source": [
         "# # Plot the shallow neural network.  We'll assume input in is range [-1,1] and output [-1,1]\n",
-        "# If the plot_all flag is set to true, then we'll plot all the intermediate stages as in Figure 3.3\n",
         "def plot_neural(x, y):\n",
         "  fig, ax = plt.subplots()\n",
         "  ax.plot(x.T,y.T)\n",
@@ -119,7 +118,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Let's define a networks.  We'll just consider the inputs and outputs over the range [-1,1].  If you set the \"plot_all\" flat to True,  you can see the details of how it was created."
+        "Let's define a network.  We'll just consider the inputs and outputs over the range [-1,1].  If you set the \"plot_all\" flat to True,  you can see the details of how it was created."
       ],
       "metadata": {
         "id": "LxBJCObC-NTY"

Notebooks/Chap05/5_1_Least_Squares_Loss.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOJeBMhN9fXO8UepZ4+Pbg6",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -433,12 +432,25 @@
       "cell_type": "code",
       "source": [
         "# Now let's plot the likelihood, negative log likelihood, and least squares as a function the value of the offset beta1\n",
-        "fig, ax = plt.subplots(1,3)\n",
+        "fig, ax = plt.subplots(1,2)\n",
-        "fig.set_size_inches(10.5, 3.5)\n",
+        "fig.set_size_inches(10.5, 5.5)\n",
-        "fig.tight_layout(pad=3.0)\n",
+        "fig.tight_layout(pad=10.0)\n",
-        "ax[0].plot(beta_1_vals, likelihoods); ax[0].set_xlabel('beta_1[0]$'); ax[0].set_ylabel('likelihood')\n",
+        "likelihood_color = 'tab:red'\n",
-        "ax[1].plot(beta_1_vals, nlls); ax[1].set_xlabel('beta_1[0]'); ax[1].set_ylabel('negative log likelihood')\n",
+        "nll_color = 'tab:blue'\n",
-        "ax[2].plot(beta_1_vals, sum_squares); ax[2].set_xlabel('beta_1[0]'); ax[2].set_ylabel('sum of squares')\n",
+        "\n",
+        "ax[0].set_xlabel('beta_1[0]')\n",
+        "ax[0].set_ylabel('likelihood', color = likelihood_color)\n",
+        "ax[0].plot(beta_1_vals, likelihoods, color = likelihood_color)\n",
+        "ax[0].tick_params(axis='y', labelcolor=likelihood_color)\n",
+        "\n",
+        "ax00 = ax[0].twinx()\n",
+        "ax00.plot(beta_1_vals, nlls, color = nll_color)\n",
+        "ax00.set_ylabel('negative log likelihood', color = nll_color)\n",
+        "ax00.tick_params(axis='y', labelcolor = nll_color)\n",
+        "\n",
+        "plt.axvline(x = beta_1_vals[np.argmax(likelihoods)], linestyle='dotted')\n",
+        "\n",
+        "ax[1].plot(beta_1_vals, sum_squares); ax[1].set_xlabel('beta_1[0]'); ax[1].set_ylabel('sum of squares')\n",
         "plt.show()"
       ],
       "metadata": {
@@ -519,12 +531,26 @@
       "cell_type": "code",
       "source": [
         "# Now let's plot the likelihood, negative log likelihood, and least squares as a function the value of the standard divation sigma\n",
-        "fig, ax = plt.subplots(1,3)\n",
+        "fig, ax = plt.subplots(1,2)\n",
-        "fig.set_size_inches(10.5, 3.5)\n",
+        "fig.set_size_inches(10.5, 5.5)\n",
-        "fig.tight_layout(pad=3.0)\n",
+        "fig.tight_layout(pad=10.0)\n",
-        "ax[0].plot(sigma_vals, likelihoods); ax[0].set_xlabel('$\\sigma$'); ax[0].set_ylabel('likelihood')\n",
+        "likelihood_color = 'tab:red'\n",
-        "ax[1].plot(sigma_vals, nlls); ax[1].set_xlabel('$\\sigma$'); ax[1].set_ylabel('negative log likelihood')\n",
+        "nll_color = 'tab:blue'\n",
-        "ax[2].plot(sigma_vals, sum_squares); ax[2].set_xlabel('$\\sigma$'); ax[2].set_ylabel('sum of squares')\n",
+        "\n",
+        "\n",
+        "ax[0].set_xlabel('sigma')\n",
+        "ax[0].set_ylabel('likelihood', color = likelihood_color)\n",
+        "ax[0].plot(sigma_vals, likelihoods, color = likelihood_color)\n",
+        "ax[0].tick_params(axis='y', labelcolor=likelihood_color)\n",
+        "\n",
+        "ax00 = ax[0].twinx()\n",
+        "ax00.plot(sigma_vals, nlls, color = nll_color)\n",
+        "ax00.set_ylabel('negative log likelihood', color = nll_color)\n",
+        "ax00.tick_params(axis='y', labelcolor = nll_color)\n",
+        "\n",
+        "plt.axvline(x = sigma_vals[np.argmax(likelihoods)], linestyle='dotted')\n",
+        "\n",
+        "ax[1].plot(sigma_vals, sum_squares); ax[1].set_xlabel('sigma'); ax[1].set_ylabel('sum of squares')\n",
         "plt.show()"
       ],
       "metadata": {
@@ -539,8 +565,8 @@
         "# Hopefully, you can see that the maximum of the likelihood fn is at the same position as the minimum negative log likelihood\n",
         "# The least squares solution does not depend on sigma, so it's just flat -- no use here.\n",
         "# Let's check that:\n",
-        "print(\"Maximum likelihood = %3.3f, at beta_1=%3.3f\"%( (likelihoods[np.argmax(likelihoods)],sigma_vals[np.argmax(likelihoods)])))\n",
+        "print(\"Maximum likelihood = %3.3f, at sigma=%3.3f\"%( (likelihoods[np.argmax(likelihoods)],sigma_vals[np.argmax(likelihoods)])))\n",
-        "print(\"Minimum negative log likelihood = %3.3f, at beta_1=%3.3f\"%( (nlls[np.argmin(nlls)],sigma_vals[np.argmin(nlls)])))\n",
+        "print(\"Minimum negative log likelihood = %3.3f, at sigma=%3.3f\"%( (nlls[np.argmin(nlls)],sigma_vals[np.argmin(nlls)])))\n",
         "# Plot the best model\n",
         "sigma= sigma_vals[np.argmin(nlls)]\n",
         "y_model = shallow_nn(x_model, beta_0, omega_0, beta_1, omega_1)\n",

Notebooks/Chap05/5_2_Binary_Cross_Entropy_Loss.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOlPP7m+YTLyMPaN0WxRdrb",
+      "authorship_tag": "ABX9TyOSb+W2AOFVQm8FZcHAb2Jq",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -66,7 +66,7 @@
         "  return activation\n",
         "\n",
         "# Define a shallow neural network\n",
-        "def shallow_nn(x, beta_0, omega_0, beta_1, omaga_1):\n",
+        "def shallow_nn(x, beta_0, omega_0, beta_1, omega_1):\n",
         "    # Make sure that input data is (1 x n_data) array\n",
         "    n_data = x.size\n",
         "    x = np.reshape(x,(1,n_data))\n",
@@ -378,12 +378,25 @@
     {
       "cell_type": "code",
       "source": [
-        "# Now let's plot the likelihood, and negative log likelihoods as a function the value of the offset beta1\n",
+        "# Now let's plot the likelihood, negative log likelihood, and least squares as a function the value of the offset beta1\n",
-        "fig, ax = plt.subplots(1,2)\n",
+        "fig, ax = plt.subplots()\n",
-        "fig.set_size_inches(10.5, 3.5)\n",
+        "fig.tight_layout(pad=5.0)\n",
-        "fig.tight_layout(pad=3.0)\n",
+        "likelihood_color = 'tab:red'\n",
-        "ax[0].plot(beta_1_vals, likelihoods); ax[0].set_xlabel('beta_1[0]'); ax[0].set_ylabel('likelihood')\n",
+        "nll_color = 'tab:blue'\n",
-        "ax[1].plot(beta_1_vals, nlls); ax[1].set_xlabel('beta_1[0]'); ax[1].set_ylabel('negative log likelihood')\n",
+        "\n",
+        "\n",
+        "ax.set_xlabel('beta_1[0]')\n",
+        "ax.set_ylabel('likelihood', color = likelihood_color)\n",
+        "ax.plot(beta_1_vals, likelihoods, color = likelihood_color)\n",
+        "ax.tick_params(axis='y', labelcolor=likelihood_color)\n",
+        "\n",
+        "ax1 = ax.twinx()\n",
+        "ax1.plot(beta_1_vals, nlls, color = nll_color)\n",
+        "ax1.set_ylabel('negative log likelihood', color = nll_color)\n",
+        "ax1.tick_params(axis='y', labelcolor = nll_color)\n",
+        "\n",
+        "plt.axvline(x = beta_1_vals[np.argmax(likelihoods)], linestyle='dotted')\n",
+        "\n",
         "plt.show()"
       ],
       "metadata": {

Notebooks/Chap05/5_3_Multiclass_Cross_entropy_Loss.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyPNAZtbS+8jYc+tZqhDHNev",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap05/5_3_Multiclass_Cross_entropy_Loss.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "jSlFkICHwHQF"
+      },
       "source": [
         "# **Notebook 5.3 Multiclass Cross-Entropy Loss**\n",
         "\n",
@@ -36,10 +25,7 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "jSlFkICHwHQF"
-      }
     },
     {
       "cell_type": "code",
@@ -61,6 +47,11 @@
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Fv7SZR3tv7mV"
+      },
+      "outputs": [],
       "source": [
         "# Define the Rectified Linear Unit (ReLU) function\n",
         "def ReLU(preactivation):\n",
@@ -77,15 +68,15 @@
         "    h1 = ReLU(np.matmul(beta_0,np.ones((1,n_data))) + np.matmul(omega_0,x))\n",
         "    model_out = np.matmul(beta_1,np.ones((1,n_data))) + np.matmul(omega_1,h1)\n",
         "    return model_out"
-      ],
+      ]
-      "metadata": {
-        "id": "Fv7SZR3tv7mV"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "pUT9Ain_HRim"
+      },
+      "outputs": [],
       "source": [
         "# Get parameters for model -- we can call this function to easily reset them\n",
         "def get_parameters():\n",
@@ -103,15 +94,15 @@
         "  omega_1[2,0] = 16.0; omega_1[2,1] = -8.0; omega_1[2,2] =-8\n",
         "\n",
         "  return beta_0, omega_0, beta_1, omega_1"
-      ],
+      ]
-      "metadata": {
-        "id": "pUT9Ain_HRim"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "NRR67ri_1TzN"
+      },
+      "outputs": [],
       "source": [
         "# Utility function for plotting data\n",
         "def plot_multiclass_classification(x_model, out_model, lambda_model, x_data = None, y_data = None, title= None):\n",
@@ -148,26 +139,27 @@
         "      if y_data[i] ==2:\n",
         "        ax[1].plot(x_data[i],-0.05, 'b.')\n",
         "  plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "NRR67ri_1TzN"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "PsgLZwsPxauP"
+      },
       "source": [
         "# Multiclass classification\n",
         "\n",
         "For multiclass classification, the network must predict the probability of $K$ classes, using $K$ outputs.  However, these probability must be non-negative and sum to one, and the network outputs can take arbitrary values.  Hence, we pass the outputs through a softmax function which maps $K$ arbitrary values to $K$ non-negative values that sum to one."
-      ],
+      ]
-      "metadata": {
-        "id": "PsgLZwsPxauP"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "uFb8h-9IXnIe"
+      },
+      "outputs": [],
       "source": [
         "# Softmax function that maps a vector of arbitrary values to a vector of values that are positive and sum to one.\n",
         "def softmax(model_out):\n",
@@ -184,15 +176,15 @@
         "  softmax_model_out = np.ones_like(model_out)/ exp_model_out.shape[0]\n",
         "\n",
         "  return softmax_model_out"
-      ],
+      ]
-      "metadata": {
-        "id": "uFb8h-9IXnIe"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "VWzNOt1swFVd"
+      },
+      "outputs": [],
       "source": [
         "\n",
         "# Let's create some 1D training data\n",
@@ -214,62 +206,64 @@
         "model_out= shallow_nn(x_model, beta_0, omega_0, beta_1, omega_1)\n",
         "lambda_model = softmax(model_out)\n",
         "plot_multiclass_classification(x_model, model_out, lambda_model, x_train, y_train)\n"
-      ],
+      ]
-      "metadata": {
-        "id": "VWzNOt1swFVd"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue)   The dots at the bottom show the training data with the same color scheme.  So we want the red curve to be high where there are red dots, the green curve to be high where there are green dots, and the blue curve to be high where there are blue dots  We'll compute the the likelihood and the negative log likelihood."
-      ],
       "metadata": {
         "id": "MvVX6tl9AEXF"
-      }
+      },
+      "source": [
+        "The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue)   The dots at the bottom show the training data with the same color scheme.  So we want the red curve to be high where there are red dots, the green curve to be high where there are green dots, and the blue curve to be high where there are blue dots  We'll compute the the likelihood and the negative log likelihood."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "YaLdRlEX0FkU"
+      },
+      "outputs": [],
       "source": [
         "# Return probability under Categorical distribution for input x\n",
         "# Just take value from row k of lambda param where y =k,\n",
         "def categorical_distribution(y, lambda_param):\n",
         "    return np.array([lambda_param[row, i] for i, row in enumerate (y)])"
-      ],
+      ]
-      "metadata": {
-        "id": "YaLdRlEX0FkU"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4TSL14dqHHbV"
+      },
+      "outputs": [],
       "source": [
         "# Let's double check we get the right answer before proceeding\n",
         "print(\"Correct answer = %3.3f, Your answer = %3.3f\"%(0.2,categorical_distribution(np.array([[0]]),np.array([[0.2],[0.5],[0.3]]))))\n",
         "print(\"Correct answer = %3.3f, Your answer = %3.3f\"%(0.5,categorical_distribution(np.array([[1]]),np.array([[0.2],[0.5],[0.3]]))))\n",
         "print(\"Correct answer = %3.3f, Your answer = %3.3f\"%(0.3,categorical_distribution(np.array([[2]]),np.array([[0.2],[0.5],[0.3]]))))\n",
         "\n"
-      ],
+      ]
-      "metadata": {
-        "id": "4TSL14dqHHbV"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's compute the likelihood using this function"
-      ],
       "metadata": {
         "id": "R5z_0dzQMF35"
-      }
+      },
+      "source": [
+        "Now let's compute the likelihood using this function"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "zpS7o6liCx7f"
+      },
+      "outputs": [],
       "source": [
         "# Return the likelihood of all of the data under the model\n",
         "def compute_likelihood(y_train, lambda_param):\n",
@@ -280,15 +274,15 @@
         "  likelihood = 0\n",
         "\n",
         "  return likelihood"
-      ],
+      ]
-      "metadata": {
-        "id": "zpS7o6liCx7f"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1hQxBLoVNlr2"
+      },
+      "outputs": [],
       "source": [
         "# Let's test this\n",
         "beta_0, omega_0, beta_1, omega_1 = get_parameters()\n",
@@ -299,27 +293,28 @@
         "likelihood = compute_likelihood(y_train, lambda_train)\n",
         "# Let's double check we get the right answer before proceeding\n",
         "print(\"Correct answer = %9.9f, Your answer = %9.9f\"%(0.000000041,likelihood))"
-      ],
+      ]
-      "metadata": {
-        "id": "1hQxBLoVNlr2"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "HzphKgPfOvlk"
+      },
       "source": [
         "You can see that this gives a very small answer, even for this small 1D dataset, and with the model fitting quite well.  This is because it is the product of several probabilities, which are all quite small themselves.\n",
         "This will get out of hand pretty quickly with real datasets -- the likelihood will get so small that we can't represent it with normal finite-precision math\n",
         "\n",
         "This is why we use negative log likelihood"
-      ],
+      ]
-      "metadata": {
-        "id": "HzphKgPfOvlk"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "dsT0CWiKBmTV"
+      },
+      "outputs": [],
       "source": [
         "# Return the negative log likelihood of the data under the model\n",
         "def compute_negative_log_likelihood(y_train, lambda_param):\n",
@@ -329,15 +324,15 @@
         "  nll = 0\n",
         "\n",
         "  return nll"
-      ],
+      ]
-      "metadata": {
-        "id": "dsT0CWiKBmTV"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "nVxUXg9rQmwI"
+      },
+      "outputs": [],
       "source": [
         "# Let's test this\n",
         "beta_0, omega_0, beta_1, omega_1 = get_parameters()\n",
@@ -349,24 +344,25 @@
         "nll = compute_negative_log_likelihood(y_train, lambda_train)\n",
         "# Let's double check we get the right answer before proceeding\n",
         "print(\"Correct answer = %9.9f, Your answer = %9.9f\"%(17.015457867,nll))"
-      ],
+      ]
-      "metadata": {
-        "id": "nVxUXg9rQmwI"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's investigate finding the maximum likelihood / minimum log likelihood solution.  For simplicity, we'll assume that all the parameters are fixed except one and look at how the likelihood and log likelihood change as we manipulate the last parameter.  We'll start with overall y_offset, beta_1 (formerly phi_0)"
-      ],
       "metadata": {
         "id": "OgcRojvPWh4V"
-      }
+      },
+      "source": [
+        "Now let's investigate finding the maximum likelihood / minimum log likelihood solution.  For simplicity, we'll assume that all the parameters are fixed except one and look at how the likelihood and log likelihood change as we manipulate the last parameter.  We'll start with overall y_offset, $\\beta_1$ (formerly $\\phi_0$)"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "pFKtDaAeVU4U"
+      },
+      "outputs": [],
       "source": [
         "# Define a range of values for the parameter\n",
         "beta_1_vals = np.arange(-2,6.0,0.1)\n",
@@ -391,32 +387,45 @@
         "    model_out = shallow_nn(x_model, beta_0, omega_0, beta_1, omega_1)\n",
         "    lambda_model = softmax(model_out)\n",
         "    plot_multiclass_classification(x_model, model_out, lambda_model, x_train, y_train, title=\"beta1[0,0]=%3.3f\"%(beta_1[0,0]))\n"
-      ],
+      ]
-      "metadata": {
-        "id": "pFKtDaAeVU4U"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "# Now let's plot the likelihood, negative log likelihood as a function the value of the offset beta1\n",
-        "fig, ax = plt.subplots(1,2)\n",
-        "fig.set_size_inches(10.5, 3.5)\n",
-        "fig.tight_layout(pad=3.0)\n",
-        "ax[0].plot(beta_1_vals, likelihoods); ax[0].set_xlabel('beta_1[0,0]'); ax[0].set_ylabel('likelihood')\n",
-        "ax[1].plot(beta_1_vals, nlls); ax[1].set_xlabel('beta_1[0,0]'); ax[1].set_ylabel('negative log likelihood')\n",
-        "plt.show()"
-      ],
       "metadata": {
         "id": "UHXeTa9MagO6"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "# Now let's plot the likelihood, negative log likelihood, and least squares as a function the value of the offset beta1\n",
+        "fig, ax = plt.subplots()\n",
+        "fig.tight_layout(pad=5.0)\n",
+        "likelihood_color = 'tab:red'\n",
+        "nll_color = 'tab:blue'\n",
+        "\n",
+        "\n",
+        "ax.set_xlabel('beta_1[0, 0]')\n",
+        "ax.set_ylabel('likelihood', color = likelihood_color)\n",
+        "ax.plot(beta_1_vals, likelihoods, color = likelihood_color)\n",
+        "ax.tick_params(axis='y', labelcolor=likelihood_color)\n",
+        "\n",
+        "ax1 = ax.twinx()\n",
+        "ax1.plot(beta_1_vals, nlls, color = nll_color)\n",
+        "ax1.set_ylabel('negative log likelihood', color = nll_color)\n",
+        "ax1.tick_params(axis='y', labelcolor = nll_color)\n",
+        "\n",
+        "plt.axvline(x = beta_1_vals[np.argmax(likelihoods)], linestyle='dotted')\n",
+        "\n",
+        "plt.show()"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "aDEPhddNdN4u"
+      },
+      "outputs": [],
       "source": [
         "# Hopefully, you can see that the maximum of the likelihood fn is at the same position as the minimum negative log likelihood solution\n",
         "# Let's check that:\n",
@@ -428,24 +437,36 @@
         "model_out = shallow_nn(x_model, beta_0, omega_0, beta_1, omega_1)\n",
         "lambda_model = softmax(model_out)\n",
         "plot_multiclass_classification(x_model, model_out, lambda_model, x_train, y_train, title=\"beta1[0,0]=%3.3f\"%(beta_1[0,0]))\n"
-      ],
+      ]
-      "metadata": {
-        "id": "aDEPhddNdN4u"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "771G8N1Vk5A2"
+      },
       "source": [
         "They both give the same answer. But you can see from the likelihood above that the likelihood is very small unless the parameters are almost correct.  So in practice, we would work with the negative log likelihood.<br><br>\n",
         "\n",
         "Again, to fit the full neural model we would vary all of the 16 parameters of the network in the $\\boldsymbol\\beta_{0},\\boldsymbol\\omega_{0},\\boldsymbol\\beta_{1},\\boldsymbol\\omega_{1}$ until we find the combination that have the maximum likelihood / minimum negative log likelihood.<br><br>\n",
         "\n"
+      ]
+    }
   ],
   "metadata": {
-        "id": "771G8N1Vk5A2"
+    "colab": {
+      "authorship_tag": "ABX9TyOPv/l+ToaApJV7Nz+8AtpV",
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
     }
-    }
+  },
-  ]
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap06/6_2_Gradient_Descent.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyN2N4cCnlIobOZXEjcwAvZ5",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap06/6_2_Gradient_Descent.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "el8l05WQEO46"
+      },
       "source": [
         "# **Notebook 6.2 Gradient descent**\n",
         "\n",
@@ -37,10 +26,7 @@
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
         "\n"
-      ],
+      ]
-      "metadata": {
-        "id": "el8l05WQEO46"
-      }
     },
     {
       "cell_type": "code",
@@ -59,34 +45,39 @@
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4cRkrh9MZ58Z"
+      },
+      "outputs": [],
       "source": [
         "# Let's create our training data 12 pairs {x_i, y_i}\n",
         "# We'll try to fit the straight line model to these data\n",
         "data = np.array([[0.03,0.19,0.34,0.46,0.78,0.81,1.08,1.18,1.39,1.60,1.65,1.90],\n",
         "                 [0.67,0.85,1.05,1.00,1.40,1.50,1.30,1.54,1.55,1.68,1.73,1.60]])"
-      ],
+      ]
-      "metadata": {
-        "id": "4cRkrh9MZ58Z"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "WQUERmb2erAe"
+      },
+      "outputs": [],
       "source": [
         "# Let's define our model -- just a straight line with intercept phi[0] and slope phi[1]\n",
         "def model(phi,x):\n",
         "  y_pred = phi[0]+phi[1] * x\n",
         "  return y_pred"
-      ],
+      ]
-      "metadata": {
-        "id": "WQUERmb2erAe"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "qFRe9POHF2le"
+      },
+      "outputs": [],
       "source": [
         "# Draw model\n",
         "def draw_model(data,model,phi,title=None):\n",
@@ -102,39 +93,40 @@
         "  if title is not None:\n",
         "    ax.set_title(title)\n",
         "  plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "qFRe9POHF2le"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "TXx1Tpd1Tl-I"
+      },
+      "outputs": [],
       "source": [
         "# Initialize the parameters to some arbitrary values and draw the model\n",
         "phi = np.zeros((2,1))\n",
         "phi[0] = 0.6      # Intercept\n",
         "phi[1] = -0.2      # Slope\n",
         "draw_model(data,model,phi, \"Initial parameters\")\n"
-      ],
+      ]
-      "metadata": {
-        "id": "TXx1Tpd1Tl-I"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now lets create compute the sum of squares loss for the training data"
-      ],
       "metadata": {
         "id": "QU5mdGvpTtEG"
-      }
+      },
+      "source": [
+        "Now lets create compute the sum of squares loss for the training data"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "I7dqTY2Gg7CR"
+      },
+      "outputs": [],
       "source": [
         "def compute_loss(data_x, data_y, model, phi):\n",
         "  # TODO -- Write this function -- replace the line below\n",
@@ -145,45 +137,47 @@
         "  loss = 0\n",
         "\n",
         "  return loss"
-      ],
+      ]
-      "metadata": {
-        "id": "I7dqTY2Gg7CR"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Let's just test that we got that right"
-      ],
       "metadata": {
         "id": "eB5DQvU5hYNx"
-      }
+      },
+      "source": [
+        "Let's just test that we got that right"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "loss = compute_loss(data[0,:],data[1,:],model,np.array([[0.6],[-0.2]]))\n",
-        "print('Your loss = %3.3f, Correct loss = %3.3f'%(loss, 12.367))"
-      ],
       "metadata": {
         "id": "Ty05UtEEg9tc"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "loss = compute_loss(data[0,:],data[1,:],model,np.array([[0.6],[-0.2]]))\n",
+        "print('Your loss = %3.3f, Correct loss = %3.3f'%(loss, 12.367))"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's plot the whole loss function"
-      ],
       "metadata": {
         "id": "F3trnavPiHpH"
-      }
+      },
+      "source": [
+        "Now let's plot the whole loss function"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "K-NTHpAAHlCl"
+      },
+      "outputs": [],
       "source": [
         "def draw_loss_function(compute_loss, data,  model, phi_iters = None):\n",
         "  # Define pretty colormap\n",
@@ -210,39 +204,40 @@
         "  ax.set_ylim([1,-1])\n",
         "  ax.set_xlabel('Intercept $\\phi_{0}$'); ax.set_ylabel('Slope, $\\phi_{1}$')\n",
         "  plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "K-NTHpAAHlCl"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "draw_loss_function(compute_loss, data, model)"
-      ],
       "metadata": {
         "id": "l8HbvIupnTME"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "draw_loss_function(compute_loss, data, model)"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "s9Duf05WqqSC"
+      },
       "source": [
         "Now let's compute the gradient vector for a given set of parameters:\n",
         "\n",
         "\\begin{equation}\n",
         "\\frac{\\partial L}{\\partial \\boldsymbol\\phi} = \\begin{bmatrix}\\frac{\\partial L}{\\partial \\phi_0} \\\\\\frac{\\partial L}{\\partial \\phi_1} \\end{bmatrix}.\n",
         "\\end{equation}"
-      ],
+      ]
-      "metadata": {
-        "id": "s9Duf05WqqSC"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "UpswmkL2qwBT"
+      },
+      "outputs": [],
       "source": [
         "# These are in the lecture slides and notes, but worth trying to calculate them yourself to\n",
         "# check that you get them right.  Write out the expression for the sum of squares loss and take the\n",
@@ -254,31 +249,32 @@
         "\n",
         "    # Return the gradient\n",
         "    return np.array([[dl_dphi0],[dl_dphi1]])"
-      ],
+      ]
-      "metadata": {
-        "id": "UpswmkL2qwBT"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "RS1nEcYVuEAM"
+      },
       "source": [
         "We can check we got this right using a trick known as **finite differences**.  If we evaluate the function and then change one of the parameters by a very small amount and normalize by that amount, we get an approximation to the gradient, so:\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
         "\\frac{\\partial L}{\\partial \\phi_{0}}&\\approx & \\frac{L[\\phi_0+\\delta, \\phi_1]-L[\\phi_0, \\phi_1]}{\\delta}\\\\\n",
         "\\frac{\\partial L}{\\partial \\phi_{1}}&\\approx & \\frac{L[\\phi_0, \\phi_1+\\delta]-L[\\phi_0, \\phi_1]}{\\delta}\n",
-        "\\end{eqnarray}\n",
+        "\\end{align}\n",
         "\n",
         "We can't do this when there are many parameters;  for a million parameters, we would have to evaluate the loss function two million times, and usually computing the gradients directly is much more efficient."
-      ],
+      ]
-      "metadata": {
-        "id": "RS1nEcYVuEAM"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "QuwAHN7yt-gi"
+      },
+      "outputs": [],
       "source": [
         "# Compute the gradient using your function\n",
         "gradient = compute_gradient(data[0,:],data[1,:], phi)\n",
@@ -291,28 +287,29 @@
         "                    compute_loss(data[0,:],data[1,:],model,phi))/delta\n",
         "print(\"Approx gradients: (%3.3f,%3.3f)\"%(dl_dphi0_est,dl_dphi1_est))\n",
         "# There might be small differences in the last significant figure because finite gradients is an approximation\n"
-      ],
+      ]
-      "metadata": {
-        "id": "QuwAHN7yt-gi"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now we are ready to perform gradient descent.  We'll need to use our line search routine from noteboo 6.1, which I've reproduced here plus the helper function loss_function_1D that converts from a 2D problem to a 1D problem"
-      ],
       "metadata": {
         "id": "5EIjMM9Fw2eT"
-      }
+      },
+      "source": [
+        "Now we are ready to perform gradient descent.  We'll need to use our line search routine from notebook 6.1, which I've reproduced here plus the helper function loss_function_1D that maps the search along the negative gradient direction in 2D space to a 1D problem (distance along this direction)"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "XrJ2gQjfw1XP"
+      },
+      "outputs": [],
       "source": [
-        "def loss_function_1D(dist_prop, data, model, phi_start, gradient):\n",
+        "def loss_function_1D(dist_prop, data, model, phi_start, search_direction):\n",
         "  # Return the loss after moving this far\n",
-        "  return compute_loss(data[0,:], data[1,:], model, phi_start+ gradient * dist_prop)\n",
+        "  return compute_loss(data[0,:], data[1,:], model, phi_start+ search_direction * dist_prop)\n",
         "\n",
         "def line_search(data, model, phi, gradient, thresh=.00001, max_dist = 0.1, max_iter = 15, verbose=False):\n",
         "    # Initialize four points along the range we are going to search\n",
@@ -363,15 +360,15 @@
         "\n",
         "    # Return average of two middle points\n",
         "    return (b+c)/2.0"
-      ],
+      ]
-      "metadata": {
-        "id": "XrJ2gQjfw1XP"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "YVq6rmaWRD2M"
+      },
+      "outputs": [],
       "source": [
         "def gradient_descent_step(phi, data,  model):\n",
         "  # TODO -- update Phi with the gradient descent step (equation 6.3)\n",
@@ -380,15 +377,15 @@
         "  # 3. Update the parameters phi based on the gradient and the step size alpha.\n",
         "\n",
         "  return phi"
-      ],
+      ]
-      "metadata": {
-        "id": "YVq6rmaWRD2M"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "tOLd0gtdRLLS"
+      },
+      "outputs": [],
       "source": [
         "# Initialize the parameters and draw the model\n",
         "n_steps = 10\n",
@@ -410,12 +407,22 @@
         "\n",
         "# Draw the trajectory on the loss function\n",
         "draw_loss_function(compute_loss, data, model,phi_all)\n"
+      ]
+    }
   ],
   "metadata": {
-        "id": "tOLd0gtdRLLS"
+    "colab": {
+      "include_colab_link": true,
+      "provenance": []
     },
-      "execution_count": null,
+    "kernelspec": {
-      "outputs": []
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
     }
-  ]
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap06/6_3_Stochastic_Gradient_Descent.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyNk5FN4qlw3pk8BwDVWw1jN",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap06/6_3_Stochastic_Gradient_Descent.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "el8l05WQEO46"
+      },
       "source": [
         "# **Notebook 6.3: Stochastic gradient descent**\n",
         "\n",
@@ -39,10 +28,7 @@
         "\n",
         "\n",
         "\n"
-      ],
+      ]
-      "metadata": {
-        "id": "el8l05WQEO46"
-      }
     },
     {
       "cell_type": "code",
@@ -61,6 +47,11 @@
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4cRkrh9MZ58Z"
+      },
+      "outputs": [],
       "source": [
         "# Let's create our training data 30 pairs {x_i, y_i}\n",
         "# We'll try to fit the Gabor model to these data\n",
@@ -74,15 +65,15 @@
         "                  -2.365e-02,5.098e-01,-2.777e-01,3.367e-01,1.927e-01,-2.222e-01,\n",
         "                  6.352e-02,6.888e-03,3.224e-02,1.091e-02,-5.706e-01,-5.258e-02,\n",
         "                  -3.666e-02,1.709e-01,-4.805e-02,2.008e-01,-1.904e-01,5.952e-01]])"
-      ],
+      ]
-      "metadata": {
-        "id": "4cRkrh9MZ58Z"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "WQUERmb2erAe"
+      },
+      "outputs": [],
       "source": [
         "# Let's define our model\n",
         "def model(phi,x):\n",
@@ -90,15 +81,15 @@
         "  gauss_component = np.exp(-(phi[0] + 0.06 * phi[1] * x) * (phi[0] + 0.06 * phi[1] * x) / 32)\n",
         "  y_pred= sin_component * gauss_component\n",
         "  return y_pred"
-      ],
+      ]
-      "metadata": {
-        "id": "WQUERmb2erAe"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "qFRe9POHF2le"
+      },
+      "outputs": [],
       "source": [
         "# Draw model\n",
         "def draw_model(data,model,phi,title=None):\n",
@@ -113,39 +104,40 @@
         "  if title is not None:\n",
         "    ax.set_title(title)\n",
         "  plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "qFRe9POHF2le"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "TXx1Tpd1Tl-I"
+      },
+      "outputs": [],
       "source": [
         "# Initialize the parameters and draw the model\n",
         "phi = np.zeros((2,1))\n",
         "phi[0] =  -5     # Horizontal offset\n",
         "phi[1] =  25     # Frequency\n",
         "draw_model(data,model,phi, \"Initial parameters\")\n"
-      ],
+      ]
-      "metadata": {
-        "id": "TXx1Tpd1Tl-I"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now lets create compute the sum of squares loss for the training data"
-      ],
       "metadata": {
         "id": "QU5mdGvpTtEG"
-      }
+      },
+      "source": [
+        "Now lets create compute the sum of squares loss for the training data"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "I7dqTY2Gg7CR"
+      },
+      "outputs": [],
       "source": [
         "def compute_loss(data_x, data_y, model, phi):\n",
         "  # TODO -- Write this function -- replace the line below\n",
@@ -155,45 +147,47 @@
         "  loss = 0\n",
         "\n",
         "  return loss"
-      ],
+      ]
-      "metadata": {
-        "id": "I7dqTY2Gg7CR"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Let's just test that we got that right"
-      ],
       "metadata": {
         "id": "eB5DQvU5hYNx"
-      }
+      },
+      "source": [
+        "Let's just test that we got that right"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "loss = compute_loss(data[0,:],data[1,:],model,np.array([[0.6],[-0.2]]))\n",
-        "print('Your loss = %3.3f, Correct loss = %3.3f'%(loss, 16.419))"
-      ],
       "metadata": {
         "id": "Ty05UtEEg9tc"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "loss = compute_loss(data[0,:],data[1,:],model,np.array([[0.6],[-0.2]]))\n",
+        "print('Your loss = %3.3f, Correct loss = %3.3f'%(loss, 16.419))"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's plot the whole loss function"
-      ],
       "metadata": {
         "id": "F3trnavPiHpH"
-      }
+      },
+      "source": [
+        "Now let's plot the whole loss function"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "K-NTHpAAHlCl"
+      },
+      "outputs": [],
       "source": [
         "def draw_loss_function(compute_loss, data,  model, phi_iters = None):\n",
         "  # Define pretty colormap\n",
@@ -220,39 +214,40 @@
         "  ax.set_ylim([2.5,22.5])\n",
         "  ax.set_xlabel('Offset $\\phi_{0}$'); ax.set_ylabel('Frequency, $\\phi_{1}$')\n",
         "  plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "K-NTHpAAHlCl"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "draw_loss_function(compute_loss, data, model)"
-      ],
       "metadata": {
         "id": "l8HbvIupnTME"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "draw_loss_function(compute_loss, data, model)"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "s9Duf05WqqSC"
+      },
       "source": [
         "Now let's compute the gradient vector for a given set of parameters:\n",
         "\n",
         "\\begin{equation}\n",
         "\\frac{\\partial L}{\\partial \\boldsymbol\\phi} = \\begin{bmatrix}\\frac{\\partial L}{\\partial \\phi_0} \\\\\\frac{\\partial L}{\\partial \\phi_1} \\end{bmatrix}.\n",
         "\\end{equation}"
-      ],
+      ]
-      "metadata": {
-        "id": "s9Duf05WqqSC"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "UpswmkL2qwBT"
+      },
+      "outputs": [],
       "source": [
         "# These came from writing out the expression for the sum of squares loss and taking the\n",
         "# derivative with respect to phi0 and phi1. It was a lot of hassle to get it right!\n",
@@ -281,31 +276,32 @@
         "    dl_dphi1 = gabor_deriv_phi1(data_x, data_y, phi[0],phi[1])\n",
         "    # Return the gradient\n",
         "    return np.array([[dl_dphi0],[dl_dphi1]])"
-      ],
+      ]
-      "metadata": {
-        "id": "UpswmkL2qwBT"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "RS1nEcYVuEAM"
+      },
       "source": [
         "We can check we got this right using a trick known as **finite differences**.  If we evaluate the function and then change one of the parameters by a very small amount and normalize by that amount, we get an approximation to the gradient, so:\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
         "\\frac{\\partial L}{\\partial \\phi_{0}}&\\approx & \\frac{L[\\phi_0+\\delta, \\phi_1]-L[\\phi_0, \\phi_1]}{\\delta}\\\\\n",
         "\\frac{\\partial L}{\\partial \\phi_{1}}&\\approx & \\frac{L[\\phi_0, \\phi_1+\\delta]-L[\\phi_0, \\phi_1]}{\\delta}\n",
-        "\\end{eqnarray}\n",
+        "\\end{align}\n",
         "\n",
         "We can't do this when there are many parameters;  for a million parameters, we would have to evaluate the loss function two million times, and usually computing the gradients directly is much more efficient."
-      ],
+      ]
-      "metadata": {
-        "id": "RS1nEcYVuEAM"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "QuwAHN7yt-gi"
+      },
+      "outputs": [],
       "source": [
         "# Compute the gradient using your function\n",
         "gradient = compute_gradient(data[0,:],data[1,:], phi)\n",
@@ -317,24 +313,25 @@
         "dl_dphi1_est = (compute_loss(data[0,:],data[1,:],model,phi+np.array([[0],[delta]])) - \\\n",
         "                    compute_loss(data[0,:],data[1,:],model,phi))/delta\n",
         "print(\"Approx gradients: (%3.3f,%3.3f)\"%(dl_dphi0_est,dl_dphi1_est))\n"
-      ],
+      ]
-      "metadata": {
-        "id": "QuwAHN7yt-gi"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now we are ready to perform gradient descent.  We'll need to use our line search routine from Notebook 6.1, which I've reproduced here plus the helper function loss_function_1D that converts from a 2D problem to a 1D problem"
-      ],
       "metadata": {
         "id": "5EIjMM9Fw2eT"
-      }
+      },
+      "source": [
+        "Now we are ready to perform gradient descent.  We'll need to use our line search routine from Notebook 6.1, which I've reproduced here plus the helper function loss_function_1D that converts from a 2D problem to a 1D problem"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "XrJ2gQjfw1XP"
+      },
+      "outputs": [],
       "source": [
         "def loss_function_1D(dist_prop, data, model, phi_start, gradient):\n",
         "  # Return the loss after moving this far\n",
@@ -389,15 +386,15 @@
         "\n",
         "    # Return average of two middle points\n",
         "    return (b+c)/2.0"
-      ],
+      ]
-      "metadata": {
-        "id": "XrJ2gQjfw1XP"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "YVq6rmaWRD2M"
+      },
+      "outputs": [],
       "source": [
         "def gradient_descent_step(phi, data,  model):\n",
         "  # Step 1:  Compute the gradient\n",
@@ -406,15 +403,15 @@
         "  alpha = line_search(data, model, phi, gradient*-1, max_dist = 2.0)\n",
         "  phi = phi - alpha * gradient\n",
         "  return phi"
-      ],
+      ]
-      "metadata": {
-        "id": "YVq6rmaWRD2M"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "tOLd0gtdRLLS"
+      },
+      "outputs": [],
       "source": [
         "# Initialize the parameters\n",
         "n_steps = 21\n",
@@ -435,41 +432,41 @@
         "    draw_model(data,model,phi_all[:,c_step+1], \"Iteration %d, loss = %f\"%(c_step+1,loss))\n",
         "\n",
         "draw_loss_function(compute_loss, data, model,phi_all)\n"
-      ],
+      ]
-      "metadata": {
-        "id": "tOLd0gtdRLLS"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "# TODO Experiment with starting the optimization in the previous cell in different places\n",
-        "# and show that it heads to a local minimum if we don't start it in the right valley"
-      ],
       "metadata": {
         "id": "Oi8ZlH0ptLqA"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "# TODO Experiment with starting the optimization in the previous cell in different places\n",
+        "# and show that it heads to a local minimum if we don't start it in the right valley"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4l-ueLk-oAxV"
+      },
+      "outputs": [],
       "source": [
         "def gradient_descent_step_fixed_learning_rate(phi, data, alpha):\n",
         "  # TODO -- fill in this routine so that we take a fixed size step of size alpha without using line search\n",
         "\n",
         "  return phi"
-      ],
+      ]
-      "metadata": {
-        "id": "4l-ueLk-oAxV"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "oi9MX_GRpM41"
+      },
+      "outputs": [],
       "source": [
         "# Initialize the parameters\n",
         "n_steps = 21\n",
@@ -490,47 +487,47 @@
         "    draw_model(data,model,phi_all[:,c_step+1], \"Iteration %d, loss = %f\"%(c_step+1,loss))\n",
         "\n",
         "draw_loss_function(compute_loss, data, model,phi_all)\n"
-      ],
+      ]
-      "metadata": {
-        "id": "oi9MX_GRpM41"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "In6sQ5YCpMqn"
+      },
+      "outputs": [],
       "source": [
         "# TODO Experiment with the learning rate, alpha.\n",
         "# What happens if you set it too large?\n",
         "# What happens if you set it too small?"
-      ],
+      ]
-      "metadata": {
-        "id": "In6sQ5YCpMqn"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "VKTC9-1Gpm3N"
+      },
+      "outputs": [],
       "source": [
         "def stochastic_gradient_descent_step(phi, data, alpha, batch_size):\n",
         "  # TODO -- fill in this routine so that we take a fixed size step of size alpha but only using a subset (batch) of the data\n",
         "  # at each step\n",
         "  # You can use the function np.random.permutation to generate a random permutation of the n_data = data.shape[1] indices\n",
         "  # and then just choose the first n=batch_size of these indices.  Then compute the gradient update\n",
-        "  # from just the data with these indices.   More properly, you should sample with replacement, but this will do for now.\n",
+        "  # from just the data with these indices.   More properly, you should sample without replacement, but this will do for now.\n",
         "\n",
         "\n",
         "  return phi"
-      ],
+      ]
-      "metadata": {
-        "id": "VKTC9-1Gpm3N"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "469OP_UHskJ4"
+      },
+      "outputs": [],
       "source": [
         "# Set the random number generator so you always get same numbers (disable if you don't want this)\n",
         "np.random.seed(1)\n",
@@ -553,34 +550,45 @@
         "    draw_model(data,model,phi_all[:,c_step+1], \"Iteration %d, loss = %f\"%(c_step+1,loss))\n",
         "\n",
         "draw_loss_function(compute_loss, data, model,phi_all)"
-      ],
+      ]
-      "metadata": {
-        "id": "469OP_UHskJ4"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "# TODO -- Experiment with different learning rates, starting points, batch sizes, number of steps.  Get a feel for this."
-      ],
       "metadata": {
         "id": "LxE2kTa3s29p"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "# TODO -- Experiment with different learning rates, starting points, batch sizes, number of steps.  Get a feel for this."
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "# TODO -- Add a learning rate schedule.  Reduce the learning rate by a factor of beta every M iterations"
-      ],
       "metadata": {
         "id": "lw4QPOaQTh5e"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
-    }
+        "# TODO -- Add a learning rate schedule.  Reduce the learning rate by a factor of beta every M iterations"
       ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "authorship_tag": "ABX9TyNk5FN4qlw3pk8BwDVWw1jN",
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap06/6_4_Momentum.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyMLS4qeqBTVHGdg9Sds9jND",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -377,6 +376,15 @@
       },
       "execution_count": null,
       "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Note that for this case, Nesterov momentum does not improve the result."
+      ],
+      "metadata": {
+        "id": "F-As4hS8s2nm"
+      }
     }
   ]
 }

Notebooks/Chap07/7_1_Backpropagation_in_Toy_Model.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyOjXmTmoff61y15VqEB5sDW",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap07/7_1_Backpropagation_in_Toy_Model.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "pOZ6Djz0dhoy"
+      },
       "source": [
         "# **Notebook 7.1: Backpropagation in Toy Model**\n",
         "\n",
@@ -36,68 +25,67 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "pOZ6Djz0dhoy"
-      }
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "1DmMo2w63CmT"
+      },
       "source": [
         "We're going to investigate how to take the derivatives of functions where one operation is composed with another, which is composed with a third and so on.  For example, consider the model:\n",
         "\n",
         "\\begin{equation}\n",
-        "     \\mbox{f}[x,\\boldsymbol\\phi] = \\beta_3+\\omega_3\\cdot\\cos\\Bigl[\\beta_2+\\omega_2\\cdot\\exp\\bigl[\\beta_1+\\omega_1\\cdot\\sin[\\beta_0+\\omega_0x]\\bigr]\\Bigr],\n",
+        "     \\text{f}[x,\\boldsymbol\\phi] = \\beta_3+\\omega_3\\cdot\\cos\\Bigl[\\beta_2+\\omega_2\\cdot\\exp\\bigl[\\beta_1+\\omega_1\\cdot\\sin[\\beta_0+\\omega_0x]\\bigr]\\Bigr],\n",
         "\\end{equation}\n",
         "\n",
         "with parameters $\\boldsymbol\\phi=\\{\\beta_0,\\omega_0,\\beta_1,\\omega_1,\\beta_2,\\omega_2,\\beta_3,\\omega_3\\}$.<br>\n",
         "\n",
         "This is a composition of the functions $\\cos[\\bullet],\\exp[\\bullet],\\sin[\\bullet]$.   I chose these just because you probably already know the derivatives of these functions:\n",
         "\n",
-        "\\begin{eqnarray*}\n",
+        "\\begin{align}\n",
         " \\frac{\\partial \\cos[z]}{\\partial z} = -\\sin[z] \\quad\\quad \\frac{\\partial \\exp[z]}{\\partial z} = \\exp[z] \\quad\\quad \\frac{\\partial \\sin[z]}{\\partial z} = \\cos[z].\n",
-        "\\end{eqnarray*}\n",
+        "\\end{align}\n",
         "\n",
         "Suppose that we have a least squares loss function:\n",
         "\n",
         "\\begin{equation*}\n",
-        "\\ell_i = (\\mbox{f}[x_i,\\boldsymbol\\phi]-y_i)^2,\n",
+        "\\ell_i = (\\text{f}[x_i,\\boldsymbol\\phi]-y_i)^2,\n",
         "\\end{equation*}\n",
         "\n",
         "Assume that we know the current values of $\\beta_{0},\\beta_{1},\\beta_{2},\\beta_{3},\\omega_{0},\\omega_{1},\\omega_{2},\\omega_{3}$, $x_i$ and $y_i$. We could obviously calculate $\\ell_i$.   But we also want to know how $\\ell_i$ changes when we make a small change to $\\beta_{0},\\beta_{1},\\beta_{2},\\beta_{3},\\omega_{0},\\omega_{1},\\omega_{2}$, or $\\omega_{3}$.  In other words, we want to compute the eight derivatives:\n",
         "\n",
-        "\\begin{eqnarray*}\n",
+        "\\begin{align}\n",
-        "\\frac{\\partial \\ell_i}{\\partial \\beta_{0}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\beta_{1}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\beta_{2}}, \\quad \\frac{\\partial \\ell_i }{\\partial \\beta_{3}},  \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{0}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{1}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{2}},  \\quad\\mbox{and} \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{3}}.\n",
+        "\\frac{\\partial \\ell_i}{\\partial \\beta_{0}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\beta_{1}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\beta_{2}}, \\quad \\frac{\\partial \\ell_i }{\\partial \\beta_{3}},  \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{0}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{1}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{2}},  \\quad\\text{and} \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{3}}.\n",
-        "\\end{eqnarray*}"
+        "\\end{align}"
-      ],
+      ]
-      "metadata": {
-        "id": "1DmMo2w63CmT"
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "# import library\n",
-        "import numpy as np"
-      ],
       "metadata": {
         "id": "RIPaoVN834Lj"
       },
-      "execution_count": 1,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "# import library\n",
+        "import numpy as np"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Let's first define the original function for $y$ and the loss term:"
-      ],
       "metadata": {
         "id": "32-ufWhc3v2c"
-      }
+      },
+      "source": [
+        "Let's first define the original function for $y$ and the loss term:"
+      ]
     },
     {
       "cell_type": "code",
-      "execution_count": 2,
+      "execution_count": null,
       "metadata": {
         "id": "AakK_qen3BpU"
       },
@@ -112,121 +100,129 @@
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now we'll choose some values for the betas and the omegas and x and compute the output of the function:"
-      ],
       "metadata": {
         "id": "y7tf0ZMt5OXt"
-      }
+      },
+      "source": [
+        "Now we'll choose some values for the betas and the omegas and x and compute the output of the function:"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "pwvOcCxr41X_",
+        "outputId": "9541922c-dfc4-4b2e-dfa3-3298812155ce"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "l_i=0.139\n"
+          ]
+        }
+      ],
       "source": [
         "beta0 = 1.0; beta1 = 2.0; beta2 = -3.0; beta3 = 0.4\n",
         "omega0 = 0.1; omega1 = -0.4; omega2 = 2.0; omega3 = 3.0\n",
         "x = 2.3; y =2.0\n",
         "l_i_func = loss(x,y,beta0,beta1,beta2,beta3,omega0,omega1,omega2,omega3)\n",
         "print('l_i=%3.3f'%l_i_func)"
-      ],
-      "metadata": {
-        "id": "pwvOcCxr41X_",
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "outputId": "9541922c-dfc4-4b2e-dfa3-3298812155ce"
-      },
-      "execution_count": 3,
-      "outputs": [
-        {
-          "output_type": "stream",
-          "name": "stdout",
-          "text": [
-            "l_i=0.139\n"
-          ]
-        }
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "u5w69NeT64yV"
+      },
       "source": [
         "# Computing derivatives by hand\n",
         "\n",
         "We could compute expressions for the derivatives by hand and write code to compute them directly but some have very complex expressions, even for this relatively simple original equation. For example:\n",
         "\n",
-        "\\begin{eqnarray*}\n",
+        "\\begin{align}\n",
         "\\frac{\\partial \\ell_i}{\\partial \\omega_{0}} &=& -2 \\left( \\beta_3+\\omega_3\\cdot\\cos\\Bigl[\\beta_2+\\omega_2\\cdot\\exp\\bigl[\\beta_1+\\omega_1\\cdot\\sin[\\beta_0+\\omega_0\\cdot x_i]\\bigr]\\Bigr]-y_i\\right)\\nonumber \\\\\n",
         "&&\\hspace{0.5cm}\\cdot \\omega_1\\omega_2\\omega_3\\cdot x_i\\cdot\\cos[\\beta_0+\\omega_0 \\cdot x_i]\\cdot\\exp\\Bigl[\\beta_1 + \\omega_1 \\cdot \\sin[\\beta_0+\\omega_0\\cdot x_i]\\Bigr]\\nonumber\\\\\n",
         "&& \\hspace{1cm}\\cdot \\sin\\biggl[\\beta_2+\\omega_2\\cdot \\exp\\Bigl[\\beta_1 + \\omega_1 \\cdot \\sin[\\beta_0+\\omega_0\\cdot x_i]\\Bigr]\\biggr].\n",
-        "\\end{eqnarray*}"
+        "\\end{align}"
-      ],
+      ]
-      "metadata": {
-        "id": "u5w69NeT64yV"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7t22hALp5zkq"
+      },
+      "outputs": [],
       "source": [
         "dldbeta3_func = 2 * (beta3 +omega3 * np.cos(beta2 + omega2 * np.exp(beta1+omega1 * np.sin(beta0+omega0 * x)))-y)\n",
         "dldomega0_func = -2 *(beta3 +omega3 * np.cos(beta2 + omega2 * np.exp(beta1+omega1 * np.sin(beta0+omega0 * x)))-y) * \\\n",
         "              omega1 * omega2 * omega3 * x * np.cos(beta0 + omega0 * x) * np.exp(beta1 +omega1 * np.sin(beta0 + omega0 * x)) *\\\n",
         "              np.sin(beta2 + omega2 * np.exp(beta1+ omega1* np.sin(beta0+omega0 * x)))"
-      ],
+      ]
-      "metadata": {
-        "id": "7t22hALp5zkq"
-      },
-      "execution_count": 4,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Let's make sure this is correct using finite differences:"
-      ],
       "metadata": {
         "id": "iRh4hnu3-H3n"
-      }
+      },
+      "source": [
+        "Let's make sure this is correct using finite differences:"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "dldomega0_fd = (loss(x,y,beta0,beta1,beta2,beta3,omega0+0.00001,omega1,omega2,omega3)-loss(x,y,beta0,beta1,beta2,beta3,omega0,omega1,omega2,omega3))/0.00001\n",
-        "\n",
-        "print('dydomega0: Function value = %3.3f, Finite difference value = %3.3f'%(dldomega0_func,dldomega0_fd))"
-      ],
       "metadata": {
-        "id": "1O3XmXMx-HlZ",
         "colab": {
           "base_uri": "https://localhost:8080/"
         },
+        "id": "1O3XmXMx-HlZ",
         "outputId": "389ed78e-9d8d-4e8b-9e6b-5f20c21407e8"
       },
-      "execution_count": 5,
       "outputs": [
         {
-          "output_type": "stream",
           "name": "stdout",
+          "output_type": "stream",
           "text": [
             "dydomega0: Function value = 5.246, Finite difference value = 5.246\n"
           ]
         }
+      ],
+      "source": [
+        "dldomega0_fd = (loss(x,y,beta0,beta1,beta2,beta3,omega0+0.00001,omega1,omega2,omega3)-loss(x,y,beta0,beta1,beta2,beta3,omega0,omega1,omega2,omega3))/0.00001\n",
+        "\n",
+        "print('dydomega0: Function value = %3.3f, Finite difference value = %3.3f'%(dldomega0_func,dldomega0_fd))"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "The code to calculate $\\partial l_i/ \\partial \\omega_0$ is a bit of a nightmare.  It's easy to make mistakes, and you can see that some parts of it are repeated (for example, the $\\sin[\\bullet]$ term), which suggests some kind of redundancy in the calculations.  The goal of this practical is to compute the derivatives in a much simpler way.  There will be three steps:"
-      ],
       "metadata": {
         "id": "wS4IPjZAKWTN"
-      }
+      },
+      "source": [
+        "The code to calculate $\\partial l_i/ \\partial \\omega_0$ is a bit of a nightmare.  It's easy to make mistakes, and you can see that some parts of it are repeated (for example, the $\\sin[\\bullet]$ term), which suggests some kind of redundancy in the calculations.  The goal of this practical is to compute the derivatives in a much simpler way.  There will be three steps:"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "8UWhvDeNDudz"
+      },
       "source": [
         "**Step 1:** Write the original equations as a series of intermediate calculations.\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
         "f_{0} &=& \\beta_{0} + \\omega_{0} x_i\\nonumber\\\\\n",
         "h_{1} &=& \\sin[f_{0}]\\nonumber\\\\\n",
         "f_{1} &=& \\beta_{1} + \\omega_{1}h_{1}\\nonumber\\\\\n",
@@ -235,16 +231,18 @@
         "h_{3} &=& \\cos[f_{2}]\\nonumber\\\\\n",
         "f_{3} &=& \\beta_{3} + \\omega_{3}h_{3}\\nonumber\\\\\n",
         "l_i &=& (f_3-y_i)^2\n",
-        "\\end{eqnarray}\n",
+        "\\end{align}\n",
         "\n",
         "and compute and store the values of all of these intermediate values.  We'll need them to compute the derivatives.<br>  This is called the **forward pass**."
-      ],
+      ]
-      "metadata": {
-        "id": "8UWhvDeNDudz"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZWKAq6HC90qV"
+      },
+      "outputs": [],
       "source": [
         "# TODO compute all the f_k and h_k terms\n",
         "# Replace the code below\n",
@@ -257,38 +255,22 @@
         "h3 = 0\n",
         "f3 = 0\n",
         "l_i = 0\n"
-      ],
+      ]
-      "metadata": {
-        "id": "ZWKAq6HC90qV"
-      },
-      "execution_count": 6,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "# Let's check we got that right:\n",
-        "print(\"f0: true value = %3.3f, your value = %3.3f\"%(1.230, f0))\n",
-        "print(\"h1: true value = %3.3f, your value = %3.3f\"%(0.942, h1))\n",
-        "print(\"f1: true value = %3.3f, your value = %3.3f\"%(1.623, f1))\n",
-        "print(\"h2: true value = %3.3f, your value = %3.3f\"%(5.068, h2))\n",
-        "print(\"f2: true value = %3.3f, your value = %3.3f\"%(7.137, f2))\n",
-        "print(\"h3: true value = %3.3f, your value = %3.3f\"%(0.657, h3))\n",
-        "print(\"f3: true value = %3.3f, your value = %3.3f\"%(2.372, f3))\n",
-        "print(\"like original = %3.3f, like from forward pass = %3.3f\"%(l_i_func, l_i))\n"
-      ],
       "metadata": {
-        "id": "ibxXw7TUW4Sx",
         "colab": {
           "base_uri": "https://localhost:8080/"
         },
+        "id": "ibxXw7TUW4Sx",
         "outputId": "4575e3eb-2b16-4e0b-c84e-9c22b443c3ce"
       },
-      "execution_count": 7,
       "outputs": [
         {
-          "output_type": "stream",
           "name": "stdout",
+          "output_type": "stream",
           "text": [
             "f0: true value = 1.230, your value = 0.000\n",
             "h1: true value = 0.942, your value = 0.000\n",
@@ -300,17 +282,32 @@
             "like original = 0.139, like from forward pass = 0.000\n"
           ]
         }
+      ],
+      "source": [
+        "# Let's check we got that right:\n",
+        "print(\"f0: true value = %3.3f, your value = %3.3f\"%(1.230, f0))\n",
+        "print(\"h1: true value = %3.3f, your value = %3.3f\"%(0.942, h1))\n",
+        "print(\"f1: true value = %3.3f, your value = %3.3f\"%(1.623, f1))\n",
+        "print(\"h2: true value = %3.3f, your value = %3.3f\"%(5.068, h2))\n",
+        "print(\"f2: true value = %3.3f, your value = %3.3f\"%(7.137, f2))\n",
+        "print(\"h3: true value = %3.3f, your value = %3.3f\"%(0.657, h3))\n",
+        "print(\"f3: true value = %3.3f, your value = %3.3f\"%(2.372, f3))\n",
+        "print(\"like original = %3.3f, like from forward pass = %3.3f\"%(l_i_func, l_i))\n"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "jay8NYWdFHuZ"
+      },
       "source": [
         "**Step 2:** Compute the derivatives of $\\ell_i$ with respect to the intermediate quantities that we just calculated, but in reverse order:\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
         "\\quad \\frac{\\partial \\ell_i}{\\partial f_3}, \\quad \\frac{\\partial \\ell_i}{\\partial h_3}, \\quad \\frac{\\partial \\ell_i}{\\partial f_2}, \\quad\n",
-        "\\frac{\\partial \\ell_i}{\\partial h_2}, \\quad \\frac{\\partial \\ell_i}{\\partial f_1}, \\quad \\frac{\\partial \\ell_i}{\\partial h_1},  \\quad\\mbox{and} \\quad \\frac{\\partial \\ell_i}{\\partial f_0}.\n",
+        "\\frac{\\partial \\ell_i}{\\partial h_2}, \\quad \\frac{\\partial \\ell_i}{\\partial f_1}, \\quad \\frac{\\partial \\ell_i}{\\partial h_1},  \\quad\\text{and} \\quad \\frac{\\partial \\ell_i}{\\partial f_0}.\n",
-        "\\end{eqnarray}\n",
+        "\\end{align}\n",
         "\n",
         "The first of these derivatives is straightforward:\n",
         "\n",
@@ -328,7 +325,7 @@
         "\n",
         "We can continue in this way, computing the derivatives of the output with respect to these intermediate quantities:\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
         "\\frac{\\partial \\ell_i}{\\partial f_{2}} &=& \\frac{\\partial h_{3}}{\\partial f_{2}}\\left(\n",
         "\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}} \\right)\n",
         "\\nonumber \\\\\n",
@@ -336,16 +333,18 @@
         "\\frac{\\partial \\ell_i}{\\partial f_{1}} &=& \\frac{\\partial h_{2}}{\\partial f_{1}}\\left( \\frac{\\partial f_{2}}{\\partial h_{2}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}} \\right)\\nonumber \\\\\n",
         "\\frac{\\partial \\ell_i}{\\partial h_{1}} &=& \\frac{\\partial f_{1}}{\\partial h_{1}}\\left(\\frac{\\partial h_{2}}{\\partial f_{1}} \\frac{\\partial f_{2}}{\\partial h_{2}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}} \\right)\\nonumber \\\\\n",
         "\\frac{\\partial \\ell_i}{\\partial f_{0}} &=& \\frac{\\partial h_{1}}{\\partial f_{0}}\\left(\\frac{\\partial f_{1}}{\\partial h_{1}}\\frac{\\partial h_{2}}{\\partial f_{1}} \\frac{\\partial f_{2}}{\\partial h_{2}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}} \\right).\n",
-        "\\end{eqnarray}\n",
+        "\\end{align}\n",
         "\n",
         "In each case, we have already computed all of the terms except the last one in the previous step, and the last term is simple to evaluate.  This is called the **backward pass**."
-      ],
+      ]
-      "metadata": {
-        "id": "jay8NYWdFHuZ"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gCQJeI--Egdl"
+      },
+      "outputs": [],
       "source": [
         "# TODO -- Compute the derivatives of the output with respect\n",
         "# to the intermediate computations h_k and f_k (i.e, run the backward pass)\n",
@@ -358,37 +357,22 @@
         "dldf1 = 1\n",
         "dldh1 = 1\n",
         "dldf0 = 1\n"
-      ],
+      ]
-      "metadata": {
-        "id": "gCQJeI--Egdl"
-      },
-      "execution_count": 8,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "# Let's check we got that right\n",
-        "print(\"dldf3: true value = %3.3f, your value = %3.3f\"%(0.745, dldf3))\n",
-        "print(\"dldh3: true value = %3.3f, your value = %3.3f\"%(2.234, dldh3))\n",
-        "print(\"dldf2: true value = %3.3f, your value = %3.3f\"%(-1.683, dldf2))\n",
-        "print(\"dldh2: true value = %3.3f, your value = %3.3f\"%(-3.366, dldh2))\n",
-        "print(\"dldf1: true value = %3.3f, your value = %3.3f\"%(-17.060, dldf1))\n",
-        "print(\"dldh1: true value = %3.3f, your value = %3.3f\"%(6.824, dldh1))\n",
-        "print(\"dldf0: true value = %3.3f, your value = %3.3f\"%(2.281, dldf0))"
-      ],
       "metadata": {
-        "id": "dS1OrLtlaFr7",
         "colab": {
           "base_uri": "https://localhost:8080/"
         },
+        "id": "dS1OrLtlaFr7",
         "outputId": "414f0862-ae36-4a0e-b68f-4758835b0e23"
       },
-      "execution_count": 9,
       "outputs": [
         {
-          "output_type": "stream",
           "name": "stdout",
+          "output_type": "stream",
           "text": [
             "dldf3: true value = 0.745, your value = -4.000\n",
             "dldh3: true value = 2.234, your value = -12.000\n",
@@ -399,33 +383,25 @@
             "dldf0: true value = 2.281, your value = 1.000\n"
           ]
         }
+      ],
+      "source": [
+        "# Let's check we got that right\n",
+        "print(\"dldf3: true value = %3.3f, your value = %3.3f\"%(0.745, dldf3))\n",
+        "print(\"dldh3: true value = %3.3f, your value = %3.3f\"%(2.234, dldh3))\n",
+        "print(\"dldf2: true value = %3.3f, your value = %3.3f\"%(-1.683, dldf2))\n",
+        "print(\"dldh2: true value = %3.3f, your value = %3.3f\"%(-3.366, dldh2))\n",
+        "print(\"dldf1: true value = %3.3f, your value = %3.3f\"%(-17.060, dldf1))\n",
+        "print(\"dldh1: true value = %3.3f, your value = %3.3f\"%(6.824, dldh1))\n",
+        "print(\"dldf0: true value = %3.3f, your value = %3.3f\"%(2.281, dldf0))"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "**Step 3:**  Finally, we consider how the loss~$\\ell_{i}$ changes when we change the parameters $\\beta_{\\bullet}$ and $\\omega_{\\bullet}$. Once more, we apply the chain rule:\n",
-        "\n",
-        "\n",
-        "\n",
-        "\n",
-        "\\begin{eqnarray}\n",
-        "\\frac{\\partial \\ell_i}{\\partial \\beta_{k}} &=& \\frac{\\partial f_{k}}{\\partial \\beta_{k}}\\frac{\\partial \\ell_i}{\\partial f_{k}}\\nonumber \\\\\n",
-        "\\frac{\\partial \\ell_i}{\\partial \\omega_{k}} &=& \\frac{\\partial f_{k}}{\\partial \\omega_{k}}\\frac{\\partial \\ell_i}{\\partial f_{k}}.\n",
-        "\\end{eqnarray}\n",
-        "\n",
-        "\\noindent In each case, the second term on the right-hand side was computed in step 2. When $k>0$, we have~$f_{k}=\\beta_{k}+\\omega_k \\cdot h_{k}$, so:\n",
-        "\n",
-        "\\begin{eqnarray}\n",
-        "\\frac{\\partial f_{k}}{\\partial \\beta_{k}} = 1 \\quad\\quad\\mbox{and}\\quad \\quad \\frac{\\partial f_{k}}{\\partial \\omega_{k}} &=& h_{k}.\n",
-        "\\end{eqnarray}"
-      ],
-      "metadata": {
-        "id": "FlzlThQPGpkU"
-      }
-    },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1I2BhqZhGMK6"
+      },
+      "outputs": [],
       "source": [
         "# TODO -- Calculate the final derivatives with respect to the beta and omega terms\n",
         "\n",
@@ -437,38 +413,22 @@
         "dldomega1 = 1\n",
         "dldbeta0 = 1\n",
         "dldomega0 = 1\n"
-      ],
+      ]
-      "metadata": {
-        "id": "1I2BhqZhGMK6"
-      },
-      "execution_count": 10,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "# Let's check we got them right\n",
-        "print('dldbeta3: Your value = %3.3f, True value = %3.3f'%(dldbeta3, 0.745))\n",
-        "print('dldomega3: Your value = %3.3f, True value = %3.3f'%(dldomega3, 0.489))\n",
-        "print('dldbeta2: Your value = %3.3f, True value = %3.3f'%(dldbeta2, -1.683))\n",
-        "print('dldomega2: Your value = %3.3f, True value = %3.3f'%(dldomega2, -8.530))\n",
-        "print('dldbeta1: Your value = %3.3f, True value = %3.3f'%(dldbeta1, -17.060))\n",
-        "print('dldomega1: Your value = %3.3f, True value = %3.3f'%(dldomega1, -16.079))\n",
-        "print('dldbeta0: Your value = %3.3f, True value = %3.3f'%(dldbeta0, 2.281))\n",
-        "print('dldomega0: Your value = %3.3f, Function value = %3.3f, Finite difference value = %3.3f'%(dldomega0, dldomega0_func, dldomega0_fd))"
-      ],
       "metadata": {
-        "id": "38eiOn2aHgHI",
         "colab": {
           "base_uri": "https://localhost:8080/"
         },
+        "id": "38eiOn2aHgHI",
         "outputId": "1a67a636-e832-471e-e771-54824363158a"
       },
-      "execution_count": 11,
       "outputs": [
         {
-          "output_type": "stream",
           "name": "stdout",
+          "output_type": "stream",
           "text": [
             "dldbeta3: Your value = 1.000, True value = 0.745\n",
             "dldomega3: Your value = 1.000, True value = 0.489\n",
@@ -480,16 +440,44 @@
             "dldomega0: Your value = 1.000, Function value = 5.246, Finite difference value = 5.246\n"
           ]
         }
+      ],
+      "source": [
+        "# Let's check we got them right\n",
+        "print('dldbeta3: Your value = %3.3f, True value = %3.3f'%(dldbeta3, 0.745))\n",
+        "print('dldomega3: Your value = %3.3f, True value = %3.3f'%(dldomega3, 0.489))\n",
+        "print('dldbeta2: Your value = %3.3f, True value = %3.3f'%(dldbeta2, -1.683))\n",
+        "print('dldomega2: Your value = %3.3f, True value = %3.3f'%(dldomega2, -8.530))\n",
+        "print('dldbeta1: Your value = %3.3f, True value = %3.3f'%(dldbeta1, -17.060))\n",
+        "print('dldomega1: Your value = %3.3f, True value = %3.3f'%(dldomega1, -16.079))\n",
+        "print('dldbeta0: Your value = %3.3f, True value = %3.3f'%(dldbeta0, 2.281))\n",
+        "print('dldomega0: Your value = %3.3f, Function value = %3.3f, Finite difference value = %3.3f'%(dldomega0, dldomega0_func, dldomega0_fd))"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Using this method, we can compute the derivatives quite easily without needing to compute very complicated expressions.  In the next practical, we'll apply this same method to a deep neural network."
-      ],
       "metadata": {
         "id": "N2ZhrR-2fNa1"
-      }
+      },
-    }
+      "source": [
+        "Using this method, we can compute the derivatives quite easily without needing to compute very complicated expressions.  In the next practical, we'll apply this same method to a deep neural network."
       ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "authorship_tag": "ABX9TyN7JeDgslwtZcwRCOuGuPFt",
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap07/7_2_Backpropagation.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOlKB4TrCJnt91TnHOrfRSJ",
+      "authorship_tag": "ABX9TyM2kkHLr00J4Jeypw41sTkQ",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -311,10 +311,16 @@
         "    network_output_2, *_ = compute_network_output(net_input, all_weights, all_biases)\n",
         "    dl_dbias[row] = (least_squares_loss(network_output_1, y) - least_squares_loss(network_output_2,y))/delta_fd\n",
         "  all_dl_dbiases_fd[layer] = np.array(dl_dbias)\n",
+        "  print(\"-----------------------------------------------\")\n",
         "  print(\"Bias %d, derivatives from backprop:\"%(layer))\n",
         "  print(all_dl_dbiases[layer])\n",
         "  print(\"Bias %d, derivatives from finite differences\"%(layer))\n",
         "  print(all_dl_dbiases_fd[layer])\n",
+        "  if np.allclose(all_dl_dbiases_fd[layer],all_dl_dbiases[layer],rtol=1e-05, atol=1e-08, equal_nan=False):\n",
+        "    print(\"Success!  Derivatives match.\")\n",
+        "  else:\n",
+        "    print(\"Failure!  Derivatives different.\")\n",
+        "\n",
         "\n",
         "\n",
         "# Test the derivatives of the weights matrices\n",
@@ -330,10 +336,15 @@
         "      network_output_2, *_ = compute_network_output(net_input, all_weights, all_biases)\n",
         "      dl_dweight[row][col] = (least_squares_loss(network_output_1, y) - least_squares_loss(network_output_2,y))/delta_fd\n",
         "  all_dl_dweights_fd[layer] = np.array(dl_dweight)\n",
+        "  print(\"-----------------------------------------------\")\n",
         "  print(\"Weight %d, derivatives from backprop:\"%(layer))\n",
         "  print(all_dl_dweights[layer])\n",
         "  print(\"Weight %d, derivatives from finite differences\"%(layer))\n",
-        "  print(all_dl_dweights_fd[layer])"
+        "  print(all_dl_dweights_fd[layer])\n",
+        "  if np.allclose(all_dl_dweights_fd[layer],all_dl_dweights[layer],rtol=1e-05, atol=1e-08, equal_nan=False):\n",
+        "    print(\"Success!  Derivatives match.\")\n",
+        "  else:\n",
+        "    print(\"Failure!  Derivatives different.\")"
       ],
       "metadata": {
         "id": "PK-UtE3hreAK"

Notebooks/Chap08/8_1_MNIST_1D_Performance.ipynb

@@ -5,7 +5,7 @@
     "colab": {
       "provenance": [],
       "gpuType": "T4",
-      "authorship_tag": "ABX9TyNLj3HOpVB87nRu7oSLuBaU",
+      "authorship_tag": "ABX9TyOuKMUcKfOIhIL2qTX9jJCy",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -84,7 +84,7 @@
       "cell_type": "code",
       "source": [
         "args = mnist1d.data.get_dataset_args()\n",
-        "data = mnist1d.data.get_dataset(args, path='./mnist1d_data.pkl', download=False, regenerate=False)\n",
+        "data = mnist1d.data.get_dataset(args, path='./sample_data/mnist1d_data.pkl', download=False, regenerate=False)\n",
         "\n",
         "# The training and test input and outputs are in\n",
         "# data['x'], data['y'], data['x_test'], and data['y_test']\n",

Notebooks/Chap08/8_2_Bias_Variance_Trade_Off.ipynb

@@ -4,7 +4,6 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyPz1B8kFc21JvGTDwqniloA",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -185,10 +184,8 @@
         "          if A[i,j] < 0:\n",
         "              A[i,j] = 0;\n",
         "\n",
-        "  ATA = np.matmul(np.transpose(A), A)\n",
+        "  beta_omega = np.linalg.lstsq(A, y, rcond=None)[0]\n",
-        "  ATAInv = np.linalg.inv(ATA)\n",
+        "\n",
-        "  ATAInvAT = np.matmul(ATAInv, np.transpose(A))\n",
-        "  beta_omega = np.matmul(ATAInvAT,y)\n",
         "  beta = beta_omega[0]\n",
         "  omega = beta_omega[1:]\n",
         "\n",

Notebooks/Chap08/8_4_High_Dimensional_Spaces.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyPXPDEQiwNw+kYhWfg4kjz6",
+      "authorship_tag": "ABX9TyPAKqlf9VxztHXKylyJwqe8",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -145,7 +145,7 @@
       "source": [
         "def volume_of_hypersphere(diameter, dimensions):\n",
         "  # Formula given in Problem 8.7 of the book\n",
-        "  # You will need sci.special.gamma()\n",
+        "  # You will need sci.gamma()\n",
         "  # Check out:    https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.gamma.html\n",
         "  # Also use this value for pi\n",
         "  pi = np.pi\n",

Notebooks/Chap09/9_3_Ensembling.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyNuR7X+PMWRddy+WQr4gr5f",
+      "authorship_tag": "ABX9TyOAC7YLEqN5qZhJXqRj+aHB",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -184,7 +184,9 @@
         "  A = np.ones((n_data, n_hidden+1))\n",
         "  for i in range(n_data):\n",
         "      for j in range(1,n_hidden+1):\n",
+        "          # Compute preactivation\n",
         "          A[i,j] = x[i]-(j-1)/n_hidden\n",
+        "          # Apply the ReLU function\n",
         "          if A[i,j] < 0:\n",
         "              A[i,j] = 0;\n",
         "\n",

Notebooks/Chap09/9_4_Bayesian_Approach.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyMB8B4269DVmrcLoCWrhzKF",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap09/9_4_Bayesian_Approach.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "el8l05WQEO46"
+      },
       "source": [
         "# **Notebook 9.4: Bayesian approach**\n",
         "\n",
@@ -36,10 +25,7 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
-      ],
+      ]
-      "metadata": {
-        "id": "el8l05WQEO46"
-      }
     },
     {
       "cell_type": "code",
@@ -58,20 +44,25 @@
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "3hpqmFyQNrbt"
+      },
+      "outputs": [],
       "source": [
         "# The true function that we are trying to estimate, defined on [0,1]\n",
         "def true_function(x):\n",
         "    y = np.exp(np.sin(x*(2*3.1413)))\n",
         "    return y"
-      ],
+      ]
-      "metadata": {
-        "id": "3hpqmFyQNrbt"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "skZMM5TbNwq4"
+      },
+      "outputs": [],
       "source": [
         "# Generate some data points with or without noise\n",
         "def generate_data(n_data, sigma_y=0.3):\n",
@@ -86,15 +77,15 @@
         "        y[i] = true_function(x[i])\n",
         "        y[i] += np.random.normal(0, sigma_y, 1)\n",
         "    return x,y"
-      ],
+      ]
-      "metadata": {
-        "id": "skZMM5TbNwq4"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ziwD_R7lN0DY"
+      },
+      "outputs": [],
       "source": [
         "# Draw the fitted function, together win uncertainty used to generate points\n",
         "def plot_function(x_func, y_func, x_data=None,y_data=None, x_model = None, y_model =None, sigma_func = None, sigma_model=None):\n",
@@ -117,15 +108,15 @@
         "    ax.set_xlabel('Input, $x$')\n",
         "    ax.set_ylabel('Output, $y$')\n",
         "    plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "ziwD_R7lN0DY"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2CgKanwaN3NM"
+      },
+      "outputs": [],
       "source": [
         "# Generate true function\n",
         "x_func = np.linspace(0, 1.0, 100)\n",
@@ -139,15 +130,15 @@
         "\n",
         "# Plot the function, data and uncertainty\n",
         "plot_function(x_func, y_func, x_data, y_data, sigma_func=sigma_func)"
-      ],
+      ]
-      "metadata": {
-        "id": "2CgKanwaN3NM"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gorZ6i97N7AR"
+      },
+      "outputs": [],
       "source": [
         "# Define model -- beta is a scalar and omega has size n_hidden,1\n",
         "def network(x, beta, omega):\n",
@@ -165,15 +156,14 @@
         "    y = y + beta\n",
         "\n",
         "    return y"
-      ],
+      ]
-      "metadata": {
-        "id": "gorZ6i97N7AR"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "i8T_QduzeBmM"
+      },
       "source": [
         "Now let's compute a probability distribution over the model parameters using Bayes's rule:\n",
         "\n",
@@ -184,69 +174,73 @@
         "We'll define the prior $Pr(\\boldsymbol\\phi)$ as:\n",
         "\n",
         "\\begin{equation}\n",
-        "Pr(\\boldsymbol\\phi) = \\mbox{Norm}_{\\boldsymbol\\phi}\\bigl[\\mathbf{0},\\sigma^2_p\\mathbf{I}\\bigr]\n",
+        "Pr(\\boldsymbol\\phi) = \\text{Norm}_{\\boldsymbol\\phi}\\bigl[\\mathbf{0},\\sigma^2_p\\mathbf{I}\\bigr]\n",
         "\\end{equation}\n",
         "\n",
         "where $\\phi=[\\omega_1,\\omega_2\\ldots \\omega_n, \\beta]^T$ and $\\sigma^2_{p}$  is the prior variance.\n",
         "\n",
         "The likelihood term $\\prod_{i=1}^{I} Pr(\\mathbf{y}_{i}|\\mathbf{x}_{i},\\boldsymbol\\phi)$ is given by:\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
-        "\\prod_{i=1}^{I} Pr(\\mathbf{y}_{i}|\\mathbf{x}_{i},\\boldsymbol\\phi) &=& \\prod_{i=1}^{I} \\mbox{Norm}_{y_i}\\bigl[\\mbox{f}[\\mathbf{x}_{i},\\boldsymbol\\phi],\\sigma_d^2\\bigr]\\\\\n",
+        "\\prod_{i=1}^{I} Pr(\\mathbf{y}_{i}|\\mathbf{x}_{i},\\boldsymbol\\phi) &=& \\prod_{i=1}^{I} \\text{Norm}_{y_i}\\bigl[\\text{f}[\\mathbf{x}_{i},\\boldsymbol\\phi],\\sigma_d^2\\bigr]\\\\\n",
-        "&=& \\prod_{i=1}^{I} \\mbox{Norm}_{y_i}\\bigl[\\boldsymbol\\omega\\mathbf{h}_i+\\beta,\\sigma_d^2\\bigr]\\\\\n",
+        "&=& \\prod_{i=1}^{I} \\text{Norm}_{y_i}\\bigl[\\boldsymbol\\omega\\mathbf{h}_i+\\beta,\\sigma_d^2\\bigr]\\\\\n",
-        "&=& \\mbox{Norm}_{\\mathbf{y}}\\bigl[\\mathbf{H}^T\\boldsymbol\\phi,\\sigma^2\\mathbf{I}\\bigr].\n",
+        "&=& \\text{Norm}_{\\mathbf{y}}\\bigl[\\mathbf{H}^T\\boldsymbol\\phi,\\sigma^2\\mathbf{I}\\bigr].\n",
-        "\\end{eqnarray}\n",
+        "\\end{align}\n",
         "\n",
         "where $\\sigma^2$ is the measurement noise and $\\mathbf{h}_{i}$ is the column vector of hidden variables for the $i^{th}$ input.  Here the vector $\\mathbf{y}$ and matrix $\\mathbf{H}$ are defined as:\n",
         "\n",
         "\\begin{equation}\n",
-        "\\mathbf{y} = \\begin{bmatrix}y_1\\\\y_2\\\\\\vdots\\\\y_{I}\\end{bmatrix}\\quad\\mbox{and}\\quad \\mathbf{H} = \\begin{bmatrix}\\mathbf{h}_{1}&\\mathbf{h}_{2}&\\cdots&\\mathbf{h}_{I}\\\\1&1&\\cdots &1\\end{bmatrix}.\n",
+        "\\mathbf{y} = \\begin{bmatrix}y_1\\\\y_2\\\\\\vdots\\\\y_{I}\\end{bmatrix}\\quad\\text{and}\\quad \\mathbf{H} = \\begin{bmatrix}\\mathbf{h}_{1}&\\mathbf{h}_{2}&\\cdots&\\mathbf{h}_{I}\\\\1&1&\\cdots &1\\end{bmatrix}.\n",
         "\\end{equation}\n"
-      ],
+      ]
-      "metadata": {
-        "id": "i8T_QduzeBmM"
-      }
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "JojV6ueRk49G"
+      },
       "source": [
         "To make progress we use the change of variable relation (Appendix C.3.4 of the book) to rewrite the likelihood term as a normal distribution in the parameters $\\boldsymbol\\phi$:\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
         "\\prod_{i=1}^{I} Pr(\\mathbf{y}_{i}|\\mathbf{x}_{i},\\boldsymbol\\phi+\\beta)\n",
-        "&=& \\mbox{Norm}_{\\mathbf{y}}\\bigl[\\mathbf{H}^T\\boldsymbol\\phi,\\sigma^2\\bigr]\\\\\n",
+        "&=& \\text{Norm}_{\\mathbf{y}}\\bigl[\\mathbf{H}^T\\boldsymbol\\phi,\\sigma^2\\bigr]\\\\\n",
-        "&\\propto& \\mbox{Norm}_{\\boldsymbol\\phi}\\bigl[(\\mathbf{H}\\mathbf{H}^T)^{-1}\\mathbf{H}\\mathbf{y},\\sigma^2(\\mathbf{H}\\mathbf{H}^t)^{-1}\\bigr]\n",
+        "&\\propto& \\text{Norm}_{\\boldsymbol\\phi}\\bigl[(\\mathbf{H}\\mathbf{H}^T)^{-1}\\mathbf{H}\\mathbf{y},\\sigma^2(\\mathbf{H}\\mathbf{H}^t)^{-1}\\bigr]\n",
-        "\\end{eqnarray}\n"
+        "\\end{align}\n"
-      ],
+      ]
-      "metadata": {
-        "id": "JojV6ueRk49G"
-      }
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "YX0O_Ciwp4W1"
+      },
       "source": [
         "Finally, we can combine the likelihood and prior terms using the product of two normal distributions relation (Appendix C.3.3).\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
         " Pr(\\boldsymbol\\phi|\\{\\mathbf{x}_{i},\\mathbf{y}_{i}\\}) &\\propto& \\prod_{i=1}^{I} Pr(\\mathbf{y}_{i}|\\mathbf{x}_{i},\\boldsymbol\\phi) Pr(\\boldsymbol\\phi)\\\\\n",
-        " &\\propto&\\mbox{Norm}_{\\boldsymbol\\phi}\\bigl[(\\mathbf{H}\\mathbf{H}^T)^{-1}\\mathbf{H}\\mathbf{y},\\sigma^2(\\mathbf{H}\\mathbf{H}^T)^{-1}\\bigr] \\mbox{Norm}_{\\boldsymbol\\phi}\\bigl[\\mathbf{0},\\sigma^2_p\\mathbf{I}\\bigr]\\\\\n",
+        " &\\propto&\\text{Norm}_{\\boldsymbol\\phi}\\bigl[(\\mathbf{H}\\mathbf{H}^T)^{-1}\\mathbf{H}\\mathbf{y},\\sigma^2(\\mathbf{H}\\mathbf{H}^T)^{-1}\\bigr] \\text{Norm}_{\\boldsymbol\\phi}\\bigl[\\mathbf{0},\\sigma^2_p\\mathbf{I}\\bigr]\\\\\n",
-        " &\\propto&\\mbox{Norm}_{\\boldsymbol\\phi}\\biggl[\\frac{1}{\\sigma^2}\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\biggr].\n",
+        " &\\propto&\\text{Norm}_{\\boldsymbol\\phi}\\biggl[\\frac{1}{\\sigma^2}\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\biggr].\n",
-        "\\end{eqnarray}\n",
+        "\\end{align}\n",
         "\n",
         "In fact, since this already a normal distribution, the constant of proportionality must be one and we can write\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
-        " Pr(\\boldsymbol\\phi|\\{\\mathbf{x}_{i},\\mathbf{y}_{i}\\}) &=& \\mbox{Norm}_{\\boldsymbol\\phi}\\biggl[\\frac{1}{\\sigma^2}\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\biggr].\n",
+        " Pr(\\boldsymbol\\phi|\\{\\mathbf{x}_{i},\\mathbf{y}_{i}\\}) &=& \\text{Norm}_{\\boldsymbol\\phi}\\biggl[\\frac{1}{\\sigma^2}\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\biggr].\n",
-        "\\end{eqnarray}\n",
+        "\\end{align}\n",
         "\n",
         "TODO -- On a piece of paper, use the relations in Appendix C.3.3 and C.3.4 to fill in the missing steps and establish that this is the correct formula for the posterior."
-      ],
+      ]
-      "metadata": {
-        "id": "YX0O_Ciwp4W1"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "nF1AcgNDwm4t"
+      },
+      "outputs": [],
       "source": [
         "def compute_H(x_data, n_hidden):\n",
         "  psi1 = np.ones((n_hidden+1,1));\n",
@@ -280,24 +274,25 @@
         "\n",
         "\n",
         "  return phi_mean, phi_covar"
-      ],
+      ]
-      "metadata": {
-        "id": "nF1AcgNDwm4t"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now we can draw samples from this distribution"
-      ],
       "metadata": {
         "id": "GjPnlG4q0UFK"
-      }
+      },
+      "source": [
+        "Now we can draw samples from this distribution"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "K4vYc82D0BMq"
+      },
+      "outputs": [],
       "source": [
         "# Define parameters\n",
         "n_hidden = 5\n",
@@ -313,15 +308,15 @@
         "x_model = x_func\n",
         "y_model_mean = network(x_model, phi_mean[-1], phi_mean[0:n_hidden])\n",
         "plot_function(x_func, y_func, x_data, y_data, x_model, y_model_mean)"
-      ],
+      ]
-      "metadata": {
-        "id": "K4vYc82D0BMq"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "TVIjhubkSw-R"
+      },
+      "outputs": [],
       "source": [
         "# TODO Draw two samples from the normal distribution over the parameters\n",
         "# Replace these lines\n",
@@ -336,37 +331,42 @@
         "# Draw the two models\n",
         "plot_function(x_func, y_func, x_data, y_data, x_model, y_model_sample1)\n",
         "plot_function(x_func, y_func, x_data, y_data, x_model, y_model_sample2)"
-      ],
+      ]
-      "metadata": {
-        "id": "TVIjhubkSw-R"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "GiNg5EroUiUb"
+      },
       "source": [
         "Now we need to perform inference for a new data points $\\mathbf{x}^*$ with corresponding hidden values $\\mathbf{h}^*$.  Instead of having a single estimate of the parameters, we have a distribution over the possible parameters.  So we marginalize (integrate) over this distribution to account for all possible values:\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
         "Pr(y^*|\\mathbf{x}^*)  &=& \\int Pr(y^{*}|\\mathbf{x}^*,\\boldsymbol\\phi)Pr(\\boldsymbol\\phi|\\{\\mathbf{x}_{i},\\mathbf{y}_{i}\\}) d\\boldsymbol\\phi\\\\\n",
-        "&=& \\int \\mbox{Norm}_{y^*}\\bigl[\\begin{bmatrix}\\mathbf{h}^{*T}&1\\end{bmatrix}\\boldsymbol\\phi,\\sigma^2]\\cdot\\mbox{Norm}_{\\boldsymbol\\phi}\\biggl[\\frac{1}{\\sigma^2}\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\biggr]d\\boldsymbol\\phi\\\\\n",
+        "&=& \\int \\text{Norm}_{y^*}\\bigl[[\\mathbf{h}^{*T},1]\\boldsymbol\\phi,\\sigma^2\\bigr]\\cdot\\text{Norm}_{\\boldsymbol\\phi}\\biggl[\\frac{1}{\\sigma^2}\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\biggr]d\\boldsymbol\\phi\\\\\n",
-        "&=& \\mbox{Norm}_{y^*}\\biggl[\\frac{1}{\\sigma^2} \\begin{bmatrix}\\mathbf{h}^{*T}&1\\end{bmatrix}\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},  \\begin{bmatrix}\\mathbf{h}^{*T}&1\\end{bmatrix}\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\n",
+        "&=& \\text{Norm}_{y^*}\\biggl[\\frac{1}{\\sigma^2} [\\mathbf{h}^{*T},1]\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\\mathbf{H}\\mathbf{y},  [\\mathbf{h}^{*T},1]\\left(\\frac{1}{\\sigma^2}\\mathbf{H}\\mathbf{H}^T+\\frac{1}{\\sigma_p^2}\\mathbf{I}\\right)^{-1}\n",
-        "\\begin{bmatrix}\\mathbf{h}^*\\\\1\\end{bmatrix}\\biggr]\n",
+        "[\\mathbf{h}^*;1]\\biggr]\n",
-        "\\end{eqnarray}\n",
+        "\\end{align}\n",
+        "\n",
+        "\n",
+        "\n",
         "\n",
         "To compute this, we reformulated the integrand using the relations from appendices\n",
         "C.3.3 and C.3.4 as the product of a normal distribution in $\\boldsymbol\\phi$ and a constant with respect\n",
         "to $\\boldsymbol\\phi$. The integral of the normal distribution must be one, and so the final result is just the constant. This constant is itself a normal distribution in $y^*$. <br>\n",
         "\n",
-        "If you feel so inclined you can work through the math of this yourself."
+        "If you feel so inclined you can work through the math of this yourself.\n",
-      ],
+        "\n"
-      "metadata": {
+      ]
-        "id": "GiNg5EroUiUb"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ILxT4EfW2lUm"
+      },
+      "outputs": [],
       "source": [
         "# Predict mean and variance of y_star from x_star\n",
         "def inference(x_star, x_data, y_data, sigma_sq, sigma_p_sq, n_hidden):\n",
@@ -381,15 +381,15 @@
         "  y_star_var =  1\n",
         "\n",
         "  return y_star_mean, y_star_var"
-      ],
+      ]
-      "metadata": {
-        "id": "ILxT4EfW2lUm"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "87cjUjMaixHZ"
+      },
+      "outputs": [],
       "source": [
         "x_model = x_func\n",
         "y_model = np.zeros_like(x_model)\n",
@@ -401,24 +401,36 @@
         "\n",
         "# Draw the model\n",
         "plot_function(x_func, y_func, x_data, y_data, x_model, y_model, sigma_model=y_model_std)\n"
-      ],
+      ]
-      "metadata": {
-        "id": "87cjUjMaixHZ"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "8Hcbe_16sK0F"
+      },
       "source": [
         "TODO:\n",
         "\n",
         "1.  Experiment running this again with different numbers of hidden units.  Make a prediction for what will happen when you increase / decrease them.\n",
         "2.  Experiment with what happens if you make the prior variance $\\sigma^2_p$ to a smaller value like 1.  How do you explain the results?"
+      ]
+    }
   ],
   "metadata": {
-        "id": "8Hcbe_16sK0F"
+    "colab": {
+      "authorship_tag": "ABX9TyMB8B4269DVmrcLoCWrhzKF",
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
     }
-    }
+  },
-  ]
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap09/9_5_Augmentation.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyM3wq9CHLjekkIXIgXRxueE",
+      "authorship_tag": "ABX9TyM38ZVBK4/xaHk5Ys5lF6dN",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -208,14 +208,14 @@
     {
       "cell_type": "code",
       "source": [
-        "def augment(data_in):\n",
+        "def augment(input_vector):\n",
         "  # Create output vector\n",
-        "  data_out = np.zeros_like(data_in)\n",
+        "  data_out = np.zeros_like(input_vector)\n",
         "\n",
         "  # TODO:  Shift the input data by a random offset\n",
         "  # (rotating, so points that would go off the end, are added back to the beginning)\n",
         "  # Replace this line:\n",
-        "  data_out = np.zeros_like(data_in) ;\n",
+        "  data_out = np.zeros_like(input_vector) ;\n",
         "\n",
         "  # TODO:    # Randomly scale the data by a factor drawn from a uniform distribution over [0.8,1.2]\n",
         "  # Replace this line:\n",

Notebooks/Chap10/10_1_1D_Convolution.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyPTidpnPhn4O5QF011gt0cz",
+      "authorship_tag": "ABX9TyML7rfAGE4gvmNUEiK5x3PS",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -41,6 +41,17 @@
         "id": "el8l05WQEO46"
       }
     },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "NOTE!!\n",
+        "\n",
+        "If you have the first edition of the printed book, it mistakenly refers to a convolutional filter with no spaces between the elements (i.e. a normal filter without dilation) as having dilation zero.  Actually, the convention is (weirdly) that this has dilation one.  And when there is one space between the elements, this is dilation two.   This notebook reflects the correct convention and so will be out of sync with the printed book.  If this is confusing, check the [errata](https://github.com/udlbook/udlbook/blob/main/UDL_Errata.pdf) document."
+      ],
+      "metadata": {
+        "id": "ggQrHkFZcUiV"
+      }
+    },
     {
       "cell_type": "code",
       "source": [
@@ -50,7 +61,7 @@
       "metadata": {
         "id": "nw7k5yCtOzoK"
       },
-      "execution_count": null,
+      "execution_count": 1,
       "outputs": []
     },
     {
@@ -330,7 +341,7 @@
         "# Compute matrix in figure 10.4 d\n",
         "def get_conv_mat_3_1_1_zp(n_out, omega):\n",
         "  omega_mat = np.zeros((n_out,n_out))\n",
-        "  # TODO Fill in this matix\n",
+        "  # TODO Fill in this matrix\n",
         "  # Replace this line:\n",
         "  omega_mat = omega_mat\n",
         "\n",

Notebooks/Chap10/10_2_Convolution_for_MNIST_1D.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyN1v/yg9PtdSVOWlYJ7bgkz",
+      "authorship_tag": "ABX9TyNJodaaCLMRWL9vTl8B/iLI",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -141,6 +141,9 @@
         "# https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html\n",
         "# https://pytorch.org/docs/1.13/generated/torch.nn.Linear.html?highlight=linear#torch.nn.Linear\n",
         "\n",
+        "# NOTE THAT THE CONVOLUTIONAL LAYERS NEED TO TAKE THE NUMBER OF INPUT CHANNELS AS A PARAMETER\n",
+        "# AND NOT THE INPUT SIZE.\n",
+        "\n",
         "# Replace the following function:\n",
         "model = nn.Sequential(\n",
         "nn.Flatten(),\n",
@@ -185,9 +188,9 @@
         "scheduler = StepLR(optimizer, step_size=20, gamma=0.5)\n",
         "# create 100 dummy data points and store in data loader class\n",
         "x_train = torch.tensor(train_data_x.transpose().astype('float32'))\n",
-        "y_train = torch.tensor(train_data_y.astype('long'))\n",
+        "y_train = torch.tensor(train_data_y.astype('long')).long()\n",
         "x_val= torch.tensor(val_data_x.transpose().astype('float32'))\n",
-        "y_val = torch.tensor(val_data_y.astype('long'))\n",
+        "y_val = torch.tensor(val_data_y.astype('long')).long()\n",
         "\n",
         "# load the data into a class that creates the batches\n",
         "data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=100, shuffle=True, worker_init_fn=np.random.seed(1))\n",

Notebooks/Chap10/10_3_2D_Convolution.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyMmbD0cKYvIHXbKX4AupA1x",
+      "authorship_tag": "ABX9TyNDaU2KKZDyY9Ea7vm/fNxo",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -114,6 +114,11 @@
         "    # Create output\n",
         "    out = np.zeros((batchSize, channelsOut, imageHeightOut, imageWidthOut), dtype=np.float32)\n",
         "\n",
+        "    # !!!!!! NOTE THERE IS A SUBTLETY HERE !!!!!!!!\n",
+        "    # I have padded the image with zeros above, so it is surrouned by a \"ring\" of zeros\n",
+        "    # That means that the image indexes are all off by one\n",
+        "    # This actually makes your code simpler\n",
+        "\n",
         "    for c_y in range(imageHeightOut):\n",
         "      for c_x in range(imageWidthOut):\n",
         "        for c_kernel_y in range(kernelHeight):\n",

Notebooks/Chap11/11_2_Residual_Networks.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyObut1y9atNUuowPT6dMY+I",
+      "authorship_tag": "ABX9TyMXS3SPB4cS/4qxix0lH/Hq",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -144,10 +144,10 @@
         "  def count_params(self):\n",
         "    return sum([p.view(-1).shape[0] for p in self.parameters()])\n",
         "\n",
-        "# # TODO -- Add residual connections to this model\n",
+        "# TODO -- Add residual connections to this model\n",
-        "# # The order of operations should similar to figure 11.5b\n",
+        "# The order of operations within each block should similar to figure 11.5b\n",
-        "# # linear1 first, ReLU+linear2 in first residual block, ReLU+linear3 in second residual block), linear4 at end\n",
+        "# ie., linear1 first, ReLU+linear2 in first residual block, ReLU+linear3 in second residual block), linear4 at end\n",
-        "# # Replace this function\n",
+        "# Replace this function\n",
         "  def forward(self, x):\n",
         "    h1 = self.linear1(x).relu()\n",
         "    h2 = self.linear2(h1).relu()\n",

Notebooks/Chap11/11_3_Batch_Normalization.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyOZaNcBrdZ9yCHhjLOwSi69",
+      "authorship_tag": "ABX9TyPVeAd3eDpEOCFh8CVyr1zz",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -267,7 +267,7 @@
         "# Use the torch function nn.BatchNorm1d\n",
         "class ResidualNetworkWithBatchNorm(torch.nn.Module):\n",
         "  def __init__(self, input_size, output_size, hidden_size=100):\n",
-        "    super(ResidualNetwork, self).__init__()\n",
+        "    super(ResidualNetworkWithBatchNorm, self).__init__()\n",
         "    self.linear1 = nn.Linear(input_size, hidden_size)\n",
         "    self.linear2 = nn.Linear(hidden_size, hidden_size)\n",
         "    self.linear3 = nn.Linear(hidden_size, hidden_size)\n",

Notebooks/Chap12/12_1_Self_Attention.ipynb

@@ -31,7 +31,7 @@
       "source": [
         "# **Notebook 12.1: Self Attention**\n",
         "\n",
-        "This notebook builds a self-attnetion mechanism from scratch, as discussed in section 12.2 of the book.\n",
+        "This notebook builds a self-attention mechanism from scratch, as discussed in section 12.2 of the book.\n",
         "\n",
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",

Notebooks/Chap12/12_4_Decoding_Strategies.ipynb

@@ -4,7 +4,7 @@
   "metadata": {
     "colab": {
       "provenance": [],
-      "authorship_tag": "ABX9TyNPrHfkLWjy3NfDHRhGG3IE",
+      "authorship_tag": "ABX9TyPsZjfqVeHYh95Hzt+hCIO7",
       "include_colab_link": true
     },
     "kernelspec": {
@@ -409,7 +409,7 @@
         "  print(\"Choosing from %d tokens\"%(thresh_index))\n",
         "  # TODO:  Find the probability value to threshold\n",
         "  # Replace this line:\n",
-        "  thresh_prob = sorted_probs_decreasing[thresh_index]\n",
+        "  thresh_prob = 0.5\n",
         "\n",
         "\n",
         "\n",

Notebooks/Chap13/13_2_Graph_Classification.ipynb

@@ -57,7 +57,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Let's build a model that maps a chemical structure to a binary decision.  This model might be used to predict whether a chemical is liquid at room temparature or not.  We'll start by drawing the chemical structure."
+        "Let's build a model that maps a chemical structure to a binary decision.  This model might be used to predict whether a chemical is liquid at room temperature or not.  We'll start by drawing the chemical structure."
       ],
       "metadata": {
         "id": "UNleESc7k5uB"
@@ -191,7 +191,7 @@
       "source": [
         "# Let's test this network\n",
         "f = graph_neural_network(A,X, Omega0, beta0, Omega1, beta1, Omega2, beta2, omega3, beta3)\n",
-        "print(\"Your value is %3f: \"%(f[0,0]), \"True value of f: 0.498010\")"
+        "print(\"Your value is %3f: \"%(f[0,0]), \"True value of f: 0.310843\")"
       ],
       "metadata": {
         "id": "X7gYgOu6uIAt"
@@ -221,7 +221,7 @@
         "X_permuted = np.copy(X)\n",
         "\n",
         "f = graph_neural_network(A_permuted,X_permuted, Omega0, beta0, Omega1, beta1, Omega2, beta2, omega3, beta3)\n",
-        "print(\"Your value is %3f: \"%(f[0,0]), \"True value of f: 0.498010\")"
+        "print(\"Your value is %3f: \"%(f[0,0]), \"True value of f: 0.310843\")"
       ],
       "metadata": {
         "id": "F0zc3U_UuR5K"

Notebooks/Chap13/13_3_Neighborhood_Sampling.ipynb

@@ -268,7 +268,7 @@
       "source": [
         "# TODO Find the nodes in hidden layer 1 that connect to the nodes in hidden layer 2\n",
         "# using the adjacency matrix.  Then sample n_sample of these nodes randomly without\n",
-        "# replacement.  Make sure not to sample nodes that were already included in hidden layer 2 our the ouput layer.\n",
+        "# replacement.  Make sure not to sample nodes that were already included in hidden layer 2 our the output layer.\n",
         "# The nodes at hidden layer 1 are the union of these nodes and the nodes in hidden layer 2\n",
         "\n",
         "# Replace this line:\n",

Notebooks/Chap16/16_3_Contraction_Mappings.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyNeCWINUqqUGKMcxsqPFTAh",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap16/16_3_Contraction_Mappings.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      },
       "source": [
         "# **Notebook 16.3: Contraction mappings**\n",
         "\n",
@@ -36,38 +25,40 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "t9vk9Elugvmi"
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "import numpy as np\n",
-        "import matplotlib.pyplot as plt"
-      ],
       "metadata": {
         "id": "OLComQyvCIJ7"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4Pfz2KSghdVI"
+      },
+      "outputs": [],
       "source": [
         "# Define a function that is a contraction mapping\n",
         "def f(z):\n",
         "    return 0.3 + 0.5 *z + 0.02 * np.sin(z*15)"
-      ],
+      ]
-      "metadata": {
-        "id": "4Pfz2KSghdVI"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "zEwCbIx0hpAI"
+      },
+      "outputs": [],
       "source": [
         "def draw_function(f, fixed_point=None):\n",
         "  z = np.arange(0,1,0.01)\n",
@@ -84,35 +75,36 @@
         "  ax.set_xlabel('Input, $z$')\n",
         "  ax.set_ylabel('Output, f$[z]$')\n",
         "  plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "zEwCbIx0hpAI"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "draw_function(f)"
-      ],
       "metadata": {
         "id": "k4e5Yu0fl8bz"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "draw_function(f)"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's find where $\\mbox{f}[z]=z$ using fixed point iteration"
-      ],
       "metadata": {
         "id": "DfgKrpCAjnol"
-      }
+      },
+      "source": [
+        "Now let's find where $\\text{f}[z]=z$ using fixed point iteration"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bAOBvZT-j3lv"
+      },
+      "outputs": [],
       "source": [
         "# Takes a function f and a starting point z\n",
         "def fixed_point_iteration(f, z0):\n",
@@ -125,115 +117,117 @@
         "\n",
         "\n",
         "  return z_out"
-      ],
+      ]
-      "metadata": {
-        "id": "bAOBvZT-j3lv"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's test that and plot the solution"
-      ],
       "metadata": {
         "id": "CAS0lgIomAa0"
-      }
+      },
+      "source": [
+        "Now let's test that and plot the solution"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "EYQZJdNPk8Lg"
+      },
+      "outputs": [],
       "source": [
         "# Now let's test that\n",
         "z = fixed_point_iteration(f, 0.2)\n",
         "draw_function(f, z)"
-      ],
+      ]
-      "metadata": {
-        "id": "EYQZJdNPk8Lg"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4DipPiqVlnwJ"
+      },
+      "outputs": [],
       "source": [
         "# Let's define another function\n",
         "def f2(z):\n",
         "    return 0.7 + -0.6 *z + 0.03 * np.sin(z*15)\n",
         "draw_function(f2)"
-      ],
+      ]
-      "metadata": {
-        "id": "4DipPiqVlnwJ"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "tYOdbWcomdEE"
+      },
+      "outputs": [],
       "source": [
         "# Now let's test that\n",
         "# TODO Before running this code, predict what you think will happen\n",
         "z = fixed_point_iteration(f2, 0.9)\n",
         "draw_function(f2, z)"
-      ],
+      ]
-      "metadata": {
-        "id": "tYOdbWcomdEE"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Mni37RUpmrIu"
+      },
+      "outputs": [],
       "source": [
         "# Let's define another function\n",
         "# Define a function that is a contraction mapping\n",
         "def f3(z):\n",
         "    return -0.2 + 1.5 *z + 0.1 * np.sin(z*15)\n",
         "draw_function(f3)"
-      ],
+      ]
-      "metadata": {
-        "id": "Mni37RUpmrIu"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "agt5mfJrnM1O"
+      },
+      "outputs": [],
       "source": [
         "# Now let's test that\n",
         "# TODO Before running this code, predict what you think will happen\n",
         "z = fixed_point_iteration(f3, 0.7)\n",
         "draw_function(f3, z)"
-      ],
+      ]
-      "metadata": {
-        "id": "agt5mfJrnM1O"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Finally, let's invert a problem of the form $y = z+ f[z]$  for a given value of $y$. What is the $z$ that maps to it?"
-      ],
       "metadata": {
         "id": "n6GI46-ZoQz6"
-      }
+      },
+      "source": [
+        "Finally, let's invert a problem of the form $y = z+ f[z]$  for a given value of $y$. What is the $z$ that maps to it?"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "def f4(z):\n",
-        "   return -0.3 + 0.5 *z + 0.02 * np.sin(z*15)"
-      ],
       "metadata": {
         "id": "dy6r3jr9rjPf"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "def f4(z):\n",
+        "   return -0.3 + 0.5 *z + 0.02 * np.sin(z*15)"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "GMX64Iz0nl-B"
+      },
+      "outputs": [],
       "source": [
         "def fixed_point_iteration_z_plus_f(f, y, z0):\n",
         "  # TODO -- write this function\n",
@@ -241,15 +235,15 @@
         "  z_out = 1\n",
         "\n",
         "  return z_out"
-      ],
+      ]
-      "metadata": {
-        "id": "GMX64Iz0nl-B"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "uXxKHad5qT8Y"
+      },
+      "outputs": [],
       "source": [
         "def draw_function2(f, y, fixed_point=None):\n",
         "  z = np.arange(0,1,0.01)\n",
@@ -267,15 +261,15 @@
         "  ax.set_xlabel('Input, $z$')\n",
         "  ax.set_ylabel('Output, z+f$[z]$')\n",
         "  plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "uXxKHad5qT8Y"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mNEBXC3Aqd_1"
+      },
+      "outputs": [],
       "source": [
         "# Test this out and draw\n",
         "y = 0.8\n",
@@ -283,12 +277,23 @@
         "draw_function2(f4,y,z)\n",
         "# If you have done this correctly, the red dot should be\n",
         "# where the cyan curve has a y value of 0.8"
+      ]
+    }
   ],
   "metadata": {
-        "id": "mNEBXC3Aqd_1"
+    "colab": {
+      "authorship_tag": "ABX9TyNeCWINUqqUGKMcxsqPFTAh",
+      "include_colab_link": true,
+      "provenance": []
     },
-      "execution_count": null,
+    "kernelspec": {
-      "outputs": []
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
     }
-  ]
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap17/17_1_Latent_Variable_Models.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyMBYNsjj1iTgHUYhAXqUYJd",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap17/17_1_Latent_Variable_Models.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      },
       "source": [
         "# **Notebook 17.1: Latent variable models**\n",
         "\n",
@@ -36,72 +25,76 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "t9vk9Elugvmi"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "outputs": [],
       "source": [
         "import numpy as np\n",
         "import matplotlib.pyplot as plt\n",
         "import scipy\n",
         "from matplotlib.colors import ListedColormap\n",
         "from matplotlib import cm"
-      ],
+      ]
-      "metadata": {
-        "id": "OLComQyvCIJ7"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "IyVn-Gi-p7wf"
+      },
       "source": [
         "We'll assume that our base distribution over the latent variables is a 1D standard normal so that\n",
         "\n",
         "\\begin{equation}\n",
-        "Pr(z) = \\mbox{Norm}_{z}[0,1]\n",
+        "Pr(z) = \\text{Norm}_{z}[0,1]\n",
         "\\end{equation}\n",
         "\n",
         "As in figure 17.2, we'll assume that the output is two dimensional, we we need to define a function that maps from the 1D latent variable to two dimensions.  Usually, we would use a neural network, but in this case, we'll just define an arbitrary relationship.\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
         "x_{1} &=& 0.5\\cdot\\exp\\Bigl[\\sin\\bigl[2+ 3.675 z \\bigr]\\Bigr]\\\\\n",
         "x_{2} &=& \\sin\\bigl[2+ 2.85 z \\bigr]\n",
-        "\\end{eqnarray}"
+        "\\end{align}"
-      ],
+      ]
-      "metadata": {
-        "id": "IyVn-Gi-p7wf"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZIfQwhd-AV6L"
+      },
+      "outputs": [],
       "source": [
         "# The function that maps z to x1 and x2\n",
         "def f(z):\n",
         "  x_1 = np.exp(np.sin(2+z*3.675)) * 0.5\n",
         "  x_2 = np.cos(2+z*2.85)\n",
         "  return x_1, x_2"
-      ],
+      ]
-      "metadata": {
-        "id": "ZIfQwhd-AV6L"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Let's plot the 3D relation between the two observed variables $x_{1}$ and $x_{2}$ and the latent variables $z$ as in figure 17.2 of the book.  We'll use the opacity to represent the prior probability $Pr(z)$."
-      ],
       "metadata": {
         "id": "KB9FU34onW1j"
-      }
+      },
+      "source": [
+        "Let's plot the 3D relation between the two observed variables $x_{1}$ and $x_{2}$ and the latent variables $z$ as in figure 17.2 of the book.  We'll use the opacity to represent the prior probability $Pr(z)$."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "lW08xqAgnP4q"
+      },
+      "outputs": [],
       "source": [
         "def draw_3d_projection(z,pr_z, x1,x2):\n",
         "  alpha = pr_z / np.max(pr_z)\n",
@@ -118,28 +111,28 @@
         "  ax.set_zlim(-1,1)\n",
         "  ax.set_box_aspect((3,1,1))\n",
         "  plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "lW08xqAgnP4q"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "9DUTauMi6tPk"
+      },
+      "outputs": [],
       "source": [
         "# Compute the prior\n",
         "def get_prior(z):\n",
         "  return scipy.stats.multivariate_normal.pdf(z)"
-      ],
+      ]
-      "metadata": {
-        "id": "9DUTauMi6tPk"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "PAzHq461VqvF"
+      },
+      "outputs": [],
       "source": [
         "# Define the latent variable values\n",
         "z = np.arange(-3.0,3.0,0.01)\n",
@@ -149,40 +142,41 @@
         "x1,x2 = f(z)\n",
         "# Plot the function\n",
         "draw_3d_projection(z,pr_z, x1,x2)"
-      ],
+      ]
-      "metadata": {
-        "id": "PAzHq461VqvF"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "The likelihood is defined as:\n",
-        "\\begin{eqnarray}\n",
-        " Pr(x_1,x_2|z) &=&  \\mbox{Norm}_{[x_1,x_2]}\\Bigl[\\mathbf{f}[z],\\sigma^{2}\\mathbf{I}\\Bigr]\n",
-        "\\end{eqnarray}\n",
-        "\n",
-        "so we will also need to define the noise level $\\sigma^2$"
-      ],
       "metadata": {
         "id": "sQg2gKR5zMrF"
-      }
+      },
+      "source": [
+        "The likelihood is defined as:\n",
+        "\\begin{align}\n",
+        " Pr(x_1,x_2|z) &=&  \\text{Norm}_{[x_1,x_2]}\\Bigl[\\mathbf{f}[z],\\sigma^{2}\\mathbf{I}\\Bigr]\n",
+        "\\end{align}\n",
+        "\n",
+        "so we will also need to define the noise level $\\sigma^2$"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "sigma_sq = 0.04"
-      ],
       "metadata": {
         "id": "In_Vg4_0nva3"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "sigma_sq = 0.04"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6P6d-AgAqxXZ"
+      },
+      "outputs": [],
       "source": [
         "# Draws a heatmap to represent a probability distribution, possibly with samples overlaed\n",
         "def plot_heatmap(x1_mesh,x2_mesh,y_mesh, x1_samples=None, x2_samples=None, title=None):\n",
@@ -207,15 +201,15 @@
         "  ax.set_xlabel('$x_1$'); ax.set_ylabel('$x_2$')\n",
         "  plt.show()\n",
         "\n"
-      ],
+      ]
-      "metadata": {
-        "id": "6P6d-AgAqxXZ"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "diYKb7_ZgjlJ"
+      },
+      "outputs": [],
       "source": [
         "# Returns the likelihood\n",
         "def get_likelihood(x1_mesh, x2_mesh, z_val):\n",
@@ -226,24 +220,25 @@
         "  mn = scipy.stats.multivariate_normal([x1, x2], [[sigma_sq, 0], [0, sigma_sq]])\n",
         "  pr_x1_x2_given_z_val = mn.pdf(np.dstack((x1_mesh, x2_mesh)))\n",
         "  return pr_x1_x2_given_z_val"
-      ],
+      ]
-      "metadata": {
-        "id": "diYKb7_ZgjlJ"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's plot the likelihood $Pr(x_1,x_2|z)$ as in fig 17.3b in the book."
-      ],
       "metadata": {
         "id": "0X4NwixzqxtZ"
-      }
+      },
+      "source": [
+        "Now let's plot the likelihood $Pr(x_1,x_2|z)$ as in fig 17.3b in the book."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "hWfqK-Oz5_DT"
+      },
+      "outputs": [],
       "source": [
         "# Choose some z value\n",
         "z_val = 1.8\n",
@@ -253,33 +248,34 @@
         "pr_x1_x2_given_z_val = get_likelihood(x1_mesh,x2_mesh, z_val)\n",
         "\n",
         "# Plot the result\n",
-        "plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2_given_z_val, title=\"Conditional distribution $Pr(x1,x2|z)$\")\n",
+        "plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2_given_z_val, title=\"Conditional distribution $Pr(x_1,x_2|z)$\")\n",
         "\n",
         "# TODO -- Experiment with different values of z and make sure that you understand the what is happening."
-      ],
+      ]
-      "metadata": {
-        "id": "hWfqK-Oz5_DT"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "25xqXnmFo-PH"
+      },
       "source": [
         "The data density is found by marginalizing over the latent variables $z$:\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
         " Pr(x_1,x_2) &=& \\int Pr(x_1,x_2, z) dz \\nonumber \\\\\n",
         " &=& \\int Pr(x_1,x_2 | z) \\cdot Pr(z)dz\\nonumber \\\\\n",
-        " &=& \\int \\mbox{Norm}_{[x_1,x_2]}\\Bigl[\\mathbf{f}[z],\\sigma^{2}\\mathbf{I}\\Bigr]\\cdot \\mbox{Norm}_{z}\\left[\\mathbf{0},\\mathbf{I}\\right]dz.\n",
+        " &=& \\int \\text{Norm}_{[x_1,x_2]}\\Bigl[\\mathbf{f}[z],\\sigma^{2}\\mathbf{I}\\Bigr]\\cdot \\text{Norm}_{z}\\left[\\mathbf{0},\\mathbf{I}\\right]dz.\n",
-        "\\end{eqnarray}"
+        "\\end{align}"
-      ],
+      ]
-      "metadata": {
-        "id": "25xqXnmFo-PH"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "H0Ijce9VzeCO"
+      },
+      "outputs": [],
       "source": [
         "# TODO Compute the data density\n",
         "# We can't integrate this function in closed form\n",
@@ -292,25 +288,26 @@
         "\n",
         "\n",
         "# Plot the result\n",
-        "plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2, title=\"Data density $Pr(x1,x2)$\")\n"
+        "plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2, title=\"Data density $Pr(x_1,x_2)$\")\n"
-      ],
+      ]
-      "metadata": {
-        "id": "H0Ijce9VzeCO"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's draw some samples from the model"
-      ],
       "metadata": {
         "id": "W264N7By_h9y"
-      }
+      },
+      "source": [
+        "Now let's draw some samples from the model"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Li3mK_I48k0k"
+      },
+      "outputs": [],
       "source": [
         "def draw_samples(n_sample):\n",
         "  # TODO Write this routine to draw n_sample samples from the model\n",
@@ -320,37 +317,38 @@
         "  x1_samples=0; x2_samples = 0;\n",
         "\n",
         "  return x1_samples, x2_samples"
-      ],
+      ]
-      "metadata": {
-        "id": "Li3mK_I48k0k"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Let's plot those samples on top of the heat map."
-      ],
       "metadata": {
         "id": "D7N7oqLe-eJO"
-      }
+      },
+      "source": [
+        "Let's plot those samples on top of the heat map."
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "x1_samples, x2_samples = draw_samples(500)\n",
-        "# Plot the result\n",
-        "plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2, x1_samples, x2_samples, title=\"Data density $Pr(x1,x2)$\")\n"
-      ],
       "metadata": {
         "id": "XRmWv99B-BWO"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "x1_samples, x2_samples = draw_samples(500)\n",
+        "# Plot the result\n",
+        "plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2, x1_samples, x2_samples, title=\"Data density $Pr(x_1,x_2)$\")\n"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "PwOjzPD5_1OF"
+      },
+      "outputs": [],
       "source": [
         "# Return the posterior distribution\n",
         "def get_posterior(x1,x2):\n",
@@ -364,15 +362,15 @@
         "\n",
         "\n",
         "  return z, pr_z_given_x1_x2"
-      ],
+      ]
-      "metadata": {
-        "id": "PwOjzPD5_1OF"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "PKFUY42K-Tp7"
+      },
+      "outputs": [],
       "source": [
         "x1 = 0.9; x2 = -0.9\n",
         "z, pr_z_given_x1_x2 = get_posterior(x1,x2)\n",
@@ -385,12 +383,23 @@
         "ax.set_xlim([-3,3])\n",
         "ax.set_ylim([0,1.5 * np.max(pr_z_given_x1_x2)])\n",
         "plt.show()"
+      ]
+    }
   ],
   "metadata": {
-        "id": "PKFUY42K-Tp7"
+    "colab": {
+      "authorship_tag": "ABX9TyOSEQVqxE5KrXmsZVh9M3gq",
+      "include_colab_link": true,
+      "provenance": []
     },
-      "execution_count": null,
+    "kernelspec": {
-      "outputs": []
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
     }
-  ]
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap17/17_2_Reparameterization_Trick.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyOxO2/0DTH4n4zhC97qbagY",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap17/17_2_Reparameterization_Trick.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      },
       "source": [
         "# **Notebook 17.2: Reparameterization trick**\n",
         "\n",
@@ -36,30 +25,31 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "t9vk9Elugvmi"
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "import numpy as np\n",
-        "import matplotlib.pyplot as plt"
-      ],
       "metadata": {
         "id": "OLComQyvCIJ7"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "paLz5RukZP1J"
+      },
       "source": [
-        "The reparameterization trick computes the derivative of an expectation of a function $\\mbox{f}[x]$:\n",
+        "The reparameterization trick computes the derivative of an expectation of a function $\\text{f}[x]$:\n",
         "\n",
         "\\begin{equation}\n",
-        "\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\bigl[\\mbox{f}[x]\\bigr],\n",
+        "\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\bigl[\\text{f}[x]\\bigr],\n",
         "\\end{equation}\n",
         "\n",
         "with respect to the parameters $\\boldsymbol\\phi$ of the distribution $Pr(x|\\boldsymbol\\phi)$ that the expectation is over.\n",
@@ -67,21 +57,23 @@
         "Let's consider a simple concrete example, where:\n",
         "\n",
         "\\begin{equation}\n",
-        "Pr(x|\\phi) = \\mbox{Norm}_{x}\\Bigl[\\mu, \\sigma^2\\Bigr]=\\mbox{Norm}_{x}\\Bigl[\\phi^3,(\\exp[\\phi])^2\\Bigr]\n",
+        "Pr(x|\\phi) = \\text{Norm}_{x}\\Bigl[\\mu, \\sigma^2\\Bigr]=\\text{Norm}_{x}\\Bigl[\\phi^3,(\\exp[\\phi])^2\\Bigr]\n",
         "\\end{equation}\n",
         "\n",
         "and\n",
         "\n",
         "\\begin{equation}\n",
-        "\\mbox{f}[x] = x^2+\\sin[x]\n",
+        "\\text{f}[x] = x^2+\\sin[x]\n",
         "\\end{equation}"
-      ],
+      ]
-      "metadata": {
-        "id": "paLz5RukZP1J"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "FdEbMnDBY0i9"
+      },
+      "outputs": [],
       "source": [
         "# Let's approximate this expectation for a particular value of phi\n",
         "def compute_expectation(phi, n_samples):\n",
@@ -96,15 +88,15 @@
         "\n",
         "\n",
         "  return expected_f_given_phi"
-      ],
+      ]
-      "metadata": {
-        "id": "FdEbMnDBY0i9"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "FTh7LJ0llNJZ"
+      },
+      "outputs": [],
       "source": [
         "# Set the seed so the random numbers are all the same\n",
         "np.random.seed(0)\n",
@@ -119,24 +111,25 @@
         "n_samples = 10000000\n",
         "expected_f_given_phi2 = compute_expectation(phi2, n_samples)\n",
         "print(\"Your value: \", expected_f_given_phi2, \", True value:  0.8176793102849222\")"
-      ],
+      ]
-      "metadata": {
-        "id": "FTh7LJ0llNJZ"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Le't plot this expectation as a function of phi"
-      ],
       "metadata": {
         "id": "r5Hl2QkimWx9"
-      }
+      },
+      "source": [
+        "Le't plot this expectation as a function of phi"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "05XxVLJxmkER"
+      },
+      "outputs": [],
       "source": [
         "phi_vals = np.arange(-1.5,1.5, 0.05)\n",
         "expected_vals = np.zeros_like(phi_vals)\n",
@@ -149,15 +142,14 @@
         "ax.set_xlabel('Parameter $\\phi$')\n",
         "ax.set_ylabel('$\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "05XxVLJxmkER"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "zTCykVeWqj_O"
+      },
       "source": [
         "It's this curve that we want to find the derivative of (so for example, we could run gradient descent and find the minimum.\n",
         "\n",
@@ -166,28 +158,30 @@
         "The answer is the reparameterization trick.  We note that:\n",
         "\n",
         "\\begin{equation}\n",
-        "\\mbox{Norm}_{x}\\Bigl[\\mu, \\sigma^2\\Bigr]=\\mbox{Norm}_{x}\\Bigl[0, 1\\Bigr] \\times \\sigma + \\mu\n",
+        "\\text{Norm}_{x}\\Bigl[\\mu, \\sigma^2\\Bigr]=\\text{Norm}_{x}\\Bigl[0, 1\\Bigr] \\times \\sigma + \\mu\n",
         "\\end{equation}\n",
         "\n",
         "and so:\n",
         "\n",
         "\\begin{equation}\n",
-        "\\mbox{Norm}_{x}\\Bigl[\\phi^3,(\\exp[\\phi])^2\\Bigr]  = \\mbox{Norm}_{x}\\Bigl[0, 1\\Bigr] \\times \\exp[\\phi]+ \\phi^3\n",
+        "\\text{Norm}_{x}\\Bigl[\\phi^3,(\\exp[\\phi])^2\\Bigr]  = \\text{Norm}_{x}\\Bigl[0, 1\\Bigr] \\times \\exp[\\phi]+ \\phi^3\n",
         "\\end{equation}\n",
         "\n",
-        "So, if we draw a sample $\\epsilon^*$ from $\\mbox{Norm}_{\\epsilon}[0, 1]$, then we can compute a sample $x^*$ as:\n",
+        "So, if we draw a sample $\\epsilon^*$ from $\\text{Norm}_{\\epsilon}[0, 1]$, then we can compute a sample $x^*$ as:\n",
         "\n",
-        "\\begin{eqnarray*}\n",
+        "\\begin{align}\n",
         "x^* &=& \\epsilon^* \\times \\sigma + \\mu \\\\\n",
         "&=& \\epsilon^* \\times \\exp[\\phi]+ \\phi^3\n",
-        "\\end{eqnarray*}"
+        "\\end{align}"
-      ],
+      ]
-      "metadata": {
-        "id": "zTCykVeWqj_O"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "w13HVpi9q8nF"
+      },
+      "outputs": [],
       "source": [
         "def compute_df_dx_star(x_star):\n",
         "  # TODO Compute this derivative (function defined at the top)\n",
@@ -222,15 +216,15 @@
         "\n",
         "\n",
         "  return df_dphi"
-      ],
+      ]
-      "metadata": {
-        "id": "w13HVpi9q8nF"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ntQT4An79kAl"
+      },
+      "outputs": [],
       "source": [
         "# Set the seed so the random numbers are all the same\n",
         "np.random.seed(0)\n",
@@ -241,15 +235,15 @@
         "\n",
         "deriv = compute_derivative_of_expectation(phi1, n_samples)\n",
         "print(\"Your value: \", deriv, \", True value:  5.726338035051403\")"
-      ],
+      ]
-      "metadata": {
-        "id": "ntQT4An79kAl"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "t0Jqd_IN_lMU"
+      },
+      "outputs": [],
       "source": [
         "phi_vals = np.arange(-1.5,1.5, 0.05)\n",
         "deriv_vals = np.zeros_like(phi_vals)\n",
@@ -262,37 +256,37 @@
         "ax.set_xlabel('Parameter $\\phi$')\n",
         "ax.set_ylabel('$\\partial/\\partial\\phi\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "t0Jqd_IN_lMU"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "This should look plausibly like the derivative of the function we plotted above!"
-      ],
       "metadata": {
         "id": "ASu4yKSwAEYI"
-      }
+      },
+      "source": [
+        "This should look plausibly like the derivative of the function we plotted above!"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "xoFR1wifc8-b"
+      },
       "source": [
-        "The reparameterization trick computes the derivative of an expectation of a function $\\mbox{f}[x]$:\n",
+        "The reparameterization trick computes the derivative of an expectation of a function $\\text{f}[x]$:\n",
         "\n",
         "\\begin{equation}\n",
-        "\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\bigl[\\mbox{f}[x]\\bigr],\n",
+        "\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\bigl[\\text{f}[x]\\bigr],\n",
         "\\end{equation}\n",
         "\n",
         "with respect to the parameters $\\boldsymbol\\phi$ of the distribution $Pr(x|\\boldsymbol\\phi)$ that the expectation is over. This derivative can also be computed as:\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
-        "\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\bigl[\\mbox{f}[x]\\bigr] &=& \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\left[\\mbox{f}[x]\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\log\\bigl[ Pr(x|\\boldsymbol\\phi)\\bigr]\\right]\\nonumber \\\\\n",
+        "\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\bigl[\\text{f}[x]\\bigr] &=& \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\left[\\text{f}[x]\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\log\\bigl[ Pr(x|\\boldsymbol\\phi)\\bigr]\\right]\\nonumber \\\\\n",
-        "&\\approx & \\frac{1}{I}\\sum_{i=1}^{I}\\mbox{f}[x_i]\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\log\\bigl[ Pr(x_i|\\boldsymbol\\phi)\\bigr].\n",
+        "&\\approx & \\frac{1}{I}\\sum_{i=1}^{I}\\text{f}[x_i]\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\log\\bigl[ Pr(x_i|\\boldsymbol\\phi)\\bigr].\n",
-        "\\end{eqnarray}\n",
+        "\\end{align}\n",
         "\n",
         "This method is known as the REINFORCE algorithm or score function estimator.  Problem 17.5 asks you to prove this relation.  Let's use this method to compute the gradient and compare.\n",
         "\n",
@@ -301,13 +295,15 @@
         "\\begin{equation}\n",
         " Pr(x|\\mu,\\sigma^2) = \\frac{1}{\\sqrt{2\\pi\\sigma^{2}}}\\exp\\left[-\\frac{(x-\\mu)^{2}}{2\\sigma^{2}}\\right].\n",
         "\\end{equation}\n"
-      ],
+      ]
-      "metadata": {
-        "id": "xoFR1wifc8-b"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4TUaxiWvASla"
+      },
+      "outputs": [],
       "source": [
         "def d_log_pr_x_given_phi(x,phi):\n",
         "  # TODO -- fill in this function\n",
@@ -333,15 +329,15 @@
         "\n",
         "\n",
         "  return deriv"
-      ],
+      ]
-      "metadata": {
-        "id": "4TUaxiWvASla"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "0RSN32Rna_C_"
+      },
+      "outputs": [],
       "source": [
         "# Set the seed so the random numbers are all the same\n",
         "np.random.seed(0)\n",
@@ -352,15 +348,15 @@
         "\n",
         "deriv = compute_derivative_of_expectation_score_function(phi1, n_samples)\n",
         "print(\"Your value: \", deriv, \", True value:  5.724609927313369\")"
-      ],
+      ]
-      "metadata": {
-        "id": "0RSN32Rna_C_"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "EM_i5zoyElHR"
+      },
+      "outputs": [],
       "source": [
         "phi_vals = np.arange(-1.5,1.5, 0.05)\n",
         "deriv_vals = np.zeros_like(phi_vals)\n",
@@ -373,24 +369,25 @@
         "ax.set_xlabel('Parameter $\\phi$')\n",
         "ax.set_ylabel('$\\partial/\\partial\\phi\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "EM_i5zoyElHR"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "This should look the same as the derivative that we computed with the reparameterization trick.  So, is there any advantage to one way or the other?  Let's compare the variances of the estimates\n"
-      ],
       "metadata": {
         "id": "1TWBiUC7bQSw"
-      }
+      },
+      "source": [
+        "This should look the same as the derivative that we computed with the reparameterization trick.  So, is there any advantage to one way or the other?  Let's compare the variances of the estimates\n"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "vV_Jx5bCbQGs"
+      },
+      "outputs": [],
       "source": [
         "n_estimate = 100\n",
         "n_sample = 1000\n",
@@ -403,21 +400,33 @@
         "\n",
         "print(\"Variance of reparameterization estimator\", np.var(reparam_estimates))\n",
         "print(\"Variance of score function estimator\", np.var(score_function_estimates))"
-      ],
+      ]
-      "metadata": {
-        "id": "vV_Jx5bCbQGs"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "The variance of the reparameterization estimator should be quite a bit lower than the score function estimator which is why it is preferred in this situation."
-      ],
       "metadata": {
         "id": "d-0tntSYdKPR"
-      }
+      },
-    }
+      "source": [
+        "The variance of the reparameterization estimator should be quite a bit lower than the score function estimator which is why it is preferred in this situation."
       ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "authorship_tag": "ABX9TyOxO2/0DTH4n4zhC97qbagY",
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap17/17_3_Importance_Sampling.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyMvae+1cigwg2Htl6vt1Who",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap17/17_3_Importance_Sampling.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      },
       "source": [
         "# **Notebook 17.3: Importance sampling**\n",
         "\n",
@@ -36,25 +25,26 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "t9vk9Elugvmi"
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "import numpy as np\n",
-        "import matplotlib.pyplot as plt"
-      ],
       "metadata": {
         "id": "OLComQyvCIJ7"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "f7a6xqKjkmvT"
+      },
       "source": [
         "Let's approximate the expectation\n",
         "\n",
@@ -65,7 +55,7 @@
         "where\n",
         "\n",
         "\\begin{equation}\n",
-        "Pr(y)=\\mbox{Norm}_y[0,1]\n",
+        "Pr(y)=\\text{Norm}_y[0,1]\n",
         "\\end{equation}\n",
         "\n",
         "by drawing $I$ samples $y_i$ and using the formula:\n",
@@ -73,13 +63,15 @@
         "\\begin{equation}\n",
         "\\mathbb{E}_{y}\\Bigl[\\exp\\bigl[- (y-1)^4\\bigr]\\Bigr] \\approx \\frac{1}{I} \\sum_{i=1}^I \\exp\\bigl[-(y-1)^4 \\bigr]\n",
         "\\end{equation}"
-      ],
+      ]
-      "metadata": {
-        "id": "f7a6xqKjkmvT"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "VjkzRr8o2ksg"
+      },
+      "outputs": [],
       "source": [
         "def f(y):\n",
         "  return np.exp(-(y-1) *(y-1) *(y-1) * (y-1))\n",
@@ -95,15 +87,15 @@
         "ax.set_xlabel(\"$y$\")\n",
         "ax.legend()\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "VjkzRr8o2ksg"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "LGAKHjUJnWmy"
+      },
+      "outputs": [],
       "source": [
         "def compute_expectation(n_samples):\n",
         "  # TODO -- compute this expectation\n",
@@ -114,15 +106,15 @@
         "\n",
         "\n",
         "  return expectation"
-      ],
+      ]
-      "metadata": {
-        "id": "LGAKHjUJnWmy"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "nmvixMqgodIP"
+      },
+      "outputs": [],
       "source": [
         "# Set the seed so the random numbers are all the same\n",
         "np.random.seed(0)\n",
@@ -131,26 +123,27 @@
         "n_samples = 100000000\n",
         "expected_f= compute_expectation(n_samples)\n",
         "print(\"Your value: \", expected_f, \", True value:  0.43160702267383166\")"
-      ],
+      ]
-      "metadata": {
-        "id": "nmvixMqgodIP"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "Jr4UPcqmnXCS"
+      },
       "source": [
         "Let's investigate how the variance of this approximation decreases as we increase the number of samples $N$.\n",
         "\n",
         "\n"
-      ],
+      ]
-      "metadata": {
-        "id": "Jr4UPcqmnXCS"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "yrDp1ILUo08j"
+      },
+      "outputs": [],
       "source": [
         "def compute_mean_variance(n_sample):\n",
         "  n_estimate = 10000\n",
@@ -158,15 +151,15 @@
         "  for i in range(n_estimate):\n",
         "    estimates[i] = compute_expectation(n_sample.astype(int))\n",
         "  return np.mean(estimates), np.var(estimates)"
-      ],
+      ]
-      "metadata": {
-        "id": "yrDp1ILUo08j"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "BcUVsodtqdey"
+      },
+      "outputs": [],
       "source": [
         "# Compute the mean and variance for 1,2,... 20 samples\n",
         "n_sample_all = np.array([1.,2,3,4,5,6,7,8,9,10,15,20,25,30,45,50,60,70,80,90,100,150,200,250,300,350,400,450,500])\n",
@@ -175,15 +168,15 @@
         "for i in range(len(n_sample_all)):\n",
         "  print(\"Computing mean and variance for expectation with %d samples\"%(n_sample_all[i]))\n",
         "  mean_all[i],variance_all[i] = compute_mean_variance(n_sample_all[i])"
-      ],
+      ]
-      "metadata": {
-        "id": "BcUVsodtqdey"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "feXmyk0krpUi"
+      },
+      "outputs": [],
       "source": [
         "fig,ax = plt.subplots()\n",
         "ax.semilogx(n_sample_all, mean_all,'r-',label='mean estimate')\n",
@@ -193,38 +186,40 @@
         "ax.plot([0,500],[0.43160702267383166,0.43160702267383166],'k--',label='true value')\n",
         "ax.legend()\n",
         "plt.show()\n"
-      ],
+      ]
-      "metadata": {
-        "id": "feXmyk0krpUi"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "As you might expect, the more samples that we use to compute the approximate estimate, the lower the variance of the estimate."
-      ],
       "metadata": {
         "id": "XTUpxFlSuOl7"
-      }
+      },
+      "source": [
+        "As you might expect, the more samples that we use to compute the approximate estimate, the lower the variance of the estimate."
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "6hxsl3Pxo1TT"
+      },
       "source": [
         " Now consider the function\n",
         " \\begin{equation}\n",
         " \\mbox{f}[y]= 20.446\\exp\\left[-(y-3)^4\\right],\n",
         " \\end{equation}\n",
         "\n",
-        "which decreases rapidly as we move away from the position $y=4$."
+        "which decreases rapidly as we move away from the position $y=3$."
-      ],
+      ]
-      "metadata": {
-        "id": "6hxsl3Pxo1TT"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "znydVPW7sL4P"
+      },
+      "outputs": [],
       "source": [
         "def f2(y):\n",
         "  return 20.446*np.exp(- (y-3) *(y-3) *(y-3) * (y-3))\n",
@@ -236,46 +231,47 @@
         "ax.set_xlabel(\"$y$\")\n",
         "ax.legend()\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "znydVPW7sL4P"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "G9Xxo0OJsIqD"
+      },
       "source": [
         "Let's again, compute the expectation:\n",
         "\n",
-        "\\begin{eqnarray}\n",
+        "\\begin{align}\n",
-        "\\mathbb{E}_{y}\\left[\\mbox{f}[y]\\right] &=& \\int \\mbox{f}[y] Pr(y) dy\\\\\n",
+        "\\mathbb{E}_{y}\\left[\\text{f}[y]\\right] &=& \\int \\text{f}[y] Pr(y) dy\\\\\n",
-        "&\\approx& \\frac{1}{I} \\mbox{f}[y]\n",
+        "&\\approx& \\frac{1}{I} \\text{f}[y]\n",
-        "\\end{eqnarray}\n",
+        "\\end{align}\n",
         "\n",
-        "where $Pr(y)=\\mbox{Norm}_y[0,1]$ by approximating with samples $y_{i}$.\n"
+        "where $Pr(y)=\\text{Norm}_y[0,1]$ by approximating with samples $y_{i}$.\n"
-      ],
+      ]
-      "metadata": {
-        "id": "G9Xxo0OJsIqD"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "l8ZtmkA2vH4y"
+      },
+      "outputs": [],
       "source": [
         "def compute_expectation2(n_samples):\n",
         "  y = np.random.normal(size=(n_samples,1))\n",
         "  expectation = np.mean(f2(y))\n",
         "\n",
         "  return expectation"
-      ],
+      ]
-      "metadata": {
-        "id": "l8ZtmkA2vH4y"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "dfUQyJ-svZ6F"
+      },
+      "outputs": [],
       "source": [
         "# Set the seed so the random numbers are all the same\n",
         "np.random.seed(0)\n",
@@ -284,26 +280,27 @@
         "n_samples = 100000000\n",
         "expected_f2= compute_expectation2(n_samples)\n",
         "print(\"Expected value: \", expected_f2)"
-      ],
+      ]
-      "metadata": {
-        "id": "dfUQyJ-svZ6F"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "2sVDqP0BvxqM"
+      },
       "source": [
         "I deliberately chose this function, because it's expectation is roughly the same as for the previous function.\n",
         "\n",
         "Again, let's look at the mean and the  variance of the estimates"
-      ],
+      ]
-      "metadata": {
-        "id": "2sVDqP0BvxqM"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mHnILRkOv0Ir"
+      },
+      "outputs": [],
       "source": [
         "def compute_mean_variance2(n_sample):\n",
         "  n_estimate = 10000\n",
@@ -318,15 +315,15 @@
         "for i in range(len(n_sample_all)):\n",
         "  print(\"Computing variance for expectation with %d samples\"%(n_sample_all[i]))\n",
         "  mean_all2[i], variance_all2[i] = compute_mean_variance2(n_sample_all[i])"
-      ],
+      ]
-      "metadata": {
-        "id": "mHnILRkOv0Ir"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "FkCX-hxxAnsw"
+      },
+      "outputs": [],
       "source": [
         "fig,ax1 = plt.subplots()\n",
         "ax1.semilogx(n_sample_all, mean_all,'r-',label='mean estimate')\n",
@@ -348,39 +345,41 @@
         "ax2.set_title(\"Second function\")\n",
         "ax2.legend()\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "FkCX-hxxAnsw"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "EtBP6NeLwZqz"
+      },
       "source": [
         "You can see that the variance of the estimate of the second function is considerably worse than the estimate of the variance of estimate of the first function\n",
         "\n",
         "TODO:  Think about why this is."
-      ],
+      ]
-      "metadata": {
-        "id": "EtBP6NeLwZqz"
-      }
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "_wuF-NoQu1--"
+      },
       "source": [
         " Now let's repeat this experiment with the second function, but this time use importance sampling with auxiliary distribution:\n",
         "\n",
         " \\begin{equation}\n",
-        "   q(y)=\\mbox{Norm}_{y}[3,1]\n",
+        "   q(y)=\\text{Norm}_{y}[3,1]\n",
         " \\end{equation}\n"
-      ],
+      ]
-      "metadata": {
-        "id": "_wuF-NoQu1--"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "jPm0AVYVIDnn"
+      },
+      "outputs": [],
       "source": [
         "def q_y(y):\n",
         "  return (1/np.sqrt(2*np.pi)) * np.exp(-0.5 * (y-3) * (y-3))\n",
@@ -395,15 +394,15 @@
         "  expectation = 0\n",
         "\n",
         "  return expectation"
-      ],
+      ]
-      "metadata": {
-        "id": "jPm0AVYVIDnn"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "No2ByVvOM2yQ"
+      },
+      "outputs": [],
       "source": [
         "# Set the seed so the random numbers are all the same\n",
         "np.random.seed(0)\n",
@@ -412,15 +411,15 @@
         "n_samples = 100000000\n",
         "expected_f2= compute_expectation2b(n_samples)\n",
         "print(\"Your value: \", expected_f2,\", True value:  0.43163734204459125 \")"
-      ],
+      ]
-      "metadata": {
-        "id": "No2ByVvOM2yQ"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6v8Jc7z4M3Mk"
+      },
+      "outputs": [],
       "source": [
         "def compute_mean_variance2b(n_sample):\n",
         "  n_estimate = 10000\n",
@@ -435,15 +434,15 @@
         "for i in range(len(n_sample_all)):\n",
         "  print(\"Computing variance for expectation with %d samples\"%(n_sample_all[i]))\n",
         "  mean_all2b[i], variance_all2b[i] = compute_mean_variance2b(n_sample_all[i])"
-      ],
+      ]
-      "metadata": {
-        "id": "6v8Jc7z4M3Mk"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "C0beD4sNNM3L"
+      },
+      "outputs": [],
       "source": [
         "fig,ax1 = plt.subplots()\n",
         "ax1.semilogx(n_sample_all, mean_all,'r-',label='mean estimate')\n",
@@ -476,21 +475,33 @@
         "ax2.set_title(\"Second function with importance sampling\")\n",
         "ax2.legend()\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "C0beD4sNNM3L"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "You can see that the importance sampling technique has reduced the amount of variance for any given number of samples."
-      ],
       "metadata": {
         "id": "y8rgge9MNiOc"
-      }
+      },
-    }
+      "source": [
+        "You can see that the importance sampling technique has reduced the amount of variance for any given number of samples."
       ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "authorship_tag": "ABX9TyNecz9/CDOggPSmy1LjT/Dv",
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap18/18_1_Diffusion_Encoder.ipynb

@@ -1,26 +1,10 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyMpC8kgLnXx0XQBtwNAQ4jJ",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_1_Diffusion_Encoder.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
@@ -28,6 +12,9 @@
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      },
       "source": [
         "# **Notebook 18.1: Diffusion Encoder**\n",
         "\n",
@@ -36,27 +23,29 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "t9vk9Elugvmi"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "outputs": [],
       "source": [
         "import numpy as np\n",
         "import matplotlib.pyplot as plt\n",
         "from matplotlib.colors import ListedColormap\n",
         "from operator import itemgetter"
-      ],
+      ]
-      "metadata": {
-        "id": "OLComQyvCIJ7"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4PM8bf6lO0VE"
+      },
+      "outputs": [],
       "source": [
         "#Create pretty colormap as in book\n",
         "my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
@@ -66,28 +55,28 @@
         "b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
         "my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
         "my_colormap = ListedColormap(my_colormap_vals)"
-      ],
+      ]
-      "metadata": {
-        "id": "4PM8bf6lO0VE"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ONGRaQscfIOo"
+      },
+      "outputs": [],
       "source": [
         "# Probability distribution for normal\n",
         "def norm_pdf(x, mu, sigma):\n",
         "    return np.exp(-0.5 * (x-mu) * (x-mu) / (sigma * sigma)) / np.sqrt(2*np.pi*sigma*sigma)"
-      ],
+      ]
-      "metadata": {
-        "id": "ONGRaQscfIOo"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gZvG0MKhfY8Y"
+      },
+      "outputs": [],
       "source": [
         "# True distribution is a mixture of four Gaussians\n",
         "class TrueDataDistribution:\n",
@@ -108,15 +97,15 @@
         "    mu_list = list(itemgetter(*hidden)(self.mu))\n",
         "    sigma_list = list(itemgetter(*hidden)(self.sigma))\n",
         "    return mu_list + sigma_list * epsilon"
-      ],
+      ]
-      "metadata": {
-        "id": "gZvG0MKhfY8Y"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "qXmej3TUuQyp"
+      },
+      "outputs": [],
       "source": [
         "# Define ground truth probability distribution that we will model\n",
         "true_dist = TrueDataDistribution()\n",
@@ -130,24 +119,24 @@
         "ax.set_ylim(0,1.0)\n",
         "ax.set_xlim(-3,3)\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "qXmej3TUuQyp"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "Let's first implement the forward process"
-      ],
       "metadata": {
         "id": "XHdtfRP47YLy"
-      }
+      },
+      "source": [
+        "Let's first implement the forward process"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "hkApJ2VJlQuk"
+      },
+      "outputs": [],
       "source": [
         "# Do one step of diffusion (equation 18.1)\n",
         "def diffuse_one_step(z_t_minus_1, beta_t):\n",
@@ -157,24 +146,24 @@
         "  z_t = np.zeros_like(z_t_minus_1)\n",
         "\n",
         "  return z_t"
-      ],
+      ]
-      "metadata": {
-        "id": "hkApJ2VJlQuk"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "Now let's run the diffusion process for a whole bunch of samples"
-      ],
       "metadata": {
         "id": "ECAUfHNi9NVW"
-      }
+      },
+      "source": [
+        "Now let's run the diffusion process for a whole bunch of samples"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "M-TY5w9Q8LYW"
+      },
+      "outputs": [],
       "source": [
         "# Generate some samples\n",
         "n_sample = 10000\n",
@@ -192,24 +181,24 @@
         "\n",
         "for t in range(T):\n",
         "  samples[t+1,:] = diffuse_one_step(samples[t,:], beta)"
-      ],
+      ]
-      "metadata": {
-        "id": "M-TY5w9Q8LYW"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "Let's, plot the evolution of a few paths as in figure 18.2"
-      ],
       "metadata": {
         "id": "jYrAW6tN-gJ4"
-      }
+      },
+      "source": [
+        "Let's, plot the evolution of a few paths as in figure 18.2"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4XU6CDZC_kFo"
+      },
+      "outputs": [],
       "source": [
         "fig, ax = plt.subplots()\n",
         "t_vals = np.arange(0,101,1)\n",
@@ -223,24 +212,24 @@
         "ax.set_xlabel('value')\n",
         "ax.set_ylabel('z_{t}')\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "4XU6CDZC_kFo"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "Notice that the samples have a tendency to move toward the center.  Now let's look at the histogram of the samples at each stage"
-      ],
       "metadata": {
         "id": "SGTYGGevAktz"
-      }
+      },
+      "source": [
+        "Notice that the samples have a tendency to move toward the center.  Now let's look at the histogram of the samples at each stage"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bn5E5NzL-evM"
+      },
+      "outputs": [],
       "source": [
         "def draw_hist(z_t,title=''):\n",
         "  fig, ax = plt.subplots()\n",
@@ -248,17 +237,17 @@
         "  plt.hist(z_t , bins=np.arange(-3,3, 0.1), density = True)\n",
         "  ax.set_xlim([-3,3])\n",
         "  ax.set_ylim([0,1.0])\n",
-        "  ax.set_title('title')\n",
+        "  ax.set_title(title)\n",
         "  plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "bn5E5NzL-evM"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "pn_XD-EhBlwk"
+      },
+      "outputs": [],
       "source": [
         "draw_hist(samples[0,:],'Original data')\n",
         "draw_hist(samples[5,:],'Time step 5')\n",
@@ -267,33 +256,33 @@
         "draw_hist(samples[40,:],'Time step 40')\n",
         "draw_hist(samples[80,:],'Time step 80')\n",
         "draw_hist(samples[100,:],'Time step 100')"
-      ],
+      ]
-      "metadata": {
-        "id": "pn_XD-EhBlwk"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "You can clearly see that as the diffusion process continues, the data becomes more Gaussian."
-      ],
       "metadata": {
         "id": "skuLfGl5Czf4"
-      }
+      },
+      "source": [
+        "You can clearly see that as the diffusion process continues, the data becomes more Gaussian."
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "Now let's investigate the diffusion kernel as in figure 18.3 of the book.\n"
-      ],
       "metadata": {
         "id": "s37CBSzzK7wh"
-      }
+      },
+      "source": [
+        "Now let's investigate the diffusion kernel as in figure 18.3 of the book.\n"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "vL62Iym0LEtY"
+      },
+      "outputs": [],
       "source": [
         "def diffusion_kernel(x, t, beta):\n",
         "    # TODO -- write this function\n",
@@ -301,15 +290,15 @@
         "    dk_mean = 0.0 ; dk_std = 1.0\n",
         "\n",
         "    return dk_mean, dk_std"
-      ],
+      ]
-      "metadata": {
-        "id": "vL62Iym0LEtY"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "KtP1KF8wMh8o"
+      },
+      "outputs": [],
       "source": [
         "def draw_prob_dist(x_plot_vals, prob_dist, title=''):\n",
         "  fig, ax = plt.subplots()\n",
@@ -363,47 +352,47 @@
         "    draw_prob_dist(x_plot_vals, diffusion_kernels[20,:],'$q(z_{20}|x)$')\n",
         "    draw_prob_dist(x_plot_vals, diffusion_kernels[40,:],'$q(z_{40}|x)$')\n",
         "    draw_prob_dist(x_plot_vals, diffusion_kernels[80,:],'$q(z_{80}|x)$')"
-      ],
+      ]
-      "metadata": {
-        "id": "KtP1KF8wMh8o"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "x = -2\n",
-        "compute_and_plot_diffusion_kernels(x, T, beta, my_colormap)"
-      ],
       "metadata": {
         "id": "g8TcI5wtRQsx"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "x = -2\n",
+        "compute_and_plot_diffusion_kernels(x, T, beta, my_colormap)"
+      ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "TODO -- Run this for different version of $x$ and check that you understand how the graphs change"
-      ],
       "metadata": {
         "id": "-RuN2lR28-hK"
-      }
+      },
+      "source": [
+        "TODO -- Run this for different version of $x$ and check that you understand how the graphs change"
+      ]
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "n-x6Whz2J_zy"
+      },
       "source": [
         "Finally, let's estimate the marginal distributions empirically and visualize them as in figure 18.4 of the book. This is only tractable because the data is in one dimension and we know the original distribution.\n",
         "\n",
         "The marginal distribution at time t is the sum of the diffusion kernels for each position x, weighted by the probability of seeing that value of x in the true distribution."
-      ],
+      ]
-      "metadata": {
-        "id": "n-x6Whz2J_zy"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "YzN5duYpg7C-"
+      },
+      "outputs": [],
       "source": [
         "def diffusion_marginal(x_plot_vals, pr_x_true, t, beta):\n",
         "    # If time is zero then marginal is just original distribution\n",
@@ -414,11 +403,11 @@
         "    marginal_at_time_t = np.zeros_like(pr_x_true);\n",
         "\n",
         "\n",
-        "    # TODO Write ths function\n",
+        "    # TODO Write this function\n",
         "    # 1. For each x (value in x_plot_vals):\n",
         "      # 2. Compute the mean and variance of the diffusion kernel at time t\n",
         "      # 3. Compute pdf of this Gaussian at every x_plot_val\n",
-        "      # 4. Weight Gaussian by probability at position x and by 0.01 to compensate for bin size\n",
+        "      # 4. Weight Gaussian by probability at position x and by 0.01 to componensate for bin size\n",
         "      # 5. Accumulate weighted Gaussian in marginal at time t.\n",
         "    # 6. Multiply result by 0.01 to compensate for bin size\n",
         "    # Replace this line:\n",
@@ -427,15 +416,15 @@
         "\n",
         "\n",
         "    return marginal_at_time_t"
-      ],
+      ]
-      "metadata": {
-        "id": "YzN5duYpg7C-"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "OgEU9sxjRaeO"
+      },
+      "outputs": [],
       "source": [
         "x_plot_vals = np.arange(-3,3,0.01)\n",
         "marginal_distributions = np.zeros((T+1,len(x_plot_vals)))\n",
@@ -460,12 +449,23 @@
         "draw_prob_dist(x_plot_vals, marginal_distributions[0,:],'$q(z_{0})$')\n",
         "draw_prob_dist(x_plot_vals, marginal_distributions[20,:],'$q(z_{20})$')\n",
         "draw_prob_dist(x_plot_vals, marginal_distributions[60,:],'$q(z_{60})$')"
+      ]
+    }
   ],
   "metadata": {
-        "id": "OgEU9sxjRaeO"
+    "colab": {
+      "authorship_tag": "ABX9TyMpC8kgLnXx0XQBtwNAQ4jJ",
+      "include_colab_link": true,
+      "provenance": []
     },
-      "execution_count": null,
+    "kernelspec": {
-      "outputs": []
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
     }
-  ]
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap18/18_2_1D_Diffusion_Model.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyM4DdZDGoP1xGst+Nn+rwvt",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_2_1D_Diffusion_Model.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      },
       "source": [
         "# **Notebook 18.2: 1D Diffusion Model**\n",
         "\n",
@@ -36,13 +25,15 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "t9vk9Elugvmi"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "outputs": [],
       "source": [
         "import numpy as np\n",
         "import matplotlib.pyplot as plt\n",
@@ -50,15 +41,15 @@
         "from operator import itemgetter\n",
         "from scipy import stats\n",
         "from IPython.display import display, clear_output"
-      ],
+      ]
-      "metadata": {
-        "id": "OLComQyvCIJ7"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4PM8bf6lO0VE"
+      },
+      "outputs": [],
       "source": [
         "#Create pretty colormap as in book\n",
         "my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
@@ -68,28 +59,28 @@
         "b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
         "my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
         "my_colormap = ListedColormap(my_colormap_vals)"
-      ],
+      ]
-      "metadata": {
-        "id": "4PM8bf6lO0VE"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ONGRaQscfIOo"
+      },
+      "outputs": [],
       "source": [
         "# Probability distribution for normal\n",
         "def norm_pdf(x, mu, sigma):\n",
         "    return np.exp(-0.5 * (x-mu) * (x-mu) / (sigma * sigma)) / np.sqrt(2*np.pi*sigma*sigma)"
-      ],
+      ]
-      "metadata": {
-        "id": "ONGRaQscfIOo"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gZvG0MKhfY8Y"
+      },
+      "outputs": [],
       "source": [
         "# True distribution is a mixture of four Gaussians\n",
         "class TrueDataDistribution:\n",
@@ -110,15 +101,15 @@
         "    mu_list = list(itemgetter(*hidden)(self.mu))\n",
         "    sigma_list = list(itemgetter(*hidden)(self.sigma))\n",
         "    return mu_list + sigma_list * epsilon"
-      ],
+      ]
-      "metadata": {
-        "id": "gZvG0MKhfY8Y"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "iJu_uBiaeUVv"
+      },
+      "outputs": [],
       "source": [
         "# Define ground truth probability distribution that we will model\n",
         "true_dist = TrueDataDistribution()\n",
@@ -133,25 +124,26 @@
         "ax.set_ylim(0,1.0)\n",
         "ax.set_xlim(-3,3)\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "iJu_uBiaeUVv"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "DRHUG_41i4t_"
+      },
       "source": [
         "To train the model to describe this distribution, we'll need to generate pairs of samples drawn from $Pr(z_t|x)$ (diffusion kernel) and $q(z_{t-1}|z_{t},x)$ (equation 18.15).\n",
         "\n"
-      ],
+      ]
-      "metadata": {
-        "id": "DRHUG_41i4t_"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "x6B8t72Ukscd"
+      },
+      "outputs": [],
       "source": [
         "# The diffusion kernel returns the parameters of Pr(z_{t}|x)\n",
         "def diffusion_kernel(x, t, beta):\n",
@@ -180,24 +172,25 @@
         "      z_tminus1 = np.random.normal(size=x_train.shape) * cd_std + cd_mean\n",
         "\n",
         "    return z_t, z_tminus1"
-      ],
+      ]
-      "metadata": {
-        "id": "x6B8t72Ukscd"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "We also need models $\\mbox{f}_t[z_{t},\\phi_{t}]$ that map from $z_{t}$ to the mean of the distribution at time $z_{t-1}$.  We're just going to use a very hacky non-parametric model (basically a lookup table) that tells you the result based on the (quantized) input."
-      ],
       "metadata": {
         "id": "aSG_4uA8_zZ-"
-      }
+      },
+      "source": [
+        "We also need models $\\text{f}_t[z_{t},\\phi_{t}]$ that map from $z_{t}$ to the mean of the distribution at time $z_{t-1}$.  We're just going to use a very hacky non-parametric model (basically a lookup table) that tells you the result based on the (quantized) input."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZHViC0pL_yy5"
+      },
+      "outputs": [],
       "source": [
         "# This code is really ugly!  Don't look too closely at it!\n",
         "# All you need to know is that it is a model that trains from pairs zt, zt_minus1\n",
@@ -223,15 +216,15 @@
         "      bin_index = np.floor((zt+self.max_val)/self.inc)\n",
         "      bin_index = np.clip(bin_index,0, len(self.model)-1).astype('uint32')\n",
         "      return zt + self.model[bin_index]"
-      ],
+      ]
-      "metadata": {
-        "id": "ZHViC0pL_yy5"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "CzVFybWoBygu"
+      },
+      "outputs": [],
       "source": [
         "# Sample data from distribution (this would usually be our collected training set)\n",
         "n_sample = 100000\n",
@@ -249,24 +242,25 @@
         "    all_models.append(NonParametricModel())\n",
         "    # The model at index t maps data from z_{t+1} to z_{t}\n",
         "    all_models[t].train(zt,zt_minus1)"
-      ],
+      ]
-      "metadata": {
-        "id": "CzVFybWoBygu"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now that we've learned the model, let's draw some samples from it.  We start at $z_{100}$ and use the model to predict $z_{99}$, then $z_{98}$ and so on until finally we get to $z_{1}$ and then $x$ (represented as $z_{0}$ here). We'll store all of the intermediate stages as well, so we can plot the trajectories.  See equations 18.16."
-      ],
       "metadata": {
         "id": "ZPc9SEvtl14U"
-      }
+      },
+      "source": [
+        "Now that we've learned the model, let's draw some samples from it.  We start at $z_{100}$ and use the model to predict $z_{99}$, then $z_{98}$ and so on until finally we get to $z_{1}$ and then $x$ (represented as $z_{0}$ here). We'll store all of the intermediate stages as well, so we can plot the trajectories.  See equations 18.16."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "A-ZMFOvACIOw"
+      },
+      "outputs": [],
       "source": [
         "def sample(model, T, sigma_t, n_samples):\n",
         "    # Create the output array\n",
@@ -295,24 +289,25 @@
         "            samples[t-1,:] = samples[t-1,:]\n",
         "\n",
         "    return samples"
-      ],
+      ]
-      "metadata": {
-        "id": "A-ZMFOvACIOw"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's run the diffusion process for a whole bunch of samples"
-      ],
       "metadata": {
         "id": "ECAUfHNi9NVW"
-      }
+      },
+      "source": [
+        "Now let's run the diffusion process for a whole bunch of samples"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "M-TY5w9Q8LYW"
+      },
+      "outputs": [],
       "source": [
         "sigma_t=0.12288\n",
         "n_samples = 100000\n",
@@ -329,24 +324,25 @@
         "plt.hist(sampled_data, bins=bins, density =True)\n",
         "ax.set_ylim(0, 0.8)\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "M-TY5w9Q8LYW"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Let's, plot the evolution of a few of the paths as in figure 18.7 (paths are from bottom to top now)."
-      ],
       "metadata": {
         "id": "jYrAW6tN-gJ4"
-      }
+      },
+      "source": [
+        "Let's, plot the evolution of a few of the paths as in figure 18.7 (paths are from bottom to top now)."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4XU6CDZC_kFo"
+      },
+      "outputs": [],
       "source": [
         "fig, ax = plt.subplots()\n",
         "t_vals = np.arange(0,101,1)\n",
@@ -360,21 +356,33 @@
         "ax.set_xlabel('value')\n",
         "ax.set_ylabel('z_{t}')\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "4XU6CDZC_kFo"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Notice that the samples have a tendency to move from positions that are near the center at time 100 to positions that are high in the true probability distribution at time 0"
-      ],
       "metadata": {
         "id": "SGTYGGevAktz"
-      }
+      },
-    }
+      "source": [
+        "Notice that the samples have a tendency to move from positions that are near the center at time 100 to positions that are high in the true probability distribution at time 0"
       ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "authorship_tag": "ABX9TyM4DdZDGoP1xGst+Nn+rwvt",
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap18/18_3_Reparameterized_Model.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyNd+D0/IVWXtU2GKsofyk2d",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_3_Reparameterized_Model.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      },
       "source": [
         "# **Notebook 18.3: 1D Reparameterized model**\n",
         "\n",
@@ -36,13 +25,15 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "t9vk9Elugvmi"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "outputs": [],
       "source": [
         "import numpy as np\n",
         "import matplotlib.pyplot as plt\n",
@@ -50,15 +41,15 @@
         "from operator import itemgetter\n",
         "from scipy import stats\n",
         "from IPython.display import display, clear_output"
-      ],
+      ]
-      "metadata": {
-        "id": "OLComQyvCIJ7"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4PM8bf6lO0VE"
+      },
+      "outputs": [],
       "source": [
         "#Create pretty colormap as in book\n",
         "my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
@@ -68,28 +59,28 @@
         "b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
         "my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
         "my_colormap = ListedColormap(my_colormap_vals)"
-      ],
+      ]
-      "metadata": {
-        "id": "4PM8bf6lO0VE"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ONGRaQscfIOo"
+      },
+      "outputs": [],
       "source": [
         "# Probability distribution for normal\n",
         "def norm_pdf(x, mu, sigma):\n",
         "    return np.exp(-0.5 * (x-mu) * (x-mu) / (sigma * sigma)) / np.sqrt(2*np.pi*sigma*sigma)"
-      ],
+      ]
-      "metadata": {
-        "id": "ONGRaQscfIOo"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gZvG0MKhfY8Y"
+      },
+      "outputs": [],
       "source": [
         "# True distribution is a mixture of four Gaussians\n",
         "class TrueDataDistribution:\n",
@@ -110,15 +101,15 @@
         "    mu_list = list(itemgetter(*hidden)(self.mu))\n",
         "    sigma_list = list(itemgetter(*hidden)(self.sigma))\n",
         "    return mu_list + sigma_list * epsilon"
-      ],
+      ]
-      "metadata": {
-        "id": "gZvG0MKhfY8Y"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "iJu_uBiaeUVv"
+      },
+      "outputs": [],
       "source": [
         "# Define ground truth probability distribution that we will model\n",
         "true_dist = TrueDataDistribution()\n",
@@ -133,25 +124,26 @@
         "ax.set_ylim(0,1.0)\n",
         "ax.set_xlim(-3,3)\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "iJu_uBiaeUVv"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "DRHUG_41i4t_"
+      },
       "source": [
         "To train the model to describe this distribution, we'll need to generate pairs of samples drawn from $Pr(z_t|x)$ (diffusion kernel) and $q(z_{t-1}|z_{t},x)$ (equation 18.15).\n",
         "\n"
-      ],
+      ]
-      "metadata": {
-        "id": "DRHUG_41i4t_"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "x6B8t72Ukscd"
+      },
+      "outputs": [],
       "source": [
         "# Return z_t (the argument of g_{t}[] in the loss function in algorithm 18.1) and epsilon\n",
         "def get_data_pairs(x_train,t,beta):\n",
@@ -161,24 +153,25 @@
         "    z_t = np.ones_like(x_train)\n",
         "\n",
         "    return z_t, epsilon"
-      ],
+      ]
-      "metadata": {
-        "id": "x6B8t72Ukscd"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "We also need models $\\mbox{g}_t[z_{t},\\phi_{t}]$ that map from $z_{t}$ to the noise $\\epsilon$ that was added.  We're just going to use a very hacky non-parametric model (basically a lookup table) that tells you the result based on the (quantized) input."
-      ],
       "metadata": {
         "id": "aSG_4uA8_zZ-"
-      }
+      },
+      "source": [
+        "We also need models $\\text{g}_t[z_{t},\\phi_{t}]$ that map from $z_{t}$ to the noise $\\epsilon$ that was added.  We're just going to use a very hacky non-parametric model (basically a lookup table) that tells you the result based on the (quantized) input."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZHViC0pL_yy5"
+      },
+      "outputs": [],
       "source": [
         "# This code is really ugly!  Don't look too closely at it!\n",
         "# All you need to know is that it is a model that trains from pairs zt, zt_minus1\n",
@@ -204,15 +197,15 @@
         "      bin_index = np.floor((zt+self.max_val)/self.inc)\n",
         "      bin_index = np.clip(bin_index,0, len(self.model)-1).astype('uint32')\n",
         "      return self.model[bin_index]"
-      ],
+      ]
-      "metadata": {
-        "id": "ZHViC0pL_yy5"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "CzVFybWoBygu"
+      },
+      "outputs": [],
       "source": [
         "# Sample data from distribution (this would usually be our collected training set)\n",
         "n_sample = 100000\n",
@@ -230,24 +223,25 @@
         "    all_models.append(NonParametricModel())\n",
         "    # The model at index t maps data from z_{t+1} to epsilon\n",
         "    all_models[t].train(zt,epsilon)"
-      ],
+      ]
-      "metadata": {
-        "id": "CzVFybWoBygu"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now that we've learned the model, let's draw some samples from it.  We start at $z_{100}$ and use the model to predict $z_{99}$, then $z_{98}$ and so on until finally we get to $z_{1}$ and then $x$ (represented as $z_{0}$ here). We'll store all of the intermediate stages as well, so we can plot the trajectories.  See algorithm 18.2"
-      ],
       "metadata": {
         "id": "ZPc9SEvtl14U"
-      }
+      },
+      "source": [
+        "Now that we've learned the model, let's draw some samples from it.  We start at $z_{100}$ and use the model to predict $z_{99}$, then $z_{98}$ and so on until finally we get to $z_{1}$ and then $x$ (represented as $z_{0}$ here). We'll store all of the intermediate stages as well, so we can plot the trajectories.  See algorithm 18.2"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "A-ZMFOvACIOw"
+      },
+      "outputs": [],
       "source": [
         "def sample(model, T, sigma_t, n_samples):\n",
         "    # Create the output array\n",
@@ -277,24 +271,25 @@
         "            samples[t-1,:] = samples[t-1,:]\n",
         "\n",
         "    return samples"
-      ],
+      ]
-      "metadata": {
-        "id": "A-ZMFOvACIOw"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's run the diffusion process for a whole bunch of samples"
-      ],
       "metadata": {
         "id": "ECAUfHNi9NVW"
-      }
+      },
+      "source": [
+        "Now let's run the diffusion process for a whole bunch of samples"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "M-TY5w9Q8LYW"
+      },
+      "outputs": [],
       "source": [
         "sigma_t=0.12288\n",
         "n_samples = 100000\n",
@@ -311,24 +306,25 @@
         "plt.hist(sampled_data, bins=bins, density =True)\n",
         "ax.set_ylim(0, 0.8)\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "M-TY5w9Q8LYW"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Let's, plot the evolution of a few of the paths as in figure 18.7 (paths are from bottom to top now)."
-      ],
       "metadata": {
         "id": "jYrAW6tN-gJ4"
-      }
+      },
+      "source": [
+        "Let's, plot the evolution of a few of the paths as in figure 18.7 (paths are from bottom to top now)."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4XU6CDZC_kFo"
+      },
+      "outputs": [],
       "source": [
         "fig, ax = plt.subplots()\n",
         "t_vals = np.arange(0,101,1)\n",
@@ -342,21 +338,33 @@
         "ax.set_xlabel('value')\n",
         "ax.set_ylabel('z_{t}')\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "4XU6CDZC_kFo"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Notice that the samples have a tendency to move from positions that are near the center at time 100 to positions that are high in the true probability distribution at time 0"
-      ],
       "metadata": {
         "id": "SGTYGGevAktz"
-      }
+      },
-    }
+      "source": [
+        "Notice that the samples have a tendency to move from positions that are near the center at time 100 to positions that are high in the true probability distribution at time 0"
       ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "authorship_tag": "ABX9TyNd+D0/IVWXtU2GKsofyk2d",
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap18/18_4_Families_of_Diffusion_Models.ipynb

@@ -1,33 +1,22 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyNFSvISBXo/Z1l+onknF2Gw",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_4_Families_of_Diffusion_Models.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      },
       "source": [
         "# **Notebook 18.4: Families of diffusion models**\n",
         "\n",
@@ -36,13 +25,15 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "t9vk9Elugvmi"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "OLComQyvCIJ7"
+      },
+      "outputs": [],
       "source": [
         "import numpy as np\n",
         "import matplotlib.pyplot as plt\n",
@@ -50,15 +41,15 @@
         "from operator import itemgetter\n",
         "from scipy import stats\n",
         "from IPython.display import display, clear_output"
-      ],
+      ]
-      "metadata": {
-        "id": "OLComQyvCIJ7"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4PM8bf6lO0VE"
+      },
+      "outputs": [],
       "source": [
         "#Create pretty colormap as in book\n",
         "my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
@@ -68,28 +59,28 @@
         "b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
         "my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
         "my_colormap = ListedColormap(my_colormap_vals)"
-      ],
+      ]
-      "metadata": {
-        "id": "4PM8bf6lO0VE"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ONGRaQscfIOo"
+      },
+      "outputs": [],
       "source": [
         "# Probability distribution for normal\n",
         "def norm_pdf(x, mu, sigma):\n",
         "    return np.exp(-0.5 * (x-mu) * (x-mu) / (sigma * sigma)) / np.sqrt(2*np.pi*sigma*sigma)"
-      ],
+      ]
-      "metadata": {
-        "id": "ONGRaQscfIOo"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "gZvG0MKhfY8Y"
+      },
+      "outputs": [],
       "source": [
         "# True distribution is a mixture of four Gaussians\n",
         "class TrueDataDistribution:\n",
@@ -110,15 +101,15 @@
         "    mu_list = list(itemgetter(*hidden)(self.mu))\n",
         "    sigma_list = list(itemgetter(*hidden)(self.sigma))\n",
         "    return mu_list + sigma_list * epsilon"
-      ],
+      ]
-      "metadata": {
-        "id": "gZvG0MKhfY8Y"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "iJu_uBiaeUVv"
+      },
+      "outputs": [],
       "source": [
         "# Define ground truth probability distribution that we will model\n",
         "true_dist = TrueDataDistribution()\n",
@@ -133,25 +124,26 @@
         "ax.set_ylim(0,1.0)\n",
         "ax.set_xlim(-3,3)\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "iJu_uBiaeUVv"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "DRHUG_41i4t_"
+      },
       "source": [
         "To train the model to describe this distribution, we'll need to generate pairs of samples drawn from $Pr(z_t|x)$ (diffusion kernel) and $q(z_{t-1}|z_{t},x)$ (equation 18.15).\n",
         "\n"
-      ],
+      ]
-      "metadata": {
-        "id": "DRHUG_41i4t_"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "x6B8t72Ukscd"
+      },
+      "outputs": [],
       "source": [
         "# Return z_t (the argument of g_{t}[] in the loss function in algorithm 18.1) and epsilon\n",
         "def get_data_pairs(x_train,t,beta):\n",
@@ -161,24 +153,25 @@
         "    z_t = x_train * np.sqrt(alpha_t) + np.sqrt(1-alpha_t) * epsilon\n",
         "\n",
         "    return z_t, epsilon"
-      ],
+      ]
-      "metadata": {
-        "id": "x6B8t72Ukscd"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "We also need models $\\mbox{g}_t[z_{t},\\phi_{t}]$ that map from $z_{t}$ to the noise $\\epsilon$ that was added.  We're just going to use a very hacky non-parametric model (basically a lookup table) that tells you the result based on the (quantized) input."
-      ],
       "metadata": {
         "id": "aSG_4uA8_zZ-"
-      }
+      },
+      "source": [
+        "We also need models $\\text{g}_t[z_{t},\\phi_{t}]$ that map from $z_{t}$ to the noise $\\epsilon$ that was added.  We're just going to use a very hacky non-parametric model (basically a lookup table) that tells you the result based on the (quantized) input."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ZHViC0pL_yy5"
+      },
+      "outputs": [],
       "source": [
         "# This code is really ugly!  Don't look too closely at it!\n",
         "# All you need to know is that it is a model that trains from pairs zt, zt_minus1\n",
@@ -204,15 +197,15 @@
         "      bin_index = np.floor((zt+self.max_val)/self.inc)\n",
         "      bin_index = np.clip(bin_index,0, len(self.model)-1).astype('uint32')\n",
         "      return self.model[bin_index]"
-      ],
+      ]
-      "metadata": {
-        "id": "ZHViC0pL_yy5"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "CzVFybWoBygu"
+      },
+      "outputs": [],
       "source": [
         "# Sample data from distribution (this would usually be our collected training set)\n",
         "n_sample = 100000\n",
@@ -230,15 +223,14 @@
         "    all_models.append(NonParametricModel())\n",
         "    # The model at index t maps data from z_{t+1} to epsilon\n",
         "    all_models[t].train(zt,epsilon)"
-      ],
+      ]
-      "metadata": {
-        "id": "CzVFybWoBygu"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "ZPc9SEvtl14U"
+      },
       "source": [
         "Now that we've learned the model, let's draw some samples from it.  We start at $z_{100}$ and use the model to predict $z_{99}$, then $z_{98}$ and so on until finally we get to $z_{1}$ and then $x$ (represented as $z_{0}$ here). We'll store all of the intermediate stages as well, so we can plot the trajectories.\n",
         "\n",
@@ -247,17 +239,19 @@
         "One such model is the denoising diffusion implicit model, which has a sampling step:\n",
         "\n",
         "\\begin{equation}\n",
-        "\\mathbf{z}_{t-1} = \\sqrt{\\alpha_{t-1}}\\left(\\frac{\\mathbf{z}_{t}-\\sqrt{1-\\alpha_{t}}\\mbox{g}_t[\\mathbf{z}_{t},\\boldsymbol\\phi]}{\\sqrt{\\alpha_{t}}}\\right) + \\sqrt{1-\\alpha_{t-1}-\\sigma^2}\\mbox{g}_t[\\mathbf{z}_{t},\\boldsymbol\\phi]+\\sigma\\epsilon\n",
+        "\\mathbf{z}_{t-1} = \\sqrt{\\alpha_{t-1}}\\left(\\frac{\\mathbf{z}_{t}-\\sqrt{1-\\alpha_{t}}\\text{g}_t[\\mathbf{z}_{t},\\boldsymbol\\phi]}{\\sqrt{\\alpha_{t}}}\\right) + \\sqrt{1-\\alpha_{t-1}-\\sigma^2}\\text{g}_t[\\mathbf{z}_{t},\\boldsymbol\\phi]+\\sigma\\epsilon\n",
         "\\end{equation}\n",
         "\n",
         "(see equation 12 of the denoising [diffusion implicit models paper ](https://arxiv.org/pdf/2010.02502.pdf).\n"
-      ],
+      ]
-      "metadata": {
-        "id": "ZPc9SEvtl14U"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "A-ZMFOvACIOw"
+      },
+      "outputs": [],
       "source": [
         "def sample_ddim(model, T, sigma_t, n_samples):\n",
         "    # Create the output array\n",
@@ -283,24 +277,25 @@
         "        if t>0:\n",
         "            samples[t-1,:] = samples[t-1,:]+ np.random.standard_normal(n_samples) * sigma_t\n",
         "    return samples"
-      ],
+      ]
-      "metadata": {
-        "id": "A-ZMFOvACIOw"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's run the diffusion process for a whole bunch of samples"
-      ],
       "metadata": {
         "id": "ECAUfHNi9NVW"
-      }
+      },
+      "source": [
+        "Now let's run the diffusion process for a whole bunch of samples"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "M-TY5w9Q8LYW"
+      },
+      "outputs": [],
       "source": [
         "# Now we'll set the noise to a MUCH smaller level\n",
         "sigma_t=0.001\n",
@@ -318,24 +313,25 @@
         "plt.hist(sampled_data, bins=bins, density =True)\n",
         "ax.set_ylim(0, 0.8)\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "M-TY5w9Q8LYW"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Let's, plot the evolution of a few of the paths as in figure 18.7 (paths are from bottom to top now)."
-      ],
       "metadata": {
         "id": "jYrAW6tN-gJ4"
-      }
+      },
+      "source": [
+        "Let's, plot the evolution of a few of the paths as in figure 18.7 (paths are from bottom to top now)."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4XU6CDZC_kFo"
+      },
+      "outputs": [],
       "source": [
         "fig, ax = plt.subplots()\n",
         "t_vals = np.arange(0,101,1)\n",
@@ -349,35 +345,37 @@
         "ax.set_xlabel('value')\n",
         "ax.set_ylabel('z_{t}')\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "4XU6CDZC_kFo"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "The samples have a tendency to move from positions that are near the center at time 100 to positions that are high in the true probability distribution at time 0"
-      ],
       "metadata": {
         "id": "SGTYGGevAktz"
-      }
+      },
+      "source": [
+        "The samples have a tendency to move from positions that are near the center at time 100 to positions that are high in the true probability distribution at time 0"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "Z-LZp_fMXxRt"
+      },
       "source": [
         "Let's now sample from the accelerated model, that requires fewer models.  Again, we don't need to learn anything new -- this is just the reverse process that corresponds to a different forward process that is compatible with the same diffusion kernel.\n",
         "\n",
         "There's nothing to do here except read the code.  It uses the same DDIM model as you just implemented in the previous step, but it jumps timesteps five at a time."
-      ],
+      ]
-      "metadata": {
-        "id": "Z-LZp_fMXxRt"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "3Z0erjGbYj1u"
+      },
+      "outputs": [],
       "source": [
         "def sample_accelerated(model, T, sigma_t, n_steps, n_samples):\n",
         "    # Create the output array\n",
@@ -403,24 +401,25 @@
         "      if t>0:\n",
         "            samples[c_step-1,:] = samples[c_step-1,:]+ np.random.standard_normal(n_samples) * sigma_t\n",
         "    return samples"
-      ],
+      ]
-      "metadata": {
-        "id": "3Z0erjGbYj1u"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's draw a bunch of samples from the model"
-      ],
       "metadata": {
         "id": "D3Sm_WYrcuED"
-      }
+      },
+      "source": [
+        "Now let's draw a bunch of samples from the model"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "UB45c7VMcGy-"
+      },
+      "outputs": [],
       "source": [
         "sigma_t=0.11\n",
         "n_samples = 100000\n",
@@ -438,15 +437,15 @@
         "plt.hist(sampled_data, bins=bins, density =True)\n",
         "ax.set_ylim(0, 0.9)\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "UB45c7VMcGy-"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Luv-6w84c_qO"
+      },
+      "outputs": [],
       "source": [
         "fig, ax = plt.subplots()\n",
         "step_increment = 100/ n_steps\n",
@@ -464,21 +463,32 @@
         "ax.set_xlabel('value')\n",
         "ax.set_ylabel('z_{t}')\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "Luv-6w84c_qO"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [],
+      "execution_count": null,
       "metadata": {
         "id": "LSJi72f0kw_e"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": []
     }
-  ]
+  ],
+  "metadata": {
+    "colab": {
+      "authorship_tag": "ABX9TyNFSvISBXo/Z1l+onknF2Gw",
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap19/19_1_Markov_Decision_Processes.ipynb

@@ -598,7 +598,7 @@
       "source": [
         "def markov_decision_process_step_deterministic(state, transition_probabilities_given_action, reward_structure, policy):\n",
         "  # TODO -- complete this function.\n",
-        "  # For each state, theres is a corresponding action.\n",
+        "  # For each state, there's is a corresponding action.\n",
         "  # Draw the next state based on the current state and that action\n",
         "  # and calculate the reward\n",
         "  # Replace this line:\n",
@@ -683,7 +683,7 @@
       "source": [
         "def markov_decision_process_step_stochastic(state, transition_probabilities_given_action, reward_structure, stochastic_policy):\n",
         "  # TODO -- complete this function.\n",
-        "  # For each state, theres is a corresponding distribution over actions\n",
+        "  # For each state, there's is a corresponding distribution over actions\n",
         "  # Draw a sample from that distribution to get the action\n",
         "  # Draw the next state based on the current state and that action\n",
         "  # and calculate the reward\n",

Notebooks/Chap21/21_1_Bias_Mitigation.ipynb

@@ -1,77 +1,69 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "authorship_tag": "ABX9TyNQPfTDV6PFG7Ctcl+XVNlz",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    }
-  },
   "cells": [
     {
+      "attachments": {},
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap21/21_1_Bias_Mitigation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "t9vk9Elugvmi"
+      },
       "source": [
         "# **Notebook 21.1: Bias mitigation**\n",
         "\n",
-        "This notebook investigates a post-processing method for bias mitigation (see figure 21.2 in the book). It based on this [blog](https://www.borealisai.com/research-blogs/tutorial1-bias-and-fairness-ai/) that I wrote for Borealis AI in 2019, which itself was derirved from [this blog](https://research.google.com/bigpicture/attacking-discrimination-in-ml/) by Wattenberg, Viégas, and Hardt.\n",
+        "This notebook investigates a post-processing method for bias mitigation (see figure 21.2 in the book). It based on this [blog](https://www.borealisai.com/research-blogs/tutorial1-bias-and-fairness-ai/) that I wrote for Borealis AI in 2019, which itself was derived from [this blog](https://research.google.com/bigpicture/attacking-discrimination-in-ml/) by Wattenberg, Viégas, and Hardt.\n",
         "\n",
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n"
-      ],
+      ]
-      "metadata": {
-        "id": "t9vk9Elugvmi"
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "import numpy as np\n",
-        "import matplotlib.pyplot as plt"
-      ],
       "metadata": {
         "id": "yC_LpiJqZXEL"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "2FYo1dWGZXgg"
+      },
       "source": [
         "# Worked example: loans\n",
         "\n",
-        "Consider the example of an algorithm $c=\\mbox{f}[\\mathbf{x},\\boldsymbol\\phi]$ that predicts credit rating scores $c$ for loan decisions.  There are two pools of loan applicants identified by the variable $p\\in\\{0,1\\}$ that we’ll describe as the blue and yellow populations. We assume that we are given historical data, so we know both the credit rating and whether the applicant actually defaulted on the loan ($y=0$) or\n",
+        "Consider the example of an algorithm $c=\\text{f}[\\mathbf{x},\\boldsymbol\\phi]$ that predicts credit rating scores $c$ for loan decisions.  There are two pools of loan applicants identified by the variable $p\\in\\{0,1\\}$ that we’ll describe as the blue and yellow populations. We assume that we are given historical data, so we know both the credit rating and whether the applicant actually defaulted on the loan ($y=0$) or\n",
         " repaid it ($y=1$).\n",
         "\n",
         "We can now think of four groups of data corresponding to (i) the blue and yellow populations and (ii) whether they did or did not repay the loan. For each of these four groups we have a distribution of credit ratings (figure 1). In an ideal world, the two distributions for the yellow population would be exactly the same as those for the blue population. However, as figure 1 shows, this is clearly not the case here."
-      ],
+      ]
-      "metadata": {
-        "id": "2FYo1dWGZXgg"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "O_0gGH9hZcjo"
+      },
+      "outputs": [],
       "source": [
         "# Class that can describe interesting curve shapes based on the input parameters\n",
-        "# Details dont' matter\n",
+        "# Details don't matter\n",
         "class FreqCurve:\n",
         "  def __init__(self, weight, mean1, mean2, sigma1, sigma2, prop):\n",
         "    self.mean1 = mean1\n",
@@ -86,30 +78,30 @@
         "                * 1.0 / np.sqrt(2*np.pi*self.sigma1*self.sigma1) \\\n",
         "          + self.weight * (1-self.prop) * np.exp(-0.5 * (x-self.mean2) * (x-self.mean2) / (self.sigma2 * self.sigma2)) \\\n",
         "                * 1.0 / np.sqrt(2*np.pi*self.sigma2*self.sigma2)\n"
-      ],
+      ]
-      "metadata": {
-        "id": "O_0gGH9hZcjo"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Bkp7vffBbrNW"
+      },
+      "outputs": [],
       "source": [
         "credit_scores = np.arange(-4,4,0.01)\n",
         "freq_y0_p0 = FreqCurve(800, -1.5, -2.5, 0.8, 0.6, 0.6).freq(credit_scores)\n",
         "freq_y1_p0 = FreqCurve(500, 0.1, 0.7, 1.5, 0.8, 0.4 ).freq(credit_scores)\n",
         "freq_y0_p1 = FreqCurve(400, 0.2, -0.1, 0.8, 0.6, 0.3).freq(credit_scores)\n",
         "freq_y1_p1 = FreqCurve(650, 0.6, 1.6, 1.2, 0.7, 0.6 ).freq(credit_scores)\n"
-      ],
+      ]
-      "metadata": {
-        "id": "Bkp7vffBbrNW"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Jf7uqyRyhVdS"
+      },
+      "outputs": [],
       "source": [
         "\n",
         "fig = plt.figure\n",
@@ -136,15 +128,14 @@
         "ax.legend()\n",
         "\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "Jf7uqyRyhVdS"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "CfZ-srQtmff2"
+      },
       "source": [
         "Why might the distributions for blue and yellow populations be different? It could be that the behaviour of the populations is identical, but the credit rating algorithm is biased; it may favor one population over another or simply be more noisy for one group. Alternatively, it could be that that the populations genuinely behave differently. In practice, the differences in blue and yellow distributions are probably attributable to a combination of these factors.\n",
         "\n",
@@ -153,96 +144,102 @@
         " to go on, the best we can do is to assign different thresholds $\\tau_{1}$\n",
         " and $\\tau_{2}$\n",
         " for the blue and yellow populations so that the loan is granted if the credit score $c$ generated by the model exceeds $\\tau_0$ for the blue population and $\\tau_1$ for the yellow population."
-      ],
+      ]
-      "metadata": {
-        "id": "CfZ-srQtmff2"
-      }
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's investiate how to set these thresholds to fulfil different criteria."
-      ],
       "metadata": {
         "id": "569oU1OtoFz8"
-      }
+      },
+      "source": [
+        "Now let's investiate how to set these thresholds to fulfil different criteria."
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "bE7yPyuWoSUy"
+      },
       "source": [
         "# Blindness to protected attribute\n",
         "\n",
-        "We'll first do the simplest possible thing.  We'll choose the same threshold for both blue and yellow populations so that $\\tau_0$ = $\\tau_1$.  Basically, we'll ingore what we know about the group membership.  Let's see what the ramifications of that."
+        "We'll first do the simplest possible thing.  We'll choose the same threshold for both blue and yellow populations so that $\\tau_0$ = $\\tau_1$.  Basically, we'll ignore what we know about the group membership.  Let's see what the ramifications of that."
-      ],
+      ]
-      "metadata": {
-        "id": "bE7yPyuWoSUy"
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "# Set the thresholds\n",
-        "tau0 = tau1 = 0.0"
-      ],
       "metadata": {
         "id": "WIG8I-LvoFBY"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "# Set the thresholds\n",
+        "tau0 = tau1 = 0.0"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2EvkCvVBiCBn"
+      },
+      "outputs": [],
       "source": [
         "def compute_probability_get_loan(credit_scores, frequencies, threshold):\n",
         "  # TODO - Write this function\n",
-        "  # Return the probability that somemone from this group loan based on the frequencies of each\n",
+        "  # Return the probability that someone from this group loan based on the frequencies of each\n",
         "  # credit score for this group\n",
         "  # Replace this line:\n",
         "  prob = 0.5\n",
         "\n",
         "\n",
         "  return prob"
-      ],
+      ]
-      "metadata": {
-        "id": "2EvkCvVBiCBn"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "First let's see what the overall probability of getting the loan is for the yellow and blue populations."
-      ],
       "metadata": {
         "id": "AGT40q6_qfpv"
-      }
+      },
+      "source": [
+        "First let's see what the overall probability of getting the loan is for the yellow and blue populations."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4nI-PR_wqWj6"
+      },
+      "outputs": [],
       "source": [
         "pr_get_loan_p0 = compute_probability_get_loan(credit_scores, freq_y0_p0+freq_y1_p0, tau0)\n",
         "pr_get_loan_p1 = compute_probability_get_loan(credit_scores, freq_y0_p1+freq_y1_p1, tau1)\n",
         "print(\"Probability blue group gets loan = %3.3f\"%(pr_get_loan_p0))\n",
         "print(\"Probability yellow group gets loan = %3.3f\"%(pr_get_loan_p1))"
-      ],
+      ]
-      "metadata": {
-        "id": "4nI-PR_wqWj6"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "Now let's plot a receiver operating characteristic (ROC) curve.  This shows the rate of true positives $Pr(\\hat{y}=1|y=1)$ (people who got loan and paid it back) and false alarms $Pr(\\hat{y}=1|y=0)$ (people who got the loan but didn't pay it back) for all possible thresholds."
-      ],
       "metadata": {
         "id": "G2pEa6h6sIyu"
-      }
+      },
+      "source": [
+        "Now let's plot a receiver operating characteristic (ROC) curve.  This shows the rate of true positives $Pr(\\hat{y}=1|y=1)$ (people who got loan and paid it back) and false alarms $Pr(\\hat{y}=1|y=0)$ (people who got the loan but didn't pay it back) for all possible thresholds."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2C7kNt3hqwiu"
+      },
+      "outputs": [],
       "source": [
         "def plot_roc(credit_scores, freq_y0_p0, freq_y1_p0, freq_y0_p1, freq_y1_p1, tau0, tau1):\n",
         "  true_positives_p0 = np.zeros_like(credit_scores)\n",
@@ -272,61 +269,64 @@
         "  ax.set_aspect('equal')\n",
         "\n",
         "  plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "2C7kNt3hqwiu"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "plot_roc(credit_scores, freq_y0_p0, freq_y1_p0, freq_y0_p1, freq_y1_p1, tau0, tau1)"
-      ],
       "metadata": {
         "id": "h3OOQeTsv8uS"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "plot_roc(credit_scores, freq_y0_p0, freq_y1_p0, freq_y0_p1, freq_y1_p1, tau0, tau1)"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "UCObTsa57uuC"
+      },
       "source": [
         "On this plot, the true positive and false alarm rate for the particular thresholds ($\\tau_0=\\tau_{1}=0$) that we chose are indicated by the circles.\n",
         "\n",
         "This criterion is clearly not great.  The blue and yellow groups get given loans at different rates overall, and (for this threshold), the false alarms and true positives are also different, so it's not even fair when we consider whether the loans really were paid back.  \n",
         "\n",
-        "TODO -- investigate setting a different threshols $\\tau_{0}=\\tau_{1}$.  Is it possible to make the overall rates that loans are given the same?  Is it possible to make the false alarm rates the same?  Is it possible to make the true positive rates the same?"
+        "TODO -- investigate setting a different threshold $\\tau_{0}=\\tau_{1}$.  Is it possible to make the overall rates that loans are given the same?  Is it possible to make the false alarm rates the same?  Is it possible to make the true positive rates the same?"
-      ],
+      ]
-      "metadata": {
-        "id": "UCObTsa57uuC"
-      }
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "Yhrxv5AQ-PWA"
+      },
       "source": [
         "# Equality of odds\n",
         "\n",
         "This definition of fairness proposes that the false positive and true positive rates should be the same for both populations. This also sounds reasonable, but the ROC curve shows that it is not possible for this example. There is no combination of thresholds that can achieve this because the ROC curves do not intersect. Even if they did, we would be stuck giving loans based on the particular false positive and true positive rates at the intersection which might not be desirable."
-      ],
+      ]
-      "metadata": {
-        "id": "Yhrxv5AQ-PWA"
-      }
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "l6yb8vjX-gdi"
+      },
       "source": [
         "Demographic parity\n",
         "\n",
         "The thresholds can be chosen so that the same proportion of each group are classified as $\\hat{y}=1$ and given loans. We make an equal number of loans to each group despite the different tendencies of each to repay. This has the disadvantage that the true positive and false positive rates might be completely different in different populations. From the perspective of the lender, it is desirable to give loans in proportion to people’s ability to pay them back. From the perspective of an individual in a more reliable group, it may seem unfair that the other group gets offered the same number of loans despite the fact they are less reliable."
-      ],
+      ]
-      "metadata": {
-        "id": "l6yb8vjX-gdi"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "syjZ2fn5wC9-"
+      },
+      "outputs": [],
       "source": [
         "# TO DO -- try to change the two thresholds so the overall probability of getting the loan is 0.6 for each group\n",
         "# Change the values in these lines\n",
@@ -340,55 +340,58 @@
         "pr_get_loan_p1 = compute_probability_get_loan(credit_scores, freq_y0_p1+freq_y1_p1, tau1)\n",
         "print(\"Probability blue group gets loan = %3.3f\"%(pr_get_loan_p0))\n",
         "print(\"Probability yellow group gets loan = %3.3f\"%(pr_get_loan_p1))"
-      ],
+      ]
-      "metadata": {
-        "id": "syjZ2fn5wC9-"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "This is good, because now both groups get roughly the same amount of loans.  But hold on... let's look at the ROC curve:"
-      ],
       "metadata": {
         "id": "5QrtvZZlHCJy"
-      }
+      },
+      "source": [
+        "This is good, because now both groups get roughly the same amount of loans.  But hold on... let's look at the ROC curve:"
+      ]
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "plot_roc(credit_scores, freq_y0_p0, freq_y1_p0, freq_y0_p1, freq_y1_p1, tau0, tau1)"
-      ],
       "metadata": {
         "id": "VApyl_58GUQb"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "plot_roc(credit_scores, freq_y0_p0, freq_y1_p0, freq_y0_p1, freq_y1_p1, tau0, tau1)"
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "The blue dot is waaay above the yellow dot. The proportion of people who are given a load and do pay it back from the blue population is much higher than that from the yellow population.  From another perspective, that's unfair... it seems like the yellow population are 'allowed' to default more often than the blue. This leads to another possibility."
-      ],
       "metadata": {
         "id": "_GgX_b6yIE4W"
-      }
+      },
+      "source": [
+        "The blue dot is waaay above the yellow dot. The proportion of people who are given a load and do pay it back from the blue population is much higher than that from the yellow population.  From another perspective, that's unfair... it seems like the yellow population are 'allowed' to default more often than the blue. This leads to another possibility."
+      ]
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
+      "metadata": {
+        "id": "WDnaqetXHhlv"
+      },
       "source": [
         "# Equal opportunity:\n",
         "\n",
         "The thresholds are chosen so that so that the true positive rate is is the same for both population. Of the people who pay back the loan, the same proportion are offered credit in each group. In terms of the two ROC curves, it means choosing thresholds so that the vertical position on each curve is the same without regard for the horizontal position."
-      ],
+      ]
-      "metadata": {
-        "id": "WDnaqetXHhlv"
-      }
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "zEN6HGJ7HJAZ"
+      },
+      "outputs": [],
       "source": [
         "# TO DO -- try to change the two thresholds so the true positive are 0.8 for each group\n",
         "# Change the values in these lines so that both points on the curves have a height of 0.8\n",
@@ -397,45 +400,58 @@
         "\n",
         "\n",
         "plot_roc(credit_scores, freq_y0_p0, freq_y1_p0, freq_y0_p1, freq_y1_p1, tau0, tau1)"
-      ],
+      ]
-      "metadata": {
-        "id": "zEN6HGJ7HJAZ"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "This seems fair -- people who are given loans default at the same rate (20%) for both groups.  But hold on... let's look at the overall loan rate between the two populations:"
-      ],
       "metadata": {
         "id": "JsyW0pBGJ24b"
-      }
+      },
+      "source": [
+        "This seems fair -- people who are given loans default at the same rate (20%) for both groups.  But hold on... let's look at the overall loan rate between the two populations:"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2a5PXHeNJDjg"
+      },
+      "outputs": [],
       "source": [
         "# Compute overall probability of getting loan\n",
         "pr_get_loan_p0 = compute_probability_get_loan(credit_scores, freq_y0_p0+freq_y1_p0, tau0)\n",
         "pr_get_loan_p1 = compute_probability_get_loan(credit_scores, freq_y0_p1+freq_y1_p1, tau1)\n",
         "print(\"Probability blue group gets loan = %3.3f\"%(pr_get_loan_p0))\n",
         "print(\"Probability yellow group gets loan = %3.3f\"%(pr_get_loan_p1))"
-      ],
+      ]
-      "metadata": {
-        "id": "2a5PXHeNJDjg"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
+      "attachments": {},
       "cell_type": "markdown",
-      "source": [
-        "The conclusion from all this is that (i) definitions of fairness are quite subtle and (ii) it's not possible to satisfy them all simultaneously."
-      ],
       "metadata": {
         "id": "tZTM7N6jKC7q"
-      }
+      },
-    }
+      "source": [
+        "The conclusion from all this is that (i) definitions of fairness are quite subtle and (ii) it's not possible to satisfy them all simultaneously."
       ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "authorship_tag": "ABX9TyNQPfTDV6PFG7Ctcl+XVNlz",
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap21/21_2_Explainability.ipynb

@@ -400,7 +400,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "This model is easilly intepretable.  The k'th coeffeicient tells us the how much (and in which direction) changing the value of the k'th input will change the output.  This is only valid in the vicinity of the input $x$.\n",
+        "This model is easily interpretable.  The k'th coefficient tells us the how much (and in which direction) changing the value of the k'th input will change the output.  This is only valid in the vicinity of the input $x$.\n",
         "\n",
         "Note that a more sophisticated version of LIME would weight the training points according to how close they are to the original data point of interest."
       ],

index.html

@@ -11,15 +11,18 @@
     <div>
         <h1 style="margin: 0; font-size: 36px">Understanding Deep Learning</h1>
         by Simon J.D. Prince
-        <br>To be published by MIT Press Dec 5th 2023.<br>
+        <br>Published by MIT Press Dec 5th 2023.<br>
         <ul>
             <li>
                 <p style="font-size: larger; margin-bottom: 0">Download draft PDF Chapters 1-21 <a
-                        href="https://github.com/udlbook/udlbook/releases/download/v1.15/UnderstandingDeepLearning_23_10_23_C.pdf">here</a>
+                        href="https://github.com/udlbook/udlbook/releases/download/v1.19/UnderstandingDeepLearning_16_12_23_C.pdf">here</a>
-                </p>2023-23-23. CC-BY-NC-ND license<br>
+                </p>2024-01-16. CC-BY-NC-ND license<br>
                 <img src="https://img.shields.io/github/downloads/udlbook/udlbook/total" alt="download stats shield">
             </li>
-            <li> Report errata via <a href="https://github.com/udlbook/udlbook/issues">github</a>
+            <li> Order your copy from  <a href="https://mitpress.mit.edu/9780262048644/understanding-deep-learning/">here </a></li>
+            <li> Known errata can be found here:   <a
+            href="https://github.com/udlbook/udlbook/raw/main/UDL_Errata.pdf">PDF</a></li>
+            <li> Report new errata via <a href="https://github.com/udlbook/udlbook/issues">github</a>
                 or contact me directly at udlbookmail@gmail.com
             <li> Follow me on <a href="https://twitter.com/SimonPrinceAI">Twitter</a> or <a
                     href="https://www.linkedin.com/in/simon-prince-615bb9165/">LinkedIn</a> for updates.
@@ -169,7 +172,9 @@
                 Figures</a>
         <li> Chapter 18 - Diffusion models: <a
                 href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap18PDF.zip">PDF Figures</a> / <a
-                href="https://docs.google.com/presentation/d/1x_ufIBtVPzWUvRieKMkpw5SdRjXWwdfR/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">
+                href="https://drive.google.com/uc?export=download&id=1A-pIGl4PxjVMYOKAUG3aT4a8wD3G-q_r"> SVG Figures</a>
+            /
+            <a href="https://docs.google.com/presentation/d/1x_ufIBtVPzWUvRieKMkpw5SdRjXWwdfR/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">
             PowerPoint Figures</a>
         <li> Chapter 19 - Deep reinforcement learning: <a
                 href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap19PDF.zip">PDF Figures</a> / <a
@@ -200,6 +205,23 @@
     Instructions for editing figures / equations can be found <a
         href="https://drive.google.com/file/d/1T_MXXVR4AfyMnlEFI-UVDh--FXI5deAp/view?usp=sharing">here</a>.
 
+    <p> My slides for 20 lecture undergraduate deep learning course:</p>
+    <ul>
+        <li><a href="https://drive.google.com/uc?export=download&id=17RHb11BrydOvxSFNbRIomE1QKLVI087m">1. Introduction</a></li>
+        <li><a href="https://drive.google.com/uc?export=download&id=1491zkHULC7gDfqlV6cqUxyVYXZ-de-Ub">2. Supervised Learning</a></li>
+        <li><a href="https://drive.google.com/uc?export=download&id=1XkP1c9EhOBowla1rT1nnsDGMf2rZvrt7">3. Shallow Neural Networks</a></li>
+        <li><a href="https://drive.google.com/uc?export=download&id=1e2ejfZbbfMKLBv0v-tvBWBdI8gO3SSS1">4. Deep Neural Networks</a></li>
+        <li><a href="https://drive.google.com/uc?export=download&id=1fxQ_a1Q3eFPZ4kPqKbak6_emJK-JfnRH">5. Loss Functions</a></li>
+        <li><a href="https://drive.google.com/uc?export=download&id=17QQ5ZzXBtR_uCNCUU1gPRWWRUeZN9exW">6. Fitting Models</a></li>
+        <li><a href="https://drive.google.com/uc?export=download&id=1hC8JUCOaFWiw3KGn0rm7nW6mEq242QDK">7. Computing Gradients</a></li>
+        <li><a href="https://drive.google.com/uc?export=download&id=1tSjCeAVg0JCeBcPgDJDbi7Gg43Qkh9_d">7b. Initialization</a></li>
+        <li><a href="https://drive.google.com/uc?export=download&id=1RVZW3KjEs0vNSGx3B2fdizddlr6I0wLl">8. Performance</a></li>
+        <li><a href="https://drive.google.com/uc?export=download&id=1LTicIKPRPbZRkkg6qOr1DSuOB72axood">9. Regularization</a></li>
+        <li><a href="https://drive.google.com/uc?export=download&id=1bGVuwAwrofzZdfvj267elIzkYMIvYFj0">10. Convolutional Networks</a></li>
+        <li><a href="https://drive.google.com/uc?export=download&id=14w31QqWRDix1GdUE-na0_E0kGKBhtKzs">11. Image Generation</a></li>
+        <li><a href="https://drive.google.com/uc?export=download&id=1af6bTTjAbhDYfrDhboW7Fuv52Gk9ygKr">12. Transformers and LLMs</a></li>
+    </ul>
+
     <h2>Resources for students</h2>
 
     <p>Answers to selected questions: <a