Add files via upload

Notebook 9_5_Augmentation - Removed duplicate weight initialization

Blogs/BorealisGradientFlow.ipynb

@@ -31,7 +31,7 @@
       "source": [
         "# Gradient flow\n",
         "\n",
-        "This notebook replicates some of the results in the the Borealis AI [blog](https://www.borealisai.com/research-blogs/gradient-flow/) on gradient flow.  \n"
+        "This notebook replicates some of the results in the Borealis AI [blog](https://www.borealisai.com/research-blogs/gradient-flow/) on gradient flow.  \n"
       ],
       "metadata": {
         "id": "ucrRRJ4dq8_d"

Blogs/BorealisNTK.ipynb

@@ -166,7 +166,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Routines to calculate the empirical and analytical NTK (i.e. the NTK with infinite hidden units) for the the shallow network"
+        "Routines to calculate the empirical and analytical NTK (i.e. the NTK with infinite hidden units) for the shallow network"
       ],
       "metadata": {
         "id": "mxW8E5kYIzlj"

CM20315/CM20315_Coursework_IV.ipynb

@@ -128,7 +128,7 @@
         "\n",
         "In part (b) of the practical we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius.  You will find that the volume decreases to almost nothing in high dimensions.  All of the volume is in the corners of the unit hypercube (which always has volume 1). Double weird.\n",
         "\n",
-        "Note that you you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
+        "Note that you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
       ],
       "metadata": {
         "id": "b2FYKV1SL4Z7"

CM20315/CM20315_Loss_II.ipynb

@@ -199,7 +199,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "The left is model output and the right is the model output after the sigmoid has been applied, so it now lies in the range [0,1] and represents the probability, that y=1.  The black dots show the training data.  We'll compute the the likelihood and the negative log likelihood."
+        "The left is model output and the right is the model output after the sigmoid has been applied, so it now lies in the range [0,1] and represents the probability, that y=1.  The black dots show the training data.  We'll compute the likelihood and the negative log likelihood."
       ],
       "metadata": {
         "id": "MvVX6tl9AEXF"

CM20315/CM20315_Loss_III.ipynb

@@ -218,7 +218,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue)   The dots at the bottom show the training data with the same color scheme.  So we want the red curve to be high where there are red dots, the green curve to be high where there are green dotsmand the blue curve to be high where there are blue dots  We'll compute the the likelihood and the negative log likelihood."
+        "The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue)   The dots at the bottom show the training data with the same color scheme.  So we want the red curve to be high where there are red dots, the green curve to be high where there are green dotsmand the blue curve to be high where there are blue dots  We'll compute the likelihood and the negative log likelihood."
       ],
       "metadata": {
         "id": "MvVX6tl9AEXF"

CM20315_2023/CM20315_Coursework_IV.ipynb

@@ -128,7 +128,7 @@
         "\n",
         "In part (b) of the practical we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius.  You will find that the volume decreases to almost nothing in high dimensions.  All of the volume is in the corners of the unit hypercube (which always has volume 1). Double weird.\n",
         "\n",
-        "Note that you you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
+        "Note that you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
       ],
       "metadata": {
         "id": "b2FYKV1SL4Z7"

CM20315_2023/CM20315_Coursework_V_2023.ipynb

@@ -214,7 +214,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Compute the derivative of the the loss with respect to the function output f_val\n",
+        "# Compute the derivative of the loss with respect to the function output f_val\n",
         "def dl_df(f_val,y):\n",
         "  # Compute sigmoid of network output\n",
         "  sig_f_val = sig(f_val)\n",

Notebooks/Chap04/4_1_Composing_Networks.ipynb

@@ -28,7 +28,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "#Notebook 4.1 -- Composing networks\n",
+        "# Notebook 4.1 -- Composing networks\n",
         "\n",
         "The purpose of this notebook is to understand what happens when we feed one neural network into another. It works through an example similar to 4.1 and varies both networks\n",
         "\n",

Notebooks/Chap04/4_2_Clipping_functions.ipynb

@@ -29,7 +29,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "#Notebook 4.2 -- Clipping functions\n",
+        "# Notebook 4.2 -- Clipping functions\n",
         "\n",
         "The purpose of this notebook is to understand how a neural network with two hidden layers build more complicated functions by clipping and recombining the representations at the intermediate hidden variables.\n",
         "\n",

Notebooks/Chap05/5_3_Multiclass_Cross_entropy_Loss.ipynb

@@ -211,7 +211,7 @@
         "id": "MvVX6tl9AEXF"
       },
       "source": [
-        "The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue).  The dots at the bottom show the training data with the same color scheme.  So we want the red curve to be high where there are red dots, the green curve to be high where there are green dots, and the blue curve to be high where there are blue dots  We'll compute the the likelihood and the negative log likelihood."
+        "The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue).  The dots at the bottom show the training data with the same color scheme.  So we want the red curve to be high where there are red dots, the green curve to be high where there are green dots, and the blue curve to be high where there are blue dots  We'll compute the likelihood and the negative log likelihood."
       ]
     },
     {

Notebooks/Chap08/8_1_MNIST_1D_Performance.ipynb

@@ -1,28 +1,10 @@
 {
-  "nbformat": 4,
-  "nbformat_minor": 0,
-  "metadata": {
-    "colab": {
-      "provenance": [],
-      "gpuType": "T4",
-      "authorship_tag": "ABX9TyOuKMUcKfOIhIL2qTX9jJCy",
-      "include_colab_link": true
-    },
-    "kernelspec": {
-      "name": "python3",
-      "display_name": "Python 3"
-    },
-    "language_info": {
-      "name": "python"
-    },
-    "accelerator": "GPU"
-  },
   "cells": [
     {
       "cell_type": "markdown",
       "metadata": {
-        "id": "view-in-github",
+        "colab_type": "text",
-        "colab_type": "text"
+        "id": "view-in-github"
       },
       "source": [
         "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap08/8_1_MNIST_1D_Performance.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
@@ -30,6 +12,9 @@
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "L6chybAVFJW2"
+      },
       "source": [
         "# **Notebook 8.1: MNIST_1D_Performance**\n",
         "\n",
@@ -38,25 +23,27 @@
         "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
         "\n",
         "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
-      ],
+      ]
-      "metadata": {
-        "id": "L6chybAVFJW2"
-      }
     },
     {
       "cell_type": "code",
-      "source": [
+      "execution_count": null,
-        "# Run this if you're in a Colab to install MNIST 1D repository\n",
-        "%pip install git+https://github.com/greydanus/mnist1d"
-      ],
       "metadata": {
         "id": "ifVjS4cTOqKz"
       },
-      "execution_count": null,
+      "outputs": [],
-      "outputs": []
+      "source": [
+        "# Run this if you're in a Colab to install MNIST 1D repository\n",
+        "%pip install git+https://github.com/greydanus/mnist1d"
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "qyE7G1StPIqO"
+      },
+      "outputs": [],
       "source": [
         "import torch, torch.nn as nn\n",
         "from torch.utils.data import TensorDataset, DataLoader\n",
@@ -64,44 +51,42 @@
         "import numpy as np\n",
         "import matplotlib.pyplot as plt\n",
         "import mnist1d"
-      ],
+      ]
-      "metadata": {
-        "id": "qyE7G1StPIqO"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "Let's generate a training and test dataset using the MNIST1D code.  The dataset gets saved as a .pkl file so it doesn't have to be regenerated each time."
-      ],
       "metadata": {
         "id": "F7LNq72SP6jO"
-      }
+      },
+      "source": [
+        "Let's generate a training and test dataset using the MNIST1D code.  The dataset gets saved as a .pkl file so it doesn't have to be regenerated each time."
+      ]
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "YLxf7dJfPaqw"
+      },
+      "outputs": [],
       "source": [
-        "!mkdir ./sample_data\n",
-        "\n",
         "args = mnist1d.data.get_dataset_args()\n",
-        "data = mnist1d.data.get_dataset(args, path='./sample_data/mnist1d_data.pkl', download=False, regenerate=False)\n",
+        "data = mnist1d.data.get_dataset(args, path='./mnist1d_data.pkl', download=False, regenerate=False)\n",
         "\n",
         "# The training and test input and outputs are in\n",
         "# data['x'], data['y'], data['x_test'], and data['y_test']\n",
         "print(\"Examples in training set: {}\".format(len(data['y'])))\n",
         "print(\"Examples in test set: {}\".format(len(data['y_test'])))\n",
         "print(\"Length of each example: {}\".format(data['x'].shape[-1]))"
-      ],
+      ]
-      "metadata": {
-        "id": "YLxf7dJfPaqw"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "FxaB5vc0uevl"
+      },
+      "outputs": [],
       "source": [
         "D_i = 40    # Input dimensions\n",
         "D_k = 100   # Hidden dimensions\n",
@@ -122,15 +107,15 @@
         "\n",
         "# Call the function you just defined\n",
         "model.apply(weights_init)\n"
-      ],
+      ]
-      "metadata": {
-        "id": "FxaB5vc0uevl"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "_rX6N3VyyQTY"
+      },
+      "outputs": [],
       "source": [
         "# choose cross entropy loss function (equation 5.24)\n",
         "loss_function = torch.nn.CrossEntropyLoss()\n",
@@ -139,9 +124,9 @@
         "# object that decreases learning rate by half every 10 epochs\n",
         "scheduler = StepLR(optimizer, step_size=10, gamma=0.5)\n",
         "x_train = torch.tensor(data['x'].astype('float32'))\n",
-        "y_train = torch.tensor(data['y'].transpose().astype('long'))\n",
+        "y_train = torch.tensor(data['y'].transpose().astype('int64'))\n",
         "x_test= torch.tensor(data['x_test'].astype('float32'))\n",
-        "y_test = torch.tensor(data['y_test'].astype('long'))\n",
+        "y_test = torch.tensor(data['y_test'].astype('int64'))\n",
         "\n",
         "# load the data into a class that creates the batches\n",
         "data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=100, shuffle=True, worker_init_fn=np.random.seed(1))\n",
@@ -186,15 +171,15 @@
         "\n",
         "  # tell scheduler to consider updating learning rate\n",
         "  scheduler.step()"
-      ],
+      ]
-      "metadata": {
-        "id": "_rX6N3VyyQTY"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "yI-l6kA_EH9G"
+      },
+      "outputs": [],
       "source": [
         "# Plot the results\n",
         "fig, ax = plt.subplots()\n",
@@ -215,25 +200,38 @@
         "ax.set_title('Train loss %3.2f, Test loss %3.2f'%(losses_train[-1],losses_test[-1]))\n",
         "ax.legend()\n",
         "plt.show()"
-      ],
+      ]
-      "metadata": {
-        "id": "yI-l6kA_EH9G"
-      },
-      "execution_count": null,
-      "outputs": []
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "q-yT6re6GZS4"
+      },
       "source": [
         "**TO DO**\n",
         "\n",
         "Play with the model -- try changing the number of layers, hidden units, learning rate, batch size, momentum or anything else you like.  See if you can improve the test results.\n",
         "\n",
         "Is it a good idea to optimize the hyperparameters in this way?  Will the final result be a good estimate of the true test performance?"
-      ],
+      ]
-      "metadata": {
-        "id": "q-yT6re6GZS4"
-      }
     }
-  ]
+  ],
+  "metadata": {
+    "accelerator": "GPU",
+    "colab": {
+      "authorship_tag": "ABX9TyOuKMUcKfOIhIL2qTX9jJCy",
+      "gpuType": "T4",
+      "include_colab_link": true,
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
 }

Notebooks/Chap08/8_4_High_Dimensional_Spaces.ipynb

@@ -134,7 +134,7 @@
       "source": [
         "# Volume of a hypersphere\n",
         "\n",
-        "In the second part of this notebook we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius.  Note that you you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
+        "In the second part of this notebook we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius.  Note that you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
       ],
       "metadata": {
         "id": "b2FYKV1SL4Z7"

Notebooks/Chap09/9_5_Augmentation.ipynb

@@ -107,10 +107,7 @@
         "  # Initialize the parameters with He initialization\n",
         "  if isinstance(layer_in, nn.Linear):\n",
         "    nn.init.kaiming_uniform_(layer_in.weight)\n",
-        "    layer_in.bias.data.fill_(0.0)\n",
+        "    layer_in.bias.data.fill_(0.0)\n"
-        "\n",
-        "# Call the function you just defined\n",
-        "model.apply(weights_init)"
       ],
       "metadata": {
         "id": "JfIFWFIL33eF"

Notebooks/Chap19/19_2_Dynamic_Programming.ipynb

@@ -393,7 +393,7 @@
     {
       "cell_type": "code",
       "source": [
-        "# Update the state values for the current policy, by making the values at at adjacent\n",
+        "# Update the state values for the current policy, by making the values at adjacent\n",
         "# states compatible with the Bellman equation (equation 19.11)\n",
         "def policy_evaluation(policy, state_values, rewards,  transition_probabilities_given_action, gamma):\n",
         "\n",

Notebooks/Chap21/21_1_Bias_Mitigation.ipynb

@@ -137,7 +137,7 @@
         "id": "CfZ-srQtmff2"
       },
       "source": [
-        "Why might the distributions for blue and yellow populations be different? It could be that the behaviour of the populations is identical, but the credit rating algorithm is biased; it may favor one population over another or simply be more noisy for one group. Alternatively, it could be that that the populations genuinely behave differently. In practice, the differences in blue and yellow distributions are probably attributable to a combination of these factors.\n",
+        "Why might the distributions for blue and yellow populations be different? It could be that the behaviour of the populations is identical, but the credit rating algorithm is biased; it may favor one population over another or simply be more noisy for one group. Alternatively, it could be that the populations genuinely behave differently. In practice, the differences in blue and yellow distributions are probably attributable to a combination of these factors.\n",
         "\n",
         "Let’s assume that we can’t retrain the credit score prediction algorithm; our job is to adjudicate whether each individual is refused the loan ($\\hat{y}=0$)\n",
         " or granted it ($\\hat{y}=1$). Since we only have the credit score\n",
@@ -382,7 +382,7 @@
       "source": [
         "# Equal opportunity:\n",
         "\n",
-        "The thresholds are chosen so that so that the true positive rate is is the same for both population. Of the people who pay back the loan, the same proportion are offered credit in each group. In terms of the two ROC curves, it means choosing thresholds so that the vertical position on each curve is the same without regard for the horizontal position."
+        "The thresholds are chosen so that so that the true positive rate is the same for both population. Of the people who pay back the loan, the same proportion are offered credit in each group. In terms of the two ROC curves, it means choosing thresholds so that the vertical position on each curve is the same without regard for the horizontal position."
       ]
     },
     {

index.html

@@ -10,6 +10,7 @@
             href="https://fonts.googleapis.com/css2?family=Encode+Sans+Expanded:wght@400;700&display=swap"
             rel="stylesheet"
         />
+        
         <title>Understanding Deep Learning</title>
     </head>
     <body>