dr_domi/udlbook
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'main' of https://github.com/udlbook/udlbook

Browse Source 
Trying to fix website problems
This commit is contained in:
Simon Prince
2025-01-23 16:25:08 -05:00
parent 2f0339341c f8acbaab82
commit 8f2ef53eab
40 changed files with 11250 additions and 201 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
Blogs/BorealisGradientFlow.ipynb
View File

@@ -31,7 +31,7 @@
"source": [
"# Gradient flow\n",
"\n",
"This notebook replicates some of the results in the the Borealis AI [blog](https://www.borealisai.com/research-blogs/gradient-flow/) on gradient flow. \n"
"This notebook replicates some of the results in the Borealis AI [blog](https://www.borealisai.com/research-blogs/gradient-flow/) on gradient flow. \n"
],
"metadata": {
"id": "ucrRRJ4dq8_d"

2
Blogs/BorealisNTK.ipynb
View File

@@ -166,7 +166,7 @@
{
"cell_type": "markdown",
"source": [
"Routines to calculate the empirical and analytical NTK (i.e. the NTK with infinite hidden units) for the the shallow network"
"Routines to calculate the empirical and analytical NTK (i.e. the NTK with infinite hidden units) for the shallow network"
],
"metadata": {
"id": "mxW8E5kYIzlj"

2
CM20315/CM20315_Coursework_IV.ipynb
View File

@@ -128,7 +128,7 @@
"\n",
"In part (b) of the practical we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius. You will find that the volume decreases to almost nothing in high dimensions. All of the volume is in the corners of the unit hypercube (which always has volume 1). Double weird.\n",
"\n",
"Note that you you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
"Note that you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
],
"metadata": {
"id": "b2FYKV1SL4Z7"

2
CM20315/CM20315_Loss_II.ipynb
View File

@@ -199,7 +199,7 @@
{
"cell_type": "markdown",
"source": [
"The left is model output and the right is the model output after the sigmoid has been applied, so it now lies in the range [0,1] and represents the probability, that y=1. The black dots show the training data. We'll compute the the likelihood and the negative log likelihood."
"The left is model output and the right is the model output after the sigmoid has been applied, so it now lies in the range [0,1] and represents the probability, that y=1. The black dots show the training data. We'll compute the likelihood and the negative log likelihood."
],
"metadata": {
"id": "MvVX6tl9AEXF"

2
CM20315/CM20315_Loss_III.ipynb
View File

@@ -218,7 +218,7 @@
{
"cell_type": "markdown",
"source": [
"The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue) The dots at the bottom show the training data with the same color scheme. So we want the red curve to be high where there are red dots, the green curve to be high where there are green dotsmand the blue curve to be high where there are blue dots We'll compute the the likelihood and the negative log likelihood."
"The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue) The dots at the bottom show the training data with the same color scheme. So we want the red curve to be high where there are red dots, the green curve to be high where there are green dotsmand the blue curve to be high where there are blue dots We'll compute the likelihood and the negative log likelihood."
],
"metadata": {
"id": "MvVX6tl9AEXF"

2
CM20315_2023/CM20315_Coursework_IV.ipynb
View File

@@ -128,7 +128,7 @@
"\n",
"In part (b) of the practical we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius. You will find that the volume decreases to almost nothing in high dimensions. All of the volume is in the corners of the unit hypercube (which always has volume 1). Double weird.\n",
"\n",
"Note that you you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
"Note that you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
],
"metadata": {
"id": "b2FYKV1SL4Z7"

2
CM20315_2023/CM20315_Coursework_V_2023.ipynb
View File

@@ -214,7 +214,7 @@
{
"cell_type": "code",
"source": [
"# Compute the derivative of the the loss with respect to the function output f_val\n",
"# Compute the derivative of the loss with respect to the function output f_val\n",
"def dl_df(f_val,y):\n",
" # Compute sigmoid of network output\n",
" sig_f_val = sig(f_val)\n",

2
Notebooks/Chap03/3_1_Shallow_Networks_I.ipynb
View File

File diff suppressed because one or more lines are too long

5
Notebooks/Chap03/3_3_Shallow_Network_Regions.ipynb
View File

@@ -4,7 +4,6 @@
"metadata": {
"colab": {
"provenance": [],
"authorship_tag": "ABX9TyNioITtfAcfxEfM3UOfQyb9",
"include_colab_link": true
},
"kernelspec": {
@@ -62,7 +61,7 @@
"source": [
"The number of regions $N$ created by a shallow neural network with $D_i$ inputs and $D$ hidden units is given by Zaslavsky's formula:\n",
"\n",
"\\begin{equation}N = \\sum_{j=0}^{D_{i}}\\binom{D}{j}=\\sum_{j=0}^{D_{i}} \\frac{D!}{(D-j)!j!} \\end{equation} <br>\n",
"\\begin{equation}N = \\sum_{j=0}^{D_{i}}\\binom{D}{j}=\\sum_{j=0}^{D_{i}} \\frac{D!}{(D-j)!j!} \\end{equation} \n",
"\n"
],
"metadata": {
@@ -221,7 +220,7 @@
{
"cell_type": "code",
"source": [
"# Now let's plot the graph from figure 3.9a (takes ~1min)\n",
"# Now let's plot the graph from figure 3.9b (takes ~1min)\n",
"dims = np.array([1,5,10,50,100])\n",
"regions = np.zeros((dims.shape[0], 200))\n",
"params = np.zeros((dims.shape[0], 200))\n",

23
Notebooks/Chap04/4_3_Deep_Networks.ipynb
View File

@@ -4,7 +4,6 @@
"metadata": {
"colab": {
"provenance": [],
"authorship_tag": "ABX9TyO2DaD75p+LGi7WgvTzjrk1",
"include_colab_link": true
},
"kernelspec": {
@@ -31,7 +30,7 @@
"source": [
"# **Notebook 4.3 Deep neural networks**\n",
"\n",
"This network investigates converting neural networks to matrix form.\n",
"This notebook investigates converting neural networks to matrix form.\n",
"\n",
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
"\n",
@@ -150,7 +149,7 @@
{
"cell_type": "markdown",
"source": [
"Now we'll define the same neural network, but this time, we will use matrix form. When you get this right, it will draw the same plot as above."
"Now we'll define the same neural network, but this time, we will use matrix form as in equation 4.15. When you get this right, it will draw the same plot as above."
],
"metadata": {
"id": "XCJqo_AjfAra"
@@ -176,8 +175,8 @@
"n1_in_mat = np.reshape(n1_in,(n_dim_in,n_data))\n",
"\n",
"# This runs the network for ALL of the inputs, x at once so we can draw graph\n",
"h1 = ReLU(np.matmul(beta_0,np.ones((1,n_data))) + np.matmul(Omega_0,n1_in_mat))\n",
"n1_out = np.matmul(beta_1,np.ones((1,n_data))) + np.matmul(Omega_1,h1)\n",
"h1 = ReLU(beta_0 + np.matmul(Omega_0,n1_in_mat))\n",
"n1_out = beta_1 + np.matmul(Omega_1,h1)\n",
"\n",
"# Draw the network and check that it looks the same as the non-matrix case\n",
"plot_neural(n1_in, n1_out)"
@@ -247,9 +246,9 @@
"n1_in_mat = np.reshape(n1_in,(n_dim_in,n_data))\n",
"\n",
"# This runs the network for ALL of the inputs, x at once so we can draw graph (hence extra np.ones term)\n",
"h1 = ReLU(np.matmul(beta_0,np.ones((1,n_data))) + np.matmul(Omega_0,n1_in_mat))\n",
"h2 = ReLU(np.matmul(beta_1,np.ones((1,n_data))) + np.matmul(Omega_1,h1))\n",
"n1_out = np.matmul(beta_2,np.ones((1,n_data))) + np.matmul(Omega_2,h2)\n",
"h1 = ReLU(beta_0 + np.matmul(Omega_0,n1_in_mat))\n",
"h2 = ReLU(beta_1 + np.matmul(Omega_1,h1))\n",
"n1_out = beta_2 + np.matmul(Omega_2,h2)\n",
"\n",
"# Draw the network and check that it looks the same as the non-matrix version\n",
"plot_neural(n1_in, n1_out)"
@@ -291,10 +290,10 @@
"\n",
"\n",
"# If you set the parameters to the correct sizes, the following code will run\n",
"h1 = ReLU(np.matmul(beta_0,np.ones((1,n_data))) + np.matmul(Omega_0,x));\n",
"h2 = ReLU(np.matmul(beta_1,np.ones((1,n_data))) + np.matmul(Omega_1,h1));\n",
"h3 = ReLU(np.matmul(beta_2,np.ones((1,n_data))) + np.matmul(Omega_2,h2));\n",
"y = np.matmul(beta_3,np.ones((1,n_data))) + np.matmul(Omega_3,h3)\n",
"h1 = ReLU(beta_0 + np.matmul(Omega_0,x));\n",
"h2 = ReLU(beta_1 + np.matmul(Omega_1,h1));\n",
"h3 = ReLU(beta_2 + np.matmul(Omega_2,h2));\n",
"y = beta_3 + np.matmul(Omega_3,h3)\n",
"\n",
"if h1.shape[0] is not D_1 or h1.shape[1] is not n_data:\n",
" print(\"h1 is wrong shape\")\n",

2
Notebooks/Chap05/5_3_Multiclass_Cross_entropy_Loss.ipynb
View File

@@ -211,7 +211,7 @@
"id": "MvVX6tl9AEXF"
},
"source": [
"The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue). The dots at the bottom show the training data with the same color scheme. So we want the red curve to be high where there are red dots, the green curve to be high where there are green dots, and the blue curve to be high where there are blue dots We'll compute the the likelihood and the negative log likelihood."
"The left is model output and the right is the model output after the softmax has been applied, so it now lies in the range [0,1] and represents the probability, that y=0 (red), 1 (green) and 2 (blue). The dots at the bottom show the training data with the same color scheme. So we want the red curve to be high where there are red dots, the green curve to be high where there are green dots, and the blue curve to be high where there are blue dots We'll compute the likelihood and the negative log likelihood."
]
},
{

3
Notebooks/Chap06/6_1_Line_Search.ipynb
View File

@@ -130,7 +130,8 @@
"\n",
" print('Iter %d, a=%3.3f, b=%3.3f, c=%3.3f, d=%3.3f'%(n_iter, a,b,c,d))\n",
"\n",
" # Rule #1 If the HEIGHT at point A is less than the HEIGHT at points B, C, and D then halve values of B, C, and D\n",
" # Rule #1 If the HEIGHT at point A is less than the HEIGHT at points B, C, and D then move them to they are half\n",
" # as far from A as they start\n",
" # i.e. bring them closer to the original point\n",
" # TODO REPLACE THE BLOCK OF CODE BELOW WITH THIS RULE\n",
" if (0):\n",

24
Notebooks/Chap06/6_2_Gradient_Descent.ipynb
View File

@@ -1,18 +1,16 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "view-in-github"
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap06/6_2_Gradient_Descent.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "el8l05WQEO46"
@@ -111,7 +109,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "QU5mdGvpTtEG"
@@ -140,7 +137,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "eB5DQvU5hYNx"
@@ -162,7 +158,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "F3trnavPiHpH"
@@ -218,7 +213,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "s9Duf05WqqSC"
@@ -252,7 +246,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "RS1nEcYVuEAM"
@@ -290,7 +283,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "5EIjMM9Fw2eT"
@@ -333,11 +325,11 @@
" print('Iter %d, a=%3.3f, b=%3.3f, c=%3.3f, d=%3.3f'%(n_iter, a,b,c,d))\n",
" print('a %f, b%f, c%f, d%f'%(lossa,lossb,lossc,lossd))\n",
"\n",
" # Rule #1 If point A is less than points B, C, and D then halve points B,C, and D\n",
" # Rule #1 If point A is less than points B, C, and D then halve distance from A to points B,C, and D\n",
" if np.argmin((lossa,lossb,lossc,lossd))==0:\n",
" b = b/2\n",
" c = c/2\n",
" d = d/2\n",
" b = a+ (b-a)/2\n",
" c = a+ (c-a)/2\n",
" d = a+ (d-a)/2\n",
" continue;\n",
"\n",
" # Rule #2 If point b is less than point c then\n",
@@ -412,8 +404,8 @@
],
"metadata": {
"colab": {
"include_colab_link": true,
"provenance": []
"provenance": [],
"include_colab_link": true
},
"kernelspec": {
"display_name": "Python 3",

25
Notebooks/Chap06/6_3_Stochastic_Gradient_Descent.ipynb
View File

@@ -1,18 +1,16 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "view-in-github"
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap06/6_3_Stochastic_Gradient_Descent.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "el8l05WQEO46"
@@ -122,7 +120,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "QU5mdGvpTtEG"
@@ -150,7 +147,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "eB5DQvU5hYNx"
@@ -172,7 +168,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "F3trnavPiHpH"
@@ -228,7 +223,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "s9Duf05WqqSC"
@@ -279,7 +273,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "RS1nEcYVuEAM"
@@ -316,7 +309,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "5EIjMM9Fw2eT"
@@ -359,11 +351,11 @@
" print('Iter %d, a=%3.3f, b=%3.3f, c=%3.3f, d=%3.3f'%(n_iter, a,b,c,d))\n",
" print('a %f, b%f, c%f, d%f'%(lossa,lossb,lossc,lossd))\n",
"\n",
" # Rule #1 If point A is less than points B, C, and D then halve points B,C, and D\n",
" # Rule #1 If point A is less than points B, C, and D then change B,C,D so they are half their current distance from A\n",
" if np.argmin((lossa,lossb,lossc,lossd))==0:\n",
" b = b/2\n",
" c = c/2\n",
" d = d/2\n",
" b = a+ (b-a)/2\n",
" c = a+ (c-a)/2\n",
" d = a+ (d-a)/2\n",
" continue;\n",
"\n",
" # Rule #2 If point b is less than point c then\n",
@@ -577,9 +569,8 @@
],
"metadata": {
"colab": {
"authorship_tag": "ABX9TyNk5FN4qlw3pk8BwDVWw1jN",
"include_colab_link": true,
"provenance": []
"provenance": [],
"include_colab_link": true
},
"kernelspec": {
"display_name": "Python 3",

15
Notebooks/Chap07/7_2_Backpropagation.ipynb
View File

@@ -4,7 +4,6 @@
"metadata": {
"colab": {
"provenance": [],
"authorship_tag": "ABX9TyM2kkHLr00J4Jeypw41sTkQ",
"include_colab_link": true
},
"kernelspec": {
@@ -68,7 +67,7 @@
"# Set seed so we always get the same random numbers\n",
"np.random.seed(0)\n",
"\n",
"# Number of layers\n",
"# Number of hidden layers\n",
"K = 5\n",
"# Number of neurons per layer\n",
"D = 6\n",
@@ -115,7 +114,7 @@
{
"cell_type": "markdown",
"source": [
"Now let's run our random network. The weight matrices $\\boldsymbol\\Omega_{1\\ldots K}$ are the entries of the list \"all_weights\" and the biases $\\boldsymbol\\beta_{1\\ldots K}$ are the entries of the list \"all_biases\"\n",
"Now let's run our random network. The weight matrices $\\boldsymbol\\Omega_{0\\ldots K}$ are the entries of the list \"all_weights\" and the biases $\\boldsymbol\\beta_{0\\ldots K}$ are the entries of the list \"all_biases\"\n",
"\n",
"We know that we will need the preactivations $\\mathbf{f}_{0\\ldots K}$ and the activations $\\mathbf{h}_{1\\ldots K}$ for the forward pass of backpropagation, so we'll store and return these as well.\n"
],
@@ -142,7 +141,7 @@
"\n",
" # Run through the layers, calculating all_f[0...K-1] and all_h[1...K]\n",
" for layer in range(K):\n",
" # Update preactivations and activations at this layer according to eqn 7.16\n",
" # Update preactivations and activations at this layer according to eqn 7.17\n",
" # Remember to use np.matmul for matrix multiplications\n",
" # TODO -- Replace the lines below\n",
" all_f[layer] = all_h[layer]\n",
@@ -230,8 +229,8 @@
"# We'll need the indicator function\n",
"def indicator_function(x):\n",
" x_in = np.array(x)\n",
" x_in[x_in>=0] = 1\n",
" x_in[x_in<0] = 0\n",
" x_in[x_in>0] = 1\n",
" x_in[x_in<=0] = 0\n",
" return x_in\n",
"\n",
"# Main backward pass routine\n",
@@ -300,7 +299,7 @@
"delta_fd = 0.000001\n",
"\n",
"# Test the dervatives of the bias vectors\n",
"for layer in range(K):\n",
"for layer in range(K+1):\n",
" dl_dbias = np.zeros_like(all_dl_dbiases[layer])\n",
" # For every element in the bias\n",
" for row in range(all_biases[layer].shape[0]):\n",
@@ -324,7 +323,7 @@
"\n",
"\n",
"# Test the derivatives of the weights matrices\n",
"for layer in range(K):\n",
"for layer in range(K+1):\n",
" dl_dweight = np.zeros_like(all_dl_dweights[layer])\n",
" # For every element in the bias\n",
" for row in range(all_weights[layer].shape[0]):\n",

146
Notebooks/Chap08/8_1_MNIST_1D_Performance.ipynb
View File

@@ -1,28 +1,10 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"gpuType": "T4",
"authorship_tag": "ABX9TyOuKMUcKfOIhIL2qTX9jJCy",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
"colab_type": "text",
"id": "view-in-github"
},
"source": [
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap08/8_1_MNIST_1D_Performance.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
@@ -30,6 +12,9 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "L6chybAVFJW2"
},
"source": [
"# **Notebook 8.1: MNIST_1D_Performance**\n",
"\n",
@@ -38,25 +23,27 @@
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
"\n",
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
],
"metadata": {
"id": "L6chybAVFJW2"
}
]
},
{
"cell_type": "code",
"source": [
"# Run this if you're in a Colab to install MNIST 1D repository\n",
"%pip install git+https://github.com/greydanus/mnist1d"
],
"execution_count": null,
"metadata": {
"id": "ifVjS4cTOqKz"
},
"execution_count": null,
"outputs": []
"outputs": [],
"source": [
"# Run this if you're in a Colab to install MNIST 1D repository\n",
"%pip install git+https://github.com/greydanus/mnist1d"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qyE7G1StPIqO"
},
"outputs": [],
"source": [
"import torch, torch.nn as nn\n",
"from torch.utils.data import TensorDataset, DataLoader\n",
@@ -64,44 +51,42 @@
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import mnist1d"
],
"metadata": {
"id": "qyE7G1StPIqO"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"source": [
"Let's generate a training and test dataset using the MNIST1D code. The dataset gets saved as a .pkl file so it doesn't have to be regenerated each time."
],
"metadata": {
"id": "F7LNq72SP6jO"
}
},
"source": [
"Let's generate a training and test dataset using the MNIST1D code. The dataset gets saved as a .pkl file so it doesn't have to be regenerated each time."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "YLxf7dJfPaqw"
},
"outputs": [],
"source": [
"!mkdir ./sample_data\n",
"\n",
"args = mnist1d.data.get_dataset_args()\n",
"data = mnist1d.data.get_dataset(args, path='./sample_data/mnist1d_data.pkl', download=False, regenerate=False)\n",
"data = mnist1d.data.get_dataset(args, path='./mnist1d_data.pkl', download=False, regenerate=False)\n",
"\n",
"# The training and test input and outputs are in\n",
"# data['x'], data['y'], data['x_test'], and data['y_test']\n",
"print(\"Examples in training set: {}\".format(len(data['y'])))\n",
"print(\"Examples in test set: {}\".format(len(data['y_test'])))\n",
"print(\"Length of each example: {}\".format(data['x'].shape[-1]))"
],
"metadata": {
"id": "YLxf7dJfPaqw"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "FxaB5vc0uevl"
},
"outputs": [],
"source": [
"D_i = 40 # Input dimensions\n",
"D_k = 100 # Hidden dimensions\n",
@@ -122,15 +107,15 @@
"\n",
"# Call the function you just defined\n",
"model.apply(weights_init)\n"
],
"metadata": {
"id": "FxaB5vc0uevl"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_rX6N3VyyQTY"
},
"outputs": [],
"source": [
"# choose cross entropy loss function (equation 5.24)\n",
"loss_function = torch.nn.CrossEntropyLoss()\n",
@@ -139,9 +124,9 @@
"# object that decreases learning rate by half every 10 epochs\n",
"scheduler = StepLR(optimizer, step_size=10, gamma=0.5)\n",
"x_train = torch.tensor(data['x'].astype('float32'))\n",
"y_train = torch.tensor(data['y'].transpose().astype('long'))\n",
"y_train = torch.tensor(data['y'].transpose().astype('int64'))\n",
"x_test= torch.tensor(data['x_test'].astype('float32'))\n",
"y_test = torch.tensor(data['y_test'].astype('long'))\n",
"y_test = torch.tensor(data['y_test'].astype('int64'))\n",
"\n",
"# load the data into a class that creates the batches\n",
"data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=100, shuffle=True, worker_init_fn=np.random.seed(1))\n",
@@ -186,15 +171,15 @@
"\n",
" # tell scheduler to consider updating learning rate\n",
" scheduler.step()"
],
"metadata": {
"id": "_rX6N3VyyQTY"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yI-l6kA_EH9G"
},
"outputs": [],
"source": [
"# Plot the results\n",
"fig, ax = plt.subplots()\n",
@@ -215,25 +200,38 @@
"ax.set_title('Train loss %3.2f, Test loss %3.2f'%(losses_train[-1],losses_test[-1]))\n",
"ax.legend()\n",
"plt.show()"
],
"metadata": {
"id": "yI-l6kA_EH9G"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "q-yT6re6GZS4"
},
"source": [
"**TO DO**\n",
"\n",
"Play with the model -- try changing the number of layers, hidden units, learning rate, batch size, momentum or anything else you like. See if you can improve the test results.\n",
"\n",
"Is it a good idea to optimize the hyperparameters in this way? Will the final result be a good estimate of the true test performance?"
],
"metadata": {
"id": "q-yT6re6GZS4"
}
}
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"authorship_tag": "ABX9TyOuKMUcKfOIhIL2qTX9jJCy",
"gpuType": "T4",
"include_colab_link": true,
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}

70
Notebooks/Chap08/8_3_Double_Descent.ipynb
View File

@@ -99,7 +99,7 @@
"# data['x'], data['y'], data['x_test'], and data['y_test']\n",
"print(\"Examples in training set: {}\".format(len(data['y'])))\n",
"print(\"Examples in test set: {}\".format(len(data['y_test'])))\n",
"print(\"Length of each example: {}\".format(data['x'].shape[-1]))"
"print(\"Dimensionality of each example: {}\".format(data['x'].shape[-1]))"
],
"metadata": {
"id": "PW2gyXL5UkLU"
@@ -147,7 +147,7 @@
{
"cell_type": "code",
"source": [
"def fit_model(model, data):\n",
"def fit_model(model, data, n_epoch):\n",
"\n",
" # choose cross entropy loss function (equation 5.24)\n",
" loss_function = torch.nn.CrossEntropyLoss()\n",
@@ -164,9 +164,6 @@
" # load the data into a class that creates the batches\n",
" data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=100, shuffle=True, worker_init_fn=np.random.seed(1))\n",
"\n",
" # loop over the dataset n_epoch times\n",
" n_epoch = 1000\n",
"\n",
" for epoch in range(n_epoch):\n",
" # loop over batches\n",
" for i, batch in enumerate(data_loader):\n",
@@ -203,6 +200,18 @@
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"def count_parameters(model):\n",
" return sum(p.numel() for p in model.parameters() if p.requires_grad)"
],
"metadata": {
"id": "AQNCmFNV6JpV"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
@@ -226,19 +235,27 @@
"# This code will take a while (~30 mins on GPU) to run! Go and make a cup of coffee!\n",
"\n",
"hidden_variables = np.array([2,4,6,8,10,14,18,22,26,30,35,40,45,50,55,60,70,80,90,100,120,140,160,180,200,250,300,400]) ;\n",
"\n",
"errors_train_all = np.zeros_like(hidden_variables)\n",
"errors_test_all = np.zeros_like(hidden_variables)\n",
"total_weights_all = np.zeros_like(hidden_variables)\n",
"\n",
"# loop over the dataset n_epoch times\n",
"n_epoch = 1000\n",
"\n",
"# For each hidden variable size\n",
"for c_hidden in range(len(hidden_variables)):\n",
" print(f'Training model with {hidden_variables[c_hidden]:3d} hidden variables')\n",
" # Get a model\n",
" model = get_model(hidden_variables[c_hidden]) ;\n",
" # Count and store number of weights\n",
" total_weights_all[c_hidden] = count_parameters(model)\n",
" # Train the model\n",
" errors_train, errors_test = fit_model(model, data)\n",
" errors_train, errors_test = fit_model(model, data, n_epoch)\n",
" # Store the results\n",
" errors_train_all[c_hidden] = errors_train\n",
" errors_test_all[c_hidden]= errors_test"
" errors_test_all[c_hidden]= errors_test\n",
"\n"
],
"metadata": {
"id": "K4OmBZGHWXpk"
@@ -249,12 +266,29 @@
{
"cell_type": "code",
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"\n",
"# Assuming data['y'] is available and contains the training examples\n",
"num_training_examples = len(data['y'])\n",
"\n",
"# Find the index where total_weights_all is closest to num_training_examples\n",
"closest_index = np.argmin(np.abs(np.array(total_weights_all) - num_training_examples))\n",
"\n",
"# Get the corresponding value of hidden variables\n",
"hidden_variable_at_num_training_examples = hidden_variables[closest_index]\n",
"\n",
"# Plot the results\n",
"fig, ax = plt.subplots()\n",
"ax.plot(hidden_variables, errors_train_all, 'r-', label='train')\n",
"ax.plot(hidden_variables, errors_test_all, 'b-', label='test')\n",
"ax.set_ylim(0,100);\n",
"ax.set_xlabel('No hidden variables'); ax.set_ylabel('Error')\n",
"\n",
"# Add a vertical line at the point where total weights equal the number of training examples\n",
"ax.axvline(x=hidden_variable_at_num_training_examples, color='g', linestyle='--', label='N(weights) = N(train)')\n",
"\n",
"ax.set_ylim(0, 100)\n",
"ax.set_xlabel('No. hidden variables')\n",
"ax.set_ylabel('Error')\n",
"ax.legend()\n",
"plt.show()\n"
],
@@ -263,6 +297,24 @@
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [],
"metadata": {
"id": "KT4X8_hE5NFb"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [],
"metadata": {
"id": "iGKZSfVF2r4z"
},
"execution_count": null,
"outputs": []
}
]
}

2
Notebooks/Chap08/8_4_High_Dimensional_Spaces.ipynb
View File

@@ -134,7 +134,7 @@
"source": [
"# Volume of a hypersphere\n",
"\n",
"In the second part of this notebook we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius. Note that you you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
"In the second part of this notebook we calculate the volume of a hypersphere of radius 0.5 (i.e., of diameter 1) as a function of the radius. Note that you can check your answer by doing the calculation for 2D using the standard formula for the area of a circle and making sure it matches."
],
"metadata": {
"id": "b2FYKV1SL4Z7"

5
Notebooks/Chap09/9_5_Augmentation.ipynb
View File

@@ -107,10 +107,7 @@
" # Initialize the parameters with He initialization\n",
" if isinstance(layer_in, nn.Linear):\n",
" nn.init.kaiming_uniform_(layer_in.weight)\n",
" layer_in.bias.data.fill_(0.0)\n",
"\n",
"# Call the function you just defined\n",
"model.apply(weights_init)"
" layer_in.bias.data.fill_(0.0)\n"
],
"metadata": {
"id": "JfIFWFIL33eF"

2
Notebooks/Chap10/10_4_Downsampling_and_Upsampling.ipynb
View File

@@ -31,7 +31,7 @@
"source": [
"# **Notebook 10.4: Downsampling and Upsampling**\n",
"\n",
"This notebook investigates the down sampling and downsampling methods discussed in section 10.4 of the book.\n",
"This notebook investigates the upsampling and downsampling methods discussed in section 10.4 of the book.\n",
"\n",
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
"\n",

15
Notebooks/Chap10/10_5_Convolution_For_MNIST.ipynb
View File

@@ -4,7 +4,7 @@
"metadata": {
"colab": {
"provenance": [],
"authorship_tag": "ABX9TyNAcc98STMeyQgh9SbVHWG+",
"authorship_tag": "ABX9TyORZF8xy4X1yf4oRhRq8Rtm",
"include_colab_link": true
},
"kernelspec": {
@@ -65,10 +65,19 @@
"source": [
"# Run this once to load the train and test data straight into a dataloader class\n",
"# that will provide the batches\n",
"\n",
"# (It may complain that some files are missing because the files seem to have been\n",
"# reorganized on the underlying website, but it still seems to work). If everything is working\n",
"# properly, then the whole notebook should run to the end without further problems\n",
"# even before you make changes.\n",
"batch_size_train = 64\n",
"batch_size_test = 1000\n",
"\n",
"# TODO Change this directory to point towards an existing directory\n",
"myDir = '/files/'\n",
"\n",
"train_loader = torch.utils.data.DataLoader(\n",
" torchvision.datasets.MNIST('/files/', train=True, download=True,\n",
" torchvision.datasets.MNIST(myDir, train=True, download=True,\n",
" transform=torchvision.transforms.Compose([\n",
" torchvision.transforms.ToTensor(),\n",
" torchvision.transforms.Normalize(\n",
@@ -77,7 +86,7 @@
" batch_size=batch_size_train, shuffle=True)\n",
"\n",
"test_loader = torch.utils.data.DataLoader(\n",
" torchvision.datasets.MNIST('/files/', train=False, download=True,\n",
" torchvision.datasets.MNIST(myDir, train=False, download=True,\n",
" transform=torchvision.transforms.Compose([\n",
" torchvision.transforms.ToTensor(),\n",
" torchvision.transforms.Normalize(\n",

2
Notebooks/Chap12/12_2_Multihead_Self_Attention.ipynb
View File

@@ -28,7 +28,7 @@
{
"cell_type": "markdown",
"source": [
"# **Notebook 12.1: Multihead Self-Attention**\n",
"# **Notebook 12.2: Multihead Self-Attention**\n",
"\n",
"This notebook builds a multihead self-attention mechanism as in figure 12.6\n",
"\n",

2
Notebooks/Chap13/13_4_Graph_Attention_Networks.ipynb
View File

@@ -109,7 +109,7 @@
"# Choose random values for the parameters\n",
"omega = np.random.normal(size=(D,D))\n",
"beta = np.random.normal(size=(D,1))\n",
"phi = np.random.normal(size=(1,2*D))"
"phi = np.random.normal(size=(2*D,1))"
],
"metadata": {
"id": "79TSK7oLMobe"

1
Notebooks/Chap15/15_2_Wasserstein_Distance.ipynb
View File

@@ -86,6 +86,7 @@
"cell_type": "code",
"source": [
"# TODO Define the distance matrix from figure 15.8d\n",
"# The index should be normalized before being used in the distance calculation.\n",
"# Replace this line\n",
"dist_mat = np.zeros((10,10))\n",
"\n",

2
Notebooks/Chap17/17_1_Latent_Variable_Models.ipynb
View File

@@ -55,7 +55,7 @@
"Pr(z) = \\text{Norm}_{z}[0,1]\n",
"\\end{equation}\n",
"\n",
"As in figure 17.2, we'll assume that the output is two dimensional, we we need to define a function that maps from the 1D latent variable to two dimensions. Usually, we would use a neural network, but in this case, we'll just define an arbitrary relationship.\n",
"As in figure 17.2, we'll assume that the output is two dimensional, we need to define a function that maps from the 1D latent variable to two dimensions. Usually, we would use a neural network, but in this case, we'll just define an arbitrary relationship.\n",
"\n",
"\\begin{align}\n",
"x_{1} &=& 0.5\\cdot\\exp\\Bigl[\\sin\\bigl[2+ 3.675 z \\bigr]\\Bigr]\\\\\n",

2
Notebooks/Chap19/19_2_Dynamic_Programming.ipynb
View File

@@ -393,7 +393,7 @@
{
"cell_type": "code",
"source": [
"# Update the state values for the current policy, by making the values at at adjacent\n",
"# Update the state values for the current policy, by making the values at adjacent\n",
"# states compatible with the Bellman equation (equation 19.11)\n",
"def policy_evaluation(policy, state_values, rewards, transition_probabilities_given_action, gamma):\n",
"\n",

17
Notebooks/Chap20/20_3_Lottery_Tickets.ipynb
View File

@@ -44,7 +44,8 @@
},
"source": [
"# Run this if you're in a Colab to install MNIST 1D repository\n",
"!pip install git+https://github.com/greydanus/mnist1d"
"!pip install git+https://github.com/greydanus/mnist1d\n",
"!git clone https://github.com/greydanus/mnist1d"
],
"execution_count": null,
"outputs": []
@@ -95,6 +96,12 @@
"id": "I-vm_gh5xTJs"
},
"source": [
"from mnist1d.data import get_dataset, get_dataset_args\n",
"from mnist1d.utils import set_seed, to_pickle, from_pickle\n",
"\n",
"import sys ; sys.path.append('./mnist1d/notebooks')\n",
"from train import get_model_args, train_model\n",
"\n",
"args = mnist1d.get_dataset_args()\n",
"data = mnist1d.get_dataset(args=args) # by default, this will download a pre-made dataset from the GitHub repo\n",
"\n",
@@ -210,7 +217,7 @@
" # we would return [1,1,0,0,1]\n",
" # Remember that these are torch tensors and not numpy arrays\n",
" # Replace this function:\n",
" mask = torch.ones_like(scores)\n",
" mask = torch.ones_like(absolute_weights)\n",
"\n",
"\n",
" return mask"
@@ -237,7 +244,6 @@
"def find_lottery_ticket(model, dataset, args, sparsity_schedule, criteria_fn=None, **kwargs):\n",
"\n",
" criteria_fn = lambda init_params, final_params: final_params.abs()\n",
"\n",
" init_params = model.get_layer_vecs()\n",
" stats = {'train_losses':[], 'test_losses':[], 'train_accs':[], 'test_accs':[]}\n",
" models = []\n",
@@ -253,7 +259,7 @@
" model.set_layer_masks(masks)\n",
"\n",
" # training process\n",
" results = mnist1d.train_model(dataset, model, args)\n",
" results = train_model(dataset, model, args)\n",
" model = results['checkpoints'][-1]\n",
"\n",
" # store stats\n",
@@ -291,7 +297,8 @@
},
"source": [
"# train settings\n",
"model_args = mnist1d.get_model_args()\n",
"from train import get_model_args, train_model\n",
"model_args = get_model_args()\n",
"model_args.total_steps = 1501\n",
"model_args.hidden_size = 500\n",
"model_args.print_every = 5000 # print never\n",

4
Notebooks/Chap21/21_1_Bias_Mitigation.ipynb
View File

@@ -137,7 +137,7 @@
"id": "CfZ-srQtmff2"
},
"source": [
"Why might the distributions for blue and yellow populations be different? It could be that the behaviour of the populations is identical, but the credit rating algorithm is biased; it may favor one population over another or simply be more noisy for one group. Alternatively, it could be that that the populations genuinely behave differently. In practice, the differences in blue and yellow distributions are probably attributable to a combination of these factors.\n",
"Why might the distributions for blue and yellow populations be different? It could be that the behaviour of the populations is identical, but the credit rating algorithm is biased; it may favor one population over another or simply be more noisy for one group. Alternatively, it could be that the populations genuinely behave differently. In practice, the differences in blue and yellow distributions are probably attributable to a combination of these factors.\n",
"\n",
"Let’s assume that we can’t retrain the credit score prediction algorithm; our job is to adjudicate whether each individual is refused the loan ($\\hat{y}=0$)\n",
" or granted it ($\\hat{y}=1$). Since we only have the credit score\n",
@@ -382,7 +382,7 @@
"source": [
"# Equal opportunity:\n",
"\n",
"The thresholds are chosen so that so that the true positive rate is is the same for both population. Of the people who pay back the loan, the same proportion are offered credit in each group. In terms of the two ROC curves, it means choosing thresholds so that the vertical position on each curve is the same without regard for the horizontal position."
"The thresholds are chosen so that so that the true positive rate is the same for both population. Of the people who pay back the loan, the same proportion are offered credit in each group. In terms of the two ROC curves, it means choosing thresholds so that the vertical position on each curve is the same without regard for the horizontal position."
]
},
{

51
Trees/LinearRegression_LeastSquares.ipynb Normal file
View File

@@ -0,0 +1,51 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"authorship_tag": "ABX9TyM1pe3HkxLrjbeKezq1MlM5",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Trees/LinearRegression_LeastSquares.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"# Least Squares Loss"
],
"metadata": {
"id": "uORlKyPv02ge"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bbF6SE_F0tU8"
},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt"
]
}
]
}

51
Trees/LinearRegression_LossFunction.ipynb Normal file
View File

@@ -0,0 +1,51 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"authorship_tag": "ABX9TyMIJ9DpOBppPZXAJ5wms6s8",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Trees/LinearRegression_LossFunction.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"# Loss function"
],
"metadata": {
"id": "uORlKyPv02ge"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bbF6SE_F0tU8"
},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt"
]
}
]
}

BIN
UDL_Answer_Booklet_Students.pdf
View File

Binary file not shown.

2229
UDL_Equations.tex Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
UDL_Errata.pdf
View File

Binary file not shown.

1
index.html
View File

@@ -10,6 +10,7 @@
href="https://fonts.googleapis.com/css2?family=Encode+Sans+Expanded:wght@400;700&display=swap"
rel="stylesheet"
/>
<title>Understanding Deep Learning</title>
</head>
<body>

0
README.md → src/README.md
View File

8672
understanding-deep-learning-final.bib Normal file
View File

File diff suppressed because it is too large Load Diff