Compare commits
32 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
e9fa659fee | ||
|
|
5db96e8a3d | ||
|
|
6ac37b5548 | ||
|
|
9acd27c8b3 | ||
|
|
a481425005 | ||
|
|
41e60e8597 | ||
|
|
ec748ecc18 | ||
|
|
467a36bed8 | ||
|
|
351b050e7e | ||
|
|
5f19405c51 | ||
|
|
ad5339a972 | ||
|
|
1d8a9af439 | ||
|
|
55a96289fe | ||
|
|
d72143ff42 | ||
|
|
a1154e2262 | ||
|
|
b419f52791 | ||
|
|
8c0bec30ff | ||
|
|
87cfb0a846 | ||
|
|
0fcfa4fe36 | ||
|
|
8e6fd1bf81 | ||
|
|
7a2a7cd2e8 | ||
|
|
f46ae30bf2 | ||
|
|
24206afa50 | ||
|
|
77ec015669 | ||
|
|
1c40b09e48 | ||
|
|
972e035974 | ||
|
|
d7a792785f | ||
|
|
80ef390e45 | ||
|
|
f70dd6e60a | ||
|
|
a7d840f204 | ||
|
|
a32260c39f | ||
|
|
6b2288665f |
@@ -4,7 +4,7 @@
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyPD+qTkgmZCe+VessXM/kIU",
|
||||
"authorship_tag": "ABX9TyNk2dAhwwRxGpfVSC3b2Owv",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
@@ -182,7 +182,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now we'll extend this model to have two outputs $y_1$ and $y_2$, each of which can be visualized with a separate heatmap. You will now have sets of parameters $\\phi_{10}, \\phi_{11},\\phi_{12}$ and $\\phi_{2}, \\phi_{21},\\phi_{22}$ that correspond to each of these outputs."
|
||||
"Now we'll extend this model to have two outputs $y_1$ and $y_2$, each of which can be visualized with a separate heatmap. You will now have sets of parameters $\\phi_{10}, \\phi_{11},\\phi_{12}$ and $\\phi_{20}, \\phi_{21},\\phi_{22}$ that correspond to each of these outputs."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Xl6LcrUyM7Lh"
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyMdflMfWi9hu9ZEg/80HCd8",
|
||||
"authorship_tag": "ABX9TyNioITtfAcfxEfM3UOfQyb9",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
@@ -79,7 +79,7 @@
|
||||
"source": [
|
||||
"def number_regions(Di, D):\n",
|
||||
" # TODO -- implement Zaslavsky's formula\n",
|
||||
" # You will need to use math.factorial() https://www.geeksforgeeks.org/factorial-in-python/\n",
|
||||
" # You can use math.com() https://www.w3schools.com/python/ref_math_comb.asp\n",
|
||||
" # Replace this code\n",
|
||||
" N = 1;\n",
|
||||
"\n",
|
||||
@@ -191,7 +191,7 @@
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Now let's compute and plot the number of regions as a function of the number of parameters as in figure 3.9b\n",
|
||||
"# First let's write a function that computes the number of parameters as a function of the input dimension and number of hidden layers (assuming just one output)\n",
|
||||
"# First let's write a function that computes the number of parameters as a function of the input dimension and number of hidden units (assuming just one output)\n",
|
||||
"\n",
|
||||
"def number_parameters(D_i, D):\n",
|
||||
" # TODO -- replace this code with the proper calculation\n",
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyMbJGN6f2+yKzzsVep/wi5U",
|
||||
"authorship_tag": "ABX9TyPyaqr0yJlxfIcTpfLSHDrP",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
@@ -274,7 +274,7 @@
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# define sizes\n",
|
||||
"D_i=4; D_1=5; D_2=2; D_3=1; D_o=1\n",
|
||||
"D_i=4; D_1=5; D_2=2; D_3=4; D_o=1\n",
|
||||
"# We'll choose the inputs and parameters of this network randomly using np.random.normal\n",
|
||||
"# For example, we'll set the input using\n",
|
||||
"n_data = 4;\n",
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyNkBMOVt5gO7Awn9JMn4N8Z",
|
||||
"authorship_tag": "ABX9TyPX88BLalmJTle9GSAZMJcz",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
@@ -307,7 +307,7 @@
|
||||
"# Return the negative log likelihood of the data under the model\n",
|
||||
"def compute_negative_log_likelihood(y_train, mu, sigma):\n",
|
||||
" # TODO -- compute the likelihood of the data -- don't use the likelihood function above -- compute the negative sum of the log probabilities\n",
|
||||
" # Bottom line of equation 5.3 in the notes\n",
|
||||
" # Equation 5.4 in the notes\n",
|
||||
" # You will need np.sum(), np.log()\n",
|
||||
" # Replace the line below\n",
|
||||
" nll = 0\n",
|
||||
|
||||
@@ -341,7 +341,7 @@
|
||||
"source": [
|
||||
"# Computes the regularization term\n",
|
||||
"def compute_reg_term(phi0,phi1):\n",
|
||||
" # TODO compute the regularization term (term in large brackets in the above equstion)\n",
|
||||
" # TODO compute the regularization term (term in large brackets in the above equation)\n",
|
||||
" # Replace this line\n",
|
||||
" reg_term = 0.0\n",
|
||||
"\n",
|
||||
|
||||
396
Notebooks/Chap17/17_1_Latent_Variable_Models.ipynb
Normal file
396
Notebooks/Chap17/17_1_Latent_Variable_Models.ipynb
Normal file
@@ -0,0 +1,396 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyMBYNsjj1iTgHUYhAXqUYJd",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "view-in-github",
|
||||
"colab_type": "text"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap17/17_1_Latent_Variable_Models.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# **Notebook 17.1: Latent variable models**\n",
|
||||
"\n",
|
||||
"This notebook investigates a non-linear latent variable model similar to that in figures 17.2 and 17.3 of the book.\n",
|
||||
"\n",
|
||||
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
|
||||
"\n",
|
||||
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "t9vk9Elugvmi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import scipy\n",
|
||||
"from matplotlib.colors import ListedColormap\n",
|
||||
"from matplotlib import cm"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "OLComQyvCIJ7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"We'll assume that our base distribution over the latent variables is a 1D standard normal so that\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"Pr(z) = \\mbox{Norm}_{z}[0,1]\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"As in figure 17.2, we'll assume that the output is two dimensional, we we need to define a function that maps from the 1D latent variable to two dimensions. Usually, we would use a neural network, but in this case, we'll just define an arbitrary relationship.\n",
|
||||
"\n",
|
||||
"\\begin{eqnarray}\n",
|
||||
"x_{1} &=& 0.5\\cdot\\exp\\Bigl[\\sin\\bigl[2+ 3.675 z \\bigr]\\Bigr]\\\\\n",
|
||||
"x_{2} &=& \\sin\\bigl[2+ 2.85 z \\bigr]\n",
|
||||
"\\end{eqnarray}"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "IyVn-Gi-p7wf"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# The function that maps z to x1 and x2\n",
|
||||
"def f(z):\n",
|
||||
" x_1 = np.exp(np.sin(2+z*3.675)) * 0.5\n",
|
||||
" x_2 = np.cos(2+z*2.85)\n",
|
||||
" return x_1, x_2"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ZIfQwhd-AV6L"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's plot the 3D relation between the two observed variables $x_{1}$ and $x_{2}$ and the latent variables $z$ as in figure 17.2 of the book. We'll use the opacity to represent the prior probability $Pr(z)$."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "KB9FU34onW1j"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def draw_3d_projection(z,pr_z, x1,x2):\n",
|
||||
" alpha = pr_z / np.max(pr_z)\n",
|
||||
" ax = plt.axes(projection='3d')\n",
|
||||
" fig = plt.gcf()\n",
|
||||
" fig.set_size_inches(18.5, 10.5)\n",
|
||||
" for i in range(len(z)-1):\n",
|
||||
" ax.plot([z[i],z[i+1]],[x1[i],x1[i+1]],[x2[i],x2[i+1]],'r-', alpha=pr_z[i])\n",
|
||||
" ax.set_xlabel('$z$',)\n",
|
||||
" ax.set_ylabel('$x_1$')\n",
|
||||
" ax.set_zlabel('$x_2$')\n",
|
||||
" ax.set_xlim(-3,3)\n",
|
||||
" ax.set_ylim(0,2)\n",
|
||||
" ax.set_zlim(-1,1)\n",
|
||||
" ax.set_box_aspect((3,1,1))\n",
|
||||
" plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "lW08xqAgnP4q"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Compute the prior\n",
|
||||
"def get_prior(z):\n",
|
||||
" return scipy.stats.multivariate_normal.pdf(z)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "9DUTauMi6tPk"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Define the latent variable values\n",
|
||||
"z = np.arange(-3.0,3.0,0.01)\n",
|
||||
"# Find the probability distribution over z\n",
|
||||
"pr_z = get_prior(z)\n",
|
||||
"# Compute x1 and x2 for each z\n",
|
||||
"x1,x2 = f(z)\n",
|
||||
"# Plot the function\n",
|
||||
"draw_3d_projection(z,pr_z, x1,x2)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "PAzHq461VqvF"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"The likelihood is defined as:\n",
|
||||
"\\begin{eqnarray}\n",
|
||||
" Pr(x_1,x_2|z) &=& \\mbox{Norm}_{[x_1,x_2]}\\Bigl[\\mathbf{f}[z],\\sigma^{2}\\mathbf{I}\\Bigr]\n",
|
||||
"\\end{eqnarray}\n",
|
||||
"\n",
|
||||
"so we will also need to define the noise level $\\sigma^2$"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "sQg2gKR5zMrF"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"sigma_sq = 0.04"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "In_Vg4_0nva3"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Draws a heatmap to represent a probability distribution, possibly with samples overlaed\n",
|
||||
"def plot_heatmap(x1_mesh,x2_mesh,y_mesh, x1_samples=None, x2_samples=None, title=None):\n",
|
||||
" # Define pretty colormap\n",
|
||||
" my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
|
||||
" my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
|
||||
" r = np.floor(my_colormap_vals_dec/(256*256))\n",
|
||||
" g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
|
||||
" b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
|
||||
" my_colormap = ListedColormap(np.vstack((r,g,b)).transpose()/255.0)\n",
|
||||
"\n",
|
||||
" fig,ax = plt.subplots()\n",
|
||||
" fig.set_size_inches(8,8)\n",
|
||||
" ax.contourf(x1_mesh,x2_mesh,y_mesh,256,cmap=my_colormap)\n",
|
||||
" ax.contour(x1_mesh,x2_mesh,y_mesh,8,colors=['#80808080'])\n",
|
||||
" if title is not None:\n",
|
||||
" ax.set_title(title);\n",
|
||||
" if x1_samples is not None:\n",
|
||||
" ax.plot(x1_samples, x2_samples, 'c.')\n",
|
||||
" ax.set_xlim([-0.5,2.5])\n",
|
||||
" ax.set_ylim([-1.5,1.5])\n",
|
||||
" ax.set_xlabel('$x_1$'); ax.set_ylabel('$x_2$')\n",
|
||||
" plt.show()\n",
|
||||
"\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "6P6d-AgAqxXZ"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Returns the likelihood\n",
|
||||
"def get_likelihood(x1_mesh, x2_mesh, z_val):\n",
|
||||
" # Find the corresponding x1 and x2 values\n",
|
||||
" x1,x2 = f(z_val)\n",
|
||||
"\n",
|
||||
" # Calculate the probability for a mesh of x1,x2 values.\n",
|
||||
" mn = scipy.stats.multivariate_normal([x1, x2], [[sigma_sq, 0], [0, sigma_sq]])\n",
|
||||
" pr_x1_x2_given_z_val = mn.pdf(np.dstack((x1_mesh, x2_mesh)))\n",
|
||||
" return pr_x1_x2_given_z_val"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "diYKb7_ZgjlJ"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's plot the likelihood $Pr(x_1,x_2|z)$ as in fig 17.3b in the book."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "0X4NwixzqxtZ"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Choose some z value\n",
|
||||
"z_val = 1.8\n",
|
||||
"\n",
|
||||
"# Compute the conditional distribution on a grid\n",
|
||||
"x1_mesh, x2_mesh = np.meshgrid(np.arange(-0.5,2.5,0.01), np.arange(-1.5,1.5,0.01))\n",
|
||||
"pr_x1_x2_given_z_val = get_likelihood(x1_mesh,x2_mesh, z_val)\n",
|
||||
"\n",
|
||||
"# Plot the result\n",
|
||||
"plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2_given_z_val, title=\"Conditional distribution $Pr(x1,x2|z)$\")\n",
|
||||
"\n",
|
||||
"# TODO -- Experiment with different values of z and make sure that you understand the what is happening."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "hWfqK-Oz5_DT"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"The data density is found by marginalizing over the latent variables $z$:\n",
|
||||
"\n",
|
||||
"\\begin{eqnarray}\n",
|
||||
" Pr(x_1,x_2) &=& \\int Pr(x_1,x_2, z) dz \\nonumber \\\\\n",
|
||||
" &=& \\int Pr(x_1,x_2 | z) \\cdot Pr(z)dz\\nonumber \\\\\n",
|
||||
" &=& \\int \\mbox{Norm}_{[x_1,x_2]}\\Bigl[\\mathbf{f}[z],\\sigma^{2}\\mathbf{I}\\Bigr]\\cdot \\mbox{Norm}_{z}\\left[\\mathbf{0},\\mathbf{I}\\right]dz.\n",
|
||||
"\\end{eqnarray}"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "25xqXnmFo-PH"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# TODO Compute the data density\n",
|
||||
"# We can't integrate this function in closed form\n",
|
||||
"# So let's approximate it as a sum over the z values (z = np.arange(-3,3,0.01))\n",
|
||||
"# You will need the functions get_likelihood() and get_prior()\n",
|
||||
"# To make this a valid probability distribution, you need to divide\n",
|
||||
"# By the z-increment (0.01)\n",
|
||||
"# Replace this line\n",
|
||||
"pr_x1_x2 = np.zeros_like(x1_mesh)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Plot the result\n",
|
||||
"plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2, title=\"Data density $Pr(x1,x2)$\")\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "H0Ijce9VzeCO"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's draw some samples from the model"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "W264N7By_h9y"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def draw_samples(n_sample):\n",
|
||||
" # TODO Write this routine to draw n_sample samples from the model\n",
|
||||
" # First draw a random value of z from the prior (a standard normal distribution)\n",
|
||||
" # Then draw a sample from Pr(x1,x2|z)\n",
|
||||
" # Replace this line\n",
|
||||
" x1_samples=0; x2_samples = 0;\n",
|
||||
"\n",
|
||||
" return x1_samples, x2_samples"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Li3mK_I48k0k"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's plot those samples on top of the heat map."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "D7N7oqLe-eJO"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"x1_samples, x2_samples = draw_samples(500)\n",
|
||||
"# Plot the result\n",
|
||||
"plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2, x1_samples, x2_samples, title=\"Data density $Pr(x1,x2)$\")\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "XRmWv99B-BWO"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Return the posterior distribution\n",
|
||||
"def get_posterior(x1,x2):\n",
|
||||
" z = np.arange(-3,3, 0.01)\n",
|
||||
" # TODO -- write this function\n",
|
||||
" # Again, we can't integrate, but we can sum\n",
|
||||
" # We don't know the constant in the denominator of equation 17.19, but we can just normalize\n",
|
||||
" # by the sum of the numerators for all values of z\n",
|
||||
" # Replace this line:\n",
|
||||
" pr_z_given_x1_x2 = np.ones_like(z)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return z, pr_z_given_x1_x2"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "PwOjzPD5_1OF"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"x1 = 0.9; x2 = -0.9\n",
|
||||
"z, pr_z_given_x1_x2 = get_posterior(x1,x2)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"fig, ax = plt.subplots()\n",
|
||||
"ax.plot(z, pr_z_given_x1_x2, 'r-')\n",
|
||||
"ax.set_xlabel(\"Latent variable $z$\")\n",
|
||||
"ax.set_ylabel(\"Posterior probability $Pr(z|x_{1},x_{2})$\")\n",
|
||||
"ax.set_xlim([-3,3])\n",
|
||||
"ax.set_ylim([0,1.5 * np.max(pr_z_given_x1_x2)])\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "PKFUY42K-Tp7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
}
|
||||
]
|
||||
}
|
||||
423
Notebooks/Chap17/17_2_Reparameterization_Trick.ipynb
Normal file
423
Notebooks/Chap17/17_2_Reparameterization_Trick.ipynb
Normal file
@@ -0,0 +1,423 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyOxO2/0DTH4n4zhC97qbagY",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "view-in-github",
|
||||
"colab_type": "text"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap17/17_2_Reparameterization_Trick.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# **Notebook 17.2: Reparameterization trick**\n",
|
||||
"\n",
|
||||
"This notebook investigates the reparameterization trick as described in section 17.7 of the book.\n",
|
||||
"\n",
|
||||
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
|
||||
"\n",
|
||||
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "t9vk9Elugvmi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "OLComQyvCIJ7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"The reparameterization trick computes the derivative of an expectation of a function $\\mbox{f}[x]$:\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\bigl[\\mbox{f}[x]\\bigr],\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"with respect to the parameters $\\boldsymbol\\phi$ of the distribution $Pr(x|\\boldsymbol\\phi)$ that the expectation is over.\n",
|
||||
"\n",
|
||||
"Let's consider a simple concrete example, where:\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"Pr(x|\\phi) = \\mbox{Norm}_{x}\\Bigl[\\mu, \\sigma^2\\Bigr]=\\mbox{Norm}_{x}\\Bigl[\\phi^3,(\\exp[\\phi])^2\\Bigr]\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"and\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\mbox{f}[x] = x^2+\\sin[x]\n",
|
||||
"\\end{equation}"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "paLz5RukZP1J"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Let's approximate this expecctation for a particular value of phi\n",
|
||||
"def compute_expectation(phi, n_samples):\n",
|
||||
" # TODO complete this function\n",
|
||||
" # 1. Compute the mean of the normal distribution, mu\n",
|
||||
" # 2. Compute the standard deviation of the normal distribution, sigma\n",
|
||||
" # 3. Draw n_samples samples using np.random.normal(mu, sigma, size=(n_samples, 1))\n",
|
||||
" # 4. Compute f[x] for each of these samples\n",
|
||||
" # 4. Approximate the expectation by taking the average of the values of f[x]\n",
|
||||
" # Replace this line\n",
|
||||
" expected_f_given_phi = 0\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return expected_f_given_phi"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "FdEbMnDBY0i9"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Set the seed so the random numbers are all the same\n",
|
||||
"np.random.seed(0)\n",
|
||||
"\n",
|
||||
"# Compute the expectation for two values of phi\n",
|
||||
"phi1 = 0.5\n",
|
||||
"n_samples = 10000000\n",
|
||||
"expected_f_given_phi1 = compute_expectation(phi1, n_samples)\n",
|
||||
"print(\"Your value: \", expected_f_given_phi1, \", True value: 2.7650801613563116\")\n",
|
||||
"\n",
|
||||
"phi2 = -0.1\n",
|
||||
"n_samples = 10000000\n",
|
||||
"expected_f_given_phi2 = compute_expectation(phi2, n_samples)\n",
|
||||
"print(\"Your value: \", expected_f_given_phi2, \", True value: 0.8176793102849222\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "FTh7LJ0llNJZ"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Le't plot this expectation as a function of phi"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "r5Hl2QkimWx9"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"phi_vals = np.arange(-1.5,1.5, 0.05)\n",
|
||||
"expected_vals = np.zeros_like(phi_vals)\n",
|
||||
"n_samples = 1000000\n",
|
||||
"for i in range(len(phi_vals)):\n",
|
||||
" expected_vals[i] = compute_expectation(phi_vals[i], n_samples)\n",
|
||||
"\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"ax.plot(phi_vals, expected_vals,'r-')\n",
|
||||
"ax.set_xlabel('Parameter $\\phi$')\n",
|
||||
"ax.set_ylabel('$\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "05XxVLJxmkER"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"It's this curve that we want to find the derivative of (so for example, we could run gradient descent and find the minimum.\n",
|
||||
"\n",
|
||||
"This is tricky though -- if you look at the computation that you performed, then there is a sampling step in the procedure (step 3). How do we compute the derivative of this?\n",
|
||||
"\n",
|
||||
"The answer is the reparameterization trick. We note that:\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\mbox{Norm}_{x}\\Bigl[\\mu, \\sigma^2\\Bigr]=\\mbox{Norm}_{x}\\Bigl[0, 1\\Bigr] \\times \\sigma + \\mu\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"and so:\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\mbox{Norm}_{x}\\Bigl[\\phi^3,(\\exp[\\phi])^2\\Bigr] = \\mbox{Norm}_{x}\\Bigl[0, 1\\Bigr] \\times \\exp[\\phi]+ \\phi^3\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"So, if we draw a sample $\\epsilon^*$ from $\\mbox{Norm}_{\\epsilon}[0, 1]$, then we can compute a sample $x^*$ as:\n",
|
||||
"\n",
|
||||
"\\begin{eqnarray*}\n",
|
||||
"x^* &=& \\epsilon^* \\times \\sigma + \\mu \\\\\n",
|
||||
"&=& \\epsilon^* \\times \\exp[\\phi]+ \\phi^3\n",
|
||||
"\\end{eqnarray*}"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "zTCykVeWqj_O"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def compute_df_dx_star(x_star):\n",
|
||||
" # TODO Compute this derivative (function defined at the top)\n",
|
||||
" # Replace this line:\n",
|
||||
" deriv = 0;\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return deriv\n",
|
||||
"\n",
|
||||
"def compute_dx_star_dphi(epsilon_star, phi):\n",
|
||||
" # TODO Compute this derivative\n",
|
||||
" # Replace this line:\n",
|
||||
" deriv = 0;\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return deriv\n",
|
||||
"\n",
|
||||
"def compute_derivative_of_expectation(phi, n_samples):\n",
|
||||
" # Generate the random values of epsilon\n",
|
||||
" epsilon_star= np.random.normal(size=(n_samples,1))\n",
|
||||
" # TODO -- write\n",
|
||||
" # 1. Compute dx*/dphi using the function defined above\n",
|
||||
" # 2. Compute x*\n",
|
||||
" # 3. Compute df/dx* using the function you wrote above\n",
|
||||
" # 4. Compute df/dphi = df/x* * dx*dphi\n",
|
||||
" # 5. Average the samples of df/dphi to get the expectation.\n",
|
||||
" # Replace this line:\n",
|
||||
" df_dphi = 0\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return df_dphi"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "w13HVpi9q8nF"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Set the seed so the random numbers are all the same\n",
|
||||
"np.random.seed(0)\n",
|
||||
"\n",
|
||||
"# Compute the expectation for two values of phi\n",
|
||||
"phi1 = 0.5\n",
|
||||
"n_samples = 10000000\n",
|
||||
"\n",
|
||||
"deriv = compute_derivative_of_expectation(phi1, n_samples)\n",
|
||||
"print(\"Your value: \", deriv, \", True value: 5.726338035051403\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ntQT4An79kAl"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"phi_vals = np.arange(-1.5,1.5, 0.05)\n",
|
||||
"deriv_vals = np.zeros_like(phi_vals)\n",
|
||||
"n_samples = 1000000\n",
|
||||
"for i in range(len(phi_vals)):\n",
|
||||
" deriv_vals[i] = compute_derivative_of_expectation(phi_vals[i], n_samples)\n",
|
||||
"\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"ax.plot(phi_vals, deriv_vals,'r-')\n",
|
||||
"ax.set_xlabel('Parameter $\\phi$')\n",
|
||||
"ax.set_ylabel('$\\partial/\\partial\\phi\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "t0Jqd_IN_lMU"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"This should look plausibly like the derivative of the function we plotted above!"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ASu4yKSwAEYI"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"The reparameterization trick computes the derivative of an expectation of a function $\\mbox{f}[x]$:\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\bigl[\\mbox{f}[x]\\bigr],\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"with respect to the parameters $\\boldsymbol\\phi$ of the distribution $Pr(x|\\boldsymbol\\phi)$ that the expectation is over. This derivative can also be computed as:\n",
|
||||
"\n",
|
||||
"\\begin{eqnarray}\n",
|
||||
"\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\bigl[\\mbox{f}[x]\\bigr] &=& \\mathbb{E}_{Pr(x|\\boldsymbol\\phi)}\\left[\\mbox{f}[x]\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\log\\bigl[ Pr(x|\\boldsymbol\\phi)\\bigr]\\right]\\nonumber \\\\\n",
|
||||
"&\\approx & \\frac{1}{I}\\sum_{i=1}^{I}\\mbox{f}[x_i]\\frac{\\partial}{\\partial \\boldsymbol\\phi} \\log\\bigl[ Pr(x_i|\\boldsymbol\\phi)\\bigr].\n",
|
||||
"\\end{eqnarray}\n",
|
||||
"\n",
|
||||
"This method is known as the REINFORCE algorithm or score function estimator. Problem 17.5 asks you to prove this relation. Let's use this method to compute the gradient and compare.\n",
|
||||
"\n",
|
||||
"Recall that the expression for a univariate Gaussian is:\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
" Pr(x|\\mu,\\sigma^2) = \\frac{1}{\\sqrt{2\\pi\\sigma^{2}}}\\exp\\left[-\\frac{(x-\\mu)^{2}}{2\\sigma^{2}}\\right].\n",
|
||||
"\\end{equation}\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "xoFR1wifc8-b"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def d_log_pr_x_given_phi(x,phi):\n",
|
||||
" # TODO -- fill in this function\n",
|
||||
" # Compute the derivative of log[Pr(x|phi)]\n",
|
||||
" # Replace this line:\n",
|
||||
" deriv =0;\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return deriv\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def compute_derivative_of_expectation_score_function(phi, n_samples):\n",
|
||||
" # TODO -- Compute this function\n",
|
||||
" # 1. Calculate mu from phi\n",
|
||||
" # 2. Calculate sigma from phi\n",
|
||||
" # 3. Generate n_sample random samples of x using np.random.normal\n",
|
||||
" # 4. Calculate f[x] for all of the samples\n",
|
||||
" # 5. Multiply f[x] by d_log_pr_x_given_phi\n",
|
||||
" # 6. Take the average of the samples\n",
|
||||
" # Replace this line:\n",
|
||||
" deriv = 0\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return deriv"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "4TUaxiWvASla"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Set the seed so the random numbers are all the same\n",
|
||||
"np.random.seed(0)\n",
|
||||
"\n",
|
||||
"# Compute the expectation for two values of phi\n",
|
||||
"phi1 = 0.5\n",
|
||||
"n_samples = 100000000\n",
|
||||
"\n",
|
||||
"deriv = compute_derivative_of_expectation_score_function(phi1, n_samples)\n",
|
||||
"print(\"Your value: \", deriv, \", True value: 5.724609927313369\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "0RSN32Rna_C_"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"phi_vals = np.arange(-1.5,1.5, 0.05)\n",
|
||||
"deriv_vals = np.zeros_like(phi_vals)\n",
|
||||
"n_samples = 1000000\n",
|
||||
"for i in range(len(phi_vals)):\n",
|
||||
" deriv_vals[i] = compute_derivative_of_expectation_score_function(phi_vals[i], n_samples)\n",
|
||||
"\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"ax.plot(phi_vals, deriv_vals,'r-')\n",
|
||||
"ax.set_xlabel('Parameter $\\phi$')\n",
|
||||
"ax.set_ylabel('$\\partial/\\partial\\phi\\mathbb{E}_{Pr(x|\\phi)}[f[x]]$')\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "EM_i5zoyElHR"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"This should look the same as the derivative that we computed with the reparameterization trick. So, is there any advantage to one way or the other? Let's compare the variances of the estimates\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "1TWBiUC7bQSw"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"n_estimate = 100\n",
|
||||
"n_sample = 1000\n",
|
||||
"phi = 0.3\n",
|
||||
"reparam_estimates = np.zeros((n_estimate,1))\n",
|
||||
"score_function_estimates = np.zeros((n_estimate,1))\n",
|
||||
"for i in range(n_estimate):\n",
|
||||
" reparam_estimates[i]= compute_derivative_of_expectation(phi, n_samples)\n",
|
||||
" score_function_estimates[i] = compute_derivative_of_expectation_score_function(phi, n_samples)\n",
|
||||
"\n",
|
||||
"print(\"Variance of reparameterization estimator\", np.var(reparam_estimates))\n",
|
||||
"print(\"Variance of score function estimator\", np.var(score_function_estimates))"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "vV_Jx5bCbQGs"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"The variance of the reparameterization estimator should be quite a bit lower than the score function estimator which is why it is preferred in this situation."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "d-0tntSYdKPR"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
496
Notebooks/Chap17/17_3_Importance_Sampling.ipynb
Normal file
496
Notebooks/Chap17/17_3_Importance_Sampling.ipynb
Normal file
@@ -0,0 +1,496 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyMvae+1cigwg2Htl6vt1Who",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "view-in-github",
|
||||
"colab_type": "text"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap17/17_3_Importance_Sampling.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# **Notebook 17.3: Importance sampling**\n",
|
||||
"\n",
|
||||
"This notebook investigates importance sampling as described in section 17.8.1 of the book.\n",
|
||||
"\n",
|
||||
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
|
||||
"\n",
|
||||
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "t9vk9Elugvmi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "OLComQyvCIJ7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's approximate the expectation\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\mathbb{E}_{y}\\Bigl[\\exp\\bigl[- (y-1)^4\\bigr]\\Bigr] = \\int \\exp\\bigl[- (y-1)^4\\bigr] Pr(y) dy,\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"where\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"Pr(y)=\\mbox{Norm}_y[0,1]\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"by drawing $I$ samples $y_i$ and using the formula:\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\mathbb{E}_{y}\\Bigl[\\exp\\bigl[- (y-1)^4\\bigr]\\Bigr] \\approx \\frac{1}{I} \\sum_{i=1}^I \\exp\\bigl[-(y-1)^4 \\bigr]\n",
|
||||
"\\end{equation}"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "f7a6xqKjkmvT"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def f(y):\n",
|
||||
" return np.exp(-(y-1) *(y-1) *(y-1) * (y-1))\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def pr_y(y):\n",
|
||||
" return (1/np.sqrt(2*np.pi)) * np.exp(-0.5 * y * y)\n",
|
||||
"\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"y = np.arange(-10,10,0.01)\n",
|
||||
"ax.plot(y, f(y), 'r-', label='f$[y]$');\n",
|
||||
"ax.plot(y, pr_y(y),'b-',label='$Pr(y)$')\n",
|
||||
"ax.set_xlabel(\"$y$\")\n",
|
||||
"ax.legend()\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "VjkzRr8o2ksg"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def compute_expectation(n_samples):\n",
|
||||
" # TODO -- compute this expectation\n",
|
||||
" # 1. Generate samples y_i using np.random.normal\n",
|
||||
" # 2. Approximate the expectation of f[y]\n",
|
||||
" # Replace this line\n",
|
||||
" expectation = 0\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return expectation"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "LGAKHjUJnWmy"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Set the seed so the random numbers are all the same\n",
|
||||
"np.random.seed(0)\n",
|
||||
"\n",
|
||||
"# Compute the expectation with a very large number of samples (good estimate)\n",
|
||||
"n_samples = 100000000\n",
|
||||
"expected_f= compute_expectation(n_samples)\n",
|
||||
"print(\"Your value: \", expected_f, \", True value: 0.43160702267383166\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "nmvixMqgodIP"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's investigate how the variance of this approximation decreases as we increase the number of samples $N$.\n",
|
||||
"\n",
|
||||
"\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Jr4UPcqmnXCS"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def compute_mean_variance(n_sample):\n",
|
||||
" n_estimate = 10000\n",
|
||||
" estimates = np.zeros((n_estimate,1))\n",
|
||||
" for i in range(n_estimate):\n",
|
||||
" estimates[i] = compute_expectation(n_sample.astype(int))\n",
|
||||
" return np.mean(estimates), np.var(estimates)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "yrDp1ILUo08j"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Compute the mean and variance for 1,2,... 20 samples\n",
|
||||
"n_sample_all = np.array([1.,2,3,4,5,6,7,8,9,10,15,20,25,30,45,50,60,70,80,90,100,150,200,250,300,350,400,450,500])\n",
|
||||
"mean_all = np.zeros_like(n_sample_all)\n",
|
||||
"variance_all = np.zeros_like(n_sample_all)\n",
|
||||
"for i in range(len(n_sample_all)):\n",
|
||||
" print(\"Computing mean and variance for expectation with %d samples\"%(n_sample_all[i]))\n",
|
||||
" mean_all[i],variance_all[i] = compute_mean_variance(n_sample_all[i])"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "BcUVsodtqdey"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"ax.semilogx(n_sample_all, mean_all,'r-',label='mean estimate')\n",
|
||||
"ax.fill_between(n_sample_all, mean_all-2*np.sqrt(variance_all), mean_all+2*np.sqrt(variance_all))\n",
|
||||
"ax.set_xlabel(\"Number of samples\")\n",
|
||||
"ax.set_ylabel(\"Mean of estimate\")\n",
|
||||
"ax.plot([0,500],[0.43160702267383166,0.43160702267383166],'k--',label='true value')\n",
|
||||
"ax.legend()\n",
|
||||
"plt.show()\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "feXmyk0krpUi"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"As you might expect, the more samples that we use to compute the approximate estimate, the lower the variance of the estimate."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "XTUpxFlSuOl7"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
" Now consider the function\n",
|
||||
" \\begin{equation}\n",
|
||||
" \\mbox{f}[y]= 20.446\\exp\\left[-(y-3)^4\\right],\n",
|
||||
" \\end{equation}\n",
|
||||
"\n",
|
||||
"which decreases rapidly as we move away from the position $y=4$."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "6hxsl3Pxo1TT"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def f2(y):\n",
|
||||
" return 20.446*np.exp(- (y-3) *(y-3) *(y-3) * (y-3))\n",
|
||||
"\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"y = np.arange(-10,10,0.01)\n",
|
||||
"ax.plot(y, f2(y), 'r-', label='f$[y]$');\n",
|
||||
"ax.plot(y, pr_y(y),'b-',label='$Pr(y)$')\n",
|
||||
"ax.set_xlabel(\"$y$\")\n",
|
||||
"ax.legend()\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "znydVPW7sL4P"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's again, compute the expectation:\n",
|
||||
"\n",
|
||||
"\\begin{eqnarray}\n",
|
||||
"\\mathbb{E}_{y}\\left[\\mbox{f}[y]\\right] &=& \\int \\mbox{f}[y] Pr(y) dy\\\\\n",
|
||||
"&\\approx& \\frac{1}{I} \\mbox{f}[y]\n",
|
||||
"\\end{eqnarray}\n",
|
||||
"\n",
|
||||
"where $Pr(y)=\\mbox{Norm}_y[0,1]$ by approximating with samples $y_{i}$.\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "G9Xxo0OJsIqD"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def compute_expectation2(n_samples):\n",
|
||||
" y = np.random.normal(size=(n_samples,1))\n",
|
||||
" expectation = np.mean(f2(y))\n",
|
||||
"\n",
|
||||
" return expectation"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "l8ZtmkA2vH4y"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Set the seed so the random numbers are all the same\n",
|
||||
"np.random.seed(0)\n",
|
||||
"\n",
|
||||
"# Compute the expectation with a very large number of samples (good estimate)\n",
|
||||
"n_samples = 100000000\n",
|
||||
"expected_f2= compute_expectation2(n_samples)\n",
|
||||
"print(\"Expected value: \", expected_f2)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "dfUQyJ-svZ6F"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"I deliberately chose this function, because it's expectation is roughly the same as for the previous function.\n",
|
||||
"\n",
|
||||
"Again, let's look at the mean and the variance of the estimates"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "2sVDqP0BvxqM"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def compute_mean_variance2(n_sample):\n",
|
||||
" n_estimate = 10000\n",
|
||||
" estimates = np.zeros((n_estimate,1))\n",
|
||||
" for i in range(n_estimate):\n",
|
||||
" estimates[i] = compute_expectation2(n_sample.astype(int))\n",
|
||||
" return np.mean(estimates), np.var(estimates)\n",
|
||||
"\n",
|
||||
"# Compute the variance for 1,2,... 20 samples\n",
|
||||
"mean_all2 = np.zeros_like(n_sample_all)\n",
|
||||
"variance_all2 = np.zeros_like(n_sample_all)\n",
|
||||
"for i in range(len(n_sample_all)):\n",
|
||||
" print(\"Computing variance for expectation with %d samples\"%(n_sample_all[i]))\n",
|
||||
" mean_all2[i], variance_all2[i] = compute_mean_variance2(n_sample_all[i])"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "mHnILRkOv0Ir"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"fig,ax1 = plt.subplots()\n",
|
||||
"ax1.semilogx(n_sample_all, mean_all,'r-',label='mean estimate')\n",
|
||||
"ax1.fill_between(n_sample_all, mean_all-2*np.sqrt(variance_all), mean_all+2*np.sqrt(variance_all))\n",
|
||||
"ax1.set_xlabel(\"Number of samples\")\n",
|
||||
"ax1.set_ylabel(\"Mean of estimate\")\n",
|
||||
"ax1.plot([1,500],[0.43160702267383166,0.43160702267383166],'k--',label='true value')\n",
|
||||
"ax1.set_ylim(-5,6)\n",
|
||||
"ax1.set_title(\"First function\")\n",
|
||||
"ax1.legend()\n",
|
||||
"\n",
|
||||
"fig2,ax2 = plt.subplots()\n",
|
||||
"ax2.semilogx(n_sample_all, mean_all2,'r-',label='mean estimate')\n",
|
||||
"ax2.fill_between(n_sample_all, mean_all2-2*np.sqrt(variance_all2), mean_all2+2*np.sqrt(variance_all2))\n",
|
||||
"ax2.set_xlabel(\"Number of samples\")\n",
|
||||
"ax2.set_ylabel(\"Mean of estimate\")\n",
|
||||
"ax2.plot([0,500],[0.43160428638892556,0.43160428638892556],'k--',label='true value')\n",
|
||||
"ax2.set_ylim(-5,6)\n",
|
||||
"ax2.set_title(\"Second function\")\n",
|
||||
"ax2.legend()\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "FkCX-hxxAnsw"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"You can see that the variance of the estimate of the second function is considerably worse than the estimate of the variance of estimate of the first function\n",
|
||||
"\n",
|
||||
"TODO: Think about why this is."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "EtBP6NeLwZqz"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
" Now let's repeat this experiment with the second function, but this time use importance sampling with auxiliary distribution:\n",
|
||||
"\n",
|
||||
" \\begin{equation}\n",
|
||||
" q(y)=\\mbox{Norm}_{y}[3,1]\n",
|
||||
" \\end{equation}\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "_wuF-NoQu1--"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def q_y(y):\n",
|
||||
" return (1/np.sqrt(2*np.pi)) * np.exp(-0.5 * (y-3) * (y-3))\n",
|
||||
"\n",
|
||||
"def compute_expectation2b(n_samples):\n",
|
||||
" # TODO -- complete this function\n",
|
||||
" # 1. Draw n_samples from auxiliary distribution\n",
|
||||
" # 2. Compute f[y] for those samples\n",
|
||||
" # 3. Scale the results by pr_y / q_y\n",
|
||||
" # 4. Compute the mean of these weighted samples\n",
|
||||
" # Replace this line\n",
|
||||
" expectation = 0\n",
|
||||
"\n",
|
||||
" return expectation"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "jPm0AVYVIDnn"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Set the seed so the random numbers are all the same\n",
|
||||
"np.random.seed(0)\n",
|
||||
"\n",
|
||||
"# Compute the expectation with a very large number of samples (good estimate)\n",
|
||||
"n_samples = 100000000\n",
|
||||
"expected_f2= compute_expectation2b(n_samples)\n",
|
||||
"print(\"Your value: \", expected_f2,\", True value: 0.43163734204459125 \")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "No2ByVvOM2yQ"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def compute_mean_variance2b(n_sample):\n",
|
||||
" n_estimate = 10000\n",
|
||||
" estimates = np.zeros((n_estimate,1))\n",
|
||||
" for i in range(n_estimate):\n",
|
||||
" estimates[i] = compute_expectation2b(n_sample.astype(int))\n",
|
||||
" return np.mean(estimates), np.var(estimates)\n",
|
||||
"\n",
|
||||
"# Compute the variance for 1,2,... 20 samples\n",
|
||||
"mean_all2b = np.zeros_like(n_sample_all)\n",
|
||||
"variance_all2b = np.zeros_like(n_sample_all)\n",
|
||||
"for i in range(len(n_sample_all)):\n",
|
||||
" print(\"Computing variance for expectation with %d samples\"%(n_sample_all[i]))\n",
|
||||
" mean_all2b[i], variance_all2b[i] = compute_mean_variance2b(n_sample_all[i])"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "6v8Jc7z4M3Mk"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"fig,ax1 = plt.subplots()\n",
|
||||
"ax1.semilogx(n_sample_all, mean_all,'r-',label='mean estimate')\n",
|
||||
"ax1.fill_between(n_sample_all, mean_all-2*np.sqrt(variance_all), mean_all+2*np.sqrt(variance_all))\n",
|
||||
"ax1.set_xlabel(\"Number of samples\")\n",
|
||||
"ax1.set_ylabel(\"Mean of estimate\")\n",
|
||||
"ax1.plot([1,500],[0.43160702267383166,0.43160702267383166],'k--',label='true value')\n",
|
||||
"ax1.set_ylim(-5,6)\n",
|
||||
"ax1.set_title(\"First function\")\n",
|
||||
"ax1.legend()\n",
|
||||
"\n",
|
||||
"fig2,ax2 = plt.subplots()\n",
|
||||
"ax2.semilogx(n_sample_all, mean_all2,'r-',label='mean estimate')\n",
|
||||
"ax2.fill_between(n_sample_all, mean_all2-2*np.sqrt(variance_all2), mean_all2+2*np.sqrt(variance_all2))\n",
|
||||
"ax2.set_xlabel(\"Number of samples\")\n",
|
||||
"ax2.set_ylabel(\"Mean of estimate\")\n",
|
||||
"ax2.plot([0,500],[0.43160428638892556,0.43160428638892556],'k--',label='true value')\n",
|
||||
"ax2.set_ylim(-5,6)\n",
|
||||
"ax2.set_title(\"Second function\")\n",
|
||||
"ax2.legend()\n",
|
||||
"plt.show()\n",
|
||||
"\n",
|
||||
"fig2,ax2 = plt.subplots()\n",
|
||||
"ax2.semilogx(n_sample_all, mean_all2b,'r-',label='mean estimate')\n",
|
||||
"ax2.fill_between(n_sample_all, mean_all2b-2*np.sqrt(variance_all2b), mean_all2b+2*np.sqrt(variance_all2b))\n",
|
||||
"ax2.set_xlabel(\"Number of samples\")\n",
|
||||
"ax2.set_ylabel(\"Mean of estimate\")\n",
|
||||
"ax2.plot([0,500],[ 0.43163734204459125, 0.43163734204459125],'k--',label='true value')\n",
|
||||
"ax2.set_ylim(-5,6)\n",
|
||||
"ax2.set_title(\"Second function with importance sampling\")\n",
|
||||
"ax2.legend()\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "C0beD4sNNM3L"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"You can see that the importance sampling technique has reduced the amount of variance for any given number of samples."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "y8rgge9MNiOc"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
471
Notebooks/Chap18/18_1_Diffusion_Encoder.ipynb
Normal file
471
Notebooks/Chap18/18_1_Diffusion_Encoder.ipynb
Normal file
@@ -0,0 +1,471 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyMpC8kgLnXx0XQBtwNAQ4jJ",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "view-in-github",
|
||||
"colab_type": "text"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_1_Diffusion_Encoder.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# **Notebook 18.1: Diffusion Encoder**\n",
|
||||
"\n",
|
||||
"This notebook investigates the diffusion encoder as described in section 18.2 of the book.\n",
|
||||
"\n",
|
||||
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
|
||||
"\n",
|
||||
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "t9vk9Elugvmi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from matplotlib.colors import ListedColormap\n",
|
||||
"from operator import itemgetter"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "OLComQyvCIJ7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"#Create pretty colormap as in book\n",
|
||||
"my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
|
||||
"my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
|
||||
"r = np.floor(my_colormap_vals_dec/(256*256))\n",
|
||||
"g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
|
||||
"b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
|
||||
"my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
|
||||
"my_colormap = ListedColormap(my_colormap_vals)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "4PM8bf6lO0VE"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Probability distribution for normal\n",
|
||||
"def norm_pdf(x, mu, sigma):\n",
|
||||
" return np.exp(-0.5 * (x-mu) * (x-mu) / (sigma * sigma)) / np.sqrt(2*np.pi*sigma*sigma)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ONGRaQscfIOo"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# True distribution is a mixture of four Gaussians\n",
|
||||
"class TrueDataDistribution:\n",
|
||||
" # Constructor initializes parameters\n",
|
||||
" def __init__(self):\n",
|
||||
" self.mu = [1.5, -0.216, 0.45, -1.875]\n",
|
||||
" self.sigma = [0.3, 0.15, 0.525, 0.075]\n",
|
||||
" self.w = [0.2, 0.3, 0.35, 0.15]\n",
|
||||
"\n",
|
||||
" # Return PDF\n",
|
||||
" def pdf(self, x):\n",
|
||||
" return(self.w[0] *norm_pdf(x,self.mu[0],self.sigma[0]) + self.w[1] *norm_pdf(x,self.mu[1],self.sigma[1]) + self.w[2] *norm_pdf(x,self.mu[2],self.sigma[2]) + self.w[3] *norm_pdf(x,self.mu[3],self.sigma[3]))\n",
|
||||
"\n",
|
||||
" # Draw samples\n",
|
||||
" def sample(self, n):\n",
|
||||
" hidden = np.random.choice(4, n, p=self.w)\n",
|
||||
" epsilon = np.random.normal(size=(n))\n",
|
||||
" mu_list = list(itemgetter(*hidden)(self.mu))\n",
|
||||
" sigma_list = list(itemgetter(*hidden)(self.sigma))\n",
|
||||
" return mu_list + sigma_list * epsilon"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "gZvG0MKhfY8Y"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Define ground truth probability distribution that we will model\n",
|
||||
"true_dist = TrueDataDistribution()\n",
|
||||
"# Let's visualize this\n",
|
||||
"x_vals = np.arange(-3,3,0.01)\n",
|
||||
"pr_x_true = true_dist.pdf(x_vals)\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"ax.plot(x_vals, pr_x_true, 'r-')\n",
|
||||
"ax.set_xlabel(\"$x$\")\n",
|
||||
"ax.set_ylabel(\"$Pr(x)$\")\n",
|
||||
"ax.set_ylim(0,1.0)\n",
|
||||
"ax.set_xlim(-3,3)\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "qXmej3TUuQyp"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's first implement the forward process"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "XHdtfRP47YLy"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Do one step of diffusion (equation 18.1)\n",
|
||||
"def diffuse_one_step(z_t_minus_1, beta_t):\n",
|
||||
" # TODO -- Implement this function\n",
|
||||
" # Use np.random.normal to generate the samples epsilon\n",
|
||||
" # Replace this line\n",
|
||||
" z_t = np.zeros_like(z_t_minus_1)\n",
|
||||
"\n",
|
||||
" return z_t"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "hkApJ2VJlQuk"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's run the diffusion process for a whole bunch of samples"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ECAUfHNi9NVW"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Generate some samples\n",
|
||||
"n_sample = 10000\n",
|
||||
"np.random.seed(6)\n",
|
||||
"x = true_dist.sample(n_sample)\n",
|
||||
"\n",
|
||||
"# Number of time steps\n",
|
||||
"T = 100\n",
|
||||
"# Noise schedule has same value at every time step\n",
|
||||
"beta = 0.01511\n",
|
||||
"\n",
|
||||
"# We'll store the diffused samples in an array\n",
|
||||
"samples = np.zeros((T+1, n_sample))\n",
|
||||
"samples[0,:] = x\n",
|
||||
"\n",
|
||||
"for t in range(T):\n",
|
||||
" samples[t+1,:] = diffuse_one_step(samples[t,:], beta)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "M-TY5w9Q8LYW"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's, plot the evolution of a few paths as in figure 18.2"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "jYrAW6tN-gJ4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"fig, ax = plt.subplots()\n",
|
||||
"t_vals = np.arange(0,101,1)\n",
|
||||
"ax.plot(samples[:,0],t_vals,'r-')\n",
|
||||
"ax.plot(samples[:,1],t_vals,'g-')\n",
|
||||
"ax.plot(samples[:,2],t_vals,'b-')\n",
|
||||
"ax.plot(samples[:,3],t_vals,'c-')\n",
|
||||
"ax.plot(samples[:,4],t_vals,'m-')\n",
|
||||
"ax.set_xlim([-3,3])\n",
|
||||
"ax.set_ylim([101, 0])\n",
|
||||
"ax.set_xlabel('value')\n",
|
||||
"ax.set_ylabel('z_{t}')\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "4XU6CDZC_kFo"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Notice that the samples have a tendencey to move toward the center. Now let's look at the histogram of the samples at each stage"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "SGTYGGevAktz"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def draw_hist(z_t,title=''):\n",
|
||||
" fig, ax = plt.subplots()\n",
|
||||
" fig.set_size_inches(8,2.5)\n",
|
||||
" plt.hist(z_t , bins=np.arange(-3,3, 0.1), density = True)\n",
|
||||
" ax.set_xlim([-3,3])\n",
|
||||
" ax.set_ylim([0,1.0])\n",
|
||||
" ax.set_title('title')\n",
|
||||
" plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "bn5E5NzL-evM"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"draw_hist(samples[0,:],'Original data')\n",
|
||||
"draw_hist(samples[5,:],'Time step 5')\n",
|
||||
"draw_hist(samples[10,:],'Time step 10')\n",
|
||||
"draw_hist(samples[20,:],'Time step 20')\n",
|
||||
"draw_hist(samples[40,:],'Time step 40')\n",
|
||||
"draw_hist(samples[80,:],'Time step 80')\n",
|
||||
"draw_hist(samples[100,:],'Time step 100')"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "pn_XD-EhBlwk"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"You can clearly see that as the diffusion process continues, the data becomes more Gaussian."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "skuLfGl5Czf4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's investigate the diffusion kernel as in figure 18.3 of the book.\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "s37CBSzzK7wh"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def diffusion_kernel(x, t, beta):\n",
|
||||
" # TODO -- write this function\n",
|
||||
" # Replace this line:\n",
|
||||
" dk_mean = 0.0 ; dk_std = 1.0\n",
|
||||
"\n",
|
||||
" return dk_mean, dk_std"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "vL62Iym0LEtY"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def draw_prob_dist(x_plot_vals, prob_dist, title=''):\n",
|
||||
" fig, ax = plt.subplots()\n",
|
||||
" fig.set_size_inches(8,2.5)\n",
|
||||
" ax.plot(x_plot_vals, prob_dist, 'b-')\n",
|
||||
" ax.set_xlim([-3,3])\n",
|
||||
" ax.set_ylim([0,1.0])\n",
|
||||
" ax.set_title(title)\n",
|
||||
" plt.show()\n",
|
||||
"\n",
|
||||
"def compute_and_plot_diffusion_kernels(x, T, beta, my_colormap):\n",
|
||||
" x_plot_vals = np.arange(-3,3,0.01)\n",
|
||||
" diffusion_kernels = np.zeros((T+1,len(x_plot_vals)))\n",
|
||||
" dk_mean_all = np.ones((T+1,1))*x\n",
|
||||
" dk_std_all = np.zeros((T+1,1))\n",
|
||||
" for t in range(T):\n",
|
||||
" dk_mean_all[t+1], dk_std_all[t+1] = diffusion_kernel(x,t+1,beta)\n",
|
||||
" diffusion_kernels[t+1,:] = norm_pdf(x_plot_vals, dk_mean_all[t+1], dk_std_all[t+1])\n",
|
||||
"\n",
|
||||
" samples = np.ones((T+1, 5))\n",
|
||||
" samples[0,:] = x\n",
|
||||
"\n",
|
||||
" for t in range(T):\n",
|
||||
" samples[t+1,:] = diffuse_one_step(samples[t,:], beta)\n",
|
||||
"\n",
|
||||
" fig, ax = plt.subplots()\n",
|
||||
" fig.set_size_inches(6,6)\n",
|
||||
"\n",
|
||||
" # Plot the image containing all the kernels\n",
|
||||
" plt.imshow(diffusion_kernels, cmap=my_colormap, interpolation='nearest')\n",
|
||||
"\n",
|
||||
" # Plot +/- 2 standard deviations\n",
|
||||
" ax.plot((dk_mean_all -2 * dk_std_all)/0.01 + len(x_plot_vals)/2, np.arange(0,101,1),'y--')\n",
|
||||
" ax.plot((dk_mean_all +2 * dk_std_all)/0.01 + len(x_plot_vals)/2, np.arange(0,101,1),'y--')\n",
|
||||
"\n",
|
||||
" # Plot a few trajectories\n",
|
||||
" ax.plot(samples[:,0]/0.01 + + len(x_plot_vals)/2, np.arange(0,101,1), 'r-')\n",
|
||||
" ax.plot(samples[:,1]/0.01 + + len(x_plot_vals)/2, np.arange(0,101,1), 'g-')\n",
|
||||
" ax.plot(samples[:,2]/0.01 + + len(x_plot_vals)/2, np.arange(0,101,1), 'b-')\n",
|
||||
" ax.plot(samples[:,3]/0.01 + + len(x_plot_vals)/2, np.arange(0,101,1), 'c-')\n",
|
||||
" ax.plot(samples[:,4]/0.01 + + len(x_plot_vals)/2, np.arange(0,101,1), 'm-')\n",
|
||||
"\n",
|
||||
" # Tidy up and plot\n",
|
||||
" ax.set_ylabel(\"$Pr(z_{t}|x)$\")\n",
|
||||
" ax.get_xaxis().set_visible(False)\n",
|
||||
" ax.set_xlim([0,601])\n",
|
||||
" ax.set_aspect(601/T)\n",
|
||||
" plt.show()\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" draw_prob_dist(x_plot_vals, diffusion_kernels[20,:],'$q(z_{20}|x)$')\n",
|
||||
" draw_prob_dist(x_plot_vals, diffusion_kernels[40,:],'$q(z_{40}|x)$')\n",
|
||||
" draw_prob_dist(x_plot_vals, diffusion_kernels[80,:],'$q(z_{80}|x)$')"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "KtP1KF8wMh8o"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"x = -2\n",
|
||||
"compute_and_plot_diffusion_kernels(x, T, beta, my_colormap)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "g8TcI5wtRQsx"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"TODO -- Run this for different version of $x$ and check that you understand how the graphs change"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "-RuN2lR28-hK"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Finally, let's estimate the marginal distributions empirically and visualize them as in figure 18.4 of the book. This is only tractable because the data is in one dimension and we know the original distribution.\n",
|
||||
"\n",
|
||||
"The marginal distribution at time t is the sum of the diffusion kernels for each position x, weighted by the probability of seeing that value of x in the true distribution."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "n-x6Whz2J_zy"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def diffusion_marginal(x_plot_vals, pr_x_true, t, beta):\n",
|
||||
" # If time is zero then marginal is just original distribution\n",
|
||||
" if t == 0:\n",
|
||||
" return pr_x_true\n",
|
||||
"\n",
|
||||
" # The thing we are computing\n",
|
||||
" marginal_at_time_t = np.zeros_like(pr_x_true);\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" # TODO Write ths function\n",
|
||||
" # 1. For each x (value in x_plot_vals):\n",
|
||||
" # 2. Compute the mean and variance of the diffusion kernel at time t\n",
|
||||
" # 3. Compute pdf of this Gaussian at every x_plot_val\n",
|
||||
" # 4. Weight Gaussian by probability at position x and by 0.01 to componensate for bin size\n",
|
||||
" # 5. Accumulate weighted Gaussian in marginal at time t.\n",
|
||||
" # 6. Multiply result by 0.01 to compensate for bin size\n",
|
||||
" # Replace this line:\n",
|
||||
" marginal_at_time_t = marginal_at_time_t\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return marginal_at_time_t"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "YzN5duYpg7C-"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"x_plot_vals = np.arange(-3,3,0.01)\n",
|
||||
"marginal_distributions = np.zeros((T+1,len(x_plot_vals)))\n",
|
||||
"\n",
|
||||
"for t in range(T+1):\n",
|
||||
" marginal_distributions[t,:] = diffusion_marginal(x_plot_vals, pr_x_true , t, beta)\n",
|
||||
"\n",
|
||||
"fig, ax = plt.subplots()\n",
|
||||
"fig.set_size_inches(6,6)\n",
|
||||
"\n",
|
||||
"# Plot the image containing all the kernels\n",
|
||||
"plt.imshow(marginal_distributions, cmap=my_colormap, interpolation='nearest')\n",
|
||||
"\n",
|
||||
"# Tidy up and plot\n",
|
||||
"ax.set_ylabel(\"$Pr(z_{t})$\")\n",
|
||||
"ax.get_xaxis().set_visible(False)\n",
|
||||
"ax.set_xlim([0,601])\n",
|
||||
"ax.set_aspect(601/T)\n",
|
||||
"plt.show()\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"draw_prob_dist(x_plot_vals, marginal_distributions[0,:],'$q(z_{0})$')\n",
|
||||
"draw_prob_dist(x_plot_vals, marginal_distributions[20,:],'$q(z_{20})$')\n",
|
||||
"draw_prob_dist(x_plot_vals, marginal_distributions[60,:],'$q(z_{60})$')"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "OgEU9sxjRaeO"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
}
|
||||
]
|
||||
}
|
||||
380
Notebooks/Chap18/18_2_1D_Diffusion_Model.ipynb
Normal file
380
Notebooks/Chap18/18_2_1D_Diffusion_Model.ipynb
Normal file
@@ -0,0 +1,380 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyM4DdZDGoP1xGst+Nn+rwvt",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "view-in-github",
|
||||
"colab_type": "text"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_2_1D_Diffusion_Model.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# **Notebook 18.2: 1D Diffusion Model**\n",
|
||||
"\n",
|
||||
"This notebook investigates the diffusion encoder as described in section 18.3 and 18.4 of the book.\n",
|
||||
"\n",
|
||||
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
|
||||
"\n",
|
||||
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "t9vk9Elugvmi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from matplotlib.colors import ListedColormap\n",
|
||||
"from operator import itemgetter\n",
|
||||
"from scipy import stats\n",
|
||||
"from IPython.display import display, clear_output"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "OLComQyvCIJ7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"#Create pretty colormap as in book\n",
|
||||
"my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
|
||||
"my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
|
||||
"r = np.floor(my_colormap_vals_dec/(256*256))\n",
|
||||
"g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
|
||||
"b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
|
||||
"my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
|
||||
"my_colormap = ListedColormap(my_colormap_vals)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "4PM8bf6lO0VE"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Probability distribution for normal\n",
|
||||
"def norm_pdf(x, mu, sigma):\n",
|
||||
" return np.exp(-0.5 * (x-mu) * (x-mu) / (sigma * sigma)) / np.sqrt(2*np.pi*sigma*sigma)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ONGRaQscfIOo"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# True distribution is a mixture of four Gaussians\n",
|
||||
"class TrueDataDistribution:\n",
|
||||
" # Constructor initializes parameters\n",
|
||||
" def __init__(self):\n",
|
||||
" self.mu = [1.5, -0.216, 0.45, -1.875]\n",
|
||||
" self.sigma = [0.3, 0.15, 0.525, 0.075]\n",
|
||||
" self.w = [0.2, 0.3, 0.35, 0.15]\n",
|
||||
"\n",
|
||||
" # Return PDF\n",
|
||||
" def pdf(self, x):\n",
|
||||
" return(self.w[0] *norm_pdf(x,self.mu[0],self.sigma[0]) + self.w[1] *norm_pdf(x,self.mu[1],self.sigma[1]) + self.w[2] *norm_pdf(x,self.mu[2],self.sigma[2]) + self.w[3] *norm_pdf(x,self.mu[3],self.sigma[3]))\n",
|
||||
"\n",
|
||||
" # Draw samples\n",
|
||||
" def sample(self, n):\n",
|
||||
" hidden = np.random.choice(4, n, p=self.w)\n",
|
||||
" epsilon = np.random.normal(size=(n))\n",
|
||||
" mu_list = list(itemgetter(*hidden)(self.mu))\n",
|
||||
" sigma_list = list(itemgetter(*hidden)(self.sigma))\n",
|
||||
" return mu_list + sigma_list * epsilon"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "gZvG0MKhfY8Y"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Define ground truth probability distribution that we will model\n",
|
||||
"true_dist = TrueDataDistribution()\n",
|
||||
"# Let's visualize this\n",
|
||||
"x_vals = np.arange(-3,3,0.01)\n",
|
||||
"pr_x_true = true_dist.pdf(x_vals)\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"fig.set_size_inches(8,2.5)\n",
|
||||
"ax.plot(x_vals, pr_x_true, 'r-')\n",
|
||||
"ax.set_xlabel(\"$x$\")\n",
|
||||
"ax.set_ylabel(\"$Pr(x)$\")\n",
|
||||
"ax.set_ylim(0,1.0)\n",
|
||||
"ax.set_xlim(-3,3)\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "iJu_uBiaeUVv"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"To train the model to describe this distribution, we'll need to generate pairs of samples drawn from $Pr(z_t|x)$ (diffusion kernel) and $q(z_{t-1}|z_{t},x)$ (equation 18.15).\n",
|
||||
"\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "DRHUG_41i4t_"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# The diffusion kernel returns the parameters of Pr(z_{t}|x)\n",
|
||||
"def diffusion_kernel(x, t, beta):\n",
|
||||
" alpha = np.power(1-beta,t)\n",
|
||||
" dk_mean = x * np.sqrt(alpha)\n",
|
||||
" dk_std = np.sqrt(1-alpha)\n",
|
||||
" return dk_mean, dk_std\n",
|
||||
"\n",
|
||||
"# Compute mean and variance q(z_{t-1}|z_{t},x)\n",
|
||||
"def conditional_diffusion_distribution(x,z_t,t,beta):\n",
|
||||
" # TODO -- Implement this function\n",
|
||||
" # Replace this line\n",
|
||||
" cd_mean = 0; cd_std = 1\n",
|
||||
"\n",
|
||||
" return cd_mean, cd_std\n",
|
||||
"\n",
|
||||
"def get_data_pairs(x_train,t,beta):\n",
|
||||
" # Find diffusion kernel for every x_train and draw samples\n",
|
||||
" dk_mean, dk_std = diffusion_kernel(x_train, t, beta)\n",
|
||||
" z_t = np.random.normal(size=x_train.shape) * dk_std + dk_mean\n",
|
||||
" # Find conditional diffusion distribution for each x_train, z pair and draw samlpes\n",
|
||||
" cd_mean, cd_std = conditional_diffusion_distribution(x_train,z_t,t,beta)\n",
|
||||
" if t == 1:\n",
|
||||
" z_tminus1 = x_train\n",
|
||||
" else:\n",
|
||||
" z_tminus1 = np.random.normal(size=x_train.shape) * cd_std + cd_mean\n",
|
||||
"\n",
|
||||
" return z_t, z_tminus1"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "x6B8t72Ukscd"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"We also need models $\\mbox{f}_t[z_{t},\\phi_{t}]$ that map from $z_{t}$ to the mean of the distribution at time $z_{t-1}$. We're just going to use a very hacky non-parametric model (basically a lookup table) that tells you the result based on the (quantized) input."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "aSG_4uA8_zZ-"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# This code is really ugly! Don't look too closely at it!\n",
|
||||
"# All you need to know is that it is a model that trains from pairs zt, zt_minus1\n",
|
||||
"# And can then predict zt\n",
|
||||
"class NonParametricModel():\n",
|
||||
" # Constructor initializes parameters\n",
|
||||
" def __init__(self):\n",
|
||||
"\n",
|
||||
" self.inc = 0.01\n",
|
||||
" self.max_val = 3.0\n",
|
||||
" self.model = []\n",
|
||||
"\n",
|
||||
" # Learns a model that predicts z_t_minus1 given z_t\n",
|
||||
" def train(self, zt, zt_minus1):\n",
|
||||
" zt = np.clip(zt,-self.max_val,self.max_val)\n",
|
||||
" zt_minus1 = np.clip(zt_minus1,-self.max_val,self.max_val)\n",
|
||||
" bins = np.arange(-self.max_val,self.max_val+self.inc,self.inc)\n",
|
||||
" numerator, *_ = stats.binned_statistic(zt, zt_minus1-zt, statistic='sum',bins=bins)\n",
|
||||
" denominator, *_ = stats.binned_statistic(zt, zt_minus1-zt, statistic='count',bins=bins)\n",
|
||||
" self.model = numerator / (denominator + 1)\n",
|
||||
"\n",
|
||||
" def predict(self, zt):\n",
|
||||
" bin_index = np.floor((zt+self.max_val)/self.inc)\n",
|
||||
" bin_index = np.clip(bin_index,0, len(self.model)-1).astype('uint32')\n",
|
||||
" return zt + self.model[bin_index]"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ZHViC0pL_yy5"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Sample data from distribution (this would usually be our collected training set)\n",
|
||||
"n_sample = 100000\n",
|
||||
"x_train = true_dist.sample(n_sample)\n",
|
||||
"\n",
|
||||
"# Define model parameters\n",
|
||||
"T = 100\n",
|
||||
"beta = 0.01511\n",
|
||||
"\n",
|
||||
"all_models = []\n",
|
||||
"for t in range(0,T):\n",
|
||||
" clear_output(wait=True)\n",
|
||||
" display(\"Training timestep %d\"%(t))\n",
|
||||
" zt,zt_minus1 = get_data_pairs(x_train,t+1,beta)\n",
|
||||
" all_models.append(NonParametricModel())\n",
|
||||
" # The model at index t maps data from z_{t+1} to z_{t}\n",
|
||||
" all_models[t].train(zt,zt_minus1)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "CzVFybWoBygu"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now that we've learned the model, let's draw some samples from it. We start at $z_{100}$ and use the model to predict $z_{99}$, then $z_{98}$ and so on until finally we get to $z_{1}$ and then $x$ (represented as $z_{0}$ here). We'll store all of the intermediate stages as well, so we can plot the trajectories. See equations 18.16."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ZPc9SEvtl14U"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def sample(model, T, sigma_t, n_samples):\n",
|
||||
" # Create the output array\n",
|
||||
" # Each row represents a time step, first row will be sampled data\n",
|
||||
" # Each column represents a different sample\n",
|
||||
" samples = np.zeros((T+1,n_samples))\n",
|
||||
"\n",
|
||||
" # TODO -- Initialize the samples z_{T} at samples[T,:] from standard normal distribution\n",
|
||||
" # Replace this line\n",
|
||||
" samples[T,:] = np.zeros((1,n_samples))\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" # For t=100...99..98... ...0\n",
|
||||
" for t in range(T,0,-1):\n",
|
||||
" clear_output(wait=True)\n",
|
||||
" display(\"Predicting z_{%d} from z_{%d}\"%(t-1,t))\n",
|
||||
" # TODO Predict samples[t-1,:] from samples[t,:] using the appropriate model\n",
|
||||
" # Replace this line:\n",
|
||||
" samples[t-1,:] = np.zeros((1,n_samples))\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" # If not the last time step\n",
|
||||
" if t>0:\n",
|
||||
" # TODO Add noise to the samples at z_t-1 we just generated with mean zero, standard deviation sigma_t\n",
|
||||
" # Replace this line\n",
|
||||
" samples[t-1,:] = samples[t-1,:]\n",
|
||||
"\n",
|
||||
" return samples"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "A-ZMFOvACIOw"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's run the diffusion process for a whole bunch of samples"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ECAUfHNi9NVW"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"sigma_t=0.12288\n",
|
||||
"n_samples = 100000\n",
|
||||
"samples = sample(all_models, T, sigma_t, n_samples)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Plot the data\n",
|
||||
"sampled_data = samples[0,:]\n",
|
||||
"bins = np.arange(-3,3.05,0.05)\n",
|
||||
"\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"fig.set_size_inches(8,2.5)\n",
|
||||
"ax.set_xlim([-3,3])\n",
|
||||
"plt.hist(sampled_data, bins=bins, density =True)\n",
|
||||
"ax.set_ylim(0, 0.8)\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "M-TY5w9Q8LYW"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's, plot the evolution of a few of the paths as in figure 18.7 (paths are from bottom to top now)."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "jYrAW6tN-gJ4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"fig, ax = plt.subplots()\n",
|
||||
"t_vals = np.arange(0,101,1)\n",
|
||||
"ax.plot(samples[:,0],t_vals,'r-')\n",
|
||||
"ax.plot(samples[:,1],t_vals,'g-')\n",
|
||||
"ax.plot(samples[:,2],t_vals,'b-')\n",
|
||||
"ax.plot(samples[:,3],t_vals,'c-')\n",
|
||||
"ax.plot(samples[:,4],t_vals,'m-')\n",
|
||||
"ax.set_xlim([-3,3])\n",
|
||||
"ax.set_ylim([101, 0])\n",
|
||||
"ax.set_xlabel('value')\n",
|
||||
"ax.set_ylabel('z_{t}')\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "4XU6CDZC_kFo"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Notice that the samples have a tendency to move from positions that are near the center at time 100 to positions that are high in the true probability distribution at time 0"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "SGTYGGevAktz"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
362
Notebooks/Chap18/18_3_Reparameterized_Model.ipynb
Normal file
362
Notebooks/Chap18/18_3_Reparameterized_Model.ipynb
Normal file
@@ -0,0 +1,362 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyNd+D0/IVWXtU2GKsofyk2d",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "view-in-github",
|
||||
"colab_type": "text"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_3_Reparameterized_Model.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# **Notebook 18.3: 1D Reparameterized model**\n",
|
||||
"\n",
|
||||
"This notebook investigates the reparameterized model as described in section 18.5 of the book and implements algorithms 18.1 and 18.2.\n",
|
||||
"\n",
|
||||
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
|
||||
"\n",
|
||||
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "t9vk9Elugvmi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from matplotlib.colors import ListedColormap\n",
|
||||
"from operator import itemgetter\n",
|
||||
"from scipy import stats\n",
|
||||
"from IPython.display import display, clear_output"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "OLComQyvCIJ7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"#Create pretty colormap as in book\n",
|
||||
"my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
|
||||
"my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
|
||||
"r = np.floor(my_colormap_vals_dec/(256*256))\n",
|
||||
"g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
|
||||
"b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
|
||||
"my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
|
||||
"my_colormap = ListedColormap(my_colormap_vals)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "4PM8bf6lO0VE"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Probability distribution for normal\n",
|
||||
"def norm_pdf(x, mu, sigma):\n",
|
||||
" return np.exp(-0.5 * (x-mu) * (x-mu) / (sigma * sigma)) / np.sqrt(2*np.pi*sigma*sigma)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ONGRaQscfIOo"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# True distribution is a mixture of four Gaussians\n",
|
||||
"class TrueDataDistribution:\n",
|
||||
" # Constructor initializes parameters\n",
|
||||
" def __init__(self):\n",
|
||||
" self.mu = [1.5, -0.216, 0.45, -1.875]\n",
|
||||
" self.sigma = [0.3, 0.15, 0.525, 0.075]\n",
|
||||
" self.w = [0.2, 0.3, 0.35, 0.15]\n",
|
||||
"\n",
|
||||
" # Return PDF\n",
|
||||
" def pdf(self, x):\n",
|
||||
" return(self.w[0] *norm_pdf(x,self.mu[0],self.sigma[0]) + self.w[1] *norm_pdf(x,self.mu[1],self.sigma[1]) + self.w[2] *norm_pdf(x,self.mu[2],self.sigma[2]) + self.w[3] *norm_pdf(x,self.mu[3],self.sigma[3]))\n",
|
||||
"\n",
|
||||
" # Draw samples\n",
|
||||
" def sample(self, n):\n",
|
||||
" hidden = np.random.choice(4, n, p=self.w)\n",
|
||||
" epsilon = np.random.normal(size=(n))\n",
|
||||
" mu_list = list(itemgetter(*hidden)(self.mu))\n",
|
||||
" sigma_list = list(itemgetter(*hidden)(self.sigma))\n",
|
||||
" return mu_list + sigma_list * epsilon"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "gZvG0MKhfY8Y"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Define ground truth probability distribution that we will model\n",
|
||||
"true_dist = TrueDataDistribution()\n",
|
||||
"# Let's visualize this\n",
|
||||
"x_vals = np.arange(-3,3,0.01)\n",
|
||||
"pr_x_true = true_dist.pdf(x_vals)\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"fig.set_size_inches(8,2.5)\n",
|
||||
"ax.plot(x_vals, pr_x_true, 'r-')\n",
|
||||
"ax.set_xlabel(\"$x$\")\n",
|
||||
"ax.set_ylabel(\"$Pr(x)$\")\n",
|
||||
"ax.set_ylim(0,1.0)\n",
|
||||
"ax.set_xlim(-3,3)\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "iJu_uBiaeUVv"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"To train the model to describe this distribution, we'll need to generate pairs of samples drawn from $Pr(z_t|x)$ (diffusion kernel) and $q(z_{t-1}|z_{t},x)$ (equation 18.15).\n",
|
||||
"\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "DRHUG_41i4t_"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Return z_t (the argument of g_{t}[] in the loss function in algorithm 18.1) and epsilon\n",
|
||||
"def get_data_pairs(x_train,t,beta):\n",
|
||||
" # TODO -- write this function\n",
|
||||
" # Replace these lines\n",
|
||||
" epsilon = np.ones_like(x_train)\n",
|
||||
" z_t = np.ones_like(x_train)\n",
|
||||
"\n",
|
||||
" return z_t, epsilon"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "x6B8t72Ukscd"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"We also need models $\\mbox{g}_t[z_{t},\\phi_{t}]$ that map from $z_{t}$ to the noise $\\epsilon$ that was added. We're just going to use a very hacky non-parametric model (basically a lookup table) that tells you the result based on the (quantized) input."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "aSG_4uA8_zZ-"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# This code is really ugly! Don't look too closely at it!\n",
|
||||
"# All you need to know is that it is a model that trains from pairs zt, zt_minus1\n",
|
||||
"# And can then predict zt\n",
|
||||
"class NonParametricModel():\n",
|
||||
" # Constructor initializes parameters\n",
|
||||
" def __init__(self):\n",
|
||||
"\n",
|
||||
" self.inc = 0.01\n",
|
||||
" self.max_val = 3.0\n",
|
||||
" self.model = []\n",
|
||||
"\n",
|
||||
" # Learns a model that predicts epsilon given z_t\n",
|
||||
" def train(self, zt, epsilon):\n",
|
||||
" zt = np.clip(zt,-self.max_val,self.max_val)\n",
|
||||
" epsilon = np.clip(epsilon,-self.max_val,self.max_val)\n",
|
||||
" bins = np.arange(-self.max_val,self.max_val+self.inc,self.inc)\n",
|
||||
" numerator, *_ = stats.binned_statistic(zt, epsilon, statistic='sum',bins=bins)\n",
|
||||
" denominator, *_ = stats.binned_statistic(zt, epsilon, statistic='count',bins=bins)\n",
|
||||
" self.model = numerator / (denominator + 1)\n",
|
||||
"\n",
|
||||
" def predict(self, zt):\n",
|
||||
" bin_index = np.floor((zt+self.max_val)/self.inc)\n",
|
||||
" bin_index = np.clip(bin_index,0, len(self.model)-1).astype('uint32')\n",
|
||||
" return self.model[bin_index]"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ZHViC0pL_yy5"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Sample data from distribution (this would usually be our collected training set)\n",
|
||||
"n_sample = 100000\n",
|
||||
"x_train = true_dist.sample(n_sample)\n",
|
||||
"\n",
|
||||
"# Define model parameters\n",
|
||||
"T = 100\n",
|
||||
"beta = 0.01511\n",
|
||||
"\n",
|
||||
"all_models = []\n",
|
||||
"for t in range(0,T):\n",
|
||||
" clear_output(wait=True)\n",
|
||||
" display(\"Training timestep %d\"%(t))\n",
|
||||
" zt,epsilon= get_data_pairs(x_train,t,beta)\n",
|
||||
" all_models.append(NonParametricModel())\n",
|
||||
" # The model at index t maps data from z_{t+1} to epsilon\n",
|
||||
" all_models[t].train(zt,epsilon)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "CzVFybWoBygu"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now that we've learned the model, let's draw some samples from it. We start at $z_{100}$ and use the model to predict $z_{99}$, then $z_{98}$ and so on until finally we get to $z_{1}$ and then $x$ (represented as $z_{0}$ here). We'll store all of the intermediate stages as well, so we can plot the trajectories. See algorithm 18.2"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ZPc9SEvtl14U"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def sample(model, T, sigma_t, n_samples):\n",
|
||||
" # Create the output array\n",
|
||||
" # Each row represents a time step, first row will be sampled data\n",
|
||||
" # Each column represents a different sample\n",
|
||||
" samples = np.zeros((T+1,n_samples))\n",
|
||||
"\n",
|
||||
" # TODO -- Initialize the samples z_{T} at samples[T,:] from standard normal distribution\n",
|
||||
" # Replace this line\n",
|
||||
" samples[T,:] = np.zeros((1,n_samples))\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" # For t=100...99..98... ...0\n",
|
||||
" for t in range(T,0,-1):\n",
|
||||
" clear_output(wait=True)\n",
|
||||
" display(\"Predicting z_{%d} from z_{%d}\"%(t-1,t))\n",
|
||||
" # TODO Predict samples[t-1,:] from samples[t,:] using the appropriate model\n",
|
||||
" # Replace this line:\n",
|
||||
" samples[t-1,:] = np.zeros((1,n_samples))\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" # If not the last time step\n",
|
||||
" if t>0:\n",
|
||||
" # TODO Add noise to the samples at z_t-1 we just generated with mean zero, standard deviation sigma_t\n",
|
||||
" # Replace this line\n",
|
||||
" samples[t-1,:] = samples[t-1,:]\n",
|
||||
"\n",
|
||||
" return samples"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "A-ZMFOvACIOw"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's run the diffusion process for a whole bunch of samples"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ECAUfHNi9NVW"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"sigma_t=0.12288\n",
|
||||
"n_samples = 100000\n",
|
||||
"samples = sample(all_models, T, sigma_t, n_samples)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Plot the data\n",
|
||||
"sampled_data = samples[0,:]\n",
|
||||
"bins = np.arange(-3,3.05,0.05)\n",
|
||||
"\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"fig.set_size_inches(8,2.5)\n",
|
||||
"ax.set_xlim([-3,3])\n",
|
||||
"plt.hist(sampled_data, bins=bins, density =True)\n",
|
||||
"ax.set_ylim(0, 0.8)\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "M-TY5w9Q8LYW"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's, plot the evolution of a few of the paths as in figure 18.7 (paths are from bottom to top now)."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "jYrAW6tN-gJ4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"fig, ax = plt.subplots()\n",
|
||||
"t_vals = np.arange(0,101,1)\n",
|
||||
"ax.plot(samples[:,0],t_vals,'r-')\n",
|
||||
"ax.plot(samples[:,1],t_vals,'g-')\n",
|
||||
"ax.plot(samples[:,2],t_vals,'b-')\n",
|
||||
"ax.plot(samples[:,3],t_vals,'c-')\n",
|
||||
"ax.plot(samples[:,4],t_vals,'m-')\n",
|
||||
"ax.set_xlim([-3,3])\n",
|
||||
"ax.set_ylim([101, 0])\n",
|
||||
"ax.set_xlabel('value')\n",
|
||||
"ax.set_ylabel('z_{t}')\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "4XU6CDZC_kFo"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Notice that the samples have a tendency to move from positions that are near the center at time 100 to positions that are high in the true probability distribution at time 0"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "SGTYGGevAktz"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
484
Notebooks/Chap18/18_4_Families_of_Diffusion_Models.ipynb
Normal file
484
Notebooks/Chap18/18_4_Families_of_Diffusion_Models.ipynb
Normal file
@@ -0,0 +1,484 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyNFSvISBXo/Z1l+onknF2Gw",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "view-in-github",
|
||||
"colab_type": "text"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap18/18_4_Families_of_Diffusion_Models.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# **Notebook 18.4: Families of diffusion models**\n",
|
||||
"\n",
|
||||
"This notebook investigates the reparameterized model as described in section 18.5 of the book and computers the results shown in figure 18.10c-f. These models are based on the paper \"Denoising diffusion implicit models\" which can be found [here](https://arxiv.org/pdf/2010.02502.pdf).\n",
|
||||
"\n",
|
||||
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
|
||||
"\n",
|
||||
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "t9vk9Elugvmi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from matplotlib.colors import ListedColormap\n",
|
||||
"from operator import itemgetter\n",
|
||||
"from scipy import stats\n",
|
||||
"from IPython.display import display, clear_output"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "OLComQyvCIJ7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"#Create pretty colormap as in book\n",
|
||||
"my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
|
||||
"my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
|
||||
"r = np.floor(my_colormap_vals_dec/(256*256))\n",
|
||||
"g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
|
||||
"b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
|
||||
"my_colormap_vals = np.vstack((r,g,b)).transpose()/255.0\n",
|
||||
"my_colormap = ListedColormap(my_colormap_vals)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "4PM8bf6lO0VE"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Probability distribution for normal\n",
|
||||
"def norm_pdf(x, mu, sigma):\n",
|
||||
" return np.exp(-0.5 * (x-mu) * (x-mu) / (sigma * sigma)) / np.sqrt(2*np.pi*sigma*sigma)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ONGRaQscfIOo"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# True distribution is a mixture of four Gaussians\n",
|
||||
"class TrueDataDistribution:\n",
|
||||
" # Constructor initializes parameters\n",
|
||||
" def __init__(self):\n",
|
||||
" self.mu = [1.5, -0.216, 0.45, -1.875]\n",
|
||||
" self.sigma = [0.3, 0.15, 0.525, 0.075]\n",
|
||||
" self.w = [0.2, 0.3, 0.35, 0.15]\n",
|
||||
"\n",
|
||||
" # Return PDF\n",
|
||||
" def pdf(self, x):\n",
|
||||
" return(self.w[0] *norm_pdf(x,self.mu[0],self.sigma[0]) + self.w[1] *norm_pdf(x,self.mu[1],self.sigma[1]) + self.w[2] *norm_pdf(x,self.mu[2],self.sigma[2]) + self.w[3] *norm_pdf(x,self.mu[3],self.sigma[3]))\n",
|
||||
"\n",
|
||||
" # Draw samples\n",
|
||||
" def sample(self, n):\n",
|
||||
" hidden = np.random.choice(4, n, p=self.w)\n",
|
||||
" epsilon = np.random.normal(size=(n))\n",
|
||||
" mu_list = list(itemgetter(*hidden)(self.mu))\n",
|
||||
" sigma_list = list(itemgetter(*hidden)(self.sigma))\n",
|
||||
" return mu_list + sigma_list * epsilon"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "gZvG0MKhfY8Y"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Define ground truth probability distribution that we will model\n",
|
||||
"true_dist = TrueDataDistribution()\n",
|
||||
"# Let's visualize this\n",
|
||||
"x_vals = np.arange(-3,3,0.01)\n",
|
||||
"pr_x_true = true_dist.pdf(x_vals)\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"fig.set_size_inches(8,2.5)\n",
|
||||
"ax.plot(x_vals, pr_x_true, 'r-')\n",
|
||||
"ax.set_xlabel(\"$x$\")\n",
|
||||
"ax.set_ylabel(\"$Pr(x)$\")\n",
|
||||
"ax.set_ylim(0,1.0)\n",
|
||||
"ax.set_xlim(-3,3)\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "iJu_uBiaeUVv"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"To train the model to describe this distribution, we'll need to generate pairs of samples drawn from $Pr(z_t|x)$ (diffusion kernel) and $q(z_{t-1}|z_{t},x)$ (equation 18.15).\n",
|
||||
"\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "DRHUG_41i4t_"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Return z_t (the argument of g_{t}[] in the loss function in algorithm 18.1) and epsilon\n",
|
||||
"def get_data_pairs(x_train,t,beta):\n",
|
||||
"\n",
|
||||
" epsilon = np.random.standard_normal(x_train.shape)\n",
|
||||
" alpha_t = np.power(1-beta,t)\n",
|
||||
" z_t = x_train * np.sqrt(alpha_t) + np.sqrt(1-alpha_t) * epsilon\n",
|
||||
"\n",
|
||||
" return z_t, epsilon"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "x6B8t72Ukscd"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"We also need models $\\mbox{g}_t[z_{t},\\phi_{t}]$ that map from $z_{t}$ to the noise $\\epsilon$ that was added. We're just going to use a very hacky non-parametric model (basically a lookup table) that tells you the result based on the (quantized) input."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "aSG_4uA8_zZ-"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# This code is really ugly! Don't look too closely at it!\n",
|
||||
"# All you need to know is that it is a model that trains from pairs zt, zt_minus1\n",
|
||||
"# And can then predict zt\n",
|
||||
"class NonParametricModel():\n",
|
||||
" # Constructor initializes parameters\n",
|
||||
" def __init__(self):\n",
|
||||
"\n",
|
||||
" self.inc = 0.01\n",
|
||||
" self.max_val = 3.0\n",
|
||||
" self.model = []\n",
|
||||
"\n",
|
||||
" # Learns a model that predicts epsilon given z_t\n",
|
||||
" def train(self, zt, epsilon):\n",
|
||||
" zt = np.clip(zt,-self.max_val,self.max_val)\n",
|
||||
" epsilon = np.clip(epsilon,-self.max_val,self.max_val)\n",
|
||||
" bins = np.arange(-self.max_val,self.max_val+self.inc,self.inc)\n",
|
||||
" numerator, *_ = stats.binned_statistic(zt, epsilon, statistic='sum',bins=bins)\n",
|
||||
" denominator, *_ = stats.binned_statistic(zt, epsilon, statistic='count',bins=bins)\n",
|
||||
" self.model = numerator / (denominator + 1)\n",
|
||||
"\n",
|
||||
" def predict(self, zt):\n",
|
||||
" bin_index = np.floor((zt+self.max_val)/self.inc)\n",
|
||||
" bin_index = np.clip(bin_index,0, len(self.model)-1).astype('uint32')\n",
|
||||
" return self.model[bin_index]"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ZHViC0pL_yy5"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Sample data from distribution (this would usually be our collected training set)\n",
|
||||
"n_sample = 100000\n",
|
||||
"x_train = true_dist.sample(n_sample)\n",
|
||||
"\n",
|
||||
"# Define model parameters\n",
|
||||
"T = 100\n",
|
||||
"beta = 0.01511\n",
|
||||
"\n",
|
||||
"all_models = []\n",
|
||||
"for t in range(0,T):\n",
|
||||
" clear_output(wait=True)\n",
|
||||
" display(\"Training timestep %d\"%(t))\n",
|
||||
" zt,epsilon= get_data_pairs(x_train,t,beta)\n",
|
||||
" all_models.append(NonParametricModel())\n",
|
||||
" # The model at index t maps data from z_{t+1} to epsilon\n",
|
||||
" all_models[t].train(zt,epsilon)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "CzVFybWoBygu"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now that we've learned the model, let's draw some samples from it. We start at $z_{100}$ and use the model to predict $z_{99}$, then $z_{98}$ and so on until finally we get to $z_{1}$ and then $x$ (represented as $z_{0}$ here). We'll store all of the intermediate stages as well, so we can plot the trajectories.\n",
|
||||
"\n",
|
||||
"This is the same model we learned last time. The whole point of this is that it is compatible with any forward process with the same diffusion kernel.\n",
|
||||
"\n",
|
||||
"One such model is the denoising diffusion implicit model, which has a sampling step:\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\mathbf{z}_{t-1} = \\sqrt{\\alpha_{t-1}}\\left(\\frac{\\mathbf{z}_{t}-\\sqrt{1-\\alpha_{t}}\\mbox{g}_t[\\mathbf{z}_{t},\\boldsymbol\\phi]}{\\sqrt{\\alpha_{t}}}\\right) + \\sqrt{1-\\alpha_{t-1}-\\sigma^2}\\mbox{g}_t[\\mathbf{z}_{t},\\boldsymbol\\phi]+\\sigma\\epsilon\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"(see equation 12 of the denoising [diffusion implicit models paper ](https://arxiv.org/pdf/2010.02502.pdf).\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ZPc9SEvtl14U"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def sample_ddim(model, T, sigma_t, n_samples):\n",
|
||||
" # Create the output array\n",
|
||||
" # Each row represents a time step, first row will be sampled data\n",
|
||||
" # Each column represents a different sample\n",
|
||||
" samples = np.zeros((T+1,n_samples))\n",
|
||||
" samples[T,:] = np.random.standard_normal(n_samples)\n",
|
||||
"\n",
|
||||
" # For t=100...99..98... ...0\n",
|
||||
" for t in range(T,0,-1):\n",
|
||||
" clear_output(wait=True)\n",
|
||||
" display(\"Predicting z_{%d} from z_{%d}\"%(t-1,t))\n",
|
||||
"\n",
|
||||
" alpha_t = np.power(1-beta,t+1)\n",
|
||||
" alpha_t_minus1 = np.power(1-beta,t)\n",
|
||||
"\n",
|
||||
" # TODO -- implement the DDIM sampling step\n",
|
||||
" # Note the final noise term is already added in the \"if\" statement below\n",
|
||||
" # Replace this line:\n",
|
||||
" samples[t-1,:] = samples[t-1,:]\n",
|
||||
"\n",
|
||||
" # If not the last time step\n",
|
||||
" if t>0:\n",
|
||||
" samples[t-1,:] = samples[t-1,:]+ np.random.standard_normal(n_samples) * sigma_t\n",
|
||||
" return samples"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "A-ZMFOvACIOw"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's run the diffusion process for a whole bunch of samples"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ECAUfHNi9NVW"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Now we'll set the noise to a MUCH smaller level\n",
|
||||
"sigma_t=0.001\n",
|
||||
"n_samples = 100000\n",
|
||||
"samples_low_noise = sample_ddim(all_models, T, sigma_t, n_samples)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Plot the data\n",
|
||||
"sampled_data = samples_low_noise[0,:]\n",
|
||||
"bins = np.arange(-3,3.05,0.05)\n",
|
||||
"\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"fig.set_size_inches(8,2.5)\n",
|
||||
"ax.set_xlim([-3,3])\n",
|
||||
"plt.hist(sampled_data, bins=bins, density =True)\n",
|
||||
"ax.set_ylim(0, 0.8)\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "M-TY5w9Q8LYW"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's, plot the evolution of a few of the paths as in figure 18.7 (paths are from bottom to top now)."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "jYrAW6tN-gJ4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"fig, ax = plt.subplots()\n",
|
||||
"t_vals = np.arange(0,101,1)\n",
|
||||
"ax.plot(samples_low_noise[:,0],t_vals,'r-')\n",
|
||||
"ax.plot(samples_low_noise[:,1],t_vals,'g-')\n",
|
||||
"ax.plot(samples_low_noise[:,2],t_vals,'b-')\n",
|
||||
"ax.plot(samples_low_noise[:,3],t_vals,'c-')\n",
|
||||
"ax.plot(samples_low_noise[:,4],t_vals,'m-')\n",
|
||||
"ax.set_xlim([-3,3])\n",
|
||||
"ax.set_ylim([101, 0])\n",
|
||||
"ax.set_xlabel('value')\n",
|
||||
"ax.set_ylabel('z_{t}')\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "4XU6CDZC_kFo"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"The samples have a tendency to move from positions that are near the center at time 100 to positions that are high in the true probability distribution at time 0"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "SGTYGGevAktz"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's now sample from the accelerated model, that requires fewer models. Again, we don't need to learn anything new -- this is just the reverse process that corresponds to a different forward process that is compatible with the same diffusion kernel.\n",
|
||||
"\n",
|
||||
"There's nothing to do here except read the code. It uses the same DDIM model as you just implemented in the previous step, but it jumps timesteps five at a time."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Z-LZp_fMXxRt"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def sample_accelerated(model, T, sigma_t, n_steps, n_samples):\n",
|
||||
" # Create the output array\n",
|
||||
" # Each row represents a sample (i.e. fewer than the time steps), first row will be sampled data\n",
|
||||
" # Each column represents a different sample\n",
|
||||
" samples = np.zeros((n_steps+1,n_samples))\n",
|
||||
" samples[n_steps,:] = np.random.standard_normal(n_samples)\n",
|
||||
"\n",
|
||||
" # For each sampling step\n",
|
||||
" for c_step in range(n_steps,0,-1):\n",
|
||||
" # Find the corresponding time step and previous time step\n",
|
||||
" t= int(T * c_step/n_steps)\n",
|
||||
" tminus1 = int(T * (c_step-1)/n_steps)\n",
|
||||
" display(\"Predicting z_{%d} from z_{%d}\"%(tminus1,t))\n",
|
||||
"\n",
|
||||
" alpha_t = np.power(1-beta,t+1)\n",
|
||||
" alpha_t_minus1 = np.power(1-beta,tminus1+1)\n",
|
||||
" epsilon_est = all_models[t-1].predict(samples[c_step,:])\n",
|
||||
"\n",
|
||||
" samples[c_step-1,:]=np.sqrt(alpha_t_minus1)*(samples[c_step,:]-np.sqrt(1-alpha_t) * epsilon_est)/np.sqrt(alpha_t) \\\n",
|
||||
" + np.sqrt(1-alpha_t_minus1 - sigma_t*sigma_t) * epsilon_est\n",
|
||||
" # If not the last time step\n",
|
||||
" if t>0:\n",
|
||||
" samples[c_step-1,:] = samples[c_step-1,:]+ np.random.standard_normal(n_samples) * sigma_t\n",
|
||||
" return samples"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "3Z0erjGbYj1u"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's draw a bunch of samples from the model"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "D3Sm_WYrcuED"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"sigma_t=0.11\n",
|
||||
"n_samples = 100000\n",
|
||||
"n_steps = 20 # i.e. sample 5 times as fast as before -- should be a divisor of 100\n",
|
||||
"samples_accelerated = sample_accelerated(all_models, T, sigma_t, n_steps, n_samples)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Plot the data\n",
|
||||
"sampled_data = samples_accelerated[0,:]\n",
|
||||
"bins = np.arange(-3,3.05,0.05)\n",
|
||||
"\n",
|
||||
"fig,ax = plt.subplots()\n",
|
||||
"fig.set_size_inches(8,2.5)\n",
|
||||
"ax.set_xlim([-3,3])\n",
|
||||
"plt.hist(sampled_data, bins=bins, density =True)\n",
|
||||
"ax.set_ylim(0, 0.9)\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "UB45c7VMcGy-"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"fig, ax = plt.subplots()\n",
|
||||
"step_increment = 100/ n_steps\n",
|
||||
"t_vals = np.arange(0,101,5)\n",
|
||||
"\n",
|
||||
"for i in range(len(t_vals)-1):\n",
|
||||
" ax.plot( (samples_accelerated[i,0],samples_accelerated[i+1,0]), (t_vals[i], t_vals[i+1]),'r.-')\n",
|
||||
" ax.plot( (samples_accelerated[i,1],samples_accelerated[i+1,1]), (t_vals[i], t_vals[i+1]),'g.-')\n",
|
||||
" ax.plot( (samples_accelerated[i,2],samples_accelerated[i+1,2]), (t_vals[i], t_vals[i+1]),'b.-')\n",
|
||||
" ax.plot( (samples_accelerated[i,3],samples_accelerated[i+1,3]), (t_vals[i], t_vals[i+1]),'c.-')\n",
|
||||
" ax.plot( (samples_accelerated[i,4],samples_accelerated[i+1,4]), (t_vals[i], t_vals[i+1]),'m.-')\n",
|
||||
"\n",
|
||||
"ax.set_xlim([-3,3])\n",
|
||||
"ax.set_ylim([101, 0])\n",
|
||||
"ax.set_xlabel('value')\n",
|
||||
"ax.set_ylabel('z_{t}')\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Luv-6w84c_qO"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [],
|
||||
"metadata": {
|
||||
"id": "LSJi72f0kw_e"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
}
|
||||
]
|
||||
}
|
||||
736
Notebooks/Chap19/19_1_Markov_Decision_Processes.ipynb
Normal file
736
Notebooks/Chap19/19_1_Markov_Decision_Processes.ipynb
Normal file
@@ -0,0 +1,736 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyPg3umHnqmIXX6jGe809Nxf",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "view-in-github",
|
||||
"colab_type": "text"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap19/19_1_Markov_Decision_Processes.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# **Notebook 19.1: Markov Decision Processes**\n",
|
||||
"\n",
|
||||
"This notebook investigates Markov decision processes as described in section 19.1 of the book.\n",
|
||||
"\n",
|
||||
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
|
||||
"\n",
|
||||
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "t9vk9Elugvmi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from PIL import Image"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "OLComQyvCIJ7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Get local copies of components of images\n",
|
||||
"!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Empty.png\n",
|
||||
"!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Hole.png\n",
|
||||
"!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Fish.png\n",
|
||||
"!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Penguin.png"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ZsvrUszPLyEG"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Ugly class that takes care of drawing pictures like in the book.\n",
|
||||
"# You can totally ignore this code!\n",
|
||||
"class DrawMDP:\n",
|
||||
" # Constructor initializes parameters\n",
|
||||
" def __init__(self, n_row, n_col):\n",
|
||||
" self.empty_image = np.asarray(Image.open('Empty.png'))\n",
|
||||
" self.hole_image = np.asarray(Image.open('Hole.png'))\n",
|
||||
" self.fish_image = np.asarray(Image.open('Fish.png'))\n",
|
||||
" self.penguin_image = np.asarray(Image.open('Penguin.png'))\n",
|
||||
" self.fig,self.ax = plt.subplots()\n",
|
||||
" self.n_row = n_row\n",
|
||||
" self.n_col = n_col\n",
|
||||
"\n",
|
||||
" my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
|
||||
" my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
|
||||
" r = np.floor(my_colormap_vals_dec/(256*256))\n",
|
||||
" g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
|
||||
" b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
|
||||
" self.colormap = np.vstack((r,g,b)).transpose()/255.0\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" def draw_text(self, text, row, col, position, color):\n",
|
||||
" if position == 'bc':\n",
|
||||
" self.ax.text( 83*col+41,83 * (row+1) -10, text, horizontalalignment=\"center\", color=color, fontweight='bold')\n",
|
||||
" if position == 'tl':\n",
|
||||
" self.ax.text( 83*col+5,83 * row +5, text, verticalalignment = 'top', horizontalalignment=\"left\", color=color, fontweight='bold')\n",
|
||||
"\n",
|
||||
" # Draws a set of states\n",
|
||||
" def draw_path(self, path, color1, color2):\n",
|
||||
" for i in range(len(path)-1):\n",
|
||||
" row_start = np.floor(path[i]/self.n_col)\n",
|
||||
" row_end = np.floor(path[i+1]/self.n_col)\n",
|
||||
" col_start = path[i] - row_start * self.n_col\n",
|
||||
" col_end = path[i+1] - row_end * self.n_col\n",
|
||||
"\n",
|
||||
" color_index = int(np.floor(255 * i/(len(path)-1.)))\n",
|
||||
" self.ax.plot([col_start * 83+41 + i, col_end * 83+41 + i ],[row_start * 83+41 + i, row_end * 83+41 + i ], color=(self.colormap[color_index,0],self.colormap[color_index,1],self.colormap[color_index,2]))\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" # Draw deterministic policy\n",
|
||||
" def draw_deterministic_policy(self,i, action):\n",
|
||||
" row = np.floor(i/self.n_col)\n",
|
||||
" col = i - row * self.n_col\n",
|
||||
" center_x = 83 * col + 41\n",
|
||||
" center_y = 83 * row + 41\n",
|
||||
" arrow_base_width = 10\n",
|
||||
" arrow_height = 15\n",
|
||||
" # Draw arrow pointing upward\n",
|
||||
" if action ==0:\n",
|
||||
" triangle_indices = np.array([[center_x, center_y-arrow_height/2],\n",
|
||||
" [center_x - arrow_base_width/2, center_y+arrow_height/2],\n",
|
||||
" [center_x + arrow_base_width/2, center_y+arrow_height/2]])\n",
|
||||
" # Draw arrow pointing right\n",
|
||||
" if action ==1:\n",
|
||||
" triangle_indices = np.array([[center_x + arrow_height/2, center_y],\n",
|
||||
" [center_x - arrow_height/2, center_y-arrow_base_width/2],\n",
|
||||
" [center_x - arrow_height/2, center_y+arrow_base_width/2]])\n",
|
||||
" # Draw arrow pointing downward\n",
|
||||
" if action ==2:\n",
|
||||
" triangle_indices = np.array([[center_x, center_y+arrow_height/2],\n",
|
||||
" [center_x - arrow_base_width/2, center_y-arrow_height/2],\n",
|
||||
" [center_x + arrow_base_width/2, center_y-arrow_height/2]])\n",
|
||||
" # Draw arrow pointing left\n",
|
||||
" if action ==3:\n",
|
||||
" triangle_indices = np.array([[center_x - arrow_height/2, center_y],\n",
|
||||
" [center_x + arrow_height/2, center_y-arrow_base_width/2],\n",
|
||||
" [center_x + arrow_height/2, center_y+arrow_base_width/2]])\n",
|
||||
" self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
|
||||
"\n",
|
||||
" # Draw stochastic policy\n",
|
||||
" def draw_stochastic_policy(self,i, action_probs):\n",
|
||||
" row = np.floor(i/self.n_col)\n",
|
||||
" col = i - row * self.n_col\n",
|
||||
" offset = 20\n",
|
||||
" # Draw arrow pointing upward\n",
|
||||
" center_x = 83 * col + 41\n",
|
||||
" center_y = 83 * row + 41 - offset\n",
|
||||
" arrow_base_width = 15 * action_probs[0]\n",
|
||||
" arrow_height = 20 * action_probs[0]\n",
|
||||
" triangle_indices = np.array([[center_x, center_y-arrow_height/2],\n",
|
||||
" [center_x - arrow_base_width/2, center_y+arrow_height/2],\n",
|
||||
" [center_x + arrow_base_width/2, center_y+arrow_height/2]])\n",
|
||||
" self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
|
||||
"\n",
|
||||
" # Draw arrow pointing right\n",
|
||||
" center_x = 83 * col + 41 + offset\n",
|
||||
" center_y = 83 * row + 41\n",
|
||||
" arrow_base_width = 15 * action_probs[1]\n",
|
||||
" arrow_height = 20 * action_probs[1]\n",
|
||||
" triangle_indices = np.array([[center_x + arrow_height/2, center_y],\n",
|
||||
" [center_x - arrow_height/2, center_y-arrow_base_width/2],\n",
|
||||
" [center_x - arrow_height/2, center_y+arrow_base_width/2]])\n",
|
||||
" self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
|
||||
"\n",
|
||||
" # Draw arrow pointing downward\n",
|
||||
" center_x = 83 * col + 41\n",
|
||||
" center_y = 83 * row + 41 +offset\n",
|
||||
" arrow_base_width = 15 * action_probs[2]\n",
|
||||
" arrow_height = 20 * action_probs[2]\n",
|
||||
" triangle_indices = np.array([[center_x, center_y+arrow_height/2],\n",
|
||||
" [center_x - arrow_base_width/2, center_y-arrow_height/2],\n",
|
||||
" [center_x + arrow_base_width/2, center_y-arrow_height/2]])\n",
|
||||
" self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
|
||||
"\n",
|
||||
" # Draw arrow pointing left\n",
|
||||
" center_x = 83 * col + 41 -offset\n",
|
||||
" center_y = 83 * row + 41\n",
|
||||
" arrow_base_width = 15 * action_probs[3]\n",
|
||||
" arrow_height = 20 * action_probs[3]\n",
|
||||
" triangle_indices = np.array([[center_x - arrow_height/2, center_y],\n",
|
||||
" [center_x + arrow_height/2, center_y-arrow_base_width/2],\n",
|
||||
" [center_x + arrow_height/2, center_y+arrow_base_width/2]])\n",
|
||||
" self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" def draw(self, layout, state, draw_state_index= False, rewards=None, policy=None, state_values=None, action_values=None,path1=None, path2 = None):\n",
|
||||
" # Construct the image\n",
|
||||
" image_out = np.zeros((self.n_row * 83, self.n_col * 83, 4),dtype='uint8')\n",
|
||||
" for c_row in range (self.n_row):\n",
|
||||
" for c_col in range(self.n_col):\n",
|
||||
" if layout[c_row * self.n_col + c_col]==0:\n",
|
||||
" image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.empty_image\n",
|
||||
" elif layout[c_row * self.n_col + c_col]==1:\n",
|
||||
" image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.hole_image\n",
|
||||
" else:\n",
|
||||
" image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.fish_image\n",
|
||||
" if state == c_row * self.n_col + c_col:\n",
|
||||
" image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.penguin_image\n",
|
||||
"\n",
|
||||
" # Draw the image\n",
|
||||
" plt.imshow(image_out)\n",
|
||||
" self.ax.get_xaxis().set_visible(False)\n",
|
||||
" self.ax.get_yaxis().set_visible(False)\n",
|
||||
" self.ax.spines['top'].set_visible(False)\n",
|
||||
" self.ax.spines['right'].set_visible(False)\n",
|
||||
" self.ax.spines['bottom'].set_visible(False)\n",
|
||||
" self.ax.spines['left'].set_visible(False)\n",
|
||||
"\n",
|
||||
" if draw_state_index:\n",
|
||||
" for c_cell in range(layout.size):\n",
|
||||
" self.draw_text(\"%d\"%(c_cell), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'tl','k')\n",
|
||||
"\n",
|
||||
" # Draw the policy as triangles\n",
|
||||
" if policy is not None:\n",
|
||||
" # If the policy is deterministic\n",
|
||||
" if len(policy) == len(layout):\n",
|
||||
" for i in range(len(layout)):\n",
|
||||
" self.draw_deterministic_policy(i, policy[i])\n",
|
||||
" # Else it is stochastic\n",
|
||||
" else:\n",
|
||||
" for i in range(len(layout)):\n",
|
||||
" self.draw_stochastic_policy(i,policy[:,i])\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" if path1 is not None:\n",
|
||||
" # self.draw_path(path1, np.array([0.81, 0.51, 0.38]), np.array([1.0, 0.2, 0.5]))\n",
|
||||
" self.draw_path(path1, np.array([1.0, 0.0, 0.0]), np.array([0.0, 1.0, 1.0]))\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Gq1HfJsHN3SB"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Let's draw the initial situation with the penguin in top right\n",
|
||||
"n_rows = 4; n_cols = 4\n",
|
||||
"layout = np.zeros(n_rows * n_cols)\n",
|
||||
"initial_state = 0\n",
|
||||
"mdp_drawer = DrawMDP(n_rows, n_cols)\n",
|
||||
"mdp_drawer.draw(layout, state = initial_state, draw_state_index = True)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "eBQ7lTpJQBSe"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Note that the states are indexed from 0 rather than 1 as in the book to make\n",
|
||||
"the code neater."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "P7P40UyMunKb"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Define the state probabilities\n",
|
||||
"transition_probabilities = np.array( \\\n",
|
||||
"[[0.00 , 0.33, 0.00, 0.00, 0.33, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.50 , 0.00, 0.33, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.33, 0.00, 0.50, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.33, 0.00, 0.00, 0.00, 0.00, 0.33, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.50 , 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.33, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.34, 0.00, 0.00, 0.33, 0.00, 0.25, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.34, 0.00, 0.00, 0.25, 0.00, 0.33, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.50, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.33, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.34, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.33, 0.00, 0.25, 0.00, 0.00, 0.33, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.25, 0.00, 0.33, 0.00, 0.00, 0.33, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.34, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.50 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.34, 0.00, 0.00, 0.00, 0.00, 0.33, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.50, 0.00, 0.33, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.34, 0.00, 0.50 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.34, 0.00, 0.00, 0.34, 0.00 ],\n",
|
||||
"])\n",
|
||||
"initial_state = 0"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "wgFcIi4YQJWI"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Define a step from the Markov process"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "axllRDDuDDLS"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def markov_process_step(state, transition_probabilities):\n",
|
||||
" # TODO -- update the state according to the appropriate transition probabilities\n",
|
||||
" # One way to do this is to use np.random.choice\n",
|
||||
" # Replace this line:\n",
|
||||
" new_state = 0\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return new_state"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "FrSZrS67sdbN"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Run the Markov process for 10 steps and visualise the results"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "uTj7rN6LDFXd"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"np.random.seed(0)\n",
|
||||
"T = 10\n",
|
||||
"states = np.zeros(T, dtype='uint8')\n",
|
||||
"states[0] = 0\n",
|
||||
"for t in range(T-1):\n",
|
||||
" states[t+1] = markov_process_step(states[t], transition_probabilities)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"print(\"Your States:\", states)\n",
|
||||
"print(\"True States: [ 0 4 8 9 10 9 10 9 13 14]\")\n",
|
||||
"mdp_drawer = DrawMDP(n_rows, n_cols)\n",
|
||||
"mdp_drawer.draw(layout, state = states[0], path1=states, draw_state_index = True)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "lRIdjagCwP62"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Define a Markov one step of a reward process."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "QLyjyBjjDMin"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def markov_reward_process_step(state, transition_probabilities, reward_structure):\n",
|
||||
"\n",
|
||||
" # TODO -- write this function\n",
|
||||
" # Update the state. Return a reward of +1 if the Penguin lands on the fish\n",
|
||||
" # or zero otherwise.\n",
|
||||
" # Replace this line\n",
|
||||
" new_state = 0; reward = 0\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return new_state, reward"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "YPHSJRKx-pgO"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Run the Markov reward process for 10 steps and visualise the results"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "AIz8QEiRFoCm"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Set up the reward structure so it matches figure 19.2\n",
|
||||
"reward_structure = np.zeros((16,1))\n",
|
||||
"reward_structure[3] = 1; reward_structure[8] = 1; reward_structure[10] = 1\n",
|
||||
"\n",
|
||||
"# Initialize random numbers\n",
|
||||
"np.random.seed(0)\n",
|
||||
"T = 10\n",
|
||||
"# Set up the states, so the fish are in the same positions as figure 19.2\n",
|
||||
"states = np.zeros(T, dtype='uint8')\n",
|
||||
"rewards = np.zeros(T, dtype='uint8')\n",
|
||||
"\n",
|
||||
"states[0] = 0\n",
|
||||
"for t in range(T-1):\n",
|
||||
" states[t+1],rewards[t+1] = markov_reward_process_step(states[t], transition_probabilities, reward_structure)\n",
|
||||
"\n",
|
||||
"print(\"Your States:\", states)\n",
|
||||
"print(\"Your Rewards:\", rewards)\n",
|
||||
"print(\"True Rewards: [0 0 1 0 1 0 1 0 0 0]\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Draw the figure\n",
|
||||
"layout = np.zeros(n_rows * n_cols)\n",
|
||||
"layout[3] = 2; layout[8] = 2 ; layout[10] = 2\n",
|
||||
"mdp_drawer = DrawMDP(n_rows, n_cols)\n",
|
||||
"mdp_drawer.draw(layout, state = states[0], path1=states, draw_state_index = True)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "0p1gCpGoFn4M"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's calculate the return -- the sum of discounted future rewards"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "lyz47NWrITfj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def calculate_return(rewards, gamma):\n",
|
||||
" # TODO -- you write this function\n",
|
||||
" # It should compute one return for the start of the sequence (i.e. G_1)\n",
|
||||
" # Replace this line\n",
|
||||
" return_val = 0.0\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return return_val"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "4fEuBRPnFm_N"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"gamma = 0.9\n",
|
||||
"for t in range(len(states)):\n",
|
||||
" print(\"Return at time %d = %3.3f\"%(t, calculate_return(rewards[t:],gamma)))\n",
|
||||
"\n",
|
||||
"# Reality check!\n",
|
||||
"print(\"True return at time 0: 1.998\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "o19lQgM3JrOz"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's define the state transition function $Pr(s_{t+1}|s_{t},a)$ in full where $a$ is the actions. Here $a=0$ means try to go upward, $a=1$, right, $a=2$ down and $a=3$ right. However, the ice is slippery, so we don't always go the direction we want to.\n",
|
||||
"\n",
|
||||
"Note that as for the states, we've indexed the actions from zero (unlike in the book, so they map to the indices of arrays better)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Fhc6DzZNOjiC"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"transition_probabilities_given_action1 = np.array(\\\n",
|
||||
"[[0.00 , 0.33, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.50 , 0.00, 0.33, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.33, 0.00, 0.50, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.33, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.50 , 0.00, 0.00, 0.00, 0.00, 0.17, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.34, 0.00, 0.00, 0.25, 0.00, 0.17, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.34, 0.00, 0.00, 0.17, 0.00, 0.25, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.50, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.17, 0.00, 0.00, 0.75, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.25, 0.00, 0.17, 0.00, 0.00, 0.50, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.17, 0.00, 0.25, 0.00, 0.00, 0.50, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.75 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.25, 0.00, 0.25, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.25, 0.00, 0.25 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.25, 0.00 ],\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"transition_probabilities_given_action2 = np.array(\\\n",
|
||||
"[[0.00 , 0.25, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.75 , 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.50, 0.00, 0.50, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.33, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.25 , 0.00, 0.00, 0.00, 0.00, 0.17, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.25, 0.00, 0.00, 0.50, 0.00, 0.17, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.25, 0.00, 0.00, 0.50, 0.00, 0.33, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.50, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.33, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.17, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.50, 0.00, 0.17, 0.00, 0.00, 0.25, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.50, 0.00, 0.33, 0.00, 0.00, 0.25, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.34, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.50 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.75, 0.00, 0.25, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.50, 0.00, 0.50 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.34, 0.00, 0.00, 0.50, 0.00 ],\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"transition_probabilities_given_action3 = np.array(\\\n",
|
||||
"[[0.00 , 0.25, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.25 , 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.25, 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.75 , 0.00, 0.00, 0.00, 0.00, 0.17, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.50, 0.00, 0.00, 0.25, 0.00, 0.17, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.50, 0.00, 0.00, 0.16, 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.75, 0.00, 0.00, 0.16, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.17, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.25, 0.00, 0.17, 0.00, 0.00, 0.33, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.16, 0.00, 0.25, 0.00, 0.00, 0.33, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.16, 0.00, 0.00, 0.00, 0.00, 0.50 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.33, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.50, 0.00, 0.33, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.34, 0.00, 0.50 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.34, 0.00 ],\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"transition_probabilities_given_action4 = np.array(\\\n",
|
||||
"[[0.00 , 0.25, 0.00, 0.00, 0.33, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.50 , 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.50, 0.00, 0.75, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.50 , 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.33, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.25, 0.00, 0.00, 0.33, 0.00, 0.50, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.50, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.34, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.33, 0.00, 0.50, 0.00, 0.00, 0.25, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.17, 0.00, 0.50, 0.00, 0.00, 0.25, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.25 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.34, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.50, 0.00, 0.50, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.25, 0.00, 0.75 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.25, 0.00 ],\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"# Store all of these in a three dimension array\n",
|
||||
"# Pr(s_{t+1}=2|s_{t}=1, a_{t}=3] is stored at position [2,1,3]\n",
|
||||
"transition_probabilities_given_action = np.concatenate((np.expand_dims(transition_probabilities_given_action1,2),\n",
|
||||
" np.expand_dims(transition_probabilities_given_action2,2),\n",
|
||||
" np.expand_dims(transition_probabilities_given_action3,2),\n",
|
||||
" np.expand_dims(transition_probabilities_given_action4,2)),axis=2)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "l7rT78BbOgTi"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Now we need a policy. Let's start with the deterministic policy in figure 19.5a:\n",
|
||||
"policy = [2,2,1,1, 2,1,1,1, 1,1,0,2, 1,0,1,1]\n",
|
||||
"\n",
|
||||
"# Let's draw the policy first\n",
|
||||
"layout = np.zeros(n_rows * n_cols)\n",
|
||||
"layout[15] = 2\n",
|
||||
"mdp_drawer = DrawMDP(n_rows, n_cols)\n",
|
||||
"mdp_drawer.draw(layout, state = states[0], policy = policy, draw_state_index = True)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "8jWhDlkaKj7Q"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def markov_decision_process_step_deterministic(state, transition_probabilities_given_action, reward_structure, policy):\n",
|
||||
" # TODO -- complete this function.\n",
|
||||
" # For each state, theres is a corresponding action.\n",
|
||||
" # Draw the next state based on the current state and that action\n",
|
||||
" # and calculate the reward\n",
|
||||
" # Replace this line:\n",
|
||||
" new_state = 0; reward = 0;\n",
|
||||
"\n",
|
||||
" return new_state, reward\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "dueNbS2SUVUK"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Set up the reward structure so it matches figure 19.2\n",
|
||||
"reward_structure = np.zeros((16,1))\n",
|
||||
"reward_structure[15] = 1\n",
|
||||
"\n",
|
||||
"# Initialize random number seed\n",
|
||||
"np.random.seed(3)\n",
|
||||
"T = 10\n",
|
||||
"# Set up the states, so the fish are in the same positions as figure 19.5\n",
|
||||
"states = np.zeros(T, dtype='uint8')\n",
|
||||
"rewards = np.zeros(T, dtype='uint8')\n",
|
||||
"\n",
|
||||
"states[0] = 0\n",
|
||||
"for t in range(T-1):\n",
|
||||
" states[t+1],rewards[t+1] = markov_decision_process_step_deterministic(states[t], transition_probabilities_given_action, reward_structure, policy)\n",
|
||||
"\n",
|
||||
"print(\"Your States:\", states)\n",
|
||||
"print(\"True States: [ 0 4 8 9 13 14 15 11 7 3]\")\n",
|
||||
"print(\"Your Rewards:\", rewards)\n",
|
||||
"print(\"True Rewards: [0 0 0 0 0 0 1 0 0 0]\")\n",
|
||||
"\n",
|
||||
"mdp_drawer = DrawMDP(n_rows, n_cols)\n",
|
||||
"mdp_drawer.draw(layout, state = states[0], path1=states, policy = policy, draw_state_index = True)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "4Du5aUfd2Lci"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"You can see that the Penguin usually follows the policy, (heads in the direction of the cyan arrows (when it can). But sometimes, the penguin \"slips\" to a different neighboring state\n",
|
||||
"\n",
|
||||
"Now let's investigate a stochastic policy"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "bLEd8xug33b-"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"np.random.seed(0)\n",
|
||||
"# Let's now choose a random policy. We'll generate a set of random numbers and pass\n",
|
||||
"# them through a softmax function\n",
|
||||
"stochastic_policy = np.random.normal(size=(4,n_rows*n_cols))\n",
|
||||
"stochastic_policy = np.exp(stochastic_policy) / (np.ones((4,1))@ np.expand_dims(np.sum(np.exp(stochastic_policy), axis=0),0))\n",
|
||||
"np.set_printoptions(precision=2)\n",
|
||||
"print(stochastic_policy)\n",
|
||||
"\n",
|
||||
"# Let's draw the policy first\n",
|
||||
"layout = np.zeros(n_rows * n_cols)\n",
|
||||
"layout[15] = 2\n",
|
||||
"mdp_drawer = DrawMDP(n_rows, n_cols)\n",
|
||||
"mdp_drawer.draw(layout, state = states[0], path1=states, policy = stochastic_policy, draw_state_index = True)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "o7T0b3tyilDc"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def markov_decision_process_step_stochastic(state, transition_probabilities_given_action, reward_structure, stochastic_policy):\n",
|
||||
" # TODO -- complete this function.\n",
|
||||
" # For each state, theres is a corresponding distribution over actions\n",
|
||||
" # Draw a sample from that distribution to get the action\n",
|
||||
" # Draw the next state based on the current state and that action\n",
|
||||
" # and calculate the reward\n",
|
||||
" # Replace this line:\n",
|
||||
" new_state = 0; reward = 0;action = 0\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" return new_state, reward, action"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "T68mTZSe6A3w"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Set up the reward structure so it matches figure 19.2\n",
|
||||
"reward_structure = np.zeros((16,1))\n",
|
||||
"reward_structure[15] = 1\n",
|
||||
"\n",
|
||||
"# Initialize random number seed\n",
|
||||
"np.random.seed(0)\n",
|
||||
"T = 10\n",
|
||||
"# Set up the states, so the fish are in the same positions as figure 19.5\n",
|
||||
"states = np.zeros(T, dtype='uint8')\n",
|
||||
"rewards = np.zeros(T, dtype='uint8')\n",
|
||||
"actions = np.zeros(T-1, dtype='uint8')\n",
|
||||
"\n",
|
||||
"states[0] = 0\n",
|
||||
"for t in range(T-1):\n",
|
||||
" states[t+1],rewards[t+1],actions[t] = markov_decision_process_step_stochastic(states[t], transition_probabilities_given_action, reward_structure, stochastic_policy)\n",
|
||||
"\n",
|
||||
"print(\"Actions\", actions)\n",
|
||||
"print(\"Your States:\", states)\n",
|
||||
"print(\"Your Rewards:\", rewards)\n",
|
||||
"\n",
|
||||
"mdp_drawer = DrawMDP(n_rows, n_cols)\n",
|
||||
"mdp_drawer.draw(layout, state = states[0], path1=states, policy = stochastic_policy, draw_state_index = True)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "hMRVYX2HtqMg"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
}
|
||||
]
|
||||
}
|
||||
769
Notebooks/Chap19/19_2_Dynamic_Programming.ipynb
Normal file
769
Notebooks/Chap19/19_2_Dynamic_Programming.ipynb
Normal file
File diff suppressed because one or more lines are too long
826
Notebooks/Chap19/19_3_Monte_Carlo_Methods.ipynb
Normal file
826
Notebooks/Chap19/19_3_Monte_Carlo_Methods.ipynb
Normal file
File diff suppressed because one or more lines are too long
521
Notebooks/Chap19/19_4_Temporal_Difference_Methods.ipynb
Normal file
521
Notebooks/Chap19/19_4_Temporal_Difference_Methods.ipynb
Normal file
@@ -0,0 +1,521 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyNEAhORON7DFN1dZMhDK/PO",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "view-in-github",
|
||||
"colab_type": "text"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap19/19_4_Temporal_Difference_Methods.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# **Notebook 19.4: Temporal difference methods**\n",
|
||||
"\n",
|
||||
"This notebook investigates temporal differnece methods for tabular reinforcement learning as described in section 19.3.3 of the book\n",
|
||||
"\n",
|
||||
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
|
||||
"\n",
|
||||
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "t9vk9Elugvmi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from PIL import Image"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "OLComQyvCIJ7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Get local copies of components of images\n",
|
||||
"!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Empty.png\n",
|
||||
"!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Hole.png\n",
|
||||
"!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Fish.png\n",
|
||||
"!wget https://raw.githubusercontent.com/udlbook/udlbook/main/Notebooks/Chap19/Penguin.png"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ZsvrUszPLyEG"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Ugly class that takes care of drawing pictures like in the book.\n",
|
||||
"# You can totally ignore this code!\n",
|
||||
"class DrawMDP:\n",
|
||||
" # Constructor initializes parameters\n",
|
||||
" def __init__(self, n_row, n_col):\n",
|
||||
" self.empty_image = np.asarray(Image.open('Empty.png'))\n",
|
||||
" self.hole_image = np.asarray(Image.open('Hole.png'))\n",
|
||||
" self.fish_image = np.asarray(Image.open('Fish.png'))\n",
|
||||
" self.penguin_image = np.asarray(Image.open('Penguin.png'))\n",
|
||||
" self.fig,self.ax = plt.subplots()\n",
|
||||
" self.n_row = n_row\n",
|
||||
" self.n_col = n_col\n",
|
||||
"\n",
|
||||
" my_colormap_vals_hex =('2a0902', '2b0a03', '2c0b04', '2d0c05', '2e0c06', '2f0d07', '300d08', '310e09', '320f0a', '330f0b', '34100b', '35110c', '36110d', '37120e', '38120f', '39130f', '3a1410', '3b1411', '3c1511', '3d1612', '3e1613', '3f1713', '401714', '411814', '421915', '431915', '451a16', '461b16', '471b17', '481c17', '491d18', '4a1d18', '4b1e19', '4c1f19', '4d1f1a', '4e201b', '50211b', '51211c', '52221c', '53231d', '54231d', '55241e', '56251e', '57261f', '58261f', '592720', '5b2821', '5c2821', '5d2922', '5e2a22', '5f2b23', '602b23', '612c24', '622d25', '632e25', '652e26', '662f26', '673027', '683027', '693128', '6a3229', '6b3329', '6c342a', '6d342a', '6f352b', '70362c', '71372c', '72372d', '73382e', '74392e', '753a2f', '763a2f', '773b30', '783c31', '7a3d31', '7b3e32', '7c3e33', '7d3f33', '7e4034', '7f4134', '804235', '814236', '824336', '834437', '854538', '864638', '874739', '88473a', '89483a', '8a493b', '8b4a3c', '8c4b3c', '8d4c3d', '8e4c3e', '8f4d3f', '904e3f', '924f40', '935041', '945141', '955242', '965343', '975343', '985444', '995545', '9a5646', '9b5746', '9c5847', '9d5948', '9e5a49', '9f5a49', 'a05b4a', 'a15c4b', 'a35d4b', 'a45e4c', 'a55f4d', 'a6604e', 'a7614e', 'a8624f', 'a96350', 'aa6451', 'ab6552', 'ac6552', 'ad6653', 'ae6754', 'af6855', 'b06955', 'b16a56', 'b26b57', 'b36c58', 'b46d59', 'b56e59', 'b66f5a', 'b7705b', 'b8715c', 'b9725d', 'ba735d', 'bb745e', 'bc755f', 'bd7660', 'be7761', 'bf7862', 'c07962', 'c17a63', 'c27b64', 'c27c65', 'c37d66', 'c47e67', 'c57f68', 'c68068', 'c78169', 'c8826a', 'c9836b', 'ca846c', 'cb856d', 'cc866e', 'cd876f', 'ce886f', 'ce8970', 'cf8a71', 'd08b72', 'd18c73', 'd28d74', 'd38e75', 'd48f76', 'd59077', 'd59178', 'd69279', 'd7937a', 'd8957b', 'd9967b', 'da977c', 'da987d', 'db997e', 'dc9a7f', 'dd9b80', 'de9c81', 'de9d82', 'df9e83', 'e09f84', 'e1a185', 'e2a286', 'e2a387', 'e3a488', 'e4a589', 'e5a68a', 'e5a78b', 'e6a88c', 'e7aa8d', 'e7ab8e', 'e8ac8f', 'e9ad90', 'eaae91', 'eaaf92', 'ebb093', 'ecb295', 'ecb396', 'edb497', 'eeb598', 'eeb699', 'efb79a', 'efb99b', 'f0ba9c', 'f1bb9d', 'f1bc9e', 'f2bd9f', 'f2bfa1', 'f3c0a2', 'f3c1a3', 'f4c2a4', 'f5c3a5', 'f5c5a6', 'f6c6a7', 'f6c7a8', 'f7c8aa', 'f7c9ab', 'f8cbac', 'f8ccad', 'f8cdae', 'f9ceb0', 'f9d0b1', 'fad1b2', 'fad2b3', 'fbd3b4', 'fbd5b6', 'fbd6b7', 'fcd7b8', 'fcd8b9', 'fcdaba', 'fddbbc', 'fddcbd', 'fddebe', 'fddfbf', 'fee0c1', 'fee1c2', 'fee3c3', 'fee4c5', 'ffe5c6', 'ffe7c7', 'ffe8c9', 'ffe9ca', 'ffebcb', 'ffeccd', 'ffedce', 'ffefcf', 'fff0d1', 'fff2d2', 'fff3d3', 'fff4d5', 'fff6d6', 'fff7d8', 'fff8d9', 'fffada', 'fffbdc', 'fffcdd', 'fffedf', 'ffffe0')\n",
|
||||
" my_colormap_vals_dec = np.array([int(element,base=16) for element in my_colormap_vals_hex])\n",
|
||||
" r = np.floor(my_colormap_vals_dec/(256*256))\n",
|
||||
" g = np.floor((my_colormap_vals_dec - r *256 *256)/256)\n",
|
||||
" b = np.floor(my_colormap_vals_dec - r * 256 *256 - g * 256)\n",
|
||||
" self.colormap = np.vstack((r,g,b)).transpose()/255.0\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" def draw_text(self, text, row, col, position, color):\n",
|
||||
" if position == 'bc':\n",
|
||||
" self.ax.text( 83*col+41,83 * (row+1) -5, text, horizontalalignment=\"center\", color=color, fontweight='bold')\n",
|
||||
" if position == 'tc':\n",
|
||||
" self.ax.text( 83*col+41,83 * (row) +10, text, horizontalalignment=\"center\", color=color, fontweight='bold')\n",
|
||||
" if position == 'lc':\n",
|
||||
" self.ax.text( 83*col+2,83 * (row) +41, text, verticalalignment=\"center\", color=color, fontweight='bold', rotation=90)\n",
|
||||
" if position == 'rc':\n",
|
||||
" self.ax.text( 83*(col+1)-5,83 * (row) +41, text, horizontalalignment=\"right\", verticalalignment=\"center\", color=color, fontweight='bold', rotation=-90)\n",
|
||||
" if position == 'tl':\n",
|
||||
" self.ax.text( 83*col+5,83 * row +5, text, verticalalignment = 'top', horizontalalignment=\"left\", color=color, fontweight='bold')\n",
|
||||
" if position == 'tr':\n",
|
||||
" self.ax.text( 83*(col+1)-5, 83 * row +5, text, verticalalignment = 'top', horizontalalignment=\"right\", color=color, fontweight='bold')\n",
|
||||
"\n",
|
||||
" # Draws a set of states\n",
|
||||
" def draw_path(self, path, color1, color2):\n",
|
||||
" for i in range(len(path)-1):\n",
|
||||
" row_start = np.floor(path[i]/self.n_col)\n",
|
||||
" row_end = np.floor(path[i+1]/self.n_col)\n",
|
||||
" col_start = path[i] - row_start * self.n_col\n",
|
||||
" col_end = path[i+1] - row_end * self.n_col\n",
|
||||
"\n",
|
||||
" color_index = int(np.floor(255 * i/(len(path)-1.)))\n",
|
||||
" self.ax.plot([col_start * 83+41 + i, col_end * 83+41 + i ],[row_start * 83+41 + i, row_end * 83+41 + i ], color=(self.colormap[color_index,0],self.colormap[color_index,1],self.colormap[color_index,2]))\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" # Draw deterministic policy\n",
|
||||
" def draw_deterministic_policy(self,i, action):\n",
|
||||
" row = np.floor(i/self.n_col)\n",
|
||||
" col = i - row * self.n_col\n",
|
||||
" center_x = 83 * col + 41\n",
|
||||
" center_y = 83 * row + 41\n",
|
||||
" arrow_base_width = 10\n",
|
||||
" arrow_height = 15\n",
|
||||
" # Draw arrow pointing upward\n",
|
||||
" if action ==0:\n",
|
||||
" triangle_indices = np.array([[center_x, center_y-arrow_height/2],\n",
|
||||
" [center_x - arrow_base_width/2, center_y+arrow_height/2],\n",
|
||||
" [center_x + arrow_base_width/2, center_y+arrow_height/2]])\n",
|
||||
" # Draw arrow pointing right\n",
|
||||
" if action ==1:\n",
|
||||
" triangle_indices = np.array([[center_x + arrow_height/2, center_y],\n",
|
||||
" [center_x - arrow_height/2, center_y-arrow_base_width/2],\n",
|
||||
" [center_x - arrow_height/2, center_y+arrow_base_width/2]])\n",
|
||||
" # Draw arrow pointing downward\n",
|
||||
" if action ==2:\n",
|
||||
" triangle_indices = np.array([[center_x, center_y+arrow_height/2],\n",
|
||||
" [center_x - arrow_base_width/2, center_y-arrow_height/2],\n",
|
||||
" [center_x + arrow_base_width/2, center_y-arrow_height/2]])\n",
|
||||
" # Draw arrow pointing left\n",
|
||||
" if action ==3:\n",
|
||||
" triangle_indices = np.array([[center_x - arrow_height/2, center_y],\n",
|
||||
" [center_x + arrow_height/2, center_y-arrow_base_width/2],\n",
|
||||
" [center_x + arrow_height/2, center_y+arrow_base_width/2]])\n",
|
||||
" self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
|
||||
"\n",
|
||||
" # Draw stochastic policy\n",
|
||||
" def draw_stochastic_policy(self,i, action_probs):\n",
|
||||
" row = np.floor(i/self.n_col)\n",
|
||||
" col = i - row * self.n_col\n",
|
||||
" offset = 20\n",
|
||||
" # Draw arrow pointing upward\n",
|
||||
" center_x = 83 * col + 41\n",
|
||||
" center_y = 83 * row + 41 - offset\n",
|
||||
" arrow_base_width = 15 * action_probs[0]\n",
|
||||
" arrow_height = 20 * action_probs[0]\n",
|
||||
" triangle_indices = np.array([[center_x, center_y-arrow_height/2],\n",
|
||||
" [center_x - arrow_base_width/2, center_y+arrow_height/2],\n",
|
||||
" [center_x + arrow_base_width/2, center_y+arrow_height/2]])\n",
|
||||
" self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
|
||||
"\n",
|
||||
" # Draw arrow pointing right\n",
|
||||
" center_x = 83 * col + 41 + offset\n",
|
||||
" center_y = 83 * row + 41\n",
|
||||
" arrow_base_width = 15 * action_probs[1]\n",
|
||||
" arrow_height = 20 * action_probs[1]\n",
|
||||
" triangle_indices = np.array([[center_x + arrow_height/2, center_y],\n",
|
||||
" [center_x - arrow_height/2, center_y-arrow_base_width/2],\n",
|
||||
" [center_x - arrow_height/2, center_y+arrow_base_width/2]])\n",
|
||||
" self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
|
||||
"\n",
|
||||
" # Draw arrow pointing downward\n",
|
||||
" center_x = 83 * col + 41\n",
|
||||
" center_y = 83 * row + 41 +offset\n",
|
||||
" arrow_base_width = 15 * action_probs[2]\n",
|
||||
" arrow_height = 20 * action_probs[2]\n",
|
||||
" triangle_indices = np.array([[center_x, center_y+arrow_height/2],\n",
|
||||
" [center_x - arrow_base_width/2, center_y-arrow_height/2],\n",
|
||||
" [center_x + arrow_base_width/2, center_y-arrow_height/2]])\n",
|
||||
" self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
|
||||
"\n",
|
||||
" # Draw arrow pointing left\n",
|
||||
" center_x = 83 * col + 41 -offset\n",
|
||||
" center_y = 83 * row + 41\n",
|
||||
" arrow_base_width = 15 * action_probs[3]\n",
|
||||
" arrow_height = 20 * action_probs[3]\n",
|
||||
" triangle_indices = np.array([[center_x - arrow_height/2, center_y],\n",
|
||||
" [center_x + arrow_height/2, center_y-arrow_base_width/2],\n",
|
||||
" [center_x + arrow_height/2, center_y+arrow_base_width/2]])\n",
|
||||
" self.ax.fill(triangle_indices[:,0], triangle_indices[:,1],facecolor='cyan', edgecolor='darkcyan', linewidth=1)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" def draw(self, layout, state=None, draw_state_index= False, rewards=None, policy=None, state_values=None, state_action_values=None,path1=None, path2 = None):\n",
|
||||
" # Construct the image\n",
|
||||
" image_out = np.zeros((self.n_row * 83, self.n_col * 83, 4),dtype='uint8')\n",
|
||||
" for c_row in range (self.n_row):\n",
|
||||
" for c_col in range(self.n_col):\n",
|
||||
" if layout[c_row * self.n_col + c_col]==0:\n",
|
||||
" image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.empty_image\n",
|
||||
" elif layout[c_row * self.n_col + c_col]==1:\n",
|
||||
" image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.hole_image\n",
|
||||
" else:\n",
|
||||
" image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.fish_image\n",
|
||||
" if state is not None and state == c_row * self.n_col + c_col:\n",
|
||||
" image_out[c_row*83:c_row*83+83, c_col*83:c_col*83+83,:] = self.penguin_image\n",
|
||||
"\n",
|
||||
" # Draw the image\n",
|
||||
" plt.imshow(image_out)\n",
|
||||
" self.ax.get_xaxis().set_visible(False)\n",
|
||||
" self.ax.get_yaxis().set_visible(False)\n",
|
||||
" self.ax.spines['top'].set_visible(False)\n",
|
||||
" self.ax.spines['right'].set_visible(False)\n",
|
||||
" self.ax.spines['bottom'].set_visible(False)\n",
|
||||
" self.ax.spines['left'].set_visible(False)\n",
|
||||
"\n",
|
||||
" if draw_state_index:\n",
|
||||
" for c_cell in range(layout.size):\n",
|
||||
" self.draw_text(\"%d\"%(c_cell), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'tl','k')\n",
|
||||
"\n",
|
||||
" # Draw the policy as triangles\n",
|
||||
" if policy is not None:\n",
|
||||
" # If the policy is deterministic\n",
|
||||
" if len(policy) == len(layout):\n",
|
||||
" for i in range(len(layout)):\n",
|
||||
" self.draw_deterministic_policy(i, policy[i])\n",
|
||||
" # Else it is stochastic\n",
|
||||
" else:\n",
|
||||
" for i in range(len(layout)):\n",
|
||||
" self.draw_stochastic_policy(i,policy[:,i])\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" if path1 is not None:\n",
|
||||
" self.draw_path(path1, np.array([1.0, 0.0, 0.0]), np.array([0.0, 1.0, 1.0]))\n",
|
||||
"\n",
|
||||
" if rewards is not None:\n",
|
||||
" for c_cell in range(layout.size):\n",
|
||||
" self.draw_text(\"%d\"%(rewards[c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'tr','r')\n",
|
||||
"\n",
|
||||
" if state_values is not None:\n",
|
||||
" for c_cell in range(layout.size):\n",
|
||||
" self.draw_text(\"%2.2f\"%(state_values[c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'bc','black')\n",
|
||||
"\n",
|
||||
" if state_action_values is not None:\n",
|
||||
" for c_cell in range(layout.size):\n",
|
||||
" self.draw_text(\"%2.2f\"%(state_action_values[0, c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'tc','black')\n",
|
||||
" self.draw_text(\"%2.2f\"%(state_action_values[1, c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'rc','black')\n",
|
||||
" self.draw_text(\"%2.2f\"%(state_action_values[2, c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'bc','black')\n",
|
||||
" self.draw_text(\"%2.2f\"%(state_action_values[3, c_cell]), np.floor(c_cell/self.n_col), c_cell-np.floor(c_cell/self.n_col)*self.n_col,'lc','black')\n",
|
||||
"\n",
|
||||
" plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Gq1HfJsHN3SB"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# We're going to work on the problem depicted in figure 19.10a\n",
|
||||
"n_rows = 4; n_cols = 4\n",
|
||||
"layout = np.zeros(n_rows * n_cols)\n",
|
||||
"reward_structure = np.zeros(n_rows * n_cols)\n",
|
||||
"layout[9] = 1 ; reward_structure[9] = -2\n",
|
||||
"layout[10] = 1; reward_structure[10] = -2\n",
|
||||
"layout[14] = 1; reward_structure[14] = -2\n",
|
||||
"layout[15] = 2; reward_structure[15] = 3\n",
|
||||
"initial_state = 0\n",
|
||||
"mdp_drawer = DrawMDP(n_rows, n_cols)\n",
|
||||
"mdp_drawer.draw(layout, state = initial_state, rewards=reward_structure, draw_state_index = True)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "eBQ7lTpJQBSe"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"For clarity, the black numbers are the state number and the red numbers are the reward for being in that state. Note that the states are indexed from 0 rather than 1 as in the book to make the code neater."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "6Vku6v_se2IG"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's define the state transition function $Pr(s_{t+1}|s_{t},a)$ in full where $a$ is the actions. Here $a=0$ means try to go upward, $a=1$, right, $a=2$ down and $a=3$ right. However, the ice is slippery, so we don't always go the direction we want to.\n",
|
||||
"\n",
|
||||
"Note that as for the states, we've indexed the actions from zero (unlike in the book) so they map to the indices of arrays better"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Fhc6DzZNOjiC"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"transition_probabilities_given_action0 = np.array(\\\n",
|
||||
"[[0.00 , 0.33, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.50 , 0.00, 0.33, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.33, 0.00, 0.50, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.33, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.50 , 0.00, 0.00, 0.00, 0.00, 0.17, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.34, 0.00, 0.00, 0.25, 0.00, 0.17, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.34, 0.00, 0.00, 0.17, 0.00, 0.25, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.50, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.17, 0.00, 0.00, 0.75, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.25, 0.00, 0.17, 0.00, 0.00, 0.50, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.17, 0.00, 0.25, 0.00, 0.00, 0.50, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.75 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.25, 0.00, 0.25, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.25, 0.00, 0.25 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.25, 0.00 ],\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"transition_probabilities_given_action1 = np.array(\\\n",
|
||||
"[[0.00 , 0.25, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.75 , 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.50, 0.00, 0.50, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.33, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.25 , 0.00, 0.00, 0.00, 0.00, 0.17, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.25, 0.00, 0.00, 0.50, 0.00, 0.17, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.25, 0.00, 0.00, 0.50, 0.00, 0.33, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.50, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.33, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.17, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.50, 0.00, 0.17, 0.00, 0.00, 0.25, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.50, 0.00, 0.33, 0.00, 0.00, 0.25, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.34, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.50 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.75, 0.00, 0.25, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.50, 0.00, 0.50 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.34, 0.00, 0.00, 0.50, 0.00 ],\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"transition_probabilities_given_action2 = np.array(\\\n",
|
||||
"[[0.00 , 0.25, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.25 , 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.25, 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.75 , 0.00, 0.00, 0.00, 0.00, 0.17, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.50, 0.00, 0.00, 0.25, 0.00, 0.17, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.50, 0.00, 0.00, 0.16, 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.75, 0.00, 0.00, 0.16, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.17, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.25, 0.00, 0.17, 0.00, 0.00, 0.33, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.16, 0.00, 0.25, 0.00, 0.00, 0.33, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.16, 0.00, 0.00, 0.00, 0.00, 0.50 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.33, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.50, 0.00, 0.33, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.34, 0.00, 0.50 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.34, 0.00 ],\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"transition_probabilities_given_action3 = np.array(\\\n",
|
||||
"[[0.00 , 0.25, 0.00, 0.00, 0.33, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.50 , 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.50, 0.00, 0.75, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.50, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.50 , 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.33, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.25, 0.00, 0.00, 0.33, 0.00, 0.50, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.50, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.34, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00, 0.50, 0.00, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.33, 0.00, 0.50, 0.00, 0.00, 0.25, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.17, 0.00, 0.50, 0.00, 0.00, 0.25, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.17, 0.00, 0.00, 0.00, 0.00, 0.25 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.34, 0.00, 0.00, 0.00, 0.00, 0.50, 0.00, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.50, 0.00, 0.50, 0.00 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.16, 0.00, 0.00, 0.25, 0.00, 0.75 ],\n",
|
||||
" [0.00 , 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.25, 0.00, 0.00, 0.25, 0.00 ],\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"# Store all of these in a three dimension array\n",
|
||||
"# Pr(s_{t+1}=2|s_{t}=1, a_{t}=3] is stored at position [2,1,3]\n",
|
||||
"transition_probabilities_given_action = np.concatenate((np.expand_dims(transition_probabilities_given_action0,2),\n",
|
||||
" np.expand_dims(transition_probabilities_given_action1,2),\n",
|
||||
" np.expand_dims(transition_probabilities_given_action2,2),\n",
|
||||
" np.expand_dims(transition_probabilities_given_action3,2)),axis=2)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "l7rT78BbOgTi"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def q_learning_step(state_action_values, reward, state, new_state, action, gamma, alpha = 0.1):\n",
|
||||
" # TODO -- write this function\n",
|
||||
" # Replace this line\n",
|
||||
" state_action_values_after = np.copy(state_action_values)\n",
|
||||
"\n",
|
||||
" return state_action_values_after"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "5pO6-9ACWhiV"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# This takes a single step from an MDP which just has a completely random policy\n",
|
||||
"def markov_decision_process_step(state, transition_probabilities_given_action, reward_structure):\n",
|
||||
" # Pick action\n",
|
||||
" action = np.random.randint(4)\n",
|
||||
" # Update the state\n",
|
||||
" new_state = np.random.choice(a=np.arange(0,transition_probabilities_given_action.shape[0]),p = transition_probabilities_given_action[:,state,action])\n",
|
||||
" # Return the reward -- here the reward is for leaving the state\n",
|
||||
" reward = reward_structure[state]\n",
|
||||
"\n",
|
||||
" return new_state, reward, action"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "akjrncMF-FkU"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Initialize the state-action values to random numbers\n",
|
||||
"np.random.seed(0)\n",
|
||||
"n_state = transition_probabilities_given_action.shape[0]\n",
|
||||
"n_action = transition_probabilities_given_action.shape[2]\n",
|
||||
"state_action_values = np.random.normal(size=(n_action, n_state))\n",
|
||||
"gamma = 0.9\n",
|
||||
"\n",
|
||||
"policy = np.argmax(state_action_values, axis=0).astype(int)\n",
|
||||
"mdp_drawer = DrawMDP(n_rows, n_cols)\n",
|
||||
"mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values, rewards = reward_structure)\n",
|
||||
"\n",
|
||||
"# Now let's simulate a single Q-learning step\n",
|
||||
"initial_state = 9\n",
|
||||
"print(\"Initial state = \", initial_state)\n",
|
||||
"new_state, reward, action = markov_decision_process_step(initial_state, transition_probabilities_given_action, reward_structure)\n",
|
||||
"print(\"Action = \", action)\n",
|
||||
"print(\"New state = \", new_state)\n",
|
||||
"print(\"Reward = \", reward)\n",
|
||||
"\n",
|
||||
"state_action_values_after = q_learning_step(state_action_values, reward, initial_state, new_state, action, gamma)\n",
|
||||
"print(\"Your value:\",state_action_values_after[action, initial_state])\n",
|
||||
"print(\"True value: 0.27650262412468796\")\n",
|
||||
"\n",
|
||||
"policy = np.argmax(state_action_values, axis=0).astype(int)\n",
|
||||
"mdp_drawer = DrawMDP(n_rows, n_cols)\n",
|
||||
"mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values_after, rewards = reward_structure)\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Fu5_VjvbSwfJ"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's run this for a while and watch the policy improve"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Ogh0qucmb68J"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Initialize the state-action values to random numbers\n",
|
||||
"np.random.seed(0)\n",
|
||||
"n_state = transition_probabilities_given_action.shape[0]\n",
|
||||
"n_action = transition_probabilities_given_action.shape[2]\n",
|
||||
"state_action_values = np.random.normal(size=(n_action, n_state))\n",
|
||||
"# Hard code termination state of finding fish\n",
|
||||
"state_action_values[:,n_state-1] = 3.0\n",
|
||||
"gamma = 0.9\n",
|
||||
"\n",
|
||||
"# Draw the initial setup\n",
|
||||
"policy = np.argmax(state_action_values, axis=0).astype(int)\n",
|
||||
"mdp_drawer = DrawMDP(n_rows, n_cols)\n",
|
||||
"mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values, rewards = reward_structure)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"state= np.random.randint(n_state-1)\n",
|
||||
"\n",
|
||||
"# Run for a number of iterations\n",
|
||||
"for c_iter in range(10000):\n",
|
||||
" new_state, reward, action = markov_decision_process_step(state, transition_probabilities_given_action, reward_structure)\n",
|
||||
" state_action_values_after = q_learning_step(state_action_values, reward, state, new_state, action, gamma)\n",
|
||||
" # If in termination state, reset state randomly\n",
|
||||
" if new_state==15:\n",
|
||||
" state= np.random.randint(n_state-1)\n",
|
||||
" else:\n",
|
||||
" state = new_state\n",
|
||||
" # Update the policy\n",
|
||||
" state_action_values = np.copy(state_action_values_after)\n",
|
||||
" policy = np.argmax(state_action_values, axis=0).astype(int)\n",
|
||||
"\n",
|
||||
"# Draw the final situation\n",
|
||||
"mdp_drawer = DrawMDP(n_rows, n_cols)\n",
|
||||
"mdp_drawer.draw(layout, policy = policy, state_action_values = state_action_values, rewards = reward_structure)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "qQFhwVqPcCFH"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
}
|
||||
]
|
||||
}
|
||||
170
Notebooks/Chap19/19_5_Control_Variates.ipynb
Normal file
170
Notebooks/Chap19/19_5_Control_Variates.ipynb
Normal file
@@ -0,0 +1,170 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyO6CLgMIO5bUVAMkzPT3z4y",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "view-in-github",
|
||||
"colab_type": "text"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap19/19_5_Control_Variates.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# **Notebook 19.5: Control variates**\n",
|
||||
"\n",
|
||||
"This notebook investigates the method of control variates as described in figure 19.16\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
|
||||
"\n",
|
||||
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "t9vk9Elugvmi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "OLComQyvCIJ7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Genearate from our two variables, $a$ and $b$. We are interested in estimating the mean of $a$, but we can use $b$$ to improve our estimates if it is correlated"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "uwmhcAZBzTRO"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Sample from two variables with mean zero, standard deviation one, and a given correlation coefficient\n",
|
||||
"def get_samples(n_samples, correlation_coeff=0.8):\n",
|
||||
" a = np.random.normal(size=(1,n_samples))\n",
|
||||
" temp = np.random.normal(size=(1, n_samples))\n",
|
||||
" b = correlation_coeff * a + np.sqrt(1-correlation_coeff * correlation_coeff) * temp\n",
|
||||
" return a, b"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "bC8MBXPawQJ3"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"N = 10000000\n",
|
||||
"a,b, = get_samples(N)\n",
|
||||
"\n",
|
||||
"# Verify that these two variables have zero mean and unit standard deviation\n",
|
||||
"print(\"Mean of a = %3.3f, Std of a = %3.3f\"%(np.mean(a),np.std(a)))\n",
|
||||
"print(\"Mean of b = %3.3f, Std of b = %3.3f\"%(np.mean(b),np.std(b)))"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "1cT66nbRyW34"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's samples $N=10$ examples from $a$ and estimate their mean $\\mathbb{E}[a]$. We'll do this 1000000 times and then compute the variance of those estimates."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "PWoYRpjS0Nlf"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"n_estimate = 1000000\n",
|
||||
"\n",
|
||||
"N = 5\n",
|
||||
"\n",
|
||||
"# TODO -- sample N examples of variable $a$\n",
|
||||
"# Compute the mean of each\n",
|
||||
"# Compute the mean and variance of these estimates of the mean\n",
|
||||
"# Replace this line\n",
|
||||
"mean_of_estimator_1 = -1; std_of_estimator_1 = -1\n",
|
||||
"\n",
|
||||
"print(\"Standard estimator mean = %3.3f, Standard estimator variance = %3.3f\"%(mean_of_estimator_1, std_of_estimator_1))"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "n6Uem2aYzBp7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now let's estimate the mean $\\mathbf{E}[a]$ of $a$ by computing $a-b$ where $b$ is correlated with $a$"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "F-af86z13TFc"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"n_estimate = 1000000\n",
|
||||
"\n",
|
||||
"N = 5\n",
|
||||
"\n",
|
||||
"# TODO -- sample N examples of variables $a$ and $b$\n",
|
||||
"# Compute $c=a-b$ for each and then compute the mean of $c$\n",
|
||||
"# Compute the mean and variance of these estimates of the mean of $c$\n",
|
||||
"# Replace this line\n",
|
||||
"mean_of_estimator_2 = -1; std_of_estimator_2 = -1\n",
|
||||
"\n",
|
||||
"print(\"Control variate estimator mean = %3.3f, Control variate estimator variance = %3.3f\"%(mean_of_estimator_2, std_of_estimator_2))"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "MrEVDggY0IGU"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Note that they both have a very similar mean, but the second estimator has a lower variance. \n",
|
||||
"\n",
|
||||
"TODO -- Experiment with different samples sizes $N$ and correlation coefficients."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Jklzkca14ofS"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
BIN
Notebooks/Chap19/Empty.png
Normal file
BIN
Notebooks/Chap19/Empty.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
BIN
Notebooks/Chap19/Fish.png
Normal file
BIN
Notebooks/Chap19/Fish.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
BIN
Notebooks/Chap19/Hole.png
Normal file
BIN
Notebooks/Chap19/Hole.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 9.2 KiB |
BIN
Notebooks/Chap19/Penguin.png
Normal file
BIN
Notebooks/Chap19/Penguin.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
305
Notebooks/Chap20/20_1_Random_Data.ipynb
Normal file
305
Notebooks/Chap20/20_1_Random_Data.ipynb
Normal file
@@ -0,0 +1,305 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyPkSYbEjOcEmLt8tU6HxNuR",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "view-in-github",
|
||||
"colab_type": "text"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap20/20_1_Random_Data.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# **Notebook 20.1: Random Data**\n",
|
||||
"\n",
|
||||
"This notebook investigates training the network with random data, as illustrated in figure 20.1.\n",
|
||||
"\n",
|
||||
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
|
||||
"\n",
|
||||
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions.\n",
|
||||
"\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "t9vk9Elugvmi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Run this if you're in a Colab to make a local copy of the MNIST 1D repository\n",
|
||||
"!git clone https://github.com/greydanus/mnist1d"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "D5yLObtZCi9J"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import os\n",
|
||||
"import torch, torch.nn as nn\n",
|
||||
"from torch.utils.data import TensorDataset, DataLoader\n",
|
||||
"from torch.optim.lr_scheduler import StepLR\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import mnist1d\n",
|
||||
"import random\n",
|
||||
"from IPython.display import display, clear_output"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "YrXWAH7sUWvU"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"args = mnist1d.data.get_dataset_args()\n",
|
||||
"data = mnist1d.data.get_dataset(args, path='./mnist1d_data.pkl', download=False, regenerate=False)\n",
|
||||
"\n",
|
||||
"# The training and test input and outputs are in\n",
|
||||
"# data['x'], data['y'], data['x_test'], and data['y_test']\n",
|
||||
"print(\"Examples in training set: {}\".format(len(data['y'])))\n",
|
||||
"print(\"Examples in test set: {}\".format(len(data['y_test'])))\n",
|
||||
"print(\"Length of each example: {}\".format(data['x'].shape[-1]))"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "twI72ZCrCt5z"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Define the network"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "_sFvRDGrl4qe"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"D_i = 40 # Input dimensions\n",
|
||||
"D_k = 300 # Hidden dimensions\n",
|
||||
"D_o = 10 # Output dimensions\n",
|
||||
"\n",
|
||||
"model = nn.Sequential(\n",
|
||||
"nn.Linear(D_i, D_k),\n",
|
||||
"nn.ReLU(),\n",
|
||||
"nn.Linear(D_k, D_k),\n",
|
||||
"nn.ReLU(),\n",
|
||||
"nn.Linear(D_k, D_k),\n",
|
||||
"nn.ReLU(),\n",
|
||||
"nn.Linear(D_k, D_k),\n",
|
||||
"nn.ReLU(),\n",
|
||||
"nn.Linear(D_k, D_o))"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "FslroPJJffrh"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# He initialization of weights\n",
|
||||
"def weights_init(layer_in):\n",
|
||||
" if isinstance(layer_in, nn.Linear):\n",
|
||||
" nn.init.kaiming_uniform_(layer_in.weight)\n",
|
||||
" layer_in.bias.data.fill_(0.0)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "YgLaex1pfhqz"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def train_model(train_data_x, train_data_y, n_epoch):\n",
|
||||
" # choose cross entropy loss function (equation 5.24 in the loss notes)\n",
|
||||
" loss_function = nn.CrossEntropyLoss()\n",
|
||||
" # construct SGD optimizer and initialize learning rate and momentum\n",
|
||||
" optimizer = torch.optim.SGD(model.parameters(), lr = 0.02, momentum=0.9)\n",
|
||||
" # object that decreases learning rate by half every 20 epochs\n",
|
||||
" scheduler = StepLR(optimizer, step_size=20, gamma=0.5)\n",
|
||||
" # create 100 dummy data points and store in data loader class\n",
|
||||
" x_train = torch.tensor(train_data_x.transpose().astype('float32'))\n",
|
||||
" y_train = torch.tensor(train_data_y.astype('long'))\n",
|
||||
"\n",
|
||||
" # load the data into a class that creates the batches\n",
|
||||
" data_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=100, shuffle=True, worker_init_fn=np.random.seed(1))\n",
|
||||
"\n",
|
||||
" # Initialize model weights\n",
|
||||
" model.apply(weights_init)\n",
|
||||
"\n",
|
||||
" # store the loss and the % correct at each epoch\n",
|
||||
" losses_train = np.zeros((n_epoch))\n",
|
||||
"\n",
|
||||
" for epoch in range(n_epoch):\n",
|
||||
" # loop over batches\n",
|
||||
" for i, data in enumerate(data_loader):\n",
|
||||
" # retrieve inputs and labels for this batch\n",
|
||||
" x_batch, y_batch = data\n",
|
||||
" # zero the parameter gradients\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" # forward pass -- calculate model output\n",
|
||||
" pred = model(x_batch)\n",
|
||||
" # compute the loss\n",
|
||||
" loss = loss_function(pred, y_batch)\n",
|
||||
" # backward pass\n",
|
||||
" loss.backward()\n",
|
||||
" # SGD update\n",
|
||||
" optimizer.step()\n",
|
||||
"\n",
|
||||
" # Run whole dataset to get statistics -- normally wouldn't do this\n",
|
||||
" pred_train = model(x_train)\n",
|
||||
" _, predicted_train_class = torch.max(pred_train.data, 1)\n",
|
||||
" losses_train[epoch] = loss_function(pred_train, y_train).item()\n",
|
||||
" if epoch % 5 == 0:\n",
|
||||
" clear_output(wait=True)\n",
|
||||
" display(\"Epoch %d, train loss %3.3f\"%(epoch, losses_train[epoch]))\n",
|
||||
"\n",
|
||||
" # tell scheduler to consider updating learning rate\n",
|
||||
" scheduler.step()\n",
|
||||
"\n",
|
||||
" return losses_train"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "NYw8I_3mmX5c"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Load in the data\n",
|
||||
"train_data_x = data['x'].transpose()\n",
|
||||
"train_data_y = data['y']\n",
|
||||
"# Print out sizes\n",
|
||||
"print(\"Train data: %d examples (columns), each of which has %d dimensions (rows)\"%((train_data_x.shape[1],train_data_x.shape[0])))"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "4FE3HQ_vedXO"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Compute loss for proper data and plot\n",
|
||||
"n_epoch = 60\n",
|
||||
"loss_true_labels = train_model(train_data_x, train_data_y, n_epoch)\n",
|
||||
"# Plot the results\n",
|
||||
"fig, ax = plt.subplots()\n",
|
||||
"ax.plot(loss_true_labels,'r-',label='true_labels')\n",
|
||||
"# ax.set_ylim(0,0.7); ax.set_xlim(0,n_epoch)\n",
|
||||
"ax.set_xlabel('Epoch'); ax.set_ylabel('Loss')\n",
|
||||
"ax.legend()\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "b56wdODqemF1"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# TODO -- Randomize the input data (train_data_x), but retain overall mean and variance\n",
|
||||
"# Replace this line\n",
|
||||
"train_data_x_randomized = np.copy(train_data_x)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "SbPCiiUKgTLw"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Compute loss for true labels and plot\n",
|
||||
"n_epoch = 60\n",
|
||||
"loss_randomized_data = train_model(train_data_x_randomized, train_data_y, n_epoch)\n",
|
||||
"# Plot the results\n",
|
||||
"fig, ax = plt.subplots()\n",
|
||||
"ax.plot(loss_true_labels,'r-',label='true_labels')\n",
|
||||
"ax.plot(loss_randomized_data,'b-',label='random_data')\n",
|
||||
"# ax.set_ylim(0,0.7); ax.set_xlim(0,n_epoch)\n",
|
||||
"ax.set_xlabel('Epoch'); ax.set_ylabel('Loss')\n",
|
||||
"ax.legend()\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "y7CcCJvvjLnn"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# TODO -- Permute the labels\n",
|
||||
"# Replace this line:\n",
|
||||
"train_data_y_permuted = np.copy(train_data_y)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ojaMTrzKj_74"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Compute loss for true labels and plot\n",
|
||||
"n_epoch = 60\n",
|
||||
"loss_permuted_labels = train_model(train_data_x, train_data_y_permuted, n_epoch)\n",
|
||||
"# Plot the results\n",
|
||||
"fig, ax = plt.subplots()\n",
|
||||
"ax.plot(loss_true_labels,'r-',label='true_labels')\n",
|
||||
"ax.plot(loss_randomized_data,'b-',label='random_data')\n",
|
||||
"ax.plot(loss_permuted_labels,'g-',label='random_labels')\n",
|
||||
"# ax.set_ylim(0,0.7); ax.set_xlim(0,n_epoch)\n",
|
||||
"ax.set_xlabel('Epoch'); ax.set_ylabel('Loss')\n",
|
||||
"ax.legend()\n",
|
||||
"plt.show()"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "LaYCSjyMo9LQ"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
}
|
||||
]
|
||||
}
|
||||
485
index.html
485
index.html
@@ -1,24 +1,31 @@
|
||||
<h1>Understanding Deep Learning</h1>
|
||||
by Simon J.D. Prince
|
||||
<br>
|
||||
To be published by MIT Press Dec 5th 2023.<br>
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<title>udlbook</title>
|
||||
<link rel="stylesheet" href="style.css">
|
||||
</head>
|
||||
|
||||
<img src="https://raw.githubusercontent.com/udlbook/udlbook/main/UDLCoverSmall.jpg" alt="front cover">
|
||||
|
||||
<h2> Download draft PDF </h2>
|
||||
|
||||
<a href="https://github.com/udlbook/udlbook/releases/download/v1.1.3/UnderstandingDeepLearning_01_10_23_C.pdf">Draft PDF Chapters 1-21</a><br> 2023-10-01. CC-BY-NC-ND license
|
||||
<br>
|
||||
<body>
|
||||
<div id="head">
|
||||
<div>
|
||||
<h1 style="margin: 0; font-size: 36px">Understanding Deep Learning</h1>
|
||||
by Simon J.D. Prince
|
||||
<br>To be published by MIT Press Dec 5th 2023.<br>
|
||||
<ul>
|
||||
<li>
|
||||
<p style="font-size: larger; margin-bottom: 0">Download draft PDF Chapters 1-21 <a
|
||||
href="https://github.com/udlbook/udlbook/releases/download/v1.14/UnderstandingDeepLearning_13_10_23_C.pdf">here</a>
|
||||
</p>2023-10-13. CC-BY-NC-ND license<br>
|
||||
<img src="https://img.shields.io/github/downloads/udlbook/udlbook/total" alt="download stats shield">
|
||||
<br>
|
||||
<ul>
|
||||
<li> Appendices and notebooks coming soon
|
||||
<li> Report errata via <a href="https://github.com/udlbook/udlbook/issues">github</a> or contact me directly at udlbookmail@gmail.com
|
||||
<li> Follow me on <a href="https://twitter.com/SimonPrinceAI">Twitter</a> or <a href="https://www.linkedin.com/in/simon-prince-615bb9165/">LinkedIn</a> for updates.
|
||||
</ul>
|
||||
|
||||
<h2>Table of contents</h2>
|
||||
<ul>
|
||||
</li>
|
||||
<li> Report errata via <a href="https://github.com/udlbook/udlbook/issues">github</a>
|
||||
or contact me directly at udlbookmail@gmail.com
|
||||
<li> Follow me on <a href="https://twitter.com/SimonPrinceAI">Twitter</a> or <a
|
||||
href="https://www.linkedin.com/in/simon-prince-615bb9165/">LinkedIn</a> for updates.
|
||||
</ul>
|
||||
<h2>Table of contents</h2>
|
||||
<ul>
|
||||
<li> Chapter 1 - Introduction
|
||||
<li> Chapter 2 - Supervised learning
|
||||
<li> Chapter 3 - Shallow neural networks
|
||||
@@ -40,122 +47,330 @@ To be published by MIT Press Dec 5th 2023.<br>
|
||||
<li> Chapter 19 - Deep reinforcement learning
|
||||
<li> Chapter 20 - Why does deep learning work?
|
||||
<li> Chapter 21 - Deep learning and ethics
|
||||
</ul>
|
||||
</ul>
|
||||
</div>
|
||||
<div id="cover">
|
||||
<img src="https://raw.githubusercontent.com/udlbook/udlbook/main/UDLCoverSmall.jpg"
|
||||
alt="front cover">
|
||||
</div>
|
||||
</div>
|
||||
<div id="body">
|
||||
<h2>Resources for instructors </h2>
|
||||
<p>Instructor answer booklet available with proof of credentials via <a
|
||||
href="https://mitpress.mit.edu/9780262048644/understanding-deep-learning"> MIT Press</a>.</p>
|
||||
<p>Figures in PDF (vector) / SVG (vector) / Powerpoint (images):
|
||||
<ul>
|
||||
<li> Chapter 1 - Introduction: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap1PDF.zip">PDF
|
||||
Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1udnl5pUOAc8DcAQ7HQwyzP9pwL95ynnv">
|
||||
SVG
|
||||
Figures</a> / <a
|
||||
href="https://docs.google.com/presentation/d/1IjTqIUvWCJc71b5vEJYte-Dwujcp7rvG/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 2 - Supervised learning: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap2PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1VSxcU5y1qNFlmd3Lb3uOWyzILuOj1Dla"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/1Br7R01ROtRWPlNhC_KOommeHAWMBpWtz/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 3 - Shallow neural networks: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap3PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=19kZFWlXhzN82Zx02ByMmSZOO4T41fmqI"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/1e9M3jB5I9qZ4dCBY90Q3Hwft_i068QVQ/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 4 - Deep neural networks: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap4PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1ojr0ebsOhzvS04ItAflX2cVmYqHQHZUa"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/1LTSsmY4mMrJbqXVvoTOCkQwHrRKoYnJj/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 5 - Loss functions: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap5PDF.zip">PDF
|
||||
Figures</a> / <a href="https://drive.google.com/uc?export=download&id=17MJO7fiMpFZVqKeqXTbQ36AMpmR4GizZ">
|
||||
SVG
|
||||
Figures</a> / <a
|
||||
href="https://docs.google.com/presentation/d/1gcpC_3z9oRp87eMkoco-kdLD-MM54Puk/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 6 - Training models: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap6PDF.zip">PDF
|
||||
Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1VPdhFRnCr9_idTrX0UdHKGAw2shUuwhK">
|
||||
SVG
|
||||
Figures</a> / <a
|
||||
href="https://docs.google.com/presentation/d/1AKoeggAFBl9yLC7X5tushAGzCCxmB7EY/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 7 - Gradients and initialization: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap7PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1TTl4gvrTvNbegnml4CoGoKOOd6O8-PGs"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/11zhB6PI-Dp6Ogmr4IcI6fbvbqNqLyYcz/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 8 - Measuring performance: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap8PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=19eQOnygd_l0DzgtJxXuYnWa4z7QKJrJx"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/1SHRmJscDLUuQrG7tmysnScb3ZUAqVMZo/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 9 - Regularization: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap9PDF.zip">PDF
|
||||
Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1LprgnUGL7xAM9-jlGZC9LhMPeefjY0r0">
|
||||
SVG
|
||||
Figures</a> / <a
|
||||
href="https://docs.google.com/presentation/d/1VwIfvjpdfTny6sEfu4ZETwCnw6m8Eg-5/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 10 - Convolutional networks: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap10PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1-Wb3VzaSvVeRzoUzJbI2JjZE0uwqupM9"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/1MtfKBC4Y9hWwGqeP6DVwUNbi1j5ncQCg/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 11 - Residual networks: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap11PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1Mr58jzEVseUAfNYbGWCQyDtEDwvfHRi1"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/1saY8Faz0KTKAAifUrbkQdLA2qkyEjOPI/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 12 - Transformers: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap12PDF.zip">PDF
|
||||
Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1txzOVNf8-jH4UfJ6SLnrtOfPd1Q3ebzd">
|
||||
SVG
|
||||
Figures</a> / <a
|
||||
href="https://docs.google.com/presentation/d/1GVNvYWa0WJA6oKg89qZre-UZEhABfm0l/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 13 - Graph neural networks: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap13PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1lQIV6nRp6LVfaMgpGFhuwEXG-lTEaAwe"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/1YwF3U82c1mQ74c1WqHVTzLZ0j7GgKaWP/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 14 - Unsupervised learning: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap14PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1aMbI6iCuUvOywqk5pBOmppJu1L1anqsM"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/1A-lBGv3NHl4L32NvfFgy1EKeSwY-0UeB/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">
|
||||
PowerPoint Figures</a>
|
||||
<li> Chapter 15 - Generative adversarial networks: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap15PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1EErnlZCOlXc3HK7m83T2Jh_0NzIUHvtL"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/10Ernk41ShOTf4IYkMD-l4dJfKATkXH4w/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 16 - Normalizing flows: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap16PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1B9bxtmdugwtg-b7Y4AdQKAIEVWxjx8l3"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/1nLLzqb9pdfF_h6i1HUDSyp7kSMIkSUUA/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 17 - Variational autoencoders: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap17PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1SNtNIY7khlHQYMtaOH-FosSH3kWwL4b7"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/1lQE4Bu7-LgvV2VlJOt_4dQT-kusYl7Vo/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Chapter 18 - Diffusion models: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap18PDF.zip">PDF Figures</a> / <a
|
||||
href="https://docs.google.com/presentation/d/1x_ufIBtVPzWUvRieKMkpw5SdRjXWwdfR/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">
|
||||
PowerPoint Figures</a>
|
||||
<li> Chapter 19 - Deep reinforcement learning: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap19PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1a5WUoF7jeSgwC_PVdckJi1Gny46fCqh0"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/1TnYmVbFNhmMFetbjyfXGmkxp1EHauMqr/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">
|
||||
PowerPoint Figures </a>
|
||||
<li> Chapter 20 - Why does deep learning work?: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap20PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1M2d0DHEgddAQoIedKSDTTt7m1ZdmBLQ3"> SVG Figures</a>
|
||||
/
|
||||
<a href="https://docs.google.com/presentation/d/1coxF4IsrCzDTLrNjRagHvqB_FBy10miA/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">
|
||||
PowerPoint Figures</a>
|
||||
<li> Chapter 21 - Deep learning and ethics: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap21PDF.zip">PDF Figures</a> / <a
|
||||
href="https://drive.google.com/uc?export=download&id=1jixmFfwmZkW_UVYzcxmDcMsdFFtnZ0bU"> SVG Figures</a>/
|
||||
<a
|
||||
href="https://docs.google.com/presentation/d/1EtfzanZYILvi9_-Idm28zD94I_6OrN9R/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint
|
||||
Figures</a>
|
||||
<li> Appendices - <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLAppendixPDF.zip">PDF
|
||||
Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1k2j7hMN40ISPSg9skFYWFL3oZT7r8v-l">
|
||||
SVG
|
||||
Figures</a> / <a
|
||||
href="https://docs.google.com/presentation/d/1_2cJHRnsoQQHst0rwZssv-XH4o5SEHks/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">Powerpoint
|
||||
Figures</a>
|
||||
</ul>
|
||||
|
||||
Instructions for editing figures / equations can be found <a
|
||||
href="https://drive.google.com/file/d/1T_MXXVR4AfyMnlEFI-UVDh--FXI5deAp/view?usp=sharing">here</a>.
|
||||
|
||||
<h2>Resources for students</h2>
|
||||
|
||||
<p>Answers to selected questions: <a
|
||||
href="https://github.com/udlbook/udlbook/raw/main/UDL_Answer_Booklet_Students.pdf">PDF</a>
|
||||
</p>
|
||||
<p>Python notebooks: (Early ones more thoroughly tested than later ones!)</p>
|
||||
|
||||
<ul>
|
||||
<li> Notebook 1.1 - Background mathematics: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap01/1_1_BackgroundMathematics.ipynb">ipynb/colab</a>
|
||||
</li>
|
||||
<li> Notebook 2.1 - Supervised learning: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap02/2_1_Supervised_Learning.ipynb">ipynb/colab</a>
|
||||
</li>
|
||||
<li> Notebook 3.1 - Shallow networks I: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_1_Shallow_Networks_I.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 3.2 - Shallow networks II: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_2_Shallow_Networks_II.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 3.3 - Shallow network regions: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_3_Shallow_Network_Regions.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 3.4 - Activation functions: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_4_Activation_Functions.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 4.1 - Composing networks: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap04/4_1_Composing_Networks.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 4.2 - Clipping functions: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap04/4_2_Clipping_functions.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 4.3 - Deep networks: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap04/4_3_Deep_Networks.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 5.1 - Least squares loss: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap05/5_1_Least_Squares_Loss.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 5.2 - Binary cross-entropy loss: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap05/5_2_Binary_Cross_Entropy_Loss.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 5.3 - Multiclass cross-entropy loss: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap05/5_3_Multiclass_Cross_entropy_Loss.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 6.1 - Line search: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_1_Line_Search.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 6.2 - Gradient descent: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_2_Gradient_Descent.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 6.3 - Stochastic gradient descent: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_3_Stochastic_Gradient_Descent.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 6.4 - Momentum: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_4_Momentum.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 6.5 - Adam: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_5_Adam.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 7.1 - Backpropagation in toy model: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap07/7_1_Backpropagation_in_Toy_Model.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 7.2 - Backpropagation: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap07/7_2_Backpropagation.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 7.3 - Initialization: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap07/7_3_Initialization.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 8.1 - MNIST-1D performance: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_1_MNIST_1D_Performance.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 8.2 - Bias-variance trade-off: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_2_Bias_Variance_Trade_Off.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 8.3 - Double descent: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_3_Double_Descent.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 8.4 - High-dimensional spaces: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_4_High_Dimensional_Spaces.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 9.1 - L2 regularization: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_1_L2_Regularization.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 9.2 - Implicit regularization: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_2_Implicit_Regularization.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 9.3 - Ensembling: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_3_Ensembling.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 9.4 - Bayesian approach: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_4_Bayesian_Approach.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 9.5 - Augmentation <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_5_Augmentation.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 10.1 - 1D convolution: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_1_1D_Convolution.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 10.2 - Convolution for MNIST-1D: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_2_Convolution_for_MNIST_1D.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 10.3 - 2D convolution: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_3_2D_Convolution.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 10.4 - Downsampling & upsampling: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_4_Downsampling_and_Upsampling.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 10.5 - Convolution for MNIST: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_5_Convolution_For_MNIST.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 11.1 - Shattered gradients: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap11/11_1_Shattered_Gradients.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 11.2 - Residual networks: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap11/11_2_Residual_Networks.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 11.3 - Batch normalization: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap11/11_3_Batch_Normalization.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 12.1 - Self-attention: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_1_Self_Attention.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 12.2 - Multi-head self-attention: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_2_Multihead_Self_Attention.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 12.3 - Tokenization: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_3_Tokenization.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 12.4 - Decoding strategies: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_4_Decoding_Strategies.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 13.1 - Encoding graphs: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap13/13_1_Graph_Representation.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 13.2 - Graph classification : <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap13/13_2_Graph_Classification.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 13.3 - Neighborhood sampling: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap13/13_3_Neighborhood_Sampling.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 13.4 - Graph attention: <a
|
||||
href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap13/13_4_Graph_Attention_Networks.ipynb">ipynb/colab </a>
|
||||
</li>
|
||||
<li> Notebook 15.1 - GAN toy example: (coming soon)</li>
|
||||
<li> Notebook 15.2 - Wasserstein distance: (coming soon)</li>
|
||||
<li> Notebook 16.1 - 1D normalizing flows: (coming soon)</li>
|
||||
<li> Notebook 16.2 - Autoregressive flows: (coming soon)</li>
|
||||
<li> Notebook 16.3 - Contraction mappings: (coming soon)</li>
|
||||
<li> Notebook 17.1 - Latent variable models: (coming soon)</li>
|
||||
<li> Notebook 17.2 - Reparameterization trick: (coming soon)</li>
|
||||
<li> Notebook 17.3 - Importance sampling: (coming soon)</li>
|
||||
<li> Notebook 18.1 - Diffusion encoder: (coming soon)</li>
|
||||
<li> Notebook 18.2 - 1D diffusion model: (coming soon)</li>
|
||||
<li> Notebook 18.3 - Reparameterized model: (coming soon)</li>
|
||||
<li> Notebook 18.4 - Families of diffusion models: (coming soon)</li>
|
||||
<li> Notebook 19.1 - Markov decision processes: (coming soon)</li>
|
||||
<li> Notebook 19.2 - Dynamic programming: (coming soon)</li>
|
||||
<li> Notebook 19.3 - Monte-Carlo methods: (coming soon)</li>
|
||||
<li> Notebook 19.4 - Temporal difference methods: (coming soon)</li>
|
||||
<li> Notebook 19.5 - Control variates: (coming soon)</li>
|
||||
<li> Notebook 20.1 - Random data: (coming soon)</li>
|
||||
<li> Notebook 20.2 - Full-batch gradient descent: (coming soon)</li>
|
||||
<li> Notebook 20.3 - Lottery tickets: (coming soon)</li>
|
||||
<li> Notebook 20.4 - Adversarial attacks: (coming soon)</li>
|
||||
<li> Notebook 21.1 - Bias mitigation: (coming soon)</li>
|
||||
<li> Notebook 21.2 - Explainability: (coming soon)</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h2>Resources for instructors </h2>
|
||||
|
||||
<p></p>Instructor answer booklet available with proof of credentials via <a href="https://mitpress.mit.edu/9780262048644/understanding-deep-learning"/> MIT Press</a></p>
|
||||
|
||||
<p></p>Figures in PDF (vector) / SVG (vector) / Powerpoint (images):
|
||||
|
||||
<ul>
|
||||
<li> Chapter 1 - Introduction: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap1PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1udnl5pUOAc8DcAQ7HQwyzP9pwL95ynnv"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1IjTqIUvWCJc71b5vEJYte-Dwujcp7rvG/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 2 - Supervised learning: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap2PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1VSxcU5y1qNFlmd3Lb3uOWyzILuOj1Dla"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1Br7R01ROtRWPlNhC_KOommeHAWMBpWtz/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 3 - Shallow neural networks: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap3PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=19kZFWlXhzN82Zx02ByMmSZOO4T41fmqI"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1e9M3jB5I9qZ4dCBY90Q3Hwft_i068QVQ/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 4 - Deep neural networks: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap4PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1ojr0ebsOhzvS04ItAflX2cVmYqHQHZUa"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1LTSsmY4mMrJbqXVvoTOCkQwHrRKoYnJj/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 5 - Loss functions: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap5PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=17MJO7fiMpFZVqKeqXTbQ36AMpmR4GizZ"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1gcpC_3z9oRp87eMkoco-kdLD-MM54Puk/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 6 - Training models: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap6PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1VPdhFRnCr9_idTrX0UdHKGAw2shUuwhK"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1AKoeggAFBl9yLC7X5tushAGzCCxmB7EY/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 7 - Gradients and initialization: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap7PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1TTl4gvrTvNbegnml4CoGoKOOd6O8-PGs"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/11zhB6PI-Dp6Ogmr4IcI6fbvbqNqLyYcz/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 8 - Measuring performance: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap8PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=19eQOnygd_l0DzgtJxXuYnWa4z7QKJrJx"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1SHRmJscDLUuQrG7tmysnScb3ZUAqVMZo/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 9 - Regularization: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap9PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1LprgnUGL7xAM9-jlGZC9LhMPeefjY0r0"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1VwIfvjpdfTny6sEfu4ZETwCnw6m8Eg-5/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 10 - Convolutional networks: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap10PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1-Wb3VzaSvVeRzoUzJbI2JjZE0uwqupM9"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1MtfKBC4Y9hWwGqeP6DVwUNbi1j5ncQCg/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 11 - Residual networks: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap11PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1Mr58jzEVseUAfNYbGWCQyDtEDwvfHRi1"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1saY8Faz0KTKAAifUrbkQdLA2qkyEjOPI/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 12 - Transformers: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap12PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1txzOVNf8-jH4UfJ6SLnrtOfPd1Q3ebzd"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1GVNvYWa0WJA6oKg89qZre-UZEhABfm0l/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 13 - Graph neural networks: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap13PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1lQIV6nRp6LVfaMgpGFhuwEXG-lTEaAwe"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1YwF3U82c1mQ74c1WqHVTzLZ0j7GgKaWP/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 14 - Unsupervised learning: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap14PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1aMbI6iCuUvOywqk5pBOmppJu1L1anqsM"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1A-lBGv3NHl4L32NvfFgy1EKeSwY-0UeB/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true"> Powerpoint Figures</a>
|
||||
<li> Chapter 15 - Generative adversarial networks: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap15PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1EErnlZCOlXc3HK7m83T2Jh_0NzIUHvtL"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/10Ernk41ShOTf4IYkMD-l4dJfKATkXH4w/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 16 - Normalizing flows: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap16PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1B9bxtmdugwtg-b7Y4AdQKAIEVWxjx8l3"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1nLLzqb9pdfF_h6i1HUDSyp7kSMIkSUUA/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 17 - Variational autoencoders: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap17PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1SNtNIY7khlHQYMtaOH-FosSH3kWwL4b7"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1lQE4Bu7-LgvV2VlJOt_4dQT-kusYl7Vo/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Chapter 18 - Diffusion models: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap18PDF.zip">PDF Figures</a> / <a href="https://docs.google.com/presentation/d/1x_ufIBtVPzWUvRieKMkpw5SdRjXWwdfR/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true"> PowerPoint Figures</a>
|
||||
<li> Chapter 19 - Deep reinforcement learning: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap19PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1a5WUoF7jeSgwC_PVdckJi1Gny46fCqh0"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1TnYmVbFNhmMFetbjyfXGmkxp1EHauMqr/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true"> PowerPoint Figures </a>
|
||||
<li> Chapter 20 - Why does deep learning work?: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap20PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1M2d0DHEgddAQoIedKSDTTt7m1ZdmBLQ3"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1coxF4IsrCzDTLrNjRagHvqB_FBy10miA/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true"> PowerPoint Figures</a>
|
||||
<li> Chapter 21 - Deep learning and ethics: <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLChap21PDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1jixmFfwmZkW_UVYzcxmDcMsdFFtnZ0bU"> SVG Figures</a>/ <a href="https://docs.google.com/presentation/d/1EtfzanZYILvi9_-Idm28zD94I_6OrN9R/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">PowerPoint Figures</a>
|
||||
<li> Appendices - <a href="https://github.com/udlbook/udlbook/raw/main/PDFFigures/UDLAppendixPDF.zip">PDF Figures</a> / <a href="https://drive.google.com/uc?export=download&id=1k2j7hMN40ISPSg9skFYWFL3oZT7r8v-l"> SVG Figures</a> / <a href="https://docs.google.com/presentation/d/1_2cJHRnsoQQHst0rwZssv-XH4o5SEHks/edit?usp=drive_link&ouid=110441678248547154185&rtpof=true&sd=true">Powerpoint Figures</a>
|
||||
</ul>
|
||||
|
||||
Instructions for editing figures / equations can be found <a href="https://drive.google.com/file/d/1T_MXXVR4AfyMnlEFI-UVDh--FXI5deAp/view?usp=sharing">here</a>.</p>
|
||||
|
||||
<h2>Resources for students</h2>
|
||||
|
||||
<p>Answers to selected questions: <a href="https://github.com/udlbook/udlbook/raw/main/UDL_Answer_Booklet_Students.pdf">PDF</a></p>
|
||||
<p>Python notebooks:</p>
|
||||
|
||||
<ul>
|
||||
<li> Notebook 1.1 - Background mathematics: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap01/1_1_BackgroundMathematics.ipynb">ipynb/colab</a>
|
||||
<li> Notebook 2.1 - Supervised learning: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap02/2_1_Supervised_Learning.ipynb"">ipynb/colab</a>
|
||||
<li> Notebook 3.1 - Shallow networks I: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_1_Shallow_Networks_I.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 3.2 - Shallow networks II: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_2_Shallow_Networks_II.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 3.3 - Shallow network regions: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_3_Shallow_Network_Regions.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 3.4 - Activation functions: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap03/3_4_Activation_Functions.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 4.1 - Composing networks: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap04/4_1_Composing_Networks.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 4.2 - Clipping functions: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap04/4_2_Clipping_functions.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 4.3 - Deep networks: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap04/4_3_Deep_Networks.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 5.1 - Least squares loss: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap05/5_1_Least_Squares_Loss.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 5.2 - Binary cross-entropy loss: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap05/5_2_Binary_Cross_Entropy_Loss.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 5.3 - Multiclass cross-entropy loss: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap05/5_3_Multiclass_Cross_entropy_Loss.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 6.1 - Line search: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_1_Line_Search.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 6.2 - Gradient descent: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_2_Gradient_Descent.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 6.3 - Stochastic gradient descent: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_3_Stochastic_Gradient_Descent.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 6.4 - Momentum: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_4_Momentum.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 6.5 - Adam: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap06/6_5_Adam.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 7.1 - Backpropagtion in toy model: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap07/7_1_Backpropagation_in_Toy_Model.ipynb"">ipynb/colab </a>
|
||||
<li> Notebook 7.2 - Backpropagation: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap07/7_2_Backpropagation.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 7.3 - Initialization: <a href=""https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap07/7_3_Initialization.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 8.1 - MNIST-1D performance: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_1_MNIST_1D_Performance.ipynb"">ipynb/colab </a>
|
||||
<li> Notebook 8.2 - Bias-variance trade-off: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_2_Bias_Variance_Trade_Off.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 8.3 - Double descent: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_3_Double_Descent.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 8.4 - High-dimensional spaces: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap08/8_4_High_Dimensional_Spaces.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 9.1 - L2 regularization: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_1_L2_Regularization.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 9.2 - Implicit regularization: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_2_Implicit_Regularization.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 9.3 - Ensembling: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_3_Ensembling.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 9.4 - Bayesian approach: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_4_Bayesian_Approach.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 9.5 - Augmentation <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap09/9_5_Augmentation.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 10.1 - 1D convolution: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_1_1D_Convolution.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 10.2 - Convolution for MNIST-1D: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_2_Convolution_for_MNIST_1D.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 10.3 - 2D convolution: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_3_2D_Convolution.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 10.4 - Downsampling & upsampling: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_4_Downsampling_and_Upsampling.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 10.5 - Convolution for MNIST: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap10/10_5_Convolution_For_MNIST.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 11.1 - Shattered gradients: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap11/11_1_Shattered_Gradients.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 11.2 - Residual networks: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap11/11_2_Residual_Networks.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 11.3 - Batch normalization: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap11/11_3_Batch_Normalization.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 12.1 - Self-attention: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_1_Self_Attention.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 12.2 - Multi-head self-attention: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_2_Multihead_Self_Attention.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 12.3 - Tokenization: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_3_Tokenization.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 12.4 - Decoding strategies: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap12/12_4_Decoding_Strategies.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 13.1 - Encoding graphs: <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap13/13_1_Graph_Representation.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 13.2 - Graph classification : <a href="https://github.com/udlbook/udlbook/blob/main/Notebooks/Chap13/13_2_Graph_Classification.ipynb">ipynb/colab </a>
|
||||
<li> Notebook 13.3 - Neighborhood sampling: (coming soon)
|
||||
<li> Notebook 13.4 - Graph attention: (coming soon)
|
||||
<li> Notebook 15.1 - GAN toy example: (coming soon)
|
||||
<li> Notebook 15.2 - Wasserstein distance: (coming soon)
|
||||
<li> Notebook 16.1 - 1D normalizing flows: (coming soon)
|
||||
<li> Notebook 16.2 - Autoregressive flows: (coming soon)
|
||||
<li> Notebook 16.3 - Contraction mappings: (coming soon)
|
||||
<li> Notebook 17.1 - Latent variable models: (coming soon)
|
||||
<li> Notebook 17.2 - Reparameterization trick: (coming soon)
|
||||
<li> Notebook 17.3 - Importance sampling: (coming soon)
|
||||
<li> Notebook 18.1 - Diffusion encoder: (coming soon)
|
||||
<li> Notebook 18.2 - 1D diffusion model: (coming soon)
|
||||
<li> Notebook 18.3 - Reparameterized model: (coming soon)
|
||||
<li> Notebook 18.4 - Families of diffusion models: (coming soon)
|
||||
<li> Notebook 19.1 - Markov decision processes: (coming soon)
|
||||
<li> Notebook 19.2 - Dynamic programming: (coming soon)
|
||||
<li> Notebook 19.3 - Monte-Carlo methods: (coming soon)
|
||||
<li> Notebook 19.4 - Temporal difference methods: (coming soon)
|
||||
<li> Notebook 19.5 - Control variates: (coming soon)
|
||||
<li> Notebook 20.1 - Random data: (coming soon)
|
||||
<li> Notebook 20.2 - Full-batch gradient descent: (coming soon)
|
||||
<li> Notebook 20.3 - Lottery tickets: (coming soon)
|
||||
<li> Notebook 20.4 - Adversarial attacks: (coming soon)
|
||||
<li> Notebook 21.1 - Bias mitigation: (coming soon)
|
||||
<li> Notebook 21.2 - Explainability: (coming soon)
|
||||
</ul>
|
||||
|
||||
|
||||
<br>
|
||||
<h2>Citation:</h2>
|
||||
<pre><code>
|
||||
<br>
|
||||
<h2>Citation</h2>
|
||||
<pre><code>
|
||||
@book{prince2023understanding,
|
||||
author = "Simon J.D. Prince",
|
||||
title = "Understanding Deep Learning",
|
||||
@@ -163,4 +378,6 @@ Instructions for editing figures / equations can be found <a href="https://drive
|
||||
year = 2023,
|
||||
url = "http://udlbook.com"
|
||||
}
|
||||
</code></pre>
|
||||
</code></pre>
|
||||
</div>
|
||||
</body>
|
||||
23
style.css
Normal file
23
style.css
Normal file
@@ -0,0 +1,23 @@
|
||||
body {
|
||||
font-size: 17px;
|
||||
margin: 2% 10%;
|
||||
}
|
||||
|
||||
#head {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
flex-wrap: wrap-reverse;
|
||||
justify-content: space-between;
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
#cover {
|
||||
justify-content: center;
|
||||
display: flex;
|
||||
width: 30%;
|
||||
}
|
||||
|
||||
#cover img {
|
||||
width: 100%;
|
||||
height: min-content;
|
||||
}
|
||||
Reference in New Issue
Block a user