dr_domi/udlbook
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add files via upload

Browse Source
This commit is contained in:
udlbook
2024-01-02 13:09:22 -05:00
committed by GitHub
parent db836826f6
commit c19e2411c5
3 changed files with 471 additions and 442 deletions
Download Patch File Download Diff File Expand all files Collapse all files

301
Notebooks/Chap17/17_1_Latent_Variable_Models.ipynb
View File

@@ -1,33 +1,22 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"authorship_tag": "ABX9TyOSEQVqxE5KrXmsZVh9M3gq",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
"colab_type": "text",
"id": "view-in-github"
},
"source": [
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap17/17_1_Latent_Variable_Models.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "t9vk9Elugvmi"
},
"source": [
"# **Notebook 17.1: Latent variable models**\n",
"\n",
@@ -36,72 +25,76 @@
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
"\n",
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
],
"metadata": {
"id": "t9vk9Elugvmi"
}
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OLComQyvCIJ7"
},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import scipy\n",
"from matplotlib.colors import ListedColormap\n",
"from matplotlib import cm"
],
"metadata": {
"id": "OLComQyvCIJ7"
},
"execution_count": null,
"outputs": []
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "IyVn-Gi-p7wf"
},
"source": [
"We'll assume that our base distribution over the latent variables is a 1D standard normal so that\n",
"\n",
"\\begin{equation}\n",
"Pr(z) = \\mbox{Norm}_{z}[0,1]\n",
"Pr(z) = \\text{Norm}_{z}[0,1]\n",
"\\end{equation}\n",
"\n",
"As in figure 17.2, we'll assume that the output is two dimensional, we we need to define a function that maps from the 1D latent variable to two dimensions. Usually, we would use a neural network, but in this case, we'll just define an arbitrary relationship.\n",
"\n",
"\\begin{eqnarray}\n",
"\\begin{align}\n",
"x_{1} &=& 0.5\\cdot\\exp\\Bigl[\\sin\\bigl[2+ 3.675 z \\bigr]\\Bigr]\\\\\n",
"x_{2} &=& \\sin\\bigl[2+ 2.85 z \\bigr]\n",
"\\end{eqnarray}"
],
"metadata": {
"id": "IyVn-Gi-p7wf"
}
"\\end{align}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ZIfQwhd-AV6L"
},
"outputs": [],
"source": [
"# The function that maps z to x1 and x2\n",
"def f(z):\n",
" x_1 = np.exp(np.sin(2+z*3.675)) * 0.5\n",
" x_2 = np.cos(2+z*2.85)\n",
" return x_1, x_2"
],
"metadata": {
"id": "ZIfQwhd-AV6L"
},
"execution_count": null,
"outputs": []
]
},
{
"attachments": {},
"cell_type": "markdown",
"source": [
"Let's plot the 3D relation between the two observed variables $x_{1}$ and $x_{2}$ and the latent variables $z$ as in figure 17.2 of the book. We'll use the opacity to represent the prior probability $Pr(z)$."
],
"metadata": {
"id": "KB9FU34onW1j"
}
},
"source": [
"Let's plot the 3D relation between the two observed variables $x_{1}$ and $x_{2}$ and the latent variables $z$ as in figure 17.2 of the book. We'll use the opacity to represent the prior probability $Pr(z)$."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "lW08xqAgnP4q"
},
"outputs": [],
"source": [
"def draw_3d_projection(z,pr_z, x1,x2):\n",
" alpha = pr_z / np.max(pr_z)\n",
@@ -118,28 +111,28 @@
" ax.set_zlim(-1,1)\n",
" ax.set_box_aspect((3,1,1))\n",
" plt.show()"
],
"metadata": {
"id": "lW08xqAgnP4q"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9DUTauMi6tPk"
},
"outputs": [],
"source": [
"# Compute the prior\n",
"def get_prior(z):\n",
" return scipy.stats.multivariate_normal.pdf(z)"
],
"metadata": {
"id": "9DUTauMi6tPk"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PAzHq461VqvF"
},
"outputs": [],
"source": [
"# Define the latent variable values\n",
"z = np.arange(-3.0,3.0,0.01)\n",
@@ -149,40 +142,41 @@
"x1,x2 = f(z)\n",
"# Plot the function\n",
"draw_3d_projection(z,pr_z, x1,x2)"
],
"metadata": {
"id": "PAzHq461VqvF"
},
"execution_count": null,
"outputs": []
]
},
{
"attachments": {},
"cell_type": "markdown",
"source": [
"The likelihood is defined as:\n",
"\\begin{eqnarray}\n",
" Pr(x_1,x_2|z) &=& \\mbox{Norm}_{[x_1,x_2]}\\Bigl[\\mathbf{f}[z],\\sigma^{2}\\mathbf{I}\\Bigr]\n",
"\\end{eqnarray}\n",
"\n",
"so we will also need to define the noise level $\\sigma^2$"
],
"metadata": {
"id": "sQg2gKR5zMrF"
}
},
"source": [
"The likelihood is defined as:\n",
"\\begin{align}\n",
" Pr(x_1,x_2|z) &=& \\text{Norm}_{[x_1,x_2]}\\Bigl[\\mathbf{f}[z],\\sigma^{2}\\mathbf{I}\\Bigr]\n",
"\\end{align}\n",
"\n",
"so we will also need to define the noise level $\\sigma^2$"
]
},
{
"cell_type": "code",
"source": [
"sigma_sq = 0.04"
],
"execution_count": null,
"metadata": {
"id": "In_Vg4_0nva3"
},
"execution_count": null,
"outputs": []
"outputs": [],
"source": [
"sigma_sq = 0.04"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6P6d-AgAqxXZ"
},
"outputs": [],
"source": [
"# Draws a heatmap to represent a probability distribution, possibly with samples overlaed\n",
"def plot_heatmap(x1_mesh,x2_mesh,y_mesh, x1_samples=None, x2_samples=None, title=None):\n",
@@ -207,15 +201,15 @@
" ax.set_xlabel('$x_1$'); ax.set_ylabel('$x_2$')\n",
" plt.show()\n",
"\n"
],
"metadata": {
"id": "6P6d-AgAqxXZ"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "diYKb7_ZgjlJ"
},
"outputs": [],
"source": [
"# Returns the likelihood\n",
"def get_likelihood(x1_mesh, x2_mesh, z_val):\n",
@@ -226,24 +220,25 @@
" mn = scipy.stats.multivariate_normal([x1, x2], [[sigma_sq, 0], [0, sigma_sq]])\n",
" pr_x1_x2_given_z_val = mn.pdf(np.dstack((x1_mesh, x2_mesh)))\n",
" return pr_x1_x2_given_z_val"
],
"metadata": {
"id": "diYKb7_ZgjlJ"
},
"execution_count": null,
"outputs": []
]
},
{
"attachments": {},
"cell_type": "markdown",
"source": [
"Now let's plot the likelihood $Pr(x_1,x_2|z)$ as in fig 17.3b in the book."
],
"metadata": {
"id": "0X4NwixzqxtZ"
}
},
"source": [
"Now let's plot the likelihood $Pr(x_1,x_2|z)$ as in fig 17.3b in the book."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hWfqK-Oz5_DT"
},
"outputs": [],
"source": [
"# Choose some z value\n",
"z_val = 1.8\n",
@@ -256,30 +251,31 @@
"plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2_given_z_val, title=\"Conditional distribution $Pr(x_1,x_2|z)$\")\n",
"\n",
"# TODO -- Experiment with different values of z and make sure that you understand the what is happening."
],
"metadata": {
"id": "hWfqK-Oz5_DT"
},
"execution_count": null,
"outputs": []
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "25xqXnmFo-PH"
},
"source": [
"The data density is found by marginalizing over the latent variables $z$:\n",
"\n",
"\\begin{eqnarray}\n",
"\\begin{align}\n",
" Pr(x_1,x_2) &=& \\int Pr(x_1,x_2, z) dz \\nonumber \\\\\n",
" &=& \\int Pr(x_1,x_2 | z) \\cdot Pr(z)dz\\nonumber \\\\\n",
" &=& \\int \\mbox{Norm}_{[x_1,x_2]}\\Bigl[\\mathbf{f}[z],\\sigma^{2}\\mathbf{I}\\Bigr]\\cdot \\mbox{Norm}_{z}\\left[\\mathbf{0},\\mathbf{I}\\right]dz.\n",
"\\end{eqnarray}"
],
"metadata": {
"id": "25xqXnmFo-PH"
}
" &=& \\int \\text{Norm}_{[x_1,x_2]}\\Bigl[\\mathbf{f}[z],\\sigma^{2}\\mathbf{I}\\Bigr]\\cdot \\text{Norm}_{z}\\left[\\mathbf{0},\\mathbf{I}\\right]dz.\n",
"\\end{align}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "H0Ijce9VzeCO"
},
"outputs": [],
"source": [
"# TODO Compute the data density\n",
"# We can't integrate this function in closed form\n",
@@ -293,24 +289,25 @@
"\n",
"# Plot the result\n",
"plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2, title=\"Data density $Pr(x_1,x_2)$\")\n"
],
"metadata": {
"id": "H0Ijce9VzeCO"
},
"execution_count": null,
"outputs": []
]
},
{
"attachments": {},
"cell_type": "markdown",
"source": [
"Now let's draw some samples from the model"
],
"metadata": {
"id": "W264N7By_h9y"
}
},
"source": [
"Now let's draw some samples from the model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Li3mK_I48k0k"
},
"outputs": [],
"source": [
"def draw_samples(n_sample):\n",
" # TODO Write this routine to draw n_sample samples from the model\n",
@@ -320,37 +317,38 @@
" x1_samples=0; x2_samples = 0;\n",
"\n",
" return x1_samples, x2_samples"
],
"metadata": {
"id": "Li3mK_I48k0k"
},
"execution_count": null,
"outputs": []
]
},
{
"attachments": {},
"cell_type": "markdown",
"source": [
"Let's plot those samples on top of the heat map."
],
"metadata": {
"id": "D7N7oqLe-eJO"
}
},
"source": [
"Let's plot those samples on top of the heat map."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XRmWv99B-BWO"
},
"outputs": [],
"source": [
"x1_samples, x2_samples = draw_samples(500)\n",
"# Plot the result\n",
"plot_heatmap(x1_mesh, x2_mesh, pr_x1_x2, x1_samples, x2_samples, title=\"Data density $Pr(x_1,x_2)$\")\n"
],
"metadata": {
"id": "XRmWv99B-BWO"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PwOjzPD5_1OF"
},
"outputs": [],
"source": [
"# Return the posterior distribution\n",
"def get_posterior(x1,x2):\n",
@@ -364,15 +362,15 @@
"\n",
"\n",
" return z, pr_z_given_x1_x2"
],
"metadata": {
"id": "PwOjzPD5_1OF"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PKFUY42K-Tp7"
},
"outputs": [],
"source": [
"x1 = 0.9; x2 = -0.9\n",
"z, pr_z_given_x1_x2 = get_posterior(x1,x2)\n",
@@ -385,12 +383,23 @@
"ax.set_xlim([-3,3])\n",
"ax.set_ylim([0,1.5 * np.max(pr_z_given_x1_x2)])\n",
"plt.show()"
],
"metadata": {
"id": "PKFUY42K-Tp7"
},
"execution_count": null,
"outputs": []
]
}
]
}
],
"metadata": {
"colab": {
"authorship_tag": "ABX9TyOSEQVqxE5KrXmsZVh9M3gq",
"include_colab_link": true,
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}