Created using Colaboratory
This commit is contained in:
udlbook
525
CM20315_2023/CM20315_Coursework_V_2023.ipynb
Normal file
525
CM20315_2023/CM20315_Coursework_V_2023.ipynb
Normal file
@@ -0,0 +1,525 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"authorship_tag": "ABX9TyM4zwYuFtnFcaVHhdLoTGCj",
|
||||
"include_colab_link": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "view-in-github",
|
||||
"colab_type": "text"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/CM20315_2023/CM20315_Coursework_V_2023.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# **Coursework: Backpropagation in Toy Model**\n",
|
||||
"\n",
|
||||
"This notebook computes the derivatives of a toy function similar (but different from) that in section 7.3 of the book.\n",
|
||||
"\n",
|
||||
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions. At various points, you will get an answer that you need to copy into Moodle to be marked.\n",
|
||||
"\n",
|
||||
"Post to the content forum if you find any mistakes or need to clarify something."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "pOZ6Djz0dhoy"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Problem setting\n",
|
||||
"\n",
|
||||
"We're going to investigate how to take the derivatives of functions where one operation is composed with another, which is composed with a third and so on. For example, consider the model:\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
" \\mbox{f}[x,\\boldsymbol\\phi] = \\beta_3+\\omega_3\\cdot\\mbox{PReLU}\\Bigl[\\gamma, \\beta_2+\\omega_2\\cdot\\mbox{PReLU}\\bigl[\\gamma, \\beta_1+\\omega_1\\cdot\\mbox{PReLU}[\\gamma, \\beta_0+\\omega_0x]\\bigr]\\Bigr],\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"with parameters $\\boldsymbol\\phi=\\{\\beta_0,\\omega_0,\\beta_1,\\omega_1,\\beta_2,\\omega_2,\\beta_3,\\omega_3\\}$, where\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\mbox{PReLU}[\\gamma, z] = \\begin{cases} \\gamma\\cdot z & \\quad z <0 \\\\ z & \\quad z\\geq 0\\end{cases}.\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"Suppose that we have a binary cross-entropy loss function (equation 5.20 from the book):\n",
|
||||
"\n",
|
||||
"\\begin{equation*}\n",
|
||||
"\\ell_i = -(1-y_{i})\\log\\Bigl[1-\\mbox{sig}[\\mbox{f}[\\mathbf{x}_i,\\boldsymbol\\phi]]\\Bigr] - y_{i}\\log\\Bigl[\\mbox{sig}[\\mbox{f}[\\mathbf{x}_i,\\boldsymbol\\phi]]\\Bigr].\n",
|
||||
"\\end{equation*}\n",
|
||||
"\n",
|
||||
"Assume that we know the current values of $\\beta_{0},\\beta_{1},\\beta_{2},\\beta_{3},\\omega_{0},\\omega_{1},\\omega_{2},\\omega_{3}$, $\\gamma$, $x_i$ and $y_i$. We want to know how $\\ell_i$ changes when we make a small change to $\\beta_{0},\\beta_{1},\\beta_{2},\\beta_{3},\\omega_{0},\\omega_{1},\\omega_{2}$, or $\\omega_{3}$. In other words, we want to compute the eight derivatives:\n",
|
||||
"\n",
|
||||
"\\begin{eqnarray*}\n",
|
||||
"\\frac{\\partial \\ell_i}{\\partial \\beta_{0}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\beta_{1}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\beta_{2}}, \\quad \\frac{\\partial \\ell_i }{\\partial \\beta_{3}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{0}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{1}}, \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{2}}, \\quad\\mbox{and} \\quad \\frac{\\partial \\ell_i}{\\partial \\omega_{3}}.\n",
|
||||
"\\end{eqnarray*}"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "1DmMo2w63CmT"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# import library\n",
|
||||
"import numpy as np"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "RIPaoVN834Lj"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's first define the original function and the loss term:"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "32-ufWhc3v2c"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "AakK_qen3BpU"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Defines the activation function\n",
|
||||
"def paramReLU(gamma,x):\n",
|
||||
" if x > 0:\n",
|
||||
" return x\n",
|
||||
" else:\n",
|
||||
" return x * gamma\n",
|
||||
"\n",
|
||||
"# Defines the main function\n",
|
||||
"def fn(x, beta0, beta1, beta2, beta3, omega0, omega1, omega2, omega3, gamma):\n",
|
||||
" return beta3+omega3 * paramReLU(gamma, beta2 + omega2 * paramReLU(gamma, beta1 + omega1 * paramReLU(gamma, beta0 + omega0 * x)))\n",
|
||||
"\n",
|
||||
"# Logistic sigmoid\n",
|
||||
"def sig(z):\n",
|
||||
" return 1./(1+np.exp(-z))\n",
|
||||
"\n",
|
||||
"# The loss function (equation 5.20 from book)\n",
|
||||
"def loss(f,y):\n",
|
||||
" sig_net_out = sig(f)\n",
|
||||
" l = -(1-y) * np.log(1-sig_net_out) - y * np.log(sig_net_out)\n",
|
||||
" return l"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now we'll choose some values for the betas and the omegas and x and compute the output of the function:"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "y7tf0ZMt5OXt"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"beta0 = 1.0; beta1 = -2.0; beta2 = -3.0; beta3 = 0.4\n",
|
||||
"omega0 = 0.1; omega1 = -0.4; omega2 = 2.0; omega3 = -3.0\n",
|
||||
"gamma = 0.2\n",
|
||||
"x = 2.3; y =1.0\n",
|
||||
"f_val = fn(x,beta0,beta1,beta2,beta3,omega0,omega1,omega2,omega3, gamma)\n",
|
||||
"l_i_func = loss(f_val, y)\n",
|
||||
"print('Loss full function = %3.3f'%l_i_func)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "pwvOcCxr41X_"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Forward pass\n",
|
||||
"\n",
|
||||
"We compute a series of intermediate values $f_0, h_0, f_1, h_1, f_2, h_2, f_3$, and finally the loss $\\ell$"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "W6ZP62T5fU64"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"x = 2.3; y =1.0\n",
|
||||
"gamma = 0.2\n",
|
||||
"# Compute all the f_k and h_k terms\n",
|
||||
"# I've done the first two for you\n",
|
||||
"f0 = beta0+omega0 * x\n",
|
||||
"h1 = paramReLU(gamma, f0)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# TODO: Replace the code below\n",
|
||||
"f1 = 0\n",
|
||||
"h2 = 0\n",
|
||||
"f2 = 0\n",
|
||||
"h3 = 0\n",
|
||||
"f3 = 0\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Compute the loss and print\n",
|
||||
"# The answer should be the same as when we computed the full function above\n",
|
||||
"l_i = loss(f3, y)\n",
|
||||
"print(\"Loss forward pass = %3.3f\"%(l_i))\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "z-BckTpMf5PL"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Backward pass: Derivative of loss function with respect to function output\n",
|
||||
"\n",
|
||||
"Now, we'll compute the derivative $\\frac{dl}{df_3}$ of the loss function with respect to the network output $f_3$. In other words, we are asking how does the loss change as we make a small change in the network output.\n",
|
||||
"\n",
|
||||
"Since the loss it itself a function of $\\mbox{sig}[f_3]$ we'll compute this using the chain rule:\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\frac{dl}{df_3} = \\frac{d\\mbox{sig}[f_3]}{df_3}\\cdot \\frac{dl}{d\\mbox{sig}[f_3]}\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"Your job is to compute the two quantities on the right hand side.\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "TbFbxz64Xz4I"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Compute the derivative of the the loss with respect to the function output f_val\n",
|
||||
"def dl_df(f_val,y):\n",
|
||||
" # Compute sigmoid of network output\n",
|
||||
" sig_f_val = sig(f_val)\n",
|
||||
" # Compute the derivative of loss with respect to network output using chain rule\n",
|
||||
" dl_df_val = dsig_df(f_val) * dl_dsigf(sig_f_val, y)\n",
|
||||
" # Return the derivative\n",
|
||||
" return dl_df_val"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ZWKAq6HC90qV"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# MOODLE ANSWER # Notebook V 1a: Copy this code when you have finished it.\n",
|
||||
"\n",
|
||||
"# Compute the derivative of the logistic sigmoid function with respect to its input (as a closed form solution)\n",
|
||||
"def dsig_df(f_val):\n",
|
||||
" # TODO Write this function\n",
|
||||
" # Replace this line:\n",
|
||||
" return 1\n",
|
||||
"\n",
|
||||
"# Compute the derivative of the loss with respect to the logistic sigmoid (as a closed form solution)\n",
|
||||
"def dl_dsigf(sig_f_val, y):\n",
|
||||
" # TODO Write this function\n",
|
||||
" # Replace this line:\n",
|
||||
" return 1"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "lIngYAgPq-5I"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's run that for some f_val, y. Check previous practicals to see how you can check whether your answer is correct."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Q-j-i8khXzbK"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"y = 0.0\n",
|
||||
"dl_df3 = dl_df(f3,y)\n",
|
||||
"print(\"Moodle Answer Notebook V 1b: dldh3=%3.3f\"%(dl_df3))\n",
|
||||
"\n",
|
||||
"y= 1.0\n",
|
||||
"dl_df3 = dl_df(f3,y)\n",
|
||||
"print(\"Moodle Answer Notebook V 1c: dldh3=%3.3f\"%(dl_df3))"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Z7Lb5BibY50H"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Backward pass: Derivative of activation function with respect to preactivations\n",
|
||||
"\n",
|
||||
"Write a function to compute the derivative $\\frac{\\partial h}{\\partial f}$ of the activation function (parametric ReLU) with respect to its input.\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "BA7QaOzejzZw"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# MOODLE ANSWER Notebook V 2a: Copy this code when you have finished it.\n",
|
||||
"\n",
|
||||
"def dh_df(gamma, f_val):\n",
|
||||
" # TODO: Write this function\n",
|
||||
" # Replace this line:\n",
|
||||
" return 1\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "bBPfPg04j-Qw"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's run that for some values of f_val. Check previous practicals to see how you can check whether your answer is correct."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "QRNCM0EGk9-w"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"f_val_test = 0.6\n",
|
||||
"dh_df_val = dh_df(gamma, f_val_test)\n",
|
||||
"print(\"Moodle Answer Notebook V 2b: dhdf=%3.3f\"%(dh_df_val))\n",
|
||||
"\n",
|
||||
"f_val_test = -0.4\n",
|
||||
"dh_df_val = dh_df(gamma, f_val_test)\n",
|
||||
"print(\"Moodle Answer Notebook V 2c: dhdf=%3.3f\"%(dh_df_val))"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "bql8VZIGk8Wy"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
" # Backward pass: Compute the derivatives of $l_i$ with respect to the intermediate quantities but in reverse order:\n",
|
||||
"\n",
|
||||
"\\begin{eqnarray}\n",
|
||||
"\\frac{\\partial \\ell_i}{\\partial h_3}, \\quad \\frac{\\partial \\ell_i}{\\partial f_2}, \\quad\n",
|
||||
"\\frac{\\partial \\ell_i}{\\partial h_2}, \\quad \\frac{\\partial \\ell_i}{\\partial f_1}, \\quad\\frac{\\partial \\ell_i}{\\partial h_1}, \\quad\\mbox{and} \\quad \\frac{\\partial \\ell_i}{\\partial f_0}.\n",
|
||||
"\\end{eqnarray}\n",
|
||||
"\n",
|
||||
"The first of these derivatives can be calculated using the chain rule:\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\frac{\\partial \\ell_i}{\\partial h_{3}} =\\frac{\\partial f_{3}}{\\partial h_{3}} \\frac{\\partial \\ell_i}{\\partial f_{3}} .\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"The left-hand side asks how $\\ell_i$ changes when $h_{3}$ changes. The right-hand side says we can decompose this into (i) how $\\ell_i$ changes when $f_{3}$ changes and how $f_{3}$ changes when $h_{3}$ changes. So you get a chain of events happening: $h_{3}$ changes $f_{3}$, which changes $\\ell_i$, and the derivatives represent the effects of this chain. Notice that we computed the first of these derivatives already. The second term is the derivative of $\\beta_{3} + \\omega_{3}h_{3}$ with respect to $h_3$ which is simply $\\omega_3$. \n",
|
||||
"\n",
|
||||
"We can continue in this way, computing the derivatives of the output with respect to these intermediate quantities:\n",
|
||||
"\n",
|
||||
"\\begin{eqnarray}\n",
|
||||
"\\frac{\\partial \\ell_i}{\\partial f_{2}} &=& \\frac{\\partial h_{3}}{\\partial f_{2}}\\left(\n",
|
||||
"\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}} \\right)\n",
|
||||
"\\nonumber \\\\\n",
|
||||
"\\frac{\\partial \\ell_i}{\\partial h_{2}} &=& \\frac{\\partial f_{2}}{\\partial h_{2}}\\left(\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}}\\right)\\nonumber \\\\\n",
|
||||
"\\frac{\\partial \\ell_i}{\\partial f_{1}} &=& \\frac{\\partial h_{2}}{\\partial f_{1}}\\left( \\frac{\\partial f_{2}}{\\partial h_{2}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}} \\right)\\nonumber \\\\\n",
|
||||
"\\frac{\\partial \\ell_i}{\\partial h_{1}} &=& \\frac{\\partial f_{1}}{\\partial h_{1}}\\left(\\frac{\\partial h_{2}}{\\partial f_{1}} \\frac{\\partial f_{2}}{\\partial h_{2}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}} \\right)\\nonumber \\\\\n",
|
||||
"\\frac{\\partial \\ell_i}{\\partial f_{0}} &=& \\frac{\\partial h_{1}}{\\partial f_{0}}\\left(\\frac{\\partial f_{1}}{\\partial h_{1}}\\frac{\\partial h_{2}}{\\partial f_{1}} \\frac{\\partial f_{2}}{\\partial h_{2}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial \\ell_i}{\\partial f_{3}} \\right).\n",
|
||||
"\\end{eqnarray}\n",
|
||||
"\n",
|
||||
"In each case, we have already computed all of the terms except the last one in the previous step, and the last term is simple to evaluate. This is called the **backward pass**."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "jay8NYWdFHuZ"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"x = 2.3; y =1.0\n",
|
||||
"dldf3 = dl_df(f3,y)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "RSC_2CIfKF1b"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# MOODLE ANSWER Notebook V 3a: Copy this code when you have finished it.\n",
|
||||
"# TODO -- Compute the derivatives of the output with respect\n",
|
||||
"# to the intermediate computations h_k and f_k (i.e, run the backward pass)\n",
|
||||
"# I've done the first two for you. You replace the code below:\n",
|
||||
"# Replace the code below\n",
|
||||
"dldh3 = 1\n",
|
||||
"dldf2 = 1\n",
|
||||
"dldh2 = 1\n",
|
||||
"dldf1 = 1\n",
|
||||
"dldh1 = 1\n",
|
||||
"dldf0 = 1"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "gCQJeI--Egdl"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Finally, we consider how the loss~$\\ell_{i}$ changes when we change the parameters $\\beta_{\\bullet}$ and $\\omega_{\\bullet}$. Once more, we apply the chain rule:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\\begin{eqnarray}\n",
|
||||
"\\frac{\\partial \\ell_i}{\\partial \\beta_{k}} &=& \\frac{\\partial f_{k}}{\\partial \\beta_{k}}\\frac{\\partial \\ell_i}{\\partial f_{k}}\\nonumber \\\\\n",
|
||||
"\\frac{\\partial \\ell_i}{\\partial \\omega_{k}} &=& \\frac{\\partial f_{k}}{\\partial \\omega_{k}}\\frac{\\partial \\ell_i}{\\partial f_{k}}.\n",
|
||||
"\\end{eqnarray}\n",
|
||||
"\n",
|
||||
"\\noindent In each case, the second term on the right-hand side was computed in step 2. When $k>0$, we have~$f_{k}=\\beta_{k}+\\omega_k \\cdot h_{k}$, so:\n",
|
||||
"\n",
|
||||
"\\begin{eqnarray}\n",
|
||||
"\\frac{\\partial f_{k}}{\\partial \\beta_{k}} = 1 \\quad\\quad\\mbox{and}\\quad \\quad \\frac{\\partial f_{k}}{\\partial \\omega_{k}} &=& h_{k}.\n",
|
||||
"\\end{eqnarray}"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "FlzlThQPGpkU"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# MOODLE ANSWER Notebook V 3b: Copy this code when you have finished it.\n",
|
||||
"# TODO -- Calculate the final derivatives with respect to the beta and omega terms\n",
|
||||
"# Replace these terms\n",
|
||||
"dldbeta3 = 1\n",
|
||||
"dldomega3 = 1\n",
|
||||
"dldbeta2 = 1\n",
|
||||
"dldomega2 = 1\n",
|
||||
"dldbeta1 = 1\n",
|
||||
"dldomega1 = 1\n",
|
||||
"dldbeta0 = 1\n",
|
||||
"dldomega0 = 1"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "1I2BhqZhGMK6"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Print the last two values out (enter these into Moodle). Again, think about how you can test whether these are correct.\n",
|
||||
"print('Moodle Answer Notebook V 3c: dldbeta0=%3.3f'%(dldbeta0))\n",
|
||||
"print('Moodle Answer Notebook V 3d: dldOmega0=%3.3f'%(dldomega0))"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "38eiOn2aHgHI"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Compute the derivatives of $\\ell_i$ with respect to the parmeter $\\gamma$ of the parametric ReLU function. \n",
|
||||
"\n",
|
||||
"In other words, compute:\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\frac{d\\ell_i}{d\\gamma}\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"Along the way, we will need to compute derivatives\n",
|
||||
"\n",
|
||||
"\\begin{equation}\n",
|
||||
"\\frac{dh_k(\\gamma,f_{k-1})}{d\\gamma}\n",
|
||||
"\\end{equation}\n",
|
||||
"\n",
|
||||
"This is quite difficult and not worth many marks, so don't spend too much time on it if you are confused!"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "lhD5AoUHx3DS"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Computes how an activation changes with a small change in gamma assuming preactivations are f\n",
|
||||
"# MOODLE ANSWER # Notebook V 4a: Copy this code when you have finished it.\n",
|
||||
"def dhdgamma(gamma, f):\n",
|
||||
" # TODO -- Write this function\n",
|
||||
" # Replace this line\n",
|
||||
" return 1"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "yC-9MTQevliP"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Compute how the loss changes with gamma\n",
|
||||
"# Replace this line:\n",
|
||||
"# MOODLE ANSWER # Notebook V 4b: Copy this code when you have finished it.\n",
|
||||
"dldgamma = 1"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "DiNQrveoLuHR"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"print(\"Moodle Answer Notebook V 4c: dldgamma = %3.3f\"%(dldgamma))"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "YHxmAEnxzy3O"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user