{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "provenance": [], "authorship_tag": "ABX9TyPZzptvvf7OPZai8erQ/0xT", "include_colab_link": true }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "view-in-github", "colab_type": "text" }, "source": [ "\"Open" ] }, { "cell_type": "markdown", "source": [ "#Notebook 4.2 -- Clipping functions\n", "\n", "The purpose of this notebook is to understand how a neural network with two hidden layers build more complicated functions by clipping and recombining the representations at the intermediate hidden variables.\n", "\n", "Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n", "\n", "Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions" ], "metadata": { "id": "MaKn8CFlzN8E" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "8ClURpZQzI6L" }, "outputs": [], "source": [ "# Imports math library\n", "import numpy as np\n", "# Imports plotting library\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "source": [ "# Define the Rectified Linear Unit (ReLU) function\n", "def ReLU(preactivation):\n", " activation = preactivation.clip(0.0)\n", " return activation" ], "metadata": { "id": "YdmveeAUz4YG" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "# Define a deep neural network with, one input, one output, two hidden layers and three hidden units (eqns 4.7-4.9)\n", "# To make this easier, we store the parameters in ndarrays, so phi_0 = phi[0] and psi_3,3 = psi[3,3] etc.\n", "def shallow_1_1_3_3(x, activation_fn, phi, psi, theta):\n", "\n", " # TODO -- You write this function\n", " # Replace the skeleton code below.\n", "\n", " # ANSWER\n", " # Preactivations at layer 1 (terms in brackets in equation 4.7)\n", " layer1_pre_1 = np.zeros_like(x) ;\n", " layer1_pre_2 = np.zeros_like(x) ;\n", " layer1_pre_3 = np.zeros_like(x) ;\n", "\n", " # Activation functions (rest of equation 4.7)\n", " h1 = activation_fn(layer1_pre_1)\n", " h2 = activation_fn(layer1_pre_2)\n", " h3 = activation_fn(layer1_pre_3)\n", "\n", " # Preactivations at layer 2 (terms in brackets in equation 4.8)\n", " layer2_pre_1 = np.zeros_like(x) ;\n", " layer2_pre_2 = np.zeros_like(x) ;\n", " layer2_pre_3 = np.zeros_like(x) ;\n", "\n", " # Activation functions (rest of equation 4.8)\n", " h1_prime = activation_fn(layer2_pre_1)\n", " h2_prime = activation_fn(layer2_pre_2)\n", " h3_prime = activation_fn(layer2_pre_3)\n", "\n", " # Weighted outputs by phi (three last terms of equation 4.9)\n", " phi1_h1_prime = np.zeros_like(x) ;\n", " phi2_h2_prime = np.zeros_like(x) ;\n", " phi3_h3_prime = np.zeros_like(x) ;\n", "\n", " # Combine weighted activation and add y offset (summing terms of equation 4.9)\n", " y = np.zeros_like(x) ;\n", "\n", "\n", " # Return everything we have calculated\n", " return y, layer2_pre_1, layer2_pre_2, layer2_pre_3, h1_prime, h2_prime, h3_prime, phi1_h1_prime, phi2_h2_prime, phi3_h3_prime" ], "metadata": { "id": "ximCLwIfz8kj" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "# # Plot two layer neural network as in figure 4.5\n", "def plot_neural_two_layers(x, y, layer2_pre_1, layer2_pre_2, layer2_pre_3, h1_prime, h2_prime, h3_prime, phi1_h1_prime, phi2_h2_prime, phi3_h3_prime):\n", "\n", " fig, ax = plt.subplots(3,3)\n", " fig.set_size_inches(8.5, 8.5)\n", " fig.tight_layout(pad=3.0)\n", " ax[0,0].plot(x,layer2_pre_1,'r-'); ax[0,0].set_ylabel(r'$\\psi_{10}+\\psi_{11}h_{1}+\\psi_{12}h_{2}+\\psi_{13}h_3$')\n", " ax[0,1].plot(x,layer2_pre_2,'b-'); ax[0,1].set_ylabel(r'$\\psi_{20}+\\psi_{21}h_{1}+\\psi_{22}h_{2}+\\psi_{23}h_3$')\n", " ax[0,2].plot(x,layer2_pre_3,'g-'); ax[0,2].set_ylabel(r'$\\psi_{30}+\\psi_{31}h_{1}+\\psi_{32}h_{2}+\\psi_{33}h_3$')\n", " ax[1,0].plot(x,h1_prime,'r-'); ax[1,0].set_ylabel(r\"$h_{1}^{'}$\")\n", " ax[1,1].plot(x,h2_prime,'b-'); ax[1,1].set_ylabel(r\"$h_{2}^{'}$\")\n", " ax[1,2].plot(x,h3_prime,'g-'); ax[1,2].set_ylabel(r\"$h_{3}^{'}$\")\n", " ax[2,0].plot(x,phi1_h1_prime,'r-'); ax[2,0].set_ylabel(r\"$\\phi_1 h_{1}^{'}$\")\n", " ax[2,1].plot(x,phi2_h2_prime,'b-'); ax[2,1].set_ylabel(r\"$\\phi_2 h_{2}^{'}$\")\n", " ax[2,2].plot(x,phi3_h3_prime,'g-'); ax[2,2].set_ylabel(r\"$\\phi_3 h_{3}^{'}$\")\n", "\n", " for plot_y in range(3):\n", " for plot_x in range(3):\n", " ax[plot_y,plot_x].set_xlim([0,1]);ax[plot_x,plot_y].set_ylim([-1,1])\n", " ax[plot_y,plot_x].set_aspect(0.5)\n", " ax[2,plot_y].set_xlabel(r'Input, $x$');\n", " plt.show()\n", "\n", " fig, ax = plt.subplots()\n", " ax.plot(x,y)\n", " ax.set_xlabel(r'Input, $x$'); ax.set_ylabel(r'Output, $y$')\n", " ax.set_xlim([0,1]);ax.set_ylim([-1,1])\n", " ax.set_aspect(0.5)\n", " plt.show()" ], "metadata": { "id": "ZB2HTalOE40X" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "Now let's define the parameters and visualize the network" ], "metadata": { "id": "LxBJCObC-NTY" } }, { "cell_type": "code", "source": [ "# Define parameters (note first dimension of theta and phi is padded to make indices match\n", "# notation in book)\n", "theta = np.zeros([4,2])\n", "psi = np.zeros([4,4])\n", "phi = np.zeros([4,1])\n", "\n", "theta[1,0] = 0.3 ; theta[1,1] = -1.0\n", "theta[2,0]= -1.0 ; theta[2,1] = 2.0\n", "theta[3,0] = -0.5 ; theta[3,1] = 0.65\n", "psi[1,0] = 0.3; psi[1,1] = 2.0; psi[1,2] = -1.0; psi[1,3]=7.0\n", "psi[2,0] = -0.2; psi[2,1] = 2.0; psi[2,2] = 1.2; psi[2,3]=-8.0\n", "psi[3,0] = 0.3; psi[3,1] = -2.3; psi[3,2] = -0.8; psi[3,3]=2.0\n", "phi[0] = 0.0; phi[1] = 0.5; phi[2] = -1.5; phi [3] = 2.2\n", "\n", "# Define a range of input values\n", "x = np.arange(0,1,0.01)\n", "\n", "# Run the neural network\n", "y, layer2_pre_1, layer2_pre_2, layer2_pre_3, h1_prime, h2_prime, h3_prime, phi1_h1_prime, phi2_h2_prime, phi3_h3_prime \\\n", " = shallow_1_1_3_3(x, ReLU, phi, psi, theta)\n", "\n", "# And then plot it\n", "plot_neural_two_layers(x, y, layer2_pre_1, layer2_pre_2, layer2_pre_3, h1_prime, h2_prime, h3_prime, phi1_h1_prime, phi2_h2_prime, phi3_h3_prime)" ], "metadata": { "id": "JRebvurv22pT" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "To do: To test your understanding of this, consider:\n", "\n", "1. What would happen if we increase $\\psi_{1,0}$?\n", "2. What would happen if we multiplied $\\psi_{2,0}, \\psi_{2,1}, \\psi_{2,2}, \\psi_{2,3}$ by -1?\n", "3. What would happen if set $\\phi_{3}$ to -1?\n", "\n", "You can rerun the code to see if you were correct.\n", "\n" ], "metadata": { "id": "GcjUUHbXf25D" } } ] }