366 lines
45 KiB
Plaintext
366 lines
45 KiB
Plaintext
{
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0,
|
|
"metadata": {
|
|
"colab": {
|
|
"provenance": [],
|
|
"authorship_tag": "ABX9TyPBNztJrxnUt1ELWfm1Awa3",
|
|
"include_colab_link": true
|
|
},
|
|
"kernelspec": {
|
|
"name": "python3",
|
|
"display_name": "Python 3"
|
|
},
|
|
"language_info": {
|
|
"name": "python"
|
|
}
|
|
},
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "view-in-github",
|
|
"colab_type": "text"
|
|
},
|
|
"source": [
|
|
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap03/3_1_Shallow_Networks_I.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"# **Notebook 3.1 -- Shallow neural networks I**\n",
|
|
"\n",
|
|
"The purpose of this notebook is to gain some familiarity with shallow neural networks with 1D inputs. It works through an example similar to figure 3.3 and experiments with different activation functions. <br>\n",
|
|
"\n",
|
|
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and write code to complete the functions. There are also questions interspersed in the text.\n",
|
|
"\n",
|
|
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
|
|
],
|
|
"metadata": {
|
|
"id": "1Z6LB4Ybn1oN"
|
|
}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "hAM55ZjSncOk"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Imports math library\n",
|
|
"import numpy as np\n",
|
|
"# Imports plotting library\n",
|
|
"import matplotlib.pyplot as plt"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"Let's first construct the shallow neural network with one input, three hidden units, and one output described in section 3.1 of the book."
|
|
],
|
|
"metadata": {
|
|
"id": "wQDy9UzXpnf5"
|
|
}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Define the Rectified Linear Unit (ReLU) function\n",
|
|
"def ReLU(preactivation):\n",
|
|
" # TODO write code to implement the ReLU and compute the activation at the\n",
|
|
" # hidden unit from the preactivation\n",
|
|
" # This should work on every element of the ndarray \"preactivation\" at once\n",
|
|
" # One way to do this is with the ndarray \"clip\" function\n",
|
|
" # https://numpy.org/doc/stable/reference/generated/numpy.ndarray.clip.html\n",
|
|
" activation = np.zeros_like(preactivation);\n",
|
|
"\n",
|
|
" return activation"
|
|
],
|
|
"metadata": {
|
|
"id": "OT7h7sSwpkrt"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Make an array of inputs\n",
|
|
"z = np.arange(-5,5,0.1)\n",
|
|
"RelU_z = ReLU(z)\n",
|
|
"\n",
|
|
"# Plot the ReLU function\n",
|
|
"fig, ax = plt.subplots()\n",
|
|
"ax.plot(z,RelU_z,'r-')\n",
|
|
"ax.set_xlim([-5,5]);ax.set_ylim([-5,5])\n",
|
|
"ax.set_xlabel('z'); ax.set_ylabel('ReLU[z]')\n",
|
|
"ax.set_aspect('equal')\n",
|
|
"plt.show()"
|
|
],
|
|
"metadata": {
|
|
"id": "okwJmSw9pVNF"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Define a shallow neural network with, one input, one output, and three hidden units\n",
|
|
"def shallow_1_1_3(x, activation_fn, phi_0,phi_1,phi_2,phi_3, theta_10, theta_11, theta_20, theta_21, theta_30, theta_31):\n",
|
|
" # TODO Replace the lines below to compute the three initial lines\n",
|
|
" # (figure 3.3a-c) from the theta parameters. These are the preactivations\n",
|
|
" pre_1 = np.zeros_like(x)\n",
|
|
" pre_2 = np.zeros_like(x)\n",
|
|
" pre_3 = np.zeros_like(x)\n",
|
|
"\n",
|
|
" # Pass these through the ReLU function to compute the activations as in\n",
|
|
" # figure 3.3 d-f\n",
|
|
" act_1 = activation_fn(pre_1)\n",
|
|
" act_2 = activation_fn(pre_2)\n",
|
|
" act_3 = activation_fn(pre_3)\n",
|
|
"\n",
|
|
" # TODO Replace the code below to weight the activations using phi1, phi2 and phi3\n",
|
|
" # To create the equivalent of figure 3.3 g-i\n",
|
|
" w_act_1 = np.zeros_like(x)\n",
|
|
" w_act_2 = np.zeros_like(x)\n",
|
|
" w_act_3 = np.zeros_like(x)\n",
|
|
"\n",
|
|
" # TODO Replace the code below to combining the weighted activations and add\n",
|
|
" # phi_0 to create the output as in figure 3.3 j\n",
|
|
" y = np.zeros_like(x)\n",
|
|
"\n",
|
|
" # Return everything we have calculated\n",
|
|
" return y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3"
|
|
],
|
|
"metadata": {
|
|
"id": "epk68ZCBu7uJ"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Plot the shallow neural network. We'll assume input in is range [0,1] and output [-1,1]\n",
|
|
"# If the plot_all flag is set to true, then we'll plot all the intermediate stages as in Figure 3.3\n",
|
|
"def plot_neural(x, y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3, plot_all=False, x_data=None, y_data=None):\n",
|
|
"\n",
|
|
" # Plot intermediate plots if flag set\n",
|
|
" if plot_all:\n",
|
|
" fig, ax = plt.subplots(3,3)\n",
|
|
" fig.set_size_inches(8.5, 8.5)\n",
|
|
" fig.tight_layout(pad=3.0)\n",
|
|
" ax[0,0].plot(x,pre_1,'r-'); ax[0,0].set_ylabel('Preactivation')\n",
|
|
" ax[0,1].plot(x,pre_2,'b-'); ax[0,1].set_ylabel('Preactivation')\n",
|
|
" ax[0,2].plot(x,pre_3,'g-'); ax[0,2].set_ylabel('Preactivation')\n",
|
|
" ax[1,0].plot(x,act_1,'r-'); ax[1,0].set_ylabel('Activation')\n",
|
|
" ax[1,1].plot(x,act_2,'b-'); ax[1,1].set_ylabel('Activation')\n",
|
|
" ax[1,2].plot(x,act_3,'g-'); ax[1,2].set_ylabel('Activation')\n",
|
|
" ax[2,0].plot(x,w_act_1,'r-'); ax[2,0].set_ylabel('Weighted Act')\n",
|
|
" ax[2,1].plot(x,w_act_2,'b-'); ax[2,1].set_ylabel('Weighted Act')\n",
|
|
" ax[2,2].plot(x,w_act_3,'g-'); ax[2,2].set_ylabel('Weighted Act')\n",
|
|
"\n",
|
|
" for plot_y in range(3):\n",
|
|
" for plot_x in range(3):\n",
|
|
" ax[plot_y,plot_x].set_xlim([0,1]);ax[plot_x,plot_y].set_ylim([-1,1])\n",
|
|
" ax[plot_y,plot_x].set_aspect(0.5)\n",
|
|
" ax[2,plot_y].set_xlabel('Input, $x$');\n",
|
|
" plt.show()\n",
|
|
"\n",
|
|
" fig, ax = plt.subplots()\n",
|
|
" ax.plot(x,y)\n",
|
|
" ax.set_xlabel('Input, $x$'); ax.set_ylabel('Output, $y$')\n",
|
|
" ax.set_xlim([0,1]);ax.set_ylim([-1,1])\n",
|
|
" ax.set_aspect(0.5)\n",
|
|
" if x_data is not None:\n",
|
|
" ax.plot(x_data, y_data, 'mo')\n",
|
|
" for i in range(len(x_data)):\n",
|
|
" ax.plot(x_data[i], y_data[i],)\n",
|
|
" plt.show()"
|
|
],
|
|
"metadata": {
|
|
"id": "CAr7n1lixuhQ"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Now lets define some parameters and run the neural network\n",
|
|
"theta_10 = 0.3 ; theta_11 = -1.0\n",
|
|
"theta_20 = -1.0 ; theta_21 = 2.0\n",
|
|
"theta_30 = -0.5 ; theta_31 = 0.65\n",
|
|
"phi_0 = -0.3; phi_1 = 2.0; phi_2 = -1.0; phi_3 = 7.0\n",
|
|
"\n",
|
|
"# Define a range of input values\n",
|
|
"x = np.arange(0,1,0.01)\n",
|
|
"\n",
|
|
"# We run the neural network for each of these input values\n",
|
|
"y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3 = \\\n",
|
|
" shallow_1_1_3(x, ReLU, phi_0,phi_1,phi_2,phi_3, theta_10, theta_11, theta_20, theta_21, theta_30, theta_31)\n",
|
|
"# And then plot it\n",
|
|
"plot_neural(x, y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3, plot_all=True)"
|
|
],
|
|
"metadata": {
|
|
"id": "SzIVdp9U-JWb"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"If your code is correct, then the final output should look like this:\n",
|
|
"<img src=\"data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjwhRE9DVFlQRSBzdmcgUFVCTElDICItLy9XM0MvL0RURCBTVkcgMS4xLy9FTiIKICAiaHR0cDovL3d3dy53My5vcmcvR3JhcGhpY3MvU1ZHLzEuMS9EVEQvc3ZnMTEuZHRkIj4KPCEtLSBDcmVhdGVkIHdpdGggbWF0cGxvdGxpYiAoaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pIC0tPgo8c3ZnIGhlaWdodD0iMjg4cHQiIHZlcnNpb249IjEuMSIgdmlld0JveD0iMCAwIDQzMiAyODgiIHdpZHRoPSI0MzJwdCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayI+CiA8ZGVmcz4KICA8c3R5bGUgdHlwZT0idGV4dC9jc3MiPgoqe3N0cm9rZS1saW5lY2FwOmJ1dHQ7c3Ryb2tlLWxpbmVqb2luOnJvdW5kO30KICA8L3N0eWxlPgogPC9kZWZzPgogPGcgaWQ9ImZpZ3VyZV8xIj4KICA8ZyBpZD0icGF0Y2hfMSI+CiAgIDxwYXRoIGQ9Ik0gMCAyODggCkwgNDMyIDI4OCAKTCA0MzIgMCAKTCAwIDAgCnoKIiBzdHlsZT0iZmlsbDojZmZmZmZmOyIvPgogIDwvZz4KICA8ZyBpZD0iYXhlc18xIj4KICAgPGcgaWQ9InBhdGNoXzIiPgogICAgPHBhdGggZD0iTSAxMTIuNjggMjUyIApMIDMzMC4xMiAyNTIgCkwgMzMwLjEyIDM0LjU2IApMIDExMi42OCAzNC41NiAKegoiIHN0eWxlPSJmaWxsOiNmZmZmZmY7Ii8+CiAgIDwvZz4KICAgPGcgaWQ9Im1hdHBsb3RsaWIuYXhpc18xIj4KICAgIDxnIGlkPSJ4dGlja18xIj4KICAgICA8ZyBpZD0ibGluZTJkXzEiPgogICAgICA8ZGVmcz4KICAgICAgIDxwYXRoIGQ9Ik0gMCAwIApMIDAgMy41IAoiIGlkPSJtMWRhYTNiZmM1MCIgc3R5bGU9InN0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowLjg7Ii8+CiAgICAgIDwvZGVmcz4KICAgICAgPGc+CiAgICAgICA8dXNlIHN0eWxlPSJzdHJva2U6IzAwMDAwMDtzdHJva2Utd2lkdGg6MC44OyIgeD0iMTEyLjY4IiB4bGluazpocmVmPSIjbTFkYWEzYmZjNTAiIHk9IjI1MiIvPgogICAgICA8L2c+CiAgICAgPC9nPgogICAgIDxnIGlkPSJ0ZXh0XzEiPgogICAgICA8IS0tIDAuMCAtLT4KICAgICAgPGRlZnM+CiAgICAgICA8cGF0aCBkPSJNIDMxLjc4MTI1IDY2LjQwNjI1IApRIDI0LjE3MTg3NSA2Ni40MDYyNSAyMC4zMjgxMjUgNTguOTA2MjUgClEgMTYuNSA1MS40MjE4NzUgMTYuNSAzNi4zNzUgClEgMTYuNSAyMS4zOTA2MjUgMjAuMzI4MTI1IDEzLjg5MDYyNSAKUSAyNC4xNzE4NzUgNi4zOTA2MjUgMzEuNzgxMjUgNi4zOTA2MjUgClEgMzkuNDUzMTI1IDYuMzkwNjI1IDQzLjI4MTI1IDEzLjg5MDYyNSAKUSA0Ny4xMjUgMjEuMzkwNjI1IDQ3LjEyNSAzNi4zNzUgClEgNDcuMTI1IDUxLjQyMTg3NSA0My4yODEyNSA1OC45MDYyNSAKUSAzOS40NTMxMjUgNjYuNDA2MjUgMzEuNzgxMjUgNjYuNDA2MjUgCnoKTSAzMS43ODEyNSA3NC4yMTg3NSAKUSA0NC4wNDY4NzUgNzQuMjE4NzUgNTAuNTE1NjI1IDY0LjUxNTYyNSAKUSA1Ni45ODQzNzUgNTQuODI4MTI1IDU2Ljk4NDM3NSAzNi4zNzUgClEgNTYuOTg0Mzc1IDE3Ljk2ODc1IDUwLjUxNTYyNSA4LjI2NTYyNSAKUSA0NC4wNDY4NzUgLTEuNDIxODc1IDMxLjc4MTI1IC0xLjQyMTg3NSAKUSAxOS41MzEyNSAtMS40MjE4NzUgMTMuMDYyNSA4LjI2NTYyNSAKUSA2LjU5Mzc1IDE3Ljk2ODc1IDYuNTkzNzUgMzYuMzc1IApRIDYuNTkzNzUgNTQuODI4MTI1IDEzLjA2MjUgNjQuNTE1NjI1IApRIDE5LjUzMTI1IDc0LjIxODc1IDMxLjc4MTI1IDc0LjIxODc1IAp6CiIgaWQ9IkRlamFWdVNhbnMtNDgiLz4KICAgICAgIDxwYXRoIGQ9Ik0gMTAuNjg3NSAxMi40MDYyNSAKTCAyMSAxMi40MDYyNSAKTCAyMSAwIApMIDEwLjY4NzUgMCAKegoiIGlkPSJEZWphVnVTYW5zLTQ2Ii8+CiAgICAgIDwvZGVmcz4KICAgICAgPGcgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTA0LjcyODQzOCAyNjYuNTk4NDM3KXNjYWxlKDAuMSAtMC4xKSI+CiAgICAgICA8dXNlIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTQ4Ii8+CiAgICAgICA8dXNlIHg9IjYzLjYyMzA0NyIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDYiLz4KICAgICAgIDx1c2UgeD0iOTUuNDEwMTU2IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00OCIvPgogICAgICA8L2c+CiAgICAgPC9nPgogICAgPC9nPgogICAgPGcgaWQ9Inh0aWNrXzIiPgogICAgIDxnIGlkPSJsaW5lMmRfMiI+CiAgICAgIDxnPgogICAgICAgPHVzZSBzdHlsZT0ic3Ryb2tlOiMwMDAwMDA7c3Ryb2tlLXdpZHRoOjAuODsiIHg9IjE1Ni4xNjgiIHhsaW5rOmhyZWY9IiNtMWRhYTNiZmM1MCIgeT0iMjUyIi8+CiAgICAgIDwvZz4KICAgICA8L2c+CiAgICAgPGcgaWQ9InRleHRfMiI+CiAgICAgIDwhLS0gMC4yIC0tPgogICAgICA8ZGVmcz4KICAgICAgIDxwYXRoIGQ9Ik0gMTkuMTg3NSA4LjI5Njg3NSAKTCA1My42MDkzNzUgOC4yOTY4NzUgCkwgNTMuNjA5Mzc1IDAgCkwgNy4zMjgxMjUgMCAKTCA3LjMyODEyNSA4LjI5Njg3NSAKUSAxMi45Mzc1IDE0LjEwOTM3NSAyMi42MjUgMjMuODkwNjI1IApRIDMyLjMyODEyNSAzMy42ODc1IDM0LjgxMjUgMzYuNTMxMjUgClEgMzkuNTQ2ODc1IDQxLjg0Mzc1IDQxLjQyMTg3NSA0NS41MzEyNSAKUSA0My4zMTI1IDQ5LjIxODc1IDQzLjMxMjUgNTIuNzgxMjUgClEgNDMuMzEyNSA1OC41OTM3NSAzOS4yMzQzNzUgNjIuMjUgClEgMzUuMTU2MjUgNjUuOTIxODc1IDI4LjYwOTM3NSA2NS45MjE4NzUgClEgMjMuOTY4NzUgNjUuOTIxODc1IDE4LjgxMjUgNjQuMzEyNSAKUSAxMy42NzE4NzUgNjIuNzAzMTI1IDcuODEyNSA1OS40MjE4NzUgCkwgNy44MTI1IDY5LjM5MDYyNSAKUSAxMy43NjU2MjUgNzEuNzgxMjUgMTguOTM3NSA3MyAKUSAyNC4xMjUgNzQuMjE4NzUgMjguNDIxODc1IDc0LjIxODc1IApRIDM5Ljc1IDc0LjIxODc1IDQ2LjQ4NDM3NSA2OC41NDY4NzUgClEgNTMuMjE4NzUgNjIuODkwNjI1IDUzLjIxODc1IDUzLjQyMTg3NSAKUSA1My4yMTg3NSA0OC45MjE4NzUgNTEuNTMxMjUgNDQuODkwNjI1IApRIDQ5Ljg1OTM3NSA0MC44NzUgNDUuNDA2MjUgMzUuNDA2MjUgClEgNDQuMTg3NSAzMy45ODQzNzUgMzcuNjQwNjI1IDI3LjIxODc1IApRIDMxLjEwOTM3NSAyMC40NTMxMjUgMTkuMTg3NSA4LjI5Njg3NSAKegoiIGlkPSJEZWphVnVTYW5zLTUwIi8+CiAgICAgIDwvZGVmcz4KICAgICAgPGcgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTQ4LjIxNjQzOCAyNjYuNTk4NDM3KXNjYWxlKDAuMSAtMC4xKSI+CiAgICAgICA8dXNlIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTQ4Ii8+CiAgICAgICA8dXNlIHg9IjYzLjYyMzA0NyIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDYiLz4KICAgICAgIDx1c2UgeD0iOTUuNDEwMTU2IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy01MCIvPgogICAgICA8L2c+CiAgICAgPC9nPgogICAgPC9nPgogICAgPGcgaWQ9Inh0aWNrXzMiPgogICAgIDxnIGlkPSJsaW5lMmRfMyI+CiAgICAgIDxnPgogICAgICAgPHVzZSBzdHlsZT0ic3Ryb2tlOiMwMDAwMDA7c3Ryb2tlLXdpZHRoOjAuODsiIHg9IjE5OS42NTYiIHhsaW5rOmhyZWY9IiNtMWRhYTNiZmM1MCIgeT0iMjUyIi8+CiAgICAgIDwvZz4KICAgICA8L2c+CiAgICAgPGcgaWQ9InRleHRfMyI+CiAgICAgIDwhLS0gMC40IC0tPgogICAgICA8ZGVmcz4KICAgICAgIDxwYXRoIGQ9Ik0gMzcuNzk2ODc1IDY0LjMxMjUgCkwgMTIuODkwNjI1IDI1LjM5MDYyNSAKTCAzNy43OTY4NzUgMjUuMzkwNjI1IAp6Ck0gMzUuMjAzMTI1IDcyLjkwNjI1IApMIDQ3LjYwOTM3NSA3Mi45MDYyNSAKTCA0Ny42MDkzNzUgMjUuMzkwNjI1IApMIDU4LjAxNTYyNSAyNS4zOTA2MjUgCkwgNTguMDE1NjI1IDE3LjE4NzUgCkwgNDcuNjA5Mzc1IDE3LjE4NzUgCkwgNDcuNjA5Mzc1IDAgCkwgMzcuNzk2ODc1IDAgCkwgMzcuNzk2ODc1IDE3LjE4NzUgCkwgNC44OTA2MjUgMTcuMTg3NSAKTCA0Ljg5MDYyNSAyNi43MDMxMjUgCnoKIiBpZD0iRGVqYVZ1U2Fucy01MiIvPgogICAgICA8L2RlZnM+CiAgICAgIDxnIHRyYW5zZm9ybT0idHJhbnNsYXRlKDE5MS43MDQ0MzggMjY2LjU5ODQzNylzY2FsZSgwLjEgLTAuMSkiPgogICAgICAgPHVzZSB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00OCIvPgogICAgICAgPHVzZSB4PSI2My42MjMwNDciIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTQ2Ii8+CiAgICAgICA8dXNlIHg9Ijk1LjQxMDE1NiIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNTIiLz4KICAgICAgPC9nPgogICAgIDwvZz4KICAgIDwvZz4KICAgIDxnIGlkPSJ4dGlja180Ij4KICAgICA8ZyBpZD0ibGluZTJkXzQiPgogICAgICA8Zz4KICAgICAgIDx1c2Ugc3R5bGU9InN0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowLjg7IiB4PSIyNDMuMTQ0IiB4bGluazpocmVmPSIjbTFkYWEzYmZjNTAiIHk9IjI1MiIvPgogICAgICA8L2c+CiAgICAgPC9nPgogICAgIDxnIGlkPSJ0ZXh0XzQiPgogICAgICA8IS0tIDAuNiAtLT4KICAgICAgPGRlZnM+CiAgICAgICA8cGF0aCBkPSJNIDMzLjAxNTYyNSA0MC4zNzUgClEgMjYuMzc1IDQwLjM3NSAyMi40ODQzNzUgMzUuODI4MTI1IApRIDE4LjYwOTM3NSAzMS4yOTY4NzUgMTguNjA5Mzc1IDIzLjM5MDYyNSAKUSAxOC42MDkzNzUgMTUuNTMxMjUgMjIuNDg0Mzc1IDEwLjk1MzEyNSAKUSAyNi4zNzUgNi4zOTA2MjUgMzMuMDE1NjI1IDYuMzkwNjI1IApRIDM5LjY1NjI1IDYuMzkwNjI1IDQzLjUzMTI1IDEwLjk1MzEyNSAKUSA0Ny40MDYyNSAxNS41MzEyNSA0Ny40MDYyNSAyMy4zOTA2MjUgClEgNDcuNDA2MjUgMzEuMjk2ODc1IDQzLjUzMTI1IDM1LjgyODEyNSAKUSAzOS42NTYyNSA0MC4zNzUgMzMuMDE1NjI1IDQwLjM3NSAKegpNIDUyLjU5Mzc1IDcxLjI5Njg3NSAKTCA1Mi41OTM3NSA2Mi4zMTI1IApRIDQ4Ljg3NSA2NC4wNjI1IDQ1LjA5Mzc1IDY0Ljk4NDM3NSAKUSA0MS4zMTI1IDY1LjkyMTg3NSAzNy41OTM3NSA2NS45MjE4NzUgClEgMjcuODI4MTI1IDY1LjkyMTg3NSAyMi42NzE4NzUgNTkuMzI4MTI1IApRIDE3LjUzMTI1IDUyLjczNDM3NSAxNi43OTY4NzUgMzkuNDA2MjUgClEgMTkuNjcxODc1IDQzLjY1NjI1IDI0LjAxNTYyNSA0NS45MjE4NzUgClEgMjguMzc1IDQ4LjE4NzUgMzMuNTkzNzUgNDguMTg3NSAKUSA0NC41NzgxMjUgNDguMTg3NSA1MC45NTMxMjUgNDEuNTE1NjI1IApRIDU3LjMyODEyNSAzNC44NTkzNzUgNTcuMzI4MTI1IDIzLjM5MDYyNSAKUSA1Ny4zMjgxMjUgMTIuMTU2MjUgNTAuNjg3NSA1LjM1OTM3NSAKUSA0NC4wNDY4NzUgLTEuNDIxODc1IDMzLjAxNTYyNSAtMS40MjE4NzUgClEgMjAuMzU5Mzc1IC0xLjQyMTg3NSAxMy42NzE4NzUgOC4yNjU2MjUgClEgNi45ODQzNzUgMTcuOTY4NzUgNi45ODQzNzUgMzYuMzc1IApRIDYuOTg0Mzc1IDUzLjY1NjI1IDE1LjE4NzUgNjMuOTM3NSAKUSAyMy4zOTA2MjUgNzQuMjE4NzUgMzcuMjAzMTI1IDc0LjIxODc1IApRIDQwLjkyMTg3NSA3NC4yMTg3NSA0NC43MDMxMjUgNzMuNDg0Mzc1IApRIDQ4LjQ4NDM3NSA3Mi43NSA1Mi41OTM3NSA3MS4yOTY4NzUgCnoKIiBpZD0iRGVqYVZ1U2Fucy01NCIvPgogICAgICA8L2RlZnM+CiAgICAgIDxnIHRyYW5zZm9ybT0idHJhbnNsYXRlKDIzNS4xOTI0MzggMjY2LjU5ODQzNylzY2FsZSgwLjEgLTAuMSkiPgogICAgICAgPHVzZSB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00OCIvPgogICAgICAgPHVzZSB4PSI2My42MjMwNDciIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTQ2Ii8+CiAgICAgICA8dXNlIHg9Ijk1LjQxMDE1NiIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNTQiLz4KICAgICAgPC9nPgogICAgIDwvZz4KICAgIDwvZz4KICAgIDxnIGlkPSJ4dGlja181Ij4KICAgICA8ZyBpZD0ibGluZTJkXzUiPgogICAgICA8Zz4KICAgICAgIDx1c2Ugc3R5bGU9InN0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowLjg7IiB4PSIyODYuNjMyIiB4bGluazpocmVmPSIjbTFkYWEzYmZjNTAiIHk9IjI1MiIvPgogICAgICA8L2c+CiAgICAgPC9nPgogICAgIDxnIGlkPSJ0ZXh0XzUiPgogICAgICA8IS0tIDAuOCAtLT4KICAgICAgPGRlZnM+CiAgICAgICA8cGF0aCBkPSJNIDMxLjc4MTI1IDM0LjYyNSAKUSAyNC43NSAzNC42MjUgMjAuNzE4NzUgMzAuODU5Mzc1IApRIDE2LjcwMzEyNSAyNy4wOTM3NSAxNi43MDMxMjUgMjAuNTE1NjI1IApRIDE2LjcwMzEyNSAxMy45MjE4NzUgMjAuNzE4NzUgMTAuMTU2MjUgClEgMjQuNzUgNi4zOTA2MjUgMzEuNzgxMjUgNi4zOTA2MjUgClEgMzguODEyNSA2LjM5MDYyNSA0Mi44NTkzNzUgMTAuMTcxODc1IApRIDQ2LjkyMTg3NSAxMy45Njg3NSA0Ni45MjE4NzUgMjAuNTE1NjI1IApRIDQ2LjkyMTg3NSAyNy4wOTM3NSA0Mi44OTA2MjUgMzAuODU5Mzc1IApRIDM4Ljg3NSAzNC42MjUgMzEuNzgxMjUgMzQuNjI1IAp6Ck0gMjEuOTIxODc1IDM4LjgxMjUgClEgMTUuNTc4MTI1IDQwLjM3NSAxMi4wMzEyNSA0NC43MTg3NSAKUSA4LjUgNDkuMDc4MTI1IDguNSA1NS4zMjgxMjUgClEgOC41IDY0LjA2MjUgMTQuNzE4NzUgNjkuMTQwNjI1IApRIDIwLjk1MzEyNSA3NC4yMTg3NSAzMS43ODEyNSA3NC4yMTg3NSAKUSA0Mi42NzE4NzUgNzQuMjE4NzUgNDguODc1IDY5LjE0MDYyNSAKUSA1NS4wNzgxMjUgNjQuMDYyNSA1NS4wNzgxMjUgNTUuMzI4MTI1IApRIDU1LjA3ODEyNSA0OS4wNzgxMjUgNTEuNTMxMjUgNDQuNzE4NzUgClEgNDggNDAuMzc1IDQxLjcwMzEyNSAzOC44MTI1IApRIDQ4LjgyODEyNSAzNy4xNTYyNSA1Mi43OTY4NzUgMzIuMzEyNSAKUSA1Ni43ODEyNSAyNy40ODQzNzUgNTYuNzgxMjUgMjAuNTE1NjI1IApRIDU2Ljc4MTI1IDkuOTA2MjUgNTAuMzEyNSA0LjIzNDM3NSAKUSA0My44NDM3NSAtMS40MjE4NzUgMzEuNzgxMjUgLTEuNDIxODc1IApRIDE5LjczNDM3NSAtMS40MjE4NzUgMTMuMjUgNC4yMzQzNzUgClEgNi43ODEyNSA5LjkwNjI1IDYuNzgxMjUgMjAuNTE1NjI1IApRIDYuNzgxMjUgMjcuNDg0Mzc1IDEwLjc4MTI1IDMyLjMxMjUgClEgMTQuNzk2ODc1IDM3LjE1NjI1IDIxLjkyMTg3NSAzOC44MTI1IAp6Ck0gMTguMzEyNSA1NC4zOTA2MjUgClEgMTguMzEyNSA0OC43MzQzNzUgMjEuODQzNzUgNDUuNTYyNSAKUSAyNS4zOTA2MjUgNDIuMzkwNjI1IDMxLjc4MTI1IDQyLjM5MDYyNSAKUSAzOC4xNDA2MjUgNDIuMzkwNjI1IDQxLjcxODc1IDQ1LjU2MjUgClEgNDUuMzEyNSA0OC43MzQzNzUgNDUuMzEyNSA1NC4zOTA2MjUgClEgNDUuMzEyNSA2MC4wNjI1IDQxLjcxODc1IDYzLjIzNDM3NSAKUSAzOC4xNDA2MjUgNjYuNDA2MjUgMzEuNzgxMjUgNjYuNDA2MjUgClEgMjUuMzkwNjI1IDY2LjQwNjI1IDIxLjg0Mzc1IDYzLjIzNDM3NSAKUSAxOC4zMTI1IDYwLjA2MjUgMTguMzEyNSA1NC4zOTA2MjUgCnoKIiBpZD0iRGVqYVZ1U2Fucy01NiIvPgogICAgICA8L2RlZnM+CiAgICAgIDxnIHRyYW5zZm9ybT0idHJhbnNsYXRlKDI3OC42ODA0MzcgMjY2LjU5ODQzNylzY2FsZSgwLjEgLTAuMSkiPgogICAgICAgPHVzZSB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00OCIvPgogICAgICAgPHVzZSB4PSI2My42MjMwNDciIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTQ2Ii8+CiAgICAgICA8dXNlIHg9Ijk1LjQxMDE1NiIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNTYiLz4KICAgICAgPC9nPgogICAgIDwvZz4KICAgIDwvZz4KICAgIDxnIGlkPSJ4dGlja182Ij4KICAgICA8ZyBpZD0ibGluZTJkXzYiPgogICAgICA8Zz4KICAgICAgIDx1c2Ugc3R5bGU9InN0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowLjg7IiB4PSIzMzAuMTIiIHhsaW5rOmhyZWY9IiNtMWRhYTNiZmM1MCIgeT0iMjUyIi8+CiAgICAgIDwvZz4KICAgICA8L2c+CiAgICAgPGcgaWQ9InRleHRfNiI+CiAgICAgIDwhLS0gMS4wIC0tPgogICAgICA8ZGVmcz4KICAgICAgIDxwYXRoIGQ9Ik0gMTIuNDA2MjUgOC4yOTY4NzUgCkwgMjguNTE1NjI1IDguMjk2ODc1IApMIDI4LjUxNTYyNSA2My45MjE4NzUgCkwgMTAuOTg0Mzc1IDYwLjQwNjI1IApMIDEwLjk4NDM3NSA2OS4zOTA2MjUgCkwgMjguNDIxODc1IDcyLjkwNjI1IApMIDM4LjI4MTI1IDcyLjkwNjI1IApMIDM4LjI4MTI1IDguMjk2ODc1IApMIDU0LjM5MDYyNSA4LjI5Njg3NSAKTCA1NC4zOTA2MjUgMCAKTCAxMi40MDYyNSAwIAp6CiIgaWQ9IkRlamFWdVNhbnMtNDkiLz4KICAgICAgPC9kZWZzPgogICAgICA8ZyB0cmFuc2Zvcm09InRyYW5zbGF0ZSgzMjIuMTY4NDM3IDI2Ni41OTg0Mzcpc2NhbGUoMC4xIC0wLjEpIj4KICAgICAgIDx1c2UgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDkiLz4KICAgICAgIDx1c2UgeD0iNjMuNjIzMDQ3IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00NiIvPgogICAgICAgPHVzZSB4PSI5NS40MTAxNTYiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTQ4Ii8+CiAgICAgIDwvZz4KICAgICA8L2c+CiAgICA8L2c+CiAgICA8ZyBpZD0idGV4dF83Ij4KICAgICA8IS0tIElucHV0LCAkeSQgLS0+CiAgICAgPGRlZnM+CiAgICAgIDxwYXRoIGQ9Ik0gOS44MTI1IDcyLjkwNjI1IApMIDE5LjY3MTg3NSA3Mi45MDYyNSAKTCAxOS42NzE4NzUgMCAKTCA5LjgxMjUgMCAKegoiIGlkPSJEZWphVnVTYW5zLTczIi8+CiAgICAgIDxwYXRoIGQ9Ik0gNTQuODkwNjI1IDMzLjAxNTYyNSAKTCA1NC44OTA2MjUgMCAKTCA0NS45MDYyNSAwIApMIDQ1LjkwNjI1IDMyLjcxODc1IApRIDQ1LjkwNjI1IDQwLjQ4NDM3NSA0Mi44NzUgNDQuMzI4MTI1IApRIDM5Ljg0Mzc1IDQ4LjE4NzUgMzMuNzk2ODc1IDQ4LjE4NzUgClEgMjYuNTE1NjI1IDQ4LjE4NzUgMjIuMzEyNSA0My41NDY4NzUgClEgMTguMTA5Mzc1IDM4LjkyMTg3NSAxOC4xMDkzNzUgMzAuOTA2MjUgCkwgMTguMTA5Mzc1IDAgCkwgOS4wNzgxMjUgMCAKTCA5LjA3ODEyNSA1NC42ODc1IApMIDE4LjEwOTM3NSA1NC42ODc1IApMIDE4LjEwOTM3NSA0Ni4xODc1IApRIDIxLjM0Mzc1IDUxLjEyNSAyNS43MDMxMjUgNTMuNTYyNSAKUSAzMC4wNzgxMjUgNTYgMzUuNzk2ODc1IDU2IApRIDQ1LjIxODc1IDU2IDUwLjA0Njg3NSA1MC4xNzE4NzUgClEgNTQuODkwNjI1IDQ0LjM0Mzc1IDU0Ljg5MDYyNSAzMy4wMTU2MjUgCnoKIiBpZD0iRGVqYVZ1U2Fucy0xMTAiLz4KICAgICAgPHBhdGggZD0iTSAxOC4xMDkzNzUgOC4yMDMxMjUgCkwgMTguMTA5Mzc1IC0yMC43OTY4NzUgCkwgOS4wNzgxMjUgLTIwLjc5Njg3NSAKTCA5LjA3ODEyNSA1NC42ODc1IApMIDE4LjEwOTM3NSA1NC42ODc1IApMIDE4LjEwOTM3NSA0Ni4zOTA2MjUgClEgMjAuOTUzMTI1IDUxLjI2NTYyNSAyNS4yNjU2MjUgNTMuNjI1IApRIDI5LjU5Mzc1IDU2IDM1LjU5Mzc1IDU2IApRIDQ1LjU2MjUgNTYgNTEuNzgxMjUgNDguMDkzNzUgClEgNTguMDE1NjI1IDQwLjE4NzUgNTguMDE1NjI1IDI3LjI5Njg3NSAKUSA1OC4wMTU2MjUgMTQuNDA2MjUgNTEuNzgxMjUgNi40ODQzNzUgClEgNDUuNTYyNSAtMS40MjE4NzUgMzUuNTkzNzUgLTEuNDIxODc1IApRIDI5LjU5Mzc1IC0xLjQyMTg3NSAyNS4yNjU2MjUgMC45NTMxMjUgClEgMjAuOTUzMTI1IDMuMzI4MTI1IDE4LjEwOTM3NSA4LjIwMzEyNSAKegpNIDQ4LjY4NzUgMjcuMjk2ODc1IApRIDQ4LjY4NzUgMzcuMjAzMTI1IDQ0LjYwOTM3NSA0Mi44NDM3NSAKUSA0MC41MzEyNSA0OC40ODQzNzUgMzMuNDA2MjUgNDguNDg0Mzc1IApRIDI2LjI2NTYyNSA0OC40ODQzNzUgMjIuMTg3NSA0Mi44NDM3NSAKUSAxOC4xMDkzNzUgMzcuMjAzMTI1IDE4LjEwOTM3NSAyNy4yOTY4NzUgClEgMTguMTA5Mzc1IDE3LjM5MDYyNSAyMi4xODc1IDExLjc1IApRIDI2LjI2NTYyNSA2LjEwOTM3NSAzMy40MDYyNSA2LjEwOTM3NSAKUSA0MC41MzEyNSA2LjEwOTM3NSA0NC42MDkzNzUgMTEuNzUgClEgNDguNjg3NSAxNy4zOTA2MjUgNDguNjg3NSAyNy4yOTY4NzUgCnoKIiBpZD0iRGVqYVZ1U2Fucy0xMTIiLz4KICAgICAgPHBhdGggZD0iTSA4LjUgMjEuNTc4MTI1IApMIDguNSA1NC42ODc1IApMIDE3LjQ4NDM3NSA1NC42ODc1IApMIDE3LjQ4NDM3NSAyMS45MjE4NzUgClEgMTcuNDg0Mzc1IDE0LjE1NjI1IDIwLjUgMTAuMjY1NjI1IApRIDIzLjUzMTI1IDYuMzkwNjI1IDI5LjU5Mzc1IDYuMzkwNjI1IApRIDM2Ljg1OTM3NSA2LjM5MDYyNSA0MS4wNzgxMjUgMTEuMDMxMjUgClEgNDUuMzEyNSAxNS42NzE4NzUgNDUuMzEyNSAyMy42ODc1IApMIDQ1LjMxMjUgNTQuNjg3NSAKTCA1NC4yOTY4NzUgNTQuNjg3NSAKTCA1NC4yOTY4NzUgMCAKTCA0NS4zMTI1IDAgCkwgNDUuMzEyNSA4LjQwNjI1IApRIDQyLjA0Njg3NSAzLjQyMTg3NSAzNy43MTg3NSAxIApRIDMzLjQwNjI1IC0xLjQyMTg3NSAyNy42ODc1IC0xLjQyMTg3NSAKUSAxOC4yNjU2MjUgLTEuNDIxODc1IDEzLjM3NSA0LjQzNzUgClEgOC41IDEwLjI5Njg3NSA4LjUgMjEuNTc4MTI1IAp6Ck0gMzEuMTA5Mzc1IDU2IAp6CiIgaWQ9IkRlamFWdVNhbnMtMTE3Ii8+CiAgICAgIDxwYXRoIGQ9Ik0gMTguMzEyNSA3MC4yMTg3NSAKTCAxOC4zMTI1IDU0LjY4NzUgCkwgMzYuODEyNSA1NC42ODc1IApMIDM2LjgxMjUgNDcuNzAzMTI1IApMIDE4LjMxMjUgNDcuNzAzMTI1IApMIDE4LjMxMjUgMTguMDE1NjI1IApRIDE4LjMxMjUgMTEuMzI4MTI1IDIwLjE0MDYyNSA5LjQyMTg3NSAKUSAyMS45Njg3NSA3LjUxNTYyNSAyNy41OTM3NSA3LjUxNTYyNSAKTCAzNi44MTI1IDcuNTE1NjI1IApMIDM2LjgxMjUgMCAKTCAyNy41OTM3NSAwIApRIDE3LjE4NzUgMCAxMy4yMzQzNzUgMy44NzUgClEgOS4yODEyNSA3Ljc2NTYyNSA5LjI4MTI1IDE4LjAxNTYyNSAKTCA5LjI4MTI1IDQ3LjcwMzEyNSAKTCAyLjY4NzUgNDcuNzAzMTI1IApMIDIuNjg3NSA1NC42ODc1IApMIDkuMjgxMjUgNTQuNjg3NSAKTCA5LjI4MTI1IDcwLjIxODc1IAp6CiIgaWQ9IkRlamFWdVNhbnMtMTE2Ii8+CiAgICAgIDxwYXRoIGQ9Ik0gMTEuNzE4NzUgMTIuNDA2MjUgCkwgMjIuMDE1NjI1IDEyLjQwNjI1IApMIDIyLjAxNTYyNSA0IApMIDE0LjAxNTYyNSAtMTEuNjI1IApMIDcuNzE4NzUgLTExLjYyNSAKTCAxMS43MTg3NSA0IAp6CiIgaWQ9IkRlamFWdVNhbnMtNDQiLz4KICAgICAgPHBhdGggaWQ9IkRlamFWdVNhbnMtMzIiLz4KICAgICAgPHBhdGggZD0iTSAyNC44MTI1IC01LjA3ODEyNSAKUSAxOC41NjI1IC0xNS41NzgxMjUgMTQuNjI1IC0xOC4xODc1IApRIDEwLjY4NzUgLTIwLjc5Njg3NSA0LjU5Mzc1IC0yMC43OTY4NzUgCkwgLTIuNDg0Mzc1IC0yMC43OTY4NzUgCkwgLTAuOTg0Mzc1IC0xMy4yODEyNSAKTCA0LjIwMzEyNSAtMTMuMjgxMjUgClEgNy45NTMxMjUgLTEzLjI4MTI1IDEwLjU5Mzc1IC0xMS4yMzQzNzUgClEgMTMuMjM0Mzc1IC05LjE4NzUgMTYuNSAtMy4yMTg3NSAKTCAxOS4yODEyNSAyIApMIDcuMTcxODc1IDU0LjY4NzUgCkwgMTYuNzAzMTI1IDU0LjY4NzUgCkwgMjUuNzgxMjUgMTIuNzk2ODc1IApMIDUwLjg3NSA1NC42ODc1IApMIDYwLjI5Njg3NSA1NC42ODc1IAp6CiIgaWQ9IkRlamFWdVNhbnMtT2JsaXF1ZS0xMjEiLz4KICAgICA8L2RlZnM+CiAgICAgPGcgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMjAyLjMgMjgwLjI1NjI1KXNjYWxlKDAuMSAtMC4xKSI+CiAgICAgIDx1c2UgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCAwLjA5Mzc1KSIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNzMiLz4KICAgICAgPHVzZSB0cmFuc2Zvcm09InRyYW5zbGF0ZSgyOS40OTIxODggMC4wOTM3NSkiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTExMCIvPgogICAgICA8dXNlIHRyYW5zZm9ybT0idHJhbnNsYXRlKDkyLjg3MTA5NCAwLjA5Mzc1KSIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtMTEyIi8+CiAgICAgIDx1c2UgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTU2LjM0NzY1NiAwLjA5Mzc1KSIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtMTE3Ii8+CiAgICAgIDx1c2UgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMjE5LjcyNjU2MiAwLjA5Mzc1KSIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtMTE2Ii8+CiAgICAgIDx1c2UgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMjU4LjkzNTU0NyAwLjA5Mzc1KSIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDQiLz4KICAgICAgPHVzZSB0cmFuc2Zvcm09InRyYW5zbGF0ZSgyOTAuNzIyNjU2IDAuMDkzNzUpIiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy0zMiIvPgogICAgICA8dXNlIHRyYW5zZm9ybT0idHJhbnNsYXRlKDMyMi41MDk3NjYgMC4wOTM3NSkiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLU9ibGlxdWUtMTIxIi8+CiAgICAgPC9nPgogICAgPC9nPgogICA8L2c+CiAgIDxnIGlkPSJtYXRwbG90bGliLmF4aXNfMiI+CiAgICA8ZyBpZD0ieXRpY2tfMSI+CiAgICAgPGcgaWQ9ImxpbmUyZF83Ij4KICAgICAgPGRlZnM+CiAgICAgICA8cGF0aCBkPSJNIDAgMCAKTCAtMy41IDAgCiIgaWQ9Im1hNDAwZWU0MmIxIiBzdHlsZT0ic3Ryb2tlOiMwMDAwMDA7c3Ryb2tlLXdpZHRoOjAuODsiLz4KICAgICAgPC9kZWZzPgogICAgICA8Zz4KICAgICAgIDx1c2Ugc3R5bGU9InN0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowLjg7IiB4PSIxMTIuNjgiIHhsaW5rOmhyZWY9IiNtYTQwMGVlNDJiMSIgeT0iMjUyIi8+CiAgICAgIDwvZz4KICAgICA8L2c+CiAgICAgPGcgaWQ9InRleHRfOCI+CiAgICAgIDwhLS0g4oiSMS4wMCAtLT4KICAgICAgPGRlZnM+CiAgICAgICA8cGF0aCBkPSJNIDEwLjU5Mzc1IDM1LjUgCkwgNzMuMTg3NSAzNS41IApMIDczLjE4NzUgMjcuMjAzMTI1IApMIDEwLjU5Mzc1IDI3LjIwMzEyNSAKegoiIGlkPSJEZWphVnVTYW5zLTg3MjIiLz4KICAgICAgPC9kZWZzPgogICAgICA8ZyB0cmFuc2Zvcm09InRyYW5zbGF0ZSg3NS4wMzQ2ODggMjU1Ljc5OTIxOSlzY2FsZSgwLjEgLTAuMSkiPgogICAgICAgPHVzZSB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy04NzIyIi8+CiAgICAgICA8dXNlIHg9IjgzLjc4OTA2MiIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDkiLz4KICAgICAgIDx1c2UgeD0iMTQ3LjQxMjEwOSIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDYiLz4KICAgICAgIDx1c2UgeD0iMTc5LjE5OTIxOSIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDgiLz4KICAgICAgIDx1c2UgeD0iMjQyLjgyMjI2NiIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDgiLz4KICAgICAgPC9nPgogICAgIDwvZz4KICAgIDwvZz4KICAgIDxnIGlkPSJ5dGlja18yIj4KICAgICA8ZyBpZD0ibGluZTJkXzgiPgogICAgICA8Zz4KICAgICAgIDx1c2Ugc3R5bGU9InN0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowLjg7IiB4PSIxMTIuNjgiIHhsaW5rOmhyZWY9IiNtYTQwMGVlNDJiMSIgeT0iMjI0LjgyIi8+CiAgICAgIDwvZz4KICAgICA8L2c+CiAgICAgPGcgaWQ9InRleHRfOSI+CiAgICAgIDwhLS0g4oiSMC43NSAtLT4KICAgICAgPGRlZnM+CiAgICAgICA8cGF0aCBkPSJNIDguMjAzMTI1IDcyLjkwNjI1IApMIDU1LjA3ODEyNSA3Mi45MDYyNSAKTCA1NS4wNzgxMjUgNjguNzAzMTI1IApMIDI4LjYwOTM3NSAwIApMIDE4LjMxMjUgMCAKTCA0My4yMTg3NSA2NC41OTM3NSAKTCA4LjIwMzEyNSA2NC41OTM3NSAKegoiIGlkPSJEZWphVnVTYW5zLTU1Ii8+CiAgICAgICA8cGF0aCBkPSJNIDEwLjc5Njg3NSA3Mi45MDYyNSAKTCA0OS41MTU2MjUgNzIuOTA2MjUgCkwgNDkuNTE1NjI1IDY0LjU5Mzc1IApMIDE5LjgyODEyNSA2NC41OTM3NSAKTCAxOS44MjgxMjUgNDYuNzM0Mzc1IApRIDIxLjk2ODc1IDQ3LjQ2ODc1IDI0LjEwOTM3NSA0Ny44MjgxMjUgClEgMjYuMjY1NjI1IDQ4LjE4NzUgMjguNDIxODc1IDQ4LjE4NzUgClEgNDAuNjI1IDQ4LjE4NzUgNDcuNzUgNDEuNSAKUSA1NC44OTA2MjUgMzQuODEyNSA1NC44OTA2MjUgMjMuMzkwNjI1IApRIDU0Ljg5MDYyNSAxMS42MjUgNDcuNTYyNSA1LjA5Mzc1IApRIDQwLjIzNDM3NSAtMS40MjE4NzUgMjYuOTA2MjUgLTEuNDIxODc1IApRIDIyLjMxMjUgLTEuNDIxODc1IDE3LjU0Njg3NSAtMC42NDA2MjUgClEgMTIuNzk2ODc1IDAuMTQwNjI1IDcuNzE4NzUgMS43MDMxMjUgCkwgNy43MTg3NSAxMS42MjUgClEgMTIuMTA5Mzc1IDkuMjM0Mzc1IDE2Ljc5Njg3NSA4LjA2MjUgClEgMjEuNDg0Mzc1IDYuODkwNjI1IDI2LjcwMzEyNSA2Ljg5MDYyNSAKUSAzNS4xNTYyNSA2Ljg5MDYyNSA0MC4wNzgxMjUgMTEuMzI4MTI1IApRIDQ1LjAxNTYyNSAxNS43NjU2MjUgNDUuMDE1NjI1IDIzLjM5MDYyNSAKUSA0NS4wMTU2MjUgMzEgNDAuMDc4MTI1IDM1LjQzNzUgClEgMzUuMTU2MjUgMzkuODkwNjI1IDI2LjcwMzEyNSAzOS44OTA2MjUgClEgMjIuNzUgMzkuODkwNjI1IDE4LjgxMjUgMzkuMDE1NjI1IApRIDE0Ljg5MDYyNSAzOC4xNDA2MjUgMTAuNzk2ODc1IDM2LjI4MTI1IAp6CiIgaWQ9IkRlamFWdVNhbnMtNTMiLz4KICAgICAgPC9kZWZzPgogICAgICA8ZyB0cmFuc2Zvcm09InRyYW5zbGF0ZSg3NS4wMzQ2ODggMjI4LjYxOTIxOSlzY2FsZSgwLjEgLTAuMSkiPgogICAgICAgPHVzZSB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy04NzIyIi8+CiAgICAgICA8dXNlIHg9IjgzLjc4OTA2MiIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDgiLz4KICAgICAgIDx1c2UgeD0iMTQ3LjQxMjEwOSIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDYiLz4KICAgICAgIDx1c2UgeD0iMTc5LjE5OTIxOSIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNTUiLz4KICAgICAgIDx1c2UgeD0iMjQyLjgyMjI2NiIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNTMiLz4KICAgICAgPC9nPgogICAgIDwvZz4KICAgIDwvZz4KICAgIDxnIGlkPSJ5dGlja18zIj4KICAgICA8ZyBpZD0ibGluZTJkXzkiPgogICAgICA8Zz4KICAgICAgIDx1c2Ugc3R5bGU9InN0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowLjg7IiB4PSIxMTIuNjgiIHhsaW5rOmhyZWY9IiNtYTQwMGVlNDJiMSIgeT0iMTk3LjY0Ii8+CiAgICAgIDwvZz4KICAgICA8L2c+CiAgICAgPGcgaWQ9InRleHRfMTAiPgogICAgICA8IS0tIOKIkjAuNTAgLS0+CiAgICAgIDxnIHRyYW5zZm9ybT0idHJhbnNsYXRlKDc1LjAzNDY4OCAyMDEuNDM5MjE5KXNjYWxlKDAuMSAtMC4xKSI+CiAgICAgICA8dXNlIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTg3MjIiLz4KICAgICAgIDx1c2UgeD0iODMuNzg5MDYyIiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00OCIvPgogICAgICAgPHVzZSB4PSIxNDcuNDEyMTA5IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00NiIvPgogICAgICAgPHVzZSB4PSIxNzkuMTk5MjE5IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy01MyIvPgogICAgICAgPHVzZSB4PSIyNDIuODIyMjY2IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00OCIvPgogICAgICA8L2c+CiAgICAgPC9nPgogICAgPC9nPgogICAgPGcgaWQ9Inl0aWNrXzQiPgogICAgIDxnIGlkPSJsaW5lMmRfMTAiPgogICAgICA8Zz4KICAgICAgIDx1c2Ugc3R5bGU9InN0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowLjg7IiB4PSIxMTIuNjgiIHhsaW5rOmhyZWY9IiNtYTQwMGVlNDJiMSIgeT0iMTcwLjQ2Ii8+CiAgICAgIDwvZz4KICAgICA8L2c+CiAgICAgPGcgaWQ9InRleHRfMTEiPgogICAgICA8IS0tIOKIkjAuMjUgLS0+CiAgICAgIDxnIHRyYW5zZm9ybT0idHJhbnNsYXRlKDc1LjAzNDY4OCAxNzQuMjU5MjE5KXNjYWxlKDAuMSAtMC4xKSI+CiAgICAgICA8dXNlIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTg3MjIiLz4KICAgICAgIDx1c2UgeD0iODMuNzg5MDYyIiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00OCIvPgogICAgICAgPHVzZSB4PSIxNDcuNDEyMTA5IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00NiIvPgogICAgICAgPHVzZSB4PSIxNzkuMTk5MjE5IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy01MCIvPgogICAgICAgPHVzZSB4PSIyNDIuODIyMjY2IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy01MyIvPgogICAgICA8L2c+CiAgICAgPC9nPgogICAgPC9nPgogICAgPGcgaWQ9Inl0aWNrXzUiPgogICAgIDxnIGlkPSJsaW5lMmRfMTEiPgogICAgICA8Zz4KICAgICAgIDx1c2Ugc3R5bGU9InN0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowLjg7IiB4PSIxMTIuNjgiIHhsaW5rOmhyZWY9IiNtYTQwMGVlNDJiMSIgeT0iMTQzLjI4Ii8+CiAgICAgIDwvZz4KICAgICA8L2c+CiAgICAgPGcgaWQ9InRleHRfMTIiPgogICAgICA8IS0tIDAuMDAgLS0+CiAgICAgIDxnIHRyYW5zZm9ybT0idHJhbnNsYXRlKDgzLjQxNDM3NSAxNDcuMDc5MjE5KXNjYWxlKDAuMSAtMC4xKSI+CiAgICAgICA8dXNlIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTQ4Ii8+CiAgICAgICA8dXNlIHg9IjYzLjYyMzA0NyIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDYiLz4KICAgICAgIDx1c2UgeD0iOTUuNDEwMTU2IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00OCIvPgogICAgICAgPHVzZSB4PSIxNTkuMDMzMjAzIiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00OCIvPgogICAgICA8L2c+CiAgICAgPC9nPgogICAgPC9nPgogICAgPGcgaWQ9Inl0aWNrXzYiPgogICAgIDxnIGlkPSJsaW5lMmRfMTIiPgogICAgICA8Zz4KICAgICAgIDx1c2Ugc3R5bGU9InN0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowLjg7IiB4PSIxMTIuNjgiIHhsaW5rOmhyZWY9IiNtYTQwMGVlNDJiMSIgeT0iMTE2LjEiLz4KICAgICAgPC9nPgogICAgIDwvZz4KICAgICA8ZyBpZD0idGV4dF8xMyI+CiAgICAgIDwhLS0gMC4yNSAtLT4KICAgICAgPGcgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoODMuNDE0Mzc1IDExOS44OTkyMTkpc2NhbGUoMC4xIC0wLjEpIj4KICAgICAgIDx1c2UgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDgiLz4KICAgICAgIDx1c2UgeD0iNjMuNjIzMDQ3IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00NiIvPgogICAgICAgPHVzZSB4PSI5NS40MTAxNTYiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTUwIi8+CiAgICAgICA8dXNlIHg9IjE1OS4wMzMyMDMiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTUzIi8+CiAgICAgIDwvZz4KICAgICA8L2c+CiAgICA8L2c+CiAgICA8ZyBpZD0ieXRpY2tfNyI+CiAgICAgPGcgaWQ9ImxpbmUyZF8xMyI+CiAgICAgIDxnPgogICAgICAgPHVzZSBzdHlsZT0ic3Ryb2tlOiMwMDAwMDA7c3Ryb2tlLXdpZHRoOjAuODsiIHg9IjExMi42OCIgeGxpbms6aHJlZj0iI21hNDAwZWU0MmIxIiB5PSI4OC45MiIvPgogICAgICA8L2c+CiAgICAgPC9nPgogICAgIDxnIGlkPSJ0ZXh0XzE0Ij4KICAgICAgPCEtLSAwLjUwIC0tPgogICAgICA8ZyB0cmFuc2Zvcm09InRyYW5zbGF0ZSg4My40MTQzNzUgOTIuNzE5MjE5KXNjYWxlKDAuMSAtMC4xKSI+CiAgICAgICA8dXNlIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTQ4Ii8+CiAgICAgICA8dXNlIHg9IjYzLjYyMzA0NyIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDYiLz4KICAgICAgIDx1c2UgeD0iOTUuNDEwMTU2IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy01MyIvPgogICAgICAgPHVzZSB4PSIxNTkuMDMzMjAzIiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00OCIvPgogICAgICA8L2c+CiAgICAgPC9nPgogICAgPC9nPgogICAgPGcgaWQ9Inl0aWNrXzgiPgogICAgIDxnIGlkPSJsaW5lMmRfMTQiPgogICAgICA8Zz4KICAgICAgIDx1c2Ugc3R5bGU9InN0cm9rZTojMDAwMDAwO3N0cm9rZS13aWR0aDowLjg7IiB4PSIxMTIuNjgiIHhsaW5rOmhyZWY9IiNtYTQwMGVlNDJiMSIgeT0iNjEuNzQiLz4KICAgICAgPC9nPgogICAgIDwvZz4KICAgICA8ZyBpZD0idGV4dF8xNSI+CiAgICAgIDwhLS0gMC43NSAtLT4KICAgICAgPGcgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoODMuNDE0Mzc1IDY1LjUzOTIxOSlzY2FsZSgwLjEgLTAuMSkiPgogICAgICAgPHVzZSB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00OCIvPgogICAgICAgPHVzZSB4PSI2My42MjMwNDciIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTQ2Ii8+CiAgICAgICA8dXNlIHg9Ijk1LjQxMDE1NiIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNTUiLz4KICAgICAgIDx1c2UgeD0iMTU5LjAzMzIwMyIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNTMiLz4KICAgICAgPC9nPgogICAgIDwvZz4KICAgIDwvZz4KICAgIDxnIGlkPSJ5dGlja185Ij4KICAgICA8ZyBpZD0ibGluZTJkXzE1Ij4KICAgICAgPGc+CiAgICAgICA8dXNlIHN0eWxlPSJzdHJva2U6IzAwMDAwMDtzdHJva2Utd2lkdGg6MC44OyIgeD0iMTEyLjY4IiB4bGluazpocmVmPSIjbWE0MDBlZTQyYjEiIHk9IjM0LjU2Ii8+CiAgICAgIDwvZz4KICAgICA8L2c+CiAgICAgPGcgaWQ9InRleHRfMTYiPgogICAgICA8IS0tIDEuMDAgLS0+CiAgICAgIDxnIHRyYW5zZm9ybT0idHJhbnNsYXRlKDgzLjQxNDM3NSAzOC4zNTkyMTkpc2NhbGUoMC4xIC0wLjEpIj4KICAgICAgIDx1c2UgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNDkiLz4KICAgICAgIDx1c2UgeD0iNjMuNjIzMDQ3IiB4bGluazpocmVmPSIjRGVqYVZ1U2Fucy00NiIvPgogICAgICAgPHVzZSB4PSI5NS40MTAxNTYiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTQ4Ii8+CiAgICAgICA8dXNlIHg9IjE1OS4wMzMyMDMiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTQ4Ii8+CiAgICAgIDwvZz4KICAgICA8L2c+CiAgICA8L2c+CiAgICA8ZyBpZD0idGV4dF8xNyI+CiAgICAgPCEtLSBPdXRwdXQsICR5JCAtLT4KICAgICA8ZGVmcz4KICAgICAgPHBhdGggZD0iTSAzOS40MDYyNSA2Ni4yMTg3NSAKUSAyOC42NTYyNSA2Ni4yMTg3NSAyMi4zMjgxMjUgNTguMjAzMTI1IApRIDE2LjAxNTYyNSA1MC4yMDMxMjUgMTYuMDE1NjI1IDM2LjM3NSAKUSAxNi4wMTU2MjUgMjIuNjA5Mzc1IDIyLjMyODEyNSAxNC41OTM3NSAKUSAyOC42NTYyNSA2LjU5Mzc1IDM5LjQwNjI1IDYuNTkzNzUgClEgNTAuMTQwNjI1IDYuNTkzNzUgNTYuNDIxODc1IDE0LjU5Mzc1IApRIDYyLjcwMzEyNSAyMi42MDkzNzUgNjIuNzAzMTI1IDM2LjM3NSAKUSA2Mi43MDMxMjUgNTAuMjAzMTI1IDU2LjQyMTg3NSA1OC4yMDMxMjUgClEgNTAuMTQwNjI1IDY2LjIxODc1IDM5LjQwNjI1IDY2LjIxODc1IAp6Ck0gMzkuNDA2MjUgNzQuMjE4NzUgClEgNTQuNzM0Mzc1IDc0LjIxODc1IDYzLjkwNjI1IDYzLjkzNzUgClEgNzMuMDkzNzUgNTMuNjU2MjUgNzMuMDkzNzUgMzYuMzc1IApRIDczLjA5Mzc1IDE5LjE0MDYyNSA2My45MDYyNSA4Ljg1OTM3NSAKUSA1NC43MzQzNzUgLTEuNDIxODc1IDM5LjQwNjI1IC0xLjQyMTg3NSAKUSAyNC4wMzEyNSAtMS40MjE4NzUgMTQuODEyNSA4LjgyODEyNSAKUSA1LjYwOTM3NSAxOS4wOTM3NSA1LjYwOTM3NSAzNi4zNzUgClEgNS42MDkzNzUgNTMuNjU2MjUgMTQuODEyNSA2My45Mzc1IApRIDI0LjAzMTI1IDc0LjIxODc1IDM5LjQwNjI1IDc0LjIxODc1IAp6CiIgaWQ9IkRlamFWdVNhbnMtNzkiLz4KICAgICA8L2RlZnM+CiAgICAgPGcgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoNjguOTM0Njg3IDE2Ni44Mylyb3RhdGUoLTkwKXNjYWxlKDAuMSAtMC4xKSI+CiAgICAgIDx1c2UgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCAwLjc4MTI1KSIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtNzkiLz4KICAgICAgPHVzZSB0cmFuc2Zvcm09InRyYW5zbGF0ZSg3OC43MTA5MzggMC43ODEyNSkiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTExNyIvPgogICAgICA8dXNlIHRyYW5zZm9ybT0idHJhbnNsYXRlKDE0Mi4wODk4NDQgMC43ODEyNSkiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTExNiIvPgogICAgICA8dXNlIHRyYW5zZm9ybT0idHJhbnNsYXRlKDE4MS4yOTg4MjggMC43ODEyNSkiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTExMiIvPgogICAgICA8dXNlIHRyYW5zZm9ybT0idHJhbnNsYXRlKDI0NC43NzUzOTEgMC43ODEyNSkiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTExNyIvPgogICAgICA8dXNlIHRyYW5zZm9ybT0idHJhbnNsYXRlKDMwOC4xNTQyOTcgMC43ODEyNSkiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTExNiIvPgogICAgICA8dXNlIHRyYW5zZm9ybT0idHJhbnNsYXRlKDM0Ny4zNjMyODEgMC43ODEyNSkiIHhsaW5rOmhyZWY9IiNEZWphVnVTYW5zLTQ0Ii8+CiAgICAgIDx1c2UgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMzc5LjE1MDM5MSAwLjc4MTI1KSIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtMzIiLz4KICAgICAgPHVzZSB0cmFuc2Zvcm09InRyYW5zbGF0ZSg0MTAuOTM3NSAwLjc4MTI1KSIgeGxpbms6aHJlZj0iI0RlamFWdVNhbnMtT2JsaXF1ZS0xMjEiLz4KICAgICA8L2c+CiAgICA8L2c+CiAgIDwvZz4KICAgPGcgaWQ9ImxpbmUyZF8xNiI+CiAgICA8cGF0aCBjbGlwLXBhdGg9InVybCgjcGQyYmE0MzY4ZmUpIiBkPSJNIDExMi42OCAxMTAuNjY0IApMIDExNC44NTQ0IDExMi44Mzg0IApMIDExNy4wMjg4IDExNS4wMTI4IApMIDExOS4yMDMyIDExNy4xODcyIApMIDEyMS4zNzc2IDExOS4zNjE2IApMIDEyMy41NTIgMTIxLjUzNiAKTCAxMjUuNzI2NCAxMjMuNzEwNCAKTCAxMjcuOTAwOCAxMjUuODg0OCAKTCAxMzAuMDc1MiAxMjguMDU5MiAKTCAxMzIuMjQ5NiAxMzAuMjMzNiAKTCAxMzQuNDI0IDEzMi40MDggCkwgMTM2LjU5ODQgMTM0LjU4MjQgCkwgMTM4Ljc3MjggMTM2Ljc1NjggCkwgMTQwLjk0NzIgMTM4LjkzMTIgCkwgMTQzLjEyMTYgMTQxLjEwNTYgCkwgMTQ1LjI5NiAxNDMuMjggCkwgMTQ3LjQ3MDQgMTQ1LjQ1NDQgCkwgMTQ5LjY0NDggMTQ3LjYyODggCkwgMTUxLjgxOTIgMTQ5LjgwMzIgCkwgMTUzLjk5MzYgMTUxLjk3NzYgCkwgMTU2LjE2OCAxNTQuMTUyIApMIDE1OC4zNDI0IDE1Ni4zMjY0IApMIDE2MC41MTY4IDE1OC41MDA4IApMIDE2Mi42OTEyIDE2MC42NzUyIApMIDE2NC44NjU2IDE2Mi44NDk2IApMIDE2Ny4wNCAxNjUuMDI0IApMIDE2OS4yMTQ0IDE2Ny4xOTg0IApMIDE3MS4zODg4IDE2OS4zNzI4IApMIDE3My41NjMyIDE3MS41NDcyIApMIDE3NS43Mzc2IDE3My43MjE2IApMIDE3Ny45MTIgMTc1Ljg5NiAKTCAxODAuMDg2NCAxNzUuODk2IApMIDE4Mi4yNjA4IDE3NS44OTYgCkwgMTg0LjQzNTIgMTc1Ljg5NiAKTCAxODYuNjA5NiAxNzUuODk2IApMIDE4OC43ODQgMTc1Ljg5NiAKTCAxOTAuOTU4NCAxNzUuODk2IApMIDE5My4xMzI4IDE3NS44OTYgCkwgMTk1LjMwNzIgMTc1Ljg5NiAKTCAxOTcuNDgxNiAxNzUuODk2IApMIDE5OS42NTYgMTc1Ljg5NiAKTCAyMDEuODMwNCAxNzUuODk2IApMIDIwNC4wMDQ4IDE3NS44OTYgCkwgMjA2LjE3OTIgMTc1Ljg5NiAKTCAyMDguMzUzNiAxNzUuODk2IApMIDIxMC41MjggMTc1Ljg5NiAKTCAyMTIuNzAyNCAxNzUuODk2IApMIDIxNC44NzY4IDE3NS44OTYgCkwgMjE3LjA1MTIgMTc1Ljg5NiAKTCAyMTkuMjI1NiAxNzUuODk2IApMIDIyMS40IDE3NS44OTYgCkwgMjIzLjU3NDQgMTc4LjA3MDQgCkwgMjI1Ljc0ODggMTgwLjI0NDggCkwgMjI3LjkyMzIgMTgyLjQxOTIgCkwgMjMwLjA5NzYgMTg0LjU5MzYgCkwgMjMyLjI3MiAxODYuNzY4IApMIDIzNC40NDY0IDE4OC45NDI0IApMIDIzNi42MjA4IDE5MS4xMTY4IApMIDIzOC43OTUyIDE5My4yOTEyIApMIDI0MC45Njk2IDE5NS40NjU2IApMIDI0My4xNDQgMTk3LjY0IApMIDI0NS4zMTg0IDE5OS44MTQ0IApMIDI0Ny40OTI4IDIwMS45ODg4IApMIDI0OS42NjcyIDIwNC4xNjMyIApMIDI1MS44NDE2IDIwNi4zMzc2IApMIDI1NC4wMTYgMjA4LjUxMiAKTCAyNTYuMTkwNCAyMTAuNjg2NCAKTCAyNTguMzY0OCAyMTIuODYwOCAKTCAyNjAuNTM5MiAyMTUuMDM1MiAKTCAyNjIuNzEzNiAyMTcuMjA5NiAKTCAyNjQuODg4IDIxOS4zODQgCkwgMjY3LjA2MjQgMjIxLjU1ODQgCkwgMjY5LjIzNjggMjIzLjczMjggCkwgMjcxLjQxMTIgMjI1LjkwNzIgCkwgMjczLjU4NTYgMjI4LjA4MTYgCkwgMjc1Ljc2IDIzMC4yNTYgCkwgMjc3LjkzNDQgMjMyLjQzMDQgCkwgMjgwLjEwODggMjM0LjIyNDI4IApMIDI4Mi4yODMyIDIzMS40NTE5MiAKTCAyODQuNDU3NiAyMjguNjc5NTYgCkwgMjg2LjYzMiAyMjUuOTA3MiAKTCAyODguODA2NCAyMjMuMTM0ODQgCkwgMjkwLjk4MDggMjIwLjM2MjQ4IApMIDI5My4xNTUyIDIxNy41OTAxMiAKTCAyOTUuMzI5NiAyMTQuODE3NzYgCkwgMjk3LjUwNCAyMTIuMDQ1NCAKTCAyOTkuNjc4NCAyMDkuMjczMDQgCkwgMzAxLjg1MjggMjA2LjUwMDY4IApMIDMwNC4wMjcyIDIwMy43MjgzMiAKTCAzMDYuMjAxNiAyMDAuOTU1OTYgCkwgMzA4LjM3NiAxOTguMTgzNiAKTCAzMTAuNTUwNCAxOTUuNDExMjQgCkwgMzEyLjcyNDggMTkyLjYzODg4IApMIDMxNC44OTkyIDE4OS44NjY1MiAKTCAzMTcuMDczNiAxODcuMDk0MTYgCkwgMzE5LjI0OCAxODQuMzIxOCAKTCAzMjEuNDIyNCAxODEuNTQ5NDQgCkwgMzIzLjU5NjggMTc4Ljc3NzA4IApMIDMyNS43NzEyIDE3Ni4wMDQ3MiAKTCAzMjcuOTQ1NiAxNzMuMjMyMzYgCiIgc3R5bGU9ImZpbGw6bm9uZTtzdHJva2U6IzFmNzdiNDtzdHJva2UtbGluZWNhcDpzcXVhcmU7c3Ryb2tlLXdpZHRoOjEuNTsiLz4KICAgPC9nPgogICA8ZyBpZD0icGF0Y2hfMyI+CiAgICA8cGF0aCBkPSJNIDExMi42OCAyNTIgCkwgMTEyLjY4IDM0LjU2IAoiIHN0eWxlPSJmaWxsOm5vbmU7c3Ryb2tlOiMwMDAwMDA7c3Ryb2tlLWxpbmVjYXA6c3F1YXJlO3N0cm9rZS1saW5lam9pbjptaXRlcjtzdHJva2Utd2lkdGg6MC44OyIvPgogICA8L2c+CiAgIDxnIGlkPSJwYXRjaF80Ij4KICAgIDxwYXRoIGQ9Ik0gMzMwLjEyIDI1MiAKTCAzMzAuMTIgMzQuNTYgCiIgc3R5bGU9ImZpbGw6bm9uZTtzdHJva2U6IzAwMDAwMDtzdHJva2UtbGluZWNhcDpzcXVhcmU7c3Ryb2tlLWxpbmVqb2luOm1pdGVyO3N0cm9rZS13aWR0aDowLjg7Ii8+CiAgIDwvZz4KICAgPGcgaWQ9InBhdGNoXzUiPgogICAgPHBhdGggZD0iTSAxMTIuNjggMjUyIApMIDMzMC4xMiAyNTIgCiIgc3R5bGU9ImZpbGw6bm9uZTtzdHJva2U6IzAwMDAwMDtzdHJva2UtbGluZWNhcDpzcXVhcmU7c3Ryb2tlLWxpbmVqb2luOm1pdGVyO3N0cm9rZS13aWR0aDowLjg7Ii8+CiAgIDwvZz4KICAgPGcgaWQ9InBhdGNoXzYiPgogICAgPHBhdGggZD0iTSAxMTIuNjggMzQuNTYgCkwgMzMwLjEyIDM0LjU2IAoiIHN0eWxlPSJmaWxsOm5vbmU7c3Ryb2tlOiMwMDAwMDA7c3Ryb2tlLWxpbmVjYXA6c3F1YXJlO3N0cm9rZS1saW5lam9pbjptaXRlcjtzdHJva2Utd2lkdGg6MC44OyIvPgogICA8L2c+CiAgPC9nPgogPC9nPgogPGRlZnM+CiAgPGNsaXBQYXRoIGlkPSJwZDJiYTQzNjhmZSI+CiAgIDxyZWN0IGhlaWdodD0iMjE3LjQ0IiB3aWR0aD0iMjE3LjQ0IiB4PSIxMTIuNjgiIHk9IjM0LjU2Ii8+CiAgPC9jbGlwUGF0aD4KIDwvZGVmcz4KPC9zdmc+Cg==\">"
|
|
],
|
|
"metadata": {
|
|
"id": "T34bszToImKQ"
|
|
}
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"Now let's play with the parameters to make sure we understand how they work. The original parameters were:\n",
|
|
"\n",
|
|
"$\\theta_{10} = 0.3$ ; $\\theta_{11} = -1.0$<br>\n",
|
|
"$\\theta_{20} = -1.0$ ; $\\theta_{21} = 2.0$<br>\n",
|
|
"$\\theta_{30} = -0.5$ ; $\\theta_{31} = 0.65$<br>\n",
|
|
"$\\phi_0 = -0.3; \\phi_1 = 2.0; \\phi_2 = -1.0; \\phi_3 = 7.0$"
|
|
],
|
|
"metadata": {
|
|
"id": "jhaBSS8oIWSX"
|
|
}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# TODO\n",
|
|
"# 1. Predict what effect changing phi_0 will have on the network.\n",
|
|
"\n",
|
|
"# 2. Predict what effect multiplying phi_1, phi_2, phi_3 by 0.5 would have. Check if you are correct\n",
|
|
"\n",
|
|
"# 3. Predict what effect multiplying phi_1 by -1 will have. Check if you are correct.\n",
|
|
"\n",
|
|
"# 4. Predict what effect setting theta_20 to -1.2 will have. Check if you are correct.\n",
|
|
"\n",
|
|
"# 5. Change the parameters so that there are only two \"joints\" (including outside the range of the plot)\n",
|
|
"# There are actually three ways to do this. See if you can figure them all out\n",
|
|
"\n",
|
|
"# 6. With the original parameters, the second line segment is flat (i.e. has slope zero)\n",
|
|
"# How could you change theta_10 so that all of the segments have non-zero slopes\n",
|
|
"\n",
|
|
"# 7. What do you predict would happen if you multiply theta_20 and theta21 by 0.5, and phi_2 by 2.0?\n",
|
|
"# Check if you are correct.\n",
|
|
"\n",
|
|
"# 8. What do you predict would happen if you multiply theta_20 and theta21 by -0.5, and phi_2 by -2.0?\n",
|
|
"# Check if you are correct.\n",
|
|
"\n",
|
|
"theta_10 = 0.3 ; theta_11 = -1.0\n",
|
|
"theta_20 = -1.0 ; theta_21 = 2.0\n",
|
|
"theta_30 = -0.5 ; theta_31 = 0.65\n",
|
|
"phi_0 = -0.3; phi_1 = 2.0; phi_2 = -1.0; phi_3 = 7.0\n",
|
|
"\n",
|
|
"# Define a range of input values\n",
|
|
"x = np.arange(0,1,0.01)\n",
|
|
"\n",
|
|
"# We run the neural network for each of these input values\n",
|
|
"y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3 = \\\n",
|
|
" shallow_1_1_3(x, ReLU, phi_0,phi_1,phi_2,phi_3, theta_10, theta_11, theta_20, theta_21, theta_30, theta_31)\n",
|
|
"# And then plot it\n",
|
|
"plot_neural(x, y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3, plot_all=True)"
|
|
],
|
|
"metadata": {
|
|
"id": "ur4arJ8KAQWe"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"# Least squares loss\n",
|
|
"\n",
|
|
"Now let's consider fitting the network to data. First we need to define the loss function. We'll use the least squares loss:\n",
|
|
"\n",
|
|
"\\begin{equation}\n",
|
|
"L[\\boldsymbol\\phi] = \\sum_{i=1}^{I}(y_{i}-\\mbox{f}[x_{i},\\boldsymbol\\phi])^2\n",
|
|
"\\end{equation}\n",
|
|
"\n",
|
|
"where $(x_i,y_i)$ is an input/output training pair and $\\mbox{f}[\\bullet,\\boldsymbol\\phi]$ is the neural network with parameters $\\boldsymbol\\phi$. The first term in the brackets is the ground truth output and the second term is the prediction of the model"
|
|
],
|
|
"metadata": {
|
|
"id": "osonHsEqVp2I"
|
|
}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Least squares function\n",
|
|
"def least_squares_loss(y_train, y_predict):\n",
|
|
" # TODO Replace the line below to compute the sum of squared\n",
|
|
" # differences between the real values of y and the predicted values from the model f[x_i,phi]\n",
|
|
" # (see figure 2.2 of the book)\n",
|
|
" # you will need to use the function np.sum\n",
|
|
" loss = 0\n",
|
|
"\n",
|
|
" return loss"
|
|
],
|
|
"metadata": {
|
|
"id": "14d5II-TU46w"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Now lets define some parameters, run the neural network, and compute the loss\n",
|
|
"theta_10 = 0.3 ; theta_11 = -1.0\n",
|
|
"theta_20 = -1.0 ; theta_21 = 2.0\n",
|
|
"theta_30 = -0.5 ; theta_31 = 0.65\n",
|
|
"phi_0 = -0.3; phi_1 = 2.0; phi_2 = -1.0; phi_3 = 7.0\n",
|
|
"\n",
|
|
"# Define a range of input values\n",
|
|
"x = np.arange(0,1,0.01)\n",
|
|
"\n",
|
|
"x_train = np.array([0.09291784,0.46809093,0.93089486,0.67612654,0.73441752,0.86847339,\\\n",
|
|
" 0.49873225,0.51083168,0.18343972,0.99380898,0.27840809,0.38028817,\\\n",
|
|
" 0.12055708,0.56715537,0.92005746,0.77072270,0.85278176,0.05315950,\\\n",
|
|
" 0.87168699,0.58858043])\n",
|
|
"y_train = np.array([-0.15934537,0.18195445,0.451270150,0.13921448,0.09366691,0.30567674,\\\n",
|
|
" 0.372291170,0.40716968,-0.08131792,0.41187806,0.36943738,0.3994327,\\\n",
|
|
" 0.019062570,0.35820410,0.452564960,-0.0183121,0.02957665,-0.24354444, \\\n",
|
|
" 0.148038840,0.26824970])\n",
|
|
"\n",
|
|
"# We run the neural network for each of these input values\n",
|
|
"y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3 = \\\n",
|
|
" shallow_1_1_3(x, ReLU, phi_0,phi_1,phi_2,phi_3, theta_10, theta_11, theta_20, theta_21, theta_30, theta_31)\n",
|
|
"# And then plot it\n",
|
|
"plot_neural(x, y, pre_1, pre_2, pre_3, act_1, act_2, act_3, w_act_1, w_act_2, w_act_3, plot_all=True, x_data = x_train, y_data = y_train)\n",
|
|
"\n",
|
|
"# Run the neural network on the training data\n",
|
|
"y_predict, *_ = shallow_1_1_3(x_train, ReLU, phi_0,phi_1,phi_2,phi_3, theta_10, theta_11, theta_20, theta_21, theta_30, theta_31)\n",
|
|
"\n",
|
|
"# Compute the least squares loss and print it out\n",
|
|
"loss = least_squares_loss(y_train,y_predict)\n",
|
|
"print("Your Loss = %3.3f, True value = 9.385"%(loss))\n",
|
|
"\n",
|
|
"# TODO. Manipulate the parameters (by hand!) to make the function\n",
|
|
"# fit the data better and try to reduce the loss to as small a number\n",
|
|
"# as possible. The best that I could do was 0.181\n",
|
|
"# Tip... start by manipulating phi_0.\n",
|
|
"# It's not that easy, so don't spend too much time on this!\n"
|
|
],
|
|
"metadata": {
|
|
"id": "o6GXjtRubZ2U"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
}
|
|
]
|
|
}
|