315 lines
11 KiB
Plaintext
315 lines
11 KiB
Plaintext
{
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0,
|
|
"metadata": {
|
|
"colab": {
|
|
"provenance": [],
|
|
"authorship_tag": "ABX9TyNXqwmC4yEc1mGv9/74b0jY",
|
|
"include_colab_link": true
|
|
},
|
|
"kernelspec": {
|
|
"name": "python3",
|
|
"display_name": "Python 3"
|
|
},
|
|
"language_info": {
|
|
"name": "python"
|
|
}
|
|
},
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "view-in-github",
|
|
"colab_type": "text"
|
|
},
|
|
"source": [
|
|
"<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/Notebooks/Chap13/13_3_Neighborhood_Sampling.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"# **Notebook 13.3: Neighborhood sampling**\n",
|
|
"\n",
|
|
"This notebook investigates neighborhood sampling of graphs as in figure 13.10 from the book.\n",
|
|
"\n",
|
|
"Work through the cells below, running each cell in turn. In various places you will see the words \"TO DO\". Follow the instructions at these places and make predictions about what is going to happen or write code to complete the functions.\n",
|
|
"\n",
|
|
"Contact me at udlbookmail@gmail.com if you find any mistakes or have any suggestions."
|
|
],
|
|
"metadata": {
|
|
"id": "t9vk9Elugvmi"
|
|
}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import networkx as nx"
|
|
],
|
|
"metadata": {
|
|
"id": "OLComQyvCIJ7"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"Let's construct the graph from figure 13.10, which has 23 nodes."
|
|
],
|
|
"metadata": {
|
|
"id": "UNleESc7k5uB"
|
|
}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Define adjacency matrix\n",
|
|
"A = np.array([[0,1,1,1,0, 0,0,0,0,0, 0,0,0,0,0, 0,0,0,0,0, 0,0,0],\n",
|
|
" [1,0,1,0,0, 0,0,0,1,1, 0,0,0,0,0, 0,0,0,0,0, 0,0,0],\n",
|
|
" [1,1,0,1,0, 0,0,0,0,1, 0,0,0,0,0, 0,0,0,0,0, 0,0,0],\n",
|
|
" [1,0,1,0,1, 0,1,1,0,0, 0,0,0,0,0, 0,0,0,0,0, 0,0,0],\n",
|
|
" [0,0,0,1,0, 1,0,1,0,0, 0,0,0,0,0, 0,0,0,0,0, 0,0,0],\n",
|
|
" [0,0,0,0,1, 0,0,1,0,0, 0,0,0,0,0, 0,0,0,0,0, 0,0,0],\n",
|
|
" [0,0,0,1,0, 0,0,1,0,1, 1,0,0,0,0, 0,0,0,0,0, 0,0,0],\n",
|
|
" [0,0,0,1,1, 1,1,0,0,0, 1,0,0,1,0, 0,0,0,0,0, 0,0,0],\n",
|
|
" [0,1,0,0,0, 0,0,0,0,1, 0,0,0,0,0, 0,0,0,0,0, 0,0,0],\n",
|
|
" [0,1,1,0,0, 0,1,0,1,0, 0,1,1,0,0, 0,1,0,0,0, 0,0,0],\n",
|
|
" [0,0,0,0,0, 0,1,1,0,0, 0,0,1,0,0, 0,0,0,0,0, 0,0,0],\n",
|
|
" [0,0,0,0,0, 0,0,0,0,1, 0,0,0,0,1, 1,1,0,0,0, 0,0,0],\n",
|
|
" [0,0,0,0,0, 0,0,0,0,1, 1,0,0,1,0, 0,1,1,0,0, 0,0,0],\n",
|
|
" [0,0,0,0,0, 0,0,1,0,0, 0,0,1,0,0, 0,0,1,1,0, 0,0,0],\n",
|
|
" [0,0,0,0,0, 0,0,0,0,0, 0,1,0,0,0, 1,0,0,0,1, 0,0,0],\n",
|
|
" [0,0,0,0,0, 0,0,0,0,0, 0,1,0,0,1, 0,1,0,0,1, 0,0,0],\n",
|
|
" [0,0,0,0,0, 0,0,0,0,1, 0,1,1,0,0, 1,0,1,0,1, 0,0,0],\n",
|
|
" [0,0,0,0,0, 0,0,0,0,0, 0,0,1,1,0, 0,1,0,1,0, 1,1,1],\n",
|
|
" [0,0,0,0,0, 0,0,0,0,0, 0,0,0,1,0, 0,0,1,0,0, 0,0,1],\n",
|
|
" [0,0,0,0,0, 0,0,0,0,0, 0,0,0,0,1, 1,1,0,0,0, 1,0,0],\n",
|
|
" [0,0,0,0,0, 0,0,0,0,0, 0,0,0,0,0, 0,0,1,0,1, 0,1,0],\n",
|
|
" [0,0,0,0,0, 0,0,0,0,0, 0,0,0,0,0, 0,0,1,0,0, 1,0,1],\n",
|
|
" [0,0,0,0,0, 0,0,0,0,0, 0,0,0,0,0, 0,0,1,1,0, 0,1,0]]);\n",
|
|
"print(A)"
|
|
],
|
|
"metadata": {
|
|
"id": "fHgH5hdG_W1h"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Routine to draw graph structure, highlighting original node (brown in fig 13.10)\n",
|
|
"# and neighborhood nodes (orange in figure 13.10)\n",
|
|
"def draw_graph_structure(adjacency_matrix, original_node, neighborhood_nodes=None):\n",
|
|
"\n",
|
|
" G = nx.Graph()\n",
|
|
" n_node = adjacency_matrix.shape[0]\n",
|
|
" for i in range(n_node):\n",
|
|
" for j in range(i):\n",
|
|
" if adjacency_matrix[i,j]:\n",
|
|
" G.add_edge(i,j)\n",
|
|
"\n",
|
|
" color_map = []\n",
|
|
"\n",
|
|
" for node in G:\n",
|
|
" if original_node[node]:\n",
|
|
" color_map.append('brown')\n",
|
|
" else:\n",
|
|
" if neighborhood_nodes[node]:\n",
|
|
" color_map.append('orange')\n",
|
|
" else:\n",
|
|
" color_map.append('white')\n",
|
|
"\n",
|
|
" nx.draw(G, nx.spring_layout(G, seed = 7), with_labels=True,node_color=color_map)\n",
|
|
" plt.show()"
|
|
],
|
|
"metadata": {
|
|
"id": "TIrihEw-7DRV"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"n_nodes = A.shape[0]\n",
|
|
"\n",
|
|
"# Define a single output layer node\n",
|
|
"output_layer_nodes=np.zeros((n_nodes,1)); output_layer_nodes[16]=1\n",
|
|
"# Define the neighboring nodes to draw (none)\n",
|
|
"neighbor_nodes = np.zeros((n_nodes,1))\n",
|
|
"print(\"Output layer:\")\n",
|
|
"draw_graph_structure(A, output_layer_nodes, neighbor_nodes)"
|
|
],
|
|
"metadata": {
|
|
"id": "gKBD5JsPfrkA"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"Let's imagine that we want to form a batch for a node labelling task that consists of just node 16 in the output layer (highlighted). The network consists of the input, hidden layer 1, hidden layer2, and the output layer."
|
|
],
|
|
"metadata": {
|
|
"id": "JaH3g_-O-0no"
|
|
}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# TODO Find the nodes in hidden layer 2 that connect to node 16 in the output layer\n",
|
|
"# using the adjacency matrix\n",
|
|
"# Replace this line:\n",
|
|
"hidden_layer2_nodes = np.zeros((n_nodes,1));\n",
|
|
"\n",
|
|
"print(\"Hidden layer 2:\")\n",
|
|
"draw_graph_structure(A, output_layer_nodes, hidden_layer2_nodes)"
|
|
],
|
|
"metadata": {
|
|
"id": "9oSiuP3B3HNS"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# TODO - Find the nodes in hidden layer 1 that connect that connect to node 16 in the output layer\n",
|
|
"# using the adjacency matrix\n",
|
|
"# Replace this line:\n",
|
|
"hidden_layer1_nodes = np.zeros((n_nodes,1));\n",
|
|
"\n",
|
|
"print(\"Hidden layer 1:\")\n",
|
|
"draw_graph_structure(A, output_layer_nodes, hidden_layer1_nodes)"
|
|
],
|
|
"metadata": {
|
|
"id": "zZFxw3m1_wWr"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# TODO Find the nodes in the input layer that connect to node 16 in the output layer\n",
|
|
"# using the adjacency matrix\n",
|
|
"# Replace this line:\n",
|
|
"input_layer_nodes = np.zeros((n_nodes,1));\n",
|
|
"\n",
|
|
"print(\"Input layer:\")\n",
|
|
"draw_graph_structure(A, output_layer_nodes, input_layer_nodes)"
|
|
],
|
|
"metadata": {
|
|
"id": "EL3N8BXyCu0F"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"This is bad news. This is a fairly sparsely connected graph (i.e. adjacency matrix is mostly zeros) and there are only two hidden layers. Nonetheless, we have to involve almost all the nodes in the graph to compute the loss at this output.\n",
|
|
"\n",
|
|
"To resolve this problem, we'll use neighborhood sampling. We'll start again with a single node in the output layer."
|
|
],
|
|
"metadata": {
|
|
"id": "CE0WqytvC7zr"
|
|
}
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"print(\"Output layer:\")\n",
|
|
"draw_graph_structure(A, output_layer_nodes, neighbor_nodes)"
|
|
],
|
|
"metadata": {
|
|
"id": "59WNys3KC5y6"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# Define umber of neighbors to sample\n",
|
|
"n_sample = 3"
|
|
],
|
|
"metadata": {
|
|
"id": "uCoJwpcTNFdI"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# TODO Find the nodes in hidden layer 2 that connect to node 16 in the output layer\n",
|
|
"# using the adjacency matrix. Then sample n_sample of these nodes randomly without\n",
|
|
"# replacement.\n",
|
|
"\n",
|
|
"# Replace this line:\n",
|
|
"hidden_layer2_nodes = np.zeros((n_nodes,1));\n",
|
|
"\n",
|
|
"draw_graph_structure(A, output_layer_nodes, hidden_layer2_nodes)"
|
|
],
|
|
"metadata": {
|
|
"id": "_WEop6lYGNhJ"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# TODO Find the nodes in hidden layer 1 that connect to the nodes in hidden layer 2\n",
|
|
"# using the adjacency matrix. Then sample n_sample of these nodes randomly without\n",
|
|
"# replacement. Make sure not to sample nodes that were already included in hidden layer 2 our the output layer.\n",
|
|
"# The nodes at hidden layer 1 are the union of these nodes and the nodes in hidden layer 2\n",
|
|
"\n",
|
|
"# Replace this line:\n",
|
|
"hidden_layer1_nodes = np.zeros((n_nodes,1));\n",
|
|
"\n",
|
|
"draw_graph_structure(A, output_layer_nodes, hidden_layer1_nodes)\n"
|
|
],
|
|
"metadata": {
|
|
"id": "k90qW_LDLpNk"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"source": [
|
|
"# TODO Find the nodes in the input layer that connect to the nodes in hidden layer 1\n",
|
|
"# using the adjacency matrix. Then sample n_sample of these nodes randomly without\n",
|
|
"# replacement. Make sure not to sample nodes that were already included in hidden layer 1,2, or the output layer.\n",
|
|
"# The nodes at the input layer are the union of these nodes and the nodes in hidden layers 1 and 2\n",
|
|
"\n",
|
|
"# Replace this line:\n",
|
|
"input_layer_nodes = np.zeros((n_nodes,1));\n",
|
|
"\n",
|
|
"draw_graph_structure(A, output_layer_nodes, input_layer_nodes)"
|
|
],
|
|
"metadata": {
|
|
"id": "NDEYUty_O3Zr"
|
|
},
|
|
"execution_count": null,
|
|
"outputs": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"source": [
|
|
"If you did this correctly, there should be 9 yellow nodes in the figure. The \"receptive field\" of node 16 in the output layer increases much more slowly as we move back through the layers of the network."
|
|
],
|
|
"metadata": {
|
|
"id": "vu4eJURmVkc5"
|
|
}
|
|
}
|
|
]
|
|
}
|