3. Hubs and Paths (Student).ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "import matplotlib.pyplot as plt\n",
    "from collections import Counter\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "# Load the pickled data without the new individuals added in the previous notebook.\n",
    "\n",
    "G = nx.read_gpickle('Synthetic Social Network.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hubs: How do we evaluate the importance of some individuals in a network?\n",
    "\n",
    "Within a social network, there will be certain individuals which perform certain important functions. For example, there may be hyper-connected individuals who are connected to many, many more people. They would be of use in the spreading of information. Alternatively, if this were a disease contact network, identifying them would be useful in stopping the spread of diseases. How would one identify these people?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Approach 1: Neighbors\n",
    "\n",
    "One way we could compute this is to find out the number of people an individual is conencted to. NetworkX let's us do this by giving us a `G.neighbors(node)` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Let's find out the number of neighbors that individual #7 has.\n",
    "G.neighbors(9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Can you create a ranked list of the importance of each individual, based on the number of neighbors they have? \n",
    "\n",
    "Hint: One suggested output would be a list of tuples, where the first element in each tuple is the node ID (an integer number), and the second element is a list of its neighbors.\n",
    "\n",
    "Hint: Python's `sorted(iterable, key=lambda x:...., reverse=True)` function may be of help here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Approach 2: Degree Centrality\n",
    "\n",
    "The number of other nodes that one node is connected to is a measure of its centrality. NetworkX implements a **degree centrality**, which is defined as the number of neighbors that a node has normalized to the number of individuals it could be connected to in the entire graph. This is accessed by using `nx.degree_centrality(G)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "nx.degree_centrality(G)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If you inspect the dictionary closely, you will find that node 19 is the one that has the highest degree centrality, just as we had measured by counting the number of neighbors.\n",
    "\n",
    "There are other measures of centrality, namely **betweenness centrality**, **flow centrality** and **load centrality**. You can take a look at their definitions on the NetworkX API docs and their cited references. You can also define your own measures if those don't fit your needs, but that is an advanced topic that won't be dealt with here.\n",
    "\n",
    "The NetworkX API docs that document the centrality measures are here: http://networkx.github.io/documentation/networkx-1.9.1/reference/algorithms.centrality.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercises\n",
    "\n",
    "1. Can you create a histogram of the distribution of degree centralities?\n",
    "2. Can you create a histogram of the distribution of number of neighbors?\n",
    "3. Can you create a scatterplot of the degree centralities against number of neighbors?\n",
    "4. If I have `n` nodes, then how many possible edges are there in total, assuming self-edges are allowed? What if self-edges are not allowed?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Hint: You may want to use:\n",
    "\n",
    "    plt.hist(list_of_values)\n",
    "\n",
    "and \n",
    "\n",
    "    plt.scatter(x_values, y_values)\n",
    "\n",
    "If you know the Matplotlib API, feel free to get fancy :)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Answer to question 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Answer to question 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Answer to question 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Paths in a Network\n",
    "\n",
    "Graph traversal is akin to walking along the graph, node by node, restricted by the edges that connect the nodes. Graph traversal is particularly useful for understanding the local structure (e.g. connectivity, retrieving the exact relationships) of certain portions of the graph and for finding paths that connect two nodes in the network. \n",
    "\n",
    "Using the synthetic social network, we will figure out how to answer the following questions:\n",
    "\n",
    "1. How long will it take for a message to spread through this group of friends? (making some assumptions, of course)\n",
    "2. How do we find the shortest path to get from individual A to individual B?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Shortest Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "nx.draw(G, with_labels=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's say we wanted to find the shortest path between two nodes. How would we approach this? One approach is what one would call a **breadth-first search** (http://en.wikipedia.org/wiki/Breadth-first_search). While not necessarily the fastest, it is the easiest to conceptualize. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The approach is essentially as such:\n",
    "\n",
    "1. Begin with a queue of the starting node.\n",
    "2. Add the neighbors of that node to the queue.\n",
    "    1. If destination node is present in the queue, end.\n",
    "    2. If destination node is not present, proceed.\n",
    "3. For each node in the queue:\n",
    "    1. Remove node from the queue.\n",
    "    2. Add neighbors of the node to the queue. Check if destination node is present or not.\n",
    "    3. If destination node is present, repeat Step 3.\n",
    "    4. If destination node is not present, continue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Try implementing this algorithm in a function called `path_exists(node1, node2, G)`.\n",
    "\n",
    "The function should take in two nodes, `node1` and `node2`, and the graph `G` that they belong to, and return a Boolean that indicates whether a path exists between those two nodes or not. For convenience, also print out whether a path exists or not between the two node.s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def path_exists(node1, node2, G):\n",
    "    \"\"\"\n",
    "    This function checks whether a path exists between two nodes (node1, node2) in graph G.\n",
    "    \"\"\"\n",
    "    # Fill in code below\n",
    "    queue = [______]\n",
    "    destination = _______\n",
    "    \n",
    "    for _____ in ______:\n",
    "        ______ = G.neighbors(_____)\n",
    "        if _______ in ______:\n",
    "            print('Path exists between nodes {0} and {1}'.format(node1, node2))\n",
    "            return True\n",
    "            break\n",
    "        else:\n",
    "            _______.remove(_______)\n",
    "            _______.extend(_______)\n",
    "        if len(_______) == ____:\n",
    "            print('Path does not exist between nodes {0} and {1}'.format(node1, node2))\n",
    "            return False\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "And testing the function on a few test cases:\n",
    "\n",
    "- 18 and any other node (should return `False`)\n",
    "- 29 and 26 (should return `True`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "path_exists(18, 5, G)\n",
    "path_exists(29, 26, G)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If you write an algorithm that runs breadth-first, the recursion pattern is likely to follow what we have done above. If you do depth-first, the recursion pattern is likely to look a bit different. We can figure that out together later, if there is time.\n",
    "\n",
    "Meanwhile... thankfully, NetworkX has a function for us to use, titled `has_path`, so we don't have to implement this on our own. :-)\n",
    "\n",
    "http://networkx.lanl.gov/reference/generated/networkx.algorithms.shortest_paths.generic.has_path.html#networkx.algorithms.shortest_paths.generic.has_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "nx.has_path(G, 18, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "NetworkX also has other shortest path algorithms implemented. \n",
    "\n",
    "https://networkx.github.io/documentation/latest/reference/generated/networkx.algorithms.shortest_paths.unweighted.predecessor.html#networkx.algorithms.shortest_paths.unweighted.predecessor\n",
    "\n",
    "We can build upon these to build our own graph query functions. Let's see if we can trace the shortest path from one node to another."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`nx.shortest_path(G, source, target)` gives us a list of nodes that exist within one of the shortest paths between the two nodes. (Not all paths are guaranteed to be found.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "nx.shortest_path(G, 4, 14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Incidentally, the node list is in order as well - we will travel through 19 and 17 in that order to get from 14 from 4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Write a function that extracts the edges in the shortest path between two nodes and puts them into a new graph, and draws it to the screen. It should also return an error if there is no path between the two nodes.\n",
    "\n",
    "Hint: You may want to use `G.subgraph(iterable_of_nodes)` to extract just the nodes and edges of interest from the graph `G`. You might want to use the following line of code somewhere:\n",
    "\n",
    "    newG = G.subgraph(nodes_of_interest)\n",
    "    \n",
    "`newG` will be a new graph, of the same type as `G`, comprised of the nodes of interest and the edges that connect them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Possible Answer:\n",
    "\n",
    "def extract_path_edges(G, source, target):\n",
    "    # Check to make sure that a path does exists between source and target.\n",
    "    \n",
    "    # Fill in answer below.\n",
    "        \n",
    "newG = extract_path_edges(G, 1, 14)\n",
    "nx.draw(newG, with_labels=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Since we've been drawing some graphs to screen, we might as well draw a few other things while we're on a roll.\n",
    "\n",
    "Write a function that extracts only node, its neighbors, and the edges between that node and its neighbors as a new graph. Then, draw the new graph to screen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Possible Answer\n",
    "\n",
    "def extract_neighbor_edges(G, node):\n",
    "    # Fill in code below\n",
    "    \n",
    "    \n",
    "    return newG\n",
    "\n",
    "fig = plt.figure(0)\n",
    "newG = extract_neighbor_edges(G, 19)\n",
    "nx.draw(newG, with_labels=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def extract_neighbor_edges2(G, node):\n",
    "    # Fill in code below\n",
    "    \n",
    "    \n",
    "    return newG\n",
    "\n",
    "fig = plt.figure(1)\n",
    "newG = extract_neighbor_edges2(G, 19)\n",
    "nx.draw(newG, with_labels=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Challenge Exercises (optional)\n",
    "\n",
    "Let's try some other problems that build on the NetworkX API.\n",
    "\n",
    "Refer to the following for the relevant functions:\n",
    "\n",
    "https://networkx.github.io/documentation/latest/reference/algorithms.shortest_paths.html\n",
    "\n",
    "1. If we want a message to go from one person to another person, and we assume that the message takes 1 day for the initial step and 1 additional day per step in the transmission chain (i.e. the first step takes 1 day, the second step takes 2 days etc.), how long will the message take to spread from any two given individuals? Write a function to compute this.\n",
    "2. What is the distribution of message spread times from person to person? What about chain lengths?\n",
    "3. Are there certain individuals who consistently show up in the chain? (Hint: you might wish to use the following functions/objects:\n",
    "    1. `Counter` object from the `collections` module \n",
    "    2. `combinations` function from the `itertools` module.\n",
    "    3. `all_shortest_paths(G, node1, node2)` which is part of the networkX algorithms.\n",
    "4. As a bonus, if you were able to compute the answer to question 3, can you plot a histogram of the number of times each node shows up in a connecting path?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Hint to Question 1:\n",
    "# All we need here is the length of the path.\n",
    "\n",
    "def compute_transmission_time(G, source, target):\n",
    "    \"\"\"\n",
    "    Fill in code below.\n",
    "    \"\"\"\n",
    "\n",
    "    return time\n",
    "\n",
    "compute_transmission_time(G, 14, 4)       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Hint to Question 2:\n",
    "# We need to know the length of every single shortest path between every pair of nodes.\n",
    "# If we don't put a source and target into the nx.shortest_path_length(G) function call, then\n",
    "# we get a dictionary of dictionaries, where all source-->target-->lengths are shown.\n",
    "\n",
    "# Fill in code below:\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Fill in code below for Question 3:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Hint to Question 4:\n",
    "# Use plt.bar()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hubs Revisited\n",
    "\n",
    "It looks like individual 19 is an important person of some sorts - if a message has to be passed through the network in the shortest time possible, then usually it'll go through person 19. Such a person has a high **betweenness centrality**. This is implemented as one of NetworkX's centrality algorithms. Check out the Wikipedia page for a further description.\n",
    "\n",
    "http://en.wikipedia.org/wiki/Betweenness_centrality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "btws = nx.betweenness_centrality(G, normalized=False)\n",
    "plt.bar(btws.keys(), btws.values())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Plot betweeness centrality against degree centrality for the synthetic social network above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Possible answer:\n",
    "deg_cent_sorted = _______\n",
    "btw_cent_sorted = _______\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Make the plot\n",
    "plt.scatter(deg_cent_sorted, btw_cent_sorted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Think about it...**\n",
    "\n",
    "From the scatter plot, we can see that the dots don't all fall on the same line. Degree centrality and betweenness centrality don't necessarily correlate. Can you think of a reason why?\n",
    "\n",
    "What would be the degree centrality and betweenness centrality of the middle connecting node in the **barbell graph** below?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "nx.draw(nx.barbell_graph(5, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}