slides-reduced.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 1 + 1 = 1 or Record Deduplication with Python\n",
    "Flávio Juvenal  \n",
    "[@flaviojuvenal](https://twitter.com/flaviojuvenal)  \n",
    "\n",
    "<p>\n",
    "    Partner at <a href=\"http://www.vinta.com.br?dedupe-slides=1\"><img class=\"vinta_logo\" style=\"display: inline-block; margin-left: 0;\" src=\"vinta.png\" width=300></a>\n",
    "</p>\n",
    "\n",
    "Check the other Vinta talks at: http://bit.ly/vinta-pygotham18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import warnings; warnings.simplefilter('ignore')\n",
    "import logging; logging.disable(level=logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<p style=\"font-size: 2em; text-weight: bold\">\n",
    "    Slides available at: <a href=\"http://bit.ly/pygotham-dup\">bit.ly/pygotham-dup</a>\n",
    "</p>\n",
    "<p>\n",
    "    Jupyter Notebook source at: <a href=\"https://github.com/vintasoftware/deduplication-slides/\">github.com/vintasoftware/deduplication-slides/</a>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Real world data is a mess!\n",
    "Probably you dealt with data like this before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = [\n",
    "    (\"Art's Delicatessen\",\"12224 Ventura Blvd\",\"Studio City\"),\n",
    "    (\"Arts Deli\",\"12224 Ventura Blvd.\",\"LA\"),\n",
    "    (\"Art's Deli and Restaurant\",\"12224 Ventura Boulevard\",\"Los Angeles\"),\n",
    "    (\"Art's Delicatessen & Restaurant\",\"12220 Ventura Blvd.\",\"LA\"),\n",
    "    (\"Fringale\", \"570 4th St.\", \"San Francisco\"),\n",
    "    (\"Fringale Restaurant\", \"570 Fourth Street\", \"SF\"),\n",
    "]\n",
    "\n",
    "df = pd.DataFrame(data, columns=['restaurant', 'address', 'city'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>restaurant</th>\n",
       "      <th>address</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Art's Delicatessen</td>\n",
       "      <td>12224 Ventura Blvd</td>\n",
       "      <td>Studio City</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Arts Deli</td>\n",
       "      <td>12224 Ventura Blvd.</td>\n",
       "      <td>LA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Art's Deli and Restaurant</td>\n",
       "      <td>12224 Ventura Boulevard</td>\n",
       "      <td>Los Angeles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Art's Delicatessen &amp; Restaurant</td>\n",
       "      <td>12220 Ventura Blvd.</td>\n",
       "      <td>LA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fringale</td>\n",
       "      <td>570 4th St.</td>\n",
       "      <td>San Francisco</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Fringale Restaurant</td>\n",
       "      <td>570 Fourth Street</td>\n",
       "      <td>SF</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        restaurant                  address           city\n",
       "0               Art's Delicatessen       12224 Ventura Blvd    Studio City\n",
       "1                        Arts Deli      12224 Ventura Blvd.             LA\n",
       "2        Art's Deli and Restaurant  12224 Ventura Boulevard    Los Angeles\n",
       "3  Art's Delicatessen & Restaurant      12220 Ventura Blvd.             LA\n",
       "4                         Fringale              570 4th St.  San Francisco\n",
       "5              Fringale Restaurant        570 Fourth Street             SF"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The solution is to perform **Deduplication** (aka [Record Linkage](https://en.wikipedia.org/wiki/Record_linkage)). It works by joining records in a **fuzzy way** using data like names, addresses, phone numbers, dates, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Fuzzy comparison of strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import jellyfish"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8833333333333334"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jellyfish.jaro_winkler(\"Art's Delicatessen\", \"Arts Deli\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4583333333333333"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jellyfish.jaro_winkler(\"Arts Deli\", \"Fringale\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Fuzzy comparison of addresses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Geocoding street addresses, i.e., converting them to **latitude/longitude** is very useful for matching because geocoders can:\n",
    "- clean irrelevant address variations\n",
    "- enable the calculation of geometric distance using lat/lng"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import geocoder\n",
    "\n",
    "full_addresses = [\n",
    "    \"20863 Stevens Creek Blvd., ste 330, 95015, CUPERTINO, CA, USA\",\n",
    "    \"20863 STEVENS CREEK Boulevard STE 330, 95014, Cupertio, CA, USA\",\n",
    "]\n",
    "\n",
    "full_addresses_latlng = []\n",
    "with requests.Session() as session:\n",
    "    for a in full_addresses:\n",
    "        a_geocoded = geocoder.google(a, session=session)\n",
    "        full_addresses_latlng.append((a_geocoded.latlng, a_geocoded.postal))\n",
    "\n",
    "address_latlng = list(zip(full_addresses, full_addresses_latlng))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('20863 Stevens Creek Blvd., ste 330, 95015, CUPERTINO, CA, USA',\n",
      "  ([37.3241563, -122.0387297], '95014')),\n",
      " ('20863 STEVENS CREEK Boulevard STE 330, 95014, Cupertio, CA, USA',\n",
      "  ([37.3241563, -122.0387297], '95014'))]\n"
     ]
    }
   ],
   "source": [
    "import pprint\n",
    "\n",
    "pprint.pprint(address_latlng)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Google geocoder is able to ignore the following variations:\n",
    "- Lower/upper case\n",
    "- Blvd. vs Boulevard\n",
    "- Zipcode 95015 vs 95014\n",
    "- Cupertino vs Cupertio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 0/4 - Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Custom dataset based on [Restaurant](https://www.cs.utexas.edu/users/ml/riddle/data.html) dataset:\n",
    "- a well-known dataset used by researchers\n",
    "- made of 881 restaurant records from the Fodor's and Zagat's restaurant guides\n",
    "- contains 150 duplicates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "\\* We've introduced some changes that you can check by doing a `diff restaurant.original.csv restaurant.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>phone</th>\n",
       "      <th>type</th>\n",
       "      <th>cluster</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arnie morton's of chicago</td>\n",
       "      <td>435 s. la cienega blv.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>310/246-1501</td>\n",
       "      <td>american</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arnie morton's of chicago</td>\n",
       "      <td>435 s. la cienega blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>310-246-1501</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arnie morton</td>\n",
       "      <td>435 s. la cienega boulevard</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>310-246-1501</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art's delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>818/762-1221</td>\n",
       "      <td>american</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art's deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>818-762-1221</td>\n",
       "      <td>delis</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>art's deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>818-762-1221</td>\n",
       "      <td>delis</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>hotel bel-air</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>310/472-1211</td>\n",
       "      <td>californian</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bel-air hotel</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>310-472-1211</td>\n",
       "      <td>californian</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bel-air</td>\n",
       "      <td>701 stone canyon road</td>\n",
       "      <td>bel air</td>\n",
       "      <td>(310) 472-1211</td>\n",
       "      <td>american</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        name                         addr         city  \\\n",
       "0  arnie morton's of chicago       435 s. la cienega blv.  los angeles   \n",
       "1  arnie morton's of chicago      435 s. la cienega blvd.  los angeles   \n",
       "2               arnie morton  435 s. la cienega boulevard  los angeles   \n",
       "3         art's delicatessen          12224 ventura blvd.  studio city   \n",
       "4                 art's deli          12224 ventura blvd.  studio city   \n",
       "5                 art's deli          12224 ventura blvd.  los angeles   \n",
       "6              hotel bel-air         701 stone canyon rd.      bel air   \n",
       "7              bel-air hotel         701 stone canyon rd.      bel air   \n",
       "8                    bel-air        701 stone canyon road      bel air   \n",
       "\n",
       "            phone         type  cluster  \n",
       "0    310/246-1501     american        0  \n",
       "1    310-246-1501  steakhouses        0  \n",
       "2    310-246-1501  steakhouses        0  \n",
       "3    818/762-1221     american        1  \n",
       "4    818-762-1221        delis        1  \n",
       "5    818-762-1221        delis        1  \n",
       "6    310/472-1211  californian        2  \n",
       "7    310-472-1211  californian        2  \n",
       "8  (310) 472-1211     american        2  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_with_truth = pd.read_csv('restaurant.csv', skip_blank_lines=True)\n",
    "df_with_truth.head(9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The dataset comes with the true matches indicated by the `cluster` column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We'll also remove the `phone` and `type` to makes things more difficult:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arnie morton's of chicago</td>\n",
       "      <td>435 s. la cienega blv.</td>\n",
       "      <td>los angeles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arnie morton's of chicago</td>\n",
       "      <td>435 s. la cienega blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arnie morton</td>\n",
       "      <td>435 s. la cienega boulevard</td>\n",
       "      <td>los angeles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art's delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art's deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>art's deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>hotel bel-air</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bel-air hotel</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bel-air</td>\n",
       "      <td>701 stone canyon road</td>\n",
       "      <td>bel air</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        name                         addr         city\n",
       "0  arnie morton's of chicago       435 s. la cienega blv.  los angeles\n",
       "1  arnie morton's of chicago      435 s. la cienega blvd.  los angeles\n",
       "2               arnie morton  435 s. la cienega boulevard  los angeles\n",
       "3         art's delicatessen          12224 ventura blvd.  studio city\n",
       "4                 art's deli          12224 ventura blvd.  studio city\n",
       "5                 art's deli          12224 ventura blvd.  los angeles\n",
       "6              hotel bel-air         701 stone canyon rd.      bel air\n",
       "7              bel-air hotel         701 stone canyon rd.      bel air\n",
       "8                    bel-air        701 stone canyon road      bel air"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_with_truth.drop(columns=['cluster', 'phone', 'type'])\n",
    "df.head(9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we'll clean the column values! Cleaning `name`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arnie morton s of chicago</td>\n",
       "      <td>435 s. la cienega blv.</td>\n",
       "      <td>los angeles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arnie morton s of chicago</td>\n",
       "      <td>435 s. la cienega blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arnie morton</td>\n",
       "      <td>435 s. la cienega boulevard</td>\n",
       "      <td>los angeles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art s delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art s deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>art s deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>hotel bel air</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bel air hotel</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bel air</td>\n",
       "      <td>701 stone canyon road</td>\n",
       "      <td>bel air</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        name                         addr         city\n",
       "0  arnie morton s of chicago       435 s. la cienega blv.  los angeles\n",
       "1  arnie morton s of chicago      435 s. la cienega blvd.  los angeles\n",
       "2               arnie morton  435 s. la cienega boulevard  los angeles\n",
       "3         art s delicatessen          12224 ventura blvd.  studio city\n",
       "4                 art s deli          12224 ventura blvd.  studio city\n",
       "5                 art s deli          12224 ventura blvd.  los angeles\n",
       "6              hotel bel air         701 stone canyon rd.      bel air\n",
       "7              bel air hotel         701 stone canyon rd.      bel air\n",
       "8                    bel air        701 stone canyon road      bel air"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "irrelevant_regex = re.compile(r'[^a-z0-9\\s]')\n",
    "multispace_regex = re.compile(r'\\s\\s+')\n",
    "\n",
    "def assign_no_symbols_name(df):\n",
    "    return df.assign(\n",
    "        name=df['name']\n",
    "             .str.replace(irrelevant_regex, ' ')\n",
    "             .str.replace(multispace_regex, ' '))\n",
    "\n",
    "df = assign_no_symbols_name(df)\n",
    "df.head(9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Geocoding `addr`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "881 819\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "all_addresses = df['addr'].str.cat(df['city'], sep=', ').values\n",
    "unique_addresses = np.unique(all_addresses)\n",
    "print(len(all_addresses), len(unique_addresses))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Failed to get postal from 7\n",
      "Failed to get latlng from 0\n"
     ]
    }
   ],
   "source": [
    "import os.path\n",
    "import json\n",
    "\n",
    "geocoding_filename = 'address_to_geocoding.json'\n",
    "\n",
    "def geocode_addresses(address_to_geocoding):\n",
    "    remaining_addresses = (\n",
    "        set(unique_addresses) -\n",
    "        set(k for k, v in address_to_geocoding.items() if v is not None and 'lat' in v))\n",
    "\n",
    "    with requests.Session() as session:\n",
    "        for i, address in enumerate(remaining_addresses):\n",
    "            print(f\"Geocoding {i + 1}/{len(remaining_addresses)}\")\n",
    "            geocode_result = geocoder.google(address, session=session)\n",
    "            address_to_geocoding[address] = geocode_result.json\n",
    "\n",
    "        with open(geocoding_filename, 'w') as f:\n",
    "            json.dump(address_to_geocoding, f, indent=4)\n",
    "\n",
    "if not os.path.exists(geocoding_filename):\n",
    "    address_to_geocoding = {}\n",
    "    geocode_addresses(address_to_geocoding)\n",
    "else:\n",
    "    with open(geocoding_filename) as f:\n",
    "        address_to_geocoding = json.load(f)\n",
    "    geocode_addresses(address_to_geocoding)\n",
    " \n",
    "address_to_postal = {\n",
    "    k: v['postal']\n",
    "    for k, v in address_to_geocoding.items()\n",
    "    if v is not None and 'postal' in v\n",
    "}\n",
    "address_to_latlng = {\n",
    "    k: (v['lat'], v['lng'])\n",
    "    for k, v in address_to_geocoding.items()\n",
    "    if v is not None\n",
    "}\n",
    "print(f\"Failed to get postal from {len(address_to_geocoding) - len(address_to_postal)}\")\n",
    "print(f\"Failed to get latlng from {len(address_to_geocoding) - len(address_to_latlng)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arnie morton s of chicago</td>\n",
       "      <td>435 s. la cienega blv.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.070609</td>\n",
       "      <td>-118.376722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arnie morton s of chicago</td>\n",
       "      <td>435 s. la cienega blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.070609</td>\n",
       "      <td>-118.376722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arnie morton</td>\n",
       "      <td>435 s. la cienega boulevard</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.070609</td>\n",
       "      <td>-118.376722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art s delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art s deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>art s deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        name                         addr         city postal  \\\n",
       "0  arnie morton s of chicago       435 s. la cienega blv.  los angeles  90048   \n",
       "1  arnie morton s of chicago      435 s. la cienega blvd.  los angeles  90048   \n",
       "2               arnie morton  435 s. la cienega boulevard  los angeles  90048   \n",
       "3         art s delicatessen          12224 ventura blvd.  studio city  91604   \n",
       "4                 art s deli          12224 ventura blvd.  studio city  91604   \n",
       "5                 art s deli          12224 ventura blvd.  los angeles  91604   \n",
       "\n",
       "         lat         lng  \n",
       "0  34.070609 -118.376722  \n",
       "1  34.070609 -118.376722  \n",
       "2  34.070609 -118.376722  \n",
       "3  34.142966 -118.399469  \n",
       "4  34.142966 -118.399469  \n",
       "5  34.142966 -118.399469  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def assign_postal_lat_lng(df):\n",
    "    addresses = df['addr'].str.cat(df['city'], sep=', ')\n",
    "    addresses_to_postal = [address_to_postal.get(a) for a in addresses]\n",
    "    addresses_to_lat = [address_to_latlng[a][0] if a in address_to_latlng else None for a in addresses]\n",
    "    addresses_to_lng = [address_to_latlng[a][1] if a in address_to_latlng else None for a in addresses]\n",
    "\n",
    "    return df.assign(postal=addresses_to_postal, lat=addresses_to_lat, lng=addresses_to_lng)\n",
    "\n",
    "df = assign_postal_lat_lng(df)\n",
    "df.head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1/4 - Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the next steps, we'll use the library **recordlinkage**, aka **[Python Record Linkage Toolkit](http://recordlinkage.readthedocs.io/en/stable/)**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We have the cleaned records, now we want the **pairs** to compare to find matches.\n",
    "\n",
    "To **produce** the pairs, we could do a \"full\" index, i.e., all records against all records:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:recordlinkage:indexing - performance warning - A full index can result in large number of record pairs.\n",
      "Full index: 881 records, 387640 pairs\n"
     ]
    }
   ],
   "source": [
    "import recordlinkage as rl\n",
    "from recordlinkage.index import Full\n",
    "\n",
    "full_indexer = Full()\n",
    "pairs = full_indexer.index(df)\n",
    "\n",
    "print(f\"Full index: {len(df)} records, {len(pairs)} pairs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The number of pairs grows too fast as the number of records grows:  \n",
    "it **grows quadratically** with the formula `len(df) * (len(df) - 1) / 2`\n",
    "\n",
    "To avoid wasting too much time, we need **indexing**. We need to produce only pairs that are good candidates of being duplicates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A basic way to index is **blocking**: produce pairs that have **some value in common**.\n",
    "\n",
    "Below we compute pairs that have **equal values for postal** code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Postal index: 6577 pairs\n"
     ]
    }
   ],
   "source": [
    "from recordlinkage.index import Block\n",
    "\n",
    "postal_indexer = Block('postal')\n",
    "pairs = postal_indexer.index(df)\n",
    "\n",
    "print(f\"Postal index: {len(pairs)} pairs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# We'll swap the levels of pairs MultiIndex returned by recordlinkage\n",
    "# to make it simpler to interpret. Here we get the upper triangle of the pairs matrix.\n",
    "pairs = pairs.swaplevel().sortlevel()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0,   1],\n",
       "       [  0,   2],\n",
       "       [  0,  35],\n",
       "       [  0,  36],\n",
       "       [  0, 262],\n",
       "       [  0, 266],\n",
       "       [  0, 273],\n",
       "       [  0, 293],\n",
       "       [  0, 685],\n",
       "       [  0, 695]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pairs.to_frame()[:10].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2/4 - Comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we want to run comparisons on the indexed pairs to produce **a comparison vector for each pair**. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>latlng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <th>200</th>\n",
       "      <td>0.5</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.9</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         name  addr  postal  latlng\n",
       "100 200   0.5   0.8     0.9       1"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame([[0.5, 0.8, 0.9, 1]],\n",
    "             columns=['name', 'addr', 'postal', 'latlng'],\n",
    "             index=pd.MultiIndex.from_arrays([[100], [200]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The pair of records `(100, 200)` has:\n",
    "- Low similarity on `name`s\n",
    "- Some similarity on `addr`s\n",
    "- High similarity on `postal`s\n",
    "- Equal `latlng`s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To compute the comparison vectors, we define a **similarity function** for each column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "comp = rl.Compare()\n",
    "comp.string('name', 'name', method='jarowinkler', label='name')\n",
    "comp.string('addr', 'addr', method='jarowinkler', label='addr')\n",
    "comp.string('postal', 'postal', method='jarowinkler', label='postal')\n",
    "comp.geo('lat', 'lng', 'lat', 'lng', method='exp', scale=0.1, offset=0.01, label='latlng');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we'll compute the **comparison vectors**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>latlng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th>1</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.985507</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.896000</td>\n",
       "      <td>0.910774</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>0.505476</td>\n",
       "      <td>0.528075</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.031305</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>0.505476</td>\n",
       "      <td>0.478788</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.031305</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>262</th>\n",
       "      <td>0.413333</td>\n",
       "      <td>0.743613</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.016543</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           name      addr  postal    latlng\n",
       "0 1    1.000000  0.985507     1.0  1.000000\n",
       "  2    0.896000  0.910774     1.0  1.000000\n",
       "  35   0.505476  0.528075     1.0  0.031305\n",
       "  36   0.505476  0.478788     1.0  0.031305\n",
       "  262  0.413333  0.743613     1.0  0.016543"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comparison_vectors = comp.compute(pairs, df)\n",
    "comparison_vectors.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now, with our comparison vectors, we'll explore **different ways** to classify them as matches and nonmatches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 3/4 - Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Threshold-Based Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A simple way to classify comparison vectors as matches or nonmatches is to compute a **weighted average** over the vectors to get a **score**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>latlng</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th>1</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.985507</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.996047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.896000</td>\n",
       "      <td>0.910774</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.928393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>0.505476</td>\n",
       "      <td>0.528075</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.031305</td>\n",
       "      <td>0.470383</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>0.505476</td>\n",
       "      <td>0.478788</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.031305</td>\n",
       "      <td>0.456941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>262</th>\n",
       "      <td>0.413333</td>\n",
       "      <td>0.743613</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.016543</td>\n",
       "      <td>0.484599</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           name      addr  postal    latlng     score\n",
       "0 1    1.000000  0.985507     1.0  1.000000  0.996047\n",
       "  2    0.896000  0.910774     1.0  1.000000  0.928393\n",
       "  35   0.505476  0.528075     1.0  0.031305  0.470383\n",
       "  36   0.505476  0.478788     1.0  0.031305  0.456941\n",
       "  262  0.413333  0.743613     1.0  0.016543  0.484599"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = np.average(\n",
    "    comparison_vectors.values,\n",
    "    axis=1,\n",
    "    weights=[50, 30, 10, 20])\n",
    "scored_comparison_vectors = comparison_vectors.assign(score=scores)\n",
    "scored_comparison_vectors.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "By looking at the data, we know record `0` truly matches `1` and `2`, so we'll **classify as a match** any pair with score `>= 0.9`. That's our **threshold**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arnie morton s of chicago</td>\n",
       "      <td>435 s. la cienega blv.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.070609</td>\n",
       "      <td>-118.376722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arnie morton s of chicago</td>\n",
       "      <td>435 s. la cienega blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.070609</td>\n",
       "      <td>-118.376722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arnie morton</td>\n",
       "      <td>435 s. la cienega boulevard</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.070609</td>\n",
       "      <td>-118.376722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art s delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art s deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        name                         addr         city postal  \\\n",
       "0  arnie morton s of chicago       435 s. la cienega blv.  los angeles  90048   \n",
       "1  arnie morton s of chicago      435 s. la cienega blvd.  los angeles  90048   \n",
       "2               arnie morton  435 s. la cienega boulevard  los angeles  90048   \n",
       "3         art s delicatessen          12224 ventura blvd.  studio city  91604   \n",
       "4                 art s deli          12224 ventura blvd.  studio city  91604   \n",
       "\n",
       "         lat         lng  \n",
       "0  34.070609 -118.376722  \n",
       "1  34.070609 -118.376722  \n",
       "2  34.070609 -118.376722  \n",
       "3  34.142966 -118.399469  \n",
       "4  34.142966 -118.399469  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>latlng</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">0</th>\n",
       "      <th>1</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.985507</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.996047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.896000</td>\n",
       "      <td>0.910774</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.928393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <td>0.896000</td>\n",
       "      <td>0.923779</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.931940</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>4</th>\n",
       "      <td>0.911111</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.959596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.911111</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.959596</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         name      addr  postal  latlng     score\n",
       "0 1  1.000000  0.985507     1.0     1.0  0.996047\n",
       "  2  0.896000  0.910774     1.0     1.0  0.928393\n",
       "1 2  0.896000  0.923779     1.0     1.0  0.931940\n",
       "3 4  0.911111  1.000000     1.0     1.0  0.959596\n",
       "  5  0.911111  1.000000     1.0     1.0  0.959596"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matches = scored_comparison_vectors[\n",
    "    scored_comparison_vectors['score'] >= 0.9]\n",
    "matches.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Since we have the true match status on `cluster` column, we can **evaluate** how well our threshold classification did:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Golden pairs: 150\n"
     ]
    }
   ],
   "source": [
    "golden_pairs = Block('cluster').index(df_with_truth)\n",
    "golden_pairs = golden_pairs.swaplevel().sortlevel()[0]\n",
    "print(\"Golden pairs:\", len(golden_pairs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "true_positives total: 128\n",
      "false_positives total: 2\n",
      "false_negatives total: 22\n"
     ]
    }
   ],
   "source": [
    "found_pairs_set = set(matches.index)\n",
    "\n",
    "golden_pairs_set = set(golden_pairs)\n",
    "\n",
    "true_positives = golden_pairs_set & found_pairs_set\n",
    "false_positives = found_pairs_set - golden_pairs_set\n",
    "false_negatives = golden_pairs_set - found_pairs_set\n",
    "\n",
    "print('true_positives total:', len(true_positives))\n",
    "print('false_positives total:', len(false_positives))\n",
    "print('false_negatives total:', len(false_negatives))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We've got a small number of **false positives**. Some are really tricky cases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False positives:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>ritz carlton cafe buckhead</td>\n",
       "      <td>3434 peachtree rd. ne</td>\n",
       "      <td>30326</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>ritz carlton dining room buckhead</td>\n",
       "      <td>3434 peachtree rd. ne</td>\n",
       "      <td>30326</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   name                   addr postal  \\\n",
       "196         ritz carlton cafe buckhead   3434 peachtree rd. ne  30326   \n",
       "198  ritz carlton dining room buckhead   3434 peachtree rd. ne  30326   \n",
       "\n",
       "           lat        lng  \n",
       "196  33.850807 -84.364227  \n",
       "198  33.850807 -84.364227  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>ritz carlton restaurant</td>\n",
       "      <td>181 peachtree st.</td>\n",
       "      <td>30303</td>\n",
       "      <td>33.758579</td>\n",
       "      <td>-84.387066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>839</th>\n",
       "      <td>ritz carlton cafe atlanta</td>\n",
       "      <td>181 peachtree st.</td>\n",
       "      <td>30303</td>\n",
       "      <td>33.758579</td>\n",
       "      <td>-84.387066</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           name               addr postal        lat  \\\n",
       "200     ritz carlton restaurant  181 peachtree st.  30303  33.758579   \n",
       "839  ritz carlton cafe atlanta   181 peachtree st.  30303  33.758579   \n",
       "\n",
       "           lng  \n",
       "200 -84.387066  \n",
       "839 -84.387066  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(f\"False positives:\")\n",
    "for false_positive_pair in false_positives:\n",
    "    display(df.loc[list(false_positive_pair)][['name', 'addr', 'postal', 'lat', 'lng']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On the other hand, we got a lot of **false negatives**. We've **missed** a lot of matches!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False negatives (sample 10 of 22):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>brasserie le coze</td>\n",
       "      <td>3393 peachtree rd.  lenox square mall  near ne...</td>\n",
       "      <td>30326</td>\n",
       "      <td>33.846181</td>\n",
       "      <td>-84.364109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>brasserie le coze</td>\n",
       "      <td>3393 peachtree rd.</td>\n",
       "      <td>30326</td>\n",
       "      <td>33.846869</td>\n",
       "      <td>-84.362512</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  name                                               addr  \\\n",
       "170  brasserie le coze  3393 peachtree rd.  lenox square mall  near ne...   \n",
       "171  brasserie le coze                                 3393 peachtree rd.   \n",
       "\n",
       "    postal        lat        lng  \n",
       "170  30326  33.846181 -84.364109  \n",
       "171  30326  33.846869 -84.362512  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>heera of india</td>\n",
       "      <td>595 piedmont ave.  rio shopping mall</td>\n",
       "      <td>30324</td>\n",
       "      <td>33.795817</td>\n",
       "      <td>-84.370602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>heera of india</td>\n",
       "      <td>595 piedmont ave.</td>\n",
       "      <td>30308</td>\n",
       "      <td>33.770495</td>\n",
       "      <td>-84.381425</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               name                                  addr postal        lat  \\\n",
       "182  heera of india  595 piedmont ave.  rio shopping mall  30324  33.795817   \n",
       "183  heera of india                     595 piedmont ave.  30308  33.770495   \n",
       "\n",
       "           lng  \n",
       "182 -84.370602  \n",
       "183 -84.381425  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>abruzzi</td>\n",
       "      <td>2355 peachtree rd.  peachtree battle shopping ...</td>\n",
       "      <td>30305</td>\n",
       "      <td>33.820137</td>\n",
       "      <td>-84.387280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>165</th>\n",
       "      <td>abruzzi</td>\n",
       "      <td>2355 peachtree rd. ne</td>\n",
       "      <td>30305</td>\n",
       "      <td>33.824647</td>\n",
       "      <td>-84.387453</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        name                                               addr postal  \\\n",
       "164  abruzzi  2355 peachtree rd.  peachtree battle shopping ...  30305   \n",
       "165  abruzzi                              2355 peachtree rd. ne  30305   \n",
       "\n",
       "           lat        lng  \n",
       "164  33.820137 -84.387280  \n",
       "165  33.824647 -84.387453  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>restaurant katsu</td>\n",
       "      <td>1972 n. hillhurst ave.</td>\n",
       "      <td>90027</td>\n",
       "      <td>34.107405</td>\n",
       "      <td>-118.28719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>katsu</td>\n",
       "      <td>1972 hillhurst ave.</td>\n",
       "      <td>90027</td>\n",
       "      <td>34.107405</td>\n",
       "      <td>-118.28719</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name                    addr postal        lat        lng\n",
       "28  restaurant katsu  1972 n. hillhurst ave.  90027  34.107405 -118.28719\n",
       "29             katsu     1972 hillhurst ave.  90027  34.107405 -118.28719"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w 3rd</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.073417</td>\n",
       "      <td>-118.381096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>locanda</td>\n",
       "      <td>w. third st.</td>\n",
       "      <td>None</td>\n",
       "      <td>34.068944</td>\n",
       "      <td>-118.322451</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name          addr postal        lat         lng\n",
       "36  locanda veneta    8638 w 3rd  90048  34.073417 -118.381096\n",
       "37         locanda  w. third st.   None  34.068944 -118.322451"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>the palm</td>\n",
       "      <td>9001 santa monica blvd.</td>\n",
       "      <td>90069</td>\n",
       "      <td>34.083064</td>\n",
       "      <td>-118.387282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>palm the los angeles</td>\n",
       "      <td>9001 santa monica blvd.</td>\n",
       "      <td>90069</td>\n",
       "      <td>34.083064</td>\n",
       "      <td>-118.387282</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     name                     addr postal        lat  \\\n",
       "40               the palm  9001 santa monica blvd.  90069  34.083064   \n",
       "41  palm the los angeles   9001 santa monica blvd.  90069  34.083064   \n",
       "\n",
       "           lng  \n",
       "40 -118.387282  \n",
       "41 -118.387282  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>136</th>\n",
       "      <td>shun lee west</td>\n",
       "      <td>43 w. 65th st.</td>\n",
       "      <td>10023</td>\n",
       "      <td>40.772900</td>\n",
       "      <td>-73.981348</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>shun lee palace</td>\n",
       "      <td>155 e. 55th st.</td>\n",
       "      <td>10022</td>\n",
       "      <td>40.759428</td>\n",
       "      <td>-73.969068</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name             addr postal        lat        lng\n",
       "136    shun lee west   43 w. 65th st.  10023  40.772900 -73.981348\n",
       "137  shun lee palace  155 e. 55th st.  10022  40.759428 -73.969068"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>3rd st.</td>\n",
       "      <td>None</td>\n",
       "      <td>33.494790</td>\n",
       "      <td>-112.069374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w. third st.</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.073417</td>\n",
       "      <td>-118.381096</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name               addr postal        lat         lng\n",
       "34  locanda veneta            3rd st.   None  33.494790 -112.069374\n",
       "35  locanda veneta  8638 w. third st.  90048  34.073417 -118.381096"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>l orangerie</td>\n",
       "      <td>903 n. la cienega blvd.</td>\n",
       "      <td>90069</td>\n",
       "      <td>34.087098</td>\n",
       "      <td>-118.376626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>l orangerie</td>\n",
       "      <td>903 n. la cienega blvd.</td>\n",
       "      <td>90028</td>\n",
       "      <td>34.100833</td>\n",
       "      <td>-118.325613</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           name                     addr postal        lat         lng\n",
       "30  l orangerie  903 n. la cienega blvd.  90069  34.087098 -118.376626\n",
       "31  l orangerie  903 n. la cienega blvd.  90028  34.100833 -118.325613"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w. third st.</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.073417</td>\n",
       "      <td>-118.381096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>locanda</td>\n",
       "      <td>w. third st.</td>\n",
       "      <td>None</td>\n",
       "      <td>34.068944</td>\n",
       "      <td>-118.322451</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name               addr postal        lat         lng\n",
       "35  locanda veneta  8638 w. third st.  90048  34.073417 -118.381096\n",
       "37         locanda       w. third st.   None  34.068944 -118.322451"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(f\"False negatives (sample 10 of {len(false_negatives)}):\")\n",
    "for false_negative_pair in list(false_negatives)[:10]:\n",
    "    display(df.loc[list(false_negative_pair)][['name', 'addr', 'postal', 'lat', 'lng']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Bue we've set the weights and the threshold **by guessing**, could we do any better?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Supervised Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of trying to guess weights and thresholds, we can train a **classifier** to learn how to classify matches and nonmatches based on some training data we provide:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>cluster</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>3rd st.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w. third st.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w 3rd</td>\n",
       "      <td>st los angeles</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cafe lalo</td>\n",
       "      <td>201 w. 83rd st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cafe lalo</td>\n",
       "      <td>201 w. 83rd st.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>les celebrites</td>\n",
       "      <td>160 central park s</td>\n",
       "      <td>new york</td>\n",
       "      <td>42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>les celebrites</td>\n",
       "      <td>155 w. 58th st.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>second avenue deli</td>\n",
       "      <td>156 2nd ave. at 10th st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>second avenue deli</td>\n",
       "      <td>156 second ave.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>smith &amp; wollensky</td>\n",
       "      <td>201 e. 49th st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>smith &amp; wollensky</td>\n",
       "      <td>797 third ave.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>chin's</td>\n",
       "      <td>3200 las vegas blvd. s</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>chin's</td>\n",
       "      <td>3200 las vegas blvd. s.</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>toulouse</td>\n",
       "      <td>b peachtree rd.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>toulouse</td>\n",
       "      <td>293-b peachtree rd.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>rose pistola</td>\n",
       "      <td>532 columbus ave.</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>rose pistola</td>\n",
       "      <td>532 columbus ave.</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>bistro garden</td>\n",
       "      <td>176 n. canon dr.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>remi</td>\n",
       "      <td>3rd st. promenade</td>\n",
       "      <td>santa monica</td>\n",
       "      <td>159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>remi</td>\n",
       "      <td>145 w. 53rd st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>334</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>west</td>\n",
       "      <td>63rd street steakhouse 44 w. 63rd st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>bistro</td>\n",
       "      <td>3400 las vegas blvd. s</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>mikado</td>\n",
       "      <td>3400 las vegas blvd. s</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>l'osteria del forno</td>\n",
       "      <td>519 columbus ave.</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>490</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>stars</td>\n",
       "      <td>150 redwood alley</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>stars cafe</td>\n",
       "      <td>500 van ness ave.</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>belvedere  the</td>\n",
       "      <td>9882 little santa monica blvd.</td>\n",
       "      <td>beverly hills</td>\n",
       "      <td>537</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>bernard's</td>\n",
       "      <td>515 s. olive st.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>bistro 45</td>\n",
       "      <td>45 s. mentor ave.</td>\n",
       "      <td>pasadena</td>\n",
       "      <td>540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>cafe '50s</td>\n",
       "      <td>838 lincoln blvd.</td>\n",
       "      <td>venice</td>\n",
       "      <td>545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>cafe blanc</td>\n",
       "      <td>9777 little santa monica blvd.</td>\n",
       "      <td>beverly hills</td>\n",
       "      <td>546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>la cachette</td>\n",
       "      <td>10506 little santa monica blvd.</td>\n",
       "      <td>century city</td>\n",
       "      <td>568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>moongate</td>\n",
       "      <td>3400 las vegas blvd. s.</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>666</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   name                                   addr  \\\n",
       "0        locanda veneta                                3rd st.   \n",
       "1        locanda veneta                      8638 w. third st.   \n",
       "2        locanda veneta                             8638 w 3rd   \n",
       "3             cafe lalo                        201 w. 83rd st.   \n",
       "4             cafe lalo                        201 w. 83rd st.   \n",
       "5        les celebrites                     160 central park s   \n",
       "6        les celebrites                        155 w. 58th st.   \n",
       "7    second avenue deli               156 2nd ave. at 10th st.   \n",
       "8    second avenue deli                        156 second ave.   \n",
       "9     smith & wollensky                        201 e. 49th st.   \n",
       "10    smith & wollensky                         797 third ave.   \n",
       "11               chin's                 3200 las vegas blvd. s   \n",
       "12               chin's                3200 las vegas blvd. s.   \n",
       "13             toulouse                        b peachtree rd.   \n",
       "14             toulouse                    293-b peachtree rd.   \n",
       "15         rose pistola                      532 columbus ave.   \n",
       "16         rose pistola                      532 columbus ave.   \n",
       "17        bistro garden                       176 n. canon dr.   \n",
       "18                 remi                      3rd st. promenade   \n",
       "19                 remi                        145 w. 53rd st.   \n",
       "20                 west  63rd street steakhouse 44 w. 63rd st.   \n",
       "21               bistro                 3400 las vegas blvd. s   \n",
       "22               mikado                 3400 las vegas blvd. s   \n",
       "23  l'osteria del forno                      519 columbus ave.   \n",
       "24                stars                      150 redwood alley   \n",
       "25           stars cafe                      500 van ness ave.   \n",
       "26       belvedere  the         9882 little santa monica blvd.   \n",
       "27            bernard's                       515 s. olive st.   \n",
       "28            bistro 45                      45 s. mentor ave.   \n",
       "29            cafe '50s                      838 lincoln blvd.   \n",
       "30           cafe blanc         9777 little santa monica blvd.   \n",
       "31          la cachette        10506 little santa monica blvd.   \n",
       "32             moongate                3400 las vegas blvd. s.   \n",
       "\n",
       "              city  cluster  \n",
       "0      los angeles       13  \n",
       "1      los angeles       13  \n",
       "2   st los angeles       13  \n",
       "3         new york       26  \n",
       "4    new york city       26  \n",
       "5         new york       42  \n",
       "6    new york city       42  \n",
       "7         new york       58  \n",
       "8    new york city       58  \n",
       "9         new york       62  \n",
       "10   new york city       62  \n",
       "11       las vegas       67  \n",
       "12       las vegas       67  \n",
       "13         atlanta       92  \n",
       "14         atlanta       92  \n",
       "15   san francisco      111  \n",
       "16   san francisco      111  \n",
       "17     los angeles      115  \n",
       "18    santa monica      159  \n",
       "19        new york      334  \n",
       "20        new york      375  \n",
       "21       las vegas      429  \n",
       "22       las vegas      446  \n",
       "23   san francisco      490  \n",
       "24   san francisco      514  \n",
       "25   san francisco      515  \n",
       "26   beverly hills      537  \n",
       "27     los angeles      539  \n",
       "28        pasadena      540  \n",
       "29          venice      545  \n",
       "30   beverly hills      546  \n",
       "31    century city      568  \n",
       "32       las vegas      666  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_training = pd.read_csv('restaurant-training.csv', skip_blank_lines=True)\n",
    "df_training = df_training.drop(columns=['type', 'phone'])\n",
    "df_training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We need to **preprocess** our training data too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>cluster</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>3rd st.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>13</td>\n",
       "      <td>None</td>\n",
       "      <td>33.494790</td>\n",
       "      <td>-112.069374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w. third st.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>13</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.073417</td>\n",
       "      <td>-118.381096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w 3rd</td>\n",
       "      <td>st los angeles</td>\n",
       "      <td>13</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.073417</td>\n",
       "      <td>-118.381096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cafe lalo</td>\n",
       "      <td>201 w. 83rd st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>26</td>\n",
       "      <td>10024</td>\n",
       "      <td>40.785981</td>\n",
       "      <td>-73.976727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cafe lalo</td>\n",
       "      <td>201 w. 83rd st.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>26</td>\n",
       "      <td>10024</td>\n",
       "      <td>40.785981</td>\n",
       "      <td>-73.976727</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             name               addr            city  cluster postal  \\\n",
       "0  locanda veneta            3rd st.     los angeles       13   None   \n",
       "1  locanda veneta  8638 w. third st.     los angeles       13  90048   \n",
       "2  locanda veneta         8638 w 3rd  st los angeles       13  90048   \n",
       "3       cafe lalo    201 w. 83rd st.        new york       26  10024   \n",
       "4       cafe lalo    201 w. 83rd st.   new york city       26  10024   \n",
       "\n",
       "         lat         lng  \n",
       "0  33.494790 -112.069374  \n",
       "1  34.073417 -118.381096  \n",
       "2  34.073417 -118.381096  \n",
       "3  40.785981  -73.976727  \n",
       "4  40.785981  -73.976727  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_training = assign_no_symbols_name(df_training)\n",
    "df_training = assign_postal_lat_lng(df_training)\n",
    "df_training.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We'll feed a **Support Vector Machine classifier** with our training data. SVMs are good for deduplication because they:\n",
    "- Are resilient to noise\n",
    "- Can handle correlated features (like 'addr' and 'latlng')\n",
    "- Are robust to imbalanced training sets (and that's natural on deduplication projects) [3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:recordlinkage:indexing - performance warning - A full index can result in large number of record pairs.\n"
     ]
    }
   ],
   "source": [
    "all_training_pairs = Full().index(df_training)\n",
    "matches_training_pairs = Block('cluster').index(df_training)\n",
    "\n",
    "training_vectors = comp.compute(all_training_pairs, df_training)\n",
    "\n",
    "svm = rl.SVMClassifier()\n",
    "svm.fit(training_vectors, matches_training_pairs);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "true_positives total: 128\n",
      "false_positives total: 2\n",
      "false_negatives total: 22\n",
      "\n",
      "svm_true_positives total: 133\n",
      "svm_false_positives total: 2\n",
      "svm_false_negatives total: 17\n"
     ]
    }
   ],
   "source": [
    "svm_pairs = svm.predict(comparison_vectors)\n",
    "svm_found_pairs_set = set(svm_pairs)\n",
    "\n",
    "svm_true_positives = golden_pairs_set & svm_found_pairs_set\n",
    "svm_false_positives = svm_found_pairs_set - golden_pairs_set\n",
    "svm_false_negatives = golden_pairs_set - svm_found_pairs_set\n",
    "\n",
    "print('true_positives total:', len(true_positives))\n",
    "print('false_positives total:', len(false_positives))\n",
    "print('false_negatives total:', len(false_negatives))\n",
    "print()\n",
    "print('svm_true_positives total:', len(svm_true_positives))\n",
    "print('svm_false_positives total:', len(svm_false_positives))\n",
    "print('svm_false_negatives total:', len(svm_false_negatives))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are [other classifiers](https://recordlinkage.readthedocs.io/en/stable/notebooks/classifiers.html) from recordlinkage library we could try, but the truth is:\n",
    "- It's **very difficult to build a good training set** that takes in account all important cases of matches/nonmatches;\n",
    "- And we're not even sure **if the indexing rules we used are really sane**. We could be:\n",
    "  * missing matches due blocking\n",
    "  * introducing mismatches due blocking + forgiving classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The **alternative** to that uncertainty is...\n",
    "\n",
    "### Active Learning Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Active Learning methods identify **training examples** that \"lead to maximal accuracy improvements\" [3], to train both:\n",
    "- optimal **classifier weights**\n",
    "- optimal **indexing/blocking rules**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A Python library called [**Dedupe**](https://docs.dedupe.io/en/latest/index.html) implements that. Let's see it in practice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import logging; logging.disable(level=logging.NOTSET)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from svm_dedupe import SVMDedupe\n",
    "import dedupe\n",
    "\n",
    "fields = [\n",
    "    {\n",
    "        'field': 'name',\n",
    "        'variable name': 'name',\n",
    "        'type': 'JaroWinkler',\n",
    "    },\n",
    "    {\n",
    "        'field': 'addr',\n",
    "        'variable name': 'addr',\n",
    "        'type': 'JaroWinkler',\n",
    "    },\n",
    "    {\n",
    "        'field': 'postal',\n",
    "        'variable name': 'postal',\n",
    "        'type': 'JaroWinkler'\n",
    "    },\n",
    "    {\n",
    "        'field': 'latlng',\n",
    "        'variable name': 'latlng',\n",
    "        'type': 'ExpLatLong'\n",
    "    },\n",
    "]\n",
    "\n",
    "deduper = SVMDedupe(fields)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:dedupe.api:((SimplePredicate: (firstIntegerPredicate, addr), SimplePredicate: (sameFiveCharStartPredicate, name)), (SimplePredicate: (latLongGridPredicate, latlng), SimplePredicate: (oneGramFingerprint, name)))\n"
     ]
    }
   ],
   "source": [
    "settings_filename = 'dedupe-simple-settings.pickle'\n",
    "if os.path.exists(settings_filename):\n",
    "    with open(settings_filename, 'rb') as sf:\n",
    "        deduper = dedupe.StaticDedupe(sf, num_cores=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "We need to **adapt** the data a bit to the format Dedupe wants:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "data_for_dedupe = df.assign(cluster=df_with_truth['cluster']).to_dict('index')\n",
    "for record in data_for_dedupe.values():\n",
    "    # Change nans to None\n",
    "    for k, v in record.items():\n",
    "        if isinstance(v, float) and np.isnan(v):\n",
    "            record[k] = None\n",
    "    \n",
    "    # Move lat and lng to a single field latlng\n",
    "    lat = record.pop('lat')\n",
    "    lng = record.pop('lng')\n",
    "    if lat is not None and lng is not None:\n",
    "        record['latlng'] = (lat, lng)\n",
    "    else:\n",
    "        record['latlng'] = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We train Dedupe by **labeling** the examples of matches and nonmatches it asks us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name : philippe the original\n",
      "addr : 1001 north alameda\n",
      "postal : 90012\n",
      "latlng : (34.059721, -118.237025)\n",
      "\n",
      "name : pisces\n",
      "addr : 95 ave. a\n",
      "postal : 10009\n",
      "latlng : (40.7256332, -73.984031)\n",
      "\n",
      "0/10 positive, 0/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "\n",
      "name : philippe the original\n",
      "addr : 1001 n. alameda st.\n",
      "postal : 90012\n",
      "latlng : (34.059721, -118.237025)\n",
      "\n",
      "name : mon kee seafood restaurant\n",
      "addr : 679 n. spring st.\n",
      "postal : 90012\n",
      "latlng : (34.0595568, -118.2382488)\n",
      "\n",
      "0/10 positive, 1/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "\n",
      "name : caffe vivaldi\n",
      "addr : 32 jones st. at bleecker st.\n",
      "postal : 10014\n",
      "latlng : (40.7317316, -74.00298049999999)\n",
      "\n",
      "name : patria\n",
      "addr : 250 park ave. s at 20th st.\n",
      "postal : 10003\n",
      "latlng : (40.7382552, -73.988214)\n",
      "\n",
      "0/10 positive, 2/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "\n",
      "name : i trulli\n",
      "addr : 122 e. 27th st.  between lexington and park aves.\n",
      "postal : 10028\n",
      "latlng : (40.77961699999999, -73.95631999999999)\n",
      "\n",
      "name : otabe\n",
      "addr : 68 e. 56th st.\n",
      "postal : 10022\n",
      "latlng : (40.7611775, -73.9720541)\n",
      "\n",
      "0/10 positive, 3/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "\n",
      "name : viva mercado s\n",
      "addr : 6182 w. flamingo rd.\n",
      "postal : 89103\n",
      "latlng : (36.1149027, -115.2269398)\n",
      "\n",
      "name : cafe con leche\n",
      "addr : 424 amsterdam ave.\n",
      "postal : 10024\n",
      "latlng : (40.7841454, -73.9778061)\n",
      "\n",
      "0/10 positive, 4/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "\n",
      "name : faz\n",
      "addr : 161 sutter st.\n",
      "postal : 94104\n",
      "latlng : (37.78973999999999, -122.4032937)\n",
      "\n",
      "name : splendido embarcadero\n",
      "addr : 4\n",
      "postal : None\n",
      "latlng : (37.7779649, -122.3962019)\n",
      "\n",
      "0/10 positive, 5/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "\n",
      "name : le bernardin\n",
      "addr : 155 w. 51st st.\n",
      "postal : 10019\n",
      "latlng : (40.7615691, -73.98180479999999)\n",
      "\n",
      "name : le bernardin\n",
      "addr : 155 w. 51st st.\n",
      "postal : 10019\n",
      "latlng : (40.7615691, -73.98180479999999)\n",
      "\n",
      "0/10 positive, 6/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "y\n",
      "\n",
      "name : cafe lalo\n",
      "addr : 201 w. 83rd st.\n",
      "postal : 10024\n",
      "latlng : (40.78598119999999, -73.97672659999999)\n",
      "\n",
      "name : cafe lalo\n",
      "addr : 201 w. 83rd st.\n",
      "postal : 10024\n",
      "latlng : (40.78598119999999, -73.97672659999999)\n",
      "\n",
      "1/10 positive, 6/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "y\n",
      "\n",
      "INFO:dedupe.training:Final predicate set:\n",
      "INFO:dedupe.training:(SimplePredicate: (fingerprint, addr), SimplePredicate: (wholeFieldPredicate, name))\n",
      "name : hi life restaurant and lounge\n",
      "addr : 1340 1st ave. at 72nd st.\n",
      "postal : 10021\n",
      "latlng : (40.7675807, -73.95580029999999)\n",
      "\n",
      "name : trattoria dell arte\n",
      "addr : 900 7th ave.  between 56th and 57th sts.\n",
      "postal : 10106\n",
      "latlng : (40.7654454, -73.9805358)\n",
      "\n",
      "2/10 positive, 6/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "\n",
      "name : lattanzi ristorante\n",
      "addr : 361 w. 46th st.\n",
      "postal : 10036\n",
      "latlng : (40.7608494, -73.990016)\n",
      "\n",
      "name : pomaire\n",
      "addr : 371 w. 46th st.  off 9th ave.\n",
      "postal : 10036\n",
      "latlng : (40.7609632, -73.9902681)\n",
      "\n",
      "2/10 positive, 7/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "\n",
      "name : empire korea\n",
      "addr : 6 e. 32nd st.\n",
      "postal : 10016\n",
      "latlng : (40.7465392, -73.9849772)\n",
      "\n",
      "name : tu lan\n",
      "addr : 8 sixth st.\n",
      "postal : 94103\n",
      "latlng : (37.7818759, -122.41013)\n",
      "\n",
      "2/10 positive, 8/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "\n",
      "name : aquavit\n",
      "addr : 13 w. 54th st.\n",
      "postal : 10019\n",
      "latlng : (40.7616767, -73.976345)\n",
      "\n",
      "name : aquavit\n",
      "addr : 13 w. 54th st.\n",
      "postal : 10019\n",
      "latlng : (40.7616767, -73.976345)\n",
      "\n",
      "2/10 positive, 9/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "y\n",
      "\n",
      "name : mesa grill\n",
      "addr : 102 5th ave.  between 15th and 16th sts.\n",
      "postal : 10011\n",
      "latlng : (40.7370445, -73.9931189)\n",
      "\n",
      "name : fiorello s roman cafe\n",
      "addr : 1900 broadway  between 63rd and 64th sts.\n",
      "postal : 10023\n",
      "latlng : (40.7715867, -73.98138519999999)\n",
      "\n",
      "3/10 positive, 9/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "\n",
      "name : second avenue deli\n",
      "addr : 156 2nd ave. at 10th st.\n",
      "postal : 10003\n",
      "latlng : (40.7296096, -73.9867012)\n",
      "\n",
      "name : second avenue deli\n",
      "addr : 156 second ave.\n",
      "postal : 10003\n",
      "latlng : (40.7296096, -73.9867012)\n",
      "\n",
      "3/10 positive, 10/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "y\n",
      "\n",
      "name : second street grill\n",
      "addr : 200 e. fremont st.\n",
      "postal : 89101\n",
      "latlng : (36.1709271, -115.1431516)\n",
      "\n",
      "name : paty s\n",
      "addr : 10001 riverside dr.\n",
      "postal : 91602\n",
      "latlng : (34.1524376, -118.3496191)\n",
      "\n",
      "4/10 positive, 10/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "\n",
      "INFO:dedupe.training:Final predicate set:\n",
      "INFO:dedupe.training:(SimplePredicate: (firstIntegerPredicate, addr), SimplePredicate: (wholeFieldPredicate, name))\n",
      "name : cafe ritz carlton buckhead\n",
      "addr : 3434 peachtree rd.\n",
      "postal : 30326\n",
      "latlng : (33.8508073, -84.364227)\n",
      "\n",
      "name : ritz carlton cafe buckhead \n",
      "addr : 3434 peachtree rd. ne\n",
      "postal : 30326\n",
      "latlng : (33.8508073, -84.364227)\n",
      "\n",
      "4/10 positive, 11/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "y\n",
      "\n",
      "INFO:dedupe.training:Final predicate set:\n",
      "INFO:dedupe.training:(SimplePredicate: (fingerprint, name), SimplePredicate: (firstIntegerPredicate, addr))\n",
      "name : le colonial\n",
      "addr : 149 e. 57th st.\n",
      "postal : 10022\n",
      "latlng : (40.7608569, -73.9683494)\n",
      "\n",
      "name : cassell s\n",
      "addr : 3266 w. sixth st.\n",
      "postal : 90020\n",
      "latlng : (34.0634809, -118.2936285)\n",
      "\n",
      "5/10 positive, 11/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "\n",
      "name : cafe ritz carlton buckhead\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "#     deduper.sample(data_for_dedupe)  \n",
    "#     dedupe.consoleLabel(deduper)\n",
    "#     deduper.train(index_predicates=False)\n",
    "\n",
    "with open('training-simple-input-output.txt') as t:\n",
    "    print('\\n'.join(t.read().split('\\n')[:262]))\n",
    "    print('...')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "You can check the full training at `training-simple-input-output.txt`.\n",
    "\n",
    "If you want to train it yourself, do a `rm dedupe-simple-settings.pickle dedupe-simple-training.json` and run this whole Active Learning session again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from dedupe.convenience import unique\n",
    "\n",
    "def auto_label(deduper):\n",
    "    # based on: dedupe.convenience.consoleLabel\n",
    "    fields = unique(field.field\n",
    "                    for field\n",
    "                    in deduper.data_model.primary_fields)\n",
    "    n_match = 0\n",
    "    n_distinct = 0\n",
    "    buffer_len = 1\n",
    "    examples_buffer = []\n",
    "    uncertain_pairs = []\n",
    "    \n",
    "    while n_match < 12 or n_distinct < 20:\n",
    "        if not uncertain_pairs:\n",
    "            uncertain_pairs = deduper.uncertainPairs()\n",
    "        record_pair = uncertain_pairs.pop()\n",
    "        \n",
    "        n_match = (len(deduper.training_pairs['match']) +\n",
    "                   sum(label == 'match' for _, label in examples_buffer))\n",
    "        n_distinct = (len(deduper.training_pairs['distinct']) +\n",
    "                      sum(label == 'distinct' for _, label in examples_buffer))\n",
    "        \n",
    "        for pair in record_pair:\n",
    "            for field in fields:\n",
    "                line = \"%s : %s\" % (field, pair[field])\n",
    "                print(line)\n",
    "            print()\n",
    "        print(f\"{n_match}/10 positive, {n_distinct}/10 negative\")\n",
    "        print(\"Do these records refer to the same thing?\")\n",
    "        print(\"(y)es / (n)o / (u)nsure / (f)inished\")\n",
    "        \n",
    "        if record_pair[0]['cluster'] == record_pair[1]['cluster']:\n",
    "            print('y')\n",
    "            examples_buffer.insert(0, (record_pair, 'match'))\n",
    "        else:\n",
    "            print('n')\n",
    "            examples_buffer.insert(0, (record_pair, 'distinct'))\n",
    "        print()\n",
    "\n",
    "        if len(examples_buffer) > buffer_len:\n",
    "            record_pair, label = examples_buffer.pop()\n",
    "            if label in ['distinct', 'match']:\n",
    "                examples = {'distinct': [], 'match': []}\n",
    "                examples[label].append(record_pair)\n",
    "                deduper.markPairs(examples)\n",
    "    \n",
    "    for record_pair, label in examples_buffer:\n",
    "        if label in ['distinct', 'match']:\n",
    "            examples = {'distinct': [], 'match': []}\n",
    "            examples[label].append(record_pair)\n",
    "            deduper.markPairs(examples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "if not isinstance(deduper, dedupe.StaticDedupe):\n",
    "    deduper.sample(data_for_dedupe)\n",
    "    \n",
    "    training_filename = 'dedupe-simple-training.json'\n",
    "    if os.path.exists(training_filename):\n",
    "        with open(training_filename) as tf:\n",
    "            deduper.readTraining(tf)\n",
    "\n",
    "#     dedupe.consoleLabel(deduper)\n",
    "    auto_label(deduper)\n",
    "    \n",
    "    with open(training_filename, 'w') as tf:\n",
    "        deduper.writeTraining(tf)\n",
    "    \n",
    "    deduper.train(index_predicates=False)\n",
    "    \n",
    "    with open(settings_filename, 'wb') as sf:\n",
    "        deduper.writeSettings(sf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "After training, we can see which **blocking predicates** (indexing rules) the deduper learned from our training input. It's good to do that to check if we trained enough:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((SimplePredicate: (firstIntegerPredicate, addr),\n",
       "  SimplePredicate: (sameFiveCharStartPredicate, name)),\n",
       " (SimplePredicate: (latLongGridPredicate, latlng),\n",
       "  SimplePredicate: (oneGramFingerprint, name)))"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deduper.predicates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The deduper selected those predicates from this extense list of **possible predicates**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{LevenshteinCanopyPredicate: (1, addr),\n",
       " LevenshteinCanopyPredicate: (1, name),\n",
       " LevenshteinCanopyPredicate: (1, postal),\n",
       " LevenshteinCanopyPredicate: (2, addr),\n",
       " LevenshteinCanopyPredicate: (2, name),\n",
       " LevenshteinCanopyPredicate: (2, postal),\n",
       " LevenshteinCanopyPredicate: (3, addr),\n",
       " LevenshteinCanopyPredicate: (3, name),\n",
       " LevenshteinCanopyPredicate: (3, postal),\n",
       " LevenshteinCanopyPredicate: (4, addr),\n",
       " LevenshteinCanopyPredicate: (4, name),\n",
       " LevenshteinCanopyPredicate: (4, postal),\n",
       " SimplePredicate: (alphaNumericPredicate, addr),\n",
       " SimplePredicate: (alphaNumericPredicate, name),\n",
       " SimplePredicate: (alphaNumericPredicate, postal),\n",
       " SimplePredicate: (commonFourGram, addr),\n",
       " SimplePredicate: (commonFourGram, name),\n",
       " SimplePredicate: (commonFourGram, postal),\n",
       " SimplePredicate: (commonIntegerPredicate, addr),\n",
       " SimplePredicate: (commonIntegerPredicate, name),\n",
       " SimplePredicate: (commonIntegerPredicate, postal),\n",
       " SimplePredicate: (commonSixGram, addr),\n",
       " SimplePredicate: (commonSixGram, name),\n",
       " SimplePredicate: (commonSixGram, postal),\n",
       " SimplePredicate: (commonThreeTokens, addr),\n",
       " SimplePredicate: (commonThreeTokens, name),\n",
       " SimplePredicate: (commonThreeTokens, postal),\n",
       " SimplePredicate: (commonTwoTokens, addr),\n",
       " SimplePredicate: (commonTwoTokens, name),\n",
       " SimplePredicate: (commonTwoTokens, postal),\n",
       " SimplePredicate: (doubleMetaphone, addr),\n",
       " SimplePredicate: (doubleMetaphone, name),\n",
       " SimplePredicate: (doubleMetaphone, postal),\n",
       " SimplePredicate: (fingerprint, addr),\n",
       " SimplePredicate: (fingerprint, name),\n",
       " SimplePredicate: (fingerprint, postal),\n",
       " SimplePredicate: (firstIntegerPredicate, addr),\n",
       " SimplePredicate: (firstIntegerPredicate, name),\n",
       " SimplePredicate: (firstIntegerPredicate, postal),\n",
       " SimplePredicate: (firstTokenPredicate, addr),\n",
       " SimplePredicate: (firstTokenPredicate, name),\n",
       " SimplePredicate: (firstTokenPredicate, postal),\n",
       " SimplePredicate: (hundredIntegerPredicate, addr),\n",
       " SimplePredicate: (hundredIntegerPredicate, name),\n",
       " SimplePredicate: (hundredIntegerPredicate, postal),\n",
       " SimplePredicate: (hundredIntegersOddPredicate, addr),\n",
       " SimplePredicate: (hundredIntegersOddPredicate, name),\n",
       " SimplePredicate: (hundredIntegersOddPredicate, postal),\n",
       " SimplePredicate: (latLongGridPredicate, latlng),\n",
       " SimplePredicate: (metaphoneToken, addr),\n",
       " SimplePredicate: (metaphoneToken, name),\n",
       " SimplePredicate: (metaphoneToken, postal),\n",
       " SimplePredicate: (nearIntegersPredicate, addr),\n",
       " SimplePredicate: (nearIntegersPredicate, name),\n",
       " SimplePredicate: (nearIntegersPredicate, postal),\n",
       " SimplePredicate: (oneGramFingerprint, addr),\n",
       " SimplePredicate: (oneGramFingerprint, name),\n",
       " SimplePredicate: (oneGramFingerprint, postal),\n",
       " SimplePredicate: (sameFiveCharStartPredicate, addr),\n",
       " SimplePredicate: (sameFiveCharStartPredicate, name),\n",
       " SimplePredicate: (sameFiveCharStartPredicate, postal),\n",
       " SimplePredicate: (sameSevenCharStartPredicate, addr),\n",
       " SimplePredicate: (sameSevenCharStartPredicate, name),\n",
       " SimplePredicate: (sameSevenCharStartPredicate, postal),\n",
       " SimplePredicate: (sameThreeCharStartPredicate, addr),\n",
       " SimplePredicate: (sameThreeCharStartPredicate, name),\n",
       " SimplePredicate: (sameThreeCharStartPredicate, postal),\n",
       " SimplePredicate: (sortedAcronym, addr),\n",
       " SimplePredicate: (sortedAcronym, name),\n",
       " SimplePredicate: (sortedAcronym, postal),\n",
       " SimplePredicate: (suffixArray, addr),\n",
       " SimplePredicate: (suffixArray, name),\n",
       " SimplePredicate: (suffixArray, postal),\n",
       " SimplePredicate: (tokenFieldPredicate, addr),\n",
       " SimplePredicate: (tokenFieldPredicate, name),\n",
       " SimplePredicate: (tokenFieldPredicate, postal),\n",
       " SimplePredicate: (twoGramFingerprint, addr),\n",
       " SimplePredicate: (twoGramFingerprint, name),\n",
       " SimplePredicate: (twoGramFingerprint, postal),\n",
       " SimplePredicate: (wholeFieldPredicate, addr),\n",
       " SimplePredicate: (wholeFieldPredicate, name),\n",
       " SimplePredicate: (wholeFieldPredicate, postal),\n",
       " TfidfNGramCanopyPredicate: (0.2, addr),\n",
       " TfidfNGramCanopyPredicate: (0.2, name),\n",
       " TfidfNGramCanopyPredicate: (0.2, postal),\n",
       " TfidfNGramCanopyPredicate: (0.4, addr),\n",
       " TfidfNGramCanopyPredicate: (0.4, name),\n",
       " TfidfNGramCanopyPredicate: (0.4, postal),\n",
       " TfidfNGramCanopyPredicate: (0.6, addr),\n",
       " TfidfNGramCanopyPredicate: (0.6, name),\n",
       " TfidfNGramCanopyPredicate: (0.6, postal),\n",
       " TfidfNGramCanopyPredicate: (0.8, addr),\n",
       " TfidfNGramCanopyPredicate: (0.8, name),\n",
       " TfidfNGramCanopyPredicate: (0.8, postal)}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deduper.data_model.predicates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To proceed with the deduplication, we compute the **clustering threshold** and call the actual **match**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:dedupe.api:Maximum expected recall and precision\n",
      "INFO:dedupe.api:recall: 0.999\n",
      "INFO:dedupe.api:precision: 0.705\n",
      "INFO:dedupe.api:With threshold: 0.209\n"
     ]
    }
   ],
   "source": [
    "import itertools\n",
    "\n",
    "threshold = deduper.threshold(data_for_dedupe, recall_weight=2)\n",
    "clustered_dupes = deduper.match(data_for_dedupe, threshold)\n",
    "\n",
    "dedupe_found_pairs_set = set()\n",
    "for cluster, __ in clustered_dupes:  # we'll explain that later\n",
    "    for pair in itertools.combinations(cluster, 2):\n",
    "        dedupe_found_pairs_set.add(tuple(pair))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we'll **evaluate** how it performed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "true_positives total: 128\n",
      "false_positives total: 2\n",
      "false_negatives total: 22\n",
      "\n",
      "svm_true_positives total: 133\n",
      "svm_false_positives total: 2\n",
      "svm_false_negatives total: 17\n",
      "\n",
      "dedupe_true_positives total: 141\n",
      "dedupe_false_positives total: 3\n",
      "dedupe_false_negatives total: 9\n"
     ]
    }
   ],
   "source": [
    "dedupe_true_positives = golden_pairs_set & dedupe_found_pairs_set\n",
    "dedupe_false_positives = dedupe_found_pairs_set - golden_pairs_set\n",
    "dedupe_false_negatives = golden_pairs_set - dedupe_found_pairs_set\n",
    "\n",
    "print('true_positives total:', len(true_positives))\n",
    "print('false_positives total:', len(false_positives))\n",
    "print('false_negatives total:', len(false_negatives))\n",
    "print()\n",
    "print('svm_true_positives total:', len(svm_true_positives))\n",
    "print('svm_false_positives total:', len(svm_false_positives))\n",
    "print('svm_false_negatives total:', len(svm_false_negatives))\n",
    "print()\n",
    "print('dedupe_true_positives total:', len(dedupe_true_positives))\n",
    "print('dedupe_false_positives total:', len(dedupe_false_positives))\n",
    "print('dedupe_false_negatives total:', len(dedupe_false_negatives))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4/4 - Clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Using the Threshold or the SVM, we got the **matching pairs**. But what Dedupe returned to us were **clusters of matches**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[((0, 1, 2), array([0.71874997, 0.71926606, 0.68507835])),\n",
       " ((3, 4, 5), array([0.70119166, 0.72756007, 0.72756007])),\n",
       " ((6, 7, 8), array([0.23452896, 0.58216786, 0.34763891])),\n",
       " ((9, 10, 11, 12), array([0.75352858, 0.75352858, 0.75352858, 0.75272876])),\n",
       " ((13, 14), (0.7539285, 0.7539285))]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clustered_dupes[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Dedupe went one step further on the process and **merged the matching pairs into clusters**! Why is that important? Because the following can happen:\n",
    "- We have the records `A`, `B`, and `C`\n",
    "- By deduplicating, we find:\n",
    "- `(A, B)` match\n",
    "- `(B, C)` match\n",
    "- `(A, C)` nonmatch\n",
    "- And that doesn't make sense!\n",
    "\n",
    "The solution for that ambiguity is computing the **Transitive Closure** through clustering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Using some private methods from Dedupe, it's possible to get the **unclustered pairs**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from dedupe.core import scoreDuplicates\n",
    "\n",
    "candidate_records = itertools.chain.from_iterable(deduper._blockedPairs(deduper._blockData(data_for_dedupe)))\n",
    "dedupe_matches = scoreDuplicates(candidate_records,\n",
    "                                 deduper.data_model,\n",
    "                                 deduper.classifier,\n",
    "                                 deduper.num_cores)\n",
    "dedupe_unclustered_pairs_score_dict = {tuple(pair): score for (pair, score) in dedupe_matches}\n",
    "dedupe_unclustered_found_pairs_set = set(dedupe_unclustered_pairs_score_dict.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's **evaluate** those unclustered pairs against the clustered pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dedupe_true_positives total: 141\n",
      "dedupe_false_positives total: 3\n",
      "dedupe_false_negatives total: 9\n",
      "\n",
      "dedupe_unclustered_true_positives total: 139\n",
      "dedupe_unclustered_false_positives total: 5\n",
      "dedupe_unclustered_false_negatives total: 11\n"
     ]
    }
   ],
   "source": [
    "dedupe_unclustered_true_positives = golden_pairs_set & dedupe_unclustered_found_pairs_set\n",
    "dedupe_unclustered_false_positives = dedupe_unclustered_found_pairs_set - golden_pairs_set\n",
    "dedupe_unclustered_false_negatives = golden_pairs_set - dedupe_unclustered_found_pairs_set\n",
    "\n",
    "print('dedupe_true_positives total:', len(dedupe_true_positives))\n",
    "print('dedupe_false_positives total:', len(dedupe_false_positives))\n",
    "print('dedupe_false_negatives total:', len(dedupe_false_negatives))\n",
    "print()\n",
    "print('dedupe_unclustered_true_positives total:', len(dedupe_unclustered_true_positives))\n",
    "print('dedupe_unclustered_false_positives total:', len(dedupe_unclustered_false_positives))\n",
    "print('dedupe_unclustered_false_negatives total:', len(dedupe_unclustered_false_negatives))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Unclustered pairs `!=` Clustered pairs.\n",
    "\n",
    "Though necessary, the clustering process can both **create new matches** and **drop found matches**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{(6, 8), (40, 41), (120, 471), (121, 471), (196, 198), (199, 839)}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "diff_set = dedupe_found_pairs_set ^ dedupe_unclustered_found_pairs_set\n",
    "display(diff_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>the palm</td>\n",
       "      <td>9001 santa monica blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>90069</td>\n",
       "      <td>34.083064</td>\n",
       "      <td>-118.387282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>palm the los angeles</td>\n",
       "      <td>9001 santa monica blvd.</td>\n",
       "      <td>w. hollywood</td>\n",
       "      <td>90069</td>\n",
       "      <td>34.083064</td>\n",
       "      <td>-118.387282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>the palm los angeles</td>\n",
       "      <td>9001 sta monica boulevard</td>\n",
       "      <td>hollywood</td>\n",
       "      <td>90069</td>\n",
       "      <td>34.083064</td>\n",
       "      <td>-118.387282</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     name                       addr          city postal  \\\n",
       "40               the palm    9001 santa monica blvd.   los angeles  90069   \n",
       "41  palm the los angeles     9001 santa monica blvd.  w. hollywood  90069   \n",
       "42   the palm los angeles  9001 sta monica boulevard     hollywood  90069   \n",
       "\n",
       "          lat         lng  \n",
       "40  34.083064 -118.387282  \n",
       "41  34.083064 -118.387282  \n",
       "42  34.083064 -118.387282  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlsAAAFBCAYAAABAYGoRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmYXVWd7//3JyMQ5klAhsggQxC9aCMiKo6ogAP6U9uJdoRHu39OtPdqO6CtojZ221zsFqUF5xlHZBBkUgRBxkASAiQhEEiAkAAhJCSs+8faxTmUlaQqqVOnhvfrec6TOnvvs8/3nArm49prr29KKUiSJKkzxnW7AEmSpNHMsCVJktRBhi1JkqQOMmxJkiR1kGFLkiSpgwxbkiRJHWTYkjSsJJmbpPTjMXeQ33dckhOax9v72P+5tvc+dDDfW9LoNqHbBUjSMDEO+HTz8wXAd7pYi6RRxLAlaVgppUxtf56ktO1Lf86RZKNSyiODXJokrRcvI0oasZJ8r+3S3nOTnJlkKXBDs/+Pzb5VvV73hO1J3g082nbIi9vOe34fb71pkpOTLEyyNMm5Sfbs1OeUNLI5siVptPgVsE3z8z0dfq8zgCe1PX8Z8KskB5RSVnf4vSWNMI5sSRotlgAHA5sArxrIC0sppwET2zZdUEpJ83hJHy95GHg68GRgVrNtP+DAAVctadQzbEkaLT5eSrmilLK8lHJTh9/ry6WU60spC4Bz2rbv1uH3lTQCGbYkjRbXDODYDZ1CMavt52VtP2+0geeVNAoZtiSNFsv72Lai+XN8kglQ19MCpvZxbOlj25q0T6YfyOskjUGGLUmj2by2n49q/vwgT5zcDkAzsX1p83Rqki07XJukMcKwJWk0+37bz2cmeRD4Cn2PggFc3vy5B3B/s/TDJzpZoKTRz7AladQqpVwAHAvMpl5SnA0cDVy9hpe8nzrh/f4hKVDSmJBSnG4gSZLUKY5sSZIkdZBhS5IkqYMMW5IkSR1k2JIkSeogw5YkSVIHGbYkSZI6yLAlSZLUQYYtSZKkDjJsSZIkdZBhS5IkqYMMW5IkSR1k2JIkSeogw5YkSVIHGbYkSZI6yLAlSZLUQYYtSZKkDjJsSZIkdZBhS5IkqYMMW5IkSR1k2JIkSeogw5YkSVIHGbYkSZI6yLAlSZLUQYYtSZKkDjJsSZIkdZBhS5IkqYMMW5IkSR1k2JIkSeogw5YkSVIHGbYkSZI6yLAlSZLUQYYtSZKkDjJsSZIkdZBhS5IkqYMMW5IkSR1k2JIkSeogw5YkSVIHGbYkSZI6aEK3C5AGRTIZ2BnYCpgErATuB+ZTyspuliZJGtsMWxrZki2BfYFp1L/PK4HVwHhq6FpFMh2YSSlLulanJGnM8jKi1ijJGUl+O0TvNTVJSfKsAbxoN+ANwH7AvcCdwD3A4ubPO4F7A3/+CHyxOX5UG8rfmSSpfwxbw0SSjzVh45Re249Ocm6Se5r9h3WpxEEzKIGgBqcjqMFqEXU0qy+rAR6Ch4AjxkLgkiQNL4atYSDJwcB7gev72D0FuAz48JAWNZzVS4eHU0PWiv685FF4tDn+8Ob1kiQNCcNWlyXZAvg+8E7qhO4nKKV8t5TyGeDsAZzzsGYU7Mgk1yZ5JMlfkzyz7ZhtkvwwyR1Jlie5Mck71nHei5L8d5KvJFncjLZ9IMnkJF9LsiTJ7UnetpZznAAcAxzR1Nh7tG63JL9P8nCSm5K8tNfr99sezpoEJ28Ep+0Jx/8B+hueVgCrT4BXJjm/+dyLm5G2Ldre42lJLkjyQJKHklyX5IVr+UwvT3Jpkvub852bZN+2/T2XSF+3js92RJJZze/rkiRval43te2YQ5Jc3Jzjzub3sflaakuSjya5tfm8NyR5a69jPpVkXpIVSe5O8p1+fp+SpH4wbHXfN4CflVIuXN8TNGFhbh+7TgL+N/As4Dbgt0k2afZtBFwNHEmdXP6fwKlJXryOt3sL8CDwbOCLwFeBXwI3N+/zbeC0JDuu4fUnAT8Bzgd2bB6Xte3/PHAy8HTgSuBHSTZtPueOwCXT4IFT4fiT4JMrYeM3wCdWQNZRNwBz4KF/g/8aB8uAg4DXAocA32o77AfAXc3+ZwAnAI+s5bRTmu/hIOAwYCnwmySTeh23ts+2K3AmcFaz/2Tgy+0vTvI04Dzg180xRzf1tdfe2+eAdwHvp85tO5H6ez6iOefrgOOB9wF7Uf8+/GUt55MkDVQpxUeXHsB7gL8CE5vnFwGnrOHYbYECHNbHvhOBC9qeH9Yc+5a2bZsCS4B3r6WeHwGntT0/A/ht2/OLgD+3PQ91Ivqv27ZNpN4R+Pq1vM8Tzttsm9rUfGzbtic32w5tnn92ClxW4LgCRxU46mp4E1BOhA/3bOv9AMo74MQCR70G/u8kWH42HNDH97Vn8/wB4JgN+L1Ooc4VO3QAn+1EYEav83y8OWZq8/w7wP/0OuYZzTHb9/5umzqWA8/r9ZqvAr9rfv4wMKvn76APHz58+Bj8h0s/dEmSvYEvUP+xfXRDzlVK+dgadv257ZiHktxAHd0gyXjg/wBvpP7DP5m6VMJF63i7x+eVlVJKkkXADW3bHk1yP7D9wD/JE88PLGj+7DnXMx+GgybCgaUGjMdNryNks1fAuBUwbnNY1dfJ58Eu28H8l9dQ2OMy4DHqd3ML8O/U0bljgAuAn5dSZq6p4CR7AP9KHe3bjjpiPA7YdQCfbR/qaFe7K3o9fyawZ5I3tr998+ce1Dlp7fajjmCek6T9+5oIzG1+/inwAWBOknOBc6jhuV9z4SRJ62bY6p7nUEerbkwevwI2Hnh+kuOAKR3+B+944CPUf2hvoN6t9wXWHZJ6B8Oyhm3re4n68XM1YY62c43bHi47FX6zrF6qe9wz66gdN8BWp8Ob9oLr9oV5z22FmvbiCjVc9rELSiknJPk+8ArqRPxPJzmulLKmy3W/Be4AjqUuN7EKuIkaXvv72fpjHHAa8B997LtzDcdDHd27va9aSinzm+D/YuAlwFeon/fZpZRlA6hNkrQGhq3u+SVwVa9tpwOzqaFnMFY9P5g6V4skU4D9qZeiAA4FflNK+W6zP8BTaUJLh62kBsuBunopvO1FsGQzuLuvA54F9/0Ilk6FBdfBnpfUeVQ8BFuvgHG7wfwb4WV/gfEHtV52CDWYzOjZUEqZTf1dnJzkv4F308fcqCTbUEel3leaeXdJDmTg/23NBF7da9tBvZ5fDUwrpdzSz3PeRL0pYLdSyh/WdFAp5RHqXLGzknyR+t0+lzo/TJK0gQxbXVLqauZPCDZJlgGLSynT27ZtTb0c1XPH3Z5JlgB3l1Lubo45ETiolNJ7cvsnktxDHd35FDXk/KDZdzPwxiSHUhcE/SfgKcA1g/cp12gu8IpmROU+eo1SrcXXHoXjDoL3vQ2+twcsvRJ2OBsOPRO+tXedn8Q4WP2aGjJvWwAbfQGOeQB2/iy89jlw9+9h1cvgk0uTj1Db+5wKnFlKuSXJxtRJ/D9t6nwSNZj2vqTX437q9/eeJPOpl2T/jTVcxlyLrwMfTnIS8E3qTQvHNvt6LgF+Cbg8ydebmh+kBr2jSinH9jofpZQHm/Od1ITpS6hz9w4GHiulfCPJP1D/d+AK6ujmG6mjXrMHWL8kaQ28G3H4exU1APXcrfjN5vlxbcfsSJ2z09v/oV4WuprmTrO2S0Ofo951djb1H+Fl1CUohsI3qaNIV1En2D+3Py8qpSyYCs+jLt/wmTfD174Bx02AR7f920uZAOzU3EW4E1z/Xjh7d1j8PrjwAdi9ef/fUOe2vbN5yWpqADuDOnH8F83+Ptc5K6U8Rg0oBwDTga8Bn6Sf63+1nWce8Drq7/s64EPAZ5rdjzTHXA88nzrh/uLmuBOBhWs59Sepd1MeD9wI/L55nznN/iXUuxUvbep/HXB0KWXO35xJkrReUkpZ91EaUZp1qy4Etiul3NvlcgZf8hzq5O/eE8IB+Cj8/Zfhh2t49fbA9NSRm2nUS6sbUy+5TQfuLMPkP4okHwA+C2w5XGqSJA2clxE1Es0Ankad5D6QEaTJ1Llis5rLuJcClybZjhq8XgNMSG1cPR1YOJQhJ8n7qXck3kO91PdJ4AyDliSNbIYtjTylLKEuU3AE/W/ZM5k6qnUWNWi1na7cA1yU5GLqHK39qet3rWqC143NMZ22J3VtrW2odzd+nTqyJUkawbyMqJGrNpU+nDrP6r7mz96XEcdTw8t44Fzq3Kh+nDoBdqIGr/2Bh6mjXTeWUhYP5seQJI1ujmxp5CplHslPgL2plxUnAI9uV+dgbU9dvHMVdR2xWb1HtNZ+6lKoa1fdmeQ86h2h+wPvau4G7QleDwzmR5IkjT6ObGl0qH0IdwG2/Ad43xnwX9Q77eZTymCsWda8TcZRl8iYBuxLnV81HbiplPLQYL2PJGn0MGxp1ElyQinlhCF4n/HUJTf2py4Iu4AavGaUUpZ3+v0lSSODlxGl9VRKWU1dHPbmJBOpa5lNAw5Pcjs1eM20z6AkjW2GLWkQNM3EbwJuSjKZOtK1P/DKJLdRg9fsMoiXNCVJI4NhSxpkzUjWDcANTfuffYADgaOS3EINXreUUgba0keSNAIZtqQOauZuXQNc0zQD35e6YOmrk8yiBq85zSVJSdIoZNiShkjTl/Iq4Kokm1Hndx0GHJ1kBjV4zWv6LUqSRgnDltQFpZQHgcuBy5NsSQ1eLwM2S3IjNXjdYaseSRr5DFtSlzV9Gv8E/CnJNtSJ9a8CJrUFr7sMXpI0Mhm2pGGklHIfcHGSS6ir4E8DXg/Q0yC7lLKoiyVKkgbIsCUNQ80o1kJgYZILgR2pI15vSbICuJEavO7rYpmSpH4wbEnDXBO8FgALkvye2pZoGvCOJA/S6tPY796PkqShY9iSRpAmeN0O3J7kXGA36ojXe5PcR6tP44NdLFOS1MawJY1QzRIRc4A5SX4H7E4d8XphkrtpBa+Hu1imJI15hi1pFGgWRZ0NzE4yAdiTOuL1kiR30OrT+EgXy5SkMcmwJY0yTRugmcDMJJOofRqnAa9IMpcavGbZp1GShoZhSxrFmkA1HZieZCNgb+AA4MhefRof7WKZkjSqGbakMaK5hHgdcF2STah9Gg+i9mm8mRq8brVPoyQNLsOWNAY1k+b/Cvw1yabAfsChwGuSzKQGr7n2aZSkDWfYksa4UspDwF+AvyTZgjq/6yXAFkluogav220XJEnrx7Al6XGllKXAZcBlSbamBq9XAhu39WlcYPCSpP4zbEnqUyllMXApcGmS7ahLSRwNjGsLXgsNXpK0doYtSetUSrkHuDDJRcAO1BGvNwGr2hpk39vFEiVp2DJsSeq3ZhTrLuCuJBcAT6aOeB2TZBmtPo33d7FMSRpWDFuS1ksTvO4A7khyHrArdcTr3UmW0ApeD3SxTEnqOsOWpA3WLBExF5ib5GzgKdQRr+cnWUSrT+Oy7lUpSd1h2JI0qJrgdStwa5KzgD2oI14vTrKAGrxmlFKWd7FMSRoyhi1JHdP0aZwFzEoyEdiLOuJ1eJJ5tPo0ruhimZLUUYYtSUOi6b94E3BTksnUPo37A0ckuY0avG62T6Ok0cawJWnINSNZ1wPXJ9kY2Ac4EDgqyWxafRpXdbFMSRoUhi1JXdXM3boGuCbJFGqfxkOofRpnUYPXHBtkSxqpDFuSho3mbsUrgSuTbE4NXocBr00yg1afRhtkSxoxDFuShqVmfa7LgcuTbEW9o/HlwJS2Btl32C5I0nBn2JI07DUr0v8R+GOSbanB69XAhLY+jXcbvCQNR4YtSSNK04Px4iSXANtT72h8A/BYT/AqpSzqZo2S1M6wJWlEakaxFgILk/wB2JEavN6a5BFa7YLu62KZkmTYkjTyNcFrAbAgye+BXajB6x1JHqAVvJZ2sUxJY5RhS9Ko0gSv24Hbk5wD7EYNXscmuY9Wn8YHu1impDHEsCVp1GqWiJgDzEnyO2B3avB6YZK7aPVpfLiLZUoa5QxbksaEZlHU2cDsJBOofRqnAS9LMp8avGaWUh7pYpmSRiHDlqQxp2kDNAOYkWQS8FTqiNcrksyh1adxZRfLlDRKGLYkjWlNoJoOTE+yEbVP49OpfRpvafbNtk+jpPVl2JKkRnMJ8Vrg2iSbAPsCBwGvbuvTeJt9GiUNhGFLkvrQTJr/K/DXJJtR+zQ+n9qncSY1eM21T6OkdTFsSdI6NMtEXAFckWQL6sT6lwCbt/VpnG+7IEl9MWxJ0gA0C6NeBlyWZGvqxPojgY3a+jQuMHhJ6mHYkqT1VEpZDFwCXJJke+qI19HAuCTTqcFrkcFLGtsMW5I0CJrm14uSXATsQB3xejOwsq1B9r1dLFFSlxi2JGkQNaNYdwF3JTkf2Jk64nVMkmW0+jTe38UyJQ0hw5YkdUgTvOYD85OcB+xKHfF6T5LFtPo0PtDFMiV1mGFLkoZAs0TEXGBukrOBp1BHvF6QZBGt4LWse1VK6gTDliQNsWZR1FuAW5KcBexBHfF6cZI7afVpXN7FMiUNEsOWJHVR0wZoFjAryURqn8ZpwMuTzKMGr1mllBVdLFPSBjBsSdIwUUp5FLgRuDHJZGBv6ojXEUlupdWn8dEulilpgAxbkjQMNSNZ1wPXJ9mY2qfxmcCrksymBq9bbZAtDX+GLUka5pq5W1cDVyfZlBq8DgFe09ancY59GqXhybAlSSNIKeUh4ErgyiSbU+d3vQjYMskMavC63eAlDR+GLUkaoZr1uf4M/DnJVjQT64EpbX0a77RdkNRdhi1JGgWaFen/CPwxybbUifWvASa0Ba+7DV7S0DNsSdIo0/RgvCjJxcD21OD1BuCxngbZpZR7ulmjNJYYtiRplGpGsRYCC5P8AdiJGrzelmQ5dZmJ6aWUxV0sUxr1DFuSNAY0wetO4M62Po3TgHclWUqrQfbSLpYpjUqGLUkaY5rgNQ+Yl+QcYCp1xOu4JPfQ6tP4UPeqlEYPw5YkjWHNEhG3Abe19WmcBrwoyV3U4DWjlPJwF8uURjTDliQJeLxB9s3AzU2fxj2pI14vS3I7rT6Nj3SxTGnEMWxJkv5G039xBjAjySRqn8ZpwCuTzKEGr5tLKSu7WKY0Ihi2JElr1QSqG4AbkmwE7AM8AziqrU/jLfZplPpm2JIk9VtzCfFa4NokmwD7AQcDr05yMzV43dZckpSEYUuStJ6aSfNXAVcl2YwavJ4PvLatT+M8+zRqrDNsSZI2WCnlQeAK4IokW1Lnd70M2CzJTdTgNd92QRqLDFuSpEFVSlkC/An4U5JtqMHrSGCjnnZBwF0GL40Vhi1JUseUUu4DLgEuSdLTp/H1AG0NshcZvDSaGbYkSUOilLII+EOSC4EdqMHrzcDKtgbZ93WzRqkTDFuSpCHVjGLdBdyV5HxgZ2rwekeSB2k1yF7SxTKlQWPYkiR1TRO85gPzk5xLbZC9P/DeJItp9Wl8oItlShvEsCVJGhaaJSLmAnOTnA08hRq8XpBkIa0+jcu6V6U0cIYtSdKw0yyKegtwS5IJ1D6N04CXJLmTVvAanD6NyWTq5cytgEnASuB+YD62JNIGMmxJkoa1pg3QTGBm06dxL+qI18uTzKXVp3HFgE9e1wTblxrkJlBD1mpgPDV0raJO3p+Jc8i0ngxbkqQRo+nTeCNwY+po1D7A04Ajk9xKDV6zm0baNGHslFLKSX9zsmQ34HBquLq3+bO38dQgdgDJuZQyb7A+S5LDgAuB7Uop9w7WeTX8jOt2AZKksSnJx5KUJKf02p4kJyRZkGR5kouSTOv9+lLKilLKdaWUH1DD0hnAs4CPJDk6yVPX8ua7AUcAi4FF9B20aLYvao47onmdNCCGLUnSkEtyMPBe4Po+dn8U+AjwT8DfUcPO75v+i2vyKPBYKeU7wCnAHcBzgc2BaUn2SDKuefMtqSNai4D+Xnpc0Rx/ePN6qd8MW5KkIZVkC+D7wDupk9Db9wX4IPDFUsrPSynTgWOAzagLoPZ1vsOA04EpSQrwIPDKUsrpzc8rgW8BDye593/BVx+rI1YrAKbDJtPg/RvBdyfCj58EX/i3OiG/txXU1+2dZG4z+va9JA8luTvJ8b3q+nCS65MsS3JnktOylqCW5B+ac70iycwkDyf5dZItkrw+yewkS5N8N8nG6/iaNYwYtiRJQ+0bwM9KKRf2se8p1NXlz+vZUEpZTm35c0jPtubS4kXN08uoAe1hYMfm0TNHq1DbA/0UOGQS/OBaOObN8JpvwoFXwDYvgk8/ANucAJ/9Nnxwd7jx4/D58+udib3dR50jBvBhYAZwIPBp4AtJjm479rGmrmnUoHgQ8H/X8d1Mpo7qvQV4MfWy6M+pgfN1wGuofSbft47zaBhxgrwkacgkeQ911Oitazhkh+bPhb22LwSe3Pb89p4fSikrkyytP5a7+zjneaWUU5oClm4Fb5oPyyfDyi/C0Ythr/+Az78A7pkGS94M398KDjoRXvgSOLPXuVYDEybAhFVwRSnl8832m5P8HTWAndnU9dW2181N8lHgV0mOadYU68sE4P2llFm13PwA+BDwpJ5J9El+BbwQ+MoazqFhxrAlSRoSSfYGvgAc2nO34Poqpbx9AIe3zwvbalO4fxls/HaY/m3YazWM/xB8jHq1J8Cqx2Di3a3g19uj42HcKvhzr+1/Bh4f2UryIup59wW2oLWcxA7AgjWce0VP0GosBO7udbfiQmC/tX5iDSuGLUnSUHkOsC112YaebeOB5yc5DpgC9IxMPYm20avmeV+jVv3RHuwmAY89VkMVq2HiZHjwnfDz5bDFZHjoGXDV5rBil3pZsi+raV6/Jql3LZ4FfBP4FPXy44HAD5sa1mRVr+elV/0925wGNIIYtiRJQ+WXwFW9tp0OzKaOeK0E5lBD1UuBKwGSbAQ8D/jntZx7JTW4rctKYNxq2OwkeMFm8KQVsPkuMOef4IZN17wERLvx1MBzcK/tB1PncEGdazUJ+FCzGj5JjuzHuTUKGbYkSUOi1BXYn7AKe5JlwOLmrsOebV8FPp5kJnAz8AngIeAHbcd8pzlnz+XEucBGSV4KXAM8XEp5uO34ScBeR9dJ9ls/Bsv2g7nvgD/tC0/+ErzrLjjjeXDHTNjyHHjmi+Daf4Wb+vgoE1fXye8HJ/kY8DPgMODt1IntUAPkOOCDSc6kBrEPDvxb02hg2JIkDTdfBjYGvka9I/AK4GWllAfbjtm1/QWllMuSfJ16mW4b4DNJPgdMpN49+GHgzqvhcuD+yXDXK2EewKXwmTfCW0+DfzwFttgIluwIM6bBH/qobTywalW93PfvwAHAvwDLgE+VUn7W1HN9kg8A/xv4HPWOyeOBH2/416ORJqWUbtcgDaokJ5RSTuh2HZKGXpLx1OUj9gf2pk4m72lavaw56DnUCeaL1uMttgemp4amvtsASb04siVJGtGaleF3owasfamtdaYDF/QaDesxgzraNZn+ryBPc/x4YNa6DpTaGbYkSSNOs9L8ztSANY26Uvx04BvN3LA1K2UJybnU3oj9bdkzmTqqdVbz+g2oXmONYUuSNCI0AWtHWgFrJTVgnV5KuW9AJytlHslZ1B6Jq6lLM/R1J+J46hyw8dSgNa++vExdv0+hsciwJUka1pJsTw1Y+zebplPvTFxUNmTicQ1cP6HO7Xoa9d/ER6mhazx1cv0q4AZgFusaMZPWwLAlSRp2kmxDHb3an3oJ70bqEgt3bVDA6q0GqCtIrgF2AbakNZdrCTCfUlYO2vtpTDJsSZKGhSRb0gpYm1ED1m+B+YMasPpSA9WtHX0PjVmGLUlS1yTZjLoMw/7UuVEzgPOAeWtp1iyNKIYtSdKQSrIJrYC1AzATuAS4rae1jTSaGLYkSR3X9DfchxqwdqG2s7kcuKWU0rv5sjSqGLYkSR3R9CPcmxqwplKbTF8L/KQ46VxjiGFLkjRokkwE9qQGrD2B26lLNfyilPJIN2uTusWwJUnaIE0/wj2oAeupwF3UgHVWKeXhbtYmDQeGLUnSgDX9CKdSA9Y+wL3UgHVeKeWhLpYmDTuGLUlSvzTtcnalBqz9gKXUgHVqKWVpN2uThjPDliRpjZqAtROtfoTLqQHrf0opi7tZmzRSGLYkSU/QBKwn0QpYj1ED1ndLKfd0szZpJDJsSZIASLItrYbPE6gB6yfA3R1vlyONYoYtSRrDkmxFqx/hFGo/wl8CdxqwpMFh2JKkMSbJ5rQC1pbATcA5wO32I5QGn2FLksaAJFNo9SPcntqP8A/AHAOW1FmGLUkapZJsDOxLDVg7ATcDlwG32o9QGjqGLUkaRZJMptWPcDfgVuAqYHYp5dFu1iaNVYYtSRrhmn6ET6UGrN2BedQ7CX9eSlnRzdokGbYkaURKMoFWP8K9gDupAevXpZTl3axN0hMZtiRphGgaPj+FGrD2BhZRA9Y5pZRl3axN0poZtiRpGGsaPvf0I9wXuJ8asP5QSnmgm7VJ6h/DliQNM027nJ2pa2FNA5ZRA9ZppZT7u1mbpIEzbEnSMNAErB1o9SNcBdwAfLuUcm83a5O0YQxbktRFSbaj1Y9wHHUE64fAItvlSKODYUuShliSrWkFrI2oAetMYIEBSxp9DFuSNASSbEGrH+Hm1H6EvwXmG7Ck0c2wJUkdkmRTWgFrG2o/wvOBufYjlMYOw5YkDaIkm9DqR7gjMAu4BLitlLK6m7VJ6g7DliRtoCQbAftQA9YuwC3AX6j9CG34LI1xhi1JWg9JJtHqR/gUYA5wLfCTUsrKbtYmaXgxbElSPzX9CPeiBqw9gDuodxL+spTySDdrkzR8GbYkaS2afoS70+pHeBc1YJ1VSnm4m7VJGhkMW5LUS9OPcCr1TsJ9gfuoAev8UsqDXSxN0ghk2JIkHm+Xswt1BGs/4AFqwDq1lLK0m7VJGtkMW5LGrCZg7URrLaxHqAHrW6WUxd2sTdLoYdiSNKY0AWt7Wu1yCjVgfa+UsqibtUkanQxbksaEJNvSGsGaRA1YPwHutl2OpE4ybEkatZJsSWsEawq1H+GvgTsMWJKGimFL0qiSZHPqBPf9ga2pAesc4Hb7EUrqBsOWpBEvyRRaAWt7aj/Ci4A59iMgfSokAAAORUlEQVSU1G2GLUkjUpKNafUjfDIwG7gMuNV+hJKGE8OWpBEjyWTqKu7TqIuO3gZcDfyolPJoF0uTpDUybEka1pJM5In9COdR7yQ8s5Syopu1SVJ/GLYkDTtNw+c9qCNYTwUWUAPWb0opy7tZmyQNlGFL0rDQ9CN8CnUEax9gETVgnVdKeaibtUnShjBsSeqaZjX33agjWPsBS6gB68JSygPdrE2SBothS9KQagLWk6kjWNOAZcCNwGmllPu7WZskdYJhS1LHNQHrSbRWc19FHcH6dinl3m7WJkmdZtiS1DFJtqMVsMZRR7B+BCy0XY6kscKwJWlQJdmaVsPnjakB60xggQFL0lhk2JK0wZJsQStgbUHtR/g7aj9CA5akMc2wJWm9JNmUVj/C7YAZwPnAXBs+S1KLYUtSvyXZBNiXOoq1E3Az8EdqP0IbPktSHwxbktYqyUbUfoT7A7sCtwBXArfYj1CS1s2wJelvJJlEbZMzDdgdmAtcD/y0lLKyi6VJ0ohj2JIEPN6PcE/qCNaewB3UtbB+VUp5pJu1SdJIZtiSxrAk46kjV9OolwoXUgPW2aWUZd2sTZJGC8OWNMY0DZ93o45g7QvcR10L64JSyoPdrE2SRiPDljQGNO1ydqGOYE0DHqSOYH2jlLKkm7VJ0mhn2JJGqSZg7Uir4fNKasA6vZRyXzdrk6SxxLAljTJJtqfVjxBqwPp+KWVR96qSpLHLsCWNAkm2oRWwJlHnYP0MuMt2OZLUXYYtaYRKsiWtfoSbUQPWr4E7DFiSNHwYtqQRJMlmtCa5b0PtR3geMM9+hJI0PBm2pGEuyRTqEg37AzsAM4GLgTn2I5Sk4c+wJQ1DTT/CnobPO1P7EV5O7Ue4qpu1SZIGxrAlDRNNP8Kehs9TgduAa4Cf2I9QkkYuw5bURUkmAntRR7D2BG6nLtXwC/sRStLoYNiShljTj3AP6gjWU4EF1DsJzyqlPNzN2iRJg8+wJQ2Bph/hVFr9CO+hjmCdV0p5qIulSZI6zLAldUjTLmdXasDaD1hCHcH6eillaTdrkyQNHcOWNIiagLUTrX6Ey6kjWP9TSlnczdokSd1h2JI2UBOwnkQrYD1GDVjfLaXc083aJEndZ9iS1lOSbWn1I5xADVg/BhbaLkeS1MOwJQ1Akq1ojWBNoc7B+iVwpwFLktQXw5ZGh2QydaX1rY6BXUieBdwPzGcDFwRNsjmths9bAjcBZwO3G7AkSeti2NLIlmxJq63NBGDlNNgaOACYBKwimQ7MpJQl/T9tNqXeQTgN2J7aj/AP1H6ENnyWJPWbYWsMSnIRML2U8o/drqW3JFOBOcDflVKuWsfBuwGHA6uBe5s/uafeAdgzMX08NTAdkORU4P8rpfxsDe+9Ma2GzzsBNwN/Am4daQ2fk5wBbFtKObLbtUjSWDeu2wXoiZJ8LElJckrbtolJvpTk+iTLktyV5AdJdl3HuQ5rzrVt5ysfYjVoHQEsBhbRBK0+rG72LwY4EJ7wXSSZnOSAJG8GPkhtmXMV8JVSypmllJtHWtCSJA0vjmwNI0kOBt4LXN9r1ybAgcDngWuBLYCvAOckOaCUsmpIC+22eunwcGqIWtHPV60AOAyecVmy7XNbSzXsDswFbgB+Xkrp7/kkSeoXR7aGiSRbAN8H3kmd2P24UsrSUspLSyk/LqXMKqX8BTiWeslr3zWcbypwYfP0nmaE64y2Q8Yl+UKSe5MsSnJS01Km5/WTmtG0O5I8nOTKJIev4zNclOTrSf4zyf3N4996nfetzbkebN73p0mevJZz9ozOvSLJX5Ms3wouugQ2/xfYays4eQL8ZBf45F9gszWdZ3nzd30RTD0VPkMNr48BLwW+BVwMnNr8Hnre+2lJLkjyQJKHklyX5IVrqfXlSS5tPvfiJOcm2bdt/9Tms7wuye+b7/WmJC/tdZ4jksxK8kiSS5K8qXnd1LZjDklycXOOO5P8dzORf021JclHk9yaZHmSG5K8tdcxn0oyL8mKJHcn+c6azidJ6j/D1vDxDeBnpZQL13lk1fMP6+PBLMkZSeY2T+cDr2t+ngbsCHyg7fVvAVYBhwD/SL2E9sa2/acDLwDeTB0B+jbwmyRPX0ddb6H+vXoONRC+tzl3j0nAp4GnA0dSL+v9cB3nhBqQPrgzHDoetns9vOtb8KZ/hlNOhI8vhl3fA3/f/oIVMO5i2PEUePbn4LUAj8DCr8JNb4TfAP8JPAAcRN1/CDV49fgBcFez/xnACfUUazQF+Gpz/GHAUup3NqnXcZ8HTm6+gyuBHzUT8mkuDZ8JnNXsPxn4cvuLkzwNOA/4dXPM0U197bX39jngXcD7qRP/T6SGyyOac74OOB54H7AX9Xfzl7WcT5LUX6UUH11+AO8B/gpMbJ5fBJyyluMnUSdu/7rX9hOBC9qeHwYU6kTp9uMuAv7ca9vvgdOan/egjvrs2uuYXwL/tZa6LqJOKk/btk8Ad6zlNfs0Ne7cPJ/aPH9Wr89weCmFAnu8BX4ElK/ABwocVeCoQ+AHm8O8lfCq/x9+9V/w2X+B0z8PJ/8Yjr8d3gCUd8CJBY7bHz5ODUOb9fF97dk8fwA4ZgN+r1Ooc8YO7fXZjm075snNtp5jTgRm9DrPx5tjpjbPv0Nt/9N+zDOaY7Zvnp8B/LatjuXA83q95qvA75qfPwzM6vk76MOHDx8+Bu/hnK0uS7I38AXqP7aP9uP4CcD3qOs9vap9XynlYwN4697zwhZQlziAeoktwE21E83jJlOXP1iby0sp7WtP/Rn41ySbl1IeSHIgdWTrGdQlGnreYFfgjn7Uu9U2zWT3d8K8np1bw5IVsMU1sM0DsOMW8KeXwbl7wLI+zvXo8jq6c30p5cG27ZdRQ+Z+wC3AvwOnJTkGuIA6p2vmmgpMsgfwr8Czge2oI3zjms/W12eB+r1D67vfhzra1e6KXs+fCeyZpH0ksud73IM6l63dfsBG1Dl+7b+bidT5agA/pY58zklyLnAONcw7h02SNpBhq/ueQ72UdmNbsBkPPD/JccCUnn/wmqD1Q+BpwGGllPs24H17B7tC67LyuOb53/Vx3PL1fcMkU4BzgfOBt1FDwbbApdTRuv7UO4nmzsMt2+5ADJQC4w6Ce2fCL94MM9ZyrtUT6ne8JgWglHJCku8Dr6BOyP90kuNKKWu6XPdbamA8FriTepn2pj4+2+PfaSmlNL/3gVzSHwecBvxHH/vuXMPxAEcBt/dVSyllfhP8Xwy8hHoDxqeTPLuU0ldglST1k2Gr+35JXWqg3enAbOqI10qoyz9QL5/tTw1ad/fj3D0rp68tWPTlGupIyQ6l/3PIejw7SdpGtw4GFjSjWs+khquPl1LmACQ5eoDnXzluHcHk7bVH4dqM3xZumQWvSLJZ2+jWIdRzPx7USimzqb+Lk5P8N/Bu+pgblWQb6qjU+3q+s2YUb6D/jc0EXt1r20G9nl8NTCul3NLPc95EvRtzt1LKGkcmSymPUOeKnZXki8DdwHOp88MkSevJCfJdVkpZUkqZ3v6gXvpa3DwvzYjWT6nB5e+BkmSH5rFxz7mSnJjkgrbTz6OO0hyRZLueSdj9qOlm6p2RZyR5fZLdkzwryfH9CEc7AV9NsneS1wP/TGsE5nbqP/r/2JzzCOplt4G4f/yG/5+EiV+qwfVh4DvNXYfPB04Fziyl3JJk4yRfa+6GnJrk2cCh1ODSZ13UhVXfk2TPJC8Avk4d3RqIrwN7pN4dunfzfR/b7OsJsF8CDkq98/N/Ne93ZOqirX+jCZMnAScleWdz/DOSHJfkvQBJ/iHJu5vv4inAO6ijXrMHWL8kqRfD1siwM3W0YyfqRPq72h7t83Z2pM7ZAaCUcid1ftTngYXAKfTfO6gjbF+mjrb8Fng+bfOk1uD71JG0K4BvAv9DE7ZKKfcAxwCvoYaWT1MnZg/EHY/VeVXrpRkVW/XcGiIOp97V+RfgV9T5Ze9sDl0NbEWdaD4L+EWzv896S23h80Zqm6DpwNeAT9L/dcB6zjOPehfpq4DrgA9R78SE5k7IUsr11N/FVOqSFddRJ9YvXMupP0m9m/J4avPs3zfvM6fZv4R6t+KlTf2vA47uGYGUJK2/PHEus7T+MlRtgJLnUCd9954I3h/bA9Mppfek82EryQeAzwJbFv+DlaQRxzlbGolmUG8SmMzARo4mU0fdZnWiqMGS5P3UOxLvoV46/iRwhkFLkkYmLyNq5CllCfWuxu2pAao/JjfHn9u8fjjbk3rZcgZ1TtvXqXPfJEkjkJcRNXLVZtSHU+dX3UffzajHA9s0f55LnRMlSdKQMWxpZKtNqfemXlacQL2DbjU1XE2k3g14AzBrBIxoSZJGIcOWRofaf3AX6sr6PXO5lgDzKWXl2l4qSVInGbYkSZI6yAnykiRJHWTYkiRJ6iDDliRJUgcZtiRJkjrIsCVJktRBhi1JkqQOMmxJkiR1kGFLkiSpgwxbkiRJHWTYkiRJ6iDDliRJUgcZtiRJkjrIsCVJktRBhi1JkqQOMmxJkiR1kGFLkiSpgwxbkiRJHWTYkiRJ6iDDliRJUgcZtiRJkjrIsCVJktRBhi1JkqQOMmxJkiR1kGFLkiSpgwxbkiRJHWTYkiRJ6iDDliRJUgcZtiRJkjrIsCVJktRBhi1JkqQOMmxJkiR1kGFLkiSpgwxbkiRJHWTYkiRJ6iDDliRJUgcZtiRJkjrIsCVJktRBhi1JkqQOMmxJkiR1kGFLkiSpgwxbkiRJHWTYkiRJ6iDDliRJUgcZtiRJkjrIsCVJktRBhi1JkqQOMmxJkiR1kGFLkiSpgwxbkiRJHWTYkiRJ6iDDliRJUgcZtiRJkjrIsCVJktRBhi1JkqQOMmxJkiR10P8DMowU+QIrY7MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlsAAAFBCAYAAABAYGoRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzs3Xd8VfX9x/HXJyEEwt4iKwIKAoo4EBkWVKDUVcE6cKFoVbS/qgWttQooCipatFKwiqLWgQMnCiKIWBxoQVmCgw1hbwiZ398f35PkEpMQIIebXN7Px+M+cu85537P996AvP1Oc84hIiIiIuGIi3YFRERERGKZwpaIiIhIiBS2REREREKksCUiIiISIoUtERERkRApbImIiIiESGFLRAplZkPMzAWPIcU9V8J1GB9xn+Sw7hPca0jw6BfmfUqCmS3P+V6iXRcRKVq5aFdARKQUGRz8/AwYH8V6iEgMUcuWiMhhYmYVol0HETn8FLZEpMTk6/LraGb/MbOtZrbZzN4ys6PyXV/ezP5iZt+a2U4zSzWzn8zssf3cp2tBXZhFHG9rZhPNbI2ZpQX1mWNmT5tZgpn1y9cd95uIcmZElHO8mb0UlJNuZhvM7E0zO7GI76FLcM124IeIazqY2dtmtt7MMsxsbfC+5HxlmZn91cxWBt/Pf83s1GL8OkSklFA3ooiEZRJQPeJ1b6AacA7ktvJ8AnTK977mQB/gLyVRCTNLCu5TO+JwzeDRrrj3MbPOwMdAxYjDdYK6nmtmPZxznxfw1reBWsHzrUFZlwCvAPER19UHrgHON7OOzrklwfHB5HVvgv++pgNWnHqLSPSpZUtEwrIMaAYcB2wIjp1tZvWD5/9HXtBaBJwBVAJaA/8qwXocT17QuhOogA9JnYHhQKZzbrxzLjK8fOacs+DRNTj2DD5orQBOARLxYW1jUOboQu6/A//ZKgK/C8LfGHzQmgO0DMrqBqTjQ+CjAGZWHbgrKCcN6IkPsM8AlQ/iuxCRKFDLloiE5T7n3FIAM/sc3wIE0ARIAS6IuHaAc+6r4Pmi4FFSVgGZ+P/eXYYPPYuBOc65vxWnADM7Fh+KwNf/fwVcdoKZHeWcW5fv+N8jP5uZdccHKoCTg7rk1z34eQY+yAF86Jz7OKjP34E/AQnFqb+IRJdatkSkKHsjniflOxf5OrWA9y6JeL474nlOeKgXcaykwtWv/gfSObcBuAnYhA83Q4EJwE9m9rmZVS1GuXWLef9aBRybexBlVTCzSvnKW53zxDmXCmwuZp1EJMoUtkSkKKsinrfOd65NIdflyIh4XtBaUOsjnh9/gPVKi3geOcOvaUEXO+fGAUcBJ+Bb2J4MTnUGbinG/TZEPP8koosx9wHEOecWFvDe/EE0sqxniyhrNz4g5miY88TMKlJwsBORUkhhS0SK8gl+HBHAb83sXjM7y8zuxY8fAh98ph1E2e9FPB9tZu3NrKKZtTCzQft574qI593NrIKZ1QNuz3+hmdU2s5H4LrkNwPvAuxGXNI54ntNa1MTMauQcdM79BPwYvDzbzG4zs+rBfdua2X3Aa/upc44vCAbKA1ebWV8zq2xmlczsdDN7FBgVnP+KvNbF35lZj6AlbhjqQhQpMxS2RKRQzrn15M2EiwPuxwer+8n778eQ4LoD9SQwK3jeBvga2IMfw1Rka5Nzbi0wI3h5Mj4krWHf4JSjAn7G4ef41rR09g2HUyKe54ytSga25FtC4o/44GPAP/CBKRX4Dt81GdktWlTdd+M/XzZQHngZ2AnsCu4/ED9rE+fcNuDh4K2JQV23AzfjvysRKQMUtkSkSM65EfhlGz4BtgBZwc9PgIuC8wdT7l7gLGAQfsD5bnwr2c/AW8Uo4gp8C9U2fAh6AehXwHVb8eHoG3y3XBY+3HwB9HXOvRNx7Z+AD8lreYqs72f4WYgv4sdPZeC/h3nAP4FiDbYPynoV34X5Fj4AZuJnNX6LD1eR64wNBe4O7pmGD6Xdg+tFpAww57StloiIiEhY1LIlIiIiEiKFLREREZEQKWyJiIiIhEhhS0RERCREClsiIiIiIVLYEhEREQmRwpaIiIhIiBS2REREREKksCUiIiISIoUtERERkRApbImIiIiESGFLREREJEQKWyIiIiIhUtgSERERCZHCloiIiEiIFLZEREREQqSwJSIiIhIihS0RERGREClsiYiIiIRIYUtEREQkRApbIiIiIiFS2BIREREJkcKWiIiISIgUtkRERERCpLAlIiIiEiKFLREREZEQKWyJiIiIhEhhS0RERCREClsiIiIiIVLYEhEREQmRwpaIiIhIiBS2REREREKksCUiIiISIoUtERERkRApbImIiIiESGFLREREJEQKWyIiIiIhUtgSERERCVG5aFdApESYJQINgRpAeSAd2Aqswrn0aFZNRESObApbUraZVQeOB1rj/zynA1lAPD50ZWK2AFiMc9uiVk8RETliqRtRCmVm483sg8N0r2Qzc2Z26gG8qQlwCdAK2ASsATYCW4Kfa4BNBl/+BUYE18e0w/k7ExGR4lHYKiXM7O4gbDyV73hvM5tiZhuD812jVMUSUyKBwAenc/HBagO+NasgWQC7YBdw7pEQuEREpHRR2CoFzKwD8EdgXgGnKwFfAHcc1kqVZr7rsCc+ZKUV5y0ZkBFc3zN4v4iIyGGhsBVlZlYNeBm4Dj+gex/OuZecc0OBjw6gzK5BK9h5Zvadme01s/+Z2SkR19Qys1fNbLWZpZrZQjO7dj/lzjCzMWb2mJltCVrb/mxmiWY22sy2mdlKM7uqiDKGANcA5wZ1zN9a18TMpprZHjNbZGbd872/VV2YVB6erADPNoeB06G44SkNyBoCvzOzT4LPvSVoaasWcY8TzGyame0ws11m9r2ZdSviM/3WzD43s61BeVPM7PiI8zldpH3289nONbMlwe9rppldFrwvOeKajmb2WVDGmuD3UbWIupmZ3WlmvwSfd76ZXZnvmvvMbIWZpZnZOjN7sZjfp4iIFIPCVvT9G3jTOffpwRYQhIXlBZwaCdwFnAosBT4ws6TgXAVgDnAefnD5E8DTZnb2fm53BbATOB0YAYwC3gF+DO7zAvCsmdUv5P0jgdeBT4D6weOLiPMPAk8CbYFvgNfMrHLwOesDM1vDjqdh4Ei4Nx0qXgJ/TwPbT70BWAa7HoV/xcFuoD1wEdAReC7isleAlOD8ScAQYG8RxVYKvof2QFdgO/C+mZXPd11Rn60xMBGYFJx/Engk8s1mdgLwMfBecE3voH6Rdc9vGNAfuAU/tm04/vd8blBmH2AgMAA4Fv/nYXYR5YmIyIFyzukRpQdwA/A/ICF4PQN4qpBrawMO6FrAueHAtIjXXYNrr4g4VhnYBlxfRH1eA56NeD0e+CDi9Qzgy4jXhh+I/l7EsQT8jMCLi7jPPuUGx5KDOt8YcaxBcKxz8Pr+SvCFg5scnO/g/DlwGeCGwx05x/I/AHctDHdw/u/hn+Uh9SM4sYDvq3nwegdwzSH8Xivhx4p1PoDPNhz4IV85fwuuSQ5evwiMy3fNScE1dfN/t0E9UoEu+d4zCvgweH4HsCTnz6Aeeuihhx4l/9DSD1FiZi2Ah/D/2GYcSlnOubsLOfVlxDW7zGw+vnUDM4sH/gpciv+HPxG/VMKM/dwud1yZc86Z2QZgfsSxDDPbCtQ98E+yb/nA2uBnTlmn7IH2CXCy8wEj1wLfQvbT/gpfAY3qwKrf+lCY4wsgG//d/Aw8jm+duwaYBrzlnFtcWJlm1gx4AN/aVwffYhwHND6Az9YS39oV6et8r08BmpvZpZG3D342w49Ji9QK34I52cwiv68EYHnw/A3gz8AyM5sCTMaH52KNhRMRkf1T2IqeM/CtVQvNcnvA4oEzzewmoFLI/+ANBP6C/4d2Pn623kPsPyTlD4aukGMH20WdW1YQ5ogoK64ufPE0vL/bd9XlOsW32gHwIrS5GhYUdoMgqCUWfAqcc0PM7GWgF34g/mAzu8k5V1h33QfAauBG/HITmcAifHgt7mcrjjjgWeAfBZxbU8j14Fv3VhZUF+fcqiD4nw2cAzyG/7ynO+d2H0DdRESkEApb0fMO8G2+Y8/jW2cewnfFHaoO+LFamFkloA2+KwqgM/C+c+6l4LwBxxERWkKUjg+WB2rOdrjqLNhWBdYVdtECOIFCwlYTWLUQesyG+PZ5hzvig8kPOQeccz/hfxdPmtkY4HoKGBtlZrXwrVIDXDDuzsxO5sD/bi0GLsx3rH2+13OA1s65n4tZ5iL8pIAmzrnphV3knNuLHys2ycxG4L/bTvjxYSIicog0QD5KnHPbnHMLIh/4Qdtbgtd+UJRZTTM7CR+UwHcjnWRmR+WUZWbDzWxaAbf5u5l1N7PW+KCQjh/8DX5A+9lm1tnMWgJPAceE82l/ZTnQxsxamFltM0vY3xsCozOgUnsY8BAcNwHqDYS2reGWJVCxOAWMghkJkN4D7g1mHZ4JPA1MdM79bGYVg5mVXYNZhKfjg+miQorcil9Q9QYza25mvwHG4lu3DsRYoJmZjQy+l974ljLI6zJ9GGhvZmPNrF1wv/PM7OmCCnTO7cRPSBhpZtcF159kZjeZ2R8BzKyfmV0ffBfHANfiW7322yUrIiLFo7BV+l0AzAVyZis+E7y+KeKa+vgxO/n9Fd8tNIdgpllE19Aw/Kyzj4CZ+KD3cklXvhDP4FuRvsUPsO9UnDc559YmQxf88g1D+8Lof8NN5SCj9q+7MgvUBLKehn/s9C1Ps4F38WPbrgsuycLvrzgeP3D87eB8geucOeey8ePeTsS3po0G7qWY639FlLMC6IP/fX8P3A4MDU7vDa6ZB5yJH3D/WXDdcGB9EUXfi59NORBYCEwN7rMsOL8NP1vx86D+fYDezrllvypJREQOigUNKBJDzK9b9SlQxzm3KcrVKXlmZ+AHf+cfEA7AnXD5I/BqIe+uCyzAufyDz0sdM/szcD9Q3ekvqohImaUxW1IW/YAfl5XIgbUgJeLHii0Jo1KHysxuwc9I3Igfb3cvMF5BS0SkbFM3opQ9zm0DpuBbqQqaVViQxOD6KcH7S6Pm+G7LH/BLSYwFBkW1RiIicsjUshWDnHMzKOaK6mWWcyswm4RfmiEL2EzBm1HHA7WCn5PwY6NKJefc7fixWiIiEkMUtqTs8oHrdaAFvluxHJBRx89MrItfvDMTv47YklLcoiUiIjFMYUvKNh+gvsZsLtAIqL7Qr5v1HX6m3SqcK4k1y0RERA6KZiNKzDGzIc65IdGuh4iICGiAvIiIiEioFLZEREREQqSwJSIiIhIihS0RERGREClsiYiIiIRIYUtEREQkRApbIiIiIiFS2BIREREJkcKWiIiISIgUtkRERERCpLAlIiIiEiKFLREREZEQKWyJiIiIhEhhS0RERCREClsiIiIiIVLYEhEREQmRwpaIiIhIiBS2REREREKksCUiIiISIoUtERERkRApbImIiIiESGFLREREJEQKWyIiIiIhUtgSERERCZHCloiIiEiIFLZEREREQqSwJSIiIhIihS0RERGREClsiYiIiIRIYUtEREQkRApbIiIiIiFS2BIREREJkcKWiIiISIgUtkRERERCpLAlIiIiEiKFLREREZEQKWyJiIiIhEhhS0RERCREClsiIiIiIVLYEhEREQmRwpaIiIhIiBS2REREREKksCUiIiISIoUtERERkRApbImIiIiESGFLyoqawMPAKiATyC7skZ2dPbiI86nATKD3Ya6/iIgcocw5F+06iOxPDWA20BgoX0Jl7gHuAx4rofJEREQKpJYtKQtuARpRckELIAl4EKhWgmWKiIj8isKWlAVXAYkhlJsO9AqhXBERkVwKW1IW1A+p3Ar4rkkREZHQKGxJWRDWn9N4oFxIZYuIiAD6h0ZiVEZGBps2bQKgdu3aJCQkRLlGIiJypFLLlsSU119/nQ4dOpCUlESDBg1o0KABSUlJdOjQgQkTJkS7eiIicgRSy5bEjLFjx3LbbbfRr18/Bg0aRL169QBYv349U6dO5aqrrmLz5s0MGDAgyjUVEZEjidbZkrJgF1Bpfxc1bdqUe+65h/79+xd4/rnnnmPYsGEsXbo051A2cC/wUElVVEREJD91I0rMSElJoVOnToWe79SpEykpKYexRiIiIgpbEkPatGnDmDFjCj0/evRoWrdufRhrJCIiojFbEkNGjRpFr169mDx5Mt27d99nzNbHH39MSkoKH330UZRrKSIiRxqN2ZKyoFhjtgBWr17N2LFjmTVrFuvWrQPgqKOOomPHjtx88800bNgw8nKN2RIRkdApbElZUOywdYAUtkREJHQasyUxo1u3blx99dXRroaIiMg+FLYkZsTFxWFm0a6GiIjIPjRAXmLGtGnTol0FERGRX1HLloiIiEiI1LIlMWX37t288sorzJo1i5SUFOLi4mjatCkXXnghPXr0iHb1RETkCKSwJTHjp59+4pxzzmHz5s0cffTRrFmzhuzsbLKysrj00kvp1q0bEyZMICEhIdpVFRGRI4i6ESVm/OlPf6JDhw5s2LCBH3/8kfXr13POOedw2mmn8csvv7B27VoeeOCBaFdTRESOMFpnS8qCYq2zVblyZWbPnk2rVq1yj33zzTf07t2bVatWMX36dK6//nptRC0iIoeVWrYkZtSoUYOtW7fuc2zbtm1UrFgRgObNm7N27dpoVE1ERI5gClsSMy666CJuuukmpk2bxurVq5k6dSq33norvXr1AvweiUcddVSUaykiIkcaDZCXmDF8+HD69+9Pjx49cM5hZvTp04eHHvK9hKmpqQwcODDKtRQRkSONxmxJWXBAeyNu2rSJFStW0LhxY+rUqVPUpRqzJSIioVPLlsSc2rVrU7t27WhXQ0REBNCYLYkhM2fOZPbs2bmvN23axBVXXEHt2rVp0KABd911F5mZmVGsoYiIHInUsiUx4+abb2bYsGG5r//4xz+yZMkShg8fTlpaGsOHD6dSpUrcd999UayliJRKZolAQ6AGUB5IB7YCq3AuPZpVk7JPY7akLCjWmK2kpCQWLlzIMcccQ3p6OklJSSxatIjjjjsOgMmTJ3Prrbfy888/57xFY7ZEjnRm1YHjgdb4Boh0IAuIx4euTGABsBjntkWrmlK2qRtRYkblypVZvXo1AJs3byYuLo5jjz0293zLli1Zs2ZNtKonIlFgZsvNrOBpyGZNgEuAVsAmYA2wEdgS/FwTHG8NXBJcX5J162pmzsw0yDTGKWxJzDj33HMZOnQoGRkZ1K9fn2bNmvHJJ5/knp88eTItW7aMYg1FJJKZ3R2EjafyHTczG2Jma80s1cxmmFnr/ZTVz8x2HcDNmwDn4oPVBnxrVkGygvNbgHNLOnDJkUFjtiRmDB8+nM6dO9OqVSv69OnDOeecw+9//3t69+5NamoqH374IW+//Xa0qykigJl1AP4IzCvg9J3AX4B+wBLgPmCqmbVwzu0sgZtXB3riQ1RaMd+VFlzfE7PX1aUoB0ItWxIzjjrqKObMmcNll13GpEmTeO6558jOzmbWrFkkJSXx5Zdf0rNnz2hXU+SIZ2bVgJeB6/CD0CPPGXAbMMI595ZzbgFwDVAF6FtIeV2B54FKQUuZM7MhEZdUMLOnzWyHma1uDw/gW6zSABZAUmu4pQK8lAAT6sFDj0LzAm6VFryvRdA9OcTM/mNmu8xsXf7uSjO7w8zmmdluM1tjZs+aD3qFfS/9grJ6mdliM9tjZu+ZWTUzu9jMfjKz7Wb2kplVLPwbltJGYUtiStWqVXnggQeYP38+u3fvJjU1laVLl/Liiy/Stm3baFdPRLx/A2865z4t4NwxwFHAxzkHnHOpwEygY86xoGtxRvDyC3xA2wPUDx4jI8q8HZgPnFwRHvsGbn0IaoMf/X4WDN4BtYbA/S/AbU1h4d/gwU/8zMT8NgMnBM/vAH4ATgYGAw+ZWe+Ia7ODerXGB8X2wD+L/GYgEd+qdwVwNnAq8BY+cPYBfg+cBwzYTzlSiqgbUWJSRkYGmzZtAvwipwkJCVGukYgAmNkN+FajKwu5JGcD0/X5jq8HGkS8XpnzxDmXbmbb/VO3roAyP3bOPRVU4L0acPdkaPM3WHQPnLgFjlkOVzb0MxHpCy/XgPbDods5MDFfWVlAuXJQLhO+ds49GBz/0cxOwwewiUG9RkW8b7mZ3Qm8a2bXOOeyC/n85YBbnHNLfHXtFXxYrOec2xQcexfoBjxWSBlSyqhlS2LK66+/TocOHUhKSqJBgwY0aNCApKQkOnTowIQJE6JdPZEjmpm1wC+10tc5l3EoZTnnrnbOXV3MyyPHhdWoDFt3QHWA/0GzLEhMhv+Ug9fLwZvl4PXt0GRdXvDLLyPe//v5Zb7jX+JnNgJgZmeZ2VQzW21mO/EhrDyFlwuQlhO0AuuBdTlBK+JY3aI+sJQuatmSmDF27Fhuu+02+vXrx6BBg6hXrx4A69evZ+rUqVx11VVs3ryZAQPU+i4SJWfgu+8W+qFZgF/P6kwzuwm/nl5Oy1Q9IlqvgtcFtVoVR2SwKw9kZ4MBZEJCIuy8Dt5KhWrVYEMrWFQZ0hr5bsmCZBG8vzDmZy1OAp7BD/DfjO9ufDWoQ2Hyb3Ph8tU/55gaS8oQhS2JGY888gijR4+mf//+vzrXp08f2rdvz7BhwxS2RKLnHeDbfMeeB37Ct3ilA8vwoao78A2AmVUAugCDiig7HR/c9icdiMuCKiPhN1WgXhpUbQTL/gTzKxe+BESkeHzg6ZDveAf8GC7wY63KA7c757KCz3FeMcqWGKSwJTEjJSWFTp06FXq+U6dOpKSkHMYaiUgk55dL2GfJBDPbDWwJZh3mHBsF/M3MFgM/An/H7yTxSsQ1LwZl5nQlLsfPOuwOzAX2OOf2RFxfHji2tx9kXzMbdreC5dfCrOOhwcPQPwXGd4HVi6H6ZDjlLPjuAVhUwEdJyPKD3zuY2d3Am0BX4Gr8wHbwATIOuM3MJuKD2G0H/q1JLFAzpMSMNm3aMGbMmELPjx49mtati1wXUURKh0eAfwCj8S1h9YEe+dbYahw8AHDOfQGMxXfTbQTuNLNyQAJ+9uAdwMlz4CtgayKk/A5W1ILMz2FoMsx7Fm69FMYMh7vWQYPWfiHT/OKBzEzf3fc4cCI+3A0D7nPOvRnUZx7w5+C+i4DrgYJXspeYp70RpSwo1t6Is2bNolevXtSvX5/u3bvvM2br448/JiUlhY8++ojOnTvnvEV7I4rEGDOLxy8f0QZogR9MvgD4wTm3O7joDPxA9g0HcYu6wAKDCcBTzrmR+3uDiLoRJWZ06tSJhQsX8vTTTzNr1iymTZsG+MVO//CHP3DzzTfTsGHDKNdSREqamcUBTfAB63h8i9QCYFohK87/gG/tSqT4K8gTXB+PX9VepNjUsiVlQbFatg6CWrZEyqhgpfmG+IDVGtiJD1gLXXG20snbG7G4W/Yk4lu1JuHcCjNbjlq2pJjUsiUiImVCELDqkxew0vEB63nn3OYDKsy5FZhNwu+RmIVfmqGgmYjxQK3g5yScW+Hf7pIP7lPIkUhhS2LG0KFDuemmm3LHauX3xBNPkJSUxA033HCYayYih8LM6uIDVpvg0AL8zMQN7lC6Z3zgeh0/tusE/L+JGfjQFY8fXJ+J3+pniTafloOlbkQpC4rVjRgXF8fcuXML3QPx4YcfZsqUKUyfPj3nkLoRRUopM6uFb71qg+/CW4gPWSmHFLAKv2F5oBF+ZfmcsVzbgFU4l17i95Mjilq2JGaYGQsXLiQzM/8CzF7FihWZO3fuYa6ViBSXmVUnL2BVwQesD4BVoQSsSD5Q/RLqPeSIpZYtKQuK1bIVHx+Pc46IbUB+xTlHdnbu/q9q2RKJMjOrgl+GoQ1+bNQP+BasFUVs1ixSpqhlS2LKpEmTtHCpSClnZknkBayjgMXATGBpztY2IrFEYUtiytFHH03jxo33f6GIHFbB/oYt8QGrEX47m6+An51zBff9i8QIhS2JGc8//7yClkgpEuxH2AIfsJLxm0x/B7zuNOhcjiAasyVlgRY1FSkjzCwBaI4PWM2BlfgxWEucc3ujWTeRaFHLlsSMuXPnkpaWRocOHaJdFZEjSrAfYTN8wDoOSMEHrEnOuT3RrJtIaaCwJTFjwIABXHrppfuErVGjRvHEE0/kvnbOsXz58ijUTiS2BPsRJuMDVktgEz5gfeyc2xXFqomUOgpbEjOWLFlCt27d9jnWrl070tLSGDhwILt27WLIkCHRqZxIDAi2y2mMD1itgO34gPW0c257NOsmUpopbEnMSEtLo1q1avscq1WrFunp6dxxxx1s3LhRYUvkAAUB62jy9iNMxQescc65LdGsm0hZobAlMaNJkybMnj2b5OTk3GOzZ8+mUaNGgO9CFJH9CwJWPfICVjY+YL3knNsYzbqJlEUKWxIzLr/8cu644w4SExNp164dX3/9NXfddRe333577jVFrS4vcqQzs9rkbfhcDh+wXgfWhb5djkgMU9iSmDFo0CDmzp1L7969c7ft6d+/P4MGDQKgcuXK3HXXXVGupUjpYmY1yNuPsBJ+P8J3gDUKWCIlQ+tsSVlwQOtsrV69mpSUFJo3b06NGjWKulTrbMkRycyqkhewqgOL8CFrpfYjFCl5atmSmNOwYUMaNmwY7WqIlCpmVom8/Qjr4vcjnA4sU8ASCZfClsS0qVOnMmXKFBITE7nkkkto27ZttKskctiYWUXgeHzAOhr4EfgC+EX7EYocPnHRroBISTn77LMZOXJk7utRo0bRq1cvvvzySyZPnszpp5/OtGnTolhDEQAqAr8B/gb8B/gHcBFQIrM3zCzRzE40s77Abfgtc74FHnPOTXTOLVHQEjm8NGZLyoJijdmqWbMmM2fOpE2bNgDUqFGDp556iiuuuAKA+++/nylTpjBr1qyct2jMlhwO1YBOQDfgt/iNmfcASUBCcM1ufItTLyDrQG8Q7Ed4HL4Fqymwgrz9CNMOsf4icogUtqQsKFbYqlSpEt988w2tWrVi3bp1NGrUiL179xIfHw/A0qVLOemkk9ixY0fOWxS2JAz1gC7AOUB3oCF+IdDKQHyYFehfAAAdwElEQVQR79sN3AWMLs5NzKwcefsRHguswQesxc651IOtvIiUPI3ZkpjRtm1b3nrrLVq1akWNGjWoUKEC69ato0GDBgCsWbOGypUrR7mWEmMMvz/gmfhgdRZQE0gDqpDXNVi+GGVVAoYA/wIK/L/gYMPnY/ABqwWwAR+wJjvndh/kZxCRkClsScwYMmQIF1xwARkZGfTt25d777039+fevXu58847ufrqq6NdTSnb4vAz+rrgu/w6A4n4VtLIJJ94kOVXxHcHLsk5EGz4nLMf4fHAVnzAmu6c21FQISJSuqgbUcqCYq+z9f777/OnP/2JlStXAnkrxleoUIFbbrmFESNGEBeXOy9E3YiyPwnAyfhw9TugPb7VKQ4/5qqk7QZuM7Nx+O7H1sFjNz5gLXTObQ3hviISIoUtKQsOaFFT5xxz5sxh6dKlZGdnU79+fU455RQqVfpVEQpbkl8ScDp+tuDvgBPxXYLlgQqHowIrV678b5MmTd4CMoH5+IC16XDcW0TCobAlZcE2/IyukpYB3A08FkLZUjZUx88UPAvoie/Cyz9TsMRkZGQwZ84cqlSpQqtWrQq8Jj09fXtiYmILYIO2yxGJDRqzJWXBT8CpIZSbil9FW44c9fFdgmfjB7Q34NczBUs82M+fP5+BAwfyxRdfUKtWLerUqUO3bt145JFHcvfxzFG+fPlE51x5ChkkLyJljxY1lbJgHL61oaTFA5+EUK6UDoZfGqEf8CqQAiwDngFuwM/qK48PV0UtyVBsGzZsYN68eaSnp+ce27lzJ0OGDCEpKYn58+fz7bffcuWVVzJy5EgWLFiwT9AKZOADoYjECHUjSlmQALyNH0dTEms3pOPHw/QGppRAeVI6xOEHk5+JnynYCR+m8s8ULFF79+7l0UcfZdy4cWzfvp3GjRvTunVrxowZQ7Vq1diwYQNHHXUU06dPp2vXrgCsXbuWxo0b88UXX9C+ffv8RTrgBeDasOosIoeXuhGlLMgAfg+cD1yFH1dT6HiajRs31qpTp87mQk7vBCYDLxExvV7KpATgFHwr0LnAafhgFdZMQbZv3061avv2Mk6YMIE33niDBx98kN/+9rd8//339OzZkzZt2nDXXXdRqVIl6tSpw3/+8x9atWpFZmYm99xzD3fccQennHJKQbcx/BgyEYkRatmSmGNmQ5xzQ6JdDylxSUAHoCu+5eoE/EzBRA5+XasipaSk8Oabb/LKK6+wdu1aWrZsyeWXX84ll1xCUpLPc6eddhotWrTgueeeo3x5v3bp6aefzsknn8yYMWMA+PTTT5k4cSIvvPACGRkZ1KxZk8aNG3PBBRdw9913F3TrdPzG0YX9T4OIlCEasyUipVUNfGvm48Ai/KzUicBf8RMmEoGqlFDQSk9PZ9asWSxfvhyA7OxsHn/8cd544w0uvPBCJkyYQIsWLbj99tsZP3587vvatGnDnDlzWLRoEQCvvfYatWvX5pZbbsm9plOnTuzZs4c+ffqwfPlyvvvuO/r378/999/Pu+++W1B19uIXTBWRWOCc00OPmHoAQ6JdBz0O6nG0c+5S59yzzrllzrk059w251ymC9G0adPcGWec4SpWrOgaN27sXnjhBZeRkeHS0tLcE0884T744IPca7Oyslz//v1dly5dco/98ssv7t5773WtW7d28fHxrnLlyu744493AwYMcKmpqc455+bNm+dq1qzp1q5du8+9mzZt6u6//36Xmfmrj5jlnHvCRf93ooceepTAQ2O2RCQaDGiOH2/VA+iGnxV4MHsKFsuaNWvIysqicePGZGVlER8fz+LFi3n44Yfp2LEjL7/8MrVq1SI1NRUzo3z58vzf//3fPmVkZWXhnKNKlSq5x5o2bcrpp5/OpEmTmDRpEqeddhrff/89V155JRUqVGD48OGsWLEC8GO+6tevD8CyZcvYtGkTSUlJuZulR4jDr/slIjFA3YgicjjEA22BW4EP8fv7fQc8AVwK1CWvW/BXayEcjJSUFP75z39yzjnnUKNGDRo1asRrr70GkLtl0zvvvMOOHTsYOXIkxxxzDDt27KBmzZr7hJ/s7GzA9wLs2bOHDz/8kOuvv36fc4MHD6Zv37707NmTSpUq0a1bN/r168c333zD5s2bOfXUUznuuOPo3bs3r732Gm+++SY33ngjZ555JpdeemlhH6EpIc6iFJHDRy1bIhKG8viZgr/BD2Y/Db/cRjn8ZsslbuXKlVSpUoUaNWqwevVq+vfvz9SpUxk8eDB33nknnTt3zh3Ubmbs3buXDz74gN69ezN48GDGjRtHtWrVaNGiBX/9619zl2SIi4sjMzOTcuXK8de//pUWLVrQvXv33HObN29m165dpKam+g8eDJL/+eef2bNnD7Vr1yYhIYHRo0czfvx4hg4dipnRo0cPbrjhBho2bFjYR0rFTwjQWnAiZZzCloiUpE7AMOAMQp4pCL5l6vHHH2fOnDm0adOGBx54gO7du1O3bl3atWtHamoqgwcPBiAzMzP3fdnZ2VSoUIG1a9cyceJEGjZsyLhx46hZsya33HILAwcOZOTIkbRv35709HTKly/Pp59+yvvvv8+4ceOoXLkyGRkZJCQkUK1aNfr06cOwYcOoUqUKnTt35t133+XHH39k0KBBJCT4VUpOPvlk2rRpQ3x8fEHdhgVJBNqgsCVS5qkbUURKykPAx/ilGUp8puDGjRv3Ofa///2PkSNH0rFjR2bMmMGrr75KixYtyM7Opnz58jRr1owff/yRyy+/nC5dunD22Wfz6KOPsmzZstxuxE6dOvH111/TrVs3evbsyWmnncZzzz2HmfHmm28CvqUqNTWVe++9l549e9Kzpx9KlROiypUrx6BBgxg4cCAvvfQS5513HjNmzODPf/4zF1988T51Ll++fHGDFvg1w9L3e5WIlHoKWyJSEk4GbqMEFxNdsWIFd955J82aNaNp06Y888wzZGRk5J6/7bbb6NChAyNGjODUU0+lXr16NG7cODdINW/enLZt25KVlcW1117L2WefzWOPPcZ1113HqlWrAL8eFsAxxxyTW25ycjLJycksXbo099iLL76Y20o2depU7rzzTk488UQ++OADAKpXr86wYcP46KOPSElJYebMmfTr1y+3S/EgZQE/H0oBIlI6KGyJSEnoRQnNHMyZKj18+HDmz5/P4MGDmTRpEueff35u2FqyZAmpqam0atWKkSNH0rRpU8477zweeeQR1qxZA0DLli0ZPHgwY8eO5brrruO+++5j+vTprFy5kldffRWACy64gIoVK/L999/vU4eZM2fSoUMHsrOz2bt3L3fffTdz587l2GOP5eKLL2bBggVcddVVdOrUaZ/31alTpyS+AvBdsAuAqSVVoIhEj8ZsiUhJqM8BbOacM+A8P+ccZsaYMWP48ssvmTRpUu4A8pxzAOvWrSM+Pp533nmHhIQERo0axS+//MKoUaP46quvmDhxIvXq1ctdZiFHq1ataNSoET///DNpaWk0btyYAQMG8Nhjj1G5cmXOOeccJk6cSO3atenZsydxcXHs2LGDe+65h8aNG3PmmWdSr169Q/ia9msX/r/LLwN/we+TKCJlnMKWiJSE5fjxRQW2bmVlZfH+++/zj3/8g1WrVnHRRRdx5ZVX0q5du31CVM7P7777jubNm7Nt2zb69u1LRkYGHTt25JprruHEE0+kRYsWLFu2jIULFzJt2rTc7sCWLVty7rnnsnz5cpKTk/epQ3Z2NnFxcfzyyy907tyZxEQ/nGzo0KFUr16dCRMmcPfdd9O8eXMeeOABTjjhBADq1q3LX/7yl5L/xnyQ2oX/zrYDn+LHvE0G1oZxQxGJDu2NKDFHeyNGRWdgEn5QfK6cIPXf//6XW2+9lR49etC2bVvGjBnDhg0bePvtt2nduvU+gWvnzp0MGDCAdevWkZCQwBlnnEGzZs147LHH2L59O1999RW1a9emS5cuLF26lMWLF+cuMrphwwZatWrFmDFj+MMf/sD27dvZtWsXDRo0YNeuXYwYMYLXX3+dcePG0aVLl33um5KSQp06dQpscSsh2fiN0Cviw9Q0fDfh5yhcicQ0jdkSkZLwDVAh/0EzIyMjg1deeYXs7GxGjBjBFVdcwYcffkjVqlV5+OGHc6/LUaVKFbKzs5k2bVrumld9+/Zl8uTJ7N69m2eeeQaAHj16kJiYyDfffJP73uXLl1O1atXcNa++/fZbrrvuOs4880yaNGnCu+++y/3330+XLl1+dd/69euXdNDKBHYAGcAP+AVcr8J3uR4DXA9MQEFLJOYpbIlISUjDB4pfiY+P57///S+XXHJJ7kzBqlWr0rdvX77++uvcWX/OudwV2du2bQtA165dSUhIIDMzkzp16nDWWWcxc+ZMAHr37s2xxx7L7bffzty5c9myZQsvvfQSxxxzDL169QKgXbt2dOrUicsuu4zPPvuM+fPnc9lll4X5HewIfn4LDAcuwm+o3Qq4A3gf2BJWBUSkdNKYLREpKR/hF+HcZ6B8XFwce/fuZc+ePbl7EoIPQnFxcXz++ec0bdqUrKys3Jamjh070qJFCz799FMuvPDC3BaynJmKAK1bt+aJJ55g0KBBXHbZZaxevZoTTjiBESNG5M4KrFmzJvfdd19Yn3cPvmswDt+y9yEwE5iD1scSkQgKWyJSUj4FbsZvKL2Pk046ie+//36fsNWwYUMaNmzIvHnzAPbpwjv55JO55JJLePTRRzn77LPp2bMn06dP59tvv2X48OG517Vs2ZKJEycyd+5ckpOTqVu3bpifbxc+WKUDs/DhciawEB+6REQKpLAlIiXlSwpZ1LRHjx7cfffdpKSk0KRJE8CPkcrMzMztOty+fTtLly6lXbt2JCUlMXToUJYtW8bf//53brzxRjIyMrjpppu46KKL9ik7ISEhdx/DEuTwg9kT8ZtmzwCm4AezL0VLMojIAdBsRIk5mo0YVT8Cx+Y/uGXLFho0aMDjjz/OzTffDPgteGrXrs2YMWO44ooruOWWW3jvvff45JNPaNGiRe5MwdmzZ5ORkUGHDh0OZKubA5UF7MYP8l+D34/wE3y4SgnrpiJyZFDLloiUpClAM/JNvqlZsyY33HADI0aMwMzo06cP//znP0lOTubEE08E4IQTTmDnzp25gSpn/FYIrVbgZwjuwS/D8FNQ7+nAF/iWLBGREqOWLYk5atmKqouA8eRbbwtg69atPPTQQ0ybNo0ff/yRGjVq8Pjjj/OHP/zhcNQrLXgkAvPwg9k/A77Ghy4RkdAobEnMUdiKqjrOudVmVug+iYsXL6ZSpUo0atQozHpEzhT8mryZgnPxrVoiIoeNuhFF5JCZWTWgNdBm+/btqVWrVi00bLVs2TKMKuTMFEwD/oufKfg5sAjNFBSRKFPYEpGDYmaV8Yt1tgHq4Bc1/aRy5crvAleHeOvImYKb8TMFP8a3XC1HMwVFpJRR2BKRYjOzJOB4fCvW0fjZh/8FfnHOZQWXfYwfu1WlhG6bhW+5qgisxu8n+Elw33UldA8RkdAobIlIkcysAtAC34LVGPgZv2L6z865gsY/fQYU2o1YDBlAKj5cLQEm4xdM/QLYdgjliohEhcKWiPxKMMD9OHwLVlN899w84A3n3P62olkNLADaUbz9V/fiV2VPBL7Dj7eaAczGhy4RkTJNYUtEADCzckBzfAtWc/JC07vOub0HWNy1+FmAFQs4t4e8cVVfA5Pwg9nnApkHXnMRkdJNYUvkCGZm8fiWq9b4rsL1+ID1kXNu9yEUPR84CxgXlGvAFvyegpPxg9kXo5mCInIEUNgSOcKYWRzQBN+CdTx+Rt9CYJpzbmcJ3uorfIgzfBfhgbaOiYjEBIUtkSOA+b1vGuHDT2v80gkLgH8758IedO5Q0BKRI5jClkiMCgJWfXwLVmv8IPQFwPPOuc3RrJuIyJFEYUskxphZXXzAahMcWgC87JzbEL1aiYgcuRS2RGKAmdUiL2CVx4/BehNIcdoAVUQkqhS2RMooM6tOsB8hfrX2hcB7wGoFLBGR0kNhS6QMMbMq5A1yr4Xfj/BjYIVzTssoiIiUQgpbIqWcmVXCL9HQBjgKvz7VZ8CyiP0IRUSklFLYEimFgv0IczZ8bojfj/Ar/H6EWmVdRKQMUdgSKSWC/QhzNnxOBpbit7B5vRj7EYqISCmlsCUSRWaWAByLb8FqDqzEL9Xw9kHsRygiIqWQwpbIYRbsR9gM34J1HLAWP5NwknNuTzTrJiIiJU9hS+QwCPYjTCZvP8KN+Basj51zu6JYNRERCZnClkhIgu1yGuMDVitgG74Fa6xzbns06yYiIoePwpZICQoC1tHk7UeYim/BGuec2xLNuomISHQobIkcoiBg1SMvYGXjA9ZLzrmN0aybiIhEn8KWyEEys9rk7UdYDh+wJgDrtV2OiIjkUNgSOQBmVoO8FqxK+DFY7wBrFLBERKQgClsSG8wS8Sut17gGGmF2KrAVWMUhLghqZlXJ2/C5OrAI+AhYqYAlIiL7o7AlZZtZdfK2tSkHpLeGmsCJQHkgE7MFwGKc21b8Yq0yfgZha6Aufj/C6fj9CLXhs4iIFJvC1hHIzGYAC5xzt0a7LvmZWTKwDDjNOfftfi5uAvQEsoBNwU82+hmAOQPT4/GB6UQzexr4g3PuzULuXZG8DZ+PBn4EZgG/lLUNn81sPFDbOXdetOsiInKki4t2BWRfZna3mTkzeyriWIKZPWxm88xst5mlmNkrZtZ4P2V1DcqqHX7NDzMftM4FtgAbCIJWAbKC81sAToZ9vgszSzSzE82sL3Abfsucb4HHnHMTnXM/lrWgJSIipYtatkoRM+sA/BGYl+9UEnAy8CDwHVANeAyYbGYnOucyD2tFo813HfbEh6i0Yr4rDaArnPSFWe1OeUs1NAWWA/OBt5xzxS1PRESkWNSyVUqYWTXgZeA6/MDuXM657c657s65Cc65Jc652cCN+C6v4wspLxn4NHi5MWjhGh9xSZyZPWRmm8xsg5mNDLaUyXl/+aA1bbWZ7TGzb8ys534+wwwzG2tmT5jZ1uDxaL5yrwzK2hnc9w0za1BEmTmtc73M7H9mlloDZsyEqvfAsTXgyXLweiO4dzZUKayc1ODP+gZIfhqG4sNrNtAdeA74DHg6+D3k3PsEM5tmZjvMbJeZfW9m3Yqo62/N7PPgc28xsylmdnzE+eTgs/Qxs6nB97rIzLrnK+dcM1tiZnvNbKaZXRa8Lznimo5m9llQxhozGxMM5C+sbmZmd5rZL2aWambzzezKfNfcZ2YrzCzNzNaZ2YuFlSciIsWnsFV6/Bt40zn36X6v9HL+Yc0NZmY23syWBy9XAX2C562B+sCfI95/BZAJdARuxXehXRpx/nngN0BffAvQC8D7ZtZ2P/W6Av/n6gx8IPxjUHaO8sBgoC1wHr5b79X9lAk+IN3WEDrHQ52Lof9zcNkgeGo4/G0LNL4BLo98QxrEfQb1n4LTh8FFAHth/ShYdCm8DzwB7ADa4893xAevHK8AKcH5k4AhvohCVQJGBdd3Bbbjv7Py+a57EHgy+A6+AV4LBuQTdA1PBCYF558EHol8s5mdAHwMvBdc0zuoX2Td8xsG9AduwQ/8H44Pl+cGZfYBBgIDgGPxv5vZRZQnIiLF5ZzTI8oP4Abgf0BC8HoG8FQR15fHD9x+L9/x4cC0iNddAYcfKB153Qzgy3zHpgLPBs+b4Vt9Gue75h3gX0XUawZ+ULlFHPs7sLqI97QM6tgweJ0cvD4132fo6ZzDQbMr4DXAPQZ/dnC+g/M7witVYUU6XPB/8O6/4P574PkH4ckJMHAlXAK4a2G4g5vawN/wYahKAd9X8+D1DuCaQ/i9VsKPGeuc77PdGHFNg+BYzjXDgR/ylfO34Jrk4PWL+O1/Iq85KbimbvB6PPBBRD1SgS753jMK+DB4fgewJOfPoB566KGHHiX30JitKDOzFsBD+H9sM4pxfTngP/j1ni6IPOecu/sAbp1/XNha/BIH4LvYDFjkd6LJlYhf/qAoXznnItee+hJ4wMyqOud2mNnJ+Jatk/BLNOTcoDGwuhj1rVErGOx+HazIOVkTtqVBtblQawfUrwazesCUZrC7gLIyUn3rzjzn3M6I41/gQ2Yr4GfgceBZM7sGmIYf07W4sAqaWTPgAeB0oA6+hS8u+GwFfRbw3zvkffct8a1dkb7O9/oUoLmZRbZE5nyPzfBj2SK1Airgx/hF/m4S8OPVAN7At3wuM7MpwGR8mNcYNhGRQ6SwFX1n4LvSFkYEm3jgTDO7CaiU8w9eELReBU4AujrnNh/CffMHO0det3Jc8Pq0Aq5LPdgbmlklYArwCXAVPhTUBj7Ht9YVp77lCWYeVo+YgWjgHMS1h02L4e2+8EMRZWWV899xYRyAc26Imb0M9MIPyB9sZjc55wrrrvsAHxhvBNbgu2kXFfDZcr9T55wLfu8H0qUfBzwL/KOAc2sKuR7gfGBlQXVxzq0Kgv/ZwDn4CRiDzex051xBgVVERIpJYSv63sEvNRDpeeAnfItXOvjlH/DdZ23wQWtdMcrOWTm9qGBRkLn4lpKjXPHHkOU43cwsonWrA7A2aNU6BR+u/uacWwZgZr0PsPz0uP0Ek6v9HoVFia8NPy+BXmZWJaJ1qyO+7Nyg5pz7Cf+7eNLMxgDXU8DYKDOrhW+VGpDznQWteAf6d2wxcGG+Y+3zvZ4DtHbO/VzMMhfhZ2M2cc4V2jLpnNuLHys2ycxGAOuATvjxYSIicpA0QD7KnHPbnHMLIh/4rq8twWsXtGi9gQ8ulwPOzI4KHhVzyjKz4WY2LaL4FfhWmnPNrE7OIOxi1OlH/MzI8WZ2sZk1NbNTzWxgMcLR0cAoM2thZhcDg8hrgVmJ/0f/1qDMc/Hdbgdia/yh/09CwsM+uO4BXgxmHZ4JPA1MdM79bGYVzWx0MBsy2cxOBzrjg0uB9cIvrHqDmTU3s98AY/GtWwdiLNDM/OzQFsH3fWNwLifAPgy0Nz/zs11wv/PML9r6K0GYHAmMNLPrgutPMrObzOyPAGbWz8yuD76LY4Br8a1ePx1g/UVEJB+FrbKhIb6142j8QPqUiEfkuJ36+DE7ADjn1uDHRz0IrAeeoviuxbewPYJvbfkAOJOIcVKFeBnfkvY18AwwjiBsOec2AtcAv8eHlsH4gdkHYnW2H1d1UIJWscxOPkT0xM/qnA28ix9fdl1waRZQAz/QfAnwdnC+wPo6v4XPpfhtghYAo4F7Kf46YDnlrMDPIr0A+B64HT8TE4KZkM65efjfRTJ+yYrv8QPr1xdR9L342ZQD8ZtnTw3usyw4vw0/W/HzoP59gN45LZAiInLwbN+xzCIHzw7XNkBmZ+AHfecfCF4cdYEFOJd/0HmpZWZ/Bu4Hqjv9hRURKXM0ZkvKoh/wkwQSObCWo0R8q9uSMCpVUszsFvyMxI34ruN7gfEKWiIiZZO6EaXscW4bflZjXXyAKo7E4PopwftLs+b4bssf8GPaxuLHvomISBmkbkQpu/xm1D3x46s2U/Bm1PFAreDnFPyYKBERkcNGYUvKNr8pdQt8t2I5/Ay6LHy4SsDPBpwPLCkDLVoiIhKDFLYkNvj9BxvhV9bPGcu1DViFc+lFvVVERCRMClsiIiIiIdIAeREREZEQKWyJiIiIhEhhS0RERCREClsiIiIiIVLYEhEREQmRwpaIiIhIiBS2REREREKksCUiIiISIoUtERERkRApbImIiIiESGFLREREJEQKWyIiIiIhUtgSERERCZHCloiIiEiIFLZEREREQqSwJSIiIhIihS0RERGREClsiYiIiIRIYUtEREQkRApbIiIiIiFS2BIREREJkcKWiIiISIgUtkRERERCpLAlIiIiEiKFLREREZEQKWyJiIiIhEhhS0RERCREClsiIiIiIVLYEhEREQmRwpaIiIhIiBS2REREREKksCUiIiISIoUtERERkRApbImIiIiESGFLREREJEQKWyIiIiIhUtgSERERCZHCloiIiEiIFLZEREREQqSwJSIiIhIihS0RERGREClsiYiIiIRIYUtE/r/dOhYAAAAAGORvPYp9RREAI9kCABjJFgDASLYAAEayBQAwki0AgJFsAQCMZAsAYCRbAAAj2QIAGMkWAMBItgAARrIFADAKsFSnipPYuhgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlsAAAFBCAYAAABAYGoRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzs3Xl4VdX59vHvkxACYUYGEQizQYLgiAhoQRGkzmAdUCuKVkX71gG01iqgVFDRYtWiP4tjq6IWtYqiyCAKKiAo8zxDmGcIGdf7x9pJDjGBADmc5OT+XNe5SPbeZ+11AsrNWmuvx5xziIiIiEh4xES6AyIiIiLRTGFLREREJIwUtkRERETCSGFLREREJIwUtkRERETCSGFLREREJIwUtkSk2JlZCzN70cwWmtleM9tjZovM7FUzax9cs8rMnJmtCnNfGpvZoODVOZz3OlZBX13wmhzp/ohI8SgX6Q6ISHQxs1uAkUB8vlNJwas2cOVx7FJjYGDI95OP471FRDSyJSLFx8wuAP6FD1oOGAI0DL4/GfgLsCNiHQwTM6sQ6T6ISMmlsCUixWkoef9f+Ydz7lHn3DrnXLpzbqlzbihwe2FvDqb6cqbROhfheE8z+9bMtphZmpltNLMpZvZAcP4NYFLILQaGtDMopJ2LzexLM9tuZunBFOcLZlYrX/9ypz7N7Dwzm2ZmqcDLIdf0DvqwK+jTEjP7m5kl5Gurqpn9y8x2mNluM3sXqFvEn7OIlCKaRhSRYmFmdYB2IYeeKeg651xmMd3vHOADDv5HY93glQA8W8R2HgCG5zvcCLgHuMTM2jvnNuc7Xxv4CjhoRMvMXgjeF6oFfkSvm5md75xLNTMDPga6hFx3HfCbovRZREoXjWyJSHFpHPL1bufc+jDfrxN5/w87FygPNAAuA94FcM714eBAM9g5Z8FrkJk1xI/GAYzDh6wK+OAD0AT4awH3TgCmAE2BysDfgoX/OUHrDeDE4LoBwbGzgLuCr7uF9GsZ0BKoBywt8qcXkVJDI1siUlqtDPn6z8B3wCLgR+fcZ0Vs42IgLuTr1QVc062Q997inNsQfL3UzPqEnOsTvApq6znggpBjLzjnFgOY2ePA10XpuIiUHhrZEpHisirk66pmdlIxtl3QPww/Av4JpAFX4KctPwVSzOzFIrZbpwjXnFDAsc0hQeto2gptc10hX4tIlFDYEpFiEaxrmh5yaEBB15nZoUbU00K+Dl0P1bSA+znn3N1ADeAc4EbgCyAWuNvMzs259BD3C12L9deQKcbcFwWHqNTDtHVjIW3lrGnbGnJtg0K+FpEoobAlIsXpESA7+Pr/BU8RnmRmccFGp38BXj3E+0On8S6D3IXwV+W/0Mx+Y2YP4beUWAL8F/g+5JLE4NdtIcdOMbPQ/b++BHIW7D8QPJWYEDwp+Bszexl46JCfOE/o1OUTZtbRzOLNrGbQ7jvADcH50Cck7zGzJDM7EXi0iPcSkVLEnDvUP/pERI6Mmd2Gn96LK+SST5xzVwY7xzcCVjvnGgfvrY5fi1U9uHYvfgH6fvxic4AuzrnJZnYj8HYh99gHJDnn1gd7YK0FauW7JqedB4GnDvGRBjvnBgX9+1WfQ5nZP8lbBF+QW5xzbwRPI04EOuc7vzWkn9845/KfF5FSSCNbIlKsnHP/AtrgA9cS/JTbPmAxMAoYdoj37gR+i5+OTMWPSj0K/L2Ay38CXgcWAruALHxY+QzonPM0pHPuAHBtcP3+Au75dHDPL4L7ZQIbgWn4neffPILP3g8/nflN0KcM/DqsScCDwT1w/l+5VwKvBdftBcZQwAieiJR+GtkSERERCSONbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBgpbImIiIiEkcKWiIiISBiVi3QHRIqFWTzQAKgBlAfSgR3AWpxLj2TXRESkbFPYktLNrDpwCpCM//OcDmQBsfjQlYnZPGARzu2MWD9FRKTM0jSiFMrM3jCzz47TvRqbmTOzs47gTY2Aa4BWwFZgPbAF2B78uh7YavD9AzAsuD6qHc/fMxERKRqFrRLCzB4OwsaL+Y73NLMvzWxLcL5zhLpYbIolEPjgdAk+WG3Gj2YVJAtgL+wFLikLgUtEREoWha0SwMzaA38A5hRwuhIwDbj/uHaqJPNTh93xISutKG/JgIzg+u7B+0VERI4Lha0IM7NqwH+AW/ELug/inHvbOTcY+OII2uwcjIJdamY/m9kBM/vJzM4MueYEM3vXzNaZWaqZzTezWw7T7mQzG2lmz5rZ9mC07U9mFm9mL5nZTjNbY2Y3HaKNQcDNwCVBH/OP1jUys/Fmtt/MFpjZRfne36oOjC0P/6gA/2oO/SdCUcNTGpA1CH5rZl8Hn3t7MNJWLeQep5rZBDPbbWZ7zewXM+tyiM90sZl9a2Y7gva+NLNTQs7nTJH2Osxnu8TMFge/X1PM7LrgfY1DrulgZt8EbawPfj+qHqJvZmYPmtny4PPONbMb813zmJmtNrM0M9toZm8V8ecpIiJFoLAVef8HfOicm3S0DQRhYVUBp4YDDwFnASuAz8wsIThXAZgFXIpfXP488IqZXXiY290A7AHOAYYBI4CPgSXBfd4E/mVm9Qp5/3DgfeBroF7wmhZy/m/AP4C2wAzgPTOrHHzOesCUZNj9CvQfDo+mQ8Vr4K9pYIfpNwArYe8z8M8Y2Ae0A64COgCvhVz2DpASnD8NGAQcOESzlYKfQzugM7AL+NTMyue77lCfLREYA4wNzv8DeDr0zWZ2KvAV8L/gmp5B/0L7nt8QoC9wN35t21D87/MlQZu9gP5AP6AF/s/D9EO0JyIiR8o5p1eEXsDtwE9AXPD9ZODFQq6tBTigcwHnhgITQr7vHFx7Q8ixysBO4LZD9Oc94F8h378BfBby/WTg+5DvDb8Q/X8hx+LwTwRefYj7HNRucKxx0Oc7Qo7VD451Cr5/vBJMc3Cng8scXDYLrgPcULg/51j+F+BugaEOLrsSXigPqV9AmwJ+Xs2D73cDNx/D72sl/FqxTkfw2YYCC/O185fgmsbB928Bo/Jdc1pwTZ38P9ugH6nAefneMwL4PPj6fmBxzp9BvfTSSy+9iv+lrR8ixMySgCfxf9lmHEtbzrmHCzn1fcg1e81sLn50AzOLBf4MXIv/iz8ev1XC5MPcLnddmXPOmdlmYG7IsQwz2wHUOfJPcnD7wIbg15y2ztwP7eLgDOcDRq55foRsaRrEpEFMVcgsqPHV0LA2rL3Yh8Ic04Bs/M9mGfAcfnTuZmAC8F/n3KLCOmxmzYAn8KN9tfEjxjFA4hF8tpb40a5QP+b7/kyguZldG3r74Ndm+DVpoVrhRzDHmVnozysOWBV8/QHwJ2ClmX0JjMOH5yKthRMRkcNT2Iqcc/GjVfPNcmfAYoHzzexOoFKY/8LrDzyA/4t2Lv5pvSc5fEjKHwxdIceOdoo6t60gzBHSVkwdmPYKfLrPT9XlOtOP2jEXarwO17WAX06B1R3zQk1o5xw+XBZwCpxzg8zsP0AP/EL8gWZ2p3OusOm6z4B1wB347SYygQX48FrUz1YUMcC/gL8XcG59IdeDH91bU1BfnHNrg+B/IdAVeBb/ec9xzu07gr6JiEghFLYi52NgZr5jrwNL8aGnOHY9b49fq4WZVQJa46eiADoBnzrn3g7OG3AyQWgJs3R8sDxSs3bBTRfAziqwsaALzoJt78GuxrDhF2g+xa+jYi/UTIOYRrB2PnSbDrHt8t7WAR9MFuYccM4txf9e/MPMRgK3UcDaKDM7AT8q1c8F6+7M7AyO/L+tRcAV+Y61y/f9LCDZObesiG0uwD8U0Mg5N7Gwi5xzB/Brxcaa2TD8z7Yjfn2YiIgcI4WtCHF+N/ODgo2Z7QO2O+fmhRyriZ+OynnirrmZ7QQ2Ouc2BtcMBdo55/Ivbv+rmW3Bj+48hg857wTnlgDXmlkn/IagfwSaALOL71MWahXQIxhR2Ua+UapDeCkD7mwH/W6CfzeDXTPgxC+g0xh4LcmvTyIGsq70IXPFBqjwJNy8Gxo8DledCxvHQ2Y3eHSX2QP48j6vAGOcc8vMrCJ+Ef8HQT/r4oNp/im9HDvwP7/bzWwtfkr2GQqZxjyEl4H7zWw48Cr+oYU7gnM5U4BPAT+Y2ctBn/fgg95lzrk78rWHc25P0N7wIExPwa/daw9kO+f+z8z64P8/8CN+dPNa/KjX0iPsv4iIFEJPI5Z8l+MDUM7Tiq8G398Zck09/Jqd/P6MnxaaRfCkWcjU0BD8U2df4P8S3offguJ4eBU/ijQTv8C+Y1He5Jzb0BjOw2/fMLg3vPR/cGc5yKj166lMAE4KniI8Ceb8Ab5oCtv7waTd0DS4/6f4tW23Bm/JwgewN/ALxz8Kzhe4z5lzLhsfUNoA84CXgEcp4v5fIe2sBnrhf79/Ae4DBgenDwTXzAHOxy+4/ya4biiw6RBNP4p/mrI/MB8YH9xnZXB+J/5pxW+D/vcCejrnVv6qJREROSrmnDv8VVKqBPtWTQJqO+e2Rrg7xc/sXPzi7/wLwgF4EK5/Gt4t5N11gHnmR26S8VOrFfFTbvOA9a6E/EdhZn8CHgeql5Q+iYjIkdM0opRGC4FT8Yvcj2QEKR6/VmxxMI37LfCtmdXGB68rgXLmC1fPAzYdz5BjZnfjn0jcgp/qexR4Q0FLRKR0U9iS0se5nfhtCi6h6CV74vGjWmPxQSukObcFmGxm3+DXaLXG79+VGQSv+cE14dYcv7fWCfinG1/Gj2yJiEgppmlEKb18Uenu+HVW24Jf808jxuLDSyzwJX5tVBGaNgNOwgev1sB+/GjXfOfc9uL8GCIiEt00siWll3OrMXsfSMJPK5YDMmr7NVh18Jt3ZuL3EVucf0Tr0E07h9+7ar2ZfYV/IrQ10Dd4GjQneO0uzo8kIiLRRyNbEh18HcKGQPU+0O8N+Cf+Sbu1OFcce5YFt7EY/BYZycAp+PVV84AFzrm9xXUfERGJHgpbEnXMbJBzbtBxuE8sfsuN1vgNYTfgg9dC51xquO8vIiKlg6YRRY6Scy4LvznsEjOLw+9llgx0N7M1+OC1SHUGRUTKNoUtkWIQFBNfACwws3j8SFdr4LdmtgIfvJa6YpzSFBGR0kFhS6SYBSNZc4G5QfmflsAZwGVmtgwfvJY55460pI+IiJRCClsiYRSs3ZoNzA6KgZ+C37D0CjNbjA9eK4MpSRERiUIKWyLHSVCXciYw08yq4Nd3dQZ6mtlCfPBaHdRbFBGRKKGwJRIBzrk9wA/AD2ZWHR+8ugFVzGw+PnitU6keEZHST2FLJMKCOo1TgalmdgJ+Yf3lQPmQ4JWi4CUiUjopbImUIM65bcA3ZjYFvwt+MnA1QE6BbOfc5gh2UUREjpDClkgJFIxibQI2mdkkoB5+xOsGM0sD5uOD17YIdlNERIpAYUukhAuC1wZgg5mNx5clSgZuMbM95NVpLHLtRxEROX4UtkRKkSB4rQHWmNmXQCP8iNcfzGwbeXUa90SwmyIiEkJhS6SUCraIWAmsNLPPgab4Ea8uZraRvOC1P4LdFBEp8xS2RKJAsCnqUmCpmZUDmuNHvLqa2Try6jQeiGA3RUTKJIUtkSgTlAFaBCwys/L4Oo3JQA8zW4UPXotVp1FE5PhQ2BKJYkGgmgfMM7MKQBLQBrg0X53GjAh2U0QkqilsiZQRwRTiL8AvZpaAr9PYDl+ncQk+eC1XnUYRkeKlsCVSBgWL5n8CfjKzykAroBNwpZktwgevVarTKCJy7BS2RMo459xeYDow3cyq4dd3dQWqmdkCfPBao3JBIiJHR2FLRHI553YB04BpZlYTH7x+C1QMqdO4QcFLRKToFLZEpEDOue3At8C3ZlYbv5VETyAmJHhtUvASETk0hS0ROSzn3BZgkplNBk7Ej3hdB2SGFMjeGsEuioiUWApbIlJkwShWCpBiZhOA+vgRr5vNbB95dRp3RLCbIiIlisKWiByVIHitA9aZ2VdAIn7E6zYz20le8NodwW6KiEScwpaIHLNgi4hVwCoz+wJogh/xOt/MNpNXp3Ff5HopIhIZClsiUqyC4LUcWG5mY4Fm+BGvC81sAz54LXTOpUawmyIix43CloiETVCncTGw2MzigBb4Ea/uZraavDqNaRHspohIWClsichxEdRfXAAsMLN4fJ3G1sAlZrYCH7yWqE6jiEQbhS0ROe6Ckaw5wBwzqwi0BM4ALjOzpeTVacyMYDdFRIqFwpaIRFSwdms2MNvMKuHrNHbA12lcjA9eK1UgW0RKK4UtESkxgqcVZwAzzKwqPnh1Bq4ys4Xk1WlUgWwRKTUUtkSkRAr25/oB+MHMauCfaLwYqBRSIHudygWJSEmnsCUiJV6wI/13wHdmVgsfvK4AyoXUadyo4CUiJZHCloiUKkENxm/MbApQB/9E4zVAdk7wcs5tjmQfRURCKWyJSKkUjGJtAjaZ2USgHj543WhmB8grF7Qtgt0UEVHYEpHSLwheG4ANZjYeaIgPXreY2W7ygteuCHZTRMoohS0RiSpB8FoDrDGzcUAjfPC6w8y2kVencU8EuykiZYjClohErWCLiJXASjP7HGiKD15dzCyFvDqN+yPYTRGJcgpbIlImBJuiLgWWmlk5fJ3GZKCbma3FB69FzrkDEeymiEQhhS0RKXOCMkALgYVmVh44GT/i1cPMVpJXpzE9gt0UkSihsCUiZVoQqOYB88ysAr5OY1t8ncZlwbmlqtMoIkdLYUtEJBBMIf4M/GxmCcApQDvgipA6jStUp1FEjoTClohIAYJF8z8BP5lZFXydxvPxdRoX4YPXKtVpFJHDUdgSETmMYJuIH4EfzawafmF9V6BqSJ3GtSoXJCIFUdgSETkCwcao04BpZlYTv7D+UqBCSJ3GDQpeIpJDYUtE5Cg557YDU4ApZlYHP+LVE4gxs3n44LVZwUukbDP9P0BKiZrAQ0BvfA28mMIudM6ZmRX2BzsNmAGMAMYUdydFzMyAE/EjXq2BdCCnQPbWSPZNRCJDYUtKgxrAdCARKF9Mbe4HHgOeLab2RH4lCF4N8CNeycA+8uo07ohk30Tk+FHYktLgr8ErvpjbTQPqAipOLGFnZjH4fzC0xj/ZuJ28Oo27I9k3EQkvhS0pDRbjd/gubnuAPwDvhaFtkUKZWSzQBD/a1RLYTF7w2hfJvolI8dMCeSkN6oWp3Qr4kQaR4yrYFHUZsMzMxgLN8CNeF5rZevLqNKZGsJsiUkwUtqQ0KHQx/DGKRf8NSIQFZYAWA4vNLA4/ipsMXGxmq/HBa7FzLi2C3RSRY6C/aCQqZWRksHWrf/CrVq1axMXFRbhHIofnnMvAP7k438zigST8iNclZracvDqNGRHspogcoXCNGIhExPvvv0/79u1JSEigfv361K9fn4SEBNq3b8/o0aMj3T2RInPOpTnn5jjn3sFvVbIMOBN4wMx6mVmSmekfzCKlgP5Dlajx8ssvc++999KnTx8GDBhA3bp1Adi0aRPjx4/npptuYtu2bfTr1y/CPRU5MsHarVnALDOrjC+Q3QG4MqRO40rVaRQpmfQ0opQGe4FKh7uoadOmPPLII/Tt27fA86+99hpDhgxhxYoVOYeygUeBJ4uroyLHk5lVxa/vag1UBxbig9caBS+RkkMjWxI1UlJS6NixY6HnO3bsSEpKynHskUh4BftzfQ98b2Y1CBbWA5VC6jSuV7kgkcjSmi2JGq1bt2bkyJGFnn/ppZdITk4+jj0SOX6cczucc985514G3gQOAFcCfzKzi8ysXrCjvYgcZ5pGlNKgSNOIU6dOpUePHtSrV4+LLrrooDVbX331FSkpKXzxxRd06tQp5y2aRpSoFoSrOuTVaczGj3bNc85tiWTfRMoShS0pDYoUtgDWrVvHyy+/zNSpU9m4cSMAJ554Ih06dOCuu+6iQYMGoZcrbEmZEQSvk/ChKxlIJa9A9vZI9k0k2ilsSWlQ5LB1hBS2pEwKglcieQWyd5FXIFu1QkWKmRbIS9To0qULDRs25K233op0V0RKtGDB/GpgtZmNAxrjR7zuNLMt5NVp3Bu5XopED4UtiRoxMTFo/a/IkQm2iFgBrAip05gMXGBmKfjgtdA5tz+C3RQp1TSNKKWBphFFjrOgTmNz/IhXc2ANeXUaD0SybyKljUa2RETkV4L6iwuBhWZWHl+nMRn4rZmtxAevJc659Ah2U6RUUNiSqLJv3z7eeecdpk6dSkpKCjExMTRt2pQrrriCbt26Rbp7IqVSEKjmAnPNrALQEjgNuMzMluKD1zLnXGYEuylSYilsSdRYunQpXbt2Zdu2bZx00kmsX7+e7OxssrKyuPbaa+nSpQujR48mLi4u0l0VKbWCKcSfgZ/NLAFoBbQHrjCzJfjgtcI5lxXBboqUKNpBXqLGH//4R9q3b8/mzZtZsmQJmzZtomvXrpx99tksX76cDRs28MQTT0S6myJRwzm33zk30zn3BvBPYANwPvCAmV1mZk3MTH/PSJmnBfJSGhRpgXzlypWZPn06rVq1yj02Y8YMevbsydq1a5k4cSK33XabClGLhJmZVSevQHYVYAF+xGut6jRKWaRpRIkaNWrUYMeOHQcd27lzJxUrVgSgefPmbNiwIRJdEylTnHM7ganAVDM7AR+8LgUqmNk8fPBKUfCSskJhS6LGVVddxZ133smIESNISkpi4cKF3HPPPfTo0QPwNRJPPPHECPdSpGxxzm0DpgBTzCynTuPVAGY2Hx+8Nit4STRT2JKoMXToUPr27Uu3bt1wzmFm9OrViyef9LOEqamp9O/fP8K9FCm7nHObgYlmNgk4ER+8egPpOSNeQTgTiSpasyWlwRFtarp161ZWr15NYmIitWvXPtSlWrMlEmFBncYG5BXI3kNegeydkeybSHFR2JLSQDvIi5QBwZOLifjg1QrYTl6dxt2R7JvIsdAjuRI1pkyZwvTp03O/37p1KzfccAO1atWifv36PPTQQ2Rmas9FkZLKOZftnFvlnPsMeBaYjJ9uvMvM+pjZWWYWjn94iYSV1mxJ1LjrrrsYMmRI7vd/+MMfWLx4MUOHDiUtLY2hQ4dSqVIlHnvssQj2UkSKItgUdRmwzMzK4eszJgNdzWw9eQWyi6dOo1k8fjqzBlAeSAd2AGtRSSI5RppGlNKgSNOICQkJzJ8/nyZNmpCenk5CQgILFizg5JNPBmDcuHHcc889LFu2LOctmkYUKWWCOo0t8FONTYFV5NVpTDuKBqsDp+CDXDl8yMoCYvGhKzNofxFaQyZHSSNbEjUqV67MunXraNKkCdu2bSMmJoYWLVrknm/ZsiXr16+PYA9F5FgFdRrnA/PNj0a1BE4FLjWz5fhgtDQopI2ZrQJedM4N/1VjZo2A7vhwtTX4Nb9YfBBrg9mXOLe6uD6LmXUGJgG1nXNbi6tdKXm0ZkuixiWXXMLgwYPJyMigXr16NGvWjK+//jr3/Lhx42jZsmUEeygioczsYTNzZvZivuNmZoPMbIOZpZrZZDNLzv9+51yac+4X59w7+LD0BnAWvlxQTzM7+RA3bwRcgl+Ev5mCgxbB8c3BdZcE7xM5IhrZkqgxdOhQOnXqRKtWrejVqxddu3blyiuvpGfPnqSmpvL555/z0UcfRbqbIgKYWXvgD8CcAk4/CDwA9AEWA48B480syTm3p5AmM4Bs59xbZlYZ/zRjR6AqkGxmzYCVzrnsYOqwOz5EFXXqMS24vjtm72tKUY6ERrYkapx44onMmjWL6667jrFjx/Laa6+RnZ3N1KlTSUhI4Pvvv6d79+6R7qZImWdm1YD/ALfiF6GHnjPgXmCYc+6/zrl5wM34Gou9C2mvM/A6UMnMHH6vrt86514Pvk4HXgP2m9nW02FEth+xSgOYBwnJcHcFeDsORteFJ5/xC/LzS8O/L8nMVgWjb/82s71mttHMDto12czuN7M5ZrbPzNab2b+CupGF/Vz6BG31MLNFZrbfzP5nZtXM7GozW2pmu8zsbTOreJgfs5QgWiAvpYH22RKJImY2GljlnHvIzCbjNzC9JzjXFFgOtHPOzQh5z1hgq3Pu5uD7yQDOuc7Bovm78P8tNwvestc5tzdYs1UFGAhMKw990uGP18JnF8LMNrD6MngwHvbdDe8mwt4X4IKZcPkXcGfXfGEQv4arlsFfgZrAU8AHQBfgBeA659yYoI/34kfuVgCNgvO/OOduCs53JmTNlpn1AV4FvgEewi/Q/y++kHcq8AhwAjAGGOKce/YofwvkONM0okSljIwMtm71601r1apFXFxchHskIgBmdjt+1OjGQi7JKWC6Kd/xTUD9kO/X5HzhnEs3s13+S7exgDa/cs69GHRgVw24bi2kxkP6MOi5HVr8Hf72G9iSDDt7w39qQLuh0KWrDzahsoBy5aBcJvzonPtbcHyJmZ0N3E/wHufciJD3rTKzB4FPzOxm51x2IZ+/HHC3c26x7669A9wH1M1ZRG9mn+DDncJWKaFpRIkq77//Pu3btychIYH69etTv359EhISaN++PaNHj45090TKNDNLwo8+9c55WvBoOed+75z7fREvD10XVqMy7NgHFX8P83bD1iyIvQ8ePgNei4ePysEHu6DRxrzgl19GrP/78/t8x7/HrxUDwMwuMLPxZrbOzPbgQ1h5Cm8XIC0naAU2ARvzPa24CahzmM8sJYhGtiRqvPzyy9x777306dOHAQMGULduXQA2bdrE+PHjuemmm9i2bRv9+vWLcE9FyqxzgVr4bRtyjsUC55vZnfjlAjkjU3XQadkCAAAgAElEQVQJGb0Kvi9o1KooQoNdeSA7GwwgC+LiYc+t8N9UqBYPe0+DmVUhrSHsL6S9LIL3F8b8U4tj8dOCjwHbgDOAd4M+FCZ/mQuXr/85xzRYUooobEnUePrpp3nppZfo27fvr8716tWLdu3aMWTIEIUtkcj5GJiZ79jrwFL8iFc6sBIfqi4CZgCYWQXgPGDAIdpOxwe3w0kHYrKgynD4TRWomwZVG8LKP8LcyoVvAREqFh942uc73h5YGHx9Fj5U3Rfsho+ZXVqEtiUKKWxJ1EhJSaFjx46Fnu/YsSMpKSnHsUciEsr57RIO2jLBzPYB24OnDnOOjQD+YmaLgCX4xeh7gXdCrnkraDNnKnEVUMHMLgJmA/udc/tDri8PtOgJHYCa2bCvFay6BaaeAvWfgr4p8MZ5sG4RVB8HZ14APz/hF6fnF5flH7Bpb2YPAx8CnYHfAzcE1yzFjz7da2Zj8EHs3iP/qUk00DCkRI3WrVszcuTIQs+/9NJLJCf/al9EESl5ngb+DryEHwmrB3TLt8dWYvACwDk3DXgZP023BXgwqKkYh99h/n7gjFnwA7AjHlJ+C6tPgMxvYXBjmPMvuOdaGDkUHtoI9ZP9Rqb5xQKZmX667zmgDT7cDQEec859GPRnDvCn4L4LgNuA/gW0J2WAtn6Q0qBIWz9MnTqVHj16UK9ePS666KKD1mx99dVXpKSk8MUXX9CpU6ect2jrB5EoY2axQBN87cQk/GLynKLV+4KLzsUvZN98FLeoA8wzGE1hZYBE8tE0okSNjh07Mn/+fF555RWmTp3KhAkTAL/Z6e9+9zvuuusuGjRoEOFeikhxM7MY/D5WrfFFpbfjA9aEQnacX4gf7Yqn6DvIE1wfi9/VXqTINLIlpYE2NRWRgwQ7zTfAB6xk/E7x84D5riildPJqIxa1ZE88flRrLM6tPmSBa5F8NLIlIiKlQhCw6pEXsNLxAet159y2I2rMudX4Xem7459A3EbBTyLG4ndtjyUIWv7trvHRfQopixS2JGoMHjyYO++8M3etVn7PP/88CQkJ3H777ce5ZyJyLMysDj5gtQ4OzcM/mbjZHcv0jA9c7+PXdp2K/zsxAx+6YvGL6zOBucBiFZ+Wo6VpRCkNijSNGBMTw+zZs2nbtm2B55966im+/PJLJk6cmHNI04giJZSZnYAfvWqNn8Kbjw9ZKccUsAq/YXmgIVCdvLVcO4G1OJde7PeTMkUjWxI1zIz58+eTmZl/A2avYsWKzJ49+zj3SkSKysyqkxewquAD1mfA2rAErFA+UC0P6z2kzNLIlpQGRRrZio2NxTlHSBmQX3HOkZ2dW/9VI1siEWZmVfDbMLTGr41aiB/BWn2IYs0ipYpGtiSqjB07VhuXipRwZpZAXsA6EVgETAFW5JS2EYkmClsSVU466SQSExMPf6GIHFdBfcOW+IDVEF/O5gdgmXOu4Ll/kSihsCVR4/XXX1fQEilBgnqESfiA1RhfZPpn4H2nRedShmjNlpQG2tRUpJQwszigOT5gNQfW4NdgLXbOHYhk30QiRSNbEjVmz55NWloa7du3j3RXRMqUoB5hM3zAOhlIwQessc65/ZHsm0hJoLAlUaNfv35ce+21B4WtESNG8Pzzz+d+75xj1apVEeidSHQJ6hE2xgeslsBWfMD6yjm3N4JdEylxFLYkaixevJguXbocdOz0008nLS2N/v37s3fvXgYNGhSZzolEgaBcTiI+YLUCduED1ivOuV2R7JtISaawJVEjLS2NatWqHXTshBNOID09nfvvv58tW7YobIkcoSBgnURePcJUfMAa5ZzbHsm+iZQWClsSNRo1asT06dNp3Lhx7rHp06fTsGFDwE8hisjhBQGrLnkBKxsfsN52zm2JZN9ESiOFLYka119/Pffffz/x8fGcfvrp/Pjjjzz00EPcd999udccand5kbLOzGqRV/C5HD5gvQ9sDHu5HJEoprAlUWPAgAHMnj2bnj175pbt6du3LwMGDACgcuXKPPTQQxHupUjJYmY1yKtHWAlfj/BjYL0Clkjx0D5bUhoc0T5b69atIyUlhebNm1OjRo1DXap9tqRMMrOq5AWs6sACfMhao3qEIsVPI1sSdRo0aECDBg0i3Q2REsXMKpFXj7AOvh7hRGClApZIeClsSVQbP348X375JfHx8VxzzTW0bds20l0SOW7MrCJwCj5gnQQsAaYBy1WPUOT4iYl0B0SKy4UXXsjw4cNzvx8xYgQ9evTg+++/Z9y4cZxzzjlMmDAhgj0UAaAi8BvgL8C/gb8DVwHF8vSGmcWbWRsz6w3ciy+ZMxN41jk3xjm3WEFL5PjSmi0pDYq0ZqtmzZpMmTKF1q1bA1CjRg1efPFFbrjhBgAef/xxvvzyS6ZOnZrzFq3ZkuOhGtAR6AJcjC/MvB9IAOKCa/bhR5x6AFlHeoOgHuHJ+BGspsBq8uoRph1j/0XkGClsSWlQpLBVqVIlZsyYQatWrdi4cSMNGzbkwIEDxMbGArBixQpOO+00du/enfMWhS0Jh7rAeUBX4CKgAX4j0MpA7CHetw94CHipKDcxs3Lk1SNsAazHB6xFzrnUo+28iBQ/rdmSqNG2bVv++9//0qpVK2rUqEGFChXYuHEj9evXB2D9+vVUrlw5wr2UKGP4+oDn44PVBUBNIA2oQt7UYPkitFUJGAT8EyjwX8FBwecm+ICVBGzGB6xxzrl9R/kZRCTMFLYkagwaNIjLL7+cjIwMevfuzaOPPpr764EDB3jwwQf5/e9/H+luSukWg3+i7zz8lF8nIB4/Shqa5OOPsv2K+OnAxTkHgoLPOfUITwF24APWROfc7oIaEZGSRdOIUhoUeZ+tTz/9lD/+8Y+sWbMGyNsxvkKFCtx9990MGzaMmJjc50I0jSiHEwecgQ9XvwXa4UedYvBrrorbPuBeMxuFn35MDl778AFrvnNuRxjuKyJhpLAlpcERbWrqnGPWrFmsWLGC7Oxs6tWrx5lnnkmlSr9qQmFL8ksAzsE/LfhboA1+SrA8UOF4dGDNmjXfNWrU6L9AJjAXH7C2Ho97i0h4KGxJabAT/0RXccsAHgaeDUPbUjpUxz8peAHQHT+Fl/9JwWKTkZHBrFmzqFKlCq1atSrwmvT09F3x8fFJwGaVyxGJDlqzJaXBUuCsMLSbit9FW8qOevgpwQvxC9rr8+snBYs92M+dO5f+/fszbdo0TjjhBGrXrk2XLl14+umnc+t45ihfvny8c648hSySF5HSR5uaSmkwCj/aUNxiga/D0K6UDIbfGqEP8C6QAqwEXgVuxz/VVx4frg61JUORbd68mTlz5pCenp57bM+ePQwaNIiEhATmzp3LzJkzufHGGxk+fDjz5s07KGgFMvCBUESihKYRpTSIAz7Cr6Mpjr0b0vHrYXoCXxZDe1IyxOAXk5+Pf1KwIz5M5X9SsFgdOHCAZ555hlGjRrFr1y4SExNJTk5m5MiRVKtWjc2bN3PiiScyceJEOnfuDMCGDRtITExk2rRptGvXLn+TDngTuCVcfRaR40vTiFIaZABXApcBN+HX1RS6nmbLli0n1K5de1shp/cA44C3CXm8XkqlOOBM/CjQJcDZ+GAVricF2bVrF9WqHTzLOHr0aD744AP+9re/cfHFF/PLL7/QvXt3WrduzUMPPUSlSpWoXbs2//73v2nVqhWZmZk88sgj3H///Zx55pkF3cbwa8hEJEpoZEuijpkNcs4NinQ/pNglAO2BzviRq1PxTwrGc/T7Wh1SSkoKH374Ie+88w4bNmygZcuWXH/99VxzzTUkJPg8d/bZZ5OUlMRrr71G+fJ+79JzzjmHM844g5EjRwIwadIkxowZw5tvvklGRgY1a9YkMTGRyy+/nIcffrigW6fjC0cX9o8GESlFtGZLREqqGvjRzOeABfinUscAf8Y/MBEPVKWYglZ6ejpTp05l1apVAGRnZ/Pcc8/xwQcfcMUVVzB69GiSkpK47777eOONN3Lf17p1a2bNmsWCBQsAeO+996hVqxZ333137jUdO3Zk//799OrVi1WrVvHzzz/Tt29fHn/8cT755JOCunMAv2GqiEQD55xeekXVCxgU6T7odVSvk5xz1zrn/uWcW+mcS3PO7XTOZbowmjBhgjv33HNdxYoVXWJionvzzTddRkaGS0tLc88//7z77LPPcq/Nyspyffv2deedd17useXLl7tHH33UJScnu9jYWFe5cmV3yimnuH79+rnU1FTnnHNz5sxxNWvWdBs2bDjo3k2bNnWPP/64y8z81UfMcs497yL/e6KXXnoVw0trtkQkEgxojl9v1Q3ogn8q8GhqChbJ+vXrycrKIjExkaysLGJjY1m0aBFPPfUUHTp04D//+Q8nnHACqampmBnly5fn//2//3dQG1lZWTjnqFKlSu6xpk2bcs455zB27FjGjh3L2WefzS+//MKNN95IhQoVGDp0KKtXrwb8mq969eoBsHLlSrZu3UpCQkJusfQQMfh9v0QkCmgaUUSOh1igLXAP8Dm+vt/PwPPAtUAd8qYFf7UXwtFISUnhhRdeoGvXrtSoUYOGDRvy3nvvAeSWbPr444/ZvXs3w4cPp0mTJuzevZuaNWseFH6ys7MBPwuwf/9+Pv/8c2677baDzg0cOJDevXvTvXt3KlWqRJcuXejTpw8zZsxg27ZtnHXWWZx88sn07NmT9957jw8//JA77riD888/n2uvvbawj9CUMD5FKSLHj0a2RCQcyuOfFPwNfjH72fjtNsrhiy0XuzVr1lClShVq1KjBunXr6Nu3L+PHj2fgwIE8+OCDdOrUKXdRu5lx4MABPvvsM3r27MnAgQMZNWoU1apVIykpiT//+c+5WzLExMSQmZlJuXLl+POf/0xSUhIXXXRR7rlt27axd+9eUlNT/QcPFskvW7aM/fv3U6tWLeLi4njppZd44403GDx4MGZGt27duP3222nQoEFhHykV/0CA9oITKeUUtkSkOHUEhgDnEuYnBcGPTD333HPMmjWL1q1b88QTT3DRRRdRp04dTj/9dFJTUxk4cCAAmZmZue/Lzs6mQoUKbNiwgTFjxtCgQQNGjRpFzZo1ufvuu+nfvz/Dhw+nXbt2pKenU758eSZNmsSnn37KqFGjqFy5MhkZGcTFxVGtWjV69erFkCFDqFKlCp06deKTTz5hyZIlDBgwgLg4v0vJGWecQevWrYmNjS1o2rAg8UBrFLZESj1NI4pIcXkS+Aq/NUOxPym4ZcuWg4799NNPDB8+nA4dOjB58mTeffddkpKSyM7Opnz58jRr1owlS5Zw/fXXc95553HhhRfyzDPPsHLlytxpxI4dO/Ljjz/SpUsXunfvztlnn81rr72GmfHhhx8CfqQqNTWVRx99lO7du9O9u19KlROiypUrx4ABA+jfvz9vv/02l156KZMnT+ZPf/oTV1999UF9Ll++fFGDFvg9w9IPe5WIlHgKWyJSHM4A7qUYNxNdvXo1Dz74IM2aNaNp06a8+uqrZGRk5J6/9957ad++PcOGDeOss86ibt26JCYm5gap5s2b07ZtW7Kysrjlllu48MILefbZZ7n11ltZu3Yt4PfDAmjSpEluu40bN6Zx48asWLEi99hbb72VO0o2fvx4HnzwQdq0acNnn30GQPXq1RkyZAhffPEFKSkpTJkyhT59+uROKR6lLGDZsTQgIiWDwpaIFIceFNOTgzmPSg8dOpS5c+cycOBAxo4dy2WXXZYbthYvXkxqaiqtWrVi+PDhNG3alEsvvZSnn36a9evXA9CyZUsGDhzIyy+/zK233spjjz3GxIkTWbNmDe+++y4Al19+ORUrVuSXX345qA9Tpkyhffv2ZGdnc+DAAR5++GFmz55NixYtuPrqq5k3bx433XQTHTt2POh9tWvXLo4fAfgp2HnA+OJqUEQiR2u2RKQ41OMIijnnLDjPzzmHmTFy5Ei+//57xo4dm7uAPOccwMaNG4mNjeXjjz8mLi6OESNGsHz5ckaMGMEPP/zAmDFjqFu3bu42CzlatWpFw4YNWbZsGWlpaSQmJtKvXz+effZZKleuTNeuXRkzZgy1atWie/fuxMTEsHv3bh555BESExM5//zzqVu37jH8mA5rL/7/y/8BHsDXSRSRUk5hS0SKwyr8+qICR7eysrL49NNP+fvf/87atWu56qqruPHGGzn99NMPClE5v/788880b96cnTt30rt3bzIyMujQoQM333wzbdq0ISkpiZUrVzJ//nwmTJiQOx3YsmVLLrnkElatWkXjxo0P6kN2djYxMTEsX76cTp06ER/vl5MNHjyY6tWrM3r0aB5++GGaN2/OE088wamnngpAnTp1eOCBB4r/J+aD1F78z2wXMAm/5m0csCEcNxSRyFBtRIk6qo0YEZ2AsfhF8blygtR3333HPffcQ7du3Wjbti0jR45k8+bNfPTRRyQnJx8UuPbs2UO/fv3YuHEjcXFxnHvuuTRr1oxnn32WXbt28cMPP1CrVi3OO+88VqxYwaJFi3I3Gd28eTOtWrVi5MiR/O53v2PXrl3s3buX+vXrs3fvXoYNG8b777/PqFGjOO+88w66b0pKCrVr1y5wxK2YZOMLoVfEh6kJ+GnCb1G4EolqWrMlIsVhBlAh/0EzIyMjg3feeYfs7GyGDRvGDTfcwOeff07VqlV56qmncq/LUaVKFbKzs5kwYULunle9e/dm3Lhx7Nu3j1dffRWAbt26ER8fz4wZM3Lfu2rVKqpWrZq759XMmTO59dZbOf/882nUqBGffPIJjz/+OOedd96v7luvXr3iDlqZwG4gA1iI38D1JvyUaxPgNmA0CloiUU9hS0SKQxo+UPxKbGws3333Hddcc03uk4JVq1ald+/e/Pjjj7lP/Tnncndkb9u2LQCdO3cmLi6OzMxMateuzQUXXMCUKVMA6NmzJy1atOC+++5j9uzZbN++nbfffpsmTZrQo0cPAE4//XQ6duzIddddxzfffMPcuXO57rrrwvkz2B38OhMYClyFL6jdCrgf+BTYHq4OiEjJpDVbIlJcvsBvwnnQQvmYmBgOHDjA/v37c2sSgg9CMTExfPvttzRt2pSsrKzckaYOHTqQlJTEpEmTuOKKK3JHyHKeVARITk7m+eefZ8CAAVx33XWsW7eOU089lWHDhuU+FVizZk0ee+yxcH3e/fipwRj8yN7nwBRgFtofS0RCKGyJSHGZBNyFLyh9kNNOO41ffvnloLDVoEEDGjRowJw5cwAOmsI744wzuOaaa3jmmWe48MIL6d69OxMnTmTmzJkMHTo097qWLVsyZswYZs+eTePGjalTp044P99efLBKB6biw+UUYD4+dImIFEhhS0SKy/cUsqlpt27dePjhh0lJSaFRo0aAXyOVmZmZO3W4a9cuVqxYwemnn05CQgKDBw9m5cqV/PWvf+WOO+4gIyODO++8k6uuuuqgtuPi4nLrGBYjh1/MHo8vmj0Z+BK/mH0F2pJBRI6AnkaUqKOnESNqCdAi/8Ht27dTv359nnvuOe666y7Al+CpVasWI0eO5IYbbuDuu+/mf//7H19//TVJSUm5TwpOnz6djIwM2rdvfySlbo5UFrAPv8h/Pb4e4df4cJUSrpuKSNmgkS0RKU5fAs3I9/BNzZo1uf322xk2bBhmRq9evXjhhRdo3Lgxbdq0AeDUU09lz549uYEqZ/1WGEatwD8huB+/DcPSoN8TgWn4kSwRkWKjkS2JOhrZiqirgDfIt98WwI4dO3jyySeZMGECS5YsoUaNGjz33HP87ne/Ox79Sgte8cAc/GL2b4Af8aFLRCRsFLYk6ihsRVRt59w6Myu0TuKiRYuoVKkSDRs2DGc/Qp8U/JG8JwVn40e1RESOG00jisgxM7NqQDLQeteuXalVq1YtNGy1bNkyHF3IeVIwDfgO/6Tgt8AC9KSgiESYwpaIHBUzq4zfrLM1UBu/qenXlStX/gT4fRhvHfqk4Db8k4Jf4UeuVqEnBUWkhFHYEpEiM7ME4BT8KNZJ+KcPvwOWO+eygsu+wq/dqlJMt83Cj1xVBNbh6wl+Hdx3YzHdQ0QkbBS2ROSQzKwCkIQfwUoEluF3TF/mnCto/dM3QKHTiEWQAaTiw9ViYBx+w9RpwM5jaFdEJCIUtkTkV4IF7ifjR7Ca4qfn5gAfOOcOV4pmHTAPOJ2i1V89gN+VPR74Gb/eajIwHR+6RERKNYUtEQHAzMoBzfEjWM3JC02fOOcOHGFzt+CfAqxYwLn95K2r+hEYi1/MPhvIPPKei4iUbApbImWYmcXiR66S8VOFm/AB6wvn3L5jaHoucAEwKmjXgO34moLj8IvZF6EnBUWkDFDYEiljzCwGaIQfwToF/0TffGCCc25PMd7qB3yIM/wU4ZGOjomIRAWFLZEywHztm4b48JOM3zphHvB/zrlwLzp3KGiJSBmmsCUSpYKAVQ8/gpWMX4Q+D3jdObctkn0TESlLFLZEooyZ1cEHrNbBoXnAf5xzmyPXKxGRskthSyQKmNkJ5AWs8vg1WB8CKU4FUEVEIkphS6SUMrPqBPUI8bu1zwf+B6xTwBIRKTkUtkRKETOrQt4i9xPw9Qi/AlY757SNgohICaSwJVLCmVkl/BYNrYET8ftTfQOsDKlHKCIiJZTClkgJFNQjzCn43ABfj/AHfD1C7bIuIlKKKGyJlBBBPcKcgs+NgRX4EjbvF6EeoYiIlFAKWyIRZGZxQAv8CFZzYA1+q4aPjqIeoYiIlEAKWyLHWVCPsBl+BOtkYAP+ScKxzrn9keybiIgUP4UtkeMgqEfYmLx6hFvwI1hfOef2RrBrIiISZgpbImESlMtJxAesVsBO/AjWy865XZHsm4iIHD8KWyLFKAhYJ5FXjzAVP4I1yjm3PZJ9ExGRyFDYEjlGQcCqS17AysYHrLedc1si2TcREYk8hS2Ro2RmtcirR1gOH7BGA5tULkdERHIobIkcATOrQd4IViX8GqyPgfUKWCIiUhCFLYkOZvH4ndZr3AwNMTsL2AGs5Rg3BDWzquQVfK4OLAC+ANYoYImIyOEobEnpZladvLI25YD0ZKgJtAHKA5mYzQMW4dzOojdrlfFPECYDdfD1CCfi6xGq4LOIiBSZwlYZZGaTgXnOuXsi3Zf8zKwxsBI42zk38zAXNwK6A1nA1uBXtvgnAHMWpsfiA1MbM3sF+J1z7sNC7l2RvILPJwFLgKnA8tJW8NnM3gBqOecujXRfRETKuphId0AOZmYPm5kzsxdDjsWZ2VNmNsfM9plZipm9Y2aJh2mrc9BWrfD3/DjzQesSYDuwmSBoFSArOL8d4Aw46GdhZvFm1sbMegP34kvmzASedc6Ncc4tKW1BS0REShaNbJUgZtYe+AMwJ9+pBOAM4G/Az0A14FlgnJm1cc5lHteORpqfOuyOD1FpRXxXGkBnOG2aWa2OeVs1NAVWAXOB/zrnitqeiIhIkWhkq4Qws2rAf4Bb8Qu7cznndjnnLnLOjXbOLXbOTQfuwE95nVJIe42BScG3W4IRrjdCLokxsyfNbKuZbTaz4UFJmZz3lw9G09aZ2X4zm2Fm3Q/zGSab2ctm9ryZ7Qhez+Rr98agrT3BfT8ws/qHaDNndK6Hmf1kZqk1YPIUqPoItKgB/ygH7zeER6dDlcLaSQ3+rG+Gxq/AYHx4zQYuAl4DvgFeCX4fcu59qplNMLPdZrbXzH4xsy6H6OvFZvZt8Lm3m9mXZnZKyPnGwWfpZWbjg5/rAjO7KF87l5jZYjM7YGZTzOy64H2NQ67pYGbfBG2sN7ORwUL+wvpmZvagmS03s1Qzm2tmN+a75jEzW21maWa20czeKqw9EREpOoWtkuP/gA+dc5MOe6WX8xdrbjAzszfMbFXw7VqgV/B1MlAP+FPI+28AMoEOwD34KbRrQ86/DvwG6I0fAXoT+NTM2h6mXzfg/1ydiw+EfwjazlEeGAi0BS7FT+u9e5g2wQekextAp1iofTX0fQ2uGwAvDoW/bIfE2+H60DekQcw3UO9FOGcIXAVwADaNgAXXwqfA88BuoB3+fAd88MrxDpASnD8NGOSbKFQlYERwfWdgF/5nVj7fdX8D/hH8DGYA7wUL8gmmhscAY4Pz/wCeDn2zmZ0KfAX8L7imZ9C/0L7nNwToC9yNX/g/FB8uLwna7AX0B/oBLfC/N9MP0Z6IiBSV+//t3X/InWUdx/H3dz9cMvzVdKiZrTaYmMFK2/JHSygaolthUlTU2tA2KpJi/dFqLBRbhpWI2qKFQ7IfCKmxomGDgZQZlGhrP9pgbG6mmDUWtZqtqz++19nuTmdP59m8nI+8X3Cz3edc5z73uc8ens++9/e671JcTvAC3AD8BphY1zcCd44w/iSycfvHfY+vAjZ01q8ECtko3R23EXi077GHgTX179PJqs/5fWMeBO4eYb82kk3l0Xnsi8CeEV5zQd3H8+r6tLp+Sd9nmFdKocD0D8MPgPI1uLHA/ALzL4PvnQq7DsKCT8NDd8NNX4B7boE7fgjLdsP7gbIIVhVYehEsJ8PQKQOO14y6vh9YeBzf62SyZ+yKvs+2pDPmNfWx3phVwJa+7SyvY6bV9XvJ2/90x8yqY6bW9bXAus5+HADe3vea24Gf1r9/FtjW+zfo4uLi4vLiLfZsnWARMRP4MvnL9oUhxk8Avkte72lB97lSyudH8db9fWFPk5c4gDzFFsDmvBPNYZPIyx+M5FellO61px4Fbo6IU0sp+yPiLWRlaxZ5iYbeG5wP7Blif8+YUpvdF8Ou3pOvhn3/hNMehyn74ZzT4BfvhvXT4W8DtvXCgazuPFlK+Wvn8V+SIfNCYAfwdWBNRCwENpA9XVuPtoMRMR24GZgDnEVW+MbVzzbos0Aedzhy7C8gq11dj/WtXwzMiIhuJbJ3HKeTvWxdFwKvInv8ut/NRLJfDeB+svK5MwCWbK4AAAVfSURBVCLWAz8jw7w9bJJ0nAxbJ96l5Km033eCzXhgbkQsBSb3fuHVoPV94E3AlaWU54/jffuDXeHIaeVxdf2tA8YdONY3jIjJwHrg58BHyFBwJvAIWa0bZn9Pos48PL0zAzGgFBg3G/60FR74EGwZYVuHJuQxPpoCUEr5UkTcB1xFNuSvjIilpZSjna5bRwbGJcBe8jTt5gGf7fAxLaWU+r2P5pT+OGAN8I0Bz+09yniA+cDuQftSSnmqBv93Au8iJ2CsjIg5pZRBgVWSNCTD1on3IHmpga57gO1kxesg5OUfyNNnF5FB65khtt27cvpIwWKQx8lKydll+B6ynjkREZ3q1tuAp2tV62IyXC0vpewEiIhrR7n9g+P+TzD5aN6jcCTjz4Qd2+CqiDilU926jNz24aBWStlOfhd3RMQ3gesZ0BsVEVPIqtQnesesVvFG+zO2FXhP32Oz+9Z/C7yxlLJjyG1uJmdjvq6UctTKZCnlH2Sv2E8i4ivAM8DlZH+YJOkY2SB/gpVS9pVSNnUX8tTXn+t6qRWt+8ng8kGgRMTZdTm5t62IWBURGzqb30VWaa6OiLN6TdhD7NMfyJmRayPiuoh4Q0RcEhHLhghH5wK3R8TMiLgO+BxHKjC7yV/6n6rbvJo87TYafxl//P9JmHhrBte/A/fWWYdzgW8BPyql7IiIkyPirjobclpEzAGuIIPLwP0iL6x6Q0TMiIh3AKvJ6tZorAamR84OnVmP95L6XC/A3grMjpz5+eb6ftdEXrT1f9QweRtwW0QsruNnRcTSiPg4QER8LCKur8fi9cAisuq1fZT7L0nqY9gaG84jqx3nko30f+ws3b6dc8ieHQBKKXvJ/qhbgGeBOxneIrLC9lWy2rIOmEunT+oo7iMraY8B3wa+Qw1bpZTngIXAe8nQspJszB6NPf/OvqpjUqti/7o8Q8Q8clbnr4GHyP6yxXXoIeAMstF8G/BAfX7g/pa8hc8HyNsEbQLuAlYw/HXAetvZRc4iXQA8AXyGnIkJdSZkKeVJ8ruYRl6y4gmysf7ZETa9gpxNuYy8efbD9X121uf3kbMVH6n7/z7g2l4FUpJ07OK/e5mlYxcv1W2AIi4lm777G8GHMRXYRCn9TecvWxFxI3ATcHrxB1aSxhx7tjQWbSEnCUxidJWjSWTVbVuLnXqxRMQnyRmJz5GnjlcAaw1akjQ2eRpRY08p+8hZjVPJADWMSXX8+vr6l7MZ5GnLLWRP22qy902SNAZ5GlFjV96Meh7ZX/U8g29GPR6YUv9cT/ZESZL0kjFsaWzLm1LPJE8rTiBn0B0iw9VEcjbg74BtY6CiJUl6BTJs6ZUh7z/4WvLK+r1ern3AU5RycKSXSpLUkmFLkiSpIRvkJUmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIb+Axh52Il3MTMkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from graph_utils import show_cluster_graphs\n",
    "\n",
    "diff_ids = {40, 41, 42}\n",
    "show_cluster_graphs(\n",
    "    df,\n",
    "    golden_pairs_set, dedupe_found_pairs_set, dedupe_unclustered_found_pairs_set,\n",
    "    dedupe_unclustered_pairs_score_dict,\n",
    "    diff_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Next Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Make sure to learn about other pre-processing methods, indexing techniques, similarity functions, classifiers, etc. Check the talk **The Art and Science of Data Matching** [1] and read the book **Data Matching** [2]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Once we have the clusters, how to consolidate data from many records into one? Check for material on **Data Fusion**:\n",
    "  - [Open issue at Python Record Linkage Toolkit](https://github.com/J535D165/recordlinkage/issues/27)\n",
    "  - Christen, 6.12 Merging Matches [2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What if new records arrive? Should we merge, unmerge, move records from clusters? Check for material on **Incremental Record Linkage**:\n",
    "  - Dedupe approach is to either add to an existing cluster or create a new cluster. You can use a [Gazetteer](https://docs.dedupe.io/en/latest/API-documentation.html#gazetteer-objects) with previously deduped data, then merge new matches into it, and index new nonmatches.\n",
    "  - Other approaches, check papers:\n",
    "      * [Online Correlation Clustering](https://arxiv.org/pdf/1001.0920.pdf)\n",
    "      * [Incremental Record Linkage](http://www.vldb.org/pvldb/vol7/p697-gruenheid.pdf)\n",
    "      * [Heuristic-based approaches for speeding up incremental record linkage](https://www.sciencedirect.com/science/article/pii/S0164121217302972)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Also worth checking the **Privacy** implications of Record Linkage:\n",
    "- Christen, chapter 8 [2]\n",
    "- [k-anonymity](https://desfontain.es/privacy/k-anonymity.html), [k-map](https://desfontain.es/privacy/k-map.html), [δ-presence](https://desfontain.es/privacy/delta-presence.html)\n",
    "- [Why differential privacy is awesome](https://desfontain.es/privacy/differential-privacy-awesomeness.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Referenced** by this talk:\n",
    "    * [1] [Mike Mull, The Art and Science of Data Matching](https://www.youtube.com/watch?v=Y-nYEOgq3YE) (slides [here](https://github.com/mikemull/Notebooks/blob/master/PyDataNYCSlides.ipynb))\n",
    "    * [2] [Peter Christen, Data Matching](http://www.springer.com/us/book/9783642311635)\n",
    "    * [3] [Mikhail Yuryevich Bilenko, Learnable Similarity Functions and Their Application to Record Linkage and Clustering](http://www.cs.utexas.edu/~ml/papers/marlin-dissertation-06.pdf)\n",
    "- Other **talks** about Record Linkage with Python:\n",
    "    * [Andrew Rowe, Big Data Deduplication and data matching using Python](https://www.youtube.com/watch?v=Z6mlvrYEYnk)\n",
    "    * [Rhydwyn Mcguire, Record linkage: Join for real life](https://www.youtube.com/watch?v=cEcVIjyHfiQ)\n",
    "- [Basics of Entity Resolution with Python and Dedupe](https://medium.com/district-data-labs/basics-of-entity-resolution-with-python-and-dedupe-bc87440b64d4)\n",
    "- [Automating your Data Cleanup with Python](https://blog.kjamistan.com/automating-your-data-cleanup-with-python/)\n",
    "- [Comparison of Data Matching software](https://github.com/J535D165/data-matching-software)\n",
    "- [Compilation of Record Linkage resources](https://github.com/ropeladder/record-linkage-resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Thank you!\n",
    "flavio@vinta.com.br  \n",
    "[@flaviojuvenal](https://twitter.com/flaviojuvenal)  \n",
    "[vinta.com.br](https://www.vinta.com.br?dedupe-slides=1)\n",
    "\n",
    "Special thanks to Russell Keith-Magee [@freakboy3742](https://twitter.com/freakboy3742), Forest Timothy Gregg [@forestgregg](https://twitter.com/forestgregg), and Jonathan de Bruin [@J535D165](https://github.com/J535D165)."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}