fdaDevice.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for interacting with API of openFDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plan\n",
    "In this project, we are looking for device adverse events. We are particularly interested in:\n",
    "    - Cause of failure\n",
    "    - Date of failure (ie age of device)\n",
    "We want to see which is the most common cause for given device to fail and at which stage of use this occurs.\n",
    "Relevant fields could be (for full reference see https://open.fda.gov/device/event/reference/):\n",
    "##### Event\n",
    "device_date_of_manufacturer\n",
    "date_of_event, date_report, date_received\n",
    "previous_use_code, remedial_action\n",
    "single_use_flag\n",
    "##### Source\n",
    "reprocessed_and_reused_flag\n",
    "##### Device\n",
    "device.generic_name\n",
    "device.expiration_date_of_device, device.device_age_text\n",
    "device.implant_flag, device.date_removed_flag\n",
    "device.manufacturer_d_name, device.manufacturer_d_state, device.manufacturer_d_country\n",
    "##### Patient\n",
    "patient.sequence_number_outcome, patient.sequence_number_treatment\n",
    "##### Report Text\n",
    "mdr_text.text, mdr_text.text_type_code\n",
    "##### Reporter Dependent Fields\n",
    "report_date\n",
    "event_location\n",
    "manufacturer_name, manufacturer_country\n",
    "manufacturer_g1_name, manufacturer_g1_state\n",
    "##### OpenFDA fields\n",
    "device_class\n",
    "##### Further interesting fields:\n",
    "Source: reporter_occupation_code\n",
    "Device: device.device_operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import modules\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import json\n",
    "import requests\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import Normalizer\n",
    "from sklearn.cluster import KMeans, MiniBatchKMeans\n",
    "import logging\n",
    "import sys\n",
    "from time import time\n",
    "from datetime import datetime, timedelta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Build base query\n",
    "baseurl = 'https://api.fda.gov/device/event.json?'\n",
    "time_period = 7\n",
    "initial_date = \"20150101\"\n",
    "final_date = datetime.strptime(\"20150701\", \"%Y%m%d\")\n",
    "skip = 0\n",
    "limit = 100\n",
    "\n",
    "apikey = ''\n",
    "with open('apikey.txt', 'r') as myfile:\n",
    "    apikey = myfile.read().replace('\\n', '')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Features to check for existence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "list_features = ['device_date_of_manufacturer', \n",
    "                 'date_of_event',\n",
    "                 #'date_report',\n",
    "                 #'date_received',\n",
    "                 'previous_use_code',\n",
    "                 #'remedial_action',\n",
    "                 #'single_use_flag',\n",
    "                 #'reprocessed_and_reused_flag',\n",
    "                 #'reporter_occupation_code',\n",
    "                 #'device.date_received',\n",
    "                 #'device.generic_name' # this allows for empty string! \n",
    "                ]\n",
    "\n",
    "# possibility to further narrow donw the search\n",
    "# list_features_specific = ['device.openfda.device_name:\"sensor\"', \n",
    "#                   #'device.implant_flag:\"Y\"',\n",
    "#                   #'previous_use_code:\"I\"', # I - initial use, R - reuse, U - unknown, * - invalid data\n",
    "#                   #'device.manufacturer_d_country:\"US\"' # SZ - Switzerland\n",
    "#                  ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List of usable device names "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "list_device_names = [\"pump\",\n",
    "                    \"sensor\",\n",
    "                    \"prosthesis\",\n",
    "                    \"defibrilator\",\n",
    "                    \"pacemaker\",\n",
    "                    \"catheter\",\n",
    "                    \"electrode\",\n",
    "                    #\"wearable\",\n",
    "                     \"stent\",\n",
    "                     \"ray\",\n",
    "                     \"ventilator\",\n",
    "                     \"bed\",\n",
    "                     \"implant\",\n",
    "                     \"lens\",\n",
    "                     #\"mds\" # https://www.cancer.org/cancer/myelodysplastic-syndrome/about/what-is-mds.html\n",
    "                     \"dialysis\",\n",
    "                     \"graft\",\n",
    "                    ]\n",
    "                  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build full query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100\n",
      "200\n",
      "300\n",
      "400\n",
      "448\n",
      "Week Done\n",
      "548\n",
      "648\n",
      "748\n",
      "848\n",
      "948\n",
      "1023\n",
      "Week Done\n",
      "1123\n",
      "1223\n",
      "1323\n",
      "1423\n",
      "1434\n",
      "Week Done\n",
      "1534\n",
      "1634\n",
      "1734\n",
      "1834\n",
      "1878\n",
      "Week Done\n",
      "1978\n",
      "2078\n",
      "2178\n",
      "2278\n",
      "2378\n",
      "Week Done\n",
      "2478\n",
      "2578\n",
      "2678\n",
      "2778\n",
      "2851\n",
      "Week Done\n",
      "2951\n",
      "3051\n",
      "3151\n",
      "3251\n",
      "3343\n",
      "Week Done\n",
      "3443\n",
      "3543\n",
      "3643\n",
      "3743\n",
      "3843\n",
      "3875\n",
      "Week Done\n",
      "3975\n",
      "4075\n",
      "4175\n",
      "4275\n",
      "4355\n",
      "Week Done\n",
      "4455\n",
      "4555\n",
      "4655\n",
      "4755\n",
      "4831\n",
      "Week Done\n",
      "4931\n",
      "5031\n",
      "5131\n",
      "5231\n",
      "5312\n",
      "Week Done\n",
      "5412\n",
      "5512\n",
      "5612\n",
      "5712\n",
      "5812\n",
      "5905\n",
      "Week Done\n",
      "6005\n",
      "6105\n",
      "6205\n",
      "6305\n",
      "6405\n",
      "6457\n",
      "Week Done\n",
      "6557\n",
      "6657\n",
      "6757\n",
      "6857\n",
      "6957\n",
      "7036\n",
      "Week Done\n",
      "7136\n",
      "7236\n",
      "7336\n",
      "7436\n",
      "7536\n",
      "7552\n",
      "Week Done\n",
      "7652\n",
      "7752\n",
      "7852\n",
      "7952\n",
      "8052\n",
      "8136\n",
      "Week Done\n",
      "8236\n",
      "8336\n",
      "8436\n",
      "8536\n",
      "8636\n",
      "8642\n",
      "Week Done\n",
      "8742\n",
      "8842\n",
      "8942\n",
      "9042\n",
      "9142\n",
      "9151\n",
      "Week Done\n",
      "9251\n",
      "9351\n",
      "9451\n",
      "9551\n",
      "9651\n",
      "9674\n",
      "Week Done\n",
      "9774\n",
      "9874\n",
      "9974\n",
      "10074\n",
      "10174\n",
      "10202\n",
      "Week Done\n",
      "10302\n",
      "10402\n",
      "10502\n",
      "10602\n",
      "10702\n",
      "10769\n",
      "Week Done\n",
      "10869\n",
      "10969\n",
      "11069\n",
      "11169\n",
      "11269\n",
      "11352\n",
      "Week Done\n",
      "11452\n",
      "11552\n",
      "11652\n",
      "11735\n",
      "Week Done\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'\\n# adding date range\\nquery = query+\"date_of_event:[\"\"+start_date+\"\"\"+\"+TO+\"+\"\"\"+end_date+\"\"]\"\\n\\n# checking features for existence\\nfor x in list_features:\\n    query = query + \"+AND+_exists_:\" + x\\n\\n# # Possibility to furhter narrow down the search\\n# for y in list_features_specific:\\n#     query = query + \"+AND+\" + y\\n\\n# Search for specific device, see the list abvoe\\nname_query = \\'+AND+device.openfda.device_name:\\' + device_name ;\\n\\n# Construct final query\\n# q1 = baseurl + \\'api_key=\\' + apikey + \\'&\\' + query + \\'&\\' + \\'limit=\\' + str(limit)\\nq2 = baseurl + \\'api_key=\\' + apikey + \\'&\\' + query + name_query + \\'&\\' +\\'limit=\\' + str(limit)\\n'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start_date = datetime.strptime(initial_date, \"%Y%m%d\")\n",
    "end_date = start_date +  timedelta(days=time_period)\n",
    "device_name = list_device_names[4]\n",
    "results = []\n",
    "\n",
    "while True:\n",
    "    skip = 0\n",
    "    \n",
    "    while (skip<=5000):\n",
    "        query = 'search=device.generic_name:'+device_name+'+AND+'\n",
    "        # adding date range\n",
    "        start = str(start_date.date()).replace(\"_\", \"\")\n",
    "        end = str(end_date.date()).replace(\"_\", \"\")\n",
    "        query = query+\"date_of_event:[\\\"\"+start+\"\\\"\"+\"+TO+\"+\"\\\"\"+end+\"\\\"]\"\n",
    "        \n",
    "        # checking features for existence\n",
    "        for x in list_features:\n",
    "            query = query + \"+AND+_exists_:\" + x\n",
    "            \n",
    "        # # Possibility to furhter narrow down the search\n",
    "        # for y in list_features_specific:\n",
    "        #     query = query + \"+AND+\" + y\n",
    "\n",
    "        \n",
    "        q1 = baseurl + 'api_key=' + apikey + '&' + query + '&' + 'limit=' + str(limit) + '&' + 'skip=' + str(skip)\n",
    "        dq1 = requests.get(q1)\n",
    "        # dq1.json()['results']\n",
    "        data = json.loads(dq1.text)\n",
    "\n",
    "        if \"results\" in data:\n",
    "            result = data['results']\n",
    "            results = results + result\n",
    "            print(len(results))\n",
    "            skip = skip + limit\n",
    "        else:\n",
    "            break\n",
    "\n",
    "    print(\"Week Done\")\n",
    "    start_date = end_date + timedelta(days=1)\n",
    "    end_date = start_date +  timedelta(days=time_period)\n",
    "    if start_date > final_date:\n",
    "        break\n",
    "        \n",
    "    if end_date > final_date:\n",
    "        end_date = final_date\n",
    "\n",
    "'''\n",
    "# adding date range\n",
    "query = query+\"date_of_event:[\\\"\"+start_date+\"\\\"\"+\"+TO+\"+\"\\\"\"+end_date+\"\\\"]\"\n",
    "\n",
    "# checking features for existence\n",
    "for x in list_features:\n",
    "    query = query + \"+AND+_exists_:\" + x\n",
    "\n",
    "# # Possibility to furhter narrow down the search\n",
    "# for y in list_features_specific:\n",
    "#     query = query + \"+AND+\" + y\n",
    "\n",
    "# Search for specific device, see the list abvoe\n",
    "name_query = '+AND+device.openfda.device_name:' + device_name ;\n",
    "\n",
    "# Construct final query\n",
    "# q1 = baseurl + 'api_key=' + apikey + '&' + query + '&' + 'limit=' + str(limit)\n",
    "q2 = baseurl + 'api_key=' + apikey + '&' + query + name_query + '&' +'limit=' + str(limit)\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\" # Checking the number of data points for each device\\ndq = requests.get(q2)\\n# dq1.json()['results']\\ndata = json.loads(dq.text)\\nnumber = data['meta']['results']['total'] # check number of matching entries\\nresults = data['results']\\nnumber\\n\""
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "''' # Checking the number of data points for each device\n",
    "dq = requests.get(q2)\n",
    "# dq1.json()['results']\n",
    "data = json.loads(dq.text)\n",
    "number = data['meta']['results']['total'] # check number of matching entries\n",
    "results = data['results']\n",
    "number\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11735"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Can also spare some structuring effort when loading data by calling normalization method\n",
    "# dftest = pd.io.json.json_normalize(results)\n",
    "len(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'adverse_event_flag': 'Y',\n",
       " 'date_manufacturer_received': '20150224',\n",
       " 'date_of_event': '20150107',\n",
       " 'date_received': '20150126',\n",
       " 'date_report': '20150224',\n",
       " 'device': [{'brand_name': 'CAPSUREEPI',\n",
       "   'catalog_number': '4968-25',\n",
       "   'date_received': '20150126',\n",
       "   'date_removed_flag': '',\n",
       "   'date_returned_to_manufacturer': '20150112',\n",
       "   'device_age_text': 'DA',\n",
       "   'device_availability': 'Device was returned to manufacturer',\n",
       "   'device_evaluated_by_manufacturer': 'Y',\n",
       "   'device_event_key': '',\n",
       "   'device_operator': 'LAY USER/PATIENT',\n",
       "   'device_report_product_code': 'DTB',\n",
       "   'device_sequence_number': ' 1.0',\n",
       "   'expiration_date_of_device': '20121128',\n",
       "   'generic_name': 'ELECTRODE, PACEMAKER, PERMANENT',\n",
       "   'implant_flag': '',\n",
       "   'lot_number': '',\n",
       "   'manufacturer_d_address_1': '8200 CORAL SEA STREET NE',\n",
       "   'manufacturer_d_address_2': '',\n",
       "   'manufacturer_d_city': 'MOUNDS VIEW',\n",
       "   'manufacturer_d_country': 'US',\n",
       "   'manufacturer_d_name': 'MEDTRONIC, INC.',\n",
       "   'manufacturer_d_postal_code': '55112',\n",
       "   'manufacturer_d_state': 'MN',\n",
       "   'manufacturer_d_zip_code': '55112',\n",
       "   'manufacturer_d_zip_code_ext': '',\n",
       "   'model_number': '4968-25',\n",
       "   'openfda': {'device_class': '3',\n",
       "    'device_name': 'Permanent Pacemaker Electrode',\n",
       "    'medical_specialty_description': 'Cardiovascular',\n",
       "    'regulation_number': '870.3680'},\n",
       "   'other_id_number': ''}],\n",
       " 'device_date_of_manufacturer': '20110325',\n",
       " 'distributor_address_1': '',\n",
       " 'distributor_address_2': '',\n",
       " 'distributor_city': '',\n",
       " 'distributor_name': '',\n",
       " 'distributor_state': '',\n",
       " 'distributor_zip_code': '',\n",
       " 'distributor_zip_code_ext': '',\n",
       " 'event_key': '',\n",
       " 'event_location': '',\n",
       " 'event_type': 'Injury',\n",
       " 'health_professional': 'Y',\n",
       " 'initial_report_to_fda': 'Unknown',\n",
       " 'manufacturer_address_1': '',\n",
       " 'manufacturer_address_2': '',\n",
       " 'manufacturer_city': '',\n",
       " 'manufacturer_contact_address_1': '8200 CORAL SEA ST NE',\n",
       " 'manufacturer_contact_address_2': '',\n",
       " 'manufacturer_contact_area_code': '763',\n",
       " 'manufacturer_contact_city': 'MOUNDS VIEW',\n",
       " 'manufacturer_contact_country': 'US',\n",
       " 'manufacturer_contact_exchange': '763',\n",
       " 'manufacturer_contact_extension': '',\n",
       " 'manufacturer_contact_f_name': 'ANNE',\n",
       " 'manufacturer_contact_l_name': 'SCHILLING',\n",
       " 'manufacturer_contact_pcity': '76350520',\n",
       " 'manufacturer_contact_pcountry': '',\n",
       " 'manufacturer_contact_phone_number': '7635',\n",
       " 'manufacturer_contact_plocal': '7635052036',\n",
       " 'manufacturer_contact_postal_code': '55112',\n",
       " 'manufacturer_contact_state': 'MN',\n",
       " 'manufacturer_contact_t_name': '',\n",
       " 'manufacturer_contact_zip_code': '55112',\n",
       " 'manufacturer_contact_zip_ext': '',\n",
       " 'manufacturer_country': '',\n",
       " 'manufacturer_g1_address_1': '8200 CORAL SEA ST NE',\n",
       " 'manufacturer_g1_address_2': '',\n",
       " 'manufacturer_g1_city': 'MOUNDS VIEW',\n",
       " 'manufacturer_g1_country': 'US',\n",
       " 'manufacturer_g1_name': 'MEDTRONIC CARDIAC RHYTHM HEART FAILURE',\n",
       " 'manufacturer_g1_postal_code': '55112',\n",
       " 'manufacturer_g1_state': 'MN',\n",
       " 'manufacturer_g1_zip_code': '55112',\n",
       " 'manufacturer_g1_zip_code_ext': '',\n",
       " 'manufacturer_link_flag': 'Y',\n",
       " 'manufacturer_name': '',\n",
       " 'manufacturer_postal_code': '',\n",
       " 'manufacturer_state': '',\n",
       " 'manufacturer_zip_code': '',\n",
       " 'manufacturer_zip_code_ext': '',\n",
       " 'mdr_report_key': '4453261',\n",
       " 'mdr_text': [{'mdr_text_key': '13068578',\n",
       "   'patient_sequence_number': '1',\n",
       "   'text': 'PRODUCT EVENT SUMMARY :THE PROXIMAL SEGMENT OF THE LEAD WAS RETURNED, ANALYZED AND THE OUTER INSULATION OF THE LEAD DEVELOPED A BREACH DUE TO A DEPRESSION WHILE IN VIVO. THE OUTER INSULATION OF THE LEAD DEVELOPED A COSMETIC DEPRESSION WHILE IN VIVO. THE OUTER INSULATION OF THE LEAD WAS OBSERVED TO HAVE BLOOD INGRESSION. (B)(4).',\n",
       "   'text_type_code': 'Additional Manufacturer Narrative'},\n",
       "  {'mdr_text_key': '12973157',\n",
       "   'patient_sequence_number': '1',\n",
       "   'text': 'THE INFORMATION SUBMITTED REFLECTS ALL RELEVANT DATA RECEIVED. IF ADDITIONAL RELEVANT INFORMATION IS RECEIVED, A SUPPLEMENTAL REPORT WILL BE SUBMITTED.',\n",
       "   'text_type_code': 'Additional Manufacturer Narrative'},\n",
       "  {'mdr_text_key': '5395576',\n",
       "   'patient_sequence_number': '1',\n",
       "   'text': 'IT WAS REPORTED THAT DURING AN UPGRADE PROCEDURE WHEN THE POCKET WAS OPENED AND THE RIGHT VENTRICULAR (RV) LEAD WAS FREED UP, AN INSULATION BREACH WAS OBSERVED JUST DISTAL TO THE PIN SECTION OF THE LEAD. THE LEAD WAS EXPLANTED AND REPLACED. NO PERFORMANCE ISSUES WERE NOTED WITH THE LEAD WHILE IMPLANTED. NO FURTHER PATIENT COMPLICATIONS HAVE BEEN REPORTED AS A RESULT OF THIS EVENT.',\n",
       "   'text_type_code': 'Description of Event or Problem'}],\n",
       " 'number_devices_in_event': '',\n",
       " 'number_patients_in_event': '',\n",
       " 'patient': [{'date_received': '20150126',\n",
       "   'patient_sequence_number': '1',\n",
       "   'sequence_number_outcome': ['1. R'],\n",
       "   'sequence_number_treatment': ['5620 ST JUDE IPG']},\n",
       "  {'date_received': '20150126',\n",
       "   'patient_sequence_number': '1',\n",
       "   'sequence_number_outcome': ['1. R'],\n",
       "   'sequence_number_treatment': ['5620 ST JUDE IPG']}],\n",
       " 'previous_use_code': 'I',\n",
       " 'product_problem_flag': 'Y',\n",
       " 'remedial_action': [''],\n",
       " 'removal_correction_number': '',\n",
       " 'report_number': '2182208-2015-00023',\n",
       " 'report_source_code': 'Manufacturer report',\n",
       " 'report_to_fda': '',\n",
       " 'report_to_manufacturer': '',\n",
       " 'reporter_occupation_code': 'PHYSICIAN',\n",
       " 'reprocessed_and_reused_flag': 'N',\n",
       " 'single_use_flag': 'Y',\n",
       " 'source_type': ['Health Professional', 'Company representation'],\n",
       " 'type_of_report': ['Initial submission', 'Followup']}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check that you obtained what you expected\n",
    "results[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Fields of Interest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fois_result = ['device_date_of_manufacturer',\n",
    "               'date_of_event',\n",
    "               'previous_use_code',\n",
    "               'single_use_flag',\n",
    "               'reprocessed_and_reused_flag',\n",
    "               #'reporter_occupation_code'\n",
    "              ]\n",
    "fois_device = [#'generic_name', \n",
    "               'expiration_date_of_device', \n",
    "               #'device_age_text', \n",
    "               #'implant_flag', \n",
    "               #'date_removed_flag', \n",
    "               #'manufacturer_d_name', \n",
    "               #'manufacturer_d_state',\n",
    "               'manufacturer_d_country',\n",
    "               #'device_operator'\n",
    "              ]\n",
    "fois_patient = [#'sequence_number_outcome', # problematic to factorize as it contains lists\n",
    "                #'sequence_number_treatment' # problematic to factorize as it contains lists\n",
    "              ]\n",
    "fois_mdrText = ['text',\n",
    "                'text_type_code']\n",
    "fois_openfda = ['device_name',\n",
    "                'device_class',\n",
    "                'medical_specialty_description']\n",
    "\n",
    "# device = data['results'][0]['device'][0]\n",
    "device = [x['device'][0] for x in results]\n",
    "# patient = data['results'][0]['patient'][0]\n",
    "patient = [x['patient'][0] for x in results]\n",
    "# mdrText = data['results'][0]['mdr_text'][0] # there may be more items in the list! \n",
    "mdrText = [x['mdr_text'] for x in results]\n",
    "#mdrText = [y['text'] for y in [x['mdr_text'][0] for x in data['results']]]\n",
    "# openfda = data['results'][0]['device'][0]['openfda']\n",
    "openfda = [x['device'][0]['openfda'] for x in results]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create sub dataframes for non-multiple columns\n",
    "df_results = pd.DataFrame(results, index = range(len(results)), columns = fois_result)\n",
    "df_openfda = pd.DataFrame(openfda, index = range(len(results)),columns = fois_openfda)\n",
    "df_device = pd.DataFrame(device, index = range(len(results)),columns = fois_device)\n",
    "df_patient = pd.DataFrame(patient, index = range(len(results)),columns = fois_patient)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Pull out all relevant text fields\n",
    "fillDic = {'mdr_text_key': '', 'patient_sequence_number': '', 'text': np.nan, 'text_type_code': np.nan}\n",
    "a = [x[0] if len(x) > 0 else fillDic for x in mdrText]\n",
    "b = [x[1] if len(x) > 1 else fillDic for x in mdrText]\n",
    "c = [x[2] if len(x) > 2 else fillDic for x in mdrText] \n",
    "\n",
    "\n",
    "a = pd.DataFrame(a, index = range(len(results)),columns = fois_mdrText)\n",
    "b = pd.DataFrame(b, index = range(len(results)),columns = fois_mdrText)\n",
    "c = pd.DataFrame(c, index = range(len(results)),columns = fois_mdrText)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Rename duplicate columns:\n",
    "fois_mdrText_a = [x + '_0' for x in fois_mdrText]\n",
    "fois_mdrText_b = [x + '_1' for x in fois_mdrText]\n",
    "fois_mdrText_c = [x + '_2' for x in fois_mdrText]\n",
    "\n",
    "columns_a = dict(zip(fois_mdrText, fois_mdrText_a))\n",
    "columns_b = dict(zip(fois_mdrText, fois_mdrText_b))\n",
    "columns_c = dict(zip(fois_mdrText, fois_mdrText_c))\n",
    "\n",
    "a.rename(columns = columns_a, inplace = True)\n",
    "b.rename(columns = columns_b, inplace = True)\n",
    "c.rename(columns = columns_c, inplace = True)\n",
    "\n",
    "# I haven't seen a four-field entry. If there were some, we would need to deal with them in smarter way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Construct final data frame\n",
    "df_mdrText = pd.concat([a, b, c], axis = 1)\n",
    "\n",
    "# Concatenate into final dataframe\n",
    "df = pd.concat([df_results, df_device, df_patient, df_mdrText, df_openfda], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Determine age of device in days\n",
    "df['age_of_device_days'] = pd.to_datetime(df['date_of_event'], format='%Y%m%d') \\\n",
    "- pd.to_datetime(df['device_date_of_manufacturer'], format='%Y%m%d')\n",
    "\n",
    "# Determine timedelta from manufacture to specified expiry date\n",
    "df['days_to_expiry'] = pd.to_datetime(df['expiration_date_of_device'], format='%Y%m%d') \\\n",
    "- pd.to_datetime(df['device_date_of_manufacturer'], format='%Y%m%d')\n",
    "df = df.drop(['date_of_event','device_date_of_manufacturer', 'expiration_date_of_device'], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Write to file for manual inspection\n",
    "\n",
    "# file_name = device_name + '_mdrTextClasses.csv'\n",
    "# # cols_to_write = ['text', 'text_type_code', 'device_name',\n",
    "# #        'medical_specialty_description', 'age_of_device_days']\n",
    "# df.to_csv(file_name, mode = 'w', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check that you have what you expect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['previous_use_code', 'single_use_flag', 'reprocessed_and_reused_flag',\n",
       "       'manufacturer_d_country', 'text_0', 'text_type_code_0', 'text_1',\n",
       "       'text_type_code_1', 'text_2', 'text_type_code_2', 'device_name',\n",
       "       'device_class', 'medical_specialty_description', 'age_of_device_days',\n",
       "       'days_to_expiry'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>previous_use_code</th>\n",
       "      <th>single_use_flag</th>\n",
       "      <th>reprocessed_and_reused_flag</th>\n",
       "      <th>manufacturer_d_country</th>\n",
       "      <th>text_0</th>\n",
       "      <th>text_type_code_0</th>\n",
       "      <th>text_1</th>\n",
       "      <th>text_type_code_1</th>\n",
       "      <th>text_2</th>\n",
       "      <th>text_type_code_2</th>\n",
       "      <th>device_name</th>\n",
       "      <th>device_class</th>\n",
       "      <th>medical_specialty_description</th>\n",
       "      <th>age_of_device_days</th>\n",
       "      <th>days_to_expiry</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6638</th>\n",
       "      <td>R</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>US</td>\n",
       "      <td>PRODUCT EVENT SUMMARY: ANALYSIS CONFIRMED THE ...</td>\n",
       "      <td>Additional Manufacturer Narrative</td>\n",
       "      <td>IT WAS REPORTED THAT THE RADIO FREQUENCY (RF) ...</td>\n",
       "      <td>Description of Event or Problem</td>\n",
       "      <td>THE INFORMATION SUBMITTED REFLECTS ALL RELEVAN...</td>\n",
       "      <td>Additional Manufacturer Narrative</td>\n",
       "      <td>Programmer, Pacemaker</td>\n",
       "      <td>3</td>\n",
       "      <td>Cardiovascular</td>\n",
       "      <td>1331 days</td>\n",
       "      <td>NaT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>I</td>\n",
       "      <td>Y</td>\n",
       "      <td>N</td>\n",
       "      <td>US</td>\n",
       "      <td>(B)(4).</td>\n",
       "      <td>Additional Manufacturer Narrative</td>\n",
       "      <td>IT WAS REPORTED THAT IN CLINIC, THE PULSE GENE...</td>\n",
       "      <td>Description of Event or Problem</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Implantable Pulse Generator, Pacemaker (Non-Crt)</td>\n",
       "      <td>3</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>3064 days</td>\n",
       "      <td>562 days</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6168</th>\n",
       "      <td>I</td>\n",
       "      <td>Y</td>\n",
       "      <td>N</td>\n",
       "      <td>US</td>\n",
       "      <td>IT WAS REPORTED THAT THE RIGHT ATRIAL (RA) LEA...</td>\n",
       "      <td>Description of Event or Problem</td>\n",
       "      <td>THE INFORMATION SUBMITTED REFLECTS ALL RELEVAN...</td>\n",
       "      <td>Additional Manufacturer Narrative</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Permanent Pacemaker Electrode</td>\n",
       "      <td>3</td>\n",
       "      <td>Cardiovascular</td>\n",
       "      <td>67 days</td>\n",
       "      <td>719 days</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     previous_use_code single_use_flag reprocessed_and_reused_flag  \\\n",
       "6638                 R               N                           N   \n",
       "51                   I               Y                           N   \n",
       "6168                 I               Y                           N   \n",
       "\n",
       "     manufacturer_d_country  \\\n",
       "6638                     US   \n",
       "51                       US   \n",
       "6168                     US   \n",
       "\n",
       "                                                 text_0  \\\n",
       "6638  PRODUCT EVENT SUMMARY: ANALYSIS CONFIRMED THE ...   \n",
       "51                                              (B)(4).   \n",
       "6168  IT WAS REPORTED THAT THE RIGHT ATRIAL (RA) LEA...   \n",
       "\n",
       "                       text_type_code_0  \\\n",
       "6638  Additional Manufacturer Narrative   \n",
       "51    Additional Manufacturer Narrative   \n",
       "6168    Description of Event or Problem   \n",
       "\n",
       "                                                 text_1  \\\n",
       "6638  IT WAS REPORTED THAT THE RADIO FREQUENCY (RF) ...   \n",
       "51    IT WAS REPORTED THAT IN CLINIC, THE PULSE GENE...   \n",
       "6168  THE INFORMATION SUBMITTED REFLECTS ALL RELEVAN...   \n",
       "\n",
       "                       text_type_code_1  \\\n",
       "6638    Description of Event or Problem   \n",
       "51      Description of Event or Problem   \n",
       "6168  Additional Manufacturer Narrative   \n",
       "\n",
       "                                                 text_2  \\\n",
       "6638  THE INFORMATION SUBMITTED REFLECTS ALL RELEVAN...   \n",
       "51                                                  NaN   \n",
       "6168                                                NaN   \n",
       "\n",
       "                       text_type_code_2  \\\n",
       "6638  Additional Manufacturer Narrative   \n",
       "51                                  NaN   \n",
       "6168                                NaN   \n",
       "\n",
       "                                           device_name device_class  \\\n",
       "6638                             Programmer, Pacemaker            3   \n",
       "51    Implantable Pulse Generator, Pacemaker (Non-Crt)            3   \n",
       "6168                     Permanent Pacemaker Electrode            3   \n",
       "\n",
       "     medical_specialty_description  age_of_device_days  days_to_expiry  \n",
       "6638                Cardiovascular           1331 days             NaT  \n",
       "51                         Unknown           3064 days        562 days  \n",
       "6168                Cardiovascular             67 days        719 days  "
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Factorize selected columns\n",
    "This will convert nominal (string) entries into categorical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Columns that we want to translate into categories\n",
    "factCols = ['previous_use_code',\n",
    "            'single_use_flag',\n",
    "            'reprocessed_and_reused_flag',\n",
    "            #'reporter_occupation_code',\n",
    "            #'implant_flag',\n",
    "            #'manufacturer_d_name',\n",
    "            'manufacturer_d_country',\n",
    "            #'device_operator',\n",
    "            #'sequence_number_outcome', # problematic to factorize as it contains lists\n",
    "            #'sequence_number_treatment', # problematic to factorize as it contains lists\n",
    "            'text_type_code_0',\n",
    "            'text_type_code_1', \n",
    "            'text_type_code_2',\n",
    "            'device_name',\n",
    "            'medical_specialty_description']\n",
    "\n",
    "# copy dataframe to preserve original;\n",
    "df1 = df.copy()\n",
    "\n",
    "# This also works but will not assign consistent labeling across multiple columns\n",
    "df1[factCols] = df1[factCols].apply(lambda x: pd.factorize(x)[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# # This will allow you to have consistency for columns that should contain same categories. Ideally avoid using this by removing\n",
    "# the\n",
    "# # http://stackoverflow.com/questions/39390160/pandas-factorize-on-an-entire-data-frame\n",
    "# def categorise(df):\n",
    "#     categories = {k: v for v, k in enumerate(df.stack().unique())}\n",
    "#     return df.replace(categories)\n",
    "# df[factCols] = categorise(df[factCols])\n",
    "\n",
    "# Later we may need to adress the issue of rather random (although consistent) numbering of categories. Both are not currently crucial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# If using pd.factorize, this is likely redundant (but not harmful)\n",
    "\n",
    "# http://stackoverflow.com/questions/28910851/python-pandas-changing-some-column-types-to-categories\n",
    "# http://pandas.pydata.org/pandas-docs/stable/categorical.html\n",
    "\n",
    "# df[factCols2].astype('category')\n",
    "df1[factCols] = df1[factCols].apply(lambda x: x.astype('category'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Check that you have what you want"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>previous_use_code</th>\n",
       "      <th>single_use_flag</th>\n",
       "      <th>reprocessed_and_reused_flag</th>\n",
       "      <th>manufacturer_d_country</th>\n",
       "      <th>text_0</th>\n",
       "      <th>text_type_code_0</th>\n",
       "      <th>text_1</th>\n",
       "      <th>text_type_code_1</th>\n",
       "      <th>text_2</th>\n",
       "      <th>text_type_code_2</th>\n",
       "      <th>device_name</th>\n",
       "      <th>device_class</th>\n",
       "      <th>medical_specialty_description</th>\n",
       "      <th>age_of_device_days</th>\n",
       "      <th>days_to_expiry</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3513</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>ALL INFORMATION PROVIDED BY MANUFACTURER, NO M...</td>\n",
       "      <td>0</td>\n",
       "      <td>THIS REPORT IS TO ADVISE OF AN EVENT OBSERVED ...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>409 days</td>\n",
       "      <td>563 days</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5897</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>IT WAS REPORTED THAT THE RIGHT VENTRICULAR (RV...</td>\n",
       "      <td>1</td>\n",
       "      <td>.</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1932 days</td>\n",
       "      <td>721 days</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     previous_use_code single_use_flag reprocessed_and_reused_flag  \\\n",
       "3513                 0               0                           0   \n",
       "5897                 0               0                           0   \n",
       "\n",
       "     manufacturer_d_country  \\\n",
       "3513                      0   \n",
       "5897                      0   \n",
       "\n",
       "                                                 text_0 text_type_code_0  \\\n",
       "3513  ALL INFORMATION PROVIDED BY MANUFACTURER, NO M...                0   \n",
       "5897  IT WAS REPORTED THAT THE RIGHT VENTRICULAR (RV...                1   \n",
       "\n",
       "                                                 text_1 text_type_code_1  \\\n",
       "3513  THIS REPORT IS TO ADVISE OF AN EVENT OBSERVED ...                0   \n",
       "5897                                                  .                1   \n",
       "\n",
       "     text_2 text_type_code_2 device_name device_class  \\\n",
       "3513    NaN               -1           4            3   \n",
       "5897    NaN               -1           1            3   \n",
       "\n",
       "     medical_specialty_description  age_of_device_days  days_to_expiry  \n",
       "3513                             0            409 days        563 days  \n",
       "5897                             1           1932 days        721 days  "
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.sample(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert mdr_text columns into class assignements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# estabilish variables\n",
    "lsa = 4\n",
    "n_features = 10000\n",
    "k = 5\n",
    "\n",
    "# pull out relevant subset\n",
    "# here we use regular expression to match all column names starting with \"text_#\" (ie we pick 3 columns)\n",
    "df_text = df1[df1.columns[df1.columns.to_series().str.contains('^text_[0-9]$')]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# comvert text data into single dimensional iterable [list]\n",
    "\n",
    "# save structuring information for possible later use\n",
    "IDXS = df_text.index\n",
    "COLS = df_text.columns\n",
    "# join texts into one string, droping nans\n",
    "df_text_cat = df_text.apply(lambda x: '; '.join(x.dropna().values.tolist()), axis=1)\n",
    "text_np_cat = df_text_cat.values\n",
    "\n",
    "# text_np = df_text.values\n",
    "# text_np_cat = ['; '.join(map(str, x)) for x in text_np]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Do Tfidf vectorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done in 1.318166s\n",
      "n_samples: 11735, n_features: 4123\n"
     ]
    }
   ],
   "source": [
    "# Do Tfidf vectorization\n",
    "t0 = time()\n",
    "vectorizer = TfidfVectorizer(max_df=0.5, max_features=n_features,\n",
    "                                 min_df=2, stop_words='english',\n",
    "                                 use_idf=True)\n",
    "X = vectorizer.fit_transform(text_np_cat)\n",
    "\n",
    "print(\"done in %fs\" % (time() - t0))\n",
    "print(\"n_samples: %d, n_features: %d\" % X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Do clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clustering sparse data with MiniBatchKMeans(batch_size=1000, compute_labels=True, init='k-means++',\n",
      "        init_size=1000, max_iter=100, max_no_improvement=10, n_clusters=5,\n",
      "        n_init=1, random_state=None, reassignment_ratio=0.01, tol=0.0,\n",
      "        verbose=False)\n",
      "done in 0.277s\n"
     ]
    }
   ],
   "source": [
    "#clustering\n",
    "km = MiniBatchKMeans(n_clusters=k, init='k-means++', n_init=1,\n",
    "                         init_size=1000, batch_size=1000, verbose=False)\n",
    "print(\"Clustering sparse data with %s\" % km)\n",
    "t0 = time()\n",
    "km.fit(X)\n",
    "print(\"done in %0.3fs\" % (time() - t0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 1744, 1: 3670, 2: 1446, 3: 2163, 4: 2712}"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See the cluster assignements\n",
    "uniqs, uniq_counts = np.unique(km.labels_, return_counts = True)\n",
    "dict(zip(uniqs,uniq_counts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEWCAYAAACe8xtsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHK5JREFUeJzt3X2UHXWd5/H3hwAhPiBg2hiSYACDGpgxShsZdHYQZYio\nE5wzQnwCd5Howvi8KnFcB13ZYXdEGWYET1BOgjBk4iBDBkENCOIThA4TDOFhiQQ2aUMSVAzoGkn4\n7B/1a7lpO923Yt++N92f1zl1uu6vflX1vdW376fr4daVbSIiIurYq90FRETEnifhERERtSU8IiKi\ntoRHRETUlvCIiIjaEh4REVFbwiPGJEmTJN0q6XFJF9SYb7okS9q7lfV1Akm3SHr3CKznXZK+P8j0\nGySd3uo6op5R/wcQw0PS24APAy8GHgdWAefZ3uUf/TCt18AM22uHedHzgUeB/Z0PO3U0269vdw3x\n+7LnEUOS9GHgQuB/ApOAQ4AvAn/Rzrr+QC8A7klwtJ4qea8ZZfILjUFJeg7wGeBs21+3/SvbT9q+\nzvbHSp/xki6U9NMyXChpfJn2e4ckymGfF5bxRZK+KOkb5RDS7ZIOL9NuLbPcJekJSadKmijpOkmP\nSfq5pO/t6o1J0rGS7pD0y/Lz2L51AqcDHyvLfd0A806QdIGkh8v835c0YYB+/1nSvaX2ByW9p2Ha\nLmuV9HFJvWW++yW9dhfPYZGki8uhmyck/UDS88s2/oWk+yS9rKH/wZKulrRF0jpJ72+Ydq6kr0m6\noqx3taQjJC2QtFnSekl/3q+EwyWtkLRV0rWSDmpY3jGSflie312SjmuYdouk8yT9APg1cFh5LTxY\n1r1O0tv7PdfPlee0TtLr+y3r3WX8XWUb/FP5vdy3q20XLWY7Q4ZdDsAcYDuw9yB9PgPcBjwP6AJ+\nCPyPMu1dwPf79TfwwjK+CPgZMJvqMOqVwJKB+pbHfwd8CdinDH8KaICaDgJ+AbyzLPet5fFzG9b7\n2UGe0xeBW4ApwDjgWGA8ML3UtHfp9wbgcEDAn1G9Ub58sFqBFwHrgYNLv+nA4buoYxHV4bWjgf2A\n7wDrgNNKXZ8Fbi599wJWAp8C9gUOAx4ETizTzwV+A5xYtsnlZVl/U+o7E1jXsO5bgF7gKOCZwNXA\nFWXalPJ7O6ms94TyuKth3v8LHFnW9RxgK/CiMn0ycGTDa+TJsv5xwH8Fftr3ey3LendD3+3Ah0rN\npwK/BA5q99/KWBuy5xFDeS7wqO3tg/R5O/AZ25ttbwE+TfWm3axrbK8o67gSmDVI3yep3nhe4GoP\n6Hsu7yr9vAF4wPZXbW+3fRVwH/CmoYopewf/BfiA7V7bO2z/0Pa2/n1tf8P2T1z5LvBtqpAYrNYd\nVEE0U9I+th+y/ZNBSrrG9krbvwGuAX5j+3LbO4B/Afr2PF5B9eb9Gdu/tf0gcCkwr2FZ37P9rbKt\nv0YV9ufbfhJYAkyXdEBD/6/avtv2r4D/DpwiaRzwDuB629fbfsr2cqCHKkz6LLK9pqxrO/AUcJSk\nCbY32l7T0Pdh25eW57S4bLdJu9gem4ELyzb9F+B+qt93jKCERwzlZ8BEDX510cHAww2PHy5tzXqk\nYfzXwLMG6fv3wFrg2+UQyDlN1tRX15Qm6plI9V/+YG/oAEh6vaTbymGpx6jePCcOVqurk/8fpNoT\n2CxpiaTBttemhvH/N8Djvu31AuDgchjpsVLPJ9j5Tbj/vI+WN+y+x7Dz9l/fMP4w1X/7E8u63tJv\nXa+metP/vXlL+JwKvBfYWA5Tvrih7yMNfX89QB2Nevv9w1D39RbDIOERQ/kRsA04eZA+P6V6M+lz\nSGkD+BXwjL4Jkp7/hxRj+3HbH7F9GNUJ+w/v4ph3/5r66uptYjWPUh3eOXywTqrO61wNfA6YZPsA\n4HqqQ1OD1mr7n22/utRo4H81UddQ1lMddjqgYXi27ZOGnHPXpjWMH0K1N/VoWddX+63rmbbPb+i/\n0x5h2eM5gSpg7qPaK9odUySpX10/3VXnaI2ERwzK9i+pjqF/UdLJkp4haZ/yH/f/Lt2uAj4pqUvS\nxNL/ijLtLuBISbMk7Uf133Ydm6iO3QMg6Y2SXljePH5JdQjoqQHmux44QtLbJO0t6VRgJnBdE8/5\nKeAy4PPlBPQ4SX9SwqLRvlSHn7YA28tJ3t+dcN5VrZJeJOn4srzfUP3HP9BzqGsF8Hg5GT+h1H2U\npFf8Act8h6SZkp5BdW7rX8ueyhXAmySdWNazn6TjJE0daCGqPlczV9Izqf4ZeYLdf87PA95fXodv\nAV5C9fuOEZTwiCHZvoDqMx6fpHqjXA/8NfBvpctnqY53/xhYDdxZ2rD9f6jedG4EHgDqfi7kXGBx\nOTRyCjCjLOsJqr2ii23fPEDNPwPeCHyE6tDbx4A32n60yfX+t/Jc7gB+TrVnsNPfi+3HgfcDS6lO\nxr8NWNbQZVe1jgfOp/oP/hGqN8MFTda1S+VN/Y1U54zWleV/mepk9e76KtVJ+0eoDuW9v6xrPTCX\n6rBY32vio+z6PWUvqtfQT6m2559RnRjfHbdTbdtHgfOAvyq/7xhBfVczRER0PEnvorry6tXtrmWs\ny55HRETUlvCIiIjactgqIiJqy55HRETUNmrvqjtx4kRPnz693WVEROxRVq5c+ajtrqH6jdrwmD59\nOj09Pe0uIyJijyKp/50ZBpTDVhERUVvCIyIiakt4REREbQmPiIioLeERERG1JTwiIqK2hEdERNSW\n8IiIiNoSHhERUduo/YR5DKOdvvGzTXIDz4iOkj2PiIioLeERERG1JTwiIqK2hEdERNSW8IiIiNpa\nFh6S9pO0QtJdktZI+nRpP1dSr6RVZTipYZ4FktZKul/SiQ3tR0taXaZdJHXC5T8REWNXKy/V3QYc\nb/sJSfsA35d0Q5n2Bdufa+wsaSYwDzgSOBi4UdIRtncAlwBnArcD1wNzgBuIiIi2aNmehytPlIf7\nlGGwi/XnAktsb7O9DlgLzJY0Gdjf9m22DVwOnNyquiMiYmgtPechaZykVcBmYLnt28uk90n6saTL\nJB1Y2qYA6xtm31DappTx/u0DrW++pB5JPVu2bBnW5xIREU9raXjY3mF7FjCVai/iKKpDUIcBs4CN\nwAXDuL6Ftrttd3d1Dfn97RERsZtG5Gor248BNwNzbG8qofIUcCkwu3TrBaY1zDa1tPWW8f7tERHR\nJq282qpL0gFlfAJwAnBfOYfR583A3WV8GTBP0nhJhwIzgBW2NwJbJR1TrrI6Dbi2VXVHRMTQWnm1\n1WRgsaRxVCG11PZ1kr4qaRbVyfOHgPcA2F4jaSlwD7AdOLtcaQVwFrAImEB1lVWutIqIaCN5lN6t\ntLu72z09Pe0uY3TohI/VjNLXaUSnkbTSdvdQ/fIJ84iIqC3hERERtSU8IiKitoRHRETUlvCIiIja\nEh4REVFbwiMiImpLeERERG0Jj4iIqC3hERERtSU8IiKitoRHRETUlvCIiIjaEh4REVFbwiMiImpL\neERERG0Jj4iIqC3hERERtSU8IiKitpaFh6T9JK2QdJekNZI+XdoPkrRc0gPl54EN8yyQtFbS/ZJO\nbGg/WtLqMu0iqRO+VDsiYuxq5Z7HNuB42y8FZgFzJB0DnAPcZHsGcFN5jKSZwDzgSGAOcLGkcWVZ\nlwBnAjPKMKeFdUdExBBaFh6uPFEe7lMGA3OBxaV9MXByGZ8LLLG9zfY6YC0wW9JkYH/bt9k2cHnD\nPBER0QYtPechaZykVcBmYLnt24FJtjeWLo8Ak8r4FGB9w+wbStuUMt6/PSIi2qSl4WF7h+1ZwFSq\nvYij+k031d7IsJA0X1KPpJ4tW7YM12IjIqKfEbnayvZjwM1U5yo2lUNRlJ+bS7deYFrDbFNLW28Z\n798+0HoW2u623d3V1TW8TyIiIn6nlVdbdUk6oIxPAE4A7gOWAaeXbqcD15bxZcA8SeMlHUp1YnxF\nOcS1VdIx5Sqr0xrmiYiINti7hcueDCwuV0ztBSy1fZ2kHwFLJZ0BPAycAmB7jaSlwD3AduBs2zvK\nss4CFgETgBvKEBERbaLqtMPo093d7Z6ennaXMTp0wsdqRunrNKLTSFppu3uofvmEeURE1JbwiIiI\n2hIeERFRW8IjIiJqS3hERERtCY+IiKgt4REREbUlPCIioraER0RE1JbwiIiI2hIeERFRWytvjBgR\nMXZ0wj3gYMTuA5c9j4iIqC3hERERtSU8IiKitoRHRETUlvCIiIjaEh4REVFbwiMiImpLeERERG0t\nCw9J0yTdLOkeSWskfaC0nyupV9KqMpzUMM8CSWsl3S/pxIb2oyWtLtMukjrl0zgREWNTKz9hvh34\niO07JT0bWClpeZn2Bdufa+wsaSYwDzgSOBi4UdIRtncAlwBnArcD1wNzgBtaWHtERAyiZXsetjfa\nvrOMPw7cC0wZZJa5wBLb22yvA9YCsyVNBva3fZttA5cDJ7eq7oiIGNqInPOQNB14GdWeA8D7JP1Y\n0mWSDixtU4D1DbNtKG1Tynj/9oHWM19Sj6SeLVu2DOMziIiIRi0PD0nPAq4GPmh7K9UhqMOAWcBG\n4ILhWpfthba7bXd3dXUN12IjIqKfloaHpH2oguNK218HsL3J9g7bTwGXArNL915gWsPsU0tbbxnv\n3x4REW3SyqutBHwFuNf25xvaJzd0ezNwdxlfBsyTNF7SocAMYIXtjcBWSceUZZ4GXNuquiMiYmit\nvNrqVcA7gdWSVpW2TwBvlTQLMPAQ8B4A22skLQXuobpS6+xypRXAWcAiYALVVVa50ioioo3kEfri\nkJHW3d3tnp6edpcxOnTCx2pG6es0RpFO+DuBP/hvRdJK291D9csnzCMioraER0RE1JbwiIiI2hIe\nERFRW8IjIiJqS3hERERtCY+IiKgt4REREbUlPCIioraER0RE1JbwiIiI2mqHh6QDJf1xK4qJiIg9\nQ1PhIekWSftLOgi4E7hU0ueHmi8iIkanZvc8nlO+BfAvgcttvxJ4XevKiog9gtQZQ4y4ZsNj7/Il\nTqcA17WwnoiI2AM0Gx6fBr4FrLV9h6TDgAdaV1ZERHSyZr9JcKPt350kt/1gznlERIxdze55/GOT\nbRERMQYMuuch6U+AY4EuSR9umLQ/MK6VhUVEROcaas9jX+BZVCHz7IZhK/BXg80oaZqkmyXdI2mN\npA+U9oMkLZf0QPl5YMM8CyStlXS/pBMb2o+WtLpMu0jK5RUREe006J6H7e8C35W0yPbDNZe9HfiI\n7TslPRtYKWk58C7gJtvnSzoHOAf4uKSZwDzgSOBg4EZJR9jeAVwCnAncDlwPzAFuqFlPREQMk2ZP\nmI+XtBCY3jiP7eN3NYPtjcDGMv64pHuBKcBc4LjSbTFwC/Dx0r7E9jZgnaS1wGxJDwH7274NQNLl\nwMkkPCIi2qbZ8Pga8CXgy8COuiuRNB14GdWew6QSLACPAJPK+BTgtobZNpS2J8t4//aB1jMfmA9w\nyCGH1C0zIiKa1Gx4bLd9ye6sQNKzgKuBD9re2ni6wrYleXeWOxDbC4GFAN3d3cO23IiI2Fmzl+r+\nu6SzJE0uJ7wPKve5GpSkfaiC40rbXy/Nm8qn1Sk/N5f2XmBaw+xTS1tvGe/fHhERbdJseJwOfBT4\nIbCyDD2DzVCuiPoKcK/txg8ULivL61vutQ3t8ySNl3QoMANYUQ5xbZV0TFnmaQ3zREREGzR12Mr2\nobux7FcB7wRWS1pV2j4BnA8slXQG8DDV/bKwvUbSUuAeqiu1zi5XWgGcBSwCJlCdKM/J8oiINpI9\n9KkBSacN1G778mGvaJh0d3e7p2fQnaNoVid8rKaJ12m0QSe8NqAzXh+jZFtIWmm7e6h+zZ4wf0XD\n+H7Aa6m+16NjwyMiIlqn2cNW72t8LOkAYElLKoqIiI63u99h/itgd86DRETEKNDUnoekfwf6DqSN\nA14CLG1VURER0dmaPefxuYbx7cDDtjfsqnNERIxuTR22KjdIvI/qjroHAr9tZVEREdHZmgoPSacA\nK4C3UH0u43ZJg96SPSIiRq9mD1v9DfAK25sBJHUBNwL/2qrCIiKiczUbHnv1BUfxM3b/Sq3ON0o+\n7BMR0SrNhsc3JX0LuKo8PpXqS5kiImIMGuo7zF9I9f0bH5X0l8Cry6QfAVe2uriIiOhMQ+15XAgs\nACi3VP86gKQ/KtPe1NLqIiKiIw113mKS7dX9G0vb9JZUFBERHW+o8DhgkGkThrOQiIjYcwwVHj2S\nzuzfKOndVF8IFRERY9BQ5zw+CFwj6e08HRbdwL7Am1tZWEREdK5Bw8P2JuBYSa8BjirN37D9nZZX\nFhERHavZ7/O4Gbi5xbVERMQeYvR+SjwiIlqmZeEh6TJJmyXd3dB2rqReSavKcFLDtAWS1kq6X9KJ\nDe1HS1pdpl0kdcq9QyIixq5W7nksAuYM0P4F27PKcD2ApJnAPODIMs/FksaV/pcAZwIzyjDQMiMi\nYgS1LDxs3wr8vMnuc4EltrfZXgesBWZLmgzsb/s22wYuB05uTcUREdGsdpzzeJ+kH5fDWgeWtinA\n+oY+G0rblDLev31AkuZL6pHUs2XLluGuOyIiipEOj0uAw4BZwEbgguFcuO2Ftrttd3d1dQ3noiMi\nosGIhoftTbZ32H4KuBSYXSb1AtMauk4tbb1lvH97RES00YiGRzmH0efNQN+VWMuAeZLGSzqU6sT4\nCtsbga2SjilXWZ0GXDuSNUfsROqMIaLNmv0yqNokXQUcB0yUtAH4W+A4SbMAAw8B7wGwvUbSUuAe\nYDtwtu0dZVFnUV25NQG4oQwREdFG8ij9qtPu7m739PTs3syd8p9dp/xuOmF7ZFvsLNtjZ52wPUbJ\ntpC00nb3UP3yCfOIiKgt4REREbUlPCIioraER0RE1JbwiIiI2hIeERFRW8IjIiJqS3hERERtCY+I\niKgt4REREbUlPCIioraER0RE1JbwiIiI2hIeERFRW8IjIiJqS3hERERtCY+IiKgt4REREbUlPCIi\noraWhYekyyRtlnR3Q9tBkpZLeqD8PLBh2gJJayXdL+nEhvajJa0u0y6SOuWLgiMixq5W7nksAub0\nazsHuMn2DOCm8hhJM4F5wJFlnosljSvzXAKcCcwoQ/9lRkTECGtZeNi+Ffh5v+a5wOIyvhg4uaF9\nie1tttcBa4HZkiYD+9u+zbaByxvmiYiINhnpcx6TbG8s448Ak8r4FGB9Q78NpW1KGe/fPiBJ8yX1\nSOrZsmXL8FUdERE7adsJ87In4WFe5kLb3ba7u7q6hnPRERHRYKTDY1M5FEX5ubm09wLTGvpNLW29\nZbx/e0REtNFIh8cy4PQyfjpwbUP7PEnjJR1KdWJ8RTnEtVXSMeUqq9Ma5omIiDbZu1ULlnQVcBww\nUdIG4G+B84Glks4AHgZOAbC9RtJS4B5gO3C27R1lUWdRXbk1AbihDBER0UaqTj2MPt3d3e7p6dm9\nmTvloySd8rvphO2RbbGzbI+ddcL2GCXbQtJK291D9csnzCMioraER0RE1JbwiIiI2hIeERFRW8Ij\nIiJqS3hERERtCY+IiKgt4REREbUlPCIioraER0RE1JbwiIiI2hIeERFRW8IjIiJqS3hERERtCY+I\niKgt4REREbUlPCIioraER0RE1JbwiIiI2toSHpIekrRa0ipJPaXtIEnLJT1Qfh7Y0H+BpLWS7pd0\nYjtqjoiIp7Vzz+M1tmc1fNH6OcBNtmcAN5XHSJoJzAOOBOYAF0sa146CIyKi0kmHreYCi8v4YuDk\nhvYltrfZXgesBWa3ob6IiCjaFR4GbpS0UtL80jbJ9sYy/ggwqYxPAdY3zLuhtEVERJvs3ab1vtp2\nr6TnAcsl3dc40bYlue5CSxDNBzjkkEOGp9KIiPg9bdnzsN1bfm4GrqE6DLVJ0mSA8nNz6d4LTGuY\nfWppG2i5C2132+7u6upqVfkREWPeiIeHpGdKenbfOPDnwN3AMuD00u104NoyvgyYJ2m8pEOBGcCK\nka06IiIateOw1STgGkl96/9n29+UdAewVNIZwMPAKQC210haCtwDbAfOtr2jDXVHREQx4uFh+0Hg\npQO0/wx47S7mOQ84r8WlRUREkzrpUt2IiNhDJDwiIqK2hEdERNSW8IiIiNoSHhERUVvCIyIiakt4\nREREbQmPiIioLeERERG1JTwiIqK2hEdERNSW8IiIiNoSHhERUVvCIyIiakt4REREbQmPiIioLeER\nERG1JTwiIqK2hEdERNSW8IiIiNr2mPCQNEfS/ZLWSjqn3fVERIxle0R4SBoHfBF4PTATeKukme2t\nKiJi7NojwgOYDay1/aDt3wJLgLltrikiYszau90FNGkKsL7h8Qbglf07SZoPzC8Pn5B0/wjUtisT\ngUf/oCVIw1NJ+2Vb7CzbY2fZHk/rhG3xgmY67Snh0RTbC4GF7a4DQFKP7e5219EJsi12lu2xs2yP\np+1J22JPOWzVC0xreDy1tEVERBvsKeFxBzBD0qGS9gXmAcvaXFNExJi1Rxy2sr1d0l8D3wLGAZfZ\nXtPmsobSEYfPOkS2xc6yPXaW7fG0PWZbyHa7a4iIiD3MnnLYKiIiOkjCIyIiakt4DLPcRuVpki6T\ntFnS3e2upRNImibpZkn3SFoj6QPtrqldJO0naYWku8q2+HS7a+oEksZJ+g9J17W7lqEkPIZRbqPy\nexYBc9pdRAfZDnzE9kzgGODsMfz62AYcb/ulwCxgjqRj2lxTJ/gAcG+7i2hGwmN45TYqDWzfCvy8\n3XV0Ctsbbd9Zxh+nepOY0t6q2sOVJ8rDfcowpq/ekTQVeAPw5XbX0oyEx/Aa6DYqY/LNIQYnaTrw\nMuD29lbSPuUQzSpgM7Dc9pjdFsWFwMeAp9pdSDMSHhEjTNKzgKuBD9re2u562sX2DtuzqO4YMVvS\nUe2uqV0kvRHYbHtlu2tpVsJjeOU2KjEoSftQBceVtr/e7no6ge3HgJsZ2+fHXgX8haSHqA53Hy/p\nivaWNLiEx/DKbVRilyQJ+Apwr+3Pt7uedpLUJemAMj4BOAG4r71VtY/tBban2p5O9b7xHdvvaHNZ\ng0p4DCPb24G+26jcCyzdA26j0jKSrgJ+BLxI0gZJZ7S7pjZ7FfBOqv8qV5XhpHYX1SaTgZsl/Zjq\nn67ltjv+8tR4Wm5PEhERtWXPIyIiakt4REREbQmPiIioLeERERG1JTwiIqK2hEdEDZKeL2mJpJ9I\nWinpeklH5M7BMdbsEV9DG9EJyof8rgEW255X2l4KTGprYRFtkD2PiOa9BnjS9pf6GmzfRcPNMCVN\nl/Q9SXeW4djSPlnSreWDgXdL+tNyY8BF5fFqSR8qfQ+X9M2yZ/M9SS8u7W8pfe+SdOvIPvWInWXP\nI6J5RwFD3bhuM3CC7d9ImgFcBXQDbwO+Zfu88r0vz6D6Hospto8C6LtdB7AQeK/tByS9ErgYOB74\nFHCi7d6GvhFtkfCIGF77AP8kaRawAziitN8BXFZujPhvtldJehA4TNI/At8Avl3uuHss8LXqKBkA\n48vPHwCLJC0FclPFaKsctopo3hrg6CH6fAjYBLyUao9jX/jdF2P9J6q7LC+SdJrtX5R+twDvpfoS\noL2Ax2zPahheUpbxXuCTVHduXinpucP8/CKalvCIaN53gPGS5vc1SPpjdr4N/3OAjbaforoJ4rjS\n7wXAJtuXUoXEyyVNBPayfTVVKLy8fL/HOklvKfOpnJRH0uG2b7f9KWBLv/VGjKiER0STXN1F9M3A\n68qlumuAvwMeaeh2MXC6pLuAFwO/Ku3HAXdJ+g/gVOAfqL5l8pbybXpXAAtK37cDZ5RlrOHprzL+\n+3Ji/W7gh8BdrXmmEUPLXXUjIqK27HlERERtCY+IiKgt4REREbUlPCIioraER0RE1JbwiIiI2hIe\nERFR2/8HLH/4AwV7Q/kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ea3f80ba8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "width = 0.5       # the width of the bars\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "rects1 = ax.bar(uniqs, uniq_counts, width, color='r')\n",
    "\n",
    "# add some text for labels, title and axes ticks\n",
    "ax.set_ylabel('Counts')\n",
    "ax.set_xlabel('Classes')\n",
    "ax.set_title('Counts of class membership')\n",
    "ax.set_xticks(uniqs)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Observe which classes you have actually obtained\n",
    "Here you should be able to actually see a pattern beind those!\n",
    "Try looking at some class and ranking words by occurance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 'THE INFORMATION SUBMITTED REFLECTS ALL RELEVANT DATA RECEIVED. IF ADDITIONAL RELEVANT INFORMATION IS RECEIVED, A SUPPLEMENTAL REPORT WILL BE SUBMITTED. CONCOMITANT MEDICAL PRODUCT: PRODUCT ID: ADDR01, IPG, IMPLANTED: 2007-(B)(6). (B)(4).; IT WAS REPORTED THAT DURING A ROUTINE DEVICE REPLACEMENT PROCEDURE, THE PHYSICIAN OBSERVED INSULATION DAMAGE ON THE RIGHT VENTRICULAR (RV) LEAD. THE LEAD WAS CAPPED AND REPLACED. NO PATIENT COMPLICATIONS HAVE BEEN REPORTED AS A RESULT OF THIS EVENT.',\n",
       "       'THE INFORMATION SUBMITTED REFLECTS ALL RELEVANT DATA RECEIVED. IF ADDITIONAL RELEVANT INFORMATION IS RECEIVED, A SUPPLEMENTAL REPORT WILL BE SUBMITTED. (B)(4).; IT WAS REPORTED BY THE PATIENT\\'S PARENT THAT THE DEVICE SYSTEM WAS REPLACED BECAUSE \"SOMETHING WENT WRONG\". ADDITIONAL INFORMATION OBTAINED REPORTED THERE WAS A VISIBLE FRACTURE OF THE VENTRICULAR LEAD. IT WAS FURTHER REPORTED THAT THERE WAS HIGH ATRIAL LEAD CAPTURE THRESHOLD AND THAT THE LEAD WAS NOT CAPTURING APPROPRIATELY. THE LEADS WERE REMOVED AND REPLACED. THE DEVICE WAS ALSO EXPLANTED AND REPLACED DUE TO CONCERN OF INFECTION. NO FURTHER PATIENT COMPLICATIONS HAVE BEEN REPORTED AS A RESULT OF THIS EVENT.',\n",
       "       'IT WAS REPORTED THAT THE PATIENT HAD SEPSIS. THE IMPLANTABLE CARDIOVERTER DEFIBRILLATOR (ICD)\\xa0SYSTEM WAS EXPLANTED. NO FURTHER PATIE NT COMPLICATIONS HAVE BEEN REPORTED AS A RESULT OF THIS EVENT.; THE INFORMATION SUBMITTED REFLECTS ALL RELEVANT DATA RECEIVED. IF ADDITIONAL RELEVANT INFORMATION IS RECEIVED, A SUPPLEMENTAL REPORT WILL BE SUBMITTED. CONCOMITANT MEDICAL PRODUCTS: D314DRG, ICD, IMPLANTED:(B)(6) 2011. (B)(4).',\n",
       "       'IT WAS REPORTED THAT THE LEFT VENTRICULAR (LV) LEAD HAD NO CAPTURE. THE LEAD WAS CAPPED AND REPLACED. NO PATIENT COMPLICATIONS HAVE BEEN REPORTED AS A RESULT OF THIS EVENT.; THE INFORMATION SUBMITTED REFLECTS ALL RELEVANT DATA RECEIVED. IF ADDITIONAL RELEVANT INFORMATION IS RECEIVED, A SUPPLEMENTAL REPORT WILL BE SUBMITTED. (B)(4).'], dtype=object)"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#df[df.columns[df.columns.to_series().str.contains('^text_*')]][km.labels_ == 0].sample(2)\n",
    "np.random.choice(text_np_cat[km.labels_ == 1], size = 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Replace the former text values with a class assigenemnt and check\n",
    "here we use regular expression to match all column names starting with \"text\" (ie we drop alltogether 6 columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df2 = df1.drop(df1.columns[df1.columns.to_series().str.contains('^text_*')], axis = 1)\n",
    "df2['mdr_text_class'] = km.labels_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>previous_use_code</th>\n",
       "      <th>single_use_flag</th>\n",
       "      <th>reprocessed_and_reused_flag</th>\n",
       "      <th>manufacturer_d_country</th>\n",
       "      <th>text_0</th>\n",
       "      <th>text_type_code_0</th>\n",
       "      <th>text_1</th>\n",
       "      <th>text_type_code_1</th>\n",
       "      <th>text_2</th>\n",
       "      <th>text_type_code_2</th>\n",
       "      <th>device_name</th>\n",
       "      <th>device_class</th>\n",
       "      <th>medical_specialty_description</th>\n",
       "      <th>age_of_device_days</th>\n",
       "      <th>days_to_expiry</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3640</th>\n",
       "      <td>I</td>\n",
       "      <td>Y</td>\n",
       "      <td>N</td>\n",
       "      <td>US</td>\n",
       "      <td>IT WAS REPORTED THAT DURING IMPLANT ATTEMPT, T...</td>\n",
       "      <td>Description of Event or Problem</td>\n",
       "      <td>THE INFORMATION SUBMITTED REFLECTS ALL RELEVAN...</td>\n",
       "      <td>Additional Manufacturer Narrative</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Permanent Pacemaker Electrode</td>\n",
       "      <td>3</td>\n",
       "      <td>Cardiovascular</td>\n",
       "      <td>1323 days</td>\n",
       "      <td>1461 days</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9643</th>\n",
       "      <td>I</td>\n",
       "      <td>Y</td>\n",
       "      <td>N</td>\n",
       "      <td>US</td>\n",
       "      <td>THE INFORMATION SUBMITTED REFLECTS ALL RELEVAN...</td>\n",
       "      <td>Additional Manufacturer Narrative</td>\n",
       "      <td>IT WAS REPORTED THAT DURING A FOLLOW-UP VISIT ...</td>\n",
       "      <td>Description of Event or Problem</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Permanent Pacemaker Electrode</td>\n",
       "      <td>3</td>\n",
       "      <td>Cardiovascular</td>\n",
       "      <td>169 days</td>\n",
       "      <td>722 days</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10452</th>\n",
       "      <td>I</td>\n",
       "      <td>Y</td>\n",
       "      <td>N</td>\n",
       "      <td>US</td>\n",
       "      <td>IT WAS REPORTED THAT THE LEAD AND DEVICE WERE ...</td>\n",
       "      <td>Description of Event or Problem</td>\n",
       "      <td>THE INFORMATION SUBMITTED REFLECTS ALL RELEVAN...</td>\n",
       "      <td>Additional Manufacturer Narrative</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Implantable Pacemaker Pulse-Generator</td>\n",
       "      <td>3</td>\n",
       "      <td>Cardiovascular</td>\n",
       "      <td>425 days</td>\n",
       "      <td>711 days</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      previous_use_code single_use_flag reprocessed_and_reused_flag  \\\n",
       "3640                  I               Y                           N   \n",
       "9643                  I               Y                           N   \n",
       "10452                 I               Y                           N   \n",
       "\n",
       "      manufacturer_d_country  \\\n",
       "3640                      US   \n",
       "9643                      US   \n",
       "10452                     US   \n",
       "\n",
       "                                                  text_0  \\\n",
       "3640   IT WAS REPORTED THAT DURING IMPLANT ATTEMPT, T...   \n",
       "9643   THE INFORMATION SUBMITTED REFLECTS ALL RELEVAN...   \n",
       "10452  IT WAS REPORTED THAT THE LEAD AND DEVICE WERE ...   \n",
       "\n",
       "                        text_type_code_0  \\\n",
       "3640     Description of Event or Problem   \n",
       "9643   Additional Manufacturer Narrative   \n",
       "10452    Description of Event or Problem   \n",
       "\n",
       "                                                  text_1  \\\n",
       "3640   THE INFORMATION SUBMITTED REFLECTS ALL RELEVAN...   \n",
       "9643   IT WAS REPORTED THAT DURING A FOLLOW-UP VISIT ...   \n",
       "10452  THE INFORMATION SUBMITTED REFLECTS ALL RELEVAN...   \n",
       "\n",
       "                        text_type_code_1 text_2 text_type_code_2  \\\n",
       "3640   Additional Manufacturer Narrative    NaN              NaN   \n",
       "9643     Description of Event or Problem    NaN              NaN   \n",
       "10452  Additional Manufacturer Narrative    NaN              NaN   \n",
       "\n",
       "                                 device_name device_class  \\\n",
       "3640           Permanent Pacemaker Electrode            3   \n",
       "9643           Permanent Pacemaker Electrode            3   \n",
       "10452  Implantable Pacemaker Pulse-Generator            3   \n",
       "\n",
       "      medical_specialty_description  age_of_device_days  days_to_expiry  \n",
       "3640                 Cardiovascular           1323 days       1461 days  \n",
       "9643                 Cardiovascular            169 days        722 days  \n",
       "10452                Cardiovascular            425 days        711 days  "
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df2['mdr_text_class'] == 1].sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Condition dataframe into ndarray\n",
    "To feed it to sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# X\n",
    "df2['days_to_expiry'] = df2['days_to_expiry'].dt.days\n",
    "\n",
    "X = df2.ix[:, df2.columns != 'age_of_device_days'].values\n",
    "days = df2['age_of_device_days']\n",
    "\n",
    "# y\n",
    "y = days.dt.days.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# load modules\n",
    "from sklearn.preprocessing import Imputer\n",
    "from sklearn import preprocessing\n",
    "from sklearn import linear_model\n",
    "from sklearn.model_selection import KFold\n",
    "\n",
    "# impute missing entries (due to missing expiry date) with mean\n",
    "imp = Imputer(missing_values='NaN', strategy='mean', axis=0)\n",
    "X_imputed = imp.fit_transform(X)\n",
    "\n",
    "# Nomralize\n",
    "X_normalized = preprocessing.normalize(X_imputed, norm='l2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train a predictor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean squared error: 2635849.17\n",
      "Mean squared error: 3256613.67\n",
      "Mean squared error: 2614457.05\n",
      "Mean squared error: 3006008.21\n",
      "Mean squared error: 2641854.05\n",
      "Mean squared error: 2588334.60\n",
      "Mean squared error: 2790047.01\n",
      "Mean squared error: 2714763.71\n",
      "Mean squared error: 2413903.84\n",
      "Mean squared error: 2580651.37\n"
     ]
    }
   ],
   "source": [
    "kf = KFold(n_splits=10, shuffle=True)\n",
    "for train_index, test_index in kf.split(X_normalized):\n",
    "    #print(\"TRAIN:\", train_index, \"TEST:\", test_index)\n",
    "    X_train, X_test = X_normalized[train_index], X_normalized[test_index]\n",
    "    y_train, y_test = y[train_index], y[test_index]\n",
    "    regr = linear_model.LinearRegression()\n",
    "    regr.fit(X_train, y_train)\n",
    "    print(\"Mean squared error: %.2f\" % np.mean((regr.predict(X_test) - y_test) ** 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stop and think\n",
    "    -We have only very few features that are very poor\n",
    "    -So far we haven't tried training on some substantial amount of samples (e.g. 50'000 +)\n",
    "    -Could perhaps branch off a bit and analyse something different?\n",
    "        -Device provenance based on its type?\n",
    "        -Date of expiry based on the type?\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}