Import from Cloudant Python example.ipynb

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "source": "# Python example using Spark SQL over Cloudant as a source\n\nThis sample notebook is written in Python and expects the Python 3.5 or higher runtime. Make sure the kernel is started and you are connected to it when executing this notebook.\nThe data source for this example can be found at: http://examples.cloudant.com/crimes/. Replicate the database into your own Cloudant account before you execute this script.\n\nThis Python notebook showcases how to use the SQL-Cloudant connector. This code reads Cloudant data, creates a DataFrame from the Cloudant data, filters that data down to only crime incidents with the nature code for a public disturbance, and then writes those 7 documents to another Cloudant database.", 
            "cell_type": "markdown", 
            "metadata": {}
        },
        {
            "source": "## 1. Work with SparkSession\n\nImport and initialize SparkSession.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": true
            }, 
            "outputs": [], 
            "source": "from pyspark.sql import SparkSession"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": true
            }, 
            "outputs": [], 
            "source": "spark = SparkSession.builder.getOrCreate()"
        }, 
        {
            "source": "## 2. Work with a Cloudant database\n\nA Dataframe object can be created directly from a Cloudant database. To configure the database as source, pass these options:\n\n1 - package name that provides the classes (like CloudantDataSource) implemented in the connector to extend BaseRelation. For the SQL-Cloudant connector this will be org.apache.bahir.cloudant\n\n2 - cloudant.host parameter to pass the Cloudant account name\n\n3 - cloudant.user parameter to pass the Cloudant user name\n\n4 - cloudant.password parameter to pass the Cloudant account password\n\n5 - the database to load\n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": true
            }, 
            "outputs": [], 
            "source": "cloudantdata = spark.read.format(\"org.apache.bahir.cloudant\")\\\n    .option(\"cloudant.host\",\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-bluemix.cloudant.com\")\\\n    .option(\"cloudant.username\", \"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-bluemix\")\\\n    .option(\"cloudant.password\",\"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\")\\\n    .load(\"crimes\")"
        }, 
        {
            "source": "## 3. Work with a Dataframe\n\nAt this point, all transformations and functions should behave as specified with Spark SQL. (http://spark.apache.org/sql/)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# This code prints the schema and a record count\ncloudantdata.printSchema()\ncloudantdata.count()"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# This code displays the values of the naturecode field\ncloudantdata.select(\"properties.naturecode\").show()"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# This code filters the data to just those records with a naturecode of \"DISTRB\", and then displays that data\ndisturbDF = cloudantdata.filter(\"properties.naturecode = 'DISTRB'\")\ndisturbDF.show()"
        }, 
        {
        	"execution_count": null,
        	"cell_type": "code",
        	"metadata": {
        		"collapsed": true
        	},
        	"outputs": [],
        	"source": "# This code writes the filtered data to a Cloudant database called crimes_filtered. If the Cloudant database exists, the documents will be added to the database.\n# If the database does not exist, set the createDBOnSave option to 'true'.\ndisturbDF.select(\"properties\").write.format(\"org.apache.bahir.cloudant\")\\\n     .option(\"cloudant.host\",\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-bluemix.cloudant.com\")\\\n     .option(\"cloudant.username\", \"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-bluemix\")\\\n     .option(\"cloudant.password\",\"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\")\\\n     .option(\"createDBOnSave\", \"true\")\\\n     .save(\"crimes_filtered\")"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# Next, you'll see how to create a visualization of the crimes data. \n# First, this line creates a DataFrame containing all of the naturecodes and a count of the crime incidents for each code.\nreducedValue = cloudantdata.groupBy(\"properties.naturecode\").count()\nreducedValue.printSchema()"
        }, 
        {
            "source": "## 4. Generate visualizations", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": true
            }, 
            "outputs": [], 
            "source": "# This line imports two Python modules. The pprint module helps to produce pretty representations of data structures, \n# and the counter subclass from the collections module helps to count hashable objects.\nimport pprint\nfrom collections import Counter"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": true
            }, 
            "outputs": [], 
            "source": "# This line imports PySpark classes for Spark SQL and DataFrames.\nfrom pyspark.sql import *\nfrom pyspark.sql.functions import udf, asc, desc\nfrom pyspark import SparkContext, SparkConf\nfrom pyspark.sql.types import IntegerType"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# This line converts an Apache Spark DataFrame to a Panda DataFrame, and also sorts the DataFrame by count first, \n# and then by naturecode second in order to produce a sorted graph later.\nimport pandas as pd\npandaDF = reducedValue.orderBy(desc(\"count\"), asc(\"naturecode\")).toPandas()\nprint(pandaDF)"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": true
            }, 
            "outputs": [], 
            "source": "# This is needed to actually see the plots\n%matplotlib inline\n\n# This line imports matplotlib.pyplot which is a collection of command style functions that make matplotlib work like MATLAB\nimport matplotlib.pyplot as plt"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# These lines assign the data to the values and labels objects.\nvalues = pandaDF['count']\nlabels = pandaDF['naturecode']\n\n# These lines provide the format for the plot.\nplt.gcf().set_size_inches(16, 12, forward=True)\nplt.title('Number of crimes by type')\n\n# These lines specify that the plot should display as a horizontal bar chart with values being for the x axis \n# and labels for the y axis\nplt.barh(range(len(values)), values)\nplt.yticks(range(len(values)), labels)\n\n# This last line displays the plot\nplt.show()"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "collapsed": true
            }, 
            "outputs": [], 
            "source": ""
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5", 
            "name": "python3", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "3.5.5", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython3", 
            "codemirror_mode": {
                "version": 3, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}