2_spark_maps_and_lazy_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Maps\n",
    "\n",
    "In Spark, maps take data as input and then transform that data with whatever function you put in the map. They are like directions for the data telling how each input should get to the output.\n",
    "\n",
    "The first code cell creates a SparkContext object. With the SparkContext, you can input a dataset and parallelize the data across a cluster (since you are currently using Spark in local mode on a single machine, technically the dataset isn't distributed yet).\n",
    "\n",
    "Run the code cell below to instantiate a SparkContext object and then read in the log_of_songs list into Spark. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "### \n",
    "# You might have noticed this code in the screencast.\n",
    "#\n",
    "#import findspark\n",
    "#findspark.init('spark-2.3.2-bin-hadoop2.7')\n",
    "#\n",
    "# The findspark Python module makes it easier to install\n",
    "# Spark in local mode on your computer. This is convenient\n",
    "# for practicing Spark syntax locally. \n",
    "# However, the workspaces already have Spark installed and you do not\n",
    "# need to use the findspark module\n",
    "#\n",
    "###\n",
    "\n",
    "import pyspark\n",
    "sc = pyspark.SparkContext(appName=\"maps_and_lazy_evaluation_example\")\n",
    "\n",
    "log_of_songs = [\n",
    "        \"Despacito\",\n",
    "        \"Nice for what\",\n",
    "        \"No tears left to cry\",\n",
    "        \"Despacito\",\n",
    "        \"Havana\",\n",
    "        \"In my feelings\",\n",
    "        \"Nice for what\",\n",
    "        \"despacito\",\n",
    "        \"All the stars\"\n",
    "]\n",
    "\n",
    "# parallelize the log_of_songs to use with Spark\n",
    "distributed_song_log = sc.parallelize(log_of_songs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next code cell defines a function that converts a song title to lowercase. Then there is an example converting the word \"Havana\" to \"havana\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'havana'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def convert_song_to_lowercase(song):\n",
    "    return song.lower()\n",
    "\n",
    "convert_song_to_lowercase(\"Havana\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code cells demonstrate how to apply this function using a map step. The map step will go through each song in the list and apply the convert_song_to_lowercase() function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PythonRDD[1] at RDD at PythonRDD.scala:53"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distributed_song_log.map(convert_song_to_lowercase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that this code cell ran quite quickly. This is because of lazy evaluation. Spark does not actually execute the map step unless it needs to.\n",
    "\n",
    "\"RDD\" in the output refers to resilient distributed dataset. RDDs are exactly what they say they are: fault-tolerant datasets distributed across a cluster. This is how Spark stores data. \n",
    "\n",
    "To get Spark to actually run the map step, you need to use an \"action\". One available action is the collect method. The collect() method takes the results from all of the clusters and \"collects\" them into a single list on the master node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['despacito',\n",
       " 'nice for what',\n",
       " 'no tears left to cry',\n",
       " 'despacito',\n",
       " 'havana',\n",
       " 'in my feelings',\n",
       " 'nice for what',\n",
       " 'despacito',\n",
       " 'all the stars']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distributed_song_log.map(convert_song_to_lowercase).collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note as well that Spark is not changing the original data set: Spark is merely making a copy. You can see this by running collect() on the original dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Despacito',\n",
       " 'Nice for what',\n",
       " 'No tears left to cry',\n",
       " 'Despacito',\n",
       " 'Havana',\n",
       " 'In my feelings',\n",
       " 'Nice for what',\n",
       " 'despacito',\n",
       " 'All the stars']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distributed_song_log.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You do not always have to write a custom function for the map step. You can also use anonymous (lambda) functions as well as built-in Python functions like string.lower(). \n",
    "\n",
    "Anonymous functions are actually a Python feature for writing functional style programs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['despacito',\n",
       " 'nice for what',\n",
       " 'no tears left to cry',\n",
       " 'despacito',\n",
       " 'havana',\n",
       " 'in my feelings',\n",
       " 'nice for what',\n",
       " 'despacito',\n",
       " 'all the stars']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distributed_song_log.map(lambda song: song.lower()).collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['despacito',\n",
       " 'nice for what',\n",
       " 'no tears left to cry',\n",
       " 'despacito',\n",
       " 'havana',\n",
       " 'in my feelings',\n",
       " 'nice for what',\n",
       " 'despacito',\n",
       " 'all the stars']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distributed_song_log.map(lambda x: x.lower()).collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}