optimizing_code_common_books_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimizing Code: Common Books\n",
    "Let's go through an example scenario where we optimize some code to be more efficient. Say we are managing books for a store, and we want to find all the books published within the last two years about code. We have a file that lists all the ids of books published in the last two years, `books_published_last_two_years.txt`, as well as a file for all coding books, `all_coding_books.txt`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what the first few lines of each file looks like.\n",
    "#### `books_published_last_two_years.txt`\n",
    "```txt\n",
    "1262771\n",
    "9011996\n",
    "2007022\n",
    "9389522\n",
    "8181760\n",
    "...\n",
    "```\n",
    "#### `all_coding_books.txt`\n",
    "```txt\n",
    "382944\n",
    "483549\n",
    "103957\n",
    "590274\n",
    "045832\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we want to find all the coding books published within the last two years, we'd want to find the book ids included in both of these files. Your coworker came up with one approach, and shows you this code to find the books in both files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('books_published_last_two_years.txt') as f:\n",
    "    recent_books = f.read().split('\\n')\n",
    "    \n",
    "with open('all_coding_books.txt') as f:\n",
    "    coding_books = f.read().split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "96\n",
      "Duration: 28.02896738052368 seconds\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "recent_coding_books = []\n",
    "\n",
    "for book in recent_books:\n",
    "    if book in coding_books:\n",
    "        recent_coding_books.append(book)\n",
    "\n",
    "print(len(recent_coding_books))\n",
    "print('Duration: {} seconds'.format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Their strategy is to loop through each book in the first file, check if it's contained in the second file, and if so - add it to the final list. This makes sense and is an intuitive first approach. However, there are several things we can do to make this more efficient. Here are some tips."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tip #1: Use vector operations over loops when possible\n",
    "Numpy and pandas are your best friends for this. There are MANY cases in which you can replace loops with Numpy and pandas that use vector operations to make your computations a LOT faster. Sometimes there is a method that does exactly what you need. Other times, you need to be a little creative. This example in particular has a useful method you can use.\n",
    "\n",
    "Let me show you how I would approach this. No joke, I google: \"how to find common elements in two Numpy arrays\" and here are the results I get!\n",
    "\n",
    "In the Jupyter notebook quiz on the next page, use Numpy's `intersect1d` method to get the intersection of the `recent_books` and `coding_books` arrays. I'll give you this same notebook, and I'll put a cell right here with code to record the time it takes to run. Write your line of code in between these time start and time end lines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "start = time.time()\n",
    "recent_coding_books = \n",
    "print(len(recent_coding_books))\n",
    "print('Duration: {} seconds'.format(time.time() - start))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tip #2: Know your data structures and which methods are faster\n",
    "In addition to relying on Numpy and pandas, it's often good to double check whether there's a data structure or method in Python you can use to accomplish your task more effectively. For example, in this case do you recall a data structure in Python that stores a group of unique elements and can quickly compute intersections and unions of different groups? You can read more about why sets are more efficient than lists for this task in the link on the bottom of this page.\n",
    "\n",
    "Also, remember how I said I googled everything? Last time, I was googling how to find common elements in specifically Numpy arrays. But you can go more general and google something like \"how to find common elements in two lists python\" and you'll see posts like [this](https://stackoverflow.com/questions/2864842/common-elements-comparison-between-2-lists) that share and compare different answers. And you can see the set being introduced here.\n",
    "\n",
    "This seems to have a lot of great explanation and votes, but ultimately we should try different methods and compare their efficiency for our example. Because different methods perform differently in different contexts. So it's smart to always test for yourself. In the next cell of the Jupyter notebook, find out how long it takes to compute the common elements of `recent_books` and `coding_books` using Python's `set.intersection` method. Here again is some code to measure how long this takes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "start = time.time()\n",
    "recent_coding_books = \n",
    "print(len(recent_coding_books))\n",
    "print('Duration: {} seconds'.format(time.time() - start))\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('books_published_last_two_years.txt') as f:\n",
    "    recent_books = f.read().split('\\n')\n",
    "\n",
    "with open('all_coding_books.txt') as f:\n",
    "    coding_books = f.read().split('\\n')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "96\n",
      "Duration: 0.08395099639892578 seconds\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "recent_coding_books = np.intersect1d(recent_books, coding_books)\n",
    "print(len(recent_coding_books))\n",
    "print('Duration: {} seconds'.format(time.time() - start))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "ad2bdc8ecc057115af97d19610ffacc2b4e99fae6737bb82f5d7fb13d2f2c186"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}