jupyter-labs-webscraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"https://gitlab.com/ibm/skills-network/courses/placeholder101/-/raw/master/labs/module%201/images/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Space X  Falcon 9 First Stage Landing Prediction**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Web scraping Falcon 9 and Falcon Heavy Launches Records from Wikipedia\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **40** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, you will be performing web scraping to collect Falcon 9 historical launch records from a Wikipedia page titled `List of Falcon 9 and Falcon Heavy launches`\n",
    "\n",
    "[https://en.wikipedia.org/wiki/List_of_Falcon\\_9\\_and_Falcon_Heavy_launches](https://en.wikipedia.org/wiki/List_of_Falcon\\_9\\_and_Falcon_Heavy_launches?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDS0321ENSkillsNetwork26802033-2021-01-01)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DS0321EN-SkillsNetwork/labs/module\\_1\\_L2/images/Falcon9\\_rocket_family.svg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Falcon 9 first stage will land successfully\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0701EN-SkillsNetwork/api/Images/landing\\_1.gif)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Several examples of an unsuccessful landing are shown here:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0701EN-SkillsNetwork/api/Images/crash.gif)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More specifically, the launch records are stored in a HTML table shown below:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DS0321EN-SkillsNetwork/labs/module\\_1\\_L2/images/falcon9-launches-wiki.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "\n",
    "Web scrap Falcon 9 launch records with `BeautifulSoup`:\n",
    "\n",
    "*   Extract a Falcon 9 launch records HTML table from Wikipedia\n",
    "*   Parse the table and convert it into a Pandas data frame\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's import required packages for this lab\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install beautifulsoup4\n",
    "!pip3 install requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "import unicodedata\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and we will provide some helper functions for you to process web scraped HTML table\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def date_time(table_cells):\n",
    "    \"\"\"\n",
    "    This function returns the data and time from the HTML  table cell\n",
    "    Input: the  element of a table data cell extracts extra row\n",
    "    \"\"\"\n",
    "    return [data_time.strip() for data_time in list(table_cells.strings)][0:2]\n",
    "\n",
    "def booster_version(table_cells):\n",
    "    \"\"\"\n",
    "    This function returns the booster version from the HTML  table cell \n",
    "    Input: the  element of a table data cell extracts extra row\n",
    "    \"\"\"\n",
    "    out=''.join([booster_version for i,booster_version in enumerate( table_cells.strings) if i%2==0][0:-1])\n",
    "    return out\n",
    "\n",
    "def landing_status(table_cells):\n",
    "    \"\"\"\n",
    "    This function returns the landing status from the HTML table cell \n",
    "    Input: the  element of a table data cell extracts extra row\n",
    "    \"\"\"\n",
    "    out=[i for i in table_cells.strings][0]\n",
    "    return out\n",
    "\n",
    "\n",
    "def get_mass(table_cells):\n",
    "    mass=unicodedata.normalize(\"NFKD\", table_cells.text).strip()\n",
    "    if mass:\n",
    "        mass.find(\"kg\")\n",
    "        new_mass=mass[0:mass.find(\"kg\")+2]\n",
    "    else:\n",
    "        new_mass=0\n",
    "    return new_mass\n",
    "\n",
    "\n",
    "def extract_column_from_header(row):\n",
    "    \"\"\"\n",
    "    This function returns the landing status from the HTML table cell \n",
    "    Input: the  element of a table data cell extracts extra row\n",
    "    \"\"\"\n",
    "    if (row.br):\n",
    "        row.br.extract()\n",
    "    if row.a:\n",
    "        row.a.extract()\n",
    "    if row.sup:\n",
    "        row.sup.extract()\n",
    "        \n",
    "    colunm_name = ' '.join(row.contents)\n",
    "    \n",
    "    # Filter the digit and empty names\n",
    "    if not(colunm_name.strip().isdigit()):\n",
    "        colunm_name = colunm_name.strip()\n",
    "        return colunm_name    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep the lab tasks consistent, you will be asked to scrape the data from a snapshot of the  `List of Falcon 9 and Falcon Heavy launches` Wikipage updated on\n",
    "`9th June 2021`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "static_url = \"https://en.wikipedia.org/w/index.php?title=List_of_Falcon_9_and_Falcon_Heavy_launches&oldid=1027686922\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, request the HTML page from the above URL and get a `response` object\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TASK 1: Request the Falcon9 Launch Wiki page from its URL\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's perform an HTTP GET method to request the Falcon9 Launch HTML page, as an HTTP response.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use requests.get() method with the provided static_url\n",
    "page=requests.get(static_url)\n",
    "page.status_code\n",
    "# assign the response to a object\n",
    "# response.json()\n",
    "#data = pd.json_normalize(response.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a `BeautifulSoup` object from the HTML `response`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use BeautifulSoup() to create a BeautifulSoup object from a response text content\n",
    "soup = BeautifulSoup(page.content, 'html.parser')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the page title to verify if the `BeautifulSoup` object was created properly\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<title>List of Falcon 9 and Falcon Heavy launches - Wikipedia</title>\n"
     ]
    }
   ],
   "source": [
    "# Use soup.title attribute\n",
    "print(soup.title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TASK 2: Extract all column/variable names from the HTML table header\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we want to collect all relevant column names from the HTML table header\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to find all tables on the wiki page first. If you need to refresh your memory about `BeautifulSoup`, please check the external reference link towards the end of this lab\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the find_all function in the BeautifulSoup object, with element type `table`\n",
    "# Assign the result to a list called `html_tables`\n",
    "html_tables=soup.find_all('tr')\n",
    "#print(html_tables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Starting from the third table is our target table contains the actual launch records.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tr>\n",
      "<th scope=\"col\">Flight No.\n",
      "</th>\n",
      "<th scope=\"col\">Date and<br/>time (<a href=\"/wiki/Coordinated_Universal_Time\" title=\"Coordinated Universal Time\">UTC</a>)\n",
      "</th>\n",
      "<th scope=\"col\"><a href=\"/wiki/List_of_Falcon_9_first-stage_boosters\" title=\"List of Falcon 9 first-stage boosters\">Version,<br/>Booster</a> <sup class=\"reference\" id=\"cite_ref-booster_11-0\"><a href=\"#cite_note-booster-11\">[b]</a></sup>\n",
      "</th>\n",
      "<th scope=\"col\">Launch site\n",
      "</th>\n",
      "<th scope=\"col\">Payload<sup class=\"reference\" id=\"cite_ref-Dragon_12-0\"><a href=\"#cite_note-Dragon-12\">[c]</a></sup>\n",
      "</th>\n",
      "<th scope=\"col\">Payload mass\n",
      "</th>\n",
      "<th scope=\"col\">Orbit\n",
      "</th>\n",
      "<th scope=\"col\">Customer\n",
      "</th>\n",
      "<th scope=\"col\">Launch<br/>outcome\n",
      "</th>\n",
      "<th scope=\"col\"><a href=\"/wiki/Falcon_9_first-stage_landing_tests\" title=\"Falcon 9 first-stage landing tests\">Booster<br/>landing</a>\n",
      "</th></tr>\n"
     ]
    }
   ],
   "source": [
    "# Let's print the third table and check its content\n",
    "first_launch_table = html_tables[2]\n",
    "print(first_launch_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should able to see the columns names embedded in the table header elements `<th>` as follows:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "<tr>\n",
    "<th scope=\"col\">Flight No.\n",
    "</th>\n",
    "<th scope=\"col\">Date and<br/>time (<a href=\"/wiki/Coordinated_Universal_Time\" title=\"Coordinated Universal Time\">UTC</a>)\n",
    "</th>\n",
    "<th scope=\"col\"><a href=\"/wiki/List_of_Falcon_9_first-stage_boosters\" title=\"List of Falcon 9 first-stage boosters\">Version,<br/>Booster</a> <sup class=\"reference\" id=\"cite_ref-booster_11-0\"><a href=\"#cite_note-booster-11\">[b]</a></sup>\n",
    "</th>\n",
    "<th scope=\"col\">Launch site\n",
    "</th>\n",
    "<th scope=\"col\">Payload<sup class=\"reference\" id=\"cite_ref-Dragon_12-0\"><a href=\"#cite_note-Dragon-12\">[c]</a></sup>\n",
    "</th>\n",
    "<th scope=\"col\">Payload mass\n",
    "</th>\n",
    "<th scope=\"col\">Orbit\n",
    "</th>\n",
    "<th scope=\"col\">Customer\n",
    "</th>\n",
    "<th scope=\"col\">Launch<br/>outcome\n",
    "</th>\n",
    "<th scope=\"col\"><a href=\"/wiki/Falcon_9_first-stage_landing_tests\" title=\"Falcon 9 first-stage landing tests\">Booster<br/>landing</a>\n",
    "</th></tr>\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we just need to iterate through the `<th>` elements and apply the provided `extract_column_from_header()` to extract column name one by one\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_names = []\n",
    "\n",
    "# Apply find_all() function with `th` element on first_launch_table\n",
    "# Iterate each th element and apply the provided extract_column_from_header() to get a column name\n",
    "# Append the Non-empty column name (`if name is not None and len(name) > 0`) into a list called column_names\n",
    "for row in first_launch_table.find_all('th'):\n",
    "    name = extract_column_from_header(row)\n",
    "    if (name != None and len(name) > 0):\n",
    "        column_names.append(name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the extracted column names\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Flight No.', 'Date and time ( )', 'Launch site', 'Payload', 'Payload mass', 'Orbit', 'Customer', 'Launch outcome']\n"
     ]
    }
   ],
   "source": [
    "print(column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TASK 3: Create a data frame by parsing the launch HTML tables\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will create an empty dictionary with keys from the extracted column names in the previous task. Later, this dictionary will be converted into a Pandas dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "launch_dict= dict.fromkeys(column_names)\n",
    "\n",
    "# Remove an irrelvant column\n",
    "del launch_dict['Date and time ( )']\n",
    "\n",
    "# Let's initial the launch_dict with each value to be an empty list\n",
    "launch_dict['Flight No.'] = []\n",
    "launch_dict['Launch site'] = []\n",
    "launch_dict['Payload'] = []\n",
    "launch_dict['Payload mass'] = []\n",
    "launch_dict['Orbit'] = []\n",
    "launch_dict['Customer'] = []\n",
    "launch_dict['Launch outcome'] = []\n",
    "# Added some new columns\n",
    "launch_dict['Version Booster']=[]\n",
    "launch_dict['Booster landing']=[]\n",
    "launch_dict['Date']=[]\n",
    "launch_dict['Time']=[]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we just need to fill up the `launch_dict` with launch records extracted from table rows.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually, HTML tables in Wiki pages are likely to contain unexpected annotations and other types of noises, such as reference links `B0004.1[8]`, missing values `N/A [e]`, inconsistent formatting, etc.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify the parsing process, we have provided an incomplete code snippet below to help you to fill up the `launch_dict`. Please complete the following code snippet with TODOs or you can choose to write your own logic to parse all launch tables:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F9 v1.0B0003.1\n",
      "F9 v1.0B0004.1\n",
      "F9 v1.0B0005.1\n",
      "F9 v1.0B0006.1\n",
      "F9 v1.0B0007.1\n",
      "F9 v1.1B1003\n",
      "F9 v1.1\n",
      "F9 v1.1\n",
      "F9 v1.1\n",
      "F9 v1.1\n",
      "F9 v1.1\n",
      "F9 v1.1\n",
      "F9 v1.1\n",
      "F9 v1.1\n",
      "F9 v1.1\n",
      "F9 v1.1\n",
      "F9 v1.1\n",
      "F9 v1.1\n",
      "F9 v1.1\n",
      "F9 FT\n",
      "F9 v1.1\n",
      "F9 FT\n",
      "F9 FT\n",
      "F9 FT\n",
      "F9 FT\n",
      "F9 FT\n",
      "F9 FT\n",
      "F9 FT\n",
      "F9 FT\n",
      "F9 FT\n",
      "F9 FT\n",
      "F9 FT♺\n",
      "F9 FT\n",
      "F9 FT\n",
      "F9 FT\n",
      "F9 FTB1029.2\n",
      "F9 FT\n",
      "F9 FT\n",
      "F9 B4\n",
      "F9 FT\n",
      "F9 B4\n",
      "F9 B4\n",
      "F9 FTB1031.2\n",
      "F9 B4\n",
      "F9 FTB1035.2\n",
      "F9 FTB1036.2\n",
      "F9 B4\n",
      "F9 FTB1032.2\n",
      "F9 FTB1038.2\n",
      "F9 B4\n",
      "F9 B4B1041.2\n",
      "F9 B4B1039.2\n",
      "F9 B4\n",
      "F9 B5B1046.1\n",
      "F9 B4B1043.2\n",
      "F9 B4B1040.2\n",
      "F9 B4B1045.2\n",
      "F9 B5\n",
      "F9 B5B1048\n",
      "F9 B5B1046.2\n",
      "F9 B5\n",
      "F9 B5B1048.2\n",
      "F9 B5B1047.2\n",
      "F9 B5B1046.3\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5B1049.2\n",
      "F9 B5B1048.3\n",
      "F9 B5[268]\n",
      "F9 B5\n",
      "F9 B5B1049.3\n",
      "F9 B5B1051.2\n",
      "F9 B5B1056.2\n",
      "F9 B5B1047.3\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5B1056.3\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5B1058.2\n",
      "F9 B5\n",
      "F9 B5B1049.6\n",
      "F9 B5\n",
      "F9 B5B1060.2\n",
      "F9 B5B1058.3\n",
      "F9 B5B1051.6\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5\n",
      "F9 B5 ♺\n",
      "F9 B5 ♺\n",
      "F9 B5 ♺\n",
      "F9 B5 ♺\n",
      "F9 B5\n",
      "F9 B5B1051.8\n",
      "F9 B5B1058.5\n"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'NoneType' object has no attribute 'string'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-d42b74696891>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     60\u001b[0m             \u001b[0;31m# Customer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     61\u001b[0m             \u001b[0;31m# TODO: Append the customer into launch_dict with key `Customer`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 62\u001b[0;31m             \u001b[0mcustomer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrow\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m6\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstring\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     63\u001b[0m             \u001b[0;31m#print(customer)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     64\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'string'"
     ]
    }
   ],
   "source": [
    "extracted_row = 0\n",
    "#Extract each table \n",
    "for table_number,table in enumerate(soup.find_all('table',\"wikitable plainrowheaders collapsible\")):\n",
    "   # get table row \n",
    "    for rows in table.find_all(\"tr\"):\n",
    "        #check to see if first table heading is as number corresponding to launch a number \n",
    "        if rows.th:\n",
    "            if rows.th.string:\n",
    "                flight_number=rows.th.string.strip()\n",
    "                flag=flight_number.isdigit()\n",
    "        else:\n",
    "            flag=False\n",
    "        #get table element \n",
    "        row=rows.find_all('td')\n",
    "        #if it is number save cells in a dictonary \n",
    "        if flag:\n",
    "            extracted_row += 1\n",
    "            # Flight Number value\n",
    "            # TODO: Append the flight_number into launch_dict with key `Flight No.`\n",
    "            #print(flight_number)\n",
    "            datatimelist=date_time(row[0])\n",
    "            \n",
    "            # Date value\n",
    "            # TODO: Append the date into launch_dict with key `Date`\n",
    "            date = datatimelist[0].strip(',')\n",
    "            #print(date)\n",
    "            \n",
    "            # Time value\n",
    "            # TODO: Append the time into launch_dict with key `Time`\n",
    "            time = datatimelist[1]\n",
    "            #print(time)\n",
    "              \n",
    "            # Booster version\n",
    "            # TODO: Append the bv into launch_dict with key `Version Booster`\n",
    "            bv=booster_version(row[1])\n",
    "            if not(bv):\n",
    "                bv=row[1].a.string\n",
    "            print(bv)\n",
    "            \n",
    "            # Launch Site\n",
    "            # TODO: Append the bv into launch_dict with key `Launch Site`\n",
    "            launch_site = row[2].a.string\n",
    "            #print(launch_site)\n",
    "            \n",
    "            # Payload\n",
    "            # TODO: Append the payload into launch_dict with key `Payload`\n",
    "            payload = row[3].a.string\n",
    "            #print(payload)\n",
    "            \n",
    "            # Payload Mass\n",
    "            # TODO: Append the payload_mass into launch_dict with key `Payload mass`\n",
    "            payload_mass = get_mass(row[4])\n",
    "            #print(payload)\n",
    "            \n",
    "            # Orbit\n",
    "            # TODO: Append the orbit into launch_dict with key `Orbit`\n",
    "            orbit = row[5].a.string\n",
    "            #print(orbit)\n",
    "            \n",
    "            # Customer\n",
    "            # TODO: Append the customer into launch_dict with key `Customer`\n",
    "            customer = row[6].a.string\n",
    "            #print(customer)\n",
    "            \n",
    "            # Launch outcome\n",
    "            # TODO: Append the launch_outcome into launch_dict with key `Launch outcome`\n",
    "            launch_outcome = list(row[7].strings)[0]\n",
    "            #print(launch_outcome)\n",
    "            \n",
    "            # Booster landing\n",
    "            # TODO: Append the launch_outcome into launch_dict with key `Booster landing`\n",
    "            booster_landing = landing_status(row[8])\n",
    "            #print(booster_landing)\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you have fill in the parsed launch record values into `launch_dict`, you can create a dataframe from it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=pd.DataFrame(launch_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now export it to a <b>CSV</b> for the next section, but to make the answers consistent and in case you have difficulties finishing this lab.\n",
    "\n",
    "Following labs will be using a provided dataset to make each lab independent.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('spacex_web_scraped.csv', index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.linkedin.com/in/yan-luo-96288783/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDS0321ENSkillsNetwork26802033-2021-01-01\">Yan Luo</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.linkedin.com/in/nayefaboutayoun/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDS0321ENSkillsNetwork26802033-2021-01-01\">Nayef Abou Tayoun</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Date (YYYY-MM-DD) | Version | Changed By | Change Description          |\n",
    "| ----------------- | ------- | ---------- | --------------------------- |\n",
    "| 2021-06-09        | 1.0     | Yan Luo    | Tasks updates               |\n",
    "| 2020-11-10        | 1.0     | Nayef      | Created the initial version |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2021 IBM Corporation. All rights reserved.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}