multimedia_popularity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "established-fever",
   "metadata": {},
   "source": [
    "# RickyRenuncia Project\n",
    "\n",
    "The team of the RickyRenuncia Project managed multiple adquicision procedures to preserve the incidents that occured during the summer 2019 related to the leave of office of Ex-governor Ricardo Rosello Nevarez.\n",
    "\n",
    "\n",
    "## Physical Media vs Digital Donations\n",
    "\n",
    "The team collected artifacts and banners used during the demonstrations. When ever possible the artifacts where accompanied by audio interview and/or photograph of the demonstrators that produced and used this artifacts. Through social media and online word-of-mouth the team also contacted the community requesting imagery and content related to the activities of that summer.\n",
    "\n",
    "## Twitter Data Collection\n",
    "\n",
    "In order to have a broad view of the many activities and demonstratiosn around the globe, one of the team members, Joel Blanco, decided to capture records of tweet activity in the web. This data was captured life during the days of the incident and requires processing and analysis to provide a valid interpretation of the information adquired.\n",
    "\n",
    "A cleaned version of this dataset occupies over `7 gigabytes` but fits into `777 megabytes` when <ins>compressed</ins> using `gzip`. Full text data can generally be easily compressed. Bellow we calculate the benefit of compressing this specific dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0300249e-bad5-471f-9536-96f111d326a6",
   "metadata": {},
   "source": [
    "# Notebook Requirements\n",
    "\n",
    "Before utilizing this notebook the user will need initialize environment variables as specified at [Developer_Registration.ipynb](./Developer_Registration.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "important-default",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Calculare the storage benefits of compression\n",
    "\n",
    "# Observations\n",
    "original_size_G = 7\n",
    "final_size_M = 777\n",
    "\n",
    "# Unit transformation\n",
    "giga_to_mega_rate = 1024.0\n",
    "original_size_M = original_size_G * giga_to_mega_rate\n",
    "\n",
    "# Calculate percent change\n",
    "new_size_to_old_size = final_size_M / original_size_M\n",
    "new_size_percent = new_size_to_old_size * 100.0\n",
    "space_freed_percent = 100 - new_size_percent\n",
    "\n",
    "print(\n",
    "    \"The storage was reduced to {:.1f}%.\\nAfter compression, {:.1f}% from the originaly occupied space was freed.\".\n",
    "    format(new_size_percent, space_freed_percent)\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "quarterly-restaurant",
   "metadata": {},
   "source": [
    "The benefits can be very big specially for long term storage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "broad-optimum",
   "metadata": {},
   "source": [
    "# Twitter Data: What we collected?\n",
    "\n",
    "It is important to understand the type of data that is collected from a social media API (application programable interface). The file `Data/Joel/tweetsRickyRenuncia-final.jsonl` is of jsonl format. If you are familiar with json files then this format is a composition of multiple `json` strings each in a new line, the 'L' stands for line (`jsonl = json-l = json-line`).\n",
    "\n",
    "This data set was collected from Twitter in 2019. The Twitter API rescently went through an update, however this data uses the previous API conventions. We will use Pythons `json` library to parse a random line from the source data to help you visualize the structure of this data. Observe that some of the content is readily availble (text field), while others are harder to parse (media url).\n",
    "\n",
    "The full list of tweet ids is available [here](https://ia601005.us.archive.org/31/items/tweetsRickyRenuncia-final/tweetsRickyRenuncia-final.txt).\n",
    "\n",
    "Bellow we show how a try/except and while loops can be used to loop through the data until a post with images is found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "obvious-surfing",
   "metadata": {},
   "outputs": [],
   "source": [
    "dir_path = os.getcwd()\n",
    "print(dir_path)\n",
    "#os.chdir(\"/home/torrien/\")\n",
    "#dir_path = os.getcwd()\n",
    "#print(dir_path)\n",
    "#print(os.listdir())\n",
    "#print(os.listfile())\n",
    "JL_DATA=\"/home/rickyrenuncia/tweetsRickyRenuncia-final.jsonl\"\n",
    "\n",
    "# Get the SAMPLE_SIZE\n",
    "SAMPLE_SIZE = 0.\n",
    "with open(JL_DATA, \"r\") as data_handler:\n",
    "    for line in data_handler:\n",
    "        if line != \"\\n\":\n",
    "            SAMPLE_SIZE += 1.\n",
    "print(f\"Sample Size:{int(SAMPLE_SIZE)}\\n\\n\")\n",
    "\n",
    "\n",
    "# Get a random integer to skip before taking single sample\n",
    "# Try seeds 1 and 16 or any you want to test\n",
    "seed(1)\n",
    "skip_lines=randint(0,int(SAMPLE_SIZE-1))\n",
    "\n",
    "\n",
    "# Reopen file using the with-open-as style and print out a single sample\n",
    "with open(JL_DATA, 'r') as data_handler:\n",
    "    # Use next to skip a line, the for loop allows skipping multiple lines\n",
    "    for _ in range(skip_lines):\n",
    "        next(data_handler)\n",
    "    \n",
    "    while True:\n",
    "        # Loop until a tweet with media.\n",
    "        try:\n",
    "            # Capture string\n",
    "            raw_data = data_handler.readline()\n",
    "            \n",
    "            # Verify if the json has any 'meda_url_https' keys.\n",
    "            if 'media_url_https' not in raw_data:\n",
    "                continue\n",
    "            data = json.loads(raw_data)\n",
    "        except:\n",
    "            break\n",
    "        try:\n",
    "            i = 0\n",
    "            while True:\n",
    "                try:\n",
    "                    media_url = data['retweeted_status']['entities']['media'][i]['media_url_https']\n",
    "                except:\n",
    "                    i += 1\n",
    "                    if i > 10:\n",
    "                        media_url = \"Could not quickly find a tweet with media.\"\n",
    "                        raise #Pass error to previous try/except.\n",
    "                    continue\n",
    "                break   \n",
    "        except:\n",
    "            continue\n",
    "        \n",
    "        \n",
    "        print(\"Text:\", data['text'])\n",
    "        # The Tweet URL is a twitter convention where both the tweet ID and the user's screen_name are required to access the status.\n",
    "        print(\"Tweet URL using user's screen_name:\", f\"https://twitter.com/{data['user']['screen_name']}/status/{data['id_str']}\")\n",
    "        print(\"Tweet URL using user's ID         :\", f\"https://twitter.com/{data['user']['id_str']}/status/{data['id_str']}\")\n",
    "        print(\"Media:\", media_url)\n",
    "#         print(f\"In replay to: {json.dumps(data['retweeted_status'], indent=1)}\")\n",
    "        print(\"\\n\")\n",
    "        # The indent and sort_keys in json.dumps \"prettify\" the output. Still not pretty.\n",
    "#         print(\"Raw Data:\")\n",
    "#         print(\"#\"*50)\n",
    "#         print(json.dumps(data, indent=4, sort_keys=True))\n",
    "#         print(\"#\"*50)\n",
    "        break\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "presidential-brazil",
   "metadata": {},
   "source": [
    "retweeted_posts, handler)## Study the old Twitter API\n",
    "Documentation on the old twitter API version 1.1 can be found [here](https://developer.twitter.com/en/docs/twitter-api/v1) and a sample [over here](https://developer.twitter.com/en/docs/twitter-api/v1/tweets/sample-realtime/api-reference/get-statuses-sample)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "subject-tomato",
   "metadata": {},
   "source": [
    "# What data is available\n",
    "\n",
    "As data analysts we need to understand the data before we can set goals. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "alike-wyoming",
   "metadata": {},
   "outputs": [],
   "source": [
    "SAMPLE_SIZE = 1113758\n",
    "data = TweetJLAnalyzer(JL_DATA, reset=True, local_media=False, cache_size=2000)\n",
    "size=getsizeof(data)\n",
    "print(str(size))\n",
    "print(str(size/1024.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "committed-nightmare",
   "metadata": {},
   "outputs": [],
   "source": [
    "most_retweeted_media = data.get_most_retweeted_media(40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "legendary-workstation",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Ammount found: \", len(most_retweeted_media))\n",
    "for rt_count, m_id, m in most_retweeted_media[15:21]:\n",
    "    print(m)\n",
    "    print(\"*\"*20 + \"\\n\" + str(rt_count) + \" - \" + str(m_id) + \"\\n\" + \"*\"*20 + \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adaptive-green",
   "metadata": {},
   "outputs": [],
   "source": [
    "most_retweeted_posts = data.get_most_retweeted(100,has_media=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bigger-seller",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save populars posts\n",
    "with open(\"100_most_retweeted_posts.pickle\",'wb') as handler:\n",
    "    pickle.dump(most_retweeted_posts, handler)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "patient-plain",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recall popular posts\n",
    "with open(\"100_most_retweeted_posts.pickle\",'rb') as handler:\n",
    "    most_retweeted_posts = pickle.load(handler)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "overhead-admission",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "print(\"Ammount found: \", len(most_retweeted_posts))\n",
    "for rt_count, tweet_id, key in random.sample(most_retweeted_posts[11:21], 10):\n",
    "    tweet = data.fetch_by_id(tweet_id)\n",
    "    if \"renuncia\" in tweet.data[\"text\"].lower() or \"puerto rico\" in tweet.data[\"text\"].lower() or \"ricky\" in tweet.data[\"text\"].lower() or \"rosell\" in tweet.data[\"text\"].lower():\n",
    "        print(tweet)\n",
    "        print(\"*\"*20 + \"\\n\" + str(rt_count) + \" - \" + str(tweet_id) + \" - \" + str(key) + \"\\n\" + \"*\"*20 + \"\\n\\n\")\n",
    "    else:\n",
    "        # print(tweet.data[\"text\"])\n",
    "        print(tweet)\n",
    "        print(\"*\"*10 + \"\\n\" + str(rt_count) + \" - \" + str(tweet_id) + \" - \" + str(key) + \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "stone-miniature",
   "metadata": {},
   "outputs": [],
   "source": [
    "# randint(0,SAMPLE_SIZE-6)\n",
    "# print(data.head(5, 40, sep=\"\\n\" + \"*\"*100 + \"\\n\\n\"))\n",
    "#RickyRenuncia\n",
    "#RickyVeteYa\n",
    "print(data.head(5, randint(0,SAMPLE_SIZE-6), sep=\"\\n\" + \"*\"*100 + \"\\n\\n\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "charged-basis",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data.head(2, sep=\"\\n*************\\n\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "amino-vanilla",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(data.retweet_cache))\n",
    "print(str(data.retweet_cache.keys())[:400])\n",
    "print(str(data.retweet_cache)[:400])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "superior-orange",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data.retweet_cache[0][0])\n",
    "print(str(data.quoteOf)[:400])\n",
    "print(str(data.retweetOf)[:400])\n",
    "print(str(data.retweet_cache)[:400])\n",
    "retweet_counts = list(data.retweet_cache.keys())\n",
    "retweet_counts.sort(reverse=True)\n",
    "quote_counts = list(data.quote_cache.keys())\n",
    "quote_counts.sort(reverse=True)\n",
    "print(str(retweet_counts)[:400])\n",
    "print(str(quote_counts)[:400])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "compressed-spank",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_t = data.fetch_by_position(112)\n",
    "print(json.dumps(sample_t.data, indent=4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cooperative-guarantee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find a video tweet\n",
    "SAMPLE_SIZE = 1113758\n",
    "count = 0\n",
    "media_ids=[]\n",
    "with open(JL_DATA,'r') as data_file:\n",
    "    for _ in range(SAMPLE_SIZE):\n",
    "        count+=1\n",
    "        if count%200000 == 0:\n",
    "            print(f\"Done with: {count}\")\n",
    "        tweet = TweetAnalyzer(data_file.readline())\n",
    "        if tweet.hasMedia:\n",
    "#             print(\"HasMedia\",tweet.hasMedia)\n",
    "            if len(tweet.media) > 0:\n",
    "                for m in tweet.media:\n",
    "                    if m.mtype().lower() != \"photo\" and m.id not in media_ids:\n",
    "                        media_ids.append(m.id)\n",
    "                        print(m.id, m.mtype(), m.url())\n",
    "#                         print(m.data)\n",
    "            else:\n",
    "                print(\"Length 0??\")\n",
    "                try:\n",
    "                    print(tweet.data[\"entities\"][\"media\"])\n",
    "                except:\n",
    "                    print(\"No Media at HERE\")\n",
    "                    try:\n",
    "                        print(tweet.data[\"retweeted_status\"][\"entities\"][\"media\"])\n",
    "                    except:\n",
    "                        print(\"No Media at RETWEET_STATUS\")\n",
    "                        print(json.dumps(tweet.data))\n",
    "                        break\n",
    "print(f\"DONE: {count}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lesser-canal",
   "metadata": {},
   "source": [
    "## Beautiful Imagery\n",
    "<div style=\"text-align:center; width:400px\"><h3>Title</h3><img src=\"https://pbs.twimg.com/media/EAAqz9aXYAIkkbi.jpg\" style=\"max-height:400px;\"><h4>Foot Notes</h4></div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "floral-disco",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f04f2736-4bfa-408b-8322-80a20f1a9629",
   "metadata": {},
   "outputs": [],
   "source": [
    "def h(p, q):\n",
    "    return (p, q)\n",
    "\n",
    "interact(h, p=10, q=fixed(20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10c86397-bfee-4441-bea6-d18610b4cc10",
   "metadata": {},
   "outputs": [],
   "source": [
    "interact(f, x=IntSlider(min=0, max=30, step=1, value=15))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbf3a2b9-0c45-4e2f-9748-d59391e7320c",
   "metadata": {},
   "outputs": [],
   "source": [
    "@interact(x=(0.0,20.0,0.5))\n",
    "def h(x=5.5):\n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7ae6c2d-7fd7-46e9-a6c7-666ef285388f",
   "metadata": {},
   "outputs": [],
   "source": [
    "@interact(x=(8,20))\n",
    "def aTitle(x=12):\n",
    "    display(HTML(f\"<h1 style='font-size:{x}px'>Hello!</h1>\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4142a3b-6d9d-45d7-8983-666fe85a5612",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipywidgets as widgets\n",
    "from IPython.display import display\n",
    "button = widgets.Button(description=\"Click Me!\")\n",
    "output = widgets.Output()\n",
    "\n",
    "display(button, output)\n",
    "output.my_n = 0\n",
    "def on_button_clicked(b):\n",
    "    with output:\n",
    "        output.clear_output()\n",
    "        output.my_n+=1\n",
    "        print(f\"Button clicked. {output.my_n}\")\n",
    "\n",
    "button.on_click(on_button_clicked)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a50d054d-45ec-4b1b-b24f-5560acd6b487",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding Required Libraries\n",
    "import ipywidgets as widgets\n",
    "from IPython.core.display import display, HTML, update_display\n",
    "import json, os, pickle\n",
    "from random import seed, randint\n",
    "from tweet_rehydrate.analysis import TweetJLAnalyzer, TweetAnalyzer, getsizeof\n",
    "from tweet_rehydrate.display import TweetInteractiveClassifier, JsonLInteractiveClassifier, TSess, prepare_google_credentials\n",
    "from twitter_secrets import C_API_KEY, C_API_SECRET, ACCESS_TOKEN, ACCESS_TOKEN_SECRET, C_BEARER_TOKEN \n",
    "JL_DATA=\"/home/rickyrenuncia/tweetsRickyRenuncia-final.jsonl\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8754bd32-7db8-4c0a-b896-53f6dd2df8ab",
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: '/path/to/key.json'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-afc3c9b63e0b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      6\u001b[0m         \u001b[0mhash_split\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m     )\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0mgoogle_credentials\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mprepare_google_credentials\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcredentials_file\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"./RickyRenuncia-case-module_shared/google_translate_keys.json\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/home/rickyrenuncia/RickyRenuncia-case-module_shared/tweet_rehydrate/display.py\u001b[0m in \u001b[0;36mprepare_google_credentials\u001b[0;34m(credentials_file)\u001b[0m\n\u001b[1;32m     34\u001b[0m         \u001b[0mcredentials_file\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menviron\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"GOOGLE_APPLICATION_CREDENTIALS\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     35\u001b[0m     \u001b[0;32massert\u001b[0m \u001b[0mcredentials_file\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 36\u001b[0;31m     \u001b[0mgoogle_credentials\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mservice_account\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mCredentials\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     37\u001b[0m         \u001b[0mfrom_service_account_file\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'/path/to/key.json'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     38\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mgoogle_credentials\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.virtualenvs/jupyterhub/lib/python3.8/site-packages/google/oauth2/service_account.py\u001b[0m in \u001b[0;36mfrom_service_account_file\u001b[0;34m(cls, filename, **kwargs)\u001b[0m\n\u001b[1;32m    236\u001b[0m                 \u001b[0mcredentials\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    237\u001b[0m         \"\"\"\n\u001b[0;32m--> 238\u001b[0;31m         info, signer = _service_account_info.from_filename(\n\u001b[0m\u001b[1;32m    239\u001b[0m             \u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequire\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"client_email\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"token_uri\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    240\u001b[0m         )\n",
      "\u001b[0;32m~/.virtualenvs/jupyterhub/lib/python3.8/site-packages/google/auth/_service_account_info.py\u001b[0m in \u001b[0;36mfrom_filename\u001b[0;34m(filename, require)\u001b[0m\n\u001b[1;32m     70\u001b[0m             \u001b[0minfo\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0ma\u001b[0m \u001b[0msigner\u001b[0m \u001b[0minstance\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     71\u001b[0m     \"\"\"\n\u001b[0;32m---> 72\u001b[0;31m     \u001b[0;32mwith\u001b[0m \u001b[0mio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"r\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"utf-8\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mjson_file\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     73\u001b[0m         \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjson_file\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     74\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfrom_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequire\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrequire\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: '/path/to/key.json'"
     ]
    }
   ],
   "source": [
    "tweet_session = TSess(\n",
    "        C_BEARER_TOKEN, \n",
    "        compression_level=5, \n",
    "        sleep_time=3, \n",
    "        cache_dir=\"./.tweet_cache_split/\", \n",
    "        hash_split=True\n",
    "    )\n",
    "google_credentials = prepare_google_credentials(credentials_file=\"./google_translate_keys.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8390093a-d2ea-4e68-916c-63dc742a1470",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ic| 'Need to request value'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading Tweet...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ic| 'Need to request value'\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<blockquote class=\"twitter-tweet\"><p lang=\"es\" dir=\"ltr\">Si ven # como Ric**SeQueda o de esa índole: NO LE DEN REPLY, NO LE DEN RETWEET, NO LE DEN QUOTE. Esto ayuda a impulsar su tag. Lxs que apoyan a Ricky saben que le responderíamos enojadxs a sus tweets, promocionando su popularidad. NO PERMITAN QUE ESTO PASE, PORQUE <a href=\"https://twitter.com/hashtag/RICKYRENUNCIA?src=hash&amp;ref_src=twsrc%5Etfw\">#RICKYRENUNCIA</a></p>&mdash; Lain (@agridvlce) <a href=\"https://twitter.com/agridvlce/status/1151335670868324352?ref_src=twsrc%5Etfw\">July 17, 2019</a></blockquote>\n",
       "<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "\n",
      "            What should we do? \n",
      "                1)Accept\n",
      "                2)Reject\n",
      "                3)Skip\n",
      "                4)Exit\n",
      "             4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ic| 'Skipped', kwargs.get(\"value\", \"\"): ''\n"
     ]
    }
   ],
   "source": [
    "# jl_display = JsonLInteractiveClassifier(\n",
    "#     tweet_ids_file=\"tweetsRickyRenuncia-final.txt\", \n",
    "#     session=tweet_session, mute=False)\n",
    "# Flier Boletin Promocion\n",
    "# 30 de Abril\n",
    "jl_display = JsonLInteractiveClassifier(\n",
    "    tweet_ids_file=\"tweetsRickyRenuncia-final.txt\", \n",
    "    session=tweet_session,pre_initialized=True, sqlite_db = \".tweetsRickyRenuncia-final.txt.db\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8cb5649e-779a-4854-ac17-5cdf089f4e6a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'jl_display' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-49c762080679>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mjl_display\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisplay_another\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'jl_display' is not defined"
     ]
    }
   ],
   "source": [
    "jl_display.display_another()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b4a00fd1-d224-4574-8dd0-799cb07305e8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ic| 'Value in Cache'\n",
      "ic| 'Value in Cache'\n"
     ]
    }
   ],
   "source": [
    "test_tweet = TweetInteractiveClassifier(tweet_id=\"1150943952616468486\", session=tweet_session)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e0fb2b60-7040-4553-81b4-94e5aadf449f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://twitter.com/any_user/status/1150943952616468486'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_tweet.url()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ffaf6d5e-ec8e-40b2-9749-9db5633b8aab",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ic| 'Value in Cache'\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<blockquote class=\"twitter-tweet\"><p lang=\"en\" dir=\"ltr\">More than 20,000 took the streets today -the fourth consecutive day- demanding that <a href=\"https://twitter.com/ricardorossello?ref_src=twsrc%5Etfw\">@ricardorossello</a> step down or that the House put begin with the impeachment proceedings. <a href=\"https://twitter.com/hashtag/RickyRenuncia?src=hash&amp;ref_src=twsrc%5Etfw\">#RickyRenuncia</a> <a href=\"https://t.co/wMSHJAgKIF\">pic.twitter.com/wMSHJAgKIF</a></p>&mdash; M Rodriguez Banchs (@mrbanchs) <a href=\"https://twitter.com/mrbanchs/status/1150943952616468486?ref_src=twsrc%5Etfw\">July 16, 2019</a></blockquote>\n",
       "<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# output = widgets.Output()\n",
    "html = test_tweet.oEmbeded()\n",
    "# print(html)\n",
    "# with output:\n",
    "display(HTML(html))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4cc34489-806f-4440-9a4d-eda31b392ea8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "More than 20,000 took the streets today -the fourth consecutive day- demanding that @ricardorossello step down or that the House put begin with the impeachment proceedings. #RickyRenuncia https://t.co/wMSHJAgKIF\n",
      "True\n",
      "True\n",
      "dict_keys(['created_at', 'id', 'id_str', 'full_text', 'truncated', 'display_text_range', 'entities', 'extended_entities', 'source', 'in_reply_to_status_id', 'in_reply_to_status_id_str', 'in_reply_to_user_id', 'in_reply_to_user_id_str', 'in_reply_to_screen_name', 'user', 'geo', 'coordinates', 'place', 'contributors', 'is_quote_status', 'retweet_count', 'favorite_count', 'favorited', 'retweeted', 'possibly_sensitive', 'lang'])\n",
      "{'hashtags': [{'text': 'RickyRenuncia', 'indices': [173, 187]}], 'symbols': [], 'user_mentions': [{'screen_name': 'ricardorossello', 'name': 'Ricardo Rosselló', 'id': 80013913, 'id_str': '80013913', 'indices': [84, 100]}], 'urls': [], 'media': [{'id': 1150943327073816581, 'id_str': '1150943327073816581', 'indices': [188, 211], 'media_url': 'http://pbs.twimg.com/ext_tw_video_thumb/1150943327073816581/pu/img/NEJsf-B609d2c93F.jpg', 'media_url_https': 'https://pbs.twimg.com/ext_tw_video_thumb/1150943327073816581/pu/img/NEJsf-B609d2c93F.jpg', 'url': 'https://t.co/wMSHJAgKIF', 'display_url': 'pic.twitter.com/wMSHJAgKIF', 'expanded_url': 'https://twitter.com/mrbanchs/status/1150943952616468486/video/1', 'type': 'photo', 'sizes': {'thumb': {'w': 150, 'h': 150, 'resize': 'crop'}, 'small': {'w': 383, 'h': 680, 'resize': 'fit'}, 'large': {'w': 720, 'h': 1280, 'resize': 'fit'}, 'medium': {'w': 675, 'h': 1200, 'resize': 'fit'}}}]}\n",
      "{'media': [{'id': 1150943327073816581, 'id_str': '1150943327073816581', 'indices': [188, 211], 'media_url': 'http://pbs.twimg.com/ext_tw_video_thumb/1150943327073816581/pu/img/NEJsf-B609d2c93F.jpg', 'media_url_https': 'https://pbs.twimg.com/ext_tw_video_thumb/1150943327073816581/pu/img/NEJsf-B609d2c93F.jpg', 'url': 'https://t.co/wMSHJAgKIF', 'display_url': 'pic.twitter.com/wMSHJAgKIF', 'expanded_url': 'https://twitter.com/mrbanchs/status/1150943952616468486/video/1', 'type': 'video', 'sizes': {'thumb': {'w': 150, 'h': 150, 'resize': 'crop'}, 'small': {'w': 383, 'h': 680, 'resize': 'fit'}, 'large': {'w': 720, 'h': 1280, 'resize': 'fit'}, 'medium': {'w': 675, 'h': 1200, 'resize': 'fit'}}, 'video_info': {'aspect_ratio': [9, 16], 'duration_millis': 16430, 'variants': [{'bitrate': 2176000, 'content_type': 'video/mp4', 'url': 'https://video.twimg.com/ext_tw_video/1150943327073816581/pu/vid/720x1280/bwitkY71BNkxtVNl.mp4?tag=10'}, {'content_type': 'application/x-mpegURL', 'url': 'https://video.twimg.com/ext_tw_video/1150943327073816581/pu/pl/ZqRRTPZyiioDzUOF.m3u8?tag=10'}, {'bitrate': 632000, 'content_type': 'video/mp4', 'url': 'https://video.twimg.com/ext_tw_video/1150943327073816581/pu/vid/320x568/mFbp7Wn5ejQw23LD.mp4?tag=10'}, {'bitrate': 832000, 'content_type': 'video/mp4', 'url': 'https://video.twimg.com/ext_tw_video/1150943327073816581/pu/vid/360x640/4BaXjVBo5F_Kl46e.mp4?tag=10'}]}, 'additional_media_info': {'monetizable': False}}]}\n"
     ]
    }
   ],
   "source": [
    "print(test_tweet.text())\n",
    "print(test_tweet.hasMedia)\n",
    "print(test_tweet.hasLocalMedia)\n",
    "print(test_tweet.data.keys())\n",
    "print(test_tweet.data.get(\"entities\", {}))\n",
    "print(test_tweet.data.get(\"extended_entities\", {}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28b45e86-58ea-4da3-a665-8d1e49973b70",
   "metadata": {},
   "outputs": [],
   "source": [
    "(test_tweet.url(),test_tweet.isRetweet, test_tweet.retweeted_status.url())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bedc4f1-0920-46a4-bc15-5b8b3c7fb65a",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_tweet.display()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "332b4c27-34b7-4926-81f4-6480a14545e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['created_at', 'id', 'id_str', 'full_text', 'truncated', 'display_text_range', 'entities', 'extended_entities', 'source', 'in_reply_to_status_id', 'in_reply_to_status_id_str', 'in_reply_to_user_id', 'in_reply_to_user_id_str', 'in_reply_to_screen_name', 'user', 'geo', 'coordinates', 'place', 'contributors', 'is_quote_status', 'retweet_count', 'favorite_count', 'favorited', 'retweeted', 'possibly_sensitive', 'lang'])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_tweet.data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef3b1db2-43cb-4380-b98a-97e82919519c",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_tweet.data[]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}