Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Patch HI to use Google cache for #600 #602

Merged
merged 2 commits into from
Jan 17, 2024
Merged

Patch HI to use Google cache for #600 #602

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e529756
Patch HI to use Google cache for #600
stucka Jan 17, 2024
38f7a1c
Update hi.py
stucka Jan 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions warn/scrapers/hi.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
import datetime
import logging
from pathlib import Path
from urllib.parse import quote

from bs4 import BeautifulSoup

from .. import utils

__authors__ = ["Ash1R", "stucka"]
__tags__ = ["html"]
__tags__ = ["html", "pdf"]
__source__ = {
"name": "Workforce Development Hawaii",
"url": "https://labor.hawaii.gov/wdc/real-time-warn-updates/",
Expand All @@ -28,15 +29,17 @@ def scrape(
cache_dir -- the Path where results can be cached (default WARN_CACHE_DIR)
Returns: the Path where the file is written
"""
firstpage = utils.get_url("https://labor.hawaii.gov/wdc/real-time-warn-updates/")
cacheprefix = "https://webcache.googleusercontent.com/search?q=cache%3A" # Use Google Cache, per #600

firstpage = utils.get_url(cacheprefix + quote("https://labor.hawaii.gov/wdc/real-time-warn-updates/"))
soup = BeautifulSoup(firstpage.text, features="html5lib")
pagesection = soup.select("div.primary-content")[0]
subpageurls = []
for atag in pagesection.find_all("a"):
href = atag["href"]
if href.endswith("/"):
href = href[:-1]
subpageurls.append(href)
href = href # [:-1]
subpageurls.append(cacheprefix + quote(href))

headers = ["Company", "Date", "PDF url", "location", "jobs"]
data = [headers]
Expand Down Expand Up @@ -85,8 +88,8 @@ def scrape(
row.append(dates[i])

row.append(url)
row.append(None) # location
row.append(None) # jobs
row.append(None) # location
row.append(None) # jobs
data.append(row)

output_csv = data_dir / "hi.csv"
Expand Down
Loading