Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

updated scraper to use new api #1150

Merged
merged 10 commits into from
Sep 5, 2024
Merged

updated scraper to use new api #1150

merged 10 commits into from
Sep 5, 2024

Conversation

martanman
Copy link
Contributor

Scrapes postgrad items as well but currently still only saves undergrad courses.

@martanman martanman requested a review from a team as a code owner June 14, 2024 07:19
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 14, 2024 •
edited
Loading

Test Results

156 tests   - 5   79 ✅  - 18   52s ⏱️ +7s
 13 suites ±0    2 💤 ± 0 
  2 files   ±0   75 ❌ +13 

For more details on these failures, see this check.

Results for commit 49a3ec4. ± Comparison against base commit a599307.

This pull request removes 5 tests. 
server.tests.planner.test_validate_term_planner ‑ test_validateTermPlanner_past_term_suppress_warnings
server.tests.user.test_toggle_warnings ‑ test_toggleWarning
server.tests.user.test_toggle_warnings ‑ test_toggleWarningMultiple
src/components/ResetModal/ResetModal.test.tsx ‑ ResetModal > should reset state when reset button is clicked
src/components/ResetModal/ResetModal.test.tsx ‑ ResetModal > should show modal when degree wizard is complete

♻️ This comment has been updated with latest results.

ollibowers
ollibowers requested changes Jun 19, 2024
View reviewed changes
backend/data/scrapers/programs_formatting.py Outdated Show resolved Hide resolved
backend/data/scrapers/payload.py Show resolved Hide resolved
backend/data/scrapers/payload.py Outdated Show resolved Hide resolved
backend/data/scrapers/payload.py Outdated Show resolved Hide resolved
backend/data/scrapers/payload.py Outdated Show resolved Hide resolved
backend/data/processors/specialisations_processing.py Outdated Show resolved Hide resolved
@martanman martanman requested a review from ollibowers July 20, 2024 02:53
ollibowers
ollibowers requested changes Aug 2, 2024
View reviewed changes
Copy link
Collaborator

@ollibowers ollibowers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be banger to only keep undergrad programs and specialisations as well (since not super sure if this breaks things)

also run condition reparsing for pkl files

@martanman martanman requested a review from ollibowers September 1, 2024 07:21
ollibowers
ollibowers approved these changes Sep 5, 2024
View reviewed changes
Copy link
Collaborator

@ollibowers ollibowers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

banger bange rbange rbagna

@ollibowers ollibowers merged commit 851997b into 331-staging Sep 5, 2024
8 checks passed
@ollibowers ollibowers deleted the update-scraper branch September 5, 2024 07:26
justjo3l pushed a commit that referenced this pull request Sep 20, 2024
@martanman @justjo3l
updated scraper to use new api (#1150)
0efed3a 
* updated scraper to use new api and scrape postgrad items

* filter in only undergrad courses

* remove debug prints

* fix pylint, remove old (unused) create_payload function

* fix pylint again

* fix pylint fr this time

* rerun format and process courses

* scrape only ugrd prgms and spcs

* re pickle conditions and check certain courses exist before changing condition

* fix be tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ollibowers ollibowers ollibowers approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@martanman @ollibowers