", .x, "
", as.character(year_table), @@ -220,9 +157,35 @@ make_tt_dataset_list_html <- function(x, file = tempfile(fileext = ".html")) { "Archive of datasets and articles from the 2018 series of #TidyTuesday events.
Data comes from The Economist GitHub. The following information was taken directly from their GitHub readme.
+ +These are the data behind the "space launches" article, The space race is dominated by new contenders.
+Principal data came from the Jonathan McDowell's JSR Launch Vehicle Database, available online at http://www.planet4589.org/space/lvdb/index.html.
+ +| File | +Contents | +Source | +
|---|---|---|
| agencies | +Space launch providers | +Jonathan McDowell; The Economist + | +
| launches | +Individual space launches | +Jonathan McDowell; The Economist + | +
| variable | +definition | +
|---|---|
| tag | +Harvard or COSPAR id of launch | +
| JD | ++Julian Date of launch | +
| launch_date | +date of launch | +
| launch_year | +year of launch | +
| type | +type of launch vehicle | +
| variant | +variant of launch vehicle | +
| mission | ++ |
| agency | +launching agency | +
| state_code | +launching agency's state | +
| category | +success (O) or failure (F) | +
| agency_type | +type of agency | +
| variable | +definition | +
|---|---|
| agency | +org phase code | +
| count | +number of launches | +
| ucode | +org Ucode | +
| state_code | +responsible state | +
| type | +type of org | +
| class | +class of org | +
| tstart | +org/phase founding date | +
| tstop | +org/phase ending date | +
| short_name | +short name | +
| name | +full name | +
| location | +plain english location | +
| longitude | ++ |
| latitude | ++ |
| error | +uncertainty in long/lat | +
| parent | +parent org | +
| short_english_name | +english short name | +
| english_name | +english full name | +
| unicode_name | +unicode full name | +
| agency_type | +type of agency | +
"Seattle Department of Transportation has 12 bike counters (four of which also count pedestrians) located on neighborhood greenways, multi-use trails, at the Fremont Bridge and on SW Spokane Street. The counters are helping us create a ridership baseline in 2014 that can be used to assess future years and make sure our investments are helping us to reach our goal of quadrupling ridership by 2030. Read our Bicycle Master Plan to learn more about what Seattle is doing to create a citywide bicycle network."
+The Seattle Times recently covered What we can learn from Seattle's bike-counter data. They have some elegant data-visualizations there!
+ +bike_traffic <- readr::read_csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2019/2019-04-02/bike_traffic.csv")
+| variable | +class | +description | +
|---|---|---|
| date | +date (mdy hms am/pm) | +Date of data upload | +
| crossing | +character | +The Street crossing/intersection | +
| direction | +character | +North/South/East/West - varies by crossing | +
| bike_count | +double | +Number of bikes counted for each hour window | +
| ped_count | +double | +Number of pedestrians counted for each hour window | +
A few exciting Tennis-related datasets this week, all of which come courtesy of Wikipedia! "All records are based on data from the Association of Tennis Professionals (ATP), the International Tennis Federation (ITF), and the official websites of the four Grand Slam tournaments." - Wikipedia
+"The Open Era is the current era of professional tennis. It began in 1968 when the Grand Slam tournaments allowed professional players to compete with amateurs, ending the division that had persisted since the dawn of the sport in the 19th century." - Wikipedia
+"The Grand Slam tournaments, also called majors, are the four most important annual tennis events. They offer the most ranking points, prize money, public and media attention, the greatest strength and size of field, and greater number of "best of" sets for men. The Grand Slam itinerary consists of the Australian Open in mid January, the French Open around late May through early June, Wimbledon in June-July, and the US Open in August-September. Each tournament is played over a two-week period. The Australian and United States tournaments are played on hard courts, the French on clay, and Wimbledon on grass." - Wikipedia
+The court surface could be an interesting additional piece of data that I have left out, you could add it in with some clever case_when() calls.
I have tamed the datasets pretty thoroughly but there are several ways you could combine, summarize, or otherwise plot the data for this week. I have a spoiler/hint at the bottom if you get stuck with combining! I also left my relatively rough cleaning and data collection .rmd in, and have uploaded it.
+ +The Grand Slam tournaments happen in a rough 2-week timeframe as mentioned above, however I was unable to find a nice dataset that covered the specific day of both the men's and women's finals. As such I have used a static date that estimates the date of the championship match, and likely has an error of a few days for each tournament. However, this is still useful for determining the approximate age of the player at each tournament.
+ +Women's Timeline
Men's Timeline
Date of Birth/First Title
Women's Singles Champs
Men's Singles Champs
The Financial Times Article has lots of great inspiration plots, but uses different datasets. Specifically the author used match wins vs tournament wins and tennis rankings over time rather than tournament placing. The author John Burn-Murdoch is a great follow for DataViz and visual storytelling resources, including R and D3.
+Additionally a gist and Tweet from John Burn-Murdoch goes through the process of collecting some men's tennis data, and then how he iterated across several plots. It's worth taking a look at for either code-inspiration for this week or as a general example of plot iteration for publication.
+ +player_dob <- readr::read_csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2019/2019-04-09/player_dob.csv")
+
+grand_slams <- readr::read_csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2019/2019-04-09/grand_slams.csv")
+
+grand_slam_timeline <- readr::read_csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2019/2019-04-09/grand_slam_timeline.csv")
+-
+
grand_slam_timeline
+
| variable | +class | +description | +
|---|---|---|
| player | +character | +Player Name | +
| year | +integer | +Tournament Year | +
| tournament | +character | +Tournament name | +
| outcome | +character | +Outcome - where player was eliminated, was absent, or won - there are some NAs | +
| gender | +character | +Male/Female for this dataset | +
-
+
grand_slams
+
| variable | +class | +description | +
|---|---|---|
| year | +integer | +Tournament Year | +
| grand_slam | +character | +Tournament name | +
| name | +character | +Player Name | +
| rolling_win_count | +integer | +Rolling win = cumulative sum of wins across time/player | +
| tournament_date | +date | ++Approximate Tournament Date (ymd) | +
| gender | +character | +Male/Female for this dataset | +
-
+
player_dob
+
| variable | +class | +description | +
|---|---|---|
| name | +character | +Player name | +
| grand_slam | +character | +Tournament for first major title win | +
| date_of_birth | +date | +date of birth (ymd) | +
| date_of_first_title | +date | +date of first major title win (ymd) | +
| age | +double | +Age in days | +
To get the tournament performance at age rather than simply across time we need to join the Date of Birth dataset with the grandslam dataset.
age_slams_comb <- left_join(grand_slams, player_dob, by = c("name")) %>%
+ mutate(age = tournament_date - date_of_birth) %>% # needs to be datetime
+ group_by(name, age, gender) %>%
+ summarize(counts = n()) %>%
+ group_by(name) %>%
+ mutate(total_wins = cumsum(counts)) %>%
+ arrange(desc(total_wins))
+
+# test plot
+age_slams_comb %>%
+ ggplot(aes(x = age, y = total_wins, group = name)) +
+ geom_point() +
+ geom_step() +
+ facet_wrap(~gender)
+
A weekly data project aimed at the R ecosystem. As this project was borne out of the R4DS Online Learning Community and the R for Data Science textbook, an emphasis was placed on understanding how to summarize and arrange data to make meaningful charts with ggplot2, tidyr, dplyr, and other tools in the tidyverse ecosystem. However, any code-based methodology is welcome - just please remember to share the code used to generate the results.
+
Join the R4DS Online Learning Community in the weekly #TidyTuesday event! Every week we post a raw dataset, a chart or article related to that dataset, and ask you to explore the data. While the dataset will be "tamed", it will not always be tidy! As such you might need to apply various R for Data Science techniques to wrangle the data into a true tidy format. The goal of TidyTuesday is to apply your R skills, get feedback, explore other's work, and connect with the greater #RStats community! As such we encourage everyone of all skills to participate!
We will have many sources of data and want to emphasize that no causation is implied. There are various moderating variables that affect all data, many of which might not have been captured in these datasets. As such, our guidelines are to use the data provided to practice your data tidying and plotting techniques. Participants are invited to consider for themselves what nuancing factors might underlie these relationships.
+The intent of Tidy Tuesday is to provide a safe and supportive forum for individuals to practice their wrangling and data visualization skills independent of drawing conclusions. While we understand that the two are related, the focus of this practice is purely on building skills with real-world data.
+All data will be posted on the data sets page on Monday. It will include the link to the original article (for context) and to the data set.
+We welcome all newcomers, enthusiasts, and experts to participate, but be mindful of a few things:
+-
+
- The data set comes from the source article or the source that the article credits. Be mindful that the data is what it is and Tidy Tuesday is designed to help you practice data visualization and basic data wrangling in R. +
- Again, the data is what it is! You are welcome to explore beyond the provided dataset, but the data is provided as a "toy" dataset to practice techniques on. +
- This is NOT about criticizing the original article or graph. Real people made the graphs, collected or acquired the data! Focus on the provided dataset, learning, and improving your techniques in R. +
- This is NOT about criticizing or tearing down your fellow
#RStatspractitioners or their code! Be supportive and kind to each other! Like other's posts and help promote the#RStatscommunity!
+ - Use the hashtag #TidyTuesday on Twitter if you create your own version and would like to share it. +
- Include a picture of the visualisation when you post to Twitter. +
- Include a copy of the code used to create your visualization when you post to Twitter. Comment your code wherever possible to help yourself and others understand your process! +
- Focus on improving your craft, even if you end up with something simple! +
- Give credit to the original data source whenever possible. +
+ +
Want to submit an interesting dataset? Please open an Issue and post a link to the article (or blogpost, etc) using the data, then we can discuss adding it to a future TidyTuesday Event!
++ +
+
+2018
+ +
+
+2019
+ ++ +
| Link | +Description | +
|---|---|
| Link | +The R4DS Online Learning Community Website | +
| Link | +The R for Data Science textbook | +
| Link | +Carbon for sharing beautiful code pics | +
| Link | +Post gist to Carbon from RStudio | +
| Link | +Post to Carbon from RStudio | +
| Link | +Join GitHub! | +
| Link | +Basics of GitHub | +
| Link | +Learn how to use GitHub with R | +
| Link | +Save high-rez ggplot2 images |
+
| Link | +Description | +
|---|---|
| Link | +Data is Plural collection | +
| Link | +BuzzFeedNews GitHub | +
| Link | +The Economist GitHub | +
| Link | +The fivethirtyeight data package |
+
| Link | +The Upshot by NY Times | +
| Link | +The Baltimore Sun Data Desk | +
| Link | +The LA Times Data Desk | +
| Link | +Open News Labs | +
| Link | +BBC Data Journalism team | +
+ +
Only books available freely online are sourced here. Feel free to add to the list
+| Link | +Description | +
|---|---|
| Link | +Fundamentals of Data Viz by Claus Wilke | +
| Link | +The Art of Data Science by Roger D. Peng & Elizabeth Matsui | +
| Link | +Tidy Text Mining by Julia Silge & David Robinson | +
| Link | +Geocomputation with R by Robin Lovelace, Jakub Nowosad, Jannes Muenchow | +
| Link | +Data Visualization by Kieran Healy | +
| Link | +
+ggplot2 cookbook by Winston Chang |
+
| Link | +BBC Data Journalism team | +
h/t to Bob Rudis for sharing the data source, and to Roel Hogervorst for the guide to scraping this data. He provided the bulk of the scraping code, and I added bit of additional data cleaning. The data this week comes from Privacy Affairs.
+I have also included all the raw text (gdpr_text.tsv) for the actual GDPR legal documents, in case someone was interested in parsing through them or using them along with the violations.
Per Wikipedia GDPR is:
+++ +The General Data Protection Regulation (EU) 2016/679 (GDPR) is a regulation in EU law on data protection and privacy in the European Union (EU) and the European Economic Area (EEA). It also addresses the transfer of personal data outside the EU and EEA areas. The GDPR aims primarily to give control to individuals over their personal data and to simplify the regulatory environment for international business by unifying the regulation within the EU.[1] Superseding the Data Protection Directive 95/46/EC, the regulation contains provisions and requirements related to the processing of personal data of individuals (formally called data subjects in the GDPR) who reside in the EEA, and applies to any enterprise—regardless of its location and the data subjects' citizenship or residence—that is processing the personal information of data subjects inside the EEA.
+
# Get the Data
+
+gdpr_violations <- readr::read_tsv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-04-21/gdpr_violations.tsv')
+gdpr_text <- readr::read_tsv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-04-21/gdpr_text.tsv')
+
+# Or read in with tidytuesdayR package (https://github.com/thebioengineer/tidytuesdayR)
+# PLEASE NOTE TO USE 2020 DATA YOU NEED TO USE the tidytuesdayR version after Jan 2020.
+
+# Either ISO-8601 date or year/week works!
+
+# Install via devtools::install_github("thebioengineer/tidytuesdayR")
+
+tuesdata <- tidytuesdayR::tt_load('2020-04-21')
+tuesdata <- tidytuesdayR::tt_load(2020, week = 17)
+
+
+gdpr_violations <- tuesdata$gdpr_violations
+| variable | +class | +description | +
|---|---|---|
| id | +integer | +Idetifier for fine/violation | +
| picture | +character | +SVG image of violation country flag | +
| name | +character | +Name of country where violation was enforced | +
| price | +integer | +Fine price in Euros (€) | +
| authority | +character | +Authority that enacted the violation | +
| date | +character | +Date of violation | +
| controller | +character | +Controller of data - the violator | +
| article_violated | +character | +Specific GDPR Article violated (see the gdpr_text.tsv data for specifics) |
+
| type | +character | +Type of violation | +
| source | +character | +Original source (URL) of fine data | +
| summary | +character | +Summary of violation | +
| variable | +class | +description | +
|---|---|---|
| chapter | +double | +GDPR Chapter Number | +
| chapter_title | +character | +Chapter title | +
| article | +double | +GDPR Article number | +
| article_title | +character | +Article title | +
| sub_article | +double | +Sub article number | +
| gdpr_text | +character | +Raw text of article/subarticle | +
| href | +character | +URL to the raw text itself | +
library(tidyverse)
+library(rvest)
+
+# Note the following code was adapted from
+# https://blog.rmhogervorst.nl/blog/2020/04/08/scraping-gdpr-fines/
+
+link <- "https://www.privacyaffairs.com/gdpr-fines/"
+page <- read_html(link)
+
+
+temp <- page %>% html_nodes("script") %>%
+ .[9] %>%
+ rvest::html_text()
+
+ends <- str_locate_all(temp, "\\]")
+starts <- str_locate_all(temp, "\\[")
+
+table1 <- temp %>%
+ stringr::str_sub(start = starts[[1]][1,2], end = ends[[1]][1,1]) %>%
+ str_remove_all("\\\n") %>%
+ str_remove_all("\\\r") %>%
+ jsonlite::fromJSON() %>%
+ as_tibble() %>%
+ mutate(summary = str_remove_all(summary,"<p>|</p>|\n"))
+
+
+table2 <- temp %>%
+ stringr::str_sub(start = starts[[1]][2,2], end = ends[[1]][2,1]) %>%
+ str_remove_all("\\\n") %>%
+ str_remove_all("\\\r") %>%
+ jsonlite::fromJSON() %>%
+ as_tibble() %>%
+ mutate(summary = str_remove_all(summary,"<p>|</p>|\n"))
+
+
+all_df <- bind_rows(table1, table2) %>%
+ janitor::clean_names() %>%
+ mutate(
+ authority = str_remove(authority, "\t"),
+ article_violated = str_remove(article_violated, '<a href="https://www.privacy-regulation.eu/en/32.htm">') %>%
+ str_remove('</a>'),
+ article_violated = str_replace_all(article_violated, ", Art", "|Art"),
+ type = str_remove(type, '<a href="https://www.privacy-regulation.eu/en/32.htm">') %>%
+ str_remove('</a>')
+ )
+
+# most frequent articles violated
+all_df %>%
+ separate_rows(article_violated, sep = "\\|") %>%
+ count(article_violated, sort = T)
+
+all_df %>%
+ write_tsv("2020/2020-04-21/gdpr_violations.tsv")
+
+
+# Getting the actual article text -----------------------------------------
+
+raw_article <- "https://gdpr-info.eu/" %>%
+ read_html()
+
+# Get all the urls for specific articles/chapters
+gdpr_href <- raw_article %>%
+ html_node(xpath = '//*[@id="tablepress-12"]') %>%
+ html_nodes("a") %>%
+ html_attr("href")
+
+# pull the titles as well
+gdpr_titles <- raw_article %>%
+ html_node(xpath = '//*[@id="tablepress-12"]') %>%
+ html_nodes("a") %>%
+ html_attr("data-title")
+
+# pull the numbers of article/chapters
+gdpr_numbers <- raw_article %>%
+ html_node(xpath = '//*[@id="tablepress-12"]') %>%
+ html_nodes("a") %>%
+ html_text()
+
+# put it all into a df
+gdpr_df <- tibble(
+ article = gdpr_numbers,
+ title = str_trim(gdpr_titles),
+ href = gdpr_href
+)
+
+# Tidy up the data, create chapters vs articles
+clean_gdpr <- gdpr_df %>%
+ mutate(chapter = if_else(str_length(article) > 3, article, NA_character_),
+ chapter_title = if_else(str_length(article) > 3, title, NA_character_)) %>%
+ fill(chapter, chapter_title) %>%
+ filter(!str_detect(article, "Chapter")) %>%
+ mutate(article = as.double(article)) %>%
+ filter(!is.na(article)) %>%
+ select(starts_with("chapter"), article, article_title = title, href)
+
+clean_gdpr
+
+# LONG running outcome
+# Get all the raw html from each of the urls for each article
+all_articles <- clean_gdpr %>%
+ mutate(raw_html = map(href, read_html))
+
+# function to take raw html and turn it into text for that specific article
+get_gdpr_text <- function(html_in){
+
+ test_var <- html_in %>%
+ html_node(".entry-content") %>%
+ html_nodes("ol") %>%
+ html_text()
+
+ if (length(test_var) == 0){
+ text <- html_in %>%
+ html_node(".entry-content > p") %>%
+ html_text() %>%
+ str_remove("^[:digit:]")
+ } else {
+ text <- html_in %>%
+ html_node(".entry-content") %>%
+ html_nodes("ol") %>%
+ html_text() %>%
+ .[[1]] %>%
+ str_replace_all(";\n", "\t") %>%
+ str_replace_all(":\n", "\t") %>%
+ str_split("\n") %>%
+ .[[1]] %>%
+ .[. != ""] %>%
+ str_replace_all("\t", "\n") %>%
+ str_remove("^[:digit:]")
+ }
+
+
+ text
+
+}
+
+# Test
+get_gdpr_text(read_html("http://gdpr-info.eu/art-2-gdpr/"))
+
+# unnest the list column of text
+clean_articles <- all_articles %>%
+ mutate(gdpr_text = map(raw_html, get_gdpr_text)) %>%
+ unnest_longer(gdpr_text)
+
+# final dataframe
+final_articles <- clean_articles %>%
+ group_by(article) %>%
+ mutate(sub_article = row_number()) %>%
+ relocate(sub_article, .after = "article_title") %>%
+ relocate(gdpr_text, .after = "sub_article") %>%
+ ungroup() %>%
+ mutate(chapter = str_extract(chapter, "[:digit:]+")) %>%
+ mutate_at(vars(chapter, article, sub_article), as.double) %>%
+ select(-raw_html)
+
+final_articles %>% view()
+
+write_tsv(final_articles, "2020/2020-04-21/gdpr_text.tsv")
+
+Archive of datasets and articles from the 2020 series of #TidyTuesday events.
Archive of datasets and articles from the 2021 series of #TidyTuesday events.
Please add alt text (alternative text) to all of your posted graphics for #TidyTuesday.
Twitter provides guidelines for how to add alt text to your images.
+The DataViz Society/Nightingale by way of Amy Cesal has an article on writing good alt text for plots/graphs.
+++Here's a simple formula for writing alt text for data visualization:
+ +It's helpful for people with partial sight to know what chart type it is and gives context for understanding the rest of the visual. Example: Line graph
+ +What data is included in the chart? The x and y axis labels may help you figure this out. Example: number of bananas sold per day in the last year
+ +Think about why you're including this visual. What does it show that's meaningful. There should be a point to every visual and you should tell people what to look for. Example: the winter months have more banana sales
+ +Don't include this in your alt text, but it should be included somewhere in the surrounding text. People should be able to click on a link to view the source data or dig further into the visual. This provides transparency about your source and lets people explore the data. Example: Data from the USDA
+
Penn State has an article on writing alt text descriptions for charts and tables.
+++Charts, graphs and maps use visuals to convey complex images to users. But since they are images, these media provide serious accessibility issues to colorblind users and users of screen readers. See the examples on this page for details on how to make charts more accessible.
+
The {rtweet} package includes the ability to post tweets with alt text programatically.
Need a reminder? There are extensions that force you to remember to add Alt Text to Tweets with media.
+ +The data this week comes from Post45 Data by way of Sara Stoudt.
+See their Data Description for full details.
+++ +Each peer-reviewed dataset has an accompanying curatorial statement, which provides an overview of the data that explains its contents, construction, and some possible uses.
+Please cite data from the Post45 Data Collective and use the following six components in your citation:
++
+- +
+author name(s)
+- +
+date published in the Post45 repository
+- +
+title
+- +
+global persistent > identifier: [DOI](https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F> %2Fwww.doi.org%2F&data=04%7C01%7Ckayla.shipp.kamibayashi%40emory.edu%7C31069808f> 9074a2a59ad08d8ce96abd0%7Ce004fb9cb0a4424fbcd0322606d5df38%7C0%7C0%7C63748649381> 8763862%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h> aWwiLCJXVCI6Mn0%3D%7C1000&sdata=3EOVEbBZrdRlmq3n%2BFQkyf8cEK7jdW99oo8LFoSqXEo%3D> &reserved=0 "Original URL: http://www.doi.org/. Click or tap if you trust this > link.")
+- +
+Post45 Data Collective
+- +
+version number
+Example replication data citation from The Program Era Project, Kelly, White, and Glass, 2021:
+Kelly, Nicholas; White, Nicole, Glass, Loren, 03/01/2021, "The Program Era > Project," DOI: https://doi.org/10.18737/CNJV1733p4520210415, Post45 Data > Collective, V1.
+
# Get the Data
+
+# Read in with tidytuesdayR package
+# Install from CRAN via: install.packages("tidytuesdayR")
+# This loads the readme and all the datasets for the week of interest
+
+# Either ISO-8601 date or year/week works!
+
+tuesdata <- tidytuesdayR::tt_load('2022-05-10')
+tuesdata <- tidytuesdayR::tt_load(2022, week = 19)
+
+nyt_titles <- tuesdata$nyt_titles
+
+# Or read in the data manually
+
+nyt_titles <- readr::read_tsv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2022/2022-05-10/nyt_titles.tsv')
+nyt_full <- readr::read_tsv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2022/2022-05-10/nyt_full.tsv')
+
+This is a subset/summary of the larger dataset.
+| variable | +class | +description | +
|---|---|---|
| id | +double | +Book id | +
| title | +character | +Title of book | +
| author | +character | +Author | +
| year | +double | +Year | +
| total_weeks | +double | +Total weeks on best sellers list | +
| first_week | +double | +First week on list (date) | +
| debut_rank | +double | +Debut rank on the list | +
| best_rank | +double | +Best rank | +
| variable | +class | +description | +
|---|---|---|
| year | +double | +Year | +
| week | +double | +Week (as date) | +
| rank | +double | +Rank (1 > 18) | +
| title_id | +double | +Title ID | +
| title | +character | +Title of book | +
| author | +character | +Author | +
Archive of datasets and articles from the 2022 series of #TidyTuesday events.
Archive of datasets and articles from the 2023 series of #TidyTuesday events.
Archive of datasets and articles from the 2024 series of #TidyTuesday events.