From 9981d79cd44ad9123b81a270b5af7afe1d3b977b Mon Sep 17 00:00:00 2001 From: Jenna Landy Date: Sun, 10 Dec 2023 00:11:42 -0500 Subject: [PATCH] compiling book --- docs/R/data-table.html | 49 ++++++---- docs/dataviz/dataviz-in-practice.html | 87 +++++++++--------- .../figure-html/bland-altman-1.png | Bin 83516 -> 83830 bytes .../figure-html/correct-transformation-1.png | Bin 91714 -> 91793 bytes .../installing-r-and-rstudio.html | 26 +++--- docs/sitemap.xml | 56 +++++------ docs/wrangling/data-table-wrangling.html | 36 ++++---- docs/wrangling/text-analysis.html | 49 +++++----- docs/wrangling/web-scraping.html | 2 +- 9 files changed, 161 insertions(+), 144 deletions(-) diff --git a/docs/R/data-table.html b/docs/R/data-table.html index 65bdda5..6b2ee04 100644 --- a/docs/R/data-table.html +++ b/docs/R/data-table.html @@ -2,7 +2,7 @@ - + Introduction to Data Science - 5  data.table @@ -88,7 +88,8 @@ "search-more-matches-text": "more matches in this document", "search-clear-button-title": "Clear", "search-detached-cancel-button-title": "Cancel", - "search-submit-button-title": "Submit" + "search-submit-button-title": "Submit", + "search-label": "Search" } } @@ -103,7 +104,7 @@ - @@ -113,7 +114,7 @@
Introduction to Data Science
- +
@@ -294,7 +295,7 @@
  • - 19  Installing Git + 19  Terminal Access and Installing Git
  • @@ -374,7 +375,7 @@

    library(dslabs) library(data.table)

    • -

    We will provide example code showing the data.table approaches to dplyr’s mutate, filter, select, group_by, and summarize shown in Chapter Chapter 4. As in that chapter, we will use the murders dataset:

    +

    We will provide example code showing the data.table approaches to dplyr’s mutate, filter, select, group_by, and summarize shown in Chapter 4. As in that chapter, we will use the murders dataset:

    The first step when using data.table is to convert the data frame into a data.table object using the as.data.table function:

    murders_dt <- as.data.table(murders)
    @@ -400,7 +401,7 @@ 

    murders <- mutate(murders, rate = total / population * 100000)

    -

    data.table uses an approach that avoids a new assignment (update by reference). This can help with large datasets that take up most of your computer’s memory. The data.table :=` function permits us to do this:

    +

    data.table uses an approach that avoids a new assignment (update by reference). This can help with large datasets that take up most of your computer’s memory. The data.table := function permits us to do this:

    murders_dt[, rate := total / population * 100000]
    @@ -416,7 +417,7 @@ 

    x <- data.table(a = 1)
 y <- x

    -

    y is actually referencing x, it is not an new opject: y just another name for x. Until you change y, a new object will not be made. However, the := function changes by reference so if you change x, a new object is not made and y continues to be just another name for x:

    +

    y is actually referencing x, it is not an new object: y just another name for x. Until you change y, a new object will not be made. However, the := function changes by reference so if you change x, a new object is not made and y continues to be just another name for x:

    x[,a := 2]
 y
@@ -462,11 +463,11 @@ 
    
 
    filter(murders, rate <= 0.7)
    -

    With data.table, we again use an approach similar to subsetting matrices, except data.table knows that rate refers to a column name and not an object in the R environment:

    +

    With data.table, we again use an approach similar to subsetting matrices, except like dplyr, data.table knows that rate refers to a column name and not an object in the R environment:

    murders_dt[rate <= 0.7]
    -

    Notice that we can combine the filter and select into one succint command. Here are the state names and rates for those with rates below 0.7.

    +

    Notice that we can combine the filter and select into one succinct command. Here are the state names and rates for those with rates below 0.7.

    murders_dt[rate <= 0.7, .(state, rate)]
 #>            state  rate
@@ -496,7 +497,7 @@ 
    
 
    heights_dt <- as.data.table(heights)
    -

    In data.table, we can call functions inside .() and they will be applied to rows. So the equivalent of:

    +

    In data.table, we can call functions inside .() and they will be applied to columns So the equivalent of:

    s <- heights |> summarize(avg = mean(height), sd = sd(height))
    @@ -516,7 +517,7 @@

    5.2.1 Multiple summaries

    -

    In Chapter 4, we defined the follwing function to permit multiple column summaries in dplyer:

    +

    In Chapter 4, we defined the following function to permit multiple column summaries in dplyr:

    median_min_max <- function(x){
   qs <- quantile(x, c(0.5, 0, 1))
@@ -542,16 +543,26 @@ 
    
 
    murders_dt[order(population)]
    -

    N To sort the table in descending order, we can order by the negative of population or use the decreasing argument:

    +

    To sort the table in descending order, we can order by the negative of population or use the decreasing argument:

    murders_dt[order(population, decreasing = TRUE)]

    5.3.1 Nested sorting

    -

    Similarly, we can perform nested ordering by including more than one variable in order

    +

    Similarly, we can perform nested ordering by including more than one variable in order:

    murders_dt[order(region, rate)]
    +
    +
    +
    + +
    +
    +

    You are ready to do exercises 8-12.

    +
    +
    +

    5.4 Exercises

    1. Load the data.table package and the murders dataset and convert it to data.table object:

    @@ -576,7 +587,7 @@ 

    murders_dt[state == "New York"]

    You can use other logical vectors to filter rows.

    -

    Show the top 5 states with the highest murder rates. After we add murder rate and rank, do not change the murders dataset, just show the result. Remember that you can filter based on the rank column.

    +

    Show the top 5 states with the highest murder rates. From here on, do not change the murders dataset, just show the result. Remember that you can filter based on the rank column.

    5. We can remove rows using the != operator. For example, to remove Florida, we would do this:

    no_florida <- murders_dt[state != "Florida"]
    @@ -596,16 +607,16 @@ 

    library(NHANES)

    -

    8. We will provide some basic facts about blood pressure. First let’s select a group to set the standard. We will use 20-to-29-year-old females. AgeDecade is a categorical variable with these ages. Note that the category is coded like ” 20-29”, with a space in front! Use the data.table package to compute the average and standard deviation of systolic blood pressure as saved in the BPSysAve variable. Save it to a variable called ref.

    +

    8. We will provide some basic facts about blood pressure. First let’s select a group to set the standard. We will use 20-to-29-year-old females. AgeDecade is a categorical variable with these ages. Note that the category is coded like " 20-29", with a space in front! Use the data.table package to compute the average and standard deviation of systolic blood pressure as saved in the BPSysAve variable. Save it to a variable called ref.

    9. Report the min and max values for the same group.

    -

    10. Compute the average and standard deviation for females, but for each age group separately rather than a selected decade as in question 1. Note that the age groups are defined by AgeDecade.

    -

    11. Repeat exercise 3 for males.

    +

    10. Compute the average and standard deviation for females, but for each age group separately rather than a selected decade as in exercise 8. Note that the age groups are defined by AgeDecade.

    +

    11. Repeat exercise 10 for males.

    12. For males between the ages of 40-49, compare systolic blood pressure across race as reported in the Race1 variable. Order the resulting table from lowest to highest average systolic blood pressure.

      -

    1. https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html↩︎

      2. +

    2. https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html↩︎

    @@ -104,7 +105,7 @@ - @@ -114,7 +115,7 @@
    Introduction to Data Science
    - +
    @@ -295,7 +296,7 @@
  • - 19  Installing Git + 19  Terminal Access and Installing Git
  • @@ -397,7 +398,7 @@

    Chapter 8 Chapter 9 and learn how to augment the code to perfect the plots. As we go through our case study, we will describe relevant general data visualization principles and learn concepts such as faceting, time series plots, transformations, and ridge plots.

    +

    In this chapter, we will demonstrate how relatively simple ggplot2 code can create insightful and aesthetically pleasing plots. As motivation we will create plots that help us better understand trends in world health and economics. We will implement what we learned in Chapter 8 and Chapter 9 and learn how to augment the code to perfect the plots. As we go through our case study, we will describe relevant general data visualization principles and learn concepts such as faceting, time series plots, transformations, and ridge plots.

    10.1 Case study 1: new insights on poverty

    Hans Rosling1 was the co-founder of the Gapminder Foundation2, an organization dedicated to educating the public by using data to dispel common myths about the so-called developing world. The organization uses data to show how actual trends in health and economics contradict the narratives that emanate from sensationalist media coverage of catastrophes, tragedies, and other unfortunate events. As stated in the Gapminder Foundation’s website:

    @@ -408,7 +409,7 @@

    3 and The Best Stats You’ve Ever Seen4. Specifically, in this section, we use data to attempt to answer the following two questions:

    +

    Hans Rosling conveyed actual data-based trends in a dramatic way of his own, using effective data visualization. This section is based on two talks that exemplify this approach to education: New Insights on Poverty3 and The Best Stats You’ve Ever Seen4. Specifically, in this section, we use data to attempt to answer the following two questions:

    1. Is it a fair characterization of today’s world to say it is divided into western rich nations and the developing world in Africa, Asia, and Latin America?
    2. Has income inequality across countries worsened during the last 40 years?
      3. @@ -546,7 +547,7 @@ 

      filter(gapminder, year %in% c(1962, 2012)) |>
   ggplot(aes(fertility, life_expectancy, col = continent)) +
@@ -626,7 +627,7 @@ 
      
 
      countries <- c("South Korea", "Germany")
 
@@ -691,8 +692,9 @@ 
      10.5 Data transformations
      
 
      We now shift our attention to the second question related to the commonly held notion that wealth distribution across the world has become worse during the last decades. When general audiences are asked if poor countries have become poorer and rich countries become richer, the majority answers yes. By using stratification, histograms, smooth densities, and boxplots, we will be able to understand if this is in fact the case. First we learn how transformations can sometimes help provide more informative summaries and plots.
      
 
      The gapminder data table includes a column with the countries’ gross domestic product (GDP). GDP measures the market value of goods and services produced by a country in a year. The GDP per person is often used as a rough summary of a country’s wealth. Here we divide this quantity by 365 to obtain the more interpretable measure dollars per day. Using current US dollars as a unit, a person surviving on an income of less than $2 a day is defined to be living in absolute poverty. We add this variable to the data table:
      
-
      
-
      gapminder <- gapminder |>  mutate(dollars_per_day = gdp/population/365)
      
+
      
+
      gapminder <- gapminder |>  
+  mutate(dollars_per_day = gdp/population/365)
      
 
      
 
      The GDP values are adjusted for inflation and represent current US dollars, so these values are meant to be comparable across the years. Of course, these are country averages and within each country there is much variability. All the graphs and insights described below relate to country averages and not to individuals.
      
 
      
@@ -713,7 +715,7 @@ 
       argument to draw a boundary and clearly distinguish the bins.
      
 
      In this plot, we see that for the majority of countries, averages are below $10 a day. However, the majority of the x-axis is dedicated to the 35 countries with averages above $10. So the plot is not very informative about countries with values below $10 a day.
      
-
      It might be more informative to quickly be able to see how many countries have average daily incomes of about $1 (extremely poor), $2 (very poor), $4 (poor), $8 (middle), $16 (well off), $32 (rich), $64 (very rich) per day. These changes are multiplicative and log transformations convert multiplicative changes into additive ones: when using base 2, a doubling of a value turns into an increase by 1.
      
+
      It might be more informative to quickly be able to see how many countries have average daily incomes of about $1 (extremely poor), $2 (very poor), $4 (poor), $8 (middle), $16 (well off), $32 (rich), $64 (very rich) per day. These changes are multiplicative, and log transformations convert multiplicative changes into additive ones: when using base 2, a doubling of a value turns into an increase by 1.
      
 
      Here is the distribution if we apply a log base 2 transform:
      
 
      
 
      gapminder |> 
@@ -731,9 +733,9 @@ 
      
 10.5.2 Which base?
      
 
      In the case above, we used base 2 in the log transformations. Other common choices are base \(\mathrm{e}\) (the natural log) and base 10.
      
-
      In general, we do not recommend using the natural log for data exploration and visualization. This is because while \(2^2, 2^3, 2^4, \dots\) or \(10^2, 10^3, \dots\) are easy to mentally compute, but the same is not true for \(\mathrm{e}^2, \mathrm{e}^3, \dots\). So natural log scale is not intuitive or easy to interpret.
      
-
      In the dollars per day example, we used base 2 instead of base 10 because the resulting range is easier to interpret. The range of the values being plotted is 0.3269426, 48.8852142.
      
-
      In base 10, this turns into a range that includes very few integers: just 0 and 1. With base two, our range includes -2, -1, 0, 1, 2, 3, 4, and 5. It is easier to compute \(2^x\) and \(10^x\) when \(x\) is an integer and between -10 and 10, so we prefer to have smaller integers in the scale. Another consequence of a limited range is that choosing the binwidth is more challenging. With log base 2, we know that a binwidth of 1 will translate to a bin with range \(x\) to \(2x\).
      
+
      In general, we do not recommend using the natural log for data exploration and visualization. This is because while \(2^2, 2^3, 2^4, \dots\) or \(10^2, 10^3, \dots\) are easy to mentally compute, but the same is not true for \(\mathrm{e}^2, \mathrm{e}^3, \dots\). The natural log scale is not intuitive or easy to interpret.
      
+
      In the dollars per day example, we used base 2 instead of base 10 because the resulting range is easier to interpret. The range of the untransformed values is 0.3269426, 48.8852142.
      
+
      In base 10, this turns into a range that includes very few integers: just 0 and 1. With base 2, our range includes -2, -1, 0, 1, 2, 3, 4, and 5. It is easier to compute \(2^x\) and \(10^x\) when \(x\) is an integer and between -10 and 10, so we prefer to have smaller integers in the scale. Another consequence of a limited range is that choosing the binwidth is more challenging. With log base 2, we know that a binwidth of 1 will translate to a bin with range \(x\) to \(2x\).
      
 
      For an example in which base 10 makes more sense, consider population sizes. A log base 10 is preferable since the range for these is:
      
 
      
 
      filter(gapminder, year == past_year) |>
@@ -761,7 +763,7 @@ 
      \(x\) is about 1.5. If the scales are logged:
      
 
      ----10---x---100------1000---
      
-
      then, to determine x, we need to compute \(10^{1.5}\), which is not easy to do in our heads. The advantage of using logged scales is that we see the original values on the axes. However, the advantage of showing logged scales is that the original values are displayed in the plot, which are easier to interpret. For example, we would see “32 dollars a day” instead of “5 log base 2 dollars a day”.
      
+
      then, to determine x, we need to compute \(10^{1.5}\), which is not easy to do in our heads. However, the advantage of showing logged scales is that the original values are displayed in the plot, which are easier to interpret. For example, we would see “32 dollars a day” instead of “5 log base 2 dollars a day”.
      
 
      As we learned earlier, if we want to scale the axis with logs, we can use the scale_x_continuous function. Instead of logging the values first, we apply this layer:
      
 
      
 
      gapminder |> 
@@ -781,7 +783,7 @@ 
      
 10.6 Multimodal distributions
      
 
      In the histogram above we see two bumps: one at about 4 and another at about 32. In statistics these bumps are sometimes referred to as modes. The mode of a distribution is the value with the highest frequency. The mode of the normal distribution is the average. When a distribution, like the one above, doesn’t monotonically decrease from the mode, we call the locations where it goes up and down again local modes and say that the distribution has multiple modes.
      
-
      The histogram above suggests that the 1970 country income distribution has two modes: one at about 2 dollars per day (1 in the log 2 scale) and another at about 32 dollars per day (5 in the log 2 scale). This bimodality is consistent with a dichotomous world made up of countries with average incomes less than $8 (3 in the log 2 scale) a day and countries above that.
      
+
      The histogram above suggests that the 1970 country income distribution has two modes: one at about 2 dollars per day (1 in the log 2 scale) and another at about 32 dollars per day (5 in the log 2 scale). This bimodality is consistent with a dichotomous world made up of countries with average incomes less than 8 dollars per day (3 in the log 2 scale) and countries above that.
      
 
      
 10.7 Comparing distributions
      
 
      A histogram showed us that the 1970 income distribution values show a dichotomy. However, the histogram does not show us if the two groups of countries are west versus the developing world.
      
@@ -800,7 +802,7 @@ 
      
 
      gapminder <- gapminder |> 
   mutate(group = case_when(
@@ -824,7 +826,7 @@ 
      
 10.7.1 Boxplots
      
-
      The exploratory data analysis above has revealed two characteristics about average income distribution in 1970. Using a histogram, we found a bimodal distribution with the modes relating to poor and rich countries. We now want to compare the distribution across these five groups to confirm the “west versus the rest” dichotomy. The number of points in each category is large enough that a summary plot may be useful. We could generate five histograms or five density plots, but it may be more practical to have all the visual summaries in one plot. We therefore start by stacking boxplots next to each other. Note that we add the layer theme(axis.text.x = element_text(angle = 90, hjust = 1)) to turn the group labels vertical, since they do not fit if we show them horizontally, and remove the axis label to make space.
      
+
      The exploratory data analysis above has revealed two characteristics about average income distribution in 1970. Using a histogram, we found a bimodal distribution with the modes relating to poor and rich countries. We now want to compare the distribution across these five groups to confirm the “west versus the rest” dichotomy. The number of points in each category is large enough that a summary plot may be useful. We could generate five histograms or five density plots, but it may be more practical to have all the visual summaries in one plot. We therefore start by stacking boxplots next to each other. Note that we add the layer theme(axis.text.x = element_text(angle = 90, hjust = 1)) to turn the group labels vertical, since they do not fit if we show them horizontally, and we remove the axis label to make space.
      
 
      
 
      p <- gapminder |> 
   filter(year == past_year & !is.na(gdp)) |>
@@ -903,7 +905,7 @@ 
      
 10.7.3 Example: 1970 versus 2010 income distributions
      
-
      Data exploration clearly shows that in 1970 there was a “west versus the rest” dichotomy. But does this dichotomy persist? Let’s use facet_grid and see how the distributions have changed. To start, we will focus on two groups: the west and the rest. We make four histograms. We make this plot only for countries with data in both 1970 and 2010. Note that several countries were founded after 1970, for example, the Soviet Union divided into several countries during the 1990s. We also note tat that data was available for more countries in 2010.
      
+
      Data exploration clearly shows that in 1970 there was a “west versus the rest” dichotomy. But does this dichotomy persist? Let’s use facet_grid and see how the distributions have changed. To start, we will focus on two groups: the west and the rest. We make four histograms. We make this plot only for countries with data in both 1970 and 2010. Note that several countries were founded after 1970; for example, the Soviet Union divided into several countries during the 1990s. We also note that that data was available for more countries in 2010.
      
 
      We therefore make the plot only for countries with data in both years:
      
 
      
 
      past_year <- 1970
@@ -916,7 +918,7 @@ 
      filter(n == 2) |>
   pull(country)
      
 
      
-
      These 108 account for 86% of the world population, so this subset should be representative. We can compare the distributions using this code:
      
+
      These 108 countries account for 86% of the world population, so this subset should be representative. We can compare the distributions using this code:
      
 
      
 
      gapminder |> 
   filter(year %in% years & country %in% country_list) |>
@@ -951,7 +953,7 @@ 
      
 
      gapminder |> 
   filter(year %in% years & country %in% country_list) |>
@@ -986,7 +988,7 @@ 
      
 10.7.4 Accessing computed variables
      
 
      To have the areas of these densities be proportional to the size of the groups, we can simply multiply the y-axis values by the size of the group. From the geom_density help file, we see that the functions compute a variable called count that does exactly this. We want this variable to be on the y-axis rather than the density.
      
@@ -1053,7 +1055,7 @@ 
      
 10.7.5 Weighted densities
      
@@ -1079,7 +1081,7 @@ 
      
 10.8.1 Logistic transformation
      
@@ -1102,7 +1104,7 @@ 
      
 10.9 Case study 3: vaccines and infectious diseases
      
-
      Vaccines have helped save millions of lives. In the 19th century, before herd immunization was achieved through vaccination programs, deaths from infectious diseases, such as smallpox and polio, were common. However, today vaccination programs have become somewhat controversial despite all the scientific evidence for their importance.
      
+
      Vaccines have helped save millions of lives. In the 19th century, before herd immunization was achieved through vaccination programs, deaths from infectious diseases, such as smallpox and polio, were common. Today however, vaccination programs have become somewhat controversial despite all the scientific evidence for their importance.
      
 
      The controversy started with a paper5 published in 1988 and led by Andrew Wakefield claiming there was a link between the administration of the measles, mumps, and rubella (MMR) vaccine and the appearance of autism and bowel disease. Despite much scientific evidence contradicting this finding, sensationalist media reports and fear-mongering from conspiracy theorists led parts of the public into believing that vaccines were harmful. As a result, many parents ceased to vaccinate their children. This dangerous practice can be potentially disastrous given that the Centers for Disease Control (CDC) estimates that vaccinations will prevent more than 21 million hospitalizations and 732,000 deaths among children born in the last 20 years (see Benefits from Immunization during the Vaccines for Children Program Era — United States, 1994-2013, MMWR6). The 1988 paper has since been retracted and Andrew Wakefield was eventually “struck off the UK medical register, with a statement identifying deliberate falsification in the research published in The Lancet, and was thereby barred from practicing medicine in the UK.” (source: Wikipedia7). Yet misconceptions persist, in part due to self-proclaimed activists who continue to disseminate misinformation about vaccines.
      
 
      Effective communication of data is a strong antidote to misinformation and fear-mongering. Earlier we used an example provided by a Wall Street Journal article8 showing data related to the impact of vaccines on battling infectious diseases. Here we reconstruct that example.
      
 
      
@@ -1141,7 +1143,7 @@ 
      10.
      
 
      Now can we show data for all states in one plot? We have three variables to show: year, state, and rate. In the WSJ figure, they use the x-axis for year, the y-axis for state, and color hue to represent rates. However, the color scale they use, which goes from yellow to blue to green to orange to red, can be improved.
      
 
      In our example, we want to use a sequential palette since there is no meaningful center, just low and high rates.
      
 
      We use the geometry geom_tile to tile the region with colors representing disease rates. We use a square root transformation to avoid having the really high counts dominate the plot. Notice that missing values are shown in grey. Note that once a disease was pretty much eradicated, some states stopped reporting cases all together. This is why we see so much grey after 1980.
      
@@ -1171,22 +1173,20 @@ 
      sum(population, na.rm = TRUE) * 10000)
      
 
      
 
      Now to make the plot we simply use the geom_line geometry:
      
-
      
+
      
 
      dat |> 
   filter(!is.na(rate)) |>
     ggplot() +
   geom_line(aes(year, rate, group = state),  color = "grey50", 
-            show.legend = FALSE, alpha = 0.2, size = 1) +
-  geom_line(mapping = aes(year, us_rate),  data = avg, size = 1) +
+            show.legend = FALSE, alpha = 0.2, linewidth = 1) +
+  geom_line(mapping = aes(year, us_rate),  data = avg, linewidth = 1) +
   scale_y_continuous(trans = "sqrt", breaks = c(5, 25, 125, 300)) + 
   ggtitle("Cases per 10,000 by state") + 
   xlab("") + ylab("") +
   geom_text(data = data.frame(x = 1955, y = 50), 
             mapping = aes(x, y, label = "US average"), 
             color = "black") + 
-  geom_vline(xintercept = 1963, col = "blue")
-#> Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
-#> ℹ Please use `linewidth` instead.
      
+  geom_vline(xintercept = 1963, col = "blue")
      
 
      
 
      
 
      
@@ -1201,21 +1201,22 @@ 
      
 
        
-
      1. https://en.wikipedia.org/wiki/Hans_Rosling↩︎
        2. 
-
      2. http://www.gapminder.org/↩︎
        3. 
-
      3. https://www.ted.com/talks/hans_rosling_reveals_new_insights_on_poverty?language=en↩︎
        4. 
-
      4. https://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen↩︎
        5. 
-
      5. http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(97)11096-0/abstract↩︎
        6. 
-
      6. https://www.cdc.gov/mmwr/preview/mmwrhtml/mm6316a4.htm↩︎
        7. 
-
      7. https://en.wikipedia.org/wiki/Andrew_Wakefield↩︎
        8. 
-
      8. http://graphics.wsj.com/infectious-diseases-and-vaccines/↩︎
        9. 
-
      9. http://www.tycho.pitt.edu/↩︎
        10. 
+
      10. https://en.wikipedia.org/wiki/Hans_Rosling↩︎
        11. 
+
      11. http://www.gapminder.org/↩︎
        12. 
+
      12. https://www.ted.com/talks/hans_rosling_reveals_new_insights_on_poverty?language=en↩︎
        13. 
+
      13. https://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen↩︎
        14. 
+
      14. http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(97)11096-0/abstract↩︎
        15. 
+
      15. https://www.cdc.gov/mmwr/preview/mmwrhtml/mm6316a4.htm↩︎
        16. 
+
      16. https://en.wikipedia.org/wiki/Andrew_Wakefield↩︎
        17. 
+
      17. http://graphics.wsj.com/infectious-diseases-and-vaccines/↩︎
        18. 
+
      18. http://www.tycho.pitt.edu/↩︎
        19. 
+
      19. Control, Centers for Disease; Prevention (2014). CDC health information for international travel 2014 (the yellow book). p. 250. ISBN 9780199948505↩︎
        20. 
 
      
 
@@ -79,7 +80,7 @@
       
       
-      
     
      
@@ -93,7 +94,7 @@
     
      
       Introduction to Data Science 
         
      
-    
+    
   
   
      
   
@@ -274,7 +275,7 @@
           
    3. 
   
       
   
- 19  Installing Git
+ 19  Terminal Access and Installing Git
   
      
 
      4. 
           
    4. 
@@ -332,12 +333,12 @@ 
      https://www.google.com/chrome/.
      
+
      The instructions below include screen shots from the installation process in which we used the Chrome browser which, although not necessary, you can freely download and install1.
      
 
      
 
      18.1 Installing R
      
 
      RStudio is an interactive desktop environment, but it is not R, nor does it include R when you download and install it. Therefore, to use RStudio, we first need to install R.
      
 
        
-
      1. You can download R from the Comprehensive R Archive Network (CRAN)1. Search for CRAN on your browser:
        2. 
+
      2. You can download R from the Comprehensive R Archive Network (CRAN)2. Search for CRAN on your browser:
        3. 
 
      
 
      
 
      
@@ -354,7 +355,7 @@ 
      
 
      Here we show screenshots for Windows, but the process is similar for the other platforms. When they differ, we will also show screenshots for Mac OS X.
      
 
        
-
      1. Once at the CRAN download page, you will have several choices. You want to install the base subdirectory. This installs the basic packages you need to get started. We will later learn how to install other needed packages from within R, rather than from this webpage.
        2. 
+
      2. Once at the CRAN download page, you will have several choices. You want to install the base subdirectory. This installs the basic packages you need to get started. Section 1.5 discusses how to install other needed packages from within R, rather than from this webpage.
        3. 
 
      
 
      
 
      
@@ -482,7 +483,7 @@ 
      
 
      
 
        
-
      1. You should find the RStudio website as shown above. Once there, click on Download RStudio.
        2. 
+
      2. You should find the Posit website3 as shown above. Once there, click on Download RStudio.
        3. 
 
      
 
      
 
      
@@ -542,7 +543,6 @@ 
      
 
      
 
      
-
      On the Mac, there are fewer clicks. You basically drag and drop the RStudio icon into the Applications folder icon here:
      
 
      
 
      
 
      
@@ -572,7 +572,9 @@ 
      
 
      
 
        
-
      1. https://cran.r-project.org/↩︎
        2. 
+
      2. https://www.google.com/chrome/↩︎
        3. 
+
      3. https://cran.r-project.org/↩︎
        4. 
+
      4. https://posit.co/products/open-source/rstudio/↩︎
        5. 
 
      
 
      
 
@@ -818,7 +820,7 @@ 
      
   
      
       
-        19  Installing Git 
+        19  Terminal Access and Installing Git 
       
   
      
 
diff --git a/docs/sitemap.xml b/docs/sitemap.xml
index 5046aca..411912c 100644
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@@ -2,114 +2,114 @@
 
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/index.html
-    2023-12-10T04:20:33.662Z
+    2023-12-10T05:11:09.513Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/intro.html
-    2023-12-10T04:20:33.672Z
+    2023-12-10T05:11:09.523Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/R/intro-to-R.html
-    2023-12-10T04:20:33.682Z
+    2023-12-10T05:11:09.532Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/R/getting-started.html
-    2023-12-10T04:20:33.699Z
+    2023-12-10T05:11:09.550Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/R/R-basics.html
-    2023-12-10T04:20:33.814Z
+    2023-12-10T05:11:09.663Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/R/programming-basics.html
-    2023-12-10T04:20:33.850Z
+    2023-12-10T05:11:09.696Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/R/tidyverse.html
-    2023-12-10T04:20:33.960Z
+    2023-12-10T05:11:09.797Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/R/data-table.html
-    2023-12-10T04:20:33.998Z
+    2023-12-10T05:11:09.833Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/R/importing-data.html
-    2023-12-10T04:20:34.030Z
+    2023-12-10T05:11:09.864Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/dataviz/intro-dataviz.html
-    2023-12-10T04:20:34.044Z
+    2023-12-10T05:11:09.877Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/dataviz/distributions.html
-    2023-12-10T04:20:34.071Z
+    2023-12-10T05:11:09.903Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/dataviz/ggplot2.html
-    2023-12-10T04:20:34.133Z
+    2023-12-10T05:11:09.963Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/dataviz/dataviz-principles.html
-    2023-12-10T04:20:34.174Z
+    2023-12-10T05:11:10.004Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/dataviz/dataviz-in-practice.html
-    2023-12-10T04:20:34.288Z
+    2023-12-10T05:11:10.095Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/wrangling/intro-to-wrangling.html
-    2023-12-10T04:20:34.305Z
+    2023-12-10T05:11:10.107Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/wrangling/reshaping-data.html
-    2023-12-10T04:20:34.351Z
+    2023-12-10T05:11:10.138Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/wrangling/joining-tables.html
-    2023-12-10T04:20:34.409Z
+    2023-12-10T05:11:10.179Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/wrangling/dates-and-times.html
-    2023-12-10T04:20:34.453Z
+    2023-12-10T05:11:10.201Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/wrangling/data-table-wrangling.html
-    2023-12-10T04:20:34.510Z
+    2023-12-10T05:11:10.228Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/wrangling/web-scraping.html
-    2023-12-10T04:20:34.568Z
+    2023-12-10T05:11:10.255Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/wrangling/string-processing.html
-    2023-12-10T04:20:34.683Z
+    2023-12-10T05:11:10.363Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/wrangling/text-analysis.html
-    2023-12-10T04:20:34.734Z
+    2023-12-10T05:11:10.411Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/productivity/intro-productivity.html
-    2023-12-10T04:20:34.742Z
+    2023-12-10T05:11:10.420Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/productivity/installing-r-and-rstudio.html
-    2023-12-10T04:20:34.759Z
+    2023-12-10T05:11:10.437Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/productivity/installing-git.html
-    2023-12-10T04:20:34.773Z
+    2023-12-10T05:11:10.450Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/productivity/unix.html
-    2023-12-10T04:20:34.802Z
+    2023-12-10T05:11:10.478Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/productivity/git.html
-    2023-12-10T04:20:34.828Z
+    2023-12-10T05:11:10.504Z
   
   
     http://rafalab.dfci.harvard.edu/dsbook-part-1/productivity/reproducible-projects.html
-    2023-12-10T04:20:34.854Z
+    2023-12-10T05:11:10.530Z
   
 
diff --git a/docs/wrangling/data-table-wrangling.html b/docs/wrangling/data-table-wrangling.html
index 2518f33..a3d259f 100644
--- a/docs/wrangling/data-table-wrangling.html
+++ b/docs/wrangling/data-table-wrangling.html
@@ -2,7 +2,7 @@
 
 
 
-
+
 
 
 Introduction to Data Science - 14  Wrangling with data.table
@@ -88,7 +88,8 @@
     "search-more-matches-text": "more matches in this document",
     "search-clear-button-title": "Clear",
     "search-detached-cancel-button-title": "Cancel",
-    "search-submit-button-title": "Submit"
+    "search-submit-button-title": "Submit",
+    "search-label": "Search"
   }
 }
 
@@ -103,7 +104,7 @@
       
       
-      
     
      5. 
@@ -113,7 +114,7 @@
     
      
       Introduction to Data Science 
         
      
-    
+    
   
   
      
   
@@ -294,7 +295,7 @@
           
    5. 
   
       
   
- 19  Installing Git
+ 19  Terminal Access and Installing Git
   
      
 
      6. 
           
    6. 
@@ -367,7 +368,7 @@ 
      
 
      
 
 
      
-
      Here we show the data.table version of some of the tidyverse commands we previously showed. The data.table functions are faster and more efficient with memory. In general, eveyrthing you can with tidyverse has a way to do it with **data.table* and R base which, although perhaps harder to read, it is often more flexible, faster, and more efficient. Here we show just a few examples, but you can learn others using internet searches or code generation tools.
      
+
      Here we show the data.table version of some of the tidyverse commands we previously showed. The data.table functions are faster and more efficient with memory. In general, everything you can do with tidyverse can be done with data.table and base R which, although perhaps harder to read, it is often more flexible, faster, and more efficient. Here we show just a few examples, but you can learn others using internet searches or code generation tools.
      
 
      
 14.1 Reshaping data
      
 
      Previously we used this example:
      
@@ -379,13 +380,13 @@ 
      
 
      
 14.1.1 pivot_longer is melt
 
      
-
      If in tidyeverse we write
      
+
      If in tidyverse we write
      
 
      
 
      wide_data <- read_csv(filename)
 new_tidy_data <- wide_data |>
   pivot_longer(-1, names_to = "year", values_to = "fertility")
      
 
      
-
      in data.table we use the melt function
      
+
      in data.table we use the melt function.
      
 
      
 
      dt_wide_data <- fread(filename) 
 dt_new_tidy_data  <- melt(dt_wide_data, 
@@ -396,12 +397,12 @@ 
      
 
      
 14.2 pivot_wider is dcast
 
      
-
      If in tidyeverse we write
      
+
      If in tidyverse we write
      
 
      
 
      new_wide_data <- new_tidy_data |> 
   pivot_wider(names_from = year, values_from = fertility)
      
 
      
-
      in data.table we write:
      
+
      in data.table we use the dcast function.
      
 
      
 
      dt_new_wide_data <- dcast(dt_new_tidy_data, formula = ... ~ year,
                           value.var = "fertility")
      
@@ -436,23 +437,24 @@

    14.3 Joins

    -

    In *tidyverse** we joined two table like this:

    +

    In tidyverse we joined two tables with left_join:

    tab <- left_join(murders, results_us_election_2016, by = "state")

    In data.table the merge functions works similarly:

    -
    -
    tab <- merge(murders, results_us_election_2016, by = "state", all.x = TRUE)
    +
    +
    tab <- merge(murders, results_us_election_2016, 
+             by = "state", all.x = TRUE)

    Instead of defining different functions for the different type of joins, merge uses the the logical arguments all (full join), all.x (left join), and all.y (right join).

    14.4 Dates and times

    -

    The data.table package also includes some of the functionality to lubridate. For example, it includes the mday, month, and year functions:

    +

    The data.table package also includes some of the same functionality as lubridate. For example, it includes the mday, month, and year functions:

    data.table::mday(now())
-#> [1] 7
+#> [1] 28
 data.table::month(now())
-#> [1] 12
+#> [1] 11
 data.table::year(now())
 #> [1] 2023
    @@ -460,7 +462,7 @@

    The package also includes the class IDate and ITime, which store dates and times more efficiently, convenient for large files with date stamps. You convert dates in the usual R format using as.IDate and as.ITime.

    14.5 Exercises

    -

    Repear exercises in Chapter 11, Section 12.1, and Chapter 13 using data.table instead of tidyverse.

    +

    Repeat exercises in Chapter 11, Section 12.1, and Chapter 13 using data.table instead of tidyverse.

    @@ -103,7 +104,7 @@ -

    • @@ -113,7 +114,7 @@
    Introduction to Data Science
    - +
    @@ -294,7 +295,7 @@
  • - 19  Installing Git + 19  Terminal Access and Installing Git
  • @@ -360,7 +361,7 @@

    library(textdata) library(dslabs)

    • -

    X.com (formerly twitter) provides an API that permits downloading tweets. Brendan Brown runs the trump archive^[https://www.thetrumparchive.com/], which compiles tweet data from Trump’s account. The dslabs package includes tweets from the following range:

    +

    X, formerly known as twitter, provides an API that permits downloading tweets. Brendan Brown runs the trump archive4, which compiles tweet data from Trump’s account. The dslabs package includes tweets from the following range:

    range(trump_tweets$created_at)
 #> [1] "2009-05-04 13:54:25 EST" "2018-01-01 08:37:52 EST"
    @@ -389,7 +390,7 @@

    #> 4 TweetDeck 468 #> 5 TwitLonger Beta 288

    -

    We are interested in what happened during the 2016 campaign, so for this analysis we will focus on what was tweeted between the day Trump announced his campaign and election day. We define the following table containing just the tweets from that time period. We remove the Twitter for part of the source and filter out retweets.

    +

    We are interested in what happened during the 2016 campaign, so for this analysis we will focus on what was tweeted between the day Trump announced his campaign and election day. We define the following table containing just the tweets from that time period. We remove the Twitter for part of the source, only keep tweets from Android or iPhone, and filter out retweets.

    campaign_tweets <- trump_tweets |> 
   filter(source %in% paste("Twitter for", c("Android", "iPhone")) &
@@ -400,7 +401,7 @@ 
    
   arrange(created_at) |> 
   as_tibble()
    -

    We can now use data visualization to explore the possibility that two different groups were tweeting from these devices. For each tweet, we will extract the hour, East Coast time (EST), it was tweeted and then compute the proportion of tweets tweeted at each hour for each device:

    +

    We can now use data visualization to explore the possibility that two different groups were tweeting from these devices. For each tweet, we will extract the hour, Eastern Standard Time (EST), it was tweeted and then compute the proportion of tweets tweeted at each hour for each device:

    campaign_tweets |>
   mutate(hour = hour(with_tz(created_at, "EST"))) |>
@@ -421,11 +422,11 @@

    We notice a big peak for the Android in the early hours of the morning, between 6 and 8 AM. There seems to be a clear difference in these patterns. We will therefore assume that two different entities are using these two devices.

    -

    We will now study how the tweets differ when we compare Android to iPhone. To do this, we introduce the tidytext package.

    +

    We will now study how the text of the tweets differ when we compare Android to iPhone. To do this, we introduce the tidytext package.

    17.2 Text as data

    The tidytext package helps us convert free form text into a tidy table. Having the data in this format greatly facilitates data visualization and the use of statistical techniques.

    -

    The main function needed to achieve this is unnest_tokens. A token refers to a unit that we are considering to be a data point. The most common token will be words, but they can also be single characters, n-grams, sentences, lines, or a pattern defined by a regex. The functions will take a vector of strings and extract the tokens so that each one gets a row in the new table. Here is a simple example:

    +

    The main function needed to achieve this is unnest_tokens. A token refers to a unit that we are considering to be a data point. The most common token will be words, but they can also be single characters, n-grams, sentences, lines, or a pattern defined by a regex. The function will take a vector of strings and extract the tokens so that each one gets a row in the new table. Here is a simple example:

    poem <- c("Roses are red,", "Violets are blue,", 
           "Sugar is sweet,", "And so are you.")
@@ -450,7 +451,7 @@ 
    
 #> 5     2 are    
 #> # ℹ 8 more rows
    -

    Now let’s look at trump tweet. We will look at tweet number 3008 because it will later permit us to illustrate a couple of points:

    +

    Now let’s look at Trump’s tweets. We will look at tweet number 3008 because it will later permit us to illustrate a couple of points:

    i <- 3008
 campaign_tweets$text[i] |> str_wrap(width = 65) |> cat()
@@ -478,7 +479,7 @@ 
    
 #> [13] "davenport"      "this"           "past"           "winter"        
 #> [17] "maga"
    -

    Now we are now ready to extract the words for all our tweets.

    +

    Now we are now ready to extract the words from all our tweets.

    tweet_words <- campaign_tweets |> 
   mutate(text = str_remove_all(text, links_to_pics))  |>
@@ -520,15 +521,14 @@ 
    
   filter(!word %in% stop_words$word )

    we end up with a much more informative set of top 10 tweeted words:

    -
    +
    tweet_words |> 
   count(word) |>
   top_n(10, n) |>
-  mutate(word = reorder(word, n)) |>
   arrange(desc(n))
 #> # A tibble: 10 × 2
 #>   word                      n
-#>   <fct>                 <int>
+#>   <chr>                 <int>
 #> 1 trump2016               415
 #> 2 hillary                 407
 #> 3 people                  304
@@ -546,7 +546,7 @@ 
    
   mutate(word = str_replace(word, "^'", ""))

    Now that we have all our words in a table, along with information about what device was used to compose the tweet they came from, we can start exploring which words are more common when comparing Android to iPhone.

    -

    For each word, we want to know if it is more likely to come from an Android tweet or an iPhone tweet. We therefore compute, for each word, what proportion of all words it represent for Android and iPhone, respectively.

    +

    For each word, we want to know if it is more likely to come from an Android tweet or an iPhone tweet. We therefore compute, for each word, its frequency among words tweeted from Android and iPhone, respectively.

    android_vs_iphone <- tweet_words |>
   count(word, source) |>
@@ -586,7 +586,7 @@

    17.3 Sentiment analysis

    In sentiment analysis, we assign a word to one or more “sentiments”. Although this approach will miss context-dependent sentiments, such as sarcasm, when performed on large numbers of words, summaries can provide insights.

    -

    The first step in sentiment analysis is to assign a sentiment to each word. As we demonstrate, the tidytext package includes several maps or lexicons. We textdata package includes several of these lexicons.

    +

    The first step in sentiment analysis is to assign a sentiment to each word. As we demonstrate, the tidytext package includes several maps or lexicons. The textdata package includes several of these lexicons.

    The bing lexicon divides words into positive and negative sentiments. We can see this using the tidytext function get_sentiments:

    @@ -675,7 +675,7 @@

    #> # ℹ 6 more rows

    So we do see some differences and the order is interesting: the largest three sentiments are disgust, anger, and negative!

    -

    If we are interested in exploring which specific words are driving these differences, we can refer back to our android_iphone_or object:

    +

    If we are interested in exploring which specific words are driving these differences, we can refer back to our android_vs_iphone object:

    android_vs_iphone |> inner_join(nrc, by = "word") |>
   filter(sentiment == "disgust") |>
@@ -699,7 +699,7 @@
    -

    This is just a simple example of the many analyses one can perform with tidytext. To learn more, we again recommend the Tidy Text Mining book4.

    +

    This is just a simple example of the many analyses one can perform with tidytext. To learn more, we again recommend the Tidy Text Mining book5.

    17.4 Exercises

    Project Gutenberg is a digital archive of public domain books. The R package gutenbergr facilitates the importation of these texts into R.

    @@ -713,7 +713,7 @@ 

    gutenberg_metadata

    1. Use str_detect to find the ID of the novel Pride and Prejudice.

    -

    2. We notice that there are several versions. The gutenberg_works() function filters this table to remove replicates and include only English language works. Read the help file and use this function to find the ID for Pride and Prejudice.

    +

    2. We notice that there are several versions. The gutenberg_works() function filters this table to remove replicates and include only English language works. Read the help file and use this function to find the ID for Pride and Prejudice.

    3. Use the gutenberg_download function to download the text for Pride and Prejudice. Save it to an object called book.

    4. Use the tidytext package to create a tidy table with all the words in the text. Save the table in an object called words

    5. We will later make a plot of sentiment versus location in the book. For this, it will be useful to add a column with the word number to the table.

    @@ -725,10 +725,11 @@

      -

    1. https://twitter.com/tvaziri/status/762005541388378112/photo/1↩︎

      2. -

    2. http://varianceexplained.org/r/trump-tweets/↩︎

      3. -

    3. https://www.tidytextmining.com/↩︎

      4. -

    4. https://www.tidytextmining.com/↩︎

      5. +

    5. https://twitter.com/tvaziri/status/762005541388378112/photo/1↩︎

      6. +

    6. http://varianceexplained.org/r/trump-tweets/↩︎

      7. +

    7. https://www.tidytextmining.com/↩︎

      8. +

    8. https://www.thetrumparchive.com/↩︎

      9. +

    9. https://www.tidytextmining.com/↩︎