Charles_Dickens.Rmd

---
title: "Text Mining Charles Dicken's Novels"
output: html_document
---

<iframe src="//giphy.com/embed/l3q2zHt5ntBe3FfZm" width="366" height="480" frameBorder="0" class="giphy-embed" allowFullScreen></iframe><p><a href="https://giphy.com/gifs/usnationalarchives-vintage-throwback-l3q2zHt5ntBe3FfZm">via GIPHY</a></p>

Dickens wrote fourteen and a half novels. I will start from  analyzing five of them - "A Tale of Two Cities", "Great Expectations", "A Christmas Carol in Prose; Being a Ghost Story of Christmas", "Oliver Twist" and "Hard Times". 

[Project Gutenberg](https://www.gutenberg.org/) offers over 53,000 free books. I will download Dickens' novels in UTF-8 encoded texts from there using `gutenbergr` package developed by [David Robinson](http://varianceexplained.org/). Besides, I will be using the following packages for this project. 

```{r global_options, include=FALSE}
knitr::opts_chunk$set(echo=FALSE, warning=FALSE, message=FALSE)
```

```{r}
library(dplyr)
library(tm.plugin.webmining)
library(purrr)
library(tidytext)
library(gutenbergr)
library(ggplot2)
```

```{r}
dickens <- gutenberg_download(c(98, 1400, 46, 730, 786))
```

Download Dickens' five novels by Project Gutenberg ID numbers.

```{r}
tidy_dickens <- dickens %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words)
```

The `unnest_tokens` package is used to split each row so that there is one token (word) in each row of the new data frame (tidy_dickens). Then remove stop words with an `anti_join` function.

```{r}
tidy_dickens %>%
  count(word, sort = TRUE)
```

After removing the stop words, here is a list of words starts from the most frequent.

```{r}
tidy_dickens %>%
  count(word, sort = TRUE) %>%
  filter(n > 600) %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n)) +
  geom_col() +
  xlab(NULL) +
  coord_flip() + ggtitle("The Most Common Words in Charles Dickens' Novels")
```

### Sentiment in Dickens' Five Novels

`tidytext` package contains several sentiment lexicons, I am using ["bing"](https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html) for the following tasks. 

```{r}
bing_word_counts <- tidy_dickens %>%
  inner_join(get_sentiments("bing")) %>%
  count(word, sentiment, sort = TRUE) %>%
  ungroup()

bing_word_counts
```

Here I got the sentiment categories of Dickens' words.

```{r}
bing_word_counts %>%
  group_by(sentiment) %>%
  top_n(10) %>%
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~sentiment, scales = "free_y") +
  labs(y = "Words Contribute to sentiment",
       x = NULL) +
  coord_flip()
```

The word "miss" is the most frequent negative word here, but it is used to describe unmarried women in Dickens' works. In particualr, Miss Havisham is a significant character in the Charles Dickens novel "Great Expectations". Dickens describes her as looking like "the witch of the place". In this case, probably "miss"" should be a negative word.

<iframe src="//giphy.com/embed/5AD8Z7w7HLGqQ" width="480" height="264.96" frameBorder="0" class="giphy-embed" allowFullScreen></iframe><p><a href="https://giphy.com/gifs/ghost-bbc-hat-5AD8Z7w7HLGqQ">via GIPHY</a></p>

Oh poor Pip!

```{r}
library(wordcloud)
tidy_dickens %>%
  anti_join(stop_words) %>%
  count(word) %>%
  with(wordcloud(word, n, max.words = 100))
```

World cloud is a good idea to identify trends and patterns that would otherwise be unclear or difficult to see in a tabular format.

```{r}
library(reshape2)
tidy_dickens %>%
  inner_join(get_sentiments("bing")) %>%
  count(word, sentiment, sort = TRUE) %>%
  acast(word ~ sentiment, value.var = "n", fill = 0) %>%
  comparison.cloud(colors = c("#F8766D", "#00BFC4"),
                   max.words = 100)
```

And compare most frequent positive and negative words in word cloud.

### Relationships between words

```{r}
dickens_bigrams <- dickens %>%
  unnest_tokens(bigram, text, token = "ngrams", n = 2)
dickens_bigrams
```

Each token now represents a bigram (two words paired). if one of the words in the bigram is a stop word, this word will be removed. After filtering out stop words, what are the most frequent bigrams?

```{r}
library(tidyr)
bigrams_separated <- dickens_bigrams %>%
  separate(bigram, c("word1", "word2"), sep = " ")
bigrams_filtered <- bigrams_separated %>%
  filter(!word1 %in% stop_words$word) %>%
  filter(!word2 %in% stop_words$word)
bigram_counts <- bigrams_filtered %>% 
  count(word1, word2, sort = TRUE)
bigram_counts
```

Names are the most common paired words in Dickens' novels.

```{r}
bigrams_united <- bigrams_filtered %>%
  unite(bigram, word1, word2, sep = " ")
bigrams_united
```

Before visualize it, I have to make these bigram back to united.

```{r}
bigram_tf_idf <- bigrams_united %>%
  count(bigram)
bigram_tf_idf <- bigram_tf_idf %>% filter(n>30)
ggplot(aes(x = reorder(bigram, n), y=n), data=bigram_tf_idf) + geom_bar(stat = 'identity') + ggtitle("The Most Common Bigrams in Dickens' novels") + coord_flip()
```

Yes, the most frequent bigrams in Dickens' works are names. I also notice some pairings of a common verb such as "wine shop" from "A Tale of Two Cities" and "oliver twist".

At last, visualizing a network of bigrams of Dickens' five novels.

```{r}
library(igraph)
bigram_graph <- bigram_counts %>%
  filter(n > 20) %>%
  graph_from_data_frame()

bigram_graph
```

```{r}
library(ggraph)
set.seed(2017)

ggraph(bigram_graph, layout = "fr") +
  geom_edge_link() +
  geom_node_point(color = "darkslategray4", size = 3) +
  geom_node_text(aes(label = name), vjust = 1.8) + ggtitle("Common Bigrams in Dickens' five Novels")
```

Wow, that was so much fun! I don't want to finish as yet. I want to look into one of these five novels - "A Tale of Two Cities". 

This time I will download the plain text file for "A Tale of Two Cities" only, leave out the Project Gutenberg header and footer information, then concatenate these lines into paragraphs as following. 

```{r}
library(readr)
library(stringr)
raw_tale <- read_lines("ta98-0.txt", skip = 30, n_max = 15500)
tale <- character()
for (i in seq_along(raw_tale)) {
        if (i%%10 == 1) tale[ceiling(i/10)] <- str_c(raw_tale[i], 
                                                     raw_tale[i+1],
                                                     raw_tale[i+2],
                                                     raw_tale[i+3],
                                                     raw_tale[i+4],
                                                     raw_tale[i+5],
                                                     raw_tale[i+6],
                                                     raw_tale[i+7],
                                                     raw_tale[i+8],
                                                     raw_tale[i+9], sep = " ")
}
```

```{r}
tale[9:10]
```

### Sentiment in "A Tale of Two Cities"

Apply [NRC sentiment dictionary](http://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm) to this novel.

```{r}
library(syuzhet)
tale_nrc <- cbind(linenumber = seq_along(tale), get_nrc_sentiment(tale))
```

Create a data frame combine the line number of the book with the sentiment score, then extract positive and negative scores for visualization. 

```{r}
tale_nrc$negative <- -tale_nrc$negative
pos_neg <- tale_nrc %>% select(linenumber, positive, negative) %>% 
        melt(id = "linenumber")
names(pos_neg) <- c("linenumber", "sentiment", "value")
```

```{r}
library(ggthemes)
ggplot(data = pos_neg, aes(x = linenumber, y = value, fill = sentiment)) +
        geom_bar(stat = 'identity', position = position_dodge()) + theme_minimal() +
        ylab("Sentiment") + 
        ggtitle("Positive and Negative Sentiment in A Tale of Two Cities") +
  scale_color_manual(values = c("orange", "blue")) +
  scale_fill_manual(values = c("orange", "blue"))
```

Seems the positive scores are higher than the negative scores overall, it does make sense given the content of the novel. 

```{r}
emotions <- tale_nrc %>% select(linenumber, anger, anticipation, 
                                      disgust, fear, joy, sadness, surprise, 
                                      trust) %>% 
        melt(id = "linenumber")
names(emotions) <- c("linenumber", "sentiment", "value")
emotions_group <- group_by(emotions, sentiment)
by_emotions <- summarise(emotions_group, 
                         values=sum(value))
ggplot(aes(reorder(x=sentiment, values), y=values, fill=sentiment), data = by_emotions) +
  geom_bar(stat = 'identity') + ggtitle('Sentiment in A Tale of Two Cities') +
  coord_flip() + theme(legend.position="none")
```

### The End 

Again, that was so much fun, my post just touched a bit of it on text mining. There are many more things to do such as comparing text across different novelists, save it to the next time. 

Source code that created this post can be found [here](https://github.com/susanli2016/Data-Analysis-with-R/blob/master/Charles_Dickens.Rmd). I am happy to hear any feedback and questions. 

References: 

[Text Mining With R](http://tidytextmining.com/) By [Julia Silge](http://juliasilge.com/) and [David Robinson](http://varianceexplained.org/).