Suits.Rmd

---
title: 'Suits(USA network): Text Mining with R'
author: "R. Mukamuri"
date: "July 8, 2016"
output: pdf_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```
Load the required libraries
```{r}
library(rvest)
```
Select tv show
```{r}
tvshow <- "suits"
```
Create download directory and change to it
```{r}
directory = paste("~/Data Analysis/files/", tvshow,
sep="")
dir.create(directory, recursive = TRUE, showWarnings = FALSE)
setwd(directory)
```
Setting base url and complete url
```{r}
baseurl <- "http://www.springfieldspringfield.co.uk/"
url <- paste(baseurl, "episode_scripts.php?tv-show=",
             tvshow, sep="")

```
Read the HTML page
```{r}
scrape_url <- read_html(url)
# node selector
s_selector <- ".season-episode-title"
# scrape href nodes in .season-episode-title
all_urls_season <- html_nodes(scrape_url, s_selector) %>%
  html_attr("href")
# Show some structure of the all_url_seasons.
str(all_urls_season)
# first 6 episodes
head(all_urls_season)
# last 6
tail(all_urls_season)

```
We have 76 url episodes, Now we have all the variables and season urls,lets harvest the scripts and save them to seperate text files for doing our text mining
```{r}
# Loop through all the season urls
for (i in all_urls_season){
  uri <- read_html(paste(baseurl, i, sep = "/"))
  # same thing here first check which node we need to select, so first do an inspection of the site
script_selector <- ".scrolling-script-container"
# scrape all script text to a variable
text <- html_nodes(uri, script_selector) %>%
  html_text()
# Get last five characters of all_url_season as season for saving this to seperate text files
substrRight <- function(x, n) {
  substr(x, nchar(x)-n+1, nchar(x))
}
seasons <- substrRight(i, 5)
# Write each script to a seperate text file
write.csv(text, file = paste(directory, "/", tvshow,
                             "_", seasons, ".txt", sep = ""), row.names = FALSE)
}

```
Start the text mining
```{r}
#load library
library(tm)
# set filepath to scripts
cname <- file.path(directory)
# see if the filepath contains our scripts
(docname <- dir(cname))

```
```{r}
# Crete a Corpus of the text files so we can do some analysis
docs <- Corpus(DirSource(cname), readerControl = list(id=docname))
# Show summary of the Corpus, we have 40 document in our Corpus
summary(docs)
# Inspect the first document, it has 12958 characters
inspect(docs[1])


```
There is a lot of information in the script we do not need and is not useful for text mining. We need to clean it up. We remove all numbers, convert text to lowercase, remove punctuation and stopwords, in this case english.
```{r}
docs <- tm_map(docs, tolower)
docs <- tm_map(docs, removePunctuation)
docs <- tm_map(docs, removeNumbers)
docs <- tm_map(docs, removeWords, stopwords("english"))

```
Now we will perform stemming, a stem is a form to which affixes can be attached. An example of this is wait, waits, waited, waiting, all of them are common to wait.
```{r}
library(SnowballC)
docs <- tm_map(docs, stemDocument)
```
We have removed a lot of characters which resulted in a lot of whitespaces, we remove this also.
```{r}
docs <- tm_map(docs, stripWhitespace)
#Let's have a look to our first document.

inspect(docs[1])

```
I have hash it because wordpress has problems with editing the post.We are ready with preprosessing the data and turn the document back as plain text documents.
```{r}
docs <- tm_map(docs, PlainTextDocument)

```
Create a Term Document Matrix of our documents. Which reflects the number of times each term in the corpus is found in each of the documents. And add some readable columnnmes
```{r}
# Create a tdm
tdm <- TermDocumentMatrix(docs)
# Add readable columnnames, in our case the document filename
docname <- gsub("suits_", "",docname)
docname <- gsub(".txt", "",docname)
docname <- paste("s",docname, sep="")
colnames(tdm) <- docname
# Show and inspect the tdm
tdm

inspect(tdm[1:10,1:6])

```
Do the same for a Document Term Matrix (this is a transpose of a tdm)
```{r}
dtm <- DocumentTermMatrix(docs)
rownames(dtm) <- docname
dtm
inspect(dtm[1:10,1:6])

```
Term frequency
Let have a look of the most frequently terms first and show the top 20.
```{r}
freq <- sort(colSums(as.matrix(dtm)), decreasing=TRUE)
head(freq,20)

```
Plotting the terms frequencies

Add is to a data frame so we can plot it and show the top 20.
```{r}
tf <- data.frame(term=names(freq), freq=freq)   
head(tf,20)

#Let's plot it.

# descending sort of teh tf by freq
tf$term <- factor(tf$term, levels = tf$term[order(-tf$freq)])
library(ggplot2)
p <- ggplot(subset(tf, freq>1000), aes(term, freq))    
p <- p + geom_bar(stat="identity")   
p <- p + theme(axis.text.x=element_text(angle=45, hjust=1))   
p

```
The most frequent term in "know"" followe by "don't", Harvey, Lewis and Mike are the most mentioned.As we can see in our first look at the tdm, we have a lot op sparse terms in our documents (97%). That is a lot, lets remove these.
```{r}
tdm.common = removeSparseTerms(tdm, sparse = 0.014)
tdm
tdm.common

```
That is a 90% less sparsity. See how many terms we had and now have
```{r}
dim(tdm)
dim(tdm.common)

```
Hmm from 12798 terms to only 186 terms, we inspect the first 10 terms of the first 6 documents.
```{r}
inspect(tdm.common[1:10,1:6])

```
Let visualize these most common terms in a heatmap with ggplot. As ggplot works with a matrix we need to convert the tdm.comon to a matrix because the tdm is a spare matrix.
```{r}
tdm.dense <- as.matrix(tdm.common)
dim(tdm.dense)

```
We need the data as a normal matrix in order to produce the visualisation.
```{r}
library(reshape2)
tdm.dense.m <- melt(tdm.dense, value.name = "count")
head(tdm.dense.m)

```
Make the heatmap visualization.
```{r}
library(ggplot2)
ggplot(tdm.dense.m, aes(x = Docs, y = Terms, fill = log10(count))) +
     geom_tile(colour = "white") +
     scale_fill_gradient(high="steelblue" , low="white")+
     ylab("") +
     theme(panel.background = element_blank()) +
     theme(axis.text.x = element_text(angle = 90, hjust = 1, vjust = 0.5))

```
was expecting "bullshit" to be among the most common words haha.
Now we plot a correlogram of the episodes.Note: Correlogram is a graph of correlation matrix. It is very useful to highlight the most correlated variables in a data table. In this plot, correlation coefficients is colored according to the value. Correlation matrix can be also reordered according to the degree of association between variables.
```{r}
corr <- cor(tdm.dense)
library(corrplot)
corrplot(corr, method = "circle", type = "upper", tl.col="black", tl.cex=0.7)

```
Transpose the tdm.dense so we can plot a correlogram of the terms.
```{r}
tdm.dense.t <- t(tdm.dense)
corr.t <- cor(tdm.dense.t)
corrplot(corr.t,method = "circle", type = "upper", tl.col="black", tl.cex=0.7)

```
To be continued...