R : Webcrawler Parser with Try-Catch

Well, I felt the need to do some analysis over content hosted on some initial set of web-sites and then aggregate it, plot. I created a very simple and easy parser which would crawl and parse the data read from these websites. I used the try-catch block for fault tolerance and resilience from error (website down, unavailable or trust error, etc). Here is a sample code, where I have used tryCatch and readLines methods :

  
>myUrlStats <- function(urlToCrawl) {
    statData <- tryCatch(
               {
   dataReadFromUrls <- readLines(con=urlToCrawl)
   myWebCrawlParser(dataReadFromUrls)
  },
        error=function(errorMessageStr) {
            message(paste("URL does not seem to exist:", urlToCrawl))
            message("Error message:")
            message(errorMessageStr)
            return(-1)
        },
        warning=function(warningMessageStr) {
            message(paste("URL caused a warning:", urlToCrawl))
            message("Warning message:")
            message(warningMessageStr)
            # Choose a return value in case of warning
            return(NULL)
        },
        finally={
   ##Clean up code
  }
 )
    return(statData)
}


> myWebCrawlParser <- function(dataReadFromURL){
# Do your analysis parsing here
# also like you can mine other outlinks from this data read for further traversing the web-links
return(1)
}


> urlToCrawl <- c(
  "http://superdevresources.com",
     "http://superbloggingresources.com"
     )
> finalReslt<- mapply(myUrlStats, urlToCrawl)

Happy programming!

R: Alternative and easy approach to Data concatenation

Sometimes in R you have the requirement of concatenating/appending data from two sources to create one Super Set, for eg. You may have a list of countries and economic growth indicators, and you want to create a data set which is a super set of this data (AxB), here is how you can do it via simple apply operation:

> myCustColNames
  MyColNames
1          A
2          B
3          C
4          D
5          E
> growthIndicator
[1] "GDP"        "Inflation"  "Population"
> apply(myCustColNames,1, paste, growthIndicator, sep="_")
     [,1]           [,2]           [,3]           [,4]           [,5]          
[1,] "A_GDP"        "B_GDP"        "C_GDP"        "D_GDP"        "E_GDP"       
[2,] "A_Inflation"  "B_Inflation"  "C_Inflation"  "D_Inflation"  "E_Inflation" 
[3,] "A_Population" "B_Population" "C_Population" "D_Population" "E_Population"
> 

Another example can be:

> myCustColNames<-data.frame(MyColNames=LETTERS[1:5])
> myCustColNames
  MyColNames
1          A
2          B
3          C
4          D
5          E
 > apply(myCustColNames,1, paste,seq(1:6), sep="")
     [,1] [,2] [,3] [,4] [,5]
[1,] "A1" "B1" "C1" "D1" "E1"
[2,] "A2" "B2" "C2" "D2" "E2"
[3,] "A3" "B3" "C3" "D3" "E3"
[4,] "A4" "B4" "C4" "D4" "E4"
[5,] "A5" "B5" "C5" "D5" "E5"
[6,] "A6" "B6" "C6" "D6" "E6"
> 

R: Converting a list to a dataframe

Sometimes a function returns a list data type, which you often need to convert to a data.frame data type. To convert the list to data.frame ran the following statement:

 compositeRowOfDataFrame<-do.call(rbind.data.frame, listData)

R deleting rows in a dataframe

To remove all rows from a data frame in R, you can use any of the following ways:

> b
   x  y   Result
1  1  1   B
2  1  2   B
3  1  3   C
4  1  4   A
5  1  5   B
6  1  6   A
7  1  7   C
8  1  8   C
9  1  9   B
10 1 10   C
> deleteRowsVec=c(FALSE)
> b[deleteRowsVec,]
[1] x   y   Result
<0 rows> (or 0-length row.names)

R: How to get current working directory and change it

If you are working on a R project, and you wish to load/save your data in some file, it is important to know which directory you are currently working in, so that you can easily browse or save your data to that directory or may be change it to a specific location. Here are the list of commands to get you going :

> getwd()
[1] "C:/Users/parag/Documents"
> ?setwd
> setwd("X:/Share")
> getwd()
[1] "X:/Share"
> 

R: Selecting unique values out of dataset

Many a times you have a dataset or a column which has a range of values and you wish to know the domain or range of values the column assumes. To do so run the following command:

>  y<-c('A', 'A','A','A','B', 'B','B','B','B','B','B','B','B', 'C','C','D','D', 'E','E', 'E','E','E' ,'E','E','E' ,'E','F','F','F', 'F','F', 'F','F', 'F','F')
> levels(factor(y))
[1] "A" "B" "C" "D" "E" "F"
> unique(y)
[1] "A" "B" "C" "D" "E" "F"

Renumbering rows after ordering in R

Many a times after data cleaning and reordering, the dataset row numbers get jumbled up. To solve the issue simple run the following command:

> row.names(myDataFrame) <- 1:nrow(myDataFrame)

Happy programming!!

R : Adding column names to data

Sometimes you have a huge data with multiple fields and the data stored in your file is not marked with header information (i.e. column names).
To add column names to your data frame or matrix after it is read into a variable run the following command:

> CustColName <- c("time", "Age", "Surname", "Standard", "RollNo", "Name")
> colnames(StudData) <- CustColName

R : Removing empty rows from Data

Often the data we have collected in R has empty rows or has some rows which have the interesting or mandatory columns missing or NA.

To remove such rows from your data run the following:

 myDF <- myDF[!is.na(studentData$RollNo),]

R : Subsetting data

To subset a data frame or matrix:

> x <- c(1,2,3,4,5,6)

> y <- c(3,5,2,4,1,4)

> z <- c(2,3,4,3,2,1)

> dataF<-data.frame(x,y,z)

> dataF

  x y z

1 1 3 2

2 2 5 3

3 3 2 4

4 4 4 3

5 5 1 2

6 6 4 1

> dataF[2,]

  x y

2 2 5

> dataF[2:5,]

  x y

2 2 5

3 3 2

4 4 4

5 5 1

> dataF$x <- NULL

> dataF

  y

1 3

2 5

3 2

4 4

5 1

6 4

> 

If you are beginning with the R language, and need tips and help feel free to put down a comment below. 

R: Renaming Row numbers (Identifiers)

Many a times in R you have a dataset for which R provides a default numerical representation for identifying the individual rows :

> myData <- cbind(x = 3, y = c(4:1, 2:5))
> myData
     x  y
[1,]  3  4
[2,]  3  3
[3,]  3  2
[4,]  3  1
[5,]  3  2
[6,]  3  3
[7,]  3  4
[8,]  3  5

If you want to rename the rows as per your needs, you can do so by running the following command:

> dimnames(myData)[[1]] <- letters[1:8]
> myData
   x y
a  3  4
b  3  3
c  3  2
d  3  1
e  3  2
f  3  3
g  3  4
h  3  5

Happy programming!

R: Appending rows to create new data (appending columns)

If you have a requirement of appending rows from different sources to create a new data row (for eg. Professor and Department he works in, to create an information snippet) here is the simplest trick, use cbind:

> profSharma<-professors(professors$Pid==3,)
> phyDepartment<-department(department$Did==19,)
> cbind(profSharma,phyDepartment)

Happy Programming!

R: Removing columns from a Dataset

If you need to remove certain columns from your dataset, you can do it the following way:

> iris[1:5,]
  Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1          5.1         3.5          1.4         0.2  setosa
2          4.9         3.0          1.4         0.2  setosa
3          4.7         3.2          1.3         0.2  setosa
4          4.6         3.1          1.5         0.2  setosa
5          5.0         3.6          1.4         0.2  setosa
> iris[,-5][1:5,]
  Sepal.Length Sepal.Width Petal.Length Petal.Width
1          5.1         3.5          1.4         0.2
2          4.9         3.0          1.4         0.2
3          4.7         3.2          1.3         0.2
4          4.6         3.1          1.5         0.2
5          5.0         3.6          1.4         0.2
> 

R : Using grepl

Sometimes the need is for selecting and displaying all columns of a dataset except for 1 or 2 columns.

Identifying the columns by index no may not always be possible. If the data is correctly labelled and column names are aptly named, then using index number hinders the code readability. To overcome such scenario use the following code:

 > studentData <- data.frame(ID=paste0("Student",1:50),
+                           Math=sample(100,50),
+                           Science=sample(100,50),
+                           History=sample(100,50),
+                           Final=sample(100,50))
> studentData
          ID Math Science History Final
1   Student1   15      78      41    93
2   Student2   85      46      52    75
3   Student3   10      12      17    99.....

> studentData[,!grepl("ID",colnames(studentData))]
   Math Science History Final
1    15      78      41    93
2    85      46      52    75
3    10      12      17    99...

Happy programming!!

R: Initializing an empty list

If you are new to R, there is a thick chance that you may want to grow your dataset (dataframe) dynamically. As we all in know that dynamic memory allocation is a tricky business and in R it is more so.

In R, dynamically allocated memory in loops and function affects the performance of the system(program). A vector object’s growth in each iteration of a loop takes its own time for the loop to complete, which decreases the program speed.
The best solution to get rid of such speed issues is to predefined size of vector and fill it up with the values inside for loop, whenever possible.

For example:

>emptyData <- rep(NA, 100000)
>emptyData <- rep(1:100)
>emptyData <- rep(1:10, times=3)

Happy programming!

R : Reading data from CSV file

To read data from a csv file, you can use the use the read.table function in R:

> myData <- read.table("C:/Downloads/R_Tutorial/18MarchStudentInfo.csv", header=TRUE, sep=",")

The above command assumes your csv file has a header field at the begining of the file describing the column names. If your CSV does not contain a row describing the column names you can use it as :

> mydata <- read.table("C:/Downloads/R_Tutorial/18MarchStudentInfo.csv", header=FALSE, sep=",")

Change sep field appropriately depending on your column delimiter in your CSV

R : Beginners Guide

If you are starting with the R language and want to know the basics, here is good link which I came across.  Here you can start with understanding the basic types, assignments, functions:

http://www.johndcook.com/blog/r_language_for_programmers/

Happy Programming!!