Introduction to R

• About me
• What is R?
• Installation and IDE
• Basic Programming Tips in R
  • Fundamentals
  • Type and Class
  • Control Strucure
  • Function
• I/O
• Visualization
• Final Remarks

• A general programming language that is…
  • interpreted
  • designed for statistics
  • with a powerful and active academic community
  • able to run in both interactive and batch mode
  • applicaple to all modern operating systems
  • FREE! (open source)
• Some useRs consider it general due to its high flexibility to develop applications for integration with other infrastructures…
  • ODBC with in-line SQL
  • high-level functions to deal with curl, json, XML…
  • Hadoop integration
  • And many others

• R is a GNU project based on S language, developed by AT&T (1976)
• S was then sold (so there is S-PLUS)
• Okay. That's it.

• Among all over statistical packages such as…
  • SAS Extremely expensive with surprisingly little capability.
  • SPSS Not programmatically friendly. Not free.
  • Stata
    • Powerful for economics and epidemiology
    • But with limited flexibility and scalability
    • Not free
  • Matlab Too fat. Not free.
  • GAUSS
    • General Apparatus for Users to Suicide Systematically (I mean it.)

• For Windows
  • Download and execute the binary
• For Mac
  • Download and execute the binary
• For Linux, depending on the distribution…
  • sudo apt-get install r-base-core
  • sudo yum install R-core (recommend using EPEL)
  • Compile it on your own (don't look at me)
• R on the cloud
  • StatAce: A startup that allows you to conduct analysis on the cloud

• Choose 64-bit version whenever you can
  • R heavily depends on RAM
  • Some advanced applications only support 64-bit platform, e.g., RHadoop
• Prefer Linux/Mac over Windows
  • There is a troublesome Unicode encoding issue in Windows not yet solved
    • if you are to analyze data coming from a variety of sources the world over, you WILL be in trouble
  • Some useful packages do not have pre-compiled version of or simply cannot run in Windows, e,g, bigmemory, rPython

• http://www.rstudio.com/
• For all platforms
  • RStudio-Server for Linux server
• Well integrated with Git
• Markdown language interface customized for R
  • Slides for this lecture are purely writen in RStudio
• Module used to develop web app of R based on interactive graphing
• And yes, it's FREE!

• http://ipython.org/
• Generally a Pyhton IDE
• But you can easily execute R codes
  • try the magic function: %load_ext rmagic

Your role matters!

• In general, R is an OO language
• As a software developer:
  • You write your application in OO style
  • You are possibly a contributor to CRAN
• As a data analyst:
  • You hack into data with procedural or functional programming
  • You only need to know basic concept of OO
• After all, it's up to you!
  • I personally never write OO-style codes for data analysis purpose

sessionInfo() # type ?sessionInfo for document help

• Where is my working directory?

getwd()

• WIN users be careful: backslash path must be escaped

setwd('C:\\Dropbox')
# or simply:
setwd('C:/Dropbox')

• Save environment and quit

save.image()  # default to destined file '.RData'
q()

• Configure startup script

edit(file='C:/Program Files/R/R-3.0.0/etc/Rprofile.site')

• Get/set search paths for libraries (packages)

.libPaths()

[1] "C:/Users/everdark/Documents/R/win-library/3.0"
[2] "C:/Program Files/R/R-3.0.2/library"           

(.libPaths(c(.libPaths(), getwd())))

[1] "C:/Users/everdark/Documents/R/win-library/3.0"
[2] "C:/Program Files/R/R-3.0.2/library"           
[3] "C:/Dropbox/R/Course"                          

• R is case-senstive
• Comment character is # only (no multi-line comment char)
• In general, all objects are put into ram
  • Yes, there are exceptions
  • use object.size to check memory usage of object(s)
• No line breaker required
  • Use semi-colon (;) to put multiple statements on one line
• Use gc for garbage collection
  • In general R will handle this automatically
• Use library or require to load external packages
• Use install.packages to install CRAN packages
• Use as function for type convertion

x <- 999L; y = 0 # append 'L' to force integer type
x                # the same as print(x)

[1] 999

print(y)

[1] 0

• Use '<-' for assignment
  • The Google R style guide indeed prohibits '=' as assignment
  • Better readability
• Use '=' for parameter association (in function call)
  • Avoid unpredictable result

• Parameter assignment is local-scoped when using '=', but not '<-'

power2 <- function(z) z^2  
power2(z = 3); z           # Error: object 'z' not found
power2(z <- 4); z          # z is assigned numeric 4

• What's more, due to lazy evaluation…

z <- 'whatever remains'
tellTheTruth <- function(z) return(TRUE)
tellTheTruth(z <- 'I won\'t appear.')

[1] TRUE

z

[1] "whatever remains"

• Assign value globally within local function (What's the point?)

changeX <- function(x) {
    x <- x + 100
    gvar <<- 'I am a GLOBAL variable.'
    x
}
x <- 0; changeX(x=x)

[1] 100

x

[1] 0

gvar

[1] "I am a GLOBAL variable."

• Types (of Vectors)
• Matrices
• Lists
• Data Frames
• Factors

• Oops! There is no such thing called scalar in R
• The fundemantal building block is vector
  • 1L is an integer vector of length 1
  • Matrices are vectors
  • Arrays (multidimensional Matrices) are vectors
• Index is 1-based

V <- c(10L,20,30)   # c for concatenate
typeof(V)           # notice for the implicit casting

[1] "double"

V[2]                # try ?'['

[1] 20

• Types include…
  • logical TRUE, FALSE, T, F
  • integer, double, complex
  • character
  • list
  • closure, special, builtin (these are functions)
  • NULL
  • …
• Use typeof or mode to check the variable type
  • Difference? See ?mode

• Every object has a mode(type) and a class
  • Mode suggests how it is stored in memory
  • Class suggest its abstract type (utilized in OOP)

x <- integer()
c(typeof(x), class(x))           # try also storage.mode(x)

[1] "integer" "integer"

y <- data.frame()
c(typeof(y), class(y))

[1] "list"       "data.frame"

• Elements of a (atomic) vector must be all of the same type

(mixed <- c('123', 123)) # implicit casting

[1] "123" "123"

• No insertion or deletion

(mixed <- c(mixed, 4000)[2:4])   # V points to a new vector

[1] "123"  "4000" NA    

• Can be named

names(mixed) <- c('1st', '2nd', '3rd'); mixed

   1st    2nd    3rd 
 "123" "4000"     NA 

• Operation that requires equal-lengthed vectors generally cause R to recycle the short one to necessary length

c(1,2) * c(100,100,100) # '*' is element-wise, see ?'*'

Warning: longer object length is not a multiple of shorter object length

[1] 100 200 100

• Apply to Matrices as well (Matrices are vectors)
  • Matrices are expanded to vectors by column

• Implemented by %*%
• No recycling: non-comfortables will generate ERROR

c(1,2,3) %*% c(100,100,100) # inner product

     [,1]
[1,]  600

c(1,2,3) %o% c(100,100,100) # outer product

     [,1] [,2] [,3]
[1,]  100  100  100
[2,]  200  200  200
[3,]  300  300  300

• Vector indexing is done by vector

V <- c(10,20,30,40)
V[c(2,4)]

[1] 20 40

V[2:length(V)]    # see ?seq for a generalized ':' function

[1] 20 30 40

bool_index <- c(TRUE,FALSE,FALSE,FALSE)
V[bool_index]

[1] 10

V1 <- c(10,20,30,40)
(V2 <- V1[c(rep(1,2),2:4)])  # take the 1st element twice

[1] 10 10 20 30 40

V2[-c(2,4)]                  # negative indexing

[1] 10 20 40

• Application?

V2[V2 > mean(V2)]            # indeed, '>'(V2, mean(V2))

[1] 30 40

• Can be done by names (if any)

names(V1) <- c('one', 'two', 'three', 'four')
str(V1) # a very useful function to check structure of an object

 Named num [1:4] 10 20 30 40
 - attr(*, "names")= chr [1:4] "one" "two" "three" "four"

V1[c('two', 'three')]

  two three 
   20    30 

V1[grep('^t', names(V1))]   # see also ?grepl and ?regex

  two three 
   20    30 

• A new class: vectors with additional attribute 'dimension'

(M <- matrix(c(1,2,3,4), nrow=2, byrow=FALSE))

     [,1] [,2]
[1,]    1    3
[2,]    2    4

str(M)           # simply a numeric vector with 2 dimensions

 num [1:2, 1:2] 1 2 3 4

dim(M)           # try length(M)

[1] 2 2

• Can be named on either rows or columns

colnames(M) <- c('c1', 'c2')
rownames(M) <- c('r1', 'r2')

• Indexed

identical(M[,2], M[1:2,'c2']) # test exact equality

[1] TRUE

• Editted

M[1,1] <- 0; M

   c1 c2
r1  0  3
r2  2  4

• Be careful! By default R will drop dimension when possible

dim(M[1,])

NULL

str(M[1,])             # rowname attribute is dropped, too

 Named num [1:2] 0 3
 - attr(*, "names")= chr [1:2] "c1" "c2"

str(M[1,,drop=FALSE])  # you'll need this one day

 num [1, 1:2] 0 3
 - attr(*, "dimnames")=List of 2
  ..$ : chr "r1"
  ..$ : chr [1:2] "c1" "c2"

• Append by cbind or rbind

M1 <- matrix(1:4, 2, 2)       # use positional argument
M2 <- matrix(0, 2, 2)         # recycle occurs
cbind(M1, M2)

     [,1] [,2] [,3] [,4]
[1,]    1    3    0    0
[2,]    2    4    0    0

unique.matrix(rbind(M1, M2))  # distinct by row

     [,1] [,2]
[1,]    1    3
[2,]    2    4
[3,]    0    0

There are many implementations of sparse matrix in R. The base package Matrix provide a fairly flexible class dgCMatrix (which, of course, extends class matrix).

• A document example

i <- c(1,3:8); j <- c(2,9,6:10); x <- 7 * (1:7)
(A <- Matrix::sparseMatrix(i, j, x = x))

8 x 10 sparse Matrix of class "dgCMatrix"

[1,] . 7 . . .  .  .  .  .  .
[2,] . . . . .  .  .  .  .  .
[3,] . . . . .  .  .  . 14  .
[4,] . . . . . 21  .  .  .  .
[5,] . . . . .  . 28  .  .  .
[6,] . . . . .  .  . 35  .  .
[7,] . . . . .  .  .  . 42  .
[8,] . . . . .  .  .  .  . 49

• Similar to dict in Python
• Still vectors, but recursive (contrary to atomic)
  • Can contain elements of different types

mylist <- list(name='Kyle', gender='male', 18)
str(mylist)         # notice that tag name is not required

List of 3
 $ name  : chr "Kyle"
 $ gender: chr "male"
 $       : num 18

• Nesting is possible
  • Notice that this is why the name recursive

nested <- list(old=mylist, new='new')

• Atomic vectors can't be broken down, and can't have multiple types in its elements
  • numeric, character, logical, …

(atomic <- c(c(1,2,3), 4, 5))              # no nesting

[1] 1 2 3 4 5

• Not the case for recursive vectors

str(recursive <- list(list('a',2), TRUE))

List of 2
 $ :List of 2
  ..$ : chr "a"
  ..$ : num 2
 $ : logi TRUE

• List indexing: take individual element

c(
    mylist$name,      # use '$' operator
    mylist[['name']], # use '[[' with tag name
    mylist[[1]]       # use '[[' with numeric index
    )

[1] "Kyle" "Kyle" "Kyle"

• List indexing: slicing (always return a list)

mylist[1:2]           # try also mylist[c('name', 'gender')]

$name
[1] "Kyle"

$gender
[1] "male"

• Insertion and deletion made possible!

mylist$newtag <- 'something new'
mylist[[3]] <- NULL
mylist[['gender']] <- 'female'
mylist

$name
[1] "Kyle"

$gender
[1] "female"

$newtag
[1] "something new"

• Use unlist to convert recursive vectors into atomic one

• Join
  • use merge for inner/outter join
• Append
  • use rbind, cbind (these functions are polymorphic)
• Aggregate (group-by operation)
  • use aggregate, sweep
• Pivot
  • use reshape
• Partition
  • use split.data.frame
• Order-by
  • use order

Package data.table provides a new class 'data.table' which extends 'data.frame' but with more efficient computing capability for fairly large dataset.

• Specifically, it provides:
  • Indexed data frame with binary search implemented
  • by-reference data manipulation
  • fast group-by operator
• See quick introduction for more info

• To model nominal or categorical data
  • Yes, they are vectors
  • Seemingly character, but numeric

(fchar <- factor(sample(letters[1:5], 5, replace=TRUE)))

[1] d b b c c
Levels: b c d

c(class(fchar), typeof(fchar))

[1] "factor"  "integer"

str(fchar)

 Factor w/ 3 levels "b","c","d": 3 1 1 2 2

x <- factor(c(1, 10, 100, 10))
levels(x)

[1] "1"   "10"  "100"

as.character(x)

[1] "1"   "10"  "100" "10" 

as.numeric(x)

[1] 1 2 3 2

grep('00', x)    # grep use implicit casting

[1] 3

• Not exist, has no mode (or mode of its own)
• Not to confused with NA, which represents missing values

c(length(NA), length(NULL))

[1] 1 0

• Not to confused with NaN, which represents not-a-number

0/0

[1] NaN

• if-else
• repeat
• while
• for
• try

• Syntax

if (expression_A) {
    code_block_A
} else if (expression_B) {
    code_block_B
} else {
    code_block_C
}

• Vectorized version: ifelse

x <- c(1, 2, 3)
ifelse(x > mean(x), 'Yes', 'NO')

[1] "NO"  "NO"  "Yes"

• Be aware of the difference between:
  • Element-wise AND/OR: & and |
  • Scalar AND/OR: && and ||

x <- c(1, 0, 1, 1)
y <- c(1, 0, 0, 0)
x & y             # return Boolean vector

[1]  TRUE FALSE FALSE FALSE

x && y            # only the FIRST element is tested

[1] TRUE

• Reminder: non-zero numerics are casted as TRUE
  • if (1) print(TRUE) is valid (not for Java you know…)

i <- 1
repeat {
    i <- i + 1
    if (i > 10) break
}
i

[1] 11

• Simply a while (TRUE) loop
• Escape keywords:
  • next to skip the current iteration
  • break to break the entire loop
• Not required for indent but recommended
• {} is required for multi-line block

i <- 1
while (i < 10) {
    i <- i + 1
}
i

[1] 10

• Check the condition at the begining of each iteration
• There is no 'until' loop in R

• Iterate over numeric/character vector

for (i in 1:10) print(i)                 # result not printed
for (c in letters) print(c)

• Iterate over a list of objects: use get

A <- c(1, 2, 3)
B <- c(100, 200, 300)
for (obj in c('A', 'B')) print(mean(get(obj)))

[1] 2
[1] 200

• Iterate over a list of objects: put object into a list

for (obj in list(A,B)) print(mean(obj))  # result not printed

Nothing more than a specific typed object, but deserve its own section cause it is the core of R programming.

Or skip to next section

doSomething <- function(x) {
    x    # this is the same as return(x)
}

• function itself is a function object to create function
  • Actually, { is a function, too
    • Try ?'{'
• Formal name: closure
  • consisting of arguments, body, and its environment (scope)

environment(doSomething)

<environment: R_GlobalEnv>

• Use ls to list all variables in global (default) scope

• return is not require
  • the function will by default return the last object in body code, and only the last one
    • use list to return multiple objects
  • use return for a conditional break
  • or use stop to break the function with error message
  • and use warning to generate warning message

justInteger <- function(x) 
    if ( !is.integer(x) ) stop('please give me integer')
justInteger(1L)
justInteger('1')

Error: please give me integer

• It is possilbe (and sometimes desirable) to write a function within a function
  • This results in scope hierarchy
  • Inner-most scope have highest priority
• Functions have no side effect
  • Only local copies will be changed
  • You can NOT change a variable in-place
  • A reminder: there are no pointer things in base R
    • variables are copied by assignment
    • but with lazy evaluation

• Try ?'names<-' and mode(get('names<-'))

V1 <- c(10,20,30,40)
V1 <- 'names<-'(V1, value=c('one', 'two', 'three', 'four'))
V1

  one   two three  four 
   10    20    30    40 

• There are something called replacement functions in R
• Remember this?
  • V1[1] <- 11 will replace the first element of V1 with numeric 11
  • Indeed: try ?'[<-'
  • Also: '$<-' for list operation

• Highly readable
• Ready for parallelization
• No side effect (compared to explicit loop)
• Useful members:
  • apply
    • Apply a function to each row/column of a matrix
  • lapply
    • Apply a function to each element of a list
    • Use sapply for a vector version
  • mapply
    • Apply a function with each parameter given in a vector
  • And many others

• Actually, all control-flow constructs are formed by functions

c(mode(get('if')), mode(get('for')))

[1] "function" "function"

• And all operators are functions, too

c(mode(get('+')), mode(get('&&')), mode(get('<-')))

[1] "function" "function" "function"

• Everything is an object, and many of them functions

mode(get('{'))

[1] "function"

Now we shall go back to the section for error handling.

A function applied to a vector is actually applied to each element individually (i.e., A map function)

• 'Apply' family functions are NOT vectorized
• Most of the built-in operators are vectorized
• The principle for writing loops in R:
  • NOT to write loops in R (whenever possible)
  • Use vectorization first
• But why?
  • R interpreter is slow
  • Vectorization is implemented in C or Fortran (they're faster)
  • Side effects

• Looping add in Python

import time
mil = range(1, 1000001)
start_time = time.time()
mil = [m + 100 for m in mil]
print time.time() - start_time, 'seconds'

0.101000070572 seconds

• Looping add in R

mil <- c(1:1000000)
system.time( # explicit loop with side effect
    for ( i in seq_len(length(mil)) ) mil[i] <- mil[i] + 100
    )

   user  system elapsed 
   3.12    0.00    3.13 

• Looping add in R (one more try!)

mil <- c(1:1000000)
system.time( # high-level loop wraper without side effect
    mil <- sapply(mil, function(x) x + 100)
    )

   user  system elapsed 
   3.89    0.03    3.93 

• Vectorized add in R

mil <- c(1:10000000) # one more zero
system.time(         # What the hell are you two just doing?
    mil <- mil + 100
    )

   user  system elapsed 
   0.15    0.00    0.16 

Here comes the data.

• Commonly used functions include…(See ?files for more)
  • dir: list files under the curent working directory
  • file.info: examine information of a given file
  • file.exist: check the existence of a file
• Or simply use system to execute your system commmand
  • stdout will be redirected to R console
  • system('ls') is conceptually the same as dir
  • Limitation: pipes not allowed
• Use shell for more flexible shell command

dir()[grep('\\.html$', dir())]
# is the same as
shell('ls | grep "\\.html$"', intern=TRUE)

• Read the source line-by-line into character vector
  • Source could be an external file or a connection
  • Connection can be established purely in R
  • Write to a file via writeLines

temp_file <- file()
cat(
'this is the first line
and this is the second line
this is the end of file
', file=temp_file)   # be ware of the quoting position
readLines(temp_file) # connection will be auto-closed

[1] "this is the first line"      "and this is the second line"
[3] "this is the end of file"    

• ?connection
• Connection in R document is used to describe the mechanism of I/O operation.
• There are lots of different kind of connection:
  • a file connection established by file
  • a compressed file connection by gzfile
    • such taht you don't need to manually uncompress the data before analysis
  • a web page connection by url
    • analyze online content with other packages such as XML, rjson
  • many others

• More configurable than readLines
  • (Optionally) Use pre-defined schema to load and convert data type
  • Can parse structurally delimited raw data
• The main function to establish high-level load function in R

temp_file <- file()
cat('1,,three', file=temp_file)
scan(temp_file, what=list(col1=1L,col2=1L,col=''), sep=',')

$col1
[1] 1

$col2
[1] NA

$col
[1] "three"

• See ?read.table
  • Many variants exist: read.csv, read.fwf, …
• Some parameter suggestions…
  • use nrows=100 to read the initial 100 rows for testing purpose
  • use comment.char='' to disable comment char if you are dealing with messy string data to prevent from long string crash
  • use stringsAsFactors=FALSE if you don't like characters to be converted into factors
  • use colClasses to pre-defined data schema for speedup
  • use fill=TRUE if your data is unbalanced (have missing columns for some rows)
• The returned object is data.frame

There are a bunch of ways to do data visualization in R, with either base environment or leveraing other add-on pacakges and even other languages– for example, JavaScript.

• Do visualiztion in R is kinda high-level stuff
  • We have plot, barplot, boxplot, matplot, …
    • plot is a generic function doing X-Y ploting
    • barplot is used to generate barplot (what else?)
    • boxplot for boxplot
    • matplot for multi-line ploting
    • and still others, e.g., rgl::plot3d
  • These are mainly built-in graphing functions
  • Many advanced add-on packages available

• Let's generate some sample data to plot!

case <- iris[,3:5]
colnames(case) = gsub('\\.', '', colnames(case))
case$Name <- paste('N', 
                   round(runif(nrow(case)),3)*1000, 
                   sep='')
head(case)

  PetalLength PetalWidth Species Name
1         1.4        0.2  setosa N904
2         1.4        0.2  setosa N503
3         1.3        0.2  setosa N133
4         1.5        0.2  setosa N858
5         1.4        0.2  setosa N659
6         1.7        0.4  setosa N741

• Possible solutions:
  1. Do data analysis in R, render graph by JS
     • rcharts, d3network, googleVis, …
     • These packages provide high-level interface to graphing libraries outside R
  2. Do data analysis and graphing in R, add interactivity by JS
     • gridSVG, SVGAnnotation, …
     • These packages provide high-level interface to JavaScript functionality
  3. Use web app devtool shiny (a RStudio product)
• Either case, automation done purely in R environment is possible
  • You may need some JS knowledge

library(rCharts)
nvd3 <- nvd3Plot(
    PetalLength ~ PetalWidth, data=case, type='scatterChart', 
    group='Species', 
    xAxis=list(axisLabel='PetalWidth'), 
    yAxis=list(axisLabel='PetalLength'),
    chart=list(showDistX=TRUE, showDistY=TRUE)
); nvd3

see my RPubs article for the source code.

• This time we create the plot by R device

library(ggplot2)
library(gridSVG)
AES <- aes(x=PetalLength, 
           y=PetalWidth, 
           group=Species, 
           color=Species)
gg <- ggplot(case, AES) + 
    geom_point(size=3) + 
    theme(legend.position='top')
ggsvg <- grid.export("ggplot_scatter.html", 
                     xmldecl=NULL, addClasses=TRUE)
file.show('ggplot_scatter.html')

Zoom out to check scalability!

• On-disk computing
  • ff, ffbase, sqldf
• Parallel computing
  • parallel for embarrassingly parallel problem
  • many others (including GPU computing)
• Machine learning
  • randomForest, arules, e1071, …
• Database interface
  • RODBC
• Hadoop
  • rmr2, rhdfs, rhbase, …
• See CRAN task views for more info

• Chang (2013), R Graphics Cookbook
• Matloff (2011), The Art of R Programming
• Murray (2013), Interactive Data Visualization for the Web