Subsetting data

Learning Objectives

To be able to subset vectors and data frames

To be able to extract individual and multiple elements:

by index,

by name,

using comparison operations

To be able to skip and remove elements from various data structures.

R has many powerful subset operators and mastering them will allow you to easily perform complex operations on any kind of dataset.

There are six different ways we can subset any kind of object, and three different subsetting operators for the different data structures.

Let’s start with the workhorse of R: atomic vectors.

x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
x

##   a   b   c   d   e 
## 5.4 6.2 7.1 4.8 7.5

So now that we’ve created a dummy vector to play with, how do we get at its contents?

Accessing elements using their indices

To extract elements of a vector we can give their corresponding index, starting from one:

x[1]

##   a 
## 5.4

x[4]

##   d 
## 4.8

The square brackets operator is just like any other function. For atomic vectors (and matrices), it means “get me the nth element”.

We can ask for multiple elements at once:

x[c(1, 3)]

##   a   c 
## 5.4 7.1

Or slices of the vector:

x[1:4]

##   a   b   c   d 
## 5.4 6.2 7.1 4.8

the : operator just creates a sequence of numbers from the left element to the right. I.e. x[1:4] is equivalent to x[c(1,2,3,4)].

We can ask for the same element multiple times:

x[c(1,1,3)]

##   a   a   c 
## 5.4 5.4 7.1

If we ask for a number outside of the vector, R will return missing values:

x[6]

## <NA> 
##   NA

This is a vector of length one containing an NA, whose name is also NA.

If we ask for the 0th element, we get an empty vector:

x[0]

## named numeric(0)

Vector numbering in R starts at 1

In many programming languages (C and python, for example), the first element of a vector has an index of 0. In R, the first element is 1.

Skipping and removing elements

If we use a negative number as the index of a vector, R will return every element except for the one specified:

x[-2]

##   a   c   d   e 
## 5.4 7.1 4.8 7.5

We can skip multiple elements:

x[c(-1, -5)]  # or x[-c(1,5)]

##   b   c   d 
## 6.2 7.1 4.8

Tip: Order of operations

A common trip up for novices occurs when trying to skip slices of a vector. Most people first try to negate a sequence like so:
x[-1:3]
## Error in x[-1:3]: only 0's may be mixed with negative subscripts
This gives a somewhat cryptic error:

But remember the order of operations. : is really a function, so what happens is it takes its first argument as -1, and second as 3, so generates the sequence of numbers: c(-1, 0, 1, 2, 3).

The correct solution is to wrap that function call in brackets, so that the - operator applies to the results:
x[-(1:3)]
##   d   e 
## 4.8 7.5

To remove elements from a vector, we need to assign the results back into the variable:

x <- x[-4]
x

##   a   b   c   e 
## 5.4 6.2 7.1 7.5

Challenge 1

Given the following code:
x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
print(x)
##   a   b   c   d   e 
## 5.4 6.2 7.1 4.8 7.5
Come up with at least 3 different commands that will produce the following output:
##   b   c   d 
## 6.2 7.1 4.8
Compare notes with your neighbour. Did you have different strategies?

Subsetting by name

We can extract elements by using their name, instead of index:

x[c("a", "c")]

##   a   c 
## 5.4 7.1

This is usually a much more reliable way to subset objects: the position of various elements can often change when chaining together subsetting operations, but the names will always remain the same!

Unfortunately we can’t skip or remove elements so easily.

To skip (or remove) a single named element:

x[-which(names(x) == "a")]

##   b   c   d   e 
## 6.2 7.1 4.8 7.5

The which function returns the indices of all TRUE elements of its argument. Remember that expressions evaluate before being passed to functions. Let’s break this down so that its clearer what’s happening.

First this happens:

names(x) == "a"

## [1]  TRUE FALSE FALSE FALSE FALSE

The condition operator is applied to every name of the vector x. Only the first name is “a” so that element is TRUE.

which then converts this to an index:

which(names(x) == "a")

## [1] 1

Only the first element is TRUE, so which returns 1. Now that we have indices the skipping works because we have a negative index!

Skipping multiple named indices is similar, but uses a different comparison operator:

x[-which(names(x) %in% c("a", "c"))]

##   b   d   e 
## 6.2 4.8 7.5

The %in% goes through each element of its left argument, in this case the names of x, and asks, “Does this element occur in the second argument?”.

Challenge 2

Run the following code to define vector x as above:
x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
print(x)
##   a   b   c   d   e 
## 5.4 6.2 7.1 4.8 7.5
Given this vector x, what would you expect the following to do?
x[-which(names(x) == "g")]
Try out this command and see what you get. Did this match your expectation? Why did we get this result? (Tip: test out each part of the command on it’s own like we just did above - this is a useful debugging strategy)

Which of the following are true:

if there are no TRUE values passed to which, an empty vector is returned

if there are no TRUE values passed to which, an error message is shown

integer() is an empty vector

making an empty vector negative produces an “everything” vector

x[] gives the same result as x[integer()]

Tip: Non-unique names

You should be aware that it is possible for multiple elements in a vector to have the same name. (For a data frame, columns can have the same name — although R tries to avoid this — but row names must be unique.) Consider these examples:

x <- 1:3
x

## [1] 1 2 3

names(x) <- c('a', 'a', 'a')  
x

## a a a 
## 1 2 3

x['a']  # only returns first value

## a 
## 1

x[which(names(x) == 'a')]  # returns all three values

## a a a 
## 1 2 3

Tip: Getting help for operators

Remember you can search for help on operators by wrapping them in quotes: help("%in%") or ?"%in%".

So why can’t we use == like before? That’s an excellent question.

Let’s take a look at just the comparison component:

names(x) == c('a', 'c')

## Warning in names(x) == c("a", "c"): longer object length is not a multiple
## of shorter object length

## [1]  TRUE FALSE  TRUE

Obviously “c” is in the names of x, so why didn’t this work? == works slightly differently than %in%. It will compare each element of its left argument to the corresponding element of its right argument.

Here’s a mock illustration:

c("a", "b", "c", "e")  # names of x
   |    |    |    |    # The elements == is comparing
c("a", "c")

When one vector is shorter than the other, it gets recycled:

c("a", "b", "c", "e")  # names of x
   |    |    |    |    # The elements == is comparing
c("a", "c", "a", "c")

In this case R simply repeats c("a", "c") twice. If the longer vector length isn’t a multiple of the shorter vector length, then R will also print out a warning message:

names(x) == c('a', 'c', 'e')

## [1]  TRUE FALSE FALSE

This difference between == and %in% is important to remember, because it can introduce hard to find and subtle bugs!

Subsetting through other logical operations

We can also more simply subset through logical operations:

x[c(TRUE, TRUE, FALSE, FALSE)]

## a a 
## 1 2

Note that in this case, the logical vector is also recycled to the length of the vector we’re subsetting!

x[c(TRUE, FALSE)]

## a a 
## 1 3

Since comparison operators evaluate to logical vectors, we can also use them to succinctly subset vectors:

x[x > 7]

## named integer(0)

Tip: Combining logical operations

There are many situations in which you will wish to combine multiple conditions. To do so several logical operations exist in R:

| logical OR: returns TRUE, if either the left or right are TRUE.

& logical AND: returns TRUE if both the left and right are TRUE

! logical NOT: converts TRUE to FALSE and FALSE to TRUE

&& and || compare the individual elements of two vectors. Recycling rules also apply here.

Challenge 3

Given the following code:
x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
print(x)
##   a   b   c   d   e 
## 5.4 6.2 7.1 4.8 7.5
Write a subsetting command to return the values in x that are greater than 4 and less than 7.

Handling special values

At some point you will encounter functions in R which cannot handle missing, infinite, or undefined data.

There are a number of special functions you can use to filter out this data:

is.na will return all positions in a vector, matrix, or data.frame containing NA.
likewise, is.nan, and is.infinite will do the same for NaN and Inf.
is.finite will return all positions in a vector, matrix, or data.frame that do not contain NA, NaN or Inf.
na.omit will filter out all missing values from a vector

Data frames

Remember the data frames are lists underneath the hood, so similar rules apply. However they are also two dimensional objects:

[ with one argument will act the same was as for lists, where each list element corresponds to a column. The resulting object will be a data frame:

head(gapminder[3])

##        pop
## 1  8425333
## 2  9240934
## 3 10267083
## 4 11537966
## 5 13079460
## 6 14880372

Similarly, [[ will act to extract a single column:

head(gapminder[["lifeExp"]])

## [1] 28.801 30.332 31.997 34.020 36.088 38.438

And $ provides a convenient shorthand to extract columns by name:

head(gapminder$year)

## [1] 1952 1957 1962 1967 1972 1977

With two arguments, [ behaves the same way as for matrices:

gapminder[1:3,]

##       country year      pop continent lifeExp gdpPercap
## 1 Afghanistan 1952  8425333      Asia  28.801  779.4453
## 2 Afghanistan 1957  9240934      Asia  30.332  820.8530
## 3 Afghanistan 1962 10267083      Asia  31.997  853.1007

If we subset a single row, the result will be a data frame (because the elements are mixed types):

gapminder[3,]

##       country year      pop continent lifeExp gdpPercap
## 3 Afghanistan 1962 10267083      Asia  31.997  853.1007

But for a single column the result will be a vector (this can be changed with the third argument, drop = FALSE).

Challenge 4

Fix each of the following common data frame subsetting errors:

Extract observations collected for the year 1957
gapminder[gapminder$year = 1957,]
Extract all columns except 1 through to 4
gapminder[,-1:4]
Extract the rows where the life expectancy is longer the 80 years
gapminder[gapminder$lifeExp > 80]
Extract the first row, and the fourth and fifth columns (lifeExp and gdpPercap).
gapminder[1, 4, 5]
Advanced: extract rows that contain information for the years 2002 and 2007
gapminder[gapminder$year == 2002 | 2007,]

Challenge 5

Why does gapminder[1:20] return an error? How does it differ from gapminder[1:20, ]?

Create a new data.frame called gapminder_small that only contains rows 1 through 9 and 19 through 23. You can do this in one or two steps.

Challenge solutions

Solutions to challenges.

Solution to challenge 1

Given the following code:
x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
print(x)
##   a   b   c   d   e 
## 5.4 6.2 7.1 4.8 7.5
Come up with at least 3 different commands that will produce the following output:
##   b   c   d 
## 6.2 7.1 4.8
x[2:4] 
x[-c(1,5)]
x[c("b", "c", "d")]
x[c(2,3,4)]

Solution to challenge 2

Run the following code to define vector x as above:
x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
print(x)
##   a   b   c   d   e 
## 5.4 6.2 7.1 4.8 7.5
Given this vector x, what would you expect the following to do?
x[-which(names(x) == "g")]
Try out this command and see what you get. Did this match your expectation?

Why did we get this result? (Tip: test out each part of the command on it’s own like we just did above - this is a useful debugging strategy)

Which of the following are true:

if there are no TRUE values passed to “which”, an empty vector is returned

if there are no TRUE values passed to “which”, an error message is shown

integer() is an empty vector

making an empty vector negative produces an “everything” vector

x[] gives the same result as x[integer()]

Answer: A and C are correct.

The which command returns the index of every TRUE value in its input. The names(x) == "g" command didn’t return any TRUE values. Because there were no TRUE values passed to the which command, it returned an empty vector. Negating this vector with the minus sign didn’t change its meaning. Because we used this empty vector to retrieve values from x, it produced an empty numeric vector. It was a named numeric empty vector because the vector type of x is “named numeric” since we assigned names to the values (try str(x) ).

Solution to challenge 4

Fix each of the following common data frame subsetting errors:

Extract observations collected for the year 1957
# gapminder[gapminder$year = 1957,]
gapminder[gapminder$year == 1957,]
Extract all columns except 1 through to 4
# gapminder[,-1:4]
gapminder[,-c(1:4)]
Extract the rows where the life expectancy is longer the 80 years
# gapminder[gapminder$lifeExp > 80]
gapminder[gapminder$lifeExp > 80,]
Extract the first row, and the fourth and fifth columns (lifeExp and gdpPercap).
# gapminder[1, 4, 5]
gapminder[1, c(4, 5)]
Advanced: extract rows that contain information for the years 2002 and 2007
# gapminder[gapminder$year == 2002 | 2007,]
gapminder[gapminder$year == 2002 | gapminder$year == 2007,]
gapminder[gapminder$year %in% c(2002, 2007),]

Solution to challenge 5

Why does gapminder[1:20] return an error? How does it differ from gapminder[1:20, ]?

Answer: gapminder is a data.frame so needs to be subsetted on two dimensions. gapminder[1:20, ] subsets the data to give the first 20 rows and all columns.

Create a new data.frame called gapminder_small that only contains rows 1 through 9 and 19 through 23. You can do this in one or two steps.
gapminder_small <- gapminder[c(1:9, 19:23),]