Cheat Sheet R Dplyr



  • Authors: Ahmed Hasan
  • Research field: Genomics
  • Lesson topic: Data wrangling with dplyr and magrittr
  • Lesson content URL: https://github.com/UofTCoders/studyGroup/tree/gh-pages/lessons/r/dplyrmagrittr
  • Lesson video stream: coming soon!

You'll be a dplyr expert in no time! You will want to keep this dplyr cheat sheet open in a separate window to remind you about the syntax: dplyr cheat sheet Also, remember: if you need to know the variables in a data.frame called biopics you can always use. R with dplyr and tidyr cheat sheet. Whenever I used R for my data analyses, I had to write a lot of codes to manipulate my data, and sometimes the codes cannot be maintainable. Thanks to dplyr and tidyr packages I no logner need to write long and redundant codes. This blog is where I write some tricks of using dplyr and tidyr. There are lots of Venn diagrams re: SQL joins on the internet, but I wanted R examples. Those diagrams also utterly fail to show what’s really going on vis-a-vis rows AND columns. Other great places to read about joins: The dplyr vignette on Two-table verbs. The Relational data chapter in R for Data Science (Wickham and Grolemund 2016. With dplyr and tidyr Cheat Sheet. Converts data to tbl class. Tbl’s are easier to examine than data frames. R displays only the data that fits onscreen: dplyr::glimpse(iris) Information dense summary of tbl data. Utils::View(iris) View data set in spreadsheet-like display (note capital V).

Tibble cheat sheet

Why dplyr and magrittr?

dplyr is a package for data wrangling and manipulation developed primarily by Hadley Wickham as part of his ‘tidyverse’ group of packages. It provides a powerful suite of functions that operate specifically on data frame objects, allowing for easy subsetting, filtering, sampling, summarising, and more.

The overall philosophy binding dplyr together is rather simple - save for a select few exceptions, nearly every function in the package follows the convention of taking in a data frame as input and returning a modified data frame as output. Coupled with the pipe operator from the magrittr package, chaining dplyr functions together makes data frame manipulation an absolute breeze.

Meet the pipe

The R pipe, or %>% (Ctrl/Cmd + Shift + M in RStudio) initially began life outside of dplyr, finding its R beginnings in the magrittr package instead. Its creator, Stefan Milton Bache, was inspired by the pipe operator in F# (|>) in addition to its Unix shell equivalent (|).

Much like in those examples, the idea behind the magrittr pipe is also relatively simple: it essentially stands for ‘evaluate the left hand side, and feed the result as input to the function on the right hand side’. For functions that take multiple arguments, the pipe will feed it to the first one by default. Like so:

Of course, sometimes we want our LHS to be fed into a different argument that just so happens to not be the first one. In these cases, we use . to tell R where to send the LHS instead:

Given the dplyr concept of each function taking in a data frame and returning a modified version, it made a lot of sense to integrate the pipe into the dplyr workflow. This way, a given data frame would be ‘piped’ through a series of functions one after the other in order to obtain a specific desired output.

To summarise:

Tidy data

One final thing to note before we delve into dplyr itself is the concept of tidy data. This basically describes a data set where each column is a variable or feature of the data, and every row is a single observation. Fisher and Anderson’s iris, which we’ll be using in today’s lesson, is a good example of a tidy dataset.

The iris dataset gives measurements of sepal length/width and petal length/width for 50 flower samples from each of Iris setosa, Iris versicolor, and Iris virginica. Notice how each row corresponds to measurements from a single flower sample, and each column represents a specific feature of that flower.

dplyr is optimized for use with tidy data. However, real world data are rarely in a tidy format from the get-go. For this reason, Hadley and co. have also developed the tidyr package, which provides useful functions for reshaping data. We won’t be covering tidyr in today’s lesson, but a wealth of tutorials are available online.

Alright, let’s dive right into dplyr!

filter - subsetting rows

filter is the first dplyr verb we’ll be looking at. At its core, and much like all dplyr functions, filter will take an input data frame as its first argument. Following that, we can define a set of conditions that we want to filter the rows of our data frame by.

To begin with, let’s look at all the flowers that have a sepal length greater than or equal to 7.0 cm.

We can simply tack on any further conditions we might want to filter by as separate arguments:

select - subsetting columns

select operates in a similar fashion to filter, but allows for subsetting columns instead. Since select is primarily given column names (which can be thought of as text strings) for input, it also allows for easy globbing of multiple columns at once.

A quick aside - we are also going to convert iris to a tibble from this point onwards. Tibbles are another feature of dplyr; they’re essentially data frame objects with a number of under-the-hood improvements, but those are somewhat beyond the scope of this lesson. What’s relevant to us here is that they prevent R from printing out the entire data frame when it’s called, and are also previewed in an informative format that tells us about the object types stored within each data frame column and more. (More details here if you’re interested.)

We can select multiple columns by simply providing their names as further arguments. dplyr also allows use of the : operator for consecutive columns.

The - operator can be used to exclude a column:

Finally, select features the helper functions contains, starts_with, and ends_with for globbing column names. Note that while we generally treat column names as objects in the body of dplyr functions, these helper functions require strings as input.

Here comes the pipe

But what if we want to select certain columns, and then filter the resultant rows by a certain condition? (Or vice versa)

We could certainly nest functions, like so:

But this is messy enough as is, and that’s just with a single instance of nesting. Imagine we had three or four nested functions! We’d fast approach having more parentheses than actual code.

An alternative is using intermediate objects:

But that can be a drain on memory if we’re working with large amounts of data; not to mention that the number of intermediates would only grow as our analysis goes on, increasingly cluttering our workspace.

However, recall that every dplyr function both takes in a data frame object as its first argument, and returns one as output. With this being the case, the pipe allows us to easily chain dplyr functions together.

Notice how that code is both easy to conceptualize and understand after the fact, and that neither of the dplyr functions have the data frame actually listed as part of their arguments. The overall code block effectively reads like:

In using the pipe with dplyr functions, we can write up simple ‘recipes’ that manipulate data frame objects in a quick and straightforward manner.

Of course, there’s more to dplyr than just subsetting -

arrange - sort your data

Dplyr cheat sheet in r

Much like the name implies, arrange allows us to sort data. Not much to it:

R studio dplyr cheat sheet

Sorting in descending order is not much harder:

Finally, feeding two (or more) arguments to arrange will be interpreted as ‘sort by first column and then by second column’.

mutate - make new variables

mutate is arguably where dplyr really comes to life. So far, we’ve just been playing around with existing data, subsetting it and arranging it how we please. Which is certainly very useful! But mutate allows us to create entirely new variables (i.e. columns) in our data frame object out of operations performed on the data frame itself.

mutate employs window functions in order to do this. Unlike an aggregation function (such as mean) which takes in $n$ values and returns one value, a window function will take in $n$ elements as input and return $n$ elements as output. In the case of mutate, this means performing an operation on every single row of a data frame and returning the resultant values in a new column.

Let’s use mutate to calculate the approximate area of sepals and petals in our dataset.

We could also get the differences between sepal length and petal length for each flower:

Gone are the days of clumsy loops over data frame rows!

Of course, if we are performing some kind of complex operation that mutate is behaving strangely with, we can use rowwise to evaluate the operation one row at a time.

Finally, it is sometimes useful to drop all other columns once we’ve performed a mutate operation. Instead of having to use select after the fact to keep the columns we need, transmute is a version of mutate that only keeps the variables you create.

Notice how the Species column is also gone in the example above.

summarise - summarise your data

Where mutate is all about window functions, summarise returns us to the familiar world of aggregate functions, outputting a single value for however many values are fed as input. These should generally be quite familiar:

The important thing to note is that in using summarise, we are obtaining our summary values within a data frame structure still. Using something like mean outside of summarise would simply return a float.

group_by - group your data

group_by is primarily a helper function itself, but it provides an immense amount of added utility to mutate and summarise. It does so by adding grouping information in an under the hood fashion. While this does not necessarily do anything explicit to the data, both summarise and mutate behave quite differently when presented with a grouped data frame.

For instance, let’s redo the summarise function we did above, but now in a grouped manner:

We can also quickly obtain the total number of records for each species:

Or the number of distinct values in a given column:

group_by also works with mutate for grouped operations. For instance, if we want to examine the ratio of each flower’s sepal length from the mean for that species, we could try something like this:

Cheat

In this example, mean(Sepal.Length) is computed for each species, instead of from the entire dataset.

Finally, grouping information can be removed with the ungroup function. This is useful when performing one grouped operation and then performing a second in the same dplyr chain but off of a different column grouping.

To summarise

  • select allows you to subset by column
  • filter allows you to subset rows that meet one or more conditions
  • arrange allows you to sort your data
  • group_by allows you to group your data for grouped downstream operations
  • mutate allows you to compute new variables from your data
  • summarise allows you to apply summary functions on your data

The official Data Wrangling with R cheat sheet is a stellar reference for working with all these functions and more. Chain them together however you please using the pipe, and watch your adventures wrangling data frames become substantially more straightforward!

Dplyr

Some magrittr tricks

While we may have covered the basics of dplyr and how the pipe synergizes with it, it’s still worth going over some of the unique tricks magrittr itself offers.

The tee operator - %T>%

Another operator that derives its inspiration from the Unix shell equivalent, the tee operator will still pass on LHS to RHS, but returns LHS to stdout instead of RHS.

In the context of a dplyr chain of commands, this is especially useful when we are saving the final result of said commands back to a data frame, but also have some kind of ancillary function tacked on at the very end (i.e. plot()) that we don’t want saved to the object.

Here, the tee operator allows us to view a plot of our data, but without the output of plot ending up saved to the sepalsonly object. plot is therefore still part of our command chain, but not in an obtrusive way.

Tibble Cheat Sheet

The compound assignment operator - %<>%

The compound assignment operator also comes in handy when we’re performing a series of dplyr commands on a data frame and saving the results back to it.

Tidyr Cheat Sheet

Where we might do:

Dplyr Cheat Sheet R

The compound assignment operator makes for a useful bit of shorthand: