# Tidyverse {#dplyr}

## Learning Objectives

The `dplyr` ecosystem is composed of three main components:

- The pipe operator `%>%`, previously discussed
- The five verbs of `dplyr`:
  - `filter()`,
  - `arrange()`,
  - `select()`,
  - `mutate()`,
  - `summarise()`, and
- `group_by()`, previously discussed

## Split-Apply-Combine with the dplyr Package {#sec:dplyr}

_Split-apply-combine_ refers to a series of actions that are often repeated in data analysis and statistics. A data-set is

- `Split` into sub-groups defined by a categorical (aka factor) variable. A function is then
- `Applied` to each individual sub-set, and the results are
- `Combined` into a _new_ data-set or _added_ onto the original data-set.

There are _many_ ways to perform _split-apply-combine_ operations in R. Actually, we already saw this in action on the first day of the workshop (\@ref(PG)).

```{r}
PlantGrowth %>%
  group_by(group) %>%
  summarise(avg = mean(weight),
            stdev = sd(weight))
```

The apply family of functions are powerful are useful `base` package functions. Typical `base` package functions include `apply`, `tapply()`, `sapply()`, `lapply()`, `mapply()`, `by()`, and `aggregrate()`. However, they proved difficult to master for new-comers since the class for the input and output of each function was different, and not obvious. Recall, one of the biggest problems you'll encounter is having data in the wrong class! There has been a major effort within the R community to develop easier tools for performing these tasks. The `dplyr` package has now come to dominate _split-apply-combine_ tasks, largely because the commands are intuitive and syntactically uniform. The many packages which have moved R into this direction are called the _tidyverse_.

The `dplyr` package takes the analogy of the program language closer to spoken language by referring to the data frames as nouns and the actions performed as verbs.

There are three main components to understanding `dplyr`. ^[`dplyr` is the data-frame centric cousin of an earlier package called `plyr`. The name `plyr` brings to mind the word _plyer_, a hand-held tool used to hold, compress and transform all variety of materials.]

- The pipe operator, as discussed previously
- The five verbs plus the helper functions
- An adverb


## The Five Verbs

There are five _verbs_ which we can use to act upon our noun:

Table: (\#tab:dplyrverbs) The five `dplyr` verbs.

| Verb                        |  Operates on | Description                                             | Section                     |
|-----------------------------|--------------|---------------------------------------------------------|-----------------------------|
| `filter()`                  | Observations | Filter observations given specific criteria.            | Section \@ref(dplyrfilter)  |
| `arrange()`                 | Observations | Rearrange observations given specific criteria.         | Section \@ref(dplyrarrange) |
| `select()`                  | Variables    | Select variables meeting specific criteria.             | Section \@ref(dplyrselect)  |
| `mutate()`                  | Variables    | Apply transformation functions on selected variables.   | Section \@ref(dplyrmutate)  |
| `summarise()`               | Variables    | Apply aggregation functions on selected variables.      | Section \@ref(dplyrmutate)  |


Let's take a look at these functions in the context of operations we already understand

### filter() {#dplyrfilter}

Recall that we can use `filter()` or `[]` to filter a data frame.

```{r eval = FALSE}
diamonds %>% 
  filter(clarity == "VVS1")

# is the same as
diamonds[diamonds$clarity == "VVS1", ]

```

The above functions are equivalent. 

The advantage is that it's easy to combine many logical expressions with a comma:

```{r eval = FALSE, tidy = FALSE}

diamonds %>% 
  filter(clarity == "VVS1", price < 1000)

```

### arrange() {#dplyrarrange}

We can imagine various definitons of _top_ values, such as:

- Top values defined by a certain cut-off, i.e. values above 35 (use a logical expression with the value)
- Top values according to a quantile, i.e. frac of the total (use a logical expression and define the cutoff dynamicall)
- Top n number of values, i.e. highest 5 values (`arrange()` and take the `head()`, or use `top_n()`)

We discussed the first two cases already. Here, we'll use the `arrange()` function: ^[Notice the use of the third possible assign operator here, `->`, which allows us to read `dplyr` commands like grammatically correct sentences: subject \%>\% verb -> object.]

```{r tidy = FALSE}
diamonds %>%
  top_n(20, price) %>% 
  select(cut) -> top20cut
```

Alternatively, we can do this more explicitly:

```{r tidy = FALSE}
diamonds %>%
    arrange(desc(price)) %>%
    select(cut) %>%
    .[1:20,] -> top20cut
```

Some alternatives for the last line in the above commands:

```
slice(1:20) -> top20cut
```

\dots{}or\dots{}

```
head(20)  -> top20cut
```

The result is a one-column data frame. Remember, `dplyr` is very data frame-centric.

```{r}
glimpse(top20cut)
```

### select() {#dplyrselect}

`select()` is used for choosing specific columns.

```{r tidy = FALSE, eval = FALSE}

diamonds %>%
    filter(clarity == "VVS1", price < 1000) %>% 
    select(cut, carat, price)

```


```{r}
# recall indexing:
foo_df[1:2] # first two columns
foo_df[-3] # all except the 3rd
```

This is all fine and good, but we also have a number of helper functions that we can use to access columns.

Table: (\#tab:dplyrhelper) The `dplyr` helper functions.

| Helper function          | Description                                      |
|--------------------------|--------------------------------------------------|
| `starts_with(x)`         | Names starts with x.                             |
| `ends_with(x)`           | Names ends in x.                                 |
| `contains(x)`            | Selects variable names containing x.             |
| `matches(x)`             | Selects all variable names matching the regex x. |
| `num_range("x", 1:5)`    | Selects x1 to x5.                                |
| `one_of("x", "y", "z")`  | Selects variables in a character vector.         |
| `everything()`           | Selects all variables.                           |


```{r tidy = FALSE, eval = FALSE}

diamonds %>%
    select(starts_with("c"))

```

Remember, we can also use the `-` notation (like we did in `[]`) to _not_ select specific columns. ^[The `-` notation is like how we use `!=` in logical expressions in in `[]` to _not_ select specific observations.]

```{r tidy = FALSE, eval = FALSE}

diamonds %>%
    select(-starts_with("c"))

```

OK, so now that we have our columns, we can pass them along to some functions:

```{r tidy = FALSE, eval = FALSE}

diamonds %>%
    select(price, carat) %>%
    log10()

```

In this very simplistic view, we need to attach the original data frame back to the $log_2$ transformed columns. This is doable, but there are better ways. For example, we can use the `dplyr` function `bind_cols()`. This is a `dplyr` version of `cbind()`.

```{r tidy = FALSE, eval = FALSE}
diamonds %>%
    select(price, carat) %>%
    log10() %>%
    bind_cols(diamonds)

```

There are even better ways, which we'll see in the next section.

### mutate() \& summarise() {#dplyrmutate}

Remember, we have two basic types of functions we want to apply to our data:

- **Transformation** using `mutate()`
- **Aggregations** using `summarise()`

Some common normalisations (transformations) include z-scores and scaling on a min-max range [0,1]:

```{r tidy = FALSE, eval = FALSE, echo = TRUE}

diamonds %>%
  mutate(price_log10 = log10(price),
         carat_log10 = log10(carat))


```

Descriptive statistics are aggregation functions:

```{r}

diamonds %>%
  group_by(color) %>% 
  summarise(ave = mean(price))

```


