# Case-study: Diamonds

Now that we're familiar with some of the basic concepts of functional and object-oriented programming, let's work on some real data!

## Importing Data

It is likely that most of your data is found in an Excel or tab-delimited table, like the text files provided for the workshop. Use these base package functions to import your data and save them as a variable.

Table: (#tab:data-table-functions) Base package functions for importing data. These functions will handle most of your general input/output needs.

| Function                | Description                                               |
|-------------------------|-----------------------------------------------------------|
| `read.table(file)`      | Read file in table format into data frame.                |
| `read.csv(file)`        | Read comma-separated data.                                |
| `read.csv2(file)`       | Read semicolon-separated data.                            |
| `read.delim(file)`      | Read tab-delimited data into data frame.                  |
| `read.delim2(file)`     | Read tab-delimited data where the decimal place is a `,`. |

#### stringsAsFactors {#sub:sAFactors2}

An important thing to consider here is something that I mentioned at the end of the dataframes section. R version 4.0, released (April, 2020), changed the default value of these import functions. The argument `stringsAsFactors` is now `FALSE` by default, istead of `TRUE`. If you use these functions in older versions of R, columns that contain characters will be converted to factors. We'll discuss factors in later on. 

This behavior was for a good, but out-dated, reason and became a long-standing complaint in the R community. Although the community wanted to have this default behavior changed, it does create some confusion since you may inherit old R scripts from colleagues that are aware of this behavior and utilize factors in their script. If you run those scripts using an upgraded version of R, you may encounter problems.

An alternative to the base package functions given above, the `readr` package, part of the tidyverse, can make things easier and faster in some cases. In particular, it doesn't convert characters to factors when importing, so older and newer scripts will work in the same way.

Table: (#tab:readrFunc) `readr` package functions for importing data.

| Function       | Description                     |
|----------------|---------------------------------|
| `read_csv()`   | Read comma-separated data.      |
| `read_csv2()`  | Read semicolon-separated data.  |
| `read_tsv()`   | Read tab-delimited data.        |
| `read_delim()` | Reads data with any delimiter.  |

The `rio` package (R Input/Output) offers the `import()` function for generic import of many different file formats. This is a good starting point if you are going to read in an Excel file. Nonetheless, I would encourage you to _not_ begin working in R using Excel files. It's my experience that the Excel files that students like to use are often times contaminated with summary statistics, multiple tables, plots, colors & lines that encode information, multiple worksheets and inconsistent use of white space. All of these things can be dealt with in R, but they make your life more difficult than it needs to be. Learning R is already a lot of work, so make your life easier by starting with a dataset that that is going to be easy to read into R and has not been modified after it was generated.

```{r echo = FALSE}
jems <- read_csv("data/diamonds.csv")
```

The `diamonds.csv` file, contains data on over 50,000 diamonds. The data set actually comes from the `ggplot2` package. 

```{exercise, name="Import and Examine"}

Here we are going to use the RStudio cloud platform. So I know that the file to import is in your working directory.

If you set up a new project in RStudio Desktop, then you can move your data files into that folder on your computer and they will also be in your working directly.

Import the _comma-separated values_ file called `diamonds.csv`, found inside the data folder, and save it as an object called `jems`. See the notes below if your having a hard time.

```

A very common point of confusion is what should be the argument for any of the import functions given above. Many students just do something like:

```{r eval = FALSE}
read_csv(diamonds.csv)
```

This will produce an error saying that the `diamonds.csv` object is not found. That makes perfect sense because you gave an argument pointing to some object in your environment not to a file on your computer (or in this case the cloud). So you need to give the name of the file using `""`, and more than that, you need to specify which sub-directory it's in. Try `"data/diamonds.csv"`

The data set has the following variables:

| Variable  | Description                                                                                       |
|-----------|---------------------------------------------------------------------------------------------------|
| `price`   | Price in US dollars ($326 – $18,823)                                                              |
| `carat`   | Weight of the diamond (0.2 – 5.01)                                                                  |
| `cut`     | Quality of the cut (Fair, Good, Very Good, Premium, Ideal)                                        |
| `color`   | Diamond colour, from D (best) to J (worst)                                                        |
| `clarity` | A measurement of how clear the diamond is (I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best)) |
| `x`       | Length in mm (0 – 10.74)                                                                            |
| `y`       | Width in mm (0 – 58.9)                                                                              |
| `z`       | Depth in mm (0 – 31.8)                                                                              |
| `depth`   | Total depth percentage = z / mean(x, y) = 2 * z / (x + y) (43–79)                                 |
| `table`   | Width of top of diamond relative to widest point (43–95)                                          |

```{exercise, name = "Examine structure"}
What type of data is contained in each column? Use some of the functions we introduced in the dataframes section to explore the basic structure of our new object.
```

Let's begin exploring our data by looking at some basic plots and doing some transformations. 

## Exercises for Plotting, transforming and EDA

After you have familiarized yourself with the `jems` data set, proceed with the following exercises. Try to use the tools we discussed in the chapters on logical expressions and indexing, plus the functions from the tidyverse that we presented, like `filter()` and  `[]`.  

```{exercise, name="Counting individual groups"}

- How many diamonds with a clarity of category "IF" are present in the data-set?
- What fraction of the total do they represent?
  
```

```{exercise, name="Summarizing proportions"}

- What proportion of the whole is made up of each category of clarity?

```


```{exercise, name="Find specific diamonds prices"}

- What is the cheapest diamond price overall?
- What is the range of diamond prices?
- What is the average diamond price in each category of cut and color?

```


```{exercise, name="Basic plotting"}

Make a scatter plot that shows the price of a diamond as described by another continous variable, like the carat. 

```

Your plot should look like this:

```{r echo = FALSE}
ggplot(jems, aes(carat, price)) +
  geom_point()
```

```{exercise, name="Interpreting plots"}

What can you say about the relationship between these two variables? Do you think that you can use the carat weight of a diamond to predict its price?

```

It looks like we would have a better relationship if we transformed our variables. There are many ways to go about applying transformations to your data set. Here we can use older base package methods. The reason I show this is because it's very simple and you will see it a lot in help pages and scripts, so it's worthwhile being familiar with both base package and tidyverse ways of doing things.

```{exercise, name="Applying transformations"}

Using the functions we discuss earlier, and in class, apply a log10 transformation to both the price and carat. You can save these as new columns in the data set called `price_log10` and `carat_log10`.

```

```{exercise, name="Basic plotting"}
Make a scatter plot that shows the price of a diamond as described by another continous variable, like the carat. 
```

Your plot should now look like this:

```{r echo = FALSE}
ggplot(jems, aes(log10(carat), log10(price))) +
  geom_point() +
  labs(x = "carat_log10", y = "price_log10")
```
```{exercise, name="Viewing models"}

At the beginnin of the course we used the `PlantGrowth` data set to produce a model. Can you use the same function we used earlier, `lm()` to recreate a model that describes the relatioship shown in the plot? We'll get into the details of exactly what that model is doing later on. For now, we'll just take a look at it in action. 

```


```{exercise, name="Displaying models"}

Now that we've described the diamond price given a single variable, can you display that on the plot? Tryto use the `geom_smooth()` function to add this new layer.

```

You plot should look like this?

```{r echo = FALSE}
ggplot(jems, aes(log10(carat), log10(price))) +
  geom_point() +
  geom_smooth(method = "lm", color = "red") +
  labs(x = "carat_log10", y = "price_log10")
```


## Saving a data frame to a file {#exporting}

There are several ways to save an R object outside of the environment. If you have a data frame, The most straightforward way is to use the `rio` package:

| Function           | Description                    |
|--------------------|--------------------------------|
| `export(df, file)` | Save the data frame `df` as a file. |

This will save the file type according to the file extension. For example, `"myData.txt"` will result in a tab-separated file. 

Alternatively, you'll also see base package versions:

| Function                | Description                                             |
|-------------------------|---------------------------------------------------------|
| `write.table(df, file)` | Save the data frame `df` as a file.                     |
| `write.csv(df, file)`   | Save the data frame `df` as a comma-separated file.     |
| `write.csv2(df, file)`  | Save the data frame `df` as a semicolon-separated file. |

Try using the form `write.table(df, "data.txt", sep = "\t", row.names = F)` to produce tab-delimited files. You can see that `export()` takes care of a lot of the settings for you, just by giving a proper extension to your file.

