# Effects of Meditation Case Study

```{block, type="learningObj"}

In this case study we'll look at data simulated from a real-world, published experiment. We'll explore some new key concepts in this chapter: rearranging data to make it easier to work with.
```

As an exercise in manipulating data, consider the _meditation_ data-set. This data-set contains the expression values of 6 pro-inflammatory genes. These are results from a published study. The researchers wanted to test if these genes are deferentially expressed after meditation in people who regularly meditate, compared to non-meditating control individuals. Expression levels were measured in the morning (time 1) and after (time 2) a full day of meditation, by experienced meditators, or leisure, by the control group.

```{r excel, echo=FALSE, fig.cap="The `medi` data-set as it would be typically stored as an Excel spreadsheet. Although this is _human-readable_, it is _not_ an efficient way of storing data!", out.width = '50%'}

knitr::include_graphics("figures/Excel_Example_trans.jpg")

```

```{exercise, name="Messy versus Tidy"}
What is the problem with the data format in figure \@ref(fig:excel)?
```

Figure \@ref(fig:excel) is an example of a typical Excel spreadsheet. Here, column headers are found in the middle of the spread sheet and the results for each measured gene is contained in its own table on the same worksheet. In addition, column headers are colour-coded and there is plenty of white space for formatting purposes.

As a first step towards tidy data, we will remove all white space, colour-coding and floating labels. The data-set in Figure \@ref(fig:ExpBetter) resembles the long list file we have already encountered. It's not perfect, but it's a good start.

```{r ExpBetter, echo=FALSE, fig.cap="The `medi` data-set in a format ready for importing into R.", out.width = '100%'}

knitr::include_graphics("figures/ExpBetterFormat.jpg")

```

## Tidy data

Reshaping and tidying our data (Tidy Data Paper)[http://vita.had.co.nz/papers/tidy-data.pdf]

- Each row is an observation
- Each column is a variable
- Each type of observational unit forms a table

```{python}

df = pd.DataFrame({
            'name': ["John Smith", "Jane Doe", "Mary Johnson"],
            'treatment_a': [None, 16, 3], 
            'treatment_b': [2, 11, 1]})
print(df)
df_melt = pd.melt(df, id_vars='name')
print(df_melt)

```

Tidy pivot_table

```{python}

df_melt_pivot = pd.pivot_table(df_melt,
                               index='name',
                               columns='variable',
                               values='value')
                               
print(df_melt_pivot)

```

This results in a hierarchical index, so to get a regular flat data frame, call

```{python}

df_melt_pivot.reset_index()
print(df_melt_pivot)

```

## groupby operations

- groupby: split-apply-combine
  - _split_ data into separate partitions
  - _apply_ a function on each partition
  - _combine_ the results
  
```{python}
print(df_melt)
df_melt.groupby('name')['value'].mean()
```

```{exercise, name="Import data"}
Import `Expression.txt`. Save it as an object called `medi`.
```

```{exercise, name="Tidy data"}
Convert the data set to a _tidy_ format.
```

```{exercise, name="Calculate Statistics"}

Calculate each of the following statistics for each of the unique 24 combinations of gene, treatment and time:

- `average` the mean of the value
- `n` the number of observations in each group
- `SEM` The standard error of the mean
- `CIerror` The 95% CI error defined by the t distribution
- `lower95` The upper 95% CI limit
- `upper95` The upper 95% CI limit
```


```{exercise, name="Export data"}
Now that you've processed your data, refer to the following table and save a file on your computer that contains the summary statistics.
```

## Wrap-up

```{block, type="chapRecap"}

So far in our journey we've seen how to do all the common steps from R in the Python way, including using tidy data.

```
