# Parsing data

```{block, type="learningObj"}
Our learning objectives in this session are to understand how to access information using indexing for position and logical expressions.
```

### Indexing

Finding information, or sub-setting, is called indexing. Every item in a data frame has a position and some have names.

We select positions using `[]` notation after a pandas data frame.

Recall that to get a specific column, we can call it by name:

```{python eval = FALSE}
# A DataFrame, ss defined previously
foo_df
```

```{python}
# Using []
foo_df['tissue']
```

```{python}
# using . notation
foo_df.tissue
```

We can also select items using using position. e.g. we can index rows by index position with `.iloc[]`:

```{python eval = FALSE}
foo_df

# First row, as a Series
foo_df.iloc[0] 

# First row, as a DataFrame
foo_df.iloc[[0]] 
```

```{python}
# a list of integers, the first two rows
foo_df.iloc[[0, 1]] 
```

But more explicitly, we can use `[ <rows> , <columns> ]` notation. In this case we must also use `:` notation to specify ranges, even if we want all rows or columns.

```{python eval = FALSE}
# To get all columns, use : after the comma
foo_df.iloc[0, :] 
```

```{python}
# a list of integers, the first two rows
foo_df.iloc[[0, 1], :]
```

```{python}
# The first two columns, all rows
foo_df.iloc[:,:2]
```

```{python}
# A single column, all rows
foo_df.iloc[:,1:2]
```

These simple examples show us some important concepts in indexing:

- Indexing in Python begins at 0!
- If no `,` is present then we retrieve rows
- We specify a range using the `:` operator, as per `start:end`. If using `,` this must be included
- The end position is _exclusive_, i.e. _not_ included in the series!
- If no `start` or `end` position is given, then we take the beginning to the end, respectively.
- Using a negative number, i.e. `-1` begins counting in the _reverse_ direction.  

```{exercise}

Using `foo_df`, what commands would I use to get: 

- The 2nd to 3rd rows?
- The last 2 rows?
- A random row in `foo_df`?
- From the 4th to the last row? But without hard-coding, i.e. regardless of how many rows my data frame contains 

```

<!-- Like columns, we can also index rows by label (loc) -->

<!-- ```{python eval = FALSE} -->
<!-- foo_df.loc['liver'] # The xth row -->

<!-- # foo_df.loc[['x', 'y']] # The xth and yth row -->
<!-- ``` -->

<!-- And or course we can combine both rows and columns -->

<!-- ```{python eval = FALSE} -->
<!-- # foo_df.loc['x', 'A'] -->
<!-- # foo_df.loc[['x', 'y'], ['A', 'B']] -->
<!-- ``` -->

<!-- - List all the possible objects we can use inside `loc[]` -->



```{exercise}

List all the possible objects we can use inside `iloc[]`

e.g. When can we use:
- Integers?
- Floats?
- Characters?
- A heterogenous list?
- A homogenous list?
```

Although we typically just use `seq[start:end]`, the complete notation is `seq[start:end:step]`. The step operation tells how at what interval to sample from the data set. We can reverse the sequence using [::-1]

```{python}
range(10)[::2]
```

```{exercise, name="Indexing at intervals"}
Use indexing to obtain all the odd and even rows only from the foo_df data frame.
```

```{python, eval = FALSE, echo = FALSE}
foo_df.iloc[::2]
foo_df.iloc[1:,:][::2]
```

## Logical Expressions

So far, so good! we saw that we can find information by name and by position. But the real power comes in using logical expressions!

Logical expressions simply mean asking and combining "yes/no" questions. If you think about it, all that computers understand are Yes/No questions. At the end of the day, all computations eventually boil down to "yes/no" questions.

### Relational Operators

Relational operators ask "yes/no" questions. In place of yes and no, Python uses type `boolean`, `True/False` or `1/0`, to provide a positive or negative answer. There are really only 6 kinds of "yes/no" questions. _Relational operators_ are listed in table \@ref(tab:RelOp):

Table: (\#tab:RelOp) A summary of relational operators.

| Operator    | Description                      |
|-------------|----------------------------------|
| `<`  | Less than                               |
| `<=` | Less than or equal to                   |
| `>`  | Greater than                            |
| `>=` | Greater than or equal to                |
| `==` | Exactly equal to                        |
| `!=` | Not equal to, i.e. the opposite of `==` |
| `~x` | Not x (logical negation)                |

### Logical Operators

A collection of logical expressions can be combined using the following _logical operators_.

Table: (\#tab:LogOp) A summary of logical operators in Python. `x` and `y` are logical vectors, e.g. the output from _relational operators_.

| Operator | Description |
|----------|-------------|
| `x | y`  | x OR y      |
| `x & y`  | x AND y     |

### Conditional sub-setting

```{python}

foo_df[foo_df.quantity == 233]

foo_df[(foo_df.tissue == "Heart") | (foo_df.quantity == 233)]

```

## Exercises for Parsing data

For the following exercises, find all rows in `foo_df` that contain:

```{exercise}

Subset for boolean data:

- Only "healthy" samples.
- Only "unhealthy" samples.

```

```{exercise}

Subset for numerical data:

- Only low quantity samples, those below 100.
- Quantity between 100 and 1000,
- Quantity below 100 and beyond 1000.

```

```{exercise}

Subset for strings:

- Only "heart" samples.
- "Heart" and "liver" samples
- Everything except "intestines"

```

## Other common operators

There are some other typical operators that you'll encounter, but we won't go into more detail here.

### Using in / not in

| Operator | Meaning	                                             |
|----------|-------------------------------------------------------|
| `in`	   | `True` if value/variable is found in the sequence     |
| `not in` | `True` if value/variable is not found in the sequence |

```{python}
cities = ['Munich', 'Paris', 'Amsterdam', 'Madrid', 'Istanbul']
dist = [584, 1054, 653, 2301, 2191]

'Paris' in cities
'Tehran' in cities

'Paris' not in cities
'Tehran' not in cities
```

### Using and / or

The evaluation using the `and` and `or` operators follow these rules:

- `and` and `or` evaluates expression from left to right.
- With `and`, if all values are `True`, returns the last evaluated value. If any value is `False`, returns the first one.
- With `or`, we return the first `True` value. If all are `False`, returns the last value

| Operator       | Meaning                                             |
|----------------|-----------------------------------------------------|
| `x and y`	     | Returns `x` if `x` is `False`, `y` otherwise        |
| `x or y`	     | Returns `y` if `x` is `False`, `x` otherwise        |


```{python}
'Paris' in cities and 'Tehran' not in cities
'Paris' in cities and 'Tehran' in cities

'Paris' in cities or 'Tehran' not in cities
'Paris' in cities or 'Tehran' in cities
```


## Wrap-up

```{block, type="chapRecap"}

In this chapter we added to our data science knowledge by understanding how to parse information in a data frame. We can use `[]` to parse according to name, position or Boolean list.
```


