# Comprehensions

```{block, type="learningObj"}
Now that we've seen iterations in action, let's take a look at how list and dict comprehensions make our lives easier.
```

## List comprehensions

List comprehensions are used to create lists from other lists, DataFrame columns and other data containers. Comprehensions are useful and common as they allow you to rapidly iterate over sets of objects. They allow us to perform complex operations in a single line of code. 

For example, this for loop is pretty tedious:

```{python set1}
vals = [6, 8, 4, 2, 5, 6, 7, 3, 5]
vals
```

```{python}
new_vals = []

for num in vals:
    new_vals.append(num + 10)
    
new_vals
```

A much easier way is to use a list comprehension:

```{python}
new_vals2 = [num + 10 for num in vals]
new_vals2
```

Can you see the direct relationship between the above list comprehension and the for loop? 

Well, to be honest, in this case you would just use a `NumPy` array.

```{python}
vals = np.array([6, 8, 4, 2, 5, 6, 7, 3, 5])
vals
vals + 10
```

But that will not work for everything you want to do, since comprehensions are not just for lists, it works over any _iterable._ Remember that a range object is _iterable_: 

```{python}
# like a range object
[num + 10 for num in range(10)]
```

where I don't have to convert to a `NumPy` array first. To perform list comprehensions, we need:

1. An _iterable_
1. An _iterator_ variable to represent the members of the iterable
1. The _output expression_

List comprehensions use the following syntax `[[output expression] for iterator variable in iterable]`.

This also works for nested for loops. Take a look at this typical example:

```{python}
pairs_1 = []

for num1 in range(0,2):
    for num2 in range(6,8):
        pairs_1.append((num1, num2))
        
pairs_1
```

It's better as list comprehensions:

```{python}
pairs_2 = [(num1, num2) for num1 in range(0,2) for num2 in range(6,8)]
pairs_2
```

It is a little less readable at first, but once you get the hang of it, it's a nice syntax. For example:

```{python}
[i**2 for i in range(0,10)]
```

Let's look at a more interesting example, with a matrix. Remember a matrix is just a 2-dimenaional `NumPy` array. In Python a matrix is represented as a list of lists, all having the same type:

```{python}
matrix = [[0, 1, 2, 3, 4],
          [0, 1, 2, 3, 4],
          [0, 1, 2, 3, 4],
          [0, 1, 2, 3, 4],
          [0, 1, 2, 3, 4]]
```

Can we produce this using nested list comprehensions?

You can create one of the rows of the matrix with a single list comprehension, then, to create the list of lists, you simply have to supply the list comprehension as the output expression of the overall list comprehension.

That is, the _output expression_, as we see in the generic syntax `[[output expression] for iterator variable in iterable]`, is itself a list comprehension.

Here's the nested for loop solution:

```{python}
matrix = [] 
  
for i in range(5): 
      
    # Append an empty sublist inside the list 
    matrix.append([]) 
      
    for j in range(5): 
        matrix[i].append(j) 
          
matrix
```

And as a list comprehension:

```{python}
# Nested list comprehension 
matrix = [[j for j in range(5)] for i in range(5)] 
# matrix = [[col for col in range(5)] for row in range(5)]
print(matrix) 
```

So that is certainly cleaner code, and you can appreciate why list comprehensions are so popular.

## Using conditionals

In the chapter on Logical Expressions, we saw how to use relational and logical operators to result in Boolean objects. Here, we can use them to subset lists.

`[` _output expression_ `for` _iterator variable_ `in` _iterable_ `if` _predicate expression_ `]`

For example:

```{python}
[num ** 2 for num in range(10) if num % 2 == 0]
```

Get only long names:
```{exercise}
Using the `cities` list, below, extract only those cities with long names, over 6 characters long.
```

```
cities = ['Munich', 'Paris', 'Amsterdam', 'Madrid', 'Istanbul']
```

```{python eval = FALSE, echo = FALSE}
cities = ['Munich', 'Paris', 'Amsterdam', 'Madrid', 'Istanbul']

[number for number in cities if len(number) > 6]

```

As an example, recall the modulo operator to check if a value is even? We can include a conditional on the output:

```{python}
# Square all the values and return only the even ones.
[num ** 2 if num % 2 == 0 else 0 for num in range(10)]
```

```{exercise}
Revisit the previous exercise, but change the position of the logical expression.
```

```{python eval = FALSE, echo = FALSE}
cities = ['Munich', 'Paris', 'Amsterdam', 'Madrid', 'Istanbul']

[name if len(name) >=7 else "" for name in cities]
```

## Dictionary comprehensions

In addition to lists, we can also use dictionary comprehensions. There are two key differences:

1. Use `{}`, not `[]`.
2. The key and value are separated by a `:` in the output expressions

```{python}
pos_neg = {num: -num for num in range(10)}
pos_neg
```

```{python}
pos_neg = {num: num**2 for num in range(10)}
pos_neg
```

```{python}
cities = ['Munich', 'Paris', 'Amsterdam', 'Madrid', 'Istanbul']
{names: len(names) for names in cities}
```

### Generators

Generators are like comprehensions, except that they don't store the solution in memory. This is practical when working on large data sets. The difference is in notation, just use `()` notation instead of `[]`.

## Wrap-up

<!-- Recall `zip()` accepts an arbitrary number of iterables and returns an iterator of tuples -->

```{block, type="chapRecap"}

So far in our journey in iterations we've seen:

- Iterators
- Associated iteration functions
- Generators (very briefly)
- List and dict comprehensions
```