---
title: "Lec 08 - tidyr"
subtitle: "<br/> Statistical Programming"
author: "Sta 323 | Spring 2022"
date: "<br/> Dr. Colin Rundel"
output:
  xaringan::moon_reader:
    css: ["slides.css"]
    lib_dir: libs
    nature:
      highlightStyle: solarized-light
      highlightLines: true
      countIncrementalSlides: false
      ratio: "16:9"

---
exclude: true

```{r setup, message=FALSE, warning=FALSE, include=FALSE}
options(
  htmltools.dir.version = FALSE, # for blogdown
  width=80
)

library(tidyverse)
```

---
class: middle

.center[
```{r echo=FALSE, out.width="45%"}
knitr::include_graphics('imgs/hex-tidyr.png')
```
]

---
class: middle, center

# Reshaping data (Wide vs. Long)

---

## Wide -> Long

```{r echo=FALSE, out.width="60%", fig.align="center"}
knitr::include_graphics('imgs/tidyr_gather.png')
```

.center[
`pivot_longer` (previously `gather`)
]

.footnote[ From [data import cheatsheet](https://github.com/rstudio/cheatsheets/blob/master/tidyr.pdf) ]

---

## Syntax

```{r}
(d = tibble::tribble(
  ~country, ~"1999",  ~"2000",
        "A", "0.7K",     "2K",
        "B",  "37K",    "80K",
        "C", "212K",   "213K"
))
```

--

```{r}
pivot_longer(d, cols = "1999":"2000", names_to = "year", values_to = "cases")
```


---

## Long -> Wide

```{r echo=FALSE, out.width="70%", fig.align="center"}
knitr::include_graphics('imgs/tidyr_spread.png')
```

.center[ .larger[
`pivot_wider` (previously `spread`)
] ]

.footnote[ From [data import cheatsheet](https://github.com/rstudio/cheatsheets/blob/master/data-import.pdf) ]

---

## Syntax 

.pull-left[
```{r}
d = tibble::tribble(
  ~country, ~year,   ~type, ~count,
       "A",  1999, "cases", "0.7K",
       "A",  1999,   "pop",  "19M",
       "A",  2000, "cases",   "2K",
       "A",  2000,   "pop",  "20M",
       "B",  1999, "cases",  "37K",
       "B",  1999,   "pop", "172M",
       "B",  2000, "cases", " 80K",
       "B",  2000,   "pop", "174M",
       "C",  1999, "cases", "212K",
       "C",  1999,   "pop",   "1T",
       "C",  2000, "cases", "213K",
       "C",  2000,   "pop",   "1T"
)
```
]

.pull-right[
```{r}
d
```
]

--

```{r}
pivot_wider(d, id_cols = country:year, names_from = type, values_from = count)
```


---

## Separate

```{r echo=FALSE, out.width="70%", fig.align="center"}
knitr::include_graphics('imgs/tidyr_separate.png')
```

--

```r
separate(d, rate, sep = "/", into = c("cases", "pop"))
```

.footnote[ From [data import cheatsheet](https://github.com/rstudio/cheatsheets/blob/master/tidyr.pdf) ]

---

## Unite

```{r echo=FALSE, out.width="70%", fig.align="center"}
knitr::include_graphics('imgs/tidyr_unite.png')
```

--

```r
unite(d, century, year, col = "year", sep = "")
```

.footnote[ From [data import cheatsheet](https://github.com/rstudio/cheatsheets/blob/master/tidyr.pdf) ]

---

## Example 1 - tidy grades

Is the following data tidy?

```{r}
(grades = tibble::tribble(
    ~name, ~hw_1, ~hw_2, ~hw_3, ~hw_4, ~proj_1, ~proj_2,
  "Alice",    19,    19,    18,    20,      89,      95,
  "Bob",      18,    20,    18,    16,      77,      88,
  "Carol",    18,    20,    18,    17,      96,      99,
  "Dave",     19,    19,    18,    19,      86,      82
))
```

--

<br/>

How would we calculate a final score based on the following formula,
$$\text{score} = 0.5\,\frac{\sum_i\text{hw}_i}{80} + 0.5\,\frac{\sum_j\text{proj}_j}{200}$$

---

## Semi-tidy approach

```{r}
grades %>%
  mutate(
    hw_avg = (hw_1+hw_2+hw_3+hw_4)/4,
    proj_avg = (proj_1+proj_2)/2
  ) %>%
  mutate(
    overall = 0.5*(proj_avg/100) + 0.5*(hw_avg/20)
  )
```

---

## pivot_longer (Wide -> Long)

```{r}
tidyr::pivot_longer(
  grades, 
  cols = hw_1:proj_2, 
  names_to = "assignment", 
  values_to = "score"
)
```

--

.center[
What does this get us?
]

---

```{r}
tidyr::pivot_longer(
  grades, 
  cols = hw_1:proj_2, 
  names_to = c("type", "id"), 
  names_sep = "_", 
  values_to = "score"
)
```



---

## Tidy approach?

```{r}
grades %>%
  tidyr::pivot_longer(
    cols = hw_1:proj_2, 
    names_to = c("type", "id"),
    names_sep = "_", 
    values_to = "score"
  ) %>% 
  group_by(name, type) %>%
  summarize(
    total = sum(score),
    .groups = "drop"
  )
```

---

## pivot_wider - (Long -> Wide)

```{r}
grades %>%
  tidyr::pivot_longer(
    cols = hw_1:proj_2, 
    names_to = c("type", "id"), 
    names_sep = "_", 
    values_to = "score"
  ) %>% 
  group_by(name, type) %>%
  summarize(
    total = sum(score),
    .groups = "drop"
  ) %>%
  tidyr::pivot_wider(
    names_from = type, 
    values_from = total
  )
```

---

## Wrapping up

```{r}
grades %>%
  tidyr::pivot_longer(
    cols = hw_1:proj_2, 
    names_to = c("type", "id"), 
    names_sep = "_", 
    values_to = "score"
  ) %>% 
  group_by(name, type) %>%
  summarize(
    total = sum(score),
    .groups = "drop"
  ) %>%
  tidyr::pivot_wider(
    names_from = type, 
    values_from = total
  ) %>%
  mutate(
    score = 0.5*(hw/80) + 0.5*(proj/200)
  )
```

---

## Exercise 1

The `palmerpenguin` package contains measurement data on various penguin species on islands near Palmer Station in Antarctica. The code below shows the # of each species measured on each of the three islands (missing island, penguin pairs implies that species does not occur on that island).

```{r}
palmerpenguins::penguins %>%
  count(island, species)
```

Starting from these data construct a contingency table of counts for island (rows) by species (columns) using the pivot functions we've just discussed.




---
class: center, middle

# Rectangling

---

## Star Wars & repurrrsive

`repurrrsive` is a package that contains a number of interesting example data sets that are stored in a hierarchical format. Many come from web-based APIs which provide results as JSON. 

```{r}
str(repurrrsive::sw_people)
```

---

```r
View(repurrrsive::sw_people)
```

```{r out.width="75%", echo=FALSE, fig.align="center"}
knitr::include_graphics("imgs/sw_people_View.png")
```



---

## Tidy data from nested lists

Recent versions of `tidyr` have added several functions that are designed to aide in the tidying of hierarchical data. Since they are part of `tidyr` all of the following functions work with data frames.

From `tidyr` - `hoist()`, `unnest_longer()`, and `unnest_wider()` provide tools for rectangling, collapsing deeply nested lists into regular columns.

---

## Lists as columns

.pull-left[
```{r}
(sw_df = tibble::tibble(
  people = repurrrsive::sw_people
))
```
]

--

.pull-right[
```{r}
sw_df %>%
  as.data.frame() %>%
  head()
```
]

--

```{r}
is.data.frame(sw_df)
nrow(sw_df)
```

---

## Unnesting

```{r}
sw_df %>%
  unnest_wider(people)
```

---

## More list columns

.pull-left[
```{r}
sw_df %>%
  unnest_wider(people) %>% 
  select(name, starships)
```
]

--

.pull-right[
```{r}
sw_df %>%
  unnest_wider(people) %>% 
  select(name, starships) %>%
  pull(starships) %>%
  str()
```
]

---

## Unnest Longer

```{r}
unnest_wider(sw_df, people) %>% 
  select(name, starships) %>%
  unnest_longer(starships)
```

---

## Aside - sw_starships

```{r}
(ships = tibble(ships = repurrrsive::sw_starships) %>%
   unnest_wider(ships) %>%
   select(ship = name, url)
)
```

---
class: middle

.center[
```{r echo=FALSE, out.width="40%"}
knitr::include_graphics('imgs/hex-dplyr.png')
```

.large[(Joins)]
]

---

## Joins (left)

```{r echo=FALSE, out.width="50%", fig.align="center"}
knitr::include_graphics("imgs/left-join-extra.gif")
```

.footnote[
From [gadenbuie/tidyexplain](https://github.com/gadenbuie/tidyexplain)
]

---

## Joins (right)

```{r echo=FALSE, out.width="50%", fig.align="center"}
knitr::include_graphics("imgs/right-join.gif")
```

.footnote[
From [gadenbuie/tidyexplain](https://github.com/gadenbuie/tidyexplain)
]

---

## Joins (full / outer)

```{r echo=FALSE, out.width="50%", fig.align="center"}
knitr::include_graphics("imgs/full-join.gif")
```

.footnote[
From [gadenbuie/tidyexplain](https://github.com/gadenbuie/tidyexplain)
]


---

## Joins (inner)

```{r echo=FALSE, out.width="50%", fig.align="center"}
knitr::include_graphics("imgs/inner-join.gif")
```

.footnote[
From [gadenbuie/tidyexplain](https://github.com/gadenbuie/tidyexplain)
]

---

## Joining people and starships

```{r}
sw_df %>%
  unnest_wider(people) %>% 
  select(name, starships) %>%
  unnest_longer(starships) %>%
  left_join(ships, by = c("starships" = "url"))
```

---

## Putting it together

```{r}
sw_df %>%
  unnest_wider(people) %>% 
  select(name, starships) %>%
  unnest_longer(starships) %>%
  inner_join(ships, by = c("starships" = "url")) %>%
  select(-starships) %>%
  group_by(name) %>%
  summarize(ships = list(ship), .groups = "drop")
```

---

```{r}
sw_df %>%
  unnest_wider(people) %>% 
  select(name, starships) %>%
  unnest_longer(starships) %>%
  inner_join(ships, by = c("starships" = "url")) %>%
  select(-starships) %>%
  group_by(name) %>%
  summarize(ships = paste(ship, collapse = ", "), .groups = "drop")
```

---

## Exercise 2

1. Which planet appeared in the most starwars film (according to the data in `sw_planet`)?


2. Which planet was the homeworld of the most characters in the starwars films?
