Load the AI survey

ai <- read_csv("data/harvard/threat_ai_jobs.csv")
glimpse(ai)

• item identifies the survey question.
• group identifies all respondents or a party group.
• response gives the answer category.
• percent gives the weighted percentage choosing that answer within the group.
• base_n gives the number of respondents in the group.

How many call AI a serious threat?

What percentage of all respondents consider AI a “Very serious” or “Somewhat serious” threat?

ai |> filter(group == "All") |> select(response, percent)

# Calculate the overall serious-threat percentage.

Visualize the overall responses

overall <- ai |> filter(group == "All")

overall |>
  ggplot(aes(response)) +
  geom_bar()

Do opinions about AI differ by political affiliation?

Investigate the response distributions for Democrats, Republicans, and independents.

by_party <- ai |> filter(group != "All")

# Investigate how the responses differ across parties.

Is concern about AI independent of party?

We don’t have the raw respondent-level data, only the summarized percentages, so estimate counts as round(base_n * percent / 100). Omit Not sure and Refused.

party_counts <- ai |>
  filter(group %in% c("Democrat", "Republican"),
         !response %in% c("Not sure", "Refused")) |>
  mutate(estimated_n = round(base_n * percent / 100))

# Create a serious/not-serious column.
# Sum the counts into a table with:
# xtabs(estimated_n ~ serious + group, data = ...)

Get all the data into one table

We have sixteen questions, each in its own file. We want to analyze them together. Here are a few rows from the combined table.

# A tibble: 4 × 5
  item               group response         percent base_n
  <chr>              <chr> <chr>              <dbl>  <dbl>
1 Large corporations All   Very serious        40.1   2024
2 Large corporations All   Somewhat serious    28.1   2024
3 AI replacing jobs  All   Very serious        35.7   2024
4 AI replacing jobs  All   Somewhat serious    28.7   2024

The columns stay the same. We stack the rows from each file.

Strategy 1: type out each filename

bind_rows() stacks tables, matching their columns by name.

survey_manual <- bind_rows(
  read_csv("data/harvard/threat_billionaires.csv", show_col_types = FALSE),
  read_csv("data/harvard/threat_corporations.csv", show_col_types = FALSE),

  read_csv("data/harvard/threat_immigration_jobs.csv", show_col_types = FALSE),
  read_csv("data/harvard/threat_ai_jobs.csv", show_col_types = FALSE),
  read_csv("data/harvard/threat_cost_of_living.csv", show_col_types = FALSE),
  read_csv("data/harvard/threat_national_debt.csv", show_col_types = FALSE),
  read_csv("data/harvard/threat_trade_wars.csv", show_col_types = FALSE),
  read_csv("data/harvard/policy_medicare_for_all.csv", show_col_types = FALSE),
  read_csv("data/harvard/policy_wealth_tax.csv", show_col_types = FALSE),
  read_csv("data/harvard/policy_deportations.csv", show_col_types = FALSE),
  read_csv("data/harvard/policy_israel_aid.csv", show_col_types = FALSE),
  read_csv("data/harvard/policy_transgender_sports.csv", show_col_types = FALSE),
  read_csv("data/harvard/policy_stock_trading.csv", show_col_types = FALSE),
  read_csv("data/harvard/policy_voter_id.csv", show_col_types = FALSE),
  read_csv("data/harvard/policy_abortion.csv", show_col_types = FALSE),
  read_csv("data/harvard/policy_tariffs.csv", show_col_types = FALSE)

)
survey_manual |> count(item)

What would you have to change if Harvard added ten more questions?

Use iteration

The filenames are already listed in items.csv. We will study ways to read them all efficiently.

survey_files <- paste0("data/harvard/", items$file)

survey_files

Strategy 2: a for loop

A for loop runs the code inside its braces once for each file. file takes the next filename on each pass.

for (file in survey_files) {
  survey <- read_csv(
    file, show_col_types = FALSE)
}
survey |> count(item)

# Repair the loop to keep every question.

Keep each table the loop reads

A list can hold all sixteen tables. survey_tables[[file]] stores a table under its filename.

survey_tables <- list()
for (file in survey_files) {
  survey_tables[[file]] <- read_csv(file, show_col_types = FALSE)
}
survey_loop <- list_rbind(survey_tables)
survey_loop |> count(item)

list_rbind() stacks the tables in the list into one data frame.

Apply a function to each element

Each colored box is an input. The same function f is applied to each one.

Our inputs are filenames, f is read_csv(), and each output is a table.

Hadley Wickham, Advanced R

Strategy 3: map

map() calls a function on each input and collects its outputs in a list. Here the inputs are filenames and the function is read_csv().

survey_tables <- map(survey_files, read_csv, show_col_types = FALSE)
survey_tables[[1]]

[[1]] displays the first table in the list. Which part of the loop is map() handling for us?

Collect all sixteen questions

Use map() and list_rbind() to create the combined table.

survey_files

# Read every file and combine its rows into survey.

Check the combined table

survey <- survey_files |> map(read_csv, show_col_types = FALSE) |> 
  list_rbind()
survey |> count(item)

Six response categories for four groups: 24 rows per question.

Separate threats and policies

items identifies seven financial-security questions and nine policy questions.

survey_with_types <- survey |>
  left_join(items |> select(item, kind), by = "item")

threats <- survey_with_types |> filter(kind == "threat")
policies <- survey_with_types |> filter(kind == "policy")

survey_with_types |> count(kind)

Use threats for questions about financial security and policies for questions about policy support.

What is your biggest financial worry?

What is each party’s biggest financial worry?

Among the seven questions in threats, which financial-security issue concerns each party most? Find which financial-security issue has the highest combined percentage of Very serious and Somewhat serious responses for each party.

threats |> count(response)

# Find the biggest concern within each party group.

Find the top concern in each party

threat_results <- threats |>
  filter(response %in% c("Very serious", "Somewhat serious")) |>

  group_by(item, group) |>
  summarize(percent = sum(percent), .groups = "drop")

threat_results |>
  filter(group != "All") |>
  group_by(group) |>
  top_n(1, percent)

Cost of living tops the list for all three groups.

Which concerns divide Democrats and Republicans?

Define net serious as the percentage saying Very serious or Somewhat serious minus the percentage saying Not very serious or Not a threat.

How do party groups differ on this score?

threats |>
  filter(response %in% c("Very serious", "Somewhat serious",
                         "Not very serious", "Not a threat"))

# Calculate net-serious scores and find the largest party gap.

Ranking the party differences

threat_net <- threats |>
  filter(response %in% c("Very serious", "Somewhat serious",
                         "Not very serious", "Not a threat")) |>
  mutate(contribution = ifelse(
    response %in% c("Very serious", "Somewhat serious"), percent, -percent)) |>
  group_by(item, group) |>
  summarize(net_serious = sum(contribution), .groups = "drop")

threat_comparison <- threat_net |>
  pivot_wider(names_from = group, values_from = net_serious) |>

  mutate(party_gap = Democrat - Republican, abs_gap = abs(party_gap))

threat_comparison |>
  arrange(desc(party_gap)) |>
  select(item, Democrat, Republican, Independent, party_gap)

Check the ranking

threat_comparison |>
  arrange(desc(abs_gap)) |>
  select(item, Democrat, Republican, Independent, abs_gap)

Immigration is near the top because the Republican score is much higher.

What concerns are shared? Which are divisive?

Make a plot of the net-serious scores for each issue across dems, republicans, and independents.

threat_net

# Plot the scores for each party group.

What does the full set show?

Which policies win a majority in every party?

• What proposals are supported by more than 50% of Democrats, Republicans, and independents?
• The questions include healthcare, taxes, stock trading, voting rules, immigration, abortion, school sports, military aid, and tariffs.

Find the policies with support across parties

Calculate support for all nine policies, then find those that pass the 50% threshold in all three party groups.

policies |> count(response)

# Calculate policy_results and find majority support across parties.

Can support be greater than 100%?

policy_attempt <- policies |>
  filter(response %in% c("Strongly support", "Somewhat support")) |>
  group_by(group) |>
  summarize(percent = sum(percent), .groups = "drop")
policy_attempt

Calculate support within each group

policy_results <- policies |>
  filter(response %in% c("Strongly support", "Somewhat support")) |>

  group_by(item, group) |>
  summarize(percent = sum(percent), .groups = "drop")

policy_results |> count(item)

Check which policies passed

policy_comparison <- policy_results |>
  pivot_wider(names_from = group, values_from = percent) |>

  mutate(party_gap = Democrat - Republican, abs_gap = abs(party_gap))

policy_comparison |>
  filter(Democrat > 50, Republican > 50, Independent > 50) |>
  select(item, Democrat, Republican, Independent)

Did either result surprise you?

Which policies have the biggest and smallest party gaps?

policy_comparison

# Find the largest and smallest absolute party gaps.

Congressional stock trading

The proposal would prevent members of Congress and their spouses from trading individual stocks while in office.

# A tibble: 1 × 5
  item                              All Democrat Republican Independent
  <chr>                           <dbl>    <dbl>      <dbl>       <dbl>
1 Congressional stock-trading ban  63.3     70.7       62.4        62.4

A majority supports it in each party group. It also has the smallest Democratic–Republican gap among these nine policies.

Where do the parties disagree?

Popular but controversial policies?

Which policies have majority support overall but fall below 50% in at least one party group?

policy_comparison

# Find policies that meet the condition above.

Which policies did you find?

policy_comparison |>
  filter(All > 50,
         Democrat < 50 | Republican < 50 | Independent < 50) |>
  select(item, All, Democrat, Republican, Independent)

Do independents track Democrats or Republicans?

We have each group’s support for nine policies.

Across those policies, does the pattern among independents resemble Democrats’ or Republicans’ more closely?

Use scatterplots and correlation to investigate.

Investigate independent opinion

independent_plot_data <- policy_comparison |>
  select(item, Independent, Democrat, Republican) |>
  pivot_longer(c(Democrat, Republican),
               names_to = "party", values_to = "support")

# Compare each party with independents.

Which pattern is more similar?

Does higher correlation mean closer agreement?

policy_comparison |>
  select(item, Democrat, Independent, Republican)

# Find a counterexample.

What we practiced

• Read a collection of files with a loop or map().
• Combine the returned tables with list_rbind().
• Join question types from items and separate threats from policies.
• Use scatterplots and correlation to study party agreement.
• Communicate a finding with a table, plot, and explanation.