Crossover Designs

Model-Based Inference

Randomization-based Inference for Crossover Design

Midterm Data

midterms

# A tibble: 74 × 4
   sections version   Q21   Q23
      <dbl> <chr>   <dbl> <dbl>
 1      101 ac          6     6
 2      101 ac          4     6
 3      101 bd          6     6
 4      101 bd          6     4
 5      101 bd          6     6
 6      101 ac          6     4
 7      101 bd          6     6
 8      101 bd          6     2
 9      101 bd          6     6
10      101 bd          4     2
# ℹ 64 more rows

midterms <- midterms |>
    mutate(Y = ifelse(version == "ac", Q23 - Q21, Q21 - Q23))
midterms

# A tibble: 74 × 5
   sections version   Q21   Q23     Y
      <dbl> <chr>   <dbl> <dbl> <dbl>
 1      101 ac          6     6     0
 2      101 ac          4     6     2
 3      101 bd          6     6     0
 4      101 bd          6     4     2
 5      101 bd          6     6     0
 6      101 ac          6     4    -2
 7      101 bd          6     6     0
 8      101 bd          6     2     4
 9      101 bd          6     6     0
10      101 bd          4     2     2
# ℹ 64 more rows

obs_stat <- mean(midterms$Y)
obs_stat

[1] 0.2972973

count(midterms, sections, version)

# A tibble: 4 × 3
  sections version     n
     <dbl> <chr>   <int>
1      101 ac         18
2      101 bd         18
3      102 ac         19
4      102 bd         19

Randomization Test

\[ H_0: Y_{23}(S) = Y_{23}(C), Y_{21}(S) = Y_{21}(C) \quad \text{for all students} \]

1. Within each section, randomly assign the version (ac or bd) to the students.
2. Compute the test statistic (mean difference) for the simulated data.
3. Repeat steps 1 and 2 many times to get the null distribution of the test statistic.
4. Compute the p-value as the proportion of simulated statistics that are as extreme as or more extreme than the observed statistic.

flip_sign <- function(df) {
    df2 <- df |>
        group_by(sections) |> # block on section
        mutate(version = sample(version)) |> # randomly assign version
        mutate(Ysim = ifelse(version == "ac", Q23 - Q21, Q21 - Q23))

    mean(df2$Ysim)
}

set.seed(4523)
stats <- replicate(1000, flip_sign(midterms))
stats

   [1]  0.13513514 -0.02702703  0.13513514  0.08108108  0.24324324  0.29729730
   [7] -0.24324324  0.08108108  0.02702703  0.35135135 -0.13513514 -0.08108108
  [13] -0.13513514  0.18918919 -0.08108108 -0.18918919 -0.08108108 -0.35135135
  [19] -0.35135135 -0.08108108  0.40540541  0.08108108  0.08108108  0.02702703
  [25] -0.35135135  0.02702703  0.24324324 -0.02702703  0.35135135 -0.40540541
  [31]  0.08108108  0.24324324  0.02702703  0.08108108  0.08108108  0.08108108
  [37] -0.24324324 -0.29729730 -0.24324324  0.13513514  0.13513514 -0.29729730
  [43]  0.13513514 -0.02702703 -0.02702703 -0.35135135  0.02702703 -0.24324324
  [49]  0.02702703  0.02702703  0.18918919  0.35135135  0.13513514  0.18918919
  [55] -0.02702703 -0.08108108 -0.18918919  0.29729730  0.35135135 -0.29729730
  [61]  0.24324324  0.08108108 -0.02702703  0.18918919  0.29729730 -0.08108108
  [67] -0.08108108 -0.24324324  0.13513514  0.18918919 -0.51351351  0.08108108
  [73]  0.13513514 -0.13513514  0.02702703 -0.13513514 -0.13513514 -0.08108108
  [79] -0.13513514 -0.08108108 -0.35135135 -0.18918919 -0.13513514  0.24324324
  [85]  0.02702703 -0.35135135  0.02702703  0.13513514 -0.18918919 -0.02702703
  [91]  0.13513514 -0.08108108  0.18918919  0.24324324 -0.08108108 -0.02702703
  [97] -0.13513514  0.18918919  0.08108108 -0.13513514  0.02702703  0.24324324
 [103]  0.02702703 -0.24324324  0.29729730 -0.13513514 -0.02702703  0.08108108
 [109] -0.18918919  0.13513514  0.13513514  0.13513514 -0.02702703  0.18918919
 [115]  0.02702703  0.08108108  0.45945946  0.24324324 -0.24324324 -0.18918919
 [121]  0.24324324 -0.08108108  0.02702703 -0.08108108  0.24324324 -0.02702703
 [127] -0.02702703  0.18918919 -0.08108108  0.13513514 -0.18918919 -0.02702703
 [133]  0.13513514 -0.18918919 -0.02702703 -0.13513514  0.18918919  0.02702703
 [139]  0.02702703  0.02702703  0.08108108  0.24324324 -0.24324324 -0.02702703
 [145]  0.18918919  0.24324324  0.08108108 -0.02702703 -0.08108108 -0.13513514
 [151] -0.24324324  0.13513514 -0.02702703 -0.29729730 -0.29729730 -0.08108108
 [157]  0.18918919 -0.02702703 -0.13513514 -0.13513514  0.08108108 -0.13513514
 [163] -0.08108108 -0.08108108  0.18918919 -0.02702703  0.02702703 -0.29729730
 [169]  0.24324324 -0.08108108 -0.02702703  0.02702703 -0.45945946 -0.02702703
 [175]  0.24324324 -0.35135135 -0.02702703  0.45945946 -0.08108108  0.13513514
 [181] -0.24324324  0.08108108  0.08108108 -0.08108108 -0.13513514 -0.08108108
 [187]  0.13513514  0.18918919 -0.13513514 -0.24324324 -0.02702703 -0.08108108
 [193]  0.18918919 -0.24324324  0.56756757 -0.29729730  0.08108108  0.18918919
 [199]  0.18918919  0.29729730 -0.02702703 -0.13513514  0.02702703 -0.02702703
 [205] -0.24324324  0.18918919 -0.13513514 -0.08108108  0.24324324 -0.08108108
 [211] -0.40540541 -0.08108108 -0.02702703 -0.24324324  0.18918919 -0.02702703
 [217] -0.13513514 -0.02702703 -0.02702703 -0.02702703 -0.02702703 -0.24324324
 [223]  0.13513514  0.24324324 -0.29729730 -0.18918919 -0.35135135  0.18918919
 [229]  0.24324324 -0.13513514 -0.13513514 -0.24324324  0.08108108 -0.13513514
 [235]  0.18918919  0.13513514 -0.02702703  0.02702703 -0.02702703 -0.02702703
 [241] -0.08108108  0.18918919 -0.13513514  0.02702703 -0.08108108 -0.02702703
 [247]  0.24324324 -0.13513514  0.13513514  0.13513514 -0.02702703  0.02702703
 [253] -0.02702703 -0.29729730  0.08108108 -0.29729730 -0.13513514  0.18918919
 [259]  0.08108108  0.13513514  0.13513514 -0.13513514  0.18918919  0.02702703
 [265] -0.02702703  0.08108108 -0.29729730 -0.24324324 -0.02702703  0.35135135
 [271] -0.35135135  0.08108108 -0.18918919  0.02702703 -0.45945946  0.02702703
 [277] -0.18918919  0.35135135 -0.08108108 -0.18918919 -0.08108108  0.24324324
 [283]  0.13513514 -0.02702703 -0.02702703 -0.08108108 -0.35135135 -0.35135135
 [289]  0.08108108  0.35135135 -0.08108108  0.02702703 -0.56756757  0.02702703
 [295]  0.08108108 -0.18918919  0.08108108  0.13513514 -0.18918919 -0.08108108
 [301]  0.18918919  0.13513514  0.24324324 -0.02702703 -0.13513514  0.02702703
 [307] -0.13513514  0.45945946  0.08108108 -0.08108108  0.02702703  0.08108108
 [313]  0.08108108  0.02702703  0.18918919  0.13513514 -0.35135135  0.08108108
 [319]  0.40540541 -0.02702703 -0.18918919 -0.24324324 -0.29729730  0.02702703
 [325] -0.24324324 -0.18918919  0.02702703  0.13513514  0.24324324  0.13513514
 [331]  0.02702703 -0.24324324 -0.18918919 -0.08108108 -0.24324324 -0.67567568
 [337] -0.18918919  0.08108108  0.08108108  0.29729730  0.18918919 -0.02702703
 [343] -0.24324324 -0.29729730  0.18918919  0.18918919  0.40540541  0.13513514
 [349]  0.08108108 -0.08108108 -0.18918919 -0.02702703 -0.13513514 -0.29729730
 [355] -0.08108108  0.08108108  0.08108108  0.18918919  0.13513514  0.13513514
 [361]  0.08108108 -0.02702703 -0.29729730 -0.08108108  0.18918919  0.08108108
 [367]  0.08108108  0.18918919 -0.29729730 -0.08108108 -0.08108108 -0.02702703
 [373] -0.18918919 -0.02702703 -0.02702703 -0.18918919  0.08108108 -0.24324324
 [379] -0.02702703 -0.18918919  0.02702703  0.18918919 -0.08108108 -0.08108108
 [385] -0.08108108 -0.13513514  0.08108108  0.13513514  0.08108108  0.08108108
 [391]  0.13513514  0.02702703 -0.13513514 -0.02702703 -0.02702703  0.24324324
 [397]  0.13513514  0.29729730 -0.02702703  0.02702703  0.29729730 -0.40540541
 [403] -0.08108108 -0.02702703 -0.02702703 -0.08108108  0.13513514  0.13513514
 [409]  0.24324324  0.02702703 -0.02702703 -0.24324324  0.02702703 -0.24324324
 [415] -0.29729730  0.13513514  0.18918919 -0.29729730 -0.35135135  0.02702703
 [421]  0.13513514 -0.08108108 -0.08108108 -0.13513514 -0.13513514 -0.18918919
 [427]  0.02702703 -0.29729730 -0.13513514 -0.02702703 -0.13513514  0.02702703
 [433] -0.18918919 -0.13513514  0.02702703 -0.18918919 -0.02702703  0.29729730
 [439] -0.08108108  0.29729730  0.13513514 -0.02702703  0.13513514 -0.29729730
 [445] -0.35135135  0.18918919 -0.18918919 -0.29729730  0.45945946  0.08108108
 [451]  0.18918919 -0.13513514 -0.35135135 -0.18918919 -0.24324324 -0.24324324
 [457]  0.08108108  0.18918919  0.08108108  0.08108108 -0.13513514 -0.13513514
 [463] -0.08108108  0.18918919 -0.24324324 -0.02702703  0.18918919 -0.02702703
 [469]  0.13513514  0.08108108  0.18918919 -0.18918919  0.51351351 -0.24324324
 [475]  0.02702703  0.13513514 -0.08108108 -0.08108108 -0.02702703 -0.02702703
 [481] -0.18918919 -0.08108108  0.13513514 -0.18918919  0.24324324 -0.13513514
 [487]  0.02702703 -0.02702703  0.08108108  0.35135135 -0.13513514  0.08108108
 [493] -0.02702703 -0.13513514 -0.02702703 -0.35135135 -0.24324324 -0.08108108
 [499] -0.02702703  0.02702703  0.35135135  0.45945946  0.08108108 -0.35135135
 [505] -0.24324324 -0.08108108  0.02702703  0.13513514 -0.02702703 -0.08108108
 [511] -0.02702703  0.24324324 -0.08108108 -0.24324324 -0.18918919  0.29729730
 [517]  0.24324324  0.13513514  0.18918919 -0.02702703 -0.02702703  0.08108108
 [523]  0.18918919  0.08108108 -0.08108108 -0.13513514  0.35135135 -0.18918919
 [529] -0.08108108 -0.18918919  0.02702703  0.08108108  0.08108108  0.18918919
 [535]  0.08108108 -0.13513514  0.02702703  0.24324324 -0.18918919  0.18918919
 [541]  0.02702703  0.02702703 -0.08108108 -0.24324324  0.18918919  0.02702703
 [547] -0.18918919 -0.18918919  0.08108108 -0.08108108  0.13513514  0.40540541
 [553] -0.24324324 -0.08108108 -0.08108108  0.35135135  0.13513514 -0.18918919
 [559] -0.08108108  0.18918919  0.18918919 -0.18918919 -0.13513514 -0.02702703
 [565]  0.08108108  0.08108108  0.08108108  0.40540541 -0.18918919  0.24324324
 [571] -0.18918919  0.51351351 -0.29729730  0.08108108 -0.13513514 -0.08108108
 [577]  0.45945946  0.13513514 -0.45945946  0.08108108  0.29729730 -0.13513514
 [583]  0.02702703 -0.02702703  0.02702703 -0.02702703 -0.08108108 -0.56756757
 [589]  0.13513514  0.08108108  0.08108108 -0.18918919 -0.02702703  0.02702703
 [595] -0.29729730 -0.02702703 -0.13513514  0.02702703 -0.13513514  0.02702703
 [601]  0.24324324 -0.18918919  0.13513514 -0.24324324  0.02702703  0.02702703
 [607]  0.08108108 -0.02702703  0.08108108 -0.13513514  0.18918919  0.13513514
 [613] -0.02702703  0.18918919  0.13513514  0.24324324  0.08108108  0.02702703
 [619]  0.29729730  0.08108108 -0.29729730 -0.02702703  0.13513514  0.29729730
 [625]  0.18918919  0.24324324  0.13513514  0.29729730  0.40540541 -0.02702703
 [631] -0.24324324  0.18918919 -0.08108108 -0.08108108 -0.24324324  0.08108108
 [637] -0.08108108  0.08108108 -0.18918919 -0.02702703  0.13513514  0.08108108
 [643]  0.08108108  0.24324324 -0.08108108  0.29729730  0.08108108  0.29729730
 [649] -0.40540541  0.40540541  0.29729730 -0.13513514  0.08108108  0.13513514
 [655] -0.24324324 -0.18918919  0.24324324 -0.08108108  0.02702703  0.24324324
 [661] -0.13513514 -0.02702703  0.18918919  0.18918919 -0.08108108 -0.08108108
 [667] -0.18918919  0.18918919 -0.29729730  0.08108108  0.24324324  0.24324324
 [673]  0.13513514 -0.13513514  0.29729730  0.18918919 -0.29729730  0.02702703
 [679] -0.13513514 -0.13513514  0.13513514 -0.24324324 -0.24324324 -0.18918919
 [685] -0.40540541 -0.08108108 -0.24324324  0.02702703 -0.02702703 -0.02702703
 [691] -0.02702703 -0.40540541 -0.18918919  0.13513514  0.29729730 -0.18918919
 [697]  0.24324324 -0.08108108 -0.18918919 -0.40540541 -0.02702703  0.24324324
 [703]  0.18918919  0.08108108  0.08108108  0.13513514  0.02702703 -0.18918919
 [709]  0.02702703  0.02702703  0.08108108  0.02702703 -0.24324324 -0.13513514
 [715]  0.02702703  0.13513514  0.51351351  0.08108108 -0.40540541 -0.13513514
 [721] -0.02702703  0.08108108  0.51351351 -0.40540541 -0.08108108 -0.02702703
 [727]  0.45945946 -0.02702703 -0.02702703 -0.08108108 -0.24324324 -0.08108108
 [733]  0.08108108  0.35135135 -0.02702703 -0.35135135 -0.08108108  0.02702703
 [739]  0.02702703 -0.02702703 -0.35135135  0.18918919 -0.13513514 -0.51351351
 [745] -0.18918919  0.18918919 -0.08108108 -0.13513514  0.24324324 -0.24324324
 [751] -0.18918919 -0.08108108  0.35135135 -0.35135135 -0.08108108  0.18918919
 [757]  0.08108108 -0.24324324  0.08108108 -0.35135135  0.62162162  0.02702703
 [763] -0.02702703  0.24324324  0.08108108  0.35135135  0.24324324 -0.08108108
 [769] -0.08108108 -0.02702703  0.02702703 -0.72972973  0.02702703  0.08108108
 [775]  0.02702703  0.13513514 -0.02702703  0.24324324 -0.08108108  0.18918919
 [781] -0.02702703 -0.29729730 -0.45945946  0.02702703  0.13513514  0.24324324
 [787]  0.18918919  0.13513514  0.02702703 -0.24324324  0.24324324  0.02702703
 [793]  0.02702703  0.02702703  0.02702703 -0.13513514  0.08108108 -0.18918919
 [799]  0.08108108  0.35135135  0.08108108  0.08108108  0.29729730 -0.18918919
 [805]  0.24324324  0.40540541  0.18918919  0.18918919 -0.29729730  0.02702703
 [811]  0.08108108  0.02702703 -0.13513514  0.18918919  0.08108108 -0.45945946
 [817] -0.08108108  0.02702703 -0.08108108  0.08108108  0.45945946 -0.24324324
 [823] -0.24324324  0.13513514  0.18918919  0.18918919  0.08108108  0.18918919
 [829]  0.13513514 -0.29729730  0.13513514 -0.08108108 -0.35135135  0.08108108
 [835]  0.13513514 -0.02702703 -0.13513514 -0.35135135 -0.29729730 -0.35135135
 [841] -0.40540541 -0.35135135  0.08108108 -0.18918919 -0.29729730  0.08108108
 [847] -0.02702703 -0.13513514 -0.18918919  0.08108108 -0.02702703 -0.08108108
 [853] -0.08108108  0.02702703 -0.02702703  0.35135135  0.13513514  0.18918919
 [859]  0.08108108 -0.02702703  0.02702703  0.02702703 -0.08108108  0.13513514
 [865] -0.02702703 -0.24324324 -0.35135135 -0.35135135 -0.13513514  0.13513514
 [871]  0.13513514  0.24324324 -0.08108108 -0.13513514  0.02702703 -0.13513514
 [877] -0.29729730  0.45945946 -0.18918919  0.18918919 -0.08108108  0.13513514
 [883]  0.35135135  0.02702703  0.13513514 -0.13513514 -0.18918919 -0.18918919
 [889]  0.18918919 -0.02702703  0.29729730  0.08108108 -0.13513514 -0.18918919
 [895]  0.40540541 -0.02702703 -0.08108108  0.24324324 -0.18918919  0.02702703
 [901]  0.24324324  0.02702703 -0.29729730 -0.08108108 -0.08108108 -0.13513514
 [907]  0.02702703  0.02702703  0.02702703 -0.08108108  0.08108108 -0.29729730
 [913]  0.02702703  0.18918919  0.13513514 -0.13513514 -0.18918919 -0.29729730
 [919] -0.35135135 -0.02702703 -0.13513514 -0.18918919  0.08108108 -0.24324324
 [925]  0.08108108 -0.35135135 -0.13513514  0.13513514 -0.18918919  0.02702703
 [931]  0.35135135 -0.35135135 -0.02702703  0.13513514 -0.02702703 -0.08108108
 [937] -0.24324324 -0.08108108 -0.02702703  0.13513514  0.24324324 -0.24324324
 [943] -0.08108108 -0.29729730 -0.02702703  0.13513514  0.35135135  0.45945946
 [949] -0.18918919 -0.29729730  0.08108108 -0.02702703  0.24324324 -0.02702703
 [955]  0.18918919 -0.08108108 -0.08108108 -0.24324324  0.18918919 -0.24324324
 [961] -0.02702703 -0.08108108 -0.08108108 -0.08108108 -0.08108108 -0.13513514
 [967]  0.08108108  0.18918919 -0.18918919 -0.35135135  0.13513514  0.02702703
 [973]  0.13513514  0.24324324 -0.40540541 -0.29729730 -0.24324324  0.08108108
 [979] -0.24324324  0.02702703 -0.08108108 -0.08108108  0.24324324  0.08108108
 [985] -0.35135135 -0.13513514  0.29729730  0.08108108 -0.08108108 -0.02702703
 [991]  0.02702703  0.08108108  0.08108108  0.13513514 -0.24324324 -0.02702703
 [997] -0.02702703 -0.13513514  0.08108108 -0.40540541

mean(abs(stats) >= obs_stat)

[1] 0.16

Model-Based Inference

The Data

midterms_long

# A tibble: 148 × 5
   student format question section score
   <fct>   <chr>  <chr>    <fct>   <dbl>
 1 1       C      Q21      101         6
 2 1       S      Q23      101         6
 3 2       C      Q21      101         4
 4 2       S      Q23      101         6
 5 3       S      Q21      101         6
 6 3       C      Q23      101         6
 7 4       S      Q21      101         6
 8 4       C      Q23      101         4
 9 5       S      Q21      101         6
10 5       C      Q23      101         6
# ℹ 138 more rows

Model 1: Ignore (almost) all structure

m1 <- lm(score ~ format, data = midterms_long)
summary(m1)

Call:
lm(formula = score ~ format, data = midterms_long)

Residuals:
    Min      1Q  Median      3Q     Max 
-5.0811 -1.0811  0.9189  1.2162  1.2162 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   4.7838     0.1866  25.641   <2e-16 ***
formatS       0.2973     0.2638   1.127    0.262    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 1.605 on 146 degrees of freedom
Multiple R-squared:  0.008621,  Adjusted R-squared:  0.001831 
F-statistic:  1.27 on 1 and 146 DF,  p-value: 0.2617

Model 2: Add question and section effects

m2 <- lm(score ~ format + question + section, data = midterms_long)
summary(m2)

Call:
lm(formula = score ~ format + question + section, data = midterms_long)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.9474 -0.9474  0.7860  1.0833  1.3806 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   4.9167     0.2663  18.461   <2e-16 ***
formatS       0.2973     0.2645   1.124    0.263    
questionQ23  -0.2973     0.2645  -1.124    0.263    
section102    0.0307     0.2646   0.116    0.908    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 1.609 on 144 degrees of freedom
Multiple R-squared:  0.01733,   Adjusted R-squared:  -0.003138 
F-statistic: 0.8467 on 3 and 144 DF,  p-value: 0.4705

Model 3: Add student effects

m3 <- lm(score ~ format + question + section + student, data = midterms_long)
summary(m3)

Call:
lm(formula = score ~ format + question + section + student, data = midterms_long)

Residuals:
    Min      1Q  Median      3Q     Max 
-2.2973 -0.2973  0.0000  0.2973  2.2973 

Coefficients: (1 not defined because of singularities)
              Estimate Std. Error t value Pr(>|t|)    
(Intercept)  6.000e+00  8.590e-01   6.985 1.18e-09 ***
formatS      2.973e-01  1.971e-01   1.509  0.13578    
questionQ23 -2.973e-01  1.971e-01  -1.509  0.13578    
section102  -2.862e-15  1.199e+00   0.000  1.00000    
student2    -1.000e+00  1.199e+00  -0.834  0.40692    
student3    -9.658e-16  1.199e+00   0.000  1.00000    
student4    -1.000e+00  1.199e+00  -0.834  0.40692    
student5    -1.739e-15  1.199e+00   0.000  1.00000    
student6    -1.000e+00  1.199e+00  -0.834  0.40692    
student7    -1.307e-15  1.199e+00   0.000  1.00000    
student8    -2.000e+00  1.199e+00  -1.668  0.09957 .  
student9    -3.795e-16  1.199e+00   0.000  1.00000    
student10   -3.000e+00  1.199e+00  -2.503  0.01460 *  
student11   -1.166e-15  1.199e+00   0.000  1.00000    
student12   -6.805e-16  1.199e+00   0.000  1.00000    
student13   -3.000e+00  1.199e+00  -2.503  0.01460 *  
student14   -1.000e+00  1.199e+00  -0.834  0.40692    
student15   -8.830e-16  1.199e+00   0.000  1.00000    
student16   -1.000e+00  1.199e+00  -0.834  0.40692    
student17   -1.000e+00  1.199e+00  -0.834  0.40692    
student18   -1.329e-15  1.199e+00   0.000  1.00000    
student19   -2.000e+00  1.199e+00  -1.668  0.09957 .  
student20   -2.868e-15  1.199e+00   0.000  1.00000    
student21   -3.000e+00  1.199e+00  -2.503  0.01460 *  
student22   -5.878e-16  1.199e+00   0.000  1.00000    
student23   -2.000e+00  1.199e+00  -1.668  0.09957 .  
student24   -7.210e-16  1.199e+00   0.000  1.00000    
student25   -1.430e-15  1.199e+00   0.000  1.00000    
student26   -4.000e+00  1.199e+00  -3.337  0.00134 ** 
student27   -1.000e+00  1.199e+00  -0.834  0.40692    
student28   -2.000e+00  1.199e+00  -1.668  0.09957 .  
student29   -1.000e+00  1.199e+00  -0.834  0.40692    
student30   -1.000e+00  1.199e+00  -0.834  0.40692    
student31   -2.000e+00  1.199e+00  -1.668  0.09957 .  
student32   -2.000e+00  1.199e+00  -1.668  0.09957 .  
student33   -4.000e+00  1.199e+00  -3.337  0.00134 ** 
student34   -7.627e-16  1.199e+00   0.000  1.00000    
student35   -3.846e-16  1.199e+00   0.000  1.00000    
student36   -1.000e+00  1.199e+00  -0.834  0.40692    
student37    3.978e-16  1.199e+00   0.000  1.00000    
student38   -4.000e+00  1.199e+00  -3.337  0.00134 ** 
student39    2.214e-15  1.199e+00   0.000  1.00000    
student40    7.904e-16  1.199e+00   0.000  1.00000    
student41    2.331e-15  1.199e+00   0.000  1.00000    
student42   -1.000e+00  1.199e+00  -0.834  0.40692    
student43   -1.000e+00  1.199e+00  -0.834  0.40692    
student44    1.955e-15  1.199e+00   0.000  1.00000    
student45   -3.000e+00  1.199e+00  -2.503  0.01460 *  
student46   -1.000e+00  1.199e+00  -0.834  0.40692    
student47   -3.000e+00  1.199e+00  -2.503  0.01460 *  
student48   -1.000e+00  1.199e+00  -0.834  0.40692    
student49   -1.000e+00  1.199e+00  -0.834  0.40692    
student50    2.473e-15  1.199e+00   0.000  1.00000    
student51    3.473e-15  1.199e+00   0.000  1.00000    
student52   -1.000e+00  1.199e+00  -0.834  0.40692    
student53   -1.000e+00  1.199e+00  -0.834  0.40692    
student54    1.729e-15  1.199e+00   0.000  1.00000    
student55    3.237e-15  1.199e+00   0.000  1.00000    
student56    2.650e-15  1.199e+00   0.000  1.00000    
student57    1.871e-15  1.199e+00   0.000  1.00000    
student58    8.195e-16  1.199e+00   0.000  1.00000    
student59    1.459e-15  1.199e+00   0.000  1.00000    
student60   -3.000e+00  1.199e+00  -2.503  0.01460 *  
student61   -1.000e+00  1.199e+00  -0.834  0.40692    
student62   -1.000e+00  1.199e+00  -0.834  0.40692    
student63   -4.000e+00  1.199e+00  -3.337  0.00134 ** 
student64   -1.000e+00  1.199e+00  -0.834  0.40692    
student65   -1.000e+00  1.199e+00  -0.834  0.40692    
student66    2.729e-15  1.199e+00   0.000  1.00000    
student67    1.846e-15  1.199e+00   0.000  1.00000    
student68   -6.000e+00  1.199e+00  -5.005 3.82e-06 ***
student69   -2.000e+00  1.199e+00  -1.668  0.09957 .  
student70    8.497e-16  1.199e+00   0.000  1.00000    
student71    1.994e-15  1.199e+00   0.000  1.00000    
student72   -4.000e+00  1.199e+00  -3.337  0.00134 ** 
student73    1.876e-15  1.199e+00   0.000  1.00000    
student74           NA         NA      NA       NA    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 1.199 on 72 degrees of freedom
Multiple R-squared:  0.7273,    Adjusted R-squared:  0.4431 
F-statistic:  2.56 on 75 and 72 DF,  p-value: 4.233e-05

Model 4: Student as random effect

library(lme4)
m4 <- lmer(score ~ format + question + section + (1 | student), data = midterms_long)
summary(m4)

Linear mixed model fit by REML ['lmerMod']
Formula: score ~ format + question + section + (1 | student)
   Data: midterms_long

REML criterion at convergence: 545.6

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.3651 -0.5693  0.3374  0.5854  1.6128 

Random effects:
 Groups   Name        Variance Std.Dev.
 student  (Intercept) 1.152    1.073   
 Residual             1.437    1.199   
Number of obs: 148, groups:  student, 74

Fixed effects:
            Estimate Std. Error t value
(Intercept)   4.9167     0.2671  18.405
formatS       0.2973     0.1971   1.509
questionQ23  -0.2973     0.1971  -1.509
section102    0.0307     0.3180   0.097

Correlation of Fixed Effects:
            (Intr) formtS qstQ23
formatS     -0.369              
questionQ23 -0.369  0.000       
section102  -0.611  0.000  0.000

Model 4: Student as random effect (continued)

library(lmerTest)
m4 <- lmer(score ~ format + question + section + (1 | student), data = midterms_long)
summary(m4)

Linear mixed model fit by REML. t-tests use Satterthwaite's method [
lmerModLmerTest]
Formula: score ~ format + question + section + (1 | student)
   Data: midterms_long

REML criterion at convergence: 545.6

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.3651 -0.5693  0.3374  0.5854  1.6128 

Random effects:
 Groups   Name        Variance Std.Dev.
 student  (Intercept) 1.152    1.073   
 Residual             1.437    1.199   
Number of obs: 148, groups:  student, 74

Fixed effects:
            Estimate Std. Error       df t value Pr(>|t|)    
(Intercept)   4.9167     0.2671 119.2292  18.405   <2e-16 ***
formatS       0.2973     0.1971  72.0000   1.509    0.136    
questionQ23  -0.2973     0.1971  72.0000  -1.509    0.136    
section102    0.0307     0.3180  72.0000   0.097    0.923    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Correlation of Fixed Effects:
            (Intr) formtS qstQ23
formatS     -0.369              
questionQ23 -0.369  0.000       
section102  -0.611  0.000  0.000