New paper with @dianamejordan.bsky.social and sky-less Trent Ollerenshaw! We provide large-N tests of repeated measure designs in survey experiments, showing that they slightly attenuate ATEs relative to post-only designs, but provide large gains to precision. Thread below. Preprint: osf.io/q6czp - ThreadSky

atrexler.com • 36 days ago

New paper with @dianamejordan.bsky.social and sky-less Trent Ollerenshaw! We provide large-N tests of repeated measure designs in survey experiments, showing that they slightly attenuate ATEs relative to post-only designs, but provide large gains to precision. Thread below.

Preprint: https://osf.io/q6czp

Comments

atrexler.com•36 days ago

We partially replicate & extend Clifford, Sheagley, & Piston (2021). They find no evidence of design effects in 6 studies & recommend repeated measure designs as a default design. We field 6 experiments in 3 large samples (18 studies, N=79k obs), expanding the evidence base.
https://doi.org/10.1017/S0003055421000241

atrexler.com•36 days ago

In contrast to CSP, we observe ~20% attenuation of ATEs under repeated measure designs. But the RM design offers much better precision, shrinking the SEs by ~50%, making hypothesis testing easier despite slight bias. Results hold across multiple sampling designs & experiment types.

atrexler.com•36 days ago

We also find limited evidence that ATE attenuation varies by how far apart repeated measure items are placed, meaning this design is suitable even for very short surveys/modules. Similarly, we find no evidence that respondent attention/recall alters the design effect.

atrexler.com•36 days ago

Caveats! Because attenuation of the ATE is non-zero, some cases where (less precise) post-only is preferred. Large-N studies, large treatment effects, and (perhaps) item/topic sensitivity push the attenuation-precision tradeoff towards post-only design.

Thanks for reading! Full paper: https://osf.io/q6czp

astrezh.bsky.social•32 days ago

Out of curiosity - is the pre-treatment adjustment for Studies 1-3 done with the fully-interacted-with-treatment Lin regression? Theoretically, there should be no bias in either case since treatment is randomized, but you can induce some finite sample bias with the OLS specification.

astrezh.bsky.social•32 days ago

Ah okay - reading more closely - the story/concern is priming by eliciting pre- measures.

atrexler.com•32 days ago

Yes, and/or consistency/demand pressures. We do not use Lin's adjustment currently (neither did CSP). The experiments are all 2-arm and our samples are 4,000+, so the small-sample bias of adjustment should be nearly zero. But Lin's is def good practice for small-N experiments w/RM.

astrezh.bsky.social•32 days ago

Gotcha! You might still get some precision improvement with the interaction, but yeah - 4k seems enough. I got confused by the use of “bias” since technically these are two different estimands, but the “design effects” framing is clearer. Impressive cross-randomization setup over 6 designs!

atrexler.com•32 days ago

Yes, true! Though somewhat orthogonal to our main purpose of estimating the design effect---for most experiments the post-only regression also includes some (non-interacted) covariate adjustment, so there would be precision gains there too. (We follow each original study we were replicating.)

atrexler.com•32 days ago

And thanks re: bias versus design effect framing. The draft has gone back and forth a bit on "bias" in the sense of the traditional concerns with RM versus the technical meaning; I agree that "design effect" is clearer so we have tried to use that language more.

astrezh.bsky.social•32 days ago

Right - I got confused on first read thinking the comparison was “post only-v-pre/post when pre- is asked” and not “what’s the design effect of asking ‘pre’ in the first place?”

andydebarros.bsky.social•32 days ago

https://documents.worldbank.org/en/publication/documents-reports/documentdetail/333121468147852171/beyond-baseline-and-follow-up-the-case-for-more-t-in-experiments

Comments

Posting Rules

Reply