At today's #2024APPAM Rossi Award Lecture, I’ll argue the substantial impact of open-science practices has been far too narrow. It’s time to adapt & adopt open-science practices—especially preregistration—for quasi-experimental designs (QEDs). #EconSky #EduSky #APPAM2024 @appam.bsky.social 🧵… - ThreadSky

tomdee.bsky.social • 169 days ago

At today's #2024APPAM Rossi Award Lecture, I’ll argue the substantial impact of open-science practices has been far too narrow.

It’s time to adapt & adopt open-science practices—especially
preregistration—for quasi-experimental designs (QEDs).

#EconSky #EduSky #APPAM2024 @appam.bsky.social

🧵…

Comments

ryanreedhill.bsky.social•169 days ago

I’m on board with the benefits, but verification of true pre-reg seems to be an obstacle to implementation. At least with an RCT, an editor could plausibly verify the study was pre-reg before the study went into the field. Hard to do that with analyses on existing datasets

ryanreedhill.bsky.social•169 days ago

And I’m not trying to be a cynic. I think it’s useful to think about scientist incentives when designing science policy. You’re working on an important problem!

tomdee.bsky.social•169 days ago

👍 Christensen & Miguel (2018) make the verification critique too.

My rejoinder is
- Evidence & commentary suggests the vast majority of p-hacking is due to motivated reasoning+HARKing, not fraud
- It lets perfect be the enemy of "better"
- It holds QEDs to an anti-fraud standard not all RCTs meet

anbrown.bsky.social•168 days ago

We made these arguments a decade ago when we established the Registry for International Development Impact Evaluations at #3ie with the help of @rand.org. #RIDIE accepts quasi-experimental and experimental designs focused on international development. https://ridie.3ieimpact.org/

tomdee.bsky.social•168 days ago

👍QED preregistration is a feasible response to a clear & critical problem. Registries already exist (e.g., https://osf.io/registries, https://sreereg.icpsr.umich.edu/sreereg/, #RIDIE) & norms for ex-ante data exploration can be formed

The immediate challenge is broad awareness of the problem & motivation to change

tomdee.bsky.social•169 days ago

The core motivation for preregistration is that researcher discretion + motivated reasoning + hindsight bias ➡️ p-hacking

But QED researchers have much more discretion than experimentalists! Especially now that preregistration is the RCT norm

Two types of empirical evidence underscore the problem

tomdee.bsky.social•169 days ago

Multiple “many analyst” studies show a wide variance in results when different research teams answer the same question with the same data.

For example, what did 73 different research teams conclude about the impact of immigration on public support for social policies (Breznau et al., 2022, PNAS)?
😳

tomdee.bsky.social•169 days ago

And, second, the distribution of test-statistics in some QEDs shows a particularly suspicious distribution of missingness just before statistical significance and bunching at statistical significance (Brodeur et al., 2022)

tomdee.bsky.social•169 days ago

QED preregistration is a highly feasible way to improve the fundamental epistemic validity of our work. And it won’t inhibit discovery!

Instead, it will promote enhanced & useful transparency about predicted “confirmatory” vs exploratory findings.

Slides @ https://dee.stanford.edu

Comments welcome!

andydebarros.bsky.social•169 days ago

+1! Adding: I found this to be super practical for a research practice partnership. Everyone agrees on what we're looking at (and how). Here is our registration of a prospective dif-in-dif with Morocco's Ministry of Ed https://osf.io/zg5ry/

tomdee.bsky.social•169 days ago

I agree! In my experience, good RPPs dovetail nicely with preregistration because researchers and practitioners have extensive and mutualistic conversations around goals, the theory of change, and aligned measurement before digging into the data.

Comments

Posting Rules

Reply