For the last 10 years there has been a LOT of confusion about what quantities survey experiments can estimate. Often they are just measurement devices and do not try to estimate any effects. If unsure, a good diagnostic question is: *could* I have measured both conditions for the same subject? - ThreadSky

macartan.bsky.social • 82 days ago

For the last 10 years there has been a LOT of confusion about what quantities survey experiments can estimate.

Often they are just measurement devices and do not try to estimate any effects.

If unsure, a good diagnostic question is: *could* I have measured both conditions for the same subject?

Comments

nmalhotra.bsky.social•82 days ago

I refer to this as the Kramer moviefone problem: https://www.youtube.com/watch?v=L2DqcXeGTyc (The master of using survey experiments as measurement is Mike Tomz)

macartan.bsky.social•82 days ago

Or: am I doing this because I really face the fundamental problem of causal inference? Or is it rather for efficiency or to address measurement issues?

The effect / measurement distinction is not about external validity but about the nature of the estimand

mikedenly.com•82 days ago

Might you be able to clarify more or provide some example(s)? Is your point about the control that survey experiments provide researchers? If so, would your argument also apply in some way to lab experiments?

macartan.bsky.social•82 days ago

Hi Mike it’s not about control it’s about whether the aim is to infer an effect or to measure sthg. Sometimes treatments are delivered via surveys. But v often not

Compare

1. Interest in the effect on candidates from doing X
2. Interest in the effect on voters about learnng X about a candidate…

macartan.bsky.social•82 days ago

3. Interest in whether voters evaluate candidates with feature X differently

1. Might be done with a field experiment
2. … by an information experiment embedded in a survey
3. By a survey experiment (or perhaps just a survey questin)

For 3 you might simultan’ly ask about candidates w/ and w/out X

mikedenly.com•82 days ago

Thank you so much, Macartan--this is very helpful! At least the closest that I have seen to your argument is this piece (https://doi.org/10.1093/oxfordhb/9780192868282.013.81) by @tompepinsky.bsky.social

scottfabramson.bsky.social•82 days ago

I believe that many political scientists think the world is just a collection of ATEs and that their job is to collect them all, Pokemon style. I don't know how we learn about politics this way.

macartan.bsky.social•74 days ago

As we put it here, too often political scientists are doing parachute experiments from grounded aircraft without even knowing it

https://macartan.github.io/assets/pdf/papers/2019_field_experiment.pdf with anna Wilke

jknowles.bsky.social•74 days ago

Such a good point. Lots of social sciences are chasing ATEs at the expense of other ways of learning.

klauspforr.bsky.social•64 days ago

📌

klauspforr.bsky.social•64 days ago

The same goes holds for many cases where we look at revealed preferences instead of stated preferences

Comments

Posting Rules

Reply