Keep thinking about this. Why don't we have a similar "total experiment error" framework as a default setting for inference from social science experiments? Oftentimes we pretend all we need to account for is sampling error even though we know our measurements, treatments, etc. are full of error. - ThreadSky

devezer.bsky.social • 138 days ago

Keep thinking about this. Why don't we have a similar "total experiment error" framework as a default setting for inference from social science experiments? Oftentimes we pretend all we need to account for is sampling error even though we know our measurements, treatments, etc. are full of error.

Reposted from Andrew Mercer

So in my field, we have this idea of “total survey error” where we enumerate all the many factors that can cause your estimate to deviate from the true population value. Most of them can’t be measured or quantified except in rare situations. But we tend to approach estimates with this in mind.

Comments

jacob-long.com•137 days ago

I have a paper that was originally written to argue TSE was transportable to other empirical methods. Reviewed made me downplay TSE a bit but it's still there: https://www.tandfonline.com/doi/full/10.1080/23808985.2021.1979421

devezer.bsky.social•137 days ago

Cool, thanks for sharing!

devezer.bsky.social•138 days ago

I both want to write something about this and feel like there's gotta be much more qualified people for the job. Then again, where are they and why haven't they already written about it?

moreanonhandle.bsky.social•137 days ago

The revealing one for this in my area was realizing that seemingly ~nobody had ever thought about error in Decennial Census aggregate data until the Bureau introduced differential privacy, at which point everyone freaked out.

moreanonhandle.bsky.social•137 days ago

I don’t think differential privacy was actually necessary, but it was very clear that many data users assumed the Bureau was just tallying and reporting before

danhicks.bsky.social•138 days ago

One of the odd things about Deborah Mayo's career is that /Error and the Growth of Experimental Knowledge/ is very good on this, and then the later stuff/uptake is much more narrowly focused on p-values

devezer.bsky.social•138 days ago

It is odd but perhaps she followed the path of least resistance.

shermandorn.com•138 days ago

If you think about the right side, that corresponds VERY roughly to validity. One of the things I take away from Kane (1992) is that validity is an argument, NOT a term of art. I agree that we can think about global sources of error usefully, as long as we know it's incomplete.

devezer.bsky.social•138 days ago

Validity is sometimes used as a weasel word to hand wave at a lot of modeling decisions we choose to bypass. We gotta do much better than thinking about it as an argument, and start properly accounting for it in our models and inference.

yvonnezlam.bsky.social•138 days ago

This sounds to me like a Safety II-ish situation. I hate gaslighting, so not accounting for known sources of error would bother me a lot. And yet, people might find the thought of dealing with *all* the errors overwhelming.

https://www.england.nhs.uk/signuptosafety/wp-content/uploads/sites/16/2015/10/safety-1-safety-2-whte-papr.pdf

yvonnezlam.bsky.social•138 days ago

I don't always agree with Safety II or how people talk about it or try to use it, but I do think there's something to be said for coming to terms with the idea that at some point, things come down to human judgement which might be wrong.

https://how.complexsystems.fail/

yvonnezlam.bsky.social•138 days ago

I guess what I am trying to say is that perhaps people are trying to build an overly simple system to immunize against failure, scofflaws, etc., and maybe other approaches are worth trying out? (note to the audience: I'm not a metascientist, or even a scientist, only an interested layperson.)

devezer.bsky.social•138 days ago

That's a very interesting perspective. I'll need to think about it some more to reconcile with how I view things. Part of the insistence on simplicity could be that but there's also a considerable amount of wishful thinking and confirmation bias involved. I mean dealing with errors is just as

devezer.bsky.social•138 days ago

difficult in other fields yet they do not feign total ignorance. It's even not the case that we do not know the errors in some cases. We do and we knowingly choose to ignore because it's below an ad hoc threshold so we conveniently assume it's zero instead.

grimalkina.bsky.social•138 days ago

I do wonder (like Yvonne also defer to your metascience expertise!) if subfields and their areas of concern can feel substantively different here. I actually felt I spent a HUGE amount of time in the applied fields of psych talking holistically about reducing errors in every step of the process

yvonnezlam.bsky.social•138 days ago

So it's more like a social convention that we don't talk about *those* errors?

michael.is-post.ing•138 days ago

when I do bootstrap sampling, after sampling data points I like to also sample the values according to an error distribution if it’s known. seems like a simple way to incorporate error but don’t think I’ve seen literature on it!

devezer.bsky.social•138 days ago

It's interesting how there's gotta be some authoritative literature on these things yet...

awmercer.bsky.social•138 days ago

It is **such** a helpful way to approach inference, but I have not seen anything similar in other fields.

devezer.bsky.social•138 days ago

In experimental social sciences we have a way of creating myths about valid inference and convincing ourselves that they're true. It's not just helpful imho, it's a necessary perspective because of how ignored errors accumulate like an avalanche at different levels.

awmercer.bsky.social•138 days ago

I think people usually assume they’re negligible (in my field too), but you stack enough negligibles on top of each other and they start to add up.

devezer.bsky.social•138 days ago

Exactly! They add up within a single study, then they add up across a series of studies within a paper, and then they continue to add up across a whole literature. Yet scientists happily continue to assume that their frequentist "error controls" work because they didn't p-hack!

boryslaw.bsky.social•137 days ago

See here, on page 230
https://czasopisma.uwm.edu.pl/index.php/pp/article/view/9731/7171

Comments

Posting Rules

Reply