Great thread. I've come to view many-analyst study designs as crisis-mining. They're commonly interpreted in a way to cause alarm for a result they're designed to generate. Metascience needs to move beyond the crisis framing and stop stripping science from its context. Domain knowledge matters. - ThreadSky

devezer.bsky.social • 4 hours ago

Great thread. I've come to view many-analyst study designs as crisis-mining. They're commonly interpreted in a way to cause alarm for a result they're designed to generate. Metascience needs to move beyond the crisis framing and stop stripping science from its context. Domain knowledge matters.

Reposted from Joe Bak-Coleman

I spoke with @cathleenogrady.bsky.social for this story on the many analysts paper in ecology. To me the concerning thing is the interpretation, not the (expected) finding.

www.science.org/content/arti...

Comments

devezer.bsky.social•4 hours ago

Relatedly, a thread I wrote about many-analyst studies before. How many more of them will it take for us to learn not to feign surprise at model misspecification leading to varying inferences?

thelasttheorist.bsky.social•3 hours ago

I'd read a paper about 248 biologists independently baking cookies from the same recipe.

devezer.bsky.social•3 hours ago

Wouldn't volunteer to taste the cookies tbh 👀

thelasttheorist.bsky.social•3 hours ago

We would finally have a way to discern good science though.

pawellenartowicz.bsky.social•3 hours ago

In this case it's the same shelf with ingredients (data) to bake cookies (research question). The recipe (like the statistical model) is to be invented.

devezer.bsky.social•3 hours ago

Don't forget to act surprised when the cookies don't resemble each other!

pawellenartowicz.bsky.social•3 hours ago

At least they achieve high reproducibility (which is the cornerstone of science!).

thelasttheorist.bsky.social•3 hours ago

And they're very unlikely to have occurred randomly from the ingredients.

crispapoll.bsky.social•4 hours ago

I think popular media reporting on these studies is harmful and seemingly just crisis baiting. But the studies themselves can be useful for individual fields that don't have strong stats/modeling training but rely on those methods anyway. Sometimes people don't believe it until they see it

devezer.bsky.social•4 hours ago

I tend to think it creates an unreal situation where a lot of non-experts are given some dataset about a research question they didn't ask. I am hesitant to believe this is an accurate description of what's going on in these fields in general.

crispapoll.bsky.social•3 hours ago

That's a great point. I shouldn't pretend to make general claims about *these* studies. I'm more familiar with these *kinds* of studies in the context of model intercomparisons where the research question is actually shared across modeling groups

Comments

Posting Rules

Reply