Another large-scale replication study that doesn't improve our understanding of anything. I know this is harsh but at some point we need to start asking questions that matter and focus on study quality and theoretical models instead of automatically replicating results in flashy mega-studies.
Comments
This is just an irresponsible and an factually wrong thing to say. At least some of these authors have read our work. They're completely and conveniently ignoring the theory.
It's also interesting that MTurk is not systematically worse than other mechanisms.
My intuition is you'd want to do replications in whatever part of the theory space will give you the most information but @devezer.bsky.social probably has a better idea
replication that can adjudicate between two theories where you don't have high certainty which is better? = higher new information
(I'm just naively analogizing this to multi-armed bandit)
But yes, re: theory, see also my other reply. Theory has to be part of the picture as well.
It's a study about one aspect of the problem, not a set of guidelines presented to The Replication Council.
But this study is not really about any relevant aspect of the problem. It's ignoring the whole problem. I really recommend all our papers on this. It's not like we don't know better.
But in areas of science where theory is very weak (like this one!) I think sanity checks can be helpful.