I don’t have time for round two, but you should compare the paper to the prereg for this one. www.nature.com/articles/s41... - ThreadSky

jbakcoleman.bsky.social • 65 days ago

I don’t have time for round two, but you should compare the paper to the prereg for this one.

https://www.nature.com/articles/s41562-024-01961-1

Comments

devezer.bsky.social•65 days ago

Not again!

Also, with apologies, I have to ask: Why would anyone need to urgently "predict" what psych manipulations would work during COVID (or another pandemic) without studying it properly? I'm scratching my head about the motivation for these hypotheses and studies.

jbakcoleman.bsky.social•65 days ago

I spent all of five minutes on this one, here's one of two preregistered hypotheses as reported in the text. Claims made based on TOST.

jbakcoleman.bsky.social•65 days ago

Next step, check the preregistration... absolutely no mention of TOST and instead the inference criteria is hinging on whether there's overlap of the 95% bootstrapped confidence interval. So, for the primary (two) preregistered hypotheses there is post-hoc tost swapped out.

https://osf.io/5kcny

jbakcoleman.bsky.social•65 days ago

(Updating this bit of the thread to clarify some things I missed).

If you read the text, it's not clear that the claims are made based on exploratory TOST throughout. The significance value associated with exploratory TOST seems to make claims in the abstract, swapping out that inference crit.

jbakcoleman.bsky.social•65 days ago

The authors instead just note that they used TOST, but don't indicate this was post hoc, not a priori. It's subtle but it leads to distinct affirmative claims in the abstract that are different from what you'd get if you just took the prereg.

jbakcoleman.bsky.social•65 days ago

There is a table I had missed on first read of what was and wasn't preregistered (hypotheses, not stats). This is a big improvement over Protzko, but the text makes no such distinctions and doesn't indicate the post hoc TOST.

jbakcoleman.bsky.social•65 days ago

There's still the co-author competing interests of one reviewer (Wagenmakers) from a paper published last month, and the issues with Lakens reviewing after (during?) a bout of reviewer misconduct evaluating concerns on Protzko.

https://www.chronicle.com/article/this-study-was-hailed-as-a-win-for-science-reform-now-its-being-retracted

klonskylab.bsky.social•65 days ago

Preregistration mandates have done a great job teaching the field to put the word “preregistration” in our articles. Mission accomplished.

mehr.nz•65 days ago

The calculus is hilarious now. Even when a prereg is totally unnecessary and adds no evidentiary value may as well go for it for extra credit with some reviewers

mehr.nz•65 days ago

I actually mentioned this to some grad students on a job interview a few yrs back — had a curr bio paper where we prereg'd after having run the same experiment in piloting so knew the result already, just did it to make review easier. The students were like 👀 hahahh

klonskylab.bsky.social•65 days ago

I mean, it’s good to replicate pilot findings, so there’s that!!

steveharoz.com•64 days ago

Isn't that a good practice though? Pilot, pilot, pilot until you have an experiment with all the kinks ironed out and a decent idea of the effect. Then run the full thing.

Seems like a waste of resources to NOT do that.

michaeljkane.bsky.social•64 days ago

The peer review reports are interesting. Of 3 reviewers, Lakens is the only 1 who notes checking the prereg & data. Shocking that 2 reviewers on a metascience ms like this didn’t rigorously review these (or if they did, explicitly confirm this to Editor).

jbakcoleman.bsky.social•64 days ago

Despite being less than fully enthusiastic about preregistration, I *always* check and quite often find some pretty substantial deviations.

michaeljkane.bsky.social•64 days ago

Me too. And when data are available the least I do is open the files and see if there’s an adequate codebook.

Comments

Posting Rules

Reply