New preprint: When and How to Deviate from a Preregistration osf.io/preprints/ps... I hope this helps researchers to reflect on the consequence of deviations from preregistrations by evaluating the test's severity and the validity of the inference. - ThreadSky

lakens.bsky.social • 438 days ago

New preprint: When and How to Deviate from a Preregistration https://osf.io/preprints/psyarxiv/ha29k I hope this helps researchers to reflect on the consequence of deviations from preregistrations by evaluating the test's severity and the validity of the inference.

Comments

paulallen.ca•435 days ago

Another principled, practical contribution.

lakens.bsky.social•435 days ago

🙏

cdsamii.bsky.social•436 days ago

Great to see this. Is the concept of “validity” here derived from the usual Shadish, Cook, and Campbell definitions? It wasn’t clear to me in the draft.

lakens.bsky.social•436 days ago

Yes, that is why I cite them :)

cdsamii.bsky.social•436 days ago

Right, they are cited on line 126, but the concept and even Table 1 are already pretty deep in working with the concept prior to that. A clear definition or reference to one earlier would help, perhaps right at line 98.

cdsamii.bsky.social•436 days ago

I’m seeing an analogy to measurement theory, and its concepts reliability and validity. But those definitions relate to the formal bias^2 + variance decomposition of MSE. It’s not clear such a decomposition works here, although maybe it does. Just a thought.

lakens.bsky.social•436 days ago

Thanks, I will move the citation up to line 98. I guess I assume this is such a basic concept everyone understands it (I teach it in intro psych) but I should clearly not assume this.

cdsamii.bsky.social•436 days ago

It’s one of those things where you have a plain language meaning and then a reference-specific meaning, and so it’s good to distinguish. Thanks for engaging.

lakens.bsky.social•438 days ago

This preprint was in part inspired by podcast episodes by Quantitude (https://quantitudepod.org/s3e07-in-defense-of-researcher-degrees-of-freedom/) and Within & Between (https://podcasters.spotify.com/pod/show/withinandbetween/episodes/S4E11-Revisiting-Preregistration-and-Registered-Reports-e2cserb). These made me realize there should be better (principled!) explanations of when and how to deviate from preregistrations.

steveharoz.com•438 days ago

This is great Daniel. Have you considered what authors should do when a paper has so many deviations that practically none of the preregistration remains? Is it valid to still describe the study as preregistered?

lakens.bsky.social•438 days ago

It would be intriguing to see an example! In essence, you would then have a paper where all deviations (if they follow my guidelines) would have a very careful reflection on the severity of the test and validity of the inference. Still better than most papers I see published ;)

lakens.bsky.social•438 days ago

I briefly state how I think terms such as 'confirmatory' and 'exploratory' are a bit distracting - and the same goes for 'preregistered' - maybe I should add this! What should be the main focus is how severe the test is, which is a continuous evaluation.

steveharoz.com•438 days ago

I'll send you an email with an example, and some of my notes

Comments

Posting Rules

Reply