This is a really fascinating paper highlighting how open science reforms, motivated by replicability, are not evidence-based. "It is striking that researchers, policymakers, journals and others are often content to implement large-scale changes without rigorous exploration of their efficacy" - ThreadSky

jbakcoleman.bsky.social • 214 days ago

This is a really fascinating paper highlighting how open science reforms, motivated by replicability, are not evidence-based.

"It is striking that researchers, policymakers, journals and others are often content to implement large-scale changes without rigorous exploration of their efficacy"

Reposted from

Open Science interventions to improve reproducibility and replicability of research: a scoping review preprint: http://osf.io/a8rmu/

Comments

devezer.bsky.social•214 days ago

I am kinda incredulous that they have not cited our formal methodology paper because come on!! Huh.

joachim.cidlab.com•214 days ago

Hard to make a case there's no formal papers if you go citing formal papers

devezer.bsky.social•214 days ago

Haha indeed! Hadn't thought of it that way but yeah, evidence-based advice is there. That it's not heeded is a different issue.

devezer.bsky.social•214 days ago

Especially since this is meant as a scoping review, however, the exclusion of relevant literature seems even more egregious to me tbh. Not a single citation to Mark Rubin's work or any of our work or a number of other authors/papers that brought up similar issues before???

joachim.cidlab.com•214 days ago

At least from the title it seems like they're focusing on empirical evaluations. I'm not all that concerned, really -- our paper is findable to people who are interested

devezer.bsky.social•214 days ago

Yeah you're right. I'm a little too sensitive about this and trying not to be.

jbakcoleman.bsky.social•214 days ago

The replication crisis and reforms began in earnest 10-13 years ago, and it's somewhat compelling/telling that the authors can't find any rigorous literature suggesting that adopting these reforms would address replication rates motivating them in the first place.

jbakcoleman.bsky.social•214 days ago

It should strike us all as strange that these are not evidence based interventions! There may be normative, social or ethical reasons to adopt aspects of the interventions but the sales pitch that these are necessary or required for replicability is not scientifically established.

jbakcoleman.bsky.social•214 days ago

Which leads to a much more difficult to answer question… why not? If they have their ostensible power in improving replicability there should be glowing signal of improved replicability over time. Boyce in 2023 looked and no dice.

https://osf.io/preprints/psyarxiv/dpyn6

jbakcoleman.bsky.social•214 days ago

We also might expect that there are differences in the distribution of p-values between rejected and accepted research that these interventions are intended to work on.

No dice there either.

jbakcoleman.bsky.social•214 days ago

Or that preregistration should alter the distribution of p-values. No so much there either!

https://www.econstor.eu/bitstream/10419/281136/1/I4R-DP101.pdf

jbakcoleman.bsky.social•214 days ago

Of course it'd be great to see an RCT of these interventions, or even something like OSC 2015 on a newer year with prereg/not-prereg as a predictor of replicability. It wouldn't be fully causal but it should at least point in the right direction.

saladcookies.bsky.social•214 days ago

Is improving replicability a common argument for open access publications? I don't know that I've seen that before now

jbakcoleman.bsky.social•214 days ago

Not for Open Access (which predates the replication crisis quite a bit) but for "open science" such as sharing data/code, preregistration etc...

saladcookies.bsky.social•214 days ago

Ahhhh, I see, my mistake!

jbakcoleman.bsky.social•214 days ago

Common point of confusion!

penders.bsky.social•213 days ago

Prescriptions on how to do science are political, they are policies. When policy decisions are required, often the evidence simply isn't there. That doesn't remove the need for decisions. Usually, we account for that by being clear about what does inform the decisions: value systems & ideologies. 1/

penders.bsky.social•213 days ago

The question then becomes why is this not (or rarely) happening in metascience? Absent evidence is fairly standard in worldly matters, but that does not excuse one from accountability. Why are we so afraid of talking about values, morals and ideologies and try to dress everything up as evidence? 2/2

jbakcoleman.bsky.social•213 days ago

I don’t see how there couldn’t be a shared, evidence based understanding of likely outcomes even if decisions to implement them as policy are implicitly political. We have evidence that policies which result in fewer carbon emissions will mitigate climate change.

penders.bsky.social•213 days ago

I'm not saying that we can't have any evidence. We definitely can. Most of the decisions are just made in situations where either that evidence is not yet there, or in situations where the evidence does not create a shared understanding (evidence tends to do that). 1/

penders.bsky.social•213 days ago

In all of those cases, we need *more* than evidence to account for decisions. 2/2

danhicks.bsky.social•213 days ago

The value-free ideal is a helluva drug (and means scientists aren't trained to recognize values, morals, and ideologies when they show up in scientific practice)

comic-sans-soleil.bsky.social•213 days ago

I guess in scientific reform values ("rigour", "truth") do drive focus on evidence as primary form of accountability. It's an excessively singular focus and we should be clearer abt those values, discuss and address a wider range of them. But ...

comic-sans-soleil.bsky.social•213 days ago

Barring a massive culture shift that makes science a quite different activity, there will surely always be a stronger emphasis on evidence there than in other policy areas?

drandreahoward.bsky.social•181 days ago

The preprint found over a hundred interventions to summarize for their systematic review. Results are a bit vaguely reported, but over half had “generally positive” (as in “it worked”) conclusions, so it seems there are quite a few studies on this?

jbakcoleman.bsky.social•181 days ago

I think this is the important bit here.

drandreahoward.bsky.social•181 days ago

Yeah I saw that bit, although without a more detailed summary of the included studies’ evidence I’m hesitant to conclude selective reporting. Half positive half neutral seems *not* especially selective

jbakcoleman.bsky.social•181 days ago

Well, that, but also that there are lots of proxy measures, such as evaluating whether people found a paper more trustworthy—not directly whether it increases replicability or whatnot.

So it's hard to disentangle the perception that they're working from whether they're working.

adam42smith.bsky.social•212 days ago

I dislike the replication crisis framing of open science reforms by both skeptics & proponents (itself an interesting case study in how scandals & psych drive discourse).
Every major OS reform (open data, methods, study registration, open access) predates the replication crisis, some several decades

jbakcoleman.bsky.social•212 days ago

Agreed... down thread, I note there may be really good reasons to do this. Even if all of the published research is correct, sharing data/code can facilitate reuse. Preprints and self-archiving improve accessibility and allow for feedback before publication. Registration is great in medicine.

jbakcoleman.bsky.social•212 days ago

I think these are *very* different questions from whether these practices make a) findings more likely to be correct/accurate/useful or b) science better able to progress by some measure of progression.

adam42smith.bsky.social•212 days ago

This is not to say that more and better evidence aren't a good idea -- I'm very much in favor -- but "replicability" is a very limited outcome & mainly of interested to a very specific subset of science (both social & otherwise)

danhicks.bsky.social•214 days ago

Did you and @devezer.bsky.social ever get a clear resolution to the "this paper on preregistration was not actually preregistered" thing?

awmercer.bsky.social•214 days ago

Would love to know the answer to this question while also dreading a revival of all the nastiness that surrounded it the first time around.

devezer.bsky.social•214 days ago

The journal's process is still ongoing. No updates we can share yet unfortunately.

jbakcoleman.bsky.social•214 days ago

As @devezer.bsky.social notes, no updates we can share!

I'm very much with you in hoping there's no nastiness ahead.

everydayhuman.bsky.social•213 days ago

It seems like you getting a decent look at all the angles.
You and others out there.😃
🙏
It’s part of the process I suppose. Thank you for explaining all you could of the process so well.

🙏 🍀
Cheers

everydayhuman.bsky.social•213 days ago

*You are, I swear I spell checked it! Sorry🥲

luscatescu.bsky.social•192 days ago

@briannosek.bsky.social would love to hear your take on this

briannosek.bsky.social•191 days ago

Yes, there is so much work to do on evaluation of open science reforms. That paper does a nice job of highlighting the gaps. It would be nice to have a more discoverability of the many lines of work underway so that people could start pro-actively addressing gaps in the evidence-not-yet-here.

Comments

Posting Rules

Reply