I also reject the notion that "most biomedical literature is unreliable/untrue." This is simply not the case. The vast majority of biomedical research is so reliable we don't even think about it. And yes, some studies and even subfields have issues, but let's acknowledge all the things that work. 🧵 - ThreadSky

clauswilke.com • 5 days ago

I also reject the notion that "most biomedical literature is unreliable/untrue." This is simply not the case. The vast majority of biomedical research is so reliable we don't even think about it. And yes, some studies and even subfields have issues, but let's acknowledge all the things that work. 🧵

Reposted from Richard Sever

"Create a journal for replication work". 🙄

This comes up time again as if there aren't existing places to put replications. PLOSOne will taken them, plenty of other journals will, and bioRxiv has had an article type called "Confirmatory Results" since we launched in 2013!

Comments

clauswilke.com•5 days ago

For example, crystal structures. We simply assume any published crystal structure is right, because there are so few issues. In fact, go to the PDB and you'll find hundreds of replications of popular structures (HIV-1 protease) and they all look the same. 1/

clauswilke.com•5 days ago

AlphaFold and other AI methods. They just work. People throw their favorite proteins at them and expect the results to be reliable. Because by and large they are. (I think people are overly confident with these methods, but that's a different question. There's no problem with replication.) 2/

clauswilke.com•5 days ago

Computational methods more generally. Most computational methods have all code available and can be run and produce the same results every time. And again, there can be disputes over the correctness or interpretation of results (see single-cell work) but that's different from lack of replication. 3/

clauswilke.com•5 days ago

Sequencing methods, such as next-gen sequencing or nanopore sequencing. You just send your samples to sequencing and they come back and you trust the results. These are advanced methods that not that long ago were at the cutting edge of science. And they have replicated over and over. 4/

clauswilke.com•5 days ago

CRISPR/Cas and other gene editors. Every lab uses them, because they work. 5/

clauswilke.com•5 days ago

Gene synthesis and assembly methods. These days, you just order the sequence fragments you want. And then you stitch them together with Golden Gate assembly. People just take it for granted that it works. 6/

brooklynkid53bskys.bsky.social•4 days ago

well, iirc, according to the science citation index, something like 50% of papers have zero or one citation, so most papers in the literature maybe right, but they are ignored, IMO, cause they are boring or hard to read or wrong

clauswilke.com•4 days ago

These stats don't match my personal experience. Almost every paper I've written has gotten some citations. I'd like to see stats by field, or by journal, or (most importantly) by whether the study was NIH-supported. Do 50% of papers that acknowledge NIH support never get cited? Seems high.

clauswilke.com•4 days ago

Also, of course, Science Citation Index omits large parts of the scientific literature from its database so has to be taken with a grain of salt. Lack of citations for the less impactful papers may simply be lack of coverage of the citation index.

brooklynkid53bskys.bsky.social•4 days ago

It would seem that you are more correct then I

ok, it looks like 20% or so is correct, see here
https://www.science.org/content/blog-post/not-so-many-uncited-papers-actually

clauswilke.com•4 days ago

20% seems more reasonable. Though in general, I think we're way too obsessed with the existence of inconsequential papers. They're part of the process and trying to get rid of them won't do much to improve science. I wrote a blog post about this a long time ago.
https://clauswilke.com/blog/2013/11/03/no-one-reads-your-paper-either/#fnref5

lotalibre.bsky.social•4 days ago

‘Lots of things work’ is a very different claim to ‘most of the literature is reliable.’ Is there empirical evidence for the latter?

castdeath97.bsky.social•4 days ago

(Evo)Psych and its consequences tbh

traceydepellegrin.bsky.social•5 days ago

JB just repeated this claim in his Senate Appropriations testimony, saying that most people don't know about the issues. Rinse and repeat. Congress isn't holding him to account by probing his statement or providing counterpoints.

mikethemadbiol.bsky.social•5 days ago

📌

thecakeisnotalie.bsky.social•5 days ago

I can't agree with this take. Most biomedical literature is reliable *for a given cohort*. We have a lot of trouble in ensuring diversity of test subjects, even when working with cell cultures! We don't test our results on a wide enough diversity in vitro, let alone in vivo!

thecakeisnotalie.bsky.social•5 days ago

This doesn't make the individual studies worthless - far from it - but it does make their applicability at the scale of *populations* extremely limited, and in some cases downright dangerous! (See: differences in treatment applicability between schizophrenics and autistics, for example!)

richardsever.bsky.social•5 days ago

💯 - a huge amount of damage has been done by that claim and amplification of it both by the well intentioned but naive and by the less well intentioned

jkpritch.bsky.social•5 days ago

Well, it does seem logically flawed to build a movement based on assuming the correctness of a published research paper entitled "Why most published research findings are false" ...

jtleek.bsky.social•4 days ago

Also empirical estimates don't agree with the simulation....

https://pubmed.ncbi.nlm.nih.gov/24068246/

wheelerlab.org•4 days ago

https://www.youtube.com/watch?v=4WRtqmHpLvg

richardsever.bsky.social•4 days ago

https://bsky.app/profile/richardsever.bsky.social/post/3lrbmqq3cuc2l

robbuc.bsky.social•4 days ago

It does, doesn't it? Also, the main author of that paper did not perform so flawlessly in their forays into epidemiological research during the COVID era... & lots of damage was caused at the time, which persists to this day. Remember the "Great Barrington" declaration? @richardsever.bsky.social

richardsever.bsky.social•4 days ago

Bertrand Russell would be proud

Comments

Posting Rules

Reply