I wrote some thoughts about why peer review matters It shapes scientific standards, maintains field coherence & trains new researchers Yes, it needs improvement—but it's the glue that holds scientific progress together briscoelab.org/2024/12/11/i... - ThreadSky

jamesbriscoe.bsky.social • 167 days ago

I wrote some thoughts about why peer review matters

It shapes scientific standards, maintains field coherence & trains new researchers

Yes, it needs improvement—but it's the glue that holds scientific progress together

https://briscoelab.org/2024/12/11/in-praise-of-peer-review/

Comments

arjunraj.bsky.social•166 days ago

Totally respect what you are doing at Development :), but I disagree on the benefits. Moreover, I don't think the cost of peer review is adequately covered in this piece. Often, 1/3 of project time is spent on needless reviewer requests. Further, what is not measured is the human cost… 1/

arjunraj.bsky.social•166 days ago

… of trainees who are soured by an arbitrary and tedious process that is increasingly slow when science is going increasingly fast. This is not how we will keep our best and brightest in academia. 2/2

katholt.bsky.social•166 days ago

What you’re referring to there are not necessary or universal aspects of peer review, they are examples of bad practice.

arjunraj.bsky.social•166 days ago

Sure, but it is what it typically looks like in actual practice :)

katholt.bsky.social•166 days ago

Typically?
Not in my experience.
I accept some aspects of this are field dependent, but I have coauthored >100 papers and chief-edited a journal and have never spent 1/3 of a project time on responding to reviews, nor seen other authors asked to do so.

katholt.bsky.social•166 days ago

I preprint everything & support new models of peer review. What we have this moment is not the best and only way.

But @jamesbriscoe.bsky.social article resonates very strongly & I shudder to imagine the flood of garbage we would be subjected to if authors thought they would never be peer reviewed.

acritschristoph.bsky.social•166 days ago

arjun I wonder if this is subfield specific or person specific or what.

I've heard a lot of people with your view (usually from mol cell bio or biochem, I think?). I've been part of only ~30 review processes or so but all of them have been pretty productive and reasonable.

ruvinskyilya.bsky.social•166 days ago

Is 1/3 estimate due to going to "fancy" journals? In my experience, the average time in review (particularly considering bouncing from one journal to another) is well correlated with how fancy the first place of submission is. So there is value in IF - it's a predictor of time in review.

devoevomed.bsky.social•166 days ago

I suspect this is right. Especially for CNS(NG)!

ruvinskyilya.bsky.social•166 days ago

CN(G)S

cyrilpedia.bsky.social•167 days ago

RE: "When reviewers evaluate manuscripts, they engage in a dialogue with authors and editors", a frequent complaint I hear is not about reviewers, but about editors who fail to engage in dialog & to give useful feedback to authors to guide a resubmission.

leopoldroth.bsky.social•167 days ago

Yes, in many cases, editors mostly forward e-mails, while their contribution is, what peer-review needs to be coherent and productive. Otherwise, integrating two different opinions, which are blind to each other, can be chaotic and frustrating.

cyrilpedia.bsky.social•167 days ago

That's the editor's job, in a healthy process, to review the reviews.
On the plus side, one feature that is increasingly common is reviewer cross-commenting, so that at many journals, they are not blind to each other.

leopoldroth.bsky.social•167 days ago

Which implies that paying editors and not overloading them is critical for the entire process.

cyrilpedia.bsky.social•167 days ago

Yes, in the academic journals that I know best, the professional editors (I was one for a couple of years) came after the academic editors simply could not handle the load anymore (in the cases I know, at the end of the 90s).

tuomaspernu.bsky.social•167 days ago

Precisely this: I find the idea that there's a "dialogue" very naive and misguided. Should there be a "dialogue"? Maybe. But then we need to put conscious effort and invest resources to it.

https://bsky.app/profile/tuomaspernu.bsky.social/post/3ld2dufw4qc2j

sproullab.bsky.social•167 days ago

Nice opinion piece. I'm inclined to agree, when done well peer review can be really helpful in strengthening work and serves a vital role. However, it is not perfect and particularly when donee badly with many negative consequences.

peiferlabunc.bsky.social•167 days ago

I was interested to see you did not comment my post about our recent peer review experience at your journal...

peiferlabunc.bsky.social•167 days ago

and still didn't....

jamesbriscoe.bsky.social•167 days ago

Mark, I'm not going to discuss individual editorial decisions publicly

It would not be seemly or edifying

Nor would it be possible in 300 characters

peiferlabunc.bsky.social•166 days ago

You are correct but I do hope you realize your high minded aims have to translate into reality in individual decisions....

thomasgregor.bsky.social•167 days ago

This is a splendid piece with many important points raised, most of which I fully endorse. Journals should be held to make the peer review public (with or without names). The discussion is part of the scientific discourse and has important scientific and educational value.

martinhumphries.bsky.social•167 days ago

I concur James, but couldn't what you say be achieved with post-publication peer review? This certainly needs a major culture change, but the outcome would be more of a community review as opposed to a process involving few people. One day, we also need to get away from the concept of journals! 😉

sproullab.bsky.social•167 days ago

I really like the model of open peer review of pre-prints but l think very few people find the time. I last year but it kept falling by the wayside of other stuff. Making a commitment to someone external like an editor seems to help me actually get peer review done.

martinhumphries.bsky.social•167 days ago

That's the culture change that is needed

jamesbriscoe.bsky.social•167 days ago

Possibly. But I don't yet see a post-pub model that is really working and likely to be sustainable. Would require a major cultural change for us, which is unlikely to happen rapidly, given how conservative we all seem to be.

martinhumphries.bsky.social•167 days ago

Agreed, but some elements of the current system are broken and change is needed urgently. We need a route that gets us from A to B, and I haven't yet seen a consensus view. @richardsever.bsky.social ?

richardsever.bsky.social•167 days ago

Lots to agree with in that article - in particular, concern that lack of the prospect of a peer review barrier could gradually erode standards. And as always I emphasize pro-preprint =/ anti-peer-review; the goal should be to improve critical assessment not abandon it. But a few quick thoughts. 1/n

robinsall.bsky.social•167 days ago

"the goal should be to improve critical assessment not abandon it".

Yes! But it's a problem with the madness of crowds. You need the field to agree on how assessment should be improved. One journal or group of journals doing it is no use - it just moves people towards less rigorous places.

richardsever.bsky.social•167 days ago

Essential to emphasize current "peer review" is a recent invention (are we saying all Nature papers pre-1970s are problematic b /c they didn't undergo it?). Meanwhile decades of arXiv suggest the threat of reputational damage is sufficient to maintain standards https://www.embopress.org/doi/full/10.15252/embj.201695531 2/n

richardsever.bsky.social•167 days ago

We must also distinguish well-conducted peer review at a single journal like Development from the systemic issue: most papers end up published somewhere; and most papers are little different from the preprint. Both of which suggest James's desired peer review dialog is not really occurring 3/n

lincolnskayak.bsky.social•167 days ago

As a non-scientist (who only knows what peer review is because I listen to TWiV podcasts), I would say peer review is the MOST important part of the scientific process. The last thing the general public needs is a layperson cherrypicking from a preprint to support a nefarious political movement.

lincolnskayak.bsky.social•167 days ago

Binge listening to TWiV & other https://microbe.tv podcasts (at start of the pandemic) was my introduction to what a “preprint” is. Listening to scientists discuss preprints was comforting to me. The general public was getting some dangerously stupid info from political leaders back then. I 💙 peer review!

tuomaspernu.bsky.social•167 days ago

This is starry-eyed. "When reviewers evaluate manuscripts, they engage in a dialogue with authors and editors" - what?!

Of course the question is: how should we change? Two things:
- No journal under 20-30% acceptance rate.
- Resources for editors to facilitate dialogue.

https://bsky.app/profile/jamesbriscoe.bsky.social/post/3lczj6tevt22i

tuomaspernu.bsky.social•167 days ago

I get the point about continuity etc., and I don't 100% disagree But some might use a different word here: bias. That's why I say: we need to get rid of artificially low acceptance rates. They only create bias. In fact, much of the things in the piece make sense if we'd have the limit around 20-30%.

tuomaspernu.bsky.social•167 days ago

The job of editors and reviewers amount basically to shooting down submissions. The processes are far from constructive "dialogues". *If that's what we want, then we need to start changing the acceptance rates. And this requires resources - but given the profit margins this should not be an issue.

tuomaspernu.bsky.social•167 days ago

Excellent points below. In particular:

most papers end up published somewhere, with little difference to preprints
-> dialogue don't happen

quality signal is assigned on the basis of subjective opinions of a few people

non-peer-review functions of journals are critical

https://bsky.app/profile/richardsever.bsky.social/post/3lczmhyp2rs22

tuomaspernu.bsky.social•167 days ago

A positive word about the piece: I do think the threat of fragmentation is real. And there are many problems here. However, at the same time, some might say this only one more instance of bias, and idea of a coherent project was an illusion to begin with. But I think we should proceed carefully.

remisramos.bsky.social•167 days ago

Agreed, but can you elaborate on the acceptance rate constraint? A new, non-profit journal (not indexed, with little incentive for competent authors to submit) has a smaller pool to choose from so it's natural to start with a higher rate, but keeping it high might not be sustainable in the long run.

tuomaspernu.bsky.social•167 days ago

This should be more analytically considered, of course. But the fact is that we are currently creating a drastic bias. The low rates emanate from a time when one had limits to what one can print. Not so anymore. This is the core of the OA model too: make basic checks, then let the audience decide.

remisramos.bsky.social•167 days ago

I get it. The PDF-only model makes it way cheaper than it used to be! Context: I worked at a small journal (funded by a small Uni, with no APC or fees, Open Access) that got recently indexed in Scopus. We are now overwhelmed by submissions, and we still have to pay the proofreading and typesetting.

remisramos.bsky.social•167 days ago

So, we went from publishing 18 articles a year to publishing 30 in 2024, and... well, I published a meme about it this morning. Funding is hard, and now our team is collapsing under the burden xD

tuomaspernu.bsky.social•167 days ago

I get that. And of course we cannot impose any universal rules. However, we could agree on a general idea, and try and make sure that all the bigger players implement it. Like I say in the thread: I don't think it's really about resources, considering the current profit margins.

robinsall.bsky.social•167 days ago

Agree with much of this. And the concept.

But "Peer review plays a crucial role in maintaining the integrity of scientific fields over time". Really? In the global sense - peer review has already failed. Peer reviewed journals *in general* have strikingly poor integrity, for a variety of reasons.

leopoldroth.bsky.social•167 days ago

I agree! If the people who review use questionable practices as well, how do we expect them to check others?

robinsall.bsky.social•167 days ago

I actually think Development & JCS have some of the best refereeing around. But in JCS' case (I know that field better) - journals with much worse integrity are much more highly regarded for careers & funding; authors can see that more integrity = less success.
Needless to say this is catastrophic.

tlemberger.bsky.social•166 days ago

Totally agree that peer review is (should be) a dialogue. Maybe it is facilitated with journal-agnostic peer review (see @reviewcommons.org) when review and editorial decisions are separated. To be a dialogue, author's responses must be integral to the process. Crucial for public reviewed preprint

timelfen.bsky.social•167 days ago

These are all important epistemic & pedagogical functions of peer review but I think you miss the crucial social function of reactivating the moral economy scholars (against all those forces degrading the conditions for a mutually obligated communities of inquiry). https://doi.org/10.1177/01622439211068798

Comments

Posting Rules

Reply