Nature published a misguided op-ed on using LLM's in peer review. The author suggested LLM's could be an aid. But I was interested in a larger question. Can ChatGPT do a peer reivew? So I uploaded a first-submission manuscript for it to review. I also have human reviews for comparison. - ThreadSky

rkapitany.bsky.social • 115 days ago

Nature published a misguided op-ed on using LLM's in peer review. The author suggested LLM's could be an aid.

But I was interested in a larger question. Can ChatGPT do a peer reivew?

So I uploaded a first-submission manuscript for it to review. I also have human reviews for comparison.

Comments

rkapitany.bsky.social•115 days ago

The paper in question is a 2-study package produced with sevearl MSc's over several years. It's not a world-changing manuscript, but - given the conditions of it's creation - it's a reasonable mansucript that I stand behind. It's basic workaday research.

https://osf.io/preprints/8bz2x

rkapitany.bsky.social•115 days ago

The instructions I gave ChatGPT:

"I am going to upload a scientific manuscript fit for publication. I would like you to review the document and provide a 'peer-review' of it. The journal is of modest prestige - the journal is neither exceptional, nor poor.

I would like you to constructively...

rkapitany.bsky.social•115 days ago

... critique the manuscript, including (but not limited to) the quality of prose, the clarity of the hypotheses, the justification of the hypotheses, the accuracy of the statistics and results, and the coherency and validity of the conclusions....

rkapitany.bsky.social•115 days ago

... Be professional and colleagial. Write approximately 2000 words."

This is a short prompt that I think covers the major issues. It may not be a perfect prompt, and it's less instruction than I would give a PhD student in training, but it looks fair to my eye.

So... was it any good?

rkapitany.bsky.social•115 days ago

Absolutely not. You should be able to access it here.

You don't even need to have skimmed the original paper to see why it's bad (though if you think the review isn't bad, then I'm sorry, you're not doing very good peer reviews).

Why is it bad?...

https://chatgpt.com/canvas/shared/67c9a08e0558819188ded56fd2e9030f

rkapitany.bsky.social•115 days ago

The most obvious way in which it's bad is that the critiques are framed as questions. Here are two obvious examples.

I would argue that those questions are superficially valid. They are bad criqitues because I have already addressed them in text.

..

Comments

Posting Rules

Reply