So here's something I'm really impressed by: I put this preprint out just about a year ago. Three or four others came out about the same time, saying a) we know authors are using LLMs a lot, b) it's really obvious from word choice; c) no, *really* obvious. What happened next? Well... - ThreadSky

generalising.bsky.social • 3 days ago

So here's something I'm really impressed by: I put this preprint out just about a year ago. Three or four others came out about the same time, saying a) we know authors are using LLMs a lot, b) it's really obvious from word choice; c) no, *really* obvious.

What happened next? Well...

Reposted from Andrew

I have a preprint out! Evidence for extensive appearance of chatGPT/LLM derived text in scholarly papers, signalled by particular words that mysteriously became a lot more popular in 2023 - eg "commendable". I estimate upwards of 60,000 papers last year (and accelerating) arxiv.org/abs/2403.16887

Comments

generalising.bsky.social•3 days ago

What happened next? Well, those papers got publicised. Then, mysteriously, a bundle of the words we identified as strong indicators suddenly stopped growing, and in some cases ("intricate") declined again. It's truly impressive.

generalising.bsky.social•3 days ago

This is from a great study a couple of weeks back looking at the terms in arXiv abstracts - https://arxiv.org/abs/2502.09606 - these can be pinned down to precise days, and they tend to have very short writing-to-posting lags, so the effects show up quickly.

generalising.bsky.social•3 days ago

But there are other markers that didn't change - so this seems to show LLM use is still growing steadily, but people are editing the results, or tweaking the prompts, or even improving the models themselves, to sound "less obvious". Which is a bit of a double-edged sword.

generalising.bsky.social•3 days ago

On the one hand, if we're going to have LLM-edited material, at least it can be a bit less awkward to read. On the other hand, if people aren't disclosing they've used these tools, then disguising it better makes it harder to spot and to draw inferences from that. So, swings and roundabouts.

generalising.bsky.social•3 days ago

I expected to see this sort of development, but I was absolutely astounded to see it happen so promptly. I guess ... did our little group of papers actually leave a distinctive mark in the literature? What an amazing thought.

generalising.bsky.social•3 days ago

I am working on a followup to this at the moment - if this week goes well, should have some new graphs to share shortly. Watch this space...

Comments

Posting Rules

Reply