i'm followed by a lot of academics, so let me just get this one PSA out there: there is no tool which detects AI-generated text, and it is plausible that no such tool is possible. revision history, however, is more or less adequate proof of human generation. - ThreadSky

theophite.bsky.social • 47 days ago

i'm followed by a lot of academics, so let me just get this one PSA out there: there is no tool which detects AI-generated text, and it is plausible that no such tool is possible. revision history, however, is more or less adequate proof of human generation.

Comments

apearlma.bsky.social•47 days ago

Also, not as if every document needs to be checked, just when suspicious

christianh.bsky.social•47 days ago

My previous university in late 2023 adopted an undisclosed "AI detector" whose assessments would be considered binding (and if positive disqualifying) on theses, something which I don't think is particularly unique and probably an endless fountain for student lawsuits in the next couple of years.

detenber.bsky.social•47 days ago

On the front lines here, and most of us understand there is no reliable AI detection system for writing (yet). So punishing students based on their output is wrong. Although it will improve, AI at this point is so bad that I just grade it on the merits. Basically no better than a B-.

sentientpotato.bsky.social•47 days ago

Yeah that's something that's nice about it, Chat GPT isn't gonna get them a good grade at least lol

concatnonsense.bsky.social•47 days ago

That approach only works if the draft and final document are the same, many work in disparate documents then paste together the final release which would appear the same as pasting in large segments from a GenAI prompt response.

jamestiv.bsky.social•46 days ago

Wtf is that kind of process? Like writing each section of the paper as a separate document?

concatnonsense.bsky.social•46 days ago

That or pulling together from multiple other R&D documents, it's incredibly common.

wallymcfly.bsky.social•47 days ago

How feasible would it be to embed a fingerprinting tool into a word processor that functions like captcha - keeping a hash of changes, delays between key strokes, writing session length, etc? Seems like verification is going to have to be on the human side, with gen AI as the default assumption

eean.dev•47 days ago

Just using existing change tracking features of word processors and graders doing spot checks makes sense.

If I was a student today I'd keep all this on-hand in case I'm ever accused of using AI, thanks to profs using those almost fraudalent AI detectors.

jamesscheibner.bsky.social•47 days ago

It’s easier just to look at how long the document has been edited for, and check for dodgy references.

wallymcfly.bsky.social•47 days ago

I can understand how that’s easier on an individual level in the short term. I’m thinking more theoretically about proving human authorship overall

gizmotronic.bsky.social•47 days ago

I honestly do not believe that there is a technological fix here. We are either going to learn to deal with the AI use culturally or we're going to go through an endless series of technological "solutions" that will only empower the knuckeheads creating the problem in the first place.

wallymcfly.bsky.social•47 days ago

What’s the cultural fix?

piss.beauty•47 days ago

just sign documents with your private key & publish the public key. it's significantly easier than this surveillance scheme

eean.dev•47 days ago

We're talking about kids cheating at writing assignments, not impersonation.

wallymcfly.bsky.social•47 days ago

And even beyond that, right? If there’s genuinely not even a theoretical way identify text as AI-generated, then the only path forward is to verify human authorship or just accept there will never be a way to tell the difference again

thefoodandwinediva.bsky.social•46 days ago

Brilliant!

bealeiderman.bsky.social•47 days ago

Everything old is new again. As a Spanish teacher ages ago I used version history to see if kids were using Google translate. I didn’t mind if they looked up single words, but copypasting full sentences and paragraphs was a no-no. It was easy to spot sometimes, but it depended on the kid, really.

algoresmilkymams.bsky.social•46 days ago

"¿inteligencia artificial en la clase de espanol? ¡que irresponsable!"

squorch.com•46 days ago

In está economia?

saintbrendan.bsky.social•45 days ago

Ooh, meet necessarily. Generative Adversarial Networks (GAN) are weirdly good at studying it whatever metrics you're using to spot "human generation-ness", and mimic it. Including timed revision history with more flaws in earlier versions.

GANs are spooky. Spooky *because* they are so effective.

neotiamat99.bsky.social•47 days ago

So, I 100% believe you, but I don't suppose you can point to any research paper or similar I can use to bludgeon some of my less tech-savvy colleagues over the head?

The US History teacher at my school thinks AI is straight-up magic and I am contemplating homicide.

willcjohns.bsky.social•47 days ago

The easiest way to do it is take a few old papers of theirs and put them through "AI detection software". It'll usually say some percent is AI generated.

annarogers.bsky.social•46 days ago

Here's my blog post describing a recent paper led by @gpucce.bsky.social:
https://hackingsemantics.xyz/2024/ai-news/
It outlines the main approaches to detection and problems with them.

neotiamat99.bsky.social•46 days ago

Oh this is *exactly* what I am looking for. Thank you very much!

Comments

Posting Rules

Reply