Apparently ~40% of crowdsource workers use LLMs when completing surveys Relatively simple fix for text responses: Add Javascript to Qs to prevent pasting Credit to Jiabi Wang for idea + Kaushal Addanki for code below github.com/rafmbatista/... - ThreadSky

rafmbatista.bsky.social • 510 days ago

Apparently ~40% of crowdsource workers use LLMs when completing surveys

Relatively simple fix for text responses: Add Javascript to Qs to prevent pasting

Credit to Jiabi Wang for idea + Kaushal Addanki for code below

https://github.com/rafmbatista/functions/blob/675cd7dc449ab966f282050b773b78bd550a279b/no-paste-Js

Comments

regretlab.bsky.social•509 days ago

Thank you, this is incredibly useful!

mattgraham.bsky.social•509 days ago

Source for 40% claim?

rafmbatista.bsky.social•509 days ago

https://arxiv.org/abs/2306.07899

hcobb.bsky.social•509 days ago

Or use the LLM itself to generate a one way key into the uncanny valley that require human levels of sensibility to escape instead of just waiting a few minutes for your JS to be reverse engineered.

chrisxrodgers.bsky.social•509 days ago

It drives me nuts when websites block copy/paste to require me to enter my email or password twice in a row. A hack that almost always works is to select text in another window and drag the selected text into the textfield that presents pasting.

christhomas.bsky.social•509 days ago

Thank you for sharing this -- I am fielding a crowdsourced survey soon and had been worried about how to deal with LLM fake responses. This seems like an elegant fix.

rafmbatista.bsky.social•509 days ago

You're welcome

Yeah, i figure for most tasks the work involved with coming up with your own answer will be less than the effort needed to manually copy and paste from ChatGPT :)

yeli.bsky.social•510 days ago

When I asked the CloudResearch table at ACR about this problem, they said they had done some internal research on this and another straightforward solution that seems to work reasonably well is to just ask people not to use ChatGPT.

larkenciel.bsky.social•509 days ago

Turning off copy paste is anti accessibility.

mariaa.bsky.social•509 days ago

As long as it’s question-specific, I think it’s fine? I don’t think it should interfere with voice to text, for example.

larkenciel.bsky.social•509 days ago

People use all kinds of tools for all kinds of accessibility issues. Preventing paste prevents people from using the tools they rely on.

mariaa.bsky.social•509 days ago

Could you give an example? I’ve previously asked participants about this and read up on my own and couldn’t find any issue around accessibility, as long as paste was disabled for specific boxes and not the whole site. If there’s an issue, I definitely want to know about it!

larkenciel.bsky.social•509 days ago

I almost never type anything more than a few words in forms. I type in my own software with my own settings and then copy paste in.

rafmbatista.bsky.social•509 days ago

Can you say more? Is it because of how it interacts with dictation software?

larkenciel.bsky.social•509 days ago

People use all kinds of tools for all kinds of accessibility issues. Preventing paste prevents people from using the tools they rely on.

lauraewallace.bsky.social•510 days ago

Very cool. Do you tell participants you are doing this from the start?

rafmbatista.bsky.social•509 days ago

I haven't in my studies

But here's another version of the Javascript (thanks GPT!) that provides a pop-up notification when participants try to copy+paste it tells them it's not allowed for that question

https://github.com/rafmbatista/functions/blob/675cd7dc449ab966f282050b773b78bd550a279b/no-paste-Js-2

mariaa.bsky.social•509 days ago

It’s nice to do this at the beginning, in the study description or somewhere else’s obvious, because some participants type in a separate window to avoid losing their work. It’d be very annoying to type it all out and only then realize you can’t paste it into the answer.

christhomas.bsky.social•509 days ago

Would it be possible to write something that also flags respondents who try to copy/paste? I would still pay them, but I would discard those responses since it seems suggestive of inattentive participation.

rafmbatista.bsky.social•509 days ago

I believe it is. You'll need an embedded variable in that case.

Ironically, GPT should be able to come up with that code for you. I've tried different versions of this one that GPT wrote and they all seem to do okay

mmilkowski.bsky.social•508 days ago

They will learn about AutoHotKey. And then all is lost.

rafmbatista.bsky.social•509 days ago

Here's the working paper (not mine):

https://arxiv.org/abs/2306.07899

rafmbatista.bsky.social•509 days ago

Here's a follow-up by same authors looking at prevention strategies:

https://arxiv.org/abs/2310.15683

Comments

Posting Rules

Reply