Is anyone else noticing a sudden sharp drop-off in participant quality on @joinprolific.bsky.social in the last week or so? We are suddenly getting over 50% fails on rudimentary attention checks #cogsci #psychscisky
Comments
Log in with your Bluesky account to leave a comment
I also saw posts on the Prolific subreddit that low-value tasks (e.g. with payouts of 15 cents) were getting grabbed so fast they got errors when trying to click on them, so I suspect there are indeed some shenanigans afoot and people are noticing.
I noticed that too! And very different demographics - many more people self-identifying as Republicans and Black (often both together, which is dubious). Like to the point where we had more conservative-leaning than not, which is extremely strange for Prolific. Wasn't sure what to make of it.
Hi @andyperfors.bsky.social, the team would really like to hear more about any strange patterns you might have seen. Would you mind also getting in touch regarding data quality and letting us know here if you do? 🙏 https://researcher-help.prolific.com/en/article/ce2961
Anywhere between 10 and 33%, depending on the task and the nature of the attention check. What struck me in this case is that it's an easy check and there are other cues to low quality (e.g. always pressing the same response key on every trial)
interesting.... and prolific doesn't actually do anything to verify participants when they create accounts right? like they fill out a screener but that can also be garbage data?\
It has before, I'm pretty sure. I've done some pretty rigorous comparison studies (imho) and the quality has been high. Which is why this most recent thing is very surprising. I don't know if their quality control has changed. I'd thought it was a weird blip we had until reading the post above.
There have been 2-3 instances in the past where they get hit by a signup wave of spammers that slip through their checks (or not even intentional spammers in one case, just a tiktok that went viral) and things go haywire for a week or two as they clean up. Hoping it's just that and they're on it.
Every participant is rigorously verified with bank-grade identity checks, location verification, qualitative response assessment in onboarding alone @mehr.nz. We also have our own always-on data protection system Protocol for behavior and response monitoring 🙂
Our system picks up account inconsistencies and suspicious activity which could include indicators of a compromised account, though proving an account sale vs. compromised account can be challenging! This is something our engineers are looking at improving currently.
Hey Jonathan, this does sound strange. Happy to help you out with this and answer any questions in more detail. Could you raise a ticket and data quality report to us so we can investigate the accounts? https://researcher-help.prolific.com/en/article/ce2961
I have doubts about the representativeness of samples that self-select into crowdsource research participation platforms regardless, so I’m inclined to call it a blessing in disguise. Convenience is a great quality for a store. A study, not so much.
These were data downloaded yesterday I think, study ran last week-ish. So more recently. Not good. Participants being aggressive to the researcher too, which is not OK
Comments
Let us know here if you do so we can check ASAP!
#meta
https://journals.sagepub.com/doi/full/10.1177/1948550619875149?casa_token=QDH-miZZqmsAAAAA:npqSW4-C0MKwyPZS12Q16LMlNA0O6MXzQzjSdHtat2hsq_9HKHYNZ-Lz8R2r78CWyZIZnbAnz7kgsQ