Good 9:30 AM turnout for "Using LLMs for Judgment and Decision Making research" @hopeschroeder.bsky.social @dgrand.bsky.social @kobihackenburg.bsky.social @sjdm-tweets.bsky.social - ThreadSky

dggoldst.bsky.social • 94 days ago

Good 9:30 AM turnout for "Using LLMs for Judgment and Decision Making research"

@hopeschroeder.bsky.social @dgrand.bsky.social
@kobihackenburg.bsky.social

@sjdm-tweets.bsky.social

Comments

dggoldst.bsky.social•94 days ago

Can large language models categorize your free text data accurately?

LLMs are highly effective in categorizing (i.e., coding) free-text responses from participants, often outperforming human coders sourced from crowdsourcing platforms
says @sdpbht.bsky.social

@sjdm-tweets.bsky.social

blasimon.bsky.social•94 days ago

Would love a copy if there’s a working paper?

profgaelle.bsky.social•94 days ago

Super pertinent as usual @sdpbht.bsky.social do you have a reference for this? I've seen it done but in a rather ad-hoc way from some of your Penn neighbours https://arxiv.org/pdf/2309.10771. I am looking for a rigorous approach to prompt for this, including validation prompts.

joesimons.bsky.social•94 days ago

We have some thoughts on the topic of assessing performance in the case of extracting psychological features here https://arxiv.org/abs/2405.07447

joesimons.bsky.social•94 days ago

Motivated in part by seeing attempts to use LLMs to not just code text features but infer deeper latent variables

dggoldst.bsky.social•94 days ago

Very interesting! Thanks for sharing. Would be nice to see it compared to, say, Prolific workers

joesimons.bsky.social•94 days ago

Yes, and one feature of our approach is you can make a genuine comparison - ie there is a way to detect if machines outperform human raters. Practically this seems highly unlikely, but conceptually I've always been a bit uncomfortable with taking human ratings as ground truth.

dggoldst.bsky.social•94 days ago

This paper also seems relevant

https://www.pnas.org/doi/10.1073/pnas.2308950121

dggoldst.bsky.social•94 days ago

I figured out the posted image is from https://arxiv.org/abs/2104.14337

... but it's not specific to text classification. Will let you know if I find one.

sdpbht.bsky.social•93 days ago

Thanks @dggoldst.bsky.social for listing the reference. The image involves nlp and vision benchmarks. One of these is GLUE (https://gluebenchmark.com/) which has many tasks, some of which I would roughly consider to be "structure extraction from text". It also has ground truth.

sdpbht.bsky.social•93 days ago

PS another task is SQUAD (https://rajpurkar.github.io/SQuAD-explorer/) which is about extracting content from text. Again with ground truth.

dggoldst.bsky.social•94 days ago

Next up @xehu.bsky.social @sdpbht.bsky.social @mikeyeomans.bsky.social

dggoldst.bsky.social•94 days ago

Tom Costello ( @tomcostello.bsky.social ) has made freely available tools that let you build LLM interaction into Qualtrics

https://publish.obsidian.md/qualtrics-documentation/Documentation+for+Using+the+Human-AI+Interaction+Qualtrics+File/Home

tomcostello.bsky.social•94 days ago

Thanks for sharing Dan. Also do check out https://www.vegapunkdoc.dev/

Which is a much easier to use version of out tool.

jackalshorns.bsky.social•94 days ago

Plans to integrate with other survey platforms? Or is it really just for Qualtrics?

dggoldst.bsky.social•94 days ago

Best practices for having LLMs label your data by @hopeschroeder.bsky.social

dggoldst.bsky.social•94 days ago

Qualtrics can make calls to an LLM, get answers, and pipe them back to participants

smcgrath.phd•94 days ago

Interesting! #AcademicSky #MLSky

dgrand.bsky.social•94 days ago

@tomcostello.bsky.social 's Qualitrics materials and tutorial video for integrating LLMs into Qualtrics can be accessed at https://publish.obsidian.md/qualtrics-documentation/Documentation+for+Using+the+Human-AI+Interaction+Qualtrics+File/Home

steveread.bsky.social•94 days ago

📌

jmping.bsky.social•94 days ago

📌

sjdm-tweets.bsky.social•94 days ago

michaelschulte.bsky.social•94 days ago

much lower turnout at open science publishing … guess LLMs will save us

anubrata.bsky.social•94 days ago

Will there be a video for this talk?

anubrata.bsky.social•94 days ago

Nvm, found it!

jackalshorns.bsky.social•94 days ago

Can you post the linky?

dggoldst.bsky.social•94 days ago

I don't know what you guys are talking about lol there is no video

jackalshorns.bsky.social•94 days ago

I'll just have to imagine it, then. Thanks for the clarification.

dggoldst.bsky.social•94 days ago

Sorry about that! The conference wasn't recorded. However you could contact the authors. I have put their handles in previous posts.

anubrata.bsky.social•94 days ago

Right, sorry for being unclear. I saw your comment sharing the Qualtrics integration tutorial with a video. https://bsky.app/profile/dggoldst.bsky.social/post/3lbrvnmkns226

dggoldst.bsky.social•94 days ago

Gotcha!

dggoldst.bsky.social•94 days ago

But there is no video!

alainstarke.bsky.social•94 days ago

Martijn was texting me SJDM does not like applied research; well, well, well...

dggoldst.bsky.social•94 days ago

SJDM welcomes all kinds of judgment and decision making research. Acceptance rate was only 13% this year so folks shouldn't read much into rejections. Keep submitting, everyone!

alainstarke.bsky.social•94 days ago

He has a poster, so no worries :-)

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply