In a paper we’re workshopping this week at #NeurIPS2024, Ned Cooper, @wesleydeng.bsky.social, and Ben Hutchinson and I ask: what is the model of societal impacts reflected in efforts to evaluate GenAI systems? Paper: arxiv.org/abs/2410.22985 Workshop: evaleval.github.io 1/5 - ThreadSky

About ThreadSky

glenberman.bsky.social • 78 days ago

In a paper we’re workshopping this week at #NeurIPS2024, Ned Cooper, @wesleydeng.bsky.social, and Ben Hutchinson and I ask: what is the model of societal impacts reflected in efforts to evaluate GenAI systems?

Paper: https://arxiv.org/abs/2410.22985
Workshop: https://evaleval.github.io/
1/5

Comments

glenberman.bsky.social•78 days ago

To answer, we look at taxonomies of societal impact, developed to support GenAI evaluations. We ask: how do these taxonomies focus attention on which social factors are capable, and deserving, of measurement during GenAI evaluation?
2/5

glenberman.bsky.social•78 days ago

We put forward 3 premises for rethinking societal impacts & their relationship to the introduction of GenAI in a particular social context. Societal impacts should be thought of as application- and context- specific, incommensurable with each other, and about questions of power.
3/5

glenberman.bsky.social•78 days ago

And, we argue that understanding societal impacts in this way means accepting that predictive evaluations of GenAI impacts are inherently limited. We suggest a ‘governance-first’ approach may be more effective at meaningfully managing harmful consequences of GenAI.
4/5

glenberman.bsky.social•78 days ago

Made it this far? Soon we’ll be starting an interview study with Renee Shelby to explore the development & use of taxonomies of societal impact . If you’d like to be kept in the loop please follow (https://glenberman.bsky.social).

Read more: https://arxiv.org/abs/2410.22985
5/5

struthious.bsky.social•77 days ago

Excited to learn more about this at the workshop!

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply