Are LLMs 🤖 as creative as humans 👩‍🎓? Not quite! Introducing CREATIVITY INDEX: a metric that quantifies the linguistic creativity of a text by reconstructing it from existing text snippets on the web. Spoiler: professional human writers like Hemingway are still far more creative than LLMs! 😲 - ThreadSky

gximing.bsky.social • 98 days ago

Are LLMs 🤖 as creative as humans 👩‍🎓? Not quite!

Introducing CREATIVITY INDEX: a metric that quantifies the linguistic creativity of a text by reconstructing it from existing text snippets on the web. Spoiler: professional human writers like Hemingway are still far more creative than LLMs! 😲

Comments

johnegan.bsky.social•95 days ago

hi ximing,

am trying to develop options for probabilistic firewalls
Q: what is/are the best security measure(s) that you are aware of to help stop or mitigate probabilistic injection ?
the simplest form of probabilistic injection is a ‘prompt injection’

gximing.bsky.social•47 days ago

I'm not particularly familiar with this field, but here's a survey paper on jailbreak attacks and defenses against LLMs that might be relevant.

https://arxiv.org/pdf/2407.04295

gximing.bsky.social•98 days ago

TLDR: We found the seemingly remarkable creativity of LLMs 🤖can be attributable in large part to the creativity of human-written texts on the web. In contrast, works by distinguished human authors 👩‍🎓cannot be easily replicated by merely assembling snippets from other works.

gximing.bsky.social•98 days ago

We define L-uniqueness for a text as the proportion of its words, outside of n-grams (n ≥ L), that appear in a vast reference corpus (e.g., RedPajama).

The CREATIVITY INDEX is then defined as the area under the L-uniqueness curve across a range of minimum n-gram lengths L.

leonderczynski.bsky.social•97 days ago

How does this connect to traditional measures and quantities of creativity?

gximing.bsky.social•98 days ago

To compute CREATIVITY INDEX efficiently, we introduce DJ SEARCH, a novel dynamic programming algorithm that can efficiently search for verbatim and near-verbatim matches of text snippets (i.e. n-grams) from a given document against a vast reference corpus in linear runtime.

gximing.bsky.social•98 days ago

We found CREATIVITY INDEX of human authors—specifically professional writers and historical figures—is on average 66.2% higher than that of LLMs. This gap is consistent across various domains—novel snippets, modern poems, and speech transcripts—at both verbatim and semantic levels.

gximing.bsky.social•98 days ago

Moreover, we found that RLHF dramatically reduces the CREATIVITY INDEX of LLMs, by an average of 30.1%. This reduction is more significant at the verbatim level than the semantic level, indicating that LLMs may have converged to certain linguistic style preferred by humans during alignment.

gximing.bsky.social•98 days ago

Furthermore, we explore creativity differences among various groups of humans. Despite in-group variance, famous authors of classic literature, like Hemingway and Dickens, exhibit the highest levels of creativity, consistent with their levels of renown.

yetanotheruseless.com•97 days ago

I need to read this, but are you just comparing to world-renowned authors like Hemingway, or also, eg regular people like someone with a bachelors degree in History or Literature - a journalist or not-well-know fiction writer?

gximing.bsky.social•47 days ago

In our paper, we compare LLMs to professional human writers, ranging from world-renowned figures like Hemingway to less famous and newer-generation authors.

gximing.bsky.social•47 days ago

We're curious: with LLMs having consumed vastly more text than any human could ever read—including the works of distinguished writers and historic figures—could they, by standing on the shoulders of giants, create novel text that reaches new heights of linguistic sophistication and creativity?

Comments

Posting Rules

Reply