Been waiting for someone to test this and see if it works - can multiple AI agents fact-checking each other reduce hallucinations? The answer appears to be yes - using 3 agents with a structured review process reduced hallucination scores by 96% across 310 test cases. arxiv.org/pdf/2501.13946 - ThreadSky

emollick.bsky.social • 24 days ago

Been waiting for someone to test this and see if it works - can multiple AI agents fact-checking each other reduce hallucinations?

The answer appears to be yes - using 3 agents with a structured review process reduced hallucination scores by 96% across 310 test cases. https://arxiv.org/pdf/2501.13946

1 / 3

Comments

gluonspring.bsky.social•24 days ago

Good to see you and over here, BTW. Let’s hope the scientific and technical discussions pick up here.

gluonspring.bsky.social•24 days ago

I have recently been doing some experiments along these lines (summarizing genetic information) and *anecdotally* this kind of cross checking does seem to filter out a fair amount of junk. Decomposing the results and having the LLM evaluate each chunk separately also seems to help.

sebastianmaki.bsky.social•24 days ago

There you go. Ethical regulator theorem and third order cybernetics for the win! (Sure not exactly a one-to-one comparison, but close enough)

richardhowes.tech•24 days ago

That intuitively makes sense.

I don't understand the mechanism well, so maybe this is happening already, but I would have thought the ability to search the web needs to be somehow integrated into removing hallucinations. 🤷‍♂️

emollick.bsky.social•24 days ago

They report a 28x decrease as 2800% which is a little confusing. Normally we would call that 96% so I deleted and reposted

nathanfavero.com•24 days ago

This still isn’t intuitive to me. They have an index that can take on positive or negative values, and 0 isn’t really a meaningful baseline so far as I can tell. So multiples of how far from zero don’t really help me understand effect size? Maybe I am missing something