"Large language models surpass human experts in predicting neuroscience results" w @ken-lxl.bsky.social and braingpt.org. LLMs integrate a noisy yet interrelated scientific literature to forecast outcomes. nature.com/articles/s41... 1/8 - ThreadSky

profdata.bsky.social • 88 days ago

"Large language models surpass human experts in predicting neuroscience results" w @ken-lxl.bsky.social
and http://BrainGPT.org. LLMs integrate a noisy yet interrelated scientific literature to forecast outcomes. https://nature.com/articles/s41562-024-02046-9 1/8

Comments

hugospiers.bsky.social•87 days ago

Congratulations Brad! Very cool.

profdata.bsky.social•88 days ago

To test, we created BrainBench, a forward-looking benchmark that stresses prediction over retrieval of facts, avoiding LLM's "hallucination" issue. The task was to predict which version of a Journal of Neuroscience abstract gave the actual result. 2/6

profdata.bsky.social•88 days ago

All 15 LLMs considered crushed human experts at BrainBench's predictive task. LLMs correctly predicted neuroscience results (across all sub areas) dramatically better than human experts, including those with decades of experience. 3/8

profdata.bsky.social•88 days ago

There were no signs of leakage from the training to test set. We performed standard checks. In follow-up work, we trained an LLM from scratch to rule out leakage; even this smaller model was superhuman on BrainBench https://arxiv.org/abs/2405.09395 4/8

profdata.bsky.social•88 days ago

In the Nature HB paper, both human experts and LLMs were well calibrated - when they were more certain of their decisions, they were more likely to be correct. Calibration is beneficial for human-machine teaming. 5/8

profdata.bsky.social•88 days ago

Indeed, follow-up work on teaming finds that joint LLM and human teams outperform either alone, because LLMs and humans make different types of errors. We offer a simple method to combine confidence-weighted judgements.
https://arxiv.org/abs/2408.08083 6/8

profdata.bsky.social•88 days ago

Finally, LLMs can be augmented with neuroscience knowledge for better performance. We tuned Mistral on 20 years of the neuroscience literature using LoRA. The tuned model, which we refer to as BrainGPT, performed better on BrainBench. 7/8

drussellmrichie.bsky.social•87 days ago

Have you done any analysis of the errors? What does the LLM get right and the human wrong, and vice versa?

laurituominen.bsky.social•87 days ago

Amazing! Could you use this to rate grant proposals?

manpiazza.bsky.social•87 days ago

If I get this right, the system gets the “intro” and “methods” and guess the correct abstract. Does this not “simply” depend on the fact that when authors write their introduction they discuss the literature in a way to enhances the expectations of towards the results that they actually get?

lauragrima.bsky.social•87 days ago

This was also my intuition (but maybe partly also the converse, when authors set up some existing model/state of the field and then discuss how their results are counter to that and are thus ‘novel’?)

humblyalex.bsky.social•88 days ago

Nice job! Was there a key reason for choosing Mistral?

profdata.bsky.social•87 days ago

A 7B is small enough to train efficiently on 4 A100s (thanks Microsoft) and at the time Mistral performed relatively well for its size.

humblyalex.bsky.social•87 days ago

Thank you! I've heard good things. If it worked well for you, I think that's the last evidence I need to go with for a couple of my own projects.

Do you mind me asking what it cost for the fine-tuning?

peiferlabunc.bsky.social•87 days ago

Low bar.... 😉

arktemis.bsky.social•87 days ago

Hope this is applied one day soon to neurological disorders like migraine.

ananeurogeek.bsky.social•87 days ago

📌

vinuarumugham.bsky.social•85 days ago

https://docs.google.com/presentation/d/1E0dzpL3RxUP649Rr9KHdkx-StOGrqYRPjAo3W1e_wX4/edit#slide=id.g3196325d0d0_0_0

norcalneuro.bsky.social•87 days ago

Did you partition human experts by field? A cognitive neuroscientist who scores 100% on that category, but chance on the rest, would look pretty bad overall. Is it the case that "neuroscience" is simply too broad a field? Sorry if I missed this in the paper somewhere..

pessoabrain.bsky.social•88 days ago

Need to take a careful look but is BrainGPT available to users??

jrtannock.bsky.social•87 days ago

https://Braingpt.org

profdata.bsky.social•87 days ago

Yes, the model weights and all materials are openly available. We really want to offer easy to use tools people can use through the web without hassle. To do that, we need to do more work (will be announcing an open source effort soon) and need some funding for hosting a model endpoint.

pessoabrain.bsky.social•87 days ago

congrats on the work!

ericleonardis.bsky.social•87 days ago

@danieljbutler.bsky.social

annabhlr.bsky.social•87 days ago

Congrats! It is so cool to see this out! Happy that I could lend a hand creating test cases.

Comments

Posting Rules

Reply