Are Vision Language Models ready for scientific research? 🧑‍🔬🧪 We compared leading VLLMs on the three pillars of chemical and material science discovery: data extraction, lab experimentation and data interpretation. arxiv.org/abs/2411.16955 - ThreadSky

About ThreadSky

Are Vision Language Models ready for scientific research?
🧑‍🔬🧪

We compared leading VLLMs on the three pillars of chemical and material science discovery: data extraction, lab experimentation and data interpretation.
https://arxiv.org/abs/2411.16955

Comments

jablonkagroup.bsky.social•96 days ago

We and M3RG-Group from IIT Delhi created MaCBench: a multimodal materials and chemistry benchmark. (2137 questions)

We focus on the tasks we consider crucial for scientific development, practical lab scenarios, Spectral Analysis, US patents, and more.

jablonkagroup.bsky.social•96 days ago

We observed a striking disparity in performance across tasks. Models can identify lab equipment but struggle with identifying safety violations in real-life laboratory scenarios.

jablonkagroup.bsky.social•96 days ago

But we did not stop there! We dug deeper with ablations to understand the bottlenecks in applicability.
We compared different modalities, multi-step vs single step reasoning, guided prompting, etc.

jablonkagroup.bsky.social•96 days ago

For instance, one would expect vision models to perform very well and better than text models on spatial reasoning - such as identifying the correct isomeric relation between two compounds.

But this is not the case!

jablonkagroup.bsky.social•96 days ago

Supported by Carl Zeiss Foundation, Intel, Merck, Alexander von Humboldt Foundation, Friedrich-Schiller-Universität Jena, IIT Delhi.

📜Manuscript: https://arxiv.org/abs/2411.16955
👩‍💻GitHub: https://github.com/lamalab-org/mac-bench

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply