(1) Vision language models can explain complex charts & decode memes, but struggle with simple tasks young kids find easy - like counting objects or finding items in cluttered scenes! Our 🆒🆕 #NeurIPS2024 paper shows why: they face the same 'binding problem' that constrains human vision! 🧵👇 - ThreadSky

(1) Vision language models can explain complex charts & decode memes, but struggle with simple tasks young kids find easy - like counting objects or finding items in cluttered scenes! Our 🆒🆕 #NeurIPS2024 paper shows why: they face the same 'binding problem' that constrains human vision! 🧵👇

Comments

thisisadax.bsky.social•137 days ago

(2) The binding problem refers to difficulties in maintaining correct associations between features (like color & shape 🖍️⬛️) when representing multiple objects over the same representational substrate. These difficulties are a consequence of interference in parallel processing systems.

thisisadax.bsky.social•137 days ago

(3) To investigate this, we tested VLMs on classic visual search tasks. They excel at finding unique objects (e.g., one green shape among red shapes 🔴🟢🔴🔴). But searching for specific feature combinations? Performance drops substantially - similar to people when under time pressure.

thisisadax.bsky.social•137 days ago

(4) Both multimodal LMs & text-to-image models show strict capacity limits - similar to human 'subitizing' limits during rapid parallel processing. Key finding: They improve with visually distinct objects, suggesting failures stem from feature interference.

thisisadax.bsky.social•137 days ago

(5) We developed a scene description benchmark inspired by visual working memory tasks to more directly evaluate how feature overlap affects performance. Key finding: Errors spike when objects share overlapping features - driven by 'illusory conjunctions' where features get mixed up!

thisisadax.bsky.social•137 days ago

(6) Finally, we found that breaking 🪚🔨 visual analogy tasks into smaller chunks (i.e. performing object segmentation) to mitigate the influence of feature interference improves performance on those tasks.

thisisadax.bsky.social•137 days ago

(7) The punchline? Capacity limits aren't just about the number of objects - they stem from interference between representations when processing multiple things at once. This 'binding problem' creates fundamental constraints on parallel processing in both humans and VLMs🧍‍♂️🤖 .

dmewes.com•135 days ago

arXiv link in case anyone else is looking for it: https://arxiv.org/abs/2411.00238v1

dmewes.com•135 days ago

I think the feature binding problem is not given nearly the attention is deserves. Interesting work!

dmewes.com•135 days ago

(incidentally, I'm working on a blog post about the binding problem - hopefully will be out in the next couple weeks at https://amongai.com)

davidhuang.blog•135 days ago

📌

tachikoma.elsewhereunbound.com•136 days ago

if i understand correctly, a model like o1 that can think for longer should be able to overcome such limitations?

nhlism.bsky.social•136 days ago

If it can repeatedly attend over the image tokens, yes. Maybe vision models need their “think step by step” moment

najarvg.bsky.social•137 days ago

Excellent work! Thanks for sharing. Skeets like this make me hanker for a bookmark feature in #bluesky

Comments

Posting Rules

Reply