🤔Do Vision and Language Models Share Concepts? 🚀
We present an empirical evaluation and find that language models partially converge towards representations isomorphic to those of vision models. #EMNLP
📃 https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00698/124631
We present an empirical evaluation and find that language models partially converge towards representations isomorphic to those of vision models. #EMNLP
📃 https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00698/124631
Comments
how can they not understand everything in the same way?
🎯We measure the alignment between vision models and LMs by mapping their vector spaces and evaluating retrieval precision on held-out data.
🧵(2/8)
✨LMs converge toward the geometry of visual models as they grow bigger and better.
🧵(3/8)
🔍Our experiments show the alignability of LMs
and vision models is sensitive to image and language dispersion, polysemy, and frequency.
🧵(4/8)
🧵(5/8)
- the LM understanding debate
- the study of emergent properties
- philosophy
🧵(6/8)
1. Representation spaces of LMs and VMs grow more partially similar with model size.
2. Lower frequency, polysemy, dispersion can be easier to align.
3. Shared concepts between LMs and VMs might extend beyond nouns.
🧵(7/8)
#NLP #NLProc