[LLMs]
@t3rminus Vectors (in this context), are numerical representations of words, capturing their meaning.
An analogy someone gave me: "Boy" and "Girl" can be represented as points in a multi-dimensional space. The difference between these points can be expressed as a vector, essentially […]
@t3rminus Vectors (in this context), are numerical representations of words, capturing their meaning.
An analogy someone gave me: "Boy" and "Girl" can be represented as points in a multi-dimensional space. The difference between these points can be expressed as a vector, essentially […]
Comments
@anthropy hmmm. That’s a good analogy! So the vectors are more specially relationships between words. (Or I presume “tokens”.)
So a string of input text becomes a graph of vectors of relationships between tokens?
Egh. I need to think about this. It’s wild.
@t3rminus Yea basically! Each vector has one or more dimensions, which are a feature of the word(/token). Dimensions could be gender, royalty, etc.
A word's vector is like its coordinates in this multi-dimensional "meaning space".
"King" would have high values in "royalty" and "male […]
@t3rminus this of course, still leaves out parts though, because how does it turn this into e.g a followup sentence?
That's where the 'multi headed attention mechanism' comes in: It allows the LLM to weigh the importance of each word in the input relative to all others, across multiple […]