In the first stage, 'symbol abstraction heads' convert to input tokens to abstract variables, based on their relations with other tokens. Interestingly, this implements an emergent form of the 'Abstractor' architecture that we recently proposed for relational reasoning arxiv.org/abs/2304.00195 (5/N) - ThreadSky

taylorwwebb.bsky.social • 21 days ago

In the first stage, 'symbol abstraction heads' convert to input tokens to abstract variables, based on their relations with other tokens. Interestingly, this implements an emergent form of the 'Abstractor' architecture that we recently proposed for relational reasoning https://arxiv.org/abs/2304.00195 (5/N)

Comments

taylorwwebb.bsky.social•21 days ago

In the second stage, 'symbolic induction heads' predict the abstract variable associated with the next token, implementing a symbolic variant of the 'induction head' mechanism that has been tied to in-context learning https://arxiv.org/abs/2209.11895 (6/N)

taylorwwebb.bsky.social•21 days ago

In the third stage, 'retrieval heads' retrieve the token associated with the predicted variable (effectively inverting the symbol abstraction heads). (7/N)

taylorwwebb.bsky.social•21 days ago

Using causal mediation analyses, we confirmed the presence of these heads in a series of three stages: abstraction heads were present in early layers, symbolic induction heads were present in middle layers, and retrieval heads were present in later layers. (8/N)

taylorwwebb.bsky.social•21 days ago

Using attention analyses, we confirmed that these heads' attention patterns were consistent with the hypothesized mechanisms. For instance, for symbolic induction heads, attention was primarily directed to tokens that instantiated the same abstract variable as the next token. (9/N)

taylorwwebb.bsky.social•21 days ago

Using representational similarity analyses, we confirmed that the outputs of these attention heads represented the hypothesized variables: the abstraction heads and symbolic induction heads primarily represented abstract symbols, whereas the retrieval heads primarily represented tokens. (10/N)

taylorwwebb.bsky.social•21 days ago

Finally, using ablation analyses, we confirmed that these attention heads were necessary to perform the abstract reasoning task. (11/N)

Comments

Posting Rules

Reply