Can we learn protein biology from a language model? In new work led by @liambai.bsky.social and me, we explore how sparse autoencoders can help us understand biology—going from mechanistic interpretability to mechanistic biology. - ThreadSky

etowah0.bsky.social • 17 days ago

Can we learn protein biology from a language model?

In new work led by @liambai.bsky.social and me, we explore how sparse autoencoders can help us understand biology—going from mechanistic interpretability to mechanistic biology.

Comments

anush-chp.bsky.social•16 days ago

Congratulations! Really nice and exciting!

etowah0.bsky.social•17 days ago

For more background on mechanistic interpretability and protein language models, check out this previous thread on our work (X link, sorry) https://x.com/liambai21/status/1852765669080879108

etowah0.bsky.social•17 days ago

Using sparse autoencoders (SAEs), we find many interpretable features in a protein language model (pLM), ESM-2, ranging from secondary structure, to conserved domains, to context specific properties. Explore them at https://interprot.com!

etowah0.bsky.social•17 days ago

(Regarding the recent discourse surrounding SAEs, we refer to this thread from @banburismus_)
https://x.com/banburismus_/status/1886125330055782710?t=U0xV3YRnrO1kTg2xVBuphA&s=19

etowah0.bsky.social•17 days ago

We find many features appear highly specific to certain protein families, suggesting pLMs contain an internal notion of protein families which mirrors known homology-based families. Other features correspond to broader, family-independent coevolutionary signals.

etowah0.bsky.social•17 days ago

We categorize SAE latents by family specificity and activation pattern by layer. We find more family specific features in middle layers, and more single token activating (point) features in layer layers. We also study how SAE hyperparameters change what features are uncovered.

etowah0.bsky.social•17 days ago

Having established methods to interpret features, we sought to associate features with functional properties and more generally understand pLM performance on downstream tasks. We train linear models on SAE features and analyze features that contribute the most to task performance.

Comments

Posting Rules

Reply