Static Embedding models have been around since before Transformers (e.g. GLoVe, word2vec), they work with pre-computed word embeddings from a mapping.

I apply this simple architecture, but train it like a modern embedding model: Contrastive Learning with Matryoshka support.

🧵