Meet OLMo 2, the best fully open language model to date, including a family of 7B and 13B models trained up to 5T tokens. OLMo 2 outperforms other fully open models and competes with open-weight models like Llama 3.1 8B — As always, we released our data, code, recipes and more 🎁 - ThreadSky

ai2.bsky.social • 213 days ago

Meet OLMo 2, the best fully open language model to date, including a family of 7B and 13B models trained up to 5T tokens. OLMo 2 outperforms other fully open models and competes with open-weight models like Llama 3.1 8B — As always, we released our data, code, recipes and more 🎁

Comments

mechanicaldirk.bsky.social•213 days ago

Absolutely incredible effort. Many fun war stories to come (in the write-up!), but the best part of this job is how the team comes together when it's crunch time.

ai2.bsky.social•213 days ago

Read about the OLMo 2 recipe in our blog: https://allenai.org/blog/olmo2
Download the full OLMo 2 collection, including model weights and data on HuggingFace: https://huggingface.co/collections/allenai/olmo-2-674117b93ab84e98afc72edc
Access the training code on GitHub: https://github.com/allenai/OLMo

ai2.bsky.social•213 days ago

We built OLMo 2 by improving the stability of our long pretraining runs with changes to model architecture and training recipe. We also employ staged training, using learning rate annealing and introducing new, high-quality data (which we call Dolmino) at the end of pretraining.

ai2.bsky.social•213 days ago

Applying our state of the art Tülu 3 post-training recipe, we also built OLMo 2 Instruct, which are competitive with even the best open-weight models—OLMo 2 13B Instruct outperforms Qwen 2.5 14B instruct, Tülu 3 8B, and Llama 3.1 8B Instruct models.