Profile avatar
neelrajani.bsky.social
PhD student in Responsible NLP at the University of Edinburgh, passionate about MechInterp
21 posts 563 followers 467 following
Getting Started
Conversation Starter
Posts 5 Comments 18

Caught off-guard by the Llama 3.3 release? This is the loss of Llama-3.3-70B-Instruct (4bit quantized) on its own Twitter release thread. It really didn't like ' RL' (loss of 13.47) and wanted the text to instead go "... progress in online learning, which allows the model to adapt"

submitted 81 days ago • 0 comments

Do instruct models store factual associations differently than base models? 🤔 Doesn't look like it! When adapting ROME's causal tracing code to Llama 3.1 8B, the plots look very similar (base on top, instruct at the bottom). Note the larger sample size for instruct: If the "correct prediction" 1/3

submitted 87 days ago • 1 comment

How do LLMs learn to reason from data? Are they ~retrieving the answers from parametric knowledge🦜? In our new preprint, we look at the pretraining data and find evidence against this: Procedural knowledge in pretraining drives LLM reasoning ⚙️🔢 🧵⬇️

submitted 100 days ago • 38 comments

Hello to all #ICLR reviewers on #MLsky

submitted 95 days ago • 0 comments

1/2 The original 2022 ROME paper by Meng et al.:

submitted 95 days ago • 1 comment