New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes Previous record: 5.03 minutes Changelog: - FlexAttention blocksize warmup - hyperparameter tweaks - ThreadSky

About ThreadSky

fernbear.bsky.social • 91 days ago

New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes

Previous record: 5.03 minutes
Changelog:
- FlexAttention blocksize warmup
- hyperparameter tweaks

Comments

ethansmith2000.com•91 days ago

Absolutely sick!

fernbear.bsky.social•91 days ago

Reproducible log:
https://github.com/KellerJordan/modded-nanogpt/blob/master/records/112424_WindowWarmup/ba299b7e-a36a-4fd8-a268-25bb772010dd.txt

fernbear.bsky.social•91 days ago

This record modifies the previous FlexAttention record by adding a warmup to the sliding window size over the course of training. Since our network can't necessarily utilize long-term connections at the start of training, this yields a pretty good boost to training speed!

fernbear.bsky.social•91 days ago

The previous record used a sliding window size of 1024, whereas this record starts at a window size of 64 and linearly anneals it to 1792 over the course of training. This means we can spend more time learning longer context lengths where it matters -- at the end of training!

fernbear.bsky.social•91 days ago

(This also means that for this record, not only is the number of steps is lower -- the average stepsize is faster as well!)

fernbear.bsky.social•91 days ago

As a result of the better match between context length during training and the network's learning dynamics, we can train a bit faster! With the increased context length, we are able to remove 125 steps, making this network 6.7% more data efficient than before.

fernbear.bsky.social•91 days ago

(Now, a note on scoring -- this is because the modded-nanoGPT speedrun scores the model by the lowest possible validation loss as long as it is causal, vs a fixed attention context length...

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply