New blogpost: "Training as we know it might end". It was originally a panorama of the new methods of synthetic generation but the stakes are now much higher and I openly wonder if model training is not soon going to change forever. vintagedata.org/blog/posts/t... - ThreadSky

dorialexander.bsky.social • 49 days ago

New blogpost: "Training as we know it might end".
It was originally a panorama of the new methods of synthetic generation but the stakes are now much higher and I openly wonder if model training is not soon going to change forever. https://vintagedata.org/blog/posts/training-as-we-know-it-will-end

Comments

dorialexander.bsky.social•49 days ago

One of the most significant development has been the emergence on complex synthetic pipelines/playground, initially to deal with the issue of training agents on unwritten data and now appearing as a shortcut to train directly models on reasoning exercises and heuristics.

dorialexander.bsky.social•49 days ago

All this makes me reconsider radically what future models could be. Likely much smaller, data effective and opinionated. And directly trained on simulated data without requiring pretraining.

tedunderwood.me•49 days ago

Fascinating read.

I see that "The physics of language models" is one key source for this line of thought.

Is another source, perhaps, recent papers showing that a small number of excellent examples of a heuristic can have huge benefits in training data? I unfortunately forget the citations.

austegard.com•49 days ago

I had Claude compare your post to the Absolute Zero paper from yesterday and its conclusion was “Blog Post as Prophecy”…

https://arxiv.org/pdf/2505.03335

bnerlich.bsky.social•49 days ago

Is 'synthetic playground' a new concept in AI? Who used it first?

dorialexander.bsky.social•49 days ago

Kind of. Really seeing it described in detail for the first time in Physics of Language Models 4.1. I used "emulators" with roughly the same meaning in a previous post.

slckl.bsky.social•48 days ago

The synthetic approaches do make the capabilities less magical. What previously magically fell out of web-scale data can now be traced back to an intentional training pipeline.

slckl.bsky.social•48 days ago

I feel like something has to give if we drop most of pretraining (data). Does creativity suffer? Can you bootstrap creativity via a synth pipeline? Is there a synth pipeline version of human empathy? Seemingly hard to verify?

Maybe we'll find out...

dorialexander.bsky.social•48 days ago

"Can you bootstrap creativity via a synth pipeline?" => yes, though not with pure synth.

jebediah98.bsky.social•48 days ago

Two thoughts: 1. Pre training will remain I think. There is evidence that RL is just rebiasing what the model already knows. 2. Phi is already trained on synth data and is actually really good.

I agree on data selection being v important. Not too sure everything is gonna be RL gyms.

dorialexander.bsky.social•48 days ago

For 1. true but that’s why I’m thinking that what is currently mid-training will replace pretraining. 2. Phi 4 is more synth/organic hybrid, currently similar to Qwen 3.

Comments

Posting Rules

Reply