The Scaling Paradox: AI capabilities have improved remarkably quickly, fuelled by the explosive scale-up of resources being used to train the leading models. But the scaling laws that inspired this rush actually show very poor returns to scale. What’s going on? 1/ www.tobyord.com/writing/the-... - ThreadSky

About ThreadSky

tobyord.bsky.social • 46 days ago

The Scaling Paradox:
AI capabilities have improved remarkably quickly, fuelled by the explosive scale-up of resources being used to train the leading models. But the scaling laws that inspired this rush actually show very poor returns to scale. What’s going on?
1/
https://www.tobyord.com/writing/the-scaling-paradox

Comments

davidmanheim.alter.org.il•43 days ago

See my response here:
https://davidmanheim.substack.com/p/is-the-scaling-paradox-meaningful

tobyord.bsky.social•46 days ago

On the one hand, AI Scaling is Shockingly Impressive.
AI scientists such as Ilya Sutskever and Dario Amodei saw the scaling laws and doubled down on them, rapidly improving cutting edge AI capabilities just via more and more data and compute.
2/

tobyord.bsky.social•46 days ago

This catapulted OpenAI into the lead and the public spotlight, while other labs and companies raced to adopt this same strategy to keep up.
3/

tobyord.bsky.social•46 days ago

Scaling has clearly worked, leading to very impressive gains in AI capabilities over the last five years. Many papers and AI labs trumpet the success of this scaling in their latest results and show graphs suggesting that the gains from scaling will continue (possibly via inference compute).
4/

tobyord.bsky.social•46 days ago

For example, here is a chart from OpenAI’s release of their o1 model, captioned: ‘o1 performance smoothly improves with both train-time and test-time compute’.
5/

tobyord.bsky.social•46 days ago

But because this chart has a logarithmic x-axis, it is actually showing us that performance is scaling roughly logarithmically. This means exponentially more resources are needed to keep performance increasing at a linear rate.
6/

tobyord.bsky.social•46 days ago

Exponential resource usage for linear gains is typically considered to be a sign the problem is intractable, or at least is intractable to the approach you are using. Indeed, exponential resource use is characteristic of a brute force approach.
7/

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply