Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵 - ThreadSky

jfrankle.com • 76 days ago

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

Comments

muledupape.bsky.social•76 days ago

Hi, do people worry about climate change there ? (in general and consequences of energy-greedy IA ?)

jfrankle.com•76 days ago

TLDR: AI progress will continue. Scaling laws, formally, may or may not, but that doesn't matter commercially. What matters is ROI: are lifecycle costs of AI worth it? That's hard to measure and predict. Even in the worst case, though, I think we're in for decades of innovation.

jfrankle.com•76 days ago

None of this is novel. Smarter people than me are saying the same thing. But it merits repeating as everyone (students, VCs, scientists, practitioners) nervously swings from hype to "it's all over" to "we're so back." Admire the progress. Focus on making this stuff as useful as possible

jfrankle.com•76 days ago

1. Of course AI is going to keep getting better. Technological progress isn't going to stop or "hit a wall." That's very safe to say. There's a ton of low-hanging fruit.

jfrankle.com•76 days ago

2. Scaling "laws" ("trends"?) as formally defined (log loss decreases linearly with log training FLOPs when properly allocated between parameters and data) may or may not continue. There's no reason to doubt that it will continue, yet extrapolating far out is uncertain.

lzamparo.bsky.social•76 days ago

No reason to doubt where? In some domains there's reasons to doubt: https://arxiv.org/abs/2411.02796

yetanotheruseless.com•76 days ago

Is this even relevant to the original concept of the Chinchilla scaling law? These laws never said that every possible task would uniformly get better as the self-supervised next token prediction got better, right?

They have generally tended to improve but there was never a “task-independent” law

jfrankle.com•76 days ago

3. Even if this precise (and very aggressive) scaling trend didn't continue for whatever reason (and there could be many reasons), the technology will continue to improve. Just at a slower rate as measured in this very specific way. We may find new ways to scale with nice properties. We may not.

jfrankle.com•76 days ago

4. Either way, this particular trend doesn't directly say much about the overall success of AI. It doesn't account for the full cost or benefits. It ignores the full cost of training (data, etc), inference costs (how much is it used), and value (what is reducing loss getting you on real tasks?)

mercer-kernel.bsky.social•76 days ago

great thoughtful post @jfrankle.com. respectfully, don't agree that "bitter lesson is over". sepp hochreiter, no doubt a giant of the field, missed it. below is what another giant of the field (michael i jordan) said about deep learning for NLP in his reddit AMA back in 2015 and here we are ...

mercer-kernel.bsky.social•76 days ago

"Although current deep learning research tends to claim to encompass NLP, I’m (1) much less convinced about the strength of the results, compared to the results in, say, vision;

mercer-kernel.bsky.social•76 days ago

(2) much less convinced in the case of NLP than, say, vision, the way to go is to couple huge amounts of data with black-box learning architectures."

Comments

Posting Rules

Reply