In April I predicted LLMs were reaching diminishing returns. 𝗔𝗹𝗹 𝘀𝗶𝗴𝗻𝘀 𝗻𝗼𝘄 𝘀𝘂𝗽𝗽𝗼𝗿𝘁 𝘁𝗵𝗮𝘁. 6 related predictions below for 2024 also were correct. That leaves 1 open: 𝗱𝗶𝗺𝗶𝗻𝗶𝘀𝗵𝗶𝗻𝗴 𝗿𝗲𝘁𝘂𝗿𝗻𝘀 𝗹𝗲𝗮𝗱 𝘁𝗼 𝗰𝗼𝗹𝗹𝗮𝗽𝘀𝗲 𝗼𝗳 𝗚𝗲𝗻𝗔𝗜 𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻𝘀. Hasn’t happened yet but could—as the implications of slowdown set in. - ThreadSky

garymarcus.bsky.social • 108 days ago

In April I predicted LLMs were reaching diminishing returns. 𝗔𝗹𝗹 𝘀𝗶𝗴𝗻𝘀 𝗻𝗼𝘄 𝘀𝘂𝗽𝗽𝗼𝗿𝘁 𝘁𝗵𝗮𝘁. 6 related predictions below for 2024 also were correct.

That leaves 1 open: 𝗱𝗶𝗺𝗶𝗻𝗶𝘀𝗵𝗶𝗻𝗴 𝗿𝗲𝘁𝘂𝗿𝗻𝘀 𝗹𝗲𝗮𝗱 𝘁𝗼 𝗰𝗼𝗹𝗹𝗮𝗽𝘀𝗲 𝗼𝗳 𝗚𝗲𝗻𝗔𝗜 𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻𝘀.

Hasn’t happened yet but could—as the implications of slowdown set in.

Comments

johnsmith4real.bsky.social•105 days ago

I predicted diminishing returns in Aug 2023. But the diminishing returns are on parameter count. There are still plenty of ways to expand LLM performance. Attention and Agents are the two directions I would keep an eye on.

goldsmif.bsky.social•108 days ago

Short sellers are circling already.

doriantaylor.com•108 days ago

thinking that the invention of the transformer enabled people to just throw money at the situation and that very quickly maxed out; will need an invention of similar proportions to get out of that rut

doriantaylor.com•108 days ago

(and like what is that, optical compute? nano-ram? quantum?)

shituationist.bsky.social•108 days ago

The problem is philosophical, not something computer science can address without giving up the computationalist aim of colonizing all other academic disciplines.

johnsontoddr.bsky.social•105 days ago

I see two areas: (1) using what we have in a collaborative fashion with humans, similar to a several year old distill paper out of Google; and (2) exploring use of LLMs and related tech as parts of cognitive architectures ala SOAR and ACT-R.

doriantaylor.com•105 days ago

what is the principal bottleneck for either of these?

johnsontoddr.bsky.social•105 days ago

Both Soar and ACT-R are inherently symbolic, though ACT-R has an underlying statistical component. But they represent the more symbolic, deliberative models of cognition, not the "subsymbolic". Embeddings and DL perceptual models could augment their basic framework.

worldnotwords.bsky.social•105 days ago

Gary, what are your honest thoughts on the active inference approach to AI? Is this path based on how the human brain works a better path to AGI?

johnsontoddr.bsky.social•105 days ago

Perhaps another AI Winter? Though I suspect we will have a long Spring, since most of our PhD students now seem to be giving presentations with titles like "Application of LLMs to... [pick your favorite ML task in healthcare]"

jkirshbaum.bsky.social•103 days ago

@garymarcus.bsky.social in 2019 you were claiming that LLMs couldn't learn language because of the infinite possibilities of grammatically correct sentences:
https://open.spotify.com/episode/0vIojNeWzo3EGLuUSZgZUY?si=YkkRc32UROuO0Q_DZfAE-Q

In 2018 you were claiming that deep learning couldn't learn hierarchical concepts:
https://arxiv.org/abs/1801.00631

jkirshbaum.bsky.social•103 days ago

No issue being wrong (don't ask me what I was saying about AI in 2018...basically nothing lol), but you seem to have jumped from being incorrect about limitations to alarmist about capabilities. It comes off as disingenuous. Where do you update and recalibrate? Did you?

cameronwilliams.bsky.social•108 days ago

While I agree with you on the real valuation, I’ve learned over the years it’s not safe to put a timeline for market value.

The market timing is never that certain.

garymarcus.bsky.social•108 days ago

Agreed

danielbayley.dev•107 days ago

The market can remain irrational longer than you can remain solvent. But some Michael Burry type will make a killing shorting NVDA stock.

cameronwilliams.bsky.social•107 days ago

Exactly. That’s why I fully agree with the intent behind the last prediction, but think setting a timeline isn’t that simple.

In the case of OpenAI, they do a great job at making success always look close. It’ll be a very long time before they’re priced on fundamentals.

brucemarino.bsky.social•104 days ago

I understand your point . But whatever.. llms are very useful (although they do not think as humans). I use copilot everyday and helps me to improve my performance.. so why so much pessimistic wave?

Comments

Posting Rules

Reply