Wrote up some notes on Microsoft's new Phi-4 LLM. They trained it on a LOT of synthetic data, and the details of how and why they did that are really interesting. simonwillison.net/2024/Dec/15/... - ThreadSky

simonwillison.net • 83 days ago

Wrote up some notes on Microsoft's new Phi-4 LLM. They trained it on a LOT of synthetic data, and the details of how and why they did that are really interesting.
https://simonwillison.net/2024/Dec/15/phi-4-technical-report/

Comments

rigbythedog.bsky.social•82 days ago

Will this one take all of our jobs or fire nuclear missiles?

simonwillison.net•82 days ago

It's a 9GB download that's basically a souped up version of an iPhone predictive text keyboard, so I remain skeptical of its abilities to pull a Terminator

rigbythedog.bsky.social•82 days ago

Then you guys need to get to it! Develop something that’ll take all the jobs or destroy the planet, and hurry up!

ttkciar.bsky.social•82 days ago

There are other open-weight models which pose a moderate threat to some jobs, like Gemma2-27B, Qwen2.5-72B, and Mistral-Large-Instruct-2411, but overall this threat is overhyped.

In the short term, too, there is a deficit of deployed compute infrastructure for putting these models to work.

rigbythedog.bsky.social•82 days ago

You cannot convince me that computer technology is not fucking everything up.

ttkciar.bsky.social•83 days ago

My hypothesis is that they opened the weights of the Phi family of models in order to demonstrate the effectiveness of their synthetic training dataset technology (in particular Evol-Instruct), which they plan on licensing to other model authoring companies.

masoudmaani.bsky.social•83 days ago

it's a 33.6GBs model as it goes by your evals, probably larger than 14B as 14B would be 27GBs or sth.
I wonder why microsoft always under-sales its models parameter numbers. and calls it a small model too. smh.
synthetic data is now industry standard, glad everyone caught up to 2018 finally.

deeksbro.bsky.social•83 days ago

TL;DR Excerpt Author highlight/
Msft AI Foundry: why synth data is beneficial

In organic datasets… relationship btwn tokens is complex & indirect…[while] that generated by a lang model is 👉by definition predicted by the preceding tokens…easier for a model to follow the resulting reasoning patterns👈

Comments

Posting Rules

Reply