✨ How good are AIs at writing Svelte 5 code? I decided to test the newest models and find out. More info soon, but the tldr is: Gemini 2.5 Pro is best, all OpenAI models do poorly. Base results: khromov.se/svelte-bench With llms-small.txt: khromov.se/svelte-bench... #svelte - ThreadSky

About ThreadSky

khromov.se • 29 days ago

✨ How good are AIs at writing Svelte 5 code? I decided to test the newest models and find out. More info soon, but the tldr is: Gemini 2.5 Pro is best, all OpenAI models do poorly.

Base results:
https://khromov.se/svelte-bench

With llms-small.txt:
https://khromov.se/svelte-bench-context

#svelte

Comments

khromov.se•29 days ago

If you're curious about the code and benchmarks, it's available here:
https://github.com/khromov/svelte-bench

stuartbritt.bsky.social•28 days ago

Those numbers are depressing. I really love svelte, the model just sits so much easier with me than React. But I'm trying to write something that's going to require a *lot* of code, and I can't help feeling that React with AI help will get me there quicker, even though I don't like it 😞

realfakenerd.bsky.social•26 days ago

Well, I'm using Gemini 2.5 with the llm-fullk.txt and it gets right most of the code it writes for repetitive tasks

ollema.xyz•28 days ago

interesting that $inspect tests are never passing! wonder why that is so hard. maybe it’s not that common in the wild yet

thiagolino8.bsky.social•28 days ago

I bet not much code is commited to github with $inspect

benmccann.com•27 days ago

Yeah, it's also because $inspect.trace must be the first line in a function or it fails. I just added a comment to the docs to help the next version of Gemini. Really you're never going to want this generated in production code though, so I think it's not that important relative to snippets, etc.

ollema.xyz•27 days ago

@khromov.se maybe some weighting of the tests would make sense if you are going to add some aggregated scores at some point. $state is more important than $inspect and so on

khromov.se•27 days ago

I'm open to it, there's no "final score" currently but one can be calculated. I'm also interested in expanding the test suite because it's pretty basic currently. You are welcome to help out!

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply