How clear is this headline? If you've never heard of Evalite, what questions do you have? - ThreadSky

mattpocock.com • 81 days ago

How clear is this headline?

If you've never heard of Evalite, what questions do you have?

Comments

I think you could learn some neat copywriting tricks from this site to spice it up a bit https://marketingexamples.com. 👀

andwrobs.com•80 days ago

why no API key required?

free?

mattpocock.com•80 days ago

Free!

Open source!

Most of the competitors in this space require API keys and credit card details.

andwrobs.com•80 days ago

with zero context my follow-up question is: how can I run an eval on the models I use without providing my API key, even if locally, to the tool

not sure if helpful, cheers!

mattpocock.com•80 days ago

Thanks!

Ofc you need an API key for your models, but not to access the eval platform itself (most require an API key).

mattpocock.com•81 days ago

Assume that many of you will ask 'what are evals'.

If so:

https://www.aihero.dev/what-are-evals

i.am.brandonfannin.com•80 days ago

This was a helpful read. My main question is, is this a local vite server only? Could evalite be added to a build pipeline in circle ci for instance to run evals when relevant code is changed?

josef.dev•81 days ago

ngl this whole time I thought you were talking about eval() and I was wondering what you were cooking 😅

samalrasmussen.bsky.social•80 days ago

Yeah I need this article to even know why or if I should even care.

iamaamins.bsky.social•80 days ago

A full landing page would help me understand the product better.

I am in tech and building stuff with AI, still don't understand what I need Evalite for. Maybe I am not in the target audience.

No hard feelings, mate.

mattpocock.com•80 days ago

Do you run evals?

iamaamins.bsky.social•80 days ago

I don't! Now I get it why the titles didn't make sense to me.

nimobeeren.com•80 days ago

Where do I try it?

mattpocock.com•80 days ago

Here!

https://github.com/mattpocock/evalite

tlbvr.com•81 days ago

If this targets language models specifically, perhaps worth clarifying? Unless it's done afterwards.

mattpocock.com•81 days ago

I'm hoping that the phrase "GenAI-powered apps" carries that weight.

I.e. that when you integrate GenAI into your apps (i.e. calling OpenAI) you need some mechanism of testing the outputs. That's what Evalite does.

tlbvr.com•81 days ago

Yeah I guess to me Gen AI isn't only text generations. Images and other content as well would be included in generative ai. I haven't seen too much of what you're building so this is just my two cents. Happy to see more TS products in the space.

mattpocock.com•81 days ago

Ah I understand your question better now - yes, you're right. Evalite will be targeting all GenAI, not just LLM's.

tlbvr.com•81 days ago

In that case, i think the headline is clear!

andy-barbour.bsky.social•80 days ago

Are GenAI-powered apps ones that integrate API calls to AI agents or LLMs into the application itself (as opposed to applications built through prompt engineering?)

mattpocock.com•80 days ago

Yes, exactly

often.codes•81 days ago

i bet it's not that hard right?

adam21.bsky.social•81 days ago

Maybe just me but "The TypeScript-native, local-first tool for testing LLM-powered apps" made more sense

mattpocock.com•80 days ago

"local-first" is suboptimal. Currently it's local-only.

yleflour.me•56 days ago

I think it's missing the fact that it's "plug & play" and doesn't require you to re-write your entire architecture to accommodate to it

Comments

Posting Rules

Reply