In conversation at #CHR2024 the question that came up often was: If we solve a problem by prompting, how do we know when we have optimized the solution? You can’t do a grid search to find optimal hyperparameters if your search space is All of Language. - ThreadSky

tedunderwood.me • 83 days ago

In conversation at #CHR2024 the question that came up often was: If we solve a problem by prompting, how do we know when we have optimized the solution? You can’t do a grid search to find optimal hyperparameters if your search space is All of Language.

Comments

mrdrozdov.com•83 days ago

You should checkout dspy!

The correct answer is not to do a grid search at all, but treat this like an optimization problem.

lauraknelson.bsky.social•83 days ago

We can also use some sort of saturation measure as well (similar to what qualitative researchers use)? Keep prompting until you're not getting anything new, and then search the output space for convergences. (E.g., in qual research stop interviewing when you're getting no new insights from n+1).

tedunderwood.me•83 days ago

Yep, that's a good idea. I feel like there's a methods article to be written here eventually.

lauraknelson.bsky.social•83 days ago

💯

lucy3.bsky.social•83 days ago

grid search < random search for typical model hyperparameters, and prompts are now just another one, so when searching through prompts I've had collaborators create a spreadsheet and instructed everyone to "go nuts" to propose prompts

tedunderwood.me•83 days ago

Whoa, this is a cool insight. How did I miss the news that grid search < random search? How long have we known that?

I guess this could be a viable solution as long as we feel some confidence that our prompts are spread out well in the space of possibility, not all clustered in one corner.

yetanotheruseless.com•83 days ago

I think there’s a great old-days YouTube by Andrew Ng which talks about this (at least that’s where I first encountered it, circa maybe 2012 or so?)

yetanotheruseless.com•83 days ago

Of course: doing this well still usually implies searching on a hyper parameter space with a good metric: you want to do some kind of knn+expand epsilon-balls around the best few points (or centroids between)

For many hyperparam spaces deciding which axes are log and which are linear is nonobvious

lauraknelson.bsky.social•83 days ago

Not to be that person (but actually I'm happy to be that person), the metric needs to also include insight. As we've known since the ancient times of topic modeling, the mathematically optimal solution is not always the solution that helps answer the (social science) question (DiMaggio et al. 2013).

yetanotheruseless.com•83 days ago

100%! The metric isn’t a metric if nearby hyper parameter points aren’t expected to behave similarly!

lucy3.bsky.social•83 days ago

I learned this in college (I remember in one of my ML classes, we spent like 80% of our time hyperparameter tuning until our numbers hit what they told us to hit).

Random Search for Hyper-Parameter Optimization 2012 https://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf

tedunderwood.me•83 days ago

Thanks!

lucy3.bsky.social•83 days ago

but both grid search and random search only work if the search space is defined, which is where the "All of Language" challenge comes into play (but really is it all of language? we kind of can smudge some boundaries over possibilities, e.g. what is reasonable to instruct people when doing tasks)

tedunderwood.me•83 days ago

Got it. Your second post anticipated my reply.

tedunderwood.me•83 days ago

Someone at CHR — I forget if David or Maria or someone else — was also telling me that DSPy is designed to automate this kind of search, even if the implementation is imperfect.

arnicas.bsky.social•83 days ago

it was lynn :)

tonyhschu.bsky.social•82 days ago

This was the spark I needed to go look at DSPy again. Thank you!

tedunderwood.me•83 days ago

Aha! Credit where due.

Comments

Posting Rules

Reply