Having an artificial dataset would be really challenging to maintain. We have a really diverse set of data and users. I can’t imagine we could reflect that group of people accurately. This is important especially now as we’re looking to dramatically change how documents are indexed. - ThreadSky

taidesu.bsky.social • 9 days ago

Having an artificial dataset would be really challenging to maintain.

We have a really diverse set of data and users. I can’t imagine we could reflect that group of people accurately.

This is important especially now as we’re looking to dramatically change how documents are indexed.

Comments

taidesu.bsky.social•9 days ago

I’m almost to the point where I think it would be best to create a really small fine-tuned data set for developing relevancy.

Then we’ll just need to rely on online live relevancy metrics to monitor their success.

dep4b.bsky.social•7 days ago

You could just refresh your static set every quarter. You are right that constant change is hard when you want reproducibility. I have just frozen the dataset and the date and used that.

taidesu.bsky.social•7 days ago

Right but refreshing the data every quarter would also mean re-evaluating judgements. As I anticipate many of them would be dramatically different.

After typing all this out over the last few days I’m feeling like synthetic data may be the best way to go.

taidesu.bsky.social•7 days ago

It’ll give us a stable baseline and we can represent some typical and some challenging retrieval examples that we’ve seen in the past.

dep4b.bsky.social•7 days ago

My concern w synthetic is how do you know it reflects your actual data and queries? If you know it does then heck yeah.

dep4b.bsky.social•7 days ago

Yeah, that is part of the workflow, and if you reevaluate every day or once per quarter or even per year doesn’t change that work.

dep4b.bsky.social•7 days ago

This is why folks looove click based judgements. Versus human.

dep4b.bsky.social•7 days ago

And why https://www.ubisearch.dev exists…. To facilitate that, among other use cases.

dep4b.bsky.social•7 days ago

Then, if you are doing that then you can also do smart statistical sampling so you dont need ALL the data.. which makes experimenting easier.

dep4b.bsky.social•7 days ago

We don’t test brand new drug compounds on people, why do we test relevancy changes in production??

Comments

Posting Rules

Reply