I think a lot of these math benchmarks are absurd. It seems people are chasing the score without thinking about what we're actually trying to do. Not getting 100% on 3 digit times a 3 digit but getting a 6 digit x 4 digit should make us question everything. Something fundamental is wrong. - ThreadSky

About ThreadSky

swalton.ai • 9 days ago

I think a lot of these math benchmarks are absurd. It seems people are chasing the score without thinking about what we're actually trying to do.

Not getting 100% on 3 digit times a 3 digit but getting a 6 digit x 4 digit should make us question everything.

Something fundamental is wrong.

Comments

swalton.ai•9 days ago

The point of these benchmarks isn't to get a score, it is to see generalization. You shouldn't need to feed a learning machine millions of multiplications for it to find the patterns. We want it to learn the actual rules, which is discoverable easily within 3 digits.

swalton.ai•9 days ago

While something like this is better it still highlights the fundamental problem: the rules aren't being learned.

Note: the original image is a gif. All scores start bad but improve. They improve top left to bottom right.
Rules still aren't learned

swalton.ai•9 days ago

We have to be very careful when evaluating things. You'll be deceived if you just think "better score = better model".

Benchmarks are far from enough and the better we get the harder they are to evaluate. Benchmarks mean nothing if you don't understand their purpose or limitations.

swalton.ai•9 days ago

It isn't just around this specific topic, but I worry our community is making these same mistakes more generally. Too often I see papers rejected for poor benchmarks and papers accepted that fail to substantiate their hypotheses. This is completely backwards. The benchmarks aren't the important part

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply