Looking for a principled evaluation method for ranking of *general* agents or models, i.e. that get evaluated across a myriad of different tasks? I’m delighted to tell you about our new paper, Soft Condorcet Optimization (SCO) for Ranking of General Agents, to be presented at AAMAS 2025! 🧵 1/N - ThreadSky

sharky6000.bsky.social • 94 days ago

Looking for a principled evaluation method for ranking of *general* agents or models, i.e. that get evaluated across a myriad of different tasks?

I’m delighted to tell you about our new paper, Soft Condorcet Optimization (SCO) for Ranking of General Agents, to be presented at AAMAS 2025! 🧵 1/N

Comments

sharky6000.bsky.social•94 days ago

In this paper, we build on recent evaluation methodologies based voting from computational social choice known as Vote N’ Rank or Voting-as-Evaluation (VasE): https://bsky.app/profile/sharky6000.bsky.social/post/3laomwxwsjs2q 2/N

sharky6000.bsky.social•94 days ago

Imagine this perspective of voting: there is an underlying ground truth and votes are rankings sampled from an noisy distribution over this ground truth with points closer to the truth being more likely. 3/N

sharky6000.bsky.social•94 days ago

Now what if I told you some voting rules could be interpreted as maximum likelihood estimators of this ground truth?

In 1785 (!), Marquis de Condorcet proposed a model for the ideal voting system, and it had four requirements: 4/N

sharky6000.bsky.social•94 days ago

… but it had some problems in its formalization, so Young proposed an alternative formulation which resolved these problems. (For details see Sec 8.3 of the Handbook of Computational Social Choice: https://www.cambridge.org/core/books/handbook-of-computational-social-choice/8AF63E87F76A5FC974D5E73536C52BD6) 5/N

sharky6000.bsky.social•94 days ago

Under Young’s interpretation, this ground truth is the optimal ranking: one that has minimal the average Kendall-tau distance to the votes (evaluation data), which also corresponds to the Kemeny voting rule. https://en.wikipedia.org/wiki/Kemeny%E2%80%93Young_method 6/N

sharky6000.bsky.social•94 days ago

We interpret the number of misclassifications as a loss function. Problem is, this discrete loss function is not differentiable. So we propose a sigmoid loss function in its place, resulting in a smooth & differentiable Kendall-tau distance that can be optimized using gradient descent! 7/N

Comments

Posting Rules

Reply