Hot take: 90% of what ACs/SACs do could in principle already be automated (with the remaining 10% being process oversight and borderline decision making). At least right now, it seems like reviewers have the more important job for the most part. - ThreadSky

tkipf.bsky.social • 97 days ago

Hot take: 90% of what ACs/SACs do could in principle already be automated (with the remaining 10% being process oversight and borderline decision making).

At least right now, it seems like reviewers have the more important job for the most part.

Reposted from Hilde Kuehne

No worries. I also got a reviewer invite… I can highly recommend it… I already decided for ICLR to review instead of ACing. It’s actually a nice break. And people always forget that in the end it’s reviewers who are making calls on papers, not ACs.

Comments

sattlertorsten.bsky.social•94 days ago

I tend to (partially) disagree. The only part that could currently be automated is sending reminders (in my experience, people are more likely to respond to personalized requests). The other parts (assigning reviewers, leading discussions, checking reviews & rebuttals, justifying and checking ...

sattlertorsten.bsky.social•94 days ago

... decisions) can certainly be made easier (e.g., don't require meta reviews for clear cases), but can't be fully automated. I don't think we can automate anything close to 90%.

I agree that getting good reviewers (knowledgeable about the field, willing to put in work) is important. But ...

sattlertorsten.bsky.social•94 days ago

... getting good reviewers often means getting a good AC who is familiar with the field and know the right people to review the paper. Getting a good AC often means the PCs doing a good job with assigning papers to ACs. All parts of the chain are important.

tkipf.bsky.social•94 days ago

I think that depends on the conference. ICLR pretty much already automated the reviewer assignment part using a new bidding system that seemed to work pretty well. Manual AC assignments were heavily discouraged, only minor adjustments were needed.

tkipf.bsky.social•94 days ago

I think there is still *a lot* of headroom for automation while ultimately reducing potential for human error (or just laziness on the AC part).

tkipf.bsky.social•94 days ago

Sending reminders really shouldn’t be something we have to deal with manually. Clearly there’s headroom in designing better incentive structures.

sattlertorsten.bsky.social•93 days ago

Indeed. CV conferences traditionally ask ACs to submit reviewer preferences. Certainly more work, especially since OpenReview and TPMS scores are not always reliable. Probably less chances for collusions than with reviewer bidding though.

ericbrachmann.bsky.social•97 days ago

90% of what AC's do is sending increasingly reproachful emails. If you automate that, the social pressure goes away - the only thing that might convince a negligent reviewer to act.

tkipf.bsky.social•97 days ago

Yeah, it'll have to be a tightly kept secret among people who enter the exclusive AC circle 🙃

davidpicard.bsky.social•96 days ago

My thoughts exactly: how do you automate blackmailing people into doing the work they promised to do but do not deliver?
Or we put the names of famous researchers in a list and randomly sample one for the automated reminders?
"Dear R2,
[threatening reminder]
Your AC, Jürgen Schmidhuber."

atlaswang.bsky.social•96 days ago

The idea has my vote!

jvgemert.bsky.social•96 days ago

Actually, instead of threatening, I usually try to convince.

I try to appeal to their sense of community by expressing the extra burden and "if it was your paper" reasoning.

It's not "guilt tripping" when it's a genuine responsibility, no? 😅

csprofkgd.bsky.social•96 days ago

That’s my starting point and then …

jonbarron.bsky.social•96 days ago

Just have high profile people tweet out the names of the people who ignore the robo-emails, it would totally work.

ericbrachmann.bsky.social•96 days ago

I can already see the hellish future where I am flagged because my robo-AC thinks my review did not "delve into" the paper enough, my social reviewer score drops and, henceforth, I am only allowed to submit to the local "gathering of AI enthusiasts".

jonbarron.bsky.social•96 days ago

Yeah there'd need to be a lot of human oversight to prevent failure modes like that. This is probably true for a lot of the future world we're building towards.

mniepert.bsky.social•96 days ago

Don’t know about SAC but as AC I typically have several borderline papers where I have to write an additional review and poke reviewers/authors to address certain points I might disagree with. For last NeurIPS I went through all borderline papers with SAC on Zoom.

mniepert.bsky.social•96 days ago

But yeah: reviewing for ICLR now and that’s also good to have more senior people in the mix there.

jonbarron.bsky.social•96 days ago

Completely agreed, AC is the role in the process that is most suited to automation.

josephdviviano.bsky.social•96 days ago

If the reviewers do their jobs properly the ACs job is dead simple. In my experience though that's far from the case, < 50% of reviewers can be left to their own devices, and you need to resolve discrepancies during the meta review which is even more work than a typical review.

fguney.bsky.social•96 days ago

I understand the importance reviewing, it’s not like I claimed that it’s less important. I actually like reviewing more but I care about my role as an AC. I might be wrong -I hope I’m wrong- but I think I was the only AC from TR at our conferences. not to mention f/m ratio. representation matters.

fguney.bsky.social•96 days ago

and the worst part is that I agree with the decision. I didn’t have a paper at CVPR for so long. every year, we at least submitted one or two papers but got rejected. this year, we couldn’t even submit one. this worries me more than any hot-take on the internet.

hildekuehne.bsky.social•96 days ago

I would even worry more about the quality then about the representation when they kick you out, but I get your point 🤗.

hildekuehne.bsky.social•96 days ago

When you look at the raw todo list, this might be true, but usually those 90% are fixed pretty quickly. So when you look at which tasks needs how much time, it’s more the remaining 10% that take 90% of time. And I don’t want to automate those. 😁

Comments

Posting Rules

Reply