ICLR should not allow public discussion until after decisions. It allows Oscars-style award/accept campaigns by high-profile allies. Even for great papers, that can’t be part of the review process. The only reason it didn’t happen before is because it was against community norms, which are changing. - ThreadSky

nsaphra.bsky.social • 103 days ago

ICLR should not allow public discussion until after decisions. It allows Oscars-style award/accept campaigns by high-profile allies. Even for great papers, that can’t be part of the review process. The only reason it didn’t happen before is because it was against community norms, which are changing.

Comments

kaiserwholearns.bsky.social•103 days ago

Agree. Oth it might be helpful as a way to receive report and doubt. There is one user reported that the authors of a paper I was reviewing violate the anonymity policy by posting their submissions in public.

nsaphra.bsky.social•103 days ago

Is that banned by ICLR? You mean linking to the openreview page? Not sure why that’s different from linking publicly to a deanonymized arxiv version, which is allowed?

davidpicard.bsky.social•103 days ago

Or adversarial comments: get friends to destroy competing papers in public comments. I remember seeing a comment that was something like "I doubt this method works and after implementing it, I get 10% less than your paper. How do you explain that?". It was circa 2018-2019.

kylem.bsky.social•103 days ago

To OP’s credit public comment should certainly wait until after reviewer decision/feedback, but the root issue here is the insane over demand for access to “prestigious” conferences. I think RLC does this better (just checks for technical rigor/correctness) but admittedly it’s a very new conference.

nsaphra.bsky.social•103 days ago

I like that there are venues that just target rigor to prevent redundant potentially useful work. However, in the interests of my extremely finite lifespan, I like that there are filter systems for quality/excitement as well.

sash-4.bsky.social•103 days ago

Totally agree with you on the filtering point, but we're all are pretty bad at predicting what papers will be useful in future e.g PPO was rejected.

So maybe only reviewing for soundness is a good thing?

kylem.bsky.social•103 days ago

that’s what the Twitter thread version of the paper is for ;)

nsaphra.bsky.social•103 days ago

Those just reveal level of author microcelebrity!

kylem.bsky.social•103 days ago

let the YouTube algorithm decide? tongue in cheek I do see your point lol

nsaphra.bsky.social•103 days ago

Ok, enough subskeeting. I commented in the open.

saxon.me•103 days ago

Epistemic status: certain

nolovedeeplearning.bsky.social•103 days ago

it should be subflapping!

parskatt.bsky.social•103 days ago

Haha I see we commented the same paper!

nsaphra.bsky.social•103 days ago

I actually contacted you through your website appreciatively!

parskatt.bsky.social•103 days ago

Oh my... You have alerted me to the fact I haven't checked that in about a year. I have some unread messages. Double thanks!

chriswolfvision.bsky.social•103 days ago

Yep. Not recommended and I don't know why this is allowed.

tuhinchakr.bsky.social•103 days ago

I am a fan after reading your comment.

processorofnl.bsky.social•103 days ago

Considering the experience the median reviewer has, these will 100% sway reviews.

parskatt.bsky.social•103 days ago

This has been going on for some time,
the earliest case I remember seeing was from Ian Goodfellow at ICLR 2018 (note: I do really like the paper):

https://openreview.net/forum?id=B1QRgziT-

parskatt.bsky.social•103 days ago

And I have no idea why I remember that Ian championed a paper in the public comments in 2018 but can't remember the name of a person I met 5 minutes ago...

gsarti.com•103 days ago

Completely agree, although re: the deanonymization point raised by @parskatt.bsky.social I'd argue that training SAEs on closed-source GPT-4 activations is already enough to identify authors' affiliation

parskatt.bsky.social•103 days ago

That's probably true, and I assume that if you're knowledgable in the field you could probably guess the authors even without the activations.

I just think the less the better.

parskatt.bsky.social•103 days ago

I do quite like double-blind, and would hate to see it go.

mraginsky.bsky.social•92 days ago

Stuff like this, from 2017, is why I have been, and remain, skeptical about ICLR in particular: https://openreview.net/forum?id=BkjLkSqxg

nsaphra.bsky.social•92 days ago

Ehhh, it didn’t have an effect on the acceptance though?

mraginsky.bsky.social•92 days ago

No, but it’s hard to argue it didn’t have any downstream effects.

nsaphra.bsky.social•92 days ago

Yeah I just mean from the conference side, it doesn’t make me doubt ICLR’s decisions much

mraginsky.bsky.social•92 days ago

“Community norms” are what the already famous can get away with. Rich get richer, etc.

iclr-conf.bsky.social•102 days ago

Thanks for the feedback — we understand the concerns. ICLR was founded on a principle of being maximally open, but we will do a post mortem after the review process to make recommendations to the community for future conferences.

Comments

Posting Rules

Reply