Adding my love letter to arxiv.org/pdf/2304.01315 Empirical Design in Reinforcement Learning by Andrew Patterson, Samuel Neumann, Martha White, Adam White JMLR 25 (2024) 1-63 #ReinforcementLearning These aren’t the heroes we deserve, but they are the heroes we need. - ThreadSky

brandonrohrer.com • 97 days ago

Adding my love letter to

https://arxiv.org/pdf/2304.01315

Empirical Design in Reinforcement Learning
by
Andrew Patterson, Samuel Neumann, Martha White, Adam White

JMLR 25 (2024) 1-63
#ReinforcementLearning

These aren’t the heroes we deserve, but they are the heroes we need.

Comments

brandonrohrer.com•97 days ago

I mean, look at this opening paragraph. This is written to be read and understood, not to impress and overwhelm. Not for the handful of reinforcement learning researchers in the world, but for the slightly larger handful of people trying to make it work on their machine.

brandonrohrer.com•97 days ago

The paper is refreshingly concrete, fully loaded with examples, case studies, recommendations, and pitfalls. It's a map to the landmines in reinforcement learning. It's what I would have gifted my younger self.

brandonrohrer.com•97 days ago

It's easy to lie to your readers in any ML work, but in reinforcement learning it's also easy to lie to yourself. The authors shine a bright light on all these sneaky backchannels. Designer's curse. Untuned baselines. Hypothesis After Results Known (HARK). Cherrypicking of all sorts.

brandonrohrer.com•97 days ago

And for the love of the great Flying Spaghetti Monster DO NOT TUNE ON YOUR SEED. Just don't.

brandonrohrer.com•97 days ago

As an unexpected treat the authors have genuinely grown-up conversations with us about statistics. Reinforcement learning runs are so computationally intense and then add in hyperparameter sweeps, it's easy to see how representing distributions and confidence intervals falls by the wayside.
No more.

brandonrohrer.com•97 days ago

But the ride doesn't stop there. The authors take an ax to the root of the tree of ML leaderboard-based publishing. They make a strong case that reinforcement learning comparisons are all but infeasible to do in a meaningful way.

This low key refutes the entire ML conference industrial complex.

jan.hermann.name•97 days ago

What if I tune it to find the worst possible performance and then try to improve that metric?

brandonrohrer.com•97 days ago

well i don't see any problem with that

capncooknl.bsky.social•96 days ago

#databs right there

leocus.bsky.social•97 days ago

Very interesting paper! I was wondering though: what do you think about figure 1? Is the last "verify" stage really needed, or can we replace it with e.g., well-known non-parametric hypothesis tests on 30 runs? This may avoid running too many experiments just to know that two methods are comparable