Focus Groups and Academic Reviews

I've expressed the ideas that follow on Twitter, but for the sake of posterity I'll repeat my thoughts here.

As I prepare to respond to NeurIPS reviews, I've been reflecting on the analogy of conference proceedings to a marketplace, where ideas (products) are produced by research labs (competing firms) and exchanged for time/attention (currency). Each time your lab attempts to launch a new product, it first has to pass muster with a small focus group, namely your reviewers. The focus group evaluates your product rather subjectively, and in comparison to the substitutes available.

You don't need me to tell you that every year, there are more and more apparent substitutes for your product in the ML research marketplace. If you're a young researcher, launching your career is akin to entering a crowded, fiercely competitive attention price war. The goal of scientific publication (aside from personal incentives) may ultimately be collaborative, but the process is fundamentally competitive. Acceptance thresholds at conferences are driven by scarcity of attention, not scarcity of good ideas.

That's why I don't think introducing lower tiers of acceptance would change much. It might look nicer than "preprint" on your CV, but it wouldn't change the fact that it would fail to signal to employers and researchers that your work is particularly worthy of attention. I do think that there is much that could be changed about the conference publication process that could greatly improve both the participation experience and output. Eliminating toxic high-stakes deadlines and compensating reviewers are good examples.

I find it helpful to remember that conference publications are reserved almost exclusively for ideas that fare well w.r.t. the impact/attention ratio, and not the magnitude of the impact itself. There's two ways to make your ideas more competitive in the short term -- innovate more (very hard) and explain better (less hard). Conference feedback seems almost useless for the former, and somewhat helpful for the latter. In the long term you could try to keep outcompeting more established labs, or you could take a step back and consider where your competitive advantages lie. Behind every flashy conference paper there are critical weaknesses that the authors will do their best to disguise.

You might be more likely to hit upon a disruptive idea when asking yourself "In what frame is method X weak?", than "How can I make method X better in the conventional frame?" At any rate, I think it's worth a shot.