Machine learning's reproducibility crisis is getting worse, and the massive shortage of qualified researchers has driven top salaries over $1,000,000, bringing in all kinds of cowboys and pretenders.
Google AI researcher Ali Rahimi got a standing ovation at a machine learning conference when he called the field "alchemy," criticizing the unsystematic reliance on rules of thumb, trial and error, and superstition.
Rahimi is the co-author of Winner's Curse? On Pace, Progress and Empirical Rigor, a paper at the 2018 International Conference on Learning Representations, in which he sharpens his critique of the field.
He shows that the trial-and-error method produces worse outcomes than empirical research into the best ways to tune algorithms for different purposes, and argues that publication bias is behind the crisis, with AI researchers drawing the most interest when they produce algorithms that perform better, rather than algorithms that are better-understood.
Ben Recht, a computer scientist at the University of California, Berkeley, and coauthor of Rahimi's alchemy keynote talk, says AI needs to borrow from physics, where researchers often shrink a problem down to a smaller "toy problem." "Physicists are amazing at devising simple experiments to root out explanations for phenomena," he says. Some AI researchers are already taking that approach, testing image recognition algorithms on small black-and-white handwritten characters before tackling large color photos, to better understand the algorithms' inner mechanics.Csaba Szepesvári, a computer scientist at DeepMind in London, says the field also needs to reduce its emphasis on competitive testing. At present, a paper is more likely to be published if the reported algorithm beats some benchmark than if the paper sheds light on the software's inner workings, he says. That's how the fancy translation algorithm made it through peer review. "The purpose of science is to generate knowledge," he says. "You want to produce something that other people can take and build on."
AI researchers allege that machine learning is alchemy [Matthew Hutson/Science]
