Machine learning's reproducibility crisis is getting worse, and the massive shortage of qualified researchers has driven top salaries over $1,000,000, bringing in all kinds of cowboys and pretenders.
Google AI researcher Ali Rahimi got a standing ovation at a machine learning conference when he called the field "alchemy," criticizing the unsystematic reliance on rules of thumb, trial and error, and superstition.
Rahimi is the co-author of Winner's Curse? On Pace, Progress and Empirical Rigor, a paper at the 2018 International Conference on Learning Representations, in which he sharpens his critique of the field.
He shows that the trial-and-error method produces worse outcomes than empirical research into the best ways to tune algorithms for different purposes, and argues that publication bias is behind the crisis, with AI researchers drawing the most interest when they produce algorithms that perform better, rather than algorithms that are better-understood.
Ben Recht, a computer scientist at the University of California, Berkeley, and coauthor of Rahimi's alchemy keynote talk, says AI needs to borrow from physics, where researchers often shrink a problem down to a smaller "toy problem." "Physicists are amazing at devising simple experiments to root out explanations for phenomena," he says. Some AI researchers are already taking that approach, testing image recognition algorithms on small black-and-white handwritten characters before tackling large color photos, to better understand the algorithms' inner mechanics.
Csaba Szepesvári, a computer scientist at DeepMind in London, says the field also needs to reduce its emphasis on competitive testing. At present, a paper is more likely to be published if the reported algorithm beats some benchmark than if the paper sheds light on the software's inner workings, he says. That's how the fancy translation algorithm made it through peer review. "The purpose of science is to generate knowledge," he says. "You want to produce something that other people can take and build on."
AI researchers allege that machine learning is alchemy [Matthew Hutson/Science]
In PrinTracker: Fingerprinting 3D Printers using Commodity Scanners (Scihub mirror), a paper to be presented at the ACM SIGSAC Conference on Computer and Communications Security conference in Toronto this month, a group of U Buffalo and Northeastern researchers present a model for uniquely identifying which 3D printer produced a given manufactured object, which may allow […]
More than 40% of US federal judges have attended Manne seminars, a notionally "bipartisan" educational conference presented by a Florida "Law and Economics" institute whose invited ideological allies explained to judges why pollution is good for minorities (polluted neighborhoods are cheaper and therefore affordable by poor people), unions are bad, monopolies are economically efficient, discrimination […]
The Authors Alliance is a nonprofit that advocates for authors, libraries, readers and scholars (I'm on their advisory board); they've done a ton of great work, notably a tool for authors to claim their copyrights back from publishers, even when the original contract specified that the rights were signed away "in perpetuity."
Speed reading isn’t just an innate skill possessed by a lucky few. Anyone can learn to speed read, and the benefits are endless. The brain can process more information than most people have time to soak up, but you can make that time now with the 2018 Award-Winning Speed Reading Bundle. The first half of […]
Sure, you could use the same old PowerPoint templates for your next business presentation. It’s not like you have bosses or investors to impress. Oh wait, you do? Time to augment that slideshow with Slideshop – the presentation tool that can individualize your pitch while saving you time. Compatible with PowerPoint, Keynote and Google Slides, […]
Multinational companies have used the no-nonsense methodologies of Six Sigma and Lean Six Sigma to oil a smooth-running operation for years. What is it? Six Sigma (and its offshoot, Lean Six Sigma) apply the principles of science to business, teaching managers to methodically target waste, maximize output and streamline the flow from producer to consumer. […]