Data-mining sucks: official report

A multi-year National Research Council review of data-mining as a means of discovering terrorists has concluded that this just doesn't work very well, and that it ends up harming and harassing -- and terrorizing -- innocents whose only crime is to have a profile that some database-designer thinks is hinky.
The report was written by a committee whose members include William Perry, a professor at Stanford University; Charles Vest, the former president of MIT; W. Earl Boebert, a retired senior scientist at Sandia National Laboratories; Cynthia Dwork of Microsoft Research; R. Gil Kerlikowske, Seattle's police chief; and Daryl Pregibon, a research scientist at Google.

They admit that far more Americans live their lives online, using everything from VoIP phones to Facebook to RFID tags in automobiles, than a decade ago, and the databases created by those activities are tempting targets for federal agencies. And they draw a distinction between subject-based data mining (starting with one individual and looking for connections) compared with pattern-based data mining (looking for anomalous activities that could show illegal activities).

But the authors conclude the type of data mining that government bureaucrats would like to do--perhaps inspired by watching too many episodes of the Fox series 24--can't work. "If it were possible to automatically find the digital tracks of terrorists and automatically monitor only the communications of terrorists, public policy choices in this domain would be much simpler. But it is not possible to do so."

As a Slashdot poster says, "Can't we just go back to probable cause?" Government report: Data mining doesn't work well (via /.)

Update: Ennis sez, "That's Bill Perry, former SecDef from 93-97! It's not just some ivory tower analysis then .... "