Brandon Mayfield was a US Army veteran and an attorney in Portland, OR. After the 2004 Madrid train bombing, his fingerprint was partially matched to one belonging to one of the suspected bombers, but the match was a poor one. But by this point, the FBI was already convinced they had their man, so they rationalized away the non-matching elements of the print, and set in motion a train of events that led to Mayfield being jailed without charge; his home and office burgled by the FBI; his client-attorney privilege violated; his life upended.

At every turn, the FBI treated evidence that contradicted their theory as evidence that confirmed it. Mayfield's passport had expired and he couldn't possibly have been in Madrid? Proof that he was a terrorist: he must be using his connections with Al Qaeda to get false papers so that his own passport isn't recorded as crossing any borders. Mayfield starts to freak out once he realizes he's under surveillance? Aha! Only the guilty worry about having their homes burgled by G-men!

The FBI was so sure of their theory that they lied to a judge during their campaign against him. His story is the perfect embodiment of "confirmation bias" -- the tendency of human beings to give undue weight to evidence that confirms their existing belief and to discount evidence that rebuts it. Confirmation bias is one of the underappreciated problems of mass surveillance: gather enough facts about anyone's life and you can find facts that confirm whatever theory you have about them.

Or, as Cardinal De Richelieu said: "If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him." This line is the epitaph in my story Scroogled (here's Wil Wheaton's reading of it), about the risks of automated, unaccountable attributions of guilt based on algorithms that are not subject to scrutiny. But as bad as the automated attribution as guilt can be, it's nothing compared to the directed attribution of guilt from cops who are absolutely sure that they have their man.

Read the rest