Why forensic science is failing us — and why tonight's NOVA documentary doesn't quite cut it

In 2009, the National Academies of Science published a massive report on forensics. For many Americans, forensics is possibly the most familiar of all the sciences. It's the one we welcome into our living rooms every night, along with TV crime dramas and murder mysteries. But the report's conclusions might surprise you.

For one thing, it's hard to even generalize about the state of forensic science in the United States, because everything from standard practices to accreditation varies widely by sub-discipline, law-enforcement agency, and whether the law enforcement is happening at a local, state, or federal level. Worse, it's not at all clear that some of those sub-disciplines have a sound, scientific basis. For instance, DNA analysis tends to be pretty well-supported by evidence, while fingerprint analysis remains an art, more dependent on the person looking at the fingerprint than on hard laws of anatomy. Of course, the report also found that there simply hasn't been enough research done to determine how scientific most disciplines of forensic science are to begin with. Meanwhile, the Supreme Court has allowed trial judges to certify forensic techniques as reliable even though we don't know whether they they are or not — and those decisions have been made in a haphazard, inconsistent way from one judge to the next.

Given how much our legal system relies on this stuff, we should all be feeling more than a little uncomfortable right about now. The state of forensic science, combined with its importance, virtually guarantees that there are innocent people behind bars (or worse) and criminals on the loose.

Tonight, on PBS, NOVA will premier a documentary on the flaws of forensics and how they might be solved. I liked the show and I think it's definitely worth watching. That said, I think NOVA took an angle on this information that made the show less useful (and less important) than it might have been.

The NOVA documentary, "Forensics on Trial", is engaging and fun to watch. The basic structure of the show presents three real-world cases where forensics failed us, explains what went wrong, and introduces us to cutting-edge technologies that might be able to make crime scene investigations more reliable in the future.

For instance, a partial fingerprint on a plastic bag briefly made Brandon Mayfield — an American lawyer and convert to Islam — suspect number one in the 2004 Madrid train bombings. The fingerprint wasn't an exact match (although the FBI claimed at the time that it was) and, in fact, there were 20 people with fingerprints similar enough to the one found in Madrid that the FBI opened investigations on them. Mayfield was held without charge for more than two weeks, until Spanish authorities convinced the FBI that they had other suspects who made more sense as potential bombers, and whose fingerprints more closely matched the one that was found.

So what do we do when one fingerprint could reasonably belong to enough different people that the suspects could play a game of softball against each other while we figure it out?

NOVA suggests improving fingerprinting technology, using a high-tech film that can capture all the lines and ridges and whorls at a much-more-detailed level.

Would the people who do fingerprint analysis like to have more-detailed prints to work from? Probably. But that wouldn't solve the much-more basic problem highlighted in the National Academies report. Techniques like fingerprint analysis — which are based on humans or computers attempting to find and match patterns — are incredibly prone to bias. Not the kind of bias where the analyst has it in for the suspect personally, but the kind that happens when the analyst is thinking about what she's heard about the case from other people; when the analyst knows that other analysts have already called the prints a match; or when the analyst is under pressure to quickly find the dangerous terrorist. Here's a description of a study on this topic, as related in the National Academies report:

Recent research provided additional evidence of this sort of bias through an experiment in which experienced fingerprint examiners were asked to analyze fingerprints that, unknown to them, they had analyzed previously in their careers. For half the examinations, contextual biasing was introduced. For example, the instructions accompanying the latent prints included information such as the “suspect confessed to the crime” or the “suspect was in police custody at the time of the crime.” In 6 of the 24 examinations that included contextual manipulation, the examiners reached conclusions that were consistent with the biasing information and different from the results they had reached when examining the same prints in their daily work.

Without also reading the National Academies report, it's easy to come away from the NOVA documentary thinking that all we need to do is lay some improved technology on top of the existing forensic science system. In the notes I took while watching the screener, I wrote, "Is the problem here a lack of adequate technology, a mishandling of technology (bad methods/training/application), or a flawed foundation to begin with?"

NOVA doesn't really provide a clear answer. Meanwhile, the reply from the National Academies report appears to be, "Yes. All of that."

The NOVA documentary did a good job of alerting me to a problem. But it could have done a better job of explaining the true nature of the problem. More importantly, it left a big empty hole in terms of what comes next. If this isn't something that's just going to be solved by incremental improvements in technology (and it seems like that's the case), well then what? Has this report had any impact on the courts since 2009? Have the forensic sciences begun to improve standardization of methods, training, and accreditation? Did Congress appropriate any funds for research that would help us understand whether some of these sub-disciplines can be trusted at all?

And if not, why not?

Maybe this is a job for Frontline. But, from my perspective, leaving those issues less-than-well-addressed knocks "Forensics on Trial" down from a "must-see" to an "interesting way to spend an hour".

Hey, at least it's better than CSI.

• "Forensics on Trial" premiers tonight. Check your local listings.

Read the full National Academies report on the failures of forensic science.

Read a 2010 report on how these findings might affect the practice of law.

Image: Courtesy of ©Providence Pictures


  1. I learned a lot about forensics recently when talking to a lawyer in my family. He was explaining a point of law, and used a stabbing he had recently worked on as an example. I asked whose fingerprints were on the knife, and he laughed and explained that the police don’t bother lifting prints in a situation like that one.

    1. I’ve noticed some lately, maybe cop shows make it worse, that some folks seem to think that every investigative technology is going to be used on every crime.  Like I’ve known people lately who absolutely expected that the officers would be taking finger prints in a shoplifting incident and that they would do a DNA search and tire tread analysis on a cigarette butt and tire marks left after a house break in. 

      1.  I know, cop shows make it worse, and yet…

        … we had a guy walk in the back door of our house, toss our entire basement (including my room), and then run when he came upstairs and my roommate confronted him.  He touched all kinds of stuff in my bedroom but they didn’t take any prints.  Fortunately he pulled the same thing elsewhere within the day and they caught him, but it turned out he had a long record.  Why didn’t they take any prints before he was caught?  Was there no interest in charging someone who was wildly breaking his parole with a new offense and taking someone off the streets?

        … at another place, we had several break-ins to the building’s “secure” garage and several cars including mine.  The second time, they swiped a 12-pack of soda in my trunk (the only thing of value in the car) and drank one, then threw the empty in the back seat.  DNA and fingerprint gold mine.  Forensics was never sent.  Thieves were never caught and continued to work area garages for the next couple years until they were interrupted en medias res, but of course the previous break-ins couldn’t be added to their charges because… no forensics.

        Why is there no interest in establishing a pattern of behavior among habitual criminals, to adequately distinguish the one-offs (dumb kid choices, stupid drunk decisions, moments of desperation) from the methodical career criminals?

        1.  i think you miss the point of the comment to which you replied, entirely.  There are no resources to prosecute, let alone investigate these (literally) petty crimes you are describing.  I feel for your losing your stuff and feeling unsafe (my car got broken into last month, right in my driveway), but i still don’t think we should dedicate much funding to these types of things.  classical methods work, and crime is at an all time low, in most cases, particularly violent crime.
          There is interest in punishing habitual perpetrators.  it’s called the judicial system.

          1.  I get the point of the comment.  What I don’t get is the phenomenon.  If violent crime is down, shouldn’t attention be paid to the lower-level (yet still FELONY) crimes perpetrated by habitual criminals?  If someone had walked into any of the burgled garages while the crime was in progress, were the guys carrying weapons, would there have been an assault or murder?  Thank goodness the dude who walked in our back door wasn’t carrying a stolen gun when our roommate confronted him or when he walked in on a lady down the block who got the cops there in time to arrest him.  Because they didn’t take any forensic evidence from my room, they had to rely on my roommate to pick him out of a lineup (the LEAST reputable of all evidence, as the program above pointed out I believe), which not only wasted a bunch of my roommate’s time but also gave the defense wiggle room to bargain for all the charges other than the one where he was caught on the property to be thrown out because “there isn’t any hard evidence proving he’s the one who was there.”

            “Broken Windows” style policing has its problems but it also has positive results, and when you take repetitive career criminals off the streets, you reduce crime by a lot more than just randomly grabbing the low-hanging fruits who are dumb enough to get caught in flagrante.  The judicial system can do nothing without evidence, and if police refuse to even collect evidence, there is no way of saying “actually, Your Honor, this defendant is linked not only to this garage break-in, but to 15 others in the same area over the past year, so no, we don’t think he should get (bail, probation, a slap on the wrist, community service, etc.)”

  2. Time for blinded fingerprint testing, along with similar changes to the  methodology of eyewitness lineups. It makes no sense to have cops who know which person is the suspect interacting with witnesses.

    1.  the cops interacting with witnesses part is illegal…
      agree on your first point, for federal cases.

  3. Fingerprints actually has a pretty long history of research and study – from the make-up of the skin to the fetal development of friction ridges to the components of the residue transferred to the development techniques allowing examiners to visualize prints, over 100 years of evidence supporting the validity of fingerprint examination is on record.  However, because there are no numbers associated with the determinations of latent print examiners, it becomes “unscientific.” 

    Dror’s studies on bias are an interesting avenue for further study, but they are far from conclusive, in spite of what he likes to put forth on screen.  In addition, he found similar issues with that gold standard, DNA analysis.    http://www.guardian.co.uk/science/2012/oct/14/vaughan-bell-on-science-forensics
    IMHO, the problem is that every determination, scientific or otherwise, requires some interpretation of data.  Bias can certainly creep in, but it is certainly not as pervasive as Dr. Dror would have us believe. 

    As a latent print examiner, I say, please – bring on further research.  Bring on advanced technology.  Fund more studies to make my job easier.  But, at the same time, stop trying to find a magic forensic bullet – none of the disciplines are infallible.

    1. Couldn’t all fingerprint examining take place in what you might call a ‘double-blind’ manner? The examiner shouldn’t know anything about the owner of the prints he’s trying to match to avoid these biases?

    2. However, because there are no numbers associated with the determinations of latent print examiners, it becomes “unscientific.”

      Despite the scare quotes, this seems to me like it actually is a serious problem.  You can’t fix a probability that a print belongs to a particular person without some numbers and if you can’t give a confidence interval then it really isn’t scientific.

      The OP points out that at least 20 different people matched the print from the Madrid bombing; it also mentions that some people were better matches than others.  This all implies that a certain amount of quantification is possible and that identification on the basis of fingerprints really should have a confidence interval attached based on how many people in a given geographical area might be a better match than the suspect.

      So why aren’t there any numbers associated with the determinations of latent print examiners?

  4. Forensic science parts of a case shouldn’t be/ usually aren’t, the only parts of the case used to make a decision.  Yeah, a finger print might not be irrefutable proof a defendant was a killer.  But that there was a finger print, a motive, an opportunity, additional evidence,etc. all together are a compelling argument.  Same for exonerating people.  If a dozen people saw me go into a house carrying an axe screaming that I was going to axe murder the person inside, and saw me leave covered head to toe in blood, and police find my clothes soaked in the victims blood at my house surrounded by creepy dioramas  of me killing the victim, and it turns out that me and the victim were bitter rivals in a love triangle, and that my whereabouts for the time of the murder can’t be verified, and there’s surveillance footage of me an hour before the murder at the hardware store buying an axe identical to the murder weapon, I’m not likely to dodge conviction because my prints aren’t on the axe handle or that someone else’s are. 

    The justice system’s not perfect.  Science isn’t perfect.  We have to try and do the best we can to make the best decisions we have with every tool and technology and type of evidence we can. 

    1. The problem is that the forensic science is being portrayed (and popularly understood) as infallible, and sometimes that’s the only bit of evidence that leads to the conviction. And this is despite known, but unaddressed, issues with tester bias.  This can also work against prosecutors – I heard from someone who sat on a jury that some of the jurors simply refused to convict because the police hadn’t done DNA testing (this was for a simple robbery case where DNA testing would never be done in the normal course of things), even though the prosecution’s case was strong.

    2. In terms of the US’ incarceration rate, we’re rivaled only by the Soviet Union in its darkest days. That’s not a matter of imperfection, that’s a social cancer with no end in sight. 

  5. DNA tests are now being pushed to the point where tester bias is a big issue, particularly when it comes to matching a mixed or partial DNA sample.  
    I was reading about a case where they had four different suspects.  They sent the crime-scene DNA and DNA from each of the suspects to four different crime labs (with each lab only getting DNA from one suspect), asking if the (different) suspects were a match.  In each case the lab made a match to the crime-scene DNA, even though each lab was working with DNA from a different person, and even though the DNA profiles didn’t exactly match up in any case.  The thing was, in each case, the lab was told that the sample they were trying to match came from “the” suspect.  Having been told that, they obediently ignored contradictory evidence to make a match.

    1. I am not aware of the case that you refer to however what you state is simply not possible as you have described it. If the crime scene DNA was profile originated from a single individual it either matches the suspect or it does not (allowing for the statistical possibility that two unrelated individuals share the same DNA profile), it could not match 4 different individuals. However, if the profile from the crime scene was a mixture of more than one individual then it is quite possible that profiles from suspects cannot be eliminated as being a contributor to that mixture. The more contributors there are to a mixture, the higher the probability of the components of an individual’s profile appearing in that mixture. I strongly suspect this is the scenario that you mentioned. Imagine a mixture of all of the digits from 4 or 5 people’s phone numbers, it is quite likely thast you could pull the digits of your own phone number out of that mixture, even though your own number never went in to the bag in the first instance. There is a huge difference between reporting a match and stating that an individual’s profile cannot be eliminated as a possible contributor to a mixture.

    2.  Do you remember where you read about it?  I’d be curious to see the details given Jonny Sokal’s response.

  6. “The state of forensic science, combined with its importance, virtually guarantees that there are innocent people behind bars (or worse) and criminals on the loose.”

    Sorry, but wrongly convicting innocent people is FAR worse than letting guilty people get away. Not only does it undermine the entire concept of justice, it destroys the lives of real individuals and families, and often ends in murder by the state. 

    The widely held belief that it’s acceptable to risk locking up an innocent person rather than take the chance that a guilty one might go free is the exact reason that prosecutors get away with shoddy work, dubious science and dishonest practices.

      1. You grammar was not poor.  The word “and” separates the phrase “people behind bars” from the phrase “criminals on the loose,” therefore the parenthetical “(or worse)” is obviously a modifier of the phrase “people behind bars.”  I would argue that the parentheses are unnecessary but not strictly incorrect.  The misunderstanding here was the fault of the reader, not the writer. 

        Let this be a lesson to those who feel threatened by those of us who know how to write (and read) correctly and dare to correct those who cannot.  We are not “grammar nazis.”  We are caretakers of language precision.  Precise use of language is not about form or adhering to rules for their own sake.  It is about clear communication, saying what you mean, having it understood as such and, most importantly, preserving our freedom. 

        This is not hyperbole. It should be obvious in this election season that language can be used as a weapon.  It can be used to manipulate and control.  Only by being aware of how language is properly used can we be aware when it is being purposely misused.

  7. It seems like blinding forensic samples would be an easy way of handling this. The main disadvantage as I see it is, for this to be worthwhile, examiners would have to be presented with significant numbers of false tests. In other words, if most tests an examiner get are from a law enforcement body that is looking for a suspect, the examiner will have reasonable belief that most samples are from suspects, and will be biased to a false positive. The way to do it would be to load the examiner up with meaningless, random tests, and only slip in a certain percentage of actual evidence. Of course, this would increase the work-load and decrease the efficiency of the examiner, and it would provide a quantitative metric of the examiner’s ability to correctly analyze samples, which is why it will probably never be done.

  8. Part of the problem with forensics is the fact that most law enforcement agencies cannot afford to run all these tests on every case.  So no fingerprinting of shoplifters, etc as mentioned previously.  These tests are fairly expensive and local labs do not have all the fancy, expensive test equipment you see on a CSI or NCIS shows.  

  9. I’m a criminal defense lawyer and I can tell you forensic science is a shabby art in almost all jurisdictions.  I worked in Chicago for years and their evidence techs were generally retired detectives collecting a second pension and barely cared about their job.  On top of that, it is absolutely the case that it is rarely used except for the times where it is used speciously.

    I have a very jaundiced eye towards forensic science and find that it’s almost always used only when the state’s case is so weak they really wouldn’t be able to put a case together otherwise.  

    For things as simple as photo ID, it has taken years of hammering at police stations to use a non-biased interviewer to give a six photo array.  And even after the Chicago PD was convinced to do it this way, they somehow came to the conclusion that sequential, double blind ID was not better than their old ways (the line up, which is a terrible way to ID someone).


    So yeah, do I buy forensic science?  Some of it.  Do I trust it?  None of it.  Look to the Supreme Court in Bullcoming for the premise that we really can’t just let scientists (or cops) come in and say, “Here’s our report…trust us.”


  10. judging from the comments I guess the question should better be about the state of forensic science in the US rather than in general. I work in the forensic field in germany and at least over here a lot of double blind testing has been implemented. for example in DNA testing there is never a full name, just a code, given and the status (accused, victim,…) is never revealed to the examiner. when police want to generate a table of pictures, the computer will generate from files a random list of people that have a lot of features in common. sure, there are flaws. but you must not forget that a) forensics in the field is never “lab environment”, samples are almost never pure, tissue mostly in bad shape and so on. b) judges will evaluate the strength of the argument, even more so if long term jail or death penalty are at stake c) do not confuse conviction with accusation. often forensic science is used to narrow down suspects after which witnesses, money transfers, motive, alibi and a good interrogation follows – and possibly conviction. those cases are the most powerful in underlining the methods themselves – if a bullet independently provides evidence for being fired from a glock and you later find out a suspect owns a glock, that is a very strong argument. every arguments following this argument usually separate the professional forensic people from “tv audience”.

Comments are closed.