TSA "behavior detection" is wrong more than 99 percent of the time

Remember when the TSA rolled out its "behavior detection" system whereby slack-jawed, water-confiscating security officers would be trained to recognize your "micro expressions" and single you out on the basis of a twitchy eyelid or a sweaty upper lip? Turns out that over 99 percent of the IDs generated by the system are false positives -- less than one percent lead to arrests (and the article doesn't say how many convictions come out of those).
"That's an awful lot of people being pulled aside and inconvenienced," said Carnegie Mellon scientist Stephen Fienberg, who studied the TSA program and other counterterrorism efforts. "I think it's a sham. We have no evidence it works."...

The TSA has not publicly said if it has caught a terrorist through the program. The agency says that some who are arrested, particularly on fake ID charges, may be scouting an airport for a possible attack.

Some scientists say the TSA effort is just as likely to flag a nervous traveler as a terrorist.

"The use of these technologies for the purpose that the TSA is interested in moves into an area where we don't have proven science," said Robert Levenson, a psychologist at the University of California-Berkeley.

Although observers can perceive whether someone appears anxious or is acting deceptively, they can't tell whether that person is planning an attack or something such as an extramarital affair, Levenson said.

Levenson and Fienberg were part of a National Academy of Sciences team whose report last month said "behavioral surveillance" has "enormous potential for violating" privacy.

TSA's 'behavior detection' leads to few arrests


  1. Had to fly last week, for the first time in almost three years.

    I would guess that their BD system does not false-positive on contempt mixed with pity, because I was not hassled much more than having my 3″ pliers measured to be sure they were not over the limit for tools. That limit is 7″, and I will leave to others to joke about a woman who has trouble distinguishing between those two.

  2. Although observers can perceive whether someone appears anxious or is acting deceptively, they can’t tell whether that person is planning an attack or something such as an extramarital affair, Levenson said.

    Or, quite possibly, whether they’re afraid of flying. Funny how that relatively common scenario escaped their minds…

  3. I think passengers are afraid of losing their tempers when being bullied by cretins they wouldn’t hire to wash their car.

  4. A quick lesson in Psychology would teach you that a person who has resolved to give up his/her life and is planning to blow themselves to bits within the next couple of hours will hardly be phased by the scrutinous looks of any security officer.

    In all likelyhood, they will be the coolest cucumbers in line.

    People get nervous if there is something to lose. If you have already given up everything, nothing matters anymore.

  5. which goes back to the undeniable fact that the sole purpose of the TSA circus is to create a docile serf population that won’t cry when the rest of their rights are taken away.

  6. Gee. Perhaps I should sell them my “probability engine”. It is only wrong 50% of the time, is shatterproof and highly portable. Does not even use electricity. My cost is five cents, but with the help of the right Washington lobbyist, I could probably sell my product to the TSA for $2,500.

  7. Of course they’re getting false positives. They don’t know what they’re looking for.

    Due to the dearth of actual, real terrorists, they don’t have much in the way of subjects to observe in order to work out what the behavioural traits of a typical terrorist are. So how are they coming up with this list of stuff to watch out for? Well, they must be making it up, based on how they think terrorists might behave. So obviously it’s going to flag up a lot of innocent people.

  8. Really? 99% of the people it lets through are actually terrorists? Just reverse the output!

    Oh, 99% of positive results are wrong.

  9. Before I quit my job as a road warrior, I had to go through airports twice a week for seven years. After 9-11, it became more and more onerous. One fine day, when I was up too early, had not had enough cigarettes or coffee (I have given up cigarettes since), I was in a bad mood and the TSA agent asked me how I was doing while I waited in line, my belt and shoes in one hand, my unsheathed laptop in the other, my jacket and backpack held between my knees as I waddled towards the x-ray station.

    “Lousy,” I replied.

    That was not the right answer. Suddenly, the TSA was concerned about my ‘attitude’. I got pulled aside and was questioned about all manner of things, including being asked twice where I was going (it was printed on the boarding pass they had taken from me).

    The words of my grade school teachers, who had counseled me that my ‘bad attitude’ would one day ‘get me in trouble’ came flooding back. How right they were.

    “Is it against the law now to be in a bad mood?” I finally asked.

    “Yes, sir, it is. If you’re in a federal security zone.”

    That was the most honest answer I ever got from a TSA agent. He said it with a straight face.

    Their parting comment to me, as they released me to hobble towards my plane, thoroughly debagged and radished, was “Have a nice day, sir,” to which I mentally added “and that’s an order.”

    I take the train now, or drive. I have zero interest in getting on a plane anymore. My mood is no longer of federal concern.

  10. “It definitely gets at things that other layers of security might miss,” [TSA spokeswoman Ellen] Howe said. … Studies are underway that analyze the program’s effectiveness, she added. (emphasis mine)

    How do they know that it definitely catches things other layers of security missed if they haven’t analyzed the program’s effectiveness? Thank you for playing, Ms. Howe. Get some internal consistency and try again.

  11. The problem is that identifying “hinky” behaviour is one of the best tools good first-line security people have to catch bad guys. This is how they catch folks with trunks full of fertilizer crossing borders, for instance.

    “Going with your gut” has a lot of potential when mixed with the right sort of observational skills.

    So teaching this sort of skill is the holy grail of security. Unfortunately, it is a very, very hard skill to teach. You either got it or you don’t, and the there is no way to know if you are really being accurate.

    Most (rent-a-)cops simply go into the mode where we are all guilty of something all the time. It’s just easier and it makes them feel like they are actually succeeding.

  12. TSA is a CYA farce. More and more people are starting to understand this, but we don’t yet have the “critical mass” of public opinion that it will take to get rid of them.

    Hijacking a passenger airplane stopped being a viable technique sometime around 9:45 a.m. on 11 September 2001. Passengers are no longer docilely compliant: if anyone does *anything* squirrelly on an airplane, they are certain to be beaten and subdued by the passengers. And the armored cockpit doors have made the “take over the plane” scenario a very difficult proposition; pre 9-11, gaining control of the aircraft was nearly a sure thing.

    So the only significant legacy threat is someone bringing the plane down with a bomb. If someone is serious about that task, they will be successful. And by “serious”, I mean “willing to shove two pounds of plastique up their ass.” Everyone, around the world, seems to be willing to accept that risk.

    So, why the hell are we telling grandmothers they can’t take their knitting needles into the airplane? Security theater, that’s why. Nothing else.

    In South Africa, a hugely crime-ridden and violent country with a not insignificant history of domestic terrorism, airport screening is quick and courteous: the bag goes through X-ray, you walk through the magnetometer, and you’re done. The shoes don’t come off. Liquids aren’t banned. There’s no shouting and bullying of passengers. And their airplanes aren’t any less safe than ours.

  13. Not that I am in the slightest bit okay with all the post 9-11 airport BS, effective or not, but 99% of positives being false would not be that many if the 1% true positives were actual terrorists rather than, say, someone breaking parole or some whatever. If you assume maybe 1 in a million fliers is a real terrorist (I’m guessing that is a high estimate), that’d be a total of a mere 100 people pulled aside out of a million to catch the 1 real terrorist. If you have a test that is less than perfect, and you are testing for something with low incidence rate, you better hope you have a high false positive rate, because if you don’t you almost surely have an astronomically high false negative rate.

  14. Isn’t a definition of a psychotic, a person who repeats an action while expecting a different outcome. “A loss of contact with reality” pretty much describes the TSA.

  15. I’m more afraid of the arbitrary authority given to the TSA screeners than I am of flying.

    I no longer assume the delays I may face will be the results of the best efforts of professionals.

  16. I certainly don’t want to be a TSA apologist, but:

    1) I think there is pretty good evidence that the micro-expressions identified by Paul Ekman are a reliable measure of affect, though this does not mean that a group of people who, as one poster noted, seem to be poor observers anyway are going to be able to become skilled readers of these micro-expressions. And if they do, they can make more money in Vegas anyway.

    2) This is all a numbers game. There will, of course, be false positives. The number of non-terrorists caught by metal detectors is probably *way* more than 99%. I have personally seen hundreds of people who have run afoul of this machine, who I assume were not (successful, immediate) terrorists. Yet I don’t see a lot of calls to do away with metal detectors, in part because most people don’t want to sit next to a drunk jerk on a plane who is also packing a six-shooter.

    So, the question is whether they should randomly or purposively sample fliers, and if the latter, how. I think random sampling makes a lot more sense, but as long as they are going the former route, suspicious behavior seems at least as worthwhile a red flag as suspicious bag content.

    The real solution: let the airlines decide. The trade-offs between large numbers of small planes (less people killed if they go down, but more opportunity for collisions and deliberate crashes), and large planes (successfully downing increases impact, and encourages attackers, but at least it requires a smaller number of trusted pilots on the whole, and reduces the total number of objects in the sky), is an interesting one to consider.

  17. If this behavior detection works so great, maybe TSA can start using it on their employees to figure out who keeps stealing my partner’s possessions from his luggage.

    Last week, once again, he was robbed by TSA. This time it was an expensive sailor’s knife I had engraved as a gift to him – replaceable but somehow not the same from a sentimental standpoint.

    If they can’t screen their own employees for criminals, how can they screen millions of passengers for terrorists? If they can’t keep their employees from taking stolen items out of the airport, how can they keep actual dangerous items from going *into* airports?

    Of course, that’s not really the point is it? What a waste of money these guys are…

  18. The blanket insults of security personnel are unwarranted and unfeeling. These are people working a low paying job…enforcing rules they did not have ANY role in creating….and who (for the most part) have some sense of pride that they keep people safe (even if they are wrong about that assumption).
    Calling them slack jawed and fascists and bullies is overly simplistic and ignorant. Of course there are those who take what little power they have been given and become little Stalins……that happens in every situation where power is given, at all levels of society…but encouraging and participating in ridiculous stereotyping of people who are by and large trying to do a positive thing while trying to make a living should be beneath the level of discourse this site appears to aspire to.

  19. Frankly, I find that the TSA security theater makes me very very nervous, as I’m afraid of being pulled for a false positive, which ends up looking like a vicious cycle.

    Bleh. Flying sucks.

  20. I’m as anti-TSA as any other Bruce Schneier fan, but… let’s remember some basic statistics.

    “99% false positives” is not by itself a bad thing, not necessarily, if it means “99% of the people it picks out were not planning on causing trouble”. The real question is; What percentage of real intended troublemakers does it NOT catch? If a tiny fraction of air travelers (I dunno, 0.01%) intend to cause trouble, and this method manages to identify all of them (or most of them) along with all those false positives, then it may be a worthwhile inconvenience. I don’t THINK this method is that effective at catching bad guys (it may be, even if it is very inefficient, i.e. catches lots of non-bad guys too), but who knows. Worth a shot. (Although somehow I have a feeling that if the method fails to work, the TSA will just keep trying it…)

  21. To Quote the man in the hot air balloon with the cape and goggles:

    “If you ever decide to do something as stupid as build an automatic terrorism detector, here’s a math lesson you need to learn first. It’s called “the paradox of the false positive,” and it’s a doozy.

    Say you have a new disease, called Super-AIDS. Only one in a million people gets Super-AIDS. You develop a test for Super-AIDS that’s 99 percent accurate. I mean, 99 percent of the time, it gives the correct result — true if the subject is infected, and false if the subject is healthy. You give the test to a million people.

    One in a million people have Super-AIDS. One in a hundred people that you test will generate a “false positive” — the test will say he has Super-AIDS even though he doesn’t. That’s what “99 percent accurate” means: one percent wrong.

    What’s one percent of one million?

    1,000,000/100 = 10,000

    One in a million people has Super-AIDS. If you test a million random people, you’ll probably only find one case of real Super-AIDS. But your test won’t identify one person as having Super-AIDS. It will identify 10,000 people as having it.

    Your 99 percent accurate test will perform with 99.99 percent inaccuracy.

    That’s the paradox of the false positive. When you try to find something really rare, your test’s accuracy has to match the rarity of the thing you’re looking for. If you’re trying to point at a single pixel on your screen, a sharp pencil is a good pointer: the pencil-tip is a lot smaller (more accurate) than the pixels. But a pencil-tip is no good at pointing at a single atom in your screen. For that, you need a pointer — a test — that’s one atom wide or less at the tip.

    This is the paradox of the false positive, and here’s how it applies to terrorism:

    Terrorists are really rare. In a city of twenty million like New York, there might be one or two terrorists. Maybe ten of them at the outside. 10/20,000,000 = 0.00005 percent. One twenty-thousandth of a percent.

    That’s pretty rare all right. Now, say you’ve got some software that can sift through all the bank-records, or toll-pass records, or public transit records, or phone-call records in the city and catch terrorists 99 percent of the time.

    In a pool of twenty million people, a 99 percent accurate test will identify two hundred thousand people as being terrorists. But only ten of them are terrorists. To catch ten bad guys, you have to haul in and investigate two hundred thousand innocent people.

    Guess what? Terrorism tests aren’t anywhere close to 99 percent accurate. More like 60 percent accurate. Even 40 percent accurate, sometimes.

    What this all meant was that the Department of Homeland Security had set itself up to fail badly. They were trying to spot incredibly rare events — a person is a terrorist — with inaccurate systems.

    Is it any wonder we were able to make such a mess?”

  22. Maturin:

    The blanket insults of Nazi footsoldiers are unwarranted and unfeeling. These are people working a low paying job…enforcing rules they did not have ANY role in creating….and who (for the most part) have some sense of pride that they keep people safe (even if they are wrong about that assumption).

    Do you see the fallacy now?

    Citizens have a right and a duty to not participate in the wholesale forfeiture of the standards of American civilisation, even if it means not taking the twelve-dollar-per-hour (TSA screener entry-level pay is 23,400 per year and can go up to 35,000, or eighteen an hour) job that requires them to violate people’s privacy without reason and without warrant.

  23. I’m surprised it’s that good. I mean… If 1 in 100 people it’s catching are actually trying to thwart airport security in some way, it must not be flagging very many people at all, right?

    Oh, wait… Actually this statistic is meaningless out of context. What percentage of people are “breaking a law” at airport security? Do 1 in 100 people have a known unauthorized item? Perhaps they’re trying to exceed their duty free limit, or have 3.5oz of shampoo?

    We don’t know if this is better or worse than random searches and interrogations unless we have a baseline.

    I wouldn’t be surprised if it does worse than random.

  24. The truth is that stories of TSA failures or incompetence only make me smirk now where I used to get angry. I have outrage fatigue, which is probably what they’re going for.

    I am by no means a “road warrior” as I only travel 5 or 6 times a year. However, the idiocy and lack of consistency no longer surprise me. I no longer fly America West because Phoenix, Sky Harbor is the home of the most stupid, knuckle-draggers in TSA.

    I will praise both ATL and MCO for having numerous security lanes that are reasonable staffed. ATL is especially nice because even if you are pulled aside for security screening, you can see your belongings at all times.

    I am officially a senseless drone.

    I simply do not know how to respond to a comparison between baggage screeners and Nazi footsoldiers.

    No I do not see the fallacy of my initial statement that is it wrong to make blanket insults of a profession that by and large seeks to achieve a societal good (even if I question the usefulness of some of their rules). There is a middle ground between these people being a total success and your histrionics about their Nazi agenda. That middle ground does not require calling them slack-jawed and fascist.

    I too think that the actions of the TSA are inept and and often misguided. They are not without reason.

  26. Government agencies do not produce, by and large, “low paying jobs” unless your definition is different that the norm.

    TSA should be abolished, or at the very least, trimmed heavily. It is exactly the type of agency that would not be missed at all and would save billions of government dollars. Go, Obama administration!

  27. Should I comment, really? this is reason #XXXXXXX why the TSA is a fully ineffective joke. Bin Laden had probably flown across the US a dozen times without getting noticed because they’re busy studying the facial expressions of philandering husbands on a “business” trip.

    How to find a needle in the haystack? add more hay!

  28. and there is a point of attack: if Phoenix Sky Harbour is indeed the very worst as Crysharris points out, then publicize it here. Do a user survey on BB of TSA rating. Let travelers converge by survey to make a list by airport of the Worst TSA In The Country. Give it an award name. Spread it. This could be a crystal to precipitate change around. Giving them a name (by airport) could be a first step to making the TSA publicly accountable – as opposed to a vast, shapeless oppressor. If nothing else, it will attach some shame to those who work at and thereby create, the worst offender. Maybe even force their management to take a good look at them. Assuming their management actually does anything.

  29. ….need an award title name…. who is the epitome of officious, petty martinet stupidity in current pop culture these days? A name all will recognize… must consider the flying demographic…

  30. i’ve read boingboing’s numerous TSA horror stories for years, and while i totally sympathize, i’m disturbed by the blanket-labelling of TSA employees as incompetent, or “slack-jawed.” My brother in law works for TSA, and is in fact a behavior screener. I’ve thrown him many an insult or insinuation about his job, but he’s taken it in stride and tells me he’s not picking on Arabs or putting a terminal in lock-down cuz someone’s in a bad mood. While he hasn’t busted any terrorists, he’s found several people running copious amounts of narcotics. Despite all of bb’s horror stories, he’s given me an example of a judicious, intelligent, and diligent TSA employee.

    The 3oz. liquid rule is still total bullshit though.

  31. – So, if the passanger weights the same as a duck, he floats and must be made of wood…
    – And…
    – HE IS A TERRORIST. Hold him indefinitely without trial.

  32. Jericho’s post at #37 is telling, I think. I know I don’t believe that the wiretapping and screening is done to catch terrorists – judge by results: it is done to catch criminals. It was sold to the public as a means to fight terrorism, but it is being used to fight crime, principally drug crime.

  33. So remember kids, don’t support the terrorists – buy only locally grown!
    (easy for me, a Californian, to say)

  34. Some time ago, when the airports touted the eye-scan as the newest technology to solve all the issues of identity-checks, there was a stated average of incorrect results at some few percent. Even if just say, 1 percent, just taking one day of passengers at Heathrow alone, that volume of incorrect hits will be incredible to handle. Add to that the “expert-training” of these highly paid screeners…well… It’s just adding more and more technological half-steps on the way to dealing with the real issues of “security”.

  35. I agree with Marcel.

    ++reminds me of an interview test described by a not very smart person.

    Any test, though it may have been Invented by a Genius who can tell you within 30 seconds of meeting a couple if they’re going to divorce, is only as good as the Intelligence and Refinement of those Administering it.


    ex: If you Do run into 1 guy who is abusing his job, where do you file the Complaint with his/her name and badge number?

    +++ If the TSA employees can “screen” people in any way, where is the set of employment + psych exams that screen out: Criminals, Sociopaths, Bullies, & Psychopaths???

    I’ve met 1 or 2 nutty people over the years, and my impression was the last thing they’d need was a job with any kind of Power.

Comments are closed.