Round Up the Usual Suspects

As a birthday present to me, the FBI's Terrorist Watch List database added its one millionth entry this week, according to the ACLU's estimate.

I've been waiting for this event, because the one millionth entry gives us a nice round number to do the calculations which demonstrate that the terrorist watch list is as close to completely useless as it's possible for a manmade artifact to get. (Note that the database doesn't actually contain a million identities; it's got a million records representing - at a guess - about 400,000 distinct individuals. But since it's my birthday we're gonna pretend that there are a million identities, in order to make all the math turn out nice and pretty.)

Let's assume that we know the names of 1,000 terrorists, and that there are another 9,000 people with terrorist intent whose names we don't know.

According to the Department of Commerce, about 46 million international travellers visited the United States in 2004; the number seems to be holding pretty steady at about 4 million visitors a month. For ease of calculation we'll round this up very slightly to 50 million a year.

When you add this to more than 25 million Americans travelling abroad each year (it's really more than that; 25 million is just for the summer), you've got in excess of 75 million people crossing the borders in a year.

Not a lot of them are terrorists; let's say we get 500 terrorists a year trying to get into the country, and that of these 500, 10% (50) are known bad guys and 450 are new recruits who we don't yet know are bad guys.

So to sum up, each year we expect to have:

• 1,000 known terrorists
• 9,000 unknown terrorists
• 10,000 total terrorists
• 1,000,000 watch list entries
• at least 1,000 and no more than 10,000 actual terrorists on the watch list
• at least 990,000 non-terrorists on the watch list
• 75,000,000 total border crossers
• 500 terrorist border crossers
• 50 known terrorist border crossers
• 450 unknown terrorist border crossers
• 74,999,500 total non-terrorist border crossers

IMPORTANT NOTE: If you don't like my numbers, I invite you to plug your own numbers into the calculations below. For any plausible set of numbers, the conclusion will remain the same. If you prefer an implausible set of numbers, there are lots of conspiracy theory blogs which will probably make you a happier person than you'll be if you keep reading here - or you could wait a week or two and catch the opening of "The X-Files: I Want To Believe".

Let's assume that all variables are independent and random, and let's assume that there's never any error in matching a person against a name on the watch list. There are six cases of interest at a border crossing:

1. Non-terrorist is checked and does not match an entry on the list. Since there are 990,000 non-terrorists on the watch list, there are (74,999,500) - (990,000) = 74,009,500 non-terrorists who are not on the watch list, and this event happens in (74,009,500) / (75,000,000) = 98.7 percent of the cases.
2. Non-terrorist is checked and matches an entry on the list. Since there are 990,000 non-terrorists on the list, this event happens in (990,000) / (75,000,000) = 1.3 percent of the cases.
3. Known terrorist is checked and does not match an entry on the list. We don't make matching mistakes, so this event doesn't ever happen.
4. Known terrorist is checked and matches an entry on the list. This happens every time a known bad guy tries to enter the country. On the other hand, only 500 bad guys enter the country, and only 50 of them are known bad guys, so this event happens (50) / (75,000,000) = .00007 percent of the cases.
5. Unknown terrorist is checked and matches an entry on the list. Since this guy is an unknown terrorist, the probability that his name is on the list is the same as the probability that an innocent civilian's name is on the list. That probability is 1 in 75 (1,000,000 names on the list; 75,000,000 total travellers). There are 450 unknown terrorists crossing the border, so (450) / (75) = 6 of their names are on the list. So this event happens in (6) / (75,000,000) = 0.000008 percent of the cases
6. Unknown terrorist is checked and does not match an entry on the list. 444 unknown terrorists who try to cross the border aren't on the list, so this happens in (444) / (75,000,000) = 0.0006 percent of the cases

Now let's have a look at the results in rank order.

Outcome	Occurrences
Innocent non-match	74,009,500
Innocent match	990,000
Unknown terrorist non-match	444
Known terrorist match	50
Unknown terrorist match	6
Known terrorist non-match	0

We match (50 + 6) / (444 + 50 + 6) = 11.2% of terrorists using this scheme.

Of the people matched, (50 + 6) / (990,000 + 50 + 6) = 0.006% are terrorists. Put another way, 99.994% of all people matched are innocent.

It's bad enough that we're letting 90% of the terrorists cross our border without additional checks, and that we're putting 990,000 innocent people through unecessary additional checks.

What's worse is that we're probably arresting some of those 990,000 innocent people because they matched the list and "seem suspicious" (ask Brandon Mayfield about this!)

What's even worse than this is that we're training the people who operate the system to ignore the real terrorist matches when they happen. 9999 out of every 10,000 matches is a false match. After the first 5,000 or so false matches, normal humans start to assume that every match is a false match (this is called "habituation", or "the fallacy of induction"). When that one true match (a real terrorist) sets off the alarm, the operator's natural tendency is just to turn the alarm off and wave the guy through.

But what's worst of all is that this system is trivially easy for even the dumbest terrorist to circumvent. It doesn't take a genius to figure out that the thing to do to defeat this system is stop sending known terrorists through it. Catching a new recruit without a terrorist history happens only by accident, and it happens with very low probability.

We're spending God knows how many millions of dollars on this list, and it cannot possibly do the job for which it's intended. We could "fix" the system to the extent that we provide a way to take innocent people off it, but it will always be the case that the huge majority of people checked against the list are innocent. As long as this is the case, the base rate fallacy will make the system essentially worthless for catching bad guys. And this is even if the bad guys are dumb enough to enter the country at controlled border crossings and send known terrorists using papers issued under their real names.

I realize that it's bureaucratically impossible to dismantle a large government system which has been publicly criticized, so in a helpful and public-spirited gesture I'll offer the following alternative suggestion:

Put everybody on the list.

It's cheap, it's fast, it's inevitable eventually anyway as long as the list continues to grow at its current rate, and it makes checking people against the list really easy (you can do it even without a computer!).

After you've put everybody on the list, implement something that might actually work as your secondary screening process.

Bruce blogged about the absurdity of the watch list here.