The claim seems to be that if you collect data overzealously, you a) increase the time to process your query, and b) increase the likelihood of false positives.

For a), it seems that’s a pretty trivial problem, with the most obvious solution being “bigger computers”. Surely it’s possible to crunch through a larger data set, it’s just hard for the current hardware. More expensive tech, or future tech can easily solve that problem.

For b), false positives can be reduced by failing to collect data, or by collecting it but filtering that data out in your query. Most such filters are difficult or ineffective now only due to the speed or storage limitations of particular systems. Basically, false positives can be reduced by improving the queries, which again, is a relatively trivial problem.

My concern is that these kinds of objections are easy to resolve, and suggest that opposition to invasive surveillance stems from a complaint about it “not being good enough”. And that’s not really an intractable problem. After all, even if our data mining techniques are crude now, there’s no harm in gathering the data for some eventuality when we have fast enough computers and good enough queries to mine it effectively….

Humans don’t operate in binary. We’re not black and white.

I believe that the Japanese have now optimized their line-ups to only one suspect. Efficient.

In reality, there is only a small chance that your target is in the original dataset, and an even smaller (ok, really, REALLY small) chance that it’s in an expanded dataset. However, there’s still a possibility, which means that in reality, adding more “hay” does help, because it’s possible that the needle was in the hay you added.

Again, I’m not arguing in favor of this kind of surveillance, I’m just saying this isn’t a good way to argue against it.

“No one is innocent, citizen. We are merely here to determine the level of your guilt.” [Judge Dredd]

@7: You say that the cameras installed in local schools are OK because they’re intended as a documentation system, not a security system. The intentions of surveillance is always presented as benign, and if they were always used benignly there would likely be no problem. The outstanding question is whether the authorities in control of the surveillance can be trusted to use it fairly and responsibly.

In other words, it’s not the quantity of data you have on someone, nor is it necessarily the quality: it’s the relevance.

I really am genuinely curious as to the economics of it all. Is it better and easier to jail a large segment than to find them work? CCTV is just a small part of a vast feeder system to fill up the prisons. I think they need to get properly organized on this.

3 percent!?! Aside from the civi liberties shredding, that’s an awful return on investment for the millions spent putting up these electro-eyes.

As security expert Bruce Schneier has said, when you’re looking for a needle it a haystack, it doesn’t help to add more hay

This is also complete balls. It sounds snappy and must be the kind of thing news outlets would lap up. Now OBVIOUSLY if we’re blindly groping around for a positive, then adding more negatives will reduce our conditional probability of success. But we don’t get to pick how much hay is in the haystack. We collect some data, see if it supports our ideas, and then adjust both accordingly. But if less data was always enough, we’d just look at gender and be done with it.

Evidence on how these things can be bad exist in the facts of several government agencies in Britain, Canada, and the US and other places losing huge amounts of personal info (and fingerprints can be faked, so access to inncent people’s fingerprints would be useful). If they can lose medical and credit info they can lose this.

And for false positives one need look no further than the TSA no fly list, which would appear by all evidence to be no more than a first name and a last name with all similar names being flagged. Resulting in everyone from toddlers to war heros to security personnel being refused access to planes.

We were safer ten years ago.

Or we could try analyzing the data without a preconceived notion about the results. Nah.

As with Zikzak’s analysis, it all breaks down at the human point of contact. All we need is a society of infallible human beings, and everything will be okay. But we aren’t infallible, and neither are those who we appoint to take responsibility for our governance. Engineering better hardware is the trivial issue; engineering a better society is the challenge.

They managed that one pretty well already with Sacco and Vanzetti.

As we’ve seen with the public photos situation, there are few consequences for authorities breaking laws… though they’ll look pretty stupid if they keep it up.

All the same, I imagine that even if Homeland Security acknowledged this problem, their solution would be to build bigger computers, or try to upgrade the AI, rather than collect more efficient intelligence.

Those anecdotes are accurate for the way that a human mind processes information. That’s probably the reason this idea of “too much information makes it harder to find what you’re looking for” seems so intuitive – for human beings, it’s very true.

It’s not true for computers though, or at least a LOT less true. If I’m searching for a value in a database table, it doesn’t much matter whether the database has 1,000 rows or 100,000 rows. It matters marginally, in terms of disk speed and processor time, but that’s trivially solved by buying more disks or processors.

Obviously there’s a cutoff point. Trillions of terabytes of data will be significantly slower to datamine than 1 gig, but the point is the cutoff is so high as to be mostly irrelevant. If a large amount of data is making your queries less effective, it’s almost certainly because your data-mining methods need to be improved.

@9: Your view on profiling sounds made-up to me. Reviewing the published literature on the subject shows that broad use (e.g., traffic stops) may not be effective. However, I see no evidence that it is ineffective for highly specific use. This information is extremely helpful when used to predict behavior in all kinds of areas: consumer behavior, health behavior, etc. So the idea that it is helpful in predicting criminal behavior is not unreasonable. Whether those with the data have the capability to use it effectively is an entirely separate issue (and where the true concern should be).

Of course a competent analysis also has to look at the difference in consequences of diagnosis failures, or in a case like this the consequence of not having that information gathering.

Which will never happen in our OMG MUST TAPE EVERYTHING culture, because looking into that would require confronting the fact that (a) CCTV shows no indicating of decreasing crime, merely moving it and (2) we’d have to look into how rarely it is helpful considering the equipment and labor costs. That doesn’t even confront the lower quality of life (privacy) and opportunity costs of having law enforcement watching crime on the tube rather than in person, where they might do some good.