I've got a guest column in the new edition of The Bookseller, the trade magazine for the UK publishing industry. It's called "Tangible Assets," and it points out that of all the fights that publishing has had with the ebook sector -- DRM, pricing, promotion -- the one they've missed is access to data. Whatever else is going on with publishers and Amazon, Google, Apple, et al, the fact that publishing knows almost nothing about its ebook customers and has no realtime view into its ebook sales; and that the ebook channel knows almost everything, instantaneously, is untenable and unsustainable.

I just came off a US tour for my YA novel Homeland, which Tor Teen published in the US in February, and which Titan will publish this coming September in the UK. I went to 23 cities in 25 days, a kind of bleary and awesome whirlwind where I got to see friends from across the USA—Internet People to a one—for about 8.5 minutes each, in a caffeinated, exhausted rush.

Inevitably, I had this conversation: "How's the book doing?" and I got to say: "Oh, awesome! It's a New York Times and Indienet bestseller!" (It stayed on the NYT list for four weeks, so I got to say this a lot). And then, always: "So, how many copies does that come out to?" And my answer was always, "No one knows."

This is where the Internet People began to boggle. "No one knows?"

"Oh, there's some Nielsen reporting from the tills of participating booksellers—you can get that if you spend a fortune. But there's no realtime e-book numbers given to the publishers. We'll all find out exactly how the book performed in a couple of months."

And that's where they lost their minds. The irate squawks that emerged from their throats were audible for miles. "You mean Amazon, Apple and Google knows exactly who comes to their stores, how they find their way to your books, where they're coming in from, how many devices they use and when, and they don't tell the publishers?"

Tangible assets

Big Data is a new book from Viktor Mayer-Schonberger, a respected Internet governance theorist; and Kenneth Cukier, a long-time technology journalist who's been on the Economist for many years. As the title and pedigree imply, this is a business-oriented book about "Big Data," a computational approach to business, regulation, science and entertainment that uses data-mining applied to massive, Internet-connected data-sets to learn things that previous generations weren't able to see because their data was too thin and diffuse.

Big Data is an eminently practical and sensible book, but it's also an exciting and excitable text, one that conveys enormous enthusiasm for the field and its fruits. The authors use well-chosen examples to show how everything from shipping logistics to video-game design to healthcare stand to benefit from studying the whole data-set, rather than random samples. They even pose this as a simple way of thinking of big data versus "small data." Small data relies on statistical sampling, and emphasises the reliability and accuracy of each measurement. With big data, you sample the entire pool of activities -- all the books sold, all the operations performed -- and worry less about inaccuracies and anomalies in individual measurements, because these are drowned out by the huge numbers of observations performed.

As you'd expect, Big Data is particularly fascinating when it explores the business implications of all this: the changing leverage between firms that own data versus the firms that know how to make sense of it, and why sometimes data is best processed by unaffiliated third parties who can examine data from rival firms and find out things from which all parties stand to benefit, but which none of them could have discovered on their own. They also cover some of the bigger Big Data business blunders through history -- companies whose culture blinkered them to the opportunities in their data, which were exploited by clever rivals.

The last fifth of the book is dedicated to issues of governance, regulation, and public policy. This is some of the most interesting material in the book and probably needs to be expanded into its own volume. As it is, there's a real sense that the authors are just scraping the surface. For example, many of the stories told in the book have deep privacy implications, and the authors make a point of touching on these, cabining them with phrases like "so long as the data is anonymized" or "adhering to privacy policy, of course." But in the final third, the authors examine the transcendental difficulty of real-world anonymization, and the titanic business blunders committed by firms that believed they'd stripped out the personal information from the data, only to have the data "de-anonymized" and their customers' privacy invaded in small and large ways. These two facts -- that many of the opportunities require effective anonymization and that no one knows how to do anonymization -- are a pretty big stumbling block in the world of Big Data, but the authors don't explicitly acknowledge the conundrum.

While Big Data is an excellent primer on the opportunities of the field, it's thin on the risks, overall. For example, Big Data is rightly fascinated with stories about how we can look at data sets and find predictors of consequential things: for example, when Google mined its query-history and compared it with CDC data on flu outbreaks, it found that it could predict flu outbreaks ahead of the CDC, which is amazingly useful. However, all those search-strings were entered by people who didn't expect to have them mined for subsequent action. If searching for "scratchy throat" and "runny nose" gets your neighborhood quarantined (or gets it extra healthcare dollars), you might get all your friends to search on those terms over and over -- or not at all. Google knows this -- or it should -- because when it started measuring the number of links between sites to define the latent authority of different parts of the Internet, it got great results, but immediately triggered a whole scummy ecosystem of linkfarms and other SEO tricks that create links whose purpose is to produce more of the indicators Google is searching for.

Another important subject is looking at algorithmic prediction in domains where the outcome is punishment, instead of reward. British Airways may get great results from using an algorithm to pick out passengers for upgrades, trying to find potential frequent fliers. But we should be very cautious about applying the same algorithm to building the TSA's No-Fly list. If BA's algorithm fails 20% of the time, it just means that a few lucky people get to ride up front of the plane. If the TSA has a 20% failure rate, it means that one in five "potential terrorists" is an innocent whose fundamental travel rights have been compromised by a secretive and unaccountable algorithm.

Secrecy and accountability are the third important area for examination in a Big Data world. Cukier and Mayer-Schonberger propose a kind of inspector-general for algorithms who'll make sure they're not corrupted to punish the undeserving or line someone's pockets unjustly. But they also talk about the fact that these algorithms are likely to be illegible -- the product of a continuously evolving machine-learning system -- and that no one will be able to tell you why a certain person was denied credit, refused insurance, kept out of a university, or blackballed for a choice job. And when you get into a world where you can't distinguish between an algorithm that gets it wrong because the math is unreliable (a "fair" wrong outcome) from an algorithm that gets it wrong because its creators set out to punish the innocent or enrich the undeserving, then we can't and won't have justice. We know that computers make mistakes, but when we combine the understandable enthusiasm for Big Data's remarkable, counterintuitive recommendations with the mysterious and oracular nature of the algorithms that produce those conclusions, then we're taking on a huge risk when we put these algorithms in charge of anything that matters.

Big Data: A Revolution That Will Transform How We Live, Work, and Think

Previously: Book about big data, predictive behavior, and decision making

Raytheon's "RIOT" (Rapid Information Overlay Technology) is intended to help governments all over the world by providing a "Google for spies" that mines multiple online sources to build up detailed pictures of the personal activities of their citizens:

The sophisticated technology demonstrates how the same social networks that helped propel the Arab Spring revolutions can be transformed into a "Google for spies" and tapped as a means of monitoring and control.

Using Riot it is possible to gain an entire snapshot of a person's life – their friends, the places they visit charted on a map – in little more than a few clicks of a button.

In the video obtained by the Guardian, it is explained by Raytheon's "principal investigator" Brian Urch that photographs users post on social networks sometimes contain latitude and longitude details – automatically embedded by smartphones within so-called "exif header data."

Riot pulls out this information, showing not only the photographs posted onto social networks by individuals, but also the location at which the photographs were taken.

"We're going to track one of our own employees," Urch says in the video, before bringing up pictures of "Nick," a Raytheon staff member used as an example target. With information gathered from social networks, Riot quickly reveals Nick frequently visits Washington Nationals Park, where on one occasion he snapped a photograph of himself posing with a blonde haired woman.

"We know where Nick's going, we know what Nick looks like," Urch explains, "now we want to try to predict where he may be in the future."

Riot can display on a spider diagram the associations and relationships between individuals online by looking at who they have communicated with over Twitter. It can also mine data from Facebook and sift GPS location information from Foursquare, a mobile phone app used by more than 25 million people to alert friends of their whereabouts. The Foursquare data can be used to display, in graph form, the top 10 places visited by tracked individuals and the times at which they visited them.

The video shows that Nick, who posts his location regularly on Foursquare, visits a gym frequently at 6am early each week. Urch quips: "So if you ever did want to try to get hold of Nick, or maybe get hold of his laptop, you might want to visit the gym at 6am on a Monday."

The associated patent says that Raytheon believes that its software can judge whether its subjects constitute a "security risk"

Software that tracks people on social media created by defence firm [Guardian/Ryan Gallagher]

Here's a lovely video shot at the XS4ALL ISP data-center outside of Amsterdam, in which the many twinkling, blinking lights are synchronized to a sprightly piano score.

De achterkant van het Internet (Thanks, Neils!)

On Twitter's engineering blog, a fascinating description of how Twitter uses a blend of machine intelligence and Mechanical Turk tasks to figure out, in real time, what is going on in the world:

Before we delve into the details, here's an overview of how the system works.

1. First, we monitor for which search queries are currently popular.
   Behind the scenes: we run a Storm topology that tracks statistics on search queries.
   For example, the query [Big Bird] may suddenly see a spike in searches from the US.
   
   
2. As soon as we discover a new popular search query, we send it to our human evaluators, who are asked a variety of questions about the query.
   Behind the scenes: when the Storm topology detects that a query has reached sufficient popularity, it connects to a Thrift API that dispatches the query to Amazon's Mechanical Turk service, and then polls Mechanical Turk for a response.
   For example: as soon as we notice "Big Bird" spiking, we may ask judges on Mechanical Turk to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps us serve relevant Tweets and ads.
   
   
3. Finally, after a response from an evaluator is received, we push the information to our backend systems, so that the next time a user searches for a query, our machine learning models will make use of the additional information. For example, suppose our evaluators tell us that [Big Bird] is related to politics; the next time someone performs this search, we know to surface ads by @barackobama or @mittromney, not ads about Dora the Explorer.

Improving Twitter search with real-time human computation (via Waxy)

Jeremy Kun, a mathematics PhD student at the University of Illinois in Chicago, has posted a wonderful primer on probability theory for programmers on his blog. It's a subject vital to machine learning and data-mining, and it's at the heart of much of the stuff going on with Big Data. His primer is lucid and easy to follow, even for math ignoramuses like me.

For instance, suppose our probability space is and is defined by setting for all (here the “experiment” is rolling a single die). Then we are likely interested in more exquisite kinds of outcomes; instead of asking the probability that the outcome is 4, we might ask what is the probability that the outcome is even? This event would be the subset , and if any of these are the outcome of the experiment, the event is said to occur. In this case we would expect the probability of the die roll being even to be 1/2 (but we have not yet formalized why this is the case).

As a quick exercise, the reader should formulate a two-dice experiment in terms of sets. What would the probability space consist of as a set? What would the probability mass function look like? What are some interesting events one might consider (if playing a game of craps)?

Probability Theory — A Primer

(Image: Dice, a Creative Commons Attribution (2.0) image from artbystevejohnson's photostream)