David Weinberger took great notes from what sounds like a barn-burner of a talk by Anil Dash at Harvard's Berkman Center on what has happened to the net, and where it's headed:

“We have a lot of software that forbids journalism.” He refers to the IoS [iphone operating system] Terms of Service for app developers that includes text that says, literally: “If you want to criticize a religion, write a book.” You can distribute that book through the Apple bookstore, but Apple doesn’t want you writing apps that criticize religion. Apple enforces an anti-journalism rule, banning an app that shows where drone strikes have been.

Less visibly, the laws is being bent “to make our controlling our data illegal.” All the social networks operate as common carriers — neutral substrates — except when it comes to monetizing. The boundaries are unclear: I can sing “Happy Birthday” to a child at home, and I can do it over FaceTime, but I can’t put it up at YouTube [because of copyright]. It’s very open-ended and difficult to figure. “Now we have the industry that creates the social network implicitly interested in getting involved in how IP laws evolve.” When the Google home page encourages visitors to call their senators against SOPA/PIPA, we have what those of us against Citizens United oppose: now we’re asking a big company to encourage people to act politically in a particular way. At the same time, we’re letting these companies capture our words and works and put them under IP law.

A decade ago, metadata was all the rage among the geeks. You could tag, geo-tag, or machine-tag Flickr photos. Flickr is from the old community. That’s why you can still do Creative Commons searches at Flickr. But you can’t on Instagram. They don’t care about metadata. From an end-user point of view, RSS is out of favor. The new companies are not investing in creating metadata to make their work discoverable and shareable.

[berkman] Anil Dash on “The Web We Lost” (via Beyond the Beyond)

In Wired, Steven Levy has a long profile of the fascinating field of algorithmic news-story generation. Levy focuses on Narrative Science, and its competitor Automated Insights, and discusses how the companies can turn "data rich" streams into credible news-stories whose style can be presented as anything from sarcastic blogger to dry market analyst. Narrative Science's cofounder, Kristian Hammond, claims that 90 percent of all news will soon be algorithmically generated, but that this won't be due to computers stealing journalists' jobs -- rather, it will be because automation will enable the creation of whole classes of news stories that don't exist today, such as detailed, breezy accounts of every little league game in the country.

Narrative Science’s writing engine requires several steps. First, it must amass high-quality data. That’s why finance and sports are such natural subjects: Both involve the fluctuations of numbers—earnings per share, stock swings, ERAs, RBI. And stats geeks are always creating new data that can enrich a story. Baseball fans, for instance, have created models that calculate the odds of a team’s victory in every situation as the game progresses. So if something happens during one at-bat that suddenly changes the odds of victory from say, 40 percent to 60 percent, the algorithm can be programmed to highlight that pivotal play as the most dramatic moment of the game thus far. Then the algorithms must fit that data into some broader understanding of the subject matter. (For instance, they must know that the team with the highest number of “runs” is declared the winner of a baseball game.) So Narrative Science’s engineers program a set of rules that govern each subject, be it corporate earnings or a sporting event. But how to turn that analysis into prose? The company has hired a team of “meta-writers,” trained journalists who have built a set of templates. They work with the engineers to coach the computers to identify various “angles” from the data. Who won the game? Was it a come-from-behind victory or a blowout? Did one player have a fantastic day at the plate? The algorithm considers context and information from other databases as well: Did a losing streak end?

Then comes the structure. Most news stories, particularly about subjects like sports or finance, hew to a pretty predictable formula, and so it’s a relatively simple matter for the meta-writers to create a framework for the articles. To construct sentences, the algorithms use vocabulary compiled by the meta-writers. (For baseball, the meta-writers seem to have relied heavily on famed early-20th-century sports columnist Ring Lardner. People are always whacking home runs, swiping bags, tallying runs, and stepping up to the dish.) The company calls its finished product “the narrative.”

Both companies claim that they'll be able to make sense of less-quantifiable subjects in the future, and will be able to generate stories about them, too.

Can an Algorithm Write a Better News Story Than a Human Reporter?

Benjamin Mako Hill writes, "Last year, I participated in a discussion on Wikipedia that led to the deletion of an article about the "Institute for Cultural Diplomacy." Because I edit Wikipedia using my real name, the ICD was able to track me down. Over the last month or so, they threated me with legal action and have now gotten their lawyers involved. I've documented the whole sad saga on my blog. I think the issue raises some important concerns about Wikipedia in general."

Donfried has made it very clear that his organization really wants a Wikipedia article and that they believe they are being damaged without one. But the fact that he wants one doesn’t mean that Wikipedia’s policies mean he should have one. Anonymous editors in Berlin and in unknown locations have made it clear that they really want a Wikipedia article about the ICD that does not include criticism. Not only do Wikipedia’s policies and principles not guarantee them this, Wikipedia might be hurt as a project when this happens.

The ICD claims to want to foster open dialogue and criticism. I think they sound like a pretty nice group working toward issues I care about personally. I wish them success.

But there seems to be a disconnect between their goals and the actions of both their leader and proponents. Because I used my real name and was skeptical about the organization on discussion pages on Wikipedia, I was tracked down and threatened. Donfried insinuated that I was motivated to “sabotage” his organization and threatened legal action if I do not answer his questions. The timing of his first letter — the day after the ICD page was recreated — means that I was unwilling to act on my commitment to Wikipedia and its policies.

The Institute for Cultural Diplomacy and Wikipedia

My latest Guardian column is "Copyright wars are damaging the health of the internet" and it looks at what we really need from proposed solutions to the copyright wars:

I've sat through more presentations about the way to solve the copyright wars than I've had hot dinners, and all of them has fallen short of the mark. That's because virtually everyone with a solution to the copyright wars is worried about the income of artists, while I'm worried about the health of the internet.

Oh, sure, I worry about the income of artists, too, but that's a secondary concern. After all, practically everyone who ever set out to earn a living from the arts has failed – indeed, a substantial portion of those who try end up losing money in the bargain. That's nothing to do with the internet: the arts are a terrible business, one where the majority of the income accrues to a statistically insignificant fraction of practitioners – a lopsided long tail with a very fat head. I happen to be one of the extremely lucky lotto winners in this strange and improbable field – I support my family with creative work – but I'm not parochial enough to think that my destiny and the destiny of my fellow 0.0000000000000000001 percenters are the real issue here.

What is the real issue here? Put simply, it's the health of the internet.

Copyright wars are damaging the health of the internet

Bruce Schneier presents a very cogent and convincing argument that "security awareness training" is a waste of money -- specifically, because the benefits of "security" are intangible, while the benefits of getting your work done are apparent.

To those who think that training users in security is a good idea, I want to ask: "Have you ever met an actual user?" They're not experts, and we can't expect them to become experts. The threats change constantly, the likelihood of failure is low, and there is enough complexity that it's hard for people to understand how to connect their behavior to eventual outcomes. So they turn to folk remedies that, while simple, don't really address the threats.

Even if we could invent an effective computer security training program, there's one last problem. HIV prevention training works because affecting what the average person does is valuable. Even if only half the population practices safe sex, those actions dramatically reduce the spread of HIV. But computer security is often only as strong as the weakest link. If four-fifths of company employees learn to choose better passwords, or not to click on dodgy links, one-fifth still get it wrong and the bad guys still get in. As long as we build systems that are vulnerable to the worst case, raising the average case won't make them more secure.

The whole concept of security awareness training demonstrates how the computer industry has failed. We should be designing systems that won't let users choose lousy passwords and don't care what links a user clicks on. We should be designing systems that conform to their folk beliefs of security, rather than forcing them to learn new ones. Microsoft has a great rule about system messages that require the user to make a decision. They should be NEAT: necessary, explained, actionable, and tested. That's how we should be designing security interfaces. And we should be spending money on security training for developers. These are people who can be taught expertise in a fast-changing environment, and this is a situation where raising the average behavior increases the security of the overall system.

Security Awareness Training

Marijke Visser from the ALA Office for Information Technology Policy writes with this provocative question:

What could a library do with a gigabit broadband connection? What kinds of services could they do that they can’t without that big of a connection? Thinking way away from the typical services libraries offer now, what are some really big ideas that would need that much connectivity? These services could happen outside the library walls, in relationship to other community organizations and/or government agencies… How would a library hooked up to a gig benefit its community?

Well?

The Internet has been groaning under the weight of a massive distributed denial of service (DDoS) attack on the Domain Name Service, apparently aimed at anti-spam vigilantes Spamhaus, in retaliation for their blacklisting of Dutch free speech hosting provider Cyberbunker. At 300 mbps, the DDoS is the worst in public Internet history.

“These things are essentially like nuclear bombs,” said Matthew Prince, chief executive of Cloudflare. “It’s so easy to cause so much damage.”

The so-called distributed denial of service, or DDoS, attacks have reached previously unknown magnitudes, growing to a data stream of 300 billion bits per second.

“It is a real number,” Mr. Gilmore said. “It is the largest publicly announced DDoS attack in the history of the Internet.”

Spamhaus, one of the most prominent groups tracking spammers on the Internet, uses volunteers to identify spammers and has been described as an online vigilante group.

In the past, blacklisted sites have retaliated against Spamhaus with denial-of-service attacks, in which they flood Spamhaus with traffic requests from personal computers until its servers become unreachable. But in recent weeks, the attackers hit back with a far more powerful strike that exploited the Internet’s core infrastructure, called the Domain Name System, or DNS.

As bad as this is, it could be a lot worse. An anonymous paper called Internet Census 2012: Port scanning /0 using insecure embedded devices reports on a researcher's project to scan every IPv4 address for publicly available machines that will accept a telnet connection and yield up a root login to a default password. The researcher reports that 1.2 million such devices are available online (s/he compromised many of these machines in order to run the census). These machines are things like printers and routers with badly secured firmware, visible on the public net. They are often running an old version of GNU/Linux and can be hijacked to form part of a staggeringly large botnet that would be virtually unkillable, since the owners of these devices are vanishingly unlikely to notice that they are silently running attackware, and the devices themselves are completely unregarded.

Firm Is Accused of Sending Spam, and Fight Jams Internet [NYT/John Markoff & Nicole Perlroth]

(via Hacker News)

Ars Technica's Nate Anderson decided to try cracking passwords (from a leaked file of MD5 hashes), to see how difficult it was. After a very long false start (he forgot to decompress the word-list file) that's covered in a little too much detail, Anderson settles down to cracking hashes in earnest, and provides some good data on the nuts and bolts of password security:

By this point I had puzzled out how Hashcat worked, so I dumped the GUI and switched back to the command-line version running on my much faster MacBook Air. My goal was to figure out how many hashes I could crack in, say, under 30 minutes, as well as which attacks were most efficient. I began again on my 17,000-hash file, this time having Hashcat remove each hash from the file once it was cracked. This way I knew exactly how many hashes each attack solved.

This set of attacks brought the number of uncracked MD5 hashes down from 17,000 to 8,790, but clearly the best "bang for the buck" came from running the RockYou list with the best64.rule iterations. In just 90 seconds, this attack would uncover 45 percent of the hashed passwords; additional attacks did little more, even those that took 16 minutes to run.

Cracking a significant number of the remaining passwords would take some much more serious effort. Applying the complex d3ad0ne.rule file to the massive RockYou dictionary, for instance, would require more than two hours of fan-spinning number-crunching. And brute force attacks using 6-character passwords only picked up a few additional results.

The point, really, is that if you want to understand the relative security of different password-generation techniques, you need to understand what's involved in state-of-the-art password cracking techniques.

How I became a password cracker

A post on ConversionXL sums up a bunch of experiments on pricing and suggests ways of combining them to best effect. All electronic goods can be had for free, so every person who buys an electronic good is essentially entering into a voluntary transaction. Getting pricing right is the best way to convince (rather than coerce) customers to pay, and to frame that payment so that it's as large as possible.

Researches found that sale price markers (with the old price mentioned) were more powerful than mere prices ending with the number nine. In the following split test, the left one won:

9 not so magical after all? Not so fast!

Then they they split tested the winner above with a similar tag, but which had $39 instead of $40:

This had the strongest effect of all.

I’m wondering whether the effect of this price tag could be increased by reducing the font size of $39. Say what?

Marketing professors at Clark University and The University of Connecticut found that consumers perceive sale prices to be a better value when the price is written in a small font rather than a large, bold typeface. In our minds, physical magnitude is related to numerical magnitude.

Pricing Experiments You Might Not Know, But Can Learn From (via O'Reilly Radar)