On Twitter's engineering blog, a fascinating description of how Twitter uses a blend of machine intelligence and Mechanical Turk tasks to figure out, in real time, what is going on in the world:
Before we delve into the details, here's an overview of how the system works.
First, we monitor for which search queries are currently popular.
Behind the scenes: we run a Storm topology that tracks statistics on search queries.
For example, the query [Big Bird] may suddenly see a spike in searches from the US.
As soon as we discover a new popular search query, we send it to our human evaluators, who are asked a variety of questions about the query.
Behind the scenes: when the Storm topology detects that a query has reached sufficient popularity, it connects to a Thrift API that dispatches the query to Amazon's Mechanical Turk service, and then polls Mechanical Turk for a response.
For example: as soon as we notice "Big Bird" spiking, we may ask judges on Mechanical Turk to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps us serve relevant Tweets and ads.
Finally, after a response from an evaluator is received, we push the information to our backend systems, so that the next time a user searches for a query, our machine learning models will make use of the additional information. For example, suppose our evaluators tell us that [Big Bird] is related to politics; the next time someone performs this search, we know to surface ads by @barackobama or @mittromney, not ads about Dora the Explorer.
In 1996, in the midst of the Clinton administration’s attack on the Internet and cryptography, Grateful Dead lyricist and EFF co-founder John Perry Barlow sat down in Davos, Switzerland, where he’d been addressing world leaders on the subject of the Internet and human rights, and wrote one of net-culture’s formative documents: The Declaration of Independence […]
Vulture presents a lengthy (and very funny) annotated history of “100 jokes that shaped modern comedy,” with embedded audio (and sometimes video) of the jokes themselves, going all the way back to 1906’s Nobody by Bert Williams — transferred from wax cylinder to shellac disc to Youtube.
Emily Sears has a longstanding, devastatingly effecting procedure for handling the unsolicited dick pix, wanking videos, and sexist come-ons she receives from creepy Internet randos: she researches their girlfriends and messages them with screengrabs of the whole thing.
You travel around a lot. It might be that jet set life from New York to LA to London to Tokyo, or it might be back and forth from the coffee shop to the office, or from the kitchen to the couch. Any which way, you’re mobile and that’s the way to live. When you […]
It’s 2016 and we like our technology really small. Our phones fit in our pockets, our remotes are lighter than ever, and even our cars seem to be shrinking. So your new drone shouldn’t be an exception. This Axis VIDIUS Drone is 21% off right now and it’s so little, your biggest problem won’t be […]
You’ve heard the news: cyber security is the new and very scary frontier. Hackers are out there just waiting for you to relax for a second and let them in. But that’s not going to happen to you. With a lifetime premium subscription to ZenMate VPN, you’re completely protected from anyone out there who wants […]