On Twitter's engineering blog, a fascinating description of how Twitter uses a blend of machine intelligence and Mechanical Turk tasks to figure out, in real time, what is going on in the world:
Before we delve into the details, here's an overview of how the system works.
- First, we monitor for which search queries are currently popular.
Behind the scenes: we run a Storm topology that tracks statistics on search queries.
For example, the query [Big Bird] may suddenly see a spike in searches from the US.
- As soon as we discover a new popular search query, we send it to our human evaluators, who are asked a variety of questions about the query.
Behind the scenes: when the Storm topology detects that a query has reached sufficient popularity, it connects to a Thrift API that dispatches the query to Amazon's Mechanical Turk service, and then polls Mechanical Turk for a response.
For example: as soon as we notice "Big Bird" spiking, we may ask judges on Mechanical Turk to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps us serve relevant Tweets and ads.
- Finally, after a response from an evaluator is received, we push the information to our backend systems, so that the next time a user searches for a query, our machine learning models will make use of the additional information. For example, suppose our evaluators tell us that [Big Bird] is related to politics; the next time someone performs this search, we know to surface ads by @barackobama or @mittromney, not ads about Dora the Explorer.
Improving Twitter search with real-time human computation
Google is downranking websites that use pejorative, racist terms like n*gger, so the awful people of 4chan and /pol/ are replacing that word with “google.”
It’s been more than 20 years since the publication of Making Book, Teresa Nielsen Hayden’s collection of essays, mostly drawn from the pre-online days of fanzines and letters columns; this year, in honor of Teresa’s stint as Fan Guest of Honor at Midamericon II, the 74th World Science Fiction Convention, NESFA Press has published a second volume: Making Conversation, a collection of essays drawn from the online world on subjects as varied as moderation and trolling, cooking, hamster-rearing, fanfic, narcolepsy, the engineering marvels of the IBM Selectric, and more.
Someone — possibly the government of China — has launched a series of probing attacks on the internet’s most critical infrastructure, using carefully titrated doses of denial-of-service to precisely calibrate a tool for shutting down the whole net.
With this comprehensive course in App & Game Development for iOS and Android, you’ll be able to take full advantage of this career opportunity without committing to going back to school full time. You’ll learn how to build immersive, interactive games and apps from start to finish using Python, C#, Unity, and HTML—some of the most in-demand programming […]
CloudPress is a responsive WordPress theme builder that allows you to create a whole site in less than 30 minutes. CloudPress comes with tools like pre-built headers, content blocks, and footers—all you have to do is pick what you like, and drag and drop. With your subscription, you get access to 13 professionally designed WordPress themes, over 80 […]
If you own a dog, you’ve most likely heard of BarkBox – the monthly subscription box for dogs. What started as a simple idea to try out the subscription model on pet owners has since developed a cult following of dog lovers. If you haven’t given it a try yet, this one month free deal is the […]