On Twitter's engineering blog, a fascinating description of how Twitter uses a blend of machine intelligence and Mechanical Turk tasks to figure out, in real time, what is going on in the world:
Before we delve into the details, here's an overview of how the system works.
First, we monitor for which search queries are currently popular.
Behind the scenes: we run a Storm topology that tracks statistics on search queries.
For example, the query [Big Bird] may suddenly see a spike in searches from the US.
As soon as we discover a new popular search query, we send it to our human evaluators, who are asked a variety of questions about the query.
Behind the scenes: when the Storm topology detects that a query has reached sufficient popularity, it connects to a Thrift API that dispatches the query to Amazon's Mechanical Turk service, and then polls Mechanical Turk for a response.
For example: as soon as we notice "Big Bird" spiking, we may ask judges on Mechanical Turk to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps us serve relevant Tweets and ads.
Finally, after a response from an evaluator is received, we push the information to our backend systems, so that the next time a user searches for a query, our machine learning models will make use of the additional information. For example, suppose our evaluators tell us that [Big Bird] is related to politics; the next time someone performs this search, we know to surface ads by @barackobama or @mittromney, not ads about Dora the Explorer.
A long time ago, Veronica Belmont was featured in a blooper reel for her old TV show in which she clowned around with a Cthulhu t-shirt, wiggling back and forth and saying “So lifelike.” A creepy Internet person turned the moment into a GIF that has followed her around ever since, so that other creepy […]
Last February, Lenovo shocked its security-conscious customers by pre-installing its own, self-signed root certificates on the machines it sold. These certificates, provided by a spyware advertising company called Superfish, made it possible for attackers create “secure” connections to undetectable fake versions of banking sites, corporate intranets, webmail providers, etc.
These knitted gloves are here to save the day (and your hands) with an ultra-comfy, double-layer that will allow you to stay warm and use your phone. Now you can take photos on the fly, text, Tinder, and more without letting freezing temperatures get in your way. Plus they work with all touchscreens, so no […]
Carrying this EDC card is like slinging around a handheld toolbox wherever you go. Its minimal design is small enough to fit in your wallet’s billfold, and it’s TSA-compliant so you’ll never leave it behind. It’s got hex wrenches, metric and imperial rulers, flathead and Phillip’s screwdrivers, and a bottle opener so that you’re ready […]