On Twitter's engineering blog, a fascinating description of how Twitter uses a blend of machine intelligence and Mechanical Turk tasks to figure out, in real time, what is going on in the world:
Before we delve into the details, here's an overview of how the system works.
First, we monitor for which search queries are currently popular.
Behind the scenes: we run a Storm topology that tracks statistics on search queries.
For example, the query [Big Bird] may suddenly see a spike in searches from the US.
As soon as we discover a new popular search query, we send it to our human evaluators, who are asked a variety of questions about the query.
Behind the scenes: when the Storm topology detects that a query has reached sufficient popularity, it connects to a Thrift API that dispatches the query to Amazon's Mechanical Turk service, and then polls Mechanical Turk for a response.
For example: as soon as we notice "Big Bird" spiking, we may ask judges on Mechanical Turk to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps us serve relevant Tweets and ads.
Finally, after a response from an evaluator is received, we push the information to our backend systems, so that the next time a user searches for a query, our machine learning models will make use of the additional information. For example, suppose our evaluators tell us that [Big Bird] is related to politics; the next time someone performs this search, we know to surface ads by @barackobama or @mittromney, not ads about Dora the Explorer.
Doubtless you’ve laughed at the ideological war between the Judean People’s Front and the People’s Front of Judea. I laughed along with you: having grown up in politics, I know firsthand about the enmities that fester between groups that should be allies — groups whose differences can only be parsed after months of study, but who are seemingly more at odds with one another than their obvious political opponents on the “other side” of the debate.
In 2013, Lavabit — famous for being the privacy-oriented email service chosen by Edward Snowden to make contact with journalists while he was contracting for the NSA — shut down under mysterious, abrupt circumstances, leaving 410,000 users wondering what had just happened to their email addresses.
A new phishing attack hops from one Gmail account to the next by searching through compromised users’ previous emails for messages with attachments, then replies them from the compromised account, replacing the link to the attachment with a lookalike that sends you to a fake Google login page (they use some trickery to hide the […]
Looking to upgrade your weekend? Here are three randomly awesome products on my mind this week.#3 FRESHeBUDS Pro Magnetic Bluetooth EarbudsAs more and more phones and gadgets switch to Bluetooth-only compatibility, you’ll need to get Bluetooth headphones like the rest of us. I’ve been super impressed with these affordable magnetic headphones. Pull the magnetic earbuds apart to auto-connect […]
Traditional folding wallets are designed for paper bills—but these days, carrying cash is rarely a necessity. More often than not, I don’t carry cash at all. This Bogui Clik Wallet is the best answer I’ve found for avoiding the hassle of those tight-fitting credit card pockets.This attractive, minimalist wallet features a protective lip, so my cards don’t […]
Using my iPhone while it’s charging is always a hassle. With tucked-away outlets and the meager length of included lightning cables, comfortable scrolling while plugged in is annoying. These 10-Ft MFi-Certified Lightning Cables are super convenient and probably the best iPhone accessory purchase I’ve made.At over three times the length of normal cables, these reach anywhere you […]