On Twitter's engineering blog, a fascinating description of how Twitter uses a blend of machine intelligence and Mechanical Turk tasks to figure out, in real time, what is going on in the world:
Before we delve into the details, here's an overview of how the system works.
First, we monitor for which search queries are currently popular.
Behind the scenes: we run a Storm topology that tracks statistics on search queries.
For example, the query [Big Bird] may suddenly see a spike in searches from the US.
As soon as we discover a new popular search query, we send it to our human evaluators, who are asked a variety of questions about the query.
Behind the scenes: when the Storm topology detects that a query has reached sufficient popularity, it connects to a Thrift API that dispatches the query to Amazon's Mechanical Turk service, and then polls Mechanical Turk for a response.
For example: as soon as we notice "Big Bird" spiking, we may ask judges on Mechanical Turk to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps us serve relevant Tweets and ads.
Finally, after a response from an evaluator is received, we push the information to our backend systems, so that the next time a user searches for a query, our machine learning models will make use of the additional information. For example, suppose our evaluators tell us that [Big Bird] is related to politics; the next time someone performs this search, we know to surface ads by @barackobama or @mittromney, not ads about Dora the Explorer.
A study by the Department of Commerce’s National Telecommunications and Information Administration found that half of American Internet users are “deterred” from engaging in online transactions because of fears over privacy and security breaches.
FBI Director James Comey told reporters that “viral video effect” (which is his latest term for what used to be called the “Ferguson effect”) is responsible for increased violent crime in some US cities, in that police are scared to do their jobs because they might end up on Youtube in an unflattering video.
The World Wide Web Consortium (W3C) has made a sea-change: now, in addition to making open web standards that anyone can implement, they’re creating a video DRM standard designed to prevent people from implementing it unless they have permission from the big movie and TV companies, by invoking the notorious Digital Millennium Copyright Act and […]
Jared Sinclair developed the RSS reader app Unread, which made $10,000 in its first 24 hours on the iOS market. And we’ve all heard the story of Flappy Bird developer Dong Nguyen, whose creation was reportedly earning $50,000 a day at the height of its 2013 explosion. While those are rare examples, they’re also testament to the […]
If you or your company’s IT system are besieged by black hat cyber attacks, an ethical hacker might be all that stands between crippling damage and a company’s long-term prosperity. It’s no wonder that the market for IT security specialists is exploding. Certification is the key – so learn the tenets of ethical hacking and get […]
Your laptop and mobile devices are top of the line…so why are you trotting out that raggedy decades-old suitcase when you go somewhere? Time to up your travel game with a complete 5-piece Herschel Travel Luggage bundle…and we’ll even give it to you for free!Of course, you’ve got to win the Ultimate Herschel Travel Bundle […]