On Twitter's engineering blog, a fascinating description of how Twitter uses a blend of machine intelligence and Mechanical Turk tasks to figure out, in real time, what is going on in the world:
Before we delve into the details, here's an overview of how the system works.
First, we monitor for which search queries are currently popular.
Behind the scenes: we run a Storm topology that tracks statistics on search queries.
For example, the query [Big Bird] may suddenly see a spike in searches from the US.
As soon as we discover a new popular search query, we send it to our human evaluators, who are asked a variety of questions about the query.
Behind the scenes: when the Storm topology detects that a query has reached sufficient popularity, it connects to a Thrift API that dispatches the query to Amazon's Mechanical Turk service, and then polls Mechanical Turk for a response.
For example: as soon as we notice "Big Bird" spiking, we may ask judges on Mechanical Turk to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps us serve relevant Tweets and ads.
Finally, after a response from an evaluator is received, we push the information to our backend systems, so that the next time a user searches for a query, our machine learning models will make use of the additional information. For example, suppose our evaluators tell us that [Big Bird] is related to politics; the next time someone performs this search, we know to surface ads by @barackobama or @mittromney, not ads about Dora the Explorer.
Though the October polls that predicted a great showing for the Pirate Party in the Icelandic elections turned out to be wrong, that election did end with a deeply divided parliament that has been unable to find enough common ground upon which to form a new government.
The University of Toronto’s Citizen Lab (previously) continues its excellent work, this time with a deep investigative piece on a sneaky form of censorship in China’s popular We Chat service, where messages posted to group chats that contain words on a government blacklist are made invisible to other participants in the chat, while the original […]
Deji writes, “Gambia is a small country but this story is pretty crazy. The president, who is seeking his 6th term, is using Trump rhetoric surrounding the ‘rigging of elections.’ People are voting by using marbles. Meanwhile, opposition activists and journalists have been arrested — and the government STILL shut off the internet. It seems […]
Loot Crate is a totally different kind of subscription service that mails subscribers monthly boxes filled with curated geek, pop culture, and gamer paraphernalia. Its cult following awaits a box every month filled with everything from bobble heads to T-shirts to special edition collectibles. But nothing gets Loot Crate fans as excited as the limited […]
The ARMOR-X Mini Flexible Phone Tripod is a smartphone tripod that is designed with flexible legs to rest on virtually any type of surface. Other tripods have proved useless unless I conveniently have a flat surface in front of me, which is why this particular tripod was appealing enough to try out. The ARMOR-X is compact and easy […]
You don’t need to get an advanced degree and take out massive loans to become a coder. This bundle of 10 courses was designed to teach anyone to code at home for less than it costs to go out for dinner. I was particularly impressed with this new 2017 bundle because it includes courses on […]