Raffi Krikorian is a smart engineer and technical manager; I've know him for decades, but didn't realize that he was currently Vice President for Platform Engineering at Twitter. This interview with him at InfoQ gives a really fascinating flavor for how Twitter's reliability engineering is baked in at the systems level:
So Decider is one of our runtime configuration mechanisms at Twitter. What I mean by that is that we can turn off features and software in Twitter without doing a deploy. So every single service at Twitter is constantly looking to the Decider system as to what are the current runtime values of Twitter right now. How that practically maps is I could say the discover homepage, for example, has a Decider value that wraps it, and that Decider value tells discover whether it's on or off right now.
So I can deploy discover into Twitter and have it deployed in the state that Decider says it should be off. So this point we don't get an inconsistent state. The discover, for example, or any feature at Twitter runs across many machines. It doesn't run on one machine, so you don't want to get in the inconsistent state where some of the machines have the feature and some of them don't. So we can deploy it off using Decider and then when is on all the machines that we want it to be on, we can turn it on atomically across the data center by flipping a Decider switch.
This also gives us the ability to do a percentage-based control. So I can say actually now that it's on all of the machines, I only want 50% of users to get it. I can actually make that decision as opposed to it being a side effect of the way that things are being deployed in Twitter. So this allows us to really have a runtime control over Twitter without having to push code. Pushing code is actually a dangerous thing, like the highest correlation to failure in a system like ours, not just Twitter but any big system, is software development error. So this way we can actually deploy software in a relatively safe way because it's off. Turn it on really slowly, purposely, make sure it's good and then ramp it up as fast as I want.
Interview with Raffi Krikorian on Twitter's Infrastructure
(via O'Reilly Radar)
Earlier this month, I gave the afternoon keynote at the Internet Archive’s Decentralized Web Summit, and my talk was about how the people who founded the web with the idea of having an open, decentralized system ended up building a system that is increasingly monopolized by a few companies — and how we can prevent the same things from happening next time.
Corrections Corporation of America (CCA) is one of the world’s largest private jailers; it runs prisons and immigration detention centers across the USA (and is diversifying into halfway houses, mental health center, and surveillance for poor neighborhoods). Mother Jones’s Shane Bauer went undercover at CCA’s Winn Prison in Louisiana, the state with the highest incarceration […]
Steven Levy is in characteristic excellent form in a long piece on Medium about the internal vogue for machine learning at Google; drawing on the contacts he made with In the Plex, his must-read 2012 biography of the company, Levy paints a picture of a company that’s being utterly remade around newly ascendant machine learning […]
Experienced shutterbugs with DSLR cameras have boatloads of lens options for capturing the moment. Unfortunately, smartphone photographers often get stuck with their one crummy lens, which means limited zoom and focus for their final image.Step up your smartphone’s photographic power with the Acesori 5-Piece Smartphone Camera Lens Kit, now just $9.99 in the Boing Boing Store.Magnetic rings easily […]
Some truths are universal. For one, your phone will always run out of power when you most need it. For another, the charging cords that come packaged with your Apple device will fray, split, and rip faster than Usain Bolt in a game of tag.Instead, pick up a charging cord that anyone would have a tough […]
Some people say magic tricks are nerdy and best left to your 12-year-old asthmatic cousin. But others see value in perfecting the slight of hand and showmanship associated with a perfectly executed routine. We’re firmly in the latter camp. And now, we’re giving you the ability to put a few parlor tricks up your sleeve with the Penguin […]