Raffi Krikorian is a smart engineer and technical manager; I've know him for decades, but didn't realize that he was currently Vice President for Platform Engineering at Twitter. This interview with him at InfoQ gives a really fascinating flavor for how Twitter's reliability engineering is baked in at the systems level:
So Decider is one of our runtime configuration mechanisms at Twitter. What I mean by that is that we can turn off features and software in Twitter without doing a deploy. So every single service at Twitter is constantly looking to the Decider system as to what are the current runtime values of Twitter right now. How that practically maps is I could say the discover homepage, for example, has a Decider value that wraps it, and that Decider value tells discover whether it's on or off right now.
So I can deploy discover into Twitter and have it deployed in the state that Decider says it should be off. So this point we don't get an inconsistent state. The discover, for example, or any feature at Twitter runs across many machines. It doesn't run on one machine, so you don't want to get in the inconsistent state where some of the machines have the feature and some of them don't. So we can deploy it off using Decider and then when is on all the machines that we want it to be on, we can turn it on atomically across the data center by flipping a Decider switch.
This also gives us the ability to do a percentage-based control. So I can say actually now that it's on all of the machines, I only want 50% of users to get it. I can actually make that decision as opposed to it being a side effect of the way that things are being deployed in Twitter. So this allows us to really have a runtime control over Twitter without having to push code. Pushing code is actually a dangerous thing, like the highest correlation to failure in a system like ours, not just Twitter but any big system, is software development error. So this way we can actually deploy software in a relatively safe way because it's off. Turn it on really slowly, purposely, make sure it's good and then ramp it up as fast as I want.
Interview with Raffi Krikorian on Twitter's Infrastructure
(via O'Reilly Radar)
Uganda is so poor that few can afford medical care, giving it one of the lowest life-expectancies on the planet — this toxic combination made the country ripe for infiltration by Tiens, a Chinese Multi-Level-Marketing “nutritional supplements” cult whose members set up fake medical clinics that diagnose fake ailments and proscribe fake medicines, then rope […]
Yahoo’s sale to Verizon means that Yahoo’s sub-companies — Flickr, Tumblr and a host of others — are now divisions of a phone company, and as you might expect, being on the payroll of a notorious neutracidal maniac with a long history of sleazy, invasive, privacy-destroying, monopolistic, deceptive, anti-competitive, scumbag shakedowns has changed the public […]
What was last week posed as an indefinite leave of absence is now for good: Travis Kalanick, CEO of scandal-wracked rideshare company Uber, announced that he is leaving the company. “I love Uber more than anything in the world and at this difficult moment in my personal life I have accepted the investors request to […]
Although flagship smartphones are unlikely to adopt heavy-duty outer casing anytime soon, you can always prepare your device for the outdoors with a beefy case and and an external battery like this Nomad Tile Trackable PowerPack, available in the Boing Boing Store for $119.95.The Nomad Tile can fully recharge an iPhone 7 over three times […]
Even though credit cards now feature an EMV chip for securing transactions, they still have to include the magnetic strip for compatibility with older point of sale systems. Because of this, there’s no way for the chip’s new security capabilities to protect against card skimmers in the wild.How do you protect yourself from legacy-technology-induced fraud? […]
As the old saying goes, “You should sit in meditation for 30 minutes every day. Unless you are too busy, in which case you should meditate for an hour.” Since most of us have an endless list of things to do and people to see, carving out quiet time can feel impossible, especially when most […]