SpaceX mission control vs. NASA mission control (photo comparison)

bcsizemo

Or maybe it is because NASA has been almost 50 years longer than SpaceX. They didn’t exactly have terabyte store arrays back then.

Honestly I’m more curious to see the other side of the NASA desks…I wonder if there still are CRT’s under there.

Arclight

Nope. This photo’s actually about six years old, and everything looked pretty modern even back then: https://upload.wikimedia.org/wikipedia/commons/4/45/ISS_Flight_Control_Room_2006.jpg
- joshhaglund
  
  I wonder what mission critical technology those 3 hand-held IR control devices work on?
  - Arclight
    
    Well, one’s probably for what looks like a JVC combo VHS/DVD set sitting on the guy’s desk.
    
    Which I think is an absolutely hilarious thing to have had even six years ago. I mean… why the hell is that there? DVD I could understand – maybe you need to watch launch video or something. But VHS? Weird.
    - C. Froggy May
      
      Institutional memory? “Hey, we had a similar problem on a mission back in ’81; let’s check the tape.”
      
      Until last year NASA was still dealing with the Space Shuttle which had its first mission in the Reagan administration, and probes to the far planets are missions that can take decades. I’d imagine there’d be substantial reasons to keep data recorded on obsolete formats readily available.
      
      Come to think of it, that may explain why so much paper hard copies. (20 something new team member can get the original specs right out of the 3-ring binder without having to know BASIC, DOS or FORTRAN. )
    - Diogenes
      
      Just because you, as a casual observer, may not be able to understand the need for what you assume is a certain device, doesn’t mean that a need does not exist, or that it’s even the device you assume it to be.
      
      What’s “absolutely hilarious” is people assuming that they’re light years ahead of the crew that runs the planet’s only space station.
      
      Show us your space station, Arclight.
    - nachoproblem
      
      Redundancy. It’s not just for breakfast.
- daen
  
  Ah, but if you look to the right, you can see binders peeking out — they’re stowed in the front of each desk.
  
  Also, the absence of cool down-lit acrylic signs obviously renders the six-year-old Mission Control center hopelessly obsolete compared to today’s version.
  - awjt
    
    The binders are used to follow procedures while what’s displayed on the screen is the real-time stuff. I’ve been there. I thought, why not just have ANOTHER screen with the binder stuff on it? But anyways, I didn’t think another screen would have helped much, because there are so many screens already.
    
    Also, the shuttle control room looked a lot like the one you have a picture of. But the ISS control room looks a lot more like the SpaceX one, with less paper in binders and a more open concept.
    - ophmarketing
      
      Because when the entire computer system goes down and you’ve got seven guys in space depending on you, the binders will still work.
Robert Daggett

Binders don’t crash
- http://profiles.google.com/chudez Ted Bautista
  
  This. The same reason why pilots with “glass cockpits” still have printed manuals and emergency procedures and a handy penlight.
- benher
  
  Exactly. Why else would a Soviet-era “Buzzer” station with a freaking analog mic still be broadcasting over AM in 2012… not saying that NASA has any “dead man switches” in operation out there, but they still have probes etc. from the 60s out there that probably need the occasional checkup.
http://codeflow.org/ Florian Bösch

And little electrical lights that go *blink* and toggles that go *click*. It’s known you can’t run a respectable mission control whithout all those things that go *blink* and *click* and *whirr*. If the government sends inspectors, to see where all the funding vanishes, and they ask what you do, you go like “well we do this, which makes *click* and then we do a lot of that, which goes *whirr* and there’s lot of stuff that *blinks*, we’re very busy you see, any other questions?”
- Felton / Moderator
  
  Blinking and beeping and flashing…
http://profiles.google.com/carboncow robert feller

I agree…it’s just in their culture to have paper resources…I’m quite sure these guys can find specs on their servers quick too…but some of the old dogs can find it quicker in a book!

RadioSilence

I noticed that watching the live feed. There’s a printer for every two people in the NASA control room, they must spend half their time printing stuff out to need that many. Either that or they think walking to the other side of room to get your printout is a bigger waste of resources than simply buying a shit-ton of printers.

Christian Vogel

I have never worked even near anything related to space flight, but at our tiny particle phytics experiment – when I was still at university – the control-room monitors had a print button at almost every display, graph, table, … (things like particle count rates, cooling water temperature, vacuum pressure, beam intensities, … which could be interlinked or invluence each other)

Whenever something strange or noteworthy happened you printed it out, glued (parts of) plots in your lab-book and added observations. After your shift you could then analyze what had happened.

And that’s 1000 times faster than copy/paste/save as .png/import into word or powerpoint, add caption, draw arrow, write email. Especially when you are under stress for keeping the experiment running.

And that’s why I think they have that many printers at NASA: The engineer should have a *immediate* access to getting a hardcopy to make notes on.
http://mjfgates.myopenid.com/ mjfgates

Given that a printer costs, what, $2-300… yeah, making engineers walk across the room to stand in line for printouts IS a waste of resources, especially if there’s any chance at all that they’re doing something that Will Kill People If It Doesn’t Get Done Right Now.
- RadioSilence
  
  @Christian Vogel
  @mjfgates
  Yeah, that makes sense.

http://pulse.yahoo.com/_LTATG2IJEC2UGDALLY5JFMZDRY Jesus

Lest we forget that NASA is also managing or monitoring dozens of other currently active missions. SpaceX just the one. NASA is also working out of a legacy facility that probably outgrew its document storage capacity in the early 80s. SpaceX is in a shiny new facility built by/for modern tech specifications. But yes, the breadth and depth of NASA’s documented procedures and protocols for everything they do is dauntingly ginormous.

bzishi

I don’t think it is surprising. For operational control of a machine that could potentially vent humans into space, I would expect that some commonly used procedures would be near at hand.

This is not a tech library, nor is it anything close to that.

capnmarrrrk

No ironic t-shirts to be found anywhere.

Chris Peters

After years of using paper you tend to have stuff bookmarked, notes in the margins, etc which just isn’t transferable to digital formats. Plus I know there are days at my work where they are rebooting computer systems, or the server is just being slow where you really miss an old paper manual that you could just flip open instead of clicking on multiple links to try and remember where can find some obscure data.

GeorgeSmith1

Cool article, Xeni! So you brought up a very important Question regarding binders on People’s desk–

Here’s the thing- NASA engineers were funded by the taxpayer.

But at SpaceX, if anything goes wrong, an insurance company will replace the rocket ship because it’s likely insured by a private company and no one can call their congressman to request an inquiry. The insurance company will likey just raise rates on things like home, car, or other products you and I are legally obligated to insure to cover a loss.

The proper insurance company will get in an airplane and then zoom across the country kind of like Edward Kelly’s entire team in Boston does and his replacement, Ted at Liberty Mutual wants to do.

They have an entire fleet of private aircraft, and with the office upgrades recently in an article at BOSTON GLOBE, probably woo’ed SpaceX for premium coverage relating to replacement.

It’s similar to the way Willis Insurance insured the Twin Towers and they were quickly replaced. (yawn).

Liberty Mutual, by the way, is well equipped to depreciate any type of vehicles, but
don’t give anyone at Liberty Mutual the Keyes to the spaceship, they may accidentally mail them to a different person on accident.

This type of thing– mailing someone else’s keys, social security, and title to others seems
To happen often. I suggest getting title insurance for the vehicle through a different company.

But that’s it– Liberty Mutual’s own aircraft is likely insured by another insurance company, who doesn’t depreciate vehicles. Likely its The Hartford. I suggest SpaceX use them as well. You can find out more about them at CostCo and you’ll save money with an Costco Executive Membership. It’s reasonable service and payments. Plus they call you back.

As for the binders, well, the binders contain all the necessary operating procedures for the computer. If you look carefully, one of those binders, which I recognize is an Assembly Ptogramming Language Manual. ASM was real fast back in the day, but required a technical team, somewhat complicated.

http://thisisonlya.blogspot.com robcat2075

It’s very unlikely that the first launching of an experimental rocket has been insured by a conventional insurer.
- http://boingboing.net/ The Life Of Bryan
  
  At least not with a reasonable deductible.
mcscruffington

What… the hell … are you talking about? Costco, Liberty Mutual, Boston Globe, Hartford… is this a new trend in corporate-sponsored commenting?
- http://vertigo25.tumblr.com/ vertigo25
  
  I’m surprised there’s not link to timecube…
- GeorgeSmith1
  
  No. I actually have rocket scientists in the family and also, I am making a comedy script about an office filled with claims agents called “Rushmore Insurance”.
  
  The kicker was The article in The Globe where all the VPs at liberty mutual each have their own jet airplane to help customers with their claims and also grease wheels to prevent regulatory involvement.
  
  In this tv show pilot I am writing, based on twitter comments of how people get regularly screwed by insurance companies, will come together.
  
  Basically the parody script is an office, like The Office, but claims are handled either on a jet or a rocketship insured by the insurance company. Parts happen in a strip club next door to the Autobody place where everyone is recommended to total their vehicle out.
- niktemadur
  
  Is this a new trend in corporate-sponsored commenting?
  
  Either that or some sort of weird OCD post with a dash of Asperger’s, I was half expecting a comment on the brand of binders going on to extoll the virtues and byzantine internal procedures of Office Depot in four paragraphs. Or something.
  
  But seriously, I thought it was a pretty funny post delivered with a big poker face.

http://www.facebook.com/profile.php?id=711162691 Scott Hiland

As a gummint employee, I can tell you that it is probably less about the bureaucracy than it is about the standard media.

The SpaceX folks probably had almost as many procedures, but they are documented online in some (either in-house or canned) content management system and the NASA folks would have had to pay for a big ol’ contract to get a CMS. Paper’s easier and cheaper than an extended procurement.

http://littleredtent.net/LRT Edie Howe

Keep in mind that NASA has to answer to taxpayers; They are also held to a much higher standard than a private company.

http://twitter.com/OhMeadhbh Meadhbh Octopodidae

me? personally? i think i would prefer an interface with physical switches, LED lamps and the occasional piezoelectric chirp. #JustSayin

Robert Dobbs

I’m sure it’s purely aesthetic.

sean duffy

Pretty sure NASA is required to have a paper backup of all procedures in case something did happen to their system. If the computers crashed, NASA could still jump on the radio and tell the crew what to do. If the computers crashed at SpaceX (however unlikely that they would all go down), you’d be flying by the seat of your pants.

It’s one of the main reasons why a lot of pilots still fly with printed manuals

DewiMorgan

Yes, I couldn’t help wondering “What’s SpaceX’ procedure for when someone trips the breaker and all the computers go down and need rebooting?” – I’m sure they DO have a procedure. I imagine it’s right there, in the computers.

Nasa has had *plenty* of equipment malfunctions, and I imagine they are very careful to be sure that no equipment is relied on to be guaranteed working.

There is probably redundancy in SpaceX: those look like terminals with the computers in another room, securely stuck away. Possibly failover machines in an adjacent room. And likely multiple separate data centers, and they can probably hand over fairly quickly.

But the thing about redundancy is that it’s only as redundant as the point of highest convergence. Multiple redundant network connections all came into my server host, the cables converging through a single underground access duct as they entered the building. And someone with a back-hoe was doing some work, and suddenly the ISP was down. For days.

Another time, I worked for a large TelCo which had a fire in one of its data-carrying underground passages. Splicing all those cables took days, again (there’s a limit to the number of splicing engineers you can fit per foot, and when there are thousands of fibers to splice…).

All that said: not ALL that paper can be mission critical procedures for use in case of computer failure.
- http://glitch.tl/ Michael Smith
  
  There is also a strong philosophy in aerospace of having your HMI nodes (the ones on the desk) as isolated as possible from the back end. In other words you you would not do the whole thing with a remote desktop. Processing has to be local and the HMI has to be able to connect to a degraded mode backend processing system on its own.
http://vertigo25.tumblr.com/ vertigo25

It is exactly this. Sure… maybe that has to do with bureaucracy, but given the cost of the vehicles and payloads, it’s also probably a damn good idea (especially when there are humans up there).
http://twitter.com/fossilfuels Funk Daddy

This is what I was taught too, they have every possible tech spec on hardcopy right there, and super redundancies on communications. So long as communications can be analog, they can do it in the dark.
SomeGuyNamedMark

They also have lots of “in case of x check y” protocol docs all heavily indexed.

Ramone

To be fair, NASA is accountable to many more people than a private company.

Simo-Pekka Nordman

Or they have stuff printed out in case something would happen to the computers. I know I’d feel awfully silly if someone screamed in a panicy voice asking what to do and I’d have to answer “Gee, that sounds like a problem right there. I’d love to check the files, but the network went down and Steve from IT says we’ll have it back in an hour or so. So you just hang in there and… try falling slower?”

journey46

Another possibility is that the binders are a non digital procedure back up system for emergencies when only radio contact is possible.

Of course SpaceX is not sending human passengers yet but when they do such a back up will be essential.

Xof

And, I’ll just mention, a binder gives you another 11″ x 17″ of screen real estate at a cost below that of a monitor.

t3kna2007

+1 Insightful. Every time you open a binder, it’s an instant extra display surface.
- http://boingboing.net/ The Life Of Bryan
  
  Every time you open a binder, an LCD loses a pixel.
  
  Please, think of the LCDs.

Grahamers2002

Or maybe…just maybe… NASA has considered 1) what might need to happen should they lose power in their control center during an important moment, 2) what might need to happen should they have some of their monitors in their control room break for a variety of reason, 3) realized that even today, versions of important info that can have physical properties like bookmarks and tabs etc, and can be carried anywhere very quickly during an emergency may have its uses, 4) etc.. I know…I know… ”But we have iPads and UPS’s and other redundancies….etc.” When peoples’ lives depend on it, having a paper backup is NEVER a bad idea.

Or did you not even think of that?

James Holmes

But its… paper… bound in a binder, no less! How blasé!
- http://www.jimdraws.com Thorzdad
  
  Yet, how perfectly accessible!
- Robert
  
  Do you mean “how gauche”?
Xof

Don’t worry, the Invisible Hand will guide them safely to a landing.

darkjayson

In an emergency grabbing the correct binder and flipping to the correct page with some details you need is much quicker than trying to search an online database, also what happens if the system is down? The emergency will not wait till you get it back up and running.

Give it time and space x will have there own paper binders.

Also remember there is stuff up there that have been working long before any sort of online databases systems existed i bet a lot of the technical documents of stuff like Voyager other such equipment are stored in those boxes.

joshhaglund

In a disaster paper can be handy.

But SpaceX can walk to the other room and grab the engineers who designed the thing, and ask them what’s up. NASA would have to call various contractors.

I think it’s hard to fairly compare the two organizations.
- http://www.facebook.com/profile.php?id=1153537686 Pierce Nichols
  
  NASA has been around long enough that they might have to call a nursing home to speak to the original designer(s)… or hold a seance.
  - http://boingboing.net/ The Life Of Bryan
    
    See that skinny gray binder, third from the left? That’s got the procedure for both orbital and terrestrial séances. (And an amazing brownie recipe, if the rumors are to be believed.)
  - Diogenes
    
    Ya, old people is stoopid!
- Diogenes
  
  Unless they laid those engineers off, after their design work was done, in order to protect the bonuses of the executive officers.

http://www.linkedin.com/in/petersaal Peter Saal

Where there are suits, there’s paper.

Robert Dobbs

Paper won’t crash.

HowieFeltersnatch

This is correct. As a research scientist who has spent the last decade studying situational awareness, common operating pictures and joint operations centers, if there’s one thing I’ve learned it’s that there’s no easier ticket to ride the fail train than not having hard copies of information. The idea that you would want no hard copies in your command center is breathtaking in its naïveté.
- timquinn
  
  Alright then, first thing we need someone to make a run to the library, with a cart, maybe two guys. Who volunteers?
- daen
  
  I liked your comment, and your spelling of “naïveté.” Well done, sir.

andygates

Picture the scene: Power fails at Mission Control, and they can’t do something important. The mission objective is scrubbed – or worse, people get hurt. Now, who’s clamouring “didn’t they even have a freaking hardcopy in a freaking binder?”

http://shortround.net/ Stephen Gerstacker

All I could think about when seeing the SpaceX control room is that they were just having a LAN party.

http://twitter.com/ProteinWrangler Carl Carruthers

is it really fair to compare a new MCC that controls a capsule against a ~17 year old Flight Control Room for a multinational space station?

Andrew McKay

I work as an engineer in a small agile engineering company and have contracted and partnered with a large engineering firm that shall remain nameless but was started by Thomas Edison. We see similar disparity in our procedures and policies just like this picture demonstrates. It’s not a government thing, it a large organization thing.

They can’t stop on a dime and change direction. They have documented procedures and policies (Programs like ISO and Six-Sigma) and any changes to those procedures and policies require lots of meetings and documentation as to what the new procedures are. If NASA has a procedure saying that all documentation for systems has to be available on paper to mission control the it’s probably not just a simple case of deciding next mission they will have it on computer. That decision would have to go through planning, risk analysis, and multiple levels of committee and administration reviews.

This is part of the reason Space-X and the company I work for get contracted by large organizations. We’re smaller, more agile and have less aversion to risk. Though there is a layer in between as a check and balance between the organizations. At some point the engineering effort has to satisfy all the checks and balances of the larger organization. It’s just they can turn a blind-eye to what happens between contract award and delivery of final product as long as final product meets all the requirements of the contract.

Diogenes

“We’re smaller, more agile and have less aversion to risk.”

Wow, that launch felt so cool! It was cheap, ahead of schedule, and (BOOM)….uh, did you hear that?
- blurgh
  
  As opposed to expensive, behind schedule and BOOM?
  - Diogenes
    
    I’ll bet my life on NASA’s safty record over anyone else, given where they’ve gone, and how many passengers they’ve brought back safely.
- Andrew McKay
  
  There is some truth to that. the ultimate goal is that small and large engineering firms are working together to reach a middle ground of what is an appropriate amount of process.
  
  The fact is large organizations get themselves locked into an attitude of “but that’s the way we’ve always done things” when sometimes there is actually no benefit to doing them.
  
  Small organizations a lot of the time don’t perceive the risks that the large ones do and pick up necessary processes to cover their butts.
  
  Definitely not being critical of NASA or other large organizations. They are that way out of necessity. However they do benefit from contracting smaller agile companies.
  
  I think that’s what Space-X and NASA are doing together today. Trust me, NASA will be putting them through the ringer over the next couple of years before they get “man-rated”. However they did develop a rocket and get to launch a lot faster than NASA ever would.

rockguitardude

If you have all of the procedures stored locally on each computer, the odds of being unable access the data is negligible. Say they had 50 people with computers there. To not be able to access this critical data, it would require the simultaneous failure of 50 computers, and likely a central backup UPS and standby generator.

Space X knows what they are doing.

Diogenes

Virus.
- rockguitardude
  
  It’s safe to assume that this system wouldn’t be connected to any outside network, therefore, threat of a virus is essentially zero. I’m sure operating procedures would prohibit use of any thumb drives too.
  
  Physical disconnection from the internet is pretty fool proof. With the nature of my work, I’ve seen many secure areas. Where security is of the utmost concern, you just physically disconnect from the outside network.
  - nox
    
    Yes, a disconnected network is very secure, but as you said it still has vulnerabilities that need to be covered by operating procedures.
    
    Nothing is perfect. There are so many possible vectors. It may be a very small chance, but the cost of prudence is negligible.
  - glenn gass
    
    My job one month required me to walk around to all the computers in a certain area and hot glue the front panel usb ports and firewire ports in the rack. And all the ports on desktops.
    
    It made backing up portable drive, using usb mouse/keyboards and pretty much anything the ENG head hadn’t thought up very difficult.
  - Petzl
    
    USB Virus.
  - Diogenes
    
    It’s not safe to assume anything of the sort. All it takes is one file from a USB key, a CD, or a DVD and you’re all infected.
    - rockguitardude
      
      It’s safe to assume these people are professional. These people aren’t going to be clods redditing and watching youtube videos while they’re preparing for launch.
    - Diogenes
      
      Your faith in groups is stronger than mine.
Robert

You don’t know how often standby generators and UPSs fail when you need them, do you?
- rockguitardude
  
  Do you? Any standby power system can be designed with sufficient redundancy to make the chance of failure negligible. Paralleled generator sets, cascaded transfer switches, etc. The only question is, how much money are you willing to spend? My bet is that all the systems in the building are so mission critical that no expense is spared when it comes to redundancy. I’d expect the system to be redundant to the point where if there was a failure, it wouldn’t matter if you had a manual or not (i.e. the building burns to the ground, tsunami, etc.)
  - Diogenes
    
    If you see the value of sufficient redundancy for the computers, why are you unable to see the same value in the redundancy of printed manuals? It’s paper, a cheap handy renewable resource that you can read with 1 candlepower if necessary.
    - rockguitardude
      
      I understand your point, but the culture of NASA is likely much different from Space X. NASA sprouted up at a time where you would have tons of printed manuals. People who worked there likely are used to having them around and due to that familiarity would know where to look for something in a pinch.
      
      On the other hand, you have the Space X which is relatively new. There may not even be printed manuals in existence. Even if there was, they are likely not familiar with all the ins and outs of them. If you have an instantly searchable file vs. a manual you’ve never used, which is faster?
      
      Every time you fix a computer, do you whip out a manual or do you Google the problem? (Or in Space X’s case, their technical documentation).
    - Diogenes
      
      We ran out the reply string, so I’ll answer here:
      
      My point is that if your computer is down, you don’t have “an instantly searchable file”; you have a boot screen, if you’re lucky. In the meantime, the humans in freefall would like you to get on the radio and offer some help.
      
      How do you Google the problem when your computer is broken or your network has crashed? If there are no printed manuals in existence, it’s because you didn’t print them, right?
http://glitch.tl/ Michael Smith

If your screens are full of data, your binder is like an extra free screen, right there on your desk.
TheMadLibrarian

I did notice, during the launch, that almost every SpaceX person in their control center had a laptop open beside them, running on battery. I wonder if that’s their equivalent of a binder?

Halloween_Jack

Wait until SpaceX has their own Apollo 13 type incident, and someone’s looking for a crucial bit of info that they scribbled on a sticky note and stuck to their monitor but now it’s gone because someone in facilities management told the cleaning crew to throw away Post-Its because the last time they had a photographer in they bitched about all the Post-Its because they looked tacky. Hopefully, no one has to die as a result of this, but…

irksome

I find the abundance of hard-copy back-up reassuring and its lack, oddly naive.

Diogenes

Ya, but what if the paper manuals get a virus?
- Petzl
  
  Fungus.
  - Diogenes
    
    True, but I can still read a mildewed manual.

stephenl123

Paper backup is good, it’s cheap, and it is not getting in the way. NASA also has some laptop computers in the room. I suspect that may be for additional redundancy.

glenn gass

More likely to run software that can’t be run on Official Hardware. You know like windows xp.

anansi133

The difference is that the NASA room must also handle human rated missions, and Space-X need not.

glenn gass

yet don’t forget the yet.

http://www.facebook.com/peter.r.rasmussen Peter Ravn Rasmussen

Not only “simple-minded”, but a cheap shot. These people are professionals doing their job — which involves getting people and equipment safely through a process that is still pretty damn risky.

Turning that dedication and methodical attention to detail into an opportunity for a dig at “government bureaucracy” is cheap and mean-spirited on your part.

And when you’re in LEO, you can’t afford to be sloppy, because mistakes are potentially dangerous, expensive and time-consuming.

When Dragon was captured by the Canadarm2 grapple, and preparations were being made to move it into berthing position, Flight Engineer Pettit noticed that the numbers on his display were different from the numbers indicated in his official manual. For 5-10 mnutes, the entire procedure was paused, while everything was checked, and Pettit was instructed to reboot his display to make sure the numbers were correct. Everything done with caution — an ounce of prevention being worth far more than a pound of cure, in space.

Again, these guys are pros, and deserve better than to be hitched to the wagon of your pet peeve about bureaucrats.

deviceofmind

Yeah, I agree, a cheap shot. But when I wrote “simple minded” to Xeni I was referring to my own explanation regarding bureaucratic tendency. -msw
- Diogenes
  
  You might want to ask Xeni to update the post to indicate your unintended error. Shame to make people dig down into the comments to see it wasn’t just some callow snark.
  - deviceofmind
    
    Yeah, I guess I could do that but I wonder if the perceived snark is what fueled so much commenting. In that sense, maybe it was worth it?
    - Diogenes
      
      Yes, if you value comment-count over communication then it was worth it. If you prefer respect and credibility you should clarify. But that’s up to you.

http://twitter.com/Carson__17 Carson 

When/If your computer screen goes down you’re going to thank god that you have a binder sitting there for back up.

http://twitter.com/mundoplano Ethan Soutar-Rau

The Front side of similar NASA desks (6 years ago)

http://en.wikipedia.org/wiki/File:ISS_Flight_Control_Room_2006.jpg

The NASA room is purpose built for the job with the accumulated wisdom of more than half a century of space flight; the SpaceX one is put together from off the shelf furniture, yet it’s also inspired by that same wisdom. There is no need for SpaceX or NASA mission controllers to be in those rooms from a technological point of view. They are there because it’s a useful social space.

Look at the way people are working:

http://www.evernote.com/shard/s166/sh/b5e491fa-dece-4aa4-b9ce-34fcefc2df62/48919f49a84abe75b736a0160ea913ca

Are the scenes really that much different?

michaelrbn

And besides, it’s “their” mission control, not “there”.

http://profiles.google.com/clawmaster Dave Mahoney

If it’s not there, then where is it?

traalfaz

Paper does not crash in the middle of a critical procedure. It’s probably a backup.

http://twitter.com/mundoplano Ethan Soutar-Rau

The first time grammar brings down a spaceship their is going to be hell to pay.

atarifan2600

http://en.wikipedia.org/wiki/Mariner_1
- Bob_Dobbs
  
  Not to mention contractors failing to realize a mass number for fuel is in Pounds Vs Kilograms on a Billion dollar didnt make it to Mars probe
DewiMorgan

Hell toupee, you say?

http://twitter.com/VagabondAstro Robert Little

Raise you a North Korean mission control center…
http://cdn.theatlantic.com/static/infocus/nkorea041712/s_n17_RTR30KT4.jpg

penguinchris

That’s a great photo… looks like something out of a 70′s sci-fi dystopia film.
- Bob_Dobbs
  
  Sanyo product placement win/fail?

http://www.facebook.com/people/Doug-Lucchetti/1132787135 Doug Lucchetti

Not to worry, given enough time and presumably enough success, SpaceX too will one day be clogged with the overburden of itsown experience and the pressure from management above….and some other new tech crew attempting to be the first to terraform a newly discovered planet will be snickering at the dinosaur SpaceX will have become by then, while we see an image of them as they stare at nothing we can see since they’ll likely all have optical eye implants and they’ll be at home wearing what passes for pajamas, eating cereal or some new snack food. Time marches on.

http://twitter.com/RickeeGaray Rickee Garay

Fair enough. NASA is a bureaucracy indeed. Been around since the 50′s has it not?

That being said, the Space X command center also resembles a fly by night – day trading office in New Jersey that could fold up shop and disappear in less than 12 hours if the Fuzz shows up. Jus sayin.

ab

HAHA look at the old fools with their paper n’ stuff!! Don’t they have an app for that? LOLZ BRB.

Punchcard

Ah, BoingBoing, where we can extol the virtues of artisinal hand-set editions of Pride and Prejudice one day, and the deride NASA for having *gasp* binders the next. I have a couple of reference books that I have both as ebooks and dead tree editions. The electronic version is great due to searchability, but by god if I am familiar with a reference book I can whip out the hard copy and flip/scan to the right page in no time. I can then pass it to someone if I need to. All faster than I do it with the electronic edition.

The newest things are not always the best things.

daen

OK, let’s see …

SpaceX: one rocket, one capsule, no humans, limited duration mission.

NASA: one 1,000,000lb space station with, at any given time, between three and six (soon to be seven) souls aboard, some of whom occasionally do EVAs, with ships docking and undocking all the time, mission constantly in operation.

We’re not comparing like-for-like here. I’m not surprised there’s a bit of paperwork at Mission Control.

None of which takes anything away from SpaceX — congratulations to them!

http://www.facebook.com/fred.vonblume Fred von Blume

I’m quite sure the difference in photos is not that significant but the criticism of Space X has been rather large and constant and generally the charge against much of all the criticism of ”outsourcing.” Essentially the charge is that government regulation and control has very little value and, by operating without it, Space X can vastly reduce costs without having these unnecessary expenses. This has avoided the overhead charge backs, workplace standards, accounting requirements, etc. required of others because Space X was a contract “launch provider.” Now that they are a vertically integrated direct competitor the competitors are complaining loudly and bitterly. This is risky business and it’s all the marbles. Vertical integration requires flexible demand because those costs are internal. It’s not known, nor knowable, how flexible demand is. Space X has had a string of failures. Should this fail, this will be another high tech industry America can/will no longer compete in.

http://www.facebook.com/profile.php?id=1113420524 Rebecca Cutri-Kohart

I worked in NASA Mission Control for the space shuttle. On my console, you would indeed see paper. However, everything I had on paper – mainly procedures, and reference notes, etc (I kept a binder of “goodies” of things I wanted to reference immediately in an emergency) were available electronically on the computer sitting next to me if I chose to operate that way – it would be a fallacy to assume we were weighed down with “TPS reports” just because you saw paper. Some things were just easier with paper in front of me – say a quick checkist. Some things weren’t – say an 1000 page reference document that I wanted to be searchable – and for those documents I used the computer (or later in my career, my kindle as well). My husband flies an airplane in a high tech cockpit, and he does a lot of things electronically, but you will still see paper checklists, for good reason, in aircraft as well – mainly emergency checklists … Ironically about this post, is occasionally our public affairs folks would bug us about getting the paper off our consoles when we were going to be video’ed or photog’ed because it didn’t seem “high tech” enough, but luckily doing our job expediently usually won out over these edicts.

That said I am incredibly awed, and impressed with SpaceX (and its competitors) and what they are doing to make space travel more accessible.

http://pulse.yahoo.com/_UWDFQHLRCTYZIOWFK3UZK223WI Rusty Shackleford

I love you space lady. I hope you’re at mission control for a next gen shuttle someday.
- rockguitardude
  
  +1 to “I love you space lady.”
Kaleberg

So the difference is that the Space X public affairs people got everyone to stash their binders for this photo.

http://boingboing.net/ The Life Of Bryan

The paperless mission control room will be preceded by the paperless office and the paperless bathroom.

LikesTurtles

Sounds like SpaceX installed Japanese washlet seats on every chair in mission control.

http://www.jimdraws.com Thorzdad

If it weren’t for those “bureaucrats” at NASA with their binders full of paper, SpaceX would still be sitting in their cubicles, scratching their heads trying to figure out this rocket science stuff.

timquinn

Simple, cargo or people. You wouldn’t ship your new born by ups just because she fit in the box, no? We handle these two things differently.

http://onthewingsblog.com/ Matt Saler

Seems like more people need to read Craig Nelson’s “Rocket Men” or something similar to get an idea of just how complicated NASA operations are and how necessary redundancies like possible paper printouts of procedures might be. Not everything is needless waste in government.

vne147

Books don’t crash and stop working I’m time critical situations. NASA might also have some books on grammar in “there” Mission Control Center.

gedsudski

I for one would prefer the well seasoned, oiled, and backed up NASA mission control team backing me up if the asteroids ever hit the fan. Sleek, clean design has it’s place… but not in a mission control room!

vne147

I saw someone post that they wondered what the other side of those consoles looked like. Here’s a picture I took during my 6 years on the other side of those consoles.

http://db.tt/KIAhfajh

Adela Doiron

Geeky squeal of delight. It’s like a data buffet
http://pulse.yahoo.com/_UWDFQHLRCTYZIOWFK3UZK223WI Rusty Shackleford

Love for all NASA people!

Diogenes

”But this seems simple minded to me.”

Then again, NASA figured out how to commute to the Moon while some of us were busy soiling our diapers, so maybe we should reserve judgment for a few more missions.

Personally, I don’t think it’s a bad idea to have procedures you can read with a flashlight.

deviceofmind

Actually, when I wrote “simple minded” to Xeni I was referring to my own explanation – not the use of binders or paper. Guess that got lost in the translation. – Michael Smith-Welch

Manny

When I go into space, I want my controller to have paper procedures, a graphing calculator, a slide rule, paper & pencil, and–just because you never know–a world atlas and a CRC handbook. I don’t want to die in orbit because someone trips over a cable at the wrong moment.

Hosidax Hosidax

Say it with me: IKEA

http://twitter.com/SOS_Sam Sam Wise

Another thing missing in this discussion — SpaceX’s mission control is just for the Dragon, a single spacecraft built by a single company. NASA’s (actually, JSC’s) mission control is for the ISS — a *far* more complicated beast, built in a bunch of pieces by a number of companies in multiple countries.

The ISS has also been in orbit for years (the first bits of it were launched in 1998); Dragon just a few days — so ISS is more likely to have something break and need humans to fix it. All those binders are full of detailed documentation on all the ins and outs of the ISS hardware and software, complete histories of their maintenance, etc.

Aaron Frith

I currently work in the Mission Control Center at Johnson Space Center (Ground Controller – Call sign “ISS GC”). There are a lot of interesting comments, and I can answer a lot of questions about my job and working with ALL of those binders (hot topic I guess).

Those binders are indeed our printed Flight Rules and other procedures. My console has a lot of drawings/schematics and at times it is easier to follow a procedure on paper than on a monitor. We have an extensive database online and we also back our procedures up on a thumb drive. There, I took all of the mystery out, I think.#StationGC

deviceofmind

When I sent this question to Xeni (and by coincidence she was with Miles O’Brien at a space conference in DC) this was ultimately the kind of answer I was looking for. Thanks! – msw

aeon

A slightly different viewpoint, from someone who doesn’t work in spaceflight or use office space very often. I hate open-plan office layouts with a passion and can’t generally work in them — a not uncommon feature of many geeky, introverted individuals.

When I see NASA’s ground control layout I see office space that I could bear to work in — it gives an element of psychological security by surrounding each workspace with ‘walls’, while still allowing the movement and easy voice communication between individuals required by the job. SpaceX control room is way too open plan and exposed — it would freak me out. I suspect NASA is likely to have a high proportion of geeky introverts in its ground staff and may well have gone to the trouble to design a workspace that allows us to function effectively.

penguinchris

When I was watching the various live coverage of Dragon I was disappointed in what the SpaceX control room looks like. It’s all off-the-shelf office furniture, and their “big board” is just a projector screen. Take the computer screens out and you can imagine the room being repurposed for lectures down at the community college.

But… that’s what tech startups always look like and should look like because they shouldn’t be spending their money on nicer stuff until they’ve proven themselves.

And while everyone’s attacked the premise here with good arguments – there are many good reasons they want things on paper in binders – nobody’s asked if maybe SpaceX does have things on paper.

How do we know they don’t have everything they might need on paper – probably in binders – ready to be rolled in on library carts in an adjacent room?

This isn’t the greatest photo but it’s the best I could find quickly – a shot from an adjecent room to the SpaceX control center that gives a bit of context, the control center is part of a large structure with lots of other rooms nearby where binders could easily be stored.

They just built their control center cheaply and decided on clutter-free, very simple desks instead of custom-built desks with room for binders, because their workflow is designed to be almost completely digital. Paper is a backup so it’s stored elsewhere (but nearby).

This is all speculation, but I trust that they know what they’re doing.

Diogenes

“How do we know they don’t have everything they might need on paper – probably in binders – ready to be rolled in on library carts in an adjacent room?”

If they do, they’re going to knock over those cheesy conference tables and dump everyone’s laptops and monitors on the floor.

kartwaffles

NASA here. Hardcopies are still *infinitely* preferable vs. SharePoint or (FSM forbid) Windchill dox. I’m a systems engineer, and while we may do some things well, document and configuration management is not one of them, especially when you’re cooperating/advising stuff that connects to private industry like a SpaceX flight.

Oh, and by the way, good job SpaceX. All politics aside, y’all kicked ass with this flight.

Gary O’Brien

Keep in mind, too, that those binders may very well be highly personalized down to the work station and job. When one flight control team goes off duty in the transition mission rules are passed from one controller to another with notes and explanations about events, changes, anomalies, what have you. If you go back through NASA history those binders with personal notes have saved some butts.

Gary O’Brien

That being said I don’t want to upset Elon in anyway. Dude built a rocket and a space craft and made it work. I don’t want his next step to be a death ray.

Tom Morgan

Because SpaceX has to manage one piece of hardware, and one mission profile, and the Nasa guys are responsible for a whole range of legacy hardwares, some of which are based on venerable TKS modules (I’m looking at you Zveda), interoperability aspects and the broad mission profiles associated with the ISS?

But, nah. It’s gummint waste, of course.

rtresco

Just to infuse some mellodrama: Many comments have touched on the need for binders in case something goes wrong, which I agree with, but when I see a picture of a room full of binders, I don’t think “precaution”, I think “follow-up”. So, at my highly unimportant job, if something goes wrong in the system – deadlines slip and money is lost. No big whoop - everyone lives. But still, the next day it begins – documenting what went wrong. Root cause analysis. What am I going to do differently going forward, how can this never happen – ever again?! Reports! Reports! Reports! And yet at NASA, when something goes wrong - it is a national tragedy. I can’t help but think that if I make a typo at work and it results in who knows how much new process, that after 50 yrs and some national tragedies, that binders are an inevitable result. Lord knows I don’t wish binders on anyone, but SpaceX just hasn’t had someone ask for corrective actions to be documented yet.

http://www.facebook.com/people/Delton-Slusher/100001664532262 Delton Slusher

The NASA center has a lot of legacy equipment built back when 50MB was a big hard drive. They are bound by law to follow contracting and procurement procedures that make it so difficult to buy equipment that whatever they buy is obsolete by the time they put it in operation. That’s why the Space Shuttles launched with laptop computers on board – because by the time they started flying, the built in onboard computers were *way* outdated. And the NASA consoles aren’t running Windows or Linux – they’re running specialized purpose-built software. They can’t fiddle with email or Google Earth during working hours. And think of this – that equipment is still working after more than 40 years. How long is the MTBF of the gear in the Space X center?

And as for the manuals spread out on top of the NASA consoles – great. What good does it do you to have your procedure guides available only on the laptop that has just crashed? Physical manuals are immune to power hiccups. And they damn well SHOULD be doing everything exactly “by the book.” That’s the only way to be sure. It’s vital when human lives are on the line.

http://www.facebook.com/profile.php?id=1113420524 Rebecca Cutri-Kohart

You’d be surprised, about Google Earth, it was a Mission Controller who built this tool, for instance: http://www.nasa.gov/mission_pages/shuttle/shuttlemissions/shuttle_google_earth.html.

(And look closely – those are Linux and Windows machines in those pictures – no one is inventing new operating systems there.. :) )

LikesTurtles

Microsoft Research did a study a few years ago on why people still use paper so much in a world of design software, word processors, cheap multiple monitor set ups, etc. The number of reasons they came up with was astonishing but most of them were legit reasons that weren’t covered with current technology. The bottom line was that though some changes could be made to reduce the need for paper, it isn’t going to disappear any time soon, especially for ad hoc type of uses.

Bob_Dobbs

Some other things to consider most of the monitors on the consuls arent there to do look ups of procedure etc theyre there to monitor real time data if you want to review 3 possible procedure paths on a task prior to the start of a task or compare multiple pages you cant retask all the monitors and leave yourself blind to what your supposed to be watching Books are good for prep and research while keeping an eye out for the red blinkie text.
Also there is the added bonus of revision history visual ques, you can look over at the person beside you and tell which version theyre on by the paper/tab/cover colour

http://profile.yahoo.com/5KMT2N5U62T67TO3NCULPY2MRQ Claire

Like a couple of the commentators, I also work at ISS Mission Control (taking care of life support and thermal). I can confirm that those binders are for critical items that we wouldn’t want to be without in case of a network failure, things like flight rules, procedures and detailed system references.

These are all things we have electronically. Speaking personally, I always use the electronic versions, but there are times when it’s nice to operate out of a paper copy to save monitor real estate and to jot down notes (not uncommon for special or one-off tasks).

There have been people on the ISS continuously for the last twelve years or so, and MCC-H has been on line 24 hours a day every day since then. There are many opportunities for computer/network problems. And indeed, the team has had to resort to paper references in the past.

We’ve been actively paring down the number of paper copies we carry over the last five years since now we have the experience to decide what’s critical and what’s just nice-to-have. We’re doing the same thing with the crew onboard. You can imagine how hard it is to print things up there (they have to do their own printer repair) and how expensive hard copies are to fly. But despite the fact that everything is available to them electronically, there are some things, like emergency response procedures, that they will never give up their paper copies of.

MPlainguet

I was at a SpaceX conference and I can tell you they have a great interest in doing things cheap. There’s lot of talk about how NASA needs to have higher standards because taxpayers are paying for it and there’s more accountability. SpaceX believes in having a “good enough” standard to save money despite higher risk of failure. Looking at these pictures and making assumptions reveals nothing.

http://profiles.google.com/keithdtyler Keith Tyler

Gee, maybe if the computer systems crash. Score 1 for NASA.

And say what you want, but NASA Mission Control looks tons cooler. That’s 2.

divmstr95

The bottom line is that some things, like procedures written on paper instead of a computer screen, are so tried and true that there is no need to change them.

What advantage did the Space-X control center have based on the visual? They look cooler.

What the comparison does is show your ignorance of NASA’s history and your understanding of the execution of a complex technical procedure.

Wonder how many of the folks in the Space-X picture are capable of pulling out a calculator and doing course-corrections on a pad of paper?

You might want to consider applying for a job at MSNBC. I hear they are good about playing select versions of 911 calls and showing partial photos of people to hide their true identity.

Diogenes

And another thing!
A binder can hold a large fold-out schematic or diagram. Multiple people can lean over it and brainstorm tracing paths and marking locations. Paper can be just plain handy.

http://www.facebook.com/profile.php?id=604461989 Mark Drummond

Colour me obvious. Hardcopy procedure manuals.

http://www.facebook.com/profile.php?id=8646282 Andrew Lawson

I don’t trust skinny chefs, mechanics with clean hands, tattoo artists without visible tattoos, AND engineers without binders on their desks.

http://www.facebook.com/people/Dave-Nullstein/100003753218370 Dave Nullstein

It’s not the binders we need to be worried about. It’s the neckties.
Every single male in the NASA mission control room is wearing one. Creeps me the fuck out.

deviceofmind

Great observation – definitely more symbolic. – msw

http://www.penguinsix.com/ penguinsix

Did anyone else notice that some of the Space X guys had laptops as well on the side. Guess they are tweeting while doing their job or something.

Plut0

While there is always a small change things go wrong, I don’t think there are many computer problems in mission control.

I think the binders provide instant ready documentation about the complete system(s) flying. When having one team, completely dedicated to building, engineering and flying on system, probably a digital version will do, however when having a dedicated mission control team, flying several different missions and systems, this information needs to be on hand.

A second thing you could wonder is which responsibilities where present where. There are several control teams involved in this. There are SpaceX launch control, SpaceX mission control and NASA mission control (and maybe also a NASA launch control)

peeatwork

“The only conclusion I can draw from this…” Really? There’s no other possible explanation for binders with paper in them?

deviceofmind

That’s why I said my explanation was simple-minded in my original e-mail to Xeni. What I was asking her and Miles O’brien in my e-mail was, “What’s in those things and why doesn’t SpaceX have ‘em?”

I got 150+ answers once Xeni put it up.

Ryan Lenethen

I’d say smart. Hell even when you go to do your stupid power point presentation, you take a hard copy just in case of technical failure.

Seems that makes sense for a multi-billion dollar space launch, could just be me. I mean sure you have backups for your backups for your computers. But do you really want zero recourse when something goes horribly wrong, and you somehow lose all your alternatives. At the very least if you can maintain communication, you could have someone going through a prearranged procedure that has been worked out in the event of issues.

nexon

Gosh… I guess no one has had to deal with Document Control? Let’s say you have an important binder: “Emergency Procedures”, and you’ve distributed 100 copies around the Mission Control Room. What do you do when the content changes? Print out 100 new copies and distribute them around the room… hoping that the Engineer will throw out the old one? From what I’ve read above, the Engineer will most likely keep the old copy since it has “post-it notes, margin notes, bookmarks,…”. The result being that the Engineer will be using obsolete or inaccurate procedures.

Or, you could try serializing the document, and tracking down and destroying all 100 copies, replacing them with the update. What happens when you can’t find all 100 copies? What if the document has been distributed beyond the Control Room? What’s the chance that the obsolete copies will re-appear at a critical moment, and lead to mission failure?

My favorite: update the document with a “change package”. Usually a shrink-wrapped set of updated document pages and instructions on how to apply the changes to the document (e.g. “replace pages 10-12 with pages 10-12.4″, etc). No one wants to waste time updating the document, so the change packages just accumulate at the back of the binder. It’s only when you actually need to use the binder that you realize that you’ve got a year’s worth of changes that haven’t been applied…

No, sorry: the binders must go. In the event that the Engineer needs to print out the document for some reason, the pages will be watermarked “Printed Document Uncontrolled”… meaning basically that it should be removed from the work area and destroyed.

Antinous / Moderator

Having worked in a hospital, I know the drill. Open Policy Manual Volume 5. Find Section 3, Subsection B, Sub-subsection xiiii. Remove pages 3-11. Replace with pages 3-17. Open Policy Manual Volume 7. Find Section 14…

And usually done monthly.

TangledThorns

I work in an ISP ops center and thankfully my office is more like Space X’s. I can’t believe how outdated NASA’s looks.

hadlockk

So, decompression in module X, and by the way, we’re at war with China, who’s just nuked Houston. Johnston Space Center mission control has survived, because it was designed to withstand exactly that sort of attack on the nearby major population center. But the internet is down, and they don’t have online access to their emergency procedures. Fortunately, they have a paper backup, which they can read with flashlights (there are no windows in mission control) and give instructions via backup radio to the station in orbit. Good thing they had a dead tree backup, they just saved the lives of some of the smartest scientists on the planet (and in orbit).

NASA: Failure is not an option.

Human spaceflight is serious business. Those are lives up there that can’t be easily rescued.

nexon

Putting the documentation “on-line” doesn’t necessarily mean “putting it on the internet”. If the Mission Control room is intact, and the computers are able to process incoming telemetry from the Station, then surely it could also serve up a few PDF’s with emergency procedures?

On the other hand: if all the Mission Control rooms and all the computers are down, you would have no status or telemetry from the Station: Mission Control would be blind. If all you have is the flashlight app on your iPhone and a month old (or older!) paper copy of the Emergency Procedures… then the Station is pretty much on its own anyway.

Wouldn’t it make more sense to put the Emergency Procedures on an SD drive and keep it on the Station? The Station crew would notify Mission Control that ‘Emergency Decompression Procedure’ had been activated, not the other way around.
- http://www.facebook.com/eugene.lavoe Eugene LaVoe
  
  Absolute worst case scenario, you’ve still got contact with the ISS (or spacecraft) via good, old-fashioned HF and VHF radio, wither voice or digital modes. You can talk to the ISS on a handheld nowadays.

teapot

Maybe NASA has all those binders to ensure they know the correct usage of there/their/they’re?

< /snark >

http://twitter.com/IamFused Adam Falshaw

They have a neat and tidy storage system that enables them to keep hard copies well organised, all within an arm’s reach of their main workstation, and the only thing you can draw from it is that the entire company is simple-minded and riddled with bureaucracy?

It blows my mind that people from this site’s community (one I usually consider fairly intelligent and reasonable) can compare stills of two working environments and deduct major flaws in the work ethic of one of the world’s most successful engineering teams.

deviceofmind

Actually, the simple-minded bit was a reference to my own explanation I posited in my original query to Xeni. I wouldn’t harsh on the community as a whole here. If anyone was unintelligent or unreasonable, it was me. From my perspective I learned a lot about how these large teams of engineers work from the Boing Boing community. Particularly the NASA folks who chimed in. I didn’t really see anyone in their comments discuss major flaws in how NASA operates. For me, I was just curious what Xeni and Miles O’brien thought about the presence of binders in one, and a lack of them in another. Poor wording/grammatical errors aside, I find it interesting that so many people felt compelled to write their view of the differences in the images.

mmm3point14

No one is looking at the binders. The binders are behind them for reference while the engineers are actively looking at computer monitors.

deviceofmind

In the image no one is looking at binders, but I did see via NASA TV during the docking that some of the mission specialists were taking down binders, jotting notes, etc. while the robotic arm reached out to grab Dragon. 99% of them are just sitting back there for reference, but some were being used.

SpaceX mission control vs. NASA mission control (photo comparison)

More at Boing Boing