How Pixar almost lost Toy Story 2 to a bad backup

[Video Link] From Jamie & Adam Tested: "Pixar's Oren Jacob and Galyn Susman recount how the files for Toy Story 2 were almost lost due to an accidental Linux command and bad backup. It's probably a little over-dramatized from what actually happened, but is a good reminder to back up your important files!"

Mark Frauenfelder is the founder of Boing Boing and the editor-in-chief of MAKE and Cool Tools. Twitter: @frauenfelder. Come and hear Mark speak at the ALA conference in Chicago on July 1.

More at Boing Boing

Ants and Stars: Bruce Sterling and Jasmina Tesanovic visit the Sardinia Radio Telescope in Italy

The Snowden Principle

http://twitter.com/AntiBoredomTeam Dan Century

Now I have a good rational for fighting the no-telecommuting/no-thumb drive policy at work.
swankgd

No, it’s a good reminder to TEST your backups and TEST your restore procedure.
http://twitter.com/lampbane Kris N

The Pixar Studio Stories are some of the best reasons to get the Blu-ray editions of their films. Well, that and the films themselves…
chrisspurgeon

HERE is the big lesson everyone should learn from this… your backup strategy does not work unless you CAN ACTUALLY RESTORE YOUR STUFF. At the last three companies I’ve worked at we’ve learned the hard way that when it came time to restore vital lost information from a backup the damn restore didn’t work for one reason or another (backup files were actually blank, restore software not compatible with current system, backup hard drive corrupted).

PROVE that your backup system works by doing regular restore exercises. Don’t be complacent. Don’t find out the hard way that your restore strategy has a fatal flaw.
arcfinn

Kind of a shame they got it back… Opinions may vary.
http://twitter.com/incarnedine_v Dan Hibiki

who gives users root access any way?
- Guest
  
  No one said they trashed the OS, just that someone did an rm * in the directory containing the movie (I’m guessing the rendering models)
Nylund

I was using a server to do run some statistical models with a big data set (12 gigs or so?). At one point, I decided I should back up that data set, but after I started the process, I got called on to go do something else. I was unsure what would happen to the backup process on this communal server if I left and someone else showed up, so I canceled it before it was complete. I got some prompt asking me what to do with the files. I assumed it meant the files that I’d partially, but not completely, backed up. Knowing I was going to be returning soon, I figured I’d just start the backup from scratch again, so I selected “delete” only to then realize it meant the ORIGINAL files, not the partial backup I’d interrupted. It did a true delete (not just trash/recycle bin delete like in Windows or OSX). I was able to recover most of it, which, in tandem with an out of date copy on my personal laptop, I was ale to get everything kosher again (after many hours of work).

Or, in other words, I was a giant idiot. I felt really dumb, especially since the whole thing started because I was trying to be responsible about making backups. I just really messed up in the process.
- http://halfbakedmaker.org Robert Baruch
  
  Sounds to me more like shitty UI design. Why would anyone ever want to delete the original files when they haven’t been securely (even if partially) backed up? And even if you might, “the files” is kind of ambiguous.
xzzy

“rm*”? Here, let me try that:

-bash-3.2$ rm*

-bash: rm*: command not found

The command actually used was probably “rm -rf *”, though the asterisk is technically unneeded. “rm -rf .” would work too. Or a slash. Or a directory name.

It would be a better documentary if it was fact checked!
- http://egypt.urnash.com Egypt Urnash
  
  Do you want every kid watching the extras on the disc to learn the RIGHT command?
- traalfaz
  
  Actually RM* not rm* so it’s even less likely that the person doing this animation actually knew any unix commands.
  - taras
    
    *And* the screen only shows a couple of characters at a time. What is this, 1965? This movie is less accurate than Braveheart. F-!
    
    OR we could all chill out and enjoy it.
- http://www.facebook.com/felipe.esquivel Felipe Esquivel Reed
  
  At the end of the story says: “99% true, as far as we recall!!”
http://twitter.com/cbuchner1 Christian Buchner

We once had an employee who mapped our company’s SMB documents share to a folder in his user account on a Windows laptop. He later went on to delete this user account, which he had only created for testing purposes.

This wiped our entire documents drive when Windows deleted the account folder recursively. A backup procedure had not yet been put in place (but the needed backup scripts had just been developed). Everyone was pretty pissed, but it’s hard to blame this guy. ;)
poisonborz

Urm, I know this is just a short summary above, but it’s a bit too shallow to be complete. Employee copied the whole source of the movie for home? One would think that the source of a fully CG feature is more than a few terabytes. And even if it would fit onto a home HDD, why would anyone need the whole source? And even if one needs that, how is that secure?

Questions, questions.
- http://twitter.com/kennedy2 Bob Kennedy
  
  I assume that since they said “Woody’s hat disappeared, then his boots…” etc, that what had really been deleted was the wireframes or 3D models of things, then whole characters went away, then some parts of scenes. I don’t think that it was necessarily fully-rendered scenes that went away.
http://twitter.com/bigmark Mark Cohen

I did this exact same thing when I was at IBM in 1994–I was working in a group that compressed TV shows and movies for video-on-demand trials. I was in a hurry and went to clear out a temp directory. But I didn’t realize I was logged-in as Root…I typed rm -r (-r is recursive) and, since I was at the root level of the machine, erased EVERYTHING on that machine. This included all our compressed video in-progress as well as the rest of the machine.

After restoring everything from back-ups and system install disks, our system administrator revoked my root privileges. Wahhh!
Ito Kagehisa

What I got out of this is Pixar spends lots of money on animators and hardware and very little on highly competent sysadmins. You gotta pay six figures at least, you cheapskates, or you may as well just hire talented high school students.
- Brainspore
  
  You seem to be under the impression that paying a six-figure salary guarantees that you’ll get a highly competent person who never makes a really stupid screw-up. I think the existence of Wall Street pretty much disproves that one.
  
  I don’t know if they skimped on paying for decent sysadmins or not, but assuming they did based on this incident would be jumping to conclusions.
  - Ito Kagehisa
    
    Ha! You definitely schooled me on that one. I stand corrected and admonished.
    
    Still, you must agree any sysadmin who could allow this to happen is not competent. No backups for a month? And nobody knows? And people are copying the entire corporate product off-site without anyone knowing?
    
    But you’re right, paying well is no guarantee. It’s just a better bet than going cheap.
    - Brainspore
      
      No argument there. The crazy thing is that I’ve seen their facilities and was really impressed by the (apparent) safeguards they had in place; they even have gigantic fuel tanks and a big big bank of industrial generators so they don’t have to interrupt production in the event of a major blackout.
http://egypt.urnash.com Egypt Urnash

Okay I’m testing the offsite backups I’ve started keeping RIGHT NOW. I haven’t yet, it took a while to get everything up there.
Brainspore

Soon afterward, this incident became the inspiration for the scene at the end of “WALL•E” where the protagonist almost loses his soul due to insufficiently robust backup procedures.
- Graham Fluet
  
  he did lose it, as his soul was not valid code and so was not included in the reinitialization process. Fortunately EVE had a backup copy.
  - http://profile.yahoo.com/S5O3QRPYMSAXUDE4MGWKVLS6EI Jeff Imparato
    
    EVA
http://obsidian.kokolis.net Chloramphenicol

This is why my new techs don’t get root until they’ve proven that they are responsible enough to be given it. Hell, they don’t even get sudo until they’ve shown that they can avoid breaking a system.
Donald Petersen

I showed this video to a few people at work last week, since so many other TV shows have been switching from shooting on film to shooting on Red or Arri Alexa digital cameras. The footage looks fine, and the process is cheaper and more streamlined, which certainly makes it a tempting choice. But the lack of actual physical source material and the need for multiple backups and long-lasting, easily accessible archival media are not lightly-dismissible concerns. A few early adopters have belatedly discovered that, to their sorrow.
Donald Petersen

I showed this video to a few people at work last week, since so many other TV shows have been switching from shooting on film to shooting on Red or Arri Alexa digital cameras. The footage looks fine, and the process is cheaper and more streamlined, which certainly makes it a tempting choice. But the lack of actual physical source material and the need for multiple backups and long-lasting, easily accessible archival media are not lightly-dismissible concerns. A few early adopters have belatedly discovered that, to their sorrow.
Donald Petersen

I showed this video to a few people at work last week, since so many other TV shows have been switching from shooting on film to shooting on Red or Arri Alexa digital cameras. The footage looks fine, and the process is cheaper and more streamlined, which certainly makes it a tempting choice. But the lack of actual physical source material and the need for multiple backups and long-lasting, easily accessible archival media are not lightly-dismissible concerns. A few early adopters have belatedly discovered that, to their sorrow.
traalfaz

Every COMPETENT IT person I’ve talked to will tell you, if you’re not TESTING your backups, then you’re not backing up.
Teqnoire

Remember people, it’s NOT a backup system, it’s a RESTORE system!
obeyken

So… I assume at least 2 people got fired?
jkg

but why would she be taking home assets like hat, boots, etc to show her kid? it was implied she was just taking home rendered frames…that’s the part that doesn’t jibe here, cute animation not withstanding.
- http://twitter.com/cjporkchop cjporkchop
  
  I think she was taking assets home to work on while caring for the newborn, not taking them home to show to her children.
- ocker3
  
  It wasn’t to Show her kids, it was because she had kids and wanted to do work at home, subtle difference. I kept wondering why she was showing the unfinished stuff to an infant, then realised I was making assumptions.
David Carroll

Why isn’t don’t use Linux the moral of this story? ;)
- http://profiles.google.com/robertbos Rob Bos
  
  The answer to “a chainsaw cut off my leg” isn’t “use safety scissors to cut down trees”.
- http://www.facebook.com/profile.php?id=624187476 Jesse Carlson
  
  You can do the same thing in any OS. The only difference is, Linux is industry standard, and… Free.
koko szanel

This is a cute BS story, the kind you tell your kids. I actually feel offended by the stupidity of that clip.
oasisob1

Everybody’s missing the real story. That lady made a copy of a movie and kept it at home. She owes somebody millions of dollars for that. Millions (and she should be thrown in jail). And let that be a lesson to anyone else who wants to ‘keep a copy at home so they can work on it’.
putty

Back in the day, the older Unix variants did not automatically convert
rm *
to
rm -i *
(the -i parameter stands for interactive, telling the system to ask “are you sure you want to delete file x?” for each individual file matching the pattern).

To combat this we used to add a line to all new user’s login scripts (.cshrc, .bashrc etc) that went something like:
alias rm “rm -i”
which had the same effect of forcing a user to confirm each file even if they issued an rm statement with a wildcard in a large folder.

Unix admins sometimes find themselves faced with files (accidentally created by some poor lost soul) whose names include oddball characters and nasty things like * and ? or even a series of backspaces that make your directory listing look like the title scene from The Matrix. The trick to deleteing those files safely without having to try to figure out how to actually specify the names of the files is to use
rm -i *
and just say no to every file except the ones with the funky names that you want to remove.
- http://disqus.com/Kimmoth/ Kimmo
  
  That’s the kind of post I clicked on this thread to read.
- Ito Kagehisa
  
  Back in the day, the older Unix variants did not automatically convert rm * to rm -i *
  
  They still don’t, really. Some distros (e.g. Red Hat) have alias rm=’rm -i’ in the default profile or /etc/skel or bashrc, but certainly not all of them.
  
  Another way to delete files with metacharacters in the names is single-quoting or using a question mark (single character glob), that way you won’t have to say “no” quite so many times.
http://tomcat.pip.verisignlabs.com/ Tommy

I had a similar scare a few years back while working in Telecommunications, where a mistype in combination with a bug in jpsoft 4NT started erasing our entire source code archive. Turned out that our new outsourced IT department had completely forgot about this server when migrating to a new backup solution.

Luckily I had aborted the command before it deleted anything recent, so we were able to restore it using a year old backup and local copies of some files.
Neil B

This reminded me of the time we moved into a new office. We bought the biggest, honkinest UPS money could buy. This thing was to keep ALL of the machines running for like an hour if power failed. It was so large and difficult to site that the installers built a fake room around it, and plastered and wallpapered that mother in.

The payoff of this story is obvious. The first brownout we had, all servers and computers went down. We had to physically break down the constructed wall to discover that the UPS batteries had been installed incorrectly.
- phuzz
  
  Or you could do what I did, and forget to replace the batteries in our main UPS.
  
  Last weekend the power failed for about a minute. Unfortunately the UPS decided it had insufficient runtime after just one second and shut down.
  :(
  
  We have new batteries already :)
  
  (we’re a small company, so it wasn’t a major problem, just a bit embarassing)
DewiMorgan

Back when Solaris was called SunOS, emacs backed up working copies of the files that you edited, saving them as the same filename, but with an appended tilde: “dissertation.doc~” and so on.

Sometimes, since you had to store all your work within your quota, and the quota was only five megs, you wanted to delete all the backup files to make space. You can do that with:
rm -rf *~

Sometimes, an incautious typist such as myself had cause to mourn the fact that Sun machines had the tilde key right next to the enter key.
http://www.facebook.com/profile.php?id=624187476 Jesse Carlson

Yeahhhh, this story sounds amazingly… Dumb, and so unlikely.

First of all, it would be most likely that she had the still images that she had taken home to show her kids, not the entire show directories including all rendered frames, anim renders, digital assets, textures, environments, props, etc. As already pointed out, that would be Terabytes worth of data, not easily transported on a thumb drive or anything like that. Secondly, she wouldn’t be taking all of the digital assets home, since they have custom in house software that I’m pretty sure would be unusable outside of the facility, so, no point to take it home to begin with. Third, all facilities I’ve worked in won’t let user’s simply rm a directory, as they are version controlled. Meaning, you have to have the system run a rm command, which stores a numerical version of each iteration of the file which you can browse and recover at any time. This keeps a history of all operations, on all important files. Fourth, all facilities I’ve worked at (Sony, Dreamworks Animation, Walt Disney Animation, Digital Domain, etc) have hourly backups. How do all your backups for a whole month just “not work”? You either have the file or you don’t. If it halted halfway through save it, that file might be corrupted, but that’s an extremely rare occurrence.

The whole thing just sounds like a dumb internet rumor to me. Besides, if she had the entire movie directory backed up on her home computer, that would have to have been a *serious* computer to be able to store that much information. Pixar would definitely have been holding all of that data on multiple server machines, each with many times more storage on them than any single home computer would ever have.