How Pixar almost lost Toy Story 2 to a bad backup

Discuss

49 Responses to “How Pixar almost lost Toy Story 2 to a bad backup”

  1. Dan Century says:

    Now I have a good rational for  fighting the no-telecommuting/no-thumb drive policy at work.

  2. swankgd says:

    No, it’s a good reminder to TEST your backups and TEST your restore procedure.

  3. Kris N says:

    The Pixar Studio Stories are some of the best reasons to get the Blu-ray editions of their films. Well, that and the films themselves…

  4. chrisspurgeon says:

    HERE is the big lesson everyone should learn from this… your backup strategy does not work unless you CAN ACTUALLY RESTORE YOUR STUFF.  At the last three companies I’ve worked at we’ve learned the hard way that when it came time to restore vital lost information from a backup the damn restore didn’t work for one reason or another (backup files were actually blank, restore software not compatible with current system, backup hard drive corrupted).

    PROVE that your backup system works by doing regular restore exercises. Don’t be complacent. Don’t find out the hard way that your restore strategy has a fatal flaw.

  5. arcfinn says:

     Kind of a shame they got it back… Opinions may vary. 

  6. Dan Hibiki says:

    who gives users root access any way?

    • Guest says:

       No one said they trashed the OS, just that someone did an rm * in the directory containing the movie (I’m guessing the rendering models)

  7. Nylund says:

    I was using a server to do run some statistical models with a big data set (12 gigs or so?).  At one point, I decided I should back up that data set, but after I started the process, I got called on to go do something else.  I was unsure what would happen to the backup process on this communal server if I left and someone else showed up, so I canceled it before it was complete.  I got some prompt asking me what to do with the files.  I assumed it meant the files that I’d partially, but not completely, backed up.  Knowing I was going to be returning soon, I figured I’d just start the backup from scratch again, so I selected “delete” only to then realize it meant the ORIGINAL files, not the partial backup I’d interrupted.  It did a true delete (not just trash/recycle bin delete like in Windows or OSX).  I was able to recover most of it, which, in tandem with an out of date copy on my personal laptop, I was ale to get everything kosher again (after many hours of work).

    Or, in other words, I was a giant idiot.  I felt really dumb, especially since the whole thing started because I was trying to be responsible about making backups.  I just really messed up in the process.

    • Sounds to me more like shitty UI design. Why would anyone ever want to delete the original files when they haven’t been securely (even if partially) backed up? And even if you might, “the files” is kind of ambiguous.

  8. xzzy says:

    “rm*”? Here, let me try that:

    -bash-3.2$ rm*

    -bash: rm*: command not found

    The command actually used was probably “rm -rf *”, though the asterisk is technically unneeded. “rm -rf .” would work too. Or a slash. Or a directory name.

    It would be a better documentary if it was fact checked!

  9. We once had an employee who mapped our company’s SMB documents share to a folder in his user account on a Windows laptop. He later went on to delete this user account, which he had only created for testing purposes.

    This wiped our entire documents drive when Windows deleted the account folder recursively. A backup procedure had not yet been put in place (but the needed backup scripts had just been developed). Everyone was pretty pissed, but it’s hard to blame this guy. ;)

  10. poisonborz says:

    Urm, I know this is just a short summary above, but it’s a bit too shallow to be complete. Employee copied the whole source of the movie for home? One would think that the source of a fully CG feature is more than a few terabytes. And even if it would fit onto a home HDD, why would anyone need the whole source? And even if one needs that, how is that secure?

    Questions, questions.

    • Bob Kennedy says:

       I assume that since they said “Woody’s hat disappeared, then his boots…” etc, that what had really been deleted was the wireframes or 3D models of things, then whole characters went away, then some parts of scenes. I don’t think that it was necessarily fully-rendered scenes that went away.

  11. Mark Cohen says:

    I did this exact same thing when I was at IBM in 1994–I was working in a group that compressed TV shows and movies for video-on-demand trials. I was in a hurry and went to clear out a temp directory. But I didn’t realize I was logged-in as Root…I typed rm -r  (-r is recursive) and, since I was at the root level of the machine, erased EVERYTHING on that machine. This included all our compressed video in-progress as well as the rest of the machine.

    After restoring everything from back-ups and system install disks, our system administrator revoked my root privileges. Wahhh!

  12. Ito Kagehisa says:

    What I got out of this is Pixar spends lots of money on animators and hardware and very little on highly competent sysadmins.  You gotta pay six figures at least, you cheapskates, or you may as well just hire talented high school students.

    • Brainspore says:

      You seem to be under the impression that paying a six-figure salary guarantees that you’ll get a highly competent person who never makes a really stupid screw-up. I think the existence of Wall Street pretty much disproves that one.

      I don’t know if they skimped on paying for decent sysadmins or not, but assuming they did based on this incident would be jumping to conclusions.

      • Ito Kagehisa says:

         Ha!  You definitely schooled me on that one.  I stand corrected and admonished.

        Still, you must agree any sysadmin who could allow this to happen is not competent.  No backups for a month?  And nobody knows?  And people are copying the entire corporate product off-site without anyone knowing?

        But you’re right, paying well is no guarantee.  It’s just a better bet than going cheap.

        • Brainspore says:

          No argument there. The crazy thing is that I’ve seen their facilities and was really impressed by the (apparent) safeguards they had in place; they even have gigantic fuel tanks and a big big bank of industrial generators so they don’t have to interrupt production in the event of a major blackout.

  13. Egypt Urnash says:

    Okay I’m testing the offsite backups I’ve started keeping RIGHT NOW. I haven’t yet, it took a while to get everything up there.

  14. Brainspore says:

    Soon afterward, this incident became the inspiration for the scene at the end of “WALL•E” where the protagonist almost loses his soul due to insufficiently robust backup procedures.

  15. This is why my new techs don’t get root until they’ve proven that they are responsible enough to be given it.  Hell, they don’t even get sudo until they’ve shown that they can avoid breaking a system.

  16. Donald Petersen says:

    I showed this video to a few people at work last week, since so many other TV shows have been switching from shooting on film to shooting on Red or Arri Alexa digital cameras.  The footage looks fine, and the process is cheaper and more streamlined, which certainly makes it a tempting choice.  But the lack of actual physical source material and the need for multiple backups and long-lasting, easily accessible archival media are not lightly-dismissible concerns.  A few early adopters have belatedly discovered that, to their sorrow.

  17. Donald Petersen says:

    I showed this video to a few people at work last week, since so many other TV shows have been switching from shooting on film to shooting on Red or Arri Alexa digital cameras.  The footage looks fine, and the process is cheaper and more streamlined, which certainly makes it a tempting choice.  But the lack of actual physical source material and the need for multiple backups and long-lasting, easily accessible archival media are not lightly-dismissible concerns.  A few early adopters have belatedly discovered that, to their sorrow.

  18. Donald Petersen says:

    I showed this video to a few people at work last week, since so many other TV shows have been switching from shooting on film to shooting on Red or Arri Alexa digital cameras.  The footage looks fine, and the process is cheaper and more streamlined, which certainly makes it a tempting choice.  But the lack of actual physical source material and the need for multiple backups and long-lasting, easily accessible archival media are not lightly-dismissible concerns.  A few early adopters have belatedly discovered that, to their sorrow.

  19. traalfaz says:

    Every COMPETENT IT person I’ve talked to will tell you, if you’re not TESTING your backups, then you’re not backing up.

  20. Teqnoire says:

    Remember people, it’s NOT a backup system, it’s a RESTORE system!

  21. obeyken says:

    So… I assume at least 2 people got fired?  

  22. jkg says:

    but why would she be taking home assets like hat, boots, etc to show her kid? it was implied she was just taking home rendered frames…that’s the part that doesn’t jibe here, cute animation not withstanding.

    • cjporkchop says:

       I think she was taking assets home to work on while caring for the newborn, not taking them home to show to her children.

    • ocker3 says:

       It wasn’t to Show her kids, it was because she had kids and wanted to do work at home, subtle difference. I kept wondering why she was showing the unfinished stuff to an infant, then realised I was making assumptions.

  23. David Carroll says:

    Why isn’t don’t use Linux the moral of this story?  ;) 

  24. koko szanel says:

    This is a cute BS story, the kind you tell your kids. I actually feel offended by the stupidity of that clip.

  25. oasisob1 says:

    Everybody’s missing the real story. That lady made a copy of a movie and kept it at home. She owes somebody millions of dollars for that. Millions (and she should be thrown in jail). And let that be a lesson to anyone else who wants to ‘keep a copy at home so they can work on it’.

  26. putty says:

    Back in the day, the older Unix variants did not automatically convert
    rm *
    to
    rm -i *
    (the -i parameter stands for interactive, telling the system to ask “are you sure you want to delete file x?” for each individual file matching the pattern).

    To combat this we used to add a line to all new user’s login scripts (.cshrc, .bashrc etc) that went something like:
    alias rm “rm -i”   
    which had the same effect of forcing a user to confirm each file even if they issued an rm statement with a wildcard in a large folder.

    Unix admins sometimes find themselves faced with files (accidentally created by some poor lost soul) whose names include oddball characters and nasty things like * and ? or even a series of backspaces that make your directory listing look like the title scene from The Matrix.  The trick to deleteing those files safely without having to try to figure out how to actually specify the names of the files is to use
    rm -i *
    and just say no to every file except the ones with the funky names that you want to remove.

    • Kimmo says:

      That’s the kind of post I clicked on this thread to read.

    • Ito Kagehisa says:

      Back in the day, the older Unix variants did not automatically convert rm * to rm -i *

       They still don’t, really.   Some distros (e.g. Red Hat) have alias rm=’rm -i’ in the default profile or /etc/skel or bashrc, but certainly not all of them.

      Another way to delete files with metacharacters in the names is single-quoting or using a question mark (single character glob), that way you won’t have to say “no” quite so many times.

  27. Tommy says:

    I had a similar scare a few years back while working in Telecommunications, where a mistype in combination with a bug in jpsoft 4NT started erasing our entire source code archive. Turned out that our new outsourced IT department had completely forgot about this server when migrating to a new backup solution.

    Luckily I had aborted the command before it deleted anything recent, so we were able to restore it using a year old backup and local copies of some files.

  28. Neil B says:

    This reminded me of the time we moved into a new office. We bought the biggest, honkinest UPS money could buy. This thing was to keep ALL of the machines running for like an hour if power failed. It was so large and difficult to site that the installers built a fake room around it, and plastered and wallpapered that mother in.

    The payoff of this story is obvious. The first brownout we had, all servers and computers went down. We had to physically break down the constructed wall to discover that the UPS batteries had been installed incorrectly.

    • phuzz says:

       Or you could do what I did, and forget to replace the batteries in our main UPS.

      Last weekend the power failed for about a minute.  Unfortunately the UPS decided it had insufficient runtime after just one second and shut down.
      :(

      We have new batteries already :)

      (we’re a small company, so it wasn’t a major problem, just a bit embarassing)

  29. DewiMorgan says:

    Back when Solaris was called SunOS, emacs backed up working copies of the files that you edited, saving them as the same filename, but with an appended tilde: “dissertation.doc~” and so on.

    Sometimes, since you had to store all your work within your quota, and the quota was only five megs, you wanted to delete all the backup files to make space. You can do that with:
    rm -rf *~

    Sometimes, an incautious typist such as myself had cause to mourn the fact that Sun machines had the tilde key right next to the enter key.

  30. Yeahhhh, this story sounds amazingly…  Dumb, and so unlikely.

    First of all, it would be most likely that she had the still images that she had taken home to show her kids, not the entire show directories including all rendered frames, anim renders, digital assets, textures, environments, props, etc.  As already pointed out, that would be Terabytes worth of data, not easily transported on a thumb drive or anything like that.  Secondly, she wouldn’t be taking all of the digital assets home, since they have custom in house software that I’m pretty sure would be unusable outside of the facility, so, no point to take it home to begin with.  Third, all facilities I’ve worked in won’t let user’s simply rm a directory, as they are version controlled.  Meaning, you have to have the system run a rm command, which stores a numerical version of each iteration of the file which you can browse and recover at any time.  This keeps a history of all operations, on all important files.  Fourth, all facilities I’ve worked at (Sony, Dreamworks Animation, Walt Disney Animation, Digital Domain, etc) have hourly backups.  How do all your backups for a whole month just “not work”?  You either have the file or you don’t.  If it halted halfway through save it, that file might be corrupted, but that’s an extremely rare occurrence.

    The whole thing just sounds like a dumb internet rumor to me.  Besides, if she had the entire movie directory backed up on her home computer, that would have to have been a *serious* computer to be able to store that much information.  Pixar would definitely have been holding all of that data on multiple server machines, each with many times more storage on them than any single home computer would ever have.

Leave a Reply