Computers piece together millions of shredded Stasi documents

A moving and intriguing Wired feature tells the story of the activists, hackers and engineers who are working to un-shred millions of hand-shredded secret files that the East German Stasi ripped to pieces in the run-up to the collapse of the Berlin Wall. The secret police panicked when they realized that they were about to lose their tight rein on power and shredded as much as they could -- but they had collected more files than any other bureaucracy in the history of the world, and they couldn't shred fast enough. So they assigned a detail to go into the basement, into a secure, copper-lined computing room, and hand-tear the most sensitive documents, all day long, millions of them.

The liberators of the Stasi's archive saved the hand-shredded material and now computer scientists are working to piece it all back together, using clever algorithms reminiscent of the systems described in Vernor Vinge's groundbreaking novel Rainbows End. The description of the files themselves are incredible -- one activist had sixty binders compiled on her, comprising every movement she took (she was followed constantly by crew-cut secret police in white vans who'd crawl the curb a few metres behind her as she walked down the street).

The data for the 400-bag pilot project is stored on 22 terabytes worth of hard drives, but the system is designed to scale. If work on all 16,000 bags is approved, there may be hundreds of scanners and processors running in parallel by 2010. (Right now they're analyzing actual documents, but still mostly vetting and refining the system.) Then, once assembly is complete, archivists and historians will probably spend a decade sorting and organizing. "People who took the time to rip things up that small had a reason," Nickolay says. "This isn't about revenge but about understanding our history." And not just Germany's – Nickolay has been approached by foreign officials from Poland and Chile with an interest in reconstructing the files damaged or destroyed by their own repressive regimes...

The truth is, for Poppe the reconstructed documents haven't contained bombshells that are any bigger than the information in the rest of her file. She chooses a black binder and sets it down on the glass coffee table in her living room. After lighting a Virginia Slim, she flips to a page-long list of snitches who spied on her. She was able to match codenames like Carlos, Heinz, and Rita to friends, coworkers, and even colleagues in the peace movement. She even tracked down the Stasi officer who managed her case, and after she set up a sort of ambush for him at a bar – he thought he was there for a job interview – they continued to get together. Over the course of half a dozen meetings, they talked about what she found in her files, why the Stasi was watching her, what they thought she was doing. For months, it turned out, an agent was assigned to steal her baby stroller and covertly let the air out of her bicycle tires when she went grocery shopping with her two toddlers. "If I had told anyone at the time that the Stasi was giving me flat tires, they would have laughed at me," she says. "It was a way to discredit people, make them seem crazy. I doubted my own sanity sometimes." Eventually, the officer broke off contact, but continued to telephone Poppe – often drunk, often late at night, sometimes complaining about his failing marriage. He eventually committed suicide.

Link

(Image: Ministry for State Security HQ, a Creative Commons Attribution licensed photo from Mintyboy's Flickr stream)

See also:
David Byrne's trip report from Berlin's Stasi museum
Stasi chief was an Orwell fan, bent reality to get room 101
Stasi smell museum

• [ernie]

  #7, Yeah, but

  “For months, it turned out, an agent was assigned to steal her baby stroller and covertly let the air out of her bicycle tires when she went grocery shopping with her two toddlers.”

  This is on a different level entirely.

• [lordbordz]

  Same story was already done in NYT almost 5 years ago:
  http://query.nytimes.com/gst/fullpage.html?res=9803E3D7123CF934A25754C0A9659C8B63

• [Takuan]

  I wonder what will happen when the Cheney presidency ends and America perhaps begins to emerge from her long night? Will there be similar tales and confessions?

• [noen]

  It has only just started here and I think we have a ways to go. But I bet there is plenty of info compiled on activists here in the US. My God they are so predictably paranoid aren’t they?

• [tix]

  I first read about the “Puzzle Women” in Ana Funder’s Stasiland a few years ago. The book is well worth grabbing if you found this story interesting.

• [Chris]

  This is *such* good news. Like TIX I read about the puzzle women in Stasiland; and that made it clear how heartbreakingly slow their work was, and how inadequate their numbers, given the mountains of paper still to be processed. Now maybe those people still searching for tortured/murdered/disappeared loved ones might have a chance of finding out the truth before they die of old age.

• [deckard]

  Vernor Vinge is an absolute genius to place such a moral dilemma in that example Cory mentioned: would you destroy all the paper books in order to secure high-fidelity and complete digital copies that you cannot touch but you can data-mine forever, or would you feel bewildered?

  When I bought my copy of Rainbow’s End, I typed the first sentence of the book into the search engine and got a link where the complete book was pasted, by the author himself, in plaintext. Maybe it was meant to be hidden through simple steganography, but got harvested by search engines, but I can see it’s still there: http://www.vrinimi.org/rainbowsend.html.

  So, if you wanted to see what Cory meant in his example, open the page and search (ctrl-f) for Librareome… and then buy the book :)

• [paulatz]

  If you are interested is the history of Stasi, or you just like nice films, you should really watch “The Lives of Others”.

• [vaporlock]

  I’m not trying to be a troll with this, but this history is not accurately framed in my opinion. The STASI were not engaged in policing that was significantly more intense than the current regime in Washington. The major difference was that they existed in the age of paper records, and we exist in the digital age. Eastern Germany really WAS faced with an external threat, that really DID exert a covert force to overthrow their government. The outside force consisted of Billions of dollars in espionage efforts by much of the Western world. This threat was real and much like our created “war on terror” it perverted a government and society to do “evil” things. Things that our own government does everyday. This was exceptional only in the personal lives of people who were affected. The people persecuted by the STASI have the right to be pissed, but the rest of us should look in the mirror and see the true terrorists. Read a book about Central America in the 1980′s if you want to see what real persecution is.

• [Halloween Jack]

  There’s something bizarre and almost funny about a former secret police drunk-dialing their former prey; imagine an out-of-work O’Brien from 1984 getting sloppy on Victory Gin and weeping in front of Winston Smith about how much he missed Big Brother. There’s a cornucopia of one-act plays to be found in this story, I think.