Brewster Kahle explains the inner workings of the Internet Wayback Machine on the O'Reilly Network.
What's amazing to me is the fact that the hardware is free. For doing things even in the hundreds of terabytes, it costs in the hundreds of thousands of dollars. When you talk to most people in IT departments, they spend a couple hundred thousand dollars just on a CPU, much less a terabyte of disk storage. You buy from EMC a terabyte for maybe $300,000. That's just the storage for 1 TB. We can buy 100 TBs with 250 CPUs to work on it, all on a high-speed switch with redundancy built in. Something has changed by using these modern constructs that are heavily used at Google, Hotmail, here, Transmeta. There's a whole sector of companies that are more cost-constrained than say, banks, that just buy Oracle and Sun and EMC.
