Fair use is the smallest part of the public's side of copyright. Sure, we have the right to quote, time-shift, format-shift, transform and so on. But the big piece is all the stuff that copyright shouldn't/doesn't touch: selling used media, watching your movies no matter what country you're in, the right to privacy in the books you read, and so on. All these are under fire in the copyright wars (I once had an argument with a MPAA vice president who wanted to make it impossible to watch a movie anonymously).
If you want to do something to preserve the public's side of the copyright bargain, check out the Access to Knowledge Treaty, a proposed treaty that the UN's World Intellectual Property Organization has recently green-lighted. Link (Thanks, Evan!)