Neatorama posted the origins of several common symbols, reprinted from a book titled Uncle John's Supremely Satisfying Bathroom Reader.
Exclamation Point Origin: Like the question mark, the exclamation point was invented by stacking letters. The mark comes from the Latin word io, meaning "exclamation of joy." Written vertically, with the i above the o, it forms the exclamation point we use today.