Oona Räisänen has written a thorough and engrossing article about the noises emitted by dial-up modems while they connect and handshake, and the accompanying graphic (ZOMG HUGE) is nothing short of spectacular. It would make a great full-size poster -- maybe a framed art-print.
Now the modems must address the problem of echo suppression. When humans talk, only one of them is usually talking while the other one listens. The telephone network exploits this fact and temporarily silences the return channel to suppress any confusing echoes of the talker's own voice.
Modems don't like this at all, as they can very well talk at the same time (it's called full-duplex). The answering modem now puts on a special answer tone that will disable any echo suppression circuits on the line. The tone also has periodic "snaps" (180° phase transitions) that aim to disable yet another type of circuit called echo canceller.
Now the modems will list their supported modulation modes and try to find one that both know. They also probe the line with test tones to see how it responds to tones of different frequencies, and how much it attenuates the signal. They exchange their test results and decide a speed that is suitable for the line.
After this, the modems will go to scrambled data. They put their data through a special scrambling formula before transmission to make its power distribution more even and to make sure there are no patterns that are suboptimal for transfer. They listen to each other sending a series of binary 1's and adjust their equalizers to optimally shape the incoming signal.