Command mode and Data mode

Command mode and Data mode refers to the two modes in which a computer modem may operate. These modes are defined in the Hayes command set, which is the de facto standard for all modems. These modes exist because there is only one channel of communication between the modem and the computer, which must carry both the computer's commands to the modem, as well as the data that the modem is enlisted to transmit to the remote party over the telephone line.

When a modem is in command mode, any characters sent to it are interpreted as commands for the modem to execute, per the Hayes command set. A command is preceded by the letters 'AT', which stand for 'Attention'. For example, if a modem receives 'ATDT5551212' while in the command mode, it interprets that as an instruction to dial the numbers 5551212 on the telephone, using touch-tone dialing. During a command mode operation, the modem may send responses back to the computer indicating the outcome of the command. For example, the modem may respond with the word "BUSY" in response to the ATDT command, if it hears a busy signal after dialing and is configured to listen for busy signals. Any communication in command mode (in both directions) is terminated by a carriage return.

When a modem is in data mode, any characters sent to the modem are intended to be transmitted to the remote party. The modem enters data mode immediately after it makes a connection. For example, if ATDT5551212 resulted in a phone call that was answered by another computer modem, the modem would report the word "CONNECT" and then switch to data mode. Any further characters received over the serial link are deemed to be from the remote party, and any characters sent are transmitted to the remote party.

When a voice-capable modem is in "voice data" mode, any data sent to the modem is interpreted as audio data to be played over the phone line, rather than character bytes to be transmitted digitally to the other party.

Switching between modes
Modems always start out in command mode when powered up. Here are the ways a modem can switch to data mode:
 * After a successful dial-out connection in response to an "ATD" dial command in which the modem reaches another modem.
 * After answering the phone with the "ATA" answer command, if another modem is on the other end.
 * After answering the phone automatically for some pre-configured reason (such as auto-answer), and connecting to another modem. (Almost all modems support auto-answering when given the command "ATS0=1".)
 * After being given the "ATO" (that's three letters A-T-Oh, not A-T-zero) command after being put back in the command mode with an escape sequence (see below).
 * In response to similar dialing or connecting commands for fax or voice communications.

Modems switch back into command mode from data mode for the following reasons:
 * The connection got broken (for example, the other party hung up).
 * The computer issued an escape command, which is usually a 1-second pause, then the three characters "+++", then another 1-second pause. The connection remains, but the modem can accept commands, such as "ATH" for hangup.  The computer can issue the "ATO" command to return to data mode.
 * The computer instructed the modem to terminate the call by setting the Data Terminal Ready (DTR) pin to an "off" state.

The DTR pin is usually how computers invoke the termination of a modem call nowadays. The +++ escape sequence is used, and may even be disabled to avoid malfunction in case these characters are legitimately a part of the data stream (ignoring the 1-second pauses).

Practical contemporary use
Today, most modems are configured with the characters "&C1&D2" in the initialization string, or otherwise behave this way by default. This causes the following behavior:
 * The modem uses the Data Carrier Detect (DCD) pin to signal whether it's connected to a host. The computer can generally just read the DCD pin at any time and always know if the modem is in command or data mode.  DCD high means data mode, and low means command mode.  DCD is a signal sent from the modem to the computer.  (Note that DCD also remains high if in command mode after a +++ escape sequence, but in practice, software uses this so rarely, if at all, so it's not really an issue.)
 * The modem interprets the Data Terminal Ready (DTR) pin as a signal from the computer to know when it wants to terminate a call. DTR is a signal from the computer to the modem.  The computer keeps DTR high at all times until it wants to terminate a call, at which time the computer lowers DTR for a second or two.  The computer also keeps DTR low when no programs are running that want to use the modem - this keeps the modem from answering calls due to auto-answer or otherwise doing something unexpected or undesired.  The computer may safely assume that after DTR has been brought low for a couple seconds, that the modem will be in command mode.

Data Link Escape (DLE) messages
For normal dial-up data communications, modems enter data mode only once - starting when the session connects, and ending when the session disconnects. However, when modems are used for fax and voice (audio) communication, they rapidly switch between command and data modes several times during a call. This is because the role of the modem changes more frequently - rather than simply handing bytes from point A to point B, it is either negotiating parameters and pages with a fax machine, or switching between recording, pausing, and playback audio modes.

In fax and voice data modes, sometimes events occur that the modem wishes to signal to the computer regardless of whether it's in command or data mode. An example of such an event is a caller pressing a touch-tone key in voice mode. Other common out-of-band messages are notifications from the modem that data was lost because the computer is sending data either too slow or too fast, or that the modem hears an unexpected dial-tone on the line (meaning the caller probably hung up), or that the extension handset was picked up or hung up.

This type of event is signalled with a DLE message, which is a single-character message preceded by a Data Link Escape code. DLE is a character sent from the modem to the computer whose definition is unrelated to either command or data mode, and which uses a specific ASCII code (0x10) which never occurs in any AT commands or responses, so that it can be isolated from the command/response stream on that merit alone. In these modes, whenever a DLE (0x10) is sent, the character following it has a special non-command and non-data meaning. For example, a DLE followed by the number 2 means the caller pressed the number 2 on his telephone keypad. This could happen in either command or data mode so long as the phone line is in use by the modem and someone is on the other end of the line.

The DLE character could appear in fax or audio data. To avoid misinterpreting a 0x10 data byte as a DLE "bit stuffing" is used. Stuffing is a common encoding scheme for modems and used to escape character in streams. In this case stuffing means two DLE's characters in a row are interpreted as one literal byte with value 0x10.

DLE is also used lightly in communication from the computer to the modem. One specific DLE event signals the end of a fax page, or the end of an audio file. That event returns the modem back into command mode. Unlike in standard dial-up data mode, dropping DTR isn't an appropriate way to resume command mode since a hangup is not desired, and an escape code with mandatory pauses isn't suitable either. Because of this, literal 0x10 bytes in data are doubled from the computer to the modem as well.

DLE is never used in standard modem-to-modem data modes such as the one used for dial-up Internet access, at least not by the modem itself. In these modes, DLE is passed over the line just like any other character. DLE (and DLE-escaped messages) are only used by a modem for fax and voice applications, when it is specifically placed in a fax or voice mode.