 MODEM PROTOCOL OVERVIEW   = 1/1/82 by Ward Christensen.  I will maintain a master copy of = this.  Please pass on changes or suggestions via CBBS/Chicago ; at (312) 545-8086, CBBS/CPMUG (312) 849-1132 or by voice at  (312) 849-6279.   : Rev: 08/09/82 Ward C. Change ACK to 06H (was error as 05H)  $ 1/13/85 by John Byrns -- CRC option.D Please pass on any reports of errors in this document or suggestionsD for improvement to me via Ward's/CBBS at (312) 849-1132, or by voice at (312) 885-1105.  4 4/17/85 by Keith Petersen -- Merge of the two above.     	    Table of Contents    1. DEFINITIONS &  2. TRANSMISSION MEDIUM LEVEL PROTOCOL   3. MESSAGE BLOCK LEVEL PROTOCOL  4. FILE LEVEL PROTOCOL .  5. DATA FLOW EXAMPLE INCLUDING ERROR RECOVERY  6. PROGRAMMING TIPS.   7. OVERVIEW OF CRC OPTION*  8. MESSAGE BLOCK LEVEL PROTOCOL, CRC MODE  9. CRC CALCULATION 2 10. FILE LEVEL PROTOCOL, CHANGES FOR COMPATIBILITY& 11. DATA FLOW EXAMPLES WITH CRC OPTION   -------- 1. DEFINITIONS.     <soh> 01H    <eot> 04H  | <ack> 06H    <nak> 15H    <can> 18H 	   <C>	43H   . -------- 2. TRANSMISSION MEDIUM LEVEL PROTOCOL  3 Asynchronous, 8 data bits, no parity, one stop bit.   ?     The protocol imposes no restrictions on the contents of the = data being transmitted.  No control characters are looked for ? in the 128-byte data messages.  Absolutely any kind of data may < be sent - binary, ASCII, etc.  The protocol has not formally; been adopted to a 7-bit environment for the transmission of ? ASCII-only (or unpacked-hex) data , although it could be simply < by having both ends agree to AND the protocol-dependent data> with 7F hex before validating it.  I specifically am referring6 to the checksum, and the block numbers and their ones- complement. <     Those wishing to maintain compatibility of the CP/M file> structure, i.e. to allow modemming ASCII files to or from CP/M' systems should follow this data format: ,   * ASCII tabs used (09H); tabs set every 8.'   * Lines terminated by CR/LF (0DH 0AH) 4   * End-of-file indicated by ^Z, 1AH.  (one or more)8   * Data is variable length, i.e. should be considered a9     continuous stream of data bytes, broken into 128-byte 2     chunks purely for the purpose of transmission.7   * A CP/M "peculiarity": If the data ends exactly on a 7     128-byte boundary, i.e. CR in 127, and LF in 128, a 8     subsequent sector containing the ^Z EOF character(s)5     is optional, but is preferred.  Some utilities or 6     user programs still do not handle EOF without ^Zs.9   * The last block sent is no different from others, i.e.      there is no "short block".    ( -------- 3. MESSAGE BLOCK LEVEL PROTOCOL  '  Each block of the transfer looks like: 2 <SOH><blk #><255-blk #><--128 data bytes--><cksum>
     in which:  <SOH>	    = 01 hex> <blk #>     = binary number, starts at 01 increments by 1, and$ 	      wraps 0FFH to 00H (not to 01); <255-blk #> = blk # after going thru 8080 "CMA" instr, i.e. 7 	      each bit complemented in the 8-bit block number. / 	      Formally, this is the "ones complement". > <cksum>     = the sum of the data bytes only.  Toss any carry.   -------- 4. FILE LEVEL PROTOCOL   , ---- 4A. COMMON TO BOTH SENDER AND RECEIVER:  ?     All errors are retried 10 times.  For versions running with ? an operator (i.e. NOT with XMODEM), a message is typed after 10 6 errors asking the operator whether to "retry or quit".9     Some versions of the protocol use <can>, ASCII ^X, to > cancel transmission.  This was never adopted as a standard, as8 having a single "abort" character makes the transmission? susceptible to false termination due to an <ack> <nak> or <soh> 9 being corrupted into a <can> and cancelling transmission. >     The protocol may be considered "receiver driven", that is,? the sender need not automatically re-transmit, although it does  in the current implementations.   ( ---- 4B. RECEIVE PROGRAM CONSIDERATIONS:  ;     The receiver has a 10-second timeout.  It sends a <nak> = every time it times out.  The receiver's first timeout, which = sends a <nak>, signals the transmitter to start.  Optionally, ? the receiver could send a <nak> immediately, in case the sender : was ready.  This would save the initial 10 second timeout.? However, the receiver MUST continue to timeout every 10 seconds   in case the sender wasn't ready.;     Once into a receiving a block, the receiver goes into a ? one-second timeout for each character and the checksum.  If the 8 receiver wishes to <nak> a block for any reason (invalid= header, timeout receiving data), it must wait for the line to ( clear.  See "programming tips" for ideas;     Synchronizing:  If a valid block number is received, it ? will be: 1) the expected one, in which case everything is fine; = or 2) a repeat of the previously received block.  This should = be considered OK, and only indicates that the receivers <ack> ? got glitched, and the sender re-transmitted; 3) any other block = number indicates a fatal loss of synchronization, such as the > rare case of the sender getting a line-glitch that looked like2 an <ack>.  Abort the transmission, sending a <can>  ( ---- 4C. SENDING PROGRAM CONSIDERATIONS.  ;     While waiting for transmission to begin, the sender has 8 only a single very long timeout, say one minute.  In the; current protocol, the sender has a 10 second timeout before = retrying.  I suggest NOT doing this, and letting the protocol < be completely receiver-driven.  This will be compatible with existing programs.<     When the sender has no more data, it sends an <eot>, and; awaits an <ack>, resending the <eot> if it doesn't get one. = Again, the protocol could be receiver-driven, with the sender 5 only having the high-level 1-minute timeout to abort.   6 -------- 5. DATA FLOW EXAMPLE INCLUDING ERROR RECOVERY  = Here is a sample of the data flow, sending a 3-block message. = It includes the two most common line hits - a garbaged block, 9 and an <ack> reply getting garbaged.  <xx> represents the  checksum byte.   SENDER					RECEIVER  				times out after 10 seconds,  			<---		<nak> <soh> 01 FE -data- <xx> ---> 			<---		<ack>/ <soh> 02 FD -data- xx	--->	(data gets line hit)  			<---		<nak> <soh> 02 FD -data- xx	---> 			<---		<ack> <soh> 03 FC -data- xx	--->#    (ack gets garbaged)  <---		<ack> ! <soh> 03 FC -data- xx	--->		<ack>  <eot>			---> 			<---		<ack>   -------- 6. PROGRAMMING TIPS.   : * The character-receive subroutine should be called with a8 parameter specifying the number of seconds to wait.  The? receiver should first call it with a time of 10, then <nak> and  try again, 10 times.9   After receiving the <soh>, the receiver should call the = character receive subroutine with a 1-second timeout, for the > remainder of the message and the <cksum>.  Since they are sent< as a continuous stream, timing out of this implies a serious? like glitch that caused, say, 127 characters to be seen instead  of 128.   = * When the receiver wishes to <nak>, it should call a "PURGE" = subroutine, to wait for the line to clear.  Recall the sender 9 tosses any characters in its UART buffer immediately upon ; completing sending a block, to ensure no glitches were mis-  interpreted.6   The most common technique is for "PURGE" to call the< character receive subroutine, specifying a 1-second timeout,? and looping back to PURGE until a timeout occurs.  The <nak> is . then sent, ensuring the other end will see it.  ; * You may wish to add code recommended by John Mahr to your < character receive routine - to set an error flag if the UART< shows framing error, or overrun.  This will help catch a few= more glitches - the most common of which is a hit in the high = bits of the byte in two consecutive bytes.  The <cksum> comes ; out OK since counting in 1-byte produces the same result of * adding 80H + 80H as with adding 00H + 00H.  " -------- 7. OVERVIEW OF CRC OPTION  F The CRC used in the Modem Protocol is an alternate form of block checkF which provides more robust error detection than the original checksum.A Andrew S. Tanenbaum says in his book, Computer Networks, that the F CRC-CCITT used by the Modem Protocol will detect all single and doubleF bit errors, all errors with an odd number of bits, all burst errors ofA length 16 or less, 99.997% of 17-bit error bursts, and 99.998% of  18-bit and longer bursts.   F The changes to the Modem Protocol to replace the checksum with the CRCB are straight forward. If that were all that we did we would not beE able to communicate between a program using the old checksum protocol E and one using the new CRC protocol. An initial handshake was added to E solve this problem. The handshake allows a receiving program with CRC D capability to determine whether the sending program supports the CRCB option, and to switch it to CRC mode if it does. This handshake isD designed so that it will work properly with programs which implement> only the original protocol. A description of this handshake is presented in section 10.  2 -------- 8. MESSAGE BLOCK LEVEL PROTOCOL, CRC MODE  3  Each block of the transfer in CRC mode looks like: ; <SOH><blk #><255-blk #><--128 data bytes--><CRC hi><CRC lo> 
     in which:  <SOH>		  = 01 hex> <blk #>     = binary number, starts at 01 increments by 1, and$ 	      wraps 0FFH to 00H (not to 01)' <255-blk #> = ones complement of blk #. E <CRC hi>    = byte containing the 8 hi order coefficients of the CRC. E <CRC lo>    = byte containing the 8 lo order coefficients of the CRC. 0 	      See the next section for CRC calculation.   -------- 9. CRC CALCULATION   1 ---- 9A. FORMAL DEFINITION OF THE CRC CALCULATION   E To calculate the 16 bit CRC the message bits are considered to be the > coefficients of a polynomial. This message polynomial is first? multiplied by X^16 and then divided by the generator polynomial C (X^16 + X^12 + X^5 + 1) using modulo two arithemetic. The remainder D left after the division is the desired CRC. Since a message block inD the Modem Protocol is 128 bytes or 1024 bits, the message polynomialB will be of order X^1023. The hi order bit of the first byte of theE message block is the coefficient of X^1023 in the message polynomial. = The lo order bit of the last byte of the message block is the - coefficient of X^0 in the message polynomial.   0 ---- 9B. EXAMPLE OF CRC CALCULATION WRITTEN IN C   /*= This function calculates the CRC used by the "Modem Protocol" @ The first argument is a pointer to the message block. The secondA argument is the number of bytes in the message block. The message 4 block used by the Modem Protocol contains 128 bytes.C The function return value is an integer which contains the CRC. The E lo order 16 bits of this integer are the coefficients of the CRC. The 8 The lo order bit is the lo order coefficient of the CRC. */  . int calcrc(ptr, count) char *ptr; int count; {       int crc, i;        crc = 0;     while(--count >= 0) {  	crc = crc ^ (int)*ptr++ << 8; 	for(i = 0; i < 8; ++i)  	    if(crc & 0x8000)  		crc = crc << 1 ^ 0x1021;	 	    else  		crc = crc << 1;  	}     return (crc & 0xFFFF);     }   ; -------- 10. FILE LEVEL PROTOCOL, CHANGES FOR COMPATIBILITY   - ---- 10A. COMMON TO BOTH SENDER AND RECEIVER:   D The only change to the File Level Protocol for the CRC option is theD initial handshake which is used to determine if both the sending andF the receiving programs support the CRC mode. All Modem Programs should@ support the checksum mode for compatibility with older versions.E A receiving program that wishes to receive in CRC mode implements the F mode setting handshake by sending a <C> in place of the initial <nak>.F If the sending program supports CRC mode it will recognize the <C> andE will set itself into CRC mode, and respond by sending the first block @ as if a <nak> had been received. If the sending program does notA support CRC mode it will not respond to the <C> at all. After the D receiver has sent the <C> it will wait up to 3 seconds for the <soh>D that starts the first block. If it receives a <soh> within 3 secondsE it will assume the sender supports CRC mode and will proceed with the C file exchange in CRC mode. If no <soh> is received within 3 seconds D the receiver will switch to checksum mode, send a <nak>, and proceed@ in checksum mode. If the receiver wishes to use checksum mode itF should send an initial <nak> and the sending program should respond toC the <nak> as defined in the original Modem Protocol. After the mode A has been set by the initial <C> or <nak> the protocol follows the D original Modem Protocol and is identical whether the checksum or CRC is being used.  ) ---- 10B. RECEIVE PROGRAM CONSIDERATIONS:   C There are at least 4 things that can go wrong with the mode setting 
 handshake.,   1. the initial <C> can be garbled or lost.&   2. the initial <soh> can be garbled./   3. the initial <C> can be changed to a <nak>. @   4. the initial <nak> from a receiver which wants to receive in&      checksum can be changed to a <C>.  B The first problem can be solved if the receiver sends a second <C>? after it times out the first time. This process can be repeated F several times. It must not be repeated a too many times before sendingC a <nak> and switching to checksum mode or a sending program without C CRC support may time out and abort. Repeating the <C> will also fix E the second problem if the sending program cooperates by responding as ; if a <nak> were received instead of ignoring the extra <C>.   A It is possible to fix problems 3 and 4 but probably not worth the D trouble since they will occur very infrequently. They could be fixedA by switching modes in either the sending or the receiving program C after a large number of successive <nak>s. This solution would risk  other problems however.   ) ---- 10C. SENDING PROGRAM CONSIDERATIONS.   @ The sending program should start in the checksum mode. This willE insure compatibility with checksum only receiving programs. Anytime a C <C> is received before the first <nak> or <ack> the sending program > should set itself into CRC mode and respond as if a <nak> wereF received. The sender should respond to additional <C>s as if they were> <nak>s until the first <ack> is received. This will assist theC receiving program in determining the correct mode when the <soh> is F lost or garbled. After the first <ack> is received the sending program should ignore <C>s.   / -------- 11. DATA FLOW EXAMPLES WITH CRC OPTION   1 ---- 11A. RECEIVER HAS CRC OPTION, SENDER DOESN'T   D Here is a data flow example for the case where the receiver requestsD transmission in the CRC mode but the sender does not support the CRC? option. This example also includes various transmission errors. " <xx> represents the checksum byte.   SENDER					RECEIVER  			<---		<C> 				times out after 3 seconds, 			<---		<nak> <soh> 01 FE -data- <xx> ---> 			<---		<ack>1 <soh> 02 FD -data- <xx> --->	(data gets line hit)  			<---		<nak> <soh> 02 FD -data- <xx> ---> 			<---		<ack> <soh> 03 FC -data- <xx> --->#    (ack gets garbaged)  <---		<ack>  				times out after 10 seconds,  			<---		<nak> <soh> 03 FC -data- <xx> ---> 			<---		<ack> <eot>			---> 			<---		<ack>  2 ---- 11B. RECEIVER AND SENDER BOTH HAVE CRC OPTION  D Here is a data flow example for the case where the receiver requestsD transmission in the CRC mode and the sender supports the CRC option.7 This example also includes various transmission errors. " <xxxx> represents the 2 CRC bytes.   SENDER					  RECEIVER  			  <---		  <C> <soh> 01 FE -data- <xxxx> ---> 			  <---		  <ack>5 <soh> 02 FD -data- <xxxx> --->	  (data gets line hit)  			  <---		  <nak> <soh> 02 FD -data- <xxxx> ---> 			  <---		  <ack> <soh> 03 FC -data- <xxxx> --->&    (ack gets garbaged)	  <---		  <ack>! 				  times out after 10 seconds,  			  <---		  <nak> <soh> 03 FC -data- <xxxx> ---> 			  <---		  <ack> <eot>			  ---> 			  <---		  <ack>   			-----END-----