<<< LASSIE::UCXAXP$DKA0:[NOTES$LIBRARY]UCX.NOTE;1 >>> -< DEC TCP/IP Services for OpenVMS >- ================================================================================ Note 5457.0 An SMTP README for those supportinG UCX SMTP No replies UCXAXP::ZIELONKO 3232 lines 23-APR-1997 05:29 -------------------------------------------------------------------------------- Hi, For a long time the lack of complete UCX SMTP documentation suitable for someone who really wants to know what's going on has been bugging me. To this end I have finally bitten the bullet and written up a brain dump on UCX SMTP - an SMTP_README. Alot of this stuff is mentioned in other notes here and there and every where. It contains "internals" type stuff, problem diagnopsisi type stuff, configuration information and tips. It is intended mostlly for people solving UCX SMTP problems. It got to be very long - over 3000 lines so you may want to print it out and read it in bits. As changes are made to the code this file will be updated. I have passed this information on to the UCX Doc's team to reduce and include in the UCX SMTP doc's. Karol And awaaaaaaayy we go-o-o-o-!  Date: April, 14 1997 Author: Karol Zielonko UCX-Version: V4.1 ECO 4, V4.2 SSB ------------ Contents ------------ Introduction Part 1: SMTP Overview What is SMTP? What is MIME? SMTP Protocol Overview SMTP Protocol Overview SMTP Client commands SMTP Server Replies SMTP Message Format SMTP Mail Addresses Local-part Notes Domain Notes Miscellaneous Addressing Topics Source Routing MX Record Overview MX Example 1 MX Example 2 MX Example 3 Part 2: UCX SMTP UCX SMTP's interface with VMS mail environment Format of VMS mail message Storing SMTP messages in VMS mail: Big square peg and small round hole. UCX SMTP and callable VMS mail Callable VMS mail overview UCX SMTP's use of mail$send routines of callable mail VMS mail Foreign Transport/MAIL$PROTOCOL Interface VMS mail Foreign Transport MAIL$PROTOCOL Interface UCX SMTP Control File CF Filenames - what they mean. Control file layout Address Area Header Area Text Area Components of UCX SMTP The UCX SMTP symbiont/queues The UCX SMTP Receiver The UCX SMTP MAIL$PROTOCOL code The UCX SMTP control file code UCX MX Lookup Displaying MX records with UCX Local vs. non-local SMTP mail The SMTP Alternate Gateway/Zone How UCX decides where to send non-local mail How the symbiont sends a mail using the UCX MX Lookup gateway list How the symbiont sends a mail through the alternate gateway Further enhancements in worklist for mail destined for alternate gateway Forwarding to an SMTP address UCX SMTP and non-SMTP mail SMTPfication of non-SMTP addresses Example SMTPfications When is SMTPfication necessary? Mail from non-SMTP to SMTP An example SMTPfication as simple, generic mechanism for gatewaying mail Mail from SMTP to non-SMTP Another SMTP to non-SMTP case Miscellaneous SMTP/non-SMTP issues SMTP/non-SMTP worklist items Undeliverable Mail Configuring UCX SMTP UCX SMTP Common directory Local Aliases File UCX SMTP Configuration database entry (UCX SET CONFIG SMTP) Postmaster field UCX SMTP configuration logicals Miscellaneous VMS mail notes VMS mail native support for SMTP addresses Setting forwarding to an SMTP address Miscellaneous SET FORWARD/USER tricks The UCX SMTP cent sign hack User use of the cent sign hack UCX SMTP mailing lists VMS Majordomo Miscellaneous Stuff Queue watcher program Terms  -------------------------------------------------------------------------------- Introduction -------------------------------------------------------------------------------- This document is for system managers who will be supporting VMS systems running the UCX SMTP product. It is divided into two parts. The first part gives a general overview of SMTP and the second part discusses UCX SMTP specifics. If you are already familiar with SMTP and Internet mail you may want to skim or skip the first part entirely. This document mentions "UCX worklist items". These are meant to inform and not to be taken as a commitment to do any particular item according to any time schedule.  -------------------------------------------------------------------------------- Part 1: SMTP Overview -------------------------------------------------------------------------------- This part gives a general overview of SMTP and Internet mail. It provides information that is not specific to any operating system or implementation of SMTP. It is a summary, a primer, an overview. It does not cover every detail. Those who wish to know more are urged to look at the RFCs. ---------------------------------------- What is SMTP? ---------------------------------------- SMTP (for Simple Mail Transfer Protocol) is the Internet's mail transfer protocol. The protocol itself is specified in RFC 821. The format for the text of a mail message carried by an SMTP MTA is specified in RFC822. These RFCs are amended by RFC 1123 Chapter 5. ---------------------------------------- What is MIME? ---------------------------------------- MIME (for Multipurpose Internet Mail Extensions) specifies an extension to RFC 822 to allow for mail message content other than simple 7 bit ASCII. It is specified in RFCs 1521, 1522 and 1523. A MIME message is by definition RFC 822 compliant. UCX SMTP currently has no special handling for MIME content and doesn't need to since UCX SMTP is an MTA and creation/interpretation of mail message content is not the job of an MTA. Therefore, discussion of MIME is outside the scope of this document. ---------------------------------------- SMTP Protocol Overview ---------------------------------------- SMTP is a store and forward mail protocol which means that mail is routed** from the originating host through 0 or more intermediate hosts before it reaches its final destination host. Contrast this with DECnet's MAIL11 protocol where mail is sent directly from the originating node to the destination node. ** This document uses the terms "route" and "relay" interchangably to refer to SMTP application layer transfer of mail messages from one host to the next to the next, etc. The term routing does not denote IP layer routing. Some advantages of a store and forward mail system are: Mail can be sent between hosts that cannot connect directly to one another, for example when the originating and destination hosts are on opposite sides of a firewall. If the destination host is not available when mail is sent to it the mail can be queued up on one of the intermediate hosts (or the originating host). The MTA on the host where the mail is queued up will retry periodically and when the destination host is available again the mail will go through. Conventionally MTA's will continue to try to send a mail for some locally configurable period of time after which the mail is bounced as undeliverable. One advantage of a non store and forward mail system is that the sending user knows immediately whether or not a mail message has been delivered to the destination. This feature however can also be provided by SMTP if both the originating and destination hosts support the Return-Receipt: message header. Intermediate hosts need not support Return-Receipt. Currently UCX SMTP doesn't support Return-Receipt but it is in the worklist. The SMTP protocol is a set of simple ASCII commands and replies. A TCP connection is requested by the SMTP client and accepted by the SMTP server listening on port 25. The SMTP client sends the commands and the server replies to them. The SMTP protocol uses TCP as its transport protocol and an SMTP server listens on port 25. -------------------- SMTP Client commands -------------------- There are five commands which are essential for the transmission of SMTP mail. They are "HELO", "MAIL FROM", "RCPT TO", "DATA" and "QUIT". Command descriptions follow. The commands appear in upper case and the command parameters (where applicable) in lower case. HELO domain domain The name of the sending (ie. client) host. With the HELO command the client identifies itself to the server. MAIL FROM: The mail address to which the mail should be bounced if it cannot be delivered. When the mail is delivered to the destination host mailbox the address in the reverse-path becomes the Return-Path mail header. The angle brackets are required. With the MAIL FROM command the client tells the server where to bounce the mail. Only one MAIL FROM command per mail message is allowed. It *is* allowed for the MAIL FROM to be "null". That is, "MAIL FROM:<>". This is required when a mail message is bounced to prevent bounced mail about bounced mail. That is, when an SMTP MTA tries to deliver a mail message and can't do so the message must be "bounced" or sent back to the sender. The MTA that bounces the mail creates a bounced mail message and the MAIL FROM address used in the SMTP protocol dialog when sending this bounced mail is MAIL FROM:<>. This indicates to any SMTP MTA that receives the bounced mail message that it canot be further bounced. How that SMTP MTA deals with the situation where it has a message with a null return path (ie. a null MAIL FROM address) is up to the SMTP MTA. RCPT TO: A mail address to which the mail message should be delivered. The angle brackets are required. With the RCPT TO commands the client tells the server where to deliver the mail. For cases where mail is to be sent to multiple recipients there must be one RCPT TO command sent for each recipient. That is, the RCPT TO command cannot contain multiple addresses. Note that although the mail message text itself may contain a To: header (or other destination type header, eg Resent-To:) it is the address(es) in the RCPT TO command(s) that the SMTP MTA sends the mail to. DATA The DATA command signals the end of the RCPT TO commands and tells the server to prepare to receive the mail message itself. When the client sends the DATA command the server returns a 354 Intermediate reply (see "SMTP Server Replies"). At this point the client sends the lines of the mail message. The end of each line of text is indicated by a . The client indicates the end of the mail message by sending a '.' on a line by itself (.). Note that the '.' and the following are not part of the mail message itself. They simply constitute an SMTP protocol "end of file" marker. Once the SMTP server receives the . indicating the end of the mail message it must acknowledge the receipt of the message and accept responsibility for its delivery. It does so by returning a "250 OK" reply. When the client receives this reply it knows that responsibility for delivery of the mail now belongs to the server. At this point the client is free to send another MAIL FROM command for the next mail message or to send a QUIT command. QUIT With the QUIT command the client indicates it has no more commands to send. On receipt of this command the server must send an OK reply and close the connection. The other commands described in RFC 821 are either non-essential or inconsequential to our discussion of SMTP here. Miscellaneous Protocol Notes: - The SMTP client commands must end in . - The SMTP server replies must end in . - Text lines within the mail message must end in . - Each text line in the mail message has a maximum length of 1000 characters including the . - The client must send the commands in this order - HELO, MAIL FROM, RCPT TO and DATA. - The addresses in the MAIL FROM and RCPT TO commands make up what is often refered to as the "envelope". As stated above the addresses in these commands hold the real destination and return (ie. bounce) addresses for the mail. An SMTP MTA must save this information outside of the message itself. It is not required that the addresses from the envelope match any addresses in the headers sent with the mail. Take the case of a mailing list where the RFC To: header is the name of the list itself but the RCPT TO addresses are the names of the individual recipients. The From: field is the username of the sender and the Reply-To field is the list itself. But the MAIL FROM address is the username of the "owner" of the mailing list so that if any of the recipients bounce the owner of the list (not the sender or the list itself) receives the bounced mail. - The Return-Path header is special in one respect that bears repeating. No Return-Path header is ever sent via an SMTP protocol dialog in the SMTP message headers. The Return-Path header is manually added into the list of headers when the message reaches its final destination. When depositing an SMTP mail message into the destination mailbox the destination host SMTP MTA manually constructs a Return-Path header using the address of the MAIL FROM command and puts that header into the mail message's headers. In the case where a message is being routed by an SMTP MTA to another host the MAIL FROM address is propagated as it was received when the mail came in. -------------------- SMTP Server Replies -------------------- Each SMTP command elicits a reply from the SMTP server. The format for the SMTP server reply is "nnn text" where "nnn" is a string of three ASCII digits known as the reply code. "text" is free form ASCII text for the convenience of humans. The contents of text are not specified by any RFC. The individual digits in the reply code each have their own significance and are interpreted separately. Only the first digit must be interpreted by the client. It has four legal values - 2, 3, 4 and 5. The meanings of the first digit of the reply code follow: Value Meaning Comments 2 Postive Completion Requested action completed successfully 3 Positive Intermediate Requested action accepted. More info needed from client. ** 4 Transient Negative Completion Request failed due to temporary error condition. Eg. Disk full. Client should try again later. 5 Permanent Negative Completion Request failed due to permanent error condition. Eg. No such user as the user specified in the RCPT TO address. Client should bounce the mail message. ** value 3 is a special case. It is sent only for the DATA command. The "additional info" required is the text of the mail message itself. Reply codes are explained in detail in Appendix E of RFC 821. ---------------------------------------- SMTP Message Format ---------------------------------------- RFC 822 specifies the format for an SMTP mail message. (ie. the format of the data sent after the SMTP DATA client command). In a nutshell a mail message consists of mail headers followed by a blank line followed by the mail body. (The body is the text of the message.) Mail headers are fixed format text lines which contain addressing, control, diagnostic, identification and other miscellaneous information about the mail message. All header lines must conform to a general header syntax specified by RFC 822. Each header has a field name followed by a ":" followed by a field body followed by a . For example with the header Date: Fri, 28 Mar 1997 09:46:15 -0500 The field name is "Date". The field body is "Fri, 28 Mar 1997 09:46:15 -0500". If the body of the header is to be extended accross multiple lines of text a may be inserted into the field body followed by at least one "linear white space" (TAB, space) character. An example of a multi line header is: Received: by mysystem.someplace.dec.com with Microsoft Mail id <01BC3A8A.16162860@asystem.someplace.dec.com>; Thu, 27 Mar 1997 08:37:20 -0500 Most header bodies have additional syntactic structure imposed on them by RFC 822. The body of the Received: header above is a good example of such a complex syntax. Note that the at the end of lines is required by the SMTP protocol. However, no requirement is made on how the mail is actually *stored* in the local mailstore on the destination host. If a mail system stores the mail with as the end of line character it must change the 's to 's when transmitting the mail via SMTP. ---------------------------------------- SMTP Mail Addresses ---------------------------------------- As we have seen SMTP addresses appear in some of the SMTP client commands. They also appear in many of the SMTP message headers. Understanding SMTP addressing is essential for the diagnosis and resolution of mail problems. RFC 822 specifies a comprehensive sytnax for SMTP addresses. An SMTP address consists of two parts - the local-part and the domain. Syntactically they are separated by an at sign (@). Ie. local-part@domain. local-part The local-part identifies a local delivery point for a mail message. It is sometimes refered to as the user, username or left hand side. It is also refered to by RFC 822 as the "domain-dependent string" since its significance depends on what is understood by the MTA serving the domain specified in the domain of the address. It should be interpreted only by the final destination host not by any intermediate hosts through which the mail is routed. domain The domain is sometimes refered to as the right hand side or the hostname. It specifies either a hostname (or an IP address) where the recipient's mailbox resides or an MX destination domain where the recipients mailbox is served. (See "MX Record Overview".) This part of the address is used by the originating host and any intermediate hosts to decide where to route the mail. -------------------- Local-part Notes -------------------- Quoted and unquoted local parts: The syntax allowed for a local-part allows a local part to take one of two forms - a quoted string or an unquoted string. A quoted local part begins and ends with a quotation mark (aka. double quote (") ). A local part that is not quoted does not. Here are some example addresses with unquoted local parts: user@xyzcorp.com user.name@xyzcorp.com Here are some example addresses with quoted local parts: "SUMNOD::BORIS"@xyzcorp.com "XYZMTS::MRGATE::\"XYZMTS::MRGATE::NATASHA\""@xyzcorp.com Rules for quoting the local part: RFC 822 defines a set of characters which it calls "special characters". They are: ()<>@,.;:\"[] If the local part needs to contain any one of these characters except a dot (.) then it MUST be quoted. (The dot character is a special character but it is allowed in an unquoted local part. This syntactical exception is what allows an address like a.b@c.d.) For example, if the SMTP MTA on xyzcorp.com wants to create an address with the local part being a DECnet mail address SUMNOD::BORIS it must quote the local part. Eg: "SUMNOD::BORIS"@xyzcorp.com because the local part contains special characters - namely colons. Nested double quotes in local part: Say the SMTP MTA on xyzcorp.com needed to make an address with the local part being an MTS address of XYZMTS::MRGATE::"XYZMTS::MRGATE::NATASHA" Not only does this address contain the colons but it also contains double quotes. This raises the issue of how double quotes are nested. RFC 822 specifies a 'C' like "escape" mechanism to allow for nested double quotes in quoted strings. To nest a double quote in a quoted string the nested double quote is preceded with a backslash character (\). So, given the desired local part above, the address created by the MTA on xyzcorp.com would be: "XYZMTS::MRGATE::\"XYZMTS::MRGATE::NATASHA\""@xyzcorp.com UCX SMTP in fact does create such complicated address as the ones in this section. The reasons for this are discussed in "UCX SMTP and non-SMTP mail". -------------------- Domain Notes -------------------- Syntactically the domain part takes one of the following two forms: 1) A DNS type hierarchical name as in "something.xyzcorp.com" in this address: jones@something.xyzcorp.com 2) An IP address as in "[1.2.3.4]" in this address: jones@[1.2.3.4] The use of IP addresses in SMTP mail addresses is allowed but strongly discouraged. The square brackets are required. -------------------- Miscellaneous Addressing Topics -------------------- --------------- Source Routing --------------- SMTP allows (although discourages) two ways for the sending mail user to specify a particalur route for their mail to take as it goes from the originating host to the destination host. This capability is called source routing. One syntax is provided for explicitly in RFC 821. The other is an alternative that is a defacto standard. RFC 821 source routing: RFC 821 and 822 provide for a syntax for source routing: @a,@b:username@c where a, b, and c are hosts. With such an address the final destination for the mail is host c but it first must go to host a and then to host b. Percent sign hack source routing: A defacto standard for source routing is also generally supported by most SMTP MTA's. It is called the %-hack. Its syntax is: username%c%b@a where a, b, and c are hosts. With such an address the final destination for the mail is host c but it first must go to host a and then to host b. Ie. this example accomplishes the exact same thing as the first. Both of these syntaxes are supported by UCX SMTP. ---------------------------------------- MX Record Overview ---------------------------------------- SMTP MTA's make use of DNS MX records to assist them in the routing of mail messages. An MX record tells an MTA where to route mail for a particular destination domain. The DNS maintains MX records but SMTP MTA's *use* them. An MX record has the following fields: destination domain This field matches the domain part of the address to which a mail is going. It is essentially the "key" field of the MX record. For example, if mail is to be sent to jones@xyzcorp.com an MX lookup is done on the destination domain "xyzcorp.com". Multiple MX records for the same destination domain are allowed. Taking the key analogy further this is like saying the destination domain field allows duplicate keys. gateway hostname The name of the host to which mail sent to the destination domain should be routed. preference Prioritizes multiple MX records for the same destination domain. The lower the preference value the higher the priority for the MX record. That is, lower preference MX records are tried before higher preference MX records. It is allowed to have multiple MX records for the same destination domain with the same preference. Some advantages of multiple MX records for the same destination domain are: 1) They enable load balancing between mail routers. This is the case where there are multiple MX records for the same destination domain with the same preference. 2) They allow one mail router to be taken down with another left up so that mail can continue to go through. 3) They provide MX based routes for mail internal and external to a firewall. -------------------- MX Example 1 -------------------- For example, imagine the following three MX records for the destination domain "xyzcorp.com": Destination domain: Gateway: Preference: xyzcorp.com mailhub1.xyzcorp.com 10 xyzcorp.com mailhub2.xyzcorp.com 10 xyzcorp.com otherhost.xyzcorp.com 20 Given these MX records, any SMTP MTA trying to send mail to the destination domain xyzcorp.com using MX records should first try to send the mail to either mailhub1.xyzcorp.com or mailhub2.xyzcorp.com. If that fails the MTA should try the other of the two "mailhub" hosts. If that too fails it should try otherhost.xyzcorp.com. If that fails the MTA should requeue the mail message in its internal queueing mechanism and retry later. The MX records in this example provide for the advantages of multiple MX records listed above as number 1 and 2: 1) The fact that mailhub1 and mailhub2 are at the same MX preference provides mail load balancing between the two hosts. This is because the DNS returns MX records in a different order each time so that records with the same preference are not always seen in the same order by the SMTP MTA. 2) Since there are multiple MX records for the domain xyzcorp.com, mail sent to xyzcorp.com can still be relayed from a sending host to one of the MTA's within the xyzcorp network as long as any of the xyzcorp hosts are alive. Mailhub1 can be taken down for maintenance and mail will still continue to flow through mailhub2. Given the example MX records it is assumed that at least the lowest preference mail hubs for the xyzcorp.com domain (ie. mailhub1 and mailhub2) are configured internally to know how to deliver mail to each user in the xyzcorp.com domain. For example user jones@xyzcorp.com mail reside on host zzz.xyzcorp.com. Something on the mailhub system has to exist so that it knows to send mail to jones@xyzcorp.com to jones on the zzz.xyzcorp.com host. It could be that xyzcorp's network is small enough so that this can be accomplished with a simple alias file for all corporate users. (That would be a SET FORWARD/USER=JONES SMTP%"""jones@zzz.xyzcorp.com""" for a VMS system.) Otherwise more complex mechanisms will need to be configured. These mechanisms depend on the SMTP implementation running on the hosts and so are beyond the scope of this document. (The creation of additional MX records as in MX Example 3 is useful in getting mail from SMTP mail hubs to internal hosts on a large network.) -------------------- MX Example 2 -------------------- What if both mailhub1 and mailhub2 were down? In this case remote MTA's will try and fail to send the mail to the mailhub hosts and will succeed in sending the mail to otherhost.xyzcorp.com. One of two possible scenarioes will occur depending on how otherhost is configured: 1) If otherhost is configured internally to know how to deliver mail to each user in the xyzcorp.com domain (as are mailhub1 and mailhub2) it will deliver mail using this mechanism without further use of MX records. 2) If otherhost is not configured to know how to deliver mail to each user in the xyzcorp.com domain it will act merely as a holding place for all mail sent to xyzcorp.com. It will continually retry to route the mail to mailhub1 and mailhub2. As long as both of these systems are down it will continue to requeue and retry the mail. Eventually either one of the mailhub hosts will become available again and the mail will be routed there or the retry time limit on otherhost will expire and the mail will bounce. -------------------- MX Example 3 -------------------- In this example we switch gears from looking at imaginary xyzcorp's MX records to Digital's own MX record setup. Here we illustrate the third advantage listed of multiple MX records for the same destination domain which has to do with firewall issues. We also illustrate a use of MX records that allows a setup where corporate mailhubs do not have to know where each individual user lives thus avoiding the problems mentioned at the end of Example 1. Here are the MX records for the SMTP destination domain ucx.lkg.dec.com: Destination domain: Gateway: Preference: ucx.lkg.dec.com ucxaxp.ucx.lkg.dec.com 50 ucx.lkg.dec.com mail2.digital.com 100 ucx.lkg.dec.com mail1.digital.com 100 ucx.lkg.dec.com crl.dec.com 200 SMTP mail internal to corporate firewall: Say user boris on host mrbig.ucx.lkg.dec.com sends mail to jones@ucx.lkg.dec.com. Assuming the SMTP MTA on host mrbig uses MX records it will first try to route the mail to ucxaxp.ucx.lkg.dec.com since that system is in sole possesion of the lowest preference (ie. highest priority) MX record for the destination domain of ucx.lkg.dec.com. Since both systems are within the corporate firewall the smtp connection from mrbig to ucxaxp can be made and the mail is transfered to ucxaxp. Once there it is delivered as local mail to user jones. Note: ucxaxp is a VMS UCX system serving mail for the domain ucx.lkg.dec.com. See "Configuring UCX SMTP" and "Local vs. non-local SMTP mail" for more information on how UCX SMTP can be configured to accept mail as local for a domain that isn't the same as the UCX hostname.) SMTP mail coming from outside corporate firewall: Say user bullwinkle on host moose.wassamadda-u.edu sends mail to rocky@ucx.lkg.dec.com. Assuming the SMTP MTA on host moose uses MX records it will first try to route the mail to ucxaxp.ucx.lkg.dec.com just as mrbig.ucx.lkg.dec.com does. However, in this case connection to ucxaxp fails because the two hosts are on opposite sides of the Digital firewall. Once the connection to ucxaxp fails the next MX record is tried. In this case the next MX record will be for either mail2.digital.com or mail1.digital.com since they both have the same preference level. Which one is tried depends on which MX record is returned first from DNS. The connection works because mail1 and mail2 can accept SMTP connections from outside the firewall. At this point say the connection is to mail1.digital.com and that the mail is transfered successfully. Once the transfer is complete the SMTP MTA on mail1.digital.com repeats the MX lookup process from the start. It sees that the lowest preference MX record for mail addressed to the domain ucx.lkg.dec.com is for ucxaxp.ucx.lkg.dec.com and so tries to connect to that host. This time the direct connaction to ucxaxp works because mail1 can connect to hosts within the firewall. So, the mail is transfered to ucxaxp and delivered locally to uses jones as with the previous example. Note: This example assumes that moose.wassamadda-u.edu is not inside its own wassamadda-u firewall. However if it is inside a firewall then MX records alone won't work since moose.wassamadda-u.edu won't be able to connect to any hosts in the MX records for ucx.lkg.dec.com. In this case the moose host must be configured to send mail destined to the world outside its firewall to some host similar to mail1.digital.com that *can* connect directly to the outside world. How this is done on host moose depends on the SMTP implementation running there. See "Configuring UCX SMTP" and "How UCX decides where to send non-local mail" for information on how to do this for UCX SMTP. These two examples show how MX records can be used to create different mail routes depending on whether mail is within or accross the boundaries of firewalls. It also demonstrates one way that a mail hub straddling a firewall can be used to accept inbound mail from outside the firewall and route it to the host inside the firewall that knows how to deliever it without the mailhub knowing anything about individual users. MX Record Notes: - The fields of MX records and handling of them are not case sensitive. - An MX destination domain may be a DNS host name but it is not requried to be. - For information on how to display MX records using UCX see "Displaying MX records with UCX".  -------------------------------------------------------------------------------- Part 2: UCX SMTP -------------------------------------------------------------------------------- This section is intended for - The VMS system manager who is tasked with maintaining a system running UCX SMTP. - Anyone tasked with diagnosing problems with UCX SMTP. The types of information included are: - VMS implementation details and UCX SMTP implementation details where they may help problem solvers diagnose problems through understanding of underlying operations. - Handy hints and tricks - Configuration suggestions - Examples to illustrate points ---------------------------------------- UCX SMTP's interface with VMS mail environment ---------------------------------------- As a VMS based SMTP MTA UCX SMTP has an interface with the VMS mail environment. It uses this interface to: - Deliver a mail message to VMS mail coming from SMTP - Receive a mail message from VMS mail to send out via SMTP It uses two distinct VMS mail interfaces to accomplish these tasks. To deliver mail from SMTP to VMS mail it uses what is called "callable VMS mail". To accept a mail message from VMS mail to send out via SMTP UCX SMTP implements a VMS mail "foreign transport" using the the undocumented MAIL$PROTOCOL interface. This section discusses these two interfaces and details about UCX SMTP's use of them and VMS mail. -------------------- Format of VMS mail message -------------------- Just as an SMTP messages has a format so does a VMS mail message. Unlike SMTP messages which have a rich set of headers VMS mail messages have only a small set of headers. They are From, To, Subj and CC. As with SMTP the headers of a VMS mail message are separated from the body of the VMS mail message by a single blank line. In the case of an SMTP mail message delivered to VMS mail the body of the VMS mail message is the entire text of the SMTP mail message - the SMTP headers followed by blank line followed by the SMTP message body.** ** This assumes that UCX SMTP has been configured with /OPTION=TOP_HEADERS. See "Configuring UCX SMTP". --------------- Storing SMTP messages in VMS mail: Big square peg and small round hole. --------------- Because of VMS mail's lack of support for the rich set of headers provided for by SMTP messages there is bound to be a "loss of fidelity" when an SMTP message is passed to VMS mail. The biggest loss of fidelity is with the "originator" headers from the SMTP mail message. SMTP messages can have many different "originator" type header fields - From, Reply-To, Return-Path - each with a different purpose. However VMS mail supports only one originator type header, the From header, so when an SMTP message is passed to VMS mail UCX SMTP has to make a choice of which of the SMTP originator fields to pass to VMS mail's From header. UCX uses a priority list of all the possible SMTP originator headers to do this. When deciding which of the originator headers to use in the VMS mail From line UCX SMTP starts at the top of the priority list scanning the SMTP message headers for the particular header from the priority list. If the header from the priority list is is found in the message that header is used in the VMS mail From line. If not the scan goes to the next header from the priority list. The priority list from first choice to last choice is: Resent-Reply-To Resent-From Reply-To From Resent-Sender Sender Return-Path So, for example, if an SMTP mail message had a Reply-To, a From and a Return-Path header the VMS mail from line would be taken from the Reply-To header since Reply-To appears first in the priority list. -------------------- UCX SMTP and callable VMS mail -------------------- --------------- Callable VMS mail overview --------------- VMS provides a suite of routines collectively refered to as "callable mail". The routines are divided into four groups. There is one group of routines for manipulating mail files, one for manipulating existing messages stored in folders within a particular mailfile, one for getting and setting user profile information and one for delivering a new message. Each of the four groups of routines has a routine name prefix which all of the routines in the group start with. They are: Prefix: Purpose: mail$mailfile_... Manipulate mailfiles mail$message_... Manipulate existing messages in folders mail$user_... Get/set user profile information mail$send_... Deliver new mail to VMS mail To deliver mail from the SMTP world into the VMS mail world UCX SMTP uses the mail$send routines. UCX SMTP uses the mail$user routines to see the forwarding string of a local user. (For more information on this see "Forwarding to an SMTP address".) Callable VMS mail is documented fully in the VMS Utility Routines Reference Manual. --------------- UCX SMTP's use of mail$send routines of callable mail --------------- The mail$send routines provide a series of calls to pass a mail message to VMS mail. The mail$send routines of interest are: Routine: Use: mail$send_add_address Passes the address of a recipient for the VMS mail message. mail$send_add_attribute Passes the text of the four VMS mail header lines for the mail message. There is one separate input item for each of the four header lines - From, To, Subj, CC. Note that the text of the To line as passed in the call to mail$send_add_attribute is *not* the same as the address passed to mail$send_add_address. The text of the To line is uninterpreted by VMS mail. mail$send_add_bodypart Passes a single line of mail text to VMS mail. Called in a loop. mail$send_message Tells VMS callable mail that text of message is complete and that it should now deliver the message. Example: Say UCX system ucxsys.xyzcorp.com is delivering a mail from natasha@wassamadd-u.edu sent to boris@ucxsys.xyzcorp.com. To pass the mail to VMS mail to be delivered UCX SMTP on ucxsys performs the following VMS callable mail calls: - Calls mail$send_add_address passing the address "boris". - Calls mail$send_add_attribute passing the the text of the SMTP message's To, CC (if any), and Subject field verbatim to VMS mail for the text of the respective VMS mail header lines. - Decides which originator field to use (see "Storing SMTP messages in VMS Mail") for the VMS mail From line. Puts the SMTP% jacket around the address to make it legal for a VMS mail reply and passes it to mail$send_add_attribute as the text of the VMS mail from line. - Calls mail$send_add_bodypart for each text line of the SMTP message which includes the RFC 822 message headers, a blank line and the message text. - Calls mail$send_message to tell VMS mail to deliver the message to the user. -------------------- VMS mail Foreign Transport/MAIL$PROTOCOL Interface -------------------- Up to this point we've discussed how SMTP mail coming into UCX SMTP is delivered to VMS mail but not how mail is composed on VMS mail and sent out via UCX SMTP. The mechanism to do this is called a "foreign transport". Foreign transports are implemented using the MAIL$PROTOCOL interface. --------------- VMS mail Foreign Transport --------------- The VMS mail foreign transport mechanism allows mail to be sent over a mail protocol other than Mail11. Foreign transport addresses begin with a special prefix to identify them as such to the VMS mail address parser. It is a string, which is the identifier for the foreign transport, followed by a percent sign (%). After this comes the mail address itself in the syntax of the foreign transport. The identifier for the UCX SMTP foreign transport is the string "SMTP". So to send a mail via UCX SMTP from VMS mail the address entered in the VMS mail To: must start with SMTP%. Because of VMS mail address parsing issues there is also an extra set of double quotes needed** around an SMTP address so the whole thing looks like: MAIL> send To: SMTP%"jones@xyzcorp.com" ** See "VMS mail address parsing notes" --------------- MAIL$PROTOCOL Interface --------------- When VMS mail sees a mail addressed to a foreign transport it needs to pass the address to the foreign transport mechanism along with the mail message so that the foreign transport can send it out. VMS mail does this using the MAIL$PROTOCOL interface which is an undocumented, unsupported calling interface between VMS mail and a foreign transport. It is implemented as a routine called MAIL$PROTOCOL which resides in an image provided as part of the foreign transport mechanism. When VMS mail sees a mail message to a foreign transport it goes through a search process to locate the MAIL$PROTOCOL routine for the foreign transport and then goes through a sequence of calls to the MAIL$PROTOCOL routine to pass the address(es) and VMS header lines and message text along to the foreign transport's MAIL$PROTOCOL routine. MAIL$PROTOCOL search: When VMS mail sees a mail message to a foreign transport it needs to find the foreign transport's MAIL$PROTOCOL routine. As a first resort it looks for an image file SYS$LIBRARY:xxx_MAILSHR.EXE where "xxx" is the foreign transport string. Eg. SYS$LIBRARY:SMTP_MAILSHR.EXE. If it finds such an image it looks for a routine named MAIL$PROTOCOL in the image. If it finds it then the search is over. If no xxx_MAILSHR file is found or it doesn't contain a MAIL$PROTOCOL routine VMS mail will attempt to translate a logical MAIL$PROTOCOL_xxx where "xxx" is the foreign transport string. If it finds one then the equivalence name of the logical must be the filename (directory defaults to SYS$LIBRARY, file extension defaults to .EXE) of the image file where the MAIL$PROTOCOL routine is to be found. UCX SMTP uses this mechanism to point to its MAIL$PROTOCOL routine in SYS$LIBRARY:UCX$SMTP_MAILSHR.EXE. If no MAIL$PROTOCOL_xxx logical is found or the file specified doesn't contain a MAIL$PROTOCOL routine then VMS mail signals an error message: %MAIL-E-ERRACTRNS, error activating transport xxx MAIL$PROTOCOL Calling sequence: Once the MAIL$PROTOCOL routine is found and loaded VMS mail calls it repeatedly to pass the mail information to the foreign transport. Each call to the MAIL$PROTOCOL routine passes a function code to indicate what is to be done, a pointer to a context block (created by the MAIL$PROTOCOL routine on the first call) and eight parameters whose meanings vary depending on the function code. There is an "initialization" function code sent first for each message and one sent last to tell the MAIL$PROTOCOL routine to send the message. There is a function code for a recipient address, one function code each for the text from the From:, To:, Subj:, and CC: lines and a code for the text of the mail itself (a pointer to an RMS rab for the text of the messages is passed). MAIL$PROTOCOL process context: The MAIL$PROTOCOL code runs in the context of the process that sends mail to the foreign transport address. MAIL$PROTOCOL invocation with DECnet mail: Consider this example: MAIL> send To: SUMNOD::SMTP%"smith@someplace.xyzcorp.com" ... Such an address is entirely legal. It might be used from a system that has DECnet but no TCP/IP installed as a means to still send SMTP mail. In what process context is the MAIL$PROTOCOL interface being invoked in this example? When such an address is used the MAIL$PROTOCOL code is running in the context of the DECnet MAIL network object on the remote DECnet node. In this case that's SUMNOD. The same ecanrio occurs when DECnet mail is sent to a user who has mail forwarded to SMTP. This is important to know when debugging problems with mail being sent between DECnet and SMTP mail. See "UCX SMTP and non-SMTP mail". ---------------------------------------- UCX SMTP Control File ---------------------------------------- With UCX SMTP each mail message is packaged into a special purpose binary file called a control file. This control file is submitted to a generic UCX SMTP queue to be processed by the UCX SMTP symbiont. Each control file contains one and only one SMTP mail message. Note that an SMTP message addressed to multiple recipients is stored in one control file. -------------------- CF Filenames - what they mean. -------------------- The name of a control file and the directory it's in tell you something about what type of mail it is. The format for the name of a control file is: yymmddhhsshh_username.UCX_scsnode where "yymmddhhsshh" is a timestamp taken when the file is created. "scsnode" is the value of the SYSGEN SCSNODE parameter. "username" is the username of the process in which the CF was created. The different possiblities for "username" are: Username: Meaning: UCX_SMTP This is mail that came onto your system via SMTP. The control file was created by the UCX SMTP receiver process which runs in the UCX_SMTP account. MAIL$SERVER This is mail that came in over DECnet and was destined for an SMTP address. The control file was created by the DECnet Mail11 network object which runs in the MAIL$SERVER account. This happens when the username on your system to which the DECnet/MTS mail is destined has forwarded their mail to and SMTP address. SYSTEM This is one of two things. If the control file is sitting in the UCX_SMTP account's directory then it is a bounced mail message created by the UCX SMTP symbiont which runs in the SYSTEM account. Otherwise the control file will be in the default mail directory of the SYSTEM user in which case the control file is simply an SMTP mail sent by the SYSTEM user. regular user name This is mail composed by the user and sent to an SMTP address. -------------------- Control file layout -------------------- The UCX SMTP control file is a binary file that holds a single SMTP mail messages. It has separate areas for the destination address information, selected, pre-parsed RFC headers and the raw text itself. This section describes these three areas. --------------- Address Area --------------- The address area of the control file contains each of the SMTP recipient addresses. (ie. the addresses for the RCPT TO SMTP protocol command(s).) It is implemented as a linked list of linked lists. The primary list is a list of the domain parts of the addresses. Each domain node in the linked list points in turn to a list of addresses within that domain. Say a mail was to delivered to jones@xyzcorp.com, smith@xyzcorp.com and taylor@sumskool.edu. The list would look like this: list head | | V | +-------------------+ +---------------------+ +--->| xyzcorp.com |-->| sumskool.edu |-->End-of-domain-list +-------------------+ +---------------------+ domains | | --------------+-------------------------+----------- addresses | | V V +-------------------+ +---------------------+ | jones@xyzcorp.com | | taylor@sumskool.edu | +-------------------+ +---------------------+ | | V V +-------------------+ End-of-address-list | smith@xyzcorp.com | +-------------------+ | V End-of-address-list All nodes in the list (domain nodes and address nodes) also contain a block of status bits. There are bits to indicate, among other things, if the address is local or not (see "Local vs. non-local SMTP mail") , if the mail has been successfully delivered, needs to be bounced or needs to be requeued. --------------- Header Area --------------- As the mail message is built into the control file certain headers are pre-parsed into an area called the header area so that they can be accessed without reparsing the entire header block. Each header value parsed into the header area has a code to identify the name of the header (eg. FROM, TO, etc.) and the value associated with it. The headers which are parsed into the header area are: Received, Return-Path, Date, From, To, Subject, CC, Sender, Comments, Errors-To, Keywords, Reference, X-Organization, X-VMS-To, X-VMS-CC, Reply-To, Resent-Reply-To, Resent-From, Resent-Sender and Message-ID When you issue a UCX SHOW MAIL command the headers it shows you are actually those from the header region, not from the text area. This explains why headers other than those in the list above don't appear in the output of UCX SHOW MAIL. The value of each header in the header area is stored in its "unfolded" format. That is, multi-lined values are folded into one single line with all extraneous white space removed. Note about storage of the MAIL FROM address/Return-Path header: As said in the SMTP Protocol review section the "envelope" addresses (those from the MAIL FROM and RCPT TO commands) are to be stored separately from the message headers. The RCPT TO addresses are stored separate from the headers in the address area of the control file. However the MAIL FROM address is stored in the Return-Path header. Note that if a mail message is to be sent to another SMTP host the UCX SMTP client uses the address in the Return-Path header in the address region of the control file to form the MAIL FROM command. The Return-Path header is *not* sent by UCX SMTP as part of the RFC 822 message headers themselves. This is required by the RFCs since the Return-Path header is only to be added into the headers by the final destination host system. We only store it in the Return-Path header for the convenience of our own implementation. --------------- Text Area --------------- The text area contains the raw text of the mail message. This includes the headers, the blank line and the message body. In some cases the text area of the control file contains a pointer to a text file that contains the text. This text file is named exactly the same as the control file except that it's extension is ".UCX_scsnode_TEXT". ---------------------------------------- Components of UCX SMTP ---------------------------------------- This section introduces the different components of UCX SMTP. For purposes of this discussion we break them up into the symbiont, the receiver, the MAIL$PROTOCOL code, and the control file code. -------------------- The UCX SMTP symbiont/queues -------------------- As was said earlier each mail message is packaged into a special purpose binary file called a control file and submitted to a generic UCX SMTP queue. This generic queue points to one or more SMTP execution queues which run the UCX SMTP symbiont hereafter refered to simply as the symbiont. ** The symbiont receives jobs one at a time from the generic SMTP queue and delivers them either locally (via callable VMS mail ) or remotely (via SMTP). The UCX SMTP queues are set up for you. By default the generic SMTP queue points to one SMTP execution queue. You can change this with the SET CONFIGURATION SMTP/QUEUE command. See "Configuring UCX SMTP". The UCX SMTP queues are started by the UCX START MAIL command and stopped by the UCX STOP MAIL command. The SMTP satrtup file UCX$SMTP_STARTUP.COM does a UCX START MAIL and the SMTP shutdown file does a UCX STOP MAIL. The symbiont process (or processes for multiple execution queues) are created when the queues are started. The symbiont process runs in the SYSTEM account. The symbiont processes *all* mail on UCX SMTP. It sends outbound mail via SMTP and delivers inbound mail to users via callable VMS mail. It also decides when and if a mail message needs to be bounced and creates the bounced mail message and sends that mail too. None of the other UCX SMTP components actually send any mail. They package it into a control file and submit it to the symbiont. ** See your VMS documentation for more information about the VMS queueing system and symbiont processes. For those who care the UCX SMTP symbiont is a single streamed, asynchronous delivery SMB$ type symbiont. (See the VMS Utility Routines Reference Manual for information on the implementation of VMS symbionts.) -------------------- The UCX SMTP Receiver -------------------- The UCX SMTP receiver is invoked by the UCX Auxiliary Server when an inbound TCP/IP connect comes into port 25 (assuming the UCX SMTP service has been enabled). The Auxiliary server runs the COM file specified in the SMTP service database entry UCX$SMTP_RECV_STARTUP.COM - which runs the receiver. The receiver image is SYS$SYSTEM:UCX$SMTP_RECEIVER.EXE. The receiver process runs in the UCX_SMTP account. The receiver acts as the SMTP server in the SMTP client-server protocol dialog. It saves the mail message from the client in a control file and submits it to the symbiont once the message is received successfully. The receiver doesn't deliver mail itself. Nor does it validate the existence of users to whom the mail is being sent. The receiver process is single threaded and once it serves a mail message it exits. That is, for each new inbound connection to port 25 a new SMTP receiver process is created. -------------------- The UCX SMTP MAIL$PROTOCOL code -------------------- This was discussed extensively in the section on the MAIL$PROTOCOL interface. To review the MAIL$PROTOCOL code is invoked by VMS mail when a message is to be delivered to an SMTP address. It runs in the context of the process that is sending the mail. This is either a user who is sending SMTP mail from the VMS mail UI or the DECNet Mail11 network object running in the MAIL$SERVER account. The only additional information is that as the MAIL$PROTOCOL code receives the mail message from VMS mail it packages it up into a control file and submits the control file to the symbiont to deliver. -------------------- The UCX SMTP control file code -------------------- This code is called by each of the other four components. The receiver and MAIL$PROTOCOL code use it to create control files. The symbiont uses it to read control files and to create control files for bounced mail messages. The control file code contains routines (among others) to: - Create, delete, open and close a control file - Put new domain, address, RFC header or line of text to file ** - Read existing domain, address, RFC header or line of text from file - Read/write domain/address status blocks - Submit a control file to the queue The error reporting in this code is not up to snuff. 99% of the different problems they incur are reported as either a generic "control file error" or "Bad parameter". Fixing this is in the worklist. ** See "UCX SMTP Control File" for more information on the contents of the control file. ---------------------------------------- UCX MX Lookup ---------------------------------------- As discussed earlier an SMTP mailer needs to access DNS maintained MX records in order to help it route mail. UCX has a procedure called MX lookup to do this. It has some value added over a raw DNS MX lookup. This section documents the UCX MX lookup procedure. The UCX MX lookup procedure is a routine that takes as input the destination SMTP domain and returns a list of gateway hosts and preferences (well, sometimes preferences, keep reading) associated with them. The value added over a raw DNS MX lookup is twofold: - The UCX MX lookup returns MX records sorted in ascending order of preference (MX records returned higher priority before lower priority). When there are multiple MX records with the same preference their order in the return list is the same as returned by DNS. Since DNS returns records in a different order each time the records with the same preference will not appear in the same order in the UCX MX lookup sorted list each time. This is essential for proper functioning of multiple MX records with the same preference as a load balancing mechanism between peer mailhubs. For an example of this see "MX Record Overview - MX Example 1". - Four database search As said earlier the UCX MX lookup procedure returns a list of gateway hosts and preferences. What was not said is that the search for these gateway hosts extends beyond DNS MX records. The UCX MX lookup actually searches four "databases" (for lack of a better word) to resolve an MX query. Following are the four databases that are searched in the order they are searched: UCX Local MX database DNS MX lookup UCX local host database DNS host lookup As you can see the last two databases listed here are looking at host records not MX records. Given this the term "UCX MX Lookup" is a bit of a misnomer because the MX lookup is looking at something in addition to MX records. The UCX MX lookup takes the destination mail domain to be looked up and performs queries on this domain in each of the above four databases. In the case of the host databases it returns a preference of "0" since there is no preference value associated with a host record. The UCX SHOW MX command and the UCX SMTP code both call the UCX MX Lookup routine which lives in SYS$LIBRARY:UCX$ACCESS_SHR.EXE. Consequently if you want to see what MX records UCX SMTP sees for a particular non-local mail domain just use UCX SHOW MX. (See "Displaying MX records with UCX".) The MX lookup routine filters duplicate gateway names. It will never return a record for a gateway that it has already passed back to the caller. If it's already told you about a possible gateway there's no point in repeating it. Try doing a UCX SHOW MX on a hostname that has an MX record where the gateway name is the hostname itself. Notice that UCX SHOW MX shows you the MX record but not the host record since it would simply be a repeat of the MX record. -------------------- Displaying MX records with UCX -------------------- UCX supports two methods for MX lookup - the Unix based nslookup program and the UCX SHOW MX_RECORDS command. UCX nslookup hooks directly to the DNS and returns the MX records just as they are passed from the DNS - unsorted. The UCX SHOW MX command provides some additional functionality on top of the bare DNS MX lookup. See "UCX MX Lookup". The following examples show how to use UCX's nslookup and UCX SHOW MX to do an MX lookup on the destination domain "ucx.lkg.dec.com". Nslookup invocation: $ mcr ucx$nslookup Default Server: ucxaxp.ucx.lkg.dec.com Address: 16.20.208.53 > set type=MX > ucx.lkg.dec.com Server: ucxaxp.ucx.lkg.dec.com Address: 16.20.208.53 ucx.lkg.dec.com preference = 100, mail exchanger = mail2.digital.com ucx.lkg.dec.com preference = 200, mail exchanger = crl.dec.com ucx.lkg.dec.com preference = 50, mail exchanger = ucxaxp.ucx.lkg.dec.com ucx.lkg.dec.com preference = 75, mail exchanger = lassie.ucx.lkg.dec.com ucx.lkg.dec.com preference = 100, mail exchanger = mail1.digital.com ucx.lkg.dec.com nameserver = ucxaxp.ucx.lkg.dec.com ucx.lkg.dec.com nameserver = lassie.ucx.lkg.dec.com mail2.digital.com inet address = 204.123.2.56 crl.dec.com inet address = 192.58.206.2 ucxaxp.ucx.lkg.dec.com inet address = 16.20.208.53 lassie.ucx.lkg.dec.com inet address = 16.20.208.100 mail1.digital.com inet address = 204.123.2.50 > exit The nslookup MX lookup first gives you the name and IP address of the DNS server from which the information came. In the example it was ucxaxp.ucx.lkg.dec.com. Next it gives you a list of MX records for the destination domain you queried on. Note that the MX records are not sorted by preference. This is because nslookup hooks directly into DNS which presents the MX records to the caller in this unsorted fashion. Finally, as a niceity, nslookup shows you the IP addresses for the mail exchanger (ie. gateway hostname) fields. UCX SHO MX_RECORDS invocation: UCXAXP> ucx UCX> sho mx ucx.lkg.dec.com BIND MX database Server: 16.20.208.53 ucxaxp.ucx.lkg.dec.com Gate address Preference Gate name 16.20.208.53 50 ucxaxp.UCX.LKG.DEC.COM 16.20.208.100 75 lassie.UCX.LKG.DEC.COM 204.123.2.56 100 mail2.digital.com 204.123.2.50 100 mail1.digital.com 192.58.206.2 200 crl.dec.com BIND database Server: 16.20.208.53 ucxaxp.ucx.lkg.dec.com Host address Host name 16.20.208.208 UCX.LKG.DEC.COM UCX> exit Note that there are differences in the output of nslookup and UCX SHOW MX. These are due to the value added by the UCX MX Lookup. For more information see "UCX MX Lookup". ---------------------------------------- Local vs. non-local SMTP mail ---------------------------------------- To deliver mail to any address UCX SMTP needs to assess whether the address is "local" or not. If an address is local the symbiont strips off the domain part of the address and delivers the mail to the local-part of the address via VMS callable mail. If an address is non-local it is relayed via SMTP off to another host for it to deliver (or to relay further). UCX SMTP determines if an address is local when it initially parses the address and stores it into the control file address region. Once the address is parsed into a local-part and domain part the SMTP address parsing code looks at the domain part of the address to see if it is local. In order to determine if the domain part of the SMTP address is local or not the UCX SMTP parsing routine performs the following steps: - Compare the domain to the SMTP substitute domain. If the domain matches the SMTP substitute domain it is considered local. This step is not performed if: - The logical UCX$SMTP_NO_SUBS_DOMAIN_INBOUND is defined. See "Configuring UCX SMTP". - No substitute domain is configured. - Compare the domain case insensitively to the domains listed in the local aliases file. If a match occurs then it is considered local. - Build a list of all IP addresses for the domain and compare this list to a list of all of the local host's IP addresses (including any IP cluster alias). If a match occurs then the address is local. Miscellaneous notes: - The status blocks of the domain and address nodes in the linked list of the control file address area contain a "local" bit which is set if the address is local. After this any code that reads this address from the control file knows if it is local or not. - When talking about local and non-local mail it is important to remember that a mail message can be both. In the case where there are multiple SMTP recipients and one of them is local and another is not the mail is delivered locally to the local address and relayed out via SMTP to the non-local one. So strictly speaking it's not the mail that's local or non-local it's the address. By convention though when we are talking about delivering mail to a local address we call it local mail and when we are delivering mail to a non-local address we call it non-local mail. ---------------------------------------- The SMTP Alternate Gateway/Zone ---------------------------------------- The SMTP alternate gateway along with the SMTP zone provide an alternative to using straight MX records to send non-local mail. When you define an alternate gateway any mail destined for a domain outside the zone is sent to the alternate gateway. If an alternate gateway is defined and no zone is defined then the value of the UCX$INET_DOMAIN logical is used as the zone. This means that all mail to systems outside your DNS zone will be sent to the alternate gateway. ---------------------------------------- How UCX decides where to send non-local mail ---------------------------------------- This section describes how UCX SMTP decides where to send a mail to a non-local address since that has been a source of alot of confusion. There are a few things that affect where the symbiont tries to route the mail: - The alternate gateway and zone and the results of the UCX MX lookup on the alternate gateway. See "The SMTP Alternate Gateway/Zone" - The results of the UCX MX Lookup on the domain part of the recipient address. See "UCX MX Lookup". - Definition of a few select UCX SMTP configuration logicals: - UCX$SMTP_ALTGATE_ALWAYS - UCX$SMTP_MX_IF_NOALTGATE - UCX$SMTP_NO_MX See "Non-local mail routing options" under "UCX SMTP configuration logicals" for more information on these logicals. As described in the section about the alternate gateway and zone mail destined for a domain outside the zone is sent to the alternate gateway. If no alternate gateway is configured then all non-local mail is delivered using the UCX MX Lookup gateway list. (See "UCX MX Lookup"). If an alternate gateway is configured but the domain of the recipient address is within the zone then the message is sent to that address using the UCX MX Lookup gateway list. Modifications to this behavior: If you have configured the UCX$SMTP_ALTGATE_ALWAYS logical then all non-local mail is sent to the alternate gateway. No zone check is performed. If you have configured the UCX$SMTP_MX_IF_NOALTGATE logical then any mail that is to be sent to the alternate gateway will be sent using the UCX MX Lookup gateway list *if* the symbiont is unable to get to the alternate gateway. If you have not configured this option and UCX SMTp caanot get to the alternate gateway the message will be requeued. If you have configured the UCX$SMTP_NO_MX logical then mail that is not destined for the alternate gateway will not be sent using the UCX MX Lookup gateway list. Rather, the domain part of the address will be translated into a DNS host name directly and the mail sent directly to that address. If the domain of the address doesn't translate as a DNS hostname the mail will bounce. This feature isn't particularly useful. -------------------- How the symbiont sends a mail using the UCX MX Lookup gateway list -------------------- When the symbiont decides that mail to a particular recipient is not going out via the alternate gateway (eg. no alternate gateway is defined or the recipient address is within the zone) the symbiont must relay the mail itself. First it calls the UCX MX Lookup routine passing it the domain part of the recipient address. As described in "UCX MX Lookup" this returns a list of potential gateways for the mail in a prioritized order. The symbiont tries to send the mail via the SMTP protocol to the first gateway in the list. If this fails it tries the next gateway in the list and on and on until either the mail has been sent or the list is exhausted. If the list is exhausted then the mail is requeued to be tried at a later time. -------------------- How the symbiont sends a mail through the alternate gateway -------------------- To send a mail to the alternate gateway you might conclude that the symbiont translates the alternate gateway to an IP address using gethostbyname and connects to the address. This assumption is understandable but incorrect. To send mail to the alternate gateway the symbiont calls the UCX MX Lookup routine (see "UCX MX Lookup") on the alternate gateway and uses the resulting list of gateways to get the mail to the alternate gateway. To understand the benefit of this consider this example which is how we at UCX engineering have UCX SMTP configured on our own system. First our alternate gateway and zone: UCX> sho config smtp ... Alternate gateway: relay.dec.com ... Zone: dec.com ... What if we try to translate the alternate gateway, "relay.dec.com" as a host name? UCX> sho host relay.dec.com %UCX-W-NORECORD, Information not found -RMS-E-RNF, record not found There is no such host as relay.dec.com because relay.dec.com is just an MX destination domain with multiple records at the same preference: UCX> sho mx relay.dec.com BIND MX database Server: 16.20.208.53 ucxaxp.ucx.lkg.dec.com Gate address Preference Gate name 192.208.46.10 100 mail11.digital.com 192.208.46.20 100 mail12.digital.com 192.208.46.30 100 mail13.digital.com 204.123.2.56 200 mail2.digital.com 204.123.2.50 200 mail1.digital.com 193.56.15.21 300 mail21.digital.com When UCX SMTP on our system sees a mail destined for a domain outside of "dec.com" it uses the above list of MX records to send the mail to what is really an imaginary entity called "relay.dec.com". So, even when mail is routed through the alternate gateway the UCX MX Lookup list is used. --------------- Further enhancements in worklist for mail destined for alternate gateway --------------- This section documents some items which are in the worklist with respect to mail destined for the alternate gateway. Enhance caching of alternate gateway MX records: The symbiont calls the UCX MX Lookup procedure for the alternate gateway once at initialization time and caches the resulting list of gateways. This is done for efficiency to save the cost of an MX lookup on the alternate gateway each time mail is to be sent to the alternate gateway. The symbiont refreshes the cached list of MX records for the alternate gateway every 24 hours. There is a problem with this. Consider the case of our own relay.dec.com alternate gateway. The domain relay.dec.com has multiple MX records defined for it at the same preference. As we discussed in "MX Example 1" in the section "MX Record Overview" in part 1 this is to achieve load balancing between the gateway hosts that share the same preference. Load balancing should occur because the MX records with the same preference appear in a different order each time they are looked up. However since the MX records for the alternate gateway are cached by the UCX SMTP symbiont it uses the same MX record sequence every time and so cannot take advantage of the load balancing that has been set up for the alternate gateway. For example if the UCX SMTP symbiont has cached the MX records for relay.dec.com as above it would always try to connect to mail11.digital.com first. No mail would get sent through mail12.digital.com or mail13.digital.com. When the MX records are refreshed after 24 hours the order of the gateways with the same preference may change and then the symbiont will connect to a different host - but always the same one - until another 24 hours passes and the alternate gateway MX cache is refreshed again. The solution for this is to periodically reshuffle the order of the MX records (with the same preference) in the alternate gateway MX list. Send all mail to alternate gateway in one SMTP session: UCX SMTP has a design inefficiency with respect to sending mail via the alternate gateway for mail with multiple recipients outside the zone. This is most commonly encountered when sending mail to a mailing list in which there are multiple recipients outside the zone. With the present design of the control file the the symbiont does everything on a domain by domain basis. It sends mail to one domain and then to the next and to the next, etc. If addresses in different domains are all destined for the alternate gateway they will still be sent separately. Say for example you have a zone of z.com with an alternate gateway of mail.z.com and you have a mailing list with the addresses user1@x.com and user2@y.com. Since these two addresses are in different domains they are processed separately by the symbiont. First we connect to the alternate gateway and go through the whole SMTP dialog for user1@x.com. Eg: send: HELO whereever.z.com recv: 250 mail.z.com Hello whereever.z.com pleased to meet you send: MAIL FROM: recv: 250 ... Sender ok send: RCPT TO: recv: 250 Recipient ok send: DATA recv: 354 Enter mail, end with "." on a line by itself send: message headers send: [blank line] send: message text send: . recv: 250 Message accepted for delivery send: QUIT recv: 221 closing connection Then the link is disconnected and we go on to do the exact same thing for user2@y.com. For mailing lists with many recipients outside the zone this is costly. It would be far more effient if the mail was sent through the alternate gateway with multiple RCPT TO commands for each of the domains destined for the alternate gateway. Eg: send: HELO whereever.z.com recv: 250 mail.z.com Hello whereever.z.com pleased to meet you send: MAIL FROM: recv: 250 ... Sender ok send: RCPT TO: send: RCPT TO: recv: 250 Recipient ok send: DATA recv: 354 Enter mail, end with "." on a line by itself send: message headers send: [blank line] send: message text send: . recv: 250 Message accepted for delivery send: QUIT recv: 221 closing connection There is a work around for this. You need to trick UCX SMTP into dealing with each of the addresses destined for the alternate gateway as if they were in the same domain. You can do this by defining the addresses of users in a mailing list who are outside the zone using the percent sign hack. Taking the example above you'd create the list like so: $ Create alist.dis user1%x.com@mail.z.com user2%y.com@mail.z.com Using the percent sign hack you have given both of the addresses the same domain part - the name of your alternate gateway. As far as UCX SMTP is concerned they are in the same domain. This means that when the symbiont goes to deliver the mail each of these addresses will be in the linked list under the "mail.z.com" domain and will be dealt with together. ---------------------------------------- Forwarding to an SMTP address ---------------------------------------- Consider the case where an SMTP mail comes into your system that is to be delivered locally and the recipient has a mail forwarded to an SMTP address. This section discusses some special case handling in the symbiont to handle such forwarding to an SMTP address which is refered to as SMTP-to-SMTP forwarding. When the symbiont delivers mail locally it checks the local-part of the address to see if there is a forwarding string (ie. SET FORWARD) for it. It does so using the mail$user_* VMS callable mail routines. If there is a forwarding string the symbiont checks to see if it is an SMTP address. If it is then some special handling is in order. Let's take an example to show what would happen if there was *no* special checking for delivering SMTP mail to a local recipient with their mail forwarded to SMTP. Say user jones on a UCX system called systm2.xyzcorp.com has forwarded their mail to SMTP%"jones@othersys.xyzcorp.com". Now say mail arrives on systm2.xyzcorp.com for jones@systm2.xyzcorp.com. What would happen if the symbiont didn't make special checks for SMTP forwarding here? The symbiont would deliver mail via callable mail to the reciopient address "jones". What is VMS mail going to do with this? It sees the mail is forwarded to SMTP% and so it locates and invokes the UCX SMTP MAIL$PROTOCOL routine (see "VMS mail Foreign Transport/MAIL$PROTOCOL Interface") to deliver the message. VMS mail passes the recipient's forwarding address - jones@othersys.xyzcorp.com - to the MAIL$PROTOCOL routine as the recipient address. This happens inline in the symbiont process but is totally transparent to the symbiont code itself. As far as the symbiont code is concerned it has made a simple call to mail$send_add_address for user jones. After the call to mail$send_add_address the symbiont passes the VMS From, To, etc lines to callable mail and each call is redirected by VMS mail to the SMTP MAIL$PROTOCOL code. Finally the symbiont passess each line of text starting with the message's RFC headers and then the blank line and the text of the message.And agin this is passed to the MAIL$PROTOCOL code. Sounds fine, no? This creates a problem though. Invoking the MAIL$PROTOCOL code causes the generation of a *new* SMTP mail message with *new* SMTP headers. The entirety of the existing SMTP message that we are trying to send (including its RFC 822 headers) is now hidden away - "encapsulated" - inside the new message as the text of the message. There are now two blocks of RFC 822 headers in the new message. The first block of headers are the ones the MAIL$PROTOCOL interface has just created. They are followed by a blank line and the original RFC 822 headers, another blank line and the original body. This is obviously a pain since the recipient has more headers to read through. It is also a bug since it breaks the SMTP loop detection mechanism. The SMTP loop detection mechanism requires that each MTA that touches a message append a Received: header and that each MTA count Received: headers and bounce the mail if the number of Received headers surpasses some locally configurable limit. (This BTW is UCX SMTP's HOP_COUNT_MAX.) In this fashion mail that has taken "too many hops" will be bounced as looping. Going back to the example say that user jones had inadvertantly set their forwarding on host "othersys" right back to systm2.xyzcorp.com - the classic mail forwarding loop. In this scenario the mail would bounce back and forth between the two systems getting one additional set of headers each time it was processed by systm2. Since each new block of headers encapsulates the existing message into a new message thus making the old block of headers invisible (they become the body of the message) a Received: header is added on each time to the outermost block of headers and then this block of headers is encapsulated within the message and a new header block is created. So we have a mail message of many, many separate blocks of RFC headers each with two Received: headers (one from systm2 and one from othersys) rather than a single block of headers with *all* the Received: headers. As a consequence the code that counts Received: headers to detect looping can't work and the mail loops forwever. (BTW. UCX SMTP had this bug for a long time.) To fix this the UCX SMTP symbiont checks to see if a local user to whom it is delivering an SMTP message has mail forwarding set to an SMTP address. If it does it dynamically adds a new address - the user's forwarding address - to the control file address list. It then marks the address status block of the old address as having been successfully delivered. In the case of our example that means that the recipient address jones@xyzorp.com is marked as delivered successfully and jones's forwarding address jones@othersys.xyzcorp.com is dynamically added to the control file. The symbiont will see this address and then deliver the mail to othersys.xyzcorp.com via SMTP without the encapsulation and bugs described above. Situation where encapsulation problem can still occur: If you define a SYSTEM logical name pointing to an SMTP address and the symbiont tries to deliver mail to that address that situation is not caught yet and you'll get the header encapsulation. EG: $ DEFINE/SYSTEM JONES SMTP%"""jones@othersys.xyzcorp.com""" Try it. You'll see. Support for setting forward to SMTP address without SMTP% jacket: For some later versions of VMS you can do: MAIL> SET FORWARD/USER=jones@othersys.xyzcorp.com Ie. you don't need the SMTP jacket. UCX SMTP can handle this too. That is, its special case check to see if an address has been forwarded to SMTP detects this kind of forwarding. ---------------------------------------- UCX SMTP and non-SMTP mail ---------------------------------------- One issue for any VMS based SMTP mailer is how to deal with mail coming from or going to non-SMTP mailers. This happens mostly when users have their mail forwarded. If a user's mail is forwarded to an SMTP address and a non-SMTP mail (say DECnet) arrives that's the case where mail is going from a non-SMTP MTA to SMTP. If a user's mail is forwarded to a non-SMTP address (say DECnet) and an SMTP mail arrives it's the case of mail going from SMTP to a non-SMTP MTA. This section discusses what happens when mail moves accross the boundary between SMTP and non-SMTP MTA's. -------------------- SMTPfication of non-SMTP addresses -------------------- Before discussing the specifics of what's happening when mail moves accross the boundary between SMTP and non-SMTP MTA's it is necessary to understand a procedure implemented in UCX SMTP for transforming any non-SMTP address into an SMTP address. This procedure is called SMTPfication and is used on addresses as they cross the boundary from non-SMTP land to SMTP land. With SMTPfication the non-SMTP address is munged and this munged string becomes the local-part of the created SMTP address. The domain part of the created SMTP address becomes the UCX SMTP substitute domain or, if no substitute domain is configured, the host's hostname. The SMTPfication steps are as follows: 1) Scan the non-smtp address and prefix a backslash (\) character to each double quote character (if any). 2) Put the entire string in double quotes. 3) Append an '@' followed by the substitute domain (or hostname if no substitute domain) --------------- Example SMTPfications --------------- This section gives some examples of address being SMTPfied. They all assume that the UCX SMTP host doing the SMTPfication has a substitute domain of xyzcorp.com. Example 1 - DECnet address: Before SMTPfication: SUMNOD::SMITH After Step 1 (Prefix double quotes with a \): SUMNOD::SMITH After Step 2 (Put entire string in double quotes): "SUMNOD::SMITH" After Step 3 (append @domain): "SUMNOD::SMITH"@xyzcorp.com Example 2 - MTS address: Before SMTPfication: SUMNOD::XXXMTS::MRGATE::"YYYMTS::A1::SMITH" After Step 1 (Prefix double quotes with a \): SUMNOD::XXXMTS::MRGATE::\"YYYMTS::A1::SMITH\" After Step 2 (Put entire string in double quotes): "SUMNOD::XXXMTS::MRGATE::\"YYYMTS::A1::SMITH\"" After Step 3 (append @domain): "SUMNOD::XXXMTS::MRGATE::\"YYYMTS::A1::SMITH\""@xyzcorp.com Example 3 - another foreign transport address: Before SMTPfication: DSN%SRQ$Z123456-7890 After SMTPfication: "DSN%SRQ$Z123456-7890"@xyzcorp.com Example 4 - Quoted DECnet address: Before SMTPfication: SUMNOD::"smith@abccorp.com" After SMTPfication: "SUMNOD::\"smith@abccorp.com\""@xyzcorp.com Example 5 - X.400 --> MTS address. Yuk!: Before SMTPfication: SUMNOD::MRGATE::"XXX::1=FI::2=XXX::3=XYZ::JOE SMITH" After SMTPfication: "SUMNOD::MRGATE::\"XXX::1=FI::2=XXX::3=XYZ::JOE SMITH\""@xyzcorp.com --------------- When is SMTPfication necessary? --------------- SMTPfication becomes necessary whenever the UCX SMTP code needs to form an address from a VMS address for an RFC 822 header or an SMTP protocol command and the string to be used for the local-part of the address is not a simple username. SMTPfication is not needed for a simple VMS username because a VMS username is syntactically legal for the local-part of an RFC 822 address. For example the simple VMS mail address of JONES becomes jones@xyzcorp.com not "jones"@xyzcorp.com. However the latter form is actually legal and would work. SMTPfication is also performed by the UCX POP server when it creates the dummied up From: RFC 822 header from the VMS mail From: line. For more information on POP's dummied up RFC 822 headers see the POP documentation. -------------------- Mail from non-SMTP to SMTP -------------------- The first case we will discuss is the one where a mail comes from a non-SMTP mail environment and is to be delivered into the SMTP environment. The most common cuase for this is when a non-SMTP mail arrives for a user with mail forwarded to an SMTP address. The key issue in accepting mail from a non-SMTP MTA and passing it into the SMTP world is getting the From: and Return-Path: (er um, the MAIL FROM) addresses correct. If the From: address is not correct the recipient will not be able to reply. If the address that we put in the MAIL FROM: command is not correct and the mail needs to be bounced then it will not be bouncable. When mail arrives for a user with mail forwarded to an SMTP address VMS mail invokes the SMTP MAIL$PROTOCOL code (see "VMS mail Foreign Transport/ MAIL$PROTOCOL Interface") and passes the mail message to it to deliver. As discussed in the MAIL$PROTOCOL section one of the calls that VMS mail makes to the SMTP MAIL$PROTOCOL routine passes the VMS mail From: line. The SMTP MAIL$PROTOCOL code uses the VMS mail from: line to form the RFC 822 From: and Return-Path: headers. To do so it uses SMTPfication on the VMS mail From: line. --------------- An example --------------- Take an example where user SMITH on DECnet node SYSTM1 sends an DECnet mail to user JONES on DECnet SYSTM2. Say that DECnet node systm2's UCX host name is also systm2.xyzcorp.com and that its substitute domain is xyzcorp.com. Finally say that user JONES has their mail forwarded to SMTP%"jones@othersys.xyzcorp.com". On SYSTM1 SMITH does: MAIL> send To: SYSTM2::JONES Subject: Test test text here On SYSTM2 the MAIL$SERVER network object (running SYS$SYSTEM:MAIL_SERVER.EXE) delivers the mail to user JONES and sees that it is forwarded to SMTP%. This means the UCX SMTP MAIL$PROTOCOL routine is called to deliver the message. When the From: line is passed to MAIL$PROTOCOL it's: From: SYSTM1::SMITH At this point the MAIL$PROTOCOL routine forms an RFC 822 From: line of From: "SYSTM1::SMITH"@xyzcorp.com And the Retun-Path header (which will be put into the MAIL FROM SMTP protocol command) is the same. Ie: Return-Path: "SYSTM1::SMITH"@xyzcorp.com Note that nothing special needs to get done with the To: field. The SMTP MAIL$PROTOCOL code forms the RFC 822 To: field from the value passed to it with the recipient address function code which is whatever was inside the SMTP% jacket. In this case it's the forward string for user jones which is jones@othersys.xyzcorp.com. Ie. This scenario doesn't complicate the construction of the To: header. After the rest of the MAIL$PROTOCOL interface calls are made the mail is packaged up into a control file, submitted to the symbiont and delivered to jones@othersys.xyzcorp.com. --------------- SMTPfication as simple, generic mechanism for gatewaying mail --------------- Consider the RFC From address "NODE::USER"@dom.ain. As illustrated by the example above, the SMTPfication procedure applied to a mail address as it passes from outside SMTP to SMTP is a way preserving the path that the mail has taken within the mail address itself. Such a mechanism does produce syntactically complex addresses but it is also useful as a generic mechanism for gatewaying mail that works for any possible non-SMTP address that the UCX SMTP MAIL$PROTOCOL code could be passed. -------------------- Mail from SMTP to non-SMTP -------------------- Now consider the case where mail is sent from the SMTP world into a non-SMTP MTA. The example above is a good starting point. Say that user jones on othersys.xyzcorp.com receives the example mail from SMITH and replies. If othersys is a UCX SMTP system the VMS mail From: line will look like: From: SMTP%"˘SYSTM1::SMITH˘@xyzcorp.com" For the time being ignore the cent signs in this address. They are required because of historical address parsing deficiencies in VMS mail which are being worked on. See "The UCX SMTP cent sign hack" for more information. Suffice it to say that this how an RFC 822 From: line of `From: "SYSTM1::SMITH"@xyzcorp.com' is stuffed into the VMS mail From line by UCX SMTP. Say that user jones replies to this mail. This will cause a mail to be sent to the address in the VMS mail From: line - SMTP%"˘SYSTM1::SMITH˘@xyzcorp.com". Since this address starts with SMTP% VMS mail invokes the UCX SMTP MAIL$PROTOCOL code on othersys. In this case the construction of the From: header is easy. The From: line passed to the MAIL$PROTOCOL code is JONES so the RFC 822 From: header is simply jones@othersys.xyzcorp.com. (Let's assume that this system has no substitute domain.) When the SMTP MAIL$PROTOCOL code is passed the recipient address function code the address looks like: ˘SYSTM1::SMITH˘@xyzcorp.com. The MAIL$PROTOCOL code converts the cent signs back to double quotes and this yields an RFC 822 To: header of: To: "SYSTM1::SMITH"@xyzcorp.com This address is also put into the control file address area (so that the symbiont sends it out with a RCPT TO:<"SYSTM1::SMITH"@xyzcorp.com>). Once the rest of the message is packaged up into a control file and submitted the symbiont gets it and sends the mail using the SMTP protocol to systm2.xyzcorp.com. (Let's assume that the MX record for xyzcorp.com points to systm2.xyzcorp.com so systm2 receives the mail.) When systm2 receives the mail it sees that it's local mail because the domain of the RCPT TO: address is xyzcorp.com which matches its substitute domain. (See "Local vs. non-local SMTP mail".) So the symbiont delivers the mail locally via callable VMS mail. (See "UCX SMTP and callable VMS mail".) To construct the recipient address string to pass to callable VMS mail's mail$send_add_address routine the symbiont now un-SMTPfies the recipient address. This is just the reverse of the SMTPfication. So starting with this: "SYSTM1::SMITH"@xyzcorp.com it strips off the @domain to get this: "SYSTM1::SMITH" and strips off the outer double quotes to get this: SYSTM1::SMITH In this example there are no \" sequences but if there were it would remove the '\' characters too as the last step of un-SMTPfying the address. So, the string that the symbiont passes to callable mail as the recipient address is SYSTM1::SMITH. To form the VMS mail From: line the symbiont does nothing special. It chooses the From: RFC 822 header to be placed into the VMS mail from line and packages it into an SMTP% jacket to make it a legal VMS incarnation of an SMTP address. So the VMS mail From line that is passed to callable mail by the symbiont is: From: SMTP%"jones@othersys.xyzcorp.com" Now the rest of the mail is passed to VMS callable mail and the symbiont is done delivering the mail. At this point the MAIL11 server takes over and delivers the mail to SMITH on SYSTM1. (Actually it's done in line with the symbiont call to callable mail because, after all, MAIL11 is not a store and forward protocol but that's an unimportant detail.) Note that the VMS mail From: line that SMITH sees will be enhanced by the MAIL11 code from what the SMTP symbiont passed to callable mail. The VMS mail From: line that SMITH sees will be: From: SYSTM2::SMTP%"jones@othersys.xyzcorp.com" The SYSTM2:: is added to the front because the mail made the hop from SYSTM2 to SYSMT1 via DECnet. Now say a reply is done. This will basically do the same thing as the original send that SMITH did to SYSTM2::JONES. The only difference between sending to SYSTM2::JONES and SYSTM2::SMTP%"jones@othersys.xyzcorp.com" is that sending to SYSTM2::SMTP%"jones@othersys.xyzcorp.com" saves the extra step performed in the MAIL$SERVER on SYSTM2 of looking to see that jones has their mail forwarded SMTP%"jones@othersys.xyzcorp.com". --------------- Another SMTP to non-SMTP case --------------- Another case similar to the example above is the case where SMTP mail is to be delivered locally to a user who has their mail forwarded to a non-SMTP address. Say systm2.xyzcorp.com is delivering a mail to doe@xyzcorp.com and user DOE has mail forwarded to SYSTM1::DOE. In this case the symbiont doesn't know it's gatewaying mail. It simply passes the string "doe" to VMS callable mail as the recipient address and the VMS mail From: line is SMTP%"whoever@wherever.com". When user DOE on SYSTM1 reads their mail it the VMS mail From: line appears in exactly the same format as the previous example: From: SYSTM2::SMTP%"whoever@wherever.com" -------------------- Miscellaneous SMTP/non-SMTP issues -------------------- Not just DECnet that works like above: The examples above all used DECnet as the prototypical non-SMTP MTA. However all the same things happen with other non-SMTP address like MTS addresses and other foreign transport address too (eg. DSN%, NM%, etc.) Unix sendmail.cf files tend to munge these addresses. If some of these addresses (eg. From: "NODE::USER"@dom.ain) are passed through or delivered to Unix systems there is a high likelyhood that the Unix system will modify and corrupt the address. The cause of this is a Unix sendmail.cf file that doesn't understand the address format and so misparses it and rewrites it incorrectly. If you see this happen it is important to understand that these addresses although syntactically complex are entirely legal according to the RFC 822 address syntax. Often the corruption of the address will be such that it still manages to get delivered to where it is going but is unreplyable. Sometimes it never even gets to where it's going and has to be bounced. Usually in that case the mail can't be bounced either since the MAIL FROM address is in the same format as the From address and if the one couldn't be understood then the other won't be either. In that case it will most likely become a dead letter. Here are some examples of correct versions of these types of addresses and the types of munging that are commonly performed by bad sendmail.cf files. Correct: From: "SUMNOD::SMITH"@xyzcorp.com Incorrect variant: From: SUMNOD::SMITH@xyzcorp.com This address is incorrect because the local part contains a colon which is an RFC 822 "special" character. The RFC address syntax requires that a local-part that contains a special character must be in double quotes. Correct: From: "SUMNOD::XXXMTS::MRGATE::\"YYYMTS::A1::SMITH\""@xyzcorp.com Incorrect variants: From: "SUMNOD::XXXMTS::MRGATE::"YYYMTS::A1::SMITH""@xyzcorp.com This variant is wrong because it doesn't "escape" the nested double quotes with backslashes. From: SUMNOD::XXXMTS::MRGATE::\"YYYMTS::A1::SMITH\"@xyzcorp.com This variant is wrong for the same reason that SUMNOD::SMITH@xyzcorp.com is wrong. Special characters in a local-part require double quotes around the local part. From: SUMNOD::XXXMTS::MRGATE::"YYYMTS::A1::SMITH"@xyzcorp.com This variant is wrong for both reasons. -------------------- SMTP/non-SMTP worklist items -------------------- This section documents some items in the UCX worklist which will hopefully make life easier for users who want to send mail between non-SMTP environments and the Internet via UCX SMTP. The problem is that the complex addresses generated by UCX SMTP's SMTPfication are usually munged by Unix sendmail.cf files. Provide mechanism to hide DECnet addresses: UCX may add support for hiding DECnet addresses inside SMTP addresses. This is a commonly supported practice. The two options are 1) Have a special SMTP domain (perhaps configurable by the UCX system manager) that UCX SMTP recognizes as mail destined for DECnet. For mail coming from SMTP to this domain the SMTP mailer essentially takes whatever is before the first "." in the domain and that is the DECnet nodename. For mail coming from DECnet destined for SMTP we take the DECnet node name and tack on the special DECnet domain to form an SMTP address. Digital's own "enet.dec.com" domain is an example of this. When one sends mail to user@nodnam.enet.dec.com the mail goes via SMTP to a Digital mail hub that has a wildcard MX for *.enet.dec.com pointing to it. This mail hub transforms user@nodnam.enet.dec.com to NODNAM::USER and forwards the mail to that address via DECnet. For example: Say the new feature allows the system manager to configure their special DECnet domain and the manager of xyzcorp.com has theirs set to decnet.xyzcorp.com. Further assume they have a wildcard MX record set up so that mail to be sent to *.decnet.xyzcorp.com is to be sent to systm2.xyzcorp.com. Say a mail is sent from the Internet somewhere outside xyzcorp's corporate network to smith@systm1.decnet.xyzcorp.com. It is routed via SMTP to systm2.xyzcorp.com. At this point UCX SMTP sees that the domain of the SMTP address is the "special" DECnet domain and so transforms the recipient address from smith@systm1.decnet.xyzcorp.com to systm1::smith and delivers the mail locally via callable mail to the address systm1::smith. Coming in the other direction DECnet mail sent from SMITH to SYSTM2::JONES where jones has mail forwarded via SMTP as in the examples earlier in this section would no longer be From: "SYSTM1::SMITH"@xyzcorp.com. It would now be From: SMITH@SYSTM1.decnet.xyzcorp.com 2) The other option is just a different syntactical technique to do the same thing. With option 2 we put hooks in the mailer to support special interpretation of the percent sign hack when the domain following the percent sign indicates a particular foreign transport. For example: smith%systm1.decnet@xyzcorp.com Provide mechanism hide MTS addresses: It may be possible to do the same thing for MTS (aka Message Router, MRGATE) mail too. Eg: To: joeuser@mrgate.xyzcorp.com funnels mail off to joeuser via MTS when it arrives at xyzcorp.com. For sites where users have VMS mail forwarded to ALL-in-1 see the description of the UCX SMTP UCX$SMTP_REWRITE_MTS_FROM logical in section "Special case logicals for MTS interaction". ---------------------------------------- Undeliverable Mail ---------------------------------------- When UCX encounters a mail that it cannot deliver it must bounce it back to the sender. To do so it forms an undeliverable mail message (aka. a "bounced mail") and sends it to the address in the SMTP Return-Path header (which is from the MAIL FROM SMTP protocol command). In other words the MAIL FROM address of the mail we were not able to deliver becomes the RCPT TO address of the bounced mail message. The bounced mail message contains information about what went wrong, which addresses got the mail and which didn't (for multiple recipient mail messages) and also the entire text of the mail message being bounced. As required by the SMTP RFC's the bounced mail message is sent with a null MAIL FROM command (ie. MAIL FROM:<>) to prevent bounced mail about bounced mail. What if we can neither deliver nor bounce a mail? In the case where we discover that we cannot deliver a mail message and then try to bounce it and cannot bounce it either what to do? Under such circumstances UCX SMTP delivers the mail to the UCX_SMTP account under the assumption that UCX_SMTP is forwarded to some account that is read regularly by someone who wants to know about dead letters. (See "Postmaster field" section of "UCX SMTP Configuration database entry".) If, for some reason, the message cannot be delivered to the UCX_SMTP account it is left as an orphan control file. ie. A dead letter. A message is signalled to the UCX SMTP symbiont log. What to do if bouncing bounced mail?: Since a null MAIL FROM is legal an SMTP mailer must handle the case where it is trying to bounce a mail message and the MAIL FROM is null. This is the case where we are trying to deliver a message which is itself a bounced mail and so cannot be bounced. In this case UCX SMTP will send the message to the UCX_SMTP account. As above, if the message cannot be delivered to the UCX_SMTP account it is left as an orphan control file and a message is signalled to the UCX SMTP symbiont log. ---------------------------------------- Configuring UCX SMTP ---------------------------------------- This section documents the buttons knobs you have to configure UCX SMTP. -------------------- UCX SMTP Common directory -------------------- In earlier versions of UCX SMTP all files were stored in the UCX_SMTP system specific directories. UCX SMTP now allows the system manager to create a directory visible to all cluster nodes (or a subset) and store certain SMTP files there. To use this feature the system manager must create the directory and define a system logical UCX$SMTP_COMMON to point to it. At this time the following files may go into UCX$SMTP_COMMON: - UCX SMTP mailing list (.DIS) files. See "UCX SMTP Mailing Lists" - The Local Aliases file. (See "Local Aliases File") -------------------- Local Aliases File -------------------- The local alias feature allows the system manager to define a list of domains that UCX SMTP will interpret as local. (See "Local vs. non-local SMTP mail".) If it sees mail to any one of the domains specified as local aliases it will deliver the mail on the local system via callable VMS mail rather than forwarding it on to another system via the SMTP protocol. To define the aliases that you want to be considered local you simply use your text editor or the DCL create command to create a file called UCX$SMTP_LOCAL_ALIASES.TXT. You can put the file in either the UCX$SMTP_COMMON directory or, if you don't define UCX$SMTP_COMMON, you can put the file in SYS$SPECIFIC:[UCX_SMTP]. Each line in the file is one domain that you want UCX SMTP to recognize as local. For example: $ SET DEF UCX$SMTP_COMMON $ CREATE UCX$SMTP_LOCAL_ALIASES.TXT ! ! This is a comment. alias1.mydomain.edu alias2.mydomain.edu alias3.mydomain.edu $ @SYS$MANAGER:UCX$SMTP_SHUTDOWN.COM $ @SYS$MANAGER:UCX$SMTP_STARTUP.COM The above example tells UCX SMTP to recognize "alias1.mydomain.com", "alias2.mydomain.com" and "alias3.mydomain.com" as local. If the symbiont on your host sees mail to user@alias1.mydomain.com, user@alias2.mydomain.com or user@alias3.mydomain.com it will consider the mail local and try to deliver it to the user on your system. The entries in the local alias file must adhere to the following syntax rules. - There may be only one alias entry per line. - Comments follow the same rules of SMTP .DIS files. The comment character, a "!", must appear in the first column. - UCX SMTP does not append the local domain name to an entry that it reads with no "." in it. For example, if your local domain is "mydomain.com" and you want "alias1.mydomain.com" to be recognized as local you must enter "alias1.mydomain.com" into the local alias file. Entering "alias1" is not sufficient. Other miscellaneous notes: - When UCX SMTP compares entries in the local alias file with the domain of an address it does so case insensitively. For example, if the local alias file contains "alias1.mydomain.edu" and a mail is sent to "ALIAS1.MYDOMAIN.EDU" a match occurs and the mail is considered local. - The local aliases file protection must be W:RE. - There is a maximum of 255 aliases each of which can be a maximum of 64 characters long. - After changing the local alias file you must stop and restart UCX SMTP in order for the changes to take effect. One use of the local aliases file is in a cluster where you want each host to accept inbound mail for all of the cluster members. Simply enter the names of the cluster members into the local aliases file in UCX$SMTP_COMMON. -------------------- UCX SMTP Configuration database entry (UCX SET CONFIG SMTP) -------------------- This section suplements the other UCX SMTP documentation on the UCX SMTP CONFUGRATION database entry. --------------- Postmaster field --------------- There are too many bugs that arise when users try to change the Postmaser field in the UCX SMTP configuration database and it's inclusion as a configurable item was at best poorly executed and at worst ill-conceived. Therefore - the account that UCX SMTP runs in is UCX_SMTP. - The Postmaster field is a deprecated field. - Leave it alone. Ie. Leave it as UCX_SMTP. - Do not muck around with the UCX_SMTP account unless you have been asked to do so by Digital. All sites should also set forward for two key SMTP usernames to an account which is read regularly. The system account is the most likely candidate. Eg: MAIL> SET FORWARD/USER=UCX_SMTP SYSTEM MAIL> SET FORWARD/USER=POSTMASTER SYSTEM Setting forward for UCX_SMTP to SYSTEM means that any messages which could neither be delivered to the recipient nor bounced back to the sender are delivered to the SYSTEM user. -------------------- UCX SMTP configuration logicals -------------------- This is to document the SMTP configuration logicals which, heretofore, have been documented only in release notes. Notes about SMTP logicals: - SMTP logicals governing the behavior of the symbiont are translated at queue startup time. If the logical value is changed or deassigned then the queue must be stopped and started in order for the change to take effect. Ie. you need to do UCX STOP MAIL, define your logicals, then do UCX START MAIL. - SMTP logicals governing the behavior of the receiver are translated when the receiver image starts up. Since a new receiver process is started for each inbound connection to port 25 there is nothing to start or stop. The logicals take effect for the next receiver process created. - Some SMTP configuration logicals are "boolean" in nature in that they are associated with an SMTP configuration option that is either on or off. If the logical is defined then the option is considered to be on. If it is not defined then the option is off. The actual *value* of the logical is not examined by the symbiont although by convention these logicals are defined to "1". Defining the logical to "OFF" or "NO" or "0" will still have the effect of turning the option on! To turn the option off don't define the logical at all. - The SMTP configuration logicals must be defined as /SYSTEM except where noted. Here is a list of all of the UCX$SMTP logicals. Boolean logicals are marked as such. ******************************************************************************** * Diagnostic logicals ******************************************************************************** Name: UCX$SMTP_LOG_LEVEL Type: Numeric Values: 2,3 or 5 This logical tells the SMTP symbiont what diagnostic information to write to the log file. It can have one of three values, 2, 3 or 5 with 2 being the least verbose and 5 being the most. The values are cumulative in that all messages printed out by one logging level are printed out by each of the more verbose logging levels. A value of "3" is recommended. Value "2" - All of the SMTP configuration info is logged when the symbiont starts up. - The "next open file" message is printed giving the name of each control file before processing on it begins. - All mail headers and mail recipients in a control file are logged after control file processing is complete. Value "3" - Logs additional information about symbiont initialization and symbiont activity during control file processing without being overly verbose. Value "5" - Full symbiont diagnostics. For use only when diagnosing problems. Much of this will only be of use to Digital engineering staff but a diligent user can still get a sense of what's going on. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_NOSEY Type: Boolean Values: NA This logical tells the symbiont to print out the Subject RFC header when it logs the RFC headers if UCX$SMTP_LOG_LEVEL is defined. If this logical is not defined then the Subject header is logged as "Subject: ". ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_LOG_LINE_NUMBERS Type: Boolean Values: NA If defined line numbers are written to UCX SMTP logs. This includes the symbiont, receiver and MAIL$PROTOCOL (DEBUG.TXT) logs. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_SYMB_TRACE Type: Boolean Values: NA This logical tells the symbiont to log all messages received from and transmitted to remote SMTP servers. It allows you to trace the SMTP application layer protocol. Any unprintable and/or control characters sent/received are printed as \n where 'n' is the hex value of the character. For example, command lines and replies are terminated with a which would appear in the log like this: send buf=MAIL FROM:\d\a recv buf=250 ... Sender OK\d\a The "\d\a" is the . CR is hex 'd' and LF is hex 'a'. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_RECV_TRACE Obsoletes: UCX$SMTP_PROTO_TRACE Type: Boolean Values: NA This logical tells the receiver to log all messages received from and transmitted to remote SMTP clients. The same conventions for logging unprintable and/or control characters are used. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_RECV_DEBUG Obsoletes: UCX$SMTP_PROTO_DEBUG Type: Boolean Values: NA This logical tells the receiver to log full diagnostics. It performs a similar function to the symbiont's UCX$SMTP_LOG_LEVEL 5. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_VMSMAIL_SEND Type: Boolean Values: NA If this logical is defined the MAIL$PROTOCOL will log diagnostics to a file named DEBUG.TXT in the default directory. Much of this will only be of use to Digital engineering staff but if you are familiar with the MAIL$PROTOCOL interface you can see what is coming into the SMTP MAIL$PROTOCOL code. If you are debugging the mail$protocol code running in a user process sending mail then this logical need not be a system logical. If you are debugging the MAIL$PROTOCOL code running when DECnet mail comes in then it's running in the context of the DECnet Mail11 network object and so the logical needs to be a system (or group logical). ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_VMSMAIL_PARSE Type: Boolean Values: NA This causes the UCX SMTP address parsing code to log diagnostics. The symbiont, receiver and the MAIL$PROTOCOL code are all sensitive to this logical. This logical is primarily for the use of Digital engineering staff. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_LOG_MEMORY, UCX$SMTP_LOG_EFS Type: Boolean Values: NA These options are primarily for use of Digital staff in tracing memory leaks and event flag leaks. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_SYMB_SNAPSHOT_BLOCKS and UCX$SMTP_RECV_SNAPSHOT_BLOCKS Type: Numeric Values: 1-1000 In cases where errors are being signalled in the symbiont or receiver logs intermittently and are not reproducable at will it is often not desirable to simply run with full diagnostics turned on and wait for the error to occur again. This wastes disk space creating huge logs and slows down processing of mail dramatically. The symbiont and receiver have a feature called "snapshot logging" which allows you to run with full diagnostics turned on but only write the diagnostics to the log file if an error is signalled. This saves disk space and allows the receiver and/or symbiont to run at a normal speed. As each line of diagnostic text is generated it is saved in an internal "snapshot" buffer rather than to the disk. The buffer is "circular" in that once it fills up, new lines of text simply start to overwrite the old data already there. This provides an up to date snapshot of the last "so many" lines of diagnostic text. There are two logicals to turn this feature on. One for the symbiont and one for the receiver. They are respectvely UCX$SMTP_SYMB_SNAPSHOT_BLOCKS and UCX$SMTP_RECV_SNAPSHOT_BLOCKS. The value you assign to these logicals is the desired size of the snapshot buffer in VMS blocks (1 block being 512 bytes). Defining either one of these logicals for the given component(s) (symbiont and/or receiver) turns on the snapshot logging function. When turning this feature on you are still required to define the other SMTP diagnostic logicals that tell UCX SMTP what types of logging you want. (EG. For the symbiont you still want to define UCX$SMTP_LOG_LEVEL to 5.) Here are two examples of setting up symbiont and receiver snapshot logicals. This sets log level to 5 and turns on snapshot logging for the SMTP symbiont with a snapshot buffer of 200 blocks. $ DEFINE/SYSTEM UCX$SMTP_LOG_LEVEL 5 $ DEFINE/SYSTEM UCX$SMTP_SYMB_SNAPSHOT_BLOCKS 200 This sets all of the receiver diagnostics on and turns on snapshot logging for the receiver with a snapshot buffer of 200 blocks. $ DEFINE/SYSTEM UCX$SMTP_RECV_DEBUG 1 $ DEFINE/SYSTEM UCX$SMTP_RECV_TRACE 1 $ DEFINE/SYSTEM UCX$SMTP_RECV_SNAPSHOT_BLOCKS 200 Note that when you have snapshot buffering turned on for the symbiont it takes some time for the symbiont process to stop when you do a UCX STOP MAIL or stop the queue. This is because stopping the queue causes the symbiont to signal the "Symbiont stopping" condition which is not an error but which still causes the snapshot buffer to be dumped. The amount of time writing out the snapshot buffer takes depends on its size and the speed of the system and its disks. When the queue state is "Stopped" rather than "Stopping" the symbiont is process is really stopped. ******************************************************************************** * Miscellaneous Configuration logicals ******************************************************************************** ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_NO_SUBS_DOMAIN_INBOUND Type: Boolean Values: NA Tells UCX SMTP not to consider mail sent to the substitute domain as local. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_COMMON Type: String Values: NA This logical specifies cluster common directory where certain UCX SMTP files can be placed by the system manager. UCX SMTP will now look for .DIS files and for the local alias file in this directory if it is defined. It is the system manager's responsibility to define this logical before UCX SMTP startup and to create the directory and to move the .DIS files and/or local alias file there. If UCX$SMTP_COMMON is not defined then UCX SMTP will look for the files in SYS$SPECIFIC:[UCX_SMTP]. This logical may be a search list. You may wish to have UCX SMTP look at the clusterwide directory first and SYS$SPECIFIC:[UCX_SMTP] second. For example: $ DEFINE/SYSTEM UCX$SMTP_COMMON WORKDISK:[SMTP_DIS], - SYS$SPECIFIC:[UCX_SMTP] Note that other files like control files and logs are still kept in SYS$SPECIFIC:[UCX_SMTP] directory. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_JACKET_LOCAL Type: Boolean Values: NA This option affects the VMS mail From: line of mail delivered by the SMTP symbiont. The SMTP symbiont contains logic to detect when SMTP mail that originated on the local host is also being delivered to VMS mail on the local host. In this case the From: line text that the symbiont passes to VMS mail will contain only the username rather than SMTP%"username@host". Put another way, the symbiont won't put the VMS mail From: line in the SMTP% jacket on local-to-local mail. Since the VMS mail From: line contains only the username it is impossible to tell that the mail message went through SMTP simply from looking at the VMS mail From: line. This causes problems for the POP server which looks at the From: address when it's deciding whether or not it can safely ignore the VMS mail headers and, instead, use the SMTP RFC headers from the body of the mail. The UCX$SMTP_JACKET_LOCAL option tells the symbiont to put the SMTP jacket on local-to-local mail to short circuit the above described special case handling of local-to-local mail. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_INBOUND_NOXVMS Type: Boolean Values: NA This option affects the VMS mail To: and CC: lines of mail delivered by the SMTP symbiont. It tells the symbiont not to use the RFC X-VMS-To header (if any) as the text of the VMS mail To: line and the X-VMS-CC header (if any) as the text of the VMS mail CC: line. Instead the RFC To: and CC: lines are used. If UCX$SMTP_INBOUND_NOXVMS is *not* defined the SMTP symbiont will use the text of the X-VMS-To and X-VMS-CC lines (if present) for their respective VMS mail header lines. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_VMSDEF_TO Type: Boolean Values: NA This option affects the VMS mail To: line of mail delivered by the SMTP symbiont. It tells the symbiont not to pass any text for the To: line to VMS callable mail when delivering local mail. This causes VMS callable mail to use the default text for the To: line which will be the username. This option overrides UCX$SMTP_INBOUND_NOXVMS for the To: field. ******************************************************************************** * Special case logicals for MTS interaction ******************************************************************************** ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_MTS_ALLIN1 Type: Boolean Values: NA When relaying mail from the SMTP environment to MTS (aka. Message Router) the symbiont puts "UCX_SMTP" into the From field. This hack, the "UCX_SMTP hack", was needed because without it older versions of MR/MRGATE would purposefully bounce the mail with a "Return path too complicated" error. The UCX_SMTP hack is no longer needed provided that you are running MR V3.3A and MRGATE V3.3A and either UCX V3.2 ECO 8 or UCX V3.3 ECO 1 or UCX V4.0. Version 3.3A of MR/MRGATE will not bounce the mail and will pass it on to VMSmail with the nested double quotes. VMSmail will not reject the mail because of the nested double quotes in the from field but *the mail will be unreplyable*. With a future version of VMS mail it will be replyable. That is, nested double quotes will be supported by VMS mail. The UCX$SMTP_MTS_ALLIN1 option disables the UCX_SMTP hack. This will allow the correct From: address to be received by your All-in-1 users. In summary, if you want to short circuit the UCX_SMTP hack then you need to upgrade MR and MRGATE to version V3.3A and you need to define the UCX$SMTP_MTS_ALLIN1 logical before SMTP queue startup. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_REWRITE_MTS_FROM Type: Boolean Values: NA Many sites make heavy use of All-in-1 and MTS. They have all or many of their users forwarded to All-in-1 via MTS addresses. For example: MAIL> !All-in-1 MAIL> set forwa/user=bjarne MRGATE::A1::BJARNE This setup can cause problems when an SMTP mail comes in from the Internet, is forwarded to MTS and is replied to from the all-in-1 account. Either the reply is bounced once it makes its way onto the Internet or the mail makes it to the other end but can't be replied to because the return address has been corrupted. To fully understand what the trouble is an example is required. Say there is a UCX SMTP system with the host name xyz.com and on this system mail to user bjarne is forwarded to All-in-1 as above. Now say that user smith@abc.com sends an SMTP mail over the Internet to Bjarne@xyz.com. This mail will be relayed to the UCX xyz.com system and then the SMTP symbiont on xyz.com will hand the mail off to VMS mail which will hand it off to MTS (because of the forwarding for user bjarne) which will deliver it to the all-in-1 user on the other system. Now, say that the user BJARNE on that system decides to reply to the mail. If he does, the reply is sent via MTS back to the XYZ.COM system. At this point the SMTP symbiont on xyz.com must form a From RFC header for the mail to be sent into SMTP and out onto the Internet. It use the UCX SMTP SMTPfication procedure to do this. So the SMTP From header would look something like this: From: "MRGATE::SUMNOD::\"MRGATE::SUMNOD::BJARNE\""@XYZ.COM Once this is done the mail is sent out into the Internet and UCX SMTP is done with it. The problem is that, although this address is absolutely correct, most U*ix systems on the Internet do not have sendmail.cf files that can handle such addresses; the result, bounced mail and/or corrupted addresses. See "Miscellaneous SMTP/non-SMTP issues". The UCX$SMTP_REWRITE_MTS_FROM configuration logical allows the system manager to tell the symbiont that when it tries to form an address in the From field such as the one above it should parse the simple USERNAME out of the complex MTS address and append the local host name to that instead. It's a type of simple address rewriting. This way only a simple address gets sent out on the Internet. Since the user is forwarding their mail on XYZ.COM to MTS, a reply coming *back* to this user will be correctly relayed back to MTS. Given the above example where mail comes from the Internet to user Bjarne@xyz.com and is forwared to All-in-1 and replied to, the RFC From: header that UCX SMTP sends onto the Internet for the reply will no longer be: From: "MRGATE::SUMNOD::\"MRGATE::SUMNOD::BJARNE\""@XYZ.COM It will now be: From: BJARNE@XYZ.COM Now when the mail goes out into the net the reply path will be simple and cause no porblems. If the recipient replies the reply will come back to user bjarne on xyz.com and, because this user has their mail forwarded it will go to MTS and on to All-in-1 as desired. There is an important caveat to this feature which requires that it be used with caution. There is one scenario where it won't work right. Say a user on host xyz.com has their mail forwarded to SMTP%"mary@myhost.xyz.com" and that this user receives a mail from MTS (say it's from MRGATE::A1::JONES). If the UCX$SMTP_REWRITE_MTS_FROM logical is defined on xyz.com then user "mary" will receive the mail on myhost.xyz.com as being from JONES@xyz.com because the SMTP symbiont on xyz.com has parsed out just the username from the MTS From address. If user mary replies to this it will cause a mail to be sent to JONES@xyz.com. In this example note that we have not said whether or not user jones even has an account on xyz.com at all. If they do then the mail will be delivered to their account on host xyz.com. If the account is forwarded it will go to where it was forwarded to. If jones doesn't have an account though, which is likely, then the mail will be bounced. For this reason this feature is really only for use on sites that have most or all of their users forwarded to All-in-1 as above. ******************************************************************************** * Non-local mail routing options ******************************************************************************** "Non-local mail" is mail that is destined for another system rather than "local" mail which is to be delivered locally to VMS mail. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_ALTGATE_ALWAYS Type: Boolean Values: NA This option tells the symbiont to send all non-local mail to the alternate gateway. No zone check is performed. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_MX_IF_NOALTGATE Type: Boolean Values: NA Tells the symbiont to try to use MX records to get to a host if the alternate gateway cannot be reached. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Name: UCX$SMTP_NO_MX Type: Boolean Values: NA Ths option tells the SMTP symbiont not to use MX records to tell it where to route mail. Rather it will try to translate the domain part of each SMTP address into a host name and send directly to that address. Note that if the host name doesn't translate to an address the mail will bounce. If the host is not available then the mail will be requeued. ---------------------------------------- Miscellaneous VMS mail notes ---------------------------------------- -------------------- VMS mail native support for SMTP addresses -------------------- Version 6.2+ of VMS contains built in support for SMTP type addresses in the VMS mail To: line field without the SMTP% jacket. Eg: One can do MAIL> send To: jones@xyzorp.com rather than MAIL> send To: SMTP%"jones@xyzcorp.com" When mail is sent to an SMTP address without the SMTP% jacket it is still funneled to the MAIL$PROTOCOL interface just as it is when the SMTP% jacket is used. This new feature makes use of SMTP address more seamless. See VMS documentation for more information on VMS mail's native support for SMTP addresses. Also see the UCX User Guide chapter on SMTP. -------------------- Setting forwarding to an SMTP address -------------------- On most VMS versions one needs to set forward to an SMTP address as follows: MAIL> SET FORWARD SMTP%"""jones@xyzcorp.com""" More recent versions allow: MAIL> SET FORWARD SMTP%"jones@xyzcorp.com" or even MAIL> SET FORWARD jones@xyzcorp.com See your VMS documentation and release notes or try it yourself to see what your system supports. If you try it yourself always remember to do a SHOW FORWARD to see the result. Older systems which expect the triple double quotes on either side of the SMTP forwarding address will accept SET FORWARD SMTP%"jones@xyzcorp.com" but will result in a forwarding address of SMTP%JONES@XYZCORP.COM which then can't be parsed by VMS mail. What you want is a SHOW FORWARD result that shows one pair of double quotes around the address. Eg: MAIL> sho forw JONES has mail forwarded to SMTP%"jones@xyzcorp.com" On clusters with different versions of VMS installed on different cluster members there's no problem as long as you use the SMTP% jacket and the result of a SHOW FORWARD is as above. (Ie. one pair of double quotes.) All currently supported versions of UCX SMTP support users who've done a SET FORWARD directly to an SMTP address without the SMTP% jacket regardless of which VMS version is running. This support becomes an issue when UCX SMTP delivers mail to a user who has in turn forwarded their mail out via SMTP. See "Forwarding to an SMTP address". -------------------- Miscellaneous SET FORWARD/USER tricks -------------------- The VMS mail SET FORWARD/USER command allows one to create "aliases" for many different purposes. Its syntax is: MAIL> SET FORWARD/USER=alias-string forwarding-address The /USER qualifier requires SYSNAM to prevent unprivileged users from malisciously changing forwarding for other users. In the case where the SET FORWARD/USER capability is needed by an unprivileged user the user must simply request that the forward be set up on their behalf by a privileged user. The key to the SET FORWARD/USER command is that the alias-string need not be a username. Nor is it constrained by VMS username sytnax. The value can be up to 31 characters (compared with VMS username limit of 12 characters). Here are some examples of functionality that can be gained through use of the SET FORWARD/USER command and UCX SMTP: 1) Funnel mail for an alias to multiple users without a list file UCX SMTP supports use of SET FORW/USER to create an alias that points to an inline list of users. Say for example you wanted to create a "tech suport" alias for technical support for some reason. You want anyone to be able to send mail to this user (actually it's not a user, just an alias) on your system and have it go to all of the the technical support people. Further say there are at most only two or three peope in this role and you don't want to set up and SMTP mailing list for it. You can do this: MAIL> set forw/user=tech-support smtp%"""jones, smith@xyzcorp.com""" MAIL> sho forw /user=tech-support TECH-SUPPORT has mail forwarded to SMTP%"""jones, smith@xyzcorp.com""" Note that you can't do this with plain old VMS mail addresses since VMS mail doesn't support this. Eg. This can't be done MAIL> set forw/user=xxxx jones,smith %CLI-W-NOLIST, list of parameter values not allowed - check use of comma (,) \JONES,SMITH\ But you can do it like so: MAIL> set forw/user=xxxx SMTP%"""jones,smith""" MAIL> sho forw/user=xxxx XXXX has mail forwarded to SMTP%"jones,smith" 2) Use Firstname.Lastname syntax The alias-string can contain a dot (.) which makes it possible to receive mail where the local-part of the address contains a dot. One use of this is the Firstname.Lastname syntax. Eg: MAIL> SET FORWARD/USER=Beth.Jones SMTP%"""jones@xyzcorp.com""" 3) SHOW FORWARD/USER takes wildcards. MAIL> sho forw/user=j* Username Forwarding address JIM_KIRK ENTPRS::SPOCK JOE.BLOW SMTP%"jones@xyzcorp.com" JONATHAN.QUINCY.HORNBLOWER SMTP%"jonathan.quincy.hornblower@xyz.xyzcorp.com" JONES SMTP%"jones@xyzcorp.com" -------------------- The UCX SMTP cent sign hack -------------------- Many SMTP address formed by UCX SMTP for the RFC 822 From field use the double qoute character. For example: From: "SYSTM1::SMITH"@xyzcorp.com This has historically posed a problem when it comes time to put the RFC 822 From header into the VMS mail From: line. To put the RFC 822 From: address into the VMS mail From: line VMS mail (for most of its life) has required the address to be enclosed in the SMTP% jacket. It also has required that the address in the SMTP% jacket is itself enclosed in double quotes. Eg: From: SMTP%"user@dom.main" ^^^^^^ ^ VMS mail || | requires || | all of || | this --->--++----->------>--+ Finally, VMS mail (until only very recently) hasn't allowed nested double quotes. ** Given all this it was necesary to contrive some means to slip nested double quotes by VMS mail in the VMS mail From: line. If this wasn't done mail from SMTP with SMTP From: addresses containing double quotes would not be replyable since the VMS mail From: line would not be syntactically correct. To "mask" the nested double quotes UCX SMTP changes each one into a cent sign character before passing it to VMS callable mail as the VMS mail From: line. This has come to be known as the cent sign hack (not to be confused with the percent sign hack). Now an example to illustrate. The RFC 822 From header of: From: "SYSTM1::SMITH"@xyzcorp.com becomes the VMS mail "cent sign hacked" From: line of: From: SMTP%"˘SYSTM1::SMITH˘@xyzcorp.com" If a reply is done to this what happens? VMS mail invokes the UCX SMTP MAIL$PROTOCOL routine and passes this recipient address: ˘SYSTM1::SMITH˘@xyzcorp.com When parsing the recipient address UCX SMTP's MAIL$PROTOCOL code changes the cent signs back to double quotes and then continues normally. ** Well, more recent versions of VMS mail (VMS V7.?) have added support for nested double qoutes. It's in the UCX SMTP worklist to change our code so that you can either use the cent sign hack or some new code to properly nest double quotes. (This will be configurable so UCX SMTP can contiue to function with older versions of VMS mail that don't support nested double quotes.) --------------- User use of the cent sign hack --------------- Although the cent sign hack was invented for the reason above it can also be employed by users who need to send mail to an SMTP address that requires double quotes. (This is a rare occurence but it is still supported.) You can purposefully compose and address like this providing you know how to use your keyboard's compose key. The compose sequence for the cent sign is "C|". Working in a DECterm the use of the compose key is non-intuitive. To type a cent sign you need to 1) Hold down the compose key and then touch the space bar and release both keys. 2) type the c key 3) type the | key And voila - ˘ ---------------------------------------- UCX SMTP mailing lists ---------------------------------------- A mailing list is a list of addresses contained in a file that can be sent to as if it were a simple username. In UCX SMTP the name for a mailing list is the name of the file itself. UCX SMTP mailing list files can reside in onw of two places - either SYS$SPECIFIC:[UCX_SMTP] or UCX$SMTP_COMMON. The file must have a .DIS extension and should be readable by the UCX_SMTP account. @@@ The entries in the local alias file must adhere to the following syntax rules. - There may be only one address entry per line. - The address must be SMTP addresses, not VMS addresses. - No SMTP% jacket - To get DECnet address in a UCX SMTP.DIS file enter it as: <"NODNAM::USER"> The <" and "> are required. - The comment character, a "!", must appear in the first column. -------------------- VMS Majordomo -------------------- Majordomo is a popular Unix based mailing list manager program it. It automates maintenance of lists to a large degree. It also pre-processes each mail sent to this list and allows for: - Canned headers and text to be appended to each message sent to the list. - Users to subscribe and unsubscribe themselves from the list. - Moderated lists - Digests With a digest messages sent to the list are saved and compiled into a single message - a "digest" - and then mailed out as one large mail message. - Archives - users can see directories "served" by Majordomo and fetch files via email Majordomo has been ported to VMS/UCX and is available for use as *unsupported* software. The get more information simply send a mail message to majordomo@ucx.lkg.dec.com The subject of the mail doesn't matter. In the body of the mail enter info vms-majordomo-users ---------------------------------------- Miscellaneous Stuff ---------------------------------------- -------------------- Queue watcher program -------------------- In cases where queues are crashing it is desirable to have a workaround to let mail keep flowing without constant human intervention. UCX SMTP provides such a feature in the form of an unsupported "queue watcher" program called UCX$RESTART_SMTPQ.COM which is supplied, as is, in the UCX$EXAMPLES directory. This procedure file watches your SMTP queues and restarts them it it finds them crashed. It runs as a self-perpetuating batch job. Among it's features are: It won't restart your SMTP queues when you have shut down UCX SMTP. It will find the likely "culprit job" that caused the queue to crash and put that job on hold so that its simply doesn't come back into the queue and crash the queue again. It will optionally check the symbiont process's page file count and gracefully stop it if it is running low. (This is the case where there is a memory leak.) After the queue stops it will restart the queue transparent to you. Stopping and restarting the queue stops the old symbiont process and starts a new one with a fresh, full sized page file count again. This lets mail keep flowing in spite of memory leaks. being incurred in the symbiont process. The UCX$RESTART_SMTPQ.COM procedure file contains instructions on its use.  -------------------------------------------------------------------------------- Terms -------------------------------------------------------------------------------- address: See mail address. body: The part of the mail after the headers, separated from the headers by a single blank line. See "SMTP Message Format Basics" and "Format of VMS mail message". bounced mail: A mail message composed by an MTA indicating that another message which the MTA was charged with delivering couldn't be delivered. A bounced mail message (aka. undeliverable mail notification) may contain sections: - that show what went wrong (often an SMTP protocol dump or an error returned by the MTA's host mail system), - that show which addresses were successfully delivered and which weren't - that contain the text of the undelivered mail message including the headers. cent sign hack: The UCX SMTP practice of changing a double quote in an SMTP address to cent sign character (˘) before passing it to VMS mail as the From: line to get around VMS mail's inability to handle nested double quotes. CF: Acronym for UCX SMTP control file. control file: The binary file into which UCX SMTP packages up each SMTP mail message. See "UCX SMTP Control File". The sequence of ASCII characters composed of a carriage return (decimal 13) followed by a line feed (decimal 10). domain: The part of a mail address after the "@". Also called the right hand side or the host. In the address "jones@XYZCorp.COM" the string "XYZCorp.COM" is the domain. See "SMTP Mail Addresses". envelope: The source and destination addresses sent in the SMTP protocol's MAIL FROM and RCPT TO commands. The destination address or addresses from the RCPT TO command(s) may match certain headers contained in the mail message but are not required to. The Return-Path header appended to the mail message's headers at delivery time must take the address of the MAIL FROM command. See "SMTP Protocol Overview". foreign transport: A VMS mail term. A foreign transport is a mail transport other than VMS mail's native MAIL11 protocol through which users want to send mail. To cause a mail message to be sent over a foreign transport the foreign transport mail address in the VMS mail To: line is prefixed with a string which is the name of the foreign transport and a percent sign (%). Common examples are SMTP% and NM%. See "VMS mail Foreign Transport/ MAIL$PROTOCOL Interface". headers: The initial lines of a mail message which mus comply with a pre-defined format. SMTP and VMS mail define different legal headers. For information on SMTP headers (aka. RFC 822 headers) see "SMTP Message Format Basics". For information on VMS mail headers see "Format of VMS mail message". The syntax of SMTP headers is specified fully in RFC 822. local-part: The part of a mail address before the "@". Also called the left hand side or the user. In the address "jones@XYZCorp.COM" the string "jones" is the local-part. See "SMTP Mail Addresses". local mail: Any mail message being sent to an address where the domain part of the address is such that UCX SMTP must deliver the mail locally via callable VMS mail to the local-part of the adrdress. See "Local vs. non-local SMTP mail". Mail11: The VMS/DECnet mail protocol. MAIL$PROTOCOL Interface: The procedure call interface that VMS mail uses to pass a mail message to a foreign transport. mail message: The mail text or "payload" that is transfered by the MTA and delivered by the MUA to the destination mail address. It is not to be confused with the "envelope". The text of the mail message consists of the headers followed by a blank line followed by the body. See "header" and "body". The format of a mail message is specified in RFC 822 and enhanced by the MIME RFCs. See "SMTP Message Format Basics" and "What is MIME?". mail address: The text string used to represent a point to which an MTA delivers a mail message or to identify the point from which the mail came. Syntax is defined in RFC 822, Chapter 6.1. In its simplest form an address appears as the familiar "user@host". Rather than use the terms user and host the terms "local-part" and "domain" are used here. Mail User Agent: See MUA. Message Transfer Agent: See MTA. MUA: The Mail User Agent. The user interface component of a host's mail system. With it users read, compose and file their mail messages. MTA: Message Transfer Agent. The software that implements the SMTP protocol (client and server) to transfer mail between an originating host and a destination host. MX record: A DNS maintained resource record used by SMTP MTA's to assist in the routing of mail. See "MX Record Overview". non-local mail: Any mail message being sent to an address where the domain part of the address is such that UCX SMTP must relay the message to another SMTP host. See "Local vs. non-local SMTP mail". orphan control file: A control file for which there is no corresponding entry in an SMTP queue. This can occur when a catastrophic error was incurred while processing the control file (like a corrupt control file) *or* when the message could neither be delivered, nor bounced nor delivered to the UCX_SMTP account. See "Undeliverable Mail". outbound mail: Synonym for non-local mail. percent sign hack: Aka. BBN hack, %-hack, See "Source Routing". reply code: The three ASCII digit string sent at the start of each reply sent by the SMTP server to the client. These are explained in detail in Appendix E of RFC 821. See "SMTP Server Replies". RFC: Request For Comments. In the Internet world the specifications that define standards, like the SMTP protocol, are called RFC's. RFC 822 headers: See definition of headers. SMTP% jacket: The SMTP text around an SMTP address that identifies it as such. For most VMS systems this is the SMTP% plus the additional double quotes. Eg. SMTP%"insert@SMTP.address.here" ^^^^^^ ^ || | ++-<- SMTP% jacket -->--->--+ SMTPfication: The process of transforming any non-SMTP mail address into an SMTP legal mail address. This is strictly a UCX SMTP term. See "SMTPfication of non-SMTP addresses". source routing: The capability for mail addresses to explicitly specify the route the mail message should take as part of the address itself. See "Source Routing". stray control file: Synonym for orphan control file.