.RIGHT MARGIN 67 .CENTER Datatrieve-11 to VAX-Datatrieve Conversion Panel .BLANK 2.CENTER Joe H.#Gallagher .BLANK.CENTER Research Medical Center .BLANK.CENTER Kansas City, MO 64132 .BLANK 2.CENTER Bart#Z.#Lederman .BLANK.CENTER ITT World Communications .BLANK.CENTER New York, N.Y. 10004-2464 .BLANK 2.CENTER Session Chairman: .BLANK.CENTER Alex L.#Lamb Jr. .BLANK.CENTER USASATCOMA .BLANK.CENTER AMCPM-SC-4G .BLANK.CENTER Ft.#Monmouth, N.J. .BLANK 2.CENTER Transcribed by B.#Z.#Lederman .BLANK .PARAGRAPH This session was originally planned to be a recounting of users' experiences in converting to VAX-DTR from DTR-11. Due to various difficulties which arose between scheduling the session and the symposium (such as non-functioning equipment, and jobs being eliminated), the final panel had to speak partially from experience and partially on generic terms. Nevertheless, many important points were covered, and the information was judged to be quite useful by those attending the session. This is not intended to be an exact transcription of the session: rather, it simply presents the information in a readable form. I will not attempt do distinguish which person made what contributions. I would also like to acknowledge the presence of Suellen Harris and Bill Opalka from DEC, who sat in the audience and kept us from going astray. .BLANK 2.CENTER Major Points. .BLANK.CENTER DTR-11 is an almost perfect subset of VAX-DTR. .PARAGRAPH DTR-11 is a nearly perfect subset of VAX-DTR. (It was, of course, developed first, and VAX-DTR was developed later using the same syntax so users could migrate.)# The primary point of difficulty has to do with a difference in how the hardware addresses memory. The PDP-11 cannot address a word which starts on an odd byte boundary, while the VAX can: therefore, such data types as REAL, DOUBLE, INTEGER, DATE, and other word, double word, and quad word variables, must be aligned to word boundaries on the PDP-11. DTR-11 will automatically do this by inserting invisible bytes where needed. Consider the following record definition: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 5 01 RECORD. 10 FIELD1 PIC X. 10 FIELD2 PIC 99 USAGE IS INTEGER. ; .BLANK.JUSTIFY.FILL FIELD1 is one byte long, and FIELD2 is two bytes long. In VAX-DTR, this record would be 3 bytes long: on the PDP-11, it will be 4 bytes long, as an extra byte has to be inserted between the two fields to make FIELD2 start on an even byte boundary. DTR-11 will do this wherever necessary (and only where necessary), and does not issue warnings or error messages. This will become apparent if the data files are moved from one system to another, and the same record definition is used on the VAX without taking precautions. It is possible to make VAX-DTR allocate on word boundaries by using the clause .BLANK "ALLOCATION#IS#LEFT-RIGHT" .BLANK within the record definition; therefore, you should change the record definition to: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 8 DEFINE RECORD TEST__REC USING ALLOCATION IS LEFT-RIGHT 01 RECORD. 10 FIELD1 PIC X. 10 FIELD2 PIC 99 USAGE IS INTEGER. ; .BLANK.JUSTIFY.FILL If you do this, VAX-DTR will allocate space in the same way as DTR-11, and there should be no problems. If you don't, you may get an error message similar to "RECORD LENGTH DOES NOT MATCH" when you try to READY the new domain on the VAX, which is an indication that you may have an alignment problem. See also the section on moving data below. .PARAGRAPH If you do get these messages, and don't do anything about it before storing new data, you will corrupt the data file. It would be a good precaution to be certain that the first time you READY the domain after moving the data that you do so READ#ONLY, and look at the data. If there are any alignment problems, you will immediately see that data isn't coming out correctly, and can take corrective measures. .BLANK 2.CENTER Planning the move: more dictionary options. .PARAGRAPH The Common Data Dictionary has more functionality than the Dictionaries on the PDP-11, and some thought should be given to using this. On the PDP-11, there is often a separate dictionary for each application, and the data files often aren't shared. On the VAX, all definitions go into the CDD, and makes sharing easier. The CDD has a hierarchical structure, and if you have many separate projects you will want to plan which sub-directories should hold what definitions, and which definitions should go into a site common directory (for everyone to use), which should go into project wide directories, and which may be left in individual directories. The protection options are also greater on the VAX than on the PDP-11: if you are not using protection now, then you can move with no protection without making any changes. If, however, you have been using UIC type protection on the PDP-11, then you may want to change, as it is unlikely that you will keep the same UIC scheme on your VAX disks that you had on your PDP-11 disks. (VMS does have UICs, but most people go to named directories and hierarchical sub-directories, as these tend to be easier to use and more closely follow the way most data is organized.) Therefore, you may want to change UIC and/or password protection, and use some of the new protection options available in VMS. .PARAGRAPH You do not have to learn much about the CDD before moving if you do everything in DTR. You can put everything into one dictionary like DTR-11, but you will get better performance if you make a good division from the beginning. You can move definitions from one sub-section to another later, but it's best to make some plans first. A quick read-through of the manual to understand the concepts, or attending some DECUS sessions are good ideas. For the new user, no new training is needed to begin with, as they will see a dictionary just like what they are used to (with the addition of version numbers), and can see one big dictionary or their own dictionary, whichever matches the PDP-11 installation; but the person planning the move and organizing the application should know something about the CDD. The CDD structure can copy the existing PDP-11 setup, but usually matches the company or application organization. Arranging the dictionary well has both performance and maintenance advantages. Note also that VAX-DTR can have startup files for individual users just as DTR-11 has, to put individuals into the proper dictionary and perform READYs or start procedures, as needed: also, the dictionary that the user sees for PLOTS can be different from that for all other dictionary elements, so the users can get all of their domains, records, procedures, and tables from an individual dictionary, and at the same time get the PLOTS from the system wide dictionary. You might also give them a procedure to set their dictionary into the system wide DEMO library if you want them to play with YACHTS, EMPLOYEES, or other sample files supplied with DTR. There is also a command procedure NEWUSER.COM that comes with DTR that can be used to easily set up a dictionary for a new user. In addition, there is also a logical name that designates the CDD dictionary in which the user will start. .BLANK 2.CENTER Moving the data. .PARAGRAPH One must move both the data files, and the dictionary objects (record definitions, domain definitions, tables, etc.). You can EXTRACT individual definitions, but DTR-11 is supplied with a utility called QXTR, which will extract all of the definitions in a dictionary at one time into a single command file in a manner similar to EXTRACT#ALL in VAX-DTR. This utility inserts the "ALLOCATION#IS#LEFT-RIGHT" clause into the record definitions, which should simplify the move considerably. [This is probably true only for DTR-11 V3.0 and V3.1, and not for earlier versions of DTR. QXTR is not supplied with PRO-DTR. DTR-11 itself does not insert the ALLOCATION clause when extracting.]# If you are using individual dictionaries for different users, you must extract each dictionary individually. QXTR also gives you the option of extracting the protection qualifiers, or not, as you choose. This single file can then be moved to the VAX and invoked to re-create all of your records, tables, domains, etc. One factor not covered in the original talk is that explicit references to disks and/or directories in domain definitions will probably have to be changed, as your device names will change, and you will probably be using symbolic device names and/or named directories, as mentioned before. To actually get the information across, there are several options. .BLANK.CENTER Moving data on disks. .PARAGRAPH If you are using RSX (which in this paper includes 11M, 11M-Plus, IAS, and possibly 11D), and you have the same kind of disk drive on both systems, or are going to move the disk drive to the new system, then you can read your old disks on VMS. You will want to copy the information to a new disk, to take advantage of some VMS features like named directories, but the files can be read while on the old disk, or copied. This applies both to the data files, and the file created by QXTR (or the DATATRIEVE EXTRACT command). If you are running RSTS/E, then you are out of luck: no other operating system will read your disks. If you don't have a disk type in common on both systems and are not moving the devices, you might want to consider plugging in your old disk on the VAX just long enough to copy the data, if possible. If not, then the local DEC office may be able to copy the disks, or your local DECUS LUG may help you find a user who may let you do it, or you may be able to locate a commercial service company that will do it. .BLANK.CENTER Moving data by Network. .PARAGRAPH If both systems run DECNet (no matter what the operating system), then any file that DATATRIEVE can read can be read over the network. You can COPY (or NFT or FTS) the file with the definitions to the VAX, define your domains, copy the data file, and be ready to go. Alternatively, you can define a two domains with the same record definition, with one having a normal file specification on the VAX, the second having a file specification which includes the node name for the PDP-11 (something you can do with VAX-DTR but not DTR-11); or what may be better, if you have remote DATATRIEVE installed on the PDP-11 you can use the "READY domain AT node" feature of VAX-DTR, and simply read from the old domain on the PDP-11 to the new domain on the VAX. This will be a little slow, as DTR is not optimized for this type of operation the way DECNet utilities are, and the data will be going over the network, but hopefully it will only be done once. The advantage of doing this is that it can avoid the problems of record alignment mentioned before. By using the DTR-11 remote server on the PDP-11, the data will be read exactly as it has always been read: on the VAX side, you can define a record without the ALIGNMENT clause, and pack the data in without worrying about hidden bytes. You may also want to consider other methods of converting data mentioned below. Remote DATATRIEVE is available starting with V3.0, not in earlier versions. If you are not using DECNet, you may not have special communications devices normally used for networking. DECNet will work over normal asynchronous lines used for terminals, however. You may have to give up 8 or 16 lines for a while, as DECNet grabs the entire device on PDP-11s, and the maximum speed may be 9600 baud so transfers may be a little slow, but hopefully you will only do the conversion once. If you don't have DECNet, maybe you can persuade the local office to let you use it just for a few days while you move your data. .BLANK.CENTER Moving data on Magnetic Tape. .PARAGRAPH If both systems have magnetic tape (or you are moving the tape drive), you may be able to store the definitions and information on tape. Once again, the RSX family provides the best compatibility, with most systems writing ANSI tapes which can be read by VMS. (On earlier RSX systems, this was a SYSGEN option, so check carefully before you write out the tapes and disconnect your PDP-11.)# If not, you can write DOS-11 format tapes on all RSX family systems which can be read on the VAX with FLX (in compatibility mode) or the new EXCHANGE utility. Though you might possibly get indexed files over if transferred in image mode it will be much safer if you first convert the data to a sequential file and re-construct it on the VAX, as will be mentioned again later. If you have RSTS/E, then you are again stuck with a system which works differently than everyone else. You might be able to generate DOS-11 format tapes, or you may want to upgrade to V9.0: this is the latest version of RSTS/E, and it includes a utility which writes tapes that are compatible with the VMS BACKUP utility. For RSTS/E users this will probably be the easiest way to move files, though V9.0 is a new release and those of us on the panel have not yet had any feedback from users. Moving data from RSTS/E to other systems can be so much of a problem that some people plan to upgrade to RSTS/E primarily for the purpose of being able to use the new utility to write VMS readable tapes with the new utility. .PARAGRAPH There is an RMS utility set, BCK and RST, which is intended for backing up files to and restoring files from magnetic tape. These utilities automatically convert indexed files to a form which will store properly on magnetic tape, and place error checking and other useful information on the tape for all types of files. Because they are supplied as part of RMS, they should be available on all PDP-11 systems (that can run DATATRIEVE). The problem is that VMS may not have a utility corresponding to RST to get the data off: so unless you have compatibility mode on your VMS system (and it is now an optional layered product), this may not be a viable option. .BLANK.CENTER Other file transfer methods. .PARAGRAPH If none of the above are available, you may want to look into some of the communications packages which will operate over normal serial (terminal) lines. KERMIT is a public domain program available from the DECUS library (and elsewhere) which runs on virtually any computer and operating system, will work on serial lines, does error checking, and can transfer (in most cases) both text and binary files. It may be slow, but it will work. There are also other communications packages that perform similarly: you may even be able to use SET#HOST/DTE/LOG on the VAX to go in to the PDP-11 and type the file out (this works best if the data is all text) or a similar "dumb" text transfer. .BLANK 2.CENTER Re-Organizing your data. .PARAGRAPH VMS has some options not available on the PDP-11 for indexed files, notably Prolog-3, which allows some space saving. Since indexed files should be re-organized occasionally for best performance, and since the same data will occupy less space in a sequential file and take less time to transfer over networks, and transfer more easily on magnetic tape, the time of transfer would also be a good opportunity to re-organize and optimize the data file by converting it to sequential, and re-populating an indexed file on the VAX. Even if you can move your disk packs and can read your old files directly, re-organizing at this point is a good idea, though in this case you don't have to convert to an intermediate sequential file. .PARAGRAPH First, you need a description of the current indexed file: you can simply make a note of the record length (which you get from DTR when you define the record), and figure out where the keyed fields are in the record, but there are two RMS utilities, the older DSP and the newer DES which are designed to record file characteristics and define new files: it's a good idea to have such a description file for documentation purposes, even if you aren't anticipating a conversion. DES creates a file definition which is quite similar to that used by the VMS FDL utility: in actual tests, I was very surprised to find that FDL will actually read the descriptor file produced by DES! It may give you a few warning messages, especially if the SOURCE and TARGET fields are empty, but this shouldn't cause any serious problems. If the utility refuses to read your descriptor file and aborts, check to see that when you transferred the description file over it did not pick up trailing blanks on the description items, and that the file position qualifier says NONE with no numbers trailing. You can use a regular text editor such as EDT to go over the file before reading it with EDIT/FDL if necessary to touch it up. If you don't take this approach, then you can move the record definition, define a domain and define a file: DTR will create a file that matches the record definition. You can then look at that file with EDIT/FDL if you wish, or do the optimization shown below. If you move your disk packs FDL can probably read your original data file, and may also be able to do this over the network if you are using DECNet. .PARAGRAPH Converting the data file from indexed to sequential on the PDP-11 is quite simple: the CNV utility will perform this conversion by default. Simply give the name of the indexed file on input, and the file name you want for the sequential file on output. .PARAGRAPH Once the file description and the data file are on the VAX, you can populate the file with CONVERT: however, now would be a good time to review the file design for possible performance improvements. the FDL utility (EDIT/FDL) has an OPTIMIZE script which can make the decisions a little easier, as it will look at your data file and give you some information about what can be done to improve access. The one factor which comes up quite often is bucket size: on the PDP-11, bucket size is almost always the smallest possible value that will hold the record size you are using, as larger buckets use up pool space. On the VAX, this is no longer a problem, and larger bucket sizes are a viable option. If you often retrieve records which are next to each other, such as retrieving a record by the primary key and then reading the next several records in order, then a larger bucket size may improve performance. If, however, you retrieve records scattered all over the file in no particular order, then a larger bucket size won't help and may hurt, but a different index structure may be of benefit. If you don't know what options to take, use the FDL utility and let it optimize: it will usually take reasonable options. Once this is done, you can populate a new file again with CONVERT. .BLANK 2.CENTER Combining related domains. .PARAGRAPH Another consideration in DTR-11 is that the data is sometimes separated into several domains rather than one, to prevent the record definition and buffers from getting so large as to use up all of your pool space. When converting to VAX-DTR, you may want to recombine them by moving all of the individual files (and domain definitions) over, define a VIEW to combine them and a single record definition with all of the fields, and read from one into the other. This may be a little slow, but again, this will only be done once. You will then want to use FDL as described before to optimize the new combined file. .BLANK 2.CENTER Reports. .PARAGRAPH For reports, the VAX-DTR works very much the same as DTR-11. If you have reports which are adjusted so that field breaks occur just at the beginning or end of a page, or are otherwise 'finagled' to match a pre-printed form, you may find that you have to do a little adjustment to the lines per page or number of lines skipped qualifiers, but most straight forward reports will work with no changes. .BLANK 2.CENTER Expanded features in VAX-Datatrieve .BLANK.CENTER Tables. .PARAGRAPH While VAX-DTR has dictionary tables just like DTR-11, it also has domain tables: this allows data in a domain to also be accessed like a table. One field (preferably a keyed field for good performance) takes the place of the "left" side of the table, and another field (any one in the domain) takes the place of the table entry on the "right". Dictionary tables are faster for small tables: domain tables can be faster for large tables if keyed fields are used, can take the place of several tables, and can also be accessed as a regular domain, which makes changing the table the same as modifying the data in any other domain; therefore, tables which are modified often are easier to maintain as domain tables. .BLANK.CENTER Functions. .PARAGRAPH VAX-DTR has all of the functions (MIN, MAX, TOTAL, etc.) that DTR-11 has, plus several more (standard deviation, and running count, for example). In addition, there is a new set of functions of the FN$xxx family which can do things like convert lower case text to upper case, move sub-sections of text strings, format output strings, get system information, do mathematical operations, and many other functions. If that isn't enough, you can add your own functions. In many cases, things that you are doing, perhaps with some difficulty, with procedures or COMPUTED#BY clauses may be much easier with the extra functions available in VAX-DTR, and a review of what you are doing may reveal some areas where improvements may be made. Though you may want to wait until your application is moved over and running, this is an area where you will want to do some work quite soon after the move .BLANK 2.CENTER ^&No Pool Space Restrictions!\& .PARAGRAPH In DTR-11, procedures are often broken into small pieces to conserve pool space, and more things were done in command files. You may consider moving more of the work into DTR procedures, especially when you can get large blocks into single WHILE or BEGIN-END blocks. This may take a little longer to compile, but once compiled will execute faster than separate blocks. .PARAGRAPH In DTR-11, especially if your application was just on the edge of available pool space, things tended to be pared down to the minimum, especially in record definitions. Variable names were very short, edit-strings removed, etc. On the VAX, you have the opportunity to put more descriptive field names back in, and make the fields more self-documenting, put in query names, edit strings, and so on. Don't drop your good habits, however. Some users think they have infinite space on the VAX, and don't FINISH unused domains or RELEASE unused variables. This will cost you something eventually on the VAX (usually in paging), and it won't be as obvious as running out of pool space was in DTR-11: your application, or the whole system, just slowly degrades. You also may interfere with other users who want to access the data if you are keeping files open and are possibly locking records. You can keep domains open if you expect to use them again and save the time it would take to READY the domain, but when you are done with a domain you should FINISH it. .BLANK 2.CENTER Other improvements. .PARAGRAPH All users will be happy to learn that VAX-DTR uses EDT when editing, rather than the built-in editor of DTR-11. If you have an EDTINI.EDT command file in your account, it will be used when you edit something in VAX-DTR; keypad and all other commands will work, etc. You will also notice that with the newer versions of VAX-DTR, you have version numbers on definitions, so you can keep the old definition around until you find that the new version works. Users should be reminded to purge out old versions regularly. .PARAGRAPH An option available in VAX-DTR is the use of FMS or TDMS to have form driven screens. After your data is on the VAX and working, you may want to start thinking of converting some of your old procedures, especially if they were working like menus, to form driven screens. Something you do need to consider before you do this, however, is whether to buy FMS or TDMS: they look very much the same to DTR, so the choice is often determined by what other software you are using. Some packages will require one or the other, (for example, All-In-1 uses FMS) while DTR can work with either, one at a time. .PARAGRAPH There are various other features that are present in VAX-DTR (three types of concatenation rather than two, CROSS statements, the CHOICE statement, and all of the graphics capabilities), which you will soon discover and will want to incorporate into your applications; but none are needed immediately for conversion, and you can wait to learn about them until after the problems of conversion are over.