 










              Performance Evaluation of Transaction Processing Systems


              By Walter H. Kohler, Yun-Ping Hsu, Thomas K. Rogers, Wael H.
              Bahaa-El-Din



          Abstract                                Transaction processing
              Performance and price              systems are complex in
             /performance are important          nature and are usually
             attributes to consider when         characterized by a large
             evaluating a transaction            number of interactive
             processing system. Two              terminals and users,
             major approaches to                 a large volume of on-
             performance evaluation              line data and storage
             are measurement and                 devices, and a high
             modeling. TPC Benchmark             volume of concurrent and
             A is an industry standard           shared database accesses.
             benchmark for measuring             Transaction processing
             a transaction processing            systems require layers of
             system's performance                software components and
             and price/performance.              hardware devices to work
             Digital has implemented             in concert. Performance
             TPC Benchmark A in a                and price/performance are
             distributed transaction             two important attributes
             processing environment.             for customers to consider
             Benchmark measurements were         when selecting transaction
             performed on the VAX 9000           processing systems.
             Model 210 and the VAX 4000          Performance is important
             Model 300 systems. Further,         because transaction
             a comprehensive analytical          processing systems are
             model was developed and             frequently used to operate
             customized to model the             the customer's business
             performance behavior of TPC         or handle mission-critical
             Benchmark A on Digital's            tasks. Therefore, a certain
             transaction processing              level of throughput and
             platforms. This model was           response time guarantee are
             validated using measurement         required from the systems
             results and has proven to           during normal operation.
             be an accurate performance          Price/performance is
             prediction tool.                    the total system and
                                                 maintenance cost in
             Introduction                        dollars, normalized by the
                                                 performance metric.






                          Digital Technical Journal Vol. 3 No. 1 Winter 1991  1

 


           Performance Evaluation of Transaction Processing Systems



              The performance of a                Modeling uses simulation
             transaction processing              or analytical modeling
             system is often measured            techniques. Compared to
             by its throughput in                the measurement approach,
             transactions per second             modeling makes it easier
             (TPS) that satisfies a              to produce results and
             response time constraint.           requires less computing
             For example, 90 percent             resources. Performance
             of the transactions must            models are also flexible.
             have a response time that           Models can be used to
             is less than 2 seconds.             answer "what-if" types
             This throughput, qualified          of questions and to provide
             by the associated response          insights into the complex
             time constraint, is called          performance behavior of
             the maximum qualified               transaction processing
             throughput (MQTh). In a             systems, which is difficult
             transaction processing              (if not impossible) to
             environment, the most               observe in the measurement
             meaningful response time            environment. Performance
             definition is the end-              models are widely used in
             to-end response time,               research and engineering
             i.e., the response time             communities to provide
             observed by a user at a             valuable analysis of
             terminal. The end-to-end            design alternatives,
             response time represents            architecture evaluation,
             the time required by all            and capacity planning.
             components that compose             Simplifying assumptions
             the transaction processing          are usually made in
             system.                             the modeling approach.
              The two major approaches           Therefore, performance
             used for evaluating                 models require validation,
             transaction processing              through detailed simulation
             system performance are              or measurement, before
             measurement and modeling.           predictions from the models
             The measurement approach is         are accepted.

             the most realistic way of            This paper presents
             evaluating the performance          Digital's benchmark
             of a system. Performance            measurement and modeling
             measurement results from            approaches to transaction
             standard benchmarks have            processing system
             been the most accepted form         performance evaluation. The
             of performance assessment           paper includes an overview
             of transaction processing           of the current industry
             systems. However, due               standard transaction
             to the complexity of                processing benchmark, the
             transaction processing              TPC Benchmark A, and a
             systems, such measurements          description of Digital's
             are usually very expensive,         implementation of the
             very time-consuming, and            benchmark, including the
             difficult to perform.               distinguishing features of
                                                 the implementation and the
                                                 benchmark methodology. The


          2  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                       Performance Evaluation of Transaction Processing Systems



                                                  The benchmark can be
                                                 run in either a local
                                                 area network (LAN) or a
                                                 wide area network (WAN)
                                                 configuration. The related
             performance measurement             throughput metrics are
             results that were achieved          tpsA-Local and tpsA-Wide,
             by using the TPC Benchmark          respectively. The benchmark
             A are also presented.               specification defines
             Finally, a multilevel               the general application
             analytical model of the             requirements, database
             performance behavior of             design and scaling rules,
             transaction processing              testing and pricing
             systems with response time          guidelines, full disclosure
             constraints is presented            report requirements, and
             and validated against               an audit checklist.[1]
             measurement results.                The following sections

                                                 provide an overview of the
          TPC Benchmark a-an Overview            benchmark.

              The TPC Benchmark A                Application Environment

             simulates a simple banking           The TPC Benchmark A
             environment and exercises           workload is patterned
             key components of the               after a simplified banking
             system under test (SUT)             application. In this model,
             by using a simple, update-          the bank contains one
             intensive transaction type.         or more branches. Each
             The benchmark is intended           branch has 10 tellers and
             to simulate a class of              100,000 customer accounts.
             transaction processing              A transaction occurs when a
             application environments,           teller enters a deposit
             not the entire range of             or a withdrawal for a
             transaction processing              customer against an account
             environments. Nevertheless,         at a branch location. Each
             the single transaction              teller enters transactions
             type specified by the               at an average rate of one
             TPC Benchmark A standard            every 10 seconds. Figure 1
             provides a simple and               illustrates this simplified
             repeatable unit of work.            banking environment.














                          Digital Technical Journal Vol. 3 No. 1 Winter 1991  3

 


           Performance Evaluation of Transaction Processing Systems



             Transaction Logic                   an account, updates the

              The transaction logic              current cash position of
             of the TPC Benchmark A              the teller and branch,
             workload can be described           and makes an entry of the
             in terms of the bank                transaction in a history
             environment shown in Figure         file. The pseudocode shown
             1. A teller deposits in             in Figure 2 represents the
             or withdraws money from             transaction.

             Terminal Communication              o  The tested system must
              For each transaction, the             preserve the effects of
             originating terminal is                committed transactions
             required to transmit data              and ensure database
             to, and receive data from,             consistency after
             the system under test. The             recovering from

             data sent to the system                -  The failure of
             under test must consist of                a single durable
             at least 100 alphanumeric                 medium that contains
             data bytes, organized as                  database or recovery
             at least four distinct                    log data

             fields: Account_ID, Teller_            -  The crash and reboot
             ID, Branch_ID, and Delta.                 of the system
             The Branch_ID identifies               -  The loss of all or
             the branch where the teller               part of memory
             is located. The Delta is
             the amount to be credited           o  Eighty-five percent of
             to, or debited from, the               the accounts processed
             specified account. The data            by a teller must belong
             received from the system               to the home branch
             under test consists of                 (the one to which the
             at least 200 data bytes,               teller belongs). Fifteen
             organized as the above                 percent of the accounts
             four input fields and                  processed by a teller
             the Account_Balance that               must be owned by a
             results from the successful            remote branch (one to
             commit operation of the                which the teller does
             transaction.                           not belong). Accounts
             Implementation Constraints             must be uniformly
                                                    distributed and randomly
              The TPC Benchmark A imposes           selected.

             several conditions on the           Database Design
             test environment.
             o  The transaction                   The database consists
                processing system must           of four individual files
                support atomicity,               /tables: Branch, Teller,
                consistency, isolation,          Account, and History,
                and durability (ACID)            as defined in Table 1.
                properties during the            The overall size of the
                test.                            database is determined by
                                                 the throughput capacity of
                                                 the system. Ten tellers,
                                                 each entering transactions
                                                 at an average rate of

          4  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                     Performance Evaluation of Transaction Processing Systems



                                                store the history records
            one transaction every 10            generated during 90 eight-
            seconds, generate what is           hour days of operation at
            defined as a one-TPS load.          the published system TPS
            Therefore, each teller              capacity. For a system that
            contributes one-tenth (1            has a processing capacity
            /10) TPS. The history area          of x TPS, the database is
            must be large enough to             sized as shown in Table 2.

                                        Table 1

                                   Database Entities

          ___________________________________________________________________
                        Fields
          Record__Bytes_Required_______Description___________________________

          Branch  100   Branch_ID      Identifies the branch across the
                                       range of branches

                        Branch_        Contains the branch's current cash
                        Balance        balance

          Teller  100   Teller_ID      Identifies the teller across the
                                       range of tellers

                        Branch_ID      Identifies the branch where the
                                       teller is located

                        Teller_        Contains the teller's current cash
                        Balance        balance

          Account 100   Account_ID     Identifies the customer account
                                       uniquely for the entire database

                        Branch_ID      Identifies the branch where the
                                       account is held

                        Account_       Contains the account's current cash
                        Balance        balance

          History 50    Account_ID     Identifies the account updated by the
                                       transaction

                        Teller_ID      Identifies the teller involved in the
                                       transaction

                        Branch_ID      Identifies the branch associated with
                                       the teller

                        Amount         Contains the amount of credit or
                                       debit (delta) specified by the
                                       transaction.


                        Digital Technical Journal Vol. 3 No. 1 Winter 1991  5

 


           Performance Evaluation of Transaction Processing Systems




                                     Table 1 (Cont.)

                                    Database Entities

           ___________________________________________________________________
                         Fields
           Record__Bytes_Required_______Description___________________________

                         Time_Stamp     Contains the date and time taken
                                        between the BEGIN TRANSACTION and
           _____________________________COMMIT_TRANSACTION_statements_________

                                                  Benchmark A uses two basic
                      Table 2                    metrics:

                  Database Sizing                o  Transactions per second

             ____________________________           (TPS)-throughput in TPS,
                                   Record           subject to a response
             Number_of_Records_____Type__           time constraint, i.e.,

             1 x x                 Branch           the MQTh, is measured
                                   records          while the system is in a

                                                    sustainable steady-state
             10 xx x               Teller           condition.
                                   records       o  Price per TPS (K$/TPS)-

             100,000 x x           Account          the purchase price and
                                   records          five-year maintenance
                                                    costs associated with
             2,592,000 x x         History          one TPS.

                                   records        Transactions per Second.
          _______________________________        To guarantee that the

              For example, to process            tested system provides
             20 TPS, a system must use           fast response to on-line
             a database that includes            users, the TPC Benchmark A
             20 branch records, 200              imposes a specific response
             teller records, and                 time constraint on the
             2,000,000 account records.          benchmark. Ninety percent
             Because each teller uses            of all transactions must
             a terminal, the price of            have a response time of
             the system must include             less than two seconds. The
             200 terminals. A test that          TPC Benchmark A standard
             results in a higher TPS             defines transaction
             rate is invalid unless the          response time as the
             size of the database and            time interval between
             the number of terminals are         the transmission from the
             increased proportionately.          terminal of the first byte
                                                 of the input message to the
             Benchmark Metrics                   system under test to the
                                                 arrival at the terminal of
                                                 the last byte of the output
                                                 message from the system
                                                 under test.

          6  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                       Performance Evaluation of Transaction Processing Systems



              The reported TPS is the            o  Price of additional
             total number of committed              products required
             transactions that both                 for the operation,
             started and completed                  administration, or
             during an interval of                  maintenance of the
             steady-state performance,              priced systems.
             divided by the elapsed              o  Price of products
             time of the interval. The              required for application
             steady-state measurement               development.
             interval must be at least
             15 minutes, and 90 percent           All hardware and software
             of the transactions must            used in the tested
             have a response time of             configuration must be
             less than 2 seconds.                announced and generally
              Price per TPS. The K$              available to customers.

             /TPS price/performance
             metric measures the total        TPC Benchmark a Implementation

             system price in thousands            Digital's implementation
             of dollars, normalized              of the TPC Benchmark A
             by the TPS rating of the            goes beyond the minimum
             system. The priced system           requirements of the TPC
             includes all the components         Benchmark A standard and
             that a customer requires            uses Digital's distributed
             to achieve the reported             approach to transaction
             performance level and               processing.[2] For example,
             is defined by the TPC               Digital's TPC Benchmark A
             Benchmark A standard as             implementation includes
             the                                 forms management and

             o  Price of the system              transaction processing
                under test, including            monitor software that
                all hardware, software,          are required in most real
                and maintenance for five         transaction processing
                years.                           environments but are not
             o  Price of the terminals           required by the benchmark.
                and network components,          The following sections
                and their maintenance            provide an overview of
                for five years.                  Digital's approach and
                                                 implementation.
             o  Price of on-line storage         Transaction Processing
                for 90 days of history           Software Environment
                records at the published
                TPS rate, which amounts           The three basic functions
                to 2,592,000 records per         of a general-purpose
                TPS. A storage medium is         transaction processing
                considered to be on-line         system are the user
                if any record can be             interface (forms
                accessed randomly within         processing), applications
                one second.                      management, and database

                                                 management. Digital has
                                                 developed a distributed
                                                 transaction architecture
                                                 (DECdta) to define how

                          Digital Technical Journal Vol. 3 No. 1 Winter 1991  7

 


           Performance Evaluation of Transaction Processing Systems



             the major functions are             forms management to be
             partitioned and supported           performed at a remote
             by components that fit              location, whereas the
             together to form a complete         application is processed
             transaction processing              at a central location.
             system. Table 3 shows the           The Digital transaction
             software components in a            processing software
             typical Digital transaction         components are separable
             processing environment.             because their clearly
                                                 defined interfaces can
                      Table 3                    be layered transparently
               Transaction Processing            onto a network. How
                Software Components              these components may be
             ____________________________        partitioned in the Digital
             Component________Example____        distributed transaction
             Operating        VMS                processing environment is
             system                              illustrated in Figure 3.


             Communications   LAT,
                              DECnet

             Database         VAX Rdb
                              /VMS

             TP monitor       VAX ACMS,
                              DECintact

             Forms            DECforms

          ___Application______COBOL______

             Distributed Transaction
             Processing Approach

              Digital transaction
             processing systems can
             be distributed by placing
             one or more of the basic
             system functions (i.e.,
             user interface, application
             manager, database manager)
             on separate computers.
             In the simplest form of
             a distributed transaction
             processing system, the user
             interface component runs
             on a front-end processor,
             and the application and
             database components run
             on a back-end processor.
             The configuration
             allows terminal and


          8  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                       Performance Evaluation of Transaction Processing Systems



             TPC Benchmark A Test                initiate transactions and
             Environment                         communicate with the front-
                                                 end processors. Front-end
              The Digital TPC Benchmark          processors communicate with
             A tests are implemented in          a back-end processor using
             a distributed transaction           the DECnet protocol.
             processing environment               The test system is the
             using the transaction               configuration of components
             processing software                 used in the lab to measure
             components shown in Figure          the performance of the
             3. The user interface               target system. The test
             component runs on one or            system uses RTEs, rather
             more front-end processors,          than user terminals, to
             whereas the application             generate the workload and
             and database components             measure response time.
             run on one or more                  (Note: In previously
             back-end processors.                published reports, based
             Transactions are entered            on Digital's DebitCredit
             from teller terminals,              benchmark, the RTE emulated
             which communicate with              front-end processors.
             the front-end processors.           In the TPC Benchmark A
             The front-end processors            standard, the RTE emulates
             then communicate with the           only the user terminals.)
             back-end processors to              The RTE component
             invoke the application
             servers and perform                 o  Emulates the behavior of
             database operations.                   terminal users according
             The communications can                 to the benchmark
             take place over either                 specification (e.g.,
             a local area or a wide                 think time, transaction
             area network. However,                 parameters)
             to simplify testing, the            o  Emulates terminal
             TPC Benchmark A standard               devices (e.g.,
             allows sponsors to use                 conversion and
             remote terminal emulators              multiplexing into the
             (RTEs) rather than real                local area transport
             terminals. Therefore,                  [LAT] protocol used by
             the TPC Benchmark A tests              the DECserver terminal
             base performance and price             servers)
             /performance results on             o  Records transaction
             two distinctly configured              messages and response
             systems, the target system             times (e.g., the
             and the test system.                   starting and ending

              The target system is the              times of individual
             configuration of hardware              transactions from
             and software components                each emulated terminal
             that customers can use                 device)
             to perform transaction
             processing. With the
             Digital distributed
             transaction processing
             approach, user terminals


                          Digital Technical Journal Vol. 3 No. 1 Winter 1991  9

 


           Performance Evaluation of Transaction Processing Systems



              Figure 4 depicts the test
             system configuration in
             the LAN environment with
             one back-end processor,
             multiple front-end
             processors, and multiple
             remote terminal emulators.
















































          10  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                       Performance Evaluation of Transaction Processing Systems




          TPC Benchmark A Results                CommunicationsDECnet-VMS      1
                                                               Phase IV
              We now present the results
             of two TPC Benchmark A              TP monitor    VAX ACMS V3.1   1
             tests based on audited
             benchmark experiments               Dictionary    VAX CDD/Plus    1
             performed on the VAX 9000                         V4.1
             Model 210 and the VAX
             4000 Model 300 systems.             Application   VAX COBOL       1
             [3,4] These two systems                           V4.2
             are representative of
             Digital's large and small           Database      VAX Rdb/VMS     1
             transaction processing              system        V4.0
             platforms. The benchmark
             was implemented using               Forms         DECforms V1.2   1
             the VAX ACMS transaction         ___management__________________
             processing monitor, the VAX
             Rdb/VMS relational database
             management system, and the                   Table 5
             DECforms forms management
             system on the VMS operating        VAX 4000 Model 300 Back-end
             system. Tables 4 and 5                 System Configuration

             show the back-end system            ____________________________
             configurations for the VAX          Component______Product______  Quantity

             9000 Model 210 and the VAX          Processor      VAX 4000       1
             4000 Model 300 systems,                            Model 300
             respectively. Table 6 shows
             the system configuration of         Memory                        64 MB
             the front-end systems.

                                                 Tape drive     TK70           1

                      Table 4
            VAX 9000 Model 210 Back-end          Disk           DSSI           3
                System Configuration             controller

             ____________________________
             Component_____Product_______  QuantiDisks          RF31           18

             Processor     VAX 9000        1     Operating      VMS 5.4        1
                           Model 210             system

             Memory                        256 MBCommunications DECnet-VMS     1

             Tape drive    TA81            1                    Phase IV

             Disk          KDM70           2     TP monitor     VAX ACMS       1
             controller                                         V3.1

             Disks         RA92            16    Dictionary     VAX CDD/Plus   1
                                                                V4.1

             Operating     VMS 5.4         1
             system

                         Digital Technical Journal Vol. 3 No. 1 Winter 1991  11

 


           Performance Evaluation of Transaction Processing Systems




                  Table 5 (Cont.)

            VAX 4000 Model 300 Back-end
                System Configuration

             ____________________________
             Component______Product______  Quantity

             Application    VAX COBOL      1
                            V4.2

             Database       VAX Rdb/VMS    1
             system         V4.0

             Forms          DECforms       1
          ___management_____V1.2_________






































          12  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                       Performance Evaluation of Transaction Processing Systems




                                         Table 6

                         Front-end Run-time System Configuration

           ___________________________________________________________________
           Component___________Product____________________Quantity____________

           Processor           VAXserver 3100 Model 10    10 for VAX 9000
                                                          back-end

                                                          3 for VAX 4000
                                                          back-end

           Memory                                         16 MB for VAX 9000
                                                          back-end

                                                          12 MB for VAX 4000
                                                          back-end

           Disks               RZ23 (104 MB)              16

           Operating system    VMS 5.3                    1 for VAX 9000
                                                          back-end

                               VMS 5.4                    1 for VAX 4000
                                                          back-end

           Communications      DECnet-VMS Phase IV        1

           TP monitor          VAX ACMS V3.1              1

           Forms_management____DECforms_V1.2______________1___________________

                                         Table 7

                     VAX 9000 Model 210 Maximum Qualified Throughput

           ___________________________________________________________________
                                 ___________Response_Time_(seconds)___________

                                 TPS (tpsA-
           System________________Local)__________Average_____90_percent__Maximum

           VAX 9000 Model 210    69.4            1.20        1.74        5.82

           VAX_4000_Model_300____21.6____________1.39________1.99________4.81_

             Measurement Results                 Both configurations have
              The maximum qualified              sufficient main memory
             throughput and response             and disk drives such
             time results for the TPC            that the processors are
             Benchmark A are summarized          effectively utilized with
             in Table 7 for the VAX              no other bottleneck.
             9000 Model 210 and the VAX          Both systems achieved
             4000 Model 300 systems.             well over 90 percent CPU

                         Digital Technical Journal Vol. 3 No. 1 Winter 1991  13

 


           Performance Evaluation of Transaction Processing Systems



             utilization at the maximum          o  Response Time Frequency
             qualified throughput                   Distribution. Figure
             under the response time                6 is a graphical
             constraint. In addition to             representation of the
             the throughput and response            transaction response
             time, the TPC Benchmark A              time distribution.
             specification requires that            The average, 90th
             several other data points              percentile, and maximum
             and graphs be reported.                transaction response
             We demonstrate these data              times are also marked on
             and graphs by using the                the graph.
             VAX 9000 Model 210 TPC              o  Transactions per
             Benchmark A results.                   Second over Time.

             o  Response Time in                    The results shown in
                Relationship to TPS.                Figure 7 demonstrate
                Figure 5 shows the                  the sustainable maximum
                90th percentile and                 qualified throughput.
                average response                    The one-minute running
                times at 100 percent                average transaction
                and approximately                   throughputs during
                80 percent and 50                   the warm-up and data
                percent of the maximum              collection periods
                qualified throughput.               of the experiment are
                The mean transaction                plotted on the graph.
                response time still                 This graph shows that
                grows linearly with the             the throughput was
                transaction rate up to              steady during the period
                the 70 TPS level, but               of data collection.
                the 90th percentile              o  Average Response
                response time curve has             Time over Time. The
                started to rise quickly             results shown in Figure
                due to the high CPU                 8 demonstrate the
                utilization and random              sustainable average
                arrival of transactions.            response time in

                                                    the experiment. The
                                                    one-minute running
                                                    average transaction
                                                    response times during
                                                    the warm-up and data
                                                    collection periods
                                                    of the experiment are
                                                    plotted on the graph.
                                                    This graph shows that
                                                    the mean response time
                                                    was steady during
                                                    the period of data
                                                    collection.






          14  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                       Performance Evaluation of Transaction Processing Systems



                                                 the basic construction
          Comprehensive Analytical Model         of the model and the
                                                 customization made to
              Modeling techniques can            model the execution of TPC
             be used as a supplement             Benchmark A on Digital's
             or an alternative to the            transaction processing
             measurement approach.               systems. The model can also
             The performance behavior            be used to study different
             of complex transaction              transaction processing
             processing systems can              workloads in addition to
             be characterized by a set           the TPC Benchmark A.
             of parameters, a set of             Response Time Components
             performance metrics, and
             the relationships among              The main metric used in
             them. These parameters              the model is the maximum
             can be used to describe             qualified throughput
             the different resources             under a response time
             available in the system,            constraint. The response
             the database operations             time constraint is in
             of transactions, and                the form of "x percent of
             the workload that the               transaction response times
             transaction processing              are less than y seconds."
             system undergoes. To                 To evaluate throughput
             completely represent                under such response
             such a system, the size             time constraint,
             of the parameter set                the distribution of
             would be too huge to                transaction response
             manage. An analytical               times is determined by
             model simplifies, through           first decomposing the
             abstraction, the complex            transaction response time
             behavior of a system                into nonoverlapping and
             into a manageable set of            independent components.
             parameters and policies.            The distribution of
             Such a model, after proper          each component is then
             validation, can be a                evaluated. Finally,
             powerful tool for many              the overall transaction
             types of analysis, as well          response time distribution
             as a performance prediction         is derived from the
             tool. Results can be                mathematical convolution
             obtained quickly for any            of the component response
             combination of parameters.          time distributions.

              A comprehensive analytical          The logical flow of a
             model of the performance            transaction in a front-end
             behavior of transaction             and back-end distributed
             processing systems                  transaction processing
             with a response time                system that is used to
             constraint was developed            implement TPC Benchmark
             and validated against               A is depicted in Figure
             measurement results. This           9. The response time of
             model is hierarchical and           a transaction consists of
             flexible for extension. The         three basic components:
             following sections describe         front-end processing,


                         Digital Technical Journal Vol. 3 No. 1 Winter 1991  15

 


           Performance Evaluation of Transaction Processing Systems



                                                 o  Back-end processing
                                                    includes the execution
                                                    of application, database
                                                    access, concurrency
                                                    control, and transaction
                                                    commit processing. The
                                                    back-end processing
                                                    usually involves a high
             back-end processing, and               degree of concurrency
             communication delays.                  and many disk I/O
                                                    activities.
             o  Front-end processing             o  Communication delays
                usually includes                    primarily include the
                terminal I/O processing,            communications between
                forms/presentation                  the user terminal and
                services, and                       the front-end node,
                communication with the              and the front-end and
                back-end systems. In the            back-end interactions.
                benchmark experiments,
                no disk I/O activity              (Note: These response time
                was involved during the          components do not overlap
                front-end processing.            with each other.)

              Within the back-end system,         The model is configured in
             the transaction response            a two-level hierarchy,
             time is further decomposed          a high level and a
             into two additional                 detailed level. The use
             components, CPU delays              of a hierarchy allows
             and non-CPU, nonoverlapping         a complex and detailed
             delays. CPU delays include          model that considers many
             both the CPU service and            components and involves
             the CPU waiting times of            many parameters to be
             transactions. Non-CPU,              constructed easily.
             nonoverlapping delays               Because of the hierarchical
             include:                            approach, the model also
             o  Logging delays, which            provides flexibility
                include the time for             for modifications and
                transaction log writes           extensions, and validation
                and commit protocol              of separate submodels.

                delays                            The high-level model
             o  Database I/O delays,             assumes the decomposition
                which include both               of transaction response
                waiting and service              times, as described in the
                times for accessing              Response Time Components
                storage devices                  section, and models the
             o  Other delays, which              behavior of the transaction
                include delays that              processing system by an
                result from concurrency          open queuing system, as
                control (e.g., waiting           shown in Figure 10. The
                for locks) and waiting           queuing system consists of
                for messages                     servers and delay centers,
                                                 which are connected in a

             Two-level Approach

          16  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                       Performance Evaluation of Transaction Processing Systems



                                                 o  No intratransaction
             queuing network with the               parallelism exists
             following assumptions:                 within individual
                                                    transaction execution.
             o  The front-end processing         o  No mutual dependency
                does not involve any                exists between
                disk I/O operation, and             transaction response
                the load on the front-              time components.
                end systems is equally           o  Transaction arrivals to
                balanced.                           the processors have a

             o  The back-end is                     Poisson distribution.
                a shared-memory
                multiprocessor system             These assumptions
                with symmetrical loads           correspond to Digital's
                on all processors (or            TPC Benchmark A
                it can be simply a               testing methodology and
                uniprocessor).                   implementation.

              The front-end CPU is               distribution. The major
             modeled as an M/M/1 queuing         input parameters for this
             center, and the back-end            high-level model are the
             CPU is modeled as an M/M            o  Number of front-end
             /m queuing center. The                 systems and the front-
             transactions' CPU times on             end CPU service time per
             the front-end and back-end             transaction
             systems are assumed to be
             exponentially distributed           o  Number of CPUs in the
             (coefficient of variation              back-end system and the
             equal to 1) due to the                 back-end CPU service
             single type of transaction             time per transaction

             in the benchmark. (Note:            o  Sum of the back-end
             An approximation of M/G/m              database I/O response
             can be used to consider a              time, journaling I/O
             coefficient of variation               response time, and
             other than 1 for the                   other delay times (i.e.,
             back-end transaction CPU               the mean for the LOD
             service time, especially               delay center's 2-Erlang
             in the multiprocessor                  distribution)
             case when the bus is                o  Response time constraint
             highly utilized.)                      (in the form of x
             Database I/O, logging                  percentile less than
             I/O, and other delays are              y seconds)
             modeled as delay centers,
             with appropriate delay               The main result from the
             distributions. For the              high-level model is the
             model of the TPC Benchmark          MQTh. This high-level
             A workload, the database            model presents a global
             I/O, journaling I/O, and            picture of the performance
             other communication and             behavior and manifests the
             synchronization delays are          relationship between the
             combined into one delay             most important parameters
             center, called the LOD              of the transaction
             delay center, which is              processing system and MQTh.
             represented by a 2-Erlang

                         Digital Technical Journal Vol. 3 No. 1 Winter 1991  17

 


           Performance Evaluation of Transaction Processing Systems



              Some of the input
             parameters in the high-
             level model are dynamic.               use spin-locks as the
             The CPU service time of                mechanism for processor-
             a transaction may vary                 level synchronization,
             with the throughput or                 and the processor spins
             number of processors,                  (i.e., busy-waits) in
             and the database I/O or                the case of a conflict.
             other delays may also                  In the model, the
             depend on the throughput. A            busy-wait overhead is
             good example of a dynamic              considered to be part
             model is a tightly coupled             of the transaction
             multiprocessor system, with            code path, and such
             one bus interconnecting
             the processors and with                contention elongates the
             a shared common memory                 transaction CPU service
             (e.g., a VAX 6000 Model                time.

             440 system). Such a                  Four detailed-level
             system would run a single           submodels are used to
             copy of the symmetrical             account for the dynamic
             multiprocessing operating           behavior of these
             system (e.g., the VMS               parameters: CPU-cache-
             system). The average                bus-memory, busy-wait, I/O
             CPU service time of                 group, and LOD.
             transactions is affected by
             both hardware and software           The CPU-cache-bus-memory
             factors, such as                    submodel consists of many
             a. Hardware contention that         low-level parameters
                results from conflicting         associated with the
                accesses to the shared           workload, processor, cache,
                bus and main memory and          bus, and memory components
                that causes processor            of multiprocessor systems.
                speed degradation and            It models these components
                longer CPU service time.         by using a mixed queuing
                                                 network model that consists
             b. Processor                        of both open and closed
                synchronization overhead         chains, as shown in Figure
                that results from                11. The most important
                the serialization of             output from this submodel
                accesses to shared               is the average number
                data structures. Many            of CPU clock cycles per
                operating systems                instruction.

              The busy-wait submodel             analysis to derive busy-
             models the spin-lock                wait time. The I/O grouping
             contention that is                  submodel models the group
             associated with the two             commit and group write
             major VMS spin-locks,               mechanisms of the VAX Rdb
             called SCHED and IOLOCK8.           /VMS relational database
             This submodel divides the           management system. This
             state of a processor into           submodel affects the path
             several nonoverlapping              length of transaction
             states and uses probability         because of the amortization
                                                 of disk I/O processing

          18  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                       Performance Evaluation of Transaction Processing Systems



                                                 path length, the busy-
             among grouped transactions.         wait overhead, and the CPU
             The LOD submodel considers          utilization.

             the disk I/O times and the           By applying the initialized
             lock contention of certain          parameters to the
             critical resources in the           submodels, the values
             VAX Rdb/VMS system.                 of these parameters are
             Integrating the Two Levels          refined and input to the
             of the Model                        high-level model. The
                                                 output parameters from the
              The two levels of the model        high-level model are then
             are integrated by using             fed back to the detailed-
             an iterative procedure              level submodels, and this
             outlined in Figure 12. It           iterative process continues
             starts at the detailed-             until the MQTh converges.
             level submodels, with               In most cases, convergence
             initial values for the              is reached within a few
             MQTh, the transaction               iterations.






          Model Predictions                      end-to-end model to the
              The back-end portion of            TPC Benchmark A on two VAX
             the model was validated             platforms, the VAX 9000
             against measurement                 Model 210 and the VAX 4000
             results from numerous               Model 300 systems, and
             DebitCredit benchmarks              then compare the results.
             (Digital's precursor of             The benchmark environment
             the TPC Benchmark A) on             and implementation are
             many VAX computers with             described in the TPC
             the VMS operating system,           Benchmark A Implementation
             running VAX ACMS and VAX            section of this paper.

             Rdb/VMS software. [5]                Because both the VAX
             With sufficient detailed            9000 Model 210 and
             parameters available (such          the VAX 4000 Model 300
             as transaction instruction          systems are uniprocessor
             count, instruction cycle            systems, there is no other
             time, bus/memory access             processor contending for
             time, cache hit ratio),             the processor-memory
             the model correctly                 interconnect and memory
             estimated the MQTh and              subsystems. Such contention
             many intermediate results           effects can therefore be
             for several multiprocessor          ignored when modeling a
             VAX systems. The model was          uniprocessor system. The
             then extended to include            transaction processing
             the front-end systems. In           performance prediction
             this section, we discuss            for the VAX 9000 Model
             applying this complete              210 system is a successful


                         Digital Technical Journal Vol. 3 No. 1 Winter 1991  19

 


           Performance Evaluation of Transaction Processing Systems



             example of the application           Additional predictions
             of our analytical model.            were made later, when an
              We needed an accurate              early prototype version
             estimate of TPC Benchmark           of the VAX 9000 Model 210
             A performance on the VAX            system was available for
             9000 Model 210 system               testing. A variant of the
             before a VAX 9000 system            DebitCredit benchmark, much
             was actually available              smaller in scale and easier
             for testing. The high-              to run, was performed on
             level (MQTh) model was              the prototype system, with
             used with estimated values          the emphasis on measuring
             for the input parameters,           the CPU performance in a
             LOD and transaction CPU             transaction processing
             service time. The estimated         environment. The result
             LOD was based on previous           was used to extrapolate the
             measurement observations            CPU service time of the TPC
             from the VAX 6000 systems.          Benchmark A transactions
             The other parameter, back-          on the VAX 9000 Model 210
             end transaction CPU service         system and to refine the
             time, was derived from the          early estimate. The results
                                                 of these modifications
             o  Timing information of            supported the previous
                the VAX 9000 CPU                 high-end estimate of
             o  Memory access time and           performance of 70 TPS
                cache miss penalty of            and refined the low-end
                the VAX 9000 CPU                 performance to be 62 TPS.
             o  Prediction of cache hit          The final, audited TPC
                ratio of the VAX 9000            Benchmark A measurement
                system under the TPC             result of the VAX 9000
                Benchmark A workload             Model 210 system showed
                                                 69.4 TPS, which closely
             o  Transaction path length          matches the prediction.
                of the TPC Benchmark A           Table 8 compares the
                implementation                   results from benchmark
             o  Instruction profile              measurement and the
                of the TPC Benchmark A           analytical model outputs.

                implementation
                                                          Table 8
              The high-level model                  Measurement Compared
             predicted a range of                   to Model Predictions
             MQTh, with a high end of            ____________________________
             70 TPS and with a strong                                  Measured    Modeled
             probability that the                System                MQTh        MQTh
             high-end performance was            ____________________________
             achievable.                         VAX 9000 Model 210    69.4        70.0

                                              ___VAX_4000_Model_300____21.5__      20.8

                                                  The VAX 4000 Model 300 TPC
                                                 Benchmark A results were
                                                 also used as a validation
                                                 case. VAX 4000 Model 300


          20  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                       Performance Evaluation of Transaction Processing Systems



             systems use the same CMOS           performance by different
             chip as the VAX 6000 Model          vendors. But it is
             400 series and the same             only one transaction
             28-nanosecond (ns) CPU              processing benchmark that
             cycle time. However, in             represents a limited class
             the VAX 4000 series, the            of applications. When
             CPU-memory interconnect             evaluating transaction
             is not the XMI bus but              processing systems
             a direct primary memory             performance, a good
             interconnect. This direct           understanding of the
             memory interconnect results         targeted application
             in fast main memory access.         environment and
             The processor, cache, and           requirements is essential
             main memory subsystems              before using any available
             are otherwise the same              benchmark result.
             as in the VAX 6000 Model            Additional benchmarks
             400 systems. Therefore,             that represent a broader
             the detailed-level model            range of commercial
             and associated parameters           applications are expected
             for the VAX 6000 Model              to be standardized by the
             410 system can be used by           Transaction Processing
             ignoring the bus access             Performance Council (TPC)
             time. The TPC Benchmark             in the coming years.
             A measurement results are            Performance modeling is
             within 7 percent of the             an attractive alternative
             model prediction, which             to benchmark measurement
             means that our assumption           because it is less
             on the memory access time           expensive to perform and
             is acceptable.                      results can be compiled

                                                 more quickly. Modeling
          Conclusion                             provides more insight
              Performance is one of the          into the behavior of
             most important attributes           system components that
             in evaluating a transaction         are treated as black
             processing system. However,         boxes in most measurement
             because of the complex              experiments. Modeling
             nature of transaction               helps system designers
             processing systems, a               to better understand
             universal assessment of             performance issues and
             transaction processing              to discover existing or
             system performance is               potential performance
             impossible. The performance         problems. Modeling also
             of a transaction                    provides solutions for
             processing system is                improving performance by
             workload dependent,                 modeling different tuning
             configuration dependent,            or design alternatives. The
             and implementation                  analytical model presented
             dependent. A standard               in this paper was validated
             benchmark, like TPC                 and used extensively
             Benchmark A, is a step              in many engineering
             toward a fair comparison            performance studies.
             of transaction processing           The model also helped


                         Digital Technical Journal Vol. 3 No. 1 Winter 1991  21

 


           Performance Evaluation of Transaction Processing Systems



                                              References

             the benchmark process to            1. Transaction Processing
             size the hardware during               Performance Council,
             preparation (e.g., the                 TPC Benchmark A Standard
             number of RTE and front-               Specification (Menlo
             end systems needed, the                Park, CA: Waterside
             size of the database)                  Associates, November
             and to provide an MQTh                 1989).

             goal as a sanity check              2. Transaction Processing
             and a tuning aid. The                  Systems Handbook
             model could be extended                (Maynard: Digital
             to represent additional                Equipment Corporation,
             distributed configurations,            Order No. EC-H0650-57,
             such as shared-disk and                1990).
             "shared-nothing" back-end           3. TPC Benchmark: A Report
             transaction processing                 for the VAX 9000 Model
             systems, and could be                  210 System (Maynard:
             applied to additional                  Digital Equipment
             transaction processing                 Corporation, Order No.
             workloads.                             EC-N0302-57, 1990).

          Acknowledgments                        4. TPC Benchmark: A Report
                                                    for the VAX 4000 Model
              The Digital TPC Benchmark             300 System (Maynard:
             A implementation and                   Digital Equipment
             measurements are the result            Corporation, Order No.
             of work by many individuals            EC-N0301-57, 1990).
             within Digital. The authors         5. L. Wright, W. Kohler,
             would like especially to               and W. Zahavi, "The
             thank Jim McKenzie, Martha             Digital DebitCredit
             Ryan, Hwan Shen, and Bob               Benchmark: Methodology
             Tanski for their work                  and Results,"
             in the TPC Benchmark A                 Proceedings of the
             measurement experiments;               International Conference
             and Per Gyllstrom and                  on Management and
             Rabah Mediouni for their               Performance Evaluation
             contributions to the                   of Computer Systems
             analytical model and                   (December 1989): 84-92.
             validation.














          22  Digital Technical Journal Vol. 3 No. 1 Winter 1991
=============================================================================
Copyright 1991 Digital Equipment Corporation.  Forwarding and copying of this 
article is permitted for personal and educational purposes without fee 
provided that Digital Equipment Corporation's copyright is retained with the 
article and that the content is not modified. This article is not to be 
distributed for commercial advantage. Abstracting with credit of Digital 
Equipment Corporation's authorship is permitted.  All rights reserved.
=============================================================================
