 










              Database Availability for Transaction Processing


              By Ananth Raghavan and T.K. Rengarajan



          Abstract                                Modern businesses store
              A transaction processing           critical data in database
             system relies on its                management systems. Much
             database management                 of the daily activity
             system to supply high               of business includes
             availability. Digital               manipulation of data
             offers a network-based              in the database. As
             product, the VAX DBMS               businesses extend their
             system, and a relational            operations worldwide,
             data-based product, the             their databases are shared
             VAX Rdb/VMS database                among office locations
             system, for its transaction         in different parts of
             processing systems. These           the world. Consequently,
             database systems have               these businesses require
             several strategies to               transaction processing
             survive failures, disk              systems to be available
             head crashes, revectored            for use at all times. This
             bad blocks, database                requirement translates
             corruptions, memory                 directly to a goal of
             corruptions, and memory             perfect availability
             overwrites by faulty                for database management
             application programs.               systems.

             They use base hardware               VAX DBMS and VAX Rdb/VMS
             technologies and also               database systems are based
             employ novel software               on network and relational
             techniques, such as                 data models, respectively.
             parallel transaction                Both systems use a kernel
             recovery, recovery on               of code that is largely
             surviving nodes of a                responsible for providing
             VAXcluster system,                  high availability. This
             restore and roll-forward            layer of code is maintained
             operations on areas of the          by the KODA group. KODA
             database, on-line backup,           is the physical subsystem
             verification and repair             for VAX DBMS and VAX Rdb
             utilities, and executive            /VMS database systems.
             mode protection of trusted          It is responsible for all
             database management system          I/O, buffer management,
             code.                               concurrency control,
             Introduction                        transaction consistency,
                                                 locking, journaling, and
                                                 access methods.


                          Digital Technical Journal Vol. 3 No. 1 Winter 1991  1

 


           Database Availability for Transaction Processing



              In this paper, we define           of downtime are useful.
             database availability,              Unexpected downtime is
             and describe downtime               caused by factors that
             situations and how such             are beyond the control of
             situations can be resolved.         the transaction processing
             We then discuss the                 system. For example, a disk
             mechanisms that have                failure is quite possible
             been implemented to                 at any time during normal
             provide minimal loss of             processing of transactions.
             availability.                       However, scheduled downtime
                                                 is entirely within the
          Database Availability                  control of the database
                                                 administrator. High
              The unit of work in                availability demands that
             transaction processing              we eliminate scheduled
             systems is a transaction.           downtime and ensure fast
             We therefore define                 system recovery from
             database availability as            unexpected failures.
             the ability to execute               The layers of the software
             transactions. One way the           and hardware services which
             database management system          compose a transaction
             provides high availability          processing system are
             is by guaranteeing the              dependent on one another
             properties of transactions:         for high availability.
             atomicity, serializability,         The dependency among these
             and durability.[1] For              services is illustrated
             example, if a transaction           in Figure 1. Each service
             that has made updates to            depends on the availability
             the database is aborted,            of the service in the
             other transactions must             lower layers. Errors and
             not be allowed to see these         failures can occur in
             updates; the updates made           any layer, but may not
             by the aborted transaction          be detected immediately.
             must be removed from the            For example, in the case
             database before other               of a database management
             transactions may use that           system, the effects of a
             data. Yet, data that has            database corruption may
             not been accessed by the            not be apparent until
             aborted transaction must            long after the corruption
             continue to be available to         (error) has occurred.
             other transactions.                 Hence it is difficult to

              Downtime is the                    deal with such errors. On
             term used to refer                  the other hand, failures
             to periods when the                 are noticed immediately.
             database is unavailable.            Failures usually make the
             Downtime is caused by               system unavailable and are
             either an unexpected                the cause of unexpected
             failure (unexpected                 downtime.
             downtime) or scheduled
             maintenance on the database
             (scheduled downtime).
             Such classifications


          2  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                               Database Availability for Transaction Processing



              Each layer can provide             Application Program
             only as much availability           Exceptions

             as the immediate lower               Although transaction
             layer. Hence we can also            processing systems are
             express the perfect-                based on the client
             availability goal of a              /server architecture,
             database management system          Digital's database systems
             as the goal of matching             are process based. The
             the availability of the             privileged database
             immediately lower layer,            management system code
             which in our case is the            is packaged in a shareable
             operating system.                   library and linked with
              At the outset, it is clear         the application programs.
             that a database management          Therefore, bugs in the
             system layered on top of            applications have a
             an operating system and             good chance of affecting
             hence only as available as          the consistency of the
             the underlying operating            database. Such bugs in
             system. However, a database         applications are one type
             management system is in             of failure that can make
             general not as available            the database unavailable.

             as the underlying layer              The VAX DBMS and VAX Rdb
             because of the need to              /VMS systems guard against
             guarantee the properties of         this class of failure by
             transactions.                       executing the database
                                                 management system code
          Unexpected Downtime                    in VAX'S executive mode.
                                                 Since application programs
              In this section we discuss         execute in user mode, they
             the causes of unexpected            do not have access to data
             downtime and the techniques         structures used by the
             that minimize downtime.             database management system.
              A database monitor must be         When a faulty application
             started on a node before            program attempts such an
             a user's process running            access, the VMS operating
             on that node can access             system detects it and
             a database. The monitor             generates an exception.
             oversees all database               This exception then forces
             activity on the node.               an image rundown of the
             It allows processes to              application program.

             attach to and detach from            In general, when an image
             databases and detects               rundown is initiated,
             failures. On detecting              Digital's database
             a failure, the monitor              management products use
             starts a process to recover         the condition-handling
             the transactions that did           facility of VMS to abort
             not complete because of             the transaction. Condition
             the failure. Note that              handling of image rundown
             this database monitor               is performed at two levels.
             is different from the TP            Two condition handlers
             monitor. [2]                        are established, one in

                          Digital Technical Journal Vol. 3 No. 1 Winter 1991  3

 


           Database Availability for Transaction Processing



             user mode and the other in          is examined at different
             kernel mode. The user mode          points in the code. If any
             exit handler is usually             inconsistency is found, a
             invoked, which rolls back           bug-check utility is called
             the current transaction             that dumps the internal
             and unbinds it from the             database format to a file.
             database. In this case, the         The utility then raises an
             rest of the users on the            exception that is handled
             system are not affected at          by the monitor, and the
             all. The database remains           DBR process is started as
             available. The execution of         described above.
             the user mode exit handler           To deal with corruptions
             is, however, not guaranteed         to the database that are
             by the VMS operating                undetected with this
             system. Under some abnormal         mechanism, an explicit
             circumstances, the user             utility is provided that
             mode exit handlers may              verifies the structural
             not be executed at all.             consistency of the
             In such circumstances, the          database. This verify
             kernel mode exit handler is         utility may be executed
             invoked by the VMS system.          on-line, while users
             This handler resides in             are still accessing the
             the database monitor. The           database. Such verification
             monitor starts a database           may also be executed by
             recovery (DBR) process.             a database administrator
             It is the responsibility            (DBA) in response to a bug-
             of the DBR process to roll          check dump. Once such a
             back the effects of the             corruption is detected, an
             aborted transaction. To do          on-line utility provides
             this, the DBR process first         the ability to repair the
             establishes a database              database.
             freeze. This freeze
             prevents other processes             In general, corruption in
             from acquiring locks that           databases causes unexpected
             were held by the aborted            downtime. Digital provides
             transaction and hence see           the means of detecting
             and update uncommitted              such corruption on-line
             data. (The VMS lock manager         and repairing them on-line
             releases all locks held by          through recovery utilities.
             a process when that process         Process Failure
             dies.) The DBR process then
             proceeds to roll back the            In the VMS system, a
             aborted transaction.                process failure is always
             Code Corruptions                    preceded by image rundown
                                                 of the current image
              It is important to prevent         running as part of the
             coding mistakes within the          process. Therefore, a
             DBMS from irretrievably             process failure is detected
             corrupting the database.            by the database monitor,
             To protect the database             which then starts a DBR
             management system from              process to handle recovery.
             coding mistakes, internal           Node Failure
             data structure consistency


          4  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                               Database Availability for Transaction Processing



              Among the many mechanisms          the recovery. The only
             Digital provides for                differences in the case of
             availability is node                process, node, or cluster
             failover within a cluster.          failure is the mechanism
             When a node fails, another          by which the monitor is
             node on the cluster detects         informed of the failure.
             the failure and rolls back          Disk Head Crash
             the lost transactions from
             the failed node. Thus the            Some failures can result
             failure of one node does            in the loss or corruption
             not cause transactions              of the data on the stable
             on other active nodes               storage device (disk).
             of the cluster to come              Digital has a mechanism for
             to a halt (except for               bringing the database back
             the time the DBR process            to a consistent state in
             enforces a freeze). It is           such cases.
             the database monitor that            A disk head crash is a
             detects node failure and            failure of hardware that
             starts a recovery process           is usually characterized by
             for every lost transaction          the inability to read from
             on the failed node. The             or write to the disk. Hence
             database becomes available          database storage areas
             as soon as recovery is              residing on that disk are
             complete for all the users          unavailable and possibly
             on the failed node.                 irretrievable. A disk

             Power Failure                       head crash automatically
              Power failure is a hardware        aborts transactions that
             failure. As soon as power           need to read from or
             is restored, the VMS system         write to that disk. In
             boots. When a process               addition, recovery of these
             attaches to the database,           aborted transactions is
             a number of messages are            not possible since the
             passed between the process          recovery processes need
             that is attaching and the           access to the same disk.
             monitor. If the database is         In this case, the database
             corrupt (because of power           is shut down and access is
             failure), the monitor is so         denied until the storage
             informed by the attaching           areas on the failed disk
             process, and again the              are brought on-line. Areas
             monitor starts recovery             are restored from backups
             processes to return the             and then rolled forward
             database to a consistent            until consistent with the
             state. The database becomes         rest of the database. The
             available as soon as                after image journal (AIJ)
             recovery is complete for            files are used to roll
             all such failed users.              the areas forward. As
                                                 soon as all the areas on
              As described above,                the failed disk have been
             recovery is always                  restored onto a good disk
             accomplished by the                 and rolled forward, the
             monitor process starting            database becomes available.
             DBR processes to do                 Bad Disk Blocks


                          Digital Technical Journal Vol. 3 No. 1 Winter 1991  5

 


           Database Availability for Transaction Processing



              Bad blocks are hardware            off against the downtime
             errors that often are               needed to restore and roll
             not detected when they              forward.
             happen. The bad blocks are          Site Failure
             revectored, and the next
             time the disk block is               A site failure occurs when
             read, an error is reported.         neither the computers nor
             Bad blocks simply mean              the disks are available. A
             that the contents of a disk         site failure is usually
             block are lost forever.             caused by a natural
             The database administrator          disaster such as an
             detects the problem only            earthquake. The best
             when a database application         recourse for recovery is
             fails to fetch data on the          archival storage. Digital
             revectored block. Such an           provides mechanisms to
             error may cause a certain           back up the database and
             transaction or a set of             AIJ files to tape. These
             transactions to fail, no            tapes must then be stored
             matter how many attempts            at a site away from the
             are made to execute                 site at which the database
             the transactions. This              resides. Should a disaster
             failure constitutes reduced         happen, these backup tapes
             availability; parts of the          can be used to restore
             database are unavailable            the database. However,
             to transactions. Exactly            the recovery may not be
             how much of the database            complete. It cannot restore
             remains available depends           the effects of those
             on which blocks were                committed transactions that
             revectored.                         were not backed up to tape.

              The mechanism provided              After a disaster, the
             to reduce the possible              database can be restored
             downtime is early                   and rolled forward to the
             detection. Digital's                state of the completion
             database systems provide            of the last AIJ that
             a verification utility that         was backed up to tape.
             can be executed while users         Any transactions that
             are running transactions.           committed after the last
             The verification utility            AIJ was backed up cannot be
             checks the structural               recovered at the alternate
             consistency of the                  site. Such transaction
             database. Once a bad block          losses can be minimized by
             is detected by such a               frequently backing up the
             utility, that area of the           AIJ files.
             database may be restored            Memory Errors
             and rolled forward. These
             two operations make the              Memory errors are quite
             whole database temporarily          infrequent, and when they
             unavailable; however, the           happen, they usually are
             bad block is corrected, and         not detected. If the error
             future downtime is avoided.         happens to a data record,
             The downtime caused by the          it may never be detected
             bad block may be traded             by any utility, but may


          6  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                               Database Availability for Transaction Processing



             be seen as incorrect                 Digital's database
             data by the user. If the            systems allow two types
             verification utility is             of transactions: update
             run on-line, it may also            and "snapshot." The ability
             detect the errors. Again,           to back up data on-line
             the database may only be            depends on the snapshot
             partially available, as             transaction capability of
             in the case of bad blocks.          the database.
             However, it is possible to           Database backup is a
             repair the database while           standard way of recovering
             users are still accessing           from media failures.
             the database. Digital's             Digital's database systems
             database management                 provide the ability to
             products provide explicit           do transaction consistent
             repair facilities for               backups of data on-line
             this purpose. The loss of           while users continue to
             availability during repair          change the database.
             is not worse than the loss
             due to the memory error              The general scheme for
             itself.                             snapshot transactions is
              As explained previously,           as follows. The update
             the database monitor                transactions of the
             plays an important                  database preserve the
             part in ensuring                    previous versions of the
             database consistency                database records in the
             and availability. Most              snapshot file. All versions
             unexpected failure                  of a database record are
             scenarios are detected              chained. Only the current
             by the monitor, which then          version of the record is
             starts recovery processes.          in the database area. The
             In addition, some failures          older versions are kept
             might require the use of            in the snapshot area. The
             backup files to restore the         versions of the records are
             database.                           tagged with the transaction
                                                 numbers (TSNs). When a
                                                 snapshot transaction (for
          Scheduled Downtime                     example, a database backup)
              Most database systems              needs to read a database
             have scheduled maintenance          record, it traverses the
             operations that require a           chain for that database
             database shutdown. Database         record and then uses the
             backup for media recovery           appropriate version of the
             and verification to check           record.

             structural consistency               There are two modes
             are examples of operations          of database operation
             that may require scheduled          with respect to snapshot
             downtime. In this section           activity. In one mode,
             we describe ways to perform         all update transactions
             many of these operations            write snapshot copies of
             while the database is               any records they update.
             executing transactions.             In the deferred snapshot
             Backup                              mode, the updates cause


                          Digital Technical Journal Vol. 3 No. 1 Winter 1991  7

 


           Database Availability for Transaction Processing



             snapshot copies to be               database to check the
             written only if a snapshot          structural consistency
             transaction is active and           of the database. These
             requires old versions of            utilities may also be
             a record. In this mode, a           executed on-line through
             snapshot transaction cannot         the use of snapshot
             start until all currently           transactions.
             active update transactions          AIJ Backup
             (which are not writing
             snapshot records) have               The backup and the AIJ
             completed; that is, the             log are the two mechanisms
             snapshot transaction must           that provide media recovery
             wait for a quiet point in           for Digital's database
             time. If there are either           management products. The
             active or pending snapshot          AIJ file is continuously
             transactions when an update         written to by all user
             transaction starts, the             processes updating the
             update transaction must             database. We need to
             write snapshot copies.              provide some ability
              Here we see a trade-off            to back up the AIJ file
             between update transactions         since it monotonically
             and snapshot transactions.          increases in size and
             The database is completely          eventually fills up the
             available to snapshot               disk. Digital's database
             transactions if all update          systems offer the ability
             transactions always write           to back up the AIJ file
             snapshot copies. On the             to tape (or another
             other hand, if the deferred         device) on-line. The only
             snapshot mode is enabled,           restriction is that a quiet
             update transactions                 point must be established
             need not write snapshot             for a short period during
             copies if a snapshot                which the backup operation
             transaction in not active.          takes place. A quiet point
             This approach obviously             is defined as a point when
             results in some loss of             the database is quiescent,
             availability to snapshot            i.e., there are no active
             transactions.                       transactions.

             Verification                        On-line Schema Changes

              Database corruption                 Digital's database
             can also result in                  management systems allow
             downtime. Although database         users to change metadata
             corruption is not probable,         on-line, while users
             it is possible. Any                 are still accessing the
             database system that                database. Although this may
             supports critical data              be standard for relational
             must provide facilities             database management
             to ensure the consistency           systems, it is not standard
             of the database. Digital's          for network databases. The
             database management systems         VAX DBMS system provides a
             provide verification                utility called the database
             utilities that scan the             restructuring utility (DRU)


          8  Digital Technical Journal Vol. 3 No. 1 Winter 1991

 


                               Database Availability for Transaction Processing



             to allow for on-line schema      References
             modifications.
                                                 1. P. Bernstein, W.
          Acknowledgments                           Emberton, and V.
                                                    Trehan, "DECdta -
              Many engineers have                   Digital's Distributed
             contributed to the                     Transaction Processing
             development of the                     Architecture," Digital
             algorithms described in                Technical Journal, vol.
             this paper. We have chosen             3, no. 1 (Winter 1991,
             not to enumerate all such              this issue): 10-17.
             contributions. However,             2. T. Speer and M. Storm,
             we would like to recognize             "Digital's Transaction
             the contributions of Peter             Processing Monitors,"
             Spiro, Ashok Joshi, Jeff               Digital Technical
             Arnold, and Rick Anderson              Journal, vol. 3, no.
             who, together with the                 1 (Winter 1991, this
             authors, are members of the            issue): 18-32.
             KODA team.




































                          Digital Technical Journal Vol. 3 No. 1 Winter 1991  9
=============================================================================
Copyright 1991 Digital Equipment Corporation.  Forwarding and copying of this 
article is permitted for personal and educational purposes without fee 
provided that Digital Equipment Corporation's copyright is retained with the 
article and that the content is not modified. This article is not to be 
distributed for commercial advantage. Abstracting with credit of Digital 
Equipment Corporation's authorship is permitted.  All rights reserved.
=============================================================================
