6 Troubleshooting

This chapter examines recovery procedures for problems that might occur and actions that can be taken to fix or avoid future problems. The chapter covers the following:

Recovery of a single user file (Section 6.1)

System recovery (Section 6.2)

Recovery and restoration procedures for a number of common problems (Section 6.3)

Preventative maintenance strategies (Section 6.4)

6.1 Recovering a User File

If you do not plan for this in advance, there is no convenient way to recover a file that has been deleted. You can plan for access to files that have been removed in three ways:

Back up the directory with the vdump command and restore the file with the vrestore command. You must have write permission for the directory where the backup is saved. See Chapter 4 for more information.

Set up a fileset clone, preferably each day, to access files that have changed that day. You must have root privileges to create a clone. See Section 2.4.10 for more information.

Set up a trashcan for the directory. This feature is under user control. Root privileges are not required. See Section 2.6 for more information.

If a file has been removed and there is no backup, trashcan or clone, there is no way to recover the file. If a file is corrupted, however, running the /sbin/advfs/salvage command may recover some of its contents. See Section 6.2.6 for directions.

6.2 General Recovery Procedures

The following recovery steps apply to all domain panics and many types of system crashes. If the cause of the crash appears to be in one domain, the following steps may be needed to regain access to the filesets in the affected domain. Follow as many steps as needed. All might not be necessary. If you are recovering the root domain, see Section 6.3.4.

6.2.1 Saving Data for a Problem Report

Run the sys_check utility with the -escalate option to create escalation files for reporting problems to your technical support representative. The utility collects the output in the file /TMPDIR/escalate.tar file.

You can run the sys_check command either from the SysMan Menu utility called Support and Services — Create Escalation Report (Escalation) (see Appendix A), or from the command line.

You must be root user or have appropriate privileges to run the sys_check utility. See sys_check(8) for more information.

6.2.2 Saving Copies of System Metadata

If it appears that a domain is corrupted or it is otherwise causing problems, run the /sbin/advfs/savemeta command to save a copy of the domain's metadata for support personnel to examine. Run the savemeta command with the -f option to save structure information from the frag files for each fileset. The utility saves the log file, bitfile metadata table, each volume's storage bitmap, the domain's root tag file, and the fileset tag files.

For example, to save metadata from domain_1 on /tmp/saved_domain_1 enter:

# /sbin/advfs/savemeta domain_1 /tmp/saved_domain_1

To save metadata from the frag file in the fileset fsetN on /tmp/saved_fsetN enter:

# /sbin/advfs/savemeta domain_1 -f fsetN /tmp/saved_fsetN

You must be root user to run this command. See savemeta(8) for more information.

6.2.3 Saving Undamaged Filesets

It is a good idea to back up as many filesets as you can in the affected domain. (See Chapter 4 for a description of backup methods.) You can use this backup, other recent backups, and the output of the recovery utilities (/sbin/advfs/verify, /sbin/advfs/fixfdmn, and /sbin/advfs/salvage) to recreate your domain.

6.2.4 Verifying File System Consistency

To ensure that metadata is consistent, run the /sbin/advfs/verify command to verify the file system structure. The verify utility checks disk structures such as the bitfile metadata table (BMT), the storage bitmaps, the tag directory, and the frag file for each fileset. It verifies that the directory structure is correct, that all directory entries reference a valid file, and that all files have a directory entry. You must be the root user to run this command.

It is a good idea to run the verify command in the following situations:

When problems are evident (corruptions, domain panic, lost data, I/O errors)

Before an update installation

If your files have not been accessed in three to six months or longer, and you plan to run utilities such as balance, defragment, migrate, quotacheck, repquota, rmfset, rmvol, or vdump that access every file in a domain

Use the SysMan Menu utility called Manage an AdvFS Domain (see Appendix A), or enter the verify command from the command line:

/sbin/advfs/verify domain_name

The verify command mounts filesets in special directories. If the verify command is unable to mount a fileset due to the failure of a domain, as a last resort you can run the verify command with the -F option. The -F option mounts the fileset using the -d option of the mount command, so AdvFS initializes the transaction log file for the domain without recovering the domain.

Caution

Because no domain recovery occurs for previously incomplete operations, using the verify command with the -F option can cause data corruption.

Under some circumstances the verify command does not unmount the filesets. If this occurs, unmount the affected filesets manually and delete the mount points that were created in the /etc/fdmns file for the domain.

On machines with millions of files, sufficient swap space must be allocated for the verify utility to run to completion. If the amount of memory required by the verify utility exceeds the kernel variable proc/max_per_proc_data_size process variable, the utility does not complete. To overcome this problem, allocate up to 10% of the domain size in swap space for running the verify command. See swapon(8) for more information.

If running the verify command is successful, it is a good idea to back up your domain in case the event that caused the crash recurs.

The following example verifies the domain domainx, which contains the filesets setx and sety:

# /sbin/advfs/verify domainx
+++Domain verification+++
Domain Id 2f03b70a.000f1db0
Checking disks ...
Checking storage allocated on disk /dev/disk/dsk10g
Checking storage allocated on disk /dev/disk/dsk10a
Checking mcell list ...
Checking mcell position field ...
Checking tag directories ...
 
+++ Fileset verification +++
+++ Fileset setx +++
Checking frag file headers ...
Checking frag file type lists ...
Scanning directories and files ...
     1100
Scanning tags ...
     1100
Searching for lost files ...
     1100
 
+++ Fileset sety +++
Checking frag file headers ...
Checking frag file type lists ...
Scanning directories and files ...
     5100
Scanning tags ...
     5100
Searching for lost files ...
     5100

In this example, the verify utility finds no problems with the domain so no further recovery steps are needed. See verify(8) for more information.

6.2.5 Fixing On-Disk Metadata Corruptions with the fixfdmn Utility

The /sbin/advfs/fixfdmn utility is designed primarily to put a domain into a usable (mountable) state by repairing metadata corruptions. Unmount your fileset and run the utility in the following situations:

If the /sbin/advfs/verify utility described in Section 6.2.4 cannot run

If the verify utility detects on-disk (metadata) corruption

If you have a domain with filesets that cannot be mounted without a domain panic and you do not want to run the verify command with the -F option

The fixfdmn utility scans metadata looking for corruptions and, if enough metadata is intact, it attempts to correct the corrupt metadata. If not enough viable metadata is available, the utility attempts to bypass the corruption by moving or deleting the corrupt metadata and deleting files as necessary.

The utility does not check or repair the contents of files. If recovering data from a file is your priority or if a large portion of the domain is corrupted, see Section 6.2.6 for information on using the /sbin/advfs/salvage utility.

Once you have run the fixfdmn utility, remount the filesets. If you can remount your filesets, check that no files have been deleted. If they have, restore them from backup. If you are unable to mount the filesets, run fixfdmn with the -u option to undo the process so that the salvage command can access the unchanged data.

If running the fixfdmn command is successful, it is a good idea to back up your domain in case the event that caused the crash recurs. Running the fixfdmn command successfully means that you do not need additional recovery procedures.

You can run the fixfdmn command with the -n option to check the domain and not do any repairs. The utility will print system messages, but changes will not be written to disk.

6.2.6 Salvaging File Data from a Corrupted Domain

Run the /sbin/advfs/salvage utility to recover the files in the domain. The command extracts salvageable files from the corrupted domain and places copies of them in filesets created to hold the recovered files. Depending on the nature of the corruption, you may be able to extract all or some of the data in the corrupted domain.

If you have run the /sbin/advfs/fixfdmn command and it was not successful, before you run the salvage command, you must first run the fixfdmn command with the -u option to undo the changes that the utility has made.

You can access salvage functionality through the SysMan Menu utility called Manage an AdvFS Domain (see Appendix A) or you can enter the salvage command from the command line:

/sbin/advfs/salvage domain_name fileset_name

You can recover data to disk or to tape. The amount of data you can recover depends upon the nature of the corruption to your domain. See salvage(8) for more information.

Running the salvage command does not guarantee that you recover all files in your domain. You might be missing files, directories, file names, or parts of files. The utility generates a log file that contains the status of files that were recovered. Use the -l option to log every file that is encountered.

The salvage command places the recovered files in directories named after the filesets. The utility creates a lost+found directory for each fileset and puts files in it that have no parent directory. You can specify the pathname of the directory that is to contain the recovered fileset directories. If you do not specify a directory, the utility writes recovered filesets under the current working directory.

You can also recover data from a damaged domain in a tar format by specifying the -F tar option.

6.2.6.1 Recovering File Data

If you have a recent backup source, use the -d option of the /sbin/advfs/salvage utility to recover only the data that has changed after the date of the backup. This limits the amount of information the salvage utility must process. If you do not have a backup source, the salvage utility can look at all files in the domain you name.

If your system does not have enough space to hold the recovered files, you can recover data to tape and then write it back to your original disk location. You can also execute the salvage command using the -F and -f options to recover data in a tar format.

The process is as follows:

Unmount all filesets in the corrupted domain.

If you are saving the salvaged data to tape, install a tape on a local drive. See the Hardware Management manual for more information.

If you are not saving salvaged data in tar format, create a temporary domain and fileset to hold the recovered information and mount the fileset.

Run the salvage command.

View the salvage.log file to ensure that all necessary files were recovered. If files are missing, you will need to use a backup source for these files.

Recreate the domain and mount-point directory. Create filesets and mount them.

Restore files from backup.

Restore the salvaged files from either the temporary domain or the tar file.

The following example salvages data to disk. In this example the corrupted domain is called PERSONNEL and contains the fileset personnel_a-e mounted at /personnel_a-e. The original domain is on volume /dev/disk/dsk12c and the salvage command places output on /dev/disk/dsk3c. Because a backup tape, /dev/tape/tape0, is available, only files from the PERSONNEL domain that were modified after 1:30 PM on December 5, 2001 (the date of the last backup) are extracted from the damaged domain. The domain is restored back to the original volume.

# umount /PERSONNEL
# mkfdmn /dev/disk/dsk3c RECOVER
# mkfset RECOVER recover_fset
# mkdir /recover
# mount RECOVER#recover_fset /recover
# /sbin/advfs/salvage -d 200112051330 -D /recover PERSONNEL
salvage: Domain to be recovered 'PERSONNEL' 
salvage: Volume(s) to be used '/dev/disk/dsk12c'
salvage: Files will be restored to '/recover' 
salvage: Logfile will be placed in './salvage.log' 
salvage: Starting search of all filesets: 09-Dec-2001
salvage: Starting search of all volumes: 09-Dec-2001
salvage: Loading file names for all filesets: 09-Dec-2001
salvage: starting recovery of all filesets: 09-Dec-2001

Before running the mkfdmn command, check the salvage.log file to make sure that you have the files you want. If you are missing files, it may be helpful to run salvage inserting an earlier date for the -d option to retrieve additional information. Once you have created a new domain, you cannot run salvage again except with the -S option, as described in Section 6.2.6.3.

# mkfdmn /dev/disk/dsk12c PERSONNEL
# mkfset PERSONNEL personnel_a-e
# mkdir /personnel_a-e
# mount PERSONNEL#personnel_a-e /personnel_a-e
# vrestore /dev/tape/tape0 /personnel_a-e
# cp -Rp /RECOVER/personnel_a-e/* /personnel_a-e
# umount /recover
# rmfdmn RECOVER
rmfdmn: remove domain RECOVER [y/n] y
rmfdmn: domain RECOVER removed.

6.2.6.2 Recovering Data from a Corrupted root Domain

If your system is not bootable because the root domain is corrupt, you can boot your system from the installation CD-ROM and run the /sbin/advfs/salvage command. Follow the steps in Section 6.3.4. Depending on the nature and extent of the root domain corruption, successful file recovery may not be possible.

6.2.6.3 Recovering Data Block by Block

If you ran the salvage utility and were unable to recover a large number of files, run the salvage command with the -S option. This process is very slow because the utility reads every disk block at least once.

Note

If you have accidentally used the mkfdmn command on a good domain, running the salvage command with the -S option is the only way to recover files.

Caution

The salvage utility opens and reads block devices directly, which can present a security problem. With the -S option it might be possible to access data from older, deleted AdvFS domains while attempting to recover data from the current AdvFS domain.

The following example recovers data from a domain called PERSONNEL block by block.

# /sbin/advfs/salvage -S PERSONNEL
salvage: Domain to be recovered 'PERSONNEL'  
salvage: Volume(s) to be used '/dev/disk/dsk12c'
salvage: Files will be restored to '.'  
salvage: Logfile will be placed in './salvage.log'  
salvage: Starting sequential search of all volumes: 07-Oct-2001
salvage: Loading file names for all filesets: 07-Oct-2001
salvage: Starting recovery of all filesets: 07-Oct-2001

6.3 Fixing Problems

This section addresses a number of ways to restore your system.

6.3.1 Recovering from a Domain Panic

When a metadata write error occurs, or if corruption is detected in a single AdvFS domain, the system initiates a domain panic (rather than a system panic) on the domain. This isolates the failed domain and allows a system to continue to serve all other domains. After a domain panic, AdvFS no longer issues I/O requests to the disk controller for the affected domain. Although the domain cannot be accessed, the filesets in the domain can be unmounted.

When a domain panic occurs, an EVM event is logged ( EVM(5)) and the following message is printed to the system log and the console:

AdvFS Domain Panic; Domain name Id domain_Id

For example:

AdvFS Domain Panic; Domain staffb_domain Id 2dad7c28.0000dfbb
An AdvFS domain panic has occurred due to either a
 metadata write error or an internal inconsistency.
This domain is being rendered inaccessible.

By default, a domain panic on an active domain causes a live dump to be created and placed in the /var/adm/crash directory. Some AdvFS-related errors might also be recorded in /var/adm/binary.errlog.

To recover from a domain panic, perform the following steps:

Run the mount command with the -t advfs option and identify all mounted filesets in the affected domain.

Unmount all the filesets in the affected domain.

Save the system problem report information (Section 6.2.1).

Save the domain metadata. See savemeta(8) and Section 6.2.2 for more information.

If the problem is a hardware problem, fix it before continuing.

Run the /sbin/advfs/verify utility (Section 6.2.4) on the domain.
- If there are no errors, mount all the filesets you unmounted and resume normal operations.
- If the verify command runs but shows errors, run the /sbin/advfs/fixfdmn command (Section 6.2.5) to fix metadata corruption.

If the failure prevents complete recovery, recreate the domain (Section 6.2.6).

The following example illustrates the commands needed after a domain panic on the staffb_dmn domain:

# mount -t advfs
staffb_dmn#staff3_fs on /usr/staff3 type advfs (rw)
staffb_dmn#staff4_fs on /usr/staff4 type advfs (rw) 
# umount /usr/staff3
# umount /usr/staff4
# sys_check -escalate
# /sbin/advfs/savemeta staffb_dmn /tmp/saved_dmn
# /sbin/advfs/savemeta -f staff3_fs /tmp/saved_staff3
# /sbin/advfs/savemeta -f staff4_fs /tmp/saved_staff4
# /sbin/advfs/verify staffb_dmn

You do not need to reboot after a domain panic.

Domain panics preceded by messages that report I/O errors are usually caused by faulty hardware that must be fixed. The following is an example of an I/O error message:

   AdvFS I/O error:
       Domain#Fileset: staffb_dmn#staff3_fs
       Mounted on: /usr/staff3
       Volume: /dev/disk/dsk6c
       Tag: 0x00000006.8001
       Page: 56
       Block: 1200
       Block count: 128
       Type of operation: Write
       Error: 19
       EEI: 0x6200 (Advfs cannot retry this)
       AdvFS initiated retries: 0
       Total AdvFS retries on this volume: 0
       I/O error appears to be due to a hardware problem.
       Check the binary error log for details.
       To obtain the name of the file on which
       the error occurred, type the command:
         /sbin/advfs/tag2name /usr/staff3/.tags/6

If you have recurring domain panics that do not indicate I/O errors, first try running the fixfdmn utility (Section 6.2.5). If domain panics still occur after running fixfdmn, try adjusting the AdvfsDomainPanicLevel attribute (Section 5.15) to facilitate debugging.

6.3.2 Recovering from Filesets That are Mounted Read-Only

When a fileset is mounted, AdvFS verifies that all volumes in a domain can be accessed. The size recorded in the domain's metadata for each volume must match the size of the volume. If the sizes match, the mount proceeds. If a volume is smaller than the recorded size, AdvFS attempts to read the last block marked in use for the fileset. If this block can be read, the mount succeeds, but the fileset is marked as read-only. If the last in-use block for any volume in the domain cannot be read, the mount fails. See mount(8) for more information.

If a fileset is mounted read-only, check the labels of the flagged volumes in the error message. Two errors are common:

A disk is mislabeled on a RAID array.

A volume upon which an AdvFS domain resides was reduced from its original size.

If you have AdvFS Utilities, and if the domain consists of multiple volumes with enough free space to remove the offending (flagged) volume, you do not need to remove your filesets. However, it is a good idea to back them up before proceeding.

Remove the volume from the domain by using the rmvol command. This automatically migrates the data to the remaining volumes. If you do not have enough free space on the remaining volumes, use the addvol command to add additional space before removing the volume.

Correct the disk label of the offending volume by using the disklabel command.

Add the corrected volume back to the domain by using the addvol command.

Run the balance command to distribute the data across the new volumes.

For example, if /dev/disk/dsk2c (on a device here called disk_a) within the data5 domain is mislabeled, you can migrate your files from that volume (automatic with the rmvol command), then move them back after you have corrected and restored the volume. Note that the first addvol step is necessary only if you need space in the domain for the information on the volume that you are removing.

# addvol /dev/disk/dsk3c data5
# rmvol /dev/disk/dsk2c data5
# disklabel -z dsk2
# disklabel -rw dsk2 disk_a
# addvol /dev/disk/dsk2c data5
# balance data5

If you do not have AdvFS Utilities, the process is as follows:

Back up all filesets in the domain.

Remove the domain by using the rmfdmn command.

Correct the disk label of the volume by using the disklabel command.

Make the new domain.

Restore the filesets from the backup.

For example, if /dev/disk/dsk1c (on a device here called disk_a) containing the data3 domain is mislabeled:

# vdump -0f -u /data3
# rmfdmn data3
# disklabel -z dsk1
# disklabel -rw dsk1 disk_a
# mkfdmn data3
# mkfset data3 data3fset
# mount data3#data3fset /data3 
# vrestore -xf - /data3

6.3.3 Restoring the /etc/fdmns Directory

AdvFS must have a current /etc/fdmns directory to mount filesets. (See Section 2.3.1 for more information.) A missing or damaged /etc/fdmns directory prevents access to a domain, but the data within the domain remains intact. You can restore the /etc/fdmns directory from backup or you can recreate it.

It is preferable to restore the /etc/fdmns directory from backup if you have a current backup copy. You can use any standard backup facility (vdump, tar, or cpio) to back up the /etc/fdmns directory. To restore the directory, use the recovery procedure that is compatible with your backup process.

If you cannot restore the /etc/fdmns directory, you can reconstruct it manually (Section 6.3.3.1) or with the /sbin/advfs/advscan command (Section 6.3.3.2). The procedure for reconstructing the /etc/fdmns directory is similar for both single-volume and multivolume domains. You can construct the directory for a missing domain, missing links, or the whole directory.

6.3.3.1 Reconstructing the /etc/fdmns Directory Manually

If you choose to reconstruct the directory manually, you must know the name of each domain and its associated volumes.

Caution

Do not run the mkfdmn command to reconstruct an existing domain. Executing the command destroys the data on that volume. If you have destroyed your files, you might be able to recover some files by running the salvage utility with the -S option.

The following example reconstructs the /etc/fdmns directory and two domains. In this example the domains exist and their names are known. Each domain contains a single volume (or special device). The order of creating the links in these examples does not matter. The domains are domain1 on /dev/disk/dsk1c and domain2 on /dev/disk/dsk2c.

# mkdir /etc/fdmns
# mkdir /etc/fdmns/domain1
# cd /etc/fdmns/domain1
# ln -s /dev/disk/dsk1c .
# mkdir /etc/fdmns/domain2
# cd /etc/fdmns/domain2
# ln -s /dev/disk/dsk2c .

The following example reconstructs one multivolume domain. The domain1 domain contains the three volumes /dev/disk/dsk1c, /dev/disk/dsk2c, and /dev/disk/dsk3c.

# mkdir /etc/fdmns
# mkdir /etc/fdmns/domain1
# cd /etc/fdmns/domain1
# ln -s /dev/disk/dsk1c .
# ln -s /dev/disk/dsk2c .
# ln -s /dev/disk/dsk3c .

6.3.3.2 Reconstructing the /etc/fdmns Directory Using the advscan Command

Use the /sbin/advfs/advscan command to determine which partitions on a disk or which Logical Storage Manager (LSM) volumes are part of an AdvFS domain if you have moved disks to a new system, if device numbers have changed, or if you have lost track of a domain location. You can also use the command to rebuild all or part of your /etc/fdmns directory if you accidentally deleted it, deleted a domain it, or deleted links from a domain's subdirectory. See advscan(8) for more information.

For each domain there are three numbers that must match for the AdvFS file system to operate properly:

The number of physical partitions found by the advscan command that have the same domain ID

The domain volume count (the number stored in the AdvFS metadata that specifies the number of partitions in the domain)

The number of /etc/fdmns links to the partitions, because each partition must be represented by a link

The advscan command attempts to repair the domain if these numbers are inconsistent.

Inconsistencies can occur in these numbers for several reasons. In general, the advscan command treats the domain volume count as more reliable than the number of partitions or the number of /etc/fdmns links. The following tables list anomalies, possible causes, and corrective actions that the advscan utility can take.

Table 6-1 describes possible causes and corrective actions if the number of links in the /etc/fdmns/domain_name directory does not equal the expected value for the number of partitions and for the domain volume count.

Table 6-1: Incorrect Number of Links

Number of Links	Possible Cause	Corrective Action
Too low	The `addvol` command terminated early or a link in `/etc/fdmns/domain_name` was manually removed.	If the domain is activated before running the `advscan` command with the -f option, and the cause of the mismatch is an interrupted `addvol` command, the situation is corrected automatically. Otherwise, the `advscan` utility adds the partition to the `/etc/fdmns/domain_name` directory.
Too high	The `rmvol` command terminated early or a link in `/etc/fdmns/domain_name` was manually added.	If the domain is activated and the cause of the mismatch is an interrupted `rmvol` command, the situation is corrected automatically. If the cause is a manually added link in `/etc/fdmns/domain_name`, systematically try removing different links in the `/etc/fdmns/domain_name` directory and activating the domain. The number of links to remove is the number of links in the `/etc/fdmns/domain_name` directory minus the domain volume count displayed by the `advscan` command.

Table 6-2 shows possible causes and corrective actions if the domain volume count does not equal the expected value for the number of partitions and for the number of links in the /etc/fdmns/domain_name directory.

Table 6-2: Incorrect Domain Volume Count

Domain Volume Count	Possible Cause	Corrective Action
Too low	Cause unknown.	Cannot correct; run the `/sbin/advfs/fixfdmn` command to put your domain in a usable state.
Too high	The `addvol` command terminated early and the partition being added is missing or was reused.	Cannot correct; run the `salvage` utility to recover as much data as possible from the remaining volumes in the domain.

Table 6-3 shows possible causes and corrective actions if the number of partitions does not equal the expected value for the domain volume count and for the number of links in the /etc/fdmns/domain_name directory.

Table 6-3: Incorrect Number of Partitions

Number of Partitions	Possible Cause	Corrective Action
Too low	Partition missing.	Cannot correct; run the `fixfdmn` command to put your domain in a usable state.
Too high	The `addvol` command terminated early.	Rerun the `addvol` command.

In the following example no domains are missing. The advscan command scans devices dsk0 and dsk5 for AdvFS partitions and finds two partitions: dsk0c and dsk5c. The domain volume count reports two, and two links are entered in the /etc/fdmns directory.

# /sbin/advfs/advscan dsk0 dsk5
Scanning disks  dsk0 dsk5
Found domains:
usr_domain
                Domain Id       2e09be37.0002geb40
                Created         Thu Feb 28 09:54:15 2002
                Domain volumes          2
                /etc/fdmns links        2
                Actual partitions found:
                                        dsk0c
                                        dsk5c

In the following example, directories that define the domains that include dsk6 were removed from the /etc/fdmns directory. Thus the number of /etc/fdmns links, the number of partitions, and the domain volume counts are no longer equal. In this example the advscan command scans device dsk6 and recreates the missing domains as follows:

A partition is found containing an AdvFS domain. The domain volume count reports one, but there is no domain directory in the /etc/fdmns directory that contains this partition.

Another partition is found containing a different AdvFS domain. The domain volume count is also one. No domain directory contains this partition.

No other AdvFS partitions are found. The domain volume counts and the number of partitions found match for the two discovered domains.

The advscan command creates directories for the two domains in the /etc/fdmns directory.

The advscan command creates symbolic links for the devices in the /etc/fdmns/domain_name directories.

If a directory for a partition does not have an entry in the /etc/fdmns directory, the partition name is marked with an asterisk (*) and the utility creates a domain name for it. You must then rename the recovered domain to the name of your original domain. In this example, the original domain names were revenue and expenses.

# /sbin/advfs/advscan -r dsk6
Scanning disks  dsk6
Found domains:
*unknown*
                Domain Id       2f2421ba.0008c1c0
                Created         Sun Jan 20 13:38:02 2002
 
                Domain volumes          1
                /etc/fdmns links        0
 
                Actual partitions found:
                                        dsk6a**
unknown*
                Domain Id       2f535f8c.000b6860
                Created         Mon Feb 25 09:38:20 2002
 
               Domain volumes          1
                /etc/fdmns links       0
 
                Actual partitions found:
                                        dsk6b*

Creating /etc/fdmns/domain_dsk6a/
        linking dsk6a
 
Creating /etc/fdmns/domain_dsk6b/
        linking dsk6b
# mv /etc/fdmns/domain_dsk6a /etc/fdmns/revenue
# mv /etc/fdmns/domain_dsk6b /etc/fdmns/expenses

6.3.4 Recovering from Data Corruption of an AdvFS root Domain

Catastrophic corruption of your AdvFS root domain typically requires that you recreate your root file system to have a bootable system. If you are running a cluster configuration, see the System Administration and the Cluster Administration manuals. If your root volume is an LSM volume, see the Logical Storage Manager manual. You must be root user to reconstruct the root domain.

An installation CD-ROM or a Remote Installation Service (RIS) server is required. The root domain is recreated on the boot device. If you do not have adequate backup of the root domain, depending on the nature and extent of the corruption, you might be able to recover root files using the /sbin/advfs/salvage utility. (See Section 6.2.6 for more information.) The salvage utility might also be used to recover files that were modified or created following the most recent backup.

First identify hardware resources.

If you plan to boot your system from the installation CD-ROM, determine the name of your CD-ROM drive by entering the following at the System Reference Manual (SRM) console prompt:
```
 >>> show device  
...
DKA500        RRD47   1206   dka500.4.0.5.0
...
```
In this example the CD-ROM device name is DKA500.
If you plan to boot from a RIS server, determine the name of your network interface device by entering the following:
```
>>> show device | more
....
ewa0.0.0.8.0    EWA0     08-00-2B-C3-E3-DC
...
```
In this example the network interface device name is EWA0.

Identify the console boot device name. If your boot device is the default boot device, identify it by entering the following:
```
>>> show bootdef_dev
bootdef_dev      dkb400.4.0.5.1
```
In this example, dkb400.4.0.5.1 indicates that dkb400 is the boot device, dk indicates that the device is a SCSI disk, b indicates that the device is connected to SCSI bus b, and 400 indicates that the device's SCSI target ID is 4 and its Logical Unit (LUN) is 00. Thus, in this example, the bus/target/LUN information is 1/4/00.
If your boot device is not the default boot device, enter the show device command and identify it from the list.

Boot the system from the installation CD-ROM or the RIS server. For example, for the devices identified above, if you are using the CD-ROM, enter the following:
```
>>> boot dka500
```
If you are using the RIS server, enter the following:
```
>>> boot ewa0
```

Exit the installation. If you have a VGA graphics console, choose to exit the installation, or choose Shell Window from the File menu of the Installation and Configuration Welcome dialog box. If you have a serial console terminal, select option 3) Exit Installation.

After you have determined your system configuration, do the following:

If the root domain is mountable when you boot from the installation CD-ROM or RIS server, the installation procedure attempts to read the existing device database from the installed root domain. If the hardware database read fails, you will get an error message. If this happens, you must translate the UNIX device name assignments to the proper hardware device.
Use the /sbin/hwmgr command with the -view devices option to identify the device to be restored by its bus/target/LUN (as explained in the previous step), to locate disks that could be used for backup, and to verify that a tape device is installed.
```
# /sbin/hwmgr -view devices
HWID: Device Name    Mfg Model         Location
------------------------------------------------------------
38:/dev/disk/floppy0c    3.5in floppy     fdi0-unit-0    
41:/dev/disk/dsk0c   DEC RZ1DB-CA (C) DEC bus-1-targ-4-lun-0
42:/dev/disk/dsk1c   DEC RZ1CB-CA (C) DEC bus-1-targ-5-lun-0
43:/dev/disk/dsk2c   DEC RZ1CB-CA (C) DEC bus-1-targ-6-lun-0
44:/dev/disk/cdrom0  DEC RRD47    (C) DEC bus-0-targ-5-lun-0
46:/dev/ntape/tape0  DEC TLZ10    (C) DEC bus-1-targ-4-lun-0    
 
```
According to the hardware database, in this example the disk with bus/target/LUN of 1/4/00 is identified as dsk0. Assume that /dev/disk/dsk0a is the volume containing the corrupted root domain.
To visually confirm that you have identified the correct device, use the hwmgr command with the -flash option to cause the disk's light to flash for thirty seconds.
```
# /sbin/hwmgr -flash light -dsf /dev/disk/dsk0a
```

If you need to install a tape device for backup at this time, enter the following:
```
# /sbin/dn_setup -install_tape
```

If you have a good backup, you can restore from it. If your backup is out of date, you can restore from it and then use the salvage command with the -d option to recover later information. If your backup is good, skip to step 9. If you need to salvage files to use in conjunction with your backup, skip to step 8.

Mount the old root domain:
```
# mount old_root_domain#root /var/mnt
```
If the mount fails, skip to step 7. You cannot perform a backup.

Create a backup if possible. See Chapter 4 for more information.

Run the /sbin/advfs/verify command with the -a option. See Section 6.2.4 for more information. If this is successful and your system shows no errors, skip to Step 12.

Run the /sbin/advfs/fixfdmn command. See Section 6.2.5 for more information. If this is successful, skip to Step 12.

Recover files with the salvage command and save them to a temporary fileset or tape. If you have an old backup, run the salvage command with the -d option. If you do not specify the option, you will salvage the entire domain.
For example, to recover files from a corrupted root domain on volume /dev/disk/dsk0aand save them on the local, unused disk, /dev/disk/dsk2c, do the following:
1. Create a temporary domain and filesets to hold the recovered information and mount the filesets.
```
# mkfdmn /dev/disk/dsk2c RECOVER
# mkfset RECOVER recover_fset
# mkdir /var/recover
# mount RECOVER#recover_fset /var/recover
```
2. Run the salvage command. You must use the -V option to specify the volume that the command operates on. If you are salvaging files to be used with an older backup, use the -d option to salvage only the information that has changed since the date of the backup.
  In this example, files from the root domain are extracted and stored in filesets mounted at /recover.
```
# /sbin/advfs/salvage -D /var/recover -V /dev/disk/dsk0a
salvage: Volume(s) to be used '/dev/disk/dsk0a'
salvage: Files will be restored to '/recover' 
salvage: Logfile will be placed in './salvage.log' 
salvage: Starting search of all filesets: 09-Dec-2001
salvage: Loading file names for all filesets: 09-Dec-2001
salvage: Starting recovery of all filesets: 09-Dec-2001 
 
```
3. View the salvage.log file to ensure that all necessary files were recovered. Once you have run the mkfdmn command in the next step, you cannot rerun the salvage command to extract more information.

Create the new root domain and root fileset. Mount the fileset at /var/mnt.

# mkfdmn -r /dev/disk/dsk0a root_domain
Warning: /dev/disk/dsk0a is marked in use for AdvFS.
If you continue with the operation you can
possibly destroy existing data.
CONTINUE? [y/n] y
# mkfset root_domain root
# mkdir /var/mnt
# mount root_domain#root /var/mnt

If you have a good backup source, restore the files from it.
```
 # vrestore -xf  /dev/tape/tape0 -D /var/mnt
```

If you have salvaged files, copy them from the recovery location to the root domain and remove the recovery domain.

# cd /
# cp -Rp /recover/root/* /var/mnt
# umount /mnt /recover
# rmfdmn RECOVER
rmfdmn: remove domain RECOVER [y/n] y
rmfdmn: domain RECOVER removed.

The root domain is restored.

Halt and reboot the system.

If the procedure was not successful, you will have to reinstall the operating system and recreate your files.

6.3.5 Recovering from Hardware Corruption

Some problems occur in AdvFS because of hardware errors. For example, if a write to the file system fails due to a hardware fault, it might appear as metadata corruption. Hardware problems cannot be repaired by the AdvFS file system.

If unexplained errors occur on a volume in a multivolume domain, do one or more of the following:

Examine the /var/adm/messages file for AdvFS I/O error messages. You must be root user to look at this file. The file describes the domain, fileset, and volume on which the error occurred. It also describes how to find out which file was affected by the I/O error. The file may look similar to the following:

Dec 05 15:39:16 systemname vmunix: AdvFS I/O error: 
Dec 05 15:39:16 systemname vmunix: Domain#Fileset:test1#tstfs
Dec 05 15:39:16 systemname vmunix: Mounted on: /test1 
Dec 05 15:39:17 systemname vmunix: Volume: /dev/rz11c 
Dec 05 15:39:17 systemname vmunix: Tag: 0x00000006.8001 
Dec 05 15:39:17 systemname vmunix: Page: 76926 
Dec 05 15:39:17 systemname vmunix: Block: 5164080 
Dec 05 15:39:17 systemname vmunix: Block count: 256 
Dec 05 15:39:17 systemname vmunix: Type of operation: Read 
Dec 05 15:39:17 systemname vmunix: Error: 5 
Dec 05 15:39:17 systemname vmunix: To obtain the name of 
Dec 05 15:39:17 systemname vmunix: the file on which the 
Dec 05 15:39:17 systemname vmunix: error occurred, type the 
Dec 05 15:39:17 systemname vmunix: command 
Dec 05 15:39:17 systemname vmunix: /sbin/advfs/tag2name 
Dec 05 15:39:17 systemname vmunix: /test1/.tags/6

If you have no AdvFS I/O error messages but still have unexplained behavior on the file system, unmount the domain as soon as possible and run the /sbin/advfs/verify utility (Section 6.2.4) or the /sbin/advfs/fixfdmn command with the -n option (Section 6.2.5) to check the consistency of the domain's metadata.

Check for device driver error messages for the volume described in the AdvFS I/O error message. If there are no device driver I/O error messages that correspond to the AdvFS I/O error messages, then the file system is being affected by problems with the underlying hardware. Unmount the domain as soon as possible and run the verify utility or the fixfdmn command with the -n option to check the integrity of the domain's metadata.

If you have a multivolume domain and enough room, try to remove the faulty volume by using the rmvol utility as described in Section 2.3.5. If this succeeds, the file system problems should not recur.
If rmvol fails, you must recreate the domain.
1. If you have a recent backup, you can use it to restore the domain. If you have no backup, or if the backup is too old, use the salvage utility (Section 6.2.6) to extract the contents of the corrupted domain.
2. Remove the faulty domain by using the rmfdmn command.
3. Recreate the domain by using the mkfdmn command. The new domain has a domain version number (DVN) of 4 by default. (See Section 2.3.3 for more information.) Add volumes as needed. Do not to include the faulty volume in the new domain.
4. Restore the contents of the recreated domain using the salvaged and backup information.
5. Remount the filesets in the domain.

6.3.6 Moving an AdvFS Disk to Another Machine

If a machine has failed, you can move disks containing AdvFS domains to another computer running the AdvFS software. Connect the disks to the new machine and modify the /etc/fdmns directory so the new system recognizes the transferred volumes. You might need to reassign the disk SCSI IDs to avoid conflicts with the new system. (See your disk manufacturer instructions for more information.) You must be root user to perform this operation.

You cannot move DVN4 domains to systems running Version 4 of the operating system software. Doing so generates an error message. (See Section 6.4.6 for a discussion of compatibility.) You can move DVN3 domains from a Version 4 machine to a machine running Version 5. The newer operating system recognizes the domains created earlier.

Caution

Do not use either the addvol command or the mkfdmn command to add the volumes to the new machine. Executing the command deletes all data on the disk you are moving. See Section 6.2.6 if you have already done so.

If you do not know which partitions your domains were on, you can add the disks on the new machine and run the /sbin/advfs/advscan utility, which might be able to recreate this information. You can also look at the disk label on the disk to see which partitions in the past were made into AdvFS partitions. The disk labels do not tell you which partitions belong to which domains.

In the following example, the system has a domain, testing_domain, on two disks on the old machine called dsk3 and dsk4. This domain contains two filesets: sample1 and sample2. These filesets are mounted on /sample1 and /sample2. Assume you know that the domain that you are moving had partitions dsk3c, dsk4a, dsk4b, and dsk4g. Take the following steps to move the disks:

Shut down the working machine to which you are moving the disks.

Connect the disks from the bad machine to the good one.

Reboot. You do not need to reboot in single-user mode.

Determine the device nodes created for the new disks.
```
# /sbin/hwmgr -show scsi -full
```
Check the Device Name column to find the names for the disks that you just added. These have the highest disk numbers.

Assume the new disk IDs in this example are disk 7 and disk 8. Modify your /etc/fdmns directory to include symbolic links from the transferred domains.

# mkdir -p /etc/fdmns/testing_domain
# cd /etc/fdmns/testing_domain
# ln -s /dev/disk/dsk7c .
# ln -s /dev/disk/dsk8a .
# ln -s /dev/disk/dsk8b .
# ln -s /dev/disk/dsk8g .
# mkdir /sample1
# mkdir /sample2

Edit the /etc/fstab file to add the fileset mount-point information for the transferred filesets.
```
testing_domain#sample1 /sample1 advfs rw 1 0
testing_domain#sample2 /sample2 advfs rw 1 0
```

Mount the volumes.
```
# mount /sample1
# mount /sample2
```

6.3.7 Restoring a Multivolume usr Domain

Before you restore a multivolume /usr file system, you must first reconstruct the usr_domain domain with all of its volumes. However, restoring a multivolume domain requires the License Management Facility (LMF). LMF controls AdvFS Utilities, which includes the addvol command needed for creating multivolume domains.

First create a one volume usr domain and restore the addvol command. Then restore LMF and use it to enable the addvol command. When this is complete, you can add volumes to the usr domain and restore the complete multivolume domain.

LMF has two parts. A utility is stored in /usr/sbin/lmf and a database is stored in /var/adm/lmf. On some systems /var is a link to /usr and both directories are located in the usr fileset. If your system has this configuration, recover the addvol command and recover both parts of the LMF. On systems where the /usr and /var directories are located in separate filesets in usr_domain, recover the addvol command and the LMF utility into the usr fileset and recover the LMF database into the var fileset.

The following example shows how to restore a multivolume domain where the /var directory and the /usr directory are both in the fileset usr in usr_domain. The domain consists of the dsk1g, dsk2c, and dsk3c volumes. A backup tape of /usr and /var exists. (If you need to create a backup tape, see Chapter 4.) The procedure assumes that the root file system has already been restored. If it has not, see Section 6.3.4.

Mount the root fileset as read/write.
```
# mount -u /
```

Remove the links for the old usr_domain and create a new usr_domain using the initial volume.
```
# rm -rf /etc/fdmns/usr_domain
# mkfdmn /dev/disk/dsk1g usr_domain
```

Create and mount the /usr and /var filesets.

# mkfset usr_domain usr
# mount -t advfs usr_domain#usr /usr

Create a soft link in /usr because that is where the lmf command looks for its database.
```
# ln -s /var /usr/var
```

Insert the /usr backup tape and restore from it.

# cd /usr
# vrestore -vi
(/) add sbin/addvol 
(/) add sbin/lmf
(/) add var/adm/lmf
(/) extract
(/) quit

Reset the license database.
```
# /usr/sbin/lmf reset
```

Add the additional volumes to usr_domain.

# /usr/sbin/addvol /dev/disk/dsk2c usr_domain
# /usr/sbin/addvol /dev/disk/dsk3c usr_domain

Do a full restore of the /usr backup.
```
# cd /usr
# vrestore -xv
```

The following example shows how to restore a multivolume domain where the /usr and /var directories are in separate filesets in the same multivolume domain, usr_domain. The domain consists of the dsk1g, dsk2c, and dsk3c volumes. A backup tape of /usr and /var exists. (If you need to create a backup tape, see Chapter 4.) In this case you must mount both the /var and the /usr backup tapes. The procedure assumes that the root file system has already been restored. If it has not, see Section 6.3.4.

Mount the root fileset as read/write.
# mount -u /

Remove the links for the old usr_domain and create a new usr_domain using the initial volume.
```
# rm -rf /etc/fdmns/usr_domain
# mkfdmn /dev/disk/dsk1g usr_domain
```

Create and mount the /usr and /var filesets.

# mkfset usr_domain usr
# mkfset usr_domain var
# mount -t advfs usr_domain#usr /usr
# mount -t advfs usr_domain#var /var

Insert the /var backup tape and restore from it.

# cd /var
# vrestore -vi
(/) add adm/lmf
(/) extract
(/) quit

Insert the /usr backup tape and restore from it.

# cd /usr
# vrestore -vi
(/) add sbin/addvol
(/) add sbin/lmf
(/) extract
(/) quit

Reset the license database.
```
# /usr/sbin/lmf reset
```

Add the extra volumes to usr_domain.

# /usr/sbin/addvol /dev/disk/dsk2c usr_domain
# /usr/sbin/addvol /dev/disk/dsk3c usr_domain

Do a full restore of /usr backup.
```
# cd /usr
# vrestore -xv
```

Do a full restore of /var backup.
```
# cd /var
# vrestore -xv
```

6.3.8 Recovering from Accidental Use of the mkfdmn or addvol Command

The mkfdmn and addvol commands initialize a volume for use by AdvFS. If the volume is already part of an AdvFS domain, using these commands damages the metadata for the domain that existed on that volume. If you accidentally use one of these commands on a previously initialized volume, you might be able to recover some files by running the salvage utility with the -S option. See Section 6.2.6.3 for more information.

The more commands you execute after a mkfdmn or addvol command, the more you alter the original volume content and the less information you can recover with the salvage operation. If you cannot get a reasonable amount of information from running the salvage command, your only option is to restore the data from backup.

6.4 Preventing Problems

When each domain is mounted after a crash, the system automatically runs recovery code that checks the transaction log file to ensure that file system operations that were occurring when the system crashed are either completed or backed out. This ensures that AdvFS metadata is in a consistent state after a crash. It is not a good idea to recover your system by using an operating system other than the one that crashed.

If a system crash has left your data compromised, see Section 5.5 for ways to improve the method data is written to disk in order to prevent such problems in the future.

6.4.1 Checking Free Space and Disk Usage

You can look at the way space is allocated on a disk by file, fileset, or domain. Table 6-4 describes command-line commands that you can use to examine disk space usage.

Table 6-4: Disk Space Usage Commands

Command	Description
`df` ^{[Footnote 2]} ^{[Footnote 3]}	Displays disk space usage by fileset. Available space for a fileset is limited by the fileset quota if it is set. When both soft and hard quota limits are set, the command calculates the disk space available using the smaller of the two limits. If there is less space in the domain than is allowed by the fileset quota, the command displays the actual space available in the domain. In domains with multiple filesets and no fileset quotas, the total capacity of all filesets can be greater than 100%. The available space value used in the calculation is all the space in the domain because it is available to each fileset.
`du`	Displays information about block allocation for files. Use the `-a` option to display information for individual files.
`ls`	Displays the space used by files. The `-l` option shows the space spanned by a sparse file. The `-s` option shows the actual block usage and might be more useful for examining sparse files.
`showfdmn` ^{[Footnote 3]}	Displays the attributes and block usage for each volume in an active domain. For multivolume domains, additional volume information is displayed.
`showfile`	Displays block usage and volume information for a file or for the contents of a directory.
`showfsets` ^{[Footnote 3]}	Displays information about the filesets in a domain. Use the -q option to display fileset quota limits.
`vdf` ^{[Footnote 2]} ^{[Footnote 3]}	Reformats output from the `showfdmn`, `showfsets`, and `df` commands. The characteristics of these three commands apply to the `vdf` command.

The df and showfsets commands do not include the space used for metadata and quota files, so the value in the Used field may be smaller than that actually used by the domain. The vdf command displays disk usage that includes metadata and quota files. See Section 3.4.2.4 and the reference pages for more complete information.

Under certain conditions, the disk usage information for AdvFS can become corrupt. Run the quotacheck command with the -v option to correct the disk usage information.

For information about examining disk usage related to quotas, see Section 3.4.

6.4.2 Reusing Volumes

If you want to add storage from an existing domain (the /etc/fdmns directory entry exists) to another domain, you can remove the volume from the original domain by using the rmvol command and then add it to the other domain. AdvFS automatically updates the domain bookkeeping.

For example, if your volume is /dev/disk/dsk5c, your original domain is old_domain, and the domain you want to add the volume to is new_domain, mount all the filesets in old_domain, then enter:

# rmvol /dev/disk/dsk5c old_domain
# addvol /dev/disk/dsk5c new_domain

If the disk or disk partition you want to add is not part of an existing domain but is giving you a warning message because it is labeled, reset the disk label. You can do this without the disklabel command by answering yes to the prompt on the addvol or mkfdmn command. All information that is on the disk or disk partition that you are adding is lost. If you want to move a disk to use it on another system, see Section 6.3.6.

6.4.3 Failing Disks

Back up your data regularly and frequently, and watch for signs of impending disk failure. Try to remove files from a problem disk before it fails. If you have the AdvFS Utilities, you can use the addvol command to add a new disk then the rmvol command to remove the damaged one. See the event management information in the System Administration manual for more information about examining disk activity.

6.4.4 Controlling Disk Usage

If your system is running without any limits on resource usage, you can add quotas to limit the amount of disk space your users can access. User and group quotas, which are similar to UFS quotas, limit the amount of space a user or group can allocate for a fileset. Fileset quotas, which are unique to AdvFS, restrain a fileset from using all of the available space in a domain. See Chapter 3 for complete information on quotas.

6.4.5 Quota and Grace Period Limits

If you are close to your quota limit, or if it appears that you cannot get to that limit, you may be experiencing limitations of the quota system. See Chapter 3 for complete information on quotas.

6.4.5.1 Exceeding Quota Limits

If you are working in an editor and realize that the information you need to save exceeds your quota limit, do not abort the editor or write the file because data might be lost. Instead, do one of the following:

Remove enough files to make room for the edited file before you save it.

Save the file to another fileset, such as tmp, until you can remove files or increase quota limits for the fileset. Then move the file to your fileset.

6.4.5.2 Saving Up to Quota Limits

Hard and soft quota limits for the number of files are non-inclusive. You can only create files if you remain below the limit. For example, if your hard limit is 1000 files, you can only create 999 files.

Hard and soft quota limits for the number of blocks are similarly non-inclusive. In the rare case that you are 8K bytes below the user, group, or fileset quota and attempt to use some or all of the remaining space, you may be restricted by the quota limit. This is because AdvFS allocates storage in units of 8K bytes. If adding 8K bytes to a file meets or exceeds the quota limit, then that file is not extended.

6.4.5.3 Changing Quota Limits

If you set quota limits or grace periods for a fileset that is unmounted or does not have userquota and groupquota options in the /etc/fstab file (Section 2.4.1), the values you set are not retained. Update the /etc/fstab entry and mount your fileset before executing the edquota command.

6.4.5.4 Turning Off the Grace Period

To turn off a grace period, execute the edquota command to set the value to 1 second. Setting the value to 0 days imposes the default grace period of 7 days.

6.4.6 Avoiding Disk Structure Incompatibility

Domains created on operating system software Version 5.0 and later have a new on-disk format that is incompatible with earlier versions. (See Section 2.3.3 for more information.) The newer operating system recognizes the older disk structure, but older operating systems do not recognize the newer disk structure. If you install your Version 5 operating system software as an update to your Version 4 operating system software (not a full installation), your /root, /usr, and /var files retain a DVN of 3. If you fully install your Version 5 operating system, the /root, /usr, and /var files have a DVN of 4.

To access a DVN4 fileset from an older operating system, NFS mount the fileset from a server running Version 5.0 or later operating system software. If you try to mount a fileset belonging to a DVN4 domain when you are running a version of the operating system earlier than Version 5.0, you get an error message.

No tool exists to automatically upgrade DVN3 domains to DVN4. To upgrade a domain to DVN4, use the procedure in Section 2.3.3.4.

6.4.7 Avoiding Utility Incompatibility

Because Version 5.0 and later of the operating system have the new on-disk file formats, some AdvFS utilities from earlier releases might corrupt DVN4 domains. All statically-linked AdvFS-specific utilities from earlier operating system versions do not run on Version 5.0 and later. These utilities are usually from operating system versions prior to Version 4.0. In addition, the following dynamically-linked AdvFS utilities from earlier releases of Tru64 UNIX do not run on Version 5.0 and later:

advfsstat

balance

chvol

defragment

rmvol

showfdmn

verify

6.4.8 Avoiding Log File Inconsistency

When a system crashes, AdvFS performs recovery at reboot. Domains with filesets that were mounted at the time of the crash are recovered when one fileset in the domain is remounted. This recovery uses the AdvFS transaction log file to keep the AdvFS metadata consistent.

Since different versions of the operating system use different transaction log file structures, make sure that you recover your filesets on the version of the operating system that was running at the time of the crash. If you do not, you risk corrupting the domain metadata and/or panicking the domain.

If the system crashed because you set the AdvfsDomainPanicLevel attribute (Section 5.7) to promote a domain panic to a system panic, run the /sbin/advfs/verify command on the panicked domain to ensure that it is not damaged. If your filesets were unmounted at the time of the crash, or if you remounted them successfully and ran the verify command, you can mount the filesets on a different version of the operating system, if appropriate.

6.4.9 Increasing the Size of an AdvFS root Domain

The AdvFS root domain is limited to one volume (partition) unless you are running a cluster configuration. If you want to increase the size of the root domain, you must increase the size of the volume containing the root domain or recreate the root domain on a larger volume.

6.4.9.1 Increasing the Size of the root Volume

If you increase the size of the underlying volume containing the root domain, you must notify the system of the change by executing the mount command with the -o extend option as described in Section 2.3.4.3. If your root volume is an LSM volume, see the Logical Storage Manager manual.

6.4.9.2 Replacing the root Volume with a Larger Device

To recreate the root domain on a different device, you need the new device installed, an installation CD-ROM or a RIS server, and a backup tape or an unused disk partition to back up the root domain.

To move your root domain to a new disk, you must first install the disk and have it recognized. If the disk that is to become the root volume is already installed on the system, skip to step 5.

Shut down the system.

Install the new disk device. For more information see your hardware manuals.

Verify that the console recognizes the newly added disk. In this example, DKB300 was added.

>>> show device
polling ncr0(NCR 53C810) slot 1, bus 0 PCI, 
     hose 1 A SCSI Bus ID 7
dka500.5.0.1.1     DKA500                   RRD45     1645
polling isp0(QLogic ISP1020) slot 4, bus 0 PCI, 
     hose 1 SCSI Bus ID 7
dkb0.0.0.4.1       DKB0                     RZ1DB-CA  LYJ0
dkb100.1.0.4.1     DKB100                   RZ1CB-CA  LYJ0
dkb200.2.0.4.1     DKB200                   RZ1CB-CA  LYJ0
dkb300.3.0.4.1     DKB300                   RZ1BB-CS  0656 
mkb400.4.0.4.1     MKB400                   TLZ10     02ab
...

Reboot the system. During the boot process, the operating system recognizes the new device and updates the device information databases accordingly. Assume the operating system's device name for the added disk is dsk6.
To visually confirm that you have identified the correct device, use the /sbin/hwmgr command with the -flash option to cause the disk's light to flash for thirty seconds.
```
# /sbin/hwmgr -flash light -dsf /dev/disk/dsk6a
```

Log in as root and check the /etc/fdmns directory to determine the volume that contains root.
```
# ls -l /etc/fdmns/root_domain
total 0
lrwxrwxrwx   1 root     system 15 Feb 1 2002 dsk3a -> /dev/disk/dsk3a
```
In this example /dev/disk/dsk3a is the volume containing the root domain.

If needed, use the disklabel command with the -t advfs option or the diskconfig utility (choosing AdvFS from the Boot Block: list) to configure the new disk as AdvFS bootable. You must be the root user to perform this operation.
```
# disklabel -r dsk6 > /tmp/backupdisklabel
# disklabel -R -r -t advfs dsk6 /tmp/backupdisklabel
# rm /tmp/backupdisklabel
```

Recreate the root domain on the new device.
1. Shut down the system booted from your old root domain.
2. Boot from the operating system CD-ROM or RIS server. For example, if you are using the CD-ROM:
```
>>> boot dka500
```
  For example, if you are using the RIS server, enter the following:
```
>>> boot ewa0
```
3. Exit the installation. If you have a VGA graphics console, choose to exit the installation, or choose Shell Window from the File menu of the Installation and Configuration Welcome dialog box. If you have a serial console terminal, select option 3) Exit Installation.
4. Create a new root domain and root fileset on the new root device. Create a new entry in the /etc/fdmns directory to link to the original device.
```
# mkfdmn -r /dev/disk/dsk6a root_domain
# mkfset root_domain root 
# mkdir /var/new_root
# mount root_domain#root /var/new_root
# mkdir /etc/fdmns/orig_root
# ln -s /dev/disk/dsk3a /etc/fdmns/orig_root
# mount orig_root#root /var/orig_root
```
5. Dump the files from the old root domain and restore them to the new root domain.
```
# vdump -0f - /var/orig_root | vrestore -xf \
  -D /var/new_root
```

Update system bookkeeping to point to the new root volume. In this example, the root domain and the swap partition were moved from dsk3 to dsk6. Nothing else was changed.
Update the /etc/fdmns directory to identify the new root domain. Here dsk6a is the volume containing the new root domain and dsk3a is the volume containing the original root domain.
```
# cd /var/new_root/etc/fdmns/root_domain
# ln -s /dev/disk/dsk6a dsk6a
# rm dsk3a 
```

Halt the system and change the default boot device.

# halt
. . . 
>>> set bootdef_dev dkb300

Boot the new root domain.
```
>>> boot 
 
```

6.4.10 Memory Mapping, Direct I/O, and Data Logging Incompatibility

Unless you have turned on temporary atomic-write data logging by using the mount command with the -o adl option, memory mapping, atomic-write data logging, and direct I/O are mutually exclusive. If a file is open in one of these modes, attempting to open the same file in a conflicting mode fails. For more information see Section 5.5, Section 5.6, and mmap(2).

An file in memory-mapped mode is flushed by smoothsync, but may not be flushed according to smoothsync_age. The means that a file may remain temporarily in memory-mapped mode after being unmapped. See the System Configuration and Tuning manual for more information.

6.4.11 Preventing Access to Old Data

Obsolete data on disk might be recovered after a crash or when the salvage command is run unless old data is removed. To prevent access to this data, use the chfsets command with the -o objectsafety option. Choosing object safety prevents an application from reusing old data, but it does not guarantee complete data destruction.

When the object safety option is enabled, the pages on disk belonging to the fileset are zero-filled and forced to disk before they are available to the file. This prevents old data from being visible if a system crash occurs while the file is being written. Files that allocated storage just prior to enabling object safety initiate object safety once the buffers associated with the allocation are flushed to disk.

For example, to enable object safety in the xyz3 fileset in the xyz_domain:

# chfsets -o objectsafety xyz_domain xyz3

Object safety can degrade performance because pages must be written to disk twice.

6.4.12 Invalid or Corrupt Saveset Format

If you are restoring a saveset that has been saved on disk and get an error message that its format is invalid or corrupt, check that you have not backed the saveset up to partition a or c, which include block 0 of the disk. Block 0, the disk label block, is protected from accidental writes to it, so if you try to back to it, the saveset is not saved correctly.

To dump to a partition that starts at block 0 of a disk, you must first clear the disk label. If you do not, the output from the vdump command might appear to contain valid savesets, but when the vrestore command attempts to interpret the disk label as part of the saveset, it returns an error. See Section 4.3.7 for information about dumping to disk partitions.

6.4.13 Overcoming Poor Performance

The performance of a disk depends upon the I/O demands upon it. If you structure your domain so that heavy access is focused on one volume, system performance might degrade. If you have a multivolume domain, there are a number of ways that you can equalize the activity and increase throughput. See the System Configuration and Tuning manual and Chapter 5 for more complete information.

To discover the causes of poor performance, first check system activity. You can improve performance in a number of ways:

Upgrade domains (Section 2.3.3.4)

Eliminate disk access incompatibility (Section 5.6)

Defragment domains (Section 5.9)

Move filesets to different volumes (Section 5.14)

If you have AdvFS Utilities, you can also:

Change resource allocation by increasing the size of a domain or substituting a faster volume for a slower one.

Run the vfast utility (Section 5.8)

Balance a multivolume domain (Section 5.11)

Stripe individual files (Section 5.13)

Migrate individual files (Section 5.12)

6.4.14 The rmvol or migrate Command Will Not Start

If you are removing a volume with the rmvol command (Section 2.3.5) and abort the command, on rare occasions the volume is left in an inaccessible state and you cannot write to it. These volumes are marked as "data unavailable" in the output of the showfdmn command. If this occurs, use the chvol command with the -A option to reactivate the volume.