Recover/Start a volume that has a plex in the disabled & recover state

The vxprint -ht output gives, among other things, the kernel state (column 4) and the state (column 5) of a plex .

Consider the following case:

# vxprint -ht -g testdg

DG     NAME     NCONFIG     NLOG     MINORS     GROUP-ID
DM     NAME     DEVICE     TYPE     PRIVLEN     PUBLEN     STATE
RV     NAME     RLINK_CNT     KSTATE     STATE     PRIMARY     DATAVOLS     SRL
RL     NAME     RVG     KSTATE     STATE     REM_HOST     REM_DG     REM_RLNK
V     NAME     RVG     KSTATE     STATE     LENGTH     USETYPE     PREFPLEX RDPOL
PL     NAME     VOLUME     KSTATE     STATE     LENGTH     LAYOUT     NCOL/WID MODE
SD     NAME     PLEX     DISK     DISKOFFS     LENGTH     [COL/]OFF     DEVICE MODE
SV     NAME     PLEX     VOLNAME     NVOLLAYR     LENGTH     [COL/]OFF     AM/NM MODE

dg     testdg     default     default     84000     970356463.1203.alu

dm     testdg01     c1t4d0s2     sliced     2179     8920560     -
dm     testdg02     c1t6d0s2     sliced     2179     8920560     -

v     test     –     DISABLED     ACTIVE     17840128     fsgen     –     SELECT
pl     test-01     test     DISABLED     RECOVER     17841120     CONCAT     –     RW
sd     testdg01-01     test-01     testdg01     0     8920560     0     c1t4d0     ENA
sd     testdg02-01     test-01     testdg02     0     8920560     8920560     c1t6d0     ENA

From the above output, it can be seen that the volume test has plex test-01 in the DISABLED and RECOVER state.

To recover the volume test, use the vxmend command. This operation applies only to volumes, or to plexes associated with a volume.  This will manually reset or change the state of a plex or volume.

The following is the procedure to recover/start this volume:

1. Bring plex test-01 to the DISABLED and OFFLINE state using the following command: vxmend -o force off  <recover_plex>

For example,

# vxmend -o force off test-01

The below output shows plex test-01 in the DISABLED and OFFLINE state:

# vxprint -ht -g testdg

DG     NAME     NCONFIG     NLOG     MINORS     GROUP-ID
DM     NAME     DEVICE     TYPE     PRIVLEN     PUBLEN     STATE
RV     NAME     RLINK_CNT     KSTATE     STATE     PRIMARY     DATAVOLS     SRL
RL     NAME     RVG     KSTATE     STATE     REM_HOST     REM_DG     REM_RLNK
V     NAME     RVG     KSTATE     STATE     LENGTH     USETYPE     PREFPLEX     RDPOL
PL     NAME     VOLUME     KSTATE     STATE     LENGTH     LAYOUT     NCOL/WID     MODE
SD     NAME     PLEX     DISK     DISKOFFS     LENGTH     [COL/]OFF     DEVICE     MODE
SV     NAME     PLEX     VOLNAME     NVOLLAYR     LENGTH     [COL/]OFF     AM/NM     MODE

dg     testdg     default     default     84000     970356463.1203.alu

dm     testdg01     c1t4d0s2     sliced     2179     8920560     -
dm     testdg02     c1t6d0s2     sliced     2179     8920560     -

v     test     –     DISABLED     ACTIVE     17840128     fsgen     –     SELECT
pl     test-01     test     DISABLED     OFFLINE     17841120     CONCAT     –     RW
sd     testdg01-01     test-01     testdg01     0     8920560     0     c1t4d0     ENA
sd     testdg02-01     test-01     testdg02     0     8920560     8920560     c1t6d0     ENA

2. Bring plex test-01 to the DISABLED and STALE state using the following command: vxmend on  <recover_plex>

For example,

# vxmend on test-01

The below output shows plex test-01 in the DISABLED and STALE state:

# vxprint -ht -g testdg

DG     NAME     NCONFIG     NLOG     MINORS     GROUP-ID
DM     NAME     DEVICE     TYPE     PRIVLEN     PUBLEN     STATE
RV     NAME     RLINK_CNT     KSTATE     STATE     PRIMARY     DATAVOLS     SRL
RL     NAME     RVG     KSTATE     STATE     REM_HOST     REM_DG     REM_RLNK
V     NAME     RVG     KSTATE     STATE     LENGTH     USETYPE     PREFPLEX     RDPOL
PL     NAME     VOLUME     KSTATE     STATE     LENGTH     LAYOUT     NCOL/WID     MODE
SD     NAME     PLEX     DISK     DISKOFFS     LENGTH     [COL/]OFF     DEVICE     MODE
SV     NAME     PLEX     VOLNAME     NVOLLAYR     LENGTH     [COL/]OFF     AM/NM     MODE

dg     testdg     default     default     84000     970356463.1203.alu

dm     testdg01     c1t4d0s2     sliced     2179     8920560     -
dm     testdg02     c1t6d0s2     sliced     2179     8920560     -

v     test     –     DISABLED     ACTIVE     17840128     fsgen     –     SELECT
pl     test-01     test     DISABLED     STALE     17841120     CONCAT     –     RW
sd     testdg01-01     test-01     testdg01     0     8920560     0     c1t4d0     ENA
sd     testdg02-01     test-01     testdg02     0     8920560     8920560     c1t6d0     ENA

3. Bring plex test-01 to the DISABLED and CLEAN state using the following command: vxmend fix clean  <recover_plex>

For example,

# vxmend fix clean test-01

The below output shows plex test-01 in the DISABLED and CLEAN state:

# vxprint -ht -g testdg

DG     NAME     NCONFIG     NLOG     MINORS     GROUP-ID
DM     NAME     DEVICE     TYPE     PRIVLEN     PUBLEN     STATE
RV     NAME     RLINK_CNT     KSTATE     STATE     PRIMARY     DATAVOLS     SRL
RL     NAME     RVG     KSTATE     STATE     REM_HOST     REM_DG     REM_RLNK
V     NAME     RVG     KSTATE     STATE     LENGTH     USETYPE     PREFPLEX     RDPOL
PL     NAME     VOLUME     KSTATE     STATE     LENGTH     LAYOUT     NCOL/WID     MODE
SD     NAME     PLEX     DISK     DISKOFFS     LENGTH     [COL/]OFF     DEVICE     MODE
SV     NAME     PLEX     VOLNAME     NVOLLAYR     LENGTH     [COL/]OFF     AM/NM     MODE

dg     testdg     default     default     84000     970356463.1203.alu

dm     testdg01     c1t4d0s2     sliced     2179     8920560     -
dm     testdg02     c1t6d0s2     sliced     2179     8920560     -

v     test     –     DISABLED     ACTIVE     17840128     fsgen     –     SELECT
pl     test-01     test     DISABLED     CLEAN     17841120     CONCAT     –     RW
sd     testdg01-01     test-01     testdg01     0     8920560     0     c1t4d0     ENA
sd     testdg02-01     test-01     testdg02     0     8920560     8920560     c1t6d0     ENA

4. Once plex test-01 is in the DISABLED/CLEAN state, the volume test can be started with the following command: vxvol start  <volume>

For example,

# vxvol start test

It can be seen in the below output that the volume is now ENABLED and ACTIVE:

# vxprint -ht -g testdg

DG     NAME     NCONFIG     NLOG     MINORS     GROUP-ID
DM     NAME     DEVICE     TYPE     PRIVLEN     PUBLEN     STATE
RV     NAME     RLINK_CNT     KSTATE     STATE     PRIMARY     DATAVOLS     SRL
RL     NAME     RVG     KSTATE     STATE     REM_HOST     REM_DG     REM_RLNK
V     NAME     RVG     KSTATE     STATE     LENGTH     USETYPE     PREFPLEX     RDPOL
PL     NAME     VOLUME     KSTATE     STATE     LENGTH     LAYOUT     NCOL/WID     MODE
SD     NAME     PLEX     DISK     DISKOFFS     LENGTH     [COL/]OFF     DEVICE     MODE
SV     NAME     PLEX     VOLNAME     NVOLLAYR     LENGTH     [COL/]OFF     AM/NM     MODE

dg     testdg     default     default     84000     970356463.1203.alu

dm     testdg01     c1t4d0s2     sliced     2179     8920560     -
dm     testdg02     c1t6d0s2     sliced     2179     8920560     -

v     test     –     ENABLED     ACTIVE     17840128     fsgen     –     SELECT
pl     test-01     test     ENABLED     ACTIVE     17841120     CONCAT     –     RW
sd     testdg01-01     test-01     testdg01     0     8920560     0     c1t4d0     ENA
sd     testdg02-01     test-01     testdg02     0     8920560     8920560     c1t6d0     ENA

Read Full Post »

Backup/Recove VxVM config for a disk group

The following are instructions on how to backup VxVM configuration information from the private region of all VxVM disks and other relevant information to recover after misconfiguration or failure. This is also good to do for future reference in case of failure.

Save the outputs of the following commands in separate files (which could be retrieved later):

1.  # vxdisk list     (this contains the mapping between da (e.g., c0t0d0) and dm (e.g., disk01) records)
2.  # vxdisk -q list | awk ‘{print $1}’ | xargs vxdisk list     (this contains detailed info for each disk)
3.  # vxdisk -q list | awk ‘{if ($2 != “simple”) print “/dev/rdsk/”$1}’ | xargs prtvtoc     (this contains the VTOC info for each disk)

Also do so for each disk group. If the version of the VxVM package is 3.x or later then apply step 4b only, otherwise apply 4a only:

4a. # vxprint -g disk_group_name -hmvps   (this contains detailed info for all the volume, plex, and subdisk objects for the specified disk group)
4b. # vxprint -g disk_group_name -rhmvps  (this contains detailed info for all the volume, sub volume, plex, and subdisk objects for the specified disk group)

It will be ideal if the above output is backed up every time after a configurational change (please see the explanation at the end of this TechNote). What is not a good idea is to set up a cron job to back them up often.

For example, (if it is VxVM 2.x):

# vxdisk list > vxdisk_list
# vxdisk -q list | awk ‘{print $1}’ | xargs vxdisk list > vxdisk_list_detail
# vxdisk -q list | awk ‘{if ($2 != “simple”) print “/dev/rdsk/”$1}’ | xargs prtvtoc > prtvtoc_all_disk
# vxprint -g datadg -hmvps > vxprint_datadg_config

For example, (if it is VxVM 3.x):

# vxdisk list > vxdisk_list
# vxdisk -q list | awk ‘{print $1}’ | xargs vxdisk list > vxdisk_list_detail
# vxdisk -q list | awk ‘{if ($2 != “simple”) print “/dev/rdsk/”$1}’ | xargs prtvtoc > prtvtoc_all_disk
# vxprint -g datadg -rhmvps > vxprint_datadg_config

In the event of a non-rootdg disk group configuration failure or error (such as failure to import a disk group by all means) that requires a rebuild of the entire configuration database for the disk group, these outputs can be used to recreate the disk groups (note that step 1 is not necessary if the desired disk group is importable.)

1.  Re-initialize (or recreate) the disk group exactly according to the vxdisk list output, i.e., make sure that the mapping between the device names (disk access names (da names)) and the logical names (disk media names (dm names)) is exactly the same as the one in the vxdisk list output:

# vxdg init disk_group_name  dm_name_1=da_name_1  dm_name_2=da_name_2  …

For example,

# vxdg init datadg d1=c1t1d1 d2=c1t1d2 …

For a disk group with a large number of disks, use:

# vxdg init datadg `awk ‘{if ($4 == “datadg”) print $3″=”$1}’ vxdisk_list`

Extreme caution should be taken for this step for various reasons. For instance, if two da names are swapped (e.g., c1t1d1 and c1t1d2), mapping between the da names and dm names according to the not-yet-updated vxdisk list output will lead to a potential disaster. At this point, the detailed vxdisk list could help if the private regions of the disks have not been removed or damaged. The disk ID can be obtained from the on-disk private region for each disk (e.g., vxdisk list c1t1d1 and c1t1d2, and so on) and then can be compared with that from the saved detailed vxdisk list output,  hence, a recovery of the correct mapping.

If this step does not succeed or there are some doubts at this point, please call Customer Support immediately since it may require initializing disks, which would require careful examination of the other outputs that have been backed up (i.e., the detailed vxdisk list and the VTOC info) and the particular nature of the situation.

2.  If step 1 is successful, then execute the vxmake command using the vxprint output:

# vxmake -g disk_group_name -d config_file_name

For example,

# vxmake -g datadg -d vxprint_datadg_config

This should restore the original configuration for the disk group.

3. If step 2 completes successfully, then start all the volumes with vxvol -g disk_group_name startall or one by one with vxvol -g disk_group_name start volume_name. Now it is time to verify the data on the volume. If there is a file system on the volume then check the file systemn using fsck or mount the file system. If there is no file system on the volume then use appropriate ways to verify the data.

Configurational changes refer to any changes that alter the state or attribute of the volume manager objects, which includes the disk group, disk, volume, plex, subdisk, etc. (For example, add or remove a disk to and from a disk group, attach or detach or disassociate a plex, shrink or grow a volume, and many more things.)

Read Full Post »

Deport/Import VxVM groups

A disk group can be moved between systems along with the VERITAS Volume Manager ™ objects (except for rootdg). This will relocate the disk group configuration to another system.

The procedure is:

1. Unmount and stop all volumes in the disk group on the first system:

# umount /mntdir
# vxvol -g <diskgroup> stopall

2. Deport (disable all local access to) the disk group to be moved with this command:

# vxdg deport <diskgroup>

3. Import (enable local access to) the disk group and its disks from the second system with:

# vxdg import <diskgroup>

4.  After the disk group is imported, start all volumes in the disk group with this command:
# vxrecover -g <diskgroup> -sb

The options here indicate that VERITAS Volume Manager will start all the disabled volumes (-s) in the background (-b).

Read Full Post »

Install CLARiiON devices VxVM

#To See new disks/luns added to the system
devfsadm -v

#Label the new drives
1a. %format
1b. select disk and choose label

# Have veritas find the newly zoned volumes
vxdiskconfig

# Verify new disks were added
vxdisk list

# Add disks to veritas – Be sure to enter the appropriate disk group and “initialize” the disk — Do not encapsulate
vxdiskadd EMC_CLARiiON0_1

Add or initialize disks
Menu: VolumeManager/Disk/AddDisks

Here is the disk selected.  Output format: [Device_Name]

EMC_CLARiiON0_1

Continue operation? [y,n,q,?] (default: y) y

You can choose to add this disk to an existing disk group, a
new disk group, or leave the disk available for use by future
add or replacement operations.  To create a new disk group,
select a disk group name that does not yet exist.  To leave
the disk available for future use, specify a disk group name
of “none”.

Which disk group [<group>,none,list,q,?] (default: rootdg) vgora01

There is no active disk group named vgora01.

Create a new group named vgora01? [y,n,q,?] (default: y) y

Create the disk group as a CDS disk group? [y,n,q,?] (default: y) n

Use a default disk name for the disk? [y,n,q,?] (default: y) n

Add disk as a spare disk for vgora01? [y,n,q,?] (default: n)

Exclude disk from hot-relocation use? [y,n,q,?] (default: n)

A new disk group will be created named vgora01 and the selected disks
will be added to the disk group with disk names that you will specify
interactively.

EMC_CLARiiON0_1

Continue with operation? [y,n,q,?] (default: y)

The following disk device has a valid VTOC, but does not appear to have
been initialized for the Volume Manager.  If there is data on the disk
that should NOT be destroyed you should encapsulate the existing disk
partitions as volumes instead of adding the disk as a new disk.
Output format: [Device_Name]

EMC_CLARiiON0_1

Encapsulate this device? [y,n,q,?] (default: y) n

EMC_CLARiiON0_1

Instead of encapsulating, initialize? [y,n,q,?] (default: n) y

Initializing device EMC_CLARiiON0_1.

Enter the desired format
[cdsdisk,sliced,simple,q,?] (default: cdsdisk) sliced

Use a default private region length for this disk?
[y,n,q,?] (default: y)

Enter disk name for EMC_CLARiiON0_1
[<name>,q,?] (default: vgora0101) vgora01_01

Creating a new disk group named vgora01 containing the disk
device EMC_CLARiiON0_1 with the name vgora01_01.

Goodbye.
# vxassist make u01 20g dg=vgora01
# vxassist make u02 15g dg=vgora01
# vxassist make u03 15g dg=vgora01
# vxassist make u04 15g dg=vgora01
# vxassist make u05 14g dg=vgora01

when I specify 15g for /u06 it stated I didn’t have enough space
# vxassist maxsize dg=vgora01
Maximum volume size: 31436800 (15350Mb)
# vxassist make u06 15350m dg=vgora01

===
# Sizing for vgora02
# vxassist make u07 999m dg=vgora02
# vxassist make u08 999m dg=vgora02
# vxassist make u09 999m dg=vgora02
# vxassist make u10 999m dg=vgora02
# vxassist make u11 999m dg=vgora02
# vxassist make u12 999m dg=vgora02

## – Now Format & mount the filesystems:

# mkfs -F vxfs /dev/vx/rdsk/vgora01/u01
# mkfs -F vxfs /dev/vx/rdsk/vgora01/u02
# mkfs -F vxfs /dev/vx/rdsk/vgora01/u03 …

Read Full Post »

VxVM Replace Bad mirror

Remove mirror disk from Volume Manager running on the server

List disk name
dm rootdisk     c1t1d0s2     -        71118513 -
dm rootmirror   c1t0d0s2     -        71118513 –

In this case rootdisk is the bad disk. Remove it from the disk group.

Dissociate plexes from bad disk.
#vxplex –g rootdg dis rootvol-0X swapvol-0X var-0X opt-0X home-0X
#vxedit –g rootdg –fr rm rootvol-0X swapvol-0X var-0X opt-0X home-0X

Remove from Volume Manager control with this command.
#vxdg –g rootdg rmdisk rootdisk

Replace drive.
#format
#label

Initialize rootdisk and move into rootdg.
#/etc/vx/bin/vxdisksetup –I c1t1d0
#vxdg -g rootdg adddisk rootdisk=c1t1d0

Attach Mirrors in correct order
#/etc/vx/bin/vxrootmir rootdisk
#vxassist –g rootdg mirror swapvol rootdisk
#vxprint -g rootdg
#vxassist –g rootdg mirror var rootdisk
#vxassist –g rootdg mirror opt rootdisk
#vxassist –g rootdg mirror home rootdisk

Create the underlying partitions on the primary boot disk.
#/usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-02 1 0×03 0×01
#/usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror-02 1 0×03 0×01
#dumpadm -d /dev/dsk/c1t1d0s1

Setup OBP nvramrc settings
#ls -l /dev/dsk/c[12]t[0189]d0s0
/dev/dsk/c1t0d0s0 -> ../../devices/pci@1f,700000/scsi@2/sd@0,0:a
/dev/dsk/c1t1d0s0 -> ../../devices/pci@1f,700000/scsi@2/sd@1,0:a

#cp /var/adm/doc/20040115.eeprom.nvramrc.out /var/tmp/nv
# vi /var/tmp/nv
devalias rootmirror /pci@1f,700000/scsi@2/disk@0,0:a
devalias rootdisk /pci@1f,700000/scsi@2/disk@1,0:a

#eeprom “boot-device=rootdisk rootmirror”
#eeprom “use-nvramrc?=true”
#eeprom “nvramrc=`cat /var/tmp/nv`”

Test reboot.

Read Full Post »