If you cannot recover the data by yourself, ask Parnassusdata, the professional ORACLE database recovery team for help.
Parnassusdata Software Database Recovery Team
Service Hotline: +86 13764045638 E-mail: [email protected]
Customer has migrated oracle databases running on old SAN to new SAN using ASM rebalance operation. Customer is using External redundancy. After completing rebalance operation, customer rebooted all server and removed old SAN device entry on weekend. Customer is unable to bring databases online on One of the 4 servers. Customer is getting following error :
ORA-15032: not all alterations performed
ORA-15017: diskgroup “DATA” cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATA”
I have attached asm-kfed result for your reference. Is it possible to recover ASM header files without so customer doesn’t need to backup/restore 5 TB database ?
Total System Global Area 284008448 bytes
Fixed Size 2158616 bytes
Variable Size 256684008 bytes
ASM Cache 25165824 bytes
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"
SQL> select group_number,disk_number,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,NAME,PATH from V$asm_disk;
GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE NAME PATH
------------ ----------- ------- ------------ ------- -------- ------------------------------ ----------------------------------------
0 0 CLOSED CANDIDATE ONLINE NORMAL /dev/rdsk/c0d26s6
0 23 IGNORED MEMBER ONLINE NORMAL /dev/rdsk/c0d22s6
0 2 IGNORED MEMBER ONLINE NORMAL /dev/rdsk/san03dp_dbs05dp_ASM01
0 3 IGNORED MEMBER ONLINE NORMAL /dev/rdsk/san03dp_dbs05dp_ASM12
0 4 CLOSED CANDIDATE ONLINE NORMAL /dev/rdsk/c0d27s6
0 5 CLOSED CANDIDATE ONLINE NORMAL /dev/rdsk/c0d25s6
0 6 CLOSED CANDIDATE ONLINE NORMAL /dev/rdsk/san03dp_dbs05dp_ASM05
0 7 IGNORED MEMBER ONLINE NORMAL /dev/rdsk/san03dp_dbs05dp_ASM02
0 8 CLOSED CANDIDATE ONLINE NORMAL /dev/rdsk/san03dp_dbs05dp_ASM08
0 9 CLOSED CANDIDATE ONLINE NORMAL /dev/rdsk/c0d24s6
0 10 IGNORED MEMBER ONLINE NORMAL /dev/rdsk/san03dp_dbs05dp_ASM11
0 11 IGNORED MEMBER ONLINE NORMAL /dev/rdsk/c0d30s6
0 12 CLOSED CANDIDATE ONLINE NORMAL /dev/rdsk/san03dp_dbs05dp_ASM07
0 13 IGNORED MEMBER ONLINE NORMAL /dev/rdsk/c0d31s6
0 14 IGNORED MEMBER ONLINE NORMAL /dev/rdsk/c0d21s6
0 15 IGNORED MEMBER ONLINE NORMAL /dev/rdsk/san03dp_dbs05dp_ASM10
0 16 CLOSED CANDIDATE ONLINE NORMAL /dev/rdsk/c0d23s6
0 17 IGNORED MEMBER ONLINE NORMAL /dev/rdsk/san03dp_dbs05dp_ASM09
0 18 CLOSED CANDIDATE ONLINE NORMAL /dev/rdsk/san03dp_dbs05dp_ASM03
0 19 IGNORED MEMBER ONLINE NORMAL /dev/rdsk/c0d29s6
0 20 CLOSED CANDIDATE ONLINE NORMAL /dev/rdsk/c0d28s6
0 21 CLOSED CANDIDATE ONLINE NORMAL /dev/rdsk/san03dp_dbs05dp_ASM04
0 22 IGNORED MEMBER ONLINE NORMAL /dev/rdsk/c0d32s6
0 1 CLOSED CANDIDATE ONLINE NORMAL /dev/rdsk/san03dp_dbs05dp_ASM06
dev/rdsk/san03dp_dbs05dp_ASM03
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
10037FE00 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
/dev/rdsk/san03dp_dbs05dp_ASM04
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
10037FE00 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
/dev/rdsk/san03dp_dbs05dp_ASM05
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
10037FE00 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
/dev/rdsk/san03dp_dbs05dp_ASM06
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
10037FE00 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
/dev/rdsk/san03dp_dbs05dp_ASM07
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
10037FE00 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
/dev/rdsk/san03dp_dbs05dp_ASM08
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
10037FE00 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
ASM disks – ASM03/04/05/06/07/08 – showing the status as “CANDIDATE” is a bit worry. But if the devices were not part of DATA diskgroup before, these devices are not the major cause of ORA-15063.
Please check ASM alert.log whether these 5 devices were belong to DATA.
I’m more concerned about the following 4 devices as they show the status as “IGNORED” which indicates there are other devices showing the same disk information given asm_diskstring parameter.
– ASM01/02/0910/11
Chances are that the following devices below show the same disk information as ASM01/02/0910/11 and there is good chance that these different path point to the same physical devices.
~~
/dev/rdsk/c0d22s6
/dev/rdsk/c0d30s6
/dev/rdsk/c0d31s6
/dev/rdsk/c0d21s6
/dev/rdsk/c0d29s6
/dev/rdsk/c0d32s6
~~
Please check all disk header which disks show the duplicate ASM disk information using the following perl script in the note below
And if duplicate paths point to the same physical device, the additional device path should be disabled using “chmod 000 <device_path>”
– KFED.PL for diagnosing – ORA-15063 ORA-15042 ORA-15020 (Doc ID 1346190.1)
Leave a Reply