【Oracle ASM数据恢复】recovering COD cache read a corrupted block一例

某用户增加LUN到ASM DISKGROUP发现某个ASM Disk header KFBTYP_DISKHEAD被意外清除掉，导致该Diskgroup无法mount的问题， 后续DBA采用kfed merge等手法修复了KFBTYP_DISKHEAD block，但仍无法mount diskgroup，ALERT.log中出现如下的日志：

NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache opening disk 1 of grp 1: VOL2 label:VOL2
NOTE: cache opening disk 2 of grp 1: VOL3 label:VOL3
NOTE: cache opening disk 3 of grp 1: VOL4 label:VOL4
NOTE: cache opening disk 4 of grp 1: VOL5 label:VOL5
NOTE: cache opening disk 5 of grp 1: VOL6 label:VOL6
NOTE: cache opening disk 6 of grp 1: VOL7 label:VOL7
NOTE: cache opening disk 7 of grp 1: VOL8 label:VOL8
NOTE: cache opening disk 8 of grp 1: VOL9 label:VOL9
NOTE: cache opening disk 9 of grp 1: VOL10 label:VOL10
NOTE: cache opening disk 10 of grp 1: VOL11 label:VOL11
NOTE: cache mounting (first) group 1/0x3A2C35D6 (DG)
* allocate domain 1, invalid = TRUE 
kjbdomatt send to node 0
kjbdomatt send to node 2
Mon Jan 27 02:18:51 CST 2014
NOTE: attached to recovery domain 1
Mon Jan 27 02:18:51 CST 2014
NOTE: starting recovery of thread=1 ckpt=1712.152 group=1
NOTE: advancing ckpt for thread=1 ckpt=1712.153
NOTE: cache recovered group 1 to fcn 0.491275704
Mon Jan 27 02:18:51 CST 2014
NOTE: LGWR attempting to mount thread 1 for disk group 1
NOTE: LGWR mounted thread 1 for disk group 1
NOTE: opening chunk 1 at fcn 0.491275704 ABA 
NOTE: seq=1713 blk=154 
Mon Jan 27 02:18:51 CST 2014
NOTE: cache mounting group 1/0x3A2C35D6 (DG) succeeded
SUCCESS: diskgroup DG was mounted
Mon Jan 27 02:18:53 CST 2014
NOTE: recovering COD for group 1/0x3a2c35d6 (DG)
WARNING: cache read a corrupted block gn=1 dsk=0 blk=2817 from disk 0
NOTE: a corrupted block was dumped to the trace file
ERROR: cache failed to read dsk=0  blk=2817 from disk(s): 0
ORA-15196: invalid ASM block header [kfc.c:8281] [endian_kfbh] [2147483648] [2817] [173 != 1]
System State dumped to trace file /u01/app/oracle/admin/+ASM/bdump/+asm2_rbal_31204.trc
NOTE: cache initiating offline of disk 0  group 1
WARNING: process 31204 initiating offline of disk 0.3913073997 (VOL1) with mask 0x3 in group 1
WARNING: Disk 0 in group 1 in mode: 0x7,state: 0x2 will be taken offline
NOTE: PST update: grp = 1, dsk = 0, mode = 0x6
Mon Jan 27 02:18:54 CST 2014
ERROR: too many offline disks in PST (grp 1)
Mon Jan 27 02:18:54 CST 2014
WARNING: Disk 0 in group 1 in mode: 0x7,state: 0x2 was taken offline
Mon Jan 27 02:18:54 CST 2014
NOTE: halting all I/Os to diskgroup DG
NOTE: active pin found: 0x0x65faff60
NOTE: active pin found: 0x0x65fb0170
NOTE: active pin found: 0x0x65fb0010
NOTE: active pin found: 0x0x65fb0220
NOTE: active pin found: 0x0x65fb02d0
NOTE: active pin found: 0x0x65fb00c0
NOTE: active pin found: 0x0x65fb0380
Mon Jan 27 02:18:54 CST 2014
ERROR: ORA-15130 in COD recovery for diskgroup 1/0x3a2c35d6 (DG)
ERROR: ORA-15130 thrown in RBAL for group number 1
Mon Jan 27 02:18:54 CST 2014
Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm2_rbal_31204.trc:
ORA-15130: diskgroup "DG" is being dismounted
Mon Jan 27 02:18:54 CST 2014
ERROR: PST-initiated MANDATORY DISMOUNT of group DG
NOTE: cache dismounting group 1/0x3A2C35D6 (DG) 
Mon Jan 27 02:18:57 CST 2014
kjbdomdet send to node 0
detach from dom 1, sending detach message to node 0
kjbdomdet send to node 2
detach from dom 1, sending detach message to node 2
Mon Jan 27 02:18:57 CST 2014
Dirty detach reconfiguration started (old inc 23, new inc 23)
List of nodes:
 0 1 2
 Global Resource Directory partially frozen for dirty detach 
* dirty detach - domain 1 invalid = TRUE 
 138 GCS resources traversed, 0 cancelled
 6104 GCS resources on freelist, 6124 on array, 6124 allocated
Dirty Detach Reconfiguration complete
Mon Jan 27 02:18:57 CST 2014
freeing rdom 1
Mon Jan 27 02:18:57 CST 2014
WARNING: dirty detached from domain 1
Mon Jan 27 02:18:57 CST 2014
SUCCESS: diskgroup DG was dismounted
Mon Jan 27 02:18:57 CST 2014
WARNING: PST-initiated MANDATORY DISMOUNT of group DG not performed - group not mounted
Mon Jan 27 02:18:57 CST 2014
Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm2_b001_31755.trc:
ORA-15001: diskgroup "DG" does not exist or is not mounted
ORA-15001: diskgroup "DG" does not exist or is not mounted
ORA-15001: diskgroup "DG" does not exist or is not mounted
Mon Jan 27 02:31:00 CST 2014

这里可以看到Diskgroup mount到了recovering COD for group 1/0x3a2c35d6 (DG)阶段时，发现了一个逻辑坏块WARNING: cache read a corrupted block gn=1 dsk=0 blk=2817 from disk 0 NOTE: a corrupted block was dumped to the trace file ERROR: cache failed to read dsk=0 blk=2817 from disk(s): 0，并因为该坏块引起了ORA-15196: invalid ASM block header [kfc.c:8281] [endian_kfbh] [2147483648] [2817] [173 != 1]。

这里2817是出错的ASM metadata的block number，173是实际从endian_kfbh位置读出的值，173!=1 这里的1是该位置理论上该有的值，由于读取到block中错误的字节序endian_kfbh信息，所以这里出现了ASM ORA-600错误。

这里recovering COD for group 1/0x3a2c35d6 (DG) 里的COD 指asm metadata file number 4 COD,  Continuing Operation Directory (COD) 该metadata file 4 中记录的是在单个metadata block中无法完成的操作记录到COD中，这样当ASM instance crash时可以恢复这些操作。例如创建 删除和resize文件，这其中file number 4 blkn=1为KFBTYP_COD_RB  即回滚rollback数据，后面的数据为KFBTYP_COD_DATA。

可回滚的操作opcodes包括：

1 - Create a file
2 - Delete a file
3 - Resize a file
4 - Drop alias entry
5 - Rename alias entry
6 - Rebalance space COD
7 - Drop disks force
8 - Attribute drop
9 - Disk Resync
10 - Disk Repair Time
11 - Volume create
12 - Volume delete
13 - Attribute directory creation
14 - Set zone attributes
15 - User drop

每次ASM diskgroup 尝试mount时都会读取FILE number 4 COD中的数据来保证操作要么完成、要么回滚。

对于此类ASM file number 4 COD出现了源数据坏块的情况， 一般需要手动设置内部事件，并尝试手动Patch ASM metadata的手法才能修复。

建议遇到此类事件第一时间备份ASM disk header 100M的数据，保护现场，以便专业恢复人员介入恢复时现场不被破坏。

如果自己搞不定可以找ASKMACLEAN专业ORACLE数据库修复团队成员帮您恢复!