Our team deleted some archivelog by mistake. Rolled the database forwards by RMAN incremental recovery to an SCN. Did a manual recovery to sync it with the primary. Managed recovery is now failing.
alter database recover managed standby database disconnect
Alert log has :
Fri Jan 22 13:50:22 2010 Attempt to start background Managed Standby Recovery process MRP0 started with pid=12 MRP0: Background Managed Standby Recovery process started Media Recovery Waiting for thread 1 seq# 193389 Fetching gap sequence for thread 1, gap sequence 193389-193391 Trying FAL server: ITS Fri Jan 22 13:50:28 2010 Completed: alter database recover managed standby database d Fri Jan 22 13:53:25 2010 Failed to request gap sequence. Thread #: 1, gap sequence: 193389-193391 All FAL server has been attempted.
Managed recovery was working earlier today after the Rman incremental and resolved two gaps automatically. But it now appears hung with the standby falling behind the primary.
SQL> show parameter fal NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ fal_client string ITS_STBY fal_server string ITS [v08k608:ITS:oracle]$ tnsping ITS_STBY TNS Ping Utility for Solaris: Version 9.2.0.7.0 - Production on 22-JAN-2010 15:01:17 Copyright (c) 1997 Oracle Corporation. All rights reserved. Used parameter files: /oracle/product/9.2.0/network/admin/sqlnet.ora Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL= TCP)(Host= v08k608.am.mot.com)(Port= 1526)) (CONNECT_DATA = (SID = ITS))) OK (10 msec) [v08k608:ITS:oracle]$ tnsping ITS TNS Ping Utility for Solaris: Version 9.2.0.7.0 - Production on 22-JAN-2010 15:01:27 Copyright (c) 1997 Oracle Corporation. All rights reserved. Used parameter files: /oracle/product/9.2.0/network/admin/sqlnet.ora Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL= TCP)(Host= 187.10.68.75)(Port= 1526)) (CONNECT_DATA = (SID = ITS))) OK (320 msec) Primary has : SQL> show parameter log_archive_dest_2 log_archive_dest_2 string SERVICE=DRITS_V08K608 reopen=6 0 max_failure=10 net_timeout=1 80 LGWR ASYNC=20480 OPTIONAL NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ log_archive_dest_state_2 string ENABLE [ITS]/its15/oradata/ITS/arch> tnsping DRITS_V08K608 TNS Ping Utility for Solaris: Version 9.2.0.7.0 - Production on 22-JAN-2010 15:03:24 Copyright (c) 1997 Oracle Corporation. All rights reserved. Used parameter files: /oracle/product/9.2.0/network/admin/sqlnet.ora Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL= TCP)(Host= 10.177.13.57)(Port= 1526)) (CONNECT_DATA = (SID = ITS))) OK (330 msec)
The arch process on the primary database might hang due to a bug below so that it couldn’t ship the missing archive log
files to the standby database.
BUG 6113783 ARC PROCESSES CAN HANG INDEFINITELY ON NETWORK
[ Not published so not viewable in My Oracle Support ]
Fixed 11.2, 10.2.0.5 patchset
We could work workaround the issue by killing the arch processes on the primary site and they will be respawned
automatically immediately without harming the primary database.
[maclean@rh2 ~]$ ps -ef|grep arc maclean 8231 1 0 22:24 ? 00:00:00 ora_arc0_PROD maclean 8233 1 0 22:24 ? 00:00:00 ora_arc1_PROD maclean 8350 8167 0 22:24 pts/0 00:00:00 grep arc [maclean@rh2 ~]$ kill -9 8231 8233 [maclean@rh2 ~]$ ps -ef|grep arc maclean 8389 1 0 22:25 ? 00:00:00 ora_arc0_PROD maclean 8391 1 1 22:25 ? 00:00:00 ora_arc1_PROD maclean 8393 8167 0 22:25 pts/0 00:00:00 grep arc and alert log will have: Fri Jul 30 22:25:27 EDT 2010 ARCH: Detected ARCH process failure ARCH: Detected ARCH process failure ARCH: STARTING ARCH PROCESSES ARC0 started with pid=26, OS id=8389 Fri Jul 30 22:25:27 EDT 2010 ARC0: Archival started ARC1: Archival started ARCH: STARTING ARCH PROCESSES COMPLETE ARC1 started with pid=27, OS id=8391 Fri Jul 30 22:25:27 EDT 2010 ARC0: Becoming the 'no FAL' ARCH ARC0: Becoming the 'no SRL' ARCH Fri Jul 30 22:25:27 EDT 2010 ARC1: Becoming the heartbeat ARCH
Actually if we don’t kill some fatal process in 10g , oracle will respawn all nonfatal processes.
For example:
[maclean@rh2 ~]$ ps -ef|grep ora_|grep -v grep
maclean 14264 1 0 23:16 ? 00:00:00 ora_pmon_PROD
maclean 14266 1 0 23:16 ? 00:00:00 ora_psp0_PROD
maclean 14268 1 0 23:16 ? 00:00:00 ora_mman_PROD
maclean 14270 1 0 23:16 ? 00:00:00 ora_dbw0_PROD
maclean 14272 1 0 23:16 ? 00:00:00 ora_lgwr_PROD
maclean 14274 1 0 23:16 ? 00:00:00 ora_ckpt_PROD
maclean 14276 1 0 23:16 ? 00:00:00 ora_smon_PROD
maclean 14278 1 0 23:16 ? 00:00:00 ora_reco_PROD
maclean 14338 1 0 23:16 ? 00:00:00 ora_arc0_PROD
maclean 14340 1 0 23:16 ? 00:00:00 ora_arc1_PROD
maclean 14452 1 0 23:17 ? 00:00:00 ora_s000_PROD
maclean 14454 1 0 23:17 ? 00:00:00 ora_d000_PROD
maclean 14456 1 0 23:17 ? 00:00:00 ora_cjq0_PROD
maclean 14458 1 0 23:17 ? 00:00:00 ora_qmnc_PROD
maclean 14460 1 0 23:17 ? 00:00:00 ora_mmon_PROD
maclean 14462 1 0 23:17 ? 00:00:00 ora_mmnl_PROD
maclean 14467 1 0 23:17 ? 00:00:00 ora_q000_PROD
maclean 14568 1 0 23:18 ? 00:00:00 ora_q001_PROD
[maclean@rh2 ~]$ ps -ef|grep ora_|grep -v pmon|grep -v ckpt |grep -v lgwr|grep -v smon|grep -v grep|grep -v dbw|grep -v psp|grep -v mman |grep -v rec|awk '{print $2}'|xargs kill -9
and alert log will have:
Fri Jul 30 23:20:58 EDT 2010
ARCH: Detected ARCH process failure
ARCH: Detected ARCH process failure
ARCH: STARTING ARCH PROCESSES
ARC0 started with pid=20, OS id=14959
Fri Jul 30 23:20:58 EDT 2010
ARC0: Archival started
ARC1: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
Fri Jul 30 23:20:58 EDT 2010
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
ARC1 started with pid=21, OS id=14961
ARC1: Becoming the heartbeat ARCH
Fri Jul 30 23:21:29 EDT 2010
found dead shared server 'S000', pid = (10, 3)
found dead dispatcher 'D000', pid = (11, 3)
Fri Jul 30 23:22:29 EDT 2010
Restarting dead background process CJQ0
Restarting dead background process QMNC
CJQ0 started with pid=12, OS id=15124
Fri Jul 30 23:22:29 EDT 2010
Restarting dead background process MMON
QMNC started with pid=13, OS id=15126
Fri Jul 30 23:22:29 EDT 2010
Restarting dead background process MMNL
MMON started with pid=14, OS id=15128
MMNL started with pid=16, OS id=15132
That's all right!
Leave a Reply