查看故障时间的监听日志，如下：
…
TNS-12528: TNS:listener: all appropriate instances are blocking new connections
05-OCT-2011 14:02:34 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=33825)) * establis
h * siebeldb * 12528
TNS-12528: TNS:listener: all appropriate instances are blocking new connections
05-OCT-2011 14:02:34 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=33826)) * establis
h * siebeldb * 12528
TNS-12528: TNS:listener: all appropriate instances are blocking new connections
05-OCT-2011 14:02:34 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=46618)) * establis
h * siebeldb * 12528
05-OCT-2011 14:02:34 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=46618)) * establis
h * siebeldb * 12528
TNS-12528: TNS:listener: all appropriate instances are blocking new connections
05-OCT-2011 14:02:34 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=52618)) * establis
h * siebeldb * 12528
TNS-12528: TNS:listener: all appropriate instances are blocking new connections
05-OCT-2011 14:02:34 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=33827)) * establis
h * siebeldb * 12528
TNS-12528: TNS:listener: all appropriate instances are blocking new connections
05-OCT-2011 14:02:34 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=33828)) * establis
h * siebeldb * 12528
TNS-12528: TNS:listener: all appropriate instances are blocking new connections
05-OCT-2011 14:02:34 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=58407)) * establis
h * siebeldb * 12528
TNS-12528: TNS:listener: all appropriate instances are blocking new connections
05-OCT-2011 14:02:34 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=58408)) * establis
h * siebeldb * 12528
TNS-12528: TNS:listener: all appropriate instances are blocking new connections
05-OCT-2011 14:02:34 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=33829)) * establis
h * siebeldb * 12528
…
这部分日志说明实例在重启过程中，监听无法建立连接。
05-OCT-2011 14:03:57 * service_update * siebeldb2 * 0
05-OCT-2011 14:03:57 * service_died * siebeldb2 * 12537
05-OCT-2011 14:03:57 * service_update * siebeldb2 * 0
05-OCT-2011 14:03:57 * service_died * siebeldb2 * 12537
05-OCT-2011 14:03:57 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=56843)) * establish
* siebeldb * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
05-OCT-2011 14:03:57 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=56844)) * establish
* siebeldb * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
05-OCT-2011 14:03:57 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=56846)) * establish
* siebeldb * 12514
05-OCT-2011 14:03:57 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=56846)) * establish
* siebeldb * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
05-OCT-2011 14:03:57 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=56849)) * establish
* siebeldb * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
05-OCT-2011 14:03:57 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=56851)) * establish
* siebeldb * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
05-OCT-2011 14:03:57 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=56852)) * establish
* siebeldb * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
05-OCT-2011 14:03:57 *
….
这部分是由于监听没有对应的service注册信息，通过现场了解，由于手工启用了$ORACLE_HOME下的监听命令lsnrctl start，导致scan_listener无法完成正常的注册信息，导致监听无法建立连接。
后通过停掉手工启动的监听，重启scan_listener恢复，但是依旧存在连接超时现象。监听日志如下：
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41074)) * establish
* siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49618)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=33876)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=33877)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=35211)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=35210)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=35213)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=33879)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * service_update * siebeldb2 * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=40406)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=35215)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=40403)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41514)) * establish
* siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41510)) * establish
* siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=35218)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=33880)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=33881)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=33883)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.5)(PORT=55543)) * establish
* siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.5)(PORT=55544)) * establish
* siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.5)(PORT=55545)) * establish
* siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41517)) * establish
* siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41522)) * establish
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41517)) * establish
* siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41522)) * establish
* siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41519)) * establish
* siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41518)) * establish
* siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49669)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=33892)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41525)) * establish
* siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49673)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49671)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:38 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=33894)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.5)(PORT=55546)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49674)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41527)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.5)(PORT=55549)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.5)(PORT=55548)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41529)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41530)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41533)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41537)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49678)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49679)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49682)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49679)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49682)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49681)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49683)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.5)(PORT=46845)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.195)(PORT=49685)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.5)(PORT=46848)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.5)(PORT=46846)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.5)(PORT=46849)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41539)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.89.4)(PORT=41543)) * establish
* siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=33905)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=33901)) * establis
h * siebeldb * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(service_name=siebeldb2)(instance_name=siebeldb2)(UR=A)) * (ADDRESS=(PROTOCOL=tcp)(HOS
T=10.7.8.16)(PORT=37497)) * establish * siebeldb2 * 0
05-OCT-2011 14:40:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=siebeldb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.7.88.194)(PORT=33904)) * establis
h * siebeldb * 0
…
alert日志如下：
…
Fatal NI connect error 12560, connecting to:
(LOCAL=NO)
Fatal NI connect error 12560, connecting to:
(LOCAL=NO)
Fatal NI connect error 12560, connecting to:
(LOCAL=NO)
Fatal NI connect error 12560, connecting to:
(LOCAL=NO)
Wed Oct 05 16:07:28 2011
Time: 05-OCT-2011 16:07:27
Wed Oct 05 16:07:27 2011
Wed Oct 05 16:07:27 2011
nt OS err code: 0
Time: 05-OCT-2011 16:07:26
Wed Oct 05 16:07:28 2011
VERSION INFORMATION:
TNS for IBM/AIX RISC System/6000: Version 11.2.0.2.0 – Production
Oracle Bequeath NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.2.0 – Production
Wed Oct 05 16:07:28 2011
Fatal NI connect error 12560, connecting to:
(LOCAL=NO)
Fatal NI connect error 12560, connecting to:
(LOCAL=NO)
Wed Oct 05 16:07:27 2011
***********************************************************************Wed Oct 05 16:07:27 2011
Wed Oct 05 16:07:27 2011
Wed Oct 05 16:07:26 2011
Fatal NI connect error 12560, connecting to:
(LOCAL=NO)
…
通过alert日志中的12560错误，可以判定是由于TCP连接超时导致无法正常连接，而查看当时的监听日志发现，客户端的发起的短连接1秒非常高，大大超过20，在AIX下，监听的处理能力不超过20，当然建议保持在10以下，当并发数量太高，会导致监听无法处理，出现连接超时的现象。
10月6日9点多和11点多再次出现无法连接的故障，通过日志和现场监控来看，9点多出现的故障和10月5日相同，是由于短连接太多导致。随后，对此进行了修改，尽量使用连接池来连接。11点多和下午14：30的故障是由于CPU 100%导致。通过资源管理限制部分资源使用不合理的SQL，CPU使用恢复正常。
建议：
1、修改应用短连接为长连接，避免并发连接太高，尽量保持在10以下
2、不使用Scan地址，直接使用vip。通过下面的命令增加listener：
srvctl add listener -l LISTENER2 -k 1 -p 1522
srvctl add listener -l LISTENER3 -k 1 -p 1523
同时对local_listener作相应的修改。
alter system set local_listener='(DESCRIPTION=(ADDRESS_LIST=
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.7.88.10) (PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.7.88.10) (PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.7.88.10) (PORT = 1523))))’ sid=’siebeldb2′;
Leave a Reply