某个客户数据库在巡检的时候发现alert日志里不定期会出现ORA-609错误,大致内容如下:
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for HPUX: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for HPUX: Version 11.2.0.3.0 - Production
TCP/IP NT Protocol Adapter for HPUX: Version 11.2.0.3.0 - Production
Time: 19-OCT-2014 20:24:16
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (2734) as a result of ORA-609
Sun Oct 19 21:27:24 2014
***********************************************************************
由于ORA-609的缘故,ospid(xxxx)进程被aborting了,同时还伴随着TNS-12537的错误,连接关闭
去MOS搜了一圈,正好有篇文档是针对这个错误的,下面是描述:
适用于:
Oracle Net Services - Version 11.2.0.1 to 11.2.0.3 [Release 11.2]
Information in this document applies to any platform.
症状:
alert日志出现以上类似的内容(略)变化:
Changes in database server load, client connect descriptor, changes in network infrastructure (firewall configuration).
原因:
首先,这个“opiodr aborting process unknown ospid (2734) as a result of ORA-609”消息仅仅是说明了由于ORA-609,使Oracle数据库专用进程被关闭了
来看一段描述:
ORA-609 means "could not attach to incoming connection" so the database process was 'aborted' (closed) because it couldn't attach to the incoming connection passed to it by the listener.
ORA-609意味着不能通过监听把它附加到即将到来的连接上,因此服务器进程被终止(关闭)
The reason for this is found in the sqlnet error stack, in our case is:
TNS-12537: TNS:connection closed.
Basically the dedicated process didn't have a client connection anymore to work with.
客户端连接有6个步骤:
鉴于以上的错误,在第3步与第4步之间时,客户端连接就关闭了,此时当专有进程尝试与客户端连接时,发现连接已经关闭了
一个客户端连接整个步骤:
客户端发起一个connection连接监听
监听启动一个专属进程(服务器进程,也就是我们通常说的LOCA=NO进程)用于接收这个connection
在专属进程启动之后,监听会将这个connection传递给这个专属进程
专属进程通过这个connection来跟客户端握手
专属进程跟客户端信息交换需要建立一个session
session打开
To determine the client which hit this problem we can try to match the timestamp of the error from alert log with an entry in listener.log, but this might be difficult in case of a loaded listener with many incoming connections per second.
Server sqlnet trace will not provide any information about the client.
去确定碰到问题的client,我们可以尝试去匹配alert日志中错误发生的时间戳并且在监听日志中也有相应的条目,但当加载的监听每秒有许多连接的时候是非常困难去判断的,服务器sqlnet的trace不会提供任何该客户端的信息
We can enable sqlnet server trace to catch the error (the match is done based on the ospid found in sqlnet server trace file name and the line with ORA-609 error):
还可以启用sqlnet server的trace中抓取到ORA-609错误,匹配成功基于sqlnet server trace文件名和ORA-609错误信息中的ospid
nscon: doing connect handshake...
nscon: recving a packet
nsprecv: entry
nsprecv: reading from transport...
nttrd: entry
nttrd: exit
ntt2err: entry
ntt2err: Read unexpected EOF ERROR on 15 <<<<<<< error
ntt2err: exit
nsprecv: error exit
nserror: entry
nserror: nsres: id=0, op=68, ns=12537, ns2=12560; nt[0]=507, nt[1]=0, nt[2]=0; ora[0]=0, ora[1]=0, ora[2]=0
nscon: error exit
nsdo: nsctxrnk=0
nsdo: error exit
nsinh_hoff: error recving request
可能引起问原因:
Several possible situations can cause this to happen:
解决方案:
Because the entry from listener.log contains only CONNECT_DATA and CID related information we need to check the client configuration for any sqlnet timeouts:
sqlnet.outbound_connect_time
sqlnet.recv_timeout
sqlnet.send_timeout
tcp_connect_timeout
检查客户端目录中sqlnet.ora的超时设置,通常是这个引起的
possible timeout in client connect descriptor (hardcoded in client application or in client tnsnames.ora):
connect_timeout
检查客户端应用或客户单tnsnames.ora中的超时参数