在AIX平台安装Oracle 11gR2 RAC每次总是要出点问题,果然在前几次问题的基础上又出现新的问题,在这里做相应的记录。
>>操作系统版本:
6100-07-03-1207
>>数据库版本:11.2.0.3.4
首先成功在AIX平台安装了11.2.0.3.0版本的Grid Infrastructure软件,重启grid软件也没有任何问题;当参考GRID 11.2.0.3.4补丁中README.html文件的内容,成功完成11.2.0.3.4补丁安装之后,grid软件无法正常的启动。在Clusterware的告警日志中看到如下内容:
2014-07-09 18:34:24.252
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/oraagent'. Details at (:CRSAGF00123:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.262
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/oraagent'. Details at (:CRSAGF00126:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.283
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/orarootagent'. Details at (:CRSAGF00123:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.284
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/orarootagent'. Details at (:CRSAGF00126:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.303
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/orarootagent'. Details at (:CRSAGF00123:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.303
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/orarootagent'. Details at (:CRSAGF00126:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.322
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/orarootagent'. Details at (:CRSAGF00123:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.323
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/orarootagent'. Details at (:CRSAGF00126:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.356
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/cssdagent'. Details at (:CRSAGF00123:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.357
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/cssdagent'. Details at (:CRSAGF00126:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.376
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/cssdmonitor'. Details at (:CRSAGF00123:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.377
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/cssdmonitor'. Details at (:CRSAGF00126:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.396
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/orarootagent'. Details at (:CRSAGF00123:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.396
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/orarootagent'. Details at (:CRSAGF00126:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.415
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/orarootagent'. Details at (:CRSAGF00123:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.416
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/orarootagent'. Details at (:CRSAGF00126:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.434
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/orarootagent'. Details at (:CRSAGF00123:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
2014-07-09 18:34:24.435
[ohasd(5898404)]CRS-5828:Could not start agent '/u01/app/11.2.0/grid/bin/orarootagent'. Details at (:CRSAGF00126:) {0:0:2} in /u01/app/11.2.0/grid/log/edoracle1/ohasd/ohasd.log.
......
cssdagent、cssdmonitor、orarootagent、oraagent 4个代理进程都无法启动。
查看ohasd.log文件,发现启动过程首先出现如下的报错:
......
2014-07-09 19:22:58.096: [ A**][5912] {0:0:2} Created alert : (:CRSAGF00123:) :
Failed to start the agent
process: /u01/app/11.2.0/grid/bin/orarootagent Category: -2 Operation: waitpid Loc: usrgetgrp12 OS error: 0
Other : child returned 232
2014-07-09 19:22:58.096: [ A**][5912] {0:0:2} Created alert : (:CRSAGF00126:) : Agent start failed
......
根据这个报错在MOS中找到了如下一篇文章:
Unable to start OHASD after apply of PSU patch. CLSU-00103: error location: usrgetgrp12 (文档 ID 1562797.1)
修改时间:
2013-7-3
类型:
PROBLEM
In this Document
Symptoms |
Cause |
Solution |
Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.
Unable to start OHASD after apply of PSU patch.
CLSU-00100: Operating System function: waitpid failed with error data: 0
CLSU-00101: Operating System error message: Error 0
CLSU-00103: error location: usrgetgrp12
CLSU-00104: additional error information: child returned 232
CRS-4000: Command Start failed, or completed with errors.
Timed out waiting for ohasd to start.
Then ohasd start timed out after a while ,
2013-06-13 12:29:45: Checking the status of ohasd
2013-06-13 12:29:50: Executing cmd: /apps/grid/11.2.0/grid_1/bin/crsctl check has
2013-06-13 12:29:51: Checking the status of ohasd
2013-06-13 12:29:56: Executing cmd: /apps/grid/11.2.0/grid_1/bin/crsctl check has
2013-06-13 12:29:58: Checking the status of ohasd
2013-06-13 12:30:03: Timed out waiting for ohasd to start.
Check for files permission under folder ./files/bin/*.bin
Permission should be like below.
$ ls -ltr crsctl.bin
-rwxr-xr-x 1 <oracle os user binary owner> <oracle os user group owner> 16656282 Jul 31 2012 crsctl.bin
For this Customer, the permissions are below:
$ ls -ltr crsctl.bin
-rw-r--r-- 1 <oracle os user binary owner> <oracle os user group owner> 16656282 Jul 31 2012 crsctl.bin
Execute permissions were mising.
Probably you did unzip the patch as root user.
rootcrs.pl -unlock and rootcrs.pl -patch will not help as it only modifies those binaries owned by root user, not grid/oracle user.
A.
Modify the permission for all binaries under $GRID_HOME/bin
chmod 755 $GRID_HOME/bin/<binaries>
Start the HAS and check the behavior.
If solution A fails, try solution B
B.
1.Rollback the patch and lock the home .
2.Make sure HAS is started.
3.Download the patch, if possible extract to local folder as grid user
4.Check the permission again after the extraction
5.Unlock the home and apply the patch.
非常遗憾的是按照文章提供的解决方案,将$GRID_HOME/bin/*.bin的权限修改为755不能成功启动grid的相关服务,之后我对比11.2.0.3.0版本相应文件的权限,
将11.2.0.3.4版本下
*.bin文件的权限修改成和11.2.0.3.0版本相同
,依然无法启动。
但是从以上文章不难看出引起这个问题有两个要点:
1).需要使用grid用户来解压补丁,而不是用root用户,之后再修改解压后文件的权限。
2).出现这个问题是因为$GRID_HOME/bin目录下文件的权限问题导致的。
由于我在应用11.2.0.3.4补丁之前,将11.2.0.3.0软件做了一个tar备份,所以我很容易的将grid软件还原到了11.2.0.3.0的版本,在这个版本grid软件就很快速的启动成功。
之后我按照文档的要求使用grid用户来解压11.2.0.3.4的补丁,参考README.html的内容重新应用了补丁,依然不能启动grid服务,同样是因为不能启动任何agent进程。
但我坚信一点,这是由于$GRID_HOME/bin目录下文件的权限导致的。于是乎,我将对比两个版本的
$GRID_HOME/bin/*.bin文件扩展到对比两个版本$GRID_HOME/bin目录下的所有文件
;参考11.2.0.3.0版本文件的权限,我在11.2.0.3.4版本的$GRID_HOME/bin目录下执行了以下命令:
chmod 741 crsd.bin
chmod 741 cssdagent.bin
chmod 741 cssdmonitor.bin
chmod 741 diskmon.bin
chmod 741 gnsd.bin
chmod 751 oclskd.bin
chmod 750 oclumon.bin
chmod 741 octssd.bin
chmod 741 ohasd.bin
chmod 741 orarootagent.bin
chmod 750 osysmond.bin
#
单纯修改以上*.bin文件的权限依然无法启动grid服务。
chmod 755 adapters
chmod 750 appvipcfg.pl
chmod 755 asmcmdcore
chmod 755 asmproxy
chmod 755 crstmpl.scr
chmod 755 evt.sh
chmod 755 gennttab
chmod 755 gsd
chmod 750 nidO
chmod 755 oc4jctl.pl
chmod 755 oprocd
chmod 750 oracleO
chmod 755 plshprofO
chmod 755 racgeut
chmod 755 racgevtf
chmod 755 racgmain
chmod 755 racgvip
chmod 750 renamedg0
chmod 755 sclsspawn
chmod 755 tnslsnr0
chmod 750 wrapO
执行完上面的修改后,grid服务即能启动成功,说明之前的判断是正确的。
通过这样一个案例,我们能总结出这样一个经验:安装文件的权限和所有者同样
影响着软件的正常运行,我们不能的简单暴力的执行chmod -R 777等操作来修复这样的问题,我们也不能轻易的修改安装文件的权限,更不允许有这样的误操作,最好的办法是安装完成后对安装文件做一次tar备份。当然我们还应该养成好的习惯,安装在什么用户下的软件就用什么用户来解压。
另外打补丁的时候还有需要注意以下两点:
1).执行
<ORACLE_HOME>/OPatch/ocm/bin/emocmrsp -output <filename
>
生成一个OCM文件。
2).执行升级以前确保安装GRID软件所在文件系统
有25G左右的剩余空间,否则升级不会开始(升级过程需要这么大的临时空间,例如,将grid infrastructure从11.2.0.3.0升级到11.2.0.3.4
),参考升级命令如下:
./opatch auto /u01/soft/11.2.0.3.4 -oh /u01/app/11.2.0/grid -ocmrf /home/grid/ocm.rsp
使用root用户执行opatch命令。
最后我贴出一个正常工作的GRID软件$GRID_HOME/bin目录下所有文件的权限的例子,方便做出对比:
-rwxr-xr-x 1 grid oinstall 9172 Jun 08 2010 AgentLifeCycle.pm
-rwxr-xr-x 1 grid oinstall 2343 Jul 26 2006 AgentMisc.pm
-rwxr-xr-x 1 grid oinstall 17833 Sep 13 2007 AgentStatus.pm
-rwxr-xr-x 1 grid oinstall 3265 Jul 31 2008 AgentSubAgent.pm
-rwxr-xr-x 1 grid oinstall 2923 May 25 2007 CompEMagent.pm
-rwxr-xr-x 1 grid oinstall 5693 Jan 19 2008 CompEMcentral.pm
-rwxr-xr-x 1 grid oinstall 5199 Jan 19 2008 CompEMcore.pm
-rwxr-xr-x 1 grid oinstall 2952 Oct 17 2005 Directory.pm
-rwxr-xr-x 1 grid oinstall 24310 Sep 12 2008 EMAgent.pm
-rwxr-xr-x 1 grid oinstall 1404 Oct 17 2005 EMAgentPatch.pm
-rwxr-xr-x 1 grid oinstall 24550 Mar 24 2009 EMDeploy.pm
-rwxr-xr-x 1 grid oinstall 3451 May 01 2006 EMDiag.pm
-rwxr-xr-x 1 grid oinstall 5653 Sep 06 2006 EMSAConsoleCommon.pm
-rwxr-xr-x 1 grid oinstall 6612 Dec 14 2006 EMconnectorCmds.pm
-rwxr-xr-x 1 grid oinstall 16758 Mar 18 2009 EMomsCmds.pm
-rwxr-xr-x 1 grid oinstall 2194 Oct 17 2005 EmCommonCmdDriver.pm
-rwxr-xr-x 1 grid oinstall 6741 Oct 03 2006 EmKeyCmds.pm
-rwxr-xr-x 1 grid oinstall 44107 Jul 09 16:26 EmctlCommon.pm
-rwxr-xr-x 1 grid oinstall 16979 Jul 27 2007 LaunchEMagent.pm
-rwxr-xr-x 1 grid oinstall 1488 Oct 17 2005 Path.pm
-rwxr-xr-x 1 grid oinstall 4442 Jan 19 2008 RegisterTType.pm
-rwxr-xr-x 1 grid oinstall 28900 Jul 31 2008 SecureAgentCmds.pm
-rwxr-xr-x 1 grid oinstall 70807 Jun 23 2009 SecureOMSCmds.pm
-rwxr-xr-x 1 grid oinstall 34604 May 23 2008 SecureUtil.pm
-rwxr-x--- 1 grid oinstall 281 Oct 17 2005 StartAgent.pl
-rwxr-xr-x 1 root oinstall 1474 Jul 09 16:56 acfsdriverstate
-rwxr-xr-x 1 root oinstall 1111 Jul 09 16:56 acfsload
-rwxr-xr-x 1 root oinstall 1192 Jul 09 16:56 acfsregistrymount
-rwxr-xr-x 1 root oinstall 1567 Aug 10 2011 acfsroot
-rwxr-xr-x 1 root oinstall 1204 Jul 09 16:56 acfssinglefsmount
-rwxr-xr-x 1 grid oinstall 13137 Aug 23 2011 adapters
-rwxr-x--x 1 grid oinstall 74309 Jul 09 16:26 adrci
-rwxr-x--x 1 grid oinstall 74309 Jul 09 16:25 adrciO
-rwxr-x--x 1 grid oinstall 48179916 Jul 09 16:25 agtctl
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 agtctlO
-rwxr-x--x 1 grid oinstall 49776188 Jul 09 16:25 amdu
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 amduO
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 appagent
-rwxr-xr-x 1 grid oinstall 300348794 Oct 05 2011 appagent.bin
-rwxr-xr-x 1 root oinstall 520 Jul 09 16:52 appvipcfg
-rwxr-x--- 1 grid oinstall 9051 Oct 13 2010 appvipcfg.pl
-rwxr-x--x 1 grid oinstall 1521 Jul 09 16:24 aqxmlctl
-rw-r--r-- 1 grid oinstall 9069 Oct 05 2008 aqxmlctl.pl
-rwxr-xr-x 1 grid oinstall 6557 Jul 09 16:24 asmca
-rwxr-x--- 1 grid oinstall 5469 Apr 21 2010 asmcmd
-rwxr-xr-x 1 grid oinstall 34290 Feb 11 2011 asmcmdcore
-rwxr-xr-x 1 grid oinstall 2966 Jan 14 2010 asmproxy
-rwxr-xr-x 1 grid oinstall 2407 Jul 09 16:24 bndlchk
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 cemutlo
-rwxr-xr-x 1 grid oinstall 14615 Oct 05 2011 cemutlo.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 cemutls
-rwxr-xr-x 1 grid oinstall 22773 Oct 05 2011 cemutls.bin
-rwxr-xr-x 1 root oinstall 8138 Jul 09 16:52 clscfg
-rwxr-xr-x 1 root oinstall 150904 Oct 05 2011 clscfg.bin
-rwxr-xr-x 1 root oinstall 8138 Jul 09 16:52 clsecho
-rwxr-xr-x 1 root oinstall 74522 Oct 05 2011 clsecho.bin
-rwxr-xr-x 1 root oinstall 8138 Jul 09 16:52 clsfmt
-rwxr-xr-x 1 root oinstall 52950 Oct 05 2011 clsfmt.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 clsid
-rwxr-xr-x 1 grid oinstall 21619 Oct 05 2011 clsid.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 clssproxy
-rwxr-xr-x 1 grid oinstall 107282 Oct 05 2011 clssproxy.bin
-rwxr-xr-x 1 grid oinstall 3572 Jul 09 16:52 cluutil
-rwxr-xr-x 1 grid oinstall 8944 Jul 09 16:24 cluvfy
-rwxr-xr-x 1 grid oinstall 1303 Feb 03 2010 cluvfyrac.sh
-rwxr-xr-x 1 grid oinstall 489 Jul 09 16:26 commonenv
-rwxr-xr-x 1 grid oinstall 495 Apr 14 2006 commonenv.template
-rwxr-xr-x 1 grid oinstall 5778 Jan 01 2000 coraenv
-rwxr-x--- 1 root oinstall 695 Jul 09 16:52 crfsetenv
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 crs_getperm
-rwxr-xr-x 1 grid oinstall 268313951 Oct 05 2011 crs_getperm.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 crs_profile
-rwxr-xr-x 1 grid oinstall 268511591 Oct 05 2011 crs_profile.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 crs_register
-rwxr-xr-x 1 grid oinstall 268512569 Oct 05 2011 crs_register.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 crs_relocate
-rwxr-xr-x 1 grid oinstall 268506914 Oct 05 2011 crs_relocate.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 crs_setperm
-rwxr-xr-x 1 grid oinstall 268300143 Oct 05 2011 crs_setperm.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 crs_start
-rwxr-xr-x 1 grid oinstall 268507226 Oct 05 2011 crs_start.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 crs_stat
-rwxr-xr-x 1 grid oinstall 268513475 Oct 05 2011 crs_stat.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 crs_stop
-rwxr-xr-x 1 grid oinstall 268507052 Oct 05 2011 crs_stop.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 crs_unregister
-rwxr-xr-x 1 grid oinstall 268496980 Oct 05 2011 crs_unregister.bin
-rwxr-xr-x 1 root oinstall 8138 Jul 09 16:52 crsctl
-rwxr-xr-x 1 root oinstall 320087302 Oct 05 2011 crsctl.bin
-rwxr----x 1 root oinstall 8138 Jul 09 16:52 crsd
-rwxr----x 1 root oinstall 263994691 Oct 05 2011 crsd.bin
-rwxr-xr-x 1 grid oinstall 2823 Mar 24 2010 crsdiag.pl
-rwxr-xr-x 1 grid oinstall 13438 Sep 08 2005 crstmpl.scr
-rwxr--r-- 1 root oinstall 6189 Jul 09 16:52 crswrapexece.pl
-rwxr-xr-x 1 grid oinstall 48564583 Aug 10 2011 csscan
-rwxr----x 1 root oinstall 300176328 Oct 05 2011 cssdagent
-rwxr----x 1 root oinstall 0 Jul 09 16:52 cssdagent.bin
-rwxr----x 1 root oinstall 300176328 Oct 05 2011 cssdmonitor
-rwxr----x 1 root oinstall 0 Jul 09 16:52 cssdmonitor.bin
-rwxr-xr-x 1 root oinstall 8138 Jul 09 16:52 cssvfupgd
-rwxr-xr-x 1 root oinstall 896774 Oct 05 2011 cssvfupgd.bin
-rwxr-x--x 1 grid oinstall 48151146 Jul 09 16:25 cursize
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 cursizeO
-rwxr-xr-x 1 grid oinstall 2794 Jul 09 16:52 cvures
-rwxr-x--- 1 grid oinstall 2209602 Oct 21 2011 dbfs_client
-rwxr-x--x 1 grid oinstall 30892 Jul 09 16:25 dbfsize
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 dbfsizeO
-rw-r--r-- 1 grid oinstall 5297 Oct 03 2006 dbgeu_run_action.pl
-rwxr-xr-x 1 grid oinstall 2415 Jan 01 2000 dbhome
-rwxr-x--- 1 grid oinstall 6030 Jan 01 2000 dbshut
-rwxr-x--- 1 grid oinstall 13797 Jan 01 2000 dbstart
-rwxr-x--x 1 grid oinstall 48214563 Jul 09 16:25 dbv
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 dbvO
-rwxr-xr-x 1 grid oinstall 1680 Jul 09 16:24 deploync
-rwxr-x--x 1 grid oinstall 31736 Jul 09 16:26 dg4pwd
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 dg4pwdO
-rwxr-x--x 1 grid oinstall 48674826 Jul 09 16:26 dgmgrl
-rwxr-x--x 1 grid oinstall 48674826 Jul 09 16:25 dgmgrlO
-rwxr-xr-x 1 root oinstall 38807 May 03 2011 diagcollection.pl
-rwxr-xr-x 1 root oinstall 499 Jul 09 16:52 diagcollection.sh
-rwxr-xr-x 1 grid oinstall 7415 Jul 09 16:24 diagsetup
-rwxr----x 1 root oinstall 8138 Jul 09 16:52 diskmon
-rwxr----x 1 root oinstall 65464762 Oct 21 2011 diskmon.bin
-rwxr-xr-x 1 grid oinstall 1448 Jul 09 16:24 dropjava
-rwxr-xr-x 1 grid oinstall 48219415 Jul 09 16:26 dsml2ldif
-rwxr-x--x 1 grid oinstall 948674 Jul 09 16:25 dumpsga
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 dumpsga0
-rwxr-xr-x 1 grid oinstall 153 Nov 06 1997 echodo
-rwxr-xr-x 1 grid oinstall 3501 Jul 09 16:26 emca
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 emcrsp
-rwxr-xr-x 1 grid oinstall 269311259 Oct 05 2011 emcrsp.bin
-rwxr--r-- 1 grid oinstall 15483 Jul 09 16:26 emctl
-rwxr-xr-x 1 grid oinstall 18340 Jul 09 16:26 emctl.pl
-rwxr-xr-x 1 grid oinstall 15615 Mar 25 2010 emctl.template
-rwxr-xr-x 1 grid oinstall 1226 Oct 17 2005 emdfail.command
-rwxr-xr-x 1 grid oinstall 12557 Jul 09 16:26 emutil
-rwxr-xr-x 1 grid oinstall 10174 Jan 19 2008 emutil.bat.template
-rwxr-xr-x 1 grid oinstall 39794 Aug 05 2011 emwd.pl
-rwxr-xr-x 1 grid oinstall 1590 Jul 09 16:24 eusm
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 evmd
-rwxr-xr-x 1 grid oinstall 33079605 Oct 05 2011 evmd.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 evminfo
-rwxr-xr-x 1 grid oinstall 26654469 Oct 05 2011 evminfo.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 evmlogger
-rwxr-xr-x 1 grid oinstall 27121882 Oct 05 2011 evmlogger.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 evmmkbin
-rwxr-xr-x 1 grid oinstall 26647538 Oct 05 2011 evmmkbin.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 evmmklib
-rwxr-xr-x 1 grid oinstall 26664215 Oct 05 2011 evmmklib.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 evmpost
-rwxr-xr-x 1 grid oinstall 26680802 Oct 05 2011 evmpost.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 evmshow
-rwxr-xr-x 1 grid oinstall 26660431 Oct 05 2011 evmshow.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 evmsort
-rwxr-xr-x 1 grid oinstall 26685099 Oct 05 2011 evmsort.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 evmwatch
-rwxr-xr-x 1 grid oinstall 26691604 Oct 05 2011 evmwatch.bin
-rwxr-xr-x 1 grid oinstall 4109 Oct 17 2002 evt.sh
-rwxr-x--x 1 grid oinstall 48996507 Jul 09 16:26 exp
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 expO
-rwxr-x--x 1 grid oinstall 48314995 Jul 09 16:26 expdp
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 expdpO
-rwsr-x--- 1 root oinstall 849343 Jul 09 16:25 extjob
-rwxr-xr-x 1 grid oinstall 849182 Oct 21 2011 extjobO
-rwx------ 1 grid oinstall 849343 Jul 09 16:25 extjobo
-rwxr-xr-x 1 grid oinstall 849182 Oct 21 2011 extjoboO
-rwxr-x--x 1 grid oinstall 1742313 Jul 09 16:26 extproc
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 extprocO
-rwxr-x--- 1 grid oinstall 291 Jul 09 16:24 extusrupgrade
-rwxr-x--- 1 grid oinstall 898415 Oct 21 2011 fmputl
-rwxr-x--- 1 grid oinstall 16520 Oct 21 2011 fmputlhp
-rwxr-xr-x 1 grid oinstall 8445 Aug 26 2010 genagtsh
-rwxr-xr-x 1 grid oinstall 9803 Oct 03 2011 genclntsh
-rwxr-xr-x 1 grid oinstall 4437 Aug 03 2010 genclntst
-rwxr-x--x 1 grid oinstall 51629 Jul 09 16:26 genezi
-rwxr-xr-x 1 grid oinstall 51121 Oct 21 2011 geneziO
-rwxr-xr-x 1 grid oinstall 667254 Oct 21 2011 genksms
-rwxr-xr-x 1 grid oinstall 3813 Aug 23 2011 gennfgt
-rwxr-xr-x 1 grid oinstall 6614 Aug 23 2011 gennttab
-rwxr-xr-x 1 grid oinstall 3512 Aug 03 2010 genoccish
-rwxr-xr-x 1 grid oinstall 10227 Aug 03 2010 genorasdksh
-rwxr-xr-x 1 grid oinstall 2487 Jan 01 2000 gensyslib
-rwxr-xr-x 1 grid oinstall 8089 Aug 03 2010 genttsh
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 gipcd
-rwxr-xr-x 1 grid oinstall 841932 Oct 05 2011 gipcd.bin
-rwxr----x 1 root oinstall 8138 Jul 09 16:52 gnsd
-rwxr----x 1 root oinstall 19113 Oct 05 2011 gnsd.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 gpnpd
-rwxr-xr-x 1 grid oinstall 374640 Oct 05 2011 gpnpd.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 gpnptool
-rwxr-xr-x 1 grid oinstall 500960 Oct 05 2011 gpnptool.bin
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 grdcscan
-rwxr-xr-x 1 grid oinstall 20677 Oct 05 2011 gsd
-rwxr-xr-x 1 grid oinstall 6391 Jul 09 16:52 gsd.sh
-rwxr-xr-x 1 grid oinstall 5748 Jul 09 16:52 gsdctl
-rwxr-x--x 1 grid oinstall 329919 Jul 09 16:25 hsalloci
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 hsallociO
-rwxr-x--x 1 grid oinstall 29263 Jul 09 16:25 hsdepxa
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 hsdepxaO
-rwxr-x--x 1 grid oinstall 27619 Jul 09 16:25 hsots
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 hsotsO
-rwxr-x--x 1 grid oinstall 48544028 Jul 09 16:26 imp
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 impO
-rwxr-x--x 1 grid oinstall 48323451 Jul 09 16:26 impdp
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 impdpO
-rwsr-x--- 1 root oinstall 28644 Jul 09 16:25 jssu
-rwxr-x--x 1 grid oinstall 49726799 Jul 09 16:25 kfed
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 kfedO
-rwxr-x--x 1 grid oinstall 65318575 Jul 09 16:25 kfod
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 kfodO
-rwxr-x--x 1 grid oinstall 48187413 Jul 09 16:26 kgmgr
-rwxr-x--x 1 grid oinstall 48187413 Jul 09 16:25 kgmgrO
lrwxrwxrwx 1 grid oinstall 42 Jul 10 10:22 lbuilder -> /u01/app/11.2.0/grid/nls/lbuilder/lbuilder
-rwxr-xr-x 1 grid oinstall 1364932 Aug 04 2011 lcsscan
-rwxr-xr-x 1 grid oinstall 48258970 Jul 09 16:26 ldapadd
-rwxr-xr-x 1 grid oinstall 48273294 Jul 09 16:26 ldapaddmt
-rwxr-xr-x 1 grid oinstall 48170919 Jul 09 16:26 ldapbind
-rwxr-xr-x 1 grid oinstall 48172425 Jul 09 16:26 ldapcompare
-rwxr-xr-x 1 grid oinstall 48174110 Jul 09 16:26 ldapdelete
-rwxr-xr-x 1 grid oinstall 48171390 Jul 09 16:26 ldapmoddn
-rwxr-xr-x 1 grid oinstall 48258970 Jul 09 16:26 ldapmodify
-rwxr-xr-x 1 grid oinstall 48273294 Jul 09 16:26 ldapmodifymt
-rwxr-xr-x 1 grid oinstall 48211991 Jul 09 16:26 ldapsearch
-rwxr-xr-x 1 grid oinstall 3230 Jul 09 16:24 ldifmigrator
-rwxr-xr-x 1 grid oinstall 1459 May 12 2009 linkshlib
-rwxr-xr-x 1 grid oinstall 1105503 Aug 04 2011 lmsgen
-rwxr-xr-x 1 grid oinstall 1666 Jul 09 16:24 loadjava
-rwxr-x--x 1 grid oinstall 48169623 Jul 09 16:26 loadpsp
-rwxr-x--x 1 grid oinstall 48169623 Jul 09 16:25 loadpspO
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 lsdb
-rwxr-xr-x 1 grid oinstall 30448 Oct 05 2011 lsdb.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 lsnodes
-rwxr-xr-x 1 grid oinstall 28268 Oct 05 2011 lsnodes.bin
-rwxr-x--x 1 grid oinstall 48283224 Jul 09 16:26 lsnrctl
-rwxr-xr-x 1 grid oinstall 0 Oct 21 2011 lsnrctl0
-rwxr-xr-x 1 grid oinstall 726678 Aug 04 2011 lxchknlb
-rwxr-xr-x 1 grid oinstall 1162698 Aug 04 2011 lxegen
-rwxr-xr-x 1 grid oinstall 4447596 Aug 04 2011 lxinst
-rwxr-x--x 1 grid oinstall 947159 Jul 09 16:25 mapsga
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 mapsga0
-rwxr-x--x 1 grid oinstall 10999 Jul 09 16:25 maxmem
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 maxmemO
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 mdnsd
-rwxr-xr-x 1 grid oinstall 1049457 Oct 05 2011 mdnsd.bin
-rwxr-x--x 1 grid oinstall 112982 Jul 09 16:25 mkpatch
-rwxr-xr-x 1 grid oinstall 112693 Oct 21 2011 mkpatchO
-rwxr-xr-x 1 grid oinstall 2786 Jul 09 16:24 mkstore
-rwxr-xr-x 1 grid oinstall 1550 Jul 09 16:24 ncomp
-rwxr-x--- 1 grid oinstall 3792 Jul 09 16:52 ndfnceca
-rwxr-xr-x 1 grid oinstall 6435 Jul 09 16:24 netca
-rw-r--r-- 1 grid oinstall 95 Jul 09 16:24 netca_deinst.sh
-rwxr-xr-x 1 grid oinstall 6323 Jul 09 16:24 netmgr
-rwxr-x--- 1 grid oinstall 2835 Sep 17 2007 nfsPatchPlugin.pm
-rwxr-x--x 1 grid oinstall 48230638 Jul 09 16:25 nid
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 nidO
-rwxr-xr-x 1 grid oinstall 20236 Jul 09 16:52 oc4jctl
-rwxr-xr-x 1 grid oinstall 2933 Jul 19 2010 oc4jctl.pl
-rwxr-xr-x 1 root oinstall 8138 Jul 09 16:52 oclskd
-rwxr-x--x 1 root oinstall 16352 Oct 05 2011 oclskd.bin
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 oclsomon
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 oclsvmon
-rwxr-x--- 1 root oinstall 1556 Jul 09 16:52 oclumon
-rwxr-x--- 1 root oinstall 22700 Oct 05 2011 oclumon.bin
-rwxr-x--- 1 grid oinstall 2025 Jul 09 16:52 oclumon.pl
-rwxr-xr-x 1 root oinstall 8138 Jul 09 16:52 ocrcheck
-rwxr-xr-x 1 root oinstall 156048 Oct 05 2011 ocrcheck.bin
-rwxr-xr-x 1 root oinstall 8138 Jul 09 16:52 ocrconfig
-rwxr-xr-x 1 root oinstall 295991 Oct 05 2011 ocrconfig.bin
-rwxr-xr-x 1 root oinstall 8138 Jul 09 16:52 ocrdump
-rwxr-xr-x 1 root oinstall 166169 Oct 05 2011 ocrdump.bin
-rwxr-xr-x 1 root oinstall 8138 Jul 09 16:52 ocrpatch
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 ocssd
-rwxr-xr-x 1 grid oinstall 3442595 Oct 05 2011 ocssd.bin
-rwxr----x 1 root oinstall 8138 Jul 09 16:52 octssd
-rwxr----x 1 root oinstall 767960 Oct 05 2011 octssd.bin
-rwxr-xr-x 1 grid oinstall 1280 Jul 09 16:24 odisrvreg
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 odnsd
-rwxr-xr-x 1 grid oinstall 84021 Oct 05 2011 odnsd.bin
-rwxr-xr-x 1 grid oinstall 2365 Jan 01 2000 oerr
-rwxr-xr-x 1 grid oinstall 28012 Oct 20 2011 offset
-rwxr----x 1 root oinstall 8138 Jul 09 16:52 ohasd
-rwxr----x 1 root oinstall 268525199 Oct 05 2011 ohasd.bin
-rwxr-xr-x 1 grid oinstall 2134 Jul 09 16:24 oidca
-rwxr-xr-x 1 grid oinstall 3548 Jul 09 16:24 oidprovtool
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 oifcfg
-rwxr-xr-x 1 grid oinstall 180957 Oct 05 2011 oifcfg.bin
-rwxr-xr-x 1 grid oinstall 1377 Jul 09 16:24 ojvmjava
-rwxr-xr-x 1 grid oinstall 1260 Jul 09 16:24 ojvmtc
-rwxr-x--- 1 grid oinstall 868 Jul 09 16:52 ologdbg
-rwxr-x--- 1 grid oinstall 4265 Jul 09 16:52 ologdbg.pl
-rwxr-x--- 1 root oinstall 35422 Oct 05 2011 ologgerd
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 olsnodes
-rwxr-xr-x 1 grid oinstall 59102 Oct 05 2011 olsnodes.bin
-rwxr-xr-x 1 grid oinstall 1074 Jul 09 16:52 onsctl
-rwxr-xr-x 1 grid oinstall 87066 Oct 05 2011 oprocd
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 oraagent
-rwxr-xr-x 1 grid oinstall 300363384 Oct 05 2011 oraagent.bin
-rwxr-xr-x 1 grid oinstall 3271507 Oct 21 2011 orabase
-rwsr-s--x 1 grid oinstall 264670918 Jul 09 16:26 oracle
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 oracleO
-rw-r--r-- 1 root system 8138 Jul 09 16:52 oradaemonagent
-rwsr-x--- 1 root oinstall 28229 Oct 21 2011 oradism
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 oradnssd
-rwxr-xr-x 1 grid oinstall 123832 Oct 05 2011 oradnssd.bin
-rwxr-xr-x 1 grid oinstall 6183 Jan 01 2000 oraenv
-rwxr-xr-x 1 grid oinstall 44 Dec 04 2002 orajaxb
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 oranetmonitor
-rwxr-xr-x 1 grid oinstall 908496 Oct 05 2011 oranetmonitor.bin
-rwxr-xr-x 1 grid oinstall 3170 Jul 09 16:24 orapki
-rwxr-x--x 1 grid oinstall 1097773 Jul 09 16:25 orapwd
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 orapwdO
-rwxr----x 1 root oinstall 8138 Jul 09 16:52 orarootagent
-rwxr----x 1 root oinstall 300213265 Oct 05 2011 orarootagent.bin
-rwxr-xr-x 1 grid oinstall 6357 Oct 04 2011 oraxlc
-rwxr-xr-x 1 grid oinstall 48 Sep 25 2000 oraxml
-rwxr-xr-x 1 grid oinstall 48 Sep 25 2000 oraxsl
-rwxr-x--x 1 grid oinstall 8306084 Jul 09 16:26 orion
-rwxr-xr-x 1 grid oinstall 8305199 Oct 21 2011 orionO
-rwxr-xr-x 1 grid oinstall 9437 Jul 09 16:26 osdbagrp
-rwxr-xr-x 1 grid oinstall 9250 Oct 05 2011 osdbagrp0
-rwxr-x--- 1 grid oinstall 11390 Oct 21 2011 osh
-rwxr-x--- 1 root oinstall 8138 Jul 09 16:52 osysmond
-rwxr-x--- 1 root oinstall 35354 Oct 05 2011 osysmond.bin
-rwxr-xr-x 1 grid oinstall 6794 Jul 09 16:24 owm
-rwxr-x--- 1 grid oinstall 4472 Apr 14 2008 patchAgtStPlugin.pm
-rw-r--r-- 1 grid oinstall 3135 Jul 09 16:24 platform_common
-rwxr-x--x 1 grid oinstall 48261332 Jul 09 16:26 plshprof
-rwxr-xr-x 1 grid oinstall 0 Oct 21 2011 plshprofO
-rwxr-xr-x 1 grid oinstall 7326 Jul 09 16:52 qosctl
-rwxr-xr-x 1 grid oinstall 1725030 Oct 05 2011 racgeut
-rwxr-xr-x 1 grid oinstall 1715689 Oct 05 2011 racgevtf
-rwxr-xr-x 1 grid oinstall 1992400 Oct 05 2011 racgmain
-rwxr-xr-x 1 grid oinstall 16020 Oct 05 2011 racgvip
-rwxr-xr-x 1 grid oinstall 1594 Jul 09 16:52 racgwrap
-rwxr-xr-x 1 grid oinstall 1498 Oct 30 2009 racgwrap.sbs
-rwxr-xr-x 1 grid oinstall 11202 Jul 09 16:26 rawutl
-rwxr-xr-x 1 grid oinstall 0 Oct 21 2011 rawutl0
-rwxr-xr-x 1 grid oinstall 7024 Jul 09 16:24 rconfig
-rwxr-xr-x 1 grid oinstall 3879 Jul 09 16:52 rdtool
-rwxr-xr-x 1 grid oinstall 5770 Nov 21 2010 relink
-rwxr-x--x 1 grid oinstall 50071336 Jul 09 16:26 renamedg
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 renamedg0
-rwxr-x--x 1 grid oinstall 61112430 Jul 09 16:26 rman
-rwxr-x--x 1 grid oinstall 61112430 Jul 09 16:26 rmanO
-rwxr-xr-x 1 grid oinstall 2552 Apr 29 2008 sAgentUtils.pm
-rwxr-x--x 1 grid oinstall 1327487 Jul 09 16:25 sbttest
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 sbttestO
-rwxr-xr-x 1 grid oinstall 3163197 Aug 25 2011 schema
-rwxr-xr-x 1 grid oinstall 2835 Jul 09 16:24 schemasync
-rwxr-xr-x 1 grid oinstall 14179 Oct 05 2011 sclsspawn
-rwxr-x--- 1 grid oinstall 6923 Jul 09 16:52 scrctl
-rwxr-xr-x 1 grid oinstall 8138 Jul 09 16:52 scriptagent
-rwxr-xr-x 1 grid oinstall 300156815 Oct 05 2011 scriptagent.bin
-rwxr-x--x 1 grid oinstall 1595098 Jul 09 16:26 setasmgid
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 setasmgid0
-rwxr-xr-x 1 grid oinstall 2324 Jul 09 16:52 setasmgidwrap
-rwxr-x--x 1 grid oinstall 10569 Jul 09 16:26 skgxpinfo
-rwxr-xr-x 1 grid oinstall 0 Oct 21 2011 skgxpinfoO
-rwxr-x--x 1 grid oinstall 49628596 Jul 09 16:26 sqlldr
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 sqlldrO
-rwxr-x--x 1 grid oinstall 49650382 Jul 09 16:26 sqlplus
-rwxr-xr-x 1 grid oinstall 4749 Jul 09 16:24 srvconfig
-rwxr-xr-x 1 root oinstall 8385 Jul 09 16:52 srvctl
-rwxr-xr-x 1 grid oinstall 1504 Jul 09 16:24 statusnc
-rwxr-xr-x 1 grid oinstall 3261 Aug 18 2010 symfind
-rwxr-x--x 1 grid oinstall 1024576 Jul 09 16:26 sysresv
-rwxr-x--x 1 grid oinstall 1024576 Jul 09 16:25 sysresv0
-rwxr-xr-x 1 grid oinstall 6937 Jul 09 16:26 targetdeploy.pl
-rwxr-x--x 1 grid oinstall 48254161 Jul 09 16:26 tkprof
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 tkprofO
-rwxr-x--x 1 grid oinstall 1243968 Jul 09 16:26 tnslsnr
-rwxr-xr-x 1 grid oinstall 0 Oct 21 2011 tnslsnr0
-rwxr-x--x 1 grid oinstall 48153143 Jul 09 16:25 tnsping
-rwxr-xr-x 1 grid oinstall 0 Oct 21 2011 tnsping0
-rwxr-xr-x 1 grid oinstall 2983 Jul 09 16:24 trcasst
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 trcldr
-rwxr-x--x 1 grid oinstall 48160811 Jul 09 16:26 trcroute
-rwxr-xr-x 1 grid oinstall 0 Oct 21 2011 trcroute0
-rwxr-xr-x 1 grid oinstall 2032 Jul 09 16:24 trcsess
-rwxr-x--x 1 grid oinstall 36508 Jul 09 16:25 tstshm
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 tstshmO
-rwxr-x--x 1 grid oinstall 753705 Jul 09 16:26 uidrvci
-rwxr-x--x 1 grid oinstall 753705 Jul 09 16:25 uidrvciO
-rwxr-xr-x 1 grid oinstall 3795 Jul 09 16:24 umu
-rwxr-xr-x 1 grid oinstall 149006 Oct 04 2011 unzip
-rwxr-xr-x 1 root oinstall 1133 Jul 09 16:52 usrvip
-rwxr-xr-x 1 grid oinstall 0 Jul 09 16:52 vipca
-rwxr-x--x 1 grid oinstall 12423821 Jul 09 16:25 wrap
-rwxr-x--- 1 grid oinstall 0 Oct 21 2011 wrapO
-rwxr-x--x 1 grid oinstall 48694807 Jul 09 16:26 wrc
-rwxr-x--x 1 grid oinstall 48694807 Jul 09 16:25 wrcO
-rwxr-xr-x 1 grid oinstall 3429525 Aug 25 2011 xml
-rwx------ 1 grid oinstall 54316 Sep 02 2009 xmlwf
-rwxr-xr-x 1 grid oinstall 291654 Oct 04 2011 zip
--end--