Oracle11gRACCRS-4535/ORA-15077

oracle 11.2.0.4的rac集群，其中一个节点出现故障，集群无法启动，

使用crsctl check crs查看集群状况如下：

[grid@db2 client]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online

ps -ef | grep d.bin #下面的查询中没有crsd.bin

查看tail -100 alertdb2.log
[client(193951)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db2/client/crsctl_oracle.log.

2016-11-22 16:37:37.539:

提示不能访问asm磁盘组中的OCR信息，详细信息查看client中的日志。

[grid@db2 client]$ tail -100 crsctl_oracle.log
2016-11-22 16:36:19.295: [ OCRASM][1798887200]ASM Error Stack : ORA-15077: could not locate ASM instance serving a required diskgroup
2016-11-22 16:36:19.302: [ OCRASM][1798887200]proprasmo: kgfoCheckMount returned [7]
2016-11-22 16:36:19.302: [ OCRASM][1798887200]proprasmo: The ASM instance is down
2016-11-22 16:36:19.506: [ OCRRAW][1798887200]proprioo: Failed to open [+OCR]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2016-11-22 16:36:19.506: [ OCRRAW][1798887200]proprioo: No OCR/OLR devices are usable
2016-11-22 16:36:19.506: [ OCRASM][1798887200]proprasmcl: asmhandle is NULL
2016-11-22 16:36:19.506: [ OCRRAW][1798887200]proprinit: Could not open raw device
2016-11-22 16:36:19.506: [ OCRASM][1798887200]proprasmcl: asmhandle is NULL
2016-11-22 16:36:19.506: [ default][1798887200]a_init:7!: Backend init unsuccessful : [26]
2016-11-22 16:36:19.736: [ OCRMSG][979273504]prom_waitconnect: CONN NOT ESTABLISHED (0,29,1,2)
2016-11-22 16:36:19.736: [ OCRMSG][979273504]GIPC error [29] msg [gipcretConnectionRefused]
2016-11-22 16:36:19.736: [ OCRMSG][979273504]prom_connect: error while waiting for connection complete [24]
[ CLWAL][979273504]clsw_Initialize: OLR initlevel [30000]
提示ASM实例关闭了，不能打开OCR。
解决：
[grid@db2 client]$ asmcmd
Connected to an idle instance.
ASMCMD> startup
ASM instance started

Total System Global Area 1703624704 bytes
Fixed Size 2253904 bytes
Variable Size 1667816368 bytes
ASM Cache 33554432 bytes
ASM diskgroups mounted

启动后，本地集群资源随即正常启动。