Unable to startup CRS as ASM failed to startup with "ORA-01017: invalid username/password; logon denied" (Doc ID 1681849.1)

In this Document

Symptoms

Changes

Cause

Solution

References

---

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.2 and later
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Backup Service - Version N/A and later
Information in this document applies to any platform.

SYMPTOMS

One of the node CRS was not coming up properly.

ASM did not come up automatically - it  failed to start  with "ORA-01017: invalid username/password; logon denied" error in the <grid home>/log/<host name>/agent/ohasd/oraagent_grid/oraagent_grid.log

alert<hostname>.log
~~~~~~~~~~~~~~~~~~~~
[client(3990)]CRS-10001:10-Jun-14 14:35 ACFS-9156: Detecting control device '/dev/ofsctl'.
[client(4002)]CRS-10001:10-Jun-14 14:35 ACFS-9322: completed
2014-06-10 14:36:18.128:
[ohasd(48895)]CRS-2807:Resource 'ora.asm' failed to start automatically.
2014-06-10 14:36:18.128:
[ohasd(48895)]CRS-2807:Resource 'ora.crsd' failed to start automatically.

oraagent_grid.log
~~~~~~~~~~~~~~~~~~
2014-06-10 14:35:15.585: [ora.asm][3219122496]{0:0:2} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=<Grid Infrastructure Home>/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=<Grid Infrastructure Home>,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM1)))
2014-06-10 14:35:15.587: [ora.asm][3219122496]{0:0:2} [start] Container:start oracle home <Grid Infrastructure Home>
2014-06-10 14:35:15.587: [ora.asm][3219122496]{0:0:2} [start] InstConnection::connectInt: server not attached
2014-06-10 14:35:15.621: [ora.asm][3219122496]{0:0:2} [start] ORA-01017: invalid username/password; logon denied  <<<<<<<<<

2014-06-10 14:36:12.960: [ AGFW][2622531904]{0:0:2} Agfw Proxy Server sending the reply to PE for message:RESOURCE_CLEAN[ora.asm 1 1] ID 4100:456
2014-06-10 14:36:12.960: [ CRSPE][2543831360]{0:0:2} Received reply to action [Clean] message ID: 456
2014-06-10 14:36:12.960: [ CRSPE][2543831360]{0:0:2} Got agent-specific msg: ORA-01017: invalid username/password; logon denied

ASM instance can be started manually  using sqlplus "/as sysasm" but the CRSD fails to start with same error.

crsd.log
~~~~~~~~~
2014-06-05 11:05:28.617: [ CRSMAIN][4003182320] Initializing OCR
[ CLWAL][4003182320]clsw_Initialize: OLR initlevel [70000]
2014-06-05 11:05:29.938: [ OCRASM][4003182320]proprasmo: Error in open/create file in dg [DATA]
[ OCRASM][4003182320]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=1017, loc=kgfokge
2014-06-05 11:05:29.938: [ OCRASM][4003182320]ASM Error Stack : ORA-01017: invalid username/password; logon denied <<<<<<<<
2014-06-05 11:05:30.985: [ OCRASM][4003182320]proprasmo: kgfoCheckMount returned [7]
2014-06-05 11:05:30.985: [ OCRASM][4003182320]proprasmo: The ASM instance is down
2014-06-05 11:05:30.985: [ OCRRAW][4003182320]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2014-06-05 11:05:30.985: [ OCRRAW][4003182320]proprioo: No OCR/OLR devices are usable

CHANGES

CAUSE

 SQLNET.ORA has below entry

# sqlnet.ora.Network Configuration File:
<Grid Infrastructure Home>/network/admin/sqlnet.ora
# Generated by Oracle configuration tools.
NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
SQLNET.AUTHENTICATION_SERVICES=none <<<<<<<<<<<<<<<<<<<<
ADR_BASE = /u01/app/grid

CRS uses OS authentication method to connect ASM instance.

If   SQLNET.AUTHENTICATION_SERVICES=none then a valid username and password is need to connect to ASM instance.

SOLUTION

1) Remove "SQLNET.AUTHENTICATION_SERVICES=none" from the Grid Home SQLNET.ORA file (location $ORACLE_HOME/network/admin)
2) Restart the CRS with force.
crsctl stop crs -f
crsctl start crs