Thursday, May 31, 2018

root.sh fails Reporting 'srvctl start nodeapps -n ... failed' as ons won't start (Doc ID 1965133.1)

In this Document
Symptoms
Changes
Cause
Solution
References


APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.

SYMPTOMS

New installation of 11.2 Grid Infrastructure, root.sh is fails, reporting to screen :
/u01/app/11.2.0.4/grid/bin/srvctl start nodeapps -n Node1 ... failed
FirstNode configuration failed at /u01/app/11.2.0.4/grid/crs/install/crsconfig_lib.pm line 9379.
 /u01/app/11.2.0.4/grid/perl/bin/perl -I/u01/app/11.2.0.4/grid/perl/lib -I/u01/app/11.2.0.4/grid/crs/install /u01/app/11.2.0.4/grid/crs/install/rootcrs.pl execution failed
Rootcrs log file shows :
2015-01-26 15:26:11: nodes_to_start=hqi-ebsdb-1
2015-01-26 15:26:32: exit value of start nodeapps/vip is 1
2015-01-26 15:26:32: output for start nodeapps is  PRCR-1013 : Failed to start resource ora.ons PRCR-1064 : Failed to start resource ora.ons on node Node1 CRS-5016: Process "/u01/app/11.2.0.4/grid/opmn/bin/onsctli" spawned by agent "/u01/app/11.2.0.4/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.4/grid/log/Node1/agent/crsd/oraagent_oracle/oraagent_oracle.log" CRS-2674: Start of 'ora.ons' on 'Node1' failed
2015-01-26 15:26:32: output of startnodeapp after removing already started mesgs is PRCR-1013 : Failed to start resource ora.ons PRCR-1064 : Failed to start resource ora.ons on node Node1 CRS-5016: Process "/u01/app/11.2.0.4/grid/opmn/bin/onsctli" spawned by agent "/u01/app/11.2.0.4/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.4/grid/log/Node1/agent/crsd/oraagent_oracle/oraagent_oracle.log" CRS-2674: Start of 'ora.ons' on 'Node1' failed
2015-01-26 15:26:32: /u01/app/11.2.0.4/grid/bin/srvctl start nodeapps -n Node1 ... failed
2015-01-26 15:26:32: Running as user oracle: /u01/app/11.2.0.4/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_NODECONFIG -state FAIL
2015-01-26 15:26:32: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/11.2.0.4/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_NODECONFIG -state FAIL '
2015-01-26 15:26:33: Removing file /tmp/fileoy7dDY
2015-01-26 15:26:33: Successfully removed file: /tmp/fileoy7dDY
2015-01-26 15:26:33: /bin/su successfully executed
ONS log file shows
[2015-01-26T15:26:15+04:00] [ons] [NOTIFICATION:1] [104] [ons-internal] ONS server initiated
[2015-01-26T15:26:15+04:00] [ons] [ERROR:1] [17] [ons-listener] 172.0.0.1,6100: BIND (Cannot assign requested address)

CHANGES

 New Installation

CAUSE

Incorrect value used for localhost in /etc/hosts
IP address 127.0.0.1 should only map to localhost and/or localhost.localdomain, not anything else
 

SOLUTION

Edit localhost and ensure that entry for localhost is
127.0.0.1 localhost.localdomain localhost
How to Validate Network and Name Resolution Setup for the Clusterware and RAC (Doc ID 1054902.1)
 

REFERENCES

NOTE:1054902.1 - How to Validate Network and Name Resolution Setup for the Clusterware and RAC

Best Practices for Corruption Detection, Prevention, and Automatic Repair - in a Data Guard Configuration (Doc ID 1302539.1)

In this Document
Purpose
Scope
Details
 Causes of corrupted blocks
 Corruption Summary
 Configuration Details
 Configure at Primary Database:
 Configure at Data Guard Standby Database:
 Deploy Primary and Standby on Oracle Exadata Database Machine and Exadata Storage Servers
 Deploy Oracle Data Integrity eXtensions (DIX) with T10 Data Integrity Field (DIF) when not on Exadata
 General Guidance on Performance Trade-offs
 DB_BLOCK_CHECKSUM- Background
 DB_BLOCK_CHECKING - Background
 DB_LOST_WRITE_PROTECT - Background
 Oracle Automatic Storage Management (ASM) - Background
 Oracle Flashback Technologies - Background
 Oracle Data Guard
 Active Data Guard Automatic Block Repair - Background
 Additional Operational Practices to detect block corruptions
References

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.1.0.7 to 12.1.0.1 [Release 11.1 to 12.1]
Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Information in this document applies to any platform.

***Checked for relevance on 3-Jul-2015*** 

PURPOSE

Oracle Active Data Guard is a data protection and availability solution for the Oracle Database.  The purpose of this document is to provide Database Administrators best practice recommendations for configuring key parameters, database features, system features and operational practices to enable best corruption detection, prevention, and automatic repair, in a MAA or Data Guard configuration. This note also provides additional background information on each parameter, performance considerations, and relevant Oracle documentation references.

SCOPE

This document is intended for Database Administrators wanting to learn how to prevent, detect and automatically repair from various data block corruptions.   
corrupt block is a block that has been changed so that it differs from what Oracle Database expects to find. This note covers two data block corruption types:
  • In a physical block corruption, which is also called a media corruption, the database does not recognize the block at all: the checksum is invalid or the block contains all zeros.  An example of a more sophisticated physical block corruption is when the block header and footer do not match.
  • In a logical block corruption, the contents of the block are physically sound and pass the physical block checks; however the block can be logically inconsistent. Examples of logical corruption include incorrect block type, incorrect data or redo block sequence number, corruption of a row piece or index entry or data dictionary corruptions.    
  • Block corruptions caused by stray writes, lost writes or misdirected writes can also cause havoc to your database availability. The data block may be physically or logically correct but in this case the block’s content is older or stale or in the wrong location. 
Block corruptions can also be divided into interblock corruption and intrablock corruption:
  • In intrablock corruption, the corruption occurs in the block itself and can be either a physical or a logical corruption.
  • In an interblock corruption, the corruption occurs between blocks and can only be a logical corruption.

DETAILS

Causes of corrupted blocks

Block corruptions can be caused by various failures including, but not limited to the following:
  • Faulty disks and disk controllers
  • Faulty memory
  • Faulty network components
  • Firmware, operating system, volume manager, NFS or third party software defects
  • Oracle Database software defects
Block corruptions can be also be caused by operator errors such as copying backups over existing data files or restoring inconsistent database backups.    Since corruptions can happen anywhere in the system and software stack, MAA recommends a comprehensive list of architectural and configuration best practices.   Active Data Guard is strategic and important component to achieve the most comprehensive Oracle data protection.   

Corruption Summary

The table outlines block corruption checks for various manual operational checks and runtime and background corruption checks.    Manual checks are something that the DBA and operations team can incorporate such as running RMAN backups, RMAN "check logical" validations or running ANALYZE VALIDATE STRUCTURE command on important objects.  Manual checks are especially important to validate data that are rarely updated or queried.    Runtime checks are far superior in that it will catch corruptions almost immediately or during runtime for actively queried and updated data.   Runtime checks can prevent corruptions or automatically fix corruptions resulting in better data protection and higher application availability.  A new background check has been introduced in Exadata to automatically scan and scrub disks intelligently with no application overhead and to automatically fix physically corrupted blocks.  
Checks
Capabilties
Physical Block Corruption
Logical Block Corruption
Manual checksDbverify, AnalyzePhysical block checksLogical intra-block and inter-object consistency checks
Manual checksRMANPhysical block checks during backup and restore operationsIntra-block logical checks
Manual checksASM ScrubPhysical block checksSome logical intra-block checks
Runtime checksActive Data Guard1. Continuous physical block checking at standby during transport and apply
2. Strong database isolation eliminates single point database failure
3. Automatic repair of block corruptions
4. Automatic database failover
1.  With DB_LOST_WRITE_PROTECT enabled, detection of lost writes (11.2 and higher).   With 11.2.0.4 and Data Guard broker, ability to shutdown the primary when lost writes are detected on the primary database.
2. With DB_BLOCK_CHECKING enabled on the standby, additional intra-block logical checks
Runtime checksDatabaseWith DB_BLOCK_CHECKSUM, in-memory data block and redo checksum validationWith DB_BLOCK_CHECKING, in-memory intra-block check validation
Runtime checksASMImplicit data corruption detection for reads and writes and automatic repair if good ASM extent block pair is available during writes
Runtime checksDIX + T10 DIFChecksum validation from operating system to HBA controller to disk (firmware). Validation for reads and writes for certified Linux, HBA and disks. 
Runtime checksHardware and StorageLimited checks due to lack of Oracle integration.  Checksum is most common.Limited checks due to lack of Oracle integration.  Checksum is most common
Runtime checksExadataComprehensive HARD checks on writesHARD checks on writes
Background checksExadataAutomatic HARD disk scrub and repair.  Detects and fixes bad sectors. 

Configuration Details

Configure at Primary Database:

  • DB_BLOCK_CHECKSUM=FULL
  • DB_BLOCK_CHECKING=FULL or MEDIUM
  • DB_LOST_WRITE_PROTECT=TYPICAL
  • Enable Flashback Technologies for fast point-in-time recovery from human errors (e.g. drop table, inadvertent or malicious data changes) and for fast reinstatement of a primary database following failover.

Configure at Data Guard Standby Database:

  • DB_BLOCK_CHECKSUM=FULL
  • DB_BLOCK_CHECKING=FULL or MEDIUM
  • DB_LOST_WRITE_PROTECT=TYPICAL
  • Enable Flashback Technologies for fast point-in-time recovery from human errors (e.g. drop table, inadvertent or malicious data changes) and for fast reinstatement of a primary database following failover.
  • Use Active Data Guard to enable Automatic Block Repair (Data Guard 11.2 onward).
 Review the additional background on each of these settings provided in the sections below, especially if tests show that any of the above recommendations have a greater than acceptable impact on the performance of your application.

Deploy Primary and Standby on Oracle Exadata Database Machine and Exadata Storage Servers

In addition to the settings above that provide optimal data protection for the Oracle Database on any platform supported by Oracle, the Exadata Database Machine and Sparc Supercluster also implements comprehensive Oracle Hardware Assisted Resilient Data (HARD) specifications, providing a unique level of validation for Oracle block data structures.   The Exadata HARD checks include support for spfiles, controlfiles, log files, Oracle data files and Data Guard broker file when residing on Exadata Storage and works during an ASM rebalance or during ASM resync operations.   Oracle Exadata Storage Server Software detects corruptions introduced into the I/O path between the database and storage. It stops corrupted data from being written to disk when a HARD check fails. This eliminates a large class of failures that the database industry had previously been unable to prevent.   Examples of the Exadata HARD checks include:  1) redo and block checksum, 2) correct log sequence, 3) block type validation, 4) block number validation, 5) Oracle data structures such as block magic number, block size, sequence#, and block header and tail data structures.   Exadata HARD checks are the most comprehensive list of Oracle data block checks initiating from the storage software (cellsrv) and works transparently after enabling database's DB_BLOCK_CHECKSUM parameter.   Except for the case of Exadata storage, the Oracle HARD initiative has ended.  Most past storage HARD implementations only provided checksums and very simple data block checks.
Starting with Oracle Exadata Software 11.2.3.3.0 and Oracle Database 11.2.0.4, Oracle Exadata Storage Server Software provides Automatic Hard Disk Scrub and Repair.   This feature automatically inspects and repairs hard disks periodically when hard disks are idle. If bad sectors are detected on a hard disk, then Oracle Exadata Storage Server Software automatically sends a request to Oracle ASM to repair the bad sectors by reading the data from another mirror copy. By default, the hard disk scrub runs every two weeks.   It’s very lightweight and has enormous value add by fixing physical block corruptions even with infrequently access data.

Deploy Oracle Data Integrity eXtensions (DIX) with T10 Data Integrity Field (DIF) when not on Exadata

Oracle Linux team has collaborated with hardware vendors and Oracle database development to extend Oracle data integrity extensions from Oracle’s operating system (Linux) to various vendor’s host adapter down to the storage device.  With these extensions, DIX provides end to end data integrity for reads and writes through a checksum validation.   The prerequisite is to leverage certified storage, HBA and disk firmware.  An example of this partnership is DIX integration with Oracle Linux, Emulex or QLogic Host Bus Adapters and any T10 DIF capable storage arrays such as EMC VMAX.    Refer to the following documentation for more information.

General Guidance on Performance Trade-offs

Performance implications are discussed in each of the sections below. In general, the processing that accompanies higher levels of corruption checking, automatic repair, or fast point in time recovery, will create additional overhead on primary and standby systems. While this overhead is reduced with every Oracle release as validation and repair algorithms are enhanced, the usual recommendation for conducting thorough performance testing still applies.

DB_BLOCK_CHECKSUM- Background

This parameter determines whether DBWn and the direct loader will calculate a checksum (a number calculated from all the bytes stored in the block) and store it in the cache header of every data block and redo log when writing to disk. The checksum is used to validate that a block is not physically corrupt, detecting corruptions caused by underlying disks, storage systems, or I/O systems. If checksum validation fails when it is set to FULL, Oracle will attempt to recover the block by reading it from disk (or from another instance) and applying the redo needed to fix the block. Corruptions are recorded as ORA-600 or ORA-01578 in the database or ASM alert logs.

Checksums do not ensure logical consistency of the block contents (see DB_BLOCK_CHECKING). Checksum checks happen in memory when a process reads the data or redo block into the SGA or PGA. Prior to writing an updated or new data or redo block, a new checksum is created. Potential clients of DB_BLOCK_CHECKSUM include: all foregrounds, DBWR, LGWR, LNS, RFS, ARCH, MRP, and recovery slaves.

Possible settings include:
  • OFF or FALSE: When set to OFF, DBWn calculates checksums only for the SYSTEM tablespace, but not for user tablespaces. In addition, no log checksum is performed when this parameter is set to OFF.
  • TYPICAL or TRUE: Checksums are verified when a block is read and the last write of the block stored a checksum.
  • FULL: In addition to checks made by TYPICAL, Oracle also verifies the checksum before a change application from update/delete statements and recomputes it after the change is applied. In addition, Oracle gives every log block a checksum before writing it to the current log.
  • Starting with Oracle Database 11g, CPU and cache efficiency has been improved by having the generating foreground processes perform most of the log block checksum, while the LGWR performs the remaining work. Prior to Oracle Database 11g, the LGWR solely performed the log block checksum, it would verify the checksum of each log block generated by the foreground processes before writing it to disk.
  • Best practice is to set DB_BLOCK_CHECKSUM=FULL on both the primary and standby databases, which typically incurs from 4% to 5% overhead on the system. Overhead for a setting of TYPICAL ranges from 1% to 2% for an OLTP workload and has minimal effect on Redo Apply performance, but also provides less protection. If tests show that using FULL at the primary results in unacceptable performance impact, consider setting the primary to TYPICAL and the standby database to FULL fo an optimal tradeoff between protection and performance (and also accept the fact that settings will have to be changed following role transitions).

DB_BLOCK_CHECKING - Background

This parameter specifies whether or not Oracle performs logical intra-block checking for database blocks (memory semantic check). Block checking will check block contents, including header and user data, when changes are made to the block and prevents in-memory corruptions from being written to disk. It performs a logical validation of the integrity of a block by walking through the data on the block, making sure it is self consistent. When DB_BLOCK_CHECKING is set at MEDIUM or FULL, block corruptions that are detected in memory are automatically repaired by reading the good block from disk and applying required redo. If for any reason the corruption cannot be repaired an error will be reported and the data block write will be prevented. All corruptions are reported as ORA-600 or ORA-01578 errors in the database or ASM alert logs.

Possible settings include:
  • OFF or FALSE - No block checking is performed for blocks in user tablespaces. However, semantic block checking for SYSTEM tablespace blocks is always turned on.
  • LOW - Basic block header checks are performed after block contents change in memory (for example, after UPDATE or INSERT statements, on-disk reads, or inter-instance block transfers in Oracle RAC). Low is very limited in usefulness for detecting and preventing data block corruption as it does not perform any data layer checks.
  • MEDIUM - All LOW checks and full semantic checks are performed for all objects except indexes (whose contents can be reconstructed by a drop and rebuild upon encountering a corruption).
  • FULL or TRUE - All LOW and MEDIUM checks plus full semantic checks are performed for all objects.
Oracle recommends setting DB_BLOCK_CHECKING to FULL at both primary and standby databases. Workload specific testing is required to assess whether the performance overhead of FULL is acceptable.  If tests show unacceptable performance impact, then set DB_BLOCK_CHECKING to MEDIUM.
Performance testing is particularly important given that overhead is incurred on every block change. Block checking typically causes 1% to 10% overhead, but for update and insert intensive applications (such as Redo Apply at a standby database) the overhead can be much higher. OLTP compressed tables also require additional checks that can result in higher overhead depending on the frequency of updates to those tables.
If performance concerns prevent setting DB_BLOCK_CHECKING to either FULL or MEDIUM at a primary database, then it becomes even more important to enable this at the standby database. This protects the standby database from logical block corruption that would be undetected at the primary database.
Note: When DB_BLOCK_CHECKING is set on the primary database, end-to-end checksums introduced in Oracle Database 11g make it unnecessary to use DB_BLOCK_CHECKING at the standby to detect primary database corruption. Oracle, however, still recommends enabling this parameter on the standby database for the following reasons:
  • The parameter is always enabled regardless of role, ensuring that the standby will have the correct settings when it assumes the primary database role
  • Even though very rare, enabling the parameter on the standby will protect against logical corruptions that may be introduced on the standby database independent of the primary.

DB_LOST_WRITE_PROTECT - Background

This parameter enables lost write detection. A data block lost write occurs when an I/O subsystem acknowledges the completion of the block write, while in fact the write did not occur in the persistent storage or in some cases an older version of the block was written out instead.

Possible settings include:
  • NONE: When set to NONE on either the primary database or the standby database, no lost write detection functionality is enabled.
  • TYPICAL: When set on a primary database it will log buffer cache reads for read-write tablespaces in the redo log and will detect lost writes when the primary database performs media recovery. By also setting TYPICAL at a Data Guard physical standby database, the Data Guard MRP process will continuously check for lost writes that occurred to read-write tablespaces at the primary database, enabling immediate action to be taken and preventing corruptions from being applied to the standby database. It will also detect lost writes that may occur at a standby database independent of the primary.
  • FULL: All logging, checking, and detecting that occurs with TYPICAL, but for both read-write and read-only tablespaces.

When a primary database lost write corruption is detected by a Data Guard physical standby database, Redo Apply (MRP) will stop and the standby will signal an ORA-752 error to explicitly indicate a primary lost write has occurred. Note that a Data Guard physical standby also detects other types of corruptions that can occur at either the primary or standby database and will likewise stop Redo Apply, in these cases signaling a more generic ORA-600 [3020] error. In cases where problems originated at the primary database this prevents corruptions from being applied to the standby database and indicates immediate action must be taken to prevent further corruption and downtime. While there is no threat to the primary database in cases where problems originate at the standby database, this provides immediate warning that corrective action must be taken to maintain data protection. For more information on lost write detection and instructions for repair from either ORA-752 or ORA-600 [3020] errors, see My Oracle Support Document 1265884.1.

Primary performance overhead is negligible when lost write protection is enabled as the validation is performed by the standby database. For DSS and Data Warehouse applications, however, where there is a high volume of queries from non-READONLY tablespaces that are updated infrequently, the primary redo generation will increase. The redo apply throughput at the standby database can be impacted due to additional data block read I/Os. For most cases, however, the impact on the primary database and standby databases is negligible. The best way to estimate the impact on the DB_LOST_WRITE_PROTECT on the primary or standby database is to enable it dynamically in your test environment and eventually in your production and standby databases.

Monitor the change in the application or redo apply throughput and evaluate the database statistics described below. The primary redo rate and redo size will grow (as measured by database statistic redo size for lost write detection) after enabling this setting since block read redo (BRR) will be created to record the version of the block at subsequent reads. The size impact is mitigated since Oracle attempts to batch the BRRs in a single redo record if possible and a typical BRR redo change is only 144 bytes and redo header is 40 bytes in a 64 bit platform.  BRRs are recorded only for physical reads from disk since the goal is to detect lost writes or stale reads from disk. After enabling DB_LOST_WRITE_PROTECT on a physical standby database, you will encounter an increase in physical reads especially for data that is updated very infrequently but read semi-frequently (forcing a physical read) on the primary database.  The additional physical reads on the physical standby database will be measured via database statistic recovery blocks read for lost write detection which can be extracted from the Standby Statspack report.
Starting with Oracle 11.2.0.4, there's a new Data Guard broker configuration-level property, PrimaryLostWriteAction. This property allows the user to choose whether the primary continues operation or is shutdown. The default value is CONTINUE.   Refer to Oracle® Data Guard Broker 11g Release 2 (11.2) documentation.
Setting DB_LOST_WRITE_PROTECT on a non-Data Guard environment is still advantageous for better troubleshooting and debugging of lost writes problems.
Starting in Oracle 12.1, Active Data Guard with lost write protect can clearly detect lost writes on the standby.   When recovery process detects a lost write on the standby, the following error will be reported.
ORA-00753: recovery detected a lost write of a data block
Cause: A data block write to storage was lost during normal redo database operation on the standby database or during recovery on a primary database.
An example of the benefit of using DB_LOST_WRITE_PROTECT, refer to  Data Guard Protection From Lost-Write Corruption demo at http://www.oracle.com/technetwork/database/features/availability/demonstrations-092317.html.

Oracle Automatic Storage Management (ASM) - Background

Read errors can be the result of a loss of access to the entire disk or media corruptions on an otherwise healthy disk. Oracle ASM tries to recover from read errors on corrupted sectors on a disk. When a read error by the database or Oracle ASM triggers the Oracle ASM instance to attempt bad block remapping, Oracle ASM reads a good copy of the extent and copies it to the disk that had the read error.
  • If the write to the same location succeeds, then the underlying allocation unit (sector) is deemed healthy. This might be because the underlying disk did its own bad block reallocation.
  • If the write fails, Oracle ASM attempts to write the extent to a new allocation unit on the same disk. If this write succeeds, the original allocation unit is marked as unusable. If the write fails, the disk is taken offline.
Another benefit with Oracle ASM based mirroring is that the database instance is aware of the mirroring. For many types of physical block corruptions such as a bad checksum, the database instance proceeds through the mirror side looking for valid content and proceeds without errors. If the process in the database that encountered the read can obtain the appropriate locks to ensure data consistency, it writes the correct data to all mirror sides.
When encountering a write error, a database instance sends the Oracle ASM instance a disk offline message.
  • If database can successfully complete a write to at least one extent copy and receive acknowledgment of the offline disk from Oracle ASM, the write is considered successful.
  • If the write to all mirror side fails, database takes the appropriate actions in response to a write error such as taking the tablespace offline.
When the Oracle ASM instance receives a write error message from a database instance or when an Oracle ASM instance encounters a write error itself, the Oracle ASM instance attempts to take the disk offline. Oracle ASM consults the Partner Status Table (PST) to see whether any of the disk's partners are offline. If too many partners are offline, Oracle ASM forces the dismounting of the disk group. Otherwise, Oracle ASM takes the disk offline.
The ASMCMD remap command was introduced to address situations where a range of bad sectors exists on a disk and must be corrected before Oracle ASM or database I/O. For information about the remap command, see "remap".
When ASM detects any block corruptions, ASM logs the error to the ASM alert.log file.  The same corruption error may not appear in the database alert.log or application if ASM can correct the corruption automatically.
Starting Oracle 12c, Oracle ASM disk scrubbing checks logical data corruptions and repairs the corruptions automatically in normal and high redundancy disks groups. The feature is designed so that it does not have any impact to the regular input and output (I/O) operations in production systems. The scrubbing process repairs logical corruptions using the Oracle ASM mirror disks. Disk scrubbing uses Oracle ASM rebalancing to minimize I/O overhead.
The scrubbing process is visible in fields of the V$ASM_OPERATION view.    Refer to Oracle® Automatic Storage Management Administrator's Guide 12c Release 1 (12.1).
These ASM benefits are available for all databases using ASM.   Since every Exadata Database Machine uses ASM, all these benefits are always available for Exadata customers.  

Oracle Flashback Technologies - Background

Flashback Database is used for fast point-in-time recovery to recover from human errors that cause widespread damage to a production database. It is also used for fast reinstatement of a failed primary database as a new standby database following a Data Guard failover. Flashback Database uses flashback logs to rewind an Oracle Database to a previous point in time. See My Oracle Support Document 565535.1 for Flashback Database best practices. See Section 13.2 of Data Guard Concepts and Administration for Fast Reinstatement using Flashback Database. Note that new optimizations in Oracle Database 11.2.0.2 reduce the impact on load operations when Flashback Database enabled.

Flashback Database can also be used on a Data Guard standby database to enable point-in-time recovery of the primary database at a schema level without requiring that the entire primary database be flashed back. For example, take the case of a primary database hosting a number of applications and different schemas where an incorrect batch job was run for one of the applications. Assuming no object dependencies between schemas, you can perform a granular repair of the primary database in the following manner:
  • Stop the application where point-in-time recovery is required and let other applications continue to run.
  • Flashback the standby database to the desired point in time and extract a good copy of the affected schema.
  • Repair the primary database by replacing the schema using the good copy from the standby, and restart the application.
  • Restart redo apply on the standby database. It will resynchronize the standby by reapplying the bad batch job run and the reload that repaired it.

More granular levels of repair from human errors (e.g. drop table, inadvertent or malicious updates or DDLs) are enabled by other Oracle Flashback technologies that include: Flashback Query, Flashback Version Query, Flashback Transaction, Flashback Transaction Query, Flashback Table and Flashback Drop. Flashback Drop only requires configuring a Recycle Bin. All other features use automatic undo management. See Section 4.2.7 of Oracle Database HA best practices for detailed best practices on the full complement of Oracle Flashback Technologies.
Starting in Oracle 11g, manual RMAN block media recovery will automatically search flashback logs for good copies of blocks to help repair from physical data block corruptions quickly.

Oracle Data Guard

Oracle Data Guard ensures high availability, data protection, and disaster recovery for enterprise data.   One of the key benefits of Oracle Data Guard is its continuous Oracle-aware validation of all changes using multiple checks for physical and logical consistency of structures within an Oracle data block and redo before updates are applied to a standby database. This isolates the standby database from being impacted by data corruptions that can occur on the primary system.
Active Data Guard Automatic Block Repair - Background
If Oracle detects a physical block corruption on either the primary or a standby database in a configuration that uses Active Data Guard, Oracle will automatically repair the corrupt block using a valid copy from the other database. This repair is transparent to primary database applications and users. No application modifications are necessary. If the nature of the corruption makes it impossible to be repaired automatically (e.g. file header corruption, max block repair timeout of 60 seconds for one block repair or number of outstanding block corruptions reaching 100 block corruption incidents), an ORA-1578 error is returned to the application.    Manual block media recovery can be executed at a later time.  Automatic Block Repair requires an Active Data Guard standby and Data Guard real-time apply. The following database initialization parameters must be configured on the standby database: the following database initialization parameters are configured on the standby database:
  • The LOG_ARCHIVE_CONFIG parameter is configured with a DG_CONFIG list and a LOG_ARCHIVE_DEST_n parameter is configured for the primary database
  • The FAL_SERVER parameter is configured and its value contains an Oracle Net service name for the primary database.
Active Data Guard auto block repair can fix physical block corruptions which is the most common type of block corruptions.    It does not address logical block corruptions which are normally prevented by setting DB_BLOCK_CHECKING on the primary and standby.   If enabling DB_BLOCK_CHECKING incurs an unacceptable performance impact on the primary, we recommend enabling on your standby database.

Starting in Oracle 11g Release 2, manual RMAN block media recovery on the primary will automatically try to use a real-time query physical standby database.

For an example of the benefits of using Active Data Guard auto block repair mechanisms, refer to http://www.oracle.com/technetwork/database/features/availability/demonstrations-092317.html
  • Automatic Block Repair at a Primary Database - Active Data Guard demo
  • Automatic Block Repair at a Standby Database - Active Data Guard demo

Additional Operational Practices to detect block corruptions

  1. Query v$database_block_corruption periodically on the primary and standby databases for any persistent data block corruptions
  2. Scan and alert on any ORA-1578s in the  primary, standby and ASM alert.logs for every instance
  3. Scan for IO errors in the database alert.log,  ASM alert.log and cell alert.log files    (Example: “IO Error”  or “I/O error”)
  4. Execute RMAN backup and restore operations which will automatically check for physical block corruptions.   This is particularly helpful for data that is rarely queried or updated.
  5. Periodically execute RMAN commands with “check logical” option to detect logical block corruptions.  
  6. Periodically execute low impact ASM disk scrubbing in the background.   This is not necessary for Exadata 11.2.3.3 and higher since Exadata’s Automatic HARD scrub and repair is automatically enabled
  7. Use ANALYZE statement with the VALIDATE STRUCTURE option if you suspect corruptions due to inter-object or inter-block corruptions.    Refer to below for more information.     
To verify the integrity of the structure of a table, index, cluster, or materialized view, use the ANALYZE statement with the VALIDATE STRUCTURE option. If the structure is valid, no error is returned. However, if the structure is corrupt, you receive an error message.
For example, in rare cases such as hardware or other system failures, an index can become corrupted and not perform correctly. When validating the index, you can confirm that every entry in the index points to the correct row of the associated table. If the index is corrupt, you can drop and re-create it.
If a table, index, or cluster is corrupt, you should drop it and re-create it. If a materialized view is corrupt, perform a complete refresh and ensure that you have remedied the problem. If the problem is not corrected, drop and re-create the materialized view.

REFERENCES

NOTE:1265884.1 - Resolving ORA-752 or ORA-600 [3020] During Standby Recovery
NOTE:565535.1 - Flashback Database Best Practices & Performance
http://www.oracle.com/technetwork/database/availability/maa-datacorruption-bestpractices-396464.pdf 

Cluster Health Monitor (CHM) FAQ (Doc ID 1328466.1)

In this Document
Purpose
Questions and Answers
 What is the Cluster Health Monitor?
 What is the purpose of the Cluster Health Monitor?
 What platform does Cluster Health Monitor support and where can I get the Cluster Health Monitor?
 What is the resource name for Cluster Health Monitor in 11.2.0.2 or higher?
 Is stop/start ora.crf affecting clusterware function or cluster database function?
 Can the Cluster Health Monitor be installed on a single node, non-RAC server?
 Do Engineered Systems like Exadata have a default usage with CHM and if so, any specific version??
 Where is oclumon?
 How do I collect the Cluster Health Monitor data?
 Why does “diagcollection.pl --collect --chmos” return “Cannot parse master from output: ERROR : in reading init file” error?
 How do you get the syntax of different options and explanations for those options for diagcollection.pl and oclumon?
 What is IPD/OS?
 How is the Cluster Health Monitor different from OSWatcher?
 Is the Cluster Health Monitor replacing OSWatcher?
 How much of overhead does the Cluster Health Monitor cause?
 Does CHM on Multiple Node configurations (e.g. 4 to 8 nodes) have scaling concerns?
 Will CDB and PDB result in any new information or special conditions using CHM?
 How much of disk space is needed for the Cluster Health Monitor?
 How do I find out the size of data collected and saved by the Cluster Health Monitor in my system?
 How can I increase the size of the Cluster Health Monitor repository ?
 What platforms can I run the Cluster Health Monitor?
 What steps are needed to install 11.2.0.2 when the Cluster Health Monitor from OTN is already running?
 Where does the Cluster Health Monitor from OTN installed in Linux?
 What logs and data should I gather before logging a SR for the Cluster Health Monitor error?
 How do I increase the trace level the Cluster Health Monitor?
 Can I use procwatcher to get the pstack of the Cluster Health Monitor regularly?
 What are the processes and components for the Cluster Health Monitor?
 What is oclumon?
 What is definition of some of the files like *.bdb, _db.* , *.ldb , log.* files created by tool in the BDB (Berkeley Database) location directory ?
 Because it takes many days / weeks to resolve a problem like the node reboot or performance degradation, is there any way to keep the Cluster Health Monitor data for that long so that it can be replayed any time later when needed ?
 Where is the location for the log files for the Cluster Health Monitor from OTN (pre 11.2.0.2)?
 How do I fix the problem that the time in the oclumon report is in UTC time zone instead of the time zone of my server?
 Can I install CHM from OTN on 11.2.0.2? What if I stop and disable CHM resource (ora.crf) on 11.2.0.2?
 Where is the trace file for client like oclumon? How do I increase the trace level for oclumon?
 Can the Directory path to the CHM Repository be same on all nodes if shared storage is used?
 How much of data (how long in time) does the node store CHM data locally when it cannot communicate with the master?
 How often does CHM collect the system metric data? Can this be changed?
 What is the default CHM retention time? 
 How can you reduce the size of bdb file that became big for any reason?
 Can you set up CHM to run locally on each node?
 Can CHM be used on a single node non-RAC server?
 How to start and stop CHM that is installed as a part of GI in 11.2 and higher?
 What is the reason for following alerts in cluster on all the nodes?
 

<Internal_Only> What does "oclumon showtrail ..." and "oclumon showobjects ..." show? What are the meaning of trail and what are the objects that are listed by showobjects?
 Database - RAC/Scalability Community
References

APPLIES TO:

Oracle Database - Enterprise Edition - Version 10.1.0.2 to 12.1.0.2 [Release 10.1 to 12.1]
Information in this document applies to any platform.

PURPOSE


The Cluster Health Monitor FAQ is an evolving document that answers common questions about the Cluster Health Monitor

QUESTIONS AND ANSWERS

What is the Cluster Health Monitor?

The Cluster Health Monitor collects OS statistics (system metrics) such as memory and swap space usage, processes, IO usage, and network related data. The Cluster Health Monitor collects information in real time and usually once a second. The Cluster Health Monitor collects OS statistics using OS API to gain performance and reduce the CPU usage overhead. The Cluster Health Monitor collects as much of system metrics and data as feasible that is restricted by the acceptable level of resource consumption by the tool.

What is the purpose of the Cluster Health Monitor?

The Cluster Health Monitor is developed to provide system metrics and data for troubleshooting many different types of problems such as node reboot and hang, instance eviction and hang, severe performance degradation, and any other problems that need the system metrics and data.

By monitoring the data constantly, users can use the Cluster Health Monitor detect potential problem areas such as CPU load, memory constraints, and spinning processes before the problem causes an unwanted outage.

What platform does Cluster Health Monitor support and where can I get the Cluster Health Monitor?

The Cluster Health Monitor is NOT supported on Linux Itanium,  and IBM Linux Z and HP-UX.

The Cluster Health Monitor is integrated part of 11.2.0.2 Oracle Grid Infrastructure for Linux (not on Linux Itanium and IBM Linux Z) and Solaris (Sparc 64 and x86-64 only), so installing 11.2.0.2 Oracle Grid Infrastructure on those platforms will automatically install the Cluster Health Monitor. AIX will have the Cluster Health Monitor starting from 11.2.0.3. The Cluster Health Monitor is also enabled for Windows (except Windows Itanium) in 11.2.0.3.

Prior to 11.2.0.2 on Linux (not on Linux Itanium and IBM Linux Z), the Cluster Health Monitor can be downloaded from OTN.

http://www-content.oracle.com/technetwork/products/clustering/downloads/ipd-download-homepage-087212.html

The OTN version for Windows is not available.  Please upgrade to 11.2.0.3 if you need CHM for Windows.

What is the resource name for Cluster Health Monitor in 11.2.0.2 or higher?

ora.crf is the Cluster Health Monitor resource name that ohasd manages. Issue “crsctl stat res –t –init” to check the current status of the Cluster Health Monitor.

Is stop/start ora.crf affecting clusterware function or cluster database function?

No, stop/start ora.crf resource will stop and start Cluster Health Monitor and its data collection, it will not affect clusterware or database functionality.

Can the Cluster Health Monitor be installed on a single node, non-RAC server?

The Cluster Health Monitor Standalone for LINUX x86 and x86-64 can be downloaded from OTN, it can be installed on a single node, non-RAC server without the need to install Grid Infrastructure or CRS. For other platform, it is required to install Grid Infrastructure or CRS to get CHM/OS.

Do Engineered Systems like Exadata have a default usage with CHM and if so, any specific version??

Engineered systems use the default GI stack that includes CHM functionality as shipped on standard platforms. At this time there are no specific extensions for Engineered systems but this may change in future releases.

Where is oclumon?

If the CHM is installed as a part of 11.2 installation on the supported platform, then the location of oclumon is in GI_HOME/bin directory.

If the CHM is manually installed using the CHM file from OTN, then the location of oclumon is in:
Linux : /usr/lib/oracrf/bin
Windows : C:\Program Files\oracrf\bin

How do I collect the Cluster Health Monitor data?

As grid user, using command “<GI_HOME>/bin/diagcollection.pl --collect --chmos” will produce output for all data that is collected in the repository. There may be too much data and may take long time, so the suggestion is limit the query to an interesting time interval.

For example, issue “<GI_HOME>/bin/diagcollection.pl --collect --crshome $ORA_CRS_HOME --chmos --incidenttime <start time of interesting time period> --incidentduration 05:00”

The above outputs the report that covers 5 hours from the time specified by incidenttime.
The incidenttime must be in MM/DD/YYYYHH:MN:SS where MM is month, DD is date, YYYY is year, HH is hour in 24 hour format, MN is minute, and SS is second. For example, if you want to put the incident time to start from 10:15 PM on June 01, 2011, the incident time is 06/01/201122:15:00. The incidenttime and incidentduration can be changed to capture more data.

Alternatively, ‘oclumon dumpnodeview -allnodes -v -last "11:59:59" > your-filename’ if diagcollection.pl fails with any reason. This will generate a report from the repository up to last 12 hours. The -last value can be changed to get more or less data.

Another example of using oclumon is 'oclumon dumpnodeview -allnodes -v -s "2012-06-01 22:15:00" -e "2012-06-02 03:15:00" > /tmp/chm.log '.  The difference in this command is that it specifies the start (-s flag) and end time (-e flag).
In this case, the time format used is "YYYY-MM-DD HH24:MI:SS" like "2007-11-12 23:05:00".

Why does “diagcollection.pl --collect --chmos” return “Cannot parse master from output: ERROR : in reading init file” error?

This is due to bug 10048487 that affects 11.2.0.2. As a result, the bug in the script causes the diagcollection.pl to never be able to retrieve the master node.

The workaround for this is to issue
oclumon dumpnodeview -allnodes -v -last “amount of data needed”
For example, oclumon dumpnodeview -allnodes -v -last “01:00:00”
will provide last one hour of data from all nodes.

How do you get the syntax of different options and explanations for those options for diagcollection.pl and oclumon?

Issue “<GI_HOME>/bin/diagcollection.pl –h” and “oclumon –h”. You may need to drill down further to get information for different options.

What is IPD/OS?

The IPD/OS is an old name for the Cluster Health Monitor. The names can be used interchangeably although Oracle now calls the tool Cluster Health Monitor.

How is the Cluster Health Monitor different from OSWatcher?

OSWatcher collects OS statistics by running regular unix commands such as vmstat, top, ps, iostat, netstat, mpstat, and meminfo. The private.net file can be configured in OSWatcher to issue traceroute command over the private interconnect to test the private interconnect. OSWatcher also runs in a user priority, so OSWatcher often cannot run when CPU load is heavy.

Is the Cluster Health Monitor replacing OSWatcher?

The Cluster Health Monitor has many advantages over OSWatcher, and the most significant is that the Cluster Health Monitor runs in real time and usually once a second, so the Cluster Health Monitor will collect data even when OSWatcher cannot. However, there are some information such as top, traceroute, and netstat that the Cluster Health Monitor does not collect, so running the Cluster Health Monitor while running OSWatcher is ideal. Both tools complement each other rather than supplement.
On the other hand, if only one of the tools can be used, then Oracle recommends that the Cluster Health Monitor is used.

How much of overhead does the Cluster Health Monitor cause?

In today's server environment, the Cluster Health Monitor uses approximately less than 3% of the server's capacity for CPU. The overhead of using the Cluster Health Monitor is minimal.  However. CHM on the server with large number of disks or IO devices and more CPUs/memory would use more CPU than CHM on a server that does not have many disks and CPUs/memory.

Does CHM on Multiple Node configurations (e.g. 4 to 8 nodes) have scaling concerns?

CHM functionality is designed to scale automatically with the cluster. While each node hosts an osysmond daemon, the ologgerd daemon services multiple osysmonds. Should a cluster grow large enough another ologgerd daemon is spawned to manage the increased load. The user is responsible for increasing the CHM data repository size as nodes are added to ensure sufficient retention time is maintained. This is recommended to be 72 hours.

Will CDB and PDB result in any new information or special conditions using CHM?

As CHM is collecting OS metrics there currently are no CDB or PDB specific metrics collected. There are currently no special conditions that are triggered when hosting a multitenant (CDB) database.

How much of disk space is needed for the Cluster Health Monitor?

The Cluster Health Monitor takes up 1GB space by default on all nodes in the cluster. The approximate amount of data collected is 0.5 GB per node per day. The size of the repository can increase to collect and save data up to 3 days, and this will increase the disk usage appropriately.

How do I find out the size of data collected and saved by the Cluster Health Monitor in my system?

“oclumon manage -get repsize” will show the size in seconds.
To estimate the space required, use the following formula:

# of nodes * 720MB * 3 = Size required for 3 days retention 
eg. for 4 node cluster: 4 * 720 * 3 = 8,640MB (8.4GB)

How can I increase the size of the Cluster Health Monitor repository ?

“oclumon manage -repos resize <number in seconds less than 259200>”. Setting the value to 259200 will collect and save the data for 72 hours (3 days). It is recommended to set 72 hours of retention based on above formula. This space needs to be available on all node in the cluster. Please resize the repositories or moving them if necessary in order to achieve 72 hours of retention.

What platforms can I run the Cluster Health Monitor?

11.2.0.1 and earlier: Linux only (download from OTN)
11.2.0.2: Solaris (Sparc 64 and x86-64 only), and Linux.
11.2.0.3: AIX, Solaris (Sparc 64 and x86-64 only), Linux, and Windows.

Cluster Health Monitor is NOT available for any Itanium platform such as Linux Itanium and Windows Itanium.

What steps are needed to install 11.2.0.2 when the Cluster Health Monitor from OTN is already running?

Remove the Cluster Health Monitor from OTN before upgrading the CRS or installing Grid Infrastructure.

Where does the Cluster Health Monitor from OTN installed in Linux?

$CRF_HOME is set /usr/lib/oracrf on Linux by default if the Cluster Health Monitor is from OTN. This is the Cluster Health Monitor home location.

What logs and data should I gather before logging a SR for the Cluster Health Monitor error?

1) provide 3-4 pstack outputs over a minute for osysmond.bin
2) output of strace -v for osysmond.bin about 2 minutes.
3) strace -cp <osysmond.bin pid> for about 2 min
4) oclumon dumpnodeview -v output for that node for 2 min.
5) output of "uname -a"
6) outpuft of "ps -eLf | grep osysmond.bin"
7) The ologgerd and sysmond log files in the CRS_HOME/log/<host name> directory from all nodes

How do I increase the trace level the Cluster Health Monitor?

Increase the log level for the daemons using,
oclumon debug log all allcomp:<trace level from 0 to 3>

Higher the trace level, more detailed tracing is done, so do not forget to reset the trace level back to 1 (the default trace level when the CHM is first installed) by issuing "oclumon debug log all allcomp:1"

Can I use procwatcher to get the pstack of the Cluster Health Monitor regularly?

Procwatcher version 030810 can now be used to monitor IPD procs. Just add the proc names to the CLUSTERPROCS list. The change is that Procwatcher is now smarter about picking the path of the executable so now it can find the IPD daemons if it is looking for them.

What are the processes and components for the Cluster Health Monitor?

Cluster Logger Service (ologgerd)
1) in 11.1, there is a master ologgerd that receives the data from other nodes and saves them in the repository (Berkeley database). It compresses the data before persisting to save the disk space. In an environment with multiple nodes, a replica ologgerd is also started on a node where the master ologgerd is not running. The master ologgerd will sync the data with replica ologgerd by sending the data to the replica ologgerd. The replica ologgerd takes over if the master ologgerd dies. A new replica ologgerd starts when the replica ologgerd dies. There is only one master ologgerd and one replica ologgerd per cluster.

To find the master olggerd, one can use the following command:
oclumon manage -get master

2) in 12.1 and above, because the CHM no longer stores the data to BDB but stores to the management database, CHM no longer needs to run ologgerd on multiple nodes, so ologgerd runs only on the node where the management database runs.

System Monitor Service (osysmond) – the sysmond process collects the system statistics of the local node and sends the data to the master ologgerd. A sysmond process runs on every node and collects the system statistics including CPU, memory usage, platform info, disk info, nic info, process info, and filesystem info.

What is oclumon?

OCLUMON command-line tool - use oclumon command line to query the CHM repository to display node-specific metrics for a specified time period.

You can also use oclumon to query and print the durations and the states for a resource on a node during a specified time period. These states are based on predefined thresholds for each resource metric and are denoted as red, orange, yellow, and green, indicating decreasing order of criticality.

What is definition of some of the files like *.bdb, _db.* , *.ldb , log.* files created by tool in the BDB (Berkeley Database) location directory ?

*.bdb & _db.* - These are files created for the berkeley db which stores the data collected.

log.* - These are berkeley bdb logfiles which preserve changes before making them to the db files. We have checkpointing setup and it reuses the log files.

*.ldb - This is the local logging file and MUST be present on all servers.

Do not delete above files except in case of trying to reduce the size of bdb file that get grow to a large size.  To reduce the size of bdb file, refer to the question "How can you reduce the size of bdb file that became big for any reason?" in this document.

Because it takes many days / weeks to resolve a problem like the node reboot or performance degradation, is there any way to keep the Cluster Health Monitor data for that long so that it can be replayed any time later when needed ?

The Cluster Health Monitor is designed to store data up to 3 days as best as it can by increasing the size of the repository up to 2GB. If you want to store data more than that, one way is to zip the output from ‘oclumon dumpnodeviews’ or ‘diagcollection’ regularly (like every hour).

Before 12.1.0.2, another way is to archive the whole BDB regularly (like every day) by making a copy of BDB file in the BDB location directory.

The way that CHMOS reads archived BDB is to start it in debug mode. It starts by using
ologdbg -d <bdb location>
After it starts, issue the oclumon dumpnodeview to get the data from the archived BDB.
For example, issue
oclumon dumpnodeview -n <node name> -s <start time> -e <end time> -v

Where is the location for the log files for the Cluster Health Monitor from OTN (pre 11.2.0.2)?

Check directory /usr/lib/oracrf/log/* for the alert<nodename>.log and other subdir for each daemons (SYSMOND, LOGGERD, OPROXYD) log.

How do I fix the problem that the time in the oclumon report is in UTC time zone instead of the time zone of my server?

The time in the repository is in UTC, and by default, oclumon shows the time in UTC. Check README, it shows UTC if ORACRF_TZ not set. Setting ORACRF_TZ should fix the time zone issue.

Can I install CHM from OTN on 11.2.0.2? What if I stop and disable CHM resource (ora.crf) on 11.2.0.2?

You cannot install CHM from OTN if there is any conflicting install, so installing CHM from OTN on servers that has 11.2.0.2 Grid Infrastructure will not work.  Disabling CHM resource (ora.crf) on 11.2.0.2 will still keep the installation; hence, OTN install will fail.

Where is the trace file for client like oclumon? How do I increase the trace level for oclumon?

The 'log' file for oclumon is in log/<hostname>/clients/oclumon.log.

Generally its not generated because, at the log level 0, there is no log data.
To see logs at higher log level one needs to do the following
1. oclumon [Enter the interactive mode]
2. query> debug log all allcomp:3

After this, any command execution will produce finer logs in oclumon.log

Can the Directory path to the CHM Repository be same on all nodes if shared storage is used?

One can set CHM repository at a shared storage under the same directory although it is recommended not to do so. One reason is the performance issue. In such a case, each node’s repository location is under the directory named as its hostname.

How much of data (how long in time) does the node store CHM data locally when it cannot communicate with the master?

The local repository size is small for nodes that need to save the local CHM data when it cannot communicate with the master.

With a sampling interval of 1 second, ideally it will be around 1 hour of data. With 11.2.0.3, we have moved to sampling interval of 5 seconds, hence, in that case the data that can be retained is 4-5 hours of data.

How often does CHM collect the system metric data? Can this be changed?

In pre-11.2.0.3, the CHM collection interval is usually once a second, but this can change depending on the the amount of data getting collected.  In 11.2.0.3, the CHM collection interval is changed to once every 5 seconds.

Currently, the collection interval can not be changed.

What is the default CHM retention time? 

In pre-11.2.0.2 CHM available from OTN, the default data retention time was 24 hours.

In 11.2.0.2, the retention time is determined by the size.   The default size has changed to 1GB. Depending on how large the cluster is, the default retention time is different.  For example, it is usually 6.9 hours for a one-node cluster when sampling interval is 1 second.   Please issue "oclumon manage -get repsize" to find out the retention time of your cluster.  The output is in seconds.

With sampling interval moving to 5 seconds in 11.2.0.3, the retention time becomes 5 times retention time with sampling interval 1 second.

It is recommended to set 72hours retention time.

How can you reduce the size of bdb file that became big for any reason?

You can manage repository size in terms of space using below command. This feature is present from 11.2.0.3.

oclumon manage -repos changesize <memsize>.

As a temporary work around, you can kill ologgerd and delete the contents in the BDB directory. osysmond should respawn ologgerd and new bdb file will get created. The past data is lost when this is done.

Please note the minimum size must be >= 1024 MB (1 GB), otherwise CRS-9100 "Error setting Cluster Health Monitor repository size" will be reported.

Can you set up CHM to run locally on each node?

On OTN, one can do that by installing CHM on each node independently although it is not recommended.

The Cluster Health Monitor that comes with the Grid Infrastructure install image must run with only one master ologgerd, so it can not be set  up to run locally on each node.

Can CHM be used on a single node non-RAC server?

The CHM available on OTN can be used on a single node non-RAC server, but only Linux and Windows version of CHM is available from OTN.  The CHM that comes with GI in 11.2 and higher must run with GI (RAC)

How to start and stop CHM that is installed as a part of GI in 11.2 and higher?

The ora.crf resource in 11.2 GI (and higher) is the resource for CHM, and the ora.crf resource is managed by ohasd. Starting and stopping ora.crf resource starts and stops CHM.

To stop CHM (or ora.crf resource managed by ohasd)
$GRID_HOME/bin/crsctl stop res ora.crf -init

To start CHM (or ora.crf resource managed by ohasd)
$GRID_HOME/bin/crsctl start res ora.crf -init

What is the reason for following alerts in cluster on all the nodes?


The ologgerd to fails to start, so the agent tries to restart ologgerd

alert.log host01
===================
2017-01-11 17:55:23.636 [OLOGGERD(14559)]CRS-8500: Oracle Clusterware OLOGGERD process is starting with operating system process ID 14559
2017-01-11 17:57:03.675 [OLOGGERD(15133)]CRS-8500: Oracle Clusterware OLOGGERD process is starting with operating system process ID 15133
2017-01-11 17:58:48.689 [OLOGGERD(15930)]CRS-8500: Oracle Clusterware OLOGGERD process is starting with operating system process ID 15930
2017-01-11 18:00:33.705 [OLOGGERD(16687)]CRS-8500: Oracle Clusterware OLOGGERD process is starting with operating system process ID 16687
The issue is caused because GIMR/mgmtdb is not created.
It's mandatory to create this database in 12.1.0.2 as it is used to store CHM log files. Ref - Doc ID 1568402.1
Please follow step "3B. For 12.1.0.2 only:" from Doc ID 1589394.1 to create mgmtdb.





Database - RAC/Scalability Community

To discuss this topic further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Database - RAC/Scalability Community


REFERENCES

NOTE:1554116.1 - Cluster Health Monitor (CHM/OS) osysmond.bin High Resource (CPU, Memory and FD etc) Usage