All about Databases - Oracle, PostgeSQL, Mysql, Big Data .....

Wednesday, May 15, 2019

"OPTION WARNING Database option mismatch: PDB installed version NULL" in PDB_PLUG_IN_VIOLATIONS (Doc ID 2020172.1)

In this Document

Symptoms

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Backup Service - Version N/A and later
Information in this document applies to any platform.

Symptoms

After converting a non-CDB to a PDB and running the noncdb_to_pdb.sql script, you see messages of this form in PDB_PLUG_IN_VIOLATIONS. In this case, the PDB name is "<PDB> ".

<PDB> OPTION WARNING Database option APS mismatch: PDB installed version NULL. CDB installed version 12.1.0.2.0. PENDING Fix the database option in the PDB or the CDB
<PDB> OPTION WARNING Database option CATJAVA mismatch: PDB installed version NULL. CDB installed version 12.1.0.2.0. PENDING Fix the database option in the PDB or the CDB
<PDB> OPTION WARNING Database option CONTEXT mismatch: PDB installed version NULL. CDB installed version 12.1.0.2.0. PENDING Fix the database option in the PDB or the CDB
<PDB> OPTION WARNING Database option DV mismatch: PDB installed version NULL. CDB installed version 12.1.0.2.0. PENDING Fix the database option in the PDB or the CDB
<PDB> OPTION WARNING Database option JAVAVM mismatch: PDB installed version NULL. CDB installed version 12.1.0.2.0. PENDING Fix the database option in the PDB or the CDB
<PDB> OPTION WARNING Database option OLS mismatch: PDB installed version NULL. CDB installed version 12.1.0.2.0. PENDING Fix the database option in the PDB or the CDB
<PDB> OPTION WARNING Database option ORDIM mismatch: PDB installed version NULL. CDB installed version 12.1.0.2.0. PENDING Fix the database option in the PDB or the CDB
<PDB> OPTION WARNING Database option OWM mismatch: PDB installed version NULL. CDB installed version 12.1.0.2.0. PENDING Fix the database option in the PDB or the CDB
<PDB> OPTION WARNING Database option SDO mismatch: PDB installed version NULL. CDB installed version 12.1.0.2.0. PENDING Fix the database option in the PDB or the CDB
<PDB> OPTION WARNING Database option XML mismatch: PDB installed version NULL. CDB installed version 12.1.0.2.0. PENDING Fix the database option in the PDB or the CDB
<PDB> OPTION WARNING Database option XOQ mismatch: PDB installed version NULL. CDB installed version 12.1.0.2.0. PENDING Fix the database option in the PDB or the CDB
<PDB> APEX WARNING APEX mismatch: PDB installed version NULL CDB installed version 4.2.5.00.08 PENDING Please contact Oracle Support.

Changes

You converted a non-CDB to a PDB.

Cause

These messages mean that the option is not installed in the PDB, but is installed in the CDB.

Solution

Warnings in PDB_PLUG_IN_VIOLATIONS do not prevent you from actually opening the PDB in READ WRITE mode. You can ignore WARNING messages (you cannot ignore ERROR messages). It is okay for a PDB to have a subset (fewer) options installed than the CDB into which it is plugged. (The reverse is NOT true, however -- the CDB must always have the same or more options as its PDBs).
Unpublished Bug 16192980 : NO SIMPLE WAY TO CLEAN ROWS FROM PDB_PLUG_IN_VIOLATIONS AFTER DBMS_PDB CALL has been filed to request a way to clear out no-impact warnings from PDB_PLUG_IN_VIOLATIONS. This enhancement is implemented in versions > 12.2.0.1.

Note: Do not disregard such warnings about PDB$SEED. PDB$SEED should have all the same options as the CDB. It should not be opened/modified by users. Oracle tools, such as datapatch, will open/close it as needed. You do need to address warnings of this format for PDB$SEED.

Monday, April 22, 2019

How to Complete Grid Infrastructure Configuration Assistant(Plug-in) if OUI is not Available (Doc ID 1360798.1)

In this Document

Community Discussions

References

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Information in this document applies to any platform.

Purpose

This note provides instructions to complete 11gR2/12c Grid Infrastructure configuration assistant(Plug-in) if Oracle Universal Installer session is not available anymore. OUI session can be gone for a number of reasons, i.e. node rebooted, session timed out due to security setting etc.

The note can only be executed after 11.2/12c Grid Infrastructure root script (root.sh or rootupgrade.sh) is finished successfully on all nodes.

Scope

This document is intended for Oracle Clusterware/Oracle RAC Database Administrators and Oracle support engineers.

Details

A. Pre-12.2

1. Verify whether "$GRID_HOME/cfgtoollogs/configToolAllCommands" exists or not, if not, as grid user, create a script file named configToolAllCommands with the following contents, for example:

/ocw/b202/oui/bin/runConfig.sh ORACLE_HOME=/ocw/b202 MODE=perform ACTION=configure RERUN=true $*

Note: in this example new GRID home is /ocw/b202; when creating, replace /ocw/b202 with the real new GRID_HOME path in your environment
If any configuration assistant(Plug-in) finished successfully previously and you'd like to skip it, set RERUN to false before executes it.

2. configToolAllCommands script would require a response file cfgrsp.properties if the configuration involves installing ASM(new installation), or upgrading ASM from a release earlier than 11.2.0.1.0.
The content of the file should be as follows:
(1) If installing ASM: (assuming that the ASMPASSWORD and ASMMONITORPASSWORD will be set to oracle)

oracle.assistants.asm|S_ASMPASSWORD=oracle
oracle.assistants.asm|S_ASMMONITORPASSWORD=oracle

(2) If upgrading ASM from a release earlier than 11.2.0.1.0:(assuming that the ASMMONITORPASSWORD will be set to oracle)

oracle.assistants.asm|S_ASMPASSWORD= ==>>>>leave blank
oracle.assistants.asm|S_ASMMONITORPASSWORD=oracle

3. Execute configToolAllCommands as grid user on first node:
If a response file was created in Step 2, execute configToolAllCommands as follows:

$GRID_HOME/cfgtoollogs/configToolAllCommands RESPONSE_FILE=<grid-home>/cfgtoollogs/cfgrsp.properties

Otherwise, execute:

$GRID_HOME/cfgtoollogs/configToolAllCommands

The logfile will be $GRID_HOME/cfgtoollogs/oui/configActions{TIMESTAMP}.log, it should be reviewed to find out whether there's any failures.

If on Windows platform, the syntax will be slightly different, i.e.:
<GI_HOME>\oui\bin\runconfig.bat ORACLE_HOME=<GI_HOME> ACTION=configure MODE=perform RERUN=true $*

B. 12.2

For 12.2, use the following:

gridSetup.sh -executeConfigTools -responseFile <responsefile.rsp>

For details, refer to:

A.7.1 Using the Installation Response File for Postinstallation Configuration

C. 18c

For 18c, use the following:

gridSetup.sh -executeConfigTools -responseFile <responsefile.rsp>
(Use the response file created during installation to also complete postinstallation configuration)
For details, refer to:

Using the Installation Response File for Postinstallation Configuration

Thursday, February 7, 2019

OBIEE 12c: BI Configuration Assistant Fails at Create Expanded Domain Step (Doc ID 2388947.1)

Applies to:

Business Intelligence Suite Enterprise Edition - Version 12.2.1.4.0 and later
Information in this document applies to any platform.

Symptoms

While configuring OBIEE 12.2.1.4 , The BI Configuration Assistant Fails at "Create expanded domain" step with below error stack in

java.lang.IllegalStateException: Action:BI_Configuration failed with error:Configure BI Failed with Execution of [/app/oracle/obi12214/Oracle_Home/oracle_common/common/bin/wlst.sh, /app/oracle/obi12214/Oracle_Home/bi/modules/oracle.bi.configassistant/create_base_domain.py, /app/oracle/obi12214/Oracle_Config/user_projects/domains/bi, weblogic, true, 9500, 9501, 9506, obieehost] failed with exit value 134
at oracle.as.install.engine.modules.configuration.client.ConfigAction.fail(ConfigAction.java:281)
at oracle.bi.install.config.actions.BIConfigAction.doExecute(BIConfigAction.java:137)
at oracle.as.install.engine.modules.configuration.client.ConfigAction.execute(ConfigAction.java:405)
at oracle.as.install.engine.modules.configuration.action.TaskPerformer.run(TaskPerformer.java:88)
at oracle.as.install.engine.modules.configuration.action.TaskPerformer.startConfigAction(TaskPerformer.java:108)
at oracle.as.install.engine.modules.configuration.action.ActionRequest.perform(ActionRequest.java:15)
at oracle.as.install.engine.modules.configuration.action.RequestQueue.performSequentialExecution(RequestQueue.java:284)
at oracle.as.install.engine.modules.configuration.action.RequestQueue.perform(RequestQueue.java:260)
at oracle.as.install.engine.modules.configuration.standard.StandardConfigActionManager.start(StandardConfigActionManager.java:185)
at oracle.as.install.engine.modules.configuration.boot.ConfigurationExtension.kickstart(ConfigurationExtension.java:82)
at oracle.as.install.engine.modules.configuration.ConfigurationModule.run(ConfigurationModule.java:87)
at java.lang.Thread.run(Thread.java:748)
In Config Module Finish Event...

Changes

Cause

urandom needs to be set

Solution

Go to path $ORACLE_HOME/oracle_common/common/bin
Edit the file setWlstEnv_internal.sh
Change below line

JVM_ARGS="${WLST_PROPERTIES} ${JVM_D64} ${UTILS_MEM_ARGS} ${CONFIG_JVM_ARGS}"
to

JVM_ARGS="-Djava.security.egd=file:///dev/../dev/urandom ${WLST_PROPERTIES} ${JVM_D64} ${UTILS_MEM_ARGS} ${CONFIG_JVM_ARGS}
Save the file
Restart BI Configuration

Wednesday, January 23, 2019

'DB_UNKNOWN' directory was created when using asmcmd pwcopy (Doc ID 2329386.1)

In this Document

APPLIES TO:

Oracle Database - Enterprise Edition - Version 12.2.0.1 and later
Information in this document applies to any platform.

SYMPTOMS

'DB_UNKNOWN' directory was created when using asmcmd pwcopy to copy the password file.

ASMCMD> pwcopy /tmp/pworcl +DATAP/orcl/PASSWORD/pwdorcl
copying /tmp/pworcl -> +DATAP/orcl/PASSWORD/pwdorclASMCMD> ls -l
Type Redund Striped Time Sys Name
PASSWORD UNPROT COARSE NOV 16 16:00:00 N pwdorcl => +DATAP/DB_UNKNOWN/PASSWORD/pwddb_unknown.256.960224193

CHANGES

DB resource is not present on CRS when coping password file by pwcopy.

[grid@duke1 ~]$ crsctl stat res ora.orcl.db -t
CRS-2613: Could not find resource 'ora.orcl.db'.

CAUSE

This is expected behavior.

When DB resource is not present on CRS and "--dbuniquename" option is not specified in pwcopy command,

the password file will be created within 'DB_UNKNOWN' directory as pwcopy can not get DB name.

if "--dbuniquename" was specified, the password file will be created within the DB name directory that specified by "--dbuniquename" option.

When DB resource is present on CRS, 'DB_UNKNOWN' was not observed even "--dbuniquename" option is not specified in pwcopy command

as the DB name can be got from CRS.

SOLUTION

Specifie DB name by "--dbuniquename" option when DB resource is not present on CRS.

[grid@duke1 ~]$ asmcmd
ASMCMD> pwcopy --dbuniquename orcl /tmp/pworcl +DATAP/orcl/PASSWORD/pwdorcl
PRCD-1120 : The resource for database orcl could not be found.
PRCR-1001 : Resource ora.orcl.db does not exist
ASMCMD-9457: an internally issued srvctl command failed with errors
copying /tmp/pworcl -> +DATAP/orcl/PASSWORD/pwdorcl
ASMCMD-9453: failed to register password file as a CRS resourceNote: These errors are expected due to "orcl" is not present on CRS.

ASMCMD> ls -l
Type Redund Striped Time Sys Name
PASSWORD UNPROT COARSE NOV 16 17:00:00 N pwdorcl => +DATAP/orcl/PASSWORD/pwdorcl.256.960224879
PASSWORD UNPROT COARSE NOV 16 17:00:00 Y pwdorcl.256.960224879

Copy password file by pwcopy after ran "srvctl add database" to add database to CRS.

[oracle@duke1 ~]$ srvctl add database -d orcl -o $ORACLE_HOME -n orcl -p "+DATAP/ORCL/PARAMETERFILE/spfile.268.959095489"
[oracle@duke1 ~]$ su - grid
[grid@duke1 ~]$ asmcmd
ASMCMD> pwcopy /tmp/pworcl +DATAP/orcl/PASSWORD/pwdorcl
copying /tmp/pworcl -> +DATAP/orcl/PASSWORD/pwdorcl
ASMCMD> ls -l
Type Redund Striped Time Sys Name
PASSWORD UNPROT COARSE NOV 16 17:00:00 N pwdorcl => +DATAP/orcl/PASSWORD/pwdorcl.256.960226829
PASSWORD UNPROT COARSE NOV 16 17:00:00 Y pwdorcl.256.960226829

Monday, December 3, 2018

All about HugePages for Oracle Database

4.1.5 Restrictions for HugePages and Transparent HugePages Configurations
Review the following guidelines for HugePages and Transparent HugePages. Oracle recommends that you disable Transparent HugePages, because they may
causes delays in accessing memory that can result in node restarts in Oracle RAC environments, or performance issues or delays for Oracle Database
single instances. Oracle continues to recommend using standard HugePages for Linux. Transparent HugePages memory is enabled by default with Red Hat
Enterprise Linux 6, Red Hat Enterprise Linux 7, SUSE 11, Oracle Linux 6, and Oracle Linux 7 with earlier releases of Oracle Linux Unbreakable
Enterprise Kernel 2 (UEK2) kernels.
Transparent HugePages memory is disabled by default in later releases of UEK2 and later UEK kernels.
Note: Although Transparent HugePages is disable on UEK2 and later UEK kernels, Transparent HugePages may be enabled by default on your Linux system.
Transparent Hugepages are similar to standard HugePages. However, while standard HugePages allocate memory at startup, Transparent Hugepages memory uses the
khugepaged thread in the kernel to allocate memory dynamically during runtime,using swappable HugePages.
HugePages allocates non-swappable memory for large page tables using memorymapped files. HugePages are not enabled by default. If you enable HugePages, then
you should deduct the memory allocated to HugePages from the available RAM before calculating swap space. Refer to your distribution documentation and to Oracle
Technology Network and My Oracle Support for more information.
During Oracle Grid Infrastructure installation, the Grid Infrastructure Management Repository (GIMR) is configured to use HugePages. Because the Grid Infrastructure
Management Repository database starts before all other databases installed on the cluster, if the space allocated to HugePages is insufficient,
then the System Global Area (SGA) of one or more databases may be mapped to regular pages, instead of Hugepages, which can adversely affect performance.
Configure the HugePages memory allocation to a size large enough to accommodate the sum of the SGA sizes of all the databases you intend to install on
the cluster, as well as the Grid Infrastructure Management Repository.
For detailed explanation check the references.

To check transparent HugePages
[oracle@testrac1 dbs]$ egrep 'trans|thp' /proc/vmstat
nr_anon_transparent_hugepages 0
thp_fault_alloc 0
thp_fault_fallback 0
thp_collapse_alloc 0
thp_collapse_alloc_failed 0
thp_split 0
thp_zero_page_alloc 0
thp_zero_page_alloc_failed 0

To check HugePages Size
[oracle@testrac1 ~]$ grep Hugepagesize /proc/meminfo
Hugepagesize: 2048 kB

To check HugePage
[oracle@testrac1 dbs]$ grep HugePages /proc/meminfo
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0

Add value to configuration file and apply it
[root@testrac1 ~]#vi /etc/sysctl.conf
# HugePages
vm.nr_hugepages =2864

[root@testrac1 ~]# sysctl -p

[root@testrac1 ~]# grep HugePages /proc/meminfo
AnonHugePages: 0 kB
HugePages_Total: 2864
HugePages_Free: 2864
HugePages_Rsvd: 0
HugePages_Surp: 0

References:
https://www.carajandb.com/en/blog/2016/7-easy-steps-to-configure-hugepages-for-your-oracle-database-server/
https://oracle-base.com/articles/linux/configuring-huge-pages-for-oracle-on-linux-64
https://www.thegeekdiary.com/centos-rhel-67-how-to-configure-hugepages/
https://docs.oracle.com/cd/E37670_01/E37355/html/ol_config_hugepages.html
https://docs.oracle.com/cd/E11882_01/server.112/e10839/appi_vlm.htm#UNXAR393
https://wadhahdaouehi.tn/2016/10/enable-hugepages-rhel-7ol-7-within-oracle-database-12c/
Metalinks :
https://mehmandba.blogspot.com/2018/05/hugepages-on-linux-what-it-is-and-what.html
https://mehmandba.blogspot.com/2018/05/oracle-linux-shell-script-to-calculate.html

Friday, November 30, 2018

ORA-27090 - Unable to Reserve Kernel Resources for Asynchronous Disk I/O (Doc ID 579108.1)

In this Document

Applies to:

Oracle Database - Enterprise Edition - Version 10.2.0.4 to 12.1.0.2 [Release 10.2 to 12.1]
Linux x86-64
Linux x86
***Checked for relevance on 02-Sep-2016***
Oracle Server Enterprise Edition - Version: 10.2.0.4
10.2.0.4
***Checked for relevance on 26-AUG-2014***

Symptoms

You discover messages of this nature in your alert log:

ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA

Additional information: 3
Additional information: 128
Additional information: 65536

- OR -

ORA-27090: UNABLE TO RESERVE KERNEL RESOURCES FOR ASYNCHRONOUS DISK I/O

Additional information: 3
Additional information: 128
Additional information: 65536

- OR -

ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536

Strace against DBWR shows:

io_setup(4096, 0x66e1588) = -1 EAGAIN (Resource temporarily unavailable),

which is related to "aio-max-nr" kernel limit.

Cause

The "aio-max-nr" kernel limit is too low.

Solution

The "aio-max-nr" kernel limit should be adjusted according to Oracle recommendations which are available in this document:

Oracle Validated Configuration:
http://linux.oracle.com/pls/apex/f?p=102:2:2007385298933639::NO::P2_VC_ID:426

Also, this parameter should be set as follows:
-------------------------------------
fs.aio-max-nr= 3145728
-------------------------------------

NOTE:
Per bug Bug 13554729 - CORRECT FS.AIO-MAX-NR = 3145728
the published limit in the Oracle Documentation at http://docs.oracle.com/cd/E11882_01/install.112/e24326/toc.htm#BHCCADGD is incorrect.
The Oracle documentation lists the minimum value as the max value.

Oracle Database Related Kernel Parameters

We add kernel parameters to /etc/sysctl.conf manually or automatically with preinstall before Oracle installation started. Or we can modify these parameters after installation.

[root@rac1 fs]# cat /etc/sysctl.conf

# oracle-database-server-12cR2-preinstall setting for fs.file-max is 6815744

fs.file-max = 6815744

# oracle-database-server-12cR2-preinstall setting for kernel.sem is '250 32000 100 128'

kernel.sem = 250 32000 100 256

# oracle-database-server-12cR2-preinstall setting for kernel.shmmni is 4096

kernel.shmmni = 4096

# oracle-database-server-12cR2-preinstall setting for kernel.shmall is 1073741824 on x86_64

kernel.shmall = 1073741824

# oracle-database-server-12cR2-preinstall setting for kernel.shmmax is 4398046511104 on x86_64

kernel.shmmax = 4398046511104

# oracle-database-server-12cR2-preinstall setting for kernel.panic_on_oops is 1 per Orabug 19212317

kernel.panic_on_oops = 1

# oracle-database-server-12cR2-preinstall setting for net.core.rmem_default is 262144

net.core.rmem_default = 262144

# oracle-database-server-12cR2-preinstall setting for net.core.rmem_max is 4194304

net.core.rmem_max = 4194304

# oracle-database-server-12cR2-preinstall setting for net.core.wmem_default is 262144

net.core.wmem_default = 262144

# oracle-database-server-12cR2-preinstall setting for net.core.wmem_max is 1048576

net.core.wmem_max = 1048576

# oracle-database-server-12cR2-preinstall setting for net.ipv4.conf.all.rp_filter is 2

net.ipv4.conf.all.rp_filter = 2

# oracle-database-server-12cR2-preinstall setting for net.ipv4.conf.default.rp_filter is 2

net.ipv4.conf.default.rp_filter = 2

# oracle-database-server-12cR2-preinstall setting for fs.aio-max-nr is 1048576

fs.aio-max-nr = 1048576

# oracle-database-server-12cR2-preinstall setting for net.ipv4.ip_local_port_range is 9000 65500

net.ipv4.ip_local_port_range = 9000 65500

Apply kernel parameters without server restart.

/sbin/sysctl -p

sysctl -p /etc/sysctl.conf

We can also check the status of parameters in Linux with following commands:

Example: Check aio status - current and max

[oracle@rac2 ~]$ sysctl -a | grep aio

fs.aio-max-nr = 1048576 --> This is the max number of allowable concurrent requests.

fs.aio-nr = 2088704 --> This is the actual usage

[oracle@rac2 ~]$ cat /proc/sys/fs/aio-nr

2088704

As we can see from outputs actual usage is bigger than the max limit. So we have to optimize the kernel parameter. This is related to ASM and Oracle explained in following links:

https://blog.pythian.com/importance-oracle-database-related-kernel-parameters-aio-max-nr-bonus-track/

https://www.kernel.org/doc/Documentation/sysctl/fs.txt

Error related to this parameter: ORA-27090 - Unable to Reserve Kernel Resources for Asynchronous Disk I/O