ORA-27090 - Unable to Reserve Kernel Resources for Asynchronous Disk I/O (Doc ID 579108.1)

In this Document Symptoms Cause Solution References Applies to: Oracle Database - Enterprise Edition - Version 10.2.0.4 to 12.1.0.2 [Release 10.2 to 12.1] Linux x86-64 Linux x86 ***Checked for relevance on 02-Sep-2016*** Oracle Server Enterprise Edition - Version: 10.2.0.4 10.2.0.4 ***Checked for relevance on 26-AUG-2014*** Symptoms You discover messages of this nature in your alert log: ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA Additional information: 3 Additional information: 128 Additional information: 65536 - OR - ORA-27090: UNABLE TO RESERVE KERNEL RESOURCES FOR ASYNCHRONOUS DISK I/O Additional information: 3 Additional information: 128 Additional information: 65536 - OR - ORA-27090: Message 27090 not found;  product=RDBMS; facility=ORA Linux-x86_64 Error: 4: Interrupted system call Additional information: 3 Additional information: 128 Additional information: 65536 Strace against DBWR shows: io_setup(4096, 0x66e1588) = -1 EAGAIN (Resource temporarily unavailable), which is related to "aio-max-nr" kernel limit. Cause The "aio-max-nr" kernel limit is too low. Solution The "aio-max-nr" kernel limit should be adjusted according to Oracle recommendations which are available in this document: Oracle Validated Configuration: http://linux.oracle.com/pls/apex/f?p=102:2:2007385298933639::NO::P2_VC_ID:426 Also, this parameter should be set as follows: ------------------------------------- fs.aio-max-nr= 3145728 ------------------------------------- NOTE: Per bug Bug 13554729 - CORRECT FS.AIO-MAX-NR = 3145728 the published limit in the Oracle Documentation at http://docs.oracle.com/cd/E11882_01/install.112/e24326/toc.htm#BHCCADGD is incorrect. The Oracle documentation lists the minimum value as the max value.

Oracle Database Related Kernel Parameters

We add kernel parameters to /etc/sysctl.conf  manually or automatically with preinstall before Oracle installation started. Or we can modify these parameters after installation.

[root@rac1 fs]# cat /etc/sysctl.conf 

# oracle-database-server-12cR2-preinstall setting for fs.file-max is 6815744

fs.file-max = 6815744

# oracle-database-server-12cR2-preinstall setting for kernel.sem is '250 32000 100 128'

kernel.sem = 250 32000 100 256

# oracle-database-server-12cR2-preinstall setting for kernel.shmmni is 4096

kernel.shmmni = 4096

# oracle-database-server-12cR2-preinstall setting for kernel.shmall is 1073741824 on x86_64

kernel.shmall = 1073741824

# oracle-database-server-12cR2-preinstall setting for kernel.shmmax is 4398046511104 on x86_64

kernel.shmmax = 4398046511104

# oracle-database-server-12cR2-preinstall setting for kernel.panic_on_oops is 1 per Orabug 19212317

kernel.panic_on_oops = 1

# oracle-database-server-12cR2-preinstall setting for net.core.rmem_default is 262144

net.core.rmem_default = 262144

# oracle-database-server-12cR2-preinstall setting for net.core.rmem_max is 4194304

net.core.rmem_max = 4194304

# oracle-database-server-12cR2-preinstall setting for net.core.wmem_default is 262144

net.core.wmem_default = 262144

# oracle-database-server-12cR2-preinstall setting for net.core.wmem_max is 1048576

net.core.wmem_max = 1048576

# oracle-database-server-12cR2-preinstall setting for net.ipv4.conf.all.rp_filter is 2

net.ipv4.conf.all.rp_filter = 2

# oracle-database-server-12cR2-preinstall setting for net.ipv4.conf.default.rp_filter is 2

net.ipv4.conf.default.rp_filter = 2

# oracle-database-server-12cR2-preinstall setting for fs.aio-max-nr is 1048576

fs.aio-max-nr = 1048576

# oracle-database-server-12cR2-preinstall setting for net.ipv4.ip_local_port_range is 9000 65500

net.ipv4.ip_local_port_range = 9000 65500

Apply kernel parameters without server restart.

/sbin/sysctl -p

or

sysctl -p /etc/sysctl.conf

We can also check the status of parameters in Linux with following commands:

Example: Check aio status - current and max

[oracle@rac2 ~]$ sysctl -a | grep aio

fs.aio-max-nr = 1048576         --> This is the max number of allowable concurrent requests.

fs.aio-nr = 2088704                 --> This is the actual usage

[oracle@rac2 ~]$ cat /proc/sys/fs/aio-nr

2088704

As we can see from outputs actual usage is bigger than the max limit. So we have to optimize the kernel parameter. This is related to ASM and Oracle explained in following links:

https://blog.pythian.com/importance-oracle-database-related-kernel-parameters-aio-max-nr-bonus-track/

https://www.kernel.org/doc/Documentation/sysctl/fs.txt

 Error related to this parameter: ORA-27090 - Unable to Reserve Kernel Resources for Asynchronous Disk I/O