All about Databases - Oracle, PostgeSQL, Mysql, Big Data .....: June 2018

Database Upgrade Assistant (DBUA) interactively steps you through the upgrade process. configures the database for the new Oracle Database 12c release 2. It is the recommended method for performing a major release upgrade or patchset release upgrade.
DBUA automates the upgrade process by performing all of the tasks. DBUA makes appropriate recommendations for configuration options and then you can act on these recommendations.
DBUA provides support for Oracle Real Application Clusters (Oracle RAC) databases. In an Oracle RAC environment, DBUA upgrade all the database and configuration files on all nodes in the cluster.
DBUA, graphical user interface must be invoked within the new Oracle home where the Oracle Database 12c R2 software has been installed.
For windows, Only an Administrator or Installed owner should invoke DBUA for Windows systems.
DBUA starts the Pre-Upgrade Tool, which automatically fixes some configuration settings to the values required for the upgrade. For example, the Pre-Upgrade Tool can change initialization parameters to values required for the upgrade. The Pre-Upgrade Tool also provides you with a list of items that you need to fix manually before you can continue with the upgrade.
It also gives certain recommendations on certain areas belonging to the database. The recommendations can then be acted on making the upgrade process user friendly and easy.
Once, you address / fix the pre-upgrade recommendation / warnings /errors and continue with the upgrade, DBUYA shows the progress of the upgrade for each component of source database.
As with previous releases of DBUA, 12c DBUA restricts the carry over of hidden parameters since Oracle recommends not to have hidden parameters other than those suggested via support during the upgrade.
To view existing hidden parameters execute the following command while connected AS SYSDBA:
SELECT name,description from SYS.V$PARAMETER WHERE name LIKE '\_%' ESCAPE '\';
DBUA performs some of the checks before actually starting the database upgrade. Some of the checks can be done manually to reduce downtime for the upgrade.
DBUA provides below options:
- Upgrade timezone. The default timezone vetrsion in 12.2.0.1 is 26.- Gather dictionary statistics before upgrade.
- Make user tablespaces read only.
- Take RMAN backup before upgrade.
- Restore database backup to rollback upgrade
- Option to execute Custom scripts before and after upgrade
- show the location of DBUA logs and Alert log files.
- Option to upgrade existing listener to 12c home or create a new listener in 12.2 target home.
Starting with Oracle Database 12c release 2 (12.2), you can upgrade the database without disabling Oracle Database Vault. However, if you disabled Oracle Database Vault, then you must enable it manually after an upgrade.

Upgrade Path / Compatibility Matrix for 12.2 Oracle Database.

DBUA can upgrade only supported versions of direct upgrade.

Direct Upgrade to 12.2:

Source Database	Target Database
11.2.0.3 and higher	12.2.0.x
12.1.0.x (12.1.0.1 - 12.1.0.2)	12.2.0.x

Indirect Upgrade to 12.2:

Source Database	Upgrade Path	Target Database
7.3.3 (lower)	7.3.4 --> 9.2.0.8 --> 11.2.0.3 or higher	12.2.0.x
8.0.5 ( or lower )	8.0.6 --> 9.2.0.8 --> 11.2.0.3 or higher	12.2.0.x
8.1.7 ( or lower )	8.1.7 --> 9.2.0.8 --> 11.2.0.3 or higher	12.2.0.x
9.0.1.3 ( or lower )	9.0.1.3 --> 9.2.0.8 --> 11.2.0.3 or higher	12.2.0.x
9.2.0.7 ( or lower )	9.2.0.7 --> 11.2.0.3 or higher	12.2.0.x
10.2.0.4 ( or lower )	10.2.0.4 --> 11.2.0.3 or higher	12.2.0.x
11.1.0.6	11.1.0.6 --> 11.2.0.3 or higher	12.2.0.x
11.2.0.1	11.2.0.1 --> 11.2.0.3 or higher	12.2.0.x

Requirements and Recommendation for source database.

Ensure that all database components/objects provided by Oracle are VALID in the source database prior to starting the upgrade.
Before you start an upgrade or downgrade process, Oracle strongly recommends that you update both your earlier release and your new release (12.2) Oracle Database to the latest Oracle bundle patch or patch set update (BP or PSU).
Ensure that you do not have duplicate objects in the SYS and SYSTEM schema. For 1 and 2 refer to:
Doc Id 556610.1 - Script to Collect DB Upgrade/Migrate Diagnostic Information (dbupgdiag.sql)dbupgdiag.sql script is a set of sql statements intended to provide a user friendly output to diagnose the status of the database either before (or) after upgrade. The script will create a output file called db_upg_diag_<sid>_<timestamp>.log
Make sure to have a valid backup of source database prior to upgrade.
Disable any custom triggers that would get executed before / after DDL statements. Re-enable after the upgrade.
Check the database server upgrade/downgrade compatibility matrix before upgrading the database.
Set Archive Log ON during upgrade. Oracle recommends that you set Archive Log ON in order for DBUA to create and update the log file for the upgrade process.
For Oracle RAC, if you upgrade a cluster database using DBUA, then you must leave the CLUSTER_DATABASE initialization parameter set to TRUE.
Ensure to run the pre-upgrade utility prior to upgrading the database.
Examine and follow the recommendation given in the preupgrade log file.
Materialized views in source database should be stopped before upgrade
Doc ID 1406586.1 - How to Handle Materialized Views When You Upgrade or Clone a Database
Disable scheduled database custom jobs / cron jobs.

Requirements and Recommendations for Target database

Verify whether your operating system / platform is certified for 12.2 release.
Download and Install Oracle 12c Release 2 (12.2.0.1) in a new Oracle_Home and make sure there are no binary relinking errors.
Download and Install the latest available Patch Set Update (PSU) or Bundle Patch (BP) (If available) from My Oracle Support (MOS).
Make sure to set the ORACLE_HOME, PATH, LD_LIBRARY_PATH, LIBPATH etc. to 12.2 target home.

Recommendation for Oracle Spatial Component:

Execute the below query on source database prior to 12.2 upgrade:
col owner format a15
col table_name format a30
col column_name format a30
SELECT owner,table_name,column_name
FROM dba_tab_columns
WHERE data_type = 'SDO_GEOMETRY'
AND owner != 'MDSYS' ORDER BY 1,2,3;

If the above query return row(s) then below patch need to be applied on target 12.2 Oracle Home before performing the database upgrade.

Link : https://updates.oracle.com/Orion/Download/process_form/p25293022_122010_Generic.zip?file_id=95607293&aru=21422588
FileName : p25293022_122010_Generic.zip

If the patch apply will be skipped on target 12.2 home then utlrp.sql script during post upgrade phase may take more time to complete and could impact the outage.

Prerequisites for Preparing Oracle Home on Windows

Your system must meet these requirements before you can upgrade Oracle Database on Microsoft Windows platforms. For security reasons, different Microsoft Windows user accounts configured as Oracle
home users for different Oracle homes are not allowed to share the same Oracle Base.

Database upgrade is supported when the same Windows user account is used as the Oracle home user in both the source and destination Oracle homes.
Database upgrade is supported when the Oracle home from which the database is being upgraded uses the Windows Built-in Account. Releases earlier than Oracle Database 12c (release 11.2 and earlier) only supported the built-in account option for the Oracle home user on Windows.
The Oracle home user may not have access to files outside its own Oracle Base and Oracle home. If that is the case, then if you choose a different Oracle Base during upgrade, it is possible that Oracle Database services cannot access files in the older Oracle Base. Using DBUA for database upgrade ensures that the Oracle home user has access to files outside of its own Oracle Base and its own Oracle home.

Pre-Upgrade

$Earlier_release_Oracle_home/jdk/bin/java -jar $New_release_Oracle_home/rdbms/admin/preupgrade.jar [FILE|TERMINAL] [TEXT|XML] [DIR output_dir]

FILE|TERMINAL - Use this option to direct script output to a file. Use TERMINAL to direct output to the terminal. If it is not specified then defaultr is FILE.
TEXT - Use this option to specify log should be in Text format. Use TEXT to specify text output. Use XML to specify XML output. If you do not specify an output type, then the default is text.
DIR - Logs will be created under <output_dir>. Directs the output to a specific directory. If you do not specify an output directory with the DIR option, then the output is directed to one of the default locations: If you define ORACLE_BASE environment variable then the generated scripts and log files will be created under $ORACLE_BASE/cfgtoollogs/<dbname>/preupgrade/ location else it will create under $ORACLE_HOME/cfgtoollogs/db_name/preupgrade/.

For example,

source Oracle Home : /u01/app/oracle/product/12.1.0.1/dbgome_1

target Oracle Home : /u01/app/oracle/product/12.2.0.1/dbhome_1

$ export ORACLE_SID=orcl
$ export ORACLE_BASE=/u01/app/oracle
$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.1/dbhome_1

$ $ORACLE_HOME/jdk/bin/java -jar /u01/app/oracle/product/12.2.0.1/dbhome_1/rdbms/admin/preupgrade.jar FILE TEXT
Preupgrade generated files:
/u01/app/oracle/cfgtoollogs/orcl/preupgrade/preupgrade.log
/u01/app/oracle/cfgtoollogs/orcl/preupgrade/preupgrade_fixups.sql
/u01/app/oracle/cfgtoollogs/orcl/preupgrade/postupgrade_fixups.sql
$

Examine the preupgrade.log file and follow the recommendation.

The latest preupgrade utility for 12.2 can be found from :
How to Download and Run Oracle's Database Pre-Upgrade Utility (Doc ID 884522.1)

Check for Invalid Objects / Components:

set pagesize500
set linesize 100
select substr(comp_name,1,40) comp_name, status, substr(version,1,10) version from dba_registry order by comp_name;

select substr(object_name,1,40) object_name,substr(owner,1,15) owner,object_type from dba_objects where status='INVALID' order by owner,object_type;

select owner,object_type,count(*) from dba_objects where status='INVALID' group by owner,object_type order by owner,object_type ;

If you find invalid objects and/or database components then try to VALIDATE the invalid objects and/or database components by executing the following steps:

Run $ORACLE_HOME/rdbms/admin/utlrp.sql to validate the invalid objects in the database. You can execute the utlrp.sql scripts multiple times to validate the invalid objects.

$ sqlplus "/ as sysdba"
SQL> @$ORACLE_HOME/rdbms/admin/utlrp.sql

Gathering Optimizer Statistics to Decrease Oracle Database Downtime

Oracle strongly recommends that you use this procedure to gather statistics before performing Oracle Database upgrades.Oracle recommends that you use the DBMS_STATS.GATHER_DICTIONARY_STATS procedure to gather these statistics. For example, enter the following SQL statement:

SQL> EXEC DBMS_STATS.GATHER_DICTIONARY_STATS;

Verifying Materialized View Refreshes are Complete Before Upgrade

Use this procedure to query the system to determine if there are any materialized view refreshes still in progress. Before upgrading Oracle Database, you must wait until all materialized views have
completed refreshing.

SQL> SELECT o.name FROM sys.obj$ o, sys.user$ u, sys.sum$ s WHERE o.type# = 42 AND bitand(s.mflags, 8) =8;

How to Handle Materialized Views When You Upgrade or Clone a Database (Doc ID 1406586.1)

Check of TIMESTAMP WITH TIMEZONE Datatype

The time zone files that are supplied with Oracle Database 12c release 2 (12.2) is version 26.

Case 1 Timezone version of source database is lower or equal 26.

If the source database is using a timezone file lower than version 26 then there is no DST patch to apply in source oracle home or target 12cR2 home.

Case 2 Timezone version of source database is higher than 26.

If the source database uses a Timezone version higher than 26 then BEFORE the upgrade you MUST patch the target 12cR2 $ORACLE_HOME with a timezone data file of the SAME version as the one used in the source release database.

Ensuring That No Files Are in Backup Mode and no files need media recovery Before Upgrading

Execute below query to check for the status of the backup:

SQL> SELECT * FROM v$backup WHERE status != 'NOT ACTIVE';

Ensure that no files require media recovery:

SQL> SELECT * FROM v$recover_file;

Purging Recycle Bin before upgrade

 SQL> PURGE DBA_RECYCLEBIN;

we can purge the recycle bin via DBUA.

Starting with Oracle Database 12c release 2 (12.2), you can upgrade the database without disabling Oracle Database Vault.

Starting with Oracle Database 12c release 2 (12.2), you can upgrade the database without disabling Oracle Database Vault. However, if you disabled Oracle Database Vault, then you must enable it manually after an upgrade.
Starting with Oracle Database 12c release 2 (12.2), if you have Oracle Database Vault enabled, then you can upgrade the Oracle Database without first disabling Oracle Database Vault. After the upgrade, Oracle Database Vault is enabled by default with the enforcement settings that you had in place before the upgrade. If you manually disable Oracle Database Vault before the upgrade, then you must enable Oracle Database Vault manually after the upgrade.

Enable Oracle Database Vault in the upgraded database by using the procedure dvsys.dbms_macadm.enable_dv(). Run this procedure with a user account that is granted either DV_OWNER or DV_ADMIN. After you run the procedure, restart the database instance so that the procedure takes effect.

Requirements for Upgrading Databases That Use Oracle Label Security and Oracle Database Vault (For Oracle Database releases earlier than 12.1)

You must complete these tasks before starting an upgrade with a database using Oracle Label Security or Oracle Database Vault.

Audit Table Preupgrade

If you are upgrading from a database earlier than Oracle Database release 12.1 that uses Oracle Label Security (OLS) and Oracle Database Vault, then you must first run the OLS preprocess script, olspreupgrade.sql, to process the aud$ table contents. The OLS upgrade moves the aud$ table from the SYSTEM schema to the SYS schema. The olspreupgrade.sql script is a preprocessing script required for this move.

Running olspreupgrade.sql on Oracle Database Release 11.2

If Oracle Label Security is installed in the earlier release that you are upgrading, then you must run the OLS preprocess olspreupgrade.sql script.

If Oracle Database Vault is not installed with your release 11.2 database, then you can skip steps 2, 3, 6, and 7 in this section.

To run the OLS preprocess script on a release 11.2 database before upgrading:

1. Copy the ORACLE_HOME/rdbms/admin/olspreupgrade.sql script from the newly installed Oracle home to the Oracle home of the database that you want to upgrade.
2. Start SQL*Plus and connect as DVOWNER to the database that you want to upgrade.
3. Run the following statement:
SQL> GRANT DV_PATCH_ADMIN to SYS;
4. At the system prompt, connect SYS as SYSDBA:
CONNECT SYS AS SYSDBA
5. Run the OLS preprocess script:
ORACLE_HOME/rdbms/admin/olspreupgrade.sql
You may continue to run your applications on the database while the OLS preprocess script is running.
6. After the olspreupgrade.sql completes its run successfully, start SQL*Plus and connect to the database as DVOWNER.
7. Run the following SQL statement:
SQL> REVOKE DV_PATCH_ADMIN from SYS;

Save OEM DB Control Configuration (For Oracle Database releases earlier than 12.1)

If you plan to downgrade database after upgrading to 12.2, then before starting the upgrade save the DB Control configuration and data using emdwgrd utility, so that you can restore the files after downgrading.

Steps to Save data:

     1. Install the software for the new Oracle Database 12c release.
     2. Set ORACLE_HOME to your old Oracle home.
     3. Set ORACLE_SID to the SID of the database being upgraded.
     4. Set PATH, LD_LIBRARY_PATH, and SHLIB_PATH to point to the upgraded Oracle Database Oracle home.
     5. Change directory to the new Oracle Database release Oracle home.
     6. Run emdwgrd

Single-instance databases:

emdwgrd[sh|bat] -save -sid old_SID -path save_directory

Oracle Real Application Clusters (Oracle RAC) databases:

Remote copy must be enabled across all cluster member nodes. Use EM_REMCP environment variable, to indicate which remote copy is configured, for example: export EM_REMCP /usr/bin/scp

emdwgrd -save -cluster -sid old_SID -path save_directory

7. Enter the SYS password for the database that you want to upgrade.

Removing DB Control with emremove.sql (For Oracle Database releases earlier than 12.1)

Warning :
The below step should not be executed during the upgrade of Enterprise Manager Cloud Control Repository database. Doing so your EM Cloud Control Repository DB will become unusable.

Starting with Oracle Database 12c release 1 (12.1), DB Control is removed as part of the upgrade process.

Stop DB Console:

emctl stop dbconsole

Execute emremove.sql script. The script willbe located in target 12.2 OH/rdbms/admin/.

SQL>SET ECHO ON
SQL>SET SERVEROUTPUT ON
SQL>@emremove.sql >> Script located in new 12c ORACLE_HOME/rdbms/admin

After emremove.sql completes, you must manually remove ORACLE_HOME/HOSTNAME_SID and ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_HOSTNAME_SID directories from your file system.

Desupport of OLAP Catalog (OLAP AMD) (For Oracle Database releases earlier than 12.1)

Starting with Oracle Database 12c, the OLAP Catalog (OLAP AMD) is desupported and will be removed from the database during the database upgrade. This step can be manually performed before the upgrade to reduce downtime. It can be removed by running the SQL script $ORACLE_HOME/olap/admin/catnoamd.sql script. Do not run the script in UPGRADE mode.

Drop JSON-Enabled Context search Indexes (Only for the upgrade of 12.1 DB to 12.2)

If you are upgrading from Oracle Database 12c release 1 (12.1) to 12c release 2 (12.2), then Oracle recommends that you drop the JSON-enabled context index before upgrading.
Oracle recommends that you drop your existing JSON-enabled index.

If you created a JSON search index using Oracle Database 12c Release 1 (12.1.0.2) then Oracle recommends that you drop that index and create a new search index for use with later releases, using CREATE SEARCH INDEX statement. Please refer "Oracle Database JSON Developer’s Guide" for more details.

Copying Transparent Encryption Oracle Wallets

If you use Oracle wallet with Transparent Data Encryption (TDE), and you use Database Upgrade Assistant (DBUA) to upgrade the database, then copy thesqlnet.ora and wallet file to the new 12.2 Oracle home.

You must copy the sqlnet.ora and the wallet file manually before starting the upgrade.
1. Log in as an authorized user.
2. Manually copy the sqlnet.ora file, and the wallet file, ewallet.p12, to the new release Oracle home.
3. Open the Oracle wallet in mount.

For example:
SQL> STARTUP MOUNT;
SQL> ALTER SYSTEM SET ENCRYPTION WALLET OPEN;

Check the accounts use Case-Insensitive password version

Log in to SQL*Plus as an administrative user, and enter the following SQL query
SELECT USERNAME,PASSWORD_VERSIONS FROM DBA_USERS;

If there are any 10g versions, suggest you to refer Oracle documentation to fix 10g versions, failing to do so, user accounts with LOCKED after upgrade is completed.

About Password Case Sensitivity

Ensure that you do not have the deprecated parameter SEC_CASE_SENSITIVE_LOGON set to FALSE.

Requirement for Upgrading Databases that Use Oracle Warehouse Builder (OWB)

OWB releases earlier than release 11.2.0.3 do not work with Oracle Database 12c. OWB is not installed as part of the software for Oracle Database 12c, and OWB components that may exist in earlier releases are not upgraded as part of the Oracle Database upgrade process. However, you can use OWB release 11.2.0.3 with Oracle Database 12c.

Invoke DBUA

run DBUA, once all Prerequisite checks are successful. please make sure that environment variables are pointing to target 12.2 home.

unset the ORACLE_PATH environment variable, if it is already set.

DBUA (Step 1 of 9)

we need to choose the database SID for upgrade to 12.2.0.1. In this case, it is orcl database.

DBUA (step 2of 9)

In below screen, DBUA will perform various pre-checks. It will execute the preupgrade script (preupgrade.jar) and return warnings / errors.

DBUA (Step 3 of 9)

once the pre-upgrade warnings has been adressed, next screen will show various options like Enable Parallel Upgrade, Recompile invalid objects in post upgrade phase, upgrading timezone, gather statistics before the upgrade, setting the user tablespaces to read only during upgrade. also, you can provide any SQL scripts before or after the upgrade.

DBUA (Step 4 of 9)

After selecting option, The next screen will show various Recovery options where it will create Guaranteed Restore Point, or Take an RMAN backup or the backup is already taken outside DBUA then "I have my own backup and restore stratege " option can be chosen.

DBUA (Step 5 of 9)

in this screen, we can create a new listener or upgrade the existing listener to 12.2.0.1.

DBUA (Step 6 of 9)

This screen is for configure EM express or register the upgraded database with EM Cloud control.

DBUA (step 7 of 9)

This is the summary screen before the actual upgrade starts. Click on Finish to proceed with the upgrade.

DBUA (Step 8 of 9)

The DBUA will start the upgrade process of orcl database. It will perform pre-upgrade, database upgrade and post upgrade steps.

DBUA will perform the upgrade of database components like Oracle server, XML etc.

DBUA Step 9 of 9)

The final screen will show the result of upgrade.

It will show the log file location and the time taken by various steps of upgrade.

Post-Upgrade

Execute dbupgdiag.sql script to verify status of objects and components. If there are invalid objects then run utlrp.sql to recompile the invalid objects as follows:

sql> connect / as sysdba

sql> @?/rdbms/admin/utlrp.sql

Known Issues

Below are the known issues:

Doc ID 2279575.1 - Upgrade to 12.2.0.1 Fails With Error "ORA-00001: Unique Constraint (SYS.I_DAM_LAST_ARCH_TS$) Violated"

Datapatch: Database 12c Post Patch SQL Automation (Doc ID 1585822.1)

In this Document

Datapatch in a Multitenant Environment:

Changes for DB release 12.1.0.2

APPLIES TO:

Oracle Database - Standard Edition - Version 12.1.0.1 and later
Oracle Database - Enterprise Edition - Version 12.1.0.1 and later
Oracle Database - Personal Edition - Version 12.1.0.1 and later
Information in this document applies to any platform.

PURPOSE

Explain Database 12c Post Patch SQL Automation

SCOPE

Database release 12c extends patch install automation for patches that contain post-patch SQL instructions.
Prior to Oracle 12c such patches required manual intervention to complete the post-patch SQL instructions after restarting the database.
Datapatch is the new tool that enables automation of post-patch SQL actions for RDBMS patches.
Datapatch resides in the opatch directory i.e $ORACLE_HOME/OPatch folder (On Windows platforms : %ORACLE_HOME%\OPatch).

Datapatch can be executed to complete post-patch SQL actions after the database is restarted following patch application.
For patches that do not have post-patch SQL actions to be performed, calling datapatch is a no-op.
For patches that do have post-patch SQL instructions to be invoked on the database instance, datapatch will automatically detect ALL pending actions (from one installed patch or multiple installed patches) and complete the actions as appropriate.

Further details on datapatch are provided towards the end of this document.
Enterprise Manager and OPatchAuto provide further automation of database patches by calling datapatch automatically after applying the binary patch.

Following section describes how various patching tools implement this automation flow.

DETAILS

Enterprise Manager:

Starting version 12.1 Enterprise Manager now calls datapatch to complete post patch actions upon any 12c or later database restart.
As noted above, datapatch contains the logic to identify if any post-patch SQL actions are pending.

OPatchAuto :

OPatchAuto calls datapatch to complete post patch actions upon installation of the binary patch and restart of the database.
As noted earlier, datapatch identifies the post install instructions which are necessary and automatically completes the same.
Post patch activity include both the application and the removal or rollback of SQL changes in the database.

OPatch :

Datapatch integration with OPatch is not possible as OPatch is executed when the database is down and
datapatch requires the database to be opened to complete its activity.
When patches are installed or rollbacked using OPatch then datapatch needs to be explicitly invoked if instructed to do so in the patch readme.
Datapatch is designed to be idempotent and can be run for all patches and
it is recommended that datapatch be run post any patch apply or rollback activity when OPatch is used.
(In computing, an idempotent operation is one that has no additional effect if it is called more than once with the same input parameters.)

Datapatch :

Datapatch is the patching tool to complete the post-patch SQL actions for RDBMS patches.

Windows specific : On Windows platforms the script name is datapatch.bat.

RAC specific: For a RAC environment, after the binary patch has been applied on all nodes run Datapatch to complete the post-install SQL deployment for the PSU only from one node. Datapatch need not be run on all the nodes.

Datapatch determines the requisite apply/rollback actions by matching an internal repository with the patch inventory.
Datapatch should be invoked when the database is restarted after a patching session.
Enterprise Manager and OPatchAuto call datapatch automatically during every patching session.
If OPatch is used to install RDBMS patches then datapatch has to be explictly called to complete any patching actions after database restart.
Datapatch will automatically determine the set of patches that need to be installed and a set of patches that need to be rolled back.

Datapatch ensures that a patch has been installed/rolled back in all RAC instances before to initiate any post-patch SQL actions on the database.
In Oracle Multitenant environment datapatch will patch the root and any pluggable databases that are opened.
In order to patch all pluggable databases, it should be ensured that before to invoke datapatch all pluggable databases are opened.
If an unpatched pluggable database is opened in Oracle Multitenant then calling datapatch will complete the pending patch actions.

Datapatch usage :

All arguments are optional, if there are no arguments then datapatch will automatically determine which SQL scripts need to
be run in order to complete the installation of any patches that contain post-patch SQL instructions.

Optional arguments:

-db <sid>
Use the specified database's SID rather than $ORACLE_SID

-apply <patch1,patch2,...,patchn>
Only consider the specified patch list for apply operations

-rollback <patch1,patch2,...,patchn>
Only consider the specified patch list for rollback operations

-force
Run the apply and/or rollback scripts even if not necessary per the SQL registry

-prereq
Run prerequisite checks only, do not actually run any scripts

-oh <oracle_home value>
Use the specified directory to determine what patches are installed

-verbose
Output additional information used for debugging

-help
Output usage information and exit

-version
Output build information and exit

Datapatch in a Multitenant Environment:

When datapatch is run to invoke any SQL actions in a Multitenant environment, datapatch only applies the changes to the ROOT, SEED and any open PDBs based on the pending actions. For any PDB that is not currently open, the SQL patch will not be applied. When such a PDB is reopened, it will be opened in restricted mode as there will be a patch level mismatch between the PDB and the ROOT.

Call DBMS_PDB.CHECK_PLUG_COMPATIBILITY on 12.1.0.1. PDB xml file will return these two violations, which are expected.
SQL Patch ERROR

SQL patch bug # mismatch: Installed in the PDB but not in the CDB. Install the SQL patch in the PDB or the CDB

This error can be cleared by running datapatch again and closing and reopening the affected PDB(s)

1) Invoke datapatch:

% cd $ORACLE_HOME/OPatch
% datapatch

2) Close and reopen the affected PDB

Connect as SYS
alter pluggable database <PDB Name> close instances =all;
alter pluggable database <PDB Name> open read write instances =all;

3) Check again for plugin violations (by querying pdb_plug_in_violations in cdb$root), there should be none and the pluggable database should be opened in normal mode

Changes for DB release 12.1.0.2

Miscellaneous

Catbundle Integration: Datapatch now assumes the role of catbundle when applying bundles/PSU. As a result of this change catbundle.sql has been deprecated and patch registry is maintained only in registry$sqlpatch. Application of the PSU does not update registry$history table anymore. The status for patch application is now entirely maintained in registry$sqlpatch
Support for patch UID: Datapatch now uses patch-ID and patch-UID to uniquely identify a patch. With this change, datapatch now distinguishes between different versions of a single patch. This allows datapatch to handle the case of re-issued patches correctly.
Support for –force –rollback all: Datapatch now provides the option to rollback all patches from a given container DB using the –force option. This is useful for unplug/plug of container databases
Integration with upgrade/downgrade scripts
Improved error checking for PLS errors and warnings.

Prerequisite Checks

In 12.1.0.2 Datapatch now has pre-req checks for a number of conditions. These pre-req checks prevent patching and have the following reasons and potential issues:

Upgrade mode check:

If a patch requires upgrade mode startup, datapatch enforces that via a pre-req check. To overcome this error, DB/PDB needs to be re-started in Upgrade/Migrate mode to complete patching. Please note that when there are a number of simultaneous patch applications pending, datapatch will fail the upgrade mode pre-req check if any of the pending patches requires upgrade mode startup. The database being started in Upgrade mode will have no impact on other patches being applied which do not require Upgrade mode.

Alternatively, any patches not requiring upgrade mode can be installed by invoking datapatch with the –apply < bug # > option
Check for existence of Apply/Rollback scripts:

Datapatch now ensures that apply/rollback scripts are present for any pending patch apply/rollback. If the files are not available datapatch will fail with a pre-req error. For out of place rollback please copy any sqlpatch files to the $OH in order to overcome this check and allow rollback to proceed
Queryable Inventory connectivity :

Datapatch flags a Queryable inventory error if for some reason Queryable Inventory returns an error status. Further details can be derived by calling select dbms_qopatch.get_pending_activity from dual and noting the error message. Further details on resolving Queryable Inventory issues can be found at Note 1530108.1, "Oracle Database 12.1 : FAQ on Queryable Patch Inventory"
DB connectivity: datapatch has a pre-req check to ensure DB connectivity.
Pre-req to ensure we can create directories under cfgtoollogs dir.