Oracle RAC dSource node addition or deletion

Managing Oracle RAC node changes in Delphix

The addition and removal of cluster nodes from an Oracle Cluster must be considered where Delphix has dSources attached to RAC databases running from the same cluster environment or RAC VDB's provisioned out to those cluster nodes. Each scenario needs to be considered in its own right as dSources and VDBs require specific changes be made to accommodate the removal of a cluster node.

The changes required by Delphix and made within Oracle that need to be considered when the Oracle cluster node configuration alters and the process for removing or adding an Oracle cluster node is best demonstrated through an example.

The oracle cluster is comprised of 2 cluster nodes as seen from olsnodes.

CODE


[oracle@oelc9n1 ~]$ su - grid
Password: 
Last login: Thu Nov 30 17:15:36 AEDT 2017 on pts/1

[grid@oelc9n1 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid

[grid@oelc9n1 ~]$  olsnodes -s -t
oelc9n1 Active  Unpinned
oelc9n2 Active  Unpinned

Node 2 in the cluster oelc9n2 is the node targeted for removal from the Oracle cluster.

Delphix dSources and removing Oracle cluster nodes

In this example, Delphix has a 2 Node Oracle RAC database being ingested as a dSource. The sample database name is db121 comprised of 2 RAC instances db1211 and db1212. The instance impacted by the removal of node oelc1n2 is db1212. The dSource configuration appears as follows:

The cluster node, its associated listeners and instances must be deleted from the cluster through Oracles RAC RDBMS and Clusterware software using the processes described in the Oracle documentation.

This begins with the removal of the RAC instance using the process detailed in the Oracle RAC documentation. This example is operating against an Oracle 12.1 RAC and Cluster and the documentation link this specific release is the following:

Removing RAC instances and homes

https://docs.oracle.com/database/121/RACAD/GUID-CE995C7B-B420-4A4E-B4EA-E7A972277588.htm#RACAD7903

Removal of the RAC instance is typically executed through the database configuration assistant dbca and using the UI provided through dbca the instance db1212 is removed from node oelc9n2. After the instance has been deleted using dbca, Oracles srvctl utility will show one instance only (db1211) running from node oelc9n1.

CODE


[oracle@oelc9n1 ~]$ srvctl config database -d db121

Database unique name: db121
Database name: db121
Oracle home: /u01/app/oracle/12.1
Oracle user: oracle
Spfile: +DATA/DB121/PARAMETERFILE/spfile.289.961441955
Password file: +DATA/DB121/PASSWORD/pwddb121.277.961441737
Domain: 
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: 
Disk Groups: FRA,DATA
Mount point paths: 
Services: 
Type: RAC
Start concurrency: 
Stop concurrency: 
OSDBA group: dba
OSOPER group: 
Database instances: db1211
Configured nodes: oelc9n1
Database is administrator managed

Following the removal of the instance, the inventory on the node to be removed is updated to reflect the removal of the Oracle Home that was running this dSource instance on the node. This too must be performed according to the Oracle documentation. Oracles utility runInstaller is used for Oracle inventory maintenance. This must be performed using the operating system owner of the RDBMS Home (in this case oracle)

With the RAC instance and RAC Home removed the node itself can be removed using the process detailed in Oracles Clusterware Administration documentation,

Removing Oracle cluster nodes

https://docs.oracle.com/database/121/CWADD/GUID-8ADA9667-EC27-4EF9-9F34-C8F65A757F2A.htm#CWADD90992

This must be performed as the Oracle Clusterware home operating system owner (in this case grid).

Removal of the node from within Oracles RDBMS and Clusterware software does not alter the fact that Delphix still sees the Oracle Cluster and RAC configuration as it was the last time an environment refresh was performed against the cluster.

The Environment in Delphix shows both cluster nodes are still present along with any node listeners that may have been discovered running on that node.

The dSource shows the node and both instances still present this needs to be cleaned up from a Delphix perspective.

Aligning Delphix with the new Oracle cluster configuration

In order for Delphix to pick up on the latest changes to the environment (OELC9) in this case, the removal of an entire node from the Oracle Cluster an environment refresh must be performed.

This is performed through the environments management panel in the Delphix Management application and the refresh icon present on the top right-hand side of the panel (highlighted in red).

Clicking on the icon presents a prompt indicating that an environment refresh is about to be initiated.

Click on refresh to kick off the environment refresh action.

The environment refresh will force Delphix to re-examine the Oracle Cluster.

This will result in the environment and the dSource being updated based on the cluster's current node, instance and listener configuration where only one node oelc9n1 is present.

After the node removal, the cluster environment in Delphix will look as follows and only oelc9n1 appears in the cluster node list.

The dSource will now be comprised of only one instance db1211 running from node oelc9n1.

Adding Oracle cluster nodes

You can add a node through the Environments management panel in the Delphix Management application and the Refresh icon present on the top right-hand side of the panel. Clicking on the icon presents a prompt indicating that an environment refresh is about to be initiated.

Click on the Refresh option to kick off the environment refresh action. The environment refresh will force Delphix to re-examine the Oracle Cluster. This will result in the environment and the dSource being updated based on the cluster's current node, instance, and listener configuration.

Delphix configuration steps to assign a VDB to a new cluster instance

Login to the Delphix Management application.
In the Oracle RDBMS, add the required number of nodes using the process detailed in the Oracle RAC documentation
In the Delphix Management Engine, select the VDB you want to edit.
From the right-side pane select the Configuration tab.
Refresh the VDB to view new nodes.
In the upper right-hand corner, from the Actions menu (...) select Disable.
In the Disable dialog select Disable.
Click the pencil icon next to Cluster Instances to be able to add new nodes.
In the Include in Cluster column, check or uncheck the check-box to the left to add or remove a node for your RAC VDB.
Complete editing and then confirm using the tick icon.
From the Action menu (...), select Enable.