Oracle RAC dSource node addition or deletion

Managing Oracle RAC node changes in the Delphix Continuous Data Engine

The addition and removal of cluster nodes from an Oracle cluster must be considered where Delphix Continuous Data Engine has dSources attached to RAC databases running from the same cluster environment or RAC VDB's provisioned out to those cluster nodes. Each scenario needs to be considered in its own right as dSources and VDBs require specific changes be made to accommodate the removal of a cluster node.

The changes required by Delphix Continuous Data Engine and made within Oracle that need to be considered when the Oracle cluster node configuration alters and the process for removing or adding an Oracle cluster node is best demonstrated through an example. 

The oracle cluster is comprised of 2 cluster nodes as seen from RAC nodes.

[oracle@rac1338dd541 ~]$ su - grid
Password: 
Last login: Thu Nov 30 17:15:36 AEDT 2023 on pts/1

[grid@rac1338dd541 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid

[grid@rac1338dd541 ~]$  olsnodes -s -t
rac1338dd541 Active  Unpinned
rac1338dd542 Active  Unpinned

Node 2 in the cluster rac1338dd542 is the node targeted for removal from the Oracle cluster.

Delphix Continuous Data Engine dSources and removing Oracle cluster nodes

In this example, Delphix Continuous Data Engine has 2 Nodes, Oracle RAC database is ingested as a dSource. The sample database name is DBOMSRB3 comprised of 2 RAC instances DBOMSRB331B31 and DBOMSRB331B32. The instance impacted by the removal of node rac1338dd542 is DBOMSRB331B32. The dSource configuration appears as follows:

The cluster node, its associated listeners and instances must be deleted from the cluster through Oracle RAC RDBMS and Clusterware software using the processes described in the Oracle documentation.

This begins with the removal of the RAC instance using the process detailed in the Oracle RAC documentation. This example is operating against an Oracle 19.0 RAC and Cluster and the documentation link for this specific release is the following:

Removing RAC instances and homes

https://docs.oracle.com/database/121/RACAD/GUID-CE995C7B-B420-4A4E-B4EA-E7A972277588.htm#RACAD7903

Removal of the RAC instance is typically executed through the database configuration assistant dbca and using the UI provided through dbca the instance DBOMSRB331B32 is removed from node rac1338dd542. After the instance has been deleted using dbca, Oracles srvctl utility will show one instance only (DBOMSRB331B31) running from node rac1338dd541.

[oracle@rac1338dd541 ~]$ srvctl config database -d DBOMSRB331B3
Database unique name: DBOMSRB331B3
Database name: DBOMSRB3
Oracle home: /u01/app/oracle/product/19.0.0.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/DBOMSRB331B3/PARAMETERFILE/spfile.304.1150593529
Password file: +DATA/DBOMSRB331B3/PASSWORD/pwddbomsrb331b3.292.1150592865
Domain: 
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: 
Disk Groups: REDO,DATA
Mount point paths: 
Services: 
Type: RAC
Start concurrency: 
Stop concurrency: 
OSDBA group: dba
OSOPER group: oper
Database instances: DBOMSRB331B31
Configured nodes: rac1338dd541
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services: 
Database is administrator managed

Following the removal of the instance, the inventory on the node to be removed is updated to reflect the removal of the Oracle Home that was running this dSource instance on the node. This too must be performed according to the Oracle documentation. Oracles utility runInstaller is used for Oracle inventory maintenance. This must be performed using the operating system owner of the RDBMS Home (in this case oracle)

With the RAC instance and RAC Home removed the node itself can be removed using the process detailed in Oracles Clusterware Administration documentation,

Removing Oracle cluster nodes

https://docs.oracle.com/database/121/CWADD/GUID-8ADA9667-EC27-4EF9-9F34-C8F65A757F2A.htm#CWADD90992

This must be performed as the Oracle Clusterware home operating system owner (in this case grid).

Removal of the node from within Oracles RDBMS and Clusterware software does not alter the fact that Delphix still sees the Oracle Cluster and RAC configuration as it was the last time an environment refresh was performed against the cluster.

The Environment in Delphix shows both cluster nodes are still present along with any node listeners that may have been discovered running on that node.

The dSource shows the node and both instances still present this needs to be cleaned up from a Delphix perspective.

Aligning Delphix Continuous Data Engine with the new Oracle cluster configuration

In order for Delphix Continuous Data Engine to pick up on the latest changes to the environment (rac1338dd540) in this case, the removal of an entire node from the Oracle Cluster and an environment refresh must be performed.

This is performed through the environments management panel in the Delphix Management application and the refresh icon present on the top right-hand side of the panel (highlighted in red)

Clicking on the icon presents a prompt indicating that an environment refresh is about to be initiated.

Click on refresh to kick off the environment refresh action.

The environment refresh will force Delphix to re-examine the Oracle Cluster.

This will result in the environment and the dSource being updated based on the cluster's current node, instance and listener configuration where only one node rac1338dd541 is present.

After the node removal, the cluster environment in Delphix will look as follows and only rac1338dd541 will appear in the cluster node list.

The dSource will now comprise only one instance DBOMSRB331B31 is running from node rac1338dd541.

Adding Oracle cluster nodes

You can add a node through the Environments management panel in the Delphix Management application and the Refresh icon is present on the top right-hand side of the panel. Clicking on the icon presents a prompt indicating that an environment refresh is about to be initiated.

Click on the Refresh option to kick off the environment refresh action. The environment refresh will force Delphix to re-examine the Oracle Cluster. This will result in the environment and the dSource being updated based on the cluster's current node, instance, and listener configuration.

Delphix configuration steps to assign a VDB to a new cluster instance

1. Login to the Delphix Management application. 

2. In the Oracle RDBMS, add the required number of nodes using the process detailed in the Oracle RAC documentation

3. In the Delphix Management Engine, select the VDB you want to edit.

4. From the right-side pane select the Configuration tab. 

5. Refresh the VDB to view new nodes.

6. In the upper right-hand corner, from the Actions menu (...) select Disable.

   

7. In the Disable dialog select Disable.

8. Click the pencil icon next to Cluster Instances to be able to add new nodes.

   

9. In the Include in Cluster column, check or uncheck the check-box to the left to add or remove a node for your RAC VDB.

   

10. Complete editing and then confirm using the tick icon.

11. From the Action menu (...), select Enable.