Linking a dSource from a Db2 database: An overview

This topic describes basic concepts behind the creation of dSources from Db2 instances.

Database level operation

Associated with an instance is the concept of an instance owner. This is the user who "owns" that instance and has SYSADM authority over the instance and all databases inside that instance. SYSADM authority is the highest level of authority in Db2 and lets this user perform several database management activities such as upgrade, restore, and editing configurations. More information about instances is located in the IBM knowledge center

Db2 database level support

Delphix operates at the database level and requires that

• The staging and target hosts must have the empty instances created prior to Delphix using them

• The desired OS user to execute commands related to dSource and VDB operation for each instance has been added as an environment user

Note:

• Make sure the staging instance used for linking doesn’t have an existing restoring database with the same name. 

• The dSource ingestion or VDB provisioning should not be performed on the source instance to maintain the integrity of the source database.

Data Ingestion

Db2 for Delphix ingests data by using a staging database created on a discovered Standby instance of Db2. The dSource uses the staging database to stay in sync with the production database. This is done by first going through the linking process during which a full backup is used to recover an initial copy of the production database to the staging database. Storage for the staging database is provided by Delphix via an NFS mount to storage exported by the Delphix system.

A single standby instance can contain data from multiple source databases.

Pipelining logic for implementing parallel restores

The plugin employs a parallel pipeline methodology so that the restore operation of non-catalog partitions can be performed in parallel in the Database Partitioning Feature (DPF). The number of parallel restores is determined by the value of “restorePipelineLimit” (default value is 10) in <Toolkit directory>/advanceConfFlag.conf. For more details on this please refer to Requirements for Db2 Hosts and Databases. The parameter “restorePipelineLimit” is configurable by end-users. The plugin performs parallel restore for all non-catalog partitions. For e.g. If the total number of non-catalog partitions is 15, and the "restorePipelineLimit" parameter is set to 10, the first set of 10 restores will happen in parallel. The plugin will track the restore of each partition present in the pipe. Whenever a restore of a partition completes, it will move out from the pipe and a new partition will enter into that pipe. Thus, the plugin ensures that the pipe will always have the user-configured number of partitions being restored (default=10). 

In case of a restore failure for a particular partition, the plugin will track that failure and will print the same in the plugin-generated logs so that the user can take the necessary action. If the user received failure during restore operation for a few partitions and then decided to manually initiate the restores for failed partitions, the plugin will verify the datapaths during snapshot operation. In case it finds wrong path information (such as paths outside the mount point), then the plugin will error out of the operation.

The plugin will track the status of each restore in a file named <DB Name>_metadata.json The plugin can handle a scenario where the staging host reboots in-between the parallel restores. In this case, the user can perform a “Resynchronisation” operation again. At this time, the remaining restores will resume from scratch. For more details on this please refer to Requirements for Db2 Hosts and Databases

Data synchronization

Delphix provides the following options to keep your dSource in sync with the source database:

1. During the linking process, you can set up a HADR connection between the original source database and the Standby instance. This allows the Standby instance using HADR for log shipping to always keep its databases in sync with the source. It is important to note that a single Standby instance (dSource) can contain multiple databases from multiple different servers and instances as long as each database has a unique name.

2. Users can opt to configure SSL with the HADR connection to enhance security by encrypting data-in-transit between the primary and the standby databases.

3. The users can opt for the "Staging Push" feature by selecting the checkbox "Use Staging Push". With this feature, the user can manage the restore and roll forward of a staging database without involving the Delphix Engine.

4. Users can opt for a non-HADR method of database ingestion. The database will be ingested to a standalone staging host and the user can trigger a resynchronization operation with a most recent online full backup to keep the staging database up to date with the source database.

5. Users can opt for the Delphix Recovery with the Customer Supplied Archive Logs method. During an update, the dSource user can trigger a snapshot operation where user-provided archive logs will get applied to the staging database.

6. Optionally, when the DPF feature is in use, users can use the “Customer Supplied Archive Logs” feature. Users need to place archive logs inside a folder with a name as NODE<Partition number> where <Partition number> is a four-digit number. For example, if the source environment has two partitions then the user-provided log path will have folder names NODE0000 and NODE0001. Both the folders will have respective archive logs. Snapshot operation will be used to apply the archive logs.

Delphix HADR standby configuration may not meet Db2 DR requirements and should only be used for Delphix use cases.