IBM Db2 staging push

The staging push transfers the responsibility of preparing the staging database to the end user. This ingestion method allows you to manually push the source data into the mount path backed by the Delphix Continuous Data Engine storage. It offers an alternative approach to the staging pull model where the Delphix Continuous Data Engine takes care of preparing the staging database.

Backups for staging push ingestion

Although the method used for pushing the source data into the mount point can be completely your choice, there are a few limitations when it comes to the type of backups used for the ingestion. The backups will have to be online backups since Delphix Continuous Data Engine relies on the archive logs to rollforward the database to specific timestamps and offline backups can be created by databases running circular logging rather than an archive logging mechanism.

The staging push ingestion method has been verified with the following mechanisms for both DPF and non-DPF databases:

• Backups Only with rollforward to MRT (Minimum Recovery Time)

• Backups with Archive logs (Rollforward to PIT)

• HADR (Only for non-DPF databases)

The first snapshot created will be an empty directory and doesn’t contain any database and hence a VDB cannot be provisioned using this VDB.

Prerequisites

Ensure that you meet the following requirements before you begin creating a dSource.

• The environment requirements are met by the source and staging environments as described Requirements for staging/target environments.

• To use privilege elevation, you must meet the Privilege elevation requirements.

For more information, refer to Requirements and Prerequisites. 

Limitation

• You cannot rollforward the staging database to the end of logs.

Procedure

Perform the following steps to create a dSource using the staging push ingestion method.

1. Login to the Delphix Management application.

2. Navigate to the Manage > Environments page.

3. On the Databases tab, choose a repository for your dSource and click the plus icon.

4. In the Add Database dialog window, provide the name of your source configuration and click Add.

5. Select your source configuration and click the Add dSource link located on the right.

6. In the Source tab, do the following:

   1. Provide the database name.

   2. Select the Use Staging Push checkbox. To use the Database partition feature along with staging push also check the Use DPF checkbox.

   3. Click Next. 

7. In the dSource Configuration tab, provide a dSource name and then select a group for your dSource. Adding a dSource to a dataset group allows you to manage user permissions for that database and its objects, including snapshots. For more information, refer to the topics under Users and Groups.

8. In the Data Management tab, specify your staging environment, user, and snapshot parameters. Select the Resynchronize dSource checkbox if you want to resynchronize the dSource. This operation will force a non-incremental load of data from the source. This operation is similar to creating a new dSource but prevents duplication of the storage requirements and maintains timeflow history. Click Next to proceed.

9. In the Policies tab, apply policy details to the dSource if needed, and then click Next to continue.

10. In the Hooks tab, select a Hook Point and then click + to add a script that you want to run at that hook point. You have the flexibility to define scripts to run at multiple hook points in the process.

11. In the Summary tab, review the configuration profile for your dSource.

12. Click Submit. The Continuous Data Engine will initiate two jobs, namely DB_Link and DB_Sync to create the dSource. You can monitor the progress of these jobs on the Running Actions section on the top right panel or by navigating to System > Jobs. Upon successful completion of the jobs, the database icon will change to a dSource icon on the Environments > Host > Databases screen. The dSource will also be listed under Datasets within its assigned group.

13. Look in connector logs generated at <toolkit path>/Delphix_COMMON_<long id>/plugin/DB2_18f4ff11-b758-4bf2-9a37-719a22f5a4b8/logs/<instance_name>/<database_name>.diag.log and follow the steps mentioned under Steps to restore and rollforward the staging database to restore and rollforward the database and bring the database to a consistent state.

14. Once the database has been restored and rollforwarded to a specific point in time, take a snapshot of the dSource as explained here.

Staging push with HADR

Staging push ingestion can also be used along with HADR to make sure the staging database is always in sync with the source database. The advantage of this method is that the user doesn’t have to provide the path to archive logs and run rollforward commands manually. Instead, the staging database starts syncing with the source database once it is connected to the source database by following the steps provided in here.

Prerequisites

Ensure that you meet the following requirements before you begin creating a dSource.

• The environment requirements are met by the source and staging environments as described Requirements for staging/target environments.

• To use privilege elevation, you must meet the Privilege elevation requirements. 

For more information, refer to Requirements and prerequisites. 

Limitations

• HADR configuration should be set at source and staging manually

• You need to stop HADR on the staging database before performing any operation on the dsource.

Procedure

1. Create staging push dSource as described in Procedure above.

2. Set up HADR configuration at source and staging as mentioned here.

3. Start HADR on source followed by staging:

   1. For Source: db2 start hadr on db <DB_NAME> as primary by force.

   2. For staging: db2 start hadr on db <DB_NAME> as standby.

4. Once the logsync is complete or satisfactory between the source and staging or when the customer is ready to take a snapshot, HADR needs to be stopped in standby by running:

   1. db2 deactivate db <DB_NAME>

   2. db2 stop hadr on db <DB_NAME>

The logsync between primary (production) and staging can be verified by running db2pd -hadr on the staging host and looking for the parameter “HADR_CONNECT_STATUS” and “HADR_LOG_GAP”.

5. Take a snapshot of the the dSource by following the steps here.