dSource operations

Overview

In order to use dSource snapshots, you must create a virtual database (VDB), an independent, writable copy of a dSource snapshot.

When integrating an environment into the Continuous Data Engine, the system automatically locates and identifies any data sources within the host, listing them as dSources under the Databases tab specific for each environment. This section outlines the management of dSources and the synchronization of data from your sources, involving operations and concepts such as:

Link/Sync: Establishing a connection with a data source.
Timeflows: Describes the timeline of data of a virtual database or dSource.
Snapshots: Captured states of a dSource at specific moments.
SnapSync and LogSync: Processes to capture and synchronize data changes.
Enable/Disable: Activating or deactivating a dSource.
Detach/Reattach: Temporarily disconnecting and then reconnecting a dSource.
Delete: Removing a dSource.
Upgrade: Updating a dSource to a new version.

Adding an environment configuration

In the Delphix Continuous Data Engine, environments are composed of repositories, which correspond with your respective database management software (DBMS) installations. An environment can be linked with an unlimited number of these repositories.

To establish a dSource, you must define its configuration with a source environment configuration. Repositories serve as a foundation for creating this kind of object, each of which corresponds to a recognized database instance within the system.

For connectors, it is necessary to set up the environment configuration manually.

Steps to create an environment configuration in the staging environment

Access the Delphix Management application by logging in.
Navigate to the Manage section.
Go to Environments and select the appropriate repository.
Click the plus (+) icon to initiate a new configuration.
In the Add database area, fill in the necessary details.
Provide a name for your dSource in the Name field.

Setting up an environment configuration using the Command Line Interface (CLI)

To configure SourceConfig via CLI, follow these instructions:

Securely log into your Delphix Continuous Data Engine using the admin account with SSH:

CODE

ssh admin@<CONTINUOUS-DATA-FQDN-or-IP>

Execute the following commands:

CODE

sourceconfig
create
set type=<AppDataStagedSourceConfig>
set databaseName=<dbname>
set repository="<repository name>"
set parameters="{\"name\":\"<source config name>\"}"
commit

This process will create a SourceConfig for your dSource using the specified repository details.

Disabling a dSource

To ensure certain operations can be carried out, a dSource may need to be disabled. Disabling halts communication between the Delphix Continuous Data Engine and the source database without removing the underlying configuration.

Steps to disable a dSource

Log in to the Delphix Management application.
Select the dSource you wish to disable.
In the upper right-hand corner, click the Actions menu (...) and choose Disable.
In the confirmation dialog, click Disable again.

Disabling is a prerequisite for operations such as:

Database migration
dSource metadata upgrading post-data source upgrade
Restoring the source database from backup

Disabling will cease any ongoing Delphix Continuous Data Engine operations related to the dSource. When you are ready to re-enable the dSource, simply select Enable from the Actions menu (...), and it will resume normal function.

Force disabling a dSource

There might be circumstances that require a dSource to be forcefully disabled, stopping all related Delphix Continuous Data Engine operations immediately.

Steps to force disable a dSource

Access the Delphix Management application.
Choose the dSource in question.
Click on the Actions menu (...) in the upper right corner and opt for Disable.
Select the Force Disable checkbox in the dialog that appears.

Detach and reattach a dSource

In certain scenarios, you may need to detach a dSource from its source database and later reattach it, possibly to a different source. This process is essential for maintaining the flexibility of your data management within the Delphix Continuous Data Engine.

Understanding dSource detachment

Detaching a dSource breaks its link with the current source database but preserves the existing data and snapshots within the Delphix Continuous Data Engine. Here are some items to note about detached dSources:

Continued use for VDBs: A detached dSource can still provision a Virtual Database (VDB).
Re-linking capability: You can re-link the original source database as a new dSource.
Child VDB refresh limitations: VDBs that were created from the detached dSource can only refresh from the last snapshot taken before detachment.
Provisioning from new snapshots: If a newer snapshot is needed, you will have to provision a new VDB. Subsequently, you may delete the outdated VDBs and the old dSource when they are no longer required.

Steps to detach a dSource

To detach a dSource from its source database:

Log into the Delphix Management application.
Select the dSource you wish to detach.
Click on the Actions menu (...) and choose Unlink dSource.
A warning message will pop up—confirm the action by clicking Unlink.
The status of the dSource will update to Detached.

Steps to reattach a dSource

When ready to re-establish a connection between a dSource and a source database:

Go back to the Delphix Management application.
From the Actions menu (...) next to the detached dSource, select Link dSource.
Follow the prompts to complete the re-linking process.

Remember, reattaching a dSource re-establishes its connection with a source database, allowing it to resume its role in data management operations according to your configurations.

Delete a dSource

Deleting a dSource will remove its metadata, snapshots, logs, and policies from the Delphix Continuous Data Engine. This action does not affect the source database.

Prerequisites for deleting a dSource

There must be no dependent virtual databases (VDBs) or virtual Pluggable databases (vPDBs). Ensure all dependent databases are deleted as per instructions in the Deleting virtual databases under Provisioning and managing virtual databases.

Steps to delete a dSource

Log in to the Delphix Management application.
Select the dSource you wish to delete.
From the Actions menu (...), select Delete.
Click Delete to confirm the action.

SnapSync a dSource

SnapSync is the process that captures the complete dataset on the initial load and only the incremental changes thereafter, creating efficient storage snapshots.

Understanding SnapSync

The initial SnapSync pulls the complete data set using standard database protocols.
Subsequent SnapSync operations fetch only incremental changes.
Each SnapSync operation results in a snapshot, serving as a baseline for VDB provisioning.

Resync a dSource

Resync is a process similar to the initial SnapSync but is executed at intervals to ensure the dSource is updated with the complete data set from the source database.

Prerequisites for Resync

The data directory under the mount path must be empty before starting a Resync operation.

Steps to Resync a dSource

If needed, ensure the data directory is empty to avoid errors.
The Resync will capture the complete data set, akin to the initial SnapSync process.

LogSync a dSource

LogSync complements SnapSync by periodically capturing log files from the source database, allowing for more granular recovery points between SnapSync snapshots.

LogSync details

Log files are transferred using standard database protocols and stored separately from SnapSync data.
These logs facilitate the provisioning of VDBs from points in time between SnapSync snapshots.
The time taken to provision from these intermediate points is proportional to the volume of data change on the source database and the time elapsed since the last snapshot.

Upgrading a dSource

When the source database connected to a dSource is upgraded, it becomes necessary to update the dSource's metadata to match this new version. This section guides you through upgrading the dSource to align with the updated database.

The upgrade of a dSource is a crucial operation that ensures the Delphix Continuous Data Engine recognizes and correctly associates with the updated version of the source database. Note that this process does not upgrade the database itself; it merely updates the dSource metadata.

Prerequisites for upgrading a dSource

Before you begin the upgrade process:

Confirm that the source database instance has been upgraded to the newer version.
Ensure that the dSource is disabled. This is a mandatory step to proceed with the upgrade.

Upgrade procedure

The upgrade process will involve modifying the dSource to use the installation path of the new database version. The general steps are as follows:

Disable the dSource. This step is crucial and must be completed before you start the upgrade process. For details on disabling a dSource, refer to the Disabling a dSource section above.
Once the dSource is disabled, you can proceed with the upgrade according to the specific database you are working with.

Database-specific upgrade instructions

Each database has its own set of instructions for upgrading a dSource. Refer to the respective connector documentation for detailed procedures: