Managing data sources and syncing data

A dSource is a virtualized representation of a physical or logical source database, which cannot be accessed using database tools. In order to use dSource snapshots, you must create a virtual database (VDB), an independent, writable copy of a dSource snapshot.

Once an environment has been added, Delphix will automatically discover any data sources on the host. These appear as dSources on the Databases page of each environment. This section will describe how to manage these dSources and sync data from your sources. The following operations and concepts exist for dSources:

• Link/Sync

• Timeflows

• Snapshots

• SnapSync and LogSync

• Enable/Disable

• Detach/Reattach

• Delete

• Upgrade

Linking dSources

Once you have discovered your data source environments, you will see a list of data sources on that host. The first step to sync data is to link a dSource from the host.

Linking a dSource will ingest data from the source and create a dSource object on the engine. The dSource is an object that the Delphix Virtualization Engine uses to create and update virtual copies of your database. As a virtualized representation of your source data, it cannot be managed, manipulated, or examined by database tools. 

Once a dSource has been linked, you can begin to perform dSource operations on it, such as enable, detach, delete, and more.

The dSource timeflow

Both dSources and VDBs have Timeflows, which are a collection of snapshots for a particular data set. Virtual databases can be provisioned from any snapshots in a Timeflow. In the UI, a Timeflow is represented vertically with a series of individual snapshots as rows.

The dSource snapshot

Snapshots of dSources preserve specific points in time for use later during the virtual database provisioning workflow. Taking a snapshot will create a new snapshot entry in the dSource’s Timeflow. 

SnapSync and LogSync

This dSource Timeflow described above is maintained through the use of SnapSync and LogSync.

SnapSync

SnapSync will pull over the complete data set during the initial load using standard database protocols. 

Subsequent SnapSync operations will pull only the incremental changes and store them with optimal storage efficiency. At the end of each SnapSync operation, a snapshot is created that serves as the base point for provisioning operations.

When provisioning a VDB, the closer the origin point is to a snapshot created via SnapSync, the sooner the provisioning operation will complete. 

LogSync

In addition to SnapSync, LogSync will periodically connect to the host(s) running the source database via standard protocols and pull over any log files associated with the database. These log files are stored separately from the SnapSync data and are used to provision from points in between SnapSync snapshots.

When provisioning from a point between snapshots, the additional time it takes to provision is directly proportional to the time difference between the provision point and the last snapshot. The rate of change on the source database dictates the amount of data that must be replayed to bring a virtual database to the correct point in time.

Enabling and disabling dSources

Some operations, such as dSource upgrade, are not available unless the dSource is disabled. Disabling a dSource turns off communication between the Delphix Engine and the source database, but it does not remove the configuration that enables communication and updates data. When a disabled dSource is later enabled, it will resume communication and incremental data updates from the source database according to the original policies and data management configurations that you set.

Disabling a dSource is also a prerequisite for several other operations, such as: 

• database migration

• upgrading the dSource metadata after the upgrade of the associated data source

• restoring the source database from a backup

Disabling a dSource will stop further operations on the Delphix Engine related to the dSource.

1. Login to the Delphix Management application.

2. Select the dSource you want to disable.

3. In the upper right-hand corner, from the Actions menu (...) select Disable.

4. In the Disable dialog, select Disable.

When you are ready to enable the dSource again, from the Actions menu (...) select Enable, and the dSource will continue to function as it did previously.

Detaching and reattaching dSources

Each dSource contains metadata that associates it with the source database, as well as the data it has ingested from the source database in the form of snapshots up to that point.

You may want to detach a dSource from a data source and reattach it to another source. It is possible to detach, or unlink, a dSource from its source database. This breaks the association with the source database without affecting the data within the Delphix Engine. 

Detached dSources and their source databases have these properties:

• A detached dSources can still be used to provision a virtual database (VDB).

• You can re-link the source database as a different dSource.

• Any child VDBs that were provisioned from this dSource will only be able to be refreshed from the most recent snapshot available on the dSource.

• If you need a VDB from a newer snapshot, you will need to provision a new VDB. Once you have provisioned the new VDB you can delete the old VDBs provisioned from this dSource. You can delete the old dSource when it is no longer needed.

Detaching a dSource

1. Login to the Delphix Management application.

2. Select the dSource you want to detach.

3. From the Actions menu (...), select Unlink dSource.
   A warning message will appear. 

4. Click Unlink to confirm. The status of the dSource will show as Detached.

When you are ready to attach the dSource again, from the Actions menu (...) select Link dSource, and the dSource.

Deleting dSources

Deleting a dSource will delete the dSource metadata for a particular source database, along with all associated snapshots, logs, and policies stored in Delphix. If a dSource is deleted, it will not affect your source database. 

Prerequisites

You cannot delete a dSource that has dependent virtual databases (VDBs). Before deleting a dSource, ensure that you have deleted all dependent VDBs as described in Deleting a VDB  on Provisioning and Managing Virtual Databases. Otherwise, the delete option will be grayed out.

For dSources, you cannot delete a dSource that has dependent virtual databases (VDBs) or virtual Pluggable databases (vPDBs). Before deleting a dSource, ensure that you have deleted all dependent VDBs or vPDBs. Otherwise, the delete option will be grayed out. When you delete all the dSources of PDBs, the associated CDBs are automatically deleted.

Procedure

1. Login to the Delphix Management application.

2. Select the dSource you want to delete.

3. From the Actions menu (...), select Delete.

4. Click Delete to confirm.

Upgrading dSources

Occasionally, you may upgrade the source database that a dSource is associated with. In this case, you will need to update the dSource’s metadata to associate a dSource with a more recent database version, thus performing an ‘upgrade’ of the dSource’s known version. 

While this feature does not upgrade the actual database, it is important that Delphix associates this dSource with the correct version of the source database to function properly. 

This process of upgrading a dSource will modify the database installation path to use the path of the newer database installation.

The process for the upgrade of dSources will vary depending on the database. Refer to the following pages for database-specific procedures:

• Upgrading dSources after an Oracle Upgrade

• Upgrading a dSource after a SQL Server Upgrade

• Upgrading a SAP ASE dSource

Prerequisites

• The data source instance has been upgraded to a more recent installation version.

• The dSource must be disabled.