Policies for scheduled jobs

Introduction

Creating policies is a great way to automate the management of datasets in Delphix. Whether for syncing data, taking snapshots, or refreshing virtual databases, policies ensure that data is ready when needed.

In order to create policies, navigate to the ‘Manage’ dropdown and select ‘Policies’.

There are five categories of policies that the Delphix Engine uses in conjunction with datasets objects:

• SnapSync – How often snapshots of a source database are taken for a dSource.

• VDB snapshot – How often snapshots are taken of the virtual database (VDB).

• Retention – How long snapshots and log files are retained for dSources and VDBs.

• VDB refresh – Automatic refresh of a VDB, either with the most recent snapshot or latest Timeflow logs. The default setting for this policy is None.

• Replica retention – How long snapshots are retained on replicated namespaces for dSources and VDBs after they have been deleted on the replication source. Retention of snapshot applies as long as the database is not deleted on the source. If the database is deleted, then the next replication update will delete the database and the retained snapshots on the replication target.

Avoid adding conflicting policies

The following example case can occur if the Snapshot policy is set to NONE, this has a direct impact on the retention policy and potential VDB data growth: An engine encountered a domain0 space issue that resulted from the archive logs on a single VDB taking up 25% of the pool. That engine was set to use a ‘None’ Snapshot policy along with a 1-week log retention policy. Because the archive logs were needed to provision from the one and only snapshot that existed, the engine was unable to enforce the retention policy.

There can be default or custom policies for each of these categories.

Setting different policies for objects in a group

Policies applied at the group level will affect all objects in that group. To set different policies for objects in a group, apply the policies at the group level first, then apply policies at the object level.

SnapSync Policies

SnapSync policies determine how often snapshots of the source database are taken. 

In the Default SnapSync policy, a snapshot is taken daily at 3:30 AM local time and will cancel if not completed within four hours. If SnapSync does not complete within this four hour period, it will resume at the next scheduled daily time until the process is complete. 

Click the 'Edit' icon to change the Default SnapSync policy, or click the 'Add' icon to create a new SnapSync policy.

Policies may be configured using the provided Schedule date picker, by Interval, or by using a Quartz cron expression when selecting Custom.

Retention policy

A Retention Policy defines how long the Delphix Engine retains snapshots and log files, which are used to rewind or provision objects from past points in time. The retention time for snapshots must be equal to or longer than the retention time for logs.

To support longer retention times, more storage may need to be allocated to the Delphix Engine. The retention policy – in combination with the SnapSync policy – can have a significant impact on the performance and storage consumption of the Delphix Engine. Retention policies can be customized to retain snapshots and logs for longer periods, enabling usage to specific points further back in time.

Replica retention policy

Replica Retention policy defines how long the snapshots are retained on replicated namespaces for dSources and VDBs, after they have been deleted on the replication source.

Normally, the snapshots that have been deleted on the replication source engine are also deleted on the replication target engine. A new retention policy is introduced to provide an extended lifetime of such snapshots on the replication target. The Replica Retention Policy is targetted and defined on the replication target. This policy can be applied to an entire replica namespace or could target specific groups or dSource/VDBs within the replica namespace.

Use the replica snapshots on the replication target to provision or refresh VDBs. Point-in-time provisioning may not be possible for these snapshots, this is done to optimize the disk space usage on the replication target.

The replica retention policy can only be used to extend (not to reduce) the lifetime of the replica snapshots.

Extended replica snapshots can be deleted in the following cases:

• Similar to replicated snapshots, Extended replica snapshots cannot be deleted manually. They can only be removed by adjusting the policy duration so the snapshot is no longer covered.

• If the entire dSource or VDB is deleted, the extended replica snapshot will be deleted on the replication target as well.

• Oracle virtual PDBs require their CDB to be present and, if the CDB is deleted or is not replicated, then the extended replica snapshots on both the deleted CDB as well as the dependent virtual PDB will be deleted on the replication target. This can happen in a rare scenario where an Oracle virtual PDB is migrated from one CDB to another. Once the virtual PDBs snapshots that reference the old CDB are deleted, the old CDB can also be deleted. The snapshots on the virtual PDB that depend on the old CDB will not be covered by extended replica retention on the replication target.

The replica retention policy runs automatically on a schedule to cleanup expiring snapshots or whenever a replication receive job is executed.

Benefits of longer retention

With increased retention time for snapshots and logs, a longer (older) rollback period for data is allowed. 

Common use cases for longer retention include:

• SOX compliance

• Frequent application changes and development

• Caution and controlled progression of data

• Reduction of project risk

• Speed of rollback or restoring to older points in time