Delphix continuous vault

Overview

The Delphix Engine has a base feature set that is compelling as a data protection solution. An enhancement is being introduced that can further prevent snapshot and database loss in the event of a ransomware attack.

Delphix continuous vault for ransomware protection allows organizations to recover their application data access much faster than traditional backup solutions, in case of malicious attacks.

The Continuous Vault solution protects against ransomware attacks by frequently ingesting production application data (dSource). By leveraging Delphix VDB provisioning capabilities, you can instantly recover applications to a specific time before an encryption attack. (Depending on the applicable data platform and sync settings, the difference or offset between the latest production application state and Delphix dSource can be reduced to seconds).

There are two variants of Delphix Continuous Vault:

Replica Continuous Vault, which replicates critical business DB data stored on Delphix Engines to a new target Engine called Replica Continuous Vault.
Single Engine Continuous Vault, which protects critical business DB data stored on a Delphix Engine by preventing manual deletion of protected sources or snapshots.

Once securely stored on the Continuous Vault, the DB data can be used to recover business applications with very low RTO and RPO.

Requires Technical Services Consult

Delphix requires a Technical Services assessment prior to deployment and configuration of Continuous Vault. The process of configuring a Delphix Continuous Vault replication profile is simple; the assessment is required because each application has specific data protection and recovery requirements and we must ensure that Delphix can respond to them appropriately. To schedule an assessment, please contact your Customer Success Manager.

Replica continuous vault

The Replica Continuous Vault feature is available via CLI or via the Continuous Vault UI section of the Replication page. This UI provides functions for creating Continuous Vault replication profiles from scratch and converting existing profiles to be locked.

Advantages

The Replica Continuous Vault variant provides the following advantages:

Creates a separation of responsibilities between the two Delphix Engines
- One engine is used for regular Virtualization cases (ingestion, VDBs, SDD)
- Another engine is used for ransomware protection.
Creates a physical separation by allowing the admin to isolate and secure the locked Delphix Engine.
- Only the DSP port has to be open for replication.
- No ports are needed for JDBC, NFS, or SSH until VDBs need to be created to export data. This also prevents attack vectors related to any of those protocols.
Can assist with making deployments and security reviews easier to pass since the locked Delphix Engine is isolated and has a single purpose to reduce potential attack vectors.

In the event of a ransomware attack on a primary engine source being compromised or corrupted, you can provision a VDB on the replica in the locked namespace of the replication target – similar to the normal replication namespace. This process can be further outlined in the Provisioning From Replicated Data Sources or VDBs article. If a complete recovery of the primary engine is needed, please contact Delphix Support.

Implementation

This feature adds a property to the replication namespace and specifications called “locked”. Additional dSources, VDBs, groups, and domains can be added to locked replication specs, but data sources cannot be removed after doing so. Failover on the target Delphix Engine is not allowed if the namespace is part of the locked replication spec. The retention policy duration on a locked namespace can be modified as long as the duration is either being increased, or it is being decreased to a minimum of 100 days.

The time configuration on Delphix engines with Continuous Vault enabled cannot be changed. This is to prevent attempts at bypassing retention policies in order to try and delete snapshots on the target. Also, the factory reset operation is forbdiden when at least one locked replication specification or namespace is present.

A fault is now generated on the Continuous Vault target for a locked namespace that has not received a successful replication update in 12 hours. Upon request, Delphix Support can change this value. New replication specs must also have automatic replication enabled and a satisfactory replication schedule.

CLI functions

Create a locked replication profile.

CODE

[user.hostname]> replication spec
[user.hostname] replication spec> create
[user.hostname] replication spec create *> set name=locked-spec-1
[user.hostname] replication spec create *> set objectSpecification.objects=Untitled/dbname
[user.hostname] replication spec create *> set targetHost=example.delphix.com
[user.hostname] replication spec create *> set targetPrincipal=admin
[user.hostname] replication spec create *> set targetCredential.password=delphix
[user.hostname] replication spec create *> set lockedProfile=true
[user.hostname] replication spec create *> commit
    `REPLICATION_SPEC-3
[user.hostname] replication spec> select locked-spec-1
[user.hostname] replication spec 'locked-spec-1'> get
    type: ReplicationSpec
    name: locked-spec-1
    automaticReplication: false
    bandwidthLimit: 0
    description: (unset)
    encrypted: false
    lockedProfile: true <------------------------------ LOCKED
    numberOfConnections: 1
    objectSpecification:
        type: ReplicationList
        name: (unset)
        objects: Untitled/dbname
    reference: REPLICATION_SPEC-3
    runtime:
        type: ReplicationSpecRuntime
    schedule: (unset)
    tag: 5570be25-dbcf-48c3-b2d2-dd2c65eb98b7
    targetCredential:
        type: PasswordCredential
        password: ********
    targetHost: example.delphix.com
    targetPort: 8415
    targetPrincipal: admin
    useSystemSocksSetting: false
[user.hostname] replication spec 'locked-spec-1'> cd ..
[user.hostname] replication spec>

Lock an unlocked replication profile.

CODE

[user.hostname]> replication spec create
[user.hostname] replication spec create *> set name=locked-spec-2
[user.hostname] replication spec create *> set objectSpecification.objects=Untitled/dbname
[user.hostname] replication spec create *> set targetHost=example.delphix.com
[user.hostname] replication spec create *> set targetPrincipal=admin
[user.hostname] replication spec create *> set targetCredential.password=delphix
[user.hostname] replication spec create *> commit
    `REPLICATION_SPEC-4
[user.hostname]> replication spec select locked-spec-2
[user.hostname] replication spec 'locked-spec-2'> get
    type: ReplicationSpec
    name: locked-spec-2
    automaticReplication: false
    bandwidthLimit: 0
    description: (unset)
    encrypted: false
    lockedProfile: false
    numberOfConnections: 1
    objectSpecification:
        type: ReplicationList
        name: (unset)
        objects: Untitled/dbname
    reference: REPLICATION_SPEC-4
    runtime:
        type: ReplicationSpecRuntime
    schedule: (unset)
    tag: e8608d05-0693-440d-8a2b-8c6cbfe06a62
    targetCredential:
        type: PasswordCredential
        password: ********
    targetHost: example.delphix.com
    targetPort: 8415
    targetPrincipal: admin
    useSystemSocksSetting: false
[user.hostname] replication spec 'locked-spec-2'> update 
[user.hostname] replication spec 'locked-spec-2' update *> set lockedProfile=true
[user.hostname] replication spec 'locked-spec-2' update *> commit
[user.hostname] replication spec 'locked-spec-2'> get lockedProfile
    true
[user.hostname] replication spec 'locked-spec-2'>

Verify the locked status of a namespace.

CODE

[user.hostname]> namespace 
[user.hostname] namespace> list
NAME            
[user.hostname]-1
[user.hostname]-3
[user.hostname] namespace> select [user.hostname]-3
[user.hostname] namespace '[user.hostname]-3'> get 
    type: Namespace
    name: [user.hostname]-3
    description: (unset)
    failedOver: false
    locked: true   <------------------------------ LOCKED
    namespaceType: REPLICATION
    reference: NAMESPACE-4
    secureNamespace: false
    tag: 5570be25-dbcf-48c3-b2d2-dd2c65eb98b7
[user.hostname] namespace '[user.hostname]-3'>

Verify that the namespace cannot be deleted or failed over.

CODE

[user.hostname] namespace '[user.hostname]-1'> delete
[user.hostname] namespace '[user.hostname]-1' delete *> commit
   Error: Namespace "[user.hostname]-1" is locked and cannot be deleted.
  Action: Cannot delete a locked namespace.
[user.hostname] namespace '[user.hostname]-1' delete *> discard 
[user.hostname] namespace '[user.hostname]-1'> failover 
[user.hostname] namespace '[user.hostname]-1' failover *> commit
   Error: Namespace "[user.hostname]-1" is locked and cannot be failed over.
  Action: Cannot failover a locked namespace.
[user.hostname] namespace '[user.hostname]-1' failover *> discard

Verify that the replication profile cannot be deleted or modified. Objects can still be added to the replication profile.

CODE

[user.hostname]> replication spec 
[user.hostname] replication spec> select locked-spec-1
[user.hostname] replication spec 'locked-spec-1'> delete 
[user.hostname] replication spec 'locked-spec-1' delete *> commit
   Error: The replication profile is locked and cannot be deleted.
  Action: Select an unlocked profile to delete.
[user.hostname] replication spec 'locked-spec-1' delete *> discard
[user.hostname] replication spec 'locked-spec-1'> update
[user.hostname] replication spec 'locked-spec-1' update *> set automaticReplication=true
[user.hostname] replication spec 'locked-spec-1' update *> commit
   Error: The replication profile is locked and cannot be updated.
  Action: Select an unlocked profile to update.
[user.hostname] replication spec 'locked-spec-1' update *> discard
[user.hostname] replication spec 'locked-spec-1'> update
[user.hostname] replication spec 'locked-spec-1' update *> set objectSpecification.objects=Untitled/dbname,Group:/Untitled
[user.hostname] replication spec 'locked-spec-1' update *> commit
[user.hostname] replication spec 'locked-spec-1'> update
[user.hostname] replication spec 'locked-spec-1' update *> set objectSpecification.objects=Untitled/dbname
[user.hostname] replication spec 'locked-spec-1' update *> commit
   Error: Objects cannot be removed from a locked replication profile.
  Action: Select an unlocked profile to update.
[user.hostname] replication spec 'locked-spec-1' update *> discard
[user.hostname] replication spec 'locked-spec-1'>

Create a replica retention policy and apply it to the locked namespace.

CODE

[user.hostname]> policy 
[user.hostname] policy> createAndApply 
[user.hostname] policy createAndApply *> set policy.type=ReplicaRetentionPolicy 
[user.hostname] policy createAndApply *> set policy.duration=6
[user.hostname] policy createAndApply *> set policy.durationUnit=YEAR 
[user.hostname] policy createAndApply *> set target=Namespace:/[user.hostname]-1 
[user.hostname] policy createAndApply *> set policy.name="Six Years"
[user.hostname] policy createAndApply *> get
    type: PolicyCreateAndApplyParameters
    policy:
        type: ReplicaRetentionPolicy (*)
        name: Six Years (*)
        customized: false
        duration: 6 (*)
        durationUnit: YEAR (*)
    target: Namespace:/[user.hostname]-1 (*)
[user.hostname] policy createAndApply *> commit
    `POLICY_REPLICA_RETENTION-30
[user.hostname] policy>

Verify that the replica retention policy cannot be deleted or modified.

CODE

[user.hostname] policy> select POLICY_REPLICA_RETENTION-30
[user.hostname] policy 'Six Years'> delete 
[user.hostname] policy 'Six Years' delete *> commit
   Error: The replica retention policy "Six Years" could not be removed because the target namespace "[user.hostname]-1" is locked.
[user.hostname] policy 'Six Years' delete *> discard
[user.hostname] policy 'Six Years'> update 
[user.hostname] policy 'Six Years' update *> set duration=4
[user.hostname] policy 'Six Years' update *> commit
   Error: The replica retention policy "Six Years" could not be modified because the target namespace "[user.hostname]-1" is locked.
[user.hostname] policy 'Six Years' update *> discard
[user.hostname] policy 'Six Years'> unapply 
[user.hostname] policy 'Six Years' unapply *> set target=Namespace:/[user.hostname]-1
[user.hostname] policy 'Six Years' unapply *> commit
   Error: The replica retention policy "Six Years" could not be removed because the target namespace "[user.hostname]-1" is locked.
[user.hostname] policy 'Six Years' unapply *> discard
[user.hostname] policy 'Six Years'>

Single engine continuous vault

This feature is available in versions 6.0.14.0 and above. CLI and UI functions are available for locking dSources.

Advantages

The Single Engine Continuous Vault provides effective protection against ransomware attacks in a standalone Delphix Engine. This option may be preferable for deployments where maintaining two separate engines is not architecturally necessary.

Implementation

This feature adds a “locked” property to sources. Once the locked property is enabled, the source cannot be removed. Locked sources are required to have a SnapSync policy defined that refreshes data at least once daily. Furthermore, an alert is raised if no new snapshot or log data is received in the last 12 hours. Upon request, Delphix Support can change this value.

To protect Continuous Vault application data from accidental deletion or malicious attack, snapshots of locked sources may not be manually deleted. These snapshots are managed by a retention policy that must be configured for locked sources. The retention policy duration can be modified as long as retention satisfies the minimum duration (100 days).

As with the Continuous Vault Replication implementation, the time configuration of a Single Engine Continuous Vault cannot be changed. This is to prevent attempts at bypassing retention policies in order to try and delete snapshots on the Continuous Vault. Also, the factory reset operation is forbidden when at least one locked source is present.

By default, locked sources are not required to have LogSync enabled. However, upon request, Delphix Support can configure an engine-wide setting that prevents LogSync from being disabled on a locked source and, optionally, requires LogSync to be enabled from the very beginning—before locking a source.

CLI functions

Locking a source.

CODE

sedv> source
sedv source> select src10
sedv source 'src10'> lock
sedv source 'src10' lock *> commit
sedv source 'src10'>

Verifying the locked status of a source.

CODE

sedv source 'src10'> ls
Properties
    type: OracleLinkedSource
    name: src10
    container: src10
    externalFilePath: (unset)
    linked: true
    locked: true     <-------------- LOCKED
    logCollectionEnabled: false
    operations:
...

Verify that a locked source cannot be disabled.

CODE

sedv source 'src10'> disable
sedv source 'src10' disable *> commit
   Error: The source "src10" cannot be disabled because it is locked.
  Action: Contact Delphix support.
sedv source 'src10' disable *> discard

Verify that source locking requires a SnapSync policy that refreshes data at least once daily.

CODE

# An insufficiently-frequent SnapSync policy: runs at 03:00 on Sundays
sedv policy 'snapsync_weekly'> ls
Properties
    type: SyncPolicy
    name: snapsync_weekly
    customized: false
    default: false
    effectiveType: DIRECT_APPLIED
    reference: POLICY_SYNC-7
    scheduleList:
        0:
            type: Schedule
            cronString: 0 0 3 ? * 1
            cutoffTime: 14400sec
    timezone:
        type: TimeZone
        id: America/New_York
        offset: 240
        offsetString: UTC -04:00

Operations
delete
update
apply
unapply


sedv> source
sedv source> select src10
sedv source 'src10'> lock
sedv source 'src10' lock *> commit
   Error: Insufficient or unrecognized day coverage in schedule that affects locked sources.
  Action: Cover either all days of the month or all days of the week when locked sources are affected. Check the documentation for examples.

Verify that source locking requires a retention policy that retains 100 days of snapshots.

CODE

# A one-month retention policy
sedv policy 'retention_one_month'> ls
Properties
    type: RetentionPolicy
    name: retention_one_month
    customized: false
    dataDuration: 1
    dataUnit: MONTH
    dayOfMonth: 1
    dayOfWeek: MONDAY
    dayOfYear: Jan 1
    default: false
    effectiveType: DIRECT_APPLIED
    logDuration: 1
    logUnit: MONTH
    numOfDaily: 0
    numOfMonthly: 0
    numOfWeekly: 0
    numOfYearly: 0
    reference: POLICY_RETENTION-8

Operations
delete
update
apply
unapply

sedv> source
sedv source> select src10
sedv source 'src10'> lock
sedv source 'src10' lock *> commit
   Error: The retention policy is less than the minimum "100" days required when applied to locked sources.
  Action: Set retention parameters to preserve at least "100" days of data, and try again.

Verify that a snapshot from a locked source cannot be manually deleted.

CODE

sedv snapshot ''@2022-05-05T22:41:24.045Z''> delete
sedv snapshot ''@2022-05-05T22:41:24.045Z'' delete *> commit
   Error: The selected snapshot cannot be deleted because the source associated with its container "Untitled/src10" is locked.
  Action: Wait for the snapshot to be automatically deleted based on its retention policy, or Contact Delphix support.

Continuous vault alert system

In addition to the data-protection rules described in the previous sections, Continuous Vaults have a special alert system that notifies administrators about all events that can affect the ability to ingest and replicate locked data or even the ability to send such alerts.

There are two categories of Continuous Vault alerts: domain alerts and system alerts which are emailed to Engine Administrators and System administrators, respectively. To receive alerts, an administrator must have an email address configured and SMTP must be enabled.

All Continuous Vault alerts are sent also via SNMP, Syslog and Splunk/Fluentd when those services are enabled and their configured severity levels allow for each alert level.

Continuous Vault domain alerts are generated for the following events:

Locking a source, replication specification or namespace (informational level).
All actions on locked replication specifications and namespaces, such as changes in replication schedules, and all actions on locked sources and related objects that can affect them, such as changes in environment settings. All action alerts are audit-level alerts.
Deleting a Engine Administrator or changing their email address. These are warning-level alerts emailed to the addresses before the change takes effect.

Continuous Vault system alerts are generated for the following events:

Modifying, disabling or deleting services used for delivering alerts: SMTP, SNMP, Syslog and Splunk/Fluentd. These are warning-level alerts sent using the service configurations before the change takes effect.
Creating or enabling such a service (informational level).
Deleting a System administrator or changing their email address. These are warning-level alerts emailed to the addresses before the change takes effect.

By default, system administrators are allowed to change these service configurations. Continuous Vaults only notify when those changes happen. However, upon request, Delphix Support can enable locking these services such that, once an administrator enables a service, that service cannot be changed (except for subsequent changes requested to Delphix Support).