Repave Delphix Engine

Overview

As the rate of new security vulnerabilities rise, Delphix continues developing solutions to mitigate potential risks. Repave is an added solution applied to a Delphix Engine to get rid of any potentially compromising complications. It transfers datasets and required metadata to a fresh, new target engine, without carrying over any compromised binaries. The target engine will function identically to the source engine.

Prerequisites

Repave is supported on Continuous Data (virtualization) engine with block storage as of version 13.0.0.0 and Continuous Compliance (masking) engine with block storage as of version 16.0.0.0.
You would need to enable the feature flag REPAVE before using it on engine version 13.0. After 14.0, the feature flag is enabled by default.
The target engine shall be in the same version as the source engine.

How to Repave a Delphix Engine

A sysadmin initiates repave/prepare on the source engine. All Delphix datasets are disabled and the Delphix management service is shut down.
A sysadmin detaches storage devices from the Delphix engine and attaches them to a new Delphix engine. This can be done through cloud APIs or manual operation.
A sysadmin calls repave/apply on the new engine. All previously disabled data sources are enabled again.

Repave API calls

POST repave/prepare initiates the repave preparation on the source engine
POST repave/apply starts the process of applying repave on the target engine.
GET repave returns the repave state, configurable metadata, and engine summary of the current engine.
POST repave/preview provides a preview of the configurable metadata and engine summary of the source engine, as well as determines its eligibility for applying repave on the target engine.

The representation below is a general API workflow of repave.

You might need to call APIs Establish Session, Login and Enable Feature Flag before calling repave APIs.

Tips and usage guidelines

repave/prepare and repave/apply returns a job id. However, when the Delphix management service is down, the job API will not work. The other way to check the repave status is by calling GET repave.
All running sources on the engine will be disabled by repave. However, if an environment was disabled before repave, repave might fail to enable the corresponding dSources and vDBs back. Manually enabling the environment and then enabling the dSources and vDBs should resolve the problem.
If an engine gets stuck in an ongoing repave state for a very long time (like a few hours), restarting the Delphix management stack can forcibly reset repave to a failed state so users are able to try again.
If the source engine has a hot fix installed, the target engine shall have the same hotfix installed as well before applying repave.

POST calls

Establish session

http://{{delphix_engine_url}}/resources/json/delphix/session

Establish an API session with Delphix, which is required before calling repave APIs.

JSON

{
	"type": "APISession",
	"version": {
		"type": "APIVersion",
		"major": {{api_version_major}},
		"minor": {{api_version_minor}},
		"micro": {{api_version_micro}}
	}
}

http://{{delphix_engine_url}}/resources/json/delphix/login

Login as sysadmin, which is required before calling repave APIs.

JSON

{
   "type": "LoginRequest",
   "username": "{{sysadmin_user}}",
   "password": "{{sysadmin_pwd}}"
}

Enable feature flag

http://{{delphix_engine_url}}/resources/json/delphix/system/enableFeatureFlag

Enable feature flag REPAVE, which is required before calling repave APIs.

JSON

{
   "type": "FeatureFlagParameters",
   "name": "REPAVE"
}

Repave prepare

http://{{delphix_engine_url}}/resources/json/delphix/repave/prepare

Introduction

Delphix sysadmin calls repave/prepare to start preparing for repave on the source engine compromised.

All the sources will be disabled.
All datasets and required metadata will be transferred to domain storage devices.
The Delphix management service will be shut down.

Request parameters

ignoreDisableSourcesFailures
If true, a failure to disable sources will not block the repave. The default is false, if not provided.
enableSourcesOnFailure
If true, when repave fails, data source disabled by repave will be enabled again. The default is false, if not provided.

Response

A job id and action id will be returned if the API is called successfully.

Please make sure repave state is PREPARE_SUCCESSFUL before detaching domain0 storage devices.

Technical details

The Delphix engine will go through four main steps during this phase:

Quiesce engine
All domain users will be forcibly kicked out. All automatic replication will be turned off. All running jobs will be cancelled or suspended. All policies will be paused. All sources will be disabled. A Repave state PREPARE_QUIESCE_ENGINE_FAILED indicates that the engine is failed to quiesce the engine.
Clean up environments
Windows environments will be cleaned up, this step is mainly to clean up the iSCSI setting of environment hosts. Any failure will be ignored since this step does not block repave, but users might see legacy iSCSI settings on their Windows environment hosts.
Extract metadata
Metadata that presents the identity of the current engine will be extracted and stored in domain0. A Repave state PREPARE_EXTRACT_METADATA_FAILED indicates that the engine has failed to extract metadata.
Export Domain0
Services like Delphix management stack and Postgres will be stopped before exporting the domain0 pool. The repave state PREPARE_EXPORT_DOMAIN0_FAILED indicates that the current engine has failed to stop services or export the domain0 pool.

JSON

{
    "type": "RepavePrepareParameters",
    "ignoreDisableSourcesFailures":false,
    "enableSourcesOnFailure":false
}

Repave apply

http://{{delphix_engine_url}}/resources/json/delphix/repave/apply

Introduction

Delphix sysadmin calls repave/apply on the new engine. The Delphix management service will be restarted and all previously disabled data sources will be enabled again.

Before calling this API, make sure all domain0 pool storage devices have been attached to the target engine correctly.

Request parameters

One type of parameter is supported for version 13.0.

BlockStorageRepaveApplyParameters is for Delphix Continuous Data engines with block storage.

Response

A job id and action id will be returned if the API is called successfully.

Technical Details

The engine will go through six main steps during the this phase. APPLY_SUCCESSFUL indicates the engine has applied repave successfully. You can always call repave/apply again if there is any failure during repaving.

Import Domain0
The domain0 pool from the source engine will be imported to the target engine. APPLY_IMPORT_DOMAIN0_FAILED indicates the repave apply failed to import domain0. Delphix management stack will be stopped after this step.
Check eligibility
Check if the target engine is eligible to apply repave. If the target engine version is different from the source engine, or any required hotfix is not installed, or not enough space is in rpool storage, checking eligibility will fail.
Setup MDS
Repave will shut down the Delphix management stack and call a backend task to set up MDS (MetaData Service) in this step, the MDS was originally from the source engine. It will also rsync /var/delphix, snapshot-based metadata and system tunable from domain0 to rpool. APPLY_SETUP_MDS_FAILED indicates repave fails in setting up MDS. The Delphix management stack will be restarted after this step.
Generate metadata
The configurable metadata stored in domain0 will be generated on the target engine so the target engine will behave identically as the source engine. APPLY_GENERATE_METADATA_FAILED indicates repave fails to generate metadata.
Refresh environments
All environments will be refreshed in parallel during this step. New iSCSI settings will be set up again on Windows environment hosts. APPLY_REFRESH_ENV_FAILED indicates repave fails to refresh environments.
Unquiesce engine
In this step, all sources disabled by repave will be enabled again, all scheduled replication jobs will be resumed and policy execution will be resumed. APPLY_UNQUIESCE_ENGINE_FAILED indicates repave fails to unquiesce engine.

JSON

{
    "type": "BlockStorageRepaveApplyParameters"
}

Repave status

http://{{delphix_engine_url}}/resources/json/delphix/repave

Introduction

GET /repave will show:

Latest state of Repave.
Summary of the current engine.
Metadata of the current engine that will be migrated to the target engine.

Tips

Always use GET /repave to check repave progress.
When the Delphix management service is restarting, GET /repave might not respond, wait for a few minutes and try again.
Use the repave state transition diagram below for reference.

Repave preview

http://{{delphix_engine_url}}/resources/json/delphix/repave/preview

Introduction

repave/preview is for users to preview the summary and metadata of a source engine.
The APPLY_ELIGIBLE state indicates the current engine is eligible to apply repave.
The APPLY_CHECK_ELIGIBILITY_FAILED state indicates the current engine is not eligible to apply repave, the actual reason can be found from stateDetail.

Request parameters

One type of parameter is supported for version 13.0.

BlockStorageRepavePreviewParameters is for engines with block storage.

Response

The engine summary and metadata of the source engine.

Technical details

The engine summary and metadata of the source engine is stored in the domain0 rpool. repave/preview will try to import domain0 pool before previewing them, then domain0 pool will be exported again.

JSON

{
    "type": "BlockStorageRepavePreviewParameters"
}