Repave Delphix Engine
Overview
As the rate of new security vulnerabilities rise, Delphix continues developing solutions to mitigate potential risks. Repave is an added solution applied to a Delphix Engine to get rid of any potentially compromising complications. It transfers datasets and required metadata to a fresh, new target engine, without carrying over any compromised binaries. The target engine will function identically to the source engine.
Prerequisites
Repave is supported on Continuous Data (virtualization) engine with block storage as of version 13.0.0.0 and Continuous Compliance (masking) engine with block storage as of version 16.0.0.0.
You would need to enable the feature flag
REPAVE
before using it on engine version 13.0. After 14.0, the feature flag is enabled by default.The target engine shall be in the same version as the source engine.
How to Repave a Delphix Engine
![](../../__attachments/47522738/repave1.png?inst-v=9f5cd9ef-94fe-4677-b3c4-6c5d1617af5c)
A sysadmin initiates repave/prepare on the source engine. All Delphix datasets are disabled and the Delphix management service is shut down.
A sysadmin detaches storage devices from the Delphix engine and attaches them to a new Delphix engine. This can be done through cloud APIs or manual operation.
A sysadmin calls repave/apply on the new engine. All previously disabled data sources are enabled again.
Repave API calls
POST repave/prepare
initiates the repave preparation on the source enginePOST repave/apply
starts the process of applying repave on the target engine.GET repave
returns the repave state, configurable metadata, and engine summary of the current engine.POST repave/preview
provides a preview of the configurable metadata and engine summary of the source engine, as well as determines its eligibility for applying repave on the target engine.
The representation below is a general API workflow of repave.
You might need to call APIs Establish Session
, Login
and Enable Feature Flag
before calling repave APIs.
![](../../__attachments/47522738/image-20230817-171354.png?inst-v=9f5cd9ef-94fe-4677-b3c4-6c5d1617af5c)
Tips and usage guidelines
repave/prepare
andrepave/apply
returns a job id. However, when the Delphix management service is down, the job API will not work. The other way to check the repave status is by callingGET repave
.All running sources on the engine will be disabled by repave. However, if an environment was disabled before repave, repave might fail to enable the corresponding dSources and vDBs back. Manually enabling the environment and then enabling the dSources and vDBs should resolve the problem.
If an engine gets stuck in an ongoing repave state for a very long time (like a few hours), restarting the Delphix management stack can forcibly reset repave to a failed state so users are able to try again.
If the source engine has a hot fix installed, the target engine shall have the same hotfix installed as well before applying repave.
POST calls
Establish session
http://{{delphix_engine_url}}/resources/json/delphix/session
Establish an API session with Delphix, which is required before calling repave APIs.
{
"type": "APISession",
"version": {
"type": "APIVersion",
"major": {{api_version_major}},
"minor": {{api_version_minor}},
"micro": {{api_version_micro}}
}
}
Login
http://{{delphix_engine_url}}/resources/json/delphix/login
Login as sysadmin
, which is required before calling repave APIs.
{
"type": "LoginRequest",
"username": "{{sysadmin_user}}",
"password": "{{sysadmin_pwd}}"
}
Enable feature flag
http://{{delphix_engine_url}}/resources/json/delphix/system/enableFeatureFlag
Enable feature flag REPAVE
, which is required before calling repave APIs.
{
"type": "FeatureFlagParameters",
"name": "REPAVE"
}
Repave prepare
http://{{delphix_engine_url}}/resources/json/delphix/repave/prepare
Introduction
Delphix sysadmin calls repave/prepare to start preparing for repave on the source engine compromised.
All the sources will be disabled.
All datasets and required metadata will be transferred to domain storage devices.
The Delphix management service will be shut down.
Request parameters
ignoreDisableSourcesFailures
Iftrue
, a failure to disable sources will not block the repave. The default isfalse
, if not provided.enableSourcesOnFailure
Iftrue
, when repave fails, data source disabled by repave will be enabled again. The default isfalse
, if not provided.
Response
A job id and action id will be returned if the API is called successfully.
Please make sure repave state is PREPARE_SUCCESSFUL
before detaching domain0 storage devices.
Technical details
The Delphix engine will go through four main steps during this phase:
Quiesce engine
All domain users will be forcibly kicked out. All automatic replication will be turned off. All running jobs will be cancelled or suspended. All policies will be paused. All sources will be disabled. A Repave statePREPARE_QUIESCE_ENGINE_FAILED
indicates that the engine is failed to quiesce the engine.Clean up environments
Windows environments will be cleaned up, this step is mainly to clean up the iSCSI setting of environment hosts. Any failure will be ignored since this step does not block repave, but users might see legacy iSCSI settings on their Windows environment hosts.Extract metadata
Metadata that presents the identity of the current engine will be extracted and stored in domain0. A Repave statePREPARE_EXTRACT_METADATA_FAILED
indicates that the engine has failed to extract metadata.Export Domain0
Services like Delphix management stack and Postgres will be stopped before exporting the domain0 pool. The repave statePREPARE_EXPORT_DOMAIN0_FAILED
indicates that the current engine has failed to stop services or export the domain0 pool.
{
"type": "RepavePrepareParameters",
"ignoreDisableSourcesFailures":false,
"enableSourcesOnFailure":false
}
Repave apply
http://{{delphix_engine_url}}/resources/json/delphix/repave/apply
Introduction
Delphix sysadmin calls repave/apply on the new engine. The Delphix management service will be restarted and all previously disabled data sources will be enabled again.
Before calling this API, make sure all domain0 pool storage devices have been attached to the target engine correctly.
Request parameters
One type of parameter is supported for version 13.0.
BlockStorageRepaveApplyParameters
is for Delphix Continuous Data engines with block storage.
Response
A job id and action id will be returned if the API is called successfully.
Technical Details
The engine will go through six main steps during the this phase. APPLY_SUCCESSFUL
indicates the engine has applied repave successfully. You can always call repave/apply
again if there is any failure during repaving.
Import Domain0
The domain0 pool from the source engine will be imported to the target engine.APPLY_IMPORT_DOMAIN0_FAILED
indicates the repave apply failed to import domain0. Delphix management stack will be stopped after this step.Check eligibility
Check if the target engine is eligible to apply repave. If the target engine version is different from the source engine, or any required hotfix is not installed, or not enough space is in rpool storage, checking eligibility will fail.Setup MDS
Repave will shut down the Delphix management stack and call a backend task to set up MDS (MetaData Service) in this step, the MDS was originally from the source engine. It will also rsync/var/delphix
, snapshot-based metadata and system tunable from domain0 to rpool.APPLY_SETUP_MDS_FAILED
indicates repave fails in setting up MDS. The Delphix management stack will be restarted after this step.Generate metadata
The configurable metadata stored in domain0 will be generated on the target engine so the target engine will behave identically as the source engine.APPLY_GENERATE_METADATA_FAILED
indicates repave fails to generate metadata.Refresh environments
All environments will be refreshed in parallel during this step. New iSCSI settings will be set up again on Windows environment hosts.APPLY_REFRESH_ENV_FAILED
indicates repave fails to refresh environments.Unquiesce engine
In this step, all sources disabled by repave will be enabled again, all scheduled replication jobs will be resumed and policy execution will be resumed.APPLY_UNQUIESCE_ENGINE_FAILED
indicates repave fails to unquiesce engine.
{
"type": "BlockStorageRepaveApplyParameters"
}
Repave status
http://{{delphix_engine_url}}/resources/json/delphix/repave
Introduction
GET /repave
will show:
Latest state of Repave.
Summary of the current engine.
Metadata of the current engine that will be migrated to the target engine.
Tips
Always use
GET /repave
to check repave progress.When the Delphix management service is restarting,
GET /repave
might not respond, wait for a few minutes and try again.Use the repave state transition diagram below for reference.
![](../../__attachments/47522738/repave3.png?inst-v=9f5cd9ef-94fe-4677-b3c4-6c5d1617af5c)
Repave preview
http://{{delphix_engine_url}}/resources/json/delphix/repave/preview
Introduction
repave/preview
is for users to preview the summary and metadata of a source engine.The
APPLY_ELIGIBLE
state indicates the current engine is eligible to apply repave.The
APPLY_CHECK_ELIGIBILITY_FAILED
state indicates the current engine is not eligible to apply repave, the actual reason can be found fromstateDetail
.
Request parameters
One type of parameter is supported for version 13.0.
BlockStorageRepavePreviewParameters
is for engines with block storage.
Response
The engine summary and metadata of the source engine.
Technical details
The engine summary and metadata of the source engine is stored in the domain0 rpool.
repave/preview
will try to import domain0 pool before previewing them, then domain0 pool will be exported again.
{
"type": "BlockStorageRepavePreviewParameters"
}