Understanding data containers
Delphix Self-Service data container overview
Data containers are provisioned from data templates by administrators and assigned to a user. A data container represents a socket that is capable of making any data within the data template accessible. The user controls what data they want to access.
Delphix Self-Service users have effectively been provisioned a set of "physical" resources, such as a database on a host that consumes some set of resources. A data container is comprised of a virtual database (VDB) or vFiles provisioned from each source in the data template from which it is created. The data container manages these VDBs, and the data operations performed on a data container will only impact these VDBs.
Data containers represent the separation between IT infrastructure and end-users. IT determines the set of VDBs or vFiles to allocate to a data container, and users determine the data that they want accessible in the containers allocated to them.
Data containers can be used to access any data within a single data template, but not across templates. Users have the ability to populate the data within their data container from any point in time on the data template, the data container's history, or shared bookmarks from other data containers.
Although operations are all accomplished by performing Timeflow operations on the underlying VDBs, the data containers hide the VDBs and their underlying properties from users. None of the data container operations require provisioning additional VDBs; everything is accomplished using the resources assigned when the data container is created.
Operation | Description |
|---|---|
Refresh | This is the same basic concept as Refresh in VDBs. In Delphix Self-Service, Refresh will update the data on the active branch of a user's data container. The user will then have the latest data in the sources of the data template from which the container was provisioned. |
Restore | Restore allows a Delphix Self-Service user to update the data on the active branch of their data container to one of the following:
This operation effectively means, "take me to the data at this time." |
Reset | Reset is a simplified version of Restore, built to support the notion of "undo." It allows a user to reset the state of their application container to the latest operation. This can be useful for testing workflows where, after each test, users want to reset the state of their environment. |
Branch | A branch represents a logical timeline, effectively a task on which a user is working. Only one branch can be active at a time, but a user can use multiple branches to track logically separate tasks. Delphix Self-Service branches do not require the allocation of a new VDB; instead, they are comprised of a collection of TimeFlows within a VDB. |
Activate | This allows the user to select which branch they want to be active. Only a single branch within a data container can be active at a time. |
Bookmark | This creates a semantic name for a point in time and prevents this data from being removed by the retention policy. Bookmarks can be annotated with tags to make them easier to search for. In addition to tags, bookmarks allow a user to enter a description of what the bookmark represents. |
Share | Bookmarks can be shared, which allows them to be seen by users who own data containers that have been provisioned from the same data template. This allows users to share data, providing a way for other users to either restore their existing timeline or create a new branch from these shared points. |
Once a data operation has been selected its progress can be viewed from the Action sidebar in the Management application. From the Action sidebar, users cannot cancel the enable, disable, or activate branch operations.
Automatic retry for data operations
To make operations on data containers more robust, Delphix Self-Service supports automatically retrying failed sources during data operations. You can specify a maximum number of retry attempts so that if an operation fails on any individual data source within a data container, it will be automatically retried until it succeeds, or until the retry limit is reached.
Automatic retry applies to any Delphix Self-Service operation that changes the data in the data container, such as Refresh, Restore, Reset, or Create Branch. This setting can be especially useful in scenarios where there are a large number of sources in a data container, and some sources fail to update the first time. If the reason for the failures was intermittent, an automatic retry may allow the sources that failed to succeed, and the operation can still complete successfully. The default number of retry attempts is 1.
To change the number of automatic retries:
Click the user icon in the upper right-hand corner of the screen.
From the drop-down menu, select Settings.
In the field Automatically retry data sources..., enter a new maximum; alternatively, use the arrows to increase or decrease the number.
Click Save.
Settings in the user drop-down menu.
Delphix Self-Service data container recovery
Data containers consistency
Delphix Self-Service allows you to group multiple datasets in the same data container. This makes it easy for you to access entire applications such as PeopleSoft, including binaries and code.
If a data container represents an application, then there are likely to be dependencies between the application's datasets. For example, the vFiles data source containing the code will depend on a specific version of the database's schema. Therefore, it is important that all dataset sources are drawn from the same point in time. If they are, the data container is in a "consistent" state; if they are out of sync, or "inconsistent," errors will occur. For example, if the vFiles data source containing the code has been updated more recently than the database's schema, the dependency cannot work.
Delphix Self-Service currently has no way to determine whether the application is consistent. However, it attempts to minimize the chance that dataset sources are out of sync whenever it performs a data operation such as refresh, restore, or reset.
When performing a data operation, Delphix Self-Service attempts to snapshot all dataset sources from a point in time as close as possible to the desired time. If at least one of the data sources fails to go to the desired point, then Delphix Self-Service considers the data container to be in an inconsistent state.
The application as a whole may still be working, but Delphix Self-Service assumes that the failed dataset's data is not the correct version. To return to a consistent state, you must perform a recovery operation on the data container.
Data container recovery
Prior to performing any data operation, Delphix Self-Service takes snapshots of all datasets. Recovery is the process of rolling back a data container to a snapshot, thereby restoring it to a consistent state. When a failure occurs, you will see the following screen:
You can either perform recovery or use a different data container. Whether the recovery will fail or succeed depends on why the data operation failed in the first place.
If the problem was intermittent, such as a temporary network problem causing SSH failure, then performing recovery should work. If the problem is persistent – for example, the target host is out of space – then intervention is required; recovery will not succeed until you address the underlying root cause of the failure.
Admins can see the underlying failure in the Actions sidebar or the Job History dashboard. The Actions sidebar is the preferred place to view the failure; it has a hierarchical display that makes diagnosing the failure more straightforward.
Preserving independent containers in Delphix Self-Service during replication
Replication is used for data backup and recovery as well as for managing and sharing data across remote data centers. Delphix Self-Service users can preserve their data after replication jobs.
In the past, if replication occurred on templates in containers, users would lose the data in their containers. Now, admin users can preserve containers to be used independently of replication jobs.
Independent containers behave in the same way as other containers, with two exceptions:
You cannot refresh them.
The bookmarks created on them cannot be shared, because they do not have a template reference.
The functional overview of independent containers seen below represents the flow of steps between the source engine and the target engine. A description of what is occurring between each of the steps appears below the diagram. See the corresponding number below the diagram to find out more details.
Functional overview of independent containers.
In Delphix Self-Service, you can create a template on the source engine and then replicate the template to the target engine.
The target engine, an admin can use the replicated template to create new containers and assign them to users. You cannot change the replicated template’s name or the names of the containers with which it was replicated over.
Due to an update, the replicated template is deleted from the source engine
The deleted replicated template will be removed from the target engine. Any new container created in step 2 loses the reference to the deleted template and becomes an independent container.
Creating independent containers
Prerequisites
The replication source and the replication target must be running identical versions of the Data Engine – for example, Data Engine version 5.1.
The target Delphix Engine must be reachable from the source engine.
The target Delphix Engine must have sufficient free storage to receive the replicated data.
The user must have administrative privileges on the source and the target engines.
For more information, see Configuring Replication and Understanding Data Templates.
Limitations of this functionality
You can find independent containers in Delphix Self-Service on the target engine under the Independent Containers tab. They have the following characteristics:
They cannot be refreshed, because they are no longer bound to a template.
You can create bookmarks on them, but you cannot share those bookmarks because there is no common template.
You can use them for branching, restoring, resetting, starting, and stopping.
Procedure
To create an independent container, complete the following steps:
On the source engine, create a template with a container.
From the user drop-down menu, select Management.
From the System menu, select Replication.
Next to Replicated Profiles, select the plus icon to Create New Profile.
Under Objects Being Replicated, select your Self-Service template and its associated container.
Enter your profile information.
Click Create Profile.
When replicating templates, you can select all, some, or none of their associated containers in the replication profile. This is done by selecting the checkbox next to the container's name in the Create New Profile window. When replicating a container, you must also replicate its associated template. Replicated objects cannot be modified on the target engine unless they are failed over, so you cannot modify the names of replicated data containers and templates.
Once the profile has been created, click Replicate Now.
On the target engine, click the user menu.
Select Self-Service.
The replicated template will appear in Self-Service on the target engine. The replica name is displayed next to the template name. You can edit regular templates by clicking the pencil icon next to the template name.
Select the replicated template, then select Containers.
In the Containers window, click Add Container. In order to complete this action, you will need to ensure that there is data available from each data source in the template. This means that VDBs must have been provisioned from each replica dSource or VDB in the template. After the container is created, your replicated template should have the new container you just created and the original container created in step 1.
On the source engine in Self-Service, delete your template.
From the user menu, select Management.
From the System menu, select Replication.
Replicate your profile to create a new independent container.
On your target engine, select Self-Service.
The new container is created. To find it:
Login to the target engine.
Click the user menu.
Select Self-Service.
In the Overview page, select the Independent Containers tab.
