Best practices for Staging environments and databases

Staging is the process of reconstructing a full copy of one or more databases, entirely separate and independent from the original source, for the Delphix Continuous Data Engine to ingest. The reconstructing process is performed on a staging database that runs on a staging environment. An environment is made up of one or more hosts, or VMs. The Delphix Continuous Data Engine provides the physical storage to the staging database which enables the various Staging pull and push ingestion processes. The Staging environment is similar to a Target environment, such as the database application and the remote storage mounts to the Delphix Continuous Data Engine. However, because its core purpose is to help with data ingestion, and not data provisioning, it frequently has a different set of requirements.

The Staging environment is a requirement for data sources that Delphix supports, except Oracle. However, Oracle does optionally utilize a Staging environment to run validated sync.

Minimum system requirements

The required CPU and Memory for a Staging environment are heavily dependent on the database type, size, configuration, and ingestion being performed. We recommend looking at your production environment as a starting point and then shrinking based on differences in the expected load. Review your chosen database’s staging pull, staging push, or validated sync mechanism to better understand the load your environment might experience.

The required storage for a Staging Environment is minimal when compared to the engine’s requirements. Because the Staging environment leverages remote storage over the network from the Delphix Continuous Data Engine, the environment only needs enough disk capacity for the OS, database application, and any relevant logs or tools. Occasionally, additional space is needed to store a temporary backup. See below and consult the connector documentation for more information.

If the Staging environment is being shared, such as with other databases or as a Target environment, ensure that resources can support the combined load or alternate ingestion times using policies or automation.

General guidance for staging servers (Multi-platform)

1. Delphix recommends dedicated Staging hosts for role/architecture separation. However, any Target host can be used as a Staging host.

2. In cases where the same host is used as both a Staging host and a Target host, we strongly recommend a dedicated instance/install for staging to avoid confusion.

3. Delphix recommends at least one Staging host per Delphix Continuous Data Engine to avoid the possibility of a single point of failure across multiple engines.

4. If a Staging host is shared among multiple Delphix Continuous Data Engines, please ensure that a dedicated database Instance is created for each Delphix Continuous Data Engine.

5. Configuration/performance factors:

   1. Transaction log generation rate.

   2. The number of VDBs (if the host is a shared Staging / Target host).

   3. The number of dSources (if the host is a dedicated Staging host or a shared Staging / Target host).

Precise guidance on these items has not yet been defined. In general, if there is a heavy log generation rate and few VDBs, the ideal recommendation is to have at least 1 Staging host per Delphix Continuous Data Engine.

Disk / local storage

1. The only local storage needed is for the OS, the application with default databases, and any temporary logs and tools

2. Storage for a Staging database is provided from the Delphix Continuous Data Engine, which is mounted over the network similar to any Target host (NFS/iSCSI).

Network requirements

1. The Staging host engages in network data transfers with the Source host backup shared location as well as with the Delphix Continuous Data Engine.

2. The Staging host is also a Target host, and as such should have < 1ms latency to the Delphix Continuous Engine (and low latency to the Source backup, when possible).

3. If the change rate on the Source database(s) is > 1 Gb/sec, the recommended network bandwidth to support network transfers is 10 Gb/sec.

4. In cases where only 1 Gb/sec network bandwidth is available, segregation of each network is recommended to reduce network contention.

5. Ensure that the virtual NIC is using the standard vmxnet3 adapter and not Intel for VMWare based clients. Logical IO errors have been reported while using Intel instead of vmxnet3 adapter.

Windows and MSSQL-specific

1. The SQL Server Instance used for Staging should not be clustered.

2. The SQL Server Instances hosted on the Staging host should have a Maximum Memory set. Also always ensure that at least 10% of total memory is available for OS operations.

3. Only system databases (Master/MSDB/Temp/MSDB) are kept on local storage. All other data is read/written to the Delphix Continuous Data Engine.

4. Ensure that Receive Side Scaling (RSS) is enabled on every network interface that Delphix will be connecting to.

Other database guidance

Each database type has differences in how a Staging environment must be configured. Consult the other connectors’ documentation for database-specific guidance.