Deployment for VMware

Overview

This article outlines the requirements for deploying the Delphix Engine on VMware (including supported versions and instance configurations), as well as recommended configuration parameters for optimal performance.

The Delphix Engine is an intensive platform, both from a network and a storage perspective. If the Delphix Engine competes with other virtual machines on the same host for resources it will result in increased latency for all operations. As such, it is crucial that the ESXi host is not over-subscribed, as this eliminates the possibility of a lack of resources for the Delphix Engine. This includes allowing a percentage of CPU resources for the hypervisor itself as it can de-schedule an entire VM if the hypervisor is needed for managing IO or compute resources.

Delphix disk storage capabilities remain seamlessly operable. There are no additional limitations to your storage, unless otherwise imposed by the [respective] Hypervisor or Cloud environment.

Supported ESX versions

Requirements	Notes
VMware Cloud VMware ESXi 8.0, 8.0 U1 VMware ESXi 7.0, 7.0 U1, 7.0 U2, 7.0 U3 VMware ESXi 8.0, 8.0 U1, 8.0 U2	If a minor release version is listed as supported, then all patch releases applicable to that minor release are supported. Public cloud deployments such as VMware cloud on AWS and Azure VMware solution are supported. While ESXi 7.0 U3 is supported, this is only for patch releases 3c and newer, due to critical issues in the earlier patch versions.

Virtual machine hardware versions

The Delphix Engine VM is distributed as an OVA for VMware, and is configured with the hardware version corresponding to the lowest ESXi version that release is qualified for. As the Engine is upgraded, the ESXi versions supported may change, but the hardware version may remain the same based on the original deployment.

Users who wish to upgrade the VM hardware version for compatibility, enhanced feature support, or other reasons may feel free to do so, with consideration for any compatibility concerns that may arise in environments where multiple ESXi versions are present, as an upgraded hardware version can affect other VMware operations (vMotion, etc). VMware support and documentation should be consulted before committing any hardware version upgrade for a guest VM, but Delphix does not maintain any version requirements in this regard.

Virtual CPUs

Requirements

Notes

8 vCPUs

CPU resource shortfalls can occur both on an over-committed host as well as competition for host resources during high IO utilization.
CPU reservations are strongly recommended for the Delphix VM so that Delphix is guaranteed the full complement of vCPUs even when resources are overcommitted.
It is suggested to use a single core per socket unless there are specific requirements for other VMs on the same ESXi host.

Never allocate all available physical CPUs to virtual machines

CPU for the ESXi Server to perform hypervisor activities must be set aside before assigning vCPUs to Delphix and other VMs.
We recommend that a minimum of 8-10% of the CPUs available are reserved for hypervisor operation. (e.g. 12 vCPUs on a 128 vCore system).

Memory

Requirements

Notes

128 GB vRAM (recommended)

64GB vRAM (minimum)

The Delphix Engine uses its memory to cache database blocks. More memory will provide better read performance.
Memory reservations are required for the Delphix VM. The performance of the Delphix Engine will be significantly impacted by the over-commitment of memory resources in the ESX Server.
Reservations ensure that the Delphix Engine will not be forced to swap pages during times of memory pressure on the host. A swapped page will require orders of magnitude more time to be brought back to physical memory from the ESXi swap device.

Never allocate all the available physical memory to virtual machines.

The default ESX minimum free memory requirement is 6% of the total RAM. When free memory falls below 6%, ESX starts swapping out the Delphix guest OS. Delphix recommends leaving about 8-10% free to avoid swapping.
For example, when running on an ESX Host with 512GB of physical memory, no more than 470GB (92%) should be allocated to the Delphix VM (and all other VMs on that host).

Memory for the ESX Server to perform hypervisor activities must be set aside before assigning memory to Delphix and other VMs.

Failure to ensure sufficient memory for the host can result in a hard memory state for all VMs on the host which will result in a block for memory allocations.

Network

Requirements	Notes
The ova is pre-configured to use one virtual ethernet adapter of type VMXNET 3.	Jumbo frames are highly recommended to reduce CPU utilization, decrease latency and increase network throughput. (typically 10-20% throughput improvement) If additional virtual network adapters are desired, they should also be of type VMXNET 3.
A 10GbE NIC in the ESX Server is recommended.	For VMs having only gigabit networks, it is possible to aggregate several physical 1GbE NICs together to increase network bandwidth (but not necessarily to reduce latency). Refer to the VMware Knowledge Base article NIC Teaming in ESXi and ESX. However, it is not recommended to aggregate NICs in the Delphix Engine VM.
If the network load in the ESX Server hosting the Delphix engine VM is high, dedicate one or more physical NICs to the Delphix Engine.	Adding NICs only works if VMs are discovered using different interfaces. The NFS/iSCSI mounts will only use the network associated with the discovery. See General Network and Connectivity Requirements for information about specific port configurations, and Network Performance Configuration Options for information about network performance tuning

To bootstrap a networking configuration to reach the Delphix Engine, after deploying it into your environment for the first time, you can use one of the following options:

DHCP
Cloud-init for public clouds
You can login to the serial console to configure networking via the CLI
OVF guest customizations to pass in network configuration to the VM before it has a network connection. For the customization steps:
- Type the name of the profile and click Next
- The domain will be ignored by Delphix. Click Next
- The time zone setting will be ignored by Delphix, Click Next
- Delphix doesn’t allow scripts, click Next
- Manually select custom settings, select a NIC and click Edit
  - Provide Netmask and Gateway
  - Select one of the options for IP
  - Confirm changes and click Next
- Add DNS
- Confirm creation of profile, click Finish
- You can use this template to clone a VM

SCSI controller

Requirements

Notes

PVSCSI (default)/ LSI Logic Parallel

When adding virtual disks make sure that they are evenly distributing the load across the maximum of 4 virtual SCSI controllers. Spreading the disks across available SCSI controllers evenly will ensure optimal IO performance from the disks. For example, a VM with 4 SCSI controllers and 8 virtual disks should distribute the disks across the controllers as follows:

disk0 = SCSI(0:0) - System Disk on Controller 0 Port 0

(ignore for purposes of load balancing)

disk1 = SCSI(0:1) - Data Disk on Controller 0 Port 1

disk2 = SCSI(1:1) - Data Disk on Controller 1 Port 1

disk3 = SCSI(2:1) - Data Disk on Controller 2 Port 1

disk4 = SCSI(3:1) - Data Disk on Controller 3 Port 1

disk1 = SCSI(0:2) - Data Disk on Controller 0 Port 2

disk2 = SCSI(1:2) - Data Disk on Controller 1 Port 2

disk3 = SCSI(2:2) - Data Disk on Controller 2 Port 2

disk4 = SCSI(3:2) - Data Disk on Controller 3 Port 2

For load purposes, we generally focus on the DB storage and ignore the controller placement of the system disk.

IDE and SATA controllers are not supported on VMware platforms

General storage

VMware offers options for disk storage, which include "Independent - Persistent" and "Independent - Non-persistent". The non-persistent setting is catastrophic to the Delphix platform when the VM reboots, thus, Independent - Persistent is required for installation.

Requirements

Notes

Storage used for Delphix must be provisioned from storage that provides data protection.

For example, using RAID levels with data protection features, or equivalent technology.

The Delphix engine product does not protect against data loss originating at the hypervisor or SAN layers.

For more information refer to, Optimal Storage Configuration Parameters for the Delphix Engine.

Delphix storage options

There are three types of data that Delphix stores on disk, which is:

Delphix VM Configuration Storage: stores data related to the configuration of the Delphix VM. VM Configuration Storage includes the VMware ESX configuration data as well as log files.
Delphix Engine System Disk Storage: stores data related to the Delphix Engine system data, such as the Delphix .ova settings.
Database Storage: stores data used by Delphix objects such as dSources and virtual databases (VDBs).

Delphix VM configuration storage

The Delphix VM configuration should be stored in a VMFS volume (often called a "datastore").

Requirements	Notes
The VMFS volume should have enough available space to hold all ESX configuration and log files associated with the Delphix Engine.	If a memory reservation is not enabled for the Delphix Engine (in violation of memory requirements stated above), then space for a paging area equal to the Delphix Engine's VM memory must be added to the VMFS volume containing the Delphix VM configuration data.

Delphix engine system disk storage

The VMFS volume must be located on shared storage in order to use vMotion and HA features.

Requirements	Notes
The Delphix Engine system disk should be stored in a VMDK.	The VMDK for the Delphix Engine System Disk Storage is often created in the same VMFS volume as the Delphix VM definition. In that case, the datastore must have sufficient space to hold the Delphix VM Configuration, the VDMK for the system disk, and a paging area if a memory reservation was not enabled for the Delphix Engine.
The Delphix .ova file is configured for a 127GB system drive.	The VMFS volume where the .ova is deployed should, therefore, have at least 127GB of free space prior to deploying the .ova.

Database storage

Option 1: Block Storage for database storage

Shared storage is required in order to use vMotion and HA features. In addition to making sure the latest VMware patches have been applied, check with the organizations hardware vendor for updates specific to the hardware configurations. VMDKs (Virtual Machine Disks) or RDMs (Raw Device Mappings) operating in virtual compatibility mode can be used for database storage.

Requirements	Notes
A minimum of 4 VMDKs or RDMs should be allocated for database storage.	Allocating a minimum of 4 VMDKs or RDMs for database storage enables the Delphix File System (DxFS) to make sure that its file systems are always consistent on disk without additional serialization. This also enables the Delphix Engine to achieve higher I/O rates by queueing more I/O operations to its storage.
If using VMDKs: Each VMDK should be the only VMDK in its VMFS volume The VMFS volumes should be assigned to dedicated physical LUNs on redundant storage. The VMDKs should be created with the Thick Provision Lazy Zeroed option.	Provisioning VMDKs from isolated VMFS volumes on dedicated physical LUNs: Reduces contention for the underlying physical LUNs Eliminates contention for locks on the VMFS volumes from other VMs and/or the ESX Server Console Enables higher availability of the Delphix VM by allowing vSphere to vMotion the VM to a different ESX host in the event of a failure of the Delphix ESX host
The quantity and size of VDMKs or RDMs assigned must be identical across all 4 controllers	If the underlying storage array allocates physical LUNs by carving them from RAID groups, the LUNs should be allocated from different RAID groups. This eliminates contention for the underlying disks in the RAID groups as the Delphix engine distributes IO across its storage devices.
The physical LUNs used for VMFS volumes and RDMs should be of the same type in terms of performance characteristics such as latency, RPMs, and RAID level.	The total number of disk drives that comprise the set of physical LUNs should be capable of providing the desired aggregate I/O throughput (MB/sec) and IOPS (Input/Output Operations per Second) for all virtual databases that will be hosted by the Delphix Engine.
The physical LUNs used for VMFS volumes can be thin-provisioned in the storage array.	If the storage array allocates physical LUNs from storage pools comprising dozens of disk drives, the LUNs should be distributed evenly across the available pools.
For best performance, the LUNs used for RDMs should not be thin-provisioned in the storage array but should be thick-provisioned with a size equal to the amount of storage that will be initially allocated to the Delphix Engine. The RDM can be expanded in the future when more storage is needed.	Using thin-provisioned LUNs in the storage array for VMFS volumes can be useful if adding storage to the Delphix engine in the future is anticipated. In this case, the LUNs should be thin-provisioned with a size larger than the amount of storage that will be initially allocated to the Delphix Engine. When appropriate, to add more storage to the Delphix engine, use vSphere to expand the size of the VMDKs. Be sure to specify that the additional storage is also thick-provisioned and eager-zeroed.

In addition to making sure the latest VMware patches have been applied, check with the organization's hardware vendor for updates specific to the hardware configurations.

Option 2: Elastic Data where Object storage is used for database storage and block storage is used for cache.

We support Elastic Data with on prem object storage for the database storage. Traditional disks as cache are used to reduce latencies for frequently read data and as temporary storage for synchronous writes before the writes are sent to object storage.

For on prem object storage we currently support storage vendors that confirm to the following
- s3 REST API compatibility
- strong read-after-write consistency
- supports s3 key id and secret access key authentication
- perpetual key support
On-premise object storage may require a security certificate to create secure connections between the Delphix engine and object storage. If a certificate is required, then it must be installed on the Delphix engine prior to configuring the storage. Refer to Certificate management for more information.
Refer to Initial setup for additional details for setting up the Elastic Data Engine.

Additional VMware configuration notes

Running Delphix inside of vSphere is supported.
Using vMotion on a Delphix VM is supported.
Device passthrough is not supported.

Procedure to install an OVA

Use the Delphix-supplied OVA file to install the Delphix Engine. The OVA file is configured with many of the minimum system requirements. The underlying storage for the install is assumed to be redundant SAN storage.

Download the OVA file from https://download.delphix.com. You will need a support login from your sales team or a welcome letter.
1. Navigate to the Delphix Product Releases/<Current Version>/Appliance Images page.
Login using the vSphere client to the vSphere server (or vCenter Server) where you want to install the Delphix Engine.
In the vSphere Client, click File.
Select Deploy OVA Template.
Browse to the OVA file.
Click Next.
Select a hostname for the Delphix Engine.
This hostname will also be used in configuring the Delphix Engine network.
Select the data center where the Delphix Engine will be located.
Select the cluster and the ESX host.
Select one (1) data store for the Delphix OS. This datastore can be thin-provisioned and must have enough free space to accommodate the 127GB comprising the Delphix operating system.
1. For traditional block storage engines
  1. Select four (4) or more data stores for Database Storage for the Delphix Engine. The Delphix Engine will stripe all of the Database Storage across these VMDKs, so for optimal I/O performance, each VMDK must be equal in size and be configured Thick Provisioned - Eager Zeroed. Additionally, these VMDKs should be distributed as evenly as possible across all four SCSI I/O controllers.
2. For Elastic data
  1. The VMDK will be used as cache to reduce latencies for frequently read data and as temporary storage for synchronous writes before the writes are sent to object storage. For optimal I/O performance, each VMDK must be equal in size and be configured Thick Provisioned - Eager Zeroed. Make sure the disks satisfy the I/O needs of the engine. Eg: At 500 IOPS per 1GiB ratio, a 32 GiB volume can be configured to have the 16K IOPS limit. Two devices would be sufficient for the instance that requires 30K IOPS. You can always add disks later if the cache is insufficient. You can only reduce the cache by removing disks, so if you think you are over provisioning the cache, increase the number of disks used and reduce the size of each disk so that removal is possible at a later point in time.
Select the virtual network you want to use.
If using multiple physical NICs for link aggregation, you must use vSphere NIC teaming. Do not add multiple virtual NICs to the Delphix Engine itself. The Delphix Engine should use a single virtual network. For more information, see Optimal network architecture for the Delphix engine.
Click Finish.
The installation will begin and the Delphix Engine will be created in the location you specified.
Once the installation has completed, power on the Delphix Engine and proceed with the initial system configuration as described in Setting up network access to the Delphix engine