Best practices for hypervisor host and VM guest

Hypervisor best practices

To begin, visit the Deployment for VMware page to see supported ESXi versions. 

1. ESXi overhead guidelines

Resources required for hypervisor cannot be reserved, they must be left unallocated.

• Memory overhead: 8-10% of available RAM should be reserved for hypervisor operations (not allocated to guest VMs). 

  • For example, when running on an ESXi host with 512GB of physical memory, no more than 470GB (92%) should be allocated to the Delphix VM (and all other VMs on that host). 

• CPU overhead: 8-10% of available CPUs should be reserved for hypervisor operations (not allocated to guest VMs). 

  • For example, if 128 vCores are available on the ESXi host, allocate 116 to the virtual machine(s), leaving 12 for the hypervisor.

This is because certain hypervisor functions require precedence over any virtualized system. If a hypervisor needs more CPU than the amount currently available, it can de-schedule all other virtual processes to ensure adequate CPU resources for the hypervisor. Ensuring the hypervisor will not have to de-schedule any running virtual processes (worlds) by setting aside and not over-subscribing CPUs for virtual functions will leave them available for hypervisor use. 

• Even if the Delphix VM is the only VM on a host, the hypervisor is still active and essential; and still needs resources. 

2. BIOS power management should be set to “high performance” in places where ESXi controls power management. 

• This can be impacted by VMware KB 1018206, which is poor VM application performance caused by power management settings. 

• Ensure that all BIOS managed C-States other than C0 are disabled if power management is hardware controlled. 

• Ensure that all ACPI sleep states above S0 are disabled in the BIOS. 

• Examples for popular server lines from Cisco, HP, and Dell are noted below. 
  Specific models will vary, use the appropriate spec sheet. 

  • UCS: Disable Processor Power States, disable Power Technology, set Energy Performance to "Performance".

  • HP Proliant: Set HP Power Regulator to “Static High Performance" mode.

  • Dell: Set BIOS System Profile to "Performance Optimized" mode.

3. VMware HA can be enabled; VMware DRS is generally disabled. 

4. Blade/Rack server firmware and ESXi drivers should be updated to latest versions. 

Virtual machine guest best practices

To begin, visit the Virtual Machine requirements for the VMware platform page for VM settings.

1. VMware Guest specifications

• Minimum: 8 vCPU x 64 GB 

• Small: 8 vCPU x 128GB 

• Medium: 16 vCPU x 256 GB 

• Large: 24 vCPU x 512 GB 

• Delphix recommends resource reservations of 100% of RAM and CPU.

  • If the ESX host is dedicated to Delphix, CPU and RAM reservations are advised, but not necessary. However, swap space will be required on the hypervisor to compensate for the lack of reserved RAM. 

2. Assign single-core sockets for vCPUs in all cases.

• If there is a compelling reason to use multi-core CPUs, reference the following article from VMware which describes matching virtual multi-core sockets to the hardware ESX is running in this VMware KB article.

  • Example: ESXi Host has 2 socket x 18 core Intel Xeon, Delphix Engine wants 16 vCPU. 
    Configure Delphix VM with 2 Virtual Sockets, 8 Cores Per Socket to utilize hardware architecture. 

3. Set the number of vCPUs per virtual machine via the vSphere client.

• To do this, please see the “Virtual CPU Configuration" section in the vSphere Administration guide.  

  • Delphix VM CPU utilization: This Delphix knowledge base article outlines what makes Delphix VMs similar to other resource-intensive applications.

  • Here is an ESXTOP Reference and related Blog. 

4. Avoid placing other extremely active VMs on the same ESX host. 

5. Set up vSphere threshold alerts for CPU, Network, and Disk Capacity.

• The Delphix Engine uses the bulk of available RAM as a read cache for frequently accessed filesystem data. Because of this, monitoring memory usage at the vSphere level is not useful and may cause vSphere to generate false alerts. 

6. Ensure that the latest available VMware drivers and firmware versions are installed.

• Primarily for HBAs, NICs and any other hardware components configured on the Delphix virtual machine. This is a critical step that can have a massive impact on the performance and robustness of our solution. 

Frequently asked questions 

Why does Delphix require a minimum of 8 vCPUs and recommend 128 GB per 8 vCPUs? 

• 8 vCPUs are key to meeting 10Gbps single-engine throughput potential and help to sustain low latency for VDBs. 

• As with CPU, cache memory is required to drive peak loads on the Delphix Engine. More memory allows for more blocks to be read from the cache rather than going to less performant disks. Delphix stores cached data in a compressed format and only keeps a single copy of unique blocks in memory. These features give read performance across multiple VDBs provisioned from a single source dramatic improvements in speed, scalability and memory utilization. 

Why does Delphix advise reservations for CPU and memory? 

• Delphix performance can be greatly impacted there is contention over CPU or RAM. Reservations allow the engine to explicitly control those resources and avoids the possibility of contention with other VMs, even when resources are overcommitted. 

Why does Delphix request 4 controllers, and why must the storage be identical between them? 

• To provide optimal storage performance, you must spread data equally over the maximum (4) virtual SCSI controllers. To provide consistent performance between each of the four controllers, you need to ensure storage is identical between them. 

Why must virtual disks (VMDKs) be thick provisioned and eager-zeroed? 

• Thick provisioning and eager zero ensure performance is top-notch from the start with no hiccups from expanding virtual resources. 

Why is 15% free space required? 

• While the ZFS file system has a lot of features leveraged by Delphix, it loses efficiency as space decreases. 15% is the minimum that must be available for best performance. 

Why does Delphix request you reserve CPU and RAM for Hypervisor overhead? 

• Based on VMWare’s resource management guide and our own experiences with high IO throughput. Note there is no specific mechanism to assign resources to the hypervisor, the only way to preserve overhead is by not allocating resources to guests. 

Why does Delphix generally want VMWare HA enabled, but DRS disabled? 

• VMware HA (High Availability) addresses outages that occur when a physical host goes down or is completely offline, by migrating the guest(s) to another physical host and restarting them. There is no real downside, it simply brings unavailable servers back online. 

• VMware DRS (Distributed Resource Scheduler) is for load balancing host resources in a cluster. Because of high IO and best practices configuration for optimal performance, our engine is typically not a good candidate for relocation. 

Why does Delphix request you set power management to High-Performance Mode? 

• This will ensure power management will never impact performance by entering into a lower power state (also known as c-state).