Architecture for performance - hypervisors and host

The Delphix Engine is a "virtual appliance": a virtual machine guest running within a hypervisor on a physical host.
There are a few key best practices we need to keep in mind as we consider this architecture.

Architecture best practices hypervisor host ESX

Hypervisor

1. ESXi 6.x or 7.0 is recommended. ESXi 5.5 or earlier is no longer supported. 

2. HyperThreading (HT) for Intel®-based servers (no HT on AMD CPUs).

   1. Disable HT in BIOS, on the ESXi Host, and disable HT Sharing on the Delphix VM for consistency. This is our best practice, disable at all levels.
      Any other combination may result in a non-deterministic performance.

   2. When HT cannot be turned off for both the Host and Delphix VM, it should be turned on at all levels, not run in a "mixed-mode".

      1. HT disablement at a guest level only can result in non-deterministic performance.

      2. A dedicated ESXi host, cluster, or DRA is recommended where consistent VDB performance is paramount.

   3. VM migration to a new host (e.g. VMware HA or vMotion) can create mismatched HT settings.

3. ESXi overhead (resources required for hypervisor cannot be reserved, they must be left unallocated).

   1. Memory Overhead - 10% of available RAM must not be allocated to guest VMs.

      1. Example: 256GB RAM, allocate 230GB to Delphix VM, leave 26GB for ESX

   2. CPU Overhead - At least 2 cores (ideally 4) must not be allocated to guest VMs.

      1. Example: If 16 physical cores are available, allocate 12 to the virtual machines, leaving 4 for the hypervisor

      2. Why 4? Certain hypervisor functions require precedence over any virtualized system. If a hypervisor needs more CPU than the amount currently available, it can de-schedule all other virtual processes to ensure adequate CPU resources for the hypervisor. Ensuring the hypervisor will not have to de-schedule any running virtual processes (worlds) by setting aside and not over-subscribing CPUs for virtual functions will leave them available for hypervisor use.

   3. Even if the Delphix VM is the only VM on a host, the hypervisor is still active and essential; and still needs resources.

4. BIOS Power Management should be set to High Performance where ESXi controls power management.

   1. It can be impacted by VMware KB 1018206 - poor VM application performance caused by power management settings.

   2. Ensure that all BIOS-managed C-States other than C0 are disabled if power management is hardware controlled.

   3. Ensure that all ACPI sleep states above S0 are disabled in the BIOS.

   4. Examples of popular server lines from Cisco, HP, Dell are below. Specific models will vary, use the appropriate spec sheet.

      1. UCS: disable the Processor Power States, disable Power Technology, set Energy Performance to "Performance"

      2. HP Proliant: set HP Power Regulator to HP "Static High Performance" mode

      3. Dell: set BIOS System Profile to "Performance Optimized" mode

5. VMware HA can be enabled; VMware DRS is generally disabled.

6. Blade/Rack Server Firmware and ESXi Drivers should be updated to the latest versions.

7. For Intel®-based servers with E5-2600 v2 processors.

8. Two typical server configurations:

   1. Blade Farm 

   2. Rack Server 
      Hyper-Converged configurations are possible for high performance.

Virtual machine guest

1. For VM machine settings, see Virtual Machine Requirements for VMware Platform.

2. VMWare Guest Specifications:

   1. Minimum: 8 vCPU x 64 GB
      Small: 8 vCPU x 128GB
      Medium: 16 vCPU x 256 GB
      Large: 24 vCPU x 512 GB

   2. Reserve 100% of RAM and CPU:

      1. If the ESX host is dedicated to Delphix, CPU and RAM reservations are advised but not necessary, however, swap space will be required on the hypervisor to compensate for the lack of reserved RAM.

   3. Hyperthreading - See the ESX host section at the top. Disable HT Sharing on VM, disable HT on ESX Host.

3. Assign single-core sockets for vCPUs in all cases. If there is a compelling reason to use multi-core vCPUs, reference the following article from VMware which describes matching virtual multi-core sockets to the hardware ESX is running on.
   VMware Article on CoresPerSocket
   Example: ESXi Host has 2 socket x 18 core Intel Xeon, Delphix Engine wants 16 vCPU.
   Configure Delphix VM with 2 Virtual Sockets, 8 Cores Per Socket to utilize hardware architecture.

4. Avoid placing other extremely active VMs on the same ESX host.

5. Monitoring - vSphere Threshold Alerts for CPU, Network, Memory, Capacity.

6. To set the number of vCPUs per virtual machine via the vSphere client, please see "Virtual CPU Configuration" in the Administration guide:

   1. Delphix VM CPU Utilization - Delphix KB article on what makes Delphix VMs similar to other resource-intensive applications

   2. Exchange on VMware Best Practices - VMworld 2013 session

   3. ESXTOP Reference, Blog

7. Ensure that the latest available VMware drivers and firmware versions are installed for HBAs, NICs, and any other hardware components configured on the Delphix virtual machine. This is a critical step that can have a massive impact on the performance and robustness of our solution.