FrankBrix

Good news, I was just browsing the VMware website and I saw that vSphere 5.5 is available for download. I have been part of the beta since early this year and there are so many features I have been looking forward to. Just to name a few.

• Much improved vCenter Web Client
• vFlash – Put local SSD/Flash in your ESXi servers and cache most reads to offload your storage array
• Improved Single Sign On (SSO) Server
• Support for 62TB .vmdk files

You can read about what else is new here

Head over to the VMware website and start your download. Personally I can’t wait to start working with this version!

It is important to set the right load balancing policy on your standard or distributed vSwitch. In this article I will talk about how each policy work and give my general recommendation. 

In vSphere we have the following policies to choose from:

• Route based on originating virtual port ID
  
• Route based on IP hash
• Route based on source MAC hash
• Use Explicit failover order
• Route based on physical NIC load (Distributed Switch only)

The originating virtual port ID is the default policy and there is a good reason for that. This policy does not require any special physical switch configuration. In fact you can use all policies except the one based on IP hash without configuring your physical switches with cisco etherchannels or hp trunks. The IP hash policy differs in the way that it let a virtual machine spread its traffic over two or more different vmnic’s (physical network adapter). Because of this the MAC address of the virtual machine will appear on multiple physical switch ports simultaneously. This is why it is required to use ether-channel or trunk the links. 

Lets take an example. We have a virtual switch with a port group. We have 4 virtual machines connected to the port group and 3 physical NICS connected to the virtual switch.

Route based on originating virtual port ID works kind of like round robin. VM-A will use vmnic1, VM-B will use vmnic-2, VM-C will use vmnic-3 and VM-D will use vmnic1. The virtual machines are just balanced over the physical available NICS. The virtual machines will have its network running on the designated physical NIC and only that NIC. Of course if we have a failure it will fail over to a different vmnic. 

The good about this configuration is that it is easy to configure. It requires not special physical switch configuration.  The bad is that VM-A and VM-D how are both utilizing the same vmnic could be the two most network intensive virtual machines and they compete for bandwidth while VM-B and VM-C does nothing on their respective vmnics.

Route based on IP hash requires physical switch configuration. (Ether-channel). In this configuration the traffic flows according to the source IP and destination IP. In this case if VM-A is communicating with three different IP’s (clients, other servers etc.) the traffic can use all available vmnic’s. If the VM-B is only communicating with one destination the traffic will not be shared.

Route based on phyiscal NIC load is a newer addition. It quite similar to virtual port ID. But with the addition that it actually looks at the physical NIC load! This means that if VM-A and VM-B both are using a lot of bandwidth on the same physical NIC one of the will be moved to another! If you are running the originating virtual port ID policy on a distributed switch you should change to physical nic load right away.

There are a lof of information out there on this subject. This was just a brief introduction.

Introduction

A key concept that is very important to understand for the VMware administrator is how Shares, Limits and Reservations work. Why do we have these controls? Well to put it simply if we are over provisioning our ESXi host on memory and CPU we need a tool to make sure that the right machines get the correct amount of resources. An example could be: Make sure the important ERP system always gets 3000MHZ (Reservaton) and make sure that the Test system never gets more than 1000MHZ (Limit). This article will focus on the three concepts on Memory and CPU. You can also apply some of this on storage and network. But that is out of scope for this article.

Rerservations

A reservation is a guarantee on either memory or cpu for a virtual machine. You define the reservation in MB or MHZ. It works kind of different on the two compute resources.

On memory it is a guarantee for access to physical memory for the virtual machine. Remember, every virtual machine has a swap file. The swap file size is defined as (swap file = configured memory – memory reservation). When the virtual machine is running the vmkernel allocates memory to it when the VM requests it. The vmkernel will always try to map the memory to physical memory, but if the ESXi host is running low on memory the memory has to come from the swap file. This has a huge performance penalty. Lets take two examples:

• Example 1: You have a virtual machine configured with 2 GB memory and you configure a 1 GB reservation
  • When the virtual machine powers on a 1 GB swap file (.vswp) is created on a datastore
  • The 1 GB reservation guarantees that the VM will always a  least get 1 GB of physical memory. If the ESXi host is running low the remaining 1 GB can come from the swap file on disk!
• Example 2: You have a virtual machine configured with 4 GB memory and you configure a 4 GB reservation
  • When the virtual machine powers on a swap file with zero size is created
  • The 4 GB reservation guarantees that the VM will get ALL its memory from physical memory and it will never do hypervisor swapping or ballooning.

On cpu the reservation is a guarantee for clock cycles. You define the reservation in MHZ. If you give a virtual machine a reservation it means the vmkernel CPU scheduler will give it at least that amount of resources. If a virtual machine is not using its resources the CPU cycles are not wasted on the physical host. Other machines can use it. What you do with CPU reservations is making sure that a VM will always get access to physical cpu in a committed environment. 

Limits

A limit is a a limit 🙂 Defined on either Memory or CPU It is defined in MB or MHZ.

On memory the limit defines what is the maximum amount of physical memory the virtual machine can use. This is a very dangerous setting! If you set the limit lower than the configured memory for a VM it will cause swapping and balloon activity for the virtual machine.

• Example 1: You have a virtual machine configured with 4 GB memory and you configure a 1 GB limit
  • What you have done is BAD, unless you want bad performance. The virtual machine guest operating system sees 4GB of memory (inside windows task manager for instance) but the ESXi is not allowed to give it more than 1 GB of physical memory. The virtual machine will probably request more that 1 GB for its application and when this happens ballooning and hypervisor swapping will start.

Limits on memory can have huge consequences. As a consultant I always look for virtual machines with a limit on memory, more often than you would imagine you find it configured on a few virtual machines! 

On CPU the limit defines how much access a virtual cpu can get on a physical cpu (core). This is used to make sure a virtual machine is not using too much resources on a host. By limiting the vCPU you essentially also limits the performance of the virtual machine. Even though capacity is available on the ESXi host the limit will still be enforced.

Shares

Shares is a different approach to performance tuning in a virtual environment. Shares define how much access you get to a resource compared to something else. Every virtual machine has 1000 shares configured pr. vCPU as a default. So you are already using them! All virtual machine are equal from a hypervisor perspective unless you change the shares and tell it which machines are really more important. What is important to know about shares is that they only is considered in case of contention! If you have available capacity for all machines it does not help performance to increase the shares on some machines.

• Example 1: VM A has 1000 shares and VM B has 1000 shares and they are both competing for the same physical cpu core. In this case the vmkernel CPU scheduler will give each machine 1/2 or 50% access and they will have the same performance
• Example 2: VM A has 3000 shares and VM B has 1000 shares and they are both competing for the same physical cpu core the result would be that VM A gets 3/4 or 75% access and VM B gets 1/4 or 25% access.
• Example 3: VM A has 3000 shares and VM B has 1000 shares and they are not competing for the same physical cpu core. In this case both machines will get 100% access to physical CPU. Remember shares is only handled when we have contention!

Summary

Reservations, limits and shares are powerful resource controls in a virtual environment. Just make sure to think about how you implement it. Especially watch out for putting a memory limit on your virtual machine.

If you have any questions, please use the comments on this page.

VMware ballooning is one of the hard concept to grasp. There are a lot of misunderstanding out there about this feature. I have been discussing this feature with customers and students during the last 5 years. This is my attempt to explain balloning.

VMware ballooning is a memory reclamation  technique used when and ESXi host is running low on memory. You should not see balloning if your hosts is performing like it should. To understand ballooning we would have to take a look at the following picture:

 This picture shows the three levels of memory in a virtual environment. In a physical world we would only have the two top levels (virtual memory & guest physical memory) but in the virtual world we also have the host physical memory. What is important to know is that the hypervisor (ESXi) has no knowledge of what is happening inside the virtual machine (grey area). The hypervisor maps memory when the virtual machines asks for it. The hypervisor will then give it memory from “host physical memory” but only if memory is available. If memory is not available the memory can med mapped to the .vswp file on a vmfs or nfs datastore. The virtual machine has no knowledge if the memory is mapped to physical memory or to a disk. This is called hypervisor swapping, and this is the last resort for the vmkernel to use this mechanism.

Ballooning in short is a process where the hypervisor reclaims  memory back from the virtual machine. Ballooning is an activity that happens when the ESXi host is running out of physical memory. The demand of the virtual machine is too high for the host to handle.

Lets take a high level example:

1. Inside a virtual machine you start an application. For instance solitaire
2. solitaire as an application will ask the guest operating system (in this case windows) for memory. Windows will give it memory and map it from the virtual memory -> guest physical memory 
3. what happens next is that the hypervisor sees the request for memory and the hypervisor maps guest physical memory -> host physical memory
4. Now everything is perfect. You play solataire for a few hours. And then you close it down.
5. When you close solitaire the guest operating system will mark the memory as “free” and make it available for other applications. BUT since the hypervisor does not have access to Windows’ “free memory” list the memory will still be mapped in “host physical memory” and putting memory load on the ESXi host.
6. This is where ballooning comes into place. In case of an ESXi host running low on memory the hypervisor will ask the “balloon” driver installed inside the virtual machine (with VMware Tools) to “inflate”
7. The balloon driver will inflate and because it is “inside” the operating system it will start by getting memory from the “free list”. The hypervisor will detect what memory the balloon driver has reclaimed and will free it up on the “host physical memory” layer!

The balloon driver can inflate up to a maximum of 65%. For instance a VM with 1000MB memory the balloon can inflate to 650MB. The way to avoid ballooning is not to uninstall the balloon driver but to create a “Memory Reservation” for the virtual machine. In case of full inflation for this particular VM the result is the hypervisor gets 650MB memory reclaimed. The downfall of this is that you risk your VM to do Guest OS Swapping to its page file! Just remember page file swapping is better than hypervisor swapping. Hypervisor swapping happens without the guest operating system is aware of it. Page file swapping it is the OS that decides what pages to swap to disk!

To check for ballooning you can either open ESXTOP or the vCenter Performance Graphs.