Expert Reference Series of White Papers
Recommended Use of
Resource Pools in
VMware vSphere
DRS Clusters
1-800-COURSESwww.globalknowledge.com
Recommended Use of Resource Pools
in VMware vSphere DRS Clusters
John A. Davis, VMware Certified Instructor (VCI), VCP5-DV, VCP5-DT,
VCAP5-DCA, VCAP5-DCD
Introduction
Resource Pools are often misunderstood, disliked, and untrusted by vSphere Administrators, who have been
burned due to unexpected results of improperly configured resource pools. Symptoms of such results include
the inability to power on required virtual machines (VMs), the accidental inhibition from accessing available
processor and memory resources, and the granting of high priority to VMs that were intended to get low priority.
Because of such results, many administrators, perhaps most, avoid utilizing resource pools. However, resource
pools can be very useful tools for administrators, who want to configure resource management without having to individually configure each VM. This leads to the administrator’s desire to explore the proper usage of
resource pools.
This white paper examines several scenarios based on actual customer implementations where resource pools
were expected to be useful. Some scenarios describe examples of poorly configured pools. These examples
include a description of the undesired results and recommendations for correcting the situation. Other scenarios
describe examples of resource pools that were well-configured to obtain a desired result. First, here’s a brief
explanation of how Shares, Limits, and Reservations settings on a resource pool are applied.
Resource Pool – Basic Concepts
Resource pools are a type of container that can be utilized
to organize VMs, much like folders. But what makes
resource pools unique, is that they can be used to implement resource controls, including Shares, Limits, and
Reservations on CPU and RAM usage. Limits establish a
hard cap on the resource usage. For example, a resource
pool whose CPU Limit is set to 2 GHz, restricts the concurrent CPU usage of all VMs in the pool to a maximum of
2 GHz collectively, although some physical CPU capacity
may remain unused. Reservations establish a minimum
guarantee of resource usage. For example, a resource pool
whose RAM Reservation is set to 2 GB guarantees the
Figure 1
concurrent RAM usage of all VMs in the pool to a minimum of 2 GB collectively, regardless of the demand from
VMs outside the resource pool. Shares establish a relative priority on resource usage that is only applied during
periods of resource contention. For example, one VM is configured with 500 CPU Shares and another is config-
Copyright ©2013 Global Knowledge Training LLC. All rights reserved.
2
ured with four times as many, or 2000, CPU Shares, as shown in Figure 1. These settings are completely ignored
unless CPU contention occurs. During contention, the VM with 2000 CPU Shares will be granted four times the
available CPU cycles than the VM with 500 CPU Shares.
For more details on controlling resources using Shares, Limits, and Reservations, see Chapter 2 – “Configuring
Resource Allocations” in the vSphere Resource Management Guide, which is available for download from the
vSphere 5.1 Documentation Center web portal. (http://pubs.vmware.com/vsphere-51/index.jsp).
Scenario One – Prioritizing VMs
One reoccurring issue that has plagued many administrators is related to the attempt to assign higher CPU or
RAM priority to a set of VMs. For example, an administrator created two resource pools, one named “High Priority” and one named “Low Priority,” and configured CPU and RAM Shares on each pool that corresponds to their
names. The CPU and RAM Shares on the “High Priority” pool are set to High, and the Shares on the “Low Priority” pool are set to Low. The administrator understood that Shares apply a priority only when the corresponding
resources are under contention, so he expected that under normal conditions, all VMs get the CPU and RAM
resources they request. But, he expected that if processor or memory contention occurs, then each VM in the
High Priority pool would get a greater share of the resources than each VM in the Low Priority pool. He expected
that during contention, the performance of the High Priority VMs may remain rather steady, while the Low Priority VMs may noticeably begin to move slowly. Unfortunately, the opposite result was realized, where the VMs in
the Low Priority pool actually began to run faster than the VMs in the High Priority Pool.
The real cause of this problem was that CPU and RAM shares were configured for the resource pools without
the administrator fully understanding the impact of shares. In this case, the administrator assumed that setting
High CPU Shares on a resource pool containing 50 VMs is equivalent to setting High CPU Shares on each VM
individually. But this assumption is incorrect. To understand how Shares were truly applied, consider the following example.
The administrator creates two resource pools that are configured as follows and illustrated in Figure 2.
• H
igh Priority Pool – CPU share are set to
“High.”
• L ow Priority Pool – CPU shares are set to
“Low.”
• T he High Priority Pool contains 100 VMs, each
with one virtual CPU and default resource
settings.
• T he Low Priority Pool contains 10 VMs, each
with one virtual CPU and default resource
settings.
Figure 2
Copyright ©2013 Global Knowledge Training LLC. All rights reserved.
3
Naturally, under normal conditions, where no CPU contention exists, these Share settings are ignored and have
no impact on the performance of the VMs. Whenever CPU contention does occur, where one or more VMs are in
active competition with other VMs for available CPU time, then the Shares are applied relatively, based on the
number of Shares for each component. When Shares are applied, they are automatically applied to objects or
containers at the same level of the inventory hierarchy, then the Shares are applied to the sub-level or children
objects.
In this example, the only two objects that reside at the top level of the cluster are the resource pools, High Priority and Low Priority. The value of the Shares at the High Priority is set to High, which is the equivalent of 8,000
shares. The value of the Shares at the Low Priority is set to Low, which is the equivalent of 2000 shares. This
guarantees that under CPU contention, the High Priority Pool will receive at least 80 percent of the available
CPU resources, while the Low Priority Pool, will receive at least 20 percent of the available CPU. If the High Priority pool contains 100 VMs, then each VM in the High Pool competes for the CPU time that has been assigned
to the pool. In this case, each VM receives 1 percent of the CPU time assigned to the High Priority Pool or 0.8
percent. If the Low Priority resource pool contains just ten VMs, then each VM in the pool competes for the CPU
time that has been assigned to the entire pool. In this case, each VM receives 2 percent of the available CPU,
which is equivalent to 250 percent of the assignment to each High Priority VM.
The main issue is that the administrator intended to give higher priority and higher resource access to a set
of VMs, but wound up actually providing lower priority to each these VMs. The High Priority Pool did, in fact,
receive four times the amount the amount of CPU than the Low Priority Pool, or 80 percent of the available
CPU. But, since the High Priority Pool contained ten times the number of VMs than Low Priority pool, the CPU resources in the Production Pool had to be evenly divided and spread to all its VMs, resulting in lower per-virtualCPU allocation. Notice that the use of the High Priority and Low Priority pools in this example would actually
work nicely, if each pool contained nearly the same number of VMs.
Recommendation: In this scenario, a better approach may be for the administrator to avoid creating any
resource pools. Instead, simply configure each individual VM with either High or Low Shares. This plan requires
one extra configuration step while deploying each new VM, but it is likely the best method.
Another option to consider is to customize the Shares on the two pools to account for the fact that the High
Priority Pool contains tens times the number of VMs as the Low Priority Pool. In this case, multiply the number of
Shares on the High Priority Pool by ten, because the High Priority pool contains tens of times the number of VMs
as the Test pool. In other words, modify the CPU and RAM shares on the High Priority pool from High (8,000
shares) to a custom value of 80,000. This would provide each High Priority VM with 80 CPU shares or four times
the CPU Shares of each Low Priority VM. Note, that this plan requires attention, such that if the ratio of High
Priority VMs to Low Priority VMs changes significantly, then the Shares values should be adjusted accordingly.
Scenario Two – Misplacement
Another common issue is that VMs are placed outside of any resource pools, leaving them at the same level as
the highest level of user-created resource pools. For example, an administrator created two VMs named VM-1
and VM-2. The CPU shares on these VMs were left at the default “Normal” value, which is equivalent to 1000
CPU shares. These two VMs were placed at the top level of a DRS Cluster along side two resources pools, as
Copyright ©2013 Global Knowledge Training LLC. All rights reserved.
4
shown in Figure 3. One resource pool was named Sales and the other was named Finance, each having Normal
Shares, which is equivalent to 4,000 CPU shares. The Limit and Reservation settings of each pool and VMs were
left at default values. The administrator simply intended to implement containers for Sales and Finance VMs that
could be used in the future to configure resources, but currently he did not intend to actually make any resource
control settings.
Figure 3
The administrator was shocked that when under a period of heavy CPU usage, the Sales and Finance VMs appeared to drag excessively, yet VM-1 and VM 2 continued working normally. Eventually, the problem was traced
to the allocation of the CPU Shares. Each VM in the Sales and Finance Pools had obtained a much smaller number of CPU Shares than VM-1 and VM-2.
Analysis: This resulting total of shares at the cluster level was 10,000 (1000 CPU shares for each of the two
VMs and 4,000 CPU shares for each of the two resource pools). This means that each of the resource pools was
guaranteed to get at least 40 percent of the available CPU capacity of the cluster and each of the two VMs was
guaranteed to get 10 percent. Fifty VMs were placed in each of pools. Each of the VMs placed inside the pools
was guaranteed to get 40 percent divided by 50 equals 0.8 percent of the available CPU capacity, while VM-1
and VM-2 were each guaranteed 10 percent. Another way to view this is that each of the 50 VMs in the Sales
pool received 4000 CPU Shares divided by 50 VMs or 80 CPU Shares. (Note: Each of these VMs was configured
with one CPU and default CPU Shares, so they shared the pool’s CPU Shares evenly). VM-1 owned 12.5 times as
many CPU Shares as each of the Sales VMs. Under contention, the VM-1 was granted 12.5 times the amount of
CPU time as the Sales VMs.
Recommendation: If resource pools are used, ensure that each VM is placed in a resource pool and not outside of all resource pools. If nested pools (child pools) are used, then extend the recommendation to the parent
pool. For example, if a parent pool named Production contains two child pools named Tier-1 and Tier-2, then do
not place any VMs directly in the Production pool; instead, place them in either the Tier-1 or Tier-2 child pool.
Copyright ©2013 Global Knowledge Training LLC. All rights reserved.
5
Scenario Three – Organizing VMs
Another common issue is where administrators intend to use resource pools strictly for organization and administration. They did not intend to affect the resource usage of the VMs in any manner, so each pool is set with
default Shares, Reservation, and Limit for both CPU and RAM. The administrator utilizes the pool for administration purposes, such as to configure permissions and alarms, but the pools are not used to configure resource settings. This results in each pool having the same Shares for CPU and Memory, but if one pool contains twice the
number of VMs that another pool, then each VM in the first pool is effectively guaranteed only 50 percent of the
amount of CPU and RAM is guaranteed to VMs in the second pool.
Recommendation: In this scenario, instead of configuring resource pools for administration and organization
purposes, the administrator should configure VM folders. Permissions and alarms could be configured on each
folder instead of on resource pools. If the administrator also plans to configure resource controls as well as permissions and alarms, then resource pools could be utilized instead of folders, but careful attention must be paid
to setting the Shares, Limits, and Reservations. In this case, it may be simplest to manually configure the Reservation on each pool to guarantee a minimum amount of CPU and RAM that is expected to be required for the
VMs in the pool. This approach ensures the pool gets a configured minimum of CPU and RAM without having to
compete for it by applying Shares.
Scenario Four – Paying for Resources
Resource Pools have been great tools to manage resources for large companies, where each department was
used to buying their own resources, but was forced to move to a shared resources model. In this case, each
department was accustomed to purchasing their own servers that the Information Technology (IT) Department
managed. In the new model, all servers were virtualized onto the same hardware and each department was
charged for the CPU, RAM, and other resources they expected to use. The managers of the departments wanted
some assurance that they would be provided the resources for which they paid.
For example, a certain large health care company,
with dozens of departments, needed to address this
challenge. The Human Resources (HR) Department
paid four times as much for IT as the Sales Department. Naturally, HR expected to obtain four times
as many resources. IT created two resource pools,
one for each of these departments, where the CPU
and RAM Reservation of the HR pool was configured
to be four times the corresponding values of the
Sales Pool, as shown in Figure 4. (The actual values
were based on the results of an assessment of the
expected CPU and RAM usage for each department).
Figure 4
Additionally, IT decided to configure Limits on each
department’s resource pool to ensure they could not access more CPU and RAM than for which they paid.
Copyright ©2013 Global Knowledge Training LLC. All rights reserved.
6
Fundamentally, this approach is similar to what VMware vCloud Director does for tenants in a cloud driven by
vCloud and vSphere. It automatically creates resource pools for each tenant and applies appropriate resource
control settings. It also includes a chargeback feature to charge customers for what they actually used. For
vSphere environments that do not have vCloud or other chargeback systems, manually configuring resource
pools can be very effective, as it was for this health care company.
Scenario Five – New Test Environment
A customer had 100 traditional, physical, Windows servers in production that they wanted to migrate into a new
vSphere environment. They never had a dedicated test environment, but now they wish to create a test environment utilizing VMs. They intended to duplicate each production server to create 100 test virtual servers, each
configured identically to its production counterpart, except they would run in an isolated network.
An assessment of the existing environment provides the following capacity and peak usage data. (The peak usage was based the highest measured values taken at 5-minute samples during a 30-day period)
• The total capacity of all the CPUs in the currently utilized environment was 480 GHz.
• The total capacity of the entire RAM in the currently utilized environment was 400 GB.
• The peak measured CPU utilization was 160 GHz.
• The peak measured RAM utilization was 180 GB.
Although each test VM was configured with the same CPU and RAM capacity as their production counterparts,
the customer did not expect the test VMs to be under the same workload. In fact, they estimated that the typical
test VM workload would be about 25 percent of the production workload.
Based the their goals to provide plenty of resources to meet the need for peak demand, expected growth,
overhead, and spare capacity, the customer calculated they should implement 224 GHz total CPU and 270 GB
RAM for production use. Since they expected that the new test environment would require 25 percent of the
resources required by production, they decided to add 56 GHz and 68 GB for test, resulting in totals of 280 GHz
and 338 GB.
To ensure that the Test VMs and Production VMs would run nicely on the same hardware, the administrator
planned to utilize resource pools. During design sessions, several options for configuring the pools were considered. Each option involved creating two resource pools, one named Production and one named Test, at the top
level of a DRS Cluster. Each of these options were configured and tested in a lab environment using software to
produce processor and memory workloads in sets of VMs, causing heavy resource demand and contention. This
allowed the administrator to compare the results of each configuration and make the best design choice.
Option 1
The pools were configured as follows and illustrated in Figure 5.
• Production Pool: CPU Shares = High (8,000 Shares)
• Production Pool: RAM Shares = High (8,000 Shares)
Copyright ©2013 Global Knowledge Training LLC. All rights reserved.
7
• Test Pool: CPU Shares = Low (2,000 Shares)
• Test Pool: RAM Shares = Low (2,000 Shares)
The tests showed that this configuration ensured that
during periods of contention, the Production VMs
collectively received at least 80 percent of the CPU
and RAM that was available. Also, the Test VMs collectively received 25 percent of the amount accessed
by the Production VMs. Yet, during periods of heavy
demand in the Production VMs and idle workload in
Figure 5
the Test VMs, the Production VMs were able to access
resource beyond what was originally planned. In
other words, nothing prevented Production VMs from accessing CPU and RAM that were originally planned and
added for the Test VMs. Likewise, during periods of idle workload in the Production VMs, the Test VMs were able
to access resources that were originally planned for Production VMs.
Option 2
The pools were configured as follows and illustrated in Figure 6.
• Production Pool: CPU Reservation = 224 GHz
• Production Pool: RAM Reservation = 270 GB
•
Test Pool: CPU Limit = 56 GHz
• Test Pool: RAM Limit = 68 GB
The tests showed that this configuration ensured the
Production Pool always received at least 224 GHz CPU
and 270 GB RAM whenever its VMs attempted to access that much or more. The Production Pool was able
to access additional CPU and RAM during periods of
Figure 6
low demand from Test VMs; however, the Test Pool was
never able to utilize more than 56 GHz CPU and 68GB RAM, even during periods of low demand from Production. In other words, this configuration ensured that adequate memory was guaranteed for the entire set of
Production VMs and a hard cap was established for the entire set of Test VMs. Although this configuration could
be useful, an argument can be made that these settings are somewhat redundant. In this configuration, the Test
VMs still may not access CPU beyond its limit, even if the current Production workload is low and available CPU
resource exist. This means that Test VMs could be running slowly although unused CPU is available.
Option 3
The pools were configured as follows and illustrated in Figure 7.
• Production CPU Reservation – 160 GHz
• Production RAM reservation – 180GB
• Production CPU and RAM Shares – High
Copyright ©2013 Global Knowledge Training LLC. All rights reserved.
8
• Test CPU reservation – 40 GHz
• Test RAM reservation – 45 GB
• Test CPU and RAM Shares – Low
The Production pool’s CPU Reservations were set to
match to the estimated peak CPU and RAM utilization, without including the growth, overhead, and
spare capacity. The CPU and RAM Reservations for
the Test pool were set to be 25 percent of the corresponding reservation for the production pool. AddiFigure 7
tionally the pools were configured to allow pools to
access resources beyond their reservation, such that
they would compete for available resources during periods of contention, based on priority.
The tests showed that this configuration ensured that each pool was able to access at least their reserved
resources during periods of heavy workload. Additionally, each pool was able to access more as needed (as
expected for growth and overhead). Whenever the workload grew to a point to produce resource contention,
the Production Pool was allocated 80 percent of the remaining resources and the Test Pool was allocated 20
percent.
Option 4
The pools were configured as follows and illustrated in Figure 8.
• Production CPU Reservation – 160 GHz
• Production RAM Reservation – 180 GB
• Production CPU Limit – 224
• Production RAM Limit - 270
• Test CPU Reservation – 40 GHz
• Test RAM Reservation – 45 GB
• Test CPU Limit – 56 GHz
• Test RAM Limit – 68 GB
Figure 8
The tests showed that this configuration ensured that the Production and Test servers were guaranteed sufficient resources corresponding to the expected peak workload to be generated within the VMs. Additionally,
each pool was able to access additional resources that were planned for overhead, growth, and spare capacity.
The test showed that neither pool could access more CPU and RAM than their configured Limits, which corresponded to the amount of resources that were originally planned for each pool, including overhead, growth, and
redundancy.
Copyright ©2013 Global Knowledge Training LLC. All rights reserved.
9
Eventually, Option 1 was selected, where the Production Pool was assigned High CPU and RAM Shares, while
the Test Pool was assigned Low CPU and RAM Shares. It was selected because it was considered to be simple
and flexible, yet effective. One key is that each of the pools would always contain the identical number of VMs,
configured in the same manner. This meant the issue described previously in Scenario 1 – Prioritizing VMs would
not apply.
Option 3 was considered to be the next-best choice, but it was expected that the Reservations on each pool
would have to be modified over time, as new VMs were added. Option 4 was considered to be too complex.
Scenario Six – Building Blocks
An assessment revealed that in the customer’s current physical environment, the total RAM usage of the Windows servers during times of peak demand was 128 GB, which was 67 percent of the total RAM capacity of 192
GB. This led to a decision to allow 150 percent RAM over commitment for the Windows VMs. After analyzing the
existing servers and planning for growth for three years, the predicted total configured RAM for all VMs was 450
GB. But, to allow 150 percent over commitment, only 67 percent of the estimated, or 300 GB, was procured for
direct use by VMs. More RAM was actually purchased to allow overhead and spare capacity for ESXi downtimes,
but only 300 GB was planned for direct VM usage.
Initially, the administrator was concerned over commitment, so to build
some trust, resource pools were implemented. The first phase of VM
deployment involved just ten Windows servers. He chose to create a
resource pool named Pool-01 with RAM Reservation set to 67 percent
of the total configured RAM for the VMs. This was done with the intent
to guarantee the pool will be able to access real memory that its VMs
are expected to need. He set the RAM Limit to equal the Reservation
plus extra RAM to allow for the overhead of each VM. Since the total
configured RAM of the first ten VMs was 15 GB, the Reservation was set
to 10 GB (67 percent of 15 GB) and the Limit was set to 11 GB, allowing
100 MB overhead per VM, as illustrated in Figure 9.
Figure 9
Although access to available CPU and RAM resources were limited on the VMs, the administrator saw that the
performance was still very satisfactory under actual workload.
The administrator chose to continue using this “build block” approach going forward. For each new wave of
servers that he deployed, he chose to create a resource pool, whose RAM Reservation and Limit was calculated
and set using the same formula. Later, he expanded the concept to allow for other over commitment targets. For
example, when he analyzed a particular set of physical Web Servers, he saw that they currently only utilized 50
percent of the configured RAM during times of peak usage. He chose to configure a pool for these VMs whose
limit was set to 50 percent of the total configured RAM plus room for overhead.
Although this approach is not commonly used in the field, it was effective for this administrator. It allowed him
to proactively determine if the VMs would behave well in the future, even if RAM contention occurred. Basi-
Copyright ©2013 Global Knowledge Training LLC. All rights reserved.
10
cally, he configured limits to force contention to the planned maximum level and verified that the applications
still performed well. Whenever a set of VMs did not perform well in their pool, he would increase the Limit and
would accordingly adjust his capacity plans. Such adjustments could have led to deploying fewer VMs to the
cluster than originally planned or purchasing more hardware.
Summary
Resource pools can be very useful tools, but care must be given to planning and configuring the pools. In many
cases, it may be best not to create any resource pools. But in the examples provided in this paper, resource pools
were very helpful in meeting a specific need. In other examples, lessons were learned concerning the functionality and proper use of resource pools.
Learn More
To learn more about how you can improve productivity, enhance efficiency, and sharpen your competitive edge,
Global Knowledge suggests the following courses:
VMware vSphere: Install, Configure and Manage [V5.1]
VMware vSphere: Fast Track [V5.1]
VMware vSphere: Optimize and Scale [V5.1]
Visit www.globalknowledge.com or call 1-800-COURSES (1-800-268-7737) to speak with a Global
Knowledge training advisor.
About the Author
John A. Davis has been a VMware Certified Instructor (VCI) and VMware Certified Professional (VCP) since 2004,
when only a dozen or so VCIs existed in the USA. He has traveled to many cities in the USA, Canada, Singapore,
Japan, Australia, and New Zealand to teach. He splits his time between teaching and delivering professional
consulting services that are 100 percent focused on VMware technology.
Copyright ©2013 Global Knowledge Training LLC. All rights reserved.
11