Dynamic Control of Electricity Cost with Power Demand Smoothing and Peak
Shaving for Distributed Internet Data Centers
Jianguo Yao, Xue Liu, Wenbo He and Ashikur Rahman
School of Computer Science
McGill University
Montreal, Canada
Email: {jianguo,xueliu,wenbohe,ashikur}@cs.mcgill.ca
Abstract—Internet based service providers, such as Amazon,
Google, Yahoo etc, build their data centers (IDC) across
multiple regions to provide reliable and low latency of services to clients. Ever-increasing service demand, complexity
of services and growing client population cause enormous
power consumptions by these IDCs incuring a major part
of their running costs. Modern electric power grid provides
a feasible way to dynamically and efficiently manage the
electricity cost of distributed IDCs based on the Locational
Marginal Pricing (LMP) policy. While recent works exploit
LMP by electricity-price based geographic load distribution,
the dynamic workload and high volatility of electricity prices
induce highly volatile power demand and critical power peak
problem. The benefit of cost minimization via geographic load
distribution is counterbalanced with the high cost incurred by
violating the peak power. In this paper, we study the dynamic
control of electricity cost to provide low volatility in power
demand and shaving of power peaks. To this end, a Model
Predictive Control (MPC) electricity cost minimization problem
is formulated based on a time-continuous differential model.
The proposed solution minimizes electricity costs, provides
low variation in power demand by penalizing the change
in workload and alleviates the power peaks by tracking the
available power budget. By providing extensive simulation
results based on real-life electricity price traces we show the
effectiveness of our approach.
I. I NTRODUCTION
The computing needs of large-scale Internet service
providers are often supported by dedicated Internet data
centers (IDC) containing massive number of servers, largescale storage and heterogeneous networking elements together with an infrastructure to distribute power and provide
cooling. As computation and storage continue to move into
the cloud IDCs computing infrastructure sizes and energy
consumptions are rapidly increasing in concert. Recent studies show that large-scale IDCs consume a power as high
as dozens of MW incurring a cost of dollar 5.6M [1], [2]; the
Environmental Protection Agency (EPA) of USA estimated
the annual data center electricity consumptions to be over
100 billion kWh at a cost of dollar 7.4 billion by 2011
[3]. Thus minimizing electricity consumptions and costs of
various IDCs are of paramount importance and provide great
benefits to the Internet based service operators.
Although research on minimizing power consumptions
has passed almost a decade, the research on minimizing
power costs is still in its early stage. Some of the recent
works on minimizing power costs [5], [6], [7] are based on
an interesting observation on electricity prices which is as
follows. The price of electricity in United States deregulated
markets exhibit both temporal and spatial diversity across
multiple regions. As IDCs of Internet service operators are
also geographically distributed across multiple regions (in
order to provide better QoS and reliability of services),
a potential electricity cost savings could be achieved if
workload is distributed based on the electricity price information of the regions where IDCs are located. Based
on this observation various optimization frameworks for
load balancing over geographically distributed data centers
are developed. However these solutions can only partially
minimize the power costs and bring some new problems as
we address next.
First of all, due to electricity-price based biased workload distribution, the IDCs located at relatively cheaper
electricity price regions tend to meet high power peaks.
As the electricity prices are highly volatile in multi-region
electricity markets this high power peak problem becomes
a frequent phenomena for some of the IDCs. With high
power peaks, those IDCs need to subscribe for higher
power delivery capacity which greatly increases their capital
cost for building the infrastructure. Moreover, high spikes
in power demand causes occasional power outages and
potential hazards to physically constrained smart power grid.
In addition, some electricity suppliers impose a peak power
limit on the amount of power draw from the grid that
arises due to some transmission limitations and penalize
those IDCs heavily if this limit is exceeded [10]. Power
peak problem also makes power demand of different IDCs
difficult to predict in advance; consequently IDC operators
become unable to qualify for price rebates by signing up
advance-contracts with the power retailer or hedge against
uncertainty in order to minimize operational risks.
Secondly, migrating instantaneous workload from high
price regions to low price regions does not necessarily
reduce total electricity cost due to the inter-dependency of
IDCs’ power demand and electricity price levels in the multiregion electricity market. The price of electricity based on
real time pricing (RTP) varies during the day depending
on usage pattern. Consequently, being a massive consumer
of electricity, the IDCs are in a position to influence the
electricity price levels during the day. When the power
demand of an IDC is adjusted in one time instance, it affects
the price levels in the wholesale market for the next time
instance. Therefore the data center operators no longer act as
simple passive consumers rather they are active consumers
in a sense that they can also influence the price level by
manipulating their demands. This behaviour was first observed in [10]. In addition, we like to argue that geographical
load balancing creates a vicious cycle among electricity
demand, cost and price which can be described as follows.
The volatility of electricity price level causes IDC operators
to adjust their power demands across multiple regions to
minimize the cost which is achieved via geographic load
balancing. The newly adjusted power demand causes the
price level to become more volatile which in turn requires
further adjustment of power demand via redistribution of
workload based on new price levels. This mutual interactions
of demand, cost and price paradoxically continues to create
a vicious cycle. As a result the price levels tend to oscillate
and some times turn to be unacceptably high.
In this paper we consider both power peak problem and
high volatility (i.e. fluctuations) of power demand problem
under the same framework. The volatility of power demand
is formally defined as the rate of change in power demand
and the power peak is the power demand at peak load
during a day. We propose a novel approach of dynamic
electricity cost control for minimizing the electricity cost
with low volatility in power demand and power peak shaving
for distributed IDC under real-time electricity price market.
To this end, a Model Predictive Control (MPC) is exerted
where we formulate the electricity cost model using timecontinuous differential equations. Based on this model the
total electricity cost function is derived as a combination of
electricity cost and the volatility in power demand. Then, the
dynamic control is achieved by the closed-loop MPC based
control by minimizing this cost function. The power peak is
shaved by setting a control reference of MPC at the target
power budget for each IDCs. The feasibility and stability
of this scheme is also illustrated and proved. Moreover,
through extensive simulations based on real-life electricity
price traces we evaluate the efficacy of the proposed scheme.
The rest of this paper is organized as follows. Section
II describes related research works. Section III presents
backgrounds, models and problem formulation. Section IV
provides a feedback control solution scheme with detailed
description of control model formulation, two-time scale
control architecture, MPC control design, optimal control
reference solution and stability analysis. Section V evaluates
the proposed architecture and shows that the effectiveness of
the proposed solution in reducing volatility of power demand
and shaving power peaks. Finally, Section VI concludes the
paper.
II. R ELATED WORKS
Data center power management is an active area of
research over the past decade. Based on the general trend
of all related research works, we can broadly classify them
into two groups:–(i) power consumption management, and
(ii) power cost management.
The research works within the first group operate at
the intra-data center level and aims at reducing power
consumptions by the server firms within a data center.
Most commonly proposed H/W and S/W level power optimization techniques include dynamic voltage and frequency
scaling, processor sleep scheduling, core parking, memorybank parking, hard drive segment parking, virtualization,
server consolidation, load balancing etc. However, minimizing power consumptions at the data center level is beyond
the scope of this work. For a detailed overview of design
challenges for power consumption management at IDCs
please see [12] and the references therein.
The second group of works targets minimizing total
power cost and mostly works at the inter-data center level.
There are myriad of works focusing on electricity cost
minimization. Qureshi et al. [13] present the cost minimizing
techniques in a whole-sale market environment for large
distributed systems. Rao et al. [5] studied the same problem
but in multi-region electricity markets to better capture
the spatial diversity of electricity prices across multiple
regions. Zhang et al. [10] introduce the idea of capping
electricity cost of Internet-scale data centers by classifying
users into premium and ordinary category and providing
QoS to premium users and best-effort services to occasional
users. Yao et al. [9] proposes a solution to reduce power cost
of delay tolerant workloads. The target applications that can
generate delay tolerant workloads are based on MapReduce
programming and include searching, social networking, data
analytics etc. By exploiting temporal and spatial variations
of both workload and electricity prices they provide a power
cost-delay trade off which is further exploited to minimize
power expenses at the cost of service delay. In [4] an
online algorithm for migrating jobs between data centers
based on electricity prices is proposed. The job migration
requires bandwidth intensive migration of the application’s
state and data, consequently, they incorporate bandwidth cost
in the electricity cost optimization problem. In [8] Sankaranarayanan et al. exploit the heterogeneity of data centers for
achieving additional energy efficiency and power savings via
intelligent scheduling of requests to heterogeneous servers at
each data center. In [7] the authors propose an optimization
framework for throughput-intensive applications like web
search engine. They propose a workload shifting algorithm
considering both electricity prices, (i.e. to reduce the energy
IDC 1
cost) and workload of data centers at the time of shifting (i.e.
to reduce response time). Liu et al. [6] address whether the
geographic load balancing can additionally encourage the
use of green energy and reduce the use of brown energy.
To solve this issue they develop a simple power cost model
which is a linear combination of variable electricity prices
and lost revenue due to reduced response time of the system
arising from both network propagational delay and service
delay.
However, none of these works have studied the dynamic
control of electricity costs that can provide controlled variation in power demand and power peak shaving under the
real-time pricing policy for modern power grid which is the
main focus of this paper.
C
L1
λ1 = ∑ λi1
λ11
i =1
M1 = 6
λ1 j
λ1N
m1 = 4
IDC j
λi1
C
λ j = ∑ λij
λij
Li
i =1
Mj =5
λiN
λC1
LC
mj = 3
IDC N
λCj
λCN
C
λN = ∑ λiN
i =1
MN = 4
mN = 2
Figure 1.
Workload allocation architecture for IDCs.
III. M ODELS AND PRELIMINARIES
In this section, we summarize the models to capture the
behaviour of electricity consumption and cost for Internet
Data Centers (IDCs).
A. Workload allocation architecture for IDCs
The workload allocation architecture for IDCs is shown
in Fig. 1. This architecture comprises two types of elements:
the front-end Web portals and IDCs. When a front-end
Web portal receives requests from clients, it distributes
the requests to different IDCs for processing. Without loss
of generality we may assume that there are C front-end
Web portals, and N IDCs located across different regions
in the world. In each IDC j, there are Mj servers. We
further assume a homogeneous system, where each server
has the similar configuration in terms of CPU, memory and
power requirements etc. Each IDC turns ON mj servers to
deal with the client requests. Then we have the following
constraints:
mj ≤ Mj , ∀j = 1, 2, · · · , N.
(1)
The client requests offer a workload Li for each front-end
Web portal i = 1, 2, · · · , C. Only a portion of the workload
λij , i = 1, 2, · · · , C, j = 1, 2, · · · , N is forwarded from
the front-end Web portal i to IDC j. The total workload
constraints for each front-end Web portal is given by:
Li =
N
∑
λij , ∀i ∈ 1, 2, · · · , C.
(2)
j=1
λij ≥ 0.
(3)
For each IDC j, the total received workload from all the
front-end Web portals is denoted by λj which is represented
as follows:
λj =
C
∑
i=1
λij , ∀j = 1, 2, · · · , N.
(4)
B. Energy consumption for each IDC
The power consumption P for an individual active sever
mainly depends on two variables: CPU utilization Ucpu and
frequency f . Other variables indirectly affect the power
consumption through CPU utilization and/or frequency. To
derive a model, mapping CPU utilization and frequency to
power consumption, the curve fitting method is exerted [14]
through a set of experiments, in which the power is measured
by running a server under different CPU utilization and
frequencies. After the experimentation it was shown that
the the power consumption has a linear relationship with
CPU utilization and frequency when other variables are kept
constant. The derived power consumption model for each
server in [14] is given by1 :
P (f, Ucpu ) = a3 f Ucpu + a2 f + a1 Ucpu + a0 ,
(5)
where a0 , a1 , a2 and a3 are the fitting parameters using a
specific curve fitting method.
CPU utilization can be approximated by the workload λ
using the equation Ucpu = λ/f [14]. Hence, the mapping
relationship between the power consumption P (λ) and the
received workload λ for each server is given by:
P (λ) = b1 λ + b0 ,
(6)
where the new fitting parameters b0 and b1 are deduced by
b0 = a2 f + a0 and b1 = a3 + a1 /f , respectively.
Assuming that the frequency of CPU for each server is
fixed and all the frequencies of CPUs are the same in each
IDC, we derive a linear model of total power consumption
Pj (λj ) for IDC j (mj active severs) with respect to offered
total workload λj :
Pj (λj ) = b1 λj + mj b0 .
(7)
1 The total power is consumed by servers, cooling systems and environment components (e.g. network devices and UPS). This paper only considers the power consumption by servers since traditional design separates
the three subsystems.
0
i=1
C. Total Electricity Cost for IDC in modern electric power
grids
The modern electric power grid in North America allows
dynamic pricing of electricity based on different regions,
time of the day and the power demand. The dynamic
electricity price is a promising mechanism to improve the
power grid efficiency, reduce the peak load, and mitigate
wholesale price volatility. Most large-scale IDCs are built in
regions where dynamic electricity prices are available under
competitive electricity market. The dynamic price could be a
function of different regions, peak load, and different hours
in a day etc. In this paper, we use a bottom-up bid-based
stochastic price model [17] to represent the dynamic price:
Prj = function (region, time, load) .
(9)
Note that here we assume that the electricity price only
depends on regions, power load and different hours in a day.
The real-time prices are shown in Fig. 2 for three different
regions in North America namely Michigan, Minnesota and
Wisconsin on October 03, 2011. The electricity prices are
adjusted every hour according to current power load.
100
Michigan
Minnesota
Wisconsin
80
Hence, the total electricity cost for a system comprising
N number of IDCs is summarized as follows:
C̄(t) =
N
∑
Cj (t).
(11)
j=1
D. Workload modelling
Modelling Internet workload is fundamental in predicting
the future workload arrival to take appropriate actions in
advance. The workload is usually modelled as a stochastic
process which captures the future uncertain evolution. Many
class of random processes, such as Markov Modulated Poisson Process (MMPP) [15] and Markovian Arrival Processes
(MAP) [16], are used for fitting the actual web service
workload. In this paper, we use a time-varying pth-order
autoregressive model to model the arrival of future workload,
which is a widely used technique to capture the Markov
process behaviour:
µi (k) =
p
∑
αpi µi (k − s) + ε(k),
(12)
s=1
where α1i (k), · · · , αpi (k) are the fitting parameters, and the
innovations ε(k) are independent and identically distributed
white noise.
The prediction precision mainly depends on the accuracy
of the workload model. To this end, we use Recursive Least
Square (RLS) method for online estimation of the workload
arrival. The RLS method is described in [18].
2000
original
predicted
1500
Request rate
The electricity energy consumption is the time integral
of power consumption. Hence we formulate the electricity
energy consumption Ej (t) through the time integral of total
power consumption Pj (λj ), which is time varying due to the
varying nature of workload λj . The formulation of Ej (t) is
represented by:
∫ t
Pj (λj )dt
Ej (t) =
0
)
∫ t( ∑
C
b1
λij + mj b0 dt.
(8)
=
1000
500
Price ($/MWh)
60
0
40
0
5
10
15
20
25
Hours
20
0
Figure 3.
Original workload vs. predicted workload.
−20
−40
0
5
10
15
20
25
Hour
Figure 2.
Real-time electricity prices.
The electricity cost for each IDC j is the time integral
of the product of real-time electricity price and electricity
energy consumption:
∫ t
Cj (t) =
Prj Ej (t)dt.
(10)
0
We assume that the estimated workload model is represented as:
p
∑
µ̂i (k) =
α̂pi µi (k − s),
(13)
s=1
α̂1i (k), · · ·
, α̂pi (k)
where
are parameters estimated with RLS
method. Given the previous p samplings of workload Li (k−
s), s ∈ 1, · · · p, we can predict the workload for the next
period.
To evaluate the workload prediction method, a numeric
simulation is conducted based on the workload data to an
EPA server on Aug. 30th 19952 . The original and predicted
workloads are drawn in Fig. 3, which clearly shows that
the prediction model can accurately capture the workload
characteristics.
E. Service latency model
In the Internet service, the latency is a major parameter
to evaluate the Quality of Service (QoS). For each IDC,
different queueing models result in different QoS. We use the
M/M/n queueing model to process the incoming workload
from front-end Web portal. According to this model, the
average latency D is represented as D = PQ /(nµ − λ)
where n is the number of servers , λ is the workload arrival
rate, µ is the service rate and PQ is the probability of clients
waiting in the queue. Without loss of generality, we may
assume that there are always client requests waiting in the
queue and the servers at IDCs are always busy to handle the
requests. Hence, we have PQ equals 1.
The actual average latency for IDC j is represented as
follows:
Dja =
1
mj µj − λj
.
(14)
where µj is the service rate for each server in IDC j.
To meet the QoS requirements for all Internet clients,
the following constraints are put on the average processing
latency for IDC j as:
Dja ≤ Dj .
(15)
where Dj is a latency bound for IDC j.
In this section, we formulate the control problem of electricity power consumption and cost. The detailed controller
design is described and analyzed.
A. Control modelling formulation
The time-continuous model of electricity consumption for
IDC j is represented as
Ej (t)
= b1
λij + mj b0 .
dt
i=1
(16)
The time-continuous model of electricity cost for IDC j
is represented as
C̄j (t)
= Prj Ej (t).
(17)
dt
The time-continuous model of total electricity cost for all
IDCs is represented as:
C̄(t) ∑
=
Prj Ej (t).
dt
j=1
N
2 The
Internet Traffic Archive http://ita.ee.lbl.gov/.
Ẋ(t) = AX + BU (t) + F V,
Y (t) = W X(t),
(18)
(19)
(20)
where the system model matrices A ∈ ℜ(N +1)×(N +1) , B ∈
ℜ(N +1)×N C , F ∈ ℜ(N +1)×N and W ∈ ℜ1×(N +1) are given
by:


0 Pr1 Pr2 · · · PrN
 0 0
0 ···
0 


 0 0
0 ···
0 
A=
,
 ..
..
..
.. 
..
 .
.
.
.
. 



B=

0
0
···
0
O1×C
′
b1 I1×C
..
.
O1×C
O1×C
..
.
···
···
..
.
O1×C
O1×C
···




F =


W =
IV. F EEDBACK C ONTROL S OLUTION
C
∑
The state vector is chosen as X(t)
=
[C̄(t), E1 (t), E2 (t), · · · , EN (t)]T , the control input
is given by U (t) = [λij ]T
∈ ℜ(N C)×1 and
V = [m1 , m2 , · · · , mN ]T , and the output is Y (t) = C̄(t).
The state space model is represented as follows:
(
0
b0
0
..
.
0
0
b0
..
.
0
0
1
0
···
···
···
..
.
0
O1×C
O1×C
..
.



,

′
b1 I1×C
0
0
0
..
.




,


· · · b0
)
··· 0 ,
′
where I1×C
= [1, · · · , 1]1×C and O1×C = [0, · · · , 0]1×C .
For the purpose of simplifying design and analysis while
facilitating the implementation, we convert the continuoustime system model to the discrete-time form using digitization method. Most of the engineering applications using
a ZOH with a sampling period Ts , can be represented as
follows:
X(k) = ΦX(k − 1) + ḠU (k − 1) + ΓV (k − 1),
Y (k) = W X(k),
(21)
(22)
where X(k) , X(kTs ) is the system state at the kth sampling period, and the discrete-time system model matrices
are given by:
Φ = eATs ,
∫ h
Ḡ =
eAs Bds,
0
∫ Ts
Γ=
eAs F ds.
(23)
(24)
(25)
0
The control system suffers from the input constraints
which comes from workload and service latency. The workload constraints defined in (2) is represented in matrix form
as:
HU = h,
where the matrix H ∈ ℜC×N C
are given by:

H1 H1
 H2 H2

H= .
..
 ..
.
(26)
and the vector h ∈ ℜC×1
···
···
..
.
H1
H2
..
.



,

HC · · · HC
H i (j) = 1, i = j;
H i (j) = 0, i ̸= j,
)T
L2 · · · LC
.
(27)
HC
{
i
H1×C
h=
(
=
L1
i=1
(28)
(29)
To facilitate representing the delay constraints with respect to inputs, we rewrite (15) as follows:
C
∑
λij ≤ µj (mj −
i=1
1
).
µj Dj
(30)
The matrix form of (30) is given by:
ΨU ≤ φ,
(31)
where the matrix Ψ ∈ ℜN ×N C and the vector φ ∈ ℜN ×1
are represented as:



Ψ=

′
I1×C
O1×C
..
.
O1×C
′
I1×C
..
.
···
···
..
.
O1×C · · ·
1
)].
φ = [µj (mj −
µj Dj
O1×C
O1×C
O1×C
..
.



,

(32)
′
I1×C
(33)
In addition, the assignment workload must be nonnegative. Hence, the following constraint is added:
U ≥ ON C×1
subject to the constraints in (1).
Sleep (ON/OFF) Controllability Condition: This condition is derived from constraints perspective. The workload
capacity for IDC j is λ̄j = Mj µj1−Dj with the assurance
of service latency bound Dj , where Mj indicates that all
available servers are turned ON. The IDCs are capable to
deal with the arriving workload with the assurance of service
latency bound Dj if the arriving workload is less than the
C
N
∑
∑
λ̄j .
µi ≤
sum of workload capacity for all IDCs, i.e.
(34)
B. Server Sleep (ON/OFF) control design
In the dynamic control problem of electricity cost for distributed Internet data centers, two types of tunable variables
are used to adjust the electricity energy consumption and
cost online, i.e. mj denoting the number of activated servers
in IDC j, and λij denoting the allocated workload in each
IDC j. The most effective way to save power is turning
OFF servers that are not used. Note that a change in mj is
decided based on the current workload assigned to the IDC
j. The inputs mj of the server sleep control is decided by
the received workload λj for IDC j. We use the following
equation to get the slow loop inputs mj :
∑

C
λij
 i=1
1 

mj = 
(35)
 µj + µj Dj  , ∀j = 1, · · · , N.




j=1
C. Dynamic workload control design
Model predictive control (MPC) is predication and optimization based control theory which targets to achieve high
performance for complex systems. MPC predicts the future
changes of output using the current output measurement, the
current dynamic models and the current control inputs. By
solving an optimization problem to minimize the errors between the references and the predicted outputs, MPC derives
the future control input series and implement the first one as
current control input. After repeating the calculation, MPC
will achieve the convergence stability and good tracking
performance.
Before designing the controller for the workload control
loop, we verify the workload control loop controllability
condition, which is described as follows.
Workload Loop Controllability Condition: This condition
comes from the control theory perspective. The system (19)
is completely controllable based on the condition that, given
an initial state X0 (t0 ) and a destination state Xf (tf ), there
exists a tf ≥ t0 and a control solution U (t) to make the
system run from the initial state to the destination state.
The sufficient and necessary
condition )for the fast loop
(
controllability is rank B AB · · · AM B = M + 1, which
is ensured since Prj > 0 and b1 > 0.
Using sleep (ON/OFF) control (35), the system (36) is
represented as:
X(k) = ΦX(k − 1) + GU (k − 1) + Ω,
(36)
where the new input matrix G is F + Γµ̄Ψ, and
µ̄ = diag[ µ11 , µ12 , · · · , µ1N ]. The disturbance Ω is
[ µ11D1 , µ21D2 , · · · , µN1DN ]T
The MPC controller computes the control input U (k) to
minimize the following cost function, which is given by:
J(k) =
+
β1
∑
∥Y (k + s|k) − r(k + s|k)∥2Q(s)
s=1
β∑
2 −1
∥U (k + s|k) − r(k + s − 1|k)∥2R(s) ,
(37)
s=1
where β1 and β2 denote the prediction horizon and the
control horizon, respectively. X(k + s|k) is the state prediction for next s sampling period based on the current
state X(k). Similarly, r(k + s|k) and U (k + s|k) gets their
corresponding meaning. Q(s) and R(s) are the weighting
matrices penalized on the tracking errors and control inputs.
Power Demand Smoothing Through Penalizing Inputs:
The power demand jumping is produced by dynamic
changing of allocated arriving workload and online turned
ON/OFF servers for each IDC, and it can be represented
by:
∆P = b1 ∆λj + ∆mj b0 .
where ∆λj =
C
∑
(38)
∆λij is the change of allocated arriving
i=1
workload for IDC j. ∆mj is the change of turned ON severs
for IDC j. Note that the power demand can be smoothed
by slowly adjusting the allocated arriving workload ∆λj
smoothly, i.e. penalizing inputs U (k). The cost function 37
can be interpreted as a compromise of electricity cost and
power demand jumping. The relative magnitudes of Q and
R provide a way to trade-off minimizing electricity cost for
smaller changes in volatile power demand.
To transform the MPC problem (37) to a standard leastsquares problem, we rewrite the system model (36) as
follows:
′
′
′
′
W ′ = diag
Ω̄ =





Y ′ (k) = 







X(k)

 .. 

 . 




 X(k) 
Y (k + β2 |k) 


,
X̄(k)
=
 X(k)  ,
Y (k + β2 + 1|k) 




 . 
..

 .. 
.
Y (k + β1 |k)
(
Ξ(k) =
G ···
β∑
2 −1
G
s=0

G
2G
..
.




 β
 ∑2
Θ=
 s=0 G


..

.

 β∑
1 −1
G
s=0
X(k)
β2
∑
G
···
s=0
O
G
..
.
β∑
2 −1
···
···
..
.
O
O
..
.
G ···
2G
s=0
..
G
s=0
s=0
..
.
β∑
1 −2
β∑
1 −1
)T
.
G ···
..
.
β1∑
−β2
s=0

G






,






···
···
Ω
W
)T
)
,
,



.

Now, we define a new vector Π ∈ ℜ(N +1)β1 ×1 which is
as follows:
Π(k) = Υ(k) − W ′ X̄(k) − W ′ ΞU (k − 1) − W ′ Ω(k), (40)
where Υ(k) is the reference trajectory with the prediction
horizon β1 defined by:
(
)T
Υ(k) = r(k + 1|k) r(k + 2|k) · · · r(k + β1 |k)
(41)
Then we transform the MPC problem (37) to a standard
least-squares problem as follows:
(β
)
β∑
1
2 −1
∑
min
∥W ′ Θ∆U (k) − Π(k)∥2Q(s) +
∥∆U (k)∥2R(s) ,
∆U (k)
s=1
s=1
(42)
subject to constraints:
where the matrices Y (k) ∈ ℜβ1 ×1 , X̄(k), Ω̄ ∈ ℜ(N +1)β1 ×1 ,
Ξ(k) ∈ ℜ(N +1)β1 ×N C , Θ ∈ ℜ(N +1)β1 ×N Cβ2 , ∆U (k) ∈
ℜN Cβ2 ×1 and W ′ ∈ ℜβ1 ×(N +1)β1 are given by:
Y (k + 1|k)
..
.
Ω Ω
W
U (k + β2 − 1|k) − U (k + β2 − 2|k)
′

W
U (k|k) − U (k − 1)
U (k + 1|k) − U (k|k)
..
.


∆U (k) = 

′
Y (k) = W X̄(k) + W ΞU (k − 1) + W Θ∆U (k) + W Ω̄,
(39)
(
(
¯
Ψ̄I∆U
(k) ≤ −Ψ̄U (k − 1) + φ̄,
(43)
−∆U (k) ≤ U (k − 1),
¯
H̄ I∆U
(k) = −H̄U (k − 1) + h̄,
(44)
(45)
where the matrices Ψ̄ ∈ ℜN β2 ×N C , I¯ ∈ ℜN C×N Cβ2 , φ̄ ∈
ℜN β2 ×1 , H̄ ∈ ℜCβ2 ×N C and h̄ ∈ ℜCβ2 ×1 are derived based
on (30) as follows:




Ψ O ··· O
I1 O · · · O
 O Ψ ··· O 



 ¯  I1 I1 · · · O 
Ψ̄ =  .
,I =  .
,

.
.
.
.
.
.
..
. . .. 
..
. . .. 
 ..
 ..




φ̄ = 




h̄ = 

O ··· Ψ
φ(k)
φ(k + 1|k)
..
.
I1 I1 · · · I1


H O ··· O

 O H ··· O 



 , H̄ =  ..
.. . .
. ,

 .
. .. 
.
φ(k + β2 − 1|k)
O O ··· H

h(k)

h(k + 1|k)

.
..

.
O

h(k + β2 − 1|k)
After transforming the MPC formulation to a standard
constrained least-square optimization problem described by
(42),(43), (44) and (45), we can use the standard leastsquares solvers to solve the electricity cost control problem.
It is worth noting that the solution is based on the electricity
cost model and the workload constraints. The electricity
price and the arriving workload are the parameters in the
model.
W ORKLOAD FOR
i
Li
1
30000
FIVE
2
15000
Table I
F RONT- END
3
15000
PORTAL SEVERS
4
20000
5
20000
D. Optimization solution of control reference
The objective of MPC controller for electricity cost is
to track the electricity cost reference value as accurately
as possible. For cost savings, the minimum electricity cost
and consumption references are preferred with the given electricity price and workload. Hence, an optimization
problem is needed to derive the minimum electricity cost
and consumption references. In this optimization problem,
the price penalty of power peak is set to 0 to minimize
the electricity cost. This problem has been studied in [5],
which minimizes the electricity cost through solving a linear
programming formulation given by:
min
mj ,λij
N
∑
Prj Pj (λij ),
(46)
j=1
subject to the constrains (1), (2), and (15).
This problem is solved using leverage Brenner’s fast
polynomial-time algorithm. For more details please see [5].
Note that the prediction of control reference is needed in
MPC controller. To this end, the optimization is conducted
based on the predicted workload as discussed in Sec.III-D.
Power Peak Shaving Through Setting Control References:
To enforce actual power demand to fall under desired budget,
we set the control power reference as Pr = Pro if Pro ≤ Prb
and as Pr = Prb if Pro > Prb , where Pro is the control power
reference derived by the optimization method (46), and Prb
is the maximum power budget constrained by power grid
market.
E. Stability analysis
The problem formulation in this paper is a standard
MPC with constraints. The stability for such a constrained
MPC closed-loop system does not rely on the poles of the
closed-loop transfer matrix of linear systems. Mayne et al.
proved the closed-loop stability for constrained MPC using
the contraction mapping theorem [21]. For more detailed
stability proofs see [21].
V. S IMULATION EXPERIMENTS
In this section, we present the evaluation of the proposed
control scheme using simulation experiments.
A. Experimental setup
To evaluate the proposed dynamic control of electricity
cost minimization, we set up a simulation environment
consisting of five front-end Web portal servers and three
IDCs (i.e. C = 5, N = 3) located at three different locations:
Michigan, Minnesota and Wisconsin. The Internet clients
send requests to the front-end Web portal servers which
then forward the requests to different IDCs for processing.
C ONFIGURATION
j
1
2
3
OF
µj
2
1.25
1.75
Table II
IDC S IN THREE DIFFERENT LOCATIONS
PjL
285
285
285
PjH
130
130
130
Mj
30000
40000
20000
Dj
0.001
0.001
0.001
Table III
E LECTRICITY PRICE IN THREE DIFFERENT LOCATIONS
Time
6H
7H
Michigan
43.2600
49.9000
Minnesota
30.2600
29.4700
Wisconsin
19.0600
77.9700
The workload is represented by the average number of
requests per second and is summarized in Table I. The delay
constraint at each IDC is assumed to be 1ms; the processing
speed for each server at three locations are 2.0 requests
per second, 1.25 requests per second and 1.75 requests
per second, respectively. Each server in the IDCs consumes
150 Watts when idle and 285 Watts when running at peak
processing speed [19]. The configuration of IDCs in three
different locations is shown in Table. II.
B. Performance evaluations of power demand smoothing
The power consumption is emulated from 6:00 to 6:10
on October 03, 2011. The electricity price in three different
locations is shown in Table III. To compare the performance
of dynamic control with only power demand smoothing and
the optimal allocation policy for electricity cost minimization presented in [5], we conduct numeric simulation and
show the power consumption results in Fig. 4(a)-4(c) for
three IDCs in three different locations: Michigan, Minnesota
and Wisconsin, respectively. The electricity consumptions
for optimal workload allocation policy are 2.1375MWH,
11.4MWH and 5.7MWH at 6H for three IDCs, respectively.
The power demand of optimal policy jumps to 5.7MWH,
11.4MWH and 1.628775MWH at 7H, respectively. We can
clearly see that the dynamic control effectively smooths the
power demand when the workload and electricity price vary
at each IDC. The number of turned ON servers for three
different IDCs are plotted in Fig. 5(a)-Fig. 5(c). The number
of turned ON severs are 7, 500, 40, 000 and 20, 000 at 6H for
three IDCs, respectively. However the number of turned ON
severs for optimal policy jumps to 20, 000, 40, 000 (no jump)
and 5, 715, respectively. The sudden increase in number of
turned ON servers for each IDC will drastically increase
the power demand. To avoid power demand jumping, the
dynamic control approach turns ON or turns OFF servers
gradually as shown in the figures. Although from the figures
12.5
4
control method
optimal method
3
2
0
2
4
6
Time (Min)
8
11.5
11
10.5
10
10
6
control method
optimal method
12
0
2
4
6
Time (Min)
(a)
Figure 4.
8
3
2
0
2
4
6
Time (Min)
8
10
(c)
Power consumption evaluations of power demand smoothing for three IDCs: (a) Michigan; (b) Minnesota; (c) Wisconsin.
4
4
4
x 10
4.0001
x 10
x 10
control method
optimal method
1
0
2
4
6
Time (Min)
8
Number
Number
1.5
2
control method
optimal method
4
Number
4
1
10
control method
optimal method
5
(b)
2
4
3.9999
10
0
2
4
6
Time (Min)
8
1
0.5
10
control method
optimal method
1.5
4
(a)
0
2
(b)
4
6
Time (Min)
8
10
(c)
Number of turned on servers evaluations of power demand smoothing for three IDCs: (a) Michigan; (b) Minnesota; (c) Wisconsin.
11.5
5
4
control method
optimal method
power budget
3
11
10.5
10
2
0
2
4
6
Time (Min)
8
0
2
10
(a)
Figure 6.
4
6
Time (Min)
2
10
0
2
4
6
Time (Min)
8
10
(c)
4
x 10
2
4
Number
control method
optimal method
1
2
4
6
Time (Min)
8
x 10
control method
optimal method
control method
optimal method
3.9
Number
3
4
x 10
0
4
Power consumption evaluations of power peak shaving for three IDCs: (a) Michigan; (b) Minnesota; (c) Wisconsin.
1.5
3.8
1.5
1
3.7
10
3.6
0
2
(a)
Figure 7.
8
control method
optimal method
power budget
5
(b)
4
2
6
control method
optimal method
power budget
Power (MWH)
Power (MWH)
Power (MWH)
6
Number
Figure 5.
Power (MWH)
5
Power (MWH)
Power (MWH)
6
4
6
Time (Min)
(b)
8
10
0.5
0
2
4
6
Time (Min)
8
10
(c)
Number of turned on servers evaluations of power peak shaving for three IDCs: (a) Michigan; (b) Minnesota; (c) Wisconsin.
it is apparently visible that by avoiding peak powers a
system may end up with more total electricity costs, the
modern smart grid prefers stable power consumptions and
may ultimately charge much more electricity costs for large
power demand jumping.
C. Performance evaluations of power peak shaving
As discussed in V-B, the optimal electricity power demand jumps to 5.7MWH, 11.4MWH and 1.628775MWH
at 7H at Michigan, Minnesota and Wisconsin, respectively.
Let us assume that the available power budgets at 7H
are 5.13MWH, 10.26MWH and 4.275MWH at the three
locations, respectively. Under such power budgets the power
demand of Michigan IDC is obviously attainable but the
Minnesota IDC and Wisconsin IDC violate their corresponding power budgets. To evaluate the performance of proposed
approach, the simulations with power budgets constraints are
conducted and the power consumption for three different
IDCs are drawn in Fig. 6(a)-Fig. 6(c). The results show that
the proposed control scheme is able to successfully track
the power consumptions of Michigan and Minnesota to keep
below their corresponding budgets but power consumptions
derived by the optimal policy exceed the budgets. The power
consumption of Wisconsin converges to the value between
its power budget and the power consumption values derived
from the optimal policy. The number of turned ON servers
for three different IDCs are drawn in Fig. 7(a)-Fig. 7(c).
VI. C ONCLUSION
This paper studies the dynamic control of electricity cost
considering the power demand smoothing and the power
peak shaving for distributed Internet data centers. The electricity cost is minimized using the real-time electricity pricing policy in modern power grid. The formulated electricity
cost problem using differential equations is solved by the
standard constrained model predictive control approach, in
which the power demand is smoothed through penalizing the
control inputs, i.e. the change of workload, and the power
peak is shaved by setting power control reference below the
desired budget. The extensive simulations based on real-life
electricity prices verify that the proposed control scheme
is able to achieve three goals together: minimizing the
electricity cost, smoothing the power demand and shaving
the power peak.
R EFERENCES
[1] X. Fan, W. Weber, and L. Barroso. Power provisioning for a
warehouse-sized computer. In Proceedings of the 34th annual
international symposium on Computer architecture, New York,
NY, USA, 2007.
[9] Y. Yao, L. Huang, A. Sharma, L. Golubchik, and M. J. Neely.
Data centers power reduction: A two time scale approach
for delay tolerant workloads. Technical Reports, University of
Southern California, 2011.
[10] Y. Zhang, Y. Wang, and X. Wang. Capping the electricity cost
of cloud-scale data centers with impacts on power markets.
In Proceedings of the 20th International ACM Symposium on
High-Performance Parallel and Distributed Computing, San
Jose, California, 2011
[11] M.Roozbehani, M.Dahleh, S.Mitter. On the stability of wholesale electricity markets under real-time pricing. In Proceedings
of the 49th IEEE Conference on Decision and Control (CDC),Atlanta, Georgia USA, 2010
[12] J. Liu, F. Zhao, X. Liu, W. He. Challenges Towards Elastic
Power Management in Internet Data Centers, In Proceedings
of the 29th IEEE International Conference on Distributed
Computing Systems Workshops, Montreal, Quebec, Canada,
2009
[13] A. Qureshi, R. Weber, H. Balakrishnan, J. Guttag, and B.
Maggs. Cutting the Electric Bill for Internet-Scale Systems, In
Proceedings of ACM SIGCOMM, Barcelona, Spain, 2009.
[14] T. Horvath and K. Skadron. Multi-mode energy management
for multitier server clusters. In Proceedings of the 17th international conference on Parallel architectures and compilation
techniques, Toronto, Ontario, Canada, 2008.
[2] J.Hamilton. Cost of power in large-scale data centers. Avaibale
at http://perspectives.mvdirona.com.
[15] G. Latouche and V. Ramaswami. Introduction to matrix
analytic methods in stochastic modeling. ASA-SIAM, 1999.
[3] United states environmental protection agency. Report to
congress on server and data center energy efficiency, 2007.
[16] S.Pacheco-Sanchez, G. Casale, B. Scotney, S. McClean, G.
Parr, and S. Dawson. Markovian Workload Characterization
for QoS Prediction in the Cloud. In Proceedings of the
2011 IEEE International Conference on Cloud Computing
(CLOUD), Washington, USA, 2011.
[4] N. Buchbinder, N. Jain, and I. Menache. Online job-migration
for reducing the electricity bill in the cloud. In Proceedings of
the 10th international IFIP TC 6 conference on Networking Volume Part I, Berlin, Heidelberg, 2011.
[5] L. Rao, X. Liu, L. Xie, W. Liu. Minimizing Electricity
Cost: Optimization of Distributed Internet Data Centers in
a Multi-Electricity-Market Environment. In Proceedings of
IEEE Conference on Computer Communications 2010 (INFOCOM’2010), San Diego, USA, 2010.
[6] Z. Liu, M.Lin, A. Wierman, S.Low, and L. Andrew. Greening
geographical load balancing. In Proceedings of International
Conference on Measurement and Modeling of Computer Systems, San Jose, California, 2011.
[7] E. Kayaaslan, B. Cambazoglu, R. Blanco, F. Junqueira, and C.
Aykanat. Energy-price-driven query processing in multi-center
web search engines. In Proceedings of the 34th Annual International ACM SIGIR Conference on Research and Development
in Information Retrieval, Beijing, China, 2011.
[8] A. Sankaranarayanan, S. Sharangi, and A. Fedorova. Global
cost diversity aware dispatch algorithm for heterogeneous
data centers. In Proceeding of the second joint WOSP/SIPEW
international conference on Performance engineering, New
York, NY, USA, 2011.
[17] P.Skantze, M.Ilic and J.Chapman. Stochastic modeling of
electric power prices in a mult-market environment.In Proceedings of Power Engineering Society Winter Meeting, Singapore,
2000.
[18] J. Yao, X.Liu, Z. Gu, X. Wang, J. Li. Online adaptive utilization control for real-time embedded multiprocessor systems.
Journal of Systems Architecture - Embedded Systems Design
56(9): 463-473, 2010.
[19] J. Moore, J. Chase, P. Ranganathan, and R. Sharma. Making
scheduling ”Cool”: temperature-aware workload placement in
data centers. In Proceedings of USENIX Annual Technical
Conference, USENIX Association Berkeley, CA, USA, 2005.
[20] N. Moiseev and F. Chernousko. Asymptotic methods in the
theory of optimal control, IEEE Trans. Automat. Control26(5):
pp.993-1000, 1981.
[21] D. Mayne, J. Rawlings, C. Rao and P. Scokaert. Constrained
model predictive control: stability and optimality. Automatica
36 (6): pp.789-814, 2000.