Greening Geographical Load Balancing .

Greening Geographical Load Balancing

Zhenhua Liu, Minghong Lin, Lachlan L. H. Andrew Adam Wierman, Steven H. Low

Faculty of ICT

CMS, California Institute of Technology, Pasadena Swinburne University of Technology, Australia

{zliu2,mhlin,adamw,slow}@caltech.edu landrew@swin.edu.au

ABSTRACT

A different stream of research has focused on exploiting Energy expenditure has become a significant fraction of data

the geographical diversity of Internet-scale systems to re- center operating costs. Recently, “geographical load balanc-

duce the energy cost. Specifically, a system with clusters at ing” has been suggested to reduce energy cost by exploiting

tens or hundreds of locations around the world can dynam- the electricity price differences across regions. However, this

ically route requests/jobs to clusters based on proximity to reduction of cost can paradoxically increase total energy use.

the user, load, and local electricity price. Thus, dynamic ge- This paper explores whether the geographical diversity of

ographical load balancing can balance the revenue lost due to Internet-scale systems can additionally be used to provide

increased delay against the electricity costs at each location. environmental gains. Specifically, we explore whether geo-

In recent years, many papers have illustrated the poten- graphical load balancing can encourage use of “green” renew-

tial of geographical load balancing to provide significant cost able energy and reduce use of “brown” fossil fuel energy. We

savings for data centers, e.g., [24, 28, 31, 32, 34, 39] and the make two contributions. First, we derive two distributed al-

references therein. The goal of the current paper is differ- gorithms for achieving optimal geographical load balancing.

ent. Our goal is to explore the social impact of geographical Second, we show that if electricity is dynamically priced in

load balancing systems. In particular, geographical load bal- proportion to the instantaneous fraction of the total energy

ancing aims to reduce energy costs, but this can come at the that is brown, then geographical load balancing provides sig-

expense of increased total energy usage: by routing to a data nificant reductions in brown energy use. However, the ben-

center farther from the request source to use cheaper energy, efits depend strongly on the degree to which systems accept

the data center may need to complete the job faster, and dynamic energy pricing and the form of pricing used.

so use more service capacity, and thus energy, than if the request was served closer to the source.

In contrast to this negative consequence, geographical load

Categories and Subject Descriptors

balancing also provides a huge opportunity for environmental

C.2.4 [Computer-Communication Networks]: Distributed benefit as the penetration of green, renewable energy sources Systems

increases. Specifically, an enormous challenge facing the elec- tric grid is that of incorporating intermittent, unpredictable renewable sources such as wind and solar. Because genera-

General Terms

tion supplied to the grid must be balanced by demand (i) in- Algorithms, Performance

stantaneously and (ii) locally (due to transmission losses), renewable sources pose a significant challenge. A key tech- nique for handling the unpredictability of renewable sources

is demand-response, which entails the grid adjusting the de- Increasingly, web services are provided by massive, geo-

1. INTRODUCTION

mand by changing the electricity price [2]. However, demand graphically diverse “Internet-scale”distributed systems, some

response entails a local customer curtailing use. In contrast, having several data centers each with hundreds of thousands

the demand of Internet-scale systems is flexible geographi- of servers. Such data centers require many megawatts of

cally; thus traffic can be routed to different regions to “follow electricity and so companies like Google and Microsoft pay

the renewables”, providing demand-response without service tens of millions of dollars annually for electricity [31].

interruption. Since data centers represent a significant and The enormous, and growing energy demands of data cen-

growing fraction of total electricity consumption, and the IT ters have motivated research both in academia and industry

infrastructure is already in place, geographical load balanc- on reducing energy usage, for both economic and environ-

ing has the potential to provide an extremely inexpensive mental reasons. Engineering advances in cooling, virtualiza-

approach for enabling large scale, global demand-response. tion, multi-core servers, DC power, etc. have led to signifi-

The key to realizing the environmental benefits above is for cant improvements in the Power Usage Effectiveness (PUE)

data centers to move from the fixed price contracts that are of data centers; see [6, 37, 19, 21]. Such work focuses on re-

now typical toward some degree of dynamic pricing, with ducing the energy use of data centers and their components.

lower prices when green energy is available. The demand response markets currently in place provide a natural way

Permission to make digital or hard copies of all or part of this work for for this transition to occur, and there is already evidence of personal or classroom use is granted without fee provided that copies are

some data centers participating in such markets [2]. not made or distributed for profit or commercial advantage and that copies

The contribution of this paper is twofold. (1) We develop bear this notice and the full citation on the first page. To copy otherwise, to

distributed algorithms for geographical load balancing with republish, to post on servers or to redistribute to lists, requires prior specific

provable optimality guarantees. (2) We use the proposed al- permission and/or a fee.

gorithms to explore the feasibility and consequences of using SIGMETRICS’11, June 7–11, 2011, San Jose, California, USA.

Contribution (1): To derive distributed geographical Our model for data center costs focuses on the server costs load balancing algorithms we use a simple but general model,

of the data center. 1 We model costs by combining the energy described in detail in Section 2. In it, each data center mini-

cost and the delay cost (in terms of lost revenue). Note that, mizes its cost, which is a linear combination of an energy cost

to simplify the model, we do not include the switching costs and the lost revenue due to the delay of requests (which in-

associated with cycling servers in and out of power-saving cludes both network propagation delay and load-dependent

modes; however the approach of [24] provides a natural way queueing delay within a data center). The geographical load

to incorporate such costs if desired. balancing algorithm must then dynamically define both how

Energy cost. To capture the geographic diversity and requests should be routed to data centers and how to allo-

variation over time of energy costs, we let g i (t, m i ,λ i ) de- cate capacity in each data center (i.e., how many servers are

note the energy cost for data center i during timeslot t given kept in active/energy-saving states).

m i active servers and arrival rate λ i . For every fixed t, we as- In Section 3, we characterize the optimal geographical load

sume that g i (t, m i ,λ i ) is continuously differentiable in both balancing solutions and show that they have practically ap-

m i and λ i , strictly increasing in m i , non-decreasing in λ i , pealing properties, such as sparse routing tables. Then, in

and convex in m i . This formulation is quite general, and Section 4, we use the previous characterization to give two

captures, for example, the common charging plan of a fixed distributed algorithms which provably compute the optimal

price per kWh plus an additional “demand charge” for the routing and provisioning decisions, and which require differ-

peak of the average power used over a sliding 15 minute win- ent types of coordination of computation. Finally, we eval-

dow [27]. Additionally, it can capture a wide range of models uate the distributed algorithms in a trace-driven numeric

for server power consumption, e.g., energy costs as an affine simulation of a realistic, distributed, Internet-scale system

function of the load, see [14], or as a polynomial function of (Section 5). The results show that a cost saving of over 40%

the speed, see [40, 5].

during light-traffic periods is possible. Defining λ i (t) = P j∈J λ ij (t), the total energy cost of data Contribution (2): In Section 6 we evaluate the feasibility

center i during timeslot t, E i (t), is simply and benefits of using geographical load balancing to facilitate

the integration of renewable sources into the grid. We do

E i (t) = g i (t, m i (t), λ i (t)). (1) this using a trace-driven numeric simulation of a realistic,

distributed Internet-scale system in combination with models Delay cost. The delay cost captures the lost revenue for the availability of wind and solar energy over time.

incurred because of the delay experienced by the requests. When the data center incentive is aligned with the social

To model this, we define r(d) as the lost revenue associated objective or reducing brown energy by dynamically pricing

with a job experiencing delay d. We assume that r(d) is electricity proportionally to the fraction of the total energy

strictly increasing and convex in d. coming from brown sources, we show that “follow the renew-

To model the delay, we consider its two components: the ables” routing ensues (see Figure 5), causing significant social

network delay experienced while the request is outside of the benefit. In contrast, we also determine the wasted brown en-

data center and the queueing delay experienced while the ergy when prices are static, or are dynamic but do not align

request is at the data center.

data center and social objectives. To model the network delay, we let d ij (t) denote the network delay experienced by a request from source j to data center i during timeslot t. We make no requirements on the

2. MODEL AND NOTATION

structure of the d ij (t).

We now introduce the workload and data center models, To model the queueing delay, we let f i (m i ,λ i ) denote the followed by the geographical load balancing problem.

queueing delay at data center i given m i active servers and an arrival rate of λ i . We assume that f i is strictly decreasing in m i , strictly increasing in λ i , and strictly convex in both

m i and λ i . Further, for stability, we must have that λ i = 0 or We consider a discrete-time model whose timeslot matches

2.1 The workload model

λ i <m i µ i , where µ i is the service rate of a server at data cen- the timescale at which routing decisions and capacity provi-

ter i. Thus, we define f i (m i ,λ i ) = ∞ for λ i ≥m i µ i . Else- sioning decisions can be updated. There is a (possibly long)

where, we assume f i is finite, continuous and differentiable. interval of interest t ∈ {1, . . . , T }. There are |J| geographi-

Note that these assumptions are satisfied by most standard cally concentrated sources of requests, i.e., “cities”, and the

queueing formulae, e.g., the mean delay under M/GI/1 Pro- mean arrival rate from source j at time t is L j (t). Job inter-

cessor Sharing (PS) queue and the 95th percentile of delay arrival times are assumed to be much shorter than a timeslot,

under the M/M/1. Further, the convexity of f i in m i models so that provisioning can be based on the average arrival rate

the law of diminishing returns for parallelism. during a slot. In practice, T could be a month and a slot

Combining the above gives the following model for the length could be 1 hour. Our analytic results make no as-

total delay cost D i (t) at data center i during timeslot t: sumptions on L j (t); however to provide realistic estimates we use real-world traces to define L X j (t) in Sections 5 and 6. D

i (t) =

j∈J λ ij (t)r (f i (m i (t), λ i (t)) + d ij (t)) . (2)

2.2 The data center cost model

2.3 The geographical load balancing problem

We model an Internet-scale system as a collection of |N | Given the cost models above, the goal of geographical load geographically diverse data centers, where data center i is

balancing is to choose the routing policy λ ij (t) and the num- modeled as a collection of M i homogeneous servers. The

ber of active servers in each data center m i (t) at each time model focuses on two key control decisions of geographical

t in order minimize the total cost during [1, T ]. This is cap- load balancing: (i) determining λ ij (t), the amount of traffic

tured by the following optimization problem: routed from source j to data center i; and (ii) determining

m i (t) ∈ {0, . . . , M i }, the number of active servers at data

center i. The system seeks to choose λ ij (t) and m i (t) in order

i∈N (E i (t) + D i (t)) (3a) to minimize cost during [1, T ]. Depending on the system

min

m(t),λ(t)

t=1

design these decisions may be centralized or decentralized.

1 Minimizing server energy consumption also reduces cooling Algorithms for these decisions are the focus of Section 4.

and power distribution costs.

subject to (4b)–(4d) and the additional constraint s.t. X

λ i ≤m i µ i ∀i ∈ N. (6b) λ ij (t) ≥ 0,

i∈N λ ij (t) = L j (t),

∀j ∈ J

(3b)

We refer to this optimization as GLB-Q. 0≤m i (t) ≤ M i ,

Additional Notation. Throughout the paper we use |S| to denote the cardinality of a set S and bold symbols to de-

note vectors or tuples. In particular, λ j = (λ ij ) i∈N denotes To simplify (3), note that Internet data centers typically con-

the tuple of λ ij from source j, and λ −j = (λ ik ) i∈N,k∈J \{j} tain thousands of active servers. So, we can relax the integer

denotes the tuples of the remaining λ ik , which forms a ma- constraint in (3) and round the resulting solution with min-

trix. Similarly m = (m i ) i∈N and λ = (λ ij ) i∈N,j∈J . imal increase in cost. Also, because this model neglects the

We also need the following in discussing the algorithms. cost of turning servers on or off, the optimization decouples

Define F i (m P i ,λ i )=g i (m i ,λ i ) + βλ i f i (m i ,λ i ), and define into independent sub-problems for each timeslot t. For the

i∈N F i (m i ,λ i )+Σ ij λ ij d ij . Further, let ˆ m i (λ i ) analysis we consider only a single interval and omit the ex-

F (m, λ) =

be the unconstrained optimal m i at data center i given fixed plicit time dependence. 2 Thus (3) becomes

λ i , i.e., the unique solution to ∂F i (m i ,λ i )/∂m i = 0.

X X min X

g i (m i ,λ i )+

λ ij r(d ij +f i (m i ,λ i )) (4a)

2.4 Practical considerations

m,λ

i∈N i∈N j∈J

X Our model assumes there exist mechanisms for dynami-

s.t. i∈N λ ij =L j ,

∀j ∈ J

(4b)

cally (i) provisioning capacity of data centers, and (ii) adapt- ing the routing of requests from sources to data centers.

λ ij ≥ 0,

With respect to (i), many dynamic server provisioning 0≤m i ≤M i ,

techniques are being explored by both academics and industry, e.g., [4, 11, 16, 38]. With respect to (ii), there are also a

We refer to this formulation as GLB. Note that GLB is variety of protocol-level mechanisms employed for data cen- jointly convex in λ ij and m i and can be efficiently solved

ter selection today. They include, (a) dynamically generated centrally. However, a distributed solution algorithm is usu-

DNS responses, (b) HTTP redirection, and (c) using per- ally required, such as those derived in Section 4.

sistent HTTP proxies to tunnel requests. Each of these has In contrast to prior work studying geographical load bal-

been evaluated thoroughly, e.g., [12, 25, 30], and though DNS ancing, it is important to observe that this paper is the first,

has drawbacks it remains the preferred mechanism for many to our knowledge, to incorporate jointly optimizing the total

industry leaders such as Akamai, possibly due to the added energy cost and the end-to-end user delay with consideration

latency due to HTTP redirection and tunneling [29]. Within of both price diversity and network delay diversity.

the GLB model, we have implicitly assumed that there exists GLB provides a general framework for studying geographi-

a proxy/DNS server co-located with each source. cal load balancing. However, the model ignores many aspects

Our model also assumes that the network delays, d ij can of data center design, e.g., reliability and availability, which

be estimated, which has been studied extensively, including are central to data center service level agreements. Such is-

work on reducing the overhead of such measurements, e.g., sues are beyond the scope of this paper; however our designs

[35], and mapping and synthetic coordinate approaches, e.g., merge nicely with proposals such as [36] for these goals.

[22, 26]. We discuss the sensitivity of our algorithms to error The GLB model is too broad for some of our analytic re-

in these estimates in Section 5.

sults and thus we often use two restricted versions. Linear lost revenue. This model uses a lost revenue function r(d) = βd, for constant β. Though it is difficult

3. CHARACTERIZING THE OPTIMA

to choose a “universal” form for the lost revenue associated We now provide characterizations of the optimal solutions with delay, there is evidence that it is linear within the range

to GLB, which are important for proving convergence of the of interest for sites such as Google, Bing, and Shopzilla [13].

distributed algorithms of Section 4. They are also necessary GLB then simplifies to

because, a priori, one might worry that the optimal solu-

tion requires a very complex routing structure, which would

X X X X be impractical; or that the set of optimal solutions is very min

fragmented, which would slow convergence in practice. The

results here show that such worries are unwarranted. subject to (4b)–(4d). We call this optimization GLB-LIN.

i∈N i∈N

i∈N j∈J

Queueing-based delay. We occasionally specify the form

Uniqueness of optimal solution.

of f and g using queueing models. This provides increased To begin, note that GLB has at least one optimal solu- intuition about the distributed algorithms presented.

tion. This can be seen by applying Weierstrass theorem [7], If the workload is perfectly parallelizable, and arrivals are

since the objective function is continuous and the feasible set Poisson, then f i (m i ,λ i ) is the average delay of m i paral-

is compact subset of R n . Although the optimal solution is lel queues, each with arrival rate λ i /m i . Moreover, if each

generally not unique, there are natural aggregate quantities queue is an M/GI/1 Processor Sharing (PS) queue, then

unique over the set of optimal solutions, which is a convex

f i (m i ,λ i ) = 1/(µ i −λ i /m i ). We also assume g i (m i ,λ i )= set. These are the focus of this section. p i m i , which implies that the increase in energy cost per

A first result is that for the GLB-LIN formulation, under timeslot for being in an active state, rather than a low-power

weak conditions on f i and g i , we have that λ i is common state, is m i regardless of λ i .

across all optimal solutions. Thus, the input to the data Under these restrictions, the GLB formulation becomes:

center provisioning optimization is unique.

X X X 1 Theorem 1. Consider the GLB-LIN formulation. Sup- min

pose that for all i, F i (m i ,λ i ) is jointly convex in λ i and m i ,

i∈N j∈J i∈N

and continuously differentiable in λ . Further, suppose that

Time-dependence of L j and prices is re-introduced for, and m ˆ i (λ i ) is strictly convex. Then, for each i, λ i is common for central to, the numeric results in Sections 5 and 6.

all optimal solutions.

The proof is in the Appendix. Theorem 1 implies that λ j := (λ ij , i ∈ N ). This allows all data centers as a group the server arrival rates at each data center, i.e., λ i /m i , are

and each proxy j to iteratively solve for optimal m and λ j common among all optimal solutions.

in a distributed manner, and communicate their intermedi- Though the conditions on F i and ˆ m i are weak, they do

ate results to each other. Though distributed, Algorithm 1 not hold for GLB-Q. In that case, ˆ m i (λ i ) is linear, and thus

requires each proxy to solve an optimization problem. not strictly convex. Although the λ i are not common across

To highlight the separation between data centers and prox- all optimal solutions in this setting, the server arrival rates

ies, we reformulate GLB-Q as:

remain common across all optimal solutions. X βλ i Theorem 2. For each data center i, the server arrival X

+β λ ij d ij (7) rates, λ i

λ min j ∈Λ j m min i ∈M i

µ i /m −λ

, are common across all optimal solutions to

GLB-Q. M X i := [0, M i ] Λ j := {λ j |λ j ≥ 0, λ ij =L j } (8)

i∈N

Sparsity of routing.

It would be impractical if the optimal solutions to GLB Since the objective and constraints M i and Λ j are separable, required that traffic from each source was divided up among

this can be solved separately by data centers i and proxies j. (nearly) all of the data centers. In general, each λ ij could be

The iterations of the algorithm are indexed by τ , and are non-zero, yielding |N | × |J| flows of traffic from sources to

assumed to be fast relative to the timeslots t. Each iteration data centers, which would lead to significant scaling issues.

τ is divided into |J| + 1 phases. In phase 0, all data centers i Luckily, there is guaranteed to exist an optimal solution with

concurrently calculate m i (τ + 1) based on their own arrival extremely sparse routing. Specifically:

rates λ i (τ ), by minimizing (7) over their own variables m i : Theorem 3. There exists an optimal solution to GLB

βλ i (τ ) with at most (|N | + |J| − 1) of the λ ij strictly positive.

m min i ∈M i p i m i +

(9) µ i −λ i (τ )/m i

Though Theorem 3 does not guarantee that every optimal solution is sparse, the proof is constructive. Thus, it pro-

In phase j of iteration τ , proxy j minimizes (7) over its own vides an approach which allows one to transform an optimal

variable by setting λ j (τ +1) as the best response to m(τ +1) solution into a sparse optimal solution.

and the most recent values of λ −j := (λ k , k 6= j). This works The following result further highlights the sparsity of the

because proxy j depends on λ −j only through their aggregate routing: any source will route to at most one data center that

arrival rates at the data centers:

is not fully active, i.e., where there exists at least a server in

λ il (τ + 1) + λ il (τ ) (10) power-saving mode.

Theorem 4. Consider GLB-Q where power costs p i are drawn from an arbitrary continuous distribution. If any source

To compute λ i (τ, j), proxy j need not obtain individual j ∈ J has its traffic split between multiple data centers N ′ ⊆

λ il (τ ) or λ il (τ + 1) from other proxies l. Instead, every data N in an optimal solution, then, with probability 1, at most

center i measures its local arrival rate λ i (τ, j) + λ (τ ) in one data center i ∈ N ′

has m i <M i . every phase j of the iteration τ and sends this to proxy j at the beginning of phase j. Then proxy j obtains λ i (τ, j) by

subtracting its own λ ij 4. (τ ) from the value received from data ALGORITHMS

center i. When there are fewer data centers than proxies, We now focus on GLB-Q and present two distributed al-

this has less overhead than direct messaging. gorithms that solve it, and prove their convergence.

In summary, the algorithm is as follows (noting that the Since GLB-Q is convex, it can be efficiently solved cen-

minimization (9) has a closed form). Here, [x] a := min{x, a}. trally if all necessary information can be collected at a single

point, as may be possible if all the proxies and data cen- Algorithm 1. Starting from a feasible initial allocation ters were owned by the same system. However there is a

λ (0) and the associated m(λ(0)), let strong case for Internet-scale systems to outsource route se-

· i low are decentralized and allow each data center and proxy (τ ) (11) to optimize based on partial information.

# lection [39]. To meet this need, the algorithms presented be- M i 1 λ

m i (τ + 1) :=

µ i These algorithms seek to fill a notable hole in the grow-

pp i /β

X λ i (τ, j) + λ ij ing literature on algorithms for geographical load balancing.

j (τ + 1) := arg min λ j ∈Λ j

i∈N µ i − (λ i (τ, j) + λ ij )/m i Specifically, they have provable optimality guarantees for a (τ + 1) performance objective that includes both energy and delay,

(12) network propagation delay information. The most closely

i∈N where route decisions are made using both energy price and λ ij d ij .

Since GLB-Q generally has multiple optimal λ related work [32] investigates the total electricity cost for ∗ j , Algo- data centers in a multi-electricity-market environment. It

rithm 1 is not guaranteed to converge to one optimal so- contains the queueing delay inside the data center (assumed

lution, i.e., for each proxy j, the allocation λ ij (τ ) of job j to be an M/M/1 queue) but neglects the end-to-end user

to data centers i may oscillate among multiple optimal al- delay. Conversely, [39] uses a simple, efficient algorithm to

locations. However, both the optimal cost and the optimal coordinate the “replica-selection” decisions, but assumes the

per-server arrival rates to data centers will converge. capacity at each data center is fixed. Other related works,

Theorem 5. Let (m(τ ), λ(τ )) be a sequence generated by e.g., [32, 34, 28], either do not provide provable guarantees

Algorithm 1 when applied to GLB-Q. Then or ignore diverse network delays and/or prices. (i) Every limit point of (m(τ ), λ(τ )) is optimal.

Algorithm 1: Gauss-Seidel iteration

(ii) F (m(τ ), λ(τ )) converges to the optimal value. Algorithm 1 is motivated by the observation that GLB-Q

(iii) The per-server arrival rates (λ i (τ )/m i (τ ), i ∈ N ) to is separable in m i , and, less obviously, also separable in

data centers converge to their unique optimal values.

The proof of Theorem 5 follows from the fact that Algo- where λ i (τ, −j) := P l6=j λ il (τ ) is the arrival rate to data cen- rithm 1 is a modified Gauss-Seidel iteration. This is also the

ter i, excluding arrivals from proxy j. Even though ˆ Λ (τ ) reason for the requirement that the proxies update sequen-

involves λ i (τ, −j) for all i, proxy j can easily calculate these tially. The details of the proof are in Appendix B.

quantities from the measured arrival rates λ i (τ ) it is told by Algorithm 1 assumes that there is a common clock to syn-

data centers i, as done in Algorithm 1 (cf. (10) and the dis- chronize all actions. In practice, updates will likely be asyn-

cussion thereafter), and does not need to communicate with chronous, with data centers and proxies updating with dif-

ferent frequencies using possibly outdated information. The other proxies. Hence, given λ i (τ, −j) from data centers i, algorithm generalizes easily to this setting, though the con-

each proxy can project into ˆ Λ j (τ ) to compute the next it- vergence proof is more difficult.

erate λ j (τ + 1) without the need to coordinate with other The convergence rate of Algorithm 1 in a realistic scenario

proxies. 3 Moreover, if λ(0) ∈ Λ then λ(τ ) ∈ Λ for all itera- is illustrated numerically in Section 5.

tions τ . In summary, Algorithm 2 is as follows.

Algorithm 2: Distributed gradient projection Algorithm 2. Starting from a feasible initial allocation

λ (0) and the associated m(λ(0)), each proxy j computes, in Algorithm 2 reduces the computational load on the proxies.

each iteration τ :

In each iteration, instead of each proxy solving a constrained minimization (12) as in Algorithm 1, Algorithm 2 takes a sin-

z j (τ + 1) := [λ j (τ ) − γ j (∇F j (λ(τ )))] Λ ˆ j (τ ) (16) gle step in a descent direction. Also, while the proxies com-

pute their λ

1 j (τ +1) sequentially in |J| phases in Algorithm 1,

|J| − 1

λ j (τ ) + z j (τ + 1) (17) they perform their updates all at once in Algorithm 2.

λ j (τ + 1) :=

|J| To achieve this, rewrite GLB-Q as

|J|

where γ j > 0 is a stepsize and ∇F j (λ(τ )) is given by

(µ i −λ i (τ )/m i (λ i (τ ))) 2 where F (λ) is the result of minimization of (7) over m i ∈M i given λ i . As explained in the definition of Algorithm 1, this

Implicit in the description is the requirement that all data minimization is easy: if we denote the solution by (cf. (11)):

centers i compute m i (λ i (τ )) according to (14) in each iteration τ . Each data center i measures the local arrival rate

λ i (τ ), calculates m i (λ i (τ )), and broadcasts these values to m i (λ i ) :=

all proxies at the beginning of iteration τ + 1 for the proxies

pp i /β

to compute their λ j (τ + 1).

Algorithm 2 has the same convergence property as Algo- then

rithm 1, provided the stepsize is small enough.

Theorem 6. Let (m(τ ), λ(τ )) be a sequence generated by

Algorithm 2 when applied to GLB-Q. If, for all j, 0 < γ j < min i∈N β 2 µ 2 i M i 4 /(|J|(φ + βM i ) 3 ), then

We now sketch the two key ideas behind Algorithm 2. The first is the standard gradient projection idea: move in the

(i) Every limit point of (m(τ ), λ(τ )) is optimal. steepest descent direction

(ii) F (m(τ ), λ(τ )) converges to the optimal value.

(iii) The per-server arrival rates (λ i (τ )/m i (τ ), i ∈ N ) to −∇F j (λ) := −

∂F (λ)

data centers converge to their unique optimal values.

∂λ 1j

∂λ |N|j

and then project the new point into the feasible set Q Λ Theorem 6 is proved in Appendix B. The key novelty

of the proof is (i) handling the fact that the objective is not The standard gradient projection algorithm will converge if

Lipshitz and (ii) allowing distributed computation of the pro- ∇F (λ) is Lipschitz over our feasible set Q j Λ j . This condi-

jection. The bound on γ j in Theorem 6 is more conservative tion, however, does not hold for our F because of the term

than necessary for large systems. Hence, a larger stepsize βλ i /(µ i −λ i /m i ). The second idea is to construct a com- Q

can be choosen to accelerate convergence. The convergence pact and convex subset Λ of the feasible set

rate is illustrated in a realistic setting in Section 5. following properties: (i) if the algorithm starts in Λ, it stays in Λ; (ii) Λ contains all optimal allocations; (iii) ∇F (λ) is Lipschitz over Λ. The algorithm then projects into Λ in each

j Λ j with the

5. CASE STUDY

iteration instead of Q j Λ j . This guarantees convergence. The remainder of the paper evaluates the algorithms pre- Specifically, fix a feasible initial allocation λ(0) ∈ Q j Λ j

sented in the previous section under a realistic workload. and let φ := F (λ(0)) be the initial objective value. Define

This section considers the data center perspective (i.e., cost minimization) and Section 6 considers the social perspective

(i.e., brown energy usage).

5.1 Experimental setup

Even though the Λ defined in (15) indeed has the desired We aim to use realistic parameters in the experimental properties (see Appendix B), the projection into Λ requires

setup and provide conservative estimates of the cost sav- coordination of all proxies and is thus impractical. In order

ings resulting from optimal geographical load balancing. The for each proxy j to perform its update in a decentralized

setup models an Internet-scale system such as Google within manner, we define proxy j’s own constraint subset:

the United States.

3 The projection to the nearest point in ˆ Λ j (τ ) is defined by Λ j (τ ) := Λ j ∩

φM i µ i

j λ i (τ, −j) + λ ij ≤

, ∀i

φ + βM i

[λ] Λ ˆ j (τ ) := arg min y∈ ˆ Λ j (τ ) ky − λk 2 .

0.2 0.2 Cost function parameters.

0.15 0.15 To model the costs of the system, we use the GLB-Q formulation. We set µ i = 1 for all i, so that the servers at each

0.1 0.1 location are equivalent. We assume the energy consumption

of an active server in one timeslot is normalized to 1. We

arrival rate

0.05 0.05 set constant electricity prices using the industrial electricity price of each state in May 2010 [18]. Specifically, the price

0 12 (cents per kWh) is 10.41 in California; 3.73 in Washington; 24 36 48 0 12 24 36 48

5.87 in Oregon, 7.48 in Illinois; 5.86 in Georgia; 6.67 in Vir- (a) Original trace

time(hour)

(b) Adjusted trace

ginia; 6.44 in Texas; 8.60 in Florida; 6.03 in North Carolina; and 5.49 in South Carolina. In this section, we set β = 1; however Figure 3 illustrates the impact of varying β.

Figure 1: Hotmail trace used in numerical results.

Algorithm benchmarks.

x 10 4 15 x 10 4 To provide benchmarks for the performance of the algo-

rithms presented here, we consider three baselines, which are

approximations of common approaches used in Internet-scale

10 systems. They also allow implicit comparisons with prior

work such as [32]. The approaches use different amounts

of information to perform the cost minimization. Note that

each approach must use queueing delay (or capacity information); otherwise the routing may lead to instability.

0 0 100 Baseline 1 uses network delays but ignores energy price 200 300 400 500 0 1000 2000 3000 4000 5000 6000 7000

when minimizing its costs. This demonstrates the impact of price-aware routing. It also shows the importance of dynamic

iteration

phase

(a) Static setting

(b) Dynamic setting

capacity provisioning, since without using energy cost in the optimization, every data center will keep every server active.

Figure 2: Convergence of Algorithm 1 and 2. Baseline 2 uses energy prices but ignores network delay. This illustrates the impact of location aware routing on the

Workload description.

data center costs. Further, it allows us to understand the To build our workload, we start with a trace of traffic

performance improvement of Algorithms 1 and 2 compared from Hotmail, a large Internet service running on tens of

to those such as [32, 34] that neglect network delays in their thousands of servers. The trace represents the I/O activity

formulations.

from 8 servers over a 48-hour period, starting at midnight Baseline 3 uses neither network delay information nor en- (PDT) on August 4, 2008, averaged over 10 minute intervals.

ergy price information when performing its cost minimiza- The trace has strong diurnal behavior and has a fairly small

tion. Thus, the traffic is routed so as to balance the delays peak-to-mean ratio of 1.64. Results for this small peak-to-

within the data centers. Though naive, designs such as this mean ratio provide a lower bound on the cost savings under

are still used by systems today; see [3]. workloads with larger peak-to-mean ratios. As illustrated in Figure 1(a), the Hotmail trace contains significant nightly ac-

5.2 Performance evaluation

tivity due to maintenance processes; however the data center The evaluation of our algorithms and the cost savings is provisioned for the peak foreground traffic. This creates a

due to optimal geographic load balancing will be organized dilemma about whether to include the maintenance activity

around the following topics.

or not. We have performed experiments with both, but report only the results with the spike removed (as illustrated

Convergence.

in Figure 1(b)) because this leads to a more conservative We start by considering the convergence of each of the dis- estimate of the cost savings.

tributed algorithms. Figure 2(a) illustrates the convergence Building on this trace, we construct our workload by plac-

of each of the algorithms in a static setting for t = 11am, ing a source at the geographical center of each mainland US

where load and electricity prices are fixed and each phase in state, co-located with a proxy or DNS server (as described in

Algorithm 1 is considered as an iteration. It validates the Section 2.4). The trace is shifted according to the time-zone

convergence analysis for both algorithms. Note here Algo- of each state, and scaled by the size of the population in the

rithm 2 used a step size γ = 10; this is much larger than that state that has an Internet connection [1].

used in the convergence analysis, which is quite conservative, and there is no sign of causing lack of convergence.

Data center description.

To demonstrate the convergence in a dynamic setting, Fig- To model an Internet-scale system, we have 14 data cen-

ure 2(b) shows Algorithm 1’s response to the first day of the ters, one at the geographic center of each state known to have

Hotmail trace, with loads averaged over one-hour intervals Google data centers [17]: California, Washington, Oregon,

for brevity. One iteration (51 phases) is performed every 10 Illinois, Georgia, Virginia, Texas, Florida, North Carolina,

minutes. This figure shows that even the slower algorithm, and South Carolina.

Algorithm 1, converges fast enough to provide near-optimal We merge the data centers in each state and set M i propor-

cost. Hence, the remaining plots show only the optimal so- tional to the number of data centers in that state, while keep-

lution.

ing Σ i∈N M i µ i twice the total peak workload, max t Σ j∈J L j (t). The network delays, d ij , between sources and data centers

Energy versus delay tradeoff.

are taken to be proportional to the distances between the The optimization objective we have chosen to model the centers of the two states and comparable to queueing delays.

data center costs imposes a particular tradeoff between the This lower bound on the network delay ignores delay due to

delay and the energy costs, β. It is important to understand congestion or indirect routes.

the impact of this factor. Figure 3 illustrates how the delay

9 x 10 4 to “follow the renewables”; thus providing increased usage of

t=7am 8 t=8am

green energy and decreased brown energy usage. However,

t=9am

such benefits are only possible if data centers forgo static

7 energy contracts for dynamic energy pricing (either through demand-response programs or real-time markets). The ex- energy cost 6 periments in this section show that if dynamic pricing is done

5 optimally, then geographical load balancing can provide significant social benefits.

x 10 5 6.1 Experimental setup

delay

To explore the social impact of geographical load balanc- Figure 3: Pareto frontier of the GLB-Q formulation

ing, we use the setup described in Section 5. However, we as a function of β for three different times (and thus

add models for the availability of renewable energy, the pric- arrival rates), PDT. Circles, x-marks, and triangles

ing of renewable energy, and the social objective. correspond to β = 2.5, 1, and 0.4, respectively.

and energy cost trade off under the optimal solution as β

The availability of renewable energy.

changes. Thus, the plot shows the Pareto frontier for the To model the availability of renewable energy we use stan- GLB-Q formulation. The figure highlights that there is a

dard models of wind and solar from [15, 20]. Though simple, smooth convex frontier with a mild ‘knee’.

these models capture the average trends for both wind and solar accurately. Since these models are smoother than ac- tual intermittent renewable sources, especially wind, they

Cost savings.

conservatively estimate the benefit due to following renew- To evaluate the cost savings of geographical load balanc-

ables.

ing, Figure 4 compares the optimal costs to those incurred We consider two settings (i) high wind penetration, where under the three baseline strategies described in the experi-

90% of renewable energy comes from wind and (ii) high so- mental setup. The overall cost, shown in Figures 4(a) and

lar penetration, where 90% of renewable energy comes from 4(b), is significantly lower under the optimal solution than

solar. The availability given by these models is shown in all of the baselines (nearly 40% during times of light traffic).

Figure 5(a). Setting (i) is motivated by studies such as [18]. Recall that Baseline 2 is the state of the art, studied in recent

Setting (ii) is motivated by the possibility of on-site or locally papers such as [32, 34].

contracted solar, which is increasingly common. To understand where the benefits are coming from, let us

Building on these availability models, for each location we consider separately the two components of cost: delay and

let α i (t) denote the fraction of the energy that is from renew- energy. Figures 4(c) and 4(d) show that the optimal algo-

i∈N α i (t) costs individually. In particular, Baseline 1 provides a lower

P rithm performs well with respect to both delay and energy P able sources at time t, and let ¯ α = (|N | T ) −1 T

be the “penetration” of renewable energy. We take ¯ α = 0.30, bound on the achievable delay costs, and the optimal algo-

t=1

which is on the progressive side of the renewable targets rithm nearly matches this lower bound. Similarly, Baseline 2

among US states [10].

provides a natural bar for comparing the achievable energy Finally, when measuring the brown/green energy usage of P

cost. At periods of light traffic the optimal algorithm pro-

a data center at time t, we use simply i∈N α i (t)m i (t) as the green energy usage and vides nearly the same energy cost as this baseline, and (per- P i∈N (1 − α i (t))m i (t) as the brown

haps surprisingly) during periods of heavy-traffic the optimal energy usage. This models the fact that the grid cannot algorithm provides significantly lower energy costs. The ex-

differentiate the source of the electricity provided. planation for this is that, when network delay is considered

by the optimal algorithm, if all the close data centers have

Demand response and dynamic pricing.

all servers active, a proxy might still route to them; how- Internet-scale systems have flexibility in energy usage that ever when network delay is not considered, a proxy is more

is not available to traditional energy consumers; thus they likely to route to a data center that is not yet running at full

are well positioned to take advantage of demand-response capacity, thereby adding to the energy cost.

and real-time markets to reduce both their energy costs and their brown energy consumption.

Sensitivity analysis.

To provide a simple model of demand-response, we use Given that the algorithms all rely on estimates of the L j

time-varying prices p i (t) in each time-slot that depend on and d ij it is important to perform a sensitivity analysis to

the availability of renewable resources α i (t) in each location. understand the impact of errors in these parameters on the

The way p i (t) is chosen as a function of α i (t) will be of achieved cost. We have performed such a sensitivity analysis

fundamental importance to the social impact of geographical but omit the details for brevity. The results show that even

load balancing. To highlight this, we consider a parameter- when the algorithms have very poor estimates of d ij and L j

ized “differentiated pricing” model that uses a price p b for there is little effect on cost. Baseline 2 can be thought of

brown energy and a price p g for green energy. Specifically, as applying the optimal algorithm to very poor estimates of

d ij (namely d ij = 0), and so the Figure 4(a) provides some p i (t) = p b (1 − α i (t)) + p g α i (t). illustration of the effect of estimation error.

Note that p g =p b corresponds to static pricing, and we show in the next section that p g = 0 corresponds to socially

optimal pricing. Our experiments vary p g ∈ [0, p b ]. We now shift focus from the cost savings of the data center

6. SOCIAL IMPACT

operator to the social impact of geographical load balancing.

The social objective.

We focus on the impact of geographical load balancing on To model the social impact of geographical load balanc- the usage of “brown” non-renewable energy by Internet-scale

ing we need to formulate a social objective. Like the GLB systems, and how this impact depends on pricing.

formulation, this must include a tradeoff between the energy Intuitively, geographical load balancing allows the traffic

usage and the delay users of the system experience, because usage and the delay users of the system experience, because

Baseline 3 5 Baseline 1

Baseline 2

Baseline 3 Baseline 2

15 Baseline 2 Optimal

Baseline 1 Optimal

total cost

3 40 delay cost 2

energy cost

1 % total cost improvement

12 time(hour) 24 36 48 0 12 24 36 time(hour) 48

time(hour)

(a) Total cost

(b) Total cost (% improve-

(d) Energy cost

ment vs. Baseline 3) Figure 4: Impact of information used on the cost incurred by geographical load balancing.

West Coast high solar

1.2 high wind

West Coast

East Coast

East Coast 1

0.4 active servers 2000

active servers 2000

renewable availability

0 12 24 36 48 0 12 24 36 48 0 12 24 36 48 0 12 24 36 48 time(hour)

time(hour)

(d) Static pricing Figure 5: Geographical load balancing “following the renewables” under optimal pricing. (a) Availability

(a) Renewable availability

(b) High solar penetration

of wind and solar. (b)–(d) Capacity provisioning of east coast and west coast data centers when there are renewables, high solar penetration, and high wind penetration, respectively.

purely minimizing brown energy use requires all m i = 0. The

p g key difference between the GLB formulation and the social =0

p g =0

p g =0.25 p =0.50p b 1.5 p g =0.25 p formulation is that the cost of energy is no longer relevant. b 10 g b p =0.50p Instead, the environmental impact is important, and thus

p g =0.75p

g b b p g =0.75p b

the brown energy usage should be minimized. This leads to

the following simple model for the social objective:

X X E i (t)

% social cost improvement

m(t),λ(t) min (1 − α i (t))

weight for brown energy usage

(b) High wind penetration where D i (t) is the delay cost defined in (2), E i (t) is the energy

(a) High solar penetration

cost defined in (1), and ˜ β is the relative valuation of delay versus energy. Further, we have imposed that the energy

Figure 6: Reduction in social cost from dynamic cost follows from the pricing of p i (t) cents/kWh in timeslot

pricing compared to static pricing as a function of t. Note that, though simple, our choice of D i (t) to model the

the weight for brown energy usage, 1/ ˜ β, under (a) disutility of delay to users is reasonable because lost revenue

high solar penetration and (b) high wind penetra- captures the lack of use as a function of increased delay.

tion.

An immediate observation about the above social objective is that to align the data center and social goals, one needs

back when solar energy is less available. Similarly, Figure to set p i (t) = (1 − α i (t))/ ˜ β, which corresponds to choosing

5(c) shows a shift, though much smaller, of service capacity toward the evenings, when wind is more available. Though

p b = 1/ ˜ β and p g = 0 in the differentiated pricing model not explicit in the figures, this “follow the renewables” rout- above. We refer to this as the “optimal” pricing model.

ing has the benefit of significantly reducing the brown energy

6.2 usage since energy use is more correlated with the availabil- The importance of dynamic pricing

ity of renewables. Thus, geographical load balancing pro- To begin our experiments, we illustrate that optimal pric-

vides the opportunity to aid the incorporation of renewables ing can lead geographical load balancing to “follow the re-

into the grid.

newables.” Figure 5 shows this in the case of high solar Figure 5 assumed the optimal dynamic pricing, but cur- penetration and high wind penetration for ˜ β = 0.1. By

rently data centers negotiate fixed price contracts. Although comparing Figures 5(b) and 5(c) to Figure 5(d), which uses

there are many reasons why grid operators will encourage static pricing, the change in capacity provisioning, and thus

data center operators to transfer to dynamic pricing over energy usage, is evident. For example, Figure 5(b) shows

the coming years, this is likely to be a slow process. Thus,

a clear shift of service capacity from the east coast to the it is important to consider the impact of partial adoption of west coast as solar energy becomes highly available and then

dynamic pricing in addition to full, optimal dynamic pricing.

Figure 6 focuses on this issue. To model the partial adop- [2] Server and data center energy efficiency, Final Report to

Congress, U.S. Environmental Protection Agency, 2007. 6(a) shows that the benefits provided by dynamic pricing are

tion of dynamic pricing, we can consider p g ∈ [0, p b ]. Figure

[3] V. K. Adhikari, S. Jain, and Z.-L. Zhang. YouTube traffic

dynamics and its interplay with a tier-1 ISP: An ISP when there is high solar penetration. Figure 6(b) suggests

moderate but significant, even at partial adoption (high p g ),

perspective. In ACM IMC, pages 431–443, 2010. that there would be much less benefit if renewable sources

[4] S. Albers. Energy-efficient algorithms. Comm. of the ACM,

were dominated by wind with only diurnal variation, because [5] L. L. H. Andrew, M. Lin, and A. Wierman. Optimality, the availability of solar energy is much more correlated with

fairness and robustness in speed scaling designs. In Proc. the traffic peaks. Specifically, the three hour gap in time

ACM Sigmetrics , 2010.

zones means that solar on the west coast can still help with [6] A. Beloglazov, R. Buyya, Y. C. Lee, and A. Zomaya. A the high traffic period of the east coast, but the peak average

Greening Geographical Load Balancing .