Simon Bramfitt

Data Center Power Management and XenApp Workloads - IV update

I spoke last week with Clemens Pfeiffer CTO of PowerAssure who are another up and coming player in the data center power management world.  Unlike the other companies I have looked at, PowerAssure is offering their power management solution through a SaaS model. From what I have seen so far it looks like their service has a significant overlap with that provided by Cassat's Active Response Standard Edition product, although for the moment it appears that PowerAssure are a couple of development cycles behind Cassatt. The SaaS element comes into play most significantly in the pricing strategy that PowerAssure have adopted, where they charge their customers a percentage of any savings achieved. While I can't begin to assess how this will work out in the long term, it certainly looks as if the SaaS model may be attractive to some people.

I will be having a followup meeting with Clemens this week to take a closer look at the company and how their technology works and I've updated my previous post to account provide a better description of PowerAssure and will do so again after my visit.

Data Center Power Management and XenApp Workloads - IV

With an annual cost to business of $6 billion, servers and data centers represent 1.5% of the total US electricity consumption.  That’s a significant chunk of anyone’s budget; so when you consider that the easiest way to reduce the size of the data center electricity bill is simply to switch stuff off when you aren’t using it, it is both surprising and disappointing to see that there are so few products specifically designed to assist in this task. To date I’ve only been able to identify the following six products that play in this space (I’m only including products that are actually capable of powering servers off and on, products such as HP’s Insight Manager, IBM’s PowerExecutive and Intel’s Node Manager do not qualify as they only provide for real-time monitoring, power capping and CPU throttling capabilities). 

For what it's worth, I'll provide a rating of each product’s ability to meet the needs to control data center power consumption in relation to XenApp workloads. Most of the products listed below are not specifically targeted at XenApp workloads and may rate significantly higher as general purpose platform agnostic solutions. 

Vendor Alpiron

Product Alpiron Suite

Summary Alpiron appear to be unique in offering a power management product specifically targeted at Application Virtualization hosting platforms. Alpiron Suite provides policy based management with support for scheduling commands as well as the ability to create SLAs controlling the number of servers that are active at any one time.

Rating B+. While well suited to small to medium sized environments, it lacks some of the management and availability features that would be required in enterprise environments, and in attempting to cater for multiple products it has adopted a server-centric model that does not provide for application-centric SLAs that are possible using products such as XenApp - this may present adoption challenges for established Citrix customers.

Vendor Citrix

Product PowerSmart

Summary PowerSmart adds basic server power management capabilities to Citrix’s core XenApp platform. By enabling administrators to power down servers after all users have logged off for the day it provides the opportunity of significant power savings without forcing users to follow a fixed schedule. Its primary weakness is that it is unable to response to unexpected changes in demand once it has powered servers down at the end of the normal working day.

Rating C. PowerSmart is an excellent example of how an application vendor can add value to existing products by thinking outside the box a little and in so doing enable customers to reduce their electricity bills without any capital outlay. It would have scored higher but for its status as a beta product lacking formal support. 

Vendor Cassatt

Product Active Response Standard Edition

Summary Active Response provides a policy-based approach to power management that is extremely extensible and flexible.  It can readily integrate almost any server platform (physical and virtual) and can manage legacy servers through external IP-addressable PDUs.  Policies can be defined based on time, internal and external events (including power utility ‘demand response’ programs) as well as application load and server resource availability. While not fully realized, it is the furthest down the path towards enabling an enterprise to reduce its data center power consumption without putting XenApp SLAs at risk. 

Rating B-. Active Response is a true enterprise class product that can scale from basic power management capabilities all the way through to a full dynamic data center. Lack of native support for XenApp load level monitoring resulted in a lower score than it might otherwise have achieved. 

Vendor PowerAssure

Product N/A

Summary PowerAssure is a ‘Green IT’ startup that is bringing a SaaS based perspective to data center power management. PowerAssure provide a portfolio of services ranging from performing infrastructure assessments, through to monitoring server workloads and making recommendations for change. PowerAssure’s top tier service builds from monitoring to providing active management of servers and server workloads, shifting workloads between servers and powering down excess capacity to reduce overall power consumption. The SaaS element comes into play most significantly in the pricing strategy the PowerAssure have adopted where they charge their customers a percentage of any savings achieved. 

Rating N/A. I’ve not had the opportunity yet to explore the technology behind PowerAssure power management service or assess it with respect to XenApp workloads, so I’ll hold off on providing a rating until I have the opportunity to look more closely.

Vendor Virtual Iron

Product LivePower

Summary LivePower was introduced as an ‘experimental’ feature in VI-Center 4.4 and looks very much as though it is Virtual Iron’s response to VMware’s Distributed Power Manger (see below). It’s only notable claim over DPM is that it can inter-operate with Intel’s Node Manager for a deeper insight into actual CPU power consumption.  This may enable it to make more informed decisions about when and where to migrate workloads between servers to maximize the work done per kW.

Rating C. While it is heartening to see virtualization system vendors look beyond the immediate power saving opportunities that virtualization offers, LivePower is limited to managing hosts running Virtual Iron’s VI-Center. Owners of workloads running on other virtual infrastructure technologies or (heaven forbid) physical servers need not apply.  

Vendor VMware

Product Distributed Power Management

Summary Another ‘experimental’ feature ‘not intended for production use’, DPM is an extension to VMWare ESX(i) 3.5 Distributed Resource Scheduler. DPM enhances DRM to provide the ability to make server power management recommendations, or when in automatic mode to go ahead and implement recommendations so that guest workloads are consolidated onto the smallest number of servers needed to support the total workload at any given moment. Server’s are shutdown under operating system (read Hypervisor) control and restarted using Wake on LAN.

Rating C. See above. 

And that's it for now, if you are aware of any other apps that fit this description please let me know. 

Last updated  - Jan 20th 2009

Shortcuts on the road to a greener data center?

I previously drew attention to a white paper by The Green Grid that made the assertion that the risk to server reliability presented by powering servers off when not in use is less than most people perceive:

"...conventional wisdom is that servers must be kept running 24x7x52 because restarting them poses a potential downtime risk. However, research data suggests that this perception is false. ...

In a series of 3 laboratory tests over a 5 month period, a total of 123 servers were restarted several times daily by disconnecting and reconnecting the power utilizing an automated power strip outlet. Out of 18,826 restarts, not a single component failure occurred."

I decided to take a closer look at the source data behind this statement and was not comforted by what I found. On closer inspection it looks like the author of the white paper took a questionable shortcut in his reporting. The Green Grid's only cited source of information on the topic appears to be a blog post made by Vinay Pai who at the time was VP of Product Engineering for Cassatt and so hardly disinterested in the adoption of power management software. Hardly the level of thoroughness expected of 

" ...a global consortium dedicated to advancing energy efficiency in data centers and business computing ecosystems [ with a goal of " defining meaningful, user-centric models and metrics; developing standards, measurement methods, processes and new technologies to improve data center performance against the defined metrics... ".

To be clear, I don't doubt Vinay's word that this information is accurate, nor do I doubt The Green Grid's intention in sharing this information, but they would have been better served in commissioning their own tests using an independent auditor. With Dell, HP, IBM and Sun all having seats on their board it should not be too difficult to pull together enough equipment to provide a suitable test environment. 

Let's hope that this type of error is not repeated.

Data Center Power Management and XenApp Workloads - III

In a previous post on data center power management I showed that between the hours of 11 pm and 6 am it should be possible to meet 100% of demand with only 10-15% of the full production server compliment running.

Powering down any unneeded servers for 7 hours each weekday night and for 55 hours during the weekend would reduce server and data center power consumption by something like 25% (we can do the detailed math on that later).  So on first inspection this sounds like a significant win; just identify servers that aren't running any sessions, power them down for the night, and make sure they are back on-line again for 6 am when the day shift starts. 

The operational work-flow required to implement this basic power management solution looks something like this:

1. Disable new log-ons on managed server(s)*. 
2. Start monitoring load (session count) on managed server(s).
3. If the load on a server is zero, power it off.
4. Repeat until end of management window.
5. Restart all servers.
6. Re-enable log-ons to the managed server(s).

* Leave at least one server per silo unmanaged if you need to provide 24x7 application availability.  

And that's exactly how Citrix PowerSmart does it.  

Citrix launched PowerSmart a little over a year ago as a free utility targeted at addressing power management for idle servers. PowerSmart operates at the farm level with a single schedule to control when to enable active power management and when to restart the servers.  This works fine for simple environments, but starts to fall apart if you have more complex requirements. For example if you need to apply different schedules to different applications or server silos there's no way to do it except by placing the servers in different farms.  Then as I mentioned in a previous post while users may not need any servers at night, this doesn't mean that the servers are not being used. If you work for an organization that has an open change window every night (but doesn't always need to use it) PowerSmart's lack of flexibility could present significant challenges. More than anything else though, PowerSmart does not have any means of starting servers in response to an increase in application demand (or reduction in available capacity). So if the marketing department decides it needs to work over a weekend to support a new product launch, they will be out of luck until the operations team manually restart the servers. It's in this area of real-time response to SLA driven demand where PowerSmart really starts to show its limitations; and it's at this point I have to concede that without support for multiple server groups, manual overrides or the ability to power on servers in response to changes in demand or capacity, PowerSmart isn't able to meet my needs.

I don't really mean to knock PowerSmart, almost every XenApp customer I've ever worked with could achieve a significant reduction in their servers' power consumption using PowerSmart as it stands, but for larger organizations with more demanding needs, something extra is needed.

Looking forward though, the picture from Citrix is starting to get better.  Last year Citrix delivered a tech preview of WorkFlow Studio which has the makings of a very useful addition to their product portfolio.

If the tech preview passed you by; WorkFlow Studio is a new product that provides a management framework for process automation services linking multiple Citrix and 3rd party products using PowerShell and Windows Workflow Foundation. I'll follow-up on WorkFlow Studio in some more depth after it is released.

From what I've seen of WorkFlow Studio so far it should be possible to reproduce PowerSmart's functionality without any difficulty (although the tech preview is missing any out of the box integration with server power control systems like HP's iLO2), so maybe PowerSmart's days are numbered and we will see it reappear as a template in WorkFlow Studio.  Either way, we should find out before too long; the WorkFlow Studio tech preview expired on 31/12/08 and I expect the finished product will ship very soon. 

The only downside to WorkFlow Studio is that, as with all other work-flow automation tools, you have to invest time and effort up-front in mapping, designing and debugging your work-flows before you see any return. As such, it may see some resistance from organizations who feel more at home with off-the-shelf solutions. However; as I previously indicated, a short payback time is a key factor in obtaining buy-in for my power management initiative and so as as yet unreleased framework product can't be my number one choice as the key enabling technology for a power management solution.

I could look to mature workflow automation tools such as BMC's BladeLogic Orchestration Manager, and IBM's Tivoli Intelligent Orchestrator to substitute for Citrix's offering, but as far as I am aware they all suffer from a similar deficit in that they cost an arm and a leg and don't natively understand the concept of a XenApp Published Application and can't measure or respond to changes in XenApp load level without significant upfront development time (or without using Citrix's WorkFlow Studio to do the hard work, which doesn't really solve my problem). 

The one alternative left is to consider the possibility of using a dedicated server power management software application. For clarity, by server power management I don't mean server power monitoring, enclosure power capping or CPU clock frequency limiting, I mean having the capability to simply turn a server off when it is not needed and turn it back on when it is.  Remarkably it would appear that there are very few products on the market that provide this capability. After some extensive investigation I am only aware of five applications (six if you include Citrix PowerSmart) that can do this today, and only two of these has the ability to control physical servers in response to real time changes in server workload.  

I'll provide a brief review of the applications available to me today in part 4.

Switching off a server isn't as simple as it might sound

The simplest, most dramatic power saving measure to be made in any data center is to turn stuff off, and in the XenApp world were each application is going to be hosted on many servers powering off a server or two, or two hundred for that matter, should be pretty straight froward. Unfortunately turning off servers it isn't as simple as it might sound.

 

The first problems are cultural rather than technical; there is a deeply ingrained perception that servers are not designed to tolerate frequent power cycles. I can still recall the feeling of nervous anticipation that always attended the restart of the servers I was responsible for in the late 80's and early 90's. But that was years ago, manufacturing tolerances and techniques have changed  and today's servers are far more reliable and more accepting of (controlled) shutdowns than many might think.

The Green Grid makes the following statement on the matter:

"...conventional wisdom is that servers must be kept running 24x7x52 because restarting them poses a potential downtime risk. However, research data suggests that this perception is false. ...

In a series of 3 laboratory tests over a 5 month period, a total of 123 servers were restarted several times daily by disconnecting and reconnecting the power utilizing an automated power strip outlet. Out of 18,826 restarts, not a single component failure occurred."

Even with independent testing like this; I would still recommend that you go to your equipment manufacturer explain what you will be doing and seek out any information they have on server power cycling.  The MTBF numbers published on the server spec sheet may assume that the server is left on 24x7.  

 

Getting vendor confirmation that powering off servers is safe is one thing, but that isn't the end of the matter. You must also look at the data center power and cooling system. Speak to your facilities team; make sure that they understand what is being proposed and seek confirmation from them that the data center power and cooling systems can accept any anticipated fluctuation in demand. Current generation high-density blade systems can have very high heat density and short thermal rise times (they get hot quickly).  A report by AMD states that 

"A data center with an average heat load of 40W per square foot can cause a thermal rise of 25˚F in 10 minutes, while an average heat load of 300W per square foot can cause the same rise in less than a minute". 

Powering on large numbers of blades in response to a sudden increase in demand may present challenges for a cooling system designed on the assumption that load levels will fluctuate slowly. 

Power however should not be an issue, data center power distribution systems should be able to handle fluctuations in demand without problem, but it still needs to be discussed.  I know of one organization where the suggestion to shutdown servers over the weekend in a large engineering lab was met with some concern. The data center manager didn't want the servers down over the weekend because that was when he measured power consumption for capacity planning purposes.  

Most importantly; consult with your operation team.  While users may not need server access at night, that doesn't mean the servers are idle.  This is the time when the operations team are at their busiest, performing all the planned and unplanned maintenance; patching servers, performing software updates, pushing the latest anti-virus signature files, removing end of life servers and deploying new ones, all the unsung work that keeps the environment operating smoothly.  Clearly you can't shut down the server when all this activity is underway. Some means of accommodating both operational imperatives and the desire to reduce power consumption needs to be found.

Data Center Power Management and XenApp Workloads - II

... So with the economy in the middle of the worst recession in decades (and me with an $800,000 utility bill), it's time to assess if there's anything that can be done to cut my data center power bill without cutting services.

Where to start? 

I'm going to set a goal of reducing my monthly data center power consumption by 25% in two years (any less and the financial savings are unlikely to justify the cost and risks associated with the effort). I'm also going to set a target of reducing power consuption by 10% in year one, and to ensure that the payback time for any investment required is no more than 6 months. These are rather constraining goals, but not without reason.  No-one is going to accept new IT spending this year if it doesn't deliver a real cost saving in the short term (especially if it isn't already on the books). 

If I uncover new information that offers an opportunity to realistically increase this figure I'll adjust upwards accordingly. If, as I progress, it looks like I won't be able to meet the target, I'll re-assess and consider if the idea should be dropped or the scope reduced.  I don't have sufficient instrumentation or data logging capability to measure power consumption at the server or rack level in all data centers and I don't have accurate PUE factors either, so to provide a workable estimate of savings made I'm going to simplify things and instead of measuring power consumed directly, I'm going to cheat a little and assume that point measurements will substitute for 100% coverage that that individual data center's PUE numbers are all about the same. I'm reasonably confident about that first assumption, but the second may need to be revised if I get better data.

What else can I do to define some ground rules?  How does this look?

Meeting SLAs comes before reducing power consumption.  This stands to reason, IT is here to support business, and the goal here is to save power without cutting services. 

Look for the low hanging fruit. Again this should be obvious, don't attempt dynamic power management of mission critical systems if you have retired servers still running that somehow never got shutdown. This is cheaper, simpler, much lower in risk than anything else that can be done. The same goes for basic virtualization services.

Think strategically, but if that fails act tactically.  If, like me you, are responsible for a single technology silo you can get too close to your problem and find yourself implementing a solution that isn't going to be extensible to other technologies or integrate with your monitoring or workflow automation tools. But this needs to be balanced with the need for action and if the payback period is short enough you can throw a point solution away in a year or two if need be.

Think outside the box. I know it's a cliche, but it's a good one.  From the outset it's clear that new problems require new solutions; either new processes built on existing technology or new technologies designed specifically for the task in hand. 

Where does this put me?

The pertinent data points surrounding my environment are as follows:

• My environment consists of over 2,100 servers made up from about 1,300 production servers, 600 Business Continuity servers and serveral of hundred servers used for engineering, 'development', production support, QA performance and scalability testing and training.   
• The Business Continuity SLA requires that these servers are available 24x7
• Production systems are siloed in either an N + 15% or N + 10% configuration depending on the service they support.  Anticipating a 15% hardware failure rate may sound a little over cautious, but that is to provide a comprehensive contingency capability covering a lot more than just server hardware failure. (moe on this later).Supporting users in multiple time zones. 

As with most other business environments, server workloads peak in mid to late morning, dip during lunch, and then slowly tail-off into the evening with employees connecting from home for a couple of extra hours work, a relatively small number of connections, about 10% of peak load, are sustained through the night.  

• Planned down-time is infrequent at best.
• Unplanned down-time is not acceptable (ever). 

Now that is clear, what can I do? The obvious low hanging fruit have already been picked. We have already buried all the dead servers. The  non-production systems have already been moved to a virtual infrastructure, the production systems are heavily siloed with tens of servers dedicated to a single application so virtualisation is not an option for them.

The load distribution graph looks like an inviting place to start.  As well as providing a visual indicator of server workload, it also provides a clear indication of the amount of excess capacity available at any moment in time. Between about 11 pm and 6 am it should be possible to meet 100% of demand with only about 10-15% of the the full production server compliment.  Turning off 90% of any server farm for 6-7 hrs each night would offer a significant reduction in the utility bill if it could be done safely, and there's an even bigger saving on offer if that time can be extended to cover the entire weekend. 

Can we power down this excess capacity overnight and reap some benefit?

The second inviting opportunity is the business continuity environment, do we really need a hot standby environment running 24 x 7, is there any way to provide the holy grail of capacity on demand?

Let's see how that works in part 3.

Data Center Power Management and XenApp Workloads - I

Prior to 2 years ago the general perception held by many of my peers was that data center space was like air, it would never run out, and since we never saw the bill essentially free.  Since then we have had to go through a painful adjustment in perception; data center space is actually a valuable resource and one that is in very limited supply.  According to a report published by the Uptime Institute in March 2008, 41% of participants report that they will exceed current data center power capacity within 12-24 months. With the lead time to build and equip a new data center being about 18 months many of us are either already in deep trouble or about to be.

 

Even those businesses that have room for expansion within their data centers are facing a problem. Data center power consumption is becoming an increasing drag on business at a time when it can be least afforded.  An August 2007 report by the US Environmental Protection Agency found that in 2006 US servers and data centers accounted for 61 billion kilowatt-hours or about 1.5% of the country’s total electricity consumption, at a cost of about $4.5 billion.  This was more than double the amount of energy consumed in 2000.  The report goes on to indicate that by 2009 energy consumption by servers and data centers would increase to 85 billion kWh, and by 2011 energy consumption may reach as much as 107 billion kWh, representing a $7.4 billion annual electricity cost. If you look at this from the perspective of power generating capacity; the growth in data center power demand from 2009 to 2011 represents an increase equivalent to 10% of all new planned power generating capacity in the US over the same period.  As bad as this sounds, some people are already painting a blacker picture. Ken Brill of the Uptime Institute has suggested that the EPA's growth forecasts for 2011 are already being exceeded by many data centers. 

Putting a personal perspective on this, some back of the envelope calculations suggest that the 2,100 or so server blades I'm responsible for consume a little over 5 million kWh at an annual cost of almost $400,000.  I don't have the site specific data needed to accurately calculate the additional energy consumed by cooling system components (chillers, computer room air conditioning units (CRACs), direct expansion air handler (DX) units, pumps, and cooling towers) or the losses inherent in power delivery systems (UPS, switch gear, PDUs, batteries, etc.) so I may be a little bit off here, but if I use a Power Usage Effectiveness (PUE) factor of 2.0, I find I'm probably responsible for about 10 million kWh at an annual cost of almost $800,000. 

So with the economy in the middle of the worst recession in decades, it's time to assess if there's anything that can be done to cut the data center power bill without cutting services.  

More on this in part 2.