Simon Bramfitt

Desktop Virtualization Performance Testing

Citrix released their latest XenDesktop single server scalability test results today; showing a dual socket Intel Xeon x5570 (Nehalem) server with 72GB RAM running 130 Windows XP VMs per host. This is a significant improvement on Citrix’s previously published results, and certainly enough to bring it back into line with results VMware previously published. 

More important than the results though is the test methodology that Citrix has adopted this time around.

 For the past year I have been trying to encourage desktop virtualization vendors to adopt a standard testing methodology for virtual desktop workloads and I am very happy to be able to say that it looks like Citrix has come into line with my way of thinking. Rather than relying on an in-house developed test, Citrix have adopted Login VSI 2.0 from Log•in Consultants as the core of their testing process. Login VSI is a specifically designed benchmark for server-based computing and desktop virtualization environments. You may know it better as part of Project Virtual Reality Check the joint project by Log•in Consultants and PQR. Project VRC is the only fully independent benchmark of desktop virtualization workloads running on diverse physical and virtual platforms. By establishing themselves as trustworthy independent subject matter experts, the Project VRC team has benefited not just their customers but a global audience of IT organizations who can take advantage of the research performed under the Project VRC banner. Even more importantly the breadth and quality of information presented through Project VRC is acting to improve the quality and credibility of all server hosted desktop virtualization vendor issued performance benchmarks.

 Now that Citrix have seen the light, I hope that all server hosted desktop virtualization vendors will follow suit and adopt Login VSI as their testing benchmark. Anyone who doesn’t should not be surprised if their performance data is called into question.

Take XenApp 5.0 Advanced Administration Exam Free - Update!

A quick update on the status of the Citrix Education offer of a free seat on the XenApp 5.0 Advanced Admin Exam.

Citrix are reporting that all places for this offer have now been taken. 

New Tolly Group performance tests show XenServer ahead by a length

Exhibiting only slightly less tension than the Oxford Cambridge boat race; the Citrix XenServer Blues triumph over VMware in the latest round of the XenServer/ESX battle over which platform has the most convincing XenApp performance testing figures.

Tolly Group have release data from a series of tests (sponsored by Citrix) that showed that XenServer 5 out performed ESX 3.5 by supporting 41% more concurrent users; so no surprises there. Well maybe one surprise; this time out VMware didn't cry foul and demand request that Tolly strike the results from the record 

"Testing focused on system scalability and user quality-of-experience. This test report was approved for publication by VMware. The VMware End User License Agreement (EULA) requires such approval".  

So maybe VMware are more sporting than Simon Crosby would have us believe, that or some of his barbs have struck home.

Setting that tomfoolery aside, don't you think it's about time we customers had some independant benchmarks of virtual server workloads that we could trust.

Citrix blur the lines between app and workload delivery - I

Citrix have always had a to face the challenge presented by having to stay a couple of steps ahead of Microsoft's Terminal Services feature set and more recently by competitors looking to steal parts of Citrix's core customer base.  So after making it clear to VMware yesterday that they have a fight on their hands for the data center and cloud virtualization markets, Citrix went on to make it clear that they haven't forgotten their base either by announcing Feature Pack for XenApp 5 and blurring the lines between application delivery and workload delivery.

Sridhar Mullapudi and Vinny Sosa blogged about the change yesterday, and provided the following chart highlighting the changes 

	Advanced	Enterprise	Platinum
Citrix XenServer	X	X	X
Application streaming	X	X	X
EasyCall voice services	X	X	X
Workflow Studio orchestration	X	X	X
Load testing services		X	X
Profile management		X	X
Provisioning services			X
Single sign-on			X

It should be obvious to everyone by now that XenServer is not really a part of XenApp, so it's arguable that it should not be included here.  However, there is one key difference that Citrix have made that might not be immediately visible.  By making XenServer a part of XenApp, it means that XenServer is now supported by Citrix as part of any existing XenApp support contract, so even the smallest of XenApp users can now implement XenServer as a fully supported production platform for no extra cost. This is huge in itself, and is another direct attack on VMware who have been making a play for hosting XenApp on ESX for some time. OK, it should be noted that officially this support status only applies to using XenServer to host a XenApp workload, but if ever a rule was there to be broken this one is. 

Moving App Streaming down to XenApp Advanced is a strong way to shore up sales of Advanced to make it attractive for customers to stay with Citrix rather than jump ship to cheaper products or forego their Subscription Advantage agreement at a time when IT budgets are under pressure. The same could be said for the other major additions, Workflow Studio, Load Testing Services and Provisioning Server. Citrix came under substantial criticism last year for a perceived lack of new features in XenApp 5.0, this update should go along way to win back customer loyalty. 

It leaves me wondering though, if Citrix are in a position to announce this many changes before VMworld Europe.  What will we see at Synergy in May?

More later.

Not yet time for an open standard for measuring XenApp performance on virtual server platforms

I previously commented on the negative response to a posting on VMware's community site reporting some surprising results from a performance test that showed ESX outperforming XenServer when running a XenApp test workload. Jeff Buell posted a follow-up on his blog seeking out opinions on "What characteristics would make for a credible, reliable, objective, and repeatable benchmark". 

I tried to engage Jeff on this front, indicating that I am very willing to work to define such a benchmark, but after a couple of exchanges I'm getting the impression that Jeff isn't quite as as willing as I first thought he was.

Still some good has clearly come of the exchange. Jeff has made it clear that  "The work published on our blog was our first cut at [a XenApp] workload and was never advertised as being complete. "  and that VMware  "are also planning scale-out tests and more complete documentation".  If nothing else it this is a significant step back from the black and white statement  "The results show that ESX runs common desktop applications on XenApp ... with significantly better performance than XenServer."  that was made in the inital blog post.

Regardless of anything else, I am certainly looking forwards to seeing the results of any scale-out test that they publish and the response to them that the broader community will provide.  

Time for an open standard for measuring XenApp performance on virtual server platforms?

There's been a lot of commentary in the last few days regarding a blog post made by a member of VMware's performance team that made some interesting claims about ESX having a performance adantage over XenServer when running a XenApp. Now, VMware didn't publish the fine detail of how their results were gathered so I'm in no position to question the accuracy of the results, but it was clear that the balance of opinion was that the results didn't align with other peoples' experiences.

VMware curtailed any additional discussion of the topic by quickly closing comments on the initial blog post, which is understandable given the uniformly negative sentiment that was expressed, but in a separate post Jeff Buell offered an olive branch on his blog to "those interested in a rational discussion of the performance issues surrounding Terminal Services and XenApp workloads in virtualized environments" by seeking out opinions on "What characteristics would make for a credible, reliable, objective, and repeatable benchmark". 

I have a clear desire to see such a benchmark, but given the obvious competition between VMware and Citrix I can't accept the notion that it be developed on a VMWare owned web site whose terms of use grant VMware the right to delete any content that they don't agree with. So with that said I've asked Jeff if he'll agree to such a venture in a more transparent venue.