Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

Written on Mar 08 2010 Filed under: Teradici PC-over-IP, Display Protocols, Citrix XenDesktop, VMware, VMware View, Citrix 22,801 views, 36 comments

by Brian Madden

A few weeks ago, a company called Miercom released a report (download PDF) that was sponsored by Citrix to compare XenDesktop 4 remoting (ICA protocol) to VMware View 4 remoting via PCoIP. Then last Friday, VMware's Mike Coleman wrote blog post claiming the report was biased and flawed, and offered a point-by-point rebuttal. (Oh, did I mention the Miercom report showed ICA as the victor? ;)

I actually hadn't seen the Miercom report before reading Coleman's blog post, so I checked it out after the fact. I agree with some of Coleman's criticism's of the Miercom report, but on other points I think Miercom was right and Coleman's points are not accurate.

So based on that I present today's article: a complete point-by-point review of the Citrix/Miercom claims, the VMware/Coleman responses, and my own analysis of what's real and what's crap.

I'll follow the same outline both authors used and address each of the four areas:

• User experience
• Bandwidth consumption
• Flash performance
• Resource consumption & scalability

User experience

What Citrix/Miercom said:

"In a comparison of Virtual Desktop Infrastructure (VDI) implementations, Citrix XenDesktop 4 provided better overall performance when compared to VMware View 4."

What VMware/Coleman said:

I'm not sure I'm allowed to cut-and-paste Coleman's entire response here, so I'll try to summarize it the best I can:

• Miercom did not consider the image quality of office applications. Citrix uses "lossy client side image caching compared to VMware View 4’s lossless techniques, which favor the View 4 user experience."
• "Many customers insist on lossless image delivery – image compression artifacts are totally unacceptable to them."
• PC-over-IP provides progressive image refinement that builds the desktop image to a lossless image.
• "View is consistently chosen to deliver a high-quality desktop experience."

My analysis:

Coleman's statement about XenDesktop using "lossy client side image caching" is not technically accurate. Citrix does offer lossy compression (which can be enabled or disabled via policies for individual users, connections, network thresholds, etc.). But that lossy compression is different that the client-side bitmap caching, (which can be lossy or lossless). The bigger point, though, is that Coleman's main claim is that good user experience = lossless compression. And that's just simply not true. But just as important, the opposite is not true either.

What makes good user experience? Depends on the user, the app, the connection, and the use case. Just like I've said a thousand times, there is no "one size fits all." Unfortunately software vendors often just talk about the specific technique they use and then brainwash customers into thinking theirs is the only correct approach.

So VMware only offers lossless image compression  (well, they "build to lossless," which I'll address in a bit), which means they of course think a good user experience is defined by a perfect image (no matter how long it took to get there).

Citrix, on the other hand, offers lossy compression which can give a more fluid experience over slower links at the expense of image quality. So as expected, Citrix thinks that fast image loads equals a good experience, regardless of how the image looks.

VMware argues that theirs is better because it always ends up lossless, so it's the best of both worlds (Lossy for the speed, eventual lossless for the quality). Citrix argues theirs is better because you're not always popping back-and-forth and waiting for lossless.)

So who's right? It depends on your use case! Personally I don't think that lossy compression is a bad thing. Remember, probably every photo you've ever seen from a digital camera is encoded via lossy compression. (So it's not "lossy" that's bad per se.. it's more of a question of "how lossy?") And if I'm using Microsoft Office and scrolling around a document or working with PowerPoint, I probably wouldn't mind being able to scroll fast in exchange for some loss in image quality. And of course, lossy compression is just an option for Citrix, so you can always turn it off if you don't want it or configure it so that it only kicks in over slower connections.

Coleman goes on to talk about View's "build to lossless" and says that customers insist on lossless image quality. The problem is that "build to lossless" means that as you're scrolling around on the screen, you'll see a sort of "low res" version of everything, and then when you stop it pops to high res. In this case I have to agree with Miercom that while this is cool for 3D apps and stuff, in everyday practice with Office it's kind of annoying and weird. And I hear what Coleman is saying about customers insisting on lossless (although I'm still curious whether many would even notice), but if lossless is so important to customers, I can't see them waiting around every few seconds for PoIP to build to it. (And again, Citrix's lossy compression is just an option, so if lossless is important you don't need VMware to get it.)

The other thing is that Citrix has client-side bitmap caching, so jumping around Word or Outlook is pretty snappy. PC-over-IP doesn't have anything like that, so you have to retransmit all aspects of the screen even if the user is going back-and-forth between two things. Over a slower connection, ICA appears to be snappier than PC-over-IP.

Bandwidth Consumption

What Citrix/Miercom said:

"XenDesktop 4 used 64% less bandwidth than View 4 with PCoIP for typical tasks."

What VMware/Mike Coleman said:

• VMware's own tests showed PoIP using 40% less bandwidth on a 1Mbps network versus a 100Mbps network.
• VMware's own tests of ICA showed it used the same bandwidth on a 1Mbps network versus a 100Mbps network.
• This shows the weakness of Citrix's "fixed quality" system, "Citrix cannot take advantage of additional network resources when available."
• PoIP "dynamically adapts the image quality based on the available network resources" (So the unconstrained network meant PoIP would use the most bandwidth to deliver the best experience.)
• Better test would have been with multiple users on constrained network
• Miercom used 100Mbps for their WAN which skewed the results.
• PoIP will "fairly share the bandwidth across the active users for a given network segment."

My Analysis:

I am 100% in agreement with Mike Coleman and VMware here. In fact I'll go a step further: Citrix should be embarrassed that Miercom published this data, and if I were Citrix, I would never use Miercom again.

This whole bandwidth issue is very important and widely misunderstood. I've written about it quite a bit in the past, and it's become somewhat of a soap box issue for me. But if you're not familiar with my stance, the short version is:

• Looking how a protocol (any protocol, not just remote display) uses bandwidth in an unconstrained environment is IN NO WAY indicative of how that protocol will perform in a bandwidth-constrained environment.
• For remoting protocols, you CANNOT extrapolate a single user session out to guess how multiple users will behave.
• "User experience" and "bandwidth consumed" are totally different dimensions.

Digging into these points a bit more, if you have an unconstrained environment, looking at bandwidth consumption is irrelevant. A good protocol will use as much as it possibly can. More data across the wire means a better user experience and less load on the server and client with all the fancy compression and caching tricks. So if I'm a remote protocol and I see a wide open highway, I want to LET 'ER RIP! In fact I would say it's a bad thing if a remote protocol DOESN'T use more bandwidth when the network becomes unconstrained. It's like I want to say, "Ummm, hello? Dumbass? There's a wide open network here, so can you please take advantage of it?"

So when the network is unconstrained, that remote protocol better deliver an experience that's f'ing perfect, because if it doesn't, that means there's some problem with the protocol's ability to scale up.

That said, the unconstrained environment is not realistic for most real-world use cases. A better test would be with multiple users over a single connection. (Maybe with tests for LAN and WAN.) And that goes to my second point. Too often people try to figure out an "average" bandwidth consumption for a remote protocol. Sure, it is possible to work out what the average consumption was per user per second per connection, but it's really, really difficult and inappropriate to try to apply these averages to other environments.

For example, you might learn that ten users combined consumed an average of 700k:

• Does that mean that each user averaged 70k? Yes.
• Does that mean that a single user will have the same quality of experience over a 70k connection? No.
• Does that mean that 20 users will have the same quality of experience over a 1400k connection? No.
• Does that mean 5 users will have the same quality of experience over a 350k connection? No

I love Shawn Bass'es quote about this: "The average temperature in the desert is 70 degrees, sounds great, right? Sure, except that 'average' comes from it being 135 degrees in the day and 15 degrees at night."

Yup.

The thing is, Citrix knows this, and they're smarter than this. Too bad this report just makes Miercom (and by extension, Citrix) look like clueless idiots.

The big question of course is which remote protocol provides the best user experience for a certain number of users over a certain bandwidth? I'm going to go right back to my consulting here, and say that it depends on your users, your applications, your network, and your use case. To be honest I'm sure that ICA and PoIP are both fine. Sure, there are cases where ICA will kick PoIP's butt, and cases where PoIP will come out on top. (And wouldn't you know it, these just happen to be the perfect cases shown in each vendor's videos!)

In general I agree with Coleman about PoIP's dynamic tuning and his criticism of ICA's tuning parameters set by policy and admins instead of actual dynamic network conditions. But that's just at the policy-level. ICA itself will self-tune with regards to frame buffers and stuff.

But again it all goes back to use case. My personal unscientific opinion is that PoIP does a better job of tuning "up." That is side-by-side with a fast network, I'm going with PoIP over ICA. But in a bandwidth constrained slow environment, ICA does better. But those are both extremes... Which one works better for your particular case is something you'll have to figure out on your own.

One more thing to add here: Just as sort of a follow-on to the article I wrote on Friday, I think it's too bad that with Citrix, you have to pay for the WAN acceleration (Branch Repeater) as a separate add-on product, even though it's now just a virtual appliance you pop into your data center.

Flash Performance

What Citrix/Miercom said:

"Flash video was delivered with an average of 65% less CPU usage, 89% less bandwidth, and excellent Quality of Experience by XenDesktop 4 compared to View 4."

What VMware/Mike Coleman said:

• More talk about PCoIP's ability to throttle down, and how the Miercom test on 100Mbps is not real world.
• If bandwidth is an actual concern, you can put hard limits on PCoIP.
• Citrix uses Flash redirection, versus VMware which was Flash remoted. Not apples-to-apples.
• Packet loss in the test was 0.5 to 5%, which is unrealistically high.
• PCoIP has "no reliance whatsoever on client side driver compatibility," but this is not the case of the Citrix Flash environment, which since it's redirection needs a local flash client as well as hardware powerful enough to run flash.
• Testing Flash only was narrow. What about Silverlight, H.264, Quicktime, etc? PCoIP does all this, plus new ones that haven't been invented yet.

My Analysis

I won't go into the bandwidth numbers here (Miercom claiming 89% less bandwidth for ICA) as you're now well aware of my thoughts on bandwidth reporting.

Coleman made the right decision to point out that Citrix is doing "Flash redirection" while VMware is doing "Flash remoting."

(A quick tech note: Citrix recently added "Flash redirection" to XenDesktop. This means that if the user is browsing the web in a remote XenDesktop session and they encounter Flash on a page, if the user is connected to XenDesktop from a Windows client device with Flash installed locally, the remote XenDesktop host will send the flash source files down to the client where they will be executed locally, thus bypassing the "remote" element. Citrix has long done this with multimedia and the Windows Media Player, but this Flash redirection is new. VMware is doing Flash the way they do everything else--it runs on the host and is remoted down to the client.)

On some level, customers shouldn't care whether Flash remoting or Flash redirection is being used. But to Coleman's point, requiring Flash to be installed on the client does provide something else you have to manage at the client and means the good experience of the Flash redirection only works on clients who have the capability to run Flash locally.

I also agree that a third problem with redirection (in addition to client software and client device capabilities) is that your software vendor needs to write their product to support it. So yeah, now in 2010 we're all happy that Citrix has Flash redirection, but from 1998 to 2009 Flash was virtually unusable via Citrix even though everyone wanted it. And today if you browse to a site with Silverlight via Citrix, it's not going to be good like Flash since they don't do Silverlight redirection yet. Of course when Citrix releases Silverlight redirection that will change, but how long will we have to wait for that, and will we want to manage Silverlight on our clients in order to get it? Meanwhile you can browse to a Silverlight site via PCoIP and it will be just like Flash or anything else via PCoIP.

Miercom also called out host-side CPU utilization, claiming 4.7% for Citrix versus 71.5%. This makes sense since the Flash video has to render somewhere, and Flash is crazy-intensive. (You want to find out right now? Check out nick.com while watching CPU.. Even on your client right now. Crazy, eh?) So with Citrix Flash redirection, that CPU consumption doesn't disappear, it just moves down to the client. Of course that's great for your server user density, but to Coleman's point, you only get that if your clients have the right version of Flash installed and if they have the horsepower to take on that workload.

So which approach is better? Who "wins" Flash? Again, it goes down to use case. What kind of Flash support do you need and what kind of client devices do you want to use? What's more important to you?

Resource consumption & scalability

What Citrix/Miercom said:

"Overall, XenDesktop 4 uses system resources more efficiently and is capable of scaling more effectively."

What VMware/Coleman said:

• Real dollars are wasted with ICA since it doesn't take advantage of money spent on network resources.
• Depending on redirection solely does not provide real scalability. This doesn't handle common office apps and requires the vendor (and therefore customer) to keep on investing to redirect every new technology.
• Miercom failed to call out costs of managing client CODECs and Flash versions.
• Only View 4 supports a true zero client.

Coleman finished with, "In closing, we really do believe that VMware View is the only desktop virtualization product that has been designed from the ground up to deliver and manage virtualized desktop environments as a service. And, we do that with less cost and complexity than any other solution on the market."

My analysis

I covered Coleman's first three points in the previous section.

Coleman's statement that "Only View 4 supports a true zero client" is flat-out wrong. I'll be seeing Coleman this week during our Geek Week: VDI Challenge so I can ask him what exactly he means there.

The concept of a "zero client" is that it's a client device with "zero" management. For years, Teradici has marketed thin clients with the PC-over-IP hardware chips in them as "zero clients" because the clients didn't have a local OS, so they believed they were easier to manage than a traditional thin client.

The problem is that a Teradici hardware chip-based thin client (which is the "zero client" that Coleman is referring to) still has firmware which needs to be updated. And to me, that's not a zero client. In fact, ironically, in order to use a hardware-based PC-over-IP thin client that was built more than a few months ago, YOU HAVE TO FLASH THE FIRMWARE. How can anyone say that's a "zero client" with a straight face? How is that any different than a Linux- or CE or ThinOS client?

Is flashing the firmware difficult? Of course not. But it's establishing the precedent that these things are not "zero clients." (And hey, if they required a firmware flash for older PC-over-IP hardware clients to connect to the software-based PC-over-IP in View, what's to say that a future version of View won't require another firmware flash?)

I want to be very clear on this: I am NOT saying that firmware flashing is bad, and I think customers have no problem flashing firmware. It's just that you can't go around saying that you have a zero client when you've established precedence of requiring client firmware updates to get new feature support.

(For the record, my opinion is that there is only one true "zero client" on the market, and that's Pano Logic.) Those things can't even generate their own logon screens. They're just some bus-level remoting chips on the end of the network jack.)

Also, I don't really agree with Coleman's view that "VMware View is the only desktop virtualization product that has been designed from the ground up to deliver and manage virtualized desktop environments as a service." Is he suggesting that since Citrix grew up on Terminal Services, they're not designed from the ground up to deliver and managed virtualized desktop environments? And actually, wasn't ESX (on which View is based) designed from the ground up for server virtualization? Isn't that why ESX sucked for desktops up until the most recent v4 release a few months ago? So how can VMware make this claim?

Oooohh.. Wait. I see Coleman prefaced that statement with "we believe." (So yeah, VMware can believe it—doesn't means it's true though. :)

The final point I want to make about resource consumption & scalability is that VMware's PC-over-IP requires some serious processing power on your VM host. So you know those numbers that VMware was kicking around with View 4, like that with vSphere 4 and Nehalem you can get 16 VMs per core? Yeah, let's just say that's not with PC-over-IP enabled. So how many can you get? Again, again, again... that depends on your use case. (Because your apps and network connections will determine how hard your host has to work per VM, which is inversely proportional to user density.)

Final thoughts

So who was right? Citrix or VMware? Miercom or Mike Coleman? ICA or PC-over-IP? Who won?

Like I wrote in the title, both of these vendors are half-right. Miercom pointed out some good things that Citrix has over VMware, but they mislead people in other areas. And Mike Coleman appropriately called "BS" on some points but added his own BS for others.

At the end of the day, both Citrix ICA and VMware PC-over-IP are fine protocols. Both are better than RDP, and both have the capability to provide a good user experience. There are many use cases where one is clearly the winner over the other, but the only scenario that actually matters is yours.

In 2010, I don't see many customers choosing one desktop virtualization over another based solely on display protocol. But again, it all comes down to use case. So figure out what your use cases are and test them. Test them with real users and real apps and real clients on your real network and see which one is better for YOU!

Comments

Kata Tank wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 1:32 AM

protocol against protocol... Always same old war (it was before ICA versus RDP)... Like in the ICA versus RDP time, you forget everything around like printing, SSLizing...

I remember times where you (hte indsutry) tried to compare RDP versus ICa in term of performance but forget to mentionned that with RDP you were not able to print (or with lot's of driver problem) nor pass over Internet using SSL... Same with PCoIP versus ICA.

Could you SSL while PCoIP ? Could you print toany printer your client have ? Could you accelerate the WAN connexion (even if you have to pay for it) ? Could you get that every where, on any network, any client ? all this (and many more) are much important to me than to know who is doing lossless, semi-loosless or something-else-loosless technical tricks.

Note : OK, I'm a Citrix fan boy but my view is that we are going in the same "non sense quesitonning" like 10 years ago about b&B compareason, getting away from the entire picture...

Brian Madden wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 1:45 AM

Just to be clear, for this whole article, I was just commenting on the other blogs.. So really it's Citrix/Miercom that forgot printing, SSL, etc.

But these are all great questions that you bring up, and stuff that we'll be talking about in our geek week VDI challenge! :)

Short answers to your questions though, for PCoIP:

- No to SSL

- Yes to printing (via ThinPrint which is built-in)

- No to WAN accel since it's encrypted

Kata Tank wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 2:28 AM

thanks Brian (not too early in the US ?)...

For the printing, someone brought to me some restriction with printer (USB printer) ? have not tested it...

A.Nikolic wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 4:47 AM

Are the audio versions of the articles still available or has this experiment died a silent death..?

I liked listening to the articles when driving home from work..!

Louis wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 5:07 AM

Brian, don't you consider Sunray devices as zero clients?

A.Nikolic wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 7:03 AM

TYPO:

I think Miercom was write or I think Miercom was right?

Or did they write that they might be right.. :)

Tim Mangan wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 7:32 AM

Hopefully this bickering will lead to some positive results (even if it makes both vendors look a little stupid for a while).  Both implementations can learn from each other and improve.

Giving the admin a choice on how and/or where to impiment a function (this applies to many of the points of contention) rather than doing it only one way is better.  But best, will be when the product "reasonable and accurately" determines which way to do the function from the environment, and still gives the (choose one: admin, user) the ability to override that decision.

As to bandwidth use or not.  Tests for "constricted" situations should not be performed by slowing down the nic at one end.  Nic speed detection, while a necessary part of a "do the right thing automatically", a much more realisitic test involves constraint in the middle of the network somewhere and not on the physical ends.  Hopefully VDi week will settle on something like two test values, one for full boat "what can I get if it is available", and one for "how low can I go over a bad WAN and still be usable".

In my mind, a big part of the vaule of some independent testing is that it helps us sort through the vendor generated feature pushing and we all come to a common understanding about how to talk about and evaluate these things.

Donald (Don) Petry III wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 8:40 AM

Excellent analysis as always.  Regarding #3 - Flash Performance, it seems like a fairer test case would be to compare the performance and user experience for flash remoting.  I doubt many Citrix customers have fully converted their environment to leverage Flash redirection, so it would be nice to know how the two compare in the real world.

 Your comment in #4 - Resource Consumption & Scalability has me very curious to see a side-by-side comparison between the two products while running a common test case like those used by Project Virtual Reality Check (www.virtualrealitycheck.net).

Rob Hammersmith wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 8:41 AM

If you've got Citrix XenDesktop and want it to take advantage of your unconstrained network badnwidth, just PXE boot your devices over the network. PXE booting takes less server infrastructure than a VDI infrastrucutre does, and the user experience isn't being piped through a remote protocol. I do suggest a hosted solution like VDI or TS when the customer has a wide variety of end point devides, and other one off situations/requirements. But without some reason that PXE booting won't work for the customer, an unconstrained network is a great candidate for PXE booting.  

Michelle Pappas wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 8:53 AM

Brian-

EXPAND NETWORKS ANNOUNCE IMMEDIATE WAN OPTIMIZATION SUPPORT FOR VMWARE VIEW ™ 4 FOR BRANCH OFFICE DEPLOYMENTS

www.expand.com/.../Release.aspx

appdetective wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 11:12 AM

I continue to be amazed at how naive people are, and it just goes to show how sheep follow sheep when they have no experience in the real world.  This is a conversation about View delivering desktops, not a f'ing protocol. That's what VMware wants you to believe it is.

PCoIP is a POS for many many reasons in delivering a DESKTOP. Kata Tank lists some of them. I will tell you run multiple users over branch office links and see what happens. Run full motion video and see if the lips and video stay in synch, ICA it won't but BW vs. POSoIP ouch. Scroll outlook , word etc and see what it does to the pipe. Add to that SSL, Printing, no TS support, UDP only, ESX only, No support for Calista when it comes out, and the list goes on.

@Brian I don't get why you think adding a BR on a XD farm would help with other WAN traffic. Isn't that a good reason to charge for it? I see it as a cheap WAN Accelerator for all traffic that I don't need to part ocean's to install. So I think you are kind of missing the point, but yes I would like it for free.

PCoIP is garbage, gives the network a lot of sucky sucky but does not love it long time. Hosts don't scale as soon you run rich multimedia and TCO goes down the crapper. TS kills View for TCO. I've seen these blogs have the same stupid debate over RDP as Kata Tank also point out for years. VMware is completely f'd in the desktop with PcoIP, it will take many many years for it to fix all it's wows, Hey but perhaps they will give you Zimbra in the meantime and tell yo rewrite all your aps in Java and move to web bad declare the Desktop dead.  So for god sake, wake the F up and stop wasting time with this protocol debate, when we all already know View is sh1t and costs $$$$$$$, It's already decided in the protocol, VMware will have to do something very different to win if indeed they have any intention to which I douby given who they are buying...…

London1 wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 11:34 AM

Hi Brian,

I think the options around remote aceess and mobilty are one of the main drivers in VDI uptake and Its kind of shocking that there is no HTTP/SSL support for PCoIP forcing you to use clunky client VPN's for direct aceess

I blogged about this month and even the Citrix guys seemed suprised

virtualy-anything.blogspot.com/.../vmware-view-40-ssl-web-access.html

I think Citrix with Xendesktop HDX aligned with somthing like Secure gateway, Netscaler VPX and Access gateway are way ahead in terms in terms of simplistic and scalable secure remote access functionality.

Jim Moyle wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 1:07 PM

Thanks Michelle Pappas for the expand link, I'd like to see how that works.

Simon Bramfitt wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 7:41 PM

@Jim

Hey Jim, I can answer that one.

It doesn't.  MOre accurately while the Expand box can deliver full WAN Acceleration services for RDP connections, it can only provide QoS support for PCoIP.  It does offer SSL tunneling for PCoIP so it's better than nothing but a long way short of what we customers need from a PCoIP aware WAN Accelerator.

Regards

Simon

Simon Bramfitt wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 8:20 PM

Brian

I'll grant that the Meircom test was less than perfect, but to suggest that Citrix should never use them again is to do disservice to the industry and its customers.

This was a ground breaking test; it was the first time that anyone has ever attempted to perform objective analysis of the actual performance of remote presentation display protocols. Until Citrix engaged Meircom this type of testing has been both arbitrary and subjective, open to biased testing procedures with results based purely on the testers personal perspective. Now we have both a standard test workload in the Login Consultants VSI 2.0 script and standard quantitative scoring mechanism in VidMOS.  

You of all people should be encouraging wider adoption of independently reproducible testing programs by all vendors.

BTW - Testing a single connection on a 100 Mb WAN link is a perfectly valid test in as much as it provides insight into protocol performance when subject to latency without bandwidth constraints.

Regards

Simon

VMguy wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 8:50 PM

Very good and impartial Brian !

Again, a flood of bad comments on VMware, including the super professional and clean language of AppDetective, but you manage to do a cold blooded analysis without emotion, and hits the Citrix report on its heart: The supposed huge bandwidth advantage.

Before people starts shooting at me ,  PCoIP is indeed bw hungry, and still lacks some functionality (such as SSL gateway), but it is not even near as being a crap, and imposes Citrix decent competition (since Citrix previously did not knew the word 'competitor') :-D

Brian Madden wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 8 2010 9:48 PM

@Simon Bramfitt, I agree that 100mb WAN connection is a valid use case deserving of testing and that user experience should be considered at that speed. What I don't agree with is the focus on bandwidth consumption that that level and the implication that the 100m use case implies that one protocol requires more bandwidth than another.

And how was the Miercom test objective?

VMguy wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Tue, Mar 9 2010 6:36 AM

"And how was the Miercom test objective?"

To show how ICA is best than PCoIP, nothing more than that.

Or does anyone think they would test any condition in which ICA performs worst ? Its Citrix sponsored !

Simon Bramfitt wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Tue, Mar 9 2010 12:02 PM

@Brian VMguy

Briefly.

My calling out the Meircom test as objective in in reference to the fact that (regardless of the test performed) the measurements were gathered using objective means i.e. using a standard measuring tool rather than subjectively i.e. requesting that an observer interpret the responsiveness of the system under test and judge it.

For comparison look to the recent Winter Olympics; where downhill skiing is scored purely objectively measured with a stop watch and ice dancing is judged subjectively based on the judges personal perspective of how well the skater performed..

Further work can be done to improve the test, but the objective measurement system is what is important here and what should be encouraged.

Regards

Simon

Adam wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Tue, Mar 9 2010 12:34 PM

It was also my understanding that enabling PCoIP broke thinprint. This may have changed recently, not totally sure.

Also, on the Client side flash redirection. Citrix will automatically fall back to server side if it can't do client side redirection. And if Citrix needs to fall back to client side it will look at bandwidth and throttle flash performance accordingly if need be.

In my experience client side redirection is really great when it can be used and has a ton of benefit. true, there are some end point caveats, but generally a good piece of tech. I would not be surprised at all if VMware followed this and released something very similar.

rebecca.vanderbilt wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Tue, Mar 9 2010 10:18 PM

Brian, Miercom is one of the best testers in the market. I'm so surprised you would have such a disparaging remarks about them. Their credibility is very high in the industry and companies like Cisco, Siemens, and other world companies pay full attention to them. You are the only guy  who thinks so highly of yourself to say Miercom sucks. Miercom is widely seen as a solid test company. Not like Tully or other pay-for-play companies. You included!

Mr. Incredible wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Tue, Mar 9 2010 10:54 PM

I work for one of the companies mentioned here. I generally take a pretty dim view of this kind of sponsored benchmark because the outcome will always favour the sponsor. As a result, the usefulness of the study is vastly reduced because the customer knows that the results will be skewed toward the sponsor. This was evident when the same company was sponsored by each of the vendors to produce a report on the previous versions of the products in question. The test results weren't questionable, but the choice of tests and how the results were interpreted were questionable in both cases. I fully expect to see a response to the Miercom test come in due course, and for Citrix to respond in a blog, just as VMware have done for this report.

Vendors: Please stop this kind of sponsored report - they serve no purpose. Any valid points are outweighed by the amount of spin on the other points, and loss of credibility when the customers sees that it was commissioned by vendor xyz.

Customers: Do due diligence, speak to both vendors, have a clear vision of what you want to achieve. Actual BUSINESS requirements - NOT technical requirements based on the datasheet of the vendor that buys you the best lunch - unless of course it's me that buys the lunch ;-). Do a proof of concept with the products in the real world - Pilot the solution with real users and objectively measure the costs & benefits against your defined business requirements as well as observed benefits. Remember it's YOUR job on the line if you get it wrong, not mine.

Brian Madden wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Tue, Mar 9 2010 11:10 PM

@rebecca.vanderbilt, Did you read the few paragraphs in the article after I called Miercom's integrity into question? Because I feel I explained pretty clearly what the problems were with their testing. So I'm curious on your thoughts... do you agree that their testing was bad, but feel that I was too harsh on Miercom, or do you think their testing was fine? (If that's the case, I'd love to understand the technical reasons why you think their tests were ok.)

Also, I've never done pay-for-play. Not once ever. So I'm not sure where that comment comes from.. Perhaps you're confusing me with someone else?

MichelEhlert wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Wed, Mar 10 2010 3:08 AM

Hi,

I know the article is about ICA versus PCoverIP.

But I do want to add there are other protocols out there. RDP7 e.g. Though I admit we do not position RDP7 for usage over WAN links. (yes I'm from Microsoft). But than what about the enhancement by Quest of the RDP protocol (EOP).

I mention this to make sure people are aware there are more choices.

rebecca.vanderbilt wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Wed, Mar 10 2010 5:05 AM

Did you even contacted Miercom? I guess not. And what have you done lately to test these products thoroughly? Should I say NONE?

"Citrix should be embarrassed that Miercom published this data, and if I were Citrix, I would never use Miercom again."

If Miercom is good enough for Cisco, why not Citrix or even VMware.

David Caddick wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Wed, Mar 10 2010 5:58 AM

@Simon Bramfitt

I'm with Simon on this - at least it's objective and measurable - this is something that should be applauded and encouraged. If another Vendor doesn't like it, then publish quantifiable, repeatable and measured tests around parameters that you do like and then Vendor A can respond as they see fit?

Brian? Perhaps you could do a poll on a number of testing parameters (or methodology) that the Community feels best meets their business and user cases?

Virtual Reality Check x 20/50/100 users x 2Mb/5Mb/10Mb = poor/OK/good/excellent

But even the result there is a cop out as it's subjective right?

VMguy wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Wed, Mar 10 2010 6:25 AM

@David,

Although I am sure PCoIP is bandwidth hungry, I am sure VMware could find some scenarios in which it could perform better than ICA.

These tests were objective, but carefully selecting scenarios favorable to ICA. Come one, its Citrix sponsored !!!! This alone makes the tests suspect.

The only purpose IMO is to give Citrix sales people some ammunition, and be used by marketing purposes.

jmsazboy wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Wed, Mar 10 2010 7:17 AM

"In 2010, I don't see many customers choosing one desktop virtualization over another based solely on display protocol"

Best quote was saved for last!  Great article and this site and both of you do the rest of us a great service by providing an objective view of this technology.

That said, I could care less about how good my user's Youtube videos are and more about how easy their solution is going to snap into my environment.  

I understand video performance is important for CAD, GIS, etc but your day to day users do not generate any revenue for your company by watching flash videos.  Both Vendors would serve themselves better to focus on fusion with incumbent environments and not focus so much on their flash performance.  

Great article guys!  

Thanks!

John

VMguy wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Wed, Mar 10 2010 8:31 AM

jmsazboy,

Very good points. I see people mentioning flash video all the time, but on the majority of the situations, people do not need it for business purposes.

Some e-learnings and other corporate stuff might require flash, but this is way different from hi-def video.

Video/Audio/Multimedia performance is relevant on some scenarios only, not for the average office user, task worker.

Kata Tank wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Thu, Mar 11 2010 9:41 AM

Keep in mind that Miercom do not judge... It only "certified" that the test was conduct "this way" and that the result are "valid". If you do not like the test procedure, you could define and run others...

I could say something... Miercom could certified this is correct... but this is probably not always true. Take it for what it is...

Dan Shappir wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Thu, Mar 11 2010 6:54 PM

As various protocols and enhancements are being mentioned, I would also like to mention Ericom Blaze. Ericom Blaze was specifically designed for WAN, and compresses and accelerates RDP to a significantly greater extent than general purpose WAN accelerators. Also, it is Connection Broker independent. This means you can use it either with or without a Connection Broker.

For those who don't know - I work at Ericom.

Marek wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Sat, Mar 13 2010 8:04 PM

So many ego's in this industry!!!

Its hillarious.

Chris France wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Mon, Mar 15 2010 2:35 PM

Brian,

I don't like any of the current VDI or terminal service-like products for high-end engineering workstations.  The GPU is not virutalized and the performance is dog-slow.  As a Design Firm that has a bunch of architects and not enough computing power, how about a 3rd option? What do you think of this approach?  Pitfalls?  We have successfully pushed our workstations into the cloud.

See this article:

www.aecbytes.com/.../BIM_Cloud.html

Gunnar Berger wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Fri, Mar 19 2010 8:43 AM

Brian,

Great post.  I find reading all the comments the most entertaining part.  :)  I agree with you View/XenDesktop it all depends on the customers needs.  I've deployed both solutions to multiple clients and both solutions have met the expectations of the client.  It is a use case scenario, which is the same thing I say to those that want to battle it out between Apple and PCs, I really don't give a crap what you use, so long as you are happy with the performance/cost.

-Gunnar

rosspcs wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Tue, Mar 23 2010 2:52 PM

I am a big fan of View but PCOIP is lacking in a few other areas such as smartcard support and no support for the security gateway, so right now its use its faily limited.

pchaddha wrote re: Citrix releases report on ICA versus PC-over-IP. VMware cries foul. Both are half-right. Here's my full analysis.

on Thu, Mar 25 2010 5:21 PM

Hmmm!  Is virtualization about protocols or price?  Protcols wars, simply to satisfy the edge use-cases, are misplaced efforts.  PCs and workstations are perfectly fine for these edge use cases.  It is time that industry focuses on VDI cost (not just OPEX but also CAPEX) rather than "HD delivery over a starved network."

(Note: You must be logged in to post a comment.)

If you log in and nothing happens, delete your cookies from BrianMadden.com and try again. Sorry about that, but we had to make a one-time change to the cookie path when we migrated web servers.