A few weeks ago, a company called Miercom released a report (download PDF) that was sponsored by Citrix to compare XenDesktop 4 remoting (ICA protocol) to VMware View 4 remoting via PCoIP. Then last Friday, VMware's Mike Coleman wrote blog post claiming the report was biased and flawed, and offered a point-by-point rebuttal. (Oh, did I mention the Miercom report showed ICA as the victor? ;)
I actually hadn't seen the Miercom report before reading Coleman's blog post, so I checked it out after the fact. I agree with some of Coleman's criticism's of the Miercom report, but on other points I think Miercom was right and Coleman's points are not accurate.
So based on that I present today's article: a complete point-by-point review of the Citrix/Miercom claims, the VMware/Coleman responses, and my own analysis of what's real and what's crap.
I'll follow the same outline both authors used and address each of the four areas:
- User experience
- Bandwidth consumption
- Flash performance
- Resource consumption & scalability
What Citrix/Miercom said:
"In a comparison of Virtual Desktop Infrastructure (VDI) implementations, Citrix XenDesktop 4 provided better overall performance when compared to VMware View 4."
What VMware/Coleman said:
I'm not sure I'm allowed to cut-and-paste Coleman's entire response here, so I'll try to summarize it the best I can:
- Miercom did not consider the image quality of office applications. Citrix uses "lossy client side image caching compared to VMware View 4’s lossless techniques, which favor the View 4 user experience."
- "Many customers insist on lossless image delivery – image compression artifacts are totally unacceptable to them."
- PC-over-IP provides progressive image refinement that builds the desktop image to a lossless image.
- "View is consistently chosen to deliver a high-quality desktop experience."
Coleman's statement about XenDesktop using "lossy client side image caching" is not technically accurate. Citrix does offer lossy compression (which can be enabled or disabled via policies for individual users, connections, network thresholds, etc.). But that lossy compression is different that the client-side bitmap caching, (which can be lossy or lossless). The bigger point, though, is that Coleman's main claim is that good user experience = lossless compression. And that's just simply not true. But just as important, the opposite is not true either.
What makes good user experience? Depends on the user, the app, the connection, and the use case. Just like I've said a thousand times, there is no "one size fits all." Unfortunately software vendors often just talk about the specific technique they use and then brainwash customers into thinking theirs is the only correct approach.
So VMware only offers lossless image compression (well, they "build to lossless," which I'll address in a bit), which means they of course think a good user experience is defined by a perfect image (no matter how long it took to get there).
Citrix, on the other hand, offers lossy compression which can give a more fluid experience over slower links at the expense of image quality. So as expected, Citrix thinks that fast image loads equals a good experience, regardless of how the image looks.
VMware argues that theirs is better because it always ends up lossless, so it's the best of both worlds (Lossy for the speed, eventual lossless for the quality). Citrix argues theirs is better because you're not always popping back-and-forth and waiting for lossless.)
So who's right? It depends on your use case! Personally I don't think that lossy compression is a bad thing. Remember, probably every photo you've ever seen from a digital camera is encoded via lossy compression. (So it's not "lossy" that's bad per se.. it's more of a question of "how lossy?") And if I'm using Microsoft Office and scrolling around a document or working with PowerPoint, I probably wouldn't mind being able to scroll fast in exchange for some loss in image quality. And of course, lossy compression is just an option for Citrix, so you can always turn it off if you don't want it or configure it so that it only kicks in over slower connections.
Coleman goes on to talk about View's "build to lossless" and says that customers insist on lossless image quality. The problem is that "build to lossless" means that as you're scrolling around on the screen, you'll see a sort of "low res" version of everything, and then when you stop it pops to high res. In this case I have to agree with Miercom that while this is cool for 3D apps and stuff, in everyday practice with Office it's kind of annoying and weird. And I hear what Coleman is saying about customers insisting on lossless (although I'm still curious whether many would even notice), but if lossless is so important to customers, I can't see them waiting around every few seconds for PoIP to build to it. (And again, Citrix's lossy compression is just an option, so if lossless is important you don't need VMware to get it.)
The other thing is that Citrix has client-side bitmap caching, so jumping around Word or Outlook is pretty snappy. PC-over-IP doesn't have anything like that, so you have to retransmit all aspects of the screen even if the user is going back-and-forth between two things. Over a slower connection, ICA appears to be snappier than PC-over-IP.
What Citrix/Miercom said:
"XenDesktop 4 used 64% less bandwidth than View 4 with PCoIP for typical tasks."
What VMware/Mike Coleman said:
- VMware's own tests showed PoIP using 40% less bandwidth on a 1Mbps network versus a 100Mbps network.
- VMware's own tests of ICA showed it used the same bandwidth on a 1Mbps network versus a 100Mbps network.
- This shows the weakness of Citrix's "fixed quality" system, "Citrix cannot take advantage of additional network resources when available."
- PoIP "dynamically adapts the image quality based on the available network resources" (So the unconstrained network meant PoIP would use the most bandwidth to deliver the best experience.)
- Better test would have been with multiple users on constrained network
- Miercom used 100Mbps for their WAN which skewed the results.
- PoIP will "fairly share the bandwidth across the active users for a given network segment."
I am 100% in agreement with Mike Coleman and VMware here. In fact I'll go a step further: Citrix should be embarrassed that Miercom published this data, and if I were Citrix, I would never use Miercom again.
This whole bandwidth issue is very important and widely misunderstood. I've written about it quite a bit in the past, and it's become somewhat of a soap box issue for me. But if you're not familiar with my stance, the short version is:
- Looking how a protocol (any protocol, not just remote display) uses bandwidth in an unconstrained environment is IN NO WAY indicative of how that protocol will perform in a bandwidth-constrained environment.
- For remoting protocols, you CANNOT extrapolate a single user session out to guess how multiple users will behave.
- "User experience" and "bandwidth consumed" are totally different dimensions.
Digging into these points a bit more, if you have an unconstrained environment, looking at bandwidth consumption is irrelevant. A good protocol will use as much as it possibly can. More data across the wire means a better user experience and less load on the server and client with all the fancy compression and caching tricks. So if I'm a remote protocol and I see a wide open highway, I want to LET 'ER RIP! In fact I would say it's a bad thing if a remote protocol DOESN'T use more bandwidth when the network becomes unconstrained. It's like I want to say, "Ummm, hello? Dumbass? There's a wide open network here, so can you please take advantage of it?"
So when the network is unconstrained, that remote protocol better deliver an experience that's f'ing perfect, because if it doesn't, that means there's some problem with the protocol's ability to scale up.
That said, the unconstrained environment is not realistic for most real-world use cases. A better test would be with multiple users over a single connection. (Maybe with tests for LAN and WAN.) And that goes to my second point. Too often people try to figure out an "average" bandwidth consumption for a remote protocol. Sure, it is possible to work out what the average consumption was per user per second per connection, but it's really, really difficult and inappropriate to try to apply these averages to other environments.
For example, you might learn that ten users combined consumed an average of 700k:
- Does that mean that each user averaged 70k? Yes.
- Does that mean that a single user will have the same quality of experience over a 70k connection? No.
- Does that mean that 20 users will have the same quality of experience over a 1400k connection? No.
- Does that mean 5 users will have the same quality of experience over a 350k connection? No
I love Shawn Bass'es quote about this: "The average temperature in the desert is 70 degrees, sounds great, right? Sure, except that 'average' comes from it being 135 degrees in the day and 15 degrees at night."
The thing is, Citrix knows this, and they're smarter than this. Too bad this report just makes Miercom (and by extension, Citrix) look like clueless idiots.
The big question of course is which remote protocol provides the best user experience for a certain number of users over a certain bandwidth? I'm going to go right back to my consulting here, and say that it depends on your users, your applications, your network, and your use case. To be honest I'm sure that ICA and PoIP are both fine. Sure, there are cases where ICA will kick PoIP's butt, and cases where PoIP will come out on top. (And wouldn't you know it, these just happen to be the perfect cases shown in each vendor's videos!)
In general I agree with Coleman about PoIP's dynamic tuning and his criticism of ICA's tuning parameters set by policy and admins instead of actual dynamic network conditions. But that's just at the policy-level. ICA itself will self-tune with regards to frame buffers and stuff.
But again it all goes back to use case. My personal unscientific opinion is that PoIP does a better job of tuning "up." That is side-by-side with a fast network, I'm going with PoIP over ICA. But in a bandwidth constrained slow environment, ICA does better. But those are both extremes... Which one works better for your particular case is something you'll have to figure out on your own.
One more thing to add here: Just as sort of a follow-on to the article I wrote on Friday, I think it's too bad that with Citrix, you have to pay for the WAN acceleration (Branch Repeater) as a separate add-on product, even though it's now just a virtual appliance you pop into your data center.
What Citrix/Miercom said:
"Flash video was delivered with an average of 65% less CPU usage, 89% less bandwidth, and excellent Quality of Experience by XenDesktop 4 compared to View 4."
What VMware/Mike Coleman said:
- More talk about PCoIP's ability to throttle down, and how the Miercom test on 100Mbps is not real world.
- If bandwidth is an actual concern, you can put hard limits on PCoIP.
- Citrix uses Flash redirection, versus VMware which was Flash remoted. Not apples-to-apples.
- Packet loss in the test was 0.5 to 5%, which is unrealistically high.
- PCoIP has "no reliance whatsoever on client side driver compatibility," but this is not the case of the Citrix Flash environment, which since it's redirection needs a local flash client as well as hardware powerful enough to run flash.
- Testing Flash only was narrow. What about Silverlight, H.264, Quicktime, etc? PCoIP does all this, plus new ones that haven't been invented yet.
I won't go into the bandwidth numbers here (Miercom claiming 89% less bandwidth for ICA) as you're now well aware of my thoughts on bandwidth reporting.
Coleman made the right decision to point out that Citrix is doing "Flash redirection" while VMware is doing "Flash remoting."
(A quick tech note: Citrix recently added "Flash redirection" to XenDesktop. This means that if the user is browsing the web in a remote XenDesktop session and they encounter Flash on a page, if the user is connected to XenDesktop from a Windows client device with Flash installed locally, the remote XenDesktop host will send the flash source files down to the client where they will be executed locally, thus bypassing the "remote" element. Citrix has long done this with multimedia and the Windows Media Player, but this Flash redirection is new. VMware is doing Flash the way they do everything else--it runs on the host and is remoted down to the client.)
On some level, customers shouldn't care whether Flash remoting or Flash redirection is being used. But to Coleman's point, requiring Flash to be installed on the client does provide something else you have to manage at the client and means the good experience of the Flash redirection only works on clients who have the capability to run Flash locally.
I also agree that a third problem with redirection (in addition to client software and client device capabilities) is that your software vendor needs to write their product to support it. So yeah, now in 2010 we're all happy that Citrix has Flash redirection, but from 1998 to 2009 Flash was virtually unusable via Citrix even though everyone wanted it. And today if you browse to a site with Silverlight via Citrix, it's not going to be good like Flash since they don't do Silverlight redirection yet. Of course when Citrix releases Silverlight redirection that will change, but how long will we have to wait for that, and will we want to manage Silverlight on our clients in order to get it? Meanwhile you can browse to a Silverlight site via PCoIP and it will be just like Flash or anything else via PCoIP.
Miercom also called out host-side CPU utilization, claiming 4.7% for Citrix versus 71.5%. This makes sense since the Flash video has to render somewhere, and Flash is crazy-intensive. (You want to find out right now? Check out nick.com while watching CPU.. Even on your client right now. Crazy, eh?) So with Citrix Flash redirection, that CPU consumption doesn't disappear, it just moves down to the client. Of course that's great for your server user density, but to Coleman's point, you only get that if your clients have the right version of Flash installed and if they have the horsepower to take on that workload.
So which approach is better? Who "wins" Flash? Again, it goes down to use case. What kind of Flash support do you need and what kind of client devices do you want to use? What's more important to you?
Resource consumption & scalability
What Citrix/Miercom said:
"Overall, XenDesktop 4 uses system resources more efficiently and is capable of scaling more effectively."
What VMware/Coleman said:
- Real dollars are wasted with ICA since it doesn't take advantage of money spent on network resources.
- Depending on redirection solely does not provide real scalability. This doesn't handle common office apps and requires the vendor (and therefore customer) to keep on investing to redirect every new technology.
- Miercom failed to call out costs of managing client CODECs and Flash versions.
- Only View 4 supports a true zero client.
Coleman finished with, "In closing, we really do believe that VMware View is the only desktop virtualization product that has been designed from the ground up to deliver and manage virtualized desktop environments as a service. And, we do that with less cost and complexity than any other solution on the market."
I covered Coleman's first three points in the previous section.
Coleman's statement that "Only View 4 supports a true zero client" is flat-out wrong. I'll be seeing Coleman this week during our Geek Week: VDI Challenge so I can ask him what exactly he means there.
The concept of a "zero client" is that it's a client device with "zero" management. For years, Teradici has marketed thin clients with the PC-over-IP hardware chips in them as "zero clients" because the clients didn't have a local OS, so they believed they were easier to manage than a traditional thin client.
The problem is that a Teradici hardware chip-based thin client (which is the "zero client" that Coleman is referring to) still has firmware which needs to be updated. And to me, that's not a zero client. In fact, ironically, in order to use a hardware-based PC-over-IP thin client that was built more than a few months ago, YOU HAVE TO FLASH THE FIRMWARE. How can anyone say that's a "zero client" with a straight face? How is that any different than a Linux- or CE or ThinOS client?
Is flashing the firmware difficult? Of course not. But it's establishing the precedent that these things are not "zero clients." (And hey, if they required a firmware flash for older PC-over-IP hardware clients to connect to the software-based PC-over-IP in View, what's to say that a future version of View won't require another firmware flash?)
I want to be very clear on this: I am NOT saying that firmware flashing is bad, and I think customers have no problem flashing firmware. It's just that you can't go around saying that you have a zero client when you've established precedence of requiring client firmware updates to get new feature support.
(For the record, my opinion is that there is only one true "zero client" on the market, and that's Pano Logic.) Those things can't even generate their own logon screens. They're just some bus-level remoting chips on the end of the network jack.)
Also, I don't really agree with Coleman's view that "VMware View is the only desktop virtualization product that has been designed from the ground up to deliver and manage virtualized desktop environments as a service." Is he suggesting that since Citrix grew up on Terminal Services, they're not designed from the ground up to deliver and managed virtualized desktop environments? And actually, wasn't ESX (on which View is based) designed from the ground up for server virtualization? Isn't that why ESX sucked for desktops up until the most recent v4 release a few months ago? So how can VMware make this claim?
Oooohh.. Wait. I see Coleman prefaced that statement with "we believe." (So yeah, VMware can believe it—doesn't means it's true though. :)
The final point I want to make about resource consumption & scalability is that VMware's PC-over-IP requires some serious processing power on your VM host. So you know those numbers that VMware was kicking around with View 4, like that with vSphere 4 and Nehalem you can get 16 VMs per core? Yeah, let's just say that's not with PC-over-IP enabled. So how many can you get? Again, again, again... that depends on your use case. (Because your apps and network connections will determine how hard your host has to work per VM, which is inversely proportional to user density.)
So who was right? Citrix or VMware? Miercom or Mike Coleman? ICA or PC-over-IP? Who won?
Like I wrote in the title, both of these vendors are half-right. Miercom pointed out some good things that Citrix has over VMware, but they mislead people in other areas. And Mike Coleman appropriately called "BS" on some points but added his own BS for others.
At the end of the day, both Citrix ICA and VMware PC-over-IP are fine protocols. Both are better than RDP, and both have the capability to provide a good user experience. There are many use cases where one is clearly the winner over the other, but the only scenario that actually matters is yours.
In 2010, I don't see many customers choosing one desktop virtualization over another based solely on display protocol. But again, it all comes down to use case. So figure out what your use cases are and test them. Test them with real users and real apps and real clients on your real network and see which one is better for YOU!