I manage a farm that includes 10 Citrix Presentation Server 4.0 servers that all run very slowly at times. The slowness is usually a few second delay in the keystrokes appearing on the screen and hanging windows. These servers all are running Windows Server 2003 Standard SP2 and are fully patched. They are running on an ESX cluster that includes (2) ESXi hosts and (1) ESX 3.5 host. These ESX hosts are HP DL380 G5 boxes. There are about 300 users in the company. There are also other non-Citrix VMs on these ESX boxes. These other VMs include DCs, print servers, app servers, and file servers. They are also running Windows 2003 Stand SP2. There are about 30 VMs total (including the 10 Citrix desktop servers).
I am not sure why the Citrix servers seem to have sporadic latency issues with just about every app. The apps on these Citrix servers include Word 2003, Outlook 2003, and other lesser known apps. Users complain that keystrokes do not appear for a few seconds when replying to emails, creating Word documents, and typing in other apps. Also there are times where a window will hang before bringing up the next window or menu. When these apps are run locally they run much faster. Users connect to the Citrix desktop servers thru a Web Interface. They are then able to launch their published apps. We have about a 50/50 mix of fat clients and thin clients. Both types of workstations have the latency issues at times.
I have added a 2nd vCPU to each Citrix server and each has 2GB or RAM. When I look at performance monitoring during peak activity all 10 of the Citrix servers they are normally running <50% CPU and RAM usage. So this tells me that the servers “should” have enough CPU and RAM resources. I have moved these Citrix servers to different ESX hosts and they still run slow on the other hosts. I have also updated the ICA clients on the workstations which did not seem to help. VMWare tools are also up to date.
Any ideas on what I can check? Thanks.
The "delay in the keystrokes appearing on the screen and hanging windows" symptom is almost always network related. I'd look at the utilization on the NIC(s) on the ESX host to start. But the issue could be anywhere from the vNIC in the guest to the user's workstation. We usually find that the pipe (LAN/WAN) from the client to the host is saturated or that a device like a router or switch is discarding packets.
What is the best way to look at the utilization on the NIC(s) on the ESX hosts? Is there a good freeware tool that I can use? Thanks.
I can say that I have seen the same problem with PS Servers on VMware and have scratched my head a bit, have a look at optimising the environment:
http://www.thincomputing.net/blog/citrix-and-terminal-servers-on-vmware-esx-3.html
Also we need to remove all variables from the issue and run one PS server on one ESX host.
I use the smcconsole.exe (Citrix SDK or ThomasKoetzing.de) to track latency. This can be run at the client or server end.
Also look ast the policies you have in place on the SpeedScreen side.
--Emil
You should be able to see that in the VMWare Infrastructure Client. Click on a host, then the Performance tab and change the chart (Switch to:) to Network. Nice little graph showing your data rates comes up. Should be low and spikey. If it's high and flatish then that means it never gets a break and is over-utilized
I am doing some additional monitoring and am noticing that the msiexec.exe process is launching on some of the Citrix servers when users logon. This process then takes up about 40-50% of the CPU resources for about 5-10 seconds before going down to 0%. It doesn't happen on all 10 of the Citrix servers but I did see it happen on two of them. What I did was log on the server once and open Task Manager. I then launched a 2nd session connecting to that same desktop server. That is when I saw that process appear. I was able to make it happen multiple times on two of those servers.
I thought that msiexec.exe (Windows Installer) only ran when an app was being installed. Why would it launch when logging on to a Citrix server? Thanks.
This could be happening because there is a requierment for an application that to complete a user portion of the install (populate HKCU) Have a look in the Event Viewer to see which app is calling MSIEXEC.
Then you can either pre-populate or find another workaround.
I would also look for descrepancies between application versions on different servers(e.g. Acrobat Reader 8 and 9), to see if they are overwirting the the same keys each logon.