Hi,
We are encountering some strange problems with a XenApp 5.0 farm that runs on Windows Server 2008 SP2. The farm consist out of 2 XenApp 5.0 servers with a single server that hosts the web interface. Underneath the OS we run VMware ESX 3.5 update 4. Usually everything runs ok but's the performance is very bad. The XenApp servers have 2 vCPU's and 4GB memory. We use roaming profile with folder redirection for some folders.
WE publish the desktop to our users so we installed all application on the 2 servers en use NTFS permissions for application access.
The problems:- Performance is not ok. When users are typing in Word (Office 2007) or other applications the characters will display with some latency, not always it happens sometimes.
- One server becomes unresponsive. Application that are allready started work very slow. New application which are started from the Start Menu won't appear. After some minutes the server goes further and all application will appear. It looks like the server is sleeping for a while. When that happens the System log shows the following errors:
7011 - A timeout (30000 miliseconds) was reached while waiting for a transaction response from the RasMan service. 7011 - A timeout (30000 miliseconds) was reached while waiting for a transaction response from the NlaSvc service. 7011 - A timeout (30000 miliseconds) was reached while waiting for a transaction response from the CryptSvc service. 10010 - The server {AAC1009F-AB33-48F9-9A21-7F5B88426A2E} did not register with DCOM within the required timeout.
Kind regards,
Onno van den Berg
Looks like many people have the same problems. The problems with hanging services (timeouts) i think only occur if you're using roaming profiles and folder redirection. We have removed the roaming profiles and went back to local profiles and we don't have those issues anymore. I'm very curious how many users you guys can run on a single Citrix server with Windows 2008, XenApp and a local installation of MS Office 2007 without having performance issues. Right now we are having some performance issues on a dedicated server. All apps are installed locally on the server not many +/- 8 apps. The server is a HP DL380 G4 with 4GB memory en 2 CPU's and we're running max 12 users when we begin to receive calls about performance issues. The server is just a workaround for the problems we had in the upcomming weeks we will begin to design a new enviroment with multiple servers. The problems with hanging services must be resolved by then i hope MS will find the cause of it very soon en will release a patch.
Maybe it's an idea to collect all current open call number so all of you can refer to those numbers. It's time MS finds the solution.
We have our servers running under ESX, with a 3GB reservation for RAM and a 2GHz reservation with 2 CPU's. (Typically though, under normal load the servers grab around 4GB of RAM and around 2.5GHz-3GHz for CPU.) All of the servers are imaged from the same base image.
Most of our users are connecting via Wyse Thin Clients and run full desktops. We have quite a few apps that access resources via network shares and DFS. Normally, we sit around 15-20 users when things are balanced out well. From what we have been seeing, the issues generally start to crop up around 20+ users (if a server or two is pulled from the farm for maintenance or other reasons...), but still can happen with a lower user count.
Jamie
@Jamie
How do you present your applications to your users. Are they published by other XenApp server to their session or did you install all applications local on the servers ?. Do you run Office 2007 ?.
Onno
@OnnovdBergThe majority of our users are running full desktop sessions with the applications running local to the server that they are connected to. Currently, we are using Office 2k3, with Outlook running constantly and Word/Excel running pretty frequently.
thanks for the feedback.
we had some strange problems while users working on the XenAPP server starting to complain that their applications were slow. The memory usage of the server is indeed pretty high (mainly i think caused by windows 2008 which consumes more memory then windows 2003 and maybe because we are running Office 2007) but there were some strang things with it. just because we all have problems with the network connectivity (servers that become unresponsive because many users logon of logoff at the same time, the use of roaming profiles and folder redirection) i asked the user to copy his documents to the desktop which is stored local on the server. the performance was much better, his excel document loaded much faster than before. well the fileserver runs windows 2003 so maybe there are smb conflicts which decrease the speed and causes very strange problems. did you guys experienced the same results ?.
Just got off the phone with MS and the person that I have been working with mentioned KB 974441 as something to try because the associated hotfix contains a newer version of rdbss.sys than what previous KB hotfixes have had. I will be installing this after I get a complete dump over to them today.
@OnnoI have noticed a difference in memory consumption between 2003 and 2008 in general. Not that it uses more for everything per se, but that the kernel likes to use up as much as it can grab and only hands off what the other non-system processes absolutely need. It may have been like that in prior versions, but it seems more evident in 2008.
For us, the performance problems are not always evident. The problems really seem to start shortly before the freezes occur. As for the SMB conflict possibility, I would suggest disabling SMB 2.0 if you can and seeing if that helps. We have had it disabled for a while just to rule out any issues with it and have not noticed anything adverse from doing so.
We just recieved a call from our customer that they experienced performance issues while opening Excel doucments from the fileserver it doesn't matter if it's a big documents or small it just takes a long time to download the file. When they copy the file to the desktop it opens very fast. So i disabled the SMB2.0 on the XenAPP server. The fileserver runs Windows Server 2003. Fat clients don't have the same performance issues, so i hope that disabling SMB2.0 will resolve it.
Guys,
Can we stay on the same subject here... Please start a new post for performance issues...
Thank you
Sorry, didn't realize it was a big deal. My problems still relate to the original topic and not so much performance issues. I was simply responding back to someone asking for input...
Thanks.
Has anyone tried R2 yet???
I don't believe this issue to be performance related. We are running Win2k8x64. The servers have 32 GB available to the OS (the maximum for Win2k8x64 standard) and 16 GB available to the RAM-cache (we are using Citrix Provisioning Server so writes to the system drive are cached in RAM based on our configuration). Reads to the system drive are going over 20GB network connections to Provisioning Servers hosting the system drives on a fiber-channel SAN (backed by many RAID-10 LUNs). We have dual quad-core processors and have seen this issue with as few as 25 users. Microsoft has confirmed that they do not suspect this as being caused by a performance bottleneck (for us). We mainly have Microsoft Office 2007 installed locally on the TS'es but users can access over 80 virtualized (VMWare ThinApp'ed) applications via SMBv2 network share).
We have not tested this as a logoff-induced issue as forum contributors have indicated. However, we have tested numerous other variables as per previous postings.
Most of our file servers (and TS servers) are using SMBv2. However our roaming profiles server continues to use SMBv1. We seem to encounter the issue more frequently when the majority of our file servers are using SMBv1 (or our TS servers are forced to use the SMBv1 client). We seem to encounter the issue less frequently with our file servers (and TS servers as a client) are using SMBv2. This is somewhat contradictory to what othes are encountering.
Microsoft continues to troubleshoot this with us. We recently provided them with a lot of data and they found nothing significant. They have requested a second network (packet-level) trace upon issue reproduction (which is easy at this point... load up a server about 50 users using SMBv1... takes less than 8 hours to reproduce).
This morning Microsoft got back to us and admitted that after examining our memory dumps it appears that our problem is a known issue.
When opening our incident I noted the other incident ID that Jeauxk sent over to me as being very similar to ours and as it turns out, the private hotfix that was sent over to Jeauxk is the same that is being sent to us. So, thank you again Jeauxk for sending that ID over to me. Unfortunately, (I believe it was stated in this thread before) the hotfix won't be publicly available until sometime in December.
I'll post any findings once I have them.
@gavin
I am unable to send you a PM with the case ID for some reason. Please send me a PM so I can send it, or post a comment like you did, this time with an email address.
I'm glad you are getting the patch, I'm still error free to this day. I did re-confirm today, the patch has been slotted for December.
Just to clarify what our issue actually was, this is the description of the problem from MS:
"It is an issue that presents as a hang and is caused by a deadlock between the Memory Manager and the Redirector Service. "
We do have the same issue. Our environment is XenApp 5.1 running on W2k8 Enterprise SP2 and we use Citrix Provisioning Server 5.1. Hardware is IBM blades with 8 GB of RAM in order to fulfill memory requirements.
We did test with SMB with no luck. HotFix KB974441 is applied and this seems to have reduced the server hangs from one server per day to one per week.
Cases are open with Microsoft and Citrix, both waiting for a memory dump which is impossible to deliver thanks to provisioning. Some of our servers will be changed to local boot.
@jeauxk: would you share your call ID so I can try the private fix as well?
Kind regardsStephan