Marketplace

High Pages/Sec & Disk Transfers/sec, in the Citrix XenApp / Presentation Server forum on BrianMadden.com

rated by 0 users
This post has 0 verified answers | 73 Replies | 3 Followers

Suggest an Answer

Points 1,097

Chris Norman posted on 03-24-2008 2:29 PM

rated by 0 users

Hey folks
Hp DL360s G4p
MS 2003 Enterprise Server SP2
PS 4.0 Ro4
8 Gigs of ram and 3.40 mhz dual processors

We publish out the entire desktop (about 30 users to a server)

I'm investigating some stalls we seem to have every so often throughout the day. The users complain about latency where they type and nothing happens for a few seconds. I've tried everything I know to resolve this. I've got local text echo on (set on the web Interface)but it doesn't seem to be helping.

The normal resource monitor shows nothing out of the ordinary, the cpu is normally bouncing from 20 to 30% and the ram usage is rarely above 4gig.

Today I'm running perfmon and watching Pages/sec & Disk Transfers/sec
I'm seeing the page/sec jump up and down at times between 750 and 15,000 for an hour or so. Then it will calm down and sit at zero with some quick 1500 spikes. In some instances I see the Disk Transfers/Sec keeping the same peaks and dips. This seems crazy high to me. I do have the /PAE switch in the boot.ini file but I’m starting to wonder if the ram (above 4gig) is even being used. Is this normal to see during peak times during their day?

Senior Administrator (Citrix)
USI Holdings

No matter where I am i'm never where I want to be.

| Post Points: 140

All Replies

Points 7,633

Rick Mack replied on 05-21-2008 6:07 AM

rated by 0 users

Hi People,

Having coped with perfmon for years, I was delighted to find a freeware (slightly crippled) alternative in the Quest Software downloads that is everything that perfmon ought to be, Spotlight on Windows. It'll give you an instant feel for what's happening on your sluggish systems.

Check out:

http://www.quest.com/spotlight-on-windows/

It'll need minimal registration, but believe me, it's worth it. And that comment isn't because I'm a Quest employee.

regards,

Rick

Ulrich Mack
Quest Software
Provision Networks Division

| Post Points: 35

Points 1,097

Chris Norman replied on 05-21-2008 9:31 AM

rated by 0 users

Rick you run this on each box or from a central point?

Senior Administrator (Citrix)
USI Holdings

No matter where I am i'm never where I want to be.

| Post Points: 20

Points 1,817

Sam Rodriguez replied on 05-21-2008 11:34 AM

rated by 0 users

Spotlight-On-Windows sure is purty! Though, between Windows Performance Monitor and the Citrix Resource Manager Summary database, one should have plenty to draw from. I would say that the Resource Manager canned reports leave something to be desired and I wished it had a dynamic "Report Creation Tool" (hint, hint). Not that big of a deal, since the summary database is contained in MS-SQL2K (or 2K5). If you're any good at MS-Access, you can configure an ODBC connection, link the tables you need to query from, and create your own reports. When I was looking at sessions/user count, I was able to use the .csv export tool in CRM to grab snapshots of every server at specific times of the day (and night) over a 7-day period. I then imported the .csv's into Excel and created my graphical report (3D ribbon style) that way - I already had similar data using PerfMon, but there is no data export utility in Server 2003 PerfMon - all you can to is bring up the log, add the counters you want to view and do a screen grab of the image. Unfortunately, even when you orient the 8-1/2"x11" page to landscape, 7 days gets rather compressed. So, if anyone from Microsoft is listening...

I did test the Provision Networks "Block-IT" tool 18 months ago (to replace Trust-No-Exe on our Server2K farm, which Server2K3 DID NOT like - http://www.beyondlogic.org/solutions/trust-no-exe/trust-no-exe.htm) "Block-IT" was intuitive and I liked the interface, but for us, it added yet another console! The whole product line seemed better suited to VMWare to me. Anyway, I felt that rather than spending scarce capitol on a pretty tool, I should learn how to implement "Software Restriction Policy" - Already built in to the Microsoft AD environment.

Temporary Internet Files and Cookies are not available for redirection through GPO. I like the RAMdisk idea, but that would take some Reg hacks to make Windows store the user profiles there. It would seem that one solution is to increase disk performance to cope with it. There's no doubt in my mind now that RAID5 performs better than RAID1 in this scenario. The least expensive way to deal with this is content, or client re-direction of the IE application itself. That should essentially offload disk activity to the client. There are some considerations though. If I recall this only works over PNAgent and Thin-Clients could become an issue. Here's a pretty good article on this (van Bragt)

http://www.msterminalservices.org/articles/Client-Server-Content-Redirection-Explained.html

http://support.citrix.com/article/CTX113457

Samuel A. Rodriguez
Sr. Systems Administrator

| Post Points: 20

Points 150

Andy Friar replied on 05-21-2008 12:03 PM

rated by 0 users

you run this on each box or from a central point?

By the looks of things from a central point which pulls the counters and puts out a nice graphical view where you can drill down to see more detailed info.

Looks like a handy tool that i'll certainly start to use.

Temporary Internet Files and Cookies are not available for redirection through GPO

Theres no default GPO that would do this no, but under User Shell Folders there are both Cache and Cookies available to either script or reg2adm file and then you'll be able to set via GPO.

Rgds

Andy Friar

| Post Points: 20

Points 1,097

Chris Norman replied on 05-21-2008 12:17 PM

rated by 0 users

Theres no default GPO that would do this no, but under User Shell Folders there are both Cache and Cookies available to either script or reg2adm file and then you'll be able to set via GPO

Might want to try this. It's called the True control template and you can get it from Login.
http://www.loginconsultants.com/index.php?option=com_docman&task=cat_view&gid=20&Itemid=149
I use it to redirtect the home folder to a drive letter rather than a varable. It actually has all the shell folders wrapped up in a GPO. It also has all the settings Rick posted earier in the thread on tuning SMB. It's a great GPO to have.

Senior Administrator (Citrix)
USI Holdings

No matter where I am i'm never where I want to be.

| Post Points: 20

Points 1,817

Sam Rodriguez replied on 05-21-2008 4:42 PM

rated by 0 users

In other words - a reg hack called by a GPO. Very useful info, however the problem remains where to put the folders (and maintain or increase performance while offloading from native disk)? I knew that it could be done, was not aware that it was commonplace. Andy, Thanks. http://safari.oreilly.com/0789716747/ch07lev1sec1

I still think the best place (to re-direct) would be a RAMDisk. Not to hijack this thread, but I've only found a couple of vendors for software RAM disks that utilize installed memory.

http://www.superspeed.com/servers/ramdisk.php - Fairly expensive, but Gold Certified.

http://users.compaqnet.be/cn021945/Download/download_and_buy.htm - Very inexpensive! Developed in Belgium? Seems they have a "Full Support" version. Anyone ever try this one?

Samuel A. Rodriguez
Sr. Systems Administrator

| Post Points: 5

Points 7,633

Rick Mack replied on 05-21-2008 4:50 PM

Hi Sam,

I agree that you can get a lot out of the RM database. Jason Conger's web interface for RM and the XPS product are clear indications of what can be done and with the schema fully exposed by something like Visio Professional all the performance data is there for the taking.

The thing I like about Spotlight for Quest is the ability to "see" everything relative to the performance of a server on one screen, and the history function is a bonus. Citrix could have done something like this but RM, like IM, has never gone anywhere.

As far as consoles go if you were running the full Provision product you'd only have one console for everything, TS, VDI, DDI and even Softgrid integration. Maybe Citrix will get it all together for Xenapp server 5? :-)

On the folder redirection side of things, redirecting the cookies and favorites folders can't be done by a standard managed GP. But while the managed GP does stuff like actually moving files out of the user's profile to the redirected folder, a bit of scripting plus an unmanaged folder redirection policy works just fine. If you've ever turned on managed folder redirection in a production environment and suffered the performance hit that resulted from the file moves, the idea of a scripted after-hours migration is a whole lot less painful.

And if you use the free deskstop standard plugin, you've got fully reversible folder redirection.

I prefer unmanaged folder redirection because it actually has significantly less overhead on login and lets you do useful stuff like tie the redirected folders to a users home drive. That makes migration/consolidation etc a lot easier.

I think that thin clients will eventually morph into something half way between a fat and today's thin client. A lot of the "multimedia" functionality that we want and often need requires redirection to the client. That will mean a local browesr, media player, codecs, maybe an audio encoding engine etc.

regards,

Rick

Ulrich Mack
Quest Software
Provision Networks Division

| Post Points: 59

Points 1,817

Sam Rodriguez replied on 05-21-2008 5:01 PM

rated by 0 users

Rick, I couldn't agree more with everything you've just said.

Best regards,

Samuel A. Rodriguez
Sr. Systems Administrator

| Post Points: 5

Points 235

Ross Walker replied on 05-22-2008 11:32 AM

rated by 0 users

[quote=Andy Friar]
I suppose it depends on the type of business your in as to the need to store cookies, whilst cookies are a nicety i'd hardly call them a necessity.

I do agree that a better solution would be to somehow be selective over which cookies are stored, logistically it would be a nightmare but a blanket 30 day or 1k cookies could work.

Alternatively just redirect the cookies to the users home drive but you wouldn't really be fixing the problem just moving it to the fileserver.
[/quote]

Andy,

I discovered something else. I took one of our users with the largest amount of cookies, 6K, and I had them export their cookies, delete their cookies and all offline files and re-import their cookies and 6K cookies became 2K cookies.

From that I can only assume that 2/3 of all cookie files are dead, out-of-index files that keep getting merged back in to the users' profiles.

By moving these to the users' homes I can probably avoid that cookie accumalation, but it will be problematic, if a user were to stay logged in to a desktop system, then try to establish a terminal server connection from home that night, as the two systems would fight over the cookie folder.

If you ask me Microsoft made a HUGE mistake in storing cookies as separate text files instead of just storing the cookie data right in the index file. It uses up a ton of disk io as it is and is a total mess to manage.

If somebody could come up with a replacement DLL for IE that moved the cookie data from flat files to the index db file I would cough up dough right now for it. The DLL will of course need to defrag the file every now and then, but it could be made intelligent to track % fragmentation and defrag on open or close if it's over X%.

-Ross

| Post Points: 20

Points 150

Andy Friar replied on 05-22-2008 1:10 PM

rated by 0 users

Sam

http://www.superspeed.com/servers/ramdisk.php - Fairly expensive, but Gold Certified.

Tried this to alleviate issues with profiles on a fileserver, the IO throughput it puts out was incredible.
Ross

6K cookies became 2K cookies.

Thats a huge saving just dropping that much, 16Mb of small files from a single profile gone.

Whats your average disk queue length like on both the citrix servers and the file server where the profiles are being loading from? and do you have a BBWC on both of these?

If you ask me Microsoft made a HUGE mistake in storing cookies as separate text files instead of just storing the cookie data right in the index file. It uses up a ton of disk io as it is and is a total mess to manage.

Couldn't agree with you more.

Rgds

Andy Friar

| Post Points: 35

Points 387

Daniel Schoppmann replied on 05-26-2008 3:35 PM

rated by 0 users

Hi Rick

Thanks for the tip with Spotlight. I like that app, although I still have a lot of questions.
I described our environement in my post of April 19, 2008 in that same thread.
Since than we did something to our farm and I have a good and a bad message.
The good message is, that by further SMB-Tuning (http://blogs.technet.com/askperf/archive/2007/09/21/windows-explorer-and-smb-traffic.aspx)
our Current Commands went from over 80 down to normal. That's sounds great but guess what we still have user complaining of seconds freezes in LN-Client.
Beside additionally turned of TOE and RSS, I started using Spotlight on both the Lotus Domino Server and the File Cluster.
The first things that caught my attention were the Pages/sec. I got period (minutes) where values didn't go under serveral thousand (~5000). Looking further for the processes that possible could have a lot of page faults I only found svchost.
My question here is how many pages/sec is still good for sustained period of time. Why the treshold (red) seem to be so low in Spotlight. (starts with about 300, I guess)
2nd thing I found very interesting, was the average disk queue length on the data vol (1,8 TB) on the LN-Server. There is one disk, that hast a a average physical disk queue length of over 2 over a sutained period of time, like 10 minutes. The logical disk queue length was even higher.
On the file server the 2 Volumes I have consist of 90 disks. Interesting was, that the logical disk queue length was higher than 2 a nearly all the time, but on the physical level there was no disk that was over 2 for a sustained period of time. In my eyes I would say that I always have to take a look on the physical disk queue length and the logical disk queue length means nothing. Is that true? and does that also refer to the value disk load, because that is nearly every time at 100% in the logical view.
Rick, I would love to hear what you can say to this.

Bye, Daniel

| Post Points: 5

Points 1,817

Sam Rodriguez replied on 06-03-2008 3:20 PM

rated by 0 users

Just chiming back to check on this thread. Pretty impressive with 55 replies - just to go silent. I wonder if the BBWC upgrade resolved this issue for Chris?

Has anyone got any experience with the RAMDisk from QSoftRAMDisk?

Has anyone resolved this issue using redirection of Cookies and Favorites folders? If so, did you redirect to your SAN or implement a high speed storage array (disc enclosure)? Could you elaborate on how you configured the interconnects?

Regards,

Samuel A. Rodriguez
Sr. Systems Administrator

| Post Points: 20

Points 1,097

Chris Norman replied on 06-03-2008 4:22 PM

ok
I was holding out so I could have a better handle on what we though the real issue was. We had 30 days to resolve this problem and we were under a ton of pressure to get this resolved so may things got done it was hard to keep track of the changes.

Here's where I stand....No more text lag! It took us months of work to get under control and it appears, from where I’m sitting that it had nothing to do with the Citrix servers.

Here what I did on my end(41 Servers total all HP):
-Updated all HP firmware and drivers (normal stuff)
-Made sure they were all up to the same patch levels(again normal stuff)
-on the G4 and G5 servers I added the Battery chache to the array
-I disabled teaming on all the G5 server (to rule it out)
-Made a new image and applied it to the G5 servers (12 servers)
-disabled rss and tcp offloads on the g5 servers that had nics that supported it.

what i didnt do was put the cache in the G3 server (6 servers). I wanted to ahve some sort of control and they are going bye bye soon so I didnt want to waste the $$. When I was done I had a way to see if the cache was helping or if the new image made a difference. The users noticed no change in performance no matter what server they were on. I however could see improvements on a few things it just wasn't the silver bullet we were looking for.

I set aside a G5 server I did re-image and I did put the cache in. I put a set of users on it. they said they didn't notice any difference. I left them there so we could do further testing. It was 8 people. We will come back to these 8 people in a sec.

Now... The messaging team decided to rebuild the exchange servers as well. They took a new Dell server we had replaced the boradcom nic with intel and we replaced the Q-Logic Fiber HBAs with Emulex. They built the OS up and re-patched everything and installed Exchange. These guys like me had been batteling poor performance. They orignally had 2 servers set up as a cluster (Two brand new Dell 2950's). By the way, we are running Exchange 2007. I don't know if I mentioned that or if it really matters.

We took the 8 users I had on my test server and migrated their mail file over to the new exchange server. They saw a huge improvement in performance. So the question at this point was is the new build better or are they getting good performance because they are the only 8 people on the exchange server? We needed to get a load on the box to really tell.

So.. the past 2 weeks have been spent migrating all the users from one exchange server to the new one. This Sunday evening they completed the task. Now everyone is in the new exchange box and for the first time in months we have gotten no performance complaints.

What did it? what was the silver bullet? It's anybody’s guess at this point. I called each person involved in their respected area and asked them what exactly they did. We gained some performance when we got rid of the q-logic fiber cards on the exchange server. Everything I did on the Citrix end gained a little better performance. But I think the kicker was the Broadcom network cards and the "Scalable Networking Pack Release" that ended up on the server with SP2. This enabled some features on the newer nics that didn't play well in our environment. Broadcom released some driver updates to try and address the issue but they didn’t seem to work. So we replaced them with Intel cards and turned off the rss and tcp offload features. We all seem to agree it was the biggest problem.

Here’s the link again if you folks wanna take a gander at it.
http://msexchangeteam.com/archive/2007/07/18/446400.aspx

How would this effect Citrix? Well people live out of Outlook and outlook was having communication issues I could see this in the edgesight console. It appears that the tconnection between Outlook and Exchange wouldn't ime out but hang until it got what it wanted. Thus effecting the user experience.

Senior Administrator (Citrix)
USI Holdings

No matter where I am i'm never where I want to be.

| Post Points: 32

Points 387

Daniel Schoppmann replied on 06-05-2008 5:16 AM

rated by 0 users

Hi Chris

Thanks for you detailed answer. I still have some questions:

1: If I got you right, it would do absolutely no harm to turn of the chimney with "netsh int ip set chimney DISABLE", to find out if this solves the problem. Is this right?
2: On what servers do you do that. Only on the backend server (like Exchange or Lotus Notes)? Or also on the CITRIX Servers?
3: We also have EdgeSight and we see a lot of "Not Responding" for Lotus Notes and also other apps. On what value you saw the bad user experience for Outlook in EdgeSight?

Bye, Daniel

| Post Points: 20

Points 1,097

Chris Norman replied on 06-05-2008 8:56 AM

rated by 0 users

1. You would think so. I believe they tried that but it didn't help. Which makes me wonder if that was the complete issue. In the end it very well could have been a combination of things. They tried disabling it, loading the latest Broadcom drivers. In the end they installed the Intel Nics and that seemed to work much better.

2. I did it on my Citrix servers that had the feature. I noticed no change in performance. Where is seemed to have made a difference is on the exchange mail box servers. One thing to note here. I noticed on the Citrix servers when I would disable the feature on the HP teaming software interface and it would still be active if you look at the physical card settings. So I disabled it in both places to be safe.

3. Not responding errors in Edgesight are going to always be there. If I saw Outlook not responding every once in a while I wouldn't worry too much it's when I'd get 300+ alerts at a time that we really took notice. Two or Three times a day we'd get spammed with outlook not responding errors in mass.

One of the things we are learning (remember we were a Notes shop up until Nov 07) is that we may have scaled the Exchange environment wrong. Notes handles mail db size much better than Exchange. So we had people with 4, 5 & 6 gig mail files. When you move 2000 users like that into exchange you notice a huge problem. Also when emc scaled this out for us I don't think they took into consideration that none of these users were going to be running in cache mode. It's all Citrix. So there's a constant leach by each user on that server. We are having to build multiple servers and cap how many go one each mail box server. We are also trying to implement an archiving solution to off load all the old mail.

BTW, Just a small rant... The Notes back end is way better than exchange. We were running 2000 users on One domino server with no issues. We have invested in 10 servers for outlook and over a mil in emc consulting. Personally I’m dumfounded at the decision to switch.

Senior Administrator (Citrix)
USI Holdings

No matter where I am i'm never where I want to be.

| Post Points: 20

Previous | Next

Page 4 of 5 (74 items) < Previous 1 2 3 4 5 Next > | RSS

High Pages/Sec & Disk Transfers/sec, in the Citrix XenApp / Presentation Server forum on BrianMadden.com

rated by 0 users This post has 0 verified answers | 73 Replies | 3 Followers

All Replies

rated by 0 users
This post has 0 verified answers | 73 Replies | 3 Followers