Once you finalize your strategy (or perhaps in order to help you solidify your strategy), you will need to run performance tests on your servers to determine the number of users that they can support. Though there are dozens of Internet sites and research studies that provide server-sizing benchmarks, it's impossible to get exact results given the variations between environments, including network speed, protocols, Windows profiles, and hardware and software revisions.
One mistake that many people make after performing benchmark tests is to assume that the system will scale linearly based on high-level information available in task manager. Suppose that a server with no users has 4% processor utilization and 150 MB memory utilization. With five active users, the processor utilization rises to 14% and the memory to 200 MB. A quick estimate reveals that the processor would max out at 48 users, with 630 MB memory utilization.
However, real world tests might indicate that 25 users can connect, but any more cause the system to run extremely slow, even though plenty of memory and processing power is available. In these situations, the server bottleneck is not visible on the surface with task manager. The disks may be overused or the server's system bus may be full.
It is imperative that a detailed analysis be performed to determine the true system utilization. It should be no surprise that Citrix and Microsoft recommend purchasing the fastest server possible. In most situations, however, this is not feasible, so you will have to test your server hardware to determine your maximum load.
Server Capacity Planning
There are several tools and techniques that you can use to determine the capacity and performance of your MetaFrame XP servers. To do this, you'll need to simulate user load on your servers and record the performance of the system. From there, you'll be able to determine the system's bottlenecks and capacity limits.
Regardless of the exact tool or technique that you use, all capacity planning and testing follows the same basic methodology:
- Choose the application or applications that you would like to use for the load testing.
- Determine what tasks a user will do within that application.
- Determine what performance speed or response time is required.
- Determine how users use the application.
- Create a script or automated process that can simulate a user using the application.
- Prepare to monitor the server's performance during the test.
- Perform the test by executing the scripts.
- Analyze the results.
- Ask your boss for more money to buy a bigger server.
Results from this method of server testing should tell you two things:
- How many users the server can support.
- How the server performs when it is highly loaded. (For example, does performance lag for current sessions, or does it stop accepting new sessions?)
Let's detail each of the testing steps.
Step 1. Choose your Test Application
When testing the performance of a server, the first thing you need to do is identify an application or applications to test. Ideally, you'll be able to test the applications that are most important to your business.
Step 2. Determine Test Tasks
Once you've determined an application that you want to use for your testing, you need to think about what users will be doing with that application. Is it a line-of-business application where users will be entering data into forms and running reports on that data, or is it a spreadsheet application where users will be performing calculations? Maybe it's a word processing application where users write documents?
Step 3. Determine Appropriate Response Times
Once you figure out how you will test the application, you need to determine what the appropriate application response time is. This will help you determine whether a server is too busy. For example, if your test application is Microsoft Word, you might determine that a letter must appear on the screen within 0.2 seconds of the user pressing the key. This would be your threshold for acceptable performance. Later on in your testing process, you may find that your server can support 130 simultaneous users before crashing, although each user has to wait 0.5 seconds for the key response delay. In this case, you may find that you can only support 80 users with the 0.2 second response time. Even though your consultant says 130, you would know that your server can really only support 80 users.
As another example, your users might need to pull up reports in a line of business application. You need to determine what the appropriate wait time is for them. If you decide that a user should not have to wait more than 15 seconds for a report, then it is unacceptable to put 60 users on a server if they must each wait 20 seconds for their reports.
Step 4. Determine how Users Use the Application
Once you determine the appropriate responsiveness of your applications, you need to determine how active your users are. For example, some users enter data into a screen, then rummage around through papers at their desk, then enter more data. For these users, you might discover that they can enter the data in 10 seconds, but that they only do this once per minute. On the other hand, more active users might perform the same 10-second transaction six times per minute.
Knowing your mix of users is important, because a server that can support 75 "slow" users might only be able to support 40 active users.
Step 5. Create the Application Simulation Script
Now that you've thought about the application that you would like to test and the way that users will use the application, you can begin thinking about the testing process itself. The main technique that you'll use in your capacity planning is to have multiple users access your application at the same time. By watching the performance of the system during this time, you can determine how the system will scale.
Instead of finding a bunch of users and asking them to "use the system" while you watch the performance, most people create user simulation scripts. These scripts simulate users using the system. The nice thing about creating a script is that you (as one single person) can test hundreds of users accessing the system at the same time. The other advantage of using a script instead of test users is that you can get consistent, repeatable loads and thus you'll know if any changes you make to the server actually affect performance.
An application simulation script is essentially a batch file that automates the process of a user launching and using an application. There are dozens of tools on the market that can be used to script your user sessions. The most popular ones detailed in Figure 6.3.
- URL: www.hiddensoft.com/AutoIt)
- Cost: Free
- URL: www.winbatch.com
- Cost: US $100
- URL: www.wintask.com
- Cost: US $100
Figure 6.3: Popular Windows application usage scripting tools
These tools offer a "recording" mode that allows you to perform some functions (such as typing a document, browsing the web, using PeopleSoft, etc). Once a script is recorded, you can play it back to simulate a user using the system.
When your application simulation script is complete, you should end up with a file or files that you can launch from the command line. (For example, "autoit.exe /word" or "myappscript.cmd.") The script should launch the application and then begin "playing back" the simulated user interaction.
Step 6. Prepare to Monitor the Performance
Before you start your testing, you need to configure your system so that it records the performance of your server during the testing. It's very important that your testing data is logged and saved. While you're conducting the test, you will be focused on creating a good test. You don't want to worry about trying to view the results of the test as you're conducting it. It's much better to to record the results so that you can view them in detail at a later time.
There are two ways to measure performance in MetaFrame XP environments:
- Use the Resource Manager component of MetaFrame XPe.
- Use Windows Performance Monitor.
Resource Manager is fully discussed in Chapter 16, so we won't spend time on it here. Besides, Performance Monitor is much more appropriate for recording the results of user capacity testing.
Think back to your Windows 2000 training. Do you remember how to use Performance Monitor to record performance counters to a log file so that you can view them later?
- In Windows 2000, launch the Performance MMC (Start | Programs | Administrative Tools | Performance).
- Expand the tree under "Performance Logs and Alerts" in the left pane. Right-click on "Counter Logs" and choose "New Log Settings…"
- Type a name for your new log file. (This should be a "friendly" name, such as "50 user test with MS Word."
- Click the "Add…" button on the screen that pops up. This is where you choose the specific counters to record. (Keep reading for a list of counters that actually matter when testing MetaFrame servers.)
- Highlight each counter and instance that you would like to record and click the "Add" button to add them to the log file. After you've selected all the counters you'd like, click the "Close" button to go back to the log file settings screen.
- By default, the system is configured to record a sample of the data every 15 seconds. Depending on your test size and hard drive space, you might want to increase the frequency to every 5 seconds or so.
- If you would like to change the path of the log file, you may do so by clicking the "Log Files" tab.
- You can also click on the "Schedule" tab to configure your log file so that it automatically starts and stops at specific times, although most people don't do this when they're running specific tests.
- Once your counter log is fully configured, it will appear in list in the Performance MMC. When it's time for your testing to begin, you can start the log by right-clicking it and selecting "Start" from the context menu. The icon next to the log will change from red to green.
Now that you know how to configure and save performance logs, there's one question that needs to be answered: What performance counters should you monitor? While there are many books that list "standard" performance counters and what they do, in the real world, there are only a few counters that matter when sizing MetaFrame servers.
Performance Counters that Actually Matter
At a minimum, you should capture the following performance counters when conducting your tests:
This counter shows the number of times per second that the server looked for something in physical memory, but instead was forced to go to the paging file on the hard drive. (Technically, this is known as a "hard page fault.") Ideally this number stays around zero. Any sustained value up around 20 or 30 indicates that you need might need more memory (or fewer users).
It's important not to confuse the "Pages/sec" counter with "Page Faults/sec." The Page Faults/sec performance counter shows the total of hard and soft page faults. Soft page faults occur when the system is ultimately able to find what it needs in memory without going to disk. Soft page faults are not as bad as hard page faults. (It's like "good" cholesterol and "bad" cholesterol.) The important thing to remember here is that you track the "Pages/sec" counter, not the "Page Faults/sec" counter.
Processor: % Processor Time: _Total
This counter shows how busy the processors are. If it pegs at 100% then you definitely need more of something. If the processor is too busy, don't automatically think that you need more processing power. For example, the processor might be busy because you're running out of memory, and the processor is spending a lot of unneeded time writing to and reading from the paging file.
If you notice that the processor utilization is fairly high, you might want to track the System: Processor Queue Length counter as well. This counter shows how many requests are backed up while they wait for the processor to get freed up to service them. By tracking this, you can see if the processor is very busy or too busy. (Yes, there is a difference.) A processor that is very busy might show 100% utilization, but it will back down as soon as another request comes through. You could see this because the Processor Queue Length would be almost zero. A processor that is too busy might also show 100% utilization, except that because it's too busy it cannot service additional requests, meaning that the Processor Queue Length would begin to fill up.
Physical Disk: % Disk Time
This counter essentially shows you how busy your server's hard drives are. A value of 100% would indicate that the disks are 100% busy, meaning that you might need faster disks, more memory, or fewer users. Like with the processor counters, if your % Disk Time counter is at or near 100, you might also want to monitor the Physical Disk: Current Disk Queue Length counter. This counter will tell you how many disk requests are waiting because the disk is too busy.
If you have multiple physical disks in your server (that are not mirrored), you should record a separate instance of this counter for each disk instead of one counter for the total disks. That way, you can determine whether your disks are being used evenly, or if one disk is overworked while the other sits idle..
Network Interface: Bytes Total/sec
If you're worried about the utilization of the server's network card, you might want to record the Bytes Total/sec of the card. If you do this, keep in mind that a 10 Mb/second network interface is 10 megabits, and the performance counter tracks bytes. Since there are 8 bits in a byte, the performance counter would max out at 1.25M bytes. If you factor in physical network overhead, the actual maximum of a 10Mb network is about one megabyte per second.
In the real world, most MetaFrame servers are connected to the network at 100Mb speeds or faster, and other server hardware components max out long before the network interface.
Terminal Services: Active Sessions
This performance counter simply provides a count of the total number of ICA sessions (or RDP sessions) that are active on your server. This counter is extremely valuable when you're analyzing your performance logs after you've captured them. It lets you see exactly how many users were active at any given time during the log period.
In addition to these specific performance counters, there are six performance objects that are MetaFrame-specific and Terminal Services-specific. Most people forget that they exist. These object classes include:
- Citrix IMA Networking counters monitor IMA background communication traffic.
- Citrix MetaFrame XP counters monitor MetaFrame XP traffic such as zone elections, local host cache, IMA data store communications, and application enumerations.
- Citrix Secure Ticket Authority counters monitor Citrix Secure Gateway. (See Chapter 15 for more information about Citrix Secure Gateway.)
- ICA Session counters (available when using Feature Release 1 or 2) monitor the details of specific (or the aggregate totals of) ICA session bandwidth, including printing, compression, and the various virtual channels. (If you don't have MetaFrame XPe then you will only be able to see the latency-related ICA session counters.)
- Terminal Services counters monitor the total, active, and inactive Terminal Services (and MetaFrame) sessions.
- Terminal Services Session counters monitor the details of specific sessions from the perspective of Terminal Services. These counters are similar to the ICA Session counters, except that they monitor the Terminal Server-specific elements of each session. These elements include the percentage of processor time that a session uses, the number of handles and threads a session has open, session input and output bytes, frames, errors, and cache performance of a session.
Step 7. Conduct the Test
Now that you have your application simulation scripts created and are ready to monitor the performance of the server during the test, you can begin the actual testing process. At first, you might think that you need to connect dozens of client workstations and manually invoke your scripts on each one, but fortunately there is a cool utility from Citrix called the Citrix Server Test Kit (CSTK) that can automate this process.
The CSTK is a utility used to run application simulation scripts from multiple ICA clients. It can even launch several sessions from a single client. Basically, it allows you (as a single person) to run tests that simulate dozens or even hundreds of users on one server. From there, you can watch your server's performance with different user loads. The CSTK is free, although you have to download it from the Citrix Developer Network (www.citrix.com/cdn). It does not come on the MetaFrame XP CD.
To use the CSTK, you will need several ICA clients in addition to the server that you want to test. These can be any platform and any type of ICA client, although there are advantages (that will become apparent later) to using 32-bit Windows clients.
The CSTK can run multiple ICA sessions from one client as long as the client device can support multiple ICA sessions and it has the resources to run as many sessions as you plan to test on it. Realistically, memory is usually the limiting factor. A good rule of thumb is that you need about 12MB of memory for each session that you want to run on a client device. For example, a Windows workstation with 128MB of memory should be able to support about 10 simultaneous ICA sessions. If you want to test your server with 100 user sessions, you would need 10 workstations, each with 128MB of memory.
Several components make up a complete CSTK environment:
- The CSTK Console. This is the main interface to the CSTK. It allows you to start and stop your tests, apply simulation scripts, and configure test user accounts.
- The user simulation scripts. These are the scripts that you created back in Step 5. The CSTK comes with some generic scripts for basic tasks (such as using Internet Explorer, calculator, notepad, etc). It also comes with more robust scripts for Office 97 and Office 2000.
- The CSTK Client. This is a small program that runs in every ICA test session. It is responsible for running the user simulation scripts. It's added to the "All Users\Startup" folder when you install the CSTK.
- The Client Launcher Utility. This program can be used on 32-bit Windows clients to automatically launch multiple ICA sessions that are used to run the test scripts. In small test environments, this tool is not necessary. However, if you plan to test 20 sessions from a single workstation, this tool will save you from having to manually start and logon to 20 different sessions.
In order to understand how the CSTK works, let's review step-by-step how it's used:
- The first thing you need to do is to prepare your environment. Ideally, you'll be able to run your tests on an isolated test network. Gather your MetaFrame server and the necessary ICA client devices. (You'll probably want to activate the Citrix licenses on your test server so that an annoying popup window does not break your scripts every 10 minutes. Just remember that "officially" Citrix advises against activating your server until it is finalized. It's your call. See Chapter 14 for the gory details of licensing.)
- If you haven't done so already, install the CSTK on your server. (Remember to put your server into install mode first.) After the CSTK is installed, you'll notice that the "CSTK Client" is automatically launched whenever a user logs on. For now, you can just ignore that.
- Launch the CSTK administrative console. (Start | Programs | Citrix Server Test Kit 2.1 | CSTK Console) You will use this console to configure your testing environment and run your tests.
- Next, import the application simulation scripts you created in Step 5 into the CSTK environment. This process will make these scripts available to the CSTK. To do this, choose Tools | Add Application Scripts. You can specify anything you want for the Script Name. Use the "Browse" button to browse to the path of the executable of your script. You can specify any necessary command-line parameters in the "Parameters" box. For example, if you used AutoIt to create your script, you might need to specify AutoIt.exe in the "Program Name" box, and yoursrcriptname.txt in the "Parameters" box. Specify whether your application applies to "Normal Users" or "Power Users." Normal users will only run one script at a time, and power users will run multiple scripts simultaneously.
- After you've added the application scripts, you need to configure groups of test users that will use your scripts (User Group | Add, or click the "+" button on the toolbar). When you specify users, you're essentially indicating which application scripts run for which users when they log on. When adding a user group, the first question you're asked is whether you want to add a group of Normal Users or Power Users. Make your selection and click "OK."
- Next, you need to specify a range of usernames that will run an application script (or scripts) when they log on. Specify the usernames by entering the basename and the number of users. For example, to apply a script to users "brian1" through "brian5," you would enter "brian" as the basename and "5" as the number of users. If this is the first time that you're using the CSTK and you don't have any test user accounts created, you can click the "Create Users" button. This will create the test user accounts based on the baseline name and number of users. These user accounts are created with blank passwords.
- Before you click "OK," highlight the script or scripts that you want this user group to run and click the "Add" button. If you elected to create normal users, selecting multiple scripts will cause the users to run them one by one and the list of available scripts will only show those that you've designated for normal users. If you elected to create power users, selecting multiple scripts will cause the users to execute them all at the same time, and the list of available scripts will only show those that you've designated for power users. In a sense, normal users execute their application scripts in a "serial" fashion, and power users execute them in a "parallel" fashion.
- Once you add a group of users and click "OK," you'll see them listed on the main CSTK console screen. You can add as many groups of users as you want (as long as the basenames are not the same in two different groups).
- At this point, the CSTK is fully configured and you should save your testing environment configuration. You can save the entire configuration, including user groups and applications, by choosing File | Save Configuration File. Your settings are then saved as an INI file with a .CST file extension. You can load your settings into the CSTK so that you don't have to manually set up everything from scratch in the future. When you save a configuration file, it does not include the application script information. When adding application scripts to the CSTK, they are available until they are deleted. If you delete one, loading a configuration file where it was used will not bring it back.
- In order to begin the testing process, choose Test | Start Test or click the lightning bolt button on the toolbar. You'll notice that starting the test doesn't actually do anything. You have to log users on in order for the scripts to execute. This is also a good time to start your performance monitor logging process as described back in Step 6.
- From one of your ICA client devices, log on as one of your test users. This should be a user that is configured in one of the user groups in the CSTK console. Since a shortcut to the CSTK Client was added to the "All Users\Startup" folder when the CSTK was installed, it will launch after logon and the appropriate application script or scripts will start to run.
- In order to easily launch multiple ICA sessions from a single 32-bit Windows client device, you can use the CSTK Client Launcher. Log onto a client workstation and run CSTKlaun.exe from the "ClntLaun" folder of the CSTK directory. When you run it, it will detect the path of the ICA client executable (wfcrun.exe). Verify that this path is correct and enter the usernames that you want the sessions to be run from. The username entries follow the same baseline syntax as the groups within the CSTK. For example, if you have 10 test workstations that you plan to use for 10 sessions each, you would configure your CSTK for usernames "test1" through "test100." Then, you would configure the CSTK Client Launcher to use "test1" through "test10" on the first workstation, "test11" through "test20" on the second workstation, and so on.
- After you specify the users that will run on a workstation, you need to click the "Create Entries" button in the CSTK Client Launcher to create custom ICA connections for each user in the workstation's Program Neighborhood. Clicking this button brings up a screen that allows you to specify the default options used for each connection (such as the name of the server to connect to, protocols, etc.) Configure your options as needed and click "OK."
- Before you run your test, click the "Advanced Delay" button to specify the delay between sessions. This allows you to specify how much time passes between launching sessions. One of the nice features is that it allows you to specify progressively more time as more sessions are launched, allowing you to anticipate slower responses as the server gets more loaded.
- After you've configured the delay, you can click the "Run" button on the Launcher's main screen. ICA user sessions will begin to be launched, and they will run the scripts that you specified for the user groups in the CSTK console.
As you add users to your testing environment, you should add them in small groups. For example, if you're testing 100 users, you might want to add 10 users every five minutes for the first hour or so, and then add users one-by-one. By doing this, you'll be able to figure out how each user affects the overall system.
Don't forget to stop your performance monitor recording log once your testing is complete. Once it's stopped, you can examine it to determine the results of your test.
Step 8. Analyze the Results
Once you've stopped your performance monitor log, you can view the results within the Performance MMC console. With the "System Monitor" object highlighted in the left-hand pane, click the button with the picture of the cylinder on it and browse to your log file. This will configure the graph to pull its data from the log file instead of from the current system activity.
Even after you configure the graph to get its data from the log file, you'll notice that the graph is still blank. You need to manually add the performance counters that you want to view. Use the "+" button on the toolbar, just like you would with live data. The only difference is that when you're displaying data from a log file, the only performance counters listed will be the ones that you recorded in the log file.
As you analyze your results, keep in mind to look for the bottlenecks in your system. Every system will max out at some point. If your system seems to be running out of memory (due to high page faults) then you can probably add more. Once you do that and run the tests again, you might find out that you can support 10 additional users but then your processors start to max out (due to a consistent Processor Queue Length).
Another fact to keep in mind is that when you use the CSTK, the CSTK client utility takes about 2.8MB of server memory for each session that is tested. Also, capturing performance logs takes up system resources.
Based on the data from the log file of your test, you can probably figure out the point in which your server performance starts to fall drastically. Looking at the Active Sessions counter will tell you how many sessions there were when that performance drop occurred. Once you determine how many users your system can hold, you may want to run your tests again. For this second round of tests, you might ask live users to log on in addition to your test users. The live users will be able to tell you whether the system is usable or not.
Third Party Server Sizing Tools
If your MetaFrame environment is really important to your business, or if you work for a consulting company, there are some third-party server sizing and stress test tools that are much easier to use than the CSTK, and they produce much more accurate results. (Of course the downside is that they are expensive.)
The two most popular tools are StressTest by Scapa Technologies (www.scapatech.com) and LoadRunner by Mercury Interactive (www.mercuryinteractive.com). These two tools are similar. In addition to being easy to use, they have the ability to test the performance of a MetaFrame server from end-to-end, instead of only testing the impact of multiple user sessions and application execution as with the CSTK. They can test the aggregate affects of server load, network bandwidth, compression, encryption, and ICA virtual channel use.
Also, because they work by using dedicated testing workstations to monitor session performance, the testing itself doesn't impact the results (unlike the CSTK which itself consumes server resources).
The process by which these third party tools are used is the same as when the CSTK is used. You still have to write application simulation scripts. However, the third party tools make it easier to run the tests and interpret the data.
Both the Scapa tools and the Mercury tools are widely used, and it's up to you to determine which is more appropriate for your environment. Mercury Interactive has always positioned themselves at the top tier of testing products, and their pricing for their MetaFrame tools reflects that. (Their product is more than twice the cost of Scapa's product.) However, if you work for a consulting company, Scapa offers a "roving" license that allows you to take their testing tools from customer to customer.
Advantages of Third-Party Testing Tools
- They test the performance of the entire system, not just the server.
- They employ outside "control" testing stations, so the act of testing does not skew the testing results.
- They provide the most accurate, end-to-end MetaFrame sizing and stress testing.
Disdvantages of Third-Party Testing Tools
- They are expensive (approximately US $25,000 for Scapa and US $50,000 for Mercury)