HI all,
I'm doing the migration to terminal server infrastructure of a my customer, and we have a trouble.
The infrastructure is composed by n.2 terminal server in network load balancing, so at the moment we have a total of 60 users who connect every day to this service (about 25/30 users for each terminal server).
Our goal, at the end of the migration, is to host about 120 users in total (about 60 users for server).
Both server are Windows Server 2003 Enterprise Edition (32-bit) with 8 GB RAM.
The trouble starts when one Terminal server starts to host more than 30 users, the system become unstable, and EventID 1500 and EventID 1508 appears in the Event Application Log.
So we did registry changes as said in the following article : http://support.microsoft.com/kb/935649/en-us
This is strange, cause Brian Madden book says : “You should be able to get four or five hundred users on a system before having to think about kernel memory tuning”.
We don’t have the switch “/3GB” set in boot.ini, and we had set “Programs” in the Memory Usage section of Performance Settings window, but we still have problems.
We downloaded “Symbols” and “Debugging tools for windows” from Microsoft website, and we entered in kernel debug mode.
Here the output during the crash :
------- start ------
*** Virtual Memory Usage *** Physical Memory: 2096553 ( 8386212 Kb) Page File: \??\C:\pagefile.sys Current: 12335104 Kb Free Space: 11515888 Kb Minimum: 12335104 Kb Maximum: 14430208 Kb Available Pages: 1316111 ( 5264444 Kb) ResAvail Pages: 1965208 ( 7860832 Kb) Locked IO Pages: 177 ( 708 Kb) Free System PTEs: 125762 ( 503048 Kb) Free NP PTEs: 32329 ( 129316 Kb) Free Special NP: 0 ( 0 Kb) Modified Pages: 201 ( 804 Kb) Modified PF Pages: 58 ( 232 Kb) NonPagedPool Usage: 32631 ( 130524 Kb) NonPagedPool Max: 65215 ( 260860 Kb) PagedPool 0 Usage: 64518 ( 258072 Kb) PagedPool 1 Usage: 16215 ( 64860 Kb) PagedPool 2 Usage: 16246 ( 64984 Kb) PagedPool 3 Usage: 16160 ( 64640 Kb) PagedPool 4 Usage: 16267 ( 65068 Kb) ********** Excessive Paged Pool Usage ***** PagedPool Usage: 129406 ( 517624 Kb) PagedPool Maximum: 134144 ( 536576 Kb) ********** 84 pool allocations have failed ********** Session Commit: 49319 ( 197276 Kb) Shared Commit: 68140 ( 272560 Kb) Special Pool: 0 ( 0 Kb) Shared Process: 42372 ( 169488 Kb) PagedPool Commit: 129470 ( 517880 Kb) Driver Commit: 1920 ( 7680 Kb) Committed pages: 1152006 ( 4608024 Kb) Commit limit: 5130416 ( 20521664 Kb)
--------end-------
We started to debug pool memory usage, but without any good result.
We can’t understand what uses Paged Pool memory, and if our 32bit platform is a real bottleneck.
I attach some other logs, captured after some users logged off (without system restart):
POOLMON_G_E -> poolmon report with “mapped_driver” and totals
Memory: 8386212K Avail: 5441552K PageFlts: 13645 InRam Krnl: 3096K P:490944K Commit:4620632K Limit:20521664K Peak:5651232K Pool N:130392K P:492200K System pool information Tag Type Allocs Frees Diff Bytes Per Alloc Mapped_Driver MmSt Paged 1568213 ( 56) 1432320 ( 42) 135893 154704872 ( 1120) 1138 [nt!mm - Mm section object prototype ptes] IoNm Paged 52352765 (1102) 52224325 (1102) 128440 33573680 ( 0) 261 [nt!io - Io parsing names] File Nonp 33653591 ( 327) 33513184 ( 327) 140407 21456040 ( 0) 152 [<unknown> - File objects] Ntfr Nonp 718618 ( 0) 467274 ( 0) 251344 16086984 ( 0) 64 [ntfs.sys - ERESOURCE]se] MmCa Nonp 6284903 ( 204) 6152851 ( 190) 132052 14687680 ( 1344) 111 [nt!mm - Mm control areas for mapped files] CM35 Paged 5651 ( 0) 5471 ( 0) 180 11268096 ( 0) 62600 [nt!cm - Internal Configuration manager allocations] MmSm Paged 675853 ( 0) 550183 ( 0) 125670 8042880 ( 0) 64 [nt!mm - segments used to map data files] NtfF Paged 118211 ( 0) 110497 ( 0) 7714 7220304 ( 0) 936 [ntfs.sys - FCB_INDEX] Ntfn Nonp 679502 ( 0) 549732 ( 0) 129770 5191392 ( 0) 40 [ntfs.sys - SCB_NONPAGED]G] NtFs Nonp 1936979 ( 0) 1807560 ( 0) 129419 5179288 ( 0) 40 [ntfs.sys - StrucSup.c]ta] MmCm Nonp 100720 ( 1) 100640 ( 1) 80 4158736 ( 0) 51984 [nt!mm - Calls made to MmAllocateContiguousMemory] CMAl Paged 4293 ( 0) 3469 ( 0) 824 3375104 ( 0) 4096 [nt!cm - internal registry memory allocator pool tag] Obtb Paged 25218 ( 2) 24188 ( 0) 1030 2704848 ( 8192) 2626 [nt!ob - object tables via EX handle.c] Thre Nonp 532421 ( 4) 528291 ( 6) 4130 2577120 ( -1248) 624 [nt!ps - Thread objects]rk context] LSwi Nonp 1 ( 0) 0 ( 0) 1 2576384 ( 0) 2576384 [<unknown> - initial work context] Sob2 Nonp 31742798 ( 321) 31730196 ( 321) 12602 2200920 ( 0) 174 Unknown Driver- ACPI object data] TPLA Nonp 512 ( 0) 0 ( 0) 512 2097152 ( 0) 4096 Unknown Driver- ACPI power data] Key Paged 219228545 (7350) 219210320 (7343) 18225 1895352 ( 728) 103 [<unknown> - Key objects]s]data] sOBi Paged 47147 ( 0) 0 ( 0) 47147 1893960 ( 0) 40 Unknown Driver- Key objects]s]ta] Vad Nonp 7754311 ( 280) 7716482 ( 261) 37829 1815792 ( 912) 48 [nt!mm - Mm virtual address descriptors] TCPt Nonp 13572704 ( 728) 13572666 ( 728) 38 1662560 ( 0) 43751 [TCP - TCP/IP network protocol] FSim Paged 357928 ( 0) 345404 ( 0) 12524 1603072 ( 0) 128 [nt!fsrtl - File System Run Time Mcb Initial Mapping Lookas Even Nonp 31275027 (1260) 31243539 (1260) 31488 1552560 ( 0) 49 [<unknown> - Event objects] BCM0 Nonp 11 ( 0) 0 ( 0) 11 1518776 ( 0) 138070 Unknown Driver- Adapter objects] NtFS Paged 84420 ( 0) 78131 ( 0) 6289 1387312 ( 0) 220 [ntfs.sys - SecurSup.c]ffer] CM16 Paged 2287 ( 0) 2025 ( 0) 262 1171456 ( 0) 4471 [nt!cm - Internal Configuration manager allocations] CMDa Paged 2206319 ( 27) 2198597 ( 7) 7722 1167136 ( 2032) 151 [nt!cm - value data cache pool tag]amed buffer] Ntfo Paged 370752 ( 0) 363107 ( 0) 7645 1133648 ( 0) 148 [ntfs.sys - SCB_INDEX normalized named buffer] AtmA Paged 214340 ( 0) 198208 ( 0) 16132 1075536 ( 0) 66 [<unknown> - Atoms]data cache pool tag]er][...]
Total Nonp -2101447180 (51995) -2102462435 (51871) 1015255 101874896 ( 18992) 0 0Total Paged -463362528 (106596) -463962788 (106465) 600260 242451600 ( 17896) 0
POOLMON_G_P_P_E -> poolmon report with only paged memory, “mapped_driver” and totals
Memory: 8386212K Avail: 5472020K PageFlts: 5216 InRam Krnl: 3096K P:491140K Commit:4602920K Limit:20521664K Peak:5651232K Pool N:130396K P:492392K System pool information Tag Type Allocs Frees Diff Bytes Per Alloc Mapped_Driver MmSt Paged 1568529 ( 1) 1432599 ( 2) 135930 154814312 ( -88) 1138 [nt!mm - Mm section object prototype ptes] Ntff Paged 614967 ( 0) 493185 ( 0) 121782 99374112 ( 0) 816 [ntfs.sys - FCB_DATA] IoNm Paged 52452305 ( 34) 52323858 ( 35) 128447 33578904 ( -64) 261 [nt!io - Io parsing names] NtFs Paged 2054648 ( 0) 1866268 ( 0) 188380 20641136 ( 0) 109 [ntfs.sys - StrucSup.c] CM35 Paged 5651 ( 0) 5471 ( 0) 180 11268096 ( 0) 62600 [nt!cm - Internal Configuration manager allocations] MmSm Paged 675947 ( 0) 550221 ( 0) 125726 8046464 ( 0) 64 [nt!mm - segments used to map data files] NtfF Paged 118723 ( 0) 110903 ( 0) 7820 7319520 ( 0) 936 [ntfs.sys - FCB_INDEX] R100 Paged 93 ( 0) 49 ( 0) 44 5458000 ( 0) 124045 Unknown Driver TSwd Paged 3991 ( 0) 3835 ( 0) 156 4201336 ( 0) 26931 [rdpwd.sys - RDPWD - Hydra Winstation Driver] CMAl Paged 4293 ( 0) 3469 ( 0) 824 3375104 ( 0) 4096 [nt!cm - internal registry memory allocator pool tag] Ntf0 Paged 2099460 ( 3) 1970213 ( 4) 129247 3232232 ( -24) 25 [ntfs.sys - general pool allocation] Obtb Paged 25222 ( 0) 24194 ( 0) 1028 2696656 ( 0) 2623 [nt!ob - object tables via EX handle.c] Ntfc Paged 279460 ( 0) 248771 ( 0) 30689 2209608 ( 0) 72 [ntfs.sys - CCB_DATA] sOBi Paged 47149 ( 0) 0 ( 0) 47149 1894040 ( 0) 40 Unknown Driver Key Paged 219411335 (4192) 219393206 (4187) 18129 1885368 ( 520) 103 [<unknown> - Key objects]s] FSim Paged 357965 ( 0) 345411 ( 0) 12554 1606912 ( 0) 128 [nt!fsrtl - File System Run Time Mcb Initial Mapping Lookas NtFS Paged 84824 ( 0) 78527 ( 0) 6297 1389560 ( 0) 220 [ntfs.sys - SecurSup.c] Wmit Paged 21 ( 0) 2 ( 0) 19 1179976 ( 0) 62104 [<unknown> - Wmi Trace] CM16 Paged 2287 ( 0) 2025 ( 0) 262 1171456 ( 0) 4471 [nt!cm - Internal Configuration manager allocations] Ntfo Paged 372329 ( 0) 364601 ( 0) 7728 1149120 ( 0) 148 [ntfs.sys - SCB_INDEX normalized named buffer] AtmA Paged 214351 ( 0) 198233 ( 1) 16118 1074560 ( -88) 66 [<unknown> - Atoms] CMVa Paged 11172632 ( 163) 11157530 ( 43) 15102 992120 ( 6720) 65 [nt!cm - value cache value tag] CMDa Paged 2207201 ( 43) 2199733 ( 1) 7468 973544 ( 17832) 130 [nt!cm - value data cache pool tag] Toke Paged 14058504 ( 182) 14057276 ( 182) 1228 870128 ( 0) 708 [nt!se - Token objects] Port Paged 375389 ( 1) 371109 ( 1) 4280 793296 ( 0) 185 [<unknown> - Port objects]
[...]
Total Paged -313723193 (58140) -314803134 (57901) 1079941 380987688 ( 30744) 0 0
POOLMON_G_P_P_S_E -> poolmon reports with only paged memory for terminal sessions, “mapped_driver” and totals
Memory: 8386212K Avail: 5459532K PageFlts: 10812 InRam Krnl: 3096K P:491236K Commit:4602436K Limit:20521664K Peak:5651232K Pool N:130388K P:492476K All sessions pool information Tag Type Allocs Frees Diff Bytes Per Alloc Mapped_Driver TSdd Paged 167461 ( 27) 166255 ( 26) 1206 27765416 ( 8192) 23022 [rdpdd.sys - RDPDD - Hydra Display Driver]] Ttfd Paged 334105 ( 6) 317935 ( 6) 16170 21022288 ( 0) 1300 [<unknown> - TrueType Font driver] Gla1 Paged 31746 ( 23) 26097 ( 3) 5649 11659536 ( 41280) 2064 [win32k.sys - GDITAG_HMGR_LOOKASIDE_D Gcac Paged 42429 ( 0) 40958 ( 0) 1471 10673928 ( 0) 7256 [<unknown> - Gdi glyph cache] TSwd Paged 55 ( 0) 29 ( 0) 26 6176768 ( 0) 237568 [rdpwd.sys - RDPWD - Hydra Winstation Driver] Bmfd Paged 1927 ( 0) 1147 ( 0) 780 5507064 ( 0) 7060 [<unknown> - Font related stuff] Gla5 Paged 291709 ( 64) 278774 ( 29) 12935 5070520 ( 13720) 392 [win32k.sys - GDITAG_HMGR_LOOKASIDE_S Gla: Paged 39987 ( 21) 32355 ( 3) 7632 5006592 ( 11808) 656 [win32k.sys - GDITAG_HMGR_LOOKASIDE_L Ggb Paged 16043 ( 0) 14813 ( 0) 1230 4490344 ( 0) 3650 [<unknown> - Gdi glyph bits]] Gcwc Paged 21 ( 0) 0 ( 0) 21 3440640 ( 0) 163840 [win32k!ConvertToAndFromWideChar - GDITAG_CHAR_TO_WIDE_CHA GhO5 Paged 72299 ( 0) 71670 ( 0) 629 2432664 ( 0) 3867 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE Gpfe Paged 8949 ( 0) 1008 ( 0) 7941 2306792 ( 0) 290 [win32k!PFFMEMOBJ::bAllocPFEData - GDITAG_PFF_INDEXES] Gh05 Paged 349757 ( 70) 348939 ( 70) 818 2178184 ( 0) 2662 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE GhB5 Paged 101023 ( 152) 100629 ( 129) 394 2058328 ( 7064) 5224 [win32k.sys - GDITAG_HMGR_SPRITE_TYPEE GhD5 Paged 51388 ( 9) 50792 ( 9) 596 1996128 ( 0) 3349 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE Ghab Paged 83673 ( 0) 66658 ( 0) 17015 1905680 ( 0) 112 [win32k!FHOBJ::bAddPFELink - GDITAG_PFE_HASHBUCKET] GhT5 Paged 51977 ( 9) 51562 ( 9) 415 1860336 ( 0) 4482 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE GhZ5 Paged 51670 ( 9) 51367 ( 9) 303 1672384 ( 0) 5519 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE Gpff Paged 8949 ( 0) 1008 ( 0) 7941 1670536 ( 0) 210 [<unknown> - Gdi physical font file] Gh85 Paged 76219 ( 9) 75910 ( 9) 309 1668600 ( 0) 5400 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE Gh35 Paged 46940 ( 14) 46565 ( 14) 375 1642832 ( 0) 4380 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE Gebr Paged 54978 ( 7) 54448 ( 4) 530 1635488 ( 81264) 3085 [<unknown> - Gdi ENGBRUSH] - GDITAG_HMGR_LOOKASIDE_R Gla4 Paged 513084 ( 98) 504747 ( 76) 8337 1600704 ( 4224) 192 [win32k.sys - GDITAG_HMGR_LOOKASIDE_R GhE5 Paged 69655 ( 0) 69344 ( 0) 311 1589280 ( 0) 5110 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE GhU5 Paged 79637 ( 105) 79351 ( 105) 286 1555520 ( 0) 5438 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE GhN5 Paged 53374 ( 0) 53147 ( 0) 227 1544536 ( 0) 6804 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE GhW5 Paged 79523 ( 51) 79130 ( 51) 393 1515128 ( 0) 3855 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE GhJ5 Paged 56975 ( 9) 56614 ( 9) 361 1483544 ( 0) 4109 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE Gh]5 Paged 41231 ( 9) 40936 ( 9) 295 1443600 ( 0) 4893 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE GhH5 Paged 55012 ( 9) 54602 ( 9) 410 1440376 ( 0) 3513 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE Gh=5 Paged 35933 ( 0) 35633 ( 0) 300 1271640 ( 0) 4238 [win32k.sys - GDITAG_HMGR_SPRITE_TYPE
Total Paged 31100486 (5227) 30896864 (5055) 203622 160711472 (170336) 00
Thank you all for your help,
Hi,
To answer you question about what uses paged pool memory, sessions do. Also the registry for each session. Look through the registries in your users profiles, see if theres one in there that is just massive, maybe corroupted. That could be exhausting all of your paged pool.
And yes, definitely look in to 64 bit. See if its viable for your environment. Take in to consideration things like device drivers, odbc drivers, print drivers, etc etc... Its definitely worth looking in to.
--Mike
Hi Mike,
thank you very much for your reply.
You're right, maybe there is one user's registry corrupted, but how can I find it? Should I have to search for excessive size of their "ntuser.dat" or I can find it in a better way?
I'm searching for some article that can explain me the limit of users for 32bit systems, but I can't find one that really give me an idea to find the limits of my servers.
My users generally use :
1. Outlook + Word + Excel2. IBM Client Access3. Printers4. Internet explorerEach session use about 110 MB of RAM. The server has 8 GB RAM, but i think this is not so important.Someone have an idea about how many users of this type a 32bit server can host?
Thank you,
Mario
Might be worth worth changing the entry for PoolusageMaximum to 50 in
HKLM\system\currentcontrolset\control\session manager\Memory Management\
PoolUsageMaximum = 50
The key is responsible for the trimming frequency for paged pool.
Our environment is like yours in many ways. Its hard to get more than 30 users on one 8GB box. The performance goes down real fast at that point. I am talking usability. I am sure we can get 100 people to log in but they won't be able to do anything. I would personally shoot for 30 per server. We use HP ML-350 G3 4x72gb 15k drives, 8gb ram. 8gb page file. Not many people today understand what a page file does or how it works. You can tell who these people are when you hear them say ' I am going to put more ram in the machine so I can go with a smaller page file'. Even if you had say, 500gb ram, YOU STILL NEED A PAGE FILE. The system offloads processes to it in an effort to free ram for future use. Yes, the operating system thinks about the future. If you don't give it the page file it needs, it cannot manage the real ram the way it was designed to do.
For 32-bit servers, the free TCT GPO templates from Login Consultants are fantastic as they include almost all of the tweaks you need to mitigate user density issues with 32-bit platforms. There are tweaks for kernel memory settings, SMB settings, etc that can make a big difference.
Alan Osborne
President (MCSE, CCNA, VCP, CCA)
VCIT Consulting - Citrix/Terminal Services Remote Desktop Solutions for SMB
VCIT website My Blog
If you still have that old memory dump, open it up again in the debugger and do a !poolused c
You should see something like:
lkd> !poolused c Sorting by Session Paged Pool Consumed
Pool Used: NonPaged Paged Tag Allocs Used Allocs Used Ghs5 0 0 232 1406576 Gdi Handle manager specific object types: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys Ttfd 0 0 431 581128 TrueType Font driver TSdd 0 0 44 357936 RDPDD - Hydra Display Driver , Binary: rdpdd.sys Gla1 0 0 169 348816 Gdi handle manager specific object types allocated from lookaside memory: defined in w32\ntgdi\inc\ntgdistr.h , Binary: win32k.sys Gcac 0 0 53 280336 Gdi glyph cache
Should show you where your pooled mem is being consumed.
If you're concerned about page file tuning watch your commit limit and your commit peak. If you're no where near your limit during normal load, you can trim your page file down. Too big of a page file causes you to swap a lot.
Hope it helps,