session reliability - xte.exe keeps crashing, in the Citrix XenApp / Presentation Server forum on BrianMadden.com

Hi,

Review the error.log file under C:\Program Files\Citrix\XTE\logs - post back anything that looks useful.

Also, post your httpd.conf file under C:\Program Files\Citrix\XTE\conf

Do you ever exceed 150 concurrent connections through the XTE service? If so, read this article:

http://support.citrix.com/article/ctx107902

It sounds like remote users are using the full PN client with a custom ICA connection to connect to a desktop session.

A few questions:

- Do you have ports 1494 (ICA), 2598 (CGP for SR), and port 80 (for XML service) open on the public interface of your firewall?
- Is SR enabled on the Options tab of the custom ICA connection for each remote user?
- Are all users configured for "TCP/IP + HTTP"?
- In the Address List section of the Custom ICA connection, what FQDN is assigned and does that FQDN resolve to an IP address that matches the altaddr value on the CPS server that's NATted to this public IP?
- Do the firewall settings in the custom ICA connection have "use alternate address for firewall connection" enabled?

Have you considered using the web client with a CSG/WI server in a DMZ instead? Much easier to manage and generally easier to troubleshoot. That's because CSG proxies all the connections and allows you to use a single public IP to load balanced ICA session to your entire farm. You can configure multiple STAs for redundancy too.

Cheers,

hi alan

thx a lot for your effort.
some of the answers to your questions cannot be obtained today.
i will post a full reply tomorrow.

thx again.

hi alan

thank you for your response. below are the answers. i have removed servernames and ip addresses.
i hope this helps to understand the setup and shed some light onto this issue.

Review the error.log file under C:\Program Files\Citrix\XTE\logs - post back anything that looks useful.

these entries are showing every time the xte.exe fails with the message "faulting module unknown"

[error] SSL Library Error 45 on servername:443 with peer xxx.xxx.xxx.xxx: the cryptographic security of the SSL-connection cannot be guaranteed.

[notice] Parent: child process exited with status 3221225477 -- Restarting.

[notice] async engine: Not enough preallocated wsa buffer, allocate 132 wsa buffer
[notice] Parent: child process exited with status 3221225477 -- Restarting.

[error] SSL Library Error 20 on servername:443 with peer xxx.xxx.xxx.xxx: The operation completed sucessfully.
[notice] Parent: child process exited with status 3221225477 -- Restarting.


these entries are showing every time the xte.exe fails with the message "failing application XTE.exe, Version 4.5.0.64631,

faulting modul libaprutil.dll, Version 4.5.0.64631, fault address 0x0000101c."

[notice] Parent: child process exited with status 3221225477 -- Restarting.

[error] (20014)Error string not specified yet: AsyncEng Err: No real data in the bucket brigade
[error] (620018)APR does not understand this error code: socks_process_read_completion: failed to write data to remote IP

[xxx.xxx.xxx.xxx] session =[ 29 ]
[notice] Parent: child process exited with status 3221225477 -- Restarting.

Also, post your httpd.conf file under C:\Program Files\Citrix\XTE\conf

#Citrix_Begin
#Server Root Path
ServerRoot "C:\Programme\Citrix\XTE"
PidFile logs/xte.pid
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 3


ThreadsPerChild 150
MaxRequestsPerChild 0


ServerName localhost
MultiplexerHandshakeTimeout 100000

# Apache Modules
LoadModule access_module modules/mod_access.so
LoadModule

log_config_module modules/mod_log_config.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module

modules/mod_proxy_http.so
# XTE Modules
LoadModule socks_module modules/mod_socks.so
LoadModule winevent_log_module

modules/mod_winevent_log.so
LoadModule cgp_module modules/mod_cgp.so
LoadModule multiplexer_module

modules/mod_multiplexer.so
LoadModule schannel_module modules/mod_schannel.so
LoadModule async_engine_module

modules/mod_async_engine.so
LoadModule ticket_module modules/mod_ticket.so

AsyncWorkerThreadCount 0
RequireTicket Off

#SSL
Relay Listening Port
Listen 443
ProtocolMultiplexer *:443
ProtocolSignature SOCKSV5 \005
ProtocolSignature CGP \032CGP

#NameVirtualHost
NameVirtualHost *:443

#Http Proxy Configuration

ServerName servername
#SSL Protocol
Engine State
SSLEngine On
#Certificate hash or ID
SSLCertificateHash "have removed hashkey"
#Allowed
Protocol (SSLv3, TLSv1)
SSLProtocol +SSLv3
#Allowed CipherSuite (ALL,COM,GOV)
SSLCipherSuite ALL

ProxyPass http://127.0.0.1:80



#SOCKS Configuration

*:443>
ServerName servername
#SSL Protocol Engine State
SSLEngine On
#SOCKS Protocol
SocksProtocol On
#Certificate hash or ID
SSLCertificateHash "have removed hashkey"
#Registered Protocol (mod_multiplexer)
RegisterProtocol SOCKSV5
#Allowed Protocol (SSLv3, TLSv1)
SSLProtocol +SSLv3
#Allowed CipherSuite
(ALL,COM,GOV)
SSLCipherSuite ALL
SocksHandshakeTimeout 100000
#Destination Servers and Ports

/destination>
Order Deny,Allow
Deny to all
Allow to servername:80
Allow to
servername:1494
Allow to xxx.xxx.xxx.xxx:1494
Allow to xxx.xxx.xxx.xxx:80


#Citrix_End

Do you ever exceed 150 concurrent connections through the XTE service? If so, read this article: http://support.citrix.com/article/ctx107902
no, not that many. we grant up to 25 users access per server.
overall we have no more than 90 concurrent connections open at any time.

It sounds like remote users are using the full PN client with a custom ICA connection to connect to a desktop session.
the users are connecting but are not seeing a desktop. their active directory accounts are configured to start a certain application from a specific location (under the tab "environment"). all users are using the same application.

Do you have ports 1494 (ICA), 2598 (CGP for SR), and port 80 (for XML service) open on the public interface of your firewall?
we don't manage the firewall. according to the people who do, only port 443 is open to the public internet.

Is SR enabled on the Options tab of the custom ICA connection for each remote user?
we have instructed them to do so. however, they are scattered all over and we have not called every single one of them to have them confirm it. do you think it would matter if session reliablity is enabled on the server but the client does not use it?

Are all users configured for "TCP/IP + HTTP"?
all connections are encrypted. they are using ssl/tls + https. minimum encryption level is 128 bit.

In the Address List section of the Custom ICA connection, what FQDN is assigned and does that FQDN resolve to an IP address that matches the altaddr value on the CPS server that's NATted to this public IP?
the server's fqdn is entered within the address list of the custom ica connection and does resolve to an ip address. not sure where to check the second part of your question.

Do the firewall settings in the custom ICA connection have "use alternate address for firewall connection" enabled?
no, that option is not enabled. the only things entered here are "proxytype: none (direct connection)" and the servers fqdn is entered in the secure gateway field with port 443.

Have you considered using the web client with a CSG/WI server in a DMZ instead? Much easier to manage and generally easier to troubleshoot. That's because CSG proxies all the connections and allows you to use a single public IP to load balanced ICA session to your entire farm. You can configure multiple STAs for redundancy too.
yes, that has been considered but was discarded for several reasons.

Hi,

You need to read Chapter 5 of this guide first:

http://support.citrix.com/article/CTX106223

It sounds like the PN client is setup to talk to a CSG server in relay mode. You can confirm this by clicking on the firewalls button in the "Address List" section of a custom ICA connection and seeing if the CSG section in the lower portion is filled in. If this is the case, then you can't use SR (quote from this guide - http://support.citrix.com/article/CTX106975):

4.4.1 How it works
Relay mode is achieved by installing Secure Gateway in Proxy mode with a certificate,
which listens on port 443 for SOCKS+SSL connections. Note that a relay mode gateway
cannot be used as a reverse Web proxy for Web Interface, and Web Interface will not
foster relay-mode connections without a custom change to the launch script or ICA
override files.
4.4.2 Known limitations and issues
• Session reliability does not work through a Relay Mode gateway

hi

thx for reply. i am reading this. maybe it will shed some light onto this matter.
if it cannot be used in this way, why isn't the connection refused?

what i don't understand is why the xte.exe is crashing on only one of the servers.
it runs stable on all the other servers. they are all of the same hardware and have been installed and configured the same way.

Have you confirmed that the PN client is setup to talk to a CSG server in relay mode? I still don't fully understand your environment...

Some people have reported problems with the xte.pid file under "C:\Program Files\Citrix\XTE\logs". You could try stopping the XTE service, copying the xte.pid from another server, then restarting the XTE service. Likely won't accomplish anything but can't hurt. Also, you could try copying the httpd.conf file from another server too.

FYI, you can deploy the PNAgent together with CSG and have a very similar end-user experience to the PN client with far less headaches.

Here's one of my httpd.conf files for reference:

#Citrix_Begin
#Server Root Path
ServerRoot "C:\Program Files\Citrix\XTE"
PidFile logs/xte.pid
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 3



ThreadsPerChild 150
MaxRequestsPerChild 0


ServerName localhost
MultiplexerHandshakeTimeout 100000


# Apache Modules
LoadModule access_module modules/mod_access.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
# XTE Modules
LoadModule socks_module modules/mod_socks.so
LoadModule winevent_log_module modules/mod_winevent_log.so
LoadModule cgp_module modules/mod_cgp.so
LoadModule multiplexer_module modules/mod_multiplexer.so
LoadModule schannel_module modules/mod_schannel.so
LoadModule async_engine_module modules/mod_async_engine.so
LoadModule ticket_module modules/mod_ticket.so


AsyncWorkerThreadCount 0
RequireTicket Off

#CGP Listen Port
Listen 2598

#Max Disconnected Sessions
CgpMaxDisconnectedSessions 100

#Default ICA Local Address and Port Number
CgpDefaultIcaLocalPort 1494

#The length of the CGP cookie in bytes
CgpCookieLength 16

#Allow old clients to request for version 1 CGP cookie
CgpAllowVersionOneCookie On

CgpClientToServerKeepAlive 4000
CgpServerToClientKeepAlive 20000

#CGP Configuration


#CGP Protocol State
CgpProtocol On

#Max TCP Channels Per Session
CgpTcpChannelsPerSession 50

#Disconnected Sessions Timeout (msec)
CgpInterruptedSessionTimeout 180000
CgpHandshakeTimeout 100000
CgpInterruptedSessionsThreadWakeupInterval 60000

Order Allow,Deny
Allow to 127.0.0.1:1494



#Citrix_End

we are using the citrix ssl relay configured for port 443. according to the people managing the firewall, that is the only open port on the firewall.

our users are being granted access to a particular server and are receiving the server's fqdn along with instructions on how to configure a custom ica connection. under the tab "firewalls" they are entering the server's fqdn and the port 443 in the fields for "citrix secure gateway (relay mode)". basically, they are connecting directly to a particular server through a firewall at the place where those servers are located. in the event of a server failure, they have an alternate server (they have a second custom ica connection configured) to connect to.
the reason for this kind of setup lies in the special application being used. if we used citrix secure gateway and loadbalancing, we'd be dealing with other issues and would be annoying the users big time :-)

about the xte.pid file... it has been tried to no avail. we've also come across that while searching for a solution on the web.

i've had another look on the servers and the only difference between them is that the problematic server, which is also the server hosting the datastore, has microsoft framework 3.0 installed.
i am already contemplating to take one weekend and just wipe citrix off of the servers and set up a the farm from scratch. just citrix, not windows. and while i am at it, remove the framework 3.0 from that particular server.

I am seeing this problem on 2 of my servers (out of 45). I'm running 4.5 R02. From the error.log:
[Tue Jun 10 09:28:51 2008] [notice] async engine: Not enough preallocated wsa buffer, allocate 136 wsa buffer
[Tue Jun 10 09:28:57 2008] [notice] async engine: Not enough preallocated wsa buffer, allocate 174 wsa buffer
[Tue Jun 10 09:28:59 2008] [notice] Parent: child process exited with status 3221225477 -- Restarting.
[Tue Jun 10 09:29:06 2008] [notice] Server built: Jan 24 2007 23:31:11
[Tue Jun 10 09:29:06 2008] [notice] Parent: Created child process 32840
[Tue Jun 10 09:29:08 2008] [notice] Child 32840: Child process is running
[Tue Jun 10 09:29:08 2008] [notice] async engine initialized successfully, 8 worker threads started
[Tue Jun 10 09:29:12 2008] [notice] Child 32840: Acquired the start mutex.
[Tue Jun 10 09:29:12 2008] [notice] Child 32840: Starting 150 worker threads.
[Tue Jun 10 09:29:12 2008] [error] A CGP client attempted to reconnect to a CGP session that no longer exists
[Tue Jun 10 09:29:12 2008] [error] A CGP client attempted to reconnect to a CGP session that no longer exists
[Tue Jun 10 09:29:12 2008] [error] A CGP client attempted to reconnect to a CGP session that no longer exists

From the application event log:
Event Type: Error
Event Source: Application Error
Event Category: (100)
Event ID: 1000
Date: 6/10/2008
Time: 9:28:57 AM
User: N/A
Computer: PHOCITRIX09
Description:
Faulting application XTE.exe, version 4.5.0.64631, faulting module mod_cgp.so, version 4.5.0.64631, fault address 0x0000b027.

Anybody have an ideas?

I have been having the same issues. here are my logs and conf file

Tue May 27 09:35:24 2008] [warn] pid file C:/Program Files/Citrix/XTE/logs/xte.pid overwritten -- Unclean shutdown of previous Apache run?
[Tue May 27 09:35:25 2008] [notice] Server built: Jan 24 2007 23:31:11
[Tue May 27 09:35:25 2008] [notice] Parent: Created child process 4896
[Tue May 27 09:35:27 2008] [notice] Child 4896: Child process is running
[Tue May 27 09:35:28 2008] [notice] async engine initialized successfully, 8 worker threads started
[Tue May 27 09:35:28 2008] [notice] Child 4896: Acquired the start mutex.
[Tue May 27 09:35:28 2008] [notice] Child 4896: Starting 150 worker threads.
[Tue Jun 03 19:15:23 2008] [error] An error occurred when processing outgoing CGP downstream data
[Tue Jun 03 19:15:23 2008] [error] An error occurred when processing outgoing CGP downstream data
[Thu Jun 05 09:34:41 2008] [warn] pid file C:/Program Files/Citrix/XTE/logs/xte.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Jun 05 09:34:42 2008] [notice] Server built: Jan 24 2007 23:31:11
[Thu Jun 05 09:34:42 2008] [notice] Parent: Created child process 4896
[Thu Jun 05 09:34:44 2008] [notice] Child 4896: Child process is running
[Thu Jun 05 09:34:44 2008] [notice] async engine initialized successfully, 8 worker threads started
[Thu Jun 05 09:34:44 2008] [notice] Child 4896: Acquired the start mutex.
[Thu Jun 05 09:34:45 2008] [notice] Child 4896: Starting 150 worker threads.
[Thu Jun 05 09:34:45 2008] [error] A CGP client attempted to reconnect to a CGP session that no longer exists
[Tue Jun 10 11:10:06 2008] [warn] pid file C:/Program Files/Citrix/XTE/logs/xte.pid overwritten -- Unclean shutdown of previous Apache run?
[Tue Jun 10 11:10:07 2008] [notice] Server built: Jan 24 2007 23:31:11
[Tue Jun 10 11:10:07 2008] [notice] Parent: Created child process 4804
[Tue Jun 10 11:10:08 2008] [notice] Child 4804: Child process is running
[Tue Jun 10 11:10:08 2008] [notice] async engine initialized successfully, 8 worker threads started
[Tue Jun 10 11:10:08 2008] [notice] Child 4804: Acquired the start mutex.
[Tue Jun 10 11:10:08 2008] [notice] Child 4804: Starting 150 worker threads.
[Tue Jun 10 20:05:18 2008] [notice] async engine: Not enough preallocated wsa buffer, allocate 186 wsa buffer
[Tue Jun 10 20:06:06 2008] [notice] async engine: Not enough preallocated wsa buffer, allocate 404 wsa buffer
[Wed Jun 11 07:37:26 2008] [warn] pid file C:/Program Files/Citrix/XTE/logs/xte.pid overwritten -- Unclean shutdown of previous Apache run?
[Wed Jun 11 07:37:27 2008] [notice] Server built: Jan 24 2007 23:31:11
[Wed Jun 11 07:37:27 2008] [notice] Parent: Created child process 4300
[Wed Jun 11 07:37:28 2008] [notice] Child 4300: Child process is running
[Wed Jun 11 07:37:29 2008] [notice] async engine initialized successfully, 8 worker threads started
[Wed Jun 11 07:37:29 2008] [notice] Child 4300: Acquired the start mutex.
[Wed Jun 11 07:37:29 2008] [notice] Child 4300: Starting 150 worker threads.
[Wed Jun 11 08:04:31 2008] [warn] pid file C:/Program Files/Citrix/XTE/logs/xte.pid overwritten -- Unclean shutdown of previous Apache run?
[Wed Jun 11 08:04:32 2008] [notice] Server built: Jan 24 2007 23:31:11
[Wed Jun 11 08:04:33 2008] [notice] Parent: Created child process 4988
[Wed Jun 11 08:04:34 2008] [notice] Child 4988: Child process is running
[Wed Jun 11 08:04:34 2008] [notice] async engine initialized successfully, 8 worker threads started
[Wed Jun 11 08:04:35 2008] [notice] Child 4988: Acquired the start mutex.
[Wed Jun 11 08:04:35 2008] [notice] Child 4988: Starting 150 worker threads.


#Citrix_Begin
#Server Root Path
ServerRoot "C:\Program Files\Citrix\XTE"
PidFile logs/xte.pid
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 3



ThreadsPerChild 150
MaxRequestsPerChild 0


ServerName localhost
MultiplexerHandshakeTimeout 100000


# Apache Modules
LoadModule access_module modules/mod_access.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
# XTE Modules
LoadModule socks_module modules/mod_socks.so
LoadModule winevent_log_module modules/mod_winevent_log.so
LoadModule cgp_module modules/mod_cgp.so
LoadModule multiplexer_module modules/mod_multiplexer.so
LoadModule schannel_module modules/mod_schannel.so
LoadModule async_engine_module modules/mod_async_engine.so
LoadModule ticket_module modules/mod_ticket.so


AsyncWorkerThreadCount 0
RequireTicket Off

#CGP Listen Port
Listen 2598

#Max Disconnected Sessions
CgpMaxDisconnectedSessions 100

#Default ICA Local Address and Port Number
CgpDefaultIcaLocalPort 1494

#The length of the CGP cookie in bytes
CgpCookieLength 16

#Allow old clients to request for version 1 CGP cookie
CgpAllowVersionOneCookie On

CgpClientToServerKeepAlive 4000
CgpServerToClientKeepAlive 20000

#CGP Configuration


#CGP Protocol State
CgpProtocol On

#Max TCP Channels Per Session
CgpTcpChannelsPerSession 50

#Disconnected Sessions Timeout (msec)
CgpInterruptedSessionTimeout 180000
CgpHandshakeTimeout 100000
CgpInterruptedSessionsThreadWakeupInterval 60000

Order Allow,Deny
Allow to 127.0.0.1:1494



#Citrix_End

Have a look at this CTX forum post for some ideas:

http://support.citrix.com/forums/thread.jspa?forumID=137&threadID=98516

Alan,

Your solution, "you could try copying the httpd.conf file from another server too" worked for me.  Thanks!

Scott