i had the same problem 2 years ago when I did what you did. You can either upgrade your licenses to enterprise and use the memory feature or you might be better off buying a 4th server??
Citrix Tips How To Speed Up Citrix Logons
Well for starters, running Citrix on ESX or Xen or HyperV is not the issue so you can stop the talk about going back to metal as it won't change things. There are dozens of us running Citrix on one brand of hypervisor or another in more complex environments with no issues.
Try turning off the phishing for IE7 if its on and check for that blasted Window Search Indexer.
Start with the basic by running perfmon with no users and then load one at at time as see what happens.
How old is your new domain? Is DNS working properly? Are the Domain Controllers running ok? If an application has to wait for long responses from the network it might hold onto memory and not release. I have seen where name resolution latency led to sluggish apps on a server. It's good practice to check your DNS for all your apps and services anyways.