Word on the street is that VMware Infrastructure 3.5 Update 2 and ESXi 3.5 Update 2 have been affected by a licensing bug left over from development. This bug renders any powered off machine incapable of being turned back on, suspended machines incapable of being resumed, and admins unable to use VMotion to migrate machines, only reporting a "general error." Searches through the logs unearth this as the real problem:
By submitting your personal information, you agree that TechTarget and its partners may contact you regarding relevant content, products and special offers.
Aug 12 10:40:10.792: vmx| http://msg.License.product.expired This product has expired.
Aug 12 10:40:10.792: vmx| Be sure that your host machine's date and time are set correctly.
Aug 12 10:40:10.792: vmx| There is a more recent version available at the VMware Web site: "https://www.vmware.com/info?id=4".
The workaround involves disabling NTP and setting your clock back to August 10, and can be found in full detail here.
According the KB article at VMware.com, engineering "will reissue the various upgrade media including the ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade tar and zip files by noon, PST on August 13. These will be available from the page: https://www.vmware.com/download/vi. Until then, VMware advises against upgrading to ESX/ESXi 3.5 Update 2."
Let's hope that it actually is fixed in that amount of time, otherwise people will be resetting their VI 3.5 servers' clocks as nightly maintenance.
Update 2 was released on July 28th, and I can't begin to guess how many organizations this has affected. Update 2 contained many important and popular feature adds, like Windows Server 2008 Support, enhanced VMotion compatibility, live cloning of virtual machines, hot virtual disk extensions, 10GbE NFS and ISCSI support, and much more. All these features may have led admins to deploy the update a little earlier than normal (I'm speaking to the wait-and-see folks like myself).
The people who truly utilize all the functionality of virtualization are probably the ones who are losing the most here. Imagine having your systems dynamically reallocate virtualized hardware resources based on the time of day or day of week. Maybe the weekends are spent running batch jobs on 80% of your servers while during the week it's just 20%. In that case, those servers fired up Friday night and shut down early Monday morning just like normal. But, with the licensing bug, the servers required to get through the week (a full 60% of the servers!) can't come back up.
VMware really messed up here, but they don't need me to tell them that--I'm pretty sure the ringers in their phones wore out a few hours ago. It will be interesting to see how this pans out and how people using Update 2 in production were affected. It's sure to be a humbling experience for VMware, and how they handle it should speak volumes about the state of company and the responsibility it has to its customers.