Massive Bug Found in VMWare ESX 3.5U2

This was first reported yesterday evening on both the VMWare Community forums and on From what anyone can tell, there is a bug in the VMWare License Management code and it is causing any system that is running ESX 3.5U2 to not be able to boot this morning. VMware is attempting to figure out what happened and put out a patch, but the more important question is, "Why wasn’t this caught before it shipped?" As Matt Marlowe posted:
OK, while we’re all remaining calm….just imagine the implications that bugs like this can occur and get past QA testing….5 years down the road, nearly all server apps worldwide pretty much running in VM’s (pretty easy prediction)……some country decides to initiate cyberwarfare and manages to get a backdoor into whatever is the prevaling hypervisor of the day…..boom. All your VM’s belong to us. […]
I’d love to find out what happened here. Don’t they do any regression testing on new releases to check for date based bugs? I thought that would be pretty obvious.

There have been some updates on this situation since it broke last night:

1. Frank Wegner’s suggested workaround:
* Do nothing
* Turn DRS off
* Avoid VMotion
* Avoid to power off VM’s

I’d council against turning DRS off as that actually deletes resource pool settings….instead, set sensitivity to 5 which should effectively disable it w/ minimal impact.

2. VMware has stated they will have fixes available in 36hrs at the earliest.

3. Anand Mewalal’s suggested workaround:
We used the following workaround to power on the VM’s.
Find the host where a VM is located
run ‘ vmware-cmd -l ‘ to list the vms.
issue the commands:
service ntpd stop
date -s 08/01/2008
vmware-cmd /vmfs/volumes/
service ntpd start

4. It’s reported that there are no easily seen warnings in logs/etc or VC prior to hitting the bug. VC will continue to show the hosts as licensed and no errors will appear in vmkernel log file until you try to start up a new vm, reboot a vm, or reboot the host.

News Source:

Leave a Reply

Your e-mail address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.