I am attempting to pull together a blog post around performance, it’s going to be a four part segment on each I/O component of VMware, CPU, Memory, Storage and Networking. My goal is to try and cover the various tools that you can use to help troubleshoot performance problems that you might experience in your virtual environment.
While I was going through some of the methods, I wanted to illustrate how VMware now includes Windows Performance Counters inside a guest virtual machine to assist with performance monitoring/troubleshooting. I jumped on a test virtual machine I have, and pulled up Windows perfmon. To my dismay the VMware counters are missing! We are currently running VMware vSphere 4.0 update 1 so I checked with a few other people online like Rick Vanover (@RickVanover). It confirmed it seemed to be related to this specific release of vSphere.
I reached out to Scott Drummonds via Twitter (@drummonds), a performance systems engineer who works for VMware, and also opened a service request with support. Scott validated that he saw the same issue and was launching an investigation. Unfortunately the SR didn’t get very far as I was instructed that this was an “experimental feature and was removed from vSphere”. Uhhh ok, I knew that wasn’t right so I waited to hear back from Scott.
Scott has since written a blog post that discusses this issue. It looks like a complete uninstall of the VMware tools on the client followed by a re-install resolves the issue. This does require a reboot for those that are not familiar with this process. The problem appears to be related to mofcomp which it a tool that Microsoft provides and registers WMI information (such as VMware performance counters) with Windows.
Thanks to Scott for jumping on this so quickly and posting a fix to the issue, it’s great to see social media paying off in the real world. Thanks to Rick for helping me figure out what was going on and validating some of my assumptions. Rick has also written up an excellent blog post on this same issue. Hopefully a patch will be rolled into the next minor release of vSphere 4 that will resolve this bug going forward.