Satyam Vaghani – VMware Storage – Yesterday and Tomorrow
















Yesterday, I attended the Carolina VMware Users Summit in Charlotte.  The morning keynote speaker was Satyam Vaghani, who is really the father of VMFS.  He walked the VMUG attendees through the history of VMFS, and VMware storage as a whole.

In my opinion, this was one of the most valuable sessions at the VMUG, and although my recording is terrible (Notability on iPad), this session deserves to be shared.  I was able to reduce some of the noise in post processing, but this is definitely not broadcast quality.

If you’re really interested in VMware storage, it’s worth a listen, despite the quality.  I honestly could have sat in this keynote for another couple hours absorbing information.


Here’s an outline:

– Birth of VMFS

– In’s and Out’s of Locking

– Optimistic Locking and Performance

– VAAI Intricacies

– The Future (vVOL, VM granular storage, I/O Demux)


Here’s a link to the recording.  I will add slides as soon as I can get them.

Slides are here!!

Update: For those who exhibit lots of “Virtual Insanity” and want to know even more about VMFS, Satyam sent me a link to a white paper he wrote on VMFS. 

Bring on the Socialcast at VMworld 2012




VMworld 2012 is rapidly approaching, and believe it or not, it will be here before we know it!  Call for papers is open now, and you have till May 18th to submit your idea to the VMworld team.  Just in case you missed some of the details, VMworld US is to be hosted in San Francisco August 27-30 and VMworld Europe will be hosted in Barcelona, October 9-11.  It will be an amazing event as always, with some really awesome technology announcements from VMware.  Mark it in your calendar now, socialize the concept to your manager, inform the family, tell the neighbors, do what you need to do but get there!!



This year I wanted to participate in the creation process of the VMworld Labs as I think it’s such a remarkable component of the event.  In 2010 we delivered over 200+ thousand VM’s to customers across 27 different labs, that is an amazing accomplishment.  This years hands on labs will only be bigger, better, and even “more epic”.  I can’t reveal all the details yet but stay tuned, as we have some very exciting things in flight right now.

One of the labs that caught my interest was the Socialcast labs.  I wanted to do something outside my core knowledge base and pickup something that I haven’t had much exposure to.  Socialcast is something I have had a lot of experience with from an end user consumption aspect but nothing on the backend infrastructure perspective.  VMware has a great internal implementation of Socialcast that we have been using excessively for some time now.  I can’t underscore how important Socialcast has become for our company as a place where we can share technical product infromation, idea’s and concepts, presentations, status polls, and basic collaboration. (There is even a Pets of VMware Photo Group)  🙂




“Socialcast software unites people, information and applications across the enterprise in a collaborative community. Help employees focus on meaningful work, share knowledge and discover data in real-time. Behind the firewall or in the cloud, Socialcast enables
secure enterprise collaboration in-context. “ 

Many people reading this blog post understand the power of collaboration and social media.  It is an important component of our being to give back to the larger community to help foster idea’s and innovation.  Socialcast is a framework that allows this collaboration to exist within the confines of your own protected environment. 

I will be working with the Socailcast team over the next several months to design a impactful lab that I am hoping many of you will take.  I have reached out to our CMO, and the VMworld team, to see if we can integrate Socialcast into the website so attendees of the event can actually utilize Socialcast during the convention.  (Idea is still being considered).  I am reaching out to one of my largest customers that has one of the biggest implementations of Socialcast in production to see if they are able to present at VMworld this year as an additional topic.

I’m looking forward to seeing you at VMworld, if your there please sign up for the Socialcast lab and give it a test drive!


VDR Drama – Host Losing Connectivity to vCenter

This is a problem I have seen now in two different environments, at two different companies.  Both happened to be using VMware Data Recovery for backups.


The problem starts like this. You lose a host from vCenter, and you cannot get it to reconnect.  You do a /sbin/ restart, and still you cannot get connected to vCenter.











You CAN connect to the host locally using the vSphere Client.  Let’s look at the logs now.


This particular problem shows up in the host.d log.  To see it, go ahead and SSH into the host and type in: tail -f /var/log/hostd.log  and then go into vCenter and right click on the host to Connect.



Watching the hostd.log, if you see any messages about snapshots during the 5 minutes it takes to time out, here’s how to see if you have this issue.


In your SSH session on the affected host, type in the following:

find /vmfs/volumes/*/* -name *delta*























You’ll see a list of all snapshots for VM’s running on this host.  If you see a VM with a couple hundred snapshots, this is why your host won’t connect to vCenter.  vCenter has a database limitation, and when a VM has more than the number of snapshots vCenter can catalog in the database, the host cannot be managed by vCenter.  I haven’t figured out the exact limit for vCenter.  A VM can have 496, according to this post by William Lam, but I think vCenter breaks before you get to that point.  I had 235 on this suspect one.
To fix this, just connect locally to the host with vSphere Client and Consolidate your snapshots.














Once you’ve consolidated, your directory should look like the following.














Now, you can connect back to vCenter with no problem and no downtime!


Since this is a development environment, we didn’t pay a lot of attention to VDR, and just assumed it was working.  This particular VM happened to be out of hard drive space, so it could not be quiesced, and VDR just kept trying.  The bottom line is, pay attention to VDR errors!!!  After this, we’ll be checking it at least every few days.