Single Throat to Choke

It seems fashionable these days to use the phrase “single throat to choke” with regard to vendors, and integrated solutions.  Sadly, this seems to be used as a selling point by the big guys who are interested in bundling their offerings together.  Other than IT guys who like to repeat buzzwords to make themselves feel important, this phrase is rarely used in mainstream business.  The reason is clear.

As a customer, I can tell you that I don’t want a throat to choke.  Today’s successful business doesn’t need a throat.  What it needs is a partner.  Don’t believe me?  Take a look at BMW.  For those who have had the pleasure of touring the plant in South Carolina, there is one thing that stands out beyond the fascinating machinery, and beautiful plant.  What stands out is the way that BMW partners with their suppliers.

At each manufacturing station, there’s also a loading dock door.  Vendors back their trucks right up to the proper door with their part, and unload just enough to keep the line going until the next truck arrives.  What this means is that BMW’s suppliers aren’t “throats to choke”, but partners.  These partners enable BMW to manufacture more efficiently, cutting out non-value add processes like moving parts around giant warehouses.

If one of their partners fails to deliver, the entire plant can grind to an abrupt halt.  And when this happens, BMW might want to choke a throat, but in reality, the vendor is choking their own throat.  Just like SLA’s in our world, there are heavy fines for these kinds of mishaps in that world.  The end result is the vendor has a vested, ongoing interest in satisfying the customer.  When things go wrong, both of them take a hit, because they are partners.

We’re not manufacturing in IT, but we are delivering a product to a customer.  The more partners we have, the better.  Do we really care that we can call one phone number for a storage, network, hardware, or virtualization platform problem?  I would submit to you that we do not.  We care more that our partners can listen, and deliver what we need to be successful.  We want it delivered on time, and we want it to enhance our company, and our profitability.

While I see merit in the vBlock, SMT, and other integrated solutions, I believe the “throat to choke” argument cheapens these offerings. The vendors who are able to pitch the strengths of their products, and how they can strengthen our environments, will win out over those who focus on choking people.  The argument seems to me to be more of the fear, uncertainty, and doubt that needs to disappear from the industry in order for it to advance.



Brandon Riley is a new blogger here at Virtual Insanity as this is his second post.  He is currently working as a Sr. Distributed Systems Engineer with The Clearing House.



Become a Virtualization Linchpin



A fellow VMware Engineer recommended a book to me recently titled “Linchpin” by Seth Godin.  The book has nothing to do with VMware or virtualization but it hits home for me because it highlights a lot of topics that I find applicable to our industry.  I could not ignore this as a relevant force that has somehow affected me, so i felt I had to write something up and share some thoughts.  This post is a little more off paced from what I normally write about so  bare with me.  I think a lot of what Godin covers is present in the VMware community today, and many of you are already “Linchpins”.  I reached out to Seth to get his permission to share some of his insights, if your interested in purchasing the book just click the link above.  Here is the book synopsis:

There used to be two teams in every workplace: management and labor. Now there’s a third team, the linchpins. These people invent, lead (regardless of title), connect others, make things happen, and create order out of chaos. They figure out what to do when there’s no rule book. They delight and challenge their customers and peers. They love their work, pour their best selves into it, and turn each day into a kind of art.

Linchpins are the essential building blocks of great organizations. Like the small piece of hardware that keeps a wheel from falling off its axle, they may not be famous but they’re indispensable. And in today’s world, they get the best jobs and the most freedom. Have you ever found a shortcut that others missed? Seen a new way to resolve a conflict? Made a connection with someone others couldn’t reach? Even once? Then you have what it takes to become indispensable, by overcoming the resistance that holds people back.

Godin believes that our society has changed and we (the U.S. in this example) are no longer living in the industrial era that our parents and grandparents grew up in.  We are no longer the factory-driven-widget-producing  society that we once were, in fact most of these types of positions have been outsourced to cheap labor across the globe.  Going through the schooling process and obtaining a piece of paper no longer guarantees you will be promised a job for the next 30 years of your life.  Competition and technology have extinguished the promise of a secure job that pays well, offers health insurance, and a great retirement package when you exit. 


Sound melodramatic and doom and gloom?  It’s not really.  Godin goes on to explain that because our society is changing, we need to also identify this and change with it.  We are not cogs in a giant industrial machine.  You have a mind of your own, and have more to contribute that you might think.  Working off the same rule book is no longer going to apply if you want to be considered indispensible by forward thinking companies.

The old school of thought:  "”Keep your head down, follow instructions, show up on time, work hard, suck it up”.

The new school of thought:  “Be remarkable, be generous, create art, make judgment calls, connect people and ideas”

Become a VMware Linchpin

Here is how Wikipedia defines Art:

Art is the process or product of deliberately arranging elements in a way to affect the senses or emotions. It encompasses a diverse range of human activities, creations, and modes of expression, including music, literature, film, photography, sculpture, and paintings. The meaning of art is explored in a branch of philosophy known as aesthetics.

The new school of thought talks about becoming an artist, but don’t think of art as the class you avoided in high school.  Art is creating something from nothing, it’s also about creating something that invokes an emotional area in the brain for yourself and others.  Many virtualization evangelists are creating something from nothing, the VMware blogosphere is one of the best examples of this today.  The VMware community is alive with passionate people that are writing and creating new content daily.  Have you ever stopped to examine the VMware Planet v12n blog aggregator?  It’s really quite amazing the amount of new content created around this topic of virtualization.  Customers stepping up, and brining their content to local VMUG’s to share their personal experiences is another great example of creating this type of art. 

Twitter is now inundated with VMware virtualization metadata.  Not only can you find where this virtualization data resides but you can now make  connections with people that would have been impossible to make before.  There are experts in every form and fashion that are now open to communicating about all things that touch virtualization.  Storage experts, systems experts, networking experts, powershell experts, and perl experts are just a few that jump out.  Are you looking for a specific need that might have a benefit to others around you?  Pose the question and 9 times out of 10 someone will write the code and share it with the community at large.

Challenge Yourself

Maybe this is something you are already doing today, maybe it’s something your not doing and will never do.  That’s fine too, I’m just some guy that read a book and sharing my two cents.  I will tell you that as you start to consider some of these topics and look out at the industry in general (not just VMware) you will see this change come into play more and more.

Challenge yourself to get out of your comfort zone.  Go out of your way to make new connections.  Help someone out that might not be as skilled as you.  Write a blog.  Sign up for a Twitter account.  Stand out, create art, be noticed.  It will give you a sense of accomplishment, help define yourself as an expert in your field, and even open more opportunities down the road.


Two Garbage Cans are Better Than One – .NET

The vTrooper Report

I was asked a question about a specific use case where a second vCPU should be added to a VM in a Virtualized environment. Generally its an easy answer;

If the server can execute multiple threads and really uses the second vCPU for that other tread

then it’s probably OK to add the second vCPU to the VM

Now adding a vCPU in a server to make it SMP oriented is an elementary task in VMware, but has a few impacts:

  • It will change your metrics for reporting
  • It changes the HA slot size for your failover needs
  • It will modify your consolidation ratio per core and indirectly per socket affecting your Capacity Planning plans
  • It will make you redeploy your Ubuntu or Linux server that you forgot to compile with an SMP kernel. (Not to be taken lightly or your server won’t boot)

I was exploring the use case and impacts when a bit of information popped up:

Garbage collection on .NET applications will require a second vCPU to perform in ‘Server Mode’ versus ‘Workstation Mode’

Explaination from MSDN:

Managed code applications that use the server API receive significant benefits from using the server-optimized garbage collector (GC) instead of the default workstation GC.

Workstation is the default GC mode and the only one available on single-processor computers. Workstation GC is hosted in console and Windows Forms applications. It performs full (generation 2) collections concurrently with the running program, thereby minimizing latency. This mode is useful for client applications, where perceived performance is usually more important than raw throughput.

The server GC is available only on multiprocessor computers. It creates a separate managed heap and thread for each processor and performs collections in parallel. During collection, all managed threads are paused (threads running native code are paused only when the native call returns). In this way, the server GC mode maximizes throughput (the number of requests per second) and improves performance as the number of processors increases. Performance especially shines on computers with four or more processors.

This caught me by surprise and makes me think;  for every disk of VM’s around the world which are out of whack (mis-aligned) , there are an equal number of .NET app servers that have been virtualized with P2V tools across the globe that are starved for the correct garbage collection mechanism….


All would Perish

Hindenburg and .NET

Whoa!  I gotta settle down!


Now you are going to ask where the special override switch or Regedit value would be used to fix it.  The answer is even more easy.    There isn’t one.  .NET sees one vCPU or more and decides for the app.  You cannot override it.  You can add 4 vCPU’s to improve its performance but not turn it off.

This probably explains a few of the things that have already happened or will happen in your app development world:

  • The .NET development on dual core workstations is working fine and when you move the application to a single core VM the development process hits a hiccup in performance while GC runs.
  • VM admins who have been adverse to the second vCPU that was idle now have a reason to deploy a second vCPU but won’t like it.
  • It drives a reason to migrate to vSphere sooner than later due to the relaxed CPU scheduler that was introduced in 4.0
  • Additional vCPU’s will drive more ‘Eggs’ into your baskets – Do Not Panic

As all the worlds workloads increase its only inevitable that the number of vCPU’s would increase as well.  The push for 64-bit systems with over 4GB of RAM are driving up the size of the VM in most farms as seen by the new maxims in the VMware vSphere release.  Just remember that you can look for vCPU contention and NUMA pressure in the ESXTOP values.

With a little help from my friends

Welcome Brandon Riley

Sr. Distributed Systems Engineer

Throughout the vBlogosphere these days, a common refrain is often heard. vProphets are shouting from their vRooftops that we IT people need to radically diversify our skill sets if we hope to survive in the age of the cloud. No longer will there be “server guys”, “network guys”, and “storage guys”. It is said that in the virtual world, we need to be all of the above, and more. Anyone who has dipped so much as a toe into the virtualization pool understands this well.

Unfortunately, many companies still don’t get it. VCE coalition members are saying that some executives are deciding to buy vBlocks simply to facilitate the transition away from the isolated silos which still Balkanize IT. When executives make purchase decisions based on how they will affect corporate IT culture, there is clearly a problem.

While we sit and wait for the old guard to catch on, some vendors have brought tools to market that push us a little further down the path of convergence. Storage vendors are doing a fantastic job integrating tools into vCenter and making our lives easier. Companies like Veeam and Vizioncore are showcasing tools that give us much more insight into the virtual environment and help us shoulder this increased responsibility. Server vendors are releasing great management tools for blade servers that help us deal with that layer of abstraction. Even Microsoft is trying to help with SCVMM and other tools in the pipeline.

Notice anything missing from the list?

When it comes to making this tectonic IT shift more palatable, the networking players aren’t holding up their end of the bargain. Cisco has done a brilliant job with UCS, and UCS Manager. No sane person can argue that those products don’t vastly improve management of the server hardware layer. The 1000v is also a remarkable tool, and I know they have heavily invested in the VMware relationship. But are they doing anything to make networking more accessible to the new age of vGeneralists?

As a CCNA, I can tell you from my experience that this bare minimum level of Cisco knowledge just scratches the surface of Cisco networking, and is not enough to get a decent sized VMware environment running well. As I write this, there are many volumes of Cisco material in my bookshelf that I can call upon, but I still don’t feel even remotely confident configuring Etherchannel, PVLANs, VPC, TRILL, and all the other administrative jujuitsu an optimized virtual environment requires. The “network guys” at my company each have ten thousand or more pages of Cisco reference on each of their desks, and I am sure they do not feel a high degree of proficiency with these features.

Does the market expect the new generation of cloud architects to also be CCIE certified? I hope not. As I pointed out previously, in most of the other areas where we are expected to be competent these days, we are getting some help. There are even plenty of tools out there to help facilitate all the PowerShell and scripting we need to perform. But as the number one networking player, Cisco does not seem interested in making networking more accessible to already overburdened, hyper-qualified virtualization experts.

Don’t get me wrong. I am not looking for sympathy, and I don’t mean for this to come off as a Cisco bashing session. But unless we are able to increase our efficiency at the same rate we are increasing and diversifying our skill sets, this transition will be as tough on us as it will be for the old IT guard who are refusing to leave their silos. UCS Manager is a huge step in the right direction for Cisco. If they can bring that type of functionality to the network stack, they will only sell more products. Will they apply the same optimized method of management to the rest of their line? Or will they continue on their current track just to sell more books and bill more consulting hours for their partners?

VMware wants to hear from you about the vSphere Java API

If you have read Easy vSphere Web Apps with Grails and the VI Java API or Easy VMware Development with VI Java API and Groovy then you know I’m a big fan of Steve Jin’s vSphere Java API.   And I recently received the following email …


We are running a survey to better understand the needs of the greater community using the vSphere / VI Java API. I am referring to

The survey takes less than 5 minutes and it would help us understand what you need from VMware.  Does your organization require formal support, indemnification, training or does the current model work for you ?

This survey will be confidential; we will not disclose your name or who you work for. We just want to understand how you are using the vSphere / VI Java API and what you need from VMware.

Link to survey:

Thanks for your time and hope to see you at our Developer Days @ VMworld 2010

Pablo  Roesch

vSphere SDK Product Marketing

So if you’re a developer using the vSphere Java API – or if you would like to use it but can’t for some reason (e.g. lack of official support) – then I would ask that you take the survey.  I’ve already taken the survey, and I can assure you it’s quick and painless.  Thanks for your help!

Six P’s and None the Richer

Welcome Kelly Culwell!

Solution Architect, InterWorks

Kelly0070 (Small)

A few years ago, I attended a VMware classroom training session (I think it was VI 3 ICM) and learned a lot.  Unfortunately, I don’t remember the name of the instructor, but he included a bit of wisdom that I will carry with me always, and that is the six Ps … P’s … Pees … what have you.  Prior Proper Planning Prevents Poor Performance.  In some cases, the final two become Pissed-off People.  While this principle is applicable to every aspect of life, not just IT, it becomes crucial when planning the implementation of a virtualized environment.

I’ve talked about this concept many times before, and I think some of my co-workers are repeating it now as well.  You wouldn’t try to build a house without considering the foundation, the frame, the wiring, and the roof, so why would your infrastructure or the core of your business be any different?  I don’t know how many times I’ve looked at a new client or some other environment and immediately seen the effects of over-provisioning, or improper provisioning, of their resources.  Now don’t get me wrong–I am by no means claiming to be an expert on anything, but I’m good at some things, and I relate a lot of my experiences to my day-to-day grind.  The purpose of this post is simply to make people think.

Continue reading

Give Me the Whole Core and Nothing but the Whole Core

Greetings Virtual Insanity readers!

My name is Jeff and I am Sr. Systems Engineer / Solutions Specialist at VMware. I specialize in virtualizing messaging systems (Microsoft Exchange, Zimbra), databases (Microsoft SQL), ESX Performance, and Security. My posts are my thoughts at the time I write them and do not reflect those of my employer. I reserve the right to change my opinion as technology evolves and I learn about new ways of getting IT done. And please test any settings described in our posts in a test environment to see if how they will impact your production environment.

I recently had a conversation with a colleague of mine regarding hyper-threading which led to this post.

Here is the question: VMware’s Performance Best Practices for vSphere whitepaper recommends enabling hyper-threading in the BIOS (see page 15), but the application vendor recommends disabling hyper-threading, can I enable hyper-threading for the ESX host while disabling hyper-threading for a specific virtual machine? Yes.

The conversation centered around Microsoft’s recommendation to disable hyper-threading for production Exchange installations and configuring a virtual machine to accommodate this recommendation.

TechNet, “Hyper-threading causes capacity planning and monitoring challenges, and as a result, the expected gain in CPU overhead is likely not justified. Hyper-threading should be disabled by default for production Exchange servers and only enabled if absolutely necessary as a temporary measure to increase CPU capacity until additional hardware can be obtained.”

So, how do I go about disabling hyper threading for a specific virtual machine while leaving the option enabled in the BIOS of the ESX host?

Continue reading

Understanding ESX/ESXi Equivalency…Are we there yet?

So I am often asked the question “how does this work in ESXi?” from my customers, especially around HW or management agents equivalency. So I hope to make the answer a bit more clear for everyone…

It is public knowledge that ESX is evolving to a pure ESXi model in the future release cycles of the product, though exact timelines are still under NDA. Convergence to a “console-less” ESX provides a number of benefits to our customers, with which many of you are, by now, well acquainted . It reduces the overall footprint that requires patching (see below) as well as removes the dependency on the vestigial RHEL-based Console Operating System, and sets the stage for future enhancements and technologies yet to be introduced. (Those who are under NDA might know to what I am referring!) 🙂

Continue reading

You got somethin’ to say?!


You may have noticed over the past few years there have been a number of authors here at Virtual Insanity.  Some have contributed only once or twice.  Others contribute (or plan to contribute) on a regular basis.  In fact, there are now seven authors who call Virtual Insanity home (ordered by number of posts) …

  • Aaron Sweemer (me) – Senior Systems Engineer at VMware
  • Scott Sauer – Systems Engineer at VMware
  • Rick Westrate – Director of Cloud Services at Eastern Computer
  • John Blessing – Technical Consultant at EMC
  • Chris Everett – Senior Systems Engineer at VMware
  • Thomas Mackay (new author!) – Staff Systems Engineer at VMware
  • Jeff Szastak (new author!) – Senior Systems Engineer and Tier 1 Applications Specialist at VMware

I was having a conversation yesterday with a few of my fellow co-authors about how we could improve the blog, make it more valuable for our readers, and make it a powerful and highly recognizable industry resource.  We set some pretty aggressive goals, but to achieve them, we’ll need to produce quality content across a broad range of topics on much more frequent basis.  Unfortunately, the main obstacle each of us face is simply the lack of time to sit down and write meaningful blog posts with any sort of regular frequency. Continue reading