Scripted ESX Installation: Reconfiguring COS Networking with Kickstart

Frequently customers have specific NICs (like onboard NICs) that they’d like assigned to the COS, leaving the other NICs for VM traffic.  This is difficult, however, when using our automated kickstart deployment scripts as there is no way to explicitly define the vmnic assigned to the COS.  And to make matters worse, the VMkernel is not yet available to us during the %post section of the kickstart script, which makes COS networking configuration difficult! Recently I had a customer who was getting frustrated because …

  1. They would “rack and stack” a physical server and wire up their NICs accordingly (i.e. onboard NICs on the management VLAN, remaining NICs on production VLANs)
  2. PXE boot the server
  3. When kickstart completed, they’d lose connection to the COS.

This happens because during installation, ESX just assigns vmnic0 to the lowest PCI number, and then assigns vmnic0 to the COS. And this is often not the NIC the admin wants used for their COS. Of course, they could go back after the fact and reconfigure the COS networking, but this kind of defeats the purpose of a completely hands-free, automated deployment.

Here is one possible solution to the problem.  Below is a script I wrote to append to the %post section of a kickstart file.  Obviously, you’ll need to make modifications for your environment.

## This script should be appended to the %post section of an ESX kickstart file.
## For more info on kickstart and scripted ESX installations, see Appendix B of
## http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_installation_guide.pdf

##
##
Essentially, this is a “script that creates a script.” Because the VMkernel is
## not yet available to us during the %post section of the scripted install, we use
## %post to generate a script called /tmp/post_esx_install.sh that will launch via
## rc.local upon first boot (and only first boot).
##
## The post_esx_install.sh will first make a backup copy of esx.conf and then
## reconfigure the COS networking.  Please see the in-line comments below for
## tweaking post_esx_install.sh for your environment.
##
## If you have any questions, please email aaron [at] sweemer [dot] com.

%post

cat > /tmp/esx_post_install.sh << EOF
#!/bin/bash
cp /etc/vmware/esx.conf /etc/vmware/esx.conf.backup
/usr/sbin/esxcfg-vswitch -U vmnic0 vSwitch0
/usr/sbin/esxcfg-vswif -d vswif0

## If your kickstart file has vmportgroup=1, you *might* want to uncomment the
## next line

## /usr/sbin/esxcfg-vswitch -D “VM Network”

/usr/sbin/esxcfg-vswitch -A “VMkernel” vSwitch0

## You’ll need to find which physical NICs you want assigned to your COS.  From
## the command line of an already installed ESX server, execute
## “/usr/sbin/esxcfg-nics -l” as root and look for something unique about the
## NICs.  For example, this could be the word “Broadcom” or it could be the
## actual PCI number.  In the next line, replace “search term” with this
## text.

/usr/sbin/esxcfg-nics -l | awk ‘\$0 ~ /search term/ {print \$1}’ | xargs –n 1 /usr/sbin/esxcfg-vswitch vSwitch0 –L

## Note: if you want to test the line above from the command-line, you’ll need
## to remove the leading “\” in front of $0 and $1. The \’s need to be here so
## the esx_post_install.sh script gets properly written by kickstart. But when
## executing directly on a command line, the \’s need to be removed.

## Replace the x.x.x.x after -i with the IP address and after -n with the
## subnet mask for your COS.

/usr/sbin/esxcfg-vswif -a vswif0 -p “Service Console” -i x.x.x.x  -n x.x.x.x

## Replace the x.x.x.x after -i with the IP address and after -n with the subnet
## mask for your VMkernel port group.

/usr/sbin/esxcfg-vmknic -a -i x.x.x.x -n x.x.x.x VMkernel

## Replace x.x.x.x with the default gateway for the COS in both of the next two lines.
route add default gw x.x.x.x
echo “GATEWAY=x.x.x.x” >> /etc/sysconfig/network

mv /etc/rc.d/rc.local.save /etc/rc.d/rc.local
EOF

chmod +x /tmp/esx_post_install.sh
cp /etc/rc.d/rc.local /etc/rc.d/rc.local.save

cat >> /etc/rc.d/rc.local << EOF
cd /tmp/
/tmp/esx_post_install.sh
EOF

As an example, in my environment I have server with 4 NICs and by default, ESX assigns vmnic0, which is mapped to PCI 02:00.00, to the service console. However, what is actually physically wired to my management network is vmnic3, which is mapped to PCI 02:03.00.  In the script above, I simply searched for the number 3 (i.e. replaced search term with 3) and now my scripted ESX installation works properly.

Below is the configuration of my server before I redeployed with kickstart.  The line in red is the NIC I want assigned to the COS.  The lines in black are what ESX assigns the COS by default.

BEFORE (without %post section)


[root@vesx7 root]# esxcfg-nics -l
Name    PCI      Driver      Link Speed    Duplex MTU    Description
vmnic1  02:01.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM

vmnic2  02:02.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM
vmnic3  02:03.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM
vmnic0  02:00.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM


[root@vesx7 root]# esxcfg-vswitch -l
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks

vSwitch0       64          4           64                1500    vmnic0

PortGroup Name      VLAN ID  Used Ports  Uplinks
VM Network          0        0           vmnic0

Service Console     0        1           vmnic0

Now, here is the same output after I redeployed the server with my modifications to the %post section of the kickstart file. The scripted deployment of ESX now properly assigns vmnic3 to my service console.

AFTER (with %post section)

[root@vesx7 root]# esxcfg-nics -l
Name    PCI      Driver      Link Speed    Duplex MTU    Description
vmnic1  02:01.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM
vmnic2  02:02.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM
vmnic0  02:00.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM
vmnic3  02:03.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 82545EM

[root@vesx7 root]# esxcfg-vswitch -l
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks

vSwitch0       64          5           64                1500    vmnic3

PortGroup Name      VLAN ID  Used Ports  Uplinks
Production          0        0           vmnic3

Service Console     0        1           vmnic3

I hope this was helpful.  Let me know if you have any questions.

Well, I’d better sign off and start packing because I leave for Omaha, NE in a few hours.



5 comments
  1. Good Post, handy to know.

    We tend to assign the COS to the first onboard NIC so it usually works out OK for us, however the same search term script could be used to customise your network whatever way you want.

    one to add to the script library, thanks

  2. Good Post, handy to know.

    We tend to assign the COS to the first onboard NIC so it usually works out OK for us, however the same search term script could be used to customise your network whatever way you want.

    one to add to the script library, thanks

Comments are closed.