LILO is the only supported boot loader for ESX, so don't replace it with any other Linux loader, e.g. GRUB.

The lilo.conf file is the configuration text file that defines how the Linux OS will boot. If you are familiar with Windows, then this file is similar to BOOT.INI. However, in contrast to the Windows file, the lilo.conf text file is compiled into a binary file, and it is that binary file which is actually used by LILO at boot time.

Here is a sample section of a lilo.conf file. You can see the initrd line which specifies the ramdisk image that the boot loader uses to load the Linux service console kernel. The Linux kernel image name is vmnix and many VMware administrators use the term vmnix when referring to the service console.

image=/boot/vmlinuz-2.4.9-vmnix2
     label=esx
     root=/dev/sda2
     initrd=/boot/initrd-2.4.9-vmnix2.img
     read-only
     append="mem=272M cpci=0:*;1:*;2:*;4:*;12:;16:*;"

  If you are troubleshooting the APPEND line, then use vmkpcidivy tool. You should not have to revert to manually editing this file. If you ever do edit this file, then you need to write those changes into the boot sector by running /sbin/lilo . If you are unsure the right changes will be made, you can do a trial run with the command /sbin/lilo -t .

The pci device mask specified in the append line of lilo.conf is actually an include, not a mask out. The important thing to note is that the append line defines the physical PCI bus hardware that is visible to the service console.

ESX manages allocation of PCI devices between service console and VMkernel with the expectation of the boot loader being LILO.

You can also view PCI device allocation using the MUI, whilst logged in as root. This is found under Startup Options in the Options tab of the MUI as shown below.

Alternatively, you could use the legacy MUI web interface using the URL

http://esxserver/pcidivy

Another alternative is to used the command vmkchdev -L.

The LILO boot loader has a boot prompt as well, displayed rather inconspicuously below the red text menu. It is at this boot prompt that you can supply additional boot parameters. You may wish to restrict LILO from accepting such user-entered boot parameters unless a password is entered.

password=<password>
restricted

If you only enter the password line to the file, then a password would be required to boot the system, if you also have the restricted option then you would only need the password for making boot modifications. In the LILO boot menu, any option that requires a password has a "P" next to the image name and any option with the restricted option has an "R" next to the image name.

If it is a concern that the /etc/lilo.conf file contains a password stored in clear text, the file should be secured using permissions that only allow root access, i.e. rwx------. You can implement this with the chmod command and the 600 numeric to represent rw.

id:3:initdefault:

The run level that the service console uses is run level 3, which specifies full multi-user mode. The init process then works through the start up scripts in the appropriate directory. For run level 3, this directory would be 

/etc/rc.d/rc3.d

The file also starts up the virtual terminals on the service console, mingetty tty2 through mingetty tty5.

The mingetty process is a manager of virtual terminals for Linux; it is a minimal version of universal getty found in UNIX. It does not support to connections of serial port connected terminals and is therefore "lighter" than getty and performs the majority of most terminal needs. In the past, when UNIX was deployed on large machines and dumb terminals were connected using serial connections, the getty service was used. Nowadays, almost nobody connects to a Linux machine by the serial port, and for that reason it was decided to lighten getty, adopting a "minimum getty" in many distributions of Linux.

This may be required when a server has only 2 physical NICs, but we really want 3, so we can dedicate 1 NIC to VMotion. To do this we add the following lines to the end of the rc.local file.

insmod vmxnet_console devName=vmnic0
ifup eth0

You can use the insmod utility to load driver modules either by explicitly stating the path and module file or by just the module name and insmod will locate the correct one. In the example above, the actual driver file is

/lib/modules/2.4.9-vmnix2/misc/vmxnet_console.o

If we do need to do this, then we need to decide which network functions should share a physical NIC (pNIC), for example

So, how you share your pNICs will depend on how much management traffic there is in relation to VM traffic as well as how often VMotion operations are likely to occur.

If you need to VLAN tag the service console traffic when using the vmxnet_console module, then you just add the VLAN ID number after the device name in rc.local. For example, to place the service console on VLAN number 105, we would modify the insmod line to read

insmod vmxnet_console devName=vmnic0.105

S00vmkstart
S10network
S11vmware
S12syslog
S55sshd
S56xinetd
S91httpd.vmware

By looking at the script titles we can guess what some of them do, e.g. S55 starts the secure shell daemon (putty in now!), S56 starts xinetd which amongst other things handles remote console sessions and then S91 starts, which gives us an Apache web server, known to us as simply as the MUI. If you would like to add your own scripts, you can place them anywhere in this start-up order. For example, if you wanted a script to start after xinetd but before the MUI, you could label it something like "S60custom".

A neat trick if you are looking to temporarily disable a start up script is to rename the file from capital "S" to lowercase "s".

ntpd   0:off 1:off 2:on  3:on  4:on  5:off 6:off
syslog 0:off 1:off 2:on  3:on  4:on  5:off 6:off
snmpd  0:off 1:off 2:off 3:off 4:off 5:off 6:off

If we wanted to change a service so that it is enabled for a particular run level, then we can use chkconfig –level.

chkconfig --level 1 ntpd on

The above command would turn on ntpd for run level 1, this would not affect the run levels that ntpd was already set for. So in this example, the ntpd run levels would be

ntpd 0:off 1:on 2:on 3:on 4:on 5:off 6:off

If we just want to turn on a daemon for the current run level we can just type the name of the service we want to enable/disable with on or off as a parameter. So to turn on nfs daemon for the current run level (whatever that may be) you would type:

chkconfig nfs on

If you are not sure what runlevel you are currently in, just use the command runlevel and the current runlevel will be displayed.

service –-status-all |grep running

would produce an output similar to the following:

crond (pid 1423) is running
httpd (pid 1486 1482 1479) is running
syslogd (pid 1136) is running
sshd (pid 1208) is running

  To avoid unnecessarily rebooting an ESX server after making certain configuration changes, we can frequently just restart the appropriate daemon. For example we could restart the Apache web server for the MUI with the command:

service httpd.vmware restart

and we can also check a named service running status with

service httpd.vmware status

The /etc/rc3.d/S12syslogd file is actually a logical link to the executable file in /etc/rc.d/init.d/syslog

logger -i -t username "This test message will appear in the service console log file!"

So now we could examine the last few lines of the service console log file to see our new entry:

tail /var/log/messages

If you were setting up logging from the service console of one ESX server to a centralised log server, then this would be a great way of testing that the centralised logging was working as expected.

A quick way to restart the SSH daemon is to enter:

# /etc/init.d/sshd restart

It is important to use the full path to the ssh daemon to do this. An easier way to do this is by using the service command

service sshd restart

The configuration of the SSH server daemon is stored in the text file /etc/ssh/sshd_config. An important setting in this file is PermitRootLogin=Yes/No. You can quickly check this with a grep on the file.

# grep Permit /etc/ssh/sshd_config

If you do edit the file, make sure you restart the service for the changes to take effect.
 

We can use it to gain a command line session with a remote host, typically the service console of another ESX Server. In the following example, we are logged on to the service console of ESX server "esx01" and we are opening a command line session with the service console of ESX server "esx02".

[root@esx01 root]# ssh esx02
The authenticity of host 'esx02 (192.168.22.32)' can't be established.
RSA key fingerprint is b0:d3:5f:87:65:6d:dd:29:be:49:e2:b5:1a:8e:db:37.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'esx02,192.168.22.32' (RSA) to the list of known hosts.
root@esx02's password:
Last login: Mon Apr 17 13:25:05 2006 from 172.16.110.204
[root@esx02 root]# exit
logout
[root@esx01 root]#

Once you have established an ssh session with another host, the known_hosts file on your server is populated.

The .ssh subdirectory is not created until you make an outbound ssh or scp connection to another host.

If you rebuild one of your ESX hosts, when you try to reconnect to it over ssh you may be prevented from connecting, if the known_hosts file has cached the old key. In the following command, we examine the contents of the known_hosts file (we've truncated the length of the key here!)

[root@esx1 root]# cat .ssh/known_hosts

esx02,192.168.22.32 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAocui7IApxnJevQgIPyIynde0SvVHRS02CM7ODFF7Mc/d <snip>

The -t switch specifies type

ssh-keygen -t dsa

Originally the inetd daemon helped in controlling network connections to a computer. When a request arrives at a TCP/UDP port that is  managed by inetd, the request is forwarded to a program called tcpd (/usr/sbin/tcpd). Then tcpd decides, in accordance with the rules contained in the hosts.{allow, deny} files whether or not to grant the request. If the request is allowed, then the the corresponding server process (e.g. ftp) can be started. This mechanism is also referred to as tcp_wrapper.

xinetd provides access control capabilities similar to the ones provided by tcp_wrapper.

The daemon itself is stored in /usr/sbin/xinetd This launches the daemons that are bound to it on demand. 

/etc/xinetd.d/vmware-authd

This text file contains the settings for the VMware remote access authentication daemon. This file specifies the TCP:902 port used by remote console.

   If this port was changed here, it must also be changed in the file /etc/vmware/config. Any changes must also be reflected in the remote console client settings and VirtualCenter.

If we wanted to add Kerberos off-box authentication for MUI access, then its in the pluggable authentication module configuration file that corresponds to this daemon that we would make a change. This file is found at

/etc/pam.d/vmware-authd

We would need to change the current "auth required" to "auth sufficient" and add a last line of "auth required" using the Kerberos authentication module. Modification may be required to the /etc/krb5.conf, /var/kerberos/krb5kdc/kdc.conf for server locations and /etc/hosts to resolve these server IP addresses.

vmnix driver   Loaded by modprobe vmnixmod.o
VMkernel       Loaded by vmkloader
Logger        
VMkdump        Any dump will be copied to /root
Starts VMs     Performed by vmware-serverd

This S11vmware file is actually a logical link file to the actual script which is stored in the file /etc/rc.d/init.d/vmware

If the automatic install of this component fails, it can be installed manually by copying the appropriate RPM package from the VirtualCenter server to the ESX host which is to be VC-managed.

Copy the RPM from C:\Program Files\VMware\VirtualCenter\ccagent\

to the ESX host and then from the command line run

rpm -Uav VMware-ccagent-esx-2.5.0

The most likely reason you would need to do this manual method is when the VC server is on a separate subnet from the ESX host and there is a firewall in-between. Even if TCP:902 is open between the subnets, some dynamic ports are temporarily required for this vmware-ccagent install.

If you are running ESX Server version 2.5.2 with VirtualCenter 1.3, you will no longer see the process vmware-ccagent. The original process name vmware-serverd remains even after adding the ESX host to a VirtualCenter farm.

If you are running ESX Server version 3 with VirtualCenter 2 (not released yet!) then you'll see something completely different.

/usr/lib/vmware-mui/apache/conf/httpd.conf

This process communicates with vmware-serverd for backend data. Remember a refresh in the browser is only a refresh to Apache, to get new data, click on the refresh button to get new kernel data. Remember if the httpd.vmware service starts and then stops immediately, check your service console disk space.

The S91httpd.vmware entry in /etc/rc3.d is a logical link to /etc/rc.d/init.d/httpd.vmware
 

The HTML files for the MUI can be found in the following path

/usr/lib/vmware-mui/apache/htdocs/vmware/en

The root of the VMware MIB is enterprises 6876

The Master SNMP agent (snmpd) can be replaced with the HP Insight Agent or Dell OpenManage as required.

The MIBs are stored on an ESX Server in the directory /usr/lib/vmware/snmp/mibs

# snmpsetup.sh default

This would produce an output similar to the following:

Stopping agents.
Stopping snmpd:                                            [FAILED]
Stopping vmware-snmpd:                                     [FAILED]

Checking for main agent.

Setting up basic config file.
Do you want to enable SNMP traps for virtual machine events? (y/n) y

Default trapsink is localhost.
You can modify /etc/snmp/snmpd.conf to set up a different trap destination.

Setup finished.
Restarting agents.
Starting snmpd:                                            [ OK ]
Waiting for master agent to start.
Starting vmware-snmpd as subagent:

You could then enable the Master SNMP Agent for required run-levels with

chkconfig snmpd on

Then enable the VMware SNMP SubAgent for required run-levels with

chkconfig vmware-snmpd on

Then we can start both SNMP daemons with

# service snmpd start
# service vmware-snmpd start

Also note, that if you are configuring snmp entirely from the command line, then you will also need to update the file /etc/vmware/config to include the text

serverd.snmpdconf.subagentenabled = "TRUE"

This is the configuration file for the Master SNMP Agent.

The following is the default contents of this file after ESX has been installed.

syscontact root@localhost (edit /etc/snmp/snmpd.conf)
syslocation room1 (edit /etc/snmp/snmpd.conf)
rocommunity public
trapcommunity public
trapsink localhost

Here is the output from lsmod

Module          Size   Used by Tainted: PF
vmnixmod        177056 121
e1000           68456  0 (unused)
usb-uhci        21220  0 (unused)
usbcore         50112  1 [usb-uhci]
megaraid2       32928  6

If a module has a tainted value of 1, this denotes the driver is not covered under the GNU license. The same information that lsmod produces can also be found by inspecting the file /proc/modules. We would do this with a tool such as cat. For example:

# cat /proc/modules

There is a different command which lists the driver modules that the VMkernel is using which is called vmkload_mod and can also be found in this guide.

shutdown -h now       Halt after shutdown
shutdown -r now       Restart after shutdown

esx            Normal ESX boot
linux          Linux SMP kernel, no VMkernel load
linux-up       Linux Uni-processor kernel, no VMkernel load

If we use the cursor key at the LILO screen to select one of the three default choices, the boot prompt (displayed below the menu) changes to reflect this. This allows us to augment the boot command with an option switch.

boot: linux –s

In this case, the –s instructs Linux to boot in single user mode. A critical security point here is that in single user mode, Linux automatically logs on as root! Once in single user mode if we wish to continue into multi-user mode then we type either exit or CTRL-D. To restrict access to single user mode, check the "restricted" parameter in the configuration file /etc/lilo.conf.

# rpm -qa
mailcap-2.1.6-1
setup-2.5.7-1
basesystem-7.0-2
bdflush-1.5-17
chkconfig-1.2.24-1
cracklib-2.7-12
db2-2.4.14-7
etc!.....

If we are only interested in the VMware rpms, then we can just pipe the output of rpm -qa command into the grep search tool.

rpm -qa |grep VMware

which should yield an output something like

VMware-mui-2.5.0-11548
VMware-esx-2.5.0-11548
VMware-perftools-2.5.0-11548
VMware-ccagent-esx-2.5.0-11343

If we then want to find out more information on an individual RPM package, we can use the rpm -qi option to query a package which reports the file version, vendor, license and description.

# rpm -qi VMware-ccagent-esx-2.5.0-11343

Name : VMware-ccagent-esx                Relocations: (not relocateable)
Version     : 2.5.0                           Vendor: VMware, Inc.
Release     : 11343                       Build Date: Tue Nov 30 05:52:16 2004
Install date: Tue Apr 4 17:48:07 2006     Build Host: pa-build11.vmware.com
Group       : Applications/Emulators      Source RPM: VMware-ccagent-esx-2.5.0-11343.src.rpm
Size        : 2360792                        License: commercial
Summary     : VMware CCagent package.
Description :

If we then want to know what files are included in the rpm package, we can use query with the list option to see the files inside. For example, to see the files

# rpm -ql VMware-perftools-2.5.0-11548
/usr/bin/esxtop
/usr/bin/rrdtool
/usr/bin/vmkusage
/usr/bin/vmkusagectl
/usr/lib/vmware/vmkusage-cron.sh
/usr/share/doc/vmware/README-perf
/usr/share/doc/vmware/open_source_licenses-perf.txt
/usr/share/man/man1/esxtop.1

# rpm2cpio VMware-perftools-2.5.0-11548.rpm | cpio -idmv

i = Restore archive
d = Create landing directories
m = Create previous file modification times
v = verbose

# ifup eth0

# ifdown eth0

would take Ethernet interface “eth0” down.

If we wish to take the interface down and then up again, we can separate these two commands with a semicolon to run the commands consecutively.

ifconfig eth0

ifconfig eth0 up
ifconfig lo down
 

Media Independent Interface tool. This tool can be used to force the service console network to a particular speed or duplex.

# mii-tool -F

Doesn’t work correctly with some network cards, including Intel 1000 Pro copper NICs.

ifdown eth0; ifup eth0

The semicolon separating the two commands in the above example can be used to separate any two command line entries when you wish the commands to be executed sequentially. In the Windows command line, the same thing can be achieved by the separator "&&".

Or instead of using the word service, some use

/etc/init.d/network restart

There will be a number of lines to this file, but the one you are likely to be interested in will start "hosts:" as shown:

hosts: files nisplus dns

In the above example, the name service will use the /etc/hosts file, then NIS+ and then the DNS name server specified in the /etc/resolv.conf file.

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1          localhost.localdomain     localhost
192.168.1.10       esx1host.taupoconsulting.net    esx1host

Notice that each line has a 3rd column which specifies an alias.

hostname -i displays the IP address

and

hostname -s displays the short hostname, i.e. without domain name

/etc/resolv.conf            - search domain.com, nameserver=w.x.y.z
/etc/hosts                  - a.b.c.d    esx1.domain.com
/etc/sysconfig/network      - HOSTNAME=esx1.domain.com

This tool does not appear to let you set the DNS domain name.

search taupoconsulting.net
nameserver 192.168.1.150

This text configuration file contains the service console hostname and default gateway IP address.

NETWORKING=yes
HOSTNAME=esx1
GATEWAY=192.168.1.1

DEVICE=eth0
BOOTPROTO=static
IPADDR=192.168.1.51
NETMASK=255.255.255.0
ONBOOT=yes
  

check_link_down () {
return 1;
}

Note this update only relates to Linux Guest operating systems inside a VM, this is not a setting required for the ifcfg-eth0 file in the service console.

route                               Prints routing table
route del –net default              Deletes the default gateway
route add –net default gw w.x.y.z   Adds a new default gateway
  

┌───────┤ Network configuration ├───────┐
│                                       │
│ Would you like to set up networking?  │
│                                       │
│ ┌─────┐ ┌────┐                        │
│ │ Yes │ │ No │                        │
│ └─────┘ └────┘                        │
│                                       │
│                                       │
└─────────────────────────────────────

This utility will update the following IP configuration files for you

/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/sysconfig/network

VMware ESX Server 2.1.2
Kernel 2.4.9-vmnix2 on an i686 

Linux esx1.taupoconsulting.net 2.4.9-vmnix2 #1 Fri Aug 6 04:38:44 PDT 2004 i686

This command displays the currently active network connections.

netstat --inet -n -p -e

id robin

would reveal something like:

uid=508(robin) gid=510(robin) groups=510(robin),506(techsupport)

This output tells us that the user robin has a UID of 508, a primary group membership of robin and secondary group membership of techsupport.

alias lsf='ls -F'

The above command alias will not however persist to another login session. To have that alias available to you on next login, you would need to add this text to your .bashrc file in your home directory.

To make the alias available to all users on the system, you could add the alias definition to the file /etc/bashrc, which is referenced by the users' /home/<user>/.bashrc file, like an include.

If you just type alias without parameters, you will see a list of the aliases you have defined.

passwd <user>

Remember that passwords are not stored in the /etc/passwd file, but in the file /etc/shadow 

If you are ever needing to reset an unknown root account password, then it is this utility you would run after booting into Linux single user mode.
 

useradd sally

would add a user called sally. We could equally have created a service console user by using "Users and Groups" in the Options tab of the MUI. We can set more than the basic properties of a user account with some additional switches. The following command

useradd robin -G techsupport -s /bin/bash -d /home/robin

would add a user called robin who is a member of the techsupportusers group and has a home directory /home/robin and will receive the Linux bash shell at login.

The service console is a modified version of Red Hat Linux (RHL), and by default in RHL, when a user account is added, a group is created of exactly the same name and has only the user account as a member. This feature is called User Private Groups (UPG) and is discussed in more detail on the RedHat documentation website found here.

So, now that we know about UPGs, looking again at the command above, the command adds a user called robin whose primary group (-g) is called robin and other group (-G) membership is techsupport

We can add additional parameters to the useradd command to more fully specify the account.

useradd alistair –g Finance –s /bin/false

In the above example the users’ primary group is Finance and the shell is specified. In this case the shell is /bin/false which is a bogus shell which would prevent interactive logon by this user. By default in the service console, the shell assigned to users is the BASH shell - specified as /bin/bash (BASH stands for Bourne-Again SHell). It appears the only other Linux shell that is shipped with the service console is csh (the C shell).

groupadd esxadmins

In the above example, a new group called esxadmins is created and therefore a new line appears in /etc/group.

gpasswd –a greg esxusers

Group removal is simple with the –d switch:

gpasswd –d tony esxusers

Be very careful with this command if you intend to use it to modify a users' group membership. When used with –G to set the users group membership, it is not adding the user to a group but is actually setting the list of secondary groups a user belongs to. Therefore in the following example if bill had secondary group list of esxusers and sqladmins, then after entering:

usermod –G techsupport bill

then bill would only have a secondary group of techsupport and nothing else! We would have overwritten the entry in the /etc/group file that listed bill as a member of esxusers and sqladmins. This is why the command gpasswd is so much clearer.

It is good to use the id command to check what groups a user is a member of, before and after the user modification operation to ensure you have got it right.

groupmod -n newgroupname oldgroupname

When it used without parameters, we are specifying to switch to the user root. However, we can use the su command to switch shell to any user account. In the first example, we are logged in as the user kevin and we are switching to user ali.

[kevin@esx1host kevin]$ su ali
Password:
[ali@esx1host kevin]

In this second example, we are switching from being logged on as a user called sara to being logged on as root. Notice to switch to root, we don't need to specify a username.

[sara@esx1host sara]$ su -
Password:
[root@esx1host root]#

If we restrict the built-in user account root from logging in over the SSH protocol, then we are forcing remote users to authenticate as themselves and then su to run privileged commands if need be, thus leaving a decent audit trail. The downside being that those users would still know the root account password.

There is a single line in this file that needs to be uncommented to limit the use of su. The line is shown below as it appears it that file, all that is required is the removal of the # symbol at the start of the line.

#auth required /lib/security/pam_wheel.so use_uid

[ali@esx1 ali]$ sudo vmkfstools

The vmkfstools command would then run under the security context of the root user. The superb feature of this tool is that the user ali does not need to know or supply the root password to be able to run the delegated command. Further, we can keep an audit trail of when sudo was invoked.

But a great benefit of using visudo over vi, is that it performs some basic syntax checking for us!

alistair ALL= /bin/kill, /usr/bin/, /usr/sbin/alistair/

The best source I've found so far on detailed use and background of sudo can be found at http://aplawrence.com/Basics/sudo.html

kirsten:x:505:kirsten
esxusers:x:507:kirsten,flagship
flagship:x:508:flagship
vpxuser:x:511:
adminaccount:x:512:
JohnSmith:x:513:

This may look like a list of users, but it is a list of groups. As the service console (vmnix) is a modified version of Red Hat Linux, the Linux security configuration is the same as Red Hat. One feature of Red Hat not found in all Linux distributions is that of the user private group (UPG). Whenever you create a user, a group of the same name is created also and the user is made a member. The format of the file is:

groupname:x:user1,user2

so when we see groups like JohnSmith:x:513 we can assume the 513 is the UID for the user JohnSmith and this is his UPG.

Here is a sample section of a passwd file:

ali:x:500:500:Alistair Sutherland:/home/ali:/bin/bash
sara:x:501:501:Sara Daniels:/home/sara:/bin/bash
janice:x:502:502::/home/janice:/bin/bash
andy:x:503:503::/home/andy:/bin/bash

As shown, the format of the file is

username:x:userID:groupID:fullname:homedirectory:shell

Normally the group ID will match the user ID.

There is a command line tool to edit this file, vipw

ali:$1$tkSdSEQD$x8pXvtDZ3Xta6zza9lKqh.:12733:0:99999:7:::
sara:$1$c4jofyxg$8zjaMTXWhW2hniTXKUt7V/:12733:0:99999:7:::

If a user account has been disabled with the usermod command, a "!" will be placed in front of the encrypted password in this file.

NIS is a network lookup service which consists of databases and processes. It works where a NIS master server stores the source files for the maps such as

/etc/passwd
/etc/group
/etc/hosts

A NIS master serves a NIS domain. You can have multiple NIS servers for a domain, but only 1 is the master, other NIS servers host read-only copies, i.e. they are slaves. NIS databases are in DBM format.

The NIS master server daemon is ypserv.

NIS client machines are those which get their configuration from the NIS Master. A NIS client runs the process ypbind.

Of note are the vmkernel, vmkwarning & messages file logs. These logs can be viewed with the more, cat, head and tail command line tools. We can also access these logs via the MUI via the following link in the Options tab.

If you use the sudo tool to run a command under a different security context then the log file /var/log/secure will contain the audit trail for such activity. Check the file /etc/syslog.conf for logging settings.

You can use less /var/log/logfile and then use SHIFT-f to enable dynamic update as new data is delivered to that file.

It is sometimes useful to add a line to the end of this file

local6.*      /dev/tty3

to get real-time logging of VMkernel to tty3.

lsof |grep IPv4.\*LISTEN

             total     used     free    shared     buffers    cached
Mem:           265      259        5         0          39       135
-/+ buffers/cache:       85      180
Swap:          541        0      541

Given these results, I would be thinking about either running fewer VMs, disconnecting unused devices from VMs, stopping any unnecessary applications or increasing service console RAM.

[root@esx1 root]# fdisk -l

Disk /dev/sda: 255 heads, 63 sectors, 17816 cylinders
Units = cylinders of 16065 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sda1   *         1         6     48163+  83  Linux
/dev/sda2             7       235   1839442+  83  Linux
/dev/sda3           236       304    554242+  82  Linux swap
/dev/sda4           305     17816 140665140    f  Win95 Ext'd (LBA)
/dev/sda5           305      1834  12289693+  83  Linux
/dev/sda6          1835      2063   1839411   83  Linux
/dev/sda7          2064      2076    104391   fc  Unknown
/dev/sda8          2077     17816 126431518+  fb  Unknown

Looking at the above output of the fdisk command, the last two partitions are for the VMkernel. Partitions of type "fc" correspond to the VMKcore dump partition. Partitions of type "fb" are VMFS volumes.

If you wanted to create a new VMFS volume from the service console command, then you could use fdisk to create the custom partition type.

fdisk /dev/sdf
 

In the following example, we have added a 2nd disk to the service console (appearing as SCSI disk "b" i.e. /dev/sdb). By using fdisk we have created a primary partition. Now, to create the file system we use makefs

makefs -t ext3 /dev/sdb1

e2label

du –h /home/ali/vmware
du –h ~
du –s summary
 

# df -h
Filesystem     Size  Used    Avail  Use%  Mounted on
/dev/sda2      2.0G  640M     1.2G   34%  /
/dev/sda1       45M   12M      31M   27%  /boot
/dev/sda7      2.0G   33M     1.8G    2%  /home
none            93M    0       93M    0%  /dev/shm
/dev/sda8      2.0G   33M     1.8G    2%  /tmp
/dev/sda6      2.0G  226M     1.6G   12%  /var
/dev/sda5      9.8G  2.9G     6.5G   31%  /vmimages
//win2k/share  137G   75G      61G   55%  /root/class

This is a great tool to run when first diagnosing an ESX server. The results of this command tell us whether the server was partitioned correctly and if any partitions are constrained for disk space.

# vdf -h
Filesystem     Size  Used    Avail  Use%  Mounted on
/dev/sda2      2.0G  640M     1.2G   34%  /
/dev/sda1       45M   12M      31M   27%  /boot
/dev/sda7      2.0G   33M     1.8G    2%  /home
none            93M    0       93M    0%  /dev/shm
/dev/sda8      2.0G   33M     1.8G    2%  /tmp
/dev/sda6      2.0G  226M     1.6G   12%  /var
/dev/sda5      9.8G  2.9G     6.5G   31%  /vmimages
//win2k/share  137G   75G      61G   55%  /root/class
vmhba0:0:0:10   48G   15G      33G   31%  /vmfs/vmhba0:0:0:10
vmhba1:0:10:1 10.0G  7.0M    10.0G    0%  /vmfs/vmhba1:0:10:1
vmhba1:0:11:1 10.0G  191M     9.8G    1%  /vmfs/vmhba1:0:11:1
vmhba1:0:25:1  136G   21G     114G   15%  /vmfs/vmhba1:0:25:1
vmhba1:0:26:1  136G  8.1G     128G    5%  /vmfs/vmhba1:0:26:1
vmhba1:0:27:1   14G  3.9G      11G   26%  /vmfs/vmhba1:0:27:1
vmhba1:0:28:1   14G  7.0M      14G    0%  /vmfs/vmhba1:0:28:1

When troubleshooting, make this your first command to run. You will be able to review if each partition for the service console and the VMkernel has enough disk space. Just take a quick look down the "Avail" column and if you see a zero there's likely a problem right there, or just look at the USE% column.

dd if=/dev/cdrom of=/vmimages/new.iso bs=32k

This tool could also be used to go from ASCII to EBCDIC etc.

This tool can be used to create an additional swap file. For example, if we did not allocate a big enough swap partition for the service console during ESX installation, we can create one now in a file of 64MB.

dd if=/dev/zero of=/swapfile bs=1M count=64

If we did add a swap file, we would need to make sure it is started when ESX starts. Therefore, an entry in the file system table /etc/fstab would be needed as this file describes the local and remote file systems to mount at boot. The total amount of service console swap space is the sum of the swap partition and any swap files that are active.

Filename                Type       Size   Used  Priority
/dev/sda3               partition  554232 0     -1
/swapfile               file       65528  0     -2
 

# touch newfile

However, this can be used to touch an existing file and update its last modified or last accessed attributes. This could be scripted if required. Be careful and avoid running touch against any file stored on a VMFS volume, as there appears to be a problem there. Remember that not all Linux tools are modified for VMFS awareness.

The VMFS is not an ext3 partition. but the directory /vmfs in the service console provides mount points to the VMkernel-mounted VMFS volumes.

This command is used frequently to view the contents of a text file, exactly as the command type in DOS or Windows command line. So to view a view we could enter

# cat /etc/vmware/netmap.conf

Technically, this is the tool to concatenate files together.

We can also use this tool to create text files quickly at the command line, by entering the text and then using the key sequence CTRL-D to write to file. In the following example, we create a new bare-minimum vmx file at the command line.

# cat > newVM.vmx

guestOS = "winxppro"
config.version = "6"
virtualHW.version = "3"
CTRL-D

Writes the text following echo command to file. This could be good for quickly creating files

echo modprobe usb-uhci > S92usb
echo modprobe usb-ohci >>S92usb

Another great use of this technique is to make changes to the ESX server configuration via the /proc hierarchy, e.g. changing the number of shares for a VM

echo 2500 > /proc/vmware/vm/nnn/cpu/shares

would change the VM CPU shares to 2500. However such a change would only exist for the duration of the world created for that VM. After the VM is powered off this in memory structure is lost. To make such a change persistent, we would need to add the line

sched.cpu.shares = "2476"

to the VMX file of the virtual machine. 

head server.dsk | file -

The “–“ is crucial to making the above command work. For an ESX virtual disk we would expect to see something like standard input: x86 boot sector.

If you are using this to view the last few entries in a log file, you can use the -f switch to "follow" changes as they happen to the file.
  

sort /etc/vmware/vm-list

or to sort a basic score sheet

sort –g –k 2 scores.txt
  

Grep can be used as a command directly e.g.

grep alistair /etc/passwd

or the output of a command can be piped directly into grep, for example the output of all running processes in the service console could be searched for the string "vmware"

ps -eaf |grep vmware 

# cat /proc/vmware/vm/*/names | cut -f1-5,25- -d" "

–mount         used to ensure it doesn't traverse to remote file systems
-size            obvious
-mtime -n     modified in the last n*24
-mmin -n      modified in the last n minutes
-ls               use output format as if ls were used
-name          name the file you are looking for (you just don’t know where it is!)

find –mmin -30                files modified in last 30 minutes
find –mtime -1                files modified in last 24 hours
find –size +10000             files in excess of 10,000 bytes
find –mount –size +10000 -ls  files on non-remote file system
find –name “hosts” -ls        file called hosts
find -exec ls -al {} \;       do ls on the files found
find -perm 666                find files with exactly rw-rw-rw-
find -perm +666               find files with at least rw-rw-rw
find -user ali                find files owned by ali

vi filename

The first thing that throws you is that to enter text into your file, you need to press "i" for Insert mode. You can then enter your text just as any other text editor. When you are done with text entering, just press the Escape (Esc) key to come out of insert mode. If you are happy with your file, then we need to Write & Quit (wq). To enter commands in this command line editor, rather than having menus, we have a command prompt in the application. To reach the vi command prompt, simply enter ":" - the colon character which will automatically place your cursor at the bottom of the session. Here you can enter the "wq" command to write and quit the editor. That's it!

Here is a summary of the vi commands

i                  Changes to insert mode where you can edit the text
:wq               Write the file and quit the editor
:q!               Quit the editor without saving changes

SHIFT ZZ       Quit the editor and save any changes made - just a fast way of doing ":wq"
Esc key          Exits the current mode, e.g. out of insert mode back to view mode.

These commands are just extra if you have the inclination to learn!

/                     search - if you entered /failed then the cursor would move to the first instance of "failed in the text
$                     jumps to the end of the opened file
yy                   copy - it's y for yank!
dd                   delete a line (cut) if you precede this with a number e.g. 8dd, then it would delete 8 lines
p                     paste
%s/old/new/g    substitute any occurrences of the world "old" with the world "new"

There are some great web sites which document the features of vi in superb depth, one of them is the staff site at University of Washington which helped me. Their site is at http://staff.washington.edu/rells/R110/

wc filename

authconfig
sysntv
mouseconfig
netconfig

 

List files in a directory including hidden (also known as dot files due to their prefix) files.

ls -dl */

List directories in long format (does not display files). Could add as a shell alias, say lsd.

If you want to organise files by their modification date.

ls -ltr

If you are interested in knowing where on the disk files are stored, based on their inode, use the -i switch.

ls -lia

more /etc/ssh/sshd_config

Or, to pipe the output of a command into the more utility

ls -al |more

#chown ali solaris.vmx

However if you wish to reset both the user owner and group owner, then rather than having to use chown and then chgrp straight after it, you can set user and group ownership in one operation by specifying the user owner and group owner separated by a colon as in the example shown.

#chown ali:ali netware5.vmx

#ll
-rwxr-wr-- bill bill w2k.vmx

Now we are going to change the group owner of the file to the group called vmadmins.

#chgrp vmadmins w2k.vmx
#ll
-rwxr-xr-- bill vmadmins w2k.vmx

So, in a full file listing, when you see 2 names, e.g. bill vmadmins, the first name is the user owner and the second name is the group owner. In Red Hat Linux, we have something called user private groups, which means that for each user account, there is a group account of the same name. So if you see a file owner and group owner as the same name, these are not the same security principals, one is the user account, the other is a group of the same name.

When we look at a file listing using ls -al the file & directory permissions are shown on the left.

-rwxr-xr-- 1 ali vmadmins 345 May 7 14:22 file.txt

In the above example, the file has 3 permissions described in the -rwxr-xr-- string. These are:

rwx for the User owner  - in our example above, this is the Linux user 'ali'
r-x for the Group owner - in our example above, this is the Linux group 'vmadmins'
r for all Others             - permission for any other user who is neither the user or group owner.

In this first chmod example, we are going to change the permissions on the file.txt by removing the read & execute permission for the user owner of the file and we are also going to remove the read permission for the group owner of the file.

# chmod u-wx,g-x file.txt
# ls -al
-r--r--r-- 1 ali vmadmins 345 May 7 14:22 file.txt

Note that using + or – indicates we are adding to or removing from the existing permissions. If we wish to reset the permission we use “=” to explicitly set the object permissions, overwriting anything that was already set.

# chmod u=rx,g=r,o=r file.txt yields r-xr—-r--

Sometimes you will see a chmod command using 'a' to specify all (user, group & other), so we could quickly set read permissions by

# chmod a-wx,a+r file.txt yields r—-r-—r--

A more common way to set permission is using chmod is using numeric equivalent values (4,2,1 for r,w,x) and permutations thereof.

chmod 777 windows2k.vmx would set permission to rwxrwxrwx
chmod 754 windows2k.vmx would set permission to rwxr-xr-- (default)

Watch for chmod commands with 4 digits, e.g. chmod 0754. This refers to additional attributes as described below.

Sticky bit

When the sticky bit (t) is set on executable files, it tells Linux to keep the application in memory. The reason for this is to improve load times for other users who wish to run the same executable. This relates to the multi-user nature of UNIX/Linux. Given the speed of memory and disk access nowadays the need to keep applications in memory is much less important and so the sticky bit isn't needed so much.

When the sticky bit is turned on for a directory, users can have read and/or write permissions for that directory, but they can only remove or rename files that they own.

If you see a "t" in a file or directory permission, this indicates the sticky bit is set. You can turn on the sticky bit with the chmod tool and specify "t".

chmod +t /directory

You can then view the directory with ls -al and note that the executable permissions indicator bit is shown as a "t" showing that the directory has the sticky bit set.

drwxr-xr-t 2 root root 4096 May 7 12:02 directory
 

SETUID (set uid)

The Set User ID bit is used on an executable file, so that when it is run, it is run under the security context of the file owner and not the current user who launched that executable. So, if I have an executable file whose owner is 'root' and it has the setuid bit set, then when I run this application as a normal user, that application would still run under 'root' privilege.

To set the UID bit, we use chmod with the "s" indicator. In the following example, the Perl script called listswitch.pl is has a user owner 'ali' and a group owner 'vmadmins'. Once the user id bit is set on this file, whoever launches the executable will not in fact be the owner of the process, the user 'ali' will be the process owner.

# chmod u+s listswitch.pl
# ls -al
-rwsr--r-- 1 ali vmadmins 396 May 7 12:09 listswitch.pl

You may have already been using a program with setuid set and not even known about it! The sudo command is owned by root and has the setuid bit set. You can check if the setuid bit is set by inspecting the file permissions

---s--x--x      1   root    root   80764  Jul  23   2001 /usr/bin/sudo

Set Group ID.

Just like SUID, setting the SGID bit for a file sets your group ID to the file's group while the file is executing. So again, we use the chmod tool with 's' but this time we set it on the group permission.

# chmod g+s listswitch.pl
# ls -al
-rwxr-sr-- 1 ali vmadmins 396 May 7 12:14 listswitch.pl

The group id bit is a great feature to enable easier management of permissions on the files in that directory. When the group id bit is set on a directory, any files or subdirectories created in that directory will automatically have their group ownership set to the same as the parent directory!

As we have seen above, to set any of these 3 attributes, we can use the 't' and 's' indicators. However, often we set permissions with chmod using numerical values like 777 to represent rwx. When setting user id, group id or sticky bits using chmod and numerical values, we use a 4th digit preceding the usual 3 used with chmod. That digit is set using the following:

4 Set user ID (s)
2 Set group ID (s)
1 Set sticky bit (t)

So if we want to set a file with permission -rwxr-xr-x and set the user ID bit we could use the following:

# chmod 4755

which would result in a new file permission of -rwsr-xr-x. Notice the "x" of the user permission is now an "s" indicating the setuid bit is set.

The most frequently used umask is 022, this would take away the write permission for the group owner and others in a permission list, i.e. full permission equals 777, corresponding to read(4), write(2) and execute (1).

Full permissions       777   rwxrwxrwx
Minus the umask        022   ----w--w-
Effective permission   755   rwxr-xr-x

1 NIC assigned to Service Console
1 NIC assigned to VMkernel
1 SCSI adapter assigned to be shared between Service Console and VMkernel
1 Fibre Channel adapter assigned to VMkernel

The vmkpcidivy tool is stored in the directory /usr/sbin/vmkpcidivy. This tool asks a series of questions and should be used with the –i switch for interactive mode. To assign a PCI card to either operating system, we use the 3 characters c, v & s.

[c] Assign to Service Console
[v] Assign to VMkernel
[s] Assign as shared between Service Console and VMkernel (the boot disk controller)

To run, we just type vmkpcidivy -i If you add a new NIC, SCSI or fibre channel PCI card to your physical server, you should boot the server into Linux and run the vmkpcidivy command. This way you can correctly assign the PCI card to the right operating system and also allows you to check that the new PCI card has not changed your existing PCI assignment. Once you have saved your changes, restart the server and boot ESX Server normally. This command is also used to refreshnames and –q vmhba_devs For example, if I had a SAN LUN of vmhba1:0:25 and lets say I removed the VMFS from this LUN and now wished to use it from the service console, I’d run

# vmkpcidivy -refreshnames

and then would run vmkpcidivy again this time with the query switch (-q)

# vmkpcidivy -q vmhba_devs

to find out what device name the service console was going to use for this LUN, e.g. vmhba0:0:0 /dev/sda

A very useful feature of this tool is the ability to create a new profile. This adds a new boot option to the LILO boot menu that will have its own allocation of memory and PCI devices. If you are unsure about the changes you are making, then create a new profile e.g. esx (modified)

vmkchdev -L

lists the PCI devices and reports whether they are assigned to VMkernel or the service console. We can also get this information from running vmkpcidivy, but if we only want a quick report of which device is owned by which OS, then this is great. Notice also that the PCI device ID is reported which is very helpful where we have more than one device of the same name, e.g. you could have 2 dual port Intel ethernet cards.

-i                   to import a virtual disk to VMFS
-e                   to export a virtual disk from VMFS
-m                   to commit changes from REDO log
-s                   to re-scan for new LUNs
-S                   to set vmfs metadata volume label
-X 6000M ./file.dsk  to extend an existing DSK to 6GB
-c 4000M ./file.dsk  to create a new empty virtual disk
-C                   to create a new vmfs volume
-l vmfsname          to list virtual disks on specified vmfs
-F                   to set the access mode e.g. public/shared
-k                   to create a VMkernel swap file
-w                   to activate a VMkernel swap file
-y                   to deactivate a VMkernel swap file
-T                   to convert a vmfs1 volume to vmfs2

Remember that the vmfs parameter always goes last on this command parameter set for vmkfstools. This can be confusing for the beginner as the source and target order is different for imports and exports.

If we want to simply list the files on a vmfs volumes we use the -l switch.

vmkfstools –l /vmfs/vmhba0:0:0:8

or if we wish to use the more friendly VMFS volume label;

vmkfstools –l <vmfs-metadatalabel>

which would produce an output similar to the following

Name: VMFS2-VOL1 (public)
Capacity 129465874944 (123461 file blocks * 1048576) avail
Permission Uid Gid Attr Bytes (Blocks)    Last Modified Filename
rw-------   0   0 swap 2146435072 ( 2047) Nov 18 18:25  Swap.vswp
rw------- 500 500 disk 4194304000 ( 4000) Nov 16 14:12  VM1.vmdk
rw------- 500 500 disk 6291456000 ( 6000) Nov 23 22:19  VM2.vmdk
rw------- 500 500 disk 2621440000 ( 2500) Nov 17 23:09  VM3.vmdk
rw------- 500 500 disk 4194304000 ( 4000) Nov 24 18:11  VM4.vmdk

If we use the command with the lh switch we get the results in human readable format. Notice that file sizes are shown rounded with the "G" symbol.

[root@esx4 W2Ktest]# vmkfstools -lh vmhba0:0:0:10

Name: Local (public) Capacity 48G, 33G avail, file block size 1.0M
Permission Uid Gid Attr Bytes Last Modified Filename
rw------- 0 0 swap 1.2G Apr 26 12:30 SwapFile.vswp
rw------- 0 0 disk 2.0G Apr 26 14:07 ad1-win2000server.vmdk
rw------- 0 0 disk 2.0G Apr 27 15:21 ad2-win2000adv.vmdk
rw------- 0 0 disk 2.0G Apr 27 08:41 Clone of ad2-win2000adv.vmdk

To create a new VMFS volume, we use the -C switch. In the following example, we are creating a VMFS volume on LUN16 on host bus adapter 1, typically the fibre channel adapter.

vmkfstools -C vmfs2 vmhba1:0:16:1

If someone has created a VMFS volume with an illegal character in the volume label, you may have problems removing that volume in the MUI. If this is the case, just overwrite the VMFS volume by creating a new volume over the top of the badly named one using the -C switch.

To create a new empty virtual disk on a VMFS volume we use the -c switch

vmkfstools –c 2048M /vmfs/vmhba0:0:0:8:newdisk.vmdk

This command would create a new virtual disk (monolithic) on the specified VMFS volume. Remember it is always better to use the VMFS name as this will not change even if your hba hardware does.

To import a virtual disk into the VMFS we use vmkfstools with the -i switch. This will take a virtual disk in sparse (COW) format into monolithic format without causing excessive SCSI reservations on the LUN holding the target VMFS.

vmkfstools –i /vmimages/template.vmdk /vmfs/vmhba1:0:25:1/new-vm.vmdk

As always with this command, the parameter specifying the VMFS location is always the last parameter.

If you just wish to view the properties of a VMFS volume, you can use the -P switch to print the volume properties. You can use either the logical name for the vmhba partition or the VMFS volume label.

[root@esx1 cpu]# vmkfstools -P VMFS2-VOL1

VMFS2-VOL1 is a VMFS-2.11 volume spanning 1 physical extents.
Volume label (if any): VMFS2-VOL1
UUID (if any): 6890b365-d911e933-7286-8497e91f9b7d
Physical Extents:
vmhba0:0:0:8 

There is no man page for this tool and --help doesn't yield anything beyond simply entering the command without parameters. Some additional information is visible if you enter

vmware-cmd -h

The first thing we can look at is to registering and un-registering a VM. We use the "-s" switch to indicate we performing a server operation, as opposed to VM operation.

vmware-cmd –s register /home/user/vmware/newvm/newvm.vmx

vmware-cmd –s unregister /home/user/vmware/oldvm/oldvm.vmx

The next use of this command is to list the VMs on the server. However, this will only list the registered VMs, i.e. the VMs which are listed in the file /etc/vmware/vm-list

# vmware-cmd –l

/home/vmware/vm1/vm1.vmx
/home/vmware/vm2/vm2.vmx
/home/alistair/vmware/alisrv1/alisrv1.vmx
/home/andy/vmware/andysolaris/andysolaris.vmx

Next we are looking at connecting or disconnecting a device. Typically this will be for the connection of IDE CD-ROM ISO files or floppy image files.

vmware-cmd /home/user/vmware/vm/vm.vmx connectdevice|disconnect

To perform power operations we unsurprisingly use the start and stop parameters. A stop operation type can be soft, try soft or hard. A stop hard is the last resort and equivalent to a forced VM power off. Here is an example of starting and then soft stopping a VM.

# vmware-cmd /home/user/vmware/server/server/vmx start
start() = 1

# vmware-cmd /home/user/vmware/server/server.vmx stop soft
stop(soft) = 1

If we wish to query the current heartbeat value for a VM, the getheartbeat parameter does the trick. Remember though, that in order to draw any meaning from this, we should query the heartbeat twice to prove the value is in fact increasing! For example,

# vmware-cmd /home/user/vmware/server/server.vmx getheartbeat
getheartbeat() = 29076

# vmware-cmd /home/user/vmware/server/server.vmx getheartbeat
getheartbeat() = 29079

If we want to determine simply if the VM is powered on or not, then we can use the getstate

# vmware-cmd /home/user/vmware/server/server.vmx getstate
getstate() = on

To find out the VMID (also known as the world ID) of a VM, we can use the getid parameter. The VMID is analogous to process ID (PID) but is the unique ID that the VMkernel is using for the Virtual Machine Monitor. The VMID of a VM is normally a 3 digit number greater than 100.

# vmware-cmd /home/user/vmware/server/server.vmx getid
getid() = 145

For every VM that is running with a VMID in the VMkernel, there are a parallel set of management processes running in the service console. These processes are there to allow operators interact with the VM, for example, power on and off, gain remote console access and to maintain the per-VM logging in the file vmware.log. To find the parent process ID (PID) of the management processes that correspond to a VM, we can use the getpid parameter.

# vmware-cmd /home/user/vmware/server/server.vmx getpid
getpid() = 12163

Both the VMID and PID remain unchanged while the VM is running. Once the VM is powered off, those IDs are removed and the VM will more than likely get a new VMID and PID the next time it is powered on.

We can also use this tool to answer questions such as the commit of a REDO file to virtual disk:

vmware-cmd "/home/vmware/SPPS 2003/SPPS 2003.vmx" answer

Question (id = 694724352) :No REDO log to be committed

0) OK
Select choice. Press enter for default <0> : 0
selected 0 : OK 

vmkdump -q              Query the VM kernel for which partition it will use
vmkdump -p vmhba0:0:0:3 Set VM kernel dump partition to partition 3
vmkdump –p none:0:0:0     Set VM kernel dump partition to none

Remember the vmkcore partition does not have a mount point in the service console and is not specified as ext3. We can use the fdisk -l command to view where the core dump partition is in relation to the disk layout.
 

# vmkload_mod -l

Name      R/O Addr Length R/