VMware Customer Support Day
Broomfield, Colorado
March 2, 2010
Confidential
© 2009 VMware Inc. All rights reserved
Broomfield Support Day Agenda
 10:00 AM
 10:30 AM
 10:45 AM
 11:15 AM
 12:00 PM
 12:45 PM
 1:45 PM
 2:45 PM
 3:00 PM
 4:00 PM
2
Registration
Kick-off
Keynote - Eric Wansong, VP GSS Americas
vSphere Upgrade Best Practices
Lunch - Q&A with GSS Experts
Storage Best Practices
Networking Best Practices
Break
Performance Best Practices
Wrap-up and Give-away
Confidential
VMware Customer Support Day
 Welcome to Broomfield’s 3rd Customer Support Day
 Collaboration bringing VMware Support, Sales & our Customers
Together
 Collaboration bringing VMware Support, Sales & our Customers
Together
 Value Add
• Education: VMware Best Practices, Tips & Tricks
• Technical Support Overview
• Certification Offerings
• Product Demos
 Customer Feedback- Support Day Topics
3
Confidential
vSphere Upgrade Best Practices
Brian Pope – Install/OS Escalation Engineer, GSS
Confidential
© 2009 VMware Inc. All rights reserved
Agenda





5
Planning
vCenter
ESX/ESXi
VMware Tools / Virtual Hardware
Licensing
Confidential
vSphere Upgrade Pre-planning
 VMware vSphere Upgrade Center
• Collection of Docs, Videos, Best Practices, New Features, etc.
• http://www.vmware.com/products/vsphere/upgrade-center/resources.html
 vSphere Upgrade Guide
• http://www.vmware.com/pdf/vsphere4/r40_u1/vsp_40_u1_upgrade_guide.pdf
 Upgrading to ESX 4.0 and vCenter 4.0 Best Practices
• Knowledge Base Article 1009039
 vSphere Migration Checklist
• http://vmware.com/files/pdf/vsphere-migration-prerequisites-checklist.pdf
 Installing ESX 4.0 and vCenter 4.0 Best Practices
• Knowledge Base Article 1009080
 VMware vCenter Install Worksheet
• Knowledge Base Article 1010023
6
Confidential
vCenter Server
 Upgrade components in the following order:
• vCenter
• ESX/ESXi Hosts
• VMware Tools
• Virtual Hardware
 vCenter now supported on 64bit OS, however, requires a 32bit DSN
• Knowledge Base Article 1010401
 Backup the vCenter Database (should be doing this anyway)
 Verify dbo perms on MSDB, VC, and UM DB’s
 Allow for any new vSphere required ports
• Knowledge Base Article 1012382
 TEST TEST TEST
• Setup a test environment to test critical applications to verify functionality and
performance.
7
Confidential
ESX 4.0 / ESXi 4.0
 vSphere 4.0 offers two GUI-based applications and a script that you
can use to upgrade ESX 3.5 to ESX 4.0:
 vSphere Host Update Utility
• For standalone hosts
 VMware vCenter Update Manager
• For ESX/ESXi hosts that are managed by vCenter Server
• Use ―Host Upgrade‖ baseline vs ―Host Patch‖ baseline
 esxupgrade.sh script
• For Offline Upgrade - ESX 3.x hosts that do not have network access.
Knowledge Base Article 1009440
 Several upgrade tools were supported in previous ESX releases and are no longer
supported in the current release. These tools include graphical upgrade from CD, textmode upgrade from CD, tarball upgrade using the service console, scripted upgrade from
CD or PXE server by using esxupdate, and scripted upgrade from CD or PXE server using
kickstart commands.
8
Confidential
ESX 4.0 / ESXi 4.0
 VMware ESX 4.0 will only install and run on servers with 64-bit x86
CPUs.
• Known 64-bit processors:
• All AMD Opterons support 64 bit.
• All Intel Xeon 3000/3200, 3100/3300, 5100/5300, 5200/5400, 7100/7300, and
7200/7400 support 64 bit.
• All Intel Nehalem processors support 64 bit.
 ESX requires ~15G VMFS volume for Console VM
• The service console must be installed on a VMFS datastore that is resident on
a host's local disk or on a SAN disk that is masked and zoned to that particular
host only. The datastore cannot be shared between hosts.
 Upgrading ESXi 3.5 hosts with OEM server vendor’s specific
components to ESXi 4.0
• Knowledge Base Article 1010489
9
Confidential
VMware Tools / Virtual Hardware
 Upgrading an ESX 3.x virtual machine to ESX 4.0
• Knowledge Base Article 1010675
 VMware tools 4.0 are backwards compatible to 3.x
• Feel free to immediately upgrade VMware Tools. You will still be able to
vMotion to 3.x hosts.
• Snapshot critical VMs in the event tools upgrade is not successful.
• Clone and test VMs to ensure tools and hardware upgrade successfully.
 Virtual Hardware version 7 is NOT backwards compatible
• Once upgraded virtual hardware 7 will only run on ESX 4.0. If done before your
host are all at 4.0 you will limit migration capability.
• Virtual Hardware downgrade is NOT supported.
• Only upgrade virtual hardware for specific VM‘s needing the new features.
• Upgrade is a powered off operation.
• A full reboot following VMware Tools install is required before hardware is
upgraded.
10
Confidential
Licensing
 vSphere Licensing Information Portal
• http://www.vmware.com/products/vsphere/upgrade-center/licensing.html
•
•
•
•
What‘s New in Licensing
Preparing for Your License Upgrade
Entitlement Mapping
Licensing Troubleshooting
 Configuring a legacy license server to manage ESX/ESXi 3.x hosts
in vCenter Server 4.0
• http://kb.vmware.com/kb/1010704
11
Confidential
Questions
Confidential
© 2009 VMware Inc. All rights reserved
Lunch – Q&A
Brian Pope
Josh Newton
Install/OS Escalation Engineer
BCS Engineer
David Garcia
Paul Clark
NASA L2 Escalation Engineer
Storage Escalation Engineer
Paul Hill
Gerald Camacho
System Management Escalation Engineer
Network Escalation Engineer
Jake McDermott
BCS Engineer
Confidential
© 2009 VMware Inc. All rights reserved
Storage Best Practices
Paul Clark – Storage Escalation Engineer, GSS
Confidential
© 2009 VMware Inc. All rights reserved
Agenda
 Performance
 SCSI Reservations
 Performance Monitoring
• esxtop
 Common Storage Issues
• Snapshot LUN‘s
• Virtual Machine Snapshot
• iSCSI Multi Pathing
• All Paths Dead (APD)
15
Confidential
Performance
 Disk subsystem bottlenecks cause more performance problems
than CPU or RAM deficiencies
 Your disk subsystem is considered to be performing poorly if it is
experiencing:
• Average read and write latencies greater than 20 milliseconds
• Latency spikes greater than 50 milliseconds that last for more than a
few seconds
16
Confidential
Performance vs. Capacity
 Performance vs. Capacity comes into play at two
main levels
• Physical drive size
• Hard disk performance doesn‘t scale with drive size
• In most cases the larger the drive the lower the performance.
• LUN size
• Larger LUNs increase the number of VM‘s, which can lead to contention on that
particular LUN
• LUN size is often times related to physical drive size which can compound
performance problems
17
Confidential
Performance – Physical Drive Size
 You need 1 TB of space for an application
• 2 x 500GB 15K RPM SAS drives = ~300 IOPS
• Capacity needs satisfied, Performance low
• 8 x 146GB 15K RPM SAS drives = ~1168 IOPS
• Capacity needs satisfied, Performance high
18
Confidential
SCSI Reservations – Why?
 SCSI Reservations – when an initiator
requests/reserves exclusive use of a target(LUN)
• VMFS is a clustered file system
• Uses SCSI reservations to protect metadata
• To preserve the integrity of VMFS in multi host
deployments
• One host has complete access to the LUN exclusively
• A reboot or release command will clear the reservation
• The virtual machine monitor users SCSI-2 reservations
19
Confidential
SCSI Reservations
 What causes SCSI Reservations
• When a VMDK is created, deleted, placed in REDO
mode, has a snapshot (delta) file, is migrated
(reservations from the source ESX and from the target
ESX) or when the VM is suspended (Since there is a
suspend file written). MEDIUM ERROR – LOGICAL
UNIT NOT READY
• When VMDK is created via a template, we get SCSI
reservations on the source and target
• When a template is created from a VMDK, SCSI
reservation is generated
20
Confidential
SCSI Reservation Best Practice
• Simplify/verify deployments so that virtual machines
do not span more than one LUN
• This will ensure SCSI reservations do not impact more than one LUN
• Determine if any operations are occurring on a LUN
on which you want to perform another operation
• Snapshots
• VMotion
• Template Deployment
• Use a single ESX server as your deployment server to
limit/prevent conflicts with other ESX servers
attempting to perform similar operations
21
Confidential
SCSI Reservation Best Practice - Continued
• Inside vCenter, limit access to actions that initiate
reservations to administrators who understand the effects
of reservations to control WHO can perform such
operations
• Schedule virtual machine reboots so that only one LUN is
impacted at any given time
• A power on and power off are considered separate operations and both with create a
reservations
• VMotion
• Use care when scheduling backups. Consult the backup
provider best practices information
• Use care when scheduling Anti Virus scans and updates
22
Confidential
SCSI Reservation Monitoring
• Monitoring /var/log/vmkernel for:
• 24/0 0x0 0x0 0x0
• SYNC CR messages
• In a shared environment like ESX there will be some SCSI
reservations. This is normal. But when you see 100‘s of
them it‘s not normal.
•
•
•
•
23
Check for Virtual Machines with snapshots
Check for HP management agents still running the storage agent
Check LUN presentation for Host mode settings
Call VMware support to dig into it further
Confidential
Storage Performance Monitoring
Paul Clark – Storage Escalation Engineer, GSS
Confidential
© 2009 VMware Inc. All rights reserved
esxtop
25
Confidential
esxtop - Continued
DAVG = Raw response time from the device
KAVG = Amount of time spent in the VMkernel, aka. virtualization
overhead
GAVG = Response time that would be perceived by virtual machines
D+K=G
26
Confidential
esxtop - Continued
27
Confidential
esxtop - Continued
28
Confidential
esxtop - Continued
• What are correct values for these response times?
• As with all things revolving around performance, it is subjective
• Obviously the lower these numbers are the better
• ESX will continue to function with nearly any response time, however
how well it functions is another issue
• Any command that is not acknowledged by the SAN within 5000ms (5
seconds) will be aborted. This is where perceived disk performance
takes a sharp dive
29
Confidential
Common Storage Issues
Paul Clark – Storage Escalation Engineer, GSS
Confidential
© 2009 VMware Inc. All rights reserved
Snapshot LUNs
 How a LUN is detected as a snapshot in ESX
• When an ESX 3.x server finds a VMFS-3 LUN, it compares the SCSI_DiskID
information returned from the storage array with the SCSI_DiskID information
stored in the LVM Header.
• If the two IDs do not match, the VMFS-3 volume is not mounted.
 A VMFS volume on ESX can be detected as a snapshot for a number of
reasons:
• -LUN ID change
• -SCSI version supported by array changed (firmware upgrade)
• -Identifier type changed – Unit Serial Number vs NAA ID
31
Confidential
Snapshot LUNs - Continued
 Resignaturing Methods
 ESX 3.5
 Enable LVM Resignaturing on the first ESX host
Configuration > Advanced Settings > LVM >
LVM.EnableResignaturing to 1.
 ESX 4
 Single Volume Resignaturing
Configuration > Storage > Add Storage > Disk / LUN
Select Volume to Resignature > Select Mount, or Resignature
32
Confidential
Virtual Machine Snapshots
 What is a Virtual Machine Snapshot:
• A snapshot captures the entire state of the virtual machine at the time you take
the snapshot.
• This includes:
 Memory state – The contents of the virtual machine‘s memory.
 Settings state – The virtual machine settings.
 Disk state – The state of all the virtual machine‘s virtual disks.
33
Confidential
Virtual Machine Snapshot - Continued
 Common issues:
• Snapshots filling up a Data Store
• Offline commit
• Clone VM
• Parent has changed.
• Contact VMware Support
• No Snapshots Found
• Create a new snapshot, then commit.
34
Confidential
ESX4 iSCSI Multi-pathing
 ESX 4, Set Up Multipathing for Software iSCSI
 Prerequisites:
• Two or more NICs.
• Unique vSwtich.
• Supported iSCSI array.
• ESX 4.0 or higher
35
Confidential
ESX4 iSCSI Multi-pathing - Continued
 Using the vSphere CLI, connect the software iSCSI initiator to the
iSCSI VMkernel ports.
 Repeat this command for each port.
• esxcli swiscsi nic add -n <port_name> -d <vmhba>
Verify that the ports were added to the software iSCSI initiator by running the
following command:
• esxcli swiscsi nic list -d <vmhba>
 Use the vSphere Client to rescan the software iSCSI initiator.
36
Confidential
ESX4 iSCSI Multi-pathing - Continued
 This example shows how to connect the software iSCSI initiator
vmhba33 to VMkernel ports vmk1 and vmk2.






37
Connect vmhba33 to vmk1:
esxcli swiscsi nic add -n vmk1 -d vmhba33
Connect vmhba33 to vmk2:
esxcli swiscsi nic add -n vmk2 -d vmhba33
Verify vmhba33 configuration:
esxcli swiscsi nic list -d vmhba33
Confidential
All Paths Dead (APD)
The Issue
 You want to remove a LUN from a vSphere 4 cluster
 You move or Storage vMotion the VMs off the datastore who is being removed
(otherwise, the VMs would hard crash if you just yank out the datastore)
 After removing the LUN, VMs on OTHER datastores would become unavailable (not
crashing, but becoming periodically unavailable on the network)
 the ESX logs would show a series of errors starting with ―NMP‖
38
Confidential
All Paths Dead - Continued
 Workaround 1
 In the vSphere client, vacate the VMs from the datastore being removed (migrate or
Storage vMotion)




In the vSphere client, remove the Datastore
In the vSphere client, remove the storage device
Only then, in your array management tool remove the LUN from the host.
In the vSphere client, rescan the bus.
 Workaround 2
 Only available in ESX/ESXi 4 U1
 esxcfg-advcfg -s 1 /VMFS3/FailVolumeOpenIfAPD
39
Confidential
Questions
Confidential
© 2009 VMware Inc. All rights reserved
vSphere Networking Overview
David Garcia – NASA L2 Escalation Engineer, GSS
Confidential
© 2009 VMware Inc. All rights reserved
Agenda
 Virtual Switches
 Virtual Switch Capabilities
 NIC Teaming
 Link Aggregation
 NIC Failover
 New Adapter Types
 VLANs
 Tips & Tricks
 Troubleshooting
42
Confidential
Why Do We Need a Virtual Switch?
Non-Virtualized
Virtualized
ESX host
…
Access Ports
(single VLAN
—no tagging)
Per Host
network visibility
from each port
Virtual Switch
Access Ports
(single VLAN
—no tagging)
VLAN
Trunks
Layer 2 Access
switches
Layer 2
switches
VLAN
Trunks
VLAN
Trunks
Distribution
and core
43
L2 Virtual Switch
provides fanout and
policy control to each VM
(consistent with nonvirtualized environment)
Layer 2 Virtual Access
switch
Distribution
and core
Confidential
Virtual vs. Physical Network Management
• Separation of Network and Server
provisioning and management systems
• Virtual Center managing & provisioning
ESX hosts and virtual switches
• Physical network managed / provisioned by
vNetwork Distributed Switch
existing networking vendor‘s tools and
applications
Virtual Center
• Network visibility ends at physical
switch port
• Different interfaces
and tools
• IOS CLI for physical network
• VC GUI and esxcfg cli
Network
Management
for vSwitches
44
Confidential
vNetwork Standard Switch
 What is it?
• Virtual network living inside ESX providing
interconnectivity between VMs and the external
physical network via standard networking
protocols (Ethernet)
• Enables many VMs to share same physical NIC
and communicate directly with each other
 Standard Networking Features
 I/O Features
• L2 Ethernet switching (inter-vm traffic)
• Enhanced VMXNET, E1000, VLANCE
• VLAN Segmentation
• Checksum off-loading, TSO, Jumbo Frames,
• Rate limiting - restrict traffic generated by a VM
NetQueue
• NIC port aggregation and redundancy for
• 10GigE, FCoE
enhanced availability and load balancing of
• IB (community support)
physical network resources (VMware NIC
Teaming)
Confidential
45
Copyright © 2005 VMware, Inc. All rights reserved.
vNetwork Standard Switch – Up Close
Uplinks (physical NICs)
attached to vSwitch
Port Groups
created for each host
Virtual Machines
vNetwork Standard
Switch (vSwitch)
Standard Switch for each ESX host
46
Confidential
vNetwork Standard Switch
ESX HOST 1
W2003EE-32-A
W2003EE-32-B
Virtual Machine
Network
Virtual Switch
Virtual Switch – Host1
47
ESX HOST 2
W2003EE-32-A2
ESX HOST 3
W2003EE-32-B2
W2003EE-32-A3
Virtual Machine
Network
Virtual Switch
Virtual Switch – Host2
W2003EE-32-B3
W2003EE-32-A4
W2003EE-32-B4
Virtual Machine
Network
Virtual Machine
Network
Virtual Switch
Virtual Switch
Virtual Switch – Host3
Confidential
ESX HOST 4
Virtual Switch – Host4
vNetwork Distributed Switch (vDS)
APP
APP
APP
APP
APP
APP
APP
APP
APP
OS
OS
OS
OS
OS
OS
OS
OS
OS
vNetwork Distributed Switch
VMware vSphere™

Aggregated cluster level (and
beyond) virtual network
management

Simplified setup and change

Easy troubleshooting, monitoring
and debugging

Additional features include:

48
Confidential

Private VLANs

Bi-directional traffic shaping

Network VMotion

3rd party distributed switch
support
Bundled with vSphere Enterprise
Plus
vNetwork Distributed Switch (vDS) - Continued
ESX Host 1
A
B
G
H
C
Service
Console
D
I
E
J
vmkernel
The Data Plane remains in each
ESX host and is responsible for
frame forwarding, teaming, etc
The Virtual Switch Control
Planes are aggregated
in vCenter Server
DV Port Groups aggregated over
entire vDS and across hosts and
group ports with same
configuration and policy
49
ESX Host 3
ESX Host 2
Service
Console
vmkernel
F
K
L
Service
Console
…
vmkernel
vCenter Server
vDS Representation
A
B
C
D
E
F
G
H
I
J
K
L
Service
Console
1
Service
Console
2
Service
Console
3
vNetwork Distributed Switch
Production
Network
Confidential
Vmk
1
Vmk
2
Vmk
3
vNetwork Distributed Switch: Configuration View
DV Uplink Port Group
defines uplink policies
DV Uplinks abstract
actual physical nics
(vmnics) on hosts
DV Port Groups
span all hosts
covered by vDS
vmnics on each host
mapped to dvUplinks
50
Confidential
vSphere Networking - 3rd Party - Distributed Switch Style
Host1
W2003EE-32-A
W2003EE-32-B
Host2
W2003EE-32-A2
Host3
W2003EE-32-B2
W2003EE-32-A3
W2003EE-32-B3
Host4
W2003EE-32-A4
W2003EE-32-B4
Single
Distributed
Port
Group
3rd Party Distributed Virtual Machine Network
vSwitch
vNetwork
Single
Distributed
Virtual Switch
3rd Party Distributed
Switch Spanning
Host1, Host2,
Host3, Host4
51
Confidential
vNetwork Appliance API
•
•
•
Heavyweight filtering
in ―Slow Path‖ agent
Lightweight filtering
in ―Fast Path‖ agent
•
•
•
52
Filter driver in vmkernel to provide
security features within ESX
networking layer
vNetwork Appliance APIs available
to partners
Clients of this API may
inspect/alter/drop/inject any frame
on a given port:
• Either directly in the IO path
(fast path agent)
• Or by punting frames up to an
appliance VM (slow path agent)
State mobility for data in fast path
agent and slow path agent
Communication between slow path
and fast path agents
Bind to VM‘s vNIC or to dvswitch port
Confidential
vNetwork - 3rd Party Virtual Switches – Who does what?


3rd Party Virtual Switches enable end to end
physical and virtual networking feature parity
Network admins now able to provision and
monitor the virtual network using existing physical
network management tools
Roles and Responsibilities
53
Third Party VSwitch
vNetwork
Distributed
Switch
vNetwork
(with 3rd Party virtual
switching)
Associate VMs to virtual networks
vSphere Admin
vSphere Admin
Associate server NICs to virtual networks
vSphere Admin
vSphere Admin
Create Virtual Switches
vSphere Admin
Network Admin
Create Port Groups
vSphere Admin
Network Admin
Modify VLAN Settings (virtual)
vSphere Admin
Network Admin
Configure NIC Team
vSphere Admin
Network Admin
Monitors Virtual Network
vSphere Admin
Network Admin
Confidential
Nexus 1000V & vCenter Server Views
―show interface‖ from Nexus 1000V VSM console
View from vSphere Client to vCenter Server
―show module‖ from Nexus 1000V VSM console
―access‖ port—assigned
to single VLAN
VSM
VEM
54
Confidential
vDS Deployment Options
Original Environment
vSS
vSS
Complete Migration to vDS
vSS
vDS
 vSS, vDS and Nexus Switches
can co-exist on same host
Partial Migration to vDS
 Network VMotion only required
for Guest VMs
• Optionally leave SC, vmkernel
ports on vSS
• Note: enhanced features only on
VMs use
vDS
Service Console,
vmkernel ports
remain on vSS
vDS
vSS
vSS
vSS
vDS
KB - Migrating virtual machines between vSwitch or PortGroups to vDS or dvPortgroups (1010612)
55
Confidential
vDS Deployment Options - Continued
Original Environment
vSS
vSS
vSS
vSS
Complete Migration to Nexus 1000V
vSS
Cisco Nexus 1000V
vSS
Partial Migration to Nexus 1000V
Multiple vDS
vDS
vDS
vSS
vDS
56
Confidential
vSS
vSS
vDS Deployment Rules
 vSS, vDS, Nexus 1000V can co-exist
• Multiple vSS and vDS per host
• Maximum of one Nexus 1000V per host (VEM)
 Take note of deployment limits (subject to change!)
• Refer to published limits
 pnics (vmnics) can only belong to one virtual switch
57
Confidential
vNetwork Solution Comparisons
VMware
Standard
Switch
VMware
Distributed Switch
Cisco Nexus 1000V
Per Host
Per ―Datacenter‖
Per ―Datacenter‖
L2 Forwarding
YES
YES
VLAN Segmentation
YES
YES
802.1Q Tagging
YES
YES
NIC Teaming
YES
YES
TX Rate Limiting
YES
YES
CDP Support
YES
YES
vNetwork Appliance APIs
YES
YES
Virtual Network Model
58
Datacenter-level management
YES
RX Rate Limiting
YES
VM Network Port Block
YES
PVLAN Support
YES
Network VMotion
YES
3rd Party Distributed Switch Support
YES
Confidential
Cisco Catalyst /
Nexus Features and
Functionality
vSphere Networking Summary
•
•
What is it?
•
Virtual network (i.e., set of virtual switches) living
inside ESX providing interconnectivity between VMs
and the external physical network
•
Enables many VMs to share physical NICs and
communicate directly with each other
Virtual Networking with vSphere 4
•
•
L2 Switching Features and Management
Cluster level unified virtual network
management

Datacenter class features including VLAN,
Private VLANs, CDP, RX/TX rate limiting etc.
vSphere 4 I/O Features
•
VMXNET Generation 3 (VMXNET3)
•
HW offloading
(Checksum/TSO/LRO)
•
Jumbo Frames (VM, NFS and
SW iSCSI)
•
NetQueue v2
•
VMDirectPath
•
10GigE
•
FCoE

Built-in availability (NIC Teaming) providing pnic
redundancy, availability and load balancing

•
vNetwork Platform Extensibility
3rd Party Distributed Switch Support (Cisco
Nexus 1000-V)


•
59
VMsafe-Net Support
IPv6 Support (VM, management, VC server)
Confidential
vSphere Networking Best Practices
David Garcia – NASA L2 Escalation Engineer, GSS
Confidential
© 2009 VMware Inc. All rights reserved
ESX Virtual Switch: Capabilities
 Layer 2 switch—forwards frames based
on 48-bit destination MAC address in
frame
VM0
 MAC address known by registration
VM1
MAC
address
assigned to
vnic
vSwitch
(it knows its VMs!)—no MAC
learning required
 Can terminate VLAN trunks (VST mode)
or pass trunk through to VM (VGT mode)
 Physical NICs associated with
vSwitches
 NIC teaming (of uplinks)
• Availability: uplink to multiple
physical switches
• Load sharing: spread load
over uplinks
61
Confidential
ESX Virtual Switch: Forwarding Rules
 The vSwitch will forward frames
• VM  VM
VM0
MAC a
VM1
• VM  Uplink
MAC b
MAC c
 But not forward
• vSwitch to vSwitch
vSwitch
vSwitch
• Uplink to Uplink
 ESX vSwitch will not create
loops in the physical network
 And will not affect Spanning Tree
(STP) in the physical network
Physical
Switches
62
Confidential
Spanning Tree Protocol (STP) Considerations
 Spanning Tree Protocol used to create
VM0
VM1
loop-free L2 tree topologies
in the physical network
MAC a
• Some physical links put in ―blocking‖ state
MAC b
to construct loop-free tree
vSwitch
 ESX vSwitch does not participate
in Spanning Tree and will not create
vSwitch drops
BPDUs
loops with uplinks
• ESX Uplinks will not block and always
Physical
Switches
Switches sending
BPDUs every 2s to
construct and
maintain Spanning
Tree Topology
Blocked link
active (full use of all links)
Recommendations for Physical Network Config:
1.
Leave Spanning Tree enabled on physical network and ESX
facing ports (i.e. leave it as is!)
2.
Use ―portfast‖ or ―portfast trunk‖ on ESX facing ports
(puts ports in forwarding state immediately)
3.
Use ―bpduguard‖ to enforce STP boundary
KB - STP may cause temporary loss of network connectivity when a failover or failback event occurs (1003804)
63
Confidential
NIC Teaming for Availability and Load Sharing
 NIC Teaming aggregates multiple
VM0
physical uplinks for:
• Availability—reduce exposure
VM1
to single points of failure
(NIC, uplink, physical switch)
• Load Sharing—distribute load over
multiple uplinks (according to selected NIC
teaming algorithm)
vSwitch
 Requirements:
NIC Team
• Two or more NICs on same vSwitch
• Teamed NICs on same L2
broadcast domain
KB - NIC teaming in ESX Server (1004088)
KB - Dedicating specific NICs to portgroups while maintaining NIC teaming and failover for the vSwitch (1002722)
64
Confidential
NIC Teaming Options
Name
Algorithm—vmnic
chosen based upon:
Physical Network Considerations
Originating
Virtual Port ID
vnic port
Teamed ports in same L2 domain
(BP: team over two physical switches)
Source MAC
Address
MAC seen on vnic
Teamed ports in same L2 domain
(BP: team over two physical switches)
IP Hash*
Hash(SrcIP, DstIP)
Teamed ports configured in static
802.3ad ―Etherchannel‖
- no LACP
- Needs MEC to span 2 switches
Explicit Failover
Order
Highest order uplink
from active list
Teamed ports in same L2 domain
(BP: team over two physical switches)
Best Practice: Use Originating Virtual PortID
for VMs
*KB - ESX Server host requirements for link aggregation (1001938)
*KB - Sample configuration of EtherChannel/Link aggregation with ESX and Cisco/HP switches (1004048)
65
Confidential
NIC Teaming with vDS
Teaming Policies Are Applied in DV Port Groups to dvUplinks
esx10a.tml.local
esx09a.tml.local
esx09b.tml.local
KB - vNetwork Distributed Switch on ESX 4.x - Concepts Overview (1010555)
vDS
A
B
Service
Console
vmkernel
vmnic0 esx09a.tml.local
vmnic0 esx09b.tml.local
vmnic0 esx10a.tml.local
vmnic2 esx10b.tml.local
3
vmnic1
vmnic3
vmnic2
vmnic0
vmnic1 esx09a.tml.local
vmnic1 esx09b.tml.local
vmnic1 esx10a.tml.local
vmnic0 esx10b.tml.local
2
vmnic2 esx09a.tml.local
vmnic2 esx09b.tml.local
―Orange‖ DV Port Group
Teaming Policy
vmnic2 esx10a.tml.local
esx10b.tml.local
A
B
1
vmnic3 esx10b.tml.local
Service
Console
vmkernel
vmnic3 esx09a.tml.local
vmnic3 esx09b.tml.local
vmnic3 esx10a.tml.local
0
vmnic1 esx10b.tml.local
vmnic0
66
Confidential
vmnic1
vmnic2
vmnic3
Link Aggregation
67
Confidential
Link Aggregation - Continued
EtherChannel
is a port trunking (link aggregation is Cisco's term) technology used primarily on Cisco switches
Can be created from between two and eight active Fast Ethernet, Gigabit Ethernet, or 10 Gigabit Ethernet ports
LACP or IEEE 802.3ad
Link Aggregation Control Protocol (LACP) is included in IEEE specification as a method to control the bundling of
several physical ports together to form a single logical channel
Only supported on Nexus 1000v
EtherChannel vs. 802.3ad
EtherChannel and IEEE 802.3ad standards are very similar and accomplish the same goal
There are a few differences between the two, other than EtherChannel is Cisco proprietary and 802.3ad is an open
standard
EtherChannel Best Practice
One IP to one IP connections over multiple NICs are not supported (Host A one connection session to Host B uses
only one NIC)
Supported Cisco configuration: EtherChannel Mode ON – ( Enable Etherchannel only)
Supported HP configuration: Trunk Mode
Supported switch Aggregation algorithm: IP-SRC-DST short for (IP-Source-Destination)
The only load balancing option for vSwitch or vDistributed Switch that can be used with EtherChannel is IP HASH
Do not use beacon probing with IP HASH load balancing
Do not configure standby uplinks with IP HASH load balancing.
68
Confidential
Failover Configurations
• Link Status Only relies solely on the link status provided by the network adapter
•Detects failures such as cable pulls and physical switch power failures
•Cannot detect configuration errors
•Switch port being blocked by spanning tree
•Switch port configured for the wrong VLAN
•cable pulls on the other side of a physical switch.
• Beacon Probing sends out and listens for beacon probes
•Ethernet broadcast frames sent by physical adapters to detect upstream network
connection failures
•on all physical Ethernet adapters in the team, as shown in Figure
•Detects many of the failures mentioned above that are not detected by link status alone
•Should not be used as a substitute for a redundant Layer 2 network design
•Most useful to detect failures in the closest switch to the ESX Server hosts
•Beacon Probing Best Practice
•Use at least 3 NICs for triangulation
•If only 2 NICs in team, probe can‘t determine which link failed
•Shotgun mode results
•KB - What is beacon probing? (1005577)
•KB - ESX host network flapping error when Beacon Probing is selected (1012819)
•KB - Duplicated Packets Occur when Beacon Probing Is Selected Using vmnic and
VLAN Type 4095 (1004373)
•KB - Packets are duplicated when you configure a portgroup or a vSwitch to use a route
that is based on IP-hash and Beaconing Probing policies simultaneously (1017612)
69
Confidential
Figure — Using beacons to detect upstream
network connection failures.
Port Group Configuration
 A Port Group is a template for one or more ports with a common configuration
• Assigns VLAN to port group members
• L2 Security—select ―reject‖ to see only frames for VM MAC addr
• Promiscuous mode/MAC address change/Forged transmits
• Traffic Shaping—limit egress traffic from VM
• Load Balancing—Origin VPID, Src MAC, IP-Hash, Explicit
• Failover Policy— Link Status & Beacon Probing
• Notify Switches—‖yes‖-gratuitously tell switches of mac location
• Failback—‖yes‖ if no fear of blackholing traffic, or, …
• … use Failover Order in ―Active Adapters‖
 Distributed Virtual Port Group (vNetwork Distributed Switch)
• All above plus:
• Bidirectional traffic shaping (ingress and egress)
• Network VMotion—network port state migrated upon VMotion
70
Confidential
VMXNET3—The Para-virtualized VM Virtual NIC
• Next evolution of ―Enhanced VMXNET‖ introduced in ESX 3.5
• Adds
• MSI/MSI-X support (subject to guest operating system kernel support)
• Receive Side Scaling (supported in Windows 2008 when explicitly enabled
through the device's Advanced configuration tab)
• Large TX/RX ring sizes (configured from within the virtual machine)
• High performance emulation mode (Default)
• Supports
•
•
•
•
•
High DMA
TSO (TCP Segmentation Offload) over IPv4 and IPv6
TCP/UDP checksum offload over IPv4 and IPv6
Jumbo Frames
802.1Q tag insertion
KB - Choosing a network adapter for your virtual machine (1001805)
71
Confidential
VMDirectPath for VMs

What is it?


Device Driver

Types of workloads

I/O Appliances

High performance VMs
Details

Virtual
Layer

Enables direct assignment of PCI devices to VM
Guest controls the physical H/W
Requirements

vSphere 4

I/O MMU

I/O Device

Used for DMA Address Translation (Guest Physical 
Host Physical) and protection
Generic device reset (FLR, Link Reset, ...)
KB - Configuring VMDirectPath I/O pass-through devices on an ESX host (1010789)
72
Confidential
FCoE on ESX
 VMware ESX Support
• FCoE supported since ESX 3.5u2
• Requires Converged Network
ESX
Adapters ―CNAs‖—(see HCL) e.g.
• Emulex LP21000 Series
• Qlogic QLE8000 Series
vSwitch
Fibre
Channel
HBA
10GigE
NIC
• Appears to ESX as:
CNA—Converged
Network Adapter
• 10GigE NIC
• FC HBA
FCoE
• SFP+ pluggable transceivers
FCoE
Switch
Ethernet
73
• Copper twin-ax (<10m)
• Optical
Fibre
Channel
Confidential
Using 10GigE
Ingress (into switch)
traffic shaping policy
control on Port Group
Variable/high
b/w 2Gbps+
iSCSI
NFS
1-2G
High
b/w
Low b/w
VMotion
FT
SC
 2x 10GigE common/expected
SC#2
• 10GigE CNAs or NICs
 Possible Deployment Method
• Active/Standby on all Portgroups
vSwitch
FCoE
10GE
10GE
FCoE
Gbps
• VMs ―sticky‖ to one vmnic
10
• SC/vmk ports sticky to other
FCoE
• Use Ingress Traffic Shaping
to control traffic type per
FCoE Priority Group
bandwidth reservation
(in CNA config utility)
Port Group
• If FCoE, use Priority Group
bandwidth reservation
74
Confidential
(on CNA utility)
Traffic Types on a Virtual Network
Virtual Machine Traffic
• Traffic sourced and received from virtual machine(s)
• Isolate from each other based on service level
VMotion Traffic
• Traffic sent when moving a virtual machine from one ESX host to another
• Should be isolated
Management Traffic
• Should be isolated from VM traffic (one or two Service Consoles)
• If VMware HA is enabled, includes heartbeats
IP Storage Traffic—NFS and/or iSCSI via vmkernel interface
• Should be isolated from other traffic types
 Fault Tolerance (FT) Logging Traffic
• Low latency, high bandwidth
• Should be isolated from other traffic types
How do we maintain traffic isolation without proliferating NICs?
75
Confidential
VLAN Trunking to Server
IEEE 802.1Q VLAN Tagging
VM0
VM1
• Enables logical network partitioning
(Traffic separation)
• Scale traffic types without scaling physical NICs
PortGroup
―Blue‖
VLAN 20
Port Group
―Yellow‖
VLAN 10
• Virtual machines connect to virtual
switch ports (like access ports
vSwitch
on physical switch)
• Virtual switch ports are associated
with a particular VLAN (VST mode)—defined
VLAN Trunks
Carrying
VLANs 10, 20
in PortGroup
• Virtual switch tags packets exiting host
12-bit VLAN id
field
(0-4095)
8100
802.1Q Header
76
Confidential
VLAN Tagging Options
VST – Virtual Switch Tagging
VLAN
assigned in
Port Group
policy
vSwitch
VLAN Tags
applied in
vSwitch
Physical Switch
VGT – Virtual Guest Tagging
vSwitch
vSwitch
VLAN Tags
applied in
Guest
PortGroup
set to VLAN
―4095‖
Physical Switch
Physical Switch
External Physical
switch applies
VLAN tags
VST is the best practice and
most common method
77
EST – External Switch Tagging
Confidential
VLAN Tagging: Further Example
Access Ports
on VLAN 10
1
2
Access Ports
on VLAN 20
3
4
5
6
Access Ports
on VLAN 50
7
A
8
B
9
10
11
12
13
15
14
All VLANs
(10,20,50,90)
trunked to
VM
C
VLAN Trunks
interface GigabitEthernet1/2
description host32-vmnic0
Carrying VLANs
10, 20, 50, 90
switchport trunk encapsulation dot1q
switchport trunk native vlan 999
Example
configuration on
Physical Switch
switchport trunk allowed vlan 10,20,50,90
switchport mode trunk
spanning-tree portfast trunk
KB -Sample configuration of virtual switch VLAN tagging (VST Mode) and ESX Server (1004074)
Uplinks A, B, and C connected to trunk ports on physical switch which carry four VLANs
(e.g. VLANs 10, 20, 50, 90)
Ports 1-14 emit untagged frames, and only those frames which were tagged with their respective VLAN ID
(equivalent to ―access port‖ on physical switch)
• Port Group VLAN ID set to one of 1-4094
Port 15 emits tagged frames for all VLANs.
• Port Group VLAN ID set to 4095 (for vSS) or ―VLAN Trunking‖ on vDS DV Port Group
78
Confidential
Private VLANs: Traffic Isolation for Every VM
 Solution: PVLAN
Private VLAN traffic isolation
between guest VMs
• Place VMs on the same virtual network
but prevent them from communicating
directly with each other (saves VLANs!)
• Avoids scaling issues from assigning
one VLAN and IP subnet per VM
 Details
• Instead, configure a SINGLE DV port
group to have a SINGLE isolated*
VLAN (ONLY ONE)
• Attach all your VMs to this SINGLE
Distributed
Switch with
PVLAN
Common
Primary VLAN
on uplinks
isolated VLAN DV port group
KB - Private VLAN (PVLAN) on vNetwork Distributed Switch - Concept Overview (1010691)
79
Confidential
Private VLANs - Continued
W2003EE-32-A
PG
W2003EE-32-B
PG
W2003EE-32-A
PG
W2003EE-32-B
PG
W2003EE-32-A
PG
W2003EE-32-B
W2003EE-32-A
W2003EE-32-B
PG
PG
PG
W2003EE-32-A
PG
W2003EE-32-B
PG
W2003EE-32-A
W2003EE-32-B
PG
PG
vNetwork Distributed Switch
TOTAL COST: 12 VLANs (one per VM)
W2003EE-32-A
W2003EE-32-B
W2003EE-32-A
W2003EE-32-B
W2003EE-32-A
W2003EE-32-B
W2003EE-32-A
W2003EE-32-B
W2003EE-32-A
W2003EE-32-B
W2003EE-32-A
W2003EE-32-B
PG (with Isolated PVLAN)
vNetwork Distributed Switch
TOTAL COST: 1 PVLAN (over 90% savings…)
80
Confidential
Designing the Network
How do you design the virtual network for
performance and availability and but maintain
isolation between the various traffic types
(e.g. VM traffic, VMotion, and Management)?
• Starting point depends on:
• Number of available physical ports on server
• Required traffic types
• 2 NIC minimum for availability, 4+ NICs
per server preferred
• 802.1Q VLAN trunking highly recommended for logical
scaling (particularly with low NIC port servers)
• Previous examples are meant as guidance and do not
represent strict requirements in terms of design
• Understand your requirements and resultant traffic types and
design accordingly
81
Confidential
Tips & Tricks
• KB - Changing a MAC address in a Windows virtual machine (1008473)
• When a physical machine is converted into a virtual machine, the MAC address of the network adapter is
changed. This can pose a problem when software is installed where the licensing is tied to the MAC
address.
• KB – Configuring speed and duplex of an ESX Server host network adapter (1004089)
• ESX recommended settings for Gigabit-Ethernet speed and duplex while connecting to a physical switch
port are as following:
• Auto Negotiate <-> Auto Negotiate
• It is not recommended to mix hard-coded setting with Auto-negotiate.
• KB - Sample Configuration - Network Load Balancing (NLB) Multicast mode over routed
subnet - Cisco Switch Static ARP Configuration (1006525)
• NLB Multicast Mode – Static ARP Resolution
• Since NLB packets are unconventional, meaning the IP address is Unicast while the MAC address of it is
Multicast, switches and routers drop NLB packets
• NLB Multicast Packets get dropped by routers and switches, causing the ARP tables of switches to not get
populated with cluster IP and MAC address
• Manual ARP Resolution of NLB cluster address is required on physical switch and router interfaces
• Cluster IP and MAC static resolution is set on each switch port that connects to ESX host
82
Confidential
Tips & Tricks - Continued
• KB - Cisco Discovery Protocol (CDP) network information via command line and
VirtualCenter on an ESX host (1007069)
• Utilizing Cisco Discovery protocol (CDP) to get switch port configuration information.
• This command is utilized to troubleshoot network connectivity issues related to VLAN tagging methods on
virtual and physical port settings.
• KB - Troubleshooting network issues with the Cisco show tech-support command
(1015437)
• If you experience networking issues between vSwitch and physical switched environment, you can obtain
information about the configuration of a Cisco router or switch by running the show tech-support command
in privileged EXEC mode.
• Note: This command does not alter the configuration of the router.
83
Confidential
Troubleshooting
• KB - ESX host or virtual machines have intermittent or no network connectivity
(1004109)
• KB - Troubleshooting Nexus 1000V vDS network issues (1014977)
• KB - Cisco Nexus 1000V installation and licensing information (1013452)
• Cisco Nexus 1000V Troubleshooting Guide, Release 4.0(4)SV1(2) 20/Jan/2010
• Cisco Nexus 1000V Troubleshooting Guide, Release 4.0(4)SV(1) 21/Jan/2010
• KB - Configuring promiscuous mode on a virtual switch or portgroup (1004099)
• KB - Troubleshooting network issues by capturing and sniffing network traffic via
tcpdump (1004090)
• KB - Troubleshooting network connection issues using Address Resolution Protocol
(ARP) (1008184)
• IEEE OUI and Company id Assignments http://standards.ieee.org/regauth/oui/index.shtml
• KB - Network performance issues (1004087)
84
Confidential
Troubleshooting - Continued
• KB - Low Network Throughput in Windows Guest when Running UDP Application
(5298153)
• KB - Performance of Outgoing UDP Packets Is Poor (10172)
• KB - Poor Network File Copy performance between local VMFS and shared VMFS
(1003554)
• KB - Cannot connect to ESX 4.0 host for 30-40 minutes after boot (1012942)
• Ensure that DNS is configured and reachable from the ESX host
• KB - Identifying issues with and setting up name resolution on ESX Server (1003735)
• Note: localhost must always be present in the hosts file. Do not modify or remove the entry for localhost
• The hosts file must be identical on all ESX Servers in the cluster
• There must be an entry for every ESX Server in the cluster
• Every host must have an IP address, Fully Qualified Domain Name (FQDN), and short name
• The hosts file is case sensitive. Be sure to use lowercase throughout the environment
85
Confidential
Must Read…
Conclusion
VMXNET3, the newest generation of virtual network adapter from
VMware, offers performance on par with or better than its previous
generations in both Windows and Linux guests. Both the driver
and the device have been highly tuned to perform better on
modern systems. Furthermore, VMXNET3 introduces new
features and enhancements, such as TSO6 and RSS. TSO6
makes it especially useful for users deploying applications that
deal with IPv6 traffic, while RSS is helpful for deployments
requiring high scalability. All these features give VMXNET3
advantages that are not possible with previous generations of
virtual network adapters. Moving forward, to keep pace with an
ever‐increasing demand for network bandwidth, we recommend
customers migrate to VMXNET3 if performance is of top concern
to their deployments.
Conclusion
This study compares performance results for e1000 and
vmxnet virtual network devices on 32-bit and 64-bit guest
operating systems using the netperf benchmark. The results
show that when a virtual machine is running with software
virtualization, e1000 is better in some cases and vmxnet is
better in others. Vmxnet has lower latency, which sometimes
comes at the cost of higher CPU utilization. When hardware
virtualization is used, vmxnet clearly provides the best
performance.
86
Confidential
Break
Confidential
© 2009 VMware Inc. All rights reserved
vSphere 4
Performance Best Practices
Paul Hill - System Management Escalation Engineer, GSS
Confidential
© 2009 VMware Inc. All rights reserved
Agenda
 vSphere 4 Performance Enhancements Overview
 Virtual Center
 High Availability
 Distributed Resource Scheduler
 Fault Tolerance
 Hardware considerations and settings
 CPU Performance
 Memory Performance
 Benchmarking
 Common Support Issues
89
Confidential
vSphere 4 Performance Enhancements
 Efficiency - reduced virtualization overheads, higher consolidation ratios
 Control – Performance monitoring and management, dynamic resource sizing
providing better scalability
 Choice – Offering several options for guest OS, virtualization technologies,
broader HCL, and integration with 3rd party management tools
 Scalability Enhancements:
vCPU – 4 to 8
VM memory – 65GB to 255GB
Host CPU core – 32 to 64
Host max memory – 256GB to 1TB
Powered-on VM per host – 128 to 320
90
Confidential
Performance Enhancements (cont.)
 New Virtual Machine HW support
-SAS for MSCS 2008 support
-IDE for older guest OS
-VMXnet generation 3
-VM hotplug support: dynamically add vCPU and memory
-VMDirectPath: High i/o to physical network devices
 CPU Enhancements
-Relaxed CO-Scheduler tuned for SMP VMs
-CPU locking algorithm is improved to reduce overhead in situations
where scheduling decisions are required
-Scheduler aware of process cache to optimize CPU usage
91
Confidential
Memory Enhancements
 HW-Assisted Memory Virtualization
 VMs differ from physical servers in the respect of virtual memory address
translation
 Guest VM memory addresses need to be translated to guest
physical addresses
using the guest OS’s page tables before translating to machine physical memory
addresses
 The use of shadow page tables are required for this mapping which concurs CPU
and memory overhead
 Current processor HW is now available to improve this situation with 2nd level page
table support
-Intel EPT (Extended Page Tables)
-AMD RVI (Rapid Virtualization Indexing)
 TLB miss costs in the form of latency is higher with 2nd level PTs than with a single
PT, but the use of large pages for the guest OS will improve performance
92
Confidential
Virtual Center Scalability
 Ability to support large scale virtual datacenters
 Single VC server = manage 300 ESX hosts, 3000 virtual machines
 Ability to link multiple VC servers with ―Linked-Mode‖ to manage up to 10,000
virtual machines from a single console
 Performance Charts
-Now, view CPU, memory, disk, network from a single dashboard
-Identify top resource consumers from an aggregate view
-Thumbnail views for host, resource pools, clusters, data-stores to Quickly navigate to a specific chart
 Easily drill down at many levels of the inventory to flesh out performance issue
-Data-store utilization by file type and unused capacity
 Application Performance
- Oracle, SQL Server, SAP, Exchange, Java
 Tools - VMMark, APPspeed
93
Confidential
VirtualCenter Performance
 High CPU utilization and sluggish UI performance
Number of clients attached is high
VC needs to keep clients consistent with inventory changes
Aggressive alarm settings
 DB administration
Periodic maintenance
Recovery and log settings
Appropriate VC statistics level
 Use gigabit NICs for the service console to clone VMs
 Assign permissions appropriately
 SQL Server Express will only run well up to 5 hosts and/or 50 VMs.
Past that, VC needs to run off an Enterprise-class DB.
94
Confidential
Virtual Center Best Practices
 VC Database sizing
Estimate of the space required to store your performance statistics in the DB
 Separate Critical Files onto Separate Drives
Make sure the database and transaction log files are placed on separate
physical drives
Place the tempdb database on a separate physical drive if possible
Arrangement distributes the I/O to the DB and dramatically improves its
performance
If a third drive is not feasible, place the tempdb files on the transaction log drive
 Enable Automatic Statistics
 Keep vCenter logging level low, unless troubleshooting
 Proper scheduling of DB backups, maintenance, monitoring
 Do not run vCenter on a server that has many applications running
 vCenter Heartbeat - http://www.vmware.com/products/vcenter-serverheartbeat/
95
Confidential
High Availability (HA)
 HA network configuration check – DNS, NTP, lowercase
hostnames, HA advanced settings
 Redundancy: server hardware, shared storage, network,
management
 Test network isolation from a core switch level, and host failure
for expected outage behavior
 Critical VMs should NOT be grouped together
 Categorize VM criticality, then set the failover appropriately
 Valid VM network label names required for proper failover
 Failover capacity/Admission control may be too conservative
when host and VM sizes vary widely – slot size calculator in VC
96
Confidential
DRS (Distributed Resource Scheduler)
 Higher number of hosts => more DRS balancing options
Recommend up to 32 hosts/cluster, may vary with VC server configuration
and VM/host ratio
 Network configuration on all hosts - VMotion network: Security policies,
VMotion NIC enabled, Gig
 Reservations, Limits, and Shares
- Shares take effect during resource contention
- Low limits can lead to wasted resources
- High VM reservations may limit DRS balancing
- Overhead memory
- Use resource pools for better manageability, do not nest too deep
 Virtual CPU‘s and Memory size
High memory size and virtual CPU‘s => fewer migration opportunities
Configure VMs based on need network, etc.
97
Confidential
DRS (Cont.)
 Ensure hosts are CPU compatible
- Intel vs. AMD
- Similar CPU family/features
- Consistent server bios levels, and NX bit exposure
- Enhanced VMotion Compatibility (EVC)
- ―VMware VMotion and CPU Compatibility‖ whitepaper
- CPU incompatibility => limited DRS VM migration options
 Larger Host CPU and memory size preferred for VM placement (if all equal)
 Differences in cache or memory architecture => inconsistency in performance
 Aggressiveness threshold - Moderate threshold (default) works well for most cases
Aggressive thresholds recommended if homogenous clusters and VM demand relatively
constant and few affinity/anti-affinity rules
 Use affinity/anti-affinity rules only when needed
Affinity rules: closely interacting VMs Anti-affinity rules: I/O intensive workloads, availability
 Automatic DRS mode recommended (cluster-wide)
Manual/Partially automatic mode for location-critical VMs (per VM)
Per VM setting overrides cluster-wide setting
98
Confidential
FT - Fault Tolerance









99
FT Provides complete VM redundancy
By definition, FT doubles resource requirements
Turning on FT disables performance-enhancing features like, H/W MMU
Each time FT is enabled, it causes a live migration
Use a dedicated NIC for FT traffic
Place primaries on different hosts
Asynchronous traffic patterns
Host Failure considerations
Run FT on machines with similar characteristics
Confidential
HW Considerations and Settings
 When purchasing new servers, target MMU virtualization(EPT/RVI) processors,
or at least CPU virtualization(VT-x/AMD-V) depending on your application work
loads
 If your application workload is creating/destroying a lot of processes, or
allocating a lot of memory them MMU will help performance
 Purchase uniform, high-speed, quality memory, populate memory banks evenly
in the power of 2.
 Choosing a system for better i/o performance MSI-X is needed which allows
support for multiple queues across multiple processors to process i/o in parallel
 PCI slot configuration on the motherboard should support PCIe v/2.0 if you
intend to use 10 gb cards, otherwise you will not utilize full bandwidth
100
Confidential
HW Considerations and Settings (cont.)
 BIOS Settings
- Make sure what you paid for,… is enabled in the bios
-enable ―Turbo-Mode‖ if your processors support it
- Verify that hyper-threading is enabled – more logical CPUs
allow more options
for the VMkernel scheduler
- NUMA systems verify that node-interleaving is enabled
- Be sure to disable power management if you want to maximize performance unless
you are using DPM. Need to decide if performance out-weighs power savings
C1E halt state - This causes parts of the processor to shut down for a short period of time in order to save
energy and reduce thermal loss
-Verify VT/NPT/EPT are enabled as older Barcelona systems do not enable these by
default
-Disable any unused USB, or serial ports
101
Confidential
Resource Types - CPU
 CPU resources are the raw processing speed of a given host or
VM
 However, on a more abstract level, we are also bound by the
hosts’ ability to schedule those resources.
 We also have to account for running a VM in the most optimal
fashion, which typically means running it on the same processor
that the last cycle completed on.
102
Confidential
CPU Performance
 Some multi-threaded apps in a SMP VM may not
perform well
 Use multiple UP VMs on a multi-CPU physical machine
ESX Server
103
ESX Server
Confidential
CPU Performance
 CPU virtualization adds varying amounts of overhead
Little or no overhead for the part of the workload that can run in direct
execution
Small to significant overhead for virtualising sensitive privileged instructions
 Performance reduction vs. increase in CPU utilization
CPU-bound applications: any CPU virtualization overhead results in reduced
throughput
non-CPU-bound applications: should expect similar throughput at higher CPU
utilization
104
Confidential
CPU Performance
 ESX supports up to eight virtual processors per VM
• Use UP VMs for single-threaded applications
• Use UP HAL or UP kernel
• For SMP VMs, configure only as many VCPUs as needed
• Unused VCPUs in SMP VMs:
• Impose unnecessary scheduling constraints on ESX Server
• Waste system resources (idle looping, process migrations, etc.)
105
Confidential
CPU Performance
 Full support for 64-bit guests
 64-bit can offer better performance than 32-bit
• More registers, large kernel tables, no HIGHMEM issue in Linux
 ESX Server may experience performance problems due to shared
host interrupt lines
• Can happen with any controller; most often with USB
• Disable unused controllers
• Physically move controllers
• See KB 1290 for more details
106
Confidential
Resource Types - Memory
 When assigning a VM a ―physical‖ amount of RAM, all you are really
doing is telling ESX how much memory a given VM process will
maximally consume past the overhead.
 Whether or not that memory is physical depends on a few factors: Host
configuration, DRS shares/Limits/Reservations and host load.
 Generally speaking, it is better to OVER-commit than UNDER-commit.
107
Confidential
Memory Performance
 ESX memory space overhead
Service Console: 272 MB
VMkernel: 100 MB+
Per-VM memory space overhead increases with:
Number of VCPUs
Size of guest memory
32 or 64 bit guest OS
 ESX memory space reclamation
Page sharing
Ballooning
108
Confidential
Memory Performance
 Page tables
ESX cannot use guest page tables
ESX Server maintains shadow page tables
Translate memory addresses from virtual to machine
VA
Per process, per VCPU
VMM maintains physical (per VM) to machine maps
No overhead from ―ordinary‖ memory references
PA
 Overhead
Page table initialization and updates
Guest OS context switching
MA
109
Confidential
Memory Performance
 Avoid high active host memory over-commitment
• Total memory demand = active working sets of all VMs
+ memory overhead
– page sharing
• No ESX swapping: total memory demand < physical memory
 Right-size guest memory
• Define adequate guest memory to avoid guest swapping
• Per-VM memory space overhead grows with guest memory
110
Confidential
Memory Performance
 Increasing a VM’s memory on a NUMA machine
Will eventually force some memory to be allocated from a remote node, which
will decrease performance
Try to size the VM so both CPU and memory fit on one node
Node 0
111
Node 1
Confidential
Memory Performance
 NUMA scheduling and memory placement policies in ESX manages
all VMs transparently
No need to manually balance virtual machines between nodes
NUMA optimizations available when node interleaving is disabled
 Manual override controls available
Memory placement: 'use memory from nodes'
Processor utilization: 'run on processors'
Not generally recommended
 For best performance of VMs on NUMA systems
# of VCPUs + 1 <= # of cores per node
VM memory <= memory of one node
112
Confidential
Install VMware Tools





vmxnet – high speed networking driver
Memory balloon driver
Improved graphics – mks, screen resolution
Idler program – deschedule Netware guests when idle
Timer sponge for correct accounting of time
Experimental, manually started
www.vmware.com/pdf/vi3_esx_vmdesched.pdf
 Time Sync – syncs time with the host every minute
Manually started (KB 1318)
113
Confidential
Benchmarking Guidelines
 Carefully select benchmarks
Represents application
Documentation
Repeatability
 Define parameters being measured and their metrics
Throughput (MBps), latency (ms)
 Benchmark a specific system component
Monitor specific component metrics
Ensure no other component on the system is constrained
Or document any such constraint
 For comparisons, preferably vary single parameter at a time
114
Confidential
Benchmarking Guidelines
 Comparing native and virtual machines
# of Physical CPUs = # of Virtual CPUs
Native Kernel/HAL = VM Kernel/HAL
Physical Memory = VM Memory
Same bitness (32 or 64) of OS and application
 Timing within the VM can be inaccurate
Especially when the processor is over-committed
Use external time source (e.g., the ‗ping‘ methodology)
 Performance tools may not work accurately in a VM
115
Confidential
Benchmarking Guidelines
 VMmark: A scalable benchmark for virtualized enterprise systems
• Provides meaningful measurement of virtualization performance
• Generates metric that scales with underlying system capacity
• Used to compare the performance of different hardware and virtualization
platforms
• Employs realistic, diverse workloads running on multiple OSes
•
•
•
•
•
116
Mail server:
Windows 2003 / MS Exchange 2003 / LoadSim
Java server:
Windows 2003 / SPECjbb2005
Web server:
SLES10 / SPECweb2005
Database server:
SLES10 / MySQL / SysBench
File server:
SLES10 / DBench
Confidential
Common Support Issues – ESX/ESXi
 Snapshots are not a backup tool
 Running a VM on a snapshot may cause performance issues, and could fill up
the data stores
 VM-support logs, archive point in time
 ESXi - profile capture – syslog export – VIMA - mix with classic
 Set the COS to 800 MB
 v/4 COS: custom build of RHEL5 up3, limit non-standard RPMS
 Hardware health monitoring
 AMD Servers- NUMA memory balance
 Size guest memory appropriately to avoid guest OS swapping
117
Confidential
Common Support Issues – Troubleshooting
 Log collection at the time of the problems, and prior to support requestwebsite initiated.
 Limit configuration changes
 Call right away, always better to ask first than wait
 Stay current, be proactive, not reactive, could be too late
 Keep VMware tools up to date
 KB, documentation, technical papers
118
Confidential
Reference links
http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf
http://www.vmware.com/resources/techresources/10041
http://www.vmware.com/resources/techresources/10054
http://www.vmware.com/resources/techresources/10066
http://www.vmware.com/files/pdf/perf-vsphere-cpu_scheduler.pdf
http://www.vmware.com/pdf/RVI_performance.pdf
http://www.vmware.com/pdf/Perf_ESX_Intel-EPT-eval.pdf
http://www.vmware.com/files/pdf/perf-vsphere-fault_tolerance.pdf
119
Confidential
Questions
Confidential
© 2009 VMware Inc. All rights reserved