VMware Validated Design Reference Architecture Guide

VMware Validated Design™
for Micro-Segmentation
Reference Architecture Guide
VMware Validated Design
for Micro-Segmentation 3.0
This document supports the version of each product listed
and supports all subsequent versions until the document is
replaced by a new edition. To check for more recent
editions of this document, see
http://www.vmware.com/support/pubs.
EN-002236-00
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
The VMware Web site also provides the latest product updates.
If you have comments about this documentation, submit your feedback to:
docfeedback@vmware.com
© 2016 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright
and intellectual property laws. This product is covered by one or more patents listed at
http://www.vmware.com/download/patents.html.
VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other
jurisdictions. All other marks and names mentioned herein may be trademarks of their respective
companies.
VMware, Inc.
3401 Hillview Avenue
Palo Alto, CA 94304
www.vmware.com
© 2016 VMware, Inc. All rights reserved.
Page 2 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Contents
1.
Purpose and Intended Audience ...................................................... 9
2.
Architecture Overview .................................................................... 10
2.1 Physical Infrastructure Architecture ............................................................................. 11
2.1.1
Pod Architecture .................................................................................................................... 11
2.1.2
Physical Network Architecture ............................................................................................... 13
2.1.3
Availability Zones and Regions ............................................................................................. 19
2.2 Virtual Infrastructure Architecture ................................................................................ 21
3.
2.2.1
Infrastructure Architecture ..................................................................................................... 21
2.2.2
Virtual Infrastructure Overview .............................................................................................. 21
2.2.3
Network Virtualization Architecture ........................................................................................ 23
2.2.4
Logging Architecture ............................................................................................................. 28
2.2.5
Operations Management Architecture ................................................................................... 30
Detailed Design.............................................................................. 33
3.1 Physical Infrastructure Design ..................................................................................... 33
3.1.1
Physical Design Fundamentals ............................................................................................. 34
3.1.2
Physical Networking Design .................................................................................................. 40
3.1.3
Physical Storage Design ....................................................................................................... 49
3.2 Virtual Infrastructure Design ........................................................................................ 57
3.2.1
Virtual Infrastructure Design Overview .................................................................................. 58
3.2.2
ESXi Design .......................................................................................................................... 61
3.2.3
vCenter Server Design .......................................................................................................... 63
3.2.4
Virtualization Network Design................................................................................................ 77
3.2.5
NSX Design ........................................................................................................................... 91
3.2.6
Shared Storage Design ....................................................................................................... 111
3.3 Operations Infrastructure Design ............................................................................... 121
3.3.1
vRealize Log Insight Design ................................................................................................ 121
© 2016 VMware, Inc. All rights reserved.
Page 3 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
List of Tables
Table 4. vRealize Operations Manager Logical Node Architecture ...................................................... 32
Table 5. Regions ................................................................................................................................... 34
Table 6. Availability Zones and Regions Design Decisions .................................................................. 35
Table 7. Required Number of Racks ..................................................................................................... 36
Table 8. POD and Racks Design Decisions ......................................................................................... 37
Table 9. ESXi Host Design Decisions ................................................................................................... 39
Table 10. Host Memory Design Decision .............................................................................................. 40
Table 11. Jumbo Frames Design Decisions ......................................................................................... 44
Table 12. VLAN Sample IP Ranges ...................................................................................................... 46
Table 13. Physical Network Design Decisions...................................................................................... 48
Table 14. Additional Network Design Decisions ................................................................................... 49
Table 15. Virtual SAN Physical Storage Design Decision .................................................................... 50
Table 16. Virtual SAN Mode Design Decision ...................................................................................... 50
Table 17. Hybrid and All-Flash Virtual SAN Endurance Classes ......................................................... 52
Table 18. SSD Endurance Class Design Decisions ............................................................................. 52
Table 19. SSD Performance Classes ................................................................................................... 53
Table 20. SSD Performance Class Selection ....................................................................................... 53
Table 21. SSD Performance Class Design Decisions .......................................................................... 54
Table 22. Virtual SAN HDD Environmental Characteristics .................................................................. 54
Table 23. HDD Characteristic Selection ............................................................................................... 55
Table 24. HDD Selection Design Decisions.......................................................................................... 55
Table 25. NFS Usage Design Decisions ............................................................................................... 56
Table 26. NFS Hardware Design Decision ........................................................................................... 57
Table 27. Volume Assignment Design Decisions ................................................................................. 57
Table 28. ESXi Boot Disk Design Decision ........................................................................................... 62
Table 29. ESXi User Access Design Decisions .................................................................................... 63
Table 30. Other ESXi Host Design Decisions ....................................................................................... 63
Table 31. vCenter Server Design Decision ........................................................................................... 64
Table 32. vCenter Server Platform Design Decisions .......................................................................... 65
Table 33. Platform Service Controller Design Decisions ...................................................................... 65
Table 34. Methods for Protecting vCenter Server System and the vCenter Server Appliance ............ 66
Table 35. vCenter Server Systems Protection Design Decisions ......................................................... 67
Table 36. Logical Specification for Management vCenter Server Appliance ........................................ 67
Table 37. Logical Specification for Compute vCenter Server Appliance .............................................. 67
Table 38. vCenter Appliance Sizing Design Decisions ......................................................................... 68
Table 39. vCenter Database Design Decisions .................................................................................... 69
Table 40. vSphere HA Design Decisions .............................................................................................. 70
© 2016 VMware, Inc. All rights reserved.
Page 4 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 41. vSphere Cluster Workload Design Decisions ....................................................................... 71
Table 42. Management Cluster Design Decisions ................................................................................ 71
Table 43. Management Cluster Attributes ............................................................................................ 72
Table 44. Edge Cluster Design Decisions ............................................................................................ 73
Table 45. Shared Edge and Compute Cluster Attributes ...................................................................... 74
Table 46. Compute Cluster Design Decisions ...................................................................................... 75
Table 47. Monitor Virtual Machines Design Decisions ......................................................................... 75
Table 48. vSphere Distributed Resource Scheduling Design Decisions .............................................. 76
Table 49. VMware Enhanced vMotion Compatibility Design Decisions ............................................... 76
Table 50. vCenter Server TLS Certificate Design Decisions ................................................................ 76
Table 51. Virtual Switch Design Decisions ........................................................................................... 79
Table 52. Virtual Switch for the Management Cluster .......................................................................... 79
Table 53. vDS-Mgmt Port Group Configuration Settings ...................................................................... 80
Table 54. Management Virtual Switches by Physical/Virtual NIC......................................................... 80
Table 55. Management Virtual Switch Port Groups and VLANs .......................................................... 81
Table 56. Management VMkernel Adapter ........................................................................................... 81
Table 57. Virtual Switch for the shared Edge and Compute Cluster .................................................... 82
Table 58. vDS-Comp01 Port Group Configuration Settings ................................................................. 82
Table 59. Shared Edge and Compute Cluster Virtual Switches by Physical/Virtual NIC ..................... 83
Table 60. Edge Cluster Virtual Switch Port Groups and VLANs........................................................... 83
Table 61. Shared Edge and Compute Cluster VMkernel Adapter ........................................................ 84
Table 62. Virtual Switches for Compute Cluster Hosts ......................................................................... 84
Table 63. vDS-Comp02 Port Group Configuration Settings ................................................................. 84
Table 64. Compute Cluster Virtual Switches by Physical/Virtual NIC .................................................. 85
Table 65. Compute Cluster Virtual Switch Port Groups and VLANs .................................................... 86
Table 66. Compute Cluster VMkernel Adapter ..................................................................................... 86
Table 67. NIC Teaming and Policy ....................................................................................................... 87
Table 68. NIC Teaming Design Decision .............................................................................................. 87
Table 69. Network I/O Control Design Decision ................................................................................... 88
Table 70. VXLAN Design Decisions ..................................................................................................... 90
Table 71. NSX for vSphere Design Decision ........................................................................................ 91
Table 72. Consumption Method Design Decisions ............................................................................... 93
Table 73. NSX Controller Design Decision ........................................................................................... 94
Table 74. NSX for vSphere Physical Network Requirements ............................................................... 95
Table 75. Resource Specification of NSX Components ....................................................................... 96
Table 76. NSX Edge Service Gateway Sizing Design Decision ........................................................... 97
Table 77. vSphere Compute Cluster Split Design Decisions ................................................................ 99
Table 78. VTEP Teaming and Failover Configuration Design Decision ............................................. 101
Table 79. Logical Switch Control Plane Mode Decision ..................................................................... 102
© 2016 VMware, Inc. All rights reserved.
Page 5 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 80. Transport Zones Design Decisions ..................................................................................... 103
Table 81. Routing Model Design Decision .......................................................................................... 103
Table 82. Transit Network Design Decision ........................................................................................ 105
Table 83. Tenant Firewall Design Decision ........................................................................................ 105
Table 84. Load Balancer Features of NSX Edge Services Gateway ................................................. 106
Table 85. NSX for vSphere Load Balancer Design Decision .............................................................. 107
Table 86.Virtual to Physical Interface Type Design Decision ............................................................. 107
Table 87. Inter-Site Connectivity Design Decisions ............................................................................ 108
Table 88. Isolated Management Applications Design Decisions ........................................................ 108
Table 90. Application Virtual Network Configuration .......................................................................... 111
Table 91. Network Shared Storage Supported by ESXi Hosts ........................................................... 112
Table 92. vSphere Features Supported by Storage Type .................................................................. 112
Table 93. Storage Type Design Decisions.......................................................................................... 114
Table 94. VAAI Design Decisions ....................................................................................................... 115
Table 95. Virtual Machine Storage Policy Design Decisions .............................................................. 116
Table 96. Storage I/O Control Design Decisions ................................................................................ 116
Table 97. Resource Management Capabilities Available for Datastores ........................................... 117
Table 112. NFS Version Design Decision ........................................................................................... 118
Table 113. NFS Export Sizing ............................................................................................................. 119
Table 114. NFS Export Design Decisions ........................................................................................... 120
Table 115. NFS Datastore Design Decision ....................................................................................... 120
Table 179. Cluster Node Configuration Design Decision ................................................................... 122
Table 180. Compute Resources for a vRealize Log Insight Medium-Size Node................................ 123
Table 181. Compute Resources for the vRealize Log Insight Nodes Design Decision ...................... 123
Table 182. vRealize Log Insight Isolated Network Design Decisions ................................................. 124
Table 183. IP Subnets in the Application Isolated Networks .............................................................. 124
Table 184. IP Subnets Design Decision ............................................................................................. 125
Table 185. DNS Names of the vRealize Log Insight Nodes ............................................................... 125
Table 186. DNS Names Design Decision ........................................................................................... 125
Table 187. Virtual Disk Configuration in the vRealize Log Insight Virtual Appliance .......................... 126
Table 188. Retention Period Design Decision .................................................................................... 126
Table 189. Log Archive Policy Design Decision ................................................................................. 127
Table 190. SMTP Alert Notification Design Decision .......................................................................... 127
Table 192. Custom Role-Based User Management Design Decision ................................................ 128
Table 193. Custom Certificates Design Decision ............................................................................... 128
Table 194. Direct Log Communication to vRealize Log Insight Design Decisions ............................. 128
Table 195. Time Synchronization Design Decision ............................................................................ 129
Table 196. Syslog Protocol Design Decision ...................................................................................... 129
Table 197. Protocol for Event Forwarding across Regions Design Decision ..................................... 130
© 2016 VMware, Inc. All rights reserved.
Page 6 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
List of Figures
Figure 1. Overview of SDDC Architecture ............................................................................................ 10
Figure 2. Pods in the SDDC .................................................................................................................. 12
Figure 3. Leaf-and-Spine Physical Network Design ............................................................................. 13
Figure 4. High-level Physical and Logical Representation of a Leaf Node ........................................... 14
Figure 5. Pod Network Design .............................................................................................................. 16
Figure 6. Oversubscription in the Leaf Layer ........................................................................................ 17
Figure 7. Compensation for a Link Failure ............................................................................................ 18
Figure 8. Quality of Service (Differentiated Services) Trust Point ........................................................ 19
Figure 9. Availability Zones and Regions .............................................................................................. 20
Figure 10. Virtual Infrastructure Layer in the SDDC ............................................................................. 21
Figure 11. SDDC Logical Design .......................................................................................................... 22
Figure 12. NSX for vSphere Architecture .............................................................................................. 24
Figure 13. NSX for vSphere Universal Distributed Logical Router ....................................................... 27
Figure 19. Cluster Architecture of vRealize Log Insight ........................................................................ 29
Figure 20. vRealize Operations Manager Architecture ......................................................................... 31
Figure 21. Physical Infrastructure Design ............................................................................................. 33
Figure 22. Physical Layer within the SDDC .......................................................................................... 34
Figure 23. SDDC Pod Architecture ....................................................................................................... 35
Figure 24. Leaf-and-Spine Architecture ................................................................................................ 41
Figure 25. Example of a Small-Scale Leaf-and-Spine Architecture ..................................................... 42
Figure 26. Leaf-and-Spine and Network Virtualization ......................................................................... 43
Figure 27. Leaf Switch to Server Connection within Compute Racks .................................................. 44
Figure 28. Leaf Switch to Server Connection within Management/Shared Compute and Edge Rack . 45
Figure 29. Sample VLANs and Subnets within a Pod .......................................................................... 46
Figure 30. Oversubscription in the Leaf Switches................................................................................. 48
Figure 31. Virtual Infrastructure Layer Business Continuity in the SDDC ............................................ 58
Figure 32. SDDC Logical Design .......................................................................................................... 59
Figure 33. vSphere Data Protection Logical Design ............................................................................. 60
Figure 34. Disaster Recovery Logical Design ....................................................................................... 61
Figure 35. vCenter Server and Platform Services Controller Deployment Model ................................ 66
Figure 36. vSphere Logical Cluster Layout ........................................................................................... 69
Figure 37. Network Switch Design for Management Hosts .................................................................. 80
Figure 38. Network Switch Design for shared Edge and Compute Hosts ............................................ 83
Figure 39. Network Switch Design for Compute Hosts ......................................................................... 85
Figure 40. Architecture of NSX for vSphere .......................................................................................... 92
Figure 41. Conceptual Tenant Overview .............................................................................................. 98
Figure 42. Cluster Design for NSX for vSphere .................................................................................. 100
© 2016 VMware, Inc. All rights reserved.
Page 7 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 43. Logical Switch Control Plane in Hybrid Mode .................................................................... 102
Figure 44. Virtual Application Network Components and Design ....................................................... 109
Figure 45. Detailed Example for vRealize Automation Networking .................................................... 110
Figure 46. Logical Storage Design ...................................................................................................... 113
Figure 49. NFS Storage Exports ......................................................................................................... 119
Figure 58. Operations Infrastructure Conceptual Design ................................................................... 121
Figure 59. Logical Design of vRealize Log Insight .............................................................................. 121
Figure 60. Networking Design for the vRealize Log Insight Deployment ........................................... 124
© 2016 VMware, Inc. All rights reserved.
Page 8 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
1.
Purpose and Intended Audience
VMware Validated Design for Micro-Segmentation Reference Architecture Guide contains a
validated model of the VMware Validated Design for Micro-Segmentation use case and provides
a detailed design of each management component of the stack.
The Architecture Overview discusses the building blocks and the main principles of each layer.
The Detailed Design provides the available design options according to the design objective, and
a set of design decisions to justify selecting the path for building each component.
The VMware Validated Design for Micro-Segmentation Reference Architecture Guide is
compliant and validated with certain product versions. See the VMware Validated Design for
Micro-Segmentation Planning and Preparation Guide for more information about supported
product versions.
The VMware Validated Design for Micro-Segmentation Reference Architecture Guide is intended
for cloud architects, infrastructure administrators and cloud administrators who are familiar with
and want to use VMware software to deploy in a short time and manage an SDDC that meets the
requirements for capacity, scalability, backup and restore, and extensibility for disaster recovery
support.
Note Design decisions in this document are based on design decisions in the VMware
Validated Design Reference Architecture Guide, but some decision have been removed or
changed. As a result, the decisions are not always numbered consecutively.
Note Illustrations in this document show an architecture that includes VMware Virtual SAN.
The Validated Design for Micro-Segmentation does not include Virtual SAN, but can be
expanded to use Virtual SAN.
© 2016 VMware, Inc. All rights reserved.
Page 9 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
2.
Architecture Overview
The VMware Validated™ Design for Micro-Segmentation enables an IT organization to automate the
provisioning of common repeatable requests and to respond to business needs with more agility and
predictability.
The architecture of the full VMware Validated Design is based on a number of layers and modules,
which allows interchangeable components be part of the end solution or outcome such as the SDDC.
If a particular component design does not fit the business or technical requirements for whatever
reason, it should be able to be swapped out for another similar component. The VMware Validated
Designs are one way of putting an architecture together. They are rigorously tested to ensure stability,
scalability and compatibility. Ultimately however, the system is designed in such a way as to ensure
the desired IT outcome will be achieved.
The VMware Validated Design for Micro-Segmentation does not include all layers of the illustration
below, but can be expanded to include all layers.
Figure 1. Overview of SDDC Architecture
Physical Layer
The lowest layer of the solution is the Physical Layer, sometimes referred to as the 'core', which
consists of three main components, Compute, Network and Storage. Inside the compute component
sit the x86 based servers that run the management, edge and tenant compute workloads. There is
some guidance around the capabilities required to run this architecture, however no
recommendations on the type or brand of hardware is given. All components must be supported on
the VMware Hardware Compatibility guide.
Virtual Infrastructure Layer
Sitting on the Physical Layers infrastructure is the Virtual Infrastructure Layer. Within the Virtual
Infrastructure Layer, access to the physical underlying infrastructure is controlled and allocated to the
management and tenant workloads. The Virtual Infrastructure Layer consists primarily of the physical
host's hypervisor and the control of these hypervisors. The management workloads consist of
elements in the virtual management layer itself, along with elements in the Cloud Management Layer,
Service Management, Business Continuity and Security areas.
Cloud Management Layer
The Cloud Management Layer is the "top" layer of the stack and is where the service consumption
occurs. This layer is not part of the VMware Validated Design for Micro-Segmentation but is part of
the software-defined data center. Typically, through a UI or API, this layer calls for resources and then
© 2016 VMware, Inc. All rights reserved.
Page 10 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
orchestrates the actions of the lower layers to achieve the request. While the SDDC can stand on its
own without any other ancillary services, for a complete SDDC experience other supporting
components are needed. The Service Management, Business Continuity and Security areas complete
the architecture by providing this support.
Service Management
When building any type of IT infrastructure, portfolio and operations management play key roles in
continued day-to-day service delivery. The Service Management area of this architecture mainly
focuses on operations management in particular monitoring, alerting and log management. Portfolio
Management is not a focus of this SDDC design but may be added in future releases.
Business Continuity
To ensure a system that is enterprise ready, it must contain elements to support business continuity in
the area of backup, restore and disaster recovery. This area ensures that when data loss occurs, the
right elements are in place to ensure there is no permanent loss to the business. The design provides
comprehensive guidance on how to operate backups, restore and also the run books of how to fail
components over in the event of a disaster.
Security
All systems need to inherently be secure by design to reduce risk and increase compliance while still
providing a governance structure. The security area outlines what is needed to ensure the entire
SDDC is resilient to both internal and external threats.
2.1
Physical Infrastructure Architecture
2.1.1 Pod Architecture
The VMware Validated Design for SDDC uses a small set of common building blocks called pods.
Pods can include different combinations of servers, storage equipment, and network equipment, and
can be set up with varying levels of hardware redundancy and varying quality of components. Pods
are connected to a network core that distributes data between them. The pod is not defined by any
hard physical properties, as it is a standard unit of connected elements within the SDDC network
fabric.
2.1.1.1. Pod
A pod is a logical boundary of functionality for the SDDC platform. While each pod usually spans one
rack, it is possible to aggregate multiple pods into a single rack in smaller setups. For both small and
large setups, homogeneity and easy replication are important.
Different pods of the same type can provide different characteristics for varying requirements. For
example, one compute pod could use full hardware redundancy for each component (power supply
through memory chips) for increased availability. At the same time, another compute pod in the same
setup could use low-cost hardware without any hardware redundancy. With these variations, the
architecture can cater to the different workload requirements in the SDDC.
One of the guiding principles for such deployments is that VLANs are not spanned beyond a single
pod by the network virtualization layer. Although this VLAN restriction appears to be a simple
requirement, it has widespread impact on how a physical switching infrastructure can be built and on
how it scales.
2.1.1.2. Pod Types
The SDDC differentiates between the following types of pods:

Management pod

Shared Edge and Compute pod

Compute pod

Storage pod
© 2016 VMware, Inc. All rights reserved.
Page 11 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 2. Pods in the SDDC
2.1.1.3. Management Pod
The management pod runs the virtual machines that manage the SDDC. These virtual machines host
vCenter Server, NSX Manager, NSX Controller, vRealize Operations Management, vRealize Log
Insight, vRealize Automation, and other shared management components. Different types of
management pods can support different SLAs. Because the management pod hosts critical
infrastructure, you should consider implementing a basic level of hardware redundancy for this pod.
Management pod components must not have tenant-specific addressing.
2.1.1.4. Shared Edge and Compute Pod
The shared edge and compute pod runs the required NSX services to enable north-south routing
between the SDDC and the external network, and east-west routing inside the SDDC. This shared
pod also hosts the SDDC tenant virtual machines (sometimes referred to as workloads or payloads).
As the SDDC grows, additional compute-only pods can be added to support a mix of different types of
workloads for different types of Service Level Agreements (SLAs).
2.1.1.5. Compute Pod
Compute pods host the SDDC tenant virtual machines (sometimes referred to as workloads or
payloads). An SDDC can mix different types of compute pods and provide separate compute pools for
different types of SLAs.
2.1.1.6. Storage Pod
Storage pods provide network-accessible storage using NFS or iSCSI. Different types of storage pods
can provide different levels of SLA, ranging from just a bunch of disks (JBODs) using IDE drives with
minimal to no redundancy, to fully redundant enterprise-class storage arrays. For bandwidth-intense
IP-based storage, the bandwidth of these pods can scale dynamically.
This design does not consider Fibre Channel or Fibre Channel over Ethernet (FCoE) based storage
technology. Instead, this design focuses on technologies that can use the central Layer 3 network
fabric for primary connectivity.
2.1.1.7. Pod to Rack Mapping
Pods are not mapped one-to-one to 19" data center racks. While a pod is an atomic unit of a
repeatable building block, a rack is merely a unit of size. Because pods can have different sizes, how
pods are mapped to 19" data center racks depends on the use case.
© 2016 VMware, Inc. All rights reserved.
Page 12 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

One Pod in One Rack. One pod can occupy exactly one rack. This is typically the case for
compute pods.

Multiple Pods in One Rack. Two or more pods can occupy a single rack, for example, one
management pod and one shared edge and compute pod can be deployed to a single rack.

Single Pod Across Multiple Racks. A single pod can stretch across multiple adjacent racks. For
example, a storage pod with filer heads and disk shelves can span more than one rack.
2.1.2 Physical Network Architecture
The physical network architecture is tightly coupled with the pod-and-core architecture, and uses a
Layer 3 leaf-and-spine network instead of the more traditional 3-tier data center design.
2.1.2.1. Leaf-and-Spine Network Architecture
The design uses leaf switches and spine switches.

A leaf switch is typically located inside a rack and provides network access to the servers inside
that rack, it is also referred to as a Top of Rack (ToR) switch.

A spine switch is in the spine layer and provides connectivity between racks. Links between spine
switches are typically not required. If a link failure occurs between a spine switch and a leaf
switch, the routing protocol ensures that no traffic for the affected rack is sent to the spine switch
that has lost connectivity to that rack.
Figure 3. Leaf-and-Spine Physical Network Design
Ports that face the servers inside a rack should have a minimal configuration, shown in the following
high-level physical and logical representation of the leaf node.
Note
Each leaf node has identical VLAN configuration with unique /24 subnets assigned to each
VLAN.
© 2016 VMware, Inc. All rights reserved.
Page 13 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 4. High-level Physical and Logical Representation of a Leaf Node
2.1.2.2. Network Transport
You can implement the physical layer switch fabric for an SDDC by offering Layer 2 transport services
or Layer 3 transport services to all components. For a scalable and vendor-neutral data center
network, use Layer 3 transport.
2.1.2.3. Benefits and Drawbacks for Layer 2 Transport
In a design that uses Layer 2 transport, leaf switches and spine switches form a switched fabric,
effectively acting like one large switch. Using modern data center switching fabric products such as
Cisco FabricPath, you can build highly scalable Layer 2 multipath networks without the Spanning Tree
Protocol (STP). Such networks are particularly suitable for large virtualization deployments, private
clouds, and high-performance computing (HPC) environments.
Using Layer 2 routing has benefits and drawbacks.

The benefit of this approach is more design freedom. You can span VLANs, which is useful for
vSphere vMotion or vSphere Fault Tolerance (FT).

The drawback is that the size of such a deployment is limited because the fabric elements have to
share a limited number of VLANs. In addition, you have to rely on a specialized data center
switching fabric product from a single vendor because these products are not designed for
interoperability between vendors.
2.1.2.4. Benefits and Drawbacks for Layer 3 Transport
A design with Layer 3 transport requires these considerations.

Layer 2 connectivity is limited to within the data center rack, up to the leaf switch.

The leaf switch terminates each VLAN and provides default gateway functionality, that is, it has a
switch virtual interface (SVI) for each VLAN.

Uplinks from the leaf switch to the spine layer are routed point-to-point links. VLAN trunking on
the uplinks is not allowed.
© 2016 VMware, Inc. All rights reserved.
Page 14 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

A dynamic routing protocol—for example OSPF, ISIS, or iBGP—connects the leaf switches and
spine switches. Each leaf switch in the rack advertises a small set of prefixes, typically one per
VLAN or subnet. In turn, the leaf switch calculates equal cost paths to the prefixes it received from
other leaf switches
Using Layer 3 routing has benefits and drawbacks.

The benefit is that you can chose from a wide array of Layer 3 capable switch products for the
physical switching fabric. You can mix switches from different vendors due to general
interoperability between implementation of OSPF, ISIS or iBGP. This approach is usually more
cost effective because it uses only basic functionality of the physical switches.

The drawbacks are some design restrictions because VLANs are restricted to a single rack. This
affects vSphere vMotion, vSphere Fault Tolerance, and storage networks.
2.1.2.5. Infrastructure Network Architecture
One of the key goals of network virtualization is to provide a virtual-to-physical network abstraction.
For this, the physical fabric must provide a robust IP transport with the following characteristics.

Simplicity

Scalability

High bandwidth

Fault-tolerant transport

Support for different levels of quality of service (QoS)
Simplicity
Configuration of the switches inside a data center must be simple. General or global configuration
such as AAA, SNMP, syslog, NTP, and others should be replicated line by line, independent of the
position of the switches. A central management capability to configure all switches at once is an
alternative.
Scalability
Scalability factors include but are not limited to the following.

Number of racks supported in a fabric.

Amount of bandwidth between any two racks in a data center.

Number of paths that a leaf switch can select from when communicating with another rack.
The total number of ports available across all spine switches and the oversubscription that is
acceptable determine the number of racks supported in a fabric. Different racks might host different
types of infrastructure, which results in different bandwidth requirements.

Racks with storage systems might attract or source more traffic than other racks.

Compute racks, such as racks hosting hypervisors with workloads or virtual machines, might have
different bandwidth requirements than shared edge and compute racks, which provide
connectivity to the outside world.
Link speed and the number of links vary to satisfy different bandwidth demands. You can vary them
for each rack without sacrificing other aspects of the leaf-and-spine architecture.
© 2016 VMware, Inc. All rights reserved.
Page 15 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 5. Pod Network Design
The number of links to the spine switches dictates how many paths for traffic from this rack to another
rack are available. Because the number of hops between any two racks is consistent, the architecture
can utilize equal-cost multipathing (ECMP). Assuming traffic sourced by the servers carries a TCP or
UDP header, traffic spray can occur on a per-flow basis.
High Bandwidth
In leaf-and-spine topologies, oversubscription typically occurs at the leaf switch.
Oversubscription is equal to the total amount of bandwidth available to all servers connected to a leaf
switch divided by the aggregate amount of uplink bandwidth.
oversubscription = total bandwidth / aggregate uplink bandwidth
For example, 20 servers with one 10 Gigabit Ethernet (10 GbE) port each create up to 200 Gbps of
bandwidth. In an environment with eight 10 GbE uplinks to the spine—a total of 80 Gbps—a 2.5:1
oversubscription results shown in the Oversubscription in the Leaf Layer illustration.
2.5 (oversubscription) = 200 (total) / 80 (total uplink)
© 2016 VMware, Inc. All rights reserved.
Page 16 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 6. Oversubscription in the Leaf Layer
You can make more or less bandwidth available to a rack by provisioning more or fewer uplinks. That
means you can change the available bandwidth on a per-rack basis.
Note
The number of uplinks from a leaf switch to each spine switch must be the same to avoid
hotspots.
For example, if a leaf switch has two uplinks to spine switch A and only one uplink to spine switches
B, C and D, more traffic is sent to the leaf switch via spine switch A, which might create a hotspot.
Fault Tolerance
The larger the environment, the more switches make up the overall fabric and the greater the
possibility for one component of the data center switching fabric to fail. A resilient fabric can sustain
individual link or box failures without widespread impact.
© 2016 VMware, Inc. All rights reserved.
Page 17 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 7. Compensation for a Link Failure
For example, if one of the spine switches fails, traffic between racks continues to be routed across the
remaining spine switches in a Layer 3 fabric. The routing protocol ensures that only available paths
are chosen. Installing more than two spine switches reduces the impact of a spine switch failure.
Multipathing-capable fabrics handle box or link failures, reducing the need for manual network
maintenance and operations. If a software upgrade of a fabric switch becomes necessary, the
administrator can take the node out of service gracefully by changing routing protocol metrics, which
will quickly drain network traffic from that switch, freeing the switch for maintenance.
Depending on the width of the spine, that is, how many switches are in the aggregation or spine layer,
the additional load that the remaining switches must carry is not as significant as if there were only
two switches in the aggregation layer. For example, in an environment with four spine switches, a
failure of a single spine switch only reduces the available capacity by 25%.
Quality of Service Differentiation
Virtualized environments must carry different types of traffic, including tenant, storage and
management traffic, across the switching infrastructure. Each traffic type has different characteristics
and makes different demands on the physical switching infrastructure.

Management traffic, although typically low in volume, is critical for controlling physical and virtual
network state.

IP storage traffic is typically high in volume and generally stays within a data center.
For virtualized environments, the hypervisor sets the QoS values for the different traffic types. The
physical switching infrastructure has to trust the values set by the hypervisor. No reclassification is
necessary at the server-facing port of a leaf switch. If there is a congestion point in the physical
switching infrastructure, the QoS values determine how the physical network sequences, prioritizes,
or potentially drops traffic.
© 2016 VMware, Inc. All rights reserved.
Page 18 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 8. Quality of Service (Differentiated Services) Trust Point
Two types of QoS configuration are supported in the physical switching infrastructure:

Layer 2 QoS, also called class of service

Layer 3 QoS, also called DSCP marking.
A vSphere Distributed Switch supports both class of service and DSCP marking. Users can mark the
traffic based on the traffic type or packet classification. When the virtual machines are connected to
the VXLAN-based logical switches or networks, the QoS values from the internal packet headers are
copied to the VXLAN-encapsulated header. This enables the external physical network to prioritize
the traffic based on the tags on the external header.
2.1.2.6. Server Interfaces (NICs)
If the server has more than one server interface (NIC) of the same speed, use two as uplinks with
VLANs trunked to the interfaces.
The vSphere Distributed Switch supports many different NIC Teaming options. Load-based NIC
teaming supports optimal use of available bandwidth and supports redundancy in case of a link
failure. Use two 10 GbE connections per server along with a pair of leaf switches. 801.Q trunks are
used for carrying a small number of VLANs; for example, management, storage, VXLAN, vSphere
Replication, and VMware vSphere vMotion traffic.
2.1.3 Availability Zones and Regions
In an SDDC, availability zones are collections of infrastructure components. Regions support disaster
recovery solutions and allow you to place workloads closer to your customers. Typically, multiple
availability zones form a single region.
© 2016 VMware, Inc. All rights reserved.
Page 19 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
This VMware Validated Design uses two regions, but uses only one availability zone in each region.
The following diagram shows how the design could be expanded to include multiple availability
zones.
Figure 9. Availability Zones and Regions
2.1.3.1. Availability Zones
Each availability zone is isolated from other availability zones to stop the propagation of failure or
outage across zone boundaries. Together, multiple availability zones provide continuous availability
through redundancy, helping to avoid outages and improve SLAs. An outage that is caused by
external factors (power, cooling, physical integrity) affects only one zone, those factors most likely
don't lead to an outage in other zones except in the case of major disasters.
Each availability zone runs on its own physically distinct, independent infrastructure, and is
engineered to be highly reliable. Each zone should have independent power, cooling, network and
security. Common points of failures within a physical data center, like generators and cooling
equipment, should not be shared across availability zones. Additionally, these zones should be
physically separate; so that even uncommon disasters affect only a single availability zone.
Availability zones are usually either two distinct data centers within metro distance (latency in the
single digit range) or two safety/fire sectors (aka data halls) within the same large scale data center.
Multiple availability zones (usually two) belong to a single region. The physical distance between
availability zones can be up to approximately 50 kilometer or 30 miles, therefore offering low singledigit latency and large bandwidth - via dark fiber - between the zones. This architecture allows the
SDDC equipment across the availability zone to operate in an active/active manner as a single virtual
data center or region.
You can operate workloads across multiple availability zones within the same region as if they were
part of a single virtual data center. This supports an architecture with very high availability that is
suitable for mission critical applications. When the distance between two locations of equipment
becomes too large, these locations can no longer function as two availability zones within the same
region and need to be treated as separate regions.
2.1.3.2. Regions
Multiple regions support placing workloads closer to your customers, for example, by operating one
region on the US east coast and one region on the US west coast, or operating a region in Europe
and another region in the US. Regions are helpful in many ways.

Regions can support disaster recovery solutions: One region can be the primary site and another
region can be the recovery site.

You can use multiple regions to address data privacy laws and restrictions in certain countries by
keeping tenant data within a region in the same country.
The distance between regions can be rather large. This design uses two regions, one region is
assumed to be in San Francisco (SFO), the other region is assumed to be in Los Angeles (LAX).
© 2016 VMware, Inc. All rights reserved.
Page 20 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
2.2
Virtual Infrastructure Architecture
2.2.1 Infrastructure Architecture
The virtual infrastructure is the foundation of an operational SDDC. Within the virtual infrastructure
layer, access to the physical underlying infrastructure is controlled and allocated to the management
and tenant workloads. The virtual infrastructure layer consists primarily of the physical hosts'
hypervisors and the control of these hypervisors. The management workloads consist of elements in
the virtual management layer itself, along with elements in the cloud management layer and in the
service management, business continuity, and security areas.
Figure 10. Virtual Infrastructure Layer in the SDDC
2.2.2 Virtual Infrastructure Overview
The SDDC virtual infrastructure consists of two regions. Each region includes a management pod and
a shared edge and compute pod.
© 2016 VMware, Inc. All rights reserved.
Page 21 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 11. SDDC Logical Design
2.2.2.1. Management Pod
Management pods run the virtual machines that manage the SDDC. These virtual machines host
vCenter Server, NSX Manager, NSX Controller, vRealize Operations, vRealize Log Insight, vRealize
Automation, Site Recovery Manager and other shared management components. All management,
monitoring, and infrastructure services are provisioned to a vCenter Server High Availability cluster
which provides high availability for these critical services. Permissions on the management cluster
limit access to only administrators. This limitation protects the virtual machines that are running the
management, monitoring, and infrastructure services.
2.2.2.2. Shared Edge and Compute Pod
The shared edge and compute pod runs the required NSX services to enable north-south routing
between the SDDC and the external network and east-west routing inside the SDDC. This pod also
hosts the SDDC tenant virtual machines (sometimes referred to as workloads or payloads). As the
SDDC grows additional compute-only pods can be added to support a mix of different types of
workloads for different types of SLAs.
© 2016 VMware, Inc. All rights reserved.
Page 22 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
2.2.3 Network Virtualization Architecture
2.2.3.1. NSX for vSphere Components
VMware NSX for vSphere, the network virtualization platform, is a key solution in the SDDC
architecture. The NSX for vSphere platform consists of several components that are relevant to the
network virtualization design.
NSX for vSphere Platform
NSX for vSphere creates a network virtualization layer. All virtual networks are created on top of this
layer, which is an abstraction between the physical and virtual networks. Several components are
required to create this network virtualization layer.

vCenter Server

NSX Manager

NSX Controller

NSX Virtual Switch

NSX for vSphere API
These components are separated into different planes to create communications boundaries and
provide isolation of workload data from system control messages.

Data plane. Workload data is contained wholly within the data plane. NSX logical switches
segregate unrelated workload data. The data is carried over designated transport networks in the
physical network. The NSX Virtual Switch, distributed routing, and the distributed firewall are also
implemented in the data plane.

Control plane. Network virtualization control messages are located in the control plane. Control
plane communication should be carried on secure physical networks (VLANs) that are isolated
from the transport networks that are used for the data plane. Control messages are used to set up
networking attributes on NSX Virtual Switch instances, as well as to configure and manage
disaster recovery and distributed firewall components on each ESXi host.

Management plane. The network virtualization orchestration happens in the management plane.
In this layer, cloud management platforms such as VMware vRealize® Automation™ can request,
consume, and destroy networking resources for virtual workloads. Communication is directed
from the cloud management platform to vCenter Server to create and manage virtual machines,
and to NSX Manager to consume networking resources.
The different planes are connected through the APIs, which include REST, VMware vSphere, and
VMware VIX APIs. The API used depends on the component being controlled.
NSX Manager
NSX Manager provides the centralized management plane for the NSX for vSphere architecture and
has a one-to-one mapping with vCenter Server for workloads. NSX Manager performs the following
functions.

Provides a single point of configuration and the REST API entry points in a vSphere environment
configured for NSX for vSphere.

Responsible for deploying NSX Controller clusters, NSX Edge distributed routers, and NSX Edge
services gateways (as appliances in OVF format), guest introspection services, and so on.

Responsible for preparing ESXi hosts for NSX for vSphere by installing VXLAN, distributed
routing, and firewall kernel modules, as well as the User World Agent (UWA).

Communicates with NSX Controller clusters through REST and with the ESXi hosts through the
VMware vFabric® RabbitMQ message bus. This internal message bus is specific to NSX for
vSphere and does not require setup of additional services.

Generates certificates for the NSX Controller nodes and ESXi hosts to secure control plane
communications with mutual authentication.
© 2016 VMware, Inc. All rights reserved.
Page 23 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
VMware NSX allows linking multiple vCenter and VMware NSX deployments, and manage them from
a single NSX Manager that is designated as primary. Such a linked environment includes both an
NSX Manager primary instance, and one or more secondary instances.

The primary NSX Manager instance is linked to the primary vCenter Server instance and allows
the creation and management of universal logical switches, universal (distributed) logical routers
and universal firewall rules.

Each secondary NSX Manager instance can manage networking services that are local to itself.
Up to seven secondary NSX Manager instances can be associated with the primary NSX
Manager in a linked environment. You can configure network services on all NSX Manager
instances from one central location.
Note
A linked environment still requires one vCenter Server instance for each NSX Manager
instance.
To manage all NSX Manager instances from the primary NSX Manager in a Cross-vCenter VMware
NSX deployment, the vCenter Server instances must be connected with Platform Services Controller
nodes in Enhanced Linked Mode.
Figure 12. NSX for vSphere Architecture
Control Plane
© 2016 VMware, Inc. All rights reserved.
Page 24 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
VMware NSX supports three different replication modes to provide multiple destination
communication.

Multicast Mode. When multicast replication mode is selected for a logical switch, VMware NSX
relies on the Layer 2 and Layer 3 multicast capability of the physical network to ensure VXLAN
encapsulated multi-destination traffic is sent to all the VXLAN tunnel end points (VTEPs). The
control plane uses multicast IP addresses on the physical network in this mode.

Unicast Mode. In unicast mode, the control plane is managed by the NSX Controller instances
and all replication is done locally on the host. No multicast IP addresses or physical network
configurations are required. This mode is well suited for smaller deployments.

Hybrid Mode. Hybrid mode is an optimized version of unicast mode, where local traffic replication
for the subnet is offloaded to the physical network. This mode requires IGMP snooping on the first
hop switch, and an IGMP querier must be available. Protocol-independent multicast (PIM) is not
required.
NSX Controller
The NSX Controller cluster is the control plane component, and is responsible for managing the
switching and routing modules in the hypervisors. An NSX Controller node performs the following
functions.

Provides the control plane to distribute VXLAN and logical routing information to ESXi hosts.

Clusters nodes for scale-out and high availability.

Slices network information across nodes in a cluster for redundancy purposes.

Eliminates the need for multicast support from the physical network infrastructure.

Provides ARP-suppression of broadcast traffic in VXLAN networks.
NSX for vSphere control plane communication occurs over the management network. Network
information from the ESXi hosts and the distributed logical router control VMs is reported to NSX
Controller instances through the UWA. The NSX Controller command line supports retrieval of
VXLAN and logical routing network state information.
Data Plane
The NSX data plane consists of the NSX vSwitch, which is based on the vSphere Distributed Switch
(VDS) and includes additional components. These components include kernel modules, which run
within the ESXi kernel and provide services such as the distributed logical router (DLR) and
distributed firewall (DFW). The NSX kernel modules also enable Virtual Extensible LAN (VXLAN)
capabilities.
The NSX vSwitch abstracts the physical network and provides access-level switching in the
hypervisor. It is central to network virtualization because it enables logical networks that
are independent of physical constructs such as a VLAN. The NSX vSwitch provides multiple benefits.

Three types of overlay networking capabilities:
o
Creation of a flexible logical Layer 2 overlay over existing IP networks on existing
physical infrastructure, without the need to rearchitect the data center networks.
o
Support for east/west and north/south communication while maintaining isolation
between tenants.
o
Support for application workloads and virtual machines that operate as if they were connected
to a physical Layer 2 network.

Support for VXLAN and centralized network configuration.

A comprehensive toolkit for traffic management, monitoring and troubleshooting within a virtual
network which includes Port Mirroring, NetFlow/IPFIX, configuration backup and restore, network
health check, and Quality of Service (QoS).
In addition to the NSX vSwitch, the data plane also includes gateway devices, which can provide
Layer 2 bridging from the logical networking space (VXLAN) to the physical network (VLAN). The
© 2016 VMware, Inc. All rights reserved.
Page 25 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
gateway device is typically an NSX Edge Gateway device. NSX Edge Gateway devices offer Layer 2,
Layer 3, perimeter firewall, load-balancing, Virtual Private Network (VPN), and Dynamic Host Control
Protocol (DHCP) services.
Consumption Plane
In the consumption plane, different users of NSX for vSphere can access and manage services in
different ways:

NSX administrators can manage the NSX environment from the vSphere Web Client.

End-users can consume the network virtualization capabilities of NSX for vSphere through the
Cloud Management Platform (CMP) when deploying applications.
2.2.3.2. Network Virtualization Services
Network virtualization services include logical switches, logical routers, logical firewalls, and other
components of NSX for vSphere.
Logical Switches
NSX for vSphere logical switches create logically abstracted segments to which tenant virtual
machines can connect. A single logical switch is mapped to a unique VXLAN segment ID and is
distributed across the ESXi hypervisors within a transport zone. This allows line-rate switching in the
hypervisor without creating constraints of VLAN sprawl or spanning tree issues.
Universal Distributed Logical Router
The NSX for vSphere Universal Distributed Logical Router is optimized for forwarding in the
virtualized space (between VMs, on VXLAN- or VLAN-backed port groups). Features include:

High performance, low overhead first hop routing.

Scaling the number of hosts.

Support for up to 1,000 logical interfaces (LIFs) on each distributed logical router.
The Universal Distributed Logical Router is installed in the kernel of every ESXi host, as such it
requires a VM to provide the control plane. The universal distributed logical router Control VM is the
control plane component of the routing process, providing communication between NSX Manager and
NSX Controller cluster through the User World Agent. NSX Manager sends logical interface
information to the Control VM and NSX Controller cluster, and the Control VM sends routing updates
to the NSX Controller cluster.
© 2016 VMware, Inc. All rights reserved.
Page 26 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 13. NSX for vSphere Universal Distributed Logical Router
Designated Instance
The designated instance is responsible for resolving ARP on a VLAN LIF. There is one designated
instance per VLAN LIF. The selection of an ESXi host as a designated instance is performed
automatically by the NSX Controller cluster and that information is pushed to all other hosts. Any ARP
requests sent by the distributed logical router on the same subnet are handled by the same host. In
case of host failure, the controller selects a new host as the designated instance and makes that
information available to other hosts.
User World Agent
User World Agent (UWA) is a TCP and SSL client that enables communication between the ESXi
hosts and NSX Controller nodes, and the retrieval of information from NSX Manager through
interaction with the message bus agent.
Edge Service Gateways
While the Universal Logical Router provides VM to VM or east-west routing, the NSX Edge Service
Gateway provides north-south connectivity, by peering with upstream Top of Rack switches, thereby
enabling tenants to access public networks.
Logical Firewall
NSX for vSphere Logical Firewall provides security mechanisms for dynamic virtual data centers.

The Distributed Firewall allows you to segment virtual data center entities like virtual machines.
Segmentation can be based on VM names and attributes, user identity, vCenter objects like data
centers, and hosts, or can be based on traditional networking attributes like IP addresses, port
groups, and so on.

The Edge Firewall component helps you meet key perimeter security requirements, such as
building DMZs based on IP/VLAN constructs, tenant-to-tenant isolation in multi-tenant virtual data
centers, Network Address Translation (NAT), partner (extranet) VPNs, and user-based SSL
VPNs.
© 2016 VMware, Inc. All rights reserved.
Page 27 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
The Flow Monitoring feature displays network activity between virtual machines at the application
protocol level. You can use this information to audit network traffic, define and refine firewall policies,
and identify threats to your network.
Logical Virtual Private Networks (VPNs)
SSL VPN-Plus allows remote users to access private corporate applications. IPSec VPN offers siteto-site connectivity between an NSX Edge instance and remote sites. L2 VPN allows you to extend
your datacenter by allowing virtual machines to retain network connectivity across geographical
boundaries.
Logical Load Balancer
The NSX Edge load balancer enables network traffic to follow multiple paths to a specific destination.
It distributes incoming service requests evenly among multiple servers in such a way that the load
distribution is transparent to users. Load balancing thus helps in achieving optimal resource
utilization, maximizing throughput, minimizing response time, and avoiding overload. NSX Edge
provides load balancing up to Layer 7.
Service Composer
Service Composer helps you provision and assign network and security services to applications in a
virtual infrastructure. You map these services to a security group, and the services are applied to the
virtual machines in the security group.
Data Security provides visibility into sensitive data that are stored within your organization's virtualized
and cloud environments. Based on the violations that are reported by the NSX for vSphere Data
Security component, NSX security or enterprise administrators can ensure that sensitive data is
adequately protected and assess compliance with regulations around the world.
NSX for vSphere Extensibility
VMware partners integrate their solutions with the NSX for vSphere platform to enable an integrated
experience across the entire SDDC. Data center operators can provision complex, multi-tier virtual
networks in seconds, independent of the underlying network topology or components.
2.2.4 Logging Architecture
vRealize Log Insight provides real-time log management and log analysis with machine learningbased intelligent grouping, high-performance searching, and troubleshooting across physical, virtual,
and cloud environments.
2.2.4.1. Overview
vRealize Log Insight collects data from ESXi hosts using the syslog protocol. It connects to vCenter
Server to collect events, tasks, and alarms data, and integrates with vRealize Operations Manager to
send notification events and enable launch in context. It also functions as a collection and analysis
point for any system capable of sending syslog data. In addition to syslog data an ingestion agent can
be installed on Linux or Windows servers to collect logs. This agent approach is especially useful for
custom logs and operating systems that don't natively support the syslog protocol, such as Windows.
2.2.4.2. Installation Models
You can deploy vRealize Log Insight as a virtual appliance in one of the following configurations:

Standalone node

Highly available cluster of one master and at least two worker nodes using an internal load
balancer (ILB)
The compute and storage resources of the vRealize Log Insight instances for scale-up.
© 2016 VMware, Inc. All rights reserved.
Page 28 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
2.2.4.3. Cluster Nodes
For high availability and scalability, you can deploy several vRealize Log Insight instances in a
cluster where they can have either of the following roles:

Master Node. Required initial node in the cluster. The master node is responsible for queries and
log ingestion. The Web user interface of the master node serves as the single pane of glass for
the cluster. All queries against data are directed to the master, which in turn queries the workers
as appropriate.

Worker Node. Enables scale-out in larger environments. A worker node is responsible for
ingestion of logs. A worker node stores logs locally. If a worker node is down, the logs on that
worker becomes unavailable.
You need at least two worker nodes to form a cluster with the master node.

Integrated Load Balancer (ILB). Provides high availability (HA). The ILB runs on one of the
cluster nodes. If the node that hosts the ILB Virtual IP (VIP) address stops responding, the VIP
address is failed over to another node in the cluster.
2.2.4.4. Architecture of a Cluster
The architecture of vRealize Log Insight enables several channels for HA collection of log messages.
Figure 14. Cluster Architecture of vRealize Log Insight
vRealize Log Insight clients connect to ILB VIP address and use the Web user interface and
ingestion (via Syslog or the Ingestion API) to send logs to vRealize Log Insight.
By default, the vRealize Log Insight Solution collects data from vCenter Server systems and ESXi
hosts. For forwarding logs from NSX for vSphere, and vRealize Automation, use content packs which
contain extensions or provide integration with other systems in the SDDC.
© 2016 VMware, Inc. All rights reserved.
Page 29 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
2.2.4.5. Integration with vRealize Operations Manager
The integration with vRealize Operations Manager provides a single pane of glass for monitoring the
SDDC. vRealize Log Insight sends notification events to vRealize Operations Manager. You can also
launch vRealize Log Insight from the vRealize Operations Manager Web user interface.
2.2.4.6. Archiving
vRealize Log Insight supports data archiving on NFS shared storage that each vRealize Log Insight
node can access.
2.2.4.7. Backup
You back up each vRealize Log Insight cluster locally by using traditional virtual machine backup
solutions, such as a vSphere Storage APIs for Data Protection (VADP) compatible backup software
like vSphere Data Protection.
2.2.4.8. Multi-Region vRealize Log Insight Deployment
The scope of the SDDC design covers multiple regions. Using vRealize Log Insight in a multi-region
design can provide a syslog infrastructure in all regions of the SDDC. Using vRealize Log
Insight across multiple regions requires deploying a cluster in each region. vRealize Log Insight
supports event forwarding to other vRealize Log Insight deployments across regions in the SDDC.
Implementing failover by using vSphere Replication or disaster recovery by using Site Recovery
Manager is not necessary. The event forwarding feature adds tags to log message that identify the
source region and event filtering prevents looping messages between the regions.
2.2.5 Operations Management Architecture
vRealize Operations Manager tracks and analyzes the operation of multiple data sources within the
Software-Defined Data Center (SDDC) by using specialized analytics algorithms. These algorithms
help vRealize Operations Manager to learn and predicts the behavior of every object it monitors.
Users access this information by using views, reports, and dashboards.
2.2.5.1. Installation Models
vRealize Operations Manager is available in two different deployment models
- a preconfigured virtual appliance, or a Windows or Linux installable package. Select the installation
method according to the following considerations:

When you use the vRealize Operations Manager virtual appliance, you deploy the OVF file of the
virtual appliance once for each cluster node. You access the product to set up cluster nodes
according to their role, and log in to configure the installation.
Use virtual appliance deployment to easily create vRealize Operations Manager nodes with predefined identical size.

When you use the Windows or Linux installable package, you run the vRealize Operations
Manager installation on each cluster node. You access the product to set up cluster nodes
according to their role, and log in to configure the installation.
Use installable package deployment to create vRealize Operations Manager node with custom
identical size.
2.2.5.2. Architecture
vRealize Operations Manager contains functional elements that collaborate for data analysis and
storage, and support creating clusters of nodes with different roles.
© 2016 VMware, Inc. All rights reserved.
Page 30 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 15. vRealize Operations Manager Architecture
2.2.5.3. Types of Nodes and Clusters
For high availability and scalability, you can deploy several vRealize Operations Manager instances in
a cluster where they can have either of the following roles:

Master Node. Required initial node in the cluster. In large-scale environments the master node
manages all other nodes. In small-scale environments, the master node is the single standalone
vRealize Operations Manager node.

Master Replica Node. (Optional) Enables high availability of the master node.

Data Node. Enables scale-out of vRealize Operations Manager in larger environments. Data
nodes have adapters installed to perform collection and analysis. Data nodes also host vRealize
Operations Manager management packs.
Larger deployments usually include adapters only on data nodes, not on the master node or
replica node

Remote Collector Node. In distributed deployments, enables navigation through firewalls,
interfaces with a remote data source, reduces bandwidth across regions, or reduces the load on
the vRealize Operations Manager analytics cluster. Remote collector nodes only gather objects
for the inventory and forward collected data to the data nodes. Remote collector nodes do not
store data or perform analysis. In addition, you can install them on a different operating system
than the rest of the cluster nodes.
The master and master replica nodes are data nodes with extended capabilities.
vRealize Operations Manager can form two type of clusters according to the nodes that participate in
a cluster:

Analytics clusters. Tracks, analyzes, and predicts the operation of monitored systems. Consists
of a master node, data nodes, and optionally of a master replica node.

Remote collectors cluster. Only collects diagnostics data without storage or analysis. Consists
only of remote collector nodes.
© 2016 VMware, Inc. All rights reserved.
Page 31 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
2.2.5.4. Application Functional Components
The functional components of a vRealize Operations Manager instance interact to provide analysis of
diagnostics data from the data center and visualize the result in the Web user interface.
Table 1. vRealize Operations Manager Logical Node Architecture
Architecture Component Diagram
Description

Admin / Product UI server. The UI server is a
Web application that serves as both user and
administration interface.

REST API / Collector. The Collector collects
data from all components in the data center.

Controller. The Controller handles the data flow
the UI server, Collector, and the analytics
engine.

Analytics. The Analytics engine creates all
associations and correlations between various
data sets, handles all super metric calculations,
performs all capacity planning functions, and is
responsible for triggering alerts.

Persistence. The persistence layer handles the
read and write operations on the underlying
databases across all nodes.

FSDB. The File System Database (FSDB)
stores collected metrics in raw format. FSDB is
available in all the nodes.

xDB (HIS). The xDB stores data from the
Historical Inventory Service (HIS). This
component is available only on the master and
master replica nodes.

Global xDB. The Global xDB stores user
preferences, alerts, and alarms, and
customization that is related to the vRealize
Operations Manager. This component is
available only on the master and master replica
nodes
2.2.5.5. Management Packs
Management packs contain extensions and third-party integration software. They add dashboards,
alerts definitions, policies, reports, and other content to the inventory of vRealize Operations
Manager. You can learn more details about and download management packs from the VMware
Solutions Exchange.
2.2.5.6. Multi-Region vRealize Operations Manager Deployment
The scope of the SDDC design covers multiple regions. Using vRealize Operations Manager across
multiple regions requires deploying an analytics cluster that is protected by Site Recovery Manager,
and deploying remote collectors in each region.
© 2016 VMware, Inc. All rights reserved.
Page 32 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Detailed Design
3.
This Software-Defined Data Center (SDDC) detailed design consists of the following main sections:

Physical Infrastructure Design

Virtual Infrastructure Design

Cloud Management Platform Design

Operations Infrastructure Design
Each section is divided into subsections that include detailed discussion, diagrams, and design
decisions with justifications.
The Physical Infrastructure Design section focuses on the three main pillars of any data center,
compute, storage and network. In this section you find information about availability zones and
regions. The section also provides details on the rack and pod configuration, and on physical hosts
and the associated storage and network configurations.
In the Virtual Infrastructure Design section, you find details on the core virtualization software
configuration. This section has information on the ESXi hypervisor, vCenter Server, the virtual
network design including VMware NSX, and on software-defined storage for VMware Virtual SAN.
This section also includes details on business continuity (backup and restore) and on disaster
recovery.
The Cloud Management Platform Design section contains information on the consumption and
orchestration layer of the SDDC stack, which uses vRealize Automation and vRealize Orchestrator. IT
organizations can use the fully distributed and scalable architecture to streamline their provisioning
and decommissioning operations.
The Operations Infrastructure Design section explains how to architect, install, and configure vRealize
Operations Manager and vRealize Log Insight. You learn how to ensure that service management
within the SDDC is comprehensive. This section ties directly into the Operational Guidance section.
3.1
Physical Infrastructure Design
The physical infrastructure design includes details on decisions covering availability zones and
regions and pod layout within data center racks. This section then goes on to detail decisions related
to server, networking, and storage hardware.
Figure 16. Physical Infrastructure Design
© 2016 VMware, Inc. All rights reserved.
Page 33 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
3.1.1 Physical Design Fundamentals
The SDDC physical layer design is based on a pod architecture. The physical data center elements
include racks, servers, network elements, and storage arrays.
Figure 17. Physical Layer within the SDDC
3.1.1.1. Availability Zones and Regions
Availability zones and regions are used for different purposes.

Availability zones. An availability zone is the fault domain of the SDDC. Multiple availability
zones can provide continuous availability of an SDDC, minimize unavailability of services and
improve SLAs.

Regions. Regions provide disaster recovery across different SDDC instances. This design uses
two regions. Each region is a separate SDDC instance. The regions have a similar physical layer
design and virtual infrastructure design but different naming. For information on exceptions to this
design, see the Business Continuity / Disaster Recovery Design chapter.
Note
This design leverages a single availability zone for a one region deployment, and a single
availability zone in each region in the case of a two region deployment.
The design uses the following regions. The region identifier uses United Nations Code for Trade and
Transport Locations (UN/LOCODE) along with a numeric instance ID.
Table 2. Regions
Region
Region
Identifier
Region-specific Domain
Name
Region Description
A
SFO01
sfo01.rainpole.local
San Francisco, CA, USA based data
center
B
LAX01
lax01.rainpole.local
Los Angeles, CA, USA based data
center
© 2016 VMware, Inc. All rights reserved.
Page 34 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 3. Availability Zones and Regions Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCPHY-001
Per region, a single
availability zone that
can support all
SDDC management
components is
deployed.
A single availability zone can
support all SDDC management
and compute components for a
region. You can later add
another availability zone to
extend and scale the
management and compute
capabilities of the SDDC.
Results in limited
redundancy of the overall
solution. The single
availability zone can
become a single point of
failure and prevent highavailability design
solutions.
SDDCPHY-002
Use two regions.
Supports the technical
requirement of multi-region
failover capability as outlined in
the design objectives.
Having multiple regions
will require an increased
solution footprint and
associated costs.
3.1.1.2. Pods and Racks
The SDDC functionality is split across multiple pods. Each pod can occupy one rack or multiple racks.
The total number of pods for each pod type depends on scalability needs.
Figure 18. SDDC Pod Architecture
© 2016 VMware, Inc. All rights reserved.
Page 35 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 4. Required Number of Racks
Pod (Function)
Required
Number of
Racks
Minimum
Number of
Racks
Comment
(for full scale
deployment)
Management pod
and shared edge
and compute pod
1
1
Two half-racks are sufficient for the
management pod and shared edge and
compute pod. As the number and resource
usage of compute VMs increase adding
additional hosts to the cluster will be
required, as such extra space in the rack
should be reserved for growth.
Compute pods
6
0
With 6 compute racks, 6 compute pods
with 19 ESXi hosts each can achieve the
target size of 6000 average-sized VMs. If
an average size VM has two vCPUs with 4
GB of RAM, 6000 VMs with 20% overhead
for bursting workloads require 114 hosts.
The quantity and performance varies based
on the workloads running within the
compute pods.
Storage pods
6
0 (if using
Virtual SAN
for Compute
Pods)
Total
13
1
Storage that is not Virtual SAN storage is
hosted on isolated storage pods.
© 2016 VMware, Inc. All rights reserved.
Page 36 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 5. POD and Racks Design Decisions
Decision ID
Design Decision
Design Justification
Design Implication
SDDC-PHY-003
A single compute pod
is bound to a physical
rack.
Scaling out of the
SDDC infrastructure is
simplified by through a
1:1 relationship
between a compute
pod and the compute
resources contained
within a physical rack.
Dual power supplies
and power feeds are
required to ensure
availability of
hardware
components.
SDDC-PHY-004
The management and
the shared edge and
compute pod occupy
the same rack.
The number of
required compute
resources for the
management pod (4
ESXi servers) and
shared edge and
compute pod (4 ESXi
servers) are low and
do not justify a
dedicated rack for
each pod.
The design must
include sufficient
power and cooling to
operate the server
equipment. This
depends on the
selected vendor and
products.
On-ramp and off-ramp
connectivity to physical
networks (i.e., northsouth L3 routing on
NSX Edge virtual
appliances) can be
supplied to both the
management and
compute pods via this
management/edge
rack.
Edge resources
require external
connectivity to physical
network devices.
Placing edge
resources for
management and
compute in the same
rack will minimize
VLAN spread.
© 2016 VMware, Inc. All rights reserved.
Page 37 of 130
If the equipment in
this entire rack fails, a
second region is
needed to mitigate
downtime associated
with such an event.
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Decision ID
Design Decision
Design Justification
Design Implication
SDDC-PHY-005
Storage pods can
occupy one or more
racks.
To simplify the scale
out of the SDDC
infrastructure, the
storage pod to rack(s)
relationship has been
standardized.
The design must
include sufficient
power and cooling to
operate the storage
equipment. This
depends on the
selected vendor and
products.
It is possible that the
storage system arrives
from the manufacturer
in dedicated rack or
set of racks and a
storage system of this
type is accommodated
for in the design.
SDDC-PHY-006
Each rack has two
separate power feeds.
Redundant power
feeds increase
availability by ensuring
that failure of a power
feed does not bring
down all equipment in
a rack.
Combined with
redundant network
connections into a rack
and within a rack,
redundant power feeds
prevent failure of
equipment in an entire
rack.
SDDC-PHY-007
Mount the compute
resources (minimum
of 4 ESXi servers) for
the management pod
together in a rack.
Mounting the compute
resources for the
management pod
together can ease
physical data center
design, deployment
and troubleshooting.
Using a VM to host
ratio of more than
100:1 can lead to
availability issues.
Host numbers within
this pod should be
scaled accordingly.
© 2016 VMware, Inc. All rights reserved.
Page 38 of 130
All equipment used
must support two
separate power feeds.
The equipment must
keep running if one
power feed fails.
If the equipment of an
entire rack fails, the
cause, such as
flooding or an
earthquake, also
affects neighboring
racks. A second
region is needed to
mitigate downtime
associated with such
an event.
None.
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Decision ID
Design Decision
Design Justification
Design Implication
SDDC-PHY-008
Mount the compute
resources for the
shared edge and
compute pod
(minimum of 4 ESXi
servers) together in a
rack.
Mounting the compute
resources for the
shared edge and
compute pod together
can ease physical
datacenter design,
deployment and
troubleshooting.
None.
Using a VM to host
ratio of more than
100:1 can lead to
availability issues.
Host numbers within
this pod should be
scaled accordingly.
3.1.1.3. ESXi Host Physical Design Specifications
The physical design specifications of the ESXi host list the characteristics that were used during
deployment and testing of the VMware Validated Design.
The configuration and assembly process for each system is standardized, with all components
installed the same manner on each host. Standardizing the entire physical configuration of the ESXi
hosts is critical to providing an easily manageable and supportable infrastructure because
standardization eliminates variability. Consistent PCI card slot location, especially for network
controllers, is essential for accurate alignment of physical to virtual I/O resources. Deploy ESXi hosts
with identical configuration, including identical storage, and networking configurations, across all
cluster members. Identical configurations ensure an even balance of virtual machine storage
components across storage and compute resources.
Select all ESXi host hardware, including CPUs following the VMware Compatibility Guide.
The sizing of the physical servers for the ESXi hosts for the management and edge pods has special
consideration because it is based on the VMware document "VMware Virtual SAN Ready Nodes", as
these pod type use VMware Virtual SAN.

An average sized VM has two vCPUs with 4 GB of RAM, 6000 VMs with 20% for bursting
workloads.

A standard 2U server can host 60 average-sized VMs on a single ESXi host.
Table 6. ESXi Host Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCPHY-009
Use Virtual SAN
Ready Nodes.
Using a Virtual SAN Ready Node
ensures seamless compatibility
with Virtual SAN during the
deployment.
Might limit hardware
choices.
© 2016 VMware, Inc. All rights reserved.
Page 39 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
SDDCPHY-010
All nodes must
have uniform
configurations
across a given
cluster.
A balanced cluster delivers more
predictable performance even
during hardware failures. In
addition, performance impact
during resync/rebuild is minimal
when the cluster is balanced.
Vendor sourcing,
budgeting and
procurement
considerations for uniform
server nodes will be
applied on a per cluster
basis.
3.1.1.4. ESXi Host Memory
Note
See the VMware Virtual SAN 6.0 Design and Sizing Guide for more information about disk
groups, including design and sizing guidance. The number of disk groups and disks that an
ESXi host manages determines memory requirements. 32 GB of RAM is required to support
the maximum number of disk groups.
Table 7. Host Memory Design Decision
Decision
ID
Design Decision
Design Justification
Design
Implication
SDDCPHY-011
Set up each ESXi host in the
management and edge pods to
have a minimum 128 GB RAM.
The VMs in the management
and edge pods require a total
375 GB RAM.
None
3.1.1.5. Host Boot Device Background Considerations
Minimum boot disk size for ESXi in SCSI-based devices (SAS / SATA / SAN) is greater than 5 GB.
ESXi can be deployed using stateful local, SAN SCSI boot devices, or vSphere Auto Deploy.
What is supported depends on the version of Virtual SAN that you are using:

Virtual SAN does not support stateless vSphere Auto Deploy

Virtual SAN 5.5 supports USB/SD embedded devices for ESXi boot device (4 GB or greater).

Since Virtual SAN 6.0, there is an option to use SATADOM as a supported boot device.
Refer to the VMware Virtual SAN 6.0 Design and Sizing Guide to choose the option that best fits your
hardware.
3.1.2 Physical Networking Design
The physical network uses a leaf-and-spine design, shown in the following illustration. For additional
information, see Physical Network Architecture.
© 2016 VMware, Inc. All rights reserved.
Page 40 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 19. Leaf-and-Spine Architecture
3.1.2.1. Leaf-and-Spine and Network Virtualization Architecture
As virtualization, cloud computing, and distributed cloud become more pervasive in the data center, a
shift in the traditional three-tier networking model is taking place. This shift addresses simplicity and
scalability.
Simplicity
The traditional core-aggregate-access model is efficient for north/south traffic that travels in and out of
the data center. This model is usually built for redundancy and resiliency against failure. However, the
Spanning Tree Protocol (STP) typically blocks 50 percent of the critical network links to prevent
network loops, which means 50 percent of the maximum bandwidth is wasted until something fails.
A core-aggregate-access architecture is still widely used for service-oriented traffic that travels
north/south. However, the trends in traffic patterns are changing with the types of workloads. In
today’s data centers east/west or server-to-server traffic is common. If the servers in a cluster are
performing a resource-intensive calculation in parallel, unpredictable latency or lack of bandwidth are
undesirable. Powerful servers that perform these calculations can attempt to communicate with each
other, but if they cannot communicate efficiently because of a bottleneck in the network architecture,
wasted capital expenditure results.
One way to solve the problem is to create a leaf-and-spine architecture, also known as a distributed
core.
A leaf-and-spine architecture has two main components: spine switches and leaf switches.

Spine switches can be thought of as the core, but instead of being a large, chassis-based
switching platform, the spine consists of many high-throughput Layer 3 switches with high port
density.

Leaf switches can be treated as the access layer. Leaf switches provide network connection
points for servers and uplink to the spine switches.
Every leaf switch connects to every spine switch in the fabric. No matter which leaf switch a server is
connected to, it always has to cross the same number of devices to get to another server (unless the
other server is located on the same leaf). This design keeps the latency down to a predictable level
because a payload has to hop only to a spine switch and another leaf switch to get to its destination.
© 2016 VMware, Inc. All rights reserved.
Page 41 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 20. Example of a Small-Scale Leaf-and-Spine Architecture
Instead of relying on one or two large chassis-based switches at the core, the load is distributed
across all spine switches, making each individual spine insignificant as the environment scales out.
Scalability
Several factors, including the following, affect scale.

Number of racks that are supported in a fabric

Amount of bandwidth between any two racks in a data center

Number of paths a leaf switch can select from when communicating with another rack
The total number of available ports dictates the number of racks supported in a fabric across all spine
switches and the acceptable level of oversubscription.
Different racks might be hosting different types of infrastructure. For example, a rack might host filers
or other storage systems, which might attract or source more traffic than other racks in a data center.
In addition, traffic levels of compute racks (that is, racks that are hosting hypervisors with workloads
or virtual machines) might have different bandwidth requirements than edge racks, which provide
connectivity to the outside world. Link speed as well as the number of links vary to satisfy different
bandwidth demands.
The number of links to the spine switches dictates how many paths are available for traffic from this
rack to another rack. Because the number of hops between any two racks is consistent, equal-cost
multipathing (ECMP) can be used. Assuming traffic sourced by the servers carry a TCP or UDP
header, traffic spray can occur on a per-flow basis.
© 2016 VMware, Inc. All rights reserved.
Page 42 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 21. Leaf-and-Spine and Network Virtualization
3.1.2.2. Top of Rack Physical Switches
When configuring Top of Rack (ToR) switches, consider the following best practices:

Configure redundant physical switches to enhance availability.

Configure switch ports that connect to ESXi hosts manually as trunk ports. Virtual switches are
passive devices and do not send or receive trunking protocols, such as Dynamic Trunking
Protocol (DTP).

Modify the Spanning Tree Protocol (STP) on any port that is connected to an ESXi NIC to reduce
the time it takes to transition ports over to the forwarding state, for example using the Trunk
PortFast feature found in a Cisco physical switch.

Provide DHCP or DHCP Helper capabilities on all VLANs that are used by VMkernel ports. This
setup simplifies the configuration by using DHCP to assign IP address based on the IP subnet in
use.
3.1.2.3. Jumbo Frames
IP storage throughput can benefit from the configuration of jumbo frames. Increasing the per-frame
payload from 1500 bytes to the jumbo frame setting increases the efficiency of data transfer. Jumbo
frames must be configured end-to-end, which is easily accomplished in a LAN. When you enable
jumbo frames on an ESXi host, you have to select an MTU that matches the MTU of the physical
switch ports.
The workload determines whether it makes sense to configure jumbo frames on a virtual machine. If
the workload consistently transfers large amounts of network data, configure jumbo frames if possible.
In that case, the virtual machine operating systems and the virtual machine NICs must also support
jumbo frames.
Using jumbo frames also improves performance of vSphere vMotion.
© 2016 VMware, Inc. All rights reserved.
Page 43 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Note
VXLANs need an MTU value of at least 1600 bytes on the switches and routers that carry the
transport zone traffic.
Table 8. Jumbo Frames Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCPHYNET-001
Configure the MTU
size to 9000 bytes
(Jumbo Frames) on
the portgroups that
support the following
traffic types.
Setting the MTU to 9000 bytes
(Jumbo Frames) improves
traffic throughput.
When adjusting the MTU
packet size, the entire
network path (VMkernel
port, distributed switch,
physical switches and
routers) must also be
configured to support the
same MTU packet size.

NFS

Virtual SAN

vMotion

VXLAN

vSphere
Replication
In order to support VXLAN the
MTU setting must be increased
to a minimum of 1600 bytes,
setting this portgroup also to
9000 bytes has no effect on
VXLAN but ensures
consistency across portgroups
that are adjusted from the
default MTU size.
3.1.2.4. Leaf Switch Connectivity and Network Settings
Each ESXi host in the compute rack is connected redundantly to the SDDC network fabric ToR
switches via two 10 GbE ports, as shown in the following illustration. Configure the ToR switches to
provide all necessary VLANs via an 802.1Q trunk.
Figure 22. Leaf Switch to Server Connection within Compute Racks
© 2016 VMware, Inc. All rights reserved.
Page 44 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Each ESXi host in the management/shared edge and compute rack is connected to the SDDC
network fabric and also to the Wide Area Network (WAN) and to the Internet, as show in the following
illustration.
Figure 23. Leaf Switch to Server Connection within Management/Shared Compute and Edge
Rack
3.1.2.5. VLANs and Subnets
Each ESXi host in the compute rack and the management/edge rack uses VLANs and corresponding
subnets for internal-only traffic, as shown in the illustration below.
The leaf switches of each rack act as the Layer 3 interface for the corresponding subnet.
The management/edge rack provides externally accessible VLANs for access to the Internet and/or
MPL-based corporate networks.
© 2016 VMware, Inc. All rights reserved.
Page 45 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 24. Sample VLANs and Subnets within a Pod
Follow these guidelines:

Use only /24 subnets to reduce confusion and mistakes when dealing with IPv4 subnetting.

Use the IP address .1 as the (floating) interface with .2 and .3 for Virtual Router Redundancy
Protocol (VRPP) or Hot Standby Routing Protocol (HSRP).

Use the RFC1918 IPv4 address space for these subnets and allocate one octet by region and
another octet by function. For example, the mapping 172.regionid.function.0/24 results in the
following sample subnets.
Note
The following IP ranges are meant as samples. Your actual implementation depends on your
environment.
Table 9. VLAN Sample IP Ranges
Pod
Function
Sample VLAN
Sample IP range
Management
Management
1611 (Native)
172.1611.0/24
Management
vMotion
1612
172.16.12.0/24
Management
VXLAN
1614
172.16.14.024
Management
VSAN
1613
172.16.13.0/24
Shared Edge and Compute
Management
1631 (Native)
172.16.31.0/24
Shared Edge and Compute
vMotion
1632
172.16.32.0/24
Shared Edge and Compute
VXLAN
1634
172.16.34.0/24
Shared Edge and Compute
VSAN
1633
172.16.33.0/24
© 2016 VMware, Inc. All rights reserved.
Page 46 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
3.1.2.6. Access Port Network Settings
Configure additional network settings on the access ports that connect the leaf switch to the
corresponding servers.

Spanning-Tree Protocol (STP). Although this design does not use the spanning tree protocol,
switches usually come with STP configured by default. Designate the access ports as trunk
PortFast.

Trunking. Configure the VLANs as members of a 802.1Q trunk with the management VLAN
acting as the native VLAN.

MTU. Set MTU for all VLANS (Management, vMotion, VXLAN and Storage) to jumbo frames for
consistency purposes.

DHCP helper. Configure the VIF of the Management, vMotion and VXLAN subnet as a DHCP
proxy.

Multicast. Configure IGMP snooping on the ToR switches and include an IGMP querier on each
VLAN.
3.1.2.7. Region Interconnectivity
The SDDC management networks, VXLAN kernel ports and the edge and compute VXLAN kernel
ports of the two regions must be connected. These connections can be over a VPN tunnel, Point to
Point circuits, MPLS, etc. End users must be able to reach the public-facing network segments (public
management and tenant networks) of both regions.
The design of the connection solution is out of scope for this VMware Validated Design.
3.1.2.8. Physical Network Characteristics

Requirements. The design uses 4 spine switches with 40 GbE ports. As a result, each leaf
switch must have 4 uplink ports capable of 40 GbE.

Fault Tolerance. In case of a switch failure or scheduled maintenance, switch fabric capacity
reduction is 25% with four spine switches.

Oversubscription. Oversubscription can occur within a leaf switch. To compute the
oversubscription for a leaf switch, use this formula.
Total bandwidth available to all connected servers / aggregate amount of
uplink bandwidth
The compute rack and the management/edge rack have 19 ESXi hosts. Each ESXi host has one 10
GbE port connected to each ToR switch, creating up to 190 Gbps of bandwidth. With four 40 GbE
uplinks to the spine, you can compute oversubscription as follows (see the Oversubscription in the
Leaf Switches illustration).
190 Gbps (total bandwidth) /160 Gbps (uplink bandwidth) = 1.1875:1
© 2016 VMware, Inc. All rights reserved.
Page 47 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 25. Oversubscription in the Leaf Switches

Routing protocols. Base the selection of the external routing protocol on your current
implementation or on available expertise among the IT staff. Take performance requirements into
consideration. Possible options are OSPF, iBGP and IS-IS.

DHCP proxy. The DHCP proxy must point to a DHCP server via its IPv4 address. See the
External Service Dependencies section for details on the DHCP server.
Table 10. Physical Network Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCPHY-NET002
Racks are connected
using a leaf-and-spine
topology and Layer 3
connectivity.
A Layer 3 leaf-and-spine
architecture supports scale
out while maintaining failure
isolation.
Layer 2 traffic is reduced
to within the pod.
SDDCPHY-NET003
Only the management
and shared edge and
compute rack have
physical access to the
external network via
VLANs
Aggregating physical cabling
and network services to the
management and shared
edge and compute rack
reduces costs.
Workloads in compute
pods located in compute
racks have to use network
virtualization (NSX for
vSphere) for external
network connectivity..
SDDCPHY-NET004
Each rack uses two
ToR switches. These
switches provide
connectivity across two
10 GbE links to each
server.
This design uses two 10
GbE links to provide
redundancy and reduce
overall design complexity.
Requires two ToR
switches per rack which
can increase costs.
© 2016 VMware, Inc. All rights reserved.
Page 48 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCPHY-NET005
Use VLANs to segment
physical network
functions.
Allow for Physical network
connectivity without requiring
large number of NICs.
Uniform configuration and
presentation is required
on all the trunks made
available to the ESXi
hosts.
Segregation is needed for
the different network
functions that are required in
the SDDC. This allows for
differentiated services and
prioritization of traffic as
needed.
Table 11. Additional Network Design Decisions
Decision
ID
Design Decision
Design Justification
Design
Implication
SDDCPHY-NET006
Static IP addresses will be
assigned to all management
nodes of the SDDC
infrastructure.
Configuration of static IP
addresses avoid connection
outages due to DHCP
availability or
misconfiguration.
Accurate IP
address
management
must be in place.
SDDCPHY-NET007
DNS records to enable
forward, reverse, short and
FQDN resolution will be
created for all management
nodes of the SDDC
infrastructure.
Ensures consistent resolution
of management nodes using
both IP address (reverse
lookup) and name resolution.
None
SDDCPHY-NET008
NTP time source will be used
for all management nodes of
the SDDC infrastructure.
Critical to maintain accurate
and synchronized time
between management nodes.
None
3.1.3 Physical Storage Design
This VMware Validated Design relies on both VMware Virtual SAN storage and NFS storage. The
Shared Storage Design section explains where the SDDC uses which type of storage and gives
background information. The focus of this section is the physical storage design.
3.1.3.1. Virtual SAN Physical Design
Software-defined storage is a key technology in the SDDC. This design uses VMware Virtual SAN to
implement software-defined storage for the management clusters.
VMware Virtual SAN is a fully integrated hypervisor-converged storage software. Virtual SAN creates
a cluster of server hard disk drives and solid state drives and presents a flash-optimized, highly
resilient, shared storage datastore to hosts and virtual machines. Virtual SAN allows you to control
capacity, performance, and availability on a per virtual machine basis through the use of storage
policies.
Requirements and Dependencies
© 2016 VMware, Inc. All rights reserved.
Page 49 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
The following requirements and dependencies summarize the information in the VMware Virtual SAN
documentation. The design decisions of this VMware Validated Design fulfill these requirements.
The software-defined storage module has the following requirements and options:

Minimum of 3 hosts providing storage resources to the Virtual SAN cluster.

Virtual SAN is configured as hybrid storage or all-flash storage.

o
A Virtual SAN hybrid storage configuration requires both magnetic devices and flash caching
devices.
o
An All-Flash Virtual SAN configuration requires vSphere 6.0 or later.
Each ESXi host that provides storage resources to the cluster must meet the following
requirements:
o
Minimum of one SSD.
o
The SSD flash cache tier should be at least 10% of the size of the HDD capacity tier.
o
Minimum of two HHDs.
o
RAID controller compatible with VMware Virtual SAN.
o
10 Gbps network for Virtual SAN traffic with Multicast enabled.
o
vSphere High Availability Isolation Response set to power off virtual machines. With this
setting, no possibility of split brain conditions in case of isolation or network partition exists. In
a split-brain condition, the virtual machine might be powered on by two hosts by mistake. See
design decision SDDC-VI-VC-024 for more details.
Table 12. Virtual SAN Physical Storage Design Decision
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCPHY-STO001
Use one 200 GB SSD
and two traditional 1 TB
HDDs to create a single
disk group in the
management cluster.
Allow enough capacity
for the management
VMs with a minimum of
10% flash-based
caching.
Having only one disk group
limits the amount of striping
(performance) capability and
increases the size of the
fault domain.
Virtual SAN Background Information
vSphere offers two different Virtual SAN modes of operation, all-flash or hybrid.
Hybrid Mode
In a hybrid storage architecture, Virtual SAN pools server-attached capacity devices--in this case
magnetic devices--and caching devices, typically SSDs or PCI-e devices to create a distributed
shared datastore.
All-Flash Mode
VMware Virtual SAN can be deployed as all-flash storage. All-flash storage uses flash-based devices
(SSD or PCI-e) only as a write cache while other flash-based devices provide high endurance for
capacity and data persistence.
Table 13. Virtual SAN Mode Design Decision
Decision
ID
Design
Decision
Design Justification
© 2016 VMware, Inc. All rights reserved.
Page 50 of 130
Design Implication
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
SDDCPHY-STO002
Configure
Virtual SAN in
hybrid mode.
The VMs in the management cluster,
which are hosted within Virtual SAN,
do not require the performance or
expense of an all-flash Virtual SAN
configuration.
Virtual SAN hybrid mode
does not provide the
potential performance of
an all-flash configuration.
Hardware Considerations
You can build your own VMware Virtual SAN cluster or choose from a list of Virtual SAN Ready
Nodes.


Build Your Own. Be sure to use hardware from the VMware Compatibly Guide for Virtual SAN
(https://www.vmware.com/resources/compatibility/search.php?deviceCategory=Virtual SAN) for
the following components:
o
Solid state disks (SSDs)
o
Magnetic hard drives (HDDs)
o
I/O controllers, including Virtual SAN certified driver/firmware combinations
Use VMware Virtual SAN Ready Nodes. A Virtual SAN Ready Node is a validated server
configuration in a tested, certified hardware form factor for Virtual SAN deployment, jointly
recommended by the server OEM and VMware. See the VMware Virtual SAN Compatibility Guide
(https://www.vmware.com/resources/compatibility/pdf/vi_vsan_rn_guide.pdf). The Virtual SAN
Ready Node documentation provides examples of standardized configurations, including the
numbers of VMs supported and estimated number of 4K IOPS delivered.
As per design decision SDDC-PHY-009, the VMware Validated Design uses Virtual SAN Ready
Nodes.
3.1.3.2. Solid State Disk (SSD) Characteristics
In a VMware Virtual SAN configuration, the SSDs are used for the Virtual SAN caching layer for
hybrid deployments and for the capacity layer for all flash.

For a hybrid deployment, the use of the SSD is split between a non-volatile write cache
(approximately 30%) and a read buffer (approximately 70%). As a result, the endurance and the
number of I/O operations per second that the SSD can sustain are important performance factors.

For an all-flash model, endurance and performance have the same criteria. However, many more
write operations are held by the caching tier, thus elongating or extending the life of the SSD
capacity-tier.
SSD Endurance
This VMware Validated Design uses class D endurance class SSDs for the caching tier.
Note
All drives listed in the VMware Compatibility Guide for Virtual SAN
(https://www.vmware.com/resources/compatibility/search.php?deviceCategory=Virtual SAN)
meet the Class D requirements.
SDDC Endurance Design Decision Background
For endurance of the SSDs used for Virtual SAN, standard industry write metrics are the primary
measurements used to gauge the reliability of the drive. No standard metric exists across all vendors,
however; Drive Writes per Day (DWPD) or Petabytes Written (PBW) are the measurements normally
used.
For vSphere 5.5, the endurance class was based on Drive Writes Per Day (DWPD). For VMware
Virtual SAN 6.0, the endurance class has been updated to use Terabytes Written (TBW), based on
the vendor’s drive warranty. TBW can be used for VMware Virtual SAN 5.5 and VMware Virtual SAN
6.0 and is reflected in the VMware Compatibility Guide for Virtual SAN.
© 2016 VMware, Inc. All rights reserved.
Page 51 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
The reasoning behind using TBW is that VMware now offers the flexibility to use larger capacity drives
with lower DWPD specifications.
If a SSD vendor uses Drive Writes Per Day as a measurement, you can calculate endurance in
Terabytes Written (TBW) as follows:
TBW (over 5 years) = Drive Size x DWPD x 365 x 5
For example, if a vendor specified DWPD = 10 for a 800 GB capacity SSD, you can compute TBW as
follows:
TBW = 0.4TB X 10DWPD X 365days X 5yrs
TBW = 7300TBW
That means the SSD supports 7300TB writes over 5 years. (Higher TBW figures denote a higher
endurance class).
For SSDs that are designated for caching and all-flash capacity layers, the following table outlines
which endurance class to use for hybrid and for all-flash VMware Virtual SAN.
Table 14. Hybrid and All-Flash Virtual SAN Endurance Classes
Endurance
Class
TBW
Hybrid Caching
Tier
All-Flash Caching
Tier
All-Flash Capacity
Tier
Class A
>=365
No
No
Yes
Class B
>=1825
Yes
No
Yes
Class C
>=3650
Yes
Yes
Yes
Class D
>=7300
Yes
Yes
Yes
Note
This VMware Validated Design does not use All-Flash Virtual SAN.
Table 15. SSD Endurance Class Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCPHY-STO003
Use Class D
(>=7300TBW) SSDs
for the caching tier of
the management
cluster.
If a SSD designated for the
caching tier fails due to wear-out,
the entire VMware Virtual SAN
disk group becomes unavailable.
The result is potential data loss or
operational impact.
SSDs with higher
endurance may be
more expensive than
lower endurance
classes.
SSD Performance
There is a direct correlation between the SSD performance class and the level of Virtual SAN
performance. The highest-performing hardware results in the best performance of the solution. Cost is
therefore the determining factor. A lower class of hardware that is more cost effective might be
attractive even if the performance or size is not ideal. For optimal performance of Virtual SAN, select
class E SSDs. See the VMware Compatibility Guide for Virtual SAN
(https://www.vmware.com/resources/compatibility/search.php?deviceCategory=Virtual SAN) for detail
on the different classes.
SSD Performance Design Decision Background
© 2016 VMware, Inc. All rights reserved.
Page 52 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Select a high class of SSD for optimal performance of VMware Virtual SAN. Before selecting a drive
size, consider disk groups and sizing as well as expected future growth. VMware defines classes of
performance in the VMware Compatibility Guide for Virtual SAN
(https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan) as follows:
Table 16. SSD Performance Classes
Performance Class
Writes Per Second
Class A
2,500 – 5,000
Class B
5,000 – 10,000
Class C
10,000 – 20,000
Class D
20,000 – 30,000
Class E
30,000 – 100,100
Class F
100,000 +
Select an SSD size that is, at a minimum, 10 percent of the anticipated size of the consumed HDD
storage capacity, before failures to tolerate are considered. For example, select an SSD of at least
100 GB for 1 TB of HDD storage consumed in a 2 TB disk group.
Caching Algorithm
Both hybrid clusters and all-flash configurations adhere to the recommendation that 10% of consumed
capacity for the flash cache layer. However, there are differences between the two configurations:

Hybrid Virtual SAN. 70% of the available cache is allocated for storing frequently read disk
blocks, minimizing accesses to the slower magnetic disks. 30% of available cache is allocated to
writes.

All-Flash Virtual SAN. All-flash clusters have two types of flash: very fast and durable write
cache, and cost-effective capacity flash. Here cache is 100% allocated for writes, as read
performance from capacity flash is more than sufficient.
Use Class E SSDs for the highest possible level of performance from the VMware Virtual SAN
volume.
Table 17. SSD Performance Class Selection
Design
Quality
Option 1
Class E
Option 2
Class C
Comments
Availability
o
o
Neither design option impacts availability.
Manageability
o
o
Neither design option impacts manageability.
Performance
↑
↓
The higher the storage class that is used, the
better the performance.
Recover-ability
o
o
Neither design option impacts recoverability.
Security
o
o
Neither design option impacts security.
Legend: ↑ = positive impact on quality; ↓ = negative impact on quality; o = no impact on quality.
© 2016 VMware, Inc. All rights reserved.
Page 53 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 18. SSD Performance Class Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCPHY-STO004
Use Class E SSDs
(30,000-100,000 writes
per second) for the
management cluster.
The storage I/O performance
requirements within the
Management cluster dictate the
need for at least Class E SSDs.
Class E SSDs might
be more expensive
than lower class
drives.
3.1.3.3. Magnetic Hard Disk Drives (HDD) Characteristics
The HDDs in a VMware Virtual SAN environment have two different purposes, capacity and object
stripe width.

Capacity. Magnetic disks, or HDDs, unlike caching-tier SSDs, make up the capacity of a Virtual
SAN datastore

Stripe Width. You can define stripe width at the virtual machine policy layer. Virtual SAN might
use additional stripes when making capacity and placement decisions outside a storage policy.
Virtual SAN supports these disk types:

Serial Attached SCSI (SAS)

Near Line Serial Attached SCSI (NL-SCSI). NL-SAS can be thought of as enterprise SATA drives
but with a SAS interface.

Serial Advanced Technology Attachment (SATA). Use SATA magnetic disks only in capacitycentric environments where performance is not prioritized.
SAS and NL-SAS get you the best results. This VMware Validated Design uses 10,000 RPM drives to
achieve a balance between cost and availability.
HDD Capacity, Cost, and Availability Background Considerations
You can achieve the best results with SAS and NL-SAS.
As per the VMware Compatibility Guide for Virtual SAN, the maximum size of an SAS drive at the time
of writing is 6 TB. The VMware Virtual SAN design must consider the number of magnetic disks
required for the capacity layer, and how well the capacity layer will perform.

SATA disks typically provide more capacity per individual drive, and tend to be less expensive
than SAS drives. However, the trade-off is performance, because SATA performance is not as
good as SAS performance due to lower rotational speeds (typically 7200RPM)

Choose SAS magnetic disks instead of SATA magnetic disks in environments where performance
is critical.
Consider that failure of a larger capacity drive has operational impact on the availability and recovery
of more components.
Rotational Speed (RPM) Background Considerations
HDDs tend to be more reliable, but that comes at a cost. SAS disks can be available up to 15,000
RPM speeds.
Table 19. Virtual SAN HDD Environmental Characteristics
Characteristic
Revolutions per Minute (RPM)
Capacity
7,200
Performance
10,000
© 2016 VMware, Inc. All rights reserved.
Page 54 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Additional Performance
15,000
Cache-friendly workloads are less sensitive to disk performance characteristics; however, workloads
can change over time. HDDs with 10,000 RPM are the accepted norm when selecting a capacity tier.
For the software-defined storage module, VMware recommends that you use an HDD configuration
that is suited to the characteristics of the environment. If there are no specific requirements, selecting
10,000 RPM drives achieves a balance between cost and availability.
Table 20. HDD Characteristic Selection
Design
Quality
Option 1
7200
RPM
Option 2
10000
RPM
Option 3
15000
RPM
Comments
Availability
↑
↓
↓
Less expensive disks make it easier to
achieve more failures to tolerate without
incurring a large cost. Therefore, slower
disks are an appealing option for an
environment in which availability is
important.
Manageability
o
o
o
No design option impacts manageability.
Performance
↓
↑
↑↑
In a VMware Virtual SAN environment,
performance is best when using high-RPM
HDDs. However, a high-performing SDD
impacts performance more than a high-RPM
HDD.
Recoverability
o
o
o
No design option impacts recover-ability.
Security
o
o
o
No design option impacts security.
Legend: ↑ = positive impact on quality; ↓ = negative impact on quality; o = no impact on quality.
Table 21. HDD Selection Design Decisions
Decision
ID
Design Decision
Design Justification
Design
Implication
SDDCPHY-STO005
Use 10,000 RPM
HDDs for the
management
cluster.
10,000 RPM HDDs achieve a balance
between performance and availability
for the VMware Virtual SAN
configuration.
Slower and
potentially cheaper
HDDs are not
available.
The performance of 10,000 RPM HDDs
avoids disk drain issues. In Virtual SAN
hybrid mode, the Virtual SAN
periodically flushes uncommitted writes
to the capacity tier.
© 2016 VMware, Inc. All rights reserved.
Page 55 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
3.1.3.4.
I/O Controllers
The I/O controllers are as important to a VMware Virtual SAN configuration as the selection of disk
drives. Virtual SAN supports SAS, SATA, and SCSI adapters in either pass-through or RAID 0 mode.
Virtual SAN supports multiple controllers per host.

Multiple controllers can improve performance and mitigate a controller or SSD failure to a smaller
number of drives or Virtual SAN disk groups.

With a single controller, all disks are controlled by one device. A controller failure impacts all
storage, including the boot media (if configured).
Controller queue depth is possibly the most important aspect for performance. All I/O controllers in the
VMware Virtual SAN Hardware Compatibility Guide have a minimum queue depth of 256. Consider
normal day-to-day operations and increase of I/O due to Virtual Machine deployment operations or resync I/O activity as a result of automatic or manual fault remediation.
A Note on SAS Expanders
SAS expanders are a storage technology that lets you maximize the storage capability of your SAS
controller card. Like switches of an ethernet network, SAS expanders enable you to connect a larger
number of devices, that is, more SAS/SATA devices to a single SAS controller. Many SAS controllers
support up to 128 or more hard drives.
VMware has not extensively tested SAS expanders, as a result performance and operational
predictability are relatively unknowns at this point. Avoid configurations with SAS expanders.
3.1.3.5. NFS Physical Storage Design
Network File System (NFS) is a distributed file system protocol that allows a user on a client computer
to access files over a network much like local storage is accessed. In this case the client computer is
an ESXi host, and the storage is provided by a NFS-capable external storage array.
The management cluster uses VMware Virtual SAN for primary storage and NFS for secondary
storage. The compute clusters are not restricted to any particular storage technology. For compute
clusters, the decision on which technology to use is based on the performance, capacity, and
capabilities (replication, deduplication, compression, etc.) required by the workloads that are running
in the clusters.
Table 22. NFS Usage Design Decisions
Decision ID
Design Decision
Design Justification
Design
Implication
SDDCPHY-STO006
NFS storage is presented
to provide:
An NFS capable
external array is
required.

A datastore for backup
data
Separate primary virtual
machine storage from backup
data in case of primary storage
failure.

An export for archive
data
vRealize Log Insight archiving
requires a NFS export.

A datastore for
templates and ISOs
Requirements
Your environment must meet the following are requirements to use NFS storage in the VMware
Validated Design.

Storage arrays are connected directly to the leaf switches.

All connections are made using 10 Gb Ethernet.

Jumbo Frames are enabled.
© 2016 VMware, Inc. All rights reserved.
Page 56 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

10K SAS (or faster) drives are used in the storage array.
Different disk speeds and disk types can be combined in an array to create different performance and
capacity tiers. The management cluster uses 10K SAS drives in the RAID configuration
recommended by the array vendor to achieve the required capacity and performance.
Table 23. NFS Hardware Design Decision
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCPHY-STO007
Use 10K SAS drives
for management
cluster NFS
volumes.
10K SAS drives achieve a balance
between performance and capacity.
Faster drives can be used if
desired.
10K SAS drives are
generally more
expensive than other
alternatives.
vSphere Data Protection requires
high-performance datastores in
order to meet backup SLAs.
vRealize Automation uses NFS
datastores for its content catalog
which requires high-performance
datastores.
vRealize Log Insight uses NFS
datastores for its archive storage
which, depending on compliance
regulations, can use a large amount
of disk space.
Volumes
A volume consists of multiple disks in a storage array. RAID is applied at the volume level. The more
disks in a volume, the better the performance and the greater the capacity.
Multiple datastores can be created on a single volume, but for applications that do not have a high I/O
footprint a single volume with multiple datastores is sufficient.

For high I/O applications, such as backup applications, use a dedicated volume to avoid
performance issues.

For other applications, set up Storage I/O Control (SIOC) to impose limits on high I/O applications
so that other applications get the I/O they are requesting.
Table 24. Volume Assignment Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCPHY-STO009
Use a shared volume
for other
management
component
datastores.
Non-backup related
management applications can
share a common volume due
to the lower I/O profile of these
applications.
Enough storage space for
shared volumes and their
associated application
data must be available.
3.2
Virtual Infrastructure Design
The virtual infrastructure design includes the software components that make up the virtual
infrastructure layer and that support the business continuity of the SDDC. These components include
the software products that provide the virtualization platform hypervisor, virtualization management,
storage virtualization, network virtualization, backup and disaster recovery. VMware products in this
© 2016 VMware, Inc. All rights reserved.
Page 57 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
layer include VMware vSphere, VMware Virtual SAN, VMware NSX, vSphere Data Protection, and
VMware Site Recovery Manager.
Figure 26. Virtual Infrastructure Layer Business Continuity in the SDDC
3.2.1 Virtual Infrastructure Design Overview
The SDDC virtual infrastructure consists of two regions. Each region includes a management pod,
and a shared edge and compute pod.
© 2016 VMware, Inc. All rights reserved.
Page 58 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 27. SDDC Logical Design
3.2.1.1. Management Pod
Management pods run the virtual machines that manage the SDDC. These virtual machines
host vCenter Server, NSX Manager, NSX Controller, vRealize Operations, vRealize Log Insight,
vRealize Automation, Site Recovery Manager and other shared management components. All
management, monitoring, and infrastructure services are provisioned to a vCenter Server High
Availability cluster which provides high availability for these critical services. Permissions on the
management cluster limit access to only administrators. This protects the virtual machines running the
management, monitoring, and infrastructure services.
3.2.1.2. Shared Edge and Compute Pod
The virtual infrastructure design uses a shared edge and compute pod. The shared pod combines the
characteristics of typical edge and compute pods into a single pod. It is possible to separate these in
the future if required.
This pod provides the following main functions:

Supports on-ramp and off-ramp connectivity to physical networks

Connects with VLANs in the physical world
© 2016 VMware, Inc. All rights reserved.
Page 59 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

Optionally hosts centralized physical services

Hosts the SDDC tenant virtual machines
The shared edge and compute pod connects the virtual networks (overlay networks) provided by NSX
for vSphere and the external networks. An SDDC can mix different types of compute-only pods and
provide separate compute pools for different types of SLAs.
3.2.1.3. Business Continuity
You can support business continuity and disaster recovery (BCDR) in the SDDC by protecting
vCenter Server, NSX for vSphere, vRealize Automation, vRealize Operations Manager, and vRealize
Log Insight. Enable backup and failover to a recovery region of these management
applications to continue the delivery of infrastructure management, operations management, and
cloud platform management.
3.2.1.4. Data Protection Design
Data backup protects the data of your organization against data loss, hardware failure, accidental
deletion, or other disaster for each region. For consistent image-level backups, use backup software
that is based on the VMware Virtual Disk Development Kit (VDDK), such as vSphere Data Protection.
Figure 28. vSphere Data Protection Logical Design
3.2.1.5. Disaster Recovery Design
The SDDC disaster recovery design includes two regions:

Protected Region A in San Francisco. Region A runs the management stack virtual machine
workloads that are being protected and is referred to as the protected region in this document.

Recovery Region B in Los Angeles. Region B is the disaster recovery region and is referred to
as the recovery region.
Site Recovery Manager can automate the setup and execution of disaster recovery plans between
these two regions.
© 2016 VMware, Inc. All rights reserved.
Page 60 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Note
A region in the VMware Validated Design is equivalent to the site construct in Site Recovery
Manager.
Figure 29. Disaster Recovery Logical Design
3.2.2 ESXi Design
3.2.2.1. ESXi Hardware Requirements
You can find the ESXi hardware requirements in Physical Design Fundamentals. The following design
outlines the design of the ESXi configuration.
3.2.2.2. ESXi Manual Install and Boot Options
You can install or boot ESXi 6.0 from the following storage systems:

SATA disk drives. SATA disk drives connected behind supported SAS controllers or supported
on-board SATA controllers.

Serial-attached SCSI (SAS) disk drives. Supported for installing ESXi.

SAN. Dedicated SAN disk on Fibre Channel or iSCSI.

USB devices. Supported for installing ESXi. 16 GB or more is recommended.

FCoE (Software Fibre Channel over Ethernet)
ESXi can boot from a disk larger than 2 TB if the system firmware and the firmware on any add-in
card support it. See the vendor documentation.
© 2016 VMware, Inc. All rights reserved.
Page 61 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
3.2.2.3. ESXi Boot Disk and Scratch Configuration
For new installations of ESXi, the installer creates a 4 GB VFAT scratch partition. ESXi uses this
scratch partition to store log files persistently. By default, vm-support output, which is used by
VMware to troubleshoot issues on the ESXi host, is also stored on the scratch partition.
An ESXi installation on SD/USB media does not configure a default scratch partition. VMware
recommends that you specify a scratch partition on a VMFS volume or configure remote syslog
logging for the host.
Table 25. ESXi Boot Disk Design Decision
`
Design Decision
Design Justification
Design Implication
SDDC-VIESXi-001
Install and configure
all ESXi hosts to boot
using local USB or
SD devices.
USB or SD cards are an
inexpensive and easy to
configure option for installing
ESXi.
When you use USB or SD storage,
ESXi logs are not retained.
Configure remote syslog (such as
vRealize Log Insight) to collect
ESXi host logs.
Using local USB or SD
allows allocation of all local
HDDs to a VMware Virtual
SAN storage system.
3.2.2.4. ESXi Host Access
After installation, ESXi hosts are added to a VMware vCenter Server system and managed through
that vCenter Server system.
Direct access to the host console is still available and most commonly used for troubleshooting
purposes. You can access ESXi hosts directly using one of these three methods:

Direct Console User Interface (DCUI). Graphical interface on the console. Allows basic
administrative controls and troubleshooting options.

ESXi Shell. A Linux-style bash login on the ESXi console itself.

Secure Shell (SSH) Access. Remote command-line console access.
You can enable or disable each method. By default, the ESXi Shell and SSH are disabled to secure
the ESXi host. The DCUI is disabled only if Lockdown Mode is enabled.
3.2.2.5. ESXi User Access
By default, root is the only user who can log in to an ESXi host directly, however, you can add ESXi
hosts to an Active Directory domain. After the host has been added to an Active Directory domain,
access can be granted through Active Directory groups. Auditing who has logged into the host also
becomes easier.
© 2016 VMware, Inc. All rights reserved.
Page 62 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 26. ESXi User Access Design Decisions
Decision
ID
Design Decision
Design Justification
Design
Implication
SDDC-VIESXi-002
Add each host to the child
Active Directory domain for the
region and in which it will
reside. e.g. sfo01.rainpole.local
or lax01.rainpole.local
Using Active Directory
membership allows greater
flexibility in granting
access to ESXi hosts.
Adding hosts to
the domain can
add some
administrative
overhead.
Change the default ESX Admins
group to the SDDC-Admins
Active Directory group. Add
ESXi administrators to the
SDDC-Admins group following
standard access procedures.
Having an SDDC-Admins
group is more secure
because it removes a
known administrative
access point. In addition
different groups allow for
separation of management
tasks.
SDDC-VIESXi-003
Ensuring that users log in
with a unique user account
allows greater visibility for
auditing.
Additional
changes to the
host's advanced
settings are
required.
3.2.2.6. Virtual Machine Swap Configuration
When a virtual machine is powered on, the system creates a VMkernel swap file to serve as a
backing store for the virtual machine's RAM contents. The default swap file is stored in the same
location as the virtual machine's configuration file. This simplifies the configuration, however it can
cause an excess of replication traffic that is not needed.
You can reduce the amount of traffic that is replicated by changing the swap file location to a userconfigured location on the host. However, it can take longer to perform VMware vSphere vMotion ®
operations when the swap file has to be recreated.
Table 27. Other ESXi Host Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIESXi-004
Configure all ESXi
hosts to synchronize
time with the central
NTP servers.
Required because
deployment of vCenter
Server Appliance on an
ESXi host might fail if the
host is not using NTP.
All firewalls located
between the ESXi host and
the NTP servers have to
allow NTP traffic on the
required network ports.
3.2.3 vCenter Server Design
The Design put forward for vCenter includes both the Server component and the VMware Platform
Services Controller instances.
A VMware Platform Services Controller groups a set of infrastructure services including vCenter
Single Sign-On, License service, Lookup Service, and VMware Certificate Authority. You can deploy
the Platform Services controller and the associated vCenter Server system on the same virtual
machine (embedded Platform Services Controller) or on different virtual machines (external Platform
Services Controller).
© 2016 VMware, Inc. All rights reserved.
Page 63 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 28. vCenter Server Design Decision
Decision
ID
Design Decision
Design Justification
Design
Implication
SDDC-VIVC-001
Deploy two vCenter Server
systems in the first
availability zone of each
region.
Isolates vCenter Server failures to
management or compute
workloads.
Requires
licenses for
each vCenter
Server
instance.


One vCenter Server
supporting the SDDC
management
components.
One vCenter Server
supporting the edge
components and
compute workloads.
Isolates vCenter Server operations
between management and
compute.
Supports a scalable cluster design
where the management
components may be re-used as
additional compute needs to be
added to the SDDC.
Simplifies capacity planning for
compute workloads by eliminating
management workloads from
consideration in the the Compute
vCenter Server.
Improves the ability to upgrade the
vSphere environment and related
components by providing for
explicit separation of maintenance
windows:

Management workloads
remain available while
workloads in compute are
being addressed

Compute workloads remain
available while workloads in
management are being
addressed
Ability to have clear separation of
roles and responsibilities to ensure
that only those administrators with
proper authorization can attend to
the management workloads.
Facilitates quicker troubleshooting
and problem resolution.
Simplifies Disaster Recovery
operations by supporting a clear
demarcation between recovery of
the management components and
compute workloads.
Enables the use of two NSX
managers, one for the
management pod and the other for
the shared edge and compute pod.
Network separation of the pods in
the SDDC allows for isolation of
potential network issues.
© 2016 VMware, Inc. All rights reserved.
Page 64 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
You can install vCenter Server as a Windows-based system or deploy the Linux-based VMware
vCenter Server Appliance. The Linux-based vCenter Server Appliance is preconfigured, enables fast
deployment, and potentially results in reduced Microsoft licensing costs.
Table 29. vCenter Server Platform Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-002
Deploy all vCenter
Server instances as
Linux-based vCenter
Server Appliances.
Allows for rapid
deployment, enables
scalability, and reduces
Microsoft licensing costs.
Operational staff might
need Linux experience to
troubleshoot the Linuxbased appliances.
3.2.3.1. Platform Services Controller Design Decision Background
vCenter Server supports installation with an embedded Platform Services Controller (embedded
deployment) or with an external Platform Services Controller.

In an embedded deployment, vCenter Server and the Platform Services Controller run on the
same virtual machine. Embedded deployments are recommended for standalone
environments with only one vCenter Server system.

Environments with an external Platform Services Controller can have multiple vCenter Server
systems. The vCenter Server systems can use the same Platform Services Controller
services. For example, several vCenter Server systems can use the same instance of vCenter
Single Sign-On for authentication.

If there is a need to replicate with other Platform Services Controller instances, or if the solution
includes more than one vCenter Single Sign-On instance, you can deploy multiple external
Platform Services Controller instances on separate virtual machines.
Table 30. Platform Service Controller Design Decisions
Decision
ID
Design Decision
Design Justification
Design
Implication
SDDC-VIVC-003
Deploy each vCenter
Server with an external
Platform Services
Controller.
External Platform Services
Controllers are required for
replication between Platform
Services Controller instances.
The number of
VMs that have to
be managed
increases.
SDDC-VIVC-004
Join all Platform
Services Controller
instances to a single
vCenter Single SignOn domain.
When all Platform Services
Controller instances are joined into
a single vCenter Single Sign-On
domain, they can share
authentication and license data
across all components and regions.
Only one Single
Sign-On domain
will exist.
SDDC-VIVC-005
Create a ring topology
for the Platform
Service Controllers.
By default, Platform Service
Controllers only replicate with one
other Platform Services Controller,
that creates a single point of failure
for replication. A ring topology
ensures each Platform Service
Controller has two replication
partners and eliminates any single
point of failure.
Command-line
interface
commands must
be used to
configure the ring
replication
topology.
© 2016 VMware, Inc. All rights reserved.
Page 65 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 30. vCenter Server and Platform Services Controller Deployment Model
3.2.3.2. vCenter Server Networking
As specified in the physical networking design static IP addresses and host names must be used for
all vCenter Server systems. The IP addresses must have valid (internal) DNS registration including
reverse name resolution.
The vCenter Server systems must maintain network connections to the following components:

All VMware vSphere Client and vSphere Web Client user interfaces.

Systems running vCenter Server add-on modules.

Each ESXi host.
3.2.3.3. vCenter Server Redundancy
Protecting the vCenter Server system is important because it is the central point of management and
monitoring for the SDDC. How you protect vCenter Server depends on maximum downtime tolerated,
and on whether failover automation is required.
The following table lists methods available for protecting the vCenter Server system and the vCenter
Server Appliance.
Table 31. Methods for Protecting vCenter Server System and the vCenter Server Appliance
Redundancy
Method
Protects
vCenter Server
system
(Windows)?
Protects
Platform
Services
Controller
(Windows)?
Protects
vCenter Server
(Appliance)?
Protects
Platform
Services
Controller
(Appliance)?
Automated
protection using
vSphere HA.
Yes
Yes
Yes
Yes
Manual
configuration and
manual failover. For
example, using a
cold standby.
Yes
Yes
Yes
Yes
© 2016 VMware, Inc. All rights reserved.
Page 66 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Redundancy
Method
Protects
vCenter Server
system
(Windows)?
Protects
Platform
Services
Controller
(Windows)?
Protects
vCenter Server
(Appliance)?
Protects
Platform
Services
Controller
(Appliance)?
HA Cluster with
external load
balancer
Not Available
Yes
Not Available
Yes
Table 32. vCenter Server Systems Protection Design Decisions
Decision
ID
Design Decision
Design Justification
Design
Implication
SDDC-VIVC-006
Protect all vCenter
Server appliances by
using vSphere HA.
Supports availability objectives for
vCenter Server appliances without
a required manual intervention
during a failure event.
vCenter Server will
be unavailable
during a vSphere
HA failover.
3.2.3.4. vCenter Server Appliance Sizing
The following tables outline minimum hardware requirements for the management vCenter Server
appliance and the compute vCenter Server appliance.
Table 33. Logical Specification for Management vCenter Server Appliance
Attribute
Specification
vCenter Server version
6.0 (vCenter Server Appliance)
Physical or virtual system
Virtual (appliance)
Appliance Size
Small (up to 100 hosts / 1,000 VMs)
Platform Services Controller
External
Number of CPUs
2
Memory
16 GB
Disk Space
136 GB
Table 34. Logical Specification for Compute vCenter Server Appliance
Attribute
Specification
vCenter Server version
6.0 (vCenter Server Appliance)
Physical or virtual system
Virtual (appliance)
Appliance Size
Large (up to 1,000 hosts / 10,000 VMs)
© 2016 VMware, Inc. All rights reserved.
Page 67 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Attribute
Specification
Platform Services Controller
External
Number of CPUs
16
Memory
32 GB
Disk Space
295 GB
Table 35. vCenter Appliance Sizing Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-007
Configure the
management vCenter
Server Appliances
with the small size
setting.
Based on the number of
management VMs that are
running, a vCenter Server
Appliance installed with the
small size setting is sufficient.
If the size of the
management
environment changes,
the vCenter Server
Appliance size might
need to be increased.
SDDC-VIVC-008
Configure the compute
and edge vCenter
Server Appliances
with the large size
setting.
Based on the number of
compute and edge VMs that are
running, a vCenter Server
Appliance installed with the
large size setting is needed.
As the compute
environment grows,
additional vCenter
Server instances
might be needed.
3.2.3.5. Database Design
A vCenter Server Appliance can use either a built-in local PostgreSQL database or an external Oracle
database. Both configurations support up to 1,000 hosts or 10,000 virtual machines.
Database Design Decision Background
A vCenter Server Windows installation can use either a supported external database or a local
PostgreSQL database. The local PostgreSQL database is installed with vCenter Server and is limited
to 20 hosts and 200 virtual machines. Supported external databases include Microsoft SQL Server
2008 R2, SQL Server 2012, SQL Server 2014, Oracle Database 11g, and Oracle Database 12c.
External databases require a 64-bit DSN. DSN aliases are not supported.
vCenter Server Appliance can use either a local PostgreSQL database that is built into the appliance,
which is recommended, or an external database. Supported external databases include Oracle
Database 11g and Oracle Database 12c. External database support is being deprecated in this
release; this is the last release that supports the use of an external database with vCenter Server
Appliance.
Unlike a vCenter Windows installation, a vCenter Server Appliance that uses a local PostgreSQL
database supports up to 1,000 hosts or 10,000 virtual machines at full vCenter Server scale.
© 2016 VMware, Inc. All rights reserved.
Page 68 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 36. vCenter Database Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-009
Set up all vCenter
Server instances to
use the embedded
PostgreSQL
databases.
Reduces both overhead and
Microsoft or Oracle licensing
costs. Avoids problems with
upgrades. Support for external
databases is deprecated for
vCenter Server Appliance in the
next release.
The vCenter Server
Appliance has limited
database management
tools for database
administrators.
3.2.3.6. vSphere Cluster Design
The cluster design must take into account the workload that the cluster handles. Different cluster
types in this design have different characteristics.
vSphere Cluster Design Decision Background
The following heuristics help with cluster design decisions.

Decide to use fewer, larger hosts or more, smaller hosts.
o
A scale-up cluster has fewer, larger hosts.
o
A scale-out cluster has more, smaller hosts.
o
A virtualized server cluster typically has more hosts with fewer virtual machines per host.

Compare the capital costs of purchasing fewer, larger hosts with the costs of purchasing more,
smaller hosts. Costs vary between vendors and models.

Evaluate the operational costs of managing a few hosts with the costs of managing more hosts.

Consider the purpose of the cluster.

Consider the total number of hosts and cluster limits.
Figure 31. vSphere Logical Cluster Layout
© 2016 VMware, Inc. All rights reserved.
Page 69 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
3.2.3.7. vSphere High Availability Design
VMware vSphere High Availability (vSphere HA) protects your virtual machines in case of host failure
by restarting virtual machines on other hosts in the cluster when a host fails.
During configuration of the cluster, the hosts elect a master host. The master host communicates with
the vCenter Server system and monitors the virtual machines and secondary hosts in the cluster.
The master hosts detects different types of failure:

Host failure, for example an unexpected power failure

Host network isolation or connectivity failure

Loss of storage connectivity

Problems with virtual machine OS availability
Table 37. vSphere HA Design Decisions
Decision
ID
Design
Decision
Design Justification
Design Implication
SDDC-VIVC-010
Use vSphere
HA to protect all
clusters against
failures.
vSphere HA supports a
robust level of protection for
both host and virtual
machine availability.
Sufficient resources on the
remaining host are required to
so that virtual machines can be
migrated to those hosts in the
event of a host outage.
SDDC-VIVC-011
Set vSphere HA
Host Isolation
Response to
Power Off.
Virtual SAN requires that the
HA Isolation Response be
set to Power Off and to
restart VMs on available
hosts.
VMs are powered off in case of
a false positive and a host is
declared isolated incorrectly.
3.2.3.8. vSphere HA Admission Control Policy Configuration
The vSphere HA Admission Control Policy allows an administrator to configure how the cluster judges
available resources. In a smaller vSphere HA cluster, a larger proportion of the cluster resources are
reserved to accommodate host failures, based on the selected policy. The following policies are
available:

Host failures the cluster tolerates. vSphere HA ensures that a specified number of hosts can
fail and sufficient resources remain in the cluster to fail over all the virtual machines from those
hosts.

Percentage of cluster resources reserved. vSphere HA ensures that a specified percentage of
aggregate CPU and memory resources are reserved for failover.

Specify Failover Hosts. When a host fails, vSphere HA attempts to restart its virtual machines
on any of the specified failover hosts. If restart is not possible, for example the failover hosts have
insufficient resources or have failed as well, then vSphere HA attempts to restart the virtual
machines on other hosts in the cluster.
3.2.3.9. vSphere Cluster Workload Design
This design defines the following vSphere clusters and the workloads that they handle.
© 2016 VMware, Inc. All rights reserved.
Page 70 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 38. vSphere Cluster Workload Design Decisions
Decision ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-012
Create a single
management cluster
containing all management
hosts.
Simplifies
configuration by
isolating
management
workloads from
compute workloads.
Management of multiple
clusters and vCenter Server
instances increases
operational overhead.
Ensures that
compute workloads
have no impact on
the management
stack.
You can add ESXi
hosts to the cluster
as needed.
SDDC-VIVC-013
Create a shared edge and
compute cluster that hosts
compute workloads, NSX
Controllers and associated
NSX Edge gateway devices
used for compute
workloads.
Simplifies
configuration and
minimizes the
number of hosts
required for initial
deployment.
Ensures that the
management stack
has no impact on
compute workloads.
Management of multiple
clusters and vCenter Server
instances increases
operational overhead.
Due to the shared nature of
the cluster, when compute
workloads are added, the
cluster must be scaled out
to keep high level of
network performance.
You can add ESXi
hosts to the cluster
as needed.
3.2.3.10. Management Cluster Design
The management cluster design determines the management cluster configuration.
Table 39. Management Cluster Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-014
Create a management
cluster with 4 hosts.
Cluster redundancy is
n+2 protection for
vSphere HA which
covers outage
redundancy during
maintenance tasks.
Two hosts is generally
considered enough to support
the management components.
One host supports failover in
case of a hardware defect. One
more host allows failover if a
second host is unavailable for
scheduled maintenance.
Calculate reserved
amounts when
cluster size increases
to prevent
overprotection.
© 2016 VMware, Inc. All rights reserved.
Page 71 of 130
Additional host
resources are
required for
redundancy.
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-015
Set vSphere HA for the
management cluster
to reserve 25% of
cluster resources for
failover.
Using the percentage- based
reservation works well in
situations where virtual
machines have varying and
sometime significant CPU or
memory reservations.
If additional hosts are
added to the cluster,
more resources are
being reserved for
failover capacity.
Recalculate the
percentage of
reserved resources
when additional hosts
are added to the
cluster.
Management Cluster Logical Design Background
The following table summarizes the attributes of the management cluster logical design.
Table 40. Management Cluster Attributes
Attribute
Specification
Number of hosts required to support management hosts with no
overcommitment
2
Number of hosts recommended due to operational constraints
(Ability to take a host offline without sacrificing High Availability
capabilities)
3
Number of hosts recommended due to operational constraints,
while using Virtual SAN (Ability to take a host offline without
sacrificing High Availability capabilities)
4
Capacity for host failures per cluster
25% reserved CPU & RAM
Number of usable hosts per cluster
3 usable hosts
3.2.3.11. Shared Edge and Compute Cluster Design
Tenant workloads run on the ESXi hosts in the shared edge and compute cluster, also, due to the
shared nature of the cluster, NSX Controllers and Edge devices run in this cluster.
© 2016 VMware, Inc. All rights reserved.
Page 72 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 41. Edge Cluster Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-016
Create a shared
edge and compute
cluster for the NSX
Controllers and
NSX Edge
gateway devices.
NSX Manager requires a 1:1
relationship with a vCenter
Server system.
Each time you provision a
Compute vCenter Server
system, a new NSX
Manager is required.
SDDC-VIVC-017
Set vSphere HA
for the shared
edge and compute
cluster to reserve
25% of cluster
resources for
failover.
vSphere HA protects the NSX
Controller instances and edge
services gateway devices in
the event of a host failure.
vSphere HA powers on virtual
machines from the failed hosts
on any remaining hosts..
If one of the hosts becomes
unavailable, two Controllers
run on a single host..
SDDC-VIVC-018
Create shared
edge and compute
cluster with a
minimum of 4
hosts.
3 NSX Controllers are required
for sufficient redundancy and
majority decisions.
4 hosts is the smallest
starting point for the shared
edge and compute cluster
for redundancy and
performance thus
increasing cost over a 3
node cluster.
SDDC-VIVC-019
Set up VLANbacked port
groups for external
access and
management on
the shared edge
and compute
cluster hosts.
Edge gateways need access
to the external network in
addition to the management
network.
VLAN-backed port groups
must be configured with the
correct number of ports, or
with elastic port allocation
SDDC-VIVC-020
Create a resource
pool for the
required SDDC
NSX Controllers
and edge
appliances with a
CPU share level of
High, a memory
share of normal,
and 15 GB
memory
reservation.
The NSX components control
all network traffic in and out of
the SDDC as well as update
route information for interSDDC communication. In a
contention situation it is
imperative that these virtual
machines receive all the
resources required.
During contention SDDC
NSX components receive
more resources then all
other workloads as such
monitoring and capacity
management must be a
proactive activity.
Set anti-affinity rules to
keep each Controller on a
separate host. A 4-node
cluster allows maintenance
while ensuring that the 3
Controllers remain on
separate hosts.
One host is available for
failover and to allow for
scheduled maintenance.
© 2016 VMware, Inc. All rights reserved.
Page 73 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-021
Create a resource
pool for all user
NSX Edge devices
with a CPU share
value of Normal
and a memory
share value of
Normal.
NSX edges for users, created
by vRealize Automation,
support functions such as load
balancing for user workloads.
These edge devices do not
support the entire SDDC as
such they receive fewer
resources during contention.
During contention these
NSX edges will receive less
resources then the SDDC
edge devices as such
monitoring and capacity
management must be a
proactive activity.
SDDC-VIVC-022
Create a resource
pool for all user
virtual machines
with a CPU share
value of Normal
and a memory
share value of
Normal.
In a shared edge and compute
cluster the SDDC edge
devices must be guaranteed
resources above all other
workloads as to not impact
network connectivity. Setting
the share values to normal
gives the SDDC edges more
shares of resources during
contention ensuring network
traffic is not impacted.
During contention user
workload virtual machines
could be starved for
resources and experience
poor performance. It is
critical that monitoring and
capacity management must
be a proactive activity and
that capacity is added or a
dedicated edge cluster is
created before contention
occurs.
Shared Edge and Compute Cluster Logical Design Background
The following table summarizes the attributes of the shared edge and compute cluster logical design.
The number of VMs on the shared edge and compute cluster will start low but will grow quickly as
user workloads are created.
Table 42. Shared Edge and Compute Cluster Attributes
Attribute
Specification
Capacity for host failures per cluster
1
Number of usable hosts per cluster
3
Minimum number of hosts required to support the shared edge and compute cluster
4
3.2.3.12. Compute Cluster Design
As the SDDC grows additional compute-only clusters can be configured. Tenant workloads run on the
ESXi hosts in the compute cluster instances. Multiple compute clusters are managed by the Compute
vCenter Server instance.
© 2016 VMware, Inc. All rights reserved.
Page 74 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 43. Compute Cluster Design Decisions
Decision ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-023
The hosts in each compute
cluster are contained within a
single rack.
The spine-and-leaf
architecture dictates that
all hosts in a cluster
must be connected to
the same top-of-rack
switches.
Fault domains are
limited to each rack.
SDDC-VIVC-024
Configure vSphere HA to use
percentage-based failover
capacity to ensure n+1
availability. The exact setting
depend on the number of hosts
in the compute cluster.
Using explicit host
failover limits the total
available resources in a
cluster.
As the number of
hosts in the cluster
changes, the
percentage of failover
capacity must be
adjusted.
3.2.3.13. vCenter Server Customization
vCenter Server supports a rich set of customization options, including monitoring, virtual machine fault
tolerance, and so on. For each feature, this VMware Validated Design specifies the design decisions.
3.2.3.14. VM and Application Monitoring Service
When VM and Application Monitoring is enabled, the VM and Application Monitoring service, which
uses VMware Tools, evaluates whether each virtual machine in the cluster is running. The service
checks for regular heartbeats and I/O activity from the VMware Tools process running on guests. If
the service receives no heartbeats or I/O activity, it is likely that the guest operating system has failed
or that VMware Tools is not being allocated time for heartbeats or I/O activity. In this case, the service
determines that the virtual machine has failed and reboots the virtual machine.
Enable Virtual Machine Monitoring for automatic restart of a failed virtual machine. The application or
service that is running on the virtual machine must be capable of restarting successfully after a reboot
or the VM restart is not sufficient.
Table 44. Monitor Virtual Machines Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-025
Enable Virtual
Machine Monitoring
for each cluster.
Virtual Machine Monitoring
provides adequate in-guest
protection for most VM
workloads.
There is no downside to
enabling Virtual
Machine Monitoring.
3.2.3.15. VMware vSphere Distributed Resource Scheduling (DRS)
vSphere Distributed Resource Scheduling provides load balancing of a cluster by migrating workloads
from heavily loaded hosts to less utilized hosts in the cluster. DRS supports manual and automatic
modes.

Manual. Recommendations are made but an administrator needs to confirm the changes.

Automatic. Automatic management can be set to five different levels. At the lowest setting,
workloads are placed automatically at power on and only migrated to fulfill certain criteria, such as
entering maintenance mode. At the highest level, any migration that would provide a slight
improvement in balancing will be executed.
© 2016 VMware, Inc. All rights reserved.
Page 75 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 45. vSphere Distributed Resource Scheduling Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-026
Enable DRS on all
clusters and set it to
automatic, with the
default setting
(medium).
The default settings provide
the best trade-off between
load balancing and
excessive migration with
vMotion events.
In the event of a vCenter
outage, mapping from
virtual machines to ESXi
hosts might be more
difficult to determine.
3.2.3.16. Enhanced vMotion Compatibility (EVC)
EVC works by masking certain features of newer CPUs to allow migration between hosts containing
older CPUs. EVC works only with CPUs from the same manufacturer and there are limits to the
version difference gaps between the CPU families.
If you set EVC during cluster creation, you can add hosts with newer CPUs at a later date without
disruption. You can use EVC for a rolling upgrade of all hardware with zero downtime.
Set EVC to the highest level possible with the current CPUs in use.
Table 46. VMware Enhanced vMotion Compatibility Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-027
Enable Enhanced vMotion
Compatibility on all clusters.
Allows cluster
upgrades without
virtual machine
downtime.
You can enable EVC only
if clusters contain hosts
with CPUs from the same
vendor.
Set EVC mode to the lowest
available setting supported
for the hosts in the cluster.
3.2.3.17. Use of Transport Layer Security (TLS) Certificates
By default, vSphere 6.0 uses TLS/SSL certificates that are signed by VMCA (VMware Certificate
Authority). By default, these certificates are not trusted by end-user devices or browsers. It is a
security best practice to replace at least user-facing certificates with certificates that are signed by a
third-party or enterprise Certificate Authority (CA). Certificates for machine-to-machine communication
can remain as VMCA-signed certificates.
Table 47. vCenter Server TLS Certificate Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIVC-028
Replace the vCenter
Server machine
certificate and Platform
Services Controller
machine certificate with
a certificate signed by
a custom Certificate
Authority (CA).
Infrastructure administrators
connect to both vCenter Server
and the Platform Services
Controller via web browser to
perform configuration,
management and troubleshooting
activities. Certificate warnings
result with the default certificate.
Replacing and
managing
certificates is an
operational
overhead.
© 2016 VMware, Inc. All rights reserved.
Page 76 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
3.2.4 Virtualization Network Design
A well-designed network helps the organization meet its business goals. It prevents unauthorized
access, and provides timely access to business data.
This network virtualization design uses vSphere and VMware NSX for vSphere to implement virtual
networking.
3.2.4.1. Virtual Network Design Background Considerations
This VMware Validated Design follows high-level network design guidelines and networking best
practices.
The high-level design goals apply regardless of your environment.

Meet diverse needs. The network must meet the diverse needs of many different entities in an
organization. These entities include applications, services, storage, administrators, and users.

Reduce costs. Reducing costs is one of the simpler goals to achieve in the vSphere
infrastructure. Server consolidation alone reduces network costs by reducing the number of
required network ports and NICs, but a more efficient network design is desirable. For example,
configuring two 10 GbE NICs with VLANs might be more cost effective than configuring a dozen 1
GbE NICs on separate physical networks.

Boost performance. You can achieve performance improvement and decrease the time that is
required to perform maintenance by providing sufficient bandwidth, which reduces contention and
latency.

Improve availability. A well-designed network improves availability, typically by providing
network redundancy.

Support security. A well-designed network supports an acceptable level of security through
controlled access (where required) and isolation (where necessary).

Enhance infrastructure functionality. You can configure the network to support vSphere
features such as vSphere vMotion, vSphere High Availability, and vSphere Fault Tolerance.
Follow networking best practices throughout your environment.

Separate network services from one another to achieve greater security and better performance.

Use Network I/O Control and traffic shaping to guarantee bandwidth to critical virtual machines.
During network contention these critical virtual machines will receive a higher percentage of the
bandwidth.

Separate network services on a single vSphere Distributed Switch by attaching them to port
groups with different VLAN IDs.

Keep vSphere vMotion traffic on a separate network. When migration with vMotion occurs, the
contents of the guest operating system’s memory is transmitted over the network. You can put
vSphere vMotion on a separate network by using a dedicated vSphere vMotion VLAN.

When using passthrough devices with a Linux kernel version 2.6.20 or earlier guest OS, avoid
MSI and MSI-X modes because these modes have significant performance impact.

For best performance, use VMXNET3 virtual NICs.

Ensure that physical network adapters that are connected to the same vSphere Standard Switch
or vSphere Distributed Switch are also connected to the same physical network.

Configure all VMkernel network adapters with the same MTU. When several VMkernel network
adapters are connected to distributed switches, but those network adapters have different MTUs
configured, network connectivity problems might result.
3.2.4.2. Network Segmentation and VLANs
Separating different types of traffic is required to reduce contention and latency. Separate networks
are also required for access security.
© 2016 VMware, Inc. All rights reserved.
Page 77 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
High latency on any network can negatively affect performance. Some components are more
sensitive to high latency than others. For example, reducing latency is important on the IP storage
and the vSphere Fault Tolerance logging network because latency on these networks can negatively
affect the performance of multiple virtual machines.
Depending on the application or service, high latency on specific virtual machine networks can also
negatively affect performance. Use information gathered from the current state analysis and from
interviews with key stakeholder and SMEs to determine which workloads and networks are especially
sensitive to high latency.
Note
Configuring separate networks requires additional monitoring and administration.
3.2.4.3. Virtual Networks
Determine the number of networks or VLANs that are required depending on the type of traffic.


vSphere operational traffic.
o
Management
o
vMotion
o
Virtual SAN
o
NFS Storage
o
vSphere Replication
o
VXLAN
Traffic that supports the organization’s services and applications.
3.2.4.4. Virtual Switches
Virtual switches simplify the configuration process by providing one single pane of glass view for
performing virtual network management tasks.
Virtual Switch Design Background
A vSphere Distributed Switch (distributed switch) offers several enhancements over standard virtual
switches.

Centralized management. Because distributed switches are created and managed centrally on a
vCenter Server system, they make the switch configuration more consistent across ESXi hosts.
Centralized management saves time, reduces mistakes, and lowers operational costs.

Additional features. Distributed switches offer features that are not available on standard virtual
switches. Some of these features can be useful to the applications and services that are running
in the organization’s infrastructure. For example, NetFlow and port mirroring provide monitoring
and troubleshooting capabilities to the virtual infrastructure.
Consider the following caveats for distributed switches.

Distributed switches require a VMware vSphere Enterprise Plus Edition license.

Distributed switches are not manageable when vCenter Server is unavailable. vCenter Server
therefore becomes a tier one application.
Health Check
The health check service helps identify and troubleshoot configuration errors in vSphere distributed
switches.
Health check helps identify the following common configuration errors.

Mismatched VLAN trunks between an ESXi host and the physical switches it's connected to.

Mismatched MTU settings between physical network adapters, distributed switches, and physical
switch ports.

Mismatched virtual switch teaming policies for the physical switch port-channel settings.
© 2016 VMware, Inc. All rights reserved.
Page 78 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Health check monitors VLAN, MTU, and teaming policies.

VLANs. Checks whether the VLAN settings on the distributed switch match the trunk port
configuration on the connected physical switch ports.

MTU. For each VLAN, health check determines whether the physical access switch port's MTU
jumbo frame setting matches the distributed switch MTU setting.

Teaming policies. Health check determines whether the connected access ports of the physical
switch that participate in an EtherChannel are paired with distributed ports whose teaming policy
is IP hash.
Health check is limited to the access switch port to which the ESXi hosts' NICs connects.
Note
For VLAN and MTU checks, at least two physical NICs for the distributed switch are required.
For a teaming policy check, at least two physical NICs and two hosts are required when
applying the policy.
Number of Virtual Switches
Create fewer virtual switches, preferably just one. For each type of network traffic, configure a single
virtual switch with a port group to simplify configuration and monitoring.
Table 48. Virtual Switch Design Decisions
Decision ID
Design Decision
Design Justification
Design Implication
SDDC-VINet-001
Use vSphere
Distributed Switches
(VDS).
vSphere Distributed
Switches simplify
management.
Migration from a VSS to a VDS
requires a minimum of two
physical NICs to maintain
redundancy.
SDDC-VINet-002
Use a single VDS per
cluster.
Reduces complexity of
the network design.
None.
Management Cluster Distributed Switches
The management cluster uses a single vSphere Distributed Switch with the following configuration
settings.
Table 49. Virtual Switch for the Management Cluster
vSphere
Distributed
Switch
Name
Function
Network I/O
Control
Number of
Physical
NIC Ports
MTU
vDS-Mgmt

ESXi Management
Enabled
2
9000

Network IP Storage (NFS)

vSphere vMotion

VXLAN Tunnel Endpoint (VTEP)

Uplinks (2) to enable ECMP

External management connectivity
© 2016 VMware, Inc. All rights reserved.
Page 79 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 50. vDS-Mgmt Port Group Configuration Settings
Parameter
Setting
Failover detection
Link status only
Notify switches
Enabled
Failback
No
Failover order
Active uplinks: Uplink1, Uplink2
Figure 32. Network Switch Design for Management Hosts
This section expands on the logical network design by providing details on the physical NIC layout
and physical network attributes.
Table 51. Management Virtual Switches by Physical/Virtual NIC
vSphere Distributed Switch
vmnic
Function
vDS-Mgmt
0
Uplink
vDS-Mgmt
1
Uplink
© 2016 VMware, Inc. All rights reserved.
Page 80 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 52. Management Virtual Switch Port Groups and VLANs
vSphere Distributed
Switch
Port Group Name
Teaming
Active
Uplinks
VLAN
ID
vDS-Mgmt
vDS-MgmtManagement
Route based on physical
NIC load
0, 1
1611
vDS-Mgmt
vDS-Mgmt-vMotion
Route based on physical
NIC load
0, 1
1612
vDS-Mgmt
Auto Generated (NSX
VTEP)
Route based on SRC-ID
0, 1
1614
vDS-Mgmt
vDS-Mgmt-Uplink01
Route based on physical
NIC load
0, 1
2711
vDS-Mgmt
vDS-Mgmt-Uplink02
Route based on physical
NIC load
0, 1
2712
vDS-Mgmt
vDS-Mgmt-NFS
Route based on physical
NIC load
0, 1
1615
vDS-Mgmt
vDS-Mgmt-ExtManagement
Route based on physical
NIC load
0, 1
130
Table 53. Management VMkernel Adapter
vSphere Distributed
Switch
Network
Label
Connected Port
Group
Enabled
Services
MTU
vDS-Mgmt
Management
vDS-MgmtManagement
Management
Traffic
1500
(Default)
vDS-Mgmt
vMotion
vDS-Mgmt-vMotion
vMotion Traffic
9000
vDS-Mgmt
NFS
vDS-Mgmt-NFS
-
9000
vDS-Mgmt
VTEP
Auto Generated (NSX
VTEP)
-
9000
For more information on the physical network design specifications, see the Physical Network Design
section.
© 2016 VMware, Inc. All rights reserved.
Page 81 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Shared Edge and Compute Cluster Distributed Switches
The shared edge and compute cluster uses a single vSphere Distributed Switch with the following
configuration settings.
Table 54. Virtual Switch for the shared Edge and Compute Cluster
vSphere Distributed
Switch Name
Function
Network I/O
Control
Number of
Physical
NIC Ports
MTU
vDS-Comp01
ESXi Management
Enabled
2
9000
Network IP Storage
(NFS)
vSphere vMotion
VXLAN Tunnel Endpoint
(VTEP)
Uplinks (2) to enable
ECMP
External customer/tenant
connectivity
Table 55. vDS-Comp01 Port Group Configuration Settings
Parameter
Setting
Failover detection
Link status only
Notify switches
Enabled
Failback
No
Failover order
Active uplinks: Uplink1, Uplink2
© 2016 VMware, Inc. All rights reserved.
Page 82 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 33. Network Switch Design for shared Edge and Compute Hosts
This section expands on the logical network design by providing details on the physical NIC layout
and physical network attributes.
Table 56. Shared Edge and Compute Cluster Virtual Switches by Physical/Virtual NIC
vSphere Distributed Switch
vmnic
Function
vDS-Comp01
0
Uplink
vDS-Comp01
1
Uplink
Table 57. Edge Cluster Virtual Switch Port Groups and VLANs
vSphere Distributed
Switch
Port Group Name
Teaming
Active
Uplinks
VLAN
ID
vDS-Comp01
vDS- Comp01Management
Route based on physical
NIC load
0, 1
1631
vDS- Comp01
vDS- Comp01-vMotion
Route based on physical
NIC load
0, 1
1632
© 2016 VMware, Inc. All rights reserved.
Page 83 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
vSphere Distributed
Switch
Port Group Name
Teaming
Active
Uplinks
VLAN
ID
vDS-Comp01
vDS-Comp01-NFS
Route based on physical
NIC load
0, 1
1615
vDS- Comp01
Auto Generated (NSX
VTEP)
Route based on SRC-ID
0, 1
1634
vDS- Comp01
vDS- Comp01Uplink01
Route based on physical
NIC load
0, 1
1635
vDS- Comp01
vDS- Comp01Uplink02
Route based on physical
NIC load
0, 1
2713
Table 58. Shared Edge and Compute Cluster VMkernel Adapter
vSphere Distributed
Switch
Network
Label
Connected Port
Group
Enabled
Services
MTU
vDS- Comp01
Management
vDS- Comp01Management
Management
Traffic
1500
(Default)
vDS- Comp01
vMotion
vDS- Comp01-vMotion
vMotion Traffic
9000
vDS-Comp01
NFS
vDS-Comp01-NFS
-
9000
vDS- Comp01
VTEP
Auto Generated (NSX
VTEP)
-
9000
For more information on the physical network design, see the Physical Network Design section.
Compute Cluster Distributed Switches
A compute cluster vSphere Distributed Switch uses the following configuration settings.
Table 59. Virtual Switches for Compute Cluster Hosts
vSphere Distributed
Switch Name
Function
Network I/O
Control
Number of
Physical NIC Ports
MTU
vDS-Comp02
ESXi Management
Enabled
2
9000
Network IP Storage
(NFS)
vSphere vMotion
VXLAN Tunnel
Endpoint (VTEP)
Table 60. vDS-Comp02 Port Group Configuration Settings
Parameter
Setting
© 2016 VMware, Inc. All rights reserved.
Page 84 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Failover detection
Link status only
Notify switches
Enabled
Failback
No
Failover order
Active uplinks: Uplink1, Uplink2
Figure 34. Network Switch Design for Compute Hosts
This section expands on the logical network design by providing details on the physical NIC layout
and physical network attributes.
Table 61. Compute Cluster Virtual Switches by Physical/Virtual NIC
vSphere Distributed Switch
vmnic
Function
vDS-Comp02
0
Uplink
vDS-Comp02
1
Uplink
© 2016 VMware, Inc. All rights reserved.
Page 85 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 62. Compute Cluster Virtual Switch Port Groups and VLANs
vSphere Distributed
Switch
Port Group Name
Teaming
Active
Uplinks
VLAN
ID
vDS-Comp02
vDS-Comp02Management
Route based on
physical NIC load
0, 1
1621
vDS-Comp02
vDS-Comp02-vMotion
Route based on
physical NIC load
0, 1
1622
vDS-Comp02
Auto Generated (NSX
VTEP)
Route based on SRCID
0, 1
1624
vDS-Comp02
vDS-Comp02-NFS
Route based on
physical NIC load
0, 1
1625
Table 63. Compute Cluster VMkernel Adapter
vSphere Distributed
Switch
Network
Label
Connected Port
Group
Enabled
Services
MTU
vDS-Comp02
Management
vDS-Comp02Management
Management
traffic
1500
(Default)
vDS-Comp02
vMotion
vDS-Comp02-vMotion
vMotion traffic
9000
vDS-Comp02
NFS
vDS-Comp02-NFS
-
9000
vDS-Comp02
VTEP
Auto Generated (NSX
VTEP)
-
9000
For more information on the physical network design specifications, see the Physical Network Design
section.
3.2.4.5. NIC Teaming
You can use NIC teaming to increase the network bandwidth available in a network path, and to
provide the redundancy that supports higher availability.
NIC teaming helps avoid a single point of failure and provides options for load balancing of traffic. To
further reduce the risk of a single point of failure, build NIC teams by using ports from multiple NIC
and motherboard interfaces.
Create a single virtual switch with teamed NICs across separate physical switches.
This VMware Validated Design uses an active-active configuration using the route that is based on
physical NIC load algorithm for teaming. In this configuration, idle network cards do not wait for a
failure to occur, and they aggregate bandwidth.
NIC Teaming Design Background
For a predictable level of performance, use multiple network adapters in one of the following
configurations.

An active-passive configuration that uses explicit failover when connected to two separate
switches.
© 2016 VMware, Inc. All rights reserved.
Page 86 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

An active-active configuration in which two or more physical NICs in the server are assigned the
active role.
This validated design uses an active-active configuration.
Table 64. NIC Teaming and Policy
Design
Quality
ActiveActive
ActivePassive
Comments
Availability
↑
↑
Using teaming regardless of the option increases the
availability of the environment.
Manageability
o
o
Neither design option impacts manageability.
Performance
↑
o
An active-active configuration can send traffic across either
NIC, thereby increasing the available bandwidth. This
configuration provides a benefit if the NICs are being shared
among traffic types and Network I/O Control is used.
Recoverability
o
o
Neither design option impacts recoverability.
Security
o
o
Neither design option impacts security.
Note
Legend: ↑ = positive impact on quality; ↓ = negative impact on quality; o = no impact on
quality.
Table 65. NIC Teaming Design Decision
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VINet-003
Use the Route based on physical
NIC load teaming algorithm for all
port groups except for ones that
carry VXLAN traffic. VTEP kernel
ports and VXLAN traffic will use
Route based on SRC-ID.
Reduce complexity of
the network design
and increase resiliency
and performance.
Because NSX does not
support Route based on
physical NIC load two
different algorithms are
necessary.
3.2.4.6. Network I/O Control
When Network I/O Control is enabled, the distributed switch allocates bandwidth for the following
system traffic types.

Fault tolerance traffic

iSCSI traffic

vSphere vMotion traffic

Management traffic

VMware vSphere Replication traffic

NFS traffic

VMware Virtual SAN traffic

vSphere Data Protection backup traffic
© 2016 VMware, Inc. All rights reserved.
Page 87 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

Virtual machine traffic
How Network I/O Control Works
Network I/O Control enforces the share value specified for the different traffic types only when there is
network contention. When contention occurs Network I/O Control applies the share values set to each
traffic type. As a result, less important traffic, as defined by the share percentage, will be throttled,
allowing more important traffic types to gain access to more network resources.
Network I/O Control also allows the reservation of bandwidth for system traffic based on the capacity
of the physical adapters on a host, and enables fine-grained resource control at the virtual machine
network adapter level. Resource control is similar to the model for vCenter CPU and memory
reservations.
Network I/O Control Considerations
The following heuristics can help with design decisions.

Shares vs. Limits. When you use bandwidth allocation, consider using shares instead of limits.
Limits impose hard limits on the amount of bandwidth used by a traffic flow even when network
bandwidth is available.

Limits on Certain Resource Pools. Consider imposing limits on a given resource pool. For
example, if you put a limit on vSphere vMotion traffic, you can benefit in situations where multiple
vSphere vMotion data transfers, initiated on different hosts at the same time, result in
oversubscription at the physical network level. By limiting the available bandwidth for vSphere
vMotion at the ESXi host level, you can prevent performance degradation for other traffic.

vSphere FT Traffic. Because vSphere FT traffic is latency sensitive, keep the shares value for
this resource pool set to high. If you are using custom shares, set the shares value to a
reasonably high relative value.

Teaming Policy. When you use Network I/O Control, use Route based on physical NIC load
teaming as a distributed switch teaming policy to maximize the networking capacity utilization.
With load-based teaming, traffic might move among uplinks, and reordering of packets at the
receiver can result occasionally.

Traffic Shaping. Use distributed port groups to apply configuration policies to different traffic
types. Traffic shaping can help in situations where multiple vSphere vMotion migrations initiated
on different hosts converge on the same destination host. The actual limit and reservation also
depend on the traffic shaping policy for the distributed port group where the adapter is connected
to.
For example, if a VM network adapter requires a limit of 200 Mbps, and the average bandwidth
that is configured in the traffic shaping policy is 100 Mbps, then the effective limit becomes 100
Mbps.
Table 66. Network I/O Control Design Decision
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VINET-004
Use Network I/O
Control version 3.
Network I/O Control version 3 enables
per vNIC reservations as well as
creating pools of bandwidth that are
guaranteed to workloads in those pools.
Version 2.0 is the
default, administrators
must upgrade to
version 3.0.
SDDC-VINET-005
Enable Network
I/O Control on all
distributed
switches.
Increase resiliency and performance of
the network.
If configured incorrectly
Network I/O Control
could impact network
performance for critical
traffic types.
© 2016 VMware, Inc. All rights reserved.
Page 88 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VINET-006
Set the share
value for vMotion
traffic to Low.
During times of contention vMotion
traffic is not as important as virtual
machine or storage traffic.
During times of network
contention vMotion's
will take longer then
usual to complete.
SDDC-VINET-007
Set the share
value for vSphere
Replication traffic
to Low.
During times of contention vSphere
Replication traffic is not as important as
virtual machine or storage traffic.
During times of network
contention vSphere
Replication will take
longer and could violate
the defined SLA.
SDDC-VINET-008
Set the share
value for Virtual
SAN to High.
During times of contention Virtual SAN
traffic needs guaranteed bandwidth so
virtual machine performance does not
suffer.
None.
Set the share
value for
Management to
Normal.
By keeping the default setting of Normal
management traffic is prioritized higher
then vMotion and vSphere Replication
but lower then Virtual SAN traffic.
Management traffic is important as it
ensures the hosts can still be managed
during times of network contention.
None.
SDDC-VINET-010
Set the share
value for NFS
Traffic to Normal.
Because NFS is used for secondary
storage, such as VDP backups and
vRealize Log Insight archives it is not as
important as Virtual SAN traffic, by
prioritizing it lower Virtual SAN is not
impacted.
During times of
contention VDP
backups will be slower
than usual.
SDDC-VINET-011
Set the share
value for vSphere
Data Protection
Backup traffic to
Low.
During times of contention it is more
important that primary functions of the
SDDC continue to have access to
network resources over backup traffic.
During times of
contention VDP
backups will be slower
than usual.
SDDC-VINET-012
Set the share
value for virtual
machines to High.
Virtual machines are the most important
asset in the SDDC. Leaving the default
setting of High ensures that they will
always have access to the network
resources they need.
None.
SDDC-VINET-013
Set the share
value for Fault
Tolerance to Low.
Fault Tolerance is not used in this
design therefore it can be set to the
lowest priority.
None.
SDDC-VINET-014
Set the share
value for iSCSI
traffic to Low.
iSCSI is not used in this design
therefore it can be set to the lowest
priority.
None.
SDDC-VINET-009
© 2016 VMware, Inc. All rights reserved.
Page 89 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
3.2.4.7. VXLAN
VXLAN provides the capability to create isolated, multi-tenant broadcast domains across data center
fabrics and enables customers to create elastic, logical networks that span physical network
boundaries.
The first step in creating these logical networks is to abstract and pool the networking resources. Just
as vSphere abstracts compute capacity from the server hardware to create virtual pools of resources
that can be consumed as a service, vSphere Distributed Switch and VXLAN abstract the network into
a generalized pool of network capacity and separate the consumption of these services from the
underlying physical infrastructure. A network capacity pool can span physical boundaries, optimizing
compute resource utilization across clusters, pods, and geographically-separated data centers. The
unified pool of network capacity can then be optimally segmented into logical networks that are
directly attached to specific applications.
VXLAN works by creating Layer 2 logical networks that are encapsulated in standard Layer 3 IP
packets. A Segment ID in every frame differentiates the VXLAN logical networks from each other
without any need for VLAN tags. As a result, large numbers of isolated Layer 2 VXLAN networks can
coexist on a common Layer 3 infrastructure.
In the vSphere architecture, the encapsulation is performed between the virtual NIC of the guest VM
and the logical port on the virtual switch, making VXLAN transparent to both the guest virtual
machines and the underlying Layer 3 network. Gateway services between VXLAN and non-VXLAN
hosts (for example, a physical server or the Internet router) are performed by the NSX for vSphere
Edge gateway appliance. The Edge gateway translates VXLAN segment IDs to VLAN IDs, so that
non-VXLAN hosts can communicate with virtual machines on a VXLAN network.
The dedicated edge cluster hosts all NSX Edge instances and all Universal Distributed Logical Router
instances that are connect to the Internet or to corporate VLANs, so that the network administrator
can manage the environment in a more secure and centralized way.
Table 67. VXLAN Design Decisions
Decision ID
Design Decision
Design Justification
Design Implication
SDDC-VINet-015
Use NSX for vSphere to
introduce VXLANs for the
use of virtual application
networks and tenants
networks.
Simplify the network
configuration for each tenant
via centralized virtual network
management.
Requires NSX for
vSphere licenses.
SDDC-VINet-016
Use VXLAN along with
NSX Edge gateways and
the Universal Distributed
Logical Router (UDLR) to
provide customer/tenant
network capabilities.
Create isolated, multi-tenant
broadcast domains across
data center fabrics to create
elastic, logical networks that
span physical network
boundaries.
Transport networks
and MTU greater than
1600 bytes has to be
configured in the
reachability radius.
SDDC-VINet-017
Use VXLAN along with
NSX Edge gateways and
the Universal Distributed
Logical Router (UDLR) to
provide management
application network
capabilities.
Leverage benefits of network
virtualization in the
management pod.
Requires installation
and configuration of the
NSX for vSphere
instance in the
management pod.
© 2016 VMware, Inc. All rights reserved.
Page 90 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
3.2.5 NSX Design
3.2.5.1. Software-Defined Networking Design
This design implements software-defined networking by using VMware NSX™ for vSphere®. With
NSX for vSphere, virtualization delivers for networking what it has already delivered for compute and
storage. In much the same way that server virtualization programmatically creates, snapshots,
deletes, and restores software-based virtual machines (VMs), NSX network virtualization
programmatically creates, snapshots, deletes, and restores software-based virtual networks. The
result is a transformative approach to networking that not only enables data center managers to
achieve orders of magnitude better agility and economics, but also supports a vastly simplified
operational model for the underlying physical network. NSX for vSphere is a nondisruptive solution
because it can be deployed on any IP network, including existing traditional networking models and
next-generation fabric architectures, from any vendor.
When administrators provision workloads, network management is one of the most time-consuming
tasks. Most of the time spent provisioning networks is consumed configuring individual components in
the physical infrastructure and verifying that network changes do not affect other devices that are
using the same networking infrastructure.
The need to pre-provision and configure networks is a major constraint to cloud deployments where
speed, agility, and flexibility are critical requirements. Pre-provisioned physical networks can allow for
the rapid creation of virtual networks and faster deployment times of workloads utilizing the virtual
network. As long as the physical network that you need is already available on the host where the
workload is to be deployed, this works well. However, if the network is not available on a given host,
you must find a host with the available network and spare capacity to run your workload in your
environment.
To get around this bottleneck requires a decoupling of virtual networks from their physical
counterparts. This, in turn, requires that you can programmatically recreate all physical networking
attributes that are required by workloads in the virtualized environment. Because network
virtualization supports the creation of virtual networks without modification of the physical network
infrastructure, it allows more rapid network provisioning.
3.2.5.2. NSX for vSphere Design
Each NSX instance is tied to a vCenter Server instance. The design decision to deploy two vCenter
Server instances per region (SDDC-VI-VC-001) requires deployment of two separate NSX instances
per region.
Table 68. NSX for vSphere Design Decision
Decision
ID
Design Decision
Design Justification
Design
Implications
SDDC-VISDN-001
Use two separate NSX
instances per region. One
instance is tied to the
Management vCenter
Server, and the other
instance is tied to the
Compute vCenter Server.
SDN capabilities offered by NSX,
such as load balancing and firewalls,
are crucial for the compute/edge
layer to support the cloud
management platform operations,
and also for the management
applications in the management
stack that need these capabilities.
You must install
and perform initial
configuration of the
four NSX instances
separately.
SDDC-VISDN-002
Pair NSX Manager
instances in a primarysecondary relationship
across regions for both
management and compute
workloads.
NSX can extend the logical
boundaries of the networking and
security services across regions. As
a result, workloads can be livemigrated and failed over between
regions without reconfiguring the
network and security constructs.
You must consider
that you can pair up
to eight NSX
Manager instances.
© 2016 VMware, Inc. All rights reserved.
Page 91 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 35. Architecture of NSX for vSphere
3.2.5.3. NSX Components
The following sections describe the components in the solution and how they are relevant to the
network virtualization design.
Consumption Layer
NSX for vSphere can be consumed by the cloud management platform (CMP), represented by vRealize
Automation, by using the NSX REST API and the vSphere Web Client.
Cloud Management Platform (CMP)
NSX for vSphere is consumed by vRealize Automation. NSX offers self-service provisioning of virtual
networks and related features from a service portal. Details of the service requests and their
orchestration are outside the scope of this document and can be referenced in the Cloud
Management Platform Design document.
API
NSX for vSphere offers a powerful management interface through its REST API.

A client can read an object by making an HTTP GET request to the object’s resource URL.
© 2016 VMware, Inc. All rights reserved.
Page 92 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

A client can write (create or modify) an object with an HTTP PUT or POST request that includes a
new or changed XML document for the object.

A client can delete an object with an HTTP DELETE request.
vSphere Web Client
The NSX Manager component provides a networking and security plug-in in the vSphere Web Client.
This plug-in provides an interface to consuming virtualized networking from the NSX Manager for
users that have sufficient privileges.
Table 69. Consumption Method Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implications
SDDC-VISDN-003
For the shared edge and
compute cluster NSX
instance, consumption is
accomplished by using
the vRealize Automation
services, the vSphere
Web Client, and the NSX
REST API.
vRealize Automation services
are used for the customerfacing portal. The vSphere Web
Client consumes NSX for
vSphere resources through the
Network and Security plug-in.
The NSX REST API offers the
potential of scripting repeating
actions and operations.
Customers typically
interact only indirectly
with NSX from the
vRealize Automation
portal. Administrators
interact with NSX from
the vSphere Web Client
and API.
SDDC-VISDN-004
For the management
cluster NSX instance,
consumption is only by
provider staff via the
vSphere Web Client and
the API.
Ensures that infrastructure
components are not modified by
tenants and/or non-provider
staff.
Tenants do not have
access to the
management stack
workloads.
NSX Manager
NSX Manager provides the centralized management plane for NSX for vSphere and has a one-to-one
mapping to vCenter Server workloads.
NSX Manager performs the following functions.

Provides the single point of configuration and the REST API entry-points for NSX in a vSphere
environment.

Deploys NSX Controller clusters, Edge distributed routers, and Edge service gateways in the form
of OVF appliances, guest introspection services, and so on.

Prepares ESXi hosts for NSX by installing VXLAN, distributed routing and firewall kernel modules,
and the User World Agent (UWA).

Communicates with NSX Controller clusters over REST and with hosts over the RabbitMQ
message bus. This internal message bus is specific to NSX for vSphere and does not require
setup of additional services.

Generates certificates for the NSX Controller instances and ESXi hosts to secure control plane
communications with mutual authentication.
NSX Controller
An NSX Controller performs the following functions.

Provides the control plane to distribute VXLAN and logical routing information to ESXi hosts.

Includes nodes that are clustered for scale-out and high availability.

Slices network information across cluster nodes for redundancy.
© 2016 VMware, Inc. All rights reserved.
Page 93 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

Removes requirement of VXLAN Layer 3 multicast in the physical network.

Provides ARP suppression of broadcast traffic in VXLAN networks.
NSX control plane communication occurs over the management network.
Table 70. NSX Controller Design Decision
Decision
ID
SDDC-VISDN-005
Design Decision
Design Justification
Design Implications
Deploy NSX Controller
instances in Universal Cluster
mode with three members to
provide high availability and
scale. Provision these three
nodes through the primary
NSX Manager instance.
The high availability of
NSX Controller
reduces the downtime
period in case of
failure of one physical
host.
The secondary NSX Manager
will not have active controllers
but will automatically import
the configuration of the
Universal Controllers that are
created in the primary NSX
Manager
NSX Virtual Switch
The NSX data plane consists of the NSX virtual switch. This virtual switch is based on the vSphere
Distributed Switch (VDS) with additional components to enable rich services. The add-on NSX
components include kernel modules (VIBs) which run within the hypervisor kernel and provide
services such as distributed logical router (DLR) and distributed firewall (DFW), and VXLAN
capabilities.
The NSX virtual switch abstracts the physical network and provides access-level switching in the
hypervisor. It is central to network virtualization because it enables logical networks that are
independent of physical constructs such as VLAN. Using an NSX virtual switch includes several
benefits.

Supports overlay networking and centralized network configuration. Overlay networking enables
the following capabilities.
o
Creation of a flexible logical Layer 2 overlay over existing IP networks on existing physical
infrastructure without the need to re-architect the data center networks.
o
Provisioning of communication (east/west and north/south) while maintaining isolation
between tenants.
o
Application workloads and virtual machines that are agnostic of the overlay network and
operate as if they were connected to a physical Layer 2 network.

Facilitates massive scale of hypervisors.

Because the NSX virtual switch is based on VDS, it provides a comprehensive toolkit for traffic
management, monitoring, and troubleshooting within a virtual network through features such as
port mirroring, NetFlow/IPFIX, configuration backup and restore, network health check, QoS, and
more.
Logical Switching
NSX logical switches create logically abstracted segments to which tenant virtual machines can be
connected. A single logical switch is mapped to a unique VXLAN segment and is distributed across
the ESXi hypervisors within a transport zone. The logical switch allows line-rate switching in the
hypervisor without the constraints of VLAN sprawl or spanning tree issues.
Distributed Logical Router
The NSX distributed logical router (DLR) is optimized for forwarding in the virtualized space, that is,
forwarding between VMs on VXLAN- or VLAN-backed port groups. DLR has the following
characteristics.

High performance, low overhead first hop routing
© 2016 VMware, Inc. All rights reserved.
Page 94 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

Scales with number of hosts

Up to 1,000 Logical Interfaces (LIFs) on each DLR
Distributed Logical Router Control Virtual Machine
The distributed logical router control virtual machine is the control plane component of the routing
process, providing communication between NSX Manager and the NSX Controller cluster through the
User World Agent (UWA). NSX Manager sends logical interface information to the control virtual
machine and the NSX Controller cluster, and the control virtual machine sends routing updates to the
NSX Controller cluster.
User World Agent
The User World Agent (UWA) is a TCP (SSL) client that facilitates communication between the ESXi
hosts and the NSX Controller instances as well as the retrieval of information from the NSX Manager
via interaction with the message bus agent.
VXLAN Tunnel Endpoint
VXLAN Tunnel Endpoints (VTEPs) are instantiated within the vSphere Distributed Switch to which the
ESXi hosts that are prepared for NSX for vSphere are connected. VTEPs are responsible for
encapsulating VXLAN traffic as frames in UDP packets and for the corresponding decapsulation.
VTEPs take the form of one or more VMkernel ports with IP addresses and are used both to
exchange packets with other VTEPs and to join IP multicast groups via Internet Group Membership
Protocol (IGMP). If you use multiple VTEPs, then you must select a teaming method.
Edge Services Gateway
The NSX Edge services gateways (ESGs) primary function is north/south communication, but it also
offers support for Layer 2, Layer 3, perimeter firewall, load balancing and other services such as SSLVPN and DHCP-relay.
Distributed Firewall
NSX includes a distributed kernel-level firewall known as the distributed firewall. Security enforcement
is done at the kernel and VM network adapter level. The security enforcement implementation
enables firewall rule enforcement in a highly scalable manner without creating bottlenecks on physical
appliances. The distributed firewall has minimal CPU overhead and can perform at line rate.
The flow monitoring feature of the distributed firewall displays network activity between virtual
machines at the application protocol level. This information can be used to audit network traffic, define
and refine firewall policies, and identify botnets.
Logical Load Balancer
The NSX logical load balancer provides load balancing services up to Layer 7, allowing distribution of
traffic across multiple servers to achieve optimal resource utilization and availability. The logical load
balancer is a service provided by the NSX Edge service gateway.
Table 71. NSX for vSphere Physical Network Requirements
Requirement
Comments
Any network that carries VXLAN traffic
must have an MTU size of 1600 or greater.
VXLAN packets cannot be fragmented. The MTU size
must be large enough to support extra encapsulation
overhead.
This design uses jumbo frames for VXLAN traffic.
© 2016 VMware, Inc. All rights reserved.
Page 95 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
For the hybrid replication mode, Internet
Group Management Protocol (IGMP)
snooping must be enabled on the Layer 2
switches to which ESXi hosts that
participate in VXLAN are attached. IGMP
querier must be enabled on the connected
router or Layer 3 switch.
IGMP snooping on Layer 2 switches is a requirement of
the hybrid replication mode. Hybrid replication mode is
the recommended replication mode for broadcast,
unknown unicast, and multicast (BUM) traffic when
deploying into an environment with large scale-out
potential. The traditional requirement for Protocol
Independent Multicast (PIM) is removed.
Dynamic routing support on the upstream
Layer 3 data center switches must be
enabled.
Enable a dynamic routing protocol supported by NSX
on the upstream data center switches to establish
dynamic routing adjacency with the ESGs.
NTP server must be available.
The NSX Manager requires NTP settings that
synchronize it with the rest of the vSphere environment.
Drift can cause problems with authentication. The NSX
Manager must be in sync with the vCenter Single SignOn service on the Platform Services Controller.
Forward and reverse DNS resolution for all
management VMs must be established.
The NSX Controller nodes do not require DNS entries.
3.2.5.4. NSX Specifications
The following table lists the components involved in the NSX for vSphere solution and the
requirements for installing and running them. The compute and storage requirements have been
taken into account when sizing resources to support the NSX for vSphere solution.
Note
NSX ESG sizing can vary with tenant requirements, so all options are listed.
Table 72. Resource Specification of NSX Components
VM
vCPU
Memory
Storage
Quantity per Stack Instance
NSX Manager
4
16 GB
60 GB
1
NSX Controller
4
4 GB
20 GB
3
NSX ESG
1 (Compact)
512 MB
(Compact)
2 (Large)
1 GB (Large)
4 (Quad
Large)
1 GB (Quad
Large)
6 (X-Large)
8 GB (XLarge)
512 MB
(Compact)
512 MB
(Large)
512 MB
(Quad Large)
Optional component. Deployment of
the NSX ESG varies per use case.
4.5 GB (XLarge)
(+4 GB with
swap)
DLR control
VM
1
512 MB
512 MB
Optional component. Varies with
use case. Typically 2 per HA pair.
Guest
introspection
2
1 GB
4 GB
Optional component. 1 per ESXi
host.
© 2016 VMware, Inc. All rights reserved.
Page 96 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
VM
vCPU
Memory
Storage
Quantity per Stack Instance
NSX data
security
1
512 MB
6 GB
Optional component. 1 per ESXi
host.
NSX Edge Service Gateway Sizing
The Quad Large model is suitable for high performance firewall abilities and the X-Large is suitable
for both high performance load balancing and routing.
You can convert between NSX Edge service gateway sizes upon demand using a non-disruptive
upgrade process, so the recommendation is to begin with the Large model and scale up if necessary.
A Large NSX Edge service gateway is suitable for medium firewall performance but as detailed later,
the NSX Edge service gateway does not perform the majority of firewall functions.
Note
Edge service gateway throughput is influenced by the WAN circuit, so an adaptable
approach, that is, converting as necessary, is recommended.
Table 73. NSX Edge Service Gateway Sizing Design Decision
Decision ID
Design Decision
Design Justification
Design
Implications
SDDC-VISDN-006
Use large size NSX
Edge service
gateways.
The large size provides all the performance
characteristics needed even in the event of a
failure.
None.
A larger size would also provide the
performance required but at the expense of
extra resources that wouldn't be used.
3.2.5.5. Network Virtualization Conceptual Design
The following diagram depicts the conceptual tenant architecture components and their relationship.
© 2016 VMware, Inc. All rights reserved.
Page 97 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 36. Conceptual Tenant Overview
In this document, tenant refers to a tenant of the cloud management platform within the compute/edge
stack or to a management application within the management stack.
The conceptual design has the following key components.

External Networks. Connectivity to and from external networks is through the perimeter firewall.
The main external network is the Internet.

Perimeter Firewall. The physical firewall exists at the perimeter of the data center. Each tenant
receives either a full instance or partition of an instance to filter external traffic.

Provider Logical Router (PLR). The PLR exists behind the perimeter firewall and handles
north/south traffic that is entering and leaving tenant workloads.

NSX for vSphere Distributed Logical Router (DLR). This logical router is optimized for
forwarding in the virtualized space, that is, between VMs, on VXLAN port groups or VLAN-backed
port groups.

Internal Non-Tenant Network. A single management network, which sits behind the perimeter
firewall but not behind the PLR. Enables customers to manage the tenant environments.

Internal Tenant Networks. Connectivity for the main tenant workload. These networks are
connected to a DLR, which sits behind the PLR. These networks take the form of VXLAN-based
© 2016 VMware, Inc. All rights reserved.
Page 98 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
NSX for vSphere logical switches. Tenant virtual machine workloads will be directly attached to
these networks.
3.2.5.6. Cluster Design for NSX for vSphere
Following the vSphere design, the NSX for vSphere design consists of a management stack and a
compute/edge stack in each region.
Management Stack
In the management stack, the underlying hosts are prepared for NSX for vSphere. The management
stack has these components.

NSX Manager instances for both stacks (management stack and compute/edge stack),

NSX Controller cluster for the management stack,

NSX ESG and DLR control VMs for the management stack.
Compute/Edge Stack
In the compute/edge stack, the underlying hosts are prepared for NSX for vSphere. The
compute/edge stack has these components.

NSX Controller cluster for the compute stack

All NSX Edge service gateways and DLR control VMs of the compute stack that are dedicated to
handling the north/south traffic in the data center. A separate edge stack helps prevent VLAN
sprawl because any external VLANs need only be trunked to the hosts in this cluster.
Table 74. vSphere Compute Cluster Split Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implications
SDDC-VISDN-007
For the compute stack,
do not use a dedicated
edge cluster.
Simplifies configuration and
minimizes the number of
hosts required for initial
deployment.
The NSX Controller instances,
NSX Edge services gateways,
and DLR control VMs of the
compute stack are deployed in
the shared edge and compute
cluster.
The shared nature of the
cluster will require the cluster
to be scaled out as compute
workloads are added so as to
not impact network
performance.
SDDC-VISDN-008
For the management
stack, do not use a
dedicated edge cluster.
The number of supported
management applications
does not justify the cost of
a dedicated edge cluster in
the management stack.
The NSX Controller instances,
NSX Edge service gateways,
and DLR control VMs of the
management stack are
deployed in the management
cluster.
SDDC-VISDN-009
Apply vSphere
Distributed Resource
Scheduler (DRS) antiaffinity rules to the NSX
components in both
stacks.
Using DRS prevents
controllers from running on
the same ESXi host and
thereby risking their high
availability capability.
Additional configuration is
required to set up anti-affinity
rules.
© 2016 VMware, Inc. All rights reserved.
Page 99 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
The logical design of NSX considers the vCenter Server clusters and define the place where each
NSX component runs.
Figure 37. Cluster Design for NSX for vSphere
High Availability of NSX for vSphere Components
The NSX Manager instances of both stacks run on the management cluster. vSphere HA protects the
NSX Manager instances by ensuring that the NSX Manager VM is restarted on a different host in the
event of primary host failure.
The NSX Controller nodes of the management stack run on the management cluster. The NSX for
vSphere Controller nodes of the compute stack run on the edge cluster. In both clusters, vSphere
Distributed Resource Scheduler (DRS) rules ensure that NSX for vSphere Controller nodes do not run
on the same host.
The data plane remains active during outages in the management and control planes although the
provisioning and modification of virtual networks is impaired until those planes become available
again.
The NSX Edge service gateways and DLR control VMs of the compute stack are deployed on the
edge cluster. The NSX Edge service gateways and DLR control VMs of the management stack run on
the management cluster.
© 2016 VMware, Inc. All rights reserved.
Page 100 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
NSX Edge components that are deployed for north/south traffic are configured in equal-cost multipath (ECMP) mode that supports route failover in seconds. NSX Edge components deployed for load
balancing utilize NSX HA. NSX HA provides faster recovery than vSphere HA alone because NSX HA
uses an active/passive pair of NSX Edge devices. By default, the passive Edge device becomes
active within 15 seconds. All NSX Edge devices are also protected by vSphere HA.
Scalability of NSX Components
A one-to-one mapping between NSX Manager instances and vCenter Server instances exists. If the
inventory of either the management stack or the compute stack exceeds the limits supported by a
single vCenter Server, then you can deploy a new vCenter Server instance, and must also deploy a
new NSX Manager instance. You can extend transport zones by adding more compute and edge
clusters until you reach the vCenter Server limits. Consider the limit of 100 DLRs per ESXi host
although the environment usually would exceed other vCenter Server limits before the DLR limit.
vSphere Distributed Switch Uplink Configuration
Each ESXi host utilizes two physical 10 Gb Ethernet adapters, associated with the uplinks on the
vSphere Distributed Switches to which it is connected. Each uplink is connected to a different top-ofrack switch to mitigate the impact of a single top-of-rack switch failure and to provide two paths in and
out of the SDDC.
Table 75. VTEP Teaming and Failover Configuration Design Decision
Decision
ID
Design Decision
Design Justification
Design Implications
SDDC-VISDN-010
Set up VXLAN Tunnel
Endpoints (VTEPs) to
use Route based on
SRC-ID for teaming and
failover configuration.
Allows for the utilization of
the two uplinks of the vDS
resulting in better
bandwidth utilization and
faster recovery from
network path failures.
Link aggregation such as LACP
between the top-of-rack (ToR)
switches and ESXi host must not
be configured in order to allow
dynamic routing to peer between
the ESGs and the upstream
switches.
3.2.5.7. Logical Switch Control Plane Mode Design
The control plane decouples NSX for vSphere from the physical network and handles the broadcast,
unknown unicast, and multicast (BUM) traffic within the logical switches. The control plane is on top of
the transport zone and is inherited by all logical switches that are created within it. It is possible to
override aspects of the control plane. The following options are available.

Multicast Mode. The control plane uses multicast IP addresses on the physical network. Use
multicast mode only when upgrading from existing VXLAN deployments. In this mode, you must
configure PIM/IGMP on the physical network.

Unicast Mode. The control plane is handled by the NSX Controllers and all replication occurs
locally on the host. This mode does not require multicast IP addresses or physical network
configuration.

Hybrid Mode. This mode is an optimized version of the unicast mode where local traffic
replication for the subnet is offloaded to the physical network. Hybrid mode requires IGMP
snooping on the first-hop switch and access to an IGMP querier in each VTEP subnet. Hybrid
mode does not require PIM.
© 2016 VMware, Inc. All rights reserved.
Page 101 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 38. Logical Switch Control Plane in Hybrid Mode
This design uses hybrid mode for control plane replication.
Table 76. Logical Switch Control Plane Mode Decision
Decision
ID
SDDC-VISDN-011
Design
Decision
Use hybrid
mode for
control plane
replication.
Design Justification
Design Implications
Offloading multicast processing to the
physical network reduces pressure on
VTEPs as the environment scales out. For
large environments, hybrid mode is
preferable to unicast mode. Multicast mode
is used only when migrating from existing
VXLAN solutions.
IGMP snooping must
be enabled on the ToR
physical switch and an
IGMP querier must be
available.
3.2.5.8. Transport Zone Design
A transport zone is used to define the scope of a VXLAN overlay network and can span one or more clusters
within one vCenter Server domain. One or more transport zones can be configured in an NSX for vSphere
solution. A transport zone is not meant to delineate a security boundary.
© 2016 VMware, Inc. All rights reserved.
Page 102 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 77. Transport Zones Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implications
SDDC-VISDN-012
For the compute stack,
use a single universal
transport zone that
encompasses all shared
edge and compute, and
compute clusters from
all regions..
A single Universal Transport
zone supports extending
networks and security policies
across regions. This allows
seamless migration of
applications across regions
either by cross vCenter vMotion
or by failover recovery with Site
Recovery Manager.
You must consider that
you can pair up to eight
NSX Manager instances. If
the solution grows past
eight NSX Manager
instances, you must deploy
a new primary manager
and new transport zone.
SDDC-VISDN-013
For the management
stack, use a single
universal transport zone
that encompasses all
management clusters.
A single Universal Transport
zone supports extending
networks and security policies
across regions. This allows
seamless migration of the
management applications
across regions either by crossvCenter vMotion or by failover
recovery with Site Recovery
Manager.
You must consider that
you can pair up to eight
NSX Manager instances. If
the solution grows past
eight NSX Manager
instances, you must deploy
a new primary manager
and new transport zone.
3.2.5.9. Routing Design
The routing design has to consider different levels of routing in the environment.

North/south. The Provider Logical Router (PLR) handles the north/south traffic to and from a
tenant and management applications inside of application virtual networks.

East/west. Internal east/west routing at the layer beneath the PLR deals with the application
workloads.
This design uses a universal distributed logical router (UDLR) which is a universal object that can
cross vCenter Server boundaries. The design decision table uses this abbreviation, for clarity when
the design decisions are viewed in a different context. The rest of this page uses the term distributed
logical router (DLR) to mean the same thing.
Table 78. Routing Model Design Decision
Decision
ID
Design Decision
Design Justification
Design Implications
SDDCVI-SDN014
Deploy NSX Edge
Services Gateways in an
ECMP configuration for
north/south routing in
both management and
shared edge and
compute clusters.
The NSX ESG is the
recommended device for
managing north/south traffic.
Using ECMP provides multiple
paths in and out of the SDDC.
This results in faster failover
times than deploying Edge
service gateways in HA mode.
ECMP requires 2 VLANS
for uplinks which adds an
additional VLAN over
traditional HA ESG
configurations.
© 2016 VMware, Inc. All rights reserved.
Page 103 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Decision
ID
Design Decision
Design Justification
Design Implications
SDDCVI-SDN015
Deploy a single NSX
UDLR for the
management cluster to
provide east/west routing
across all regions.
Using the UDLR reduces the hop
count between nodes attached to
it to 1. This reduces latency and
improves performance.
DLRs are limited to 1,000
logical interfaces. When
that limit is reached, a
new UDLR must be
deployed.
SDDCVI-SDN016
Deploy a single NSX
UDLR for the shared
edge and compute, and
compute clusters to
provide east/west routing
across all regions.
Using the UDLR reduces the hop
count between nodes attached to
it to 1. This reduces latency and
improves performance.
DLRs are limited to 1,000
logical interfaces. When
that limit is reached a new
UDLR must be deployed.
SDDCVI-SDN017
Deploy all NSX UDLRs
without the local egress
option enabled.
When local egress is enabled,
control of ingress traffic, is also
necessary (for example using
NAT). This becomes hard to
manage for little to no benefit.
All north/south traffic is
routed through Region A
until those routes are no
longer available. At that
time, all traffic dynamically
changes to Region B.
SDDCVI-SDN018
Use BGP as the
dynamic routing protocol
inside the SDDC.
Using BGP as opposed to OSPF
eases the implementation of
dynamic routing. There is no
need to plan and design access
to OSPF area 0 inside the SDDC.
OSPF area 0 varies based on
customer configuration.
BGP requires configuring
each ESG and UDLR with
the remote router that it
exchanges routes with.
SDDCVI-SDN019
Configure BGP Keep
Alive Timer to 1 and
Hold Down Timer to 3
between the UDLR and
all ESGs that provide
north/south routing.
With Keep Alive and Hold Timers
between the UDLR and ECMP
ESGs set low, a failure is
detected quicker, and the routing
table is updated faster.
If an ESXi host becomes
resource constrained, the
ESG running on that host
might no longer be used
even though it is still up.
SDDCVI-SDN020
Configure BGP Keep
Alive Timer to 4 and
Hold Down Timer to 12
between the ToR
switches and all ESGs
providing north/south
routing.
This provides a good balance
between failure detection
between the ToRs and the ESGs
and overburdening the ToRs with
keep alive traffic.
By using longer timers to
detect when a router is
dead, a dead router stays
in the routing table longer
and continues to send
traffic to a dead router.
Transit Network and Dynamic Routing
Dedicated networks are needed to facilitate traffic between the universal dynamic routers and edge
gateways, and to facilitate traffic between edge gateways and the top of rack switches. These
networks are used for exchanging routing tables and for carrying transit traffic.
© 2016 VMware, Inc. All rights reserved.
Page 104 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 79. Transit Network Design Decision
Decision
ID
Design Decision
Design Justification
Design Implications
SDDCVI-SDN021
Create a universal virtual switch
for use as the transit network
between the UDLR and ESGs.
The UDLR provides north/south
routing in both compute and
management stacks.
The universal virtual switch
allows the UDLR and all
ESGs across regions to
exchange routing
information.
Only the primary NSX
Manager can create
and manage universal
objects including this
UDLR.
SDDCVI-SDN022
Create two VLANs in each region.
Use those VLANs to enable
ECMP between the north/south
ESGs and the ToR switches.
This enables the ESGs to
have multiple equal-cost
routes and provides more
resiliency and better
bandwidth utilization in the
network.
Extra VLANs are
required.
Each ToR has an SVI on each
VLAN and each north/south ESG
also has an interface on each
VLAN.
3.2.5.10. Firewall Logical Design
The NSX Distributed Firewall is used to protect all management applications attached to application
virtual networks. To secure the SDDC, only other solutions in the SDDC and approved administration
IPs can directly communicate with individual components. External facing portals are accessible via a
load balancer virtual IP (VIP). This simplifies the design by having a single point of administration for
all firewall rules. The firewall on individual ESGs is set to allow all traffic. An exception are ESGs that
provide ECMP services, which require the firewall to be disabled.
Table 80. Tenant Firewall Design Decision
Decision
ID
Design Decision
Design Justification
Design Implications
SDDC-VISDN-023
For all ESGs deployed
as load balancers, set
the default firewall rule
to allow all traffic.
Restricting and granting access is
handled by the distributed
firewall. The default firewall rule
does not have to do it.
Explicit rules to allow
access to management
applications must be
defined in the distributed
firewall.
SDDC-VISDN-024
For all ESGs deployed
as ECMP north/south
routers, disable the
firewall.
Use of ECMP on the ESGs is a
requirement. Leaving the firewall
enabled, even in allow all traffic
mode, results in sporadic network
connectivity.
Services such as NAT and
load balancing cannot be
used when the firewall is
disabled.
3.2.5.11. Load Balancer Design
The ESG implements load balancing within NSX for vSphere. The ESG has both a Layer 4 and a
Layer 7 engine that offer different features, summarized in the following table.
© 2016 VMware, Inc. All rights reserved.
Page 105 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 81. Load Balancer Features of NSX Edge Services Gateway
Feature
Layer 4 Engine
Layer 7 Engine
TCP
Protocols
HTTP
TCP
HTTPS (SSL Pass-through)
HTTPS (SSL Offload)
Round Robin
Load balancing method
Source IP Hash
Least Connection
Round Robin
Source IP Hash
Least Connection
URI
TCP
Health checks
TCP
HTTP (GET, OPTION, POST)
HTTPS (GET, OPTION, POST)
TCP: SourceIP, MSRDP
Persistence (keeping client
connections to the same
back-end server)
TCP: SourceIP
Connection throttling
No
HTTP: SourceIP, Cookie
HTTPS: SourceIP, Cookie,
ssl_session_id
Client Side: Maximum concurrent
connections, Maximum new
connections per second
Server Side: Maximum concurrent
connections
High availability
Yes
Yes
Monitoring
View VIP (Virtual IP), Pool
and Server objects and stats
via CLI and API
View VIP, Pool and Server objects
and statistics by using CLI and API
View global stats for VIP
sessions from the vSphere
Web Client
Layer 7 manipulation
No
View global statistics about VIP
sessions from the vSphere Web
Client
URL block, URL rewrite, content rewrite
© 2016 VMware, Inc. All rights reserved.
Page 106 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 82. NSX for vSphere Load Balancer Design Decision
Decision
ID
Design Decision
Design Justification
Design Implications
SDDCVI-SDN025
Use the NSX load
balancer.
The NSX load balancer can support
the needs of the management
applications. Using another load
balancer would increase cost and
add another component to be
managed as part of the SDDC.
None.
SDDCVI-SDN026
Use a single NSX
load balancer in HA
mode for all
management
applications.
All management applications that
require a load balancer are on a
single virtual wire, having a single
load balancer keeps the design
simple.
One management
application owner could
make changes to the load
balancer that impact
another application.
3.2.5.12. Bridging Physical Workloads
NSX for vSphere offers VXLAN to Layer 2 VLAN bridging capabilities with the data path contained
entirely in the ESXi hypervisor. The bridge runs on the ESXi host where the DLR control VM is
located. Multiple bridges per DLR are supported.
Table 83.Virtual to Physical Interface Type Design Decision
Decision
ID
Design Decision
Design Justification
Design
Implications
SDDCVI-SDN027
Place all virtual machines, both
management and tenant, on VXLANbacked networks unless you must
satisfy an explicit requirement to use
VLAN-backed port groups for these
virtual machines. If VLAN-backed
port groups are required, connect
physical workloads that need to
communicate to virtualized
workloads to routed VLAN LIFs on a
DLR.
Bridging and routing are not
possible on the same logical
switch. As a result, it makes
sense to attach a VLAN LIF to a
distributed router or ESG and
route between the physical and
virtual machines. Use bridging
only where virtual machines
need access only to the physical
machines on the same Layer 2.
Access to
physical
workloads is
routed via the
DLR or ESG.
3.2.5.13. Region Connectivity
Regions must be connected to each other. Connection types could be point-to-point links, MPLS,
VPN Tunnels, etc. This connection will vary by customer and is out of scope for this design.
© 2016 VMware, Inc. All rights reserved.
Page 107 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 84. Inter-Site Connectivity Design Decisions
Decision
ID
Design Decision
Design Justification
Design
Implications
SDDC-VISDN-028
Provide a connection
between regions that is
capable of routing
between each pod.
NSX universal objects require connectivity
between NSX managers and ESXi host
VTEPs.
None.
To support cross-region authentication,
the vCenter Server and Platform Services
Controller design requires a single
vCenter Single Sign-On domain.
Portability of management and compute
workloads requires connectivity between
regions.
SDDC-VISDN-029
Make sure that the
latency in the connection
between the regions is
below 150ms.
A latency below 150 ms is required for the
following features.
None.
Cross-vCenter vMotion
The NSX design for the SDDC
3.2.5.14. Application Virtual Network
Management applications, such as vRealize Log Insight, VMware vRealize Automation, VMware
vRealize Operations Manager, or VMware vRealize Orchestrator, leverage a traditional 3-tier
client/server architecture with a presentation tier (user interface), functional process logic tier, and
data tier. The validated design for micro-segmentation uses only vRealize Log Insight.
This architecture requires a load balancer for presenting end-user facing services. Implement each of
these management applications as their own trust zone (or multiple trust zones) and isolate
management applications from each other, but also from the external-to-SDDC security zone.
Table 85. Isolated Management Applications Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implications
SDDC-VISDN-030
Place the following
management
applications on an
application virtual
network.
Access to the
management
applications is only
through published
access points.
The virtual application network is
fronted by an NSX Edge device for
load balancing and the distributed
firewall to isolate applications from
each other and external users. Direct
access to virtual application networks
is controlled by distributed firewall
rules.
Using only two
application virtual
networks simplifies the
design by sharing Layer
2 networks with
applications based on
their needs.
A single /24 subnet is used for each
application virtual network. IP
management becomes critical to
ensure no shortage of IP addresses
will appear in the future.
vRealize Log Insight
SDDC-VISDN-031
Create two
application virtual
networks.
Each region has a
dedicated
application virtual
network for
management
applications .
© 2016 VMware, Inc. All rights reserved.
Page 108 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Having software-defined networking based on NSX in the management stack makes all NSX features
available to the management applications.
This approach to network virtualization service design improves security and mobility of the
management applications, and reduces the integration effort with existing customer networks.
Figure 39. Virtual Application Network Components and Design
3.2.5.15. Tenant Onboarding Process Automation
Certain configuration choices might later facilitate the tenant onboarding process.

Create the primary NSX ESG to act as the tenant PLR and the logical switch that forms the transit
network for use in connecting to the DLR.

Connect the primary NSX ESG uplinks to the external networks

Connect the primary NSX ESG internal interface to the transit network.
© 2016 VMware, Inc. All rights reserved.
Page 109 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

Create the NSX DLR to provide routing capabilities for tenant internal networks and connect the
DLR uplink to the transit network.

Create any tenant networks that are known up front and connect them to the DLR.
3.2.5.16. Virtual Network Design Example
The Detailed Example for vRealize Automation Networking illustration shows an example for
implementing a management application virtual network. The example service is vRealize
Automation, but any other 3-tier application would look similar.
Figure 40. Detailed Example for vRealize Automation Networking
The example is set up as follows.

You deploy vRealize Automation on the application virtual network that is used to fail over
applications between regions. This network is provided by a VXLAN virtual wire (orange network
in Detailed Example for vRealize Automation Networking).

The network that is used by vRealize Automation connects to external networks through NSX for
vSphere. NSX ESGs and the UDLR route traffic between the application virtual networks and the
public network.

Services such as a Web GUI, which must be available to the end users of vRealize Automation,
are accessible via the NSX Edge load balancer.
© 2016 VMware, Inc. All rights reserved.
Page 110 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
The following table shows an example of a mapping from application virtual networks to IPv4 subnets.
The actual mapping depends on the customer environment and is based on available IP subnets.
Note
The following IP ranges are samples. Your actual implementation depends on your
environment.
Table 86. Application Virtual Network Configuration
Application Virtual
Network
Management Applications
Internal IPv4 Subnet
Mgmt-RegionA01VXLAN
vRealize Log Insight
192.168.31.0/24
Mgmt-RegionB01VXLAN
vRealize Log Insight
192.168.32.0/24
3.2.6 Shared Storage Design
Well-designed shared storage provides the basis for an SDDC and has the following benefits:

Prevents unauthorized access to business data.

Protects data from hardware and software failures.

Protects data from malicious or accidental corruption.
Follow these guidelines when designing shared storage for your environment.

Optimize the storage design to meet the diverse needs of applications, services, administrators,
and users.

Strategically align business applications and the storage infrastructure to reduce costs, boost
performance, improve availability, provide security, and enhance functionality.

Provide multiple tiers of storage to match application data access to application requirements.

Design each tier of storage with different performance, capacity, and availability characteristics.
Because not every application requires expensive, high-performance, highly available storage,
designing different storage tiers reduces cost.
3.2.6.1. Shared Storage Platform
You can choose between traditional storage, VMware vSphere Virtual Volumes and VMware Virtual
SAN storage.

Traditional Storage. Fibre Channel, NFS, and iSCSI are mature and viable options to support
virtual machine needs.

VMware Virtual SAN Storage. VMware Virtual SAN is a software-based distributed storage
platform that combines the compute and storage resources of VMware ESXi hosts. When you
design and size a Virtual SAN cluster, hardware choices are more limited than for traditional
storage.

Virtual Volumes. This design does not leverage VMware vSphere Virtual Volumes because
Virtual Volumes does not support Site Recovery Manager.
3.2.6.2. Background on Traditional Storage and VMware Virtual SAN Storage
Traditional Storage
Fibre Channel, NFS, and iSCSI are mature and viable options to support virtual machine needs.
Your decision to implement one technology or another can be based on performance and
functionality, and on considerations like the following:
© 2016 VMware, Inc. All rights reserved.
Page 111 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

The organization’s current in-house expertise and installation base

The cost, including both capital and long-term operational expenses

The organization’s current relationship with a storage vendor
VMware Virtual SAN
VMware Virtual SAN is a software-based distributed storage platform that combines the compute and
storage resources of ESXi hosts. It provides a simple storage management experience for the user.
This solution makes software-defined storage a reality for VMware customers. However, you must
carefully consider supported hardware options when sizing and designing a Virtual SAN cluster.
Note: The VMware Validated Design for Micro-Segmentation does not use Virtual SAN storage.
Information about Virtual SAN is included as background information.
3.2.6.3. Storage Type Comparison
ESXi hosts support a variety of storage types. Each storage type supports different vSphere features.
Table 87. Network Shared Storage Supported by ESXi Hosts
Technology
Protocols
Transfers
Interface
Fibre Channel
FC/SCSI
Block access of
data/LUN
Fibre Channel HBA
Fibre Channel over
Ethernet
FCoE/SCSI
Block access of
data/LUN
Converged network adapter
(hardware FCoE)
NIC with FCoE support (software
FCoE)
iSCSI
IP/SCSI
Block access of
data/LUN
iSCSI HBA or iSCSI enabled NIC
(hardware iSCSI)
Network Adapter (software iSCSI)
NAS
IP/NFS
File (no direct LUN
access)
Network adapter
Virtual SAN
IP
Block access of
data
Network adapter
Table 88. vSphere Features Supported by Storage Type
Type
vSphere
vMotion
Datastore
Raw Device
Mapping
(RDM)
Application or
Block-level
Clustering
HA/DRS
Storage APIs
Data
Protection
Local Storage
Yes
VMFS
No
Yes
No
Yes
Fibre Channel
/ Fibre
Channel over
Ethernet
Yes
VMFS
Yes
Yes
Yes
Yes
iSCSI
Yes
VMFS
Yes
Yes
Yes
Yes
© 2016 VMware, Inc. All rights reserved.
Page 112 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Type
vSphere
vMotion
Datastore
Raw Device
Mapping
(RDM)
Application or
Block-level
Clustering
HA/DRS
Storage APIs
Data
Protection
NAS over NFS
Yes
NFS
No
No
Yes
Yes
Virtual SAN
Yes
Virtual
SAN
No
No
Yes
Yes
3.2.6.4. Shared Storage Logical Design
The VMware Validated Design for Micro-Segmentation uses NFS for shared storage for both clusters.
The VMware Validated Design for the Software-Defined Datacenter uses both Virtual SAN and NFS,
as follows:

Management clusters use Virtual SAN for primary storage and NFS for secondary storage

Shared edge and compute clusters can use FC/FCoE, iSCSI, NFS, or Virtual SAN storage. No
specific guidance is given as user workloads and other factors determine storage type and SLA
for user workloads
Figure 41. Logical Storage Design
© 2016 VMware, Inc. All rights reserved.
Page 113 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 89. Storage Type Design Decisions
Decision ID
Design Decision
Design Justification
Design Implication
SDDC-VIStorage001
Use NFS storage
For the VMware Validated
Design for MicroSegmentation, using only
NFS storage simplifies the
design.
The full design for the
Software-Defined Data
Center uses both Virtual
SAN and NFS. Additional
storage configuration is
needed when expanding to
that design.
SDDC-VIStorage002
In the shared edge
and compute
cluster, ensure that
at least 20% of free
space is always
available.
If the datastore runs out of
free space, services that
include the NSX Edge core
network services fail. To
prevent this, maintain
adequate free space.
Monitoring and capacity
management are critical and
must be performed
proactively.
3.2.6.5. Storage Tiering
Today’s enterprise-class storage arrays contain multiple drive types and protection mechanisms. The
storage, server, and application administrators face challenges when selecting the correct storage
configuration for each application being deployed in the environment. Virtualization can make this
problem more challenging by consolidating many different application workloads onto a small number
of large devices. Given this challenge, administrators might use single storage type for every type of
workload without regard to the needs of the particular workload. However, not all application
workloads have the same requirements, and storage tiering allows for these differences by creating
multiple levels of storage with varying degrees of performance, reliability and cost, depending on the
application workload needs.
The most mission-critical data typically represents the smallest amount of data and offline data
represents the largest amount. Details differ for different organizations.
To determine the storage tier for application data, determine the storage characteristics of the
application or service.

I/O operations per second (IOPS) requirements

Megabytes per second (MBps) requirements

Capacity requirements

Availability requirements

Latency requirements
After you determine the information for each application, you can move the application to the storage
tier with matching characteristics.

Consider any existing service-level agreements (SLAs)

Move data between storage tiers during the application life cycle as needed.
3.2.6.6. VMware Hardware Acceleration API/CLI for Storage
The VMware Hardware Acceleration API/CLI for storage (previously known as vStorage APIs for
Array Integration or VAAI), supports a set of ESXCLI commands for enabling communication between
ESXi hosts and storage devices. The APIs define a set of storage primitives that enable the ESXi host
to offload certain storage operations to the array. Offloading the operations reduces resource
overhead on the ESXi hosts and can significantly improve performance for storage-intensive
operations such as storage cloning, zeroing, and so on. The goal of hardware acceleration is to help
© 2016 VMware, Inc. All rights reserved.
Page 114 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
storage vendors provide hardware assistance to speed up VMware I/O operations that are more
efficiently accomplished in the storage hardware.
Without the use of VAAI, cloning or migration of virtual machines by the VMkernel data mover
involves software data movement. The data mover issues I/O to read and write blocks to and from the
source and destination datastores. With VAAI, the data mover can use the API primitives to offload
operations to the array when possible. For example, when you copy a virtual machine disk file
(VMDK file) from one datastore to another inside the same array, the data mover directs the array to
make the copy completely inside the array. If you invoke a data movement operation and the
corresponding hardware offload operation is enabled, the data mover first attempts to use hardware
offload. If the hardware offload operation fails, the data mover reverts to the traditional software
method of data movement.
In nearly all cases, hardware data movement performs significantly better than software data
movement. It consumes fewer CPU cycles and less bandwidth on the storage fabric. Timing
operations that use the VAAI primitives and use esxtop to track values such as CMDS/s, READS/s,
WRITES/s, MBREAD/s, and MBWRTN/s of storage adapters during the operation show performance
improvements.
Table 90. VAAI Design Decisions
Decision
ID
Design
Decision
Design Justification
Design Implication
SDDC-VIStorage003
Select an
array that
supports VAAI
over NAS
(NFS).
VAAI offloads tasks to the array
itself, enabling the ESXi hypervisor
to use its resources for application
workloads and not become a
bottleneck in the storage
subsystem.
Not all VAAI arrays support
VAAI over NFS. A plugin
from the array vendor is
required to enable this
functionality.
VAAI is required to support the
desired number of virtual machine
lifecycle operations.
3.2.6.7. Virtual Machine Storage Policies
You can create a storage policy for a virtual machine to specify which storage capabilities and
characteristics are the best match for this virtual machine.
Note
VMware Virtual SAN uses storage policies to allow specification of the characteristics of
virtual machines, so you can define the policy on an individual disk level rather than at the
volume level for Virtual SAN.
You can identify the storage subsystem capabilities by using the VMware vSphere API for Storage
Awareness or by using a user-defined storage policy.

VMware vSphere API for Storage Awareness (VASA). With vSphere API for Storage
Awareness, storage vendors can publish the capabilities of their storage to VMware vCenter
Server, which can display these capabilities in its user interface.

User-defined storage policy. Defined by using the VMware Storage Policy SDK or VMware
vSphere PowerCLI (see the Sample Scripts), or from the vSphere Web Client.
You can assign a storage policy to a virtual machine and periodically check for compliance so that the
virtual machine continues to run on storage with the correct performance and availability
characteristics.
You can associate a virtual machine with a virtual machine storage policy when you create, clone, or
migrate that virtual machine. If a virtual machine is associated with a storage policy, the vSphere Web
Client shows the datastores that are compatible with the policy. You can select a datastore or
datastore cluster. If you select a datastore that does not match the virtual machine storage policy, the
© 2016 VMware, Inc. All rights reserved.
Page 115 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
vSphere Web Client shows that the virtual machine is using non-compliant storage. See Creating and
Managing vSphere Storage Policies.
Table 91. Virtual Machine Storage Policy Design Decisions
Decision ID
Design Decision
Design Justification
Design Implication
SDDC-VIStorage-004
Do not use
customized virtual
machine storage
policies.
The default storage
policy is adequate for
the management cluster
VMs.
If 3rd party or additional VMs
have different storage
requirements, additional VM
storage policies might be
required.
3.2.6.8. vSphere Storage I/O Control Background Information
VMware vSphere Storage I/O Control allows cluster-wide storage I/O prioritization, which results in
better workload consolidation and helps reduce extra costs associated with overprovisioning.
vSphere Storage I/O Control extends the constructs of shares and limits to storage I/O resources.
You can control the amount of storage I/O that is allocated to virtual machines during periods of I/O
congestion, so that more important virtual machines get preference over less important virtual
machines for I/O resource allocation.
When vSphere Storage I/O Control is enabled on a datastore, the ESXi host monitors the device
latency when communicating with that datastore. When device latency exceeds a threshold, the
datastore is considered to be congested and each virtual machine that accesses that datastore is
allocated I/O resources in proportion to their shares. Shares are set on a per-virtual machine basis
and can be adjusted.
vSphere Storage I/O Control has several requirements, limitations, and constraints.

Datastores that are enabled with vSphere Storage I/O Control must be managed by a single
vCenter Server system.

Storage I/O Control is supported on Fibre Channel-connected, iSCSI-connected, and NFSconnected storage. RDM is not supported.

Storage I/O Control does not support datastores with multiple extents.

Before using vSphere Storage I/O Control on datastores that are backed by arrays with
automated storage tiering capabilities, check the VMware Compatibility Guide whether the
storage array has been certified a compatible with vSphere Storage I/O Control.
Table 92. Storage I/O Control Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDC-VIStorage005
Enable Storage I/O
Control with the
default values on the
NFS datastores.
Storage I/O Control ensures that
all virtual machines on a
datastore receive an equal
amount of I/O.
Virtual machines that
use more I/O are
throttled to allow other
virtual machines access
to the datastore only
when contention occurs
on the datastore.
© 2016 VMware, Inc. All rights reserved.
Page 116 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
SDDC-VIStorage006
In the shared edge
and compute cluster,
enable Storage I/O
Control with default
values.
Storage I/O Control ensures that
all virtual machines on a
datastore receive an equal
amount of I/O. For the NSX
components in this shared
cluster it is critical that they have
equal access to the datastore to
avoid network bottlenecks.
Virtual machines that
use more I/O are
throttled to allow other
virtual machines access
to the datastore only
when contention occurs
on the datastore.
3.2.6.9. Datastore Cluster Design
A datastore cluster is a collection of datastores with shared resources and a shared management
interface. Datastore clusters are to datastores what clusters are to ESXi hosts. After you create a
datastore cluster, you can use vSphere Storage DRS to manage storage resources.
vSphere datastore clusters group similar datastores into a pool of storage resources. When vSphere
Storage DRS is enabled on a datastore cluster, vSphere automates the process of initial virtual
machine file placement and balances storage resources across the cluster to avoid bottlenecks.
vSphere Storage DRS considers datastore space usage and I/O load when making migration
recommendations.
When you add a datastore to a datastore cluster, the datastore's resources become part of the
datastore cluster's resources. The following resource management capabilities are also available for
each datastore cluster.
Table 93. Resource Management Capabilities Available for Datastores
Capability
Description
Space
utilization load
balancing
You can set a threshold for space use. When space use on a datastore
exceeds the threshold, vSphere Storage DRS generates recommendations or
performs migrations with vSphere Storage vMotion to balance space use
across the datastore cluster.
I/O latency
load balancing
You can configure the I/O latency threshold to avoid bottlenecks. When I/O
latency on a datastore exceeds the threshold, vSphere Storage DRS generates
recommendations or performs vSphere Storage vMotion migrations to help
alleviate high I/O load.
Anti-affinity
rules
You can configure anti-affinity rules for virtual machine disks to ensure that the
virtual disks of a virtual machine are kept on different datastores. By default, all
virtual disks for a virtual machine are placed on the same datastore.
You can enable vSphere Storage I/O Control or vSphere Storage DRS for a datastore cluster. You
can enable the two features separately, even though vSphere Storage I/O control is enabled by
default when you enable vSphere Storage DRS.
3.2.6.10. vSphere Storage DRS Background Information
vSphere Storage DRS supports automating the management of datastores based on latency and
storage utilization. When configuring vSphere Storage DRS, verify that all datastores use the same
version of VMFS and are on the same storage subsystem. Because vSphere Storage vMotion
performs the migration of the virtual machines, confirm that all prerequisites are met.
vSphere Storage DRS provides a way of balancing usage and IOPS among datastores in a storage
cluster:

Initial placement of virtual machines is based on storage capacity.
© 2016 VMware, Inc. All rights reserved.
Page 117 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

vSphere Storage DRS uses vSphere Storage vMotion to migrate virtual machines based on
storage capacity.

vSphere Storage DRS uses vSphere Storage vMotion to migrate virtual machines based on I/O
latency.

You can configure vSphere Storage DRS to run in either manual mode or in fully automated
mode.
vSphere Storage I/O Control and vSphere Storage DRS manage latency differently.

vSphere Storage I/O Control distributes the resources based on virtual disk share value after a
latency threshold is reached.

vSphere Storage DRS measures latency over a period of time. If the latency threshold of vSphere
Storage DRS is met in that time frame, vSphere Storage DRS migrates virtual machines to
balance latency across the datastores that are part of the cluster.
When making a vSphere Storage design decision, consider these points:

Use vSphere Storage DRS where possible.

vSphere Storage DRS provides a way of balancing usage and IOPS among datastores in a
storage cluster:
o
Initial placement of virtual machines is based on storage capacity.
o
vSphere Storage vMotion is used to migrate virtual machines based on storage capacity.
o
vSphere Storage vMotion is used to migrate virtual machines based on I/O latency.
o
vSphere Storage DRS can be configured in either manual or fully automated modes
3.2.6.11.
NFS Storage Design
This NFS design does not give specific vendor or array guidance. Consult your storage vendor for the
configuration settings appropriate for your storage array.
NFS Storage Concepts
NFS (Network File System) presents file devices to an ESXi host for mounting over a network. The
NFS server or array makes its local file systems available to ESXi hosts. The ESXi hosts access the
metadata and files on the NFS array or server using a RPC-based protocol. NFS is implemented
using Standard NIC that is accessed using a VMkernel port (vmknic).
NFS Load Balancing
No load balancing is available for NFS/NAS on vSphere because it is based on single session
connections. You can configure aggregate bandwidth by creating multiple paths to the NAS array, and
by accessing some datastores via one path, and other datastores via another path. You can configure
NIC Teaming so that if one interface fails, another can take its place. However these load balancing
techniques work only in case of a network failure and might not be able to handle error conditions on
the NFS array or on the NFS server. The storage vendor is often the source for correct configuration
and configuration maximums.
NFS Versions
vSphere is compatible with both NFS version 3 and version 4.1; however, not all features can be
enabled when connecting to storage arrays that use NFS v4.1.
Table 94. NFS Version Design Decision
Decision
ID
Design Decision
Design Justification
© 2016 VMware, Inc. All rights reserved.
Page 118 of 130
Design Implication
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
SDDCNFS-001
Use NFS v3 for all
NFS hosted
datastores.
NFS v4.1 datastores are not
supported with Storage I/O Control
and with Site Recovery Manager.
NFS v3 does not
support Kerberos
authentication.
Storage Access
NFS v3 traffic is transmitted in an unencrypted format across the LAN. Therefore, best practice is to
use NFS storage on trusted networks only and to isolate the traffic on dedicated VLANs.
Many NFS arrays have some built-in security, which enables them to control the IP addresses that
can mount NFS exports. Best practice is to use this feature to determine which ESXi hosts can mount
the volumes that are being exported and have read/write access to those volumes. This prevents
unapproved hosts from mounting the NFS datastores.
Exports
All NFS exports are shared directories that sit on top of a storage volume. These exports control the
access between the endpoints (ESXi hosts) and the underlying storage system. Multiple exports can
exist on a single volume, with different access controls on each.
Table 95. NFS Export Sizing
Export Size per Region
Size
vSphere Data Protection
4 TB
vRealize Log Insight Archive
1 TB
vRealize Automation Content Library
1 TB
Figure 42. NFS Storage Exports
© 2016 VMware, Inc. All rights reserved.
Page 119 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Table 96. NFS Export Design Decisions
Decision ID
Design Decision
Design Justification
Design Implication
SDDCNFS-002
Create 3 exports to
support the management
components.
The storage requirements of
these management
components are separate
from the primary storage.
Having multiple
exports can introduce
operational overhead.
vSphere Data Protection
vRealize Log Insight
Archive
vRealize Automation
Content Library
SDDCNFS-003
Place the vSphere Data
Protection export on its
own separate volume as
per SDDC-PHY-STO008
vSphere Data Protection is I/O
intensive. vSphere Data
Protection or other
applications suffer if vSphere
Data Protection is placed on a
shared volume.
Dedicated exports can
add management
overhead to storage
administrators.
SDDCNFS-004
For each export, limit
access to only the
application VMs or hosts
requiring the ability to
mount the storage.
Limiting access helps ensure
the security of the underlying
data.
Securing exports
individually can
introduce operational
overhead.
NFS Datastores
Within vSphere environments, ESXi hosts mount NFS exports as a file-share instead of using the
VMFS clustering filesystem. For this design, only secondary storage is being hosted on NFS storage.
The datastore construct within vSphere mounts some of the exports, depending on their intended use.
For the vRealize Log Insight archive data, the application maps directly to the NFS export and no
vSphere Datastore is required.
Table 97. NFS Datastore Design Decision
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCNFS-005
Create 2 datastores
for use across the
following clusters.
The application VMs using
these data assume that all
hosts in the vSphere cluster
can access the datastores.
Do not use the NFS
datastores as primary VM
storage in the management
cluster even though that is
possible.
Management cluster:
vSphere Data
Protection
Shared Edge and
Compute cluster:
vRealize Automation
Content Library
© 2016 VMware, Inc. All rights reserved.
Page 120 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
3.3
Operations Infrastructure Design
Operations Management is a required element of a software-defined data center. Monitoring
operations support in vRealize Operations Manager and vRealize Log Insight provides capabilities for
performance and capacity management of related infrastructure and cloud management components.
Figure 43. Operations Infrastructure Conceptual Design
3.3.1 vRealize Log Insight Design
vRealize Log Insight design enables real-time logging for all components that build up the
management capabilities of the SDDC in a dual-region setup.
3.3.1.1. Logical Design
In a multi-region Software Defined Data Center (SDDC) deploy a vRealize Log Insight cluster in each
region that consists of three nodes. This allows for continued availability and increased log ingestion
rates.
Figure 44. Logical Design of vRealize Log Insight
© 2016 VMware, Inc. All rights reserved.
Page 121 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
3.3.1.2. Sources of Log Data
vRealize Log Insight collects logs as to provide monitoring information about the SDDC from a central
location.
vRealize Log Insight collects log events from the following virtual infrastructure and cloud
management components:


Management vCenter Server
o
Platform Services Controller
o
vCenter Server
Compute vCenter Server
o
Platform Services Controller
o
vCenter Server

Management, shared edge and compute ESXi hosts

NSX for vSphere for the management and for the shared compute and edge clusters


o
NSX Manager
o
NSX Controller instances
o
NSX Edge instances
vRealize Automation
o
vRealize Orchestrator
o
vRealize Automation components
vRealize Operations Manager
o
Analytics cluster nodes
3.3.1.3. Cluster Nodes
The vRealize Log Insight cluster consists of one master node and two worker nodes. You enable the
Integrated Load Balancer (ILB) on the cluster to have vRealize Log Insight to balance incoming traffic
fairly among available nodes. vRealize Log Insight clients, using both the Web user interface, and
ingestion through syslog or the Ingestion API, connect to vRealize Log Insight at the ILB address.
vRealize Log Insight cluster can scale out to 6 nodes, that is, one master and 5 worker nodes.
Table 98. Cluster Node Configuration Design Decision
Decision
ID
Design Decision
Design Justification
Design
Implication
SDDCOPS-LOG001
Deploy vRealize Log Insight in
a cluster configuration of 3
nodes with an integrated load
balancer: one master and two
worker nodes.
Provides high availability. Using
the integrated load balancer
simplifies the Log Insight
deployment, and prevents a
single point of failure.
You must size
each node
identically.
3.3.1.4. Sizing
By default, a vRealize Log Insight virtual appliance has 2 vCPUs, 4 GB of virtual memory, and 144
GB of disk space provisioned. vRealize Log Insight uses 100 GB of the disk space to store raw, index
and metadata.
Sizing Nodes
© 2016 VMware, Inc. All rights reserved.
Page 122 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
To accommodate all of log data from the products in the SDDC, you must size the Log Insight nodes
properly.
Table 99. Compute Resources for a vRealize Log Insight Medium-Size Node
Attribute
Specification
Appliance size
Medium
Number of CPUs
8
Memory
16 GB
IOPS
1,000 IOPS
Amount of processed log data
38 GB/day
Number of process log messages
7,500
Environment
Up to 250 syslog connections per node
Sizing Storage
Sizing is based on IT organization requirements, but assuming that you want to retain 7 days of data,
you can use the following calculations.
For 250 syslog sources at a rate of 150 MB of logs ingested per-day per-source over 7 days:
250 sources * 150 MB of log data ≈ 37 GB log data per-day
37 GB * 7 days ≈ 260 GB log data per vRealize Log Insight node
260 GB * 1.7 overhead index ≈ 450 GB
Based on this example, you must provide 270 GB of storage space per node when you deploy the
medium-size vRealize Log Insight virtual appliance. You must add additional space of approximately
190 GB.
Note
vRealize Log Insight supports virtual hard disks of up to 2 TB. If more capacity is needed, add
another virtual hard disk. Do not extend existing retention virtual disks.
Table 100. Compute Resources for the vRealize Log Insight Nodes Design Decision
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCOPS-LOG002
Deploy vRealize
Log Insight nodes
of medium size.
Accommodates the
number of expected
syslog connections.
You must increase the size of
the nodes if you configure Log
Insight to monitor additional
syslog sources.
SDDCOPS-LOG003
Add (190 GB)*
additional storage
per node.
Used to ensure 7 days of
data retention.
Additional storage space is
required.
3.3.1.5. Networking Design
In both regions, the vRealize Log Insight instances are connected to the region-specific management
VXLANs Mgmt-RegionA01-VXLAN and Mgmt-RegionB01-VXLAN. Each vRealize Log Insight
instance is deployed within the shared management application isolated network.
© 2016 VMware, Inc. All rights reserved.
Page 123 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Figure 45. Networking Design for the vRealize Log Insight Deployment
3.3.1.6. Application Isolated Network Design
Each of the two instances of the vRealize Log Insight deployment is installed in the shared nonfailover application network for the region. An NSX universal distributed logical router (UDLR) is
configured at the front of each shared application network to provide network isolation. This
networking design has the following features:

All nodes have routed access to the vSphere management network through the Management
NSX UDLR for the home region.

Routing to the vSphere management network and the external network is dynamic, and is based
on the Border Gateway Protocol (BGP).
For more information about the networking configuration of the application isolated networks for
vRealize Log Insight, see NSX Design.
Table 101. vRealize Log Insight Isolated Network Design Decisions
Decision ID
Design Decision
Design Justification
Design
Implication
SDDC-OPSLOG-004
Deploy vRealize Log Insight on
the shared management region
VXLAN.
Secures the vRealize Log
Insight instances.
None
Provides a consistent
deployment model for
management applications.
3.3.1.7. IP Subnets
You can allocate the following example subnets to the vRealize Log Insight deployment:
Table 102. IP Subnets in the Application Isolated Networks
vRealize Log Insight Cluster
IP Subnet
Region A
192.168.31.0/24
© 2016 VMware, Inc. All rights reserved.
Page 124 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
Region B
192.168.32.0/24
Table 103. IP Subnets Design Decision
Decision
ID
Design Decision
Design Justification
Design
Implication
SDDCOPS-LOG005
Allocate separate
subnets for each
application isolated
network.
Satisfies the requirement for log
forwarding between the two vRealize Log
Insight instances to place each instance
on its own unique subnet.
None.
3.3.1.8. DNS Names
vRealize Log Insight node name resolution uses a region-specific suffix, such as
sfo01.rainpole.local or lax01.rainpole.local, including the load balancer virtual IP
addresses (VIPs). The Log Insight components in both regions have the following node names:
Table 104. DNS Names of the vRealize Log Insight Nodes
DNS Name
Role
Region
vrli-cluster-01.sfo01.rainpole.local
Log Insight ILB VIP
A
vrli-mstr01.sfo01.rainpole.local
Master node
A
vrli-wrkr01.sfo01.rainpole.local
Worker node
A
vrli-wrkr02.sfo01.rainpole.local
Worker node
A
vrli-cluster-51.lax01.rainpole.local
Log Insight ILB VIP
B
vrli-mstr51.lax01.rainpole.local
Master node
B
vrli-wrkr51.lax01.rainpole.local
Worker node
B
vrli-wrkr52.lax01.rainpole.local
Worker node
B
Table 105. DNS Names Design Decision
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCOPSLOG-006
Configure forward and
reverse DNS records for
all vRealize Log Insight
nodes and VIPs.
All nodes are
accessible by using
fully-qualified domain
names instead of by
using IP addresses
only.
You must manually provide a
DNS record for each node and
VIP.
© 2016 VMware, Inc. All rights reserved.
Page 125 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
SDDCOPSLOG-007
For all applications that
failover between regions
(such as vRealize
Automation and vRealize
Operations Manager), use
the FQDN of the vRealize
Log Insight Region A VIP
when you configure
logging.
Support logging when
not all management
applications are failed
over to Region B. For
example, only one
application is moved to
Region B.
If vRealize Automation and
vRealize Operations Manager
are failed over to Region B
and the vRealize Log Insight
cluster is no longer available
in Region A, update the A
record on the child DNS
server to point to the vRealize
Log Insight cluster in Region
B.
3.3.1.9. Retention and Archiving
Configure archive and retention parameters of vRealize Log Insight according to the company policy
for compliance and governance.
Retention
vRealize Log Insight virtual appliances contain three default virtual disks and can use addition virtual
disks for storage, for example, hard disk 4.
Table 106. Virtual Disk Configuration in the vRealize Log Insight Virtual Appliance
Hard Disk
Size
Usage
Hard disk 1
12.125 GB
Root file system
Hard disk 2
270 GB for mediumsize deployment
Contains two partitions:
/storage/var System logs
/storage/core Storage for Collected logs.
Hard disk 3
256 MB
First boot only
Hard disk 4
(additional virtual
disk)
190 GB
Storage for collected logs. The capacity from
this disk is added to /storage/core.
Calculate the storage space that is available for log data using the following equation:
/storage/core = hard disk 2 space + hard disk 4 space - system logs space
on hard disk 2
Based on the size of the default and additional virtual disks, the storage core is equal to 440 GB.
/storage/core = 270 GB +
190 GB - 20 GB = 440 GB
Retention = /storage/core – 3% * /storage/core
If /storage/core is 425 GB, vRealize Log Insight can use 413 GB for retention.
Retention = 440 GB - 3% * 440
≈ 427 GB
Configure a retention period of 7 days for the medium-size vRealize Log Insight appliance.
Table 107. Retention Period Design Decision
Decision ID
Design Decision
Design Justification
© 2016 VMware, Inc. All rights reserved.
Page 126 of 130
Design Implication
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
SDDC-OPSLOG-008
Configure vRealize
Log Insight to retain
data for 7 days.
Accommodates logs from 750
syslog sources (250 per node) as
per the SDDC design.
You must add a
VMDK to each
appliance.
Archiving
vRealize Log Insight archives log messages as soon as possible. At the same time, they remain
retained on the virtual appliance until the free local space is almost filled. Data exists on both the
vRealize Log Insight appliance and the archive location for most of the retention period. The archiving
period must be longer than the retention period.
The archive location must be on an NFS version 3 shared storage. The archive location must be
available and must have enough capacity to accommodate the archives.
Apply an archive policy of 90 days for the medium-size vRealize Log Insight appliance. The vRealize
Log Insight appliance will use about 1 TB of shared storage. According to the business compliance
regulations of your organization, these sizes might change.
Table 108. Log Archive Policy Design Decision
Decision
ID
Design Decision
Design
Justification
Design Implication
SDDCOPS-LOG009
Provide 1 TB of NFS
version 3 shared storage
to each vRealize Log
Insight cluster.
Archives logs from
750 syslog
sources.
You must provide an NFS version
3 shared storage in addition to the
data storage for the vRealize Log
Insight cluster.
You must enforce the archive
policy directly on the shared
storage.
3.3.1.10. Alerting
vRealize Log Insight supports alerts that trigger notifications about its health. The following types of
alerts exist in vRealize Log Insight:

System Alerts. vRealize Log Insight generates notifications when an important system event
occurs, for example when the disk space is almost exhausted and vRealize Log Insight must start
deleting or archiving old log files.

Content Pack Alerts. Content packs contain default alerts that can be configured to send
notifications, these alerts are specific to the content pack and are disabled by default.

User-Defined Alerts. Administrators and users can define their own alerts based on data
ingested by vRealize Log Insight.
vRealize Log Insight handles alerts in two ways:

Send an e-mail over SMTP

Send to vRealize Operations Manager
3.3.1.11. SMTP Notification
Enable e-mail notification for alerts in vRealize Log Insight.
Table 109. SMTP Alert Notification Design Decision
Decision ID
Design
Decision
Design Justification
© 2016 VMware, Inc. All rights reserved.
Page 127 of 130
Design Implication
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
SDDC-OPSLOG-010
Enable alerting
over SMTP.
Enables administrators and operators
to receive alerts via email from
vRealize Log Insight.
Requires access to an
external SMTP server.
3.3.1.12. Security and Authentication
Protect the vRealize Log Insight deployment by providing centralized role-based authentication and
secure communication with the other components in the Software-Defined Data Center (SDDC).
Authentication
Enable role-based access control in vRealize Log Insight by using the existing rainpole.local Active
Directory domain.
Table 110. Custom Role-Based User Management Design Decision
Decision ID
Design Decision
Design Justification
Design Implication
SDDCOPS-LOG012
Use Active
Directory for
authentication.
Provides fine-grained role and
privilege-based access for
administrator and operator
roles.
You must provide access
to the Active Directory
from all Log Insight nodes.
Encryption
Replace default self-signed certificates with a CA-signed certificate to provide secure access to the
vRealize Log Insight Web user interface.
Table 111. Custom Certificates Design Decision
Decision
ID
Design Decision
Design Justification
Design
Implication
SDDCOPS-LOG013
Replace the default
self-signed certificates
with a CA-signed
certificate.
Configuring a CA-signed certificate
ensures that all communication to
the externally facing Web UI is
encrypted.
Access to a
Certificate
Authority is
required.
3.3.1.13. Configuration for Collecting Logs
Client applications can send logs to vRealize Log Insight in one of the following ways:

Directly to vRealize Log Insight over the syslog protocol.

By using vRealize Log Insight agents.
Table 112. Direct Log Communication to vRealize Log Insight Design Decisions
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCOPS-LOG014
Configure syslog
sources to send log
data directly to
vRealize Log Insight.
Simplifies the design
implementation for log
sources that are syslog
capable.
You must configure syslog
sources to forward logs to
the vRealize Log Insight
VIP.
© 2016 VMware, Inc. All rights reserved.
Page 128 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide
3.3.1.14. Time Synchronization
Time synchronization is critical for the core functionality of vRealize Log Insight. By default, vRealize
Log Insight synchronizes time with a pre-defined list of public NTP servers.
Configure consistent NTP sources on all systems that send log data (vCenter Server, ESXi, vRealize
Operation Manager). See Time Synchronization in the Planning and Preparation implementation
guide.
Table 113. Time Synchronization Design Decision
Decision
ID
Design Decision
Design
Justification
Design Implication
SDDCOPS-LOG016
Configure consistent NTP sources
on all virtual infrastructure and cloud
management applications for correct
log analysis in vRealize Log Insight.
Guarantees
accurate log
timestamps.
Requires that all
applications
synchronize time to the
same NTP time source.
3.3.1.15. Connectivity in the Cluster
All vRealize Log Insight cluster nodes must be in the same LAN with no firewall or NAT between the
nodes.
External Communication
vRealize Log Insight receives log data over the syslog TCP, syslog TLS/SSL, or syslog UDP
protocols. Use the default syslog UDP protocol because security is already designed at the level of
the management network.
Table 114. Syslog Protocol Design Decision
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCOPS-LOG017
Communicate with the syslog
clients, such as ESXi, vCenter
Server, NSX for vSphere, on the
default UDP syslog port.
Using the default syslog
port simplifies
configuration for all
syslog sources.
If the network
connection is
interrupted, the
syslog traffic is lost.
UDP syslog traffic is
not secure.
3.3.1.16. Event Forwarding between Regions
vRealize Log Insight supports event forwarding to other clusters and standalone instances. While
forwarding events, the vRealize Log Insight instance still ingests, stores and archives events locally.
Event Forwarding Protocol
You forward syslog data in vRealize Log Insight by using the Ingestion API or a native syslog
implementation.
The vRealize Log Insight Ingestion API uses TCP communication. In contrast to syslog, the
forwarding module supports the following features for the Ingestion API:

Forwarding to other vRealize Log Insight instances

Both structured and unstructured data, that is, multi-line messages.

Metadata in the form of tags
© 2016 VMware, Inc. All rights reserved.
Page 129 of 130
VMware Validated Design for Micro-Segmentation Reference Architecture Guide

Client-side compression

Configurable disk-backed queue to save events until the server acknowledges the ingestion.
Table 115. Protocol for Event Forwarding across Regions Design Decision
Decision
ID
Design Decision
Design Justification
Design Implication
SDDCOPS-LOG018
Forward log event
to the other region
by using the
Ingestion API.
Using the forwarding protocol
supports structured and
unstructured data, provides clientside compression, and event
throttling.
You must configure
each region to
forward log data to
the other.
3.3.1.17. Disaster Recovery
Each region is configured to forward log information to the vRealize Log Insight instance in the other
region. As a result, you do not have to configure failover.
© 2016 VMware, Inc. All rights reserved.
Page 130 of 130
Download PDF